GPUs: fan speed in % per card (nvidia-smi fan.speed, hwmon pwm1); 0% shows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

BUGS.md

@@ -752,3 +752,27 @@ before its own login).
     404 — harmless, but a stray.
 12. **Guest logins can open Users and Settings** — by design ("a guest can do everything you
     can"), but worth a second look now that Users includes removing accounts.
+
+# 2026-10-23: the rate obeyed, but not in step
+
+**Graphs and numbers on Top moved on different beats, whatever the rate.** Every machine was
+sampling at exactly the set rate (measured: 500 ms between samples on all five nodes), so the
+sampler was not the fault. Four things in the console were:
+
+1. ✅ **Graphs stepped once a point, not once a tick.** A point is the history window split 300
+   ways — at 5 min that is 1 s — so at 250 or 500 ms the numbers changed two to four times for
+   every move of a graph, and a narrow box (phone, net graph) moved once a *column*, every few
+   seconds. The columns now slide by a fraction of one on every tick (`columnsAt` in
+   `graphcols.ts`), the newest growing in from the right; a finished column still never changes.
+   The per-core strips keep the last 18 ticks rather than stride points.
+2. ✅ **The process table, MHz, CPU watts and every GPU box ran on their own timer** — a
+   `setInterval` on `/api/procs` with an arbitrary phase against the sensor stream, so half the
+   page changed up to a tick after the other half. The table is now read on each sample and the
+   two are committed together: one render per tick for the whole page.
+3. ✅ **GPU clients were walked at most once a second** (`gpuScanEvery`), so per-process GPU% and
+   the engine meters lagged the device's numbers. Walked every sample now.
+4. ✅ **Hidden floors on other live views:** Sensors gallery (1 s), Network live rates (1 s) and
+   speed-test needle (0.4–1 s), Docker live (2 s), ZFS ARC (2 s). All at the rate now.
+
+Not changed: container status sensors are read every 15 s (`docker ps` plus an inspect) and
+filesystem usage every 30 s — slow-moving facts behind an exec or a statfs of every mount.

PLAN.md

@@ -16,7 +16,7 @@ stub tab or a roadmap item.
 | Service | `hamlab.service`, `User=core`, `Restart=always`, binary at `/usr/local/bin/hamlab` | Same pattern as `calorie-counter.service`. Runs as `core` so `sudo` works in the terminal without a password, as on the SSH path. |
 | Listen | `10.0.0.234:8100` (free) | NPM is a container, so loopback-only won't do. Firewalling to NPM is a later hardening step. |
 | Data | `/home/core/apps/hamlab/` — `config.yaml`, `state.db` (SQLite) | `apps/` is the nightly rsync target to `grid`, so it's backed up for free. |
-| Ingress | NPM proxy host `hamlab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
+| Ingress | NPM proxy host `lab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
 | Repo | Gitea `mitchell/hamlab`, private, `main` | Created 2026-09-17. |
 | Guest logins | A **guest** is an account like any other, with `guest: true` and an expiry: `login`, sessions and prefs need no special case, and **it may do everything an account may do**. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. | Built 2026-09-27. A read-only version was built first and **cut the same day**: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is *an account*: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to. |
 | Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
@@ -37,7 +37,7 @@ stub tab or a roadmap item.
 │ Cron         │                                               │
 │ …            │                                               │
 ├──────────────┴───────────────────────────────────────────────┤
-│ footer: v0.1.0 · up 12d · hamlab.bigham.dev                   │
+│ footer: v0.1.0 · up 12d · lab.bigham.dev                   │
 └──────────────────────────────────────────────────────────────┘
 ```
 
@@ -116,7 +116,7 @@ hamlab/
 - Mobile drawer, key bar, fit-on-rotate.
 - `manifest.webmanifest`, icons (Ham logo on black, same recipe as HamTV), service
   worker caching the app shell only (never `/api`).
-- NPM proxy host for `hamlab.bigham.dev`; verify WS through NPM; force-SSL/HSTS/HTTP2.
+- NPM proxy host for `lab.bigham.dev`; verify WS through NPM; force-SSL/HSTS/HTTP2.
 - Stub groups for Services / Cron / ZFS / Dashboard.
 
 **Done when:** from the phone, open the PWA, log in, open two terminals, run
@@ -417,39 +417,9 @@ something long in one, switch away and back, refresh — it's all still there.
   snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it
   had taken left untouched and fully readable here — which is what adopting its naming
   bought. See `hamlet/docs/hosts.md`.
-- **KVM tab** — **shipped 2026-09-24** (`internal/api/kvm.go`, `KVM.tsx`, `config.KVM`).
-  Paste a device's address; its own web page appears in the tab. **Nothing in it knows
-  anything about any particular KVM** — the first device here is a Sipeed NanoKVM on `grid`
-  at `10.0.0.217`, but none of that is in the code.
-  **Why it proxies rather than framing the device directly.** Ham Lab is served over TLS and
-  a KVM usually is not, so a bare `http://…` iframe is blocked as mixed content before it
-  loads. Serving the device from Ham Lab's own origin (`/kvm/<id>/…`) fixes that, puts it
-  behind **Ham Lab's** login rather than a second one, and makes the iframe same-origin.
-  **The general problem, and the answer.** A device UI asks for its assets by absolute path
-  — `/assets/app.js`, `/api/whatever` — which no path prefix can rewrite away, and rewriting
-  bodies per device would not be general. So the prefix is only the entry point: any request
-  Ham Lab does not serve itself is routed to whichever KVM the asking page belongs to, read
-  from the **Referer**, falling back to a **cookie** when there is none (a WebSocket upgrade).
-  Ham Lab's own routes always match first, so a device can never shadow the console, and the
-  fallback is gated on a live session. The device never receives Ham Lab's session cookie,
-  and its `X-Frame-Options`/CSP/HSTS are dropped on the way out since Ham Lab is the origin
-  now. Redirects to the device's root are rewritten back through the prefix.
-  **One catch found in testing:** Ham Lab's own `frame-ancestors 'none'` forbids framing even
-  from the same origin, so `/kvm/` paths send `'self'` instead. Without that the iframe was
-  refused before it loaded.
-  **The keyboard** needs less than it sounds: a page cannot forward keys into an iframe and
-  does not need to, since one with focus receives them directly. Fullscreen plus
-  `navigator.keyboard.lock()` is what takes Escape, Tab and Ctrl-W back from the browser;
-  that is Chromium-only, and the tab says which version you are getting rather than
-  pretending.
-  **One KVM per machine, and the tab follows the machine switcher** (corrected the same day:
-  the first cut was a hub-side list you pointed at machines, which had it backwards). Switch
-  to `grid` and you get grid's; adding a second for a machine replaces the first. The
-  addresses are still kept **on the hub** rather than on each machine, because a KVM earns
-  its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub
-  proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at
-  all. Not done: holding the device's own login on the hub, so its page does not ask for a
-  second password.
+- **KVM tab** — shipped 2026-09-24, **removed 2026-11-09** at Mitchell's call: not needed —
+  the KVMs are reached over WireGuard (see `hamlet/docs/hosts.md`). It proxied a device's web
+  page through Ham Lab's origin; the code is in git history before that date.
 - **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
   Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
   type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
@@ -810,6 +780,61 @@ something long in one, switch away and back, refresh — it's all still there.
   - Tested (`TestSeekRestartsTheEncode`): a 64 s 1080p clip, a request for segment 12 right
     after the start runs from 48 s, the gap is filled, all 16 segments joined behind the one
     init decode with no hole.
+- **The player is Ham Share's; a Transcode device setting** — **2026-10-23** (v0.1.0-226,
+  `internal/media/{encode,play}.go`, `internal/api/play.go`, `Player.tsx`, `vr.ts`,
+  `Settings.tsx`). Asked for as "redo the HamLab one with the one you sent … a 'Transcode
+  Device' setting … the A380 on Grid selected … other options being the other transcode
+  devices like the iGPUs on the various CPUs that have HamLab installed". **Replaces** the
+  previous two entries' design: no codec/size picker, no single-quality re-encode, no
+  repackaging; the old streamer (`hls.go`) is gone.
+  - **YouTube's way:** Auto (hls.js reads the connection), any quality by hand, or Original
+    (the file as it is, `/api/files/read`, falling back to Auto when the browser can't). A
+    ladder by size (`Ladder`): flat 2160p/1440p/1080p/720p/480p by the short side, plus the
+    file's own size on top when it sits between steps (a 2292×960 capture gets 960p); 360
+    files by width, named as YouTube does (2160s 3840 … 720s). 720 and down at ≤30 fps.
+    H.264 High + AAC only.
+  - **Pieces made when asked for**, 3 s MPEG-TS, every quality's playlist written whole up
+    front. One ffmpeg per play: it starts at the piece asked, runs ahead, and is paused
+    (SIGSTOP; a root run through `run-ffmpeg`, which now reads `stop`/`cont` on stdin) 10
+    pieces ahead; a piece more than 3 past what's made, or at another quality, starts a new
+    run there. Keyframes only on the 3 s marks and every run's times offset alike
+    (`-output_ts_offset start+10`), so pieces from any run and quality join (`TestPieces`
+    checks it; `SRC=… DEVICE=… go test ./internal/media -run Pieces` on a machine with a GPU).
+    At most 3 runs a machine, the one asked of least lately ended for a fourth; 2 GiB of
+    pieces a play; forgotten after 30 min.
+  - **hls.js is patched** (`PatientAbr`): the wait for a piece being made isn't counted
+    against the connection, and a piece isn't abandoned before its first byte. Without it
+    Auto dropped to the bottom after every seek.
+  - **Transcode device** (Settings, hub-side, `store.Transcode`): every machine's devices
+    asked live (`GET /api/files/play-devices` on each), unusable ones greyed with the reason
+    (hamtv's UHD 600: the render permission; mac and gate: no ffmpeg). Set to **grid's Arc
+    A380**. "The machine the file is on" is the other choice. `POST /api/play` on the hub
+    opens the play where the setting says; **a machine transcoding another's file reads it
+    through the hub** at `/api/play-src/<token>` — a random token good for that one file, 12
+    h from its last use, ranges passed through to the file's machine (ffmpeg reconnects).
+    If that machine fails, the file's own makes it and the player says so.
+  - Recipes per device: same-card decode first, then CPU decode + upload, then libx264 as
+    the fallback whatever was asked. A card without `vainfo` (core) is tried anyway: core's
+    UHD encodes. **Rotation:** the card path turns a phone's picture itself
+    (`transpose_vaapi`, `-noautorotate`; ffmpeg's autorotate never applied to VAAPI frames,
+    so iPhone clips came out sideways). **HLG** is shown as it is (tonemap_vaapi refuses it);
+    HDR10 is tone-mapped on the card, else zscale/hable before the upload.
+  - 360 files: the look-around view, zoom 8°–300° (ordinary lens into stereographic, the
+    "little planet"), drag/pinch/wheel, phone motion. Full screen is the element's, or the
+    whole window on an iPhone.
+  - **Video previews follow the setting too** (same day, asked for as "it should use the
+    same video generation device"). On the device's own machine the hub says which device
+    by header (`X-Hamlab-Device`); for a file elsewhere the hub asks the file's machine's
+    cache first (`?cached=1`), else has the device's machine make it from the hub's one-file
+    URL (`POST /api/files/thumb-from`) and keeps it in the file's machine's cache (`PUT
+    /api/files/thumb`, refused if the file changed since its listing), so each is made once.
+    A failure is kept as a failure. Unreachable → made where the file is, as before. Images
+    and audio covers stay on the file's machine's CPU (a JPEG is cheaper there than a GPU
+    round trip). Measured: a `hamtv` video previewed on grid's A380 in 0.22 s, then 2 ms
+    from hamtv's cache. The previews button's tooltip names the device.
+  - Measured (`web/scripts/e2e-player.mjs`, Chromium): an iPhone HLG clip on `core` made on
+    grid's A380 — 3.2 s to play, a seek 1.0 s, a quality change 1.5 s; the 8K 360 Disney day on
+    grid — 3.1 s, 2160s on Auto, a seek 4.9 s (10 s source keyframes).
 - **Files edge sweep** — **2026-10-11** (v0.1.0-214), asked for as "a full bug sweep of
   the Files section, every example and edge case". Every handler and both UI files read
   through with odd inputs in mind, then a battery of ~60 requests against the live hub
@@ -946,6 +971,11 @@ what it has moved.
 
 ## To do
 
+- **Handle `fuse.mergerfs` in the Storage view** (2026-10-25). `grid` has a mergerfs pool at
+  `/srv/tank` (the NVMe landing area + `/tank`; `hamlet`'s `docs/tank-landing-plan.md`).
+  `fuse.mergerfs` isn't in `remoteFS` or `pseudoFS` (`internal/metrics/metrics.go`), so it shows
+  as a local filesystem whose size is `tank` and the NVMe added together. That counts both twice
+  in any storage total. Show it, labelled as a pool of its branches, and leave it out of totals.
 - **Grow the Files tab** — asked for 2026-09-23 as two items (Files polish, and deprecating
   copyparty), merged 2026-09-24. **copyparty was retired 2026-10-11** at Mitchell's call,
   ahead of these gaps rather than after them: service, files and NPM host all gone, with a

PORTABILITY.md

@@ -20,10 +20,10 @@ default or phones home to this lab · **C** cosmetic (placeholder text, comments
 | 1.1 | **B** | `internal/selfupdate/selfupdate.go:36` `DefaultURL = "https://git.bigham.dev/api/v1/repos/mitchell/hamlab"` | Every install with no `update.url` checks **this** Gitea 90 s after boot and daily (`internal/api/update.go:1951, 2016`). A stranger's install phones home here by default. | Default should be **off** (or a build-time `-X` variable set at release). Installer asks or `install.sh` passes `HAMLAB_URL` through into `update.url`. |
 | 1.2 | **B** | `install.sh:4, 27` `HAMLAB_URL` default; `README.md:23-25, 31` | The curl one-liner and the install source are this Gitea. | Default comes from the release that served the script (`@URL@` filled by `make release`), or is required. |
 | 1.3 | **B** | `scripts/gitea-release.sh:14` `REPO` default | Publishing goes to `mitchell/hamlab` on this Gitea. | Require `HAMLAB_REPO`; no default. |
-| 1.4 | **C** | `README.md:100` "On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager" | Lab deployment fact in the product README. | Move to `hamlet/docs/hosts.md`. |
+| 1.4 | **C** | `README.md:100` "On `core` it is public at `https://lab.bigham.dev` behind Nginx Proxy Manager" | Lab deployment fact in the product README. | Move to `hamlet/docs/hosts.md`. |
 | 1.5 | **C** | `examples/external-demo/index.html:8` `base: 'http://10.0.0.234:8100'` | Demo config names the hub's LAN IP. | `http://<hamlab-host>:8100`. |
-| 1.6 | **C** | `SENSORS.md:149` Prometheus target `10.0.0.234:8100`; `:187` `&machine=grid`; `:196-208` Glance examples at `hamlab.bigham.dev`, `zfs.tank.used_pct`, `/api/nodes/grid/...`; `:218-220` `grid,wyse`, `{"core":…,"grid":…,"mac":…}` | Docs use real host names, pool and domain. | Generic placeholders (`<host>`, `<machine>`, `<pool>`). |
-| 1.7 | **C** | `PLAN.md:3-4` "bare-metal Debian 13 host. First target: `core` (10.0.0.234)"; decisions table rows (Listen `10.0.0.234:8100`, Data `/home/core/apps/hamlab/`, Ingress NPM/`hamlab.bigham.dev`, Repo) | Product plan is written as this lab's plan. | Keep as history, or split lab facts out to `hamlet`. The repo is public. |
+| 1.6 | **C** | `SENSORS.md:149` Prometheus target `10.0.0.234:8100`; `:187` `&machine=grid`; `:196-208` Glance examples at `lab.bigham.dev`, `zfs.tank.used_pct`, `/api/nodes/grid/...`; `:218-220` `grid,wyse`, `{"core":…,"grid":…,"mac":…}` | Docs use real host names, pool and domain. | Generic placeholders (`<host>`, `<machine>`, `<pool>`). |
+| 1.7 | **C** | `PLAN.md:3-4` "bare-metal Debian 13 host. First target: `core` (10.0.0.234)"; decisions table rows (Listen `10.0.0.234:8100`, Data `/home/core/apps/hamlab/`, Ingress NPM/`lab.bigham.dev`, Repo) | Product plan is written as this lab's plan. | Keep as history, or split lab facts out to `hamlet`. The repo is public. |
 | 1.9 | **C** | `BUGS.md` throughout | Bug bashes narrate `core`, `grid`, `wyse`, `hamtv`, `gate`, the Mac. | History; fine to leave, but it is public. |
 | 1.10 | **C** | `cmd/hamlab/main.go:1` "Proxmox-style web console for a bare-metal Debian host"; `internal/hardware/hardware.go:5` "what Debian and its kin already have"; `internal/hardware/hardware.go:29` `// "GMKtec NucBox G3"`; `internal/metrics/metrics.go:1358` "did, on grid, 2026-10-02"; `internal/api/server.go:1382-1383` `["grid", …]`, `?machines=grid,wyse`; `internal/api/stacks.go:10` `/home/core/apps/dockhand/stacks/…`; `internal/api/dockersched.go:5` "unbound, nginx and pihole"; `internal/api/zfs.go:5` "grid already runs zfSnap"; `internal/storage/snapshots.go:6` "grid has run zfSnap v1.11.1"; `internal/api/guests.go:8` "Mitchell, 2026-09-27"; `internal/network/network.go:306` "the public address here is a DHCP lease"; `internal/metrics/gpu.go:17` "Asahi, today"; `internal/metrics/pciids.go:4-6` "the router's GPU"; `web/src/Top.tsx:522, 576`; `web/src/hashsel.ts:1-9`; `web/src/Embed.tsx:3` `&machine=grid` | Comments explain decisions by naming lab hosts. | Harmless, but rewrite the ones in package doc comments so a reader of the source doesn't need the lab to understand the code. |
 
@@ -32,7 +32,6 @@ default or phones home to this lab · **C** cosmetic (placeholder text, comments
 | # | Sev | Where | Text | General form |
 |---|---|---|---|---|
 | 2.1 | **C** | `web/src/Machines.tsx:116-118` | `placeholder="grid"`, `"https://10.0.0.214:8100"`, `"core"` | `nas`, `https://192.0.2.10:8100`, `admin` or the hub's own user. |
-| 2.2 | **C** | `web/src/KVM.tsx:166` | `placeholder="10.0.0.217"` (the NanoKVM on `grid`) | `192.0.2.20` or `kvm.local`. |
 | 2.3 | **C** | `web/src/Settings.tsx:242` | `placeholder="cpu.*, temp.*, zfs.tank.*"` | `cpu.*, temp.*, fs.*`. |
 | 2.4 | **C** | `web/src/Settings.tsx:225` | read-only token default name `'glance'` | Empty, or `dashboard`. |
 | 2.5 | **C** | `web/src/Alerts.tsx:38, 40` | `"Tank nearly full"`, `"zfs.tank.used_pct"` | `"Disk nearly full"`, `"fs.root.used_pct"`. |
@@ -146,7 +145,6 @@ Listed so the sweep is known to be complete, not because they need work.
 - `internal/nodes`, `internal/tlsauto`: hub ↔ node pairing, pinned certs, proxy, push.
 - `internal/sensors`, alerts, widgets, embeds, Prometheus: machine-agnostic by design.
 - `internal/files`, `internal/term`, `internal/tasks`, `internal/auth`, `internal/store`.
-- `internal/api/kvm.go`: "nothing in it knows anything about any particular KVM" — verified.
 - `internal/compose` (modulo 3.5).
 - The React UI: machine-switcher model, hash routing, per-account prefs, themes.
 

README.md

@@ -97,5 +97,5 @@ page flags machines running an older version and offers the same push per machin
 Mixed architectures are fine: `make deploy` also cross-compiles every other architecture,
 and the hub sends each machine the build for its own.
 
-- On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager.
+- On `core` it is public at `https://lab.bigham.dev` behind Nginx Proxy Manager.
 - Installable as a PWA.

SENSORS.md

@@ -169,6 +169,9 @@ container.<name>.down                           should be up and isn't: Docker c
                                                 restart it — *not* a container someone stopped
 updates.pending|security|reboot_required        every 5 min
 zfs.<pool>.health|healthy|size|used|used_pct|frag   every 1 min
+zfs.write.dirty|limit|delayed                       the RAM write buffer (all pools) and writes the throttle is slowing
+zfs.<pool>.write_bps|logical_bps                    bytes/s synced to disk, and written by programs
+zfs.<pool>.landing.pending_bytes|pending_files|used_pct|in_bps|drain_bps|mover|queue   a landing tier, where one fronts the pool
 smart.<dev>.passed|warnings|power_on|wear       every 10 min, never waking a sleeping disk
 ```
 
@@ -200,13 +203,13 @@ facts, never live. An `iframe` of the embed is live.
 ```yaml
 - type: iframe
   title: core CPU
-  source: https://hamlab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
+  source: https://lab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
   height: 130
 
 - type: custom-api
   title: grid
   cache: 1m
-  url: https://hamlab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
+  url: https://lab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
   headers: { Authorization: Bearer <read-only token> }
   template: |
     <div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>

cmd/hamlab/main.go

@@ -63,6 +63,7 @@ func main() {
   passwd [user]        set a user's password (makes the account if it is new; reads stdin when piped)
   users                list the accounts
   zfs-snapshot <id>    run one ZFS snapshot schedule (what its cron line calls)
+  landing-verify <new|all|id>  verify a landing tier against its mover's ledger (a schedule's id is what its cron line calls)
   docker-update <id>   run one image-update schedule (what its cron line calls)
   user-rm <user>       remove an account and sign it out everywhere
   check                verify the config and state load, then exit
@@ -150,6 +151,10 @@ func main() {
 		if err := zfsSnapshot(cfg, flag.Arg(1)); err != nil {
 			log.Fatal(err)
 		}
+	case "landing-verify":
+		if err := landingVerify(cfg, flag.Arg(1)); err != nil {
+			log.Fatal(err)
+		}
 	case "users":
 		migrateUsers(cfg)
 		if len(cfg.Users) == 0 {
@@ -254,6 +259,81 @@ func dockerUpdateJob(cfg *config.Config, id string) error {
 //
 // It deliberately does not write state.json. The daemon owns that file and would clobber
 // anything written from out here; the outcome goes beside it instead, in zfs-runs/.
+// landingVerify runs one verify pass over every landing tier on this machine: by hand with a
+// scope ("new" or "all"), or as a schedule by its id, which is what the cron line passes.
+func landingVerify(cfg *config.Config, arg string) error {
+	scope, id := arg, "manual"
+	if arg != "new" && arg != "all" {
+		if strings.TrimSpace(arg) == "" {
+			return errors.New("usage: hamlab landing-verify <new|all|schedule-id>")
+		}
+		st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
+		if err != nil {
+			return err
+		}
+		var raw json.RawMessage
+		st.View(func(s *store.State) { raw = s.ZFS })
+		var z struct {
+			Verify []storage.VerifyPolicy `json:"verify"`
+		}
+		if len(raw) > 0 {
+			if err := json.Unmarshal(raw, &z); err != nil {
+				return fmt.Errorf("reading the schedules: %w", err)
+			}
+		}
+		found := false
+		for _, p := range z.Verify {
+			if p.ID == arg {
+				scope, id, found = p.Scope, p.ID, true
+			}
+		}
+		if !found {
+			return fmt.Errorf("no verify schedule %q — it may have been removed; this cron line is stale", arg)
+		}
+	}
+	dir := cfg.DataDir()
+	unlock, err := storage.LockVerify(dir)
+	if err != nil {
+		return err
+	}
+	defer unlock()
+	defer storage.ClearVerifyProgress(dir)
+	targets := storage.LandingTargets()
+	if len(targets) == 0 {
+		r := storage.VerifyResult{ID: id, Scope: scope, Started: time.Now(), Finished: time.Now(), Issues: []storage.VerifyIssue{},
+			Error: "no landing mover is reporting (nothing in " + storage.MoverStatusDir + "), so there is nothing to verify"}
+		_ = storage.RecordVerify(dir, storage.LandingTarget{}, r)
+		return errors.New(r.Error)
+	}
+	var firstErr error
+	for _, t := range targets {
+		r := storage.Verify(t, id, scope, storage.VerifyCursor(dir, t), func(p storage.VerifyProgress) { storage.WriteVerifyProgress(dir, p) })
+		if err := storage.RecordVerify(dir, t, r); err != nil {
+			fmt.Fprintln(os.Stderr, "recording the pass:", err)
+		}
+		fmt.Printf("%s -> %s (%s): %d checked, %d match, %d mismatch, %d gone, %d unreadable, %d changed since, %d waiting, %d unhashed, %s read in %s\n",
+			t.Src, t.Dst, scope, r.Checked, r.Matched, r.Mismatched, r.Gone, r.Unreadable, r.Changed, r.Waiting, r.Unhashed,
+			humanBytes(r.BytesRead), r.Finished.Sub(r.Started).Round(time.Second))
+		if r.Error != "" && firstErr == nil {
+			firstErr = errors.New(r.Error)
+		}
+	}
+	return firstErr
+}
+
+func humanBytes(n int64) string {
+	const u = 1024
+	if n < u {
+		return fmt.Sprintf("%d B", n)
+	}
+	div, exp := int64(u), 0
+	for v := n / u; v >= u; v /= u {
+		div *= u
+		exp++
+	}
+	return fmt.Sprintf("%.1f %ciB", float64(n)/float64(div), "KMGTPE"[exp])
+}
+
 func zfsSnapshot(cfg *config.Config, id string) error {
 	if strings.TrimSpace(id) == "" {
 		return errors.New("usage: hamlab zfs-snapshot <schedule-id>")

internal/api/kvm.go

@@ -1,420 +0,0 @@
-package api
-
-// IP-KVMs: a device's own web page, shown inside Ham Lab.
-//
-// The whole difficulty is that Ham Lab is served over TLS and a KVM usually is not, so a
-// browser refuses to put the device's page in an iframe — mixed content, blocked before
-// anything loads. The fix is to serve the device from Ham Lab's own origin, which also
-// means it inherits Ham Lab's login instead of needing a second one, and makes the iframe
-// same-origin.
-//
-// Proxying an arbitrary web app under a path prefix is the hard part, and nothing here
-// knows anything about any particular device. A device UI asks for its assets by absolute
-// path — "/assets/app.js", "/api/whatever" — which a prefix cannot rewrite away. So the
-// prefix is only the entry point: any request Ham Lab does not serve itself is routed to
-// whichever KVM the asking page belongs to, worked out from the Referer, and from a cookie
-// when there is no Referer (a WebSocket upgrade, say). Ham Lab's own routes always win, so
-// a device can never shadow the console.
-
-import (
-	"crypto/rand"
-	"encoding/hex"
-	"encoding/json"
-	"errors"
-	"io"
-	"log"
-	"net/http"
-	"net/http/httputil"
-	"net/url"
-	"strings"
-	"sync"
-	"time"
-
-	"hamlab/internal/config"
-	"hamlab/internal/store"
-)
-
-const kvmPrefix = "/kvm/"
-const kvmCookie = "hamlab_kvm"
-
-type kvmProxies struct {
-	mu sync.Mutex
-	m  map[string]*httputil.ReverseProxy
-}
-
-func (s *Server) kvm(id string) (config.KVM, bool) {
-	var found config.KVM
-	ok := false
-	s.cfg.View(func(c *config.Config) {
-		for _, k := range c.KVMs {
-			if k.ID == id {
-				found, ok = k, true
-			}
-		}
-	})
-	return found, ok
-}
-
-// kvmProxy is the reverse proxy for one device, made once and kept.
-func (s *Server) kvmProxy(k config.KVM) (*httputil.ReverseProxy, error) {
-	s.kvms.mu.Lock()
-	defer s.kvms.mu.Unlock()
-	if s.kvms.m == nil {
-		s.kvms.m = map[string]*httputil.ReverseProxy{}
-	}
-	if p, ok := s.kvms.m[k.ID]; ok {
-		return p, nil
-	}
-	target, err := url.Parse(k.URL)
-	if err != nil || target.Host == "" {
-		return nil, errors.New("this KVM's address is not a URL")
-	}
-	prefix := kvmPrefix + k.ID
-	p := &httputil.ReverseProxy{
-		Director: func(r *http.Request) {
-			r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
-			// Entry through the prefix keeps the path after it; anything else arrived by its
-			// own absolute path and is passed through untouched.
-			if rest, ok := strings.CutPrefix(r.URL.Path, prefix); ok {
-				if rest == "" {
-					rest = "/"
-				}
-				r.URL.Path = rest
-			}
-			r.URL.RawPath = ""
-			r.Host = target.Host
-			// Whether the console itself is on TLS: the response rewriter needs to know, and by
-			// then only the outbound request is in hand.
-			if secure(r) {
-				r.Header.Set(kvmSecureHeader, "1")
-			} else {
-				r.Header.Del(kvmSecureHeader)
-			}
-			// The device gets its own host and none of Ham Lab's session.
-			r.Header.Del("Origin")
-			stripCookie(r, cookieName, kvmCookie)
-			// What is left is the device's own login. A browser that has it teaches the hub; one
-			// that has not is lent what the hub remembers, so the device sees a signed-in client.
-			if own := r.Cookies(); len(own) > 0 {
-				s.rememberKVMCookies(k.ID, own)
-			} else {
-				for name, val := range s.kvmCookies(k.ID) {
-					r.AddCookie(&http.Cookie{Name: name, Value: val})
-				}
-			}
-		},
-		FlushInterval: -1, // a video stream must not be buffered
-		ModifyResponse: func(resp *http.Response) error {
-			// The device's own framing and transport policy would stop it being embedded, or
-			// send a browser off to plain HTTP. Ham Lab is the origin now, and says what it
-			// serves; its own headers are added on the way out.
-			for _, h := range []string{"X-Frame-Options", "Content-Security-Policy", "Content-Security-Policy-Report-Only", "Strict-Transport-Security", "Referrer-Policy"} {
-				resp.Header.Del(h)
-			}
-			// A cookie the device sets by header is kept for Ham Lab's origin — so its Domain
-			// (the device's) goes, Secure follows the console rather than the device, and the
-			// hub remembers it too.
-			if set := resp.Cookies(); len(set) > 0 {
-				https := resp.Request != nil && resp.Request.Header.Get(kvmSecureHeader) == "1"
-				resp.Header.Del("Set-Cookie")
-				for _, c := range set {
-					c.Domain = ""
-					if c.Path == "" {
-						c.Path = "/"
-					}
-					c.Secure = c.Secure && https
-					if v := c.String(); v != "" {
-						resp.Header.Add("Set-Cookie", v)
-					}
-				}
-				s.rememberKVMCookies(k.ID, set)
-			}
-			// A redirect to the device's own root has to come back through the prefix.
-			if loc := resp.Header.Get("Location"); loc != "" {
-				if u, err := url.Parse(loc); err == nil && (u.Host == "" || u.Host == target.Host) && strings.HasPrefix(u.Path, "/") && !strings.HasPrefix(u.Path, prefix) {
-					u.Scheme, u.Host = "", ""
-					u.Path = prefix + u.Path
-					resp.Header.Set("Location", u.String())
-				}
-			}
-			return nil
-		},
-		ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
-			w.Header().Set("Content-Type", "application/json")
-			w.WriteHeader(http.StatusBadGateway)
-			_ = json.NewEncoder(w).Encode(map[string]string{"error": k.Name + " unreachable: " + err.Error()})
-		},
-	}
-	s.kvms.m[k.ID] = p
-	return p, nil
-}
-
-// kvmSecureHeader rides on the outbound request to say whether the console was reached
-// over TLS; the device ignores it.
-const kvmSecureHeader = "X-Hamlab-Secure"
-
-// kvmCookies is what the hub remembers of a device's own cookies — its login, in practice.
-func (s *Server) kvmCookies(id string) map[string]string {
-	var out map[string]string
-	s.store.View(func(st *store.State) {
-		if m := st.KVMCookies[id]; len(m) > 0 {
-			out = make(map[string]string, len(m))
-			for k, v := range m {
-				out[k] = v
-			}
-		}
-	})
-	return out
-}
-
-// rememberKVMCookies keeps the cookies a browser (or the device's own Set-Cookie) carries
-// for a KVM, so that a browser without them can be lent them. A cookie emptied or expired
-// by the device is forgotten. Written only when something changed: this runs on every
-// proxied request.
-func (s *Server) rememberKVMCookies(id string, cs []*http.Cookie) {
-	have := s.kvmCookies(id)
-	changed := false
-	for _, c := range cs {
-		if c.Name == "" || c.Name == cookieName || c.Name == kvmCookie {
-			continue
-		}
-		gone := c.Value == "" || c.MaxAge < 0 || (!c.Expires.IsZero() && c.Expires.Before(time.Now()))
-		if gone {
-			if _, ok := have[c.Name]; ok {
-				changed = true
-			}
-			continue
-		}
-		if have[c.Name] != c.Value {
-			changed = true
-		}
-	}
-	if !changed {
-		return
-	}
-	_ = s.store.Update(func(st *store.State) {
-		if st.KVMCookies == nil {
-			st.KVMCookies = map[string]map[string]string{}
-		}
-		m := st.KVMCookies[id]
-		if m == nil {
-			m = map[string]string{}
-		}
-		for _, c := range cs {
-			if c.Name == "" || c.Name == cookieName || c.Name == kvmCookie {
-				continue
-			}
-			if c.Value == "" || c.MaxAge < 0 || (!c.Expires.IsZero() && c.Expires.Before(time.Now())) {
-				delete(m, c.Name)
-			} else {
-				m[c.Name] = c.Value
-			}
-		}
-		if len(m) == 0 {
-			delete(st.KVMCookies, id)
-		} else {
-			st.KVMCookies[id] = m
-		}
-	})
-}
-
-// forgetKVMCookies drops a device's remembered login — when the device is removed.
-func (s *Server) forgetKVMCookies(id string) {
-	_ = s.store.Update(func(st *store.State) { delete(st.KVMCookies, id) })
-}
-
-// lendKVMCookies hands the browser the device cookies the hub remembers and it lacks, as
-// the device's own page would have set them: readable by its scripts, for Ham Lab's whole
-// origin, kept for half a year. The device's SPA checks for its token client-side before it
-// ever asks the server, so lending it only on the wire would still land on the login form.
-func (s *Server) lendKVMCookies(w http.ResponseWriter, r *http.Request, id string) {
-	for name, val := range s.kvmCookies(id) {
-		if _, err := r.Cookie(name); err == nil {
-			continue
-		}
-		http.SetCookie(w, &http.Cookie{Name: name, Value: val, Path: "/", MaxAge: 180 * 24 * 3600, Secure: secure(r), SameSite: http.SameSiteLaxMode})
-	}
-}
-
-// stripCookie removes named cookies from a request, so Ham Lab's session is never handed to
-// a device and the device never sees which KVM the browser is looking at.
-func stripCookie(r *http.Request, names ...string) {
-	cs := r.Cookies()
-	kept := cs[:0]
-	for _, c := range cs {
-		drop := false
-		for _, n := range names {
-			if c.Name == n {
-				drop = true
-			}
-		}
-		if !drop {
-			kept = append(kept, c)
-		}
-	}
-	r.Header.Del("Cookie")
-	for _, c := range kept {
-		r.AddCookie(c)
-	}
-}
-
-func (s *Server) forgetKVMProxy(id string) {
-	s.kvms.mu.Lock()
-	delete(s.kvms.m, id)
-	s.kvms.mu.Unlock()
-}
-
-// serveKVM is the entry point: /kvm/<id>/… . It also drops a cookie naming the device, so
-// that requests the page makes later by absolute path can be traced back to it.
-func (s *Server) serveKVM(w http.ResponseWriter, r *http.Request) {
-	id := r.PathValue("id")
-	k, ok := s.kvm(id)
-	if !ok {
-		jsonErr(w, 404, "no such KVM")
-		return
-	}
-	p, err := s.kvmProxy(k)
-	if err != nil {
-		jsonErr(w, 400, err.Error())
-		return
-	}
-	http.SetCookie(w, &http.Cookie{Name: kvmCookie, Value: id, Path: "/", HttpOnly: true, Secure: secure(r), SameSite: http.SameSiteLaxMode})
-	s.lendKVMCookies(w, r, id)
-	p.ServeHTTP(w, r)
-}
-
-// kvmFor works out which device a request belongs to when it arrived by absolute path
-// rather than through the prefix: the Referer names the page that asked, and a cookie
-// covers what has no Referer at all. Nothing device-specific: it never has to know which
-// paths a given KVM uses.
-func (s *Server) kvmFor(r *http.Request) (config.KVM, bool) {
-	if ref := r.Header.Get("Referer"); ref != "" {
-		if u, err := url.Parse(ref); err == nil {
-			if rest, ok := strings.CutPrefix(u.Path, kvmPrefix); ok {
-				if id, _, _ := strings.Cut(rest, "/"); id != "" {
-					if k, ok := s.kvm(id); ok {
-						return k, true
-					}
-				}
-			}
-		}
-		// asked for by a page that is not a KVM: not ours to answer
-		return config.KVM{}, false
-	}
-	if c, err := r.Cookie(kvmCookie); err == nil && c.Value != "" {
-		return s.kvm(c.Value)
-	}
-	return config.KVM{}, false
-}
-
-// kvmFallback answers a request Ham Lab does not serve itself, when it came from inside a
-// KVM's page. It returns false when the request has nothing to do with a KVM, and the
-// caller carries on with its own 404.
-func (s *Server) kvmFallback(w http.ResponseWriter, r *http.Request) bool {
-	k, ok := s.kvmFor(r)
-	if !ok {
-		return false
-	}
-	p, err := s.kvmProxy(k)
-	if err != nil {
-		return false
-	}
-	p.ServeHTTP(w, r)
-	return true
-}
-
-func machineLabel(m string) string {
-	if m == "" {
-		return "this host"
-	}
-	return m
-}
-
-// ---- the list ----
-
-func (s *Server) listKVMs(w http.ResponseWriter, _ *http.Request) {
-	out := []config.KVM{}
-	s.cfg.View(func(c *config.Config) { out = append(out, c.KVMs...) })
-	jsonOK(w, 200, out)
-}
-
-func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
-	var body struct{ Name, URL, Machine string }
-	if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
-		jsonErr(w, 400, "bad request")
-		return
-	}
-	raw := strings.TrimSpace(body.URL)
-	if raw != "" && !strings.Contains(raw, "://") {
-		raw = "http://" + raw // what you would type: an address, not a scheme
-	}
-	u, err := url.Parse(raw)
-	if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
-		jsonErr(w, 400, "address: http(s)://host, or just the host")
-		return
-	}
-	name := strings.TrimSpace(body.Name)
-	if name == "" {
-		name = u.Host
-	}
-	if len(name) > maxNameLen {
-		jsonErr(w, 400, "that name is too long")
-		return
-	}
-	b := make([]byte, 6)
-	if _, err := rand.Read(b); err != nil {
-		jsonErr(w, 500, err.Error())
-		return
-	}
-	machine := strings.TrimSpace(body.Machine)
-	k := config.KVM{ID: hex.EncodeToString(b), Name: name, URL: u.Scheme + "://" + u.Host + strings.TrimSuffix(u.Path, "/"), Machine: machine, Added: time.Now()}
-	// One per machine: a second one for the same host replaces the first rather than sitting
-	// beside it, since the tab shows the machine you are looking at.
-	var replaced string
-	if err := s.cfg.Update(func(c *config.Config) {
-		kept := c.KVMs[:0]
-		for _, old := range c.KVMs {
-			if old.Machine == machine {
-				replaced = old.ID
-				continue
-			}
-			kept = append(kept, old)
-		}
-		c.KVMs = append(kept, k)
-	}); err != nil {
-		jsonErr(w, 500, err.Error())
-		return
-	}
-	if replaced != "" {
-		s.forgetKVMProxy(replaced)
-	}
-	log.Printf("kvm %q added for %s (%s) by %s", k.Name, machineLabel(machine), k.URL, clientIP(r))
-	jsonOK(w, 201, k)
-}
-
-func (s *Server) removeKVM(w http.ResponseWriter, r *http.Request) {
-	id := r.PathValue("id")
-	found := false
-	if err := s.cfg.Update(func(c *config.Config) {
-		kept := c.KVMs[:0]
-		for _, k := range c.KVMs {
-			if k.ID == id {
-				found = true
-				continue
-			}
-			kept = append(kept, k)
-		}
-		c.KVMs = kept
-	}); err != nil {
-		jsonErr(w, 500, err.Error())
-		return
-	}
-	if !found {
-		jsonErr(w, 404, "no such KVM")
-		return
-	}
-	s.forgetKVMProxy(id)
-	s.forgetKVMCookies(id)
-	log.Printf("kvm %s removed by %s", id, clientIP(r))
-	w.WriteHeader(204)
-}

internal/api/kvm_test.go

@@ -1,123 +0,0 @@
-package api
-
-import (
-	"net/http"
-	"net/http/httptest"
-	"net/url"
-	"path/filepath"
-	"strings"
-	"testing"
-
-	"hamlab/internal/auth"
-	"hamlab/internal/config"
-	"hamlab/internal/store"
-)
-
-// A KVM's own login is a cookie its page sets in one browser. The hub remembers it from the
-// requests that carry it, lends it to a browser that lacks it — both on the wire to the
-// device and back to the browser, since the device's page checks for it client-side — and
-// forgets it when the device says it is gone or the KVM is removed.
-func TestKVMCookiesAreRememberedAndLent(t *testing.T) {
-	var seen []string // the Cookie header the fake device received, per request
-	device := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		seen = append(seen, r.Header.Get("Cookie"))
-		if r.URL.Path == "/api/auth/login" {
-			// a device that sets its token by header, with its own domain and Secure
-			http.SetCookie(w, &http.Cookie{Name: "nano-kvm-token", Value: "tok-1", Domain: "10.0.0.217", Secure: true})
-		}
-		if r.URL.Path == "/api/auth/logout" {
-			http.SetCookie(w, &http.Cookie{Name: "nano-kvm-token", Value: "", MaxAge: -1})
-		}
-		w.WriteHeader(200)
-	}))
-	defer device.Close()
-
-	dir := t.TempDir()
-	cfg := config.Default(filepath.Join(dir, "config.json"))
-	if err := cfg.Update(func(c *config.Config) {
-		c.KVMs = []config.KVM{{ID: "abc123", Name: "test", URL: device.URL, Machine: ""}}
-	}); err != nil {
-		t.Fatal(err)
-	}
-	st, err := store.Open(filepath.Join(dir, "state.json"))
-	if err != nil {
-		t.Fatal(err)
-	}
-	s := &Server{cfg: cfg, store: st, sessions: auth.NewSessions(st)}
-	k, _ := s.kvm("abc123")
-	p, err := s.kvmProxy(k)
-	if err != nil {
-		t.Fatal(err)
-	}
-	ask := func(path string, cookies ...*http.Cookie) *httptest.ResponseRecorder {
-		r := httptest.NewRequest("GET", path, nil)
-		r.SetPathValue("id", "abc123")
-		for _, c := range cookies {
-			r.AddCookie(c)
-		}
-		w := httptest.NewRecorder()
-		if strings.HasPrefix(path, kvmPrefix) {
-			s.serveKVM(w, r)
-		} else {
-			p.ServeHTTP(w, r)
-		}
-		return w
-	}
-
-	// 1. the device sets its token by header: it reaches the browser without the device's
-	//    domain or Secure (plain-http console), and the hub remembers it
-	w := ask("/api/auth/login", &http.Cookie{Name: cookieName, Value: "hamlab-session"})
-	var set *http.Cookie
-	for _, c := range w.Result().Cookies() {
-		if c.Name == "nano-kvm-token" {
-			set = c
-		}
-	}
-	if set == nil || set.Domain != "" || set.Secure || set.Value != "tok-1" || set.Path != "/" {
-		t.Fatalf("rewritten Set-Cookie = %+v, want nano-kvm-token=tok-1 for / without Domain or Secure", set)
-	}
-	if got := s.kvmCookies("abc123"); got["nano-kvm-token"] != "tok-1" {
-		t.Fatalf("remembered = %v, want the token", got)
-	}
-	if strings.Contains(seen[0], "hamlab_session") {
-		t.Fatalf("the device was handed Ham Lab's session: %q", seen[0])
-	}
-
-	// 2. a browser that has the cookie teaches a newer value
-	ask("/api/vm/info", &http.Cookie{Name: "nano-kvm-token", Value: "tok-2"}, &http.Cookie{Name: cookieName, Value: "x"})
-	if got := s.kvmCookies("abc123"); got["nano-kvm-token"] != "tok-2" {
-		t.Fatalf("remembered = %v, want tok-2", got)
-	}
-	if seen[1] != "nano-kvm-token=tok-2" {
-		t.Fatalf("device saw %q, want the browser's own cookie only", seen[1])
-	}
-
-	// 3. a browser without it is lent it: on the wire, and back as a Set-Cookie it can read
-	w = ask(kvmPrefix+"abc123/", &http.Cookie{Name: cookieName, Value: "x"})
-	if seen[2] != "nano-kvm-token=tok-2" {
-		t.Fatalf("device saw %q from a browser with no cookie, want the remembered token", seen[2])
-	}
-	lent := false
-	for _, c := range w.Result().Cookies() {
-		if c.Name == "nano-kvm-token" && c.Value == "tok-2" && !c.HttpOnly && c.Path == "/" && c.MaxAge > 0 {
-			lent = true
-		}
-	}
-	if !lent {
-		t.Fatalf("browser was not lent the token: %v", w.Result().Cookies())
-	}
-
-	// 4. the device logging out forgets it
-	ask("/api/auth/logout", &http.Cookie{Name: "nano-kvm-token", Value: "tok-2"})
-	if got := s.kvmCookies("abc123"); got != nil {
-		t.Fatalf("after logout remembered = %v, want nothing", got)
-	}
-
-	// 5. removing the KVM clears whatever is left
-	s.rememberKVMCookies("abc123", []*http.Cookie{{Name: "nano-kvm-token", Value: "tok-3"}})
-	s.forgetKVMCookies("abc123")
-	if got := s.kvmCookies("abc123"); got != nil {
-		t.Fatalf("after remove remembered = %v, want nothing", got)
-	}
-	_ = url.Parse
-}

internal/api/landing.go

@@ -0,0 +1,216 @@
+package api
+
+// The ZFS tab's landing-tier checks: what has waited too long on a landing branch, and verify
+// passes that read the pool's copy of every moved file back against the mover's ledger —
+// run by hand or on a schedule written to /etc/cron.d/hamlab-landing, just as snapshot
+// schedules are. A pass runs as its own process (`hamlab landing-verify`), the same one cron
+// starts, so it survives the console being restarted or upgraded mid-pass.
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"log"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"syscall"
+	"time"
+
+	"hamlab/internal/storage"
+)
+
+func (s *Server) syncVerifyCron(ctx context.Context, z zfsState) error {
+	exe, err := os.Executable()
+	if err != nil {
+		return err
+	}
+	exe, _ = filepath.EvalSymlinks(exe)
+	return s.storage.WriteVerifyCron(ctx, z.Verify, s.user, exe, s.cfg.Path())
+}
+
+// startVerify launches a pass in its own session: arg is "new", "all" or a schedule id.
+func (s *Server) startVerify(arg string) error {
+	exe, err := os.Executable()
+	if err != nil {
+		return err
+	}
+	exe, _ = filepath.EvalSymlinks(exe)
+	dir := filepath.Join(s.cfg.DataDir(), "landing-verify")
+	if err := os.MkdirAll(dir, 0o700); err != nil {
+		return err
+	}
+	out, err := os.OpenFile(filepath.Join(dir, "last.log"), os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
+	if err != nil {
+		return err
+	}
+	cmd := exec.Command(exe, "-config", s.cfg.Path(), "landing-verify", arg)
+	cmd.Stdout, cmd.Stderr = out, out
+	cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
+	if err := cmd.Start(); err != nil {
+		out.Close()
+		return err
+	}
+	go func() { _ = cmd.Wait(); out.Close() }()
+	return nil
+}
+
+func (s *Server) landingOverview(w http.ResponseWriter, r *http.Request) {
+	dir := s.cfg.DataDir()
+	now := time.Now()
+	type tier struct {
+		storage.LandingTarget
+		Cursor     int64               `json:"cursor"`
+		StaleCount int                 `json:"stale_count"`
+		StaleBytes int64               `json:"stale_bytes"`
+		Stale      []storage.StaleFile `json:"stale"`
+	}
+	tiers := []tier{}
+	for _, t := range storage.LandingTargets() {
+		files, n, b := storage.StaleFiles(t.Src, 100, now)
+		if files == nil {
+			files = []storage.StaleFile{}
+		}
+		tiers = append(tiers, tier{LandingTarget: t, Cursor: storage.VerifyCursor(dir, t), StaleCount: n, StaleBytes: b, Stale: files})
+	}
+	z := s.zfsLoad()
+	pols := make([]map[string]any, 0, len(z.Verify))
+	for i := range z.Verify {
+		p := z.Verify[i]
+		expr, _ := p.CronExpr()
+		m := map[string]any{"policy": p, "cron": expr, "last": storage.LastVerify(dir, p.ID)}
+		if p.Enabled {
+			if n := p.NextRun(now); !n.IsZero() {
+				m["next"] = n
+			}
+		}
+		pols = append(pols, m)
+	}
+	hist := storage.VerifyHistory(dir)
+	if len(hist) > 10 {
+		hist = hist[:10]
+	}
+	jsonOK(w, 200, map[string]any{
+		"tiers": tiers, "stale_after_sec": int(storage.StaleAfter.Seconds()),
+		"running": storage.RunningVerify(dir), "history": hist, "policies": pols,
+		"cron_file": storage.VerifyCronFile,
+	})
+}
+
+func (s *Server) landingVerifyNow(w http.ResponseWriter, r *http.Request) {
+	var body struct {
+		Scope string `json:"scope"`
+	}
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1024)).Decode(&body); err != nil || (body.Scope != "new" && body.Scope != "all") {
+		jsonErr(w, 400, "scope: new or all")
+		return
+	}
+	if storage.RunningVerify(s.cfg.DataDir()) != nil {
+		jsonErr(w, 409, "a verify pass is already running")
+		return
+	}
+	if len(storage.LandingTargets()) == 0 {
+		jsonErr(w, 409, "no landing mover is reporting on this machine, so there is nothing to verify")
+		return
+	}
+	if err := s.startVerify(body.Scope); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	log.Printf("landing verify (%s) started by %s", body.Scope, clientIP(r))
+	jsonOK(w, 202, map[string]string{"started": body.Scope})
+}
+
+func (s *Server) landingPolicyPut(w http.ResponseWriter, r *http.Request) {
+	var p storage.VerifyPolicy
+	if err := json.NewDecoder(io.LimitReader(r.Body, 16<<10)).Decode(&p); err != nil {
+		jsonErr(w, 400, "bad request")
+		return
+	}
+	if err := p.Validate(); err != nil {
+		jsonErr(w, 400, err.Error())
+		return
+	}
+	z := s.zfsLoad()
+	found := false
+	for i := range z.Verify {
+		if p.ID != "" && z.Verify[i].ID == p.ID {
+			z.Verify[i], found = p, true
+		}
+	}
+	if !found {
+		if p.ID == "" {
+			id, err := randomID()
+			if err != nil {
+				jsonErr(w, 500, err.Error())
+				return
+			}
+			p.ID = id
+		}
+		z.Verify = append(z.Verify, p)
+	}
+	if err := s.zfsSave(z); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	if err := s.syncVerifyCron(r.Context(), z); err != nil {
+		jsonErr(w, 500, "saved, but writing "+storage.VerifyCronFile+" failed: "+err.Error())
+		return
+	}
+	expr, _ := p.CronExpr()
+	log.Printf("landing verify schedule %q saved by %s (%q, %s)", p.Name, clientIP(r), expr, p.Scope)
+	jsonOK(w, 200, p)
+}
+
+func (s *Server) landingPolicyDelete(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	z := s.zfsLoad()
+	kept := z.Verify[:0]
+	found := false
+	for _, p := range z.Verify {
+		if p.ID == id {
+			found = true
+			continue
+		}
+		kept = append(kept, p)
+	}
+	if !found {
+		jsonErr(w, 404, "no such schedule")
+		return
+	}
+	z.Verify = kept
+	if err := s.zfsSave(z); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	if err := s.syncVerifyCron(r.Context(), z); err != nil {
+		jsonErr(w, 500, "removed, but writing "+storage.VerifyCronFile+" failed: "+err.Error())
+		return
+	}
+	storage.ForgetVerify(s.cfg.DataDir(), id)
+	log.Printf("landing verify schedule %s removed by %s", id, clientIP(r))
+	w.WriteHeader(204)
+}
+
+func (s *Server) landingPolicyRun(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	found := false
+	for _, p := range s.zfsLoad().Verify {
+		found = found || p.ID == id
+	}
+	if !found {
+		jsonErr(w, 404, "no such schedule")
+		return
+	}
+	if storage.RunningVerify(s.cfg.DataDir()) != nil {
+		jsonErr(w, 409, "a verify pass is already running")
+		return
+	}
+	if err := s.startVerify(id); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	log.Printf("landing verify schedule %s run by %s", id, clientIP(r))
+	jsonOK(w, 202, map[string]string{"started": id})
+}

internal/api/play.go

@@ -1,141 +1,162 @@
 package api
 
-// Playing a video the browser cannot play as it is: POST /api/files/play starts an HLS
-// re-encode on this machine (internal/media), GET …/play/{id}/index.m3u8 and the segments
-// feed the player, GET …/play/{id} is its progress, DELETE ends it. /api/files/play-caps
-// says which codecs this machine can make and on what. Root mode (?sudo=1) reads the file
-// and runs ffmpeg as root, through the same sudo the file reads use.
+// Playing a video in the Files viewer, YouTube's way: Auto (the player reads the connection),
+// any quality by hand, or the file itself. The qualities are made on the fly as HLS by
+// internal/media, on the device the Transcode device setting names — which may be on another
+// machine than the file.
+//
+// On every machine:
+//   POST /api/files/play[?sudo=1] {path | url, name, device}   open a play
+//   GET  /api/files/play/{id}/master.m3u8, /{rung}.m3u8, /{rung}/{n}.ts
+//   GET  /api/files/play/{id}                                    what it is doing
+//   DELETE /api/files/play/{id} (or POST …/close, for a beacon)  end it
+//   GET  /api/files/play-devices                                 what it can encode on
+// On the hub:
+//   POST /api/play {node, path, sudo}   open a play of a file on node, wherever the setting
+//                                       says, answering which machine the play is on
+//   GET|PUT /api/transcode              the setting, and every machine's devices
+//   GET  /api/play-src/{token}          a file, for a machine transcoding it, by a token
+//                                       that names that one file; ranges pass through
 
 import (
 	"context"
+	"crypto/rand"
+	"encoding/hex"
 	"encoding/json"
 	"errors"
+	"fmt"
 	"io"
 	"log"
+	"net"
 	"net/http"
+	"net/url"
 	"path/filepath"
+	"strconv"
 	"strings"
+	"sync"
 	"time"
 
 	"hamlab/internal/files"
 	"hamlab/internal/media"
+	"hamlab/internal/store"
 )
 
-const playLimit = 3 // re-encodes at once on one machine; a GPU does two or three, a CPU one
+// playRuns is how many plays may be encoding at once on one machine; a fourth ends the one
+// asked of least lately. The A380 makes about two 8K-sourced streams in real time.
+const playRuns = 3
 
-type playCaps struct {
-	Codecs    []playCodec     `json:"codecs"`
-	Qualities []media.Quality `json:"qualities"`
-	Error     string          `json:"error,omitempty"`
+type playReq struct {
+	Path   string `json:"path,omitempty"`
+	URL    string `json:"url,omitempty"` // a file on another machine, as the hub serves it
+	Name   string `json:"name,omitempty"`
+	Device string `json:"device,omitempty"`
 }
 
-type playCodec struct {
-	media.Codec
-	Engine string `json:"engine,omitempty"` // "" when nothing here encodes it
+type playOut struct {
+	Node   string           `json:"node"` // the machine the play is on: "" for the hub; filled in by the hub
+	Play   *media.Play      `json:"play"`
+	Master string           `json:"master"` // the playlist, under that machine's /api
+	Engine string           `json:"engine"`
+	Host   string           `json:"host"` // that machine's name, for the menu
+	Note   string           `json:"note,omitempty"`
+	Status media.PlayStatus `json:"status"`
 }
 
-// playCapsHandler: GET /api/files/play-caps[?sudo=1]
-func (s *Server) playCapsHandler(w http.ResponseWriter, r *http.Request) {
+// playDevices: GET /api/files/play-devices — what this machine could make the player's H.264
+// on, probed as the daemon's own user (a play from another machine never runs as root).
+func (s *Server) playDevices(w http.ResponseWriter, r *http.Request) {
+	caps, err := s.thumbs.capsFor(r.Context(), false, false)
+	if err != nil {
+		caps.Error = err.Error()
+	}
+	jsonOK(w, 200, map[string]any{"host": s.hostName(), "devices": media.Devices(caps)})
+}
+
+// playStart: POST /api/files/play[?sudo=1] — open a play of a file here, or of a URL the hub
+// serves a file on another machine at.
+func (s *Server) playStart(w http.ResponseWriter, r *http.Request) {
+	var body playReq
+	if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
+		jsonErr(w, 400, "bad request")
+		return
+	}
 	root := r.URL.Query().Get("sudo") == "1"
 	if root && s.fsSudo == nil {
 		jsonErr(w, 403, "root access needs passwordless sudo on the host")
 		return
 	}
-	caps, err := s.thumbs.capsFor(r.Context(), root, false)
-	pc := playCaps{Codecs: []playCodec{}, Qualities: media.Qualities}
+	out, code, err := s.openPlay(r.Context(), body, root)
 	if err != nil {
-		pc.Error = err.Error()
-	} else if caps.Error != "" {
-		pc.Error = caps.Error
+		jsonErr(w, code, err.Error())
+		return
 	}
-	for _, c := range media.Codecs {
-		x := playCodec{Codec: c}
-		if pc.Error == "" {
-			x.Engine = media.EngineFor(caps, c.ID)
-		}
-		pc.Codecs = append(pc.Codecs, x)
-	}
-	jsonOK(w, 200, pc)
+	jsonOK(w, 200, out)
 }
 
-// playStart: POST /api/files/play[?sudo=1] {path, codec, quality}
-func (s *Server) playStart(w http.ResponseWriter, r *http.Request) {
-	fsys, ok := s.fsFor(w, r)
-	if !ok {
-		return
+func (s *Server) openPlay(ctx context.Context, body playReq, root bool) (playOut, int, error) {
+	if root && s.fsSudo == nil && body.URL == "" {
+		return playOut{}, 403, errors.New("root access needs passwordless sudo on the host")
 	}
-	root := r.URL.Query().Get("sudo") == "1"
-	var body struct{ Path, Codec, Quality string }
-	if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
-		jsonErr(w, 400, "bad request")
-		return
+	in, name := body.Path, body.Name
+	if body.URL != "" {
+		u, err := url.Parse(body.URL)
+		if err != nil || (u.Scheme != "http" && u.Scheme != "https") || body.Path != "" {
+			return playOut{}, 400, errors.New("a play is of a path or of an http URL")
 		}
+		in, root = body.URL, false
+	} else {
 		p, err := files.Clean(body.Path)
 		if err != nil {
-		jsonErr(w, 400, err.Error())
-		return
+			return playOut{}, 400, err
 		}
 		if media.Kind(p) != "video" {
-		jsonErr(w, 415, "not a video")
-		return
+			return playOut{}, 415, errors.New("not a video")
 		}
-	if err := media.ValidChoice(body.Codec, body.Quality); err != nil {
-		jsonErr(w, 400, err.Error())
-		return
+		fsys := s.fsLocal
+		if root {
+			fsys = s.fsSudo
 		}
-	if _, err := fsys.Stat(r.Context(), p); err != nil {
-		fsErr(w, err)
-		return
+		if _, err := fsys.Stat(ctx, p); err != nil {
+			return playOut{}, 404, err
+		}
+		in = p
+		if name == "" {
+			name = filepath.Base(p)
+		}
+	}
+	caps, err := s.thumbs.capsFor(ctx, root, false)
+	if err == nil && caps.Error != "" {
+		err = errors.New(caps.Error)
 	}
-	caps, err := s.thumbs.capsFor(r.Context(), root, false)
 	if err != nil {
-		jsonErr(w, 503, err.Error())
-		return
+		return playOut{}, 503, err
 	}
-	if caps.Error != "" {
-		jsonErr(w, 503, caps.Error)
-		return
-	}
-	ictx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
-	src, err := media.Inspect(ictx, caps, p, root)
+	ictx, cancel := context.WithTimeout(ctx, 30*time.Second)
+	src, err := media.Inspect(ictx, caps, in, root)
 	cancel()
 	if err != nil {
-		playErr(w, err)
-		return
-	}
-	st, err := s.plays.Open(r.Context(), caps, root, p, body.Codec, body.Quality, src)
-	if err != nil {
-		playErr(w, err)
-		return
-	}
-	log.Printf("files: play %s as %s %s on %s (from %s)", p, st.Codec, st.Quality, st.Engine, clientIP(r))
-	jsonOK(w, 200, map[string]any{"stream": st, "url": "/api/files/play/" + st.ID + "/index.m3u8"})
-}
-
-func playErr(w http.ResponseWriter, err error) {
 		var de *media.DecodeError
-	switch {
-	case errors.As(err, &de):
-		jsonErr(w, 422, de.Msg)
-	case strings.Contains(err.Error(), "already being re-encoded"):
-		jsonErr(w, 429, err.Error())
-	case strings.HasPrefix(err.Error(), "unknown ") || strings.HasPrefix(err.Error(), "nothing on this machine encodes"):
-		jsonErr(w, 400, err.Error())
-	case errors.Is(err, context.DeadlineExceeded):
-		jsonErr(w, 504, "the file took too long to read")
-	default:
-		jsonErr(w, 500, err.Error())
+		if errors.As(err, &de) {
+			return playOut{}, 422, errors.New(de.Msg)
 		}
+		return playOut{}, 504, err
+	}
+	p, err := s.plays.Open(caps, root, in, name, src, body.Device)
+	if err != nil {
+		return playOut{}, 422, err
+	}
+	log.Printf("files: play %s on %s (%d qualities, %s)", name, p.Engine(), len(p.Rungs), map[bool]string{true: "from another machine", false: "here"}[p.Remote])
+	return playOut{Play: p, Master: "/api/files/play/" + p.ID + "/master.m3u8", Engine: p.Engine(), Host: s.hostName(), Status: p.Status()}, 200, nil
 }
 
 // playStatus: GET /api/files/play/{id}
 func (s *Server) playStatus(w http.ResponseWriter, r *http.Request) {
-	st, ok := s.plays.Find(r.PathValue("id"))
+	p, ok := s.plays.Find(r.PathValue("id"))
 	if !ok {
-		jsonErr(w, 404, "no such stream")
+		jsonErr(w, 404, "no such play")
 		return
 	}
-	jsonOK(w, 200, st.Status())
+	jsonOK(w, 200, p.Status())
 }
 
 // playStop: DELETE /api/files/play/{id}, or POST …/close — the form a page can send on its
@@ -145,33 +166,47 @@ func (s *Server) playStop(w http.ResponseWriter, r *http.Request) {
 	w.WriteHeader(204)
 }
 
-// playFile: GET /api/files/play/{id}/{file} — the playlist, the init segment, a segment.
-// A seekable stream's playlist is written whole, up front; a segment of it that isn't on disk
-// yet is waited for, or re-encoded from there if the encode isn't heading that way (a seek).
-func (s *Server) playFile(w http.ResponseWriter, r *http.Request) {
-	st, ok := s.plays.Find(r.PathValue("id"))
+// playPlaylist: GET /api/files/play/{id}/{file} — master.m3u8, or one quality's <name>.m3u8.
+func (s *Server) playPlaylist(w http.ResponseWriter, r *http.Request) {
+	p, ok := s.plays.Find(r.PathValue("id"))
 	if !ok {
-		jsonErr(w, 404, "no such stream")
+		jsonErr(w, 404, "no such play")
 		return
 	}
-	name := r.PathValue("file")
-	p, ok := st.File(name)
-	if !ok {
-		jsonErr(w, 404, "not a stream file")
+	f := r.PathValue("file")
+	var b []byte
+	if f == "master.m3u8" {
+		b = p.Master()
+	} else if name, isList := strings.CutSuffix(f, ".m3u8"); isList {
+		if b, ok = p.Media(name); !ok {
+			jsonErr(w, 404, "no such quality")
+			return
+		}
+	} else {
+		jsonErr(w, 404, "not a play file")
 		return
 	}
-	w.Header().Set("Cache-Control", "no-store")
-	w.Header().Set("X-Content-Type-Options", "nosniff")
-	if st.Seekable {
-		if name == "index.m3u8" {
 	w.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
-			_, _ = w.Write(st.Playlist())
+	w.Header().Set("Cache-Control", "no-store")
+	_, _ = w.Write(b)
+}
+
+// playPiece: GET /api/files/play/{id}/{rung}/{seg} — a piece, made first if need be.
+func (s *Server) playPiece(w http.ResponseWriter, r *http.Request) {
+	p, ok := s.plays.Find(r.PathValue("id"))
+	if !ok {
+		jsonErr(w, 404, "no such play")
 		return
 	}
-		if i, isSeg := media.SegmentIndex(name); isSeg {
-			sp, err := st.Segment(r.Context(), i)
+	ns, isTS := strings.CutSuffix(r.PathValue("seg"), ".ts")
+	n, err := strconv.Atoi(ns)
+	if !isTS || err != nil || strconv.Itoa(n) != ns {
+		jsonErr(w, 404, "not a piece")
+		return
+	}
+	file, err := p.Segment(r.Context().Done(), r.PathValue("rung"), n)
 	switch {
-			case errors.Is(err, media.ErrNotInFilm):
+	case errors.Is(err, media.ErrNotFound):
 		jsonErr(w, 404, err.Error())
 		return
 	case r.Context().Err() != nil:
@@ -180,14 +215,349 @@ func (s *Server) playFile(w http.ResponseWriter, r *http.Request) {
 		jsonErr(w, 503, err.Error())
 		return
 	}
-			p = sp
+	w.Header().Set("Content-Type", "video/mp2t")
+	w.Header().Set("Cache-Control", "private, max-age=3600")
+	http.ServeFile(w, r, file)
+}
+
+// ---- on the hub: where plays are made ----
+
+// transcodeChoice is the setting; nil is the machine the file is on.
+func (s *Server) transcodeChoice() *store.Transcode {
+	var t *store.Transcode
+	if s.store == nil {
+		return nil
+	}
+	s.store.View(func(st *store.State) {
+		if st.Transcode != nil {
+			c := *st.Transcode
+			t = &c
+		}
+	})
+	return t
+}
+
+type transcodeMachine struct {
+	Node    string         `json:"node"` // "" is the hub
+	Host    string         `json:"host"`
+	Devices []media.Device `json:"devices"`
+	Error   string         `json:"error,omitempty"` // the machine didn't answer
+}
+
+// getTranscode: GET /api/transcode — the setting, and what every machine can encode on,
+// asked of each now. ?only=selected asks only the chosen machine, for the device's name.
+func (s *Server) getTranscode(w http.ResponseWriter, r *http.Request) {
+	if r.URL.Query().Get("only") == "selected" {
+		c := s.transcodeChoice()
+		out := map[string]any{"selected": c}
+		if c != nil {
+			name := c.Device
+			ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
+			defer cancel()
+			var devs []media.Device
+			if c.Node == "" {
+				caps, _ := s.thumbs.capsFor(ctx, false, false)
+				devs = media.Devices(caps)
+			} else if code, b, err := s.nodes.Do(ctx, c.Node, "GET", "/api/files/play-devices", nil, nil); err == nil && code == 200 {
+				var d struct{ Devices []media.Device }
+				_ = json.Unmarshal(b, &d)
+				devs = d.Devices
+			}
+			for _, d := range devs {
+				if d.ID == c.Device {
+					name = d.Name
 				}
 			}
-	switch filepath.Ext(p) {
-	case ".m3u8":
-		w.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
+			host := c.Node
+			if host == "" {
+				host = s.hostName()
+			}
+			out["host"], out["name"] = host, name
+		}
+		jsonOK(w, 200, out)
+		return
+	}
+	names := []string{""}
+	if s.nodes != nil {
+		names = append(names, s.nodes.Names()...)
+	}
+	out := make([]transcodeMachine, len(names))
+	var wg sync.WaitGroup
+	for i, n := range names {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			m := transcodeMachine{Node: n, Host: n}
+			ctx, cancel := context.WithTimeout(r.Context(), 8*time.Second)
+			defer cancel()
+			if n == "" {
+				caps, err := s.thumbs.capsFor(ctx, false, false)
+				if err != nil {
+					caps.Error = err.Error()
+				}
+				m.Host, m.Devices = s.hostName(), media.Devices(caps)
+			} else {
+				code, b, err := s.nodes.Do(ctx, n, "GET", "/api/files/play-devices", nil, nil)
+				var d struct {
+					Host    string         `json:"host"`
+					Devices []media.Device `json:"devices"`
+				}
+				switch {
+				case err != nil:
+					m.Error = err.Error()
+				case code == 404:
+					m.Error = "runs an older Ham Lab"
+				case code != 200 || json.Unmarshal(b, &d) != nil:
+					m.Error = fmt.Sprintf("answered %d", code)
 				default:
-		w.Header().Set("Content-Type", "video/mp4")
+					m.Devices = d.Devices
 				}
-	http.ServeFile(w, r, p)
+			}
+			out[i] = m
+		}()
+	}
+	wg.Wait()
+	jsonOK(w, 200, map[string]any{"selected": s.transcodeChoice(), "machines": out})
+}
+
+// putTranscode: PUT /api/transcode {node, device}, or {} for the machine the file is on.
+func (s *Server) putTranscode(w http.ResponseWriter, r *http.Request) {
+	var body *store.Transcode
+	if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
+		jsonErr(w, 400, "expected {\"node\": …, \"device\": …}")
+		return
+	}
+	if body != nil && body.Device == "" {
+		body = nil
+	}
+	if body != nil {
+		known := body.Node == ""
+		if s.nodes != nil {
+			for _, n := range s.nodes.Names() {
+				known = known || n == body.Node
+			}
+		}
+		if !known {
+			jsonErr(w, 400, "no such machine")
+			return
+		}
+		if body.Device != media.CPU && !strings.HasPrefix(body.Device, "/dev/dri/") {
+			jsonErr(w, 400, "a device is a render node or \"cpu\"")
+			return
+		}
+	}
+	if err := s.store.Update(func(st *store.State) { st.Transcode = body }); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	s.getTranscode(w, r)
+}
+
+// hubPlay: POST /api/play {node, path, sudo} — open a play of a file on node ("" the hub) on
+// the device the setting names. When that is on another machine, that machine reads the file
+// through this hub, by a token good for that one file. If it can't be reached, the file's own
+// machine makes it instead, and the answer says so.
+func (s *Server) hubPlay(w http.ResponseWriter, r *http.Request) {
+	var body struct {
+		Node string `json:"node"`
+		Path string `json:"path"`
+		Sudo bool   `json:"sudo"`
+	}
+	if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
+		jsonErr(w, 400, "bad request")
+		return
+	}
+	p, err := files.Clean(body.Path)
+	if err != nil {
+		jsonErr(w, 400, err.Error())
+		return
+	}
+	if body.Node != "" && (s.nodes == nil || !has(s.nodes.Names(), body.Node)) {
+		jsonErr(w, 404, "no such machine")
+		return
+	}
+	choice := s.transcodeChoice()
+	var note string
+	if choice != nil && choice.Node != body.Node {
+		if s.nodes != nil && choice.Node != "" && !has(s.nodes.Names(), choice.Node) {
+			note = "The transcode machine " + choice.Node + " is no longer managed here; made where the file is."
+		} else {
+			src, err := s.srcURL(choice.Node, body.Node, p, body.Sudo)
+			if err == nil {
+				req := playReq{URL: src, Name: filepath.Base(p), Device: choice.Device}
+				out, code, err := s.playOn(r.Context(), choice.Node, req, false)
+				if err == nil {
+					jsonOK(w, 200, out)
+					return
+				}
+				if code == 422 || code == 415 {
+					jsonErr(w, code, err.Error())
+					return
+				}
+			}
+			log.Printf("files: play on %s failed (%v); making it where the file is", nodeLabel(choice.Node), err)
+			note = "Couldn't make it on " + nodeLabel(choice.Node) + " (" + err.Error() + "), so it is made where the file is."
+		}
+	}
+	device := ""
+	if choice != nil && choice.Node == body.Node {
+		device = choice.Device
+	}
+	out, code, err := s.playOn(r.Context(), body.Node, playReq{Path: p, Device: device}, body.Sudo)
+	if err != nil {
+		jsonErr(w, code, err.Error())
+		return
+	}
+	out.Note = note
+	jsonOK(w, 200, out)
+}
+
+func nodeLabel(n string) string {
+	if n == "" {
+		return "the hub"
+	}
+	return n
+}
+
+func has(list []string, x string) bool {
+	for _, v := range list {
+		if v == x {
+			return true
+		}
+	}
+	return false
+}
+
+// playOn opens a play on a machine: here, or through that node's API.
+func (s *Server) playOn(ctx context.Context, node string, req playReq, sudo bool) (playOut, int, error) {
+	if node == "" {
+		out, code, err := s.openPlay(ctx, req, sudo)
+		return out, code, err
+	}
+	b, _ := json.Marshal(req)
+	path := "/api/files/play"
+	if sudo {
+		path += "?sudo=1"
+	}
+	code, resp, err := s.nodes.Do(ctx, node, "POST", path, b, http.Header{"Content-Type": []string{"application/json"}})
+	if err != nil {
+		return playOut{}, 502, err
+	}
+	if code != 200 {
+		var e struct{ Error string }
+		_ = json.Unmarshal(resp, &e)
+		if e.Error == "" {
+			e.Error = fmt.Sprintf("%s answered %d", node, code)
+		}
+		return playOut{}, code, errors.New(e.Error)
+	}
+	var out playOut
+	if err := json.Unmarshal(resp, &out); err != nil {
+		return playOut{}, 502, err
+	}
+	out.Node = node
+	return out, 200, nil
+}
+
+// ---- the file, for the machine transcoding it ----
+
+type srcGrant struct {
+	node, path string
+	sudo       bool
+	until      time.Time
+}
+
+type srcGrants struct {
+	mu sync.Mutex
+	m  map[string]*srcGrant
+}
+
+const srcLife = 12 * time.Hour // from its last use: a film paused overnight still plays
+
+var grants = &srcGrants{m: map[string]*srcGrant{}}
+
+func (g *srcGrants) mint(node, path string, sudo bool) string {
+	b := make([]byte, 24)
+	_, _ = rand.Read(b)
+	tok := hex.EncodeToString(b)
+	now := time.Now()
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	for k, v := range g.m {
+		if now.After(v.until) {
+			delete(g.m, k)
+		}
+	}
+	g.m[tok] = &srcGrant{node: node, path: path, sudo: sudo, until: now.Add(srcLife)}
+	return tok
+}
+
+func (g *srcGrants) use(tok string) (srcGrant, bool) {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	v, ok := g.m[tok]
+	if !ok || time.Now().After(v.until) {
+		return srcGrant{}, false
+	}
+	v.until = time.Now().Add(srcLife)
+	return *v, true
+}
+
+// srcURL is where the machine `to` can read a file on machine `from` through this hub: this
+// hub's own address as `to` reaches it, and a token for that one file.
+func (s *Server) srcURL(to, from, path string, sudo bool) (string, error) {
+	host, port, err := net.SplitHostPort(s.cfg.Listen)
+	if err != nil {
+		return "", err
+	}
+	ip := net.ParseIP(host)
+	if host == "" || ip == nil || ip.IsUnspecified() || ip.IsLoopback() {
+		// Listening everywhere: the address this machine reaches `to` from is one it answers on.
+		target := "127.0.0.1:1"
+		if to != "" {
+			if u, err := url.Parse(s.nodes.URL(to)); err == nil && u.Host != "" {
+				target = u.Host
+			}
+		}
+		c, err := net.Dial("udp", target)
+		if err != nil {
+			return "", err
+		}
+		host = c.LocalAddr().(*net.UDPAddr).IP.String()
+		c.Close()
+	}
+	scheme := "http"
+	if s.fingerprint != "" {
+		scheme = "https"
+	}
+	return scheme + "://" + net.JoinHostPort(host, port) + "/api/play-src/" + grants.mint(from, path, sudo), nil
+}
+
+// playSrc: GET /api/play-src/{token} — the one file a token names, ranges and all, read from
+// its machine. The token is the only key: it is minted for a machine transcoding that file.
+func (s *Server) playSrc(w http.ResponseWriter, r *http.Request) {
+	g, ok := grants.use(r.PathValue("token"))
+	if !ok {
+		jsonErr(w, 404, "no such file")
+		return
+	}
+	q := url.Values{"path": {g.path}}
+	if g.sudo {
+		q.Set("sudo", "1")
+	}
+	r2 := r.Clone(r.Context())
+	r2.URL.RawQuery = q.Encode()
+	r2.Header.Del("Cookie")
+	if g.node == "" {
+		r2.URL.Path = "/api/files/read"
+		s.filesRead(w, r2)
+		return
+	}
+	p, ok := s.nodes.Proxy(g.node)
+	if !ok {
+		jsonErr(w, 404, "no such machine")
+		return
+	}
+	r2.URL.Path = "/api/nodes/" + g.node + "/files/read"
+	p.ServeHTTP(w, r2)
 }

internal/api/play_test.go

@@ -0,0 +1,93 @@
+package api
+
+import (
+	"fmt"
+	"net/http/httptest"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"hamlab/internal/config"
+	"hamlab/internal/files"
+)
+
+// The file a machine transcodes from another is read through the hub by a token naming that
+// one file: ranges pass through, the query can't point it anywhere else, and an unknown token
+// is nothing.
+func TestPlaySrc(t *testing.T) {
+	dir := t.TempDir()
+	p := filepath.Join(dir, "film.mp4")
+	if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(filepath.Join(dir, "other.mp4"), []byte("secret"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	s := &Server{fsLocal: files.Local{}, cfg: &config.Config{Listen: "10.0.0.234:8100"}}
+	u, err := s.srcURL("", "", p, false)
+	if err != nil || !strings.HasPrefix(u, "http://10.0.0.234:8100/api/play-src/") {
+		t.Fatalf("url %q %v", u, err)
+	}
+	tok := strings.TrimPrefix(u, "http://10.0.0.234:8100/api/play-src/")
+	get := func(tok, query, rng string) *httptest.ResponseRecorder {
+		r := httptest.NewRequest("GET", "/api/play-src/"+tok+query, nil)
+		r.SetPathValue("token", tok)
+		if rng != "" {
+			r.Header.Set("Range", rng)
+		}
+		w := httptest.NewRecorder()
+		s.playSrc(w, r)
+		return w
+	}
+	if w := get(tok, "", "bytes=2-4"); w.Code != 206 || w.Body.String() != "234" {
+		t.Fatalf("range: %d %q", w.Code, w.Body)
+	}
+	if w := get(tok, "?path="+filepath.Join(dir, "other.mp4"), ""); w.Body.String() != "0123456789" {
+		t.Fatalf("the query moved it: %q", w.Body)
+	}
+	if w := get("nope", "", ""); w.Code != 404 {
+		t.Fatalf("unknown token: %d", w.Code)
+	}
+}
+
+// A preview made on another machine is kept by the file's own: only while the file is still
+// the one it was made of, and a failure is kept as a failure.
+func TestThumbPut(t *testing.T) {
+	dir := t.TempDir()
+	p := filepath.Join(dir, "film.mp4")
+	if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	fi, _ := os.Stat(p)
+	s := &Server{fsLocal: files.Local{}, thumbs: newThumbSvc(dir, "")}
+	v := fmt.Sprintf("%d-%d", fi.ModTime().UnixMilli(), fi.Size())
+	put := func(v string, body string, hdr map[string]string) int {
+		r := httptest.NewRequest("PUT", "/api/files/thumb?path="+url.QueryEscape(p)+"&v="+v, strings.NewReader(body))
+		for k, x := range hdr {
+			r.Header.Set(k, x)
+		}
+		w := httptest.NewRecorder()
+		s.thumbPut(w, r)
+		return w.Code
+	}
+	get := func() *httptest.ResponseRecorder {
+		r := httptest.NewRequest("GET", "/api/files/thumb?cached=1&path="+url.QueryEscape(p)+"&v="+v, nil)
+		w := httptest.NewRecorder()
+		s.filesThumb(w, r)
+		return w
+	}
+	if w := get(); w.Code != 404 {
+		t.Fatalf("cached=1 before anything: %d", w.Code)
+	}
+	if c := put("1-10", "jpeg", nil); c != 409 {
+		t.Fatalf("a changed file: %d", c)
+	}
+	if c := put(v, "jpeg-bytes", map[string]string{"X-Hamlab-Thumb": "grid vaapi:/dev/dri/renderD129"}); c != 204 {
+		t.Fatalf("put: %d", c)
+	}
+	if w := get(); w.Code != 200 || w.Body.String() != "jpeg-bytes" || !strings.Contains(w.Header().Get("X-Hamlab-Thumb"), "grid") {
+		t.Fatalf("kept: %d %q %q", w.Code, w.Body, w.Header().Get("X-Hamlab-Thumb"))
+	}
+}

internal/api/server.go

@@ -75,11 +75,10 @@ type Server struct {
 	net         *network.Watcher
 	update      updateState
 	events      eventLog
-	kvms        kvmProxies
 	fsLocal     files.FS
 	fsSudo      files.FS // nil without passwordless sudo
 	thumbs      *thumbSvc
-	plays       *media.Streamer
+	plays       *media.Plays
 	self        string // this binary, for helpers run as tasks
 	nodes       *nodes.Manager
 	fingerprint string
@@ -103,7 +102,7 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
 	s.self = self
 	s.space = space.New(self, filepath.Join(cfg.DataDir(), "space"))
 	s.thumbs = newThumbSvc(cfg.DataDir(), self)
-	s.plays = media.NewStreamer(cfg.DataDir(), self, playLimit)
+	s.plays = media.NewPlays(cfg.DataDir(), self, playRuns)
 	s.hostName() // read it once now so the first request doesn't have to
 	if u, err := user.Current(); err == nil && u.Username != "" {
 		s.user = u.Username
@@ -276,15 +275,23 @@ func (s *Server) routes() *http.ServeMux {
 	m.HandleFunc("POST /api/files/delete", s.auth(s.filesDelete))
 	m.HandleFunc("POST /api/files/chmod", s.auth(s.filesChmod))
 	m.HandleFunc("GET /api/files/thumb", s.auth(s.filesThumb))
+	m.HandleFunc("PUT /api/files/thumb", s.auth(s.thumbPut))
+	m.HandleFunc("POST /api/files/thumb-from", s.auth(s.thumbFrom))
+	m.HandleFunc("GET /api/nodes/{name}/files/thumb", s.auth(s.hubThumb))
 	m.HandleFunc("GET /api/files/thumbs", s.auth(s.thumbsStatus))
 	m.HandleFunc("POST /api/files/thumbs/install", s.auth(s.thumbsInstall))
 	m.HandleFunc("POST /api/files/thumbs/gpu-access", s.auth(s.thumbsGPUAccess))
-	m.HandleFunc("GET /api/files/play-caps", s.auth(s.playCapsHandler))
+	m.HandleFunc("GET /api/files/play-devices", s.auth(s.playDevices))
 	m.HandleFunc("POST /api/files/play", s.auth(s.playStart))
 	m.HandleFunc("GET /api/files/play/{id}", s.auth(s.playStatus))
 	m.HandleFunc("DELETE /api/files/play/{id}", s.auth(s.playStop))
 	m.HandleFunc("POST /api/files/play/{id}/close", s.auth(s.playStop))
-	m.HandleFunc("GET /api/files/play/{id}/{file}", s.auth(s.playFile))
+	m.HandleFunc("GET /api/files/play/{id}/{file}", s.auth(s.playPlaylist))
+	m.HandleFunc("GET /api/files/play/{id}/{rung}/{seg}", s.auth(s.playPiece))
+	m.HandleFunc("POST /api/play", s.auth(s.hubPlay))
+	m.HandleFunc("GET /api/transcode", s.auth(s.getTranscode))
+	m.HandleFunc("PUT /api/transcode", s.auth(s.putTranscode))
+	m.HandleFunc("GET /api/play-src/{token}", s.playSrc) // the token is the key
 	m.HandleFunc("GET /api/hardware", s.auth(s.hardwareReport))
 	m.HandleFunc("GET /api/network", s.auth(s.networkOverview))
 	m.HandleFunc("GET /api/vpn", s.auth(s.vpnReport))
@@ -306,6 +313,11 @@ func (s *Server) routes() *http.ServeMux {
 	m.HandleFunc("DELETE /api/zfs/schedules/{id}", s.auth(s.zfsPolicyDelete))
 	m.HandleFunc("POST /api/zfs/schedules/{id}/run", s.auth(s.zfsRunNow))
 	m.HandleFunc("POST /api/zfs/snapshots/delete", s.auth(s.zfsSnapshotDelete))
+	m.HandleFunc("GET /api/zfs/landing", s.auth(s.landingOverview))
+	m.HandleFunc("POST /api/zfs/landing/verify", s.auth(s.landingVerifyNow))
+	m.HandleFunc("PUT /api/zfs/landing/schedules", s.auth(s.landingPolicyPut))
+	m.HandleFunc("DELETE /api/zfs/landing/schedules/{id}", s.auth(s.landingPolicyDelete))
+	m.HandleFunc("POST /api/zfs/landing/schedules/{id}/run", s.auth(s.landingPolicyRun))
 	m.HandleFunc("GET /api/space", s.auth(s.spaceOverview))
 	m.HandleFunc("GET /api/space/{disk}", s.auth(s.spaceStatus))
 	m.HandleFunc("POST /api/space/{disk}/scan", s.auth(s.spaceScan))
@@ -338,18 +350,7 @@ func (s *Server) routes() *http.ServeMux {
 	m.HandleFunc("PUT /api/prefs/sampler", s.auth(s.putSampler))
 	m.HandleFunc("GET /api/prefs/history", s.auth(s.getHistoryWindow))
 	m.HandleFunc("PUT /api/prefs/history", s.auth(s.putHistoryWindow))
-	m.HandleFunc("GET /api/kvms", s.auth(s.listKVMs))
-	m.HandleFunc("POST /api/kvms", s.auth(s.addKVM))
-	m.HandleFunc("DELETE /api/kvms/{id}", s.auth(s.removeKVM))
-	m.HandleFunc("/kvm/{id}/{rest...}", s.auth(s.serveKVM))
-	m.HandleFunc("/kvm/{id}", s.auth(s.serveKVM))
-	// A path Ham Lab does not serve may still belong to a KVM's page, which asks for its
-	// own assets by absolute path. Ham Lab's own routes matched first, so this can only
-	// catch what would otherwise have been a 404.
 	m.HandleFunc("/api/", s.auth(func(w http.ResponseWriter, r *http.Request) {
-		if s.kvmFallback(w, r) {
-			return
-		}
 		jsonErr(w, 404, "not found")
 	}))
 	m.HandleFunc("/", s.static)
@@ -373,16 +374,10 @@ func sensorPath(p string) bool {
 func (s *Server) Handler() http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		h := w.Header()
-		// The console must never be framed. Two exceptions: an embedded widget exists to be
-		// (a page that can only read sensors, and only with a read-only token in its URL),
-		// and a proxied KVM is framed by the console itself — 'none' forbids that even from
-		// the same origin, so those paths say 'self' instead of opening up.
-		switch {
-		case r.URL.Path == "/embed/widget":
-		case strings.HasPrefix(r.URL.Path, kvmPrefix):
-			h.Set("X-Frame-Options", "SAMEORIGIN")
-			h.Set("Content-Security-Policy", "frame-ancestors 'self'")
-		default:
+		// The console must never be framed. The one exception is an embedded widget, which
+		// exists to be (a page that can only read sensors, and only with a read-only token in
+		// its URL).
+		if r.URL.Path != "/embed/widget" {
 			h.Set("X-Frame-Options", "DENY")
 			h.Set("Content-Security-Policy", "frame-ancestors 'none'")
 		}
@@ -2641,12 +2636,6 @@ func (s *Server) static(w http.ResponseWriter, r *http.Request) {
 			return
 		}
 	}
-	// A path with no file of ours behind it may belong to a KVM's page, which asks for its
-	// own assets by absolute path. Checked here rather than in the router because this
-	// handler is the catch-all — and gated on a session, since this handler is public.
-	if s.signedIn(r) && s.kvmFallback(w, r) {
-		return
-	}
 	// SPA fallback
 	w.Header().Set("Cache-Control", "no-cache")
 	http.ServeFileFS(w, r, s.ui, "index.html")

internal/api/thumbs.go

@@ -10,10 +10,16 @@ package api
 // opens every render node, whatever groups the daemon's user has.
 
 import (
+	"bytes"
 	"context"
+	"encoding/json"
 	"errors"
+	"fmt"
+	"io"
 	"log"
 	"net/http"
+	"net/http/httptest"
+	"net/url"
 	"os"
 	"os/exec"
 	"path/filepath"
@@ -25,6 +31,7 @@ import (
 
 	"hamlab/internal/media"
 	"hamlab/internal/priv"
+	"hamlab/internal/store"
 )
 
 const thumbSize = 256
@@ -117,7 +124,9 @@ func describeCaps(c media.Caps) string {
 }
 
 // make produces a preview, sharing the work when the same file is asked for twice at once.
-func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (media.Result, error) {
+// in is a path, or a URL the hub serves another machine's file at (kind then says what it
+// is); device is the one to decode on ("" any, "cpu" none).
+func (t *thumbSvc) make(ctx context.Context, root bool, key, in, kind, device string) (media.Result, error) {
 	t.mu.Lock()
 	if c, ok := t.calls[key]; ok {
 		t.mu.Unlock()
@@ -153,19 +162,29 @@ func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (media
 		c.err = err
 		return c.res, c.err
 	}
-	o := media.Opts{Size: thumbSize, Timeout: thumbTimeout}
+	o := media.Opts{Size: thumbSize, Timeout: thumbTimeout, Kind: kind}
+	caps = caps.Only(device)
 	if root {
-		c.res, c.err = media.MakeRoot(wctx, t.self, caps, path, o)
+		c.res, c.err = media.MakeRoot(wctx, t.self, caps, in, o)
 	} else {
-		c.res, c.err = media.Make(wctx, caps, path, o)
+		c.res, c.err = media.Make(wctx, caps, in, o)
 	}
 	return c.res, c.err
 }
 
 // ---- handlers ----
 
-// filesThumb: GET /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1]. v is the client's idea of
-// the file, from its listing; it makes a cache hit free of a stat, and a changed file a new URL.
+// thumbDeviceHeader carries the device a hub wants a video preview made on: the Transcode
+// device setting, when it is on the machine the file is on.
+const thumbDeviceHeader = "X-Hamlab-Device"
+
+// filesThumb: GET /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1][&cached=1]. v is the
+// client's idea of the file, from its listing; it makes a cache hit free of a stat, and a
+// changed file a new URL. cached=1 answers from the cache or 404, and makes nothing: the hub
+// asks that before having the preview made on another machine.
+//
+// A video preview is made on the Transcode device: on the hub, by the setting (another
+// machine's device reads the file through the hub); on a node, by the hub's header.
 func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
 	fsys, ok := s.fsFor(w, r)
 	if !ok {
@@ -203,7 +222,28 @@ func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
 	if s.serveCachedThumb(w, key) {
 		return
 	}
-	res, err := s.thumbs.make(r.Context(), root, key, p)
+	if r.URL.Query().Get("cached") == "1" {
+		jsonErr(w, 404, "not made yet")
+		return
+	}
+	device := r.Header.Get(thumbDeviceHeader)
+	if c := s.transcodeChoice(); c != nil && device == "" && media.Kind(p) == "video" {
+		if c.Node != "" {
+			// The hub's own file, previewed on another machine.
+			if res, ok := s.thumbElsewhere(r.Context(), c, "", p, root, key); ok {
+				if res.Failure != "" {
+					_ = s.thumbs.cache.Put(key, media.Entry{Path: p, Failure: res.Failure})
+					jsonErr(w, 422, res.Failure)
+					return
+				}
+				s.keepThumb(w, key, p, res)
+				return
+			}
+		} else {
+			device = c.Device
+		}
+	}
+	res, err := s.thumbs.make(r.Context(), root, key, p, "", device)
 	if err != nil {
 		var de *media.DecodeError
 		switch {
@@ -223,12 +263,204 @@ func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
 		}
 		return
 	}
+	s.keepThumb(w, key, p, res)
+}
+
+// keepThumb puts a preview in the cache and answers with it.
+func (s *Server) keepThumb(w http.ResponseWriter, key, p string, res media.Result) {
 	if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}); err != nil {
 		log.Printf("thumbs: cache: %v", err)
 	}
 	writeThumb(w, key, res.JPEG, res.Engine)
 }
 
+// ---- previews made on another machine (the Transcode device) ----
+
+// thumbFrom: POST /api/files/thumb-from {url, kind, key, device} — a preview of a file on
+// another machine, read through the hub at url, made here on device. Answers the JPEG (the
+// engine in X-Hamlab-Thumb), or 422 with why the file can't be previewed.
+func (s *Server) thumbFrom(w http.ResponseWriter, r *http.Request) {
+	var body struct{ URL, Kind, Key, Device string }
+	if err := json.NewDecoder(io.LimitReader(r.Body, 8192)).Decode(&body); err != nil || !strings.HasPrefix(body.URL, "http") || body.Key == "" {
+		jsonErr(w, 400, "expected {url, kind, key, device}")
+		return
+	}
+	res, err := s.thumbs.make(r.Context(), false, "from:"+body.Key, body.URL, body.Kind, body.Device)
+	if err != nil {
+		var de *media.DecodeError
+		if errors.As(err, &de) {
+			jsonErr(w, 422, de.Msg)
+		} else {
+			jsonErr(w, 503, err.Error())
+		}
+		return
+	}
+	w.Header().Set("X-Hamlab-Thumb-Codec", res.Codec)
+	w.Header().Set("X-Hamlab-Thumb-Size", fmt.Sprintf("%dx%d", res.Width, res.Height))
+	writeThumb(w, body.Key, res.JPEG, res.Engine)
+}
+
+// thumbPut: PUT /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1] — a preview another machine
+// made of this file, kept here as if made here. The body is the JPEG, or nothing with
+// X-Hamlab-Thumb-Failure saying why the file can't be previewed. Refused if the file has
+// changed since v.
+func (s *Server) thumbPut(w http.ResponseWriter, r *http.Request) {
+	fsys, ok := s.fsFor(w, r)
+	if !ok {
+		return
+	}
+	p, ok := cleanQuery(w, r, "path")
+	if !ok {
+		return
+	}
+	mt, sz, ok := parseV(r.URL.Query().Get("v"))
+	if !ok {
+		jsonErr(w, 400, "v=<mtime>-<size> is required")
+		return
+	}
+	in, err := fsys.Stat(r.Context(), p)
+	if err != nil {
+		fsErr(w, err)
+		return
+	}
+	if in.Size != sz || in.Mtime != mt {
+		jsonErr(w, 409, "the file has changed")
+		return
+	}
+	key := media.Key(p, sz, mt)
+	e := media.Entry{Path: p, Engine: r.Header.Get("X-Hamlab-Thumb"), Codec: r.Header.Get("X-Hamlab-Thumb-Codec"), Failure: r.Header.Get("X-Hamlab-Thumb-Failure")}
+	fmt.Sscanf(r.Header.Get("X-Hamlab-Thumb-Size"), "%dx%d", &e.Width, &e.Height)
+	if e.Failure == "" {
+		if e.JPEG, err = io.ReadAll(io.LimitReader(r.Body, 4<<20)); err != nil || len(e.JPEG) == 0 {
+			jsonErr(w, 400, "no picture")
+			return
+		}
+	}
+	if err := s.thumbs.cache.Put(key, e); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	w.WriteHeader(204)
+}
+
+// thumbElsewhere has a video preview made on the Transcode device's machine c.Node, of a
+// file on machine node ("" the hub), and answers it. ok is false when that machine can't
+// (unreachable, an older Ham Lab): the caller makes it where the file is instead. A file
+// that can't be previewed is a result too, with the failure in it.
+func (s *Server) thumbElsewhere(ctx context.Context, c *store.Transcode, node, p string, root bool, key string) (media.Result, bool) {
+	src, err := s.srcURL(c.Node, node, p, root)
+	if err != nil {
+		return media.Result{}, false
+	}
+	body, _ := json.Marshal(map[string]string{"url": src, "kind": "video", "key": key, "device": c.Device})
+	ctx, cancel := context.WithTimeout(ctx, thumbTimeout+10*time.Second)
+	defer cancel()
+	var code int
+	var b []byte
+	var hdr http.Header
+	if c.Node == "" {
+		rec := httptest.NewRecorder()
+		req := httptest.NewRequest("POST", "/api/files/thumb-from", bytes.NewReader(body)).WithContext(ctx)
+		s.thumbFrom(rec, req)
+		code, b, hdr = rec.Code, rec.Body.Bytes(), rec.Header()
+	} else {
+		code, b, hdr, err = s.nodes.DoHeaders(ctx, c.Node, "POST", "/api/files/thumb-from", body, http.Header{"Content-Type": []string{"application/json"}})
+		if err != nil {
+			log.Printf("thumbs: %s on %s: %v; made where it is", p, c.Node, err)
+			return media.Result{}, false
+		}
+	}
+	switch code {
+	case 200:
+		res := media.Result{JPEG: b, Engine: c.Node + " " + hdr.Get("X-Hamlab-Thumb"), Codec: hdr.Get("X-Hamlab-Thumb-Codec")}
+		fmt.Sscanf(hdr.Get("X-Hamlab-Thumb-Size"), "%dx%d", &res.Width, &res.Height)
+		if c.Node == "" {
+			res.Engine = s.hostName() + " " + hdr.Get("X-Hamlab-Thumb")
+		}
+		return res, true
+	case 422:
+		var e struct{ Error string }
+		_ = json.Unmarshal(b, &e)
+		return media.Result{Failure: e.Error}, true
+	}
+	log.Printf("thumbs: %s on %s answered %d; made where it is", p, nodeLabel(c.Node), code)
+	return media.Result{}, false
+}
+
+// hubThumb: GET /api/nodes/{name}/files/thumb — a node's preview, through the hub. Without a
+// Transcode device setting, or for an image or a sound file's cover, it goes to the node as
+// any request does. A video is made on the setting's device: the node's own (told by header),
+// or another machine's, which reads the file through the hub; the node keeps the result in
+// its own cache either way.
+func (s *Server) hubThumb(w http.ResponseWriter, r *http.Request) {
+	node := r.PathValue("name")
+	proxy, ok := s.nodes.Proxy(node)
+	if !ok {
+		jsonErr(w, 404, "no such machine")
+		return
+	}
+	c := s.transcodeChoice()
+	q := r.URL.Query()
+	p := q.Get("path")
+	if c == nil || media.Kind(p) != "video" {
+		proxy.ServeHTTP(w, r)
+		return
+	}
+	if c.Node == node {
+		r.Header.Set(thumbDeviceHeader, c.Device)
+		proxy.ServeHTTP(w, r)
+		return
+	}
+	mt, sz, okV := parseV(q.Get("v"))
+	if !okV {
+		proxy.ServeHTTP(w, r)
+		return
+	}
+	// The node's own cache first.
+	cq := url.Values{"path": {p}, "v": {q.Get("v")}, "cached": {"1"}}
+	if q.Get("sudo") == "1" {
+		cq.Set("sudo", "1")
+	}
+	hdr := http.Header{}
+	if inm := r.Header.Get("If-None-Match"); inm != "" {
+		hdr.Set("If-None-Match", inm)
+	}
+	code, b, rh, err := s.nodes.DoHeaders(r.Context(), node, "GET", "/api/files/thumb?"+cq.Encode(), nil, hdr)
+	if err == nil && code != 404 {
+		for _, k := range []string{"Content-Type", "Cache-Control", "ETag", "X-Hamlab-Thumb"} {
+			if v := rh.Get(k); v != "" {
+				w.Header().Set(k, v)
+			}
+		}
+		w.WriteHeader(code)
+		_, _ = w.Write(b)
+		return
+	}
+	key := media.Key(p, sz, mt)
+	res, made := s.thumbElsewhere(r.Context(), c, node, p, q.Get("sudo") == "1", key)
+	if !made {
+		proxy.ServeHTTP(w, r) // the node makes it itself
+		return
+	}
+	// Kept where the file is, so it is made once.
+	ph := http.Header{"X-Hamlab-Thumb": {res.Engine}, "X-Hamlab-Thumb-Codec": {res.Codec}, "X-Hamlab-Thumb-Size": {fmt.Sprintf("%dx%d", res.Width, res.Height)}}
+	if res.Failure != "" {
+		ph.Set("X-Hamlab-Thumb-Failure", res.Failure)
+	}
+	pq := url.Values{"path": {p}, "v": {q.Get("v")}}
+	if q.Get("sudo") == "1" {
+		pq.Set("sudo", "1")
+	}
+	if code, _, _, err := s.nodes.DoHeaders(r.Context(), node, "PUT", "/api/files/thumb?"+pq.Encode(), res.JPEG, ph); err != nil || code != 204 {
+		log.Printf("thumbs: keeping %s's preview of %s: %d %v", node, p, code, err)
+	}
+	if res.Failure != "" {
+		jsonErr(w, 422, res.Failure)
+		return
+	}
+	writeThumb(w, key, res.JPEG, res.Engine)
+}
+
 // serveCachedThumb answers from the cache when it can. A cache that cannot be read is a
 // miss: the preview is made again rather than refused.
 func (s *Server) serveCachedThumb(w http.ResponseWriter, key string) bool {

internal/api/zfs.go

@@ -27,6 +27,7 @@ import (
 
 type zfsState struct {
 	Policies []storage.Policy       `json:"policies"`
+	Verify   []storage.VerifyPolicy `json:"verify"` // landing-tier verify schedules (landing.go)
 }
 
 func (s *Server) zfsLoad() zfsState {
@@ -39,6 +40,9 @@ func (s *Server) zfsLoad() zfsState {
 	if z.Policies == nil {
 		z.Policies = []storage.Policy{}
 	}
+	if z.Verify == nil {
+		z.Verify = []storage.VerifyPolicy{}
+	}
 	return z
 }
 
@@ -138,11 +142,13 @@ func (s *Server) zfsOverview(w http.ResponseWriter, r *http.Request) {
 
 // zfsARC is the cache: the sampler's live ARC (size, the hit rate over the last interval)
 // beside the storage package's fuller picture (since-boot counters, tunables, and each
-// pool's cache/log/special devices or the lack of them).
+// pool's cache/log/special devices or the lack of them), and the write path — the dirty
+// buffer in RAM, each pool's write rate, and any landing tier in front of a pool.
 func (s *Server) zfsARC(w http.ResponseWriter, r *http.Request) {
-	out := map[string]any{"detail": s.storage.ARC(r.Context()), "live": nil}
+	out := map[string]any{"detail": s.storage.ARC(r.Context()), "live": nil, "writes": nil}
 	if smp, ok := s.metrics.Latest(); ok && smp.ARC != nil {
 		out["live"] = smp.ARC
+		out["writes"] = smp.ZFSWrite
 	}
 	jsonOK(w, 200, out)
 }

internal/config/config.go

@@ -77,23 +77,6 @@ type UpdateSource struct {
 	Check string `json:"check,omitempty"`
 }
 
-// KVM is an IP-KVM: a device with a web page that gives screen, keyboard and mouse access
-// to a machine, including when that machine has no OS running. Ham Lab holds the list on
-// the hub and proxies each one, because a KVM matters exactly when the machine it is
-// attached to cannot answer for itself.
-type KVM struct {
-	ID   string `json:"id"`
-	Name string `json:"name"`
-	// URL is the device's own address, as you would type it into a browser on the LAN.
-	URL string `json:"url"`
-	// Machine is the host this KVM is attached to, and is the key: one KVM per machine, shown
-	// on that machine's KVM tab. Empty means the hub itself. The list lives on the hub even
-	// for a node's KVM, because a KVM earns its keep when the machine is down and cannot
-	// answer for itself.
-	Machine string    `json:"machine"`
-	Added   time.Time `json:"added"`
-}
-
 type Config struct {
 	Listen string `json:"listen"`
 	// PasswordHash is the single login of installs before users existed. On load it becomes
@@ -103,7 +86,6 @@ type Config struct {
 	TLS          string  `json:"tls,omitempty"` // "auto" → self-signed cert in the data dir; "" → plain HTTP
 	Tokens       []Token `json:"tokens,omitempty"`
 	Nodes        []Node  `json:"nodes,omitempty"`
-	KVMs         []KVM   `json:"kvms,omitempty"`
 	// Updates: where a newer build comes from. See UpdateSource.
 	Updates UpdateSource `json:"update,omitempty"`
 

internal/media/encode.go

@@ -1,280 +1,337 @@
 package media
 
-// What to run when a file is to be re-encoded for a browser that cannot play it as it is:
-// the codecs and sizes on offer, the output size and bit rate a choice comes to for a
-// given source, the ways this machine could do it in the order worth trying, and the
-// ffmpeg argument list for one of them. All decisions, no processes, so it can be tested.
+// What to run when a video is played in the browser: the qualities it is offered in (the
+// ladder), the ways a machine could make one of them on the device asked for, in the order
+// worth trying, and the ffmpeg argument list for one piece-making run. All decisions, no
+// processes, so it can be tested. Everything is H.264 High and AAC: what every browser plays.
 
 import (
 	"fmt"
 	"math"
 	"sort"
 	"strconv"
+	"strings"
 )
 
-// Codec is one target a stream may be re-encoded to. "copy" keeps the codec and only
-// repackages: for a browser that lacks the container, not the decoder.
-type Codec struct {
-	ID    string `json:"id"`
-	Label string `json:"label"`
+// Rung is one quality the player can pick: a size, a frame rate and a bit rate.
+type Rung struct {
+	Name    string  `json:"name"`  // in URLs: the width, "1920"
+	Label   string  `json:"label"` // what the menu says: "1080s", "720p"
+	Width   int     `json:"width"`
+	Height  int     `json:"height"`
+	FPS     float64 `json:"fps"`
+	Bitrate int     `json:"bitrate"` // video, bits a second, on average
+	Peak    int     `json:"peak"`    // what the playlist promises, sound included
+	level   string  // H.264 level, "5.2"
+	codecs  string  // for the master playlist
 }
 
-var Codecs = []Codec{{"h264", "H.264"}, {"hevc", "HEVC"}, {"av1", "AV1"}, {"copy", "Original codec, repackaged"}}
-
-// Quality is a picture height; the bit rate follows from the size the output ends up at.
-type Quality struct {
-	ID     string `json:"id"`
-	Label  string `json:"label"`
-	Height int    `json:"height"` // 0: the source's own
+// One step of the ladder: the long side for a 360 video (its width), the short side for a flat
+// one, and the bit rate (Mb/s ×10) at more than 30 frames a second and at 30 or fewer.
+type step struct {
+	size   int
+	label  string
+	hi, lo int
+	cap30  bool
 }
 
-var Qualities = []Quality{{"original", "Original size", 0}, {"1080p", "1080p", 1080}, {"720p", "720p", 720}, {"480p", "480p", 480}}
-
-func qualityByID(id string) (Quality, bool) {
-	for _, q := range Qualities {
-		if q.ID == id {
-			return q, true
-		}
-	}
-	return Quality{}, false
+// A 360 picture is seen a quarter of its width at a time, so it needs more pixels than a flat
+// one for the same sharpness; YouTube names its sizes by the flat picture they look like
+// ("1080s"), and so does this.
+var vrSteps = []step{
+	{3840, "2160s", 220, 160, false},
+	{2880, "1440s", 120, 90, false},
+	{1920, "1080s", 60, 45, false},
+	{1280, "720s", 25, 25, true},
 }
 
-// ValidChoice says whether a codec and quality are ones on offer — checked before any work.
-func ValidChoice(codec, quality string) error {
-	if !codecKnown(codec) {
-		return fmt.Errorf("unknown codec %q", codec)
-	}
-	if _, ok := qualityByID(quality); !ok {
-		return fmt.Errorf("unknown quality %q", quality)
+var flatSteps = []step{
+	{2160, "2160p", 160, 120, false},
+	{1440, "1440p", 90, 65, false},
+	{1080, "1080p", 50, 35, false},
+	{720, "720p", 25, 25, true},
+	{480, "480p", 12, 12, true},
 }
+
+const audioRate = 160_000
+
+func even(x float64) int { return int(math.Round(x/2)) * 2 }
+
+// Ladder is the qualities a video is offered in, best first: every step no bigger than the
+// picture itself, or the picture's own size if it is smaller than all of them.
+func Ladder(i Info) []Rung {
+	v := i.Video
+	if v == nil || v.Width <= 0 || v.Height <= 0 {
 		return nil
 	}
-
-func codecKnown(id string) bool {
-	for _, c := range Codecs {
-		if c.ID == id {
-			return true
+	vr := v.Spherical
+	steps := flatSteps
+	w, h := v.Shown()
+	src := min(w, h) // a flat picture is named by its short side
+	if vr {
+		steps, src = vrSteps, w
 	}
+	// A picture between two steps (a 2292×960 capture) gets its own size on top, at the bit
+	// rate of the step above, so the best on offer is never less than the file.
+	var out []Rung
+	for k, s := range steps {
+		if s.size > src {
+			continue
 		}
-	return false
+		if len(out) == 0 && s.size < src && k > 0 { // never above the top step: H.264 stops there
+			own := steps[k-1]
+			own.size, own.label, own.cap30 = src, strconv.Itoa(src)+map[bool]string{true: "s", false: "p"}[vr], false
+			out = append(out, rung(i, own))
+		}
+		out = append(out, rung(i, s))
+	}
+	if len(out) == 0 {
+		last := steps[len(steps)-1]
+		last.size, last.label = src, strconv.Itoa(src)+map[bool]string{true: "s", false: "p"}[vr]
+		out = append(out, rung(i, last))
+	}
+	return out
 }
 
-// fit is the output: never larger than the source, even dimensions, and a bit rate by the
-// size it comes to — a quarter more at 50 fps and up, less for the codecs that need less.
-func fit(q Quality, src Info, codec string) (w, h, kbps int) {
-	w, h = src.Video.Width, src.Video.Height
-	if q.Height > 0 && h > q.Height {
-		w = int(math.Round(float64(w) * float64(q.Height) / float64(h)))
-		h = q.Height
-	}
-	w, h = w&^1, h&^1
+func rung(i Info, s step) Rung {
+	v := i.Video
+	sw, sh := v.Shown()
+	var w, h int
 	switch {
-	case h >= 2160:
-		kbps = 20000
-	case h >= 1440:
-		kbps = 12000
-	case h >= 1080:
-		kbps = 8000
-	case h >= 720:
-		kbps = 4000
-	default:
-		kbps = 2000
+	case v.Spherical:
+		w, h = s.size, even(float64(s.size)*float64(sh)/float64(sw))
+	case sw >= sh:
+		h, w = s.size, even(float64(s.size)*float64(sw)/float64(sh))
+	default: // standing up
+		w, h = s.size, even(float64(s.size)*float64(sh)/float64(sw))
 	}
-	if src.Video.FPS >= 50 {
-		kbps = kbps * 5 / 4
+	fps := v.FPS
+	if fps <= 0 {
+		fps = 30
 	}
-	if codec == "hevc" || codec == "av1" {
-		kbps = kbps * 6 / 10
+	for fps > 60.5 || (s.cap30 && fps > 30.5) {
+		fps /= 2
 	}
-	return w, h, kbps
+	fps = math.Round(fps*1000) / 1000
+	br := s.lo
+	if fps > 30.5 {
+		br = s.hi
+	}
+	r := Rung{Name: strconv.Itoa(w), Label: s.label, Width: w, Height: h, FPS: fps, Bitrate: br * 100_000}
+	r.Peak = r.Bitrate*3/2 + audioRate
+	r.level = level(w, h, fps)
+	lv, _ := strconv.ParseFloat(r.level, 64)
+	r.codecs = fmt.Sprintf("avc1.6400%02x", int(math.Round(lv*10)))
+	if i.Audio != nil {
+		r.codecs += ",mp4a.40.2"
+	}
+	return r
 }
 
-// Recipe is one way to make the stream: the engine's name for the record, and the ffmpeg
-// arguments that go before the input, the video filter, and the video codec.
+// level is the lowest H.264 level that takes this many macroblocks a frame and a second.
+func level(w, h int, fps float64) string {
+	mbs := ((w + 15) / 16) * ((h + 15) / 16)
+	rate := float64(mbs) * fps
+	for _, l := range []struct {
+		name  string
+		mbps  float64
+		frame int
+	}{
+		{"3.1", 108000, 3600}, {"3.2", 216000, 5120}, {"4.0", 245760, 8192}, {"4.2", 522240, 8704},
+		{"5.0", 589824, 22080}, {"5.1", 983040, 36864}, {"5.2", 2073600, 36864},
+	} {
+		if rate <= l.mbps && mbs <= l.frame {
+			return l.name
+		}
+	}
+	return "6.2"
+}
+
+// fpsArg is a frame rate as ffmpeg's fps filter takes it, exactly for the NTSC rates.
+func fpsArg(f float64) string {
+	for _, n := range []float64{24, 30, 60, 120} {
+		if math.Abs(f-n*1000/1001) < 0.01 {
+			return fmt.Sprintf("%d/1001", int(n*1000))
+		}
+	}
+	return strings.TrimRight(strings.TrimRight(strconv.FormatFloat(f, 'f', 3, 64), "0"), ".")
+}
+
+// Recipe is one way to make a quality: the engine's name for the record, the device it runs
+// on ("cpu" for the software encoder), and the ffmpeg arguments that go before the input, the
+// video filter, and the video codec.
 type Recipe struct {
 	Engine string
+	Device string
 	Pre    []string
 	VF     string
 	Video  []string
 }
 
-// recipes are the ways to try, best first: a card on the bus before the integrated GPU;
-// on a GPU, decoding on the same card (frames never leave it) before decoding in software
-// and uploading; the software encoders last. An empty list means nothing here can.
-func recipes(c Caps, codec string, src Info, w, h, kbps int) []Recipe {
-	if codec == "copy" {
-		v := []string{"-c:v", "copy"}
-		if src.Video.Codec == "hevc" {
-			v = append(v, "-tag:v", "hvc1") // Safari reads hvc1, not hev1
+// CPU is the device name for software encoding.
+const CPU = "cpu"
+
+// Device is one thing a machine can make H.264 on, for the Transcode device setting.
+type Device struct {
+	ID         string `json:"id"`   // a render node, or "cpu"
+	Name       string `json:"name"` // "Intel Arc A380", "CPU (libx264)"
+	Integrated bool   `json:"integrated,omitempty"`
+	OK         bool   `json:"ok"`
+	Reason     string `json:"reason,omitempty"` // when not OK: why
 }
-		return []Recipe{{Engine: "Original codec", Video: v}}
+
+// Devices lists what this machine could make the player's H.264 on: every GPU, usable or not
+// (and why not), then the CPU.
+func Devices(c Caps) []Device {
+	var out []Device
+	if c.Error != "" {
+		return []Device{{ID: CPU, Name: "CPU", Reason: c.Error}}
 	}
-	var out []Recipe
-	gpus := append([]GPU(nil), c.GPUs...)
-	sort.SliceStable(gpus, func(i, j int) bool { return !gpus[i].Integrated && gpus[j].Integrated })
-	rate := []string{"-b:v", strconv.Itoa(kbps) + "k", "-maxrate", strconv.Itoa(kbps*5/4) + "k", "-bufsize", strconv.Itoa(kbps*5/2) + "k"}
-	for i := range gpus {
-		g := &gpus[i]
-		if g.API != "vaapi" || !has(g.Encodes, codec) {
-			continue
+	for _, g := range c.GPUs {
+		d := Device{ID: g.Device, Name: g.Name, Integrated: g.Integrated}
+		if d.Name == "" {
+			d.Name = g.Device
 		}
-		video := append([]string{"-c:v", codec + "_vaapi"}, rate...)
-		switch codec {
-		case "h264":
-			video = append(video, "-profile:v", "high")
-		case "hevc":
-			video = append(video, "-tag:v", "hvc1")
+		switch {
+		case g.API != "vaapi":
+			d.Reason = "only VAAPI devices encode here"
+		case g.Encodes != nil && !has(g.Encodes, "h264"):
+			d.Reason = "doesn't encode H.264"
+		default:
+			d.OK = true
 		}
+		out = append(out, d)
+	}
+	for _, g := range c.Unusable {
 		name := g.Name
 		if name == "" {
 			name = g.Device
 		}
-		scale := fmt.Sprintf("scale_vaapi=w=%d:h=%d:format=nv12", w, h)
-		if has(g.Codecs, src.Video.Codec) {
-			vf := scale
-			if src.HDR() {
-				vf = "tonemap_vaapi=format=nv12:t=bt709:p=bt709:m=bt709," + vf
+		out = append(out, Device{ID: g.Device, Name: name, Integrated: g.Integrated, Reason: g.Reason})
 	}
-			out = append(out, Recipe{Engine: name + " (vaapi)", Pre: []string{"-hwaccel", "vaapi", "-hwaccel_device", g.Device, "-hwaccel_output_format", "vaapi"}, VF: vf, Video: video})
+	d := Device{ID: CPU, Name: "CPU (libx264)", OK: has(c.Encoders, "libx264")}
+	if !d.OK {
+		d.Reason = "this ffmpeg has no libx264"
 	}
-		out = append(out, Recipe{Engine: name + " (vaapi, decoded on the CPU)", Pre: []string{"-init_hw_device", "vaapi=va:" + g.Device, "-filter_hw_device", "va"},
-			VF: fmt.Sprintf("format=nv12,scale=%d:%d,hwupload", w, h), Video: video})
+	return append(out, d)
 }
-	swVF := fmt.Sprintf("scale=%d:%d,format=yuv420p", w, h)
-	if src.HDR() {
-		swVF = fmt.Sprintf("zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable,zscale=t=bt709:m=bt709,scale=%d:%d,format=yuv420p", w, h)
+
+// recipes are the ways to make quality r of src on device, best first: on a GPU, decoding on
+// the same card (frames never leave it) before decoding in software and uploading; the CPU
+// last, as the fallback whatever was asked. device "" is any: a card on the bus before the
+// integrated GPU. An empty list means nothing here can.
+func recipes(c Caps, device string, src Info, r Rung) []Recipe {
+	var out []Recipe
+	gpus := append([]GPU(nil), c.GPUs...)
+	sort.SliceStable(gpus, func(i, j int) bool { return !gpus[i].Integrated && gpus[j].Integrated })
+	kbps := r.Bitrate / 1000
+	rate := []string{"-b:v", strconv.Itoa(kbps) + "k", "-maxrate", strconv.Itoa(kbps*3/2) + "k", "-bufsize", strconv.Itoa(kbps*2) + "k",
+		"-profile:v", "high", "-level", r.level}
+	fps := ""
+	if src.Video == nil || src.Video.FPS <= 0 || math.Abs(src.Video.FPS-r.FPS) > 0.01 {
+		fps = "fps=" + fpsArg(r.FPS) + ","
 	}
-	cap := []string{"-maxrate", strconv.Itoa(kbps*5/4) + "k", "-bufsize", strconv.Itoa(kbps*5/2) + "k"}
-	switch {
-	case codec == "h264" && has(c.Encoders, "libx264"):
-		out = append(out, Recipe{Engine: "CPU (libx264)", VF: swVF, Video: append([]string{"-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-profile:v", "high"}, cap...)})
-	case codec == "hevc" && has(c.Encoders, "libx265"):
-		out = append(out, Recipe{Engine: "CPU (libx265)", VF: swVF, Video: append([]string{"-c:v", "libx265", "-preset", "fast", "-crf", "26", "-tag:v", "hvc1"}, cap...)})
-	case codec == "av1" && has(c.Encoders, "libsvtav1"):
-		out = append(out, Recipe{Engine: "CPU (SVT-AV1)", VF: swVF, Video: append([]string{"-c:v", "libsvtav1", "-preset", "8", "-crf", "35"}, cap...)})
+	if device != CPU {
+		for i := range gpus {
+			g := &gpus[i]
+			// Encodes is vainfo's answer; without vainfo there is none, and the card is tried
+			// anyway, with the next recipe behind it.
+			if g.API != "vaapi" || (g.Encodes != nil && !has(g.Encodes, "h264")) || (device != "" && g.Device != device) {
+				continue
+			}
+			video := append(append([]string{"-c:v", "h264_vaapi"}, rate...), "-bf", "0")
+			name := g.Name
+			if name == "" {
+				name = g.Device
+			}
+			// The card doesn't turn a phone's picture by itself (ffmpeg's own turning is for
+			// pictures in memory), so it is told to, and ffmpeg is told not to.
+			scale := turn(src) + fmt.Sprintf("scale_vaapi=w=%d:h=%d:format=nv12", r.Width, r.Height)
+			// HDR10 (PQ) is tone-mapped on the card, which needs the file's mastering data;
+			// HLG is made to look right on an ordinary screen as it is, so it is only scaled
+			// (tonemap_vaapi refuses it). Either way the next recipe tone-maps in software.
+			if src.Video != nil && has(g.Codecs, src.Video.Codec) {
+				vf := fps + scale
+				if src.PQ() {
+					vf = fps + "tonemap_vaapi=format=nv12:t=bt709:p=bt709:m=bt709," + scale
+				}
+				out = append(out, Recipe{Engine: name, Device: g.Device, Pre: []string{"-noautorotate", "-hwaccel", "vaapi", "-hwaccel_device", g.Device, "-hwaccel_output_format", "vaapi"}, VF: vf, Video: video})
+			}
+			cpuVF := fmt.Sprintf("%sformat=nv12,scale=%d:%d,hwupload", fps, r.Width, r.Height)
+			if src.PQ() {
+				cpuVF = fps + toneMap + fmt.Sprintf(",scale=%d:%d,format=nv12,hwupload", r.Width, r.Height)
+			}
+			out = append(out, Recipe{Engine: name + ", decoded on the CPU", Device: g.Device, Pre: []string{"-init_hw_device", "vaapi=va:" + g.Device, "-filter_hw_device", "va"},
+				VF: cpuVF, Video: video})
+		}
+	}
+	if has(c.Encoders, "libx264") {
+		vf := fmt.Sprintf("%sscale=%d:%d,format=yuv420p", fps, r.Width, r.Height)
+		if src.PQ() {
+			vf = fps + toneMap + fmt.Sprintf(",scale=%d:%d,format=yuv420p", r.Width, r.Height)
+		}
+		out = append(out, Recipe{Engine: "CPU (libx264)", Device: CPU, VF: vf, Video: append([]string{"-c:v", "libx264", "-preset", "veryfast", "-sc_threshold", "0"}, rate...)})
 	}
 	return out
 }
 
-// EngineFor is the engine the picker will say a codec runs on, "" when none can.
-func EngineFor(c Caps, codec string) string {
-	if c.Error != "" {
+// turn is the card's filter that turns a picture the way it is meant to be shown.
+func turn(src Info) string {
+	if src.Video == nil {
 		return ""
 	}
-	if codec == "copy" {
-		return "Original codec"
+	switch src.Video.Rotation {
+	case 90:
+		return "transpose_vaapi=dir=clock,"
+	case 180:
+		return "transpose_vaapi=dir=reversal,"
+	case 270:
+		return "transpose_vaapi=dir=cclock,"
 	}
-	r := recipes(c, codec, Info{Video: &Track{Codec: "h264", Width: 1920, Height: 1080}}, 1920, 1080, 8000)
-	if len(r) == 0 {
 	return ""
 }
-	return r[0].Engine
-}
 
-// hlsArgs is the whole ffmpeg argument list for one recipe: the input, the first video and
-// audio streams, the picture through the recipe, sound as AAC unless it already is, a
-// keyframe every segment, and an event playlist of fragmented-MP4 segments in dir.
-func hlsArgs(dir, in string, src Info, r Recipe, copyCodec bool) []string {
-	a := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
-	a = append(a, r.Pre...)
-	a = append(a, "-i", in, "-map", "0:v:0", "-map", "0:a:0?", "-sn", "-dn")
-	if r.VF != "" {
-		a = append(a, "-vf", r.VF)
-	}
-	a = append(a, r.Video...)
-	switch {
-	case src.Audio == nil:
-		a = append(a, "-an")
-	case src.Audio.Codec == "aac":
-		a = append(a, "-c:a", "copy")
-	default:
-		a = append(a, "-c:a", "aac", "-b:a", "160k", "-ac", "2")
-	}
-	if !copyCodec {
-		fps := src.Video.FPS
-		if fps <= 0 {
-			fps = 30
-		}
-		gop := int(math.Round(fps * hlsSeconds))
-		if gop < 24 {
-			gop = 24
-		}
-		if gop > 300 {
-			gop = 300
-		}
-		a = append(a, "-g", strconv.Itoa(gop), "-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%d)", hlsSeconds))
-	}
-	a = append(a, "-f", "hls", "-hls_time", strconv.Itoa(hlsSeconds), "-hls_playlist_type", "event", "-hls_segment_type", "fmp4",
-		"-hls_fmp4_init_filename", "init.mp4", "-hls_segment_filename", dir+"/seg%05d.m4s", "-hls_flags", "independent_segments+temp_file",
-		"-progress", dir+"/progress", "-stats_period", "1", dir+"/index.m3u8")
-	return a
-}
+// toneMap turns an HDR10 picture into an ordinary one, in software.
+const toneMap = "zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable,zscale=t=bt709:m=bt709:r=tv"
 
-const hlsSeconds = 4
+// Seg is the length of a piece, in seconds.
+const Seg = 3.0
 
-// tsBase is added to every run's timestamps. A run from the top would otherwise start just
-// below zero (the B-frame delay) and the muxer would shift it to make room, while a run from a
-// seek point would not: with the base, no run is ever shifted, and a segment means the same
-// stretch of the film whichever run made it.
+// tsBase is added to every run's times, so pieces from different runs follow on: without it
+// a run from the top would start just below zero (the B-frame and AAC delay) and the muxer
+// would shift that run alone.
 const tsBase = 10
 
-// segArgs is one run of a seekable re-encode: segments from..to-1 of the fixed 4 s grid.
-// Every run is made the same way, the first included, so their segments are interchangeable:
-// an accurate seek to from×4 s, timestamps put back where that is in the film (+ tsBase),
-// kept as they are in each fragment (frag_discont), numbered from `from`, and an end just
-// short of segment `to`, which is already on disk, so the run never writes over it. Segments
-// land in dir as seg%05d.m4s; init is the init segment's name (only the first run's is used).
-func segArgs(dir, in string, src Info, r Recipe, from, to, n int, init string) []string {
-	fps := 30.0
-	if src.Video != nil && src.Video.FPS > 0 {
-		fps = src.Video.FPS
-	}
-	start := from * hlsSeconds
+// pieceArgs is one run making quality r from piece n on, into dir as <n>.ts: an accurate seek
+// to n×Seg, a keyframe on every Seg mark and nowhere else (so every piece starts on one, and
+// pieces from any run and quality line up), times put back where that is in the film. in is a
+// path or, for a file on another machine, an http URL.
+func pieceArgs(dir, in string, src Info, rc Recipe, r Rung, n int) []string {
+	at := strconv.FormatFloat(float64(n)*Seg, 'f', 3, 64)
 	a := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
-	a = append(a, r.Pre...)
-	a = append(a, "-ss", strconv.Itoa(start), "-i", in, "-map", "0:v:0", "-map", "0:a:0?", "-sn", "-dn")
-	if r.VF != "" {
-		a = append(a, "-vf", r.VF)
+	a = append(a, rc.Pre...)
+	if strings.HasPrefix(in, "http://") || strings.HasPrefix(in, "https://") {
+		a = append(a, "-reconnect", "1", "-reconnect_on_network_error", "1", "-reconnect_delay_max", "4")
 	}
-	a = append(a, r.Video...)
-	switch {
-	case src.Audio == nil:
+	a = append(a, "-ss", at, "-i", in, "-map", "0:v:0", "-map", "0:a:0?", "-sn", "-dn")
+	if rc.VF != "" {
+		a = append(a, "-vf", rc.VF)
+	}
+	a = append(a, rc.Video...)
+	if src.Audio == nil {
 		a = append(a, "-an")
-	case src.Audio.Codec == "aac":
-		a = append(a, "-c:a", "copy")
-	default:
+	} else {
 		a = append(a, "-c:a", "aac", "-b:a", "160k", "-ac", "2")
 	}
-	gop := int(math.Round(fps * hlsSeconds))
-	if gop < 24 {
-		gop = 24
-	}
-	if gop > 300 {
-		gop = 300
-	}
-	a = append(a, "-g", strconv.Itoa(gop), "-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%d)", hlsSeconds))
-	if to < n {
-		a = append(a, "-t", strconv.FormatFloat(float64((to-from)*hlsSeconds)-0.5/fps, 'f', 3, 64))
-	}
-	a = append(a, "-output_ts_offset", strconv.Itoa(start+tsBase),
-		"-f", "hls", "-hls_time", strconv.Itoa(hlsSeconds), "-hls_playlist_type", "event", "-hls_segment_type", "fmp4",
-		"-hls_segment_options", "movflags=+frag_discont", "-start_number", strconv.Itoa(from),
-		"-hls_fmp4_init_filename", init, "-hls_segment_filename", dir+"/seg%05d.m4s", "-hls_flags", "independent_segments+temp_file",
-		"-progress", dir+"/progress", "-stats_period", "1", dir+"/run.m3u8")
+	gop := int(math.Ceil(r.FPS*Seg)) * 4
+	a = append(a, "-g", strconv.Itoa(gop), "-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%g)", Seg),
+		"-output_ts_offset", strconv.FormatFloat(float64(n)*Seg+tsBase, 'f', 3, 64),
+		"-f", "hls", "-hls_time", strconv.FormatFloat(Seg-0.5, 'f', 2, 64), "-hls_list_size", "0",
+		"-hls_flags", "temp_file", "-hls_segment_type", "mpegts", "-start_number", strconv.Itoa(n),
+		"-hls_segment_filename", dir+"/%d.ts", dir+"/ffmpeg.m3u8")
 	return a
 }
 
-// segCount is how many 4 s segments a seekable re-encode of a file this long lists. The last
-// 0.3 s are left out of the count, so a container whose sound runs on past the picture never
-// lists a segment no frame starts.
-func segCount(duration float64) int {
-	d := duration - 0.3
-	if d < 0 {
-		d = 0
-	}
-	return int(d/hlsSeconds) + 1
-}
-
 func has(list []string, x string) bool {
 	for _, v := range list {
 		if v == x {

internal/media/encode_test.go

@@ -9,25 +9,40 @@ func src(codec string, w, h int, fps float64) Info {
 	return Info{Duration: 100, Video: &Track{Codec: codec, Width: w, Height: h, FPS: fps}, Audio: &Track{Codec: "opus", Channels: 2}}
 }
 
-func TestFit(t *testing.T) {
-	q := func(id string) Quality { x, _ := qualityByID(id); return x }
-	for _, c := range []struct {
-		q         string
-		in        Info
-		codec     string
-		w, h, kbs int
-	}{
-		{"1080p", src("hevc", 3840, 2160, 60), "h264", 1920, 1080, 10000},
-		{"original", src("hevc", 3840, 2160, 30), "h264", 3840, 2160, 20000},
-		{"720p", src("h264", 2292, 960, 60), "h264", 1718, 720, 5000},  // odd width made even
-		{"1080p", src("h264", 1280, 720, 30), "h264", 1280, 720, 4000}, // never upscaled
-		{"480p", src("h264", 1920, 1080, 30), "hevc", 852, 480, 1200},  // 853.3 rounds down, then even
-		{"original", src("av1", 640, 360, 24), "av1", 640, 360, 1200},
-	} {
-		w, h, k := fit(q(c.q), c.in, c.codec)
-		if w != c.w || h != c.h || k != c.kbs {
-			t.Errorf("%s %s %dx%d@%.0f → %dx%d %dk, want %dx%d %dk", c.q, c.codec, c.in.Video.Width, c.in.Video.Height, c.in.Video.FPS, w, h, k, c.w, c.h, c.kbs)
+func TestLadder(t *testing.T) {
+	vr := src("hevc", 7680, 3840, 50)
+	vr.Video.Spherical = true
+	r := Ladder(vr)
+	if len(r) != 4 || r[0].Name != "3840" || r[0].Height != 1920 || r[0].level != "5.2" || r[3].FPS != 25 || r[0].Label != "2160s" {
+		t.Fatalf("8K 360: %+v", r)
 	}
+	if r[0].codecs != "avc1.640034,mp4a.40.2" {
+		t.Fatalf("codecs %q", r[0].codecs)
+	}
+	r = Ladder(src("h264", 2292, 960, 60)) // a capture: its own size on top, then the steps under it
+	if len(r) != 3 || r[0].Label != "960p" || r[0].Width != 2292 || r[0].FPS != 60 || r[0].Bitrate != 5_000_000 || r[1].Label != "720p" || r[1].Width != 1720 || r[1].FPS != 30 {
+		t.Fatalf("2292×960: %+v", r)
+	}
+	r = Ladder(src("h264", 1920, 1080, 59.94))
+	if len(r) != 3 || r[0].Label != "1080p" || r[0].FPS != 59.94 || r[1].FPS != 29.97 || r[2].Width != 854 {
+		t.Fatalf("1080p: %+v", r)
+	}
+	if r = Ladder(src("h264", 1080, 1920, 30)); r[0].Width != 1080 || r[0].Height != 1920 { // standing up
+		t.Fatalf("portrait: %+v", r)
+	}
+	if r = Ladder(src("h264", 320, 240, 30)); len(r) != 1 || r[0].Width != 320 || r[0].Label != "240p" {
+		t.Fatalf("tiny: %+v", r)
+	}
+	silent := src("h264", 1280, 720, 30)
+	silent.Audio = nil
+	if r = Ladder(silent); strings.Contains(r[0].codecs, "mp4a") {
+		t.Fatalf("no sound, no AAC: %q", r[0].codecs)
+	}
+	if Ladder(Info{Duration: 10}) != nil {
+		t.Fatal("no picture, no ladder")
+	}
+	if fpsArg(29.97) != "30000/1001" || fpsArg(25) != "25" {
+		t.Fatal(fpsArg(29.97), fpsArg(25))
 	}
 }
 
@@ -47,111 +62,110 @@ func engineNames(r []Recipe) []string {
 	return out
 }
 
-func TestRecipesOrder(t *testing.T) {
+func TestRecipes(t *testing.T) {
 	c := gridCaps()
-	// h264 from hevc: the card first, decoding on it; then the iGPU the same way; then the CPU
-	got := engineNames(recipes(c, "h264", src("hevc", 3840, 2160, 60), 1920, 1080, 10000))
-	want := []string{"Intel Arc A380 (vaapi)", "Intel Arc A380 (vaapi, decoded on the CPU)", "Intel UHD Graphics 750 (vaapi)", "Intel UHD Graphics 750 (vaapi, decoded on the CPU)", "CPU (libx264)"}
-	if strings.Join(got, "|") != strings.Join(want, "|") {
-		t.Fatalf("got %v", got)
+	in := src("hevc", 3840, 2160, 60)
+	r1080 := Ladder(in)[2]
+	// any device: the card first, decoding on it; then the iGPU the same way; then the CPU
+	got := strings.Join(engineNames(recipes(c, "", in, r1080)), "|")
+	if got != "Intel Arc A380|Intel Arc A380, decoded on the CPU|Intel UHD Graphics 750|Intel UHD Graphics 750, decoded on the CPU|CPU (libx264)" {
+		t.Fatalf("any: %s", got)
 	}
-	// av1 source: neither card decodes it, so no same-card recipe, and only the Arc encodes av1
-	got = engineNames(recipes(c, "av1", src("av1", 1920, 1080, 30), 1920, 1080, 4800))
-	want = []string{"Intel Arc A380 (vaapi, decoded on the CPU)", "CPU (SVT-AV1)"}
-	if strings.Join(got, "|") != strings.Join(want, "|") {
-		t.Fatalf("av1: got %v", got)
+	// the iGPU asked for: only it, then the CPU as the fallback
+	if got = strings.Join(engineNames(recipes(c, "/dev/dri/renderD128", in, r1080)), "|"); got != "Intel UHD Graphics 750|Intel UHD Graphics 750, decoded on the CPU|CPU (libx264)" {
+		t.Fatalf("iGPU: %s", got)
 	}
-	// copy is one recipe, no filter, hvc1 for hevc
-	r := recipes(c, "copy", src("hevc", 1920, 1080, 30), 1920, 1080, 0)
-	if len(r) != 1 || r[0].VF != "" || !has(r[0].Video, "hvc1") || !has(r[0].Video, "copy") {
-		t.Fatalf("copy: %+v", r)
+	if got = strings.Join(engineNames(recipes(c, CPU, in, r1080)), "|"); got != "CPU (libx264)" {
+		t.Fatalf("cpu: %s", got)
 	}
-	// a machine with no GPU and no x264: nothing for h264, something for av1
-	bare := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac", "libsvtav1"}}
-	if len(recipes(bare, "h264", src("h264", 1280, 720, 30), 1280, 720, 4000)) != 0 {
-		t.Fatal("no x264 means no h264")
+	// an AV1 source: neither card decodes it, so no same-card recipe
+	if got = strings.Join(engineNames(recipes(c, "/dev/dri/renderD129", src("av1", 1920, 1080, 30), r1080)), "|"); got != "Intel Arc A380, decoded on the CPU|CPU (libx264)" {
+		t.Fatalf("av1: %s", got)
 	}
-	if EngineFor(bare, "av1") != "CPU (SVT-AV1)" || EngineFor(bare, "h264") != "" || EngineFor(bare, "copy") != "Original codec" {
-		t.Fatal("engineFor")
+	// a machine with no GPU and no x264: nothing
+	bare := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac"}}
+	if len(recipes(bare, "", in, r1080)) != 0 {
+		t.Fatal("no x264 means nothing")
 	}
-	if EngineFor(c, "hevc") != "Intel Arc A380 (vaapi)" {
-		t.Fatalf("engineFor on grid: %q", EngineFor(c, "hevc"))
-	}
-	// HDR gets tone-mapped on both paths
+	// 30 fps from 60 for the lower rungs; HDR tone-mapped on both paths
+	r720 := Ladder(in)[3]
 	hdr := src("hevc", 3840, 2160, 60)
 	hdr.Video.ColorTransfer = "smpte2084"
-	r = recipes(c, "h264", hdr, 1920, 1080, 10000)
-	if !strings.HasPrefix(r[0].VF, "tonemap_vaapi") || !strings.Contains(r[len(r)-1].VF, "tonemap=hable") {
-		t.Fatalf("hdr: %q … %q", r[0].VF, r[len(r)-1].VF)
+	rs := recipes(c, "", hdr, r720)
+	if !strings.HasPrefix(rs[0].VF, "fps=30,tonemap_vaapi") || !strings.Contains(rs[1].VF, "tonemap=hable") || !strings.HasSuffix(rs[1].VF, "hwupload") || !strings.Contains(rs[len(rs)-1].VF, "tonemap=hable") {
+		t.Fatalf("hdr: %q … %q … %q", rs[0].VF, rs[1].VF, rs[len(rs)-1].VF)
+	}
+	hdr.Video.ColorTransfer = "arib-std-b67" // HLG: shown as it is, on every path
+	if rs = recipes(c, "", hdr, r720); strings.Contains(rs[0].VF, "tonemap") || strings.Contains(rs[1].VF, "tonemap") {
+		t.Fatalf("hlg: %q … %q", rs[0].VF, rs[1].VF)
+	}
+	// a phone's picture, upright: the ladder by the shown size, turned on the card
+	phone := src("hevc", 1920, 1080, 30)
+	phone.Video.Rotation = 90
+	lr := Ladder(phone)
+	if lr[0].Width != 1080 || lr[0].Height != 1920 {
+		t.Fatalf("upright: %+v", lr[0])
+	}
+	if rs = recipes(c, "", phone, lr[0]); !strings.HasPrefix(rs[0].VF, "transpose_vaapi=dir=clock,scale_vaapi=w=1080:h=1920") || rs[0].Pre[0] != "-noautorotate" || strings.Contains(rs[1].VF, "transpose") {
+		t.Fatalf("turned: %q %v / %q", rs[0].VF, rs[0].Pre, rs[1].VF)
+	}
+	// no vainfo: the card is tried all the same
+	nov := gridCaps()
+	nov.GPUs[1].Encodes = nil
+	if rs = recipes(nov, "/dev/dri/renderD129", in, r1080); rs[0].Engine != "Intel Arc A380" || !Devices(nov)[1].OK {
+		t.Fatalf("no vainfo: %v", engineNames(rs))
+	}
+	if rs := recipes(c, "", in, r1080); strings.Contains(rs[0].VF, "fps=") {
+		t.Fatalf("same rate needs no fps filter: %q", rs[0].VF)
 	}
 }
 
-func TestHLSArgs(t *testing.T) {
+func TestDevices(t *testing.T) {
+	c := gridCaps()
+	c.Unusable = []GPU{{Device: "/dev/dri/renderD130", Name: "Something", Reason: "permission denied"}}
+	d := Devices(c)
+	if len(d) != 4 || !d[0].OK || !d[1].OK || d[2].OK || d[2].Reason == "" || d[3].ID != CPU || !d[3].OK {
+		t.Fatalf("%+v", d)
+	}
+	if d := Devices(Caps{Error: "ffmpeg is not installed"}); len(d) != 1 || d[0].OK {
+		t.Fatalf("no ffmpeg: %+v", d)
+	}
+}
+
+func TestPieceArgs(t *testing.T) {
 	in := src("h264", 1920, 1080, 60)
-	r := recipes(gridCaps(), "h264", in, 1920, 1080, 10000)[0]
-	a := strings.Join(hlsArgs("/d", "/v.mkv", in, r, false), " ")
-	for _, want := range []string{"-hwaccel vaapi -hwaccel_device /dev/dri/renderD129", "-i /v.mkv", "-map 0:v:0 -map 0:a:0?", "-vf scale_vaapi=w=1920:h=1080:format=nv12", "-c:v h264_vaapi -b:v 10000k -maxrate 12500k", "-c:a aac -b:a 160k -ac 2", "-g 240 -force_key_frames expr:gte(t,n_forced*4)", "-hls_segment_type fmp4", "-hls_segment_filename /d/seg%05d.m4s", "/d/index.m3u8"} {
+	r := Ladder(in)[0]
+	rc := recipes(gridCaps(), "/dev/dri/renderD129", in, r)[0]
+	a := strings.Join(pieceArgs("/d", "/v.mkv", in, rc, r, 12), " ")
+	for _, want := range []string{"-hwaccel vaapi -hwaccel_device /dev/dri/renderD129", "-ss 36.000 -i /v.mkv", "-map 0:v:0 -map 0:a:0?",
+		"-vf scale_vaapi=w=1920:h=1080:format=nv12", "-c:v h264_vaapi -b:v 5000k -maxrate 7500k", "-level 4.2", "-c:a aac -b:a 160k -ac 2",
+		"-force_key_frames expr:gte(t,n_forced*3)", "-output_ts_offset 46.000", "-hls_time 2.50", "-start_number 12", "-hls_segment_filename /d/%d.ts"} {
 		if !strings.Contains(a, want) {
 			t.Errorf("missing %q in\n%s", want, a)
 		}
 	}
-	// aac stays as it is; no audio means none; copy sets no keyframes
-	in.Audio.Codec = "aac"
-	if a := strings.Join(hlsArgs("/d", "/v", in, r, false), " "); !strings.Contains(a, "-c:a copy") {
-		t.Error("aac should be copied")
+	if strings.Contains(a, "-reconnect") {
+		t.Error("a path needs no reconnect")
+	}
+	if a := strings.Join(pieceArgs("/d", "http://10.0.0.234:8100/api/play-src/x", in, rc, r, 0), " "); !strings.Contains(a, "-reconnect 1") {
+		t.Error("a URL reconnects")
 	}
 	in.Audio = nil
-	if a := strings.Join(hlsArgs("/d", "/v", in, r, false), " "); !strings.Contains(a, " -an ") {
-		t.Error("no audio stream → -an")
-	}
-	cp := recipes(gridCaps(), "copy", in, 1920, 1080, 0)[0]
-	if a := strings.Join(hlsArgs("/d", "/v", in, cp, true), " "); strings.Contains(a, "force_key_frames") || strings.Contains(a, "-vf") {
-		t.Error("copy must not ask for keyframes or filter")
+	if a := strings.Join(pieceArgs("/d", "/v", in, rc, r, 0), " "); !strings.Contains(a, "-an") {
+		t.Error("no sound means -an")
 	}
 }
 
-func TestSegArgs(t *testing.T) {
-	in := src("h264", 1920, 1080, 25)
-	r := recipes(gridCaps(), "h264", in, 1920, 1080, 10000)[0]
-	first := strings.Join(segArgs("/d", "/v.mkv", in, r, 0, 900, 900, "init.mp4"), " ")
-	for _, want := range []string{"-ss 0 -i /v.mkv", "-output_ts_offset 10", "-start_number 0", "movflags=+frag_discont", "-hls_fmp4_init_filename init.mp4", "-hls_segment_filename /d/seg%05d.m4s", "/d/run.m3u8", "-force_key_frames expr:gte(t,n_forced*4)"} {
-		if !strings.Contains(first, want) {
-			t.Errorf("first run: missing %q in\n%s", want, first)
+func TestCapsOnly(t *testing.T) {
+	c := gridCaps()
+	if got := c.Only("/dev/dri/renderD129"); len(got.GPUs) != 1 || got.GPUs[0].Name != "Intel Arc A380" || len(c.GPUs) != 2 {
+		t.Fatalf("one card: %+v", got.GPUs)
 	}
+	if len(c.Only(CPU).GPUs) != 0 || len(c.Only("/dev/dri/renderD200").GPUs) != 0 || len(c.Only("").GPUs) != 2 {
+		t.Fatal("cpu, a device not here, any")
 	}
-	if strings.Contains(first, " -t ") {
-		t.Error("a run to the end has no -t")
-	}
-	// a seek: segments 30..44, stopping half a frame short of 45, which is on disk
-	seek := strings.Join(segArgs("/d", "/v.mkv", in, r, 30, 45, 900, "init-run.mp4"), " ")
-	for _, want := range []string{"-ss 120 -i /v.mkv", "-t 59.980", "-output_ts_offset 130", "-start_number 30", "-hls_fmp4_init_filename init-run.mp4"} {
-		if !strings.Contains(seek, want) {
-			t.Errorf("seek run: missing %q in\n%s", want, seek)
-		}
-	}
-	if strings.Index(seek, "-ss 120") > strings.Index(seek, "-i /v.mkv") || strings.Index(seek, "-hwaccel") > strings.Index(seek, "-i /v.mkv") {
-		t.Error("-ss and the decoder's options go before -i")
-	}
-	for d, n := range map[float64]int{9: 3, 8.2: 2, 8.31: 3, 64: 16, 0.1: 1, 3600: 900} {
-		if got := segCount(d); got != n {
-			t.Errorf("segCount(%v) = %d, want %d", d, got, n)
-		}
-	}
-}
-
-func TestParseProgress(t *testing.T) {
-	b := []byte("frame=10\nout_time_us=1000000\nspeed=2.5x\nprogress=continue\nframe=20\nout_time_us=2500000\nspeed=3.1x\nprogress=continue\n")
-	s, sp := parseProgress(b)
-	if s != 2.5 || sp != 3.1 {
-		t.Fatalf("got %v %v", s, sp)
-	}
-}
-
-func TestValidChoice(t *testing.T) {
-	if err := ValidChoice("h264", "720p"); err != nil {
-		t.Fatal(err)
-	}
-	if ValidChoice("vp8", "720p") == nil || ValidChoice("h264", "9p") == nil || ValidChoice("", "") == nil {
-		t.Fatal("bad choices must be refused")
+	// previews then go to that card, or the CPU
+	if g := c.Only("/dev/dri/renderD129").pick("hevc"); g == nil || g.Device != "/dev/dri/renderD129" {
+		t.Fatalf("pick: %+v", g)
 	}
 }

internal/media/ffrun.go

@@ -6,13 +6,13 @@ package media
 // transcode never outlives the console that asked for it.
 
 import (
+	"bufio"
 	"bytes"
 	"context"
 	"errors"
 	"io"
 	"os"
 	"os/exec"
-	"strings"
 	"sync"
 	"syscall"
 	"time"
@@ -92,6 +92,35 @@ func (p *ffproc) end() {
 	}
 }
 
+// pause stops ffmpeg where it is (SIGSTOP), and resume lets it go on: a stream far enough
+// ahead of its player waits rather than making the rest of the film. A root run is paused by
+// its helper, which is told on its standard input.
+func (p *ffproc) pause() error  { return p.signal(syscall.SIGSTOP, "stop\n") }
+func (p *ffproc) resume() error { return p.signal(syscall.SIGCONT, "cont\n") }
+
+func (p *ffproc) signal(sig syscall.Signal, word string) error {
+	if done, _ := p.exited(); done {
+		return errors.New("ended")
+	}
+	if p.in != nil {
+		_, err := io.WriteString(p.in, word)
+		return err
+	}
+	return syscall.Kill(-p.cmd.Process.Pid, sig)
+}
+
+// kill ends ffmpeg now, without waiting: a paused one too.
+func (p *ffproc) kill() {
+	if done, _ := p.exited(); done {
+		return
+	}
+	if p.in != nil {
+		p.in.Close() // the helper ends it, waking it first if it is paused
+		return
+	}
+	_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGKILL)
+}
+
 // lastWords is what ffmpeg said last, for the record.
 func (p *ffproc) lastWords() string { return p.tail.last() }
 
@@ -141,7 +170,17 @@ func RunFFmpegHelper(args []string, stdin io.Reader, stderr io.Writer) int {
 	}
 	ctx, cancel := context.WithCancel(context.Background())
 	go func() {
-		_, _ = io.Copy(io.Discard, stdin) // returns when the daemon lets go
+		// "stop" and "cont" pause and resume it; the end of input (the daemon letting go)
+		// ends it.
+		sc := bufio.NewScanner(stdin)
+		for sc.Scan() {
+			switch sc.Text() {
+			case "stop":
+				_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGSTOP)
+			case "cont":
+				_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGCONT)
+			}
+		}
 		cancel()
 	}()
 	done := make(chan error, 1)
@@ -151,6 +190,7 @@ func RunFFmpegHelper(args []string, stdin io.Reader, stderr io.Writer) int {
 	case werr = <-done:
 	case <-ctx.Done():
 		_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM)
+		_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGCONT) // a paused one can't act on TERM
 		select {
 		case werr = <-done:
 		case <-time.After(5 * time.Second):
@@ -181,8 +221,3 @@ func clearDir(dir string) error {
 	}
 	return nil
 }
-
-func hasSegments(playlist string) bool {
-	b, err := os.ReadFile(playlist)
-	return err == nil && strings.Contains(string(b), "#EXTINF")
-}

internal/media/hls.go

@@ -1,669 +0,0 @@
-package media
-
-// Playing what the browser cannot play as it is: an HLS stream, re-encoded on demand by
-// ffmpeg on the machine's own GPU where it has one it can use, on its CPU otherwise. A
-// stream is one file at one codec and size, in a folder of its own under
-// <data dir>/transcode/.
-//
-// A re-encode is seekable anywhere from the start. Its playlist is written up front, for
-// the whole film: a fixed grid of 4 s segments, so the player's timeline is the film's.
-// ffmpeg runs ahead from the top as fast as it can; a request for a segment it hasn't
-// reached and won't reach soon — a seek ahead, or back into a stretch never made — ends that
-// run and starts another at the segment asked for, which stops on its own where segments
-// already exist. When a run finishes, the next gap is filled, until the whole file is on
-// disk. Every run is made the same way (segArgs), so segments from different runs join.
-//
-// "Repackaged" (copy) can't cut on a fixed grid — its keyframes are the source's — so it is
-// an event playlist that grows as ffmpeg writes it, seekable within what's done. Repackaging
-// runs at disk speed, so that is all of it within seconds.
-//
-// Nothing is kept: a stream nobody has touched for a while is ended and its folder removed,
-// and the folder is emptied when the daemon starts.
-
-import (
-	"bufio"
-	"bytes"
-	"context"
-	"crypto/rand"
-	"encoding/hex"
-	"errors"
-	"fmt"
-	"log"
-	"os"
-	"path/filepath"
-	"regexp"
-	"strconv"
-	"strings"
-	"sync"
-	"time"
-)
-
-// Stream is one re-encode, in progress or finished.
-type Stream struct {
-	ID       string    `json:"id"`
-	Path     string    `json:"path"`
-	Codec    string    `json:"codec"`
-	Quality  string    `json:"quality"`
-	Engine   string    `json:"engine"`
-	Width    int       `json:"width"`
-	Height   int       `json:"height"`
-	Kbps     int       `json:"kbps"`
-	Source   Info      `json:"source"`
-	Started  time.Time `json:"started"`
-	Root     bool      `json:"root"`
-	Seekable bool      `json:"seekable"` // seeking anywhere restarts the re-encode there
-
-	key      string
-	dir      string
-	mu       sync.Mutex
-	used     time.Time
-	proc     *ffproc // the run going now (or the only one, for copy)
-	finished bool
-	failure  string
-	gone     bool // forgotten by the Streamer
-
-	// seekable streams only
-	caps    Caps
-	self    string
-	recipe  Recipe
-	n       int // segments in the playlist
-	end     int // segments past this are not in the file after all (n until a run finds out)
-	run     *segRun
-	runMu   sync.Mutex // one run change at a time; held across ending one run and starting the next
-	asked   time.Time  // when the request that last moved the run arrived: an older one never moves it back
-	initOK  bool       // a later run's init segment was checked against the first's
-	initBad bool
-}
-
-// segRun is one ffmpeg over segments from..to-1.
-type segRun struct {
-	from, to int
-	proc     *ffproc
-	stopped  bool // ended by us: what it wrote after that is not to be trusted
-}
-
-// StreamStatus is what the player asks for while the file is still going through.
-type StreamStatus struct {
-	Running  bool    `json:"running"`
-	Done     bool    `json:"done"`
-	Error    string  `json:"error,omitempty"`
-	Progress float64 `json:"progress"` // seconds of output written (copy), or on disk (seekable)
-	Speed    float64 `json:"speed"`    // × realtime
-	Segments int     `json:"segments"`
-	From     float64 `json:"from"` // seekable: where the run going now started, seconds
-}
-
-// Streamer keeps one machine's streams.
-type Streamer struct {
-	dir     string
-	self    string
-	limit   int
-	mu      sync.Mutex
-	streams map[string]*Stream
-}
-
-const streamIdle = 10 * time.Minute
-const firstSegmentWait = 40 * time.Second
-
-// segmentWait is how long a request for a segment waits for a run to write it.
-const segmentWait = 45 * time.Second // under the hub proxy's 60 s wait for a node to answer
-
-// lookahead: a segment this many past what the run has written is waited for, not
-// restarted at. A few seconds of film, which any engine here writes faster than that.
-const lookahead = 3
-
-var streamFile = regexp.MustCompile(`^(index\.m3u8|init\.mp4|seg\d{5}\.m4s)$`)
-var segName = regexp.MustCompile(`^seg(\d{5})\.m4s$`)
-
-// NewStreamer empties the folder — a daemon restart orphans every stream — and starts the
-// sweeper.
-func NewStreamer(dataDir, self string, limit int) *Streamer {
-	m := &Streamer{dir: filepath.Join(dataDir, "transcode"), self: self, limit: limit, streams: map[string]*Stream{}}
-	_ = os.MkdirAll(m.dir, 0o755)
-	_ = clearDir(m.dir)
-	go func() {
-		for range time.Tick(30 * time.Second) {
-			m.Sweep(streamIdle)
-		}
-	}()
-	return m
-}
-
-// Open starts a stream, or hands back the one already serving the same request. It
-// returns once the first segment is on disk, so the player's first fetch is answered.
-func (m *Streamer) Open(ctx context.Context, c Caps, root bool, path, codec, quality string, src Info) (*Stream, error) {
-	if c.Error != "" {
-		return nil, errors.New(c.Error)
-	}
-	if err := ValidChoice(codec, quality); err != nil {
-		return nil, err
-	}
-	q, _ := qualityByID(quality)
-	if src.Video == nil {
-		return nil, &DecodeError{Msg: "no video stream"}
-	}
-	if src.Video.Width <= 0 || src.Video.Height <= 0 {
-		return nil, &DecodeError{Msg: "the picture's size is unknown"}
-	}
-	key := fmt.Sprintf("%s|%d|%.0f|%s|%s|%v", path, src.Size, src.Duration, codec, quality, root)
-	m.mu.Lock()
-	for _, s := range m.streams {
-		if s.key == key {
-			if s.failed() { // not the stream to hand back: let it go and make a fresh one
-				delete(m.streams, s.ID)
-				_ = os.RemoveAll(s.dir)
-				break
-			}
-			m.mu.Unlock()
-			s.touch()
-			return s, nil
-		}
-	}
-	running := 0
-	for _, s := range m.streams {
-		if s.running() {
-			running++
-		}
-	}
-	if running >= m.limit {
-		m.mu.Unlock()
-		return nil, fmt.Errorf("%d videos are already being re-encoded on this machine; one must finish or be closed first", running)
-	}
-	s := &Stream{ID: newID(), Path: path, Codec: codec, Quality: quality, Source: src, Started: time.Now(), Root: root, key: key, used: time.Now(),
-		caps: c, self: m.self}
-	s.dir = filepath.Join(m.dir, s.ID)
-	// A fixed grid needs the film's length; without one it can only grow as it goes.
-	s.Seekable = codec != "copy" && src.Duration > 0
-	if s.Seekable {
-		s.n = segCount(src.Duration)
-		s.end = s.n
-	}
-	m.streams[s.ID] = s
-	m.mu.Unlock()
-
-	s.Width, s.Height, s.Kbps = fit(q, src, codec)
-	if codec == "copy" {
-		s.Width, s.Height, s.Kbps = src.Video.Width, src.Video.Height, int(src.BitRate/1000)
-	}
-	rs := recipes(c, codec, src, s.Width, s.Height, s.Kbps)
-	if len(rs) == 0 {
-		m.forget(s)
-		return nil, errors.New("nothing on this machine encodes " + codec)
-	}
-	var last string
-	for _, r := range rs {
-		if err := os.MkdirAll(s.dir, 0o755); err != nil { // the root helper's ffmpeg writes here too
-			m.forget(s)
-			return nil, err
-		}
-		_ = clearDir(s.dir)
-		var args []string
-		if s.Seekable {
-			args = segArgs(s.dir, path, src, r, 0, s.n, s.n, "init.mp4")
-		} else {
-			args = hlsArgs(s.dir, path, src, r, codec == "copy")
-		}
-		p, err := startFF(c, m.self, root, args)
-		if err != nil {
-			last = r.Engine + ": " + err.Error()
-			continue
-		}
-		s.mu.Lock()
-		s.proc, s.Engine, s.recipe = p, r.Engine, r
-		if s.Seekable {
-			s.run = &segRun{from: 0, to: s.n, proc: p}
-		}
-		s.mu.Unlock()
-		msg, ok := s.waitFirstSegment(ctx)
-		if ok {
-			if s.Seekable {
-				go s.watchRun(s.run)
-			} else {
-				go s.watch()
-			}
-			log.Printf("media: stream %s: %s → %s %s %dx%d %dk on %s", s.ID, filepath.Base(path), codec, quality, s.Width, s.Height, s.Kbps, r.Engine)
-			return s, nil
-		}
-		p.end()
-		if ctx.Err() != nil || s.closed() { // the caller left, or somebody closed it meanwhile: not the recipe's fault
-			m.forget(s)
-			return nil, errors.New("cancelled")
-		}
-		last = r.Engine + ": " + msg
-		log.Printf("media: stream %s on %s: %s", filepath.Base(path), r.Engine, msg)
-	}
-	m.forget(s)
-	return nil, &DecodeError{Msg: last}
-}
-
-// waitFirstSegment watches for the first segment, or ffmpeg giving up.
-func (s *Stream) waitFirstSegment(ctx context.Context) (string, bool) {
-	deadline := time.Now().Add(firstSegmentWait)
-	playlist := filepath.Join(s.dir, "index.m3u8")
-	for {
-		if s.Seekable && exists(s.segPath(0)) || !s.Seekable && hasSegments(playlist) {
-			return "", true
-		}
-		if done, err := s.proc.exited(); done {
-			msg := s.proc.lastWords()
-			if msg == "" && err != nil {
-				msg = err.Error()
-			}
-			if msg == "" {
-				msg = "ffmpeg ended before writing a segment"
-			}
-			return msg, false
-		}
-		if time.Now().After(deadline) {
-			return "no segment after " + firstSegmentWait.String(), false
-		}
-		select {
-		case <-ctx.Done():
-			return ctx.Err().Error(), false
-		case <-time.After(250 * time.Millisecond):
-		}
-	}
-}
-
-// watch records how the encode ended (copy streams: one run, start to end).
-func (s *Stream) watch() {
-	<-s.proc.done
-	s.mu.Lock()
-	s.finished = true
-	if s.proc.err != nil {
-		s.failure = s.proc.lastWords()
-		if s.failure == "" {
-			s.failure = s.proc.err.Error()
-		}
-	}
-	s.mu.Unlock()
-	if s.failure != "" {
-		log.Printf("media: stream %s ended early: %s", s.ID, s.failure)
-	}
-}
-
-// ---- seekable streams ----
-
-func (s *Stream) segPath(i int) string { return filepath.Join(s.dir, fmt.Sprintf("seg%05d.m4s", i)) }
-
-// written is the last segment of a run on disk, counting on from where it started.
-func (s *Stream) written(r *segRun) int {
-	i := r.from
-	for i < r.to && exists(s.segPath(i)) {
-		i++
-	}
-	return i - 1
-}
-
-// watchRun follows one run to its end. A run that ends by itself — the end of the film, or
-// the segments it was to stop short of — is followed by the next gap. One that failed fails
-// the stream; one we ended is the business of whoever ended it.
-func (s *Stream) watchRun(r *segRun) {
-	<-r.proc.done
-	s.mu.Lock()
-	if s.gone || r.stopped || s.run != r {
-		s.mu.Unlock()
-		return
-	}
-	if r.proc.err != nil {
-		s.failure = r.proc.lastWords()
-		if s.failure == "" {
-			s.failure = r.proc.err.Error()
-		}
-		s.finished = true
-		s.mu.Unlock()
-		log.Printf("media: stream %s: run from segment %d failed: %s", s.ID, r.from, s.failure)
-		return
-	}
-	if w := s.written(r); w+1 < r.to && r.to == s.n {
-		// ran to the end of the file before the end of the grid: the picture is shorter than
-		// the container says, and those last segments will never exist
-		s.end = w + 1
-		log.Printf("media: stream %s: the film ends at segment %d of %d", s.ID, s.end, s.n)
-	}
-	s.checkInit(r)
-	s.mu.Unlock()
-	s.fillNext(r.to)
-}
-
-// checkInit compares a later run's init segment with the first run's, once. They come out
-// identical (same encoder, same settings); if one day they don't, the log says so.
-func (s *Stream) checkInit(r *segRun) {
-	if r.from == 0 || s.initOK {
-		return
-	}
-	a, err1 := os.ReadFile(filepath.Join(s.dir, "init.mp4"))
-	b, err2 := os.ReadFile(filepath.Join(s.dir, "init-run.mp4"))
-	if err1 != nil || err2 != nil {
-		return
-	}
-	s.initOK = true
-	if !bytes.Equal(a, b) && !s.initBad {
-		s.initBad = true
-		log.Printf("media: stream %s: a restarted run's init segment differs from the first (%s); its segments may not join", s.ID, s.Engine)
-	}
-}
-
-// missingFrom is the first segment not on disk at or after i (wrapping round to the start),
-// or -1 when the whole film is there.
-func (s *Stream) missingFrom(i int) int {
-	end := s.end
-	for k := 0; k < end; k++ {
-		j := (i + k) % end
-		if !exists(s.segPath(j)) {
-			return j
-		}
-	}
-	return -1
-}
-
-// fillNext starts a run at the next gap after `after`; when there is none, the stream is done.
-func (s *Stream) fillNext(after int) {
-	s.runMu.Lock()
-	defer s.runMu.Unlock()
-	s.mu.Lock()
-	if s.gone || s.failure != "" || (s.run != nil && !s.run.done()) {
-		s.mu.Unlock()
-		return
-	}
-	end := s.end
-	s.mu.Unlock()
-	if after >= end {
-		after = 0
-	}
-	next := s.missingFrom(after)
-	if next < 0 {
-		s.mu.Lock()
-		s.finished, s.proc = true, nil
-		s.mu.Unlock()
-		log.Printf("media: stream %s: all %d segments on disk", s.ID, end)
-		return
-	}
-	s.startRun(next)
-}
-
-func (r *segRun) done() bool { d, _ := r.proc.exited(); return d }
-
-// startRun ends the run going now and starts one at segment from, stopping short of the next
-// segment already on disk. The run it ends is asked to stop and waited for, and whatever it
-// finished writing after being asked is removed: ffmpeg closes the segment it was in the
-// middle of on its way out, short. Called with runMu held, mu not.
-func (s *Stream) startRun(from int) {
-	s.mu.Lock()
-	old := s.run
-	end := s.end
-	s.mu.Unlock()
-	if old != nil && !old.done() {
-		had := map[int]bool{}
-		for i := old.from; i < old.to; i++ {
-			had[i] = exists(s.segPath(i))
-		}
-		s.mu.Lock()
-		old.stopped = true
-		s.mu.Unlock()
-		old.proc.end()
-		for i := old.from; i < old.to; i++ {
-			if !had[i] && exists(s.segPath(i)) {
-				_ = os.Remove(s.segPath(i))
-			}
-		}
-	}
-	to := from + 1
-	for to < end && !exists(s.segPath(to)) {
-		to++
-	}
-	s.mu.Lock()
-	if s.gone {
-		s.mu.Unlock()
-		return
-	}
-	s.mu.Unlock()
-	init := "init-run.mp4"
-	if !exists(filepath.Join(s.dir, "init.mp4")) {
-		init = "init.mp4"
-	}
-	p, err := startFF(s.caps, s.self, s.Root, segArgs(s.dir, s.Path, s.Source, s.recipe, from, to, s.n, init))
-	s.mu.Lock()
-	defer s.mu.Unlock()
-	if err != nil {
-		s.failure, s.finished = "restarting at "+strconv.Itoa(from*hlsSeconds)+" s: "+err.Error(), true
-		return
-	}
-	r := &segRun{from: from, to: to, proc: p}
-	s.run, s.proc, s.finished = r, p, false
-	go s.watchRun(r)
-	log.Printf("media: stream %s: run from %d s to %d s", s.ID, from*hlsSeconds, to*hlsSeconds)
-}
-
-// Playlist is the whole film's, written before most of it exists.
-func (s *Stream) Playlist() []byte {
-	var b strings.Builder
-	b.WriteString("#EXTM3U\n#EXT-X-VERSION:7\n")
-	fmt.Fprintf(&b, "#EXT-X-TARGETDURATION:%d\n", hlsSeconds)
-	b.WriteString("#EXT-X-MEDIA-SEQUENCE:0\n#EXT-X-PLAYLIST-TYPE:VOD\n#EXT-X-INDEPENDENT-SEGMENTS\n#EXT-X-MAP:URI=\"init.mp4\"\n")
-	for i := 0; i < s.n; i++ {
-		d := float64(hlsSeconds)
-		if i == s.n-1 {
-			d = s.Source.Duration - float64(i*hlsSeconds)
-		}
-		fmt.Fprintf(&b, "#EXTINF:%.6f,\nseg%05d.m4s\n", d, i)
-	}
-	b.WriteString("#EXT-X-ENDLIST\n")
-	return []byte(b.String())
-}
-
-// ErrNotInFilm is a segment the playlist lists but the film turned out not to reach.
-var ErrNotInFilm = errors.New("past the end of the film")
-
-// Segment is the path of segment i once it is on disk. When the run going now will write it
-// soon, it waits; otherwise this is a seek, and the run moves there — unless a newer request
-// has moved it since this one arrived.
-func (s *Stream) Segment(ctx context.Context, i int) (string, error) {
-	asked := time.Now()
-	deadline := asked.Add(segmentWait)
-	p := s.segPath(i)
-	for {
-		s.touch()
-		if exists(p) {
-			return p, nil
-		}
-		s.mu.Lock()
-		switch {
-		case s.gone:
-			s.mu.Unlock()
-			return "", errors.New("the stream was closed")
-		case i >= s.end:
-			s.mu.Unlock()
-			return "", ErrNotInFilm
-		case s.failure != "":
-			f := s.failure
-			s.mu.Unlock()
-			return "", errors.New(f)
-		}
-		r := s.run
-		coming := r != nil && !r.done() && r.from <= i && i < r.to && i <= s.written(r)+lookahead
-		move := !coming && !asked.Before(s.asked)
-		if move {
-			s.asked = asked
-		}
-		s.mu.Unlock()
-		if move {
-			s.runMu.Lock()
-			// still the newest, and still not on its way? (a run may have moved for a
-			// newer request while this one waited for the lock)
-			s.mu.Lock()
-			r = s.run
-			coming = r != nil && !r.done() && r.from <= i && i < r.to && i <= s.written(r)+lookahead
-			newest := !asked.Before(s.asked)
-			s.mu.Unlock()
-			if !coming && newest && !exists(p) {
-				log.Printf("media: stream %s: seek to %d s", s.ID, i*hlsSeconds)
-				s.startRun(i)
-			}
-			s.runMu.Unlock()
-			continue
-		}
-		if time.Now().After(deadline) {
-			return "", fmt.Errorf("segment %d didn't arrive in %s", i, segmentWait)
-		}
-		select {
-		case <-ctx.Done():
-			return "", ctx.Err()
-		case <-time.After(150 * time.Millisecond):
-		}
-	}
-}
-
-// SegmentIndex reads the number out of a segment's name.
-func SegmentIndex(name string) (int, bool) {
-	m := segName.FindStringSubmatch(name)
-	if m == nil {
-		return 0, false
-	}
-	i, err := strconv.Atoi(m[1])
-	return i, err == nil
-}
-
-// ---- both kinds ----
-
-func (m *Streamer) Find(id string) (*Stream, bool) {
-	m.mu.Lock()
-	defer m.mu.Unlock()
-	s, ok := m.streams[id]
-	return s, ok
-}
-
-// Close ends a stream and removes its folder.
-func (m *Streamer) Close(id string) {
-	m.mu.Lock()
-	s, ok := m.streams[id]
-	m.mu.Unlock()
-	if !ok {
-		return
-	}
-	s.mu.Lock()
-	s.gone = true // a run ending now is ours, not a gap to fill
-	p := s.proc
-	if s.run != nil {
-		s.run.stopped = true
-	}
-	s.mu.Unlock()
-	if p != nil {
-		p.end()
-	}
-	m.forget(s)
-}
-
-func (m *Streamer) forget(s *Stream) {
-	m.mu.Lock()
-	delete(m.streams, s.ID)
-	m.mu.Unlock()
-	s.mu.Lock()
-	s.gone = true
-	s.mu.Unlock()
-	_ = os.RemoveAll(s.dir)
-}
-
-func (s *Stream) closed() bool { s.mu.Lock(); defer s.mu.Unlock(); return s.gone }
-
-func (s *Stream) failed() bool {
-	s.mu.Lock()
-	defer s.mu.Unlock()
-	return s.finished && s.failure != ""
-}
-
-// Sweep closes what nobody has touched for a while.
-func (m *Streamer) Sweep(idle time.Duration) {
-	m.mu.Lock()
-	var old []string
-	for id, s := range m.streams {
-		s.mu.Lock()
-		if time.Since(s.used) > idle {
-			old = append(old, id)
-		}
-		s.mu.Unlock()
-	}
-	m.mu.Unlock()
-	for _, id := range old {
-		log.Printf("media: stream %s idle, closed", id)
-		m.Close(id)
-	}
-}
-
-// File is the on-disk path of one of the stream's files — the playlist (copy streams; a
-// seekable one's is Playlist), the init segment or a media segment — and nothing else.
-func (s *Stream) File(name string) (string, bool) {
-	if !streamFile.MatchString(name) {
-		return "", false
-	}
-	s.touch()
-	return filepath.Join(s.dir, name), true
-}
-
-func (s *Stream) touch() { s.mu.Lock(); s.used = time.Now(); s.mu.Unlock() }
-
-func (s *Stream) running() bool {
-	s.mu.Lock()
-	defer s.mu.Unlock()
-	return s.proc != nil && !s.finished
-}
-
-// Status is read from ffmpeg's own progress file and what is on disk.
-func (s *Stream) Status() StreamStatus {
-	s.mu.Lock()
-	st := StreamStatus{Running: s.proc != nil && !s.finished, Done: s.finished && s.failure == "", Error: s.failure}
-	seekable, end, run := s.Seekable, s.end, s.run
-	s.mu.Unlock()
-	b, err := os.ReadFile(filepath.Join(s.dir, "progress"))
-	if err == nil {
-		st.Progress, st.Speed = parseProgress(b)
-	}
-	if !seekable {
-		if b, err := os.ReadFile(filepath.Join(s.dir, "index.m3u8")); err == nil {
-			st.Segments = strings.Count(string(b), "#EXTINF")
-		}
-		return st
-	}
-	for i := 0; i < end; i++ {
-		if exists(s.segPath(i)) {
-			st.Segments++
-		}
-	}
-	st.Progress = float64(st.Segments * hlsSeconds)
-	if st.Progress > s.Source.Duration {
-		st.Progress = s.Source.Duration
-	}
-	if run != nil && st.Running {
-		st.From = float64(run.from * hlsSeconds)
-	}
-	return st
-}
-
-// parseProgress takes the last out_time_us and speed from ffmpeg's -progress output.
-func parseProgress(b []byte) (seconds, speed float64) {
-	sc := bufio.NewScanner(strings.NewReader(string(b)))
-	for sc.Scan() {
-		k, v, ok := strings.Cut(sc.Text(), "=")
-		if !ok {
-			continue
-		}
-		v = strings.TrimSpace(v)
-		switch k {
-		case "out_time_us":
-			if us, err := strconv.ParseInt(v, 10, 64); err == nil && us >= 0 {
-				seconds = float64(us) / 1e6
-			}
-		case "speed":
-			if x, err := strconv.ParseFloat(strings.TrimSuffix(v, "x"), 64); err == nil {
-				speed = x
-			}
-		}
-	}
-	return seconds, speed
-}
-
-func newID() string {
-	b := make([]byte, 8)
-	_, _ = rand.Read(b)
-	return hex.EncodeToString(b)
-}

internal/media/hls_test.go

@@ -1,220 +0,0 @@
-package media
-
-import (
-	"context"
-	"errors"
-	"math"
-	"os"
-	"os/exec"
-	"path/filepath"
-	"sort"
-	"strconv"
-	"strings"
-	"testing"
-	"time"
-)
-
-// The real thing, when ffmpeg is here: a generated clip goes through as H.264 and comes out
-// as a playlist of segments, the status follows it to the end, and closing removes it all.
-func TestStreamWithFFmpeg(t *testing.T) {
-	if _, err := exec.LookPath("ffmpeg"); err != nil {
-		t.Skip("no ffmpeg")
-	}
-	c := Probe(context.Background())
-	if c.Error != "" {
-		t.Skip(c.Error)
-	}
-	if EngineFor(c, "h264") == "" {
-		t.Skip("nothing here encodes h264")
-	}
-	dir := t.TempDir()
-	clip := filepath.Join(dir, "clip.mp4")
-	out, err := exec.Command("ffmpeg", "-nostdin", "-v", "error", "-f", "lavfi", "-i", "testsrc=size=640x360:rate=25:duration=9", "-f", "lavfi", "-i", "sine=frequency=440:duration=9",
-		"-c:v", "mpeg4", "-q:v", "5", "-c:a", "aac", "-shortest", clip).CombinedOutput()
-	if err != nil {
-		t.Skipf("could not make a clip: %s", out)
-	}
-	src, err := Inspect(context.Background(), c, clip, false)
-	if err != nil || src.Video == nil || src.Audio == nil || src.Video.Codec != "mpeg4" || src.Duration < 8 {
-		t.Fatalf("inspect: %+v %v", src, err)
-	}
-	m := NewStreamer(dir, "hamlab", 2)
-	s, err := m.Open(context.Background(), c, false, clip, "h264", "480p", src)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if s.Width != 640 || s.Height != 360 || s.Engine == "" || !s.Seekable {
-		t.Fatalf("stream: %+v", s)
-	}
-	if _, ok := s.File("index.m3u8"); !ok {
-		t.Fatal("no playlist")
-	}
-	if pl := string(s.Playlist()); !strings.Contains(pl, "#EXT-X-PLAYLIST-TYPE:VOD") || !strings.HasSuffix(pl, "#EXT-X-ENDLIST\n") || strings.Count(pl, "#EXTINF") != 3 {
-		t.Fatalf("playlist for 9 s:\n%s", pl)
-	}
-	if !exists(s.segPath(0)) {
-		t.Fatal("Open returned before the first segment")
-	}
-	for _, bad := range []string{"../x", "progress", "seg1.m4s", "index.m3u8/", "init.mp4.tmp"} {
-		if _, ok := s.File(bad); ok {
-			t.Fatalf("%q must not be served", bad)
-		}
-	}
-	// the same request is the same stream
-	if again, err := m.Open(context.Background(), c, false, clip, "h264", "480p", src); err != nil || again.ID != s.ID {
-		t.Fatal("not reused")
-	}
-	var st StreamStatus
-	for i := 0; i < 120; i++ {
-		st = s.Status()
-		if st.Done || st.Error != "" {
-			break
-		}
-		time.Sleep(250 * time.Millisecond)
-	}
-	if !st.Done || st.Error != "" || st.Segments != 3 || st.Progress < 8 {
-		t.Fatalf("status at the end: %+v", st)
-	}
-	if init, _ := s.File("init.mp4"); !exists(init) {
-		t.Fatal("no init segment")
-	}
-	m.Close(s.ID)
-	if _, ok := m.Find(s.ID); ok || exists(s.dir) {
-		t.Fatal("close must forget and remove")
-	}
-	// an unplayable source is a DecodeError, and leaves nothing behind
-	bad := filepath.Join(dir, "bad.mkv")
-	_ = os.WriteFile(bad, []byte("not a video"), 0o644)
-	if _, err := Inspect(context.Background(), c, bad, false); err == nil {
-		t.Fatal("inspect of garbage")
-	}
-	if left, _ := os.ReadDir(m.dir); len(left) != 0 {
-		t.Fatalf("left behind: %v", left)
-	}
-}
-
-// A seek ahead of the encode: the run moves there, the segment comes from a run that started
-// at it, the gaps are filled afterwards, and the segments of the different runs, joined, are
-// one film with no hole and nothing twice beyond a frame at a join.
-func TestSeekRestartsTheEncode(t *testing.T) {
-	if _, err := exec.LookPath("ffmpeg"); err != nil {
-		t.Skip("no ffmpeg")
-	}
-	c := Probe(context.Background())
-	if c.Error != "" || EngineFor(c, "h264") == "" {
-		t.Skip("nothing here encodes h264")
-	}
-	dir := t.TempDir()
-	clip := filepath.Join(dir, "clip.mkv")
-	// big and busy enough that the first run is still near the start when the seek comes
-	out, err := exec.Command("ffmpeg", "-nostdin", "-v", "error", "-f", "lavfi", "-i", "testsrc2=size=1920x1080:rate=30000/1001:duration=64", "-f", "lavfi", "-i", "sine=frequency=440:duration=64",
-		"-c:v", "libx264", "-preset", "ultrafast", "-g", "250", "-c:a", "libopus", "-shortest", clip).CombinedOutput()
-	if err != nil {
-		t.Skipf("could not make a clip: %s", out)
-	}
-	src, err := Inspect(context.Background(), c, clip, false)
-	if err != nil {
-		t.Fatal(err)
-	}
-	m := NewStreamer(dir, "hamlab", 2)
-	s, err := m.Open(context.Background(), c, false, clip, "h264", "1080p", src)
-	if err != nil {
-		t.Fatal(err)
-	}
-	defer m.Close(s.ID)
-	if s.n != 16 {
-		t.Fatalf("%d segments for 64 s", s.n)
-	}
-	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
-	defer cancel()
-	p, err := s.Segment(ctx, 12)
-	if err != nil {
-		t.Fatal(err)
-	}
-	s.mu.Lock()
-	from := s.run.from
-	s.mu.Unlock()
-	if from != 12 {
-		t.Fatalf("the run after the seek starts at segment %d, not 12 (the first run was already there?)", from)
-	}
-	if got := firstVideoPTS(t, filepath.Join(s.dir, "init.mp4"), p); math.Abs(got-(48+tsBase)) > 0.05 {
-		t.Fatalf("segment 12 starts at %.3f, want %d", got, 48+tsBase)
-	}
-	if _, err := s.Segment(ctx, 16); !errors.Is(err, ErrNotInFilm) {
-		t.Fatalf("past the end: %v", err)
-	}
-	var st StreamStatus
-	for i := 0; i < 480; i++ {
-		if st = s.Status(); st.Done || st.Error != "" {
-			break
-		}
-		time.Sleep(250 * time.Millisecond)
-	}
-	if !st.Done || st.Segments != 16 {
-		t.Fatalf("not filled in: %+v", st)
-	}
-	// the joined film: every segment in order behind the one init segment
-	joined := filepath.Join(dir, "joined.mp4")
-	f, _ := os.Create(joined)
-	for _, n := range append([]string{"init.mp4"}, segNames(16)...) {
-		b, err := os.ReadFile(filepath.Join(s.dir, n))
-		if err != nil {
-			t.Fatal(err)
-		}
-		_, _ = f.Write(b)
-	}
-	f.Close()
-	if out, err := exec.Command("ffmpeg", "-v", "error", "-i", joined, "-f", "null", "-").CombinedOutput(); err != nil || strings.Contains(string(out), "error") {
-		t.Fatalf("the joined film doesn't decode cleanly: %v %s", err, out)
-	}
-	pts := videoPTS(t, joined)
-	if len(pts) < 1900 || pts[0] > tsBase+0.1 || pts[len(pts)-1] < tsBase+63.5 {
-		t.Fatalf("%d frames, %.3f .. %.3f", len(pts), pts[0], pts[len(pts)-1])
-	}
-	for i := 1; i < len(pts); i++ {
-		if d := pts[i] - pts[i-1]; d > 0.07 { // a frame is 0.033; a join may lose up to one
-			t.Fatalf("a hole of %.3f s at %.3f", d, pts[i-1])
-		}
-	}
-}
-
-func segNames(n int) []string {
-	var out []string
-	for i := 0; i < n; i++ {
-		out = append(out, "seg"+strconv.FormatInt(int64(100000+i), 10)[1:]+".m4s")
-	}
-	return out
-}
-
-func videoPTS(t *testing.T, file string) []float64 {
-	out, err := exec.Command("ffprobe", "-v", "error", "-select_streams", "v", "-show_entries", "packet=pts_time", "-of", "csv=p=0", file).Output()
-	if err != nil {
-		t.Fatal(err)
-	}
-	var pts []float64
-	seen := map[string]bool{}
-	for _, l := range strings.Fields(string(out)) {
-		if seen[l] { // the same frame from both runs at a join
-			continue
-		}
-		seen[l] = true
-		if v, err := strconv.ParseFloat(l, 64); err == nil {
-			pts = append(pts, v)
-		}
-	}
-	sort.Float64s(pts)
-	return pts
-}
-
-func firstVideoPTS(t *testing.T, init, seg string) float64 {
-	dir := t.TempDir()
-	j := filepath.Join(dir, "one.mp4")
-	a, _ := os.ReadFile(init)
-	b, _ := os.ReadFile(seg)
-	_ = os.WriteFile(j, append(a, b...), 0o644)
-	pts := videoPTS(t, j)
-	if len(pts) == 0 {
-		t.Fatal("no frames in the segment")
-	}
-	return pts[0]
-}

internal/media/inspect.go

@@ -5,6 +5,7 @@ import (
 	"context"
 	"encoding/json"
 	"errors"
+	"math"
 	"os/exec"
 	"strconv"
 	"strings"
@@ -25,6 +26,16 @@ type Track struct {
 	ColorTransfer string  `json:"color_transfer,omitempty"` // smpte2084 / arib-std-b67 mean HDR
 	Channels      int     `json:"channels,omitempty"`
 	AttachedPic   bool    `json:"attached_pic,omitempty"`
+	Spherical     bool    `json:"spherical,omitempty"` // an equirectangular 360 picture
+	Rotation      int     `json:"rotation,omitempty"`  // degrees clockwise to turn it to be shown: 0, 90, 180, 270 (a phone held upright)
+}
+
+// Shown is the picture's size as it is shown, turned by its rotation.
+func (t *Track) Shown() (w, h int) {
+	if t.Rotation == 90 || t.Rotation == 270 {
+		return t.Height, t.Width
+	}
+	return t.Width, t.Height
 }
 
 // Info is what a file holds: the picture to take, the sound to go with it, and the whole.
@@ -37,7 +48,12 @@ type Info struct {
 	Audio     *Track  `json:"audio,omitempty"` // the first sound stream
 }
 
-// HDR reports a picture that needs tone mapping to look right on an SDR screen.
+// PQ reports HDR10 (and Dolby Vision's base layer): a picture that can't be shown on an SDR
+// screen without tone mapping. HLG, the other kind, can, if a little flat.
+func (i Info) PQ() bool { return i.Video != nil && i.Video.ColorTransfer == "smpte2084" }
+
+// HDR reports a picture that needs tone mapping to look right on an SDR screen. (The player
+// tone-maps only PQ: HLG is made to be shown on an SDR screen as it is.)
 func (i Info) HDR() bool {
 	return i.Video != nil && (i.Video.ColorTransfer == "smpte2084" || i.Video.ColorTransfer == "arib-std-b67")
 }
@@ -58,6 +74,11 @@ type probeJSON struct {
 		Disposition   struct {
 			AttachedPic int `json:"attached_pic"`
 		} `json:"disposition"`
+		SideData []struct {
+			Type       string  `json:"side_data_type"`
+			Projection string  `json:"projection"`
+			Rotation   float64 `json:"rotation"`
+		} `json:"side_data_list"`
 	} `json:"streams"`
 	Format struct {
 		FormatName string `json:"format_name"`
@@ -74,7 +95,7 @@ func Inspect(ctx context.Context, c Caps, path string, root bool) (Info, error)
 		return Info{}, errors.New("ffprobe is not installed")
 	}
 	argv := []string{c.FFprobe, "-v", "error", "-show_entries",
-		"stream=index,codec_type,codec_name,profile,pix_fmt,width,height,r_frame_rate,avg_frame_rate,color_transfer,channels,disposition:format=format_name,duration,bit_rate,size",
+		"stream=index,codec_type,codec_name,profile,pix_fmt,width,height,r_frame_rate,avg_frame_rate,color_transfer,channels,disposition:stream_side_data=side_data_type,projection,rotation:format=format_name,duration,bit_rate,size",
 		"-of", "json", path}
 	if root {
 		argv = priv.Wrap(argv...)
@@ -105,6 +126,19 @@ func Inspect(ctx context.Context, c Caps, path string, root bool) (Info, error)
 	for _, s := range pj.Streams {
 		st := &Track{Index: s.Index, Type: s.CodecType, Codec: s.CodecName, Profile: s.Profile, PixFmt: s.PixFmt, Width: s.Width, Height: s.Height,
 			ColorTransfer: s.ColorTransfer, Channels: s.Channels, AttachedPic: s.Disposition.AttachedPic != 0}
+		// A 360 video says so in its spherical tag (ffmpeg reads the old XML one and the sv3d
+		// box); an untagged one is still taken for 360 when it is 2:1 and too big for anything
+		// else.
+		for _, sd := range s.SideData {
+			if sd.Type == "Spherical Mapping" && strings.HasPrefix(sd.Projection, "equirectangular") {
+				st.Spherical = true
+			}
+			if sd.Type == "Display Matrix" {
+				// ffprobe's rotation is counter-clockwise; ffmpeg turns it back the other way
+				st.Rotation = ((-int(math.Round(sd.Rotation/90))*90)%360 + 360) % 360
+			}
+		}
+		st.Spherical = st.Spherical || (s.CodecType == "video" && s.Width == 2*s.Height && s.Width >= 3840)
 		st.FPS = fps(s.AvgFrameRate)
 		if st.FPS == 0 {
 			st.FPS = fps(s.RFrameRate)

internal/media/play.go

@@ -0,0 +1,462 @@
+package media
+
+// Playing a video in the browser at the quality the connection can take: HLS, made on the fly
+// by ffmpeg on the device asked for (the Transcode device setting), YouTube's way. The player
+// is told every quality is all there from the start, in Seg-second pieces, and each piece is
+// made when it is first asked for.
+//
+// One ffmpeg runs per play at a time. It starts at the piece asked for and runs on ahead of the
+// player, paused once it is far enough ahead; a piece asked for well past where it has got to
+// (a seek), or at another quality, ends it and starts a new one there. Every run is made the
+// same way (pieceArgs), so pieces from different runs and qualities follow on from one another.
+//
+// The file is a path on this machine or, when the device asked for is on another one, a URL
+// the hub serves it at. Nothing is kept: a play nobody asks anything of is forgotten, and the
+// folder is emptied when the daemon starts.
+
+import (
+	"bytes"
+	"crypto/rand"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"io/fs"
+	"log"
+	"math"
+	"os"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+const (
+	runAhead = 10               // pieces a run may make past the last one asked for before it is paused
+	jump     = 3                // a piece more than this far past what's made is a seek: start again there
+	idleRun  = 45 * time.Second // a run nobody has asked anything of for this long is ended
+	idlePlay = 30 * time.Minute // a play nobody has asked anything of for this long is forgotten
+	budget   = 2 << 30          // bytes of pieces one play keeps; those asked for least lately go first
+	// PieceWait is the longest a request waits for its piece: under the hub proxy's 60 s wait
+	// for a node to answer.
+	PieceWait = 45 * time.Second
+)
+
+var ErrNotFound = errors.New("no such piece")
+
+// Plays is every play on this machine.
+type Plays struct {
+	dir     string
+	self    string // this binary, for root runs through run-ffmpeg
+	maxRuns int
+
+	mu    sync.Mutex
+	plays map[string]*Play
+	stop  chan struct{}
+}
+
+// NewPlays keeps pieces under <dataDir>/transcode, which it empties first.
+func NewPlays(dataDir, self string, maxRuns int) *Plays {
+	dir := filepath.Join(dataDir, "transcode")
+	_ = os.MkdirAll(dir, 0o700)
+	_ = clearDir(dir)
+	m := &Plays{dir: dir, self: self, maxRuns: max(maxRuns, 1), plays: map[string]*Play{}, stop: make(chan struct{})}
+	go m.tend()
+	return m
+}
+
+// Play is one video opened for watching.
+type Play struct {
+	ID     string `json:"id"`
+	Name   string `json:"name"` // the file's name, for the record
+	Info   Info   `json:"info"`
+	Rungs  []Rung `json:"rungs"`  // best first
+	Device string `json:"device"` // asked for: a render node, "cpu", or "" for the best here
+	Remote bool   `json:"remote"` // the file is on another machine, read over the network
+
+	caps Caps
+	in   string
+	root bool
+	segs int
+	dir  string
+	m    *Plays
+
+	mu    sync.Mutex
+	seen  time.Time
+	run   *playRun
+	tried map[int]int // per rung: recipes that failed to make anything, skipped from now on
+}
+
+type playRun struct {
+	rung   int
+	recipe Recipe
+	start  int // the first piece this run makes
+	head   int // the last piece finished (start-1 before the first)
+	want   int // the last piece asked for
+	asked  time.Time
+	proc   *ffproc
+	paused bool
+	dead   bool // ended, or exited
+	killed bool // ended by us, not by itself
+	exited bool
+	err    error
+	tick   chan struct{} // closed when head moves or the run ends
+}
+
+func newPlayID() string {
+	b := make([]byte, 12)
+	_, _ = rand.Read(b)
+	return hex.EncodeToString(b)
+}
+
+// Open starts a play of in (a path, or an http URL for a file on another machine). Nothing
+// runs until a piece is asked for.
+func (m *Plays) Open(c Caps, root bool, in, name string, src Info, device string) (*Play, error) {
+	rungs := Ladder(src)
+	if len(rungs) == 0 || src.Duration <= 0 {
+		return nil, errors.New("this video's length or picture size isn't known, so it can't be played")
+	}
+	if len(recipes(c, device, src, rungs[0])) == 0 {
+		return nil, errors.New("nothing on this machine encodes H.264 (no GPU that does, and no libx264)")
+	}
+	segs := max(int(math.Ceil((src.Duration-0.25)/Seg)), 1) // the last piece takes in what's left
+	p := &Play{ID: newPlayID(), Name: name, Info: src, Rungs: rungs, Device: device, Remote: strings.HasPrefix(in, "http"),
+		caps: c, in: in, root: root, segs: segs, m: m, seen: time.Now(), tried: map[int]int{}}
+	p.dir = filepath.Join(m.dir, p.ID)
+	m.mu.Lock()
+	m.plays[p.ID] = p
+	m.mu.Unlock()
+	return p, nil
+}
+
+func (m *Plays) Find(id string) (*Play, bool) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	p, ok := m.plays[id]
+	return p, ok
+}
+
+// Close ends a play and deletes its pieces.
+func (m *Plays) Close(id string) {
+	m.mu.Lock()
+	p, ok := m.plays[id]
+	delete(m.plays, id)
+	m.mu.Unlock()
+	if ok {
+		p.mu.Lock()
+		p.end()
+		p.mu.Unlock()
+		go os.RemoveAll(p.dir)
+	}
+}
+
+func (p *Play) rung(name string) int {
+	for i, r := range p.Rungs {
+		if r.Name == name {
+			return i
+		}
+	}
+	return -1
+}
+
+// Master is the playlist of qualities, lowest first, so a player that starts on the first one
+// listed starts on the one that surely plays.
+func (p *Play) Master() []byte {
+	var b bytes.Buffer
+	b.WriteString("#EXTM3U\n#EXT-X-VERSION:3\n#EXT-X-INDEPENDENT-SEGMENTS\n")
+	for i := len(p.Rungs) - 1; i >= 0; i-- {
+		r := p.Rungs[i]
+		fmt.Fprintf(&b, "#EXT-X-STREAM-INF:BANDWIDTH=%d,AVERAGE-BANDWIDTH=%d,RESOLUTION=%dx%d,FRAME-RATE=%.3f,CODECS=\"%s\"\n%s.m3u8\n",
+			r.Peak, r.Bitrate+audioRate, r.Width, r.Height, r.FPS, r.codecs, r.Name)
+	}
+	return b.Bytes()
+}
+
+// Media is one quality's playlist: every piece, made or not.
+func (p *Play) Media(name string) ([]byte, bool) {
+	if p.rung(name) < 0 {
+		return nil, false
+	}
+	var b bytes.Buffer
+	last := p.Info.Duration - float64(p.segs-1)*Seg
+	fmt.Fprintf(&b, "#EXTM3U\n#EXT-X-VERSION:3\n#EXT-X-TARGETDURATION:%d\n#EXT-X-MEDIA-SEQUENCE:0\n#EXT-X-PLAYLIST-TYPE:VOD\n#EXT-X-INDEPENDENT-SEGMENTS\n",
+		int(math.Ceil(math.Max(last, Seg))))
+	for n := 0; n < p.segs; n++ {
+		d := Seg
+		if n == p.segs-1 {
+			d = last
+		}
+		fmt.Fprintf(&b, "#EXTINF:%.6f,\n%s/%d.ts\n", d, name, n)
+	}
+	b.WriteString("#EXT-X-ENDLIST\n")
+	return b.Bytes(), true
+}
+
+func (p *Play) piece(rung, n int) string {
+	return filepath.Join(p.dir, p.Rungs[rung].Name, strconv.Itoa(n)+".ts")
+}
+
+// Segment answers the file of piece n at quality name, making it first if need be: it waits
+// for the run already making it, or starts one there.
+func (p *Play) Segment(done <-chan struct{}, name string, n int) (string, error) {
+	ri := p.rung(name)
+	if ri < 0 || n < 0 || n >= p.segs {
+		return "", ErrNotFound
+	}
+	path := p.piece(ri, n)
+	deadline := time.After(PieceWait)
+	for {
+		p.mu.Lock()
+		now := time.Now()
+		p.seen = now
+		r := p.run
+		if r != nil && r.rung == ri && !r.dead {
+			r.want, r.asked = n, now
+		}
+		if exists(path) {
+			p.mu.Unlock()
+			_ = os.Chtimes(path, now, now) // the budget keeps what was asked for last
+			return path, nil
+		}
+		if r != nil && r.exited && !r.killed && r.rung == ri && r.start == n {
+			// This run started at this very piece and ended without it: that way of making
+			// it doesn't work here. The next one is tried, if there is one.
+			p.tried[ri]++
+			log.Printf("files: play %s: %s made nothing at %s piece %d: %v: %s", p.Name, r.recipe.Engine, name, n, r.err, r.proc.lastWords())
+			if p.tried[ri] >= len(recipes(p.caps, p.Device, p.Info, p.Rungs[ri])) {
+				err := fmt.Errorf("couldn't make %s: %s", p.Rungs[ri].Label, r.proc.lastWords())
+				p.mu.Unlock()
+				return "", err
+			}
+			p.run = nil
+			r = nil
+		}
+		if r == nil || r.dead || r.rung != ri || n < r.start || n <= r.head || n > r.head+jump {
+			p.mu.Unlock()
+			p.m.makeRoom(p)
+			p.mu.Lock()
+			if p.run == r { // nobody started one meanwhile
+				if err := p.start(ri, n); err != nil {
+					p.mu.Unlock()
+					return "", err
+				}
+			}
+			p.mu.Unlock()
+			continue
+		}
+		tick := r.tick
+		p.mu.Unlock()
+		select {
+		case <-tick:
+		case <-time.After(200 * time.Millisecond):
+		case <-done:
+			return "", errors.New("gone")
+		case <-deadline:
+			return "", fmt.Errorf("%s piece %d took longer than %s", name, n, PieceWait)
+		}
+	}
+}
+
+// end stops this play's run. p.mu is held.
+func (p *Play) end() {
+	if r := p.run; r != nil && !r.dead {
+		r.dead, r.killed = true, true
+		r.proc.kill()
+	}
+}
+
+// makeRoom ends the run asked of least lately, if every slot is taken by another play. It
+// never holds two plays' locks at once.
+func (m *Plays) makeRoom(except *Play) {
+	m.mu.Lock()
+	plays := make([]*Play, 0, len(m.plays))
+	for _, p := range m.plays {
+		if p != except {
+			plays = append(plays, p)
+		}
+	}
+	m.mu.Unlock()
+	var oldest *Play
+	var when time.Time
+	running := 0
+	for _, p := range plays {
+		p.mu.Lock()
+		if r := p.run; r != nil && !r.dead {
+			running++
+			if oldest == nil || r.asked.Before(when) {
+				oldest, when = p, r.asked
+			}
+		}
+		p.mu.Unlock()
+	}
+	if running >= m.maxRuns && oldest != nil {
+		oldest.mu.Lock()
+		oldest.end()
+		oldest.mu.Unlock()
+		log.Printf("files: %d plays encoding; ended %s's, asked of least lately", running, oldest.Name)
+	}
+}
+
+// start begins a run making quality ri from piece n on, ending the one before. p.mu is held.
+func (p *Play) start(ri, n int) error {
+	p.end()
+	rs := recipes(p.caps, p.Device, p.Info, p.Rungs[ri])
+	if p.tried[ri] >= len(rs) {
+		return errors.New("nothing here can make this quality")
+	}
+	rc := rs[p.tried[ri]]
+	out := filepath.Join(p.dir, p.Rungs[ri].Name)
+	if err := os.MkdirAll(out, 0o700); err != nil {
+		return err
+	}
+	proc, err := startFF(p.caps, p.m.self, p.root, pieceArgs(out, p.in, p.Info, rc, p.Rungs[ri], n))
+	if err != nil {
+		return err
+	}
+	r := &playRun{rung: ri, recipe: rc, start: n, head: n - 1, want: n, asked: time.Now(), proc: proc, tick: make(chan struct{})}
+	p.run = r
+	go func() {
+		<-proc.done
+		p.mu.Lock()
+		defer p.mu.Unlock()
+		r.dead, r.exited, r.err = true, true, proc.err
+		p.scan(r)
+		close(r.tick)
+		r.tick = make(chan struct{})
+	}()
+	return nil
+}
+
+// scan moves a run's head on past the pieces it has finished. p.mu is held.
+func (p *Play) scan(r *playRun) bool {
+	moved := false
+	for r.head+1 < p.segs && exists(p.piece(r.rung, r.head+1)) {
+		r.head++
+		moved = true
+	}
+	return moved
+}
+
+// PlayStatus is what a play is doing, for the player's menu.
+type PlayStatus struct {
+	Engine  string `json:"engine,omitempty"` // what the run going now runs on
+	Quality string `json:"quality,omitempty"`
+	From    int    `json:"from"` // seconds into the film the run started at
+	Ahead   int    `json:"ahead"`
+	Paused  bool   `json:"paused,omitempty"`
+}
+
+func (p *Play) Status() PlayStatus {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	r := p.run
+	if r == nil || r.dead {
+		return PlayStatus{}
+	}
+	return PlayStatus{Engine: r.recipe.Engine, Quality: p.Rungs[r.rung].Label, From: int(float64(r.start) * Seg), Ahead: r.head - r.want, Paused: r.paused}
+}
+
+// Engine is what this play would run on first, for the player to say before anything runs.
+func (p *Play) Engine() string {
+	rs := recipes(p.caps, p.Device, p.Info, p.Rungs[0])
+	if len(rs) == 0 {
+		return ""
+	}
+	return rs[0].Engine
+}
+
+// tend, ten times a second: notices finished pieces, pauses a run that is far enough ahead and
+// wakes it when the player catches up, ends one nobody is asking anything of; and every ten
+// seconds keeps each play inside its budget and forgets plays nobody watches.
+func (m *Plays) tend() {
+	t := time.NewTicker(100 * time.Millisecond)
+	defer t.Stop()
+	for i := 0; ; i++ {
+		select {
+		case <-m.stop:
+			return
+		case <-t.C:
+		}
+		m.mu.Lock()
+		plays := make([]*Play, 0, len(m.plays))
+		for _, p := range m.plays {
+			plays = append(plays, p)
+		}
+		m.mu.Unlock()
+		now := time.Now()
+		for _, p := range plays {
+			p.mu.Lock()
+			if r := p.run; r != nil && !r.dead {
+				if p.scan(r) {
+					close(r.tick)
+					r.tick = make(chan struct{})
+				}
+				ahead := r.head - r.want
+				switch {
+				case now.Sub(r.asked) > idleRun:
+					p.end()
+				case !r.paused && ahead >= runAhead:
+					if r.proc.pause() == nil {
+						r.paused = true
+					}
+				case r.paused && ahead < runAhead-3:
+					if r.proc.resume() == nil {
+						r.paused = false
+					}
+				}
+			}
+			idle := now.Sub(p.seen) > idlePlay
+			p.mu.Unlock()
+			if idle {
+				m.Close(p.ID)
+			} else if i%100 == 0 {
+				p.trim()
+			}
+		}
+	}
+}
+
+// trim deletes the pieces asked for least lately until the play is inside its budget, never
+// the ones its run has made ahead of the player.
+func (p *Play) trim() {
+	type piece struct {
+		path string
+		size int64
+		at   time.Time
+	}
+	var all []piece
+	var total int64
+	_ = filepath.WalkDir(p.dir, func(path string, d fs.DirEntry, err error) error {
+		if err != nil || d.IsDir() || !strings.HasSuffix(path, ".ts") {
+			return nil
+		}
+		if fi, err := d.Info(); err == nil {
+			all = append(all, piece{path, fi.Size(), fi.ModTime()})
+			total += fi.Size()
+		}
+		return nil
+	})
+	if total <= budget {
+		return
+	}
+	p.mu.Lock()
+	keep := map[string]bool{}
+	if r := p.run; r != nil && !r.dead {
+		for n := r.want - 1; n <= r.head+1; n++ {
+			if n >= 0 && n < p.segs {
+				keep[p.piece(r.rung, n)] = true
+			}
+		}
+	}
+	p.mu.Unlock()
+	sort.Slice(all, func(a, b int) bool { return all[a].at.Before(all[b].at) })
+	for _, x := range all {
+		if total <= budget*3/4 {
+			break
+		}
+		if !keep[x.path] && os.Remove(x.path) == nil {
+			total -= x.size
+		}
+	}
+}

internal/media/play_test.go

@@ -0,0 +1,129 @@
+package media
+
+import (
+	"context"
+	"encoding/json"
+	"math"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+)
+
+// inspectPiece answers when a piece's first video packet is shown, whether it is a keyframe,
+// how many keyframes the piece has, and when its last packet is.
+func inspectPiece(t *testing.T, path string) (first float64, key bool, keys int, last float64) {
+	t.Helper()
+	b, err := exec.Command("ffprobe", "-v", "error", "-show_entries", "packet=pts_time,flags,codec_type", "-of", "json", path).Output()
+	if err != nil {
+		t.Fatalf("ffprobe %s: %v", path, err)
+	}
+	var p struct {
+		Packets []struct {
+			PTS   string `json:"pts_time"`
+			Flags string `json:"flags"`
+			Type  string `json:"codec_type"`
+		} `json:"packets"`
+	}
+	_ = json.Unmarshal(b, &p)
+	first = -1
+	for _, k := range p.Packets {
+		if k.Type != "video" {
+			continue
+		}
+		x, _ := strconv.ParseFloat(k.PTS, 64)
+		if first < 0 {
+			first, key = x, strings.HasPrefix(k.Flags, "K")
+		}
+		first = math.Min(first, x)
+		last = math.Max(last, x)
+		if strings.HasPrefix(k.Flags, "K") {
+			keys++
+		}
+	}
+	return
+}
+
+// TestPieces makes pieces of a made-up video (or SRC, on DEVICE, to try the real thing on a
+// machine with a GPU) in order and after a seek, at two qualities, and checks they line up:
+// each starts on a keyframe at its own place in the film, with no other keyframe inside it.
+func TestPieces(t *testing.T) {
+	if _, err := exec.LookPath("ffmpeg"); err != nil {
+		t.Skip("no ffmpeg")
+	}
+	dir := t.TempDir()
+	in := os.Getenv("SRC")
+	if in == "" {
+		in = filepath.Join(dir, "in.mkv")
+		if out, err := exec.Command("ffmpeg", "-v", "error", "-f", "lavfi", "-i", "testsrc2=size=1920x1080:rate=50:duration=40",
+			"-f", "lavfi", "-i", "sine=frequency=440:duration=40", "-c:v", "libx264", "-preset", "ultrafast", "-g", "500",
+			"-c:a", "libopus", "-shortest", in).CombinedOutput(); err != nil {
+			t.Fatalf("making the test video: %v: %s", err, out)
+		}
+	}
+	caps := Probe(context.Background())
+	if caps.Error != "" {
+		t.Skip(caps.Error)
+	}
+	info, err := Inspect(context.Background(), caps, in, false)
+	if err != nil {
+		t.Fatal(err)
+	}
+	m := NewPlays(dir, "", 2)
+	device := os.Getenv("DEVICE")
+	if device == "" {
+		device = CPU
+	}
+	p, err := m.Open(caps, false, in, "in.mkv", info, device)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer m.Close(p.ID)
+	if !strings.Contains(string(p.Master()), "RESOLUTION=") {
+		t.Fatal(string(p.Master()))
+	}
+	low := p.Rungs[len(p.Rungs)-1].Name
+	var offset float64
+	far := min(p.segs-2, 60)
+	for i, n := range []int{0, 1, 2, far, far + 1, 3} {
+		t0 := time.Now()
+		path, err := p.Segment(nil, low, n)
+		if err != nil {
+			t.Fatalf("piece %d: %v", n, err)
+		}
+		first, key, keys, last := inspectPiece(t, path)
+		if i == 0 {
+			offset = first
+		}
+		at := first - offset
+		t.Logf("piece %d: %d ms on %s, starts %.3f, %d keyframe(s), %.2f s long", n, time.Since(t0).Milliseconds(), p.Status().Engine, at, keys, last-first)
+		if !key || keys != 1 {
+			t.Errorf("piece %d: first packet keyframe %v, %d keyframes", n, key, keys)
+		}
+		if math.Abs(at-float64(n)*Seg) > 0.05 {
+			t.Errorf("piece %d starts at %.3f, want %.3f", n, at, float64(n)*Seg)
+		}
+		if n < p.segs-1 && math.Abs(last-first-Seg) > 0.1 {
+			t.Errorf("piece %d is %.3f s long", n, last-first)
+		}
+	}
+	// another quality, at a piece already made at the first: its own run, lined up too
+	if len(p.Rungs) > 1 {
+		path, err := p.Segment(nil, p.Rungs[0].Name, 2)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if first, key, _, _ := inspectPiece(t, path); !key || math.Abs(first-offset-2*Seg) > 0.05 {
+			t.Errorf("%s piece 2 starts %.3f (keyframe %v)", p.Rungs[0].Name, first-offset, key)
+		}
+	}
+	if _, err := p.Segment(nil, low, p.segs); err != ErrNotFound {
+		t.Errorf("past the end: %v", err)
+	}
+	if _, err := p.Segment(nil, "999", 0); err != ErrNotFound {
+		t.Errorf("no such quality: %v", err)
+	}
+}

internal/media/thumbs.go

@@ -62,6 +62,24 @@ type Caps struct {
 type Opts struct {
 	Size    int           // the long edge in pixels; 256 when zero
 	Timeout time.Duration // for the whole job, ffprobe and every ffmpeg attempt; 30 s when zero
+	Kind    string        // image | video | audio; from the name when empty (a URL has none)
+}
+
+// Only is the caps with one device left to decode on: a render node, CPU for none, or ""
+// for all of them — the Transcode device setting, which previews follow too. A device this
+// machine doesn't have leaves the CPU.
+func (c Caps) Only(device string) Caps {
+	if device == "" {
+		return c
+	}
+	out := c
+	out.GPUs = nil
+	for _, g := range c.GPUs {
+		if g.Device == device {
+			out.GPUs = append(out.GPUs, g)
+		}
+	}
+	return out
 }
 
 func (o Opts) defaults() Opts {
@@ -81,6 +99,8 @@ type Result struct {
 	Codec  string `json:"codec"`
 	Width  int    `json:"width"`
 	Height int    `json:"height"`
+	// Failure is why the file can't be previewed, when another machine tried (no JPEG then).
+	Failure string `json:"-"`
 }
 
 // ErrUnsupported: not an image or a video by its name.
@@ -356,7 +376,10 @@ func (a attempt) engine() string {
 
 // Make produces the preview of one file.
 func Make(ctx context.Context, c Caps, path string, o Opts) (Result, error) {
-	kind := Kind(path)
+	kind := o.Kind
+	if kind == "" {
+		kind = Kind(path)
+	}
 	if kind == "" {
 		return Result{}, ErrUnsupported
 	}

internal/metrics/cpupower.go

@@ -95,30 +95,41 @@ func readRAPL(root string, dirs []string) ([]raplCount, bool) {
 // is carried across its range. One that went backwards with no known range, or a set of
 // counters that changed between readings, gives no answer rather than a wrong one.
 func raplWatts(prev, cur []raplCount, max map[string]uint64, dt float64) (float64, bool) {
-	if dt <= 0 || len(prev) == 0 || len(prev) != len(cur) {
-		return 0, false
+	each, ok := raplWattsEach(prev, cur, max, dt)
+	var w float64
+	for _, x := range each {
+		w += x
 	}
-	var uj float64
+	return w, ok
+}
+
+// raplWattsEach is raplWatts per counter: one package each.
+func raplWattsEach(prev, cur []raplCount, max map[string]uint64, dt float64) ([]float64, bool) {
+	if dt <= 0 || len(prev) == 0 || len(prev) != len(cur) {
+		return nil, false
+	}
+	out := make([]float64, len(cur))
 	for i := range cur {
 		if cur[i].Dir != prev[i].Dir {
-			return 0, false
+			return nil, false
 		}
 		if cur[i].UJ >= prev[i].UJ {
-			uj += float64(cur[i].UJ - prev[i].UJ)
+			out[i] = float64(cur[i].UJ-prev[i].UJ) / 1e6 / dt
 			continue
 		}
 		m := max[cur[i].Dir]
 		if m == 0 || prev[i].UJ > m {
-			return 0, false
+			return nil, false
 		}
-		uj += float64(m - prev[i].UJ + cur[i].UJ)
+		out[i] = float64(m-prev[i].UJ+cur[i].UJ) / 1e6 / dt
 	}
-	return uj / 1e6 / dt, true
+	return out, true
 }
 
 // cpuPower is the CPU packages' power since the previous sample, in watts; 0 when this
-// machine has no counters or no way to read them.
-func (c *Collector) cpuPower(now time.Time) float64 {
+// machine has no counters or no way to read them. On a machine with more than one package
+// it also gives each one's, indexed by package id (the socket).
+func (c *Collector) cpuPower(now time.Time) (float64, []float64) {
 	p := &c.power
 	if !p.probed {
 		p.probed = true
@@ -131,7 +142,7 @@ func (c *Collector) cpuPower(now time.Time) float64 {
 		}
 	}
 	if len(p.dirs) == 0 {
-		return 0
+		return 0, nil
 	}
 	var cur []raplCount
 	ok := false
@@ -148,14 +159,40 @@ func (c *Collector) cpuPower(now time.Time) float64 {
 	}
 	if !ok {
 		p.prev = nil
-		return 0
+		return 0, nil
 	}
-	w, good := raplWatts(p.prev, cur, p.max, now.Sub(p.prevAt).Seconds())
+	each, good := raplWattsEach(p.prev, cur, p.max, now.Sub(p.prevAt).Seconds())
 	p.prev, p.prevAt = cur, now
 	if !good {
-		return 0
+		return 0, nil
 	}
-	return round1(w)
+	var w float64
+	for _, x := range each {
+		w += x
+	}
+	return round1(w), perPackage(cur, each)
+}
+
+// perPackage puts each counter's watts at its package's id — intel-rapl:1 is "package-1" —
+// and is nil for a single package, whose figure is the total.
+func perPackage(cur []raplCount, each []float64) []float64 {
+	if len(cur) < 2 {
+		return nil
+	}
+	var out []float64
+	for i, c := range cur {
+		id := i
+		if n, ok := strings.CutPrefix(strings.TrimSpace(readStr(filepath.Join(raplRoot, c.Dir, "name"))), "package-"); ok {
+			if v, err := strconv.Atoi(n); err == nil && v >= 0 && v < 64 {
+				id = v
+			}
+		}
+		for len(out) <= id {
+			out = append(out, 0)
+		}
+		out[id] = round1(each[i])
+	}
+	return out
 }
 
 // energyHelper is the root child that answers each newline with the package counters, one

internal/metrics/cputemp_test.go

@@ -0,0 +1,16 @@
+package metrics
+
+import "testing"
+
+// a two-socket board: two coretemp chips, and the CPU's temperature is the hotter package
+func TestCPUTempHottestPackage(t *testing.T) {
+	temps := []Temp{
+		{Chip: "coretemp", Label: "Core 0", C: 31},
+		{Chip: "coretemp", Label: "Core 0", C: 44, Pkg: 1},
+		{Chip: "coretemp", Label: "Package id 0", C: 35},
+		{Chip: "coretemp", Label: "Package id 1", C: 41, Pkg: 1},
+	}
+	if got := cpuTemp(temps); got != 41 {
+		t.Fatalf("cpuTemp = %v, want 41", got)
+	}
+}

internal/metrics/fold.go

@@ -1,5 +1,7 @@
 package metrics
 
+import "strconv"
+
 // One kept history point is the average of the samples behind it. At the default ten-minute
 // window that bucket is a single sample and nothing is averaged; at a day it is five minutes
 // of them, and the difference matters — a graph of the last day drawn from whichever instant
@@ -43,6 +45,21 @@ func fold(b []Sample) Sample {
 	if pn > 0 {
 		out.CPUPower = round1(pw / pn)
 	}
+	// and each package's, the same way
+	out.CPUPowerPkgs = nil
+	for _, s := range b {
+		for i, w := range s.CPUPowerPkgs {
+			for len(out.CPUPowerPkgs) <= i {
+				out.CPUPowerPkgs = append(out.CPUPowerPkgs, 0)
+			}
+			out.CPUPowerPkgs[i] += w
+		}
+	}
+	if pn > 0 {
+		for i := range out.CPUPowerPkgs {
+			out.CPUPowerPkgs[i] = round1(out.CPUPowerPkgs[i] / pn)
+		}
+	}
 	out.Load[0] = sum(func(s Sample) float64 { return s.Load[0] })
 	out.Load[1] = sum(func(s Sample) float64 { return s.Load[1] })
 	out.Load[2] = sum(func(s Sample) float64 { return s.Load[2] })
@@ -70,13 +87,13 @@ func fold(b []Sample) Sample {
 	temps := avgBy(b, func(s Sample) map[string][]float64 {
 		m := make(map[string][]float64, len(s.Temps))
 		for _, x := range s.Temps {
-			m[x.Chip+"/"+x.Label] = []float64{x.C}
+			m[x.Chip+"/"+x.Label+"/"+strconv.Itoa(x.Pkg)] = []float64{x.C}
 		}
 		return m
 	})
 	out.Temps = append([]Temp(nil), out.Temps...)
 	for i, x := range out.Temps {
-		if v, ok := temps[x.Chip+"/"+x.Label]; ok {
+		if v, ok := temps[x.Chip+"/"+x.Label+"/"+strconv.Itoa(x.Pkg)]; ok {
 			out.Temps[i].C = v[0]
 		}
 	}
@@ -122,9 +139,54 @@ func fold(b []Sample) Sample {
 			out.Disks[i].ReadBps, out.Disks[i].WriteBps, out.Disks[i].UtilPct = v[0], v[1], v[2]
 		}
 	}
+	out.ZFSWrite = foldZFSWrites(b, out.ZFSWrite)
 	return out
 }
 
+// foldZFSWrites averages the write path's rates and the buffer's fill over a bucket, per pool;
+// everything else (limits, the mover's state, the last walk) is the newest sample's.
+func foldZFSWrites(b []Sample, last *ZFSWrites) *ZFSWrites {
+	if last == nil {
+		return nil
+	}
+	w := *last
+	w.Pools = append([]PoolWrites(nil), last.Pools...)
+	var dirty, delayed, over, n float64
+	for _, s := range b {
+		if s.ZFSWrite != nil {
+			dirty, delayed, over, n = dirty+float64(s.ZFSWrite.Dirty), delayed+s.ZFSWrite.DelayedPerSec, over+s.ZFSWrite.OverMaxPerSec, n+1
+		}
+	}
+	if n > 0 {
+		w.Dirty, w.DelayedPerSec, w.OverMaxPerSec = uint64(dirty/n), round1(delayed/n), round1(over/n)
+	}
+	per := avgBy(b, func(s Sample) map[string][]float64 {
+		m := map[string][]float64{}
+		if s.ZFSWrite == nil {
+			return m
+		}
+		for _, p := range s.ZFSWrite.Pools {
+			v := []float64{p.WriteBps, p.LogicalBps, 0, 0}
+			if p.Landing != nil {
+				v[2], v[3] = p.Landing.InBps, p.Landing.DrainBps
+			}
+			m[p.Pool] = v
+		}
+		return m
+	})
+	for i, p := range w.Pools {
+		if v, ok := per[p.Pool]; ok {
+			w.Pools[i].WriteBps, w.Pools[i].LogicalBps = round1(v[0]), round1(v[1])
+			if p.Landing != nil {
+				l := *p.Landing
+				l.InBps, l.DrainBps = round1(v[2]), round1(v[3])
+				w.Pools[i].Landing = &l
+			}
+		}
+	}
+	return &w
+}
+
 // avgBy averages each sample's named numbers across the bucket, per name.
 func avgBy(b []Sample, of func(Sample) map[string][]float64) map[string][]float64 {
 	sums := map[string][]float64{}

internal/metrics/gpu.go

@@ -49,6 +49,7 @@ type GPU struct {
 	PowerW     float64            `json:"power_w,omitempty"`
 	PowerCapW  float64            `json:"power_cap_w,omitempty"`
 	FanRPM     int                `json:"fan_rpm,omitempty"`
+	FanPct     *float64           `json:"fan_pct,omitempty"` // fan speed as a % of its range: nvidia-smi's fan.speed, or the hwmon's pwm1; nil when the card has no such reading (0 is a stopped fan)
 	MHz        float64            `json:"mhz,omitempty"`     // actual clock (0 when power-gated)
 	ReqMHz     float64            `json:"req_mhz,omitempty"` // what the driver asked for
 	MaxMHz     float64            `json:"max_mhz,omitempty"`
@@ -56,7 +57,7 @@ type GPU struct {
 	MemTotal   uint64             `json:"mem_total,omitempty"`
 	Clients    []GPUClient        `json:"clients"`
 	ClientsOK  bool               `json:"clients_ok"`        // false when fdinfo of other users' processes could not be read
-	Stalled    bool               `json:"stalled,omitempty"` // a sysfs read of this card never returned; its readings are paused until restart
+	Stalled    bool               `json:"stalled,omitempty"` // a read of this card has been out for gpuStallAfter; its readings are paused until it returns
 }
 
 // GPUClient is one process with the device open.
@@ -79,14 +80,24 @@ type gpuDev struct {
 	prevIdle, prevEnergy                     float64
 	prevAt                                   time.Time
 	nv                                       []GPUClient  // nvidia: the process table from the last read (pmon), fdinfo says nothing there
+	nvRead                                   []GPUClient  // the same, as the read in flight left it: only that read touches it
+	pending                                  chan gpuRead // the read in flight, nil when none is
+	pendingAt                                time.Time
+	last                                     *GPU                  // the last read that came back
+	readFn                                   func(*GPU, time.Time) // nil is read (tests stand a slow one in)
+}
+
+type gpuRead struct {
+	x  GPU
+	nv []GPUClient
 }
 
 type gpuState struct {
 	devs     []*gpuDev
 	listedAt time.Time
 	clients  map[string]clientPrev  // device key + client id → engine totals at the previous scan
-	last     []drmFD               // the newest client walk, reused between walks
-	scanAt   time.Time
+	last     []drmFD                // the newest client walk
+	scan     func() ([]drmFD, bool) // the walk; nil is scanClients (tests stand a fake list in)
 	helper   *drmHelper
 	ok       bool // the last walk saw every process
 }
@@ -97,7 +108,6 @@ type clientPrev struct {
 }
 
 const gpuRelist = time.Minute // how often the card list is re-read: hotplug is rare, the readdir is cheap
-const gpuScanEvery = time.Second // engine shares over less than this are noise
 
 // gpus is the GPU section of a sample. cpuTemp stands in for an integrated GPU's
 // temperature: it is the same die.
@@ -112,10 +122,13 @@ func (c *Collector) gpus(now time.Time, cpuTemp float64) []GPU {
 	if g.clients == nil {
 		g.clients = map[string]clientPrev{}
 	}
-	if g.scanAt.IsZero() || now.Sub(g.scanAt) >= gpuScanEvery {
-		g.last, g.ok = c.scanClients()
-		g.scanAt = now
+	// The clients are read on every sample, like everything else in it: their engine shares
+	// move on the same tick as the device's own numbers, at whatever rate is set.
+	scan := c.scanClients
+	if g.scan != nil {
+		scan = g.scan
 	}
+	g.last, g.ok = scan()
 	out := make([]GPU, 0, len(g.devs))
 	live := map[string]bool{}
 	for _, d := range g.devs {
@@ -346,28 +359,60 @@ func lspciName(desc string) string {
 	return strings.TrimSpace(vendor + " " + desc)
 }
 
-// readWithin is read with a deadline. A sysfs read that does not come back must not take
-// the whole sampler with it — every graph on every page of this machine stops — so the
-// device is marked stalled, its goroutine abandoned, and the rest of the sample goes on.
-const gpuReadWithin = 3 * time.Second
+// readWithin is read with a deadline. A read that does not come back must not take the
+// whole sampler with it — every graph on every page of this machine stops — so one still out
+// after gpuReadWithin is left running, the sample goes on with the card's last readings, and
+// no second read starts until it returns. Late is normal for nvidia-smi on a busy card; only a
+// read out for gpuStallAfter marks the card stalled, and that clears the moment it returns. A
+// sysfs read that never returns keeps it stalled, at the cost of one goroutine.
+var (
+	gpuReadWithin = 3 * time.Second
+	gpuStallAfter = 15 * time.Second // past both of readNvidia's nvidia-smi calls timing out (runOut, 5 s each)
+)
 
 func (d *gpuDev) readWithin(x *GPU, now time.Time) {
+	var r gpuRead
+	back := false
+	if d.pending == nil {
+		done := make(chan gpuRead, 1)
+		y := *x
+		read := d.read
+		if d.readFn != nil {
+			read = d.readFn
+		}
+		go func() {
+			read(&y, now)
+			done <- gpuRead{y, d.nvRead}
+		}()
+		d.pending, d.pendingAt = done, now
+		select {
+		case r, back = <-done:
+		case <-time.After(gpuReadWithin):
+		}
+	} else { // started on an earlier sample: taken if it is back, never waited on again
+		select {
+		case r, back = <-d.pending:
+		default:
+		}
+	}
+	ok := x.ClientsOK
+	defer func() { x.ClientsOK = ok }()
+	if back {
 		if d.stalled {
-		x.Stalled = true
+			log.Printf("gpu: %s (%s): its read came back after %s; readings resume", d.name, d.card, now.Sub(d.pendingAt).Round(time.Second))
+		}
+		d.pending, d.nv, d.stalled, d.last = nil, r.nv, false, &r.x
+		*x = r.x
 		return
 	}
-	done := make(chan GPU, 1)
-	go func() {
-		y := *x
-		d.read(&y, now)
-		done <- y
-	}()
-	select {
-	case y := <-done:
-		*x = y
-	case <-time.After(gpuReadWithin):
+	if d.last != nil {
+		*x = *d.last
+	}
+	if now.Sub(d.pendingAt) >= gpuStallAfter {
+		if !d.stalled {
+			log.Printf("gpu: %s (%s): a read has not returned in %s; its readings are paused until it does", d.name, d.card, gpuStallAfter)
+		}
 		d.stalled, x.Stalled = true, true
-		log.Printf("gpu: %s (%s): a sysfs read did not return within %s; its readings are paused until restart", d.name, d.card, gpuReadWithin)
 	}
 }
 
@@ -435,6 +480,14 @@ func (d *gpuDev) read(x *GPU, now time.Time) {
 			x.PowerCapW = p / 1e6
 		}
 		x.FanRPM = int(readF(d.hwmon + "/fan1_input"))
+		if pwm, err := strconv.ParseFloat(strings.TrimSpace(readStr(d.hwmon+"/pwm1")), 64); err == nil {
+			top := readF(d.hwmon + "/pwm1_max")
+			if top <= 0 {
+				top = 255
+			}
+			v := round1(pwm / top * 100)
+			x.FanPct = &v
+		}
 	}
 	d.prevAt = now
 }
@@ -718,11 +771,11 @@ func ueventValue(p, key string) string {
 // sample (~20 ms and ~75 ms on a GTX 965M), no root needed. The PCI address selects the
 // card, so two NVIDIA GPUs stay apart.
 func (d *gpuDev) readNvidia(x *GPU) {
-	if out, ok := runOut("nvidia-smi", "--id="+d.bus, "--query-gpu=utilization.gpu,utilization.encoder,utilization.decoder,memory.used,memory.total,clocks.sm,clocks.max.sm,temperature.gpu,power.draw,power.limit", "--format=csv,noheader,nounits"); ok {
+	if out, ok := runOut("nvidia-smi", "--id="+d.bus, "--query-gpu=utilization.gpu,utilization.encoder,utilization.decoder,memory.used,memory.total,clocks.sm,clocks.max.sm,temperature.gpu,power.draw,power.limit,fan.speed", "--format=csv,noheader,nounits"); ok {
 		parseNvidiaQuery(x, out)
 	}
 	if out, ok := runOut("nvidia-smi", "pmon", "-c", "1", "-s", "um", "--id="+d.bus); ok {
-		d.nv = parseNvidiaPmon(out)
+		d.nvRead = parseNvidiaPmon(out)
 	}
 }
 
@@ -737,7 +790,8 @@ func nvNum(s string) (float64, bool) {
 }
 
 // parseNvidiaQuery fills a GPU from one --query-gpu line in the field order readNvidia asks
-// for: gpu%, encoder%, decoder%, mem used MiB, mem total MiB, SM MHz, max SM MHz, °C, W, W cap.
+// for: gpu%, encoder%, decoder%, mem used MiB, mem total MiB, SM MHz, max SM MHz, °C, W, W cap,
+// fan % ("[N/A]" on a card with no fan of its own: a laptop's, a passive datacentre card).
 func parseNvidiaQuery(x *GPU, line string) {
 	f := strings.Split(strings.TrimSpace(line), ",")
 	if len(f) < 10 {
@@ -773,6 +827,11 @@ func parseNvidiaQuery(x *GPU, line string) {
 	if v, ok := nvNum(f[9]); ok {
 		x.PowerCapW = v
 	}
+	if len(f) > 10 {
+		if v, ok := nvNum(f[10]); ok {
+			x.FanPct = &v
+		}
+	}
 }
 
 // parseNvidiaPmon reads `nvidia-smi pmon -c 1 -s um`: a header naming the columns, then one

internal/metrics/gpu_stall_test.go

@@ -0,0 +1,52 @@
+package metrics
+
+import (
+	"testing"
+	"time"
+)
+
+// A read that comes back late carries the last readings meanwhile and is used when it lands;
+// only one out past gpuStallAfter marks the card stalled, and its return clears that.
+func TestGPUReadLateThenStalledThenBack(t *testing.T) {
+	defer func(w, s time.Duration) { gpuReadWithin, gpuStallAfter = w, s }(gpuReadWithin, gpuStallAfter)
+	gpuReadWithin, gpuStallAfter = 20*time.Millisecond, 10*time.Second
+	release := make(chan struct{})
+	n := 0.0 // only the read goroutine writes it, one read at a time
+	d := &gpuDev{name: "test", card: "card0"}
+	d.readFn = func(x *GPU, _ time.Time) {
+		n++
+		if n == 2 {
+			<-release
+		}
+		x.Usage = n * 10
+	}
+	t0 := time.Now()
+	sample := func(at time.Duration) GPU {
+		x := GPU{Name: "test"}
+		d.readWithin(&x, t0.Add(at))
+		return x
+	}
+	if x := sample(0); x.Usage != 10 || x.Stalled {
+		t.Fatalf("first: %+v", x)
+	}
+	if x := sample(time.Second); x.Usage != 10 || x.Stalled { // late: the last readings
+		t.Fatalf("late: %+v", x)
+	}
+	if x := sample(5 * time.Second); x.Usage != 10 || x.Stalled {
+		t.Fatalf("still late: %+v", x)
+	}
+	if x := sample(12 * time.Second); !x.Stalled {
+		t.Fatalf("out past gpuStallAfter, not stalled: %+v", x)
+	}
+	if x := sample(12*time.Second + 1); !x.Stalled {
+		t.Fatal("stall did not hold")
+	}
+	close(release)
+	time.Sleep(10 * time.Millisecond)
+	if x := sample(13 * time.Second); x.Usage != 20 || x.Stalled {
+		t.Fatalf("back: %+v", x)
+	}
+	if x := sample(14 * time.Second); x.Usage != 30 || x.Stalled { // 30: no extra read ran while one was out
+		t.Fatalf("next: %+v", x)
+	}
+}

internal/metrics/gpu_test.go

@@ -61,7 +61,7 @@ func TestGPUClientsShare(t *testing.T) {
 	c.gpu.clients = map[string]clientPrev{}
 	t0 := time.Now()
 	c.gpu.last = []drmFD{{PID: 7, Comm: "ffmpeg", Info: fdinfoA}}
-	c.gpu.scanAt = t0 // no walk: the fake list stands
+	c.gpu.scan = func() ([]drmFD, bool) { return c.gpu.last, true } // no walk: the fake list stands
 	first := c.gpus(t0, 40)
 	if len(first) != 1 || len(first[0].Clients) != 1 || first[0].Clients[0].Usage != 0 {
 		t.Fatalf("first pass: %+v", first)
@@ -73,7 +73,6 @@ func TestGPUClientsShare(t *testing.T) {
 	later = replace(later, "drm-engine-render:	1000000000 ns", "drm-engine-render:	1500000000 ns")
 	later = replace(later, "drm-engine-video:	2000000000 ns", "drm-engine-video:	4000000000 ns")
 	c.gpu.last = []drmFD{{PID: 7, Comm: "ffmpeg", Info: later}}
-	c.gpu.scanAt = t0.Add(time.Second)
 	got := c.gpus(t0.Add(time.Second), 40)
 	cl := got[0].Clients[0]
 	if cl.Engines["render"] != 50 || cl.Engines["video"] != 100 || cl.Usage != 100 {
@@ -183,6 +182,20 @@ func TestParseNvidiaQuery(t *testing.T) {
 	if g.Temp != 57 || g.TempSrc != "gpu" || g.PowerW != 23.4 || g.PowerCapW != 0 {
 		t.Errorf("temp/power (cap must stay 0 for [N/A]): %+v", g)
 	}
+	if g.FanPct != nil {
+		t.Errorf("no fan field must leave the fan unset: %+v", g)
+	}
+	var fan, nofan GPU
+	parseNvidiaQuery(&fan, "99, 0, 0, 20000, 24576, 1695, 2100, 83, 403.99, 420.00, 34\n")
+	parseNvidiaQuery(&nofan, "5, 0, 0, 500, 2048, 300, 1097, 45, 8.0, [N/A], [N/A]\n")
+	if fan.FanPct == nil || *fan.FanPct != 34 || nofan.FanPct != nil {
+		t.Errorf("fan: %v, [N/A]: %v", fan.FanPct, nofan.FanPct)
+	}
+	var stopped GPU
+	parseNvidiaQuery(&stopped, "0, 0, 0, 500, 24576, 210, 2100, 35, 20.0, 420.00, 0\n")
+	if stopped.FanPct == nil || *stopped.FanPct != 0 {
+		t.Errorf("a stopped fan must read 0, not nothing: %v", stopped.FanPct)
+	}
 	var short GPU
 	parseNvidiaQuery(&short, "garbage")
 	if short.UsageSrc != "" {

internal/metrics/metrics.go

@@ -60,6 +60,7 @@ type Temp struct {
 	C     float64 `json:"c"`
 	Max   float64 `json:"max,omitempty"`
 	Crit  float64 `json:"crit,omitempty"`
+	Pkg   int     `json:"pkg,omitempty"` // coretemp: the CPU package the chip belongs to — a two-socket board has two chips, both with a "Core 0"
 }
 
 type Fan struct {
@@ -101,6 +102,9 @@ type FS struct {
 	// said honestly: a ZFS pool or an md array spreads a filesystem over several drives, and
 	// a remote mount is on none of them.
 	Disk string `json:"disk,omitempty"`
+	// Source is the device the mount is of ("/dev/nvme0n1p1"). A bind mount has the same
+	// source as the filesystem it shows, so a sum over a drive counts each source once.
+	Source string `json:"source,omitempty"`
 }
 
 type UPS struct {
@@ -140,6 +144,7 @@ type Sample struct {
 	CPU          float64    `json:"cpu"`
 	Cores        []float64  `json:"cores"`
 	CPUPower     float64    `json:"cpu_power_w,omitempty"`      // the CPU packages' own energy counters; 0 = no reading
+	CPUPowerPkgs []float64  `json:"cpu_power_pkgs_w,omitempty"` // each package's share, by socket; only with more than one
 	Load         [3]float64 `json:"load"`
 	Mem          Mem        `json:"mem"`
 	Swap         Mem        `json:"swap"`
@@ -151,6 +156,7 @@ type Sample struct {
 	UPS          *UPS       `json:"ups"`
 	GPUs         []GPU      `json:"gpus"`
 	ARC          *ARC       `json:"arc"`        // the ZFS read cache; nil without the module
+	ZFSWrite     *ZFSWrites `json:"zfs_writes"` // the ZFS write path; nil without the module
 	Uptime       float64    `json:"uptime"`
 
 	fsT, upsT int64 // when the slow sections were really read; sensors report these, not T
@@ -182,6 +188,7 @@ type Collector struct {
 	gpu       gpuState
 	power     cpuPowerState
 	prevARC   arcPrev
+	prevZW    zfsWritePrev
 
 	subMu sync.Mutex
 	subs  map[chan struct{}]struct{} // told after every sample; see Subscribe
@@ -347,7 +354,7 @@ func (c *Collector) sample() {
 
 	s := Sample{T: now.UnixMilli(), Interval: c.Interval().Milliseconds(), Temps: []Temp{}, Fans: []Fan{}, Net: []Net{}, Disks: []Disk{}}
 	s.CPU, s.Cores = c.cpu()
-	s.CPUPower = c.cpuPower(now)
+	s.CPUPower, s.CPUPowerPkgs = c.cpuPower(now)
 	s.Load = loadavg()
 	s.Mem, s.Swap = meminfo()
 	s.Uptime = uptime()
@@ -368,6 +375,7 @@ func (c *Collector) sample() {
 	s.UPS, s.upsT = c.lastUPS, c.upsAt.UnixMilli()
 	s.GPUs = c.gpus(now, cpuTemp(s.Temps))
 	s.ARC = c.arc(now)
+	s.ZFSWrite = c.zfsWrites(now, s.ARC, s.Disks)
 
 	if first {
 		return // deltas are meaningless on the priming pass
@@ -541,6 +549,19 @@ func hwmonDir(d string) (chip string, temps []Temp, fans []Fan) {
 			fans = append(fans, Fan{Chip: chip, Label: label, RPM: int(raw)})
 		}
 	}
+	// coretemp is one chip per CPU package, and only its "Package id N" says which
+	if chip == "coretemp" {
+		for _, t := range temps {
+			if n, ok := strings.CutPrefix(t.Label, "Package id "); ok {
+				if pkg, err := strconv.Atoi(n); err == nil {
+					for i := range temps {
+						temps[i].Pkg = pkg
+					}
+				}
+				break
+			}
+		}
+	}
 	return
 }
 
@@ -556,7 +577,10 @@ func hwmon() ([]Temp, []Fan) {
 		if temps[i].Chip != temps[j].Chip {
 			return temps[i].Chip < temps[j].Chip
 		}
+		if temps[i].Label != temps[j].Label {
 			return temps[i].Label < temps[j].Label
+		}
+		return temps[i].Pkg < temps[j].Pkg
 	})
 	return temps, fans
 }
@@ -842,7 +866,7 @@ func filesystems() []FS {
 				return
 			}
 			bs := uint64(st.Bsize)
-			done <- FS{Mount: mount, Type: fs[2], Size: st.Blocks * bs, Used: (st.Blocks - st.Bfree) * bs, Remote: remoteFS[fs[2]], Disk: diskOf(fs[0])}
+			done <- FS{Mount: mount, Type: fs[2], Size: st.Blocks * bs, Used: (st.Blocks - st.Bfree) * bs, Remote: remoteFS[fs[2]], Disk: diskOf(fs[0]), Source: fs[0]}
 		}()
 		select {
 		case r := <-done:
@@ -1109,6 +1133,38 @@ func coreIDs(n int) []int {
 	return coreIDsVal[:n]
 }
 
+// corePkgs maps logical CPUs to the CPU package (socket) each is on, and is nil on a
+// one-socket machine: there is nothing to tell apart.
+var (
+	corePkgsOnce sync.Once
+	corePkgsVal  []int
+)
+
+func corePkgs(n int) []int {
+	corePkgsOnce.Do(func() {
+		seen := map[int]bool{}
+		var out []int
+		for i := 0; ; i++ {
+			b, err := os.ReadFile("/sys/devices/system/cpu/cpu" + strconv.Itoa(i) + "/topology/physical_package_id")
+			if err != nil {
+				break
+			}
+			id, err := strconv.Atoi(strings.TrimSpace(string(b)))
+			if err != nil {
+				break
+			}
+			out, seen[id] = append(out, id), true
+		}
+		if len(seen) > 1 {
+			corePkgsVal = out
+		}
+	})
+	if len(corePkgsVal) < n {
+		return nil
+	}
+	return corePkgsVal[:n]
+}
+
 // ProcTable is the Processes tab payload.
 type ProcTable struct {
 	T             int64      `json:"t"`
@@ -1119,8 +1175,10 @@ type ProcTable struct {
 	CPUMHz        float64    `json:"cpu_mhz,omitempty"`
 	CPUMaxMHz     float64    `json:"cpu_max_mhz,omitempty"`
 	CPUPowerW     float64    `json:"cpu_power_w,omitempty"`
+	CPUPowerPkgsW []float64  `json:"cpu_power_pkgs_w,omitempty"`
 	CPUTemp       float64    `json:"cpu_temp,omitempty"`
 	CoreIDs       []int      `json:"core_ids,omitempty"`  // for each logical CPU, the physical core it is a thread of (what the temperature sensors are named by)
+	CorePkgs      []int      `json:"core_pkgs,omitempty"` // for each logical CPU, the socket it is on; only on a machine with more than one
 	Load          [3]float64 `json:"load"`
 	Mem           Mem        `json:"mem"`
 	Swap          Mem        `json:"swap"`
@@ -1179,18 +1237,24 @@ func cpuMHz() float64 {
 
 // cpuTemp picks the sensor that best represents the CPU package.
 func cpuTemp(temps []Temp) float64 {
-	best, max := 0.0, 0.0
+	// the hottest package: a two-socket board has two, and the hotter one is the one to watch
+	pkg, max := 0.0, 0.0
 	for _, t := range temps {
 		l := strings.ToLower(t.Label)
 		if t.Chip == "coretemp" && strings.HasPrefix(l, "package") || t.Chip == "k10temp" && (l == "tctl" || l == "tdie") ||
 			t.Chip == "thermal" && strings.Contains(l, "cpu") || t.Chip == "cpu_thermal" || t.Chip == "soc_thermal" {
-			return t.C
+			if t.C > pkg {
+				pkg = t.C
+			}
 		}
 		if (t.Chip == "coretemp" || t.Chip == "k10temp") && t.C > max {
-			max, best = t.C, t.C
+			max = t.C
 		}
 	}
-	return best
+	if pkg > 0 {
+		return pkg
+	}
+	return max
 }
 
 var bootTime = func() int64 {
@@ -1318,8 +1382,9 @@ func (c *Collector) Processes() ProcTable {
 	t.Mem, t.Swap = meminfo()
 	if n := len(c.ring); n > 0 {
 		last := c.ring[n-1]
-		t.CPU, t.Cores, t.Net, t.CPUTemp, t.CPUPowerW = last.CPU, last.Cores, last.Net, cpuTemp(last.Temps), last.CPUPower
+		t.CPU, t.Cores, t.Net, t.CPUTemp, t.CPUPowerW, t.CPUPowerPkgsW = last.CPU, last.Cores, last.Net, cpuTemp(last.Temps), last.CPUPower, last.CPUPowerPkgs
 		t.CoreIDs = coreIDs(len(last.Cores))
+		t.CorePkgs = corePkgs(len(last.Cores))
 		from := n - 90
 		if from < 0 {
 			from = 0

internal/metrics/sensors.go

@@ -65,6 +65,10 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
 	if s.CPUPower > 0 {
 		f.add("cpu.power", "CPU power", "power", "W", s.CPUPower, t, none)
 	}
+	for i, w := range s.CPUPowerPkgs {
+		n := strconv.Itoa(i)
+		f.add("cpu.package."+n+".power", "CPU "+strconv.Itoa(i+1)+" power", "power", "W", w, t, none, "package", n)
+	}
 	for i, w := range []string{"1", "5", "15"} {
 		f.add("load."+w, "Load "+w+" min", "load", "", s.Load[i], t, none)
 	}
@@ -92,7 +96,11 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
 		if x.Crit > 0 {
 			b.crit = sensors.F(x.Crit)
 		}
-		f.add(sensors.ID("temp", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "temperature", "°C", x.C, t, b, "chip", x.Chip, "label", x.Label)
+		labels := []string{"chip", x.Chip, "label", x.Label}
+		if x.Chip == "coretemp" { // a two-socket board has a coretemp chip per CPU, both with a "Core 0"
+			labels = append(labels, "package", strconv.Itoa(x.Pkg))
+		}
+		f.add(sensors.ID("temp", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "temperature", "°C", x.C, t, b, labels...)
 	}
 	for _, x := range s.Fans {
 		f.add(sensors.ID("fan", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "fan", "rpm", float64(x.RPM), t, none, "chip", x.Chip, "label", x.Label)
@@ -129,13 +137,22 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
 		if x.Disk != "" {
 			lbl = append(lbl, "disk", x.Disk)
 		}
+		if x.Source != "" {
+			lbl = append(lbl, "source", x.Source)
+		}
 		f.add(p+".size", x.Mount+" size", "bytes", "B", float64(x.Size), s.fsT, none, lbl...)
 		f.add(p+".used", x.Mount+" used", "bytes", "B", float64(x.Used), s.fsT, upTo(float64(x.Size)), lbl...)
 		f.add(p+".used_pct", x.Mount+" used", "usage", "%", ratio(x.Used, x.Size), s.fsT, pct, lbl...)
 	}
 
+	gpuSeen := map[string]int{}
 	for _, g := range s.GPUs {
+		// two identical cards are gpu.<name> and gpu.<name>_2, every sensor of the second under
+		// its own stem (not add's suffix on the end of each id, which the page can't find)
 		p := "gpu." + sensors.Seg(g.Name)
+		if gpuSeen[p]++; gpuSeen[p] > 1 {
+			p = fmt.Sprintf("%s_%d", p, gpuSeen[p])
+		}
 		lbl := []string{"name", g.Name, "driver", g.Driver}
 		if g.Bus != "" {
 			lbl = append(lbl, "bus", g.Bus)
@@ -164,6 +181,9 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
 		if g.FanRPM > 0 {
 			f.add(p+".fan", g.Name+" fan", "fan", "rpm", float64(g.FanRPM), t, none, lbl...)
 		}
+		if g.FanPct != nil {
+			f.add(p+".fan_pct", g.Name+" fan", "fan", "%", *g.FanPct, t, pct, lbl...)
+		}
 		if g.MaxMHz > 0 {
 			f.add(p+".freq", g.Name+" clock", "frequency", "MHz", g.MHz, t, upTo(g.MaxMHz), lbl...)
 		}
@@ -203,6 +223,32 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
 		}
 	}
 
+	if w := s.ZFSWrite; w != nil {
+		f.add("zfs.write.dirty", "ZFS write buffer", "bytes", "B", float64(w.Dirty), t, upTo(float64(w.Limit)))
+		f.add("zfs.write.limit", "ZFS write buffer limit", "bytes", "B", float64(w.Limit), t, none)
+		f.add("zfs.write.delayed", "ZFS writes throttled", "rate", "/s", w.DelayedPerSec, t, none)
+		for _, p := range w.Pools {
+			pp := "zfs." + sensors.Seg(p.Pool) + "."
+			f.add(pp+"write_bps", p.Pool+" writes to disk", "rate", "B/s", p.WriteBps, t, none, "pool", p.Pool)
+			f.add(pp+"logical_bps", p.Pool+" writes by programs", "rate", "B/s", p.LogicalBps, t, none, "pool", p.Pool)
+			if l := p.Landing; l != nil {
+				lp := pp + "landing."
+				lbl := []string{"pool", p.Pool, "branch", l.Branch}
+				f.add(lp+"pending_bytes", p.Pool+" landing: waiting", "bytes", "B", float64(l.PendingBytes), t, upTo(float64(l.Size)), lbl...)
+				f.add(lp+"pending_files", p.Pool+" landing: files waiting", "count", "", float64(l.PendingFiles), t, none, lbl...)
+				if l.Size > 0 {
+					f.add(lp+"used_pct", p.Pool+" landing: used", "usage", "%", round1(float64(l.Used)/float64(l.Size)*100), t, pct, lbl...)
+				}
+				f.add(lp+"in_bps", p.Pool+" landing: arriving", "rate", "B/s", l.InBps, t, none, lbl...)
+				f.add(lp+"drain_bps", p.Pool+" landing: moving to the pool", "rate", "B/s", l.DrainBps, t, none, lbl...)
+				if m := l.Mover; m != nil {
+					f.add(lp+"mover", p.Pool+" landing: mover", "state", "", m.State, t, none, lbl...)
+					f.add(lp+"queue", p.Pool+" landing: mover queue", "count", "", float64(m.Queue), t, none, lbl...)
+				}
+			}
+		}
+	}
+
 	if u := s.UPS; u != nil {
 		ut := s.upsT
 		f.add("ups.battery", "UPS battery", "usage", "%", u.BatteryPct, ut, pct, "model", u.Model)

internal/metrics/zfswrite.go

@@ -0,0 +1,456 @@
+package metrics
+
+// The ZFS write path, for the ZFS tab: what is waiting in RAM to be written, how fast each
+// pool is writing to its disks, and — where a pool has one — the landing tier in front of it.
+//
+// A landing tier is a mergerfs pool whose first branches are fast non-ZFS storage and whose
+// last branch is a ZFS dataset: new files land on the fast branch and a mover drains them onto
+// the pool. It is detected from mergerfs's own branch list (the user.mergerfs.branches xattr on
+// the pool's .mergerfs control file), so it shows wherever one exists and nowhere else. A mover
+// that writes a status file into /run/landing-mover/ (hamlet's scripts/landing-mover.py does)
+// adds what it is doing: its queue, the file in flight, totals and the last error.
+//
+// Sources, all cheap enough for every sample:
+//   - arcstats anon_size: dirty data in RAM, for all pools together (ZFS has no live per-pool
+//     figure; txgs only records ndirty once a txg has synced)
+//   - dmu_tx: dmu_tx_dirty_delay / dmu_tx_dirty_over_max — writers slowed by the write throttle,
+//     or stopped at the hard limit
+//   - <pool>/txgs: nwritten per synced txg — bytes the pool actually wrote to its vdevs
+//   - <pool>/objset-*: nwritten per dataset — bytes programs wrote, before compression
+// The landing branch's file walk is slower and runs every landingEvery.
+
+import (
+	"encoding/json"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"syscall"
+	"time"
+)
+
+const (
+	zfsKstatDir    = "/proc/spl/kstat/zfs/"
+	zfsParamDir    = "/sys/module/zfs/parameters/"
+	moverStateDir  = "/run/landing-mover"
+	landingEvery   = 10 * time.Second
+	landingWalkMax = 200000 // entries; past this the counts are marked partial
+)
+
+// ZFSWrites is the write path as one sample sees it. Nil without the ZFS module.
+type ZFSWrites struct {
+	Dirty         uint64       `json:"dirty"`            // bytes in RAM written but not yet on disk, all pools
+	DirtyMax      uint64       `json:"dirty_max"`        // zfs_dirty_data_max
+	DelayMinPct   int          `json:"delay_min_pct"`    // zfs_delay_min_dirty_percent: the throttle starts here
+	Limit         uint64       `json:"limit"`            // what the buffer can really hold now: the smaller of dirty_max and half the ARC target
+	DelayedPerSec float64      `json:"delayed_per_sec"`  // writes the throttle slowed, per second over the interval
+	OverMaxPerSec float64      `json:"over_max_per_sec"` // writes stopped dead at the limit
+	TxgTimeoutSec int          `json:"txg_timeout_sec"`  // zfs_txg_timeout
+	Pools         []PoolWrites `json:"pools"`
+}
+
+// PoolWrites is one pool.
+type PoolWrites struct {
+	Pool       string   `json:"pool"`
+	WriteBps   float64  `json:"write_bps"`   // to its vdevs, after compression, parity included
+	LogicalBps float64  `json:"logical_bps"` // into its datasets, as programs wrote it
+	Landing    *Landing `json:"landing,omitempty"`
+}
+
+// Landing is a landing tier in front of a pool.
+type Landing struct {
+	Mount          string  `json:"mount"`  // the mergerfs pool, e.g. /srv/tank
+	Branch         string  `json:"branch"` // the landing branch, e.g. /srv/landing/tank
+	PoolBranch     string  `json:"pool_branch"`
+	Device         string  `json:"device,omitempty"` // block device under the branch, as in Sample.Disks
+	FSType         string  `json:"fstype,omitempty"`
+	Size           uint64  `json:"size"`
+	Used           uint64  `json:"used"`
+	PendingFiles   int     `json:"pending_files"`
+	PendingBytes   uint64  `json:"pending_bytes"`
+	PendingPartial bool    `json:"pending_partial,omitempty"` // the walk stopped at landingWalkMax
+	OldestMs       int64   `json:"oldest_ms,omitempty"`       // mtime of the oldest waiting file
+	CountedMs      int64   `json:"counted_ms"`                // when the walk ran
+	InBps          float64 `json:"in_bps"`                    // the landing device's writes
+	DrainBps       float64 `json:"drain_bps"`                 // the mover's progress onto the pool
+	Mover          *Mover  `json:"mover,omitempty"`
+}
+
+// Mover is what a landing mover says about itself in its status file.
+type Mover struct {
+	State        string  `json:"state"` // idle | moving | holding (a mount it needs is missing)
+	Src          string  `json:"src"`
+	Dst          string  `json:"dst"`
+	PID          int     `json:"pid"`
+	StartedMs    int64   `json:"started_ms"`
+	UpdatedMs    int64   `json:"updated_ms"`
+	Stale        bool    `json:"stale"` // not updated for a while: stopped or hung
+	Queue        int     `json:"queue"`
+	Settle       float64 `json:"settle"`
+	BWLimit      string  `json:"bwlimit"`
+	BatchFiles   int     `json:"batch_files"`
+	BatchBytes   uint64  `json:"batch_bytes"`
+	BatchStartMs int64   `json:"batch_start_ms"`
+	BatchFirst   string  `json:"batch_first"`
+	Current      string  `json:"current"`      // the file being copied, when it can tell
+	CurrentSize  uint64  `json:"current_size"` // its full size
+	Partial      uint64  `json:"partial"`      // bytes of it already on the pool
+	MovedFiles   uint64  `json:"moved_files"`  // since the mover started
+	MovedBytes   uint64  `json:"moved_bytes"`
+	LastMove     string  `json:"last_move"`
+	LastMoveMs   int64   `json:"last_move_ms"`
+	LastError    string  `json:"last_error"`
+	LastErrorMs  int64   `json:"last_error_ms"`
+}
+
+type zfsWritePrev struct {
+	at        time.Time
+	delay     uint64
+	overMax   uint64
+	txg       map[string]uint64    // pool -> highest synced txg counted
+	txgAt     map[string]time.Time // pool -> when that txg was first seen
+	txgRate   map[string]float64   // pool -> bytes/s over the last gap between syncs
+	logical   map[string]uint64    // pool -> sum of objset nwritten
+	drained   map[string]uint64    // landing branch -> moved + partial bytes, at the mover's last update
+	drainedAt map[string]int64     // landing branch -> that update's time, unix ms
+	drainRate map[string]float64   // landing branch -> bytes/s between its last two updates
+	landing   map[string]Landing   // branch -> last walk
+	landingAt time.Time
+}
+
+func readParamU(name string) uint64 {
+	v, _ := strconv.ParseUint(strings.TrimSpace(readStr(zfsParamDir+name)), 10, 64)
+	return v
+}
+
+// poolsWithKstats lists the imported pools: the directories under the kstat tree.
+func poolsWithKstats() []string {
+	ents, err := os.ReadDir(zfsKstatDir)
+	if err != nil {
+		return nil
+	}
+	var out []string
+	for _, e := range ents {
+		if e.IsDir() {
+			if _, err := os.Stat(zfsKstatDir + e.Name() + "/txgs"); err == nil {
+				out = append(out, e.Name())
+			}
+		}
+	}
+	sort.Strings(out)
+	return out
+}
+
+// parseTxgs reads a pool's txgs kstat and returns, for every committed txg above `after`, the
+// bytes it wrote, plus the highest committed txg seen.
+func parseTxgs(text string, after uint64) (written, top uint64) {
+	top = after
+	for _, line := range strings.Split(text, "\n") {
+		f := strings.Fields(line)
+		// txg birth state ndirty nread nwritten reads writes otime qtime wtime stime
+		if len(f) < 12 || f[2] != "C" {
+			continue
+		}
+		txg, err := strconv.ParseUint(f[0], 10, 64)
+		if err != nil {
+			continue
+		}
+		if txg > top {
+			top = txg
+		}
+		if txg > after {
+			n, _ := strconv.ParseUint(f[5], 10, 64)
+			written += n
+		}
+	}
+	return written, top
+}
+
+// objsetWritten sums nwritten over a pool's datasets.
+func objsetWritten(pool string) uint64 {
+	m, _ := filepath.Glob(zfsKstatDir + pool + "/objset-*")
+	var t uint64
+	for _, p := range m {
+		for _, line := range strings.Split(readStr(p), "\n") {
+			f := strings.Fields(line)
+			if len(f) == 3 && f[0] == "nwritten" {
+				v, _ := strconv.ParseUint(f[2], 10, 64)
+				t += v
+			}
+		}
+	}
+	return t
+}
+
+// mountEntry is one line of /proc/mounts.
+type mountEntry struct{ src, point, fstype string }
+
+func readMounts() []mountEntry {
+	var out []mountEntry
+	for _, line := range strings.Split(readStr("/proc/mounts"), "\n") {
+		f := strings.Fields(line)
+		if len(f) >= 3 {
+			out = append(out, mountEntry{src: unescapeMount(f[0]), point: unescapeMount(f[1]), fstype: f[2]})
+		}
+	}
+	return out
+}
+
+// unescapeMount undoes /proc/mounts' octal escapes (\040 for a space, and so on).
+func unescapeMount(s string) string {
+	if !strings.Contains(s, `\`) {
+		return s
+	}
+	var b strings.Builder
+	for i := 0; i < len(s); i++ {
+		if s[i] == '\\' && i+3 < len(s) {
+			if v, err := strconv.ParseUint(s[i+1:i+4], 8, 8); err == nil {
+				b.WriteByte(byte(v))
+				i += 3
+				continue
+			}
+		}
+		b.WriteByte(s[i])
+	}
+	return b.String()
+}
+
+// mountFor is the mount a path lives on: the longest mount point that is a prefix of it.
+func mountFor(ms []mountEntry, path string) (mountEntry, bool) {
+	var best mountEntry
+	found := false
+	for _, m := range ms {
+		if m.point == path || m.point == "/" || strings.HasPrefix(path, strings.TrimSuffix(m.point, "/")+"/") {
+			if !found || len(m.point) > len(best.point) {
+				best, found = m, true
+			}
+		}
+	}
+	return best, found
+}
+
+// parseBranches reads mergerfs's branch list: "/srv/landing/tank=RW:/tank=RW,50G".
+func parseBranches(v string) (paths []string, modes []string) {
+	for _, b := range strings.Split(strings.TrimSpace(v), ":") {
+		if b == "" {
+			continue
+		}
+		p, mode, _ := strings.Cut(b, "=")
+		mode, _, _ = strings.Cut(mode, ",")
+		if mode == "" {
+			mode = "RW"
+		}
+		paths, modes = append(paths, p), append(modes, mode)
+	}
+	return paths, modes
+}
+
+func getxattr(path, name string) string {
+	buf := make([]byte, 4096)
+	n, err := syscall.Getxattr(path, name, buf)
+	if err != nil || n <= 0 {
+		return ""
+	}
+	return string(buf[:n])
+}
+
+// findLandings maps pool -> the landing tiers in front of it.
+func findLandings(ms []mountEntry) map[string][]Landing {
+	out := map[string][]Landing{}
+	for _, m := range ms {
+		if m.fstype != "fuse.mergerfs" {
+			continue
+		}
+		paths, modes := parseBranches(getxattr(filepath.Join(m.point, ".mergerfs"), "user.mergerfs.branches"))
+		// the pool branch is the first ZFS one; landing branches are writable non-ZFS ones before it
+		for i, p := range paths {
+			bm, ok := mountFor(ms, p)
+			if !ok || bm.fstype != "zfs" {
+				continue
+			}
+			pool, _, _ := strings.Cut(bm.src, "/")
+			for j := 0; j < i; j++ {
+				lm, ok := mountFor(ms, paths[j])
+				if !ok || lm.fstype == "zfs" || modes[j] == "RO" {
+					continue
+				}
+				out[pool] = append(out[pool], Landing{Mount: m.point, Branch: paths[j], PoolBranch: p, FSType: lm.fstype, Device: blockDisk(lm.src)})
+			}
+			break
+		}
+	}
+	return out
+}
+
+// blockDisk turns a mount source into the name Sample.Disks uses: /dev/nvme0n1p1 -> nvme0,
+// /dev/sdb2 -> sdb. Empty for anything that is not a plain block device.
+func blockDisk(src string) string {
+	dev, err := filepath.EvalSymlinks(src)
+	if err != nil || !strings.HasPrefix(dev, "/dev/") {
+		return ""
+	}
+	name := filepath.Base(dev)
+	// a partition's sysfs entry sits inside its disk's
+	if p, err := filepath.EvalSymlinks("/sys/class/block/" + name); err == nil {
+		if _, err := os.Stat(p + "/partition"); err == nil {
+			name = filepath.Base(filepath.Dir(p))
+		}
+	}
+	if m := nsRe.FindStringSubmatch(name); m != nil {
+		return m[1] // nvme0n1 -> nvme0, as foldNVMe names it
+	}
+	return name
+}
+
+// walkLanding counts what is waiting on a landing branch.
+func walkLanding(l *Landing, now time.Time) {
+	var st syscall.Statfs_t
+	if syscall.Statfs(l.Branch, &st) == nil {
+		l.Size = st.Blocks * uint64(st.Bsize)
+		l.Used = (st.Blocks - st.Bfree) * uint64(st.Bsize)
+	}
+	seen := 0
+	l.PendingFiles, l.PendingBytes, l.OldestMs, l.PendingPartial = 0, 0, 0, false
+	_ = filepath.WalkDir(l.Branch, func(p string, d fs.DirEntry, err error) error {
+		if err != nil {
+			return nil
+		}
+		if seen++; seen > landingWalkMax {
+			l.PendingPartial = true
+			return filepath.SkipAll
+		}
+		if !d.Type().IsRegular() {
+			return nil
+		}
+		if fi, err := d.Info(); err == nil {
+			l.PendingFiles++
+			l.PendingBytes += uint64(fi.Size())
+			if ms := fi.ModTime().UnixMilli(); l.OldestMs == 0 || ms < l.OldestMs {
+				l.OldestMs = ms
+			}
+		}
+		return nil
+	})
+	l.CountedMs = now.UnixMilli()
+}
+
+// readMovers loads every mover status file, keyed by the branch it drains.
+func readMovers(now time.Time) map[string]*Mover {
+	out := map[string]*Mover{}
+	m, _ := filepath.Glob(moverStateDir + "/*.json")
+	for _, p := range m {
+		b, err := os.ReadFile(p)
+		if err != nil {
+			continue
+		}
+		var mv Mover
+		if json.Unmarshal(b, &mv) != nil || mv.Src == "" {
+			continue
+		}
+		mv.Stale = now.UnixMilli()-mv.UpdatedMs > 30_000
+		if mv.Stale {
+			mv.State = "stopped"
+		}
+		out[filepath.Clean(mv.Src)] = &mv
+	}
+	return out
+}
+
+// zfsWrites samples the write path; nil when this machine has no ZFS module loaded.
+func (c *Collector) zfsWrites(now time.Time, arc *ARC, disks []Disk) *ZFSWrites {
+	if arc == nil {
+		return nil
+	}
+	p := &c.prevZW
+	dt := 0.0
+	if !p.at.IsZero() {
+		dt = now.Sub(p.at).Seconds()
+	}
+	w := &ZFSWrites{Dirty: arc.Anon, DirtyMax: readParamU("zfs_dirty_data_max"),
+		DelayMinPct: int(readParamU("zfs_delay_min_dirty_percent")), TxgTimeoutSec: int(readParamU("zfs_txg_timeout")), Pools: []PoolWrites{}}
+	w.Limit = w.DirtyMax
+	if half := arc.Target / 2; half > 0 && half < w.Limit {
+		w.Limit = half
+	}
+	tx := readKstat(zfsKstatDir + "dmu_tx")
+	if dt > 0 && tx["dmu_tx_dirty_delay"] >= p.delay && tx["dmu_tx_dirty_over_max"] >= p.overMax {
+		w.DelayedPerSec = round1(float64(tx["dmu_tx_dirty_delay"]-p.delay) / dt)
+		w.OverMaxPerSec = round1(float64(tx["dmu_tx_dirty_over_max"]-p.overMax) / dt)
+	}
+	p.delay, p.overMax = tx["dmu_tx_dirty_delay"], tx["dmu_tx_dirty_over_max"]
+	if p.txg == nil {
+		p.txg, p.logical, p.drained, p.landing = map[string]uint64{}, map[string]uint64{}, map[string]uint64{}, map[string]Landing{}
+		p.txgAt, p.txgRate, p.drainedAt, p.drainRate = map[string]time.Time{}, map[string]float64{}, map[string]int64{}, map[string]float64{}
+	}
+
+	ms := readMounts()
+	landings := findLandings(ms)
+	walk := now.Sub(p.landingAt) >= landingEvery
+	if walk {
+		p.landingAt = now
+	}
+	movers := readMovers(now)
+	diskW := map[string]float64{}
+	for _, d := range disks {
+		diskW[d.Name] = d.WriteBps
+	}
+
+	for _, pool := range poolsWithKstats() {
+		pw := PoolWrites{Pool: pool}
+		// A txg records what it wrote only when it has synced — every few seconds, not every
+		// sample — so the rate is taken over the gap between syncs and held until the next one.
+		prevTop, had := p.txg[pool]
+		written, top := parseTxgs(readStr(zfsKstatDir+pool+"/txgs"), prevTop)
+		switch {
+		case !had:
+			p.txgAt[pool] = now
+		case top > prevTop:
+			if gap := now.Sub(p.txgAt[pool]).Seconds(); gap > 0 {
+				p.txgRate[pool] = round1(float64(written) / gap)
+			}
+			p.txgAt[pool] = now
+		case now.Sub(p.txgAt[pool]) > time.Duration(2*max(w.TxgTimeoutSec, 5))*time.Second:
+			p.txgRate[pool] = 0 // no sync for two timeouts: nothing is being written
+		}
+		pw.WriteBps = p.txgRate[pool]
+		p.txg[pool] = top
+		lw := objsetWritten(pool)
+		if prev, ok := p.logical[pool]; ok && dt > 0 && lw >= prev {
+			pw.LogicalBps = round1(float64(lw-prev) / dt)
+		}
+		p.logical[pool] = lw
+
+		if ls := landings[pool]; len(ls) > 0 {
+			l := ls[0] // one landing tier per pool is what the view shows
+			if last, ok := p.landing[l.Branch]; ok && !walk {
+				l.Size, l.Used, l.PendingFiles, l.PendingBytes = last.Size, last.Used, last.PendingFiles, last.PendingBytes
+				l.OldestMs, l.PendingPartial, l.CountedMs = last.OldestMs, last.PendingPartial, last.CountedMs
+			} else {
+				walkLanding(&l, now)
+				p.landing[l.Branch] = l
+			}
+			l.InBps = diskW[l.Device]
+			if mv := movers[filepath.Clean(l.Branch)]; mv != nil {
+				l.Mover = mv
+				// the mover writes its status every couple of seconds: measure between its updates
+				done := mv.MovedBytes + mv.Partial
+				if at, ok := p.drainedAt[l.Branch]; !ok || mv.UpdatedMs != at {
+					if ok && mv.UpdatedMs > at && done >= p.drained[l.Branch] {
+						p.drainRate[l.Branch] = round1(float64(done-p.drained[l.Branch]) / (float64(mv.UpdatedMs-at) / 1000))
+					}
+					p.drained[l.Branch], p.drainedAt[l.Branch] = done, mv.UpdatedMs
+				}
+				if mv.State != "moving" {
+					p.drainRate[l.Branch] = 0
+				}
+				l.DrainBps = p.drainRate[l.Branch]
+			}
+			pw.Landing = &l
+		}
+		w.Pools = append(w.Pools, pw)
+	}
+	p.at = now
+	return w
+}

internal/metrics/zfswrite_test.go

@@ -0,0 +1,86 @@
+package metrics
+
+import "testing"
+
+// real rows from grid's /proc/spl/kstat/zfs/tank/txgs, 2026-10-25
+const txgsSample = `81 1 0x01 26 7072 20029594359 13953814329135
+txg      birth            state ndirty       nread        nwritten     reads    writes   otime        qtime        wtime        stime
+15812833 13936408791659   C     0            0            0            0        0        5119980242   2040         58385        309580
+15812834 13941528771901   C     0            0            0            0        0        5119948815   8471         65826        112612
+15812835 13946648720716   C     885760       0            2293760      0        367      5120100556   1704         56762        271264374
+15812836 13951768821272   S     4194304      0            0            0        0        5120000000   1000         50000        0
+15812837 13956888821272   O     0            0            0            0        0        0            0            0            0
+`
+
+func TestParseTxgs(t *testing.T) {
+	w, top := parseTxgs(txgsSample, 0)
+	if w != 2293760 || top != 15812835 {
+		t.Fatalf("from 0: written %d top %d", w, top)
+	}
+	// only txgs synced since the last look count, and an open or syncing txg never does
+	w, top = parseTxgs(txgsSample, 15812834)
+	if w != 2293760 || top != 15812835 {
+		t.Fatalf("after 15812834: written %d top %d", w, top)
+	}
+	w, top = parseTxgs(txgsSample, 15812835)
+	if w != 0 || top != 15812835 {
+		t.Fatalf("nothing new: written %d top %d", w, top)
+	}
+}
+
+func TestParseBranches(t *testing.T) {
+	p, m := parseBranches("/srv/landing/tank=RW:/tank=NC,50G:/mnt/old=RO\n")
+	if len(p) != 3 || p[0] != "/srv/landing/tank" || p[1] != "/tank" || p[2] != "/mnt/old" {
+		t.Fatalf("paths %q", p)
+	}
+	if m[0] != "RW" || m[1] != "NC" || m[2] != "RO" {
+		t.Fatalf("modes %q", m)
+	}
+	if p, _ := parseBranches("/a:/b"); len(p) != 2 {
+		t.Fatalf("bare paths %q", p)
+	}
+}
+
+func TestMountFor(t *testing.T) {
+	ms := []mountEntry{
+		{"/dev/sdj2", "/", "ext4"},
+		{"/dev/nvme0n1p1", "/srv/landing", "ext4"},
+		{"tank", "/tank", "zfs"},
+		{"tank/media", "/tank/media", "zfs"},
+		{"tank-landing", "/srv/tank", "fuse.mergerfs"},
+	}
+	cases := map[string]string{
+		"/srv/landing/tank": "/srv/landing", "/tank": "/tank", "/tank/media/x": "/tank/media",
+		"/tankfoo": "/", "/srv/landingx": "/",
+	}
+	for path, want := range cases {
+		if m, ok := mountFor(ms, path); !ok || m.point != want {
+			t.Errorf("%s: got %q want %q", path, m.point, want)
+		}
+	}
+}
+
+func TestUnescapeMount(t *testing.T) {
+	if got := unescapeMount(`/mnt/my\040disk`); got != "/mnt/my disk" {
+		t.Fatalf("got %q", got)
+	}
+}
+
+func TestFoldZFSWrites(t *testing.T) {
+	mk := func(dirty uint64, bps, in, drain float64) Sample {
+		return Sample{ZFSWrite: &ZFSWrites{Dirty: dirty, Limit: 100, Pools: []PoolWrites{{Pool: "tank", WriteBps: bps,
+			Landing: &Landing{Branch: "/srv/landing/tank", InBps: in, DrainBps: drain, Mover: &Mover{State: "moving"}}}}}}
+	}
+	b := []Sample{mk(10, 100, 400, 0), mk(30, 300, 0, 60)}
+	got := fold(b).ZFSWrite
+	if got.Dirty != 20 || got.Pools[0].WriteBps != 200 || got.Pools[0].Landing.InBps != 200 || got.Pools[0].Landing.DrainBps != 30 {
+		t.Fatalf("folded %+v / %+v", got, got.Pools[0].Landing)
+	}
+	// the fold must not write through to the newest sample it copied from
+	if b[1].ZFSWrite.Pools[0].Landing.InBps != 0 || b[1].ZFSWrite.Dirty != 30 {
+		t.Fatalf("fold changed its input")
+	}
+	if got.Pools[0].Landing.Mover.State != "moving" {
+		t.Fatalf("mover state lost")
+	}
+}

internal/nodes/nodes.go

@@ -383,13 +383,19 @@ func (m *Manager) PushAll(ctx context.Context, binPath, ourArch string, force bo
 // WebSocket origin check (meant for browsers) doesn't reject the hub.
 // Do makes one request to a managed machine as this hub and returns the whole response.
 func (m *Manager) Do(ctx context.Context, name, method, path string, body []byte, hdr http.Header) (int, []byte, error) {
+	code, b, _, err := m.DoHeaders(ctx, name, method, path, body, hdr)
+	return code, b, err
+}
+
+// DoHeaders is Do that also answers the response's headers.
+func (m *Manager) DoHeaders(ctx context.Context, name, method, path string, body []byte, hdr http.Header) (int, []byte, http.Header, error) {
 	n, ok := m.node(name)
 	if !ok {
-		return 0, nil, errors.New("no such machine")
+		return 0, nil, nil, errors.New("no such machine")
 	}
 	req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(n.URL, "/")+path, bytes.NewReader(body))
 	if err != nil {
-		return 0, nil, err
+		return 0, nil, nil, err
 	}
 	for k, v := range hdr {
 		req.Header[k] = v
@@ -397,11 +403,17 @@ func (m *Manager) Do(ctx context.Context, name, method, path string, body []byte
 	req.Header.Set("Authorization", "Bearer "+n.Token)
 	resp, err := m.client(n).Do(req)
 	if err != nil {
-		return 0, nil, errors.New(shortErr(err))
+		return 0, nil, nil, errors.New(shortErr(err))
 	}
 	defer resp.Body.Close()
 	b, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
-	return resp.StatusCode, b, err
+	return resp.StatusCode, b, resp.Header, err
+}
+
+// URL is a managed machine's address, "" for one not managed here.
+func (m *Manager) URL(name string) string {
+	n, _ := m.node(name)
+	return n.URL
 }
 
 // Names lists the managed machines.

internal/storage/landing.go

@@ -0,0 +1,564 @@
+package storage
+
+// Verifying a landing tier: did everything that landed really reach the pool, intact?
+//
+// A landing tier is fast storage in front of a pool (a mergerfs pool, new files on the fast
+// branch first) with a mover draining it. The mover keeps a ledger — one JSON line per file it
+// moved: the path under the pool, its size and mtime, and the SHA-256 of the fast copy read
+// just before the move (hamlet's scripts/landing-mover.py writes /var/lib/landing-mover/
+// ledger.jsonl and names it in its status file). A verify pass reads the pool's copy of each
+// entry back and compares.
+//
+// What a pass can say about each file:
+//   - match:      same size, same mtime, same SHA-256 — it arrived intact
+//   - mismatch:   same size and mtime but different contents — the one real failure
+//   - changed:    the pool's copy has a different size or mtime: edited since it moved, so the
+//                 old hash says nothing about it (counted, not compared)
+//   - gone:       not on the pool and not waiting on the landing branch either — deleted or
+//                 renamed since it moved. A pass cannot tell that from a loss, so it lists them
+//   - waiting:    back on the landing branch (written again since), the mover will take it
+//   - unreadable: could not be opened as this account
+//   - unhashed:   the mover recorded no hash for it (not a regular file, or changed mid-move)
+//
+// Scope "new" checks what has moved since the last pass, "all" re-reads the whole ledger. A
+// path moved more than once is checked against its latest entry only.
+
+import (
+	"bufio"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"syscall"
+	"time"
+)
+
+// ---- schedules ----
+
+// VerifyPolicy runs a verify pass on a schedule. The when-fields mean what they do on a
+// snapshot Policy; Scope is "new" or "all".
+type VerifyPolicy struct {
+	ID      string `json:"id"`
+	Name    string `json:"name"`
+	Every   string `json:"every"`
+	At      string `json:"at"`
+	Weekday int    `json:"weekday"`
+	Day     int    `json:"day"`
+	Cron    string `json:"cron,omitempty"`
+	Scope   string `json:"scope"`
+	Enabled bool   `json:"enabled"`
+}
+
+// when borrows the snapshot Policy's schedule logic, so both kinds of schedule are written,
+// validated and read back by exactly the same code.
+func (v *VerifyPolicy) when() *Policy {
+	return &Policy{Every: v.Every, At: v.At, Weekday: v.Weekday, Day: v.Day, Cron: v.Cron}
+}
+
+func (v *VerifyPolicy) CronExpr() (string, error)       { return v.when().CronExpr() }
+func (v *VerifyPolicy) NextRun(now time.Time) time.Time { return v.when().NextRun(now) }
+
+func (v *VerifyPolicy) Validate() error {
+	if v.Scope != "new" && v.Scope != "all" {
+		return errors.New("scope: new (what moved since the last pass) or all (the whole ledger)")
+	}
+	switch v.Every {
+	case "hourly", "daily", "weekly", "monthly":
+	case "custom":
+		if strings.TrimSpace(v.Cron) == "" {
+			return errors.New("a custom schedule needs a cron expression")
+		}
+	default:
+		return errors.New("every: hourly, daily, weekly, monthly or custom")
+	}
+	if _, err := v.CronExpr(); err != nil {
+		return err
+	}
+	if _, _, err := v.when().clock(); err != nil {
+		return err
+	}
+	if v.Every == "weekly" && (v.Weekday < 0 || v.Weekday > 6) {
+		return errors.New("weekday: 0 (Sunday) to 6")
+	}
+	if v.Every == "monthly" && (v.Day < 1 || v.Day > 28) {
+		return errors.New("day: 1 to 28, so every month has one")
+	}
+	if len(v.Name) > 64 {
+		return errors.New("name: 64 characters at most")
+	}
+	return nil
+}
+
+// VerifyCronFile holds the verify schedules; Ham Lab owns it and rewrites it whole.
+const VerifyCronFile = "/etc/cron.d/hamlab-landing"
+
+// VerifyCronText renders the verify schedules as cron lines, run as Ham Lab's own account.
+func VerifyCronText(ps []VerifyPolicy, user, exe, config string) (string, error) {
+	var b strings.Builder
+	b.WriteString("# Ham Lab landing-tier verify schedules — written by the ZFS tab, rewritten whole on every change.\n")
+	b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
+	b.WriteString("SHELL=/bin/sh\n")
+	b.WriteString("PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
+	for i := range ps {
+		p := &ps[i]
+		name := strings.ReplaceAll(p.Name, "\n", " ")
+		if !p.Enabled {
+			b.WriteString("# " + name + " — paused\n")
+			continue
+		}
+		expr, err := p.CronExpr()
+		if err != nil {
+			return "", fmt.Errorf("%s: %w", p.Name, err)
+		}
+		fmt.Fprintf(&b, "# %s — verify %s\n", name, p.Scope)
+		fmt.Fprintf(&b, "%s %s %s -config %s landing-verify %s\n\n", expr, user, exe, config, p.ID)
+	}
+	return b.String(), nil
+}
+
+// WriteVerifyCron puts the verify schedules on the machine, or removes the file when none run.
+func (m *Manager) WriteVerifyCron(ctx context.Context, ps []VerifyPolicy, user, exe, config string) error {
+	live := false
+	for i := range ps {
+		live = live || ps[i].Enabled
+	}
+	if !live {
+		return m.RemoveRootFile(ctx, VerifyCronFile)
+	}
+	text, err := VerifyCronText(ps, user, exe, config)
+	if err != nil {
+		return err
+	}
+	return m.writeRoot(ctx, VerifyCronFile, text)
+}
+
+// ---- finding the landing tier ----
+
+// MoverStatusDir is where a landing mover writes what it is doing.
+const MoverStatusDir = "/run/landing-mover"
+
+// DefaultLedger is where hamlet's mover keeps its ledger when no status file names one.
+const DefaultLedger = "/var/lib/landing-mover/ledger.jsonl"
+
+// LandingTarget is one landing tier a pass can check: the branch files land on, the pool
+// directory they are moved to, and the mover's ledger.
+type LandingTarget struct {
+	Src    string `json:"src"`
+	Dst    string `json:"dst"`
+	Ledger string `json:"ledger"`
+}
+
+// LandingTargets reads the movers' status files. Each mover names its own source, target and
+// ledger; with none running, nothing can be verified, and the page says so.
+func LandingTargets() []LandingTarget {
+	var out []LandingTarget
+	m, _ := filepath.Glob(MoverStatusDir + "/*.json")
+	for _, p := range m {
+		b, err := os.ReadFile(p)
+		if err != nil {
+			continue
+		}
+		var st struct{ Src, Dst, Ledger string }
+		if json.Unmarshal(b, &st) != nil || st.Src == "" || st.Dst == "" {
+			continue
+		}
+		if st.Ledger == "" {
+			st.Ledger = DefaultLedger
+		}
+		out = append(out, LandingTarget{Src: filepath.Clean(st.Src), Dst: filepath.Clean(st.Dst), Ledger: st.Ledger})
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Src < out[j].Src })
+	return out
+}
+
+// ---- stale files ----
+
+// StaleAfter is how long a file may wait on a landing branch before it counts as stale. A file
+// older than this is not necessarily lost — a big backlog drains for hours — but it is worth a
+// look, which is what the ZFS tab offers.
+const StaleAfter = time.Hour
+
+// StaleFile is one file that has waited longer than StaleAfter.
+type StaleFile struct {
+	Path  string `json:"path"` // relative to the landing branch, which is also its path on the pool
+	Size  int64  `json:"size"`
+	MTime int64  `json:"mtime"` // unix ms
+}
+
+// StaleFiles lists what has waited on a landing branch longer than StaleAfter, oldest first,
+// at most limit of them, with the full count and size.
+func StaleFiles(branch string, limit int, now time.Time) (files []StaleFile, count int, bytes int64) {
+	cut := now.Add(-StaleAfter)
+	_ = filepath.WalkDir(branch, func(p string, d os.DirEntry, err error) error {
+		if err != nil || !d.Type().IsRegular() {
+			return nil
+		}
+		fi, err := d.Info()
+		if err != nil || !fi.ModTime().Before(cut) {
+			return nil
+		}
+		count++
+		bytes += fi.Size()
+		rel, _ := filepath.Rel(branch, p)
+		files = append(files, StaleFile{Path: rel, Size: fi.Size(), MTime: fi.ModTime().UnixMilli()})
+		return nil
+	})
+	sort.Slice(files, func(i, j int) bool { return files[i].MTime < files[j].MTime })
+	if len(files) > limit {
+		files = files[:limit]
+	}
+	return files, count, bytes
+}
+
+// ---- the pass ----
+
+// LedgerEntry is one line of the mover's ledger.
+type LedgerEntry struct {
+	T       int64  `json:"t"` // when it was moved, unix ms
+	Path    string `json:"path"`
+	Size    int64  `json:"size"`
+	MTimeNs int64  `json:"mtime_ns"`
+	SHA256  string `json:"sha256,omitempty"`
+	Note    string `json:"note,omitempty"`
+}
+
+// VerifyIssue is one file a pass wants a person to look at.
+type VerifyIssue struct {
+	Path  string `json:"path"`
+	Kind  string `json:"kind"` // mismatch | gone | unreadable
+	Moved int64  `json:"moved"`
+	Size  int64  `json:"size"`
+	Note  string `json:"note,omitempty"`
+}
+
+// VerifyResult is what one pass found.
+type VerifyResult struct {
+	ID         string        `json:"id"` // the schedule that ran it, or "manual"
+	Scope      string        `json:"scope"`
+	Started    time.Time     `json:"started"`
+	Finished   time.Time     `json:"finished"`
+	Src        string        `json:"src"`
+	Dst        string        `json:"dst"`
+	Since      int64         `json:"since"` // ledger entries after this (unix ms); 0 for all
+	Entries    int           `json:"entries"`
+	Checked    int           `json:"checked"`
+	Matched    int           `json:"matched"`
+	Mismatched int           `json:"mismatched"`
+	Changed    int           `json:"changed"`
+	Gone       int           `json:"gone"`
+	Waiting    int           `json:"waiting"`
+	Unreadable int           `json:"unreadable"`
+	Unhashed   int           `json:"unhashed"`
+	BytesRead  int64         `json:"bytes_read"`
+	Issues     []VerifyIssue `json:"issues"`
+	Error      string        `json:"error,omitempty"`
+	Cursor     int64         `json:"cursor"` // the newest entry it covered
+}
+
+// OK is true when nothing needs a look: no mismatches, nothing gone or unreadable, no error.
+func (r *VerifyResult) OK() bool {
+	return r.Error == "" && r.Mismatched == 0 && r.Gone == 0 && r.Unreadable == 0
+}
+
+// VerifyProgress is published while a pass runs.
+type VerifyProgress struct {
+	ID        string    `json:"id"`
+	Scope     string    `json:"scope"`
+	PID       int       `json:"pid"`
+	Started   time.Time `json:"started"`
+	Updated   time.Time `json:"updated"`
+	Total     int       `json:"total"`
+	Done      int       `json:"done"`
+	BytesRead int64     `json:"bytes_read"`
+	BytesAll  int64     `json:"bytes_all"`
+	Current   string    `json:"current"`
+}
+
+const maxIssues = 500
+
+// readLedger loads the ledger's entries newer than since, the latest per path, oldest first.
+func readLedger(path string, since int64) ([]LedgerEntry, error) {
+	f, err := os.Open(path)
+	if err != nil {
+		return nil, err
+	}
+	defer f.Close()
+	latest := map[string]LedgerEntry{}
+	sc := bufio.NewScanner(f)
+	sc.Buffer(make([]byte, 64<<10), 4<<20)
+	for sc.Scan() {
+		var e LedgerEntry
+		if json.Unmarshal(sc.Bytes(), &e) != nil || e.Path == "" || e.T <= since {
+			continue
+		}
+		if old, ok := latest[e.Path]; !ok || e.T >= old.T {
+			latest[e.Path] = e
+		}
+	}
+	if err := sc.Err(); err != nil {
+		return nil, err
+	}
+	out := make([]LedgerEntry, 0, len(latest))
+	for _, e := range latest {
+		out = append(out, e)
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].T < out[j].T })
+	return out, nil
+}
+
+// inside joins rel under root and refuses anything that would climb out of it.
+func inside(root, rel string) (string, bool) {
+	p := filepath.Join(root, rel)
+	return p, p == root || strings.HasPrefix(p, root+string(filepath.Separator))
+}
+
+func hashFile(p string, buf []byte, onRead func(int)) (string, error) {
+	f, err := os.Open(p)
+	if err != nil {
+		return "", err
+	}
+	defer f.Close()
+	h := sha256.New()
+	for {
+		n, err := f.Read(buf)
+		if n > 0 {
+			h.Write(buf[:n])
+			onRead(n)
+		}
+		if err == io.EOF {
+			return hex.EncodeToString(h.Sum(nil)), nil
+		}
+		if err != nil {
+			return "", err
+		}
+	}
+}
+
+// Verify runs one pass over a landing tier's ledger. progress is called about once a second.
+func Verify(t LandingTarget, id, scope string, since int64, progress func(VerifyProgress)) VerifyResult {
+	r := VerifyResult{ID: id, Scope: scope, Started: time.Now(), Src: t.Src, Dst: t.Dst, Since: since, Issues: []VerifyIssue{}, Cursor: since}
+	if scope == "all" {
+		r.Since = 0
+	}
+	entries, err := readLedger(t.Ledger, r.Since)
+	if err != nil {
+		if errors.Is(err, os.ErrNotExist) {
+			r.Error = "no ledger yet at " + t.Ledger + " — nothing has been moved since the mover started keeping one"
+		} else {
+			r.Error = "reading the ledger: " + err.Error()
+		}
+		r.Finished = time.Now()
+		return r
+	}
+	r.Entries = len(entries)
+	pr := VerifyProgress{ID: id, Scope: scope, PID: os.Getpid(), Started: r.Started, Total: len(entries)}
+	for _, e := range entries {
+		pr.BytesAll += e.Size
+	}
+	last := time.Time{}
+	tick := func(force bool) {
+		if progress != nil && (force || time.Since(last) >= time.Second) {
+			pr.Updated, last = time.Now(), time.Now()
+			progress(pr)
+		}
+	}
+	issue := func(e LedgerEntry, kind, note string) {
+		if len(r.Issues) < maxIssues {
+			r.Issues = append(r.Issues, VerifyIssue{Path: e.Path, Kind: kind, Moved: e.T, Size: e.Size, Note: note})
+		}
+	}
+	buf := make([]byte, 4<<20)
+	for _, e := range entries {
+		pr.Current = e.Path
+		tick(false)
+		if e.T > r.Cursor {
+			r.Cursor = e.T
+		}
+		r.Checked++
+		pr.Done++
+		dst, ok1 := inside(t.Dst, e.Path)
+		src, ok2 := inside(t.Src, e.Path)
+		if !ok1 || !ok2 {
+			r.Unreadable++
+			issue(e, "unreadable", "path climbs out of the pool")
+			continue
+		}
+		st, err := os.Lstat(dst)
+		if err != nil {
+			if errors.Is(err, os.ErrNotExist) {
+				if _, err := os.Lstat(src); err == nil {
+					r.Waiting++
+					continue
+				}
+				r.Gone++
+				issue(e, "gone", "not on the pool, not waiting to move: deleted or renamed since it moved?")
+				continue
+			}
+			r.Unreadable++
+			issue(e, "unreadable", err.Error())
+			continue
+		}
+		if e.SHA256 == "" {
+			r.Unhashed++
+			continue
+		}
+		mt := st.ModTime().UnixNano()
+		if st.Size() != e.Size || abs64(mt-e.MTimeNs) >= int64(time.Second) {
+			r.Changed++
+			continue
+		}
+		sum, err := hashFile(dst, buf, func(n int) { r.BytesRead += int64(n); pr.BytesRead += int64(n); tick(false) })
+		if err != nil {
+			if errors.Is(err, syscall.EACCES) || errors.Is(err, os.ErrPermission) {
+				r.Unreadable++
+				issue(e, "unreadable", "permission denied")
+			} else {
+				r.Unreadable++
+				issue(e, "unreadable", err.Error())
+			}
+			continue
+		}
+		if sum != e.SHA256 {
+			r.Mismatched++
+			issue(e, "mismatch", "contents differ from the copy that landed")
+			continue
+		}
+		r.Matched++
+	}
+	tick(true)
+	r.Finished = time.Now()
+	return r
+}
+
+func abs64(v int64) int64 {
+	if v < 0 {
+		return -v
+	}
+	return v
+}
+
+// ---- records ----
+
+func verifyDir(dataDir string) string { return filepath.Join(dataDir, "landing-verify") }
+
+func writeJSON(p string, v any) error {
+	if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
+		return err
+	}
+	b, err := json.Marshal(v)
+	if err != nil {
+		return err
+	}
+	tmp := p + ".tmp"
+	if err := os.WriteFile(tmp, b, 0o600); err != nil {
+		return err
+	}
+	return os.Rename(tmp, p)
+}
+
+func readJSON(p string, v any) bool {
+	b, err := os.ReadFile(p)
+	return err == nil && json.Unmarshal(b, v) == nil
+}
+
+// cursorKey names a landing tier's cursor file.
+func cursorKey(t LandingTarget) string {
+	return strings.Trim(strings.NewReplacer("/", "_").Replace(t.Src), "_")
+}
+
+// VerifyCursor is the newest ledger entry a finished pass covered, for scope "new".
+func VerifyCursor(dataDir string, t LandingTarget) int64 {
+	var c struct{ Cursor int64 }
+	readJSON(filepath.Join(verifyDir(dataDir), "cursor-"+cursorKey(t)+".json"), &c)
+	return c.Cursor
+}
+
+// RecordVerify keeps a finished pass: as the run's last result, in the history (newest first,
+// 30 kept), and — when it read the ledger at all — as the new cursor.
+func RecordVerify(dataDir string, t LandingTarget, r VerifyResult) error {
+	d := verifyDir(dataDir)
+	if err := writeJSON(filepath.Join(d, "run-"+r.ID+".json"), r); err != nil {
+		return err
+	}
+	var hist []VerifyResult
+	readJSON(filepath.Join(d, "history.json"), &hist)
+	hist = append([]VerifyResult{r}, hist...)
+	if len(hist) > 30 {
+		hist = hist[:30]
+	}
+	if err := writeJSON(filepath.Join(d, "history.json"), hist); err != nil {
+		return err
+	}
+	if r.Error == "" && r.Cursor > VerifyCursor(dataDir, t) {
+		return writeJSON(filepath.Join(d, "cursor-"+cursorKey(t)+".json"), map[string]int64{"cursor": r.Cursor})
+	}
+	return nil
+}
+
+// VerifyHistory is the kept passes, newest first.
+func VerifyHistory(dataDir string) []VerifyResult {
+	var hist []VerifyResult
+	readJSON(filepath.Join(verifyDir(dataDir), "history.json"), &hist)
+	if hist == nil {
+		hist = []VerifyResult{}
+	}
+	return hist
+}
+
+// LastVerify is the last pass a schedule (or "manual") ran.
+func LastVerify(dataDir, id string) *VerifyResult {
+	var r VerifyResult
+	if !readJSON(filepath.Join(verifyDir(dataDir), "run-"+id+".json"), &r) {
+		return nil
+	}
+	return &r
+}
+
+// ForgetVerify drops a removed schedule's last result (the history keeps it).
+func ForgetVerify(dataDir, id string) {
+	_ = os.Remove(filepath.Join(verifyDir(dataDir), "run-"+id+".json"))
+}
+
+func progressPath(dataDir string) string { return filepath.Join(verifyDir(dataDir), "running.json") }
+
+// WriteVerifyProgress publishes a running pass.
+func WriteVerifyProgress(dataDir string, p VerifyProgress) { _ = writeJSON(progressPath(dataDir), p) }
+
+// ClearVerifyProgress removes it once the pass ends.
+func ClearVerifyProgress(dataDir string) { _ = os.Remove(progressPath(dataDir)) }
+
+// RunningVerify is the pass in progress, or nil. A progress file whose process is gone, or
+// that has not moved for two minutes, is a pass that died: it is reported as not running.
+func RunningVerify(dataDir string) *VerifyProgress {
+	var p VerifyProgress
+	if !readJSON(progressPath(dataDir), &p) {
+		return nil
+	}
+	if p.PID <= 0 || syscall.Kill(p.PID, 0) != nil || time.Since(p.Updated) > 2*time.Minute {
+		return nil
+	}
+	return &p
+}
+
+// LockVerify takes the one-pass-at-a-time lock; release it with the returned func.
+func LockVerify(dataDir string) (func(), error) {
+	if err := os.MkdirAll(verifyDir(dataDir), 0o700); err != nil {
+		return nil, err
+	}
+	f, err := os.OpenFile(filepath.Join(verifyDir(dataDir), "lock"), os.O_CREATE|os.O_RDWR, 0o600)
+	if err != nil {
+		return nil, err
+	}
+	if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
+		f.Close()
+		return nil, errors.New("a verify pass is already running")
+	}
+	return func() { _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN); f.Close() }, nil
+}

internal/storage/landing_test.go

@@ -0,0 +1,137 @@
+package storage
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestVerifyClassifiesEveryCase(t *testing.T) {
+	root := t.TempDir()
+	src, dst := filepath.Join(root, "landing"), filepath.Join(root, "pool")
+	for _, d := range []string{src, dst + "/a", src + "/a"} {
+		if err := os.MkdirAll(d, 0o755); err != nil {
+			t.Fatal(err)
+		}
+	}
+	mtime := time.Date(2026, 10, 25, 12, 0, 0, 0, time.UTC)
+	put := func(p, body string) LedgerEntry {
+		if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
+			t.Fatal(err)
+		}
+		_ = os.Chtimes(p, mtime, mtime)
+		sum := sha256.Sum256([]byte(body))
+		return LedgerEntry{Size: int64(len(body)), MTimeNs: mtime.UnixNano(), SHA256: hex.EncodeToString(sum[:])}
+	}
+	var ledger []LedgerEntry
+	add := func(rel string, e LedgerEntry, at int64) { e.Path, e.T = rel, at; ledger = append(ledger, e) }
+
+	add("a/match.bin", put(dst+"/a/match.bin", "intact"), 10)
+	e := put(dst+"/a/mismatch.bin", "garbled") // same size and mtime as what was hashed, other bytes
+	sum := sha256.Sum256([]byte("GARBLED"))
+	e.SHA256 = hex.EncodeToString(sum[:])
+	add("a/mismatch.bin", e, 11)
+	e = put(dst+"/a/changed.bin", "edited since, longer now")
+	e.Size = 3
+	add("a/changed.bin", e, 12)
+	add("a/gone.bin", LedgerEntry{Size: 4, MTimeNs: mtime.UnixNano(), SHA256: "00"}, 13)
+	put(src+"/a/waiting.bin", "back again")
+	add("a/waiting.bin", LedgerEntry{Size: 10, MTimeNs: mtime.UnixNano(), SHA256: "00"}, 14)
+	e = put(dst+"/a/unhashed", "x")
+	e.SHA256, e.Note = "", "not a regular file"
+	add("a/unhashed", e, 15)
+	add("../escape", LedgerEntry{Size: 1, SHA256: "00"}, 16)
+	// moved twice: only the newer entry counts, and it matches
+	add("a/twice.bin", LedgerEntry{Size: 99, SHA256: "stale"}, 5)
+	add("a/twice.bin", put(dst+"/a/twice.bin", "second time"), 17)
+
+	lp := filepath.Join(root, "ledger.jsonl")
+	f, _ := os.Create(lp)
+	for _, e := range ledger {
+		b, _ := json.Marshal(e)
+		f.Write(append(b, '\n'))
+	}
+	f.WriteString("not json\n")
+	f.Close()
+
+	tg := LandingTarget{Src: src, Dst: dst, Ledger: lp}
+	r := Verify(tg, "manual", "all", 0, nil)
+	want := map[string]int{"checked": 8, "matched": 2, "mismatched": 1, "changed": 1, "gone": 1, "waiting": 1, "unhashed": 1, "unreadable": 1}
+	got := map[string]int{"checked": r.Checked, "matched": r.Matched, "mismatched": r.Mismatched, "changed": r.Changed,
+		"gone": r.Gone, "waiting": r.Waiting, "unhashed": r.Unhashed, "unreadable": r.Unreadable}
+	for k, v := range want {
+		if got[k] != v {
+			t.Errorf("%s: got %d want %d (%+v)", k, got[k], v, r)
+		}
+	}
+	if r.OK() || r.Cursor != 17 || r.Error != "" {
+		t.Errorf("ok=%v cursor=%d err=%q", r.OK(), r.Cursor, r.Error)
+	}
+	kinds := map[string]string{}
+	for _, i := range r.Issues {
+		kinds[i.Path] = i.Kind
+	}
+	if kinds["a/mismatch.bin"] != "mismatch" || kinds["a/gone.bin"] != "gone" || kinds["../escape"] != "unreadable" {
+		t.Errorf("issues %+v", r.Issues)
+	}
+
+	// "new" after a pass at cursor 14 only looks at what moved later
+	r = Verify(tg, "manual", "new", 14, nil)
+	if r.Checked != 3 || r.Matched != 1 || r.Unhashed != 1 || r.Unreadable != 1 {
+		t.Errorf("new since 14: %+v", r)
+	}
+
+	// records: the cursor only advances on a pass that read the ledger
+	dir := t.TempDir()
+	if err := RecordVerify(dir, tg, Verify(tg, "manual", "all", 0, nil)); err != nil {
+		t.Fatal(err)
+	}
+	if c := VerifyCursor(dir, tg); c != 17 {
+		t.Errorf("cursor %d", c)
+	}
+	bad := Verify(LandingTarget{Src: src, Dst: dst, Ledger: filepath.Join(root, "missing")}, "manual", "new", 0, nil)
+	if !strings.Contains(bad.Error, "no ledger yet") {
+		t.Errorf("missing ledger: %q", bad.Error)
+	}
+	if len(VerifyHistory(dir)) != 1 || LastVerify(dir, "manual") == nil {
+		t.Errorf("history/last not kept")
+	}
+}
+
+func TestVerifyPolicy(t *testing.T) {
+	p := VerifyPolicy{ID: "abc", Name: "nightly", Every: "daily", At: "05:10", Scope: "new", Enabled: true}
+	if err := p.Validate(); err != nil {
+		t.Fatal(err)
+	}
+	if x, _ := p.CronExpr(); x != "10 5 * * *" {
+		t.Fatalf("cron %q", x)
+	}
+	txt, err := VerifyCronText([]VerifyPolicy{p, {ID: "off", Name: "paused one", Every: "hourly", Scope: "all"}}, "core", "/usr/local/bin/hamlab", "/cfg.json")
+	if err != nil || !strings.Contains(txt, "10 5 * * * core /usr/local/bin/hamlab -config /cfg.json landing-verify abc") || !strings.Contains(txt, "# paused one — paused") {
+		t.Fatalf("%v\n%s", err, txt)
+	}
+	for _, b := range []VerifyPolicy{{Every: "daily", Scope: "some"}, {Every: "sometimes", Scope: "new"}, {Every: "custom", Scope: "new"}} {
+		if b.Validate() == nil {
+			t.Errorf("accepted %+v", b)
+		}
+	}
+}
+
+func TestStaleFiles(t *testing.T) {
+	d := t.TempDir()
+	now := time.Now()
+	os.MkdirAll(d+"/x", 0o755)
+	for name, age := range map[string]time.Duration{"x/old.bin": 3 * time.Hour, "x/older.bin": 5 * time.Hour, "fresh.bin": time.Minute} {
+		os.WriteFile(filepath.Join(d, name), []byte("12345"), 0o644)
+		os.Chtimes(filepath.Join(d, name), now.Add(-age), now.Add(-age))
+	}
+	files, n, b := StaleFiles(d, 1, now)
+	if n != 2 || b != 10 || len(files) != 1 || files[0].Path != "x/older.bin" {
+		t.Fatalf("%d %d %+v", n, b, files)
+	}
+}

internal/store/store.go

@@ -69,12 +69,18 @@ type State struct {
 	// orchestrator that has since been removed can still be found on disk. See
 	// internal/api/stacks.go.
 	Mounts     []RememberedMount `json:"mounts,omitempty"`
-	// KVMCookies is each KVM's own login, as the cookies its page set in a browser: kept so
-	// the next browser — or this one after it has forgotten them — is signed into the device
-	// already. See internal/api/kvm.go.
-	KVMCookies map[string]map[string]string `json:"kvm_cookies,omitempty"`
 	SamplerMs  int               `json:"sampler_ms,omitempty"`  // how often this machine samples; the hub keeps every machine on the console's refresh rate
 	HistorySec int               `json:"history_sec,omitempty"` // how far back the graphs reach; the hub keeps every machine on the same one
+	// Transcode is where videos played in the Files viewer are made (internal/api/play.go):
+	// a machine and a device on it. nil is the machine the file is on, its best device.
+	Transcode *Transcode `json:"transcode,omitempty"`
+}
+
+// Transcode names a device on a machine: Node "" is the hub itself; Device is a render node
+// or "cpu".
+type Transcode struct {
+	Node   string `json:"node"`
+	Device string `json:"device"`
 }
 
 // UserState returns the prefs of an account, making an empty one if needed.

web/package.json

@@ -7,7 +7,7 @@
     "dev": "vite",
     "build": "tsc --noEmit && vite build",
     "icons": "node scripts/icons.mjs",
-    "test": "node scripts/test-graphcols.mjs && node scripts/test-css-imports.mjs && node scripts/test-resolve-path.mjs"
+    "test": "node scripts/test-graphcols.mjs && node scripts/test-sockets.mjs && node scripts/test-css-imports.mjs && node scripts/test-resolve-path.mjs"
   },
   "dependencies": {
     "@xterm/addon-fit": "^0.10.0",

web/scripts/e2e-player.mjs

@@ -0,0 +1,164 @@
+// The Files player, end to end in Chromium against a running hub. Playwright is borrowed from
+// the wedding site.
+//
+//   BASE=http://10.0.0.234:8100 USER=… PASS=… MACHINE=hub FILE=/path/to/video node scripts/e2e-player.mjs
+//
+// Opens FILE in the Files viewer on MACHINE ("hub" for the hub itself), plays it on Auto,
+// checks the 360 view if it is one, seeks into the middle, picks the lowest quality, tries
+// Original, and reads the menu's "Made on …". Changes nothing. DEVICE="Pixel 7" for a phone;
+// TRACE=1 prints every piece asked for. Screenshots land in $SHOTS (default /tmp/hamlab-shots).
+import { createRequire } from 'node:module'
+import { mkdirSync } from 'node:fs'
+const require = createRequire(import.meta.url)
+const { chromium, devices } = require('/home/core/projects/wedding-website/node_modules/playwright')
+
+const { BASE = 'http://10.0.0.234:8100', USER, PASS, MACHINE = 'hub', FILE, DEVICE, TRACE } = process.env
+const SHOTS = process.env.SHOTS ?? '/tmp/hamlab-shots'
+if (!USER || !PASS || !FILE) { console.error('USER, PASS and FILE are required'); process.exit(2) }
+mkdirSync(SHOTS, { recursive: true })
+
+let failed = 0
+const check = (name, ok, extra = '') => { console.log(`${ok ? 'ok  ' : 'FAIL'} ${name}${extra ? ' — ' + extra : ''}`); if (!ok) failed++ }
+const browser = await chromium.launch({ args: ['--autoplay-policy=no-user-gesture-required', '--use-gl=angle', '--use-angle=swiftshader', '--enable-unsafe-swiftshader'] })
+const ctx = await browser.newContext({ ...(DEVICE ? devices[DEVICE] : { viewport: { width: 1280, height: 860 } }), colorScheme: 'dark' })
+check('signs in', (await ctx.request.post(BASE + '/api/login', { data: { username: USER, password: PASS } })).ok())
+const page = await ctx.newPage()
+const errors = []
+page.on('pageerror', (e) => errors.push(String(e)))
+page.on('console', (m) => { if (m.type() === 'error' && !/401|404|favicon/.test(m.text())) errors.push(m.text()) })
+const T0 = Date.now(), trace = []
+const short = (u) => u.replace(/.*\/files\/play\/[0-9a-f]+\//, '')
+page.on('request', (r) => { if (/\/files\/play\/.*\.ts$/.test(r.url())) trace.push(`${((Date.now() - T0) / 1000).toFixed(1)} ask ${short(r.url())}`) })
+page.on('requestfinished', (r) => { if (/\/files\/play\/.*\.ts$/.test(r.url())) trace.push(`${((Date.now() - T0) / 1000).toFixed(1)} got ${short(r.url())}`) })
+const mark = (m) => trace.push(`${((Date.now() - T0) / 1000).toFixed(1)} -- ${m}`)
+
+const dir = FILE.slice(0, FILE.lastIndexOf('/')) || '/', name = FILE.slice(FILE.lastIndexOf('/') + 1)
+await page.goto(`${BASE}/#/${MACHINE}/files/${encodeURIComponent(dir)}`)
+await page.locator('.file-row', { hasText: name }).first().click()
+await page.locator('.vp').waitFor()
+
+const vstate = () => page.evaluate(() => { const v = document.querySelector('.vp video'); return v ? { t: v.currentTime, paused: v.paused, w: v.videoWidth, h: v.videoHeight, rs: v.readyState, d: v.duration } : null })
+async function playingPast(t, ms = 60000) {
+  const t0 = Date.now()
+  while (Date.now() - t0 < ms) {
+    const s = await vstate()
+    if (s && s.t > t && !s.paused && s.rs >= 3) return Date.now() - t0
+    await page.waitForTimeout(200)
+  }
+  return -1
+}
+async function controls() {
+  if (await page.locator('.vp.bare').count()) {
+    const b = await page.locator('.vp').boundingBox()
+    if (DEVICE) await page.touchscreen.tap(b.x + 40, b.y + 40)
+    else await page.mouse.move(b.x + b.width / 2 + Math.random() * 40, b.y + b.height / 3)
+    await page.waitForTimeout(300)
+  }
+}
+
+let took = await playingPast(1)
+check('plays on Auto', took >= 0, `${((Date.now() - T0) / 1000).toFixed(1)} s from opening`)
+await page.waitForTimeout(6000)
+let s = await vstate()
+check('Auto settles on a quality', s?.w > 0, `${s?.w}×${s?.h}; “${await page.locator('.vp-q span').textContent()}”`)
+if (await page.locator('canvas.vp-sphere').count()) {
+  const px = await page.evaluate(() => { const c = document.querySelector('canvas.vp-sphere'); const g = c.getContext('webgl'); const d = new Uint8Array(64); g.readPixels(c.width >> 1, c.height >> 1, 4, 4, g.RGBA, g.UNSIGNED_BYTE, d); return d.reduce((a, b) => a + b, 0) })
+  check('the 360° view draws', px > 0)
+}
+await page.screenshot({ path: `${SHOTS}/h1-auto.png` })
+
+// No theme touches playback: for every theme the stylesheet has rules for, no full-window
+// overlay is drawn, the picture isn't filtered, and the player's lettering is plain.
+{
+  const was = await page.evaluate(() => document.documentElement.getAttribute('data-theme'))
+  const themes = await page.evaluate(() => {
+    const ids = new Set()
+    for (const sh of document.styleSheets) { let rules; try { rules = sh.cssRules } catch { continue }
+      const walk = (rs) => { for (const r of rs) { if (r.cssRules) walk(r.cssRules); for (const m of (r.selectorText ?? '').matchAll(/data-theme="([^"]+)"/g)) ids.add(m[1]) } }
+      walk(rules) }
+    return [...ids]
+  })
+  const bad = []
+  for (const t of themes) {
+    const r = await page.evaluate((t) => {
+      document.documentElement.setAttribute('data-theme', t)
+      const els = [document.documentElement, document.body, document.querySelector('.app'), document.querySelector('#root')].filter(Boolean)
+      const overlays = els.flatMap((e) => ['::before', '::after'].map((p) => getComputedStyle(e, p))).filter((c) => c.display !== 'none' && c.content !== 'none' && c.content !== 'normal' && (c.position === 'fixed' || c.position === 'absolute')).length
+      const pics = [...document.querySelectorAll('.vp video, .vp canvas')].map((e) => getComputedStyle(e).filter).filter((f) => f !== 'none').length
+      const c = getComputedStyle(document.querySelector('.vp'))
+      const b = getComputedStyle(document.querySelector('.vp button'))
+      return { overlays, pics, plain: !/mono|courier/i.test(b.fontFamily) && c.fontWeight === '400' && c.letterSpacing === 'normal' && c.textShadow === 'none' }
+    }, t)
+    if (r.overlays || r.pics || !r.plain) bad.push(`${t}: ${JSON.stringify(r)}`)
+  }
+  check(`no theme touches the video (${themes.length} themes with their own rules)`, themes.length > 0 && bad.length === 0, bad.join(' / ') || themes.join(', '))
+  await page.evaluate((t) => t ? document.documentElement.setAttribute('data-theme', t) : document.documentElement.removeAttribute('data-theme'), was)
+}
+
+// Volume: the speaker mutes and unmutes, the slider sets the level, and it is remembered.
+{
+  await controls()
+  const vs = () => page.evaluate(() => { const v = document.querySelector('.vp video'); return { muted: v.muted, volume: Math.round(v.volume * 100) / 100 } })
+  await page.getByRole('button', { name: 'Mute' }).click()
+  const a = await vs()
+  await page.getByRole('button', { name: 'Unmute' }).click()
+  const b = await vs()
+  let c = b
+  if (await page.getByLabel('Volume').isVisible()) { await page.getByLabel('Volume').fill('0.3'); c = await vs() }
+  const kept = await page.evaluate(() => Object.entries(localStorage).find(([k]) => /volume/.test(k))?.[1] ?? '')
+  check('volume: mute, unmute, level, remembered', a.muted && !b.muted && (c.volume === 0.3 || !(await page.getByLabel('Volume').isVisible())) && /level/.test(kept),
+    `${JSON.stringify(a)} → ${JSON.stringify(b)} → ${JSON.stringify(c)}, kept ${kept}`)
+  if (await page.getByLabel('Volume').isVisible()) await page.getByLabel('Volume').fill('1')
+}
+
+// The 5-second buttons: back and on, the ball with them.
+{
+  await controls()
+  const t0 = (await vstate()).t
+  await page.getByRole('button', { name: 'On 5 seconds' }).click()
+  const t1 = (await vstate()).t
+  await page.getByRole('button', { name: 'Back 5 seconds' }).click()
+  await page.getByRole('button', { name: 'Back 5 seconds' }).click()
+  const t2 = (await vstate()).t
+  const ball = await page.evaluate(() => Number(document.querySelector('.vp input[type=range]').value))
+  check('+5 and −5 buttons', t1 - t0 > 4 && t1 - t0 < 6.5 && t1 - t2 > 9 && t1 - t2 < 11 && Math.abs(ball - t2) < 1, `${t0.toFixed(1)} → ${t1.toFixed(1)} → ${t2.toFixed(1)}, ball ${ball.toFixed(1)}`)
+}
+
+const mid = Math.floor(s.d / 2)
+mark('seek')
+await page.evaluate((m) => { document.querySelector('.vp video').currentTime = m }, mid)
+took = await playingPast(mid + 0.5)
+check('a seek into the middle plays again', took >= 0, `${(took / 1000).toFixed(1)} s`)
+
+await controls()
+await page.locator('.vp-q').click()
+await page.waitForTimeout(2500) // the menu asks the machine what it is doing
+const items = page.locator('.vp-menu button')
+const n = await items.count()
+check('the menu lists Auto, Original and the qualities', n >= 3, (await items.allTextContents()).join(' | '))
+const foot = await page.locator('.vp-menu-f').textContent()
+check('it says where it is made', /Made on \S+/.test(foot), foot)
+await page.screenshot({ path: `${SHOTS}/h2-menu.png` })
+const lowLabel = await items.nth(n - 1).locator('span').nth(1).textContent()
+mark('pick ' + lowLabel)
+const before = (await vstate()).t
+const t0 = Date.now()
+await items.nth(n - 1).click()
+let ok = false
+for (let i = 0; i < 200 && !ok; i++) { await page.waitForTimeout(200); s = await vstate(); ok = s.t > before + 1 && !s.paused && (await page.locator('.vp-q span').textContent()) === lowLabel }
+check(`picking ${lowLabel} switches to it`, ok, `${s.w}×${s.h} after ${((Date.now() - t0) / 1000).toFixed(1)} s`)
+
+await controls()
+await page.locator('.vp-q').click()
+await page.locator('.vp-menu button', { hasText: 'Original' }).click()
+await page.waitForTimeout(4000)
+const label = await page.locator('.vp-q span').textContent()
+const note = await page.locator('.vp-note').count() ? await page.locator('.vp-note').textContent() : ''
+check('Original plays, or falls back to Auto and says so', label === 'Original' || (/^Auto/.test(label) && /can’t play/.test(note)), `${label}${note ? ' — ' + note : ''}`)
+check('still playing', (await playingPast((await vstate()).t + 0.5)) >= 0)
+await page.screenshot({ path: `${SHOTS}/h3-after-original.png` })
+check('no page errors', errors.length === 0, errors.slice(0, 3).join(' / '))
+if (TRACE) console.log(trace.join('\n'))
+await browser.close()
+console.log(failed ? `${failed} failed` : 'all ok')
+process.exit(failed ? 1 : 0)

web/scripts/test-graphcols.mjs

@@ -8,7 +8,7 @@ import { join } from 'node:path'
 
 const out = join(mkdtempSync(join(tmpdir(), 'graphcols-')), 'graphcols.mjs')
 buildSync({ entryPoints: [new URL('../src/graphcols.ts', import.meta.url).pathname], outfile: out, format: 'esm', logLevel: 'error' })
-const { columnsOf } = await import(pathToFileURL(out).href)
+const { columnsOf, columnsAt } = await import(pathToFileURL(out).href)
 
 const fails = []
 const check = (ok, what, detail) => { if (!ok) fails.push(what + (detail !== undefined ? ' — ' + JSON.stringify(detail).slice(0, 200) : '')) }
@@ -38,5 +38,28 @@ const wide = columnsOf(base, CAP - 1, CAP, 390)
 check(wide.length === 390 && Math.max(...wide) === 900, 'a box wider than the history is filled, nothing invented', wide.length)
 check(columnsOf([], 0, CAP, COLS).length === 0 && columnsOf([5], 0, 2, 50).length >= 1, 'empty and single-point curves')
 
+// Sliding with the ticks: four ticks to a point (a 250 ms rate, a 1 s stride). Every tick moves
+// the graph left by the same fraction of a column, the newest column grows in from the right, and
+// a column, once finished, keeps its value wherever it has slid to.
+for (const cols of [COLS, 390]) {
+  let v = base.slice(), adv = CAP - 1, prevAt = null, back = 0, jump = 0, held = 0
+  const was = new Map() // a finished column's value, by its number
+  for (let t = 0; t < 4 * 30; t++) {
+    const now = adv + ((t % 4) + 1) / 4
+    const { cols: seen, edge, last } = columnsAt(v, adv, now, CAP, cols)
+    check(edge > 0 && edge <= 1, `edge in (0, 1] (${cols} cols)`, edge)
+    check(seen.length >= cols - 2 && seen.length <= cols + 1, `the box is filled, give or take the sliver (${cols} cols)`, seen.length)
+    const at = last + edge // how far the graph has slid, in columns
+    if (prevAt !== null) { if (at < prevAt - 1e-9) back++; if (at > prevAt + 1 + 1e-9) jump++; if (Math.abs(at - prevAt) < 1e-9) held++ }
+    prevAt = at
+    seen.forEach((x, i) => { const k = last - (seen.length - 1 - i); if (k < last - 1) { if (was.has(k) && was.get(k) !== x) fails.push(`column ${k} changed after it was finished`); was.set(k, x) } })
+    if ((t % 4) === 3) { v.push(10); v.shift(); adv++ }
+  }
+  check(back === 0 && jump === 0, `the graph only slides left, less than a column a tick (${cols} cols)`, { back, jump })
+  check(held < 4 * 30 / 4, `the graph moves on almost every tick, not once a point (${cols} cols)`, held)
+}
+// a point per tick (stride == rate) is the old layout exactly
+check(JSON.stringify(columnsAt(base, CAP - 1, CAP, CAP, COLS).cols.slice(-COLS)) === JSON.stringify(columnsOf(base, CAP - 1, CAP, COLS)), 'columnsOf is columnsAt at a whole point')
+
 if (fails.length) { console.error('graphcols: FAIL\n  ' + fails.join('\n  ')); process.exit(1) }
 console.log('graphcols: ok')

web/scripts/test-sockets.mjs

@@ -0,0 +1,34 @@
+// node scripts/test-sockets.mjs — src/sockets.ts: each CPU's threads, counted under it.
+import { buildSync } from 'esbuild'
+import { pathToFileURL } from 'node:url'
+import { mkdtempSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+
+const out = join(mkdtempSync(join(tmpdir(), 'sockets-')), 'sockets.mjs')
+buildSync({ entryPoints: [new URL('../src/sockets.ts', import.meta.url).pathname], outfile: out, format: 'esm', logLevel: 'error' })
+const { socketThreads } = await import(pathToFileURL(out).href)
+
+const fails = []
+const eq = (got, want, what) => { if (JSON.stringify(got) !== JSON.stringify(want)) fails.push(`${what} — got ${JSON.stringify(got)}, want ${JSON.stringify(want)}`) }
+const range = (a, b) => Array.from({ length: b - a }, (_, i) => a + i)
+
+// nexus: 2 × E5-2680 v2, as /sys/devices/system/cpu/cpu*/topology/physical_package_id reads
+const nexus = [...Array(10).fill(0), ...Array(10).fill(1), ...Array(10).fill(0), ...Array(10).fill(1)]
+const got = socketThreads(nexus)
+eq(got.map((s) => s.pkg), [0, 1], 'two sockets, in order')
+eq(got[0].cpus, [...range(0, 10), ...range(20, 30)], "CPU 1's threads")
+eq(got[1].cpus, [...range(10, 20), ...range(30, 40)], "CPU 2's threads")
+eq(got.map((s) => s.cpus.length), [20, 20], 'twenty threads each')
+// every thread is under exactly one CPU
+eq(got.flatMap((s) => s.cpus).sort((a, b) => a - b), range(0, 40), 'every thread once')
+// a thread's number under its CPU runs 0…19 on both
+for (const s of got) eq(s.cpus.map((_, n) => n), range(0, 20), `CPU ${s.pkg + 1} counts from 0`)
+
+// a board that numbers each socket's threads together, and one whose socket ids are not 0 and 1
+eq(socketThreads([0, 0, 0, 0, 1, 1, 1, 1]), [{ pkg: 0, cpus: [0, 1, 2, 3] }, { pkg: 1, cpus: [4, 5, 6, 7] }], 'contiguous numbering')
+eq(socketThreads([3, 1, 3, 1]), [{ pkg: 1, cpus: [1, 3] }, { pkg: 3, cpus: [0, 2] }], 'socket ids sorted')
+eq(socketThreads([]), [], 'nothing')
+
+if (fails.length) { console.error('sockets: FAIL\n  ' + fails.join('\n  ')); process.exit(1) }
+console.log('sockets: ok')

web/src/Docker.tsx

@@ -402,7 +402,7 @@ export default function Docker() {
   const rate = useRefreshMs()
   const load = useCallback(async () => { try { setD(await api.docker()); setErr('') } catch (e) { setErr(e instanceof Error ? e.message : 'failed') } api.stacks().then((r) => setStacks(r.stacks)).catch(() => setStacks([])) }, [])
   useEffect(() => { load(); const t = setInterval(() => { if (document.visibilityState === 'visible') load() }, Math.max(5000, rate)); return () => clearInterval(t) }, [load, rate])
-  const live = useSensorStream<{ c: Record<string, Sensor> }>({ c: 'container.*' }, { everyMs: Math.max(2000, rate) })
+  const live = useSensorStream<{ c: Record<string, Sensor> }>({ c: 'container.*' }, { everyMs: rate })
   const sensors = live.data?.c ?? {}
   const byStack = useMemo(() => {
     const f = filter.trim().toLowerCase(); const out = new Map<string, DockerContainer[]>()

web/src/Files.tsx

@@ -1,4 +1,4 @@
-import { DragEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
+import { DragEvent, PointerEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
 import { api, FileEntry, FileInfo, TaskInfo, ThumbStatus } from './api'
 import { fmtBytes } from './fmt'
 import { Btn, Loading, Spinner, askText, askYes, setUnsaved, toast, useEscape } from './ui'
@@ -22,6 +22,7 @@ export function resolvePath(input: string, here: string): string {
 }
 const fmtTime = (ms: number) => { const d = new Date(ms); const now = new Date(); return d.toDateString() === now.toDateString() ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) : d.toLocaleDateString(undefined, { year: d.getFullYear() === now.getFullYear() ? undefined : 'numeric', month: 'short', day: '2-digit' }) }
 const isImage = (n: string) => /\.(png|jpe?g|gif|webp|svg|bmp|ico|avif)$/i.test(n)
+const isMedia = (e: FileEntry) => !!e.preview || isImage(e.name)
 const shellQuote = (s: string) => "'" + s.replace(/'/g, "'\\''") + "'"
 const MAX_EDIT = 2 * 1024 * 1024
 
@@ -198,7 +199,9 @@ function ThumbNote({ st, sudo, onChanged }: { st: ThumbStatus | null; sudo: bool
 }
 
 // ---- editor / viewer ----
-function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolean; onClose: () => void; onChanged: () => void }) {
+// steps: the files ← and → go through, in the list's order (only the pictures, videos and
+// sound when one of those is open).
+function Viewer({ path, sudo, steps, onStep, onClose, onChanged }: { path: string; sudo: boolean; steps: string[]; onStep: (p: string) => void; onClose: () => void; onChanged: () => void }) {
   const [info, setInfo] = useState<FileInfo | null>(null)
   const [text, setText] = useState<string | null>(null)
   const [orig, setOrig] = useState('')
@@ -251,10 +254,49 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
   }, [dirty, name])
   const dl = api.fileDownloadURL(path, sudo)
 
+  // Previous and next: the buttons, ← and → (unless typing, or the player has the focus and
+  // seeks with them), and a sideways swipe on a picture.
+  const at = steps.indexOf(path)
+  const prev = at > 0 ? steps[at - 1] : null
+  const next = at >= 0 && at < steps.length - 1 ? steps[at + 1] : null
+  const step = useCallback(async (to: string | null) => {
+    if (!to) return
+    if (dirty && !await askYes('Discard unsaved changes?', 'Discard')) return
+    setUnsaved(''); onStep(to)
+  }, [dirty, onStep])
+  useEffect(() => {
+    const h = (e: KeyboardEvent) => {
+      if (e.defaultPrevented || e.altKey || e.ctrlKey || e.metaKey || e.shiftKey) return
+      const t = e.target as HTMLElement | null
+      if (t?.closest('input, textarea, select, [contenteditable], .vp')) return
+      if (e.key === 'ArrowLeft' && prev) { e.preventDefault(); step(prev) }
+      else if (e.key === 'ArrowRight' && next) { e.preventDefault(); step(next) }
+    }
+    window.addEventListener('keydown', h)
+    return () => window.removeEventListener('keydown', h)
+  }, [prev, next, step])
+  // The next and previous pictures load now, so stepping onto them is instant.
+  useEffect(() => {
+    for (const p of [prev, next]) if (p && isImage(p)) new Image().src = api.fileReadURL(p, sudo)
+  }, [prev, next, sudo])
+  const swipe = useRef<{ x: number; y: number } | null>(null)
+  const swipeEnd = (e: PointerEvent) => {
+    const s = swipe.current
+    swipe.current = null
+    if (!s) return
+    const dx = e.clientX - s.x, dy = e.clientY - s.y
+    if (Math.abs(dx) > 50 && Math.abs(dx) > 1.5 * Math.abs(dy)) step(dx < 0 ? next : prev)
+  }
+
   return (
     <div className="viewer">
       <div className="viewer-bar">
         <button className="btn" onClick={close}>‹ Back</button>
+        {at >= 0 && steps.length > 1 && <>
+          <button className="btn step" disabled={!prev} title="previous (←)" onClick={() => step(prev)}>←</button>
+          <span className="muted num">{at + 1}/{steps.length}</span>
+          <button className="btn step" disabled={!next} title="next (→)" onClick={() => step(next)}>→</button>
+        </>}
         <span className="vname mono" title={path}>{name}{dirty && <span className="dirty"> •</span>}</span>
         <span className="spacer" />
         {info && <span className="muted num">{fmtBytes(info.size)} · {info.perm} · {info.user}</span>}
@@ -266,11 +308,17 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
       {info && text !== null && (
         <textarea className="editor file-editor" value={text} spellCheck={false} onChange={(e) => setText(e.target.value)} />
       )}
-      {info && text === null && isImage(name) && <div className="preview"><img src={api.fileReadURL(path, sudo)} alt={name} /></div>}
-      {/* Video: the browser's own decoder first, and a re-encode on the machine when that fails. Audio as it is. */}
-      {info && text === null && !isImage(name) && info.preview === 'video' && <Player path={path} sudo={sudo} name={name} />}
+      {info && text === null && isImage(name) && (
+        <div className="preview pic" onPointerDown={(e) => { if (e.pointerType !== 'mouse' && e.isPrimary) swipe.current = { x: e.clientX, y: e.clientY } }} onPointerUp={swipeEnd} onPointerCancel={() => { swipe.current = null }}>
+          <img src={api.fileReadURL(path, sudo)} alt={name} draggable={false} />
+          {prev && <button className="step-edge prev" title="previous (←)" onClick={() => step(prev)}>‹</button>}
+          {next && <button className="step-edge next" title="next (→)" onClick={() => step(next)}>›</button>}
+        </div>
+      )}
+      {/* Video: the player (Auto, by hand, or the file as it is), made on the Transcode device. Audio as it is. */}
+      {info && text === null && !isImage(name) && info.preview === 'video' && <Player key={path} path={path} sudo={sudo} name={name} />}
       {info && text === null && !isImage(name) && info.preview === 'audio' && !cantPlay && (
-        <div className="preview audio"><div className="audio-box"><div className="ficon">🎵</div><div className="mono small">{name}</div><audio controls preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => setCantPlay(true)} /></div></div>
+        <div className="preview audio"><div className="audio-box"><div className="ficon">🎵</div><div className="mono small">{name}</div><audio controls autoPlay preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => setCantPlay(true)} /></div></div>
       )}
       {info && text === null && !isImage(name) && info.preview === 'audio' && cantPlay && (
         <div className="notice">This browser can't play {name} as it is — the codec isn't one it decodes. Download it instead.</div>
@@ -307,6 +355,9 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
   const [thumbs, setThumbs] = useState<ThumbStatus | null>(null)
   const [listEl, setListEl] = useState<HTMLDivElement | null>(null)
   const near = useMemo(() => (listEl && previews ? new Near(listEl) : null), [listEl, previews])
+  // Video previews are made on the Transcode device (Settings), wherever it is.
+  const [tc, setTc] = useState<string>('')
+  useEffect(() => { api.transcodeSelected().then((t) => setTc(t.selected ? `videos on ${t.host}'s ${t.name} (Settings → Transcode device), images on this machine's CPU` : '')).catch(() => {}) }, [])
   const readThumbs = useCallback((refresh = false) => { api.thumbStatus(sudo, refresh).then(setThumbs).catch((e) => setThumbs({ ok: false, engine: '', summary: '', gpus: [], error: e instanceof Error ? e.message : 'failed', user: '', cache: { path: '', entries: 0, bytes: 0 } })) }, [sudo])
   useEffect(() => { try { localStorage.setItem(PREVIEWS_KEY, previews ? '1' : '0') } catch { /* private mode */ } }, [previews])
   useEffect(() => { if (previews) readThumbs(); else setThumbs(null) }, [previews, readThumbs])
@@ -396,7 +447,11 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
     if (files.length) uploadFiles(files)
   }
 
-  if (open) return <div className="files"><Viewer path={open} sudo={sudo} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
+  if (open) {
+    const cur = shown.find((e) => join(path, e.name) === open)
+    const steps = shown.filter((e) => e.type !== 'dir' && e.type !== 'other' && (!cur || !isMedia(cur) || isMedia(e))).map((e) => join(path, e.name))
+    return <div className="files"><Viewer path={open} sudo={sudo} steps={steps} onStep={setOpen} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
+  }
 
   return (
     <div className={'files' + (drag ? ' drag' : '')} onDragOver={(e) => { e.preventDefault(); setDrag(true) }} onDragLeave={() => setDrag(false)} onDrop={onDrop}>
@@ -409,7 +464,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
         <div className="files-tools">
           <button className={'btn' + (sudo ? ' on' : '')} title="browse as root" onClick={() => setSudo(!sudo)}>{sudo ? 'root' : 'sudo'}</button>
           <button className={'btn' + (hidden ? ' on' : '')} title="show hidden files" onClick={() => setHidden(!hidden)}>.hidden</button>
-          <button className={'btn' + (previews ? ' on' : '')} title={'thumbnails for images and videos' + (thumbs ? ` — ${thumbs.summary}` + (thumbs.cache.entries ? ` · ${thumbs.cache.entries} kept (${fmtBytes(thumbs.cache.bytes)})` : '') : '')} onClick={() => setPreviews(!previews)}>previews</button>
+          <button className={'btn' + (previews ? ' on' : '')} title={'thumbnails for images and videos' + (thumbs ? ` — ${tc || thumbs.summary}` + (thumbs.cache.entries ? ` · ${thumbs.cache.entries} kept (${fmtBytes(thumbs.cache.bytes)})` : '') : '')} onClick={() => setPreviews(!previews)}>previews</button>
           <select className="btn" value={sort} onChange={(e) => setSort(e.target.value as Sort)}><option value="name">name</option><option value="size">size</option><option value="mtime">modified</option></select>
           <input className="filter" placeholder="filter" value={q} onChange={(e) => setQ(e.target.value)} />
           <button className="btn" onClick={mkdir}>+ folder</button>

web/src/Gallery.tsx

@@ -37,8 +37,8 @@ export default function Gallery() {
   const rate = useRefreshMs()
   const [find, setFind] = useState('')
   useEffect(() => { fetchCatalog().then(setCat).catch((e) => setErr(e instanceof Error ? e.message : 'failed')) }, [])
-  // Values only, at most once a second: the catalog already holds everything that doesn't change.
-  const live = useSensorStream<[Record<string, Sensor['value']>]>(['*'], { full: false, everyMs: Math.max(1000, rate) })
+  // Values only, at the rate: the catalog already holds everything that doesn't change.
+  const live = useSensorStream<[Record<string, Sensor['value']>]>(['*'], { full: false, everyMs: rate })
   const now = Date.now()
   const sensors = useMemo(() => {
     const v = live.data?.[0]

web/src/Hardware.tsx

@@ -158,6 +158,10 @@ export default function Hardware() {
   const capacity = drives.reduce((n, d) => n + d.size, 0)
   const threads = cpu.threads ?? 0
   const cores = cpu.cores ?? 0
+  // a two-socket board is two processors, not one with twice the cores: count and name each
+  const sockets = Math.max(1, cpu.sockets ?? 1)
+  const multi = sockets > 1
+  const per = (n: number) => Math.round(n / sockets)
 
   return (
     <div className="dash">
@@ -171,8 +175,11 @@ export default function Hardware() {
       <div className="kpis">
         <div className="tile">
           <div className="label">Processor</div>
-          <div className="value" title={cpu.model} style={(cpu.model?.length ?? 0) > 20 ? { fontSize: 16 } : undefined}>{cpu.model}</div>
-          <div className="sub">{[cores ? `${cores} core${cores === 1 ? '' : 's'}` : '', threads && threads !== cores ? `${threads} threads` : '', cpu.max_mhz ? `up to ${ghz(cpu.max_mhz)}` : ''].filter(Boolean).join(' · ')}</div>
+          <div className="value" title={cpu.model} style={(cpu.model?.length ?? 0) > (multi ? 16 : 20) ? { fontSize: 16 } : undefined}>{multi ? `${sockets} × ${cpu.model}` : cpu.model}</div>
+          <div className="sub">{(multi
+            ? [`${sockets} CPUs`, cores ? `${per(cores)} cores${threads && threads !== cores ? ` / ${per(threads)} threads` : ''} each` : '', cpu.max_mhz ? `up to ${ghz(cpu.max_mhz)}` : '']
+            : [cores ? `${cores} core${cores === 1 ? '' : 's'}` : '', threads && threads !== cores ? `${threads} threads` : '', cpu.max_mhz ? `up to ${ghz(cpu.max_mhz)}` : '']
+          ).filter(Boolean).join(' · ')}</div>
         </div>
         <div className="tile">
           <div className="label">Memory</div>
@@ -199,9 +206,11 @@ export default function Hardware() {
           <div className="props">
             <P k="Model" v={cpu.model} />
             <P k="Vendor" v={cpu.vendor} />
-            <P k="Cores" v={cores ? `${cores}${cpu.threads_per_core && cpu.threads_per_core > 1 ? ` · ${cpu.threads_per_core} threads each` : ''}` : ''} />
+            {multi && <P k="CPUs" v={`${sockets}`} />}
+            <P k="Cores" v={cores ? `${multi ? `${per(cores)} per CPU · ${cores} in all` : cores}${cpu.threads_per_core && cpu.threads_per_core > 1 ? ` · ${cpu.threads_per_core} threads each` : ''}` : ''} />
+            {multi && threads > 0 && <P k="Threads" v={`${per(threads)} per CPU · ${threads} in all`} />}
             <P k="Clock" v={cpu.min_mhz && cpu.max_mhz ? `${ghz(cpu.min_mhz)} – ${ghz(cpu.max_mhz)}` : ghz(cpu.max_mhz)} />
-            <P k="Socket" v={(cpu.socket_names ?? []).join(', ')} />
+            <P k={multi ? 'Sockets' : 'Socket'} v={(cpu.socket_names ?? []).join(', ')} />
             <P k="Arch" v={cpu.arch} />
             <P k="Family" v={[cpu.family && `family ${cpu.family}`, cpu.model_id && `model ${cpu.model_id}`, cpu.stepping && `stepping ${cpu.stepping}`].filter(Boolean).join(' · ')} />
             <P k="Microcode" v={cpu.microcode} mono />

web/src/KVM.tsx

@@ -1,187 +0,0 @@
-// The KVM of the machine you are looking at: the device's own web page, shown in the tab.
-//
-// One per machine, and it follows the machine switcher — switch to `grid` and you get
-// grid's KVM. The address is held on the **hub**, not on the machine, because a KVM earns
-// its keep exactly when that machine is down and cannot answer for itself; the hub proxies
-// the device either way.
-//
-// Nothing here knows anything about any particular KVM. It is an iframe pointed at Ham
-// Lab's own proxy, so whatever web UI the device serves is what you get.
-//
-// On the keyboard: a page cannot capture keys and forward them into an iframe, and does not
-// need to. An iframe with focus receives them directly. What the browser itself swallows —
-// Escape, Tab, Ctrl-W, the function keys — is what `navigator.keyboard.lock()` releases,
-// and that needs fullscreen and only exists in Chromium. Everywhere else you get the plain
-// version, which is most keys, and the device's own on-screen buttons for the rest.
-import { FormEvent, useCallback, useEffect, useRef, useState } from 'react'
-import { KVM as Device, api } from './api'
-import { Btn, Loading, askYes, toast, useEscape } from './ui'
-
-// lock() releases the keys the browser would otherwise keep for itself. Chromium only; a
-// browser without it simply does not get the extra keys.
-type KeyboardLock = { lock?: (keys?: string[]) => Promise<void>; unlock?: () => void }
-const keyboardAPI = (): KeyboardLock | undefined => (navigator as unknown as { keyboard?: KeyboardLock }).keyboard
-
-// Fullscreen, across the three ways browsers spell it. Safari on a Mac wants the webkit
-// prefix; Safari on iOS has no fullscreen for an ordinary element at all — only for a
-// <video> — so there the page expands inside the viewport instead, which is the same thing
-// minus the browser's own chrome.
-type FSElement = HTMLElement & { webkitRequestFullscreen?: () => Promise<void> | void }
-type FSDocument = Document & { webkitFullscreenElement?: Element | null; webkitExitFullscreen?: () => Promise<void> | void }
-const nativeFullscreen = () => {
-  if (typeof document === 'undefined') return false
-  const el = document.documentElement as FSElement
-  return typeof el.requestFullscreen === 'function' || typeof el.webkitRequestFullscreen === 'function'
-}
-const fullscreenElement = () => (document as FSDocument).fullscreenElement ?? (document as FSDocument).webkitFullscreenElement ?? null
-async function enterFullscreen(el: FSElement) {
-  if (typeof el.requestFullscreen === 'function') return el.requestFullscreen()
-  if (typeof el.webkitRequestFullscreen === 'function') return el.webkitRequestFullscreen()
-  throw new Error('this browser has no fullscreen for a page element')
-}
-function exitFullscreen() {
-  const d = document as FSDocument
-  if (typeof d.exitFullscreen === 'function') return d.exitFullscreen()
-  if (typeof d.webkitExitFullscreen === 'function') return d.webkitExitFullscreen()
-}
-
-export default function KVM({ machine, title }: { machine: string | null; title: string }) {
-  const [kvm, setKvm] = useState<Device | null | undefined>(undefined)
-  const [adding, setAdding] = useState(false)
-  const [err, setErr] = useState('')
-  const [full, setFull] = useState(false)   // native fullscreen is on
-  const [expanded, setExpanded] = useState(false) // the fallback: filling the viewport in-page
-  const box = useRef<HTMLDivElement>(null)
-  const frame = useRef<HTMLIFrameElement>(null)
-  const key = machine ?? '' // the hub is a machine too, and its key is the empty one
-
-  const load = useCallback(async () => {
-    try { setKvm((await api.kvms()).find((k) => k.machine === key) ?? null) }
-    catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
-  }, [key])
-  useEffect(() => { load() }, [load])
-
-  // Fullscreen plus a keyboard lock is what makes Escape and Tab reach the machine rather
-  // than the browser. Both are undone together when fullscreen ends, however it ends.
-  useEffect(() => {
-    const onChange = () => {
-      const on = !!fullscreenElement()
-      setFull(on)
-      if (!on) keyboardAPI()?.unlock?.()
-    }
-    document.addEventListener('fullscreenchange', onChange)
-    document.addEventListener('webkitfullscreenchange', onChange)
-    return () => {
-      document.removeEventListener('fullscreenchange', onChange)
-      document.removeEventListener('webkitfullscreenchange', onChange)
-      keyboardAPI()?.unlock?.()
-    }
-  }, [])
-  // Expanding is in-page, so Escape is ours to handle.
-  useEscape(() => setExpanded(false), expanded)
-
-  const goBig = async () => {
-    if (!nativeFullscreen()) { setExpanded(true); frame.current?.focus(); return }
-    try {
-      if (box.current) await enterFullscreen(box.current)
-      await keyboardAPI()?.lock?.()
-      frame.current?.focus()
-    } catch {
-      setExpanded(true) // refused (an iframe policy, a permissions policy): fill the page instead
-      frame.current?.focus()
-    }
-  }
-  const goSmall = () => { setExpanded(false); if (fullscreenElement()) exitFullscreen() }
-
-  const remove = async () => {
-    if (!kvm || !await askYes(`Forget ${title}'s KVM? The device itself is not touched — this only removes its address.`, 'Remove')) return
-    try { await api.removeKVM(kvm.id); toast('removed'); await load() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
-  }
-
-  if (err && kvm === undefined) return <div className="dash"><div className="error">{err}</div></div>
-  if (kvm === undefined) return <div className="dash"><Loading label="Looking…" /></div>
-
-  const locks = !!keyboardAPI()?.lock
-  const big = full || expanded
-
-  return (
-    <div className="kvm">
-      <div className="kvm-bar">
-        <span className="dname">{title}</span>
-        {kvm
-          ? <span className="muted small mono kvm-url" title={kvm.url}>{kvm.name !== kvm.url ? `${kvm.name} · ` : ''}{kvm.url}</span>
-          : <span className="muted small">no KVM</span>}
-        <span className="spacer" />
-        {kvm && <>
-          <Btn className="btn" onClick={goBig} title={!nativeFullscreen() ? 'fill the screen — this browser has no true fullscreen for a page element' : locks ? 'fullscreen, and the browser stops keeping Escape, Tab and Ctrl-W for itself' : 'fullscreen — this browser will still keep some keys for itself'}>{nativeFullscreen() ? 'Fullscreen' : 'Expand'}</Btn>
-          <a className="btn" href={kvm.url} target="_blank" rel="noreferrer" title="the device directly, outside Ham Lab — only works on the LAN">Open ↗</a>
-          <Btn className="btn" onClick={() => setAdding(true)}>Change</Btn>
-          <Btn className="btn danger" onClick={remove}>Remove</Btn>
-        </>}
-        {!kvm && <Btn className="btn on" onClick={() => setAdding(true)}>+ Add a KVM for {title}</Btn>}
-      </div>
-
-      {kvm ? (
-        <div className={'kvm-frame' + (full ? ' full' : '') + (expanded ? ' expanded' : '')} ref={box}>
-          <iframe ref={frame} key={kvm.id} src={`/kvm/${kvm.id}/`} title={kvm.name}
-            allow="fullscreen; clipboard-read; clipboard-write" />
-          {expanded && <button className="btn kvm-shrink" onClick={goSmall} title="back to the console">Done</button>}
-        </div>
-      ) : (
-        <div className="svc-empty muted kvm-none">
-          <div>
-            <p>No KVM for <b>{title}</b>.</p>
-            <p className="small">Add one with its address and its page appears here. The address is kept on the hub,
-              so it still works when {title} does not — which is when you will want it.</p>
-          </div>
-        </div>
-      )}
-      {kvm && !big && (
-        <div className="kvm-hint muted small">
-          Click the picture, then type — the keys go to {title}.
-          {locks ? ' Fullscreen also takes Escape, Tab and Ctrl-W back from the browser.' : ' This browser keeps Escape, Tab and Ctrl-W for itself; use the device’s own on-screen keys for those, or a Chromium browser.'}
-        </div>
-      )}
-      {adding && <AddDialog machine={key} title={title} existing={kvm} onClose={() => setAdding(false)} onAdded={() => { setAdding(false); load() }} />}
-    </div>
-  )
-}
-
-function AddDialog({ machine, title, existing, onClose, onAdded }: { machine: string; title: string; existing: Device | null; onClose: () => void; onAdded: () => void }) {
-  const [url, setUrl] = useState(existing?.url ?? '')
-  const [name, setName] = useState(existing?.name ?? '')
-  const [err, setErr] = useState('')
-  const [busy, setBusy] = useState(false)
-  useEscape(onClose)
-  const save = async (e: FormEvent) => {
-    e.preventDefault(); setErr(''); setBusy(true)
-    try { await api.addKVM(name.trim(), url.trim(), machine); onAdded() }
-    catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setBusy(false) }
-  }
-  return (
-    <div className="modal-back" onClick={onClose}>
-      <form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
-        <h3>{existing ? `Change ${title}'s KVM` : `A KVM for ${title}`}</h3>
-        <label className="muted">Address — whatever you would type into a browser
-          <input autoFocus className="mono" value={url} onChange={(e) => setUrl(e.target.value)} placeholder="10.0.0.217" spellCheck={false} autoCapitalize="off" required />
-        </label>
-        <label className="muted">Name — optional; the address is used otherwise
-          <input value={name} onChange={(e) => setName(e.target.value)} placeholder={`${title} KVM`} maxLength={64} />
-        </label>
-        <div className="muted small">
-          Ham Lab serves the device from its own address, so a KVM that speaks only plain HTTP
-          still works here, and it is behind this login rather than needing a second one. The
-          address is kept on the hub, so it still reaches the device when {title} is down. The
-          device itself is not changed and is never reachable from outside.
-          {existing && <> Saving replaces the one there now — a machine has one KVM.</>}
-        </div>
-        {err && <div className="error">{err}</div>}
-        <div className="svc-actions">
-          <button type="submit" className="btn on" disabled={busy || !url.trim()}>{existing ? 'Save' : 'Add'}</button>
-          <span className="spacer" />
-          <button type="button" className="btn" onClick={onClose}>Cancel</button>
-        </div>
-      </form>
-    </div>
-  )
-}

web/src/Network.tsx

@@ -81,7 +81,7 @@ export default function Network() {
   // While a test is running the state is polled hard, because the needle should move; at rest
   // the page is a read and asks again only now and then.
   useEffect(() => {
-    const every = running ? Math.max(400, Math.min(rate, 1000)) : Math.max(5000, rate)
+    const every = running ? rate : Math.max(5000, rate)
     const t = setInterval(() => {
       if (document.visibilityState !== 'visible') return
       if (running) api.speedtest().then((r) => setD((old) => (old ? { ...old, run: r } : old))).catch(() => {})
@@ -93,7 +93,7 @@ export default function Network() {
   const phase = run?.phase
   useEffect(() => { if (phase === 'done') load() }, [phase, load])
 
-  const live = useSensorStream<{ n: Record<string, Sensor> }>({ n: 'net.*' }, { everyMs: Math.max(1000, rate) })
+  const live = useSensorStream<{ n: Record<string, Sensor> }>({ n: 'net.*' }, { everyMs: rate })
   const sens = live.data?.n ?? {}
   const rateOf = (name: string, dir: 'rx' | 'tx') => sens[`net.${name.toLowerCase().replace(/[^a-z0-9_-]+/g, '_')}.${dir}`]?.value
 

web/src/Player.tsx

@@ -1,153 +1,416 @@
-// A video in the Files viewer. First the file as it is, handed to the browser's own
-// decoder; when that fails — Safari and an MKV, HEVC without a hardware decoder — a
-// picker: codec and size, and the machine re-encodes it to HLS on its own GPU (or CPU),
-// which plays natively on Safari and through hls.js everywhere else. The same picker is
-// one tap away even when the file plays, for a 36 Mbps capture over a slow link.
-// A re-encode's timeline is the whole film from the start: seek anywhere, and the machine
-// starts encoding from there (repackaging, which is near-instant, grows as it goes instead).
+// A video in the Files viewer, YouTube's way: Auto, which reads the connection and picks the
+// quality (hls.js), any quality by hand, or Original — the file as it is, for a browser that
+// decodes it. The qualities are made on the fly, from wherever the viewer seeks to, on the device
+// the Transcode device setting names (Settings), which may be on another machine than the file.
+// A 360° video is drawn as the view from inside it (vr.ts): drag, pinch or scroll, and on a
+// phone the motion button. The same player as Ham Share's.
 import { useCallback, useEffect, useRef, useState } from 'react'
-import { api, apiPath, PlayCaps, PlayStatus, PlayStream } from './api'
-import { Btn, Spinner } from './ui'
+import Hls from 'hls.js'
+import { api, nodeAPI, type PlayOpen, type PlayStatus } from './api'
+import { canMotion, Sphere } from './vr'
 
-type Mode = 'native' | 'pick' | 'hls'
-const CODEC_KEY = 'hamlab.play.codec'
-const QUALITY_KEY = 'hamlab.play.quality'
-const remember = (k: string, v: string) => { try { localStorage.setItem(k, v) } catch { /* private mode */ } }
-const recall = (k: string, d: string) => { try { return localStorage.getItem(k) || d } catch { return d } }
+type Choice = 'auto' | 'original' | number // a number is a width: the rung's name
 
-const clock = (sec: number) => { const t = Math.floor(sec), h = Math.floor(t / 3600), m = Math.floor((t % 3600) / 60), x = t % 60; return `${h ? `${h}:${String(m).padStart(2, '0')}` : m}:${String(x).padStart(2, '0')}` }
-const fmtMbps = (bps?: number) => (bps ? `${(bps / 1e6).toFixed(bps >= 10e6 ? 0 : 1)} Mbps` : '')
-const describe = (s: PlayStream) => {
-  const v = s.source.video
-  const src = v ? `${v.codec}${v.profile ? ' ' + v.profile : ''} ${v.width}×${v.height}${v.fps ? ` ${Math.round(v.fps)} fps` : ''}${s.source.bit_rate ? ` ${fmtMbps(s.source.bit_rate)}` : ''} in ${s.source.container.split(',')[0]}` : s.source.container
-  const out = s.codec === 'copy' ? 'repackaged' : `${s.codec === 'h264' ? 'H.264' : s.codec.toUpperCase()} ${s.width}×${s.height} ${(s.kbps / 1000).toFixed(s.kbps >= 10000 ? 0 : 1)} Mbps`
-  return `${src} → ${out} on ${s.engine}`
+// The bandwidth hls.js measured last time, so the next video starts near the right quality
+// rather than at the bottom.
+const BW_KEY = 'hamlab.play.bw'
+const savedBW = () => { try { const n = Number(localStorage.getItem(BW_KEY)); return n > 0 ? n : 0 } catch { return 0 } }
+const saveBW = (n: number) => { try { if (n > 0) localStorage.setItem(BW_KEY, String(Math.round(n))) } catch { /* private mode */ } }
+
+// hls.js reads the connection from how long each piece takes. Here a piece can be slow to
+// start because the machine is still making it (a few seconds after a seek or a change of
+// quality, at 8K), which says nothing about the connection; counted, it drags Auto down to the
+// bottom, and every drop starts the making over again. So: the wait before the first byte is
+// not counted against the connection, and hls.js doesn't give up on a piece before its first
+// byte comes. Once bytes flow, its usual rules apply.
+type AbrLike = { new(hls: Hls): object }
+const Base = Hls.DefaultConfig.abrController as unknown as AbrLike
+type Stats = { loading: { start: number; first: number } }
+type Proto = { initEstimator: () => { sampleTTFB: (ms: number) => void }; onFragBuffered: (ev: unknown, d: unknown) => void }
+const proto = Base.prototype as unknown as Proto
+class PatientAbr extends Base {
+  constructor(hls: Hls) {
+    super(hls)
+    const self = this as unknown as { _abandonRulesCheck: (l?: unknown) => void; fragCurrent: { stats: Stats } | null }
+    const orig = self._abandonRulesCheck
+    self._abandonRulesCheck = (l?: unknown) => {
+      const f = self.fragCurrent
+      if (f && !f.stats.loading.first) return
+      orig(l)
+    }
+  }
+  initEstimator() {
+    const e = proto.initEstimator.call(this)
+    const sample = e.sampleTTFB.bind(e)
+    e.sampleTTFB = (ms: number) => sample(Math.min(ms, 300))
+    return e
+  }
+  onFragBuffered(ev: unknown, d: { frag: { stats: Stats }; part?: { stats: Stats & { loaded: number } } | null }) {
+    const st = d.part?.stats.loaded ? d.part.stats : d.frag.stats
+    const start = st.loading.start
+    if (st.loading.first > start) st.loading.start = st.loading.first
+    try { proto.onFragBuffered.call(this, ev, d) } finally { st.loading.start = start }
+  }
 }
 
-export default function Player({ path, sudo, name }: { path: string; sudo: boolean; name: string }) {
-  const [mode, setMode] = useState<Mode>('native')
-  const [why, setWhy] = useState('') // what brought the picker up
-  const [caps, setCaps] = useState<PlayCaps | null>(null)
-  const [codec, setCodec] = useState(() => recall(CODEC_KEY, 'h264'))
-  const [quality, setQuality] = useState(() => recall(QUALITY_KEY, '1080p'))
-  const [stream, setStream] = useState<{ s: PlayStream; url: string } | null>(null)
-  const [status, setStatus] = useState<PlayStatus | null>(null)
+// Whether the browser lets a page set the volume level (iOS doesn't: only mute).
+const canLevel = (() => { try { const a = document.createElement('audio'); a.volume = 0.5; return a.volume === 0.5 } catch { return false } })()
+
+const mbps = (b: number) => `${b >= 10e6 ? Math.round(b / 1e6) : (b / 1e6).toFixed(1)} Mb/s`
+const clock = (sec: number) => { const t = Math.max(0, Math.floor(sec)), h = Math.floor(t / 3600), m = Math.floor((t % 3600) / 60), x = t % 60; return `${h ? `${h}:${String(m).padStart(2, '0')}` : m}:${String(x).padStart(2, '0')}` }
+const size = (w?: number, h?: number, vr?: boolean) => {
+  if (!w || !h) return ''
+  if (vr) return `${w >= 7680 ? '8K' : w >= 5760 ? '5.7K' : w >= 3840 ? '4K' : w >= 3072 ? '3K' : `${w}×${h}`} 360°`
+  const s = Math.min(w, h)
+  return s >= 2160 ? '4K' : s >= 1440 ? '1440p' : s >= 1080 ? '1080p' : s >= 720 ? '720p' : `${w}×${h}`
+}
+
+const I = ({ d, size: sz = 20, fill }: { d: string; size?: number; fill?: boolean }) => (
+  <svg width={sz} height={sz} viewBox="0 0 24 24" fill={fill ? 'currentColor' : 'none'} stroke={fill ? 'none' : 'currentColor'} strokeWidth="1.8" strokeLinecap="round" strokeLinejoin="round" aria-hidden="true"><path d={d} /></svg>
+)
+const PLAY = 'M7 4.5v15l12.5-7.5z', PAUSE = 'M8 5v14M16 5v14', CHECK = 'M5 12.5l4.5 4.5L19 7.5', TUNE = 'M4 7h10M18 7h2M4 17h4M12 17h8M16 5v4M10 15v4'
+const FULL = 'M4 9V4h5M20 9V4h-5M4 15v5h5M20 15v5h-5', SMALL = 'M9 4v5H4M15 4v5h5M9 20v-5H4M15 20v-5h5', PHONE = 'M8 3h8a1 1 0 0 1 1 1v16a1 1 0 0 1-1 1H8a1 1 0 0 1-1-1V4a1 1 0 0 1 1-1zM3 9a9 9 0 0 0 0 6M21 9a9 9 0 0 1 0 6'
+
+// The speaker: waves by the level, a cross when muted.
+const Speaker = ({ off, low }: { off?: boolean; low?: boolean }) => (
+  <svg width={20} height={20} viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="1.8" strokeLinecap="round" strokeLinejoin="round" aria-hidden="true">
+    <path d="M4 9.5h3.5L12 5.5v13l-4.5-4H4z" fill="currentColor" />
+    {off ? <path d="M16 9.5l5 5M21 9.5l-5 5" /> : <path d={low ? 'M15.5 9.5a3.5 3.5 0 0 1 0 5' : 'M15.5 9.5a3.5 3.5 0 0 1 0 5M18 7a7 7 0 0 1 0 10'} />}
+  </svg>
+)
+
+// Five seconds back or on: a circling arrow with the 5 inside.
+const Skip = ({ back }: { back?: boolean }) => (
+  <svg width={20} height={20} viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="1.8" strokeLinecap="round" strokeLinejoin="round" aria-hidden="true">
+    <path d={back ? 'M12 5a7 7 0 1 1-7 7M12 5l2.5-2.5M12 5l2.5 2.5' : 'M12 5a7 7 0 1 0 7 7M12 5L9.5 2.5M12 5l-2.5 2.5'} />
+    <text x="12" y="15" textAnchor="middle" fontSize="7.5" fontWeight="700" fill="currentColor" stroke="none" fontFamily="system-ui, sans-serif">5</text>
+  </svg>
+)
+
+export default function Player({ path, sudo }: { path: string; sudo: boolean; name: string }) {
+  const [open, setOpen] = useState<PlayOpen | null>(null)
   const [err, setErr] = useState('')
+  const [note, setNote] = useState('')
+  const [choice, setChoice] = useState<Choice>('auto')
+  const [showing, setShowing] = useState(0) // the width playing now
+  const [status, setStatus] = useState<PlayStatus | null>(null)
+  const [paused, setPaused] = useState(true)
+  const [waiting, setWaiting] = useState(true)
+  const [time, setTime] = useState(0)
+  const [scrub, setScrub] = useState<number | null>(null)
+  const [buffered, setBuffered] = useState(0)
+  const [menu, setMenu] = useState(false)
+  const [chrome, setChrome] = useState(true)
+  const [motion, setMotion] = useState(false)
+  const [full, setFull] = useState(false) // the browser's full screen, or (an iPhone) the window
+  const box = useRef<HTMLDivElement>(null)
   const video = useRef<HTMLVideoElement>(null)
-  const streamRef = useRef(stream)
-  streamRef.current = stream
+  const canvas = useRef<HTMLCanvasElement>(null)
+  const hls = useRef<Hls | null>(null)
+  const sphere = useRef<Sphere | null>(null)
+  const hideTimer = useRef<number>()
+  const resume = useRef<{ at: number; play: boolean }>({ at: 0, play: true })
+  const openRef = useRef(open)
+  openRef.current = open
+  const menuRef = useRef(menu)
+  menuRef.current = menu
 
-  useEffect(() => { setMode('native'); setWhy(''); setStream(null); setStatus(null); setErr('') }, [path, sudo])
-  useEffect(() => { remember(CODEC_KEY, codec) }, [codec])
-  useEffect(() => { remember(QUALITY_KEY, quality) }, [quality])
-  useEffect(() => { if (mode === 'pick' && !caps) api.playCaps(sudo).then(setCaps).catch((e) => setCaps({ codecs: [], qualities: [], error: e instanceof Error ? e.message : 'failed' })) }, [mode, caps, sudo])
-
-  // a stream is the machine's work: let go of it when this player does — and when the page
-  // itself goes, by a beacon, the one request a browser still sends on its way out
-  const stop = useCallback(() => { const s = streamRef.current; if (s) { api.playStop(s.s.id).catch(() => { /* gone already */ }); setStream(null); setStatus(null) } }, [])
-  useEffect(() => () => stop(), [stop])
+  // Open the play; let it go when this player does, and when the page goes, by a beacon.
   useEffect(() => {
-    const h = () => { const s = streamRef.current; if (s) navigator.sendBeacon(apiPath(`/api/files/play/${s.s.id}/close`)) }
+    let live = true
+    setOpen(null); setErr(''); setNote(''); setChoice('auto'); resume.current = { at: 0, play: true }
+    api.play(path, sudo).then((o) => {
+      if (!live) { api.playStop(o.node, o.play.id).catch(() => {}); return }
+      setOpen(o)
+      if (o.note) setNote(o.note)
+    }).catch((e) => { if (live) setErr(e instanceof Error ? e.message : 'failed') })
+    return () => { live = false; const o = openRef.current; if (o) api.playStop(o.node, o.play.id).catch(() => { /* gone already */ }) }
+  }, [path, sudo])
+  useEffect(() => {
+    const h = () => { const o = openRef.current; if (o) navigator.sendBeacon(nodeAPI(o.node, `/api/files/play/${o.play.id}/close`)) }
     window.addEventListener('pagehide', h)
     return () => window.removeEventListener('pagehide', h)
   }, [])
 
-  const start = async () => {
-    setErr('')
-    try {
-      stop()
-      const r = await api.playStart(path, codec, quality, sudo)
-      setStream({ s: r.stream, url: r.url }); setMode('hls')
-    } catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
-  }
+  const info = open?.play.info
+  const vr = !!info?.video?.spherical
 
-  // feed the stream to the element: Safari plays HLS itself, the rest through hls.js
+  useEffect(() => {
+    if (!vr || !canvas.current || !video.current) return
+    try { sphere.current = new Sphere(canvas.current, video.current) } catch (e) { setErr((e as Error).message) }
+    return () => { sphere.current?.destroy(); sphere.current = null }
+  }, [vr, open])
+
+  // The source: hls.js for Auto and the chosen qualities, the file itself for Original.
   useEffect(() => {
     const v = video.current
-    if (mode !== 'hls' || !stream || !v) return
-    const url = apiPath(stream.url)
-    let hls: { destroy: () => void } | null = null
-    let gone = false
-    ;(async () => {
-      if (v.canPlayType('application/vnd.apple.mpegurl')) {
-        v.src = url
-      } else {
-        const { default: Hls } = await import('hls.js')
-        if (gone) return
-        if (!Hls.isSupported()) { setErr('this browser has neither HLS nor MediaSource — nothing can play here'); return }
-        // a segment the machine hasn't made yet is made when asked for (a seek): its first
-        // byte can take as long as the re-encode takes to start there and write it
-        const slow = { maxTimeToFirstByteMs: 50000, maxLoadTimeMs: 120000, timeoutRetry: { maxNumRetry: 2, retryDelayMs: 0, maxRetryDelayMs: 0 }, errorRetry: { maxNumRetry: 3, retryDelayMs: 1000, maxRetryDelayMs: 8000 } }
-        const h = new Hls({ maxBufferLength: 60, backBufferLength: 30, fragLoadPolicy: { default: slow } })
-        hls = h
-        h.on(Hls.Events.ERROR, (_e, d) => { if (d.fatal) setErr(`playback failed: ${d.details}`) })
-        h.loadSource(url)
-        h.attachMedia(v)
+    if (!open || !v) return
+    const { at, play } = resume.current
+    const start = () => { if (at) v.currentTime = at; if (play) v.play().catch(() => setPaused(true)) }
+    if (choice === 'original') {
+      v.src = api.fileReadURL(path, sudo)
+      v.addEventListener('loadedmetadata', start, { once: true })
+      setShowing(info?.video?.width ?? 0)
+      const bad = () => {
+        resume.current = { at: v.currentTime || at, play: true }
+        setNote('This browser can’t play the file as it is. Back on Auto.')
+        setChoice('auto')
       }
-      v.play().catch(() => { /* needs a tap on some phones */ })
-    })()
-    return () => { gone = true; hls?.destroy() }
-  }, [mode, stream])
+      v.addEventListener('error', bad, { once: true })
+      return () => { v.removeEventListener('error', bad); v.removeEventListener('loadedmetadata', start); v.removeAttribute('src'); v.load() }
+    }
+    const master = nodeAPI(open.node, open.master)
+    if (Hls.isSupported()) {
+      const h = new Hls({
+        abrController: PatientAbr as unknown as typeof Hls.DefaultConfig.abrController,
+        abrEwmaDefaultEstimate: savedBW() || 5_000_000,
+        startPosition: at || -1,
+        maxBufferLength: 30,
+        maxMaxBufferLength: 60,
+        backBufferLength: 60,
+        // A piece being made after a seek can take a while; the machine waits up to 45 s.
+        fragLoadPolicy: { default: { maxTimeToFirstByteMs: 50_000, maxLoadTimeMs: 120_000, timeoutRetry: { maxNumRetry: 2, retryDelayMs: 0, maxRetryDelayMs: 0 }, errorRetry: { maxNumRetry: 4, retryDelayMs: 1000, maxRetryDelayMs: 8000 } } },
+      })
+      hls.current = h
+      h.on(Hls.Events.MANIFEST_PARSED, () => {
+        if (typeof choice === 'number') {
+          const i = h.levels.findIndex((l) => l.width === choice)
+          if (i >= 0) { h.startLevel = i; h.currentLevel = i }
+        }
+        if (play) v.play().catch(() => setPaused(true))
+      })
+      h.on(Hls.Events.LEVEL_SWITCHED, (_, d) => setShowing(h.levels[d.level]?.width ?? 0))
+      h.on(Hls.Events.FRAG_BUFFERED, () => saveBW(h.bandwidthEstimate))
+      let recovered = 0
+      h.on(Hls.Events.ERROR, (_, d) => {
+        if (!d.fatal) return
+        if (d.type === Hls.ErrorTypes.MEDIA_ERROR && recovered++ < 3) { h.recoverMediaError(); return }
+        if (d.type === Hls.ErrorTypes.NETWORK_ERROR && recovered++ < 3) { h.startLoad(v.currentTime); return }
+        setErr(`The video stopped loading (${d.details}).`)
+      })
+      h.loadSource(master)
+      h.attachMedia(v)
+      return () => { saveBW(h.bandwidthEstimate); h.destroy(); hls.current = null }
+    }
+    // Safari without Media Source (an older iPhone): its own HLS, Auto only.
+    v.src = master
+    v.addEventListener('loadedmetadata', start, { once: true })
+    return () => { v.removeEventListener('loadedmetadata', start); v.removeAttribute('src'); v.load() }
+  }, [open, choice === 'original' ? 'o' : 'h']) // eslint-disable-line react-hooks/exhaustive-deps
 
-  // follow the re-encode while it runs
+  // A hand-picked quality, or back to Auto, while hls.js is already playing.
   useEffect(() => {
-    if (mode !== 'hls' || !stream) return
-    let stopPolling = false
-    const tick = async () => { try { const st = await api.playStatus(stream.s.id); if (!stopPolling) { setStatus(st); if (st.running) setTimeout(tick, 2000) } } catch { /* ended */ } }
-    tick()
-    return () => { stopPolling = true }
-  }, [mode, stream])
+    const h = hls.current
+    if (!h || choice === 'original' || !h.levels.length) return
+    if (choice === 'auto') { h.currentLevel = -1; return }
+    const i = h.levels.findIndex((l) => l.width === choice)
+    if (i >= 0) h.currentLevel = i
+  }, [choice])
 
-  const pick = (reason: string) => { setWhy(reason); setMode('pick') }
-  const codecs = caps?.codecs ?? []
+  // What the machine is doing, while the menu is open.
+  useEffect(() => {
+    if (!menu || !open) return
+    let live = true
+    const tick = () => api.playStatus(open.node, open.play.id).then((s) => { if (live) setStatus(s) }).catch(() => {})
+    tick()
+    const t = window.setInterval(tick, 2000)
+    return () => { live = false; window.clearInterval(t) }
+  }, [menu, open])
+
+  // Controls fade while it plays and nobody touches anything.
+  const poke = useCallback(() => {
+    setChrome(true)
+    window.clearTimeout(hideTimer.current)
+    hideTimer.current = window.setTimeout(() => { if (!video.current?.paused && !menuRef.current) setChrome(false) }, 3000)
+  }, [])
+  useEffect(() => () => window.clearTimeout(hideTimer.current), [])
+  useEffect(() => { if (!menu) poke() }, [menu, poke])
+
+  const toggle = useCallback(() => {
+    const v = video.current
+    if (!v) return
+    if (v.paused) void v.play().catch(() => {})
+    else v.pause()
+    poke()
+  }, [poke])
+  // The seek bar: while it is dragged the ball follows the finger (the input event); the seek
+  // itself is on release (the browser's own change event, which a click, a drag and the arrow
+  // keys all end with). The ball is put where the seek went at once, so it waits there while
+  // the piece is made rather than going back to where the film was.
+  const seekBar = useRef<HTMLInputElement>(null)
+  useEffect(() => {
+    const el = seekBar.current
+    if (!el) return
+    const commit = () => {
+      const x = Number(el.value), v = video.current
+      if (v && Math.abs(v.currentTime - x) > 0.05) v.currentTime = x
+      setTime(x)
+      setScrub(null)
+    }
+    el.addEventListener('change', commit)
+    return () => el.removeEventListener('change', commit)
+  })
+  // Volume: a level and mute, remembered for the next video. iOS lets a page mute but not set
+  // the level (it reads back as 1), so there the slider isn't shown.
+  const [vol, setVol] = useState(() => { try { const v = JSON.parse(localStorage.getItem('hamlab.play.volume') ?? ''); return { level: Math.min(1, Math.max(0, Number(v.level) || 0)), muted: !!v.muted } } catch { return { level: 1, muted: false } } })
+  useEffect(() => {
+    const v = video.current
+    if (v) { v.volume = vol.level; v.muted = vol.muted || vol.level === 0 }
+    try { localStorage.setItem('hamlab.play.volume', JSON.stringify(vol)) } catch { /* private mode */ }
+  }, [vol, open])
+  const volBy = useCallback((d: number) => setVol((x) => ({ level: Math.round(Math.min(1, Math.max(0, x.level + d)) * 20) / 20, muted: false })), [])
+  const muteToggle = useCallback(() => setVol((x) => (x.muted || x.level === 0 ? { level: x.level || 0.5, muted: false } : { ...x, muted: true })), [])
+
+  const seekBy = useCallback((d: number) => {
+    const v = video.current
+    if (!v) return
+    const to = Math.max(0, Math.min(info?.duration ?? v.duration, v.currentTime + d))
+    v.currentTime = to
+    setTime(to) // the ball goes there at once, made yet or not
+    poke()
+  }, [info, poke])
+  // Full screen: the browser's where it has one for an element, else (an iPhone) the window.
+  const fullscreen = useCallback(() => {
+    const el = box.current as (HTMLDivElement & { webkitRequestFullscreen?: () => void }) | null
+    if (!el) return
+    if (document.fullscreenElement) { void document.exitFullscreen(); return }
+    if (el.requestFullscreen) { void el.requestFullscreen().catch(() => setFull((f) => !f)); return }
+    if (el.webkitRequestFullscreen) { el.webkitRequestFullscreen(); return }
+    setFull((f) => !f)
+  }, [])
+  useEffect(() => {
+    const f = () => setFull(!!document.fullscreenElement)
+    document.addEventListener('fullscreenchange', f)
+    return () => document.removeEventListener('fullscreenchange', f)
+  }, [])
+
+  // Keys work while the player has the focus (a click on it gives it).
+  const onKey = (e: React.KeyboardEvent) => {
+    if (e.key === ' ' || e.key === 'k') { e.preventDefault(); toggle() }
+    else if (e.key === 'ArrowLeft' || e.key === 'j') { e.preventDefault(); seekBy(e.key === 'j' ? -10 : -5) }
+    else if (e.key === 'ArrowRight' || e.key === 'l') { e.preventDefault(); seekBy(e.key === 'l' ? 10 : 5) }
+    else if (e.key === 'f') fullscreen()
+    else if (e.key === 'm') muteToggle()
+    else if (e.key === 'ArrowUp') { e.preventDefault(); volBy(0.1) }
+    else if (e.key === 'ArrowDown') { e.preventDefault(); volBy(-0.1) }
+    else if (e.key === 'Escape') { if (menu) setMenu(false); else if (full && !document.fullscreenElement) setFull(false) }
+  }
+
+  // Looking around a 360° video: a drag turns the view, a pinch zooms, a tap shows or hides the
+  // controls.
+  const pointers = useRef(new Map<number, { x: number; y: number }>())
+  const moved = useRef(0)
+  const pinch = useRef(0)
+  const onDown = (e: React.PointerEvent) => {
+    (e.target as Element).setPointerCapture(e.pointerId)
+    pointers.current.set(e.pointerId, { x: e.clientX, y: e.clientY })
+    moved.current = 0
+    if (pointers.current.size === 2) { const [a, b] = [...pointers.current.values()]; pinch.current = Math.hypot(a.x - b.x, a.y - b.y) }
+  }
+  const onMove = (e: React.PointerEvent) => {
+    const p = pointers.current.get(e.pointerId)
+    if (!p) return
+    const dx = e.clientX - p.x, dy = e.clientY - p.y
+    pointers.current.set(e.pointerId, { x: e.clientX, y: e.clientY })
+    moved.current += Math.abs(dx) + Math.abs(dy)
+    if (pointers.current.size === 2) {
+      const [a, b] = [...pointers.current.values()]
+      const d = Math.hypot(a.x - b.x, a.y - b.y)
+      if (pinch.current) sphere.current?.zoom(pinch.current / d)
+      pinch.current = d
+    } else sphere.current?.drag(dx, dy)
+  }
+  const onUp = (e: React.PointerEvent) => {
+    pointers.current.delete(e.pointerId)
+    if (pointers.current.size < 2) pinch.current = 0
+    if (pointers.current.size === 0 && moved.current < 6) {
+      if (e.pointerType === 'mouse') toggle()
+      else if (chrome && !menu) setChrome(false)
+      else poke()
+      setMenu(false)
+    }
+  }
+
+  const v = video.current
+  const dur = info?.duration ?? 0
+  const shown = scrub ?? time
+  const rungs = open?.play.rungs ?? []
+  const label = (w: number) => rungs.find((r) => r.width === w)?.label ?? (w && w === info?.video?.width ? 'Original' : '')
+  const choiceLabel = choice === 'auto' ? `Auto${showing ? ` · ${label(showing)}` : ''}` : choice === 'original' ? 'Original' : label(choice)
+  const tooBig = vr && sphere.current && (info?.video?.width ?? 0) > sphere.current.maxTexture
+  const fileRate = info?.bit_rate || (info?.size && dur ? (info.size * 8) / dur : 0)
+  const keep = () => { if (v) resume.current = { at: v.currentTime, play: !v.paused } }
 
   return (
-    <div className="player">
-      {mode === 'native' && (
-        <>
-          <div className="play-bar"><span>as it is</span><span className="spacer" /><button className="btn" onClick={() => pick('')}>Re-encode…</button></div>
-          <div className="preview"><video controls playsInline preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => pick(`This browser can't play ${name} as it is — the container or codec isn't one it decodes.`)} /></div>
-        </>
-      )}
-      {mode === 'pick' && (
-        <div className="play-pick notice">
-          <div>{why || `Re-encode ${name} on this machine and play that instead.`}</div>
-          {caps?.error && <div className="error" style={{ marginTop: 8 }}>{caps.error}</div>}
-          {!caps && !err && <div className="muted small" style={{ marginTop: 8 }}><Spinner /> asking the machine what it can encode…</div>}
-          {caps && !caps.error && (
-            <div className="row">
-              <label>codec
-                <select value={codec} onChange={(e) => setCodec(e.target.value)}>
-                  {codecs.map((c) => <option key={c.id} value={c.id} disabled={!c.engine}>{c.label}{c.engine ? ` — ${c.engine}` : ' — not on this machine'}</option>)}
-                </select>
-              </label>
-              <label>size
-                <select value={quality} onChange={(e) => setQuality(e.target.value)} disabled={codec === 'copy'}>
-                  {caps.qualities.map((q) => <option key={q.id} value={q.id}>{q.label}</option>)}
-                </select>
-              </label>
-              <Btn className="btn on" disabled={!codecs.find((c) => c.id === codec)?.engine} onClick={start}>Play</Btn>
-              <a className="btn" href={api.fileDownloadURL(path, sudo)} download={name}>Download instead</a>
+    <div ref={box} tabIndex={0} onKeyDown={onKey} className={`vp${chrome ? '' : ' bare'}${full ? ' max' : ''}`}
+      onPointerMove={(e) => { if (e.pointerType === 'mouse') poke() }}
+      onWheel={vr ? (e) => { if (!(e.target as Element).closest('.vp-menu')) sphere.current?.zoom(e.deltaY > 0 ? 1.08 : 1 / 1.08) } : undefined}>
+      <video ref={video} className={vr ? 'vp-v vp-hidden' : 'vp-v'} playsInline preload="auto"
+        onPlay={() => { setPaused(false); poke() }} onPause={() => { setPaused(true); setChrome(true) }}
+        onWaiting={() => setWaiting(true)} onPlaying={() => setWaiting(false)} onCanPlay={() => setWaiting(false)} onSeeking={() => setWaiting(true)} onSeeked={() => setWaiting(false)}
+        onTimeUpdate={(e) => {
+          const el = e.currentTarget
+          setTime(el.currentTime)
+          resume.current = { at: el.currentTime, play: !el.paused }
+          for (let i = 0; i < el.buffered.length; i++) if (el.buffered.start(i) <= el.currentTime + 0.5) setBuffered(el.buffered.end(i))
+        }}
+        onClick={vr ? undefined : () => { if (matchMedia('(pointer: coarse)').matches) { if (chrome) setChrome(false); else poke() } else toggle() }}
+        onDoubleClick={vr ? undefined : fullscreen} />
+      {vr && <canvas ref={canvas} className="vp-v vp-sphere" onPointerDown={onDown} onPointerMove={onMove} onPointerUp={onUp} onPointerCancel={onUp} onDoubleClick={fullscreen} />}
+
+      {(waiting || !open) && !err && <div className="vp-spin" aria-label="Loading" />}
+      {err && <div className="vp-err"><p>{err}</p></div>}
+      {open && paused && !waiting && !err && <button className="vp-big" onClick={toggle} aria-label="Play"><I d={PLAY} size={36} fill /></button>}
+      {note && <div className="vp-note" role="status" onClick={() => setNote('')}>{note}</div>}
+      {vr && open && chrome && !menu && time < 8 && !err && <div className="vp-hint">Drag to look around · scroll or pinch to zoom</div>}
+
+      <div className="vp-bottom">
+        <span className="vp-time above">{clock(shown)} / {clock(dur)}</span>
+        <div className="vp-seek" style={{ ['--p' as string]: dur ? `${(shown / dur) * 100}%` : '0%', ['--b' as string]: dur ? `${(buffered / dur) * 100}%` : '0%' }}>
+          <input ref={seekBar} type="range" min={0} max={dur || 1} step={0.1} value={shown} aria-label="Seek"
+            onChange={(e) => { setScrub(Number(e.target.value)); poke() }} />
         </div>
-          )}
-          {err && <div className="error" style={{ marginTop: 8 }}>{err}</div>}
-        </div>
-      )}
-      {mode === 'hls' && stream && (
-        <>
-          <div className="play-bar">
-            <span title={describe(stream.s)}>{describe(stream.s)}</span>
-            {status && status.running && (stream.s.seekable
-              ? <span>· {status.speed ? `${status.speed.toFixed(1)}× ` : ''}from {clock(status.from ?? 0)} · {Math.min(100, Math.round(status.progress / stream.s.source.duration * 100))}% ready — seek anywhere</span>
-              : <span>· {status.speed ? `${status.speed.toFixed(1)}×` : ''} {stream.s.source.duration ? `${Math.min(100, Math.round(status.progress / stream.s.source.duration * 100))}% through` : `${Math.round(status.progress)} s done`} — seek within what's done</span>)}
-            {status?.done && <span>· all of it is ready</span>}
-            {status?.error && <span className="error">· stopped early: {status.error}</span>}
+        <div className="vp-row">
+          <button className="vp-icon" onClick={toggle} aria-label={paused ? 'Play' : 'Pause'}>{paused ? <I d={PLAY} fill /> : <I d={PAUSE} />}</button>
+          <button className="vp-icon" onClick={() => seekBy(-5)} aria-label="Back 5 seconds" title="Back 5 seconds"><Skip back /></button>
+          <button className="vp-icon" onClick={() => seekBy(5)} aria-label="On 5 seconds" title="On 5 seconds"><Skip /></button>
+          <span className="vp-vol">
+            <button className="vp-icon" onClick={muteToggle} aria-label={vol.muted || vol.level === 0 ? 'Unmute' : 'Mute'} title={vol.muted || vol.level === 0 ? 'Unmute (m)' : 'Mute (m)'}><Speaker off={vol.muted || vol.level === 0} low={vol.level < 0.5} /></button>
+            {canLevel && <input type="range" min={0} max={1} step={0.05} value={vol.muted ? 0 : vol.level} aria-label="Volume"
+              style={{ ['--v' as string]: `${(vol.muted ? 0 : vol.level) * 100}%` }}
+              onChange={(e) => { const level = Number(e.target.value); setVol({ level, muted: level === 0 }); poke() }} />}
+          </span>
+          <span className="vp-time inrow">{clock(shown)} / {clock(dur)}</span>
           <span className="spacer" />
-            <button className="btn" onClick={() => { stop(); setMode('pick'); setWhy('') }}>Change…</button>
+          {vr && canMotion() && (
+            <button className={`vp-icon${motion ? ' on' : ''}`} aria-label="Look around by moving the phone" aria-pressed={motion}
+              onClick={async () => { const s = sphere.current; if (s) setMotion(await s.setGyro(!s.gyroOn)) }}><I d={PHONE} /></button>
+          )}
+          <button className="vp-q" onClick={() => { setMenu((m) => !m); poke() }} aria-label="Quality" aria-expanded={menu}><I d={TUNE} size={16} /><span>{choiceLabel}</span></button>
+          <button className="vp-icon" onClick={fullscreen} aria-label={full ? 'Exit full screen' : 'Full screen'}><I d={full ? SMALL : FULL} /></button>
+        </div>
+      </div>
+
+      {menu && open && (
+        <div className="vp-menu" role="menu">
+          <div className="vp-menu-h">Quality</div>
+          <button role="menuitemradio" aria-checked={choice === 'auto'} onClick={() => { setChoice('auto'); setMenu(false) }}>
+            <span className="ck">{choice === 'auto' && <I d={CHECK} size={16} />}</span><span>Auto</span><small>{choice === 'auto' && showing ? label(showing) : 'fits the connection'}</small>
+          </button>
+          <button role="menuitemradio" aria-checked={choice === 'original'} disabled={!!tooBig} onClick={() => { keep(); setChoice('original'); setMenu(false) }}>
+            <span className="ck">{choice === 'original' && <I d={CHECK} size={16} />}</span><span>Original</span>
+            <small>{tooBig ? 'too big for this device' : [size(info?.video?.width, info?.video?.height, vr), fileRate ? mbps(fileRate) : ''].filter(Boolean).join(' · ')}</small>
+          </button>
+          {rungs.map((r) => (
+            <button key={r.name} role="menuitemradio" aria-checked={choice === r.width} onClick={() => { keep(); setChoice(r.width); setMenu(false) }}>
+              <span className="ck">{choice === r.width && <I d={CHECK} size={16} />}</span><span>{r.label}</span><small>{mbps(r.bitrate)}</small>
+            </button>
+          ))}
+          <div className="vp-menu-f">
+            Made on <b>{open.host}</b> · {status?.engine || open.engine}
+            {status?.engine && <> · {status.paused ? 'far enough ahead, waiting' : `from ${clock(status.from)}`}</>}
+          </div>
         </div>
-          <div className="preview"><video ref={video} controls autoPlay playsInline /></div>
-          {err && <div className="error" style={{ margin: 12 }}>{err}</div>}
-        </>
       )}
     </div>
   )

web/src/Settings.tsx

@@ -1,6 +1,6 @@
 import { useCallback, useEffect, useState } from 'react'
 import { fetchCatalog } from './sensors'
-import { api, Me, TokenInfo, UpdateStatus } from './api'
+import { api, Me, TokenInfo, TranscodeSetting, UpdateStatus } from './api'
 import { savePrefs } from './accents'
 import { applyTheme, currentThemeId, THEMES } from './themes'
 import { Btn, Loading, Spinner, askYes, toast, useEscape } from './ui'
@@ -104,6 +104,7 @@ export default function Settings({ me }: { me: Me }) {
         </div>
       </section>
 
+      <TranscodeDevice />
 
       <div className="dash-bar"><h2 className="section">Access tokens</h2></div>
       <section className="card">
@@ -248,3 +249,60 @@ function TokenDialog({ onClose, onMade }: { onClose: () => void; onMade: (t: { n
     </div>
   )
 }
+
+// Where videos played in Files are made: one device on one machine, for every machine's files.
+// Every machine's devices are asked for live; one that can't be used says why.
+function TranscodeDevice() {
+  const [t, setT] = useState<TranscodeSetting | null>(null)
+  const [err, setErr] = useState('')
+  const [busy, setBusy] = useState('')
+  useEffect(() => { api.transcode().then(setT).catch((e) => setErr(e instanceof Error ? e.message : 'failed')) }, [])
+  const set = async (node: string, device: string) => {
+    setBusy(node + '|' + device); setErr('')
+    try { setT(await api.setTranscode(node, device)) } catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
+    setBusy('')
+  }
+  const sel = t?.selected
+  return (
+    <>
+      <div className="dash-bar"><h2 className="section">Transcode device</h2><span className="spacer" /><span className="muted small">videos in Files, on every machine</span></div>
+      <section className="card">
+        {!t && !err && <div className="muted small"><Spinner /> asking every machine what it can encode on…</div>}
+        {t && (
+          <div className="tc-list">
+            <div className="tc-machine">
+              <div className="tc-host">Anywhere</div>
+              <div className="chips">
+                <button className={'chip' + (!sel ? ' on' : '')} disabled={!!busy} onClick={() => set('', '')}>The machine the file is on, its best device</button>
+              </div>
+            </div>
+            {t.machines.map((m) => (
+              <div key={m.node} className="tc-machine">
+                <div className="tc-host">{m.host}{m.node === '' && <span className="muted small"> · this hub</span>}</div>
+                {m.error && <div className="muted small">{m.error}</div>}
+                <div className="chips">
+                  {(m.devices ?? []).map((d) => {
+                    const on = !!sel && sel.node === m.node && sel.device === d.id
+                    return (
+                      <button key={d.id} className={'chip' + (on ? ' on' : '') + (d.ok ? '' : ' off')} disabled={!d.ok || !!busy} title={d.ok ? d.id : d.reason}
+                        onClick={() => set(m.node, d.id)}>
+                        {busy === m.node + '|' + d.id ? <Spinner size={10} /> : null}{d.name}{d.id !== 'cpu' && d.integrated ? ' (integrated)' : ''}{!d.ok && d.reason ? ` — ${d.reason}` : ''}
+                      </button>
+                    )
+                  })}
+                </div>
+              </div>
+            ))}
+          </div>
+        )}
+        {err && <div className="error" style={{ marginTop: 8 }}>{err}</div>}
+        <div className="muted small" style={{ marginTop: 10 }}>
+          Opening a video in Files plays it YouTube's way — Auto picks the quality the connection takes, any quality can be picked by hand,
+          or the file as it is — and every quality is made on the fly, from wherever you seek to, on this device. Video previews
+          (thumbnails) are made on it too; image previews stay on each machine's CPU. When it is on another
+          machine than the file, that machine reads the file through this hub. If it can't be reached, the file's own machine makes it instead.
+        </div>
+      </section>
+    </>
+  )
+}

web/src/Shell.tsx

@@ -15,7 +15,6 @@ import Hardware from './Hardware'
 import Network from './Network'
 import VPN from './VPN'
 import ZFS from './ZFS'
-import KVM from './KVM'
 import Files from './Files'
 import Top from './Top'
 import Machines from './Machines'
@@ -81,7 +80,6 @@ const GROUPS: Group[] = [
   { id: 'usage', label: 'Usage', icon: '▧' },
   { id: 'zfs', label: 'ZFS', icon: '⛁' },
   { id: 'docker', label: 'Docker', icon: DockerIcon },
-  { id: 'kvm', label: 'KVM', icon: '⌨' },
   { id: 'hardware', label: 'Hardware', icon: '⬡' },
   { id: 'updates', label: 'Updates', icon: '⇪' },
   { id: 'alerts', label: 'Alerts', icon: BellIcon },
@@ -91,8 +89,8 @@ const GROUPS: Group[] = [
   { id: 'settings', label: 'Settings', icon: '⚙' },
 ]
 
-type View = 'terminal' | 'widgets' | 'docker' | 'alerts' | 'services' | 'jobs' | 'updates' | 'disks' | 'hardware' | 'network' | 'vpn' | 'zfs' | 'usage' | 'files' | 'procs' | 'machines' | 'kvm' | 'users' | 'settings'
-const VIEWS: View[] = ['terminal', 'widgets', 'docker', 'alerts', 'services', 'jobs', 'updates', 'disks', 'hardware', 'network', 'vpn', 'zfs', 'usage', 'files', 'procs', 'machines', 'kvm', 'users', 'settings']
+type View = 'terminal' | 'widgets' | 'docker' | 'alerts' | 'services' | 'jobs' | 'updates' | 'disks' | 'hardware' | 'network' | 'vpn' | 'zfs' | 'usage' | 'files' | 'procs' | 'machines' | 'users' | 'settings'
+const VIEWS: View[] = ['terminal', 'widgets', 'docker', 'alerts', 'services', 'jobs', 'updates', 'disks', 'hardware', 'network', 'vpn', 'zfs', 'usage', 'files', 'procs', 'machines', 'users', 'settings']
 const HUB_ONLY: View[] = ['machines', 'users', 'settings', 'alerts']
 
 // URL hash is the source of truth for where you are: #/<machine|hub>/<view>. Back and
@@ -324,7 +322,6 @@ export default function Shell({ me: hubMe, onLoggedOut }: { me: Me; onLoggedOut:
           <PageBoundary page={GROUPS.find((g) => g.id === view)?.label ?? view}>
           {view === 'settings' && <Settings me={hubMe} />}
           {view === 'users' && <Users me={hubMe} />}
-          {view === 'kvm' && <KVM key={machine ?? 'hub'} machine={machine} title={title} />}
           {view === 'usage' && <Usage key={machine ?? ''} />}
           {view === 'machines' && <Machines hub={hubMe} onChanged={loadMachines} onOpen={(n) => { switchMachine(n); setView('procs') }} />}
           {view === 'widgets' && <Gallery />}
@@ -366,7 +363,7 @@ export default function Shell({ me: hubMe, onLoggedOut }: { me: Me; onLoggedOut:
         <span>{machine ? `${title} · ` : ''}{me.version.startsWith('v') ? me.version : 'v' + me.version}</span>
         <span>up {fmtUptime(me.uptime)}</span>
         <span className="spacer" />
-        <span className="muted">{view === 'widgets' ? 'Widgets' : view === 'docker' ? 'Docker' : view === 'alerts' ? 'Alerts' : view === 'services' ? 'Services' : view === 'jobs' ? 'Jobs' : view === 'updates' ? 'Updates' : view === 'disks' ? 'Disks' : view === 'hardware' ? 'Hardware' : view === 'network' ? 'Network' : view === 'vpn' ? 'VPN' : view === 'zfs' ? 'ZFS' : view === 'usage' ? 'Usage' : view === 'files' ? 'Files' : view === 'procs' ? 'Processes' : view === 'machines' ? 'Machines' : view === 'kvm' ? 'KVM' : view === 'users' ? 'Users' : view === 'settings' ? 'Settings' : activeTerm ? activeTerm.name : ''}</span>
+        <span className="muted">{view === 'widgets' ? 'Widgets' : view === 'docker' ? 'Docker' : view === 'alerts' ? 'Alerts' : view === 'services' ? 'Services' : view === 'jobs' ? 'Jobs' : view === 'updates' ? 'Updates' : view === 'disks' ? 'Disks' : view === 'hardware' ? 'Hardware' : view === 'network' ? 'Network' : view === 'vpn' ? 'VPN' : view === 'zfs' ? 'ZFS' : view === 'usage' ? 'Usage' : view === 'files' ? 'Files' : view === 'procs' ? 'Processes' : view === 'machines' ? 'Machines' : view === 'users' ? 'Users' : view === 'settings' ? 'Settings' : activeTerm ? activeTerm.name : ''}</span>
       </footer>
     </div>
   )

web/src/Top.tsx

@@ -3,14 +3,15 @@
 // as they climb. The numbers are Ham Lab's own — the sensor stream for everything measured and
 // /api/procs for the table — so this works on any machine the console can see, whether or not
 // btop is installed on it.
-import { createContext, memo, useCallback, useContext, useEffect, useMemo, useRef, useState } from 'react'
+import { Fragment, createContext, memo, useCallback, useContext, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
 import { api, GPU, ProcTable, Proc, Sample } from './api'
 import { Catalog, MONITOR_SHAPE, Values, mergeValues, sampleOf, stale } from './sampleAdapter'
 import { fetchCatalog, fetchHistories, useSensorStream } from './sensors'
 import { fmtMs, nextPoint, REFRESH_OPTIONS, setRefreshMs, strideMs, useHistorySec, useRefreshMs } from './prefs'
 import { Loading, askYes, toast } from './ui'
 import { Theme, currentThemeId, themeById } from './themes'
-import { columnsOf } from './graphcols'
+import { columnsAt } from './graphcols'
+import { socketThreads } from './sockets'
 import { savePrefs } from './accents'
 
 const MAX = 300 // points kept per curve, as the machines keep them
@@ -68,7 +69,7 @@ function at(g: readonly string[], t: number): string {
 
 // ---- the dot graph ----
 // One dot column per sample, filled from the bottom, each dot coloured by how high it sits.
-function Graph({ values, tick, advance, cap, max, grad, flip = false }: { values: number[]; tick: number; advance: number; cap: number; max?: number; grad: GradName; flip?: boolean }) {
+function Graph({ values, tick, advance, now, cap, max, grad, flip = false }: { values: number[]; tick: number; advance: number; now: number; cap: number; max?: number; grad: GradName; flip?: boolean }) {
   const ref = useRef<HTMLCanvasElement>(null)
   const box = useRef<HTMLDivElement>(null)
   const pal = useContext(Pal)
@@ -79,7 +80,8 @@ function Graph({ values, tick, advance, cap, max, grad, flip = false }: { values
     ro.observe(box.current)
     return () => ro.disconnect()
   }, [])
-  useEffect(() => {
+  // drawn before the browser paints, so a graph changes in the same frame as the numbers
+  useLayoutEffect(() => {
     const c = ref.current
     if (!c || w < 4 || height < 4) return
     const dpr = window.devicePixelRatio || 1
@@ -91,14 +93,16 @@ function Graph({ values, tick, advance, cap, max, grad, flip = false }: { values
     // The box is the whole window, always: `cap` points wide (what the window holds at this
     // rate) whatever the box measures, so a full history fills it edge to edge and a machine
     // that has only just started draws in from the right, as btop does. Which points each
-    // column draws — the peak of its own fixed slice of the history — is graphcols.ts.
-    const seen = columnsOf(values, advance, cap, cols)
-    const top = max ?? Math.max(1, ...seen)
+    // column draws — the peak of its own fixed slice of the history — is graphcols.ts. The
+    // columns slide by a fraction of one on every tick, so the graph moves with the numbers
+    // beside it even when its points are several ticks apart.
+    const { cols: seen, edge } = columnsAt(values, advance, now, cap, cols)
+    const top = max ?? Math.max(1, ...seen.slice(-cols))
     const stops = pal[grad]
     for (let i = 0; i < seen.length; i++) {
       // anything above nothing lights the bottom row: the curve keeps a baseline, as btop's does
       const lit = seen[i] > 0 ? Math.max(1, Math.round((seen[i] / top) * rows)) : 0
-      const x = w - (seen.length - i) * step
+      const x = Math.round((w - (seen.length - 1 - i + edge) * step) * dpr) / dpr
       for (let r = 0; r < lit; r++) {
         const t = rows > 1 ? r / (rows - 1) : 1
         g.fillStyle = at(stops, t)
@@ -106,7 +110,7 @@ function Graph({ values, tick, advance, cap, max, grad, flip = false }: { values
         g.fillRect(x, y, 2, 2)
       }
     }
-  }, [values, tick, advance, cap, w, height, max, grad, flip, pal]) // tick: the curve is appended to in place
+  }, [values, tick, advance, now, cap, w, height, max, grad, flip, pal]) // tick: the curve is appended to in place
   return <div className="tg" ref={box} data-n={values.length}><canvas ref={ref} style={{ width: '100%', height: '100%' }} /></div>
 }
 
@@ -187,12 +191,17 @@ function braille(values: number[], cells = 7, max = 100): string {
 
 // ---- the curves this page keeps ----
 const CURVES = 'cpu.usage,cpu.core.*.usage,net.*.rx,net.*.tx,disk.*.util,gpu.*.usage'
+const RECENT = 18 // ticks a per-core strip keeps: its nine braille cells, two dots each
 
 function useCurves(live: Values | null, t: number, refreshMs: number, historySec: number, resumed: number) {
   const cur = useRef<Map<string, number[]>>(new Map())
   const lastT = useRef(0)
   const binN = useRef(1)
   const adv = useRef(0) // the number of the newest point, counted since the page opened: what a graph's columns are cut by
+  const now = useRef(1) // how far along the curves are, in points: adv plus the newest point's share of its stride gone
+  // The per-core strips are a few seconds wide: they keep every tick, not the stride's points,
+  // so they move with the numbers beside them whatever the history window is.
+  const recent = useRef<Map<string, number[]>>(new Map())
   const [ticks, bump] = useState(0)
   const seed = useCallback(async () => {
     try {
@@ -202,21 +211,30 @@ function useCurves(live: Values | null, t: number, refreshMs: number, historySec
       // The stride is timed on this clock only. The history's own timestamps are the machine's:
       // measured against them, a viewer whose clock runs minutes slow would wait those minutes
       // before its curves gained a single new point.
-      cur.current = m; lastT.current = Date.now(); binN.current = 1
+      cur.current = m; lastT.current = Date.now(); binN.current = 1; now.current = adv.current + 1
       bump((x) => x + 1)
     } catch { /* the curves fill in live instead */ }
   }, [])
   useEffect(() => { seed() }, [seed, resumed, historySec]) // a new window is a new spacing: start from what the machine now keeps
-  useEffect(() => {
+  // before paint: the curves take the sample in the same frame that shows its numbers
+  useLayoutEffect(() => {
     if (!live || !t) return
-    const { fresh, start } = nextPoint(t, lastT.current, strideMs(historySec, refreshMs), refreshMs)
+    const stride = strideMs(historySec, refreshMs)
+    const { fresh, start } = nextPoint(t, lastT.current, stride, refreshMs)
     binN.current = fresh ? 1 : binN.current + 1
     lastT.current = start
     if (fresh) adv.current++
+    // this tick's share of the point: one tick in at its start, the whole point by its last
+    now.current = adv.current + Math.max(refreshMs / stride, Math.min(1, (t - start + refreshMs) / stride))
     const n = binN.current
     for (const [id, v] of Object.entries(live)) {
       if (typeof v !== 'number') continue
       if (!/^cpu\.usage$|^cpu\.core\.\d+\.usage$|^net\..+\.(rx|tx)$|^disk\..+\.util$|^gpu\..+\.usage$/.test(id)) continue
+      if (id.startsWith('cpu.core.')) {
+        const r = recent.current.get(id) ?? []
+        r.push(v); if (r.length > RECENT) r.shift()
+        recent.current.set(id, r)
+      }
       const arr = cur.current.get(id) ?? []
       if (!cur.current.has(id)) cur.current.set(id, arr)
       if (fresh || arr.length === 0) { arr.push(v); if (arr.length > MAX) arr.shift() }
@@ -224,7 +242,7 @@ function useCurves(live: Values | null, t: number, refreshMs: number, historySec
     }
     bump((x) => x + 1)
   }, [live, t, refreshMs, historySec])
-  return [cur.current, ticks, adv.current] as const
+  return [cur.current, ticks, adv.current, now.current, recent.current] as const
 }
 
 // ---- the page ----
@@ -239,18 +257,51 @@ export default function Top() {
   const [ready, setReady] = useState(false)
   const stream = useSensorStream<unknown[]>(MONITOR_SHAPE, { full: false, everyMs: refreshMs })
   useEffect(() => { fetchCatalog().then((c) => { catalog.current = new Map(c.map((x) => [x.id, x])); setReady(true) }).catch(() => setReady(true)) }, [])
-  const values = useMemo(() => (stream.data ? mergeValues(stream.data) : null), [stream.data])
+
+  // One beat for the whole page. Every sample the stream pushes is held until the process
+  // table has been read for it, and the two land together: the curves, the meters, the numbers,
+  // the GPU boxes and the table all change on the same tick, at the rate, never on two clocks
+  // that drift past each other. A sample that arrives while the table is still being read
+  // replaces the held one, so a slow read costs ticks, never order.
+  const [frame, setFrame] = useState<{ data: unknown[] | null; pt: ProcTable | null }>({ data: null, pt: null })
+  const [err, setErr] = useState('')
+  const held = useRef<unknown[] | null>(null)
+  const reading = useRef(false)
+  const mounted = useRef(true)
+  useEffect(() => { mounted.current = true; return () => { mounted.current = false } }, [])
+  useEffect(() => {
+    if (!stream.data) return
+    held.current = stream.data
+    if (reading.current) return
+    reading.current = true
+    api.procs().then((pt) => ({ pt, e: '' }), (e) => ({ pt: null, e: e instanceof Error ? e.message : 'failed' })).then(({ pt, e }) => {
+      reading.current = false
+      if (!mounted.current) return
+      const data = held.current
+      setFrame((f) => ({ data, pt: pt ?? f.pt }))
+      setErr(e)
+    })
+  }, [stream.data])
+  const pt = frame.pt
+  const values = useMemo(() => (frame.data ? mergeValues(frame.data) : null), [frame.data])
   useEffect(() => {
     if (!values || catalog.current.size === 0 || refetching.current || !stale(values, catalog.current)) return
     refetching.current = true
     fetchCatalog().then((c) => { catalog.current = new Map(c.map((x) => [x.id, x])) }).catch(() => {}).finally(() => window.setTimeout(() => { refetching.current = false }, 10000))
   }, [values])
   const s: Sample | null = useMemo(() => (values && ready ? sampleOf(values, catalog.current) : null), [values, ready])
-  const [curves, tick, advance] = useCurves(values, s?.t ?? 0, refreshMs, historySec, stream.resumed)
+  const [curves, tick, advance, now, recent] = useCurves(values, s?.t ?? 0, refreshMs, historySec, stream.resumed)
 
-  // the process table, and the CPU model, come from the same place the Processes page reads
-  const [pt, setPt] = useState<ProcTable | null>(null)
+  // the process table, and the CPU model, come from the same place the Processes page reads,
+  // on the sample's beat (above)
   const [pickedIf, setPickedIf] = useState<string | null>(null) // which NIC the net box draws
+  // which sockets' threads are folded away on a two-socket machine; remembered in this browser
+  const [folded, setFolded] = useState<number[]>(() => { try { return JSON.parse(localStorage.getItem('top.folded-cpus') ?? '[]') } catch { return [] } })
+  const fold = (pk: number) => setFolded((f) => {
+    const n = f.includes(pk) ? f.filter((x) => x !== pk) : [...f, pk]
+    try { localStorage.setItem('top.folded-cpus', JSON.stringify(n)) } catch { /* private window */ }
+    return n
+  })
   // Which pool each drive belongs to, asked once: a drive in a ZFS pool has no filesystem of
   // its own, and "nothing is on it" would be a lie about eight drives holding a NAS.
   const [pools, setPools] = useState<Record<string, string>>({})
@@ -264,14 +315,6 @@ export default function Top() {
     }).catch(() => { /* the rows simply say less */ })
     return () => { live = false }
   }, [])
-  const [err, setErr] = useState('')
-  useEffect(() => {
-    let stop = false
-    const poll = async () => { try { const r = await api.procs(); if (!stop) { setPt(r); setErr('') } } catch (e) { if (!stop) setErr(e instanceof Error ? e.message : 'failed') } }
-    poll()
-    const i = window.setInterval(poll, refreshMs) // exactly the rate, like every other page: nothing is throttled behind his back
-    return () => { stop = true; window.clearInterval(i) }
-  }, [refreshMs])
 
   // the same global rate Settings sets: saved on the hub, so it sticks and every machine's sampler follows
   const rateIdx = (() => { const i = REFRESH_OPTIONS.findIndex((x) => x >= refreshMs); return i < 0 ? REFRESH_OPTIONS.length - 1 : i })()
@@ -313,7 +356,7 @@ export default function Top() {
     try { await api.procSignal(p.pid, sig); toast(`SIG${sig} sent to ${p.pid}`) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
   }
 
-  if (!s && !pt) return <div className="top" style={pal.vars as React.CSSProperties}><Loading label={err || 'Reading the machine…'} /></div>
+  if (!s && !pt) return <div className="top" style={pal.vars as React.CSSProperties}><Loading label={err || stream.error || 'Reading the machine…'} /></div>
 
   const cpu = s?.cpu ?? pt?.cpu ?? 0
   const cores = (s?.cores?.length ? s.cores : pt?.cores) ?? []
@@ -325,7 +368,16 @@ export default function Top() {
   const model = (pt?.cpu_model ?? '').replace(/\(R\)|\(TM\)|CPU|Processor/g, '').replace(/@.*$/, '').replace(/^\d+th Gen\s*/i, '').replace(/\s+/g, ' ').trim()
   // the hottest thing that looks like a CPU sensor, and one per core where the chip names them;
   // a second thread of a core (CPU 8 on an 8-core chip with hyper-threading) wears its core's
-  const coreTemp = (i: number) => { const id = pt?.core_ids?.[i] ?? i; return s?.temps.find((t) => new RegExp(`core ${id}\\b`, 'i').test(t.label))?.c }
+  // On a two-socket board each CPU has its own sensor chip, both counting from "Core 0", so a
+  // thread's core is looked up on its own socket's chip.
+  const pkgs = pt?.core_pkgs?.length === cores.length ? pt.core_pkgs : undefined
+  // each CPU's own threads, counted from 0 under it (sockets.ts): the kernel's one numbering
+  // of the whole machine puts cpu 0–9 and 20–29 on the first and 10–19 and 30–39 on the second
+  const socks = pkgs ? socketThreads(pkgs) : []
+  const sockets = socks.map((x) => x.pkg)
+  const coreTemp = (i: number) => { const id = pt?.core_ids?.[i] ?? i, pk = pkgs?.[i]; return s?.temps.find((t) => new RegExp(`core ${id}\\b`, 'i').test(t.label) && (pk === undefined || (t.pkg ?? 0) === pk))?.c }
+  const sockTemp = (pk: number) => s?.temps.find((t) => t.chip === 'coretemp' && new RegExp(`^package id ${pk}$`, 'i').test(t.label))?.c
+  const sockUse = (pk: number) => { const v = cores.filter((_, i) => pkgs?.[i] === pk); return v.length ? v.reduce((a, b) => a + b, 0) / v.length : 0 }
   const cpuTemp = pt?.cpu_temp ?? s?.temps.find((t) => /package|tctl|tdie|cpu/i.test(t.label))?.c ?? coreTemp(0)
   // the temperature columns are there for every row once any row has one, so the % column lines up
   const anyTemp = cpuTemp !== undefined || cores.some((_, i) => coreTemp(i) !== undefined)
@@ -365,29 +417,46 @@ export default function Top() {
       </>}>
         <div className="t-cpu">
           <div className="t-cpu-graph">
-            <Graph values={curves.get('cpu.usage') ?? []} tick={tick} advance={advance} cap={cap} max={100} grad="cpu" />
+            <Graph values={curves.get('cpu.usage') ?? []} tick={tick} advance={advance} now={now} cap={cap} max={100} grad="cpu" />
             <div className="t-up">up {upfor(uptime)}</div>
           </div>
-          <div className="t-cores">
-            <div className="t-cores-head"><b>{model || 'cpu'}</b><span className="spacer" />{pt?.cpu_mhz ? `${Math.round(pt.cpu_mhz)} MHz` : ''}
-              {pt?.cpu_power_w ? <span title="the CPU's own power reading: cores, built-in graphics and memory controller, not the whole machine">{pt.cpu_mhz ? ' · ' : ''}{pt.cpu_power_w < 10 ? pt.cpu_power_w.toFixed(1) : Math.round(pt.cpu_power_w)} W</span> : null}</div>
-            <div className="t-core">
+          <div className={'t-cores' + (sockets.length > 1 ? ' multi' : '')}>
+            <div className="t-cores-head"><b>{sockets.length > 1 ? `${sockets.length} × ` : ''}{model || 'cpu'}</b><span className="spacer" />{pt?.cpu_mhz ? `${Math.round(pt.cpu_mhz)} MHz` : ''}
+              {pt?.cpu_power_w && sockets.length < 2 ? <span title="the CPU's own power reading: cores, built-in graphics and memory controller, not the whole machine">{pt.cpu_mhz ? ' · ' : ''}{pt.cpu_power_w < 10 ? pt.cpu_power_w.toFixed(1) : Math.round(pt.cpu_power_w)} W</span> : null}</div>
+            {sockets.length < 2 && <div className="t-core">
               <span className="t-core-n">CPU</span>
               <Meter pct={cpu} grad="cpu" />
               <span className="t-core-p">{Math.round(cpu)}%</span>
               {anyTemp && <><span className="t-core-t">{cpuTemp !== undefined && <Meter pct={Math.min(100, (cpuTemp / 100) * 100)} grad="temp" />}</span><span className="t-core-c">{cpuTemp !== undefined ? `${Math.round(cpuTemp)}°C` : ''}</span></>}
+            </div>}
+            {(sockets.length > 1 ? socks : [{ pkg: undefined, cpus: cores.map((_, i) => i) }]).map(({ pkg: pk, cpus }) => (
+              <Fragment key={pk ?? 'all'}>
+                {pk !== undefined && (() => {
+                  const u = sockUse(pk), t = sockTemp(pk), w = pt?.cpu_power_pkgs_w?.[pk], open = !folded.includes(pk)
+                  return (
+                    <div className="t-core t-sock" role="button" aria-expanded={open} title={open ? 'hide its threads' : 'show its threads'} onClick={() => fold(pk)}>
+                      <span className="t-core-n"><span className="t-fold">{open ? '▾' : '▸'}</span>CPU{pk + 1}</span>
+                      <span className="t-core-w" title="this CPU's own power reading: its cores, cache and memory controller">{w ? `${w < 10 ? w.toFixed(1) : Math.round(w)} W` : ''}</span>
+                      <Meter pct={u} grad="cpu" />
+                      <span className="t-core-p">{Math.round(u)}%</span>
+                      {anyTemp && <><span className="t-core-t">{t !== undefined && <Meter pct={Math.min(100, t)} grad="temp" />}</span><span className="t-core-c">{t !== undefined ? `${Math.round(t)}°C` : ''}</span></>}
                     </div>
-            {cores.map((v, i) => {
-              const t = coreTemp(i)
+                  )
+                })()}
+                {(pk !== undefined && folded.includes(pk) ? [] : cpus).map((i, n) => {
+                  // n is the thread's number under its own CPU, i the kernel's for the whole machine
+                  const t = coreTemp(i), v = cores[i] ?? 0
                   return (
                     <div className="t-core" key={i}>
-                  <span className="t-core-n">C{i}</span>
-                  <span className="t-core-g" title={pt?.core_ids && pt.core_ids[i] !== i ? `cpu ${i} · a thread of core ${pt.core_ids[i]}` : `core ${i}`}>{braille(curves.get(`cpu.core.${i}.usage`) ?? [], 9)}</span>
+                      <span className="t-core-n">C{n}</span>
+                      <span className="t-core-g" title={pk !== undefined ? `CPU ${pk + 1} · thread ${n} · cpu ${i} to the kernel${pt?.core_ids ? ` · a thread of core ${pt.core_ids[i]}` : ''}` : pt?.core_ids && pt.core_ids[i] !== i ? `cpu ${i} · a thread of core ${pt.core_ids[i]}` : `core ${i}`}>{braille(recent.get(`cpu.core.${i}.usage`) ?? [], 9)}</span>
                       <span className="t-core-p">{Math.round(v)}%</span>
                       {anyTemp && <><span className="t-core-t">{t !== undefined && <Meter pct={Math.min(100, t)} grad="temp" />}</span><span className="t-core-c">{t !== undefined ? `${Math.round(t)}°C` : ''}</span></>}
                     </div>
                   )
                 })}
+              </Fragment>
+            ))}
             <div className="t-load">Load AVG:<span className="spacer" />{load.map((x) => x.toFixed(2)).join('   ')}</div>
           </div>
         </div>
@@ -397,7 +466,7 @@ export default function Top() {
         <div className="top-gpus">
           {gpus.map(({ g, stem }, i) => (
             <Box key={stem} name={`gpu${i}`} className={'top-gpu' + (unmeasured(g) ? ' slim' : '')} right={<span className="t-gpu-name" title={[g.name, g.driver, g.bus].filter(Boolean).join(' · ')}>{g.name}</span>}>
-              {unmeasured(g) ? <GPULine g={g} /> : <GPUBox g={g} values={curves.get(stem + '.usage') ?? []} tick={tick} advance={advance} cap={cap} />}
+              {unmeasured(g) ? <GPULine g={g} /> : <GPUBox g={g} values={curves.get(stem + '.usage') ?? []} tick={tick} advance={advance} now={now} cap={cap} />}
             </Box>
           ))}
         </div>
@@ -421,7 +490,9 @@ export default function Top() {
             // pool or an md array spans several drives, so nothing is attributed and the row
             // shows what is true of the hardware — how busy it is, and how hot.
             const on = fs.filter((f) => f.disk === d.name)
-            const used = on.reduce((a, f) => a + f.used, 0)
+            // a bind mount (nexus's /var/log is /data/var-log) is the same filesystem twice:
+            // count each source device once, or the drive reads double
+            const used = [...new Map(on.map((f) => [f.source || f.mount, f.used])).values()].reduce((a, b) => a + b, 0)
             return <Disk key={d.name} name={d.name} note={d.model} size={d.size} used={on.length ? used : undefined} io={d.util_pct} temp={d.temp} mounts={on.map((f) => f.mount)} pool={pools[d.name]} />
           })}
           {(s?.disks ?? []).length === 0 && <div className="t-dim">no drives reported</div>}
@@ -441,10 +512,10 @@ export default function Top() {
             <div className="t-net-graphs">
               <div className="t-net-g">
                 <span className="t-scale">{rate(peak(`net.${segOf(iface.iface)}.rx`))}</span>
-                <Graph values={curves.get(`net.${segOf(iface.iface)}.rx`) ?? []} tick={tick} advance={advance} cap={cap} grad="down" />
+                <Graph values={curves.get(`net.${segOf(iface.iface)}.rx`) ?? []} tick={tick} advance={advance} now={now} cap={cap} grad="down" />
               </div>
               <div className="t-net-g">
-                <Graph values={curves.get(`net.${segOf(iface.iface)}.tx`) ?? []} tick={tick} advance={advance} cap={cap} grad="up" flip />
+                <Graph values={curves.get(`net.${segOf(iface.iface)}.tx`) ?? []} tick={tick} advance={advance} now={now} cap={cap} grad="up" flip />
                 <span className="t-scale t-scale-b">{rate(peak(`net.${segOf(iface.iface)}.tx`))}</span>
               </div>
             </div>
@@ -522,16 +593,14 @@ const Rows = memo(function Rows({ procs, sel, onSel, curves, gpu, empty }: { pro
 // that have it open with their share. What cannot be measured on this card is left out rather
 // than shown as zero: an Arc on i915 has no temperature sensor, Asahi has no busy counter.
 const ENGINES: [string, string][] = [['render', 'render'], ['compute', 'compute'], ['video', 'video'], ['video-enhance', 'enhance'], ['copy', 'copy']]
-function GPUBox({ g, values, tick, advance, cap }: { g: GPU; values: number[]; tick: number; advance: number; cap: number }) {
+function GPUBox({ g, values, tick, advance, now, cap }: { g: GPU; values: number[]; tick: number; advance: number; now: number; cap: number }) {
   const engines = ENGINES.filter(([k]) => g.engines && g.engines[k] !== undefined)
   const temp = g.temp !== undefined && g.temp > 0 ? g.temp : undefined
-  const how = g.stalled ? 'a sysfs read of this card stopped answering; its readings are paused until the console restarts'
-    : g.usage_src === 'idle' ? 'busy, from idle residency' : g.usage_src === 'busy' ? 'busy, as the driver reports it' : g.usage_src === 'clients' ? 'the busiest engine, summed over its clients' : 'this driver exposes no activity counter'
   return (
     <div className="t-gpu">
       <div className="t-gpu-graph">
-        <Graph values={values} tick={tick} advance={advance} cap={cap} max={100} grad="cpu" />
-        <div className="t-up">{g.usage_src ? `${Math.round(g.usage)}% ` : ''}{how}</div>
+        <Graph values={values} tick={tick} advance={advance} now={now} cap={cap} max={100} grad="cpu" />
+        {g.stalled && <div className="t-up">a read of this card stopped answering; its readings are paused until it does</div>}
       </div>
       <div className="t-gpu-stats">
         <div className="t-cores-head"><b>{g.name}</b><span className="spacer" />{g.driver}{g.integrated ? ' · integrated' : ''}</div>
@@ -555,7 +624,7 @@ function GPUBox({ g, values, tick, advance, cap }: { g: GPU; values: number[]; t
             <span className="t-fact" title={g.req_mhz ? `${Math.round(g.req_mhz)} MHz requested${g.max_mhz ? `, ${Math.round(g.max_mhz)} MHz max` : ''}` : undefined}>{Math.round(g.mhz ?? 0)}{g.max_mhz ? ` / ${Math.round(g.max_mhz)}` : ''} MHz</span></div> : null}
           {(g.power_w !== undefined && g.power_w > 0) || g.power_cap_w ? <div className="t-row t-sub"><span>Power</span><span className="spacer" />
             <span className="t-fact">{(g.power_w ?? 0).toFixed(1)}{g.power_cap_w ? ` / ${Math.round(g.power_cap_w)}` : ''} W</span></div> : null}
-          {g.fan_rpm ? <div className="t-row t-sub"><span>Fan</span><span className="spacer" /><span className="t-fact">{g.fan_rpm} rpm</span></div> : null}
+          {g.fan_rpm || g.fan_pct !== undefined ? <div className="t-row t-sub"><span>Fan</span><span className="spacer" /><span className="t-fact">{[g.fan_pct !== undefined ? `${Math.round(g.fan_pct)}%` : '', g.fan_rpm ? `${g.fan_rpm} rpm` : ''].filter(Boolean).join(' · ')}</span></div> : null}
           {g.mem_total ? <div className="t-row t-sub"><span>Mem</span><span className="t-pct">{Math.round(((g.mem_used ?? 0) / g.mem_total) * 100)}%</span><Meter pct={((g.mem_used ?? 0) / g.mem_total) * 100} grad="used" /><span className="t-amt">{bytes(g.mem_used ?? 0, 1)}</span></div>
             : g.mem_used ? <div className="t-row t-sub"><span>Mem</span><span className="spacer" /><span className="t-fact" title="resident memory of the processes that have it open">{bytes(g.mem_used, 1)}</span></div> : null}
         </div>
@@ -576,13 +645,13 @@ function GPUBox({ g, values, tick, advance, cap }: { g: GPU; values: number[]; t
 // Nothing on this GPU can be read — no busy counter, engine, temperature, clock, power, memory,
 // nor a process with it open: the Mac's Asahi, today. Its box would be a graph that can never
 // draw beside a column of dashes, so it is one line instead. A client opening it brings the box back.
-const unmeasured = (g: GPU) => !g.usage_src && !g.engines && !(g.temp && g.temp > 0) && !g.mhz && !g.max_mhz && !g.power_w && !g.power_cap_w && !g.fan_rpm && !g.mem_total && !g.mem_used && g.clients.length === 0
+const unmeasured = (g: GPU) => !g.usage_src && !g.engines && !(g.temp && g.temp > 0) && !g.mhz && !g.max_mhz && !g.power_w && !g.power_cap_w && !g.fan_rpm && g.fan_pct === undefined && !g.mem_total && !g.mem_used && g.clients.length === 0
 function GPULine({ g }: { g: GPU }) {
   return (
     <div className="t-row t-gpu-line">
       <span>{g.driver}{g.integrated ? ' · integrated' : ''}</span>
       {g.stalled
-        ? <span className="t-dim" title="a sysfs read of this card stopped answering">readings paused until the console restarts</span>
+        ? <span className="t-dim" title="a read of this card stopped answering">readings paused until it answers</span>
         : <span className="t-dim" title="the driver exposes no busy counter, clock, temperature, power or memory reading">nothing measurable</span>}
       <span className="t-dim t-gpu-who" title={g.clients_ok ? undefined : 'the console has no root here'}>{g.clients_ok ? 'no process has it open' : 'other users’ processes are hidden'}</span>
     </div>

web/src/ZFS.tsx

@@ -9,8 +9,8 @@
 // disagree over. An existing cron job is detected and shown, so the page tells the truth
 // about the machine whether Ham Lab or something else is doing the work.
 import { FormEvent, Fragment, useCallback, useEffect, useState } from 'react'
-import { api, ARCView, CacheDev, PoolCaching, ZFSExternal, ZFSInfo, ZFSOverview, ZFSPolicy, ZFSRun, ZFSSnapshot } from './api'
-import { fmtBytes } from './fmt'
+import { api, ARCView, CacheDev, Landing, LandingCheck, PoolCaching, PoolWrites, VerifyPolicy, VerifyResult, ZFSExternal, ZFSInfo, ZFSOverview, ZFSPolicy, ZFSRun, ZFSSnapshot, ZFSWrites } from './api'
+import { fmtBytes, fmtRate } from './fmt'
 import { useRefreshMs } from './prefs'
 import { fetchHistories } from './sensors'
 import Sparkline from './Sparkline'
@@ -367,22 +367,153 @@ function PoolTiers({ p, v }: { p: PoolCaching; v: ARCView }) {
   )
 }
 
+// ---- the write path: the dirty buffer in RAM, what each pool writes, and a landing tier ----
+//
+// A write reaches a pool in up to three steps. If a landing tier fronts the pool (a mergerfs
+// pool whose first branch is fast storage and whose last is the pool), new files land there
+// first and a mover drains them. Either way the data then waits in RAM as dirty buffers, and
+// each txg sync writes it to the disks. Each step's live number is here, with its history.
+
+const dur = (s: number) => (!Number.isFinite(s) || s <= 0 ? '—' : s < 90 ? `${Math.round(s)} s` : s < 5400 ? `${Math.round(s / 60)} min` : `${(s / 3600).toFixed(1)} h`)
+type WHist = Record<string, number[]>
+const wIds = (w: ZFSWrites) => ['zfs.write.dirty', ...w.pools.flatMap((p) => {
+  const b = `zfs.${p.pool}.`
+  return [b + 'write_bps', b + 'logical_bps', ...(p.landing ? [b + 'landing.pending_bytes', b + 'landing.in_bps', b + 'landing.drain_bps'] : [])]
+})]
+
+// rsync's --bwlimit: a bare number is KiB/s, a K/M/G suffix is binary (60M = 60 MiB/s); 0 is none
+const bwl = (v: string) => {
+  const m = /^(\d+(?:\.\d+)?)([KMG]?)$/i.exec((v ?? '').trim())
+  if (!m || Number(m[1]) === 0) return ''
+  return ` · rsync limited to ${m[1]} ${(m[2] || 'K').toUpperCase()}iB/s`
+}
+
+function MoverLine({ l }: { l: Landing }) {
+  const m = l.mover
+  if (!m) return <div className="notice" style={{ marginTop: 6 }}>No mover is reporting for this landing tier (nothing in <span className="mono">/run/landing-mover/</span>). Files that land here stay here until something moves them.</div>
+  const frac = m.current_size > 0 ? m.partial / m.current_size : 0
+  return (
+    <>
+      <div className="props rows" style={{ marginTop: 6 }}>
+        <div><span className="k">Mover</span><span>
+          <span className={'badge ' + (m.state === 'moving' ? 'on' : m.state === 'idle' ? '' : 'warn')}>{m.state}</span>
+          {' '}{m.state === 'moving' && m.batch_files > 1 ? `batch of ${m.batch_files} files, ${fmtBytes(m.batch_bytes)}` : m.state === 'holding' ? 'a mount it needs is missing — waiting' : m.state === 'stopped' ? `no word since ${ago(m.updated_ms)}` : `queue ${m.queue}`}
+          {m.state === 'moving' && m.queue > 0 ? ` · ${m.queue} more queued` : ''}
+        </span></div>
+        {m.state === 'moving' && m.current && <div><span className="k">Now</span><span className="mono small" title={m.current}>{m.current}{m.current_size ? ` · ${fmtBytes(m.partial)} of ${fmtBytes(m.current_size)}` : ''}</span></div>}
+        <div><span className="k">Since start</span><span>{fmtN(m.moved_files)} files · {fmtBytes(m.moved_bytes)} moved · running {ago(m.started_ms).replace(' ago', '')}</span></div>
+        {m.last_move && <div><span className="k">Last moved</span><span className="mono small" title={m.last_move}>{m.last_move} · {ago(m.last_move_ms)}</span></div>}
+        <div><span className="k">Rules</span><span className="muted">a file moves once it has been closed and left alone for {m.settle} s · oldest first{bwl(m.bwlimit)}</span></div>
+        {m.last_error && <div><span className="k">Last error</span><span className="warn-text" style={{ whiteSpace: 'normal' }}>{m.last_error} · {ago(m.last_error_ms)}</span></div>}
+      </div>
+      {m.state === 'moving' && m.current_size > 0 && <Meter frac={frac} cls="" />}
+    </>
+  )
+}
+
+function LandingBlockInner({ l, pending, inn, drain }: { l: Landing; pending: number[]; inn: number[]; drain: number[] }) {
+  const eta = l.drain_bps > 0 ? l.pending_bytes / l.drain_bps : NaN
+  const fill = l.size > 0 ? l.used / l.size : 0
+  return (
+    <div className="landing-tier">
+      <div className="arc-tier"><span className="k" title="a mergerfs pool puts new files on this branch first; a mover drains them onto the pool">Landing</span>
+        <span className="arc-tier-v"><span className="mono" title={`${l.branch} → ${l.pool_branch} via ${l.mount}`}>{l.branch}{l.device ? ` · ${l.device}` : ''}{l.fstype ? ` · ${l.fstype}` : ''} · in front of {l.pool_branch} as {l.mount}</span></span>
+      </div>
+      <div className="sub-head" style={{ marginTop: 6 }}>
+        <span className="muted small">{fmtBytes(l.used)} of {fmtBytes(l.size)} used</span><span className="spacer" />
+        <span className="num">{l.pending_files ? `${fmtN(l.pending_files)}${l.pending_partial ? '+' : ''} waiting · ${fmtBytes(l.pending_bytes)}` : 'nothing waiting'}</span>
+      </div>
+      <Meter frac={fill} />
+      <div className="props rows" style={{ marginTop: 6 }}>
+        <div><span className="k">Waiting</span><span>{l.pending_files ? `${fmtN(l.pending_files)}${l.pending_partial ? '+' : ''} files, ${fmtBytes(l.pending_bytes)} · oldest landed ${ago(l.oldest_ms)}` : 'nothing — everything that landed is on the pool'}<span className="muted small"> · counted {ago(l.counted_ms)}</span></span></div>
+        <div><span className="k">Arriving</span><span>{fmtRate(l.in_bps)}{l.device ? <span className="muted small"> · writes to {l.device}</span> : ''}</span></div>
+        <div><span className="k">To the pool</span><span>{fmtRate(l.drain_bps)}{l.pending_bytes > 0 ? ` · ${Number.isFinite(eta) ? `clear in about ${dur(eta)}` : 'not draining right now'}` : ''}</span></div>
+      </div>
+      <MoverLine l={l} />
+      <div className="arc-sparks">
+        <div>
+          <div className="muted small">Waiting on the landing tier · {fmtBytes(l.pending_bytes)}</div>
+          <Sparkline series={[{ name: 'waiting', values: pending, color: cssVar('--s1') }]} format={fmtBytes} height={40} />
+        </div>
+        <div>
+          <div className="muted small">Arriving {fmtRate(l.in_bps)} · to the pool {fmtRate(l.drain_bps)}</div>
+          <Sparkline series={[
+            { name: 'arriving', values: inn, color: cssVar('--s1') },
+            { name: 'to the pool', values: drain, color: cssVar('--ok') },
+          ]} format={fmtRate} height={40} />
+        </div>
+      </div>
+    </div>
+  )
+}
+
+function WritePath({ w, p, h }: { w: ZFSWrites; p: PoolWrites; h: WHist }) {
+  const b = `zfs.${p.pool}.`
+  const throttling = w.delayed_per_sec > 0 || w.over_max_per_sec > 0
+  const delayAt = (w.limit * w.delay_min_pct) / 100
+  const ratio = p.write_bps > 0 && p.logical_bps > 0 ? p.logical_bps / p.write_bps : 0
+  return (
+    <section className="card">
+      <div className="sub-head">
+        <span className={'dot ' + (w.over_max_per_sec > 0 ? 'bad' : throttling ? 'warn' : 'ok')} />
+        <span className="dname">{p.pool}</span>
+        <span className="badge">write path</span>
+        <span className="spacer" />
+        <span className="num">{fmtRate(p.write_bps)} to disk</span>
+      </div>
+      {p.landing
+        ? <LandingBlockWithHist l={p.landing} h={h} b={b} />
+        : <div className="arc-tiers"><div className="arc-tier"><span className="k" title="a fast tier new files land on before the pool">Landing</span><span className="arc-tier-v"><span className="none">none — writes go straight into the pool's RAM buffer</span></span></div></div>}
+      <div className="arc-sparks">
+        <div>
+          <div className="muted small">RAM write buffer · {fmtBytes(w.dirty)} of {fmtBytes(w.limit)}</div>
+          <Sparkline series={[{ name: 'waiting in RAM', values: h['zfs.write.dirty'] ?? [], color: cssVar('--s1') }]} max={w.limit || undefined} format={fmtBytes} height={44} />
+        </div>
+        <div>
+          <div className="muted small">Writes · {fmtRate(p.write_bps)} to disk · {fmtRate(p.logical_bps)} from programs</div>
+          <Sparkline series={[
+            { name: 'to disk', values: h[b + 'write_bps'] ?? [], color: cssVar('--s1') },
+            { name: 'from programs', values: h[b + 'logical_bps'] ?? [], color: cssVar('--ok') },
+          ]} format={fmtRate} height={44} />
+        </div>
+      </div>
+      <div className="props rows" style={{ marginTop: 8 }}>
+        <div><span className="k">In RAM</span><span title="dirty data: written by programs, not yet on the disks. For all pools together — ZFS keeps no live per-pool figure">{fmtBytes(w.dirty)} waiting · holds up to {fmtBytes(w.limit)} {w.limit < w.dirty_max ? `(half the ARC target; zfs_dirty_data_max is ${fmtBytes(w.dirty_max)})` : '(zfs_dirty_data_max)'}</span></div>
+        <div><span className="k">Throttle</span><span title="past zfs_delay_min_dirty_percent of the buffer, ZFS makes each write wait a little so the disks can catch up; at the limit, writes stop until a sync frees room">
+          {w.over_max_per_sec > 0 ? <span className="warn-text">{fmtN(w.over_max_per_sec)}/s writes stopped at the limit</span> : w.delayed_per_sec > 0 ? <span className="warn-text">slowing {fmtN(w.delayed_per_sec)} writes/s — the disks are the limit now</span> : 'not slowing anything'}
+          <span className="muted"> · starts at {w.delay_min_pct}% ({fmtBytes(delayAt)})</span></span></div>
+        <div><span className="k">To disk</span><span>{fmtRate(p.write_bps)} written by syncs{ratio > 0 ? ` · ${fmtRate(p.logical_bps)} from programs (${ratio >= 1 ? `${ratio.toFixed(2)}x compressed` : 'parity and metadata included'})` : ''}<span className="muted"> · a sync at least every {w.txg_timeout_sec} s, sooner as the buffer fills</span></span></div>
+      </div>
+    </section>
+  )
+}
+
+function LandingBlockWithHist({ l, h, b }: { l: Landing; h: WHist; b: string }) {
+  return <LandingBlockInner l={l} pending={h[b + 'landing.pending_bytes'] ?? []} inn={h[b + 'landing.in_bps'] ?? []} drain={h[b + 'landing.drain_bps'] ?? []} />
+}
+
 function ARCSection() {
   const [v, setV] = useState<ARCView | null>(null)
   const [hist, setHist] = useState<{ size: number[]; hit: number[] }>({ size: [], hit: [] })
+  const [whist, setWhist] = useState<WHist>({})
   const [err, setErr] = useState('')
   const rate = useRefreshMs()
   useEffect(() => {
     let stop = false
     const poll = async () => {
-      try { const r = await api.zfsARC(); if (!stop) { setV(r); setErr('') } } catch (e) { if (!stop) setErr(e instanceof Error ? e.message : 'failed') }
+      let r: ARCView | null = null
+      try { r = await api.zfsARC(); if (!stop) { setV(r); setErr('') } } catch (e) { if (!stop) setErr(e instanceof Error ? e.message : 'failed') }
       try {
-        const hs = await fetchHistories('zfs.arc.size,zfs.arc.hit_pct')
-        if (!stop) setHist({ size: hs.find((h) => h.id === 'zfs.arc.size')?.v ?? [], hit: hs.find((h) => h.id === 'zfs.arc.hit_pct')?.v ?? [] })
+        const extra = r?.writes ? wIds(r.writes) : []
+        const hs = await fetchHistories(['zfs.arc.size', 'zfs.arc.hit_pct', ...extra].join(','))
+        if (!stop) {
+          setHist({ size: hs.find((h) => h.id === 'zfs.arc.size')?.v ?? [], hit: hs.find((h) => h.id === 'zfs.arc.hit_pct')?.v ?? [] })
+          setWhist(Object.fromEntries(hs.filter((h) => extra.includes(h.id)).map((h) => [h.id, h.v])))
+        }
       } catch { /* the sparklines stay empty; the numbers still show */ }
     }
     poll()
-    const t = setInterval(poll, Math.max(2000, rate))
+    const t = setInterval(poll, rate)
     return () => { stop = true; clearInterval(t) }
   }, [rate])
 
@@ -444,7 +575,262 @@ function ARCSection() {
           <span><b>All reads</b></span><span className="num"><b>{fmtPct(hitPct(L.Hits, L.Misses))}</b></span><span className="num">{fmtN(L.Hits)}</span><span className="num">{fmtN(L.Misses)}</span>
         </div>
       </section>
-      {d.pools.length > 0 && <div className="cards" style={{ marginTop: 8 }}>{d.pools.map((p) => <PoolTiers key={p.pool} p={p} v={v} />)}</div>}
+      {d.pools.length > 0 && <div className="cards full" style={{ marginTop: 8 }}>{d.pools.map((p) => <PoolTiers key={p.pool} p={p} v={v} />)}</div>}
+      {v.writes && v.writes.pools.length > 0 && (
+        <>
+          <div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Writes</h2><span className="spacer" />
+            <span className="muted small">what is waiting to reach each pool, and how fast it is getting there</span>
+          </div>
+          <div className="cards full">{v.writes.pools.map((p) => <WritePath key={p.pool} w={v.writes!} p={p} h={whist} />)}</div>
+        </>
+      )}
+    </>
+  )
+}
+
+// ---- landing-tier checks: what has waited too long, and verify passes ----
+//
+// A verify pass reads the pool's copy of every file the mover moved back against the SHA-256
+// the mover took of the landing copy, from its ledger. "New moves" checks what has moved since
+// the last pass; "everything" re-reads the whole ledger. Passes run as their own process, the
+// same one a schedule's cron line starts, so a console restart does not cut one short.
+
+const vDescribe = (p: VerifyPolicy) => {
+  const when = p.every === 'custom' ? `on ${p.cron}`
+    : p.every === 'hourly' ? `every hour at :${(p.at.split(':')[1] ?? '00').padStart(2, '0')}`
+    : p.every === 'daily' ? `every day at ${p.at}`
+      : p.every === 'weekly' ? `every ${WEEKDAYS[p.weekday] ?? 'Sunday'} at ${p.at}`
+        : `on day ${p.day} of the month at ${p.at}`
+  return `${when} · ${p.scope === 'all' ? 'everything in the ledger' : 'what moved since the last pass'}`
+}
+const vOK = (r: VerifyResult) => !r.error && r.mismatched === 0 && r.gone === 0 && r.unreadable === 0
+const secs = (a: string, b: string) => (new Date(b).getTime() - new Date(a).getTime()) / 1000
+
+function VerifySummary({ r }: { r: VerifyResult }) {
+  if (r.error) return <span className="warn-text" style={{ whiteSpace: 'normal' }}>{r.error}</span>
+  const extra = [r.changed && `${fmtN(r.changed)} edited since (not compared)`, r.waiting && `${fmtN(r.waiting)} waiting to move again`, r.unhashed && `${fmtN(r.unhashed)} without a hash`].filter(Boolean).join(' · ')
+  return (
+    <span>
+      {vOK(r)
+        ? <>{r.checked === 0 ? 'nothing new to check' : <><b>{fmtN(r.matched)}</b> of {fmtN(r.checked)} files match</>}</>
+        : <span className="warn-text">{[r.mismatched && `${fmtN(r.mismatched)} DIFFER`, r.gone && `${fmtN(r.gone)} not on the pool`, r.unreadable && `${fmtN(r.unreadable)} unreadable`].filter(Boolean).join(' · ')} · {fmtN(r.matched)} match</span>}
+      <span className="muted"> · {fmtBytes(r.bytes_read)} read in {dur(secs(r.started, r.finished)) === '—' ? '0 s' : dur(secs(r.started, r.finished))} · {r.scope === 'all' ? 'everything' : 'new moves'} · {ago(r.finished)}</span>
+      {extra && <span className="muted"> · {extra}</span>}
+    </span>
+  )
+}
+
+function Issues({ r }: { r: VerifyResult }) {
+  const [open, setOpen] = useState(false)
+  if (!r.issues?.length) return null
+  return (
+    <div style={{ marginTop: 6 }}>
+      <button className="btn" onClick={() => setOpen(!open)}>{open ? 'Hide' : 'Show'} the {r.issues.length} file{r.issues.length === 1 ? '' : 's'} to look at</button>
+      {open && <div className="snap-list" style={{ marginTop: 6 }}>{r.issues.map((i) => (
+        <div key={i.path + i.kind} className="snap">
+          <span className={'badge ' + (i.kind === 'mismatch' ? 'bad' : 'warn')}>{i.kind === 'mismatch' ? 'differs' : i.kind === 'gone' ? 'not on pool' : 'unreadable'}</span>
+          <span className="mono snap-name" title={i.path}>{i.path}</span>
+          <span className="muted snap-when">moved {new Date(i.moved).toLocaleString()}</span>
+          <span className="num muted">{fmtBytes(i.size)}</span>
+        </div>
+      ))}</div>}
+      {open && r.gone > 0 && <div className="muted small" style={{ marginTop: 4 }}>"Not on the pool" is a file that was moved and is no longer there — deleted or renamed since, most often. The pass cannot tell that from a loss, so it lists them.</div>}
+    </div>
+  )
+}
+
+function VerifyScheduleCard({ p, cron, next, last, onChanged, onEdit }: { p: VerifyPolicy; cron: string; next?: string; last: VerifyResult | null; onChanged: () => void; onEdit: () => void }) {
+  const run = async () => {
+    try { await api.runVerifySchedule(p.id); toast('verify pass started'); onChanged() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
+  }
+  const remove = async () => {
+    if (!await askYes(`Remove the verify schedule "${p.name}"? Passes it already ran stay in the history.`, 'Remove')) return
+    try { await api.deleteVerifySchedule(p.id); toast('schedule removed'); onChanged() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
+  }
+  const toggle = async () => {
+    try { await api.saveVerifySchedule({ ...p, enabled: !p.enabled }); onChanged() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
+  }
+  return (
+    <section className="card">
+      <div className="sub-head">
+        <span className={'dot ' + (last && !vOK(last) ? 'bad' : p.enabled ? 'ok' : '')} />
+        <span className="dname">{p.name || 'verify'}</span>
+        {!p.enabled && <span className="badge">paused</span>}
+        <span className="spacer" />
+        <span className="muted small">{p.enabled ? soon(next) : 'not scheduled'}</span>
+      </div>
+      <div className="props rows" style={{ marginTop: 6 }}>
+        <div><span className="k">When</span><span>{vDescribe(p)}</span></div>
+        <div><span className="k">Cron</span><span className="mono small">{cron}</span></div>
+        <div><span className="k">Last pass</span>{last ? <VerifySummary r={last} /> : <span>never run</span>}</div>
+      </div>
+      <div className="svc-actions">
+        <Btn className="btn on" onClick={run}>Run now</Btn>
+        <button className="btn" onClick={onEdit}>Edit</button>
+        <Btn className="btn" onClick={toggle}>{p.enabled ? 'Pause' : 'Resume'}</Btn>
+        <Btn className="btn danger" onClick={remove}>Remove</Btn>
+      </div>
+    </section>
+  )
+}
+
+const VBLANK: Partial<VerifyPolicy> = { name: '', every: 'daily', at: '05:10', weekday: 0, day: 1, scope: 'new', enabled: true }
+
+function VerifyDialog({ p, onClose, onSaved }: { p: Partial<VerifyPolicy>; onClose: () => void; onSaved: () => void }) {
+  const [f, setF] = useState<Partial<VerifyPolicy>>({ ...VBLANK, ...p })
+  const [err, setErr] = useState('')
+  useEscape(onClose)
+  const set = (k: keyof VerifyPolicy, v: unknown) => setF((x) => ({ ...x, [k]: v }))
+  const save = async (e: FormEvent) => {
+    e.preventDefault(); setErr('')
+    try { await api.saveVerifySchedule({ ...f, name: (f.name || '').trim() || (f.scope === 'all' ? 'verify everything' : 'verify new moves') }); onSaved() }
+    catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
+  }
+  return (
+    <div className="modal-back" onClick={onClose}>
+      <form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
+        <h3>{f.id ? 'Edit verify schedule' : 'New verify schedule'}</h3>
+        <label className="muted">Name — what this schedule is for
+          <input value={f.name} onChange={(e) => set('name', e.target.value)} placeholder="nightly verify" maxLength={64} />
+        </label>
+        <label className="muted">What to check
+          <span className="chips">
+            <button type="button" className={'chip' + (f.scope === 'new' ? ' on' : '')} onClick={() => set('scope', 'new')}>new moves</button>
+            <button type="button" className={'chip' + (f.scope === 'all' ? ' on' : '')} onClick={() => set('scope', 'all')}>everything</button>
+          </span>
+          <span className="small">{f.scope === 'all' ? 'Re-reads every file the mover has ever moved from the pool. Thorough, and as slow as the ledger is large.' : 'Reads back what has moved since the last pass — every file checked once, soon after it lands.'}</span>
+        </label>
+        <label className="muted">How often
+          <span className="chips">
+            {(['hourly', 'daily', 'weekly', 'monthly', 'custom'] as const).map((k) => (
+              <button type="button" key={k} className={'chip' + (f.every === k ? ' on' : '')} onClick={() => set('every', k)}>{k}</button>
+            ))}
+          </span>
+        </label>
+        {f.every === 'custom' ? (
+          <label className="muted">Cron expression — minute hour day month weekday
+            <input className="mono" value={f.cron ?? ''} onChange={(e) => set('cron', e.target.value)} placeholder="0 */6 * * *" spellCheck={false} required />
+          </label>
+        ) : (
+          <label className="muted">At
+            <input type="time" value={f.at} onChange={(e) => set('at', e.target.value)} />
+          </label>
+        )}
+        {f.every === 'weekly' && (
+          <label className="muted">On
+            <select value={f.weekday} onChange={(e) => set('weekday', Number(e.target.value))}>
+              {WEEKDAYS.map((d, i) => <option key={d} value={i}>{d}</option>)}
+            </select>
+          </label>
+        )}
+        {f.every === 'monthly' && (
+          <label className="muted">Day of the month — 1 to 28, so every month has one
+            <input type="number" min={1} max={28} value={f.day} onChange={(e) => set('day', Number(e.target.value))} />
+          </label>
+        )}
+        <label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
+          <input type="checkbox" checked={!!f.enabled} onChange={(e) => set('enabled', e.target.checked)} style={{ width: 'auto' }} />
+          Enabled
+        </label>
+        {err && <div className="error">{err}</div>}
+        <div className="svc-actions">
+          <button type="submit" className="btn on">Save</button>
+          <span className="spacer" />
+          <button type="button" className="btn" onClick={onClose}>Cancel</button>
+        </div>
+      </form>
+    </div>
+  )
+}
+
+function LandingChecks() {
+  const [c, setC] = useState<LandingCheck | null>(null)
+  const [err, setErr] = useState('')
+  const [showStale, setShowStale] = useState(false)
+  const [editing, setEditing] = useState<Partial<VerifyPolicy> | null>(null)
+  const rate = useRefreshMs()
+  const load = useCallback(async () => {
+    try { setC(await api.zfsLanding()); setErr('') } catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
+  }, [])
+  useEffect(() => { load() }, [load])
+  const running = !!c?.running
+  useEffect(() => { const t = setInterval(load, running ? 2000 : Math.max(10000, rate * 5)); return () => clearInterval(t) }, [load, rate, running])
+
+  if (err && !c) return <div className="error">{err}</div>
+  if (!c || (c.tiers.length === 0 && c.policies.length === 0 && c.history.length === 0)) return null
+  const verify = async (scope: 'new' | 'all') => {
+    if (scope === 'all' && !await askYes('Re-read every file the mover has ever moved, from the pool, and check each against its hash? It reads as much as the ledger holds.', 'Verify everything')) return
+    try { await api.landingVerify(scope); toast('verify pass started'); setTimeout(load, 800) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
+  }
+  const last = c.history[0]
+  const hours = Math.round(c.stale_after_sec / 3600)
+  const r = c.running
+  return (
+    <>
+      <div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Landing tier checks</h2><span className="spacer" />
+        <span className="muted small">did everything that landed reach the pool, intact?</span>
+      </div>
+      <div className="cards full">
+        {c.tiers.length === 0 && <div className="notice">No landing mover is reporting right now (nothing in <span className="mono">/run/landing-mover/</span>), so there is nothing to check. Past passes are below.</div>}
+        {c.tiers.map((t) => (
+          <section className="card" key={t.src}>
+            <div className="sub-head">
+              <span className={'dot ' + (last && !vOK(last) ? 'bad' : t.stale_count ? 'warn' : 'ok')} />
+              <span className="dname">{t.dst}</span>
+              <span className="badge">landing checks</span>
+              <span className="spacer" />
+              <span className="muted small mono">{t.src} → {t.dst}</span>
+            </div>
+            <div className="props rows" style={{ marginTop: 6 }}>
+              <div><span className="k">Stale</span><span>
+                {t.stale_count
+                  ? <><span className="warn-text">{fmtN(t.stale_count)} file{t.stale_count === 1 ? '' : 's'}, {fmtBytes(t.stale_bytes)}, waiting over {hours} h</span>{' '}<button className="btn" onClick={() => setShowStale(!showStale)}>{showStale ? 'Hide' : 'Show'}</button></>
+                  : `nothing has waited on the landing tier over ${hours} h`}
+              </span></div>
+              <div><span className="k">Last pass</span>{last ? <VerifySummary r={last} /> : <span>never verified</span>}</div>
+              <div><span className="k">Covered</span><span>{t.cursor ? `every file moved up to ${new Date(t.cursor).toLocaleString()} has been checked at least once` : 'nothing checked yet'}<span className="muted"> · ledger {t.ledger}</span></span></div>
+            </div>
+            {showStale && t.stale.length > 0 && (
+              <div className="snap-list" style={{ marginTop: 6 }}>{t.stale.map((f) => (
+                <div key={f.path} className="snap">
+                  <span className="mono snap-name" title={f.path}>{f.path}</span>
+                  <span className="muted snap-when">landed {new Date(f.mtime).toLocaleString()}</span>
+                  <span className="muted snap-exp">waiting {dur((Date.now() - f.mtime) / 1000)}</span>
+                  <span className="num muted">{fmtBytes(f.size)}</span>
+                </div>
+              ))}{t.stale_count > t.stale.length && <div className="muted small">and {fmtN(t.stale_count - t.stale.length)} more</div>}</div>
+            )}
+            {r && (
+              <div style={{ marginTop: 8 }}>
+                <div className="sub-head"><span className="muted small">Verifying {r.scope === 'all' ? 'everything' : 'new moves'} · {fmtN(r.done)} of {fmtN(r.total)} files · {fmtBytes(r.bytes_read)} of {fmtBytes(r.bytes_all)}</span><span className="spacer" /><span className="mono small muted" title={r.current}>{r.current}</span></div>
+                <Meter frac={r.bytes_all > 0 ? r.bytes_read / r.bytes_all : r.total > 0 ? r.done / r.total : 0} cls="" />
+              </div>
+            )}
+            {last && <Issues r={last} />}
+            <div className="svc-actions">
+              <Btn className="btn on" disabled={running} onClick={() => verify('new')}>{running ? 'Verifying…' : 'Verify new moves'}</Btn>
+              <Btn className="btn" disabled={running} onClick={() => verify('all')}>Verify everything</Btn>
+            </div>
+            {c.history.length > 1 && (
+              <details style={{ marginTop: 6 }}>
+                <summary className="muted small">Earlier passes</summary>
+                <div className="props rows" style={{ marginTop: 4 }}>{c.history.slice(1).map((h, i) => (
+                  <div key={i}><span className="k">{new Date(h.finished).toLocaleString()}</span><VerifySummary r={h} /></div>
+                ))}</div>
+              </details>
+            )}
+          </section>
+        ))}
+      </div>
+
+      <div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Verify schedules</h2><span className="spacer" />
+        <Btn className="btn on" onClick={() => setEditing({ ...VBLANK })}>+ New schedule</Btn>
+      </div>
+      {c.policies.length === 0 && <div className="notice">No verify runs on a schedule. A daily pass of new moves checks every file once, soon after it lands.</div>}
+      <div className="cards full">{c.policies.map(({ policy, cron, next, last }) => (
+        <VerifyScheduleCard key={policy.id} p={policy} cron={cron} next={next} last={last} onChanged={load} onEdit={() => setEditing(policy)} />
+      ))}</div>
+      {editing && <VerifyDialog p={editing} onClose={() => setEditing(null)} onSaved={() => { setEditing(null); load() }} />}
     </>
   )
 }
@@ -475,19 +861,20 @@ export default function ZFS() {
     <div className="dash">
       <div className="dash-bar"><h2 className="section">Pools</h2></div>
       {zfs.pools.length === 0 && <div className="notice">ZFS is present but no pools are imported.</div>}
-      <div className="cards">{zfs.pools.map((p) => <PoolCard key={p.name} p={p} onChanged={load} />)}</div>
+      <div className="cards full">{zfs.pools.map((p) => <PoolCard key={p.name} p={p} onChanged={load} />)}</div>
 
       <div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Cache</h2><span className="spacer" />
         <span className="muted small">the ARC in RAM, and what each pool has in front of its disks</span>
       </div>
       <ARCSection />
+      <LandingChecks />
 
       <div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Snapshot schedules</h2><span className="spacer" />
         <Btn className="btn on" onClick={() => setEditing({ ...BLANK, dataset: zfs.pools[0]?.name ?? '' })}>+ New schedule</Btn>
       </div>
       {ov.external && <ExternalNotice x={ov.external} mine={mine} onAdopt={() => setEditing(adopt(ov.external!, zfs.pools[0]?.name ?? ''))} />}
       {ov.policies.length === 0 && !ov.external && <div className="notice">Nothing is taking snapshots on a schedule. Snapshots are written the way zfSnap writes them, with the retention in the name, so anything already here is understood.</div>}
-      <div className="cards">{ov.policies.map(({ policy, cron, next, last }) => (
+      <div className="cards full">{ov.policies.map(({ policy, cron, next, last }) => (
         <ScheduleCard key={policy.id} p={policy} cron={cron} next={next} last={last ?? {}} onChanged={load} onEdit={() => setEditing(policy)} />
       ))}</div>
 

web/src/api.ts

@@ -4,7 +4,12 @@ export class Unauthenticated extends Error { constructor(msg = 'unauthenticated'
 // is rewritten to /api/nodes/<name>/… and the hub proxies it. Hub-only calls (login,
 // the machine list itself) pass local=true and are never rewritten.
 let base = ''
-export function setMachine(name: string | null) { base = name ? `/api/nodes/${name}` : '' }
+export function setMachine(name: string | null) { base = name ? `/api/nodes/${name}` : ''; machine = name ?? '' }
+let machine = ''
+// The machine the console is looking at: '' is this host.
+export const currentMachine = () => machine
+// A path under another machine's /api, whichever one the console is looking at.
+export const nodeAPI = (node: string, p: string) => (node && p.startsWith('/api/') ? `/api/nodes/${node}${p.slice(4)}` : p)
 export const apiPath = (p: string) => (base && p.startsWith('/api/') ? base + p.slice(4) : p)
 
 // A machine that just failed to answer is left alone for a few seconds instead of
@@ -66,11 +71,11 @@ export const titleOf = (name: string | null, machines: MachineStatus[], hubHost:
 export interface MachineStatus { name: string; url: string; fingerprint?: string; added: string; online: boolean; host?: string; version?: string; load?: string; uptime?: number; reboot_needed?: boolean; last_seen?: string; error?: string }
 export interface Term { id: string; name: string; running: boolean; exit_code: number; created: string; clients: number }
 
-export interface Temp { chip: string; label: string; c: number; max?: number; crit?: number }
+export interface Temp { chip: string; label: string; c: number; max?: number; crit?: number; pkg?: number }
 export interface Fan { chip: string; label: string; rpm: number }
 export interface NetIf { iface: string; up: boolean; rx_bps: number; tx_bps: number; rx_total: number; tx_total: number }
 export interface Disk { name: string; model?: string; size: number; read_bps: number; write_bps: number; util_pct: number; temp?: number }
-export interface FS { mount: string; type: string; size: number; used: number; remote?: boolean; disk?: string }
+export interface FS { mount: string; type: string; size: number; used: number; remote?: boolean; disk?: string; source?: string }
 export interface UPS { driver: string; model: string; state: string; source: string; on_battery: boolean; in_v: number; out_v: number; battery_pct: number; runtime_min: number; load_w: number; load_pct: number; last_event?: string }
 export interface Proc { pid: number; ppid: number; name: string; cmd?: string; user: string; cpu: number; gpu?: number; cpu_time: number; rss: number; threads: number; state: string; nice: number; started: number }
 export interface MemInfo { total: number; used: number; avail: number; free?: number; cached?: number; buffers?: number }
@@ -81,11 +86,11 @@ export interface GPUClient { pid: number; name: string; usage: number; engines?:
 export interface GPU {
   name: string; driver: string; bus?: string; integrated?: boolean
   usage: number; usage_src?: 'idle' | 'busy' | 'clients'; engines?: Record<string, number>
-  temp?: number; temp_src?: 'gpu' | 'package'; power_w?: number; power_cap_w?: number; fan_rpm?: number
+  temp?: number; temp_src?: 'gpu' | 'package'; power_w?: number; power_cap_w?: number; fan_rpm?: number; fan_pct?: number
   mhz?: number; req_mhz?: number; max_mhz?: number; mem_used?: number; mem_total?: number
   clients: GPUClient[]; clients_ok: boolean; stalled?: boolean
 }
-export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_power_w?: number; cpu_temp?: number; core_ids?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
+export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_power_w?: number; cpu_power_pkgs_w?: number[]; cpu_temp?: number; core_ids?: number[]; core_pkgs?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
 export interface Sample {
   t: number; interval_ms: number; cpu: number; cores: number[]; load: [number, number, number]
   mem: { total: number; used: number; avail: number }; swap: { total: number; used: number; avail: number }
@@ -180,7 +185,6 @@ export interface SpaceNode { name: string; dir: boolean; size: number; apparent:
 export interface SpaceTree extends SpaceNode { path: string; children?: SpaceTree[]; more_n?: number; more_size?: number; total: number }
 export interface SpaceFile { path: string; size: number; apparent: number; mtime: number }
 
-export interface KVM { id: string; name: string; url: string; machine: string; added: string }
 
 // ---- the Network tab ----
 export interface IfAddr { ip: string; prefix: number; v6?: boolean }
@@ -247,15 +251,51 @@ export interface ARCDetail {
   dbuf: { Size: number; Target: number; Hits: number; Misses: number }
   pools: PoolCaching[]
 }
-export interface ARCView { detail: ARCDetail; live: ARCLive | null }
+export interface LandingMover {
+  state: 'idle' | 'moving' | 'holding' | 'stopped' | string; src: string; dst: string; pid: number; started_ms: number; updated_ms: number; stale: boolean
+  queue: number; settle: number; bwlimit: string; batch_files: number; batch_bytes: number; batch_start_ms: number; batch_first: string
+  current: string; current_size: number; partial: number; moved_files: number; moved_bytes: number
+  last_move: string; last_move_ms: number; last_error: string; last_error_ms: number
+}
+export interface Landing {
+  mount: string; branch: string; pool_branch: string; device?: string; fstype?: string; size: number; used: number
+  pending_files: number; pending_bytes: number; pending_partial?: boolean; oldest_ms?: number; counted_ms: number
+  in_bps: number; drain_bps: number; mover?: LandingMover
+}
+export interface PoolWrites { pool: string; write_bps: number; logical_bps: number; landing?: Landing }
+export interface ZFSWrites {
+  dirty: number; dirty_max: number; delay_min_pct: number; limit: number; delayed_per_sec: number; over_max_per_sec: number; txg_timeout_sec: number
+  pools: PoolWrites[]
+}
+export interface VerifyPolicy { id: string; name: string; every: 'hourly' | 'daily' | 'weekly' | 'monthly' | 'custom'; at: string; weekday: number; day: number; cron?: string; scope: 'new' | 'all'; enabled: boolean }
+export interface VerifyIssue { path: string; kind: 'mismatch' | 'gone' | 'unreadable'; moved: number; size: number; note?: string }
+export interface VerifyResult {
+  id: string; scope: 'new' | 'all'; started: string; finished: string; src: string; dst: string; since: number
+  entries: number; checked: number; matched: number; mismatched: number; changed: number; gone: number; waiting: number; unreadable: number; unhashed: number
+  bytes_read: number; issues: VerifyIssue[]; error?: string; cursor: number
+}
+export interface VerifyProgress { id: string; scope: string; pid: number; started: string; updated: string; total: number; done: number; bytes_read: number; bytes_all: number; current: string }
+export interface StaleFile { path: string; size: number; mtime: number }
+export interface LandingCheck {
+  tiers: { src: string; dst: string; ledger: string; cursor: number; stale_count: number; stale_bytes: number; stale: StaleFile[] }[]
+  stale_after_sec: number; running: VerifyProgress | null; history: VerifyResult[]; cron_file: string
+  policies: { policy: VerifyPolicy; cron: string; next?: string; last: VerifyResult | null }[]
+}
+export interface ARCView { detail: ARCDetail; live: ARCLive | null; writes: ZFSWrites | null }
 
 export interface FileEntry { name: string; type: 'dir' | 'file' | 'link' | 'other'; size: number; mode: string; perm: string; mtime: number; user: string; group: string; target?: string; broken?: boolean; preview?: 'image' | 'video' | 'audio' }
-export interface PlayCaps { codecs: { id: string; label: string; engine?: string }[]; qualities: { id: string; label: string; height: number }[]; error?: string }
-export interface PlayTrack { codec: string; profile?: string; pix_fmt?: string; width?: number; height?: number; fps?: number; channels?: number }
-export interface PlayStream { id: string; codec: string; quality: string; engine: string; width: number; height: number; kbps: number; seekable?: boolean; source: { container: string; duration: number; bit_rate?: number; video?: PlayTrack; audio?: PlayTrack } }
-// progress: seconds written (repackaged), or seconds of the film on disk (a seekable re-encode,
-// whose run now going started at `from`)
-export interface PlayStatus { running: boolean; done: boolean; error?: string; progress: number; speed: number; segments: number; from?: number }
+export interface PlayTrack { codec: string; profile?: string; pix_fmt?: string; width?: number; height?: number; fps?: number; channels?: number; spherical?: boolean }
+export interface PlayRung { name: string; label: string; width: number; height: number; fps: number; bitrate: number; peak: number }
+export interface PlayStatus { engine?: string; quality?: string; from: number; ahead: number; paused?: boolean }
+// A play: where it is made (node, '' the hub), its qualities best first, and the playlist under
+// that machine's /api.
+export interface PlayOpen {
+  node: string; master: string; engine: string; host: string; note?: string; status: PlayStatus
+  play: { id: string; name: string; device: string; remote: boolean; rungs: PlayRung[]; info: { container: string; duration: number; bit_rate?: number; size?: number; video?: PlayTrack; audio?: PlayTrack } }
+}
+export interface TranscodeDevice { id: string; name: string; integrated?: boolean; ok: boolean; reason?: string }
+export interface TranscodeMachine { node: string; host: string; devices?: TranscodeDevice[]; error?: string }
+export interface TranscodeSetting { selected: { node: string; device: string } | null; machines: TranscodeMachine[] }
 export interface ThumbGPU { api: string; device: string; name?: string; codecs?: string[]; reason?: string; permission?: boolean }
 // How previews get made on the machine: ok with an engine, or why not and what would fix it.
 export interface ThumbStatus { ok: boolean; engine: 'gpu' | 'cpu' | ''; summary: string; ffmpeg?: string; gpus: ThumbGPU[]; unusable?: ThumbGPU[]; error?: string; install?: { pm: string; package: string }; gpu_fix?: boolean; user: string; cache: { path: string; entries: number; bytes: number } }
@@ -281,6 +321,14 @@ export const api = {
     if (!r.ok) { let m = r.statusText; try { m = (await r.json()).error ?? m } catch { /* */ } throw new Error(m) }
     return r.text()
   },
+  // Playing: opened on the hub, which picks the machine (the Transcode device setting); the
+  // rest goes to that machine.
+  play: (path: string, sudo: boolean) => req<PlayOpen>('POST', '/api/play', { node: currentMachine(), path, sudo }, true),
+  playStatus: (node: string, id: string) => req<PlayStatus>('GET', nodeAPI(node, `/api/files/play/${id}`), undefined, true),
+  playStop: (node: string, id: string) => req<void>('DELETE', nodeAPI(node, `/api/files/play/${id}`), undefined, true),
+  transcode: () => req<TranscodeSetting>('GET', '/api/transcode', undefined, true),
+  transcodeSelected: () => req<{ selected: { node: string; device: string } | null; host?: string; name?: string }>('GET', '/api/transcode?only=selected', undefined, true),
+  setTranscode: (node: string, device: string) => req<TranscodeSetting>('PUT', '/api/transcode', device ? { node, device } : {}, true),
   fileReadURL: (path: string, sudo: boolean) => apiPath(`/api/files/read?${q(path, sudo)}`),
   fileDownloadURL: (path: string, sudo: boolean) => apiPath(`/api/files/download?${q(path, sudo)}`),
   // v carries the file's mtime and size, so a changed file is a new URL and an unchanged one a cached one
@@ -288,11 +336,6 @@ export const api = {
   thumbStatus: (sudo: boolean, refresh = false) => req<ThumbStatus>('GET', `/api/files/thumbs?${sudo ? 'sudo=1&' : ''}${refresh ? 'refresh=1' : ''}`),
   thumbInstall: () => req<TaskInfo>('POST', '/api/files/thumbs/install'),
   thumbGPUAccess: () => req<TaskInfo>('POST', '/api/files/thumbs/gpu-access'),
-  // playback of what the browser can't play as it is: an HLS re-encode on the machine
-  playCaps: (sudo: boolean) => req<PlayCaps>('GET', `/api/files/play-caps${sudo ? '?sudo=1' : ''}`),
-  playStart: (path: string, codec: string, quality: string, sudo: boolean) => req<{ stream: PlayStream; url: string }>('POST', `/api/files/play${sudo ? '?sudo=1' : ''}`, { path, codec, quality }),
-  playStatus: (id: string) => req<PlayStatus>('GET', `/api/files/play/${id}`),
-  playStop: (id: string) => req<void>('DELETE', `/api/files/play/${id}`),
   fileWrite: async (path: string, text: string, sudo: boolean) => {
     const r = await fetch(apiPath(`/api/files/write?${q(path, sudo)}`), { method: 'PUT', credentials: 'same-origin', body: text, headers: { 'Content-Type': 'application/octet-stream' } })
     if (r.status === 401) throw new Unauthenticated()
@@ -316,12 +359,13 @@ export const api = {
   storage: () => req<{ devices: StorageDevice[]; zfs: ZFSInfo; smartctl: boolean }>('GET', '/api/storage'),
   smart: (dev: string, force = false) => req<SmartInfo>('GET', `/api/storage/smart/${dev}${force ? '?force=1' : ''}`),
   smartTest: (dev: string, kind: 'short' | 'long') => req<void>('POST', `/api/storage/smart/${dev}/test/${kind}`),
-  // Always the hub's: it holds every machine's KVM, so the tab works when the machine does not.
-  kvms: () => req<KVM[]>('GET', '/api/kvms', undefined, true),
-  addKVM: (name: string, url: string, machine: string) => req<KVM>('POST', '/api/kvms', { name, url, machine }, true),
-  removeKVM: (id: string) => req<void>('DELETE', `/api/kvms/${encodeURIComponent(id)}`, undefined, true),
   zfsOverview: () => req<ZFSOverview>('GET', '/api/zfs'),
   zfsARC: () => req<ARCView>('GET', '/api/zfs/arc'),
+  zfsLanding: () => req<LandingCheck>('GET', '/api/zfs/landing'),
+  landingVerify: (scope: 'new' | 'all') => req<{ started: string }>('POST', '/api/zfs/landing/verify', { scope }),
+  saveVerifySchedule: (p: Partial<VerifyPolicy>) => req<VerifyPolicy>('PUT', '/api/zfs/landing/schedules', p),
+  deleteVerifySchedule: (id: string) => req<void>('DELETE', `/api/zfs/landing/schedules/${encodeURIComponent(id)}`),
+  runVerifySchedule: (id: string) => req<{ started: string }>('POST', `/api/zfs/landing/schedules/${encodeURIComponent(id)}/run`),
   saveZfsSchedule: (p: Partial<ZFSPolicy>) => req<ZFSPolicy>('PUT', '/api/zfs/schedules', p),
   deleteZfsSchedule: (id: string) => req<void>('DELETE', `/api/zfs/schedules/${encodeURIComponent(id)}`),
   runZfsSchedule: (id: string) => req<{ taken: string; pruned: string[] }>('POST', `/api/zfs/schedules/${encodeURIComponent(id)}/run`),

web/src/graphcols.ts

@@ -16,26 +16,40 @@
 // Returns one value per column, oldest first, right-aligned: a young curve has fewer columns
 // than the box and draws in from the right.
 export function columnsOf(values: number[], advance: number, cap: number, cols: number): number[] {
+  return columnsAt(values, advance, advance + 1, cap, cols).cols.slice(-cols)
+}
+
+// The same columns, placed in time. `now` is how far the curve has got in points: `advance`
+// plus the share of the newest point's stride already gone (0–1], so a curve whose points are
+// several ticks apart still moves on every tick instead of jumping once a point. `edge` is how
+// much of the newest column is showing (0–1]: the box draws it that far in from the right and
+// everything else behind it, so the whole graph slides a fraction of a column each tick. One
+// more column than the box holds comes back, to fill the sliver the slide opens on the left.
+// `last` is the newest column's number, for a caller that follows a column as it slides.
+export function columnsAt(values: number[], advance: number, now: number, cap: number, cols: number): { cols: number[]; edge: number; last: number } {
   const n = Math.min(values.length, cap)
-  if (n === 0 || cols < 1 || cap < 1) return []
+  if (n === 0 || cols < 1 || cap < 1) return { cols: [], edge: 1, last: 0 }
   const at = (a: number) => values[values.length - 1 - (advance - a)] // the point numbered a
   const oldest = advance - n + 1
+  const r = cols / cap
   const out: number[] = []
   if (cols >= cap) {
-    const kLast = Math.ceil(((advance + 1) * cols) / cap) - 1
-    for (let k = kLast - cols + 1; k <= kLast; k++) {
+    const kLast = Math.ceil(now * r) - 1
+    for (let k = kLast - cols; k <= kLast; k++) {
       const a = Math.floor((k * cap) / cols)
       if (a >= oldest && a <= advance) out.push(at(a))
     }
-    return out
+    return { cols: out, edge: Math.min(1, Math.max(0, now * r - kLast)) || 1, last: kLast }
   }
+  // the newest column is the newest point's; until the next point lands in the next column the
+  // slide waits at its edge, rather than show a column with nothing in it
   const kLast = Math.floor((advance * cols) / cap)
-  for (let k = kLast - cols + 1; k <= kLast; k++) {
+  for (let k = kLast - cols; k <= kLast; k++) {
     const a0 = Math.max(oldest, Math.ceil((k * cap) / cols)), a1 = Math.min(advance, Math.ceil(((k + 1) * cap) / cols) - 1)
     if (a1 < a0) continue
     let peak = at(a0)
     for (let a = a0 + 1; a <= a1; a++) peak = Math.max(peak, at(a))
     out.push(peak)
   }
-  return out
+  return { cols: out, edge: Math.min(1, Math.max(0, now * r - kLast)) || 1, last: kLast }
 }

web/src/sampleAdapter.ts

@@ -47,11 +47,11 @@ export function sampleOf(v: Values, cat: Catalog): Sample {
     t: Date.now(), interval_ms: num(v, 'hamlab.sample_interval'),
     cpu: num(v, 'cpu.usage'), cores, load: [num(v, 'load.1'), num(v, 'load.5'), num(v, 'load.15')],
     mem: mem('mem'), swap: mem('swap'), uptime: num(v, 'uptime'),
-    temps: hw('temp.').map(({ id, s }) => ({ chip: s.labels!.chip, label: s.labels!.label, c: num(v, id), max: s.warn, crit: s.crit })),
+    temps: hw('temp.').map(({ id, s }) => ({ chip: s.labels!.chip, label: s.labels!.label, c: num(v, id), max: s.warn, crit: s.crit, pkg: s.labels!.package !== undefined ? Number(s.labels!.package) : undefined })),
     fans: hw('fan.').map(({ id, s }) => ({ chip: s.labels!.chip, label: s.labels!.label, rpm: num(v, id) })),
     net: [...groups(v, cat, 'net.', 'iface')].map(([iface, p]) => ({ iface, up: v[p + '.up'] === true, rx_bps: num(v, p + '.rx'), tx_bps: num(v, p + '.tx'), rx_total: num(v, p + '.rx_total'), tx_total: num(v, p + '.tx_total') })),
     disks: [...groups(v, cat, 'disk.', 'device')].map(([name, p]) => ({ name, model: cat.get(p + '.size')?.labels?.model || undefined, size: num(v, p + '.size'), read_bps: num(v, p + '.read'), write_bps: num(v, p + '.write'), util_pct: num(v, p + '.util'), temp: num(v, p + '.temp') || undefined })),
-    fs: [...groups(v, cat, 'fs.', 'mount')].map(([mount, p]) => ({ mount, type: cat.get(p + '.size')?.labels?.fstype ?? '', size: num(v, p + '.size'), used: num(v, p + '.used'), remote: cat.get(p + '.size')?.labels?.remote === '1', disk: cat.get(p + '.size')?.labels?.disk })),
+    fs: [...groups(v, cat, 'fs.', 'mount')].map(([mount, p]) => ({ mount, type: cat.get(p + '.size')?.labels?.fstype ?? '', size: num(v, p + '.size'), used: num(v, p + '.used'), remote: cat.get(p + '.size')?.labels?.remote === '1', disk: cat.get(p + '.size')?.labels?.disk, source: cat.get(p + '.size')?.labels?.source })),
     ups: v['ups.battery'] === undefined ? null : {
       driver: cat.get('ups.last_event')?.labels?.driver ?? '', model: cat.get('ups.battery')?.labels?.model ?? '', state: str(v, 'ups.state'), source: str(v, 'ups.source'),
       on_battery: v['ups.on_battery'] === true, in_v: num(v, 'ups.in_v'), out_v: num(v, 'ups.out_v'), battery_pct: num(v, 'ups.battery'), runtime_min: num(v, 'ups.runtime'),

web/src/sockets.ts

@@ -0,0 +1,17 @@
+// Which threads belong to which CPU, on a machine with more than one.
+//
+// The kernel numbers every thread of the machine in one run, and on a two-socket board with
+// hyper-threading it deals them out a socket at a time and then goes round again: cpu 0–9 are
+// CPU 1's cores, 10–19 CPU 2's, 20–29 the second threads of CPU 1's, 30–39 of CPU 2's. Listed
+// under its CPU by that number, a 20-thread processor reads "C0…C9, C20…C29" and the one
+// beside it "C10…C19, C30…C39" — neither looks like twenty of anything. So each CPU's threads
+// are counted from 0 under it, in the kernel's order, the way a one-socket machine's are: the
+// first half are its cores, the second half their second threads.
+//
+// `pkgs[i]` is the socket logical CPU `i` is on. The answer is, for each socket (ascending),
+// the logical CPUs on it (ascending); a thread's number under its CPU is its place in that list.
+export function socketThreads(pkgs: number[]): { pkg: number; cpus: number[] }[] {
+  const by = new Map<number, number[]>()
+  pkgs.forEach((pk, i) => { const l = by.get(pk); if (l) l.push(i); else by.set(pk, [i]) })
+  return [...by.entries()].sort((a, b) => a[0] - b[0]).map(([pkg, cpus]) => ({ pkg, cpus }))
+}

web/src/styles.css

@@ -5,6 +5,10 @@
   --s1: #3987e5; --s2: #c98500; --track: rgba(57, 135, 229, .18);
   --btn-fg: #111; --term-bg: var(--term-bg); --font: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, "Liberation Mono", monospace;
   --header-h: 44px; --footer-h: 28px; --rail-w: 220px; --tabs-h: 34px;
+  /* iOS 26+ draws a soft blur under the status bar of a Home Screen app that reaches ~16pt
+     below it (hamlet docs/gotchas.md). Whatever starts at the safe area starts below that band
+     too. Zero where there is no status bar to clear: desktop, a Safari tab. */
+  --blur: min(env(safe-area-inset-top), 16px);
 }
 * { box-sizing: border-box; }
 html, body, #root { height: 100%; margin: 0; }
@@ -30,7 +34,7 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
 .svc-list > .error, .cron-list > .error { margin: 8px 12px; }
 
 /* layout */
-.app { height: 100%; display: flex; flex-direction: column; padding-top: env(safe-area-inset-top); }
+.app { height: 100%; display: flex; flex-direction: column; padding-top: calc(env(safe-area-inset-top) + var(--blur)); }
 .header { height: var(--header-h); display: flex; align-items: center; gap: 12px; padding: 0 12px; background: var(--bg2); border-bottom: 1px solid var(--border); }
 .brand { font-weight: 700; color: var(--accent); letter-spacing: .5px; }
 .host { color: var(--muted); font-family: ui-monospace, monospace; font-size: 13px; }
@@ -95,6 +99,7 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
 .dash > * { flex-shrink: 0; }
 .kpis { display: grid; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); gap: 12px; }
 .cards { display: grid; grid-template-columns: repeat(auto-fill, minmax(320px, 1fr)); gap: 12px; align-items: start; }
+.cards.full { grid-template-columns: minmax(0, 1fr); } /* one card per row, the width of the page */
 .tile, .card { background: var(--bg2); border: 1px solid var(--border); border-radius: 8px; padding: 12px; min-width: 0; }
 .tile .label, .card h2 { margin: 0 0 6px; font-size: 12px; font-weight: 600; color: var(--muted); text-transform: uppercase; letter-spacing: .5px; }
 .tile .value { font-size: 22px; font-weight: 600; font-variant-numeric: tabular-nums; line-height: 1.2; }
@@ -145,6 +150,9 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
 .svc-tools { padding: 10px; display: flex; flex-direction: column; gap: 8px; border-bottom: 1px solid var(--border); }
 .svc-tools input { padding: 8px 10px; background: var(--bg); border: 1px solid var(--border); border-radius: 6px; color: var(--fg); font-size: 14px; }
 .chip.on { color: var(--fg); border-color: var(--accent); } .chip.bad { color: var(--bad); }
+.chip.off { opacity: .55; cursor: default; } .chip.off:hover { color: var(--muted); border-color: var(--border); }
+.tc-list { display: flex; flex-direction: column; gap: 12px; }
+.tc-host { font-weight: 600; margin-bottom: 6px; }
 .svc-rows { flex: 1; overflow-y: auto; }
 .svc-row { display: flex; align-items: center; gap: 10px; padding: 8px 12px; cursor: pointer; border-bottom: 1px solid var(--border); }
 .svc-row:hover { background: var(--bg3); } .svc-row.active { background: var(--bg3); box-shadow: inset 3px 0 0 var(--accent); }
@@ -189,6 +197,11 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
 .props span:not(.k) { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } .mono { font-family: ui-monospace, monospace; font-size: 12px; }
 /* One property per row, label above its value. For panes like Inspect, whose values are
    long paths and lists — side by side they were half truncated and read as columns. */
+/* One property per row, label beside its value, the value wrapping rather than cut off: for
+   sentences, like the ZFS tab's write path, that columns would truncate. */
+.props.rows { grid-template-columns: minmax(0, 1fr); gap: 3px 0; }
+.props.rows .k { width: 96px; }
+.props.rows span:not(.k) { overflow: visible; white-space: normal; overflow-wrap: anywhere; }
 .props.stacked { grid-template-columns: 1fr; gap: 10px 0; }
 .props.stacked > div { flex-direction: column; gap: 2px; }
 .props.stacked .k { width: auto; font-size: 11px; letter-spacing: .04em; text-transform: uppercase; }
@@ -261,7 +274,7 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
    than what is left, and scrolls inside itself. Its parts keep their height: in a column that
    may shrink, anything with its own scrollbar (a preview, a list) is squeezed to a sliver
    instead, and the dialog never scrolls at all. */
-.modal-back { position: fixed; inset: 0; z-index: 50; background: rgba(0,0,0,.6); display: flex; align-items: center; justify-content: center; padding: max(16px, env(safe-area-inset-top)) max(16px, env(safe-area-inset-right)) max(16px, env(safe-area-inset-bottom)) max(16px, env(safe-area-inset-left)); }
+.modal-back { position: fixed; inset: 0; z-index: 50; background: rgba(0,0,0,.6); display: flex; align-items: center; justify-content: center; padding: max(16px, calc(env(safe-area-inset-top) + var(--blur))) max(16px, env(safe-area-inset-right)) max(16px, env(safe-area-inset-bottom)) max(16px, env(safe-area-inset-left)); }
 .modal { --modal-bg: var(--bg2); width: min(420px, 100%); max-height: 100%; overflow-y: auto; overscroll-behavior: contain; background: var(--modal-bg); border: 1px solid var(--border); border-radius: 10px; padding: 16px; display: flex; flex-direction: column; gap: 12px; }
 .modal > * { flex-shrink: 0; }
 /* a long dialog's buttons (and what went wrong) stay in reach while the rest scrolls */
@@ -365,13 +378,53 @@ h2.section { margin: 0; font-size: 13px; font-weight: 600; color: var(--muted);
 .preview { flex: 1; display: flex; align-items: center; justify-content: center; overflow: auto; padding: 12px; }
 .preview img { max-width: 100%; max-height: 100%; object-fit: contain; }
 .preview video { max-width: 100%; max-height: 100%; background: #000; border-radius: 4px; outline: none; }
-.player { flex: 1; min-height: 0; display: flex; flex-direction: column; }
-.player .preview { flex: 1; min-height: 0; }
-.play-bar { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; padding: 6px 12px; font-size: 12px; color: var(--muted); border-bottom: 1px solid var(--border); }
-.play-bar select { padding: 4px 8px; background: var(--bg); border: 1px solid var(--border); border-radius: 5px; color: var(--fg); font-size: 12px; }
-.play-pick { margin: 12px; }
-.play-pick .row { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; margin-top: 10px; }
-.play-pick label { display: flex; align-items: center; gap: 6px; font-size: 12px; color: var(--muted); }
+.preview.pic { position: relative; touch-action: pan-y pinch-zoom; }
+.step-edge { position: absolute; top: 50%; transform: translateY(-50%); width: 44px; height: 72px; border: 0; border-radius: 6px; background: rgba(0, 0, 0, .35); color: #fff; font-size: 34px; line-height: 1; cursor: pointer; opacity: .55; }
+.step-edge:hover { opacity: 1; } .step-edge.prev { left: 8px; } .step-edge.next { right: 8px; }
+.btn.step:disabled { opacity: .4; }
+/* the video player (Player.tsx): Ham Share's, in the viewer; full screen is the browser's, or
+   on an iPhone the whole window */
+.vp { position: relative; flex: 1; min-height: 240px; background: #000; color: #fff; overflow: hidden; user-select: none; -webkit-user-select: none; outline: none; }
+.vp.max { position: fixed; inset: 0; z-index: 1000; }
+.vp.max .vp-note { top: calc(env(safe-area-inset-top) + var(--blur) + 14px); }
+.vp-v { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: contain; background: #000; }
+.vp-hidden { opacity: 0; pointer-events: none; }
+.vp-sphere { touch-action: none; cursor: grab; }
+.vp-sphere:active { cursor: grabbing; }
+.vp-bottom { position: absolute; left: 0; right: 0; bottom: 0; padding: 26px 8px calc(env(safe-area-inset-bottom) + 4px); background: linear-gradient(transparent, rgba(0, 0, 0, .75)); transition: opacity .25s; z-index: 2; }
+.vp.bare .vp-bottom, .vp.bare .vp-hint { opacity: 0; pointer-events: none; }
+.vp.bare { cursor: none; }
+.vp-row { display: flex; align-items: center; gap: 4px; }
+.vp-icon { width: 38px; height: 38px; display: grid; place-items: center; border-radius: 19px; color: #fff; }
+.vp-icon:hover { background: rgba(255, 255, 255, .14); }
+.vp-icon.on { color: var(--accent); }
+.vp-time { font-size: 13px; font-variant-numeric: tabular-nums; padding: 0 6px; white-space: nowrap; }
+.vp-q { display: inline-flex; align-items: center; gap: 6px; padding: 7px 10px; border-radius: 17px; font-size: 13px; font-weight: 600; white-space: nowrap; color: #fff; }
+.vp-q:hover { background: rgba(255, 255, 255, .14); }
+.vp-seek { position: relative; height: 24px; margin: 0 8px; display: flex; align-items: center; }
+.vp-seek::before { content: ''; position: absolute; left: 0; right: 0; height: 4px; border-radius: 2px;
+  background: linear-gradient(to right, var(--accent) var(--p), rgba(255, 255, 255, .45) var(--p) var(--b), rgba(255, 255, 255, .2) var(--b)); }
+.vp-seek input { position: relative; width: 100%; margin: 0; background: transparent; border: 0; padding: 0; height: 24px; -webkit-appearance: none; appearance: none; cursor: pointer; }
+.vp-seek input::-webkit-slider-runnable-track { height: 4px; background: transparent; }
+.vp-seek input::-moz-range-track { height: 4px; background: transparent; }
+.vp-seek input::-webkit-slider-thumb { -webkit-appearance: none; width: 14px; height: 14px; margin-top: -5px; border-radius: 50%; background: var(--accent); border: 0; }
+.vp-seek input::-moz-range-thumb { width: 14px; height: 14px; border-radius: 50%; background: var(--accent); border: 0; }
+.vp-big { position: absolute; left: 50%; top: 50%; width: 68px; height: 68px; margin: -34px 0 0 -34px; border-radius: 50%; display: grid; place-items: center; background: rgba(0, 0, 0, .55); color: #fff; z-index: 2; padding-left: 4px; }
+.vp-spin { position: absolute; left: 50%; top: 50%; width: 46px; height: 46px; margin: -23px 0 0 -23px; border-radius: 50%; border: 4px solid rgba(255, 255, 255, .2); border-top-color: var(--accent); animation: vp-spin .8s linear infinite; z-index: 1; pointer-events: none; }
+@keyframes vp-spin { to { transform: rotate(360deg); } }
+.vp-err { position: absolute; inset: 0; display: flex; align-items: center; justify-content: center; padding: 24px; text-align: center; z-index: 1; }
+.vp-err p { margin: 0; max-width: 460px; line-height: 1.45; }
+.vp-note { position: absolute; left: 50%; top: 14px; transform: translateX(-50%); max-width: min(92%, 480px); padding: 9px 13px; border-radius: 8px; background: rgba(20, 20, 20, .92); font-size: 13px; line-height: 1.4; z-index: 3; cursor: pointer; }
+.vp-hint { position: absolute; left: 50%; bottom: 86px; transform: translateX(-50%); padding: 5px 11px; border-radius: 14px; background: rgba(0, 0, 0, .55); font-size: 12px; pointer-events: none; transition: opacity .25s; z-index: 1; white-space: nowrap; }
+.vp-menu { position: absolute; right: 8px; bottom: 60px; width: min(290px, calc(100% - 16px)); max-height: calc(100% - 76px); overflow-y: auto; padding: 6px; border-radius: 10px; background: rgba(24, 24, 24, .96); box-shadow: 0 8px 30px rgba(0, 0, 0, .5); z-index: 4; }
+.vp-menu-h { font-size: 11px; font-weight: 700; letter-spacing: .5px; text-transform: uppercase; color: rgba(255, 255, 255, .55); padding: 6px 10px 4px; }
+.vp-menu button { display: flex; align-items: center; gap: 8px; width: 100%; padding: 8px 10px; border-radius: 6px; text-align: left; font-size: 14px; color: #fff; }
+.vp-menu button:hover:not(:disabled) { background: rgba(255, 255, 255, .1); }
+.vp-menu button:disabled { opacity: .45; }
+.vp-menu button small { margin-left: auto; color: rgba(255, 255, 255, .6); font-size: 12px; white-space: nowrap; }
+.vp-menu .ck { width: 16px; flex: none; color: var(--accent); display: grid; place-items: center; }
+.vp-menu-f { border-top: 1px solid rgba(255, 255, 255, .12); margin-top: 6px; padding: 8px 10px 4px; font-size: 12px; color: rgba(255, 255, 255, .6); line-height: 1.4; }
+.vp-menu-f b { color: #fff; font-weight: 600; }
 .preview.audio { align-items: flex-start; }
 .audio-box { display: flex; flex-direction: column; align-items: center; gap: 10px; width: min(560px, 100%); margin-top: 8vh; }
 .audio-box .ficon { font-size: 48px; }
@@ -741,6 +794,11 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
 .t-cores-head b { color: var(--fg); font-weight: 400; overflow: hidden; text-overflow: ellipsis; }
 .t-core { display: flex; align-items: center; gap: 6px; }
 .t-core-n { width: 30px; } .t-core-p { width: 34px; text-align: right; }
+/* a two-socket board: a row per socket — arrow, name, watts — with its threads indented under it */
+.t-cores.multi .t-core-n { width: 50px; } .t-cores.multi .t-core:not(.t-sock) .t-core-n { padding-left: 14px; box-sizing: border-box; }
+.t-sock { color: var(--fg); margin-top: 2px; cursor: pointer; user-select: none; }
+.t-fold { display: inline-block; width: 12px; color: var(--muted); }
+.t-core-w { width: 40px; flex-shrink: 0; text-align: right; color: var(--muted); }
 .t-core-g { flex: 1; color: var(--ok); letter-spacing: 0; overflow: hidden; white-space: nowrap; }
 .t-core .tm { flex: 1; } .t-core-t { flex: 0 0 62px; display: flex; } .t-core-c { width: 42px; text-align: right; color: var(--t-blue); }
 .t-load { display: flex; margin-top: 2px; }
@@ -841,6 +899,8 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
 .arc-tier { display: flex; gap: 8px; min-width: 0; } .arc-tier .k { color: var(--muted); width: 80px; flex-shrink: 0; cursor: help; }
 .arc-tier-v { display: flex; flex-direction: column; gap: 1px; min-width: 0; } .arc-tier-v > span { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
 .arc-tier-v .none { color: var(--muted); white-space: normal; }
+.landing-tier { border: 1px solid var(--border); border-radius: 8px; padding: 8px 10px; margin: 4px 0 6px; font-size: 13px; }
+.landing-tier .props .badge { margin-right: 2px; }
 
 /* ---- Space: what is using a disk ---- */
 .space { position: absolute; inset: 0; display: flex; flex-direction: column; background: var(--bg); min-height: 0; }
@@ -908,21 +968,33 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
 .modal select, .modal input[type=time], .modal input[type=number] { padding: 9px 10px; background: var(--bg); border: 1px solid var(--border); border-radius: 6px; color: var(--fg); font-size: 16px; }
 .modal .chips { margin-top: 2px; }
 
-/* ---- KVM: a device's own web page, framed ---- */
-.kvm { position: absolute; inset: 0; display: flex; flex-direction: column; background: var(--bg); min-height: 0; }
-.kvm-bar { display: flex; align-items: center; gap: 8px; padding: 8px 12px; border-bottom: 1px solid var(--border); background: var(--bg2); flex-wrap: wrap; }
-.kvm-bar .kvm-url { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; max-width: 28ch; }
-.kvm-frame { flex: 1; min-height: 0; background: #000; position: relative; }
-.kvm-frame iframe { width: 100%; height: 100%; border: 0; display: block; background: #000; }
-.kvm-frame.full { background: #000; }
-.kvm-hint { padding: 6px 12px; border-top: 1px solid var(--border); }
-@media (max-width: 800px) { .kvm-bar .kvm-url { display: none; } }
-.kvm-none { text-align: center; } .kvm-none p { margin: 4px 0; }
-/* No true fullscreen for a page element (Safari on iOS): fill the viewport in-page instead. */
-.kvm-frame.expanded { position: fixed; inset: 0; z-index: 60; }
-.kvm-frame.expanded iframe { height: 100%; }
-.kvm-shrink { position: absolute; right: max(10px, env(safe-area-inset-right)); bottom: max(10px, env(safe-area-inset-bottom)); z-index: 61; background: var(--bg2); border: 1px solid var(--border); opacity: .75; }
-.kvm-shrink:hover, .kvm-shrink:active { opacity: 1; }
 .updot { margin-left: 6px; font-size: 11px; font-weight: 700; }
 .updot.behind { color: var(--s2); } .updot.unknown { color: var(--muted); }
 
+/* a phone: the time goes above the seek bar, so the buttons fit in one row */
+.vp-time.above { display: none; }
+@media (max-width: 560px) {
+  .vp-time.above { display: block; padding: 0 10px 2px; }
+  .vp-time.inrow { display: none; }
+}
+
+
+/* No theme touches playback, whatever it draws. A theme's full-window overlays (the Retro
+   scanlines today, any other tomorrow) are off while a player is open; the picture is never
+   filtered or blended; the player's text is plain, not the theme's lettering. Only the accent
+   colour of its controls follows the theme. */
+html:has(.vp)::before, html:has(.vp)::after, body:has(.vp)::before, body:has(.vp)::after,
+.app:has(.vp)::before, .app:has(.vp)::after, #root:has(.vp)::before, #root:has(.vp)::after { display: none !important; }
+.vp { isolation: isolate; filter: none !important; mix-blend-mode: normal !important; }
+.vp video, .vp canvas { filter: none !important; mix-blend-mode: normal !important; }
+.vp { font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; font-weight: 400; letter-spacing: normal; text-shadow: none; }
+.vp button, .vp input, .vp select { font-family: inherit !important; }
+
+/* volume: the speaker mutes; the slider beside it (not on iOS, which only mutes) sets the level */
+.vp-vol { display: inline-flex; align-items: center; }
+.vp-vol input { width: 72px; height: 24px; margin: 0 6px 0 0; padding: 0; background: transparent; border: 0; -webkit-appearance: none; appearance: none; cursor: pointer; }
+.vp-vol input::-webkit-slider-runnable-track { height: 4px; border-radius: 2px; background: linear-gradient(to right, #fff var(--v), rgba(255, 255, 255, .3) var(--v)); }
+.vp-vol input::-moz-range-track { height: 4px; border-radius: 2px; background: linear-gradient(to right, #fff var(--v), rgba(255, 255, 255, .3) var(--v)); }
+.vp-vol input::-webkit-slider-thumb { -webkit-appearance: none; width: 12px; height: 12px; margin-top: -4px; border-radius: 50%; background: #fff; border: 0; }
+.vp-vol input::-moz-range-thumb { width: 12px; height: 12px; border-radius: 50%; background: #fff; border: 0; }
+@media (max-width: 560px) { .vp-vol input { display: none; } }

web/src/vr.ts

@@ -0,0 +1,241 @@
+// The 360° view: an equirectangular video drawn onto a canvas as the flat picture a person
+// would see looking out from the middle of it. Drag (or swipe) to look around, pinch or scroll
+// to zoom, and on a phone, optionally, turn the phone. One full-canvas triangle pair; the
+// fragment shader works out, for each pixel, which way it looks and where that is in the video.
+
+const VERT = `
+attribute vec2 a;
+varying vec2 p;
+void main() { p = a; gl_Position = vec4(a, 0.0, 1.0); }`
+
+const FRAG = `
+precision highp float;
+uniform sampler2D tex;
+uniform mat3 view;  // columns: right, up, forward
+uniform vec2 aspect; // the screen's sides over its taller side: (w/h, 1) or (1, h/w)
+uniform float fov;   // radians, across the taller side
+varying vec2 p;
+const float PI = 3.14159265358979;
+void main() {
+  vec2 q = p * aspect;
+  // Narrow: an ordinary lens. Wide: stereographic, which goes on past 180 degrees to the
+  // "little planet". Between 100 and 140 degrees one turns into the other.
+  vec3 rect = normalize(vec3(q * tan(min(fov, 3.0) * 0.5), 1.0));
+  float r = length(q) * 2.0 * tan(fov * 0.25);
+  float th = 2.0 * atan(r * 0.5);
+  vec2 dir = r > 0.0 ? q / length(q) : vec2(0.0);
+  vec3 stereo = vec3(dir * sin(th), cos(th));
+  float t = smoothstep(1.745, 2.443, fov);
+  vec3 d = normalize(view * normalize(mix(rect, stereo, t)));
+  float lon = atan(d.x, -d.z);
+  float lat = asin(clamp(d.y, -1.0, 1.0));
+  gl_FragColor = texture2D(tex, vec2(lon / (2.0 * PI) + 0.5, 0.5 - lat / PI));
+}`
+
+// From a close look at something far off to the whole sphere as a little planet.
+const MIN_FOV = 8, MAX_FOV = 300
+
+type Vec3 = [number, number, number]
+type Quat = [number, number, number, number] // x, y, z, w
+
+function qmul(a: Quat, b: Quat): Quat {
+  const [ax, ay, az, aw] = a, [bx, by, bz, bw] = b
+  return [aw * bx + ax * bw + ay * bz - az * by, aw * by - ax * bz + ay * bw + az * bx, aw * bz + ax * by - ay * bx + az * bw, aw * bw - ax * bx - ay * by - az * bz]
+}
+function qrot(q: Quat, v: Vec3): Vec3 {
+  const [x, y, z, w] = q
+  const ix = w * v[0] + y * v[2] - z * v[1], iy = w * v[1] + z * v[0] - x * v[2], iz = w * v[2] + x * v[1] - y * v[0], iw = -x * v[0] - y * v[1] - z * v[2]
+  return [ix * w + iw * -x + iy * -z - iz * -y, iy * w + iw * -y + iz * -x - ix * -z, iz * w + iw * -z + ix * -y - iy * -x]
+}
+const axis = (v: Vec3, rad: number): Quat => { const s = Math.sin(rad / 2); return [v[0] * s, v[1] * s, v[2] * s, Math.cos(rad / 2)] }
+
+// The phone's orientation as a camera's, the way the W3C spec defines the angles: Z-X'-Y''
+// intrinsic, then turned to look out of the back of the phone, then for the screen's rotation.
+function deviceQuat(alpha: number, beta: number, gamma: number, screen: number): Quat {
+  const d = Math.PI / 180
+  const [a, b, g] = [alpha * d, beta * d, gamma * d]
+  let q = qmul(qmul(axis([0, 1, 0], a), axis([1, 0, 0], b)), axis([0, 0, 1], -g)) // YXZ, in three.js's frame
+  q = qmul(q, [-Math.SQRT1_2, 0, 0, Math.SQRT1_2])
+  return qmul(q, axis([0, 0, 1], -screen * d))
+}
+
+export class Sphere {
+  yaw = 0 // degrees, + is right
+  pitch = 0 // degrees, + is up
+  fov = 80 // degrees, the taller of the two sides
+  private gl: WebGLRenderingContext
+  private tex: WebGLTexture
+  private uView: WebGLUniformLocation
+  private uAspect: WebGLUniformLocation
+  private uFov: WebGLUniformLocation
+  private raf = 0
+  private frameCb = 0
+  private dirty = true
+  private hasFrame = false
+  private gyro: { alpha: number; beta: number; gamma: number } | null = null
+  private gyroYaw = 0 // what dragging adds to the phone's own heading
+  private ro: ResizeObserver
+  maxTexture: number
+
+  constructor(private canvas: HTMLCanvasElement, private video: HTMLVideoElement) {
+    const gl = canvas.getContext('webgl', { antialias: false, alpha: false, preserveDrawingBuffer: false })
+    if (!gl) throw new Error('This browser can’t draw 360° video (no WebGL).')
+    this.gl = gl
+    this.maxTexture = gl.getParameter(gl.MAX_TEXTURE_SIZE) as number
+    const sh = (type: number, src: string) => {
+      const s = gl.createShader(type)!
+      gl.shaderSource(s, src); gl.compileShader(s)
+      if (!gl.getShaderParameter(s, gl.COMPILE_STATUS)) throw new Error(gl.getShaderInfoLog(s) ?? 'shader')
+      return s
+    }
+    const prog = gl.createProgram()!
+    gl.attachShader(prog, sh(gl.VERTEX_SHADER, VERT)); gl.attachShader(prog, sh(gl.FRAGMENT_SHADER, FRAG))
+    gl.linkProgram(prog); gl.useProgram(prog)
+    const buf = gl.createBuffer()
+    gl.bindBuffer(gl.ARRAY_BUFFER, buf)
+    gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1, -1, 1, -1, -1, 1, -1, 1, 1, -1, 1, 1]), gl.STATIC_DRAW)
+    const loc = gl.getAttribLocation(prog, 'a')
+    gl.enableVertexAttribArray(loc); gl.vertexAttribPointer(loc, 2, gl.FLOAT, false, 0, 0)
+    this.tex = gl.createTexture()!
+    gl.bindTexture(gl.TEXTURE_2D, this.tex)
+    // No mipmaps (a video changes every frame) and clamped: any size of picture will do.
+    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR)
+    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR)
+    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE)
+    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE)
+    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB, 1, 1, 0, gl.RGB, gl.UNSIGNED_BYTE, new Uint8Array([0, 0, 0]))
+    this.uView = gl.getUniformLocation(prog, 'view')!
+    this.uAspect = gl.getUniformLocation(prog, 'aspect')!
+    this.uFov = gl.getUniformLocation(prog, 'fov')!
+    this.ro = new ResizeObserver(() => { this.dirty = true })
+    this.ro.observe(canvas)
+    this.watchFrames()
+    const loop = () => { this.draw(); this.raf = requestAnimationFrame(loop) }
+    this.raf = requestAnimationFrame(loop)
+  }
+
+  // A new picture from the video: on the frame callback where there is one, else every draw
+  // while it plays.
+  private watchFrames() {
+    const v = this.video as HTMLVideoElement & { requestVideoFrameCallback?: (cb: () => void) => number }
+    if (v.requestVideoFrameCallback) {
+      const cb = () => { this.upload(); this.frameCb = v.requestVideoFrameCallback!(cb) }
+      this.frameCb = v.requestVideoFrameCallback(cb)
+    }
+    v.addEventListener('loadeddata', this.upload)
+    v.addEventListener('seeked', this.upload)
+  }
+
+  private upload = () => {
+    const v = this.video
+    if (v.readyState < 2 || !v.videoWidth) return
+    const gl = this.gl
+    gl.bindTexture(gl.TEXTURE_2D, this.tex)
+    try {
+      gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB, gl.RGB, gl.UNSIGNED_BYTE, v)
+      this.hasFrame = true
+      this.dirty = true
+    } catch { /* not ready yet */ }
+  }
+
+  private draw() {
+    const v = this.video as HTMLVideoElement & { requestVideoFrameCallback?: unknown }
+    if (!v.requestVideoFrameCallback && !v.paused) this.upload()
+    if (!this.dirty && !this.gyro) return
+    this.dirty = false
+    const c = this.canvas, dpr = Math.min(window.devicePixelRatio || 1, 2)
+    const w = Math.round(c.clientWidth * dpr), h = Math.round(c.clientHeight * dpr)
+    if (!w || !h) return
+    if (c.width !== w || c.height !== h) { c.width = w; c.height = h }
+    const gl = this.gl
+    gl.viewport(0, 0, w, h)
+    if (!this.hasFrame) { gl.clearColor(0, 0, 0, 1); gl.clear(gl.COLOR_BUFFER_BIT); return }
+    // The field of view applies to the taller side; on a phone held upright that is the height.
+    let right: Vec3, up: Vec3, fwd: Vec3
+    if (this.gyro) {
+      const q = qmul(axis([0, 1, 0], (-this.gyroYaw * Math.PI) / 180), deviceQuat(this.gyro.alpha, this.gyro.beta, this.gyro.gamma, screenAngle()))
+      right = qrot(q, [1, 0, 0]); up = qrot(q, [0, 1, 0]); fwd = qrot(q, [0, 0, -1])
+    } else {
+      const y = (this.yaw * Math.PI) / 180, p = (this.pitch * Math.PI) / 180
+      fwd = [Math.sin(y) * Math.cos(p), Math.sin(p), -Math.cos(y) * Math.cos(p)]
+      right = [Math.cos(y), 0, Math.sin(y)]
+      up = [right[1] * fwd[2] - right[2] * fwd[1], right[2] * fwd[0] - right[0] * fwd[2], right[0] * fwd[1] - right[1] * fwd[0]]
+    }
+    // d = right·x + up·y + fwd·1, column-major
+    gl.uniformMatrix3fv(this.uView, false, new Float32Array([...right, ...up, ...fwd]))
+    gl.uniform2f(this.uAspect, w >= h ? 1 : w / h, w >= h ? h / w : 1)
+    gl.uniform1f(this.uFov, (this.fov * Math.PI) / 180)
+    gl.drawArrays(gl.TRIANGLES, 0, 6)
+  }
+
+  // Dragging by a pixel turns the view by the angle a pixel covers, so what is under the finger
+  // stays under it.
+  drag(dx: number, dy: number) {
+    const per = this.fov / Math.max(this.canvas.clientWidth, this.canvas.clientHeight)
+    if (this.gyro) this.gyroYaw = (this.gyroYaw - dx * per) % 360
+    else {
+      this.yaw = (this.yaw - dx * per) % 360
+      this.pitch = Math.max(-89, Math.min(89, this.pitch + dy * per))
+    }
+    this.dirty = true
+  }
+
+  zoom(factor: number) {
+    this.fov = Math.max(MIN_FOV, Math.min(MAX_FOV, this.fov * factor))
+    this.dirty = true
+  }
+
+  get gyroOn() { return this.gyro !== null }
+
+  private onOrient = (e: DeviceOrientationEvent) => {
+    if (e.alpha === null || e.beta === null || e.gamma === null) return
+    if (!this.gyro) {
+      // Keep looking where the drag left off: the phone's heading now is the drag's yaw.
+      const q = deviceQuat(e.alpha, e.beta, e.gamma, screenAngle())
+      const f = qrot(q, [0, 0, -1])
+      this.gyroYaw = this.yaw - (Math.atan2(f[0], -f[2]) * 180) / Math.PI
+    }
+    this.gyro = { alpha: e.alpha, beta: e.beta, gamma: e.gamma }
+  }
+
+  // Turning on motion asks for permission on an iPhone, which must happen in a tap.
+  async setGyro(on: boolean): Promise<boolean> {
+    if (!on) {
+      window.removeEventListener('deviceorientation', this.onOrient)
+      if (this.gyro) {
+        const q = qmul(axis([0, 1, 0], (-this.gyroYaw * Math.PI) / 180), deviceQuat(this.gyro.alpha, this.gyro.beta, this.gyro.gamma, screenAngle()))
+        const f = qrot(q, [0, 0, -1])
+        this.yaw = (Math.atan2(f[0], -f[2]) * 180) / Math.PI
+        this.pitch = (Math.asin(Math.max(-1, Math.min(1, f[1]))) * 180) / Math.PI
+      }
+      this.gyro = null
+      this.dirty = true
+      return false
+    }
+    const D = DeviceOrientationEvent as unknown as { requestPermission?: () => Promise<string> }
+    if (D.requestPermission) {
+      try { if ((await D.requestPermission()) !== 'granted') return false } catch { return false }
+    }
+    window.addEventListener('deviceorientation', this.onOrient)
+    return true
+  }
+
+  destroy() {
+    cancelAnimationFrame(this.raf)
+    const v = this.video as HTMLVideoElement & { cancelVideoFrameCallback?: (n: number) => void }
+    if (this.frameCb && v.cancelVideoFrameCallback) v.cancelVideoFrameCallback(this.frameCb)
+    v.removeEventListener('loadeddata', this.upload)
+    v.removeEventListener('seeked', this.upload)
+    window.removeEventListener('deviceorientation', this.onOrient)
+    this.ro.disconnect()
+    this.gl.getExtension('WEBGL_lose_context')?.loseContext()
+  }
+}
+
+function screenAngle(): number {
+  const o = screen.orientation?.angle
+  if (typeof o === 'number') return o
+  return (window as unknown as { orientation?: number }).orientation ?? 0
+}
+
+export const canMotion = () => typeof DeviceOrientationEvent !== 'undefined' && matchMedia('(pointer: coarse)').matches