27 Commits
Author SHA1 Message Date
Mitchell-JohnsonandClaude Opus 5.5 afd4688358 GPUs: fan speed in % per card (nvidia-smi fan.speed, hwmon pwm1); 0% shows
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:09:10 -08:00
Mitchell-JohnsonandClaude Opus 5.5 95bb026b7f Docs: Ham Lab on core is lab.bigham.dev (was hamlab.bigham.dev)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:47:48 -08:00
Mitchell-JohnsonandClaude Opus 5.5 a6d7ff436f Drive tile counts each source device once (a bind mount doubled nexus's NVMe)
fs.* sensors carry a source label; Top.tsx dedupes by it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 00:06:33 -08:00
Mitchell-JohnsonandClaude Opus 5.5 574d9855ce GPU: a slow read is late, not stalled; a stall clears when the read returns
nvidia-smi on a busy card can take past the 3 s deadline, which latched the card
as stalled until restart. A late read now carries the last readings and is used
when it lands; no second read starts while one is out; only one out for 15 s
(past both nvidia-smi timeouts) marks the card stalled, and its return clears it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 18:42:13 -08:00
Mitchell-JohnsonandClaude Opus 5.5 ad576d8c1a Top: drop the GPU busy caption; keep only the stall warning
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:55:57 -08:00
Mitchell-JohnsonandClaude c40de5067b Top: each CPU's threads are counted from 0 under it on a two-socket machine
The kernel numbers the whole machine's threads in one run and deals them out a
socket at a time, twice: nexus's first CPU listed C0–C9 and C20–C29, its second
C10–C19 and C30–C39. Each CPU now lists its own twenty as C0–C19; the kernel's
number is in the row's tooltip.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-10 12:19:49 -08:00
Mitchell-JohnsonandClaude Opus 5.5 0e57419d8b Top: the second of two identical GPUs draws its busy graph
Its sensors were gpu.<name>.usage_2 and so on; the page looks under gpu.<name>_2. Each
card now gets its own stem, as the page already assumed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 00:40:56 -08:00
Mitchell-JohnsonandClaude Opus 5.5 f1c66ce886 Top: the second CPU's core temperatures on a two-socket machine
The sensor feed dropped which socket a coretemp reading came from, so every core matched
the first CPU's chip. coretemp sensors now carry a package label, and history folding
keeps the two sockets' same-named cores apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 00:22:05 -08:00
Mitchell-JohnsonandClaude Opus 5.5 a516cf4cad Top: a two-socket machine shows only its two CPUs, no combined row or wattage
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 00:09:09 -08:00
Mitchell-JohnsonandClaude Opus 5.5 e9a87af5e8 Top: each CPU's own wattage on a two-socket machine, its threads folded under it
Per-package RAPL watts in the sample, the process table and as cpu.package.N.power
sensors. The socket rows collapse with an arrow; the fold is remembered per browser.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 00:07:24 -08:00
Mitchell-JohnsonandClaude Opus 5.5 1e2dcacb2c Two-socket machines: Hardware and Top show each CPU, not one with every core
Hardware counts CPUs, cores and threads per CPU. Top groups threads under a row
per socket, reads each core's temperature from its own socket's coretemp chip,
and the CPU temperature is the hotter package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 00:04:47 -08:00
Mitchell-JohnsonandClaude Opus 5.5 4b238ea93e Remove the KVM tab: proxy, routes, config and state fields, UI
Not needed — the KVMs are reached over WireGuard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 23:52:38 -08:00
Mitchell-JohnsonandClaude Opus 5.5 5edad2de6a Files viewer: previous/next — buttons, ← →, swipe on pictures; steps through media when media is open; audio autoplays
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 01:27:26 -07:00
Mitchell-JohnsonandClaude Opus 5.5 829617c932 ZFS tab: landing tier checks — stale files, verify passes against the mover's ledger, verify schedules
A verify pass reads the pool's copy of every file the mover moved back against the SHA-256 the
mover took of the landing copy (its ledger, named in its status file): match, mismatch, edited
since, gone, waiting, unreadable, unhashed. 'New moves' checks what moved since the last pass,
'everything' the whole ledger. Passes run as 'hamlab landing-verify' in their own session — the
same command a schedule's line in /etc/cron.d/hamlab-landing runs — so a console restart does
not cut one short. Stale files: anything waiting on the landing branch over an hour, listed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 23:16:55 -07:00
Mitchell-JohnsonandClaude Opus 5.5 f518df239f SENSORS: the ZFS write path's sensors
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 22:41:23 -07:00
Mitchell-JohnsonandClaude Opus 5.5 41cb734f7f ZFS tab: show a mover's rsync limit only when there is one, in the units rsync means (60M = 60 MiB/s)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 22:39:33 -07:00
Mitchell-JohnsonandClaude Opus 5.5 0698f07415 ZFS tab: the write path's details one per row, wrapping instead of cut off
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 22:37:22 -07:00
Mitchell-JohnsonandClaude Opus 5.5 b2528afe2a ZFS tab: snapshot schedules full width too — every box on the tab spans the page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 22:34:55 -07:00
Mitchell-JohnsonandClaude Opus 5.5 50768e2d94 ZFS tab: pools, caching tiers and writes full width; write and drain rates measured between real updates
A txg records its bytes only when it syncs and the mover reports every 2 s, so a per-sample delta
read 0 most of the time. Rates are now taken over the gap between syncs (or mover updates) and held
until the next one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 22:34:11 -07:00
Mitchell-JohnsonandClaude Opus 5.5 86cfcd2bad ZFS tab: Writes — the RAM write buffer, each pool's write rate, and a landing tier with its mover
Live from arcstats (dirty), dmu_tx (throttled writes), each pool's txgs (bytes synced to disk)
and objset kstats (bytes from programs). A landing tier — a mergerfs pool whose first branch is
fast storage and whose last is a ZFS dataset — is detected from mergerfs's branch xattr; a mover
status file in /run/landing-mover/ adds its queue, the file in flight and totals. Sensors
zfs.write.*, zfs.<pool>.write_bps|logical_bps and zfs.<pool>.landing.* carry the history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 22:28:40 -07:00
Mitchell-JohnsonandClaude Opus 5.5 1a9b31ea0a PLAN: handle fuse.mergerfs (grid's landing pool) in Storage
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 21:01:59 -07:00
Mitchell-JohnsonandClaude Opus 5.5 695e4c4f57 e2e-player: the volume slider is only there wider than a phone
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 15:49:00 -07:00
Mitchell-JohnsonandClaude Opus 5.5 432ddc0df4 files: the player's volume — mute, a level slider, remembered (as Ham Share's)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 15:43:05 -07:00
Mitchell-JohnsonandClaude Opus 5.5 1c5cd2b0de No theme touches playback, any theme; the header clears the iOS status-bar blur
Every theme's full-window overlays are off while a video is open, the picture
is never filtered or blended, the player's lettering is plain. --blur, as in To
Do and Notes: the header (the top row of the app), dialogs, the full-screen
player's note and an expanded KVM start below iOS 26's scroll-edge blur in a
Home Screen app; zero on desktop and in a Safari tab.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 00:05:04 -07:00
Mitchell-JohnsonandClaude Opus 5.5 01e0da9022 files: no theme touches playback — Retro's scanlines off while a video is open, plain font
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 00:02:15 -07:00
Mitchell-JohnsonandClaude Opus 5.5 dbf89cd5e2 files: on a phone the player's time sits above the seek bar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 23:52:35 -07:00
Mitchell-JohnsonandClaude Opus 5.5 033343793f files: the player's −5 s and +5 s buttons, as Ham Share's
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 23:48:37 -07:00
38 changed files with 2789 additions and 995 deletions
+11 -36
View File
@@ -16,7 +16,7 @@ stub tab or a roadmap item.
| Service | `hamlab.service`, `User=core`, `Restart=always`, binary at `/usr/local/bin/hamlab` | Same pattern as `calorie-counter.service`. Runs as `core` so `sudo` works in the terminal without a password, as on the SSH path. |
| Listen | `10.0.0.234:8100` (free) | NPM is a container, so loopback-only won't do. Firewalling to NPM is a later hardening step. |
| Data | `/home/core/apps/hamlab/` — `config.yaml`, `state.db` (SQLite) | `apps/` is the nightly rsync target to `grid`, so it's backed up for free. |
| Ingress | NPM proxy host `hamlab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
| Ingress | NPM proxy host `lab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
| Repo | Gitea `mitchell/hamlab`, private, `main` | Created 2026-09-17. |
| Guest logins | A **guest** is an account like any other, with `guest: true` and an expiry: `login`, sessions and prefs need no special case, and **it may do everything an account may do**. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. | Built 2026-09-27. A read-only version was built first and **cut the same day**: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is *an account*: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to. |
| Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
@@ -37,7 +37,7 @@ stub tab or a roadmap item.
│ Cron │ │
│ … │ │
├──────────────┴───────────────────────────────────────────────┤
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev │
│ footer: v0.1.0 · up 12d · lab.bigham.dev │
└──────────────────────────────────────────────────────────────┘
```
@@ -116,7 +116,7 @@ hamlab/
- Mobile drawer, key bar, fit-on-rotate.
- `manifest.webmanifest`, icons (Ham logo on black, same recipe as HamTV), service
worker caching the app shell only (never `/api`).
- NPM proxy host for `hamlab.bigham.dev`; verify WS through NPM; force-SSL/HSTS/HTTP2.
- NPM proxy host for `lab.bigham.dev`; verify WS through NPM; force-SSL/HSTS/HTTP2.
- Stub groups for Services / Cron / ZFS / Dashboard.
**Done when:** from the phone, open the PWA, log in, open two terminals, run
@@ -417,39 +417,9 @@ something long in one, switch away and back, refresh — it's all still there.
snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it
had taken left untouched and fully readable here — which is what adopting its naming
bought. See `hamlet/docs/hosts.md`.
- **KVM tab** — **shipped 2026-09-24** (`internal/api/kvm.go`, `KVM.tsx`, `config.KVM`).
Paste a device's address; its own web page appears in the tab. **Nothing in it knows
anything about any particular KVM** — the first device here is a Sipeed NanoKVM on `grid`
at `10.0.0.217`, but none of that is in the code.
**Why it proxies rather than framing the device directly.** Ham Lab is served over TLS and
a KVM usually is not, so a bare `http://…` iframe is blocked as mixed content before it
loads. Serving the device from Ham Lab's own origin (`/kvm/<id>/…`) fixes that, puts it
behind **Ham Lab's** login rather than a second one, and makes the iframe same-origin.
**The general problem, and the answer.** A device UI asks for its assets by absolute path
— `/assets/app.js`, `/api/whatever` — which no path prefix can rewrite away, and rewriting
bodies per device would not be general. So the prefix is only the entry point: any request
Ham Lab does not serve itself is routed to whichever KVM the asking page belongs to, read
from the **Referer**, falling back to a **cookie** when there is none (a WebSocket upgrade).
Ham Lab's own routes always match first, so a device can never shadow the console, and the
fallback is gated on a live session. The device never receives Ham Lab's session cookie,
and its `X-Frame-Options`/CSP/HSTS are dropped on the way out since Ham Lab is the origin
now. Redirects to the device's root are rewritten back through the prefix.
**One catch found in testing:** Ham Lab's own `frame-ancestors 'none'` forbids framing even
from the same origin, so `/kvm/` paths send `'self'` instead. Without that the iframe was
refused before it loaded.
**The keyboard** needs less than it sounds: a page cannot forward keys into an iframe and
does not need to, since one with focus receives them directly. Fullscreen plus
`navigator.keyboard.lock()` is what takes Escape, Tab and Ctrl-W back from the browser;
that is Chromium-only, and the tab says which version you are getting rather than
pretending.
**One KVM per machine, and the tab follows the machine switcher** (corrected the same day:
the first cut was a hub-side list you pointed at machines, which had it backwards). Switch
to `grid` and you get grid's; adding a second for a machine replaces the first. The
addresses are still kept **on the hub** rather than on each machine, because a KVM earns
its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub
proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at
all. Not done: holding the device's own login on the hub, so its page does not ask for a
second password.
- **KVM tab** — shipped 2026-09-24, **removed 2026-11-09** at Mitchell's call: not needed —
the KVMs are reached over WireGuard (see `hamlet/docs/hosts.md`). It proxied a device's web
page through Ham Lab's origin; the code is in git history before that date.
- **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
@@ -1001,6 +971,11 @@ what it has moved.
## To do
- **Handle `fuse.mergerfs` in the Storage view** (2026-10-25). `grid` has a mergerfs pool at
`/srv/tank` (the NVMe landing area + `/tank`; `hamlet`'s `docs/tank-landing-plan.md`).
`fuse.mergerfs` isn't in `remoteFS` or `pseudoFS` (`internal/metrics/metrics.go`), so it shows
as a local filesystem whose size is `tank` and the NVMe added together. That counts both twice
in any storage total. Show it, labelled as a pool of its branches, and leave it out of totals.
- **Grow the Files tab** — asked for 2026-09-23 as two items (Files polish, and deprecating
copyparty), merged 2026-09-24. **copyparty was retired 2026-10-11** at Mitchell's call,
ahead of these gaps rather than after them: service, files and NPM host all gone, with a
+3 -5
View File
@@ -20,10 +20,10 @@ default or phones home to this lab · **C** cosmetic (placeholder text, comments
| 1.1 | **B** | `internal/selfupdate/selfupdate.go:36` `DefaultURL = "https://git.bigham.dev/api/v1/repos/mitchell/hamlab"` | Every install with no `update.url` checks **this** Gitea 90 s after boot and daily (`internal/api/update.go:1951, 2016`). A stranger's install phones home here by default. | Default should be **off** (or a build-time `-X` variable set at release). Installer asks or `install.sh` passes `HAMLAB_URL` through into `update.url`. |
| 1.2 | **B** | `install.sh:4, 27` `HAMLAB_URL` default; `README.md:23-25, 31` | The curl one-liner and the install source are this Gitea. | Default comes from the release that served the script (`@URL@` filled by `make release`), or is required. |
| 1.3 | **B** | `scripts/gitea-release.sh:14` `REPO` default | Publishing goes to `mitchell/hamlab` on this Gitea. | Require `HAMLAB_REPO`; no default. |
| 1.4 | **C** | `README.md:100` "On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager" | Lab deployment fact in the product README. | Move to `hamlet/docs/hosts.md`. |
| 1.4 | **C** | `README.md:100` "On `core` it is public at `https://lab.bigham.dev` behind Nginx Proxy Manager" | Lab deployment fact in the product README. | Move to `hamlet/docs/hosts.md`. |
| 1.5 | **C** | `examples/external-demo/index.html:8` `base: 'http://10.0.0.234:8100'` | Demo config names the hub's LAN IP. | `http://<hamlab-host>:8100`. |
| 1.6 | **C** | `SENSORS.md:149` Prometheus target `10.0.0.234:8100`; `:187` `&machine=grid`; `:196-208` Glance examples at `hamlab.bigham.dev`, `zfs.tank.used_pct`, `/api/nodes/grid/...`; `:218-220` `grid,wyse`, `{"core":…,"grid":…,"mac":…}` | Docs use real host names, pool and domain. | Generic placeholders (`<host>`, `<machine>`, `<pool>`). |
| 1.7 | **C** | `PLAN.md:3-4` "bare-metal Debian 13 host. First target: `core` (10.0.0.234)"; decisions table rows (Listen `10.0.0.234:8100`, Data `/home/core/apps/hamlab/`, Ingress NPM/`hamlab.bigham.dev`, Repo) | Product plan is written as this lab's plan. | Keep as history, or split lab facts out to `hamlet`. The repo is public. |
| 1.6 | **C** | `SENSORS.md:149` Prometheus target `10.0.0.234:8100`; `:187` `&machine=grid`; `:196-208` Glance examples at `lab.bigham.dev`, `zfs.tank.used_pct`, `/api/nodes/grid/...`; `:218-220` `grid,wyse`, `{"core":…,"grid":…,"mac":…}` | Docs use real host names, pool and domain. | Generic placeholders (`<host>`, `<machine>`, `<pool>`). |
| 1.7 | **C** | `PLAN.md:3-4` "bare-metal Debian 13 host. First target: `core` (10.0.0.234)"; decisions table rows (Listen `10.0.0.234:8100`, Data `/home/core/apps/hamlab/`, Ingress NPM/`lab.bigham.dev`, Repo) | Product plan is written as this lab's plan. | Keep as history, or split lab facts out to `hamlet`. The repo is public. |
| 1.9 | **C** | `BUGS.md` throughout | Bug bashes narrate `core`, `grid`, `wyse`, `hamtv`, `gate`, the Mac. | History; fine to leave, but it is public. |
| 1.10 | **C** | `cmd/hamlab/main.go:1` "Proxmox-style web console for a bare-metal Debian host"; `internal/hardware/hardware.go:5` "what Debian and its kin already have"; `internal/hardware/hardware.go:29` `// "GMKtec NucBox G3"`; `internal/metrics/metrics.go:1358` "did, on grid, 2026-10-02"; `internal/api/server.go:1382-1383` `["grid", …]`, `?machines=grid,wyse`; `internal/api/stacks.go:10` `/home/core/apps/dockhand/stacks/…`; `internal/api/dockersched.go:5` "unbound, nginx and pihole"; `internal/api/zfs.go:5` "grid already runs zfSnap"; `internal/storage/snapshots.go:6` "grid has run zfSnap v1.11.1"; `internal/api/guests.go:8` "Mitchell, 2026-09-27"; `internal/network/network.go:306` "the public address here is a DHCP lease"; `internal/metrics/gpu.go:17` "Asahi, today"; `internal/metrics/pciids.go:4-6` "the router's GPU"; `web/src/Top.tsx:522, 576`; `web/src/hashsel.ts:1-9`; `web/src/Embed.tsx:3` `&machine=grid` | Comments explain decisions by naming lab hosts. | Harmless, but rewrite the ones in package doc comments so a reader of the source doesn't need the lab to understand the code. |
@@ -32,7 +32,6 @@ default or phones home to this lab · **C** cosmetic (placeholder text, comments
| # | Sev | Where | Text | General form |
|---|---|---|---|---|
| 2.1 | **C** | `web/src/Machines.tsx:116-118` | `placeholder="grid"`, `"https://10.0.0.214:8100"`, `"core"` | `nas`, `https://192.0.2.10:8100`, `admin` or the hub's own user. |
| 2.2 | **C** | `web/src/KVM.tsx:166` | `placeholder="10.0.0.217"` (the NanoKVM on `grid`) | `192.0.2.20` or `kvm.local`. |
| 2.3 | **C** | `web/src/Settings.tsx:242` | `placeholder="cpu.*, temp.*, zfs.tank.*"` | `cpu.*, temp.*, fs.*`. |
| 2.4 | **C** | `web/src/Settings.tsx:225` | read-only token default name `'glance'` | Empty, or `dashboard`. |
| 2.5 | **C** | `web/src/Alerts.tsx:38, 40` | `"Tank nearly full"`, `"zfs.tank.used_pct"` | `"Disk nearly full"`, `"fs.root.used_pct"`. |
@@ -146,7 +145,6 @@ Listed so the sweep is known to be complete, not because they need work.
- `internal/nodes`, `internal/tlsauto`: hub ↔ node pairing, pinned certs, proxy, push.
- `internal/sensors`, alerts, widgets, embeds, Prometheus: machine-agnostic by design.
- `internal/files`, `internal/term`, `internal/tasks`, `internal/auth`, `internal/store`.
- `internal/api/kvm.go`: "nothing in it knows anything about any particular KVM" — verified.
- `internal/compose` (modulo 3.5).
- The React UI: machine-switcher model, hash routing, per-account prefs, themes.
+1 -1
View File
@@ -97,5 +97,5 @@ page flags machines running an older version and offers the same push per machin
Mixed architectures are fine: `make deploy` also cross-compiles every other architecture,
and the hub sends each machine the build for its own.
- On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager.
- On `core` it is public at `https://lab.bigham.dev` behind Nginx Proxy Manager.
- Installable as a PWA.
+5 -2
View File
@@ -169,6 +169,9 @@ container.<name>.down should be up and isn't: Docker c
restart it — *not* a container someone stopped
updates.pending|security|reboot_required every 5 min
zfs.<pool>.health|healthy|size|used|used_pct|frag every 1 min
zfs.write.dirty|limit|delayed the RAM write buffer (all pools) and writes the throttle is slowing
zfs.<pool>.write_bps|logical_bps bytes/s synced to disk, and written by programs
zfs.<pool>.landing.pending_bytes|pending_files|used_pct|in_bps|drain_bps|mover|queue a landing tier, where one fronts the pool
smart.<dev>.passed|warnings|power_on|wear every 10 min, never waking a sleeping disk
```
@@ -200,13 +203,13 @@ facts, never live. An `iframe` of the embed is live.
```yaml
- type: iframe
title: core CPU
source: https://hamlab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
source: https://lab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
height: 130
- type: custom-api
title: grid
cache: 1m
url: https://hamlab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
url: https://lab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
headers: { Authorization: Bearer <read-only token> }
template: |
<div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>
+80
View File
@@ -63,6 +63,7 @@ func main() {
passwd [user] set a user's password (makes the account if it is new; reads stdin when piped)
users list the accounts
zfs-snapshot <id> run one ZFS snapshot schedule (what its cron line calls)
landing-verify <new|all|id> verify a landing tier against its mover's ledger (a schedule's id is what its cron line calls)
docker-update <id> run one image-update schedule (what its cron line calls)
user-rm <user> remove an account and sign it out everywhere
check verify the config and state load, then exit
@@ -150,6 +151,10 @@ func main() {
if err := zfsSnapshot(cfg, flag.Arg(1)); err != nil {
log.Fatal(err)
}
case "landing-verify":
if err := landingVerify(cfg, flag.Arg(1)); err != nil {
log.Fatal(err)
}
case "users":
migrateUsers(cfg)
if len(cfg.Users) == 0 {
@@ -254,6 +259,81 @@ func dockerUpdateJob(cfg *config.Config, id string) error {
//
// It deliberately does not write state.json. The daemon owns that file and would clobber
// anything written from out here; the outcome goes beside it instead, in zfs-runs/.
// landingVerify runs one verify pass over every landing tier on this machine: by hand with a
// scope ("new" or "all"), or as a schedule by its id, which is what the cron line passes.
func landingVerify(cfg *config.Config, arg string) error {
scope, id := arg, "manual"
if arg != "new" && arg != "all" {
if strings.TrimSpace(arg) == "" {
return errors.New("usage: hamlab landing-verify <new|all|schedule-id>")
}
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
if err != nil {
return err
}
var raw json.RawMessage
st.View(func(s *store.State) { raw = s.ZFS })
var z struct {
Verify []storage.VerifyPolicy `json:"verify"`
}
if len(raw) > 0 {
if err := json.Unmarshal(raw, &z); err != nil {
return fmt.Errorf("reading the schedules: %w", err)
}
}
found := false
for _, p := range z.Verify {
if p.ID == arg {
scope, id, found = p.Scope, p.ID, true
}
}
if !found {
return fmt.Errorf("no verify schedule %q — it may have been removed; this cron line is stale", arg)
}
}
dir := cfg.DataDir()
unlock, err := storage.LockVerify(dir)
if err != nil {
return err
}
defer unlock()
defer storage.ClearVerifyProgress(dir)
targets := storage.LandingTargets()
if len(targets) == 0 {
r := storage.VerifyResult{ID: id, Scope: scope, Started: time.Now(), Finished: time.Now(), Issues: []storage.VerifyIssue{},
Error: "no landing mover is reporting (nothing in " + storage.MoverStatusDir + "), so there is nothing to verify"}
_ = storage.RecordVerify(dir, storage.LandingTarget{}, r)
return errors.New(r.Error)
}
var firstErr error
for _, t := range targets {
r := storage.Verify(t, id, scope, storage.VerifyCursor(dir, t), func(p storage.VerifyProgress) { storage.WriteVerifyProgress(dir, p) })
if err := storage.RecordVerify(dir, t, r); err != nil {
fmt.Fprintln(os.Stderr, "recording the pass:", err)
}
fmt.Printf("%s -> %s (%s): %d checked, %d match, %d mismatch, %d gone, %d unreadable, %d changed since, %d waiting, %d unhashed, %s read in %s\n",
t.Src, t.Dst, scope, r.Checked, r.Matched, r.Mismatched, r.Gone, r.Unreadable, r.Changed, r.Waiting, r.Unhashed,
humanBytes(r.BytesRead), r.Finished.Sub(r.Started).Round(time.Second))
if r.Error != "" && firstErr == nil {
firstErr = errors.New(r.Error)
}
}
return firstErr
}
func humanBytes(n int64) string {
const u = 1024
if n < u {
return fmt.Sprintf("%d B", n)
}
div, exp := int64(u), 0
for v := n / u; v >= u; v /= u {
div *= u
exp++
}
return fmt.Sprintf("%.1f %ciB", float64(n)/float64(div), "KMGTPE"[exp])
}
func zfsSnapshot(cfg *config.Config, id string) error {
if strings.TrimSpace(id) == "" {
return errors.New("usage: hamlab zfs-snapshot <schedule-id>")
-420
View File
@@ -1,420 +0,0 @@
package api
// IP-KVMs: a device's own web page, shown inside Ham Lab.
//
// The whole difficulty is that Ham Lab is served over TLS and a KVM usually is not, so a
// browser refuses to put the device's page in an iframe — mixed content, blocked before
// anything loads. The fix is to serve the device from Ham Lab's own origin, which also
// means it inherits Ham Lab's login instead of needing a second one, and makes the iframe
// same-origin.
//
// Proxying an arbitrary web app under a path prefix is the hard part, and nothing here
// knows anything about any particular device. A device UI asks for its assets by absolute
// path — "/assets/app.js", "/api/whatever" — which a prefix cannot rewrite away. So the
// prefix is only the entry point: any request Ham Lab does not serve itself is routed to
// whichever KVM the asking page belongs to, worked out from the Referer, and from a cookie
// when there is no Referer (a WebSocket upgrade, say). Ham Lab's own routes always win, so
// a device can never shadow the console.
import (
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"io"
"log"
"net/http"
"net/http/httputil"
"net/url"
"strings"
"sync"
"time"
"hamlab/internal/config"
"hamlab/internal/store"
)
const kvmPrefix = "/kvm/"
const kvmCookie = "hamlab_kvm"
type kvmProxies struct {
mu sync.Mutex
m map[string]*httputil.ReverseProxy
}
func (s *Server) kvm(id string) (config.KVM, bool) {
var found config.KVM
ok := false
s.cfg.View(func(c *config.Config) {
for _, k := range c.KVMs {
if k.ID == id {
found, ok = k, true
}
}
})
return found, ok
}
// kvmProxy is the reverse proxy for one device, made once and kept.
func (s *Server) kvmProxy(k config.KVM) (*httputil.ReverseProxy, error) {
s.kvms.mu.Lock()
defer s.kvms.mu.Unlock()
if s.kvms.m == nil {
s.kvms.m = map[string]*httputil.ReverseProxy{}
}
if p, ok := s.kvms.m[k.ID]; ok {
return p, nil
}
target, err := url.Parse(k.URL)
if err != nil || target.Host == "" {
return nil, errors.New("this KVM's address is not a URL")
}
prefix := kvmPrefix + k.ID
p := &httputil.ReverseProxy{
Director: func(r *http.Request) {
r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
// Entry through the prefix keeps the path after it; anything else arrived by its
// own absolute path and is passed through untouched.
if rest, ok := strings.CutPrefix(r.URL.Path, prefix); ok {
if rest == "" {
rest = "/"
}
r.URL.Path = rest
}
r.URL.RawPath = ""
r.Host = target.Host
// Whether the console itself is on TLS: the response rewriter needs to know, and by
// then only the outbound request is in hand.
if secure(r) {
r.Header.Set(kvmSecureHeader, "1")
} else {
r.Header.Del(kvmSecureHeader)
}
// The device gets its own host and none of Ham Lab's session.
r.Header.Del("Origin")
stripCookie(r, cookieName, kvmCookie)
// What is left is the device's own login. A browser that has it teaches the hub; one
// that has not is lent what the hub remembers, so the device sees a signed-in client.
if own := r.Cookies(); len(own) > 0 {
s.rememberKVMCookies(k.ID, own)
} else {
for name, val := range s.kvmCookies(k.ID) {
r.AddCookie(&http.Cookie{Name: name, Value: val})
}
}
},
FlushInterval: -1, // a video stream must not be buffered
ModifyResponse: func(resp *http.Response) error {
// The device's own framing and transport policy would stop it being embedded, or
// send a browser off to plain HTTP. Ham Lab is the origin now, and says what it
// serves; its own headers are added on the way out.
for _, h := range []string{"X-Frame-Options", "Content-Security-Policy", "Content-Security-Policy-Report-Only", "Strict-Transport-Security", "Referrer-Policy"} {
resp.Header.Del(h)
}
// A cookie the device sets by header is kept for Ham Lab's origin — so its Domain
// (the device's) goes, Secure follows the console rather than the device, and the
// hub remembers it too.
if set := resp.Cookies(); len(set) > 0 {
https := resp.Request != nil && resp.Request.Header.Get(kvmSecureHeader) == "1"
resp.Header.Del("Set-Cookie")
for _, c := range set {
c.Domain = ""
if c.Path == "" {
c.Path = "/"
}
c.Secure = c.Secure && https
if v := c.String(); v != "" {
resp.Header.Add("Set-Cookie", v)
}
}
s.rememberKVMCookies(k.ID, set)
}
// A redirect to the device's own root has to come back through the prefix.
if loc := resp.Header.Get("Location"); loc != "" {
if u, err := url.Parse(loc); err == nil && (u.Host == "" || u.Host == target.Host) && strings.HasPrefix(u.Path, "/") && !strings.HasPrefix(u.Path, prefix) {
u.Scheme, u.Host = "", ""
u.Path = prefix + u.Path
resp.Header.Set("Location", u.String())
}
}
return nil
},
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadGateway)
_ = json.NewEncoder(w).Encode(map[string]string{"error": k.Name + " unreachable: " + err.Error()})
},
}
s.kvms.m[k.ID] = p
return p, nil
}
// kvmSecureHeader rides on the outbound request to say whether the console was reached
// over TLS; the device ignores it.
const kvmSecureHeader = "X-Hamlab-Secure"
// kvmCookies is what the hub remembers of a device's own cookies — its login, in practice.
func (s *Server) kvmCookies(id string) map[string]string {
var out map[string]string
s.store.View(func(st *store.State) {
if m := st.KVMCookies[id]; len(m) > 0 {
out = make(map[string]string, len(m))
for k, v := range m {
out[k] = v
}
}
})
return out
}
// rememberKVMCookies keeps the cookies a browser (or the device's own Set-Cookie) carries
// for a KVM, so that a browser without them can be lent them. A cookie emptied or expired
// by the device is forgotten. Written only when something changed: this runs on every
// proxied request.
func (s *Server) rememberKVMCookies(id string, cs []*http.Cookie) {
have := s.kvmCookies(id)
changed := false
for _, c := range cs {
if c.Name == "" || c.Name == cookieName || c.Name == kvmCookie {
continue
}
gone := c.Value == "" || c.MaxAge < 0 || (!c.Expires.IsZero() && c.Expires.Before(time.Now()))
if gone {
if _, ok := have[c.Name]; ok {
changed = true
}
continue
}
if have[c.Name] != c.Value {
changed = true
}
}
if !changed {
return
}
_ = s.store.Update(func(st *store.State) {
if st.KVMCookies == nil {
st.KVMCookies = map[string]map[string]string{}
}
m := st.KVMCookies[id]
if m == nil {
m = map[string]string{}
}
for _, c := range cs {
if c.Name == "" || c.Name == cookieName || c.Name == kvmCookie {
continue
}
if c.Value == "" || c.MaxAge < 0 || (!c.Expires.IsZero() && c.Expires.Before(time.Now())) {
delete(m, c.Name)
} else {
m[c.Name] = c.Value
}
}
if len(m) == 0 {
delete(st.KVMCookies, id)
} else {
st.KVMCookies[id] = m
}
})
}
// forgetKVMCookies drops a device's remembered login — when the device is removed.
func (s *Server) forgetKVMCookies(id string) {
_ = s.store.Update(func(st *store.State) { delete(st.KVMCookies, id) })
}
// lendKVMCookies hands the browser the device cookies the hub remembers and it lacks, as
// the device's own page would have set them: readable by its scripts, for Ham Lab's whole
// origin, kept for half a year. The device's SPA checks for its token client-side before it
// ever asks the server, so lending it only on the wire would still land on the login form.
func (s *Server) lendKVMCookies(w http.ResponseWriter, r *http.Request, id string) {
for name, val := range s.kvmCookies(id) {
if _, err := r.Cookie(name); err == nil {
continue
}
http.SetCookie(w, &http.Cookie{Name: name, Value: val, Path: "/", MaxAge: 180 * 24 * 3600, Secure: secure(r), SameSite: http.SameSiteLaxMode})
}
}
// stripCookie removes named cookies from a request, so Ham Lab's session is never handed to
// a device and the device never sees which KVM the browser is looking at.
func stripCookie(r *http.Request, names ...string) {
cs := r.Cookies()
kept := cs[:0]
for _, c := range cs {
drop := false
for _, n := range names {
if c.Name == n {
drop = true
}
}
if !drop {
kept = append(kept, c)
}
}
r.Header.Del("Cookie")
for _, c := range kept {
r.AddCookie(c)
}
}
func (s *Server) forgetKVMProxy(id string) {
s.kvms.mu.Lock()
delete(s.kvms.m, id)
s.kvms.mu.Unlock()
}
// serveKVM is the entry point: /kvm/<id>/… . It also drops a cookie naming the device, so
// that requests the page makes later by absolute path can be traced back to it.
func (s *Server) serveKVM(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
k, ok := s.kvm(id)
if !ok {
jsonErr(w, 404, "no such KVM")
return
}
p, err := s.kvmProxy(k)
if err != nil {
jsonErr(w, 400, err.Error())
return
}
http.SetCookie(w, &http.Cookie{Name: kvmCookie, Value: id, Path: "/", HttpOnly: true, Secure: secure(r), SameSite: http.SameSiteLaxMode})
s.lendKVMCookies(w, r, id)
p.ServeHTTP(w, r)
}
// kvmFor works out which device a request belongs to when it arrived by absolute path
// rather than through the prefix: the Referer names the page that asked, and a cookie
// covers what has no Referer at all. Nothing device-specific: it never has to know which
// paths a given KVM uses.
func (s *Server) kvmFor(r *http.Request) (config.KVM, bool) {
if ref := r.Header.Get("Referer"); ref != "" {
if u, err := url.Parse(ref); err == nil {
if rest, ok := strings.CutPrefix(u.Path, kvmPrefix); ok {
if id, _, _ := strings.Cut(rest, "/"); id != "" {
if k, ok := s.kvm(id); ok {
return k, true
}
}
}
}
// asked for by a page that is not a KVM: not ours to answer
return config.KVM{}, false
}
if c, err := r.Cookie(kvmCookie); err == nil && c.Value != "" {
return s.kvm(c.Value)
}
return config.KVM{}, false
}
// kvmFallback answers a request Ham Lab does not serve itself, when it came from inside a
// KVM's page. It returns false when the request has nothing to do with a KVM, and the
// caller carries on with its own 404.
func (s *Server) kvmFallback(w http.ResponseWriter, r *http.Request) bool {
k, ok := s.kvmFor(r)
if !ok {
return false
}
p, err := s.kvmProxy(k)
if err != nil {
return false
}
p.ServeHTTP(w, r)
return true
}
func machineLabel(m string) string {
if m == "" {
return "this host"
}
return m
}
// ---- the list ----
func (s *Server) listKVMs(w http.ResponseWriter, _ *http.Request) {
out := []config.KVM{}
s.cfg.View(func(c *config.Config) { out = append(out, c.KVMs...) })
jsonOK(w, 200, out)
}
func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
var body struct{ Name, URL, Machine string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
raw := strings.TrimSpace(body.URL)
if raw != "" && !strings.Contains(raw, "://") {
raw = "http://" + raw // what you would type: an address, not a scheme
}
u, err := url.Parse(raw)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
jsonErr(w, 400, "address: http(s)://host, or just the host")
return
}
name := strings.TrimSpace(body.Name)
if name == "" {
name = u.Host
}
if len(name) > maxNameLen {
jsonErr(w, 400, "that name is too long")
return
}
b := make([]byte, 6)
if _, err := rand.Read(b); err != nil {
jsonErr(w, 500, err.Error())
return
}
machine := strings.TrimSpace(body.Machine)
k := config.KVM{ID: hex.EncodeToString(b), Name: name, URL: u.Scheme + "://" + u.Host + strings.TrimSuffix(u.Path, "/"), Machine: machine, Added: time.Now()}
// One per machine: a second one for the same host replaces the first rather than sitting
// beside it, since the tab shows the machine you are looking at.
var replaced string
if err := s.cfg.Update(func(c *config.Config) {
kept := c.KVMs[:0]
for _, old := range c.KVMs {
if old.Machine == machine {
replaced = old.ID
continue
}
kept = append(kept, old)
}
c.KVMs = append(kept, k)
}); err != nil {
jsonErr(w, 500, err.Error())
return
}
if replaced != "" {
s.forgetKVMProxy(replaced)
}
log.Printf("kvm %q added for %s (%s) by %s", k.Name, machineLabel(machine), k.URL, clientIP(r))
jsonOK(w, 201, k)
}
func (s *Server) removeKVM(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
found := false
if err := s.cfg.Update(func(c *config.Config) {
kept := c.KVMs[:0]
for _, k := range c.KVMs {
if k.ID == id {
found = true
continue
}
kept = append(kept, k)
}
c.KVMs = kept
}); err != nil {
jsonErr(w, 500, err.Error())
return
}
if !found {
jsonErr(w, 404, "no such KVM")
return
}
s.forgetKVMProxy(id)
s.forgetKVMCookies(id)
log.Printf("kvm %s removed by %s", id, clientIP(r))
w.WriteHeader(204)
}
-123
View File
@@ -1,123 +0,0 @@
package api
import (
"net/http"
"net/http/httptest"
"net/url"
"path/filepath"
"strings"
"testing"
"hamlab/internal/auth"
"hamlab/internal/config"
"hamlab/internal/store"
)
// A KVM's own login is a cookie its page sets in one browser. The hub remembers it from the
// requests that carry it, lends it to a browser that lacks it — both on the wire to the
// device and back to the browser, since the device's page checks for it client-side — and
// forgets it when the device says it is gone or the KVM is removed.
func TestKVMCookiesAreRememberedAndLent(t *testing.T) {
var seen []string // the Cookie header the fake device received, per request
device := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
seen = append(seen, r.Header.Get("Cookie"))
if r.URL.Path == "/api/auth/login" {
// a device that sets its token by header, with its own domain and Secure
http.SetCookie(w, &http.Cookie{Name: "nano-kvm-token", Value: "tok-1", Domain: "10.0.0.217", Secure: true})
}
if r.URL.Path == "/api/auth/logout" {
http.SetCookie(w, &http.Cookie{Name: "nano-kvm-token", Value: "", MaxAge: -1})
}
w.WriteHeader(200)
}))
defer device.Close()
dir := t.TempDir()
cfg := config.Default(filepath.Join(dir, "config.json"))
if err := cfg.Update(func(c *config.Config) {
c.KVMs = []config.KVM{{ID: "abc123", Name: "test", URL: device.URL, Machine: ""}}
}); err != nil {
t.Fatal(err)
}
st, err := store.Open(filepath.Join(dir, "state.json"))
if err != nil {
t.Fatal(err)
}
s := &Server{cfg: cfg, store: st, sessions: auth.NewSessions(st)}
k, _ := s.kvm("abc123")
p, err := s.kvmProxy(k)
if err != nil {
t.Fatal(err)
}
ask := func(path string, cookies ...*http.Cookie) *httptest.ResponseRecorder {
r := httptest.NewRequest("GET", path, nil)
r.SetPathValue("id", "abc123")
for _, c := range cookies {
r.AddCookie(c)
}
w := httptest.NewRecorder()
if strings.HasPrefix(path, kvmPrefix) {
s.serveKVM(w, r)
} else {
p.ServeHTTP(w, r)
}
return w
}
// 1. the device sets its token by header: it reaches the browser without the device's
// domain or Secure (plain-http console), and the hub remembers it
w := ask("/api/auth/login", &http.Cookie{Name: cookieName, Value: "hamlab-session"})
var set *http.Cookie
for _, c := range w.Result().Cookies() {
if c.Name == "nano-kvm-token" {
set = c
}
}
if set == nil || set.Domain != "" || set.Secure || set.Value != "tok-1" || set.Path != "/" {
t.Fatalf("rewritten Set-Cookie = %+v, want nano-kvm-token=tok-1 for / without Domain or Secure", set)
}
if got := s.kvmCookies("abc123"); got["nano-kvm-token"] != "tok-1" {
t.Fatalf("remembered = %v, want the token", got)
}
if strings.Contains(seen[0], "hamlab_session") {
t.Fatalf("the device was handed Ham Lab's session: %q", seen[0])
}
// 2. a browser that has the cookie teaches a newer value
ask("/api/vm/info", &http.Cookie{Name: "nano-kvm-token", Value: "tok-2"}, &http.Cookie{Name: cookieName, Value: "x"})
if got := s.kvmCookies("abc123"); got["nano-kvm-token"] != "tok-2" {
t.Fatalf("remembered = %v, want tok-2", got)
}
if seen[1] != "nano-kvm-token=tok-2" {
t.Fatalf("device saw %q, want the browser's own cookie only", seen[1])
}
// 3. a browser without it is lent it: on the wire, and back as a Set-Cookie it can read
w = ask(kvmPrefix+"abc123/", &http.Cookie{Name: cookieName, Value: "x"})
if seen[2] != "nano-kvm-token=tok-2" {
t.Fatalf("device saw %q from a browser with no cookie, want the remembered token", seen[2])
}
lent := false
for _, c := range w.Result().Cookies() {
if c.Name == "nano-kvm-token" && c.Value == "tok-2" && !c.HttpOnly && c.Path == "/" && c.MaxAge > 0 {
lent = true
}
}
if !lent {
t.Fatalf("browser was not lent the token: %v", w.Result().Cookies())
}
// 4. the device logging out forgets it
ask("/api/auth/logout", &http.Cookie{Name: "nano-kvm-token", Value: "tok-2"})
if got := s.kvmCookies("abc123"); got != nil {
t.Fatalf("after logout remembered = %v, want nothing", got)
}
// 5. removing the KVM clears whatever is left
s.rememberKVMCookies("abc123", []*http.Cookie{{Name: "nano-kvm-token", Value: "tok-3"}})
s.forgetKVMCookies("abc123")
if got := s.kvmCookies("abc123"); got != nil {
t.Fatalf("after remove remembered = %v, want nothing", got)
}
_ = url.Parse
}
+216
View File
@@ -0,0 +1,216 @@
package api
// The ZFS tab's landing-tier checks: what has waited too long on a landing branch, and verify
// passes that read the pool's copy of every moved file back against the mover's ledger —
// run by hand or on a schedule written to /etc/cron.d/hamlab-landing, just as snapshot
// schedules are. A pass runs as its own process (`hamlab landing-verify`), the same one cron
// starts, so it survives the console being restarted or upgraded mid-pass.
import (
"context"
"encoding/json"
"io"
"log"
"net/http"
"os"
"os/exec"
"path/filepath"
"syscall"
"time"
"hamlab/internal/storage"
)
func (s *Server) syncVerifyCron(ctx context.Context, z zfsState) error {
exe, err := os.Executable()
if err != nil {
return err
}
exe, _ = filepath.EvalSymlinks(exe)
return s.storage.WriteVerifyCron(ctx, z.Verify, s.user, exe, s.cfg.Path())
}
// startVerify launches a pass in its own session: arg is "new", "all" or a schedule id.
func (s *Server) startVerify(arg string) error {
exe, err := os.Executable()
if err != nil {
return err
}
exe, _ = filepath.EvalSymlinks(exe)
dir := filepath.Join(s.cfg.DataDir(), "landing-verify")
if err := os.MkdirAll(dir, 0o700); err != nil {
return err
}
out, err := os.OpenFile(filepath.Join(dir, "last.log"), os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
if err != nil {
return err
}
cmd := exec.Command(exe, "-config", s.cfg.Path(), "landing-verify", arg)
cmd.Stdout, cmd.Stderr = out, out
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
if err := cmd.Start(); err != nil {
out.Close()
return err
}
go func() { _ = cmd.Wait(); out.Close() }()
return nil
}
func (s *Server) landingOverview(w http.ResponseWriter, r *http.Request) {
dir := s.cfg.DataDir()
now := time.Now()
type tier struct {
storage.LandingTarget
Cursor int64 `json:"cursor"`
StaleCount int `json:"stale_count"`
StaleBytes int64 `json:"stale_bytes"`
Stale []storage.StaleFile `json:"stale"`
}
tiers := []tier{}
for _, t := range storage.LandingTargets() {
files, n, b := storage.StaleFiles(t.Src, 100, now)
if files == nil {
files = []storage.StaleFile{}
}
tiers = append(tiers, tier{LandingTarget: t, Cursor: storage.VerifyCursor(dir, t), StaleCount: n, StaleBytes: b, Stale: files})
}
z := s.zfsLoad()
pols := make([]map[string]any, 0, len(z.Verify))
for i := range z.Verify {
p := z.Verify[i]
expr, _ := p.CronExpr()
m := map[string]any{"policy": p, "cron": expr, "last": storage.LastVerify(dir, p.ID)}
if p.Enabled {
if n := p.NextRun(now); !n.IsZero() {
m["next"] = n
}
}
pols = append(pols, m)
}
hist := storage.VerifyHistory(dir)
if len(hist) > 10 {
hist = hist[:10]
}
jsonOK(w, 200, map[string]any{
"tiers": tiers, "stale_after_sec": int(storage.StaleAfter.Seconds()),
"running": storage.RunningVerify(dir), "history": hist, "policies": pols,
"cron_file": storage.VerifyCronFile,
})
}
func (s *Server) landingVerifyNow(w http.ResponseWriter, r *http.Request) {
var body struct {
Scope string `json:"scope"`
}
if err := json.NewDecoder(io.LimitReader(r.Body, 1024)).Decode(&body); err != nil || (body.Scope != "new" && body.Scope != "all") {
jsonErr(w, 400, "scope: new or all")
return
}
if storage.RunningVerify(s.cfg.DataDir()) != nil {
jsonErr(w, 409, "a verify pass is already running")
return
}
if len(storage.LandingTargets()) == 0 {
jsonErr(w, 409, "no landing mover is reporting on this machine, so there is nothing to verify")
return
}
if err := s.startVerify(body.Scope); err != nil {
jsonErr(w, 500, err.Error())
return
}
log.Printf("landing verify (%s) started by %s", body.Scope, clientIP(r))
jsonOK(w, 202, map[string]string{"started": body.Scope})
}
func (s *Server) landingPolicyPut(w http.ResponseWriter, r *http.Request) {
var p storage.VerifyPolicy
if err := json.NewDecoder(io.LimitReader(r.Body, 16<<10)).Decode(&p); err != nil {
jsonErr(w, 400, "bad request")
return
}
if err := p.Validate(); err != nil {
jsonErr(w, 400, err.Error())
return
}
z := s.zfsLoad()
found := false
for i := range z.Verify {
if p.ID != "" && z.Verify[i].ID == p.ID {
z.Verify[i], found = p, true
}
}
if !found {
if p.ID == "" {
id, err := randomID()
if err != nil {
jsonErr(w, 500, err.Error())
return
}
p.ID = id
}
z.Verify = append(z.Verify, p)
}
if err := s.zfsSave(z); err != nil {
jsonErr(w, 500, err.Error())
return
}
if err := s.syncVerifyCron(r.Context(), z); err != nil {
jsonErr(w, 500, "saved, but writing "+storage.VerifyCronFile+" failed: "+err.Error())
return
}
expr, _ := p.CronExpr()
log.Printf("landing verify schedule %q saved by %s (%q, %s)", p.Name, clientIP(r), expr, p.Scope)
jsonOK(w, 200, p)
}
func (s *Server) landingPolicyDelete(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
z := s.zfsLoad()
kept := z.Verify[:0]
found := false
for _, p := range z.Verify {
if p.ID == id {
found = true
continue
}
kept = append(kept, p)
}
if !found {
jsonErr(w, 404, "no such schedule")
return
}
z.Verify = kept
if err := s.zfsSave(z); err != nil {
jsonErr(w, 500, err.Error())
return
}
if err := s.syncVerifyCron(r.Context(), z); err != nil {
jsonErr(w, 500, "removed, but writing "+storage.VerifyCronFile+" failed: "+err.Error())
return
}
storage.ForgetVerify(s.cfg.DataDir(), id)
log.Printf("landing verify schedule %s removed by %s", id, clientIP(r))
w.WriteHeader(204)
}
func (s *Server) landingPolicyRun(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
found := false
for _, p := range s.zfsLoad().Verify {
found = found || p.ID == id
}
if !found {
jsonErr(w, 404, "no such schedule")
return
}
if storage.RunningVerify(s.cfg.DataDir()) != nil {
jsonErr(w, 409, "a verify pass is already running")
return
}
if err := s.startVerify(id); err != nil {
jsonErr(w, 500, err.Error())
return
}
log.Printf("landing verify schedule %s run by %s", id, clientIP(r))
jsonOK(w, 202, map[string]string{"started": id})
}
+9 -28
View File
@@ -75,7 +75,6 @@ type Server struct {
net *network.Watcher
update updateState
events eventLog
kvms kvmProxies
fsLocal files.FS
fsSudo files.FS // nil without passwordless sudo
thumbs *thumbSvc
@@ -314,6 +313,11 @@ func (s *Server) routes() *http.ServeMux {
m.HandleFunc("DELETE /api/zfs/schedules/{id}", s.auth(s.zfsPolicyDelete))
m.HandleFunc("POST /api/zfs/schedules/{id}/run", s.auth(s.zfsRunNow))
m.HandleFunc("POST /api/zfs/snapshots/delete", s.auth(s.zfsSnapshotDelete))
m.HandleFunc("GET /api/zfs/landing", s.auth(s.landingOverview))
m.HandleFunc("POST /api/zfs/landing/verify", s.auth(s.landingVerifyNow))
m.HandleFunc("PUT /api/zfs/landing/schedules", s.auth(s.landingPolicyPut))
m.HandleFunc("DELETE /api/zfs/landing/schedules/{id}", s.auth(s.landingPolicyDelete))
m.HandleFunc("POST /api/zfs/landing/schedules/{id}/run", s.auth(s.landingPolicyRun))
m.HandleFunc("GET /api/space", s.auth(s.spaceOverview))
m.HandleFunc("GET /api/space/{disk}", s.auth(s.spaceStatus))
m.HandleFunc("POST /api/space/{disk}/scan", s.auth(s.spaceScan))
@@ -346,18 +350,7 @@ func (s *Server) routes() *http.ServeMux {
m.HandleFunc("PUT /api/prefs/sampler", s.auth(s.putSampler))
m.HandleFunc("GET /api/prefs/history", s.auth(s.getHistoryWindow))
m.HandleFunc("PUT /api/prefs/history", s.auth(s.putHistoryWindow))
m.HandleFunc("GET /api/kvms", s.auth(s.listKVMs))
m.HandleFunc("POST /api/kvms", s.auth(s.addKVM))
m.HandleFunc("DELETE /api/kvms/{id}", s.auth(s.removeKVM))
m.HandleFunc("/kvm/{id}/{rest...}", s.auth(s.serveKVM))
m.HandleFunc("/kvm/{id}", s.auth(s.serveKVM))
// A path Ham Lab does not serve may still belong to a KVM's page, which asks for its
// own assets by absolute path. Ham Lab's own routes matched first, so this can only
// catch what would otherwise have been a 404.
m.HandleFunc("/api/", s.auth(func(w http.ResponseWriter, r *http.Request) {
if s.kvmFallback(w, r) {
return
}
jsonErr(w, 404, "not found")
}))
m.HandleFunc("/", s.static)
@@ -381,16 +374,10 @@ func sensorPath(p string) bool {
func (s *Server) Handler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h := w.Header()
// The console must never be framed. Two exceptions: an embedded widget exists to be
// (a page that can only read sensors, and only with a read-only token in its URL),
// and a proxied KVM is framed by the console itself — 'none' forbids that even from
// the same origin, so those paths say 'self' instead of opening up.
switch {
case r.URL.Path == "/embed/widget":
case strings.HasPrefix(r.URL.Path, kvmPrefix):
h.Set("X-Frame-Options", "SAMEORIGIN")
h.Set("Content-Security-Policy", "frame-ancestors 'self'")
default:
// The console must never be framed. The one exception is an embedded widget, which
// exists to be (a page that can only read sensors, and only with a read-only token in
// its URL).
if r.URL.Path != "/embed/widget" {
h.Set("X-Frame-Options", "DENY")
h.Set("Content-Security-Policy", "frame-ancestors 'none'")
}
@@ -2649,12 +2636,6 @@ func (s *Server) static(w http.ResponseWriter, r *http.Request) {
return
}
}
// A path with no file of ours behind it may belong to a KVM's page, which asks for its
// own assets by absolute path. Checked here rather than in the router because this
// handler is the catch-all — and gated on a session, since this handler is public.
if s.signedIn(r) && s.kvmFallback(w, r) {
return
}
// SPA fallback
w.Header().Set("Cache-Control", "no-cache")
http.ServeFileFS(w, r, s.ui, "index.html")
+9 -3
View File
@@ -26,7 +26,8 @@ import (
)
type zfsState struct {
Policies []storage.Policy `json:"policies"`
Policies []storage.Policy `json:"policies"`
Verify []storage.VerifyPolicy `json:"verify"` // landing-tier verify schedules (landing.go)
}
func (s *Server) zfsLoad() zfsState {
@@ -39,6 +40,9 @@ func (s *Server) zfsLoad() zfsState {
if z.Policies == nil {
z.Policies = []storage.Policy{}
}
if z.Verify == nil {
z.Verify = []storage.VerifyPolicy{}
}
return z
}
@@ -138,11 +142,13 @@ func (s *Server) zfsOverview(w http.ResponseWriter, r *http.Request) {
// zfsARC is the cache: the sampler's live ARC (size, the hit rate over the last interval)
// beside the storage package's fuller picture (since-boot counters, tunables, and each
// pool's cache/log/special devices or the lack of them).
// pool's cache/log/special devices or the lack of them), and the write path — the dirty
// buffer in RAM, each pool's write rate, and any landing tier in front of a pool.
func (s *Server) zfsARC(w http.ResponseWriter, r *http.Request) {
out := map[string]any{"detail": s.storage.ARC(r.Context()), "live": nil}
out := map[string]any{"detail": s.storage.ARC(r.Context()), "live": nil, "writes": nil}
if smp, ok := s.metrics.Latest(); ok && smp.ARC != nil {
out["live"] = smp.ARC
out["writes"] = smp.ZFSWrite
}
jsonOK(w, 200, out)
}
-18
View File
@@ -77,23 +77,6 @@ type UpdateSource struct {
Check string `json:"check,omitempty"`
}
// KVM is an IP-KVM: a device with a web page that gives screen, keyboard and mouse access
// to a machine, including when that machine has no OS running. Ham Lab holds the list on
// the hub and proxies each one, because a KVM matters exactly when the machine it is
// attached to cannot answer for itself.
type KVM struct {
ID string `json:"id"`
Name string `json:"name"`
// URL is the device's own address, as you would type it into a browser on the LAN.
URL string `json:"url"`
// Machine is the host this KVM is attached to, and is the key: one KVM per machine, shown
// on that machine's KVM tab. Empty means the hub itself. The list lives on the hub even
// for a node's KVM, because a KVM earns its keep when the machine is down and cannot
// answer for itself.
Machine string `json:"machine"`
Added time.Time `json:"added"`
}
type Config struct {
Listen string `json:"listen"`
// PasswordHash is the single login of installs before users existed. On load it becomes
@@ -103,7 +86,6 @@ type Config struct {
TLS string `json:"tls,omitempty"` // "auto" → self-signed cert in the data dir; "" → plain HTTP
Tokens []Token `json:"tokens,omitempty"`
Nodes []Node `json:"nodes,omitempty"`
KVMs []KVM `json:"kvms,omitempty"`
// Updates: where a newer build comes from. See UpdateSource.
Updates UpdateSource `json:"update,omitempty"`
+52 -15
View File
@@ -95,30 +95,41 @@ func readRAPL(root string, dirs []string) ([]raplCount, bool) {
// is carried across its range. One that went backwards with no known range, or a set of
// counters that changed between readings, gives no answer rather than a wrong one.
func raplWatts(prev, cur []raplCount, max map[string]uint64, dt float64) (float64, bool) {
if dt <= 0 || len(prev) == 0 || len(prev) != len(cur) {
return 0, false
each, ok := raplWattsEach(prev, cur, max, dt)
var w float64
for _, x := range each {
w += x
}
var uj float64
return w, ok
}
// raplWattsEach is raplWatts per counter: one package each.
func raplWattsEach(prev, cur []raplCount, max map[string]uint64, dt float64) ([]float64, bool) {
if dt <= 0 || len(prev) == 0 || len(prev) != len(cur) {
return nil, false
}
out := make([]float64, len(cur))
for i := range cur {
if cur[i].Dir != prev[i].Dir {
return 0, false
return nil, false
}
if cur[i].UJ >= prev[i].UJ {
uj += float64(cur[i].UJ - prev[i].UJ)
out[i] = float64(cur[i].UJ-prev[i].UJ) / 1e6 / dt
continue
}
m := max[cur[i].Dir]
if m == 0 || prev[i].UJ > m {
return 0, false
return nil, false
}
uj += float64(m - prev[i].UJ + cur[i].UJ)
out[i] = float64(m-prev[i].UJ+cur[i].UJ) / 1e6 / dt
}
return uj / 1e6 / dt, true
return out, true
}
// cpuPower is the CPU packages' power since the previous sample, in watts; 0 when this
// machine has no counters or no way to read them.
func (c *Collector) cpuPower(now time.Time) float64 {
// machine has no counters or no way to read them. On a machine with more than one package
// it also gives each one's, indexed by package id (the socket).
func (c *Collector) cpuPower(now time.Time) (float64, []float64) {
p := &c.power
if !p.probed {
p.probed = true
@@ -131,7 +142,7 @@ func (c *Collector) cpuPower(now time.Time) float64 {
}
}
if len(p.dirs) == 0 {
return 0
return 0, nil
}
var cur []raplCount
ok := false
@@ -148,14 +159,40 @@ func (c *Collector) cpuPower(now time.Time) float64 {
}
if !ok {
p.prev = nil
return 0
return 0, nil
}
w, good := raplWatts(p.prev, cur, p.max, now.Sub(p.prevAt).Seconds())
each, good := raplWattsEach(p.prev, cur, p.max, now.Sub(p.prevAt).Seconds())
p.prev, p.prevAt = cur, now
if !good {
return 0
return 0, nil
}
return round1(w)
var w float64
for _, x := range each {
w += x
}
return round1(w), perPackage(cur, each)
}
// perPackage puts each counter's watts at its package's id — intel-rapl:1 is "package-1" —
// and is nil for a single package, whose figure is the total.
func perPackage(cur []raplCount, each []float64) []float64 {
if len(cur) < 2 {
return nil
}
var out []float64
for i, c := range cur {
id := i
if n, ok := strings.CutPrefix(strings.TrimSpace(readStr(filepath.Join(raplRoot, c.Dir, "name"))), "package-"); ok {
if v, err := strconv.Atoi(n); err == nil && v >= 0 && v < 64 {
id = v
}
}
for len(out) <= id {
out = append(out, 0)
}
out[id] = round1(each[i])
}
return out
}
// energyHelper is the root child that answers each newline with the package counters, one
+16
View File
@@ -0,0 +1,16 @@
package metrics
import "testing"
// a two-socket board: two coretemp chips, and the CPU's temperature is the hotter package
func TestCPUTempHottestPackage(t *testing.T) {
temps := []Temp{
{Chip: "coretemp", Label: "Core 0", C: 31},
{Chip: "coretemp", Label: "Core 0", C: 44, Pkg: 1},
{Chip: "coretemp", Label: "Package id 0", C: 35},
{Chip: "coretemp", Label: "Package id 1", C: 41, Pkg: 1},
}
if got := cpuTemp(temps); got != 41 {
t.Fatalf("cpuTemp = %v, want 41", got)
}
}
+64 -2
View File
@@ -1,5 +1,7 @@
package metrics
import "strconv"
// One kept history point is the average of the samples behind it. At the default ten-minute
// window that bucket is a single sample and nothing is averaged; at a day it is five minutes
// of them, and the difference matters — a graph of the last day drawn from whichever instant
@@ -43,6 +45,21 @@ func fold(b []Sample) Sample {
if pn > 0 {
out.CPUPower = round1(pw / pn)
}
// and each package's, the same way
out.CPUPowerPkgs = nil
for _, s := range b {
for i, w := range s.CPUPowerPkgs {
for len(out.CPUPowerPkgs) <= i {
out.CPUPowerPkgs = append(out.CPUPowerPkgs, 0)
}
out.CPUPowerPkgs[i] += w
}
}
if pn > 0 {
for i := range out.CPUPowerPkgs {
out.CPUPowerPkgs[i] = round1(out.CPUPowerPkgs[i] / pn)
}
}
out.Load[0] = sum(func(s Sample) float64 { return s.Load[0] })
out.Load[1] = sum(func(s Sample) float64 { return s.Load[1] })
out.Load[2] = sum(func(s Sample) float64 { return s.Load[2] })
@@ -70,13 +87,13 @@ func fold(b []Sample) Sample {
temps := avgBy(b, func(s Sample) map[string][]float64 {
m := make(map[string][]float64, len(s.Temps))
for _, x := range s.Temps {
m[x.Chip+"/"+x.Label] = []float64{x.C}
m[x.Chip+"/"+x.Label+"/"+strconv.Itoa(x.Pkg)] = []float64{x.C}
}
return m
})
out.Temps = append([]Temp(nil), out.Temps...)
for i, x := range out.Temps {
if v, ok := temps[x.Chip+"/"+x.Label]; ok {
if v, ok := temps[x.Chip+"/"+x.Label+"/"+strconv.Itoa(x.Pkg)]; ok {
out.Temps[i].C = v[0]
}
}
@@ -122,9 +139,54 @@ func fold(b []Sample) Sample {
out.Disks[i].ReadBps, out.Disks[i].WriteBps, out.Disks[i].UtilPct = v[0], v[1], v[2]
}
}
out.ZFSWrite = foldZFSWrites(b, out.ZFSWrite)
return out
}
// foldZFSWrites averages the write path's rates and the buffer's fill over a bucket, per pool;
// everything else (limits, the mover's state, the last walk) is the newest sample's.
func foldZFSWrites(b []Sample, last *ZFSWrites) *ZFSWrites {
if last == nil {
return nil
}
w := *last
w.Pools = append([]PoolWrites(nil), last.Pools...)
var dirty, delayed, over, n float64
for _, s := range b {
if s.ZFSWrite != nil {
dirty, delayed, over, n = dirty+float64(s.ZFSWrite.Dirty), delayed+s.ZFSWrite.DelayedPerSec, over+s.ZFSWrite.OverMaxPerSec, n+1
}
}
if n > 0 {
w.Dirty, w.DelayedPerSec, w.OverMaxPerSec = uint64(dirty/n), round1(delayed/n), round1(over/n)
}
per := avgBy(b, func(s Sample) map[string][]float64 {
m := map[string][]float64{}
if s.ZFSWrite == nil {
return m
}
for _, p := range s.ZFSWrite.Pools {
v := []float64{p.WriteBps, p.LogicalBps, 0, 0}
if p.Landing != nil {
v[2], v[3] = p.Landing.InBps, p.Landing.DrainBps
}
m[p.Pool] = v
}
return m
})
for i, p := range w.Pools {
if v, ok := per[p.Pool]; ok {
w.Pools[i].WriteBps, w.Pools[i].LogicalBps = round1(v[0]), round1(v[1])
if p.Landing != nil {
l := *p.Landing
l.InBps, l.DrainBps = round1(v[2]), round1(v[3])
w.Pools[i].Landing = &l
}
}
}
return &w
}
// avgBy averages each sample's named numbers across the bucket, per name.
func avgBy(b []Sample, of func(Sample) map[string][]float64) map[string][]float64 {
sums := map[string][]float64{}
+79 -22
View File
@@ -49,6 +49,7 @@ type GPU struct {
PowerW float64 `json:"power_w,omitempty"`
PowerCapW float64 `json:"power_cap_w,omitempty"`
FanRPM int `json:"fan_rpm,omitempty"`
FanPct *float64 `json:"fan_pct,omitempty"` // fan speed as a % of its range: nvidia-smi's fan.speed, or the hwmon's pwm1; nil when the card has no such reading (0 is a stopped fan)
MHz float64 `json:"mhz,omitempty"` // actual clock (0 when power-gated)
ReqMHz float64 `json:"req_mhz,omitempty"` // what the driver asked for
MaxMHz float64 `json:"max_mhz,omitempty"`
@@ -56,7 +57,7 @@ type GPU struct {
MemTotal uint64 `json:"mem_total,omitempty"`
Clients []GPUClient `json:"clients"`
ClientsOK bool `json:"clients_ok"` // false when fdinfo of other users' processes could not be read
Stalled bool `json:"stalled,omitempty"` // a sysfs read of this card never returned; its readings are paused until restart
Stalled bool `json:"stalled,omitempty"` // a read of this card has been out for gpuStallAfter; its readings are paused until it returns
}
// GPUClient is one process with the device open.
@@ -78,7 +79,17 @@ type gpuDev struct {
integrated, stalled bool
prevIdle, prevEnergy float64
prevAt time.Time
nv []GPUClient // nvidia: the process table from the last read (pmon), fdinfo says nothing there
nv []GPUClient // nvidia: the process table from the last read (pmon), fdinfo says nothing there
nvRead []GPUClient // the same, as the read in flight left it: only that read touches it
pending chan gpuRead // the read in flight, nil when none is
pendingAt time.Time
last *GPU // the last read that came back
readFn func(*GPU, time.Time) // nil is read (tests stand a slow one in)
}
type gpuRead struct {
x GPU
nv []GPUClient
}
type gpuState struct {
@@ -348,28 +359,60 @@ func lspciName(desc string) string {
return strings.TrimSpace(vendor + " " + desc)
}
// readWithin is read with a deadline. A sysfs read that does not come back must not take
// the whole sampler with it — every graph on every page of this machine stops — so the
// device is marked stalled, its goroutine abandoned, and the rest of the sample goes on.
const gpuReadWithin = 3 * time.Second
// readWithin is read with a deadline. A read that does not come back must not take the
// whole sampler with it — every graph on every page of this machine stops — so one still out
// after gpuReadWithin is left running, the sample goes on with the card's last readings, and
// no second read starts until it returns. Late is normal for nvidia-smi on a busy card; only a
// read out for gpuStallAfter marks the card stalled, and that clears the moment it returns. A
// sysfs read that never returns keeps it stalled, at the cost of one goroutine.
var (
gpuReadWithin = 3 * time.Second
gpuStallAfter = 15 * time.Second // past both of readNvidia's nvidia-smi calls timing out (runOut, 5 s each)
)
func (d *gpuDev) readWithin(x *GPU, now time.Time) {
if d.stalled {
x.Stalled = true
var r gpuRead
back := false
if d.pending == nil {
done := make(chan gpuRead, 1)
y := *x
read := d.read
if d.readFn != nil {
read = d.readFn
}
go func() {
read(&y, now)
done <- gpuRead{y, d.nvRead}
}()
d.pending, d.pendingAt = done, now
select {
case r, back = <-done:
case <-time.After(gpuReadWithin):
}
} else { // started on an earlier sample: taken if it is back, never waited on again
select {
case r, back = <-d.pending:
default:
}
}
ok := x.ClientsOK
defer func() { x.ClientsOK = ok }()
if back {
if d.stalled {
log.Printf("gpu: %s (%s): its read came back after %s; readings resume", d.name, d.card, now.Sub(d.pendingAt).Round(time.Second))
}
d.pending, d.nv, d.stalled, d.last = nil, r.nv, false, &r.x
*x = r.x
return
}
done := make(chan GPU, 1)
go func() {
y := *x
d.read(&y, now)
done <- y
}()
select {
case y := <-done:
*x = y
case <-time.After(gpuReadWithin):
if d.last != nil {
*x = *d.last
}
if now.Sub(d.pendingAt) >= gpuStallAfter {
if !d.stalled {
log.Printf("gpu: %s (%s): a read has not returned in %s; its readings are paused until it does", d.name, d.card, gpuStallAfter)
}
d.stalled, x.Stalled = true, true
log.Printf("gpu: %s (%s): a sysfs read did not return within %s; its readings are paused until restart", d.name, d.card, gpuReadWithin)
}
}
@@ -437,6 +480,14 @@ func (d *gpuDev) read(x *GPU, now time.Time) {
x.PowerCapW = p / 1e6
}
x.FanRPM = int(readF(d.hwmon + "/fan1_input"))
if pwm, err := strconv.ParseFloat(strings.TrimSpace(readStr(d.hwmon+"/pwm1")), 64); err == nil {
top := readF(d.hwmon + "/pwm1_max")
if top <= 0 {
top = 255
}
v := round1(pwm / top * 100)
x.FanPct = &v
}
}
d.prevAt = now
}
@@ -720,11 +771,11 @@ func ueventValue(p, key string) string {
// sample (~20 ms and ~75 ms on a GTX 965M), no root needed. The PCI address selects the
// card, so two NVIDIA GPUs stay apart.
func (d *gpuDev) readNvidia(x *GPU) {
if out, ok := runOut("nvidia-smi", "--id="+d.bus, "--query-gpu=utilization.gpu,utilization.encoder,utilization.decoder,memory.used,memory.total,clocks.sm,clocks.max.sm,temperature.gpu,power.draw,power.limit", "--format=csv,noheader,nounits"); ok {
if out, ok := runOut("nvidia-smi", "--id="+d.bus, "--query-gpu=utilization.gpu,utilization.encoder,utilization.decoder,memory.used,memory.total,clocks.sm,clocks.max.sm,temperature.gpu,power.draw,power.limit,fan.speed", "--format=csv,noheader,nounits"); ok {
parseNvidiaQuery(x, out)
}
if out, ok := runOut("nvidia-smi", "pmon", "-c", "1", "-s", "um", "--id="+d.bus); ok {
d.nv = parseNvidiaPmon(out)
d.nvRead = parseNvidiaPmon(out)
}
}
@@ -739,7 +790,8 @@ func nvNum(s string) (float64, bool) {
}
// parseNvidiaQuery fills a GPU from one --query-gpu line in the field order readNvidia asks
// for: gpu%, encoder%, decoder%, mem used MiB, mem total MiB, SM MHz, max SM MHz, °C, W, W cap.
// for: gpu%, encoder%, decoder%, mem used MiB, mem total MiB, SM MHz, max SM MHz, °C, W, W cap,
// fan % ("[N/A]" on a card with no fan of its own: a laptop's, a passive datacentre card).
func parseNvidiaQuery(x *GPU, line string) {
f := strings.Split(strings.TrimSpace(line), ",")
if len(f) < 10 {
@@ -775,6 +827,11 @@ func parseNvidiaQuery(x *GPU, line string) {
if v, ok := nvNum(f[9]); ok {
x.PowerCapW = v
}
if len(f) > 10 {
if v, ok := nvNum(f[10]); ok {
x.FanPct = &v
}
}
}
// parseNvidiaPmon reads `nvidia-smi pmon -c 1 -s um`: a header naming the columns, then one
+52
View File
@@ -0,0 +1,52 @@
package metrics
import (
"testing"
"time"
)
// A read that comes back late carries the last readings meanwhile and is used when it lands;
// only one out past gpuStallAfter marks the card stalled, and its return clears that.
func TestGPUReadLateThenStalledThenBack(t *testing.T) {
defer func(w, s time.Duration) { gpuReadWithin, gpuStallAfter = w, s }(gpuReadWithin, gpuStallAfter)
gpuReadWithin, gpuStallAfter = 20*time.Millisecond, 10*time.Second
release := make(chan struct{})
n := 0.0 // only the read goroutine writes it, one read at a time
d := &gpuDev{name: "test", card: "card0"}
d.readFn = func(x *GPU, _ time.Time) {
n++
if n == 2 {
<-release
}
x.Usage = n * 10
}
t0 := time.Now()
sample := func(at time.Duration) GPU {
x := GPU{Name: "test"}
d.readWithin(&x, t0.Add(at))
return x
}
if x := sample(0); x.Usage != 10 || x.Stalled {
t.Fatalf("first: %+v", x)
}
if x := sample(time.Second); x.Usage != 10 || x.Stalled { // late: the last readings
t.Fatalf("late: %+v", x)
}
if x := sample(5 * time.Second); x.Usage != 10 || x.Stalled {
t.Fatalf("still late: %+v", x)
}
if x := sample(12 * time.Second); !x.Stalled {
t.Fatalf("out past gpuStallAfter, not stalled: %+v", x)
}
if x := sample(12*time.Second + 1); !x.Stalled {
t.Fatal("stall did not hold")
}
close(release)
time.Sleep(10 * time.Millisecond)
if x := sample(13 * time.Second); x.Usage != 20 || x.Stalled {
t.Fatalf("back: %+v", x)
}
if x := sample(14 * time.Second); x.Usage != 30 || x.Stalled { // 30: no extra read ran while one was out
t.Fatalf("next: %+v", x)
}
}
+14
View File
@@ -182,6 +182,20 @@ func TestParseNvidiaQuery(t *testing.T) {
if g.Temp != 57 || g.TempSrc != "gpu" || g.PowerW != 23.4 || g.PowerCapW != 0 {
t.Errorf("temp/power (cap must stay 0 for [N/A]): %+v", g)
}
if g.FanPct != nil {
t.Errorf("no fan field must leave the fan unset: %+v", g)
}
var fan, nofan GPU
parseNvidiaQuery(&fan, "99, 0, 0, 20000, 24576, 1695, 2100, 83, 403.99, 420.00, 34\n")
parseNvidiaQuery(&nofan, "5, 0, 0, 500, 2048, 300, 1097, 45, 8.0, [N/A], [N/A]\n")
if fan.FanPct == nil || *fan.FanPct != 34 || nofan.FanPct != nil {
t.Errorf("fan: %v, [N/A]: %v", fan.FanPct, nofan.FanPct)
}
var stopped GPU
parseNvidiaQuery(&stopped, "0, 0, 0, 500, 24576, 210, 2100, 35, 20.0, 420.00, 0\n")
if stopped.FanPct == nil || *stopped.FanPct != 0 {
t.Errorf("a stopped fan must read 0, not nothing: %v", stopped.FanPct)
}
var short GPU
parseNvidiaQuery(&short, "garbage")
if short.UsageSrc != "" {
+109 -44
View File
@@ -60,6 +60,7 @@ type Temp struct {
C float64 `json:"c"`
Max float64 `json:"max,omitempty"`
Crit float64 `json:"crit,omitempty"`
Pkg int `json:"pkg,omitempty"` // coretemp: the CPU package the chip belongs to — a two-socket board has two chips, both with a "Core 0"
}
type Fan struct {
@@ -101,6 +102,9 @@ type FS struct {
// said honestly: a ZFS pool or an md array spreads a filesystem over several drives, and
// a remote mount is on none of them.
Disk string `json:"disk,omitempty"`
// Source is the device the mount is of ("/dev/nvme0n1p1"). A bind mount has the same
// source as the filesystem it shows, so a sum over a drive counts each source once.
Source string `json:"source,omitempty"`
}
type UPS struct {
@@ -135,23 +139,25 @@ type Proc struct {
}
type Sample struct {
T int64 `json:"t"` // unix ms
Interval int64 `json:"interval_ms"` // sampling interval in effect
CPU float64 `json:"cpu"`
Cores []float64 `json:"cores"`
CPUPower float64 `json:"cpu_power_w,omitempty"` // the CPU packages' own energy counters; 0 = no reading
Load [3]float64 `json:"load"`
Mem Mem `json:"mem"`
Swap Mem `json:"swap"`
Temps []Temp `json:"temps"`
Fans []Fan `json:"fans"`
Net []Net `json:"net"`
Disks []Disk `json:"disks"`
FS []FS `json:"fs"`
UPS *UPS `json:"ups"`
GPUs []GPU `json:"gpus"`
ARC *ARC `json:"arc"` // the ZFS read cache; nil without the module
Uptime float64 `json:"uptime"`
T int64 `json:"t"` // unix ms
Interval int64 `json:"interval_ms"` // sampling interval in effect
CPU float64 `json:"cpu"`
Cores []float64 `json:"cores"`
CPUPower float64 `json:"cpu_power_w,omitempty"` // the CPU packages' own energy counters; 0 = no reading
CPUPowerPkgs []float64 `json:"cpu_power_pkgs_w,omitempty"` // each package's share, by socket; only with more than one
Load [3]float64 `json:"load"`
Mem Mem `json:"mem"`
Swap Mem `json:"swap"`
Temps []Temp `json:"temps"`
Fans []Fan `json:"fans"`
Net []Net `json:"net"`
Disks []Disk `json:"disks"`
FS []FS `json:"fs"`
UPS *UPS `json:"ups"`
GPUs []GPU `json:"gpus"`
ARC *ARC `json:"arc"` // the ZFS read cache; nil without the module
ZFSWrite *ZFSWrites `json:"zfs_writes"` // the ZFS write path; nil without the module
Uptime float64 `json:"uptime"`
fsT, upsT int64 // when the slow sections were really read; sensors report these, not T
}
@@ -182,6 +188,7 @@ type Collector struct {
gpu gpuState
power cpuPowerState
prevARC arcPrev
prevZW zfsWritePrev
subMu sync.Mutex
subs map[chan struct{}]struct{} // told after every sample; see Subscribe
@@ -347,7 +354,7 @@ func (c *Collector) sample() {
s := Sample{T: now.UnixMilli(), Interval: c.Interval().Milliseconds(), Temps: []Temp{}, Fans: []Fan{}, Net: []Net{}, Disks: []Disk{}}
s.CPU, s.Cores = c.cpu()
s.CPUPower = c.cpuPower(now)
s.CPUPower, s.CPUPowerPkgs = c.cpuPower(now)
s.Load = loadavg()
s.Mem, s.Swap = meminfo()
s.Uptime = uptime()
@@ -368,6 +375,7 @@ func (c *Collector) sample() {
s.UPS, s.upsT = c.lastUPS, c.upsAt.UnixMilli()
s.GPUs = c.gpus(now, cpuTemp(s.Temps))
s.ARC = c.arc(now)
s.ZFSWrite = c.zfsWrites(now, s.ARC, s.Disks)
if first {
return // deltas are meaningless on the priming pass
@@ -541,6 +549,19 @@ func hwmonDir(d string) (chip string, temps []Temp, fans []Fan) {
fans = append(fans, Fan{Chip: chip, Label: label, RPM: int(raw)})
}
}
// coretemp is one chip per CPU package, and only its "Package id N" says which
if chip == "coretemp" {
for _, t := range temps {
if n, ok := strings.CutPrefix(t.Label, "Package id "); ok {
if pkg, err := strconv.Atoi(n); err == nil {
for i := range temps {
temps[i].Pkg = pkg
}
}
break
}
}
}
return
}
@@ -556,7 +577,10 @@ func hwmon() ([]Temp, []Fan) {
if temps[i].Chip != temps[j].Chip {
return temps[i].Chip < temps[j].Chip
}
return temps[i].Label < temps[j].Label
if temps[i].Label != temps[j].Label {
return temps[i].Label < temps[j].Label
}
return temps[i].Pkg < temps[j].Pkg
})
return temps, fans
}
@@ -842,7 +866,7 @@ func filesystems() []FS {
return
}
bs := uint64(st.Bsize)
done <- FS{Mount: mount, Type: fs[2], Size: st.Blocks * bs, Used: (st.Blocks - st.Bfree) * bs, Remote: remoteFS[fs[2]], Disk: diskOf(fs[0])}
done <- FS{Mount: mount, Type: fs[2], Size: st.Blocks * bs, Used: (st.Blocks - st.Bfree) * bs, Remote: remoteFS[fs[2]], Disk: diskOf(fs[0]), Source: fs[0]}
}()
select {
case r := <-done:
@@ -1109,27 +1133,61 @@ func coreIDs(n int) []int {
return coreIDsVal[:n]
}
// corePkgs maps logical CPUs to the CPU package (socket) each is on, and is nil on a
// one-socket machine: there is nothing to tell apart.
var (
corePkgsOnce sync.Once
corePkgsVal []int
)
func corePkgs(n int) []int {
corePkgsOnce.Do(func() {
seen := map[int]bool{}
var out []int
for i := 0; ; i++ {
b, err := os.ReadFile("/sys/devices/system/cpu/cpu" + strconv.Itoa(i) + "/topology/physical_package_id")
if err != nil {
break
}
id, err := strconv.Atoi(strings.TrimSpace(string(b)))
if err != nil {
break
}
out, seen[id] = append(out, id), true
}
if len(seen) > 1 {
corePkgsVal = out
}
})
if len(corePkgsVal) < n {
return nil
}
return corePkgsVal[:n]
}
// ProcTable is the Processes tab payload.
type ProcTable struct {
T int64 `json:"t"`
CPU float64 `json:"cpu"`
Cores []float64 `json:"cores"`
CPUHist []float64 `json:"cpu_hist"` // recent total-CPU samples, oldest first
CPUModel string `json:"cpu_model,omitempty"`
CPUMHz float64 `json:"cpu_mhz,omitempty"`
CPUMaxMHz float64 `json:"cpu_max_mhz,omitempty"`
CPUPowerW float64 `json:"cpu_power_w,omitempty"`
CPUTemp float64 `json:"cpu_temp,omitempty"`
CoreIDs []int `json:"core_ids,omitempty"` // for each logical CPU, the physical core it is a thread of (what the temperature sensors are named by)
Load [3]float64 `json:"load"`
Mem Mem `json:"mem"`
Swap Mem `json:"swap"`
Net []Net `json:"net"`
GPUs []GPU `json:"gpus"`
Uptime float64 `json:"uptime"`
Procs []Proc `json:"procs"`
Total int `json:"total"`
Running int `json:"running"`
T int64 `json:"t"`
CPU float64 `json:"cpu"`
Cores []float64 `json:"cores"`
CPUHist []float64 `json:"cpu_hist"` // recent total-CPU samples, oldest first
CPUModel string `json:"cpu_model,omitempty"`
CPUMHz float64 `json:"cpu_mhz,omitempty"`
CPUMaxMHz float64 `json:"cpu_max_mhz,omitempty"`
CPUPowerW float64 `json:"cpu_power_w,omitempty"`
CPUPowerPkgsW []float64 `json:"cpu_power_pkgs_w,omitempty"`
CPUTemp float64 `json:"cpu_temp,omitempty"`
CoreIDs []int `json:"core_ids,omitempty"` // for each logical CPU, the physical core it is a thread of (what the temperature sensors are named by)
CorePkgs []int `json:"core_pkgs,omitempty"` // for each logical CPU, the socket it is on; only on a machine with more than one
Load [3]float64 `json:"load"`
Mem Mem `json:"mem"`
Swap Mem `json:"swap"`
Net []Net `json:"net"`
GPUs []GPU `json:"gpus"`
Uptime float64 `json:"uptime"`
Procs []Proc `json:"procs"`
Total int `json:"total"`
Running int `json:"running"`
}
var cpuModel = func() string {
@@ -1179,18 +1237,24 @@ func cpuMHz() float64 {
// cpuTemp picks the sensor that best represents the CPU package.
func cpuTemp(temps []Temp) float64 {
best, max := 0.0, 0.0
// the hottest package: a two-socket board has two, and the hotter one is the one to watch
pkg, max := 0.0, 0.0
for _, t := range temps {
l := strings.ToLower(t.Label)
if t.Chip == "coretemp" && strings.HasPrefix(l, "package") || t.Chip == "k10temp" && (l == "tctl" || l == "tdie") ||
t.Chip == "thermal" && strings.Contains(l, "cpu") || t.Chip == "cpu_thermal" || t.Chip == "soc_thermal" {
return t.C
if t.C > pkg {
pkg = t.C
}
}
if (t.Chip == "coretemp" || t.Chip == "k10temp") && t.C > max {
max, best = t.C, t.C
max = t.C
}
}
return best
if pkg > 0 {
return pkg
}
return max
}
var bootTime = func() int64 {
@@ -1318,8 +1382,9 @@ func (c *Collector) Processes() ProcTable {
t.Mem, t.Swap = meminfo()
if n := len(c.ring); n > 0 {
last := c.ring[n-1]
t.CPU, t.Cores, t.Net, t.CPUTemp, t.CPUPowerW = last.CPU, last.Cores, last.Net, cpuTemp(last.Temps), last.CPUPower
t.CPU, t.Cores, t.Net, t.CPUTemp, t.CPUPowerW, t.CPUPowerPkgsW = last.CPU, last.Cores, last.Net, cpuTemp(last.Temps), last.CPUPower, last.CPUPowerPkgs
t.CoreIDs = coreIDs(len(last.Cores))
t.CorePkgs = corePkgs(len(last.Cores))
from := n - 90
if from < 0 {
from = 0
+47 -1
View File
@@ -65,6 +65,10 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
if s.CPUPower > 0 {
f.add("cpu.power", "CPU power", "power", "W", s.CPUPower, t, none)
}
for i, w := range s.CPUPowerPkgs {
n := strconv.Itoa(i)
f.add("cpu.package."+n+".power", "CPU "+strconv.Itoa(i+1)+" power", "power", "W", w, t, none, "package", n)
}
for i, w := range []string{"1", "5", "15"} {
f.add("load."+w, "Load "+w+" min", "load", "", s.Load[i], t, none)
}
@@ -92,7 +96,11 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
if x.Crit > 0 {
b.crit = sensors.F(x.Crit)
}
f.add(sensors.ID("temp", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "temperature", "°C", x.C, t, b, "chip", x.Chip, "label", x.Label)
labels := []string{"chip", x.Chip, "label", x.Label}
if x.Chip == "coretemp" { // a two-socket board has a coretemp chip per CPU, both with a "Core 0"
labels = append(labels, "package", strconv.Itoa(x.Pkg))
}
f.add(sensors.ID("temp", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "temperature", "°C", x.C, t, b, labels...)
}
for _, x := range s.Fans {
f.add(sensors.ID("fan", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "fan", "rpm", float64(x.RPM), t, none, "chip", x.Chip, "label", x.Label)
@@ -129,13 +137,22 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
if x.Disk != "" {
lbl = append(lbl, "disk", x.Disk)
}
if x.Source != "" {
lbl = append(lbl, "source", x.Source)
}
f.add(p+".size", x.Mount+" size", "bytes", "B", float64(x.Size), s.fsT, none, lbl...)
f.add(p+".used", x.Mount+" used", "bytes", "B", float64(x.Used), s.fsT, upTo(float64(x.Size)), lbl...)
f.add(p+".used_pct", x.Mount+" used", "usage", "%", ratio(x.Used, x.Size), s.fsT, pct, lbl...)
}
gpuSeen := map[string]int{}
for _, g := range s.GPUs {
// two identical cards are gpu.<name> and gpu.<name>_2, every sensor of the second under
// its own stem (not add's suffix on the end of each id, which the page can't find)
p := "gpu." + sensors.Seg(g.Name)
if gpuSeen[p]++; gpuSeen[p] > 1 {
p = fmt.Sprintf("%s_%d", p, gpuSeen[p])
}
lbl := []string{"name", g.Name, "driver", g.Driver}
if g.Bus != "" {
lbl = append(lbl, "bus", g.Bus)
@@ -164,6 +181,9 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
if g.FanRPM > 0 {
f.add(p+".fan", g.Name+" fan", "fan", "rpm", float64(g.FanRPM), t, none, lbl...)
}
if g.FanPct != nil {
f.add(p+".fan_pct", g.Name+" fan", "fan", "%", *g.FanPct, t, pct, lbl...)
}
if g.MaxMHz > 0 {
f.add(p+".freq", g.Name+" clock", "frequency", "MHz", g.MHz, t, upTo(g.MaxMHz), lbl...)
}
@@ -203,6 +223,32 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
}
}
if w := s.ZFSWrite; w != nil {
f.add("zfs.write.dirty", "ZFS write buffer", "bytes", "B", float64(w.Dirty), t, upTo(float64(w.Limit)))
f.add("zfs.write.limit", "ZFS write buffer limit", "bytes", "B", float64(w.Limit), t, none)
f.add("zfs.write.delayed", "ZFS writes throttled", "rate", "/s", w.DelayedPerSec, t, none)
for _, p := range w.Pools {
pp := "zfs." + sensors.Seg(p.Pool) + "."
f.add(pp+"write_bps", p.Pool+" writes to disk", "rate", "B/s", p.WriteBps, t, none, "pool", p.Pool)
f.add(pp+"logical_bps", p.Pool+" writes by programs", "rate", "B/s", p.LogicalBps, t, none, "pool", p.Pool)
if l := p.Landing; l != nil {
lp := pp + "landing."
lbl := []string{"pool", p.Pool, "branch", l.Branch}
f.add(lp+"pending_bytes", p.Pool+" landing: waiting", "bytes", "B", float64(l.PendingBytes), t, upTo(float64(l.Size)), lbl...)
f.add(lp+"pending_files", p.Pool+" landing: files waiting", "count", "", float64(l.PendingFiles), t, none, lbl...)
if l.Size > 0 {
f.add(lp+"used_pct", p.Pool+" landing: used", "usage", "%", round1(float64(l.Used)/float64(l.Size)*100), t, pct, lbl...)
}
f.add(lp+"in_bps", p.Pool+" landing: arriving", "rate", "B/s", l.InBps, t, none, lbl...)
f.add(lp+"drain_bps", p.Pool+" landing: moving to the pool", "rate", "B/s", l.DrainBps, t, none, lbl...)
if m := l.Mover; m != nil {
f.add(lp+"mover", p.Pool+" landing: mover", "state", "", m.State, t, none, lbl...)
f.add(lp+"queue", p.Pool+" landing: mover queue", "count", "", float64(m.Queue), t, none, lbl...)
}
}
}
}
if u := s.UPS; u != nil {
ut := s.upsT
f.add("ups.battery", "UPS battery", "usage", "%", u.BatteryPct, ut, pct, "model", u.Model)
+456
View File
@@ -0,0 +1,456 @@
package metrics
// The ZFS write path, for the ZFS tab: what is waiting in RAM to be written, how fast each
// pool is writing to its disks, and — where a pool has one — the landing tier in front of it.
//
// A landing tier is a mergerfs pool whose first branches are fast non-ZFS storage and whose
// last branch is a ZFS dataset: new files land on the fast branch and a mover drains them onto
// the pool. It is detected from mergerfs's own branch list (the user.mergerfs.branches xattr on
// the pool's .mergerfs control file), so it shows wherever one exists and nowhere else. A mover
// that writes a status file into /run/landing-mover/ (hamlet's scripts/landing-mover.py does)
// adds what it is doing: its queue, the file in flight, totals and the last error.
//
// Sources, all cheap enough for every sample:
// - arcstats anon_size: dirty data in RAM, for all pools together (ZFS has no live per-pool
// figure; txgs only records ndirty once a txg has synced)
// - dmu_tx: dmu_tx_dirty_delay / dmu_tx_dirty_over_max — writers slowed by the write throttle,
// or stopped at the hard limit
// - <pool>/txgs: nwritten per synced txg — bytes the pool actually wrote to its vdevs
// - <pool>/objset-*: nwritten per dataset — bytes programs wrote, before compression
// The landing branch's file walk is slower and runs every landingEvery.
import (
"encoding/json"
"io/fs"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"syscall"
"time"
)
const (
zfsKstatDir = "/proc/spl/kstat/zfs/"
zfsParamDir = "/sys/module/zfs/parameters/"
moverStateDir = "/run/landing-mover"
landingEvery = 10 * time.Second
landingWalkMax = 200000 // entries; past this the counts are marked partial
)
// ZFSWrites is the write path as one sample sees it. Nil without the ZFS module.
type ZFSWrites struct {
Dirty uint64 `json:"dirty"` // bytes in RAM written but not yet on disk, all pools
DirtyMax uint64 `json:"dirty_max"` // zfs_dirty_data_max
DelayMinPct int `json:"delay_min_pct"` // zfs_delay_min_dirty_percent: the throttle starts here
Limit uint64 `json:"limit"` // what the buffer can really hold now: the smaller of dirty_max and half the ARC target
DelayedPerSec float64 `json:"delayed_per_sec"` // writes the throttle slowed, per second over the interval
OverMaxPerSec float64 `json:"over_max_per_sec"` // writes stopped dead at the limit
TxgTimeoutSec int `json:"txg_timeout_sec"` // zfs_txg_timeout
Pools []PoolWrites `json:"pools"`
}
// PoolWrites is one pool.
type PoolWrites struct {
Pool string `json:"pool"`
WriteBps float64 `json:"write_bps"` // to its vdevs, after compression, parity included
LogicalBps float64 `json:"logical_bps"` // into its datasets, as programs wrote it
Landing *Landing `json:"landing,omitempty"`
}
// Landing is a landing tier in front of a pool.
type Landing struct {
Mount string `json:"mount"` // the mergerfs pool, e.g. /srv/tank
Branch string `json:"branch"` // the landing branch, e.g. /srv/landing/tank
PoolBranch string `json:"pool_branch"`
Device string `json:"device,omitempty"` // block device under the branch, as in Sample.Disks
FSType string `json:"fstype,omitempty"`
Size uint64 `json:"size"`
Used uint64 `json:"used"`
PendingFiles int `json:"pending_files"`
PendingBytes uint64 `json:"pending_bytes"`
PendingPartial bool `json:"pending_partial,omitempty"` // the walk stopped at landingWalkMax
OldestMs int64 `json:"oldest_ms,omitempty"` // mtime of the oldest waiting file
CountedMs int64 `json:"counted_ms"` // when the walk ran
InBps float64 `json:"in_bps"` // the landing device's writes
DrainBps float64 `json:"drain_bps"` // the mover's progress onto the pool
Mover *Mover `json:"mover,omitempty"`
}
// Mover is what a landing mover says about itself in its status file.
type Mover struct {
State string `json:"state"` // idle | moving | holding (a mount it needs is missing)
Src string `json:"src"`
Dst string `json:"dst"`
PID int `json:"pid"`
StartedMs int64 `json:"started_ms"`
UpdatedMs int64 `json:"updated_ms"`
Stale bool `json:"stale"` // not updated for a while: stopped or hung
Queue int `json:"queue"`
Settle float64 `json:"settle"`
BWLimit string `json:"bwlimit"`
BatchFiles int `json:"batch_files"`
BatchBytes uint64 `json:"batch_bytes"`
BatchStartMs int64 `json:"batch_start_ms"`
BatchFirst string `json:"batch_first"`
Current string `json:"current"` // the file being copied, when it can tell
CurrentSize uint64 `json:"current_size"` // its full size
Partial uint64 `json:"partial"` // bytes of it already on the pool
MovedFiles uint64 `json:"moved_files"` // since the mover started
MovedBytes uint64 `json:"moved_bytes"`
LastMove string `json:"last_move"`
LastMoveMs int64 `json:"last_move_ms"`
LastError string `json:"last_error"`
LastErrorMs int64 `json:"last_error_ms"`
}
type zfsWritePrev struct {
at time.Time
delay uint64
overMax uint64
txg map[string]uint64 // pool -> highest synced txg counted
txgAt map[string]time.Time // pool -> when that txg was first seen
txgRate map[string]float64 // pool -> bytes/s over the last gap between syncs
logical map[string]uint64 // pool -> sum of objset nwritten
drained map[string]uint64 // landing branch -> moved + partial bytes, at the mover's last update
drainedAt map[string]int64 // landing branch -> that update's time, unix ms
drainRate map[string]float64 // landing branch -> bytes/s between its last two updates
landing map[string]Landing // branch -> last walk
landingAt time.Time
}
func readParamU(name string) uint64 {
v, _ := strconv.ParseUint(strings.TrimSpace(readStr(zfsParamDir+name)), 10, 64)
return v
}
// poolsWithKstats lists the imported pools: the directories under the kstat tree.
func poolsWithKstats() []string {
ents, err := os.ReadDir(zfsKstatDir)
if err != nil {
return nil
}
var out []string
for _, e := range ents {
if e.IsDir() {
if _, err := os.Stat(zfsKstatDir + e.Name() + "/txgs"); err == nil {
out = append(out, e.Name())
}
}
}
sort.Strings(out)
return out
}
// parseTxgs reads a pool's txgs kstat and returns, for every committed txg above `after`, the
// bytes it wrote, plus the highest committed txg seen.
func parseTxgs(text string, after uint64) (written, top uint64) {
top = after
for _, line := range strings.Split(text, "\n") {
f := strings.Fields(line)
// txg birth state ndirty nread nwritten reads writes otime qtime wtime stime
if len(f) < 12 || f[2] != "C" {
continue
}
txg, err := strconv.ParseUint(f[0], 10, 64)
if err != nil {
continue
}
if txg > top {
top = txg
}
if txg > after {
n, _ := strconv.ParseUint(f[5], 10, 64)
written += n
}
}
return written, top
}
// objsetWritten sums nwritten over a pool's datasets.
func objsetWritten(pool string) uint64 {
m, _ := filepath.Glob(zfsKstatDir + pool + "/objset-*")
var t uint64
for _, p := range m {
for _, line := range strings.Split(readStr(p), "\n") {
f := strings.Fields(line)
if len(f) == 3 && f[0] == "nwritten" {
v, _ := strconv.ParseUint(f[2], 10, 64)
t += v
}
}
}
return t
}
// mountEntry is one line of /proc/mounts.
type mountEntry struct{ src, point, fstype string }
func readMounts() []mountEntry {
var out []mountEntry
for _, line := range strings.Split(readStr("/proc/mounts"), "\n") {
f := strings.Fields(line)
if len(f) >= 3 {
out = append(out, mountEntry{src: unescapeMount(f[0]), point: unescapeMount(f[1]), fstype: f[2]})
}
}
return out
}
// unescapeMount undoes /proc/mounts' octal escapes (\040 for a space, and so on).
func unescapeMount(s string) string {
if !strings.Contains(s, `\`) {
return s
}
var b strings.Builder
for i := 0; i < len(s); i++ {
if s[i] == '\\' && i+3 < len(s) {
if v, err := strconv.ParseUint(s[i+1:i+4], 8, 8); err == nil {
b.WriteByte(byte(v))
i += 3
continue
}
}
b.WriteByte(s[i])
}
return b.String()
}
// mountFor is the mount a path lives on: the longest mount point that is a prefix of it.
func mountFor(ms []mountEntry, path string) (mountEntry, bool) {
var best mountEntry
found := false
for _, m := range ms {
if m.point == path || m.point == "/" || strings.HasPrefix(path, strings.TrimSuffix(m.point, "/")+"/") {
if !found || len(m.point) > len(best.point) {
best, found = m, true
}
}
}
return best, found
}
// parseBranches reads mergerfs's branch list: "/srv/landing/tank=RW:/tank=RW,50G".
func parseBranches(v string) (paths []string, modes []string) {
for _, b := range strings.Split(strings.TrimSpace(v), ":") {
if b == "" {
continue
}
p, mode, _ := strings.Cut(b, "=")
mode, _, _ = strings.Cut(mode, ",")
if mode == "" {
mode = "RW"
}
paths, modes = append(paths, p), append(modes, mode)
}
return paths, modes
}
func getxattr(path, name string) string {
buf := make([]byte, 4096)
n, err := syscall.Getxattr(path, name, buf)
if err != nil || n <= 0 {
return ""
}
return string(buf[:n])
}
// findLandings maps pool -> the landing tiers in front of it.
func findLandings(ms []mountEntry) map[string][]Landing {
out := map[string][]Landing{}
for _, m := range ms {
if m.fstype != "fuse.mergerfs" {
continue
}
paths, modes := parseBranches(getxattr(filepath.Join(m.point, ".mergerfs"), "user.mergerfs.branches"))
// the pool branch is the first ZFS one; landing branches are writable non-ZFS ones before it
for i, p := range paths {
bm, ok := mountFor(ms, p)
if !ok || bm.fstype != "zfs" {
continue
}
pool, _, _ := strings.Cut(bm.src, "/")
for j := 0; j < i; j++ {
lm, ok := mountFor(ms, paths[j])
if !ok || lm.fstype == "zfs" || modes[j] == "RO" {
continue
}
out[pool] = append(out[pool], Landing{Mount: m.point, Branch: paths[j], PoolBranch: p, FSType: lm.fstype, Device: blockDisk(lm.src)})
}
break
}
}
return out
}
// blockDisk turns a mount source into the name Sample.Disks uses: /dev/nvme0n1p1 -> nvme0,
// /dev/sdb2 -> sdb. Empty for anything that is not a plain block device.
func blockDisk(src string) string {
dev, err := filepath.EvalSymlinks(src)
if err != nil || !strings.HasPrefix(dev, "/dev/") {
return ""
}
name := filepath.Base(dev)
// a partition's sysfs entry sits inside its disk's
if p, err := filepath.EvalSymlinks("/sys/class/block/" + name); err == nil {
if _, err := os.Stat(p + "/partition"); err == nil {
name = filepath.Base(filepath.Dir(p))
}
}
if m := nsRe.FindStringSubmatch(name); m != nil {
return m[1] // nvme0n1 -> nvme0, as foldNVMe names it
}
return name
}
// walkLanding counts what is waiting on a landing branch.
func walkLanding(l *Landing, now time.Time) {
var st syscall.Statfs_t
if syscall.Statfs(l.Branch, &st) == nil {
l.Size = st.Blocks * uint64(st.Bsize)
l.Used = (st.Blocks - st.Bfree) * uint64(st.Bsize)
}
seen := 0
l.PendingFiles, l.PendingBytes, l.OldestMs, l.PendingPartial = 0, 0, 0, false
_ = filepath.WalkDir(l.Branch, func(p string, d fs.DirEntry, err error) error {
if err != nil {
return nil
}
if seen++; seen > landingWalkMax {
l.PendingPartial = true
return filepath.SkipAll
}
if !d.Type().IsRegular() {
return nil
}
if fi, err := d.Info(); err == nil {
l.PendingFiles++
l.PendingBytes += uint64(fi.Size())
if ms := fi.ModTime().UnixMilli(); l.OldestMs == 0 || ms < l.OldestMs {
l.OldestMs = ms
}
}
return nil
})
l.CountedMs = now.UnixMilli()
}
// readMovers loads every mover status file, keyed by the branch it drains.
func readMovers(now time.Time) map[string]*Mover {
out := map[string]*Mover{}
m, _ := filepath.Glob(moverStateDir + "/*.json")
for _, p := range m {
b, err := os.ReadFile(p)
if err != nil {
continue
}
var mv Mover
if json.Unmarshal(b, &mv) != nil || mv.Src == "" {
continue
}
mv.Stale = now.UnixMilli()-mv.UpdatedMs > 30_000
if mv.Stale {
mv.State = "stopped"
}
out[filepath.Clean(mv.Src)] = &mv
}
return out
}
// zfsWrites samples the write path; nil when this machine has no ZFS module loaded.
func (c *Collector) zfsWrites(now time.Time, arc *ARC, disks []Disk) *ZFSWrites {
if arc == nil {
return nil
}
p := &c.prevZW
dt := 0.0
if !p.at.IsZero() {
dt = now.Sub(p.at).Seconds()
}
w := &ZFSWrites{Dirty: arc.Anon, DirtyMax: readParamU("zfs_dirty_data_max"),
DelayMinPct: int(readParamU("zfs_delay_min_dirty_percent")), TxgTimeoutSec: int(readParamU("zfs_txg_timeout")), Pools: []PoolWrites{}}
w.Limit = w.DirtyMax
if half := arc.Target / 2; half > 0 && half < w.Limit {
w.Limit = half
}
tx := readKstat(zfsKstatDir + "dmu_tx")
if dt > 0 && tx["dmu_tx_dirty_delay"] >= p.delay && tx["dmu_tx_dirty_over_max"] >= p.overMax {
w.DelayedPerSec = round1(float64(tx["dmu_tx_dirty_delay"]-p.delay) / dt)
w.OverMaxPerSec = round1(float64(tx["dmu_tx_dirty_over_max"]-p.overMax) / dt)
}
p.delay, p.overMax = tx["dmu_tx_dirty_delay"], tx["dmu_tx_dirty_over_max"]
if p.txg == nil {
p.txg, p.logical, p.drained, p.landing = map[string]uint64{}, map[string]uint64{}, map[string]uint64{}, map[string]Landing{}
p.txgAt, p.txgRate, p.drainedAt, p.drainRate = map[string]time.Time{}, map[string]float64{}, map[string]int64{}, map[string]float64{}
}
ms := readMounts()
landings := findLandings(ms)
walk := now.Sub(p.landingAt) >= landingEvery
if walk {
p.landingAt = now
}
movers := readMovers(now)
diskW := map[string]float64{}
for _, d := range disks {
diskW[d.Name] = d.WriteBps
}
for _, pool := range poolsWithKstats() {
pw := PoolWrites{Pool: pool}
// A txg records what it wrote only when it has synced — every few seconds, not every
// sample — so the rate is taken over the gap between syncs and held until the next one.
prevTop, had := p.txg[pool]
written, top := parseTxgs(readStr(zfsKstatDir+pool+"/txgs"), prevTop)
switch {
case !had:
p.txgAt[pool] = now
case top > prevTop:
if gap := now.Sub(p.txgAt[pool]).Seconds(); gap > 0 {
p.txgRate[pool] = round1(float64(written) / gap)
}
p.txgAt[pool] = now
case now.Sub(p.txgAt[pool]) > time.Duration(2*max(w.TxgTimeoutSec, 5))*time.Second:
p.txgRate[pool] = 0 // no sync for two timeouts: nothing is being written
}
pw.WriteBps = p.txgRate[pool]
p.txg[pool] = top
lw := objsetWritten(pool)
if prev, ok := p.logical[pool]; ok && dt > 0 && lw >= prev {
pw.LogicalBps = round1(float64(lw-prev) / dt)
}
p.logical[pool] = lw
if ls := landings[pool]; len(ls) > 0 {
l := ls[0] // one landing tier per pool is what the view shows
if last, ok := p.landing[l.Branch]; ok && !walk {
l.Size, l.Used, l.PendingFiles, l.PendingBytes = last.Size, last.Used, last.PendingFiles, last.PendingBytes
l.OldestMs, l.PendingPartial, l.CountedMs = last.OldestMs, last.PendingPartial, last.CountedMs
} else {
walkLanding(&l, now)
p.landing[l.Branch] = l
}
l.InBps = diskW[l.Device]
if mv := movers[filepath.Clean(l.Branch)]; mv != nil {
l.Mover = mv
// the mover writes its status every couple of seconds: measure between its updates
done := mv.MovedBytes + mv.Partial
if at, ok := p.drainedAt[l.Branch]; !ok || mv.UpdatedMs != at {
if ok && mv.UpdatedMs > at && done >= p.drained[l.Branch] {
p.drainRate[l.Branch] = round1(float64(done-p.drained[l.Branch]) / (float64(mv.UpdatedMs-at) / 1000))
}
p.drained[l.Branch], p.drainedAt[l.Branch] = done, mv.UpdatedMs
}
if mv.State != "moving" {
p.drainRate[l.Branch] = 0
}
l.DrainBps = p.drainRate[l.Branch]
}
pw.Landing = &l
}
w.Pools = append(w.Pools, pw)
}
p.at = now
return w
}
+86
View File
@@ -0,0 +1,86 @@
package metrics
import "testing"
// real rows from grid's /proc/spl/kstat/zfs/tank/txgs, 2026-10-25
const txgsSample = `81 1 0x01 26 7072 20029594359 13953814329135
txg birth state ndirty nread nwritten reads writes otime qtime wtime stime
15812833 13936408791659 C 0 0 0 0 0 5119980242 2040 58385 309580
15812834 13941528771901 C 0 0 0 0 0 5119948815 8471 65826 112612
15812835 13946648720716 C 885760 0 2293760 0 367 5120100556 1704 56762 271264374
15812836 13951768821272 S 4194304 0 0 0 0 5120000000 1000 50000 0
15812837 13956888821272 O 0 0 0 0 0 0 0 0 0
`
func TestParseTxgs(t *testing.T) {
w, top := parseTxgs(txgsSample, 0)
if w != 2293760 || top != 15812835 {
t.Fatalf("from 0: written %d top %d", w, top)
}
// only txgs synced since the last look count, and an open or syncing txg never does
w, top = parseTxgs(txgsSample, 15812834)
if w != 2293760 || top != 15812835 {
t.Fatalf("after 15812834: written %d top %d", w, top)
}
w, top = parseTxgs(txgsSample, 15812835)
if w != 0 || top != 15812835 {
t.Fatalf("nothing new: written %d top %d", w, top)
}
}
func TestParseBranches(t *testing.T) {
p, m := parseBranches("/srv/landing/tank=RW:/tank=NC,50G:/mnt/old=RO\n")
if len(p) != 3 || p[0] != "/srv/landing/tank" || p[1] != "/tank" || p[2] != "/mnt/old" {
t.Fatalf("paths %q", p)
}
if m[0] != "RW" || m[1] != "NC" || m[2] != "RO" {
t.Fatalf("modes %q", m)
}
if p, _ := parseBranches("/a:/b"); len(p) != 2 {
t.Fatalf("bare paths %q", p)
}
}
func TestMountFor(t *testing.T) {
ms := []mountEntry{
{"/dev/sdj2", "/", "ext4"},
{"/dev/nvme0n1p1", "/srv/landing", "ext4"},
{"tank", "/tank", "zfs"},
{"tank/media", "/tank/media", "zfs"},
{"tank-landing", "/srv/tank", "fuse.mergerfs"},
}
cases := map[string]string{
"/srv/landing/tank": "/srv/landing", "/tank": "/tank", "/tank/media/x": "/tank/media",
"/tankfoo": "/", "/srv/landingx": "/",
}
for path, want := range cases {
if m, ok := mountFor(ms, path); !ok || m.point != want {
t.Errorf("%s: got %q want %q", path, m.point, want)
}
}
}
func TestUnescapeMount(t *testing.T) {
if got := unescapeMount(`/mnt/my\040disk`); got != "/mnt/my disk" {
t.Fatalf("got %q", got)
}
}
func TestFoldZFSWrites(t *testing.T) {
mk := func(dirty uint64, bps, in, drain float64) Sample {
return Sample{ZFSWrite: &ZFSWrites{Dirty: dirty, Limit: 100, Pools: []PoolWrites{{Pool: "tank", WriteBps: bps,
Landing: &Landing{Branch: "/srv/landing/tank", InBps: in, DrainBps: drain, Mover: &Mover{State: "moving"}}}}}}
}
b := []Sample{mk(10, 100, 400, 0), mk(30, 300, 0, 60)}
got := fold(b).ZFSWrite
if got.Dirty != 20 || got.Pools[0].WriteBps != 200 || got.Pools[0].Landing.InBps != 200 || got.Pools[0].Landing.DrainBps != 30 {
t.Fatalf("folded %+v / %+v", got, got.Pools[0].Landing)
}
// the fold must not write through to the newest sample it copied from
if b[1].ZFSWrite.Pools[0].Landing.InBps != 0 || b[1].ZFSWrite.Dirty != 30 {
t.Fatalf("fold changed its input")
}
if got.Pools[0].Landing.Mover.State != "moving" {
t.Fatalf("mover state lost")
}
}
+564
View File
@@ -0,0 +1,564 @@
package storage
// Verifying a landing tier: did everything that landed really reach the pool, intact?
//
// A landing tier is fast storage in front of a pool (a mergerfs pool, new files on the fast
// branch first) with a mover draining it. The mover keeps a ledger — one JSON line per file it
// moved: the path under the pool, its size and mtime, and the SHA-256 of the fast copy read
// just before the move (hamlet's scripts/landing-mover.py writes /var/lib/landing-mover/
// ledger.jsonl and names it in its status file). A verify pass reads the pool's copy of each
// entry back and compares.
//
// What a pass can say about each file:
// - match: same size, same mtime, same SHA-256 — it arrived intact
// - mismatch: same size and mtime but different contents — the one real failure
// - changed: the pool's copy has a different size or mtime: edited since it moved, so the
// old hash says nothing about it (counted, not compared)
// - gone: not on the pool and not waiting on the landing branch either — deleted or
// renamed since it moved. A pass cannot tell that from a loss, so it lists them
// - waiting: back on the landing branch (written again since), the mover will take it
// - unreadable: could not be opened as this account
// - unhashed: the mover recorded no hash for it (not a regular file, or changed mid-move)
//
// Scope "new" checks what has moved since the last pass, "all" re-reads the whole ledger. A
// path moved more than once is checked against its latest entry only.
import (
"bufio"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"strings"
"syscall"
"time"
)
// ---- schedules ----
// VerifyPolicy runs a verify pass on a schedule. The when-fields mean what they do on a
// snapshot Policy; Scope is "new" or "all".
type VerifyPolicy struct {
ID string `json:"id"`
Name string `json:"name"`
Every string `json:"every"`
At string `json:"at"`
Weekday int `json:"weekday"`
Day int `json:"day"`
Cron string `json:"cron,omitempty"`
Scope string `json:"scope"`
Enabled bool `json:"enabled"`
}
// when borrows the snapshot Policy's schedule logic, so both kinds of schedule are written,
// validated and read back by exactly the same code.
func (v *VerifyPolicy) when() *Policy {
return &Policy{Every: v.Every, At: v.At, Weekday: v.Weekday, Day: v.Day, Cron: v.Cron}
}
func (v *VerifyPolicy) CronExpr() (string, error) { return v.when().CronExpr() }
func (v *VerifyPolicy) NextRun(now time.Time) time.Time { return v.when().NextRun(now) }
func (v *VerifyPolicy) Validate() error {
if v.Scope != "new" && v.Scope != "all" {
return errors.New("scope: new (what moved since the last pass) or all (the whole ledger)")
}
switch v.Every {
case "hourly", "daily", "weekly", "monthly":
case "custom":
if strings.TrimSpace(v.Cron) == "" {
return errors.New("a custom schedule needs a cron expression")
}
default:
return errors.New("every: hourly, daily, weekly, monthly or custom")
}
if _, err := v.CronExpr(); err != nil {
return err
}
if _, _, err := v.when().clock(); err != nil {
return err
}
if v.Every == "weekly" && (v.Weekday < 0 || v.Weekday > 6) {
return errors.New("weekday: 0 (Sunday) to 6")
}
if v.Every == "monthly" && (v.Day < 1 || v.Day > 28) {
return errors.New("day: 1 to 28, so every month has one")
}
if len(v.Name) > 64 {
return errors.New("name: 64 characters at most")
}
return nil
}
// VerifyCronFile holds the verify schedules; Ham Lab owns it and rewrites it whole.
const VerifyCronFile = "/etc/cron.d/hamlab-landing"
// VerifyCronText renders the verify schedules as cron lines, run as Ham Lab's own account.
func VerifyCronText(ps []VerifyPolicy, user, exe, config string) (string, error) {
var b strings.Builder
b.WriteString("# Ham Lab landing-tier verify schedules — written by the ZFS tab, rewritten whole on every change.\n")
b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
b.WriteString("SHELL=/bin/sh\n")
b.WriteString("PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
for i := range ps {
p := &ps[i]
name := strings.ReplaceAll(p.Name, "\n", " ")
if !p.Enabled {
b.WriteString("# " + name + " — paused\n")
continue
}
expr, err := p.CronExpr()
if err != nil {
return "", fmt.Errorf("%s: %w", p.Name, err)
}
fmt.Fprintf(&b, "# %s — verify %s\n", name, p.Scope)
fmt.Fprintf(&b, "%s %s %s -config %s landing-verify %s\n\n", expr, user, exe, config, p.ID)
}
return b.String(), nil
}
// WriteVerifyCron puts the verify schedules on the machine, or removes the file when none run.
func (m *Manager) WriteVerifyCron(ctx context.Context, ps []VerifyPolicy, user, exe, config string) error {
live := false
for i := range ps {
live = live || ps[i].Enabled
}
if !live {
return m.RemoveRootFile(ctx, VerifyCronFile)
}
text, err := VerifyCronText(ps, user, exe, config)
if err != nil {
return err
}
return m.writeRoot(ctx, VerifyCronFile, text)
}
// ---- finding the landing tier ----
// MoverStatusDir is where a landing mover writes what it is doing.
const MoverStatusDir = "/run/landing-mover"
// DefaultLedger is where hamlet's mover keeps its ledger when no status file names one.
const DefaultLedger = "/var/lib/landing-mover/ledger.jsonl"
// LandingTarget is one landing tier a pass can check: the branch files land on, the pool
// directory they are moved to, and the mover's ledger.
type LandingTarget struct {
Src string `json:"src"`
Dst string `json:"dst"`
Ledger string `json:"ledger"`
}
// LandingTargets reads the movers' status files. Each mover names its own source, target and
// ledger; with none running, nothing can be verified, and the page says so.
func LandingTargets() []LandingTarget {
var out []LandingTarget
m, _ := filepath.Glob(MoverStatusDir + "/*.json")
for _, p := range m {
b, err := os.ReadFile(p)
if err != nil {
continue
}
var st struct{ Src, Dst, Ledger string }
if json.Unmarshal(b, &st) != nil || st.Src == "" || st.Dst == "" {
continue
}
if st.Ledger == "" {
st.Ledger = DefaultLedger
}
out = append(out, LandingTarget{Src: filepath.Clean(st.Src), Dst: filepath.Clean(st.Dst), Ledger: st.Ledger})
}
sort.Slice(out, func(i, j int) bool { return out[i].Src < out[j].Src })
return out
}
// ---- stale files ----
// StaleAfter is how long a file may wait on a landing branch before it counts as stale. A file
// older than this is not necessarily lost — a big backlog drains for hours — but it is worth a
// look, which is what the ZFS tab offers.
const StaleAfter = time.Hour
// StaleFile is one file that has waited longer than StaleAfter.
type StaleFile struct {
Path string `json:"path"` // relative to the landing branch, which is also its path on the pool
Size int64 `json:"size"`
MTime int64 `json:"mtime"` // unix ms
}
// StaleFiles lists what has waited on a landing branch longer than StaleAfter, oldest first,
// at most limit of them, with the full count and size.
func StaleFiles(branch string, limit int, now time.Time) (files []StaleFile, count int, bytes int64) {
cut := now.Add(-StaleAfter)
_ = filepath.WalkDir(branch, func(p string, d os.DirEntry, err error) error {
if err != nil || !d.Type().IsRegular() {
return nil
}
fi, err := d.Info()
if err != nil || !fi.ModTime().Before(cut) {
return nil
}
count++
bytes += fi.Size()
rel, _ := filepath.Rel(branch, p)
files = append(files, StaleFile{Path: rel, Size: fi.Size(), MTime: fi.ModTime().UnixMilli()})
return nil
})
sort.Slice(files, func(i, j int) bool { return files[i].MTime < files[j].MTime })
if len(files) > limit {
files = files[:limit]
}
return files, count, bytes
}
// ---- the pass ----
// LedgerEntry is one line of the mover's ledger.
type LedgerEntry struct {
T int64 `json:"t"` // when it was moved, unix ms
Path string `json:"path"`
Size int64 `json:"size"`
MTimeNs int64 `json:"mtime_ns"`
SHA256 string `json:"sha256,omitempty"`
Note string `json:"note,omitempty"`
}
// VerifyIssue is one file a pass wants a person to look at.
type VerifyIssue struct {
Path string `json:"path"`
Kind string `json:"kind"` // mismatch | gone | unreadable
Moved int64 `json:"moved"`
Size int64 `json:"size"`
Note string `json:"note,omitempty"`
}
// VerifyResult is what one pass found.
type VerifyResult struct {
ID string `json:"id"` // the schedule that ran it, or "manual"
Scope string `json:"scope"`
Started time.Time `json:"started"`
Finished time.Time `json:"finished"`
Src string `json:"src"`
Dst string `json:"dst"`
Since int64 `json:"since"` // ledger entries after this (unix ms); 0 for all
Entries int `json:"entries"`
Checked int `json:"checked"`
Matched int `json:"matched"`
Mismatched int `json:"mismatched"`
Changed int `json:"changed"`
Gone int `json:"gone"`
Waiting int `json:"waiting"`
Unreadable int `json:"unreadable"`
Unhashed int `json:"unhashed"`
BytesRead int64 `json:"bytes_read"`
Issues []VerifyIssue `json:"issues"`
Error string `json:"error,omitempty"`
Cursor int64 `json:"cursor"` // the newest entry it covered
}
// OK is true when nothing needs a look: no mismatches, nothing gone or unreadable, no error.
func (r *VerifyResult) OK() bool {
return r.Error == "" && r.Mismatched == 0 && r.Gone == 0 && r.Unreadable == 0
}
// VerifyProgress is published while a pass runs.
type VerifyProgress struct {
ID string `json:"id"`
Scope string `json:"scope"`
PID int `json:"pid"`
Started time.Time `json:"started"`
Updated time.Time `json:"updated"`
Total int `json:"total"`
Done int `json:"done"`
BytesRead int64 `json:"bytes_read"`
BytesAll int64 `json:"bytes_all"`
Current string `json:"current"`
}
const maxIssues = 500
// readLedger loads the ledger's entries newer than since, the latest per path, oldest first.
func readLedger(path string, since int64) ([]LedgerEntry, error) {
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()
latest := map[string]LedgerEntry{}
sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 64<<10), 4<<20)
for sc.Scan() {
var e LedgerEntry
if json.Unmarshal(sc.Bytes(), &e) != nil || e.Path == "" || e.T <= since {
continue
}
if old, ok := latest[e.Path]; !ok || e.T >= old.T {
latest[e.Path] = e
}
}
if err := sc.Err(); err != nil {
return nil, err
}
out := make([]LedgerEntry, 0, len(latest))
for _, e := range latest {
out = append(out, e)
}
sort.Slice(out, func(i, j int) bool { return out[i].T < out[j].T })
return out, nil
}
// inside joins rel under root and refuses anything that would climb out of it.
func inside(root, rel string) (string, bool) {
p := filepath.Join(root, rel)
return p, p == root || strings.HasPrefix(p, root+string(filepath.Separator))
}
func hashFile(p string, buf []byte, onRead func(int)) (string, error) {
f, err := os.Open(p)
if err != nil {
return "", err
}
defer f.Close()
h := sha256.New()
for {
n, err := f.Read(buf)
if n > 0 {
h.Write(buf[:n])
onRead(n)
}
if err == io.EOF {
return hex.EncodeToString(h.Sum(nil)), nil
}
if err != nil {
return "", err
}
}
}
// Verify runs one pass over a landing tier's ledger. progress is called about once a second.
func Verify(t LandingTarget, id, scope string, since int64, progress func(VerifyProgress)) VerifyResult {
r := VerifyResult{ID: id, Scope: scope, Started: time.Now(), Src: t.Src, Dst: t.Dst, Since: since, Issues: []VerifyIssue{}, Cursor: since}
if scope == "all" {
r.Since = 0
}
entries, err := readLedger(t.Ledger, r.Since)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
r.Error = "no ledger yet at " + t.Ledger + " — nothing has been moved since the mover started keeping one"
} else {
r.Error = "reading the ledger: " + err.Error()
}
r.Finished = time.Now()
return r
}
r.Entries = len(entries)
pr := VerifyProgress{ID: id, Scope: scope, PID: os.Getpid(), Started: r.Started, Total: len(entries)}
for _, e := range entries {
pr.BytesAll += e.Size
}
last := time.Time{}
tick := func(force bool) {
if progress != nil && (force || time.Since(last) >= time.Second) {
pr.Updated, last = time.Now(), time.Now()
progress(pr)
}
}
issue := func(e LedgerEntry, kind, note string) {
if len(r.Issues) < maxIssues {
r.Issues = append(r.Issues, VerifyIssue{Path: e.Path, Kind: kind, Moved: e.T, Size: e.Size, Note: note})
}
}
buf := make([]byte, 4<<20)
for _, e := range entries {
pr.Current = e.Path
tick(false)
if e.T > r.Cursor {
r.Cursor = e.T
}
r.Checked++
pr.Done++
dst, ok1 := inside(t.Dst, e.Path)
src, ok2 := inside(t.Src, e.Path)
if !ok1 || !ok2 {
r.Unreadable++
issue(e, "unreadable", "path climbs out of the pool")
continue
}
st, err := os.Lstat(dst)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
if _, err := os.Lstat(src); err == nil {
r.Waiting++
continue
}
r.Gone++
issue(e, "gone", "not on the pool, not waiting to move: deleted or renamed since it moved?")
continue
}
r.Unreadable++
issue(e, "unreadable", err.Error())
continue
}
if e.SHA256 == "" {
r.Unhashed++
continue
}
mt := st.ModTime().UnixNano()
if st.Size() != e.Size || abs64(mt-e.MTimeNs) >= int64(time.Second) {
r.Changed++
continue
}
sum, err := hashFile(dst, buf, func(n int) { r.BytesRead += int64(n); pr.BytesRead += int64(n); tick(false) })
if err != nil {
if errors.Is(err, syscall.EACCES) || errors.Is(err, os.ErrPermission) {
r.Unreadable++
issue(e, "unreadable", "permission denied")
} else {
r.Unreadable++
issue(e, "unreadable", err.Error())
}
continue
}
if sum != e.SHA256 {
r.Mismatched++
issue(e, "mismatch", "contents differ from the copy that landed")
continue
}
r.Matched++
}
tick(true)
r.Finished = time.Now()
return r
}
func abs64(v int64) int64 {
if v < 0 {
return -v
}
return v
}
// ---- records ----
func verifyDir(dataDir string) string { return filepath.Join(dataDir, "landing-verify") }
func writeJSON(p string, v any) error {
if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
return err
}
b, err := json.Marshal(v)
if err != nil {
return err
}
tmp := p + ".tmp"
if err := os.WriteFile(tmp, b, 0o600); err != nil {
return err
}
return os.Rename(tmp, p)
}
func readJSON(p string, v any) bool {
b, err := os.ReadFile(p)
return err == nil && json.Unmarshal(b, v) == nil
}
// cursorKey names a landing tier's cursor file.
func cursorKey(t LandingTarget) string {
return strings.Trim(strings.NewReplacer("/", "_").Replace(t.Src), "_")
}
// VerifyCursor is the newest ledger entry a finished pass covered, for scope "new".
func VerifyCursor(dataDir string, t LandingTarget) int64 {
var c struct{ Cursor int64 }
readJSON(filepath.Join(verifyDir(dataDir), "cursor-"+cursorKey(t)+".json"), &c)
return c.Cursor
}
// RecordVerify keeps a finished pass: as the run's last result, in the history (newest first,
// 30 kept), and — when it read the ledger at all — as the new cursor.
func RecordVerify(dataDir string, t LandingTarget, r VerifyResult) error {
d := verifyDir(dataDir)
if err := writeJSON(filepath.Join(d, "run-"+r.ID+".json"), r); err != nil {
return err
}
var hist []VerifyResult
readJSON(filepath.Join(d, "history.json"), &hist)
hist = append([]VerifyResult{r}, hist...)
if len(hist) > 30 {
hist = hist[:30]
}
if err := writeJSON(filepath.Join(d, "history.json"), hist); err != nil {
return err
}
if r.Error == "" && r.Cursor > VerifyCursor(dataDir, t) {
return writeJSON(filepath.Join(d, "cursor-"+cursorKey(t)+".json"), map[string]int64{"cursor": r.Cursor})
}
return nil
}
// VerifyHistory is the kept passes, newest first.
func VerifyHistory(dataDir string) []VerifyResult {
var hist []VerifyResult
readJSON(filepath.Join(verifyDir(dataDir), "history.json"), &hist)
if hist == nil {
hist = []VerifyResult{}
}
return hist
}
// LastVerify is the last pass a schedule (or "manual") ran.
func LastVerify(dataDir, id string) *VerifyResult {
var r VerifyResult
if !readJSON(filepath.Join(verifyDir(dataDir), "run-"+id+".json"), &r) {
return nil
}
return &r
}
// ForgetVerify drops a removed schedule's last result (the history keeps it).
func ForgetVerify(dataDir, id string) {
_ = os.Remove(filepath.Join(verifyDir(dataDir), "run-"+id+".json"))
}
func progressPath(dataDir string) string { return filepath.Join(verifyDir(dataDir), "running.json") }
// WriteVerifyProgress publishes a running pass.
func WriteVerifyProgress(dataDir string, p VerifyProgress) { _ = writeJSON(progressPath(dataDir), p) }
// ClearVerifyProgress removes it once the pass ends.
func ClearVerifyProgress(dataDir string) { _ = os.Remove(progressPath(dataDir)) }
// RunningVerify is the pass in progress, or nil. A progress file whose process is gone, or
// that has not moved for two minutes, is a pass that died: it is reported as not running.
func RunningVerify(dataDir string) *VerifyProgress {
var p VerifyProgress
if !readJSON(progressPath(dataDir), &p) {
return nil
}
if p.PID <= 0 || syscall.Kill(p.PID, 0) != nil || time.Since(p.Updated) > 2*time.Minute {
return nil
}
return &p
}
// LockVerify takes the one-pass-at-a-time lock; release it with the returned func.
func LockVerify(dataDir string) (func(), error) {
if err := os.MkdirAll(verifyDir(dataDir), 0o700); err != nil {
return nil, err
}
f, err := os.OpenFile(filepath.Join(verifyDir(dataDir), "lock"), os.O_CREATE|os.O_RDWR, 0o600)
if err != nil {
return nil, err
}
if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
f.Close()
return nil, errors.New("a verify pass is already running")
}
return func() { _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN); f.Close() }, nil
}
+137
View File
@@ -0,0 +1,137 @@
package storage
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestVerifyClassifiesEveryCase(t *testing.T) {
root := t.TempDir()
src, dst := filepath.Join(root, "landing"), filepath.Join(root, "pool")
for _, d := range []string{src, dst + "/a", src + "/a"} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
mtime := time.Date(2026, 10, 25, 12, 0, 0, 0, time.UTC)
put := func(p, body string) LedgerEntry {
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
_ = os.Chtimes(p, mtime, mtime)
sum := sha256.Sum256([]byte(body))
return LedgerEntry{Size: int64(len(body)), MTimeNs: mtime.UnixNano(), SHA256: hex.EncodeToString(sum[:])}
}
var ledger []LedgerEntry
add := func(rel string, e LedgerEntry, at int64) { e.Path, e.T = rel, at; ledger = append(ledger, e) }
add("a/match.bin", put(dst+"/a/match.bin", "intact"), 10)
e := put(dst+"/a/mismatch.bin", "garbled") // same size and mtime as what was hashed, other bytes
sum := sha256.Sum256([]byte("GARBLED"))
e.SHA256 = hex.EncodeToString(sum[:])
add("a/mismatch.bin", e, 11)
e = put(dst+"/a/changed.bin", "edited since, longer now")
e.Size = 3
add("a/changed.bin", e, 12)
add("a/gone.bin", LedgerEntry{Size: 4, MTimeNs: mtime.UnixNano(), SHA256: "00"}, 13)
put(src+"/a/waiting.bin", "back again")
add("a/waiting.bin", LedgerEntry{Size: 10, MTimeNs: mtime.UnixNano(), SHA256: "00"}, 14)
e = put(dst+"/a/unhashed", "x")
e.SHA256, e.Note = "", "not a regular file"
add("a/unhashed", e, 15)
add("../escape", LedgerEntry{Size: 1, SHA256: "00"}, 16)
// moved twice: only the newer entry counts, and it matches
add("a/twice.bin", LedgerEntry{Size: 99, SHA256: "stale"}, 5)
add("a/twice.bin", put(dst+"/a/twice.bin", "second time"), 17)
lp := filepath.Join(root, "ledger.jsonl")
f, _ := os.Create(lp)
for _, e := range ledger {
b, _ := json.Marshal(e)
f.Write(append(b, '\n'))
}
f.WriteString("not json\n")
f.Close()
tg := LandingTarget{Src: src, Dst: dst, Ledger: lp}
r := Verify(tg, "manual", "all", 0, nil)
want := map[string]int{"checked": 8, "matched": 2, "mismatched": 1, "changed": 1, "gone": 1, "waiting": 1, "unhashed": 1, "unreadable": 1}
got := map[string]int{"checked": r.Checked, "matched": r.Matched, "mismatched": r.Mismatched, "changed": r.Changed,
"gone": r.Gone, "waiting": r.Waiting, "unhashed": r.Unhashed, "unreadable": r.Unreadable}
for k, v := range want {
if got[k] != v {
t.Errorf("%s: got %d want %d (%+v)", k, got[k], v, r)
}
}
if r.OK() || r.Cursor != 17 || r.Error != "" {
t.Errorf("ok=%v cursor=%d err=%q", r.OK(), r.Cursor, r.Error)
}
kinds := map[string]string{}
for _, i := range r.Issues {
kinds[i.Path] = i.Kind
}
if kinds["a/mismatch.bin"] != "mismatch" || kinds["a/gone.bin"] != "gone" || kinds["../escape"] != "unreadable" {
t.Errorf("issues %+v", r.Issues)
}
// "new" after a pass at cursor 14 only looks at what moved later
r = Verify(tg, "manual", "new", 14, nil)
if r.Checked != 3 || r.Matched != 1 || r.Unhashed != 1 || r.Unreadable != 1 {
t.Errorf("new since 14: %+v", r)
}
// records: the cursor only advances on a pass that read the ledger
dir := t.TempDir()
if err := RecordVerify(dir, tg, Verify(tg, "manual", "all", 0, nil)); err != nil {
t.Fatal(err)
}
if c := VerifyCursor(dir, tg); c != 17 {
t.Errorf("cursor %d", c)
}
bad := Verify(LandingTarget{Src: src, Dst: dst, Ledger: filepath.Join(root, "missing")}, "manual", "new", 0, nil)
if !strings.Contains(bad.Error, "no ledger yet") {
t.Errorf("missing ledger: %q", bad.Error)
}
if len(VerifyHistory(dir)) != 1 || LastVerify(dir, "manual") == nil {
t.Errorf("history/last not kept")
}
}
func TestVerifyPolicy(t *testing.T) {
p := VerifyPolicy{ID: "abc", Name: "nightly", Every: "daily", At: "05:10", Scope: "new", Enabled: true}
if err := p.Validate(); err != nil {
t.Fatal(err)
}
if x, _ := p.CronExpr(); x != "10 5 * * *" {
t.Fatalf("cron %q", x)
}
txt, err := VerifyCronText([]VerifyPolicy{p, {ID: "off", Name: "paused one", Every: "hourly", Scope: "all"}}, "core", "/usr/local/bin/hamlab", "/cfg.json")
if err != nil || !strings.Contains(txt, "10 5 * * * core /usr/local/bin/hamlab -config /cfg.json landing-verify abc") || !strings.Contains(txt, "# paused one — paused") {
t.Fatalf("%v\n%s", err, txt)
}
for _, b := range []VerifyPolicy{{Every: "daily", Scope: "some"}, {Every: "sometimes", Scope: "new"}, {Every: "custom", Scope: "new"}} {
if b.Validate() == nil {
t.Errorf("accepted %+v", b)
}
}
}
func TestStaleFiles(t *testing.T) {
d := t.TempDir()
now := time.Now()
os.MkdirAll(d+"/x", 0o755)
for name, age := range map[string]time.Duration{"x/old.bin": 3 * time.Hour, "x/older.bin": 5 * time.Hour, "fresh.bin": time.Minute} {
os.WriteFile(filepath.Join(d, name), []byte("12345"), 0o644)
os.Chtimes(filepath.Join(d, name), now.Add(-age), now.Add(-age))
}
files, n, b := StaleFiles(d, 1, now)
if n != 2 || b != 10 || len(files) != 1 || files[0].Path != "x/older.bin" {
t.Fatalf("%d %d %+v", n, b, files)
}
}
+3 -7
View File
@@ -68,13 +68,9 @@ type State struct {
// Mounts remembers where a container's bind mounts pointed, so a stack deployed by an
// orchestrator that has since been removed can still be found on disk. See
// internal/api/stacks.go.
Mounts []RememberedMount `json:"mounts,omitempty"`
// KVMCookies is each KVM's own login, as the cookies its page set in a browser: kept so
// the next browser — or this one after it has forgotten them — is signed into the device
// already. See internal/api/kvm.go.
KVMCookies map[string]map[string]string `json:"kvm_cookies,omitempty"`
SamplerMs int `json:"sampler_ms,omitempty"` // how often this machine samples; the hub keeps every machine on the console's refresh rate
HistorySec int `json:"history_sec,omitempty"` // how far back the graphs reach; the hub keeps every machine on the same one
Mounts []RememberedMount `json:"mounts,omitempty"`
SamplerMs int `json:"sampler_ms,omitempty"` // how often this machine samples; the hub keeps every machine on the console's refresh rate
HistorySec int `json:"history_sec,omitempty"` // how far back the graphs reach; the hub keeps every machine on the same one
// Transcode is where videos played in the Files viewer are made (internal/api/play.go):
// a machine and a device on it. nil is the machine the file is on, its best device.
Transcode *Transcode `json:"transcode,omitempty"`
+1 -1
View File
@@ -7,7 +7,7 @@
"dev": "vite",
"build": "tsc --noEmit && vite build",
"icons": "node scripts/icons.mjs",
"test": "node scripts/test-graphcols.mjs && node scripts/test-css-imports.mjs && node scripts/test-resolve-path.mjs"
"test": "node scripts/test-graphcols.mjs && node scripts/test-sockets.mjs && node scripts/test-css-imports.mjs && node scripts/test-resolve-path.mjs"
},
"dependencies": {
"@xterm/addon-fit": "^0.10.0",
+57
View File
@@ -67,6 +67,63 @@ if (await page.locator('canvas.vp-sphere').count()) {
}
await page.screenshot({ path: `${SHOTS}/h1-auto.png` })
// No theme touches playback: for every theme the stylesheet has rules for, no full-window
// overlay is drawn, the picture isn't filtered, and the player's lettering is plain.
{
const was = await page.evaluate(() => document.documentElement.getAttribute('data-theme'))
const themes = await page.evaluate(() => {
const ids = new Set()
for (const sh of document.styleSheets) { let rules; try { rules = sh.cssRules } catch { continue }
const walk = (rs) => { for (const r of rs) { if (r.cssRules) walk(r.cssRules); for (const m of (r.selectorText ?? '').matchAll(/data-theme="([^"]+)"/g)) ids.add(m[1]) } }
walk(rules) }
return [...ids]
})
const bad = []
for (const t of themes) {
const r = await page.evaluate((t) => {
document.documentElement.setAttribute('data-theme', t)
const els = [document.documentElement, document.body, document.querySelector('.app'), document.querySelector('#root')].filter(Boolean)
const overlays = els.flatMap((e) => ['::before', '::after'].map((p) => getComputedStyle(e, p))).filter((c) => c.display !== 'none' && c.content !== 'none' && c.content !== 'normal' && (c.position === 'fixed' || c.position === 'absolute')).length
const pics = [...document.querySelectorAll('.vp video, .vp canvas')].map((e) => getComputedStyle(e).filter).filter((f) => f !== 'none').length
const c = getComputedStyle(document.querySelector('.vp'))
const b = getComputedStyle(document.querySelector('.vp button'))
return { overlays, pics, plain: !/mono|courier/i.test(b.fontFamily) && c.fontWeight === '400' && c.letterSpacing === 'normal' && c.textShadow === 'none' }
}, t)
if (r.overlays || r.pics || !r.plain) bad.push(`${t}: ${JSON.stringify(r)}`)
}
check(`no theme touches the video (${themes.length} themes with their own rules)`, themes.length > 0 && bad.length === 0, bad.join(' / ') || themes.join(', '))
await page.evaluate((t) => t ? document.documentElement.setAttribute('data-theme', t) : document.documentElement.removeAttribute('data-theme'), was)
}
// Volume: the speaker mutes and unmutes, the slider sets the level, and it is remembered.
{
await controls()
const vs = () => page.evaluate(() => { const v = document.querySelector('.vp video'); return { muted: v.muted, volume: Math.round(v.volume * 100) / 100 } })
await page.getByRole('button', { name: 'Mute' }).click()
const a = await vs()
await page.getByRole('button', { name: 'Unmute' }).click()
const b = await vs()
let c = b
if (await page.getByLabel('Volume').isVisible()) { await page.getByLabel('Volume').fill('0.3'); c = await vs() }
const kept = await page.evaluate(() => Object.entries(localStorage).find(([k]) => /volume/.test(k))?.[1] ?? '')
check('volume: mute, unmute, level, remembered', a.muted && !b.muted && (c.volume === 0.3 || !(await page.getByLabel('Volume').isVisible())) && /level/.test(kept),
`${JSON.stringify(a)} → ${JSON.stringify(b)} → ${JSON.stringify(c)}, kept ${kept}`)
if (await page.getByLabel('Volume').isVisible()) await page.getByLabel('Volume').fill('1')
}
// The 5-second buttons: back and on, the ball with them.
{
await controls()
const t0 = (await vstate()).t
await page.getByRole('button', { name: 'On 5 seconds' }).click()
const t1 = (await vstate()).t
await page.getByRole('button', { name: 'Back 5 seconds' }).click()
await page.getByRole('button', { name: 'Back 5 seconds' }).click()
const t2 = (await vstate()).t
const ball = await page.evaluate(() => Number(document.querySelector('.vp input[type=range]').value))
check('+5 and −5 buttons', t1 - t0 > 4 && t1 - t0 < 6.5 && t1 - t2 > 9 && t1 - t2 < 11 && Math.abs(ball - t2) < 1, `${t0.toFixed(1)} → ${t1.toFixed(1)} → ${t2.toFixed(1)}, ball ${ball.toFixed(1)}`)
}
const mid = Math.floor(s.d / 2)
mark('seek')
await page.evaluate((m) => { document.querySelector('.vp video').currentTime = m }, mid)
+34
View File
@@ -0,0 +1,34 @@
// node scripts/test-sockets.mjs — src/sockets.ts: each CPU's threads, counted under it.
import { buildSync } from 'esbuild'
import { pathToFileURL } from 'node:url'
import { mkdtempSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
const out = join(mkdtempSync(join(tmpdir(), 'sockets-')), 'sockets.mjs')
buildSync({ entryPoints: [new URL('../src/sockets.ts', import.meta.url).pathname], outfile: out, format: 'esm', logLevel: 'error' })
const { socketThreads } = await import(pathToFileURL(out).href)
const fails = []
const eq = (got, want, what) => { if (JSON.stringify(got) !== JSON.stringify(want)) fails.push(`${what} — got ${JSON.stringify(got)}, want ${JSON.stringify(want)}`) }
const range = (a, b) => Array.from({ length: b - a }, (_, i) => a + i)
// nexus: 2 × E5-2680 v2, as /sys/devices/system/cpu/cpu*/topology/physical_package_id reads
const nexus = [...Array(10).fill(0), ...Array(10).fill(1), ...Array(10).fill(0), ...Array(10).fill(1)]
const got = socketThreads(nexus)
eq(got.map((s) => s.pkg), [0, 1], 'two sockets, in order')
eq(got[0].cpus, [...range(0, 10), ...range(20, 30)], "CPU 1's threads")
eq(got[1].cpus, [...range(10, 20), ...range(30, 40)], "CPU 2's threads")
eq(got.map((s) => s.cpus.length), [20, 20], 'twenty threads each')
// every thread is under exactly one CPU
eq(got.flatMap((s) => s.cpus).sort((a, b) => a - b), range(0, 40), 'every thread once')
// a thread's number under its CPU runs 0…19 on both
for (const s of got) eq(s.cpus.map((_, n) => n), range(0, 20), `CPU ${s.pkg + 1} counts from 0`)
// a board that numbers each socket's threads together, and one whose socket ids are not 0 and 1
eq(socketThreads([0, 0, 0, 0, 1, 1, 1, 1]), [{ pkg: 0, cpus: [0, 1, 2, 3] }, { pkg: 1, cpus: [4, 5, 6, 7] }], 'contiguous numbering')
eq(socketThreads([3, 1, 3, 1]), [{ pkg: 1, cpus: [1, 3] }, { pkg: 3, cpus: [0, 2] }], 'socket ids sorted')
eq(socketThreads([]), [], 'nothing')
if (fails.length) { console.error('sockets: FAIL\n ' + fails.join('\n ')); process.exit(1) }
console.log('sockets: ok')
+58 -6
View File
@@ -1,4 +1,4 @@
import { DragEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
import { DragEvent, PointerEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
import { api, FileEntry, FileInfo, TaskInfo, ThumbStatus } from './api'
import { fmtBytes } from './fmt'
import { Btn, Loading, Spinner, askText, askYes, setUnsaved, toast, useEscape } from './ui'
@@ -22,6 +22,7 @@ export function resolvePath(input: string, here: string): string {
}
const fmtTime = (ms: number) => { const d = new Date(ms); const now = new Date(); return d.toDateString() === now.toDateString() ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) : d.toLocaleDateString(undefined, { year: d.getFullYear() === now.getFullYear() ? undefined : 'numeric', month: 'short', day: '2-digit' }) }
const isImage = (n: string) => /\.(png|jpe?g|gif|webp|svg|bmp|ico|avif)$/i.test(n)
const isMedia = (e: FileEntry) => !!e.preview || isImage(e.name)
const shellQuote = (s: string) => "'" + s.replace(/'/g, "'\\''") + "'"
const MAX_EDIT = 2 * 1024 * 1024
@@ -198,7 +199,9 @@ function ThumbNote({ st, sudo, onChanged }: { st: ThumbStatus | null; sudo: bool
}
// ---- editor / viewer ----
function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolean; onClose: () => void; onChanged: () => void }) {
// steps: the files ← and → go through, in the list's order (only the pictures, videos and
// sound when one of those is open).
function Viewer({ path, sudo, steps, onStep, onClose, onChanged }: { path: string; sudo: boolean; steps: string[]; onStep: (p: string) => void; onClose: () => void; onChanged: () => void }) {
const [info, setInfo] = useState<FileInfo | null>(null)
const [text, setText] = useState<string | null>(null)
const [orig, setOrig] = useState('')
@@ -251,10 +254,49 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
}, [dirty, name])
const dl = api.fileDownloadURL(path, sudo)
// Previous and next: the buttons, ← and → (unless typing, or the player has the focus and
// seeks with them), and a sideways swipe on a picture.
const at = steps.indexOf(path)
const prev = at > 0 ? steps[at - 1] : null
const next = at >= 0 && at < steps.length - 1 ? steps[at + 1] : null
const step = useCallback(async (to: string | null) => {
if (!to) return
if (dirty && !await askYes('Discard unsaved changes?', 'Discard')) return
setUnsaved(''); onStep(to)
}, [dirty, onStep])
useEffect(() => {
const h = (e: KeyboardEvent) => {
if (e.defaultPrevented || e.altKey || e.ctrlKey || e.metaKey || e.shiftKey) return
const t = e.target as HTMLElement | null
if (t?.closest('input, textarea, select, [contenteditable], .vp')) return
if (e.key === 'ArrowLeft' && prev) { e.preventDefault(); step(prev) }
else if (e.key === 'ArrowRight' && next) { e.preventDefault(); step(next) }
}
window.addEventListener('keydown', h)
return () => window.removeEventListener('keydown', h)
}, [prev, next, step])
// The next and previous pictures load now, so stepping onto them is instant.
useEffect(() => {
for (const p of [prev, next]) if (p && isImage(p)) new Image().src = api.fileReadURL(p, sudo)
}, [prev, next, sudo])
const swipe = useRef<{ x: number; y: number } | null>(null)
const swipeEnd = (e: PointerEvent) => {
const s = swipe.current
swipe.current = null
if (!s) return
const dx = e.clientX - s.x, dy = e.clientY - s.y
if (Math.abs(dx) > 50 && Math.abs(dx) > 1.5 * Math.abs(dy)) step(dx < 0 ? next : prev)
}
return (
<div className="viewer">
<div className="viewer-bar">
<button className="btn" onClick={close}>‹ Back</button>
{at >= 0 && steps.length > 1 && <>
<button className="btn step" disabled={!prev} title="previous (←)" onClick={() => step(prev)}>←</button>
<span className="muted num">{at + 1}/{steps.length}</span>
<button className="btn step" disabled={!next} title="next (→)" onClick={() => step(next)}>→</button>
</>}
<span className="vname mono" title={path}>{name}{dirty && <span className="dirty"> •</span>}</span>
<span className="spacer" />
{info && <span className="muted num">{fmtBytes(info.size)} · {info.perm} · {info.user}</span>}
@@ -266,11 +308,17 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
{info && text !== null && (
<textarea className="editor file-editor" value={text} spellCheck={false} onChange={(e) => setText(e.target.value)} />
)}
{info && text === null && isImage(name) && <div className="preview"><img src={api.fileReadURL(path, sudo)} alt={name} /></div>}
{info && text === null && isImage(name) && (
<div className="preview pic" onPointerDown={(e) => { if (e.pointerType !== 'mouse' && e.isPrimary) swipe.current = { x: e.clientX, y: e.clientY } }} onPointerUp={swipeEnd} onPointerCancel={() => { swipe.current = null }}>
<img src={api.fileReadURL(path, sudo)} alt={name} draggable={false} />
{prev && <button className="step-edge prev" title="previous (←)" onClick={() => step(prev)}>‹</button>}
{next && <button className="step-edge next" title="next (→)" onClick={() => step(next)}>›</button>}
</div>
)}
{/* Video: the player (Auto, by hand, or the file as it is), made on the Transcode device. Audio as it is. */}
{info && text === null && !isImage(name) && info.preview === 'video' && <Player path={path} sudo={sudo} name={name} />}
{info && text === null && !isImage(name) && info.preview === 'video' && <Player key={path} path={path} sudo={sudo} name={name} />}
{info && text === null && !isImage(name) && info.preview === 'audio' && !cantPlay && (
<div className="preview audio"><div className="audio-box"><div className="ficon">🎵</div><div className="mono small">{name}</div><audio controls preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => setCantPlay(true)} /></div></div>
<div className="preview audio"><div className="audio-box"><div className="ficon">🎵</div><div className="mono small">{name}</div><audio controls autoPlay preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => setCantPlay(true)} /></div></div>
)}
{info && text === null && !isImage(name) && info.preview === 'audio' && cantPlay && (
<div className="notice">This browser can't play {name} as it is — the codec isn't one it decodes. Download it instead.</div>
@@ -399,7 +447,11 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
if (files.length) uploadFiles(files)
}
if (open) return <div className="files"><Viewer path={open} sudo={sudo} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
if (open) {
const cur = shown.find((e) => join(path, e.name) === open)
const steps = shown.filter((e) => e.type !== 'dir' && e.type !== 'other' && (!cur || !isMedia(cur) || isMedia(e))).map((e) => join(path, e.name))
return <div className="files"><Viewer path={open} sudo={sudo} steps={steps} onStep={setOpen} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
}
return (
<div className={'files' + (drag ? ' drag' : '')} onDragOver={(e) => { e.preventDefault(); setDrag(true) }} onDragLeave={() => setDrag(false)} onDrop={onDrop}>
+13 -4
View File
@@ -158,6 +158,10 @@ export default function Hardware() {
const capacity = drives.reduce((n, d) => n + d.size, 0)
const threads = cpu.threads ?? 0
const cores = cpu.cores ?? 0
// a two-socket board is two processors, not one with twice the cores: count and name each
const sockets = Math.max(1, cpu.sockets ?? 1)
const multi = sockets > 1
const per = (n: number) => Math.round(n / sockets)
return (
<div className="dash">
@@ -171,8 +175,11 @@ export default function Hardware() {
<div className="kpis">
<div className="tile">
<div className="label">Processor</div>
<div className="value" title={cpu.model} style={(cpu.model?.length ?? 0) > 20 ? { fontSize: 16 } : undefined}>{cpu.model}</div>
<div className="sub">{[cores ? `${cores} core${cores === 1 ? '' : 's'}` : '', threads && threads !== cores ? `${threads} threads` : '', cpu.max_mhz ? `up to ${ghz(cpu.max_mhz)}` : ''].filter(Boolean).join(' · ')}</div>
<div className="value" title={cpu.model} style={(cpu.model?.length ?? 0) > (multi ? 16 : 20) ? { fontSize: 16 } : undefined}>{multi ? `${sockets} × ${cpu.model}` : cpu.model}</div>
<div className="sub">{(multi
? [`${sockets} CPUs`, cores ? `${per(cores)} cores${threads && threads !== cores ? ` / ${per(threads)} threads` : ''} each` : '', cpu.max_mhz ? `up to ${ghz(cpu.max_mhz)}` : '']
: [cores ? `${cores} core${cores === 1 ? '' : 's'}` : '', threads && threads !== cores ? `${threads} threads` : '', cpu.max_mhz ? `up to ${ghz(cpu.max_mhz)}` : '']
).filter(Boolean).join(' · ')}</div>
</div>
<div className="tile">
<div className="label">Memory</div>
@@ -199,9 +206,11 @@ export default function Hardware() {
<div className="props">
<P k="Model" v={cpu.model} />
<P k="Vendor" v={cpu.vendor} />
<P k="Cores" v={cores ? `${cores}${cpu.threads_per_core && cpu.threads_per_core > 1 ? ` · ${cpu.threads_per_core} threads each` : ''}` : ''} />
{multi && <P k="CPUs" v={`${sockets}`} />}
<P k="Cores" v={cores ? `${multi ? `${per(cores)} per CPU · ${cores} in all` : cores}${cpu.threads_per_core && cpu.threads_per_core > 1 ? ` · ${cpu.threads_per_core} threads each` : ''}` : ''} />
{multi && threads > 0 && <P k="Threads" v={`${per(threads)} per CPU · ${threads} in all`} />}
<P k="Clock" v={cpu.min_mhz && cpu.max_mhz ? `${ghz(cpu.min_mhz)} – ${ghz(cpu.max_mhz)}` : ghz(cpu.max_mhz)} />
<P k="Socket" v={(cpu.socket_names ?? []).join(', ')} />
<P k={multi ? 'Sockets' : 'Socket'} v={(cpu.socket_names ?? []).join(', ')} />
<P k="Arch" v={cpu.arch} />
<P k="Family" v={[cpu.family && `family ${cpu.family}`, cpu.model_id && `model ${cpu.model_id}`, cpu.stepping && `stepping ${cpu.stepping}`].filter(Boolean).join(' · ')} />
<P k="Microcode" v={cpu.microcode} mono />
-187
View File
@@ -1,187 +0,0 @@
// The KVM of the machine you are looking at: the device's own web page, shown in the tab.
//
// One per machine, and it follows the machine switcher — switch to `grid` and you get
// grid's KVM. The address is held on the **hub**, not on the machine, because a KVM earns
// its keep exactly when that machine is down and cannot answer for itself; the hub proxies
// the device either way.
//
// Nothing here knows anything about any particular KVM. It is an iframe pointed at Ham
// Lab's own proxy, so whatever web UI the device serves is what you get.
//
// On the keyboard: a page cannot capture keys and forward them into an iframe, and does not
// need to. An iframe with focus receives them directly. What the browser itself swallows —
// Escape, Tab, Ctrl-W, the function keys — is what `navigator.keyboard.lock()` releases,
// and that needs fullscreen and only exists in Chromium. Everywhere else you get the plain
// version, which is most keys, and the device's own on-screen buttons for the rest.
import { FormEvent, useCallback, useEffect, useRef, useState } from 'react'
import { KVM as Device, api } from './api'
import { Btn, Loading, askYes, toast, useEscape } from './ui'
// lock() releases the keys the browser would otherwise keep for itself. Chromium only; a
// browser without it simply does not get the extra keys.
type KeyboardLock = { lock?: (keys?: string[]) => Promise<void>; unlock?: () => void }
const keyboardAPI = (): KeyboardLock | undefined => (navigator as unknown as { keyboard?: KeyboardLock }).keyboard
// Fullscreen, across the three ways browsers spell it. Safari on a Mac wants the webkit
// prefix; Safari on iOS has no fullscreen for an ordinary element at all — only for a
// <video> — so there the page expands inside the viewport instead, which is the same thing
// minus the browser's own chrome.
type FSElement = HTMLElement & { webkitRequestFullscreen?: () => Promise<void> | void }
type FSDocument = Document & { webkitFullscreenElement?: Element | null; webkitExitFullscreen?: () => Promise<void> | void }
const nativeFullscreen = () => {
if (typeof document === 'undefined') return false
const el = document.documentElement as FSElement
return typeof el.requestFullscreen === 'function' || typeof el.webkitRequestFullscreen === 'function'
}
const fullscreenElement = () => (document as FSDocument).fullscreenElement ?? (document as FSDocument).webkitFullscreenElement ?? null
async function enterFullscreen(el: FSElement) {
if (typeof el.requestFullscreen === 'function') return el.requestFullscreen()
if (typeof el.webkitRequestFullscreen === 'function') return el.webkitRequestFullscreen()
throw new Error('this browser has no fullscreen for a page element')
}
function exitFullscreen() {
const d = document as FSDocument
if (typeof d.exitFullscreen === 'function') return d.exitFullscreen()
if (typeof d.webkitExitFullscreen === 'function') return d.webkitExitFullscreen()
}
export default function KVM({ machine, title }: { machine: string | null; title: string }) {
const [kvm, setKvm] = useState<Device | null | undefined>(undefined)
const [adding, setAdding] = useState(false)
const [err, setErr] = useState('')
const [full, setFull] = useState(false) // native fullscreen is on
const [expanded, setExpanded] = useState(false) // the fallback: filling the viewport in-page
const box = useRef<HTMLDivElement>(null)
const frame = useRef<HTMLIFrameElement>(null)
const key = machine ?? '' // the hub is a machine too, and its key is the empty one
const load = useCallback(async () => {
try { setKvm((await api.kvms()).find((k) => k.machine === key) ?? null) }
catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
}, [key])
useEffect(() => { load() }, [load])
// Fullscreen plus a keyboard lock is what makes Escape and Tab reach the machine rather
// than the browser. Both are undone together when fullscreen ends, however it ends.
useEffect(() => {
const onChange = () => {
const on = !!fullscreenElement()
setFull(on)
if (!on) keyboardAPI()?.unlock?.()
}
document.addEventListener('fullscreenchange', onChange)
document.addEventListener('webkitfullscreenchange', onChange)
return () => {
document.removeEventListener('fullscreenchange', onChange)
document.removeEventListener('webkitfullscreenchange', onChange)
keyboardAPI()?.unlock?.()
}
}, [])
// Expanding is in-page, so Escape is ours to handle.
useEscape(() => setExpanded(false), expanded)
const goBig = async () => {
if (!nativeFullscreen()) { setExpanded(true); frame.current?.focus(); return }
try {
if (box.current) await enterFullscreen(box.current)
await keyboardAPI()?.lock?.()
frame.current?.focus()
} catch {
setExpanded(true) // refused (an iframe policy, a permissions policy): fill the page instead
frame.current?.focus()
}
}
const goSmall = () => { setExpanded(false); if (fullscreenElement()) exitFullscreen() }
const remove = async () => {
if (!kvm || !await askYes(`Forget ${title}'s KVM? The device itself is not touched — this only removes its address.`, 'Remove')) return
try { await api.removeKVM(kvm.id); toast('removed'); await load() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
}
if (err && kvm === undefined) return <div className="dash"><div className="error">{err}</div></div>
if (kvm === undefined) return <div className="dash"><Loading label="Looking…" /></div>
const locks = !!keyboardAPI()?.lock
const big = full || expanded
return (
<div className="kvm">
<div className="kvm-bar">
<span className="dname">{title}</span>
{kvm
? <span className="muted small mono kvm-url" title={kvm.url}>{kvm.name !== kvm.url ? `${kvm.name} · ` : ''}{kvm.url}</span>
: <span className="muted small">no KVM</span>}
<span className="spacer" />
{kvm && <>
<Btn className="btn" onClick={goBig} title={!nativeFullscreen() ? 'fill the screen — this browser has no true fullscreen for a page element' : locks ? 'fullscreen, and the browser stops keeping Escape, Tab and Ctrl-W for itself' : 'fullscreen — this browser will still keep some keys for itself'}>{nativeFullscreen() ? 'Fullscreen' : 'Expand'}</Btn>
<a className="btn" href={kvm.url} target="_blank" rel="noreferrer" title="the device directly, outside Ham Lab — only works on the LAN">Open ↗</a>
<Btn className="btn" onClick={() => setAdding(true)}>Change</Btn>
<Btn className="btn danger" onClick={remove}>Remove</Btn>
</>}
{!kvm && <Btn className="btn on" onClick={() => setAdding(true)}>+ Add a KVM for {title}</Btn>}
</div>
{kvm ? (
<div className={'kvm-frame' + (full ? ' full' : '') + (expanded ? ' expanded' : '')} ref={box}>
<iframe ref={frame} key={kvm.id} src={`/kvm/${kvm.id}/`} title={kvm.name}
allow="fullscreen; clipboard-read; clipboard-write" />
{expanded && <button className="btn kvm-shrink" onClick={goSmall} title="back to the console">Done</button>}
</div>
) : (
<div className="svc-empty muted kvm-none">
<div>
<p>No KVM for <b>{title}</b>.</p>
<p className="small">Add one with its address and its page appears here. The address is kept on the hub,
so it still works when {title} does not — which is when you will want it.</p>
</div>
</div>
)}
{kvm && !big && (
<div className="kvm-hint muted small">
Click the picture, then type — the keys go to {title}.
{locks ? ' Fullscreen also takes Escape, Tab and Ctrl-W back from the browser.' : ' This browser keeps Escape, Tab and Ctrl-W for itself; use the device’s own on-screen keys for those, or a Chromium browser.'}
</div>
)}
{adding && <AddDialog machine={key} title={title} existing={kvm} onClose={() => setAdding(false)} onAdded={() => { setAdding(false); load() }} />}
</div>
)
}
function AddDialog({ machine, title, existing, onClose, onAdded }: { machine: string; title: string; existing: Device | null; onClose: () => void; onAdded: () => void }) {
const [url, setUrl] = useState(existing?.url ?? '')
const [name, setName] = useState(existing?.name ?? '')
const [err, setErr] = useState('')
const [busy, setBusy] = useState(false)
useEscape(onClose)
const save = async (e: FormEvent) => {
e.preventDefault(); setErr(''); setBusy(true)
try { await api.addKVM(name.trim(), url.trim(), machine); onAdded() }
catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setBusy(false) }
}
return (
<div className="modal-back" onClick={onClose}>
<form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
<h3>{existing ? `Change ${title}'s KVM` : `A KVM for ${title}`}</h3>
<label className="muted">Address — whatever you would type into a browser
<input autoFocus className="mono" value={url} onChange={(e) => setUrl(e.target.value)} placeholder="10.0.0.217" spellCheck={false} autoCapitalize="off" required />
</label>
<label className="muted">Name — optional; the address is used otherwise
<input value={name} onChange={(e) => setName(e.target.value)} placeholder={`${title} KVM`} maxLength={64} />
</label>
<div className="muted small">
Ham Lab serves the device from its own address, so a KVM that speaks only plain HTTP
still works here, and it is behind this login rather than needing a second one. The
address is kept on the hub, so it still reaches the device when {title} is down. The
device itself is not changed and is never reachable from outside.
{existing && <> Saving replaces the one there now — a machine has one KVM.</>}
</div>
{err && <div className="error">{err}</div>}
<div className="svc-actions">
<button type="submit" className="btn on" disabled={busy || !url.trim()}>{existing ? 'Save' : 'Add'}</button>
<span className="spacer" />
<button type="button" className="btn" onClick={onClose}>Cancel</button>
</div>
</form>
</div>
)
}
+46 -2
View File
@@ -53,6 +53,9 @@ class PatientAbr extends Base {
}
}
// Whether the browser lets a page set the volume level (iOS doesn't: only mute).
const canLevel = (() => { try { const a = document.createElement('audio'); a.volume = 0.5; return a.volume === 0.5 } catch { return false } })()
const mbps = (b: number) => `${b >= 10e6 ? Math.round(b / 1e6) : (b / 1e6).toFixed(1)} Mb/s`
const clock = (sec: number) => { const t = Math.max(0, Math.floor(sec)), h = Math.floor(t / 3600), m = Math.floor((t % 3600) / 60), x = t % 60; return `${h ? `${h}:${String(m).padStart(2, '0')}` : m}:${String(x).padStart(2, '0')}` }
const size = (w?: number, h?: number, vr?: boolean) => {
@@ -68,6 +71,22 @@ const I = ({ d, size: sz = 20, fill }: { d: string; size?: number; fill?: boolea
const PLAY = 'M7 4.5v15l12.5-7.5z', PAUSE = 'M8 5v14M16 5v14', CHECK = 'M5 12.5l4.5 4.5L19 7.5', TUNE = 'M4 7h10M18 7h2M4 17h4M12 17h8M16 5v4M10 15v4'
const FULL = 'M4 9V4h5M20 9V4h-5M4 15v5h5M20 15v5h-5', SMALL = 'M9 4v5H4M15 4v5h5M9 20v-5H4M15 20v-5h5', PHONE = 'M8 3h8a1 1 0 0 1 1 1v16a1 1 0 0 1-1 1H8a1 1 0 0 1-1-1V4a1 1 0 0 1 1-1zM3 9a9 9 0 0 0 0 6M21 9a9 9 0 0 1 0 6'
// The speaker: waves by the level, a cross when muted.
const Speaker = ({ off, low }: { off?: boolean; low?: boolean }) => (
<svg width={20} height={20} viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="1.8" strokeLinecap="round" strokeLinejoin="round" aria-hidden="true">
<path d="M4 9.5h3.5L12 5.5v13l-4.5-4H4z" fill="currentColor" />
{off ? <path d="M16 9.5l5 5M21 9.5l-5 5" /> : <path d={low ? 'M15.5 9.5a3.5 3.5 0 0 1 0 5' : 'M15.5 9.5a3.5 3.5 0 0 1 0 5M18 7a7 7 0 0 1 0 10'} />}
</svg>
)
// Five seconds back or on: a circling arrow with the 5 inside.
const Skip = ({ back }: { back?: boolean }) => (
<svg width={20} height={20} viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="1.8" strokeLinecap="round" strokeLinejoin="round" aria-hidden="true">
<path d={back ? 'M12 5a7 7 0 1 1-7 7M12 5l2.5-2.5M12 5l2.5 2.5' : 'M12 5a7 7 0 1 0 7 7M12 5L9.5 2.5M12 5l-2.5 2.5'} />
<text x="12" y="15" textAnchor="middle" fontSize="7.5" fontWeight="700" fill="currentColor" stroke="none" fontFamily="system-ui, sans-serif">5</text>
</svg>
)
export default function Player({ path, sudo }: { path: string; sudo: boolean; name: string }) {
const [open, setOpen] = useState<PlayOpen | null>(null)
const [err, setErr] = useState('')
@@ -231,10 +250,23 @@ export default function Player({ path, sudo }: { path: string; sudo: boolean; na
el.addEventListener('change', commit)
return () => el.removeEventListener('change', commit)
})
// Volume: a level and mute, remembered for the next video. iOS lets a page mute but not set
// the level (it reads back as 1), so there the slider isn't shown.
const [vol, setVol] = useState(() => { try { const v = JSON.parse(localStorage.getItem('hamlab.play.volume') ?? ''); return { level: Math.min(1, Math.max(0, Number(v.level) || 0)), muted: !!v.muted } } catch { return { level: 1, muted: false } } })
useEffect(() => {
const v = video.current
if (v) { v.volume = vol.level; v.muted = vol.muted || vol.level === 0 }
try { localStorage.setItem('hamlab.play.volume', JSON.stringify(vol)) } catch { /* private mode */ }
}, [vol, open])
const volBy = useCallback((d: number) => setVol((x) => ({ level: Math.round(Math.min(1, Math.max(0, x.level + d)) * 20) / 20, muted: false })), [])
const muteToggle = useCallback(() => setVol((x) => (x.muted || x.level === 0 ? { level: x.level || 0.5, muted: false } : { ...x, muted: true })), [])
const seekBy = useCallback((d: number) => {
const v = video.current
if (!v) return
v.currentTime = Math.max(0, Math.min(info?.duration ?? v.duration, v.currentTime + d))
const to = Math.max(0, Math.min(info?.duration ?? v.duration, v.currentTime + d))
v.currentTime = to
setTime(to) // the ball goes there at once, made yet or not
poke()
}, [info, poke])
// Full screen: the browser's where it has one for an element, else (an iPhone) the window.
@@ -258,6 +290,9 @@ export default function Player({ path, sudo }: { path: string; sudo: boolean; na
else if (e.key === 'ArrowLeft' || e.key === 'j') { e.preventDefault(); seekBy(e.key === 'j' ? -10 : -5) }
else if (e.key === 'ArrowRight' || e.key === 'l') { e.preventDefault(); seekBy(e.key === 'l' ? 10 : 5) }
else if (e.key === 'f') fullscreen()
else if (e.key === 'm') muteToggle()
else if (e.key === 'ArrowUp') { e.preventDefault(); volBy(0.1) }
else if (e.key === 'ArrowDown') { e.preventDefault(); volBy(-0.1) }
else if (e.key === 'Escape') { if (menu) setMenu(false); else if (full && !document.fullscreenElement) setFull(false) }
}
@@ -330,13 +365,22 @@ export default function Player({ path, sudo }: { path: string; sudo: boolean; na
{vr && open && chrome && !menu && time < 8 && !err && <div className="vp-hint">Drag to look around · scroll or pinch to zoom</div>}
<div className="vp-bottom">
<span className="vp-time above">{clock(shown)} / {clock(dur)}</span>
<div className="vp-seek" style={{ ['--p' as string]: dur ? `${(shown / dur) * 100}%` : '0%', ['--b' as string]: dur ? `${(buffered / dur) * 100}%` : '0%' }}>
<input ref={seekBar} type="range" min={0} max={dur || 1} step={0.1} value={shown} aria-label="Seek"
onChange={(e) => { setScrub(Number(e.target.value)); poke() }} />
</div>
<div className="vp-row">
<button className="vp-icon" onClick={toggle} aria-label={paused ? 'Play' : 'Pause'}>{paused ? <I d={PLAY} fill /> : <I d={PAUSE} />}</button>
<span className="vp-time">{clock(shown)} / {clock(dur)}</span>
<button className="vp-icon" onClick={() => seekBy(-5)} aria-label="Back 5 seconds" title="Back 5 seconds"><Skip back /></button>
<button className="vp-icon" onClick={() => seekBy(5)} aria-label="On 5 seconds" title="On 5 seconds"><Skip /></button>
<span className="vp-vol">
<button className="vp-icon" onClick={muteToggle} aria-label={vol.muted || vol.level === 0 ? 'Unmute' : 'Mute'} title={vol.muted || vol.level === 0 ? 'Unmute (m)' : 'Mute (m)'}><Speaker off={vol.muted || vol.level === 0} low={vol.level < 0.5} /></button>
{canLevel && <input type="range" min={0} max={1} step={0.05} value={vol.muted ? 0 : vol.level} aria-label="Volume"
style={{ ['--v' as string]: `${(vol.muted ? 0 : vol.level) * 100}%` }}
onChange={(e) => { const level = Number(e.target.value); setVol({ level, muted: level === 0 }); poke() }} />}
</span>
<span className="vp-time inrow">{clock(shown)} / {clock(dur)}</span>
<span className="spacer" />
{vr && canMotion() && (
<button className={`vp-icon${motion ? ' on' : ''}`} aria-label="Look around by moving the phone" aria-pressed={motion}
+3 -6
View File
@@ -15,7 +15,6 @@ import Hardware from './Hardware'
import Network from './Network'
import VPN from './VPN'
import ZFS from './ZFS'
import KVM from './KVM'
import Files from './Files'
import Top from './Top'
import Machines from './Machines'
@@ -81,7 +80,6 @@ const GROUPS: Group[] = [
{ id: 'usage', label: 'Usage', icon: '▧' },
{ id: 'zfs', label: 'ZFS', icon: '⛁' },
{ id: 'docker', label: 'Docker', icon: DockerIcon },
{ id: 'kvm', label: 'KVM', icon: '⌨' },
{ id: 'hardware', label: 'Hardware', icon: '⬡' },
{ id: 'updates', label: 'Updates', icon: '⇪' },
{ id: 'alerts', label: 'Alerts', icon: BellIcon },
@@ -91,8 +89,8 @@ const GROUPS: Group[] = [
{ id: 'settings', label: 'Settings', icon: '⚙' },
]
type View = 'terminal' | 'widgets' | 'docker' | 'alerts' | 'services' | 'jobs' | 'updates' | 'disks' | 'hardware' | 'network' | 'vpn' | 'zfs' | 'usage' | 'files' | 'procs' | 'machines' | 'kvm' | 'users' | 'settings'
const VIEWS: View[] = ['terminal', 'widgets', 'docker', 'alerts', 'services', 'jobs', 'updates', 'disks', 'hardware', 'network', 'vpn', 'zfs', 'usage', 'files', 'procs', 'machines', 'kvm', 'users', 'settings']
type View = 'terminal' | 'widgets' | 'docker' | 'alerts' | 'services' | 'jobs' | 'updates' | 'disks' | 'hardware' | 'network' | 'vpn' | 'zfs' | 'usage' | 'files' | 'procs' | 'machines' | 'users' | 'settings'
const VIEWS: View[] = ['terminal', 'widgets', 'docker', 'alerts', 'services', 'jobs', 'updates', 'disks', 'hardware', 'network', 'vpn', 'zfs', 'usage', 'files', 'procs', 'machines', 'users', 'settings']
const HUB_ONLY: View[] = ['machines', 'users', 'settings', 'alerts']
// URL hash is the source of truth for where you are: #/<machine|hub>/<view>. Back and
@@ -324,7 +322,6 @@ export default function Shell({ me: hubMe, onLoggedOut }: { me: Me; onLoggedOut:
<PageBoundary page={GROUPS.find((g) => g.id === view)?.label ?? view}>
{view === 'settings' && <Settings me={hubMe} />}
{view === 'users' && <Users me={hubMe} />}
{view === 'kvm' && <KVM key={machine ?? 'hub'} machine={machine} title={title} />}
{view === 'usage' && <Usage key={machine ?? ''} />}
{view === 'machines' && <Machines hub={hubMe} onChanged={loadMachines} onOpen={(n) => { switchMachine(n); setView('procs') }} />}
{view === 'widgets' && <Gallery />}
@@ -366,7 +363,7 @@ export default function Shell({ me: hubMe, onLoggedOut }: { me: Me; onLoggedOut:
<span>{machine ? `${title} · ` : ''}{me.version.startsWith('v') ? me.version : 'v' + me.version}</span>
<span>up {fmtUptime(me.uptime)}</span>
<span className="spacer" />
<span className="muted">{view === 'widgets' ? 'Widgets' : view === 'docker' ? 'Docker' : view === 'alerts' ? 'Alerts' : view === 'services' ? 'Services' : view === 'jobs' ? 'Jobs' : view === 'updates' ? 'Updates' : view === 'disks' ? 'Disks' : view === 'hardware' ? 'Hardware' : view === 'network' ? 'Network' : view === 'vpn' ? 'VPN' : view === 'zfs' ? 'ZFS' : view === 'usage' ? 'Usage' : view === 'files' ? 'Files' : view === 'procs' ? 'Processes' : view === 'machines' ? 'Machines' : view === 'kvm' ? 'KVM' : view === 'users' ? 'Users' : view === 'settings' ? 'Settings' : activeTerm ? activeTerm.name : ''}</span>
<span className="muted">{view === 'widgets' ? 'Widgets' : view === 'docker' ? 'Docker' : view === 'alerts' ? 'Alerts' : view === 'services' ? 'Services' : view === 'jobs' ? 'Jobs' : view === 'updates' ? 'Updates' : view === 'disks' ? 'Disks' : view === 'hardware' ? 'Hardware' : view === 'network' ? 'Network' : view === 'vpn' ? 'VPN' : view === 'zfs' ? 'ZFS' : view === 'usage' ? 'Usage' : view === 'files' ? 'Files' : view === 'procs' ? 'Processes' : view === 'machines' ? 'Machines' : view === 'users' ? 'Users' : view === 'settings' ? 'Settings' : activeTerm ? activeTerm.name : ''}</span>
</footer>
</div>
)
+59 -25
View File
@@ -3,7 +3,7 @@
// as they climb. The numbers are Ham Lab's own — the sensor stream for everything measured and
// /api/procs for the table — so this works on any machine the console can see, whether or not
// btop is installed on it.
import { createContext, memo, useCallback, useContext, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
import { Fragment, createContext, memo, useCallback, useContext, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
import { api, GPU, ProcTable, Proc, Sample } from './api'
import { Catalog, MONITOR_SHAPE, Values, mergeValues, sampleOf, stale } from './sampleAdapter'
import { fetchCatalog, fetchHistories, useSensorStream } from './sensors'
@@ -11,6 +11,7 @@ import { fmtMs, nextPoint, REFRESH_OPTIONS, setRefreshMs, strideMs, useHistorySe
import { Loading, askYes, toast } from './ui'
import { Theme, currentThemeId, themeById } from './themes'
import { columnsAt } from './graphcols'
import { socketThreads } from './sockets'
import { savePrefs } from './accents'
const MAX = 300 // points kept per curve, as the machines keep them
@@ -294,6 +295,13 @@ export default function Top() {
// the process table, and the CPU model, come from the same place the Processes page reads,
// on the sample's beat (above)
const [pickedIf, setPickedIf] = useState<string | null>(null) // which NIC the net box draws
// which sockets' threads are folded away on a two-socket machine; remembered in this browser
const [folded, setFolded] = useState<number[]>(() => { try { return JSON.parse(localStorage.getItem('top.folded-cpus') ?? '[]') } catch { return [] } })
const fold = (pk: number) => setFolded((f) => {
const n = f.includes(pk) ? f.filter((x) => x !== pk) : [...f, pk]
try { localStorage.setItem('top.folded-cpus', JSON.stringify(n)) } catch { /* private window */ }
return n
})
// Which pool each drive belongs to, asked once: a drive in a ZFS pool has no filesystem of
// its own, and "nothing is on it" would be a lie about eight drives holding a NAS.
const [pools, setPools] = useState<Record<string, string>>({})
@@ -360,7 +368,16 @@ export default function Top() {
const model = (pt?.cpu_model ?? '').replace(/\(R\)|\(TM\)|CPU|Processor/g, '').replace(/@.*$/, '').replace(/^\d+th Gen\s*/i, '').replace(/\s+/g, ' ').trim()
// the hottest thing that looks like a CPU sensor, and one per core where the chip names them;
// a second thread of a core (CPU 8 on an 8-core chip with hyper-threading) wears its core's
const coreTemp = (i: number) => { const id = pt?.core_ids?.[i] ?? i; return s?.temps.find((t) => new RegExp(`core ${id}\\b`, 'i').test(t.label))?.c }
// On a two-socket board each CPU has its own sensor chip, both counting from "Core 0", so a
// thread's core is looked up on its own socket's chip.
const pkgs = pt?.core_pkgs?.length === cores.length ? pt.core_pkgs : undefined
// each CPU's own threads, counted from 0 under it (sockets.ts): the kernel's one numbering
// of the whole machine puts cpu 0–9 and 20–29 on the first and 10–19 and 30–39 on the second
const socks = pkgs ? socketThreads(pkgs) : []
const sockets = socks.map((x) => x.pkg)
const coreTemp = (i: number) => { const id = pt?.core_ids?.[i] ?? i, pk = pkgs?.[i]; return s?.temps.find((t) => new RegExp(`core ${id}\\b`, 'i').test(t.label) && (pk === undefined || (t.pkg ?? 0) === pk))?.c }
const sockTemp = (pk: number) => s?.temps.find((t) => t.chip === 'coretemp' && new RegExp(`^package id ${pk}$`, 'i').test(t.label))?.c
const sockUse = (pk: number) => { const v = cores.filter((_, i) => pkgs?.[i] === pk); return v.length ? v.reduce((a, b) => a + b, 0) / v.length : 0 }
const cpuTemp = pt?.cpu_temp ?? s?.temps.find((t) => /package|tctl|tdie|cpu/i.test(t.label))?.c ?? coreTemp(0)
// the temperature columns are there for every row once any row has one, so the % column lines up
const anyTemp = cpuTemp !== undefined || cores.some((_, i) => coreTemp(i) !== undefined)
@@ -403,26 +420,43 @@ export default function Top() {
<Graph values={curves.get('cpu.usage') ?? []} tick={tick} advance={advance} now={now} cap={cap} max={100} grad="cpu" />
<div className="t-up">up {upfor(uptime)}</div>
</div>
<div className="t-cores">
<div className="t-cores-head"><b>{model || 'cpu'}</b><span className="spacer" />{pt?.cpu_mhz ? `${Math.round(pt.cpu_mhz)} MHz` : ''}
{pt?.cpu_power_w ? <span title="the CPU's own power reading: cores, built-in graphics and memory controller, not the whole machine">{pt.cpu_mhz ? ' · ' : ''}{pt.cpu_power_w < 10 ? pt.cpu_power_w.toFixed(1) : Math.round(pt.cpu_power_w)} W</span> : null}</div>
<div className="t-core">
<div className={'t-cores' + (sockets.length > 1 ? ' multi' : '')}>
<div className="t-cores-head"><b>{sockets.length > 1 ? `${sockets.length} × ` : ''}{model || 'cpu'}</b><span className="spacer" />{pt?.cpu_mhz ? `${Math.round(pt.cpu_mhz)} MHz` : ''}
{pt?.cpu_power_w && sockets.length < 2 ? <span title="the CPU's own power reading: cores, built-in graphics and memory controller, not the whole machine">{pt.cpu_mhz ? ' · ' : ''}{pt.cpu_power_w < 10 ? pt.cpu_power_w.toFixed(1) : Math.round(pt.cpu_power_w)} W</span> : null}</div>
{sockets.length < 2 && <div className="t-core">
<span className="t-core-n">CPU</span>
<Meter pct={cpu} grad="cpu" />
<span className="t-core-p">{Math.round(cpu)}%</span>
{anyTemp && <><span className="t-core-t">{cpuTemp !== undefined && <Meter pct={Math.min(100, (cpuTemp / 100) * 100)} grad="temp" />}</span><span className="t-core-c">{cpuTemp !== undefined ? `${Math.round(cpuTemp)}°C` : ''}</span></>}
</div>
{cores.map((v, i) => {
const t = coreTemp(i)
return (
<div className="t-core" key={i}>
<span className="t-core-n">C{i}</span>
<span className="t-core-g" title={pt?.core_ids && pt.core_ids[i] !== i ? `cpu ${i} · a thread of core ${pt.core_ids[i]}` : `core ${i}`}>{braille(recent.get(`cpu.core.${i}.usage`) ?? [], 9)}</span>
<span className="t-core-p">{Math.round(v)}%</span>
{anyTemp && <><span className="t-core-t">{t !== undefined && <Meter pct={Math.min(100, t)} grad="temp" />}</span><span className="t-core-c">{t !== undefined ? `${Math.round(t)}°C` : ''}</span></>}
</div>
)
})}
</div>}
{(sockets.length > 1 ? socks : [{ pkg: undefined, cpus: cores.map((_, i) => i) }]).map(({ pkg: pk, cpus }) => (
<Fragment key={pk ?? 'all'}>
{pk !== undefined && (() => {
const u = sockUse(pk), t = sockTemp(pk), w = pt?.cpu_power_pkgs_w?.[pk], open = !folded.includes(pk)
return (
<div className="t-core t-sock" role="button" aria-expanded={open} title={open ? 'hide its threads' : 'show its threads'} onClick={() => fold(pk)}>
<span className="t-core-n"><span className="t-fold">{open ? '▾' : '▸'}</span>CPU{pk + 1}</span>
<span className="t-core-w" title="this CPU's own power reading: its cores, cache and memory controller">{w ? `${w < 10 ? w.toFixed(1) : Math.round(w)} W` : ''}</span>
<Meter pct={u} grad="cpu" />
<span className="t-core-p">{Math.round(u)}%</span>
{anyTemp && <><span className="t-core-t">{t !== undefined && <Meter pct={Math.min(100, t)} grad="temp" />}</span><span className="t-core-c">{t !== undefined ? `${Math.round(t)}°C` : ''}</span></>}
</div>
)
})()}
{(pk !== undefined && folded.includes(pk) ? [] : cpus).map((i, n) => {
// n is the thread's number under its own CPU, i the kernel's for the whole machine
const t = coreTemp(i), v = cores[i] ?? 0
return (
<div className="t-core" key={i}>
<span className="t-core-n">C{n}</span>
<span className="t-core-g" title={pk !== undefined ? `CPU ${pk + 1} · thread ${n} · cpu ${i} to the kernel${pt?.core_ids ? ` · a thread of core ${pt.core_ids[i]}` : ''}` : pt?.core_ids && pt.core_ids[i] !== i ? `cpu ${i} · a thread of core ${pt.core_ids[i]}` : `core ${i}`}>{braille(recent.get(`cpu.core.${i}.usage`) ?? [], 9)}</span>
<span className="t-core-p">{Math.round(v)}%</span>
{anyTemp && <><span className="t-core-t">{t !== undefined && <Meter pct={Math.min(100, t)} grad="temp" />}</span><span className="t-core-c">{t !== undefined ? `${Math.round(t)}°C` : ''}</span></>}
</div>
)
})}
</Fragment>
))}
<div className="t-load">Load AVG:<span className="spacer" />{load.map((x) => x.toFixed(2)).join(' ')}</div>
</div>
</div>
@@ -456,7 +490,9 @@ export default function Top() {
// pool or an md array spans several drives, so nothing is attributed and the row
// shows what is true of the hardware — how busy it is, and how hot.
const on = fs.filter((f) => f.disk === d.name)
const used = on.reduce((a, f) => a + f.used, 0)
// a bind mount (nexus's /var/log is /data/var-log) is the same filesystem twice:
// count each source device once, or the drive reads double
const used = [...new Map(on.map((f) => [f.source || f.mount, f.used])).values()].reduce((a, b) => a + b, 0)
return <Disk key={d.name} name={d.name} note={d.model} size={d.size} used={on.length ? used : undefined} io={d.util_pct} temp={d.temp} mounts={on.map((f) => f.mount)} pool={pools[d.name]} />
})}
{(s?.disks ?? []).length === 0 && <div className="t-dim">no drives reported</div>}
@@ -560,13 +596,11 @@ const ENGINES: [string, string][] = [['render', 'render'], ['compute', 'compute'
function GPUBox({ g, values, tick, advance, now, cap }: { g: GPU; values: number[]; tick: number; advance: number; now: number; cap: number }) {
const engines = ENGINES.filter(([k]) => g.engines && g.engines[k] !== undefined)
const temp = g.temp !== undefined && g.temp > 0 ? g.temp : undefined
const how = g.stalled ? 'a sysfs read of this card stopped answering; its readings are paused until the console restarts'
: g.usage_src === 'idle' ? 'busy, from idle residency' : g.usage_src === 'busy' ? 'busy, as the driver reports it' : g.usage_src === 'clients' ? 'the busiest engine, summed over its clients' : 'this driver exposes no activity counter'
return (
<div className="t-gpu">
<div className="t-gpu-graph">
<Graph values={values} tick={tick} advance={advance} now={now} cap={cap} max={100} grad="cpu" />
<div className="t-up">{g.usage_src ? `${Math.round(g.usage)}% ` : ''}{how}</div>
{g.stalled && <div className="t-up">a read of this card stopped answering; its readings are paused until it does</div>}
</div>
<div className="t-gpu-stats">
<div className="t-cores-head"><b>{g.name}</b><span className="spacer" />{g.driver}{g.integrated ? ' · integrated' : ''}</div>
@@ -590,7 +624,7 @@ function GPUBox({ g, values, tick, advance, now, cap }: { g: GPU; values: number
<span className="t-fact" title={g.req_mhz ? `${Math.round(g.req_mhz)} MHz requested${g.max_mhz ? `, ${Math.round(g.max_mhz)} MHz max` : ''}` : undefined}>{Math.round(g.mhz ?? 0)}{g.max_mhz ? ` / ${Math.round(g.max_mhz)}` : ''} MHz</span></div> : null}
{(g.power_w !== undefined && g.power_w > 0) || g.power_cap_w ? <div className="t-row t-sub"><span>Power</span><span className="spacer" />
<span className="t-fact">{(g.power_w ?? 0).toFixed(1)}{g.power_cap_w ? ` / ${Math.round(g.power_cap_w)}` : ''} W</span></div> : null}
{g.fan_rpm ? <div className="t-row t-sub"><span>Fan</span><span className="spacer" /><span className="t-fact">{g.fan_rpm} rpm</span></div> : null}
{g.fan_rpm || g.fan_pct !== undefined ? <div className="t-row t-sub"><span>Fan</span><span className="spacer" /><span className="t-fact">{[g.fan_pct !== undefined ? `${Math.round(g.fan_pct)}%` : '', g.fan_rpm ? `${g.fan_rpm} rpm` : ''].filter(Boolean).join(' · ')}</span></div> : null}
{g.mem_total ? <div className="t-row t-sub"><span>Mem</span><span className="t-pct">{Math.round(((g.mem_used ?? 0) / g.mem_total) * 100)}%</span><Meter pct={((g.mem_used ?? 0) / g.mem_total) * 100} grad="used" /><span className="t-amt">{bytes(g.mem_used ?? 0, 1)}</span></div>
: g.mem_used ? <div className="t-row t-sub"><span>Mem</span><span className="spacer" /><span className="t-fact" title="resident memory of the processes that have it open">{bytes(g.mem_used, 1)}</span></div> : null}
</div>
@@ -611,13 +645,13 @@ function GPUBox({ g, values, tick, advance, now, cap }: { g: GPU; values: number
// Nothing on this GPU can be read — no busy counter, engine, temperature, clock, power, memory,
// nor a process with it open: the Mac's Asahi, today. Its box would be a graph that can never
// draw beside a column of dashes, so it is one line instead. A client opening it brings the box back.
const unmeasured = (g: GPU) => !g.usage_src && !g.engines && !(g.temp && g.temp > 0) && !g.mhz && !g.max_mhz && !g.power_w && !g.power_cap_w && !g.fan_rpm && !g.mem_total && !g.mem_used && g.clients.length === 0
const unmeasured = (g: GPU) => !g.usage_src && !g.engines && !(g.temp && g.temp > 0) && !g.mhz && !g.max_mhz && !g.power_w && !g.power_cap_w && !g.fan_rpm && g.fan_pct === undefined && !g.mem_total && !g.mem_used && g.clients.length === 0
function GPULine({ g }: { g: GPU }) {
return (
<div className="t-row t-gpu-line">
<span>{g.driver}{g.integrated ? ' · integrated' : ''}</span>
{g.stalled
? <span className="t-dim" title="a sysfs read of this card stopped answering">readings paused until the console restarts</span>
? <span className="t-dim" title="a read of this card stopped answering">readings paused until it answers</span>
: <span className="t-dim" title="the driver exposes no busy counter, clock, temperature, power or memory reading">nothing measurable</span>}
<span className="t-dim t-gpu-who" title={g.clients_ok ? undefined : 'the console has no root here'}>{g.clients_ok ? 'no process has it open' : 'other users’ processes are hidden'}</span>
</div>
+395 -8
View File
@@ -9,8 +9,8 @@
// disagree over. An existing cron job is detected and shown, so the page tells the truth
// about the machine whether Ham Lab or something else is doing the work.
import { FormEvent, Fragment, useCallback, useEffect, useState } from 'react'
import { api, ARCView, CacheDev, PoolCaching, ZFSExternal, ZFSInfo, ZFSOverview, ZFSPolicy, ZFSRun, ZFSSnapshot } from './api'
import { fmtBytes } from './fmt'
import { api, ARCView, CacheDev, Landing, LandingCheck, PoolCaching, PoolWrites, VerifyPolicy, VerifyResult, ZFSExternal, ZFSInfo, ZFSOverview, ZFSPolicy, ZFSRun, ZFSSnapshot, ZFSWrites } from './api'
import { fmtBytes, fmtRate } from './fmt'
import { useRefreshMs } from './prefs'
import { fetchHistories } from './sensors'
import Sparkline from './Sparkline'
@@ -367,18 +367,149 @@ function PoolTiers({ p, v }: { p: PoolCaching; v: ARCView }) {
)
}
// ---- the write path: the dirty buffer in RAM, what each pool writes, and a landing tier ----
//
// A write reaches a pool in up to three steps. If a landing tier fronts the pool (a mergerfs
// pool whose first branch is fast storage and whose last is the pool), new files land there
// first and a mover drains them. Either way the data then waits in RAM as dirty buffers, and
// each txg sync writes it to the disks. Each step's live number is here, with its history.
const dur = (s: number) => (!Number.isFinite(s) || s <= 0 ? '—' : s < 90 ? `${Math.round(s)} s` : s < 5400 ? `${Math.round(s / 60)} min` : `${(s / 3600).toFixed(1)} h`)
type WHist = Record<string, number[]>
const wIds = (w: ZFSWrites) => ['zfs.write.dirty', ...w.pools.flatMap((p) => {
const b = `zfs.${p.pool}.`
return [b + 'write_bps', b + 'logical_bps', ...(p.landing ? [b + 'landing.pending_bytes', b + 'landing.in_bps', b + 'landing.drain_bps'] : [])]
})]
// rsync's --bwlimit: a bare number is KiB/s, a K/M/G suffix is binary (60M = 60 MiB/s); 0 is none
const bwl = (v: string) => {
const m = /^(\d+(?:\.\d+)?)([KMG]?)$/i.exec((v ?? '').trim())
if (!m || Number(m[1]) === 0) return ''
return ` · rsync limited to ${m[1]} ${(m[2] || 'K').toUpperCase()}iB/s`
}
function MoverLine({ l }: { l: Landing }) {
const m = l.mover
if (!m) return <div className="notice" style={{ marginTop: 6 }}>No mover is reporting for this landing tier (nothing in <span className="mono">/run/landing-mover/</span>). Files that land here stay here until something moves them.</div>
const frac = m.current_size > 0 ? m.partial / m.current_size : 0
return (
<>
<div className="props rows" style={{ marginTop: 6 }}>
<div><span className="k">Mover</span><span>
<span className={'badge ' + (m.state === 'moving' ? 'on' : m.state === 'idle' ? '' : 'warn')}>{m.state}</span>
{' '}{m.state === 'moving' && m.batch_files > 1 ? `batch of ${m.batch_files} files, ${fmtBytes(m.batch_bytes)}` : m.state === 'holding' ? 'a mount it needs is missing — waiting' : m.state === 'stopped' ? `no word since ${ago(m.updated_ms)}` : `queue ${m.queue}`}
{m.state === 'moving' && m.queue > 0 ? ` · ${m.queue} more queued` : ''}
</span></div>
{m.state === 'moving' && m.current && <div><span className="k">Now</span><span className="mono small" title={m.current}>{m.current}{m.current_size ? ` · ${fmtBytes(m.partial)} of ${fmtBytes(m.current_size)}` : ''}</span></div>}
<div><span className="k">Since start</span><span>{fmtN(m.moved_files)} files · {fmtBytes(m.moved_bytes)} moved · running {ago(m.started_ms).replace(' ago', '')}</span></div>
{m.last_move && <div><span className="k">Last moved</span><span className="mono small" title={m.last_move}>{m.last_move} · {ago(m.last_move_ms)}</span></div>}
<div><span className="k">Rules</span><span className="muted">a file moves once it has been closed and left alone for {m.settle} s · oldest first{bwl(m.bwlimit)}</span></div>
{m.last_error && <div><span className="k">Last error</span><span className="warn-text" style={{ whiteSpace: 'normal' }}>{m.last_error} · {ago(m.last_error_ms)}</span></div>}
</div>
{m.state === 'moving' && m.current_size > 0 && <Meter frac={frac} cls="" />}
</>
)
}
function LandingBlockInner({ l, pending, inn, drain }: { l: Landing; pending: number[]; inn: number[]; drain: number[] }) {
const eta = l.drain_bps > 0 ? l.pending_bytes / l.drain_bps : NaN
const fill = l.size > 0 ? l.used / l.size : 0
return (
<div className="landing-tier">
<div className="arc-tier"><span className="k" title="a mergerfs pool puts new files on this branch first; a mover drains them onto the pool">Landing</span>
<span className="arc-tier-v"><span className="mono" title={`${l.branch} → ${l.pool_branch} via ${l.mount}`}>{l.branch}{l.device ? ` · ${l.device}` : ''}{l.fstype ? ` · ${l.fstype}` : ''} · in front of {l.pool_branch} as {l.mount}</span></span>
</div>
<div className="sub-head" style={{ marginTop: 6 }}>
<span className="muted small">{fmtBytes(l.used)} of {fmtBytes(l.size)} used</span><span className="spacer" />
<span className="num">{l.pending_files ? `${fmtN(l.pending_files)}${l.pending_partial ? '+' : ''} waiting · ${fmtBytes(l.pending_bytes)}` : 'nothing waiting'}</span>
</div>
<Meter frac={fill} />
<div className="props rows" style={{ marginTop: 6 }}>
<div><span className="k">Waiting</span><span>{l.pending_files ? `${fmtN(l.pending_files)}${l.pending_partial ? '+' : ''} files, ${fmtBytes(l.pending_bytes)} · oldest landed ${ago(l.oldest_ms)}` : 'nothing — everything that landed is on the pool'}<span className="muted small"> · counted {ago(l.counted_ms)}</span></span></div>
<div><span className="k">Arriving</span><span>{fmtRate(l.in_bps)}{l.device ? <span className="muted small"> · writes to {l.device}</span> : ''}</span></div>
<div><span className="k">To the pool</span><span>{fmtRate(l.drain_bps)}{l.pending_bytes > 0 ? ` · ${Number.isFinite(eta) ? `clear in about ${dur(eta)}` : 'not draining right now'}` : ''}</span></div>
</div>
<MoverLine l={l} />
<div className="arc-sparks">
<div>
<div className="muted small">Waiting on the landing tier · {fmtBytes(l.pending_bytes)}</div>
<Sparkline series={[{ name: 'waiting', values: pending, color: cssVar('--s1') }]} format={fmtBytes} height={40} />
</div>
<div>
<div className="muted small">Arriving {fmtRate(l.in_bps)} · to the pool {fmtRate(l.drain_bps)}</div>
<Sparkline series={[
{ name: 'arriving', values: inn, color: cssVar('--s1') },
{ name: 'to the pool', values: drain, color: cssVar('--ok') },
]} format={fmtRate} height={40} />
</div>
</div>
</div>
)
}
function WritePath({ w, p, h }: { w: ZFSWrites; p: PoolWrites; h: WHist }) {
const b = `zfs.${p.pool}.`
const throttling = w.delayed_per_sec > 0 || w.over_max_per_sec > 0
const delayAt = (w.limit * w.delay_min_pct) / 100
const ratio = p.write_bps > 0 && p.logical_bps > 0 ? p.logical_bps / p.write_bps : 0
return (
<section className="card">
<div className="sub-head">
<span className={'dot ' + (w.over_max_per_sec > 0 ? 'bad' : throttling ? 'warn' : 'ok')} />
<span className="dname">{p.pool}</span>
<span className="badge">write path</span>
<span className="spacer" />
<span className="num">{fmtRate(p.write_bps)} to disk</span>
</div>
{p.landing
? <LandingBlockWithHist l={p.landing} h={h} b={b} />
: <div className="arc-tiers"><div className="arc-tier"><span className="k" title="a fast tier new files land on before the pool">Landing</span><span className="arc-tier-v"><span className="none">none — writes go straight into the pool's RAM buffer</span></span></div></div>}
<div className="arc-sparks">
<div>
<div className="muted small">RAM write buffer · {fmtBytes(w.dirty)} of {fmtBytes(w.limit)}</div>
<Sparkline series={[{ name: 'waiting in RAM', values: h['zfs.write.dirty'] ?? [], color: cssVar('--s1') }]} max={w.limit || undefined} format={fmtBytes} height={44} />
</div>
<div>
<div className="muted small">Writes · {fmtRate(p.write_bps)} to disk · {fmtRate(p.logical_bps)} from programs</div>
<Sparkline series={[
{ name: 'to disk', values: h[b + 'write_bps'] ?? [], color: cssVar('--s1') },
{ name: 'from programs', values: h[b + 'logical_bps'] ?? [], color: cssVar('--ok') },
]} format={fmtRate} height={44} />
</div>
</div>
<div className="props rows" style={{ marginTop: 8 }}>
<div><span className="k">In RAM</span><span title="dirty data: written by programs, not yet on the disks. For all pools together — ZFS keeps no live per-pool figure">{fmtBytes(w.dirty)} waiting · holds up to {fmtBytes(w.limit)} {w.limit < w.dirty_max ? `(half the ARC target; zfs_dirty_data_max is ${fmtBytes(w.dirty_max)})` : '(zfs_dirty_data_max)'}</span></div>
<div><span className="k">Throttle</span><span title="past zfs_delay_min_dirty_percent of the buffer, ZFS makes each write wait a little so the disks can catch up; at the limit, writes stop until a sync frees room">
{w.over_max_per_sec > 0 ? <span className="warn-text">{fmtN(w.over_max_per_sec)}/s writes stopped at the limit</span> : w.delayed_per_sec > 0 ? <span className="warn-text">slowing {fmtN(w.delayed_per_sec)} writes/s — the disks are the limit now</span> : 'not slowing anything'}
<span className="muted"> · starts at {w.delay_min_pct}% ({fmtBytes(delayAt)})</span></span></div>
<div><span className="k">To disk</span><span>{fmtRate(p.write_bps)} written by syncs{ratio > 0 ? ` · ${fmtRate(p.logical_bps)} from programs (${ratio >= 1 ? `${ratio.toFixed(2)}x compressed` : 'parity and metadata included'})` : ''}<span className="muted"> · a sync at least every {w.txg_timeout_sec} s, sooner as the buffer fills</span></span></div>
</div>
</section>
)
}
function LandingBlockWithHist({ l, h, b }: { l: Landing; h: WHist; b: string }) {
return <LandingBlockInner l={l} pending={h[b + 'landing.pending_bytes'] ?? []} inn={h[b + 'landing.in_bps'] ?? []} drain={h[b + 'landing.drain_bps'] ?? []} />
}
function ARCSection() {
const [v, setV] = useState<ARCView | null>(null)
const [hist, setHist] = useState<{ size: number[]; hit: number[] }>({ size: [], hit: [] })
const [whist, setWhist] = useState<WHist>({})
const [err, setErr] = useState('')
const rate = useRefreshMs()
useEffect(() => {
let stop = false
const poll = async () => {
try { const r = await api.zfsARC(); if (!stop) { setV(r); setErr('') } } catch (e) { if (!stop) setErr(e instanceof Error ? e.message : 'failed') }
let r: ARCView | null = null
try { r = await api.zfsARC(); if (!stop) { setV(r); setErr('') } } catch (e) { if (!stop) setErr(e instanceof Error ? e.message : 'failed') }
try {
const hs = await fetchHistories('zfs.arc.size,zfs.arc.hit_pct')
if (!stop) setHist({ size: hs.find((h) => h.id === 'zfs.arc.size')?.v ?? [], hit: hs.find((h) => h.id === 'zfs.arc.hit_pct')?.v ?? [] })
const extra = r?.writes ? wIds(r.writes) : []
const hs = await fetchHistories(['zfs.arc.size', 'zfs.arc.hit_pct', ...extra].join(','))
if (!stop) {
setHist({ size: hs.find((h) => h.id === 'zfs.arc.size')?.v ?? [], hit: hs.find((h) => h.id === 'zfs.arc.hit_pct')?.v ?? [] })
setWhist(Object.fromEntries(hs.filter((h) => extra.includes(h.id)).map((h) => [h.id, h.v])))
}
} catch { /* the sparklines stay empty; the numbers still show */ }
}
poll()
@@ -444,7 +575,262 @@ function ARCSection() {
<span><b>All reads</b></span><span className="num"><b>{fmtPct(hitPct(L.Hits, L.Misses))}</b></span><span className="num">{fmtN(L.Hits)}</span><span className="num">{fmtN(L.Misses)}</span>
</div>
</section>
{d.pools.length > 0 && <div className="cards" style={{ marginTop: 8 }}>{d.pools.map((p) => <PoolTiers key={p.pool} p={p} v={v} />)}</div>}
{d.pools.length > 0 && <div className="cards full" style={{ marginTop: 8 }}>{d.pools.map((p) => <PoolTiers key={p.pool} p={p} v={v} />)}</div>}
{v.writes && v.writes.pools.length > 0 && (
<>
<div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Writes</h2><span className="spacer" />
<span className="muted small">what is waiting to reach each pool, and how fast it is getting there</span>
</div>
<div className="cards full">{v.writes.pools.map((p) => <WritePath key={p.pool} w={v.writes!} p={p} h={whist} />)}</div>
</>
)}
</>
)
}
// ---- landing-tier checks: what has waited too long, and verify passes ----
//
// A verify pass reads the pool's copy of every file the mover moved back against the SHA-256
// the mover took of the landing copy, from its ledger. "New moves" checks what has moved since
// the last pass; "everything" re-reads the whole ledger. Passes run as their own process, the
// same one a schedule's cron line starts, so a console restart does not cut one short.
const vDescribe = (p: VerifyPolicy) => {
const when = p.every === 'custom' ? `on ${p.cron}`
: p.every === 'hourly' ? `every hour at :${(p.at.split(':')[1] ?? '00').padStart(2, '0')}`
: p.every === 'daily' ? `every day at ${p.at}`
: p.every === 'weekly' ? `every ${WEEKDAYS[p.weekday] ?? 'Sunday'} at ${p.at}`
: `on day ${p.day} of the month at ${p.at}`
return `${when} · ${p.scope === 'all' ? 'everything in the ledger' : 'what moved since the last pass'}`
}
const vOK = (r: VerifyResult) => !r.error && r.mismatched === 0 && r.gone === 0 && r.unreadable === 0
const secs = (a: string, b: string) => (new Date(b).getTime() - new Date(a).getTime()) / 1000
function VerifySummary({ r }: { r: VerifyResult }) {
if (r.error) return <span className="warn-text" style={{ whiteSpace: 'normal' }}>{r.error}</span>
const extra = [r.changed && `${fmtN(r.changed)} edited since (not compared)`, r.waiting && `${fmtN(r.waiting)} waiting to move again`, r.unhashed && `${fmtN(r.unhashed)} without a hash`].filter(Boolean).join(' · ')
return (
<span>
{vOK(r)
? <>{r.checked === 0 ? 'nothing new to check' : <><b>{fmtN(r.matched)}</b> of {fmtN(r.checked)} files match</>}</>
: <span className="warn-text">{[r.mismatched && `${fmtN(r.mismatched)} DIFFER`, r.gone && `${fmtN(r.gone)} not on the pool`, r.unreadable && `${fmtN(r.unreadable)} unreadable`].filter(Boolean).join(' · ')} · {fmtN(r.matched)} match</span>}
<span className="muted"> · {fmtBytes(r.bytes_read)} read in {dur(secs(r.started, r.finished)) === '—' ? '0 s' : dur(secs(r.started, r.finished))} · {r.scope === 'all' ? 'everything' : 'new moves'} · {ago(r.finished)}</span>
{extra && <span className="muted"> · {extra}</span>}
</span>
)
}
function Issues({ r }: { r: VerifyResult }) {
const [open, setOpen] = useState(false)
if (!r.issues?.length) return null
return (
<div style={{ marginTop: 6 }}>
<button className="btn" onClick={() => setOpen(!open)}>{open ? 'Hide' : 'Show'} the {r.issues.length} file{r.issues.length === 1 ? '' : 's'} to look at</button>
{open && <div className="snap-list" style={{ marginTop: 6 }}>{r.issues.map((i) => (
<div key={i.path + i.kind} className="snap">
<span className={'badge ' + (i.kind === 'mismatch' ? 'bad' : 'warn')}>{i.kind === 'mismatch' ? 'differs' : i.kind === 'gone' ? 'not on pool' : 'unreadable'}</span>
<span className="mono snap-name" title={i.path}>{i.path}</span>
<span className="muted snap-when">moved {new Date(i.moved).toLocaleString()}</span>
<span className="num muted">{fmtBytes(i.size)}</span>
</div>
))}</div>}
{open && r.gone > 0 && <div className="muted small" style={{ marginTop: 4 }}>"Not on the pool" is a file that was moved and is no longer there — deleted or renamed since, most often. The pass cannot tell that from a loss, so it lists them.</div>}
</div>
)
}
function VerifyScheduleCard({ p, cron, next, last, onChanged, onEdit }: { p: VerifyPolicy; cron: string; next?: string; last: VerifyResult | null; onChanged: () => void; onEdit: () => void }) {
const run = async () => {
try { await api.runVerifySchedule(p.id); toast('verify pass started'); onChanged() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
}
const remove = async () => {
if (!await askYes(`Remove the verify schedule "${p.name}"? Passes it already ran stay in the history.`, 'Remove')) return
try { await api.deleteVerifySchedule(p.id); toast('schedule removed'); onChanged() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
}
const toggle = async () => {
try { await api.saveVerifySchedule({ ...p, enabled: !p.enabled }); onChanged() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
}
return (
<section className="card">
<div className="sub-head">
<span className={'dot ' + (last && !vOK(last) ? 'bad' : p.enabled ? 'ok' : '')} />
<span className="dname">{p.name || 'verify'}</span>
{!p.enabled && <span className="badge">paused</span>}
<span className="spacer" />
<span className="muted small">{p.enabled ? soon(next) : 'not scheduled'}</span>
</div>
<div className="props rows" style={{ marginTop: 6 }}>
<div><span className="k">When</span><span>{vDescribe(p)}</span></div>
<div><span className="k">Cron</span><span className="mono small">{cron}</span></div>
<div><span className="k">Last pass</span>{last ? <VerifySummary r={last} /> : <span>never run</span>}</div>
</div>
<div className="svc-actions">
<Btn className="btn on" onClick={run}>Run now</Btn>
<button className="btn" onClick={onEdit}>Edit</button>
<Btn className="btn" onClick={toggle}>{p.enabled ? 'Pause' : 'Resume'}</Btn>
<Btn className="btn danger" onClick={remove}>Remove</Btn>
</div>
</section>
)
}
const VBLANK: Partial<VerifyPolicy> = { name: '', every: 'daily', at: '05:10', weekday: 0, day: 1, scope: 'new', enabled: true }
function VerifyDialog({ p, onClose, onSaved }: { p: Partial<VerifyPolicy>; onClose: () => void; onSaved: () => void }) {
const [f, setF] = useState<Partial<VerifyPolicy>>({ ...VBLANK, ...p })
const [err, setErr] = useState('')
useEscape(onClose)
const set = (k: keyof VerifyPolicy, v: unknown) => setF((x) => ({ ...x, [k]: v }))
const save = async (e: FormEvent) => {
e.preventDefault(); setErr('')
try { await api.saveVerifySchedule({ ...f, name: (f.name || '').trim() || (f.scope === 'all' ? 'verify everything' : 'verify new moves') }); onSaved() }
catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
}
return (
<div className="modal-back" onClick={onClose}>
<form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
<h3>{f.id ? 'Edit verify schedule' : 'New verify schedule'}</h3>
<label className="muted">Name — what this schedule is for
<input value={f.name} onChange={(e) => set('name', e.target.value)} placeholder="nightly verify" maxLength={64} />
</label>
<label className="muted">What to check
<span className="chips">
<button type="button" className={'chip' + (f.scope === 'new' ? ' on' : '')} onClick={() => set('scope', 'new')}>new moves</button>
<button type="button" className={'chip' + (f.scope === 'all' ? ' on' : '')} onClick={() => set('scope', 'all')}>everything</button>
</span>
<span className="small">{f.scope === 'all' ? 'Re-reads every file the mover has ever moved from the pool. Thorough, and as slow as the ledger is large.' : 'Reads back what has moved since the last pass — every file checked once, soon after it lands.'}</span>
</label>
<label className="muted">How often
<span className="chips">
{(['hourly', 'daily', 'weekly', 'monthly', 'custom'] as const).map((k) => (
<button type="button" key={k} className={'chip' + (f.every === k ? ' on' : '')} onClick={() => set('every', k)}>{k}</button>
))}
</span>
</label>
{f.every === 'custom' ? (
<label className="muted">Cron expression — minute hour day month weekday
<input className="mono" value={f.cron ?? ''} onChange={(e) => set('cron', e.target.value)} placeholder="0 */6 * * *" spellCheck={false} required />
</label>
) : (
<label className="muted">At
<input type="time" value={f.at} onChange={(e) => set('at', e.target.value)} />
</label>
)}
{f.every === 'weekly' && (
<label className="muted">On
<select value={f.weekday} onChange={(e) => set('weekday', Number(e.target.value))}>
{WEEKDAYS.map((d, i) => <option key={d} value={i}>{d}</option>)}
</select>
</label>
)}
{f.every === 'monthly' && (
<label className="muted">Day of the month — 1 to 28, so every month has one
<input type="number" min={1} max={28} value={f.day} onChange={(e) => set('day', Number(e.target.value))} />
</label>
)}
<label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
<input type="checkbox" checked={!!f.enabled} onChange={(e) => set('enabled', e.target.checked)} style={{ width: 'auto' }} />
Enabled
</label>
{err && <div className="error">{err}</div>}
<div className="svc-actions">
<button type="submit" className="btn on">Save</button>
<span className="spacer" />
<button type="button" className="btn" onClick={onClose}>Cancel</button>
</div>
</form>
</div>
)
}
function LandingChecks() {
const [c, setC] = useState<LandingCheck | null>(null)
const [err, setErr] = useState('')
const [showStale, setShowStale] = useState(false)
const [editing, setEditing] = useState<Partial<VerifyPolicy> | null>(null)
const rate = useRefreshMs()
const load = useCallback(async () => {
try { setC(await api.zfsLanding()); setErr('') } catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
}, [])
useEffect(() => { load() }, [load])
const running = !!c?.running
useEffect(() => { const t = setInterval(load, running ? 2000 : Math.max(10000, rate * 5)); return () => clearInterval(t) }, [load, rate, running])
if (err && !c) return <div className="error">{err}</div>
if (!c || (c.tiers.length === 0 && c.policies.length === 0 && c.history.length === 0)) return null
const verify = async (scope: 'new' | 'all') => {
if (scope === 'all' && !await askYes('Re-read every file the mover has ever moved, from the pool, and check each against its hash? It reads as much as the ledger holds.', 'Verify everything')) return
try { await api.landingVerify(scope); toast('verify pass started'); setTimeout(load, 800) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
}
const last = c.history[0]
const hours = Math.round(c.stale_after_sec / 3600)
const r = c.running
return (
<>
<div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Landing tier checks</h2><span className="spacer" />
<span className="muted small">did everything that landed reach the pool, intact?</span>
</div>
<div className="cards full">
{c.tiers.length === 0 && <div className="notice">No landing mover is reporting right now (nothing in <span className="mono">/run/landing-mover/</span>), so there is nothing to check. Past passes are below.</div>}
{c.tiers.map((t) => (
<section className="card" key={t.src}>
<div className="sub-head">
<span className={'dot ' + (last && !vOK(last) ? 'bad' : t.stale_count ? 'warn' : 'ok')} />
<span className="dname">{t.dst}</span>
<span className="badge">landing checks</span>
<span className="spacer" />
<span className="muted small mono">{t.src} → {t.dst}</span>
</div>
<div className="props rows" style={{ marginTop: 6 }}>
<div><span className="k">Stale</span><span>
{t.stale_count
? <><span className="warn-text">{fmtN(t.stale_count)} file{t.stale_count === 1 ? '' : 's'}, {fmtBytes(t.stale_bytes)}, waiting over {hours} h</span>{' '}<button className="btn" onClick={() => setShowStale(!showStale)}>{showStale ? 'Hide' : 'Show'}</button></>
: `nothing has waited on the landing tier over ${hours} h`}
</span></div>
<div><span className="k">Last pass</span>{last ? <VerifySummary r={last} /> : <span>never verified</span>}</div>
<div><span className="k">Covered</span><span>{t.cursor ? `every file moved up to ${new Date(t.cursor).toLocaleString()} has been checked at least once` : 'nothing checked yet'}<span className="muted"> · ledger {t.ledger}</span></span></div>
</div>
{showStale && t.stale.length > 0 && (
<div className="snap-list" style={{ marginTop: 6 }}>{t.stale.map((f) => (
<div key={f.path} className="snap">
<span className="mono snap-name" title={f.path}>{f.path}</span>
<span className="muted snap-when">landed {new Date(f.mtime).toLocaleString()}</span>
<span className="muted snap-exp">waiting {dur((Date.now() - f.mtime) / 1000)}</span>
<span className="num muted">{fmtBytes(f.size)}</span>
</div>
))}{t.stale_count > t.stale.length && <div className="muted small">and {fmtN(t.stale_count - t.stale.length)} more</div>}</div>
)}
{r && (
<div style={{ marginTop: 8 }}>
<div className="sub-head"><span className="muted small">Verifying {r.scope === 'all' ? 'everything' : 'new moves'} · {fmtN(r.done)} of {fmtN(r.total)} files · {fmtBytes(r.bytes_read)} of {fmtBytes(r.bytes_all)}</span><span className="spacer" /><span className="mono small muted" title={r.current}>{r.current}</span></div>
<Meter frac={r.bytes_all > 0 ? r.bytes_read / r.bytes_all : r.total > 0 ? r.done / r.total : 0} cls="" />
</div>
)}
{last && <Issues r={last} />}
<div className="svc-actions">
<Btn className="btn on" disabled={running} onClick={() => verify('new')}>{running ? 'Verifying…' : 'Verify new moves'}</Btn>
<Btn className="btn" disabled={running} onClick={() => verify('all')}>Verify everything</Btn>
</div>
{c.history.length > 1 && (
<details style={{ marginTop: 6 }}>
<summary className="muted small">Earlier passes</summary>
<div className="props rows" style={{ marginTop: 4 }}>{c.history.slice(1).map((h, i) => (
<div key={i}><span className="k">{new Date(h.finished).toLocaleString()}</span><VerifySummary r={h} /></div>
))}</div>
</details>
)}
</section>
))}
</div>
<div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Verify schedules</h2><span className="spacer" />
<Btn className="btn on" onClick={() => setEditing({ ...VBLANK })}>+ New schedule</Btn>
</div>
{c.policies.length === 0 && <div className="notice">No verify runs on a schedule. A daily pass of new moves checks every file once, soon after it lands.</div>}
<div className="cards full">{c.policies.map(({ policy, cron, next, last }) => (
<VerifyScheduleCard key={policy.id} p={policy} cron={cron} next={next} last={last} onChanged={load} onEdit={() => setEditing(policy)} />
))}</div>
{editing && <VerifyDialog p={editing} onClose={() => setEditing(null)} onSaved={() => { setEditing(null); load() }} />}
</>
)
}
@@ -475,19 +861,20 @@ export default function ZFS() {
<div className="dash">
<div className="dash-bar"><h2 className="section">Pools</h2></div>
{zfs.pools.length === 0 && <div className="notice">ZFS is present but no pools are imported.</div>}
<div className="cards">{zfs.pools.map((p) => <PoolCard key={p.name} p={p} onChanged={load} />)}</div>
<div className="cards full">{zfs.pools.map((p) => <PoolCard key={p.name} p={p} onChanged={load} />)}</div>
<div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Cache</h2><span className="spacer" />
<span className="muted small">the ARC in RAM, and what each pool has in front of its disks</span>
</div>
<ARCSection />
<LandingChecks />
<div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Snapshot schedules</h2><span className="spacer" />
<Btn className="btn on" onClick={() => setEditing({ ...BLANK, dataset: zfs.pools[0]?.name ?? '' })}>+ New schedule</Btn>
</div>
{ov.external && <ExternalNotice x={ov.external} mine={mine} onAdopt={() => setEditing(adopt(ov.external!, zfs.pools[0]?.name ?? ''))} />}
{ov.policies.length === 0 && !ov.external && <div className="notice">Nothing is taking snapshots on a schedule. Snapshots are written the way zfSnap writes them, with the retention in the name, so anything already here is understood.</div>}
<div className="cards">{ov.policies.map(({ policy, cron, next, last }) => (
<div className="cards full">{ov.policies.map(({ policy, cron, next, last }) => (
<ScheduleCard key={policy.id} p={policy} cron={cron} next={next} last={last ?? {}} onChanged={load} onEdit={() => setEditing(policy)} />
))}</div>
+40 -10
View File
@@ -71,11 +71,11 @@ export const titleOf = (name: string | null, machines: MachineStatus[], hubHost:
export interface MachineStatus { name: string; url: string; fingerprint?: string; added: string; online: boolean; host?: string; version?: string; load?: string; uptime?: number; reboot_needed?: boolean; last_seen?: string; error?: string }
export interface Term { id: string; name: string; running: boolean; exit_code: number; created: string; clients: number }
export interface Temp { chip: string; label: string; c: number; max?: number; crit?: number }
export interface Temp { chip: string; label: string; c: number; max?: number; crit?: number; pkg?: number }
export interface Fan { chip: string; label: string; rpm: number }
export interface NetIf { iface: string; up: boolean; rx_bps: number; tx_bps: number; rx_total: number; tx_total: number }
export interface Disk { name: string; model?: string; size: number; read_bps: number; write_bps: number; util_pct: number; temp?: number }
export interface FS { mount: string; type: string; size: number; used: number; remote?: boolean; disk?: string }
export interface FS { mount: string; type: string; size: number; used: number; remote?: boolean; disk?: string; source?: string }
export interface UPS { driver: string; model: string; state: string; source: string; on_battery: boolean; in_v: number; out_v: number; battery_pct: number; runtime_min: number; load_w: number; load_pct: number; last_event?: string }
export interface Proc { pid: number; ppid: number; name: string; cmd?: string; user: string; cpu: number; gpu?: number; cpu_time: number; rss: number; threads: number; state: string; nice: number; started: number }
export interface MemInfo { total: number; used: number; avail: number; free?: number; cached?: number; buffers?: number }
@@ -86,11 +86,11 @@ export interface GPUClient { pid: number; name: string; usage: number; engines?:
export interface GPU {
name: string; driver: string; bus?: string; integrated?: boolean
usage: number; usage_src?: 'idle' | 'busy' | 'clients'; engines?: Record<string, number>
temp?: number; temp_src?: 'gpu' | 'package'; power_w?: number; power_cap_w?: number; fan_rpm?: number
temp?: number; temp_src?: 'gpu' | 'package'; power_w?: number; power_cap_w?: number; fan_rpm?: number; fan_pct?: number
mhz?: number; req_mhz?: number; max_mhz?: number; mem_used?: number; mem_total?: number
clients: GPUClient[]; clients_ok: boolean; stalled?: boolean
}
export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_power_w?: number; cpu_temp?: number; core_ids?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_power_w?: number; cpu_power_pkgs_w?: number[]; cpu_temp?: number; core_ids?: number[]; core_pkgs?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
export interface Sample {
t: number; interval_ms: number; cpu: number; cores: number[]; load: [number, number, number]
mem: { total: number; used: number; avail: number }; swap: { total: number; used: number; avail: number }
@@ -185,7 +185,6 @@ export interface SpaceNode { name: string; dir: boolean; size: number; apparent:
export interface SpaceTree extends SpaceNode { path: string; children?: SpaceTree[]; more_n?: number; more_size?: number; total: number }
export interface SpaceFile { path: string; size: number; apparent: number; mtime: number }
export interface KVM { id: string; name: string; url: string; machine: string; added: string }
// ---- the Network tab ----
export interface IfAddr { ip: string; prefix: number; v6?: boolean }
@@ -252,7 +251,37 @@ export interface ARCDetail {
dbuf: { Size: number; Target: number; Hits: number; Misses: number }
pools: PoolCaching[]
}
export interface ARCView { detail: ARCDetail; live: ARCLive | null }
export interface LandingMover {
state: 'idle' | 'moving' | 'holding' | 'stopped' | string; src: string; dst: string; pid: number; started_ms: number; updated_ms: number; stale: boolean
queue: number; settle: number; bwlimit: string; batch_files: number; batch_bytes: number; batch_start_ms: number; batch_first: string
current: string; current_size: number; partial: number; moved_files: number; moved_bytes: number
last_move: string; last_move_ms: number; last_error: string; last_error_ms: number
}
export interface Landing {
mount: string; branch: string; pool_branch: string; device?: string; fstype?: string; size: number; used: number
pending_files: number; pending_bytes: number; pending_partial?: boolean; oldest_ms?: number; counted_ms: number
in_bps: number; drain_bps: number; mover?: LandingMover
}
export interface PoolWrites { pool: string; write_bps: number; logical_bps: number; landing?: Landing }
export interface ZFSWrites {
dirty: number; dirty_max: number; delay_min_pct: number; limit: number; delayed_per_sec: number; over_max_per_sec: number; txg_timeout_sec: number
pools: PoolWrites[]
}
export interface VerifyPolicy { id: string; name: string; every: 'hourly' | 'daily' | 'weekly' | 'monthly' | 'custom'; at: string; weekday: number; day: number; cron?: string; scope: 'new' | 'all'; enabled: boolean }
export interface VerifyIssue { path: string; kind: 'mismatch' | 'gone' | 'unreadable'; moved: number; size: number; note?: string }
export interface VerifyResult {
id: string; scope: 'new' | 'all'; started: string; finished: string; src: string; dst: string; since: number
entries: number; checked: number; matched: number; mismatched: number; changed: number; gone: number; waiting: number; unreadable: number; unhashed: number
bytes_read: number; issues: VerifyIssue[]; error?: string; cursor: number
}
export interface VerifyProgress { id: string; scope: string; pid: number; started: string; updated: string; total: number; done: number; bytes_read: number; bytes_all: number; current: string }
export interface StaleFile { path: string; size: number; mtime: number }
export interface LandingCheck {
tiers: { src: string; dst: string; ledger: string; cursor: number; stale_count: number; stale_bytes: number; stale: StaleFile[] }[]
stale_after_sec: number; running: VerifyProgress | null; history: VerifyResult[]; cron_file: string
policies: { policy: VerifyPolicy; cron: string; next?: string; last: VerifyResult | null }[]
}
export interface ARCView { detail: ARCDetail; live: ARCLive | null; writes: ZFSWrites | null }
export interface FileEntry { name: string; type: 'dir' | 'file' | 'link' | 'other'; size: number; mode: string; perm: string; mtime: number; user: string; group: string; target?: string; broken?: boolean; preview?: 'image' | 'video' | 'audio' }
export interface PlayTrack { codec: string; profile?: string; pix_fmt?: string; width?: number; height?: number; fps?: number; channels?: number; spherical?: boolean }
@@ -330,12 +359,13 @@ export const api = {
storage: () => req<{ devices: StorageDevice[]; zfs: ZFSInfo; smartctl: boolean }>('GET', '/api/storage'),
smart: (dev: string, force = false) => req<SmartInfo>('GET', `/api/storage/smart/${dev}${force ? '?force=1' : ''}`),
smartTest: (dev: string, kind: 'short' | 'long') => req<void>('POST', `/api/storage/smart/${dev}/test/${kind}`),
// Always the hub's: it holds every machine's KVM, so the tab works when the machine does not.
kvms: () => req<KVM[]>('GET', '/api/kvms', undefined, true),
addKVM: (name: string, url: string, machine: string) => req<KVM>('POST', '/api/kvms', { name, url, machine }, true),
removeKVM: (id: string) => req<void>('DELETE', `/api/kvms/${encodeURIComponent(id)}`, undefined, true),
zfsOverview: () => req<ZFSOverview>('GET', '/api/zfs'),
zfsARC: () => req<ARCView>('GET', '/api/zfs/arc'),
zfsLanding: () => req<LandingCheck>('GET', '/api/zfs/landing'),
landingVerify: (scope: 'new' | 'all') => req<{ started: string }>('POST', '/api/zfs/landing/verify', { scope }),
saveVerifySchedule: (p: Partial<VerifyPolicy>) => req<VerifyPolicy>('PUT', '/api/zfs/landing/schedules', p),
deleteVerifySchedule: (id: string) => req<void>('DELETE', `/api/zfs/landing/schedules/${encodeURIComponent(id)}`),
runVerifySchedule: (id: string) => req<{ started: string }>('POST', `/api/zfs/landing/schedules/${encodeURIComponent(id)}/run`),
saveZfsSchedule: (p: Partial<ZFSPolicy>) => req<ZFSPolicy>('PUT', '/api/zfs/schedules', p),
deleteZfsSchedule: (id: string) => req<void>('DELETE', `/api/zfs/schedules/${encodeURIComponent(id)}`),
runZfsSchedule: (id: string) => req<{ taken: string; pruned: string[] }>('POST', `/api/zfs/schedules/${encodeURIComponent(id)}/run`),
+2 -2
View File
@@ -47,11 +47,11 @@ export function sampleOf(v: Values, cat: Catalog): Sample {
t: Date.now(), interval_ms: num(v, 'hamlab.sample_interval'),
cpu: num(v, 'cpu.usage'), cores, load: [num(v, 'load.1'), num(v, 'load.5'), num(v, 'load.15')],
mem: mem('mem'), swap: mem('swap'), uptime: num(v, 'uptime'),
temps: hw('temp.').map(({ id, s }) => ({ chip: s.labels!.chip, label: s.labels!.label, c: num(v, id), max: s.warn, crit: s.crit })),
temps: hw('temp.').map(({ id, s }) => ({ chip: s.labels!.chip, label: s.labels!.label, c: num(v, id), max: s.warn, crit: s.crit, pkg: s.labels!.package !== undefined ? Number(s.labels!.package) : undefined })),
fans: hw('fan.').map(({ id, s }) => ({ chip: s.labels!.chip, label: s.labels!.label, rpm: num(v, id) })),
net: [...groups(v, cat, 'net.', 'iface')].map(([iface, p]) => ({ iface, up: v[p + '.up'] === true, rx_bps: num(v, p + '.rx'), tx_bps: num(v, p + '.tx'), rx_total: num(v, p + '.rx_total'), tx_total: num(v, p + '.tx_total') })),
disks: [...groups(v, cat, 'disk.', 'device')].map(([name, p]) => ({ name, model: cat.get(p + '.size')?.labels?.model || undefined, size: num(v, p + '.size'), read_bps: num(v, p + '.read'), write_bps: num(v, p + '.write'), util_pct: num(v, p + '.util'), temp: num(v, p + '.temp') || undefined })),
fs: [...groups(v, cat, 'fs.', 'mount')].map(([mount, p]) => ({ mount, type: cat.get(p + '.size')?.labels?.fstype ?? '', size: num(v, p + '.size'), used: num(v, p + '.used'), remote: cat.get(p + '.size')?.labels?.remote === '1', disk: cat.get(p + '.size')?.labels?.disk })),
fs: [...groups(v, cat, 'fs.', 'mount')].map(([mount, p]) => ({ mount, type: cat.get(p + '.size')?.labels?.fstype ?? '', size: num(v, p + '.size'), used: num(v, p + '.used'), remote: cat.get(p + '.size')?.labels?.remote === '1', disk: cat.get(p + '.size')?.labels?.disk, source: cat.get(p + '.size')?.labels?.source })),
ups: v['ups.battery'] === undefined ? null : {
driver: cat.get('ups.last_event')?.labels?.driver ?? '', model: cat.get('ups.battery')?.labels?.model ?? '', state: str(v, 'ups.state'), source: str(v, 'ups.source'),
on_battery: v['ups.on_battery'] === true, in_v: num(v, 'ups.in_v'), out_v: num(v, 'ups.out_v'), battery_pct: num(v, 'ups.battery'), runtime_min: num(v, 'ups.runtime'),
+17
View File
@@ -0,0 +1,17 @@
// Which threads belong to which CPU, on a machine with more than one.
//
// The kernel numbers every thread of the machine in one run, and on a two-socket board with
// hyper-threading it deals them out a socket at a time and then goes round again: cpu 0–9 are
// CPU 1's cores, 10–19 CPU 2's, 20–29 the second threads of CPU 1's, 30–39 of CPU 2's. Listed
// under its CPU by that number, a 20-thread processor reads "C0…C9, C20…C29" and the one
// beside it "C10…C19, C30…C39" — neither looks like twenty of anything. So each CPU's threads
// are counted from 0 under it, in the kernel's order, the way a one-socket machine's are: the
// first half are its cores, the second half their second threads.
//
// `pkgs[i]` is the socket logical CPU `i` is on. The answer is, for each socket (ascending),
// the logical CPUs on it (ascending); a thread's number under its CPU is its place in that list.
export function socketThreads(pkgs: number[]): { pkg: number; cpus: number[] }[] {
const by = new Map<number, number[]>()
pkgs.forEach((pk, i) => { const l = by.get(pk); if (l) l.push(i); else by.set(pk, [i]) })
return [...by.entries()].sort((a, b) => a[0] - b[0]).map(([pkg, cpus]) => ({ pkg, cpus }))
}
+51 -17
View File
@@ -5,6 +5,10 @@
--s1: #3987e5; --s2: #c98500; --track: rgba(57, 135, 229, .18);
--btn-fg: #111; --term-bg: var(--term-bg); --font: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, "Liberation Mono", monospace;
--header-h: 44px; --footer-h: 28px; --rail-w: 220px; --tabs-h: 34px;
/* iOS 26+ draws a soft blur under the status bar of a Home Screen app that reaches ~16pt
below it (hamlet docs/gotchas.md). Whatever starts at the safe area starts below that band
too. Zero where there is no status bar to clear: desktop, a Safari tab. */
--blur: min(env(safe-area-inset-top), 16px);
}
* { box-sizing: border-box; }
html, body, #root { height: 100%; margin: 0; }
@@ -30,7 +34,7 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
.svc-list > .error, .cron-list > .error { margin: 8px 12px; }
/* layout */
.app { height: 100%; display: flex; flex-direction: column; padding-top: env(safe-area-inset-top); }
.app { height: 100%; display: flex; flex-direction: column; padding-top: calc(env(safe-area-inset-top) + var(--blur)); }
.header { height: var(--header-h); display: flex; align-items: center; gap: 12px; padding: 0 12px; background: var(--bg2); border-bottom: 1px solid var(--border); }
.brand { font-weight: 700; color: var(--accent); letter-spacing: .5px; }
.host { color: var(--muted); font-family: ui-monospace, monospace; font-size: 13px; }
@@ -95,6 +99,7 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
.dash > * { flex-shrink: 0; }
.kpis { display: grid; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); gap: 12px; }
.cards { display: grid; grid-template-columns: repeat(auto-fill, minmax(320px, 1fr)); gap: 12px; align-items: start; }
.cards.full { grid-template-columns: minmax(0, 1fr); } /* one card per row, the width of the page */
.tile, .card { background: var(--bg2); border: 1px solid var(--border); border-radius: 8px; padding: 12px; min-width: 0; }
.tile .label, .card h2 { margin: 0 0 6px; font-size: 12px; font-weight: 600; color: var(--muted); text-transform: uppercase; letter-spacing: .5px; }
.tile .value { font-size: 22px; font-weight: 600; font-variant-numeric: tabular-nums; line-height: 1.2; }
@@ -192,6 +197,11 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
.props span:not(.k) { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } .mono { font-family: ui-monospace, monospace; font-size: 12px; }
/* One property per row, label above its value. For panes like Inspect, whose values are
long paths and lists — side by side they were half truncated and read as columns. */
/* One property per row, label beside its value, the value wrapping rather than cut off: for
sentences, like the ZFS tab's write path, that columns would truncate. */
.props.rows { grid-template-columns: minmax(0, 1fr); gap: 3px 0; }
.props.rows .k { width: 96px; }
.props.rows span:not(.k) { overflow: visible; white-space: normal; overflow-wrap: anywhere; }
.props.stacked { grid-template-columns: 1fr; gap: 10px 0; }
.props.stacked > div { flex-direction: column; gap: 2px; }
.props.stacked .k { width: auto; font-size: 11px; letter-spacing: .04em; text-transform: uppercase; }
@@ -264,7 +274,7 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
than what is left, and scrolls inside itself. Its parts keep their height: in a column that
may shrink, anything with its own scrollbar (a preview, a list) is squeezed to a sliver
instead, and the dialog never scrolls at all. */
.modal-back { position: fixed; inset: 0; z-index: 50; background: rgba(0,0,0,.6); display: flex; align-items: center; justify-content: center; padding: max(16px, env(safe-area-inset-top)) max(16px, env(safe-area-inset-right)) max(16px, env(safe-area-inset-bottom)) max(16px, env(safe-area-inset-left)); }
.modal-back { position: fixed; inset: 0; z-index: 50; background: rgba(0,0,0,.6); display: flex; align-items: center; justify-content: center; padding: max(16px, calc(env(safe-area-inset-top) + var(--blur))) max(16px, env(safe-area-inset-right)) max(16px, env(safe-area-inset-bottom)) max(16px, env(safe-area-inset-left)); }
.modal { --modal-bg: var(--bg2); width: min(420px, 100%); max-height: 100%; overflow-y: auto; overscroll-behavior: contain; background: var(--modal-bg); border: 1px solid var(--border); border-radius: 10px; padding: 16px; display: flex; flex-direction: column; gap: 12px; }
.modal > * { flex-shrink: 0; }
/* a long dialog's buttons (and what went wrong) stay in reach while the rest scrolls */
@@ -368,10 +378,15 @@ h2.section { margin: 0; font-size: 13px; font-weight: 600; color: var(--muted);
.preview { flex: 1; display: flex; align-items: center; justify-content: center; overflow: auto; padding: 12px; }
.preview img { max-width: 100%; max-height: 100%; object-fit: contain; }
.preview video { max-width: 100%; max-height: 100%; background: #000; border-radius: 4px; outline: none; }
.preview.pic { position: relative; touch-action: pan-y pinch-zoom; }
.step-edge { position: absolute; top: 50%; transform: translateY(-50%); width: 44px; height: 72px; border: 0; border-radius: 6px; background: rgba(0, 0, 0, .35); color: #fff; font-size: 34px; line-height: 1; cursor: pointer; opacity: .55; }
.step-edge:hover { opacity: 1; } .step-edge.prev { left: 8px; } .step-edge.next { right: 8px; }
.btn.step:disabled { opacity: .4; }
/* the video player (Player.tsx): Ham Share's, in the viewer; full screen is the browser's, or
on an iPhone the whole window */
.vp { position: relative; flex: 1; min-height: 240px; background: #000; color: #fff; overflow: hidden; user-select: none; -webkit-user-select: none; outline: none; }
.vp.max { position: fixed; inset: 0; z-index: 1000; }
.vp.max .vp-note { top: calc(env(safe-area-inset-top) + var(--blur) + 14px); }
.vp-v { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: contain; background: #000; }
.vp-hidden { opacity: 0; pointer-events: none; }
.vp-sphere { touch-action: none; cursor: grab; }
@@ -779,6 +794,11 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
.t-cores-head b { color: var(--fg); font-weight: 400; overflow: hidden; text-overflow: ellipsis; }
.t-core { display: flex; align-items: center; gap: 6px; }
.t-core-n { width: 30px; } .t-core-p { width: 34px; text-align: right; }
/* a two-socket board: a row per socket — arrow, name, watts — with its threads indented under it */
.t-cores.multi .t-core-n { width: 50px; } .t-cores.multi .t-core:not(.t-sock) .t-core-n { padding-left: 14px; box-sizing: border-box; }
.t-sock { color: var(--fg); margin-top: 2px; cursor: pointer; user-select: none; }
.t-fold { display: inline-block; width: 12px; color: var(--muted); }
.t-core-w { width: 40px; flex-shrink: 0; text-align: right; color: var(--muted); }
.t-core-g { flex: 1; color: var(--ok); letter-spacing: 0; overflow: hidden; white-space: nowrap; }
.t-core .tm { flex: 1; } .t-core-t { flex: 0 0 62px; display: flex; } .t-core-c { width: 42px; text-align: right; color: var(--t-blue); }
.t-load { display: flex; margin-top: 2px; }
@@ -879,6 +899,8 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
.arc-tier { display: flex; gap: 8px; min-width: 0; } .arc-tier .k { color: var(--muted); width: 80px; flex-shrink: 0; cursor: help; }
.arc-tier-v { display: flex; flex-direction: column; gap: 1px; min-width: 0; } .arc-tier-v > span { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.arc-tier-v .none { color: var(--muted); white-space: normal; }
.landing-tier { border: 1px solid var(--border); border-radius: 8px; padding: 8px 10px; margin: 4px 0 6px; font-size: 13px; }
.landing-tier .props .badge { margin-right: 2px; }
/* ---- Space: what is using a disk ---- */
.space { position: absolute; inset: 0; display: flex; flex-direction: column; background: var(--bg); min-height: 0; }
@@ -946,21 +968,33 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
.modal select, .modal input[type=time], .modal input[type=number] { padding: 9px 10px; background: var(--bg); border: 1px solid var(--border); border-radius: 6px; color: var(--fg); font-size: 16px; }
.modal .chips { margin-top: 2px; }
/* ---- KVM: a device's own web page, framed ---- */
.kvm { position: absolute; inset: 0; display: flex; flex-direction: column; background: var(--bg); min-height: 0; }
.kvm-bar { display: flex; align-items: center; gap: 8px; padding: 8px 12px; border-bottom: 1px solid var(--border); background: var(--bg2); flex-wrap: wrap; }
.kvm-bar .kvm-url { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; max-width: 28ch; }
.kvm-frame { flex: 1; min-height: 0; background: #000; position: relative; }
.kvm-frame iframe { width: 100%; height: 100%; border: 0; display: block; background: #000; }
.kvm-frame.full { background: #000; }
.kvm-hint { padding: 6px 12px; border-top: 1px solid var(--border); }
@media (max-width: 800px) { .kvm-bar .kvm-url { display: none; } }
.kvm-none { text-align: center; } .kvm-none p { margin: 4px 0; }
/* No true fullscreen for a page element (Safari on iOS): fill the viewport in-page instead. */
.kvm-frame.expanded { position: fixed; inset: 0; z-index: 60; }
.kvm-frame.expanded iframe { height: 100%; }
.kvm-shrink { position: absolute; right: max(10px, env(safe-area-inset-right)); bottom: max(10px, env(safe-area-inset-bottom)); z-index: 61; background: var(--bg2); border: 1px solid var(--border); opacity: .75; }
.kvm-shrink:hover, .kvm-shrink:active { opacity: 1; }
.updot { margin-left: 6px; font-size: 11px; font-weight: 700; }
.updot.behind { color: var(--s2); } .updot.unknown { color: var(--muted); }
/* a phone: the time goes above the seek bar, so the buttons fit in one row */
.vp-time.above { display: none; }
@media (max-width: 560px) {
.vp-time.above { display: block; padding: 0 10px 2px; }
.vp-time.inrow { display: none; }
}
/* No theme touches playback, whatever it draws. A theme's full-window overlays (the Retro
scanlines today, any other tomorrow) are off while a player is open; the picture is never
filtered or blended; the player's text is plain, not the theme's lettering. Only the accent
colour of its controls follows the theme. */
html:has(.vp)::before, html:has(.vp)::after, body:has(.vp)::before, body:has(.vp)::after,
.app:has(.vp)::before, .app:has(.vp)::after, #root:has(.vp)::before, #root:has(.vp)::after { display: none !important; }
.vp { isolation: isolate; filter: none !important; mix-blend-mode: normal !important; }
.vp video, .vp canvas { filter: none !important; mix-blend-mode: normal !important; }
.vp { font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; font-weight: 400; letter-spacing: normal; text-shadow: none; }
.vp button, .vp input, .vp select { font-family: inherit !important; }
/* volume: the speaker mutes; the slider beside it (not on iOS, which only mutes) sets the level */
.vp-vol { display: inline-flex; align-items: center; }
.vp-vol input { width: 72px; height: 24px; margin: 0 6px 0 0; padding: 0; background: transparent; border: 0; -webkit-appearance: none; appearance: none; cursor: pointer; }
.vp-vol input::-webkit-slider-runnable-track { height: 4px; border-radius: 2px; background: linear-gradient(to right, #fff var(--v), rgba(255, 255, 255, .3) var(--v)); }
.vp-vol input::-moz-range-track { height: 4px; border-radius: 2px; background: linear-gradient(to right, #fff var(--v), rgba(255, 255, 255, .3) var(--v)); }
.vp-vol input::-webkit-slider-thumb { -webkit-appearance: none; width: 12px; height: 12px; margin-top: -4px; border-radius: 50%; background: #fff; border: 0; }
.vp-vol input::-moz-range-thumb { width: 12px; height: 12px; border-radius: 50%; background: #fff; border: 0; }
@media (max-width: 560px) { .vp-vol input { display: none; } }