Compare commits
31
Commits
d1b3ef7efd
...
main
@@ -16,7 +16,7 @@ stub tab or a roadmap item.
|
||||
| Service | `hamlab.service`, `User=core`, `Restart=always`, binary at `/usr/local/bin/hamlab` | Same pattern as `calorie-counter.service`. Runs as `core` so `sudo` works in the terminal without a password, as on the SSH path. |
|
||||
| Listen | `10.0.0.234:8100` (free) | NPM is a container, so loopback-only won't do. Firewalling to NPM is a later hardening step. |
|
||||
| Data | `/home/core/apps/hamlab/` — `config.yaml`, `state.db` (SQLite) | `apps/` is the nightly rsync target to `grid`, so it's backed up for free. |
|
||||
| Ingress | NPM proxy host `hamlab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
|
||||
| Ingress | NPM proxy host `lab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
|
||||
| Repo | Gitea `mitchell/hamlab`, private, `main` | Created 2026-09-17. |
|
||||
| Guest logins | A **guest** is an account like any other, with `guest: true` and an expiry: `login`, sessions and prefs need no special case, and **it may do everything an account may do**. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. | Built 2026-09-27. A read-only version was built first and **cut the same day**: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is *an account*: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to. |
|
||||
| Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
|
||||
@@ -37,7 +37,7 @@ stub tab or a roadmap item.
|
||||
│ Cron │ │
|
||||
│ … │ │
|
||||
├──────────────┴───────────────────────────────────────────────┤
|
||||
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev │
|
||||
│ footer: v0.1.0 · up 12d · lab.bigham.dev │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -116,7 +116,7 @@ hamlab/
|
||||
- Mobile drawer, key bar, fit-on-rotate.
|
||||
- `manifest.webmanifest`, icons (Ham logo on black, same recipe as HamTV), service
|
||||
worker caching the app shell only (never `/api`).
|
||||
- NPM proxy host for `hamlab.bigham.dev`; verify WS through NPM; force-SSL/HSTS/HTTP2.
|
||||
- NPM proxy host for `lab.bigham.dev`; verify WS through NPM; force-SSL/HSTS/HTTP2.
|
||||
- Stub groups for Services / Cron / ZFS / Dashboard.
|
||||
|
||||
**Done when:** from the phone, open the PWA, log in, open two terminals, run
|
||||
@@ -417,39 +417,9 @@ something long in one, switch away and back, refresh — it's all still there.
|
||||
snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it
|
||||
had taken left untouched and fully readable here — which is what adopting its naming
|
||||
bought. See `hamlet/docs/hosts.md`.
|
||||
- **KVM tab** — **shipped 2026-09-24** (`internal/api/kvm.go`, `KVM.tsx`, `config.KVM`).
|
||||
Paste a device's address; its own web page appears in the tab. **Nothing in it knows
|
||||
anything about any particular KVM** — the first device here is a Sipeed NanoKVM on `grid`
|
||||
at `10.0.0.217`, but none of that is in the code.
|
||||
**Why it proxies rather than framing the device directly.** Ham Lab is served over TLS and
|
||||
a KVM usually is not, so a bare `http://…` iframe is blocked as mixed content before it
|
||||
loads. Serving the device from Ham Lab's own origin (`/kvm/<id>/…`) fixes that, puts it
|
||||
behind **Ham Lab's** login rather than a second one, and makes the iframe same-origin.
|
||||
**The general problem, and the answer.** A device UI asks for its assets by absolute path
|
||||
— `/assets/app.js`, `/api/whatever` — which no path prefix can rewrite away, and rewriting
|
||||
bodies per device would not be general. So the prefix is only the entry point: any request
|
||||
Ham Lab does not serve itself is routed to whichever KVM the asking page belongs to, read
|
||||
from the **Referer**, falling back to a **cookie** when there is none (a WebSocket upgrade).
|
||||
Ham Lab's own routes always match first, so a device can never shadow the console, and the
|
||||
fallback is gated on a live session. The device never receives Ham Lab's session cookie,
|
||||
and its `X-Frame-Options`/CSP/HSTS are dropped on the way out since Ham Lab is the origin
|
||||
now. Redirects to the device's root are rewritten back through the prefix.
|
||||
**One catch found in testing:** Ham Lab's own `frame-ancestors 'none'` forbids framing even
|
||||
from the same origin, so `/kvm/` paths send `'self'` instead. Without that the iframe was
|
||||
refused before it loaded.
|
||||
**The keyboard** needs less than it sounds: a page cannot forward keys into an iframe and
|
||||
does not need to, since one with focus receives them directly. Fullscreen plus
|
||||
`navigator.keyboard.lock()` is what takes Escape, Tab and Ctrl-W back from the browser;
|
||||
that is Chromium-only, and the tab says which version you are getting rather than
|
||||
pretending.
|
||||
**One KVM per machine, and the tab follows the machine switcher** (corrected the same day:
|
||||
the first cut was a hub-side list you pointed at machines, which had it backwards). Switch
|
||||
to `grid` and you get grid's; adding a second for a machine replaces the first. The
|
||||
addresses are still kept **on the hub** rather than on each machine, because a KVM earns
|
||||
its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub
|
||||
proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at
|
||||
all. Not done: holding the device's own login on the hub, so its page does not ask for a
|
||||
second password.
|
||||
- **KVM tab** — shipped 2026-09-24, **removed 2026-11-09** at Mitchell's call: not needed —
|
||||
the KVMs are reached over WireGuard (see `hamlet/docs/hosts.md`). It proxied a device's web
|
||||
page through Ham Lab's origin; the code is in git history before that date.
|
||||
- **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
|
||||
Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
|
||||
type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
|
||||
@@ -852,6 +822,16 @@ something long in one, switch away and back, refresh — it's all still there.
|
||||
- 360 files: the look-around view, zoom 8°–300° (ordinary lens into stereographic, the
|
||||
"little planet"), drag/pinch/wheel, phone motion. Full screen is the element's, or the
|
||||
whole window on an iPhone.
|
||||
- **Video previews follow the setting too** (same day, asked for as "it should use the
|
||||
same video generation device"). On the device's own machine the hub says which device
|
||||
by header (`X-Hamlab-Device`); for a file elsewhere the hub asks the file's machine's
|
||||
cache first (`?cached=1`), else has the device's machine make it from the hub's one-file
|
||||
URL (`POST /api/files/thumb-from`) and keeps it in the file's machine's cache (`PUT
|
||||
/api/files/thumb`, refused if the file changed since its listing), so each is made once.
|
||||
A failure is kept as a failure. Unreachable → made where the file is, as before. Images
|
||||
and audio covers stay on the file's machine's CPU (a JPEG is cheaper there than a GPU
|
||||
round trip). Measured: a `hamtv` video previewed on grid's A380 in 0.22 s, then 2 ms
|
||||
from hamtv's cache. The previews button's tooltip names the device.
|
||||
- Measured (`web/scripts/e2e-player.mjs`, Chromium): an iPhone HLG clip on `core` made on
|
||||
grid's A380 — 3.2 s to play, a seek 1.0 s, a quality change 1.5 s; the 8K 360 Disney day on
|
||||
grid — 3.1 s, 2160s on Auto, a seek 4.9 s (10 s source keyframes).
|
||||
@@ -991,6 +971,11 @@ what it has moved.
|
||||
|
||||
## To do
|
||||
|
||||
- **Handle `fuse.mergerfs` in the Storage view** (2026-10-25). `grid` has a mergerfs pool at
|
||||
`/srv/tank` (the NVMe landing area + `/tank`; `hamlet`'s `docs/tank-landing-plan.md`).
|
||||
`fuse.mergerfs` isn't in `remoteFS` or `pseudoFS` (`internal/metrics/metrics.go`), so it shows
|
||||
as a local filesystem whose size is `tank` and the NVMe added together. That counts both twice
|
||||
in any storage total. Show it, labelled as a pool of its branches, and leave it out of totals.
|
||||
- **Grow the Files tab** — asked for 2026-09-23 as two items (Files polish, and deprecating
|
||||
copyparty), merged 2026-09-24. **copyparty was retired 2026-10-11** at Mitchell's call,
|
||||
ahead of these gaps rather than after them: service, files and NPM host all gone, with a
|
||||
|
||||
+3
-5
@@ -20,10 +20,10 @@ default or phones home to this lab · **C** cosmetic (placeholder text, comments
|
||||
| 1.1 | **B** | `internal/selfupdate/selfupdate.go:36` `DefaultURL = "https://git.bigham.dev/api/v1/repos/mitchell/hamlab"` | Every install with no `update.url` checks **this** Gitea 90 s after boot and daily (`internal/api/update.go:1951, 2016`). A stranger's install phones home here by default. | Default should be **off** (or a build-time `-X` variable set at release). Installer asks or `install.sh` passes `HAMLAB_URL` through into `update.url`. |
|
||||
| 1.2 | **B** | `install.sh:4, 27` `HAMLAB_URL` default; `README.md:23-25, 31` | The curl one-liner and the install source are this Gitea. | Default comes from the release that served the script (`@URL@` filled by `make release`), or is required. |
|
||||
| 1.3 | **B** | `scripts/gitea-release.sh:14` `REPO` default | Publishing goes to `mitchell/hamlab` on this Gitea. | Require `HAMLAB_REPO`; no default. |
|
||||
| 1.4 | **C** | `README.md:100` "On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager" | Lab deployment fact in the product README. | Move to `hamlet/docs/hosts.md`. |
|
||||
| 1.4 | **C** | `README.md:100` "On `core` it is public at `https://lab.bigham.dev` behind Nginx Proxy Manager" | Lab deployment fact in the product README. | Move to `hamlet/docs/hosts.md`. |
|
||||
| 1.5 | **C** | `examples/external-demo/index.html:8` `base: 'http://10.0.0.234:8100'` | Demo config names the hub's LAN IP. | `http://<hamlab-host>:8100`. |
|
||||
| 1.6 | **C** | `SENSORS.md:149` Prometheus target `10.0.0.234:8100`; `:187` `&machine=grid`; `:196-208` Glance examples at `hamlab.bigham.dev`, `zfs.tank.used_pct`, `/api/nodes/grid/...`; `:218-220` `grid,wyse`, `{"core":…,"grid":…,"mac":…}` | Docs use real host names, pool and domain. | Generic placeholders (`<host>`, `<machine>`, `<pool>`). |
|
||||
| 1.7 | **C** | `PLAN.md:3-4` "bare-metal Debian 13 host. First target: `core` (10.0.0.234)"; decisions table rows (Listen `10.0.0.234:8100`, Data `/home/core/apps/hamlab/`, Ingress NPM/`hamlab.bigham.dev`, Repo) | Product plan is written as this lab's plan. | Keep as history, or split lab facts out to `hamlet`. The repo is public. |
|
||||
| 1.6 | **C** | `SENSORS.md:149` Prometheus target `10.0.0.234:8100`; `:187` `&machine=grid`; `:196-208` Glance examples at `lab.bigham.dev`, `zfs.tank.used_pct`, `/api/nodes/grid/...`; `:218-220` `grid,wyse`, `{"core":…,"grid":…,"mac":…}` | Docs use real host names, pool and domain. | Generic placeholders (`<host>`, `<machine>`, `<pool>`). |
|
||||
| 1.7 | **C** | `PLAN.md:3-4` "bare-metal Debian 13 host. First target: `core` (10.0.0.234)"; decisions table rows (Listen `10.0.0.234:8100`, Data `/home/core/apps/hamlab/`, Ingress NPM/`lab.bigham.dev`, Repo) | Product plan is written as this lab's plan. | Keep as history, or split lab facts out to `hamlet`. The repo is public. |
|
||||
| 1.9 | **C** | `BUGS.md` throughout | Bug bashes narrate `core`, `grid`, `wyse`, `hamtv`, `gate`, the Mac. | History; fine to leave, but it is public. |
|
||||
| 1.10 | **C** | `cmd/hamlab/main.go:1` "Proxmox-style web console for a bare-metal Debian host"; `internal/hardware/hardware.go:5` "what Debian and its kin already have"; `internal/hardware/hardware.go:29` `// "GMKtec NucBox G3"`; `internal/metrics/metrics.go:1358` "did, on grid, 2026-10-02"; `internal/api/server.go:1382-1383` `["grid", …]`, `?machines=grid,wyse`; `internal/api/stacks.go:10` `/home/core/apps/dockhand/stacks/…`; `internal/api/dockersched.go:5` "unbound, nginx and pihole"; `internal/api/zfs.go:5` "grid already runs zfSnap"; `internal/storage/snapshots.go:6` "grid has run zfSnap v1.11.1"; `internal/api/guests.go:8` "Mitchell, 2026-09-27"; `internal/network/network.go:306` "the public address here is a DHCP lease"; `internal/metrics/gpu.go:17` "Asahi, today"; `internal/metrics/pciids.go:4-6` "the router's GPU"; `web/src/Top.tsx:522, 576`; `web/src/hashsel.ts:1-9`; `web/src/Embed.tsx:3` `&machine=grid` | Comments explain decisions by naming lab hosts. | Harmless, but rewrite the ones in package doc comments so a reader of the source doesn't need the lab to understand the code. |
|
||||
|
||||
@@ -32,7 +32,6 @@ default or phones home to this lab · **C** cosmetic (placeholder text, comments
|
||||
| # | Sev | Where | Text | General form |
|
||||
|---|---|---|---|---|
|
||||
| 2.1 | **C** | `web/src/Machines.tsx:116-118` | `placeholder="grid"`, `"https://10.0.0.214:8100"`, `"core"` | `nas`, `https://192.0.2.10:8100`, `admin` or the hub's own user. |
|
||||
| 2.2 | **C** | `web/src/KVM.tsx:166` | `placeholder="10.0.0.217"` (the NanoKVM on `grid`) | `192.0.2.20` or `kvm.local`. |
|
||||
| 2.3 | **C** | `web/src/Settings.tsx:242` | `placeholder="cpu.*, temp.*, zfs.tank.*"` | `cpu.*, temp.*, fs.*`. |
|
||||
| 2.4 | **C** | `web/src/Settings.tsx:225` | read-only token default name `'glance'` | Empty, or `dashboard`. |
|
||||
| 2.5 | **C** | `web/src/Alerts.tsx:38, 40` | `"Tank nearly full"`, `"zfs.tank.used_pct"` | `"Disk nearly full"`, `"fs.root.used_pct"`. |
|
||||
@@ -146,7 +145,6 @@ Listed so the sweep is known to be complete, not because they need work.
|
||||
- `internal/nodes`, `internal/tlsauto`: hub ↔ node pairing, pinned certs, proxy, push.
|
||||
- `internal/sensors`, alerts, widgets, embeds, Prometheus: machine-agnostic by design.
|
||||
- `internal/files`, `internal/term`, `internal/tasks`, `internal/auth`, `internal/store`.
|
||||
- `internal/api/kvm.go`: "nothing in it knows anything about any particular KVM" — verified.
|
||||
- `internal/compose` (modulo 3.5).
|
||||
- The React UI: machine-switcher model, hash routing, per-account prefs, themes.
|
||||
|
||||
|
||||
@@ -97,5 +97,5 @@ page flags machines running an older version and offers the same push per machin
|
||||
Mixed architectures are fine: `make deploy` also cross-compiles every other architecture,
|
||||
and the hub sends each machine the build for its own.
|
||||
|
||||
- On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager.
|
||||
- On `core` it is public at `https://lab.bigham.dev` behind Nginx Proxy Manager.
|
||||
- Installable as a PWA.
|
||||
|
||||
+5
-2
@@ -169,6 +169,9 @@ container.<name>.down should be up and isn't: Docker c
|
||||
restart it — *not* a container someone stopped
|
||||
updates.pending|security|reboot_required every 5 min
|
||||
zfs.<pool>.health|healthy|size|used|used_pct|frag every 1 min
|
||||
zfs.write.dirty|limit|delayed the RAM write buffer (all pools) and writes the throttle is slowing
|
||||
zfs.<pool>.write_bps|logical_bps bytes/s synced to disk, and written by programs
|
||||
zfs.<pool>.landing.pending_bytes|pending_files|used_pct|in_bps|drain_bps|mover|queue a landing tier, where one fronts the pool
|
||||
smart.<dev>.passed|warnings|power_on|wear every 10 min, never waking a sleeping disk
|
||||
```
|
||||
|
||||
@@ -200,13 +203,13 @@ facts, never live. An `iframe` of the embed is live.
|
||||
```yaml
|
||||
- type: iframe
|
||||
title: core CPU
|
||||
source: https://hamlab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
|
||||
source: https://lab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
|
||||
height: 130
|
||||
|
||||
- type: custom-api
|
||||
title: grid
|
||||
cache: 1m
|
||||
url: https://hamlab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
|
||||
url: https://lab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
|
||||
headers: { Authorization: Bearer <read-only token> }
|
||||
template: |
|
||||
<div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>
|
||||
|
||||
@@ -63,6 +63,7 @@ func main() {
|
||||
passwd [user] set a user's password (makes the account if it is new; reads stdin when piped)
|
||||
users list the accounts
|
||||
zfs-snapshot <id> run one ZFS snapshot schedule (what its cron line calls)
|
||||
landing-verify <new|all|id> verify a landing tier against its mover's ledger (a schedule's id is what its cron line calls)
|
||||
docker-update <id> run one image-update schedule (what its cron line calls)
|
||||
user-rm <user> remove an account and sign it out everywhere
|
||||
check verify the config and state load, then exit
|
||||
@@ -150,6 +151,10 @@ func main() {
|
||||
if err := zfsSnapshot(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "landing-verify":
|
||||
if err := landingVerify(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "users":
|
||||
migrateUsers(cfg)
|
||||
if len(cfg.Users) == 0 {
|
||||
@@ -254,6 +259,81 @@ func dockerUpdateJob(cfg *config.Config, id string) error {
|
||||
//
|
||||
// It deliberately does not write state.json. The daemon owns that file and would clobber
|
||||
// anything written from out here; the outcome goes beside it instead, in zfs-runs/.
|
||||
// landingVerify runs one verify pass over every landing tier on this machine: by hand with a
|
||||
// scope ("new" or "all"), or as a schedule by its id, which is what the cron line passes.
|
||||
func landingVerify(cfg *config.Config, arg string) error {
|
||||
scope, id := arg, "manual"
|
||||
if arg != "new" && arg != "all" {
|
||||
if strings.TrimSpace(arg) == "" {
|
||||
return errors.New("usage: hamlab landing-verify <new|all|schedule-id>")
|
||||
}
|
||||
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var raw json.RawMessage
|
||||
st.View(func(s *store.State) { raw = s.ZFS })
|
||||
var z struct {
|
||||
Verify []storage.VerifyPolicy `json:"verify"`
|
||||
}
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &z); err != nil {
|
||||
return fmt.Errorf("reading the schedules: %w", err)
|
||||
}
|
||||
}
|
||||
found := false
|
||||
for _, p := range z.Verify {
|
||||
if p.ID == arg {
|
||||
scope, id, found = p.Scope, p.ID, true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return fmt.Errorf("no verify schedule %q — it may have been removed; this cron line is stale", arg)
|
||||
}
|
||||
}
|
||||
dir := cfg.DataDir()
|
||||
unlock, err := storage.LockVerify(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer unlock()
|
||||
defer storage.ClearVerifyProgress(dir)
|
||||
targets := storage.LandingTargets()
|
||||
if len(targets) == 0 {
|
||||
r := storage.VerifyResult{ID: id, Scope: scope, Started: time.Now(), Finished: time.Now(), Issues: []storage.VerifyIssue{},
|
||||
Error: "no landing mover is reporting (nothing in " + storage.MoverStatusDir + "), so there is nothing to verify"}
|
||||
_ = storage.RecordVerify(dir, storage.LandingTarget{}, r)
|
||||
return errors.New(r.Error)
|
||||
}
|
||||
var firstErr error
|
||||
for _, t := range targets {
|
||||
r := storage.Verify(t, id, scope, storage.VerifyCursor(dir, t), func(p storage.VerifyProgress) { storage.WriteVerifyProgress(dir, p) })
|
||||
if err := storage.RecordVerify(dir, t, r); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "recording the pass:", err)
|
||||
}
|
||||
fmt.Printf("%s -> %s (%s): %d checked, %d match, %d mismatch, %d gone, %d unreadable, %d changed since, %d waiting, %d unhashed, %s read in %s\n",
|
||||
t.Src, t.Dst, scope, r.Checked, r.Matched, r.Mismatched, r.Gone, r.Unreadable, r.Changed, r.Waiting, r.Unhashed,
|
||||
humanBytes(r.BytesRead), r.Finished.Sub(r.Started).Round(time.Second))
|
||||
if r.Error != "" && firstErr == nil {
|
||||
firstErr = errors.New(r.Error)
|
||||
}
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
|
||||
func humanBytes(n int64) string {
|
||||
const u = 1024
|
||||
if n < u {
|
||||
return fmt.Sprintf("%d B", n)
|
||||
}
|
||||
div, exp := int64(u), 0
|
||||
for v := n / u; v >= u; v /= u {
|
||||
div *= u
|
||||
exp++
|
||||
}
|
||||
return fmt.Sprintf("%.1f %ciB", float64(n)/float64(div), "KMGTPE"[exp])
|
||||
}
|
||||
|
||||
func zfsSnapshot(cfg *config.Config, id string) error {
|
||||
if strings.TrimSpace(id) == "" {
|
||||
return errors.New("usage: hamlab zfs-snapshot <schedule-id>")
|
||||
|
||||
@@ -1,420 +0,0 @@
|
||||
package api
|
||||
|
||||
// IP-KVMs: a device's own web page, shown inside Ham Lab.
|
||||
//
|
||||
// The whole difficulty is that Ham Lab is served over TLS and a KVM usually is not, so a
|
||||
// browser refuses to put the device's page in an iframe — mixed content, blocked before
|
||||
// anything loads. The fix is to serve the device from Ham Lab's own origin, which also
|
||||
// means it inherits Ham Lab's login instead of needing a second one, and makes the iframe
|
||||
// same-origin.
|
||||
//
|
||||
// Proxying an arbitrary web app under a path prefix is the hard part, and nothing here
|
||||
// knows anything about any particular device. A device UI asks for its assets by absolute
|
||||
// path — "/assets/app.js", "/api/whatever" — which a prefix cannot rewrite away. So the
|
||||
// prefix is only the entry point: any request Ham Lab does not serve itself is routed to
|
||||
// whichever KVM the asking page belongs to, worked out from the Referer, and from a cookie
|
||||
// when there is no Referer (a WebSocket upgrade, say). Ham Lab's own routes always win, so
|
||||
// a device can never shadow the console.
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const kvmPrefix = "/kvm/"
|
||||
const kvmCookie = "hamlab_kvm"
|
||||
|
||||
type kvmProxies struct {
|
||||
mu sync.Mutex
|
||||
m map[string]*httputil.ReverseProxy
|
||||
}
|
||||
|
||||
func (s *Server) kvm(id string) (config.KVM, bool) {
|
||||
var found config.KVM
|
||||
ok := false
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, k := range c.KVMs {
|
||||
if k.ID == id {
|
||||
found, ok = k, true
|
||||
}
|
||||
}
|
||||
})
|
||||
return found, ok
|
||||
}
|
||||
|
||||
// kvmProxy is the reverse proxy for one device, made once and kept.
|
||||
func (s *Server) kvmProxy(k config.KVM) (*httputil.ReverseProxy, error) {
|
||||
s.kvms.mu.Lock()
|
||||
defer s.kvms.mu.Unlock()
|
||||
if s.kvms.m == nil {
|
||||
s.kvms.m = map[string]*httputil.ReverseProxy{}
|
||||
}
|
||||
if p, ok := s.kvms.m[k.ID]; ok {
|
||||
return p, nil
|
||||
}
|
||||
target, err := url.Parse(k.URL)
|
||||
if err != nil || target.Host == "" {
|
||||
return nil, errors.New("this KVM's address is not a URL")
|
||||
}
|
||||
prefix := kvmPrefix + k.ID
|
||||
p := &httputil.ReverseProxy{
|
||||
Director: func(r *http.Request) {
|
||||
r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
|
||||
// Entry through the prefix keeps the path after it; anything else arrived by its
|
||||
// own absolute path and is passed through untouched.
|
||||
if rest, ok := strings.CutPrefix(r.URL.Path, prefix); ok {
|
||||
if rest == "" {
|
||||
rest = "/"
|
||||
}
|
||||
r.URL.Path = rest
|
||||
}
|
||||
r.URL.RawPath = ""
|
||||
r.Host = target.Host
|
||||
// Whether the console itself is on TLS: the response rewriter needs to know, and by
|
||||
// then only the outbound request is in hand.
|
||||
if secure(r) {
|
||||
r.Header.Set(kvmSecureHeader, "1")
|
||||
} else {
|
||||
r.Header.Del(kvmSecureHeader)
|
||||
}
|
||||
// The device gets its own host and none of Ham Lab's session.
|
||||
r.Header.Del("Origin")
|
||||
stripCookie(r, cookieName, kvmCookie)
|
||||
// What is left is the device's own login. A browser that has it teaches the hub; one
|
||||
// that has not is lent what the hub remembers, so the device sees a signed-in client.
|
||||
if own := r.Cookies(); len(own) > 0 {
|
||||
s.rememberKVMCookies(k.ID, own)
|
||||
} else {
|
||||
for name, val := range s.kvmCookies(k.ID) {
|
||||
r.AddCookie(&http.Cookie{Name: name, Value: val})
|
||||
}
|
||||
}
|
||||
},
|
||||
FlushInterval: -1, // a video stream must not be buffered
|
||||
ModifyResponse: func(resp *http.Response) error {
|
||||
// The device's own framing and transport policy would stop it being embedded, or
|
||||
// send a browser off to plain HTTP. Ham Lab is the origin now, and says what it
|
||||
// serves; its own headers are added on the way out.
|
||||
for _, h := range []string{"X-Frame-Options", "Content-Security-Policy", "Content-Security-Policy-Report-Only", "Strict-Transport-Security", "Referrer-Policy"} {
|
||||
resp.Header.Del(h)
|
||||
}
|
||||
// A cookie the device sets by header is kept for Ham Lab's origin — so its Domain
|
||||
// (the device's) goes, Secure follows the console rather than the device, and the
|
||||
// hub remembers it too.
|
||||
if set := resp.Cookies(); len(set) > 0 {
|
||||
https := resp.Request != nil && resp.Request.Header.Get(kvmSecureHeader) == "1"
|
||||
resp.Header.Del("Set-Cookie")
|
||||
for _, c := range set {
|
||||
c.Domain = ""
|
||||
if c.Path == "" {
|
||||
c.Path = "/"
|
||||
}
|
||||
c.Secure = c.Secure && https
|
||||
if v := c.String(); v != "" {
|
||||
resp.Header.Add("Set-Cookie", v)
|
||||
}
|
||||
}
|
||||
s.rememberKVMCookies(k.ID, set)
|
||||
}
|
||||
// A redirect to the device's own root has to come back through the prefix.
|
||||
if loc := resp.Header.Get("Location"); loc != "" {
|
||||
if u, err := url.Parse(loc); err == nil && (u.Host == "" || u.Host == target.Host) && strings.HasPrefix(u.Path, "/") && !strings.HasPrefix(u.Path, prefix) {
|
||||
u.Scheme, u.Host = "", ""
|
||||
u.Path = prefix + u.Path
|
||||
resp.Header.Set("Location", u.String())
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": k.Name + " unreachable: " + err.Error()})
|
||||
},
|
||||
}
|
||||
s.kvms.m[k.ID] = p
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// kvmSecureHeader rides on the outbound request to say whether the console was reached
|
||||
// over TLS; the device ignores it.
|
||||
const kvmSecureHeader = "X-Hamlab-Secure"
|
||||
|
||||
// kvmCookies is what the hub remembers of a device's own cookies — its login, in practice.
|
||||
func (s *Server) kvmCookies(id string) map[string]string {
|
||||
var out map[string]string
|
||||
s.store.View(func(st *store.State) {
|
||||
if m := st.KVMCookies[id]; len(m) > 0 {
|
||||
out = make(map[string]string, len(m))
|
||||
for k, v := range m {
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// rememberKVMCookies keeps the cookies a browser (or the device's own Set-Cookie) carries
|
||||
// for a KVM, so that a browser without them can be lent them. A cookie emptied or expired
|
||||
// by the device is forgotten. Written only when something changed: this runs on every
|
||||
// proxied request.
|
||||
func (s *Server) rememberKVMCookies(id string, cs []*http.Cookie) {
|
||||
have := s.kvmCookies(id)
|
||||
changed := false
|
||||
for _, c := range cs {
|
||||
if c.Name == "" || c.Name == cookieName || c.Name == kvmCookie {
|
||||
continue
|
||||
}
|
||||
gone := c.Value == "" || c.MaxAge < 0 || (!c.Expires.IsZero() && c.Expires.Before(time.Now()))
|
||||
if gone {
|
||||
if _, ok := have[c.Name]; ok {
|
||||
changed = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
if have[c.Name] != c.Value {
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
if !changed {
|
||||
return
|
||||
}
|
||||
_ = s.store.Update(func(st *store.State) {
|
||||
if st.KVMCookies == nil {
|
||||
st.KVMCookies = map[string]map[string]string{}
|
||||
}
|
||||
m := st.KVMCookies[id]
|
||||
if m == nil {
|
||||
m = map[string]string{}
|
||||
}
|
||||
for _, c := range cs {
|
||||
if c.Name == "" || c.Name == cookieName || c.Name == kvmCookie {
|
||||
continue
|
||||
}
|
||||
if c.Value == "" || c.MaxAge < 0 || (!c.Expires.IsZero() && c.Expires.Before(time.Now())) {
|
||||
delete(m, c.Name)
|
||||
} else {
|
||||
m[c.Name] = c.Value
|
||||
}
|
||||
}
|
||||
if len(m) == 0 {
|
||||
delete(st.KVMCookies, id)
|
||||
} else {
|
||||
st.KVMCookies[id] = m
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// forgetKVMCookies drops a device's remembered login — when the device is removed.
|
||||
func (s *Server) forgetKVMCookies(id string) {
|
||||
_ = s.store.Update(func(st *store.State) { delete(st.KVMCookies, id) })
|
||||
}
|
||||
|
||||
// lendKVMCookies hands the browser the device cookies the hub remembers and it lacks, as
|
||||
// the device's own page would have set them: readable by its scripts, for Ham Lab's whole
|
||||
// origin, kept for half a year. The device's SPA checks for its token client-side before it
|
||||
// ever asks the server, so lending it only on the wire would still land on the login form.
|
||||
func (s *Server) lendKVMCookies(w http.ResponseWriter, r *http.Request, id string) {
|
||||
for name, val := range s.kvmCookies(id) {
|
||||
if _, err := r.Cookie(name); err == nil {
|
||||
continue
|
||||
}
|
||||
http.SetCookie(w, &http.Cookie{Name: name, Value: val, Path: "/", MaxAge: 180 * 24 * 3600, Secure: secure(r), SameSite: http.SameSiteLaxMode})
|
||||
}
|
||||
}
|
||||
|
||||
// stripCookie removes named cookies from a request, so Ham Lab's session is never handed to
|
||||
// a device and the device never sees which KVM the browser is looking at.
|
||||
func stripCookie(r *http.Request, names ...string) {
|
||||
cs := r.Cookies()
|
||||
kept := cs[:0]
|
||||
for _, c := range cs {
|
||||
drop := false
|
||||
for _, n := range names {
|
||||
if c.Name == n {
|
||||
drop = true
|
||||
}
|
||||
}
|
||||
if !drop {
|
||||
kept = append(kept, c)
|
||||
}
|
||||
}
|
||||
r.Header.Del("Cookie")
|
||||
for _, c := range kept {
|
||||
r.AddCookie(c)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) forgetKVMProxy(id string) {
|
||||
s.kvms.mu.Lock()
|
||||
delete(s.kvms.m, id)
|
||||
s.kvms.mu.Unlock()
|
||||
}
|
||||
|
||||
// serveKVM is the entry point: /kvm/<id>/… . It also drops a cookie naming the device, so
|
||||
// that requests the page makes later by absolute path can be traced back to it.
|
||||
func (s *Server) serveKVM(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
k, ok := s.kvm(id)
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such KVM")
|
||||
return
|
||||
}
|
||||
p, err := s.kvmProxy(k)
|
||||
if err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
http.SetCookie(w, &http.Cookie{Name: kvmCookie, Value: id, Path: "/", HttpOnly: true, Secure: secure(r), SameSite: http.SameSiteLaxMode})
|
||||
s.lendKVMCookies(w, r, id)
|
||||
p.ServeHTTP(w, r)
|
||||
}
|
||||
|
||||
// kvmFor works out which device a request belongs to when it arrived by absolute path
|
||||
// rather than through the prefix: the Referer names the page that asked, and a cookie
|
||||
// covers what has no Referer at all. Nothing device-specific: it never has to know which
|
||||
// paths a given KVM uses.
|
||||
func (s *Server) kvmFor(r *http.Request) (config.KVM, bool) {
|
||||
if ref := r.Header.Get("Referer"); ref != "" {
|
||||
if u, err := url.Parse(ref); err == nil {
|
||||
if rest, ok := strings.CutPrefix(u.Path, kvmPrefix); ok {
|
||||
if id, _, _ := strings.Cut(rest, "/"); id != "" {
|
||||
if k, ok := s.kvm(id); ok {
|
||||
return k, true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// asked for by a page that is not a KVM: not ours to answer
|
||||
return config.KVM{}, false
|
||||
}
|
||||
if c, err := r.Cookie(kvmCookie); err == nil && c.Value != "" {
|
||||
return s.kvm(c.Value)
|
||||
}
|
||||
return config.KVM{}, false
|
||||
}
|
||||
|
||||
// kvmFallback answers a request Ham Lab does not serve itself, when it came from inside a
|
||||
// KVM's page. It returns false when the request has nothing to do with a KVM, and the
|
||||
// caller carries on with its own 404.
|
||||
func (s *Server) kvmFallback(w http.ResponseWriter, r *http.Request) bool {
|
||||
k, ok := s.kvmFor(r)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
p, err := s.kvmProxy(k)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
p.ServeHTTP(w, r)
|
||||
return true
|
||||
}
|
||||
|
||||
func machineLabel(m string) string {
|
||||
if m == "" {
|
||||
return "this host"
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ---- the list ----
|
||||
|
||||
func (s *Server) listKVMs(w http.ResponseWriter, _ *http.Request) {
|
||||
out := []config.KVM{}
|
||||
s.cfg.View(func(c *config.Config) { out = append(out, c.KVMs...) })
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name, URL, Machine string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
raw := strings.TrimSpace(body.URL)
|
||||
if raw != "" && !strings.Contains(raw, "://") {
|
||||
raw = "http://" + raw // what you would type: an address, not a scheme
|
||||
}
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
jsonErr(w, 400, "address: http(s)://host, or just the host")
|
||||
return
|
||||
}
|
||||
name := strings.TrimSpace(body.Name)
|
||||
if name == "" {
|
||||
name = u.Host
|
||||
}
|
||||
if len(name) > maxNameLen {
|
||||
jsonErr(w, 400, "that name is too long")
|
||||
return
|
||||
}
|
||||
b := make([]byte, 6)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
machine := strings.TrimSpace(body.Machine)
|
||||
k := config.KVM{ID: hex.EncodeToString(b), Name: name, URL: u.Scheme + "://" + u.Host + strings.TrimSuffix(u.Path, "/"), Machine: machine, Added: time.Now()}
|
||||
// One per machine: a second one for the same host replaces the first rather than sitting
|
||||
// beside it, since the tab shows the machine you are looking at.
|
||||
var replaced string
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.KVMs[:0]
|
||||
for _, old := range c.KVMs {
|
||||
if old.Machine == machine {
|
||||
replaced = old.ID
|
||||
continue
|
||||
}
|
||||
kept = append(kept, old)
|
||||
}
|
||||
c.KVMs = append(kept, k)
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if replaced != "" {
|
||||
s.forgetKVMProxy(replaced)
|
||||
}
|
||||
log.Printf("kvm %q added for %s (%s) by %s", k.Name, machineLabel(machine), k.URL, clientIP(r))
|
||||
jsonOK(w, 201, k)
|
||||
}
|
||||
|
||||
func (s *Server) removeKVM(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
found := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.KVMs[:0]
|
||||
for _, k := range c.KVMs {
|
||||
if k.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, k)
|
||||
}
|
||||
c.KVMs = kept
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such KVM")
|
||||
return
|
||||
}
|
||||
s.forgetKVMProxy(id)
|
||||
s.forgetKVMCookies(id)
|
||||
log.Printf("kvm %s removed by %s", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -1,123 +0,0 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// A KVM's own login is a cookie its page sets in one browser. The hub remembers it from the
|
||||
// requests that carry it, lends it to a browser that lacks it — both on the wire to the
|
||||
// device and back to the browser, since the device's page checks for it client-side — and
|
||||
// forgets it when the device says it is gone or the KVM is removed.
|
||||
func TestKVMCookiesAreRememberedAndLent(t *testing.T) {
|
||||
var seen []string // the Cookie header the fake device received, per request
|
||||
device := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
seen = append(seen, r.Header.Get("Cookie"))
|
||||
if r.URL.Path == "/api/auth/login" {
|
||||
// a device that sets its token by header, with its own domain and Secure
|
||||
http.SetCookie(w, &http.Cookie{Name: "nano-kvm-token", Value: "tok-1", Domain: "10.0.0.217", Secure: true})
|
||||
}
|
||||
if r.URL.Path == "/api/auth/logout" {
|
||||
http.SetCookie(w, &http.Cookie{Name: "nano-kvm-token", Value: "", MaxAge: -1})
|
||||
}
|
||||
w.WriteHeader(200)
|
||||
}))
|
||||
defer device.Close()
|
||||
|
||||
dir := t.TempDir()
|
||||
cfg := config.Default(filepath.Join(dir, "config.json"))
|
||||
if err := cfg.Update(func(c *config.Config) {
|
||||
c.KVMs = []config.KVM{{ID: "abc123", Name: "test", URL: device.URL, Machine: ""}}
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err := store.Open(filepath.Join(dir, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{cfg: cfg, store: st, sessions: auth.NewSessions(st)}
|
||||
k, _ := s.kvm("abc123")
|
||||
p, err := s.kvmProxy(k)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ask := func(path string, cookies ...*http.Cookie) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("GET", path, nil)
|
||||
r.SetPathValue("id", "abc123")
|
||||
for _, c := range cookies {
|
||||
r.AddCookie(c)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
if strings.HasPrefix(path, kvmPrefix) {
|
||||
s.serveKVM(w, r)
|
||||
} else {
|
||||
p.ServeHTTP(w, r)
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// 1. the device sets its token by header: it reaches the browser without the device's
|
||||
// domain or Secure (plain-http console), and the hub remembers it
|
||||
w := ask("/api/auth/login", &http.Cookie{Name: cookieName, Value: "hamlab-session"})
|
||||
var set *http.Cookie
|
||||
for _, c := range w.Result().Cookies() {
|
||||
if c.Name == "nano-kvm-token" {
|
||||
set = c
|
||||
}
|
||||
}
|
||||
if set == nil || set.Domain != "" || set.Secure || set.Value != "tok-1" || set.Path != "/" {
|
||||
t.Fatalf("rewritten Set-Cookie = %+v, want nano-kvm-token=tok-1 for / without Domain or Secure", set)
|
||||
}
|
||||
if got := s.kvmCookies("abc123"); got["nano-kvm-token"] != "tok-1" {
|
||||
t.Fatalf("remembered = %v, want the token", got)
|
||||
}
|
||||
if strings.Contains(seen[0], "hamlab_session") {
|
||||
t.Fatalf("the device was handed Ham Lab's session: %q", seen[0])
|
||||
}
|
||||
|
||||
// 2. a browser that has the cookie teaches a newer value
|
||||
ask("/api/vm/info", &http.Cookie{Name: "nano-kvm-token", Value: "tok-2"}, &http.Cookie{Name: cookieName, Value: "x"})
|
||||
if got := s.kvmCookies("abc123"); got["nano-kvm-token"] != "tok-2" {
|
||||
t.Fatalf("remembered = %v, want tok-2", got)
|
||||
}
|
||||
if seen[1] != "nano-kvm-token=tok-2" {
|
||||
t.Fatalf("device saw %q, want the browser's own cookie only", seen[1])
|
||||
}
|
||||
|
||||
// 3. a browser without it is lent it: on the wire, and back as a Set-Cookie it can read
|
||||
w = ask(kvmPrefix+"abc123/", &http.Cookie{Name: cookieName, Value: "x"})
|
||||
if seen[2] != "nano-kvm-token=tok-2" {
|
||||
t.Fatalf("device saw %q from a browser with no cookie, want the remembered token", seen[2])
|
||||
}
|
||||
lent := false
|
||||
for _, c := range w.Result().Cookies() {
|
||||
if c.Name == "nano-kvm-token" && c.Value == "tok-2" && !c.HttpOnly && c.Path == "/" && c.MaxAge > 0 {
|
||||
lent = true
|
||||
}
|
||||
}
|
||||
if !lent {
|
||||
t.Fatalf("browser was not lent the token: %v", w.Result().Cookies())
|
||||
}
|
||||
|
||||
// 4. the device logging out forgets it
|
||||
ask("/api/auth/logout", &http.Cookie{Name: "nano-kvm-token", Value: "tok-2"})
|
||||
if got := s.kvmCookies("abc123"); got != nil {
|
||||
t.Fatalf("after logout remembered = %v, want nothing", got)
|
||||
}
|
||||
|
||||
// 5. removing the KVM clears whatever is left
|
||||
s.rememberKVMCookies("abc123", []*http.Cookie{{Name: "nano-kvm-token", Value: "tok-3"}})
|
||||
s.forgetKVMCookies("abc123")
|
||||
if got := s.kvmCookies("abc123"); got != nil {
|
||||
t.Fatalf("after remove remembered = %v, want nothing", got)
|
||||
}
|
||||
_ = url.Parse
|
||||
}
|
||||
@@ -0,0 +1,216 @@
|
||||
package api
|
||||
|
||||
// The ZFS tab's landing-tier checks: what has waited too long on a landing branch, and verify
|
||||
// passes that read the pool's copy of every moved file back against the mover's ledger —
|
||||
// run by hand or on a schedule written to /etc/cron.d/hamlab-landing, just as snapshot
|
||||
// schedules are. A pass runs as its own process (`hamlab landing-verify`), the same one cron
|
||||
// starts, so it survives the console being restarted or upgraded mid-pass.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
func (s *Server) syncVerifyCron(ctx context.Context, z zfsState) error {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
exe, _ = filepath.EvalSymlinks(exe)
|
||||
return s.storage.WriteVerifyCron(ctx, z.Verify, s.user, exe, s.cfg.Path())
|
||||
}
|
||||
|
||||
// startVerify launches a pass in its own session: arg is "new", "all" or a schedule id.
|
||||
func (s *Server) startVerify(arg string) error {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
exe, _ = filepath.EvalSymlinks(exe)
|
||||
dir := filepath.Join(s.cfg.DataDir(), "landing-verify")
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := os.OpenFile(filepath.Join(dir, "last.log"), os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cmd := exec.Command(exe, "-config", s.cfg.Path(), "landing-verify", arg)
|
||||
cmd.Stdout, cmd.Stderr = out, out
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
|
||||
if err := cmd.Start(); err != nil {
|
||||
out.Close()
|
||||
return err
|
||||
}
|
||||
go func() { _ = cmd.Wait(); out.Close() }()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Server) landingOverview(w http.ResponseWriter, r *http.Request) {
|
||||
dir := s.cfg.DataDir()
|
||||
now := time.Now()
|
||||
type tier struct {
|
||||
storage.LandingTarget
|
||||
Cursor int64 `json:"cursor"`
|
||||
StaleCount int `json:"stale_count"`
|
||||
StaleBytes int64 `json:"stale_bytes"`
|
||||
Stale []storage.StaleFile `json:"stale"`
|
||||
}
|
||||
tiers := []tier{}
|
||||
for _, t := range storage.LandingTargets() {
|
||||
files, n, b := storage.StaleFiles(t.Src, 100, now)
|
||||
if files == nil {
|
||||
files = []storage.StaleFile{}
|
||||
}
|
||||
tiers = append(tiers, tier{LandingTarget: t, Cursor: storage.VerifyCursor(dir, t), StaleCount: n, StaleBytes: b, Stale: files})
|
||||
}
|
||||
z := s.zfsLoad()
|
||||
pols := make([]map[string]any, 0, len(z.Verify))
|
||||
for i := range z.Verify {
|
||||
p := z.Verify[i]
|
||||
expr, _ := p.CronExpr()
|
||||
m := map[string]any{"policy": p, "cron": expr, "last": storage.LastVerify(dir, p.ID)}
|
||||
if p.Enabled {
|
||||
if n := p.NextRun(now); !n.IsZero() {
|
||||
m["next"] = n
|
||||
}
|
||||
}
|
||||
pols = append(pols, m)
|
||||
}
|
||||
hist := storage.VerifyHistory(dir)
|
||||
if len(hist) > 10 {
|
||||
hist = hist[:10]
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"tiers": tiers, "stale_after_sec": int(storage.StaleAfter.Seconds()),
|
||||
"running": storage.RunningVerify(dir), "history": hist, "policies": pols,
|
||||
"cron_file": storage.VerifyCronFile,
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) landingVerifyNow(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Scope string `json:"scope"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1024)).Decode(&body); err != nil || (body.Scope != "new" && body.Scope != "all") {
|
||||
jsonErr(w, 400, "scope: new or all")
|
||||
return
|
||||
}
|
||||
if storage.RunningVerify(s.cfg.DataDir()) != nil {
|
||||
jsonErr(w, 409, "a verify pass is already running")
|
||||
return
|
||||
}
|
||||
if len(storage.LandingTargets()) == 0 {
|
||||
jsonErr(w, 409, "no landing mover is reporting on this machine, so there is nothing to verify")
|
||||
return
|
||||
}
|
||||
if err := s.startVerify(body.Scope); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("landing verify (%s) started by %s", body.Scope, clientIP(r))
|
||||
jsonOK(w, 202, map[string]string{"started": body.Scope})
|
||||
}
|
||||
|
||||
func (s *Server) landingPolicyPut(w http.ResponseWriter, r *http.Request) {
|
||||
var p storage.VerifyPolicy
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 16<<10)).Decode(&p); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if err := p.Validate(); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
z := s.zfsLoad()
|
||||
found := false
|
||||
for i := range z.Verify {
|
||||
if p.ID != "" && z.Verify[i].ID == p.ID {
|
||||
z.Verify[i], found = p, true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
if p.ID == "" {
|
||||
id, err := randomID()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
p.ID = id
|
||||
}
|
||||
z.Verify = append(z.Verify, p)
|
||||
}
|
||||
if err := s.zfsSave(z); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.syncVerifyCron(r.Context(), z); err != nil {
|
||||
jsonErr(w, 500, "saved, but writing "+storage.VerifyCronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
expr, _ := p.CronExpr()
|
||||
log.Printf("landing verify schedule %q saved by %s (%q, %s)", p.Name, clientIP(r), expr, p.Scope)
|
||||
jsonOK(w, 200, p)
|
||||
}
|
||||
|
||||
func (s *Server) landingPolicyDelete(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
z := s.zfsLoad()
|
||||
kept := z.Verify[:0]
|
||||
found := false
|
||||
for _, p := range z.Verify {
|
||||
if p.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, p)
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
z.Verify = kept
|
||||
if err := s.zfsSave(z); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.syncVerifyCron(r.Context(), z); err != nil {
|
||||
jsonErr(w, 500, "removed, but writing "+storage.VerifyCronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
storage.ForgetVerify(s.cfg.DataDir(), id)
|
||||
log.Printf("landing verify schedule %s removed by %s", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) landingPolicyRun(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
found := false
|
||||
for _, p := range s.zfsLoad().Verify {
|
||||
found = found || p.ID == id
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
if storage.RunningVerify(s.cfg.DataDir()) != nil {
|
||||
jsonErr(w, 409, "a verify pass is already running")
|
||||
return
|
||||
}
|
||||
if err := s.startVerify(id); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("landing verify schedule %s run by %s", id, clientIP(r))
|
||||
jsonOK(w, 202, map[string]string{"started": id})
|
||||
}
|
||||
+34
-1
@@ -225,6 +225,9 @@ func (s *Server) playPiece(w http.ResponseWriter, r *http.Request) {
|
||||
// transcodeChoice is the setting; nil is the machine the file is on.
|
||||
func (s *Server) transcodeChoice() *store.Transcode {
|
||||
var t *store.Transcode
|
||||
if s.store == nil {
|
||||
return nil
|
||||
}
|
||||
s.store.View(func(st *store.State) {
|
||||
if st.Transcode != nil {
|
||||
c := *st.Transcode
|
||||
@@ -242,8 +245,38 @@ type transcodeMachine struct {
|
||||
}
|
||||
|
||||
// getTranscode: GET /api/transcode — the setting, and what every machine can encode on,
|
||||
// asked of each now.
|
||||
// asked of each now. ?only=selected asks only the chosen machine, for the device's name.
|
||||
func (s *Server) getTranscode(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Query().Get("only") == "selected" {
|
||||
c := s.transcodeChoice()
|
||||
out := map[string]any{"selected": c}
|
||||
if c != nil {
|
||||
name := c.Device
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
|
||||
defer cancel()
|
||||
var devs []media.Device
|
||||
if c.Node == "" {
|
||||
caps, _ := s.thumbs.capsFor(ctx, false, false)
|
||||
devs = media.Devices(caps)
|
||||
} else if code, b, err := s.nodes.Do(ctx, c.Node, "GET", "/api/files/play-devices", nil, nil); err == nil && code == 200 {
|
||||
var d struct{ Devices []media.Device }
|
||||
_ = json.Unmarshal(b, &d)
|
||||
devs = d.Devices
|
||||
}
|
||||
for _, d := range devs {
|
||||
if d.ID == c.Device {
|
||||
name = d.Name
|
||||
}
|
||||
}
|
||||
host := c.Node
|
||||
if host == "" {
|
||||
host = s.hostName()
|
||||
}
|
||||
out["host"], out["name"] = host, name
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
return
|
||||
}
|
||||
names := []string{""}
|
||||
if s.nodes != nil {
|
||||
names = append(names, s.nodes.Names()...)
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -49,3 +51,43 @@ func TestPlaySrc(t *testing.T) {
|
||||
t.Fatalf("unknown token: %d", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// A preview made on another machine is kept by the file's own: only while the file is still
|
||||
// the one it was made of, and a failure is kept as a failure.
|
||||
func TestThumbPut(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "film.mp4")
|
||||
if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fi, _ := os.Stat(p)
|
||||
s := &Server{fsLocal: files.Local{}, thumbs: newThumbSvc(dir, "")}
|
||||
v := fmt.Sprintf("%d-%d", fi.ModTime().UnixMilli(), fi.Size())
|
||||
put := func(v string, body string, hdr map[string]string) int {
|
||||
r := httptest.NewRequest("PUT", "/api/files/thumb?path="+url.QueryEscape(p)+"&v="+v, strings.NewReader(body))
|
||||
for k, x := range hdr {
|
||||
r.Header.Set(k, x)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
s.thumbPut(w, r)
|
||||
return w.Code
|
||||
}
|
||||
get := func() *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("GET", "/api/files/thumb?cached=1&path="+url.QueryEscape(p)+"&v="+v, nil)
|
||||
w := httptest.NewRecorder()
|
||||
s.filesThumb(w, r)
|
||||
return w
|
||||
}
|
||||
if w := get(); w.Code != 404 {
|
||||
t.Fatalf("cached=1 before anything: %d", w.Code)
|
||||
}
|
||||
if c := put("1-10", "jpeg", nil); c != 409 {
|
||||
t.Fatalf("a changed file: %d", c)
|
||||
}
|
||||
if c := put(v, "jpeg-bytes", map[string]string{"X-Hamlab-Thumb": "grid vaapi:/dev/dri/renderD129"}); c != 204 {
|
||||
t.Fatalf("put: %d", c)
|
||||
}
|
||||
if w := get(); w.Code != 200 || w.Body.String() != "jpeg-bytes" || !strings.Contains(w.Header().Get("X-Hamlab-Thumb"), "grid") {
|
||||
t.Fatalf("kept: %d %q %q", w.Code, w.Body, w.Header().Get("X-Hamlab-Thumb"))
|
||||
}
|
||||
}
|
||||
|
||||
+12
-28
@@ -75,7 +75,6 @@ type Server struct {
|
||||
net *network.Watcher
|
||||
update updateState
|
||||
events eventLog
|
||||
kvms kvmProxies
|
||||
fsLocal files.FS
|
||||
fsSudo files.FS // nil without passwordless sudo
|
||||
thumbs *thumbSvc
|
||||
@@ -276,6 +275,9 @@ func (s *Server) routes() *http.ServeMux {
|
||||
m.HandleFunc("POST /api/files/delete", s.auth(s.filesDelete))
|
||||
m.HandleFunc("POST /api/files/chmod", s.auth(s.filesChmod))
|
||||
m.HandleFunc("GET /api/files/thumb", s.auth(s.filesThumb))
|
||||
m.HandleFunc("PUT /api/files/thumb", s.auth(s.thumbPut))
|
||||
m.HandleFunc("POST /api/files/thumb-from", s.auth(s.thumbFrom))
|
||||
m.HandleFunc("GET /api/nodes/{name}/files/thumb", s.auth(s.hubThumb))
|
||||
m.HandleFunc("GET /api/files/thumbs", s.auth(s.thumbsStatus))
|
||||
m.HandleFunc("POST /api/files/thumbs/install", s.auth(s.thumbsInstall))
|
||||
m.HandleFunc("POST /api/files/thumbs/gpu-access", s.auth(s.thumbsGPUAccess))
|
||||
@@ -311,6 +313,11 @@ func (s *Server) routes() *http.ServeMux {
|
||||
m.HandleFunc("DELETE /api/zfs/schedules/{id}", s.auth(s.zfsPolicyDelete))
|
||||
m.HandleFunc("POST /api/zfs/schedules/{id}/run", s.auth(s.zfsRunNow))
|
||||
m.HandleFunc("POST /api/zfs/snapshots/delete", s.auth(s.zfsSnapshotDelete))
|
||||
m.HandleFunc("GET /api/zfs/landing", s.auth(s.landingOverview))
|
||||
m.HandleFunc("POST /api/zfs/landing/verify", s.auth(s.landingVerifyNow))
|
||||
m.HandleFunc("PUT /api/zfs/landing/schedules", s.auth(s.landingPolicyPut))
|
||||
m.HandleFunc("DELETE /api/zfs/landing/schedules/{id}", s.auth(s.landingPolicyDelete))
|
||||
m.HandleFunc("POST /api/zfs/landing/schedules/{id}/run", s.auth(s.landingPolicyRun))
|
||||
m.HandleFunc("GET /api/space", s.auth(s.spaceOverview))
|
||||
m.HandleFunc("GET /api/space/{disk}", s.auth(s.spaceStatus))
|
||||
m.HandleFunc("POST /api/space/{disk}/scan", s.auth(s.spaceScan))
|
||||
@@ -343,18 +350,7 @@ func (s *Server) routes() *http.ServeMux {
|
||||
m.HandleFunc("PUT /api/prefs/sampler", s.auth(s.putSampler))
|
||||
m.HandleFunc("GET /api/prefs/history", s.auth(s.getHistoryWindow))
|
||||
m.HandleFunc("PUT /api/prefs/history", s.auth(s.putHistoryWindow))
|
||||
m.HandleFunc("GET /api/kvms", s.auth(s.listKVMs))
|
||||
m.HandleFunc("POST /api/kvms", s.auth(s.addKVM))
|
||||
m.HandleFunc("DELETE /api/kvms/{id}", s.auth(s.removeKVM))
|
||||
m.HandleFunc("/kvm/{id}/{rest...}", s.auth(s.serveKVM))
|
||||
m.HandleFunc("/kvm/{id}", s.auth(s.serveKVM))
|
||||
// A path Ham Lab does not serve may still belong to a KVM's page, which asks for its
|
||||
// own assets by absolute path. Ham Lab's own routes matched first, so this can only
|
||||
// catch what would otherwise have been a 404.
|
||||
m.HandleFunc("/api/", s.auth(func(w http.ResponseWriter, r *http.Request) {
|
||||
if s.kvmFallback(w, r) {
|
||||
return
|
||||
}
|
||||
jsonErr(w, 404, "not found")
|
||||
}))
|
||||
m.HandleFunc("/", s.static)
|
||||
@@ -378,16 +374,10 @@ func sensorPath(p string) bool {
|
||||
func (s *Server) Handler() http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
h := w.Header()
|
||||
// The console must never be framed. Two exceptions: an embedded widget exists to be
|
||||
// (a page that can only read sensors, and only with a read-only token in its URL),
|
||||
// and a proxied KVM is framed by the console itself — 'none' forbids that even from
|
||||
// the same origin, so those paths say 'self' instead of opening up.
|
||||
switch {
|
||||
case r.URL.Path == "/embed/widget":
|
||||
case strings.HasPrefix(r.URL.Path, kvmPrefix):
|
||||
h.Set("X-Frame-Options", "SAMEORIGIN")
|
||||
h.Set("Content-Security-Policy", "frame-ancestors 'self'")
|
||||
default:
|
||||
// The console must never be framed. The one exception is an embedded widget, which
|
||||
// exists to be (a page that can only read sensors, and only with a read-only token in
|
||||
// its URL).
|
||||
if r.URL.Path != "/embed/widget" {
|
||||
h.Set("X-Frame-Options", "DENY")
|
||||
h.Set("Content-Security-Policy", "frame-ancestors 'none'")
|
||||
}
|
||||
@@ -2646,12 +2636,6 @@ func (s *Server) static(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
}
|
||||
// A path with no file of ours behind it may belong to a KVM's page, which asks for its
|
||||
// own assets by absolute path. Checked here rather than in the router because this
|
||||
// handler is the catch-all — and gated on a session, since this handler is public.
|
||||
if s.signedIn(r) && s.kvmFallback(w, r) {
|
||||
return
|
||||
}
|
||||
// SPA fallback
|
||||
w.Header().Set("Cache-Control", "no-cache")
|
||||
http.ServeFileFS(w, r, s.ui, "index.html")
|
||||
|
||||
+239
-7
@@ -10,10 +10,16 @@ package api
|
||||
// opens every render node, whatever groups the daemon's user has.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
@@ -25,6 +31,7 @@ import (
|
||||
|
||||
"hamlab/internal/media"
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const thumbSize = 256
|
||||
@@ -117,7 +124,9 @@ func describeCaps(c media.Caps) string {
|
||||
}
|
||||
|
||||
// make produces a preview, sharing the work when the same file is asked for twice at once.
|
||||
func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (media.Result, error) {
|
||||
// in is a path, or a URL the hub serves another machine's file at (kind then says what it
|
||||
// is); device is the one to decode on ("" any, "cpu" none).
|
||||
func (t *thumbSvc) make(ctx context.Context, root bool, key, in, kind, device string) (media.Result, error) {
|
||||
t.mu.Lock()
|
||||
if c, ok := t.calls[key]; ok {
|
||||
t.mu.Unlock()
|
||||
@@ -153,19 +162,29 @@ func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (media
|
||||
c.err = err
|
||||
return c.res, c.err
|
||||
}
|
||||
o := media.Opts{Size: thumbSize, Timeout: thumbTimeout}
|
||||
o := media.Opts{Size: thumbSize, Timeout: thumbTimeout, Kind: kind}
|
||||
caps = caps.Only(device)
|
||||
if root {
|
||||
c.res, c.err = media.MakeRoot(wctx, t.self, caps, path, o)
|
||||
c.res, c.err = media.MakeRoot(wctx, t.self, caps, in, o)
|
||||
} else {
|
||||
c.res, c.err = media.Make(wctx, caps, path, o)
|
||||
c.res, c.err = media.Make(wctx, caps, in, o)
|
||||
}
|
||||
return c.res, c.err
|
||||
}
|
||||
|
||||
// ---- handlers ----
|
||||
|
||||
// filesThumb: GET /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1]. v is the client's idea of
|
||||
// the file, from its listing; it makes a cache hit free of a stat, and a changed file a new URL.
|
||||
// thumbDeviceHeader carries the device a hub wants a video preview made on: the Transcode
|
||||
// device setting, when it is on the machine the file is on.
|
||||
const thumbDeviceHeader = "X-Hamlab-Device"
|
||||
|
||||
// filesThumb: GET /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1][&cached=1]. v is the
|
||||
// client's idea of the file, from its listing; it makes a cache hit free of a stat, and a
|
||||
// changed file a new URL. cached=1 answers from the cache or 404, and makes nothing: the hub
|
||||
// asks that before having the preview made on another machine.
|
||||
//
|
||||
// A video preview is made on the Transcode device: on the hub, by the setting (another
|
||||
// machine's device reads the file through the hub); on a node, by the hub's header.
|
||||
func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
|
||||
fsys, ok := s.fsFor(w, r)
|
||||
if !ok {
|
||||
@@ -203,7 +222,28 @@ func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
|
||||
if s.serveCachedThumb(w, key) {
|
||||
return
|
||||
}
|
||||
res, err := s.thumbs.make(r.Context(), root, key, p)
|
||||
if r.URL.Query().Get("cached") == "1" {
|
||||
jsonErr(w, 404, "not made yet")
|
||||
return
|
||||
}
|
||||
device := r.Header.Get(thumbDeviceHeader)
|
||||
if c := s.transcodeChoice(); c != nil && device == "" && media.Kind(p) == "video" {
|
||||
if c.Node != "" {
|
||||
// The hub's own file, previewed on another machine.
|
||||
if res, ok := s.thumbElsewhere(r.Context(), c, "", p, root, key); ok {
|
||||
if res.Failure != "" {
|
||||
_ = s.thumbs.cache.Put(key, media.Entry{Path: p, Failure: res.Failure})
|
||||
jsonErr(w, 422, res.Failure)
|
||||
return
|
||||
}
|
||||
s.keepThumb(w, key, p, res)
|
||||
return
|
||||
}
|
||||
} else {
|
||||
device = c.Device
|
||||
}
|
||||
}
|
||||
res, err := s.thumbs.make(r.Context(), root, key, p, "", device)
|
||||
if err != nil {
|
||||
var de *media.DecodeError
|
||||
switch {
|
||||
@@ -223,12 +263,204 @@ func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
return
|
||||
}
|
||||
s.keepThumb(w, key, p, res)
|
||||
}
|
||||
|
||||
// keepThumb puts a preview in the cache and answers with it.
|
||||
func (s *Server) keepThumb(w http.ResponseWriter, key, p string, res media.Result) {
|
||||
if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}); err != nil {
|
||||
log.Printf("thumbs: cache: %v", err)
|
||||
}
|
||||
writeThumb(w, key, res.JPEG, res.Engine)
|
||||
}
|
||||
|
||||
// ---- previews made on another machine (the Transcode device) ----
|
||||
|
||||
// thumbFrom: POST /api/files/thumb-from {url, kind, key, device} — a preview of a file on
|
||||
// another machine, read through the hub at url, made here on device. Answers the JPEG (the
|
||||
// engine in X-Hamlab-Thumb), or 422 with why the file can't be previewed.
|
||||
func (s *Server) thumbFrom(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ URL, Kind, Key, Device string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 8192)).Decode(&body); err != nil || !strings.HasPrefix(body.URL, "http") || body.Key == "" {
|
||||
jsonErr(w, 400, "expected {url, kind, key, device}")
|
||||
return
|
||||
}
|
||||
res, err := s.thumbs.make(r.Context(), false, "from:"+body.Key, body.URL, body.Kind, body.Device)
|
||||
if err != nil {
|
||||
var de *media.DecodeError
|
||||
if errors.As(err, &de) {
|
||||
jsonErr(w, 422, de.Msg)
|
||||
} else {
|
||||
jsonErr(w, 503, err.Error())
|
||||
}
|
||||
return
|
||||
}
|
||||
w.Header().Set("X-Hamlab-Thumb-Codec", res.Codec)
|
||||
w.Header().Set("X-Hamlab-Thumb-Size", fmt.Sprintf("%dx%d", res.Width, res.Height))
|
||||
writeThumb(w, body.Key, res.JPEG, res.Engine)
|
||||
}
|
||||
|
||||
// thumbPut: PUT /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1] — a preview another machine
|
||||
// made of this file, kept here as if made here. The body is the JPEG, or nothing with
|
||||
// X-Hamlab-Thumb-Failure saying why the file can't be previewed. Refused if the file has
|
||||
// changed since v.
|
||||
func (s *Server) thumbPut(w http.ResponseWriter, r *http.Request) {
|
||||
fsys, ok := s.fsFor(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
p, ok := cleanQuery(w, r, "path")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
mt, sz, ok := parseV(r.URL.Query().Get("v"))
|
||||
if !ok {
|
||||
jsonErr(w, 400, "v=<mtime>-<size> is required")
|
||||
return
|
||||
}
|
||||
in, err := fsys.Stat(r.Context(), p)
|
||||
if err != nil {
|
||||
fsErr(w, err)
|
||||
return
|
||||
}
|
||||
if in.Size != sz || in.Mtime != mt {
|
||||
jsonErr(w, 409, "the file has changed")
|
||||
return
|
||||
}
|
||||
key := media.Key(p, sz, mt)
|
||||
e := media.Entry{Path: p, Engine: r.Header.Get("X-Hamlab-Thumb"), Codec: r.Header.Get("X-Hamlab-Thumb-Codec"), Failure: r.Header.Get("X-Hamlab-Thumb-Failure")}
|
||||
fmt.Sscanf(r.Header.Get("X-Hamlab-Thumb-Size"), "%dx%d", &e.Width, &e.Height)
|
||||
if e.Failure == "" {
|
||||
if e.JPEG, err = io.ReadAll(io.LimitReader(r.Body, 4<<20)); err != nil || len(e.JPEG) == 0 {
|
||||
jsonErr(w, 400, "no picture")
|
||||
return
|
||||
}
|
||||
}
|
||||
if err := s.thumbs.cache.Put(key, e); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// thumbElsewhere has a video preview made on the Transcode device's machine c.Node, of a
|
||||
// file on machine node ("" the hub), and answers it. ok is false when that machine can't
|
||||
// (unreachable, an older Ham Lab): the caller makes it where the file is instead. A file
|
||||
// that can't be previewed is a result too, with the failure in it.
|
||||
func (s *Server) thumbElsewhere(ctx context.Context, c *store.Transcode, node, p string, root bool, key string) (media.Result, bool) {
|
||||
src, err := s.srcURL(c.Node, node, p, root)
|
||||
if err != nil {
|
||||
return media.Result{}, false
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"url": src, "kind": "video", "key": key, "device": c.Device})
|
||||
ctx, cancel := context.WithTimeout(ctx, thumbTimeout+10*time.Second)
|
||||
defer cancel()
|
||||
var code int
|
||||
var b []byte
|
||||
var hdr http.Header
|
||||
if c.Node == "" {
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest("POST", "/api/files/thumb-from", bytes.NewReader(body)).WithContext(ctx)
|
||||
s.thumbFrom(rec, req)
|
||||
code, b, hdr = rec.Code, rec.Body.Bytes(), rec.Header()
|
||||
} else {
|
||||
code, b, hdr, err = s.nodes.DoHeaders(ctx, c.Node, "POST", "/api/files/thumb-from", body, http.Header{"Content-Type": []string{"application/json"}})
|
||||
if err != nil {
|
||||
log.Printf("thumbs: %s on %s: %v; made where it is", p, c.Node, err)
|
||||
return media.Result{}, false
|
||||
}
|
||||
}
|
||||
switch code {
|
||||
case 200:
|
||||
res := media.Result{JPEG: b, Engine: c.Node + " " + hdr.Get("X-Hamlab-Thumb"), Codec: hdr.Get("X-Hamlab-Thumb-Codec")}
|
||||
fmt.Sscanf(hdr.Get("X-Hamlab-Thumb-Size"), "%dx%d", &res.Width, &res.Height)
|
||||
if c.Node == "" {
|
||||
res.Engine = s.hostName() + " " + hdr.Get("X-Hamlab-Thumb")
|
||||
}
|
||||
return res, true
|
||||
case 422:
|
||||
var e struct{ Error string }
|
||||
_ = json.Unmarshal(b, &e)
|
||||
return media.Result{Failure: e.Error}, true
|
||||
}
|
||||
log.Printf("thumbs: %s on %s answered %d; made where it is", p, nodeLabel(c.Node), code)
|
||||
return media.Result{}, false
|
||||
}
|
||||
|
||||
// hubThumb: GET /api/nodes/{name}/files/thumb — a node's preview, through the hub. Without a
|
||||
// Transcode device setting, or for an image or a sound file's cover, it goes to the node as
|
||||
// any request does. A video is made on the setting's device: the node's own (told by header),
|
||||
// or another machine's, which reads the file through the hub; the node keeps the result in
|
||||
// its own cache either way.
|
||||
func (s *Server) hubThumb(w http.ResponseWriter, r *http.Request) {
|
||||
node := r.PathValue("name")
|
||||
proxy, ok := s.nodes.Proxy(node)
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such machine")
|
||||
return
|
||||
}
|
||||
c := s.transcodeChoice()
|
||||
q := r.URL.Query()
|
||||
p := q.Get("path")
|
||||
if c == nil || media.Kind(p) != "video" {
|
||||
proxy.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
if c.Node == node {
|
||||
r.Header.Set(thumbDeviceHeader, c.Device)
|
||||
proxy.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
mt, sz, okV := parseV(q.Get("v"))
|
||||
if !okV {
|
||||
proxy.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
// The node's own cache first.
|
||||
cq := url.Values{"path": {p}, "v": {q.Get("v")}, "cached": {"1"}}
|
||||
if q.Get("sudo") == "1" {
|
||||
cq.Set("sudo", "1")
|
||||
}
|
||||
hdr := http.Header{}
|
||||
if inm := r.Header.Get("If-None-Match"); inm != "" {
|
||||
hdr.Set("If-None-Match", inm)
|
||||
}
|
||||
code, b, rh, err := s.nodes.DoHeaders(r.Context(), node, "GET", "/api/files/thumb?"+cq.Encode(), nil, hdr)
|
||||
if err == nil && code != 404 {
|
||||
for _, k := range []string{"Content-Type", "Cache-Control", "ETag", "X-Hamlab-Thumb"} {
|
||||
if v := rh.Get(k); v != "" {
|
||||
w.Header().Set(k, v)
|
||||
}
|
||||
}
|
||||
w.WriteHeader(code)
|
||||
_, _ = w.Write(b)
|
||||
return
|
||||
}
|
||||
key := media.Key(p, sz, mt)
|
||||
res, made := s.thumbElsewhere(r.Context(), c, node, p, q.Get("sudo") == "1", key)
|
||||
if !made {
|
||||
proxy.ServeHTTP(w, r) // the node makes it itself
|
||||
return
|
||||
}
|
||||
// Kept where the file is, so it is made once.
|
||||
ph := http.Header{"X-Hamlab-Thumb": {res.Engine}, "X-Hamlab-Thumb-Codec": {res.Codec}, "X-Hamlab-Thumb-Size": {fmt.Sprintf("%dx%d", res.Width, res.Height)}}
|
||||
if res.Failure != "" {
|
||||
ph.Set("X-Hamlab-Thumb-Failure", res.Failure)
|
||||
}
|
||||
pq := url.Values{"path": {p}, "v": {q.Get("v")}}
|
||||
if q.Get("sudo") == "1" {
|
||||
pq.Set("sudo", "1")
|
||||
}
|
||||
if code, _, _, err := s.nodes.DoHeaders(r.Context(), node, "PUT", "/api/files/thumb?"+pq.Encode(), res.JPEG, ph); err != nil || code != 204 {
|
||||
log.Printf("thumbs: keeping %s's preview of %s: %d %v", node, p, code, err)
|
||||
}
|
||||
if res.Failure != "" {
|
||||
jsonErr(w, 422, res.Failure)
|
||||
return
|
||||
}
|
||||
writeThumb(w, key, res.JPEG, res.Engine)
|
||||
}
|
||||
|
||||
// serveCachedThumb answers from the cache when it can. A cache that cannot be read is a
|
||||
// miss: the preview is made again rather than refused.
|
||||
func (s *Server) serveCachedThumb(w http.ResponseWriter, key string) bool {
|
||||
|
||||
+8
-2
@@ -27,6 +27,7 @@ import (
|
||||
|
||||
type zfsState struct {
|
||||
Policies []storage.Policy `json:"policies"`
|
||||
Verify []storage.VerifyPolicy `json:"verify"` // landing-tier verify schedules (landing.go)
|
||||
}
|
||||
|
||||
func (s *Server) zfsLoad() zfsState {
|
||||
@@ -39,6 +40,9 @@ func (s *Server) zfsLoad() zfsState {
|
||||
if z.Policies == nil {
|
||||
z.Policies = []storage.Policy{}
|
||||
}
|
||||
if z.Verify == nil {
|
||||
z.Verify = []storage.VerifyPolicy{}
|
||||
}
|
||||
return z
|
||||
}
|
||||
|
||||
@@ -138,11 +142,13 @@ func (s *Server) zfsOverview(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// zfsARC is the cache: the sampler's live ARC (size, the hit rate over the last interval)
|
||||
// beside the storage package's fuller picture (since-boot counters, tunables, and each
|
||||
// pool's cache/log/special devices or the lack of them).
|
||||
// pool's cache/log/special devices or the lack of them), and the write path — the dirty
|
||||
// buffer in RAM, each pool's write rate, and any landing tier in front of a pool.
|
||||
func (s *Server) zfsARC(w http.ResponseWriter, r *http.Request) {
|
||||
out := map[string]any{"detail": s.storage.ARC(r.Context()), "live": nil}
|
||||
out := map[string]any{"detail": s.storage.ARC(r.Context()), "live": nil, "writes": nil}
|
||||
if smp, ok := s.metrics.Latest(); ok && smp.ARC != nil {
|
||||
out["live"] = smp.ARC
|
||||
out["writes"] = smp.ZFSWrite
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
@@ -77,23 +77,6 @@ type UpdateSource struct {
|
||||
Check string `json:"check,omitempty"`
|
||||
}
|
||||
|
||||
// KVM is an IP-KVM: a device with a web page that gives screen, keyboard and mouse access
|
||||
// to a machine, including when that machine has no OS running. Ham Lab holds the list on
|
||||
// the hub and proxies each one, because a KVM matters exactly when the machine it is
|
||||
// attached to cannot answer for itself.
|
||||
type KVM struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
// URL is the device's own address, as you would type it into a browser on the LAN.
|
||||
URL string `json:"url"`
|
||||
// Machine is the host this KVM is attached to, and is the key: one KVM per machine, shown
|
||||
// on that machine's KVM tab. Empty means the hub itself. The list lives on the hub even
|
||||
// for a node's KVM, because a KVM earns its keep when the machine is down and cannot
|
||||
// answer for itself.
|
||||
Machine string `json:"machine"`
|
||||
Added time.Time `json:"added"`
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
Listen string `json:"listen"`
|
||||
// PasswordHash is the single login of installs before users existed. On load it becomes
|
||||
@@ -103,7 +86,6 @@ type Config struct {
|
||||
TLS string `json:"tls,omitempty"` // "auto" → self-signed cert in the data dir; "" → plain HTTP
|
||||
Tokens []Token `json:"tokens,omitempty"`
|
||||
Nodes []Node `json:"nodes,omitempty"`
|
||||
KVMs []KVM `json:"kvms,omitempty"`
|
||||
// Updates: where a newer build comes from. See UpdateSource.
|
||||
Updates UpdateSource `json:"update,omitempty"`
|
||||
|
||||
|
||||
@@ -155,3 +155,17 @@ func TestPieceArgs(t *testing.T) {
|
||||
t.Error("no sound means -an")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCapsOnly(t *testing.T) {
|
||||
c := gridCaps()
|
||||
if got := c.Only("/dev/dri/renderD129"); len(got.GPUs) != 1 || got.GPUs[0].Name != "Intel Arc A380" || len(c.GPUs) != 2 {
|
||||
t.Fatalf("one card: %+v", got.GPUs)
|
||||
}
|
||||
if len(c.Only(CPU).GPUs) != 0 || len(c.Only("/dev/dri/renderD200").GPUs) != 0 || len(c.Only("").GPUs) != 2 {
|
||||
t.Fatal("cpu, a device not here, any")
|
||||
}
|
||||
// previews then go to that card, or the CPU
|
||||
if g := c.Only("/dev/dri/renderD129").pick("hevc"); g == nil || g.Device != "/dev/dri/renderD129" {
|
||||
t.Fatalf("pick: %+v", g)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -62,6 +62,24 @@ type Caps struct {
|
||||
type Opts struct {
|
||||
Size int // the long edge in pixels; 256 when zero
|
||||
Timeout time.Duration // for the whole job, ffprobe and every ffmpeg attempt; 30 s when zero
|
||||
Kind string // image | video | audio; from the name when empty (a URL has none)
|
||||
}
|
||||
|
||||
// Only is the caps with one device left to decode on: a render node, CPU for none, or ""
|
||||
// for all of them — the Transcode device setting, which previews follow too. A device this
|
||||
// machine doesn't have leaves the CPU.
|
||||
func (c Caps) Only(device string) Caps {
|
||||
if device == "" {
|
||||
return c
|
||||
}
|
||||
out := c
|
||||
out.GPUs = nil
|
||||
for _, g := range c.GPUs {
|
||||
if g.Device == device {
|
||||
out.GPUs = append(out.GPUs, g)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (o Opts) defaults() Opts {
|
||||
@@ -81,6 +99,8 @@ type Result struct {
|
||||
Codec string `json:"codec"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
// Failure is why the file can't be previewed, when another machine tried (no JPEG then).
|
||||
Failure string `json:"-"`
|
||||
}
|
||||
|
||||
// ErrUnsupported: not an image or a video by its name.
|
||||
@@ -356,7 +376,10 @@ func (a attempt) engine() string {
|
||||
|
||||
// Make produces the preview of one file.
|
||||
func Make(ctx context.Context, c Caps, path string, o Opts) (Result, error) {
|
||||
kind := Kind(path)
|
||||
kind := o.Kind
|
||||
if kind == "" {
|
||||
kind = Kind(path)
|
||||
}
|
||||
if kind == "" {
|
||||
return Result{}, ErrUnsupported
|
||||
}
|
||||
|
||||
@@ -95,30 +95,41 @@ func readRAPL(root string, dirs []string) ([]raplCount, bool) {
|
||||
// is carried across its range. One that went backwards with no known range, or a set of
|
||||
// counters that changed between readings, gives no answer rather than a wrong one.
|
||||
func raplWatts(prev, cur []raplCount, max map[string]uint64, dt float64) (float64, bool) {
|
||||
if dt <= 0 || len(prev) == 0 || len(prev) != len(cur) {
|
||||
return 0, false
|
||||
each, ok := raplWattsEach(prev, cur, max, dt)
|
||||
var w float64
|
||||
for _, x := range each {
|
||||
w += x
|
||||
}
|
||||
var uj float64
|
||||
return w, ok
|
||||
}
|
||||
|
||||
// raplWattsEach is raplWatts per counter: one package each.
|
||||
func raplWattsEach(prev, cur []raplCount, max map[string]uint64, dt float64) ([]float64, bool) {
|
||||
if dt <= 0 || len(prev) == 0 || len(prev) != len(cur) {
|
||||
return nil, false
|
||||
}
|
||||
out := make([]float64, len(cur))
|
||||
for i := range cur {
|
||||
if cur[i].Dir != prev[i].Dir {
|
||||
return 0, false
|
||||
return nil, false
|
||||
}
|
||||
if cur[i].UJ >= prev[i].UJ {
|
||||
uj += float64(cur[i].UJ - prev[i].UJ)
|
||||
out[i] = float64(cur[i].UJ-prev[i].UJ) / 1e6 / dt
|
||||
continue
|
||||
}
|
||||
m := max[cur[i].Dir]
|
||||
if m == 0 || prev[i].UJ > m {
|
||||
return 0, false
|
||||
return nil, false
|
||||
}
|
||||
uj += float64(m - prev[i].UJ + cur[i].UJ)
|
||||
out[i] = float64(m-prev[i].UJ+cur[i].UJ) / 1e6 / dt
|
||||
}
|
||||
return uj / 1e6 / dt, true
|
||||
return out, true
|
||||
}
|
||||
|
||||
// cpuPower is the CPU packages' power since the previous sample, in watts; 0 when this
|
||||
// machine has no counters or no way to read them.
|
||||
func (c *Collector) cpuPower(now time.Time) float64 {
|
||||
// machine has no counters or no way to read them. On a machine with more than one package
|
||||
// it also gives each one's, indexed by package id (the socket).
|
||||
func (c *Collector) cpuPower(now time.Time) (float64, []float64) {
|
||||
p := &c.power
|
||||
if !p.probed {
|
||||
p.probed = true
|
||||
@@ -131,7 +142,7 @@ func (c *Collector) cpuPower(now time.Time) float64 {
|
||||
}
|
||||
}
|
||||
if len(p.dirs) == 0 {
|
||||
return 0
|
||||
return 0, nil
|
||||
}
|
||||
var cur []raplCount
|
||||
ok := false
|
||||
@@ -148,14 +159,40 @@ func (c *Collector) cpuPower(now time.Time) float64 {
|
||||
}
|
||||
if !ok {
|
||||
p.prev = nil
|
||||
return 0
|
||||
return 0, nil
|
||||
}
|
||||
w, good := raplWatts(p.prev, cur, p.max, now.Sub(p.prevAt).Seconds())
|
||||
each, good := raplWattsEach(p.prev, cur, p.max, now.Sub(p.prevAt).Seconds())
|
||||
p.prev, p.prevAt = cur, now
|
||||
if !good {
|
||||
return 0
|
||||
return 0, nil
|
||||
}
|
||||
return round1(w)
|
||||
var w float64
|
||||
for _, x := range each {
|
||||
w += x
|
||||
}
|
||||
return round1(w), perPackage(cur, each)
|
||||
}
|
||||
|
||||
// perPackage puts each counter's watts at its package's id — intel-rapl:1 is "package-1" —
|
||||
// and is nil for a single package, whose figure is the total.
|
||||
func perPackage(cur []raplCount, each []float64) []float64 {
|
||||
if len(cur) < 2 {
|
||||
return nil
|
||||
}
|
||||
var out []float64
|
||||
for i, c := range cur {
|
||||
id := i
|
||||
if n, ok := strings.CutPrefix(strings.TrimSpace(readStr(filepath.Join(raplRoot, c.Dir, "name"))), "package-"); ok {
|
||||
if v, err := strconv.Atoi(n); err == nil && v >= 0 && v < 64 {
|
||||
id = v
|
||||
}
|
||||
}
|
||||
for len(out) <= id {
|
||||
out = append(out, 0)
|
||||
}
|
||||
out[id] = round1(each[i])
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// energyHelper is the root child that answers each newline with the package counters, one
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
package metrics
|
||||
|
||||
import "testing"
|
||||
|
||||
// a two-socket board: two coretemp chips, and the CPU's temperature is the hotter package
|
||||
func TestCPUTempHottestPackage(t *testing.T) {
|
||||
temps := []Temp{
|
||||
{Chip: "coretemp", Label: "Core 0", C: 31},
|
||||
{Chip: "coretemp", Label: "Core 0", C: 44, Pkg: 1},
|
||||
{Chip: "coretemp", Label: "Package id 0", C: 35},
|
||||
{Chip: "coretemp", Label: "Package id 1", C: 41, Pkg: 1},
|
||||
}
|
||||
if got := cpuTemp(temps); got != 41 {
|
||||
t.Fatalf("cpuTemp = %v, want 41", got)
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,7 @@
|
||||
package metrics
|
||||
|
||||
import "strconv"
|
||||
|
||||
// One kept history point is the average of the samples behind it. At the default ten-minute
|
||||
// window that bucket is a single sample and nothing is averaged; at a day it is five minutes
|
||||
// of them, and the difference matters — a graph of the last day drawn from whichever instant
|
||||
@@ -43,6 +45,21 @@ func fold(b []Sample) Sample {
|
||||
if pn > 0 {
|
||||
out.CPUPower = round1(pw / pn)
|
||||
}
|
||||
// and each package's, the same way
|
||||
out.CPUPowerPkgs = nil
|
||||
for _, s := range b {
|
||||
for i, w := range s.CPUPowerPkgs {
|
||||
for len(out.CPUPowerPkgs) <= i {
|
||||
out.CPUPowerPkgs = append(out.CPUPowerPkgs, 0)
|
||||
}
|
||||
out.CPUPowerPkgs[i] += w
|
||||
}
|
||||
}
|
||||
if pn > 0 {
|
||||
for i := range out.CPUPowerPkgs {
|
||||
out.CPUPowerPkgs[i] = round1(out.CPUPowerPkgs[i] / pn)
|
||||
}
|
||||
}
|
||||
out.Load[0] = sum(func(s Sample) float64 { return s.Load[0] })
|
||||
out.Load[1] = sum(func(s Sample) float64 { return s.Load[1] })
|
||||
out.Load[2] = sum(func(s Sample) float64 { return s.Load[2] })
|
||||
@@ -70,13 +87,13 @@ func fold(b []Sample) Sample {
|
||||
temps := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Temps))
|
||||
for _, x := range s.Temps {
|
||||
m[x.Chip+"/"+x.Label] = []float64{x.C}
|
||||
m[x.Chip+"/"+x.Label+"/"+strconv.Itoa(x.Pkg)] = []float64{x.C}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Temps = append([]Temp(nil), out.Temps...)
|
||||
for i, x := range out.Temps {
|
||||
if v, ok := temps[x.Chip+"/"+x.Label]; ok {
|
||||
if v, ok := temps[x.Chip+"/"+x.Label+"/"+strconv.Itoa(x.Pkg)]; ok {
|
||||
out.Temps[i].C = v[0]
|
||||
}
|
||||
}
|
||||
@@ -122,9 +139,54 @@ func fold(b []Sample) Sample {
|
||||
out.Disks[i].ReadBps, out.Disks[i].WriteBps, out.Disks[i].UtilPct = v[0], v[1], v[2]
|
||||
}
|
||||
}
|
||||
out.ZFSWrite = foldZFSWrites(b, out.ZFSWrite)
|
||||
return out
|
||||
}
|
||||
|
||||
// foldZFSWrites averages the write path's rates and the buffer's fill over a bucket, per pool;
|
||||
// everything else (limits, the mover's state, the last walk) is the newest sample's.
|
||||
func foldZFSWrites(b []Sample, last *ZFSWrites) *ZFSWrites {
|
||||
if last == nil {
|
||||
return nil
|
||||
}
|
||||
w := *last
|
||||
w.Pools = append([]PoolWrites(nil), last.Pools...)
|
||||
var dirty, delayed, over, n float64
|
||||
for _, s := range b {
|
||||
if s.ZFSWrite != nil {
|
||||
dirty, delayed, over, n = dirty+float64(s.ZFSWrite.Dirty), delayed+s.ZFSWrite.DelayedPerSec, over+s.ZFSWrite.OverMaxPerSec, n+1
|
||||
}
|
||||
}
|
||||
if n > 0 {
|
||||
w.Dirty, w.DelayedPerSec, w.OverMaxPerSec = uint64(dirty/n), round1(delayed/n), round1(over/n)
|
||||
}
|
||||
per := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := map[string][]float64{}
|
||||
if s.ZFSWrite == nil {
|
||||
return m
|
||||
}
|
||||
for _, p := range s.ZFSWrite.Pools {
|
||||
v := []float64{p.WriteBps, p.LogicalBps, 0, 0}
|
||||
if p.Landing != nil {
|
||||
v[2], v[3] = p.Landing.InBps, p.Landing.DrainBps
|
||||
}
|
||||
m[p.Pool] = v
|
||||
}
|
||||
return m
|
||||
})
|
||||
for i, p := range w.Pools {
|
||||
if v, ok := per[p.Pool]; ok {
|
||||
w.Pools[i].WriteBps, w.Pools[i].LogicalBps = round1(v[0]), round1(v[1])
|
||||
if p.Landing != nil {
|
||||
l := *p.Landing
|
||||
l.InBps, l.DrainBps = round1(v[2]), round1(v[3])
|
||||
w.Pools[i].Landing = &l
|
||||
}
|
||||
}
|
||||
}
|
||||
return &w
|
||||
}
|
||||
|
||||
// avgBy averages each sample's named numbers across the bucket, per name.
|
||||
func avgBy(b []Sample, of func(Sample) map[string][]float64) map[string][]float64 {
|
||||
sums := map[string][]float64{}
|
||||
|
||||
+77
-20
@@ -49,6 +49,7 @@ type GPU struct {
|
||||
PowerW float64 `json:"power_w,omitempty"`
|
||||
PowerCapW float64 `json:"power_cap_w,omitempty"`
|
||||
FanRPM int `json:"fan_rpm,omitempty"`
|
||||
FanPct *float64 `json:"fan_pct,omitempty"` // fan speed as a % of its range: nvidia-smi's fan.speed, or the hwmon's pwm1; nil when the card has no such reading (0 is a stopped fan)
|
||||
MHz float64 `json:"mhz,omitempty"` // actual clock (0 when power-gated)
|
||||
ReqMHz float64 `json:"req_mhz,omitempty"` // what the driver asked for
|
||||
MaxMHz float64 `json:"max_mhz,omitempty"`
|
||||
@@ -56,7 +57,7 @@ type GPU struct {
|
||||
MemTotal uint64 `json:"mem_total,omitempty"`
|
||||
Clients []GPUClient `json:"clients"`
|
||||
ClientsOK bool `json:"clients_ok"` // false when fdinfo of other users' processes could not be read
|
||||
Stalled bool `json:"stalled,omitempty"` // a sysfs read of this card never returned; its readings are paused until restart
|
||||
Stalled bool `json:"stalled,omitempty"` // a read of this card has been out for gpuStallAfter; its readings are paused until it returns
|
||||
}
|
||||
|
||||
// GPUClient is one process with the device open.
|
||||
@@ -79,6 +80,16 @@ type gpuDev struct {
|
||||
prevIdle, prevEnergy float64
|
||||
prevAt time.Time
|
||||
nv []GPUClient // nvidia: the process table from the last read (pmon), fdinfo says nothing there
|
||||
nvRead []GPUClient // the same, as the read in flight left it: only that read touches it
|
||||
pending chan gpuRead // the read in flight, nil when none is
|
||||
pendingAt time.Time
|
||||
last *GPU // the last read that came back
|
||||
readFn func(*GPU, time.Time) // nil is read (tests stand a slow one in)
|
||||
}
|
||||
|
||||
type gpuRead struct {
|
||||
x GPU
|
||||
nv []GPUClient
|
||||
}
|
||||
|
||||
type gpuState struct {
|
||||
@@ -348,28 +359,60 @@ func lspciName(desc string) string {
|
||||
return strings.TrimSpace(vendor + " " + desc)
|
||||
}
|
||||
|
||||
// readWithin is read with a deadline. A sysfs read that does not come back must not take
|
||||
// the whole sampler with it — every graph on every page of this machine stops — so the
|
||||
// device is marked stalled, its goroutine abandoned, and the rest of the sample goes on.
|
||||
const gpuReadWithin = 3 * time.Second
|
||||
// readWithin is read with a deadline. A read that does not come back must not take the
|
||||
// whole sampler with it — every graph on every page of this machine stops — so one still out
|
||||
// after gpuReadWithin is left running, the sample goes on with the card's last readings, and
|
||||
// no second read starts until it returns. Late is normal for nvidia-smi on a busy card; only a
|
||||
// read out for gpuStallAfter marks the card stalled, and that clears the moment it returns. A
|
||||
// sysfs read that never returns keeps it stalled, at the cost of one goroutine.
|
||||
var (
|
||||
gpuReadWithin = 3 * time.Second
|
||||
gpuStallAfter = 15 * time.Second // past both of readNvidia's nvidia-smi calls timing out (runOut, 5 s each)
|
||||
)
|
||||
|
||||
func (d *gpuDev) readWithin(x *GPU, now time.Time) {
|
||||
var r gpuRead
|
||||
back := false
|
||||
if d.pending == nil {
|
||||
done := make(chan gpuRead, 1)
|
||||
y := *x
|
||||
read := d.read
|
||||
if d.readFn != nil {
|
||||
read = d.readFn
|
||||
}
|
||||
go func() {
|
||||
read(&y, now)
|
||||
done <- gpuRead{y, d.nvRead}
|
||||
}()
|
||||
d.pending, d.pendingAt = done, now
|
||||
select {
|
||||
case r, back = <-done:
|
||||
case <-time.After(gpuReadWithin):
|
||||
}
|
||||
} else { // started on an earlier sample: taken if it is back, never waited on again
|
||||
select {
|
||||
case r, back = <-d.pending:
|
||||
default:
|
||||
}
|
||||
}
|
||||
ok := x.ClientsOK
|
||||
defer func() { x.ClientsOK = ok }()
|
||||
if back {
|
||||
if d.stalled {
|
||||
x.Stalled = true
|
||||
log.Printf("gpu: %s (%s): its read came back after %s; readings resume", d.name, d.card, now.Sub(d.pendingAt).Round(time.Second))
|
||||
}
|
||||
d.pending, d.nv, d.stalled, d.last = nil, r.nv, false, &r.x
|
||||
*x = r.x
|
||||
return
|
||||
}
|
||||
done := make(chan GPU, 1)
|
||||
go func() {
|
||||
y := *x
|
||||
d.read(&y, now)
|
||||
done <- y
|
||||
}()
|
||||
select {
|
||||
case y := <-done:
|
||||
*x = y
|
||||
case <-time.After(gpuReadWithin):
|
||||
if d.last != nil {
|
||||
*x = *d.last
|
||||
}
|
||||
if now.Sub(d.pendingAt) >= gpuStallAfter {
|
||||
if !d.stalled {
|
||||
log.Printf("gpu: %s (%s): a read has not returned in %s; its readings are paused until it does", d.name, d.card, gpuStallAfter)
|
||||
}
|
||||
d.stalled, x.Stalled = true, true
|
||||
log.Printf("gpu: %s (%s): a sysfs read did not return within %s; its readings are paused until restart", d.name, d.card, gpuReadWithin)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -437,6 +480,14 @@ func (d *gpuDev) read(x *GPU, now time.Time) {
|
||||
x.PowerCapW = p / 1e6
|
||||
}
|
||||
x.FanRPM = int(readF(d.hwmon + "/fan1_input"))
|
||||
if pwm, err := strconv.ParseFloat(strings.TrimSpace(readStr(d.hwmon+"/pwm1")), 64); err == nil {
|
||||
top := readF(d.hwmon + "/pwm1_max")
|
||||
if top <= 0 {
|
||||
top = 255
|
||||
}
|
||||
v := round1(pwm / top * 100)
|
||||
x.FanPct = &v
|
||||
}
|
||||
}
|
||||
d.prevAt = now
|
||||
}
|
||||
@@ -720,11 +771,11 @@ func ueventValue(p, key string) string {
|
||||
// sample (~20 ms and ~75 ms on a GTX 965M), no root needed. The PCI address selects the
|
||||
// card, so two NVIDIA GPUs stay apart.
|
||||
func (d *gpuDev) readNvidia(x *GPU) {
|
||||
if out, ok := runOut("nvidia-smi", "--id="+d.bus, "--query-gpu=utilization.gpu,utilization.encoder,utilization.decoder,memory.used,memory.total,clocks.sm,clocks.max.sm,temperature.gpu,power.draw,power.limit", "--format=csv,noheader,nounits"); ok {
|
||||
if out, ok := runOut("nvidia-smi", "--id="+d.bus, "--query-gpu=utilization.gpu,utilization.encoder,utilization.decoder,memory.used,memory.total,clocks.sm,clocks.max.sm,temperature.gpu,power.draw,power.limit,fan.speed", "--format=csv,noheader,nounits"); ok {
|
||||
parseNvidiaQuery(x, out)
|
||||
}
|
||||
if out, ok := runOut("nvidia-smi", "pmon", "-c", "1", "-s", "um", "--id="+d.bus); ok {
|
||||
d.nv = parseNvidiaPmon(out)
|
||||
d.nvRead = parseNvidiaPmon(out)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -739,7 +790,8 @@ func nvNum(s string) (float64, bool) {
|
||||
}
|
||||
|
||||
// parseNvidiaQuery fills a GPU from one --query-gpu line in the field order readNvidia asks
|
||||
// for: gpu%, encoder%, decoder%, mem used MiB, mem total MiB, SM MHz, max SM MHz, °C, W, W cap.
|
||||
// for: gpu%, encoder%, decoder%, mem used MiB, mem total MiB, SM MHz, max SM MHz, °C, W, W cap,
|
||||
// fan % ("[N/A]" on a card with no fan of its own: a laptop's, a passive datacentre card).
|
||||
func parseNvidiaQuery(x *GPU, line string) {
|
||||
f := strings.Split(strings.TrimSpace(line), ",")
|
||||
if len(f) < 10 {
|
||||
@@ -775,6 +827,11 @@ func parseNvidiaQuery(x *GPU, line string) {
|
||||
if v, ok := nvNum(f[9]); ok {
|
||||
x.PowerCapW = v
|
||||
}
|
||||
if len(f) > 10 {
|
||||
if v, ok := nvNum(f[10]); ok {
|
||||
x.FanPct = &v
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// parseNvidiaPmon reads `nvidia-smi pmon -c 1 -s um`: a header naming the columns, then one
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A read that comes back late carries the last readings meanwhile and is used when it lands;
|
||||
// only one out past gpuStallAfter marks the card stalled, and its return clears that.
|
||||
func TestGPUReadLateThenStalledThenBack(t *testing.T) {
|
||||
defer func(w, s time.Duration) { gpuReadWithin, gpuStallAfter = w, s }(gpuReadWithin, gpuStallAfter)
|
||||
gpuReadWithin, gpuStallAfter = 20*time.Millisecond, 10*time.Second
|
||||
release := make(chan struct{})
|
||||
n := 0.0 // only the read goroutine writes it, one read at a time
|
||||
d := &gpuDev{name: "test", card: "card0"}
|
||||
d.readFn = func(x *GPU, _ time.Time) {
|
||||
n++
|
||||
if n == 2 {
|
||||
<-release
|
||||
}
|
||||
x.Usage = n * 10
|
||||
}
|
||||
t0 := time.Now()
|
||||
sample := func(at time.Duration) GPU {
|
||||
x := GPU{Name: "test"}
|
||||
d.readWithin(&x, t0.Add(at))
|
||||
return x
|
||||
}
|
||||
if x := sample(0); x.Usage != 10 || x.Stalled {
|
||||
t.Fatalf("first: %+v", x)
|
||||
}
|
||||
if x := sample(time.Second); x.Usage != 10 || x.Stalled { // late: the last readings
|
||||
t.Fatalf("late: %+v", x)
|
||||
}
|
||||
if x := sample(5 * time.Second); x.Usage != 10 || x.Stalled {
|
||||
t.Fatalf("still late: %+v", x)
|
||||
}
|
||||
if x := sample(12 * time.Second); !x.Stalled {
|
||||
t.Fatalf("out past gpuStallAfter, not stalled: %+v", x)
|
||||
}
|
||||
if x := sample(12*time.Second + 1); !x.Stalled {
|
||||
t.Fatal("stall did not hold")
|
||||
}
|
||||
close(release)
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
if x := sample(13 * time.Second); x.Usage != 20 || x.Stalled {
|
||||
t.Fatalf("back: %+v", x)
|
||||
}
|
||||
if x := sample(14 * time.Second); x.Usage != 30 || x.Stalled { // 30: no extra read ran while one was out
|
||||
t.Fatalf("next: %+v", x)
|
||||
}
|
||||
}
|
||||
@@ -182,6 +182,20 @@ func TestParseNvidiaQuery(t *testing.T) {
|
||||
if g.Temp != 57 || g.TempSrc != "gpu" || g.PowerW != 23.4 || g.PowerCapW != 0 {
|
||||
t.Errorf("temp/power (cap must stay 0 for [N/A]): %+v", g)
|
||||
}
|
||||
if g.FanPct != nil {
|
||||
t.Errorf("no fan field must leave the fan unset: %+v", g)
|
||||
}
|
||||
var fan, nofan GPU
|
||||
parseNvidiaQuery(&fan, "99, 0, 0, 20000, 24576, 1695, 2100, 83, 403.99, 420.00, 34\n")
|
||||
parseNvidiaQuery(&nofan, "5, 0, 0, 500, 2048, 300, 1097, 45, 8.0, [N/A], [N/A]\n")
|
||||
if fan.FanPct == nil || *fan.FanPct != 34 || nofan.FanPct != nil {
|
||||
t.Errorf("fan: %v, [N/A]: %v", fan.FanPct, nofan.FanPct)
|
||||
}
|
||||
var stopped GPU
|
||||
parseNvidiaQuery(&stopped, "0, 0, 0, 500, 24576, 210, 2100, 35, 20.0, 420.00, 0\n")
|
||||
if stopped.FanPct == nil || *stopped.FanPct != 0 {
|
||||
t.Errorf("a stopped fan must read 0, not nothing: %v", stopped.FanPct)
|
||||
}
|
||||
var short GPU
|
||||
parseNvidiaQuery(&short, "garbage")
|
||||
if short.UsageSrc != "" {
|
||||
|
||||
@@ -60,6 +60,7 @@ type Temp struct {
|
||||
C float64 `json:"c"`
|
||||
Max float64 `json:"max,omitempty"`
|
||||
Crit float64 `json:"crit,omitempty"`
|
||||
Pkg int `json:"pkg,omitempty"` // coretemp: the CPU package the chip belongs to — a two-socket board has two chips, both with a "Core 0"
|
||||
}
|
||||
|
||||
type Fan struct {
|
||||
@@ -101,6 +102,9 @@ type FS struct {
|
||||
// said honestly: a ZFS pool or an md array spreads a filesystem over several drives, and
|
||||
// a remote mount is on none of them.
|
||||
Disk string `json:"disk,omitempty"`
|
||||
// Source is the device the mount is of ("/dev/nvme0n1p1"). A bind mount has the same
|
||||
// source as the filesystem it shows, so a sum over a drive counts each source once.
|
||||
Source string `json:"source,omitempty"`
|
||||
}
|
||||
|
||||
type UPS struct {
|
||||
@@ -140,6 +144,7 @@ type Sample struct {
|
||||
CPU float64 `json:"cpu"`
|
||||
Cores []float64 `json:"cores"`
|
||||
CPUPower float64 `json:"cpu_power_w,omitempty"` // the CPU packages' own energy counters; 0 = no reading
|
||||
CPUPowerPkgs []float64 `json:"cpu_power_pkgs_w,omitempty"` // each package's share, by socket; only with more than one
|
||||
Load [3]float64 `json:"load"`
|
||||
Mem Mem `json:"mem"`
|
||||
Swap Mem `json:"swap"`
|
||||
@@ -151,6 +156,7 @@ type Sample struct {
|
||||
UPS *UPS `json:"ups"`
|
||||
GPUs []GPU `json:"gpus"`
|
||||
ARC *ARC `json:"arc"` // the ZFS read cache; nil without the module
|
||||
ZFSWrite *ZFSWrites `json:"zfs_writes"` // the ZFS write path; nil without the module
|
||||
Uptime float64 `json:"uptime"`
|
||||
|
||||
fsT, upsT int64 // when the slow sections were really read; sensors report these, not T
|
||||
@@ -182,6 +188,7 @@ type Collector struct {
|
||||
gpu gpuState
|
||||
power cpuPowerState
|
||||
prevARC arcPrev
|
||||
prevZW zfsWritePrev
|
||||
|
||||
subMu sync.Mutex
|
||||
subs map[chan struct{}]struct{} // told after every sample; see Subscribe
|
||||
@@ -347,7 +354,7 @@ func (c *Collector) sample() {
|
||||
|
||||
s := Sample{T: now.UnixMilli(), Interval: c.Interval().Milliseconds(), Temps: []Temp{}, Fans: []Fan{}, Net: []Net{}, Disks: []Disk{}}
|
||||
s.CPU, s.Cores = c.cpu()
|
||||
s.CPUPower = c.cpuPower(now)
|
||||
s.CPUPower, s.CPUPowerPkgs = c.cpuPower(now)
|
||||
s.Load = loadavg()
|
||||
s.Mem, s.Swap = meminfo()
|
||||
s.Uptime = uptime()
|
||||
@@ -368,6 +375,7 @@ func (c *Collector) sample() {
|
||||
s.UPS, s.upsT = c.lastUPS, c.upsAt.UnixMilli()
|
||||
s.GPUs = c.gpus(now, cpuTemp(s.Temps))
|
||||
s.ARC = c.arc(now)
|
||||
s.ZFSWrite = c.zfsWrites(now, s.ARC, s.Disks)
|
||||
|
||||
if first {
|
||||
return // deltas are meaningless on the priming pass
|
||||
@@ -541,6 +549,19 @@ func hwmonDir(d string) (chip string, temps []Temp, fans []Fan) {
|
||||
fans = append(fans, Fan{Chip: chip, Label: label, RPM: int(raw)})
|
||||
}
|
||||
}
|
||||
// coretemp is one chip per CPU package, and only its "Package id N" says which
|
||||
if chip == "coretemp" {
|
||||
for _, t := range temps {
|
||||
if n, ok := strings.CutPrefix(t.Label, "Package id "); ok {
|
||||
if pkg, err := strconv.Atoi(n); err == nil {
|
||||
for i := range temps {
|
||||
temps[i].Pkg = pkg
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
@@ -556,7 +577,10 @@ func hwmon() ([]Temp, []Fan) {
|
||||
if temps[i].Chip != temps[j].Chip {
|
||||
return temps[i].Chip < temps[j].Chip
|
||||
}
|
||||
if temps[i].Label != temps[j].Label {
|
||||
return temps[i].Label < temps[j].Label
|
||||
}
|
||||
return temps[i].Pkg < temps[j].Pkg
|
||||
})
|
||||
return temps, fans
|
||||
}
|
||||
@@ -842,7 +866,7 @@ func filesystems() []FS {
|
||||
return
|
||||
}
|
||||
bs := uint64(st.Bsize)
|
||||
done <- FS{Mount: mount, Type: fs[2], Size: st.Blocks * bs, Used: (st.Blocks - st.Bfree) * bs, Remote: remoteFS[fs[2]], Disk: diskOf(fs[0])}
|
||||
done <- FS{Mount: mount, Type: fs[2], Size: st.Blocks * bs, Used: (st.Blocks - st.Bfree) * bs, Remote: remoteFS[fs[2]], Disk: diskOf(fs[0]), Source: fs[0]}
|
||||
}()
|
||||
select {
|
||||
case r := <-done:
|
||||
@@ -1109,6 +1133,38 @@ func coreIDs(n int) []int {
|
||||
return coreIDsVal[:n]
|
||||
}
|
||||
|
||||
// corePkgs maps logical CPUs to the CPU package (socket) each is on, and is nil on a
|
||||
// one-socket machine: there is nothing to tell apart.
|
||||
var (
|
||||
corePkgsOnce sync.Once
|
||||
corePkgsVal []int
|
||||
)
|
||||
|
||||
func corePkgs(n int) []int {
|
||||
corePkgsOnce.Do(func() {
|
||||
seen := map[int]bool{}
|
||||
var out []int
|
||||
for i := 0; ; i++ {
|
||||
b, err := os.ReadFile("/sys/devices/system/cpu/cpu" + strconv.Itoa(i) + "/topology/physical_package_id")
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
id, err := strconv.Atoi(strings.TrimSpace(string(b)))
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
out, seen[id] = append(out, id), true
|
||||
}
|
||||
if len(seen) > 1 {
|
||||
corePkgsVal = out
|
||||
}
|
||||
})
|
||||
if len(corePkgsVal) < n {
|
||||
return nil
|
||||
}
|
||||
return corePkgsVal[:n]
|
||||
}
|
||||
|
||||
// ProcTable is the Processes tab payload.
|
||||
type ProcTable struct {
|
||||
T int64 `json:"t"`
|
||||
@@ -1119,8 +1175,10 @@ type ProcTable struct {
|
||||
CPUMHz float64 `json:"cpu_mhz,omitempty"`
|
||||
CPUMaxMHz float64 `json:"cpu_max_mhz,omitempty"`
|
||||
CPUPowerW float64 `json:"cpu_power_w,omitempty"`
|
||||
CPUPowerPkgsW []float64 `json:"cpu_power_pkgs_w,omitempty"`
|
||||
CPUTemp float64 `json:"cpu_temp,omitempty"`
|
||||
CoreIDs []int `json:"core_ids,omitempty"` // for each logical CPU, the physical core it is a thread of (what the temperature sensors are named by)
|
||||
CorePkgs []int `json:"core_pkgs,omitempty"` // for each logical CPU, the socket it is on; only on a machine with more than one
|
||||
Load [3]float64 `json:"load"`
|
||||
Mem Mem `json:"mem"`
|
||||
Swap Mem `json:"swap"`
|
||||
@@ -1179,18 +1237,24 @@ func cpuMHz() float64 {
|
||||
|
||||
// cpuTemp picks the sensor that best represents the CPU package.
|
||||
func cpuTemp(temps []Temp) float64 {
|
||||
best, max := 0.0, 0.0
|
||||
// the hottest package: a two-socket board has two, and the hotter one is the one to watch
|
||||
pkg, max := 0.0, 0.0
|
||||
for _, t := range temps {
|
||||
l := strings.ToLower(t.Label)
|
||||
if t.Chip == "coretemp" && strings.HasPrefix(l, "package") || t.Chip == "k10temp" && (l == "tctl" || l == "tdie") ||
|
||||
t.Chip == "thermal" && strings.Contains(l, "cpu") || t.Chip == "cpu_thermal" || t.Chip == "soc_thermal" {
|
||||
return t.C
|
||||
if t.C > pkg {
|
||||
pkg = t.C
|
||||
}
|
||||
}
|
||||
if (t.Chip == "coretemp" || t.Chip == "k10temp") && t.C > max {
|
||||
max, best = t.C, t.C
|
||||
max = t.C
|
||||
}
|
||||
}
|
||||
return best
|
||||
if pkg > 0 {
|
||||
return pkg
|
||||
}
|
||||
return max
|
||||
}
|
||||
|
||||
var bootTime = func() int64 {
|
||||
@@ -1318,8 +1382,9 @@ func (c *Collector) Processes() ProcTable {
|
||||
t.Mem, t.Swap = meminfo()
|
||||
if n := len(c.ring); n > 0 {
|
||||
last := c.ring[n-1]
|
||||
t.CPU, t.Cores, t.Net, t.CPUTemp, t.CPUPowerW = last.CPU, last.Cores, last.Net, cpuTemp(last.Temps), last.CPUPower
|
||||
t.CPU, t.Cores, t.Net, t.CPUTemp, t.CPUPowerW, t.CPUPowerPkgsW = last.CPU, last.Cores, last.Net, cpuTemp(last.Temps), last.CPUPower, last.CPUPowerPkgs
|
||||
t.CoreIDs = coreIDs(len(last.Cores))
|
||||
t.CorePkgs = corePkgs(len(last.Cores))
|
||||
from := n - 90
|
||||
if from < 0 {
|
||||
from = 0
|
||||
|
||||
@@ -65,6 +65,10 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
|
||||
if s.CPUPower > 0 {
|
||||
f.add("cpu.power", "CPU power", "power", "W", s.CPUPower, t, none)
|
||||
}
|
||||
for i, w := range s.CPUPowerPkgs {
|
||||
n := strconv.Itoa(i)
|
||||
f.add("cpu.package."+n+".power", "CPU "+strconv.Itoa(i+1)+" power", "power", "W", w, t, none, "package", n)
|
||||
}
|
||||
for i, w := range []string{"1", "5", "15"} {
|
||||
f.add("load."+w, "Load "+w+" min", "load", "", s.Load[i], t, none)
|
||||
}
|
||||
@@ -92,7 +96,11 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
|
||||
if x.Crit > 0 {
|
||||
b.crit = sensors.F(x.Crit)
|
||||
}
|
||||
f.add(sensors.ID("temp", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "temperature", "°C", x.C, t, b, "chip", x.Chip, "label", x.Label)
|
||||
labels := []string{"chip", x.Chip, "label", x.Label}
|
||||
if x.Chip == "coretemp" { // a two-socket board has a coretemp chip per CPU, both with a "Core 0"
|
||||
labels = append(labels, "package", strconv.Itoa(x.Pkg))
|
||||
}
|
||||
f.add(sensors.ID("temp", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "temperature", "°C", x.C, t, b, labels...)
|
||||
}
|
||||
for _, x := range s.Fans {
|
||||
f.add(sensors.ID("fan", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "fan", "rpm", float64(x.RPM), t, none, "chip", x.Chip, "label", x.Label)
|
||||
@@ -129,13 +137,22 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
|
||||
if x.Disk != "" {
|
||||
lbl = append(lbl, "disk", x.Disk)
|
||||
}
|
||||
if x.Source != "" {
|
||||
lbl = append(lbl, "source", x.Source)
|
||||
}
|
||||
f.add(p+".size", x.Mount+" size", "bytes", "B", float64(x.Size), s.fsT, none, lbl...)
|
||||
f.add(p+".used", x.Mount+" used", "bytes", "B", float64(x.Used), s.fsT, upTo(float64(x.Size)), lbl...)
|
||||
f.add(p+".used_pct", x.Mount+" used", "usage", "%", ratio(x.Used, x.Size), s.fsT, pct, lbl...)
|
||||
}
|
||||
|
||||
gpuSeen := map[string]int{}
|
||||
for _, g := range s.GPUs {
|
||||
// two identical cards are gpu.<name> and gpu.<name>_2, every sensor of the second under
|
||||
// its own stem (not add's suffix on the end of each id, which the page can't find)
|
||||
p := "gpu." + sensors.Seg(g.Name)
|
||||
if gpuSeen[p]++; gpuSeen[p] > 1 {
|
||||
p = fmt.Sprintf("%s_%d", p, gpuSeen[p])
|
||||
}
|
||||
lbl := []string{"name", g.Name, "driver", g.Driver}
|
||||
if g.Bus != "" {
|
||||
lbl = append(lbl, "bus", g.Bus)
|
||||
@@ -164,6 +181,9 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
|
||||
if g.FanRPM > 0 {
|
||||
f.add(p+".fan", g.Name+" fan", "fan", "rpm", float64(g.FanRPM), t, none, lbl...)
|
||||
}
|
||||
if g.FanPct != nil {
|
||||
f.add(p+".fan_pct", g.Name+" fan", "fan", "%", *g.FanPct, t, pct, lbl...)
|
||||
}
|
||||
if g.MaxMHz > 0 {
|
||||
f.add(p+".freq", g.Name+" clock", "frequency", "MHz", g.MHz, t, upTo(g.MaxMHz), lbl...)
|
||||
}
|
||||
@@ -203,6 +223,32 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
|
||||
}
|
||||
}
|
||||
|
||||
if w := s.ZFSWrite; w != nil {
|
||||
f.add("zfs.write.dirty", "ZFS write buffer", "bytes", "B", float64(w.Dirty), t, upTo(float64(w.Limit)))
|
||||
f.add("zfs.write.limit", "ZFS write buffer limit", "bytes", "B", float64(w.Limit), t, none)
|
||||
f.add("zfs.write.delayed", "ZFS writes throttled", "rate", "/s", w.DelayedPerSec, t, none)
|
||||
for _, p := range w.Pools {
|
||||
pp := "zfs." + sensors.Seg(p.Pool) + "."
|
||||
f.add(pp+"write_bps", p.Pool+" writes to disk", "rate", "B/s", p.WriteBps, t, none, "pool", p.Pool)
|
||||
f.add(pp+"logical_bps", p.Pool+" writes by programs", "rate", "B/s", p.LogicalBps, t, none, "pool", p.Pool)
|
||||
if l := p.Landing; l != nil {
|
||||
lp := pp + "landing."
|
||||
lbl := []string{"pool", p.Pool, "branch", l.Branch}
|
||||
f.add(lp+"pending_bytes", p.Pool+" landing: waiting", "bytes", "B", float64(l.PendingBytes), t, upTo(float64(l.Size)), lbl...)
|
||||
f.add(lp+"pending_files", p.Pool+" landing: files waiting", "count", "", float64(l.PendingFiles), t, none, lbl...)
|
||||
if l.Size > 0 {
|
||||
f.add(lp+"used_pct", p.Pool+" landing: used", "usage", "%", round1(float64(l.Used)/float64(l.Size)*100), t, pct, lbl...)
|
||||
}
|
||||
f.add(lp+"in_bps", p.Pool+" landing: arriving", "rate", "B/s", l.InBps, t, none, lbl...)
|
||||
f.add(lp+"drain_bps", p.Pool+" landing: moving to the pool", "rate", "B/s", l.DrainBps, t, none, lbl...)
|
||||
if m := l.Mover; m != nil {
|
||||
f.add(lp+"mover", p.Pool+" landing: mover", "state", "", m.State, t, none, lbl...)
|
||||
f.add(lp+"queue", p.Pool+" landing: mover queue", "count", "", float64(m.Queue), t, none, lbl...)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if u := s.UPS; u != nil {
|
||||
ut := s.upsT
|
||||
f.add("ups.battery", "UPS battery", "usage", "%", u.BatteryPct, ut, pct, "model", u.Model)
|
||||
|
||||
@@ -0,0 +1,456 @@
|
||||
package metrics
|
||||
|
||||
// The ZFS write path, for the ZFS tab: what is waiting in RAM to be written, how fast each
|
||||
// pool is writing to its disks, and — where a pool has one — the landing tier in front of it.
|
||||
//
|
||||
// A landing tier is a mergerfs pool whose first branches are fast non-ZFS storage and whose
|
||||
// last branch is a ZFS dataset: new files land on the fast branch and a mover drains them onto
|
||||
// the pool. It is detected from mergerfs's own branch list (the user.mergerfs.branches xattr on
|
||||
// the pool's .mergerfs control file), so it shows wherever one exists and nowhere else. A mover
|
||||
// that writes a status file into /run/landing-mover/ (hamlet's scripts/landing-mover.py does)
|
||||
// adds what it is doing: its queue, the file in flight, totals and the last error.
|
||||
//
|
||||
// Sources, all cheap enough for every sample:
|
||||
// - arcstats anon_size: dirty data in RAM, for all pools together (ZFS has no live per-pool
|
||||
// figure; txgs only records ndirty once a txg has synced)
|
||||
// - dmu_tx: dmu_tx_dirty_delay / dmu_tx_dirty_over_max — writers slowed by the write throttle,
|
||||
// or stopped at the hard limit
|
||||
// - <pool>/txgs: nwritten per synced txg — bytes the pool actually wrote to its vdevs
|
||||
// - <pool>/objset-*: nwritten per dataset — bytes programs wrote, before compression
|
||||
// The landing branch's file walk is slower and runs every landingEvery.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
zfsKstatDir = "/proc/spl/kstat/zfs/"
|
||||
zfsParamDir = "/sys/module/zfs/parameters/"
|
||||
moverStateDir = "/run/landing-mover"
|
||||
landingEvery = 10 * time.Second
|
||||
landingWalkMax = 200000 // entries; past this the counts are marked partial
|
||||
)
|
||||
|
||||
// ZFSWrites is the write path as one sample sees it. Nil without the ZFS module.
|
||||
type ZFSWrites struct {
|
||||
Dirty uint64 `json:"dirty"` // bytes in RAM written but not yet on disk, all pools
|
||||
DirtyMax uint64 `json:"dirty_max"` // zfs_dirty_data_max
|
||||
DelayMinPct int `json:"delay_min_pct"` // zfs_delay_min_dirty_percent: the throttle starts here
|
||||
Limit uint64 `json:"limit"` // what the buffer can really hold now: the smaller of dirty_max and half the ARC target
|
||||
DelayedPerSec float64 `json:"delayed_per_sec"` // writes the throttle slowed, per second over the interval
|
||||
OverMaxPerSec float64 `json:"over_max_per_sec"` // writes stopped dead at the limit
|
||||
TxgTimeoutSec int `json:"txg_timeout_sec"` // zfs_txg_timeout
|
||||
Pools []PoolWrites `json:"pools"`
|
||||
}
|
||||
|
||||
// PoolWrites is one pool.
|
||||
type PoolWrites struct {
|
||||
Pool string `json:"pool"`
|
||||
WriteBps float64 `json:"write_bps"` // to its vdevs, after compression, parity included
|
||||
LogicalBps float64 `json:"logical_bps"` // into its datasets, as programs wrote it
|
||||
Landing *Landing `json:"landing,omitempty"`
|
||||
}
|
||||
|
||||
// Landing is a landing tier in front of a pool.
|
||||
type Landing struct {
|
||||
Mount string `json:"mount"` // the mergerfs pool, e.g. /srv/tank
|
||||
Branch string `json:"branch"` // the landing branch, e.g. /srv/landing/tank
|
||||
PoolBranch string `json:"pool_branch"`
|
||||
Device string `json:"device,omitempty"` // block device under the branch, as in Sample.Disks
|
||||
FSType string `json:"fstype,omitempty"`
|
||||
Size uint64 `json:"size"`
|
||||
Used uint64 `json:"used"`
|
||||
PendingFiles int `json:"pending_files"`
|
||||
PendingBytes uint64 `json:"pending_bytes"`
|
||||
PendingPartial bool `json:"pending_partial,omitempty"` // the walk stopped at landingWalkMax
|
||||
OldestMs int64 `json:"oldest_ms,omitempty"` // mtime of the oldest waiting file
|
||||
CountedMs int64 `json:"counted_ms"` // when the walk ran
|
||||
InBps float64 `json:"in_bps"` // the landing device's writes
|
||||
DrainBps float64 `json:"drain_bps"` // the mover's progress onto the pool
|
||||
Mover *Mover `json:"mover,omitempty"`
|
||||
}
|
||||
|
||||
// Mover is what a landing mover says about itself in its status file.
|
||||
type Mover struct {
|
||||
State string `json:"state"` // idle | moving | holding (a mount it needs is missing)
|
||||
Src string `json:"src"`
|
||||
Dst string `json:"dst"`
|
||||
PID int `json:"pid"`
|
||||
StartedMs int64 `json:"started_ms"`
|
||||
UpdatedMs int64 `json:"updated_ms"`
|
||||
Stale bool `json:"stale"` // not updated for a while: stopped or hung
|
||||
Queue int `json:"queue"`
|
||||
Settle float64 `json:"settle"`
|
||||
BWLimit string `json:"bwlimit"`
|
||||
BatchFiles int `json:"batch_files"`
|
||||
BatchBytes uint64 `json:"batch_bytes"`
|
||||
BatchStartMs int64 `json:"batch_start_ms"`
|
||||
BatchFirst string `json:"batch_first"`
|
||||
Current string `json:"current"` // the file being copied, when it can tell
|
||||
CurrentSize uint64 `json:"current_size"` // its full size
|
||||
Partial uint64 `json:"partial"` // bytes of it already on the pool
|
||||
MovedFiles uint64 `json:"moved_files"` // since the mover started
|
||||
MovedBytes uint64 `json:"moved_bytes"`
|
||||
LastMove string `json:"last_move"`
|
||||
LastMoveMs int64 `json:"last_move_ms"`
|
||||
LastError string `json:"last_error"`
|
||||
LastErrorMs int64 `json:"last_error_ms"`
|
||||
}
|
||||
|
||||
type zfsWritePrev struct {
|
||||
at time.Time
|
||||
delay uint64
|
||||
overMax uint64
|
||||
txg map[string]uint64 // pool -> highest synced txg counted
|
||||
txgAt map[string]time.Time // pool -> when that txg was first seen
|
||||
txgRate map[string]float64 // pool -> bytes/s over the last gap between syncs
|
||||
logical map[string]uint64 // pool -> sum of objset nwritten
|
||||
drained map[string]uint64 // landing branch -> moved + partial bytes, at the mover's last update
|
||||
drainedAt map[string]int64 // landing branch -> that update's time, unix ms
|
||||
drainRate map[string]float64 // landing branch -> bytes/s between its last two updates
|
||||
landing map[string]Landing // branch -> last walk
|
||||
landingAt time.Time
|
||||
}
|
||||
|
||||
func readParamU(name string) uint64 {
|
||||
v, _ := strconv.ParseUint(strings.TrimSpace(readStr(zfsParamDir+name)), 10, 64)
|
||||
return v
|
||||
}
|
||||
|
||||
// poolsWithKstats lists the imported pools: the directories under the kstat tree.
|
||||
func poolsWithKstats() []string {
|
||||
ents, err := os.ReadDir(zfsKstatDir)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, e := range ents {
|
||||
if e.IsDir() {
|
||||
if _, err := os.Stat(zfsKstatDir + e.Name() + "/txgs"); err == nil {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// parseTxgs reads a pool's txgs kstat and returns, for every committed txg above `after`, the
|
||||
// bytes it wrote, plus the highest committed txg seen.
|
||||
func parseTxgs(text string, after uint64) (written, top uint64) {
|
||||
top = after
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
f := strings.Fields(line)
|
||||
// txg birth state ndirty nread nwritten reads writes otime qtime wtime stime
|
||||
if len(f) < 12 || f[2] != "C" {
|
||||
continue
|
||||
}
|
||||
txg, err := strconv.ParseUint(f[0], 10, 64)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if txg > top {
|
||||
top = txg
|
||||
}
|
||||
if txg > after {
|
||||
n, _ := strconv.ParseUint(f[5], 10, 64)
|
||||
written += n
|
||||
}
|
||||
}
|
||||
return written, top
|
||||
}
|
||||
|
||||
// objsetWritten sums nwritten over a pool's datasets.
|
||||
func objsetWritten(pool string) uint64 {
|
||||
m, _ := filepath.Glob(zfsKstatDir + pool + "/objset-*")
|
||||
var t uint64
|
||||
for _, p := range m {
|
||||
for _, line := range strings.Split(readStr(p), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) == 3 && f[0] == "nwritten" {
|
||||
v, _ := strconv.ParseUint(f[2], 10, 64)
|
||||
t += v
|
||||
}
|
||||
}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// mountEntry is one line of /proc/mounts.
|
||||
type mountEntry struct{ src, point, fstype string }
|
||||
|
||||
func readMounts() []mountEntry {
|
||||
var out []mountEntry
|
||||
for _, line := range strings.Split(readStr("/proc/mounts"), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) >= 3 {
|
||||
out = append(out, mountEntry{src: unescapeMount(f[0]), point: unescapeMount(f[1]), fstype: f[2]})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// unescapeMount undoes /proc/mounts' octal escapes (\040 for a space, and so on).
|
||||
func unescapeMount(s string) string {
|
||||
if !strings.Contains(s, `\`) {
|
||||
return s
|
||||
}
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(s); i++ {
|
||||
if s[i] == '\\' && i+3 < len(s) {
|
||||
if v, err := strconv.ParseUint(s[i+1:i+4], 8, 8); err == nil {
|
||||
b.WriteByte(byte(v))
|
||||
i += 3
|
||||
continue
|
||||
}
|
||||
}
|
||||
b.WriteByte(s[i])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// mountFor is the mount a path lives on: the longest mount point that is a prefix of it.
|
||||
func mountFor(ms []mountEntry, path string) (mountEntry, bool) {
|
||||
var best mountEntry
|
||||
found := false
|
||||
for _, m := range ms {
|
||||
if m.point == path || m.point == "/" || strings.HasPrefix(path, strings.TrimSuffix(m.point, "/")+"/") {
|
||||
if !found || len(m.point) > len(best.point) {
|
||||
best, found = m, true
|
||||
}
|
||||
}
|
||||
}
|
||||
return best, found
|
||||
}
|
||||
|
||||
// parseBranches reads mergerfs's branch list: "/srv/landing/tank=RW:/tank=RW,50G".
|
||||
func parseBranches(v string) (paths []string, modes []string) {
|
||||
for _, b := range strings.Split(strings.TrimSpace(v), ":") {
|
||||
if b == "" {
|
||||
continue
|
||||
}
|
||||
p, mode, _ := strings.Cut(b, "=")
|
||||
mode, _, _ = strings.Cut(mode, ",")
|
||||
if mode == "" {
|
||||
mode = "RW"
|
||||
}
|
||||
paths, modes = append(paths, p), append(modes, mode)
|
||||
}
|
||||
return paths, modes
|
||||
}
|
||||
|
||||
func getxattr(path, name string) string {
|
||||
buf := make([]byte, 4096)
|
||||
n, err := syscall.Getxattr(path, name, buf)
|
||||
if err != nil || n <= 0 {
|
||||
return ""
|
||||
}
|
||||
return string(buf[:n])
|
||||
}
|
||||
|
||||
// findLandings maps pool -> the landing tiers in front of it.
|
||||
func findLandings(ms []mountEntry) map[string][]Landing {
|
||||
out := map[string][]Landing{}
|
||||
for _, m := range ms {
|
||||
if m.fstype != "fuse.mergerfs" {
|
||||
continue
|
||||
}
|
||||
paths, modes := parseBranches(getxattr(filepath.Join(m.point, ".mergerfs"), "user.mergerfs.branches"))
|
||||
// the pool branch is the first ZFS one; landing branches are writable non-ZFS ones before it
|
||||
for i, p := range paths {
|
||||
bm, ok := mountFor(ms, p)
|
||||
if !ok || bm.fstype != "zfs" {
|
||||
continue
|
||||
}
|
||||
pool, _, _ := strings.Cut(bm.src, "/")
|
||||
for j := 0; j < i; j++ {
|
||||
lm, ok := mountFor(ms, paths[j])
|
||||
if !ok || lm.fstype == "zfs" || modes[j] == "RO" {
|
||||
continue
|
||||
}
|
||||
out[pool] = append(out[pool], Landing{Mount: m.point, Branch: paths[j], PoolBranch: p, FSType: lm.fstype, Device: blockDisk(lm.src)})
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// blockDisk turns a mount source into the name Sample.Disks uses: /dev/nvme0n1p1 -> nvme0,
|
||||
// /dev/sdb2 -> sdb. Empty for anything that is not a plain block device.
|
||||
func blockDisk(src string) string {
|
||||
dev, err := filepath.EvalSymlinks(src)
|
||||
if err != nil || !strings.HasPrefix(dev, "/dev/") {
|
||||
return ""
|
||||
}
|
||||
name := filepath.Base(dev)
|
||||
// a partition's sysfs entry sits inside its disk's
|
||||
if p, err := filepath.EvalSymlinks("/sys/class/block/" + name); err == nil {
|
||||
if _, err := os.Stat(p + "/partition"); err == nil {
|
||||
name = filepath.Base(filepath.Dir(p))
|
||||
}
|
||||
}
|
||||
if m := nsRe.FindStringSubmatch(name); m != nil {
|
||||
return m[1] // nvme0n1 -> nvme0, as foldNVMe names it
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
// walkLanding counts what is waiting on a landing branch.
|
||||
func walkLanding(l *Landing, now time.Time) {
|
||||
var st syscall.Statfs_t
|
||||
if syscall.Statfs(l.Branch, &st) == nil {
|
||||
l.Size = st.Blocks * uint64(st.Bsize)
|
||||
l.Used = (st.Blocks - st.Bfree) * uint64(st.Bsize)
|
||||
}
|
||||
seen := 0
|
||||
l.PendingFiles, l.PendingBytes, l.OldestMs, l.PendingPartial = 0, 0, 0, false
|
||||
_ = filepath.WalkDir(l.Branch, func(p string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
if seen++; seen > landingWalkMax {
|
||||
l.PendingPartial = true
|
||||
return filepath.SkipAll
|
||||
}
|
||||
if !d.Type().IsRegular() {
|
||||
return nil
|
||||
}
|
||||
if fi, err := d.Info(); err == nil {
|
||||
l.PendingFiles++
|
||||
l.PendingBytes += uint64(fi.Size())
|
||||
if ms := fi.ModTime().UnixMilli(); l.OldestMs == 0 || ms < l.OldestMs {
|
||||
l.OldestMs = ms
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
l.CountedMs = now.UnixMilli()
|
||||
}
|
||||
|
||||
// readMovers loads every mover status file, keyed by the branch it drains.
|
||||
func readMovers(now time.Time) map[string]*Mover {
|
||||
out := map[string]*Mover{}
|
||||
m, _ := filepath.Glob(moverStateDir + "/*.json")
|
||||
for _, p := range m {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var mv Mover
|
||||
if json.Unmarshal(b, &mv) != nil || mv.Src == "" {
|
||||
continue
|
||||
}
|
||||
mv.Stale = now.UnixMilli()-mv.UpdatedMs > 30_000
|
||||
if mv.Stale {
|
||||
mv.State = "stopped"
|
||||
}
|
||||
out[filepath.Clean(mv.Src)] = &mv
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// zfsWrites samples the write path; nil when this machine has no ZFS module loaded.
|
||||
func (c *Collector) zfsWrites(now time.Time, arc *ARC, disks []Disk) *ZFSWrites {
|
||||
if arc == nil {
|
||||
return nil
|
||||
}
|
||||
p := &c.prevZW
|
||||
dt := 0.0
|
||||
if !p.at.IsZero() {
|
||||
dt = now.Sub(p.at).Seconds()
|
||||
}
|
||||
w := &ZFSWrites{Dirty: arc.Anon, DirtyMax: readParamU("zfs_dirty_data_max"),
|
||||
DelayMinPct: int(readParamU("zfs_delay_min_dirty_percent")), TxgTimeoutSec: int(readParamU("zfs_txg_timeout")), Pools: []PoolWrites{}}
|
||||
w.Limit = w.DirtyMax
|
||||
if half := arc.Target / 2; half > 0 && half < w.Limit {
|
||||
w.Limit = half
|
||||
}
|
||||
tx := readKstat(zfsKstatDir + "dmu_tx")
|
||||
if dt > 0 && tx["dmu_tx_dirty_delay"] >= p.delay && tx["dmu_tx_dirty_over_max"] >= p.overMax {
|
||||
w.DelayedPerSec = round1(float64(tx["dmu_tx_dirty_delay"]-p.delay) / dt)
|
||||
w.OverMaxPerSec = round1(float64(tx["dmu_tx_dirty_over_max"]-p.overMax) / dt)
|
||||
}
|
||||
p.delay, p.overMax = tx["dmu_tx_dirty_delay"], tx["dmu_tx_dirty_over_max"]
|
||||
if p.txg == nil {
|
||||
p.txg, p.logical, p.drained, p.landing = map[string]uint64{}, map[string]uint64{}, map[string]uint64{}, map[string]Landing{}
|
||||
p.txgAt, p.txgRate, p.drainedAt, p.drainRate = map[string]time.Time{}, map[string]float64{}, map[string]int64{}, map[string]float64{}
|
||||
}
|
||||
|
||||
ms := readMounts()
|
||||
landings := findLandings(ms)
|
||||
walk := now.Sub(p.landingAt) >= landingEvery
|
||||
if walk {
|
||||
p.landingAt = now
|
||||
}
|
||||
movers := readMovers(now)
|
||||
diskW := map[string]float64{}
|
||||
for _, d := range disks {
|
||||
diskW[d.Name] = d.WriteBps
|
||||
}
|
||||
|
||||
for _, pool := range poolsWithKstats() {
|
||||
pw := PoolWrites{Pool: pool}
|
||||
// A txg records what it wrote only when it has synced — every few seconds, not every
|
||||
// sample — so the rate is taken over the gap between syncs and held until the next one.
|
||||
prevTop, had := p.txg[pool]
|
||||
written, top := parseTxgs(readStr(zfsKstatDir+pool+"/txgs"), prevTop)
|
||||
switch {
|
||||
case !had:
|
||||
p.txgAt[pool] = now
|
||||
case top > prevTop:
|
||||
if gap := now.Sub(p.txgAt[pool]).Seconds(); gap > 0 {
|
||||
p.txgRate[pool] = round1(float64(written) / gap)
|
||||
}
|
||||
p.txgAt[pool] = now
|
||||
case now.Sub(p.txgAt[pool]) > time.Duration(2*max(w.TxgTimeoutSec, 5))*time.Second:
|
||||
p.txgRate[pool] = 0 // no sync for two timeouts: nothing is being written
|
||||
}
|
||||
pw.WriteBps = p.txgRate[pool]
|
||||
p.txg[pool] = top
|
||||
lw := objsetWritten(pool)
|
||||
if prev, ok := p.logical[pool]; ok && dt > 0 && lw >= prev {
|
||||
pw.LogicalBps = round1(float64(lw-prev) / dt)
|
||||
}
|
||||
p.logical[pool] = lw
|
||||
|
||||
if ls := landings[pool]; len(ls) > 0 {
|
||||
l := ls[0] // one landing tier per pool is what the view shows
|
||||
if last, ok := p.landing[l.Branch]; ok && !walk {
|
||||
l.Size, l.Used, l.PendingFiles, l.PendingBytes = last.Size, last.Used, last.PendingFiles, last.PendingBytes
|
||||
l.OldestMs, l.PendingPartial, l.CountedMs = last.OldestMs, last.PendingPartial, last.CountedMs
|
||||
} else {
|
||||
walkLanding(&l, now)
|
||||
p.landing[l.Branch] = l
|
||||
}
|
||||
l.InBps = diskW[l.Device]
|
||||
if mv := movers[filepath.Clean(l.Branch)]; mv != nil {
|
||||
l.Mover = mv
|
||||
// the mover writes its status every couple of seconds: measure between its updates
|
||||
done := mv.MovedBytes + mv.Partial
|
||||
if at, ok := p.drainedAt[l.Branch]; !ok || mv.UpdatedMs != at {
|
||||
if ok && mv.UpdatedMs > at && done >= p.drained[l.Branch] {
|
||||
p.drainRate[l.Branch] = round1(float64(done-p.drained[l.Branch]) / (float64(mv.UpdatedMs-at) / 1000))
|
||||
}
|
||||
p.drained[l.Branch], p.drainedAt[l.Branch] = done, mv.UpdatedMs
|
||||
}
|
||||
if mv.State != "moving" {
|
||||
p.drainRate[l.Branch] = 0
|
||||
}
|
||||
l.DrainBps = p.drainRate[l.Branch]
|
||||
}
|
||||
pw.Landing = &l
|
||||
}
|
||||
w.Pools = append(w.Pools, pw)
|
||||
}
|
||||
p.at = now
|
||||
return w
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
package metrics
|
||||
|
||||
import "testing"
|
||||
|
||||
// real rows from grid's /proc/spl/kstat/zfs/tank/txgs, 2026-10-25
|
||||
const txgsSample = `81 1 0x01 26 7072 20029594359 13953814329135
|
||||
txg birth state ndirty nread nwritten reads writes otime qtime wtime stime
|
||||
15812833 13936408791659 C 0 0 0 0 0 5119980242 2040 58385 309580
|
||||
15812834 13941528771901 C 0 0 0 0 0 5119948815 8471 65826 112612
|
||||
15812835 13946648720716 C 885760 0 2293760 0 367 5120100556 1704 56762 271264374
|
||||
15812836 13951768821272 S 4194304 0 0 0 0 5120000000 1000 50000 0
|
||||
15812837 13956888821272 O 0 0 0 0 0 0 0 0 0
|
||||
`
|
||||
|
||||
func TestParseTxgs(t *testing.T) {
|
||||
w, top := parseTxgs(txgsSample, 0)
|
||||
if w != 2293760 || top != 15812835 {
|
||||
t.Fatalf("from 0: written %d top %d", w, top)
|
||||
}
|
||||
// only txgs synced since the last look count, and an open or syncing txg never does
|
||||
w, top = parseTxgs(txgsSample, 15812834)
|
||||
if w != 2293760 || top != 15812835 {
|
||||
t.Fatalf("after 15812834: written %d top %d", w, top)
|
||||
}
|
||||
w, top = parseTxgs(txgsSample, 15812835)
|
||||
if w != 0 || top != 15812835 {
|
||||
t.Fatalf("nothing new: written %d top %d", w, top)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBranches(t *testing.T) {
|
||||
p, m := parseBranches("/srv/landing/tank=RW:/tank=NC,50G:/mnt/old=RO\n")
|
||||
if len(p) != 3 || p[0] != "/srv/landing/tank" || p[1] != "/tank" || p[2] != "/mnt/old" {
|
||||
t.Fatalf("paths %q", p)
|
||||
}
|
||||
if m[0] != "RW" || m[1] != "NC" || m[2] != "RO" {
|
||||
t.Fatalf("modes %q", m)
|
||||
}
|
||||
if p, _ := parseBranches("/a:/b"); len(p) != 2 {
|
||||
t.Fatalf("bare paths %q", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMountFor(t *testing.T) {
|
||||
ms := []mountEntry{
|
||||
{"/dev/sdj2", "/", "ext4"},
|
||||
{"/dev/nvme0n1p1", "/srv/landing", "ext4"},
|
||||
{"tank", "/tank", "zfs"},
|
||||
{"tank/media", "/tank/media", "zfs"},
|
||||
{"tank-landing", "/srv/tank", "fuse.mergerfs"},
|
||||
}
|
||||
cases := map[string]string{
|
||||
"/srv/landing/tank": "/srv/landing", "/tank": "/tank", "/tank/media/x": "/tank/media",
|
||||
"/tankfoo": "/", "/srv/landingx": "/",
|
||||
}
|
||||
for path, want := range cases {
|
||||
if m, ok := mountFor(ms, path); !ok || m.point != want {
|
||||
t.Errorf("%s: got %q want %q", path, m.point, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnescapeMount(t *testing.T) {
|
||||
if got := unescapeMount(`/mnt/my\040disk`); got != "/mnt/my disk" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFoldZFSWrites(t *testing.T) {
|
||||
mk := func(dirty uint64, bps, in, drain float64) Sample {
|
||||
return Sample{ZFSWrite: &ZFSWrites{Dirty: dirty, Limit: 100, Pools: []PoolWrites{{Pool: "tank", WriteBps: bps,
|
||||
Landing: &Landing{Branch: "/srv/landing/tank", InBps: in, DrainBps: drain, Mover: &Mover{State: "moving"}}}}}}
|
||||
}
|
||||
b := []Sample{mk(10, 100, 400, 0), mk(30, 300, 0, 60)}
|
||||
got := fold(b).ZFSWrite
|
||||
if got.Dirty != 20 || got.Pools[0].WriteBps != 200 || got.Pools[0].Landing.InBps != 200 || got.Pools[0].Landing.DrainBps != 30 {
|
||||
t.Fatalf("folded %+v / %+v", got, got.Pools[0].Landing)
|
||||
}
|
||||
// the fold must not write through to the newest sample it copied from
|
||||
if b[1].ZFSWrite.Pools[0].Landing.InBps != 0 || b[1].ZFSWrite.Dirty != 30 {
|
||||
t.Fatalf("fold changed its input")
|
||||
}
|
||||
if got.Pools[0].Landing.Mover.State != "moving" {
|
||||
t.Fatalf("mover state lost")
|
||||
}
|
||||
}
|
||||
+10
-4
@@ -383,13 +383,19 @@ func (m *Manager) PushAll(ctx context.Context, binPath, ourArch string, force bo
|
||||
// WebSocket origin check (meant for browsers) doesn't reject the hub.
|
||||
// Do makes one request to a managed machine as this hub and returns the whole response.
|
||||
func (m *Manager) Do(ctx context.Context, name, method, path string, body []byte, hdr http.Header) (int, []byte, error) {
|
||||
code, b, _, err := m.DoHeaders(ctx, name, method, path, body, hdr)
|
||||
return code, b, err
|
||||
}
|
||||
|
||||
// DoHeaders is Do that also answers the response's headers.
|
||||
func (m *Manager) DoHeaders(ctx context.Context, name, method, path string, body []byte, hdr http.Header) (int, []byte, http.Header, error) {
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
return 0, nil, errors.New("no such machine")
|
||||
return 0, nil, nil, errors.New("no such machine")
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(n.URL, "/")+path, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return 0, nil, err
|
||||
return 0, nil, nil, err
|
||||
}
|
||||
for k, v := range hdr {
|
||||
req.Header[k] = v
|
||||
@@ -397,11 +403,11 @@ func (m *Manager) Do(ctx context.Context, name, method, path string, body []byte
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
resp, err := m.client(n).Do(req)
|
||||
if err != nil {
|
||||
return 0, nil, errors.New(shortErr(err))
|
||||
return 0, nil, nil, errors.New(shortErr(err))
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
||||
return resp.StatusCode, b, err
|
||||
return resp.StatusCode, b, resp.Header, err
|
||||
}
|
||||
|
||||
// URL is a managed machine's address, "" for one not managed here.
|
||||
|
||||
@@ -0,0 +1,564 @@
|
||||
package storage
|
||||
|
||||
// Verifying a landing tier: did everything that landed really reach the pool, intact?
|
||||
//
|
||||
// A landing tier is fast storage in front of a pool (a mergerfs pool, new files on the fast
|
||||
// branch first) with a mover draining it. The mover keeps a ledger — one JSON line per file it
|
||||
// moved: the path under the pool, its size and mtime, and the SHA-256 of the fast copy read
|
||||
// just before the move (hamlet's scripts/landing-mover.py writes /var/lib/landing-mover/
|
||||
// ledger.jsonl and names it in its status file). A verify pass reads the pool's copy of each
|
||||
// entry back and compares.
|
||||
//
|
||||
// What a pass can say about each file:
|
||||
// - match: same size, same mtime, same SHA-256 — it arrived intact
|
||||
// - mismatch: same size and mtime but different contents — the one real failure
|
||||
// - changed: the pool's copy has a different size or mtime: edited since it moved, so the
|
||||
// old hash says nothing about it (counted, not compared)
|
||||
// - gone: not on the pool and not waiting on the landing branch either — deleted or
|
||||
// renamed since it moved. A pass cannot tell that from a loss, so it lists them
|
||||
// - waiting: back on the landing branch (written again since), the mover will take it
|
||||
// - unreadable: could not be opened as this account
|
||||
// - unhashed: the mover recorded no hash for it (not a regular file, or changed mid-move)
|
||||
//
|
||||
// Scope "new" checks what has moved since the last pass, "all" re-reads the whole ledger. A
|
||||
// path moved more than once is checked against its latest entry only.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ---- schedules ----
|
||||
|
||||
// VerifyPolicy runs a verify pass on a schedule. The when-fields mean what they do on a
|
||||
// snapshot Policy; Scope is "new" or "all".
|
||||
type VerifyPolicy struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Every string `json:"every"`
|
||||
At string `json:"at"`
|
||||
Weekday int `json:"weekday"`
|
||||
Day int `json:"day"`
|
||||
Cron string `json:"cron,omitempty"`
|
||||
Scope string `json:"scope"`
|
||||
Enabled bool `json:"enabled"`
|
||||
}
|
||||
|
||||
// when borrows the snapshot Policy's schedule logic, so both kinds of schedule are written,
|
||||
// validated and read back by exactly the same code.
|
||||
func (v *VerifyPolicy) when() *Policy {
|
||||
return &Policy{Every: v.Every, At: v.At, Weekday: v.Weekday, Day: v.Day, Cron: v.Cron}
|
||||
}
|
||||
|
||||
func (v *VerifyPolicy) CronExpr() (string, error) { return v.when().CronExpr() }
|
||||
func (v *VerifyPolicy) NextRun(now time.Time) time.Time { return v.when().NextRun(now) }
|
||||
|
||||
func (v *VerifyPolicy) Validate() error {
|
||||
if v.Scope != "new" && v.Scope != "all" {
|
||||
return errors.New("scope: new (what moved since the last pass) or all (the whole ledger)")
|
||||
}
|
||||
switch v.Every {
|
||||
case "hourly", "daily", "weekly", "monthly":
|
||||
case "custom":
|
||||
if strings.TrimSpace(v.Cron) == "" {
|
||||
return errors.New("a custom schedule needs a cron expression")
|
||||
}
|
||||
default:
|
||||
return errors.New("every: hourly, daily, weekly, monthly or custom")
|
||||
}
|
||||
if _, err := v.CronExpr(); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, _, err := v.when().clock(); err != nil {
|
||||
return err
|
||||
}
|
||||
if v.Every == "weekly" && (v.Weekday < 0 || v.Weekday > 6) {
|
||||
return errors.New("weekday: 0 (Sunday) to 6")
|
||||
}
|
||||
if v.Every == "monthly" && (v.Day < 1 || v.Day > 28) {
|
||||
return errors.New("day: 1 to 28, so every month has one")
|
||||
}
|
||||
if len(v.Name) > 64 {
|
||||
return errors.New("name: 64 characters at most")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// VerifyCronFile holds the verify schedules; Ham Lab owns it and rewrites it whole.
|
||||
const VerifyCronFile = "/etc/cron.d/hamlab-landing"
|
||||
|
||||
// VerifyCronText renders the verify schedules as cron lines, run as Ham Lab's own account.
|
||||
func VerifyCronText(ps []VerifyPolicy, user, exe, config string) (string, error) {
|
||||
var b strings.Builder
|
||||
b.WriteString("# Ham Lab landing-tier verify schedules — written by the ZFS tab, rewritten whole on every change.\n")
|
||||
b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
|
||||
b.WriteString("SHELL=/bin/sh\n")
|
||||
b.WriteString("PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
|
||||
for i := range ps {
|
||||
p := &ps[i]
|
||||
name := strings.ReplaceAll(p.Name, "\n", " ")
|
||||
if !p.Enabled {
|
||||
b.WriteString("# " + name + " — paused\n")
|
||||
continue
|
||||
}
|
||||
expr, err := p.CronExpr()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%s: %w", p.Name, err)
|
||||
}
|
||||
fmt.Fprintf(&b, "# %s — verify %s\n", name, p.Scope)
|
||||
fmt.Fprintf(&b, "%s %s %s -config %s landing-verify %s\n\n", expr, user, exe, config, p.ID)
|
||||
}
|
||||
return b.String(), nil
|
||||
}
|
||||
|
||||
// WriteVerifyCron puts the verify schedules on the machine, or removes the file when none run.
|
||||
func (m *Manager) WriteVerifyCron(ctx context.Context, ps []VerifyPolicy, user, exe, config string) error {
|
||||
live := false
|
||||
for i := range ps {
|
||||
live = live || ps[i].Enabled
|
||||
}
|
||||
if !live {
|
||||
return m.RemoveRootFile(ctx, VerifyCronFile)
|
||||
}
|
||||
text, err := VerifyCronText(ps, user, exe, config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return m.writeRoot(ctx, VerifyCronFile, text)
|
||||
}
|
||||
|
||||
// ---- finding the landing tier ----
|
||||
|
||||
// MoverStatusDir is where a landing mover writes what it is doing.
|
||||
const MoverStatusDir = "/run/landing-mover"
|
||||
|
||||
// DefaultLedger is where hamlet's mover keeps its ledger when no status file names one.
|
||||
const DefaultLedger = "/var/lib/landing-mover/ledger.jsonl"
|
||||
|
||||
// LandingTarget is one landing tier a pass can check: the branch files land on, the pool
|
||||
// directory they are moved to, and the mover's ledger.
|
||||
type LandingTarget struct {
|
||||
Src string `json:"src"`
|
||||
Dst string `json:"dst"`
|
||||
Ledger string `json:"ledger"`
|
||||
}
|
||||
|
||||
// LandingTargets reads the movers' status files. Each mover names its own source, target and
|
||||
// ledger; with none running, nothing can be verified, and the page says so.
|
||||
func LandingTargets() []LandingTarget {
|
||||
var out []LandingTarget
|
||||
m, _ := filepath.Glob(MoverStatusDir + "/*.json")
|
||||
for _, p := range m {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var st struct{ Src, Dst, Ledger string }
|
||||
if json.Unmarshal(b, &st) != nil || st.Src == "" || st.Dst == "" {
|
||||
continue
|
||||
}
|
||||
if st.Ledger == "" {
|
||||
st.Ledger = DefaultLedger
|
||||
}
|
||||
out = append(out, LandingTarget{Src: filepath.Clean(st.Src), Dst: filepath.Clean(st.Dst), Ledger: st.Ledger})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Src < out[j].Src })
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- stale files ----
|
||||
|
||||
// StaleAfter is how long a file may wait on a landing branch before it counts as stale. A file
|
||||
// older than this is not necessarily lost — a big backlog drains for hours — but it is worth a
|
||||
// look, which is what the ZFS tab offers.
|
||||
const StaleAfter = time.Hour
|
||||
|
||||
// StaleFile is one file that has waited longer than StaleAfter.
|
||||
type StaleFile struct {
|
||||
Path string `json:"path"` // relative to the landing branch, which is also its path on the pool
|
||||
Size int64 `json:"size"`
|
||||
MTime int64 `json:"mtime"` // unix ms
|
||||
}
|
||||
|
||||
// StaleFiles lists what has waited on a landing branch longer than StaleAfter, oldest first,
|
||||
// at most limit of them, with the full count and size.
|
||||
func StaleFiles(branch string, limit int, now time.Time) (files []StaleFile, count int, bytes int64) {
|
||||
cut := now.Add(-StaleAfter)
|
||||
_ = filepath.WalkDir(branch, func(p string, d os.DirEntry, err error) error {
|
||||
if err != nil || !d.Type().IsRegular() {
|
||||
return nil
|
||||
}
|
||||
fi, err := d.Info()
|
||||
if err != nil || !fi.ModTime().Before(cut) {
|
||||
return nil
|
||||
}
|
||||
count++
|
||||
bytes += fi.Size()
|
||||
rel, _ := filepath.Rel(branch, p)
|
||||
files = append(files, StaleFile{Path: rel, Size: fi.Size(), MTime: fi.ModTime().UnixMilli()})
|
||||
return nil
|
||||
})
|
||||
sort.Slice(files, func(i, j int) bool { return files[i].MTime < files[j].MTime })
|
||||
if len(files) > limit {
|
||||
files = files[:limit]
|
||||
}
|
||||
return files, count, bytes
|
||||
}
|
||||
|
||||
// ---- the pass ----
|
||||
|
||||
// LedgerEntry is one line of the mover's ledger.
|
||||
type LedgerEntry struct {
|
||||
T int64 `json:"t"` // when it was moved, unix ms
|
||||
Path string `json:"path"`
|
||||
Size int64 `json:"size"`
|
||||
MTimeNs int64 `json:"mtime_ns"`
|
||||
SHA256 string `json:"sha256,omitempty"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
// VerifyIssue is one file a pass wants a person to look at.
|
||||
type VerifyIssue struct {
|
||||
Path string `json:"path"`
|
||||
Kind string `json:"kind"` // mismatch | gone | unreadable
|
||||
Moved int64 `json:"moved"`
|
||||
Size int64 `json:"size"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
// VerifyResult is what one pass found.
|
||||
type VerifyResult struct {
|
||||
ID string `json:"id"` // the schedule that ran it, or "manual"
|
||||
Scope string `json:"scope"`
|
||||
Started time.Time `json:"started"`
|
||||
Finished time.Time `json:"finished"`
|
||||
Src string `json:"src"`
|
||||
Dst string `json:"dst"`
|
||||
Since int64 `json:"since"` // ledger entries after this (unix ms); 0 for all
|
||||
Entries int `json:"entries"`
|
||||
Checked int `json:"checked"`
|
||||
Matched int `json:"matched"`
|
||||
Mismatched int `json:"mismatched"`
|
||||
Changed int `json:"changed"`
|
||||
Gone int `json:"gone"`
|
||||
Waiting int `json:"waiting"`
|
||||
Unreadable int `json:"unreadable"`
|
||||
Unhashed int `json:"unhashed"`
|
||||
BytesRead int64 `json:"bytes_read"`
|
||||
Issues []VerifyIssue `json:"issues"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Cursor int64 `json:"cursor"` // the newest entry it covered
|
||||
}
|
||||
|
||||
// OK is true when nothing needs a look: no mismatches, nothing gone or unreadable, no error.
|
||||
func (r *VerifyResult) OK() bool {
|
||||
return r.Error == "" && r.Mismatched == 0 && r.Gone == 0 && r.Unreadable == 0
|
||||
}
|
||||
|
||||
// VerifyProgress is published while a pass runs.
|
||||
type VerifyProgress struct {
|
||||
ID string `json:"id"`
|
||||
Scope string `json:"scope"`
|
||||
PID int `json:"pid"`
|
||||
Started time.Time `json:"started"`
|
||||
Updated time.Time `json:"updated"`
|
||||
Total int `json:"total"`
|
||||
Done int `json:"done"`
|
||||
BytesRead int64 `json:"bytes_read"`
|
||||
BytesAll int64 `json:"bytes_all"`
|
||||
Current string `json:"current"`
|
||||
}
|
||||
|
||||
const maxIssues = 500
|
||||
|
||||
// readLedger loads the ledger's entries newer than since, the latest per path, oldest first.
|
||||
func readLedger(path string, since int64) ([]LedgerEntry, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
latest := map[string]LedgerEntry{}
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 64<<10), 4<<20)
|
||||
for sc.Scan() {
|
||||
var e LedgerEntry
|
||||
if json.Unmarshal(sc.Bytes(), &e) != nil || e.Path == "" || e.T <= since {
|
||||
continue
|
||||
}
|
||||
if old, ok := latest[e.Path]; !ok || e.T >= old.T {
|
||||
latest[e.Path] = e
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]LedgerEntry, 0, len(latest))
|
||||
for _, e := range latest {
|
||||
out = append(out, e)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].T < out[j].T })
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// inside joins rel under root and refuses anything that would climb out of it.
|
||||
func inside(root, rel string) (string, bool) {
|
||||
p := filepath.Join(root, rel)
|
||||
return p, p == root || strings.HasPrefix(p, root+string(filepath.Separator))
|
||||
}
|
||||
|
||||
func hashFile(p string, buf []byte, onRead func(int)) (string, error) {
|
||||
f, err := os.Open(p)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
for {
|
||||
n, err := f.Read(buf)
|
||||
if n > 0 {
|
||||
h.Write(buf[:n])
|
||||
onRead(n)
|
||||
}
|
||||
if err == io.EOF {
|
||||
return hex.EncodeToString(h.Sum(nil)), nil
|
||||
}
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Verify runs one pass over a landing tier's ledger. progress is called about once a second.
|
||||
func Verify(t LandingTarget, id, scope string, since int64, progress func(VerifyProgress)) VerifyResult {
|
||||
r := VerifyResult{ID: id, Scope: scope, Started: time.Now(), Src: t.Src, Dst: t.Dst, Since: since, Issues: []VerifyIssue{}, Cursor: since}
|
||||
if scope == "all" {
|
||||
r.Since = 0
|
||||
}
|
||||
entries, err := readLedger(t.Ledger, r.Since)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
r.Error = "no ledger yet at " + t.Ledger + " — nothing has been moved since the mover started keeping one"
|
||||
} else {
|
||||
r.Error = "reading the ledger: " + err.Error()
|
||||
}
|
||||
r.Finished = time.Now()
|
||||
return r
|
||||
}
|
||||
r.Entries = len(entries)
|
||||
pr := VerifyProgress{ID: id, Scope: scope, PID: os.Getpid(), Started: r.Started, Total: len(entries)}
|
||||
for _, e := range entries {
|
||||
pr.BytesAll += e.Size
|
||||
}
|
||||
last := time.Time{}
|
||||
tick := func(force bool) {
|
||||
if progress != nil && (force || time.Since(last) >= time.Second) {
|
||||
pr.Updated, last = time.Now(), time.Now()
|
||||
progress(pr)
|
||||
}
|
||||
}
|
||||
issue := func(e LedgerEntry, kind, note string) {
|
||||
if len(r.Issues) < maxIssues {
|
||||
r.Issues = append(r.Issues, VerifyIssue{Path: e.Path, Kind: kind, Moved: e.T, Size: e.Size, Note: note})
|
||||
}
|
||||
}
|
||||
buf := make([]byte, 4<<20)
|
||||
for _, e := range entries {
|
||||
pr.Current = e.Path
|
||||
tick(false)
|
||||
if e.T > r.Cursor {
|
||||
r.Cursor = e.T
|
||||
}
|
||||
r.Checked++
|
||||
pr.Done++
|
||||
dst, ok1 := inside(t.Dst, e.Path)
|
||||
src, ok2 := inside(t.Src, e.Path)
|
||||
if !ok1 || !ok2 {
|
||||
r.Unreadable++
|
||||
issue(e, "unreadable", "path climbs out of the pool")
|
||||
continue
|
||||
}
|
||||
st, err := os.Lstat(dst)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
if _, err := os.Lstat(src); err == nil {
|
||||
r.Waiting++
|
||||
continue
|
||||
}
|
||||
r.Gone++
|
||||
issue(e, "gone", "not on the pool, not waiting to move: deleted or renamed since it moved?")
|
||||
continue
|
||||
}
|
||||
r.Unreadable++
|
||||
issue(e, "unreadable", err.Error())
|
||||
continue
|
||||
}
|
||||
if e.SHA256 == "" {
|
||||
r.Unhashed++
|
||||
continue
|
||||
}
|
||||
mt := st.ModTime().UnixNano()
|
||||
if st.Size() != e.Size || abs64(mt-e.MTimeNs) >= int64(time.Second) {
|
||||
r.Changed++
|
||||
continue
|
||||
}
|
||||
sum, err := hashFile(dst, buf, func(n int) { r.BytesRead += int64(n); pr.BytesRead += int64(n); tick(false) })
|
||||
if err != nil {
|
||||
if errors.Is(err, syscall.EACCES) || errors.Is(err, os.ErrPermission) {
|
||||
r.Unreadable++
|
||||
issue(e, "unreadable", "permission denied")
|
||||
} else {
|
||||
r.Unreadable++
|
||||
issue(e, "unreadable", err.Error())
|
||||
}
|
||||
continue
|
||||
}
|
||||
if sum != e.SHA256 {
|
||||
r.Mismatched++
|
||||
issue(e, "mismatch", "contents differ from the copy that landed")
|
||||
continue
|
||||
}
|
||||
r.Matched++
|
||||
}
|
||||
tick(true)
|
||||
r.Finished = time.Now()
|
||||
return r
|
||||
}
|
||||
|
||||
func abs64(v int64) int64 {
|
||||
if v < 0 {
|
||||
return -v
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// ---- records ----
|
||||
|
||||
func verifyDir(dataDir string) string { return filepath.Join(dataDir, "landing-verify") }
|
||||
|
||||
func writeJSON(p string, v any) error {
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := p + ".tmp"
|
||||
if err := os.WriteFile(tmp, b, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, p)
|
||||
}
|
||||
|
||||
func readJSON(p string, v any) bool {
|
||||
b, err := os.ReadFile(p)
|
||||
return err == nil && json.Unmarshal(b, v) == nil
|
||||
}
|
||||
|
||||
// cursorKey names a landing tier's cursor file.
|
||||
func cursorKey(t LandingTarget) string {
|
||||
return strings.Trim(strings.NewReplacer("/", "_").Replace(t.Src), "_")
|
||||
}
|
||||
|
||||
// VerifyCursor is the newest ledger entry a finished pass covered, for scope "new".
|
||||
func VerifyCursor(dataDir string, t LandingTarget) int64 {
|
||||
var c struct{ Cursor int64 }
|
||||
readJSON(filepath.Join(verifyDir(dataDir), "cursor-"+cursorKey(t)+".json"), &c)
|
||||
return c.Cursor
|
||||
}
|
||||
|
||||
// RecordVerify keeps a finished pass: as the run's last result, in the history (newest first,
|
||||
// 30 kept), and — when it read the ledger at all — as the new cursor.
|
||||
func RecordVerify(dataDir string, t LandingTarget, r VerifyResult) error {
|
||||
d := verifyDir(dataDir)
|
||||
if err := writeJSON(filepath.Join(d, "run-"+r.ID+".json"), r); err != nil {
|
||||
return err
|
||||
}
|
||||
var hist []VerifyResult
|
||||
readJSON(filepath.Join(d, "history.json"), &hist)
|
||||
hist = append([]VerifyResult{r}, hist...)
|
||||
if len(hist) > 30 {
|
||||
hist = hist[:30]
|
||||
}
|
||||
if err := writeJSON(filepath.Join(d, "history.json"), hist); err != nil {
|
||||
return err
|
||||
}
|
||||
if r.Error == "" && r.Cursor > VerifyCursor(dataDir, t) {
|
||||
return writeJSON(filepath.Join(d, "cursor-"+cursorKey(t)+".json"), map[string]int64{"cursor": r.Cursor})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// VerifyHistory is the kept passes, newest first.
|
||||
func VerifyHistory(dataDir string) []VerifyResult {
|
||||
var hist []VerifyResult
|
||||
readJSON(filepath.Join(verifyDir(dataDir), "history.json"), &hist)
|
||||
if hist == nil {
|
||||
hist = []VerifyResult{}
|
||||
}
|
||||
return hist
|
||||
}
|
||||
|
||||
// LastVerify is the last pass a schedule (or "manual") ran.
|
||||
func LastVerify(dataDir, id string) *VerifyResult {
|
||||
var r VerifyResult
|
||||
if !readJSON(filepath.Join(verifyDir(dataDir), "run-"+id+".json"), &r) {
|
||||
return nil
|
||||
}
|
||||
return &r
|
||||
}
|
||||
|
||||
// ForgetVerify drops a removed schedule's last result (the history keeps it).
|
||||
func ForgetVerify(dataDir, id string) {
|
||||
_ = os.Remove(filepath.Join(verifyDir(dataDir), "run-"+id+".json"))
|
||||
}
|
||||
|
||||
func progressPath(dataDir string) string { return filepath.Join(verifyDir(dataDir), "running.json") }
|
||||
|
||||
// WriteVerifyProgress publishes a running pass.
|
||||
func WriteVerifyProgress(dataDir string, p VerifyProgress) { _ = writeJSON(progressPath(dataDir), p) }
|
||||
|
||||
// ClearVerifyProgress removes it once the pass ends.
|
||||
func ClearVerifyProgress(dataDir string) { _ = os.Remove(progressPath(dataDir)) }
|
||||
|
||||
// RunningVerify is the pass in progress, or nil. A progress file whose process is gone, or
|
||||
// that has not moved for two minutes, is a pass that died: it is reported as not running.
|
||||
func RunningVerify(dataDir string) *VerifyProgress {
|
||||
var p VerifyProgress
|
||||
if !readJSON(progressPath(dataDir), &p) {
|
||||
return nil
|
||||
}
|
||||
if p.PID <= 0 || syscall.Kill(p.PID, 0) != nil || time.Since(p.Updated) > 2*time.Minute {
|
||||
return nil
|
||||
}
|
||||
return &p
|
||||
}
|
||||
|
||||
// LockVerify takes the one-pass-at-a-time lock; release it with the returned func.
|
||||
func LockVerify(dataDir string) (func(), error) {
|
||||
if err := os.MkdirAll(verifyDir(dataDir), 0o700); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
f, err := os.OpenFile(filepath.Join(verifyDir(dataDir), "lock"), os.O_CREATE|os.O_RDWR, 0o600)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
|
||||
f.Close()
|
||||
return nil, errors.New("a verify pass is already running")
|
||||
}
|
||||
return func() { _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN); f.Close() }, nil
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestVerifyClassifiesEveryCase(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
src, dst := filepath.Join(root, "landing"), filepath.Join(root, "pool")
|
||||
for _, d := range []string{src, dst + "/a", src + "/a"} {
|
||||
if err := os.MkdirAll(d, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
mtime := time.Date(2026, 10, 25, 12, 0, 0, 0, time.UTC)
|
||||
put := func(p, body string) LedgerEntry {
|
||||
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Chtimes(p, mtime, mtime)
|
||||
sum := sha256.Sum256([]byte(body))
|
||||
return LedgerEntry{Size: int64(len(body)), MTimeNs: mtime.UnixNano(), SHA256: hex.EncodeToString(sum[:])}
|
||||
}
|
||||
var ledger []LedgerEntry
|
||||
add := func(rel string, e LedgerEntry, at int64) { e.Path, e.T = rel, at; ledger = append(ledger, e) }
|
||||
|
||||
add("a/match.bin", put(dst+"/a/match.bin", "intact"), 10)
|
||||
e := put(dst+"/a/mismatch.bin", "garbled") // same size and mtime as what was hashed, other bytes
|
||||
sum := sha256.Sum256([]byte("GARBLED"))
|
||||
e.SHA256 = hex.EncodeToString(sum[:])
|
||||
add("a/mismatch.bin", e, 11)
|
||||
e = put(dst+"/a/changed.bin", "edited since, longer now")
|
||||
e.Size = 3
|
||||
add("a/changed.bin", e, 12)
|
||||
add("a/gone.bin", LedgerEntry{Size: 4, MTimeNs: mtime.UnixNano(), SHA256: "00"}, 13)
|
||||
put(src+"/a/waiting.bin", "back again")
|
||||
add("a/waiting.bin", LedgerEntry{Size: 10, MTimeNs: mtime.UnixNano(), SHA256: "00"}, 14)
|
||||
e = put(dst+"/a/unhashed", "x")
|
||||
e.SHA256, e.Note = "", "not a regular file"
|
||||
add("a/unhashed", e, 15)
|
||||
add("../escape", LedgerEntry{Size: 1, SHA256: "00"}, 16)
|
||||
// moved twice: only the newer entry counts, and it matches
|
||||
add("a/twice.bin", LedgerEntry{Size: 99, SHA256: "stale"}, 5)
|
||||
add("a/twice.bin", put(dst+"/a/twice.bin", "second time"), 17)
|
||||
|
||||
lp := filepath.Join(root, "ledger.jsonl")
|
||||
f, _ := os.Create(lp)
|
||||
for _, e := range ledger {
|
||||
b, _ := json.Marshal(e)
|
||||
f.Write(append(b, '\n'))
|
||||
}
|
||||
f.WriteString("not json\n")
|
||||
f.Close()
|
||||
|
||||
tg := LandingTarget{Src: src, Dst: dst, Ledger: lp}
|
||||
r := Verify(tg, "manual", "all", 0, nil)
|
||||
want := map[string]int{"checked": 8, "matched": 2, "mismatched": 1, "changed": 1, "gone": 1, "waiting": 1, "unhashed": 1, "unreadable": 1}
|
||||
got := map[string]int{"checked": r.Checked, "matched": r.Matched, "mismatched": r.Mismatched, "changed": r.Changed,
|
||||
"gone": r.Gone, "waiting": r.Waiting, "unhashed": r.Unhashed, "unreadable": r.Unreadable}
|
||||
for k, v := range want {
|
||||
if got[k] != v {
|
||||
t.Errorf("%s: got %d want %d (%+v)", k, got[k], v, r)
|
||||
}
|
||||
}
|
||||
if r.OK() || r.Cursor != 17 || r.Error != "" {
|
||||
t.Errorf("ok=%v cursor=%d err=%q", r.OK(), r.Cursor, r.Error)
|
||||
}
|
||||
kinds := map[string]string{}
|
||||
for _, i := range r.Issues {
|
||||
kinds[i.Path] = i.Kind
|
||||
}
|
||||
if kinds["a/mismatch.bin"] != "mismatch" || kinds["a/gone.bin"] != "gone" || kinds["../escape"] != "unreadable" {
|
||||
t.Errorf("issues %+v", r.Issues)
|
||||
}
|
||||
|
||||
// "new" after a pass at cursor 14 only looks at what moved later
|
||||
r = Verify(tg, "manual", "new", 14, nil)
|
||||
if r.Checked != 3 || r.Matched != 1 || r.Unhashed != 1 || r.Unreadable != 1 {
|
||||
t.Errorf("new since 14: %+v", r)
|
||||
}
|
||||
|
||||
// records: the cursor only advances on a pass that read the ledger
|
||||
dir := t.TempDir()
|
||||
if err := RecordVerify(dir, tg, Verify(tg, "manual", "all", 0, nil)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if c := VerifyCursor(dir, tg); c != 17 {
|
||||
t.Errorf("cursor %d", c)
|
||||
}
|
||||
bad := Verify(LandingTarget{Src: src, Dst: dst, Ledger: filepath.Join(root, "missing")}, "manual", "new", 0, nil)
|
||||
if !strings.Contains(bad.Error, "no ledger yet") {
|
||||
t.Errorf("missing ledger: %q", bad.Error)
|
||||
}
|
||||
if len(VerifyHistory(dir)) != 1 || LastVerify(dir, "manual") == nil {
|
||||
t.Errorf("history/last not kept")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifyPolicy(t *testing.T) {
|
||||
p := VerifyPolicy{ID: "abc", Name: "nightly", Every: "daily", At: "05:10", Scope: "new", Enabled: true}
|
||||
if err := p.Validate(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if x, _ := p.CronExpr(); x != "10 5 * * *" {
|
||||
t.Fatalf("cron %q", x)
|
||||
}
|
||||
txt, err := VerifyCronText([]VerifyPolicy{p, {ID: "off", Name: "paused one", Every: "hourly", Scope: "all"}}, "core", "/usr/local/bin/hamlab", "/cfg.json")
|
||||
if err != nil || !strings.Contains(txt, "10 5 * * * core /usr/local/bin/hamlab -config /cfg.json landing-verify abc") || !strings.Contains(txt, "# paused one — paused") {
|
||||
t.Fatalf("%v\n%s", err, txt)
|
||||
}
|
||||
for _, b := range []VerifyPolicy{{Every: "daily", Scope: "some"}, {Every: "sometimes", Scope: "new"}, {Every: "custom", Scope: "new"}} {
|
||||
if b.Validate() == nil {
|
||||
t.Errorf("accepted %+v", b)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStaleFiles(t *testing.T) {
|
||||
d := t.TempDir()
|
||||
now := time.Now()
|
||||
os.MkdirAll(d+"/x", 0o755)
|
||||
for name, age := range map[string]time.Duration{"x/old.bin": 3 * time.Hour, "x/older.bin": 5 * time.Hour, "fresh.bin": time.Minute} {
|
||||
os.WriteFile(filepath.Join(d, name), []byte("12345"), 0o644)
|
||||
os.Chtimes(filepath.Join(d, name), now.Add(-age), now.Add(-age))
|
||||
}
|
||||
files, n, b := StaleFiles(d, 1, now)
|
||||
if n != 2 || b != 10 || len(files) != 1 || files[0].Path != "x/older.bin" {
|
||||
t.Fatalf("%d %d %+v", n, b, files)
|
||||
}
|
||||
}
|
||||
@@ -69,10 +69,6 @@ type State struct {
|
||||
// orchestrator that has since been removed can still be found on disk. See
|
||||
// internal/api/stacks.go.
|
||||
Mounts []RememberedMount `json:"mounts,omitempty"`
|
||||
// KVMCookies is each KVM's own login, as the cookies its page set in a browser: kept so
|
||||
// the next browser — or this one after it has forgotten them — is signed into the device
|
||||
// already. See internal/api/kvm.go.
|
||||
KVMCookies map[string]map[string]string `json:"kvm_cookies,omitempty"`
|
||||
SamplerMs int `json:"sampler_ms,omitempty"` // how often this machine samples; the hub keeps every machine on the console's refresh rate
|
||||
HistorySec int `json:"history_sec,omitempty"` // how far back the graphs reach; the hub keeps every machine on the same one
|
||||
// Transcode is where videos played in the Files viewer are made (internal/api/play.go):
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@
|
||||
"dev": "vite",
|
||||
"build": "tsc --noEmit && vite build",
|
||||
"icons": "node scripts/icons.mjs",
|
||||
"test": "node scripts/test-graphcols.mjs && node scripts/test-css-imports.mjs && node scripts/test-resolve-path.mjs"
|
||||
"test": "node scripts/test-graphcols.mjs && node scripts/test-sockets.mjs && node scripts/test-css-imports.mjs && node scripts/test-resolve-path.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"@xterm/addon-fit": "^0.10.0",
|
||||
|
||||
@@ -67,6 +67,63 @@ if (await page.locator('canvas.vp-sphere').count()) {
|
||||
}
|
||||
await page.screenshot({ path: `${SHOTS}/h1-auto.png` })
|
||||
|
||||
// No theme touches playback: for every theme the stylesheet has rules for, no full-window
|
||||
// overlay is drawn, the picture isn't filtered, and the player's lettering is plain.
|
||||
{
|
||||
const was = await page.evaluate(() => document.documentElement.getAttribute('data-theme'))
|
||||
const themes = await page.evaluate(() => {
|
||||
const ids = new Set()
|
||||
for (const sh of document.styleSheets) { let rules; try { rules = sh.cssRules } catch { continue }
|
||||
const walk = (rs) => { for (const r of rs) { if (r.cssRules) walk(r.cssRules); for (const m of (r.selectorText ?? '').matchAll(/data-theme="([^"]+)"/g)) ids.add(m[1]) } }
|
||||
walk(rules) }
|
||||
return [...ids]
|
||||
})
|
||||
const bad = []
|
||||
for (const t of themes) {
|
||||
const r = await page.evaluate((t) => {
|
||||
document.documentElement.setAttribute('data-theme', t)
|
||||
const els = [document.documentElement, document.body, document.querySelector('.app'), document.querySelector('#root')].filter(Boolean)
|
||||
const overlays = els.flatMap((e) => ['::before', '::after'].map((p) => getComputedStyle(e, p))).filter((c) => c.display !== 'none' && c.content !== 'none' && c.content !== 'normal' && (c.position === 'fixed' || c.position === 'absolute')).length
|
||||
const pics = [...document.querySelectorAll('.vp video, .vp canvas')].map((e) => getComputedStyle(e).filter).filter((f) => f !== 'none').length
|
||||
const c = getComputedStyle(document.querySelector('.vp'))
|
||||
const b = getComputedStyle(document.querySelector('.vp button'))
|
||||
return { overlays, pics, plain: !/mono|courier/i.test(b.fontFamily) && c.fontWeight === '400' && c.letterSpacing === 'normal' && c.textShadow === 'none' }
|
||||
}, t)
|
||||
if (r.overlays || r.pics || !r.plain) bad.push(`${t}: ${JSON.stringify(r)}`)
|
||||
}
|
||||
check(`no theme touches the video (${themes.length} themes with their own rules)`, themes.length > 0 && bad.length === 0, bad.join(' / ') || themes.join(', '))
|
||||
await page.evaluate((t) => t ? document.documentElement.setAttribute('data-theme', t) : document.documentElement.removeAttribute('data-theme'), was)
|
||||
}
|
||||
|
||||
// Volume: the speaker mutes and unmutes, the slider sets the level, and it is remembered.
|
||||
{
|
||||
await controls()
|
||||
const vs = () => page.evaluate(() => { const v = document.querySelector('.vp video'); return { muted: v.muted, volume: Math.round(v.volume * 100) / 100 } })
|
||||
await page.getByRole('button', { name: 'Mute' }).click()
|
||||
const a = await vs()
|
||||
await page.getByRole('button', { name: 'Unmute' }).click()
|
||||
const b = await vs()
|
||||
let c = b
|
||||
if (await page.getByLabel('Volume').isVisible()) { await page.getByLabel('Volume').fill('0.3'); c = await vs() }
|
||||
const kept = await page.evaluate(() => Object.entries(localStorage).find(([k]) => /volume/.test(k))?.[1] ?? '')
|
||||
check('volume: mute, unmute, level, remembered', a.muted && !b.muted && (c.volume === 0.3 || !(await page.getByLabel('Volume').isVisible())) && /level/.test(kept),
|
||||
`${JSON.stringify(a)} → ${JSON.stringify(b)} → ${JSON.stringify(c)}, kept ${kept}`)
|
||||
if (await page.getByLabel('Volume').isVisible()) await page.getByLabel('Volume').fill('1')
|
||||
}
|
||||
|
||||
// The 5-second buttons: back and on, the ball with them.
|
||||
{
|
||||
await controls()
|
||||
const t0 = (await vstate()).t
|
||||
await page.getByRole('button', { name: 'On 5 seconds' }).click()
|
||||
const t1 = (await vstate()).t
|
||||
await page.getByRole('button', { name: 'Back 5 seconds' }).click()
|
||||
await page.getByRole('button', { name: 'Back 5 seconds' }).click()
|
||||
const t2 = (await vstate()).t
|
||||
const ball = await page.evaluate(() => Number(document.querySelector('.vp input[type=range]').value))
|
||||
check('+5 and −5 buttons', t1 - t0 > 4 && t1 - t0 < 6.5 && t1 - t2 > 9 && t1 - t2 < 11 && Math.abs(ball - t2) < 1, `${t0.toFixed(1)} → ${t1.toFixed(1)} → ${t2.toFixed(1)}, ball ${ball.toFixed(1)}`)
|
||||
}
|
||||
|
||||
const mid = Math.floor(s.d / 2)
|
||||
mark('seek')
|
||||
await page.evaluate((m) => { document.querySelector('.vp video').currentTime = m }, mid)
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
// node scripts/test-sockets.mjs — src/sockets.ts: each CPU's threads, counted under it.
|
||||
import { buildSync } from 'esbuild'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { mkdtempSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
|
||||
const out = join(mkdtempSync(join(tmpdir(), 'sockets-')), 'sockets.mjs')
|
||||
buildSync({ entryPoints: [new URL('../src/sockets.ts', import.meta.url).pathname], outfile: out, format: 'esm', logLevel: 'error' })
|
||||
const { socketThreads } = await import(pathToFileURL(out).href)
|
||||
|
||||
const fails = []
|
||||
const eq = (got, want, what) => { if (JSON.stringify(got) !== JSON.stringify(want)) fails.push(`${what} — got ${JSON.stringify(got)}, want ${JSON.stringify(want)}`) }
|
||||
const range = (a, b) => Array.from({ length: b - a }, (_, i) => a + i)
|
||||
|
||||
// nexus: 2 × E5-2680 v2, as /sys/devices/system/cpu/cpu*/topology/physical_package_id reads
|
||||
const nexus = [...Array(10).fill(0), ...Array(10).fill(1), ...Array(10).fill(0), ...Array(10).fill(1)]
|
||||
const got = socketThreads(nexus)
|
||||
eq(got.map((s) => s.pkg), [0, 1], 'two sockets, in order')
|
||||
eq(got[0].cpus, [...range(0, 10), ...range(20, 30)], "CPU 1's threads")
|
||||
eq(got[1].cpus, [...range(10, 20), ...range(30, 40)], "CPU 2's threads")
|
||||
eq(got.map((s) => s.cpus.length), [20, 20], 'twenty threads each')
|
||||
// every thread is under exactly one CPU
|
||||
eq(got.flatMap((s) => s.cpus).sort((a, b) => a - b), range(0, 40), 'every thread once')
|
||||
// a thread's number under its CPU runs 0…19 on both
|
||||
for (const s of got) eq(s.cpus.map((_, n) => n), range(0, 20), `CPU ${s.pkg + 1} counts from 0`)
|
||||
|
||||
// a board that numbers each socket's threads together, and one whose socket ids are not 0 and 1
|
||||
eq(socketThreads([0, 0, 0, 0, 1, 1, 1, 1]), [{ pkg: 0, cpus: [0, 1, 2, 3] }, { pkg: 1, cpus: [4, 5, 6, 7] }], 'contiguous numbering')
|
||||
eq(socketThreads([3, 1, 3, 1]), [{ pkg: 1, cpus: [1, 3] }, { pkg: 3, cpus: [0, 2] }], 'socket ids sorted')
|
||||
eq(socketThreads([]), [], 'nothing')
|
||||
|
||||
if (fails.length) { console.error('sockets: FAIL\n ' + fails.join('\n ')); process.exit(1) }
|
||||
console.log('sockets: ok')
|
||||
+62
-7
@@ -1,4 +1,4 @@
|
||||
import { DragEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
|
||||
import { DragEvent, PointerEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
|
||||
import { api, FileEntry, FileInfo, TaskInfo, ThumbStatus } from './api'
|
||||
import { fmtBytes } from './fmt'
|
||||
import { Btn, Loading, Spinner, askText, askYes, setUnsaved, toast, useEscape } from './ui'
|
||||
@@ -22,6 +22,7 @@ export function resolvePath(input: string, here: string): string {
|
||||
}
|
||||
const fmtTime = (ms: number) => { const d = new Date(ms); const now = new Date(); return d.toDateString() === now.toDateString() ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) : d.toLocaleDateString(undefined, { year: d.getFullYear() === now.getFullYear() ? undefined : 'numeric', month: 'short', day: '2-digit' }) }
|
||||
const isImage = (n: string) => /\.(png|jpe?g|gif|webp|svg|bmp|ico|avif)$/i.test(n)
|
||||
const isMedia = (e: FileEntry) => !!e.preview || isImage(e.name)
|
||||
const shellQuote = (s: string) => "'" + s.replace(/'/g, "'\\''") + "'"
|
||||
const MAX_EDIT = 2 * 1024 * 1024
|
||||
|
||||
@@ -198,7 +199,9 @@ function ThumbNote({ st, sudo, onChanged }: { st: ThumbStatus | null; sudo: bool
|
||||
}
|
||||
|
||||
// ---- editor / viewer ----
|
||||
function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolean; onClose: () => void; onChanged: () => void }) {
|
||||
// steps: the files ← and → go through, in the list's order (only the pictures, videos and
|
||||
// sound when one of those is open).
|
||||
function Viewer({ path, sudo, steps, onStep, onClose, onChanged }: { path: string; sudo: boolean; steps: string[]; onStep: (p: string) => void; onClose: () => void; onChanged: () => void }) {
|
||||
const [info, setInfo] = useState<FileInfo | null>(null)
|
||||
const [text, setText] = useState<string | null>(null)
|
||||
const [orig, setOrig] = useState('')
|
||||
@@ -251,10 +254,49 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
|
||||
}, [dirty, name])
|
||||
const dl = api.fileDownloadURL(path, sudo)
|
||||
|
||||
// Previous and next: the buttons, ← and → (unless typing, or the player has the focus and
|
||||
// seeks with them), and a sideways swipe on a picture.
|
||||
const at = steps.indexOf(path)
|
||||
const prev = at > 0 ? steps[at - 1] : null
|
||||
const next = at >= 0 && at < steps.length - 1 ? steps[at + 1] : null
|
||||
const step = useCallback(async (to: string | null) => {
|
||||
if (!to) return
|
||||
if (dirty && !await askYes('Discard unsaved changes?', 'Discard')) return
|
||||
setUnsaved(''); onStep(to)
|
||||
}, [dirty, onStep])
|
||||
useEffect(() => {
|
||||
const h = (e: KeyboardEvent) => {
|
||||
if (e.defaultPrevented || e.altKey || e.ctrlKey || e.metaKey || e.shiftKey) return
|
||||
const t = e.target as HTMLElement | null
|
||||
if (t?.closest('input, textarea, select, [contenteditable], .vp')) return
|
||||
if (e.key === 'ArrowLeft' && prev) { e.preventDefault(); step(prev) }
|
||||
else if (e.key === 'ArrowRight' && next) { e.preventDefault(); step(next) }
|
||||
}
|
||||
window.addEventListener('keydown', h)
|
||||
return () => window.removeEventListener('keydown', h)
|
||||
}, [prev, next, step])
|
||||
// The next and previous pictures load now, so stepping onto them is instant.
|
||||
useEffect(() => {
|
||||
for (const p of [prev, next]) if (p && isImage(p)) new Image().src = api.fileReadURL(p, sudo)
|
||||
}, [prev, next, sudo])
|
||||
const swipe = useRef<{ x: number; y: number } | null>(null)
|
||||
const swipeEnd = (e: PointerEvent) => {
|
||||
const s = swipe.current
|
||||
swipe.current = null
|
||||
if (!s) return
|
||||
const dx = e.clientX - s.x, dy = e.clientY - s.y
|
||||
if (Math.abs(dx) > 50 && Math.abs(dx) > 1.5 * Math.abs(dy)) step(dx < 0 ? next : prev)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="viewer">
|
||||
<div className="viewer-bar">
|
||||
<button className="btn" onClick={close}>‹ Back</button>
|
||||
{at >= 0 && steps.length > 1 && <>
|
||||
<button className="btn step" disabled={!prev} title="previous (←)" onClick={() => step(prev)}>←</button>
|
||||
<span className="muted num">{at + 1}/{steps.length}</span>
|
||||
<button className="btn step" disabled={!next} title="next (→)" onClick={() => step(next)}>→</button>
|
||||
</>}
|
||||
<span className="vname mono" title={path}>{name}{dirty && <span className="dirty"> •</span>}</span>
|
||||
<span className="spacer" />
|
||||
{info && <span className="muted num">{fmtBytes(info.size)} · {info.perm} · {info.user}</span>}
|
||||
@@ -266,11 +308,17 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
|
||||
{info && text !== null && (
|
||||
<textarea className="editor file-editor" value={text} spellCheck={false} onChange={(e) => setText(e.target.value)} />
|
||||
)}
|
||||
{info && text === null && isImage(name) && <div className="preview"><img src={api.fileReadURL(path, sudo)} alt={name} /></div>}
|
||||
{info && text === null && isImage(name) && (
|
||||
<div className="preview pic" onPointerDown={(e) => { if (e.pointerType !== 'mouse' && e.isPrimary) swipe.current = { x: e.clientX, y: e.clientY } }} onPointerUp={swipeEnd} onPointerCancel={() => { swipe.current = null }}>
|
||||
<img src={api.fileReadURL(path, sudo)} alt={name} draggable={false} />
|
||||
{prev && <button className="step-edge prev" title="previous (←)" onClick={() => step(prev)}>‹</button>}
|
||||
{next && <button className="step-edge next" title="next (→)" onClick={() => step(next)}>›</button>}
|
||||
</div>
|
||||
)}
|
||||
{/* Video: the player (Auto, by hand, or the file as it is), made on the Transcode device. Audio as it is. */}
|
||||
{info && text === null && !isImage(name) && info.preview === 'video' && <Player path={path} sudo={sudo} name={name} />}
|
||||
{info && text === null && !isImage(name) && info.preview === 'video' && <Player key={path} path={path} sudo={sudo} name={name} />}
|
||||
{info && text === null && !isImage(name) && info.preview === 'audio' && !cantPlay && (
|
||||
<div className="preview audio"><div className="audio-box"><div className="ficon">🎵</div><div className="mono small">{name}</div><audio controls preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => setCantPlay(true)} /></div></div>
|
||||
<div className="preview audio"><div className="audio-box"><div className="ficon">🎵</div><div className="mono small">{name}</div><audio controls autoPlay preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => setCantPlay(true)} /></div></div>
|
||||
)}
|
||||
{info && text === null && !isImage(name) && info.preview === 'audio' && cantPlay && (
|
||||
<div className="notice">This browser can't play {name} as it is — the codec isn't one it decodes. Download it instead.</div>
|
||||
@@ -307,6 +355,9 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
||||
const [thumbs, setThumbs] = useState<ThumbStatus | null>(null)
|
||||
const [listEl, setListEl] = useState<HTMLDivElement | null>(null)
|
||||
const near = useMemo(() => (listEl && previews ? new Near(listEl) : null), [listEl, previews])
|
||||
// Video previews are made on the Transcode device (Settings), wherever it is.
|
||||
const [tc, setTc] = useState<string>('')
|
||||
useEffect(() => { api.transcodeSelected().then((t) => setTc(t.selected ? `videos on ${t.host}'s ${t.name} (Settings → Transcode device), images on this machine's CPU` : '')).catch(() => {}) }, [])
|
||||
const readThumbs = useCallback((refresh = false) => { api.thumbStatus(sudo, refresh).then(setThumbs).catch((e) => setThumbs({ ok: false, engine: '', summary: '', gpus: [], error: e instanceof Error ? e.message : 'failed', user: '', cache: { path: '', entries: 0, bytes: 0 } })) }, [sudo])
|
||||
useEffect(() => { try { localStorage.setItem(PREVIEWS_KEY, previews ? '1' : '0') } catch { /* private mode */ } }, [previews])
|
||||
useEffect(() => { if (previews) readThumbs(); else setThumbs(null) }, [previews, readThumbs])
|
||||
@@ -396,7 +447,11 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
||||
if (files.length) uploadFiles(files)
|
||||
}
|
||||
|
||||
if (open) return <div className="files"><Viewer path={open} sudo={sudo} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
|
||||
if (open) {
|
||||
const cur = shown.find((e) => join(path, e.name) === open)
|
||||
const steps = shown.filter((e) => e.type !== 'dir' && e.type !== 'other' && (!cur || !isMedia(cur) || isMedia(e))).map((e) => join(path, e.name))
|
||||
return <div className="files"><Viewer path={open} sudo={sudo} steps={steps} onStep={setOpen} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
|
||||
}
|
||||
|
||||
return (
|
||||
<div className={'files' + (drag ? ' drag' : '')} onDragOver={(e) => { e.preventDefault(); setDrag(true) }} onDragLeave={() => setDrag(false)} onDrop={onDrop}>
|
||||
@@ -409,7 +464,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
||||
<div className="files-tools">
|
||||
<button className={'btn' + (sudo ? ' on' : '')} title="browse as root" onClick={() => setSudo(!sudo)}>{sudo ? 'root' : 'sudo'}</button>
|
||||
<button className={'btn' + (hidden ? ' on' : '')} title="show hidden files" onClick={() => setHidden(!hidden)}>.hidden</button>
|
||||
<button className={'btn' + (previews ? ' on' : '')} title={'thumbnails for images and videos' + (thumbs ? ` — ${thumbs.summary}` + (thumbs.cache.entries ? ` · ${thumbs.cache.entries} kept (${fmtBytes(thumbs.cache.bytes)})` : '') : '')} onClick={() => setPreviews(!previews)}>previews</button>
|
||||
<button className={'btn' + (previews ? ' on' : '')} title={'thumbnails for images and videos' + (thumbs ? ` — ${tc || thumbs.summary}` + (thumbs.cache.entries ? ` · ${thumbs.cache.entries} kept (${fmtBytes(thumbs.cache.bytes)})` : '') : '')} onClick={() => setPreviews(!previews)}>previews</button>
|
||||
<select className="btn" value={sort} onChange={(e) => setSort(e.target.value as Sort)}><option value="name">name</option><option value="size">size</option><option value="mtime">modified</option></select>
|
||||
<input className="filter" placeholder="filter" value={q} onChange={(e) => setQ(e.target.value)} />
|
||||
<button className="btn" onClick={mkdir}>+ folder</button>
|
||||
|
||||
+13
-4
@@ -158,6 +158,10 @@ export default function Hardware() {
|
||||
const capacity = drives.reduce((n, d) => n + d.size, 0)
|
||||
const threads = cpu.threads ?? 0
|
||||
const cores = cpu.cores ?? 0
|
||||
// a two-socket board is two processors, not one with twice the cores: count and name each
|
||||
const sockets = Math.max(1, cpu.sockets ?? 1)
|
||||
const multi = sockets > 1
|
||||
const per = (n: number) => Math.round(n / sockets)
|
||||
|
||||
return (
|
||||
<div className="dash">
|
||||
@@ -171,8 +175,11 @@ export default function Hardware() {
|
||||
<div className="kpis">
|
||||
<div className="tile">
|
||||
<div className="label">Processor</div>
|
||||
<div className="value" title={cpu.model} style={(cpu.model?.length ?? 0) > 20 ? { fontSize: 16 } : undefined}>{cpu.model}</div>
|
||||
<div className="sub">{[cores ? `${cores} core${cores === 1 ? '' : 's'}` : '', threads && threads !== cores ? `${threads} threads` : '', cpu.max_mhz ? `up to ${ghz(cpu.max_mhz)}` : ''].filter(Boolean).join(' · ')}</div>
|
||||
<div className="value" title={cpu.model} style={(cpu.model?.length ?? 0) > (multi ? 16 : 20) ? { fontSize: 16 } : undefined}>{multi ? `${sockets} × ${cpu.model}` : cpu.model}</div>
|
||||
<div className="sub">{(multi
|
||||
? [`${sockets} CPUs`, cores ? `${per(cores)} cores${threads && threads !== cores ? ` / ${per(threads)} threads` : ''} each` : '', cpu.max_mhz ? `up to ${ghz(cpu.max_mhz)}` : '']
|
||||
: [cores ? `${cores} core${cores === 1 ? '' : 's'}` : '', threads && threads !== cores ? `${threads} threads` : '', cpu.max_mhz ? `up to ${ghz(cpu.max_mhz)}` : '']
|
||||
).filter(Boolean).join(' · ')}</div>
|
||||
</div>
|
||||
<div className="tile">
|
||||
<div className="label">Memory</div>
|
||||
@@ -199,9 +206,11 @@ export default function Hardware() {
|
||||
<div className="props">
|
||||
<P k="Model" v={cpu.model} />
|
||||
<P k="Vendor" v={cpu.vendor} />
|
||||
<P k="Cores" v={cores ? `${cores}${cpu.threads_per_core && cpu.threads_per_core > 1 ? ` · ${cpu.threads_per_core} threads each` : ''}` : ''} />
|
||||
{multi && <P k="CPUs" v={`${sockets}`} />}
|
||||
<P k="Cores" v={cores ? `${multi ? `${per(cores)} per CPU · ${cores} in all` : cores}${cpu.threads_per_core && cpu.threads_per_core > 1 ? ` · ${cpu.threads_per_core} threads each` : ''}` : ''} />
|
||||
{multi && threads > 0 && <P k="Threads" v={`${per(threads)} per CPU · ${threads} in all`} />}
|
||||
<P k="Clock" v={cpu.min_mhz && cpu.max_mhz ? `${ghz(cpu.min_mhz)} – ${ghz(cpu.max_mhz)}` : ghz(cpu.max_mhz)} />
|
||||
<P k="Socket" v={(cpu.socket_names ?? []).join(', ')} />
|
||||
<P k={multi ? 'Sockets' : 'Socket'} v={(cpu.socket_names ?? []).join(', ')} />
|
||||
<P k="Arch" v={cpu.arch} />
|
||||
<P k="Family" v={[cpu.family && `family ${cpu.family}`, cpu.model_id && `model ${cpu.model_id}`, cpu.stepping && `stepping ${cpu.stepping}`].filter(Boolean).join(' · ')} />
|
||||
<P k="Microcode" v={cpu.microcode} mono />
|
||||
|
||||
-187
@@ -1,187 +0,0 @@
|
||||
// The KVM of the machine you are looking at: the device's own web page, shown in the tab.
|
||||
//
|
||||
// One per machine, and it follows the machine switcher — switch to `grid` and you get
|
||||
// grid's KVM. The address is held on the **hub**, not on the machine, because a KVM earns
|
||||
// its keep exactly when that machine is down and cannot answer for itself; the hub proxies
|
||||
// the device either way.
|
||||
//
|
||||
// Nothing here knows anything about any particular KVM. It is an iframe pointed at Ham
|
||||
// Lab's own proxy, so whatever web UI the device serves is what you get.
|
||||
//
|
||||
// On the keyboard: a page cannot capture keys and forward them into an iframe, and does not
|
||||
// need to. An iframe with focus receives them directly. What the browser itself swallows —
|
||||
// Escape, Tab, Ctrl-W, the function keys — is what `navigator.keyboard.lock()` releases,
|
||||
// and that needs fullscreen and only exists in Chromium. Everywhere else you get the plain
|
||||
// version, which is most keys, and the device's own on-screen buttons for the rest.
|
||||
import { FormEvent, useCallback, useEffect, useRef, useState } from 'react'
|
||||
import { KVM as Device, api } from './api'
|
||||
import { Btn, Loading, askYes, toast, useEscape } from './ui'
|
||||
|
||||
// lock() releases the keys the browser would otherwise keep for itself. Chromium only; a
|
||||
// browser without it simply does not get the extra keys.
|
||||
type KeyboardLock = { lock?: (keys?: string[]) => Promise<void>; unlock?: () => void }
|
||||
const keyboardAPI = (): KeyboardLock | undefined => (navigator as unknown as { keyboard?: KeyboardLock }).keyboard
|
||||
|
||||
// Fullscreen, across the three ways browsers spell it. Safari on a Mac wants the webkit
|
||||
// prefix; Safari on iOS has no fullscreen for an ordinary element at all — only for a
|
||||
// <video> — so there the page expands inside the viewport instead, which is the same thing
|
||||
// minus the browser's own chrome.
|
||||
type FSElement = HTMLElement & { webkitRequestFullscreen?: () => Promise<void> | void }
|
||||
type FSDocument = Document & { webkitFullscreenElement?: Element | null; webkitExitFullscreen?: () => Promise<void> | void }
|
||||
const nativeFullscreen = () => {
|
||||
if (typeof document === 'undefined') return false
|
||||
const el = document.documentElement as FSElement
|
||||
return typeof el.requestFullscreen === 'function' || typeof el.webkitRequestFullscreen === 'function'
|
||||
}
|
||||
const fullscreenElement = () => (document as FSDocument).fullscreenElement ?? (document as FSDocument).webkitFullscreenElement ?? null
|
||||
async function enterFullscreen(el: FSElement) {
|
||||
if (typeof el.requestFullscreen === 'function') return el.requestFullscreen()
|
||||
if (typeof el.webkitRequestFullscreen === 'function') return el.webkitRequestFullscreen()
|
||||
throw new Error('this browser has no fullscreen for a page element')
|
||||
}
|
||||
function exitFullscreen() {
|
||||
const d = document as FSDocument
|
||||
if (typeof d.exitFullscreen === 'function') return d.exitFullscreen()
|
||||
if (typeof d.webkitExitFullscreen === 'function') return d.webkitExitFullscreen()
|
||||
}
|
||||
|
||||
export default function KVM({ machine, title }: { machine: string | null; title: string }) {
|
||||
const [kvm, setKvm] = useState<Device | null | undefined>(undefined)
|
||||
const [adding, setAdding] = useState(false)
|
||||
const [err, setErr] = useState('')
|
||||
const [full, setFull] = useState(false) // native fullscreen is on
|
||||
const [expanded, setExpanded] = useState(false) // the fallback: filling the viewport in-page
|
||||
const box = useRef<HTMLDivElement>(null)
|
||||
const frame = useRef<HTMLIFrameElement>(null)
|
||||
const key = machine ?? '' // the hub is a machine too, and its key is the empty one
|
||||
|
||||
const load = useCallback(async () => {
|
||||
try { setKvm((await api.kvms()).find((k) => k.machine === key) ?? null) }
|
||||
catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
|
||||
}, [key])
|
||||
useEffect(() => { load() }, [load])
|
||||
|
||||
// Fullscreen plus a keyboard lock is what makes Escape and Tab reach the machine rather
|
||||
// than the browser. Both are undone together when fullscreen ends, however it ends.
|
||||
useEffect(() => {
|
||||
const onChange = () => {
|
||||
const on = !!fullscreenElement()
|
||||
setFull(on)
|
||||
if (!on) keyboardAPI()?.unlock?.()
|
||||
}
|
||||
document.addEventListener('fullscreenchange', onChange)
|
||||
document.addEventListener('webkitfullscreenchange', onChange)
|
||||
return () => {
|
||||
document.removeEventListener('fullscreenchange', onChange)
|
||||
document.removeEventListener('webkitfullscreenchange', onChange)
|
||||
keyboardAPI()?.unlock?.()
|
||||
}
|
||||
}, [])
|
||||
// Expanding is in-page, so Escape is ours to handle.
|
||||
useEscape(() => setExpanded(false), expanded)
|
||||
|
||||
const goBig = async () => {
|
||||
if (!nativeFullscreen()) { setExpanded(true); frame.current?.focus(); return }
|
||||
try {
|
||||
if (box.current) await enterFullscreen(box.current)
|
||||
await keyboardAPI()?.lock?.()
|
||||
frame.current?.focus()
|
||||
} catch {
|
||||
setExpanded(true) // refused (an iframe policy, a permissions policy): fill the page instead
|
||||
frame.current?.focus()
|
||||
}
|
||||
}
|
||||
const goSmall = () => { setExpanded(false); if (fullscreenElement()) exitFullscreen() }
|
||||
|
||||
const remove = async () => {
|
||||
if (!kvm || !await askYes(`Forget ${title}'s KVM? The device itself is not touched — this only removes its address.`, 'Remove')) return
|
||||
try { await api.removeKVM(kvm.id); toast('removed'); await load() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
|
||||
}
|
||||
|
||||
if (err && kvm === undefined) return <div className="dash"><div className="error">{err}</div></div>
|
||||
if (kvm === undefined) return <div className="dash"><Loading label="Looking…" /></div>
|
||||
|
||||
const locks = !!keyboardAPI()?.lock
|
||||
const big = full || expanded
|
||||
|
||||
return (
|
||||
<div className="kvm">
|
||||
<div className="kvm-bar">
|
||||
<span className="dname">{title}</span>
|
||||
{kvm
|
||||
? <span className="muted small mono kvm-url" title={kvm.url}>{kvm.name !== kvm.url ? `${kvm.name} · ` : ''}{kvm.url}</span>
|
||||
: <span className="muted small">no KVM</span>}
|
||||
<span className="spacer" />
|
||||
{kvm && <>
|
||||
<Btn className="btn" onClick={goBig} title={!nativeFullscreen() ? 'fill the screen — this browser has no true fullscreen for a page element' : locks ? 'fullscreen, and the browser stops keeping Escape, Tab and Ctrl-W for itself' : 'fullscreen — this browser will still keep some keys for itself'}>{nativeFullscreen() ? 'Fullscreen' : 'Expand'}</Btn>
|
||||
<a className="btn" href={kvm.url} target="_blank" rel="noreferrer" title="the device directly, outside Ham Lab — only works on the LAN">Open ↗</a>
|
||||
<Btn className="btn" onClick={() => setAdding(true)}>Change</Btn>
|
||||
<Btn className="btn danger" onClick={remove}>Remove</Btn>
|
||||
</>}
|
||||
{!kvm && <Btn className="btn on" onClick={() => setAdding(true)}>+ Add a KVM for {title}</Btn>}
|
||||
</div>
|
||||
|
||||
{kvm ? (
|
||||
<div className={'kvm-frame' + (full ? ' full' : '') + (expanded ? ' expanded' : '')} ref={box}>
|
||||
<iframe ref={frame} key={kvm.id} src={`/kvm/${kvm.id}/`} title={kvm.name}
|
||||
allow="fullscreen; clipboard-read; clipboard-write" />
|
||||
{expanded && <button className="btn kvm-shrink" onClick={goSmall} title="back to the console">Done</button>}
|
||||
</div>
|
||||
) : (
|
||||
<div className="svc-empty muted kvm-none">
|
||||
<div>
|
||||
<p>No KVM for <b>{title}</b>.</p>
|
||||
<p className="small">Add one with its address and its page appears here. The address is kept on the hub,
|
||||
so it still works when {title} does not — which is when you will want it.</p>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
{kvm && !big && (
|
||||
<div className="kvm-hint muted small">
|
||||
Click the picture, then type — the keys go to {title}.
|
||||
{locks ? ' Fullscreen also takes Escape, Tab and Ctrl-W back from the browser.' : ' This browser keeps Escape, Tab and Ctrl-W for itself; use the device’s own on-screen keys for those, or a Chromium browser.'}
|
||||
</div>
|
||||
)}
|
||||
{adding && <AddDialog machine={key} title={title} existing={kvm} onClose={() => setAdding(false)} onAdded={() => { setAdding(false); load() }} />}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function AddDialog({ machine, title, existing, onClose, onAdded }: { machine: string; title: string; existing: Device | null; onClose: () => void; onAdded: () => void }) {
|
||||
const [url, setUrl] = useState(existing?.url ?? '')
|
||||
const [name, setName] = useState(existing?.name ?? '')
|
||||
const [err, setErr] = useState('')
|
||||
const [busy, setBusy] = useState(false)
|
||||
useEscape(onClose)
|
||||
const save = async (e: FormEvent) => {
|
||||
e.preventDefault(); setErr(''); setBusy(true)
|
||||
try { await api.addKVM(name.trim(), url.trim(), machine); onAdded() }
|
||||
catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setBusy(false) }
|
||||
}
|
||||
return (
|
||||
<div className="modal-back" onClick={onClose}>
|
||||
<form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
|
||||
<h3>{existing ? `Change ${title}'s KVM` : `A KVM for ${title}`}</h3>
|
||||
<label className="muted">Address — whatever you would type into a browser
|
||||
<input autoFocus className="mono" value={url} onChange={(e) => setUrl(e.target.value)} placeholder="10.0.0.217" spellCheck={false} autoCapitalize="off" required />
|
||||
</label>
|
||||
<label className="muted">Name — optional; the address is used otherwise
|
||||
<input value={name} onChange={(e) => setName(e.target.value)} placeholder={`${title} KVM`} maxLength={64} />
|
||||
</label>
|
||||
<div className="muted small">
|
||||
Ham Lab serves the device from its own address, so a KVM that speaks only plain HTTP
|
||||
still works here, and it is behind this login rather than needing a second one. The
|
||||
address is kept on the hub, so it still reaches the device when {title} is down. The
|
||||
device itself is not changed and is never reachable from outside.
|
||||
{existing && <> Saving replaces the one there now — a machine has one KVM.</>}
|
||||
</div>
|
||||
{err && <div className="error">{err}</div>}
|
||||
<div className="svc-actions">
|
||||
<button type="submit" className="btn on" disabled={busy || !url.trim()}>{existing ? 'Save' : 'Add'}</button>
|
||||
<span className="spacer" />
|
||||
<button type="button" className="btn" onClick={onClose}>Cancel</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
+65
-6
@@ -53,6 +53,9 @@ class PatientAbr extends Base {
|
||||
}
|
||||
}
|
||||
|
||||
// Whether the browser lets a page set the volume level (iOS doesn't: only mute).
|
||||
const canLevel = (() => { try { const a = document.createElement('audio'); a.volume = 0.5; return a.volume === 0.5 } catch { return false } })()
|
||||
|
||||
const mbps = (b: number) => `${b >= 10e6 ? Math.round(b / 1e6) : (b / 1e6).toFixed(1)} Mb/s`
|
||||
const clock = (sec: number) => { const t = Math.max(0, Math.floor(sec)), h = Math.floor(t / 3600), m = Math.floor((t % 3600) / 60), x = t % 60; return `${h ? `${h}:${String(m).padStart(2, '0')}` : m}:${String(x).padStart(2, '0')}` }
|
||||
const size = (w?: number, h?: number, vr?: boolean) => {
|
||||
@@ -68,6 +71,22 @@ const I = ({ d, size: sz = 20, fill }: { d: string; size?: number; fill?: boolea
|
||||
const PLAY = 'M7 4.5v15l12.5-7.5z', PAUSE = 'M8 5v14M16 5v14', CHECK = 'M5 12.5l4.5 4.5L19 7.5', TUNE = 'M4 7h10M18 7h2M4 17h4M12 17h8M16 5v4M10 15v4'
|
||||
const FULL = 'M4 9V4h5M20 9V4h-5M4 15v5h5M20 15v5h-5', SMALL = 'M9 4v5H4M15 4v5h5M9 20v-5H4M15 20v-5h5', PHONE = 'M8 3h8a1 1 0 0 1 1 1v16a1 1 0 0 1-1 1H8a1 1 0 0 1-1-1V4a1 1 0 0 1 1-1zM3 9a9 9 0 0 0 0 6M21 9a9 9 0 0 1 0 6'
|
||||
|
||||
// The speaker: waves by the level, a cross when muted.
|
||||
const Speaker = ({ off, low }: { off?: boolean; low?: boolean }) => (
|
||||
<svg width={20} height={20} viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="1.8" strokeLinecap="round" strokeLinejoin="round" aria-hidden="true">
|
||||
<path d="M4 9.5h3.5L12 5.5v13l-4.5-4H4z" fill="currentColor" />
|
||||
{off ? <path d="M16 9.5l5 5M21 9.5l-5 5" /> : <path d={low ? 'M15.5 9.5a3.5 3.5 0 0 1 0 5' : 'M15.5 9.5a3.5 3.5 0 0 1 0 5M18 7a7 7 0 0 1 0 10'} />}
|
||||
</svg>
|
||||
)
|
||||
|
||||
// Five seconds back or on: a circling arrow with the 5 inside.
|
||||
const Skip = ({ back }: { back?: boolean }) => (
|
||||
<svg width={20} height={20} viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="1.8" strokeLinecap="round" strokeLinejoin="round" aria-hidden="true">
|
||||
<path d={back ? 'M12 5a7 7 0 1 1-7 7M12 5l2.5-2.5M12 5l2.5 2.5' : 'M12 5a7 7 0 1 0 7 7M12 5L9.5 2.5M12 5l-2.5 2.5'} />
|
||||
<text x="12" y="15" textAnchor="middle" fontSize="7.5" fontWeight="700" fill="currentColor" stroke="none" fontFamily="system-ui, sans-serif">5</text>
|
||||
</svg>
|
||||
)
|
||||
|
||||
export default function Player({ path, sudo }: { path: string; sudo: boolean; name: string }) {
|
||||
const [open, setOpen] = useState<PlayOpen | null>(null)
|
||||
const [err, setErr] = useState('')
|
||||
@@ -214,10 +233,40 @@ export default function Player({ path, sudo }: { path: string; sudo: boolean; na
|
||||
else v.pause()
|
||||
poke()
|
||||
}, [poke])
|
||||
// The seek bar: while it is dragged the ball follows the finger (the input event); the seek
|
||||
// itself is on release (the browser's own change event, which a click, a drag and the arrow
|
||||
// keys all end with). The ball is put where the seek went at once, so it waits there while
|
||||
// the piece is made rather than going back to where the film was.
|
||||
const seekBar = useRef<HTMLInputElement>(null)
|
||||
useEffect(() => {
|
||||
const el = seekBar.current
|
||||
if (!el) return
|
||||
const commit = () => {
|
||||
const x = Number(el.value), v = video.current
|
||||
if (v && Math.abs(v.currentTime - x) > 0.05) v.currentTime = x
|
||||
setTime(x)
|
||||
setScrub(null)
|
||||
}
|
||||
el.addEventListener('change', commit)
|
||||
return () => el.removeEventListener('change', commit)
|
||||
})
|
||||
// Volume: a level and mute, remembered for the next video. iOS lets a page mute but not set
|
||||
// the level (it reads back as 1), so there the slider isn't shown.
|
||||
const [vol, setVol] = useState(() => { try { const v = JSON.parse(localStorage.getItem('hamlab.play.volume') ?? ''); return { level: Math.min(1, Math.max(0, Number(v.level) || 0)), muted: !!v.muted } } catch { return { level: 1, muted: false } } })
|
||||
useEffect(() => {
|
||||
const v = video.current
|
||||
if (v) { v.volume = vol.level; v.muted = vol.muted || vol.level === 0 }
|
||||
try { localStorage.setItem('hamlab.play.volume', JSON.stringify(vol)) } catch { /* private mode */ }
|
||||
}, [vol, open])
|
||||
const volBy = useCallback((d: number) => setVol((x) => ({ level: Math.round(Math.min(1, Math.max(0, x.level + d)) * 20) / 20, muted: false })), [])
|
||||
const muteToggle = useCallback(() => setVol((x) => (x.muted || x.level === 0 ? { level: x.level || 0.5, muted: false } : { ...x, muted: true })), [])
|
||||
|
||||
const seekBy = useCallback((d: number) => {
|
||||
const v = video.current
|
||||
if (!v) return
|
||||
v.currentTime = Math.max(0, Math.min(info?.duration ?? v.duration, v.currentTime + d))
|
||||
const to = Math.max(0, Math.min(info?.duration ?? v.duration, v.currentTime + d))
|
||||
v.currentTime = to
|
||||
setTime(to) // the ball goes there at once, made yet or not
|
||||
poke()
|
||||
}, [info, poke])
|
||||
// Full screen: the browser's where it has one for an element, else (an iPhone) the window.
|
||||
@@ -241,6 +290,9 @@ export default function Player({ path, sudo }: { path: string; sudo: boolean; na
|
||||
else if (e.key === 'ArrowLeft' || e.key === 'j') { e.preventDefault(); seekBy(e.key === 'j' ? -10 : -5) }
|
||||
else if (e.key === 'ArrowRight' || e.key === 'l') { e.preventDefault(); seekBy(e.key === 'l' ? 10 : 5) }
|
||||
else if (e.key === 'f') fullscreen()
|
||||
else if (e.key === 'm') muteToggle()
|
||||
else if (e.key === 'ArrowUp') { e.preventDefault(); volBy(0.1) }
|
||||
else if (e.key === 'ArrowDown') { e.preventDefault(); volBy(-0.1) }
|
||||
else if (e.key === 'Escape') { if (menu) setMenu(false); else if (full && !document.fullscreenElement) setFull(false) }
|
||||
}
|
||||
|
||||
@@ -313,15 +365,22 @@ export default function Player({ path, sudo }: { path: string; sudo: boolean; na
|
||||
{vr && open && chrome && !menu && time < 8 && !err && <div className="vp-hint">Drag to look around · scroll or pinch to zoom</div>}
|
||||
|
||||
<div className="vp-bottom">
|
||||
<span className="vp-time above">{clock(shown)} / {clock(dur)}</span>
|
||||
<div className="vp-seek" style={{ ['--p' as string]: dur ? `${(shown / dur) * 100}%` : '0%', ['--b' as string]: dur ? `${(buffered / dur) * 100}%` : '0%' }}>
|
||||
<input type="range" min={0} max={dur || 1} step={0.1} value={shown} aria-label="Seek"
|
||||
onChange={(e) => { setScrub(Number(e.target.value)); poke() }}
|
||||
onPointerUp={(e) => { if (v) v.currentTime = Number((e.target as HTMLInputElement).value); setScrub(null) }}
|
||||
onKeyUp={(e) => { if (v) v.currentTime = Number((e.target as HTMLInputElement).value); setScrub(null) }} />
|
||||
<input ref={seekBar} type="range" min={0} max={dur || 1} step={0.1} value={shown} aria-label="Seek"
|
||||
onChange={(e) => { setScrub(Number(e.target.value)); poke() }} />
|
||||
</div>
|
||||
<div className="vp-row">
|
||||
<button className="vp-icon" onClick={toggle} aria-label={paused ? 'Play' : 'Pause'}>{paused ? <I d={PLAY} fill /> : <I d={PAUSE} />}</button>
|
||||
<span className="vp-time">{clock(shown)} / {clock(dur)}</span>
|
||||
<button className="vp-icon" onClick={() => seekBy(-5)} aria-label="Back 5 seconds" title="Back 5 seconds"><Skip back /></button>
|
||||
<button className="vp-icon" onClick={() => seekBy(5)} aria-label="On 5 seconds" title="On 5 seconds"><Skip /></button>
|
||||
<span className="vp-vol">
|
||||
<button className="vp-icon" onClick={muteToggle} aria-label={vol.muted || vol.level === 0 ? 'Unmute' : 'Mute'} title={vol.muted || vol.level === 0 ? 'Unmute (m)' : 'Mute (m)'}><Speaker off={vol.muted || vol.level === 0} low={vol.level < 0.5} /></button>
|
||||
{canLevel && <input type="range" min={0} max={1} step={0.05} value={vol.muted ? 0 : vol.level} aria-label="Volume"
|
||||
style={{ ['--v' as string]: `${(vol.muted ? 0 : vol.level) * 100}%` }}
|
||||
onChange={(e) => { const level = Number(e.target.value); setVol({ level, muted: level === 0 }); poke() }} />}
|
||||
</span>
|
||||
<span className="vp-time inrow">{clock(shown)} / {clock(dur)}</span>
|
||||
<span className="spacer" />
|
||||
{vr && canMotion() && (
|
||||
<button className={`vp-icon${motion ? ' on' : ''}`} aria-label="Look around by moving the phone" aria-pressed={motion}
|
||||
|
||||
@@ -298,7 +298,8 @@ function TranscodeDevice() {
|
||||
{err && <div className="error" style={{ marginTop: 8 }}>{err}</div>}
|
||||
<div className="muted small" style={{ marginTop: 10 }}>
|
||||
Opening a video in Files plays it YouTube's way — Auto picks the quality the connection takes, any quality can be picked by hand,
|
||||
or the file as it is — and every quality is made on the fly, from wherever you seek to, on this device. When it is on another
|
||||
or the file as it is — and every quality is made on the fly, from wherever you seek to, on this device. Video previews
|
||||
(thumbnails) are made on it too; image previews stay on each machine's CPU. When it is on another
|
||||
machine than the file, that machine reads the file through this hub. If it can't be reached, the file's own machine makes it instead.
|
||||
</div>
|
||||
</section>
|
||||
|
||||
+3
-6
@@ -15,7 +15,6 @@ import Hardware from './Hardware'
|
||||
import Network from './Network'
|
||||
import VPN from './VPN'
|
||||
import ZFS from './ZFS'
|
||||
import KVM from './KVM'
|
||||
import Files from './Files'
|
||||
import Top from './Top'
|
||||
import Machines from './Machines'
|
||||
@@ -81,7 +80,6 @@ const GROUPS: Group[] = [
|
||||
{ id: 'usage', label: 'Usage', icon: '▧' },
|
||||
{ id: 'zfs', label: 'ZFS', icon: '⛁' },
|
||||
{ id: 'docker', label: 'Docker', icon: DockerIcon },
|
||||
{ id: 'kvm', label: 'KVM', icon: '⌨' },
|
||||
{ id: 'hardware', label: 'Hardware', icon: '⬡' },
|
||||
{ id: 'updates', label: 'Updates', icon: '⇪' },
|
||||
{ id: 'alerts', label: 'Alerts', icon: BellIcon },
|
||||
@@ -91,8 +89,8 @@ const GROUPS: Group[] = [
|
||||
{ id: 'settings', label: 'Settings', icon: '⚙' },
|
||||
]
|
||||
|
||||
type View = 'terminal' | 'widgets' | 'docker' | 'alerts' | 'services' | 'jobs' | 'updates' | 'disks' | 'hardware' | 'network' | 'vpn' | 'zfs' | 'usage' | 'files' | 'procs' | 'machines' | 'kvm' | 'users' | 'settings'
|
||||
const VIEWS: View[] = ['terminal', 'widgets', 'docker', 'alerts', 'services', 'jobs', 'updates', 'disks', 'hardware', 'network', 'vpn', 'zfs', 'usage', 'files', 'procs', 'machines', 'kvm', 'users', 'settings']
|
||||
type View = 'terminal' | 'widgets' | 'docker' | 'alerts' | 'services' | 'jobs' | 'updates' | 'disks' | 'hardware' | 'network' | 'vpn' | 'zfs' | 'usage' | 'files' | 'procs' | 'machines' | 'users' | 'settings'
|
||||
const VIEWS: View[] = ['terminal', 'widgets', 'docker', 'alerts', 'services', 'jobs', 'updates', 'disks', 'hardware', 'network', 'vpn', 'zfs', 'usage', 'files', 'procs', 'machines', 'users', 'settings']
|
||||
const HUB_ONLY: View[] = ['machines', 'users', 'settings', 'alerts']
|
||||
|
||||
// URL hash is the source of truth for where you are: #/<machine|hub>/<view>. Back and
|
||||
@@ -324,7 +322,6 @@ export default function Shell({ me: hubMe, onLoggedOut }: { me: Me; onLoggedOut:
|
||||
<PageBoundary page={GROUPS.find((g) => g.id === view)?.label ?? view}>
|
||||
{view === 'settings' && <Settings me={hubMe} />}
|
||||
{view === 'users' && <Users me={hubMe} />}
|
||||
{view === 'kvm' && <KVM key={machine ?? 'hub'} machine={machine} title={title} />}
|
||||
{view === 'usage' && <Usage key={machine ?? ''} />}
|
||||
{view === 'machines' && <Machines hub={hubMe} onChanged={loadMachines} onOpen={(n) => { switchMachine(n); setView('procs') }} />}
|
||||
{view === 'widgets' && <Gallery />}
|
||||
@@ -366,7 +363,7 @@ export default function Shell({ me: hubMe, onLoggedOut }: { me: Me; onLoggedOut:
|
||||
<span>{machine ? `${title} · ` : ''}{me.version.startsWith('v') ? me.version : 'v' + me.version}</span>
|
||||
<span>up {fmtUptime(me.uptime)}</span>
|
||||
<span className="spacer" />
|
||||
<span className="muted">{view === 'widgets' ? 'Widgets' : view === 'docker' ? 'Docker' : view === 'alerts' ? 'Alerts' : view === 'services' ? 'Services' : view === 'jobs' ? 'Jobs' : view === 'updates' ? 'Updates' : view === 'disks' ? 'Disks' : view === 'hardware' ? 'Hardware' : view === 'network' ? 'Network' : view === 'vpn' ? 'VPN' : view === 'zfs' ? 'ZFS' : view === 'usage' ? 'Usage' : view === 'files' ? 'Files' : view === 'procs' ? 'Processes' : view === 'machines' ? 'Machines' : view === 'kvm' ? 'KVM' : view === 'users' ? 'Users' : view === 'settings' ? 'Settings' : activeTerm ? activeTerm.name : ''}</span>
|
||||
<span className="muted">{view === 'widgets' ? 'Widgets' : view === 'docker' ? 'Docker' : view === 'alerts' ? 'Alerts' : view === 'services' ? 'Services' : view === 'jobs' ? 'Jobs' : view === 'updates' ? 'Updates' : view === 'disks' ? 'Disks' : view === 'hardware' ? 'Hardware' : view === 'network' ? 'Network' : view === 'vpn' ? 'VPN' : view === 'zfs' ? 'ZFS' : view === 'usage' ? 'Usage' : view === 'files' ? 'Files' : view === 'procs' ? 'Processes' : view === 'machines' ? 'Machines' : view === 'users' ? 'Users' : view === 'settings' ? 'Settings' : activeTerm ? activeTerm.name : ''}</span>
|
||||
</footer>
|
||||
</div>
|
||||
)
|
||||
|
||||
+51
-17
@@ -3,7 +3,7 @@
|
||||
// as they climb. The numbers are Ham Lab's own — the sensor stream for everything measured and
|
||||
// /api/procs for the table — so this works on any machine the console can see, whether or not
|
||||
// btop is installed on it.
|
||||
import { createContext, memo, useCallback, useContext, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
|
||||
import { Fragment, createContext, memo, useCallback, useContext, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
|
||||
import { api, GPU, ProcTable, Proc, Sample } from './api'
|
||||
import { Catalog, MONITOR_SHAPE, Values, mergeValues, sampleOf, stale } from './sampleAdapter'
|
||||
import { fetchCatalog, fetchHistories, useSensorStream } from './sensors'
|
||||
@@ -11,6 +11,7 @@ import { fmtMs, nextPoint, REFRESH_OPTIONS, setRefreshMs, strideMs, useHistorySe
|
||||
import { Loading, askYes, toast } from './ui'
|
||||
import { Theme, currentThemeId, themeById } from './themes'
|
||||
import { columnsAt } from './graphcols'
|
||||
import { socketThreads } from './sockets'
|
||||
import { savePrefs } from './accents'
|
||||
|
||||
const MAX = 300 // points kept per curve, as the machines keep them
|
||||
@@ -294,6 +295,13 @@ export default function Top() {
|
||||
// the process table, and the CPU model, come from the same place the Processes page reads,
|
||||
// on the sample's beat (above)
|
||||
const [pickedIf, setPickedIf] = useState<string | null>(null) // which NIC the net box draws
|
||||
// which sockets' threads are folded away on a two-socket machine; remembered in this browser
|
||||
const [folded, setFolded] = useState<number[]>(() => { try { return JSON.parse(localStorage.getItem('top.folded-cpus') ?? '[]') } catch { return [] } })
|
||||
const fold = (pk: number) => setFolded((f) => {
|
||||
const n = f.includes(pk) ? f.filter((x) => x !== pk) : [...f, pk]
|
||||
try { localStorage.setItem('top.folded-cpus', JSON.stringify(n)) } catch { /* private window */ }
|
||||
return n
|
||||
})
|
||||
// Which pool each drive belongs to, asked once: a drive in a ZFS pool has no filesystem of
|
||||
// its own, and "nothing is on it" would be a lie about eight drives holding a NAS.
|
||||
const [pools, setPools] = useState<Record<string, string>>({})
|
||||
@@ -360,7 +368,16 @@ export default function Top() {
|
||||
const model = (pt?.cpu_model ?? '').replace(/\(R\)|\(TM\)|CPU|Processor/g, '').replace(/@.*$/, '').replace(/^\d+th Gen\s*/i, '').replace(/\s+/g, ' ').trim()
|
||||
// the hottest thing that looks like a CPU sensor, and one per core where the chip names them;
|
||||
// a second thread of a core (CPU 8 on an 8-core chip with hyper-threading) wears its core's
|
||||
const coreTemp = (i: number) => { const id = pt?.core_ids?.[i] ?? i; return s?.temps.find((t) => new RegExp(`core ${id}\\b`, 'i').test(t.label))?.c }
|
||||
// On a two-socket board each CPU has its own sensor chip, both counting from "Core 0", so a
|
||||
// thread's core is looked up on its own socket's chip.
|
||||
const pkgs = pt?.core_pkgs?.length === cores.length ? pt.core_pkgs : undefined
|
||||
// each CPU's own threads, counted from 0 under it (sockets.ts): the kernel's one numbering
|
||||
// of the whole machine puts cpu 0–9 and 20–29 on the first and 10–19 and 30–39 on the second
|
||||
const socks = pkgs ? socketThreads(pkgs) : []
|
||||
const sockets = socks.map((x) => x.pkg)
|
||||
const coreTemp = (i: number) => { const id = pt?.core_ids?.[i] ?? i, pk = pkgs?.[i]; return s?.temps.find((t) => new RegExp(`core ${id}\\b`, 'i').test(t.label) && (pk === undefined || (t.pkg ?? 0) === pk))?.c }
|
||||
const sockTemp = (pk: number) => s?.temps.find((t) => t.chip === 'coretemp' && new RegExp(`^package id ${pk}$`, 'i').test(t.label))?.c
|
||||
const sockUse = (pk: number) => { const v = cores.filter((_, i) => pkgs?.[i] === pk); return v.length ? v.reduce((a, b) => a + b, 0) / v.length : 0 }
|
||||
const cpuTemp = pt?.cpu_temp ?? s?.temps.find((t) => /package|tctl|tdie|cpu/i.test(t.label))?.c ?? coreTemp(0)
|
||||
// the temperature columns are there for every row once any row has one, so the % column lines up
|
||||
const anyTemp = cpuTemp !== undefined || cores.some((_, i) => coreTemp(i) !== undefined)
|
||||
@@ -403,26 +420,43 @@ export default function Top() {
|
||||
<Graph values={curves.get('cpu.usage') ?? []} tick={tick} advance={advance} now={now} cap={cap} max={100} grad="cpu" />
|
||||
<div className="t-up">up {upfor(uptime)}</div>
|
||||
</div>
|
||||
<div className="t-cores">
|
||||
<div className="t-cores-head"><b>{model || 'cpu'}</b><span className="spacer" />{pt?.cpu_mhz ? `${Math.round(pt.cpu_mhz)} MHz` : ''}
|
||||
{pt?.cpu_power_w ? <span title="the CPU's own power reading: cores, built-in graphics and memory controller, not the whole machine">{pt.cpu_mhz ? ' · ' : ''}{pt.cpu_power_w < 10 ? pt.cpu_power_w.toFixed(1) : Math.round(pt.cpu_power_w)} W</span> : null}</div>
|
||||
<div className="t-core">
|
||||
<div className={'t-cores' + (sockets.length > 1 ? ' multi' : '')}>
|
||||
<div className="t-cores-head"><b>{sockets.length > 1 ? `${sockets.length} × ` : ''}{model || 'cpu'}</b><span className="spacer" />{pt?.cpu_mhz ? `${Math.round(pt.cpu_mhz)} MHz` : ''}
|
||||
{pt?.cpu_power_w && sockets.length < 2 ? <span title="the CPU's own power reading: cores, built-in graphics and memory controller, not the whole machine">{pt.cpu_mhz ? ' · ' : ''}{pt.cpu_power_w < 10 ? pt.cpu_power_w.toFixed(1) : Math.round(pt.cpu_power_w)} W</span> : null}</div>
|
||||
{sockets.length < 2 && <div className="t-core">
|
||||
<span className="t-core-n">CPU</span>
|
||||
<Meter pct={cpu} grad="cpu" />
|
||||
<span className="t-core-p">{Math.round(cpu)}%</span>
|
||||
{anyTemp && <><span className="t-core-t">{cpuTemp !== undefined && <Meter pct={Math.min(100, (cpuTemp / 100) * 100)} grad="temp" />}</span><span className="t-core-c">{cpuTemp !== undefined ? `${Math.round(cpuTemp)}°C` : ''}</span></>}
|
||||
</div>}
|
||||
{(sockets.length > 1 ? socks : [{ pkg: undefined, cpus: cores.map((_, i) => i) }]).map(({ pkg: pk, cpus }) => (
|
||||
<Fragment key={pk ?? 'all'}>
|
||||
{pk !== undefined && (() => {
|
||||
const u = sockUse(pk), t = sockTemp(pk), w = pt?.cpu_power_pkgs_w?.[pk], open = !folded.includes(pk)
|
||||
return (
|
||||
<div className="t-core t-sock" role="button" aria-expanded={open} title={open ? 'hide its threads' : 'show its threads'} onClick={() => fold(pk)}>
|
||||
<span className="t-core-n"><span className="t-fold">{open ? '▾' : '▸'}</span>CPU{pk + 1}</span>
|
||||
<span className="t-core-w" title="this CPU's own power reading: its cores, cache and memory controller">{w ? `${w < 10 ? w.toFixed(1) : Math.round(w)} W` : ''}</span>
|
||||
<Meter pct={u} grad="cpu" />
|
||||
<span className="t-core-p">{Math.round(u)}%</span>
|
||||
{anyTemp && <><span className="t-core-t">{t !== undefined && <Meter pct={Math.min(100, t)} grad="temp" />}</span><span className="t-core-c">{t !== undefined ? `${Math.round(t)}°C` : ''}</span></>}
|
||||
</div>
|
||||
{cores.map((v, i) => {
|
||||
const t = coreTemp(i)
|
||||
)
|
||||
})()}
|
||||
{(pk !== undefined && folded.includes(pk) ? [] : cpus).map((i, n) => {
|
||||
// n is the thread's number under its own CPU, i the kernel's for the whole machine
|
||||
const t = coreTemp(i), v = cores[i] ?? 0
|
||||
return (
|
||||
<div className="t-core" key={i}>
|
||||
<span className="t-core-n">C{i}</span>
|
||||
<span className="t-core-g" title={pt?.core_ids && pt.core_ids[i] !== i ? `cpu ${i} · a thread of core ${pt.core_ids[i]}` : `core ${i}`}>{braille(recent.get(`cpu.core.${i}.usage`) ?? [], 9)}</span>
|
||||
<span className="t-core-n">C{n}</span>
|
||||
<span className="t-core-g" title={pk !== undefined ? `CPU ${pk + 1} · thread ${n} · cpu ${i} to the kernel${pt?.core_ids ? ` · a thread of core ${pt.core_ids[i]}` : ''}` : pt?.core_ids && pt.core_ids[i] !== i ? `cpu ${i} · a thread of core ${pt.core_ids[i]}` : `core ${i}`}>{braille(recent.get(`cpu.core.${i}.usage`) ?? [], 9)}</span>
|
||||
<span className="t-core-p">{Math.round(v)}%</span>
|
||||
{anyTemp && <><span className="t-core-t">{t !== undefined && <Meter pct={Math.min(100, t)} grad="temp" />}</span><span className="t-core-c">{t !== undefined ? `${Math.round(t)}°C` : ''}</span></>}
|
||||
</div>
|
||||
)
|
||||
})}
|
||||
</Fragment>
|
||||
))}
|
||||
<div className="t-load">Load AVG:<span className="spacer" />{load.map((x) => x.toFixed(2)).join(' ')}</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -456,7 +490,9 @@ export default function Top() {
|
||||
// pool or an md array spans several drives, so nothing is attributed and the row
|
||||
// shows what is true of the hardware — how busy it is, and how hot.
|
||||
const on = fs.filter((f) => f.disk === d.name)
|
||||
const used = on.reduce((a, f) => a + f.used, 0)
|
||||
// a bind mount (nexus's /var/log is /data/var-log) is the same filesystem twice:
|
||||
// count each source device once, or the drive reads double
|
||||
const used = [...new Map(on.map((f) => [f.source || f.mount, f.used])).values()].reduce((a, b) => a + b, 0)
|
||||
return <Disk key={d.name} name={d.name} note={d.model} size={d.size} used={on.length ? used : undefined} io={d.util_pct} temp={d.temp} mounts={on.map((f) => f.mount)} pool={pools[d.name]} />
|
||||
})}
|
||||
{(s?.disks ?? []).length === 0 && <div className="t-dim">no drives reported</div>}
|
||||
@@ -560,13 +596,11 @@ const ENGINES: [string, string][] = [['render', 'render'], ['compute', 'compute'
|
||||
function GPUBox({ g, values, tick, advance, now, cap }: { g: GPU; values: number[]; tick: number; advance: number; now: number; cap: number }) {
|
||||
const engines = ENGINES.filter(([k]) => g.engines && g.engines[k] !== undefined)
|
||||
const temp = g.temp !== undefined && g.temp > 0 ? g.temp : undefined
|
||||
const how = g.stalled ? 'a sysfs read of this card stopped answering; its readings are paused until the console restarts'
|
||||
: g.usage_src === 'idle' ? 'busy, from idle residency' : g.usage_src === 'busy' ? 'busy, as the driver reports it' : g.usage_src === 'clients' ? 'the busiest engine, summed over its clients' : 'this driver exposes no activity counter'
|
||||
return (
|
||||
<div className="t-gpu">
|
||||
<div className="t-gpu-graph">
|
||||
<Graph values={values} tick={tick} advance={advance} now={now} cap={cap} max={100} grad="cpu" />
|
||||
<div className="t-up">{g.usage_src ? `${Math.round(g.usage)}% ` : ''}{how}</div>
|
||||
{g.stalled && <div className="t-up">a read of this card stopped answering; its readings are paused until it does</div>}
|
||||
</div>
|
||||
<div className="t-gpu-stats">
|
||||
<div className="t-cores-head"><b>{g.name}</b><span className="spacer" />{g.driver}{g.integrated ? ' · integrated' : ''}</div>
|
||||
@@ -590,7 +624,7 @@ function GPUBox({ g, values, tick, advance, now, cap }: { g: GPU; values: number
|
||||
<span className="t-fact" title={g.req_mhz ? `${Math.round(g.req_mhz)} MHz requested${g.max_mhz ? `, ${Math.round(g.max_mhz)} MHz max` : ''}` : undefined}>{Math.round(g.mhz ?? 0)}{g.max_mhz ? ` / ${Math.round(g.max_mhz)}` : ''} MHz</span></div> : null}
|
||||
{(g.power_w !== undefined && g.power_w > 0) || g.power_cap_w ? <div className="t-row t-sub"><span>Power</span><span className="spacer" />
|
||||
<span className="t-fact">{(g.power_w ?? 0).toFixed(1)}{g.power_cap_w ? ` / ${Math.round(g.power_cap_w)}` : ''} W</span></div> : null}
|
||||
{g.fan_rpm ? <div className="t-row t-sub"><span>Fan</span><span className="spacer" /><span className="t-fact">{g.fan_rpm} rpm</span></div> : null}
|
||||
{g.fan_rpm || g.fan_pct !== undefined ? <div className="t-row t-sub"><span>Fan</span><span className="spacer" /><span className="t-fact">{[g.fan_pct !== undefined ? `${Math.round(g.fan_pct)}%` : '', g.fan_rpm ? `${g.fan_rpm} rpm` : ''].filter(Boolean).join(' · ')}</span></div> : null}
|
||||
{g.mem_total ? <div className="t-row t-sub"><span>Mem</span><span className="t-pct">{Math.round(((g.mem_used ?? 0) / g.mem_total) * 100)}%</span><Meter pct={((g.mem_used ?? 0) / g.mem_total) * 100} grad="used" /><span className="t-amt">{bytes(g.mem_used ?? 0, 1)}</span></div>
|
||||
: g.mem_used ? <div className="t-row t-sub"><span>Mem</span><span className="spacer" /><span className="t-fact" title="resident memory of the processes that have it open">{bytes(g.mem_used, 1)}</span></div> : null}
|
||||
</div>
|
||||
@@ -611,13 +645,13 @@ function GPUBox({ g, values, tick, advance, now, cap }: { g: GPU; values: number
|
||||
// Nothing on this GPU can be read — no busy counter, engine, temperature, clock, power, memory,
|
||||
// nor a process with it open: the Mac's Asahi, today. Its box would be a graph that can never
|
||||
// draw beside a column of dashes, so it is one line instead. A client opening it brings the box back.
|
||||
const unmeasured = (g: GPU) => !g.usage_src && !g.engines && !(g.temp && g.temp > 0) && !g.mhz && !g.max_mhz && !g.power_w && !g.power_cap_w && !g.fan_rpm && !g.mem_total && !g.mem_used && g.clients.length === 0
|
||||
const unmeasured = (g: GPU) => !g.usage_src && !g.engines && !(g.temp && g.temp > 0) && !g.mhz && !g.max_mhz && !g.power_w && !g.power_cap_w && !g.fan_rpm && g.fan_pct === undefined && !g.mem_total && !g.mem_used && g.clients.length === 0
|
||||
function GPULine({ g }: { g: GPU }) {
|
||||
return (
|
||||
<div className="t-row t-gpu-line">
|
||||
<span>{g.driver}{g.integrated ? ' · integrated' : ''}</span>
|
||||
{g.stalled
|
||||
? <span className="t-dim" title="a sysfs read of this card stopped answering">readings paused until the console restarts</span>
|
||||
? <span className="t-dim" title="a read of this card stopped answering">readings paused until it answers</span>
|
||||
: <span className="t-dim" title="the driver exposes no busy counter, clock, temperature, power or memory reading">nothing measurable</span>}
|
||||
<span className="t-dim t-gpu-who" title={g.clients_ok ? undefined : 'the console has no root here'}>{g.clients_ok ? 'no process has it open' : 'other users’ processes are hidden'}</span>
|
||||
</div>
|
||||
|
||||
+395
-8
@@ -9,8 +9,8 @@
|
||||
// disagree over. An existing cron job is detected and shown, so the page tells the truth
|
||||
// about the machine whether Ham Lab or something else is doing the work.
|
||||
import { FormEvent, Fragment, useCallback, useEffect, useState } from 'react'
|
||||
import { api, ARCView, CacheDev, PoolCaching, ZFSExternal, ZFSInfo, ZFSOverview, ZFSPolicy, ZFSRun, ZFSSnapshot } from './api'
|
||||
import { fmtBytes } from './fmt'
|
||||
import { api, ARCView, CacheDev, Landing, LandingCheck, PoolCaching, PoolWrites, VerifyPolicy, VerifyResult, ZFSExternal, ZFSInfo, ZFSOverview, ZFSPolicy, ZFSRun, ZFSSnapshot, ZFSWrites } from './api'
|
||||
import { fmtBytes, fmtRate } from './fmt'
|
||||
import { useRefreshMs } from './prefs'
|
||||
import { fetchHistories } from './sensors'
|
||||
import Sparkline from './Sparkline'
|
||||
@@ -367,18 +367,149 @@ function PoolTiers({ p, v }: { p: PoolCaching; v: ARCView }) {
|
||||
)
|
||||
}
|
||||
|
||||
// ---- the write path: the dirty buffer in RAM, what each pool writes, and a landing tier ----
|
||||
//
|
||||
// A write reaches a pool in up to three steps. If a landing tier fronts the pool (a mergerfs
|
||||
// pool whose first branch is fast storage and whose last is the pool), new files land there
|
||||
// first and a mover drains them. Either way the data then waits in RAM as dirty buffers, and
|
||||
// each txg sync writes it to the disks. Each step's live number is here, with its history.
|
||||
|
||||
const dur = (s: number) => (!Number.isFinite(s) || s <= 0 ? '—' : s < 90 ? `${Math.round(s)} s` : s < 5400 ? `${Math.round(s / 60)} min` : `${(s / 3600).toFixed(1)} h`)
|
||||
type WHist = Record<string, number[]>
|
||||
const wIds = (w: ZFSWrites) => ['zfs.write.dirty', ...w.pools.flatMap((p) => {
|
||||
const b = `zfs.${p.pool}.`
|
||||
return [b + 'write_bps', b + 'logical_bps', ...(p.landing ? [b + 'landing.pending_bytes', b + 'landing.in_bps', b + 'landing.drain_bps'] : [])]
|
||||
})]
|
||||
|
||||
// rsync's --bwlimit: a bare number is KiB/s, a K/M/G suffix is binary (60M = 60 MiB/s); 0 is none
|
||||
const bwl = (v: string) => {
|
||||
const m = /^(\d+(?:\.\d+)?)([KMG]?)$/i.exec((v ?? '').trim())
|
||||
if (!m || Number(m[1]) === 0) return ''
|
||||
return ` · rsync limited to ${m[1]} ${(m[2] || 'K').toUpperCase()}iB/s`
|
||||
}
|
||||
|
||||
function MoverLine({ l }: { l: Landing }) {
|
||||
const m = l.mover
|
||||
if (!m) return <div className="notice" style={{ marginTop: 6 }}>No mover is reporting for this landing tier (nothing in <span className="mono">/run/landing-mover/</span>). Files that land here stay here until something moves them.</div>
|
||||
const frac = m.current_size > 0 ? m.partial / m.current_size : 0
|
||||
return (
|
||||
<>
|
||||
<div className="props rows" style={{ marginTop: 6 }}>
|
||||
<div><span className="k">Mover</span><span>
|
||||
<span className={'badge ' + (m.state === 'moving' ? 'on' : m.state === 'idle' ? '' : 'warn')}>{m.state}</span>
|
||||
{' '}{m.state === 'moving' && m.batch_files > 1 ? `batch of ${m.batch_files} files, ${fmtBytes(m.batch_bytes)}` : m.state === 'holding' ? 'a mount it needs is missing — waiting' : m.state === 'stopped' ? `no word since ${ago(m.updated_ms)}` : `queue ${m.queue}`}
|
||||
{m.state === 'moving' && m.queue > 0 ? ` · ${m.queue} more queued` : ''}
|
||||
</span></div>
|
||||
{m.state === 'moving' && m.current && <div><span className="k">Now</span><span className="mono small" title={m.current}>{m.current}{m.current_size ? ` · ${fmtBytes(m.partial)} of ${fmtBytes(m.current_size)}` : ''}</span></div>}
|
||||
<div><span className="k">Since start</span><span>{fmtN(m.moved_files)} files · {fmtBytes(m.moved_bytes)} moved · running {ago(m.started_ms).replace(' ago', '')}</span></div>
|
||||
{m.last_move && <div><span className="k">Last moved</span><span className="mono small" title={m.last_move}>{m.last_move} · {ago(m.last_move_ms)}</span></div>}
|
||||
<div><span className="k">Rules</span><span className="muted">a file moves once it has been closed and left alone for {m.settle} s · oldest first{bwl(m.bwlimit)}</span></div>
|
||||
{m.last_error && <div><span className="k">Last error</span><span className="warn-text" style={{ whiteSpace: 'normal' }}>{m.last_error} · {ago(m.last_error_ms)}</span></div>}
|
||||
</div>
|
||||
{m.state === 'moving' && m.current_size > 0 && <Meter frac={frac} cls="" />}
|
||||
</>
|
||||
)
|
||||
}
|
||||
|
||||
function LandingBlockInner({ l, pending, inn, drain }: { l: Landing; pending: number[]; inn: number[]; drain: number[] }) {
|
||||
const eta = l.drain_bps > 0 ? l.pending_bytes / l.drain_bps : NaN
|
||||
const fill = l.size > 0 ? l.used / l.size : 0
|
||||
return (
|
||||
<div className="landing-tier">
|
||||
<div className="arc-tier"><span className="k" title="a mergerfs pool puts new files on this branch first; a mover drains them onto the pool">Landing</span>
|
||||
<span className="arc-tier-v"><span className="mono" title={`${l.branch} → ${l.pool_branch} via ${l.mount}`}>{l.branch}{l.device ? ` · ${l.device}` : ''}{l.fstype ? ` · ${l.fstype}` : ''} · in front of {l.pool_branch} as {l.mount}</span></span>
|
||||
</div>
|
||||
<div className="sub-head" style={{ marginTop: 6 }}>
|
||||
<span className="muted small">{fmtBytes(l.used)} of {fmtBytes(l.size)} used</span><span className="spacer" />
|
||||
<span className="num">{l.pending_files ? `${fmtN(l.pending_files)}${l.pending_partial ? '+' : ''} waiting · ${fmtBytes(l.pending_bytes)}` : 'nothing waiting'}</span>
|
||||
</div>
|
||||
<Meter frac={fill} />
|
||||
<div className="props rows" style={{ marginTop: 6 }}>
|
||||
<div><span className="k">Waiting</span><span>{l.pending_files ? `${fmtN(l.pending_files)}${l.pending_partial ? '+' : ''} files, ${fmtBytes(l.pending_bytes)} · oldest landed ${ago(l.oldest_ms)}` : 'nothing — everything that landed is on the pool'}<span className="muted small"> · counted {ago(l.counted_ms)}</span></span></div>
|
||||
<div><span className="k">Arriving</span><span>{fmtRate(l.in_bps)}{l.device ? <span className="muted small"> · writes to {l.device}</span> : ''}</span></div>
|
||||
<div><span className="k">To the pool</span><span>{fmtRate(l.drain_bps)}{l.pending_bytes > 0 ? ` · ${Number.isFinite(eta) ? `clear in about ${dur(eta)}` : 'not draining right now'}` : ''}</span></div>
|
||||
</div>
|
||||
<MoverLine l={l} />
|
||||
<div className="arc-sparks">
|
||||
<div>
|
||||
<div className="muted small">Waiting on the landing tier · {fmtBytes(l.pending_bytes)}</div>
|
||||
<Sparkline series={[{ name: 'waiting', values: pending, color: cssVar('--s1') }]} format={fmtBytes} height={40} />
|
||||
</div>
|
||||
<div>
|
||||
<div className="muted small">Arriving {fmtRate(l.in_bps)} · to the pool {fmtRate(l.drain_bps)}</div>
|
||||
<Sparkline series={[
|
||||
{ name: 'arriving', values: inn, color: cssVar('--s1') },
|
||||
{ name: 'to the pool', values: drain, color: cssVar('--ok') },
|
||||
]} format={fmtRate} height={40} />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function WritePath({ w, p, h }: { w: ZFSWrites; p: PoolWrites; h: WHist }) {
|
||||
const b = `zfs.${p.pool}.`
|
||||
const throttling = w.delayed_per_sec > 0 || w.over_max_per_sec > 0
|
||||
const delayAt = (w.limit * w.delay_min_pct) / 100
|
||||
const ratio = p.write_bps > 0 && p.logical_bps > 0 ? p.logical_bps / p.write_bps : 0
|
||||
return (
|
||||
<section className="card">
|
||||
<div className="sub-head">
|
||||
<span className={'dot ' + (w.over_max_per_sec > 0 ? 'bad' : throttling ? 'warn' : 'ok')} />
|
||||
<span className="dname">{p.pool}</span>
|
||||
<span className="badge">write path</span>
|
||||
<span className="spacer" />
|
||||
<span className="num">{fmtRate(p.write_bps)} to disk</span>
|
||||
</div>
|
||||
{p.landing
|
||||
? <LandingBlockWithHist l={p.landing} h={h} b={b} />
|
||||
: <div className="arc-tiers"><div className="arc-tier"><span className="k" title="a fast tier new files land on before the pool">Landing</span><span className="arc-tier-v"><span className="none">none — writes go straight into the pool's RAM buffer</span></span></div></div>}
|
||||
<div className="arc-sparks">
|
||||
<div>
|
||||
<div className="muted small">RAM write buffer · {fmtBytes(w.dirty)} of {fmtBytes(w.limit)}</div>
|
||||
<Sparkline series={[{ name: 'waiting in RAM', values: h['zfs.write.dirty'] ?? [], color: cssVar('--s1') }]} max={w.limit || undefined} format={fmtBytes} height={44} />
|
||||
</div>
|
||||
<div>
|
||||
<div className="muted small">Writes · {fmtRate(p.write_bps)} to disk · {fmtRate(p.logical_bps)} from programs</div>
|
||||
<Sparkline series={[
|
||||
{ name: 'to disk', values: h[b + 'write_bps'] ?? [], color: cssVar('--s1') },
|
||||
{ name: 'from programs', values: h[b + 'logical_bps'] ?? [], color: cssVar('--ok') },
|
||||
]} format={fmtRate} height={44} />
|
||||
</div>
|
||||
</div>
|
||||
<div className="props rows" style={{ marginTop: 8 }}>
|
||||
<div><span className="k">In RAM</span><span title="dirty data: written by programs, not yet on the disks. For all pools together — ZFS keeps no live per-pool figure">{fmtBytes(w.dirty)} waiting · holds up to {fmtBytes(w.limit)} {w.limit < w.dirty_max ? `(half the ARC target; zfs_dirty_data_max is ${fmtBytes(w.dirty_max)})` : '(zfs_dirty_data_max)'}</span></div>
|
||||
<div><span className="k">Throttle</span><span title="past zfs_delay_min_dirty_percent of the buffer, ZFS makes each write wait a little so the disks can catch up; at the limit, writes stop until a sync frees room">
|
||||
{w.over_max_per_sec > 0 ? <span className="warn-text">{fmtN(w.over_max_per_sec)}/s writes stopped at the limit</span> : w.delayed_per_sec > 0 ? <span className="warn-text">slowing {fmtN(w.delayed_per_sec)} writes/s — the disks are the limit now</span> : 'not slowing anything'}
|
||||
<span className="muted"> · starts at {w.delay_min_pct}% ({fmtBytes(delayAt)})</span></span></div>
|
||||
<div><span className="k">To disk</span><span>{fmtRate(p.write_bps)} written by syncs{ratio > 0 ? ` · ${fmtRate(p.logical_bps)} from programs (${ratio >= 1 ? `${ratio.toFixed(2)}x compressed` : 'parity and metadata included'})` : ''}<span className="muted"> · a sync at least every {w.txg_timeout_sec} s, sooner as the buffer fills</span></span></div>
|
||||
</div>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
|
||||
function LandingBlockWithHist({ l, h, b }: { l: Landing; h: WHist; b: string }) {
|
||||
return <LandingBlockInner l={l} pending={h[b + 'landing.pending_bytes'] ?? []} inn={h[b + 'landing.in_bps'] ?? []} drain={h[b + 'landing.drain_bps'] ?? []} />
|
||||
}
|
||||
|
||||
function ARCSection() {
|
||||
const [v, setV] = useState<ARCView | null>(null)
|
||||
const [hist, setHist] = useState<{ size: number[]; hit: number[] }>({ size: [], hit: [] })
|
||||
const [whist, setWhist] = useState<WHist>({})
|
||||
const [err, setErr] = useState('')
|
||||
const rate = useRefreshMs()
|
||||
useEffect(() => {
|
||||
let stop = false
|
||||
const poll = async () => {
|
||||
try { const r = await api.zfsARC(); if (!stop) { setV(r); setErr('') } } catch (e) { if (!stop) setErr(e instanceof Error ? e.message : 'failed') }
|
||||
let r: ARCView | null = null
|
||||
try { r = await api.zfsARC(); if (!stop) { setV(r); setErr('') } } catch (e) { if (!stop) setErr(e instanceof Error ? e.message : 'failed') }
|
||||
try {
|
||||
const hs = await fetchHistories('zfs.arc.size,zfs.arc.hit_pct')
|
||||
if (!stop) setHist({ size: hs.find((h) => h.id === 'zfs.arc.size')?.v ?? [], hit: hs.find((h) => h.id === 'zfs.arc.hit_pct')?.v ?? [] })
|
||||
const extra = r?.writes ? wIds(r.writes) : []
|
||||
const hs = await fetchHistories(['zfs.arc.size', 'zfs.arc.hit_pct', ...extra].join(','))
|
||||
if (!stop) {
|
||||
setHist({ size: hs.find((h) => h.id === 'zfs.arc.size')?.v ?? [], hit: hs.find((h) => h.id === 'zfs.arc.hit_pct')?.v ?? [] })
|
||||
setWhist(Object.fromEntries(hs.filter((h) => extra.includes(h.id)).map((h) => [h.id, h.v])))
|
||||
}
|
||||
} catch { /* the sparklines stay empty; the numbers still show */ }
|
||||
}
|
||||
poll()
|
||||
@@ -444,7 +575,262 @@ function ARCSection() {
|
||||
<span><b>All reads</b></span><span className="num"><b>{fmtPct(hitPct(L.Hits, L.Misses))}</b></span><span className="num">{fmtN(L.Hits)}</span><span className="num">{fmtN(L.Misses)}</span>
|
||||
</div>
|
||||
</section>
|
||||
{d.pools.length > 0 && <div className="cards" style={{ marginTop: 8 }}>{d.pools.map((p) => <PoolTiers key={p.pool} p={p} v={v} />)}</div>}
|
||||
{d.pools.length > 0 && <div className="cards full" style={{ marginTop: 8 }}>{d.pools.map((p) => <PoolTiers key={p.pool} p={p} v={v} />)}</div>}
|
||||
{v.writes && v.writes.pools.length > 0 && (
|
||||
<>
|
||||
<div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Writes</h2><span className="spacer" />
|
||||
<span className="muted small">what is waiting to reach each pool, and how fast it is getting there</span>
|
||||
</div>
|
||||
<div className="cards full">{v.writes.pools.map((p) => <WritePath key={p.pool} w={v.writes!} p={p} h={whist} />)}</div>
|
||||
</>
|
||||
)}
|
||||
</>
|
||||
)
|
||||
}
|
||||
|
||||
// ---- landing-tier checks: what has waited too long, and verify passes ----
|
||||
//
|
||||
// A verify pass reads the pool's copy of every file the mover moved back against the SHA-256
|
||||
// the mover took of the landing copy, from its ledger. "New moves" checks what has moved since
|
||||
// the last pass; "everything" re-reads the whole ledger. Passes run as their own process, the
|
||||
// same one a schedule's cron line starts, so a console restart does not cut one short.
|
||||
|
||||
const vDescribe = (p: VerifyPolicy) => {
|
||||
const when = p.every === 'custom' ? `on ${p.cron}`
|
||||
: p.every === 'hourly' ? `every hour at :${(p.at.split(':')[1] ?? '00').padStart(2, '0')}`
|
||||
: p.every === 'daily' ? `every day at ${p.at}`
|
||||
: p.every === 'weekly' ? `every ${WEEKDAYS[p.weekday] ?? 'Sunday'} at ${p.at}`
|
||||
: `on day ${p.day} of the month at ${p.at}`
|
||||
return `${when} · ${p.scope === 'all' ? 'everything in the ledger' : 'what moved since the last pass'}`
|
||||
}
|
||||
const vOK = (r: VerifyResult) => !r.error && r.mismatched === 0 && r.gone === 0 && r.unreadable === 0
|
||||
const secs = (a: string, b: string) => (new Date(b).getTime() - new Date(a).getTime()) / 1000
|
||||
|
||||
function VerifySummary({ r }: { r: VerifyResult }) {
|
||||
if (r.error) return <span className="warn-text" style={{ whiteSpace: 'normal' }}>{r.error}</span>
|
||||
const extra = [r.changed && `${fmtN(r.changed)} edited since (not compared)`, r.waiting && `${fmtN(r.waiting)} waiting to move again`, r.unhashed && `${fmtN(r.unhashed)} without a hash`].filter(Boolean).join(' · ')
|
||||
return (
|
||||
<span>
|
||||
{vOK(r)
|
||||
? <>{r.checked === 0 ? 'nothing new to check' : <><b>{fmtN(r.matched)}</b> of {fmtN(r.checked)} files match</>}</>
|
||||
: <span className="warn-text">{[r.mismatched && `${fmtN(r.mismatched)} DIFFER`, r.gone && `${fmtN(r.gone)} not on the pool`, r.unreadable && `${fmtN(r.unreadable)} unreadable`].filter(Boolean).join(' · ')} · {fmtN(r.matched)} match</span>}
|
||||
<span className="muted"> · {fmtBytes(r.bytes_read)} read in {dur(secs(r.started, r.finished)) === '—' ? '0 s' : dur(secs(r.started, r.finished))} · {r.scope === 'all' ? 'everything' : 'new moves'} · {ago(r.finished)}</span>
|
||||
{extra && <span className="muted"> · {extra}</span>}
|
||||
</span>
|
||||
)
|
||||
}
|
||||
|
||||
function Issues({ r }: { r: VerifyResult }) {
|
||||
const [open, setOpen] = useState(false)
|
||||
if (!r.issues?.length) return null
|
||||
return (
|
||||
<div style={{ marginTop: 6 }}>
|
||||
<button className="btn" onClick={() => setOpen(!open)}>{open ? 'Hide' : 'Show'} the {r.issues.length} file{r.issues.length === 1 ? '' : 's'} to look at</button>
|
||||
{open && <div className="snap-list" style={{ marginTop: 6 }}>{r.issues.map((i) => (
|
||||
<div key={i.path + i.kind} className="snap">
|
||||
<span className={'badge ' + (i.kind === 'mismatch' ? 'bad' : 'warn')}>{i.kind === 'mismatch' ? 'differs' : i.kind === 'gone' ? 'not on pool' : 'unreadable'}</span>
|
||||
<span className="mono snap-name" title={i.path}>{i.path}</span>
|
||||
<span className="muted snap-when">moved {new Date(i.moved).toLocaleString()}</span>
|
||||
<span className="num muted">{fmtBytes(i.size)}</span>
|
||||
</div>
|
||||
))}</div>}
|
||||
{open && r.gone > 0 && <div className="muted small" style={{ marginTop: 4 }}>"Not on the pool" is a file that was moved and is no longer there — deleted or renamed since, most often. The pass cannot tell that from a loss, so it lists them.</div>}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function VerifyScheduleCard({ p, cron, next, last, onChanged, onEdit }: { p: VerifyPolicy; cron: string; next?: string; last: VerifyResult | null; onChanged: () => void; onEdit: () => void }) {
|
||||
const run = async () => {
|
||||
try { await api.runVerifySchedule(p.id); toast('verify pass started'); onChanged() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
|
||||
}
|
||||
const remove = async () => {
|
||||
if (!await askYes(`Remove the verify schedule "${p.name}"? Passes it already ran stay in the history.`, 'Remove')) return
|
||||
try { await api.deleteVerifySchedule(p.id); toast('schedule removed'); onChanged() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
|
||||
}
|
||||
const toggle = async () => {
|
||||
try { await api.saveVerifySchedule({ ...p, enabled: !p.enabled }); onChanged() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
|
||||
}
|
||||
return (
|
||||
<section className="card">
|
||||
<div className="sub-head">
|
||||
<span className={'dot ' + (last && !vOK(last) ? 'bad' : p.enabled ? 'ok' : '')} />
|
||||
<span className="dname">{p.name || 'verify'}</span>
|
||||
{!p.enabled && <span className="badge">paused</span>}
|
||||
<span className="spacer" />
|
||||
<span className="muted small">{p.enabled ? soon(next) : 'not scheduled'}</span>
|
||||
</div>
|
||||
<div className="props rows" style={{ marginTop: 6 }}>
|
||||
<div><span className="k">When</span><span>{vDescribe(p)}</span></div>
|
||||
<div><span className="k">Cron</span><span className="mono small">{cron}</span></div>
|
||||
<div><span className="k">Last pass</span>{last ? <VerifySummary r={last} /> : <span>never run</span>}</div>
|
||||
</div>
|
||||
<div className="svc-actions">
|
||||
<Btn className="btn on" onClick={run}>Run now</Btn>
|
||||
<button className="btn" onClick={onEdit}>Edit</button>
|
||||
<Btn className="btn" onClick={toggle}>{p.enabled ? 'Pause' : 'Resume'}</Btn>
|
||||
<Btn className="btn danger" onClick={remove}>Remove</Btn>
|
||||
</div>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
|
||||
const VBLANK: Partial<VerifyPolicy> = { name: '', every: 'daily', at: '05:10', weekday: 0, day: 1, scope: 'new', enabled: true }
|
||||
|
||||
function VerifyDialog({ p, onClose, onSaved }: { p: Partial<VerifyPolicy>; onClose: () => void; onSaved: () => void }) {
|
||||
const [f, setF] = useState<Partial<VerifyPolicy>>({ ...VBLANK, ...p })
|
||||
const [err, setErr] = useState('')
|
||||
useEscape(onClose)
|
||||
const set = (k: keyof VerifyPolicy, v: unknown) => setF((x) => ({ ...x, [k]: v }))
|
||||
const save = async (e: FormEvent) => {
|
||||
e.preventDefault(); setErr('')
|
||||
try { await api.saveVerifySchedule({ ...f, name: (f.name || '').trim() || (f.scope === 'all' ? 'verify everything' : 'verify new moves') }); onSaved() }
|
||||
catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
|
||||
}
|
||||
return (
|
||||
<div className="modal-back" onClick={onClose}>
|
||||
<form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
|
||||
<h3>{f.id ? 'Edit verify schedule' : 'New verify schedule'}</h3>
|
||||
<label className="muted">Name — what this schedule is for
|
||||
<input value={f.name} onChange={(e) => set('name', e.target.value)} placeholder="nightly verify" maxLength={64} />
|
||||
</label>
|
||||
<label className="muted">What to check
|
||||
<span className="chips">
|
||||
<button type="button" className={'chip' + (f.scope === 'new' ? ' on' : '')} onClick={() => set('scope', 'new')}>new moves</button>
|
||||
<button type="button" className={'chip' + (f.scope === 'all' ? ' on' : '')} onClick={() => set('scope', 'all')}>everything</button>
|
||||
</span>
|
||||
<span className="small">{f.scope === 'all' ? 'Re-reads every file the mover has ever moved from the pool. Thorough, and as slow as the ledger is large.' : 'Reads back what has moved since the last pass — every file checked once, soon after it lands.'}</span>
|
||||
</label>
|
||||
<label className="muted">How often
|
||||
<span className="chips">
|
||||
{(['hourly', 'daily', 'weekly', 'monthly', 'custom'] as const).map((k) => (
|
||||
<button type="button" key={k} className={'chip' + (f.every === k ? ' on' : '')} onClick={() => set('every', k)}>{k}</button>
|
||||
))}
|
||||
</span>
|
||||
</label>
|
||||
{f.every === 'custom' ? (
|
||||
<label className="muted">Cron expression — minute hour day month weekday
|
||||
<input className="mono" value={f.cron ?? ''} onChange={(e) => set('cron', e.target.value)} placeholder="0 */6 * * *" spellCheck={false} required />
|
||||
</label>
|
||||
) : (
|
||||
<label className="muted">At
|
||||
<input type="time" value={f.at} onChange={(e) => set('at', e.target.value)} />
|
||||
</label>
|
||||
)}
|
||||
{f.every === 'weekly' && (
|
||||
<label className="muted">On
|
||||
<select value={f.weekday} onChange={(e) => set('weekday', Number(e.target.value))}>
|
||||
{WEEKDAYS.map((d, i) => <option key={d} value={i}>{d}</option>)}
|
||||
</select>
|
||||
</label>
|
||||
)}
|
||||
{f.every === 'monthly' && (
|
||||
<label className="muted">Day of the month — 1 to 28, so every month has one
|
||||
<input type="number" min={1} max={28} value={f.day} onChange={(e) => set('day', Number(e.target.value))} />
|
||||
</label>
|
||||
)}
|
||||
<label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
|
||||
<input type="checkbox" checked={!!f.enabled} onChange={(e) => set('enabled', e.target.checked)} style={{ width: 'auto' }} />
|
||||
Enabled
|
||||
</label>
|
||||
{err && <div className="error">{err}</div>}
|
||||
<div className="svc-actions">
|
||||
<button type="submit" className="btn on">Save</button>
|
||||
<span className="spacer" />
|
||||
<button type="button" className="btn" onClick={onClose}>Cancel</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function LandingChecks() {
|
||||
const [c, setC] = useState<LandingCheck | null>(null)
|
||||
const [err, setErr] = useState('')
|
||||
const [showStale, setShowStale] = useState(false)
|
||||
const [editing, setEditing] = useState<Partial<VerifyPolicy> | null>(null)
|
||||
const rate = useRefreshMs()
|
||||
const load = useCallback(async () => {
|
||||
try { setC(await api.zfsLanding()); setErr('') } catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
|
||||
}, [])
|
||||
useEffect(() => { load() }, [load])
|
||||
const running = !!c?.running
|
||||
useEffect(() => { const t = setInterval(load, running ? 2000 : Math.max(10000, rate * 5)); return () => clearInterval(t) }, [load, rate, running])
|
||||
|
||||
if (err && !c) return <div className="error">{err}</div>
|
||||
if (!c || (c.tiers.length === 0 && c.policies.length === 0 && c.history.length === 0)) return null
|
||||
const verify = async (scope: 'new' | 'all') => {
|
||||
if (scope === 'all' && !await askYes('Re-read every file the mover has ever moved, from the pool, and check each against its hash? It reads as much as the ledger holds.', 'Verify everything')) return
|
||||
try { await api.landingVerify(scope); toast('verify pass started'); setTimeout(load, 800) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
|
||||
}
|
||||
const last = c.history[0]
|
||||
const hours = Math.round(c.stale_after_sec / 3600)
|
||||
const r = c.running
|
||||
return (
|
||||
<>
|
||||
<div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Landing tier checks</h2><span className="spacer" />
|
||||
<span className="muted small">did everything that landed reach the pool, intact?</span>
|
||||
</div>
|
||||
<div className="cards full">
|
||||
{c.tiers.length === 0 && <div className="notice">No landing mover is reporting right now (nothing in <span className="mono">/run/landing-mover/</span>), so there is nothing to check. Past passes are below.</div>}
|
||||
{c.tiers.map((t) => (
|
||||
<section className="card" key={t.src}>
|
||||
<div className="sub-head">
|
||||
<span className={'dot ' + (last && !vOK(last) ? 'bad' : t.stale_count ? 'warn' : 'ok')} />
|
||||
<span className="dname">{t.dst}</span>
|
||||
<span className="badge">landing checks</span>
|
||||
<span className="spacer" />
|
||||
<span className="muted small mono">{t.src} → {t.dst}</span>
|
||||
</div>
|
||||
<div className="props rows" style={{ marginTop: 6 }}>
|
||||
<div><span className="k">Stale</span><span>
|
||||
{t.stale_count
|
||||
? <><span className="warn-text">{fmtN(t.stale_count)} file{t.stale_count === 1 ? '' : 's'}, {fmtBytes(t.stale_bytes)}, waiting over {hours} h</span>{' '}<button className="btn" onClick={() => setShowStale(!showStale)}>{showStale ? 'Hide' : 'Show'}</button></>
|
||||
: `nothing has waited on the landing tier over ${hours} h`}
|
||||
</span></div>
|
||||
<div><span className="k">Last pass</span>{last ? <VerifySummary r={last} /> : <span>never verified</span>}</div>
|
||||
<div><span className="k">Covered</span><span>{t.cursor ? `every file moved up to ${new Date(t.cursor).toLocaleString()} has been checked at least once` : 'nothing checked yet'}<span className="muted"> · ledger {t.ledger}</span></span></div>
|
||||
</div>
|
||||
{showStale && t.stale.length > 0 && (
|
||||
<div className="snap-list" style={{ marginTop: 6 }}>{t.stale.map((f) => (
|
||||
<div key={f.path} className="snap">
|
||||
<span className="mono snap-name" title={f.path}>{f.path}</span>
|
||||
<span className="muted snap-when">landed {new Date(f.mtime).toLocaleString()}</span>
|
||||
<span className="muted snap-exp">waiting {dur((Date.now() - f.mtime) / 1000)}</span>
|
||||
<span className="num muted">{fmtBytes(f.size)}</span>
|
||||
</div>
|
||||
))}{t.stale_count > t.stale.length && <div className="muted small">and {fmtN(t.stale_count - t.stale.length)} more</div>}</div>
|
||||
)}
|
||||
{r && (
|
||||
<div style={{ marginTop: 8 }}>
|
||||
<div className="sub-head"><span className="muted small">Verifying {r.scope === 'all' ? 'everything' : 'new moves'} · {fmtN(r.done)} of {fmtN(r.total)} files · {fmtBytes(r.bytes_read)} of {fmtBytes(r.bytes_all)}</span><span className="spacer" /><span className="mono small muted" title={r.current}>{r.current}</span></div>
|
||||
<Meter frac={r.bytes_all > 0 ? r.bytes_read / r.bytes_all : r.total > 0 ? r.done / r.total : 0} cls="" />
|
||||
</div>
|
||||
)}
|
||||
{last && <Issues r={last} />}
|
||||
<div className="svc-actions">
|
||||
<Btn className="btn on" disabled={running} onClick={() => verify('new')}>{running ? 'Verifying…' : 'Verify new moves'}</Btn>
|
||||
<Btn className="btn" disabled={running} onClick={() => verify('all')}>Verify everything</Btn>
|
||||
</div>
|
||||
{c.history.length > 1 && (
|
||||
<details style={{ marginTop: 6 }}>
|
||||
<summary className="muted small">Earlier passes</summary>
|
||||
<div className="props rows" style={{ marginTop: 4 }}>{c.history.slice(1).map((h, i) => (
|
||||
<div key={i}><span className="k">{new Date(h.finished).toLocaleString()}</span><VerifySummary r={h} /></div>
|
||||
))}</div>
|
||||
</details>
|
||||
)}
|
||||
</section>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Verify schedules</h2><span className="spacer" />
|
||||
<Btn className="btn on" onClick={() => setEditing({ ...VBLANK })}>+ New schedule</Btn>
|
||||
</div>
|
||||
{c.policies.length === 0 && <div className="notice">No verify runs on a schedule. A daily pass of new moves checks every file once, soon after it lands.</div>}
|
||||
<div className="cards full">{c.policies.map(({ policy, cron, next, last }) => (
|
||||
<VerifyScheduleCard key={policy.id} p={policy} cron={cron} next={next} last={last} onChanged={load} onEdit={() => setEditing(policy)} />
|
||||
))}</div>
|
||||
{editing && <VerifyDialog p={editing} onClose={() => setEditing(null)} onSaved={() => { setEditing(null); load() }} />}
|
||||
</>
|
||||
)
|
||||
}
|
||||
@@ -475,19 +861,20 @@ export default function ZFS() {
|
||||
<div className="dash">
|
||||
<div className="dash-bar"><h2 className="section">Pools</h2></div>
|
||||
{zfs.pools.length === 0 && <div className="notice">ZFS is present but no pools are imported.</div>}
|
||||
<div className="cards">{zfs.pools.map((p) => <PoolCard key={p.name} p={p} onChanged={load} />)}</div>
|
||||
<div className="cards full">{zfs.pools.map((p) => <PoolCard key={p.name} p={p} onChanged={load} />)}</div>
|
||||
|
||||
<div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Cache</h2><span className="spacer" />
|
||||
<span className="muted small">the ARC in RAM, and what each pool has in front of its disks</span>
|
||||
</div>
|
||||
<ARCSection />
|
||||
<LandingChecks />
|
||||
|
||||
<div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Snapshot schedules</h2><span className="spacer" />
|
||||
<Btn className="btn on" onClick={() => setEditing({ ...BLANK, dataset: zfs.pools[0]?.name ?? '' })}>+ New schedule</Btn>
|
||||
</div>
|
||||
{ov.external && <ExternalNotice x={ov.external} mine={mine} onAdopt={() => setEditing(adopt(ov.external!, zfs.pools[0]?.name ?? ''))} />}
|
||||
{ov.policies.length === 0 && !ov.external && <div className="notice">Nothing is taking snapshots on a schedule. Snapshots are written the way zfSnap writes them, with the retention in the name, so anything already here is understood.</div>}
|
||||
<div className="cards">{ov.policies.map(({ policy, cron, next, last }) => (
|
||||
<div className="cards full">{ov.policies.map(({ policy, cron, next, last }) => (
|
||||
<ScheduleCard key={policy.id} p={policy} cron={cron} next={next} last={last ?? {}} onChanged={load} onEdit={() => setEditing(policy)} />
|
||||
))}</div>
|
||||
|
||||
|
||||
+41
-10
@@ -71,11 +71,11 @@ export const titleOf = (name: string | null, machines: MachineStatus[], hubHost:
|
||||
export interface MachineStatus { name: string; url: string; fingerprint?: string; added: string; online: boolean; host?: string; version?: string; load?: string; uptime?: number; reboot_needed?: boolean; last_seen?: string; error?: string }
|
||||
export interface Term { id: string; name: string; running: boolean; exit_code: number; created: string; clients: number }
|
||||
|
||||
export interface Temp { chip: string; label: string; c: number; max?: number; crit?: number }
|
||||
export interface Temp { chip: string; label: string; c: number; max?: number; crit?: number; pkg?: number }
|
||||
export interface Fan { chip: string; label: string; rpm: number }
|
||||
export interface NetIf { iface: string; up: boolean; rx_bps: number; tx_bps: number; rx_total: number; tx_total: number }
|
||||
export interface Disk { name: string; model?: string; size: number; read_bps: number; write_bps: number; util_pct: number; temp?: number }
|
||||
export interface FS { mount: string; type: string; size: number; used: number; remote?: boolean; disk?: string }
|
||||
export interface FS { mount: string; type: string; size: number; used: number; remote?: boolean; disk?: string; source?: string }
|
||||
export interface UPS { driver: string; model: string; state: string; source: string; on_battery: boolean; in_v: number; out_v: number; battery_pct: number; runtime_min: number; load_w: number; load_pct: number; last_event?: string }
|
||||
export interface Proc { pid: number; ppid: number; name: string; cmd?: string; user: string; cpu: number; gpu?: number; cpu_time: number; rss: number; threads: number; state: string; nice: number; started: number }
|
||||
export interface MemInfo { total: number; used: number; avail: number; free?: number; cached?: number; buffers?: number }
|
||||
@@ -86,11 +86,11 @@ export interface GPUClient { pid: number; name: string; usage: number; engines?:
|
||||
export interface GPU {
|
||||
name: string; driver: string; bus?: string; integrated?: boolean
|
||||
usage: number; usage_src?: 'idle' | 'busy' | 'clients'; engines?: Record<string, number>
|
||||
temp?: number; temp_src?: 'gpu' | 'package'; power_w?: number; power_cap_w?: number; fan_rpm?: number
|
||||
temp?: number; temp_src?: 'gpu' | 'package'; power_w?: number; power_cap_w?: number; fan_rpm?: number; fan_pct?: number
|
||||
mhz?: number; req_mhz?: number; max_mhz?: number; mem_used?: number; mem_total?: number
|
||||
clients: GPUClient[]; clients_ok: boolean; stalled?: boolean
|
||||
}
|
||||
export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_power_w?: number; cpu_temp?: number; core_ids?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
|
||||
export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_power_w?: number; cpu_power_pkgs_w?: number[]; cpu_temp?: number; core_ids?: number[]; core_pkgs?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
|
||||
export interface Sample {
|
||||
t: number; interval_ms: number; cpu: number; cores: number[]; load: [number, number, number]
|
||||
mem: { total: number; used: number; avail: number }; swap: { total: number; used: number; avail: number }
|
||||
@@ -185,7 +185,6 @@ export interface SpaceNode { name: string; dir: boolean; size: number; apparent:
|
||||
export interface SpaceTree extends SpaceNode { path: string; children?: SpaceTree[]; more_n?: number; more_size?: number; total: number }
|
||||
export interface SpaceFile { path: string; size: number; apparent: number; mtime: number }
|
||||
|
||||
export interface KVM { id: string; name: string; url: string; machine: string; added: string }
|
||||
|
||||
// ---- the Network tab ----
|
||||
export interface IfAddr { ip: string; prefix: number; v6?: boolean }
|
||||
@@ -252,7 +251,37 @@ export interface ARCDetail {
|
||||
dbuf: { Size: number; Target: number; Hits: number; Misses: number }
|
||||
pools: PoolCaching[]
|
||||
}
|
||||
export interface ARCView { detail: ARCDetail; live: ARCLive | null }
|
||||
export interface LandingMover {
|
||||
state: 'idle' | 'moving' | 'holding' | 'stopped' | string; src: string; dst: string; pid: number; started_ms: number; updated_ms: number; stale: boolean
|
||||
queue: number; settle: number; bwlimit: string; batch_files: number; batch_bytes: number; batch_start_ms: number; batch_first: string
|
||||
current: string; current_size: number; partial: number; moved_files: number; moved_bytes: number
|
||||
last_move: string; last_move_ms: number; last_error: string; last_error_ms: number
|
||||
}
|
||||
export interface Landing {
|
||||
mount: string; branch: string; pool_branch: string; device?: string; fstype?: string; size: number; used: number
|
||||
pending_files: number; pending_bytes: number; pending_partial?: boolean; oldest_ms?: number; counted_ms: number
|
||||
in_bps: number; drain_bps: number; mover?: LandingMover
|
||||
}
|
||||
export interface PoolWrites { pool: string; write_bps: number; logical_bps: number; landing?: Landing }
|
||||
export interface ZFSWrites {
|
||||
dirty: number; dirty_max: number; delay_min_pct: number; limit: number; delayed_per_sec: number; over_max_per_sec: number; txg_timeout_sec: number
|
||||
pools: PoolWrites[]
|
||||
}
|
||||
export interface VerifyPolicy { id: string; name: string; every: 'hourly' | 'daily' | 'weekly' | 'monthly' | 'custom'; at: string; weekday: number; day: number; cron?: string; scope: 'new' | 'all'; enabled: boolean }
|
||||
export interface VerifyIssue { path: string; kind: 'mismatch' | 'gone' | 'unreadable'; moved: number; size: number; note?: string }
|
||||
export interface VerifyResult {
|
||||
id: string; scope: 'new' | 'all'; started: string; finished: string; src: string; dst: string; since: number
|
||||
entries: number; checked: number; matched: number; mismatched: number; changed: number; gone: number; waiting: number; unreadable: number; unhashed: number
|
||||
bytes_read: number; issues: VerifyIssue[]; error?: string; cursor: number
|
||||
}
|
||||
export interface VerifyProgress { id: string; scope: string; pid: number; started: string; updated: string; total: number; done: number; bytes_read: number; bytes_all: number; current: string }
|
||||
export interface StaleFile { path: string; size: number; mtime: number }
|
||||
export interface LandingCheck {
|
||||
tiers: { src: string; dst: string; ledger: string; cursor: number; stale_count: number; stale_bytes: number; stale: StaleFile[] }[]
|
||||
stale_after_sec: number; running: VerifyProgress | null; history: VerifyResult[]; cron_file: string
|
||||
policies: { policy: VerifyPolicy; cron: string; next?: string; last: VerifyResult | null }[]
|
||||
}
|
||||
export interface ARCView { detail: ARCDetail; live: ARCLive | null; writes: ZFSWrites | null }
|
||||
|
||||
export interface FileEntry { name: string; type: 'dir' | 'file' | 'link' | 'other'; size: number; mode: string; perm: string; mtime: number; user: string; group: string; target?: string; broken?: boolean; preview?: 'image' | 'video' | 'audio' }
|
||||
export interface PlayTrack { codec: string; profile?: string; pix_fmt?: string; width?: number; height?: number; fps?: number; channels?: number; spherical?: boolean }
|
||||
@@ -298,6 +327,7 @@ export const api = {
|
||||
playStatus: (node: string, id: string) => req<PlayStatus>('GET', nodeAPI(node, `/api/files/play/${id}`), undefined, true),
|
||||
playStop: (node: string, id: string) => req<void>('DELETE', nodeAPI(node, `/api/files/play/${id}`), undefined, true),
|
||||
transcode: () => req<TranscodeSetting>('GET', '/api/transcode', undefined, true),
|
||||
transcodeSelected: () => req<{ selected: { node: string; device: string } | null; host?: string; name?: string }>('GET', '/api/transcode?only=selected', undefined, true),
|
||||
setTranscode: (node: string, device: string) => req<TranscodeSetting>('PUT', '/api/transcode', device ? { node, device } : {}, true),
|
||||
fileReadURL: (path: string, sudo: boolean) => apiPath(`/api/files/read?${q(path, sudo)}`),
|
||||
fileDownloadURL: (path: string, sudo: boolean) => apiPath(`/api/files/download?${q(path, sudo)}`),
|
||||
@@ -329,12 +359,13 @@ export const api = {
|
||||
storage: () => req<{ devices: StorageDevice[]; zfs: ZFSInfo; smartctl: boolean }>('GET', '/api/storage'),
|
||||
smart: (dev: string, force = false) => req<SmartInfo>('GET', `/api/storage/smart/${dev}${force ? '?force=1' : ''}`),
|
||||
smartTest: (dev: string, kind: 'short' | 'long') => req<void>('POST', `/api/storage/smart/${dev}/test/${kind}`),
|
||||
// Always the hub's: it holds every machine's KVM, so the tab works when the machine does not.
|
||||
kvms: () => req<KVM[]>('GET', '/api/kvms', undefined, true),
|
||||
addKVM: (name: string, url: string, machine: string) => req<KVM>('POST', '/api/kvms', { name, url, machine }, true),
|
||||
removeKVM: (id: string) => req<void>('DELETE', `/api/kvms/${encodeURIComponent(id)}`, undefined, true),
|
||||
zfsOverview: () => req<ZFSOverview>('GET', '/api/zfs'),
|
||||
zfsARC: () => req<ARCView>('GET', '/api/zfs/arc'),
|
||||
zfsLanding: () => req<LandingCheck>('GET', '/api/zfs/landing'),
|
||||
landingVerify: (scope: 'new' | 'all') => req<{ started: string }>('POST', '/api/zfs/landing/verify', { scope }),
|
||||
saveVerifySchedule: (p: Partial<VerifyPolicy>) => req<VerifyPolicy>('PUT', '/api/zfs/landing/schedules', p),
|
||||
deleteVerifySchedule: (id: string) => req<void>('DELETE', `/api/zfs/landing/schedules/${encodeURIComponent(id)}`),
|
||||
runVerifySchedule: (id: string) => req<{ started: string }>('POST', `/api/zfs/landing/schedules/${encodeURIComponent(id)}/run`),
|
||||
saveZfsSchedule: (p: Partial<ZFSPolicy>) => req<ZFSPolicy>('PUT', '/api/zfs/schedules', p),
|
||||
deleteZfsSchedule: (id: string) => req<void>('DELETE', `/api/zfs/schedules/${encodeURIComponent(id)}`),
|
||||
runZfsSchedule: (id: string) => req<{ taken: string; pruned: string[] }>('POST', `/api/zfs/schedules/${encodeURIComponent(id)}/run`),
|
||||
|
||||
@@ -47,11 +47,11 @@ export function sampleOf(v: Values, cat: Catalog): Sample {
|
||||
t: Date.now(), interval_ms: num(v, 'hamlab.sample_interval'),
|
||||
cpu: num(v, 'cpu.usage'), cores, load: [num(v, 'load.1'), num(v, 'load.5'), num(v, 'load.15')],
|
||||
mem: mem('mem'), swap: mem('swap'), uptime: num(v, 'uptime'),
|
||||
temps: hw('temp.').map(({ id, s }) => ({ chip: s.labels!.chip, label: s.labels!.label, c: num(v, id), max: s.warn, crit: s.crit })),
|
||||
temps: hw('temp.').map(({ id, s }) => ({ chip: s.labels!.chip, label: s.labels!.label, c: num(v, id), max: s.warn, crit: s.crit, pkg: s.labels!.package !== undefined ? Number(s.labels!.package) : undefined })),
|
||||
fans: hw('fan.').map(({ id, s }) => ({ chip: s.labels!.chip, label: s.labels!.label, rpm: num(v, id) })),
|
||||
net: [...groups(v, cat, 'net.', 'iface')].map(([iface, p]) => ({ iface, up: v[p + '.up'] === true, rx_bps: num(v, p + '.rx'), tx_bps: num(v, p + '.tx'), rx_total: num(v, p + '.rx_total'), tx_total: num(v, p + '.tx_total') })),
|
||||
disks: [...groups(v, cat, 'disk.', 'device')].map(([name, p]) => ({ name, model: cat.get(p + '.size')?.labels?.model || undefined, size: num(v, p + '.size'), read_bps: num(v, p + '.read'), write_bps: num(v, p + '.write'), util_pct: num(v, p + '.util'), temp: num(v, p + '.temp') || undefined })),
|
||||
fs: [...groups(v, cat, 'fs.', 'mount')].map(([mount, p]) => ({ mount, type: cat.get(p + '.size')?.labels?.fstype ?? '', size: num(v, p + '.size'), used: num(v, p + '.used'), remote: cat.get(p + '.size')?.labels?.remote === '1', disk: cat.get(p + '.size')?.labels?.disk })),
|
||||
fs: [...groups(v, cat, 'fs.', 'mount')].map(([mount, p]) => ({ mount, type: cat.get(p + '.size')?.labels?.fstype ?? '', size: num(v, p + '.size'), used: num(v, p + '.used'), remote: cat.get(p + '.size')?.labels?.remote === '1', disk: cat.get(p + '.size')?.labels?.disk, source: cat.get(p + '.size')?.labels?.source })),
|
||||
ups: v['ups.battery'] === undefined ? null : {
|
||||
driver: cat.get('ups.last_event')?.labels?.driver ?? '', model: cat.get('ups.battery')?.labels?.model ?? '', state: str(v, 'ups.state'), source: str(v, 'ups.source'),
|
||||
on_battery: v['ups.on_battery'] === true, in_v: num(v, 'ups.in_v'), out_v: num(v, 'ups.out_v'), battery_pct: num(v, 'ups.battery'), runtime_min: num(v, 'ups.runtime'),
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
// Which threads belong to which CPU, on a machine with more than one.
|
||||
//
|
||||
// The kernel numbers every thread of the machine in one run, and on a two-socket board with
|
||||
// hyper-threading it deals them out a socket at a time and then goes round again: cpu 0–9 are
|
||||
// CPU 1's cores, 10–19 CPU 2's, 20–29 the second threads of CPU 1's, 30–39 of CPU 2's. Listed
|
||||
// under its CPU by that number, a 20-thread processor reads "C0…C9, C20…C29" and the one
|
||||
// beside it "C10…C19, C30…C39" — neither looks like twenty of anything. So each CPU's threads
|
||||
// are counted from 0 under it, in the kernel's order, the way a one-socket machine's are: the
|
||||
// first half are its cores, the second half their second threads.
|
||||
//
|
||||
// `pkgs[i]` is the socket logical CPU `i` is on. The answer is, for each socket (ascending),
|
||||
// the logical CPUs on it (ascending); a thread's number under its CPU is its place in that list.
|
||||
export function socketThreads(pkgs: number[]): { pkg: number; cpus: number[] }[] {
|
||||
const by = new Map<number, number[]>()
|
||||
pkgs.forEach((pk, i) => { const l = by.get(pk); if (l) l.push(i); else by.set(pk, [i]) })
|
||||
return [...by.entries()].sort((a, b) => a[0] - b[0]).map(([pkg, cpus]) => ({ pkg, cpus }))
|
||||
}
|
||||
+51
-17
@@ -5,6 +5,10 @@
|
||||
--s1: #3987e5; --s2: #c98500; --track: rgba(57, 135, 229, .18);
|
||||
--btn-fg: #111; --term-bg: var(--term-bg); --font: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, "Liberation Mono", monospace;
|
||||
--header-h: 44px; --footer-h: 28px; --rail-w: 220px; --tabs-h: 34px;
|
||||
/* iOS 26+ draws a soft blur under the status bar of a Home Screen app that reaches ~16pt
|
||||
below it (hamlet docs/gotchas.md). Whatever starts at the safe area starts below that band
|
||||
too. Zero where there is no status bar to clear: desktop, a Safari tab. */
|
||||
--blur: min(env(safe-area-inset-top), 16px);
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body, #root { height: 100%; margin: 0; }
|
||||
@@ -30,7 +34,7 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
|
||||
.svc-list > .error, .cron-list > .error { margin: 8px 12px; }
|
||||
|
||||
/* layout */
|
||||
.app { height: 100%; display: flex; flex-direction: column; padding-top: env(safe-area-inset-top); }
|
||||
.app { height: 100%; display: flex; flex-direction: column; padding-top: calc(env(safe-area-inset-top) + var(--blur)); }
|
||||
.header { height: var(--header-h); display: flex; align-items: center; gap: 12px; padding: 0 12px; background: var(--bg2); border-bottom: 1px solid var(--border); }
|
||||
.brand { font-weight: 700; color: var(--accent); letter-spacing: .5px; }
|
||||
.host { color: var(--muted); font-family: ui-monospace, monospace; font-size: 13px; }
|
||||
@@ -95,6 +99,7 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
|
||||
.dash > * { flex-shrink: 0; }
|
||||
.kpis { display: grid; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); gap: 12px; }
|
||||
.cards { display: grid; grid-template-columns: repeat(auto-fill, minmax(320px, 1fr)); gap: 12px; align-items: start; }
|
||||
.cards.full { grid-template-columns: minmax(0, 1fr); } /* one card per row, the width of the page */
|
||||
.tile, .card { background: var(--bg2); border: 1px solid var(--border); border-radius: 8px; padding: 12px; min-width: 0; }
|
||||
.tile .label, .card h2 { margin: 0 0 6px; font-size: 12px; font-weight: 600; color: var(--muted); text-transform: uppercase; letter-spacing: .5px; }
|
||||
.tile .value { font-size: 22px; font-weight: 600; font-variant-numeric: tabular-nums; line-height: 1.2; }
|
||||
@@ -192,6 +197,11 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
|
||||
.props span:not(.k) { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } .mono { font-family: ui-monospace, monospace; font-size: 12px; }
|
||||
/* One property per row, label above its value. For panes like Inspect, whose values are
|
||||
long paths and lists — side by side they were half truncated and read as columns. */
|
||||
/* One property per row, label beside its value, the value wrapping rather than cut off: for
|
||||
sentences, like the ZFS tab's write path, that columns would truncate. */
|
||||
.props.rows { grid-template-columns: minmax(0, 1fr); gap: 3px 0; }
|
||||
.props.rows .k { width: 96px; }
|
||||
.props.rows span:not(.k) { overflow: visible; white-space: normal; overflow-wrap: anywhere; }
|
||||
.props.stacked { grid-template-columns: 1fr; gap: 10px 0; }
|
||||
.props.stacked > div { flex-direction: column; gap: 2px; }
|
||||
.props.stacked .k { width: auto; font-size: 11px; letter-spacing: .04em; text-transform: uppercase; }
|
||||
@@ -264,7 +274,7 @@ button { font: inherit; color: inherit; background: none; border: 0; cursor: poi
|
||||
than what is left, and scrolls inside itself. Its parts keep their height: in a column that
|
||||
may shrink, anything with its own scrollbar (a preview, a list) is squeezed to a sliver
|
||||
instead, and the dialog never scrolls at all. */
|
||||
.modal-back { position: fixed; inset: 0; z-index: 50; background: rgba(0,0,0,.6); display: flex; align-items: center; justify-content: center; padding: max(16px, env(safe-area-inset-top)) max(16px, env(safe-area-inset-right)) max(16px, env(safe-area-inset-bottom)) max(16px, env(safe-area-inset-left)); }
|
||||
.modal-back { position: fixed; inset: 0; z-index: 50; background: rgba(0,0,0,.6); display: flex; align-items: center; justify-content: center; padding: max(16px, calc(env(safe-area-inset-top) + var(--blur))) max(16px, env(safe-area-inset-right)) max(16px, env(safe-area-inset-bottom)) max(16px, env(safe-area-inset-left)); }
|
||||
.modal { --modal-bg: var(--bg2); width: min(420px, 100%); max-height: 100%; overflow-y: auto; overscroll-behavior: contain; background: var(--modal-bg); border: 1px solid var(--border); border-radius: 10px; padding: 16px; display: flex; flex-direction: column; gap: 12px; }
|
||||
.modal > * { flex-shrink: 0; }
|
||||
/* a long dialog's buttons (and what went wrong) stay in reach while the rest scrolls */
|
||||
@@ -368,10 +378,15 @@ h2.section { margin: 0; font-size: 13px; font-weight: 600; color: var(--muted);
|
||||
.preview { flex: 1; display: flex; align-items: center; justify-content: center; overflow: auto; padding: 12px; }
|
||||
.preview img { max-width: 100%; max-height: 100%; object-fit: contain; }
|
||||
.preview video { max-width: 100%; max-height: 100%; background: #000; border-radius: 4px; outline: none; }
|
||||
.preview.pic { position: relative; touch-action: pan-y pinch-zoom; }
|
||||
.step-edge { position: absolute; top: 50%; transform: translateY(-50%); width: 44px; height: 72px; border: 0; border-radius: 6px; background: rgba(0, 0, 0, .35); color: #fff; font-size: 34px; line-height: 1; cursor: pointer; opacity: .55; }
|
||||
.step-edge:hover { opacity: 1; } .step-edge.prev { left: 8px; } .step-edge.next { right: 8px; }
|
||||
.btn.step:disabled { opacity: .4; }
|
||||
/* the video player (Player.tsx): Ham Share's, in the viewer; full screen is the browser's, or
|
||||
on an iPhone the whole window */
|
||||
.vp { position: relative; flex: 1; min-height: 240px; background: #000; color: #fff; overflow: hidden; user-select: none; -webkit-user-select: none; outline: none; }
|
||||
.vp.max { position: fixed; inset: 0; z-index: 1000; }
|
||||
.vp.max .vp-note { top: calc(env(safe-area-inset-top) + var(--blur) + 14px); }
|
||||
.vp-v { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: contain; background: #000; }
|
||||
.vp-hidden { opacity: 0; pointer-events: none; }
|
||||
.vp-sphere { touch-action: none; cursor: grab; }
|
||||
@@ -779,6 +794,11 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
|
||||
.t-cores-head b { color: var(--fg); font-weight: 400; overflow: hidden; text-overflow: ellipsis; }
|
||||
.t-core { display: flex; align-items: center; gap: 6px; }
|
||||
.t-core-n { width: 30px; } .t-core-p { width: 34px; text-align: right; }
|
||||
/* a two-socket board: a row per socket — arrow, name, watts — with its threads indented under it */
|
||||
.t-cores.multi .t-core-n { width: 50px; } .t-cores.multi .t-core:not(.t-sock) .t-core-n { padding-left: 14px; box-sizing: border-box; }
|
||||
.t-sock { color: var(--fg); margin-top: 2px; cursor: pointer; user-select: none; }
|
||||
.t-fold { display: inline-block; width: 12px; color: var(--muted); }
|
||||
.t-core-w { width: 40px; flex-shrink: 0; text-align: right; color: var(--muted); }
|
||||
.t-core-g { flex: 1; color: var(--ok); letter-spacing: 0; overflow: hidden; white-space: nowrap; }
|
||||
.t-core .tm { flex: 1; } .t-core-t { flex: 0 0 62px; display: flex; } .t-core-c { width: 42px; text-align: right; color: var(--t-blue); }
|
||||
.t-load { display: flex; margin-top: 2px; }
|
||||
@@ -879,6 +899,8 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
|
||||
.arc-tier { display: flex; gap: 8px; min-width: 0; } .arc-tier .k { color: var(--muted); width: 80px; flex-shrink: 0; cursor: help; }
|
||||
.arc-tier-v { display: flex; flex-direction: column; gap: 1px; min-width: 0; } .arc-tier-v > span { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.arc-tier-v .none { color: var(--muted); white-space: normal; }
|
||||
.landing-tier { border: 1px solid var(--border); border-radius: 8px; padding: 8px 10px; margin: 4px 0 6px; font-size: 13px; }
|
||||
.landing-tier .props .badge { margin-right: 2px; }
|
||||
|
||||
/* ---- Space: what is using a disk ---- */
|
||||
.space { position: absolute; inset: 0; display: flex; flex-direction: column; background: var(--bg); min-height: 0; }
|
||||
@@ -946,21 +968,33 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
|
||||
.modal select, .modal input[type=time], .modal input[type=number] { padding: 9px 10px; background: var(--bg); border: 1px solid var(--border); border-radius: 6px; color: var(--fg); font-size: 16px; }
|
||||
.modal .chips { margin-top: 2px; }
|
||||
|
||||
/* ---- KVM: a device's own web page, framed ---- */
|
||||
.kvm { position: absolute; inset: 0; display: flex; flex-direction: column; background: var(--bg); min-height: 0; }
|
||||
.kvm-bar { display: flex; align-items: center; gap: 8px; padding: 8px 12px; border-bottom: 1px solid var(--border); background: var(--bg2); flex-wrap: wrap; }
|
||||
.kvm-bar .kvm-url { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; max-width: 28ch; }
|
||||
.kvm-frame { flex: 1; min-height: 0; background: #000; position: relative; }
|
||||
.kvm-frame iframe { width: 100%; height: 100%; border: 0; display: block; background: #000; }
|
||||
.kvm-frame.full { background: #000; }
|
||||
.kvm-hint { padding: 6px 12px; border-top: 1px solid var(--border); }
|
||||
@media (max-width: 800px) { .kvm-bar .kvm-url { display: none; } }
|
||||
.kvm-none { text-align: center; } .kvm-none p { margin: 4px 0; }
|
||||
/* No true fullscreen for a page element (Safari on iOS): fill the viewport in-page instead. */
|
||||
.kvm-frame.expanded { position: fixed; inset: 0; z-index: 60; }
|
||||
.kvm-frame.expanded iframe { height: 100%; }
|
||||
.kvm-shrink { position: absolute; right: max(10px, env(safe-area-inset-right)); bottom: max(10px, env(safe-area-inset-bottom)); z-index: 61; background: var(--bg2); border: 1px solid var(--border); opacity: .75; }
|
||||
.kvm-shrink:hover, .kvm-shrink:active { opacity: 1; }
|
||||
.updot { margin-left: 6px; font-size: 11px; font-weight: 700; }
|
||||
.updot.behind { color: var(--s2); } .updot.unknown { color: var(--muted); }
|
||||
|
||||
/* a phone: the time goes above the seek bar, so the buttons fit in one row */
|
||||
.vp-time.above { display: none; }
|
||||
@media (max-width: 560px) {
|
||||
.vp-time.above { display: block; padding: 0 10px 2px; }
|
||||
.vp-time.inrow { display: none; }
|
||||
}
|
||||
|
||||
|
||||
/* No theme touches playback, whatever it draws. A theme's full-window overlays (the Retro
|
||||
scanlines today, any other tomorrow) are off while a player is open; the picture is never
|
||||
filtered or blended; the player's text is plain, not the theme's lettering. Only the accent
|
||||
colour of its controls follows the theme. */
|
||||
html:has(.vp)::before, html:has(.vp)::after, body:has(.vp)::before, body:has(.vp)::after,
|
||||
.app:has(.vp)::before, .app:has(.vp)::after, #root:has(.vp)::before, #root:has(.vp)::after { display: none !important; }
|
||||
.vp { isolation: isolate; filter: none !important; mix-blend-mode: normal !important; }
|
||||
.vp video, .vp canvas { filter: none !important; mix-blend-mode: normal !important; }
|
||||
.vp { font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; font-weight: 400; letter-spacing: normal; text-shadow: none; }
|
||||
.vp button, .vp input, .vp select { font-family: inherit !important; }
|
||||
|
||||
/* volume: the speaker mutes; the slider beside it (not on iOS, which only mutes) sets the level */
|
||||
.vp-vol { display: inline-flex; align-items: center; }
|
||||
.vp-vol input { width: 72px; height: 24px; margin: 0 6px 0 0; padding: 0; background: transparent; border: 0; -webkit-appearance: none; appearance: none; cursor: pointer; }
|
||||
.vp-vol input::-webkit-slider-runnable-track { height: 4px; border-radius: 2px; background: linear-gradient(to right, #fff var(--v), rgba(255, 255, 255, .3) var(--v)); }
|
||||
.vp-vol input::-moz-range-track { height: 4px; border-radius: 2px; background: linear-gradient(to right, #fff var(--v), rgba(255, 255, 255, .3) var(--v)); }
|
||||
.vp-vol input::-webkit-slider-thumb { -webkit-appearance: none; width: 12px; height: 12px; margin-top: -4px; border-radius: 50%; background: #fff; border: 0; }
|
||||
.vp-vol input::-moz-range-thumb { width: 12px; height: 12px; border-radius: 50%; background: #fff; border: 0; }
|
||||
@media (max-width: 560px) { .vp-vol input { display: none; } }
|
||||
|
||||
Reference in New Issue
Block a user