Sensors: per-service and container sensors, pattern-limited tokens, fleet query, steady sampler

- service.<unit>.active|state for services that are enabled, up or failed
- containers.* and container.<name>.running|state|health|mem|cpu (cgroup reads, no docker stats)
- sensors:read tokens can be limited to ID patterns, enforced on every endpoint and,
  through the hub's filter header, on managed machines; the header can only narrow
- /api/fleet/sensors/query: one shaped query, every machine, one request
- the sampler waits what's left of the interval, so 250 ms means 4 a second
- Monitors asks for the families it draws instead of *
- unpowered hwmon sentinels (-93 °C) are not temperatures

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Mitchell-Johnson
2026-09-19 08:08:44 -07:00
co-authored by Claude Fable 5.1
parent 4ddf10be51
commit 4c2eb57ea6
11 changed files with 383 additions and 51 deletions
+18 -3
View File
@@ -99,6 +99,10 @@ hub talks to the node as itself, the caller's token stops at the hub), and nothi
- Send it as `Authorization: Bearer <token>`, or as `?token=<token>` where headers are
impossible (an iframe, a browser `EventSource`).
- A read-only token can be **limited to ID patterns** (`{"name": "glance", "sensors":
["cpu.*", "zfs.tank.*"]}`, or the second prompt in Settings). Everything outside them
does not exist for that token — catalog, query, history, stream, Prometheus, and on
managed machines through the hub, which passes the restriction along.
- A **full-control token is never accepted from a URL**, and a read-only token is never
accepted anywhere outside this file's endpoints.
@@ -129,6 +133,9 @@ Not everything is a /proc number. These are read in the background on their own
```
services.failed|failed_names|running every 30 s
service.<unit>.active|state one pair per service that is enabled, up or failed
containers.total|running|unhealthy every 15 s (Docker or Podman, if present)
container.<name>.running|state|health|mem|cpu mem and cpu read from the container's cgroup
updates.pending|security|reboot_required every 5 min
zfs.<pool>.health|healthy|size|used|used_pct|frag every 1 min
smart.<dev>.passed|warnings|power_on|wear every 10 min, never waking a sleeping disk
@@ -171,10 +178,18 @@ facts, never live. An `iframe` of the embed is live.
<div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>
```
## Not yet
## The whole fleet in one request
- Per-unit service sensors, containers, token restriction to ID patterns, a fleet-wide
query in one request.
On a hub, `/api/fleet/sensors/query` asks every machine (or `machines`) the same shaped
query and answers `{machine: result}`; this machine answers under its own hostname, and
one that can't be reached answers `{"error": …}` while the rest still do.
```
POST /api/fleet/sensors/query {"machines": "*", "query": {"cpu": "cpu.usage", "up": "uptime"}}
GET /api/fleet/sensors/query?machines=grid,wyse&cpu=cpu.usage&failed=services.failed
→ {"core": {"cpu": 4.8, "up": 50211}, "grid": {"cpu": 0.4, "up": 437177}, "mac": {"error": "timeout"}}
```
## What Ham Lab itself runs on