hamlab/SENSORS.md

# Sensors — the standard read interface

Everything Ham Lab measures is a **sensor**: a stable dotted ID, a little metadata, a
current value. Any consumer — Ham Lab's own dashboard, Glance, Home Assistant, a shell
script — discovers them from the catalog and asks for exactly the ones it wants.

Code: `internal/sensors` (types, registry, selectors, shaped query) and
`internal/metrics/sensors.go` (the collector as a provider). The registry measures
nothing; providers do. A new source of sensors implements `sensors.Provider` and is
added to the registry in `api.New`.

## IDs

```
cpu.usage  cpu.core.<n>.usage  cpu.temp  cpu.power  load.1|5|15  uptime
mem.total|used|avail|free|cached|buffers|used_pct     swap.total|used|avail|used_pct
temp.<chip>.<label>            fan.<chip>.<label>
net.<iface>.rx|tx|rx_total|tx_total|up
net.internet.download|upload|latency|jitter|tested|ip|edge
disk.<dev>.read|write|util|size|temp
fs.<mount>.size|used|used_pct
ups.battery|runtime|load|load_pct|in_v|out_v|on_battery|state|source
gpu.<name>.usage|render|video|video-enhance|copy|compute|temp|power|fan|freq|mem_used|mem_total|clients
zfs.arc.size|target|min|max|mru|mfu|data|metadata|hit_pct|reads|misses|lifetime_hit_pct|no_grow|compress_ratio|l2_size|l2_hit_pct
```

- Segments are lower case `[a-z0-9_-]`: `Package id 0` → `package_id_0`, `/mnt/tank` →
  `mnt_tank`, `/` → `root`. The real names are in the sensor's `labels`.
- Built from names the hardware reports, never enumeration order (`hwmon3` moves between
  boots). Two identical chips collide; the second gets `_2`.
- Which sensors exist differs per machine. That is the point of the catalog.
- `net.internet.*` is the Network tab's speed test: `download` and `upload` in Mbps,
  `latency` and `jitter` in ms, `tested` the age of the reading in seconds, and `ip` and
  `edge` the public address and the Cloudflare edge as strings. They appear only once a test
  has run on that machine, and each one's `t` is when it ran — so a widget or an alert on
  them is reading a measurement from that moment, not a live value. Nothing takes a speed
  test on its own; something has to press the button.
- `cpu.power` is the CPU packages' own power in watts, from the kernel's RAPL energy
  counters (Intel, and AMD Zen), summed over every package: cores, built-in graphics and
  memory controller, not the whole machine. The counters are root-only, so a daemon that is
  not root reads them through a helper started once with sudo or doas. It is absent on a
  machine without the counters (ARM, the Mac mini) or without a way to root.
- `gpu.<name>.*` is each graphics device, named as lspci names it (`gpu.intel_arc_a380`;
  labels carry `driver` and the PCI `bus`). `usage` is how busy the device was — from the
  GT's idle residency on Intel, `gpu_busy_percent` on amdgpu — and is absent on a driver
  with no such counter. The per-engine percentages (`render`, `video`…) are the sum over the
  processes holding the device open, read from DRM fdinfo, and exist only while something
  has it open. `temp` is the card's own sensor, or on an integrated GPU the CPU package it
  shares a die with (label `source` says which); an Arc on i915 has none and reports none.
  `freq` is the actual clock, 0 when power-gated. `clients` counts the processes using it.
- `zfs.arc.*` is the ZFS read cache in RAM, on a machine with the module loaded. `hit_pct`
  is over the last sampling interval and appears once an interval has had any reads (an idle
  interval repeats the last value); `lifetime_hit_pct` is since boot. `no_grow` is true
  while memory pressure stops the cache growing. `l2_*` appear only with cache devices.

## A sensor

```json
{ "id": "temp.coretemp.package_id_0", "name": "coretemp Package id 0",
  "kind": "temperature", "unit": "°C", "value": 51, "t": 1789826600878,
  "warn": 105, "crit": 105, "labels": { "chip": "coretemp", "label": "Package id 0" } }
```

`value` is a number, bool or string. `t` is when it was really read, in unix ms — slow
sections (filesystems 30 s, UPS 10 s) are older than the tick. `min`/`max` are the scale
(0–100 for percentages, 0–total for used bytes); `warn`/`crit` are the hardware's own
thresholds where it states them.

## Endpoints

All behind the same auth as the rest of `/api` (session cookie or `Authorization: Bearer`).
On a hub, every one is also at `/api/nodes/<machine>/sensors…` for each managed machine.

| | |
|---|---|
| `GET /api/sensors` | Catalog, with current values. `?ids=fan.*,cpu.usage` narrows it. |
| `GET /api/sensors/{id}` | One sensor. `?format=text` → the bare value as text/plain. |
| `GET /api/sensors/{id}/history` | `{id, unit, t[], v[]}` — what the ring holds (300 points, spaced by the window below). Numbers only. |
| `GET /api/sensors/history?ids=cpu.usage,net.*.rx` | Many histories in one request: `{series: [{id, unit, t[], v[]}]}`. A sensor younger than the ring has a shorter series. |
| `GET /api/prefs/history` | How far back the history reaches: `{seconds, points, stride_sec, min_seconds, max_seconds}`. |
| `PUT /api/prefs/history` | `{seconds}` — 60 to 86400. Set on the hub it is passed to every machine, so the fleet's graphs reach equally far back. |
| `POST /api/sensors/query` | Shaped query, below. |
| `GET /api/sensors/query?k=selector&…` | The same, flat, for consumers that can only fetch a URL. |

A selector is an exact ID or a glob. `*` crosses dots: `fan.*` is every fan,
`cpu.core.*.usage` every core.

## Shaped query

Send the object you want, with selectors where the values go. Get it back filled in.

```
POST /api/sensors/query
{ "cpu": "cpu.usage", "fans": "fan.*", "box": { "temp": "cpu.temp", "up": "uptime" } }

→ { "cpu": 12.4,
    "fans": { "fan.nct6775.fan1": 820, "fan.nct6775.fan2": 1140 },
    "box": { "temp": 41, "up": 80520 } }
```

- Exact ID → its value; `null` if this machine has no such sensor (so one query works
  across machines that differ).
- Glob → an object keyed by ID; `{}` if nothing matches.
- Objects and arrays are walked; anything else in a query is a 400.
- `?full=1` → every value is the whole sensor object instead of the bare value. (In the
  GET form `full` is therefore not usable as a key.)

## Live stream

`GET /api/sensors/stream` is a shaped query that keeps answering: the same object, pushed
as a server-sent event after every sample. The query is `?q=<URL-encoded JSON shape>` or
the flat form; `?full=1` as above; `?every=<ms>` spaces events out for a consumer that
doesn't want them at the sampling rate. A keepalive comment goes out every 25 s, and
`X-Accel-Buffering: no` keeps nginx from holding events back.

```js
new EventSource('/api/sensors/stream?token=…&q=' + encodeURIComponent(JSON.stringify({ cpu: 'cpu.usage', fans: 'fan.*' })))
  .onmessage = (e) => render(JSON.parse(e.data))
```

## Auth: read-only tokens

Settings → Access tokens → **+ Read-only sensor token** (or `POST /api/tokens/read
{"name"}` with a session). Scope `sensors:read`: it opens every endpoint in this file,
here and on each managed machine through the hub (`/api/nodes/<machine>/sensors…`; the
hub talks to the node as itself, the caller's token stops at the hub), and nothing else.

- Send it as `Authorization: Bearer <token>`, or as `?token=<token>` where headers are
  impossible (an iframe, a browser `EventSource`).
- A read-only token can be **limited to ID patterns** (`{"name": "glance", "sensors":
  ["cpu.*", "zfs.tank.*"]}`, or the second prompt in Settings). Everything outside them
  does not exist for that token — catalog, query, history, stream, Prometheus, and on
  managed machines through the hub, which passes the restriction along.
- A **full-control token is never accepted from a URL**, and a read-only token is never
  accepted anywhere outside this file's endpoints.

## Prometheus

`GET /metrics` (or `/api/sensors/prometheus`, which also works through the hub as
`/api/nodes/<machine>/sensors/prometheus`). One gauge family per kind of measurement,
base units, the sensor ID and its real-world names as labels:

```
hamlab_temperature_celsius{id="temp.coretemp.core_0",chip="coretemp",label="Core 0"} 50
hamlab_usage_percent{id="fs.root.used_pct",fstype="ext4",mount="/"} 10.1
hamlab_state{id="ups.state",model="CPS CP1500PFCRM2U",value="Normal"} 1
hamlab_updates_pending{id="updates.pending",manager="apt"} 3
```

```yaml
scrape_configs:
  - job_name: hamlab
    authorization: { credentials: <read-only token> }
    static_configs: [{ targets: ['10.0.0.234:8100'] }]
```

## Status sensors

Not everything is a /proc number. These are read in the background on their own clocks
(each sensor's `t` is its real age) by `internal/api/status_sensors.go`:

```
services.failed|failed_names|running            every 30 s
service.<unit>.active|state                     one pair per service that is enabled, up or failed
containers.total|running|unhealthy              every 15 s (Docker or Podman, if present)
container.<name>.running|state|health|mem|cpu   mem and cpu read from the container's cgroup
container.<name>.down                           should be up and isn't: Docker couldn't start it, it is
                                                stuck restarting, or it died non-zero with no policy to
                                                restart it — *not* a container someone stopped
updates.pending|security|reboot_required        every 5 min
zfs.<pool>.health|healthy|size|used|used_pct|frag   every 1 min
zfs.write.dirty|limit|delayed                       the RAM write buffer (all pools) and writes the throttle is slowing
zfs.<pool>.write_bps|logical_bps                    bytes/s synced to disk, and written by programs
zfs.<pool>.landing.pending_bytes|pending_files|used_pct|in_bps|drain_bps|mover|queue   a landing tier, where one fronts the pool
smart.<dev>.passed|warnings|power_on|wear       every 10 min, never waking a sleeping disk
```

## Widgets

A widget is a spec — `{type, sensor, title?}` — over this API and nothing else
(`web/src/widgets.tsx`, fed by `useSensorStream` in `web/src/sensors.ts`). Types: `stat`,
`gauge`, `bar`, `spark`, and `list` for a pattern. Scale and colour come from the
sensor's own `min`/`max`/`warn`/`crit`.

- **In Ham Lab:** Dashboard → Customize → **+ Add widget**. Saved with the layout,
  per machine; all of a page's widgets share one stream.
- **Widgets tab:** every widget type, then every sensor on the machine drawn once, live,
  grouped by family (`web/src/Gallery.tsx`). Tap a tile to copy its sensor ID — the
  quickest way to find what to put in a widget, a query or a Glance tile.
- **Anywhere else:** `/embed/widget` is one widget alone on a page, live, in its own small
  bundle (the console never loads), and the only path Ham Lab allows to be framed.

```
/embed/widget?type=gauge&sensor=cpu.usage&token=<read-only token>
   &machine=grid   &title=…   &theme=dark|light|cyberpunk|retro   &every=<ms>   &bare=1
```

### Glance

Glance's own `custom-api` widget refetches on page load / cache expiry — fine for slow
facts, never live. An `iframe` of the embed is live.

```yaml
- type: iframe
  title: core CPU
  source: https://lab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
  height: 130

- type: custom-api
  title: grid
  cache: 1m
  url: https://lab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
  headers: { Authorization: Bearer <read-only token> }
  template: |
    <div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>
```

## The whole fleet in one request

On a hub, `/api/fleet/sensors/query` asks every machine (or `machines`) the same shaped
query and answers `{machine: result}`; this machine answers under its own hostname, and
one that can't be reached answers `{"error": …}` while the rest still do.

```
POST /api/fleet/sensors/query   {"machines": "*", "query": {"cpu": "cpu.usage", "up": "uptime"}}
GET  /api/fleet/sensors/query?machines=grid,wyse&cpu=cpu.usage&failed=services.failed

→ {"core": {"cpu": 4.8, "up": 50211}, "grid": {"cpu": 0.4, "up": 437177}, "mac": {"error": "timeout"}}
```

## What counts as a disk

`disk.*` names **drives**, not the block devices the kernel lists. An NVMe controller can
present several namespaces — an Apple SSD shows `nvme0n1`, `nvme0n2` and `nvme0n3`, the last
two a few MiB of firmware — and each one arrives from `/proc/diskstats` looking like a whole
disk. They are folded into the controller (`disk.nvme0.*`), with the rates added and busy
taken as the longest of them rather than the sum, since they share one controller. This is
what the Disks page has always shown; since 2026-09-22 the dashboard, the Top tab and these
sensors agree with it.

## The Top tab

Ham Lab → **Top** is a whole machine on one screen in the style of btop: black ground, boxes
with their name set into the border, curves drawn as dots that climb green → yellow → red.
It reads nothing of its own — the sensor stream for everything measured, `/api/procs` for the
table — so it works on any machine the console can reach, whether or not btop is installed
there, and it follows the same refresh rate and history window as the rest of the console.
Its colours are the active skin's: the terminal background for the ground, the skin's sixteen
terminal colours for the curves and meters. (The look is btop's, `web/src/Top.tsx` is ours;
btop itself is Apache-2.0.)

## How often a machine samples

The console's refresh rate (Settings, or `− rate +` on Top) is also the fleet's sampling rate.
It is saved with the UI prefs on the hub, which sets its own sampler and passes the rate to
every machine it manages — `PUT /api/prefs/sampler {"ms": …}`, 100 ms to 60 s, sampled no
slower than the collector's 10 s — and repeats it at start-up. `GET` answers `{ms, min_ms,
max_ms}`. Before 2026-09-21 only an open Dashboard page told a machine its rate.

## How far back a history reaches

Every machine keeps **300 points per curve**, whatever the window — so the window sets how
far apart they are, not how much memory monitoring costs. At the default ten minutes each
point is one sample (2 s apart at the default rate); at a day each is the **average of the
five minutes behind it**, not whichever instant landed on the mark, so a long graph shows
the load that really happened. Ham Lab → Settings → **Graph history** is the one control,
1 minute to 24 hours; the hub passes it to every machine it manages.

The window can't be finer than the sampling interval — 300 points of a 1-minute window at a
2-second rate is 30 points, and that is what you get. It is also all in memory: a machine
that restarts starts its graphs again.

## What Ham Lab itself runs on

Since 2026-09-19 the Dashboard tab (stock modules and custom widgets) and the Disks tab's
live rates read only this API: the catalog once, one batch of histories, one stream —
closed while the page is hidden, refilled when it returns. `web/src/sampleAdapter.ts` is
the seam that hands the stock modules the object they were written against. The old
`/api/metrics` and `/api/metrics/history` are gone. `hamlab.sample_interval` is the
sampler's own rate, as a sensor.

## From another origin

The sensor endpoints (and only they) answer any origin: `Access-Control-Allow-Origin: *`,
preflight included. `*` is the one value browsers won't combine with cookies, so a foreign
page can reach the API with a token and can never ride a logged-in console's session.
`examples/external-demo/` is a complete outside consumer in one HTML file.