Mitchell-JohnsonandClaude Opus 5
2f59a54678
Per machine like every other tab: the interfaces as the kernel has them
(addresses, MAC, MTU, negotiated link speed, carrier, totals, live rates from
the sampler's own sensors), the default route and resolvers, what the internet
sees of this connection, and a speed test with the last fifty results kept.
The test runs in the daemon, not the browser, so the figure is the line of the
machine in the switcher — the router's, if you switch to gate — and it carries
on when the phone locks. It measures against Cloudflare's public speed
endpoints, so there is nothing to install on any host, which is what makes it
work the same on Debian and on OpenWrt where flash is scarce.
What the measurement does and does not count:
- Several streams, HTTP/2 deliberately off. One stream over one connection
measures a congestion window, and HTTP/2 would put every stream back onto
one connection and measure the same thing.
- The first second of each phase is thrown away as the ramp.
- A phase ends on 64 MiB as well as on four seconds, so a fast line says what
it is without moving a gigabyte. A minimum window keeps a very fast line
from filling the cap before there is anything to divide by.
- Latency is the round trip less the far end's own Server-Timing. A zero-byte
response is mostly Cloudflare's worker thinking: left in, it reported 31 ms
for a line the edge itself measures at 3.5. Jitter is the mean change
between consecutive probes, so one slow probe cannot run away with it.
- The far end rate-limits by volume. So: a 20 s cooldown counted from the last
attempt rather than the last success, streams that retry twice, the rate
computed over the window in which bytes were actually still arriving, and a
plain sentence when it was refused. Coasting to the end of the clock with
nothing arriving reported a third of the truth.
The last result is published as net.internet.* sensors, each stamped with when
the test ran, so the line can be a widget, a graph, an alert or a Prometheus
series like anything else. The public address is refreshed on its own quarter
hour, which is worth having here: it is a DHCP lease and it moves.
Tests are offline — the window, the byte cap, the refusal and the cancellation
run against a local server; the route parser's byte order is pinned by a case
that would have caught the bug it did. The live measurement is behind
HAMLAB_NET_TEST=1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
266 lines
13 KiB
Markdown
266 lines
13 KiB
Markdown
# Sensors — the standard read interface
|
||
|
||
Everything Ham Lab measures is a **sensor**: a stable dotted ID, a little metadata, a
|
||
current value. Any consumer — Ham Lab's own dashboard, Glance, Home Assistant, a shell
|
||
script — discovers them from the catalog and asks for exactly the ones it wants.
|
||
|
||
Code: `internal/sensors` (types, registry, selectors, shaped query) and
|
||
`internal/metrics/sensors.go` (the collector as a provider). The registry measures
|
||
nothing; providers do. A new source of sensors implements `sensors.Provider` and is
|
||
added to the registry in `api.New`.
|
||
|
||
## IDs
|
||
|
||
```
|
||
cpu.usage cpu.core.<n>.usage cpu.temp load.1|5|15 uptime
|
||
mem.total|used|avail|free|cached|buffers|used_pct swap.total|used|avail|used_pct
|
||
temp.<chip>.<label> fan.<chip>.<label>
|
||
net.<iface>.rx|tx|rx_total|tx_total|up
|
||
net.internet.download|upload|latency|jitter|tested|ip|edge
|
||
disk.<dev>.read|write|util|size|temp
|
||
fs.<mount>.size|used|used_pct
|
||
ups.battery|runtime|load|load_pct|in_v|out_v|on_battery|state|source
|
||
```
|
||
|
||
- Segments are lower case `[a-z0-9_-]`: `Package id 0` → `package_id_0`, `/mnt/tank` →
|
||
`mnt_tank`, `/` → `root`. The real names are in the sensor's `labels`.
|
||
- Built from names the hardware reports, never enumeration order (`hwmon3` moves between
|
||
boots). Two identical chips collide; the second gets `_2`.
|
||
- Which sensors exist differs per machine. That is the point of the catalog.
|
||
- `net.internet.*` is the Network tab's speed test: `download` and `upload` in Mbps,
|
||
`latency` and `jitter` in ms, `tested` the age of the reading in seconds, and `ip` and
|
||
`edge` the public address and the Cloudflare edge as strings. They appear only once a test
|
||
has run on that machine, and each one's `t` is when it ran — so a widget or an alert on
|
||
them is reading a measurement from that moment, not a live value. Nothing takes a speed
|
||
test on its own; something has to press the button.
|
||
|
||
## A sensor
|
||
|
||
```json
|
||
{ "id": "temp.coretemp.package_id_0", "name": "coretemp Package id 0",
|
||
"kind": "temperature", "unit": "°C", "value": 51, "t": 1789826600878,
|
||
"warn": 105, "crit": 105, "labels": { "chip": "coretemp", "label": "Package id 0" } }
|
||
```
|
||
|
||
`value` is a number, bool or string. `t` is when it was really read, in unix ms — slow
|
||
sections (filesystems 30 s, UPS 10 s) are older than the tick. `min`/`max` are the scale
|
||
(0–100 for percentages, 0–total for used bytes); `warn`/`crit` are the hardware's own
|
||
thresholds where it states them.
|
||
|
||
## Endpoints
|
||
|
||
All behind the same auth as the rest of `/api` (session cookie or `Authorization: Bearer`).
|
||
On a hub, every one is also at `/api/nodes/<machine>/sensors…` for each managed machine.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `GET /api/sensors` | Catalog, with current values. `?ids=fan.*,cpu.usage` narrows it. |
|
||
| `GET /api/sensors/{id}` | One sensor. `?format=text` → the bare value as text/plain. |
|
||
| `GET /api/sensors/{id}/history` | `{id, unit, t[], v[]}` — what the ring holds (300 points, spaced by the window below). Numbers only. |
|
||
| `GET /api/sensors/history?ids=cpu.usage,net.*.rx` | Many histories in one request: `{series: [{id, unit, t[], v[]}]}`. A sensor younger than the ring has a shorter series. |
|
||
| `GET /api/prefs/history` | How far back the history reaches: `{seconds, points, stride_sec, min_seconds, max_seconds}`. |
|
||
| `PUT /api/prefs/history` | `{seconds}` — 60 to 86400. Set on the hub it is passed to every machine, so the fleet's graphs reach equally far back. |
|
||
| `POST /api/sensors/query` | Shaped query, below. |
|
||
| `GET /api/sensors/query?k=selector&…` | The same, flat, for consumers that can only fetch a URL. |
|
||
|
||
A selector is an exact ID or a glob. `*` crosses dots: `fan.*` is every fan,
|
||
`cpu.core.*.usage` every core.
|
||
|
||
## Shaped query
|
||
|
||
Send the object you want, with selectors where the values go. Get it back filled in.
|
||
|
||
```
|
||
POST /api/sensors/query
|
||
{ "cpu": "cpu.usage", "fans": "fan.*", "box": { "temp": "cpu.temp", "up": "uptime" } }
|
||
|
||
→ { "cpu": 12.4,
|
||
"fans": { "fan.nct6775.fan1": 820, "fan.nct6775.fan2": 1140 },
|
||
"box": { "temp": 41, "up": 80520 } }
|
||
```
|
||
|
||
- Exact ID → its value; `null` if this machine has no such sensor (so one query works
|
||
across machines that differ).
|
||
- Glob → an object keyed by ID; `{}` if nothing matches.
|
||
- Objects and arrays are walked; anything else in a query is a 400.
|
||
- `?full=1` → every value is the whole sensor object instead of the bare value. (In the
|
||
GET form `full` is therefore not usable as a key.)
|
||
|
||
## Live stream
|
||
|
||
`GET /api/sensors/stream` is a shaped query that keeps answering: the same object, pushed
|
||
as a server-sent event after every sample. The query is `?q=<URL-encoded JSON shape>` or
|
||
the flat form; `?full=1` as above; `?every=<ms>` spaces events out for a consumer that
|
||
doesn't want them at the sampling rate. A keepalive comment goes out every 25 s, and
|
||
`X-Accel-Buffering: no` keeps nginx from holding events back.
|
||
|
||
```js
|
||
new EventSource('/api/sensors/stream?token=…&q=' + encodeURIComponent(JSON.stringify({ cpu: 'cpu.usage', fans: 'fan.*' })))
|
||
.onmessage = (e) => render(JSON.parse(e.data))
|
||
```
|
||
|
||
## Auth: read-only tokens
|
||
|
||
Settings → Access tokens → **+ Read-only sensor token** (or `POST /api/tokens/read
|
||
{"name"}` with a session). Scope `sensors:read`: it opens every endpoint in this file,
|
||
here and on each managed machine through the hub (`/api/nodes/<machine>/sensors…`; the
|
||
hub talks to the node as itself, the caller's token stops at the hub), and nothing else.
|
||
|
||
- Send it as `Authorization: Bearer <token>`, or as `?token=<token>` where headers are
|
||
impossible (an iframe, a browser `EventSource`).
|
||
- A read-only token can be **limited to ID patterns** (`{"name": "glance", "sensors":
|
||
["cpu.*", "zfs.tank.*"]}`, or the second prompt in Settings). Everything outside them
|
||
does not exist for that token — catalog, query, history, stream, Prometheus, and on
|
||
managed machines through the hub, which passes the restriction along.
|
||
- A **full-control token is never accepted from a URL**, and a read-only token is never
|
||
accepted anywhere outside this file's endpoints.
|
||
|
||
## Prometheus
|
||
|
||
`GET /metrics` (or `/api/sensors/prometheus`, which also works through the hub as
|
||
`/api/nodes/<machine>/sensors/prometheus`). One gauge family per kind of measurement,
|
||
base units, the sensor ID and its real-world names as labels:
|
||
|
||
```
|
||
hamlab_temperature_celsius{id="temp.coretemp.core_0",chip="coretemp",label="Core 0"} 50
|
||
hamlab_usage_percent{id="fs.root.used_pct",fstype="ext4",mount="/"} 10.1
|
||
hamlab_state{id="ups.state",model="CPS CP1500PFCRM2U",value="Normal"} 1
|
||
hamlab_updates_pending{id="updates.pending",manager="apt"} 3
|
||
```
|
||
|
||
```yaml
|
||
scrape_configs:
|
||
- job_name: hamlab
|
||
authorization: { credentials: <read-only token> }
|
||
static_configs: [{ targets: ['10.0.0.234:8100'] }]
|
||
```
|
||
|
||
## Status sensors
|
||
|
||
Not everything is a /proc number. These are read in the background on their own clocks
|
||
(each sensor's `t` is its real age) by `internal/api/status_sensors.go`:
|
||
|
||
```
|
||
services.failed|failed_names|running every 30 s
|
||
service.<unit>.active|state one pair per service that is enabled, up or failed
|
||
containers.total|running|unhealthy every 15 s (Docker or Podman, if present)
|
||
container.<name>.running|state|health|mem|cpu mem and cpu read from the container's cgroup
|
||
container.<name>.down should be up and isn't: Docker couldn't start it, it is
|
||
stuck restarting, or it died non-zero with no policy to
|
||
restart it — *not* a container someone stopped
|
||
updates.pending|security|reboot_required every 5 min
|
||
zfs.<pool>.health|healthy|size|used|used_pct|frag every 1 min
|
||
smart.<dev>.passed|warnings|power_on|wear every 10 min, never waking a sleeping disk
|
||
```
|
||
|
||
## Widgets
|
||
|
||
A widget is a spec — `{type, sensor, title?}` — over this API and nothing else
|
||
(`web/src/widgets.tsx`, fed by `useSensorStream` in `web/src/sensors.ts`). Types: `stat`,
|
||
`gauge`, `bar`, `spark`, and `list` for a pattern. Scale and colour come from the
|
||
sensor's own `min`/`max`/`warn`/`crit`.
|
||
|
||
- **In Ham Lab:** Dashboard → Customize → **+ Add widget**. Saved with the layout,
|
||
per machine; all of a page's widgets share one stream.
|
||
- **Widgets tab:** every widget type, then every sensor on the machine drawn once, live,
|
||
grouped by family (`web/src/Gallery.tsx`). Tap a tile to copy its sensor ID — the
|
||
quickest way to find what to put in a widget, a query or a Glance tile.
|
||
- **Anywhere else:** `/embed/widget` is one widget alone on a page, live, in its own small
|
||
bundle (the console never loads), and the only path Ham Lab allows to be framed.
|
||
|
||
```
|
||
/embed/widget?type=gauge&sensor=cpu.usage&token=<read-only token>
|
||
&machine=grid &title=… &theme=dark|light|cyberpunk|retro &every=<ms> &bare=1
|
||
```
|
||
|
||
### Glance
|
||
|
||
Glance's own `custom-api` widget refetches on page load / cache expiry — fine for slow
|
||
facts, never live. An `iframe` of the embed is live.
|
||
|
||
```yaml
|
||
- type: iframe
|
||
title: core CPU
|
||
source: https://hamlab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
|
||
height: 130
|
||
|
||
- type: custom-api
|
||
title: grid
|
||
cache: 1m
|
||
url: https://hamlab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
|
||
headers: { Authorization: Bearer <read-only token> }
|
||
template: |
|
||
<div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>
|
||
```
|
||
|
||
## The whole fleet in one request
|
||
|
||
On a hub, `/api/fleet/sensors/query` asks every machine (or `machines`) the same shaped
|
||
query and answers `{machine: result}`; this machine answers under its own hostname, and
|
||
one that can't be reached answers `{"error": …}` while the rest still do.
|
||
|
||
```
|
||
POST /api/fleet/sensors/query {"machines": "*", "query": {"cpu": "cpu.usage", "up": "uptime"}}
|
||
GET /api/fleet/sensors/query?machines=grid,wyse&cpu=cpu.usage&failed=services.failed
|
||
|
||
→ {"core": {"cpu": 4.8, "up": 50211}, "grid": {"cpu": 0.4, "up": 437177}, "mac": {"error": "timeout"}}
|
||
```
|
||
|
||
## What counts as a disk
|
||
|
||
`disk.*` names **drives**, not the block devices the kernel lists. An NVMe controller can
|
||
present several namespaces — an Apple SSD shows `nvme0n1`, `nvme0n2` and `nvme0n3`, the last
|
||
two a few MiB of firmware — and each one arrives from `/proc/diskstats` looking like a whole
|
||
disk. They are folded into the controller (`disk.nvme0.*`), with the rates added and busy
|
||
taken as the longest of them rather than the sum, since they share one controller. This is
|
||
what the Disks page has always shown; since 2026-09-22 the dashboard, the Top tab and these
|
||
sensors agree with it.
|
||
|
||
## The Top tab
|
||
|
||
Ham Lab → **Top** is a whole machine on one screen in the style of btop: black ground, boxes
|
||
with their name set into the border, curves drawn as dots that climb green → yellow → red.
|
||
It reads nothing of its own — the sensor stream for everything measured, `/api/procs` for the
|
||
table — so it works on any machine the console can reach, whether or not btop is installed
|
||
there, and it follows the same refresh rate and history window as the rest of the console.
|
||
Its colours are the active skin's: the terminal background for the ground, the skin's sixteen
|
||
terminal colours for the curves and meters. (The look is btop's, `web/src/Top.tsx` is ours;
|
||
btop itself is Apache-2.0.)
|
||
|
||
## How often a machine samples
|
||
|
||
The console's refresh rate (Settings, or `− rate +` on Top) is also the fleet's sampling rate.
|
||
It is saved with the UI prefs on the hub, which sets its own sampler and passes the rate to
|
||
every machine it manages — `PUT /api/prefs/sampler {"ms": …}`, 100 ms to 60 s, sampled no
|
||
slower than the collector's 10 s — and repeats it at start-up. `GET` answers `{ms, min_ms,
|
||
max_ms}`. Before 2026-09-21 only an open Dashboard page told a machine its rate.
|
||
|
||
## How far back a history reaches
|
||
|
||
Every machine keeps **300 points per curve**, whatever the window — so the window sets how
|
||
far apart they are, not how much memory monitoring costs. At the default ten minutes each
|
||
point is one sample (2 s apart at the default rate); at a day each is the **average of the
|
||
five minutes behind it**, not whichever instant landed on the mark, so a long graph shows
|
||
the load that really happened. Ham Lab → Settings → **Graph history** is the one control,
|
||
1 minute to 24 hours; the hub passes it to every machine it manages.
|
||
|
||
The window can't be finer than the sampling interval — 300 points of a 1-minute window at a
|
||
2-second rate is 30 points, and that is what you get. It is also all in memory: a machine
|
||
that restarts starts its graphs again.
|
||
|
||
## What Ham Lab itself runs on
|
||
|
||
Since 2026-09-19 the Dashboard tab (stock modules and custom widgets) and the Disks tab's
|
||
live rates read only this API: the catalog once, one batch of histories, one stream —
|
||
closed while the page is hidden, refilled when it returns. `web/src/sampleAdapter.ts` is
|
||
the seam that hands the stock modules the object they were written against. The old
|
||
`/api/metrics` and `/api/metrics/history` are gone. `hamlab.sample_interval` is the
|
||
sampler's own rate, as a sensor.
|
||
|
||
## From another origin
|
||
|
||
The sensor endpoints (and only they) answer any origin: `Access-Control-Allow-Origin: *`,
|
||
preflight included. `*` is the one value browsers won't combine with cookies, so a foreign
|
||
page can reach the API with a token and can never ride a logged-in console's session.
|
||
`examples/external-demo/` is a complete outside consumer in one HTML file.
|