hamlab/SENSORS.md

Sensors — the standard read interface

Everything Ham Lab measures is a sensor: a stable dotted ID, a little metadata, a current value. Any consumer — Ham Lab's own dashboard, Glance, Home Assistant, a shell script — discovers them from the catalog and asks for exactly the ones it wants.

Code: internal/sensors (types, registry, selectors, shaped query) and internal/metrics/sensors.go (the collector as a provider). The registry measures nothing; providers do. A new source of sensors implements sensors.Provider and is added to the registry in api.New.

IDs

cpu.usage  cpu.core.<n>.usage  cpu.temp  load.1|5|15  uptime
mem.total|used|avail|free|cached|buffers|used_pct     swap.total|used|avail|used_pct
temp.<chip>.<label>            fan.<chip>.<label>
net.<iface>.rx|tx|rx_total|tx_total|up
disk.<dev>.read|write|util|size|temp
fs.<mount>.size|used|used_pct
ups.battery|runtime|load|load_pct|in_v|out_v|on_battery|state|source

• Segments are lower case [a-z0-9_-]: Package id 0 → package_id_0, /mnt/tank → mnt_tank, / → root. The real names are in the sensor's labels.
• Built from names the hardware reports, never enumeration order (hwmon3 moves between boots). Two identical chips collide; the second gets _2.
• Which sensors exist differs per machine. That is the point of the catalog.

A sensor

{ "id": "temp.coretemp.package_id_0", "name": "coretemp Package id 0",
  "kind": "temperature", "unit": "°C", "value": 51, "t": 1789826600878,
  "warn": 105, "crit": 105, "labels": { "chip": "coretemp", "label": "Package id 0" } }

value is a number, bool or string. t is when it was really read, in unix ms — slow sections (filesystems 30 s, UPS 10 s) are older than the tick. min/max are the scale (0–100 for percentages, 0–total for used bytes); warn/crit are the hardware's own thresholds where it states them.

Endpoints

All behind the same auth as the rest of /api (session cookie or Authorization: Bearer). On a hub, every one is also at /api/nodes/<machine>/sensors… for each managed machine.

GET /api/sensors	Catalog, with current values. ?ids=fan.*,cpu.usage narrows it.
GET /api/sensors/{id}	One sensor. ?format=text → the bare value as text/plain.
GET /api/sensors/{id}/history	{id, unit, t[], v[]} — what the ring holds (300 points, spaced by the window below). Numbers only.
GET /api/sensors/history?ids=cpu.usage,net.*.rx	Many histories in one request: {series: [{id, unit, t[], v[]}]}. A sensor younger than the ring has a shorter series.
GET /api/prefs/history	How far back the history reaches: {seconds, points, stride_sec, min_seconds, max_seconds}.
PUT /api/prefs/history	{seconds} — 60 to 86400. Set on the hub it is passed to every machine, so the fleet's graphs reach equally far back.
POST /api/sensors/query	Shaped query, below.
GET /api/sensors/query?k=selector&…	The same, flat, for consumers that can only fetch a URL.

A selector is an exact ID or a glob. * crosses dots: fan.* is every fan, cpu.core.*.usage every core.

Shaped query

Send the object you want, with selectors where the values go. Get it back filled in.

POST /api/sensors/query
{ "cpu": "cpu.usage", "fans": "fan.*", "box": { "temp": "cpu.temp", "up": "uptime" } }

→ { "cpu": 12.4,
    "fans": { "fan.nct6775.fan1": 820, "fan.nct6775.fan2": 1140 },
    "box": { "temp": 41, "up": 80520 } }

• Exact ID → its value; null if this machine has no such sensor (so one query works across machines that differ).
• Glob → an object keyed by ID; {} if nothing matches.
• Objects and arrays are walked; anything else in a query is a 400.
• ?full=1 → every value is the whole sensor object instead of the bare value. (In the GET form full is therefore not usable as a key.)

Live stream

GET /api/sensors/stream is a shaped query that keeps answering: the same object, pushed as a server-sent event after every sample. The query is ?q=<URL-encoded JSON shape> or the flat form; ?full=1 as above; ?every=<ms> spaces events out for a consumer that doesn't want them at the sampling rate. A keepalive comment goes out every 25 s, and X-Accel-Buffering: no keeps nginx from holding events back.

new EventSource('/api/sensors/stream?token=…&q=' + encodeURIComponent(JSON.stringify({ cpu: 'cpu.usage', fans: 'fan.*' })))
  .onmessage = (e) => render(JSON.parse(e.data))

Auth: read-only tokens

Settings → Access tokens → + Read-only sensor token (or POST /api/tokens/read {"name"} with a session). Scope sensors:read: it opens every endpoint in this file, here and on each managed machine through the hub (/api/nodes/<machine>/sensors…; the hub talks to the node as itself, the caller's token stops at the hub), and nothing else.

• Send it as Authorization: Bearer <token>, or as ?token=<token> where headers are impossible (an iframe, a browser EventSource).
• A read-only token can be limited to ID patterns ({"name": "glance", "sensors": ["cpu.*", "zfs.tank.*"]}, or the second prompt in Settings). Everything outside them does not exist for that token — catalog, query, history, stream, Prometheus, and on managed machines through the hub, which passes the restriction along.
• A full-control token is never accepted from a URL, and a read-only token is never accepted anywhere outside this file's endpoints.

Prometheus

GET /metrics (or /api/sensors/prometheus, which also works through the hub as /api/nodes/<machine>/sensors/prometheus). One gauge family per kind of measurement, base units, the sensor ID and its real-world names as labels:

hamlab_temperature_celsius{id="temp.coretemp.core_0",chip="coretemp",label="Core 0"} 50
hamlab_usage_percent{id="fs.root.used_pct",fstype="ext4",mount="/"} 10.1
hamlab_state{id="ups.state",model="CPS CP1500PFCRM2U",value="Normal"} 1
hamlab_updates_pending{id="updates.pending",manager="apt"} 3

scrape_configs:
  - job_name: hamlab
    authorization: { credentials: <read-only token> }
    static_configs: [{ targets: ['10.0.0.234:8100'] }]

Status sensors

Not everything is a /proc number. These are read in the background on their own clocks (each sensor's t is its real age) by internal/api/status_sensors.go:

services.failed|failed_names|running            every 30 s
service.<unit>.active|state                     one pair per service that is enabled, up or failed
containers.total|running|unhealthy              every 15 s (Docker or Podman, if present)
container.<name>.running|state|health|mem|cpu   mem and cpu read from the container's cgroup
container.<name>.down                           should be up and isn't: Docker couldn't start it, it is
                                                stuck restarting, or it died non-zero with no policy to
                                                restart it — *not* a container someone stopped
updates.pending|security|reboot_required        every 5 min
zfs.<pool>.health|healthy|size|used|used_pct|frag   every 1 min
smart.<dev>.passed|warnings|power_on|wear       every 10 min, never waking a sleeping disk

Widgets

A widget is a spec — {type, sensor, title?} — over this API and nothing else (web/src/widgets.tsx, fed by useSensorStream in web/src/sensors.ts). Types: stat, gauge, bar, spark, and list for a pattern. Scale and colour come from the sensor's own min/max/warn/crit.

• In Ham Lab: Dashboard → Customize → + Add widget. Saved with the layout, per machine; all of a page's widgets share one stream.
• Widgets tab: every widget type, then every sensor on the machine drawn once, live, grouped by family (web/src/Gallery.tsx). Tap a tile to copy its sensor ID — the quickest way to find what to put in a widget, a query or a Glance tile.
• Anywhere else: /embed/widget is one widget alone on a page, live, in its own small bundle (the console never loads), and the only path Ham Lab allows to be framed.

/embed/widget?type=gauge&sensor=cpu.usage&token=<read-only token>
   &machine=grid   &title=…   &theme=dark|light|cyberpunk|retro   &every=<ms>   &bare=1

Glance

Glance's own custom-api widget refetches on page load / cache expiry — fine for slow facts, never live. An iframe of the embed is live.

- type: iframe
  title: core CPU
  source: https://hamlab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
  height: 130

- type: custom-api
  title: grid
  cache: 1m
  url: https://hamlab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
  headers: { Authorization: Bearer <read-only token> }
  template: |
    <div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>

The whole fleet in one request

On a hub, /api/fleet/sensors/query asks every machine (or machines) the same shaped query and answers {machine: result}; this machine answers under its own hostname, and one that can't be reached answers {"error": …} while the rest still do.

POST /api/fleet/sensors/query   {"machines": "*", "query": {"cpu": "cpu.usage", "up": "uptime"}}
GET  /api/fleet/sensors/query?machines=grid,wyse&cpu=cpu.usage&failed=services.failed

→ {"core": {"cpu": 4.8, "up": 50211}, "grid": {"cpu": 0.4, "up": 437177}, "mac": {"error": "timeout"}}

How far back a history reaches

Every machine keeps 300 points per curve, whatever the window — so the window sets how far apart they are, not how much memory monitoring costs. At the default ten minutes each point is one sample (2 s apart at the default rate); at a day each is the average of the five minutes behind it, not whichever instant landed on the mark, so a long graph shows the load that really happened. Ham Lab → Settings → Graph history is the one control, 1 minute to 24 hours; the hub passes it to every machine it manages.

The window can't be finer than the sampling interval — 300 points of a 1-minute window at a 2-second rate is 30 points, and that is what you get. It is also all in memory: a machine that restarts starts its graphs again.

What Ham Lab itself runs on

Since 2026-09-19 the Dashboard tab (stock modules and custom widgets) and the Disks tab's live rates read only this API: the catalog once, one batch of histories, one stream — closed while the page is hidden, refilled when it returns. web/src/sampleAdapter.ts is the seam that hands the stock modules the object they were written against. The old /api/metrics and /api/metrics/history are gone. hamlab.sample_interval is the sampler's own rate, as a sensor.

From another origin

The sensor endpoints (and only they) answer any origin: Access-Control-Allow-Origin: *, preflight included. * is the one value browsers won't combine with cookies, so a foreign page can reach the API with a token and can never ride a logged-in console's session. examples/external-demo/ is a complete outside consumer in one HTML file.