hamlab/SENSORS.md

Sensors — the standard read interface

Everything Ham Lab measures is a sensor: a stable dotted ID, a little metadata, a current value. Any consumer — Ham Lab's own dashboard, Glance, Home Assistant, a shell script — discovers them from the catalog and asks for exactly the ones it wants.

Code: internal/sensors (types, registry, selectors, shaped query) and internal/metrics/sensors.go (the collector as a provider). The registry measures nothing; providers do. A new source of sensors implements sensors.Provider and is added to the registry in api.New.

IDs

cpu.usage  cpu.core.<n>.usage  cpu.temp  load.1|5|15  uptime
mem.total|used|avail|free|cached|buffers|used_pct     swap.total|used|avail|used_pct
temp.<chip>.<label>            fan.<chip>.<label>
net.<iface>.rx|tx|rx_total|tx_total|up
net.internet.download|upload|latency|jitter|tested|ip|edge
disk.<dev>.read|write|util|size|temp
fs.<mount>.size|used|used_pct
ups.battery|runtime|load|load_pct|in_v|out_v|on_battery|state|source
gpu.<name>.usage|render|video|video-enhance|copy|compute|temp|power|fan|freq|mem_used|mem_total|clients

• Segments are lower case [a-z0-9_-]: Package id 0 → package_id_0, /mnt/tank → mnt_tank, / → root. The real names are in the sensor's labels.
• Built from names the hardware reports, never enumeration order (hwmon3 moves between boots). Two identical chips collide; the second gets _2.
• Which sensors exist differs per machine. That is the point of the catalog.
• net.internet.* is the Network tab's speed test: download and upload in Mbps, latency and jitter in ms, tested the age of the reading in seconds, and ip and edge the public address and the Cloudflare edge as strings. They appear only once a test has run on that machine, and each one's t is when it ran — so a widget or an alert on them is reading a measurement from that moment, not a live value. Nothing takes a speed test on its own; something has to press the button.
• gpu.<name>.* is each graphics device, named as lspci names it (gpu.intel_arc_a380; labels carry driver and the PCI bus). usage is how busy the device was — from the GT's idle residency on Intel, gpu_busy_percent on amdgpu — and is absent on a driver with no such counter. The per-engine percentages (render, video…) are the sum over the processes holding the device open, read from DRM fdinfo, and exist only while something has it open. temp is the card's own sensor, or on an integrated GPU the CPU package it shares a die with (label source says which); an Arc on i915 has none and reports none. freq is the actual clock, 0 when power-gated. clients counts the processes using it.

A sensor

{ "id": "temp.coretemp.package_id_0", "name": "coretemp Package id 0",
  "kind": "temperature", "unit": "°C", "value": 51, "t": 1789826600878,
  "warn": 105, "crit": 105, "labels": { "chip": "coretemp", "label": "Package id 0" } }

value is a number, bool or string. t is when it was really read, in unix ms — slow sections (filesystems 30 s, UPS 10 s) are older than the tick. min/max are the scale (0–100 for percentages, 0–total for used bytes); warn/crit are the hardware's own thresholds where it states them.

Endpoints

All behind the same auth as the rest of /api (session cookie or Authorization: Bearer). On a hub, every one is also at /api/nodes/<machine>/sensors… for each managed machine.

GET /api/sensors	Catalog, with current values. ?ids=fan.*,cpu.usage narrows it.
GET /api/sensors/{id}	One sensor. ?format=text → the bare value as text/plain.
GET /api/sensors/{id}/history	{id, unit, t[], v[]} — what the ring holds (300 points, spaced by the window below). Numbers only.
GET /api/sensors/history?ids=cpu.usage,net.*.rx	Many histories in one request: {series: [{id, unit, t[], v[]}]}. A sensor younger than the ring has a shorter series.
GET /api/prefs/history	How far back the history reaches: {seconds, points, stride_sec, min_seconds, max_seconds}.
PUT /api/prefs/history	{seconds} — 60 to 86400. Set on the hub it is passed to every machine, so the fleet's graphs reach equally far back.
POST /api/sensors/query	Shaped query, below.
GET /api/sensors/query?k=selector&…	The same, flat, for consumers that can only fetch a URL.

A selector is an exact ID or a glob. * crosses dots: fan.* is every fan, cpu.core.*.usage every core.

Shaped query

Send the object you want, with selectors where the values go. Get it back filled in.

POST /api/sensors/query
{ "cpu": "cpu.usage", "fans": "fan.*", "box": { "temp": "cpu.temp", "up": "uptime" } }

→ { "cpu": 12.4,
    "fans": { "fan.nct6775.fan1": 820, "fan.nct6775.fan2": 1140 },
    "box": { "temp": 41, "up": 80520 } }

• Exact ID → its value; null if this machine has no such sensor (so one query works across machines that differ).
• Glob → an object keyed by ID; {} if nothing matches.
• Objects and arrays are walked; anything else in a query is a 400.
• ?full=1 → every value is the whole sensor object instead of the bare value. (In the GET form full is therefore not usable as a key.)

Live stream

GET /api/sensors/stream is a shaped query that keeps answering: the same object, pushed as a server-sent event after every sample. The query is ?q=<URL-encoded JSON shape> or the flat form; ?full=1 as above; ?every=<ms> spaces events out for a consumer that doesn't want them at the sampling rate. A keepalive comment goes out every 25 s, and X-Accel-Buffering: no keeps nginx from holding events back.

new EventSource('/api/sensors/stream?token=…&q=' + encodeURIComponent(JSON.stringify({ cpu: 'cpu.usage', fans: 'fan.*' })))
  .onmessage = (e) => render(JSON.parse(e.data))

Auth: read-only tokens

Settings → Access tokens → + Read-only sensor token (or POST /api/tokens/read {"name"} with a session). Scope sensors:read: it opens every endpoint in this file, here and on each managed machine through the hub (/api/nodes/<machine>/sensors…; the hub talks to the node as itself, the caller's token stops at the hub), and nothing else.

• Send it as Authorization: Bearer <token>, or as ?token=<token> where headers are impossible (an iframe, a browser EventSource).
• A read-only token can be limited to ID patterns ({"name": "glance", "sensors": ["cpu.*", "zfs.tank.*"]}, or the second prompt in Settings). Everything outside them does not exist for that token — catalog, query, history, stream, Prometheus, and on managed machines through the hub, which passes the restriction along.
• A full-control token is never accepted from a URL, and a read-only token is never accepted anywhere outside this file's endpoints.

Prometheus

GET /metrics (or /api/sensors/prometheus, which also works through the hub as /api/nodes/<machine>/sensors/prometheus). One gauge family per kind of measurement, base units, the sensor ID and its real-world names as labels:

hamlab_temperature_celsius{id="temp.coretemp.core_0",chip="coretemp",label="Core 0"} 50
hamlab_usage_percent{id="fs.root.used_pct",fstype="ext4",mount="/"} 10.1
hamlab_state{id="ups.state",model="CPS CP1500PFCRM2U",value="Normal"} 1
hamlab_updates_pending{id="updates.pending",manager="apt"} 3

scrape_configs:
  - job_name: hamlab
    authorization: { credentials: <read-only token> }
    static_configs: [{ targets: ['10.0.0.234:8100'] }]

Status sensors

Not everything is a /proc number. These are read in the background on their own clocks (each sensor's t is its real age) by internal/api/status_sensors.go:

services.failed|failed_names|running            every 30 s
service.<unit>.active|state                     one pair per service that is enabled, up or failed
containers.total|running|unhealthy              every 15 s (Docker or Podman, if present)
container.<name>.running|state|health|mem|cpu   mem and cpu read from the container's cgroup
container.<name>.down                           should be up and isn't: Docker couldn't start it, it is
                                                stuck restarting, or it died non-zero with no policy to
                                                restart it — *not* a container someone stopped
updates.pending|security|reboot_required        every 5 min
zfs.<pool>.health|healthy|size|used|used_pct|frag   every 1 min
smart.<dev>.passed|warnings|power_on|wear       every 10 min, never waking a sleeping disk

Widgets

A widget is a spec — {type, sensor, title?} — over this API and nothing else (web/src/widgets.tsx, fed by useSensorStream in web/src/sensors.ts). Types: stat, gauge, bar, spark, and list for a pattern. Scale and colour come from the sensor's own min/max/warn/crit.

• In Ham Lab: Dashboard → Customize → + Add widget. Saved with the layout, per machine; all of a page's widgets share one stream.
• Widgets tab: every widget type, then every sensor on the machine drawn once, live, grouped by family (web/src/Gallery.tsx). Tap a tile to copy its sensor ID — the quickest way to find what to put in a widget, a query or a Glance tile.
• Anywhere else: /embed/widget is one widget alone on a page, live, in its own small bundle (the console never loads), and the only path Ham Lab allows to be framed.

/embed/widget?type=gauge&sensor=cpu.usage&token=<read-only token>
   &machine=grid   &title=…   &theme=dark|light|cyberpunk|retro   &every=<ms>   &bare=1

Glance

Glance's own custom-api widget refetches on page load / cache expiry — fine for slow facts, never live. An iframe of the embed is live.

- type: iframe
  title: core CPU
  source: https://hamlab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
  height: 130

- type: custom-api
  title: grid
  cache: 1m
  url: https://hamlab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
  headers: { Authorization: Bearer <read-only token> }
  template: |
    <div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>

The whole fleet in one request

On a hub, /api/fleet/sensors/query asks every machine (or machines) the same shaped query and answers {machine: result}; this machine answers under its own hostname, and one that can't be reached answers {"error": …} while the rest still do.

POST /api/fleet/sensors/query   {"machines": "*", "query": {"cpu": "cpu.usage", "up": "uptime"}}
GET  /api/fleet/sensors/query?machines=grid,wyse&cpu=cpu.usage&failed=services.failed

→ {"core": {"cpu": 4.8, "up": 50211}, "grid": {"cpu": 0.4, "up": 437177}, "mac": {"error": "timeout"}}

What counts as a disk

disk.* names drives, not the block devices the kernel lists. An NVMe controller can present several namespaces — an Apple SSD shows nvme0n1, nvme0n2 and nvme0n3, the last two a few MiB of firmware — and each one arrives from /proc/diskstats looking like a whole disk. They are folded into the controller (disk.nvme0.*), with the rates added and busy taken as the longest of them rather than the sum, since they share one controller. This is what the Disks page has always shown; since 2026-09-22 the dashboard, the Top tab and these sensors agree with it.

The Top tab

Ham Lab → Top is a whole machine on one screen in the style of btop: black ground, boxes with their name set into the border, curves drawn as dots that climb green → yellow → red. It reads nothing of its own — the sensor stream for everything measured, /api/procs for the table — so it works on any machine the console can reach, whether or not btop is installed there, and it follows the same refresh rate and history window as the rest of the console. Its colours are the active skin's: the terminal background for the ground, the skin's sixteen terminal colours for the curves and meters. (The look is btop's, web/src/Top.tsx is ours; btop itself is Apache-2.0.)

How often a machine samples

The console's refresh rate (Settings, or − rate + on Top) is also the fleet's sampling rate. It is saved with the UI prefs on the hub, which sets its own sampler and passes the rate to every machine it manages — PUT /api/prefs/sampler {"ms": …}, 100 ms to 60 s, sampled no slower than the collector's 10 s — and repeats it at start-up. GET answers {ms, min_ms, max_ms}. Before 2026-09-21 only an open Dashboard page told a machine its rate.

How far back a history reaches

Every machine keeps 300 points per curve, whatever the window — so the window sets how far apart they are, not how much memory monitoring costs. At the default ten minutes each point is one sample (2 s apart at the default rate); at a day each is the average of the five minutes behind it, not whichever instant landed on the mark, so a long graph shows the load that really happened. Ham Lab → Settings → Graph history is the one control, 1 minute to 24 hours; the hub passes it to every machine it manages.

The window can't be finer than the sampling interval — 300 points of a 1-minute window at a 2-second rate is 30 points, and that is what you get. It is also all in memory: a machine that restarts starts its graphs again.

What Ham Lab itself runs on

Since 2026-09-19 the Dashboard tab (stock modules and custom widgets) and the Disks tab's live rates read only this API: the catalog once, one batch of histories, one stream — closed while the page is hidden, refilled when it returns. web/src/sampleAdapter.ts is the seam that hands the stock modules the object they were written against. The old /api/metrics and /api/metrics/history are gone. hamlab.sample_interval is the sampler's own rate, as a sensor.

From another origin

The sensor endpoints (and only they) answer any origin: Access-Control-Allow-Origin: *, preflight included. * is the one value browsers won't combine with cookies, so a foreign page can reach the API with a token and can never ride a logged-in console's session. examples/external-demo/ is a complete outside consumer in one HTML file.