GPU boxes on the Processes tab: busy, engines, temps, and who has the card open

Every DRM device with a real GPU driver gets a box under the cpu box, laid out
like it: the busy curve, a meter for the device and one per engine, temperature,
clock, power and fan where the card has them, and the processes holding it open
with their share. The process table gains a Gpu% column and sort.

Device busy is Intel's RC6 idle residency inverted (what intel_gpu_top shows,
and readable without root) or amdgpu's gpu_busy_percent. Per-process shares are
DRM fdinfo deltas, as nvtop reads them; other users' fdinfo needs root, so a
daemon running as a user keeps one root child (hamlab drm-clients) over a pipe
rather than paying a sudo a second. Everything is also a gpu.* sensor.

What cannot be measured is left out, not shown as zero: an Arc A380 on i915 6.12
has no temperature sensor, an integrated GPU wears the package temperature
marked *, and Asahi has neither a busy counter nor fdinfo yet.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

PLAN.md

@@ -299,6 +299,21 @@ something long in one, switch away and back, refresh — it's all still there.
   id and its route are gone. A rail order saved before this keeps its shape — `railorder.ts`
   maps the retired `top` to `procs` and drops the duplicate, so the merged page sits where
   Top sat rather than falling to the bottom.
+- **GPUs on the Processes tab** — **shipped 2026-10-02** (`internal/metrics/gpu.go`, the
+  `GPUBox` in `Top.tsx`, `gpu.*` sensors). Asked for as "GPU activity and temps, like btop
+  has it, always on". Each DRM device with a real GPU driver gets a box under the cpu box:
+  busy curve, busy meter, a meter per engine (render/video/copy/enhance), temperature, clock,
+  power and fan where the card has them, and the processes that have it open with their
+  share. The process table gains a `Gpu%` column and sort. **How it is measured, and the
+  honest gaps:** device busy is Intel's RC6 idle residency inverted (the counter every GT
+  exposes without root; `intel_gpu_top` shows the same number) or amdgpu's own percent;
+  per-process and per-engine shares are DRM fdinfo deltas, the way nvtop reads them. Other
+  users' fdinfo needs root, so a daemon running as `core` keeps **one** root child
+  (`hamlab drm-clients`, over a pipe) rather than a sudo a second. The Arc A380 on i915 6.12
+  exposes power, energy and fan but **no temperature**, so its box shows none; an integrated
+  GPU shows the CPU package temperature marked `*`. Asahi (the Mac mini) has neither a busy
+  counter nor fdinfo in 6.19, so it lists the device and says so. No NVIDIA path: nothing here
+  has one to test against.
 - **Dashboard retired** — **2026-09-23**, at Mitchell's instruction, the same reasoning that
   merged Top into Processes: the app repeated itself. The customisable module list is gone
   (`Dashboard.tsx` deleted), and with it `/api/prefs/dashboard`, `applyPrefs`, and the

SENSORS.md

@@ -20,6 +20,7 @@ net.internet.download|upload|latency|jitter|tested|ip|edge
 disk.<dev>.read|write|util|size|temp
 fs.<mount>.size|used|used_pct
 ups.battery|runtime|load|load_pct|in_v|out_v|on_battery|state|source
+gpu.<name>.usage|render|video|video-enhance|copy|compute|temp|power|fan|freq|mem_used|mem_total|clients
 ```
 
 - Segments are lower case `[a-z0-9_-]`: `Package id 0` → `package_id_0`, `/mnt/tank` →
@@ -33,6 +34,14 @@ ups.battery|runtime|load|load_pct|in_v|out_v|on_battery|state|source
   has run on that machine, and each one's `t` is when it ran — so a widget or an alert on
   them is reading a measurement from that moment, not a live value. Nothing takes a speed
   test on its own; something has to press the button.
+- `gpu.<name>.*` is each graphics device, named as lspci names it (`gpu.intel_arc_a380`;
+  labels carry `driver` and the PCI `bus`). `usage` is how busy the device was — from the
+  GT's idle residency on Intel, `gpu_busy_percent` on amdgpu — and is absent on a driver
+  with no such counter. The per-engine percentages (`render`, `video`…) are the sum over the
+  processes holding the device open, read from DRM fdinfo, and exist only while something
+  has it open. `temp` is the card's own sensor, or on an integrated GPU the CPU package it
+  shares a die with (label `source` says which); an Arc on i915 has none and reports none.
+  `freq` is the actual clock, 0 when power-gated. `clients` counts the processes using it.
 
 ## A sensor
 

cmd/hamlab/main.go

@@ -82,6 +82,8 @@ func main() {
 		os.Exit(files.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
 	case "space-walk": // helper for the Space tab: measure mountpoints, stream the tree back
 		os.Exit(space.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
+	case "drm-clients": // privileged helper for the GPU readings: who has the GPU open (run as root by the daemon)
+		os.Exit(metrics.DRMClientsHelper(os.Stdin, os.Stdout))
 	case "install", "uninstall":
 		var err error
 		if cmd == "install" {

internal/metrics/gpu.go

@@ -0,0 +1,646 @@
+package metrics
+
+// GPU telemetry, the way btop and intel_gpu_top get it without a vendor library: the
+// device from sysfs, and who is using it from DRM fdinfo.
+//
+//   - The device: /sys/class/drm/card<n> with a driver we know to be a GPU. Clock from the
+//     driver's frequency files, temperature/power/fan from the hwmon under the device, and
+//     how busy it is from the one counter every Intel GT exports without root — RC6 idle
+//     residency, so busy = 100 − idle. amdgpu says it outright (gpu_busy_percent).
+//   - The clients: every process holding /dev/dri/* open has a drm-engine-<name> running
+//     total (nanoseconds) in /proc/<pid>/fdinfo/<fd>; the delta between two reads is that
+//     process's share of each engine, and the sum across clients is the engine's load. Other
+//     users' fdinfo is unreadable, so when the daemon isn't root it keeps one root child
+//     (`hamlab drm-clients`) that does the walk, asked over a pipe — one sudo per daemon
+//     start, not one per second.
+//
+// A GPU with no counter of either kind (Asahi, today) is still listed: a name is better
+// than silence about hardware the machine has.
+
+import (
+	"bufio"
+	"encoding/json"
+	"io"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"hamlab/internal/priv"
+)
+
+// GPU is one graphics device as a Sample reports it.
+type GPU struct {
+	Name       string             `json:"name"`
+	Driver     string             `json:"driver"`
+	Bus        string             `json:"bus,omitempty"` // PCI address; empty for a platform device
+	Integrated bool               `json:"integrated,omitempty"`
+	Usage      float64            `json:"usage"`               // % busy
+	UsageSrc   string             `json:"usage_src,omitempty"` // "idle" (100 − idle residency), "busy" (the driver's own %), "clients" (the busiest engine); "" when nothing measures it
+	Engines    map[string]float64 `json:"engines,omitempty"`   // render/video/copy/enhance… % from the clients, normalised by engine count
+	Temp       float64            `json:"temp,omitempty"`
+	TempSrc    string             `json:"temp_src,omitempty"` // "gpu" from the device's own sensor, "package" the CPU die an integrated GPU shares
+	PowerW     float64            `json:"power_w,omitempty"`
+	PowerCapW  float64            `json:"power_cap_w,omitempty"`
+	FanRPM     int                `json:"fan_rpm,omitempty"`
+	MHz        float64            `json:"mhz,omitempty"`     // actual clock (0 when power-gated)
+	ReqMHz     float64            `json:"req_mhz,omitempty"` // what the driver asked for
+	MaxMHz     float64            `json:"max_mhz,omitempty"`
+	MemUsed    uint64             `json:"mem_used,omitempty"` // VRAM used, or the clients' resident bytes where the driver has no VRAM counter
+	MemTotal   uint64             `json:"mem_total,omitempty"`
+	Clients    []GPUClient        `json:"clients"`
+	ClientsOK  bool               `json:"clients_ok"` // false when fdinfo of other users' processes could not be read
+}
+
+// GPUClient is one process with the device open.
+type GPUClient struct {
+	PID     int                `json:"pid"`
+	Name    string             `json:"name"`
+	Usage   float64            `json:"usage"` // its busiest engine, %
+	Engines map[string]float64 `json:"engines,omitempty"`
+	Mem     uint64             `json:"mem,omitempty"` // resident bytes
+}
+
+// gpuDrivers are DRM drivers that drive a GPU, not just a scanout (simpledrm, ast, the
+// Apple display coprocessor) — those have no engines and nothing to say.
+var gpuDrivers = map[string]bool{"i915": true, "xe": true, "amdgpu": true, "radeon": true, "nouveau": true, "nvidia-drm": true,
+	"asahi": true, "panfrost": true, "panthor": true, "v3d": true, "msm": true, "etnaviv": true, "lima": true, "powervr": true, "virtio_gpu": true}
+
+type gpuDev struct {
+	card, dev, key, bus, driver, name, hwmon string
+	integrated                               bool
+	prevIdle, prevEnergy                     float64
+	prevAt                                   time.Time
+}
+
+type gpuState struct {
+	devs     []*gpuDev
+	listedAt time.Time
+	clients  map[string]clientPrev // device key + client id → engine totals at the previous scan
+	last     []drmFD               // the newest client walk, reused between walks
+	scanAt   time.Time
+	helper   *drmHelper
+	ok       bool // the last walk saw every process
+}
+
+type clientPrev struct {
+	ns map[string]uint64
+	at time.Time
+}
+
+const gpuRelist = time.Minute    // how often the card list is re-read: hotplug is rare, the readdir is cheap
+const gpuScanEvery = time.Second // engine shares over less than this are noise
+
+// gpus is the GPU section of a sample. cpuTemp stands in for an integrated GPU's
+// temperature: it is the same die.
+func (c *Collector) gpus(now time.Time, cpuTemp float64) []GPU {
+	g := &c.gpu
+	if g.devs == nil || now.Sub(g.listedAt) >= gpuRelist {
+		g.devs, g.listedAt = listGPUs(g.devs), now
+	}
+	if len(g.devs) == 0 {
+		return []GPU{}
+	}
+	if g.clients == nil {
+		g.clients = map[string]clientPrev{}
+	}
+	if g.scanAt.IsZero() || now.Sub(g.scanAt) >= gpuScanEvery {
+		g.last, g.ok = c.scanClients()
+		g.scanAt = now
+	}
+	out := make([]GPU, 0, len(g.devs))
+	live := map[string]bool{}
+	for _, d := range g.devs {
+		x := GPU{Name: d.name, Driver: d.driver, Bus: d.bus, Integrated: d.integrated, Clients: []GPUClient{}, ClientsOK: g.ok}
+		d.read(&x, now)
+		if x.Temp == 0 && d.integrated && cpuTemp > 0 {
+			x.Temp, x.TempSrc = cpuTemp, "package"
+		}
+		// the clients: one entry per client id, under the first pid that holds it (a child
+		// that inherited the fd shares the client)
+		byID := map[string]*GPUClient{}
+		var order []string
+		engines := map[string]float64{}
+		var resident uint64
+		for _, f := range g.last {
+			fi := parseFdinfo(f.Info)
+			if fi.pdev != "" && fi.pdev != d.bus || fi.pdev == "" && fi.driver != d.driver {
+				continue
+			}
+			key := d.key + "/" + fi.client
+			live[key] = true
+			if byID[fi.client] != nil {
+				continue
+			}
+			cl := &GPUClient{PID: f.PID, Name: f.Comm, Mem: fi.mem}
+			byID[fi.client] = cl
+			order = append(order, fi.client)
+			resident += fi.mem
+			if prev, ok := g.clients[key]; ok {
+				if dt := now.Sub(prev.at).Nanoseconds(); dt > 0 {
+					for e, ns := range fi.engines {
+						p, ok := prev.ns[e]
+						if !ok || ns < p {
+							continue
+						}
+						pct := float64(ns-p) / float64(dt) * 100
+						if n := fi.capacity[e]; n > 1 {
+							pct /= float64(n)
+						}
+						pct = round1(pct)
+						if cl.Engines == nil {
+							cl.Engines = map[string]float64{}
+						}
+						cl.Engines[e] = pct
+						engines[e] += pct
+						if pct > cl.Usage {
+							cl.Usage = pct
+						}
+					}
+				}
+			}
+			g.clients[key] = clientPrev{ns: fi.engines, at: now}
+		}
+		for _, id := range order {
+			x.Clients = append(x.Clients, *byID[id])
+		}
+		sort.SliceStable(x.Clients, func(i, j int) bool { return x.Clients[i].Usage > x.Clients[j].Usage })
+		if len(engines) > 0 {
+			x.Engines = map[string]float64{}
+			for e, v := range engines {
+				if v > 100 {
+					v = 100
+				}
+				x.Engines[e] = round1(v)
+			}
+		}
+		if x.UsageSrc == "" && len(x.Engines) > 0 {
+			for _, v := range x.Engines {
+				if v > x.Usage {
+					x.Usage = v
+				}
+			}
+			x.UsageSrc = "clients"
+		}
+		if x.MemTotal == 0 && x.MemUsed == 0 {
+			x.MemUsed = resident
+		}
+		out = append(out, x)
+	}
+	for k := range g.clients { // a client that closed the device
+		if !live[k] {
+			delete(g.clients, k)
+		}
+	}
+	return out
+}
+
+// listGPUs enumerates the cards, keeping the counters of those already known.
+func listGPUs(known []*gpuDev) []*gpuDev {
+	ents, _ := os.ReadDir("/sys/class/drm")
+	old := map[string]*gpuDev{}
+	for _, d := range known {
+		old[d.card] = d
+	}
+	out := []*gpuDev{}
+	for _, e := range ents {
+		n := e.Name()
+		if !strings.HasPrefix(n, "card") || strings.Contains(n, "-") {
+			continue
+		}
+		card := "/sys/class/drm/" + n
+		dev, err := filepath.EvalSymlinks(card + "/device")
+		if err != nil {
+			continue
+		}
+		driver := ueventValue(dev+"/uevent", "DRIVER")
+		if driver == "" { // some drivers hang the DRIVER key off the parent
+			driver = filepath.Base(readLink(dev + "/driver"))
+		}
+		if !gpuDrivers[driver] {
+			continue
+		}
+		if d, ok := old[card]; ok && d.dev == dev {
+			out = append(out, d)
+			continue
+		}
+		d := &gpuDev{card: card, dev: dev, driver: driver}
+		if pci := ueventValue(dev+"/uevent", "PCI_SLOT_NAME"); pci != "" {
+			d.bus, d.key = pci, pci
+			// an on-die GPU hangs off root bus 0 (Intel's 00:02.0, an AMD APU's 00:08.0 bridge)
+			d.integrated = strings.HasPrefix(pci, "0000:00:")
+		} else {
+			d.key = "driver:" + driver
+			d.integrated = true
+		}
+		if hw, _ := filepath.Glob(dev + "/hwmon/hwmon*"); len(hw) > 0 {
+			d.hwmon = hw[0]
+		}
+		d.name = gpuName(dev, d.bus, driver)
+		out = append(out, d)
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].card < out[j].card })
+	return out
+}
+
+var gpuVendors = map[string]string{"0x8086": "Intel", "0x1002": "AMD", "0x10de": "NVIDIA", "0x1af4": "Virtio", "0x106b": "Apple"}
+
+var (
+	revRe     = regexp.MustCompile(`\s*\(rev [^)]*\)`)
+	bracketRe = regexp.MustCompile(`\[([^\]]+)\]`)
+)
+
+// gpuName is "Intel Arc A380", from lspci's bracketed marketing name when it has one.
+func gpuName(dev, bus, driver string) string {
+	if bus != "" {
+		if out, ok := runOut("lspci", "-s", bus); ok {
+			if _, desc, found := strings.Cut(out, ": "); found {
+				return lspciName(desc)
+			}
+		}
+	}
+	vendor := gpuVendors[strings.TrimSpace(readStr(dev+"/vendor"))]
+	if compat := readStr(dev + "/of_node/compatible"); compat != "" { // "apple,agx-t8103\0apple,agx-g13g"
+		first := strings.SplitN(compat, "\x00", 2)[0]
+		if v, model, ok := strings.Cut(first, ","); ok {
+			if vendor == "" && v != "" {
+				vendor = strings.ToUpper(v[:1]) + v[1:]
+			}
+			return vendor + " GPU (" + model + ")"
+		}
+	}
+	if vendor == "" && driver != "" {
+		vendor = strings.ToUpper(driver[:1]) + driver[1:]
+	}
+	if id := strings.TrimPrefix(strings.TrimSpace(readStr(dev+"/device")), "0x"); id != "" {
+		return strings.TrimSpace(vendor + " GPU " + id)
+	}
+	return strings.TrimSpace(vendor + " GPU")
+}
+
+// lspciName turns "Intel Corporation DG2 [Arc A380] (rev 05)" into "Intel Arc A380".
+func lspciName(desc string) string {
+	desc = revRe.ReplaceAllString(strings.TrimSpace(desc), "")
+	vendor := ""
+	for _, v := range []struct{ long, short string }{{"Intel Corporation", "Intel"}, {"Advanced Micro Devices, Inc. [AMD/ATI]", "AMD"}, {"Advanced Micro Devices, Inc. [AMD]", "AMD"}, {"NVIDIA Corporation", "NVIDIA"}, {"Red Hat, Inc.", "Virtio"}} {
+		if strings.HasPrefix(desc, v.long) {
+			vendor = v.short
+			desc = strings.TrimSpace(desc[len(v.long):])
+			break
+		}
+	}
+	if vendor == "" { // vendor not recognised: its first word
+		if i := strings.IndexByte(desc, ' '); i > 0 {
+			vendor, desc = desc[:i], strings.TrimSpace(desc[i+1:])
+		}
+	}
+	if m := bracketRe.FindStringSubmatch(desc); m != nil {
+		desc = m[1]
+	}
+	return strings.TrimSpace(vendor + " " + desc)
+}
+
+// read fills the device-level numbers: clock, idle/busy, hwmon.
+func (d *gpuDev) read(x *GPU, now time.Time) {
+	dt := now.Sub(d.prevAt).Seconds()
+	switch d.driver {
+	case "i915":
+		x.MHz = readF(d.card + "/gt_act_freq_mhz")
+		x.ReqMHz = readF(d.card + "/gt_cur_freq_mhz")
+		x.MaxMHz = readF(d.card + "/gt_max_freq_mhz")
+		idle, ok := readInt(d.card + "/gt/gt0/rc6_residency_ms")
+		if !ok {
+			idle, ok = readInt(d.card + "/power/rc6_residency_ms")
+		}
+		if ok {
+			d.busyFromIdle(x, float64(idle), dt)
+		}
+	case "xe":
+		gt := d.dev + "/tile0/gt0"
+		x.MHz = readF(gt + "/freq0/act_freq")
+		x.ReqMHz = readF(gt + "/freq0/cur_freq")
+		x.MaxMHz = readF(gt + "/freq0/max_freq")
+		if idle, ok := readInt(gt + "/gtidle/idle_residency_ms"); ok {
+			d.busyFromIdle(x, float64(idle), dt)
+		}
+	case "amdgpu":
+		if v, ok := readInt(d.dev + "/gpu_busy_percent"); ok {
+			x.Usage, x.UsageSrc = float64(v), "busy"
+		}
+		x.MemUsed = uint64(readF(d.dev + "/mem_info_vram_used"))
+		x.MemTotal = uint64(readF(d.dev + "/mem_info_vram_total"))
+		// "1: 1800Mhz *" — the starred level is the current one, the last the highest
+		for _, line := range strings.Split(readStr(d.dev+"/pp_dpm_sclk"), "\n") {
+			f := strings.Fields(line)
+			if len(f) < 2 {
+				continue
+			}
+			mhz := leadingNum(f[1])
+			x.MaxMHz = mhz
+			if strings.HasSuffix(line, "*") {
+				x.MHz = mhz
+			}
+		}
+	}
+	if d.hwmon != "" {
+		if t := readF(d.hwmon + "/temp1_input"); t > 0 {
+			x.Temp, x.TempSrc = t/1000, "gpu"
+		}
+		if p := readF(d.hwmon + "/power1_input"); p > 0 {
+			x.PowerW = round1(p / 1e6)
+		} else if p := readF(d.hwmon + "/power1_average"); p > 0 {
+			x.PowerW = round1(p / 1e6)
+		} else if e := readF(d.hwmon + "/energy1_input"); e > 0 { // µJ, cumulative
+			if d.prevEnergy > 0 && e >= d.prevEnergy && dt > 0 && !d.prevAt.IsZero() {
+				x.PowerW = round1((e - d.prevEnergy) / 1e6 / dt)
+			}
+			d.prevEnergy = e
+		}
+		if p := readF(d.hwmon + "/power1_cap"); p > 0 {
+			x.PowerCapW = p / 1e6
+		} else if p := readF(d.hwmon + "/power1_max"); p > 0 {
+			x.PowerCapW = p / 1e6
+		}
+		x.FanRPM = int(readF(d.hwmon + "/fan1_input"))
+	}
+	d.prevAt = now
+}
+
+// busyFromIdle: the GT's idle residency is a running ms total; the share of the interval
+// it did not cover is how busy the device was.
+func (d *gpuDev) busyFromIdle(x *GPU, idle, dt float64) {
+	x.UsageSrc = "idle"
+	if d.prevAt.IsZero() || dt <= 0 || idle < d.prevIdle {
+		d.prevIdle = idle
+		return
+	}
+	busy := 100 - (idle-d.prevIdle)/(dt*1000)*100
+	if busy < 0 {
+		busy = 0
+	}
+	if busy > 100 {
+		busy = 100
+	}
+	x.Usage = round1(busy)
+	d.prevIdle = idle
+}
+
+// ---- DRM fdinfo: who has the device open ----
+
+// drmFD is one client of a DRM device: the fdinfo text of one open fd, with its owner.
+type drmFD struct {
+	PID  int    `json:"pid"`
+	Comm string `json:"comm"`
+	Info string `json:"info"`
+}
+
+type fdinfo struct {
+	driver, pdev, client string
+	engines              map[string]uint64 // ns, cumulative
+	capacity             map[string]int
+	mem                  uint64 // bytes
+}
+
+func parseFdinfo(s string) fdinfo {
+	fi := fdinfo{engines: map[string]uint64{}, capacity: map[string]int{}}
+	var resident, memory uint64
+	for _, line := range strings.Split(s, "\n") {
+		k, v, ok := strings.Cut(line, ":")
+		if !ok || !strings.HasPrefix(k, "drm-") {
+			continue
+		}
+		v = strings.TrimSpace(v)
+		switch {
+		case k == "drm-driver":
+			fi.driver = v
+		case k == "drm-pdev":
+			fi.pdev = v
+		case k == "drm-client-id":
+			fi.client = v
+		case strings.HasPrefix(k, "drm-engine-capacity-"):
+			fi.capacity[k[len("drm-engine-capacity-"):]], _ = strconv.Atoi(v)
+		case strings.HasPrefix(k, "drm-engine-"):
+			n, _ := strconv.ParseUint(strings.TrimSuffix(v, " ns"), 10, 64)
+			fi.engines[k[len("drm-engine-"):]] = n
+		case strings.HasPrefix(k, "drm-resident-"):
+			resident += kib(v)
+		case strings.HasPrefix(k, "drm-memory-"):
+			memory += kib(v)
+		}
+	}
+	fi.mem = resident
+	if fi.mem == 0 {
+		fi.mem = memory
+	}
+	return fi
+}
+
+// kib parses "52624 KiB" (or MiB, or a bare count of bytes) to bytes.
+func kib(v string) uint64 {
+	f := strings.Fields(v)
+	if len(f) == 0 {
+		return 0
+	}
+	n, _ := strconv.ParseUint(f[0], 10, 64)
+	if len(f) > 1 {
+		switch f[1] {
+		case "KiB":
+			n <<= 10
+		case "MiB":
+			n <<= 20
+		case "GiB":
+			n <<= 30
+		}
+	}
+	return n
+}
+
+// drmScan walks /proc for every fd open on /dev/dri/*, one entry per client per process.
+// Run as root it sees everyone; as a user, only that user's processes.
+func drmScan() (out []drmFD, complete bool) {
+	ents, err := os.ReadDir("/proc")
+	if err != nil {
+		return []drmFD{}, false
+	}
+	complete = true
+	out = []drmFD{}
+	for _, e := range ents {
+		pid, err := strconv.Atoi(e.Name())
+		if err != nil {
+			continue
+		}
+		fds, err := os.ReadDir("/proc/" + e.Name() + "/fd")
+		if err != nil {
+			if !os.IsNotExist(err) {
+				complete = false
+			}
+			continue
+		}
+		seen := map[string]bool{}
+		comm := ""
+		for _, fd := range fds {
+			target, err := os.Readlink("/proc/" + e.Name() + "/fd/" + fd.Name())
+			if err != nil || !strings.HasPrefix(target, "/dev/dri/") {
+				continue
+			}
+			info := readStr("/proc/" + e.Name() + "/fdinfo/" + fd.Name())
+			_, rest, ok := strings.Cut(info, "drm-client-id:")
+			if !ok {
+				continue
+			}
+			id := strings.TrimSpace(strings.SplitN(rest, "\n", 2)[0])
+			if seen[id] {
+				continue
+			}
+			seen[id] = true
+			if comm == "" {
+				comm = strings.TrimSpace(readStr("/proc/" + e.Name() + "/comm"))
+			}
+			out = append(out, drmFD{PID: pid, Comm: comm, Info: info})
+		}
+	}
+	sort.SliceStable(out, func(i, j int) bool { return out[i].PID < out[j].PID })
+	return out, complete
+}
+
+// scanClients walks in-process when that sees everything (root), else through the helper.
+func (c *Collector) scanClients() ([]drmFD, bool) {
+	if priv.Mode() == "root" || !priv.Available() {
+		return drmScan()
+	}
+	if c.gpu.helper == nil {
+		c.gpu.helper = &drmHelper{}
+	}
+	if fds, ok := c.gpu.helper.scan(); ok {
+		return fds, true
+	}
+	return drmScan() // the helper is down: our own processes are better than nothing
+}
+
+// drmHelper is the root child that answers "scan" requests. One line in, one JSON line out.
+type drmHelper struct {
+	mu       sync.Mutex
+	cmd      *exec.Cmd
+	in       io.WriteCloser
+	out      *bufio.Reader
+	failedAt time.Time
+}
+
+const helperRetry = 30 * time.Second
+
+func (h *drmHelper) scan() ([]drmFD, bool) {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	if h.cmd == nil {
+		if !h.failedAt.IsZero() && time.Since(h.failedAt) < helperRetry {
+			return nil, false
+		}
+		if !h.start() {
+			h.failedAt = time.Now()
+			return nil, false
+		}
+	}
+	// a stuck child must not stall the sampler: it is killed, and tried again later
+	cmd := h.cmd
+	stop := time.AfterFunc(5*time.Second, func() { cmd.Process.Kill() })
+	defer stop.Stop()
+	if _, err := io.WriteString(h.in, "\n"); err != nil {
+		h.fail()
+		return nil, false
+	}
+	line, err := h.out.ReadBytes('\n')
+	if err != nil {
+		h.fail()
+		return nil, false
+	}
+	var fds []drmFD
+	if json.Unmarshal(line, &fds) != nil {
+		h.fail()
+		return nil, false
+	}
+	if fds == nil {
+		fds = []drmFD{}
+	}
+	return fds, true
+}
+
+func (h *drmHelper) start() bool {
+	self, err := os.Executable()
+	if err != nil {
+		return false
+	}
+	full := priv.Wrap(self, "drm-clients")
+	cmd := exec.Command(full[0], full[1:]...)
+	in, err := cmd.StdinPipe()
+	if err != nil {
+		return false
+	}
+	out, err := cmd.StdoutPipe()
+	if err != nil {
+		return false
+	}
+	if err := cmd.Start(); err != nil {
+		return false
+	}
+	h.cmd, h.in, h.out = cmd, in, bufio.NewReaderSize(out, 1<<20)
+	return true
+}
+
+func (h *drmHelper) fail() {
+	if h.cmd != nil {
+		h.in.Close()
+		h.cmd.Process.Kill()
+		go h.cmd.Wait()
+	}
+	h.cmd, h.in, h.out = nil, nil, nil
+	h.failedAt = time.Now()
+}
+
+// DRMClientsHelper is `hamlab drm-clients`: for every line read on stdin, one JSON line of
+// drmFD on stdout. Ends when stdin does — when the daemon that started it has gone.
+func DRMClientsHelper(stdin io.Reader, stdout io.Writer) int {
+	sc := bufio.NewScanner(stdin)
+	w := bufio.NewWriter(stdout)
+	for sc.Scan() {
+		fds, _ := drmScan()
+		b, _ := json.Marshal(fds)
+		w.Write(b)
+		w.WriteByte('\n')
+		if w.Flush() != nil {
+			return 1
+		}
+	}
+	return 0
+}
+
+// ---- small helpers ----
+
+func readF(p string) float64 {
+	v, err := strconv.ParseFloat(strings.TrimSpace(readStr(p)), 64)
+	if err != nil {
+		return 0
+	}
+	return v
+}
+
+func readLink(p string) string {
+	t, err := os.Readlink(p)
+	if err != nil {
+		return ""
+	}
+	return t
+}
+
+func ueventValue(p, key string) string {
+	for _, line := range strings.Split(readStr(p), "\n") {
+		if k, v, ok := strings.Cut(line, "="); ok && k == key {
+			return strings.TrimSpace(v)
+		}
+	}
+	return ""
+}

internal/metrics/gpu_test.go

@@ -0,0 +1,136 @@
+package metrics
+
+import (
+	"encoding/json"
+	"os"
+	"testing"
+	"time"
+)
+
+const fdinfoA = `pos:	0
+flags:	02100002
+drm-driver:	i915
+drm-client-id:	544
+drm-pdev:	0000:00:02.0
+drm-total-system0:	52624 KiB
+drm-resident-system0:	50140 KiB
+drm-resident-stolen-system0:	32 KiB
+drm-engine-render:	1000000000 ns
+drm-engine-copy:	0 ns
+drm-engine-video:	2000000000 ns
+drm-engine-capacity-video:	2
+drm-engine-video-enhance:	0 ns
+`
+
+func TestParseFdinfo(t *testing.T) {
+	fi := parseFdinfo(fdinfoA)
+	if fi.driver != "i915" || fi.pdev != "0000:00:02.0" || fi.client != "544" {
+		t.Fatalf("identity: %+v", fi)
+	}
+	if fi.engines["video"] != 2000000000 || fi.capacity["video"] != 2 {
+		t.Errorf("video engine: %+v", fi)
+	}
+	if fi.mem != (50140+32)<<10 {
+		t.Errorf("resident = %d", fi.mem)
+	}
+}
+
+func TestLspciName(t *testing.T) {
+	for in, want := range map[string]string{
+		"Intel Corporation DG2 [Arc A380] (rev 05)":                                              "Intel Arc A380",
+		"Intel Corporation RocketLake-S GT1 [UHD Graphics 750] (rev 04)":                         "Intel UHD Graphics 750",
+		"Intel Corporation Alder Lake-N [UHD Graphics]":                                          "Intel UHD Graphics",
+		"Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c7)": "AMD Radeon RX 6600/6600 XT/6600M",
+		"NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)":                    "NVIDIA GeForce RTX 3060 Lite Hash Rate",
+		"Moore Threads Technology Co.,Ltd MTT S80":                                               "Moore Threads Technology Co.,Ltd MTT S80",
+	} {
+		if got := lspciName(in); got != want {
+			t.Errorf("%q → %q, want %q", in, got, want)
+		}
+	}
+}
+
+// Two scans a second apart: a client that ran its render engine for half of it and both
+// video engines flat out reads 50% render, 100% video, and the device's engines follow.
+func TestGPUClientsShare(t *testing.T) {
+	c := NewCollector()
+	d := &gpuDev{card: "/nonexistent/card9", dev: "/nonexistent", key: "0000:00:02.0", bus: "0000:00:02.0", driver: "i915", name: "Test GPU", integrated: true}
+	c.gpu.devs = []*gpuDev{d}
+	c.gpu.listedAt = time.Now().Add(time.Hour) // keep the fake list
+	c.gpu.clients = map[string]clientPrev{}
+	t0 := time.Now()
+	c.gpu.last = []drmFD{{PID: 7, Comm: "ffmpeg", Info: fdinfoA}}
+	c.gpu.scanAt = t0 // no walk: the fake list stands
+	first := c.gpus(t0, 40)
+	if len(first) != 1 || len(first[0].Clients) != 1 || first[0].Clients[0].Usage != 0 {
+		t.Fatalf("first pass: %+v", first)
+	}
+	if first[0].Temp != 40 || first[0].TempSrc != "package" {
+		t.Errorf("an integrated GPU wears the package temperature: %+v", first[0])
+	}
+	later := fdinfoA
+	later = replace(later, "drm-engine-render:	1000000000 ns", "drm-engine-render:	1500000000 ns")
+	later = replace(later, "drm-engine-video:	2000000000 ns", "drm-engine-video:	4000000000 ns")
+	c.gpu.last = []drmFD{{PID: 7, Comm: "ffmpeg", Info: later}}
+	c.gpu.scanAt = t0.Add(time.Second)
+	got := c.gpus(t0.Add(time.Second), 40)
+	cl := got[0].Clients[0]
+	if cl.Engines["render"] != 50 || cl.Engines["video"] != 100 || cl.Usage != 100 {
+		t.Errorf("client share: %+v", cl)
+	}
+	if got[0].Engines["video"] != 100 || got[0].Usage != 100 || got[0].UsageSrc != "clients" {
+		t.Errorf("device from clients: %+v", got[0])
+	}
+	if got[0].MemUsed != cl.Mem {
+		t.Errorf("resident memory should be the clients' sum: %d vs %d", got[0].MemUsed, cl.Mem)
+	}
+	// the Processes table joins the client to its row
+	c.ring = []Sample{{T: 1, GPUs: got}}
+	c.prevProc = nil
+	table := c.Processes()
+	if len(table.GPUs) != 1 {
+		t.Errorf("table carries the GPUs: %+v", table.GPUs)
+	}
+}
+
+func TestFlattenGPU(t *testing.T) {
+	s := Sample{T: 5, GPUs: []GPU{{Name: "Intel Arc A380", Driver: "i915", Bus: "0000:03:00.0", Usage: 12, UsageSrc: "idle", Engines: map[string]float64{"video": 30}, PowerW: 9.5, PowerCapW: 55, MHz: 2450, MaxMHz: 2450, Clients: []GPUClient{{PID: 1}}}}}
+	got := map[string]any{}
+	for _, x := range flatten(s, nil) {
+		got[x.ID] = x.Value
+	}
+	for id, want := range map[string]any{"gpu.intel_arc_a380.usage": 12.0, "gpu.intel_arc_a380.video": 30.0, "gpu.intel_arc_a380.power": 9.5, "gpu.intel_arc_a380.freq": 2450.0, "gpu.intel_arc_a380.clients": 1.0} {
+		if got[id] != want {
+			t.Errorf("%s = %v, want %v", id, got[id], want)
+		}
+	}
+	if _, ok := got["gpu.intel_arc_a380.temp"]; ok {
+		t.Error("no temperature sensor, no temperature")
+	}
+}
+
+func replace(s, old, new string) string {
+	for i := 0; i+len(old) <= len(s); i++ {
+		if s[i:i+len(old)] == old {
+			return s[:i] + new + s[i+len(old):]
+		}
+	}
+	return s
+}
+
+// TestLiveGPU reads this machine's GPUs for real: HAMLAB_LIVE=1 go test -run TestLiveGPU -v ./internal/metrics
+func TestLiveGPU(t *testing.T) {
+	if os.Getenv("HAMLAB_LIVE") == "" {
+		t.Skip("set HAMLAB_LIVE=1 to read the real hardware")
+	}
+	c := NewCollector()
+	c.sample()
+	time.Sleep(1100 * time.Millisecond)
+	c.sample()
+	s, _ := c.Latest()
+	b, _ := json.MarshalIndent(s.GPUs, "", "  ")
+	t.Logf("%s", b)
+	for _, x := range flatten(s, func(id string) bool { return len(id) > 4 && id[:4] == "gpu." }) {
+		t.Logf("%s = %v %s", x.ID, x.Value, x.Unit)
+	}
+}

internal/metrics/metrics.go

@@ -123,6 +123,7 @@ type Proc struct {
 	Cmd     string  `json:"cmd,omitempty"`
 	User    string  `json:"user"`
 	CPU     float64 `json:"cpu"`           // percent of one core
+	GPU     float64 `json:"gpu,omitempty"` // its busiest GPU engine, percent; summed over the GPUs it has open
 	CPUTime float64 `json:"cpu_time"`      // seconds, lifetime
 	RSS     uint64  `json:"rss"`           // bytes
 	Threads int     `json:"threads"`
@@ -145,6 +146,7 @@ type Sample struct {
 	Disks    []Disk     `json:"disks"`
 	FS       []FS       `json:"fs"`
 	UPS      *UPS       `json:"ups"`
+	GPUs     []GPU      `json:"gpus"`
 	Uptime   float64    `json:"uptime"`
 
 	fsT, upsT int64 // when the slow sections were really read; sensors report these, not T
@@ -173,6 +175,7 @@ type Collector struct {
 	procAt    time.Time
 	lastTable *ProcTable
 	upsProbe  func() *UPS // nil when no UPS daemon is installed
+	gpu       gpuState
 
 	subMu sync.Mutex
 	subs  map[chan struct{}]struct{} // told after every sample; see Subscribe
@@ -356,6 +359,7 @@ func (c *Collector) sample() {
 		c.lastUPS, c.upsAt = c.upsProbe(), now
 	}
 	s.UPS, s.upsT = c.lastUPS, c.upsAt.UnixMilli()
+	s.GPUs = c.gpus(now, cpuTemp(s.Temps))
 
 	if first {
 		return // deltas are meaningless on the priming pass
@@ -1112,6 +1116,7 @@ type ProcTable struct {
 	Mem       Mem        `json:"mem"`
 	Swap      Mem        `json:"swap"`
 	Net       []Net      `json:"net"`
+	GPUs      []GPU      `json:"gpus"`
 	Uptime    float64    `json:"uptime"`
 	Procs     []Proc     `json:"procs"`
 	Total     int        `json:"total"`
@@ -1274,6 +1279,20 @@ func (c *Collector) Processes() ProcTable {
 		}
 	}
 	c.prevProc, c.procAt = cur, now
+	// what each process is doing on the GPUs, from the newest sample's client lists
+	gpuBy := map[int]float64{}
+	gpus := []GPU{}
+	if n := len(c.ring); n > 0 && c.ring[n-1].GPUs != nil {
+		gpus = c.ring[n-1].GPUs
+		for _, g := range gpus {
+			for _, cl := range g.Clients {
+				gpuBy[cl.PID] += cl.Usage
+			}
+		}
+	}
+	for i := range procs {
+		procs[i].GPU = round1(gpuBy[procs[i].PID])
+	}
 	sort.Slice(procs, func(i, j int) bool {
 		if procs[i].CPU != procs[j].CPU {
 			return procs[i].CPU > procs[j].CPU
@@ -1281,7 +1300,7 @@ func (c *Collector) Processes() ProcTable {
 		return procs[i].RSS > procs[j].RSS
 	})
 	t := ProcTable{T: now.UnixMilli(), Procs: procs, Total: len(procs), Uptime: uptime(), Load: loadavg(),
-		CPUModel: cpuModel, CPUMHz: round1(cpuMHz()), CPUMaxMHz: cpuMaxMHz, CPUHist: []float64{}, Net: []Net{}}
+		CPUModel: cpuModel, CPUMHz: round1(cpuMHz()), CPUMaxMHz: cpuMaxMHz, CPUHist: []float64{}, Net: []Net{}, GPUs: gpus}
 	for _, p := range procs {
 		if p.State == "R" {
 			t.Running++

internal/metrics/sensors.go

@@ -131,6 +131,48 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
 		f.add(p+".used_pct", x.Mount+" used", "usage", "%", ratio(x.Used, x.Size), s.fsT, pct, lbl...)
 	}
 
+	for _, g := range s.GPUs {
+		p := "gpu." + sensors.Seg(g.Name)
+		lbl := []string{"name", g.Name, "driver", g.Driver}
+		if g.Bus != "" {
+			lbl = append(lbl, "bus", g.Bus)
+		}
+		if g.Integrated {
+			lbl = append(lbl, "integrated", "1")
+		}
+		if g.UsageSrc != "" {
+			f.add(p+".usage", g.Name, "usage", "%", g.Usage, t, pct, lbl...)
+		}
+		for _, e := range []string{"render", "video", "copy", "video-enhance", "compute"} {
+			if v, ok := g.Engines[e]; ok {
+				f.add(p+"."+sensors.Seg(e), g.Name+" "+e, "usage", "%", v, t, pct, lbl...)
+			}
+		}
+		if g.Temp > 0 {
+			f.add(p+".temp", g.Name+" temperature", "temperature", "°C", g.Temp, t, none, append(lbl, "source", g.TempSrc)...)
+		}
+		if g.PowerW > 0 || g.PowerCapW > 0 {
+			b := none
+			if g.PowerCapW > 0 {
+				b = upTo(g.PowerCapW)
+			}
+			f.add(p+".power", g.Name+" power", "power", "W", g.PowerW, t, b, lbl...)
+		}
+		if g.FanRPM > 0 {
+			f.add(p+".fan", g.Name+" fan", "fan", "rpm", float64(g.FanRPM), t, none, lbl...)
+		}
+		if g.MaxMHz > 0 {
+			f.add(p+".freq", g.Name+" clock", "frequency", "MHz", g.MHz, t, upTo(g.MaxMHz), lbl...)
+		}
+		if g.MemTotal > 0 {
+			f.add(p+".mem_used", g.Name+" memory used", "bytes", "B", float64(g.MemUsed), t, upTo(float64(g.MemTotal)), lbl...)
+			f.add(p+".mem_total", g.Name+" memory", "bytes", "B", float64(g.MemTotal), t, none, lbl...)
+		} else if g.MemUsed > 0 {
+			f.add(p+".mem_used", g.Name+" memory used", "bytes", "B", float64(g.MemUsed), t, none, lbl...)
+		}
+		f.add(p+".clients", g.Name+" clients", "count", "", float64(len(g.Clients)), t, none, lbl...)
+	}
+
 	if u := s.UPS; u != nil {
 		ut := s.upsT
 		f.add("ups.battery", "UPS battery", "usage", "%", u.BatteryPct, ut, pct, "model", u.Model)

web/src/Top.tsx

@@ -4,7 +4,7 @@
 // /api/procs for the table — so this works on any machine the console can see, whether or not
 // btop is installed on it.
 import { createContext, memo, useCallback, useContext, useEffect, useMemo, useRef, useState } from 'react'
-import { api, ProcTable, Proc, Sample } from './api'
+import { api, GPU, ProcTable, Proc, Sample } from './api'
 import { Catalog, MONITOR_SHAPE, Values, mergeValues, sampleOf, stale } from './sampleAdapter'
 import { fetchCatalog, fetchHistories, useSensorStream } from './sensors'
 import { fmtMs, nextPoint, REFRESH_OPTIONS, setRefreshMs, strideMs, useHistorySec, useRefreshMs } from './prefs'
@@ -186,7 +186,7 @@ function braille(values: number[], cells = 7, max = 100): string {
 }
 
 // ---- the curves this page keeps ----
-const CURVES = 'cpu.usage,cpu.core.*.usage,net.*.rx,net.*.tx,disk.*.util'
+const CURVES = 'cpu.usage,cpu.core.*.usage,net.*.rx,net.*.tx,disk.*.util,gpu.*.usage'
 
 function useCurves(live: Values | null, t: number, refreshMs: number, historySec: number, resumed: number) {
   const cur = useRef<Map<string, number[]>>(new Map())
@@ -216,7 +216,7 @@ function useCurves(live: Values | null, t: number, refreshMs: number, historySec
     const n = binN.current
     for (const [id, v] of Object.entries(live)) {
       if (typeof v !== 'number') continue
-      if (!/^cpu\.usage$|^cpu\.core\.\d+\.usage$|^net\..+\.(rx|tx)$|^disk\..+\.util$/.test(id)) continue
+      if (!/^cpu\.usage$|^cpu\.core\.\d+\.usage$|^net\..+\.(rx|tx)$|^disk\..+\.util$|^gpu\..+\.usage$/.test(id)) continue
       const arr = cur.current.get(id) ?? []
       if (!cur.current.has(id)) cur.current.set(id, arr)
       if (fresh || arr.length === 0) { arr.push(v); if (arr.length > MAX) arr.shift() }
@@ -280,7 +280,7 @@ export default function Top() {
   // how many points the history window holds at this rate: the width of every graph
   const cap = Math.max(2, Math.min(MAX, Math.round((historySec * 1000) / strideMs(historySec, refreshMs))))
 
-  const [sort, setSort] = useState<'cpu' | 'mem' | 'pid' | 'name'>('cpu')
+  const [sort, setSort] = useState<'cpu' | 'gpu' | 'mem' | 'pid' | 'name'>('cpu')
   const [rev, setRev] = useState(false)
   const [filter, setFilter] = useState('')
   const [sel, setSel] = useState<number | null>(null)
@@ -303,8 +303,8 @@ export default function Top() {
     if (!pt) return []
     const f = filter.trim().toLowerCase()
     const list = pt.procs.filter((p) => !f || p.name.toLowerCase().includes(f) || (p.cmd ?? '').toLowerCase().includes(f) || p.user.toLowerCase().includes(f) || String(p.pid) === f)
-    const key = (p: Proc) => (sort === 'cpu' ? p.cpu : sort === 'mem' ? p.rss : sort === 'pid' ? p.pid : 0)
+    const key = (p: Proc) => (sort === 'cpu' ? p.cpu : sort === 'gpu' ? (p.gpu ?? 0) : sort === 'mem' ? p.rss : sort === 'pid' ? p.pid : 0)
-    list.sort((a, b) => (sort === 'name' ? a.name.localeCompare(b.name) : key(b) - key(a)))
+    list.sort((a, b) => (sort === 'name' ? a.name.localeCompare(b.name) : key(b) - key(a) || b.cpu - a.cpu))
     return rev ? list.reverse() : list
   }, [pt, filter, sort, rev])
 
@@ -342,10 +342,16 @@ export default function Top() {
   const pct = (a: number, b: number) => (b > 0 ? (a / b) * 100 : 0)
   // the top of a graph's scale: the peak of what it is drawing — the window, not everything kept
   const peak = (id: string) => Math.max(1, ...(curves.get(id) ?? []).slice(-cap))
+  // the GPUs, with each one's sensor stem: the same rule the server uses, so two identical
+  // cards are gpu.<name> and gpu.<name>_2 here as there
+  const gpus = (pt?.gpus ?? []).map((g, i, all) => {
+    const seg = segOf(g.name), n = all.slice(0, i).filter((o) => segOf(o.name) === seg).length
+    return { g, stem: 'gpu.' + seg + (n ? `_${n + 1}` : '') }
+  })
 
   return (
     <Pal.Provider value={pal}>
-    <div className="top" style={pal.vars as React.CSSProperties}>
+    <div className={'top' + (gpus.length ? ' has-gpu' : '')} style={pal.vars as React.CSSProperties}>
       <Box n={1} name="cpu" className="top-cpu" right={<>
         <span className="t-clock">{clock.toLocaleTimeString([], { hour12: false })}</span>
         <span className="t-rate">
@@ -383,6 +389,16 @@ export default function Top() {
         </div>
       </Box>
 
+      {gpus.length > 0 && (
+        <div className="top-gpus">
+          {gpus.map(({ g, stem }, i) => (
+            <Box key={stem} name={`gpu${i}`} className="top-gpu" right={<span className="t-gpu-name" title={[g.name, g.driver, g.bus].filter(Boolean).join(' · ')}>{g.name}</span>}>
+              <GPUBox g={g} values={curves.get(stem + '.usage') ?? []} tick={tick} advance={advance} cap={cap} />
+            </Box>
+          ))}
+        </div>
+      )}
+
       <Box n={2} name="mem" className="top-mem">
         <div className="t-row"><b>Total:</b><span className="spacer" />{bytes(mem.total)}</div>
         {([['Used', mem.used, 'used'], ['Available', mem.avail, 'avail'], ['Cached', cached, 'cached'], ['Free', free, 'free']] as const).map(([k, v, g]) => (
@@ -445,17 +461,17 @@ export default function Top() {
 
       <Box n={5} name="proc" className="top-proc" right={<>
         <span className="t-sorts">
-          {(['pid', 'name', 'mem', 'cpu'] as const).map((k) => <button key={k} className={sort === k ? 'on' : ''} onClick={() => setSort(k)}>{k}</button>)}
+          {(['pid', 'name', 'mem', ...(gpus.length ? ['gpu' as const] : []), 'cpu'] as const).map((k) => <button key={k} className={sort === k ? 'on' : ''} onClick={() => setSort(k)}>{k}</button>)}
           <button className={rev ? 'on' : ''} onClick={() => setRev(!rev)}>reverse</button>
         </span>
       </>}>
-        <div className="t-proc">
+        <div className={'t-proc' + (gpus.length ? ' has-gpu' : '')}>
           <input className="t-filter" value={filter} onChange={(e) => setFilter(e.target.value)} placeholder="filter" spellCheck={false} autoCapitalize="off" autoCorrect="off" />
           <div className="t-thead">
             <span className="c-pid">Pid:</span><span className="c-name">Program:</span><span className="c-cmd">Command:</span>
-            <span className="c-thr">Threads:</span><span className="c-user">User:</span><span className="c-mem">MemB</span><span className="c-g" /><span className="c-cpu">Cpu%</span>
+            <span className="c-thr">Threads:</span><span className="c-user">User:</span><span className="c-mem">MemB</span><span className="c-g" />{gpus.length > 0 && <span className="c-gpu">Gpu%</span>}<span className="c-cpu">Cpu%</span>
           </div>
-          <Rows procs={procs} sel={sel} onSel={setSel} curves={pcurve} empty={err || 'no process matches'} />
+          <Rows procs={procs} sel={sel} onSel={setSel} curves={pcurve} gpu={gpus.length > 0} empty={err || 'no process matches'} />
           <div className="t-foot">
             {sel !== null && procs.find((p) => p.pid === sel) ? (() => { const p = procs.find((x) => x.pid === sel)!; return <>
               <span className="t-dim">{p.pid} {p.name}</span><span className="spacer" />
@@ -473,7 +489,7 @@ export default function Top() {
 
 // The rows are the heaviest thing on the page — hundreds of them — so they are redrawn when the
 // table is read again, not on every tick of the curves above them.
-const Rows = memo(function Rows({ procs, sel, onSel, curves, empty }: { procs: Proc[]; sel: number | null; onSel: (p: number | null) => void; curves: React.RefObject<Map<number, number[]>>; empty: string }) {
+const Rows = memo(function Rows({ procs, sel, onSel, curves, gpu, empty }: { procs: Proc[]; sel: number | null; onSel: (p: number | null) => void; curves: React.RefObject<Map<number, number[]>>; gpu: boolean; empty: string }) {
   return (
     <div className="t-rows">
       {procs.map((p) => {
@@ -487,6 +503,7 @@ const Rows = memo(function Rows({ procs, sel, onSel, curves, empty }: { procs: P
             <span className="c-user">{p.user}</span>
             <span className="c-mem">{shortBytes(p.rss)}</span>
             <span className="c-g">{braille(c, 7, Math.max(5, ...(c.length ? c : [0])))}</span>
+            {gpu && <span className={'c-gpu' + (p.gpu ? ' lit' : '')}>{p.gpu ? p.gpu.toFixed(1) : ''}</span>}
             <span className="c-cpu">{p.cpu.toFixed(1)}</span>
           </div>
         )
@@ -496,6 +513,61 @@ const Rows = memo(function Rows({ procs, sel, onSel, curves, empty }: { procs: P
   )
 })
 
+// One GPU, laid out like the cpu box: the busy curve on the left; on the right the device's
+// meters — busy, each engine, temperature — then clock, power and memory, then the processes
+// that have it open with their share. What cannot be measured on this card is left out rather
+// than shown as zero: an Arc on i915 has no temperature sensor, Asahi has no busy counter.
+const ENGINES: [string, string][] = [['render', 'render'], ['compute', 'compute'], ['video', 'video'], ['video-enhance', 'enhance'], ['copy', 'copy']]
+function GPUBox({ g, values, tick, advance, cap }: { g: GPU; values: number[]; tick: number; advance: number; cap: number }) {
+  const engines = ENGINES.filter(([k]) => g.engines && g.engines[k] !== undefined)
+  const temp = g.temp !== undefined && g.temp > 0 ? g.temp : undefined
+  const how = g.usage_src === 'idle' ? 'busy, from idle residency' : g.usage_src === 'busy' ? 'busy, as the driver reports it' : g.usage_src === 'clients' ? 'the busiest engine, summed over its clients' : 'this driver exposes no activity counter'
+  return (
+    <div className="t-gpu">
+      <div className="t-gpu-graph">
+        <Graph values={values} tick={tick} advance={advance} cap={cap} max={100} grad="cpu" />
+        <div className="t-up">{g.usage_src ? `${Math.round(g.usage)}% ` : ''}{how}</div>
+      </div>
+      <div className="t-gpu-stats">
+        <div className="t-cores-head"><b>{g.name}</b><span className="spacer" />{g.driver}{g.integrated ? ' · integrated' : ''}</div>
+        <div className="t-core">
+          <span className="t-core-n">GPU</span>
+          <Meter pct={g.usage_src ? g.usage : 0} grad="cpu" />
+          <span className="t-core-p">{g.usage_src ? `${Math.round(g.usage)}%` : '—'}</span>
+          <span className="t-core-t">{temp !== undefined && <Meter pct={Math.min(100, temp)} grad="temp" />}</span>
+          <span className="t-core-c" title={temp !== undefined ? (g.temp_src === 'package' ? 'the CPU package: an integrated GPU shares the die, and this is its only sensor' : "the card's own sensor") : 'no temperature sensor on this device'}>{temp !== undefined ? `${Math.round(temp)}°C${g.temp_src === 'package' ? '*' : ''}` : ''}</span>
+        </div>
+        {engines.map(([k, label]) => (
+          <div className="t-core" key={k}>
+            <span className="t-core-n t-eng">{label}</span>
+            <Meter pct={g.engines![k]} grad="cpu" />
+            <span className="t-core-p">{Math.round(g.engines![k])}%</span>
+            <span className="t-core-t" /><span className="t-core-c" />
+          </div>
+        ))}
+        <div className="t-gpu-facts">
+          {(g.mhz !== undefined || g.max_mhz) ? <div className="t-row t-sub"><span>Clock</span><span className="spacer" />
+            <span className="t-fact" title={g.req_mhz ? `${Math.round(g.req_mhz)} MHz requested${g.max_mhz ? `, ${Math.round(g.max_mhz)} MHz max` : ''}` : undefined}>{Math.round(g.mhz ?? 0)}{g.max_mhz ? ` / ${Math.round(g.max_mhz)}` : ''} MHz</span></div> : null}
+          {(g.power_w !== undefined && g.power_w > 0) || g.power_cap_w ? <div className="t-row t-sub"><span>Power</span><span className="spacer" />
+            <span className="t-fact">{(g.power_w ?? 0).toFixed(1)}{g.power_cap_w ? ` / ${Math.round(g.power_cap_w)}` : ''} W</span></div> : null}
+          {g.fan_rpm ? <div className="t-row t-sub"><span>Fan</span><span className="spacer" /><span className="t-fact">{g.fan_rpm} rpm</span></div> : null}
+          {g.mem_total ? <div className="t-row t-sub"><span>Mem</span><span className="t-pct">{Math.round(((g.mem_used ?? 0) / g.mem_total) * 100)}%</span><Meter pct={((g.mem_used ?? 0) / g.mem_total) * 100} grad="used" /><span className="t-amt">{bytes(g.mem_used ?? 0, 1)}</span></div>
+            : g.mem_used ? <div className="t-row t-sub"><span>Mem</span><span className="spacer" /><span className="t-fact" title="resident memory of the processes that have it open">{bytes(g.mem_used, 1)}</span></div> : null}
+        </div>
+        <div className="t-gpu-clients">
+          <div className="t-thead"><span className="c-pid">Pid:</span><span className="c-name">Program:</span><span className="c-mem">MemB</span><span className="c-cpu">Gpu%</span></div>
+          {g.clients.map((c) => (
+            <div className="t-crow" key={c.pid} title={c.engines ? Object.entries(c.engines).map(([k, v]) => `${k} ${v.toFixed(1)}%`).join('  ') : undefined}>
+              <span className="c-pid">{c.pid}</span><span className="c-name">{c.name}</span><span className="c-mem">{c.mem ? shortBytes(c.mem) : ''}</span><span className="c-cpu">{c.usage.toFixed(1)}</span>
+            </div>
+          ))}
+          {g.clients.length === 0 && <div className="t-dim t-crow-none">{g.clients_ok ? 'no process has it open' : 'other users’ processes are hidden: the console has no root here'}</div>}
+        </div>
+      </div>
+    </div>
+  )
+}
+
 // One physical drive. `used` is the bytes of the filesystems on it, left out when none can
 // be attributed to it — a pool member's share of a pool is not a number worth inventing.
 function Disk({ name, note, size, used, io, temp, mounts, pool }: { name: string; note?: string; size: number; used?: number; io?: number; temp?: number; mounts?: string[]; pool?: string }) {

web/src/api.ts

@@ -72,9 +72,20 @@ export interface NetIf { iface: string; up: boolean; rx_bps: number; tx_bps: num
 export interface Disk { name: string; model?: string; size: number; read_bps: number; write_bps: number; util_pct: number; temp?: number }
 export interface FS { mount: string; type: string; size: number; used: number; remote?: boolean; disk?: string }
 export interface UPS { driver: string; model: string; state: string; source: string; on_battery: boolean; in_v: number; out_v: number; battery_pct: number; runtime_min: number; load_w: number; load_pct: number; last_event?: string }
-export interface Proc { pid: number; ppid: number; name: string; cmd?: string; user: string; cpu: number; cpu_time: number; rss: number; threads: number; state: string; nice: number; started: number }
+export interface Proc { pid: number; ppid: number; name: string; cmd?: string; user: string; cpu: number; gpu?: number; cpu_time: number; rss: number; threads: number; state: string; nice: number; started: number }
 export interface MemInfo { total: number; used: number; avail: number; free?: number; cached?: number; buffers?: number }
-export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_temp?: number; core_ids?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; uptime: number; procs: Proc[]; total: number; running: number }
+// A graphics device: usage_src says what measures it ("idle" residency, the driver's own "busy", or
+// the busiest "clients" engine) and is absent when nothing does; temp_src is "gpu" for the device's own
+// sensor or "package" for the CPU die an integrated GPU shares.
+export interface GPUClient { pid: number; name: string; usage: number; engines?: Record<string, number>; mem?: number }
+export interface GPU {
+  name: string; driver: string; bus?: string; integrated?: boolean
+  usage: number; usage_src?: 'idle' | 'busy' | 'clients'; engines?: Record<string, number>
+  temp?: number; temp_src?: 'gpu' | 'package'; power_w?: number; power_cap_w?: number; fan_rpm?: number
+  mhz?: number; req_mhz?: number; max_mhz?: number; mem_used?: number; mem_total?: number
+  clients: GPUClient[]; clients_ok: boolean
+}
+export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_temp?: number; core_ids?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
 export interface Sample {
   t: number; interval_ms: number; cpu: number; cores: number[]; load: [number, number, number]
   mem: { total: number; used: number; avail: number }; swap: { total: number; used: number; avail: number }

web/src/sampleAdapter.ts

@@ -8,7 +8,7 @@ export type Values = Record<string, number | boolean | string>
 export type Catalog = Map<string, Sensor>
 
 // Everything the modules draw, as one stream query.
-export const MONITOR_SHAPE = ['cpu.*', 'load.*', 'uptime', 'hamlab.*', 'mem.*', 'swap.*', 'temp.*', 'fan.*', 'net.*', 'disk.*', 'fs.*', 'ups.*']
+export const MONITOR_SHAPE = ['cpu.*', 'load.*', 'uptime', 'hamlab.*', 'mem.*', 'swap.*', 'temp.*', 'fan.*', 'net.*', 'disk.*', 'fs.*', 'ups.*', 'gpu.*']
 // The stream answers a list with a list; the modules want one flat map.
 export function mergeValues(parts: unknown[]): Values {
   const out: Values = {}

web/src/styles.css

@@ -699,16 +699,22 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
 .t-prow .c-user { text-overflow: ellipsis; }
 .c-pid, .c-thr, .c-mem, .c-cpu { text-align: right; } .t-thead .c-pid, .t-thead .c-name, .t-thead .c-cmd, .t-thead .c-user { text-align: left; }
 .c-g { color: var(--muted); text-align: right; }
+/* the Gpu% column, on a machine that has one: dim at rest, lit on the rows that are using it */
+.c-gpu { text-align: right; color: var(--muted); } .c-gpu.lit { color: var(--t-magenta); }
+.t-proc.has-gpu .t-thead, .t-proc.has-gpu .t-prow { grid-template-columns: 58px 116px minmax(0, 1fr) 62px 58px 50px 56px 42px 42px; }
 @container tproc (max-width: 640px) {
   .t-thead, .t-prow { grid-template-columns: 58px 116px minmax(0, 1fr) 62px 58px 50px 42px; }
+  .t-proc.has-gpu .t-thead, .t-proc.has-gpu .t-prow { grid-template-columns: 58px 116px minmax(0, 1fr) 62px 58px 50px 42px 42px; }
   .t-thead .c-g, .t-prow .c-g { display: none; }
 }
 @container tproc (max-width: 560px) {
   .t-thead, .t-prow { grid-template-columns: 58px 110px minmax(0, 1fr) 58px 50px 42px; }
+  .t-proc.has-gpu .t-thead, .t-proc.has-gpu .t-prow { grid-template-columns: 58px 110px minmax(0, 1fr) 58px 50px 42px 42px; }
   .t-thead .c-thr, .t-prow .c-thr { display: none; }
 }
 @container tproc (max-width: 470px) {
   .t-thead, .t-prow { grid-template-columns: 52px minmax(0, 1fr) 52px 48px 40px; gap: 6px; }
+  .t-proc.has-gpu .t-thead, .t-proc.has-gpu .t-prow { grid-template-columns: 52px minmax(0, 1fr) 52px 48px 40px 40px; }
   .t-thead .c-cmd, .t-prow .c-cmd { display: none; }
 }
 .t-foot { display: flex; align-items: center; gap: 8px; border-top: 1px solid var(--border); padding-top: 3px; margin-top: 2px; font-size: 12px; }
@@ -721,10 +727,14 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
   .top-mem, .top-disks { min-height: 220px; }
   .top-cpu { grid-column: 1 / -1; grid-row: 1; } .top-mem { grid-column: 1; grid-row: 2; } .top-disks { grid-column: 2; grid-row: 2; }
   .top-net { grid-column: 1 / -1; grid-row: 3; } .top-proc { grid-column: 1 / -1; grid-row: 4; }
+  .top.has-gpu { grid-template-rows: minmax(230px, auto) auto minmax(220px, auto) minmax(200px, auto) minmax(420px, 1fr); }
+  .top.has-gpu .top-mem, .top.has-gpu .top-disks { grid-row: 3; } .top.has-gpu .top-net { grid-row: 4; } .top.has-gpu .top-proc { grid-row: 5; }
 }
 @media (max-width: 800px) {
   .top { display: flex; flex-direction: column; font-size: 12px; }
   .top > .tbox { flex: 0 0 auto; }
+  .top-gpus { display: flex; flex-direction: column; flex: 0 0 auto; }
+  .t-gpu { flex-direction: column; } .t-gpu-graph { height: 110px; flex: 0 0 110px; } .t-gpu-stats { width: auto; max-height: none; }
   .t-cpu { flex-direction: column; } .t-cpu-graph { height: 130px; flex: 0 0 130px; } .t-cores { width: auto; max-height: 40vh; }
   .t-net { flex-direction: column; } .t-net-nums { width: auto; } .t-net-graphs { flex: 0 0 130px; }
   .top-proc { height: 70vh; min-height: 420px; }