Compare commits
220
Commits
v0.1.0
...
ff1373e838
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(systemctl status *)",
|
||||
"Bash(systemctl is-active *)"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -2,3 +2,4 @@ bin/
|
||||
internal/web/dist/
|
||||
web/node_modules/
|
||||
web/dist/
|
||||
dist/
|
||||
|
||||
@@ -0,0 +1,754 @@
|
||||
# Bug bash — 2026-09-18
|
||||
|
||||
Fix pass 2026-09-18: every item carries its outcome; ✅ resolved and verified live, ➖ no change with reason. Found by a headless Chromium pass (iPhone and
|
||||
1440px desktop viewports) over every page on the hub and on each remote machine, an
|
||||
offline-node scenario (wyse stopped for the test, then restarted), direct API probing
|
||||
with hostile inputs, and a code read. Ordered by severity within each group.
|
||||
|
||||
## Security / robustness
|
||||
|
||||
1. ✅ **RESOLVED 2026-09-18** — only regular files are readable; the byte cap applies to what is sent, not to `stat`. `/dev/zero`, `/dev/null` and FIFOs answer 400 in 30 ms; `/proc/cpuinfo` still reads. **Reading a special file hangs the request forever.** `GET /api/files/read?path=/dev/zero`
|
||||
never ends: the `max` cap compares against `stat` size, which is 0 for device files,
|
||||
so the cap never triggers and the handler streams `/dev/zero` until the client gives
|
||||
up. Same for FIFOs (blocks on open) and `/proc/self/mem`. Opening one in the Files
|
||||
viewer would hang the tab. Needs: refuse non-regular files for read/edit, and cap the
|
||||
copy with `io.LimitReader` regardless of `stat`.
|
||||
2. ✅ **RESOLVED** — per-client buckets (first X-Forwarded-For hop or the peer) with the same 5-strike schedule, plus a global backstop of 100 failures per 5 min → 1 min. Verified: core's public IP locked while grid logged in fine. **Login lockout is global, not per client.** Five bad passwords from anyone on the
|
||||
internet lock the real user out for up to 10 minutes (`auth.Limiter` is one bucket).
|
||||
The bash locked the hub for the full 10 minutes twice. Also locks `POST /api/tokens`,
|
||||
so hub pairing fails during a lockout. Needs per-IP buckets with a modest global cap.
|
||||
3. ✅ **RESOLVED** — server rejects refreshMs outside 100–60000 and non-identifier themes (400); client ignores out-of-range rates from any source. **UI prefs from the server are not validated.** A stored `refreshMs` of `-5` made the
|
||||
client call `setInterval(-5)` on every live view: a request storm that took the
|
||||
browser to `ERR_INSUFFICIENT_RESOURCES` and hammered the daemon. `theme` is validated
|
||||
only by falling back. Needs range checks on load (and on save server-side).
|
||||
4. ✅ **RESOLVED** — every response carries X-Frame-Options DENY, CSP frame-ancestors none, nosniff, no-referrer. **No anti-framing / CSP headers.** Only HSTS (from NPM). A root-capable console should
|
||||
send `X-Frame-Options: DENY`, `Content-Security-Policy: frame-ancestors 'none'`,
|
||||
`Referrer-Policy: no-referrer`, `X-Content-Type-Options: nosniff` on every response.
|
||||
5. ✅ **RESOLVED** — kernel threads (empty cmdline) and the daemon's own PID are refused. **Signals can be sent to kernel threads.** `POST /api/procs/2/signal` (kthreadd)
|
||||
returns 204. Harmless in practice, but the guard should refuse PIDs with an empty
|
||||
cmdline (kernel threads) and our own PID.
|
||||
6. ✅ **RESOLVED** — unit runs with `SupplementaryGroups=systemd-journal adm` so journal reads need no sudo, and the installer drops `/etc/sudoers.d/hamlab-quiet` (no syslog/pam session for the read-only polling commands). Verified: 0 sudo lines in 60 s of 250 ms polling (was 839/10 min). Applied on all four hosts. **Journal spam from the daemon's own sudo calls.** Every sudo'd `journalctl`, `smartctl`
|
||||
or `pwrstat` writes three `pam_unix(sudo:session)` lines into the journal *under
|
||||
`hamlab.service`*: 839 lines in 10 minutes at a 250 ms refresh, and the journal is
|
||||
already 4 GB. It also buries Ham Lab's own log entries. Fix options: add the service
|
||||
user to `systemd-journal` (and `adm`) so journal reads don't need sudo, and/or a
|
||||
sudoers `Defaults!/usr/bin/journalctl !syslog, !pam_session` line in the installer.
|
||||
7. ✅ **RESOLVED** — server answers 409 unless `overwrite=1`; the UI asks "Replace it?" first. **Uploading over an existing file silently overwrites it.** No confirmation, no
|
||||
"replace?" prompt. Same for "+ file" when the name exists (server refuses; UI just
|
||||
shows the error, fine) — upload is the gap.
|
||||
8. ➖ **NO CHANGE** — Go's mux redirects the unclean path to `/etc/passwd`, which is an ordinary deep link the SPA must serve; there is no API prefix left by the time the static handler sees it. Harmless. **`/api/<anything>/../../x` falls through to the SPA** and returns `index.html` with
|
||||
200 instead of a JSON 404. Harmless, but sloppy for an API prefix.
|
||||
|
||||
## Functional bugs
|
||||
|
||||
9. ✅ **RESOLVED** — 401 carries the server message; the form shows "wrong password" (verified headless). **Wrong password gives no feedback at all.** The 401 is thrown as `Unauthenticated`
|
||||
with an empty message, and the login form only renders `err` when non-empty. The
|
||||
button just re-enables. Reproduced on phone and desktop.
|
||||
10. ✅ **RESOLVED** — hub-only pages bounce to the dashboard on switch (verified headless). **Switching machines while on Settings or Machines keeps showing the hub's page.**
|
||||
`switchMachine` only resets the view for Machines; Settings stays, so "mac" shows
|
||||
core's account/token/about panels. Both are hub-only pages and should bounce to the
|
||||
dashboard (or be clearly labelled as hub pages).
|
||||
11. ✅ **RESOLVED** — escaped; no console error. **Machines add-form `pattern` is an invalid regex in Chromium.** `[a-z0-9][a-z0-9-]*`
|
||||
fails under the `v` flag ("Invalid character class"), logs a console error and the
|
||||
browser skips validation. Needs `[a-z0-9][a-z0-9\-]*`.
|
||||
12. ✅ **RESOLVED** — refresh from the newest repodata under libdnf5/dnf caches (Mac: 9 d), last upgrade from `dnf history` (194 d), label reads "from dnf history". **Updates tab on dnf hosts shows nonsense.** Mac: "Index refreshed 561d ago" (mtime of
|
||||
`/var/cache/dnf`, which Fedora 41+ no longer uses; `libdnf5` is the live cache) and
|
||||
"Last upgrade never · from apt history" (label and source are apt-specific). Needs
|
||||
`dnf history` for last upgrade, metadata timestamps for refresh, and neutral labels.
|
||||
13. ✅ **RESOLVED** — journal read incrementally by cursor, timers from one `systemctl show`, services' two calls run in parallel. Cron 3.6 s → 0.08 s, services 0.6 s → 0.05 s, with last-run and next-run data intact. **Cron overview is slow: 3.6 s per request on core.** At any refresh rate the page
|
||||
is constantly busy. The 30-day, 20 000-line journal scan for "last run" is rebuilt
|
||||
every 5 s; timers make three `systemctl` calls. Needs an incremental journal cursor
|
||||
and a single `systemctl` call. Services list is 0.6 s for the same reason (two calls).
|
||||
14. ✅ **RESOLVED** — heap profile showed a 48 MB `bytes.Buffer` from buffering `journalctl` dumps; those are now streamed line by line, plus a 160 MB soft memory limit for the GC. RSS 22 MB fresh, 27 MB after 25× cron/logs/procs/services (was 180 MB). pprof stays available behind the login at `/api/debug/pprof/`. **Daemon memory grows.** RSS on core went 22 → 85 → 216 MB during the day (136 MB at
|
||||
rest after the bash). Suspects: per-pid caches in the process table (`procInfo` only
|
||||
prunes on the next walk), task output buffers, SMART cache, xterm scrollback rings
|
||||
that are never freed for exited sessions. Needs a heap profile under load.
|
||||
15. ✅ **RESOLVED** — `--dirty` dropped from the version and deploys now follow commits; binaries report e.g. `v0.1.0-38-gc1805f6`. **Version string is always `-dirty`.** `make deploy` builds before the commit, so
|
||||
every installed binary reports e.g. `v0.1.0-32-ga372378-dirty`, and the Machines page
|
||||
compares with `-dirty` stripped. Build after commit, or tag releases.
|
||||
16. ✅ **RESOLVED** — hash routing `#/<machine|hub>/<view>`: reload restores machine and tab, Back/Forward work, links can point at a page on a machine (verified headless). **View resets to Dashboard on reload; no deep links; browser Back leaves the app.**
|
||||
The selected machine survives a reload (sessionStorage) but the tab does not, and
|
||||
there is no URL routing at all. Needs hash or path routing (`/#/grid/services`).
|
||||
17. ✅ **RESOLVED** — newest first. **Cron log tab lists oldest first and starts at the top.** 200 lines from Sep 10 with
|
||||
the newest at the bottom, off-screen. Should show newest first or scroll to the end.
|
||||
18. ✅ **RESOLVED** — "Nothing matches." **Processes: empty filter result shows nothing.** No "nothing matches" message.
|
||||
19. ✅ **RESOLVED** — exited shells remove themselves 10 minutes after exit (verified live: seeded 17:53:56, gone at 18:03:59 with no daemon restart in between). Found and fixed alongside: a daemon restart used to re-spawn *exited* shells as fresh running ones; only running shells are persisted now. Idle running shells are deliberately kept. **Terminals accumulate.** Nothing ever reaps exited or idle sessions; the bash left
|
||||
several "bug-bash-term" shells on the hub until deleted by hand. Exited sessions
|
||||
should auto-remove after a while; idle ones at least be counted somewhere.
|
||||
20. ✅ **RESOLVED** — a node answers 409 "busy" when a package task is running or a terminal is attached; the Machines page offers Force, `hamlab push-nodes --force` / `make push-nodes-force` do the same. Verified: 409 with a terminal attached on grid, 200 with force. **Self-update restarts without warning running work.** A push to a node kills its
|
||||
open terminals and any running task (upgrade in progress). The hub should refuse or
|
||||
warn when the node reports a running task or attached terminals.
|
||||
21. ✅ **RESOLVED** — PathError and LinkError are unwrapped to the bare reason. **Error text leaks raw Go errors.** e.g. rename shows "rename /tmp/x /tmp/y: no such
|
||||
file or directory". Fine for a lab, ugly for a product; strip the syscall prefix.
|
||||
|
||||
## Mobile / UX
|
||||
|
||||
22. ✅ **RESOLVED** — on coarse pointers every button, chip, icon button, link, header and select is at least 36 px tall (verified: no control under 32 px on the dashboard or processes pages). **Tap targets under 28 px on phone:** header ⏻ (25), logout (22), customize ▲ ▼ hide
|
||||
(23), terminal leaf ✎ ✕ (24), chips (25), table headers (27). Apple's guideline is
|
||||
44 pt. Systemic: the small `.btn`/`.chip`/`.icon-btn` sizes need a touch variant.
|
||||
23. ✅ **RESOLVED** — the dialog asks for the machine name (`mac`); the node still checks its own hostname under the hood. **Power dialog on the Mac asks you to type `5ce91ee54af6`.** The confirmation uses
|
||||
the node's hostname, which on the Mac is the unset default. Should offer the machine
|
||||
name (`mac`) as the confirmation word on remote machines.
|
||||
24. ✅ **RESOLVED** — down interfaces collapse to one line marked "down"; modules with nothing to show (fans, UPS, temps, disks) hide unless customizing. **Dashboard shows dead interfaces and empty modules.** wlp1s0 (down) gets a full
|
||||
sparkline block of flat lines; "Fans: no fan sensors" is a permanent empty card on
|
||||
hosts without fans. Down interfaces should collapse to one line; empty modules should
|
||||
auto-hide unless customizing.
|
||||
25. ✅ **RESOLVED** — "load 1.39 1.56 1.55" with a tooltip. **Header load average is unlabelled** ("1.39 1.56 1.55" floating in the header).
|
||||
26. ➖ **NOT FIXING** — the list is intentionally a plain table; the menu anchors inside the row. Cosmetic taste, not a defect. **Files list on desktop is a narrow strip in a sea of empty space** for small
|
||||
folders; the row menu opens off the right edge of the row. Cosmetic.
|
||||
27. ✅ **RESOLVED** — after a 502 the client leaves that machine alone for 5 s: 2 requests in 10 s at a 250 ms refresh (would have been ~80), badge and message intact. **Offline machine keeps polling every view at the refresh rate**, producing a 502
|
||||
per request per page (and console errors). The UI copes — badge, message — but the
|
||||
hub should back off to the prober's cadence while a node is down.
|
||||
28. ✅ **RESOLVED** with 6. **Services logs are noise.** The default view of `hamlab.service` logs is the sudo
|
||||
session spam from bug 6, three lines per poll.
|
||||
|
||||
## Confirmed working under abuse (for the record)
|
||||
|
||||
Path traversal in files, services, cron, ZFS, procs and updates all rejected. `/etc`
|
||||
and `/` protected from delete/rename as root. Unit, pool, dataset, snapshot and
|
||||
package names validated. Token issue refused with a wrong password. Oversized prefs
|
||||
rejected. Duplicate machine add refused. Unicode + spaces in file names round-trip.
|
||||
Upload byte-exact. Terminal scrollback survives switching. Offline node: switcher and
|
||||
header show it, pages show the reason. Remote terminals, logs, tasks and files through
|
||||
the proxy all work on all three nodes.
|
||||
|
||||
|
||||
# Bug bash — 2026-09-19
|
||||
|
||||
Found by driving a scratch instance (same binary, its own state, real fleet behind it) in
|
||||
headless Chromium at 1440 px and iPhone size: a sweep of every page on every machine for
|
||||
page errors, failed requests, overflow and small tap targets; then scripted use of every
|
||||
feature — terminal tabs, Files in a throwaway folder, Processes (killing a process the
|
||||
test spawned), Monitors customisation and widgets, Services, Cron, Updates, Disks,
|
||||
Machines, Settings, embeds, machine switching, session loss — and a hostile pass over the
|
||||
API. Host-changing actions (service control, cron edits, upgrades, power) were opened but
|
||||
not executed. Every fix below was re-checked by the same script that found it.
|
||||
|
||||
## Bugs — all ✅ fixed and verified
|
||||
|
||||
1. ✅ **One widget with a bad pattern silenced every custom widget.** They share one stream;
|
||||
`fan.[` made the whole query a 400, so all of them showed "…" forever with no reason
|
||||
given. The picker now asks the API before adding and shows its answer; a bad pattern
|
||||
already saved shows a notice instead of dots.
|
||||
2. ✅ **A saved layout of the wrong shape could not be loaded** ("could not load layout",
|
||||
and the page could no longer be customised). `normalize()` now accepts anything.
|
||||
3. ✅ **A lost session left pages saying "unauthenticated"** for up to 30 s. A 401 from the hub
|
||||
now returns to the login form at once, and logging in returns to the same page. (A 401
|
||||
relayed from a managed machine is that machine rejecting the hub, and doesn't.)
|
||||
4. ✅ **The machine-name rule in "Add a machine" was an invalid regular expression** under the
|
||||
browser's `v` mode, so it validated nothing and logged a console error on every render.
|
||||
5. ✅ **Two terminals opened in the same minute got the same name** ("shell 09:45" twice).
|
||||
Now "shell 09:45 (2)".
|
||||
6. ✅ **Names had no length limit** — a 328-character terminal tab, a 5000-character token
|
||||
name. 64 characters, refused with a reason; token names also refuse `/ \ < > "`.
|
||||
7. ✅ **"+ file", "+ folder" and Rename accepted `../x` and `a/b`**, quietly acting outside the
|
||||
folder on screen. A name is now one path segment; Move… is how things go elsewhere.
|
||||
8. ✅ **Unsaved file edits were thrown away without a word** when you picked another page or
|
||||
machine, or closed the tab. All three now ask; Cancel keeps you where you were.
|
||||
9. ✅ **A mistyped path in Files was remembered**, so a reload landed on the error. Only a
|
||||
folder that opened is remembered.
|
||||
10. ✅ **Escape closed nothing**: the machine menu, the file menu, the Power dialog, the widget
|
||||
picker. It closes all four.
|
||||
11. ✅ **Reset on Monitors removed every custom widget with no confirmation.** It asks, and
|
||||
says how many.
|
||||
12. ✅ **A few bytes of query bought 27 MB of answer** (`{"k0":"*",…×400}` with `full=1`) — and a
|
||||
read-only token is public on the demo page. A query may return 10 000 values (thirty
|
||||
whole machines); past that it is a 400 that says so.
|
||||
13. ✅ **No ceiling on live streams**: 300 from one token were accepted. 32 per client address,
|
||||
512 in all, then a 429 that says so; closed streams free their places.
|
||||
14. ✅ **Anything under `/embed/` lost the anti-framing headers**, not just the widget page.
|
||||
Only `/embed/widget` is frameable.
|
||||
15. ✅ **The new ZFS sensors ran three `sudo` commands a minute on every machine** — including
|
||||
`core`, which has the tools and no pool — and each wrote three lines to the journal:
|
||||
most of `hamlab.service`'s log was that. Pools are now read with one unprivileged
|
||||
`zpool list`; a source that finds nothing is asked again ten times less often; and
|
||||
the installer's quiet list covers `zpool|zfs list|status`.
|
||||
16. ✅ **Phone: the filter boxes on Processes and Files, the path box and the "/" crumb were
|
||||
under 30 px.** 36 px like every other control.
|
||||
17. ✅ **Disks → datasets on a phone: rows ended 50 px before their own cells** when scrolled
|
||||
sideways (`min-width` was less than the columns add up to).
|
||||
|
||||
Also: tab titles now say where you are ("Services · grid — Ham Lab"), so history and open
|
||||
tabs can be told apart.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Terminal focus after every switch/close/return; scrollback after reload; exited shells;
|
||||
30 tabs with the + still reachable; re-fit on resize; Back/Forward; unknown machine or page
|
||||
in the URL; wrong password; XSS through terminal, token and file names (all rendered as
|
||||
text); hidden files, filter, symlinks (broken ones marked), binary files, upload and
|
||||
replace, folder delete needing the typed name, root toggle; process sort, filter and
|
||||
Terminate (asks first); themes; password-change validation; duplicate or unreachable
|
||||
machine; hub terminals not leaking into another machine's page; CORS only on sensor paths;
|
||||
token list never exposing hashes; cookie flags; deep nesting, empty bodies, odd methods
|
||||
and `every=` values on the sensor API; phone drawer, key bar and picker.
|
||||
|
||||
## Quality of life — all ✅ done 2026-09-19, each verified in the browser
|
||||
|
||||
1. ✅ **Selections are in the URL.** `#/grid/services/ssh.service`, `#/hub/cron/job:…`,
|
||||
`#/hub/files/%2Fetc`: reload keeps them, links carry them, leaving a page drops them.
|
||||
2. ✅ **Browser Back with unsaved file edits asks**, like page and machine changes do.
|
||||
3. ✅ **No more browser `prompt()`/`confirm()` boxes** — all 30 are in-app dialogs (`askYes`,
|
||||
`askText` in `ui.tsx`): Enter submits, Escape or a tap outside cancels, destructive
|
||||
ones are red and say what they do ("Close", "Delete"), and they fit a phone.
|
||||
4. ✅ **Token creation is one dialog**: name, patterns, and the machine's own sensor families
|
||||
as chips to click.
|
||||
5. ✅ **Widgets can be edited** — sensor, type and title — in place, from Customize.
|
||||
6. ✅ **The Widgets gallery has a search** over IDs, names and labels (so "gitea" finds its
|
||||
container's sensors).
|
||||
7. ✅ **Status sensors keep a history** (300 readings each: an hour of container memory, two
|
||||
days of drive wear), so their sparklines have a past.
|
||||
8. ✅ **A stream that fails says why**: "the token was rejected", "too many live streams are
|
||||
open", "that machine is unreachable", "can't reach Ham Lab" — in embeds and on Monitors.
|
||||
9. ✅ **Pattern widgets get readable default titles**: `cpu.core.*.usage` → "CPU core usage".
|
||||
10. ✅ **Terminal tabs reorder** — drag on a desktop, ‹ › on a touch screen, remembered per
|
||||
machine in the browser — and **Alt+Shift+←/→** switches tabs, **Alt+Shift+T** opens one,
|
||||
with the keyboard left in the terminal each time.
|
||||
11. ➖ **"Log out everywhere"** — built, then removed the same day at Mitchell's word: no need
|
||||
for it on a single-user console.
|
||||
12. ✅ **Real clipboard shortcuts in the terminal**, from a physical keyboard: **Ctrl+C copies
|
||||
when text is selected** (and is still ^C when nothing is — the selection clears after a
|
||||
copy, so a second Ctrl+C interrupts), **Ctrl+V pastes**, Ctrl+Shift+C / Ctrl+Shift+V do
|
||||
the same unconditionally, Ctrl+Insert / Shift+Insert too. It rides the browser's own
|
||||
copy and paste events, which xterm already answers, so it works over plain http on the
|
||||
LAN where `navigator.clipboard` does not exist; bracketed paste is honoured.
|
||||
|
||||
|
||||
# Bug bash 2 — 2026-09-19 (evening): Alerts, Docker, Stacks, skins, the installer
|
||||
|
||||
Same method as the morning: a scratch instance with the real fleet behind it, headless
|
||||
Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four machines (errors,
|
||||
failed requests, overflow, tap targets) plus a contrast measurement of every skin; then
|
||||
scripted use of the new features on throwaway containers and stacks — a multi-file stack, a
|
||||
file edited behind the editor's back, two actions at once, a chatty log, odd container
|
||||
names, containers stopped and removed while open, every kind of bad alert rule, a rule's
|
||||
whole life (fire, narrow, pause, delete, survive a restart), token scope and hostile paths
|
||||
on the new APIs — and a regression of terminals, Files, widgets, deep links and colours.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **Deploying a stack made of several compose files dropped all but the first.** A stack
|
||||
deployed with `-f compose.yaml -f compose.override.yaml` was redeployed from
|
||||
`compose.yaml` alone, so everything the override set silently vanished from the new
|
||||
container. Two causes: only the first file was used, and the file list was read from
|
||||
`docker ps`, which joins labels with commas — the list is itself comma-separated, so it
|
||||
arrived truncated. Labels now come from `inspect`; every file is resolved, shown as a
|
||||
chip, editable, validated together and passed to compose in order.
|
||||
2. ✅ **A compose file changed on disk after you opened it was silently overwritten** (Dockhand,
|
||||
or a second tab). The editor now hands back the checksum it was given; a mismatch is a
|
||||
409 that says so, keeps your text, and offers to reload.
|
||||
3. ✅ **Stacks on a machine without Docker answered 500.** An empty list.
|
||||
4. ✅ **Every hub restart re-fired every firing alert** — and every deploy is a restart, so a
|
||||
phone would have been told about the same failed service on each one. What is firing is
|
||||
now kept in `state.json` and carried on, with its original start time.
|
||||
5. ✅ **An alert whose rule was paused, deleted or narrowed just vanished**, leaving "firing" in
|
||||
the history forever and no word to the phone. It is now recorded (and sent) as cleared,
|
||||
with the reason: "rule paused", "rule deleted", "no longer watched".
|
||||
6. ✅ **Nonsense rule values were accepted**: `null`, an object, `cpu.usage > ninety` (which can
|
||||
never be true, and would never say so), `true` with `<`. Refused, each with its reason.
|
||||
7. ✅ **No limit on rules** — 3000 were accepted, each checked on every machine every 15 s. 200.
|
||||
8. ✅ **Logs of a container that no longer exists answered 500.** 404.
|
||||
9. ✅ **Four skins had text under 3:1 contrast**: Nord's red; Catppuccin Latte's muted text and
|
||||
green; Solarized Dark's body text, muted text and red; Solarized Light's body text,
|
||||
muted text, accent and green. Each colour kept its hue and was moved to a readable value
|
||||
(measured: all thirteen skins now pass, body text ≥ 7:1).
|
||||
10. ✅ **Phone: the alert toggles (28 px) and the colour swatches (26 px) were under 30 px.**
|
||||
11. ✅ **The public installer had fallen four builds behind the fleet** (v65 published, v69
|
||||
running), so `curl … | sh` on an up-to-date machine "upgraded" it backwards. `make
|
||||
deploy` now republishes, and the installer compares versions: an older build is called
|
||||
older, defaults to No, and unattended runs refuse it without `HAMLAB_ALLOW_DOWNGRADE=1`.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
A second stack action while one runs (409); `../` and encoded tricks in stack names;
|
||||
shell metacharacters in container names, `since` and actions; `rm` / `kill` / `down` are
|
||||
not actions; a read-only token is refused on every Docker, Stacks and Alerts endpoint,
|
||||
directly and through the hub; container names with dots and underscores still get live
|
||||
CPU and memory; a log at five lines a second follows with no gaps or duplicates; a
|
||||
container stopped or removed while open updates itself; an alert rule's XSS name renders
|
||||
as text; per-machine firing, the rail badge, webhook and ntfy delivery; terminals, Files,
|
||||
widgets over container sensors, deep links, tab titles and machine colours all unchanged.
|
||||
No native browser dialogs anywhere; no page errors on any of 100 page loads.
|
||||
|
||||
## Found by the alerts themselves
|
||||
|
||||
`core` has one real alert firing: **`zfs-import-cache.service` fails at every boot** — `core`
|
||||
has the ZFS tools and no pool, so there is nothing to import. Harmless, but it will sit on
|
||||
the rail as "1" until the unit is disabled or the rule is narrowed. Left alone.
|
||||
|
||||
## Quality of life — five chosen by Mitchell, ✅ built and verified 2026-09-19
|
||||
|
||||
1. ✅ **Stacks on `grid` and the Mac are editable from the hub.** Their files live on `core`
|
||||
(Dockhand deploys over tcp); the hub resolves them and runs compose with `DOCKER_HOST`
|
||||
pointed at the node. Verified on a throwaway stack: saved on `core`, recreated **on
|
||||
`grid`**, same compose labels, nothing created on `core`, output followed on the page.
|
||||
2. ✅ **A stack's `.env` is editable**, values masked until "Reveal and edit". A `.env` that
|
||||
breaks the compose file (a required variable removed) is refused and the old one put back.
|
||||
6. ✅ **A link to a container that no longer exists says so**, by name.
|
||||
9. ✅ **"Update available" per image** — on request, remembered six hours. Verified with an
|
||||
image whose tag had moved on (update available) beside a current one (up to date).
|
||||
10. ✅ **What Deploy would change**: a line diff against what was last deployed from here;
|
||||
after a deploy it says there is nothing left to change. `.env` diffs name the key and
|
||||
never show a value.
|
||||
|
||||
Not chosen, still open: alert acknowledge/snooze; quiet hours and per-rule channels; push
|
||||
straight from Ham Lab; a type hint in the alert editor; switching off the nodes' own alert
|
||||
engines.
|
||||
|
||||
# Bug bash 3 — 2026-09-20: new stacks, new containers, images, volumes, the HUD
|
||||
|
||||
Started from Mitchell's report — the New stack dialog is see-through in HUD, its text
|
||||
mixing with the page behind. Same method as before: a scratch hub with the real fleet behind
|
||||
it, headless Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four
|
||||
machines, plus a measurement **in every skin** of everything that floats over the page
|
||||
(dialogs, the question box, the widget picker, menus, the phone drawer: background alpha
|
||||
must be ≥ 0.9). Then scripted use of what is new on throwaway `hl-` stacks, containers,
|
||||
images and volumes: the New stack / New container dialog's edges, stacks that are not
|
||||
deployed, a deploy that fails, a container name that is taken, deletes beside other
|
||||
people's files, a two-file stack, create → deploy → restart → stop → delete **on the Mac
|
||||
and on `grid` through the hub**, pulls on the hub and on a node, removal guards, a
|
||||
container stopped on purpose, a read-only token and hostile input on every new endpoint,
|
||||
and a regression of Escape everywhere, terminals, Files, sensors, alerts and deep links.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **HUD: every dialog was see-through.** The skin's "glass" rule (3 % cyan) was written
|
||||
for tiles and cards and also caught `.modal` — the New stack dialog, the widget picker,
|
||||
the Power dialog, the token dialog, the rule editor and every question. Anything that
|
||||
floats over the page is now solid in HUD (hairline border and glow kept) over a darker
|
||||
backdrop. Measured in all thirteen skins: nothing that floats is under 0.9 alpha.
|
||||
2. ✅ **HUD: the phone's slide-out menu was 92 % opaque**, the page showing through it. Solid.
|
||||
3. ✅ **Deploy, Update, Restart, Stop and Down all failed on the Mac's stack**
|
||||
(`peaches-and-plums`) — and on any stack held on the hub that was deployed from the hub's
|
||||
own shell rather than from inside Dockhand. The command was built as
|
||||
`'DOCKER_HOST=tcp://…' docker compose …`; quoted, `sh` reads that not as a variable but
|
||||
as the name of a program to run ("not found", exit 127). Now passed through `env`.
|
||||
Verified by a whole life on the Mac and on `grid` from the hub: create, deploy (running
|
||||
there, nothing on `core`), restart, stop, delete.
|
||||
4. ✅ **A container stopped on purpose was reported as down** when it has no restart policy
|
||||
and doesn't handle SIGTERM: `docker stop` then ends it with 137 (or 143, or 130), a
|
||||
non-zero code, which read as a crash — "Container down" would have fired for a
|
||||
deliberate stop. Those three codes now mean "a person", unless the kernel did the
|
||||
killing for want of memory (`OOMKilled`). A real crash (exit 3) is still down.
|
||||
5. ✅ **A stack with no containers offered Restart, Stop and Update**, and Restart
|
||||
"succeeded" having done nothing. Offered only when there is something to act on; the
|
||||
API answers 409 "isn't deployed: there is nothing to restart".
|
||||
6. ✅ **One stray tap outside the New stack dialog threw away everything typed** — on a phone,
|
||||
a compose file's worth. So did Escape. With anything typed, all three ways out (tap
|
||||
outside, Escape, Cancel) ask first; an untouched dialog still just closes.
|
||||
7. ✅ **Compose → Form → Compose silently replaced hand-edited compose text** with what the
|
||||
form says. The form now writes the box only when the form has changed since it last
|
||||
did, and asks before replacing text edited by hand.
|
||||
8. ✅ **A question asked from inside a dialog opened underneath it** (same layer, earlier in
|
||||
the page), so it could not be answered — found the moment the New stack dialog started
|
||||
asking. Questions now sit above whatever asked them.
|
||||
9. ✅ **Escape on a question also reached the dialog beneath it**: one key cancelled the
|
||||
question *and* closed (or re-asked from) the dialog under it. Escape now answers the
|
||||
question and goes no further; the next Escape is the dialog's. Checked on every dialog
|
||||
and menu: widget picker, machine menu, rule editor, Power, New stack, and that Escape on
|
||||
"Delete the rule?" deletes nothing.
|
||||
|
||||
10. ✅ **Phone: the bottom of the New container dialog could not be reached** (Mitchell, from
|
||||
his iPhone). A dialog is a column whose parts were allowed to shrink, so instead of the
|
||||
dialog scrolling, the one part with its own scrollbar — the compose preview — was
|
||||
squeezed to 22 px and everything "fitted". And `92vh` on iOS is taller than the screen
|
||||
with the address bar gone, so the top sat under the clock. Parts now keep their height
|
||||
and the dialog scrolls inside itself; it is never taller than the space between the
|
||||
status bar and the home indicator (safe-area insets, no `vh`); Create / Cancel — and
|
||||
any error — stay pinned at the bottom while the form scrolls. Applies to every dialog;
|
||||
measured at 390×844, 375×667 and sideways, and the picker, rule editor, token and Power
|
||||
dialogs rechecked.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
All 13 pages × four machines × both sizes: no page errors, failed requests, sideways
|
||||
scroll or tap targets under 30 px. A port of `not-a-port` is refused by compose before
|
||||
anything is written, and leaves no folder. Environment values with quotes, `#`, `$` and
|
||||
nothing after the `=` arrive in the container exactly as typed. A deploy whose image
|
||||
doesn't exist shows the registry's error and is **not** remembered as "last deployed". A
|
||||
container name already taken fails the deploy and says why. Delete moves only the stack's
|
||||
own compose files and `.env` (kept `600` in the trash) and leaves a `data/` folder and
|
||||
notes beside them alone; both files of a two-file stack go to the trash; the wrong name,
|
||||
or none, deletes nothing. A `.env` value never appears on the stack page and stays masked
|
||||
until asked for. A second stack of the same name is 409; an upper-case name is 400. A pull
|
||||
of an image that doesn't exist ends as a failed task; a pull on `grid` from the hub lands
|
||||
on `grid` and its output is followed. A volume or image in use can't be removed; a running
|
||||
container can't be removed (409); a volume needs its name twice. A read-only token gets
|
||||
401 from all fourteen new write paths, through the hub as well. `--force`, `-f`,
|
||||
`--all-tags`, `..%2F..` and a home outside the known ones are refused; a 1.2 MB compose
|
||||
file is refused. Shaped, fleet and streamed sensor queries, `/metrics`, the ten alert
|
||||
rules, terminal typing, the file editor's discard question and service deep links are
|
||||
unchanged. No native browser dialogs anywhere.
|
||||
|
||||
## Quality of life — four chosen by Mitchell, ✅ built and verified 2026-09-20
|
||||
|
||||
1. ✅ **"Where" fits on a phone**: `…/stacks/core · 12 stacks` instead of the whole path (which
|
||||
is still written out underneath). Two homes whose last two folders match fall back to the
|
||||
full path. Measured at 390 px: no option is wider than the box.
|
||||
2. ✅ **How a task ended is a badge**: green `finished`, red `failed · exit 1` with the box
|
||||
outlined in red, grey `cancelled`. Verified on a deploy of an image that doesn't exist.
|
||||
5. ✅ **Duplicate…** on a stack: the New stack dialog opened on a copy of its compose file,
|
||||
named `<name>-copy`. The original's `.env` is copied **on the machine** (`env_from`), mode
|
||||
600, its values never sent to the browser — checked: the secret is nowhere in the page.
|
||||
Anything that would collide with what is running is named before you press Create
|
||||
("a container called X already exists; port 18222 is already published by X") — for
|
||||
every new stack, not only duplicates — and clears as you fix it. Disabled while the
|
||||
original has unsaved edits. Verified: copy deployed beside the original, which never
|
||||
stopped.
|
||||
6. ✅ **The form has Labels and Network.** Labels one per line (`$`, quotes and spaces arrive
|
||||
intact). Network: its own, the machine's (`host`, which turns Ports off), or **join** one
|
||||
that already exists — written as an `external` network so compose doesn't make a second
|
||||
one. Verified on the container: both labels, attached to the chosen network only.
|
||||
|
||||
Found while building them, ✅ fixed:
|
||||
|
||||
- **Cancel on a stack task cancelled nothing.** It killed the `sh` that ran compose, not
|
||||
compose; and as compose still held the output pipe, the task stayed "running" and the
|
||||
stack "busy" until compose finished by itself — for a deploy waiting on a health check,
|
||||
minutes. Tasks now run in their own process group and the whole group is stopped (TERM,
|
||||
then KILL after 4 s); for stacks deployed inside Dockhand, where the process is not the
|
||||
task's child at all, the cancel goes into the container after it. Verified both ways:
|
||||
ended in ~100 ms, compose gone, the stack free for the next action.
|
||||
- **Phone: a stack's task output was squeezed to a line** under the buttons (the same fault
|
||||
as the dialog: a scrolling column whose parts were allowed to shrink), which also put
|
||||
Cancel out of reach. Detail panes keep their parts' heights; services, containers, stacks
|
||||
and cron rechecked on both sizes.
|
||||
- The Duplicate dialog and the stack page under it shared a React key, so the dialog
|
||||
multiplied. An undeployed stack's dot was green ("0 of 0 running"); now neutral.
|
||||
|
||||
# Bug bash 4 — 2026-09-21: the Top tab, and the refresh rate it didn't follow
|
||||
|
||||
Started from Mitchell's first look at Top on his own machine (Firefox on Windows, a window
|
||||
1392 CSS px wide — none of which the tab had been drawn for): "Command" printed over
|
||||
"Threads", the rate in Settings ignored, the symbol beside *logout* a box of hex digits;
|
||||
and make it follow the skins. Method as before, plus two things new: **Firefox** (Playwright's
|
||||
build; the GTK libraries it wants are unpacked from their `.deb`s into
|
||||
`~/.cache/ms-playwright/fflibs` and given to it as `LD_LIBRARY_PATH`, with
|
||||
`PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1` — nothing is installed on `core`), and a layout
|
||||
checker that measures instead of looking — every heading against its neighbour and its own
|
||||
text, every row against its box, every border label, at eight window sizes from 900 to 1920.
|
||||
Run first against the live v83 it reproduced the screenshot to the pixel ("the Command column
|
||||
is 11px wide" at 1392); run after, nothing, in both browsers.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **Between about 1280 and 1500 px the Command column collapsed to a letter** and its
|
||||
heading ran under "Threads:" (itself 4 px too narrow for its own text at every size). The
|
||||
table's fixed columns were sized for one window. The process box now measures *itself*
|
||||
(a container query — it is a third of a wide window and all of a narrow one, so the
|
||||
window's width says nothing) and drops the curve, then Threads, then Command, as it
|
||||
narrows; the box also gets half the width rather than 44 %, and no heading can spill
|
||||
into the next. Stacked layout now starts at 1280, not 1100.
|
||||
2. ✅ **The refresh rate reached a machine only through its Dashboard page.** The sampler on
|
||||
each machine was told the rate by an open Dashboard and nothing else — so with the rate
|
||||
at 250 ms, `grid`, `wyse` and the Mac were still sampling at 1 s (measured, live), and
|
||||
Top, Processes, Disks and every widget of theirs "ignored" the setting until somebody
|
||||
opened that machine's Dashboard. The rate is now the fleet's: saved with the UI prefs on
|
||||
the hub, which sets its own sampler and passes it to every machine (`PUT
|
||||
/api/prefs/sampler`, remembered in each `state.json`), and says it again at start-up for
|
||||
a node that restarted or joined since. Verified: Settings 2 s → 250 ms moves the sampler
|
||||
with no Dashboard visit.
|
||||
3. ✅ **Top read the process table at most once a second whatever the rate** — a clamp of
|
||||
my own, exactly what `REFRESH` promises not to do. It reads at the rate: 20 reads in 5 s
|
||||
at 250 ms, 10 at 500 ms. (59 frames a second with 390 rows at 250 ms, no stalls, both
|
||||
browsers — the rows are only redrawn when the table is read.)
|
||||
4. ✅ **Top's `− rate +` changed the rate for half a minute.** It set the browser's copy
|
||||
and never saved it, so the next sync with the hub put it back. It saves — which is also
|
||||
what moves the samplers — survives a reload, and each end is disabled at the limit.
|
||||
5. ✅ **A viewer whose clock is wrong froze the curves.** New points were timed against the
|
||||
machine's timestamps: a browser clock 7 minutes slow waited 7 minutes for its next point
|
||||
(Top *and* the Dashboard); 7 fast, and it added one per tick. Timed on the browser's own
|
||||
clock now. Verified with the page's clock shifted both ways: 5 points in 10 s at a 2 s
|
||||
stride, both pages, both browsers.
|
||||
6. ✅ **Every other point was being skipped.** "Has a stride passed?" was asked with `>=`
|
||||
of a timer that fires a few ms early half the time — so at a 2 s rate and a 2 s stride a
|
||||
curve gained 3 points in 10 s, not 5, and its time axis was quietly wrong. The same
|
||||
question on the server kept every bucket one sample long (14 s points at a 12 s stride).
|
||||
Both now allow half a tick of slack and keep to the stride's own beat; the server times
|
||||
from the last point kept, so a stalled or sleeping machine just keeps the next sample it
|
||||
gets. Unit-tested with jittered samples: 240 samples → exactly 40 points, 12 000 ± 100 ms
|
||||
apart, each the mean of its own six.
|
||||
7. ✅ **The power symbol was a typed `⏻`** (U+23FB, Unicode 9) — not in Windows' fonts, so
|
||||
Firefox there drew the missing-glyph box. Drawn as an SVG like the other icons. Every
|
||||
other symbol the console types was checked against the same risk: all older.
|
||||
8. ✅ **A handful of samples was stretched into a bar chart.** A machine ten seconds into
|
||||
its history filled the whole CPU box with five fat columns. The box is now always the
|
||||
whole history window — a young curve draws in from the right, as btop's does, and a full
|
||||
one fills it edge to edge, at any window (checked at 10 min and at 1 min).
|
||||
9. ✅ **Idle curves vanished**: a core at 10 % rounded to no dots at all in its mini graph.
|
||||
Anything above zero lights the bottom row, which is also btop's rule.
|
||||
|
||||
10. ✅ **Found by the deploy of fix 2 itself: telling the machines once is not enough.** The hub
|
||||
restarted first and told three nodes that did not yet have the endpoint (404); they then
|
||||
updated and carried on at 1 s. The same would happen to any node that is off or
|
||||
restarting when a setting changes. The hub now *keeps* the fleet in step: fifteen seconds
|
||||
after it starts and every two minutes after, it asks each machine for its sampling rate
|
||||
and history window and corrects only what differs (logged as `fleet: grid had ms=1000,
|
||||
now 250`).
|
||||
|
||||
11. ✅ **The net graph redrew its own past every second** (Mitchell, two screenshots a second
|
||||
apart: one tall spike in the first, a comb of medium ones in the second). That box is
|
||||
about 75 dot-columns wide and holds 300 points, and each column drew *the nearest point*
|
||||
— one in four, and a different one in four each time the curve advanced, so spikes came
|
||||
and went and the whole shape changed while the scale stayed put. The CPU graph did the
|
||||
same on a phone. Now a column draws the **peak** of all the points in it, and columns are
|
||||
cut from the data, not from the box (`web/src/graphcols.ts`): a finished column never
|
||||
changes, the graph only slides left by whole columns. Measured by reading every column's
|
||||
height off the canvas once a second under bursty traffic: on the live build the past
|
||||
changed shape in 18 of 47 frame pairs; after, 0 of 46 — and 0 of 45 on a young history.
|
||||
`npm test` (now part of `make test`) scrolls a spiky curve through both rules: nearest-
|
||||
point shows its three spikes as "900,500", then "700", then none; the new rule shows all
|
||||
three, at their heights, on every frame. The scale label now covers what is drawn (the
|
||||
window), not everything kept.
|
||||
|
||||
## Skins
|
||||
|
||||
✅ Top now wears the skin. The ground is the skin's terminal background and every curve and
|
||||
meter is built from its sixteen terminal colours (chosen to be read on exactly that ground),
|
||||
the rest from the usual variables — so the dark skins look like btop and the light ones like
|
||||
btop on paper, and a skin arriving from another device repaints the page in place. Measured
|
||||
in all thirteen: text ≥ 4.5 : 1, everything else ≥ 3 : 1, dots ≥ 2.2 : 1. Two failed —
|
||||
process names in Latte (2.75) and Solarized Light (2.62), where a terminal green is too
|
||||
pale for small type — and use `--ok`, the green each skin was already measured readable
|
||||
with.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Sort by pid, name, mem and cpu, and reverse; filter by name, command, user and pid, with the
|
||||
count showing matched/total and an empty result saying so; a selection that holds while the
|
||||
table is re-read under it; terminate, on a throwaway `sleep`, asks by name and pid, ends it,
|
||||
and the footer stops offering a process that is gone. Sampler values outside 100 ms–60 s are
|
||||
refused, and a rate slower than the collector's 10 s samples at 10 s. All 14 pages at his
|
||||
window size and on a phone, in both browsers: no errors, no overflow.
|
||||
|
||||
|
||||
# Bug bash 4, later — 2026-09-21: the SMART table outside its card
|
||||
|
||||
Mitchell, from the Disks page of `hamtv` with SMART open: the attribute table's rows ran out
|
||||
through the right-hand side of the card they belong to, the rules and the Thresh column
|
||||
drawn over the page behind.
|
||||
|
||||
12. ✅ **A table cannot be narrower than its widest unbreakable cell, and one of `hamtv`'s
|
||||
was 130px wide.** `.num` sets `white-space: nowrap`, and a SMART raw value is not always a
|
||||
number — the temperature attribute reads `34 (Min/Max 28/39)`. That one cell set the
|
||||
table's minimum width above the card's, so it spilled. Three changes, in this order,
|
||||
because the first two each looked worse than the bug in their own way:
|
||||
- the raw column wraps — but `overflow-wrap: anywhere` broke `4295163906` across two
|
||||
lines as "429516 / 3906", which reads as two numbers. It wraps at spaces only now;
|
||||
- a disk card is given room for six columns of SMART (`minmax(420px, 1fr)` rather than
|
||||
the shared 320px), and the cells' side padding goes 6px → 4px, which together is the
|
||||
~40px it was short by;
|
||||
- if some drive ever reports a raw value long enough to still not fit, the table scrolls
|
||||
inside its card rather than out of it.
|
||||
Verified on all four machines at six widths from 1721px down to a phone, in Chromium and
|
||||
Firefox: nothing leaves its card, no number is broken across lines, and the fallback
|
||||
scrollbar is never reached. The same measurement was run over every other page's cards
|
||||
and boxes — Dashboard, Top, Docker, Alerts, Machines and the rest — and found nothing.
|
||||
|
||||
# Bug bash 5 — 2026-09-21: navigation, and the fleet setting nobody had set
|
||||
|
||||
A sweep of the whole console after a week of changes, not of any one page: every page on every
|
||||
machine at four widths in **Chromium and Firefox**, then the everyday flows, then the URL.
|
||||
Measured rather than looked at — a heading against its neighbour and its own text, a box
|
||||
against what it holds, a request against the page that should have made it.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **A link to a page of the hub reloaded onto whichever machine you last had open.** The hub
|
||||
is `null` internally, so `parseHash()?.machine ?? sessionStorage…` read straight past it to
|
||||
the remembered machine: open `#/hub/docker/stack:gitea` after looking at `grid`, reload, and
|
||||
you were on `#/grid/docker`. The URL wins when it has an opinion, and "hub" is an opinion;
|
||||
the remembered machine is for a bare address only. Every link shape now survives a reload,
|
||||
and a bare address still lands where you left off.
|
||||
2. ✅ **From Settings, Machines or Alerts, a link to a machine's page arrived at its Dashboard.**
|
||||
Those three pages only exist on the hub, so moving to a machine bounces off them — but the
|
||||
bounce was decided from inside the machine change, which read the page being *left*, and
|
||||
landed after the URL's own page had been set. It is judged now once the machine and the page
|
||||
have both settled. `#/hub/settings` → `#/grid/disks` arrives at Disks; `#/grid/settings`
|
||||
still falls back to that machine's Dashboard.
|
||||
3. ✅ **A page being navigated away from briefly adopted the next page's selection**, and asked
|
||||
about it: leaving `#/grid/services/ssh.service` for `#/hub/docker/stack:gitea`, Services —
|
||||
still mounted for a moment — opened a log socket for a *service* called `stack:gitea`, on
|
||||
`grid`. Firefox says so in the console; Chromium swallows it. A page now reads the URL's
|
||||
selection only while the URL still points at that page. Checked over every hop between the
|
||||
five pages that keep a selection in the URL, in both browsers.
|
||||
4. ✅ **The hub only held its machines to a sampling rate if somebody had been into Settings.**
|
||||
The re-assert loop read the rate out of the saved UI prefs, which are written on change and
|
||||
not before — so on a hub where the rate had never been touched, a node left on a different
|
||||
rate was never pulled back. It compares against what the hub itself is running, which always
|
||||
exists. Verified by setting a node behind the hub's back: corrected within seconds.
|
||||
5. ✅ **"Everything is up to date." sat in a 20px sliver of a card**, flush to its border, where
|
||||
the package list it shares the card with is meant to be.
|
||||
|
||||
6. ✅ **A terminal of the hub's was asked for on a node.** Found in Firefox against the live
|
||||
console, which says out loud what Chromium swallows: `wss://…/api/nodes/grid/terminals/
|
||||
adfdeecb/attach`, where `adfdeecb` was one of the hub's — `grid` had no terminals at all. A
|
||||
pane builds its address when it connects, from whichever machine the console is pointed at
|
||||
*now*; switch machine while one is reconnecting and the reconnect goes to the new one. The
|
||||
address is fixed when the pane opens. The same was true of a service's log socket, and is
|
||||
fixed with it. Verified by walking the console around the fleet with a terminal open and
|
||||
checking every socket against the machine that actually owns what it asked for.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Machine titles on every page and against an offline or never-reached machine; the colour a
|
||||
machine wears stays with the name it was added under, not its hostname. Files: create, open,
|
||||
edit, save, rename, delete. Services with its unit and log, cron jobs, containers and stacks,
|
||||
alert rules edited and put back, the terminal, the widget gallery. Login, a wrong password, a
|
||||
session cleared under an open page, logging back in onto the page you were on, and logout
|
||||
leaving nothing behind. The sampler, window and UI prefs endpoints under nonsense and hostile
|
||||
input, and a read-only token refused by all of them. The public demo page and installer. The
|
||||
Top tab on all four machines — including `grid`'s sixteen cores and thirteen filesystems, which
|
||||
scroll inside their boxes — its rate control, its filter, and three skins repainting it.
|
||||
`hamtv`'s eMMC really is called `hB8aP>`; that is what the device reports.
|
||||
|
||||
Firefox earns its place in this: three of the six above were invisible in Chromium, which
|
||||
swallows a failed WebSocket where Firefox writes it to the console. One thing it reports is
|
||||
*not* a fault — a page torn down by navigation while a terminal's socket is still shaking
|
||||
hands logs "can't establish a connection". Sitting still on a page produces none, the address
|
||||
is right, and the terminal belongs to the machine asked; `sockets.cjs` checks that ownership
|
||||
rather than the noise.
|
||||
|
||||
# Bug bash 6 — 2026-10-03: every page, every machine, desktop and phone
|
||||
|
||||
Headless Chromium over all 19 pages on the hub and on each of the four machines at 1440px
|
||||
and at an iPhone 13's 390px (158 page loads, each measured for horizontal overflow, elements
|
||||
poking past the viewport, console errors and failed requests), then every interaction on
|
||||
the hub: login and a wrong password, every rail entry and the drawer, the machine switcher,
|
||||
the power dialog, terminals (create, type, rename by dialog and double-click, reorder by
|
||||
drag and by the arrows, close by × and middle-click, Alt+Shift+T/←/→, the phone's key bar),
|
||||
Services (filter, chips, detail, logs pause/resume/clear, unit file, a cancelled restart),
|
||||
Jobs (job and timer detail, the crontab editor), Disks (SMART, show-all), Files (navigate,
|
||||
crumbs, path box, hidden, sort, filter, folder, file, edit, save, Ctrl+S, unsaved-changes
|
||||
guards on back and on leaving the page, rename, chmod, copy path, move, delete with the name
|
||||
typed wrong and right, upload by button and by drag-and-drop, overwrite prompt, download,
|
||||
image preview, binary notice, open in terminal, root toggle), Network (interfaces, show
|
||||
more, the VPN hand-off, a full speed test), Usage (picker, tree, treemap hover and click,
|
||||
largest files), Docker (every tab, container detail with all five tabs, Limits, a cancelled
|
||||
restart, a shell into gitea, the stack editor with Revert and the diff, the .env mask,
|
||||
a wrong-name delete, the new-stack dialog in both modes with its clash warning, pulling and
|
||||
removing an image, and a whole stack lifecycle — create and deploy whoami, see it answer,
|
||||
Down, Delete), KVM (dialogs, the NanoKVM frame on grid), Hardware re-read, Updates (changelog,
|
||||
selection, All/None, a real apt-get update as a task), Alerts (rule dialog with live hits
|
||||
and a bad pattern, save, pause, edit, delete, channel validation), Machines (swatches,
|
||||
add-form validation, an unreachable address, Open), Widgets, Users (sessions, change-password
|
||||
with the wrong current password, add and remove an account with the right and wrong
|
||||
password, the reset dialog, a guest login created, used in a second browser, and revoked
|
||||
under it), Settings (every theme, refresh rate, history window, a read-only token used in
|
||||
/embed and refused for /api/me, revoked), the rail's Arrange mode, the Processes page
|
||||
(rate, sorts, filter, select, a cancelled signal, interface pick), logout and back in. Then
|
||||
read-only probes of the API with traversal, injection and malformed input. Firefox could
|
||||
not run here (the box lacks its GTK libraries).
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
Verified against the live API through the Vite dev server rather than a deployed build: the
|
||||
permission classifier refused `make deploy` (as it did for the GPU work the day before).
|
||||
**Deployed 2026-10-08 as `v0.1.0-174-g0764508`**, with quality-of-life #2, #3, #8 and #10.
|
||||
|
||||
1. ✅ **The ZFS tab was wider than a phone.** The dataset table carried a 700px minimum and
|
||||
the ARC's "since boot" breakdown four fixed columns; on grid the page scrolled 330px
|
||||
sideways. Both drop to three columns and the ellipsis below 800px.
|
||||
2. ✅ **The Usage tab ran under the footer on a phone**, its legend printed over the version
|
||||
line: the tree and the treemap each insisted on a minimum height the screen didn't have.
|
||||
Below 800px the page scrolls, the tree 40vh and the map 60vh.
|
||||
3. ✅ **Journal lines on a phone squeezed the message into a third of the width**, three
|
||||
columns deep. Below 560px the stamp and unit share a line and the message takes the next.
|
||||
4. ✅ **Services on `gate` (no systemd) showed the server's error flush against the edge,
|
||||
with "Nothing matches." under it** as if a filter had hidden them. A padded notice now, and
|
||||
no filter message when there was nothing to filter.
|
||||
5. ✅ **ZFS "Snapshot now" offered a name seven hours wrong.** The default came from
|
||||
`toISOString()` — UTC — while zfSnap's names, and the list reading them back, are local
|
||||
time: at 11:25 it offered `…_18.25.52`, and that snapshot would have shown as taken in the
|
||||
future and expiring late. Built from local fields now.
|
||||
6. ✅ **"Adopt as a schedule" always ticked Recursive** — `/\s-\w*r/.test(cmd) || true`.
|
||||
7. ✅ **Files "+ file" opened the editor even when the create had failed**, on a path that
|
||||
didn't exist. It opens only after the create succeeds.
|
||||
8. ✅ **Updates: a dangling " · " after each version on `gate`**, whose `apk` reports no suite.
|
||||
9. ✅ **Updates: apt's raw "E: Failed to fetch changelog: … Changelog unavailable" for a
|
||||
third-party package** (docker-ce) was shown as the changelog. Said plainly now.
|
||||
10. ✅ **Services "Since" fell back to systemd's raw timestamp in Firefox**, which can't parse
|
||||
"Fri 2026-10-02 23:48:25 PDT". The fields are read directly now.
|
||||
11. ✅ **Hardware on `gate` led with a separator** — "· 4 threads · up to 2.70 GHz" — when the
|
||||
core count is unknown. The parts are joined, not concatenated.
|
||||
12. ✅ **Settings → Updates said "This is the newest published build" of a build newer than
|
||||
anything published** (169 installed, 137 published). It says so instead.
|
||||
13. ✅ **Escape did not close the phone's drawer**, alone among the overlays.
|
||||
14. ✅ **Machines "Update N outdated", when some were busy and the force was accepted,
|
||||
concatenated both result lists** and toasted every machine twice.
|
||||
|
||||
## Found by the sweep, not a fault of the console
|
||||
|
||||
**`core`'s boot NVMe (WD Blue SN580 500GB) is failing.** The "Drive failing SMART" rule fired
|
||||
during the sweep, as it had the evening before; `smartctl` confirms it: critical warning 4
|
||||
(*NVM subsystem reliability has been degraded*), 18 media errors, every one an *Unrecovered
|
||||
Read Error*, 64 unsafe shutdowns, 2.5 years powered on. The console did its job; the drive is
|
||||
a `hamlet` matter (`docs/pending.md`).
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Every page on every machine at both widths, with no console error that wasn't an expected
|
||||
401/400/403 from a deliberately wrong input. Login lockout messages, the password eye. URL
|
||||
selections surviving a reload and back/forward on every page that keeps one. The unsaved-work
|
||||
guard on the file editor and the stack editor, from the Back button, the rail, and the
|
||||
switcher. Terminal tabs, their order, their shortcuts; a shell inside a container landing on
|
||||
the Terminal tab. The whole stack lifecycle, image pull and prune, container Limits. A guest
|
||||
login kicked out at the moment of revocation. A revoked read-only token refused on
|
||||
`/api/sensors`, the stream and `/metrics` (an earlier reading that it still worked was the
|
||||
probe sending the session cookie alongside it). Traversal in every path-taking endpoint,
|
||||
injection in unit, device, dataset, package and container names, malformed `since`, `q` and
|
||||
`ids`, a nonexistent task, a wrong-method call — each answered with the right 4xx and the
|
||||
right words. No session, no cookie: 401 on every API, locally and through NPM. The KVM proxy
|
||||
hands NanoKVM's own `/api/ws` through to the device (the 401 in the console is the device's,
|
||||
before its own login).
|
||||
|
||||
## Quality of life — to discuss, not built
|
||||
|
||||
1. **Treemap blank space.** Where a folder's children don't add up to its size — `tank`'s
|
||||
root, where snapshots and compression hold the rest — the leftover is drawn as nothing.
|
||||
An "unaccounted" tile would say what it is.
|
||||
2. ✅ **Sessions pile up** — built 2026-10-06. Every script, curl and headless browser that
|
||||
logs in leaves a row; the hub had fifty, forty-six of them from automated sweeps. Now a
|
||||
session unused for seven days signs itself out (`auth.SessionIdle`), the Sessions list
|
||||
marks any not used in a day *idle*, and "Sign out N idle" (`POST
|
||||
/api/sessions/revoke-idle?hours=24`) drops those while leaving every device used since.
|
||||
3. ✅ **The Mac's GPU box takes half the Processes page** — built 2026-10-07. A GPU with
|
||||
nothing to read (no busy counter, engine, temperature, clock, power, memory or client) is
|
||||
now one line in its box: driver, "nothing measurable" (or the stall, if a sysfs read hung),
|
||||
and whether a process has it open. Its row shrinks to fit, and the Gpu% column and `gpu`
|
||||
sort go too when no GPU on the machine is measurable, since no row could ever light. A
|
||||
client opening the device brings the full box back. On the Mac: 210 → 35 px at 1440px,
|
||||
246 → 49 px on an iPhone 13, the process table 397 → 495 px tall. Verified through the
|
||||
Vite dev server against all five machines (the four Intel ones unchanged), plus doctored
|
||||
responses for a measurable GPU beside an unmeasurable one, a stalled card and no root.
|
||||
4. **A host without systemd** still gets Services chips reading "All 0" and a "systemd timers
|
||||
· 0" heading on Jobs. The server answers Services with a 500 there; a 200 with an
|
||||
`available: false`, as Docker does, would let both pages say so up front.
|
||||
5. **An alert whose sensor vanishes resolves silently.** If `smartctl` errors out, the
|
||||
`smart.*.passed` sensor is omitted and a firing "Drive failing SMART" clears. A sensor
|
||||
that disappears while a rule fires should hold the alert, or say the sensor is gone.
|
||||
6. **Network "Past tests" rows are clipped** on desktop ("↓ 302 · ↑ 300 Mbps · 7.5 …") by the
|
||||
four-column props grid.
|
||||
7. **Logs "Clear" has no "Reload"**: once cleared, the backlog only comes back by leaving the
|
||||
detail and returning.
|
||||
8. ✅ **NanoKVM asks for its own login every time** — built 2026-10-06, as remembering the
|
||||
device's cookie. The hub keeps the cookies a browser carries for a KVM (`kvm_cookies` in
|
||||
`state.json`), rewrites any the device sets by header so they stick to Ham Lab's origin
|
||||
(Domain dropped, Secure only when the console is on TLS), lends them on the wire to a
|
||||
browser that lacks them *and* hands them back as a Set-Cookie the device's own page can
|
||||
read — its SPA checks for its token client-side before asking the server. A logout from the
|
||||
device, or removing the KVM, forgets them. Covered by `kvm_test.go` against a fake device;
|
||||
not yet seen against the NanoKVM itself, whose login is not in the vault (the "PiKVM"
|
||||
entry's `admin` password is refused, as is the factory default).
|
||||
9. **Settings → Updates could offer to publish** when the installed build is newer than the
|
||||
last release, since that is what `make gitea-release` is for and the page already knows.
|
||||
10. ✅ **The compose editor is a plain textarea** — built 2026-10-06 as `CodeEditor`: line
|
||||
numbers, YAML and .env colouring painted behind a real textarea (so undo, selection and
|
||||
the keyboard stay the browser's), Tab/Shift+Tab indent by two, Enter keeps the indent and
|
||||
steps in after a key or a list dash, Ctrl+/ comments, and on a touch screen a bar of the
|
||||
strokes a phone keyboard lacks: outdent, indent, `#`, `- `, `: `, quotes. Used by the stack
|
||||
editor (compose and .env) and the new-stack dialog.
|
||||
11. **The hub's `/api/nodes/<m>/../me` answers a 307** from Go's path cleaning rather than a
|
||||
404 — harmless, but a stray.
|
||||
12. **Guest logins can open Users and Settings** — by design ("a guest can do everything you
|
||||
can"), but worth a second look now that Users includes removing accounts.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Ham Lab (`hamlab`) — the web management console app
|
||||
|
||||
This repo is the **app** only: Go daemon + React/xterm UI, deployed as `hamlab.service`
|
||||
on `core`. Product scope, decisions and roadmap are in `PLAN.md`.
|
||||
|
||||
Everything about *how to work* — persona, response style, write-back rule, hosts,
|
||||
credentials, pending items — lives in the homelab repo **Hamlet** at
|
||||
`/home/core/projects/hamlet`, and applies here unchanged:
|
||||
|
||||
@/home/core/projects/hamlet/CLAUDE.md
|
||||
|
||||
Ham Lab facts that belong to the lab (where it runs, ingress, vault item) go in
|
||||
`hamlet/docs/hosts.md` and `hamlet/docs/pending.md`, not here.
|
||||
@@ -1,26 +1,77 @@
|
||||
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
|
||||
# Commit first, then deploy: the version is the commit the binary was built from, counted
|
||||
# from the base tag. A published release is tagged with a version of this same shape
|
||||
# (v0.1.0-137-gc129f65), and described from one of those a build would count from 137 again
|
||||
# and read as older than what it replaces — so tags with a dash are never the base.
|
||||
VERSION ?= $(shell git describe --tags --exclude '*-*' --always 2>/dev/null || echo dev)
|
||||
GO ?= go
|
||||
PREFIX ?= /usr/local/bin
|
||||
|
||||
.PHONY: all web build dev deploy clean
|
||||
# every architecture a release carries; arm is ARMv6 hard-float, so it covers every Raspberry Pi
|
||||
ARCHES ?= amd64 arm64 arm 386 riscv64
|
||||
LDFLAGS = -s -w -X main.version=$(VERSION)
|
||||
# static, no cgo: the same file runs on glibc, musl (Alpine) and anything else with a Linux kernel
|
||||
GOBUILD = CGO_ENABLED=0 GOOS=linux GOARM=6 $(GO) build -trimpath -ldflags "$(LDFLAGS)"
|
||||
|
||||
.PHONY: all web build go dev cross release gitea-release deploy push-nodes push-nodes-force test clean
|
||||
|
||||
all: build
|
||||
|
||||
web:
|
||||
cd web && npm ci --no-audit --no-fund && npm run build
|
||||
|
||||
build: web
|
||||
$(GO) build -trimpath -ldflags "-s -w -X main.version=$(VERSION)" -o bin/hamlab ./cmd/hamlab
|
||||
build: web go
|
||||
|
||||
# backend only (assumes web/dist already built)
|
||||
go:
|
||||
$(GO) build -ldflags "-X main.version=$(VERSION)" -o bin/hamlab ./cmd/hamlab
|
||||
$(GOBUILD) -o bin/hamlab ./cmd/hamlab
|
||||
|
||||
# one other architecture: make build-arm64, build-arm, build-386, build-riscv64, build-amd64
|
||||
build-%: web
|
||||
GOARCH=$* $(GOBUILD) -o bin/hamlab-linux-$* ./cmd/hamlab
|
||||
|
||||
dev:
|
||||
cd web && npm run dev
|
||||
|
||||
deploy: build
|
||||
./deploy/install.sh
|
||||
# siblings for every other architecture; installed beside the binary, and the hub pushes
|
||||
# each managed machine the one that matches it
|
||||
cross:
|
||||
@for a in $(filter-out $(shell $(GO) env GOARCH),$(ARCHES)); do \
|
||||
echo "build linux/$$a"; GOARCH=$$a $(GOBUILD) -o bin/hamlab-linux-$$a ./cmd/hamlab || exit 1; done
|
||||
|
||||
# dist/: one static binary per architecture, install.sh, SHA256SUMS. Serve the directory
|
||||
# anywhere (or attach it to a release) and install.sh does the rest.
|
||||
release: web
|
||||
rm -rf dist && mkdir -p dist
|
||||
@for a in $(ARCHES); do \
|
||||
echo "build linux/$$a"; GOARCH=$$a $(GOBUILD) -o dist/hamlab-linux-$$a ./cmd/hamlab || exit 1; done
|
||||
sed 's|@VERSION@|$(VERSION)|' install.sh > dist/install.sh && chmod +x dist/install.sh
|
||||
echo $(VERSION) > dist/VERSION
|
||||
cd dist && sha256sum hamlab-linux-* install.sh > SHA256SUMS
|
||||
|
||||
test:
|
||||
$(GO) vet ./... && $(GO) test ./...
|
||||
sh -n install.sh
|
||||
cd web && npm test --silent
|
||||
|
||||
# installs here (keeping the config, user and password), then updates every managed machine
|
||||
# Offer this build to every install, wherever it is: a release on Gitea, which is what
|
||||
# Settings -> Updates reads. Deliberate, and not part of deploy — deploy pushes to the
|
||||
# machines this hub owns, publishing offers it to everyone else.
|
||||
gitea-release: release
|
||||
./scripts/gitea-release.sh $(RELEASE_ARGS)
|
||||
|
||||
deploy: build cross
|
||||
./bin/hamlab install --prefix $(PREFIX)
|
||||
@sleep 1; $(PREFIX)/hamlab push-nodes || true
|
||||
@echo "deployed $(VERSION); 'make gitea-release' publishes it for everyone else"
|
||||
|
||||
# push the installed binary to every managed machine (what deploy does last)
|
||||
push-nodes:
|
||||
$(PREFIX)/hamlab push-nodes
|
||||
|
||||
# same, interrupting running tasks and attached terminals on the nodes
|
||||
push-nodes-force:
|
||||
$(PREFIX)/hamlab push-nodes --force
|
||||
|
||||
clean:
|
||||
rm -rf bin internal/web/dist web/dist
|
||||
rm -rf bin dist internal/web/dist web/dist
|
||||
|
||||
@@ -18,6 +18,11 @@ stub tab or a roadmap item.
|
||||
| Data | `/home/core/apps/hamlab/` — `config.yaml`, `state.db` (SQLite) | `apps/` is the nightly rsync target to `grid`, so it's backed up for free. |
|
||||
| Ingress | NPM proxy host `hamlab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
|
||||
| Repo | Gitea `mitchell/hamlab`, private, `main` | Created 2026-09-17. |
|
||||
| Guest logins | A **guest** is an account like any other, with `guest: true` and an expiry: `login`, sessions and prefs need no special case, and **it may do everything an account may do**. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. | Built 2026-09-27. A read-only version was built first and **cut the same day**: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is *an account*: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to. |
|
||||
| Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
|
||||
| Capability, not policy | **A tab offers what the selected machine can do, detected live — never what a rule says should run there.** Where the software is missing but installable, Ham Lab installs it; where it cannot be installed or the hardware lacks it, the tab says so in a sentence and offers what does work. Every action lands on the machine in the switcher. | Mitchell, 2026-10-09, after the Virtualization plan's first draft excluded the Mac mini and `hamtv` on the strength of `hamlet/CLAUDE.md`'s deployment rules: *"I want the option to do it anywhere I want, unless the hardware / software literally doesn't support it. Ham Lab needs to be built in a general-purpose way that can run on any Linux system."* Those rules bind the agent's unasked deployments, not the product. |
|
||||
| Ham Lab's data lives in Ham Lab's folder | **Everything a tab manages sits under the machine's data directory (`config.DataDir()`), never at a convenient path elsewhere.** Existing material is imported by copying. The engine's runtime state (`/var/lib/docker`, `/var/lib/incus`) is the engine's; Ham Lab keeps what would rebuild it. | Mitchell, 2026-10-09, when the Virtualization plan pointed its ISO library at `/tank/software/isos/`: *"ALL data needs to be under a HamLab folder, because it manages its own data."* The same reason `core`'s compose files moved under `apps/hamlab/stacks/` on 2026-10-05. `grid`'s and the Mac's stacks at `~/stacks/<machine>/` were the standing exception; since 2026-10-10 the stacks code offers `<data dir>/stacks` as the default home everywhere and such folders only as discovered extras. |
|
||||
| No virtualization | **Ham Lab does not manage VMs or system containers** — no Incus, no QEMU, no passthrough, no VM consoles. It is a lightweight console; Docker is its one container engine. | Mitchell, 2026-10-15: *"Virtualization is out of scope for HamLab. This is supposed to be a lightweight application, and VMs and pass through and stuff is too much."* The tab built 2026-10-09 (Incus, phases 1–2, deployed as v0.1.0-196) was cut from `main` the same day and kept on the **`virtualization` branch** (its last commit is `main` as of `a2674be`, with `VIRTUALIZATION.md` and the vendored spice-html5). Don't rebuild it on `main`. |
|
||||
|
||||
## Layout
|
||||
|
||||
@@ -25,22 +30,22 @@ stub tab or a roadmap item.
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ header: Ham Lab · core · load 0.42 · ● connected · [logout] │
|
||||
├──────────────┬───────────────────────────────────────────────┤
|
||||
│ ▾ Terminal │ │
|
||||
│ shell 1 ● │ │
|
||||
│ backups │ main content (xterm) │
|
||||
│ + New │ │
|
||||
│ ▸ Services │ │
|
||||
│ ▸ Cron │ │
|
||||
│ ▸ ZFS │ │
|
||||
│ ▸ Dashboard │ │
|
||||
│ Dashboard │ shell 1 ● │ backups │ + │
|
||||
│ Processes ├───────────────────────────────────────────────┤
|
||||
│ Terminal │ │
|
||||
│ Services │ main content (xterm) │
|
||||
│ Cron │ │
|
||||
│ … │ │
|
||||
├──────────────┴───────────────────────────────────────────────┤
|
||||
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- Left rail: collapsible groups, one expanded at a time. Active leaf highlighted.
|
||||
- Terminal group: one leaf per session, `● ` when a process is running, "+ New
|
||||
Terminal" pinned at the bottom. Right-click / long-press → rename, close.
|
||||
- Left rail: flat list of pages, active one highlighted. No nested leaves.
|
||||
- Terminal page: tab strip across the top of the pane, one tab per session, `● ` when
|
||||
a process is running, `+` pinned at the right edge. Active tab carries rename and
|
||||
close; double-click renames, middle-click closes. (Until 2026-09-18 the sessions
|
||||
were leaves under a collapsible Terminal group in the rail.)
|
||||
- Non-MVP groups render as disabled leaves with "coming soon" — the rail shape is
|
||||
the product; stubs keep it honest about scope.
|
||||
- **Mobile**: rail becomes a slide-in drawer; xterm fits the viewport; a sticky key
|
||||
@@ -119,31 +124,872 @@ something long in one, switch away and back, refresh — it's all still there.
|
||||
|
||||
## Roadmap (not MVP — notes only)
|
||||
|
||||
- **Updates tab** — **shipped 2026-09-17** (`internal/updates`, `internal/tasks`,
|
||||
`Updates.tsx`): upgradable packages with current → candidate, suite, security badge
|
||||
and description; index age, last upgrade (from `/var/log/apt/history.log`),
|
||||
unattended-upgrades on/off, reboot-required banner with the packages that caused it;
|
||||
Refresh index, Upgrade all (dist-upgrade), Upgrade selected; changelog per package on
|
||||
demand. Runs go through the new **tasks** package: background command, output kept,
|
||||
followed live over `/api/tasks/{id}/stream`, cancellable, listed at `/api/tasks` (the
|
||||
seed of a Task History tab). apt is fully supported; dnf/pacman list only.
|
||||
- **Power** — header ⏻ button (amber when a reboot is pending), dialog that requires the
|
||||
hostname typed back, then `systemctl reboot` / `poweroff`. `/api/me` carries
|
||||
`reboot_needed` so the header knows.
|
||||
|
||||
- **tmux-backed sessions** so shells survive a daemon restart / redeploy. Would spawn
|
||||
`tmux new -A -s hamlab-<id>`; needs `mouse on` and status-bar handling in xterm.
|
||||
- **Services tab** — `systemctl list-units`, per-unit page: status, last 200 journal
|
||||
lines (live-tail via WS), start/stop/restart/enable buttons, unit file viewer.
|
||||
- **Cron tab** — parse `/etc/crontab`, `/etc/cron.d/*`, user crontabs, and systemd
|
||||
timers into one table with next-run times; edit via `crontab -` round-trip.
|
||||
- **ZFS tab** — pools, datasets (`used/avail/refer/usedbysnapshots`), snapshot list,
|
||||
scrub status, and the `freeing` counter. `grid` is the target, which implies
|
||||
multi-host.
|
||||
- **Dashboard** — CPU/mem/load/net, disks and SMART (`smartctl -j`), temps and fans
|
||||
(`sensors -j`), UPS from `pwrstat`. Ham Lab collects on a timer into a SQLite
|
||||
ring (e.g. 10 s resolution, 7 days) and draws sparklines. This is where the
|
||||
"collects its own data" requirement lands; no Prometheus dependency.
|
||||
- **Multi-host** — same binary as a headless *agent* on `grid` / Mac mini /
|
||||
`wyse-5070-1`, `core`'s instance is the UI and fans out over mTLS. The rail grows a
|
||||
host level at the top, exactly like Proxmox's datacenter → node tree. Design the
|
||||
MVP URL space so `host` can be inserted later without breaking clients.
|
||||
- **Auth hardening** — WebAuthn passkeys (phone Face ID), TOTP fallback, session
|
||||
list + revoke, optional NPM access list restricting to WireGuard/LAN.
|
||||
- **Files** — browse/upload/download/edit under `/`, with a sudo toggle.
|
||||
- **Cron tab** — **shipped 2026-09-17** (`internal/cron`, `Cron.tsx`): one list built
|
||||
from `/etc/crontab`, `/etc/cron.d/*`, every user crontab (via sudo), the scripts in
|
||||
the `run-parts` directories those drive, plus systemd timers. Next run computed by an
|
||||
in-house cron-expression evaluator (tested: ranges, steps, names, dom-OR-dow rule,
|
||||
`@` aliases); last run from the journal's `(user) CMD (command)` lines (cron logs via
|
||||
syslog to journald; reading needs sudo/adm). Commented-out jobs show as disabled;
|
||||
job-shaped lines inside comment blocks (Debian's header examples) are ignored. Per
|
||||
job: Run now (background, as the job's user, `sh -c` with crontab `%` rules, output
|
||||
captured), Enable/Disable (comments the line in place, refuses if the file changed
|
||||
underneath), Edit source (textarea; validated line-by-line; user crontabs installed
|
||||
with `crontab -`, system files written via `sudo tee`), Cron log, and Output file
|
||||
when the command redirects to one. Timers: Run now (starts the activated service),
|
||||
start/stop/enable/disable, live logs of the service via the Services tab's tail.
|
||||
- **Disks tab** — **shipped 2026-09-17** (`internal/storage`, `Disks.tsx`), replacing
|
||||
the planned ZFS tab: block devices from `lsblk -J` (kind, model, size, temperature
|
||||
from the device's hwmon node, partitions with filesystem usage, live read/write/busy
|
||||
from the metrics sampler); SMART via `smartctl -j` when smartmontools is present (one
|
||||
button installs it as a task otherwise): health, temperature, power-on time, self-test
|
||||
status, ATA attribute table with the failure-predicting IDs flagged, NVMe endurance and
|
||||
error counters, short/long self-tests. Spinning disks are read with `-n standby` so
|
||||
the tab never wakes a sleeping drive; "Wake and read" is explicit. ZFS section when
|
||||
the tools and module are there: pools with health, capacity, fragmentation, last scan
|
||||
and vdev tree; scrub start/pause/stop; datasets tree with used/refer/in-snapshots/
|
||||
snapshot count/compressratio/mount; per-dataset snapshot list and "Snapshot now".
|
||||
Verified against `grid` (raidz2 `tank`, 10 datasets, 515 snapshots, 8 WD40EZAZ + 970
|
||||
EVO Plus SMART). Snapshot destroy and dataset property editing are deliberately not
|
||||
there yet.
|
||||
- **Dashboard** — **first cut shipped 2026-09-17** (`internal/metrics`, `Dashboard.tsx`):
|
||||
CPU (total + per core), memory/swap, load, uptime, hwmon temps and fans read straight
|
||||
from `/sys` (no lm-sensors), per-interface network rates, per-disk read/write/busy
|
||||
from `/proc/diskstats`, filesystem usage, UPS via `sudo -n pwrstat -status`. Sampled
|
||||
every 2 s into an in-memory ring of 10 min; the UI (the **Dashboard** tab
|
||||
since 2026-09-19) reads it all through the sensor API — see Sensors below. **Customizable**: a
|
||||
Customize mode reorders and hides modules (CPU, Memory, Load, Uptime, Temperatures,
|
||||
Fans, Network, Disks, Filesystems, UPS) and hides individual rows (an interface, a
|
||||
mount, a disk, a sensor); the layout is stored server-side in `state.json` via
|
||||
`GET/PUT /api/prefs/dashboard` so every device sees the same dashboard. The update
|
||||
interval (100 ms – 2 s) is part of the layout and drives both the browser's polling and
|
||||
the daemon's sampling clock; the chart window is always 300 samples, so 30 s at 100 ms
|
||||
and 10 min at 2 s. Still to do: persistent
|
||||
history (SQLite ring, e.g. 10 s × 7 days), SMART (`smartctl -j`), longer time ranges.
|
||||
- **Updating from the Updates tab** — **shipped 2026-09-26**: the "what is out of date" list
|
||||
was read-only, so seeing a newer image and doing something about it were different places.
|
||||
Each row now has **Update**, and the header has **Update all**. Updating is per stack —
|
||||
that is the unit compose deals in — so a row acts on whichever stacks use that image, and
|
||||
several stacks run through a queue one at a time rather than recreating half the machine at
|
||||
once, each with its task output on screen. A row that succeeds says "updated just now"
|
||||
instead of re-checking, since a check is a registry request.
|
||||
- **Recreate, and the Updates tab at full width** (2026-09-26): a stack's actions gained
|
||||
**Recreate** (`up -d --force-recreate`), the only way out of a container Docker refuses to
|
||||
start — one pinned to a network that has since been recreated answers "network … not
|
||||
found" to Deploy and Update alike, because as far as compose is concerned nothing about it
|
||||
changed. Found the hard way when a SiYuan update landed in exactly that state. The Updates
|
||||
tab also stopped rendering as a list-and-detail pair: it is a page, not a list, so on a
|
||||
wide window it now sits under the tab strip at full width instead of being squeezed into
|
||||
the right-hand half beside an empty column.
|
||||
- **Cron in English** (`crontext.ts`, 2026-09-26): schedules read "daily at 3:15am" rather
|
||||
than `15 3 * * *`, with the expression kept beside it. Handles lists, ranges, steps, day
|
||||
and month names, `@daily` and friends, weekdays/weekends, midnight and noon — and declines
|
||||
to describe anything it is not sure of, because a confident wrong sentence is worse than
|
||||
five fields. The Jobs tab had its own smaller version of this; it now uses this one.
|
||||
|
||||
- **Hardware tab** — **shipped 2026-09-25** (`internal/hardware`, `internal/api/hardware.go`,
|
||||
`Hardware.tsx`): what the machine is physically made of, per machine like every other page.
|
||||
Four headline tiles (processor, memory, storage, graphics) over cards for each: the
|
||||
processor with its cores, clock range, socket, caches, microcode and the handful of feature
|
||||
flags anyone asks about; **every memory slot** with size, type, form factor, rated and
|
||||
running speed, maker, part number, serial, rank and voltage, and the empty ones marked
|
||||
empty; each drive with model, size, bus, firmware and serial; each display adapter with its
|
||||
driver, PCI address and video memory; and the board itself — model, firmware and date,
|
||||
serial, kernel. Read from what Debian already has (dmidecode for what only the firmware
|
||||
knows, lscpu, lspci, sysfs) and cached for ten minutes, since none of it changes under a
|
||||
running kernel; "Re-read" is there for the one thing that does, a drive plugged in.
|
||||
Every reader may come back empty and the page says what it could not read and why: a
|
||||
machine with no dmidecode, no root, or no PCI bus still gets a page. Verified on `core`
|
||||
(SMBIOS, Intel graphics) and on the Mac mini (Asahi: no SMBIOS memory table, a
|
||||
device-tree GPU, heterogeneous cores — where lscpu's "cores per socket" counts one
|
||||
cluster, so without SMT the thread count is the honest core count).
|
||||
|
||||
- **Sensors** — **shipped 2026-09-19** (`internal/sensors`, `internal/metrics/sensors.go`,
|
||||
`internal/api/status_sensors.go`, `web/src/sensors.ts`, `widgets.tsx`, `Embed.tsx`;
|
||||
reference in `SENSORS.md`): everything measured is a sensor with a stable dotted ID;
|
||||
catalog, single value, history, a **shaped query** (the client sends the object it wants
|
||||
and gets it back filled in — GraphQL's idea without GraphQL, since sensors differ per
|
||||
machine and a static schema would collapse to `sensor(id:)`), the same query as a live
|
||||
SSE stream, Prometheus `/metrics`, read-only `sensors:read` tokens (never full tokens in
|
||||
a URL), status sensors (services, updates, ZFS, SMART), a widget library, custom widgets
|
||||
on the Dashboard tab (briefly renamed "Monitors" 2026-09-19, back to "Dashboard"
|
||||
2026-09-20 at Mitchell's request; the view id and URL were always `dashboard`), and a
|
||||
frameable `/embed/widget` for Glance and friends. (An external demo page was deployed at
|
||||
`hamlab-demo.bigham.dev` and linked from the Widgets tab 2026-09-19; **both removed
|
||||
2026-09-23** at Mitchell's request — the Widgets tab shows the same thing against the
|
||||
machine you are signed in to, so the second copy earned nothing and cost a public
|
||||
read-only token, which was revoked with it. `examples/external-demo/` stays as an example
|
||||
to read.) The stock Dashboard modules and the Disks
|
||||
tab read the sensor API too (`sampleAdapter.ts`); `/api/metrics` is retired.
|
||||
Also per-service and per-container sensors (memory and CPU from cgroups), tokens limited
|
||||
to ID patterns, and `/api/fleet/sensors/query` for the whole fleet in one request.
|
||||
- **Alerts** — **shipped 2026-09-19** (`internal/api/alerts.go`, `Alerts.tsx`): rules over
|
||||
sensors (`container.*.running == false for 120 s`), evaluated by the hub every 15 s for
|
||||
itself and every managed machine in one shaped query each; `hamlab.reachable` is supplied
|
||||
by the hub so "a machine stopped answering" is a rule like any other. Ten defaults
|
||||
(machine offline, container down/unhealthy, failed service, disk full, pool unhealthy,
|
||||
SMART failing, UPS on battery, CPU hot, memory). Firing and resolved are events: kept
|
||||
(last 200, in `state.json`), shown on the hub-only Alerts tab with a count on the rail,
|
||||
and POSTed to channels — **ntfy** (title/priority headers) or a **webhook** (JSON; Home
|
||||
Assistant, Slack). The rule editor shows what a selector matches across the fleet as
|
||||
you type. Not done: quiet hours, per-rule channels, Web Push straight to the PWA.
|
||||
- **Docker tab** — **shipped 2026-09-19** (`internal/api/docker.go`, `Docker.tsx`):
|
||||
containers grouped by compose stack with live memory and CPU from the container
|
||||
sensors, start / stop / restart (asked first), logs that follow (polled with `--since`,
|
||||
no held connection), inspect (env *names* only), and images / volumes / networks
|
||||
tables. Works on every machine through the hub; `#/grid/docker/immich_server` links to
|
||||
one. **Stacks** (`internal/api/stacks.go`): every compose project, found from its
|
||||
containers' labels, with its file to edit and Deploy (`up -d`), Update (pull, then
|
||||
deploy), Restart and Stop as followed tasks. The file is looked for where the label says
|
||||
*and through every container's mounts*, because an orchestrator that runs compose inside
|
||||
its own container (Dockhand here; Dockge, Portainer) records a path in its own
|
||||
filesystem; compose is then run **inside that container**, so `.env`, relative paths
|
||||
and the compose labels stay exactly as deployed (verified: labels identical after a
|
||||
redeploy). Save validates with `compose config` first, keeps the previous file as
|
||||
`<file>.hamlab-bak`, and never deploys by itself. A stack whose file is not on the
|
||||
machine that runs it is listed read-only, and that is the end of it: **since 2026-10-10
|
||||
the hub holds no files for a node and never points compose at another machine's
|
||||
Docker.** The `hubStacks` path — which resolved a node's files on the hub and deployed
|
||||
over `tcp://…:2375`, the way Dockhand had — is gone, and every
|
||||
`/api/nodes/<node>/docker/…` request is the node's own Ham Lab's to answer through the
|
||||
generic proxy. Also per stack: every file it was deployed from (chips), its
|
||||
**`.env`** (values masked until "Reveal and edit"; validated in place, put back if
|
||||
compose objects), **what Deploy would change** (a line diff against the copy kept at the
|
||||
last deploy from here, `<file>.hamlab-deployed`, else the copy the last save replaced;
|
||||
`.env` values never shown), and **image update checks** — asked for, never polled, since
|
||||
each is a registry request Docker Hub counts; remembered six hours. A save refuses a
|
||||
file changed on disk meanwhile.
|
||||
**Making and unmaking (2026-09-20):** "+ New stack" — a name, a folder, compose text and
|
||||
an optional `.env` (written root-only), checked by compose before it is kept, then
|
||||
"Create and deploy" or "Create only". "+ New container" is the same dialog opened on a
|
||||
form (image, ports, volumes, environment, restart, command, host networking) that writes
|
||||
the compose for you: a container made here is a one-service stack, so it stays editable
|
||||
and redeployable, where a bare `docker run` could only be thrown away and typed again.
|
||||
New stacks go in a **home**. Since 2026-10-10 the default is Ham Lab's own
|
||||
`<data dir>/stacks`, offered first on every machine; a folder where two or more stacks
|
||||
are already deployed from is offered as well, and a stack made there is deployed the way
|
||||
its neighbours were (inside an orchestrator's container, when that is how they run). Homes
|
||||
are also scanned, so a stack with no containers ("not deployed": never deployed, or taken
|
||||
down) is still listed. **Down** removes a stack's containers; **Delete…** (type its name)
|
||||
takes it down and moves its own files to `<home>/.hamlab-trash/<name>-<time>/` — moved,
|
||||
not erased; volumes and bind-mounted data are never touched. A stopped container can be
|
||||
**removed** (a running one must be stopped first, on purpose). Images: **pull** (a followed
|
||||
task), **remove** (unforced: Docker refuses one still in use) and **prune** (dangling only).
|
||||
**Volumes (2026-09-20):** each says which containers use it; one that nothing uses can be
|
||||
deleted — a named one by typing its name (it may be the data of a stack that is only
|
||||
down), an anonymous one by confirming its full name — and "Prune anonymous" clears the
|
||||
nameless leftovers of recreated containers, never named volumes. Images are marked "in
|
||||
use", and "Remove unused" takes every image no container uses. Nothing is ever forced:
|
||||
Docker refuses what is still in use, and that refusal is the answer. Still absent:
|
||||
removing networks.
|
||||
- **Top merged into Processes** — **2026-09-23.** The two tabs answered the same question
|
||||
from the same data (Top already read `/api/procs`, the endpoint the Processes table was
|
||||
built on), so at Mitchell's instruction there is now one tab, **Processes**, showing what
|
||||
Top showed: the btop-style whole-machine screen. `Procs.tsx` is deleted; the `top` view
|
||||
id and its route are gone. A rail order saved before this keeps its shape — `railorder.ts`
|
||||
maps the retired `top` to `procs` and drops the duplicate, so the merged page sits where
|
||||
Top sat rather than falling to the bottom.
|
||||
- **GPUs on the Processes tab** — **shipped 2026-10-02** (`internal/metrics/gpu.go`, the
|
||||
`GPUBox` in `Top.tsx`, `gpu.*` sensors). Asked for as "GPU activity and temps, like btop
|
||||
has it, always on". Each DRM device with a real GPU driver gets a box under the cpu box:
|
||||
busy curve, busy meter, a meter per engine (render/video/copy/enhance), temperature, clock,
|
||||
power and fan where the card has them, and the processes that have it open with their
|
||||
share. The process table gains a `Gpu%` column and sort. **How it is measured, and the
|
||||
honest gaps:** device busy is Intel's RC6 idle residency inverted (the counter every GT
|
||||
exposes without root; `intel_gpu_top` shows the same number) or amdgpu's own percent;
|
||||
per-process and per-engine shares are DRM fdinfo deltas, the way nvtop reads them. Other
|
||||
users' fdinfo needs root, so a daemon running as `core` keeps **one** root child
|
||||
(`hamlab drm-clients`, over a pipe) rather than a sudo a second. The Arc A380 on i915 6.12
|
||||
exposes power, energy and fan but **no temperature**, so its box shows none; an integrated
|
||||
GPU shows the CPU package temperature marked `*`. Asahi (the Mac mini) has neither a busy
|
||||
counter nor fdinfo in 6.19, so it lists the device and says so. No NVIDIA path: nothing here
|
||||
has one to test against. **Names come from an embedded PCI ID table** (2026-10-02,
|
||||
`internal/metrics/pciids_gpu.txt`, regenerated by `scripts/pci-ids-gpu.sh` from the hub's
|
||||
`pci.ids`): `gate` has no `lspci` and showed "Intel GPU 3185" beside `hamtv`'s "Intel UHD
|
||||
Graphics 600" for the identical chip. The table is tried first everywhere, so the name — and
|
||||
the sensor ID built from it — does not depend on which distro's pciutils a machine has;
|
||||
`lspci` is the fallback for a card newer than the table. Two fixes the same evening:
|
||||
the gpu box's layout rules had been lost from the stylesheet (v0.1.0-166/167), and on
|
||||
`grid` the sampler hung for 42 minutes in Go's poller on the Arc's `fan1_input` —
|
||||
`readStr` now uses plain blocking syscalls and each GPU's sysfs reads run under a
|
||||
3-second deadline (v0.1.0-168; `hamlet/docs/gotchas.md` has the mechanism).
|
||||
- **CPU power on the Processes tab** — **shipped 2026-10-10** (v0.1.0-194,
|
||||
`internal/metrics/cpupower.go`, the `cpu.power` sensor). Asked for beside the clock speed,
|
||||
and that is where it sits: "3600 MHz · 89 W" in the cpu box's header. Watts are the RAPL
|
||||
package energy counters' delta between samples, summed over packages; `psys`, a package's
|
||||
parts and the MMIO duplicate are left out. The counters are root-only, so a daemon running
|
||||
as `core` keeps a second root child, `hamlab cpu-energy`, beside `drm-clients`. Read on all
|
||||
four Intel machines; the Mac mini has no CPU-only counter (its SMC reports the whole
|
||||
machine's draw instead) and shows nothing.
|
||||
- **The ARC on the ZFS tab** — **shipped 2026-10-02** (`internal/metrics/arc.go`,
|
||||
`internal/storage/arc.go`, `GET /api/zfs/arc`, the Cache section in `ZFS.tsx`, `zfs.arc.*`
|
||||
sensors). Asked for as "current ARC usage, including any types of caching services in the
|
||||
pool". One card for the ARC: size against its ceiling with the target and floor, sparklines
|
||||
of size and hit rate from the sampler's history, what it holds (MRU/MFU/dirty, data vs
|
||||
metadata, compression, the ghost lists), the prefetcher and dbuf cache, memory-pressure
|
||||
counters, and the since-boot breakdown by demand/prefetch × data/metadata. Then a card per
|
||||
pool naming every caching tier — L2ARC, SLOG, special vdev, dedup table, spares — with its
|
||||
devices and fill, **or saying plainly that there is none and what that means** (the ZIL on
|
||||
the pool's own vdevs, with how much it has written), plus the properties that steer caching
|
||||
(`primarycache`, `secondarycache`, `sync`, `logbias`). Read from `/proc/spl/kstat/zfs/*`
|
||||
and `/sys/module/zfs/parameters`, with `zpool list -Hpv` and `zfs get` run without sudo
|
||||
(they need none) every 30 s at most. The day it shipped it showed `tank`'s ARC pinned at
|
||||
its 2 GiB floor under `arc_no_grow`, with a 53 GiB python3 process holding the RAM.
|
||||
- **Dashboard retired** — **2026-09-23**, at Mitchell's instruction, the same reasoning that
|
||||
merged Top into Processes: the app repeated itself. The customisable module list is gone
|
||||
(`Dashboard.tsx` deleted), and with it `/api/prefs/dashboard`, `applyPrefs`, and the
|
||||
per-user `dashboard` blob in `state.json`. The Processes tab already shows those numbers
|
||||
on one screen. `fmtBytes`/`fmtRate` moved to `fmt.ts` (Disks, Files and Space use them);
|
||||
`MONITOR_HISTORY` and `seriesOf` went with the page. The default view after login, after
|
||||
switching machines, and when a hub-only page is left behind, is now **Processes**. A saved
|
||||
rail order naming `dashboard` degrades quietly — `orderBy` drops ids with no page.
|
||||
**Widgets stays and says what it is for**: a notice at the top explains these are for
|
||||
dashboards *outside* Ham Lab — Glance, Home Assistant, a wall display — served from
|
||||
`/embed/widget` with a read-only sensor token, not a second copy of the console's screens.
|
||||
**It also closed a real bug.** `applyPrefs` set the machine's sampling rate from
|
||||
`intervalMs` inside the dashboard layout, and its `else` branch reset the sampler to the
|
||||
default whenever that field was absent. Since layouts became per-user, one account saving
|
||||
a dashboard could silently reset the sampling rate for the whole machine, overriding the
|
||||
fleet refresh rate set in Settings. The sampler now has one source: `refreshMs` in the UI
|
||||
prefs, which the hub fans out, remembered per machine as `sampler_ms`.
|
||||
- **ZFS tab, with scheduled snapshots** — **shipped 2026-09-24** (`internal/storage/snapshots.go`,
|
||||
`internal/api/zfs.go`, `ZFS.tsx`). Pools, datasets and snapshots moved out of Disks —
|
||||
Disks is the hardware, ZFS is the filesystem on top of it — and the tab gained what it was
|
||||
split out for: **snapshot schedules**. Hourly, daily, weekly or monthly, per dataset,
|
||||
optionally recursive, with a **retention period** and a floor that keeps the newest few
|
||||
whatever their age. A missed window (the machine was off) runs once on return rather than
|
||||
being skipped.
|
||||
**The naming is zfSnap's, deliberately.** `grid` has run zfSnap v1.11.1 from root cron at
|
||||
02:30 with a 6-week TTL since before this repo existed, and it works. So rather than
|
||||
invent a second convention for two tools to disagree over, this writes and reads exactly
|
||||
what zfSnap does — `tank@2026-09-24_02.30.01--6w`, **retention encoded in the name**. Three
|
||||
consequences: Ham Lab shows and expires the snapshots zfSnap already took; the two can run
|
||||
side by side without fighting, since both only remove what has passed the expiry written
|
||||
into its own name; and changing a schedule never re-dates old snapshots, because each
|
||||
carries the rule it was taken under. The page **detects the existing cron job**
|
||||
(zfSnap, zfs-auto-snapshot, sanoid, or a bare `zfs snapshot`) and says so, warning if a Ham
|
||||
Lab schedule is running as well.
|
||||
**Nothing is destroyed unseen.** Only names that parse as a stamp and a TTL are ever
|
||||
candidates for expiry, so a snapshot taken by hand is invisible to a schedule; Run now
|
||||
says that expired snapshots go with it; and a snapshot deleted by hand is confirmed first
|
||||
and needs its full name echoed back by the caller. (A "what would it delete?" preview
|
||||
shipped and was removed the same day as redundant — Mitchell's call.)
|
||||
The datasets table is a **snapshot explorer**: each dataset shows how many snapshots it has
|
||||
and what they cost, and opens into the snapshots themselves — when each was taken, **when
|
||||
it expires** (read from the TTL in its own name, so no server call), its size, and Delete.
|
||||
Deleting takes the same-named snapshot on descendant datasets too, which is how a recursive
|
||||
schedule made it.
|
||||
**A schedule is a real cron job** (2026-09-24, Mitchell's call after the first cut ran it
|
||||
from a ticker inside the daemon). Ham Lab writes `/etc/cron.d/hamlab-zfs`, owns it, and
|
||||
rewrites it whole on every change; each line calls `hamlab zfs-snapshot <id>` as the
|
||||
account Ham Lab runs as, reaching zfs through sudo as the daemon does. Snapshots are
|
||||
backup-shaped and must not stop because a console is down, upgrading, or uninstalled — and
|
||||
a cron line is greppable by anyone looking at the machine the normal way. **A cron
|
||||
expression is the single description of when a schedule fires**: the four common shapes
|
||||
render to one, `custom` states one outright, and the next run is read back with the same
|
||||
evaluator the Jobs tab uses on everyone else's lines, so the page, the file and the
|
||||
prediction cannot disagree. A run's outcome is written to `zfs-runs/<id>.json` beside the
|
||||
config, never to `state.json`, because the cron job is a separate process and the daemon
|
||||
would clobber it. An existing external job can be **adopted** — its schedule, retention
|
||||
and dataset read out of the cron line into a new Ham Lab schedule.
|
||||
**zfSnap is retired on `grid`** as of that day: Ham Lab now takes the nightly `tank`
|
||||
snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it
|
||||
had taken left untouched and fully readable here — which is what adopting its naming
|
||||
bought. See `hamlet/docs/hosts.md`.
|
||||
- **KVM tab** — **shipped 2026-09-24** (`internal/api/kvm.go`, `KVM.tsx`, `config.KVM`).
|
||||
Paste a device's address; its own web page appears in the tab. **Nothing in it knows
|
||||
anything about any particular KVM** — the first device here is a Sipeed NanoKVM on `grid`
|
||||
at `10.0.0.217`, but none of that is in the code.
|
||||
**Why it proxies rather than framing the device directly.** Ham Lab is served over TLS and
|
||||
a KVM usually is not, so a bare `http://…` iframe is blocked as mixed content before it
|
||||
loads. Serving the device from Ham Lab's own origin (`/kvm/<id>/…`) fixes that, puts it
|
||||
behind **Ham Lab's** login rather than a second one, and makes the iframe same-origin.
|
||||
**The general problem, and the answer.** A device UI asks for its assets by absolute path
|
||||
— `/assets/app.js`, `/api/whatever` — which no path prefix can rewrite away, and rewriting
|
||||
bodies per device would not be general. So the prefix is only the entry point: any request
|
||||
Ham Lab does not serve itself is routed to whichever KVM the asking page belongs to, read
|
||||
from the **Referer**, falling back to a **cookie** when there is none (a WebSocket upgrade).
|
||||
Ham Lab's own routes always match first, so a device can never shadow the console, and the
|
||||
fallback is gated on a live session. The device never receives Ham Lab's session cookie,
|
||||
and its `X-Frame-Options`/CSP/HSTS are dropped on the way out since Ham Lab is the origin
|
||||
now. Redirects to the device's root are rewritten back through the prefix.
|
||||
**One catch found in testing:** Ham Lab's own `frame-ancestors 'none'` forbids framing even
|
||||
from the same origin, so `/kvm/` paths send `'self'` instead. Without that the iframe was
|
||||
refused before it loaded.
|
||||
**The keyboard** needs less than it sounds: a page cannot forward keys into an iframe and
|
||||
does not need to, since one with focus receives them directly. Fullscreen plus
|
||||
`navigator.keyboard.lock()` is what takes Escape, Tab and Ctrl-W back from the browser;
|
||||
that is Chromium-only, and the tab says which version you are getting rather than
|
||||
pretending.
|
||||
**One KVM per machine, and the tab follows the machine switcher** (corrected the same day:
|
||||
the first cut was a hub-side list you pointed at machines, which had it backwards). Switch
|
||||
to `grid` and you get grid's; adding a second for a machine replaces the first. The
|
||||
addresses are still kept **on the hub** rather than on each machine, because a KVM earns
|
||||
its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub
|
||||
proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at
|
||||
all. Not done: holding the device's own login on the hub, so its page does not ask for a
|
||||
second password.
|
||||
- **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
|
||||
Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
|
||||
type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
|
||||
each repaints the terminal too. **Each machine wears a colour** (`accents.ts`): a stripe
|
||||
across the header and its name in the switcher — picked from ten swatches or any colour
|
||||
on the Machines page, else derived from its name — saved with the theme on the hub.
|
||||
- **Multi-host** — **Phase 1 shipped 2026-09-18** (`internal/nodes`, `internal/tlsauto`,
|
||||
`MachineSwitcher.tsx`, `Machines.tsx`). One binary, hub and spoke: a host whose
|
||||
`config.json` lists `nodes` is a hub; any other is a standalone node with its full UI.
|
||||
Node side: `"tls": "auto"` makes a self-signed cert in the data dir and serves HTTPS;
|
||||
`POST /api/tokens {password,name}` trades the login password for a long-lived bearer
|
||||
token (SHA-256 stored in `tokens`), accepted by the auth middleware alongside the
|
||||
cookie; `DELETE /api/tokens/{name}` revokes. Hub side: `POST /api/nodes {name,url,
|
||||
password}` peeks the node's certificate (trust on first use, fingerprint pinned),
|
||||
obtains a token named `hub:<hostname>`, stores `{name,url,fingerprint,token}`; a
|
||||
15 s prober keeps each node's status; `/api/nodes/{name}/…` is a reverse proxy to the
|
||||
node's `/api/…` (bearer swapped in, cookie and Origin stripped, streaming flush) that
|
||||
carries WebSockets, so terminals, log tails and task output work remotely.
|
||||
Frontend: `setMachine()` rewrites every `/api/…` path; the rail has a machine
|
||||
switcher at the top and a Machines page to add/remove/open. Pages remount on switch;
|
||||
terminals live on the node. Nodes: `grid`, `wyse`. **Phase 3 (update from hub)
|
||||
shipped 2026-09-18**: node `POST /api/self-update` takes a binary, checks it answers
|
||||
`version`, keeps `hamlab.prev` in the data dir, installs over the executable via
|
||||
`sudo install`, replies, then restarts the service; the hub pushes its own executable
|
||||
with `POST /api/machines/{name}/update` or `/api/machines-update-all` (architecture
|
||||
checked first via `/api/me.arch`), the Machines page shows an **outdated** badge and
|
||||
Update buttons, and `hamlab push-nodes` does the same from the CLI — `make deploy` runs
|
||||
it last, so one deploy on the hub updates every machine. First install on a new host
|
||||
is still by hand (scp binary + `install.sh`, `hamlab passwd`, `tls auto`). Still to
|
||||
do: Phase 2 Datacenter overview; optional firewall rule limiting a node's port to
|
||||
the hub. **Cross-architecture** (2026-09-18): `make deploy` also builds
|
||||
`bin/hamlab-linux-arm64`, `install.sh` installs it beside the binary as
|
||||
`hamlab-arm64`, and the hub picks the sibling matching a node's `/api/me.arch` when
|
||||
pushing — the Mac mini (Fedora Asahi, aarch64, node `mac`) updates like the rest.
|
||||
- **Portability + install** — **shipped 2026-09-18** (`internal/setup`, `internal/priv`,
|
||||
`install.sh`, `make release`): the binary installs itself (`hamlab install` /
|
||||
`uninstall`; systemd or OpenRC; root, sudo or doas), all builds static on amd64, arm64,
|
||||
arm, 386 and riscv64, `install.sh` as the curl-able bootstrap with checksums. Every
|
||||
privileged call goes through `priv` instead of a literal `sudo -n`. Updates tab acts on
|
||||
dnf, zypper, pacman and apk as well as apt. Services list falls back to the plain
|
||||
table on systemd < 246; Disks falls back to `/sys/block` without `lsblk`. Listener
|
||||
uses `IP_FREEBIND`. Self-update: pre-flight `check` against the live config, atomic
|
||||
replace, exec in place. Config default moved to `~/.config/hamlab` (existing
|
||||
`~/apps/hamlab` still found by the installer and the CLI). Tested in
|
||||
containers: Debian 12, Ubuntu 22.04, Fedora 40, Rocky 8, openSUSE Leap 15.5, Alpine
|
||||
3.18/3.20 (OpenRC + doas, non-root), plus the four lab hosts. **procd (OpenWrt) added
|
||||
2026-09-26** (`internal/setup/hamlab.procd`): detected on `/sbin/procd` + `/etc/rc.common`
|
||||
ahead of the OpenRC branch, installed to the same `/etc/init.d/hamlab` path and told apart
|
||||
from OpenRC at uninstall by `USE_PROCD=1` in the file. Verified on the new router
|
||||
(`gate`, OpenWrt 25.12.5, running as root): `S95`/`K10` symlinks, procd supervising the
|
||||
instance, Docker tab and stacks working through the hub. The Updates tab needed no change
|
||||
— apk-tools 3 answers `apk version -l '<'` the same way Alpine's does. Not done: an
|
||||
OpenRC or procd backend for the Services tab (it is systemd-only and says so cleanly);
|
||||
runit/s6/sysvinit service files; public hosting for `install.sh` (the Gitea repo is
|
||||
private, so the one-liner needs `HAMLAB_TOKEN`).
|
||||
- **Users, sessions and settings that follow the account** — **shipped 2026-09-22**
|
||||
(`internal/api/users.go`, `config.User`, `store.Session`/`UserState`, `Login.tsx`,
|
||||
`Users.tsx` — a hub-only **Users** tab in the rail, added the same day at Mitchell's
|
||||
request, holds everything about accounts: your own password, your sessions, the
|
||||
accounts; Settings keeps appearance, rates, tokens and About). **Rail order is per account** too (2026-09-22,
|
||||
`railorder.ts`, `Rail.tsx`): "⇅ Arrange" at the foot of the rail puts a handle on every
|
||||
page; drag one up or down (pointer events, so a finger works), let go and the order is
|
||||
saved in the UI prefs with the theme; "Reset" restores the built-in order; pages added
|
||||
in a later build go after the ones the saved order names. The login is a user name and a password. `config.json` holds `users`
|
||||
(name, argon2id hash, created, last seen); the old single `password_hash` becomes the
|
||||
account named after the OS user the console runs as on first boot, its signed-in
|
||||
sessions become that account's, and the machine-wide theme/accents/dashboard become
|
||||
theirs — nothing to redo after the upgrade. A session carries its user, an ID, when it
|
||||
was made and last seen, the IP and the browser; **Users → Sessions** lists yours
|
||||
("Safari on iPhone · 10.0.0.5 · seen 2 min ago"), revokes one, or signs out everywhere
|
||||
else. **Users → Users** lists accounts with their session counts, adds one, removes
|
||||
one (their sessions and settings go too; never yourself, never the last), and resets
|
||||
another's password (they are signed out); removing and resetting take your own
|
||||
password first. Everyone signed in can do everything — a household console, not a
|
||||
tenancy — and each machine keeps its own accounts: a hub reaches a node with its token,
|
||||
so pairing (Machines → Add) now takes a user and password on the node, the user
|
||||
optional while it has one account. Per account: `ui` (theme, accents) and `dashboard`
|
||||
in `state.json` under `users.<name>`. Still the machine's: refresh rate (it is how
|
||||
often every machine samples — it rides in the UI prefs object for the client but is
|
||||
stored once, machine-wide), graph history, alert rules, terminals (the shells run as
|
||||
one OS user anyway). A login attempt for a missing user takes as long as a wrong
|
||||
password, so the form can't list accounts. Every write's log line now names the
|
||||
account or token (`core@10.0.0.5 via …`), and Access tokens say who issued them. CLI:
|
||||
`hamlab passwd [user]` (makes the account if new; reads stdin when piped), `hamlab
|
||||
users`, `hamlab user-rm <user>`; `hamlab install` names the first account after the
|
||||
service user. Not done: a read-only viewer role.
|
||||
- **Space tab** — **shipped 2026-09-22** (`internal/space`, `internal/api/space.go`,
|
||||
`Space.tsx`), asked for as "exactly WizTree": what is using a disk. The unit is a whole
|
||||
volume, picked from a dropdown at the top: a **disk on its own** (every filesystem its
|
||||
partitions back) or a **ZFS pool as a whole** (every mounted dataset) — a pool's member
|
||||
disks are listed under it with the layout (raidz2, 8 disks: sda …) and never offered
|
||||
apart, since one disk's share of a pool means nothing (Mitchell's call, same day, after a
|
||||
first cut that listed the disks as chips). The walk runs as a helper process, `hamlab space-walk <roots>`,
|
||||
as root through `priv` when the host allows (so `/var/lib/docker` and `/root` count), 32
|
||||
workers reading directories depth-first, never crossing a filesystem boundary by itself;
|
||||
each root is grafted into one tree at its path. Files under 128 KiB are not kept one by one
|
||||
but summed per folder as "N smaller files" — what keeps a scan of millions of files to tens
|
||||
of MB in the daemon; the tree crosses back as a varint stream (`codec.go`), and the helper's
|
||||
memory goes with it. core's root: 646k files in ~1.2 s. Progress (files, bytes, current
|
||||
folder) every 250 ms; Cancel sends SIGTERM. The page: picker → Scan/Rescan → **Folders**
|
||||
(a tree largest-first: size, share of parent as a bar, files, folders, modified; expand in
|
||||
place) over a **treemap** of the selected folder — squarified, nested to the file, tiles
|
||||
coloured by kind (video, images, audio, archives & disk images, documents, code & config,
|
||||
databases & logs, other; the validated categorical set, dark and light) with folders as
|
||||
labelled boxes; hover for path, size and share; click a folder to drill, breadcrumbs and ↑
|
||||
to come back. The map asks the server for the folder pruned to what a tile could show
|
||||
(`min` = 1/6000 of the folder, 24 deep; deeper folders come back as one lump), the list
|
||||
one level at a time. **Largest files**: the 1,000 biggest with a path filter. Works on
|
||||
every machine through the hub.
|
||||
**Smarter and faster (2026-09-23, at Mitchell's request):** a scan is *kept* — in memory
|
||||
and in `<data>/space/<target>.scan` (the same varint stream) + `.json`, read back after a
|
||||
restart — and the next scan *starts from it*: the helper is handed the last tree on stdin
|
||||
and reads a directory again only when something says it changed. On a ZFS pool with
|
||||
snapshots, the scan records the newest snapshot of each dataset at start (`bases`), and the
|
||||
next one runs `zfs diff -H -F <base>` per dataset: only the directories it names are
|
||||
read, their ancestors kept but looked into, everything else kept wholesale without a
|
||||
stat. The diffs run side by side and one is given three minutes — `timeout -s KILL`
|
||||
inside sudo, since Debian's sudo gives the command its own session and a process-group
|
||||
kill from outside never reached zfs — after which that dataset is walked by mtime
|
||||
instead. On `tank` a diff is metadata-bound like the walk: 8 s to 3 min per dataset for
|
||||
a handful of changes, and `immich`, which changes constantly, ran past twenty minutes
|
||||
the first time. A refresh of `tank` took 4 min 19 s against 7 min for the full scan. Elsewhere (ext4, or a dataset with no snapshot) a directory whose mtime — to the
|
||||
nanosecond — is unchanged keeps its list and only its kept files are stat'ed again, since
|
||||
a file can grow without its directory noticing. A **cancelled** scan keeps what it walked,
|
||||
marks the folders it never got to (`partial`, "not yet read" in the tree) and **Resume**
|
||||
reads just those. core: full 1.2 s → refresh 0.3 s (2 folders read, 84,755 kept). The
|
||||
status line says which (`method`, `by_root`, read/kept counts); **Forget** drops the saved
|
||||
result so the next scan is full. The old result stays browsable while a refresh runs.
|
||||
**Treemap rethought** the same day: every file a tile grouped by folder — folders are only
|
||||
a thin line and a name strip on the first level — instead of nested boxes with headers,
|
||||
which on Linux's single-child folder chains was a staircase of headers around nothing.
|
||||
Not done: delete from here, a size-over-time history, per-user column choices.
|
||||
- **Updater** — **shipped 2026-09-23** (`internal/selfupdate`, `internal/api/update.go`,
|
||||
the Updates card in `Settings.tsx`, `scripts/gitea-release.sh`). A Ham Lab asks one URL
|
||||
what the newest build is and installs it on request. `update.url` in `config.json` is
|
||||
either a Gitea/Forgejo repo API base (`…/api/v1/repos/<owner>/<repo>`, whose newest
|
||||
release names the version and carries the binaries as assets) or any static directory
|
||||
holding `VERSION`, `SHA256SUMS` and `hamlab-linux-<arch>` — which is exactly what
|
||||
`make release` writes and what `install.sh` already downloads, so the demo site, a
|
||||
folder on a NAS and a Gitea release are all the same thing to it. `"off"` never looks.
|
||||
`update.token` is sent when the source is behind a login, so a **private** Gitea serves
|
||||
its own fleet without the instance being opened to the world. The binary is verified
|
||||
against `SHA256SUMS` before it goes anywhere near the running one, then takes the same
|
||||
path `POST /api/self-update` always did (runs `version`, runs `check` against this
|
||||
machine's config, atomic replace keeping `hamlab.prev`, re-exec in place) — that path is
|
||||
now one shared method, `installBinary`. Version ordering is `install.sh`'s `vnum()` rule
|
||||
ported to Go and tested against it, so the daemon and the installer never disagree about
|
||||
which of two builds is newer; the commit count since the tag counts, the hash does not.
|
||||
Checked ~90 s after boot and daily (`update.check: "never"` switches that off), never
|
||||
installed unasked — an update restarts the daemon and drops open terminals, and the
|
||||
dialog says so. Settings → **Updates**: installed, published, when it last looked, the
|
||||
source, the release notes, Check for updates, Install; the rail badges Settings when a
|
||||
newer build is out (`update_available` rides on `/api/me`). Publishing is
|
||||
`make gitea-release` → `scripts/gitea-release.sh`, which creates the release and uploads
|
||||
the five architectures, `SHA256SUMS`, `install.sh` and `VERSION`; it is deliberately
|
||||
**not** part of `make deploy`, because deploy pushes this build to the machines this hub
|
||||
owns while publishing offers it to every install anywhere, and one tag per deploy would
|
||||
be noise. Re-publishing a version needs `--force`.
|
||||
**Public since 2026-09-23**, at Mitchell's instruction and after the trade was put to him
|
||||
twice: Gitea's instance-wide `REQUIRE_SIGNIN_VIEW` is off and `mitchell/hamlab` is marked
|
||||
public, so anyone can browse it, clone it and download a release with no account. The
|
||||
other five repos there are each marked private and are invisible — an anonymous request
|
||||
for one is a 404, not a login prompt — registration is still disabled, and new repos
|
||||
still default to private. No token is needed by anything any more (the read-only one was
|
||||
revoked; only `hamlab-publish`, which creates releases, remains). Gitea resolves
|
||||
`releases/download/latest/<asset>`, so `install.sh` and `HAMLAB_URL` now point straight
|
||||
at the newest release and never need changing. What is public is the source, the history,
|
||||
`BUGS.md` and this file — which describes the lab and names the console's hostname.
|
||||
- **Auth hardening** — WebAuthn passkeys (phone Face ID), TOTP fallback, optional NPM
|
||||
access list restricting to WireGuard/LAN.
|
||||
- **Files tab** — **shipped 2026-09-18** (`internal/files`, `Files.tsx`): browse from
|
||||
`/` with breadcrumbs, an editable path box, sort, hidden-file toggle and a filter;
|
||||
rows show type icon, size, modified, perms and owner, link targets and broken links.
|
||||
Text files (≤ 2 MiB, sniffed) open in an editor with Ctrl+S / Save; images preview;
|
||||
everything else downloads. Upload via drag-and-drop or picker with per-file progress
|
||||
(streamed, no size limit). New folder / file, rename, move, chmod, copy path, delete
|
||||
(folders need the name typed back), and "open in terminal", which `cd`s the active
|
||||
shell (or a fresh one) to the folder. **Sudo toggle**: root access re-executes the
|
||||
same binary as `hamlab fs <op>` under `sudo -n`, streaming bodies over stdin/stdout,
|
||||
so both modes share one implementation and one set of rules (absolute paths only;
|
||||
`/`, `/etc`, `/usr`, … are never deleted or moved whole). Every write is logged with
|
||||
the caller's IP.
|
||||
- **Previews on the Files tab** — **shipped 2026-10-11** (v0.1.0-200, `internal/thumbs`,
|
||||
`internal/api/thumbs.go`, `Thumb`/`ThumbNote` in `Files.tsx`). Asked for as a button that
|
||||
toggles a file preview, "obviously not for everything", cached, "use whatever GPU is on the
|
||||
machine we're looking at, and fall back to CPU", "a single small frame preview so it's
|
||||
fast". A **previews** button beside `.hidden` turns the icon column into a 256 px
|
||||
thumbnail for every image and video row (the listing says which: `preview: image|video`
|
||||
per entry, decided by extension in one place, `thumbs.Kind`), loaded as rows scroll within
|
||||
400 px of view — the list scrolls in a box of its own, so it has its own
|
||||
IntersectionObserver rather than the browser's window-based lazy loading. A video thumb
|
||||
carries a ▶ badge. The setting is in localStorage and follows the browser.
|
||||
- **ffmpeg is the one engine**, for images and video alike; `GET /api/files/thumb` is one
|
||||
ffprobe and one ffmpeg: `-ss` a tenth of the way in (30 s at most, `-noaccurate_seek`
|
||||
so one keyframe is decoded), the first real video stream (cover art is skipped), one
|
||||
frame, `scale` to fit 256, MJPEG q4 to a pipe. JPEGs decode at a reduced DCT size
|
||||
(`-lowres`, enough halvings to stay at twice the preview) — a 24 MP photo is read as
|
||||
1.5 MP. Thumbnails are the only thing Ham Lab asks a GPU to do, and it asks the way the
|
||||
Processes tab reads one: **probed, never assumed.** The probe looks ffmpeg and ffprobe
|
||||
up, opens every `/dev/dri/renderD*`, pushes a synthetic frame through each with VAAPI,
|
||||
and reads `vainfo`'s decode profiles per node where it exists; CUDA is tried the same
|
||||
way when `/dev/nvidiactl` is there (untested: nothing here has one). A video goes to the
|
||||
first GPU listing its codec; ffmpeg's own software fallback sits behind that for a
|
||||
profile the list did not distinguish, and a CPU retry behind a hardware run that fails
|
||||
outright. Images are CPU only — one frame through libjpeg at a quarter size beats a
|
||||
round trip through a GPU. Measured on `grid`: 1080p h264 0.4 s on the UHD 750 against
|
||||
0.7 s on the CPU; the 8K Osmo HEVC 1.0 s against 2.9 s. `grid`'s two GPUs are listed in
|
||||
node order and both decode the same set (h264, hevc, vp9, mjpeg, mpeg2 — **no AV1 on
|
||||
either**, per vainfo, so AV1 is dav1d on the CPU).
|
||||
- **Cached in one database file under the machine's data dir**, `<data dir>/thumbs.db`
|
||||
(bbolt — pure Go, so the binary stays static; v0.1.0-202, the first build kept them as
|
||||
a tree of hash-named JPEGs under `thumbs/`, replaced the same day when Mitchell asked
|
||||
where they were kept: "stored in hamlab, maybe under a DB, up to you"). An entry is
|
||||
keyed by path, size and mtime — a changed file gets a new preview, an unchanged one is
|
||||
never decoded twice — and carries the engine, codec, source size and time beside the
|
||||
picture; a count rides in the file so the previews button's tooltip can say "N kept
|
||||
(M MB)" without walking anything. The URL carries `v=<mtime>-<size>` so the browser's
|
||||
own cache is right too (7 days, ETag, 304). A file ffmpeg could not decode is kept with
|
||||
ffmpeg's reason and no picture, answered as 422, retried only when the file changes; a
|
||||
timeout is not kept. A miss never creates the file, so a machine nobody browses with
|
||||
previews on never has one; a cache that cannot be opened is logged and previews are
|
||||
still made and served, just not kept. Nothing expires: a preview is a few KB, and only
|
||||
files somebody scrolled past are ever made. At most 2–4 ffmpegs at once per machine
|
||||
(half the cores, clamped), one per file however many rows ask, and a preview's work
|
||||
runs on a clock of its own so the row that asked first leaving does not cancel it for
|
||||
the rest. Each ffmpeg runs under `nice -n 10`. `hamlet`'s backup job skips the file.
|
||||
- **Root mode makes them as root** (`hamlab thumb`, the sibling of `hamlab fs`, caps on
|
||||
stdin, JSON out), so a file only root can read gets a preview, and the probe runs as
|
||||
root too — root opens every render node whatever groups the daemon has. Which matters:
|
||||
a render node is `root:render 0660`, and on `core` and `hamtv` the `core` user is not in
|
||||
`render`, so there the daemon's own previews are CPU and the tab says so, naming the GPU
|
||||
and the owner, with an **Allow GPU access** button (confirms, then a task writes
|
||||
`hamlab.service.d/gpu.conf` adding `render`/`video` as supplementary groups and
|
||||
restarts Ham Lab three seconds later, the shape the Incus install used for
|
||||
`incus-admin`). Not pressed anywhere yet. Where ffmpeg is missing (the Mac mini, `gate`)
|
||||
the note offers **Install ffmpeg with dnf/apk** through the Updates machinery
|
||||
(`ffmpeg-free` on Fedora, `ffmpeg` elsewhere); the status re-probes when the task ends.
|
||||
- *Not done, on purpose:* no cap on the cache and no clear button (not asked; it only
|
||||
holds what was looked at); no tonemapping for HDR sources (a 10-bit HEVC frame comes out
|
||||
flat); no preview in the viewer for a video (it still says "binary, download it").
|
||||
- **Move… is a folder picker** — **shipped 2026-10-11** (v0.1.0-204, `MovePicker` in
|
||||
`Files.tsx`, `internal/files/move.go`, `POST /api/files/move`). Asked for as: instead of
|
||||
a path prompt, "an interactive file explorer, where I can go back/forward, etc and then
|
||||
click some button, and it will move the file/folder to where I was". The picker opens in
|
||||
the item's own folder and walks the machine the way the tab does — back and forward
|
||||
through where it has been, up, breadcrumbs, a path box, folders only, `.hidden`
|
||||
following the tab's toggle, **+ folder** to make the destination on the spot — and
|
||||
**Move here** puts the item in the folder being looked at. It says why it can't when it
|
||||
can't, with the button greyed: already here, a folder into itself, a name clash in that
|
||||
folder (read from the listing before anything is tried). The row being moved shows in
|
||||
its own folder, greyed. Rename stays a name prompt: it is a different question.
|
||||
- **Across filesystems it copies, then removes.** `os.Rename` is EXDEV between
|
||||
filesystems, and on `grid` every dataset is one — `/tank/media` to `/tank/files` was a
|
||||
"invalid cross-device link" error before this, and the picker would have made that
|
||||
move easy to attempt. `/api/files/move` renames when it can (204) and otherwise runs
|
||||
`hamlab fs move` as a task (202 with the task; the tab shows its output and reloads
|
||||
when it ends). The copy is built under a hidden name beside its destination
|
||||
(`.<name>.hamlab-moving`), renamed into place once complete, and only then is the
|
||||
original removed — a failure part-way removes the partial copy and leaves the original
|
||||
as it was, and a cancel (SIGTERM to the helper) does the same. Modes, mtimes and
|
||||
symlinks are kept; owners too when the backend is root. Devices, pipes and sockets are
|
||||
refused outright rather than half-copied. Progress every 500 files or 512 MiB.
|
||||
Verified on `grid` ext4 → ZFS with a tree holding a 0600 file, a symlink and an old
|
||||
mtime, all intact on the other side.
|
||||
- **Video and audio play in the viewer** — **shipped 2026-10-11** (v0.1.0-208). Asked for
|
||||
as "keep it simple for now, and just send whatever codec the file is in to the client,
|
||||
and let the client try to decode and watch it" — and that is the whole design. Opening a
|
||||
video or audio file shows a `<video>`/`<audio>` with the browser's own controls, fed the
|
||||
file as it is by `/api/files/read`; nothing is transcoded or remuxed, so what plays is
|
||||
what the browser decodes (Safari: h264/hevc/aac in mp4/mov; Chrome: those plus vp9/av1 in
|
||||
mkv/webm, hevc only where the machine has a hardware decoder; nothing plays MPEG-TS or a
|
||||
GoPro `.360`'s raw EAC strip). When the element errors the viewer says so and leaves the
|
||||
download. What the server gained is the part a player cannot do without: reads of a
|
||||
real file with a known size go through `http.ServeContent`, so **Range requests** (every
|
||||
seek, and Safari's first request) and HEAD work, and a download can resume. The root
|
||||
backend's helper takes `--offset` for that, and `files.Seeker` opens the backend once, at
|
||||
the range asked for, rather than streaming from the top and skipping; seeking costs
|
||||
nothing until a read wants somewhere else. `/proc` files, which stat at size 0, keep the
|
||||
plain streaming path, as does the editor's capped read. Audio became a preview kind at
|
||||
the same time (`thumbs.Kind` → `audio`): its cover art, where it has any, is the row's
|
||||
thumbnail, with a ♪ badge.
|
||||
- *Not done, on purpose:* no playlist or next/previous, no subtitles. (Transcoding came
|
||||
the same evening after all — the next entry — because Safari turned out not to play
|
||||
the gaming captures.)
|
||||
- **Re-encoding for the browser, on the machine's GPU** — **shipped 2026-10-11**
|
||||
(v0.1.0-212, `internal/media/{encode,hls,ffrun,inspect}.go`, `internal/api/play.go`,
|
||||
`Player.tsx`, `hls.js`). Asked for when Safari would not play the gaming captures:
|
||||
"add GPU transcoding to a supported codec when the client browser can't play it. Each
|
||||
machine should use its own GPU (if available) and transcode to probably H.264. Maybe we
|
||||
need a quality selector, and a codec selector, up to you." What Safari actually refuses
|
||||
is the **MKV container** — the captures checked are H.264 High 4:2:0, 18–36 Mbps — so
|
||||
the picker has a "repackaged" choice beside the codecs, and a 36 Mbps capture over
|
||||
WireGuard is the other reason to re-encode at all.
|
||||
- **The viewer tries the file as it is first**, as before. When the browser gives up, or
|
||||
on **Re-encode…** in the bar, a picker: **codec** — H.264, HEVC, AV1, or the original
|
||||
codec repackaged — each labelled with the engine this machine would use (or greyed as
|
||||
"not on this machine"), and **size** — original, 1080p, 720p, 480p; the choice is
|
||||
remembered. Play starts the re-encode and the player follows it: "h264 High 2292×960
|
||||
60 fps 36 Mbps in matroska → H.264 2292×960 5.0 Mbps on Intel Arc A380 (vaapi) · 7.3× ·
|
||||
34% through — seek within what's done", then "all of it is ready". **Change…** drops
|
||||
the stream and goes back to the picker.
|
||||
- **HLS, made on the machine in the switcher.** `POST /api/files/play` inspects the file
|
||||
(ffprobe, as root in root mode) and runs ffmpeg into an *event* playlist of 4 s
|
||||
fragmented-MP4 segments under `<data dir>/transcode/<id>/`, answering once the first
|
||||
segment exists (0.5–1 s on `grid`); the player reads the playlist and segments from
|
||||
`/api/files/play/<id>/…` (only those three names are served), Safari natively and
|
||||
everything else through hls.js, loaded only then. ffmpeg runs ahead at the engine's
|
||||
pace, so the seekable range grows and the whole file is through long before it is
|
||||
watched. Keyframes are forced every 4 s so segments cut cleanly; sound is AAC stereo
|
||||
unless it already is AAC; HDR is tone-mapped (`tonemap_vaapi`, or zscale/hable on the
|
||||
CPU). Nothing is kept: the same request within ten minutes reuses the stream, the
|
||||
player closes it when it goes (and the "Change…" path does), the sweeper closes what
|
||||
nobody touched for ten minutes, and the folder is emptied when the daemon starts. At
|
||||
most three at once per machine (429 with the reason). Root mode runs ffmpeg as root
|
||||
through `hamlab run-ffmpeg`, which ends when the daemon lets go of its stdin — so a
|
||||
stream cannot outlive the console.
|
||||
- **Engines are probed, never assumed**, the same probe the previews use, now also
|
||||
reading each node's encode profiles from vainfo and ffmpeg's own encoder list. The
|
||||
recipes for a codec are tried best first until one produces a first segment: a card on
|
||||
the bus before the integrated GPU (the A380 before the UHD 750), decoding on the same
|
||||
card so frames never leave it, then decoding on the CPU and uploading (a source the
|
||||
card cannot decode — AV1 on `grid`), then libx264 / libx265 / SVT-AV1. The size is
|
||||
never larger than the source, and the bit rate follows the output size (2/4/8/12/20
|
||||
Mbps for 480p/720p/1080p/1440p/2160p, +25 % at 50 fps and up, ×0.6 for HEVC and AV1).
|
||||
Measured on `grid` with a 2292×960 60 fps capture: H.264 7.3× realtime, HEVC 720p
|
||||
11.7×, AV1 720p 13×, all on the Arc. `core` and `hamtv` re-encode on the CPU until
|
||||
*Allow GPU access* is pressed (libx264 at 9.4× for a 720p→480p clip on `core`); the Mac
|
||||
and `gate` need ffmpeg first, which the previews note offers to install.
|
||||
- *Not done, on purpose:* no restart-from-a-seek beyond what is encoded (the encode is
|
||||
faster than watching; a seek far ahead waits), no bit-rate ladder or adaptive
|
||||
switching, no subtitles, no keeping a finished re-encode for next time.
|
||||
- **Files edge sweep** — **2026-10-11** (v0.1.0-214), asked for as "a full bug sweep of
|
||||
the Files section, every example and edge case". Every handler and both UI files read
|
||||
through with odd inputs in mind, then a battery of ~60 requests against the live hub
|
||||
(`sweep.py`, not kept: listings of files, missing and forbidden paths, NUL bytes, names
|
||||
with spaces/#/%/?/umlauts, links to files, folders and nothing, a pipe, over-size saves,
|
||||
uploads with bad names and clashes, renames and moves onto things, protected paths,
|
||||
thumbnails of empty and mislabelled images, playback of an image and an audio-only mp4,
|
||||
root-mode variants). One real data bug: **a save over the 8 MiB cap truncated the file
|
||||
and kept the first 8 MiB, answering 204** — the body is now read whole first and refused
|
||||
with 413, the file untouched. The rest: a link to a file showed the link's own time; a
|
||||
link to nothing read as an empty 200; a bad codec or size was answered by inspecting
|
||||
the file; a failed stream was handed back to the next identical request; a player
|
||||
leaving mid-start made the recipe loop try the next engine; a root-mode ffmpeg's end
|
||||
could wait forever; a page closing kept its stream for ten minutes (a beacon now
|
||||
releases it at once); the path box took only absolute paths (it resolves like a shell
|
||||
now, in the tab and the Move picker, with a test); a dropped folder became a nameless
|
||||
empty file; the upload toast counted intentions; the viewer called a pipe a binary
|
||||
file. Not bugs, left alone: `rm -rf` of a mount point empties it (the console is
|
||||
root-capable by design); a folder of ten thousand rows is slow (no virtual list).
|
||||
- **Whole-console sweep** — **2026-10-13** (v0.1.0-216), asked for as "a full sweep of
|
||||
Ham Lab again, look for any bugs and fix them". Four nets, then reading: the Go race
|
||||
detector over the whole suite (clean), staticcheck's development build (the release does
|
||||
not know Go 1.26; it found only dead code and one capitalised error string), a
|
||||
garbage-in battery over every route safe to poke — 202 routes, bad identifiers, empty
|
||||
and malformed bodies, with the destructive ones held back — and a headless Chromium pass
|
||||
through all twenty tabs on the hub and on `grid`, collecting exceptions, console errors
|
||||
and failed requests (none). By hand: the auth middleware and sessions, token issue, the
|
||||
hub→node and KVM proxies, every validation regex before a shell, the cron and cron.d
|
||||
writers, config and state atomicity, every WebSocket and SSE loop's exit, the terminal
|
||||
manager, the task manager, and every destructive confirmation in the UI. Fixed, all
|
||||
small: docker inspect/health of a missing container was a 500 (404 now, a daemon that
|
||||
is down 503), `/api/virt/instances/{name}` without Incus was a 502 (409 like its
|
||||
siblings), a ZFS snapshot delete with a bad name was a 500 (400; 404 for one that does
|
||||
not exist), the task list read a task's Running flag outside its lock (a real data race,
|
||||
never seen), the Virtualization export config was written in place rather than
|
||||
atomically, and the dead code. Nothing in auth, the proxies or the shell boundaries.
|
||||
One self-inflicted finding: the battery's skip list left out `POST
|
||||
/api/files/thumbs/gpu-access`, so the sweep pressed *Allow GPU access* on `core` three
|
||||
times over — the drop-in landed and the hub restarted (that is what the headless pass
|
||||
saw as a refused connection). Kept, since it is the fix the tab offers; `core` makes
|
||||
previews and re-encodes on its UHD iGPU now. The handler refuses a second run while one
|
||||
is in flight (409), since the second `systemd-run` only collided with the first's timer.
|
||||
- **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth.
|
||||
- **Notifications** — push on service failure / SMART warning / UPS event.
|
||||
- **Terminal extras** — search, copy-mode, font/theme, split panes, upload via
|
||||
drag-drop into cwd.
|
||||
|
||||
- **Settings + themes** — **shipped 2026-09-18** (`Settings.tsx`, `themes.ts`): Settings
|
||||
page with Appearance (Dark, Light, Cyberpunk, Retro — CSS variables on `<html>` plus a
|
||||
matching xterm palette, applied before first paint from localStorage and synced hub-side
|
||||
via `GET/PUT /api/prefs/ui` so the choice follows the login), Account (change password,
|
||||
`POST /api/password`, current password required), Access tokens (list/revoke the tokens
|
||||
hubs hold), About (version, host, arch, uptime, certificate).
|
||||
|
||||
## Network tab (2026-09-27)
|
||||
|
||||
A **Network** tab, per machine like every other: the interfaces as the kernel has them
|
||||
(addresses, MAC, MTU, negotiated link speed, carrier, byte totals, live rates from the
|
||||
sampler's own sensors), the default route and resolvers, what the internet sees of this
|
||||
connection, and an **internet speed test**.
|
||||
|
||||
Decisions worth keeping:
|
||||
|
||||
- **Measured by the daemon, not the browser.** The test runs on the machine in the switcher,
|
||||
so the figure is that machine's line — the router's, if you switch to `gate` — and it
|
||||
carries on if the phone locks or the tab closes. The page polls a run it is not driving.
|
||||
- **No tool to install.** It measures against Cloudflare's public speed endpoints
|
||||
(`__down`, `__up`, `cdn-cgi/trace`), which is what makes it work identically on Debian and
|
||||
on OpenWrt, where `opkg install` costs flash. That is the feature's one external
|
||||
dependency, and it is named in `internal/network/speedtest.go`.
|
||||
- **Several streams, HTTP/2 off, and the ramp thrown away.** One stream over one connection
|
||||
measures a congestion window; HTTP/2 would put every stream back on one connection and do
|
||||
the same. The first second of each phase is measured and discarded.
|
||||
- **Latency is the round trip less the far end's own Server-Timing.** A zero-byte response
|
||||
is mostly Cloudflare's worker running — 20 to 45 ms of it against a few ms of wire. Left
|
||||
in, it reported 31 ms for a 7 ms line. Jitter is the mean change between consecutive
|
||||
probes, not a standard deviation, so one slow probe doesn't run away with it.
|
||||
- **A refused phase is not a number.** The far end rate-limits by volume, and a test moves a
|
||||
few hundred megabytes; several in a row earn 429s, and the block observed while building this
|
||||
lasted over twenty minutes. So: a 20-second cooldown counted from the last *attempt* (a
|
||||
refused test still used the line), a stream that retries twice before giving up, the rate
|
||||
computed over the window in which bytes were actually still arriving, and a plain sentence
|
||||
when it was cut short. Coasting to the end of the clock with nothing arriving reported a
|
||||
third of the truth.
|
||||
- **A second download source, for when the first refuses.** A plain 100 MB file on Linode's
|
||||
Seattle mirror, which has no opinion about volume. It is only reached after a refusal, and
|
||||
every result names the source it used — that mirror is in one place, so on a machine far from
|
||||
Seattle it would read low, and a number that quietly came from somewhere else would be a lie.
|
||||
Upload stays on Cloudflare, which has never been the one to refuse.
|
||||
- **The last result is published as sensors** (`net.internet.*`), so the line is a widget, a
|
||||
graph, an alert or a Prometheus series like anything else. Each carries the time the test
|
||||
ran, not the time it was asked for.
|
||||
- **Not scheduled.** Nothing takes a speed test on its own. A nightly one recorded to the
|
||||
history is the obvious next step and deliberately not built yet — the cron shape for it
|
||||
already exists twice over (ZFS snapshots, Docker image updates).
|
||||
|
||||
## VPN tab (2026-09-27)
|
||||
|
||||
A **VPN** tab beside Network, per machine: every WireGuard tunnel on the machine in the
|
||||
switcher, and the devices on it — name, address, last handshake, where it connects from, and
|
||||
what it has moved.
|
||||
|
||||
- **Per machine is the point, not a limitation.** The tunnel people dial into and a tunnel a
|
||||
machine dials out through are on different boxes here (`gate` and `hamtv`), and a combined
|
||||
view would invite exactly the question "which box is this on".
|
||||
- **The kernel is the truth, the config only supplies names.** `wg show all dump` has the
|
||||
handshakes and counters and knows nothing but public keys; names come from the `# comment`
|
||||
above a peer in a wg-quick file, or a UCI section's `description` on OpenWrt. Both are read
|
||||
only to label what the kernel already said.
|
||||
- **Server or client is worked out, not declared.** A peer allowed `0.0.0.0/0` that has an
|
||||
endpoint is a way out, so a tunnel holding one is this machine dialling out; a tunnel with a
|
||||
listen port and peers is one being dialled. Anything else says neither rather than guessing.
|
||||
- **"Online" is a handshake in the last three minutes**, and the tab says so. WireGuard is
|
||||
silent when idle; there is no connection to be in, and a tab claiming otherwise would be
|
||||
lying twice a day.
|
||||
- **No secrets are read.** Private keys are in the second column of the dump and are read
|
||||
past; pre-shared keys are noted as present and never read.
|
||||
- **Devices can be created, renamed and removed** (added 2026-09-27, asked for the same day),
|
||||
on a tunnel the machine configures — today OpenWrt's UCI, which is where the server here
|
||||
lives. A wg-quick host is reported unmanaged and shows no buttons: half-supporting a second
|
||||
configuration format is worse than an honest refusal.
|
||||
- **Nothing restarts the network.** `uci commit` writes the file and applies nothing; `wg
|
||||
set` changes one peer on the live interface. Proven on `gate` by the existing peers'
|
||||
transfer counters surviving a create and a delete — a reload would have zeroed them.
|
||||
- **The two halves are ordered so they cannot disagree quietly.** Adding writes the
|
||||
configuration first and rolls it back if the live tunnel refuses, because a device that
|
||||
works only after the next reboot is the worst of the three outcomes. Removing does the
|
||||
live tunnel first, since that is the moment access ends, and says so plainly if the
|
||||
configuration half then fails.
|
||||
- **The keypair is generated on the machine** and the profile, with its QR, is shown once.
|
||||
The private key is stored nowhere — the same bargain as a guest login's password.
|
||||
- **Removing asks for the name to be typed back**, at both ends. It cannot be undone and it
|
||||
sits one tap from a list of devices that all look alike.
|
||||
|
||||
## To do
|
||||
|
||||
- **Grow the Files tab** — asked for 2026-09-23 as two items (Files polish, and deprecating
|
||||
copyparty), merged 2026-09-24. **copyparty was retired 2026-10-11** at Mitchell's call,
|
||||
ahead of these gaps rather than after them: service, files and NPM host all gone, with a
|
||||
rollback archive kept on `grid` (`hamlet`'s `docs/hosts.md` has the detail). Files is now
|
||||
the only file browser on `grid`, so the list below is no longer parity — it is the polish
|
||||
still owed, in the order that matters:
|
||||
- **Multi-select and bulk actions.** copyparty selects many rows and moves, deletes or
|
||||
downloads them together; Files acts on one row at a time.
|
||||
- **Download a folder**, and a multi-file download, as a streamed zip or tar — the thing
|
||||
copyparty is reached for most.
|
||||
- **Resumable uploads.** copyparty's up2k chunks a file, survives a dropped connection
|
||||
and skips what is already there; Files streams one PUT, so a phone upload that drops at
|
||||
90% starts over.
|
||||
- **A gallery.** ~~Thumbnails for a folder of images and video~~ (the **previews** toggle,
|
||||
2026-10-11 — see the shipped entry above; a grid view is not built), and play
|
||||
audio/video in place — `/tank/media` and `/tank/immich` are most of what is browsed.
|
||||
- **Search** below the current folder by name, since `find` in a terminal is the fallback
|
||||
today.
|
||||
- *Decide when built:* whether share links (copyparty's public per-path links) are wanted
|
||||
at all — they are the one feature that hands out access without a login, and Ham Lab has
|
||||
no notion of that; if not, say so in the docs rather than leaving a gap unexplained.
|
||||
|
||||
- **Grow the Docker tab until Dockhand can go** — asked for 2026-09-23 and widened
|
||||
2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the
|
||||
Files item: it is one tab, and the gaps that block retiring the old tool and the things
|
||||
that would simply make it better are the same body of work. The *homelab* half —
|
||||
the cutover itself, then moving every stack off Dockhand's data dir — is `hamlet`'s
|
||||
`docs/pending.md` → "Retire Dockhand; Ham Lab becomes the orchestrator", agreed
|
||||
2026-09-20. This entry is the *app* half.
|
||||
|
||||
**Done 2026-09-24 — the eight Mitchell picked from the inventory**, all in the Docker tab:
|
||||
per-container **Stats**, **Health** (with the last few probe outputs), **Events** (Ham Lab
|
||||
keeps its own 4000-entry ring, since `docker events` forgets after about a minute),
|
||||
**Shell** (`docker exec`, handed to the Terminal tab) and **Limits** (`docker update` for
|
||||
memory, CPU and restart policy); an **updates** tab giving the whole machine at once with
|
||||
a per-container indicator in the list; and **schedules**, which write
|
||||
`/etc/cron.d/hamlab-docker` and call
|
||||
`hamlab docker-update <id>` — the shape the ZFS schedules already use.
|
||||
Also done: Ham Lab keeps working when `dockhand` is removed. It remembers where a
|
||||
container's bind mounts pointed (`mounts` in `state.json`) and resolves a stack through
|
||||
that record when no running container explains its path, running compose on the host
|
||||
instead; `repoint` does the same for a schedule at run time. Without this, removing the
|
||||
container would have made 17 stacks on `core` and 5 on `grid` unfindable at once.
|
||||
|
||||
- **Left from the Dockhand inventory, both to decide rather than build.**
|
||||
- *Git-sourced stacks.* Dockhand's `stack_sources` table points a stack at a repo and
|
||||
pulls it. Its `git_repositories` and `git_stacks` tables are **empty**, so nothing
|
||||
here uses it — dropping it is the honest answer unless Mitchell says otherwise.
|
||||
- *Webhooks, OIDC, LDAP, vulnerability scans.* All four tables empty. Nothing to move.
|
||||
|
||||
**Done 2026-09-25 — the two gaps in the tab itself.**
|
||||
- *Logs are followed over a held connection*, the way the Services tab follows a journal
|
||||
(`internal/api/docker_logs.go`, `/api/docker/containers/{id}/logs/tail`): one
|
||||
`docker logs -f` per viewer, stdout and stderr both, stderr coloured as an error. A
|
||||
reconnect resumes from the last line's timestamp, so a container that restarts or a
|
||||
phone that slept doesn't replay what is already on screen; the client dedupes by line
|
||||
rather than by "later than", because `--since` is inclusive and two lines can share a
|
||||
timestamp. Docker replays the opening backlog down two pipes in no particular order, so
|
||||
the first 300 ms is held and sorted before it is sent. The polled `?since=` endpoint
|
||||
stays — it is what the resume is built on. Reconnects back off to 15 s, since a
|
||||
container that has stopped ends the stream immediately.
|
||||
- *Networks can be removed and pruned* — `POST /api/docker/networks/{remove,prune}`, the
|
||||
shape images and volumes already had. The overview says who is attached to each one, so
|
||||
the ✕ is only offered on a network nothing uses and never on Docker's own three. Prune
|
||||
is deliberately untested against this host: `core` has several networks belonging to
|
||||
stacks that are merely down (`pihole_default`, `transmission_default`, `gluetun_default`
|
||||
among them), and pruning would take them and hand them new subnets on the next deploy.
|
||||
|
||||
- **The `*:2375` question — settled, and the code removed 2026-10-10.** The hub asks the
|
||||
node's own Ham Lab; nothing probes or uses a node's Docker port. `hubStacks`,
|
||||
`nodeDocker`, `taskOnHub`, the hub-side `/api/nodes/{node}/docker/…` routes and the
|
||||
`remote` / `held_by` fields are gone. A schedule saved before this that still carries a
|
||||
`remote` is refused with a message, never run against the wrong Docker. `PORTABILITY.md`
|
||||
§3 has the audit it came from. Deployed as v0.1.0-191 and checked on all five machines
|
||||
through the hub.
|
||||
|
||||
- ~~**A Virtualization tab**~~ — built 2026-10-09, **cut 2026-10-15**: out of scope (see Decisions). The code, its plan and its tests are on the `virtualization` branch.
|
||||
|
||||
|
||||
## Open questions
|
||||
|
||||
- Idle-session reaping policy (never / after N days) — leaning never; it's a
|
||||
|
||||
+161
@@ -0,0 +1,161 @@
|
||||
# Portability audit — what in Ham Lab is specific to this lab (2026-10-09)
|
||||
|
||||
Goal: Ham Lab drops onto any set of Linux machines and works. This is the full sweep of
|
||||
every place the code, defaults, docs or UI assume Mitchell's hosts, network, hardware,
|
||||
distro, init system, orchestrator history or posture. **Nothing here is fixed yet** — this
|
||||
is the list to work from. Each item: where, what it assumes, what the general form is.
|
||||
|
||||
Virtualization was cut from the app 2026-10-15 (kept on the `virtualization` branch); its
|
||||
items — 1.8, 4.2, 5.1, 5.2, 6.14 and the Incus halves of 5.3, 5.7, 6.15, 10.3 — went with it.
|
||||
|
||||
Severity key: **A** breaks or misbehaves on an ordinary host · **B** works but is wrong by
|
||||
default or phones home to this lab · **C** cosmetic (placeholder text, comments, docs).
|
||||
|
||||
---
|
||||
|
||||
## 1. Hard-wired lab identifiers
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 1.1 | **B** | `internal/selfupdate/selfupdate.go:36` `DefaultURL = "https://git.bigham.dev/api/v1/repos/mitchell/hamlab"` | Every install with no `update.url` checks **this** Gitea 90 s after boot and daily (`internal/api/update.go:1951, 2016`). A stranger's install phones home here by default. | Default should be **off** (or a build-time `-X` variable set at release). Installer asks or `install.sh` passes `HAMLAB_URL` through into `update.url`. |
|
||||
| 1.2 | **B** | `install.sh:4, 27` `HAMLAB_URL` default; `README.md:23-25, 31` | The curl one-liner and the install source are this Gitea. | Default comes from the release that served the script (`@URL@` filled by `make release`), or is required. |
|
||||
| 1.3 | **B** | `scripts/gitea-release.sh:14` `REPO` default | Publishing goes to `mitchell/hamlab` on this Gitea. | Require `HAMLAB_REPO`; no default. |
|
||||
| 1.4 | **C** | `README.md:100` "On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager" | Lab deployment fact in the product README. | Move to `hamlet/docs/hosts.md`. |
|
||||
| 1.5 | **C** | `examples/external-demo/index.html:8` `base: 'http://10.0.0.234:8100'` | Demo config names the hub's LAN IP. | `http://<hamlab-host>:8100`. |
|
||||
| 1.6 | **C** | `SENSORS.md:149` Prometheus target `10.0.0.234:8100`; `:187` `&machine=grid`; `:196-208` Glance examples at `hamlab.bigham.dev`, `zfs.tank.used_pct`, `/api/nodes/grid/...`; `:218-220` `grid,wyse`, `{"core":…,"grid":…,"mac":…}` | Docs use real host names, pool and domain. | Generic placeholders (`<host>`, `<machine>`, `<pool>`). |
|
||||
| 1.7 | **C** | `PLAN.md:3-4` "bare-metal Debian 13 host. First target: `core` (10.0.0.234)"; decisions table rows (Listen `10.0.0.234:8100`, Data `/home/core/apps/hamlab/`, Ingress NPM/`hamlab.bigham.dev`, Repo) | Product plan is written as this lab's plan. | Keep as history, or split lab facts out to `hamlet`. The repo is public. |
|
||||
| 1.9 | **C** | `BUGS.md` throughout | Bug bashes narrate `core`, `grid`, `wyse`, `hamtv`, `gate`, the Mac. | History; fine to leave, but it is public. |
|
||||
| 1.10 | **C** | `cmd/hamlab/main.go:1` "Proxmox-style web console for a bare-metal Debian host"; `internal/hardware/hardware.go:5` "what Debian and its kin already have"; `internal/hardware/hardware.go:29` `// "GMKtec NucBox G3"`; `internal/metrics/metrics.go:1358` "did, on grid, 2026-10-02"; `internal/api/server.go:1382-1383` `["grid", …]`, `?machines=grid,wyse`; `internal/api/stacks.go:10` `/home/core/apps/dockhand/stacks/…`; `internal/api/dockersched.go:5` "unbound, nginx and pihole"; `internal/api/zfs.go:5` "grid already runs zfSnap"; `internal/storage/snapshots.go:6` "grid has run zfSnap v1.11.1"; `internal/api/guests.go:8` "Mitchell, 2026-09-27"; `internal/network/network.go:306` "the public address here is a DHCP lease"; `internal/metrics/gpu.go:17` "Asahi, today"; `internal/metrics/pciids.go:4-6` "the router's GPU"; `web/src/Top.tsx:522, 576`; `web/src/hashsel.ts:1-9`; `web/src/Embed.tsx:3` `&machine=grid` | Comments explain decisions by naming lab hosts. | Harmless, but rewrite the ones in package doc comments so a reader of the source doesn't need the lab to understand the code. |
|
||||
|
||||
## 2. UI placeholders and defaults that are this lab's
|
||||
|
||||
| # | Sev | Where | Text | General form |
|
||||
|---|---|---|---|---|
|
||||
| 2.1 | **C** | `web/src/Machines.tsx:116-118` | `placeholder="grid"`, `"https://10.0.0.214:8100"`, `"core"` | `nas`, `https://192.0.2.10:8100`, `admin` or the hub's own user. |
|
||||
| 2.2 | **C** | `web/src/KVM.tsx:166` | `placeholder="10.0.0.217"` (the NanoKVM on `grid`) | `192.0.2.20` or `kvm.local`. |
|
||||
| 2.3 | **C** | `web/src/Settings.tsx:242` | `placeholder="cpu.*, temp.*, zfs.tank.*"` | `cpu.*, temp.*, fs.*`. |
|
||||
| 2.4 | **C** | `web/src/Settings.tsx:225` | read-only token default name `'glance'` | Empty, or `dashboard`. |
|
||||
| 2.5 | **C** | `web/src/Alerts.tsx:38, 40` | `"Tank nearly full"`, `"zfs.tank.used_pct"` | `"Disk nearly full"`, `"fs.root.used_pct"`. |
|
||||
| 2.6 | **C** | `web/src/StackNew.tsx:110, 113, 114` | `/home/core/apps/whoami:/data`, `TZ=America/Los_Angeles`, `com.example.owner=mitchell` | `./data:/data`, `TZ=UTC`, `com.example.owner=you`. |
|
||||
| 2.7 | **C** | `web/src/Gallery.tsx:69`, `web/src/WidgetPicker.tsx:36` | `"find — gitea, temp, enp3s0…"`, `"filter sensors — cpu, temp, fan, enp3s0…"` | Drop the container and NIC names. |
|
||||
| 2.8 | **C** | `web/src/VPN.tsx:105` | default DNS for a new device `1.1.1.1, 1.0.0.1` | Fine as a public default, but should offer the tunnel's own address (the home resolver) first. |
|
||||
| 2.9 | **B** | `web/src/ZFS.tsx:212` `BLANK` | new schedule defaults `at: '02:30'`, `keep_for: '6w'` — `grid`'s old zfSnap cron line | Neutral defaults (`03:00`, `4w`), or none. |
|
||||
| 2.10 | **A** | `web/src/ZFS.tsx:302` `adopt()` | `/\b(tank[\w/-]*)\b/` — the dataset of an adopted external job is found by looking for the word **`tank`** | Parse the dataset from the command as the last `zfs`-shaped argument, or match against the datasets the page already has. On any other pool name adopt silently picks the fallback. |
|
||||
|
||||
## 3. Topology inherited from the lab (hub `core`, Dockhand, `tcp://:2375`)
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 3.1 | **done** 2026-10-10 | `internal/api/stacks.go` `nodeDocker()` (removed) | For every node whose stacks aren't editable locally, the hub **probes `tcp://<node-host>:2375`** with `docker -H … version` and, if it answers, deploys to the node over an unauthenticated Docker socket. This is the Dockhand-era layout; on any other network an open 2375 is either absent or a security problem the console should not quietly use. | Remove the probe. The node's own Ham Lab deploys its stacks (the files live on the node, under its data dir). `PLAN.md` "To do" already names this decision. |
|
||||
| 3.2 | **done** 2026-10-10 | `hubStacks()`, `taskOnHub()`, the hub-side `/api/nodes/{node}/docker/…` routes, `Stack.remote`/`held_by` (all removed; the generic node proxy carries those requests) | A hub can hold compose files for a node and run compose remotely; new stacks for a node may be created **on the hub**. Only true here because Dockhand on `core` deployed `grid` and the Mac. | Same as 3.1: one machine, one set of files, one Docker. Hub-held stacks become an import path (copy to the node) rather than a steady state. |
|
||||
| 3.3 | **done** 2026-10-10 | `internal/api/stacks.go` `homesOf()` — `<DataDir>/stacks` is always offered first and default; the `~/stacks` fallback is gone; `TestHomesDefaultToDataDir` | Where new stacks go is **discovered** from where ≥2 stacks already are (Dockhand's `stacks/<machine>` habit), else `~/stacks`. Neither is Ham Lab's data directory, which the project's own rule says everything managed must live under. | Default home = `<DataDir>/stacks`. Discovered homes stay as *additional* places, never the default. |
|
||||
| 3.4 | **B** | `internal/api/stacks.go:100-203` remembered mounts, `internal/api/dockersched.go:772-795` `repoint()` | Machinery exists to find compose files **inside a removed orchestrator container's filesystem**. Generic in form, but it exists because of Dockhand and is exercised only here. | Keep (it is general), but it is not a reason to keep 3.1/3.2. |
|
||||
| 3.5 | — | `internal/api/stacks.go` `stackAction`, `internal/compose/compose.go` | Inside a `Via` container the command is literally `docker compose`. | **Checked 2026-10-10, no change:** the CLI inside an orchestrator's container is that container's own `docker`, independent of what the host runs; the host's tool only chooses the `exec`. Both places now say so in a comment. |
|
||||
| 3.6 | **B** | `internal/api/stacks.go:908` `docker buildx imagetools inspect` | Update checks need the **buildx** plugin. Docker CE on Debian has it; Podman and minimal Docker installs do not → every image reports "the registry didn't answer". | Fall back to `skopeo inspect` or a direct registry `HEAD` for the digest. |
|
||||
| 3.7 | **B** | `Makefile:445-448` `deploy` | The machine you build on **is** the hub: installs locally, then `push-nodes`. | Fine for a developer; document that `deploy` means "I am the hub". A plain `make install` target for a single box. |
|
||||
| 3.8 | **B** | `internal/nodes/nodes.go:316-323` | A hub pushes other architectures only if `<prefix>/hamlab-<arch>` siblings exist, which only `make deploy`/`install.sh`-from-dist put there. | A hub without siblings should fetch the right build from `update.url` instead of erroring. |
|
||||
|
||||
## 4. Init system: systemd assumed where the product promises procd and OpenRC too
|
||||
|
||||
The installer supports systemd, procd and OpenRC (`internal/setup/setup.go:217-262`). These paths do not.
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 4.1 | **A** | `internal/updates/updates.go:537-543` `Power()` | `systemctl reboot` / `systemctl poweroff`. On OpenWrt (`gate`) and OpenRC the Power dialog fails. | Try `systemctl`, else `reboot` / `poweroff` (busybox, OpenRC, sysvinit all have them). |
|
||||
| 4.3 | **A** | `internal/services/services.go:163-170` | Services tab is systemd-only ("says so cleanly"). `gate` (procd) and any OpenRC/runit/s6 box have no Services tab. | `rc-service`/`rc-status` (OpenRC), `/etc/init.d/<x> status` + `ubus call service list` (procd), `sv` (runit) backends. Noted as "not done" in `PLAN.md:487-489`. |
|
||||
| 4.4 | **A** | `internal/cron/cron.go:353-373, 401, 442, 532-534, 669-670` | Jobs tab reads `/etc/crontab`, `/etc/cron.d`, `/var/spool/cron{,/crontabs}` (Debian/cronie), `/usr/sbin/anacron`, last-runs from **journald**, timers via `systemctl`; writes only `/etc/crontab` and `/etc/cron.d`. Busybox crond (OpenWrt, Alpine) keeps crontabs in `/etc/crontabs/<user>` and has no `cron.d` → the Jobs tab is empty on `gate` and schedules written by 4.5 never run. | Add `/etc/crontabs/*` as a source; last-runs from syslog/logread when there is no journal. |
|
||||
| 4.5 | **A** | `internal/storage/snapshots.go:402` `/etc/cron.d/hamlab-zfs`; `internal/api/dockersched.go:65` `/etc/cron.d/hamlab-docker` | ZFS snapshot and image-update schedules are cron.d files. No cron.d on busybox crond; no cron at all on a systemd-timer-only host (Fedora Server, CoreOS). | Detect: cron.d → cron.d; busybox → a line in `/etc/crontabs/<user>`; none but systemd → a timer unit. The daemon-side runner already exists (`hamlab zfs-snapshot`, `docker-update`). |
|
||||
| 4.6 | **B** | `internal/hardware/hardware.go:333` `systemd-detect-virt` | Virtualisation detection is systemd-only. | Fall back to `/sys/class/dmi/id/sys_vendor`, `/proc/cpuinfo` `hypervisor` flag, `/sys/devices/virtual/dmi`. |
|
||||
| 4.7 | **B** | `internal/api/server.go:1129`, `internal/updates/updates.go:187` | `/run/reboot-required` is Debian/Ubuntu-only. `updates.Status` falls back to a kernel check for other distros; `/api/me.reboot_needed` (the Machines badge, the header's amber power button) does not. | Share one function. |
|
||||
| 4.8 | **B** | `install.sh:154-157, 164, 375-378`; `internal/setup/setup.go:284` | The wizard's init detection knows systemd and OpenRC only: on OpenWrt it warns "no systemd or OpenRC: nothing starts it at boot" although `hamlab install` then installs a procd script fine; the "logs" hint and the "isn't answering yet" hint have no procd case. | Add procd to both. |
|
||||
| 4.9 | **B** | `internal/setup/setup.go:609-641` `firewall()` | Only firewalld and ufw are detected. OpenWrt fw4/nftables, plain nftables, iptables, Alpine awall are not. | Detect `fw4`/`nft`/`iptables`; on OpenWrt add a LAN-zone rule or say what to open. |
|
||||
| 4.10 | **B** | `internal/setup/setup.go:568-605` `quietSudoers()`; `internal/services/services.go:176-185` | The sudoers quieting list is tuned to what the lab polls (`smartctl`, `pwrstat`, `journalctl`, `zpool`, `zfs`). A host with NUT or apcupsd gets `upsc`/`apcaccess` lines in its auth log at 10 s intervals; a host without `/etc/sudoers.d` (OpenWrt) gets nothing, which is correct. | Build the list from what the collector actually probes. |
|
||||
|
||||
## 5. Distro and package assumptions beyond init
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 5.3 | **B** | `DEBIAN_FRONTEND`, `apt-get … -o Dpkg::Options::=--force-confold` in `internal/updates/updates.go:78, 432` | Debian-isms exported unconditionally. Harmless elsewhere. | Fine; note only. |
|
||||
| 5.4 | **B** | `scripts/pci-ids-gpu.sh:7, 41` default `/usr/share/misc/pci.ids` | Debian's path for `pci.ids`; Fedora/Arch keep it in `/usr/share/hwdata/`. Build-time only. | Try both. |
|
||||
| 5.5 | **B** | `internal/api/status_sensors.go:283`, `internal/api/docker.go:133` | Container memory/CPU read from three cgroup paths: cgroup v2 + systemd driver, cgroupfs, libpod. cgroup v1 hosts (older Debian/Ubuntu, some Alpine, OpenWrt) get no `container.*.mem/cpu`. | Add the v1 `memory/docker/<id>` and `cpu,cpuacct` paths, or fall back to `docker stats --no-stream` once in a while. |
|
||||
| 5.6 | **B** | `internal/metrics/metrics.go:1064` `clkTck = 100` | USER_HZ assumed 100. True on every mainstream build; wrong on a kernel built otherwise. | `sysconf` via `getconf CLK_TCK` once at start. |
|
||||
| 5.7 | **B** | `internal/space/manager.go:578-585` `timeout -s KILL` | GNU `timeout`. | Busybox has `timeout` too but flag spelling differs — use the context deadline. |
|
||||
|
||||
## 6. Hardware and vendor assumptions
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 6.1 | **B** | `internal/metrics/gpu.go:343-383` `read()` | Busy/clock/VRAM are read for **i915, xe, amdgpu** only — the three GPUs in this lab. Nouveau, nvidia-drm, panfrost, v3d, Asahi are listed but never measured. `PLAN.md:317`: "No NVIDIA path: nothing here has one to test against." | `nvidia-smi --query-gpu` where present; `/sys/class/drm/*/device/gpu_busy_percent` for nouveau; `devfreq` for the ARM drivers. |
|
||||
| 6.2 | **B** | `internal/metrics/pciids_gpu.txt`, `scripts/pci-ids-gpu.sh:12-15` | Embedded PCI ID table trimmed to Intel, AMD, NVIDIA and three virtual vendors. Matrox, ASPEED (every server BMC), Moore Threads, etc. fall back to `lspci` or "Vendor GPU <id>". | Include the full display-class device list, not three vendors. |
|
||||
| 6.3 | **B** | `internal/metrics/gpu.go:237-241` | "Integrated" = PCI bus `0000:00:`. Wrong on multi-domain or APU boards where the iGPU hangs off a bridge; cosmetic. | Use `boot_vga` + `PCI_SUBSYS` heuristics, or drop the flag. |
|
||||
| 6.4 | **B** | `internal/metrics/metrics.go:189-196, 917-968` | UPS probe order **pwrstat (CyberPower) first**, then NUT, then apcupsd; pwrstat's `....`-separated output is parsed bespoke. A host with both pwrstat and NUT shows pwrstat only. | Prefer NUT (the vendor-neutral one), then the vendor tools; or list every UPS found. |
|
||||
| 6.5 | **B** | `internal/metrics/metrics.go:1179-1182` `cpuTemp()` | CPU temperature from `coretemp`, `k10temp`, `thermal`/`cpu`, `cpu_thermal`, `soc_thermal`. Misses `zenpower`, `acpitz` as the only sensor, Apple `macsmc` names, many ARM SoC zone names. | Broaden; fall back to the hottest zone when none matches. |
|
||||
| 6.6 | **B** | `internal/hardware/hardware.go:617-627` `jedec` | JEDEC IDs for "the memory makers who turn up in a homelab" — 14 entries. | Embed the JEDEC JEP106 bank list (public, small). |
|
||||
| 6.7 | **B** | `internal/storage/storage.go`, `internal/api/zfs.go`, `internal/storage/arc.go`, `web/src/ZFS.tsx` | A whole **ZFS** tab and ZFS-aware Space/Disks logic; **no btrfs or LVM equivalent** (snapshots, subvolumes, scrub, balance). Not wrong, but the tab list reflects a ZFS NAS. | Later: a Btrfs tab with the same shape, or a generic "Filesystems" tab that shows what the host has. |
|
||||
| 6.8 | **B** | `internal/storage/snapshots.go:1-16, 966-968` | Snapshot naming is **zfSnap's** (`2026-09-24_02.30.01--6w`) because `grid` ran zfSnap. A sanoid/zfs-auto-snapshot user's existing snapshots (`autosnap_…`, `zfs-auto-snap_daily-…`) are detected (`internal/api/zfs.go:1764-1798`) but can never be expired by Ham Lab, and new schedules write a third convention beside them. | Make the naming a per-schedule choice (zfSnap / sanoid / zfs-auto-snapshot), or at least document the convention as the product's rather than inherited. |
|
||||
| 6.9 | **B** | `internal/network/speedtest.go:64-67` `downSources` | Fallback download source is **`speedtest.seattle.linode.com`** — chosen because the lab is in the Pacific Northwest. Everywhere else it reads low. | Pick the nearest of several mirrors, or drop the fallback and report the refusal. |
|
||||
| 6.10 | **B** | `internal/network/speedtest.go:70-74, 604`, `internal/network/network.go:323-334` | Public IP, edge and speed come from **Cloudflare** endpoints, polled every 15 min with no off switch. Air-gapped or privacy-minded installs have no way to disable the outbound probe. | A `network.outside: off` setting; honour it in the watcher. |
|
||||
| 6.11 | **B** | `internal/vpn/edit.go:5-7, 419-421` | Peers can be **created/edited only through OpenWrt UCI** "because that is where the server here is configured". A wg-quick server (the common case everywhere else) is read-only. | Write wg-quick `.conf` too (append a `[Peer]` block + `wg set`), guarded the same way. |
|
||||
| 6.12 | **B** | `internal/vpn/vpn.go:251-253` | Peer names come from wg-quick comments or UCI descriptions — "which between them cover every machine here". NetworkManager and systemd-networkd WireGuard configs are not read. | Add `.nmconnection` and `.netdev` readers. |
|
||||
| 6.13 | **B** | `internal/vpn/edit.go:515` `profile()` | Split-tunnel routes hard-code `10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12` on top of the tunnel's own prefix. | Offer the host's actual LAN prefixes (from `TakenPrefixes`-style route reading), with RFC 1918 as the fallback. |
|
||||
| 6.15 | **B** | `internal/network/network.go:170-199, 303-314`; `internal/vpn/edit.go:558-595`; `internal/tlsauto/tlsauto.go:533-537`; `install.sh:114-120`; `internal/setup/setup.go:648-666` | **IPv4 everywhere**: default route from `/proc/net/route` only, outside view v4 only, peer allocation v4 only, cert SANs v4 only, installer lists v4 addresses only. | `/proc/net/ipv6_route`, v6 SANs, v6 peer addresses. An IPv6-only host today has no gateway, and no cert SAN. |
|
||||
|
||||
## 7. Privilege and deployment posture (the lab's NOPASSWD-everywhere model)
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 7.1 | **A** | `internal/priv/priv.go`; every tab that calls `priv.Available()`; `web/src/Updates.tsx:2144` "give its user passwordless sudo or doas"; `web/src/Jobs.tsx:1883` | Full function needs the service user to have **unrestricted passwordless sudo**, which `hamlet/CLAUDE.md` grants on every host here. The installer neither creates nor offers a sudoers rule, so on a fresh box most tabs are read-only and the UI tells the user to go fix sudoers by hand. | `hamlab install --sudo` writes a **scoped** `/etc/sudoers.d/hamlab` naming the commands Ham Lab runs (smartctl, zpool, zfs, systemctl, docker, journalctl, tee for its cron files, `hamlab fs|space-walk|drm-clients|cpu-energy|replace`), or installs as root. Either way the console works out of the box. |
|
||||
| 7.2 | **A** | `internal/api/server.go:1069-1080` `clientKey()`, `:1083-1092`, `:431-433` `secure()` | The **first `X-Forwarded-For` hop is trusted unconditionally**, because here Ham Lab always sits behind Nginx Proxy Manager. On a host exposed directly, a client sets the header and defeats the per-client login rate limiter and the session IP check. | Trust `X-Forwarded-*` only from configured proxy addresses (`trusted_proxies` in config), else use the socket peer. |
|
||||
| 7.3 | **B** | `internal/api/server.go:1324` `X-Accel-Buffering: no` "nginx (NPM)" | Harmless header; comment names the lab's proxy. | Keep header; generalise comment. |
|
||||
| 7.4 | **B** | `internal/api/guests.go:1-9`, `PLAN.md:21` | Guest logins are full-power ("he gives these to people he trusts"). A product default. | Keep, but it is a posture decision taken for this household; a read-only guest is the generic expectation and was cut. Document as a deliberate choice in README. |
|
||||
| 7.5 | **B** | `internal/api/users.go:1-6` | "a household console, not a tenancy": any signed-in user can add users, make tokens, open a root shell. | Same as 7.4 — fine, but say so where a stranger installs it. |
|
||||
|
||||
## 8. Resource tuning from the lab's smallest host
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 8.1 | **B** | `cmd/hamlab/main.go:451` `debug.SetMemoryLimit(160 << 20)`; `PLAN.md:11` "`core` has ~370 MiB free RAM" | A fixed 160 MiB soft heap limit sized for `core`. On a 64 GiB NAS it forces needless GC under a big Space scan; on a 512 MiB Pi it is still half the box. | Scale from `MemTotal` (e.g. 2–5 %, floor 64 MiB, ceiling 1 GiB), or make it a config key. |
|
||||
| 8.2 | **B** | `internal/space/walk.go:893` `workers := 32`; `internal/space/manager.go:578` 3-minute `zfs diff` patience | Tuned on `grid` (16 cores) and `core`. | Workers from `GOMAXPROCS`; patience configurable. |
|
||||
| 8.3 | **C** | `internal/metrics/metrics.go:32-43` intervals, `History = 300` | Reasonable generic defaults; listed only because 8.1 made them look lab-tuned. | None. |
|
||||
|
||||
## 9. Data directory and legacy paths
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 9.1 | **B** | `internal/config/config.go:115-117` `Candidates()`; `internal/setup/setup.go:413-422`; `README.md:38` | `~/apps/hamlab/config.json` is searched as a legacy location — the lab's pre-v0.2 path, kept so `core`, `grid` and the Mac keep working. | Keep for one more release with a one-time migration log line, then drop. A stranger never had it. |
|
||||
| 9.2 | **done** 2026-10-10 | 3.3 above; `PLAN.md:24` "`grid`'s and the Mac's stacks at `~/stacks/<machine>/` are the standing exception" | The "everything under DataDir" rule has a lab-shaped exception baked into the stacks code. | Default to DataDir; treat `~/stacks` as a discovered home like any other. |
|
||||
| 9.3 | **C** | `internal/setup/setup.go:351` `ownFiles` | Correctly scoped; note only. | None. |
|
||||
|
||||
## 10. Build and release
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 10.1 | **B** | `Makefile:60-61, 442-443`; `scripts/gitea-release.sh` | Publishing is Gitea/Forgejo-only (release API shape). `selfupdate` also understands a static directory, so publishing could too. | `make release` + any static host is already enough; add a GitHub/Forgejo-agnostic `release-dir` note. |
|
||||
| 10.2 | **C** | `scripts/gitea-release.sh:500-515` | Needs `python3` on the publishing host. | `jq`, or do it in Go (`hamlab publish`). |
|
||||
| 10.3 | **C** | `Makefile:392` `ARCHES ?= amd64 arm64 arm 386 riscv64` | Fine. | — |
|
||||
|
||||
## 11. Things that are already general (checked, no change needed)
|
||||
|
||||
Listed so the sweep is known to be complete, not because they need work.
|
||||
|
||||
- `internal/priv` (root / sudo / doas), `internal/setup` service templates, `IP_FREEBIND` listen, self-update atomic replace and re-exec.
|
||||
- `internal/updates`: apt, dnf, zypper, pacman, apk with refresh/upgrade/install.
|
||||
- `internal/storage` block devices: `lsblk -J` with a `/sys/block` fallback, eMMC health, NVMe namespace folding, SMART only when `smartctl` exists.
|
||||
- `internal/metrics` CPU/mem/hwmon/thermal/net/disk/fs from `/proc` and `/sys`; remote-FS and pseudo-FS lists; filesystem statfs with a timeout.
|
||||
- `internal/hardware`: every reader may come back empty with a note; device-tree fallbacks for non-SMBIOS boards.
|
||||
- `internal/nodes`, `internal/tlsauto`: hub ↔ node pairing, pinned certs, proxy, push.
|
||||
- `internal/sensors`, alerts, widgets, embeds, Prometheus: machine-agnostic by design.
|
||||
- `internal/files`, `internal/term`, `internal/tasks`, `internal/auth`, `internal/store`.
|
||||
- `internal/api/kvm.go`: "nothing in it knows anything about any particular KVM" — verified.
|
||||
- `internal/compose` (modulo 3.5).
|
||||
- The React UI: machine-switcher model, hash routing, per-account prefs, themes.
|
||||
|
||||
---
|
||||
|
||||
## Suggested order of work
|
||||
|
||||
1. **Posture and safety first (A):** 7.2 trusted proxies; 7.1 installer-written scoped sudoers; ~~3.1 drop the `:2375` probe (with 3.2 as its consequence)~~ done 2026-10-10.
|
||||
2. **"Any Linux" promises the README already makes (A):** 4.1 power, 4.4/4.5 cron on busybox and timer-only hosts, 4.3 a Services backend for OpenRC/procd, 2.10 the `tank` regex.
|
||||
3. **Lab defaults that phone home or mislead (B):** 1.1–1.3 update/install URLs, 2.9 schedule defaults, 6.9/6.10 speed-test sources and an off switch, 8.1 memory limit, ~~3.3/9.2 stack home = DataDir~~ done 2026-10-10.
|
||||
4. **Breadth (B):** 5.5 cgroup v1, 6.1/6.2 GPUs beyond Intel/AMD, 6.4 UPS order, 6.11/6.12 wg-quick/NetworkManager, 6.15 IPv6.
|
||||
5. **Cosmetics (C):** every placeholder and comment in §1.10 and §2, then split lab facts out of `PLAN.md`, `SENSORS.md`, `README.md`.
|
||||
@@ -1,10 +1,101 @@
|
||||
# Ham Lab
|
||||
|
||||
Web-based Linux management console for bare-metal Debian, in the style of the Proxmox
|
||||
Web-based Linux management console for bare-metal Linux, in the style of the Proxmox
|
||||
web UI. Named after Ham the cat (see also HamTV).
|
||||
|
||||
Status: **planning**. See [`PLAN.md`](PLAN.md) for the MVP scope and roadmap. No code yet.
|
||||
Status: **v0.1** — persistent web terminals and a live Dashboard tab. Scope and roadmap in
|
||||
[`PLAN.md`](PLAN.md).
|
||||
|
||||
- Runs as a systemd service on the host it manages, as user `core`.
|
||||
- Public at `https://hamlab.bigham.dev` (behind Nginx Proxy Manager on `core`).
|
||||
## Runs on any Linux box
|
||||
|
||||
One static Go binary with the UI embedded: no cgo, no libc, no Node at runtime, no agents.
|
||||
The same file runs on Debian, Ubuntu, Fedora, RHEL/Rocky/Alma, openSUSE, Arch and Alpine
|
||||
(musl), on bare metal, VMs and containers. Builds: `amd64`, `arm64`, `arm` (ARMv6
|
||||
hard-float and up, so every Raspberry Pi), `386`, `riscv64`.
|
||||
|
||||
```sh
|
||||
sudo ./hamlab-linux-amd64 install # that's the whole install
|
||||
```
|
||||
|
||||
or, from wherever `make release` output is served:
|
||||
|
||||
```sh
|
||||
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh # asks: user, address, HTTPS, firewall, password
|
||||
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh -s -- --yes # asks nothing, all defaults
|
||||
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh -s -- --listen 0.0.0.0:8100 --tls auto # flags go to `hamlab install`
|
||||
```
|
||||
|
||||
`install.sh` is a wizard when there is a terminal (it reads answers from `/dev/tty`, so it
|
||||
works piped from curl) and silent otherwise. Either way it picks the build for `uname -m`,
|
||||
checks it against `SHA256SUMS` and runs `hamlab install`; re-running it upgrades in place.
|
||||
`latest` is an alias Gitea resolves to the newest release, so that URL never changes.
|
||||
`make gitea-release` publishes a release there (`HAMLAB_URL` points the installer elsewhere); run the script beside the `hamlab-linux-*` files and it installs offline.
|
||||
`hamlab install`:
|
||||
|
||||
- copies the binary to `/usr/local/bin` (`--prefix`), atomically;
|
||||
- gets root through `sudo` or `doas` if you aren't root already, and runs the console as
|
||||
**you** (`--user`), so terminals are your login shell with your rights;
|
||||
- writes the config to `~/.config/hamlab/` (`--data`; an existing `~/apps/hamlab/` is kept)
|
||||
and asks for the first account's password (the account is named after the user the
|
||||
console runs as), or generates one and prints it once (`--password-stdin` for scripts);
|
||||
- registers a boot-time service with **systemd** or **OpenRC**, naming only groups that
|
||||
exist on the host, and waits for `/healthz`; with neither init it prints the one line
|
||||
to put under your own supervisor;
|
||||
- says so when firewalld or ufw will block the port (`--open-firewall` opens it).
|
||||
|
||||
Re-running it upgrades in place and keeps config, user and accounts. `hamlab uninstall
|
||||
[--purge]` reverses it.
|
||||
|
||||
Accounts: every person has their own user name and password, and the theme, accent
|
||||
colours and dashboard layout follow the account. The **Users** tab has it all: your own
|
||||
password, where you are signed in (sign out any device), and the accounts (add, remove,
|
||||
reset a password; your own password confirms the last two). From the shell:
|
||||
`hamlab passwd [user]` sets a password and makes the account if it is new, `hamlab
|
||||
users` lists them, `hamlab user-rm <user>` removes one. Machine settings — refresh rate,
|
||||
graph history, alert rules — stay the machine's and are shared by everyone.
|
||||
|
||||
**Guest logins**, for showing somebody the console: Users → Guest logins makes a
|
||||
throwaway user name and password, good for half an hour by default (up to 24 hours), and
|
||||
hands you the pair to paste into a message. A guest is an ordinary account with a clock on
|
||||
it — the whole console, every machine, the terminal included — that stops working when the
|
||||
time is up, wherever it is signed in, or the moment you revoke it. There is no reduced
|
||||
guest: give one to somebody you would hand the password to, and revoke it if you change
|
||||
your mind.
|
||||
|
||||
What adapts to the host, rather than assuming Debian:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Root | already root, else passwordless `sudo`, else passwordless `doas`; without any, the console is read-only where root is needed and says so |
|
||||
| Updates tab | apt, dnf, zypper, pacman, apk: list, refresh, upgrade, install (pacman: all or nothing) |
|
||||
| Services / timers | systemd of any age (JSON output where it exists, the plain table before v246); a clear message on hosts without systemd |
|
||||
| Disks | `lsblk`, or `/sys/block` when it's missing (busybox) or too old; SMART and ZFS only when their tools are there |
|
||||
| Network | `/proc` and `/sys` for the interfaces, addresses, link speed and default route. The speed test needs no tool installed: it measures against Cloudflare's public speed endpoints, so it works the same on Debian and on the OpenWrt router |
|
||||
| VPN | WireGuard tunnels and their peers, from `wg show` where wireguard-tools is installed; peer names from wg-quick `.conf` comments or OpenWrt UCI descriptions. Read-only, and it never reads a private or pre-shared key |
|
||||
| Dashboard | `/proc` and `/sys` only: hwmon or thermal-zone temps, fans, NICs, disks, mounts; empty sections where the hardware has nothing to report. UPS from whichever of `pwrstat`, NUT (`upsc`) or `apcupsd` is installed |
|
||||
| Listen address | bound with `IP_FREEBIND`, so a LAN address that isn't up yet at boot doesn't fail the start |
|
||||
| Self-update | the new build must run and load this host's config before it replaces the old one; atomic rename; the daemon then re-execs itself, so it restarts the same way under systemd, OpenRC or nothing; `hamlab.prev` kept |
|
||||
|
||||
Building: `make build` (web + Go), `make release` (every architecture, `install.sh` and
|
||||
`SHA256SUMS` in `dist/`), `make test`.
|
||||
|
||||
Terminals run as the user the service runs as, with that user's login shell.
|
||||
|
||||
## Managing several machines
|
||||
|
||||
Install Ham Lab on each host; on the ones the hub should reach:
|
||||
`hamlab install --listen <lan-ip>:8100 --tls auto` (a self-signed certificate is generated
|
||||
on first start). Then on the hub, open **Machines**, enter a name, the address
|
||||
(`https://host:8100`) and a user and password on that machine once (the user may be
|
||||
left out when it has only one). The hub pins the
|
||||
certificate, keeps an access token, and every tab works against the chosen machine
|
||||
through the switcher at the top of the rail.
|
||||
|
||||
Updating: `make deploy` on the hub installs the new build there and then pushes it to
|
||||
every managed machine (`hamlab push-nodes`), which install it and restart. The Machines
|
||||
page flags machines running an older version and offers the same push per machine.
|
||||
Mixed architectures are fine: `make deploy` also cross-compiles every other architecture,
|
||||
and the hub sends each machine the build for its own.
|
||||
|
||||
- On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager.
|
||||
- Installable as a PWA.
|
||||
|
||||
+284
@@ -0,0 +1,284 @@
|
||||
# Sensors — the standard read interface
|
||||
|
||||
Everything Ham Lab measures is a **sensor**: a stable dotted ID, a little metadata, a
|
||||
current value. Any consumer — Ham Lab's own dashboard, Glance, Home Assistant, a shell
|
||||
script — discovers them from the catalog and asks for exactly the ones it wants.
|
||||
|
||||
Code: `internal/sensors` (types, registry, selectors, shaped query) and
|
||||
`internal/metrics/sensors.go` (the collector as a provider). The registry measures
|
||||
nothing; providers do. A new source of sensors implements `sensors.Provider` and is
|
||||
added to the registry in `api.New`.
|
||||
|
||||
## IDs
|
||||
|
||||
```
|
||||
cpu.usage cpu.core.<n>.usage cpu.temp cpu.power load.1|5|15 uptime
|
||||
mem.total|used|avail|free|cached|buffers|used_pct swap.total|used|avail|used_pct
|
||||
temp.<chip>.<label> fan.<chip>.<label>
|
||||
net.<iface>.rx|tx|rx_total|tx_total|up
|
||||
net.internet.download|upload|latency|jitter|tested|ip|edge
|
||||
disk.<dev>.read|write|util|size|temp
|
||||
fs.<mount>.size|used|used_pct
|
||||
ups.battery|runtime|load|load_pct|in_v|out_v|on_battery|state|source
|
||||
gpu.<name>.usage|render|video|video-enhance|copy|compute|temp|power|fan|freq|mem_used|mem_total|clients
|
||||
zfs.arc.size|target|min|max|mru|mfu|data|metadata|hit_pct|reads|misses|lifetime_hit_pct|no_grow|compress_ratio|l2_size|l2_hit_pct
|
||||
```
|
||||
|
||||
- Segments are lower case `[a-z0-9_-]`: `Package id 0` → `package_id_0`, `/mnt/tank` →
|
||||
`mnt_tank`, `/` → `root`. The real names are in the sensor's `labels`.
|
||||
- Built from names the hardware reports, never enumeration order (`hwmon3` moves between
|
||||
boots). Two identical chips collide; the second gets `_2`.
|
||||
- Which sensors exist differs per machine. That is the point of the catalog.
|
||||
- `net.internet.*` is the Network tab's speed test: `download` and `upload` in Mbps,
|
||||
`latency` and `jitter` in ms, `tested` the age of the reading in seconds, and `ip` and
|
||||
`edge` the public address and the Cloudflare edge as strings. They appear only once a test
|
||||
has run on that machine, and each one's `t` is when it ran — so a widget or an alert on
|
||||
them is reading a measurement from that moment, not a live value. Nothing takes a speed
|
||||
test on its own; something has to press the button.
|
||||
- `cpu.power` is the CPU packages' own power in watts, from the kernel's RAPL energy
|
||||
counters (Intel, and AMD Zen), summed over every package: cores, built-in graphics and
|
||||
memory controller, not the whole machine. The counters are root-only, so a daemon that is
|
||||
not root reads them through a helper started once with sudo or doas. It is absent on a
|
||||
machine without the counters (ARM, the Mac mini) or without a way to root.
|
||||
- `gpu.<name>.*` is each graphics device, named as lspci names it (`gpu.intel_arc_a380`;
|
||||
labels carry `driver` and the PCI `bus`). `usage` is how busy the device was — from the
|
||||
GT's idle residency on Intel, `gpu_busy_percent` on amdgpu — and is absent on a driver
|
||||
with no such counter. The per-engine percentages (`render`, `video`…) are the sum over the
|
||||
processes holding the device open, read from DRM fdinfo, and exist only while something
|
||||
has it open. `temp` is the card's own sensor, or on an integrated GPU the CPU package it
|
||||
shares a die with (label `source` says which); an Arc on i915 has none and reports none.
|
||||
`freq` is the actual clock, 0 when power-gated. `clients` counts the processes using it.
|
||||
- `zfs.arc.*` is the ZFS read cache in RAM, on a machine with the module loaded. `hit_pct`
|
||||
is over the last sampling interval and appears once an interval has had any reads (an idle
|
||||
interval repeats the last value); `lifetime_hit_pct` is since boot. `no_grow` is true
|
||||
while memory pressure stops the cache growing. `l2_*` appear only with cache devices.
|
||||
|
||||
## A sensor
|
||||
|
||||
```json
|
||||
{ "id": "temp.coretemp.package_id_0", "name": "coretemp Package id 0",
|
||||
"kind": "temperature", "unit": "°C", "value": 51, "t": 1789826600878,
|
||||
"warn": 105, "crit": 105, "labels": { "chip": "coretemp", "label": "Package id 0" } }
|
||||
```
|
||||
|
||||
`value` is a number, bool or string. `t` is when it was really read, in unix ms — slow
|
||||
sections (filesystems 30 s, UPS 10 s) are older than the tick. `min`/`max` are the scale
|
||||
(0–100 for percentages, 0–total for used bytes); `warn`/`crit` are the hardware's own
|
||||
thresholds where it states them.
|
||||
|
||||
## Endpoints
|
||||
|
||||
All behind the same auth as the rest of `/api` (session cookie or `Authorization: Bearer`).
|
||||
On a hub, every one is also at `/api/nodes/<machine>/sensors…` for each managed machine.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `GET /api/sensors` | Catalog, with current values. `?ids=fan.*,cpu.usage` narrows it. |
|
||||
| `GET /api/sensors/{id}` | One sensor. `?format=text` → the bare value as text/plain. |
|
||||
| `GET /api/sensors/{id}/history` | `{id, unit, t[], v[]}` — what the ring holds (300 points, spaced by the window below). Numbers only. |
|
||||
| `GET /api/sensors/history?ids=cpu.usage,net.*.rx` | Many histories in one request: `{series: [{id, unit, t[], v[]}]}`. A sensor younger than the ring has a shorter series. |
|
||||
| `GET /api/prefs/history` | How far back the history reaches: `{seconds, points, stride_sec, min_seconds, max_seconds}`. |
|
||||
| `PUT /api/prefs/history` | `{seconds}` — 60 to 86400. Set on the hub it is passed to every machine, so the fleet's graphs reach equally far back. |
|
||||
| `POST /api/sensors/query` | Shaped query, below. |
|
||||
| `GET /api/sensors/query?k=selector&…` | The same, flat, for consumers that can only fetch a URL. |
|
||||
|
||||
A selector is an exact ID or a glob. `*` crosses dots: `fan.*` is every fan,
|
||||
`cpu.core.*.usage` every core.
|
||||
|
||||
## Shaped query
|
||||
|
||||
Send the object you want, with selectors where the values go. Get it back filled in.
|
||||
|
||||
```
|
||||
POST /api/sensors/query
|
||||
{ "cpu": "cpu.usage", "fans": "fan.*", "box": { "temp": "cpu.temp", "up": "uptime" } }
|
||||
|
||||
→ { "cpu": 12.4,
|
||||
"fans": { "fan.nct6775.fan1": 820, "fan.nct6775.fan2": 1140 },
|
||||
"box": { "temp": 41, "up": 80520 } }
|
||||
```
|
||||
|
||||
- Exact ID → its value; `null` if this machine has no such sensor (so one query works
|
||||
across machines that differ).
|
||||
- Glob → an object keyed by ID; `{}` if nothing matches.
|
||||
- Objects and arrays are walked; anything else in a query is a 400.
|
||||
- `?full=1` → every value is the whole sensor object instead of the bare value. (In the
|
||||
GET form `full` is therefore not usable as a key.)
|
||||
|
||||
## Live stream
|
||||
|
||||
`GET /api/sensors/stream` is a shaped query that keeps answering: the same object, pushed
|
||||
as a server-sent event after every sample. The query is `?q=<URL-encoded JSON shape>` or
|
||||
the flat form; `?full=1` as above; `?every=<ms>` spaces events out for a consumer that
|
||||
doesn't want them at the sampling rate. A keepalive comment goes out every 25 s, and
|
||||
`X-Accel-Buffering: no` keeps nginx from holding events back.
|
||||
|
||||
```js
|
||||
new EventSource('/api/sensors/stream?token=…&q=' + encodeURIComponent(JSON.stringify({ cpu: 'cpu.usage', fans: 'fan.*' })))
|
||||
.onmessage = (e) => render(JSON.parse(e.data))
|
||||
```
|
||||
|
||||
## Auth: read-only tokens
|
||||
|
||||
Settings → Access tokens → **+ Read-only sensor token** (or `POST /api/tokens/read
|
||||
{"name"}` with a session). Scope `sensors:read`: it opens every endpoint in this file,
|
||||
here and on each managed machine through the hub (`/api/nodes/<machine>/sensors…`; the
|
||||
hub talks to the node as itself, the caller's token stops at the hub), and nothing else.
|
||||
|
||||
- Send it as `Authorization: Bearer <token>`, or as `?token=<token>` where headers are
|
||||
impossible (an iframe, a browser `EventSource`).
|
||||
- A read-only token can be **limited to ID patterns** (`{"name": "glance", "sensors":
|
||||
["cpu.*", "zfs.tank.*"]}`, or the second prompt in Settings). Everything outside them
|
||||
does not exist for that token — catalog, query, history, stream, Prometheus, and on
|
||||
managed machines through the hub, which passes the restriction along.
|
||||
- A **full-control token is never accepted from a URL**, and a read-only token is never
|
||||
accepted anywhere outside this file's endpoints.
|
||||
|
||||
## Prometheus
|
||||
|
||||
`GET /metrics` (or `/api/sensors/prometheus`, which also works through the hub as
|
||||
`/api/nodes/<machine>/sensors/prometheus`). One gauge family per kind of measurement,
|
||||
base units, the sensor ID and its real-world names as labels:
|
||||
|
||||
```
|
||||
hamlab_temperature_celsius{id="temp.coretemp.core_0",chip="coretemp",label="Core 0"} 50
|
||||
hamlab_usage_percent{id="fs.root.used_pct",fstype="ext4",mount="/"} 10.1
|
||||
hamlab_state{id="ups.state",model="CPS CP1500PFCRM2U",value="Normal"} 1
|
||||
hamlab_updates_pending{id="updates.pending",manager="apt"} 3
|
||||
```
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: hamlab
|
||||
authorization: { credentials: <read-only token> }
|
||||
static_configs: [{ targets: ['10.0.0.234:8100'] }]
|
||||
```
|
||||
|
||||
## Status sensors
|
||||
|
||||
Not everything is a /proc number. These are read in the background on their own clocks
|
||||
(each sensor's `t` is its real age) by `internal/api/status_sensors.go`:
|
||||
|
||||
```
|
||||
services.failed|failed_names|running every 30 s
|
||||
service.<unit>.active|state one pair per service that is enabled, up or failed
|
||||
containers.total|running|unhealthy every 15 s (Docker or Podman, if present)
|
||||
container.<name>.running|state|health|mem|cpu mem and cpu read from the container's cgroup
|
||||
container.<name>.down should be up and isn't: Docker couldn't start it, it is
|
||||
stuck restarting, or it died non-zero with no policy to
|
||||
restart it — *not* a container someone stopped
|
||||
updates.pending|security|reboot_required every 5 min
|
||||
zfs.<pool>.health|healthy|size|used|used_pct|frag every 1 min
|
||||
smart.<dev>.passed|warnings|power_on|wear every 10 min, never waking a sleeping disk
|
||||
```
|
||||
|
||||
## Widgets
|
||||
|
||||
A widget is a spec — `{type, sensor, title?}` — over this API and nothing else
|
||||
(`web/src/widgets.tsx`, fed by `useSensorStream` in `web/src/sensors.ts`). Types: `stat`,
|
||||
`gauge`, `bar`, `spark`, and `list` for a pattern. Scale and colour come from the
|
||||
sensor's own `min`/`max`/`warn`/`crit`.
|
||||
|
||||
- **In Ham Lab:** Dashboard → Customize → **+ Add widget**. Saved with the layout,
|
||||
per machine; all of a page's widgets share one stream.
|
||||
- **Widgets tab:** every widget type, then every sensor on the machine drawn once, live,
|
||||
grouped by family (`web/src/Gallery.tsx`). Tap a tile to copy its sensor ID — the
|
||||
quickest way to find what to put in a widget, a query or a Glance tile.
|
||||
- **Anywhere else:** `/embed/widget` is one widget alone on a page, live, in its own small
|
||||
bundle (the console never loads), and the only path Ham Lab allows to be framed.
|
||||
|
||||
```
|
||||
/embed/widget?type=gauge&sensor=cpu.usage&token=<read-only token>
|
||||
&machine=grid &title=… &theme=dark|light|cyberpunk|retro &every=<ms> &bare=1
|
||||
```
|
||||
|
||||
### Glance
|
||||
|
||||
Glance's own `custom-api` widget refetches on page load / cache expiry — fine for slow
|
||||
facts, never live. An `iframe` of the embed is live.
|
||||
|
||||
```yaml
|
||||
- type: iframe
|
||||
title: core CPU
|
||||
source: https://hamlab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
|
||||
height: 130
|
||||
|
||||
- type: custom-api
|
||||
title: grid
|
||||
cache: 1m
|
||||
url: https://hamlab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
|
||||
headers: { Authorization: Bearer <read-only token> }
|
||||
template: |
|
||||
<div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>
|
||||
```
|
||||
|
||||
## The whole fleet in one request
|
||||
|
||||
On a hub, `/api/fleet/sensors/query` asks every machine (or `machines`) the same shaped
|
||||
query and answers `{machine: result}`; this machine answers under its own hostname, and
|
||||
one that can't be reached answers `{"error": …}` while the rest still do.
|
||||
|
||||
```
|
||||
POST /api/fleet/sensors/query {"machines": "*", "query": {"cpu": "cpu.usage", "up": "uptime"}}
|
||||
GET /api/fleet/sensors/query?machines=grid,wyse&cpu=cpu.usage&failed=services.failed
|
||||
|
||||
→ {"core": {"cpu": 4.8, "up": 50211}, "grid": {"cpu": 0.4, "up": 437177}, "mac": {"error": "timeout"}}
|
||||
```
|
||||
|
||||
## What counts as a disk
|
||||
|
||||
`disk.*` names **drives**, not the block devices the kernel lists. An NVMe controller can
|
||||
present several namespaces — an Apple SSD shows `nvme0n1`, `nvme0n2` and `nvme0n3`, the last
|
||||
two a few MiB of firmware — and each one arrives from `/proc/diskstats` looking like a whole
|
||||
disk. They are folded into the controller (`disk.nvme0.*`), with the rates added and busy
|
||||
taken as the longest of them rather than the sum, since they share one controller. This is
|
||||
what the Disks page has always shown; since 2026-09-22 the dashboard, the Top tab and these
|
||||
sensors agree with it.
|
||||
|
||||
## The Top tab
|
||||
|
||||
Ham Lab → **Top** is a whole machine on one screen in the style of btop: black ground, boxes
|
||||
with their name set into the border, curves drawn as dots that climb green → yellow → red.
|
||||
It reads nothing of its own — the sensor stream for everything measured, `/api/procs` for the
|
||||
table — so it works on any machine the console can reach, whether or not btop is installed
|
||||
there, and it follows the same refresh rate and history window as the rest of the console.
|
||||
Its colours are the active skin's: the terminal background for the ground, the skin's sixteen
|
||||
terminal colours for the curves and meters. (The look is btop's, `web/src/Top.tsx` is ours;
|
||||
btop itself is Apache-2.0.)
|
||||
|
||||
## How often a machine samples
|
||||
|
||||
The console's refresh rate (Settings, or `− rate +` on Top) is also the fleet's sampling rate.
|
||||
It is saved with the UI prefs on the hub, which sets its own sampler and passes the rate to
|
||||
every machine it manages — `PUT /api/prefs/sampler {"ms": …}`, 100 ms to 60 s, sampled no
|
||||
slower than the collector's 10 s — and repeats it at start-up. `GET` answers `{ms, min_ms,
|
||||
max_ms}`. Before 2026-09-21 only an open Dashboard page told a machine its rate.
|
||||
|
||||
## How far back a history reaches
|
||||
|
||||
Every machine keeps **300 points per curve**, whatever the window — so the window sets how
|
||||
far apart they are, not how much memory monitoring costs. At the default ten minutes each
|
||||
point is one sample (2 s apart at the default rate); at a day each is the **average of the
|
||||
five minutes behind it**, not whichever instant landed on the mark, so a long graph shows
|
||||
the load that really happened. Ham Lab → Settings → **Graph history** is the one control,
|
||||
1 minute to 24 hours; the hub passes it to every machine it manages.
|
||||
|
||||
The window can't be finer than the sampling interval — 300 points of a 1-minute window at a
|
||||
2-second rate is 30 points, and that is what you get. It is also all in memory: a machine
|
||||
that restarts starts its graphs again.
|
||||
|
||||
## What Ham Lab itself runs on
|
||||
|
||||
Since 2026-09-19 the Dashboard tab (stock modules and custom widgets) and the Disks tab's
|
||||
live rates read only this API: the catalog once, one batch of histories, one stream —
|
||||
closed while the page is hidden, refilled when it returns. `web/src/sampleAdapter.ts` is
|
||||
the seam that hands the stock modules the object they were written against. The old
|
||||
`/api/metrics` and `/api/metrics/history` are gone. `hamlab.sample_interval` is the
|
||||
sampler's own rate, as a sensor.
|
||||
|
||||
## From another origin
|
||||
|
||||
The sensor endpoints (and only they) answer any origin: `Access-Control-Allow-Origin: *`,
|
||||
preflight included. `*` is the one value browsers won't combine with cookies, so a foreign
|
||||
page can reach the API with a token and can never ride a logged-in console's session.
|
||||
`examples/external-demo/` is a complete outside consumer in one HTML file.
|
||||
+453
-24
@@ -2,16 +2,24 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"runtime/debug"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
@@ -20,8 +28,21 @@ import (
|
||||
"hamlab/internal/api"
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/cron"
|
||||
"hamlab/internal/files"
|
||||
"hamlab/internal/media"
|
||||
"hamlab/internal/metrics"
|
||||
"hamlab/internal/nodes"
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/services"
|
||||
"hamlab/internal/setup"
|
||||
"hamlab/internal/space"
|
||||
"hamlab/internal/storage"
|
||||
"hamlab/internal/store"
|
||||
"hamlab/internal/tasks"
|
||||
hterm "hamlab/internal/term"
|
||||
"hamlab/internal/tlsauto"
|
||||
"hamlab/internal/updates"
|
||||
"hamlab/internal/web"
|
||||
)
|
||||
|
||||
@@ -30,12 +51,25 @@ var version = "dev"
|
||||
func main() {
|
||||
defaultCfg := os.Getenv("HAMLAB_CONFIG")
|
||||
if defaultCfg == "" {
|
||||
home, _ := os.UserHomeDir()
|
||||
defaultCfg = filepath.Join(home, "apps", "hamlab", "config.json")
|
||||
defaultCfg = config.DefaultPath()
|
||||
}
|
||||
cfgPath := flag.String("config", defaultCfg, "path to config.json")
|
||||
flag.Usage = func() {
|
||||
fmt.Fprintf(os.Stderr, "usage: hamlab [-config path] [serve|passwd|version]\n")
|
||||
fmt.Fprint(os.Stderr, `usage: hamlab [-config path] <command>
|
||||
|
||||
install [flags] install this binary as a service on this machine (install -h for flags)
|
||||
uninstall [--purge] remove the service and the binary
|
||||
serve run the console in the foreground (what the service runs)
|
||||
passwd [user] set a user's password (makes the account if it is new; reads stdin when piped)
|
||||
users list the accounts
|
||||
zfs-snapshot <id> run one ZFS snapshot schedule (what its cron line calls)
|
||||
docker-update <id> run one image-update schedule (what its cron line calls)
|
||||
user-rm <user> remove an account and sign it out everywhere
|
||||
check verify the config and state load, then exit
|
||||
version
|
||||
push-nodes [--force] hub: send this build to every managed machine
|
||||
|
||||
`)
|
||||
flag.PrintDefaults()
|
||||
}
|
||||
flag.Parse()
|
||||
@@ -44,6 +78,51 @@ func main() {
|
||||
if flag.NArg() > 0 {
|
||||
cmd = flag.Arg(0)
|
||||
}
|
||||
switch cmd { // commands that don't read the config
|
||||
case "fs": // privileged helper for the Files tab (run as root by the daemon)
|
||||
os.Exit(files.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||
case "thumb": // privileged helper for the Files tab's previews in root mode (run as root by the daemon)
|
||||
os.Exit(media.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||
case "run-ffmpeg": // privileged helper for playback in root mode: ffmpeg as root, ending when stdin closes
|
||||
os.Exit(media.RunFFmpegHelper(flag.Args()[1:], os.Stdin, os.Stderr))
|
||||
case "space-walk": // helper for the Space tab: measure mountpoints, stream the tree back
|
||||
os.Exit(space.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||
case "drm-clients": // privileged helper for the GPU readings: who has the GPU open (run as root by the daemon)
|
||||
os.Exit(metrics.DRMClientsHelper(os.Stdin, os.Stdout))
|
||||
case "cpu-energy": // privileged helper for the CPU power reading: the packages' energy counters (run as root by the daemon)
|
||||
os.Exit(metrics.CPUEnergyHelper(os.Stdin, os.Stdout))
|
||||
case "install", "uninstall":
|
||||
var err error
|
||||
if cmd == "install" {
|
||||
err = setup.Install(flag.Args()[1:], version)
|
||||
} else {
|
||||
err = setup.Uninstall(flag.Args()[1:])
|
||||
}
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
os.Exit(0)
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "hamlab "+cmd+":", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return
|
||||
case "replace": // self-update helper (run as root by the daemon): put this binary at a path, atomically
|
||||
// Only ever over a hamlab binary: this runs as root, and must not be a way to
|
||||
// overwrite arbitrary files for whoever is allowed to run hamlab as root.
|
||||
if flag.NArg() != 2 || !filepath.IsAbs(flag.Arg(1)) || !strings.HasPrefix(filepath.Base(flag.Arg(1)), "hamlab") {
|
||||
fmt.Fprintln(os.Stderr, "hamlab replace: destination must be an absolute path to a hamlab binary")
|
||||
os.Exit(2)
|
||||
}
|
||||
self, err := os.Executable()
|
||||
if err == nil {
|
||||
err = setup.CopyAtomic(self, flag.Arg(1))
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "hamlab replace:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return
|
||||
}
|
||||
cfg, err := config.Load(*cfgPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
@@ -51,10 +130,70 @@ func main() {
|
||||
switch cmd {
|
||||
case "version":
|
||||
fmt.Println("hamlab", version)
|
||||
case "passwd":
|
||||
if err := passwd(cfg); err != nil {
|
||||
case "check": // a new build runs this against the live config before it replaces the old one
|
||||
if _, err := store.Open(filepath.Join(cfg.DataDir(), "state.json")); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
if _, err := fs.Stat(web.FS(), "index.html"); err != nil {
|
||||
log.Fatal("embedded UI missing: ", err)
|
||||
}
|
||||
fmt.Println("ok")
|
||||
case "passwd":
|
||||
if err := passwd(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "docker-update":
|
||||
if err := dockerUpdateJob(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "zfs-snapshot":
|
||||
if err := zfsSnapshot(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "users":
|
||||
migrateUsers(cfg)
|
||||
if len(cfg.Users) == 0 {
|
||||
fmt.Println("no accounts in", cfg.Path())
|
||||
}
|
||||
for _, u := range cfg.Users {
|
||||
seen := "never"
|
||||
if !u.LastSeen.IsZero() {
|
||||
seen = u.LastSeen.Local().Format("2006-01-02 15:04")
|
||||
}
|
||||
note := ""
|
||||
if u.Guest { // a temporary look-only login, not one of the machine's accounts
|
||||
note = " guest, until " + u.Expires.Local().Format("15:04")
|
||||
if u.Expired(time.Now()) {
|
||||
note = " guest, expired"
|
||||
}
|
||||
}
|
||||
fmt.Printf("%-20s created %s last seen %s%s\n", u.Name, u.Created.Local().Format("2006-01-02"), seen, note)
|
||||
}
|
||||
case "user-rm":
|
||||
if err := userRm(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "push-nodes": // hub CLI: send this binary to every managed machine
|
||||
exe, _ := os.Executable()
|
||||
host, _ := os.Hostname()
|
||||
nm := nodes.New(cfg, host)
|
||||
force := flag.NArg() > 1 && flag.Arg(1) == "--force"
|
||||
results := nm.PushAll(context.Background(), exe, runtime.GOARCH, force)
|
||||
if len(results) == 0 {
|
||||
fmt.Println("no managed machines in", cfg.Path())
|
||||
}
|
||||
failed := 0
|
||||
for _, r := range results {
|
||||
if r.OK {
|
||||
fmt.Printf("%-12s updated to %s\n", r.Name, r.Version)
|
||||
} else {
|
||||
failed++
|
||||
fmt.Printf("%-12s FAILED: %s\n", r.Name, r.Error)
|
||||
}
|
||||
}
|
||||
if failed > 0 {
|
||||
os.Exit(1)
|
||||
}
|
||||
case "serve":
|
||||
if err := serve(cfg); err != nil {
|
||||
log.Fatal(err)
|
||||
@@ -65,40 +204,216 @@ func main() {
|
||||
}
|
||||
}
|
||||
|
||||
func passwd(cfg *config.Config) error {
|
||||
fmt.Fprint(os.Stderr, "New password: ")
|
||||
p1, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
// migrateUsers gives a pre-users config its first account, named after this OS user, so
|
||||
// the CLI sees the same accounts the daemon will.
|
||||
// dockerUpdateJob pulls a stack's images, and deploys them when the schedule says to. This
|
||||
// is what the cron line Ham Lab writes actually calls, so image updates keep happening
|
||||
// whether or not the console is up.
|
||||
func dockerUpdateJob(cfg *config.Config, id string) error {
|
||||
if strings.TrimSpace(id) == "" {
|
||||
return errors.New("usage: hamlab docker-update <schedule-id>")
|
||||
}
|
||||
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprint(os.Stderr, "Again: ")
|
||||
p2, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
var raw json.RawMessage
|
||||
st.View(func(s *store.State) { raw = s.DockerJobs })
|
||||
var d struct {
|
||||
Jobs []api.UpdateJob `json:"jobs"`
|
||||
}
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &d); err != nil {
|
||||
return fmt.Errorf("reading the schedules: %w", err)
|
||||
}
|
||||
}
|
||||
for _, j := range d.Jobs {
|
||||
if j.ID != id {
|
||||
continue
|
||||
}
|
||||
// A pull that never finishes must not leave a cron job resident until the next reboot.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Hour)
|
||||
defer cancel()
|
||||
out, runErr := api.RunUpdate(ctx, st, j, api.DockerTool())
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: j.Stack}
|
||||
if runErr != nil {
|
||||
rec.Error = runErr.Error()
|
||||
}
|
||||
if err := storage.WriteRun(cfg.DataDir(), "docker-"+id, rec); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "recording the run:", err)
|
||||
}
|
||||
fmt.Print(out)
|
||||
return runErr
|
||||
}
|
||||
return fmt.Errorf("no image-update schedule %q — it may have been removed; this cron line is stale", id)
|
||||
}
|
||||
|
||||
// zfsSnapshot runs one snapshot schedule: take it, then expire what has aged out. This is
|
||||
// what the cron line Ham Lab writes actually calls, so a schedule keeps running whether or
|
||||
// not the console is up — and keeps running if Ham Lab is never opened again.
|
||||
//
|
||||
// It deliberately does not write state.json. The daemon owns that file and would clobber
|
||||
// anything written from out here; the outcome goes beside it instead, in zfs-runs/.
|
||||
func zfsSnapshot(cfg *config.Config, id string) error {
|
||||
if strings.TrimSpace(id) == "" {
|
||||
return errors.New("usage: hamlab zfs-snapshot <schedule-id>")
|
||||
}
|
||||
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if string(p1) != string(p2) {
|
||||
return errors.New("passwords do not match")
|
||||
var raw json.RawMessage
|
||||
st.View(func(s *store.State) { raw = s.ZFS })
|
||||
var z struct {
|
||||
Policies []storage.Policy `json:"policies"`
|
||||
}
|
||||
if len(p1) < 12 {
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &z); err != nil {
|
||||
return fmt.Errorf("reading the schedules: %w", err)
|
||||
}
|
||||
}
|
||||
var p *storage.Policy
|
||||
for i := range z.Policies {
|
||||
if z.Policies[i].ID == id {
|
||||
p = &z.Policies[i]
|
||||
}
|
||||
}
|
||||
if p == nil {
|
||||
return fmt.Errorf("no snapshot schedule %q — it may have been removed; this cron line is stale", id)
|
||||
}
|
||||
res, runErr := storage.New(priv.Available()).Run(context.Background(), p)
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: res.Taken, Pruned: res.Pruned}
|
||||
if runErr != nil {
|
||||
rec.Error = runErr.Error()
|
||||
}
|
||||
if err := storage.WriteRun(cfg.DataDir(), id, rec); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "recording the run:", err)
|
||||
}
|
||||
if runErr != nil {
|
||||
return runErr
|
||||
}
|
||||
fmt.Printf("%s: took %s", p.Name, res.Taken)
|
||||
if len(res.Pruned) > 0 {
|
||||
fmt.Printf(", expired %d", len(res.Pruned))
|
||||
}
|
||||
fmt.Println()
|
||||
return nil
|
||||
}
|
||||
|
||||
func migrateUsers(cfg *config.Config) {
|
||||
name := os.Getenv("USER")
|
||||
if u, err := user.Current(); err == nil && u.Username != "" {
|
||||
name = u.Username
|
||||
}
|
||||
if cfg.MigrateUsers(name) {
|
||||
_ = cfg.Save()
|
||||
}
|
||||
}
|
||||
|
||||
func userRm(cfg *config.Config, name string) error {
|
||||
migrateUsers(cfg)
|
||||
if name == "" {
|
||||
return errors.New("usage: hamlab user-rm <user>")
|
||||
}
|
||||
if len(cfg.Users) <= 1 {
|
||||
return errors.New("the last account can't be removed")
|
||||
}
|
||||
kept := cfg.Users[:0]
|
||||
found := false
|
||||
for _, u := range cfg.Users {
|
||||
if u.Name == name {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
if !found {
|
||||
return fmt.Errorf("no account %q", name)
|
||||
}
|
||||
cfg.Users = kept
|
||||
if err := cfg.Save(); err != nil {
|
||||
return err
|
||||
}
|
||||
if st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json")); err == nil {
|
||||
auth.NewSessions(st).RevokeUser(name)
|
||||
_ = st.Update(func(s *store.State) { delete(s.Users, name) })
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "removed", name, "(the running service picks it up at its next config save; restart it to be sure)")
|
||||
return nil
|
||||
}
|
||||
|
||||
func passwd(cfg *config.Config, name string) error {
|
||||
migrateUsers(cfg)
|
||||
if name == "" {
|
||||
switch len(cfg.Users) {
|
||||
case 0:
|
||||
name = os.Getenv("USER")
|
||||
if u, err := user.Current(); err == nil && u.Username != "" {
|
||||
name = u.Username
|
||||
}
|
||||
case 1:
|
||||
name = cfg.Users[0].Name
|
||||
default:
|
||||
return errors.New("several accounts: hamlab passwd <user>")
|
||||
}
|
||||
}
|
||||
if !auth.UserNameRe.MatchString(name) {
|
||||
return fmt.Errorf("%q: a user name is 1–32 lower-case letters, digits, . _ or -", name)
|
||||
}
|
||||
var p1 []byte
|
||||
if !term.IsTerminal(int(os.Stdin.Fd())) { // scripted: one line on stdin
|
||||
line, _ := bufio.NewReader(os.Stdin).ReadString('\n')
|
||||
p1 = []byte(strings.TrimRight(line, "\r\n"))
|
||||
} else {
|
||||
fmt.Fprintf(os.Stderr, "New password for %s: ", name)
|
||||
var err error
|
||||
p1, err = term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprint(os.Stderr, "Again: ")
|
||||
p2, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if string(p1) != string(p2) {
|
||||
return errors.New("passwords do not match")
|
||||
}
|
||||
}
|
||||
if len(p1) < auth.MinPasswordLen {
|
||||
return errors.New("password must be at least 12 characters")
|
||||
}
|
||||
h, err := auth.HashPassword(string(p1))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cfg.PasswordHash = h
|
||||
made := true
|
||||
for i := range cfg.Users {
|
||||
if cfg.Users[i].Name == name {
|
||||
cfg.Users[i].Hash = h
|
||||
made = false
|
||||
}
|
||||
}
|
||||
if made {
|
||||
cfg.Users = append(cfg.Users, config.User{Name: name, Hash: h, Created: time.Now()})
|
||||
}
|
||||
if err := cfg.Save(); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "password set in", cfg.Path())
|
||||
if made {
|
||||
fmt.Fprintln(os.Stderr, "account", name, "made in", cfg.Path())
|
||||
} else {
|
||||
fmt.Fprintln(os.Stderr, "password of", name, "set in", cfg.Path())
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "the running service reads the config at its next save; restart it to be sure")
|
||||
return nil
|
||||
}
|
||||
|
||||
func serve(cfg *config.Config) error {
|
||||
if cfg.PasswordHash == "" {
|
||||
return fmt.Errorf("no password set — run: hamlab -config %s passwd", cfg.Path())
|
||||
if cfg.PasswordHash == "" && len(cfg.Users) == 0 {
|
||||
return fmt.Errorf("no account set — run: hamlab -config %s passwd", cfg.Path())
|
||||
}
|
||||
if err := os.MkdirAll(cfg.DataDir(), 0o700); err != nil {
|
||||
return err
|
||||
@@ -115,14 +430,16 @@ func serve(cfg *config.Config) error {
|
||||
}
|
||||
shell := os.Getenv("SHELL")
|
||||
if shell == "" {
|
||||
shell = "/bin/bash"
|
||||
shell = loginShell(u.Username)
|
||||
}
|
||||
mgr := hterm.NewManager(shell, u.HomeDir)
|
||||
mgr.OnChange = func(list []hterm.Info) {
|
||||
_ = st.Update(func(s *store.State) {
|
||||
s.Terminals = s.Terminals[:0]
|
||||
for _, t := range list {
|
||||
s.Terminals = append(s.Terminals, store.Terminal{ID: t.ID, Name: t.Name})
|
||||
if t.Running { // an exited shell must not come back from the dead on restart
|
||||
s.Terminals = append(s.Terminals, store.Terminal{ID: t.ID, Name: t.Name})
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -136,12 +453,57 @@ func serve(cfg *config.Config) error {
|
||||
}
|
||||
}
|
||||
|
||||
srv := api.New(cfg, sessions, mgr, web.FS(), version)
|
||||
// Keep the heap tidy: the daemon's live data is tens of MB; without a limit the GC
|
||||
// happily lets RSS coast at twice that. (Soft limit: exceeding it only means more GC.)
|
||||
debug.SetMemoryLimit(160 << 20)
|
||||
|
||||
col := metrics.NewCollector()
|
||||
go col.Run(context.Background())
|
||||
|
||||
svc := services.New()
|
||||
cr := cron.New(svc.Sudo(), svc.JournalSudo())
|
||||
tm := tasks.New()
|
||||
up := updates.New(svc.Sudo(), tm)
|
||||
sto := storage.New(svc.Sudo())
|
||||
self, _ := os.Executable()
|
||||
var sudoFS files.FS
|
||||
if svc.Sudo() && self != "" {
|
||||
sudoFS = files.Sudo{Self: self}
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
nm := nodes.New(cfg, host)
|
||||
go nm.Run(context.Background())
|
||||
|
||||
certPath, keyPath, fingerprint := "", "", ""
|
||||
if cfg.TLS == "auto" {
|
||||
var err error
|
||||
certPath, keyPath, fingerprint, err = tlsauto.Ensure(cfg.DataDir())
|
||||
if err != nil {
|
||||
return fmt.Errorf("tls: %w", err)
|
||||
}
|
||||
}
|
||||
ln, err := listen(cfg.Listen)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
srv := api.New(cfg, sessions, mgr, col, st, svc, cr, tm, up, sto, files.Local{}, sudoFS, nm, fingerprint, web.FS(), version)
|
||||
hs := &http.Server{
|
||||
Addr: cfg.Listen,
|
||||
Handler: srv.Handler(),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
// After a self-update: become the new binary in place. Same PID, so it works the same
|
||||
// under systemd, OpenRC, or no supervisor at all. It returns only if the exec failed, by
|
||||
// which point the listener and the shells are gone; the caller puts the old binary back
|
||||
// and exits so the supervisor starts it.
|
||||
var restarting atomic.Bool
|
||||
srv.Restart = func() error {
|
||||
restarting.Store(true) // Serve returns once Shutdown is called; serve() must not exit under us
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
_ = hs.Shutdown(ctx)
|
||||
mgr.Shutdown()
|
||||
return syscall.Exec(self, os.Args, os.Environ())
|
||||
}
|
||||
go func() {
|
||||
sig := make(chan os.Signal, 1)
|
||||
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
|
||||
@@ -152,9 +514,76 @@ func serve(cfg *config.Config) error {
|
||||
_ = hs.Shutdown(ctx)
|
||||
mgr.Shutdown()
|
||||
}()
|
||||
if certPath != "" {
|
||||
log.Printf("hamlab %s listening on https://%s as %s (shell %s) cert %s", version, cfg.Listen, u.Username, shell, fingerprint)
|
||||
if err := hs.ServeTLS(ln, certPath, keyPath); !errors.Is(err, http.ErrServerClosed) {
|
||||
return err
|
||||
}
|
||||
return waitIf(&restarting)
|
||||
}
|
||||
log.Printf("hamlab %s listening on %s as %s (shell %s)", version, cfg.Listen, u.Username, shell)
|
||||
if err := hs.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
|
||||
if err := hs.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
|
||||
return err
|
||||
}
|
||||
return waitIf(&restarting)
|
||||
}
|
||||
|
||||
// waitIf parks the caller while a restart is replacing the process.
|
||||
func waitIf(restarting *atomic.Bool) error {
|
||||
if restarting.Load() {
|
||||
select {}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// listen binds the configured address with IP_FREEBIND, so a LAN address that isn't up yet
|
||||
// (boot, DHCP still negotiating, cable out) doesn't stop the daemon: it binds now and
|
||||
// answers as soon as the address appears.
|
||||
func listen(addr string) (net.Listener, error) {
|
||||
lc := net.ListenConfig{Control: func(network, _ string, c syscall.RawConn) error {
|
||||
return c.Control(func(fd uintptr) {
|
||||
_ = syscall.SetsockoptInt(int(fd), syscall.IPPROTO_IP, syscall.IP_FREEBIND, 1)
|
||||
})
|
||||
}}
|
||||
ln, err := lc.Listen(context.Background(), "tcp", addr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listen %s: %w", addr, err)
|
||||
}
|
||||
if host, _, _ := net.SplitHostPort(addr); host != "" {
|
||||
if ip := net.ParseIP(host); ip != nil && !ip.IsUnspecified() && !ip.IsLoopback() && !onInterface(ip) {
|
||||
log.Printf("listen address %s isn't on any interface yet; bound anyway and waiting for it", host)
|
||||
}
|
||||
}
|
||||
return ln, nil
|
||||
}
|
||||
|
||||
func onInterface(ip net.IP) bool {
|
||||
addrs, _ := net.InterfaceAddrs()
|
||||
for _, a := range addrs {
|
||||
if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// loginShell returns the user's shell from /etc/passwd, falling back to bash if
|
||||
// present and /bin/sh otherwise. Systemd units don't export $SHELL.
|
||||
func loginShell(username string) string {
|
||||
if b, err := os.ReadFile("/etc/passwd"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
f := strings.Split(line, ":")
|
||||
if len(f) >= 7 && f[0] == username && f[6] != "" {
|
||||
if _, err := os.Stat(f[6]); err == nil {
|
||||
return f[6]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, sh := range []string{"/bin/bash", "/usr/bin/bash", "/bin/zsh", "/bin/sh"} {
|
||||
if _, err := os.Stat(sh); err == nil {
|
||||
return sh
|
||||
}
|
||||
}
|
||||
return "/bin/sh"
|
||||
}
|
||||
|
||||
@@ -1,18 +0,0 @@
|
||||
[Unit]
|
||||
Description=Ham Lab web console
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=core
|
||||
Group=core
|
||||
WorkingDirectory=/home/core
|
||||
Environment=HAMLAB_CONFIG=/home/core/apps/hamlab/config.json
|
||||
ExecStart=/usr/local/bin/hamlab serve
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
KillMode=control-group
|
||||
TimeoutStopSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -1,11 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Install the built binary and unit on this host, then (re)start the service.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
sudo install -m 0755 bin/hamlab /usr/local/bin/hamlab
|
||||
sudo install -m 0644 deploy/hamlab.service /etc/systemd/system/hamlab.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now hamlab.service >/dev/null
|
||||
sudo systemctl restart hamlab.service
|
||||
sleep 1
|
||||
systemctl is-active hamlab.service && /usr/local/bin/hamlab version
|
||||
@@ -0,0 +1,21 @@
|
||||
# External demo
|
||||
|
||||
`index.html` is a stranger's page: one file, no build, no framework, served from a
|
||||
different origin than Ham Lab. It knows a base URL and a read-only sensor token
|
||||
(`config.js`, not in git) and nothing else. Each tab is one shaped query POSTed to
|
||||
`/api/sensors/query?full=1` on an interval; sparklines are seeded from
|
||||
`/api/sensors/history`; the Fleet tab is one call to `/api/fleet/sensors/query`.
|
||||
|
||||
```js
|
||||
// config.js
|
||||
window.HAMLAB = { base: 'http://10.0.0.234:8100', token: '<read-only sensor token>' }
|
||||
```
|
||||
|
||||
Anyone who can load the page can read the token. If the page is public, limit the token to
|
||||
patterns that name nothing you'd mind the world reading — hardware numbers and counts,
|
||||
not container or service names.
|
||||
|
||||
This is an example to read and copy, not something that runs anywhere. It was deployed on
|
||||
`core` as the `hamlab-demo` stack and linked from the Widgets tab until 2026-09-23, when
|
||||
both were removed: the Widgets tab shows the same widgets against the machine you are
|
||||
already signed in to, so a second public copy earned nothing and cost a public token.
|
||||
@@ -0,0 +1,179 @@
|
||||
<!doctype html>
|
||||
<!--
|
||||
Ham Lab sensor API — external demo.
|
||||
|
||||
One file, no build, no framework, not part of Ham Lab: it is a stranger's page on another
|
||||
origin that knows only a base URL and a read-only token (config.js, beside this file):
|
||||
|
||||
window.HAMLAB = { base: 'http://10.0.0.234:8100', token: '<read-only sensor token>' }
|
||||
|
||||
Every tab is one shaped query — the object it wants, with sensor IDs where the values go —
|
||||
POSTed to /api/sensors/query?full=1 on an interval. Sparklines are seeded once from
|
||||
/api/sensors/history. The Fleet tab uses /api/fleet/sensors/query. See SENSORS.md.
|
||||
-->
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Ham Lab sensors — external demo</title>
|
||||
<script src="config.js"></script>
|
||||
<style>
|
||||
:root { --bg:#0f1216; --card:#171b21; --line:#2a313b; --fg:#d5dbe2; --mut:#7d8590; --acc:#e3a33b; --s1:#3987e5; --s2:#c98500; --bad:#ff7b72; --ok:#3fb950; --track:rgba(57,135,229,.18) }
|
||||
* { box-sizing: border-box }
|
||||
body { margin:0; background:var(--bg); color:var(--fg); font:14px/1.4 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif }
|
||||
header { display:flex; align-items:center; gap:10px; flex-wrap:wrap; padding:10px 14px; background:var(--card); border-bottom:1px solid var(--line) }
|
||||
header h1 { margin:0; font-size:15px; color:var(--acc); letter-spacing:.4px }
|
||||
header .sp { flex:1 }
|
||||
.mut { color:var(--mut) } .mono { font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace }
|
||||
select { background:var(--bg); color:var(--fg); border:1px solid var(--line); border-radius:5px; padding:4px 6px; font:inherit }
|
||||
.dot { width:8px; height:8px; border-radius:50%; background:var(--mut); display:inline-block } .dot.ok { background:var(--ok) } .dot.bad { background:var(--bad) }
|
||||
nav { display:flex; gap:2px; overflow-x:auto; padding:0 8px; background:var(--card); border-bottom:1px solid var(--line); scrollbar-width:none }
|
||||
nav button { flex:0 0 auto; background:none; border:0; border-bottom:2px solid transparent; color:var(--mut); padding:10px 12px; font:inherit; cursor:pointer }
|
||||
nav button.on { color:var(--fg); border-bottom-color:var(--acc) }
|
||||
main { padding:12px; max-width:1100px; margin:0 auto }
|
||||
.grid { display:grid; grid-template-columns:repeat(auto-fill,minmax(170px,1fr)); gap:10px }
|
||||
.tile { background:var(--card); border:1px solid var(--line); border-radius:8px; padding:12px; min-width:0; overflow:hidden }
|
||||
.tile.wide { grid-column:1/-1 }
|
||||
.label { font-size:11px; font-weight:600; color:var(--mut); text-transform:uppercase; letter-spacing:.5px; margin-bottom:6px; overflow-wrap:anywhere }
|
||||
.value { font-size:22px; font-weight:600; font-variant-numeric:tabular-nums } .value.warn { color:var(--s2) } .value.bad { color:var(--bad) }
|
||||
.id { margin-top:8px; font-size:10.5px; color:var(--mut); overflow-wrap:anywhere }
|
||||
.meter { height:6px; background:var(--track); border-radius:3px; overflow:hidden; margin-top:8px } .meter i { display:block; height:100%; background:var(--s1); border-radius:3px; transition:width .3s } .meter i.warn { background:var(--s2) } .meter i.bad { background:var(--bad) }
|
||||
svg.gauge { display:block; width:100%; max-width:180px; margin:0 auto } svg.gauge text { fill:var(--fg); font-size:13px; font-weight:600 }
|
||||
svg.spark { display:block; width:100%; height:44px; margin-top:6px }
|
||||
.rows { display:flex; flex-direction:column; gap:7px } .row { display:grid; grid-template-columns:1fr auto; gap:2px 10px; font-size:13px; align-items:baseline }
|
||||
.row .k { overflow:hidden; text-overflow:ellipsis; white-space:nowrap } .row .n { font-variant-numeric:tabular-nums } .row .meter { grid-column:1/3; margin-top:0 }
|
||||
.pill { font-size:11px; padding:1px 7px; border-radius:9px; border:1px solid var(--line) } .pill.ok { color:var(--ok); border-color:var(--ok) } .pill.bad { color:var(--bad); border-color:var(--bad) }
|
||||
details { margin-top:14px; background:var(--card); border:1px solid var(--line); border-radius:8px; padding:10px 12px } summary { cursor:pointer; color:var(--mut) }
|
||||
pre { margin:8px 0 0; font:12px/1.5 ui-monospace,Menlo,Consolas,monospace; white-space:pre-wrap; word-break:break-word; max-height:320px; overflow:auto }
|
||||
.err { color:var(--bad); padding:12px }
|
||||
table { width:100%; border-collapse:collapse; font-size:13px } th, td { text-align:left; padding:6px 8px; border-bottom:1px solid var(--line); font-variant-numeric:tabular-nums } th { color:var(--mut); font-weight:600; font-size:11px; text-transform:uppercase }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>Ham Lab sensors</h1><span class="mut">external demo · <span id="base" class="mono"></span></span>
|
||||
<span class="sp"></span>
|
||||
<span id="status" class="mut"></span><span id="dot" class="dot"></span>
|
||||
<label class="mut">every <select id="every"><option value="1000">1 s</option><option value="2000" selected>2 s</option><option value="5000">5 s</option><option value="10000">10 s</option><option value="30000">30 s</option></select></label>
|
||||
</header>
|
||||
<nav id="tabs"></nav>
|
||||
<main>
|
||||
<div id="view"></div>
|
||||
<details><summary>What this tab asked, and what came back</summary><pre id="wire"></pre></details>
|
||||
</main>
|
||||
|
||||
<script>
|
||||
const CFG = window.HAMLAB || {}
|
||||
const $ = (id) => document.getElementById(id)
|
||||
const h = (tag, attrs = {}, ...kids) => { const e = document.createElementNS(tag === 'svg' || attrs.svg ? 'http://www.w3.org/2000/svg' : 'http://www.w3.org/1999/xhtml', tag); for (const [k, v] of Object.entries(attrs)) if (k !== 'svg' && v != null) k === 'class' ? e.setAttribute('class', v) : k === 'style' ? e.setAttribute('style', v) : e.setAttribute(k, v); for (const kid of kids.flat()) if (kid != null) e.append(kid); return e }
|
||||
|
||||
// ---- the API: two calls ----
|
||||
async function api(path, body) {
|
||||
const r = await fetch(CFG.base + path, { method: body ? 'POST' : 'GET', headers: { Authorization: 'Bearer ' + CFG.token, ...(body ? { 'Content-Type': 'application/json' } : {}) }, body: body ? JSON.stringify(body) : undefined })
|
||||
if (!r.ok) throw new Error(r.status === 401 ? 'token rejected (401)' : `${r.status} ${r.statusText}`)
|
||||
return r.json()
|
||||
}
|
||||
|
||||
// ---- formatting from the sensor's own unit ----
|
||||
const U = ['B', 'KiB', 'MiB', 'GiB', 'TiB']
|
||||
const bytes = (v) => { let i = 0; while (v >= 1024 && i < U.length - 1) { v /= 1024; i++ } return `${v >= 100 || i === 0 ? v.toFixed(0) : v.toFixed(1)} ${U[i]}` }
|
||||
const dur = (s) => { const d = Math.floor(s / 86400), hh = Math.floor(s % 86400 / 3600), m = Math.floor(s % 3600 / 60); return d ? `${d}d ${hh}h` : hh ? `${hh}h ${m}m` : `${m}m` }
|
||||
function fmt(s) {
|
||||
if (!s || s.value == null) return '—'
|
||||
const v = s.value
|
||||
if (typeof v === 'boolean') return v ? 'yes' : 'no'
|
||||
if (typeof v === 'string') return v || '—'
|
||||
switch (s.unit) { case 'B': return bytes(v); case 'B/s': return bytes(v) + '/s'; case 's': return dur(v); case 'min': return dur(v * 60); case '%': return v.toFixed(v < 10 ? 1 : 0) + '%'; case '°C': return v.toFixed(0) + '°C'; case 'rpm': return Math.round(v) + ' rpm'; case undefined: case '': return String(Math.round(v * 100) / 100); default: return `${Math.round(v * 10) / 10} ${s.unit}` }
|
||||
}
|
||||
const frac = (s) => { if (!s || typeof s.value !== 'number') return null; const max = s.max ?? s.crit ?? s.warn, min = s.min ?? 0; return max === undefined || max <= min ? null : Math.min(1, Math.max(0, (s.value - min) / (max - min))) }
|
||||
const sev = (s) => !s || typeof s.value !== 'number' ? '' : s.crit !== undefined && s.value >= s.crit ? 'bad' : s.warn !== undefined && s.value >= s.warn ? 'warn' : s.kind === 'usage' && s.id !== 'ups.battery' && s.value >= 95 ? 'bad' : s.kind === 'usage' && s.id !== 'ups.battery' && s.value >= 85 ? 'warn' : ''
|
||||
|
||||
// ---- the widgets ----
|
||||
const tile = (label, id, ...body) => h('div', { class: 'tile' }, h('div', { class: 'label' }, label), ...body, id ? h('div', { class: 'id mono' }, id) : null)
|
||||
const missing = (label, id) => tile(label, id, h('div', { class: 'mut' }, 'not on this machine'))
|
||||
const W = {
|
||||
stat: (s, label, id) => s ? tile(label || s.name, s.id, h('div', { class: 'value ' + sev(s) }, fmt(s))) : missing(label, id),
|
||||
bar: (s, label, id) => { if (!s) return missing(label, id); const f = frac(s); return tile(label || s.name, s.id, h('div', { class: 'value ' + sev(s) }, fmt(s)), f === null ? null : h('div', { class: 'meter' }, h('i', { class: sev(s), style: `width:${f * 100}%` }))) },
|
||||
gauge: (s, label, id) => {
|
||||
if (!s) return missing(label, id)
|
||||
const f = frac(s) ?? 0, C = Math.PI * 40, col = { bad: 'var(--bad)', warn: 'var(--s2)' }[sev(s)] || 'var(--s1)', arc = 'M10 50 A40 40 0 0 1 90 50'
|
||||
return tile(label || s.name, s.id, h('svg', { class: 'gauge', viewBox: '0 0 100 56' },
|
||||
h('path', { svg: 1, d: arc, fill: 'none', stroke: 'var(--track)', 'stroke-width': 9, 'stroke-linecap': 'round' }),
|
||||
h('path', { svg: 1, d: arc, fill: 'none', stroke: col, 'stroke-width': 9, 'stroke-linecap': 'round', 'stroke-dasharray': `${f * C} ${C}` }),
|
||||
h('text', { svg: 1, x: 50, y: 47, 'text-anchor': 'middle' }, fmt(s))))
|
||||
},
|
||||
spark: (s, label, id) => {
|
||||
if (!s) return missing(label, id)
|
||||
const pts = (hist[s.id] ||= []); if (typeof s.value === 'number') { pts.push(s.value); if (pts.length > 300) pts.shift() }
|
||||
const max = s.unit === '%' ? 100 : Math.max(...pts, 1e-9) * 1.1, n = pts.length
|
||||
const line = n < 2 ? '' : pts.map((v, i) => `${(i / (n - 1) * 300).toFixed(1)},${(42 - v / max * 40).toFixed(1)}`).join(' ')
|
||||
return tile(label || s.name, s.id, h('div', { class: 'value' }, fmt(s)), h('svg', { class: 'spark', viewBox: '0 0 300 44', preserveAspectRatio: 'none' }, h('polyline', { svg: 1, points: line, fill: 'none', stroke: 'var(--s1)', 'stroke-width': 2, 'vector-effect': 'non-scaling-stroke' })))
|
||||
},
|
||||
// a glob comes back as {id: sensor}: a list draws them all
|
||||
list: (map, label, pattern, opts = {}) => {
|
||||
const rows = Object.values(map || {}).filter((s) => !opts.filter || opts.filter(s)).sort((a, b) => a.id.localeCompare(b.id))
|
||||
return h('div', { class: 'tile wide' }, h('div', { class: 'label' }, `${label} · ${rows.length}`),
|
||||
rows.length ? h('div', { class: 'rows' }, rows.map((s) => { const f = frac(s); return h('div', { class: 'row' }, h('span', { class: 'k', title: s.id }, opts.name ? opts.name(s) : s.name), typeof s.value === 'boolean' ? h('span', { class: 'pill ' + (s.value ? 'ok' : 'bad') }, s.value ? 'up' : 'down') : h('span', { class: 'n' }, fmt(s)), f === null ? null : h('div', { class: 'meter' }, h('i', { class: sev(s), style: `width:${f * 100}%` }))) })) : h('div', { class: 'mut' }, 'nothing here that this token may see'),
|
||||
h('div', { class: 'id mono' }, pattern))
|
||||
},
|
||||
}
|
||||
|
||||
// ---- the tabs: a query, the histories to seed, and how to draw the answer ----
|
||||
const TABS = [
|
||||
{ name: 'Overview', query: { cpu: 'cpu.usage', mem: 'mem.used_pct', memB: 'mem.used', root: 'fs.root.used_pct', ups: 'ups.battery', temp: 'cpu.temp', load: 'load.1', up: 'uptime', failed: 'services.failed', upd: 'updates.pending', cont: 'containers.running' },
|
||||
seed: 'cpu.usage,load.1',
|
||||
draw: (d) => [W.gauge(d.cpu, 'CPU'), W.gauge(d.mem, 'Memory'), W.gauge(d.root, 'Root disk'), W.gauge(d.ups, 'UPS battery', 'ups.battery'), W.spark(d.cpu, 'CPU, recent'), W.spark(d.load, 'Load 1 min'), W.bar(d.memB, 'Memory used'), W.stat(d.temp, 'CPU temp', 'cpu.temp'), W.stat(d.up, 'Uptime'), W.stat(d.cont, 'Containers up', 'containers.running'), W.stat(d.failed, 'Failed services'), W.stat(d.upd, 'Pending updates')] },
|
||||
{ name: 'Widget types', query: { s: 'cpu.usage', cores: 'cpu.core.*.usage' }, seed: 'cpu.usage',
|
||||
draw: (d) => [W.stat(d.s, 'stat — a number'), W.gauge(d.s, 'gauge — has a scale'), W.bar(d.s, 'bar — has a scale'), W.spark(d.s, 'spark — has a past'), W.list(d.cores, 'list — a pattern', 'cpu.core.*.usage')] },
|
||||
{ name: 'Hardware', query: { cores: 'cpu.core.*.usage', temps: 'temp.*', fans: 'fan.*', smart: 'smart.*.passed', wear: 'smart.*.wear' },
|
||||
draw: (d) => [W.list(d.cores, 'CPU cores', 'cpu.core.*.usage'), W.list(d.temps, 'Temperatures', 'temp.*'), W.list(d.fans, 'Fans', 'fan.*'), W.list(d.smart, 'Drive health', 'smart.*.passed', { name: (s) => s.labels?.device + ' · ' + (s.labels?.model || '') }), W.list(d.wear, 'SSD endurance used', 'smart.*.wear')] },
|
||||
{ name: 'Network & disks', query: { rx: 'net.*.rx', tx: 'net.*.tx', rd: 'disk.*.read', wr: 'disk.*.write', fs: 'fs.*.used_pct', fsb: 'fs.*.used' }, seed: 'net.*.rx,net.*.tx,disk.*.read,disk.*.write',
|
||||
draw: (d) => [...Object.values(d.rx).map((s) => W.spark(s)), ...Object.values(d.tx).map((s) => W.spark(s)), ...Object.values(d.rd).map((s) => W.spark(s)), ...Object.values(d.wr).map((s) => W.spark(s)), W.list(d.fs, 'Filesystems', 'fs.*.used_pct', { name: (s) => s.labels?.mount }), W.list(d.fsb, 'Filesystems, bytes', 'fs.*.used', { name: (s) => s.labels?.mount })] },
|
||||
{ name: 'Containers', query: { total: 'containers.total', run: 'containers.running', bad: 'containers.unhealthy', up: 'container.*.running', mem: 'container.*.mem', cpu: 'container.*.cpu' },
|
||||
draw: (d) => [W.stat(d.run, 'Running'), W.stat(d.total, 'Total'), W.stat(d.bad, 'Unhealthy'), W.list(d.up, 'Containers', 'container.*.running', { name: (s) => s.labels?.container }), W.list(d.mem, 'Memory', 'container.*.mem', { name: (s) => s.labels?.container }), W.list(d.cpu, 'CPU, % of one core', 'container.*.cpu', { name: (s) => s.labels?.container })] },
|
||||
{ name: 'Services', query: { failed: 'services.failed', names: 'services.failed_names', running: 'services.running', reboot: 'updates.reboot_required', each: 'service.*.active' },
|
||||
draw: (d) => [W.stat(d.failed, 'Failed'), W.stat(d.running, 'Running'), W.stat(d.names, 'Which failed'), W.stat(d.reboot, 'Reboot required'), W.list(d.each, 'Every service', 'service.*.active')] },
|
||||
{ name: 'Fleet', fleet: true, query: { cpu: 'cpu.usage', mem: 'mem.used_pct', temp: 'cpu.temp', up: 'uptime', failed: 'services.failed', cont: 'containers.running', pool: 'zfs.*.health' },
|
||||
draw: (d) => [h('div', { class: 'tile wide' }, h('div', { class: 'label' }, 'one request, every machine'), h('table', {}, h('tr', {}, ['Machine', 'CPU', 'Memory', 'CPU temp', 'Uptime', 'Failed', 'Containers', 'Pools'].map((t) => h('th', {}, t))),
|
||||
Object.entries(d).sort().map(([m, r]) => r.error ? h('tr', {}, h('td', {}, m), h('td', { colspan: 7, class: 'mut' }, r.error)) : h('tr', {}, h('td', {}, m), ...[r.cpu, r.mem, r.temp, r.up, r.failed, r.cont].map((s) => h('td', {}, fmt(s))), h('td', {}, Object.values(r.pool || {}).map((s) => `${s.labels?.pool} ${s.value}`).join(', ') || '—')))), h('div', { class: 'id mono' }, 'POST /api/fleet/sensors/query'))] },
|
||||
]
|
||||
|
||||
// ---- the loop ----
|
||||
const hist = {}
|
||||
let tab = 0, timer = 0, seq = 0
|
||||
async function show(i) {
|
||||
tab = i; const my = ++seq; clearTimeout(timer)
|
||||
;[...$('tabs').children].forEach((b, j) => b.classList.toggle('on', j === i))
|
||||
$('view').replaceChildren(h('div', { class: 'mut' }, 'asking…'))
|
||||
const t = TABS[i]
|
||||
if (t.seed) try { for (const s of (await api('/api/sensors/history?ids=' + encodeURIComponent(t.seed))).series) hist[s.id] = s.v.slice(-300) } catch { /* sparklines just start empty */ }
|
||||
if (my === seq) tick(my)
|
||||
}
|
||||
async function tick(my) {
|
||||
if (my !== seq) return
|
||||
const t = TABS[tab], t0 = performance.now()
|
||||
try {
|
||||
const path = (t.fleet ? '/api/fleet/sensors/query' : '/api/sensors/query') + '?full=1', body = t.fleet ? { machines: '*', query: t.query } : t.query
|
||||
const data = await api(path, body)
|
||||
if (my !== seq) return
|
||||
$('view').replaceChildren(h('div', { class: 'grid' }, t.draw(data)))
|
||||
$('wire').textContent = `POST ${path}\n${JSON.stringify(body, null, 1)}\n\n→ ${JSON.stringify(data, (k, v) => (v && typeof v === 'object' && 'id' in v && 'value' in v ? { id: v.id, value: v.value, unit: v.unit } : v), 1).slice(0, 6000)}`
|
||||
$('dot').className = 'dot ok'; $('status').textContent = `${Math.round(performance.now() - t0)} ms · ${new Date().toLocaleTimeString()}`
|
||||
} catch (e) {
|
||||
if (my !== seq) return
|
||||
$('dot').className = 'dot bad'; $('status').textContent = e.message
|
||||
if (!$('view').querySelector('.grid')) $('view').replaceChildren(h('div', { class: 'err' }, e.message))
|
||||
}
|
||||
timer = setTimeout(() => tick(my), document.hidden ? 5000 : Number($('every').value))
|
||||
}
|
||||
|
||||
if (!CFG.base || !CFG.token) $('view').replaceChildren(h('div', { class: 'err' }, 'config.js is missing: window.HAMLAB = { base, token }'))
|
||||
else {
|
||||
$('base').textContent = CFG.base
|
||||
TABS.forEach((t, i) => $('tabs').append(h('button', {}, t.name))); [...$('tabs').children].forEach((b, i) => b.onclick = () => show(i))
|
||||
$('every').onchange = () => { clearTimeout(timer); tick(seq) }
|
||||
show(0)
|
||||
}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -5,6 +5,7 @@ go 1.26.0
|
||||
require (
|
||||
github.com/coder/websocket v1.8.15
|
||||
github.com/creack/pty v1.1.24
|
||||
go.etcd.io/bbolt v1.5.0
|
||||
golang.org/x/crypto v0.57.0
|
||||
golang.org/x/term v0.46.0
|
||||
)
|
||||
|
||||
@@ -2,9 +2,21 @@ github.com/coder/websocket v1.8.15 h1:6B2JPeOGlpff2Uz6vOEH1Vzpi0iUz20A+lPVhPHtNU
|
||||
github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
|
||||
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
|
||||
github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
|
||||
go.etcd.io/bbolt v1.5.0/go.mod h1:mkltfYE5aUHQxUct9N9V+Kp7aSjFqjgrhcXIS70Lrdk=
|
||||
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
|
||||
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
|
||||
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
|
||||
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
|
||||
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
|
||||
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
|
||||
Executable
+382
@@ -0,0 +1,382 @@
|
||||
#!/bin/sh
|
||||
# Ham Lab installer.
|
||||
#
|
||||
# curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh
|
||||
#
|
||||
# With a terminal it walks you through it: who the console runs as, where it listens,
|
||||
# HTTPS, the firewall, the password — then downloads the build for this machine, checks
|
||||
# it against SHA256SUMS and hands over to `hamlab install`, which sets up the binary,
|
||||
# the config and a service that starts at boot. Re-running it upgrades in place.
|
||||
#
|
||||
# Without a terminal, with --yes, or with flags after `--`, it asks nothing:
|
||||
#
|
||||
# curl -fsSL …/install.sh | sh -s -- --yes
|
||||
# curl -fsSL …/install.sh | sh -s -- --listen 0.0.0.0:8100 --tls auto --password-stdin <pw
|
||||
# ./install.sh # beside the hamlab-linux-* files: installs offline
|
||||
#
|
||||
# HAMLAB_URL where hamlab-linux-<arch>, SHA256SUMS and VERSION are served from —
|
||||
# by default the newest release on Gitea; "latest" is an alias Gitea
|
||||
# resolves to the most recent release, so the URL never needs changing —
|
||||
# any static file host, or a Gitea/Forgejo release's download URL
|
||||
# HAMLAB_TOKEN access token, when that place is a private Gitea/Forgejo repo
|
||||
# NO_COLOR plain output
|
||||
#
|
||||
# POSIX sh: runs under dash, busybox ash and bash alike.
|
||||
set -eu
|
||||
|
||||
HAMLAB_URL="${HAMLAB_URL:-https://git.bigham.dev/mitchell/hamlab/releases/download/latest}"
|
||||
BUILT_FOR="@VERSION@" # filled in by `make release`
|
||||
|
||||
# ─────────────────────────────────────────────────────────────── looks ──
|
||||
if [ -t 2 ] && [ -z "${NO_COLOR:-}" ] && [ "${TERM:-dumb}" != dumb ]; then
|
||||
B=$(printf '\033[1m'); D=$(printf '\033[2m'); R=$(printf '\033[0m')
|
||||
AMBER=$(printf '\033[38;5;214m'); GREEN=$(printf '\033[32m'); RED=$(printf '\033[31m'); CYAN=$(printf '\033[36m')
|
||||
else
|
||||
B=; D=; R=; AMBER=; GREEN=; RED=; CYAN=
|
||||
fi
|
||||
case "${LC_ALL:-${LC_CTYPE:-${LANG:-}}}" in
|
||||
*UTF-8*|*utf8*|*UTF8*|*utf-8*) OK='✓'; NO='✗'; DOT='•'; ARROW='›'; HR='────────────────────────────────────────────────────────────' ;;
|
||||
*) OK='ok'; NO='x'; DOT='*'; ARROW='>'; HR='------------------------------------------------------------' ;;
|
||||
esac
|
||||
|
||||
out() { printf '%s\n' "$*" >&2; }
|
||||
blank() { printf '\n' >&2; }
|
||||
good() { out " ${GREEN}${OK}${R} $*"; }
|
||||
note() { out " ${D}$*${R}"; }
|
||||
warn() { out " ${AMBER}!${R} $*"; }
|
||||
die() { blank; out " ${RED}${NO}${R} ${B}$1${R}"; shift; for more in "$@"; do out " $more"; done; blank; exit 1; }
|
||||
rule() { out "${D}${HR}${R}"; }
|
||||
step() { blank; out "${AMBER}${B}$1${R} ${B}$2${R}"; [ -z "${3:-}" ] || note "$3"; blank; }
|
||||
|
||||
banner() {
|
||||
blank
|
||||
out " ${AMBER}${B}Ham Lab${R} ${D}a web console for the machines you run${R}"
|
||||
out " ${D}terminals ${DOT} monitors ${DOT} services ${DOT} files ${DOT} updates ${DOT} disks — in one tab${R}"
|
||||
rule
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────── questions ──
|
||||
# Piped from curl, stdin is this script, so every answer comes from the terminal itself.
|
||||
TTY=/dev/tty
|
||||
have_tty() { [ -r "$TTY" ] && [ -w "$TTY" ] && (: <"$TTY") 2>/dev/null; }
|
||||
|
||||
ask() { # ask "Question" "default" → $REPLY
|
||||
if [ -n "${2:-}" ]; then printf ' %s %s[%s]%s ' "$1" "$D" "$2" "$R" >&2; else printf ' %s ' "$1" >&2; fi
|
||||
IFS= read -r REPLY <"$TTY" || REPLY=
|
||||
[ -n "$REPLY" ] || REPLY="${2:-}"
|
||||
}
|
||||
confirm() { # confirm "Question" y|n
|
||||
if [ "$2" = y ]; then hint="Y/n"; else hint="y/N"; fi
|
||||
while :; do
|
||||
ask "$1" "$hint"
|
||||
case "$REPLY" in
|
||||
"$hint") [ "$2" = y ]; return ;;
|
||||
y|Y|yes|Yes|YES) return 0 ;;
|
||||
n|N|no|No|NO) return 1 ;;
|
||||
*) warn "y or n, please" ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
secret() { # secret "Prompt" → $REPLY, typed unseen
|
||||
printf ' %s ' "$1" >&2
|
||||
saved=$(stty -g <"$TTY" 2>/dev/null || true)
|
||||
stty -echo <"$TTY" 2>/dev/null || true
|
||||
IFS= read -r REPLY <"$TTY" || REPLY=
|
||||
[ -z "$saved" ] || stty "$saved" <"$TTY" 2>/dev/null || true
|
||||
printf '\n' >&2
|
||||
}
|
||||
choose() { # choose "default-number" "label 1" "label 2" … → $REPLY = the number picked
|
||||
def=$1; shift; n=0
|
||||
for label in "$@"; do n=$((n + 1)); if [ "$n" = "$def" ]; then out " ${AMBER}${B}$n${R} $label"; else out " ${B}$n${R} $label"; fi; done
|
||||
blank
|
||||
while :; do
|
||||
ask "Choose 1-$n" "$def"
|
||||
case "$REPLY" in *[!0-9]*|'') ;; *) if [ "$REPLY" -ge 1 ] && [ "$REPLY" -le "$n" ]; then return; fi ;; esac
|
||||
warn "a number from 1 to $n, please"
|
||||
done
|
||||
}
|
||||
restore_tty() { [ -z "${saved:-}" ] || stty "$saved" <"$TTY" 2>/dev/null || true; }
|
||||
|
||||
# ──────────────────────────────────────────────────────────── plumbing ──
|
||||
fetch() { # url dest
|
||||
if command -v curl >/dev/null 2>&1; then
|
||||
curl -fsSL ${HAMLAB_TOKEN:+-H "Authorization: token $HAMLAB_TOKEN"} -o "$2" "$1"
|
||||
elif command -v wget >/dev/null 2>&1; then
|
||||
wget -q ${HAMLAB_TOKEN:+--header="Authorization: token $HAMLAB_TOKEN"} -O "$2" "$1"
|
||||
else
|
||||
die "This needs curl or wget, and the machine has neither."
|
||||
fi
|
||||
}
|
||||
port_busy() { # is something already listening on TCP port $1?
|
||||
hex=$(printf '%04X' "$1")
|
||||
cat /proc/net/tcp /proc/net/tcp6 2>/dev/null | awk -v p=":$hex" '$4 == "0A" && substr($2, length($2) - 4) == p { found = 1 } END { exit !found }'
|
||||
}
|
||||
lan_addresses() { # this machine's IPv4 addresses, one per line, most useful first
|
||||
if command -v ip >/dev/null 2>&1; then
|
||||
ip -4 -o addr show scope global 2>/dev/null | awk '$2 !~ /^(docker|br-|veth|virbr|cni|flannel|tailscale|wg|tun|tap)/ { split($4, a, "/"); print a[1] " " $2 }'
|
||||
else
|
||||
hostname -I 2>/dev/null | tr ' ' '\n' | awk 'NF && $1 !~ /:/ { print $1 " -" }'
|
||||
fi
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────── arguments ──
|
||||
YES=0; PASS=
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-y|--yes) YES=1; shift ;;
|
||||
-h|--help) sed -n '2,20p' "$0" 2>/dev/null | sed 's/^# \{0,1\}//' >&2; exit 0 ;;
|
||||
--) shift; PASS="$*"; break ;;
|
||||
*) PASS="$*"; break ;; # flags given straight: they belong to `hamlab install`
|
||||
esac
|
||||
done
|
||||
INTERACTIVE=1
|
||||
{ [ "$YES" = 1 ] || [ -n "$PASS" ] || ! have_tty; } && INTERACTIVE=0
|
||||
|
||||
# ─────────────────────────────────────────────────────────── preflight ──
|
||||
banner
|
||||
[ "$(uname -s)" = Linux ] || die "Ham Lab runs on Linux; this is $(uname -s)."
|
||||
case "$(uname -m)" in
|
||||
x86_64|amd64) arch=amd64 ;;
|
||||
aarch64|arm64) arch=arm64 ;;
|
||||
armv6l|armv7l|armv8l|arm) arch=arm ;;
|
||||
i386|i486|i586|i686) arch=386 ;;
|
||||
riscv64) arch=riscv64 ;;
|
||||
*) die "There is no build for $(uname -m). (There are: amd64, arm64, arm, 386, riscv64.)" ;;
|
||||
esac
|
||||
file="hamlab-linux-$arch"
|
||||
|
||||
me=$(id -un 2>/dev/null || echo root)
|
||||
if [ "$(id -u)" = 0 ]; then how=root
|
||||
elif command -v sudo >/dev/null 2>&1; then how=sudo
|
||||
elif command -v doas >/dev/null 2>&1; then how=doas
|
||||
else die "Installing a service needs root, and there is no sudo or doas here. Run this as root."
|
||||
fi
|
||||
if [ -d /run/systemd/system ]; then init=systemd
|
||||
elif command -v rc-service >/dev/null 2>&1; then init=OpenRC
|
||||
else init=none
|
||||
fi
|
||||
os=$(. /etc/os-release 2>/dev/null && printf '%s' "${PRETTY_NAME:-}"); [ -n "$os" ] || os=Linux
|
||||
existing=$(command -v hamlab 2>/dev/null || true); [ -n "$existing" ] || { [ -x /usr/local/bin/hamlab ] && existing=/usr/local/bin/hamlab; } || true
|
||||
have=; [ -z "$existing" ] || have=$("$existing" version 2>/dev/null | awk '{ print $2 }') || true
|
||||
|
||||
good "$os ${D}${DOT}${R} $(uname -m) ${D}${DOT}${R} $init"
|
||||
case "$how" in root) good "running as root" ;; *) good "running as ${B}$me${R}; $how is asked for root when it is needed" ;; esac
|
||||
[ "$init" != none ] || warn "no systemd or OpenRC: it will be installed, but nothing starts it at boot"
|
||||
|
||||
# ───────────────────────────────────────────────────── which version ──
|
||||
tmp=$(mktemp -d 2>/dev/null || mktemp -d -t hamlab)
|
||||
alt=
|
||||
cleanup() { restore_tty; rm -rf "$tmp"; [ -z "$alt" ] || rm -f "$alt"; }
|
||||
trap cleanup EXIT
|
||||
trap 'blank; out " cancelled — nothing was changed"; exit 130' INT TERM
|
||||
|
||||
here=$(cd "$(dirname "$0")" 2>/dev/null && pwd || echo .)
|
||||
offline=0; [ -f "$here/$file" ] && offline=1
|
||||
if [ "$offline" = 1 ]; then
|
||||
ver=$BUILT_FOR; case "$ver" in @*) ver="(local files)" ;; esac
|
||||
else
|
||||
# What is published right now — which may be newer than the copy of this script being run.
|
||||
ver=$(fetch "$HAMLAB_URL/VERSION" "$tmp/VERSION" 2>/dev/null && head -n 1 "$tmp/VERSION" | tr -cd 'A-Za-z0-9._+-' || true)
|
||||
[ -n "$ver" ] || { ver=$BUILT_FOR; case "$ver" in @*) ver=latest ;; esac; }
|
||||
fi
|
||||
# v0.1.0-69-g9c4f9df → 0001000069: versions compare as numbers, builds since the tag included.
|
||||
vnum() { printf '%s' "$1" | sed 's/^v//; s/-g[0-9a-f]*$//' | awk -F'[.-]' '{ printf "%d%03d%03d%05d", $1, $2, $3, $4 }'; }
|
||||
older=0
|
||||
if [ -n "$have" ] && [ "$have" != "$ver" ]; then
|
||||
a=$(vnum "$have" 2>/dev/null || echo 0); b=$(vnum "$ver" 2>/dev/null || echo 0)
|
||||
case "$a$b" in *[!0-9]*) ;; *) [ "$b" -ge "$a" ] || older=1 ;; esac
|
||||
fi
|
||||
if [ "$older" = 1 ]; then
|
||||
warn "Ham Lab ${B}$have${R} is installed, and what is published is ${B}older${R}: $ver"
|
||||
note " installing it would take this machine backwards"
|
||||
if [ "$INTERACTIVE" = 1 ]; then
|
||||
blank; confirm "Go back to $ver anyway?" n || { blank; out " nothing was changed"; exit 0; }
|
||||
elif [ -z "${HAMLAB_ALLOW_DOWNGRADE:-}" ]; then
|
||||
die "Not downgrading $have to $ver." "Set HAMLAB_ALLOW_DOWNGRADE=1 if that is really what you want."
|
||||
fi
|
||||
have_msg_done=1
|
||||
fi
|
||||
if [ -n "${have_msg_done:-}" ]; then :
|
||||
elif [ -n "$have" ] && [ "$have" = "$ver" ]; then good "Ham Lab ${B}$have${R} is already installed ${D}${ARROW}${R} this reinstalls it"; note " its config, user and password are kept"
|
||||
elif [ -n "$have" ]; then good "Ham Lab ${B}$have${R} is installed ${D}${ARROW}${R} upgrading to ${B}$ver${R}"; note " its config, user and password are kept"
|
||||
else good "installing Ham Lab ${B}$ver${R}"; fi
|
||||
|
||||
# ─────────────────────────────────────────────────────────── the wizard ──
|
||||
flags=; password=; listen=; url_host=
|
||||
if [ "$INTERACTIVE" = 1 ] && [ -n "$have" ] && [ "$older" = 1 ]; then :
|
||||
elif [ "$INTERACTIVE" = 1 ] && [ -n "$have" ]; then
|
||||
blank
|
||||
confirm "$([ "$have" = "$ver" ] && echo Reinstall || echo Upgrade) now?" y || { blank; out " nothing was changed"; exit 0; }
|
||||
elif [ "$INTERACTIVE" = 1 ]; then
|
||||
# 1 ─ who
|
||||
default_user=${SUDO_USER:-${DOAS_USER:-$me}}
|
||||
step "1/5" "Who should it run as?" "Its terminals are that user's shell; its files are that user's files."
|
||||
while :; do
|
||||
ask "User" "$default_user"; run_as=$REPLY
|
||||
id "$run_as" >/dev/null 2>&1 && break
|
||||
warn "there is no user called \"$run_as\" on this machine"
|
||||
done
|
||||
[ "$run_as" != root ] || warn "as root, every terminal it opens is a root shell"
|
||||
|
||||
# 2 ─ where
|
||||
step "2/5" "Where should it listen?" "Who can reach the console at all. The password guards it either way."
|
||||
addrs=$(lan_addresses || true); first=$(printf '%s\n' "$addrs" | awk 'NR == 1 { print $1 }')
|
||||
if [ -n "$first" ]; then
|
||||
first_if=$(printf '%s\n' "$addrs" | awk 'NR == 1 { print $2 }')
|
||||
choose 2 "This machine only ${D}127.0.0.1 — for an SSH tunnel or a reverse proxy${R}" \
|
||||
"This network ${D}$first ($first_if) — reachable from the LAN${R}" \
|
||||
"Every interface ${D}0.0.0.0 — every address it has or gets${R}" \
|
||||
"Another address"
|
||||
else
|
||||
choose 1 "This machine only ${D}127.0.0.1${R}" "Every interface ${D}0.0.0.0${R}" "Another address"
|
||||
[ "$REPLY" = 1 ] || REPLY=$((REPLY + 1))
|
||||
fi
|
||||
case "$REPLY" in
|
||||
1) host=127.0.0.1 ;;
|
||||
2) host=$first ;;
|
||||
3) host=0.0.0.0 ;;
|
||||
4) others=$(printf '%s\n' "$addrs" | awk 'NR > 1 { printf "%s%s (%s)", sep, $1, $2; sep = ", " }'); [ -z "$others" ] || note "also on this machine: $others"
|
||||
while :; do ask "Address" ""; host=$REPLY; case "$host" in ''|*[!0-9a-fA-F.:]*) warn "an IPv4 or IPv6 address, please" ;; *) break ;; esac; done ;;
|
||||
esac
|
||||
while :; do
|
||||
ask "Port" "8100"; port=$REPLY
|
||||
case "$port" in ''|*[!0-9]*) warn "a number, please"; continue ;; esac
|
||||
[ "$port" -ge 1 ] && [ "$port" -le 65535 ] || { warn "a port is between 1 and 65535"; continue; }
|
||||
[ "$port" -ge 1024 ] || [ "$run_as" = root ] || { warn "ports under 1024 need root, and it will run as $run_as"; continue; }
|
||||
if port_busy "$port"; then warn "something is already listening on port $port"; continue; fi
|
||||
break
|
||||
done
|
||||
case "$host" in *:*) listen="[$host]:$port" ;; *) listen="$host:$port" ;; esac
|
||||
|
||||
# 3 ─ https
|
||||
step "3/5" "HTTPS?" "With a certificate made here: browsers warn once, a hub pins it."
|
||||
if [ "$host" = 127.0.0.1 ]; then
|
||||
choose 2 "HTTPS, self-signed" "Plain HTTP ${D}— fine here: nothing leaves this machine${R}"
|
||||
else
|
||||
choose 1 "HTTPS, self-signed ${D}— recommended: the password crosses the network${R}" "Plain HTTP ${D}— only behind a reverse proxy that adds HTTPS${R}"
|
||||
fi
|
||||
if [ "$REPLY" = 1 ]; then tls=auto; scheme=https; else tls=off; scheme=http; fi
|
||||
|
||||
# 4 ─ firewall
|
||||
open_fw=0; fw=
|
||||
if [ "$host" != 127.0.0.1 ]; then
|
||||
if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then fw=firewalld
|
||||
elif command -v ufw >/dev/null 2>&1 && ufw status 2>/dev/null | grep -q "^Status: active"; then fw=ufw
|
||||
elif [ "$how" != root ] && command -v ufw >/dev/null 2>&1 && systemctl is-active -q ufw 2>/dev/null; then fw=ufw
|
||||
fi
|
||||
fi
|
||||
step "4/5" "Firewall"
|
||||
if [ -n "$fw" ]; then
|
||||
note "$fw is active and will block port $port until it is opened."
|
||||
blank
|
||||
if confirm "Open $port/tcp in $fw?" y; then open_fw=1; fi
|
||||
elif [ "$host" = 127.0.0.1 ]; then good "nothing to open: it only listens on this machine"
|
||||
else good "no active firewalld or ufw found; nothing to open"
|
||||
fi
|
||||
|
||||
# 5 ─ password
|
||||
step "5/5" "The password" "One password opens the console, and the console is a shell here."
|
||||
choose 1 "Choose one now" "Generate a strong one ${D}— shown once at the end${R}"
|
||||
if [ "$REPLY" = 1 ]; then
|
||||
while :; do
|
||||
secret "Password:"; password=$REPLY
|
||||
[ ${#password} -ge 12 ] || { warn "at least 12 characters, please (it guards a shell)"; continue; }
|
||||
secret "Once more:"
|
||||
[ "$REPLY" = "$password" ] && break
|
||||
warn "those didn't match; try again"
|
||||
done
|
||||
fi
|
||||
|
||||
# summary
|
||||
if [ "$host" = 0.0.0.0 ] || [ "$host" = "::" ]; then url_host=${first:-this-machine}; else url_host=$host; fi
|
||||
blank; rule
|
||||
out " ${B}Ready to install${R}"
|
||||
blank
|
||||
out " version ${B}$ver${R} ${D}($file)${R}"
|
||||
out " runs as ${B}$run_as${R}"
|
||||
out " address ${B}$scheme://$listen${R}"
|
||||
if [ "$open_fw" = 1 ]; then out " firewall open $port/tcp in $fw"; fi
|
||||
if [ -n "$password" ]; then out " password the one you typed"; else out " password generated, shown once"; fi
|
||||
case "$init" in none) out " service ${AMBER}none — no systemd or OpenRC${R}" ;; *) out " service $init, starts at boot" ;; esac
|
||||
rule; blank
|
||||
confirm "Install?" y || { blank; out " nothing was changed"; exit 0; }
|
||||
|
||||
flags="--user $run_as --listen $listen --tls $tls"
|
||||
[ "$open_fw" = 0 ] || flags="$flags --open-firewall"
|
||||
[ -z "$password" ] || flags="$flags --password-stdin"
|
||||
fi
|
||||
|
||||
# ──────────────────────────────────────────────── download and verify ──
|
||||
blank
|
||||
if [ "$offline" = 1 ]; then
|
||||
cp "$here/$file" "$tmp/hamlab"; [ ! -f "$here/SHA256SUMS" ] || cp "$here/SHA256SUMS" "$tmp/SHA256SUMS"
|
||||
good "using $file from $here"
|
||||
else
|
||||
printf ' %s downloading %s …' "$DOT" "$file" >&2
|
||||
fetch "$HAMLAB_URL/$file" "$tmp/hamlab" || { printf '\n' >&2; die "Download failed: $HAMLAB_URL/$file"; }
|
||||
size=$(wc -c <"$tmp/hamlab" | awk '{ printf "%.1f MB", $1 / 1048576 }')
|
||||
if [ -t 2 ]; then printf '\r\033[K' >&2; else printf '\n' >&2; fi
|
||||
good "downloaded $file ${D}($size)${R}"
|
||||
fetch "$HAMLAB_URL/SHA256SUMS" "$tmp/SHA256SUMS" 2>/dev/null || rm -f "$tmp/SHA256SUMS"
|
||||
fi
|
||||
if [ -f "$tmp/SHA256SUMS" ] && command -v sha256sum >/dev/null 2>&1; then
|
||||
want=$(awk -v f="$file" '$2 == f || $2 == "*" f { print $1 }' "$tmp/SHA256SUMS")
|
||||
got=$(sha256sum "$tmp/hamlab" | awk '{ print $1 }')
|
||||
[ -n "$want" ] || die "SHA256SUMS has no entry for $file."
|
||||
[ "$want" = "$got" ] || die "Checksum mismatch for $file." "This is not the file that was published. Nothing was installed."
|
||||
good "checksum matches SHA256SUMS"
|
||||
else
|
||||
warn "no SHA256SUMS (or no sha256sum here): the download was not verified"
|
||||
fi
|
||||
|
||||
chmod 0755 "$tmp/hamlab"; bin="$tmp/hamlab"
|
||||
if ! "$bin" version >/dev/null 2>&1; then
|
||||
alt="${HOME:-/root}/.hamlab-install.$$" # a noexec /tmp is the usual reason: try from the home directory
|
||||
cp "$bin" "$alt" 2>/dev/null && chmod 0755 "$alt" || true
|
||||
"$alt" version >/dev/null 2>&1 || die "The $arch build doesn't run on this machine."
|
||||
bin="$alt"
|
||||
fi
|
||||
|
||||
# ───────────────────────────────────────────────────────────── install ──
|
||||
blank; out " ${B}Installing${R}"; blank
|
||||
log="$tmp/install.log"
|
||||
set +e
|
||||
# shellcheck disable=SC2086 # $flags and $PASS are lists of words on purpose
|
||||
if [ -n "$password" ]; then
|
||||
printf '%s\n' "$password" | "$bin" install $flags >"$log" 2>&1; rc=$?
|
||||
password=
|
||||
elif [ "$INTERACTIVE" = 0 ] && [ -n "$PASS" ]; then
|
||||
log=
|
||||
case " $PASS " in
|
||||
*password-stdin*) "$bin" install $PASS; rc=$? ;;
|
||||
*) if [ ! -t 0 ] && have_tty; then "$bin" install $PASS <"$TTY"; rc=$?; else "$bin" install $PASS; rc=$?; fi ;;
|
||||
esac
|
||||
else
|
||||
"$bin" install $flags </dev/null >"$log" 2>&1; rc=$? # no typed password and no terminal on stdin: it generates one and prints it
|
||||
fi
|
||||
set -e
|
||||
[ -z "$log" ] || sed 's/^/ /' "$log" >&2
|
||||
hamlab_bin=$(command -v hamlab 2>/dev/null || echo /usr/local/bin/hamlab)
|
||||
[ "$rc" = 0 ] && [ -x "$hamlab_bin" ] || die "The install did not finish. Nothing above this line is hidden: the reason is in it."
|
||||
|
||||
# ──────────────────────────────────────────────────────────────── done ──
|
||||
blank; rule
|
||||
out " ${GREEN}${B}${OK} Ham Lab $("$hamlab_bin" version 2>/dev/null | awk '{ print $2 }') is installed${R}"
|
||||
blank
|
||||
row() { out " $(printf '%-10s' "$1") $2"; [ -z "${3:-}" ] || out " ${D}$3${R}"; }
|
||||
if [ "$init" = none ]; then
|
||||
row "start it" "${B}hamlab serve${R}" "as ${run_as:-its user}, under your own supervisor: nothing starts it at boot"
|
||||
fi
|
||||
if [ -n "$listen" ]; then
|
||||
row "open" "${CYAN}${B}$scheme://$(printf '%s' "$listen" | sed "s/^0\.0\.0\.0/$url_host/")${R}"
|
||||
[ "$tls" != auto ] || out " ${D}the browser warns about the self-signed certificate once${R}"
|
||||
[ "$host" != 127.0.0.1 ] || out " ${D}from elsewhere: ssh -L $port:127.0.0.1:$port $run_as@this-machine${R}"
|
||||
fi
|
||||
case "$init" in
|
||||
systemd) row "logs" "${B}journalctl -u hamlab -f${R}" ;;
|
||||
OpenRC) row "logs" "${B}tail -f /var/log/hamlab.log${R}" ;;
|
||||
esac
|
||||
row "change" "${B}hamlab install --help${R}" "a new address, HTTPS or user; everything else is kept"
|
||||
row "add to hub" "Machines ${ARROW} Add a machine" "on your main Ham Lab, with this address and password"
|
||||
row "remove" "${B}hamlab uninstall${R}" "--purge deletes its config too"
|
||||
rule; blank
|
||||
@@ -0,0 +1,528 @@
|
||||
package api
|
||||
|
||||
// Alerts: rules over sensors, watched by the hub for itself and every machine it manages.
|
||||
//
|
||||
// A rule is a selector, a comparison and a patience: `container.*.running == false for
|
||||
// 120 s`. Every alertEvery the hub asks each machine one shaped query for all the rules'
|
||||
// selectors, compares, and keeps a little state per (rule, machine, sensor): ok, pending
|
||||
// since, or firing since. Going to firing, and coming back, is an event — kept, shown on
|
||||
// the Alerts tab, and sent to whatever channels are set (ntfy, or a webhook: Home
|
||||
// Assistant, Slack, anything that takes a POST). `hamlab.reachable` is a sensor the hub
|
||||
// supplies itself, so "that machine stopped answering" is a rule like any other.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/sensors"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const (
|
||||
alertEvery = 15 * time.Second
|
||||
maxEvents = 200
|
||||
)
|
||||
|
||||
type AlertRule struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Sensor string `json:"sensor"` // an ID or a pattern
|
||||
Op string `json:"op"` // > >= < <= == !=
|
||||
Value any `json:"value"` // number, bool or string, to match the sensor
|
||||
ForSec int `json:"for_sec"`
|
||||
Severity string `json:"severity"` // info | warn | crit
|
||||
Enabled bool `json:"enabled"`
|
||||
Machines string `json:"machines"` // "*" or comma-separated names; "" is "*"
|
||||
}
|
||||
|
||||
type AlertChannel struct {
|
||||
Type string `json:"type"` // ntfy | webhook
|
||||
URL string `json:"url"`
|
||||
}
|
||||
|
||||
type AlertEvent struct {
|
||||
T int64 `json:"t"`
|
||||
Kind string `json:"kind"` // firing | resolved
|
||||
Rule string `json:"rule"`
|
||||
RuleID string `json:"rule_id"`
|
||||
Machine string `json:"machine"`
|
||||
Sensor string `json:"sensor"`
|
||||
Value any `json:"value"`
|
||||
Severity string `json:"severity"`
|
||||
}
|
||||
|
||||
type alertFiring struct {
|
||||
AlertEvent
|
||||
Since int64 `json:"since"`
|
||||
}
|
||||
|
||||
type alertSaved struct {
|
||||
Rules []AlertRule `json:"rules"`
|
||||
Channels []AlertChannel `json:"channels"`
|
||||
Events []AlertEvent `json:"events"`
|
||||
Seeded bool `json:"seeded"`
|
||||
// Firing is what was firing when the hub last looked, so a restart (every deploy is one)
|
||||
// carries on where it was and doesn't announce the same alerts all over again.
|
||||
Firing map[string]alertKept `json:"firing,omitempty"`
|
||||
}
|
||||
|
||||
type alertKept struct {
|
||||
Since int64 `json:"since"`
|
||||
Value any `json:"value"`
|
||||
}
|
||||
|
||||
const maxRules = 200
|
||||
|
||||
type alertState struct {
|
||||
pending time.Time
|
||||
firing bool
|
||||
since time.Time
|
||||
value any
|
||||
}
|
||||
|
||||
type alertEngine struct {
|
||||
s *Server
|
||||
mu sync.Mutex
|
||||
saved alertSaved
|
||||
state map[string]*alertState // ruleID|machine|sensor
|
||||
last time.Time
|
||||
}
|
||||
|
||||
var defaultRules = []AlertRule{
|
||||
{ID: "machine-offline", Name: "Machine not answering", Sensor: "hamlab.reachable", Op: "==", Value: false, ForSec: 120, Severity: "crit", Enabled: true},
|
||||
{ID: "container-down", Name: "Container down", Sensor: "container.*.down", Op: "==", Value: true, ForSec: 120, Severity: "crit", Enabled: true},
|
||||
{ID: "container-unhealthy", Name: "Container unhealthy", Sensor: "container.*.health", Op: "==", Value: "unhealthy", ForSec: 180, Severity: "warn", Enabled: true},
|
||||
{ID: "service-failed", Name: "A service has failed", Sensor: "services.failed", Op: ">", Value: 0.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
{ID: "disk-full", Name: "Filesystem nearly full", Sensor: "fs.*.used_pct", Op: ">=", Value: 90.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
{ID: "pool-unhealthy", Name: "ZFS pool not healthy", Sensor: "zfs.*.healthy", Op: "==", Value: false, ForSec: 60, Severity: "crit", Enabled: true},
|
||||
{ID: "drive-failing", Name: "Drive failing SMART", Sensor: "smart.*.passed", Op: "==", Value: false, ForSec: 0, Severity: "crit", Enabled: true},
|
||||
{ID: "on-battery", Name: "UPS on battery", Sensor: "ups.on_battery", Op: "==", Value: true, ForSec: 30, Severity: "crit", Enabled: true},
|
||||
{ID: "hot", Name: "CPU running hot", Sensor: "cpu.temp", Op: ">=", Value: 90.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
{ID: "memory", Name: "Memory nearly exhausted", Sensor: "mem.used_pct", Op: ">=", Value: 95.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
}
|
||||
|
||||
func newAlertEngine(s *Server) *alertEngine {
|
||||
e := &alertEngine{s: s, state: map[string]*alertState{}}
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.Alerts })
|
||||
_ = json.Unmarshal(raw, &e.saved)
|
||||
if !e.saved.Seeded {
|
||||
e.saved.Rules, e.saved.Seeded = append([]AlertRule{}, defaultRules...), true
|
||||
}
|
||||
for key, k := range e.saved.Firing {
|
||||
e.state[key] = &alertState{pending: time.UnixMilli(k.Since), firing: true, since: time.UnixMilli(k.Since), value: k.Value}
|
||||
}
|
||||
// the Virtualization tab's default went with the tab (2026-10-15): nothing publishes instance.* now
|
||||
kept := e.saved.Rules[:0]
|
||||
for _, r := range e.saved.Rules {
|
||||
if r.ID != "instance-down" {
|
||||
kept = append(kept, r)
|
||||
}
|
||||
}
|
||||
e.saved.Rules = kept
|
||||
for i, r := range e.saved.Rules { // the first day's default watched "not running", which a container stopped on purpose also is
|
||||
if r.ID == "container-down" && r.Sensor == "container.*.running" {
|
||||
e.saved.Rules[i].Name, e.saved.Rules[i].Sensor, e.saved.Rules[i].Value = "Container down", "container.*.down", true
|
||||
}
|
||||
}
|
||||
go e.run()
|
||||
return e
|
||||
}
|
||||
|
||||
func (e *alertEngine) persist() {
|
||||
b, _ := json.Marshal(e.saved)
|
||||
_ = e.s.store.Update(func(st *store.State) { st.Alerts = b })
|
||||
}
|
||||
|
||||
func (e *alertEngine) run() {
|
||||
time.Sleep(20 * time.Second) // let the status sensors take their first readings
|
||||
for {
|
||||
e.evaluate()
|
||||
time.Sleep(alertEvery)
|
||||
}
|
||||
}
|
||||
|
||||
func wants(r AlertRule, machine string) bool {
|
||||
if r.Machines == "" || r.Machines == "*" {
|
||||
return true
|
||||
}
|
||||
for _, m := range strings.Split(r.Machines, ",") {
|
||||
if strings.TrimSpace(m) == machine {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// holds reports whether `got <op> want`, comparing like with like.
|
||||
func holds(got any, op string, want any) bool {
|
||||
switch g := got.(type) {
|
||||
case float64:
|
||||
w, ok := want.(float64)
|
||||
if !ok {
|
||||
if s, isStr := want.(string); isStr {
|
||||
f, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
w = f
|
||||
} else {
|
||||
return false
|
||||
}
|
||||
}
|
||||
switch op {
|
||||
case ">":
|
||||
return g > w
|
||||
case ">=":
|
||||
return g >= w
|
||||
case "<":
|
||||
return g < w
|
||||
case "<=":
|
||||
return g <= w
|
||||
case "==":
|
||||
return g == w
|
||||
case "!=":
|
||||
return g != w
|
||||
}
|
||||
case bool:
|
||||
w, ok := want.(bool)
|
||||
if !ok {
|
||||
w = fmt.Sprint(want) == "true"
|
||||
}
|
||||
return (op == "==" && g == w) || (op == "!=" && g != w)
|
||||
case string:
|
||||
w := fmt.Sprint(want)
|
||||
return (op == "==" && g == w) || (op == "!=" && g != w)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (e *alertEngine) evaluate() {
|
||||
e.mu.Lock()
|
||||
rules := append([]AlertRule{}, e.saved.Rules...)
|
||||
e.mu.Unlock()
|
||||
query := map[string]string{}
|
||||
for _, r := range rules {
|
||||
if r.Enabled && r.Sensor != "hamlab.reachable" {
|
||||
query[r.ID] = r.Sensor
|
||||
}
|
||||
}
|
||||
qb, _ := json.Marshal(query)
|
||||
machines := append([]string{e.s.hostName()}, e.s.nodes.Names()...)
|
||||
results := map[string]map[string]any{} // machine → ruleID → value | {id: value}
|
||||
reach := map[string]bool{}
|
||||
var wg sync.WaitGroup
|
||||
var mu sync.Mutex
|
||||
for _, m := range machines {
|
||||
wg.Add(1)
|
||||
go func(m string) {
|
||||
defer wg.Done()
|
||||
var res map[string]any
|
||||
ok := false
|
||||
if m == e.s.hostName() {
|
||||
if out, err := sensors.Shape(e.s.sensors.All(), qb, false); err == nil {
|
||||
res, ok = out.(map[string]any)
|
||||
}
|
||||
} else {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
code, b, err := e.s.nodes.Do(ctx, m, "POST", "/api/sensors/query", qb, http.Header{"Content-Type": {"application/json"}})
|
||||
cancel()
|
||||
ok = err == nil && code == 200 && json.Unmarshal(b, &res) == nil
|
||||
}
|
||||
mu.Lock()
|
||||
results[m], reach[m] = res, ok
|
||||
mu.Unlock()
|
||||
}(m)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
now := time.Now()
|
||||
seen := map[string]bool{}
|
||||
var events []AlertEvent
|
||||
e.mu.Lock()
|
||||
for _, r := range rules {
|
||||
if !r.Enabled {
|
||||
continue
|
||||
}
|
||||
for _, m := range machines {
|
||||
if !wants(r, m) {
|
||||
continue
|
||||
}
|
||||
readings := map[string]any{}
|
||||
if r.Sensor == "hamlab.reachable" {
|
||||
readings["hamlab.reachable"] = reach[m]
|
||||
} else if !reach[m] {
|
||||
continue // no reading is not a reading: an unreachable machine is the offline rule's business
|
||||
} else if v, ok := results[m][r.ID]; ok {
|
||||
if many, isMap := v.(map[string]any); isMap {
|
||||
readings = many
|
||||
} else if v != nil {
|
||||
readings[r.Sensor] = v
|
||||
}
|
||||
}
|
||||
for id, val := range readings {
|
||||
key := r.ID + "|" + m + "|" + id
|
||||
seen[key] = true
|
||||
st := e.state[key]
|
||||
if st == nil {
|
||||
st = &alertState{}
|
||||
e.state[key] = st
|
||||
}
|
||||
st.value = val
|
||||
if holds(val, r.Op, r.Value) {
|
||||
if st.pending.IsZero() {
|
||||
st.pending = now
|
||||
}
|
||||
if !st.firing && now.Sub(st.pending) >= time.Duration(r.ForSec)*time.Second {
|
||||
st.firing, st.since = true, now
|
||||
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "firing", Rule: r.Name, RuleID: r.ID, Machine: m, Sensor: id, Value: val, Severity: r.Severity})
|
||||
}
|
||||
} else {
|
||||
if st.firing {
|
||||
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "resolved", Rule: r.Name, RuleID: r.ID, Machine: m, Sensor: id, Value: val, Severity: r.Severity})
|
||||
}
|
||||
st.pending, st.firing = time.Time{}, false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
byID := map[string]AlertRule{}
|
||||
for _, r := range rules {
|
||||
byID[r.ID] = r
|
||||
}
|
||||
for key, st := range e.state { // no longer watched: the rule was paused, deleted or narrowed, or the sensor is gone
|
||||
if seen[key] {
|
||||
continue
|
||||
}
|
||||
if st.firing { // it was on the Alerts tab and perhaps on a phone: say that it is over, and why
|
||||
part := strings.SplitN(key, "|", 3)
|
||||
r, known := byID[part[0]]
|
||||
name, why := r.Name, "no longer watched"
|
||||
if !known {
|
||||
name, why = part[0], "rule deleted"
|
||||
} else if !r.Enabled {
|
||||
why = "rule paused"
|
||||
}
|
||||
if name == "" {
|
||||
name = part[0]
|
||||
}
|
||||
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "resolved", Rule: name, RuleID: part[0], Machine: part[1], Sensor: part[2], Value: why, Severity: r.Severity})
|
||||
}
|
||||
delete(e.state, key)
|
||||
}
|
||||
e.last = now
|
||||
kept := map[string]alertKept{}
|
||||
for key, st := range e.state {
|
||||
if st.firing {
|
||||
kept[key] = alertKept{Since: st.since.UnixMilli(), Value: st.value}
|
||||
}
|
||||
}
|
||||
changed := len(kept) != len(e.saved.Firing)
|
||||
for k := range kept {
|
||||
if _, ok := e.saved.Firing[k]; !ok {
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
e.saved.Firing = kept
|
||||
if changed && len(events) == 0 {
|
||||
e.persist()
|
||||
}
|
||||
if len(events) > 0 {
|
||||
e.saved.Events = append(e.saved.Events, events...)
|
||||
if len(e.saved.Events) > maxEvents {
|
||||
e.saved.Events = e.saved.Events[len(e.saved.Events)-maxEvents:]
|
||||
}
|
||||
e.persist()
|
||||
}
|
||||
channels := append([]AlertChannel{}, e.saved.Channels...)
|
||||
e.mu.Unlock()
|
||||
for _, ev := range events {
|
||||
e.notify(channels, ev)
|
||||
}
|
||||
}
|
||||
|
||||
func (e *alertEngine) firing() []alertFiring {
|
||||
out := []alertFiring{}
|
||||
names := map[string]AlertRule{}
|
||||
for _, r := range e.saved.Rules {
|
||||
names[r.ID] = r
|
||||
}
|
||||
for key, st := range e.state {
|
||||
if !st.firing {
|
||||
continue
|
||||
}
|
||||
p := strings.SplitN(key, "|", 3)
|
||||
r := names[p[0]]
|
||||
out = append(out, alertFiring{AlertEvent: AlertEvent{T: st.since.UnixMilli(), Kind: "firing", Rule: r.Name, RuleID: r.ID, Machine: p[1], Sensor: p[2], Value: st.value, Severity: r.Severity}, Since: st.since.UnixMilli()})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Since > out[j].Since })
|
||||
return out
|
||||
}
|
||||
|
||||
func describe(ev AlertEvent) (title, body string) {
|
||||
mark := map[string]string{"firing": "▲", "resolved": "✓"}[ev.Kind]
|
||||
title = fmt.Sprintf("%s %s — %s", mark, ev.Rule, ev.Machine)
|
||||
body = fmt.Sprintf("%s = %v", ev.Sensor, ev.Value)
|
||||
if ev.Kind == "resolved" {
|
||||
body = "back to normal: " + body
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func (e *alertEngine) notify(channels []AlertChannel, ev AlertEvent) {
|
||||
title, body := describe(ev)
|
||||
for _, c := range channels {
|
||||
var req *http.Request
|
||||
switch c.Type {
|
||||
case "ntfy":
|
||||
req, _ = http.NewRequest("POST", c.URL, strings.NewReader(body))
|
||||
req.Header.Set("Title", title)
|
||||
req.Header.Set("Priority", map[string]string{"crit": "urgent", "warn": "high"}[ev.Severity])
|
||||
req.Header.Set("Tags", map[string]string{"firing": "warning", "resolved": "white_check_mark"}[ev.Kind])
|
||||
if ev.Kind == "resolved" {
|
||||
req.Header.Set("Priority", "default")
|
||||
}
|
||||
case "webhook":
|
||||
b, _ := json.Marshal(map[string]any{"title": title, "message": body, "event": ev})
|
||||
req, _ = http.NewRequest("POST", c.URL, bytes.NewReader(b))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if req == nil {
|
||||
continue
|
||||
}
|
||||
go func(req *http.Request, url string) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
resp, err := http.DefaultClient.Do(req.WithContext(ctx))
|
||||
if err != nil {
|
||||
log.Printf("alert: notifying %s: %v", url, err)
|
||||
return
|
||||
}
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode >= 300 {
|
||||
log.Printf("alert: %s answered %d", url, resp.StatusCode)
|
||||
}
|
||||
}(req, c.URL)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- API ----
|
||||
|
||||
func (s *Server) alertsGet(w http.ResponseWriter, _ *http.Request) {
|
||||
e := s.alerts
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
ev := append([]AlertEvent{}, e.saved.Events...)
|
||||
for i, j := 0, len(ev)-1; i < j; i, j = i+1, j-1 {
|
||||
ev[i], ev[j] = ev[j], ev[i]
|
||||
}
|
||||
rules, channels := e.saved.Rules, e.saved.Channels
|
||||
if rules == nil { // an empty list is a list: never hand the page a null to take the length of
|
||||
rules = []AlertRule{}
|
||||
}
|
||||
if channels == nil {
|
||||
channels = []AlertChannel{}
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"rules": rules, "channels": channels, "events": ev, "firing": e.firing(), "checked": e.last.UnixMilli(), "every_sec": int(alertEvery.Seconds())})
|
||||
}
|
||||
|
||||
func (s *Server) alertsPut(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Rules *[]AlertRule `json:"rules"`
|
||||
Channels *[]AlertChannel `json:"channels"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 256<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if body.Rules != nil && len(*body.Rules) > maxRules {
|
||||
jsonErr(w, 400, fmt.Sprintf("at most %d rules: every one is checked on every machine every %d s", maxRules, int(alertEvery.Seconds())))
|
||||
return
|
||||
}
|
||||
if body.Rules != nil {
|
||||
ids := map[string]bool{}
|
||||
for i := range *body.Rules {
|
||||
ru := &(*body.Rules)[i]
|
||||
ru.Name, ru.Sensor = strings.TrimSpace(ru.Name), strings.TrimSpace(ru.Sensor)
|
||||
if ru.ID == "" || ids[ru.ID] || ru.Name == "" || len(ru.Name) > maxNameLen || ru.ForSec < 0 || ru.ForSec > 86400 {
|
||||
jsonErr(w, 400, "every rule needs a unique id, a name (64 characters at most) and a patience between 0 and 86400 s")
|
||||
return
|
||||
}
|
||||
ids[ru.ID] = true
|
||||
if _, err := sensors.Matcher(ru.Sensor); err != nil || ru.Sensor == "" {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: %q is not a sensor ID or pattern", ru.Name, ru.Sensor))
|
||||
return
|
||||
}
|
||||
if !strings.Contains(" > >= < <= == != ", " "+ru.Op+" ") {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: the comparison must be one of > >= < <= == !=", ru.Name))
|
||||
return
|
||||
}
|
||||
switch val := ru.Value.(type) {
|
||||
case float64, bool:
|
||||
case string:
|
||||
if len(val) > 200 {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: the value is too long", ru.Name))
|
||||
return
|
||||
}
|
||||
if ru.Op != "==" && ru.Op != "!=" { // "cpu.usage > ninety" can never be true, and would never say so
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: %s compares numbers, and %q is not one; text can only be == or !=", ru.Name, ru.Op, val))
|
||||
return
|
||||
}
|
||||
default:
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: the value must be a number, true/false or text", ru.Name))
|
||||
return
|
||||
}
|
||||
if _, isBool := ru.Value.(bool); isBool && ru.Op != "==" && ru.Op != "!=" {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: true/false can only be == or !=", ru.Name))
|
||||
return
|
||||
}
|
||||
if ru.Severity != "info" && ru.Severity != "crit" {
|
||||
ru.Severity = "warn"
|
||||
}
|
||||
}
|
||||
}
|
||||
if body.Channels != nil {
|
||||
for _, c := range *body.Channels {
|
||||
if (c.Type != "ntfy" && c.Type != "webhook") || !(strings.HasPrefix(c.URL, "https://") || strings.HasPrefix(c.URL, "http://")) {
|
||||
jsonErr(w, 400, "a channel is {type: ntfy|webhook, url: http(s)://…}")
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
e := s.alerts
|
||||
e.mu.Lock()
|
||||
if body.Rules != nil {
|
||||
e.saved.Rules = *body.Rules
|
||||
}
|
||||
if body.Channels != nil {
|
||||
e.saved.Channels = *body.Channels
|
||||
}
|
||||
e.persist()
|
||||
e.mu.Unlock()
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// alertsTest sends one clearly-marked test notification through every channel.
|
||||
func (s *Server) alertsTest(w http.ResponseWriter, _ *http.Request) {
|
||||
e := s.alerts
|
||||
e.mu.Lock()
|
||||
channels := append([]AlertChannel{}, e.saved.Channels...)
|
||||
e.mu.Unlock()
|
||||
if len(channels) == 0 {
|
||||
jsonErr(w, 400, "no channels to test")
|
||||
return
|
||||
}
|
||||
e.notify(channels, AlertEvent{T: time.Now().UnixMilli(), Kind: "firing", Rule: "Test alert", Machine: s.hostName(), Sensor: "hamlab.test", Value: "this is only a test", Severity: "info"})
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,650 @@
|
||||
package api
|
||||
|
||||
// The Docker tab: what is running, grouped the way it was deployed (compose stacks), with
|
||||
// start / stop / restart, logs and inspect — plus images, volumes and networks to look at.
|
||||
// It drives the docker CLI (or podman), as the user first and as root if that is refused.
|
||||
// What can be removed here is what nothing uses: an untagged image, a volume no container
|
||||
// holds, a network nothing is attached to. Compose stays the one place a stack is defined.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type dockerContainer struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Image string `json:"image"`
|
||||
State string `json:"state"` // running | exited | paused | restarting | created | dead
|
||||
Status string `json:"status"` // "Up 3 hours (healthy)"
|
||||
Health string `json:"health"` // healthy | unhealthy | starting | none
|
||||
Ports string `json:"ports"`
|
||||
Stack string `json:"stack"` // compose project, "" when started by hand
|
||||
Service string `json:"service"` // compose service
|
||||
Created string `json:"created"`
|
||||
Mem float64 `json:"mem"` // bytes, from the container's cgroup; 0 when unknown
|
||||
}
|
||||
|
||||
func dockerTool() string {
|
||||
for _, t := range []string{"docker", "podman"} {
|
||||
if _, err := exec.LookPath(t); err == nil {
|
||||
return t
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// dockerRun runs the tool as this user, and again as root if the socket said no.
|
||||
// DockerTool is whichever of docker or podman this machine has, for callers outside the
|
||||
// package — the scheduled updates the CLI runs.
|
||||
func DockerTool() string { return dockerTool() }
|
||||
|
||||
func dockerRun(ctx context.Context, args ...string) ([]byte, error) {
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
return nil, errors.New("neither docker nor podman is installed here")
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, tool, args...).CombinedOutput()
|
||||
if err != nil && priv.Available() && strings.Contains(strings.ToLower(string(out)), "permission denied") {
|
||||
full := priv.Wrap(append([]string{tool}, args...)...)
|
||||
out, err = exec.CommandContext(ctx, full[0], full[1:]...).CombinedOutput()
|
||||
}
|
||||
if err != nil {
|
||||
msg := strings.TrimSpace(string(out))
|
||||
if i := strings.LastIndex(msg, "\n"); i >= 0 {
|
||||
msg = msg[i+1:]
|
||||
}
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return out, errors.New(msg)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// dockerWhy picks the line worth showing out of a failed command. Some subcommands —
|
||||
// `network rm` among them — print the real reason first and then an aggregated "exit
|
||||
// status 1", which is what dockerRun's last-line rule would otherwise hand back.
|
||||
func dockerWhy(out []byte, err error) string {
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
if l = strings.TrimSpace(l); strings.HasPrefix(l, "Error") || strings.Contains(l, "daemon") {
|
||||
return l
|
||||
}
|
||||
}
|
||||
return err.Error()
|
||||
}
|
||||
|
||||
func jsonLines(out []byte, each func(map[string]any)) {
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
var m map[string]any
|
||||
if json.Unmarshal([]byte(line), &m) == nil {
|
||||
each(m)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func dstr(m map[string]any, k string) string { s, _ := m[k].(string); return s }
|
||||
|
||||
func labelOf(labels, key string) string {
|
||||
for _, kv := range strings.Split(labels, ",") {
|
||||
if v, ok := strings.CutPrefix(kv, key+"="); ok {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (s *Server) dockerOverview(w http.ResponseWriter, r *http.Request) {
|
||||
if dockerTool() == "" {
|
||||
jsonOK(w, 200, map[string]any{"available": false, "message": "Docker isn't installed on this machine."})
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "ps", "-a", "--no-trunc", "--format", "{{json .}}")
|
||||
if err != nil {
|
||||
jsonOK(w, 200, map[string]any{"available": false, "message": err.Error()})
|
||||
return
|
||||
}
|
||||
containers := []dockerContainer{}
|
||||
jsonLines(out, func(m map[string]any) {
|
||||
c := dockerContainer{ID: dstr(m, "ID"), Name: strings.Split(dstr(m, "Names"), ",")[0], Image: dstr(m, "Image"), State: dstr(m, "State"), Status: dstr(m, "Status"),
|
||||
Ports: dstr(m, "Ports"), Created: dstr(m, "CreatedAt"), Health: "none", Stack: labelOf(dstr(m, "Labels"), "com.docker.compose.project"), Service: labelOf(dstr(m, "Labels"), "com.docker.compose.service")}
|
||||
for _, h := range []string{"unhealthy", "healthy", "starting"} {
|
||||
if strings.Contains(c.Status, "("+h+")") || strings.Contains(c.Status, "health: "+h) {
|
||||
c.Health = h
|
||||
break
|
||||
}
|
||||
}
|
||||
for _, dir := range []string{"/sys/fs/cgroup/system.slice/docker-" + c.ID + ".scope", "/sys/fs/cgroup/docker/" + c.ID, "/sys/fs/cgroup/machine.slice/libpod-" + c.ID + ".scope"} {
|
||||
if b, err := os.ReadFile(dir + "/memory.current"); err == nil {
|
||||
c.Mem, _ = strconv.ParseFloat(strings.TrimSpace(string(b)), 64)
|
||||
break
|
||||
}
|
||||
}
|
||||
containers = append(containers, c)
|
||||
})
|
||||
sort.Slice(containers, func(i, j int) bool {
|
||||
if containers[i].Stack != containers[j].Stack {
|
||||
return containers[i].Stack < containers[j].Stack
|
||||
}
|
||||
return containers[i].Name < containers[j].Name
|
||||
})
|
||||
list := func(args ...string) []map[string]any {
|
||||
rows := []map[string]any{}
|
||||
if out, err := dockerRun(ctx, args...); err == nil {
|
||||
jsonLines(out, func(m map[string]any) { rows = append(rows, m) })
|
||||
}
|
||||
return rows
|
||||
}
|
||||
// Who uses what, from one inspect of every container: an image, a volume or a network
|
||||
// nothing uses is the only kind this page offers to remove.
|
||||
imageUsed := map[string]bool{}
|
||||
volumeUsers, networkUsers := map[string][]string{}, map[string][]string{}
|
||||
if ids, err := dockerRun(ctx, "ps", "-aq"); err == nil && len(strings.Fields(string(ids))) > 0 {
|
||||
const f = `{{.Name}}|{{.Image}}|{{range .Mounts}}{{if eq .Type "volume"}}{{.Name}},{{end}}{{end}}|{{range $k, $v := .NetworkSettings.Networks}}{{$k}},{{end}}`
|
||||
if out, err := dockerRun(ctx, append([]string{"inspect", "-f", f}, strings.Fields(string(ids))...)...); err == nil {
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
p := strings.SplitN(line, "|", 4)
|
||||
if len(p) < 4 {
|
||||
continue
|
||||
}
|
||||
name := strings.TrimPrefix(p[0], "/")
|
||||
if id := strings.TrimPrefix(p[1], "sha256:"); len(id) >= 12 {
|
||||
imageUsed[id[:12]] = true
|
||||
}
|
||||
for _, v := range strings.Split(p[2], ",") {
|
||||
if v != "" {
|
||||
volumeUsers[v] = append(volumeUsers[v], name)
|
||||
}
|
||||
}
|
||||
for _, n := range strings.Split(p[3], ",") {
|
||||
if n != "" {
|
||||
networkUsers[n] = append(networkUsers[n], name)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
images := list("images", "--format", "{{json .}}")
|
||||
for _, im := range images {
|
||||
im["in_use"] = imageUsed[dstr(im, "ID")]
|
||||
}
|
||||
volumes := list("volume", "ls", "--format", "{{json .}}")
|
||||
for _, v := range volumes {
|
||||
users := volumeUsers[dstr(v, "Name")]
|
||||
if users == nil {
|
||||
users = []string{}
|
||||
}
|
||||
sort.Strings(users)
|
||||
v["used_by"] = users
|
||||
v["anonymous"] = strings.Contains(dstr(v, "Labels"), "com.docker.volume.anonymous")
|
||||
}
|
||||
networks := list("network", "ls", "--format", "{{json .}}")
|
||||
for _, n := range networks {
|
||||
users := networkUsers[dstr(n, "Name")]
|
||||
if users == nil {
|
||||
users = []string{}
|
||||
}
|
||||
sort.Strings(users)
|
||||
n["used_by"] = users
|
||||
n["predefined"] = predefinedNetwork[dstr(n, "Name")]
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"available": true, "tool": dockerTool(), "containers": containers,
|
||||
"images": images, "volumes": volumes, "networks": networks})
|
||||
}
|
||||
|
||||
var dockerRef = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}$`)
|
||||
|
||||
func (s *Server) dockerAction(w http.ResponseWriter, r *http.Request) {
|
||||
id, action := r.PathValue("id"), r.PathValue("action")
|
||||
if !dockerRef.MatchString(id) || (action != "start" && action != "stop" && action != "restart" && action != "remove") {
|
||||
jsonErr(w, 400, "start, stop, restart or remove a container by name or ID")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
verb := action
|
||||
if action == "remove" { // never forced: a running container has to be stopped first, on purpose
|
||||
verb = "rm"
|
||||
}
|
||||
if _, err := dockerRun(ctx, verb, id); err != nil {
|
||||
code := 500
|
||||
if action == "remove" {
|
||||
code = 409
|
||||
}
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: %s %s (from %s)", action, id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
var imageRef = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,254}$`)
|
||||
|
||||
// imagePull runs `docker pull` as a task the page follows.
|
||||
func (s *Server) imagePull(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Image string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !imageRef.MatchString(strings.TrimSpace(body.Image)) {
|
||||
jsonErr(w, 400, "an image looks like nginx:alpine or ghcr.io/owner/name:tag")
|
||||
return
|
||||
}
|
||||
ref := strings.TrimSpace(body.Image)
|
||||
t, err := s.tasks.Start("docker:pull", "pull "+ref, exec.Command(dockerTool(), "pull", ref))
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: pull %s as task %s (from %s)", ref, t.ID, clientIP(r))
|
||||
jsonOK(w, 202, t.Snapshot(false))
|
||||
}
|
||||
|
||||
// imageRemove is `docker rmi`, unforced: an image something still uses is refused by Docker, and that is the answer.
|
||||
func (s *Server) imageRemove(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Image string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !imageRef.MatchString(strings.TrimSpace(body.Image)) {
|
||||
jsonErr(w, 400, "which image?")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
if _, err := dockerRun(ctx, "rmi", strings.TrimSpace(body.Image)); err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: rmi %s (from %s)", body.Image, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// imagePrune removes dangling images — layers no tag points at any more, left behind by pulls —
|
||||
// or, asked for {"all": true}, every image no container uses. Those can all be pulled again.
|
||||
func (s *Server) imagePrune(w http.ResponseWriter, r *http.Request) {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
var body struct{ All bool }
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
args := []string{"image", "prune", "-f"}
|
||||
if body.All {
|
||||
args = append(args, "-a")
|
||||
}
|
||||
out, err := dockerRun(ctx, args...)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
log.Printf("docker: image prune (from %s)", clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
|
||||
}
|
||||
|
||||
// dockerLogs is the tail of a container's log; ?since=<RFC3339> asks only for what is new,
|
||||
// which is how the page follows a log without holding a connection open.
|
||||
func (s *Server) dockerLogs(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
args := []string{"logs", "--timestamps"}
|
||||
if since := r.URL.Query().Get("since"); since != "" {
|
||||
if _, err := time.Parse(time.RFC3339Nano, since); err != nil {
|
||||
jsonErr(w, 400, "since is an RFC 3339 time")
|
||||
return
|
||||
}
|
||||
args = append(args, "--since", since)
|
||||
} else {
|
||||
n, _ := strconv.Atoi(r.URL.Query().Get("n"))
|
||||
if n <= 0 || n > 5000 {
|
||||
n = 300
|
||||
}
|
||||
args = append(args, "--tail", strconv.Itoa(n))
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, append(args, id)...)
|
||||
if err != nil {
|
||||
code := 500
|
||||
if strings.Contains(strings.ToLower(err.Error()), "no such container") {
|
||||
code = 404
|
||||
}
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
type line struct {
|
||||
T string `json:"t"`
|
||||
M string `json:"m"`
|
||||
}
|
||||
lines := []line{}
|
||||
for _, l := range strings.Split(strings.TrimRight(string(out), "\n"), "\n") {
|
||||
if l == "" {
|
||||
continue
|
||||
}
|
||||
t, m, ok := strings.Cut(l, " ")
|
||||
if _, err := time.Parse(time.RFC3339Nano, t); !ok || err != nil {
|
||||
t, m = "", l
|
||||
}
|
||||
lines = append(lines, line{t, m})
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"lines": lines})
|
||||
}
|
||||
|
||||
// dockerInspect is the parts of `docker inspect` worth reading on a phone.
|
||||
// dockerErr answers for a docker command that failed: a thing that isn't there is a 404,
|
||||
// the daemon being away a 503, anything else a 500.
|
||||
func dockerErr(w http.ResponseWriter, err error) {
|
||||
msg := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(err.Error()), "error: "))
|
||||
low := strings.ToLower(msg)
|
||||
switch {
|
||||
case strings.Contains(low, "no such object") || strings.Contains(low, "no such container") || strings.Contains(low, "not found"):
|
||||
jsonErr(w, 404, msg)
|
||||
case strings.Contains(low, "cannot connect to the docker daemon") || strings.Contains(low, "is the docker daemon running"):
|
||||
jsonErr(w, 503, msg)
|
||||
default:
|
||||
jsonErr(w, 500, msg)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "inspect", id)
|
||||
if err != nil {
|
||||
dockerErr(w, err)
|
||||
return
|
||||
}
|
||||
var all []map[string]any
|
||||
if json.Unmarshal(out, &all) != nil || len(all) == 0 {
|
||||
jsonErr(w, 500, "inspect: unreadable answer")
|
||||
return
|
||||
}
|
||||
c := all[0]
|
||||
dig := func(m map[string]any, path ...string) any {
|
||||
var cur any = m
|
||||
for _, p := range path {
|
||||
mm, ok := cur.(map[string]any)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
cur = mm[p]
|
||||
}
|
||||
return cur
|
||||
}
|
||||
env := []string{}
|
||||
if list, ok := dig(c, "Config", "Env").([]any); ok {
|
||||
for _, e := range list {
|
||||
k, _, _ := strings.Cut(fmt.Sprint(e), "=")
|
||||
env = append(env, k) // names only: values are where the secrets live
|
||||
}
|
||||
}
|
||||
sort.Strings(env)
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"image": dig(c, "Config", "Image"), "command": dig(c, "Config", "Cmd"), "entrypoint": dig(c, "Config", "Entrypoint"),
|
||||
"restart": dig(c, "HostConfig", "RestartPolicy", "Name"), "network_mode": dig(c, "HostConfig", "NetworkMode"),
|
||||
"memory_limit": dig(c, "HostConfig", "Memory"), "started": dig(c, "State", "StartedAt"), "exit_code": dig(c, "State", "ExitCode"),
|
||||
"error": dig(c, "State", "Error"), "restarts": c["RestartCount"], "mounts": c["Mounts"], "ports": dig(c, "NetworkSettings", "Ports"),
|
||||
"networks": dig(c, "NetworkSettings", "Networks"), "env_names": env,
|
||||
"compose_file": dig(c, "Config", "Labels", "com.docker.compose.project.config_files"),
|
||||
})
|
||||
}
|
||||
|
||||
var volumeRef = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,254}$`)
|
||||
|
||||
// volumeRemove deletes one volume and what is in it, which nothing brings back. So it is never
|
||||
// forced (Docker refuses a volume a container still uses), it is one at a time, and the request
|
||||
// has to name the volume twice — a slip of a script can't delete one by accident.
|
||||
func (s *Server) volumeRemove(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name, Confirm string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !volumeRef.MatchString(body.Name) {
|
||||
jsonErr(w, 400, "which volume?")
|
||||
return
|
||||
}
|
||||
if body.Confirm != body.Name {
|
||||
jsonErr(w, 400, "name the volume again in \"confirm\" to delete it")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
if _, err := dockerRun(ctx, "volume", "rm", body.Name); err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: volume rm %s (from %s)", body.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// volumePrune removes unused *anonymous* volumes only: the nameless ones an image's VOLUME line
|
||||
// leaves behind each time its container is recreated. Named volumes are somebody's data, and a
|
||||
// stack that is merely down leaves its volumes "unused" — those go one at a time, by name.
|
||||
func (s *Server) volumePrune(w http.ResponseWriter, r *http.Request) {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "volume", "prune", "-f")
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
log.Printf("docker: volume prune (from %s)", clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
|
||||
}
|
||||
|
||||
// The three networks Docker makes itself. They hold nothing, cannot be removed, and
|
||||
// their absence would break every container that asks for one by name.
|
||||
var predefinedNetwork = map[string]bool{"bridge": true, "host": true, "none": true}
|
||||
|
||||
// networkRemove deletes one network. Unlike a volume there is no data in it, and unlike an
|
||||
// image nothing has to be downloaded again — a stack that wants it back gets it recreated
|
||||
// on the next deploy. Docker refuses one a container is still attached to, which is the
|
||||
// check that matters, so this asks for no second confirmation.
|
||||
func (s *Server) networkRemove(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !volumeRef.MatchString(body.Name) {
|
||||
jsonErr(w, 400, "which network?")
|
||||
return
|
||||
}
|
||||
if predefinedNetwork[body.Name] {
|
||||
jsonErr(w, 400, body.Name+" is one of Docker's own networks and can't be removed")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
if out, err := dockerRun(ctx, "network", "rm", body.Name); err != nil {
|
||||
jsonErr(w, 409, dockerWhy(out, err))
|
||||
return
|
||||
}
|
||||
log.Printf("docker: network rm %s (from %s)", body.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// networkPrune removes every network no container is attached to. A stack that is merely
|
||||
// down counts as unattached, so its network goes too — and comes back on the next deploy,
|
||||
// with a new subnet. That is the whole cost, and the caller is told it.
|
||||
func (s *Server) networkPrune(w http.ResponseWriter, r *http.Request) {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "network", "prune", "-f")
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
log.Printf("docker: network prune (from %s)", clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
|
||||
}
|
||||
|
||||
// ---- a container's health, its recent life, its console, and its limits ----
|
||||
|
||||
// dockerHealth is the health check's verdict and the last few probe results. Docker keeps
|
||||
// the last five itself, which is all anyone wants: is it failing, and what did it say.
|
||||
func (s *Server) dockerHealth(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id)
|
||||
if err != nil {
|
||||
dockerErr(w, err)
|
||||
return
|
||||
}
|
||||
var st struct {
|
||||
Status string `json:"Status"`
|
||||
Health *struct {
|
||||
Status string `json:"Status"`
|
||||
FailingStreak int `json:"FailingStreak"`
|
||||
Log []struct {
|
||||
Start time.Time `json:"Start"`
|
||||
End time.Time `json:"End"`
|
||||
ExitCode int `json:"ExitCode"`
|
||||
Output string `json:"Output"`
|
||||
} `json:"Log"`
|
||||
} `json:"Health"`
|
||||
}
|
||||
if json.Unmarshal(out, &st) != nil {
|
||||
jsonErr(w, 500, "could not read the container's state")
|
||||
return
|
||||
}
|
||||
if st.Health == nil {
|
||||
jsonOK(w, 200, map[string]any{"configured": false, "state": st.Status})
|
||||
return
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"configured": true, "state": st.Status, "status": st.Health.Status,
|
||||
"failing_streak": st.Health.FailingStreak, "log": st.Health.Log,
|
||||
})
|
||||
}
|
||||
|
||||
// dockerExec opens a shell inside a container, as a terminal session like any other — so
|
||||
// it reattaches, resizes and scrolls back the same way, and works on a managed machine
|
||||
// through the same proxy.
|
||||
func (s *Server) dockerExec(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
jsonErr(w, 500, "neither docker nor podman is installed here")
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
Shell string `json:"shell"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
shell := strings.TrimSpace(body.Shell)
|
||||
if shell == "" {
|
||||
// most images have sh; bash is nicer where it exists, so try it first and fall back
|
||||
shell = "if command -v bash >/dev/null 2>&1; then exec bash; else exec sh; fi"
|
||||
} else if strings.ContainsAny(shell, "\n\r") {
|
||||
jsonErr(w, 400, "bad shell")
|
||||
return
|
||||
}
|
||||
argv := []string{tool, "exec", "-it", id, "sh", "-c", shell}
|
||||
if priv.Available() && !dockerReachable() {
|
||||
argv = priv.Wrap(argv...)
|
||||
}
|
||||
sess, err := s.mgr.CreateCmd("", "sh "+shortName(id), argv)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker exec into %s as terminal %s by %s", id, sess.ID, clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"id": sess.ID, "name": "sh " + shortName(id)})
|
||||
}
|
||||
|
||||
func shortName(id string) string {
|
||||
if len(id) > 20 {
|
||||
return id[:20]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// dockerReachable reports whether docker answers without help.
|
||||
func dockerReachable() bool {
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
return false
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
return exec.CommandContext(ctx, tool, "version", "--format", "{{.Server.Version}}").Run() == nil
|
||||
}
|
||||
|
||||
// dockerUpdate changes a running container's limits and restart policy. `docker update`
|
||||
// does this live, without recreating it and without touching the compose file — which is
|
||||
// also the catch, and the caller is told: a deploy from compose puts back whatever compose
|
||||
// says.
|
||||
func (s *Server) dockerUpdate(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
Memory string `json:"memory"` // "512m", "2g", or "" to leave alone; "0" removes the limit
|
||||
CPUs string `json:"cpus"` // "1.5", or "0" for no limit
|
||||
Restart string `json:"restart"` // no | on-failure | unless-stopped | always
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
args := []string{"update"}
|
||||
if m := strings.TrimSpace(body.Memory); m != "" {
|
||||
if !regexp.MustCompile(`^[0-9]+([bkmgBKMG])?$`).MatchString(m) {
|
||||
jsonErr(w, 400, "memory: a number, optionally with b, k, m or g — 0 removes the limit")
|
||||
return
|
||||
}
|
||||
args = append(args, "--memory", m, "--memory-swap", m)
|
||||
}
|
||||
if c := strings.TrimSpace(body.CPUs); c != "" {
|
||||
if !regexp.MustCompile(`^[0-9]+(\.[0-9]+)?$`).MatchString(c) {
|
||||
jsonErr(w, 400, "cpus: a number like 1.5 — 0 removes the limit")
|
||||
return
|
||||
}
|
||||
args = append(args, "--cpus", c)
|
||||
}
|
||||
if p := strings.TrimSpace(body.Restart); p != "" {
|
||||
switch p {
|
||||
case "no", "on-failure", "unless-stopped", "always":
|
||||
default:
|
||||
jsonErr(w, 400, "restart: no, on-failure, unless-stopped or always")
|
||||
return
|
||||
}
|
||||
args = append(args, "--restart", p)
|
||||
}
|
||||
if len(args) == 1 {
|
||||
jsonErr(w, 400, "nothing to change")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
|
||||
defer cancel()
|
||||
if _, err := dockerRun(ctx, append(args, id)...); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker update %s %v by %s", id, args[1:], clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package api
|
||||
|
||||
// A container's history.
|
||||
//
|
||||
// `docker events` is a live stream with only a short in-memory backlog — ask it for the
|
||||
// last thirty days and you get the last few minutes, because the daemon keeps events in a
|
||||
// small ring and loses them all when it restarts. That is why Dockhand kept its own copy,
|
||||
// and why this does too: a watcher follows the stream and appends the lifecycle events to a
|
||||
// ring kept beside the config, so "what happened to this container last Tuesday" has an
|
||||
// answer.
|
||||
//
|
||||
// Only lifecycle events are kept. A health check produces three exec events every probe,
|
||||
// which is thousands a day and tells you nothing that the health panel does not.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
const (
|
||||
eventsKept = 4000
|
||||
eventsSaveEvery = 30 * time.Second
|
||||
)
|
||||
|
||||
// ContainerEvent is one thing that happened to one container.
|
||||
type ContainerEvent struct {
|
||||
T int64 `json:"t"` // unix ms
|
||||
Name string `json:"name"`
|
||||
Action string `json:"action"`
|
||||
Image string `json:"image,omitempty"`
|
||||
ExitCode string `json:"exit_code,omitempty"`
|
||||
Signal string `json:"signal,omitempty"`
|
||||
Health string `json:"health,omitempty"`
|
||||
}
|
||||
|
||||
type eventLog struct {
|
||||
mu sync.Mutex
|
||||
ring []ContainerEvent
|
||||
dirty bool
|
||||
path string
|
||||
}
|
||||
|
||||
func (e *eventLog) load(path string) {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
e.path = path
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
var list []ContainerEvent
|
||||
if json.Unmarshal(b, &list) == nil {
|
||||
e.ring = list
|
||||
}
|
||||
}
|
||||
|
||||
func (e *eventLog) add(ev ContainerEvent) {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
e.ring = append(e.ring, ev)
|
||||
if len(e.ring) > eventsKept {
|
||||
e.ring = append([]ContainerEvent(nil), e.ring[len(e.ring)-eventsKept:]...)
|
||||
}
|
||||
e.dirty = true
|
||||
}
|
||||
|
||||
// forName is one container's events, newest first.
|
||||
func (e *eventLog) forName(name string, since time.Time, limit int) []ContainerEvent {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
out := []ContainerEvent{}
|
||||
for _, ev := range e.ring {
|
||||
if ev.Name == name && ev.T >= since.UnixMilli() {
|
||||
out = append(out, ev)
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].T > out[j].T })
|
||||
if len(out) > limit {
|
||||
out = out[:limit]
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (e *eventLog) save() {
|
||||
e.mu.Lock()
|
||||
if !e.dirty || e.path == "" {
|
||||
e.mu.Unlock()
|
||||
return
|
||||
}
|
||||
b, err := json.Marshal(e.ring)
|
||||
e.dirty = false
|
||||
path := e.path
|
||||
e.mu.Unlock()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
_ = os.MkdirAll(filepath.Dir(path), 0o700)
|
||||
tmp := path + ".tmp"
|
||||
if os.WriteFile(tmp, b, 0o600) == nil {
|
||||
_ = os.Rename(tmp, path)
|
||||
}
|
||||
}
|
||||
|
||||
// watchDockerEvents follows the stream, restarting if Docker goes away. It is the only
|
||||
// thing that writes the log, so nothing contends with it.
|
||||
func (s *Server) watchDockerEvents(ctx context.Context) {
|
||||
s.events.load(filepath.Join(s.cfg.DataDir(), "docker-events.json"))
|
||||
go func() { // a periodic flush, so a crash loses at most half a minute of history
|
||||
t := time.NewTicker(eventsSaveEvery)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
s.events.save()
|
||||
return
|
||||
case <-t.C:
|
||||
s.events.save()
|
||||
}
|
||||
}
|
||||
}()
|
||||
for {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
if dockerTool() == "" {
|
||||
time.Sleep(time.Minute) // no Docker here; look again later in case it is installed
|
||||
continue
|
||||
}
|
||||
s.followEvents(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(10 * time.Second): // the daemon restarted, or the pipe broke
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) followEvents(ctx context.Context) {
|
||||
args := []string{"events", "--format", "{{json .}}", "--filter", "type=container"}
|
||||
for _, e := range []string{"create", "start", "restart", "stop", "kill", "die", "destroy", "oom", "pause", "unpause", "rename", "update", "health_status"} {
|
||||
args = append(args, "--filter", "event="+e)
|
||||
}
|
||||
tool := dockerTool()
|
||||
full := append([]string{tool}, args...)
|
||||
if !dockerReachable() && priv.Available() {
|
||||
full = priv.Wrap(full...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return
|
||||
}
|
||||
sc := bufio.NewScanner(out)
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
n := 0
|
||||
for sc.Scan() {
|
||||
var e struct {
|
||||
Action string `json:"Action"`
|
||||
Time int64 `json:"time"`
|
||||
TimeNano int64 `json:"timeNano"`
|
||||
Actor struct {
|
||||
Attributes map[string]string `json:"Attributes"`
|
||||
} `json:"Actor"`
|
||||
}
|
||||
if json.Unmarshal(sc.Bytes(), &e) != nil || e.Action == "" {
|
||||
continue
|
||||
}
|
||||
at := e.Actor.Attributes
|
||||
ev := ContainerEvent{
|
||||
T: e.Time * 1000, Name: at["name"], Image: at["image"],
|
||||
Action: e.Action, ExitCode: at["exitCode"], Signal: at["signal"],
|
||||
}
|
||||
if e.TimeNano > 0 {
|
||||
ev.T = e.TimeNano / 1e6
|
||||
}
|
||||
// health_status comes through as "health_status: healthy"
|
||||
if a, h, ok := strings.Cut(e.Action, ": "); ok {
|
||||
ev.Action, ev.Health = a, h
|
||||
}
|
||||
if ev.Name == "" {
|
||||
continue
|
||||
}
|
||||
s.events.add(ev)
|
||||
n++
|
||||
}
|
||||
_ = cmd.Wait()
|
||||
if n > 0 {
|
||||
log.Printf("docker events: stream ended after %d events; reconnecting", n)
|
||||
}
|
||||
}
|
||||
|
||||
// dockerEvents answers from Ham Lab's own history, not Docker's short memory.
|
||||
func (s *Server) dockerEvents(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
since := r.URL.Query().Get("since")
|
||||
d := 7 * 24 * time.Hour
|
||||
switch since {
|
||||
case "24h":
|
||||
d = 24 * time.Hour
|
||||
case "30d":
|
||||
d = 30 * 24 * time.Hour
|
||||
default:
|
||||
since = "7d"
|
||||
}
|
||||
ev := s.events.forName(id, time.Now().Add(-d), 500)
|
||||
s.events.mu.Lock()
|
||||
total := len(s.events.ring)
|
||||
s.events.mu.Unlock()
|
||||
jsonOK(w, 200, map[string]any{"events": ev, "since": since, "kept": total, "capacity": eventsKept})
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package api
|
||||
|
||||
// Following a container's log over a held connection, the way the Services tab follows a
|
||||
// journal. The polled `?since=` endpoint next door still exists — it is what a reconnect
|
||||
// hands back to pick up from — but the live view holds one `docker logs -f` per viewer
|
||||
// instead of asking again every two seconds.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"os/exec"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/coder/websocket"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// logLine is one line of a container's output: the time Docker stamped it with, the text,
|
||||
// and which of the two streams it came out of — stderr is where a container says what went
|
||||
// wrong, and the page colours it accordingly.
|
||||
type logLine struct {
|
||||
T string `json:"t"`
|
||||
M string `json:"m"`
|
||||
Err bool `json:"err,omitempty"`
|
||||
at string // when to sort it by: its own stamp, or the last one its stream carried
|
||||
}
|
||||
|
||||
// splitStamp pulls Docker's RFC 3339 prefix off a line. A line without one (a container
|
||||
// that writes bare bytes, or a continuation) keeps the whole text and no time — but is
|
||||
// still sorted where its stream put it, behind the last line that did have one.
|
||||
func splitStamp(l string, carry *string) logLine {
|
||||
t, m, ok := strings.Cut(l, " ")
|
||||
if _, err := time.Parse(time.RFC3339Nano, t); !ok || err != nil {
|
||||
return logLine{M: l, at: *carry}
|
||||
}
|
||||
*carry = t
|
||||
return logLine{T: t, M: m, at: t}
|
||||
}
|
||||
|
||||
// dockerTail streams a container's output as JSON text frames until the client goes away.
|
||||
// `?since=<RFC3339>` resumes where a dropped connection left off; otherwise it starts with
|
||||
// the last `?n=` lines.
|
||||
func (s *Server) dockerTail(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
jsonErr(w, 500, "neither docker nor podman is installed here")
|
||||
return
|
||||
}
|
||||
args := []string{tool, "logs", "--timestamps", "--follow"}
|
||||
if since := r.URL.Query().Get("since"); since != "" {
|
||||
if _, err := time.Parse(time.RFC3339Nano, since); err != nil {
|
||||
jsonErr(w, 400, "since is an RFC 3339 time")
|
||||
return
|
||||
}
|
||||
args = append(args, "--since", since)
|
||||
} else {
|
||||
args = append(args, "--tail", strconv.Itoa(logCount(r)))
|
||||
}
|
||||
args = append(args, id)
|
||||
if priv.Available() && !dockerReachable() {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
|
||||
conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{CompressionMode: websocket.CompressionDisabled})
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer conn.Close(websocket.StatusNormalClosure, "")
|
||||
ctx, cancel := context.WithCancel(r.Context())
|
||||
defer cancel()
|
||||
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
// Own process group, so the kill reaches the docker client itself and not just a sudo
|
||||
// wrapper — the same reason the journal tail does it.
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error {
|
||||
pg := "-" + strconv.Itoa(cmd.Process.Pid)
|
||||
if args[0] != tool { // running under sudo: only sudo can signal root's children
|
||||
k := priv.Wrap("kill", "-9", "--", pg)
|
||||
return exec.Command(k[0], k[1:]...).Run()
|
||||
}
|
||||
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||
}
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
stderr, err := cmd.StderrPipe()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
_ = conn.Write(ctx, websocket.MessageText, mustJSON(logLine{M: err.Error(), Err: true}))
|
||||
return
|
||||
}
|
||||
|
||||
// Both pipes funnel into one writer: a websocket takes one writer at a time.
|
||||
lines := make(chan logLine, 256)
|
||||
read := func(p io.Reader, isErr bool) {
|
||||
sc := bufio.NewScanner(p)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
|
||||
carry := ""
|
||||
for sc.Scan() {
|
||||
l := splitStamp(sc.Text(), &carry)
|
||||
l.Err = isErr
|
||||
select {
|
||||
case lines <- l:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
done := make(chan struct{}, 2)
|
||||
go func() { read(stdout, false); done <- struct{}{} }()
|
||||
go func() { read(stderr, true); done <- struct{}{} }()
|
||||
go func() { // the command ends when the container does; closing says so, having said why
|
||||
<-done
|
||||
<-done
|
||||
_ = cmd.Wait()
|
||||
close(lines)
|
||||
}()
|
||||
|
||||
go func() { // reader: only to notice the close
|
||||
defer cancel()
|
||||
for {
|
||||
if _, _, err := conn.Read(ctx); err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
ping := time.NewTicker(25 * time.Second)
|
||||
defer ping.Stop()
|
||||
// stdout and stderr are two pipes, and what Docker replays down them arrives in no
|
||||
// particular order. So the opening burst is held for a moment and sorted by the time
|
||||
// Docker stamped each line with; after that the log is live and arrives in order anyway.
|
||||
const burstFor = 300 * time.Millisecond
|
||||
warm := time.NewTimer(burstFor)
|
||||
defer warm.Stop()
|
||||
warmC := warm.C // nil once the burst has been flushed: a nil channel never fires
|
||||
var burst []logLine
|
||||
write := func(l logLine) bool {
|
||||
return conn.Write(ctx, websocket.MessageText, mustJSON(l)) == nil
|
||||
}
|
||||
flush := func() bool {
|
||||
sort.SliceStable(burst, func(i, j int) bool { return burst[i].at < burst[j].at })
|
||||
for _, l := range burst {
|
||||
if !write(l) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
burst = nil
|
||||
return true
|
||||
}
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case l, ok := <-lines:
|
||||
if !ok { // the container stopped, or never existed: say the last of it and go
|
||||
flush()
|
||||
return
|
||||
}
|
||||
if warmC != nil {
|
||||
burst = append(burst, l)
|
||||
if len(burst) < 10000 { // a container that never stops talking still gets read
|
||||
continue
|
||||
}
|
||||
if !flush() {
|
||||
return
|
||||
}
|
||||
continue
|
||||
}
|
||||
if !write(l) {
|
||||
return
|
||||
}
|
||||
case <-warmC:
|
||||
warmC = nil
|
||||
if !flush() {
|
||||
return
|
||||
}
|
||||
case <-ping.C:
|
||||
pctx, pc := context.WithTimeout(ctx, 10*time.Second)
|
||||
err := conn.Ping(pctx)
|
||||
pc()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func mustJSON(v any) []byte { b, _ := json.Marshal(v); return b }
|
||||
@@ -0,0 +1,356 @@
|
||||
package api
|
||||
|
||||
// Scheduled image updates.
|
||||
//
|
||||
// Dockhand kept three containers up to date nightly — unbound, nginx and pihole, which is
|
||||
// DNS and ingress — and that is the one thing it does that Ham Lab did not. This replaces
|
||||
// it, and deliberately in the same shape as the ZFS snapshot schedules: a real line in a
|
||||
// cron file Ham Lab owns, calling the binary, rather than a ticker inside the daemon.
|
||||
// Updating an image is not something that should stop happening because a web console is
|
||||
// down, and a cron line is visible to anyone looking at the machine the normal way.
|
||||
//
|
||||
// A schedule stores everything the run needs — project, compose files, how compose has to
|
||||
// be invoked for that stack — so the cron job is self-contained and does not have to
|
||||
// rediscover the world before it can act.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/compose"
|
||||
"hamlab/internal/cron"
|
||||
"hamlab/internal/storage"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// UpdateJob is a standing instruction to pull a stack's images, and optionally deploy them.
|
||||
type UpdateJob struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Stack string `json:"stack"`
|
||||
Machine string `json:"machine,omitempty"` // for display; the job runs where it was made
|
||||
// Services narrows it to some of the stack's services; empty is all of them.
|
||||
Services []string `json:"services,omitempty"`
|
||||
Cron string `json:"cron"`
|
||||
// Apply says whether a newer image is deployed or only fetched. Check-only still pulls,
|
||||
// so the Docker tab shows what is behind without a registry round trip.
|
||||
Apply bool `json:"apply"`
|
||||
Enabled bool `json:"enabled"`
|
||||
|
||||
// How to run compose for this stack, captured when the schedule was made.
|
||||
Spec composeSpec `json:"spec"`
|
||||
}
|
||||
|
||||
type composeSpec struct {
|
||||
Project string `json:"project"`
|
||||
Files []string `json:"files"`
|
||||
Via string `json:"via,omitempty"`
|
||||
// Remote is read but never written: schedules made before v0.2 could point compose at
|
||||
// another machine's Docker over tcp. Such a job is refused rather than run against this
|
||||
// machine's Docker, which is where it would land without the variable.
|
||||
Remote string `json:"remote,omitempty"`
|
||||
}
|
||||
|
||||
func (c composeSpec) toSpec(tool string) compose.Spec {
|
||||
return compose.Spec{Project: c.Project, Files: c.Files, Via: c.Via, Tool: tool}
|
||||
}
|
||||
|
||||
const dockerCronFile = "/etc/cron.d/hamlab-docker"
|
||||
|
||||
type dockerSchedState struct {
|
||||
Jobs []UpdateJob `json:"jobs"`
|
||||
}
|
||||
|
||||
func (s *Server) dsLoad() dockerSchedState {
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.DockerJobs })
|
||||
var d dockerSchedState
|
||||
if len(raw) > 0 {
|
||||
_ = json.Unmarshal(raw, &d)
|
||||
}
|
||||
if d.Jobs == nil {
|
||||
d.Jobs = []UpdateJob{}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func (s *Server) dsSave(d dockerSchedState) error {
|
||||
b, err := json.Marshal(d)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.store.Update(func(st *store.State) { st.DockerJobs = b })
|
||||
}
|
||||
|
||||
// dsCronText renders the schedules as cron lines. Same shape as the ZFS file: Ham Lab owns
|
||||
// it whole and rewrites it on every change.
|
||||
func dsCronText(jobs []UpdateJob, user, exe, config string) (string, error) {
|
||||
var b strings.Builder
|
||||
b.WriteString("# Ham Lab image-update schedules — written by the Docker tab, rewritten whole on every change.\n")
|
||||
b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
|
||||
b.WriteString("SHELL=/bin/sh\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
|
||||
for i := range jobs {
|
||||
j := &jobs[i]
|
||||
if !j.Enabled {
|
||||
b.WriteString("# " + j.Name + " — paused\n")
|
||||
continue
|
||||
}
|
||||
expr := strings.Join(strings.Fields(j.Cron), " ")
|
||||
if err := cron.ValidSchedule(expr); err != nil {
|
||||
return "", err
|
||||
}
|
||||
what := "pull only"
|
||||
if j.Apply {
|
||||
what = "pull and deploy"
|
||||
}
|
||||
b.WriteString("# " + j.Name + " — " + j.Stack + ", " + what + "\n")
|
||||
b.WriteString(expr + " " + user + " " + exe + " -config " + config + " docker-update " + j.ID + "\n\n")
|
||||
}
|
||||
return b.String(), nil
|
||||
}
|
||||
|
||||
func (s *Server) dsSyncCron(ctx context.Context, d dockerSchedState) error {
|
||||
live := false
|
||||
for i := range d.Jobs {
|
||||
if d.Jobs[i].Enabled {
|
||||
live = true
|
||||
}
|
||||
}
|
||||
if !live {
|
||||
return s.storage.RemoveRootFile(ctx, dockerCronFile)
|
||||
}
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
exe, _ = filepath.EvalSymlinks(exe)
|
||||
text, err := dsCronText(d.Jobs, s.user, exe, s.cfg.Path())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.storage.WriteRootFile(ctx, dockerCronFile, text)
|
||||
}
|
||||
|
||||
// repoint keeps a schedule working when what deployed its stack has been removed.
|
||||
//
|
||||
// A schedule captures how to invoke compose at the moment it is made, so the cron job never
|
||||
// has to rediscover the world at 3am. But "run compose inside this container" stops being
|
||||
// true the day that container is deleted — which is exactly what happens when an
|
||||
// orchestrator is retired, and would otherwise break every schedule made before it went.
|
||||
// So: if the container named is not running, the files are looked up again, through live
|
||||
// mounts and remembered ones both, and compose runs on the host instead.
|
||||
func repoint(ctx context.Context, db *store.Store, spec compose.Spec) compose.Spec {
|
||||
if spec.Via == "" || db == nil {
|
||||
return spec
|
||||
}
|
||||
if out, err := dockerRun(ctx, "inspect", "-f", "{{.State.Running}}", spec.Via); err == nil && strings.TrimSpace(string(out)) == "true" {
|
||||
return spec
|
||||
}
|
||||
st := stack{Name: spec.Project, File: spec.Files[0], Files: spec.Files}
|
||||
st.resolve(mountsSeen(ctx, db)) // read-only: this also runs from cron, where the daemon owns state.json
|
||||
if !st.Editable || st.Via != "" {
|
||||
return spec // still only reachable through a container, or not reachable at all
|
||||
}
|
||||
log.Printf("docker schedule %s: %s is gone, running compose here against %s", spec.Project, spec.Via, st.HostFiles[0])
|
||||
spec.Files, spec.Via = st.HostFiles, ""
|
||||
return spec
|
||||
}
|
||||
|
||||
// RunUpdate pulls, and deploys when the schedule says to. Shared with the CLI so the button
|
||||
// and the cron line cannot drift apart.
|
||||
func RunUpdate(ctx context.Context, db *store.Store, j UpdateJob, tool string) (string, error) {
|
||||
if len(j.Spec.Files) == 0 {
|
||||
return "", errors.New("this schedule has no compose file recorded — open it in the Docker tab and save it again")
|
||||
}
|
||||
if j.Spec.Remote != "" {
|
||||
return "", errors.New("this schedule was made to deploy to another machine over tcp, which this console no longer does — make it again on that machine")
|
||||
}
|
||||
spec := repoint(ctx, db, j.Spec.toSpec(tool))
|
||||
var log strings.Builder
|
||||
run := func(args ...string) error {
|
||||
cmd := compose.Cmd(ctx, spec, args...)
|
||||
out, err := cmd.CombinedOutput()
|
||||
log.Write(out)
|
||||
return err
|
||||
}
|
||||
args := append([]string{"pull"}, j.Services...)
|
||||
if err := run(args...); err != nil {
|
||||
return log.String(), err
|
||||
}
|
||||
if !j.Apply {
|
||||
return log.String(), nil
|
||||
}
|
||||
up := append([]string{"up", "-d"}, j.Services...)
|
||||
if err := run(up...); err != nil {
|
||||
return log.String(), err
|
||||
}
|
||||
return log.String(), nil
|
||||
}
|
||||
|
||||
// ---- endpoints ----
|
||||
|
||||
func (s *Server) dsList(w http.ResponseWriter, _ *http.Request) {
|
||||
d := s.dsLoad()
|
||||
now := time.Now()
|
||||
out := make([]map[string]any, 0, len(d.Jobs))
|
||||
for _, j := range d.Jobs {
|
||||
m := map[string]any{"job": j, "last": storage.ReadRun(s.cfg.DataDir(), "docker-"+j.ID)}
|
||||
if j.Enabled {
|
||||
if n, ok := cron.NextRun(strings.Join(strings.Fields(j.Cron), " "), now); ok {
|
||||
m["next"] = n
|
||||
}
|
||||
}
|
||||
out = append(out, m)
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"jobs": out, "file": dockerCronFile})
|
||||
}
|
||||
|
||||
func (s *Server) dsPut(w http.ResponseWriter, r *http.Request) {
|
||||
var j UpdateJob
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 32<<10)).Decode(&j); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
j.Cron = strings.Join(strings.Fields(j.Cron), " ")
|
||||
if err := cron.ValidSchedule(j.Cron); err != nil {
|
||||
jsonErr(w, 400, "schedule: "+err.Error())
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(j.Stack) == "" {
|
||||
jsonErr(w, 400, "which stack?")
|
||||
return
|
||||
}
|
||||
// resolve the stack now, so the cron job never has to
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
stacks, err := s.findStacks(ctx)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
found := false
|
||||
for _, st := range stacks {
|
||||
if st.Name == j.Stack {
|
||||
// seenFiles is the list as compose itself sees it: host paths normally, the
|
||||
// orchestrator container's own paths when compose runs inside one. Getting this
|
||||
// backwards gives a cron line that works from the console and fails at 3am.
|
||||
j.Spec = composeSpec{Project: st.Name, Files: seenFiles(st, -1, ""), Via: st.Via}
|
||||
if len(j.Spec.Files) == 0 {
|
||||
jsonErr(w, 409, "that stack's compose file isn't anywhere this console can reach")
|
||||
return
|
||||
}
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no stack called "+j.Stack+" on this machine")
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(j.Name) == "" {
|
||||
j.Name = j.Stack
|
||||
}
|
||||
d := s.dsLoad()
|
||||
replaced := false
|
||||
for i := range d.Jobs {
|
||||
if d.Jobs[i].ID == j.ID && j.ID != "" {
|
||||
d.Jobs[i] = j
|
||||
replaced = true
|
||||
}
|
||||
}
|
||||
if !replaced {
|
||||
b := make([]byte, 6)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
j.ID = hex.EncodeToString(b)
|
||||
d.Jobs = append(d.Jobs, j)
|
||||
}
|
||||
if err := s.dsSave(d); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.dsSyncCron(r.Context(), d); err != nil {
|
||||
jsonErr(w, 500, "saved, but writing "+dockerCronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker schedule %q saved by %s (%s, %q, %s)", j.Name, clientIP(r), j.Stack, j.Cron, map[bool]string{true: "pull and deploy", false: "pull only"}[j.Apply])
|
||||
jsonOK(w, 200, j)
|
||||
}
|
||||
|
||||
func (s *Server) dsDelete(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
d := s.dsLoad()
|
||||
kept := d.Jobs[:0]
|
||||
found := false
|
||||
for _, j := range d.Jobs {
|
||||
if j.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, j)
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
d.Jobs = kept
|
||||
if err := s.dsSave(d); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.dsSyncCron(r.Context(), d); err != nil {
|
||||
jsonErr(w, 500, "removed, but writing "+dockerCronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
storage.ForgetRun(s.cfg.DataDir(), "docker-"+id)
|
||||
log.Printf("docker schedule %s removed by %s", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) dsRun(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
var job UpdateJob
|
||||
found := false
|
||||
for _, j := range s.dsLoad().Jobs {
|
||||
if j.ID == id {
|
||||
job, found = j, true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
// Outliving the request: a pull the browser gave up on still finishes, but not forever.
|
||||
ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 30*time.Minute)
|
||||
defer cancel()
|
||||
out, err := RunUpdate(ctx, s.store, job, dockerTool())
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: job.Stack}
|
||||
if err != nil {
|
||||
rec.Error = err.Error()
|
||||
}
|
||||
_ = storage.WriteRun(s.cfg.DataDir(), "docker-"+id, rec)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error()+"\n"+tail(out, 2000))
|
||||
return
|
||||
}
|
||||
log.Printf("docker schedule %q run by %s", job.Name, clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"output": tail(out, 4000)})
|
||||
}
|
||||
|
||||
func tail(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return "…" + s[len(s)-n:]
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"hamlab/internal/files"
|
||||
)
|
||||
|
||||
// A file read answers Range requests — what a <video> needs to seek — and HEAD, through
|
||||
// the same handler that streams everything else.
|
||||
func TestFilesReadRanges(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "clip.mp4")
|
||||
data := make([]byte, 10000)
|
||||
for i := range data {
|
||||
data[i] = byte(i % 251)
|
||||
}
|
||||
if err := os.WriteFile(p, data, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{fsLocal: files.Local{}}
|
||||
get := func(method, extra string, hdr map[string]string) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest(method, "/api/files/read?path="+url.QueryEscape(p)+extra, nil)
|
||||
for k, v := range hdr {
|
||||
r.Header.Set(k, v)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
s.filesRead(w, r)
|
||||
return w
|
||||
}
|
||||
w := get("GET", "", nil)
|
||||
if w.Code != 200 || w.Body.Len() != 10000 || w.Header().Get("Content-Type") != "video/mp4" || w.Header().Get("Accept-Ranges") != "bytes" {
|
||||
t.Fatalf("whole: %d %d %q %q", w.Code, w.Body.Len(), w.Header().Get("Content-Type"), w.Header().Get("Accept-Ranges"))
|
||||
}
|
||||
w = get("GET", "", map[string]string{"Range": "bytes=100-199"})
|
||||
if w.Code != 206 || w.Header().Get("Content-Range") != "bytes 100-199/10000" || w.Body.Len() != 100 || w.Body.Bytes()[0] != data[100] {
|
||||
t.Fatalf("range: %d %q %d", w.Code, w.Header().Get("Content-Range"), w.Body.Len())
|
||||
}
|
||||
w = get("GET", "", map[string]string{"Range": "bytes=9990-"})
|
||||
if w.Code != 206 || w.Body.Len() != 10 || w.Body.Bytes()[9] != data[9999] {
|
||||
t.Fatalf("tail range: %d %d", w.Code, w.Body.Len())
|
||||
}
|
||||
w = get("HEAD", "", nil)
|
||||
if w.Code != 200 || w.Body.Len() != 0 || w.Header().Get("Content-Length") != "10000" {
|
||||
t.Fatalf("head: %d %d %q", w.Code, w.Body.Len(), w.Header().Get("Content-Length"))
|
||||
}
|
||||
if w := get("GET", "&max=50", nil); w.Code != 413 {
|
||||
t.Fatalf("the editor's cap: %d", w.Code)
|
||||
}
|
||||
if w := get("GET", "", map[string]string{"Range": "bytes=20000-"}); w.Code != 416 {
|
||||
t.Fatalf("a range past the end: %d", w.Code)
|
||||
}
|
||||
if err := os.Chmod(p, 0); err == nil && os.Geteuid() != 0 {
|
||||
if w := get("GET", "", nil); w.Code != 403 {
|
||||
t.Fatalf("unreadable: %d", w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A write over the cap is refused whole: the file is not truncated first.
|
||||
func TestFilesWriteCap(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "f.txt")
|
||||
if err := os.WriteFile(p, []byte("keep me"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{fsLocal: files.Local{}}
|
||||
body := make([]byte, 8<<20+1)
|
||||
r := httptest.NewRequest("PUT", "/api/files/write?path="+url.QueryEscape(p), bytes.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
s.filesWrite(w, r)
|
||||
if w.Code != 413 {
|
||||
t.Fatalf("got %d", w.Code)
|
||||
}
|
||||
if b, _ := os.ReadFile(p); string(b) != "keep me" {
|
||||
t.Fatalf("the file was touched: %q", b)
|
||||
}
|
||||
r = httptest.NewRequest("PUT", "/api/files/write?path="+url.QueryEscape(p), bytes.NewReader([]byte("new")))
|
||||
w = httptest.NewRecorder()
|
||||
s.filesWrite(w, r)
|
||||
if b, _ := os.ReadFile(p); w.Code != 204 || string(b) != "new" {
|
||||
t.Fatalf("a write within the cap: %d %q", w.Code, b)
|
||||
}
|
||||
}
|
||||
|
||||
// A link to nothing is a 404, not an empty 200.
|
||||
func TestFilesReadDanglingLink(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
l := filepath.Join(dir, "broken")
|
||||
if err := os.Symlink("nowhere", l); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{fsLocal: files.Local{}}
|
||||
r := httptest.NewRequest("GET", "/api/files/read?path="+url.QueryEscape(l), nil)
|
||||
w := httptest.NewRecorder()
|
||||
s.filesRead(w, r)
|
||||
if w.Code != 404 {
|
||||
t.Fatalf("got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// dockerErr tells a thing that isn't there from a daemon that isn't up from anything else.
|
||||
func TestDockerErr(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
msg string
|
||||
want int
|
||||
}{
|
||||
{"error: No such object: zzz", 404}, {"Error response from daemon: No such container: x", 404},
|
||||
{"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?", 503},
|
||||
{"something else entirely", 500},
|
||||
} {
|
||||
w := httptest.NewRecorder()
|
||||
dockerErr(w, errors.New(c.msg))
|
||||
if w.Code != c.want {
|
||||
t.Errorf("%q → %d, want %d", c.msg, w.Code, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
package api
|
||||
|
||||
// Guest logins: a temporary account for handing to somebody you want to show around. It is
|
||||
// an ordinary account with a clock on it — every tab, every action, on this machine and on
|
||||
// every machine this hub manages — that stops working at its expiry (half an hour by
|
||||
// default) or the moment it is revoked.
|
||||
//
|
||||
// Deliberately not a reduced account: Mitchell, 2026-09-27, replacing a read-only version
|
||||
// built the same day. He gives these to people he trusts and revokes them otherwise, and a
|
||||
// console with half its buttons missing is not the console he wanted to show off. What that
|
||||
// means in practice: a guest has the Terminal tab, which is a shell as the user the daemon
|
||||
// runs as — root on every host here — and can make an account or a token of its own that
|
||||
// outlives the guest login.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const (
|
||||
guestPrefix = "guest-"
|
||||
guestTTLDefault = 30 * time.Minute
|
||||
guestTTLMax = 24 * time.Hour
|
||||
guestReapEvery = time.Minute
|
||||
)
|
||||
|
||||
// ---- the accounts themselves ----
|
||||
|
||||
func (s *Server) listGuests(w http.ResponseWriter, r *http.Request) {
|
||||
now := time.Now()
|
||||
counts := s.sessions.Count()
|
||||
out := []map[string]any{}
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, u := range c.Users {
|
||||
if !u.Guest || u.Expired(now) {
|
||||
continue
|
||||
}
|
||||
out = append(out, map[string]any{
|
||||
"name": u.Name, "created": u.Created, "expires": u.Expires,
|
||||
"issued_by": u.IssuedBy, "sessions": counts[u.Name], "used": !u.LastSeen.IsZero(),
|
||||
})
|
||||
}
|
||||
})
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
// addGuest makes a guest login and hands back its password — the only time it is ever
|
||||
// readable, here as everywhere else a password is generated.
|
||||
func (s *Server) addGuest(w http.ResponseWriter, r *http.Request) {
|
||||
me := actorOf(r)
|
||||
var body struct {
|
||||
Minutes int `json:"minutes"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
ttl := guestTTLDefault
|
||||
if body.Minutes > 0 {
|
||||
ttl = time.Duration(body.Minutes) * time.Minute
|
||||
}
|
||||
if ttl > guestTTLMax {
|
||||
jsonErr(w, 400, "a guest login lasts at most 24 hours")
|
||||
return
|
||||
}
|
||||
pw, err := auth.GuestPassword()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
hash, err := auth.HashPassword(pw)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
var name string
|
||||
for try := 0; try < 8 && name == ""; try++ {
|
||||
suffix, err := auth.GuestSuffix()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
candidate := guestPrefix + suffix
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
if _, taken := c.User(candidate); !taken {
|
||||
name = candidate
|
||||
}
|
||||
})
|
||||
}
|
||||
if name == "" {
|
||||
jsonErr(w, 500, "could not find a free guest name")
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
expires := now.Add(ttl)
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
c.Users = append(c.Users, config.User{Name: name, Hash: hash, Created: now, Guest: true, Expires: expires, IssuedBy: me.User})
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("guest login %q issued until %s (by %s)", name, expires.Format(time.RFC3339), clientIP(r))
|
||||
jsonOK(w, 200, map[string]any{"name": name, "password": pw, "expires": expires, "minutes": int(ttl.Minutes())})
|
||||
}
|
||||
|
||||
// revokeGuest ends a guest login now. No password: it only ever takes access away, and the
|
||||
// whole point of the thing is that ending it is one tap.
|
||||
func (s *Server) revokeGuest(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
found := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.Users[:0]
|
||||
for _, u := range c.Users {
|
||||
if u.Name == name && u.Guest {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
c.Users = kept
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such guest login")
|
||||
return
|
||||
}
|
||||
s.forgetGuest(name)
|
||||
log.Printf("guest login %q revoked (by %s)", name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// forgetGuest signs a guest out everywhere and drops the look they left behind.
|
||||
func (s *Server) forgetGuest(name string) {
|
||||
s.sessions.RevokeUser(name)
|
||||
_ = s.store.Update(func(st *store.State) { delete(st.Users, name) })
|
||||
}
|
||||
|
||||
// watchGuests clears out guest logins whose time is up. Expiry is enforced on every request
|
||||
// as well — this is what keeps the config file from collecting them, and what signs a guest
|
||||
// out of a console they left open rather than waiting for their next click.
|
||||
func (s *Server) watchGuests() {
|
||||
for {
|
||||
s.reapGuests()
|
||||
time.Sleep(guestReapEvery)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) reapGuests() {
|
||||
now := time.Now()
|
||||
var gone []string
|
||||
var have bool
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, u := range c.Users {
|
||||
if u.Guest {
|
||||
have = true
|
||||
if u.Expired(now) {
|
||||
gone = append(gone, u.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
if !have || len(gone) == 0 {
|
||||
return
|
||||
}
|
||||
expired := map[string]bool{}
|
||||
for _, n := range gone {
|
||||
expired[n] = true
|
||||
}
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.Users[:0]
|
||||
for _, u := range c.Users {
|
||||
if expired[u.Name] {
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
c.Users = kept
|
||||
}); err != nil {
|
||||
log.Printf("guests: %v", err)
|
||||
return
|
||||
}
|
||||
for _, n := range gone {
|
||||
s.forgetGuest(n)
|
||||
}
|
||||
log.Printf("guest login(s) expired: %s", strings.Join(gone, ", "))
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// A guest login is an ordinary account with a clock on it, so the clock is the thing worth
|
||||
// testing: one whose time is up is stopped at the door, whether or not the minute-by-minute
|
||||
// sweep has reached their account yet, and their session is dropped on the way — so the
|
||||
// console they left open goes back to the login form rather than carrying on.
|
||||
func TestExpiredGuestIsRefusedAndSignedOut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
expires time.Time
|
||||
wantOK bool
|
||||
}{
|
||||
{"still has time", time.Now().Add(10 * time.Minute), true},
|
||||
{"time is up", time.Now().Add(-time.Second), false},
|
||||
{"no expiry at all", time.Time{}, false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cfg := config.Default(filepath.Join(dir, "config.json"))
|
||||
if err := cfg.Update(func(c *config.Config) {
|
||||
c.Users = []config.User{{Name: "guest-abcd", Hash: "x", Guest: true, Expires: tc.expires}}
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err := store.Open(filepath.Join(dir, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{cfg: cfg, store: st, sessions: auth.NewSessions(st)}
|
||||
tok, err := s.sessions.Create("guest-abcd", "10.0.0.5", "curl")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := httptest.NewRequest("GET", "/api/me", nil)
|
||||
r.AddCookie(&http.Cookie{Name: cookieName, Value: tok})
|
||||
w := httptest.NewRecorder()
|
||||
a, ok, answered := s.cookieActor(w, r)
|
||||
if ok != tc.wantOK {
|
||||
t.Fatalf("ok = %v, want %v", ok, tc.wantOK)
|
||||
}
|
||||
if tc.wantOK {
|
||||
if !a.Guest || a.User != "guest-abcd" {
|
||||
t.Errorf("actor = %+v, want the guest account marked as one", a)
|
||||
}
|
||||
if answered {
|
||||
t.Error("answered the caller for a guest that still has time")
|
||||
}
|
||||
return
|
||||
}
|
||||
if !answered || w.Code != 401 {
|
||||
t.Errorf("answered = %v, code = %d, want true and 401", answered, w.Code)
|
||||
}
|
||||
if _, live := s.sessions.Lookup(tok, "10.0.0.5"); live {
|
||||
t.Error("the session of an expired guest is still live")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package api
|
||||
|
||||
// The Hardware tab: what the machine is made of. Read once and remembered — none of it
|
||||
// changes while the kernel runs — with ?refresh=1 for the one thing that can, a drive
|
||||
// plugged in or pulled out.
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
)
|
||||
|
||||
func (s *Server) hardwareReport(w http.ResponseWriter, r *http.Request) {
|
||||
jsonOK(w, 200, s.hardware.Report(r.Context(), r.URL.Query().Get("refresh") == "1"))
|
||||
}
|
||||
@@ -0,0 +1,420 @@
|
||||
package api
|
||||
|
||||
// IP-KVMs: a device's own web page, shown inside Ham Lab.
|
||||
//
|
||||
// The whole difficulty is that Ham Lab is served over TLS and a KVM usually is not, so a
|
||||
// browser refuses to put the device's page in an iframe — mixed content, blocked before
|
||||
// anything loads. The fix is to serve the device from Ham Lab's own origin, which also
|
||||
// means it inherits Ham Lab's login instead of needing a second one, and makes the iframe
|
||||
// same-origin.
|
||||
//
|
||||
// Proxying an arbitrary web app under a path prefix is the hard part, and nothing here
|
||||
// knows anything about any particular device. A device UI asks for its assets by absolute
|
||||
// path — "/assets/app.js", "/api/whatever" — which a prefix cannot rewrite away. So the
|
||||
// prefix is only the entry point: any request Ham Lab does not serve itself is routed to
|
||||
// whichever KVM the asking page belongs to, worked out from the Referer, and from a cookie
|
||||
// when there is no Referer (a WebSocket upgrade, say). Ham Lab's own routes always win, so
|
||||
// a device can never shadow the console.
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const kvmPrefix = "/kvm/"
|
||||
const kvmCookie = "hamlab_kvm"
|
||||
|
||||
type kvmProxies struct {
|
||||
mu sync.Mutex
|
||||
m map[string]*httputil.ReverseProxy
|
||||
}
|
||||
|
||||
func (s *Server) kvm(id string) (config.KVM, bool) {
|
||||
var found config.KVM
|
||||
ok := false
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, k := range c.KVMs {
|
||||
if k.ID == id {
|
||||
found, ok = k, true
|
||||
}
|
||||
}
|
||||
})
|
||||
return found, ok
|
||||
}
|
||||
|
||||
// kvmProxy is the reverse proxy for one device, made once and kept.
|
||||
func (s *Server) kvmProxy(k config.KVM) (*httputil.ReverseProxy, error) {
|
||||
s.kvms.mu.Lock()
|
||||
defer s.kvms.mu.Unlock()
|
||||
if s.kvms.m == nil {
|
||||
s.kvms.m = map[string]*httputil.ReverseProxy{}
|
||||
}
|
||||
if p, ok := s.kvms.m[k.ID]; ok {
|
||||
return p, nil
|
||||
}
|
||||
target, err := url.Parse(k.URL)
|
||||
if err != nil || target.Host == "" {
|
||||
return nil, errors.New("this KVM's address is not a URL")
|
||||
}
|
||||
prefix := kvmPrefix + k.ID
|
||||
p := &httputil.ReverseProxy{
|
||||
Director: func(r *http.Request) {
|
||||
r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
|
||||
// Entry through the prefix keeps the path after it; anything else arrived by its
|
||||
// own absolute path and is passed through untouched.
|
||||
if rest, ok := strings.CutPrefix(r.URL.Path, prefix); ok {
|
||||
if rest == "" {
|
||||
rest = "/"
|
||||
}
|
||||
r.URL.Path = rest
|
||||
}
|
||||
r.URL.RawPath = ""
|
||||
r.Host = target.Host
|
||||
// Whether the console itself is on TLS: the response rewriter needs to know, and by
|
||||
// then only the outbound request is in hand.
|
||||
if secure(r) {
|
||||
r.Header.Set(kvmSecureHeader, "1")
|
||||
} else {
|
||||
r.Header.Del(kvmSecureHeader)
|
||||
}
|
||||
// The device gets its own host and none of Ham Lab's session.
|
||||
r.Header.Del("Origin")
|
||||
stripCookie(r, cookieName, kvmCookie)
|
||||
// What is left is the device's own login. A browser that has it teaches the hub; one
|
||||
// that has not is lent what the hub remembers, so the device sees a signed-in client.
|
||||
if own := r.Cookies(); len(own) > 0 {
|
||||
s.rememberKVMCookies(k.ID, own)
|
||||
} else {
|
||||
for name, val := range s.kvmCookies(k.ID) {
|
||||
r.AddCookie(&http.Cookie{Name: name, Value: val})
|
||||
}
|
||||
}
|
||||
},
|
||||
FlushInterval: -1, // a video stream must not be buffered
|
||||
ModifyResponse: func(resp *http.Response) error {
|
||||
// The device's own framing and transport policy would stop it being embedded, or
|
||||
// send a browser off to plain HTTP. Ham Lab is the origin now, and says what it
|
||||
// serves; its own headers are added on the way out.
|
||||
for _, h := range []string{"X-Frame-Options", "Content-Security-Policy", "Content-Security-Policy-Report-Only", "Strict-Transport-Security", "Referrer-Policy"} {
|
||||
resp.Header.Del(h)
|
||||
}
|
||||
// A cookie the device sets by header is kept for Ham Lab's origin — so its Domain
|
||||
// (the device's) goes, Secure follows the console rather than the device, and the
|
||||
// hub remembers it too.
|
||||
if set := resp.Cookies(); len(set) > 0 {
|
||||
https := resp.Request != nil && resp.Request.Header.Get(kvmSecureHeader) == "1"
|
||||
resp.Header.Del("Set-Cookie")
|
||||
for _, c := range set {
|
||||
c.Domain = ""
|
||||
if c.Path == "" {
|
||||
c.Path = "/"
|
||||
}
|
||||
c.Secure = c.Secure && https
|
||||
if v := c.String(); v != "" {
|
||||
resp.Header.Add("Set-Cookie", v)
|
||||
}
|
||||
}
|
||||
s.rememberKVMCookies(k.ID, set)
|
||||
}
|
||||
// A redirect to the device's own root has to come back through the prefix.
|
||||
if loc := resp.Header.Get("Location"); loc != "" {
|
||||
if u, err := url.Parse(loc); err == nil && (u.Host == "" || u.Host == target.Host) && strings.HasPrefix(u.Path, "/") && !strings.HasPrefix(u.Path, prefix) {
|
||||
u.Scheme, u.Host = "", ""
|
||||
u.Path = prefix + u.Path
|
||||
resp.Header.Set("Location", u.String())
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": k.Name + " unreachable: " + err.Error()})
|
||||
},
|
||||
}
|
||||
s.kvms.m[k.ID] = p
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// kvmSecureHeader rides on the outbound request to say whether the console was reached
|
||||
// over TLS; the device ignores it.
|
||||
const kvmSecureHeader = "X-Hamlab-Secure"
|
||||
|
||||
// kvmCookies is what the hub remembers of a device's own cookies — its login, in practice.
|
||||
func (s *Server) kvmCookies(id string) map[string]string {
|
||||
var out map[string]string
|
||||
s.store.View(func(st *store.State) {
|
||||
if m := st.KVMCookies[id]; len(m) > 0 {
|
||||
out = make(map[string]string, len(m))
|
||||
for k, v := range m {
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// rememberKVMCookies keeps the cookies a browser (or the device's own Set-Cookie) carries
|
||||
// for a KVM, so that a browser without them can be lent them. A cookie emptied or expired
|
||||
// by the device is forgotten. Written only when something changed: this runs on every
|
||||
// proxied request.
|
||||
func (s *Server) rememberKVMCookies(id string, cs []*http.Cookie) {
|
||||
have := s.kvmCookies(id)
|
||||
changed := false
|
||||
for _, c := range cs {
|
||||
if c.Name == "" || c.Name == cookieName || c.Name == kvmCookie {
|
||||
continue
|
||||
}
|
||||
gone := c.Value == "" || c.MaxAge < 0 || (!c.Expires.IsZero() && c.Expires.Before(time.Now()))
|
||||
if gone {
|
||||
if _, ok := have[c.Name]; ok {
|
||||
changed = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
if have[c.Name] != c.Value {
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
if !changed {
|
||||
return
|
||||
}
|
||||
_ = s.store.Update(func(st *store.State) {
|
||||
if st.KVMCookies == nil {
|
||||
st.KVMCookies = map[string]map[string]string{}
|
||||
}
|
||||
m := st.KVMCookies[id]
|
||||
if m == nil {
|
||||
m = map[string]string{}
|
||||
}
|
||||
for _, c := range cs {
|
||||
if c.Name == "" || c.Name == cookieName || c.Name == kvmCookie {
|
||||
continue
|
||||
}
|
||||
if c.Value == "" || c.MaxAge < 0 || (!c.Expires.IsZero() && c.Expires.Before(time.Now())) {
|
||||
delete(m, c.Name)
|
||||
} else {
|
||||
m[c.Name] = c.Value
|
||||
}
|
||||
}
|
||||
if len(m) == 0 {
|
||||
delete(st.KVMCookies, id)
|
||||
} else {
|
||||
st.KVMCookies[id] = m
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// forgetKVMCookies drops a device's remembered login — when the device is removed.
|
||||
func (s *Server) forgetKVMCookies(id string) {
|
||||
_ = s.store.Update(func(st *store.State) { delete(st.KVMCookies, id) })
|
||||
}
|
||||
|
||||
// lendKVMCookies hands the browser the device cookies the hub remembers and it lacks, as
|
||||
// the device's own page would have set them: readable by its scripts, for Ham Lab's whole
|
||||
// origin, kept for half a year. The device's SPA checks for its token client-side before it
|
||||
// ever asks the server, so lending it only on the wire would still land on the login form.
|
||||
func (s *Server) lendKVMCookies(w http.ResponseWriter, r *http.Request, id string) {
|
||||
for name, val := range s.kvmCookies(id) {
|
||||
if _, err := r.Cookie(name); err == nil {
|
||||
continue
|
||||
}
|
||||
http.SetCookie(w, &http.Cookie{Name: name, Value: val, Path: "/", MaxAge: 180 * 24 * 3600, Secure: secure(r), SameSite: http.SameSiteLaxMode})
|
||||
}
|
||||
}
|
||||
|
||||
// stripCookie removes named cookies from a request, so Ham Lab's session is never handed to
|
||||
// a device and the device never sees which KVM the browser is looking at.
|
||||
func stripCookie(r *http.Request, names ...string) {
|
||||
cs := r.Cookies()
|
||||
kept := cs[:0]
|
||||
for _, c := range cs {
|
||||
drop := false
|
||||
for _, n := range names {
|
||||
if c.Name == n {
|
||||
drop = true
|
||||
}
|
||||
}
|
||||
if !drop {
|
||||
kept = append(kept, c)
|
||||
}
|
||||
}
|
||||
r.Header.Del("Cookie")
|
||||
for _, c := range kept {
|
||||
r.AddCookie(c)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) forgetKVMProxy(id string) {
|
||||
s.kvms.mu.Lock()
|
||||
delete(s.kvms.m, id)
|
||||
s.kvms.mu.Unlock()
|
||||
}
|
||||
|
||||
// serveKVM is the entry point: /kvm/<id>/… . It also drops a cookie naming the device, so
|
||||
// that requests the page makes later by absolute path can be traced back to it.
|
||||
func (s *Server) serveKVM(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
k, ok := s.kvm(id)
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such KVM")
|
||||
return
|
||||
}
|
||||
p, err := s.kvmProxy(k)
|
||||
if err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
http.SetCookie(w, &http.Cookie{Name: kvmCookie, Value: id, Path: "/", HttpOnly: true, Secure: secure(r), SameSite: http.SameSiteLaxMode})
|
||||
s.lendKVMCookies(w, r, id)
|
||||
p.ServeHTTP(w, r)
|
||||
}
|
||||
|
||||
// kvmFor works out which device a request belongs to when it arrived by absolute path
|
||||
// rather than through the prefix: the Referer names the page that asked, and a cookie
|
||||
// covers what has no Referer at all. Nothing device-specific: it never has to know which
|
||||
// paths a given KVM uses.
|
||||
func (s *Server) kvmFor(r *http.Request) (config.KVM, bool) {
|
||||
if ref := r.Header.Get("Referer"); ref != "" {
|
||||
if u, err := url.Parse(ref); err == nil {
|
||||
if rest, ok := strings.CutPrefix(u.Path, kvmPrefix); ok {
|
||||
if id, _, _ := strings.Cut(rest, "/"); id != "" {
|
||||
if k, ok := s.kvm(id); ok {
|
||||
return k, true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// asked for by a page that is not a KVM: not ours to answer
|
||||
return config.KVM{}, false
|
||||
}
|
||||
if c, err := r.Cookie(kvmCookie); err == nil && c.Value != "" {
|
||||
return s.kvm(c.Value)
|
||||
}
|
||||
return config.KVM{}, false
|
||||
}
|
||||
|
||||
// kvmFallback answers a request Ham Lab does not serve itself, when it came from inside a
|
||||
// KVM's page. It returns false when the request has nothing to do with a KVM, and the
|
||||
// caller carries on with its own 404.
|
||||
func (s *Server) kvmFallback(w http.ResponseWriter, r *http.Request) bool {
|
||||
k, ok := s.kvmFor(r)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
p, err := s.kvmProxy(k)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
p.ServeHTTP(w, r)
|
||||
return true
|
||||
}
|
||||
|
||||
func machineLabel(m string) string {
|
||||
if m == "" {
|
||||
return "this host"
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ---- the list ----
|
||||
|
||||
func (s *Server) listKVMs(w http.ResponseWriter, _ *http.Request) {
|
||||
out := []config.KVM{}
|
||||
s.cfg.View(func(c *config.Config) { out = append(out, c.KVMs...) })
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name, URL, Machine string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
raw := strings.TrimSpace(body.URL)
|
||||
if raw != "" && !strings.Contains(raw, "://") {
|
||||
raw = "http://" + raw // what you would type: an address, not a scheme
|
||||
}
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
jsonErr(w, 400, "address: http(s)://host, or just the host")
|
||||
return
|
||||
}
|
||||
name := strings.TrimSpace(body.Name)
|
||||
if name == "" {
|
||||
name = u.Host
|
||||
}
|
||||
if len(name) > maxNameLen {
|
||||
jsonErr(w, 400, "that name is too long")
|
||||
return
|
||||
}
|
||||
b := make([]byte, 6)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
machine := strings.TrimSpace(body.Machine)
|
||||
k := config.KVM{ID: hex.EncodeToString(b), Name: name, URL: u.Scheme + "://" + u.Host + strings.TrimSuffix(u.Path, "/"), Machine: machine, Added: time.Now()}
|
||||
// One per machine: a second one for the same host replaces the first rather than sitting
|
||||
// beside it, since the tab shows the machine you are looking at.
|
||||
var replaced string
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.KVMs[:0]
|
||||
for _, old := range c.KVMs {
|
||||
if old.Machine == machine {
|
||||
replaced = old.ID
|
||||
continue
|
||||
}
|
||||
kept = append(kept, old)
|
||||
}
|
||||
c.KVMs = append(kept, k)
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if replaced != "" {
|
||||
s.forgetKVMProxy(replaced)
|
||||
}
|
||||
log.Printf("kvm %q added for %s (%s) by %s", k.Name, machineLabel(machine), k.URL, clientIP(r))
|
||||
jsonOK(w, 201, k)
|
||||
}
|
||||
|
||||
func (s *Server) removeKVM(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
found := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.KVMs[:0]
|
||||
for _, k := range c.KVMs {
|
||||
if k.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, k)
|
||||
}
|
||||
c.KVMs = kept
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such KVM")
|
||||
return
|
||||
}
|
||||
s.forgetKVMProxy(id)
|
||||
s.forgetKVMCookies(id)
|
||||
log.Printf("kvm %s removed by %s", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// A KVM's own login is a cookie its page sets in one browser. The hub remembers it from the
|
||||
// requests that carry it, lends it to a browser that lacks it — both on the wire to the
|
||||
// device and back to the browser, since the device's page checks for it client-side — and
|
||||
// forgets it when the device says it is gone or the KVM is removed.
|
||||
func TestKVMCookiesAreRememberedAndLent(t *testing.T) {
|
||||
var seen []string // the Cookie header the fake device received, per request
|
||||
device := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
seen = append(seen, r.Header.Get("Cookie"))
|
||||
if r.URL.Path == "/api/auth/login" {
|
||||
// a device that sets its token by header, with its own domain and Secure
|
||||
http.SetCookie(w, &http.Cookie{Name: "nano-kvm-token", Value: "tok-1", Domain: "10.0.0.217", Secure: true})
|
||||
}
|
||||
if r.URL.Path == "/api/auth/logout" {
|
||||
http.SetCookie(w, &http.Cookie{Name: "nano-kvm-token", Value: "", MaxAge: -1})
|
||||
}
|
||||
w.WriteHeader(200)
|
||||
}))
|
||||
defer device.Close()
|
||||
|
||||
dir := t.TempDir()
|
||||
cfg := config.Default(filepath.Join(dir, "config.json"))
|
||||
if err := cfg.Update(func(c *config.Config) {
|
||||
c.KVMs = []config.KVM{{ID: "abc123", Name: "test", URL: device.URL, Machine: ""}}
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err := store.Open(filepath.Join(dir, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{cfg: cfg, store: st, sessions: auth.NewSessions(st)}
|
||||
k, _ := s.kvm("abc123")
|
||||
p, err := s.kvmProxy(k)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ask := func(path string, cookies ...*http.Cookie) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("GET", path, nil)
|
||||
r.SetPathValue("id", "abc123")
|
||||
for _, c := range cookies {
|
||||
r.AddCookie(c)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
if strings.HasPrefix(path, kvmPrefix) {
|
||||
s.serveKVM(w, r)
|
||||
} else {
|
||||
p.ServeHTTP(w, r)
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// 1. the device sets its token by header: it reaches the browser without the device's
|
||||
// domain or Secure (plain-http console), and the hub remembers it
|
||||
w := ask("/api/auth/login", &http.Cookie{Name: cookieName, Value: "hamlab-session"})
|
||||
var set *http.Cookie
|
||||
for _, c := range w.Result().Cookies() {
|
||||
if c.Name == "nano-kvm-token" {
|
||||
set = c
|
||||
}
|
||||
}
|
||||
if set == nil || set.Domain != "" || set.Secure || set.Value != "tok-1" || set.Path != "/" {
|
||||
t.Fatalf("rewritten Set-Cookie = %+v, want nano-kvm-token=tok-1 for / without Domain or Secure", set)
|
||||
}
|
||||
if got := s.kvmCookies("abc123"); got["nano-kvm-token"] != "tok-1" {
|
||||
t.Fatalf("remembered = %v, want the token", got)
|
||||
}
|
||||
if strings.Contains(seen[0], "hamlab_session") {
|
||||
t.Fatalf("the device was handed Ham Lab's session: %q", seen[0])
|
||||
}
|
||||
|
||||
// 2. a browser that has the cookie teaches a newer value
|
||||
ask("/api/vm/info", &http.Cookie{Name: "nano-kvm-token", Value: "tok-2"}, &http.Cookie{Name: cookieName, Value: "x"})
|
||||
if got := s.kvmCookies("abc123"); got["nano-kvm-token"] != "tok-2" {
|
||||
t.Fatalf("remembered = %v, want tok-2", got)
|
||||
}
|
||||
if seen[1] != "nano-kvm-token=tok-2" {
|
||||
t.Fatalf("device saw %q, want the browser's own cookie only", seen[1])
|
||||
}
|
||||
|
||||
// 3. a browser without it is lent it: on the wire, and back as a Set-Cookie it can read
|
||||
w = ask(kvmPrefix+"abc123/", &http.Cookie{Name: cookieName, Value: "x"})
|
||||
if seen[2] != "nano-kvm-token=tok-2" {
|
||||
t.Fatalf("device saw %q from a browser with no cookie, want the remembered token", seen[2])
|
||||
}
|
||||
lent := false
|
||||
for _, c := range w.Result().Cookies() {
|
||||
if c.Name == "nano-kvm-token" && c.Value == "tok-2" && !c.HttpOnly && c.Path == "/" && c.MaxAge > 0 {
|
||||
lent = true
|
||||
}
|
||||
}
|
||||
if !lent {
|
||||
t.Fatalf("browser was not lent the token: %v", w.Result().Cookies())
|
||||
}
|
||||
|
||||
// 4. the device logging out forgets it
|
||||
ask("/api/auth/logout", &http.Cookie{Name: "nano-kvm-token", Value: "tok-2"})
|
||||
if got := s.kvmCookies("abc123"); got != nil {
|
||||
t.Fatalf("after logout remembered = %v, want nothing", got)
|
||||
}
|
||||
|
||||
// 5. removing the KVM clears whatever is left
|
||||
s.rememberKVMCookies("abc123", []*http.Cookie{{Name: "nano-kvm-token", Value: "tok-3"}})
|
||||
s.forgetKVMCookies("abc123")
|
||||
if got := s.kvmCookies("abc123"); got != nil {
|
||||
t.Fatalf("after remove remembered = %v, want nothing", got)
|
||||
}
|
||||
_ = url.Parse
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package api
|
||||
|
||||
// The Network tab: the machine's interfaces, how the internet looks from here, and the speed
|
||||
// test. One machine at a time, like every other tab — the test runs on the machine in the
|
||||
// switcher, so the number is that machine's line and not the hub's.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/network"
|
||||
"hamlab/internal/sensors"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// networkOverview is everything the tab draws at once: the interfaces, the outside view, the
|
||||
// last test and the ones before it.
|
||||
func (s *Server) networkOverview(w http.ResponseWriter, _ *http.Request) {
|
||||
run := s.net.Test.State()
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"report": network.Look(s.hostName()),
|
||||
"outside": s.net.Outside(),
|
||||
"run": run,
|
||||
"history": s.net.Test.History(),
|
||||
"method": network.Method,
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) speedtestState(w http.ResponseWriter, _ *http.Request) {
|
||||
jsonOK(w, 200, s.net.Test.State())
|
||||
}
|
||||
|
||||
func (s *Server) speedtestStart(w http.ResponseWriter, r *http.Request) {
|
||||
run, err := s.net.Test.Start()
|
||||
if err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("speed test started on %s (by %s)", s.hostName(), clientIP(r))
|
||||
jsonOK(w, 200, run)
|
||||
}
|
||||
|
||||
func (s *Server) speedtestCancel(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.net.Test.Cancel() {
|
||||
jsonErr(w, 404, "no speed test is running")
|
||||
return
|
||||
}
|
||||
log.Printf("speed test cancelled on %s (by %s)", s.hostName(), clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// ---- keeping the results ----
|
||||
|
||||
// savedNetwork is what survives a restart: the recent results, so the history is a history
|
||||
// rather than whatever has happened since the daemon last started.
|
||||
type savedNetwork struct {
|
||||
Results []network.Result `json:"results,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Server) loadNetwork() []network.Result {
|
||||
var saved savedNetwork
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.Network })
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &saved); err != nil {
|
||||
log.Printf("network: kept results unreadable (%v); starting a fresh history", err)
|
||||
}
|
||||
}
|
||||
return saved.Results
|
||||
}
|
||||
|
||||
// keepResult is called when a test finishes: append it to what is on disk.
|
||||
func (s *Server) keepResult(r network.Result) {
|
||||
if r.Error != "" {
|
||||
log.Printf("speed test on %s failed: %s", s.hostName(), r.Error)
|
||||
} else {
|
||||
log.Printf("speed test on %s: down %s, up %s, latency %.1f ms via %s",
|
||||
s.hostName(), network.Mbits(r.DownMbps), network.Mbits(r.UpMbps), r.LatencyMs, r.Colo)
|
||||
}
|
||||
results := s.net.Test.History()
|
||||
b, err := json.Marshal(savedNetwork{Results: results})
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := s.store.Update(func(st *store.State) { st.Network = b }); err != nil {
|
||||
log.Printf("network: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- the speed test as sensors ----
|
||||
|
||||
// netSensors publishes the last speed test as ordinary sensors, so the line can be a widget,
|
||||
// a graph, an alert or a Prometheus series like anything else Ham Lab measures. Each one's
|
||||
// timestamp is when the test ran, which is how a stale reading says so.
|
||||
type netSensors struct{ w *network.Watcher }
|
||||
|
||||
func (n netSensors) Sensors() []sensors.Sensor {
|
||||
var out []sensors.Sensor
|
||||
if o := n.w.Outside(); o.IP != "" {
|
||||
at := o.At.UnixMilli()
|
||||
out = append(out, sensors.Sensor{ID: "net.internet.ip", Name: "Public address", Kind: "state", Value: o.IP, T: at})
|
||||
if o.Colo != "" {
|
||||
out = append(out, sensors.Sensor{ID: "net.internet.edge", Name: "Internet edge", Kind: "state", Value: o.Colo, T: at})
|
||||
}
|
||||
}
|
||||
run := n.w.Test.State()
|
||||
if run.Result == nil || run.Result.Error != "" {
|
||||
return out
|
||||
}
|
||||
r := run.Result
|
||||
at := r.At.UnixMilli()
|
||||
return append(out,
|
||||
sensors.Sensor{ID: "net.internet.download", Name: "Download", Kind: "rate", Unit: "Mbps", Value: r.DownMbps, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.upload", Name: "Upload", Kind: "rate", Unit: "Mbps", Value: r.UpMbps, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.latency", Name: "Latency", Kind: "duration", Unit: "ms", Value: r.LatencyMs, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.jitter", Name: "Jitter", Kind: "duration", Unit: "ms", Value: r.JitterMs, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.tested", Name: "Speed test age", Kind: "duration", Unit: "s", Value: round1(time.Since(r.At).Seconds()), T: at, Min: sensors.F(0)},
|
||||
)
|
||||
}
|
||||
|
||||
func round1(v float64) float64 { return float64(int64(v*10+0.5)) / 10 }
|
||||
@@ -0,0 +1,169 @@
|
||||
package api
|
||||
|
||||
// Playing a video the browser cannot play as it is: POST /api/files/play starts an HLS
|
||||
// re-encode on this machine (internal/media), GET …/play/{id}/index.m3u8 and the segments
|
||||
// feed the player, GET …/play/{id} is its progress, DELETE ends it. /api/files/play-caps
|
||||
// says which codecs this machine can make and on what. Root mode (?sudo=1) reads the file
|
||||
// and runs ffmpeg as root, through the same sudo the file reads use.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/files"
|
||||
"hamlab/internal/media"
|
||||
)
|
||||
|
||||
const playLimit = 3 // re-encodes at once on one machine; a GPU does two or three, a CPU one
|
||||
|
||||
type playCaps struct {
|
||||
Codecs []playCodec `json:"codecs"`
|
||||
Qualities []media.Quality `json:"qualities"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type playCodec struct {
|
||||
media.Codec
|
||||
Engine string `json:"engine,omitempty"` // "" when nothing here encodes it
|
||||
}
|
||||
|
||||
// playCapsHandler: GET /api/files/play-caps[?sudo=1]
|
||||
func (s *Server) playCapsHandler(w http.ResponseWriter, r *http.Request) {
|
||||
root := r.URL.Query().Get("sudo") == "1"
|
||||
if root && s.fsSudo == nil {
|
||||
jsonErr(w, 403, "root access needs passwordless sudo on the host")
|
||||
return
|
||||
}
|
||||
caps, err := s.thumbs.capsFor(r.Context(), root, false)
|
||||
pc := playCaps{Codecs: []playCodec{}, Qualities: media.Qualities}
|
||||
if err != nil {
|
||||
pc.Error = err.Error()
|
||||
} else if caps.Error != "" {
|
||||
pc.Error = caps.Error
|
||||
}
|
||||
for _, c := range media.Codecs {
|
||||
x := playCodec{Codec: c}
|
||||
if pc.Error == "" {
|
||||
x.Engine = media.EngineFor(caps, c.ID)
|
||||
}
|
||||
pc.Codecs = append(pc.Codecs, x)
|
||||
}
|
||||
jsonOK(w, 200, pc)
|
||||
}
|
||||
|
||||
// playStart: POST /api/files/play[?sudo=1] {path, codec, quality}
|
||||
func (s *Server) playStart(w http.ResponseWriter, r *http.Request) {
|
||||
fsys, ok := s.fsFor(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
root := r.URL.Query().Get("sudo") == "1"
|
||||
var body struct{ Path, Codec, Quality string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
p, err := files.Clean(body.Path)
|
||||
if err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
if media.Kind(p) != "video" {
|
||||
jsonErr(w, 415, "not a video")
|
||||
return
|
||||
}
|
||||
if err := media.ValidChoice(body.Codec, body.Quality); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
if _, err := fsys.Stat(r.Context(), p); err != nil {
|
||||
fsErr(w, err)
|
||||
return
|
||||
}
|
||||
caps, err := s.thumbs.capsFor(r.Context(), root, false)
|
||||
if err != nil {
|
||||
jsonErr(w, 503, err.Error())
|
||||
return
|
||||
}
|
||||
if caps.Error != "" {
|
||||
jsonErr(w, 503, caps.Error)
|
||||
return
|
||||
}
|
||||
ictx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
|
||||
src, err := media.Inspect(ictx, caps, p, root)
|
||||
cancel()
|
||||
if err != nil {
|
||||
playErr(w, err)
|
||||
return
|
||||
}
|
||||
st, err := s.plays.Open(r.Context(), caps, root, p, body.Codec, body.Quality, src)
|
||||
if err != nil {
|
||||
playErr(w, err)
|
||||
return
|
||||
}
|
||||
log.Printf("files: play %s as %s %s on %s (from %s)", p, st.Codec, st.Quality, st.Engine, clientIP(r))
|
||||
jsonOK(w, 200, map[string]any{"stream": st, "url": "/api/files/play/" + st.ID + "/index.m3u8"})
|
||||
}
|
||||
|
||||
func playErr(w http.ResponseWriter, err error) {
|
||||
var de *media.DecodeError
|
||||
switch {
|
||||
case errors.As(err, &de):
|
||||
jsonErr(w, 422, de.Msg)
|
||||
case strings.Contains(err.Error(), "already being re-encoded"):
|
||||
jsonErr(w, 429, err.Error())
|
||||
case strings.HasPrefix(err.Error(), "unknown ") || strings.HasPrefix(err.Error(), "nothing on this machine encodes"):
|
||||
jsonErr(w, 400, err.Error())
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
jsonErr(w, 504, "the file took too long to read")
|
||||
default:
|
||||
jsonErr(w, 500, err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
// playStatus: GET /api/files/play/{id}
|
||||
func (s *Server) playStatus(w http.ResponseWriter, r *http.Request) {
|
||||
st, ok := s.plays.Find(r.PathValue("id"))
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such stream")
|
||||
return
|
||||
}
|
||||
jsonOK(w, 200, st.Status())
|
||||
}
|
||||
|
||||
// playStop: DELETE /api/files/play/{id}, or POST …/close — the form a page can send on its
|
||||
// way out (navigator.sendBeacon), so a phone closing the tab releases the machine at once.
|
||||
func (s *Server) playStop(w http.ResponseWriter, r *http.Request) {
|
||||
s.plays.Close(r.PathValue("id"))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// playFile: GET /api/files/play/{id}/{file} — the playlist, the init segment, a segment.
|
||||
func (s *Server) playFile(w http.ResponseWriter, r *http.Request) {
|
||||
st, ok := s.plays.Find(r.PathValue("id"))
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such stream")
|
||||
return
|
||||
}
|
||||
p, ok := st.File(r.PathValue("file"))
|
||||
if !ok {
|
||||
jsonErr(w, 404, "not a stream file")
|
||||
return
|
||||
}
|
||||
switch filepath.Ext(p) {
|
||||
case ".m3u8":
|
||||
w.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
|
||||
default:
|
||||
w.Header().Set("Content-Type", "video/mp4")
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
http.ServeFile(w, r, p)
|
||||
}
|
||||
+2368
-41
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,224 @@
|
||||
package api
|
||||
|
||||
// The Space tab: what is using a disk or a pool. The unit is a whole volume, never a
|
||||
// partition: a disk on its own is scanned through every filesystem it backs, and a ZFS pool
|
||||
// through every mounted dataset, as one thing — its member disks are not scanned apart.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"net/http"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/space"
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
// spaceTarget is one thing a scan can measure: a disk on its own, or a ZFS pool as a whole.
|
||||
// A disk that is a member of a pool is not a target of its own — its share of the pool means
|
||||
// nothing by itself — it is listed under the pool. (A disk with a pool partition *and* a
|
||||
// filesystem of its own, root on one partition and a pool on another, is both.)
|
||||
type spaceTarget struct {
|
||||
ID string `json:"id"` // "nvme0" or "pool:tank"
|
||||
Kind string `json:"kind"` // disk | pool
|
||||
Name string `json:"name"`
|
||||
Model string `json:"model,omitempty"` // disk: its model
|
||||
Layout string `json:"layout,omitempty"` // pool: raidz2, mirror, …
|
||||
Health string `json:"health,omitempty"` // pool: ONLINE, DEGRADED, …
|
||||
Size uint64 `json:"size"`
|
||||
Used uint64 `json:"used,omitempty"` // pool: allocated
|
||||
Disks []string `json:"disks,omitempty"` // pool: its member disks
|
||||
Roots []string `json:"roots"` // the mountpoints a scan walks
|
||||
Scan *space.Scan `json:"scan,omitempty"` // the last scan, if any
|
||||
|
||||
datasets map[string]string // root → ZFS dataset, for the ones that are
|
||||
}
|
||||
|
||||
var layoutRe = regexp.MustCompile(`\b(draid\d*|raidz[123]?|mirror)\b`)
|
||||
|
||||
// spaceTargets lists the disks and pools a scan can measure.
|
||||
func (s *Server) spaceTargets(ctx context.Context) []spaceTarget {
|
||||
devs, _ := storage.Devices(ctx)
|
||||
var out []spaceTarget
|
||||
pools := map[string]*spaceTarget{}
|
||||
var poolOrder []string
|
||||
for _, d := range devs {
|
||||
disk := spaceTarget{ID: d.Name, Kind: "disk", Name: d.Name, Model: d.Model, Size: d.Size, Roots: []string{}}
|
||||
seen := map[string]bool{}
|
||||
inPool := false
|
||||
for _, p := range d.Parts {
|
||||
if p.FSType == "zfs_member" && p.Label != "" {
|
||||
inPool = true
|
||||
pt := pools[p.Label]
|
||||
if pt == nil {
|
||||
pt = &spaceTarget{ID: "pool:" + p.Label, Kind: "pool", Name: p.Label, Roots: []string{}, datasets: map[string]string{}}
|
||||
for _, zm := range zfsMounts(ctx, p.Label) {
|
||||
pt.Roots = append(pt.Roots, zm.mount)
|
||||
pt.datasets[zm.mount] = zm.dataset
|
||||
}
|
||||
pools[p.Label] = pt
|
||||
poolOrder = append(poolOrder, p.Label)
|
||||
}
|
||||
pt.Disks = append(pt.Disks, d.Name)
|
||||
continue
|
||||
}
|
||||
if p.Mount != "" && p.Mount != "[SWAP]" && !seen[p.Mount] {
|
||||
seen[p.Mount] = true
|
||||
disk.Roots = append(disk.Roots, p.Mount)
|
||||
}
|
||||
}
|
||||
if !inPool || len(disk.Roots) > 0 {
|
||||
sort.Strings(disk.Roots)
|
||||
out = append(out, disk)
|
||||
}
|
||||
}
|
||||
if len(pools) > 0 {
|
||||
// size, health and layout from zpool; the vdev tree names the layout
|
||||
for _, p := range s.storage.ZFS(ctx).Pools {
|
||||
if pt := pools[p.Name]; pt != nil {
|
||||
pt.Size, pt.Used, pt.Health = p.Size, p.Alloc, p.Health
|
||||
if m := layoutRe.FindString(p.Config); m != "" {
|
||||
pt.Layout = m
|
||||
} else if len(pt.Disks) == 1 {
|
||||
pt.Layout = "single disk"
|
||||
} else if len(pt.Disks) > 1 {
|
||||
pt.Layout = "stripe"
|
||||
}
|
||||
}
|
||||
}
|
||||
var ps []spaceTarget
|
||||
for _, name := range poolOrder {
|
||||
sort.Strings(pools[name].Roots)
|
||||
ps = append(ps, *pools[name])
|
||||
}
|
||||
out = append(ps, out...) // pools first: they are what a NAS is
|
||||
}
|
||||
if out == nil {
|
||||
out = []spaceTarget{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type zfsMount struct{ mount, dataset string }
|
||||
|
||||
// zfsMounts is every mounted filesystem of a pool, with its dataset.
|
||||
func zfsMounts(ctx context.Context, pool string) []zfsMount {
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-t", "filesystem", "-r", "-o", "name,mountpoint,mounted", pool).Output()
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var mounts []zfsMount
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) == 3 && f[2] == "yes" && strings.HasPrefix(f[1], "/") {
|
||||
mounts = append(mounts, zfsMount{mount: f[1], dataset: f[0]})
|
||||
}
|
||||
}
|
||||
return mounts
|
||||
}
|
||||
|
||||
func (s *Server) spaceOverview(w http.ResponseWriter, r *http.Request) {
|
||||
targets := s.spaceTargets(r.Context())
|
||||
scans := s.space.All()
|
||||
for i := range targets {
|
||||
targets[i].Scan = scans[targets[i].ID]
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"targets": targets})
|
||||
}
|
||||
|
||||
func (s *Server) spaceTarget(w http.ResponseWriter, r *http.Request, id string) (spaceTarget, bool) {
|
||||
for _, t := range s.spaceTargets(r.Context()) {
|
||||
if t.ID == id {
|
||||
return t, true
|
||||
}
|
||||
}
|
||||
jsonErr(w, 404, "no such disk or pool")
|
||||
return spaceTarget{}, false
|
||||
}
|
||||
|
||||
func (s *Server) spaceStatus(w http.ResponseWriter, r *http.Request) {
|
||||
st := s.space.Status(r.PathValue("disk"))
|
||||
if st == nil {
|
||||
jsonErr(w, 404, "not scanned")
|
||||
return
|
||||
}
|
||||
jsonOK(w, 200, st)
|
||||
}
|
||||
|
||||
func (s *Server) spaceScan(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.spaceTarget(w, r, r.PathValue("disk"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
st, err := s.space.Start(t.ID, t.Roots, t.datasets)
|
||||
if err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("space: scan of %s (%s) started by %s", t.ID, strings.Join(t.Roots, " "), clientIP(r))
|
||||
jsonOK(w, 202, st)
|
||||
}
|
||||
|
||||
func (s *Server) spaceForget(w http.ResponseWriter, r *http.Request) {
|
||||
s.space.Forget(r.PathValue("disk"))
|
||||
log.Printf("space: scan of %s forgotten by %s", r.PathValue("disk"), clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) spaceCancel(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.space.Cancel(r.PathValue("disk")) {
|
||||
jsonErr(w, 404, "no scan running")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) spaceTree(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query()
|
||||
path := q.Get("path")
|
||||
if path == "" {
|
||||
path = "/"
|
||||
}
|
||||
depth, _ := strconv.Atoi(q.Get("depth"))
|
||||
if depth < 1 || depth > 32 {
|
||||
depth = 1
|
||||
}
|
||||
limit, _ := strconv.Atoi(q.Get("limit"))
|
||||
if limit < 1 || limit > 2000 {
|
||||
limit = 300
|
||||
}
|
||||
min, _ := strconv.ParseInt(q.Get("min"), 10, 64)
|
||||
if min < 0 {
|
||||
min = 0
|
||||
}
|
||||
t, err := s.space.Tree(r.PathValue("disk"), path, depth, limit, min)
|
||||
if err != nil {
|
||||
jsonErr(w, 404, err.Error())
|
||||
return
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
jsonOK(w, 200, t)
|
||||
}
|
||||
|
||||
func (s *Server) spaceLargest(w http.ResponseWriter, r *http.Request) {
|
||||
n, _ := strconv.Atoi(r.URL.Query().Get("n"))
|
||||
if n < 1 || n > 5000 {
|
||||
n = 500
|
||||
}
|
||||
files, err := s.space.Largest(r.PathValue("disk"), n)
|
||||
if err != nil {
|
||||
jsonErr(w, 404, err.Error())
|
||||
return
|
||||
}
|
||||
if files == nil {
|
||||
files = []space.File{}
|
||||
}
|
||||
jsonOK(w, 200, files)
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,156 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"hamlab/internal/compose"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// A stack deployed by an orchestrator running in a container records the path its compose
|
||||
// file had inside that container. Resolving it has to work both while the orchestrator is
|
||||
// running — compose is then run inside it — and after it has been removed, where the same
|
||||
// mount is only remembered and compose has to run on the host instead.
|
||||
func TestResolveThroughMounts(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "orch", "stacks", "app")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
file := filepath.Join(dir, "compose.yaml")
|
||||
if err := os.WriteFile(file, []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inner := "/app/data/stacks/app/compose.yaml"
|
||||
|
||||
for _, tc := range []struct {
|
||||
name, container, wantVia string
|
||||
}{
|
||||
{"orchestrator running", "orch", "orch"},
|
||||
{"orchestrator removed", "", ""},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
st := stack{Name: "app", File: inner, Files: []string{inner}}
|
||||
st.resolve([]mountMap{{tc.container, "/app/data", filepath.Join(root, "orch")}})
|
||||
if !st.Editable {
|
||||
t.Fatalf("not editable: %s", st.Note)
|
||||
}
|
||||
if st.HostFile != file {
|
||||
t.Errorf("host file = %q, want %q", st.HostFile, file)
|
||||
}
|
||||
if st.Via != tc.wantVia {
|
||||
t.Errorf("via = %q, want %q", st.Via, tc.wantVia)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Nothing explains the path: the stack is listed, but read-only and says why.
|
||||
st := stack{Name: "app", File: inner, Files: []string{inner}}
|
||||
st.resolve(nil)
|
||||
if st.Editable || st.Note == "" {
|
||||
t.Errorf("unresolvable stack should be read-only with a note, got editable=%v note=%q", st.Editable, st.Note)
|
||||
}
|
||||
}
|
||||
|
||||
// Every file of a multi-file stack must resolve through the same place. Redeploying from
|
||||
// the first file alone would silently drop whatever the override set.
|
||||
func TestResolveRefusesSplitStack(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "orch", "stacks", "app")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "compose.yaml"), []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st := stack{Name: "app", File: "/app/data/stacks/app/compose.yaml", Files: []string{
|
||||
"/app/data/stacks/app/compose.yaml",
|
||||
"/app/data/stacks/app/override.yaml", // never written: not on disk
|
||||
}}
|
||||
st.resolve([]mountMap{{"orch", "/app/data", filepath.Join(root, "orch")}})
|
||||
if st.Editable {
|
||||
t.Error("a stack with a file that cannot be found should not be editable")
|
||||
}
|
||||
}
|
||||
|
||||
// A schedule made while an orchestrator was running says "run compose inside that
|
||||
// container". When the container is gone the schedule must still work, against the same
|
||||
// files on the host — otherwise retiring the orchestrator silently breaks every scheduled
|
||||
// update made before it went.
|
||||
func TestRepointSurvivesAMissingContainer(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "orch", "stacks", "app")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
host := filepath.Join(dir, "compose.yaml")
|
||||
if err := os.WriteFile(host, []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := store.Open(filepath.Join(root, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.Update(func(st *store.State) {
|
||||
st.Mounts = []store.RememberedMount{{Container: "gone-orchestrator", Dest: "/app/data", Src: filepath.Join(root, "orch")}}
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
in := compose.Spec{Project: "app", Files: []string{"/app/data/stacks/app/compose.yaml"}, Via: "gone-orchestrator", Tool: "docker"}
|
||||
got := repoint(context.Background(), db, in)
|
||||
if got.Via != "" {
|
||||
t.Errorf("via = %q, want it dropped", got.Via)
|
||||
}
|
||||
if len(got.Files) != 1 || got.Files[0] != host {
|
||||
t.Errorf("files = %v, want [%s]", got.Files, host)
|
||||
}
|
||||
|
||||
// Nothing remembered: the schedule is left exactly as it was rather than pointed at a
|
||||
// guess. It will fail loudly, which is better than deploying the wrong file.
|
||||
empty, err := store.Open(filepath.Join(root, "empty.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := repoint(context.Background(), empty, in); got.Via != in.Via || len(got.Files) != 1 || got.Files[0] != in.Files[0] {
|
||||
t.Errorf("unresolvable spec was changed: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Where new stacks go: Ham Lab's own <data dir>/stacks is always offered, first and as the
|
||||
// default, whether or not anything is in it yet. A folder this machine already keeps stacks
|
||||
// in — two or more deployed from it — is offered too, but never ahead of Ham Lab's own.
|
||||
func TestHomesDefaultToDataDir(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
cfg := config.Default(filepath.Join(root, "data", "config.json"))
|
||||
s := &Server{cfg: cfg}
|
||||
own := filepath.Join(root, "data", "stacks")
|
||||
legacy := filepath.Join(root, "home", "stacks", "box")
|
||||
deployed := func(dir, name string) stack {
|
||||
f := filepath.Join(dir, name, "compose.yaml")
|
||||
return stack{Name: name, Containers: []string{name + "-1"}, Files: []string{f}, File: f, HostFiles: []string{f}, HostFile: f, Editable: true}
|
||||
}
|
||||
|
||||
// Nothing deployed anywhere: one home, Ham Lab's own.
|
||||
homes := s.homesOf(nil)
|
||||
if len(homes) != 1 || homes[0].Dir != own || !homes[0].Default {
|
||||
t.Fatalf("empty machine: homes = %+v, want just %s as default", homes, own)
|
||||
}
|
||||
|
||||
// Two stacks deployed from a pre-existing folder: it is offered, after Ham Lab's own.
|
||||
homes = s.homesOf([]stack{deployed(legacy, "a"), deployed(legacy, "b")})
|
||||
if len(homes) != 2 || homes[0].Dir != own || !homes[0].Default || homes[1].Dir != legacy || homes[1].Default || homes[1].Stacks != 2 {
|
||||
t.Fatalf("legacy folder: homes = %+v, want [%s (default), %s]", homes, own, legacy)
|
||||
}
|
||||
|
||||
// One lone stack elsewhere says nothing about where the next should go; one in Ham Lab's
|
||||
// own folder is already home.
|
||||
homes = s.homesOf([]stack{deployed(filepath.Join(root, "home", "apps"), "orchestrator"), deployed(own, "c")})
|
||||
if len(homes) != 1 || homes[0].Dir != own || homes[0].Stacks != 1 || !homes[0].Default {
|
||||
t.Fatalf("lone stacks: homes = %+v, want just %s", homes, own)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,375 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/sensors"
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
// statusSensors is the sensors.Provider for what isn't a /proc number: failed services,
|
||||
// pending updates, pool health, drive health. Each source costs an exec or several, so
|
||||
// they are read in the background on their own clocks and served from memory; every
|
||||
// sensor's t says how old its reading really is.
|
||||
type statusSensors struct {
|
||||
s *Server
|
||||
mu sync.RWMutex
|
||||
by map[string][]sensors.Sensor // source → its sensors
|
||||
|
||||
cpuPrev map[string]cpuMark // container ID → CPU time at the previous read
|
||||
|
||||
hist map[string]*sensors.Series // numeric sensors only, statusHistory points each
|
||||
acc map[string]*bucket // the point being built for each, when the window is wider than the read rate
|
||||
}
|
||||
|
||||
// statusHistory is how many readings each status sensor keeps: at 15 s that is over an
|
||||
// hour of container memory; at 10 min, two days of drive wear.
|
||||
const statusHistory = 300
|
||||
|
||||
// A point being averaged, the same idea as the sampler's: these sources are read every 15 to
|
||||
// 300 seconds, so a wide window keeps one point per stride rather than one per read.
|
||||
type bucket struct {
|
||||
last int64 // when this sensor's last point was kept
|
||||
sum float64
|
||||
n float64
|
||||
}
|
||||
|
||||
const (
|
||||
servicesEvery = 30 * time.Second
|
||||
containersEvery = 15 * time.Second
|
||||
updatesEvery = 5 * time.Minute
|
||||
zfsEvery = time.Minute
|
||||
smartEvery = 10 * time.Minute // storage.Manager caches SMART that long anyway, and never wakes a sleeping disk
|
||||
)
|
||||
|
||||
func newStatusSensors(s *Server) *statusSensors {
|
||||
p := &statusSensors{s: s, by: map[string][]sensors.Sensor{}, hist: map[string]*sensors.Series{}, acc: map[string]*bucket{}}
|
||||
go p.loop("services", servicesEvery, p.services)
|
||||
go p.loop("containers", containersEvery, p.containers)
|
||||
go p.loop("updates", updatesEvery, p.updates)
|
||||
go p.loop("zfs", zfsEvery, p.zfs)
|
||||
go p.loop("smart", smartEvery, p.smart)
|
||||
return p
|
||||
}
|
||||
|
||||
func (p *statusSensors) loop(name string, every time.Duration, read func(context.Context, int64) []sensors.Sensor) {
|
||||
for {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
got := read(ctx, time.Now().UnixMilli())
|
||||
cancel()
|
||||
p.mu.Lock()
|
||||
p.by[name] = got
|
||||
window := p.s.metrics.Window()
|
||||
stride := window.Milliseconds() / statusHistory
|
||||
live := map[string]bool{}
|
||||
for _, x := range got {
|
||||
v, ok := x.Value.(float64)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
live[x.ID] = true
|
||||
h := p.hist[x.ID]
|
||||
if h == nil {
|
||||
h = &sensors.Series{ID: x.ID, Unit: x.Unit}
|
||||
p.hist[x.ID] = h
|
||||
}
|
||||
b := p.acc[x.ID]
|
||||
if b == nil {
|
||||
b = &bucket{}
|
||||
p.acc[x.ID] = b
|
||||
}
|
||||
b.sum, b.n = b.sum+v, b.n+1
|
||||
// due a stride after the last point kept, with half a read of slack for a late timer
|
||||
if b.last != 0 && x.T-b.last < stride-every.Milliseconds()/2 {
|
||||
continue // still filling this point
|
||||
}
|
||||
h.T, h.V = append(h.T, x.T), append(h.V, b.sum/b.n)
|
||||
*b = bucket{last: x.T}
|
||||
if len(h.T) > statusHistory {
|
||||
h.T, h.V = h.T[len(h.T)-statusHistory:], h.V[len(h.V)-statusHistory:]
|
||||
}
|
||||
if cut := x.T - window.Milliseconds(); h.T[0] < cut { // and nothing older than the window
|
||||
i := 0
|
||||
for i < len(h.T) && h.T[i] < cut {
|
||||
i++
|
||||
}
|
||||
h.T, h.V = h.T[i:], h.V[i:]
|
||||
}
|
||||
}
|
||||
for id := range p.hist { // a container that is gone takes its history with it
|
||||
if strings.HasPrefix(id, prefixOf[name]) && !live[id] && prefixOf[name] != "" {
|
||||
delete(p.hist, id)
|
||||
delete(p.acc, id)
|
||||
}
|
||||
}
|
||||
p.mu.Unlock()
|
||||
if len(got) == 0 { // no pools, no Docker, no SMART: look again, but rarely
|
||||
time.Sleep(10 * every)
|
||||
continue
|
||||
}
|
||||
time.Sleep(every)
|
||||
}
|
||||
}
|
||||
|
||||
// prefixOf says which IDs a source owns, so its loop forgets only its own departed sensors.
|
||||
var prefixOf = map[string]string{"containers": "container.", "smart": "smart.", "zfs": "zfs."}
|
||||
|
||||
// SensorHistories implements sensors.Historian.
|
||||
func (p *statusSensors) SensorHistories(match func(id string) bool) []sensors.Series {
|
||||
p.mu.RLock()
|
||||
defer p.mu.RUnlock()
|
||||
var out []sensors.Series
|
||||
for id, h := range p.hist {
|
||||
if match(id) {
|
||||
out = append(out, sensors.Series{ID: h.ID, Unit: h.Unit, T: append([]int64{}, h.T...), V: append([]float64{}, h.V...)})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (p *statusSensors) Sensors() []sensors.Sensor {
|
||||
p.mu.RLock()
|
||||
defer p.mu.RUnlock()
|
||||
var out []sensors.Sensor
|
||||
for _, v := range p.by {
|
||||
out = append(out, v...)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (p *statusSensors) services(ctx context.Context, t int64) []sensors.Sensor {
|
||||
units, err := p.s.svc.List(ctx)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var failed, running float64
|
||||
var names []string
|
||||
var per []sensors.Sensor
|
||||
for _, u := range units {
|
||||
if u.Failed {
|
||||
failed++
|
||||
names = append(names, u.Name)
|
||||
}
|
||||
if u.Sub == "running" {
|
||||
running++
|
||||
}
|
||||
// One pair per service that is meant to be doing something: enabled, up, or broken.
|
||||
// (Every static and one-shot unit on the box would triple the catalog for nothing.)
|
||||
short, isService := strings.CutSuffix(u.Name, ".service")
|
||||
if !isService || !(u.Enabled == "enabled" || u.Active == "active" || u.Failed) {
|
||||
continue
|
||||
}
|
||||
id, l := "service."+sensors.Seg(short), map[string]string{"unit": u.Name, "description": u.Description}
|
||||
per = append(per,
|
||||
sensors.Sensor{ID: id + ".active", Name: short, Kind: "state", Value: u.Active == "active", T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".state", Name: short + " state", Kind: "state", Value: u.Sub, T: t, Labels: l})
|
||||
}
|
||||
return append(per, []sensors.Sensor{
|
||||
{ID: "services.failed", Name: "Failed services", Kind: "count", Value: failed, T: t, Warn: sensors.F(1)},
|
||||
{ID: "services.failed_names", Name: "Failed services", Kind: "state", Value: strings.Join(names, ", "), T: t},
|
||||
{ID: "services.running", Name: "Running services", Kind: "count", Value: running, T: t},
|
||||
}...)
|
||||
}
|
||||
|
||||
// containers reads Docker (or Podman) with one `ps`, then each running container's memory
|
||||
// and CPU straight from its cgroup — `docker stats` takes seconds and a daemon round-trip
|
||||
// per call; two small files per container take microseconds.
|
||||
func (p *statusSensors) containers(ctx context.Context, t int64) []sensors.Sensor {
|
||||
var out []byte
|
||||
var err error
|
||||
tool := ""
|
||||
for _, t := range []string{"docker", "podman"} {
|
||||
if _, e := exec.LookPath(t); e != nil {
|
||||
continue
|
||||
}
|
||||
tool = t
|
||||
args := []string{tool, "ps", "-a", "--no-trunc", "--format", "{{json .}}"}
|
||||
if out, err = exec.CommandContext(ctx, args[0], args[1:]...).Output(); err != nil && priv.Available() {
|
||||
full := priv.Wrap(args...) // not in the docker group: ask as root
|
||||
out, err = exec.CommandContext(ctx, full[0], full[1:]...).Output()
|
||||
}
|
||||
break
|
||||
}
|
||||
if err != nil || len(out) == 0 {
|
||||
return nil
|
||||
}
|
||||
// "Not running" is not "down": servarr stopped by hand in June is off, not broken. What is
|
||||
// broken is a container that should be up and isn't — one inspect of the stopped ones tells
|
||||
// them apart: Docker refused to start it (State.Error: yesterday's Gitea), it is stuck
|
||||
// restarting, or it died with a non-zero code that no restart policy picked up. Under
|
||||
// `always` / `unless-stopped` a crash is restarted by Docker, so "exited" there means a
|
||||
// person stopped it. With no restart policy, `docker stop` leaves 143 (the process took the
|
||||
// SIGTERM), 137 (it didn't, and Docker killed it after the grace period) or 130 (SIGINT):
|
||||
// also a person, unless the kernel did the killing for want of memory.
|
||||
type stopped struct {
|
||||
Name string
|
||||
State struct {
|
||||
Status string
|
||||
ExitCode int
|
||||
Error string
|
||||
OOMKilled bool
|
||||
}
|
||||
HostConfig struct{ RestartPolicy struct{ Name string } }
|
||||
}
|
||||
down := map[string]bool{}
|
||||
var ids []string
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
var c struct{ ID, State string }
|
||||
if json.Unmarshal([]byte(line), &c) == nil && c.ID != "" && c.State != "running" {
|
||||
ids = append(ids, c.ID)
|
||||
}
|
||||
}
|
||||
if len(ids) > 0 {
|
||||
args := append([]string{"inspect"}, ids...)
|
||||
insp, ierr := exec.CommandContext(ctx, tool, args...).Output()
|
||||
if ierr != nil && priv.Available() {
|
||||
full := priv.Wrap(append([]string{tool}, args...)...)
|
||||
insp, _ = exec.CommandContext(ctx, full[0], full[1:]...).Output()
|
||||
}
|
||||
var list []stopped
|
||||
if json.Unmarshal(insp, &list) == nil {
|
||||
for _, c := range list {
|
||||
pol := c.HostConfig.RestartPolicy.Name
|
||||
managed := pol == "always" || pol == "unless-stopped"
|
||||
code := c.State.ExitCode
|
||||
byHand := (code == 130 || code == 137 || code == 143) && !c.State.OOMKilled
|
||||
down[strings.TrimPrefix(c.Name, "/")] = c.State.Status == "restarting" || c.State.Status == "dead" || c.State.Error != "" ||
|
||||
(!managed && c.State.Status == "exited" && code != 0 && !byHand)
|
||||
}
|
||||
}
|
||||
}
|
||||
var res []sensors.Sensor
|
||||
var running, unhealthy, total float64
|
||||
seenCPU := map[string]cpuMark{}
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
var c struct{ ID, Names, Image, State, Status string }
|
||||
if json.Unmarshal([]byte(line), &c) != nil || c.Names == "" {
|
||||
continue
|
||||
}
|
||||
total++
|
||||
name := strings.Split(c.Names, ",")[0]
|
||||
id, l := "container."+sensors.Seg(name), map[string]string{"container": name, "image": c.Image}
|
||||
up := c.State == "running"
|
||||
health := "none"
|
||||
for _, h := range []string{"unhealthy", "healthy", "starting"} { // "unhealthy" first: it contains "healthy"
|
||||
if strings.Contains(c.Status, "("+h+")") || strings.Contains(c.Status, "(health: "+h+")") {
|
||||
health = h
|
||||
break
|
||||
}
|
||||
}
|
||||
if up {
|
||||
running++
|
||||
}
|
||||
if health == "unhealthy" {
|
||||
unhealthy++
|
||||
}
|
||||
res = append(res,
|
||||
sensors.Sensor{ID: id + ".running", Name: name, Kind: "state", Value: up, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".down", Name: name + " down", Kind: "state", Value: down[name], T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".state", Name: name + " state", Kind: "state", Value: c.State, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".health", Name: name + " health", Kind: "state", Value: health, T: t, Labels: l})
|
||||
if !up {
|
||||
continue
|
||||
}
|
||||
cg := ""
|
||||
for _, dir := range []string{"/sys/fs/cgroup/system.slice/docker-" + c.ID + ".scope", "/sys/fs/cgroup/docker/" + c.ID, "/sys/fs/cgroup/machine.slice/libpod-" + c.ID + ".scope"} {
|
||||
if _, e := os.Stat(dir + "/memory.current"); e == nil {
|
||||
cg = dir
|
||||
break
|
||||
}
|
||||
}
|
||||
if cg == "" {
|
||||
continue
|
||||
}
|
||||
if b, e := os.ReadFile(cg + "/memory.current"); e == nil {
|
||||
if v, e := strconv.ParseFloat(strings.TrimSpace(string(b)), 64); e == nil {
|
||||
res = append(res, sensors.Sensor{ID: id + ".mem", Name: name + " memory", Kind: "bytes", Unit: "B", Value: v, T: t, Labels: l})
|
||||
}
|
||||
}
|
||||
if b, e := os.ReadFile(cg + "/cpu.stat"); e == nil {
|
||||
for _, ln := range strings.Split(string(b), "\n") {
|
||||
if f := strings.Fields(ln); len(f) == 2 && f[0] == "usage_usec" {
|
||||
usec, _ := strconv.ParseFloat(f[1], 64)
|
||||
seenCPU[c.ID] = cpuMark{usec, t}
|
||||
if prev, ok := p.cpuPrev[c.ID]; ok && t > prev.t && usec >= prev.usec {
|
||||
// percent of one core, like the Processes tab
|
||||
pctv := (usec - prev.usec) / 1000 / float64(t-prev.t) * 100
|
||||
res = append(res, sensors.Sensor{ID: id + ".cpu", Name: name + " CPU", Kind: "usage", Unit: "%", Value: float64(int(pctv*10+0.5)) / 10, T: t, Min: sensors.F(0), Labels: l})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
p.cpuPrev = seenCPU // only this loop touches it
|
||||
return append(res,
|
||||
sensors.Sensor{ID: "containers.total", Name: "Containers", Kind: "count", Value: total, T: t},
|
||||
sensors.Sensor{ID: "containers.running", Name: "Running containers", Kind: "count", Value: running, T: t},
|
||||
sensors.Sensor{ID: "containers.unhealthy", Name: "Unhealthy containers", Kind: "count", Value: unhealthy, T: t, Warn: sensors.F(1)})
|
||||
}
|
||||
|
||||
type cpuMark struct {
|
||||
usec float64
|
||||
t int64
|
||||
}
|
||||
|
||||
func (p *statusSensors) updates(ctx context.Context, t int64) []sensors.Sensor {
|
||||
st := p.s.updates.Status(ctx)
|
||||
if st.Manager == "" {
|
||||
return nil
|
||||
}
|
||||
return []sensors.Sensor{
|
||||
{ID: "updates.pending", Name: "Pending updates", Kind: "count", Value: float64(len(st.Packages)), T: t, Labels: map[string]string{"manager": st.Manager}},
|
||||
{ID: "updates.security", Name: "Pending security updates", Kind: "count", Value: float64(st.SecurityCount), T: t, Warn: sensors.F(1)},
|
||||
{ID: "updates.reboot_required", Name: "Reboot required", Kind: "state", Value: st.RebootNeeded, T: t},
|
||||
}
|
||||
}
|
||||
|
||||
func (p *statusSensors) zfs(ctx context.Context, t int64) []sensors.Sensor {
|
||||
var out []sensors.Sensor
|
||||
for _, pool := range p.s.storage.Pools(ctx) {
|
||||
id, l := "zfs."+sensors.Seg(pool.Name), map[string]string{"pool": pool.Name}
|
||||
out = append(out,
|
||||
sensors.Sensor{ID: id + ".health", Name: pool.Name + " health", Kind: "state", Value: pool.Health, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".healthy", Name: pool.Name + " healthy", Kind: "state", Value: pool.Health == "ONLINE", T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".size", Name: pool.Name + " size", Kind: "bytes", Unit: "B", Value: float64(pool.Size), T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".used", Name: pool.Name + " used", Kind: "bytes", Unit: "B", Value: float64(pool.Alloc), T: t, Min: sensors.F(0), Max: sensors.F(float64(pool.Size)), Labels: l},
|
||||
sensors.Sensor{ID: id + ".used_pct", Name: pool.Name + " used", Kind: "usage", Unit: "%", Value: float64(pool.Cap), T: t, Min: sensors.F(0), Max: sensors.F(100), Labels: l},
|
||||
sensors.Sensor{ID: id + ".frag", Name: pool.Name + " fragmentation", Kind: "fragmentation", Unit: "%", Value: float64(pool.Frag), T: t, Min: sensors.F(0), Max: sensors.F(100), Labels: l},
|
||||
)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (p *statusSensors) smart(ctx context.Context, t int64) []sensors.Sensor {
|
||||
devs, err := storage.Devices(ctx)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []sensors.Sensor
|
||||
for _, d := range devs {
|
||||
sm := p.s.storage.Smart(ctx, d.Name, false)
|
||||
if sm.Passed == nil || sm.Standby || sm.Error != "" {
|
||||
continue // no verdict to report: no smartctl, asleep, or not a SMART device
|
||||
}
|
||||
id, l := "smart."+sensors.Seg(d.Name), map[string]string{"device": d.Name, "model": d.Model}
|
||||
out = append(out,
|
||||
sensors.Sensor{ID: id + ".passed", Name: d.Name + " health", Kind: "state", Value: *sm.Passed, T: sm.FetchedAt, Labels: l},
|
||||
sensors.Sensor{ID: id + ".warnings", Name: d.Name + " warnings", Kind: "count", Value: float64(len(sm.Warnings)), T: sm.FetchedAt, Warn: sensors.F(1), Labels: l},
|
||||
)
|
||||
if sm.PowerOnHours > 0 {
|
||||
out = append(out, sensors.Sensor{ID: id + ".power_on", Name: d.Name + " powered on", Kind: "duration", Unit: "s", Value: float64(sm.PowerOnHours) * 3600, T: sm.FetchedAt, Labels: l})
|
||||
}
|
||||
if sm.NVMe != nil {
|
||||
out = append(out, sensors.Sensor{ID: id + ".wear", Name: d.Name + " endurance used", Kind: "usage", Unit: "%", Value: float64(sm.NVMe.PercentageUsed), T: sm.FetchedAt, Min: sensors.F(0), Max: sensors.F(100), Labels: l})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package api
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestStreamGate(t *testing.T) {
|
||||
var g streamGate
|
||||
for i := 0; i < maxStreamsPerClient; i++ {
|
||||
if !g.enter("a") {
|
||||
t.Fatalf("stream %d for one client refused early", i)
|
||||
}
|
||||
}
|
||||
if g.enter("a") {
|
||||
t.Fatal("one client went past its ceiling")
|
||||
}
|
||||
if !g.enter("b") {
|
||||
t.Fatal("another client must still get in")
|
||||
}
|
||||
g.leave("a")
|
||||
if !g.enter("a") {
|
||||
t.Fatal("a closed stream must free its place")
|
||||
}
|
||||
for i := 0; i < maxStreamsPerClient+1; i++ {
|
||||
g.leave("a")
|
||||
}
|
||||
g.leave("b")
|
||||
if g.total != -1 && g.total != 0 { // leave() past zero must not wedge the gate open or shut
|
||||
t.Logf("total after over-leave: %d", g.total)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,418 @@
|
||||
package api
|
||||
|
||||
// Previews for the Files tab: GET /api/files/thumb hands back one small JPEG of an image or
|
||||
// a video, made by internal/thumbs and kept in <data dir>/thumbs.db. The work is bounded
|
||||
// two ways — a handful of ffmpegs at once, and one per file however many rows ask for it —
|
||||
// because a folder of a thousand photos asks for every row the moment previews go on.
|
||||
//
|
||||
// Root mode (?sudo=1) makes the preview through the same root helper the file reads use,
|
||||
// so a file only root can read gets a preview too, and the probe runs as root as well: root
|
||||
// opens every render node, whatever groups the daemon's user has.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/media"
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
const thumbSize = 256
|
||||
const thumbTimeout = 30 * time.Second
|
||||
const capsTTL = 10 * time.Minute // how long a probe is believed; a refresh after an install or a group change does not wait
|
||||
|
||||
type thumbSvc struct {
|
||||
cache *media.Cache
|
||||
self string
|
||||
sem chan struct{}
|
||||
mu sync.Mutex
|
||||
caps map[bool]*capsEntry // keyed by root
|
||||
calls map[string]*thumbCall
|
||||
}
|
||||
|
||||
type capsEntry struct {
|
||||
mu sync.Mutex
|
||||
caps media.Caps
|
||||
err error
|
||||
at time.Time
|
||||
}
|
||||
|
||||
type thumbCall struct {
|
||||
done chan struct{}
|
||||
res media.Result
|
||||
err error
|
||||
}
|
||||
|
||||
func newThumbSvc(dataDir, self string) *thumbSvc {
|
||||
n := runtime.NumCPU() / 2 // each ffmpeg is itself threaded; previews must not take the whole machine
|
||||
if n < 2 {
|
||||
n = 2
|
||||
}
|
||||
if n > 4 {
|
||||
n = 4
|
||||
}
|
||||
return &thumbSvc{cache: media.NewCache(filepath.Join(dataDir, "thumbs.db")), self: self, sem: make(chan struct{}, n), caps: map[bool]*capsEntry{}, calls: map[string]*thumbCall{}}
|
||||
}
|
||||
|
||||
// capsFor is what previews can be made with, probed once and kept for a while.
|
||||
func (t *thumbSvc) capsFor(ctx context.Context, root, refresh bool) (media.Caps, error) {
|
||||
t.mu.Lock()
|
||||
e := t.caps[root]
|
||||
if e == nil {
|
||||
e = &capsEntry{}
|
||||
t.caps[root] = e
|
||||
}
|
||||
t.mu.Unlock()
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
if !refresh && !e.at.IsZero() && time.Since(e.at) < capsTTL {
|
||||
return e.caps, e.err
|
||||
}
|
||||
if root {
|
||||
e.caps, e.err = media.ProbeRoot(ctx, t.self)
|
||||
} else {
|
||||
e.caps, e.err = media.Probe(ctx), nil
|
||||
}
|
||||
e.at = time.Now()
|
||||
mode := "as " + map[bool]string{false: "the daemon", true: "root"}[root]
|
||||
switch {
|
||||
case e.err != nil:
|
||||
log.Printf("thumbs: probe %s failed: %v", mode, e.err)
|
||||
case e.caps.Error != "":
|
||||
log.Printf("thumbs: %s: %s", mode, e.caps.Error)
|
||||
default:
|
||||
log.Printf("thumbs: %s: ffmpeg %s, %s", mode, e.caps.Version, describeCaps(e.caps))
|
||||
}
|
||||
return e.caps, e.err
|
||||
}
|
||||
|
||||
// describeCaps is the one line that says how previews get made here.
|
||||
func describeCaps(c media.Caps) string {
|
||||
if len(c.GPUs) == 0 {
|
||||
return "CPU only"
|
||||
}
|
||||
parts := make([]string, 0, len(c.GPUs))
|
||||
for _, g := range c.GPUs {
|
||||
name := g.Name
|
||||
if name == "" {
|
||||
name = g.Device
|
||||
}
|
||||
p := name + " (" + g.API
|
||||
if len(g.Codecs) > 0 {
|
||||
p += ": " + strings.Join(g.Codecs, ", ")
|
||||
}
|
||||
parts = append(parts, p+")")
|
||||
}
|
||||
return "GPU " + strings.Join(parts, ", ")
|
||||
}
|
||||
|
||||
// make produces a preview, sharing the work when the same file is asked for twice at once.
|
||||
func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (media.Result, error) {
|
||||
t.mu.Lock()
|
||||
if c, ok := t.calls[key]; ok {
|
||||
t.mu.Unlock()
|
||||
select {
|
||||
case <-c.done:
|
||||
return c.res, c.err
|
||||
case <-ctx.Done():
|
||||
return media.Result{}, ctx.Err()
|
||||
}
|
||||
}
|
||||
c := &thumbCall{done: make(chan struct{})}
|
||||
t.calls[key] = c
|
||||
t.mu.Unlock()
|
||||
defer func() {
|
||||
t.mu.Lock()
|
||||
delete(t.calls, key)
|
||||
t.mu.Unlock()
|
||||
close(c.done)
|
||||
}()
|
||||
// The work itself runs on a context of its own: the first requester going away must not
|
||||
// cancel a preview three other rows are waiting for.
|
||||
wctx, cancel := context.WithTimeout(context.Background(), thumbTimeout+5*time.Second)
|
||||
defer cancel()
|
||||
select {
|
||||
case t.sem <- struct{}{}:
|
||||
defer func() { <-t.sem }()
|
||||
case <-ctx.Done():
|
||||
c.err = ctx.Err()
|
||||
return c.res, c.err
|
||||
}
|
||||
caps, err := t.capsFor(wctx, root, false)
|
||||
if err != nil {
|
||||
c.err = err
|
||||
return c.res, c.err
|
||||
}
|
||||
o := media.Opts{Size: thumbSize, Timeout: thumbTimeout}
|
||||
if root {
|
||||
c.res, c.err = media.MakeRoot(wctx, t.self, caps, path, o)
|
||||
} else {
|
||||
c.res, c.err = media.Make(wctx, caps, path, o)
|
||||
}
|
||||
return c.res, c.err
|
||||
}
|
||||
|
||||
// ---- handlers ----
|
||||
|
||||
// filesThumb: GET /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1]. v is the client's idea of
|
||||
// the file, from its listing; it makes a cache hit free of a stat, and a changed file a new URL.
|
||||
func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
|
||||
fsys, ok := s.fsFor(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
p, ok := cleanQuery(w, r, "path")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if media.Kind(p) == "" {
|
||||
jsonErr(w, 415, "no preview for this kind of file")
|
||||
return
|
||||
}
|
||||
root := r.URL.Query().Get("sudo") == "1"
|
||||
if mt, sz, ok := parseV(r.URL.Query().Get("v")); ok {
|
||||
key := media.Key(p, sz, mt)
|
||||
if r.Header.Get("If-None-Match") == `"`+key+`"` {
|
||||
w.WriteHeader(304)
|
||||
return
|
||||
}
|
||||
if s.serveCachedThumb(w, key) {
|
||||
return
|
||||
}
|
||||
}
|
||||
in, err := fsys.Stat(r.Context(), p)
|
||||
if err != nil {
|
||||
fsErr(w, err)
|
||||
return
|
||||
}
|
||||
if in.Type == "dir" || in.Type == "other" || in.Broken {
|
||||
jsonErr(w, 400, "not a regular file")
|
||||
return
|
||||
}
|
||||
key := media.Key(p, in.Size, in.Mtime)
|
||||
if s.serveCachedThumb(w, key) {
|
||||
return
|
||||
}
|
||||
res, err := s.thumbs.make(r.Context(), root, key, p)
|
||||
if err != nil {
|
||||
var de *media.DecodeError
|
||||
switch {
|
||||
case errors.As(err, &de):
|
||||
if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Failure: de.Msg}); err != nil {
|
||||
log.Printf("thumbs: cache: %v", err)
|
||||
}
|
||||
log.Printf("thumbs: %s: %s", p, de.Msg)
|
||||
jsonErr(w, 422, de.Msg)
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
log.Printf("thumbs: %s: timed out", p)
|
||||
jsonErr(w, 504, "preview timed out")
|
||||
case errors.Is(err, context.Canceled):
|
||||
jsonErr(w, 499, "cancelled")
|
||||
default:
|
||||
jsonErr(w, 503, err.Error())
|
||||
}
|
||||
return
|
||||
}
|
||||
if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}); err != nil {
|
||||
log.Printf("thumbs: cache: %v", err)
|
||||
}
|
||||
writeThumb(w, key, res.JPEG, res.Engine)
|
||||
}
|
||||
|
||||
// serveCachedThumb answers from the cache when it can. A cache that cannot be read is a
|
||||
// miss: the preview is made again rather than refused.
|
||||
func (s *Server) serveCachedThumb(w http.ResponseWriter, key string) bool {
|
||||
e, found, err := s.thumbs.cache.Get(key)
|
||||
if err != nil {
|
||||
log.Printf("thumbs: cache: %v", err)
|
||||
}
|
||||
switch {
|
||||
case !found:
|
||||
return false
|
||||
case e.Failure != "":
|
||||
jsonErr(w, 422, e.Failure)
|
||||
default:
|
||||
writeThumb(w, key, e.JPEG, "cache ("+e.Engine+")")
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func writeThumb(w http.ResponseWriter, key string, jpeg []byte, engine string) {
|
||||
w.Header().Set("Content-Type", "image/jpeg")
|
||||
w.Header().Set("Content-Length", strconv.Itoa(len(jpeg)))
|
||||
w.Header().Set("Cache-Control", "private, max-age=604800")
|
||||
w.Header().Set("ETag", `"`+key+`"`)
|
||||
w.Header().Set("X-Hamlab-Thumb", engine)
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
_, _ = w.Write(jpeg)
|
||||
}
|
||||
|
||||
func parseV(v string) (mtime, size int64, ok bool) {
|
||||
a, b, found := strings.Cut(v, "-")
|
||||
if !found {
|
||||
return 0, 0, false
|
||||
}
|
||||
mtime, err1 := strconv.ParseInt(a, 10, 64)
|
||||
size, err2 := strconv.ParseInt(b, 10, 64)
|
||||
return mtime, size, err1 == nil && err2 == nil
|
||||
}
|
||||
|
||||
type thumbStatus struct {
|
||||
OK bool `json:"ok"`
|
||||
Engine string `json:"engine"` // gpu | cpu | ""
|
||||
Summary string `json:"summary"`
|
||||
FFmpeg string `json:"ffmpeg,omitempty"`
|
||||
GPUs []media.GPU `json:"gpus"`
|
||||
Unusable []media.GPU `json:"unusable,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Install *thumbInstall `json:"install,omitempty"` // ffmpeg is missing and this can put it on
|
||||
GPUFix bool `json:"gpu_fix,omitempty"` // a render node is there but closed to the daemon's user, and this can open it
|
||||
User string `json:"user"`
|
||||
Cache thumbCache `json:"cache"`
|
||||
}
|
||||
|
||||
type thumbCache struct {
|
||||
Path string `json:"path"`
|
||||
Entries int64 `json:"entries"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
|
||||
type thumbInstall struct {
|
||||
PM string `json:"pm"`
|
||||
Package string `json:"package"`
|
||||
}
|
||||
|
||||
// thumbsStatus: GET /api/files/thumbs[?sudo=1][&refresh=1] — how previews get made on this machine.
|
||||
func (s *Server) thumbsStatus(w http.ResponseWriter, r *http.Request) {
|
||||
root := r.URL.Query().Get("sudo") == "1"
|
||||
if root && s.fsSudo == nil {
|
||||
jsonErr(w, 403, "root access needs passwordless sudo on the host")
|
||||
return
|
||||
}
|
||||
caps, err := s.thumbs.capsFor(r.Context(), root, r.URL.Query().Get("refresh") == "1")
|
||||
st := thumbStatus{GPUs: []media.GPU{}, User: s.user, Cache: thumbCache{Path: s.thumbs.cache.Path}}
|
||||
st.Cache.Entries, st.Cache.Bytes = s.thumbs.cache.Stats()
|
||||
if root {
|
||||
st.User = "root"
|
||||
}
|
||||
if err != nil {
|
||||
st.Error = err.Error()
|
||||
jsonOK(w, 200, st)
|
||||
return
|
||||
}
|
||||
st.FFmpeg, st.GPUs, st.Unusable, st.Error = caps.Version, caps.GPUs, caps.Unusable, caps.Error
|
||||
if st.GPUs == nil {
|
||||
st.GPUs = []media.GPU{}
|
||||
}
|
||||
st.OK = caps.Error == ""
|
||||
switch {
|
||||
case !st.OK:
|
||||
if pkg := ffmpegPackage(s.updates.PM()); pkg != "" && priv.Available() {
|
||||
st.Install = &thumbInstall{PM: s.updates.PM(), Package: pkg}
|
||||
}
|
||||
st.Summary = st.Error
|
||||
case len(caps.GPUs) > 0:
|
||||
st.Engine = "gpu"
|
||||
st.Summary = describeCaps(caps) + " · images on the CPU"
|
||||
default:
|
||||
st.Engine = "cpu"
|
||||
st.Summary = "CPU only"
|
||||
}
|
||||
if !root {
|
||||
for _, g := range caps.Unusable {
|
||||
if g.Permission && gpuFixPossible() {
|
||||
st.GPUFix = true
|
||||
}
|
||||
}
|
||||
}
|
||||
jsonOK(w, 200, st)
|
||||
}
|
||||
|
||||
// ffmpegPackage is what to install for ffmpeg on a package manager. Fedora ships it as
|
||||
// ffmpeg-free; the others under its own name.
|
||||
func ffmpegPackage(pm string) string {
|
||||
switch pm {
|
||||
case "":
|
||||
return ""
|
||||
case "dnf":
|
||||
if b, _ := os.ReadFile("/etc/os-release"); strings.Contains(string(b), "ID=fedora") {
|
||||
return "ffmpeg-free"
|
||||
}
|
||||
}
|
||||
return "ffmpeg"
|
||||
}
|
||||
|
||||
const hamlabUnit = "/etc/systemd/system/hamlab.service"
|
||||
|
||||
// gpuFixPossible: the daemon is an ordinary user under systemd, with root to hand.
|
||||
func gpuFixPossible() bool {
|
||||
if os.Geteuid() == 0 || !priv.Available() {
|
||||
return false
|
||||
}
|
||||
_, err := os.Stat(hamlabUnit)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// thumbsInstall: POST /api/files/thumbs/install — put ffmpeg on with the machine's package manager.
|
||||
func (s *Server) thumbsInstall(w http.ResponseWriter, r *http.Request) {
|
||||
pkg := ffmpegPackage(s.updates.PM())
|
||||
if pkg == "" {
|
||||
jsonErr(w, 400, "no supported package manager on this machine")
|
||||
return
|
||||
}
|
||||
t, err := s.updates.Install([]string{pkg})
|
||||
if err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("thumbs: installing %s as task %s (from %s)", pkg, t.ID, clientIP(r))
|
||||
jsonOK(w, 202, t.Snapshot(false))
|
||||
}
|
||||
|
||||
// gpuAccessScript gives hamlab.service the groups that own the render nodes, by a drop-in,
|
||||
// then restarts it: a running process cannot gain a group.
|
||||
const gpuAccessScript = `set -e
|
||||
unit=` + hamlabUnit + `
|
||||
[ -f "$unit" ] || { echo "no $unit: this machine does not run Ham Lab under systemd"; exit 1; }
|
||||
groups=""
|
||||
for g in render video; do
|
||||
if getent group "$g" >/dev/null 2>&1; then groups="$groups $g"; fi
|
||||
done
|
||||
[ -n "$groups" ] || { echo "neither a render nor a video group exists here"; exit 1; }
|
||||
mkdir -p "$unit.d"
|
||||
printf '[Service]\nSupplementaryGroups=%s\n' "${groups# }" > "$unit.d/gpu.conf"
|
||||
echo "hamlab.service: supplementary groups${groups} ($unit.d/gpu.conf)"
|
||||
systemctl daemon-reload
|
||||
echo 'Ham Lab restarts in 3 seconds to pick up its new groups; the page reconnects by itself.'
|
||||
systemd-run --quiet --on-active=3 --unit=hamlab-gpu-restart systemctl restart hamlab.service
|
||||
`
|
||||
|
||||
// thumbsGPUAccess: POST /api/files/thumbs/gpu-access — open the render nodes to the daemon.
|
||||
func (s *Server) thumbsGPUAccess(w http.ResponseWriter, r *http.Request) {
|
||||
if !gpuFixPossible() {
|
||||
jsonErr(w, 400, "nothing to do: Ham Lab is root, has no root, or is not under systemd here")
|
||||
return
|
||||
}
|
||||
if t, busy := s.tasks.RunningOf("thumbs:gpu"); busy { // a second run would only collide with the first's restart timer
|
||||
jsonErr(w, 409, "already being done (task "+t.ID+")")
|
||||
return
|
||||
}
|
||||
full := priv.Wrap("sh", "-c", gpuAccessScript)
|
||||
t, err := s.tasks.Start("thumbs:gpu", "give Ham Lab access to the GPU", exec.Command(full[0], full[1:]...))
|
||||
if err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("thumbs: GPU access drop-in as task %s, restart follows (from %s)", t.ID, clientIP(r))
|
||||
jsonOK(w, 202, t.Snapshot(false))
|
||||
}
|
||||
@@ -0,0 +1,217 @@
|
||||
package api
|
||||
|
||||
// Updating Ham Lab itself: is there a newer build published, and install it on request.
|
||||
// The check is a plain HTTP GET against whatever `update.url` names — this project's Gitea
|
||||
// releases by default — so a fleet, a stranger's install and an air-gapped copy pointed at
|
||||
// a directory on a NAS all work the same way. Nothing is ever installed unasked: an update
|
||||
// restarts the daemon and drops open terminals, which is not a thing to do to somebody.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/selfupdate"
|
||||
)
|
||||
|
||||
type updateState struct {
|
||||
mu sync.Mutex
|
||||
checking bool
|
||||
installing bool
|
||||
checkedAt time.Time
|
||||
latest selfupdate.Release
|
||||
err string
|
||||
}
|
||||
|
||||
// source is the update source as configured, and whether looking is switched off.
|
||||
func (s *Server) source() (selfupdate.Source, bool) {
|
||||
var u config.UpdateSource
|
||||
s.cfg.View(func(c *config.Config) { u = c.Updates })
|
||||
url := strings.TrimSpace(u.URL)
|
||||
if url == "" {
|
||||
url = selfupdate.DefaultURL
|
||||
}
|
||||
if strings.EqualFold(url, "off") || strings.EqualFold(url, "none") {
|
||||
return selfupdate.Source{}, false
|
||||
}
|
||||
return selfupdate.Source{URL: url, Token: u.Token}, true
|
||||
}
|
||||
|
||||
func (s *Server) checksEnabled() bool {
|
||||
var u config.UpdateSource
|
||||
s.cfg.View(func(c *config.Config) { u = c.Updates })
|
||||
if strings.EqualFold(u.Check, "never") {
|
||||
return false
|
||||
}
|
||||
_, on := s.source()
|
||||
return on
|
||||
}
|
||||
|
||||
// Behind reports whether a newer build than this one has been seen, and which.
|
||||
func (s *Server) behind() (string, bool) {
|
||||
s.update.mu.Lock()
|
||||
defer s.update.mu.Unlock()
|
||||
v := s.update.latest.Version
|
||||
if v == "" || selfupdate.Compare(s.version, v) >= 0 {
|
||||
return "", false
|
||||
}
|
||||
return v, true
|
||||
}
|
||||
|
||||
// checkNow asks the source what the newest build is. One at a time.
|
||||
func (s *Server) checkNow(ctx context.Context) {
|
||||
src, on := s.source()
|
||||
if !on {
|
||||
return
|
||||
}
|
||||
s.update.mu.Lock()
|
||||
if s.update.checking {
|
||||
s.update.mu.Unlock()
|
||||
return
|
||||
}
|
||||
s.update.checking = true
|
||||
s.update.mu.Unlock()
|
||||
ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
|
||||
defer cancel()
|
||||
rel, err := src.Latest(ctx, runtime.GOARCH)
|
||||
s.update.mu.Lock()
|
||||
s.update.checking = false
|
||||
s.update.checkedAt = time.Now()
|
||||
if err != nil {
|
||||
s.update.err = err.Error()
|
||||
} else {
|
||||
s.update.err, s.update.latest = "", rel
|
||||
}
|
||||
s.update.mu.Unlock()
|
||||
if err != nil {
|
||||
log.Printf("update check: %v", err)
|
||||
return
|
||||
}
|
||||
if selfupdate.Compare(s.version, rel.Version) < 0 {
|
||||
log.Printf("update check: %s is published; this is %s", rel.Version, s.version)
|
||||
}
|
||||
}
|
||||
|
||||
// watchForUpdates asks once shortly after boot and then daily.
|
||||
func (s *Server) watchForUpdates(ctx context.Context) {
|
||||
t := time.NewTimer(90 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
if s.checksEnabled() {
|
||||
s.checkNow(ctx)
|
||||
}
|
||||
t.Reset(selfupdate.DefaultCheckEvery)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) updateStatus(w http.ResponseWriter, _ *http.Request) {
|
||||
src, on := s.source()
|
||||
s.update.mu.Lock()
|
||||
defer s.update.mu.Unlock()
|
||||
latest := s.update.latest.Version
|
||||
out := map[string]any{
|
||||
"current": s.version, "latest": latest, "arch": runtime.GOARCH,
|
||||
"behind": latest != "" && selfupdate.Compare(s.version, latest) < 0,
|
||||
"checking": s.update.checking,
|
||||
"installing": s.update.installing,
|
||||
"enabled": on, "auto": s.checksEnabled(), "url": src.URL,
|
||||
"notes": s.update.latest.Notes, "published": s.update.latest.Published,
|
||||
}
|
||||
if !s.update.checkedAt.IsZero() {
|
||||
out["checked"] = s.update.checkedAt
|
||||
}
|
||||
if s.update.err != "" {
|
||||
out["error"] = s.update.err
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) updateCheck(w http.ResponseWriter, r *http.Request) {
|
||||
if _, on := s.source(); !on {
|
||||
jsonErr(w, 400, "update checks are switched off (update.url is \"off\")")
|
||||
return
|
||||
}
|
||||
s.checkNow(r.Context())
|
||||
s.updateStatus(w, r)
|
||||
}
|
||||
|
||||
// updateInstall fetches the published build for this machine and puts it in place.
|
||||
func (s *Server) updateInstall(w http.ResponseWriter, r *http.Request) {
|
||||
src, on := s.source()
|
||||
if !on {
|
||||
jsonErr(w, 400, "update checks are switched off (update.url is \"off\")")
|
||||
return
|
||||
}
|
||||
force := r.URL.Query().Get("force") == "1"
|
||||
if err := s.updateAllowed(force); err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
Version string `json:"version"` // optional: refuse if what is published is not this
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
|
||||
s.update.mu.Lock()
|
||||
if s.update.installing {
|
||||
s.update.mu.Unlock()
|
||||
jsonErr(w, 409, "an update is already being installed")
|
||||
return
|
||||
}
|
||||
rel := s.update.latest
|
||||
s.update.installing = true
|
||||
s.update.mu.Unlock()
|
||||
done := func() { s.update.mu.Lock(); s.update.installing = false; s.update.mu.Unlock() }
|
||||
|
||||
ctx := context.WithoutCancel(r.Context()) // a closed browser tab must not abort an install
|
||||
if rel.Version == "" || body.Version != "" && body.Version != rel.Version {
|
||||
var err error
|
||||
if rel, err = src.Latest(ctx, runtime.GOARCH); err != nil {
|
||||
done()
|
||||
jsonErr(w, 502, err.Error())
|
||||
return
|
||||
}
|
||||
s.update.mu.Lock()
|
||||
s.update.latest, s.update.checkedAt, s.update.err = rel, time.Now(), ""
|
||||
s.update.mu.Unlock()
|
||||
}
|
||||
if body.Version != "" && body.Version != rel.Version {
|
||||
done()
|
||||
jsonErr(w, 409, "what is published is now "+rel.Version+", not "+body.Version+"; look again")
|
||||
return
|
||||
}
|
||||
if !force && selfupdate.Compare(s.version, rel.Version) >= 0 {
|
||||
done()
|
||||
jsonErr(w, 409, "this machine is already running "+s.version+"; nothing newer is published")
|
||||
return
|
||||
}
|
||||
tmp := filepath.Join(s.cfg.DataDir(), "hamlab.new")
|
||||
if err := src.Fetch(ctx, rel, runtime.GOARCH, tmp); err != nil {
|
||||
done()
|
||||
jsonErr(w, 502, err.Error())
|
||||
return
|
||||
}
|
||||
newVersion, err := s.installBinary(ctx, tmp)
|
||||
if err != nil {
|
||||
done()
|
||||
jsonErr(w, codeOf(err, 500), err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("update: installed %s (was %s), asked for by %s; restarting", newVersion, s.version, clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"version": newVersion, "previous": s.version})
|
||||
s.restartAfterUpdate()
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakeDocker puts a `docker` on PATH whose `image inspect` answers with the given RepoDigests.
|
||||
func fakeDocker(t *testing.T, digests string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
script := "#!/bin/sh\necho '" + digests + "'\n"
|
||||
if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(script), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("PATH", dir)
|
||||
}
|
||||
|
||||
// An update pulls the newer image: the cached registry answer must not keep calling it behind.
|
||||
func TestSettledSeesAPulledImage(t *testing.T) {
|
||||
cached := []imageCheck{
|
||||
{Image: "nginx:alpine", State: "update", remote: "sha256:new"},
|
||||
{Image: "redis:7", State: "current", remote: "sha256:r"},
|
||||
{Image: "local/thing", State: "unknown"},
|
||||
}
|
||||
fakeDocker(t, `["nginx@sha256:new"]`)
|
||||
got := settled(context.Background(), cached)
|
||||
if got[0].State != "current" {
|
||||
t.Fatalf("pulled image still marked %q", got[0].State)
|
||||
}
|
||||
if got[1].State != "current" || got[2].State != "unknown" {
|
||||
t.Fatalf("others changed: %+v", got)
|
||||
}
|
||||
if cached[0].State != "update" {
|
||||
t.Fatal("settled wrote through to the cache")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSettledKeepsAnImageStillBehind(t *testing.T) {
|
||||
fakeDocker(t, `["nginx@sha256:old"]`)
|
||||
got := settled(context.Background(), []imageCheck{{Image: "nginx:alpine", State: "update", remote: "sha256:new"}})
|
||||
if got[0].State != "update" {
|
||||
t.Fatalf("image not yet pulled marked %q", got[0].State)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,300 @@
|
||||
package api
|
||||
|
||||
// Accounts and their sessions. Every person has their own login, and the console's look
|
||||
// (theme, accents, dashboard layout) follows the account rather than the machine. Anyone
|
||||
// signed in may add or remove accounts — a household console, not a tenancy — but removing
|
||||
// one, or resetting another's password, asks for the caller's own password first, so a
|
||||
// picked-up unlocked phone is not enough.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// actor is who a request is from: a signed-in account (with its session), or a token.
|
||||
type actor struct {
|
||||
User string // account name, "" for a token
|
||||
Session string // session ID, "" for a token
|
||||
token string // the raw cookie token, so a handler can spare the current session
|
||||
Token string // token name, "" for a session
|
||||
// Guest marks a temporary look-around login, and Expires is when it stops working. What
|
||||
// a guest may reach is decided in the middleware, not here; see internal/api/guests.go.
|
||||
Guest bool
|
||||
Expires time.Time
|
||||
}
|
||||
|
||||
type ctxKey int
|
||||
|
||||
const actorKey ctxKey = 1
|
||||
|
||||
func withActor(r *http.Request, a actor) *http.Request {
|
||||
return r.WithContext(context.WithValue(r.Context(), actorKey, a))
|
||||
}
|
||||
|
||||
func actorOf(r *http.Request) actor {
|
||||
a, _ := r.Context().Value(actorKey).(actor)
|
||||
return a
|
||||
}
|
||||
|
||||
// who names the caller for a log line: "core@10.0.0.5" for a person, "token:hub:core@…" for a hub.
|
||||
func (a actor) String() string {
|
||||
switch {
|
||||
case a.User != "" && a.Guest:
|
||||
return "guest:" + a.User
|
||||
case a.User != "":
|
||||
return a.User
|
||||
case a.Token != "":
|
||||
return "token:" + a.Token
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- users ----
|
||||
|
||||
func (s *Server) listUsers(w http.ResponseWriter, r *http.Request) {
|
||||
me := actorOf(r).User
|
||||
counts := s.sessions.Count()
|
||||
var out []map[string]any
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, u := range c.Accounts() { // guest logins are listed on their own, see listGuests
|
||||
var seen any
|
||||
if !u.LastSeen.IsZero() {
|
||||
seen = u.LastSeen
|
||||
}
|
||||
out = append(out, map[string]any{"name": u.Name, "created": u.Created, "last_seen": seen, "sessions": counts[u.Name], "self": u.Name == me})
|
||||
}
|
||||
})
|
||||
if out == nil {
|
||||
out = []map[string]any{}
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) addUser(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name, Password string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
name := strings.TrimSpace(body.Name)
|
||||
if !auth.UserNameRe.MatchString(name) {
|
||||
jsonErr(w, 400, "a user name is 1–32 lower-case letters, digits, . _ or -, starting with a letter or digit")
|
||||
return
|
||||
}
|
||||
if strings.HasPrefix(name, guestPrefix) {
|
||||
jsonErr(w, 400, "names starting with "+guestPrefix+" belong to guest logins")
|
||||
return
|
||||
}
|
||||
if len(body.Password) < auth.MinPasswordLen {
|
||||
jsonErr(w, 400, "password must be at least 12 characters")
|
||||
return
|
||||
}
|
||||
h, err := auth.HashPassword(body.Password)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
clash := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
if _, ok := c.User(name); ok {
|
||||
clash = true
|
||||
return
|
||||
}
|
||||
c.Users = append(c.Users, config.User{Name: name, Hash: h, Created: time.Now()})
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if clash {
|
||||
jsonErr(w, 409, "a user with that name exists")
|
||||
return
|
||||
}
|
||||
log.Printf("user %q added (by %s)", name, clientIP(r))
|
||||
jsonOK(w, 200, map[string]any{"name": name})
|
||||
}
|
||||
|
||||
// confirmSelf checks the caller's own password, the gate on anything done to another account.
|
||||
func (s *Server) confirmSelf(w http.ResponseWriter, r *http.Request, current string) bool {
|
||||
me := actorOf(r).User
|
||||
if me == "" {
|
||||
jsonErr(w, 403, "only a signed-in user may do that")
|
||||
return false
|
||||
}
|
||||
var users []config.User
|
||||
s.cfg.View(func(c *config.Config) { users = append(users, c.Users...) })
|
||||
if !auth.CheckUser(users, me, current) {
|
||||
jsonErr(w, 403, "your password is wrong")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *Server) removeUser(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
var body struct{ Current string }
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
if name == actorOf(r).User {
|
||||
jsonErr(w, 400, "you can't remove the account you are signed in as")
|
||||
return
|
||||
}
|
||||
if !s.confirmSelf(w, r, body.Current) {
|
||||
return
|
||||
}
|
||||
found, last := false, false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
if len(c.Accounts()) <= 1 { // guest logins don't keep the last real account alive
|
||||
last = true
|
||||
return
|
||||
}
|
||||
kept := c.Users[:0]
|
||||
for _, u := range c.Users {
|
||||
if u.Name == name {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
c.Users = kept
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if last {
|
||||
jsonErr(w, 400, "the last account can't be removed")
|
||||
return
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such user")
|
||||
return
|
||||
}
|
||||
s.sessions.RevokeUser(name)
|
||||
_ = s.store.Update(func(st *store.State) { delete(st.Users, name) })
|
||||
log.Printf("user %q removed (by %s)", name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// resetPassword sets another user's password, on the caller's own password.
|
||||
func (s *Server) resetPassword(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
var body struct{ Current, Password string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if !s.confirmSelf(w, r, body.Current) {
|
||||
return
|
||||
}
|
||||
if len(body.Password) < auth.MinPasswordLen {
|
||||
jsonErr(w, 400, "password must be at least 12 characters")
|
||||
return
|
||||
}
|
||||
if !s.setPassword(w, name, body.Password) {
|
||||
return
|
||||
}
|
||||
if name != actorOf(r).User { // whoever held the old password is signed out
|
||||
s.sessions.RevokeUser(name)
|
||||
}
|
||||
log.Printf("password of %q reset (by %s)", name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) setPassword(w http.ResponseWriter, name, pw string) bool {
|
||||
h, err := auth.HashPassword(pw)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return false
|
||||
}
|
||||
found := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
for i := range c.Users {
|
||||
if c.Users[i].Name == name {
|
||||
c.Users[i].Hash = h
|
||||
found = true
|
||||
}
|
||||
}
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return false
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such user")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ---- sessions: where the caller's account is signed in ----
|
||||
|
||||
func (s *Server) listSessions(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
if a.User == "" {
|
||||
jsonOK(w, 200, []any{})
|
||||
return
|
||||
}
|
||||
var out []map[string]any
|
||||
for _, ses := range s.sessions.List(a.User) {
|
||||
out = append(out, map[string]any{"id": ses.ID, "created": ses.Created, "last_seen": ses.LastSeen, "expires": ses.Expires, "ip": ses.IP, "agent": ses.Agent, "current": ses.ID == a.Session})
|
||||
}
|
||||
if out == nil {
|
||||
out = []map[string]any{}
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) revokeSession(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
id := r.PathValue("id")
|
||||
if a.User == "" || !s.sessions.Revoke(a.User, id) {
|
||||
jsonErr(w, 404, "no such session")
|
||||
return
|
||||
}
|
||||
log.Printf("session %s of %q revoked (by %s)", id, a.User, clientIP(r))
|
||||
if id == a.Session {
|
||||
s.setCookie(w, r, "", -1)
|
||||
}
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// revokeIdleSessions drops the caller's sessions that have not been used for ?hours (24 by
|
||||
// default): the scripts and headless browsers that signed in once and never came back,
|
||||
// without touching the phone that was used this morning.
|
||||
func (s *Server) revokeIdleSessions(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
if a.User == "" {
|
||||
jsonErr(w, 403, "only a signed-in user may do that")
|
||||
return
|
||||
}
|
||||
hours := 24
|
||||
if h := r.URL.Query().Get("hours"); h != "" {
|
||||
n, err := strconv.Atoi(h)
|
||||
if err != nil || n < 1 || n > 24*365 {
|
||||
jsonErr(w, 400, "hours is a whole number from 1 to 8760")
|
||||
return
|
||||
}
|
||||
hours = n
|
||||
}
|
||||
n := s.sessions.RevokeIdle(a.User, a.token, time.Duration(hours)*time.Hour)
|
||||
log.Printf("%d idle session(s) of %q revoked (idle > %dh, by %s)", n, a.User, hours, clientIP(r))
|
||||
jsonOK(w, 200, map[string]int{"revoked": n})
|
||||
}
|
||||
|
||||
func (s *Server) revokeOtherSessions(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
if a.User == "" {
|
||||
jsonErr(w, 403, "only a signed-in user may do that")
|
||||
return
|
||||
}
|
||||
n := s.sessions.RevokeOthers(a.User, a.token)
|
||||
log.Printf("%d other session(s) of %q revoked (by %s)", n, a.User, clientIP(r))
|
||||
jsonOK(w, 200, map[string]int{"revoked": n})
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
package api
|
||||
|
||||
// The VPN tab: this machine's WireGuard tunnels, who is on them, and — where the tunnel is
|
||||
// one this machine configures — adding, renaming and removing the devices that dial in.
|
||||
//
|
||||
// Per machine like every other tab: the tunnels shown and edited are the ones on the machine
|
||||
// in the switcher, which is the only honest way to show them, since the inbound server and an
|
||||
// outbound tunnel are different machines here.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"hamlab/internal/vpn"
|
||||
)
|
||||
|
||||
func (s *Server) vpnReport(w http.ResponseWriter, r *http.Request) {
|
||||
jsonOK(w, 200, vpn.Look(r.Context()))
|
||||
}
|
||||
|
||||
// tunnelOf finds the tunnel a request names, so every editing handler starts from the live
|
||||
// state rather than from whatever the page last saw.
|
||||
func (s *Server) tunnelOf(ctx context.Context, w http.ResponseWriter, name string) (vpn.Tunnel, bool) {
|
||||
for _, t := range vpn.Look(ctx).Tunnels {
|
||||
if t.Name == name {
|
||||
if !t.Managed {
|
||||
jsonErr(w, 400, vpn.ErrUnmanaged.Error())
|
||||
return t, false
|
||||
}
|
||||
return t, true
|
||||
}
|
||||
}
|
||||
jsonErr(w, 404, "no such tunnel on this machine")
|
||||
return vpn.Tunnel{}, false
|
||||
}
|
||||
|
||||
// addVPNPeer makes a device and hands back its profile. The profile holds a private key that
|
||||
// is generated for this call and stored nowhere, so it is the only time it can be read — the
|
||||
// same bargain as any password this console generates.
|
||||
func (s *Server) addVPNPeer(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var body vpn.NewPeer
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 8<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
made, err := vpn.AddPeer(r.Context(), t, body)
|
||||
if err != nil {
|
||||
code := 400
|
||||
if errors.Is(err, vpn.ErrUnmanaged) {
|
||||
code = 409
|
||||
}
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
// The name and address are worth a log line; the profile is emphatically not.
|
||||
log.Printf("vpn: device %q added to %s as %s (by %s)", made.Name, t.Name, made.Address, clientIP(r))
|
||||
jsonOK(w, 200, made)
|
||||
}
|
||||
|
||||
func (s *Server) editVPNPeer(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var body struct{ Name, Address string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
section := r.PathValue("section")
|
||||
if err := vpn.EditPeer(r.Context(), t, section, body.Name, body.Address); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("vpn: device %s on %s changed (by %s)", section, t.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) removeVPNPeer(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
section := r.PathValue("section")
|
||||
var body struct{ Confirm string }
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4<<10)).Decode(&body)
|
||||
// The name has to be typed back. Removing a device is the one action here that cannot be
|
||||
// undone — the key is gone and the profile on the device stops working — and it is one tap
|
||||
// away from a list of devices that all look alike.
|
||||
var want string
|
||||
for _, p := range t.Peers {
|
||||
if p.Section == section {
|
||||
want = p.Name
|
||||
}
|
||||
}
|
||||
if want != "" && !strings.EqualFold(strings.TrimSpace(body.Confirm), want) {
|
||||
jsonErr(w, 400, "type the device's name to remove it")
|
||||
return
|
||||
}
|
||||
if err := vpn.DeletePeer(r.Context(), t, section); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("vpn: device %s (%s) removed from %s (by %s)", section, want, t.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,281 @@
|
||||
package api
|
||||
|
||||
// The ZFS tab: pools, datasets, snapshots, and a schedule for taking and expiring them.
|
||||
//
|
||||
// grid already runs zfSnap from root cron and has done for longer than this repo has
|
||||
// existed, so this does not replace it. It reads the same snapshot names, expires by the
|
||||
// same rule, and surfaces the cron job that drives it — so what is on screen is the truth
|
||||
// about the machine, whether Ham Lab or zfSnap is doing the work. A Ham Lab schedule is for
|
||||
// a machine that has no such tool, and the page says plainly when both are running.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/storage"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
type zfsState struct {
|
||||
Policies []storage.Policy `json:"policies"`
|
||||
}
|
||||
|
||||
func (s *Server) zfsLoad() zfsState {
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.ZFS })
|
||||
var z zfsState
|
||||
if len(raw) > 0 {
|
||||
_ = json.Unmarshal(raw, &z)
|
||||
}
|
||||
if z.Policies == nil {
|
||||
z.Policies = []storage.Policy{}
|
||||
}
|
||||
return z
|
||||
}
|
||||
|
||||
func (s *Server) zfsSave(z zfsState) error {
|
||||
b, err := json.Marshal(z)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.store.Update(func(st *store.State) { st.ZFS = b })
|
||||
}
|
||||
|
||||
// externalSnapshotJob finds a cron job that is already taking snapshots on this machine, so
|
||||
// the page can say so rather than let two schedules quietly double up.
|
||||
func (s *Server) externalSnapshotJob(ctx context.Context) map[string]any {
|
||||
if s.cron == nil {
|
||||
return nil
|
||||
}
|
||||
ov := s.cron.Overview(ctx)
|
||||
for _, j := range ov.Jobs {
|
||||
if !j.Enabled {
|
||||
continue
|
||||
}
|
||||
c := strings.ToLower(j.Command)
|
||||
if !strings.Contains(c, "zfsnap") && !strings.Contains(c, "zfs-auto-snapshot") && !strings.Contains(c, "sanoid") &&
|
||||
!(strings.Contains(c, "zfs ") && strings.Contains(c, "snapshot")) {
|
||||
continue
|
||||
}
|
||||
return map[string]any{
|
||||
"tool": toolName(c), "schedule": j.Schedule, "command": j.Command,
|
||||
"user": j.User, "source": j.Source, "id": j.ID, "next": j.Next, "last": j.Last,
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func toolName(cmd string) string {
|
||||
switch {
|
||||
case strings.Contains(cmd, "zfsnap"):
|
||||
return "zfSnap"
|
||||
case strings.Contains(cmd, "zfs-auto-snapshot"):
|
||||
return "zfs-auto-snapshot"
|
||||
case strings.Contains(cmd, "sanoid"):
|
||||
return "sanoid"
|
||||
}
|
||||
return "a cron job"
|
||||
}
|
||||
|
||||
// randomID is a short opaque id for a schedule.
|
||||
func randomID() (string, error) {
|
||||
b := make([]byte, 6)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(b), nil
|
||||
}
|
||||
|
||||
// ---- the cron file ----
|
||||
//
|
||||
// A schedule is a cron line, written to /etc/cron.d/hamlab-zfs, which Ham Lab owns and
|
||||
// rewrites whole whenever the schedules change. Nothing in the daemon runs them: snapshots
|
||||
// are backup-shaped and should not stop because a web console is down, being upgraded, or
|
||||
// uninstalled. The line calls `hamlab zfs-snapshot <id>` as the account Ham Lab runs as,
|
||||
// which reaches zfs through sudo exactly as the daemon does.
|
||||
|
||||
// syncCron puts the current schedules on the machine.
|
||||
func (s *Server) syncCron(ctx context.Context, z zfsState) error {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
exe, _ = filepath.EvalSymlinks(exe)
|
||||
return s.storage.WriteCronFile(ctx, z.Policies, s.user, exe, s.cfg.Path())
|
||||
}
|
||||
|
||||
// ---- endpoints ----
|
||||
|
||||
func (s *Server) zfsOverview(w http.ResponseWriter, r *http.Request) {
|
||||
z := s.zfsLoad()
|
||||
now := time.Now()
|
||||
dir := s.cfg.DataDir()
|
||||
out := make([]map[string]any, 0, len(z.Policies))
|
||||
for i := range z.Policies {
|
||||
p := z.Policies[i]
|
||||
expr, _ := p.CronExpr()
|
||||
m := map[string]any{"policy": p, "cron": expr, "last": storage.ReadRun(dir, p.ID)}
|
||||
if p.Enabled {
|
||||
if n := p.NextRun(now); !n.IsZero() {
|
||||
m["next"] = n
|
||||
}
|
||||
}
|
||||
out = append(out, m)
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"zfs": s.storage.ZFS(r.Context()), "policies": out, "external": s.externalSnapshotJob(r.Context()),
|
||||
})
|
||||
}
|
||||
|
||||
// zfsARC is the cache: the sampler's live ARC (size, the hit rate over the last interval)
|
||||
// beside the storage package's fuller picture (since-boot counters, tunables, and each
|
||||
// pool's cache/log/special devices or the lack of them).
|
||||
func (s *Server) zfsARC(w http.ResponseWriter, r *http.Request) {
|
||||
out := map[string]any{"detail": s.storage.ARC(r.Context()), "live": nil}
|
||||
if smp, ok := s.metrics.Latest(); ok && smp.ARC != nil {
|
||||
out["live"] = smp.ARC
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) zfsPolicyPut(w http.ResponseWriter, r *http.Request) {
|
||||
var p storage.Policy
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 16<<10)).Decode(&p); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if err := p.Validate(); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
z := s.zfsLoad()
|
||||
found := false
|
||||
for i := range z.Policies {
|
||||
if z.Policies[i].ID == p.ID && p.ID != "" {
|
||||
z.Policies[i] = p
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
if p.ID == "" {
|
||||
id, err := randomID()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
p.ID = id
|
||||
}
|
||||
z.Policies = append(z.Policies, p)
|
||||
}
|
||||
if err := s.zfsSave(z); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.syncCron(r.Context(), z); err != nil {
|
||||
jsonErr(w, 500, "saved, but writing "+storage.CronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
expr, _ := p.CronExpr()
|
||||
log.Printf("zfs schedule %q saved by %s (%s, %q, keep %s)", p.Name, clientIP(r), p.Dataset, expr, p.KeepFor)
|
||||
jsonOK(w, 200, p)
|
||||
}
|
||||
|
||||
func (s *Server) zfsPolicyDelete(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
z := s.zfsLoad()
|
||||
kept := z.Policies[:0]
|
||||
found := false
|
||||
for _, p := range z.Policies {
|
||||
if p.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, p)
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
z.Policies = kept
|
||||
if err := s.zfsSave(z); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.syncCron(r.Context(), z); err != nil {
|
||||
jsonErr(w, 500, "removed, but writing "+storage.CronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
storage.ForgetRun(s.cfg.DataDir(), id)
|
||||
log.Printf("zfs schedule %s removed by %s (its snapshots are left alone)", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) zfsPolicy(id string) (storage.Policy, bool) {
|
||||
for _, p := range s.zfsLoad().Policies {
|
||||
if p.ID == id {
|
||||
return p, true
|
||||
}
|
||||
}
|
||||
return storage.Policy{}, false
|
||||
}
|
||||
|
||||
func (s *Server) zfsRunNow(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
p, ok := s.zfsPolicy(id)
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
res, err := s.storage.Run(r.Context(), &p)
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: res.Taken, Pruned: res.Pruned}
|
||||
if err != nil {
|
||||
rec.Error = err.Error()
|
||||
}
|
||||
_ = storage.WriteRun(s.cfg.DataDir(), id, rec)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("zfs schedule %q run by %s: took %s, expired %d", p.Name, clientIP(r), res.Taken, len(res.Pruned))
|
||||
jsonOK(w, 200, res)
|
||||
}
|
||||
|
||||
// zfsSnapshotDelete removes one snapshot, named in full and typed back by the caller.
|
||||
func (s *Server) zfsSnapshotDelete(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Name string `json:"name"`
|
||||
Confirm string `json:"confirm"`
|
||||
Recursive bool `json:"recursive"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if body.Confirm != body.Name {
|
||||
jsonErr(w, 400, "type the snapshot's full name to confirm")
|
||||
return
|
||||
}
|
||||
if err := s.storage.DestroySnapshot(r.Context(), body.Name, body.Recursive); err != nil {
|
||||
msg := err.Error()
|
||||
switch {
|
||||
case msg == "bad snapshot name":
|
||||
jsonErr(w, 400, msg)
|
||||
case strings.Contains(msg, "does not exist"):
|
||||
jsonErr(w, 404, msg)
|
||||
default:
|
||||
jsonErr(w, 500, msg)
|
||||
}
|
||||
return
|
||||
}
|
||||
log.Printf("zfs snapshot %s destroyed by %s%s", body.Name, clientIP(r), map[bool]string{true: " (recursive)", false: ""}[body.Recursive])
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
+325
-33
@@ -3,17 +3,21 @@ package auth
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"golang.org/x/crypto/argon2"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
@@ -24,6 +28,10 @@ const (
|
||||
argonKeyLen = 32
|
||||
|
||||
SessionTTL = 30 * 24 * time.Hour
|
||||
// SessionIdle is how long a session may go unused before it is dropped, whatever its
|
||||
// expiry says. Every script, curl and headless browser that signs in leaves a session;
|
||||
// a phone that is used keeps its own alive by being used.
|
||||
SessionIdle = 7 * 24 * time.Hour
|
||||
)
|
||||
|
||||
func HashPassword(pw string) (string, error) {
|
||||
@@ -59,6 +67,118 @@ func VerifyPassword(hash, pw string) bool {
|
||||
return subtle.ConstantTimeCompare(got, want) == 1
|
||||
}
|
||||
|
||||
// ---- API tokens (hub → node) ----
|
||||
|
||||
// NewToken returns a random bearer token and its SHA-256 hex, which is what gets stored.
|
||||
func NewToken() (token, hash string, err error) {
|
||||
b := make([]byte, 32)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
token = base64.RawURLEncoding.EncodeToString(b)
|
||||
return token, HashToken(token), nil
|
||||
}
|
||||
|
||||
func HashToken(token string) string {
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// TokenMatches compares a presented token against stored hashes in constant time.
|
||||
func TokenMatches(token string, hashes []string) bool { return TokenIndex(token, hashes) >= 0 }
|
||||
|
||||
// TokenIndex is which of hashes the presented token matches, -1 for none; every hash is
|
||||
// compared whatever the answer.
|
||||
func TokenIndex(token string, hashes []string) int {
|
||||
h := []byte(HashToken(token))
|
||||
idx := -1
|
||||
for i, want := range hashes {
|
||||
if subtle.ConstantTimeCompare(h, []byte(want)) == 1 {
|
||||
idx = i
|
||||
}
|
||||
}
|
||||
return idx
|
||||
}
|
||||
|
||||
// ---- users ----
|
||||
|
||||
// UserNameRe is what an account may be called: like a Unix login, lower-case.
|
||||
var UserNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{0,31}$`)
|
||||
|
||||
const MinPasswordLen = 12
|
||||
|
||||
// dummyHash is verified against when the named user does not exist, so a login attempt
|
||||
// takes the same time either way and the form cannot be used to list accounts.
|
||||
var dummyHash = func() string {
|
||||
h, err := HashPassword("not-a-real-password")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return h
|
||||
}()
|
||||
|
||||
// CheckUser verifies a password for one of users. A missing user — or a guest login whose
|
||||
// time is up — fails in the same time as a wrong password.
|
||||
func CheckUser(users []config.User, name, pw string) bool {
|
||||
for _, u := range users {
|
||||
if u.Name == name {
|
||||
if u.Expired(time.Now()) {
|
||||
break
|
||||
}
|
||||
return VerifyPassword(u.Hash, pw)
|
||||
}
|
||||
}
|
||||
VerifyPassword(dummyHash, pw)
|
||||
return false
|
||||
}
|
||||
|
||||
// ---- guest logins ----
|
||||
|
||||
// guestAlphabet leaves out every letter and digit that can be misread in a screenshot or
|
||||
// over the phone: i, l, o, 0 and 1. A guest login is read aloud and typed by hand.
|
||||
const guestAlphabet = "abcdefghjkmnpqrstuvwxyz23456789"
|
||||
|
||||
// randomFrom returns n characters of guestAlphabet, chosen without modulo bias.
|
||||
func randomFrom(n int) (string, error) {
|
||||
out := make([]byte, 0, n)
|
||||
buf := make([]byte, n)
|
||||
limit := byte(256 - 256%len(guestAlphabet)) // the top of the last whole run of the alphabet
|
||||
for len(out) < n {
|
||||
if _, err := rand.Read(buf); err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, b := range buf {
|
||||
if b >= limit {
|
||||
continue // would favour the first few letters; ask for another byte instead
|
||||
}
|
||||
out = append(out, guestAlphabet[int(b)%len(guestAlphabet)])
|
||||
if len(out) == n {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return string(out), nil
|
||||
}
|
||||
|
||||
// GuestPassword is a password for a guest login: four groups of four, ~78 bits, in an
|
||||
// alphabet with nothing ambiguous in it.
|
||||
func GuestPassword() (string, error) {
|
||||
var groups []string
|
||||
for i := 0; i < 4; i++ {
|
||||
g, err := randomFrom(4)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
groups = append(groups, g)
|
||||
}
|
||||
return strings.Join(groups, "-"), nil
|
||||
}
|
||||
|
||||
// GuestSuffix is the tail of a generated guest name.
|
||||
func GuestSuffix() (string, error) { return randomFrom(4) }
|
||||
|
||||
// ---- sessions ----
|
||||
|
||||
// Sessions is a token store backed by the state file. Expiry is sliding.
|
||||
type Sessions struct {
|
||||
st *store.Store
|
||||
@@ -72,89 +192,261 @@ func NewSessions(st *store.Store) *Sessions {
|
||||
|
||||
func pruneExpired(state *store.State) {
|
||||
now := time.Now()
|
||||
for tok, exp := range state.Sessions {
|
||||
if exp.Before(now) {
|
||||
for tok, ses := range state.Sessions {
|
||||
if !live(ses, now) {
|
||||
delete(state.Sessions, tok)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Sessions) Create() (string, error) {
|
||||
b := make([]byte, 32)
|
||||
// live is whether a session still counts: unexpired, and used within SessionIdle. A
|
||||
// session from before visits were recorded is judged by its creation, or failing that by
|
||||
// its expiry alone.
|
||||
func live(ses store.Session, now time.Time) bool {
|
||||
if !ses.Expires.After(now) {
|
||||
return false
|
||||
}
|
||||
seen := ses.LastSeen
|
||||
if seen.IsZero() {
|
||||
seen = ses.Created
|
||||
}
|
||||
return seen.IsZero() || now.Sub(seen) <= SessionIdle
|
||||
}
|
||||
|
||||
func randomHex(n int) (string, error) {
|
||||
b := make([]byte, n)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", err
|
||||
}
|
||||
tok := hex.EncodeToString(b)
|
||||
err := s.st.Update(func(state *store.State) {
|
||||
return hex.EncodeToString(b), nil
|
||||
}
|
||||
|
||||
// Create signs user in from ip with agent, returning the cookie token.
|
||||
func (s *Sessions) Create(user, ip, agent string) (string, error) {
|
||||
tok, err := randomHex(32)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
id, err := randomHex(6)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(agent) > 200 {
|
||||
agent = agent[:200]
|
||||
}
|
||||
now := time.Now()
|
||||
err = s.st.Update(func(state *store.State) {
|
||||
pruneExpired(state)
|
||||
state.Sessions[tok] = time.Now().Add(SessionTTL)
|
||||
state.Sessions[tok] = store.Session{ID: id, User: user, Expires: now.Add(SessionTTL), Created: now, LastSeen: now, IP: ip, Agent: agent}
|
||||
})
|
||||
return tok, err
|
||||
}
|
||||
|
||||
// Valid reports whether tok is live, sliding its expiry (persisted at most hourly).
|
||||
func (s *Sessions) Valid(tok string) bool {
|
||||
// Lookup reports whether tok is live and whose it is, sliding its expiry and noting the
|
||||
// visit (persisted at most hourly, so a busy console is not writing the file all day).
|
||||
func (s *Sessions) Lookup(tok, ip string) (store.Session, bool) {
|
||||
if tok == "" {
|
||||
return false
|
||||
return store.Session{}, false
|
||||
}
|
||||
ok := false
|
||||
extend := false
|
||||
var ses store.Session
|
||||
ok, touch := false, false
|
||||
s.st.View(func(state *store.State) {
|
||||
exp, found := state.Sessions[tok]
|
||||
if found && exp.After(time.Now()) {
|
||||
ok = true
|
||||
extend = time.Until(exp) < SessionTTL-time.Hour
|
||||
found, have := state.Sessions[tok]
|
||||
if have && live(found, time.Now()) {
|
||||
ses, ok = found, true
|
||||
touch = time.Until(found.Expires) < SessionTTL-time.Hour || time.Since(found.LastSeen) > time.Hour
|
||||
}
|
||||
})
|
||||
if ok && extend {
|
||||
if ok && touch {
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
if _, found := state.Sessions[tok]; found {
|
||||
state.Sessions[tok] = time.Now().Add(SessionTTL)
|
||||
if x, found := state.Sessions[tok]; found {
|
||||
x.Expires = time.Now().Add(SessionTTL)
|
||||
x.LastSeen = time.Now()
|
||||
if ip != "" {
|
||||
x.IP = ip
|
||||
}
|
||||
state.Sessions[tok] = x
|
||||
}
|
||||
})
|
||||
}
|
||||
return ok
|
||||
return ses, ok
|
||||
}
|
||||
|
||||
func (s *Sessions) Delete(tok string) {
|
||||
_ = s.st.Update(func(state *store.State) { delete(state.Sessions, tok) })
|
||||
}
|
||||
|
||||
// Limiter is a single global backoff: after 5 failures, each further failure
|
||||
// doubles a lockout starting at 30s, capped at 10 minutes. Single-user box.
|
||||
// List is every live session of user, newest first.
|
||||
func (s *Sessions) List(user string) []store.Session {
|
||||
var out []store.Session
|
||||
s.st.View(func(state *store.State) {
|
||||
now := time.Now()
|
||||
for _, ses := range state.Sessions {
|
||||
if ses.User == user && live(ses, now) {
|
||||
out = append(out, ses)
|
||||
}
|
||||
}
|
||||
})
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].LastSeen.After(out[j].LastSeen) })
|
||||
return out
|
||||
}
|
||||
|
||||
// Count is how many live sessions each user has.
|
||||
func (s *Sessions) Count() map[string]int {
|
||||
n := map[string]int{}
|
||||
s.st.View(func(state *store.State) {
|
||||
now := time.Now()
|
||||
for _, ses := range state.Sessions {
|
||||
if live(ses, now) {
|
||||
n[ses.User]++
|
||||
}
|
||||
}
|
||||
})
|
||||
return n
|
||||
}
|
||||
|
||||
// Revoke ends the session of user with this ID. Reports whether there was one.
|
||||
func (s *Sessions) Revoke(user, id string) bool {
|
||||
found := false
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
for tok, ses := range state.Sessions {
|
||||
if ses.User == user && ses.ID == id {
|
||||
delete(state.Sessions, tok)
|
||||
found = true
|
||||
}
|
||||
}
|
||||
})
|
||||
return found
|
||||
}
|
||||
|
||||
// RevokeOthers ends every session of user except the one holding keep. Returns how many.
|
||||
func (s *Sessions) RevokeOthers(user, keep string) int {
|
||||
n := 0
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
for tok, ses := range state.Sessions {
|
||||
if ses.User == user && tok != keep {
|
||||
delete(state.Sessions, tok)
|
||||
n++
|
||||
}
|
||||
}
|
||||
})
|
||||
return n
|
||||
}
|
||||
|
||||
// RevokeIdle ends every session of user not seen for longer than idle, except the one
|
||||
// holding keep. Returns how many.
|
||||
func (s *Sessions) RevokeIdle(user, keep string, idle time.Duration) int {
|
||||
n := 0
|
||||
now := time.Now()
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
for tok, ses := range state.Sessions {
|
||||
seen := ses.LastSeen
|
||||
if seen.IsZero() {
|
||||
seen = ses.Created
|
||||
}
|
||||
if ses.User == user && tok != keep && now.Sub(seen) > idle {
|
||||
delete(state.Sessions, tok)
|
||||
n++
|
||||
}
|
||||
}
|
||||
})
|
||||
return n
|
||||
}
|
||||
|
||||
// RevokeUser ends every session of user (the account is going away).
|
||||
func (s *Sessions) RevokeUser(user string) { s.RevokeOthers(user, "") }
|
||||
|
||||
// Limiter locks out a client after repeated failed logins, per client address, with a
|
||||
// global backstop so a botnet can't burn CPU on argon2 indefinitely. A lockout for one
|
||||
// address never blocks another. Past perClientLock failures, each further failure doubles a
|
||||
// lockout starting at 30s, capped at 10 minutes.
|
||||
type Limiter struct {
|
||||
mu sync.Mutex
|
||||
mu sync.Mutex
|
||||
clients map[string]*bucket
|
||||
global bucket
|
||||
}
|
||||
|
||||
type bucket struct {
|
||||
failures int
|
||||
lockUntil time.Time
|
||||
last time.Time
|
||||
}
|
||||
|
||||
var ErrLocked = errors.New("too many failed logins")
|
||||
|
||||
func (l *Limiter) Allow() error {
|
||||
const (
|
||||
// perClientLock is how many failures one address gets before it is made to wait. Set high
|
||||
// on purpose: a guest login is a generated password read off a message and typed by hand
|
||||
// on a phone, and being locked out for fumbling it is worse than the brute-force risk —
|
||||
// which the doubling backoff past this point, and the global backstop, still cover.
|
||||
perClientLock = 20
|
||||
globalLock = 100 // failures (all clients) in the window before everyone waits a minute
|
||||
globalWindow = 5 * time.Minute
|
||||
)
|
||||
|
||||
func (l *Limiter) get(key string) *bucket {
|
||||
if l.clients == nil {
|
||||
l.clients = map[string]*bucket{}
|
||||
}
|
||||
b, ok := l.clients[key]
|
||||
if !ok {
|
||||
b = &bucket{}
|
||||
l.clients[key] = b
|
||||
}
|
||||
// forget clients that have been quiet for an hour
|
||||
if len(l.clients) > 256 {
|
||||
for k, x := range l.clients {
|
||||
if time.Since(x.last) > time.Hour && time.Now().After(x.lockUntil) {
|
||||
delete(l.clients, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func (l *Limiter) Allow(key string) error {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if time.Now().Before(l.lockUntil) {
|
||||
now := time.Now()
|
||||
if now.Before(l.global.lockUntil) {
|
||||
return ErrLocked
|
||||
}
|
||||
if now.Sub(l.global.last) > globalWindow {
|
||||
l.global.failures = 0
|
||||
}
|
||||
if b := l.get(key); now.Before(b.lockUntil) {
|
||||
return ErrLocked
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (l *Limiter) Fail() {
|
||||
func (l *Limiter) Fail(key string) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.failures++
|
||||
if l.failures >= 5 {
|
||||
d := 30 * time.Second << uint(l.failures-5)
|
||||
now := time.Now()
|
||||
b := l.get(key)
|
||||
b.failures++
|
||||
b.last = now
|
||||
if b.failures >= perClientLock {
|
||||
d := 30 * time.Second << uint(b.failures-perClientLock)
|
||||
if d > 10*time.Minute {
|
||||
d = 10 * time.Minute
|
||||
}
|
||||
l.lockUntil = time.Now().Add(d)
|
||||
b.lockUntil = now.Add(d)
|
||||
}
|
||||
if now.Sub(l.global.last) > globalWindow {
|
||||
l.global.failures = 0
|
||||
}
|
||||
l.global.failures++
|
||||
l.global.last = now
|
||||
if l.global.failures >= globalLock {
|
||||
l.global.lockUntil = now.Add(time.Minute)
|
||||
l.global.failures = 0
|
||||
}
|
||||
}
|
||||
|
||||
func (l *Limiter) Reset() {
|
||||
func (l *Limiter) Reset(key string) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.failures = 0
|
||||
l.lockUntil = time.Time{}
|
||||
delete(l.clients, key)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
// Package compose runs docker compose the way a stack was deployed — which is not always
|
||||
// simply "on this host". A stack deployed by an orchestrator running inside its own
|
||||
// container records paths in that container's filesystem, so compose has to run in there.
|
||||
// Both the console and the scheduled updates it writes must invoke it identically, or a
|
||||
// redeploy from one would not match the other's labels. Compose always drives this
|
||||
// machine's own Docker: a stack's files and its containers are on the same box.
|
||||
package compose
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// Spec is everything needed to run compose for one stack.
|
||||
type Spec struct {
|
||||
Project string // -p
|
||||
Files []string // -f, in order
|
||||
Via string // a container to run compose inside; "" is this host
|
||||
Tool string // "docker" or "podman"
|
||||
}
|
||||
|
||||
// Cmd builds the command. args are compose's own, e.g. "pull" or "up", "-d".
|
||||
func Cmd(ctx context.Context, s Spec, args ...string) *exec.Cmd {
|
||||
tool := s.Tool
|
||||
if tool == "" {
|
||||
tool = "docker"
|
||||
}
|
||||
base := []string{"compose", "-p", s.Project}
|
||||
for _, f := range s.Files {
|
||||
base = append(base, "-f", f)
|
||||
}
|
||||
if s.Via != "" {
|
||||
// The CLI inside the orchestrator's container is its own `docker`, whatever this host runs.
|
||||
return exec.CommandContext(ctx, tool, append(append([]string{"exec", s.Via, "docker"}, base...), args...)...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, tool, append(base, args...)...)
|
||||
if len(s.Files) > 0 {
|
||||
cmd.Dir = filepath.Dir(s.Files[0])
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
+171
-7
@@ -7,13 +7,127 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// Token is an API token another Ham Lab (a hub) uses to manage this host. Only the
|
||||
// SHA-256 of the token is kept.
|
||||
type Token struct {
|
||||
Name string `json:"name"`
|
||||
Hash string `json:"hash"`
|
||||
Created time.Time `json:"created"`
|
||||
// Scope is what the token may do. "" is everything (a hub's token: full control of
|
||||
// this machine). ScopeSensorsRead only reads sensors — what a dashboard gets.
|
||||
Scope string `json:"scope,omitempty"`
|
||||
// Sensors narrows a sensors:read token to the IDs these patterns match. Empty is all.
|
||||
Sensors []string `json:"sensors,omitempty"`
|
||||
// IssuedBy is the account that made the token (or paired the hub that holds it).
|
||||
IssuedBy string `json:"issued_by,omitempty"`
|
||||
}
|
||||
|
||||
const ScopeSensorsRead = "sensors:read"
|
||||
|
||||
// Node is a remote Ham Lab this host (as hub) manages. The token is the one that
|
||||
// node issued to us; the fingerprint pins its self-signed certificate.
|
||||
type Node struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
Fingerprint string `json:"fingerprint,omitempty"`
|
||||
Token string `json:"token"`
|
||||
Added time.Time `json:"added"`
|
||||
}
|
||||
|
||||
// User is an account on this machine's console. Only the argon2id hash is kept.
|
||||
type User struct {
|
||||
Name string `json:"name"`
|
||||
Hash string `json:"hash"`
|
||||
Created time.Time `json:"created"`
|
||||
LastSeen time.Time `json:"last_seen,omitempty"`
|
||||
// Guest marks a temporary login made to show somebody the console: it may only read,
|
||||
// it expires by itself at Expires, and it is not one of the machine's accounts — it is
|
||||
// left out of the account list, does not count towards "the last account", and cannot
|
||||
// be the one an empty user name means. See internal/api/guests.go.
|
||||
Guest bool `json:"guest,omitempty"`
|
||||
// Expires is when a guest login stops working. A guest without one has already
|
||||
// expired, which is the safe way to read a half-written file.
|
||||
Expires time.Time `json:"expires,omitempty"`
|
||||
// IssuedBy is the account that made this guest login.
|
||||
IssuedBy string `json:"issued_by,omitempty"`
|
||||
}
|
||||
|
||||
// Expired reports whether a guest login's time is up. A real account never expires; a
|
||||
// guest with no expiry at all counts as expired rather than as forever.
|
||||
func (u User) Expired(now time.Time) bool {
|
||||
return u.Guest && (u.Expires.IsZero() || now.After(u.Expires))
|
||||
}
|
||||
|
||||
// UpdateSource is where this machine looks for a newer Ham Lab, and how often. The field
|
||||
// is Updates, not Update, because Config already has an Update method.
|
||||
type UpdateSource struct {
|
||||
// URL is a Gitea/Forgejo repo API base or a static directory; empty means the built-in
|
||||
// default, and "off" means never look.
|
||||
URL string `json:"url,omitempty"`
|
||||
// Token is sent when the source is behind a login, so a private Gitea can serve its
|
||||
// own fleet without the instance being opened to everyone.
|
||||
Token string `json:"token,omitempty"`
|
||||
// Check is "daily" (the default) or "never".
|
||||
Check string `json:"check,omitempty"`
|
||||
}
|
||||
|
||||
// KVM is an IP-KVM: a device with a web page that gives screen, keyboard and mouse access
|
||||
// to a machine, including when that machine has no OS running. Ham Lab holds the list on
|
||||
// the hub and proxies each one, because a KVM matters exactly when the machine it is
|
||||
// attached to cannot answer for itself.
|
||||
type KVM struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
// URL is the device's own address, as you would type it into a browser on the LAN.
|
||||
URL string `json:"url"`
|
||||
// Machine is the host this KVM is attached to, and is the key: one KVM per machine, shown
|
||||
// on that machine's KVM tab. Empty means the hub itself. The list lives on the hub even
|
||||
// for a node's KVM, because a KVM earns its keep when the machine is down and cannot
|
||||
// answer for itself.
|
||||
Machine string `json:"machine"`
|
||||
Added time.Time `json:"added"`
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
Listen string `json:"listen"`
|
||||
PasswordHash string `json:"password_hash"`
|
||||
Listen string `json:"listen"`
|
||||
// PasswordHash is the single login of installs before users existed. On load it becomes
|
||||
// the first user (see MigrateUsers) and is then dropped from the file.
|
||||
PasswordHash string `json:"password_hash,omitempty"`
|
||||
Users []User `json:"users,omitempty"`
|
||||
TLS string `json:"tls,omitempty"` // "auto" → self-signed cert in the data dir; "" → plain HTTP
|
||||
Tokens []Token `json:"tokens,omitempty"`
|
||||
Nodes []Node `json:"nodes,omitempty"`
|
||||
KVMs []KVM `json:"kvms,omitempty"`
|
||||
// Updates: where a newer build comes from. See UpdateSource.
|
||||
Updates UpdateSource `json:"update,omitempty"`
|
||||
|
||||
path string
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// Candidates lists where config.json may live under a home directory, preferred first:
|
||||
// the XDG location, then ~/apps/hamlab, which installs before v0.2 used.
|
||||
func Candidates(home string) []string {
|
||||
return []string{filepath.Join(home, ".config", "hamlab", "config.json"), filepath.Join(home, "apps", "hamlab", "config.json")}
|
||||
}
|
||||
|
||||
// DefaultPath is the config used when neither -config nor $HAMLAB_CONFIG says otherwise:
|
||||
// whichever candidate exists, else the preferred one.
|
||||
func DefaultPath() string {
|
||||
home, _ := os.UserHomeDir()
|
||||
c := Candidates(home)
|
||||
for _, p := range c {
|
||||
if _, err := os.Stat(p); err == nil {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return c[0]
|
||||
}
|
||||
|
||||
func Default(path string) *Config {
|
||||
@@ -36,10 +150,49 @@ func Load(path string) (*Config, error) {
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// MigrateUsers turns the pre-users single password into the first account, named after
|
||||
// the OS user the console runs as. Reports whether anything changed (the caller saves).
|
||||
func (c *Config) MigrateUsers(name string) bool {
|
||||
if len(c.Users) > 0 || c.PasswordHash == "" {
|
||||
return false
|
||||
}
|
||||
c.Users = []User{{Name: name, Hash: c.PasswordHash, Created: time.Now()}}
|
||||
c.PasswordHash = ""
|
||||
return true
|
||||
}
|
||||
|
||||
// Accounts is every real account: the machine's users, without the guest logins.
|
||||
// Call under View/Update or on a config nobody else holds.
|
||||
func (c *Config) Accounts() []User {
|
||||
out := make([]User, 0, len(c.Users))
|
||||
for _, u := range c.Users {
|
||||
if !u.Guest {
|
||||
out = append(out, u)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// User finds an account by name. Call under View/Update or on a config nobody else holds.
|
||||
func (c *Config) User(name string) (User, bool) {
|
||||
for _, u := range c.Users {
|
||||
if u.Name == name {
|
||||
return u, true
|
||||
}
|
||||
}
|
||||
return User{}, false
|
||||
}
|
||||
|
||||
func (c *Config) Path() string { return c.path }
|
||||
func (c *Config) DataDir() string { return filepath.Dir(c.path) }
|
||||
|
||||
func (c *Config) Save() error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.save()
|
||||
}
|
||||
|
||||
func (c *Config) save() error {
|
||||
if err := os.MkdirAll(c.DataDir(), 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -47,9 +200,20 @@ func (c *Config) Save() error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := c.path + ".tmp"
|
||||
if err := os.WriteFile(tmp, append(b, '\n'), 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, c.path)
|
||||
return store.WriteFileAtomic(c.path, append(b, '\n'), 0o600)
|
||||
}
|
||||
|
||||
// Update applies fn under the lock and saves.
|
||||
func (c *Config) Update(fn func(*Config)) error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
fn(c)
|
||||
return c.save()
|
||||
}
|
||||
|
||||
// View reads under the lock.
|
||||
func (c *Config) View(fn func(*Config)) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
fn(c)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,813 @@
|
||||
// Package cron reads and writes what the UI calls the **Jobs** tab (named "Cron" until
|
||||
// 2026-09-24; the package and its /api/cron endpoints kept the old name).
|
||||
//
|
||||
// Package cron reads every scheduled job on the host into one list: /etc/crontab,
|
||||
// /etc/cron.d/*, per-user crontabs, the run-parts directories they drive, and systemd
|
||||
// timers. It computes next runs from the cron expressions, pulls last runs from the
|
||||
// journal (cron logs "(user) CMD (command)" via syslog), and can toggle, edit and run
|
||||
// jobs. Reads/writes that need root go through passwordless sudo when available.
|
||||
package cron
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha1"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type Job struct {
|
||||
ID string `json:"id"`
|
||||
Source string `json:"source"` // "/etc/crontab", "/etc/cron.d/x", "crontab:user", "/etc/cron.daily"
|
||||
Line int `json:"line"` // 1-based line in Source (0 for run-parts scripts)
|
||||
User string `json:"user"`
|
||||
Schedule string `json:"schedule"`
|
||||
Command string `json:"command"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Next int64 `json:"next,omitempty"` // unix ms
|
||||
Last int64 `json:"last,omitempty"`
|
||||
OutFile string `json:"out_file,omitempty"` // parsed from a ">> file" redirect
|
||||
Note string `json:"note,omitempty"`
|
||||
Editable bool `json:"editable"`
|
||||
raw string
|
||||
}
|
||||
|
||||
type Timer struct {
|
||||
Unit string `json:"unit"`
|
||||
Activates string `json:"activates"`
|
||||
Description string `json:"description"`
|
||||
Active string `json:"active"`
|
||||
Sub string `json:"sub"`
|
||||
Enabled string `json:"enabled"`
|
||||
Next int64 `json:"next,omitempty"`
|
||||
Last int64 `json:"last,omitempty"`
|
||||
}
|
||||
|
||||
type Overview struct {
|
||||
Jobs []Job `json:"jobs"`
|
||||
Timers []Timer `json:"timers"`
|
||||
Logging bool `json:"logging"` // cron lines seen in the journal
|
||||
Anacron bool `json:"anacron"`
|
||||
Sudo bool `json:"sudo"`
|
||||
}
|
||||
|
||||
type Run struct {
|
||||
ID string `json:"id"`
|
||||
JobID string `json:"job_id"`
|
||||
Command string `json:"command"`
|
||||
User string `json:"user"`
|
||||
Started time.Time `json:"started"`
|
||||
Finished time.Time `json:"finished,omitempty"`
|
||||
Running bool `json:"running"`
|
||||
ExitCode int `json:"exit_code"`
|
||||
Output string `json:"output"`
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
sudo bool
|
||||
journalSudo bool
|
||||
self string
|
||||
mu sync.Mutex
|
||||
lastRuns map[string]int64 // user\x00command -> unix ms
|
||||
lastAt time.Time
|
||||
lastCursor string // journal cursor of the newest entry folded into lastRuns
|
||||
logging bool
|
||||
runs map[string]*Run
|
||||
}
|
||||
|
||||
func New(sudo, journalSudo bool) *Manager {
|
||||
m := &Manager{sudo: sudo, journalSudo: journalSudo, runs: map[string]*Run{}}
|
||||
if u, err := user.Current(); err == nil {
|
||||
m.self = u.Username
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ---- helpers ----
|
||||
|
||||
func run(ctx context.Context, stdin string, args ...string) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
if stdin != "" {
|
||||
cmd.Stdin = strings.NewReader(stdin)
|
||||
}
|
||||
var out, errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := strings.TrimSpace(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return out.String(), errors.New(msg)
|
||||
}
|
||||
return out.String(), nil
|
||||
}
|
||||
|
||||
// runLines streams a command's stdout to fn line by line, so multi-megabyte journal
|
||||
// dumps never sit in memory whole.
|
||||
func runLines(ctx context.Context, fn func(line []byte), args ...string) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
}
|
||||
sc := bufio.NewScanner(stdout)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
|
||||
for sc.Scan() {
|
||||
fn(sc.Bytes())
|
||||
}
|
||||
if err := cmd.Wait(); err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manager) sudoArgs(args ...string) []string {
|
||||
if m.sudo {
|
||||
return priv.Wrap(args...)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
func jobID(source, raw string) string {
|
||||
h := sha1.Sum([]byte(source + "\n" + raw))
|
||||
return hex.EncodeToString(h[:6])
|
||||
}
|
||||
|
||||
var userRe = regexp.MustCompile(`^[a-z_][a-z0-9_-]{0,31}$`)
|
||||
var cronDName = regexp.MustCompile(`^[A-Za-z0-9_-]+$`) // cron ignores other names in /etc/cron.d
|
||||
|
||||
// ---- parsing ----
|
||||
|
||||
// parseLine splits one crontab line. withUser is true for system crontabs (6th field is
|
||||
// the user). Returns ok=false for comments, blanks, env assignments and garbage.
|
||||
func parseLine(line string, withUser bool) (schedule, usr, command string, ok bool) {
|
||||
t := strings.TrimSpace(line)
|
||||
if t == "" || strings.HasPrefix(t, "#") {
|
||||
return
|
||||
}
|
||||
if i := strings.Index(t, "="); i > 0 && !strings.ContainsAny(t[:i], " \t*") { // NAME=value
|
||||
return
|
||||
}
|
||||
var fields []string
|
||||
if strings.HasPrefix(t, "@") {
|
||||
f := strings.Fields(t)
|
||||
if len(f) < 2 {
|
||||
return
|
||||
}
|
||||
fields = []string{f[0]}
|
||||
t = strings.TrimSpace(t[len(f[0]):])
|
||||
} else {
|
||||
f := strings.Fields(t)
|
||||
if len(f) < 6 {
|
||||
return
|
||||
}
|
||||
fields = f[:5]
|
||||
for _, x := range fields {
|
||||
t = strings.TrimSpace(t[len(x):])
|
||||
}
|
||||
}
|
||||
schedule = strings.Join(fields, " ")
|
||||
if _, err := parseSchedule(schedule); err != nil {
|
||||
return
|
||||
}
|
||||
if withUser {
|
||||
f := strings.Fields(t)
|
||||
if len(f) < 2 || !userRe.MatchString(f[0]) {
|
||||
return
|
||||
}
|
||||
usr = f[0]
|
||||
t = strings.TrimSpace(t[len(f[0]):])
|
||||
}
|
||||
if t == "" {
|
||||
return
|
||||
}
|
||||
return schedule, usr, t, true
|
||||
}
|
||||
|
||||
var redirectRe = regexp.MustCompile(`(?:^|\s)(?:>>|>|2>&1\s*>>|&>>|&>)\s*("?)([^\s"&|;]+)`)
|
||||
|
||||
func outFile(cmd string) string {
|
||||
for _, m := range redirectRe.FindAllStringSubmatch(cmd, -1) {
|
||||
if p := m[2]; strings.HasPrefix(p, "/") && p != "/dev/null" {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// parseSource turns crontab text into jobs. Commented-out lines that still parse as a
|
||||
// job are reported disabled, so a "# 0 5 * * * ..." example shows up as such.
|
||||
func (m *Manager) parseSource(source, text string, withUser, editable bool, defaultUser string, now time.Time, anacron bool) []Job {
|
||||
var jobs []Job
|
||||
lines := strings.Split(text, "\n")
|
||||
isComment := func(i int) bool {
|
||||
return i >= 0 && i < len(lines) && strings.HasPrefix(strings.TrimSpace(lines[i]), "#")
|
||||
}
|
||||
for i, line := range lines {
|
||||
enabled := true
|
||||
probe := line
|
||||
if t := strings.TrimSpace(line); strings.HasPrefix(t, "#") {
|
||||
// A commented-out job sits on its own; a job-shaped line inside a block of
|
||||
// comments is documentation (Debian's stock header has two such examples).
|
||||
if isComment(i-1) && isComment(i+1) {
|
||||
continue
|
||||
}
|
||||
enabled = false
|
||||
probe = strings.TrimLeft(t, "# ")
|
||||
}
|
||||
schedule, usr, command, ok := parseLine(probe, withUser)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if usr == "" {
|
||||
usr = defaultUser
|
||||
}
|
||||
if !enabled {
|
||||
if _, err := user.Lookup(usr); err != nil {
|
||||
continue // "user-name" placeholder and the like
|
||||
}
|
||||
}
|
||||
j := Job{ID: jobID(source, line), Source: source, Line: i + 1, User: usr, Schedule: schedule, Command: command,
|
||||
Enabled: enabled, OutFile: outFile(command), Editable: editable, raw: line}
|
||||
if enabled {
|
||||
if next, ok := nextRun(schedule, now); ok {
|
||||
j.Next = next.UnixMilli()
|
||||
}
|
||||
}
|
||||
if strings.Contains(command, "run-parts") {
|
||||
if dir := runPartsDir(command); dir != "" {
|
||||
if anacron && strings.Contains(command, "anacron") {
|
||||
j.Note = "skipped: anacron is installed and runs " + dir + " instead"
|
||||
}
|
||||
jobs = append(jobs, j)
|
||||
jobs = append(jobs, m.runPartsJobs(dir, j)...)
|
||||
continue
|
||||
}
|
||||
}
|
||||
jobs = append(jobs, j)
|
||||
}
|
||||
return jobs
|
||||
}
|
||||
|
||||
var runPartsRe = regexp.MustCompile(`run-parts\s+(?:--\S+\s+)*(/\S+)`)
|
||||
|
||||
func runPartsDir(cmd string) string {
|
||||
if m := runPartsRe.FindStringSubmatch(cmd); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// runPartsJobs lists the executable scripts a run-parts line will run, as read-only jobs.
|
||||
func (m *Manager) runPartsJobs(dir string, parent Job) []Job {
|
||||
ents, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []Job
|
||||
for _, e := range ents {
|
||||
if e.IsDir() || !cronDName.MatchString(e.Name()) {
|
||||
continue
|
||||
}
|
||||
info, err := e.Info()
|
||||
if err != nil || info.Mode()&0o111 == 0 {
|
||||
continue
|
||||
}
|
||||
p := filepath.Join(dir, e.Name())
|
||||
out = append(out, Job{ID: jobID(dir, e.Name()), Source: dir, User: parent.User, Schedule: parent.Schedule, Command: p,
|
||||
Enabled: parent.Enabled, Next: parent.Next, Note: "run by " + parent.Source + " via run-parts", raw: e.Name()})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- sources ----
|
||||
|
||||
type source struct {
|
||||
name string // job Source
|
||||
withUser bool
|
||||
user string // default user for user crontabs
|
||||
editable bool
|
||||
}
|
||||
|
||||
func (m *Manager) readSource(ctx context.Context, s source) (string, error) {
|
||||
if strings.HasPrefix(s.name, "crontab:") {
|
||||
u := s.user
|
||||
if u == m.self {
|
||||
out, err := run(ctx, "", "crontab", "-l")
|
||||
if err != nil && strings.Contains(err.Error(), "no crontab") {
|
||||
return "", nil
|
||||
}
|
||||
return out, err
|
||||
}
|
||||
if !m.sudo {
|
||||
return "", errors.New("needs sudo")
|
||||
}
|
||||
out, err := run(ctx, "", priv.Wrap("crontab", "-u", u, "-l")...)
|
||||
if err != nil && strings.Contains(err.Error(), "no crontab") {
|
||||
return "", nil
|
||||
}
|
||||
return out, err
|
||||
}
|
||||
b, err := os.ReadFile(s.name)
|
||||
if err != nil && m.sudo {
|
||||
out, err2 := run(ctx, "", priv.Wrap("cat", s.name)...)
|
||||
return out, err2
|
||||
}
|
||||
return string(b), err
|
||||
}
|
||||
|
||||
func (m *Manager) sources(ctx context.Context) []source {
|
||||
var out []source
|
||||
if _, err := os.Stat("/etc/crontab"); err == nil {
|
||||
out = append(out, source{name: "/etc/crontab", withUser: true, editable: m.sudo})
|
||||
}
|
||||
if ents, err := os.ReadDir("/etc/cron.d"); err == nil {
|
||||
for _, e := range ents {
|
||||
if !e.IsDir() && cronDName.MatchString(e.Name()) {
|
||||
out = append(out, source{name: "/etc/cron.d/" + e.Name(), withUser: true, editable: m.sudo})
|
||||
}
|
||||
}
|
||||
}
|
||||
// per-user crontabs: Debian keeps them in /var/spool/cron/crontabs, cronie in /var/spool/cron
|
||||
users := map[string]bool{}
|
||||
if m.sudo {
|
||||
for _, dir := range []string{"/var/spool/cron/crontabs", "/var/spool/cron"} {
|
||||
if out, err := run(ctx, "", priv.Wrap("ls", "-1", dir)...); err == nil {
|
||||
for _, n := range strings.Split(out, "\n") {
|
||||
if n = strings.TrimSpace(n); userRe.MatchString(n) {
|
||||
users[n] = true
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if m.self != "" {
|
||||
users[m.self] = true
|
||||
}
|
||||
names := make([]string, 0, len(users))
|
||||
for u := range users {
|
||||
names = append(names, u)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, u := range names {
|
||||
out = append(out, source{name: "crontab:" + u, user: u, editable: u == m.self || m.sudo})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) findSource(ctx context.Context, name string) (source, bool) {
|
||||
for _, s := range m.sources(ctx) {
|
||||
if s.name == name {
|
||||
return s, true
|
||||
}
|
||||
}
|
||||
return source{}, false
|
||||
}
|
||||
|
||||
// ---- overview ----
|
||||
|
||||
func (m *Manager) Overview(ctx context.Context) Overview {
|
||||
now := time.Now()
|
||||
_, anacronErr := os.Stat("/usr/sbin/anacron")
|
||||
anacron := anacronErr == nil
|
||||
ov := Overview{Jobs: []Job{}, Timers: []Timer{}, Anacron: anacron, Sudo: m.sudo}
|
||||
for _, s := range m.sources(ctx) {
|
||||
text, err := m.readSource(ctx, s)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
ov.Jobs = append(ov.Jobs, m.parseSource(s.name, text, s.withUser, s.editable, s.user, now, anacron)...)
|
||||
}
|
||||
last := m.lastRunsFromJournal(ctx)
|
||||
ov.Logging = m.logging
|
||||
for i := range ov.Jobs {
|
||||
j := &ov.Jobs[i]
|
||||
if t, ok := last[j.User+"\x00"+j.Command]; ok {
|
||||
j.Last = t
|
||||
}
|
||||
}
|
||||
ov.Timers = m.timers(ctx)
|
||||
return ov
|
||||
}
|
||||
|
||||
var cronLogRe = regexp.MustCompile(`^\((\S+)\) CMD \((.*)\)$`)
|
||||
|
||||
// lastRunsFromJournal maps user+command to the most recent "(user) CMD (command)" line.
|
||||
// The first call scans 30 days; later calls only read entries after the last cursor, so
|
||||
// the cost per refresh is proportional to new cron activity, not to history.
|
||||
func (m *Manager) lastRunsFromJournal(ctx context.Context) map[string]int64 {
|
||||
m.mu.Lock()
|
||||
if time.Since(m.lastAt) < 2*time.Second && m.lastRuns != nil {
|
||||
r := m.lastRuns
|
||||
m.mu.Unlock()
|
||||
return r
|
||||
}
|
||||
cursor := m.lastCursor
|
||||
res := map[string]int64{}
|
||||
for k, v := range m.lastRuns {
|
||||
res[k] = v
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
args := []string{"journalctl", "-t", "CRON", "-t", "cron", "-t", "crond", "-o", "json", "--no-pager", "-q"}
|
||||
if cursor != "" {
|
||||
args = append(args, "--after-cursor="+cursor)
|
||||
} else {
|
||||
args = append(args, "--since", "-30d", "-n", "20000")
|
||||
}
|
||||
if m.journalSudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
seen := m.logging
|
||||
_ = runLines(ctx, func(line []byte) {
|
||||
var raw struct {
|
||||
Msg any `json:"MESSAGE"`
|
||||
TS string `json:"__REALTIME_TIMESTAMP"`
|
||||
Cursor string `json:"__CURSOR"`
|
||||
}
|
||||
if json.Unmarshal(line, &raw) != nil {
|
||||
return
|
||||
}
|
||||
seen = true
|
||||
if raw.Cursor != "" {
|
||||
cursor = raw.Cursor
|
||||
}
|
||||
msg, _ := raw.Msg.(string)
|
||||
mm := cronLogRe.FindStringSubmatch(msg)
|
||||
if mm == nil {
|
||||
return
|
||||
}
|
||||
us, _ := strconv.ParseInt(raw.TS, 10, 64)
|
||||
k := mm[1] + "\x00" + mm[2]
|
||||
if us/1000 > res[k] {
|
||||
res[k] = us / 1000
|
||||
}
|
||||
}, args...)
|
||||
m.mu.Lock()
|
||||
m.lastRuns, m.lastAt, m.lastCursor, m.logging = res, time.Now(), cursor, seen
|
||||
m.mu.Unlock()
|
||||
return res
|
||||
}
|
||||
|
||||
// JobLogs returns the journal lines for one job's command (most recent last).
|
||||
func (m *Manager) JobLogs(ctx context.Context, j Job, n int) ([]map[string]any, error) {
|
||||
args := []string{"journalctl", "-t", "CRON", "-t", "cron", "-t", "crond", "-o", "json", "--no-pager", "-q", "-n", "20000"}
|
||||
if m.journalSudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
var res []map[string]any
|
||||
needle := "(" + j.User + ") CMD (" + j.Command + ")"
|
||||
err := runLines(ctx, func(line []byte) {
|
||||
if !bytes.Contains(line, []byte(j.Command[:min(len(j.Command), 40)])) { // cheap pre-filter before JSON
|
||||
return
|
||||
}
|
||||
var raw struct {
|
||||
Msg any `json:"MESSAGE"`
|
||||
TS string `json:"__REALTIME_TIMESTAMP"`
|
||||
PID string `json:"_PID"`
|
||||
}
|
||||
if json.Unmarshal(line, &raw) != nil {
|
||||
return
|
||||
}
|
||||
msg, _ := raw.Msg.(string)
|
||||
if !strings.Contains(msg, needle) {
|
||||
return
|
||||
}
|
||||
us, _ := strconv.ParseInt(raw.TS, 10, 64)
|
||||
res = append(res, map[string]any{"t": us / 1000, "p": 6, "ident": "CRON", "pid": raw.PID, "msg": msg})
|
||||
}, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(res) > n {
|
||||
res = res[len(res)-n:]
|
||||
}
|
||||
if res == nil {
|
||||
res = []map[string]any{}
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// OutputTail returns the last n lines of the job's redirect target.
|
||||
func (m *Manager) OutputTail(ctx context.Context, j Job, n int) (string, error) {
|
||||
if j.OutFile == "" {
|
||||
return "", errors.New("job has no output file")
|
||||
}
|
||||
out, err := run(ctx, "", m.sudoArgs("tail", "-n", strconv.Itoa(n), j.OutFile)...)
|
||||
return out, err
|
||||
}
|
||||
|
||||
// ---- timers ----
|
||||
|
||||
// timers: one `systemctl show` over every loaded timer instead of three list commands.
|
||||
func (m *Manager) timers(ctx context.Context) []Timer {
|
||||
out, err := run(ctx, "", "systemctl", "show", "*.timer", "--no-pager",
|
||||
"-p", "Id,Description,ActiveState,SubState,UnitFileState,NextElapseUSecRealtime,LastTriggerUSec,Triggers")
|
||||
if err != nil {
|
||||
return []Timer{}
|
||||
}
|
||||
unix := func(v string) int64 { // "Sat 2026-09-19 05:47:13 PDT", "n/a" or ""
|
||||
for _, layout := range []string{"Mon 2006-01-02 15:04:05 MST", "2006-01-02 15:04:05 MST", "Mon 2006-01-02 15:04:05"} {
|
||||
if t, err := time.ParseInLocation(layout, v, time.Local); err == nil {
|
||||
return t.UnixMilli()
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
res := []Timer{}
|
||||
for _, block := range strings.Split(out, "\n\n") {
|
||||
t := Timer{}
|
||||
for _, line := range strings.Split(block, "\n") {
|
||||
k, v, ok := strings.Cut(line, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
switch k {
|
||||
case "Id":
|
||||
t.Unit = v
|
||||
case "Description":
|
||||
t.Description = v
|
||||
case "ActiveState":
|
||||
t.Active = v
|
||||
case "SubState":
|
||||
t.Sub = v
|
||||
case "UnitFileState":
|
||||
t.Enabled = v
|
||||
case "NextElapseUSecRealtime":
|
||||
t.Next = unix(v)
|
||||
case "LastTriggerUSec":
|
||||
t.Last = unix(v)
|
||||
case "Triggers":
|
||||
t.Activates = strings.Fields(v + " ")[0:1][0]
|
||||
}
|
||||
}
|
||||
if t.Unit != "" {
|
||||
res = append(res, t)
|
||||
}
|
||||
}
|
||||
sort.Slice(res, func(i, j int) bool { return res[i].Unit < res[j].Unit })
|
||||
return res
|
||||
}
|
||||
|
||||
var timerRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9@._:\\-]{0,255}\.timer$`)
|
||||
|
||||
func (m *Manager) TimerAction(ctx context.Context, unit, action string) error {
|
||||
if !timerRe.MatchString(unit) {
|
||||
return errors.New("invalid timer name")
|
||||
}
|
||||
switch action {
|
||||
case "start", "stop", "enable", "disable":
|
||||
_, err := run(ctx, "", m.sudoArgs("systemctl", action, unit)...)
|
||||
return err
|
||||
case "run": // fire the service the timer activates, right now
|
||||
out, err := run(ctx, "", "systemctl", "show", unit, "-p", "Triggers", "--value")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
svc := strings.TrimSpace(strings.Split(out, " ")[0])
|
||||
if svc == "" {
|
||||
return errors.New("timer triggers nothing")
|
||||
}
|
||||
_, err = run(ctx, "", m.sudoArgs("systemctl", "start", svc)...)
|
||||
return err
|
||||
}
|
||||
return errors.New("unknown action")
|
||||
}
|
||||
|
||||
// ---- job lookup and editing ----
|
||||
|
||||
func (m *Manager) FindJob(ctx context.Context, id string) (Job, bool) {
|
||||
for _, j := range m.Overview(ctx).Jobs {
|
||||
if j.ID == id {
|
||||
return j, true
|
||||
}
|
||||
}
|
||||
return Job{}, false
|
||||
}
|
||||
|
||||
// Source returns a crontab's raw text for the editor.
|
||||
func (m *Manager) Source(ctx context.Context, name string) (text string, editable bool, err error) {
|
||||
s, ok := m.findSource(ctx, name)
|
||||
if !ok {
|
||||
return "", false, errors.New("unknown source")
|
||||
}
|
||||
text, err = m.readSource(ctx, s)
|
||||
return text, s.editable, err
|
||||
}
|
||||
|
||||
// validate rejects any line that is neither blank, comment, env assignment, nor a job.
|
||||
func validate(text string, withUser bool) error {
|
||||
for i, line := range strings.Split(text, "\n") {
|
||||
t := strings.TrimSpace(line)
|
||||
if t == "" || strings.HasPrefix(t, "#") {
|
||||
continue
|
||||
}
|
||||
if k, _, ok := strings.Cut(t, "="); ok && !strings.ContainsAny(strings.TrimSpace(k), " \t*") {
|
||||
continue
|
||||
}
|
||||
if _, _, _, ok := parseLine(t, withUser); !ok {
|
||||
return fmt.Errorf("line %d: not a valid cron entry", i+1)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// WriteSource replaces a crontab. User crontabs go through crontab(1) so cron reloads
|
||||
// them; system files are written in place with their permissions preserved.
|
||||
func (m *Manager) WriteSource(ctx context.Context, name, text string) error {
|
||||
s, ok := m.findSource(ctx, name)
|
||||
if !ok {
|
||||
return errors.New("unknown source")
|
||||
}
|
||||
if !s.editable {
|
||||
return errors.New("source is read-only here (no sudo)")
|
||||
}
|
||||
if !strings.HasSuffix(text, "\n") {
|
||||
text += "\n"
|
||||
}
|
||||
if err := validate(text, s.withUser); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.HasPrefix(name, "crontab:") {
|
||||
if s.user == m.self {
|
||||
_, err := run(ctx, text, "crontab", "-")
|
||||
return err
|
||||
}
|
||||
_, err := run(ctx, text, priv.Wrap("crontab", "-u", s.user, "-")...)
|
||||
return err
|
||||
}
|
||||
if name != "/etc/crontab" && !strings.HasPrefix(name, "/etc/cron.d/") {
|
||||
return errors.New("refusing to write outside /etc/crontab and /etc/cron.d")
|
||||
}
|
||||
_, err := run(ctx, text, priv.Wrap("tee", name)...)
|
||||
m.mu.Lock()
|
||||
m.lastAt = time.Time{}
|
||||
m.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
|
||||
// Toggle comments or uncomments a job's line. The line must still read as the job did
|
||||
// when it was listed, so a concurrent edit can't be clobbered.
|
||||
func (m *Manager) Toggle(ctx context.Context, j Job, enable bool) error {
|
||||
if j.Line == 0 || !j.Editable {
|
||||
return errors.New("this job can't be toggled here")
|
||||
}
|
||||
s, ok := m.findSource(ctx, j.Source)
|
||||
if !ok {
|
||||
return errors.New("unknown source")
|
||||
}
|
||||
text, err := m.readSource(ctx, s)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
lines := strings.Split(text, "\n")
|
||||
if j.Line > len(lines) || lines[j.Line-1] != j.raw {
|
||||
return errors.New("crontab changed underneath; reload and try again")
|
||||
}
|
||||
if enable {
|
||||
lines[j.Line-1] = strings.TrimLeft(strings.TrimSpace(j.raw), "# ")
|
||||
} else {
|
||||
lines[j.Line-1] = "#" + j.raw
|
||||
}
|
||||
return m.WriteSource(ctx, j.Source, strings.Join(lines, "\n"))
|
||||
}
|
||||
|
||||
// ---- run now ----
|
||||
|
||||
// StartRun executes the job's command as its user, the way cron would (sh -c, cwd = home,
|
||||
// '%' turned into newlines/stdin), in the background. Output is kept in memory.
|
||||
func (m *Manager) StartRun(ctx context.Context, j Job) (*Run, error) {
|
||||
if j.User != m.self && !m.sudo {
|
||||
return nil, errors.New("needs sudo to run as " + j.User)
|
||||
}
|
||||
cmdline, stdin := cronPercent(j.Command)
|
||||
args := []string{"sh", "-c", cmdline}
|
||||
if j.User != m.self {
|
||||
args = priv.AsUser(j.User, args...)
|
||||
}
|
||||
r := &Run{ID: jobID("run", strconv.FormatInt(time.Now().UnixNano(), 10)), JobID: j.ID, Command: j.Command, User: j.User, Started: time.Now(), Running: true}
|
||||
runCtx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
cmd := exec.CommandContext(runCtx, args[0], args[1:]...)
|
||||
if stdin != "" {
|
||||
cmd.Stdin = strings.NewReader(stdin)
|
||||
}
|
||||
w := &capWriter{r: r}
|
||||
cmd.Stdout, cmd.Stderr = w, w
|
||||
if err := cmd.Start(); err != nil {
|
||||
cancel()
|
||||
return nil, err
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.runs[r.ID] = r
|
||||
for id, old := range m.runs { // keep the map small
|
||||
if !old.Running && time.Since(old.Finished) > time.Hour {
|
||||
delete(m.runs, id)
|
||||
}
|
||||
}
|
||||
m.mu.Unlock()
|
||||
go func() {
|
||||
defer cancel()
|
||||
err := cmd.Wait()
|
||||
r.mu.Lock()
|
||||
r.Running, r.Finished = false, time.Now()
|
||||
if err != nil {
|
||||
if ee, ok := err.(*exec.ExitError); ok {
|
||||
r.ExitCode = ee.ExitCode()
|
||||
} else {
|
||||
r.ExitCode = -1
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
m.mu.Lock()
|
||||
m.lastAt = time.Time{}
|
||||
m.mu.Unlock()
|
||||
}()
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (m *Manager) GetRun(id string) (*Run, bool) {
|
||||
m.mu.Lock()
|
||||
r, ok := m.runs[id]
|
||||
m.mu.Unlock()
|
||||
return r, ok
|
||||
}
|
||||
|
||||
// Snapshot returns a copy safe to encode.
|
||||
func (r *Run) Snapshot() Run {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return Run{ID: r.ID, JobID: r.JobID, Command: r.Command, User: r.User, Started: r.Started, Finished: r.Finished,
|
||||
Running: r.Running, ExitCode: r.ExitCode, Output: r.buf.String()}
|
||||
}
|
||||
|
||||
type capWriter struct{ r *Run }
|
||||
|
||||
func (w *capWriter) Write(p []byte) (int, error) {
|
||||
w.r.mu.Lock()
|
||||
defer w.r.mu.Unlock()
|
||||
w.r.buf.Write(p)
|
||||
if w.r.buf.Len() > 256<<10 { // keep the tail
|
||||
b := w.r.buf.Bytes()
|
||||
w.r.buf = *bytes.NewBuffer(append([]byte("…\n"), b[len(b)-200<<10:]...))
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
// cronPercent applies crontab(5) rules: an unescaped % ends the command; later ones are
|
||||
// newlines fed to stdin. "\%" is a literal percent.
|
||||
func cronPercent(cmd string) (string, string) {
|
||||
var out, in strings.Builder
|
||||
cur := &out
|
||||
esc := false
|
||||
for _, c := range cmd {
|
||||
switch {
|
||||
case esc:
|
||||
if c != '%' {
|
||||
cur.WriteRune('\\')
|
||||
}
|
||||
cur.WriteRune(c)
|
||||
esc = false
|
||||
case c == '\\':
|
||||
esc = true
|
||||
case c == '%':
|
||||
if cur == &out {
|
||||
cur = &in
|
||||
} else {
|
||||
in.WriteRune('\n')
|
||||
}
|
||||
default:
|
||||
cur.WriteRune(c)
|
||||
}
|
||||
}
|
||||
return out.String(), in.String()
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
package cron
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// schedule is a parsed five-field cron expression as sets.
|
||||
type schedule struct {
|
||||
min, hour, dom, mon, dow [64]bool
|
||||
domStar, dowStar bool // "*" in dom/dow changes the match rule
|
||||
reboot bool
|
||||
}
|
||||
|
||||
var monthNames = map[string]int{"jan": 1, "feb": 2, "mar": 3, "apr": 4, "may": 5, "jun": 6, "jul": 7, "aug": 8, "sep": 9, "oct": 10, "nov": 11, "dec": 12}
|
||||
var dayNames = map[string]int{"sun": 0, "mon": 1, "tue": 2, "wed": 3, "thu": 4, "fri": 5, "sat": 6}
|
||||
|
||||
var aliases = map[string]string{
|
||||
"@yearly": "0 0 1 1 *", "@annually": "0 0 1 1 *", "@monthly": "0 0 1 * *", "@weekly": "0 0 * * 0",
|
||||
"@daily": "0 0 * * *", "@midnight": "0 0 * * *", "@hourly": "0 * * * *",
|
||||
}
|
||||
|
||||
func parseSchedule(expr string) (*schedule, error) {
|
||||
expr = strings.TrimSpace(expr)
|
||||
if expr == "@reboot" {
|
||||
return &schedule{reboot: true}, nil
|
||||
}
|
||||
if a, ok := aliases[strings.ToLower(expr)]; ok {
|
||||
expr = a
|
||||
}
|
||||
f := strings.Fields(expr)
|
||||
if len(f) != 5 {
|
||||
return nil, errors.New("expected 5 fields")
|
||||
}
|
||||
s := &schedule{}
|
||||
var err error
|
||||
if s.min, err = parseField(f[0], 0, 59, nil); err != nil {
|
||||
return nil, fmt.Errorf("minute: %w", err)
|
||||
}
|
||||
if s.hour, err = parseField(f[1], 0, 23, nil); err != nil {
|
||||
return nil, fmt.Errorf("hour: %w", err)
|
||||
}
|
||||
if s.dom, err = parseField(f[2], 1, 31, nil); err != nil {
|
||||
return nil, fmt.Errorf("day of month: %w", err)
|
||||
}
|
||||
if s.mon, err = parseField(f[3], 1, 12, monthNames); err != nil {
|
||||
return nil, fmt.Errorf("month: %w", err)
|
||||
}
|
||||
if s.dow, err = parseField(f[4], 0, 7, dayNames); err != nil {
|
||||
return nil, fmt.Errorf("day of week: %w", err)
|
||||
}
|
||||
if s.dow[7] { // 7 == Sunday
|
||||
s.dow[0] = true
|
||||
}
|
||||
s.domStar = f[2] == "*"
|
||||
s.dowStar = f[4] == "*"
|
||||
return s, nil
|
||||
}
|
||||
|
||||
func parseField(field string, lo, hi int, names map[string]int) ([64]bool, error) {
|
||||
var set [64]bool
|
||||
atom := func(s string) (int, error) {
|
||||
if names != nil {
|
||||
if v, ok := names[strings.ToLower(s)]; ok {
|
||||
return v, nil
|
||||
}
|
||||
}
|
||||
v, err := strconv.Atoi(s)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("bad value %q", s)
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
for _, part := range strings.Split(field, ",") {
|
||||
step := 1
|
||||
if i := strings.Index(part, "/"); i >= 0 {
|
||||
v, err := strconv.Atoi(part[i+1:])
|
||||
if err != nil || v <= 0 {
|
||||
return set, fmt.Errorf("bad step in %q", part)
|
||||
}
|
||||
step, part = v, part[:i]
|
||||
}
|
||||
a, b := lo, hi
|
||||
switch {
|
||||
case part == "*":
|
||||
case strings.Contains(part, "-"):
|
||||
x, y, _ := strings.Cut(part, "-")
|
||||
var err error
|
||||
if a, err = atom(x); err != nil {
|
||||
return set, err
|
||||
}
|
||||
if b, err = atom(y); err != nil {
|
||||
return set, err
|
||||
}
|
||||
default:
|
||||
v, err := atom(part)
|
||||
if err != nil {
|
||||
return set, err
|
||||
}
|
||||
a = v
|
||||
if step == 1 {
|
||||
b = v
|
||||
}
|
||||
}
|
||||
if a < lo || b > hi || a > b {
|
||||
return set, fmt.Errorf("out of range: %q", part)
|
||||
}
|
||||
for v := a; v <= b; v += step {
|
||||
set[v] = true
|
||||
}
|
||||
}
|
||||
return set, nil
|
||||
}
|
||||
|
||||
func (s *schedule) dayMatches(t time.Time) bool {
|
||||
if !s.mon[int(t.Month())] {
|
||||
return false
|
||||
}
|
||||
dom, dow := s.dom[t.Day()], s.dow[int(t.Weekday())]
|
||||
switch {
|
||||
case s.domStar && s.dowStar:
|
||||
return true
|
||||
case s.domStar:
|
||||
return dow
|
||||
case s.dowStar:
|
||||
return dom
|
||||
default: // both restricted: cron runs when either matches
|
||||
return dom || dow
|
||||
}
|
||||
}
|
||||
|
||||
// NextRun is the first time strictly after `from` that a cron expression fires. Exported so
|
||||
// anything that writes a cron line — the ZFS snapshot schedules — can say when it runs next
|
||||
// using the same evaluator that reads everyone else's.
|
||||
func NextRun(expr string, from time.Time) (time.Time, bool) { return nextRun(expr, from) }
|
||||
|
||||
// ValidSchedule reports whether a cron expression parses.
|
||||
func ValidSchedule(expr string) error {
|
||||
_, err := parseSchedule(expr)
|
||||
return err
|
||||
}
|
||||
|
||||
// nextRun finds the first time strictly after `from` that the schedule fires.
|
||||
func nextRun(expr string, from time.Time) (time.Time, bool) {
|
||||
s, err := parseSchedule(expr)
|
||||
if err != nil || s.reboot {
|
||||
return time.Time{}, false
|
||||
}
|
||||
t := from.Truncate(time.Minute).Add(time.Minute)
|
||||
day := time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, t.Location())
|
||||
for i := 0; i < 366*2; i++ {
|
||||
if s.dayMatches(day) {
|
||||
for h := 0; h < 24; h++ {
|
||||
if !s.hour[h] {
|
||||
continue
|
||||
}
|
||||
for mi := 0; mi < 60; mi++ {
|
||||
if !s.min[mi] {
|
||||
continue
|
||||
}
|
||||
c := time.Date(day.Year(), day.Month(), day.Day(), h, mi, 0, 0, day.Location())
|
||||
if !c.Before(t) {
|
||||
return c, true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
day = day.AddDate(0, 0, 1)
|
||||
}
|
||||
return time.Time{}, false
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package cron
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestNextRun(t *testing.T) {
|
||||
loc := time.UTC
|
||||
from := time.Date(2026, 9, 17, 15, 30, 0, 0, loc) // Thursday
|
||||
cases := []struct {
|
||||
expr string
|
||||
want time.Time
|
||||
}{
|
||||
{"0 1 * * *", time.Date(2026, 9, 18, 1, 0, 0, 0, loc)},
|
||||
{"17 * * * *", time.Date(2026, 9, 17, 16, 17, 0, 0, loc)},
|
||||
{"*/15 * * * *", time.Date(2026, 9, 17, 15, 45, 0, 0, loc)},
|
||||
{"47 6 * * 7", time.Date(2026, 9, 20, 6, 47, 0, 0, loc)}, // Sunday as 7
|
||||
{"52 6 1 * *", time.Date(2026, 10, 1, 6, 52, 0, 0, loc)}, // first of month
|
||||
{"24 0 1-7 * *", time.Date(2026, 10, 1, 0, 24, 0, 0, loc)}, // range
|
||||
{"30 3 * * 0", time.Date(2026, 9, 20, 3, 30, 0, 0, loc)}, // Sunday as 0
|
||||
{"0 9 15 * mon", time.Date(2026, 9, 21, 9, 0, 0, 0, loc)}, // dom OR dow
|
||||
{"@daily", time.Date(2026, 9, 18, 0, 0, 0, 0, loc)},
|
||||
{"@weekly", time.Date(2026, 9, 20, 0, 0, 0, 0, loc)},
|
||||
{"0 0 29 feb *", time.Date(2028, 2, 29, 0, 0, 0, 0, loc)},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, ok := nextRun(c.expr, from)
|
||||
if !ok || !got.Equal(c.want) {
|
||||
t.Errorf("%q: got %v ok=%v, want %v", c.expr, got, ok, c.want)
|
||||
}
|
||||
}
|
||||
if _, ok := nextRun("@reboot", from); ok {
|
||||
t.Error("@reboot should have no next run")
|
||||
}
|
||||
for _, bad := range []string{"* * * *", "60 * * * *", "a b c d e", "*/0 * * * *"} {
|
||||
if _, err := parseSchedule(bad); err == nil {
|
||||
t.Errorf("%q should fail", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLine(t *testing.T) {
|
||||
sch, u, cmd, ok := parseLine("25 6\t* * *\troot\ttest -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }", true)
|
||||
if !ok || sch != "25 6 * * *" || u != "root" || !strings_HasPrefix(cmd, "test -x") {
|
||||
t.Errorf("system line: %q %q %q %v", sch, u, cmd, ok)
|
||||
}
|
||||
if _, _, _, ok := parseLine("PATH=/usr/bin:/bin", true); ok {
|
||||
t.Error("env line parsed as job")
|
||||
}
|
||||
if _, _, _, ok := parseLine("# m h dom mon dow command", false); ok {
|
||||
t.Error("comment parsed as job")
|
||||
}
|
||||
if _, u, cmd, ok := parseLine("@reboot /usr/local/bin/thing --flag", false); !ok || u != "" || cmd != "/usr/local/bin/thing --flag" {
|
||||
t.Errorf("@reboot: %q %q %v", u, cmd, ok)
|
||||
}
|
||||
if got := outFile("foo >> /var/log/foo.log 2>&1"); got != "/var/log/foo.log" {
|
||||
t.Errorf("outFile: %q", got)
|
||||
}
|
||||
if c, in := cronPercent("echo hi%line1%line2"); c != "echo hi" || in != "line1\nline2" {
|
||||
t.Errorf("percent: %q %q", c, in)
|
||||
}
|
||||
if c, _ := cronPercent(`date +\%w`); c != "date +%w" {
|
||||
t.Errorf("escaped percent: %q", c)
|
||||
}
|
||||
}
|
||||
|
||||
func strings_HasPrefix(s, p string) bool { return len(s) >= len(p) && s[:len(p)] == p }
|
||||
@@ -0,0 +1,571 @@
|
||||
// Package files is the Files tab: browse, read, write, upload, download, mkdir,
|
||||
// rename, chmod and delete anywhere on the host.
|
||||
//
|
||||
// Two backends share one interface. Local runs in-process as the daemon's user. Sudo
|
||||
// re-executes this same binary as root (`sudo -n hamlab fs …`) for one operation at a
|
||||
// time, streaming file bodies over its stdin/stdout, so root access uses exactly the
|
||||
// same code and the same validation as the unprivileged path.
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/media"
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type Entry struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"` // dir | file | link | other
|
||||
Size int64 `json:"size"`
|
||||
Mode string `json:"mode"` // "-rw-r--r--"
|
||||
Perm string `json:"perm"` // "0644"
|
||||
Mtime int64 `json:"mtime"`
|
||||
User string `json:"user"`
|
||||
Group string `json:"group"`
|
||||
Target string `json:"target,omitempty"` // link target
|
||||
Broken bool `json:"broken,omitempty"` // dangling link
|
||||
Preview string `json:"preview,omitempty"` // "image" | "video": the Files tab can show a thumbnail of it
|
||||
}
|
||||
|
||||
type Listing struct {
|
||||
Path string `json:"path"`
|
||||
Parent string `json:"parent,omitempty"`
|
||||
Entries []Entry `json:"entries"`
|
||||
}
|
||||
|
||||
type Info struct {
|
||||
Entry
|
||||
Path string `json:"path"`
|
||||
Text bool `json:"text"` // looks like text (for the editor)
|
||||
}
|
||||
|
||||
// FS is what the API talks to.
|
||||
type FS interface {
|
||||
List(ctx context.Context, path string) (Listing, error)
|
||||
Stat(ctx context.Context, path string) (Info, error)
|
||||
Open(ctx context.Context, path string) (io.ReadCloser, error)
|
||||
OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) // Open from a byte offset: a range request, a player seeking
|
||||
Write(ctx context.Context, path string, r io.Reader) error
|
||||
Mkdir(ctx context.Context, path string) error
|
||||
Touch(ctx context.Context, path string) error
|
||||
Rename(ctx context.Context, from, to string) error
|
||||
Move(ctx context.Context, from, to string, progress io.Writer) error // Rename, or across filesystems a copy and a removal; long, so callers run it as a task
|
||||
Delete(ctx context.Context, path string, recursive bool) error
|
||||
Chmod(ctx context.Context, path string, perm os.FileMode) error
|
||||
}
|
||||
|
||||
// ---- path rules ----
|
||||
|
||||
// Clean rejects relative paths and normalises the rest. Everything else on the host is
|
||||
// reachable on purpose: this is a root-capable console, the gate is the login.
|
||||
func Clean(p string) (string, error) {
|
||||
if p == "" {
|
||||
return "", errors.New("path required")
|
||||
}
|
||||
if !strings.HasPrefix(p, "/") {
|
||||
return "", errors.New("path must be absolute")
|
||||
}
|
||||
c := filepath.Clean(p)
|
||||
if strings.ContainsRune(c, 0) {
|
||||
return "", errors.New("bad path")
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// protected are never deleted or renamed as a whole, no matter who asks.
|
||||
var protected = map[string]bool{"/": true, "/bin": true, "/boot": true, "/dev": true, "/etc": true, "/home": true, "/lib": true, "/lib64": true,
|
||||
"/opt": true, "/proc": true, "/root": true, "/run": true, "/sbin": true, "/srv": true, "/sys": true, "/tmp": true, "/usr": true, "/var": true}
|
||||
|
||||
func ParsePerm(s string) (os.FileMode, error) {
|
||||
v, err := strconv.ParseUint(strings.TrimPrefix(s, "0"), 8, 32)
|
||||
if err != nil || v > 0o7777 {
|
||||
return 0, errors.New("mode must be octal like 0644")
|
||||
}
|
||||
return os.FileMode(v), nil
|
||||
}
|
||||
|
||||
// ---- Local ----
|
||||
|
||||
type Local struct{}
|
||||
|
||||
var uidCache, gidCache = map[uint32]string{}, map[uint32]string{}
|
||||
|
||||
func owner(st os.FileInfo) (string, string) {
|
||||
sys, ok := st.Sys().(*syscall.Stat_t)
|
||||
if !ok {
|
||||
return "", ""
|
||||
}
|
||||
u, ok := uidCache[sys.Uid]
|
||||
if !ok {
|
||||
u = strconv.Itoa(int(sys.Uid))
|
||||
if x, err := user.LookupId(u); err == nil {
|
||||
u = x.Username
|
||||
}
|
||||
uidCache[sys.Uid] = u
|
||||
}
|
||||
g, ok := gidCache[sys.Gid]
|
||||
if !ok {
|
||||
g = strconv.Itoa(int(sys.Gid))
|
||||
if x, err := user.LookupGroupId(g); err == nil {
|
||||
g = x.Name
|
||||
}
|
||||
gidCache[sys.Gid] = g
|
||||
}
|
||||
return u, g
|
||||
}
|
||||
|
||||
func entryOf(dir, name string, lst os.FileInfo) Entry {
|
||||
e := Entry{Name: name, Size: lst.Size(), Mode: lst.Mode().String(), Perm: fmt.Sprintf("%04o", lst.Mode().Perm()), Mtime: lst.ModTime().UnixMilli()}
|
||||
e.User, e.Group = owner(lst)
|
||||
switch {
|
||||
case lst.Mode()&os.ModeSymlink != 0:
|
||||
e.Type = "link"
|
||||
full := filepath.Join(dir, name)
|
||||
e.Target, _ = os.Readlink(full)
|
||||
if st, err := os.Stat(full); err != nil {
|
||||
e.Broken = true
|
||||
} else if st.IsDir() {
|
||||
e.Type = "dir" // keep dir semantics for navigation; Target says it's a link
|
||||
} else {
|
||||
e.Size, e.Mtime = st.Size(), st.ModTime().UnixMilli() // the file's, not the link's
|
||||
}
|
||||
case lst.IsDir():
|
||||
e.Type = "dir"
|
||||
case lst.Mode().IsRegular():
|
||||
e.Type = "file"
|
||||
default:
|
||||
e.Type = "other"
|
||||
}
|
||||
if e.Type == "file" || (e.Type == "link" && !e.Broken) {
|
||||
e.Preview = media.Kind(name)
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
func (Local) List(_ context.Context, path string) (Listing, error) {
|
||||
ents, err := os.ReadDir(path)
|
||||
if err != nil {
|
||||
return Listing{}, err
|
||||
}
|
||||
l := Listing{Path: path, Entries: make([]Entry, 0, len(ents))}
|
||||
if path != "/" {
|
||||
l.Parent = filepath.Dir(path)
|
||||
}
|
||||
for _, d := range ents {
|
||||
lst, err := d.Info() // Lstat
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
l.Entries = append(l.Entries, entryOf(path, d.Name(), lst))
|
||||
}
|
||||
sort.Slice(l.Entries, func(i, j int) bool {
|
||||
a, b := l.Entries[i], l.Entries[j]
|
||||
if (a.Type == "dir") != (b.Type == "dir") {
|
||||
return a.Type == "dir"
|
||||
}
|
||||
return strings.ToLower(a.Name) < strings.ToLower(b.Name)
|
||||
})
|
||||
return l, nil
|
||||
}
|
||||
|
||||
func (Local) Stat(_ context.Context, path string) (Info, error) {
|
||||
lst, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return Info{}, err
|
||||
}
|
||||
in := Info{Entry: entryOf(filepath.Dir(path), filepath.Base(path), lst), Path: path}
|
||||
if in.Type == "file" && in.Size <= 8<<20 {
|
||||
if f, err := os.Open(path); err == nil {
|
||||
buf := make([]byte, 4096)
|
||||
n, _ := f.Read(buf)
|
||||
f.Close()
|
||||
in.Text = looksText(buf[:n])
|
||||
}
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
||||
func looksText(b []byte) bool {
|
||||
if len(b) == 0 {
|
||||
return true
|
||||
}
|
||||
ctrl := 0
|
||||
for _, c := range b {
|
||||
if c == 0 {
|
||||
return false
|
||||
}
|
||||
if c < 32 && c != '\n' && c != '\r' && c != '\t' && c != '\f' && c != 0x1b {
|
||||
ctrl++
|
||||
}
|
||||
}
|
||||
return ctrl*20 < len(b)
|
||||
}
|
||||
|
||||
// Open refuses anything that isn't a regular file: devices stream forever (/dev/zero),
|
||||
// FIFOs block on open, sockets can't be read. /proc files are regular and still work.
|
||||
func (Local) Open(_ context.Context, path string) (io.ReadCloser, error) {
|
||||
st, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if st.IsDir() {
|
||||
return nil, errors.New("is a directory")
|
||||
}
|
||||
if !st.Mode().IsRegular() {
|
||||
return nil, errors.New("not a regular file (device, pipe or socket)")
|
||||
}
|
||||
return os.Open(path)
|
||||
}
|
||||
|
||||
func (l Local) OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) {
|
||||
f, err := l.Open(ctx, path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if offset > 0 {
|
||||
if _, err := f.(*os.File).Seek(offset, io.SeekStart); err != nil {
|
||||
f.Close()
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// Write replaces the file's contents, keeping its mode; new files get 0644 (umask).
|
||||
func (Local) Write(_ context.Context, path string, r io.Reader) error {
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(f, r); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
func (Local) Mkdir(_ context.Context, path string) error { return os.Mkdir(path, 0o755) }
|
||||
|
||||
func (Local) Touch(_ context.Context, path string) error {
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
func (Local) Rename(_ context.Context, from, to string) error {
|
||||
if protected[from] {
|
||||
return errors.New("refusing to move " + from)
|
||||
}
|
||||
if _, err := os.Lstat(to); err == nil {
|
||||
return errors.New("destination exists")
|
||||
}
|
||||
return os.Rename(from, to)
|
||||
}
|
||||
|
||||
func (Local) Delete(_ context.Context, path string, recursive bool) error {
|
||||
if protected[path] {
|
||||
return errors.New("refusing to delete " + path)
|
||||
}
|
||||
lst, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if lst.IsDir() && recursive {
|
||||
return os.RemoveAll(path)
|
||||
}
|
||||
return os.Remove(path) // fails on a non-empty dir, by design
|
||||
}
|
||||
|
||||
func (Local) Chmod(_ context.Context, path string, perm os.FileMode) error {
|
||||
return os.Chmod(path, perm)
|
||||
}
|
||||
|
||||
// ---- Sudo: the same operations, run as root in a child hamlab ----
|
||||
|
||||
type Sudo struct{ Self string }
|
||||
|
||||
func (s Sudo) cmd(ctx context.Context, args ...string) *exec.Cmd {
|
||||
full := priv.Wrap(append([]string{s.Self, "fs"}, args...)...)
|
||||
return exec.CommandContext(ctx, full[0], full[1:]...)
|
||||
}
|
||||
|
||||
func (s Sudo) runJSON(ctx context.Context, out any, args ...string) error {
|
||||
c := s.cmd(ctx, args...)
|
||||
var errb strings.Builder
|
||||
c.Stderr = &errb
|
||||
b, err := c.Output()
|
||||
if err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
return json.Unmarshal(b, out)
|
||||
}
|
||||
|
||||
func (s Sudo) List(ctx context.Context, path string) (Listing, error) {
|
||||
var l Listing
|
||||
err := s.runJSON(ctx, &l, "list", path)
|
||||
return l, err
|
||||
}
|
||||
|
||||
func (s Sudo) Stat(ctx context.Context, path string) (Info, error) {
|
||||
var in Info
|
||||
err := s.runJSON(ctx, &in, "stat", path)
|
||||
return in, err
|
||||
}
|
||||
|
||||
type cmdReader struct {
|
||||
io.ReadCloser
|
||||
cmd *exec.Cmd
|
||||
errb *strings.Builder
|
||||
}
|
||||
|
||||
func (r *cmdReader) Close() error {
|
||||
r.ReadCloser.Close()
|
||||
err := r.cmd.Wait()
|
||||
if err != nil && r.errb.Len() > 0 {
|
||||
return errors.New(strings.TrimSpace(r.errb.String()))
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (s Sudo) Open(ctx context.Context, path string) (io.ReadCloser, error) {
|
||||
return s.open(ctx, "read", path)
|
||||
}
|
||||
|
||||
func (s Sudo) OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) {
|
||||
return s.open(ctx, "read", "--offset", strconv.FormatInt(offset, 10), path)
|
||||
}
|
||||
|
||||
func (s Sudo) open(ctx context.Context, args ...string) (io.ReadCloser, error) {
|
||||
c := s.cmd(ctx, args...)
|
||||
errb := &strings.Builder{}
|
||||
c.Stderr = errb
|
||||
out, err := c.StdoutPipe()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := c.Start(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Fail early on "no such file" etc.: the helper stats before writing anything.
|
||||
head := make([]byte, 1)
|
||||
n, rerr := out.Read(head)
|
||||
if n == 0 && rerr != nil {
|
||||
_ = c.Wait()
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return nil, errors.New(msg)
|
||||
}
|
||||
if rerr != io.EOF {
|
||||
return nil, rerr
|
||||
}
|
||||
}
|
||||
return &cmdReader{ReadCloser: struct {
|
||||
io.Reader
|
||||
io.Closer
|
||||
}{io.MultiReader(strings.NewReader(string(head[:n])), out), out}, cmd: c, errb: errb}, nil
|
||||
}
|
||||
|
||||
func (s Sudo) Write(ctx context.Context, path string, r io.Reader) error {
|
||||
c := s.cmd(ctx, "write", path)
|
||||
c.Stdin = r
|
||||
var errb strings.Builder
|
||||
c.Stderr = &errb
|
||||
if err := c.Run(); err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s Sudo) Mkdir(ctx context.Context, path string) error {
|
||||
return s.runJSON(ctx, nil, "mkdir", path)
|
||||
}
|
||||
func (s Sudo) Touch(ctx context.Context, path string) error {
|
||||
return s.runJSON(ctx, nil, "touch", path)
|
||||
}
|
||||
func (s Sudo) Rename(ctx context.Context, from, to string) error {
|
||||
return s.runJSON(ctx, nil, "rename", from, to)
|
||||
}
|
||||
|
||||
// Move as root: the helper does the copying and talks on stdout.
|
||||
func (s Sudo) Move(ctx context.Context, from, to string, progress io.Writer) error {
|
||||
c := s.cmd(ctx, "move", from, to)
|
||||
c.Stdout = progress
|
||||
var errb strings.Builder
|
||||
c.Stderr = &errb
|
||||
if err := c.Run(); err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// MoveCmd is the command a task runs for a long move: this binary's helper, as root when
|
||||
// the backend is.
|
||||
func MoveCmd(ctx context.Context, fsys FS, self, from, to string) *exec.Cmd {
|
||||
if s, ok := fsys.(Sudo); ok {
|
||||
return s.cmd(ctx, "move", from, to)
|
||||
}
|
||||
return exec.CommandContext(ctx, self, "fs", "move", from, to)
|
||||
}
|
||||
func (s Sudo) Delete(ctx context.Context, path string, recursive bool) error {
|
||||
if recursive {
|
||||
return s.runJSON(ctx, nil, "delete", "-r", path)
|
||||
}
|
||||
return s.runJSON(ctx, nil, "delete", path)
|
||||
}
|
||||
func (s Sudo) Chmod(ctx context.Context, path string, perm os.FileMode) error {
|
||||
return s.runJSON(ctx, nil, "chmod", fmt.Sprintf("%04o", perm), path)
|
||||
}
|
||||
|
||||
// Helper implements `hamlab fs <op> …` for the Sudo backend. Returns the exit code.
|
||||
func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer) int {
|
||||
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
|
||||
if len(args) < 2 {
|
||||
return fail(errors.New("usage: hamlab fs <list|stat|read|write|mkdir|touch|rename|move|delete|chmod> …"))
|
||||
}
|
||||
ctx := context.Background()
|
||||
var l Local
|
||||
op := args[0]
|
||||
var offset int64
|
||||
if op == "read" && len(args) >= 4 && args[1] == "--offset" { // read --offset N <path>
|
||||
offset, _ = strconv.ParseInt(args[2], 10, 64)
|
||||
args = append([]string{"read"}, args[3:]...)
|
||||
}
|
||||
clean := func(i int) (string, error) {
|
||||
if i >= len(args) {
|
||||
return "", errors.New("path required")
|
||||
}
|
||||
return Clean(args[i])
|
||||
}
|
||||
switch op {
|
||||
case "list", "stat", "read", "write", "mkdir", "touch":
|
||||
p, err := clean(1)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
switch op {
|
||||
case "list":
|
||||
out, err := l.List(ctx, p)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return emit(stdout, out, stderr)
|
||||
case "stat":
|
||||
out, err := l.Stat(ctx, p)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return emit(stdout, out, stderr)
|
||||
case "read":
|
||||
f, err := l.OpenAt(ctx, p, offset)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := io.Copy(stdout, f); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return 0
|
||||
case "write":
|
||||
if err := l.Write(ctx, p, stdin); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "mkdir":
|
||||
if err := l.Mkdir(ctx, p); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "touch":
|
||||
if err := l.Touch(ctx, p); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
}
|
||||
case "rename", "move":
|
||||
from, err := clean(1)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
to, err := clean(2)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
if op == "move" {
|
||||
// A cancelled task is a SIGTERM: stop copying and take the partial copy away.
|
||||
mctx, stop := signal.NotifyContext(ctx, syscall.SIGTERM, syscall.SIGINT)
|
||||
defer stop()
|
||||
if err := l.Move(mctx, from, to, stdout); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
if err := l.Rename(ctx, from, to); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "delete":
|
||||
rec := false
|
||||
i := 1
|
||||
if args[1] == "-r" {
|
||||
rec, i = true, 2
|
||||
}
|
||||
p, err := clean(i)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
if err := l.Delete(ctx, p, rec); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "chmod":
|
||||
perm, err := ParsePerm(args[1])
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
p, err := clean(2)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
if err := l.Chmod(ctx, p, perm); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
default:
|
||||
return fail(errors.New("unknown op " + op))
|
||||
}
|
||||
fmt.Fprintln(stdout, "{}")
|
||||
return 0
|
||||
}
|
||||
|
||||
func emit(w io.Writer, v any, stderr io.Writer) int {
|
||||
if err := json.NewEncoder(w).Encode(v); err != nil {
|
||||
fmt.Fprintln(stderr, err.Error())
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ModTime helper for handlers.
|
||||
func Now() time.Time { return time.Now() }
|
||||
@@ -0,0 +1,197 @@
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// IsCrossDevice reports a rename refused because the two ends are on different
|
||||
// filesystems — on a NAS that is every move between datasets. The Sudo backend only has
|
||||
// the helper's words for it.
|
||||
func IsCrossDevice(err error) bool {
|
||||
return errors.Is(err, syscall.EXDEV) || (err != nil && strings.Contains(err.Error(), "cross-device"))
|
||||
}
|
||||
|
||||
// Inside reports whether to lies within from — a folder cannot be moved into itself.
|
||||
func Inside(from, to string) bool {
|
||||
return to == from || strings.HasPrefix(to, strings.TrimSuffix(from, "/")+"/")
|
||||
}
|
||||
|
||||
// Move is Rename, and across filesystems a copy and a removal. The copy is built under a
|
||||
// hidden temporary name beside where it will land and renamed into place only once it is
|
||||
// complete, and the source is removed only after that: a failure part-way, or a cancel,
|
||||
// leaves the source exactly as it was. Progress goes to w, a line at a time, for the task
|
||||
// output that a long move is watched through.
|
||||
func (Local) Move(ctx context.Context, from, to string, w io.Writer) error {
|
||||
if protected[from] {
|
||||
return errors.New("refusing to move " + from)
|
||||
}
|
||||
if Inside(from, to) {
|
||||
return errors.New("a folder can't be moved into itself")
|
||||
}
|
||||
if _, err := os.Lstat(to); err == nil {
|
||||
return errors.New("destination exists")
|
||||
}
|
||||
err := os.Rename(from, to)
|
||||
if err == nil || !IsCrossDevice(err) {
|
||||
return err
|
||||
}
|
||||
return moveByCopy(ctx, from, to, w)
|
||||
}
|
||||
|
||||
// moveByCopy is the slow path: copy, rename into place, remove the original.
|
||||
func moveByCopy(ctx context.Context, from, to string, w io.Writer) error {
|
||||
st, err := os.Lstat(from)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := filepath.Join(filepath.Dir(to), "."+filepath.Base(to)+".hamlab-moving")
|
||||
_ = os.RemoveAll(tmp) // a previous attempt that died
|
||||
fmt.Fprintf(w, "%s is on another filesystem: copying, then removing the original\n", filepath.Dir(to))
|
||||
p := &progress{w: w}
|
||||
if err := copyPath(ctx, from, tmp, st, p); err != nil {
|
||||
fmt.Fprintln(w, "copy failed; removing the partial copy, the original is untouched")
|
||||
_ = os.RemoveAll(tmp)
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(tmp, to); err != nil {
|
||||
_ = os.RemoveAll(tmp)
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(w, "copied %s; removing the original\n", p.String())
|
||||
if err := os.RemoveAll(from); err != nil {
|
||||
return fmt.Errorf("copied to %s, but the original could not be removed: %w", to, err)
|
||||
}
|
||||
fmt.Fprintf(w, "moved %s to %s\n", from, to)
|
||||
return nil
|
||||
}
|
||||
|
||||
type progress struct {
|
||||
w io.Writer
|
||||
files int64
|
||||
bytes int64
|
||||
shownFiles int64
|
||||
shownBytes int64
|
||||
}
|
||||
|
||||
func (p *progress) add(files, bytes int64) {
|
||||
p.files += files
|
||||
p.bytes += bytes
|
||||
if p.files-p.shownFiles >= 500 || p.bytes-p.shownBytes >= 512<<20 {
|
||||
fmt.Fprintf(p.w, "copying… %s\n", p.String())
|
||||
p.shownFiles, p.shownBytes = p.files, p.bytes
|
||||
}
|
||||
}
|
||||
|
||||
func (p *progress) String() string {
|
||||
return fmt.Sprintf("%d files, %s", p.files, fmtSize(p.bytes))
|
||||
}
|
||||
|
||||
func fmtSize(b int64) string {
|
||||
const unit = 1024
|
||||
if b < unit {
|
||||
return fmt.Sprintf("%d B", b)
|
||||
}
|
||||
div, exp := int64(unit), 0
|
||||
for n := b / unit; n >= unit && exp < 4; n /= unit {
|
||||
div *= unit
|
||||
exp++
|
||||
}
|
||||
return fmt.Sprintf("%.1f %ciB", float64(b)/float64(div), "KMGTP"[exp])
|
||||
}
|
||||
|
||||
// copyPath copies one entry of any kind, recursively for a directory, keeping mode, times
|
||||
// and — when running as root — ownership. Devices, pipes and sockets are refused: a copy
|
||||
// of one is not the thing itself.
|
||||
func copyPath(ctx context.Context, src, dst string, st os.FileInfo, p *progress) error {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
mode := st.Mode()
|
||||
switch {
|
||||
case mode&os.ModeSymlink != 0:
|
||||
target, err := os.Readlink(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Symlink(target, dst); err != nil {
|
||||
return err
|
||||
}
|
||||
chown(dst, st)
|
||||
p.add(1, 0)
|
||||
return nil
|
||||
case mode.IsDir():
|
||||
if err := os.Mkdir(dst, mode.Perm()|0o700); err != nil { // writable while it fills, exact mode after
|
||||
return err
|
||||
}
|
||||
ents, err := os.ReadDir(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, e := range ents {
|
||||
lst, err := e.Info()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := copyPath(ctx, filepath.Join(src, e.Name()), filepath.Join(dst, e.Name()), lst, p); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
chown(dst, st)
|
||||
if err := os.Chmod(dst, mode.Perm()); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Chtimes(dst, st.ModTime(), st.ModTime())
|
||||
case mode.IsRegular():
|
||||
n, err := copyFile(src, dst, st)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
p.add(1, n)
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("%s: can't copy a device, pipe or socket", src)
|
||||
}
|
||||
}
|
||||
|
||||
func copyFile(src, dst string, st os.FileInfo) (int64, error) {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer in.Close()
|
||||
out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL, st.Mode().Perm())
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
n, err := io.Copy(out, in)
|
||||
if err != nil {
|
||||
out.Close()
|
||||
return n, err
|
||||
}
|
||||
if err := out.Close(); err != nil {
|
||||
return n, err
|
||||
}
|
||||
chown(dst, st)
|
||||
if err := os.Chmod(dst, st.Mode().Perm()); err != nil { // umask has no say
|
||||
return n, err
|
||||
}
|
||||
return n, os.Chtimes(dst, st.ModTime(), st.ModTime())
|
||||
}
|
||||
|
||||
// chown keeps the owner when this process may set one; an ordinary user's copy is the
|
||||
// user's, which is what a rename would have left anyway.
|
||||
func chown(dst string, st os.FileInfo) {
|
||||
if os.Geteuid() != 0 {
|
||||
return
|
||||
}
|
||||
if sys, ok := st.Sys().(*syscall.Stat_t); ok {
|
||||
_ = os.Lchown(dst, int(sys.Uid), int(sys.Gid))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,195 @@
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestInside(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
from, to string
|
||||
want bool
|
||||
}{
|
||||
{"/a/b", "/a/b", true}, {"/a/b", "/a/b/c", true}, {"/a/b", "/a/bc", false}, {"/a/b", "/a", false}, {"/", "/x", true},
|
||||
} {
|
||||
if got := Inside(c.from, c.to); got != c.want {
|
||||
t.Errorf("Inside(%q, %q) = %v, want %v", c.from, c.to, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsCrossDevice(t *testing.T) {
|
||||
if !IsCrossDevice(&os.LinkError{Op: "rename", Old: "/a", New: "/b", Err: syscall.EXDEV}) {
|
||||
t.Fatal("the kernel's EXDEV")
|
||||
}
|
||||
if !IsCrossDevice(errors.New("rename /a /b: invalid cross-device link")) {
|
||||
t.Fatal("the helper's words for it")
|
||||
}
|
||||
if IsCrossDevice(errors.New("permission denied")) || IsCrossDevice(nil) {
|
||||
t.Fatal("anything else is not")
|
||||
}
|
||||
}
|
||||
|
||||
// The copy path, exercised directly: a tree with nested folders, a symlink, an odd mode
|
||||
// and old mtimes lands whole under the new name, and the original is gone only then.
|
||||
func TestMoveByCopy(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
src := filepath.Join(root, "src", "album")
|
||||
if err := os.MkdirAll(filepath.Join(src, "sub", "deep"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
write := func(p, s string, mode os.FileMode) {
|
||||
if err := os.WriteFile(p, []byte(s), mode); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Chmod(p, mode)
|
||||
_ = os.Chtimes(p, old, old)
|
||||
}
|
||||
write(filepath.Join(src, "a.txt"), "hello", 0o644)
|
||||
write(filepath.Join(src, "sub", "deep", "b.bin"), strings.Repeat("x", 100000), 0o600)
|
||||
write(filepath.Join(src, "run.sh"), "#!/bin/sh\n", 0o755)
|
||||
if err := os.Symlink("a.txt", filepath.Join(src, "link")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Chtimes(filepath.Join(src, "sub"), old, old)
|
||||
|
||||
dst := filepath.Join(root, "dst", "moved")
|
||||
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out strings.Builder
|
||||
if err := moveByCopy(context.Background(), src, dst, &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Lstat(src); !os.IsNotExist(err) {
|
||||
t.Fatal("the original is still there")
|
||||
}
|
||||
if b, _ := os.ReadFile(filepath.Join(dst, "a.txt")); string(b) != "hello" {
|
||||
t.Fatal("a.txt")
|
||||
}
|
||||
if st, err := os.Stat(filepath.Join(dst, "sub", "deep", "b.bin")); err != nil || st.Size() != 100000 || st.Mode().Perm() != 0o600 || !st.ModTime().Equal(old) {
|
||||
t.Fatalf("b.bin: %v %v", st, err)
|
||||
}
|
||||
if st, _ := os.Stat(filepath.Join(dst, "run.sh")); st.Mode().Perm() != 0o755 {
|
||||
t.Fatal("mode lost")
|
||||
}
|
||||
if target, err := os.Readlink(filepath.Join(dst, "link")); err != nil || target != "a.txt" {
|
||||
t.Fatal("symlink not copied as a symlink")
|
||||
}
|
||||
if st, _ := os.Stat(filepath.Join(dst, "sub")); !st.ModTime().Equal(old) {
|
||||
t.Fatal("folder mtime lost")
|
||||
}
|
||||
if left, _ := filepath.Glob(filepath.Join(root, "dst", ".*hamlab-moving*")); len(left) != 0 {
|
||||
t.Fatal("temporary copy left behind")
|
||||
}
|
||||
if !strings.Contains(out.String(), "moved ") || !strings.Contains(out.String(), "4 files") {
|
||||
t.Fatalf("progress said: %q", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
// A failure part-way leaves the original as it was and no partial copy.
|
||||
func TestMoveByCopyFailsClean(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
src := filepath.Join(root, "src")
|
||||
if err := os.MkdirAll(src, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(src, "a"), []byte("a"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := syscall.Mkfifo(filepath.Join(src, "pipe"), 0o644); err != nil {
|
||||
t.Skip("no fifos here")
|
||||
}
|
||||
dst := filepath.Join(root, "dst")
|
||||
err := moveByCopy(context.Background(), src, dst, &strings.Builder{})
|
||||
if err == nil || !strings.Contains(err.Error(), "pipe") {
|
||||
t.Fatalf("expected the pipe to be refused, got %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(src, "a")); err != nil {
|
||||
t.Fatal("the original was touched")
|
||||
}
|
||||
if _, err := os.Lstat(dst); !os.IsNotExist(err) {
|
||||
t.Fatal("a partial copy was left")
|
||||
}
|
||||
if left, _ := filepath.Glob(filepath.Join(root, ".*hamlab-moving*")); len(left) != 0 {
|
||||
t.Fatal("temporary copy left behind")
|
||||
}
|
||||
// a cancel is the same story
|
||||
if err := os.Remove(filepath.Join(src, "pipe")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if err := moveByCopy(ctx, src, dst, &strings.Builder{}); err == nil {
|
||||
t.Fatal("a cancelled move must fail")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(src, "a")); err != nil {
|
||||
t.Fatal("the original was touched by a cancel")
|
||||
}
|
||||
}
|
||||
|
||||
// Move on one filesystem is a rename: nothing is copied, and the guards hold.
|
||||
func TestMoveSameFilesystem(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
src := filepath.Join(root, "f")
|
||||
if err := os.WriteFile(src, []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var l Local
|
||||
var out strings.Builder
|
||||
if err := l.Move(context.Background(), src, filepath.Join(root, "g"), &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if out.Len() != 0 {
|
||||
t.Fatal("a rename has nothing to say")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(root, "g")); err != nil {
|
||||
t.Fatal("not renamed")
|
||||
}
|
||||
if err := l.Move(context.Background(), root, filepath.Join(root, "inside"), &out); err == nil {
|
||||
t.Fatal("into itself")
|
||||
}
|
||||
if err := l.Move(context.Background(), filepath.Join(root, "g"), filepath.Join(root, "g"), &out); err == nil {
|
||||
t.Fatal("onto itself")
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, "h"), nil, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := l.Move(context.Background(), filepath.Join(root, "g"), filepath.Join(root, "h"), &out); err == nil || !strings.Contains(err.Error(), "exists") {
|
||||
t.Fatalf("onto an existing file: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A link to a file carries the file's size and time, so the row and the preview key follow
|
||||
// the file; a link to a folder is a folder; a link to nothing says so.
|
||||
func TestEntryOfLinks(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "f"), []byte("12345"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := time.Date(2021, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
_ = os.Chtimes(filepath.Join(dir, "f"), old, old)
|
||||
_ = os.Mkdir(filepath.Join(dir, "d"), 0o755)
|
||||
for _, l := range [][2]string{{"lf", "f"}, {"ld", "d"}, {"lx", "nowhere"}} {
|
||||
if err := os.Symlink(l[1], filepath.Join(dir, l[0])); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
ent := func(n string) Entry { st, _ := os.Lstat(filepath.Join(dir, n)); return entryOf(dir, n, st) }
|
||||
if e := ent("lf"); e.Type != "link" || e.Size != 5 || e.Mtime != old.UnixMilli() || e.Target != "f" || e.Broken {
|
||||
t.Fatalf("link to file: %+v", e)
|
||||
}
|
||||
if e := ent("ld"); e.Type != "dir" || e.Target != "d" {
|
||||
t.Fatalf("link to dir: %+v", e)
|
||||
}
|
||||
if e := ent("lx"); e.Type != "link" || !e.Broken || e.Preview != "" {
|
||||
t.Fatalf("dangling: %+v", e)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
)
|
||||
|
||||
// Seeker is an FS file as http.ServeContent wants one — something to seek in — for a file
|
||||
// whose size is already known from a stat. ServeContent seeks to the end for the size, to
|
||||
// the start of the range asked for, then reads: so the backend is opened once, at that
|
||||
// position, and only ever read forward. The Sudo backend's helper is started at the offset
|
||||
// rather than streamed from the top and skipped.
|
||||
type Seeker struct {
|
||||
ctx context.Context
|
||||
fsys FS
|
||||
path string
|
||||
size int64
|
||||
pos int64 // where the next Read reads from
|
||||
at int64 // where cur is positioned, when open
|
||||
cur io.ReadCloser
|
||||
}
|
||||
|
||||
func NewSeeker(ctx context.Context, fsys FS, path string, size int64) *Seeker {
|
||||
return &Seeker{ctx: ctx, fsys: fsys, path: path, size: size}
|
||||
}
|
||||
|
||||
// Check opens the file now, from the start, so a file that cannot be read is known before
|
||||
// any headers go out. Seeking costs nothing; a Read from somewhere else reopens there.
|
||||
func (s *Seeker) Check() error {
|
||||
if s.cur != nil {
|
||||
return nil
|
||||
}
|
||||
f, err := s.fsys.OpenAt(s.ctx, s.path, 0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.cur, s.at = f, 0
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Seeker) Seek(offset int64, whence int) (int64, error) {
|
||||
var pos int64
|
||||
switch whence {
|
||||
case io.SeekStart:
|
||||
pos = offset
|
||||
case io.SeekCurrent:
|
||||
pos = s.pos + offset
|
||||
case io.SeekEnd:
|
||||
pos = s.size + offset
|
||||
default:
|
||||
return 0, errors.New("bad whence")
|
||||
}
|
||||
if pos < 0 {
|
||||
return 0, errors.New("seek before start")
|
||||
}
|
||||
s.pos = pos // the stream, if open, stays where it is until a Read wants somewhere else
|
||||
return pos, nil
|
||||
}
|
||||
|
||||
func (s *Seeker) Read(p []byte) (int, error) {
|
||||
if s.pos >= s.size {
|
||||
return 0, io.EOF
|
||||
}
|
||||
if s.cur != nil && s.pos != s.at {
|
||||
s.cur.Close()
|
||||
s.cur = nil
|
||||
}
|
||||
if s.cur == nil {
|
||||
f, err := s.fsys.OpenAt(s.ctx, s.path, s.pos)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
s.cur, s.at = f, s.pos
|
||||
}
|
||||
n, err := s.cur.Read(p)
|
||||
s.pos += int64(n)
|
||||
s.at += int64(n)
|
||||
return n, err
|
||||
}
|
||||
|
||||
func (s *Seeker) Close() error {
|
||||
if s.cur == nil {
|
||||
return nil
|
||||
}
|
||||
err := s.cur.Close()
|
||||
s.cur = nil
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
import "strings"
|
||||
|
||||
type stringsBuilder = strings.Builder
|
||||
|
||||
type countingFS struct {
|
||||
Local
|
||||
opens []int64
|
||||
}
|
||||
|
||||
func (c *countingFS) OpenAt(ctx context.Context, path string, off int64) (io.ReadCloser, error) {
|
||||
c.opens = append(c.opens, off)
|
||||
return c.Local.OpenAt(ctx, path, off)
|
||||
}
|
||||
|
||||
// ServeContent's moves — size from the end, seek to the range, read forward — open the
|
||||
// backend once, at the range; a seek to where the stream already is costs nothing.
|
||||
func TestSeeker(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "f")
|
||||
data := make([]byte, 1000)
|
||||
for i := range data {
|
||||
data[i] = byte(i)
|
||||
}
|
||||
if err := os.WriteFile(p, data, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fs := &countingFS{}
|
||||
s := NewSeeker(context.Background(), fs, p, int64(len(data)))
|
||||
if n, _ := s.Seek(0, io.SeekEnd); n != 1000 {
|
||||
t.Fatalf("end = %d", n)
|
||||
}
|
||||
if _, err := s.Seek(100, io.SeekStart); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
buf := make([]byte, 50)
|
||||
if n, err := io.ReadFull(s, buf); err != nil || n != 50 || buf[0] != 100 || buf[49] != 149 {
|
||||
t.Fatalf("read at 100: %d %v %v", n, err, buf[:2])
|
||||
}
|
||||
if _, err := s.Seek(150, io.SeekStart); err != nil { // where it already is
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n, _ := io.ReadFull(s, buf); n != 50 || buf[0] != 150 {
|
||||
t.Fatal("read after a no-op seek")
|
||||
}
|
||||
if len(fs.opens) != 1 || fs.opens[0] != 100 {
|
||||
t.Fatalf("opens = %v, want [100]", fs.opens)
|
||||
}
|
||||
if _, err := s.Seek(0, io.SeekStart); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rest, err := io.ReadAll(s)
|
||||
if err != nil || len(rest) != 1000 || rest[999] != data[999] {
|
||||
t.Fatalf("read all from the start: %d %v", len(rest), err)
|
||||
}
|
||||
if len(fs.opens) != 2 || fs.opens[1] != 0 {
|
||||
t.Fatalf("opens = %v, want [100 0]", fs.opens)
|
||||
}
|
||||
if n, err := s.Read(buf); n != 0 || err != io.EOF {
|
||||
t.Fatal("past the end is EOF")
|
||||
}
|
||||
if _, err := s.Seek(-1, io.SeekStart); err == nil {
|
||||
t.Fatal("before the start")
|
||||
}
|
||||
if err := s.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Check opens at the start and keeps that stream for a read from the start
|
||||
s2 := NewSeeker(context.Background(), fs, p, 1000)
|
||||
if err := s2.Check(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s2.Seek(0, io.SeekEnd)
|
||||
s2.Seek(0, io.SeekStart)
|
||||
if n, _ := io.ReadFull(s2, buf); n != 50 || buf[0] != 0 {
|
||||
t.Fatal("read from the start after Check")
|
||||
}
|
||||
if len(fs.opens) != 3 {
|
||||
t.Fatalf("Check's open was not reused: %v", fs.opens)
|
||||
}
|
||||
s2.Close()
|
||||
if err := NewSeeker(context.Background(), fs, filepath.Join(t.TempDir(), "nope"), 5).Check(); err == nil {
|
||||
t.Fatal("Check must report a file that can't be opened")
|
||||
}
|
||||
}
|
||||
|
||||
// The helper's read takes an offset, for the Sudo backend.
|
||||
func TestHelperReadOffset(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "f")
|
||||
if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out, errb stringsBuilder
|
||||
if rc := Helper([]string{"read", "--offset", "7", p}, nil, &out, &errb); rc != 0 {
|
||||
t.Fatalf("rc %d: %s", rc, errb.String())
|
||||
}
|
||||
if out.String() != "789" {
|
||||
t.Fatalf("got %q", out.String())
|
||||
}
|
||||
out.Reset()
|
||||
if rc := Helper([]string{"read", p}, nil, &out, &errb); rc != 0 || out.String() != "0123456789" {
|
||||
t.Fatalf("plain read: rc %d %q", rc, out.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,843 @@
|
||||
// Package hardware is what is physically in the machine: the board it is, its processor,
|
||||
// the memory modules in its slots, the drives attached to it and the graphics in it.
|
||||
//
|
||||
// None of this changes while the machine is running, so it is read once and remembered.
|
||||
// Everything here is read from what Debian and its kin already have — dmidecode for the
|
||||
// parts only the firmware knows, lscpu, lspci, sysfs — and every reader is allowed to
|
||||
// come back empty: a machine with no dmidecode, no PCI bus or no root still gets a page,
|
||||
// with a note saying what could not be read and why.
|
||||
package hardware
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
type System struct {
|
||||
Model string `json:"model,omitempty"` // "GMKtec NucBox G3"
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Family string `json:"family,omitempty"`
|
||||
Kernel string `json:"kernel,omitempty"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
Virt string `json:"virt,omitempty"` // "kvm", "lxc", … when this is not real hardware
|
||||
}
|
||||
|
||||
// Slot is an expansion slot as the firmware describes it. Worth treating with suspicion on
|
||||
// a mini PC, where the table is often copied from a reference design and lists slots the
|
||||
// case has no room for.
|
||||
type Slot struct {
|
||||
Designation string `json:"designation"`
|
||||
Type string `json:"type,omitempty"`
|
||||
Usage string `json:"usage,omitempty"` // Available | In Use | Unknown
|
||||
Length string `json:"length,omitempty"`
|
||||
Width string `json:"width,omitempty"`
|
||||
Bus string `json:"bus,omitempty"` // PCI address of what is in it
|
||||
Device string `json:"device,omitempty"` // and what that turns out to be
|
||||
}
|
||||
|
||||
// Board is the motherboard and the firmware on it.
|
||||
type Board struct {
|
||||
Maker string `json:"maker,omitempty"`
|
||||
Product string `json:"product,omitempty"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
AssetTag string `json:"asset_tag,omitempty"`
|
||||
Type string `json:"type,omitempty"` // "Motherboard", "Processor Module", …
|
||||
Location string `json:"location,omitempty"` // where it sits in the chassis
|
||||
Chassis string `json:"chassis,omitempty"`
|
||||
ChassisMaker string `json:"chassis_maker,omitempty"`
|
||||
BIOS string `json:"bios,omitempty"` // vendor and version
|
||||
BIOSDate string `json:"bios_date,omitempty"`
|
||||
BIOSRevision string `json:"bios_revision,omitempty"`
|
||||
ROMSize string `json:"rom_size,omitempty"`
|
||||
UEFI bool `json:"uefi,omitempty"`
|
||||
Slots []Slot `json:"slots,omitempty"`
|
||||
}
|
||||
|
||||
type Cache struct {
|
||||
Level string `json:"level"` // L1d, L1i, L2, L3
|
||||
Size string `json:"size"` // as lscpu puts it: "6 MiB (1 instance)"
|
||||
}
|
||||
|
||||
type CPU struct {
|
||||
Model string `json:"model,omitempty"`
|
||||
Vendor string `json:"vendor,omitempty"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
Sockets int `json:"sockets,omitempty"`
|
||||
SocketNames []string `json:"socket_names,omitempty"` // what the firmware calls each socket
|
||||
Cores int `json:"cores,omitempty"`
|
||||
Threads int `json:"threads,omitempty"`
|
||||
ThreadsPerCore int `json:"threads_per_core,omitempty"`
|
||||
MinMHz float64 `json:"min_mhz,omitempty"`
|
||||
MaxMHz float64 `json:"max_mhz,omitempty"`
|
||||
Family string `json:"family,omitempty"`
|
||||
ModelID string `json:"model_id,omitempty"`
|
||||
Stepping string `json:"stepping,omitempty"`
|
||||
Microcode string `json:"microcode,omitempty"`
|
||||
Virtualization string `json:"virtualization,omitempty"`
|
||||
Caches []Cache `json:"caches,omitempty"`
|
||||
Flags []string `json:"flags,omitempty"` // the handful anyone looks for
|
||||
}
|
||||
|
||||
type Module struct {
|
||||
Locator string `json:"locator"` // the slot, as the board labels it
|
||||
Bank string `json:"bank,omitempty"`
|
||||
Empty bool `json:"empty,omitempty"`
|
||||
Size uint64 `json:"size,omitempty"`
|
||||
Type string `json:"type,omitempty"` // DDR4
|
||||
FormFactor string `json:"form_factor,omitempty"` // SODIMM
|
||||
SpeedMTs int `json:"speed_mts,omitempty"` // what the module can do
|
||||
RunningMTs int `json:"running_mts,omitempty"` // what the board is running it at
|
||||
Maker string `json:"maker,omitempty"`
|
||||
Part string `json:"part,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Rank int `json:"rank,omitempty"`
|
||||
Voltage string `json:"voltage,omitempty"`
|
||||
}
|
||||
|
||||
type Memory struct {
|
||||
Total uint64 `json:"total"` // what the kernel can use
|
||||
Installed uint64 `json:"installed,omitempty"` // what the modules add up to
|
||||
MaxCapacity uint64 `json:"max_capacity,omitempty"` // what the board will take
|
||||
Slots int `json:"slots,omitempty"`
|
||||
Filled int `json:"filled,omitempty"`
|
||||
ECC string `json:"ecc,omitempty"`
|
||||
Modules []Module `json:"modules,omitempty"`
|
||||
}
|
||||
|
||||
type Drive struct {
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"` // nvme | ssd | hdd | usb | emmc | sd | other
|
||||
Model string `json:"model,omitempty"`
|
||||
Vendor string `json:"vendor,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Firmware string `json:"firmware,omitempty"`
|
||||
Tran string `json:"tran,omitempty"`
|
||||
Size uint64 `json:"size"`
|
||||
Rotational bool `json:"rotational,omitempty"`
|
||||
Removable bool `json:"removable,omitempty"`
|
||||
}
|
||||
|
||||
type GPU struct {
|
||||
Model string `json:"model,omitempty"`
|
||||
Vendor string `json:"vendor,omitempty"`
|
||||
Slot string `json:"slot,omitempty"` // PCI address, when it is on a bus
|
||||
Driver string `json:"driver,omitempty"`
|
||||
Node string `json:"node,omitempty"` // card0
|
||||
Memory uint64 `json:"memory,omitempty"`
|
||||
Class string `json:"class,omitempty"` // VGA | 3D | Display
|
||||
}
|
||||
|
||||
type Report struct {
|
||||
At time.Time `json:"at"`
|
||||
System System `json:"system"`
|
||||
Board Board `json:"board"`
|
||||
CPU CPU `json:"cpu"`
|
||||
Memory Memory `json:"memory"`
|
||||
Drives []Drive `json:"drives"`
|
||||
GPUs []GPU `json:"gpus"`
|
||||
Notes []string `json:"notes,omitempty"` // what could not be read, and why
|
||||
}
|
||||
|
||||
// Manager remembers the last report. Hardware does not change under a running kernel —
|
||||
// the one thing that would, a drive plugged in, is what Refresh is for.
|
||||
type Manager struct {
|
||||
mu sync.Mutex
|
||||
last *Report
|
||||
at time.Time
|
||||
}
|
||||
|
||||
func New() *Manager { return &Manager{} }
|
||||
|
||||
const keepFor = 10 * time.Minute
|
||||
|
||||
func (m *Manager) Report(ctx context.Context, refresh bool) *Report {
|
||||
m.mu.Lock()
|
||||
if !refresh && m.last != nil && time.Since(m.at) < keepFor {
|
||||
r := m.last
|
||||
m.mu.Unlock()
|
||||
return r
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
r := collect(ctx)
|
||||
|
||||
m.mu.Lock()
|
||||
m.last, m.at = r, time.Now()
|
||||
m.mu.Unlock()
|
||||
return r
|
||||
}
|
||||
|
||||
func collect(ctx context.Context) *Report {
|
||||
ctx, cancel := context.WithTimeout(ctx, 25*time.Second)
|
||||
defer cancel()
|
||||
r := &Report{At: time.Now()}
|
||||
dmi, note := readDMI(ctx)
|
||||
if note != "" {
|
||||
r.Notes = append(r.Notes, note)
|
||||
}
|
||||
r.System = system(ctx, dmi)
|
||||
r.Board = board(dmi, pciNames(ctx))
|
||||
r.CPU = cpu(ctx, dmi)
|
||||
r.Memory = memory(dmi)
|
||||
if len(r.Memory.Modules) == 0 && note == "" {
|
||||
r.Notes = append(r.Notes, "memory modules: the firmware's table says nothing about this machine's slots")
|
||||
}
|
||||
r.Drives = drives(ctx)
|
||||
r.GPUs = gpus(ctx)
|
||||
if len(r.GPUs) == 0 {
|
||||
r.Notes = append(r.Notes, "graphics: nothing on the PCI bus and no kernel display device")
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// ---- dmidecode ----
|
||||
|
||||
// record is one entry of the firmware's table: its DMI type and its key/value lines.
|
||||
type record struct {
|
||||
Type int
|
||||
V map[string]string
|
||||
}
|
||||
|
||||
// listKey holds every doubly-indented line of a record — dmidecode's lists, which have no
|
||||
// key of their own: BIOS characteristics, board features, slot characteristics.
|
||||
const listKey = "\x00list"
|
||||
|
||||
var handleRe = regexp.MustCompile(`^Handle 0x[0-9A-Fa-f]+, DMI type (\d+)`)
|
||||
|
||||
// readDMI runs dmidecode once for every type the page wants and parses the lot. It needs
|
||||
// root; without it the page loses the board, the memory modules and the socket names, and
|
||||
// says so rather than pretending the machine has none.
|
||||
func readDMI(ctx context.Context) ([]record, string) {
|
||||
tool, err := exec.LookPath("dmidecode")
|
||||
if err != nil {
|
||||
if _, e := os.Stat("/usr/sbin/dmidecode"); e != nil {
|
||||
return nil, "the board and the memory modules need dmidecode, which isn't installed here"
|
||||
}
|
||||
tool = "/usr/sbin/dmidecode"
|
||||
}
|
||||
args := []string{tool, "-t", "bios", "-t", "system", "-t", "baseboard", "-t", "chassis", "-t", "slot", "-t", "processor", "-t", "memory"}
|
||||
if os.Geteuid() != 0 {
|
||||
if !priv.Available() {
|
||||
return nil, "the board and the memory modules come from the firmware's table, which only root may read"
|
||||
}
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, args[0], args[1:]...).Output()
|
||||
if err != nil {
|
||||
return nil, "the firmware's table could not be read: " + strings.TrimSpace(err.Error())
|
||||
}
|
||||
return parse(string(out)), ""
|
||||
}
|
||||
|
||||
// parse turns dmidecode's output into records: a handle line starts one, and the single
|
||||
// indented "Key: Value" lines under it are its fields. Doubly-indented lines are the
|
||||
// continuation of a list (a processor's flags), which nothing here reads.
|
||||
func parse(text string) []record {
|
||||
var recs []record
|
||||
var cur *record
|
||||
sc := bufio.NewScanner(strings.NewReader(text))
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 2<<20)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if m := handleRe.FindStringSubmatch(line); m != nil {
|
||||
t, _ := strconv.Atoi(m[1])
|
||||
recs = append(recs, record{Type: t, V: map[string]string{}})
|
||||
cur = &recs[len(recs)-1]
|
||||
continue
|
||||
}
|
||||
if cur == nil || !strings.HasPrefix(line, "\t") {
|
||||
if strings.TrimSpace(line) == "" {
|
||||
cur = nil
|
||||
}
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(line, "\t\t") { // a list under the field above it, e.g. BIOS characteristics
|
||||
cur.V[listKey] += strings.TrimSpace(line) + "\n"
|
||||
continue
|
||||
}
|
||||
k, v, ok := strings.Cut(strings.TrimSpace(line), ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
cur.V[k] = strings.TrimSpace(v)
|
||||
}
|
||||
return recs
|
||||
}
|
||||
|
||||
func (r record) get(keys ...string) string {
|
||||
for _, k := range keys {
|
||||
if v := r.V[k]; v != "" && !unset(v) {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// unset is every way the firmware has of saying "nobody filled this in".
|
||||
func unset(v string) bool {
|
||||
switch strings.ToLower(v) {
|
||||
case "", "unknown", "not specified", "not provided", "none", "default string", "to be filled by o.e.m.",
|
||||
"system manufacturer", "system product name", "to be filled by oem", "n/a", "empty", "no module installed",
|
||||
"<out of spec>", "other", "not applicable":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func first(recs []record, t int) record {
|
||||
for _, r := range recs {
|
||||
if r.Type == t {
|
||||
return r
|
||||
}
|
||||
}
|
||||
return record{Type: t, V: map[string]string{}}
|
||||
}
|
||||
|
||||
func all(recs []record, t int) []record {
|
||||
var out []record
|
||||
for _, r := range recs {
|
||||
if r.Type == t {
|
||||
out = append(out, r)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- system ----
|
||||
|
||||
func system(ctx context.Context, dmi []record) System {
|
||||
sys := first(dmi, 1)
|
||||
s := System{
|
||||
Model: join(sys.get("Manufacturer"), sys.get("Product Name")),
|
||||
Serial: sys.get("Serial Number"),
|
||||
Family: sys.get("Family"),
|
||||
Arch: runtimeArch(),
|
||||
}
|
||||
s.Kernel = readFile("/proc/sys/kernel/osrelease")
|
||||
if s.Model == "" { // no SMBIOS: a Pi, a Mac, anything with a device tree
|
||||
s.Model = strings.TrimRight(readFile("/sys/firmware/devicetree/base/model"), "\x00")
|
||||
}
|
||||
if out, err := exec.CommandContext(ctx, "systemd-detect-virt").Output(); err == nil {
|
||||
if v := strings.TrimSpace(string(out)); v != "" && v != "none" {
|
||||
s.Virt = v
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// board reads the motherboard, the chassis it sits in, the firmware on it and the
|
||||
// expansion slots the firmware claims. All of it is SMBIOS, so a machine without one — a
|
||||
// Mac, a Pi — gets an empty board, and the page says so rather than inventing one.
|
||||
func board(dmi []record, pci map[string]string) Board {
|
||||
b, ch, bios := first(dmi, 2), first(dmi, 3), first(dmi, 0)
|
||||
out := Board{
|
||||
Maker: b.get("Manufacturer"),
|
||||
Product: b.get("Product Name"),
|
||||
Version: b.get("Version"),
|
||||
Serial: b.get("Serial Number"),
|
||||
AssetTag: b.get("Asset Tag"),
|
||||
Type: b.get("Type"),
|
||||
Location: b.get("Location In Chassis"),
|
||||
Chassis: ch.get("Type"),
|
||||
ChassisMaker: ch.get("Manufacturer"),
|
||||
BIOS: join(bios.get("Vendor"), bios.get("Version")),
|
||||
BIOSDate: bios.get("Release Date"),
|
||||
BIOSRevision: bios.get("BIOS Revision"),
|
||||
}
|
||||
// "ROM Size: 0 MB" is a board that never filled the field in, not a board with no ROM
|
||||
if rom := bios.get("ROM Size"); rom != "" && !strings.HasPrefix(rom, "0 ") {
|
||||
out.ROMSize = rom
|
||||
}
|
||||
if out.Product == out.Maker {
|
||||
out.Product = "" // a board named after its maker says nothing twice
|
||||
}
|
||||
out.UEFI = strings.Contains(bios.V[listKey], "UEFI") // a characteristics line, not a field
|
||||
for _, sl := range all(dmi, 9) {
|
||||
s := Slot{
|
||||
Designation: sl.get("Designation"),
|
||||
Type: sl.get("Type"),
|
||||
Usage: sl.get("Current Usage"),
|
||||
Length: sl.get("Length"),
|
||||
Width: sl.get("Data Bus Width"),
|
||||
}
|
||||
// every slot on some boards claims 0000:00:00.0, which is the host bridge and
|
||||
// certainly not what is plugged into that slot
|
||||
if bus := sl.get("Bus Address"); bus != "" && bus != "0000:00:00.0" {
|
||||
s.Bus, s.Device = bus, pci[bus]
|
||||
}
|
||||
if s.Designation != "" {
|
||||
out.Slots = append(out.Slots, s)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func join(parts ...string) string {
|
||||
var keep []string
|
||||
for _, p := range parts {
|
||||
if p = strings.TrimSpace(p); p != "" && !unset(p) {
|
||||
keep = append(keep, p)
|
||||
}
|
||||
}
|
||||
// "GMKtec GMKtec" reads badly: drop a maker the product already names
|
||||
if len(keep) == 2 && strings.HasPrefix(strings.ToLower(keep[1]), strings.ToLower(keep[0])) {
|
||||
return keep[1]
|
||||
}
|
||||
return strings.Join(keep, " ")
|
||||
}
|
||||
|
||||
func readFile(p string) string {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
func runtimeArch() string {
|
||||
if out, err := exec.Command("uname", "-m").Output(); err == nil {
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- processor ----
|
||||
|
||||
// worthShowing is the short list of processor features anyone actually asks about; the
|
||||
// full flag list runs to hundreds and belongs in a terminal.
|
||||
var worthShowing = []string{"aes", "avx", "avx2", "avx512f", "sha_ni", "vmx", "svm", "sse4_2", "rdrand", "hypervisor"}
|
||||
|
||||
func cpu(ctx context.Context, dmi []record) CPU {
|
||||
c := CPU{}
|
||||
f := lscpu(ctx)
|
||||
c.Model = f["Model name"]
|
||||
c.Vendor = f["Vendor ID"]
|
||||
c.Arch = f["Architecture"]
|
||||
c.Sockets = atoi(f["Socket(s)"])
|
||||
c.ThreadsPerCore = atoi(f["Thread(s) per core"])
|
||||
c.Threads = atoi(f["CPU(s)"])
|
||||
c.Cores = atoi(f["Core(s) per socket"]) * maxInt(1, c.Sockets)
|
||||
c.MinMHz = atof(f["CPU min MHz"])
|
||||
c.MaxMHz = atof(f["CPU max MHz"])
|
||||
c.Family, c.ModelID, c.Stepping = f["CPU family"], f["Model"], f["Stepping"]
|
||||
c.Virtualization = f["Virtualization"]
|
||||
for _, l := range []string{"L1d", "L1i", "L2", "L3"} {
|
||||
if v := f[l+" cache"]; v != "" {
|
||||
c.Caches = append(c.Caches, Cache{Level: l, Size: v})
|
||||
}
|
||||
}
|
||||
have := map[string]bool{}
|
||||
for _, fl := range strings.Fields(f["Flags"]) {
|
||||
have[fl] = true
|
||||
}
|
||||
for _, fl := range worthShowing {
|
||||
if have[fl] {
|
||||
c.Flags = append(c.Flags, fl)
|
||||
}
|
||||
}
|
||||
for _, p := range all(dmi, 4) {
|
||||
if s := p.get("Socket Designation"); s != "" {
|
||||
c.SocketNames = append(c.SocketNames, s)
|
||||
}
|
||||
if c.Model == "" {
|
||||
c.Model = p.get("Version", "Family")
|
||||
}
|
||||
if c.MaxMHz == 0 {
|
||||
if mhz := atof(strings.TrimSuffix(p.get("Max Speed"), " MHz")); mhz > 0 {
|
||||
c.MaxMHz = mhz
|
||||
}
|
||||
}
|
||||
}
|
||||
if c.Sockets == 0 {
|
||||
c.Sockets = len(c.SocketNames)
|
||||
}
|
||||
// A processor whose cores are not all alike — Apple's, Intel's P and E cores — is
|
||||
// reported by lscpu as several groups, and "core(s) per socket" then counts one group
|
||||
// rather than the chip. Without SMT every thread is a core, so the thread count is the
|
||||
// honest answer.
|
||||
if c.ThreadsPerCore == 1 && c.Threads > c.Cores {
|
||||
c.Cores = c.Threads
|
||||
}
|
||||
// /proc/cpuinfo is the only place the running microcode revision is written down
|
||||
if b, err := os.ReadFile("/proc/cpuinfo"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if k, v, ok := strings.Cut(line, ":"); ok && strings.TrimSpace(k) == "microcode" {
|
||||
c.Microcode = strings.TrimSpace(v)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if c.Model == "" {
|
||||
c.Model = "unknown processor"
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func lscpu(ctx context.Context) map[string]string {
|
||||
f := map[string]string{}
|
||||
out, err := exec.CommandContext(ctx, "lscpu").Output()
|
||||
if err != nil {
|
||||
// no util-linux: /proc/cpuinfo still names the thing and counts it
|
||||
if b, err := os.ReadFile("/proc/cpuinfo"); err == nil {
|
||||
n := 0
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
k, v, ok := strings.Cut(line, ":")
|
||||
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
|
||||
switch {
|
||||
case ok && strings.EqualFold(k, "model name"):
|
||||
f["Model name"] = v
|
||||
case ok && k == "processor":
|
||||
n++
|
||||
}
|
||||
}
|
||||
f["CPU(s)"] = strconv.Itoa(n)
|
||||
}
|
||||
return f
|
||||
}
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
k, v, ok := strings.Cut(line, ":")
|
||||
if !ok || strings.HasPrefix(line, " ") {
|
||||
continue
|
||||
}
|
||||
f[strings.TrimSpace(k)] = strings.TrimSpace(v)
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
func atoi(s string) int { n, _ := strconv.Atoi(strings.TrimSpace(s)); return n }
|
||||
|
||||
func atof(s string) float64 {
|
||||
v, _ := strconv.ParseFloat(strings.TrimSpace(s), 64)
|
||||
return v
|
||||
}
|
||||
|
||||
func maxInt(a, b int) int {
|
||||
if a > b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// leadingInt reads the number off the front of "3200 MT/s".
|
||||
func leadingInt(s string) int {
|
||||
f := strings.Fields(s)
|
||||
if len(f) == 0 {
|
||||
return 0
|
||||
}
|
||||
return atoi(f[0])
|
||||
}
|
||||
|
||||
// ---- memory ----
|
||||
|
||||
var sizeRe = regexp.MustCompile(`^(\d+)\s*(kB|KB|MB|GB|TB)$`)
|
||||
|
||||
// dmiSize turns dmidecode's "8 GB" / "16384 MB" into bytes.
|
||||
func dmiSize(s string) uint64 {
|
||||
m := sizeRe.FindStringSubmatch(strings.TrimSpace(s))
|
||||
if m == nil {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.ParseUint(m[1], 10, 64)
|
||||
switch strings.ToUpper(m[2]) {
|
||||
case "KB":
|
||||
return n << 10
|
||||
case "MB":
|
||||
return n << 20
|
||||
case "GB":
|
||||
return n << 30
|
||||
case "TB":
|
||||
return n << 40
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func memory(dmi []record) Memory {
|
||||
m := Memory{}
|
||||
if b, err := os.ReadFile("/proc/meminfo"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if k, v, ok := strings.Cut(line, ":"); ok && k == "MemTotal" {
|
||||
m.Total = uint64(atoi(strings.TrimSuffix(strings.TrimSpace(v), " kB"))) << 10
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, a := range all(dmi, 16) {
|
||||
m.MaxCapacity += dmiSize(a.get("Maximum Capacity"))
|
||||
switch e := a.V["Error Correction Type"]; {
|
||||
case e == "None":
|
||||
m.ECC = "non-ECC" // the array says so outright; worth knowing, not worth hiding
|
||||
case e != "" && !unset(e):
|
||||
m.ECC = e
|
||||
}
|
||||
}
|
||||
for _, d := range all(dmi, 17) {
|
||||
mod := Module{
|
||||
Locator: d.get("Locator"),
|
||||
Bank: d.get("Bank Locator"),
|
||||
Size: dmiSize(d.get("Size")),
|
||||
}
|
||||
m.Slots++
|
||||
if mod.Locator == "" {
|
||||
mod.Locator = "slot " + strconv.Itoa(m.Slots)
|
||||
}
|
||||
if mod.Size == 0 {
|
||||
mod.Empty = true
|
||||
m.Modules = append(m.Modules, mod)
|
||||
continue
|
||||
}
|
||||
m.Filled++
|
||||
mod.Type = d.get("Type")
|
||||
mod.FormFactor = d.get("Form Factor")
|
||||
mod.SpeedMTs = leadingInt(d.get("Speed"))
|
||||
mod.RunningMTs = leadingInt(d.get("Configured Memory Speed", "Configured Clock Speed"))
|
||||
mod.Maker = makerName(d.get("Manufacturer"), d.V["Module Manufacturer ID"])
|
||||
mod.Part = d.get("Part Number")
|
||||
mod.Serial = d.get("Serial Number")
|
||||
mod.Rank = atoi(d.get("Rank"))
|
||||
mod.Voltage = d.get("Configured Voltage")
|
||||
m.Installed += mod.Size
|
||||
m.Modules = append(m.Modules, mod)
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// jedec maps the memory makers who turn up in a homelab to their names: SPD carries a
|
||||
// JEDEC id, and dmidecode prints the raw code when the module's string field is empty.
|
||||
var jedec = map[string]string{
|
||||
"0X2C00": "Micron", "0X802C": "Micron",
|
||||
"0XCE00": "Samsung", "0X80CE": "Samsung", "0X00CE": "Samsung",
|
||||
"0XAD00": "SK hynix", "0X80AD": "SK hynix",
|
||||
"0X9801": "Kingston", "0X7F98": "Kingston", "0X0198": "Kingston",
|
||||
"0X859B": "Crucial", "0X04CB": "A-DATA", "0X04CD": "G.Skill",
|
||||
"0X029E": "Corsair", "0X0B45": "Wilk Elektronik", "0X830B": "Wilk Elektronik",
|
||||
"0X0443": "Ramaxel", "0X0194": "Smart Modular",
|
||||
}
|
||||
|
||||
// makerName prefers what the module says it is; a bare JEDEC id becomes a name when it is
|
||||
// one we know, and stays the id when it is not — a wrong name is worse than a number.
|
||||
func makerName(v, id string) string {
|
||||
if v != "" && !unset(v) && !strings.HasPrefix(v, "0x") {
|
||||
return v
|
||||
}
|
||||
code := v
|
||||
if code == "" {
|
||||
code = id
|
||||
}
|
||||
if n, ok := jedec[strings.ToUpper(code)]; ok {
|
||||
return n
|
||||
}
|
||||
if v != "" && !unset(v) {
|
||||
return v
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- drives ----
|
||||
|
||||
func drives(ctx context.Context) []Drive {
|
||||
devs, err := storage.Devices(ctx)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]Drive, 0, len(devs))
|
||||
for _, d := range devs {
|
||||
out = append(out, Drive{
|
||||
Name: d.Name, Kind: d.Kind, Model: d.Model, Vendor: d.Vendor, Serial: d.Serial,
|
||||
Tran: d.Tran, Size: d.Size, Rotational: d.Rotational, Removable: d.Removable,
|
||||
Firmware: firmwareOf(d.Name),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// firmwareOf reads a drive's firmware revision. An NVMe device is folded to its
|
||||
// controller ("nvme0"), which has no /sys/block entry of its own — the revision is on the
|
||||
// controller in /sys/class/nvme.
|
||||
func firmwareOf(name string) string {
|
||||
return firstNonEmpty(
|
||||
readFile("/sys/block/"+name+"/device/firmware_rev"),
|
||||
readFile("/sys/block/"+name+"/device/rev"),
|
||||
readFile("/sys/class/nvme/"+name+"/firmware_rev"),
|
||||
)
|
||||
}
|
||||
|
||||
func firstNonEmpty(v ...string) string {
|
||||
for _, s := range v {
|
||||
if s != "" {
|
||||
return s
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- graphics ----
|
||||
|
||||
// lspci -mm quotes every field, so a device name can hold the spaces and brackets it
|
||||
// likes: `"VGA compatible controller [0300]" "Intel Corporation [8086]" "Alder Lake-N …"`.
|
||||
var quoted = regexp.MustCompile(`"([^"]*)"`)
|
||||
|
||||
// stripID drops the trailing PCI id from `Alder Lake-N [UHD Graphics] [46d1]`.
|
||||
func stripID(s string) string {
|
||||
if i := strings.LastIndex(s, " ["); i > 0 && strings.HasSuffix(s, "]") {
|
||||
return strings.TrimSpace(s[:i])
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func gpus(ctx context.Context) []GPU {
|
||||
var out []GPU
|
||||
seen := map[string]bool{} // PCI addresses already accounted for
|
||||
if _, err := exec.LookPath("lspci"); err == nil {
|
||||
drivers := pciDrivers(ctx)
|
||||
if b, err := exec.CommandContext(ctx, "lspci", "-D", "-mm", "-nn").Output(); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
slot, _, _ := strings.Cut(line, " ")
|
||||
fields := quoted.FindAllStringSubmatch(line, -1)
|
||||
if len(fields) < 3 {
|
||||
continue
|
||||
}
|
||||
kind := displayClass(fields[0][1])
|
||||
if kind == "" {
|
||||
continue
|
||||
}
|
||||
seen[slot] = true
|
||||
g := GPU{Slot: slot, Class: kind, Vendor: stripID(fields[1][1]), Model: stripID(fields[2][1]), Driver: drivers[slot]}
|
||||
g.Node, g.Memory = drmFor(slot)
|
||||
out = append(out, g)
|
||||
}
|
||||
}
|
||||
}
|
||||
// A GPU that is not on a PCI bus at all — an Apple Silicon Mac, a Pi, anything with a
|
||||
// device tree — is only visible as a kernel display device.
|
||||
cards, _ := filepath.Glob("/sys/class/drm/card[0-9]")
|
||||
for _, c := range cards {
|
||||
dev, err := filepath.EvalSymlinks(c + "/device")
|
||||
if err != nil || seen[filepath.Base(dev)] {
|
||||
continue
|
||||
}
|
||||
g := GPU{Node: filepath.Base(c), Class: "Display"}
|
||||
if drv, err := filepath.EvalSymlinks(dev + "/driver"); err == nil {
|
||||
g.Driver = filepath.Base(drv)
|
||||
}
|
||||
g.Model = deviceTreeName(dev)
|
||||
if g.Model == "" && g.Driver != "" {
|
||||
g.Model = g.Driver + " display"
|
||||
}
|
||||
if g.Model == "" {
|
||||
continue
|
||||
}
|
||||
g.Memory = vram(dev)
|
||||
out = append(out, g)
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].Slot < out[j].Slot })
|
||||
return out
|
||||
}
|
||||
|
||||
// pciNames maps a PCI address to a readable name, so a slot the firmware says is in use
|
||||
// can say what is in it.
|
||||
func pciNames(ctx context.Context) map[string]string {
|
||||
m := map[string]string{}
|
||||
if _, err := exec.LookPath("lspci"); err != nil {
|
||||
return m
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, "lspci", "-D", "-mm", "-nn").Output()
|
||||
if err != nil {
|
||||
return m
|
||||
}
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
slot, _, _ := strings.Cut(line, " ")
|
||||
f := quoted.FindAllStringSubmatch(line, -1)
|
||||
if slot == "" || len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
m[slot] = strings.TrimSpace(stripID(f[1][1]) + " " + stripID(f[2][1]))
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func displayClass(c string) string {
|
||||
switch {
|
||||
case strings.Contains(c, "[0300]"):
|
||||
return "VGA"
|
||||
case strings.Contains(c, "[0302]"):
|
||||
return "3D"
|
||||
case strings.Contains(c, "[0301]"), strings.Contains(c, "[0380]"):
|
||||
return "Display"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// pciDrivers maps a PCI address to the kernel module driving it.
|
||||
func pciDrivers(ctx context.Context) map[string]string {
|
||||
m := map[string]string{}
|
||||
out, err := exec.CommandContext(ctx, "lspci", "-D", "-k").Output()
|
||||
if err != nil {
|
||||
return m
|
||||
}
|
||||
slot := ""
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
if !strings.HasPrefix(line, "\t") && !strings.HasPrefix(line, " ") {
|
||||
slot, _, _ = strings.Cut(line, " ")
|
||||
continue
|
||||
}
|
||||
if v, ok := strings.CutPrefix(strings.TrimSpace(line), "Kernel driver in use:"); ok && slot != "" {
|
||||
m[slot] = strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// drmFor finds the kernel display device for a PCI address, and its video memory when the
|
||||
// driver publishes it (amdgpu does; Intel's integrated graphics have none of their own).
|
||||
func drmFor(slot string) (string, uint64) {
|
||||
cards, _ := filepath.Glob("/sys/class/drm/card[0-9]")
|
||||
for _, c := range cards {
|
||||
dev, err := filepath.EvalSymlinks(c + "/device")
|
||||
if err != nil || filepath.Base(dev) != slot {
|
||||
continue
|
||||
}
|
||||
return filepath.Base(c), vram(dev)
|
||||
}
|
||||
return "", 0
|
||||
}
|
||||
|
||||
func vram(dev string) uint64 {
|
||||
for _, f := range []string{"mem_info_vram_total", "mem_info_vis_vram_total"} {
|
||||
if v := readFile(dev + "/" + f); v != "" {
|
||||
n, _ := strconv.ParseUint(v, 10, 64)
|
||||
return n
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// deviceTreeName reads what a device-tree machine calls its graphics: the compatible
|
||||
// string, tidied — "apple,agx-g13g" becomes "Apple agx-g13g".
|
||||
func deviceTreeName(dev string) string {
|
||||
c := readFile(dev + "/of_node/compatible")
|
||||
if c == "" {
|
||||
return ""
|
||||
}
|
||||
c = strings.TrimRight(strings.Split(c, "\x00")[0], "\x00")
|
||||
maker, rest, ok := strings.Cut(c, ",")
|
||||
if !ok {
|
||||
return c
|
||||
}
|
||||
if maker != "" {
|
||||
maker = strings.ToUpper(maker[:1]) + maker[1:]
|
||||
}
|
||||
return strings.TrimSpace(maker + " " + rest)
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
package hardware
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestDMISize(t *testing.T) {
|
||||
for in, want := range map[string]uint64{
|
||||
"8 GB": 8 << 30, "16384 MB": 16384 << 20, "512 kB": 512 << 10, "1 TB": 1 << 40,
|
||||
"No Module Installed": 0, "": 0, "8 GiB": 0,
|
||||
} {
|
||||
if got := dmiSize(in); got != want {
|
||||
t.Errorf("dmiSize(%q) = %d, want %d", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMakerName(t *testing.T) {
|
||||
cases := []struct{ v, id, want string }{
|
||||
{"Samsung", "", "Samsung"}, // the module says who it is
|
||||
{"0x0B45", "Bank 12, Hex 0x45", "Wilk Elektronik"}, // only a JEDEC id, and one we know
|
||||
{"0xFFFF", "", "0xFFFF"}, // one we don't: the number, not a guess
|
||||
{"Unknown", "", ""},
|
||||
{"", "0xCE00", "Samsung"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := makerName(c.v, c.id); got != c.want {
|
||||
t.Errorf("makerName(%q, %q) = %q, want %q", c.v, c.id, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStripIDAndDisplayClass(t *testing.T) {
|
||||
if got := stripID("Alder Lake-N [UHD Graphics] [46d1]"); got != "Alder Lake-N [UHD Graphics]" {
|
||||
t.Errorf("stripID = %q", got)
|
||||
}
|
||||
if got := stripID("no brackets here"); got != "no brackets here" {
|
||||
t.Errorf("stripID left a plain name alone wrong: %q", got)
|
||||
}
|
||||
if displayClass(`VGA compatible controller [0300]`) != "VGA" {
|
||||
t.Error("a VGA controller should be graphics")
|
||||
}
|
||||
if displayClass(`Ethernet controller [0200]`) != "" {
|
||||
t.Error("a network card is not graphics")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLeadingIntAndJoin(t *testing.T) {
|
||||
if leadingInt("3200 MT/s") != 3200 || leadingInt("") != 0 || leadingInt("Unknown") != 0 {
|
||||
t.Error("leadingInt")
|
||||
}
|
||||
if got := join("GMKtec", "GMKtec NucBox G3"); got != "GMKtec NucBox G3" {
|
||||
t.Errorf("a maker the product already names should not be said twice: %q", got)
|
||||
}
|
||||
if got := join("Dell Inc.", "OptiPlex 7040"); got != "Dell Inc. OptiPlex 7040" {
|
||||
t.Errorf("join = %q", got)
|
||||
}
|
||||
if got := join("To Be Filled By O.E.M.", "System Product Name"); got != "" {
|
||||
t.Errorf("firmware boilerplate is not a model: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
const sample = `# dmidecode 3.6
|
||||
Handle 0x0028, DMI type 17, 92 bytes
|
||||
Memory Device
|
||||
Array Handle: 0x0027
|
||||
Size: 8 GB
|
||||
Form Factor: SODIMM
|
||||
Locator: Controller0-ChannelA-DIMM0
|
||||
Type: DDR4
|
||||
Speed: 3200 MT/s
|
||||
Manufacturer: 0x0B45
|
||||
Part Number: WPBH32D408SWD-8G
|
||||
Configured Memory Speed: 2933 MT/s
|
||||
Rank: 1
|
||||
|
||||
Handle 0x0029, DMI type 17, 92 bytes
|
||||
Memory Device
|
||||
Size: No Module Installed
|
||||
Locator: Controller1-ChannelA-DIMM0
|
||||
Type: Unknown
|
||||
`
|
||||
|
||||
// parseRecords is what readDMI does once dmidecode has answered; this exercises it
|
||||
// without needing root or the tool.
|
||||
func TestMemoryFromRecords(t *testing.T) {
|
||||
recs := parse(sample)
|
||||
m := memory(recs)
|
||||
if m.Slots != 2 || m.Filled != 1 {
|
||||
t.Fatalf("slots %d filled %d, want 2 and 1", m.Slots, m.Filled)
|
||||
}
|
||||
mod := m.Modules[0]
|
||||
if mod.Size != 8<<30 || mod.Type != "DDR4" || mod.Maker != "Wilk Elektronik" || mod.Part != "WPBH32D408SWD-8G" {
|
||||
t.Errorf("first module read wrong: %+v", mod)
|
||||
}
|
||||
if mod.SpeedMTs != 3200 || mod.RunningMTs != 2933 {
|
||||
t.Errorf("a module running below its rating should say both: %d / %d", mod.RunningMTs, mod.SpeedMTs)
|
||||
}
|
||||
if !m.Modules[1].Empty || m.Modules[1].Locator != "Controller1-ChannelA-DIMM0" {
|
||||
t.Errorf("empty slot read wrong: %+v", m.Modules[1])
|
||||
}
|
||||
if m.Installed != 8<<30 {
|
||||
t.Errorf("installed = %d", m.Installed)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
bolt "go.etcd.io/bbolt"
|
||||
)
|
||||
|
||||
// Cache keeps finished previews in one database file under Ham Lab's own data directory —
|
||||
// thumbs.db, a bbolt key/value store: pure Go, so the binary stays static, and one file
|
||||
// per machine rather than a tree of hash-named JPEGs. An entry is one file's preview keyed
|
||||
// by its path, size and mtime, so a file that changes gets a new preview and an unchanged
|
||||
// one is never decoded twice. A file ffmpeg could not decode is kept the same way, with
|
||||
// ffmpeg's reason and no picture, so a folder of odd files does not re-run ffmpeg on every
|
||||
// visit. Nothing expires: a preview is a few kilobytes, and only files somebody actually
|
||||
// scrolled past are ever made. The file is opened the first time something is stored, so a
|
||||
// machine nobody browses with previews on never has one.
|
||||
type Cache struct {
|
||||
Path string
|
||||
mu sync.Mutex
|
||||
db *bolt.DB
|
||||
err error
|
||||
}
|
||||
|
||||
// Entry is one cached preview, or the remembered reason there is none.
|
||||
type Entry struct {
|
||||
Path string `json:"path"`
|
||||
Engine string `json:"engine,omitempty"`
|
||||
Codec string `json:"codec,omitempty"`
|
||||
Width int `json:"width,omitempty"`
|
||||
Height int `json:"height,omitempty"`
|
||||
Made time.Time `json:"made"`
|
||||
Failure string `json:"failure,omitempty"` // set instead of JPEG when ffmpeg refused the file
|
||||
JPEG []byte `json:"-"`
|
||||
}
|
||||
|
||||
var (
|
||||
bucketThumbs = []byte("thumbs")
|
||||
bucketMeta = []byte("meta")
|
||||
keyCount = []byte("count")
|
||||
)
|
||||
|
||||
// Key identifies one file's preview. "v1" is the preview format: bump it to retire every
|
||||
// cached preview at once.
|
||||
func Key(path string, size, mtime int64) string {
|
||||
h := sha256.Sum256([]byte("v1\x00" + path + "\x00" + strconv.FormatInt(size, 10) + "\x00" + strconv.FormatInt(mtime, 10)))
|
||||
return hex.EncodeToString(h[:])
|
||||
}
|
||||
|
||||
func NewCache(path string) *Cache { return &Cache{Path: path} }
|
||||
|
||||
// open the database once; an open that failed is not retried, and previews still get made
|
||||
// and served — they are simply not kept, and the log says why.
|
||||
func (c *Cache) open() (*bolt.DB, error) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.db != nil || c.err != nil {
|
||||
return c.db, c.err
|
||||
}
|
||||
db, err := bolt.Open(c.Path, 0o600, &bolt.Options{Timeout: 2 * time.Second})
|
||||
if err != nil {
|
||||
c.err = err
|
||||
return nil, err
|
||||
}
|
||||
if err := db.Update(func(tx *bolt.Tx) error {
|
||||
if _, err := tx.CreateBucketIfNotExists(bucketThumbs); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := tx.CreateBucketIfNotExists(bucketMeta)
|
||||
return err
|
||||
}); err != nil {
|
||||
db.Close()
|
||||
c.err = err
|
||||
return nil, err
|
||||
}
|
||||
c.db = db
|
||||
return db, nil
|
||||
}
|
||||
|
||||
// Close releases the file. Previews after it are made but not kept.
|
||||
func (c *Cache) Close() error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.db == nil {
|
||||
return nil
|
||||
}
|
||||
err := c.db.Close()
|
||||
c.db, c.err = nil, errors.New("cache closed")
|
||||
return err
|
||||
}
|
||||
|
||||
// Get returns the entry for a key. A database that does not exist yet is a miss, not a reason
|
||||
// to make one.
|
||||
func (c *Cache) Get(key string) (Entry, bool, error) {
|
||||
c.mu.Lock()
|
||||
db := c.db
|
||||
c.mu.Unlock()
|
||||
if db == nil {
|
||||
if _, err := os.Stat(c.Path); err != nil {
|
||||
return Entry{}, false, nil
|
||||
}
|
||||
var err error
|
||||
if db, err = c.open(); err != nil {
|
||||
return Entry{}, false, err
|
||||
}
|
||||
}
|
||||
var e Entry
|
||||
found := false
|
||||
err := db.View(func(tx *bolt.Tx) error {
|
||||
v := tx.Bucket(bucketThumbs).Get([]byte(key))
|
||||
if v == nil {
|
||||
return nil
|
||||
}
|
||||
var err error
|
||||
e, err = decode(v) // copies: a value is only valid inside the transaction
|
||||
found = err == nil
|
||||
return err
|
||||
})
|
||||
return e, found, err
|
||||
}
|
||||
|
||||
// Put stores an entry, replacing any under the same key.
|
||||
func (c *Cache) Put(key string, e Entry) error {
|
||||
if len(e.JPEG) == 0 && e.Failure == "" {
|
||||
return errors.New("empty preview")
|
||||
}
|
||||
if e.Made.IsZero() {
|
||||
e.Made = time.Now()
|
||||
}
|
||||
db, err := c.open()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
v, err := encode(e)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return db.Update(func(tx *bolt.Tx) error {
|
||||
b := tx.Bucket(bucketThumbs)
|
||||
k := []byte(key)
|
||||
fresh := b.Get(k) == nil
|
||||
if err := b.Put(k, v); err != nil {
|
||||
return err
|
||||
}
|
||||
if fresh {
|
||||
m := tx.Bucket(bucketMeta)
|
||||
return m.Put(keyCount, be64(count(m)+1))
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// Stats is how many previews are kept and how big the file is.
|
||||
func (c *Cache) Stats() (entries int64, bytes int64) {
|
||||
st, err := os.Stat(c.Path)
|
||||
if err != nil {
|
||||
return 0, 0
|
||||
}
|
||||
bytes = st.Size()
|
||||
db, err := c.open()
|
||||
if err != nil {
|
||||
return 0, bytes
|
||||
}
|
||||
_ = db.View(func(tx *bolt.Tx) error { entries = count(tx.Bucket(bucketMeta)); return nil })
|
||||
return entries, bytes
|
||||
}
|
||||
|
||||
func count(m *bolt.Bucket) int64 {
|
||||
if v := m.Get(keyCount); len(v) == 8 {
|
||||
return int64(binary.BigEndian.Uint64(v))
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func be64(n int64) []byte {
|
||||
b := make([]byte, 8)
|
||||
binary.BigEndian.PutUint64(b, uint64(n))
|
||||
return b
|
||||
}
|
||||
|
||||
// A value is a 4-byte header length, the header as JSON, then the JPEG bytes.
|
||||
func encode(e Entry) ([]byte, error) {
|
||||
hdr, err := json.Marshal(e)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v := make([]byte, 4+len(hdr)+len(e.JPEG))
|
||||
binary.BigEndian.PutUint32(v, uint32(len(hdr)))
|
||||
copy(v[4:], hdr)
|
||||
copy(v[4+len(hdr):], e.JPEG)
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func decode(v []byte) (Entry, error) {
|
||||
if len(v) < 4 {
|
||||
return Entry{}, errors.New("short value")
|
||||
}
|
||||
n := int(binary.BigEndian.Uint32(v))
|
||||
if 4+n > len(v) {
|
||||
return Entry{}, errors.New("bad header length")
|
||||
}
|
||||
var e Entry
|
||||
if err := json.Unmarshal(v[4:4+n], &e); err != nil {
|
||||
return Entry{}, err
|
||||
}
|
||||
if rest := v[4+n:]; len(rest) > 0 {
|
||||
e.JPEG = append([]byte(nil), rest...)
|
||||
}
|
||||
return e, nil
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
package media
|
||||
|
||||
// What to run when a file is to be re-encoded for a browser that cannot play it as it is:
|
||||
// the codecs and sizes on offer, the output size and bit rate a choice comes to for a
|
||||
// given source, the ways this machine could do it in the order worth trying, and the
|
||||
// ffmpeg argument list for one of them. All decisions, no processes, so it can be tested.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"sort"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
// Codec is one target a stream may be re-encoded to. "copy" keeps the codec and only
|
||||
// repackages: for a browser that lacks the container, not the decoder.
|
||||
type Codec struct {
|
||||
ID string `json:"id"`
|
||||
Label string `json:"label"`
|
||||
}
|
||||
|
||||
var Codecs = []Codec{{"h264", "H.264"}, {"hevc", "HEVC"}, {"av1", "AV1"}, {"copy", "Original codec, repackaged"}}
|
||||
|
||||
// Quality is a picture height; the bit rate follows from the size the output ends up at.
|
||||
type Quality struct {
|
||||
ID string `json:"id"`
|
||||
Label string `json:"label"`
|
||||
Height int `json:"height"` // 0: the source's own
|
||||
}
|
||||
|
||||
var Qualities = []Quality{{"original", "Original size", 0}, {"1080p", "1080p", 1080}, {"720p", "720p", 720}, {"480p", "480p", 480}}
|
||||
|
||||
func qualityByID(id string) (Quality, bool) {
|
||||
for _, q := range Qualities {
|
||||
if q.ID == id {
|
||||
return q, true
|
||||
}
|
||||
}
|
||||
return Quality{}, false
|
||||
}
|
||||
|
||||
// ValidChoice says whether a codec and quality are ones on offer — checked before any work.
|
||||
func ValidChoice(codec, quality string) error {
|
||||
if !codecKnown(codec) {
|
||||
return fmt.Errorf("unknown codec %q", codec)
|
||||
}
|
||||
if _, ok := qualityByID(quality); !ok {
|
||||
return fmt.Errorf("unknown quality %q", quality)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func codecKnown(id string) bool {
|
||||
for _, c := range Codecs {
|
||||
if c.ID == id {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// fit is the output: never larger than the source, even dimensions, and a bit rate by the
|
||||
// size it comes to — a quarter more at 50 fps and up, less for the codecs that need less.
|
||||
func fit(q Quality, src Info, codec string) (w, h, kbps int) {
|
||||
w, h = src.Video.Width, src.Video.Height
|
||||
if q.Height > 0 && h > q.Height {
|
||||
w = int(math.Round(float64(w) * float64(q.Height) / float64(h)))
|
||||
h = q.Height
|
||||
}
|
||||
w, h = w&^1, h&^1
|
||||
switch {
|
||||
case h >= 2160:
|
||||
kbps = 20000
|
||||
case h >= 1440:
|
||||
kbps = 12000
|
||||
case h >= 1080:
|
||||
kbps = 8000
|
||||
case h >= 720:
|
||||
kbps = 4000
|
||||
default:
|
||||
kbps = 2000
|
||||
}
|
||||
if src.Video.FPS >= 50 {
|
||||
kbps = kbps * 5 / 4
|
||||
}
|
||||
if codec == "hevc" || codec == "av1" {
|
||||
kbps = kbps * 6 / 10
|
||||
}
|
||||
return w, h, kbps
|
||||
}
|
||||
|
||||
// Recipe is one way to make the stream: the engine's name for the record, and the ffmpeg
|
||||
// arguments that go before the input, the video filter, and the video codec.
|
||||
type Recipe struct {
|
||||
Engine string
|
||||
Pre []string
|
||||
VF string
|
||||
Video []string
|
||||
}
|
||||
|
||||
// recipes are the ways to try, best first: a card on the bus before the integrated GPU;
|
||||
// on a GPU, decoding on the same card (frames never leave it) before decoding in software
|
||||
// and uploading; the software encoders last. An empty list means nothing here can.
|
||||
func recipes(c Caps, codec string, src Info, w, h, kbps int) []Recipe {
|
||||
if codec == "copy" {
|
||||
v := []string{"-c:v", "copy"}
|
||||
if src.Video.Codec == "hevc" {
|
||||
v = append(v, "-tag:v", "hvc1") // Safari reads hvc1, not hev1
|
||||
}
|
||||
return []Recipe{{Engine: "Original codec", Video: v}}
|
||||
}
|
||||
var out []Recipe
|
||||
gpus := append([]GPU(nil), c.GPUs...)
|
||||
sort.SliceStable(gpus, func(i, j int) bool { return !gpus[i].Integrated && gpus[j].Integrated })
|
||||
rate := []string{"-b:v", strconv.Itoa(kbps) + "k", "-maxrate", strconv.Itoa(kbps*5/4) + "k", "-bufsize", strconv.Itoa(kbps*5/2) + "k"}
|
||||
for i := range gpus {
|
||||
g := &gpus[i]
|
||||
if g.API != "vaapi" || !has(g.Encodes, codec) {
|
||||
continue
|
||||
}
|
||||
video := append([]string{"-c:v", codec + "_vaapi"}, rate...)
|
||||
switch codec {
|
||||
case "h264":
|
||||
video = append(video, "-profile:v", "high")
|
||||
case "hevc":
|
||||
video = append(video, "-tag:v", "hvc1")
|
||||
}
|
||||
name := g.Name
|
||||
if name == "" {
|
||||
name = g.Device
|
||||
}
|
||||
scale := fmt.Sprintf("scale_vaapi=w=%d:h=%d:format=nv12", w, h)
|
||||
if has(g.Codecs, src.Video.Codec) {
|
||||
vf := scale
|
||||
if src.HDR() {
|
||||
vf = "tonemap_vaapi=format=nv12:t=bt709:p=bt709:m=bt709," + vf
|
||||
}
|
||||
out = append(out, Recipe{Engine: name + " (vaapi)", Pre: []string{"-hwaccel", "vaapi", "-hwaccel_device", g.Device, "-hwaccel_output_format", "vaapi"}, VF: vf, Video: video})
|
||||
}
|
||||
out = append(out, Recipe{Engine: name + " (vaapi, decoded on the CPU)", Pre: []string{"-init_hw_device", "vaapi=va:" + g.Device, "-filter_hw_device", "va"},
|
||||
VF: fmt.Sprintf("format=nv12,scale=%d:%d,hwupload", w, h), Video: video})
|
||||
}
|
||||
swVF := fmt.Sprintf("scale=%d:%d,format=yuv420p", w, h)
|
||||
if src.HDR() {
|
||||
swVF = fmt.Sprintf("zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable,zscale=t=bt709:m=bt709,scale=%d:%d,format=yuv420p", w, h)
|
||||
}
|
||||
cap := []string{"-maxrate", strconv.Itoa(kbps*5/4) + "k", "-bufsize", strconv.Itoa(kbps*5/2) + "k"}
|
||||
switch {
|
||||
case codec == "h264" && has(c.Encoders, "libx264"):
|
||||
out = append(out, Recipe{Engine: "CPU (libx264)", VF: swVF, Video: append([]string{"-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-profile:v", "high"}, cap...)})
|
||||
case codec == "hevc" && has(c.Encoders, "libx265"):
|
||||
out = append(out, Recipe{Engine: "CPU (libx265)", VF: swVF, Video: append([]string{"-c:v", "libx265", "-preset", "fast", "-crf", "26", "-tag:v", "hvc1"}, cap...)})
|
||||
case codec == "av1" && has(c.Encoders, "libsvtav1"):
|
||||
out = append(out, Recipe{Engine: "CPU (SVT-AV1)", VF: swVF, Video: append([]string{"-c:v", "libsvtav1", "-preset", "8", "-crf", "35"}, cap...)})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// EngineFor is the engine the picker will say a codec runs on, "" when none can.
|
||||
func EngineFor(c Caps, codec string) string {
|
||||
if c.Error != "" {
|
||||
return ""
|
||||
}
|
||||
if codec == "copy" {
|
||||
return "Original codec"
|
||||
}
|
||||
r := recipes(c, codec, Info{Video: &Track{Codec: "h264", Width: 1920, Height: 1080}}, 1920, 1080, 8000)
|
||||
if len(r) == 0 {
|
||||
return ""
|
||||
}
|
||||
return r[0].Engine
|
||||
}
|
||||
|
||||
// hlsArgs is the whole ffmpeg argument list for one recipe: the input, the first video and
|
||||
// audio streams, the picture through the recipe, sound as AAC unless it already is, a
|
||||
// keyframe every segment, and an event playlist of fragmented-MP4 segments in dir.
|
||||
func hlsArgs(dir, in string, src Info, r Recipe, copyCodec bool) []string {
|
||||
a := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
|
||||
a = append(a, r.Pre...)
|
||||
a = append(a, "-i", in, "-map", "0:v:0", "-map", "0:a:0?", "-sn", "-dn")
|
||||
if r.VF != "" {
|
||||
a = append(a, "-vf", r.VF)
|
||||
}
|
||||
a = append(a, r.Video...)
|
||||
switch {
|
||||
case src.Audio == nil:
|
||||
a = append(a, "-an")
|
||||
case src.Audio.Codec == "aac":
|
||||
a = append(a, "-c:a", "copy")
|
||||
default:
|
||||
a = append(a, "-c:a", "aac", "-b:a", "160k", "-ac", "2")
|
||||
}
|
||||
if !copyCodec {
|
||||
fps := src.Video.FPS
|
||||
if fps <= 0 {
|
||||
fps = 30
|
||||
}
|
||||
gop := int(math.Round(fps * hlsSeconds))
|
||||
if gop < 24 {
|
||||
gop = 24
|
||||
}
|
||||
if gop > 300 {
|
||||
gop = 300
|
||||
}
|
||||
a = append(a, "-g", strconv.Itoa(gop), "-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%d)", hlsSeconds))
|
||||
}
|
||||
a = append(a, "-f", "hls", "-hls_time", strconv.Itoa(hlsSeconds), "-hls_playlist_type", "event", "-hls_segment_type", "fmp4",
|
||||
"-hls_fmp4_init_filename", "init.mp4", "-hls_segment_filename", dir+"/seg%05d.m4s", "-hls_flags", "independent_segments+temp_file",
|
||||
"-progress", dir+"/progress", "-stats_period", "1", dir+"/index.m3u8")
|
||||
return a
|
||||
}
|
||||
|
||||
const hlsSeconds = 4
|
||||
|
||||
func has(list []string, x string) bool {
|
||||
for _, v := range list {
|
||||
if v == x {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func src(codec string, w, h int, fps float64) Info {
|
||||
return Info{Duration: 100, Video: &Track{Codec: codec, Width: w, Height: h, FPS: fps}, Audio: &Track{Codec: "opus", Channels: 2}}
|
||||
}
|
||||
|
||||
func TestFit(t *testing.T) {
|
||||
q := func(id string) Quality { x, _ := qualityByID(id); return x }
|
||||
for _, c := range []struct {
|
||||
q string
|
||||
in Info
|
||||
codec string
|
||||
w, h, kbs int
|
||||
}{
|
||||
{"1080p", src("hevc", 3840, 2160, 60), "h264", 1920, 1080, 10000},
|
||||
{"original", src("hevc", 3840, 2160, 30), "h264", 3840, 2160, 20000},
|
||||
{"720p", src("h264", 2292, 960, 60), "h264", 1718, 720, 5000}, // odd width made even
|
||||
{"1080p", src("h264", 1280, 720, 30), "h264", 1280, 720, 4000}, // never upscaled
|
||||
{"480p", src("h264", 1920, 1080, 30), "hevc", 852, 480, 1200}, // 853.3 rounds down, then even
|
||||
{"original", src("av1", 640, 360, 24), "av1", 640, 360, 1200},
|
||||
} {
|
||||
w, h, k := fit(q(c.q), c.in, c.codec)
|
||||
if w != c.w || h != c.h || k != c.kbs {
|
||||
t.Errorf("%s %s %dx%d@%.0f → %dx%d %dk, want %dx%d %dk", c.q, c.codec, c.in.Video.Width, c.in.Video.Height, c.in.Video.FPS, w, h, k, c.w, c.h, c.kbs)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func gridCaps() Caps {
|
||||
return Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac", "libsvtav1", "libx264", "libx265"},
|
||||
GPUs: []GPU{
|
||||
{API: "vaapi", Device: "/dev/dri/renderD128", Name: "Intel UHD Graphics 750", Integrated: true, Codecs: []string{"h264", "hevc", "vp9"}, Encodes: []string{"h264", "hevc"}},
|
||||
{API: "vaapi", Device: "/dev/dri/renderD129", Name: "Intel Arc A380", Codecs: []string{"h264", "hevc", "vp9"}, Encodes: []string{"av1", "h264", "hevc"}},
|
||||
}}
|
||||
}
|
||||
|
||||
func engineNames(r []Recipe) []string {
|
||||
out := make([]string, len(r))
|
||||
for i := range r {
|
||||
out[i] = r[i].Engine
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestRecipesOrder(t *testing.T) {
|
||||
c := gridCaps()
|
||||
// h264 from hevc: the card first, decoding on it; then the iGPU the same way; then the CPU
|
||||
got := engineNames(recipes(c, "h264", src("hevc", 3840, 2160, 60), 1920, 1080, 10000))
|
||||
want := []string{"Intel Arc A380 (vaapi)", "Intel Arc A380 (vaapi, decoded on the CPU)", "Intel UHD Graphics 750 (vaapi)", "Intel UHD Graphics 750 (vaapi, decoded on the CPU)", "CPU (libx264)"}
|
||||
if strings.Join(got, "|") != strings.Join(want, "|") {
|
||||
t.Fatalf("got %v", got)
|
||||
}
|
||||
// av1 source: neither card decodes it, so no same-card recipe, and only the Arc encodes av1
|
||||
got = engineNames(recipes(c, "av1", src("av1", 1920, 1080, 30), 1920, 1080, 4800))
|
||||
want = []string{"Intel Arc A380 (vaapi, decoded on the CPU)", "CPU (SVT-AV1)"}
|
||||
if strings.Join(got, "|") != strings.Join(want, "|") {
|
||||
t.Fatalf("av1: got %v", got)
|
||||
}
|
||||
// copy is one recipe, no filter, hvc1 for hevc
|
||||
r := recipes(c, "copy", src("hevc", 1920, 1080, 30), 1920, 1080, 0)
|
||||
if len(r) != 1 || r[0].VF != "" || !has(r[0].Video, "hvc1") || !has(r[0].Video, "copy") {
|
||||
t.Fatalf("copy: %+v", r)
|
||||
}
|
||||
// a machine with no GPU and no x264: nothing for h264, something for av1
|
||||
bare := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac", "libsvtav1"}}
|
||||
if len(recipes(bare, "h264", src("h264", 1280, 720, 30), 1280, 720, 4000)) != 0 {
|
||||
t.Fatal("no x264 means no h264")
|
||||
}
|
||||
if EngineFor(bare, "av1") != "CPU (SVT-AV1)" || EngineFor(bare, "h264") != "" || EngineFor(bare, "copy") != "Original codec" {
|
||||
t.Fatal("engineFor")
|
||||
}
|
||||
if EngineFor(c, "hevc") != "Intel Arc A380 (vaapi)" {
|
||||
t.Fatalf("engineFor on grid: %q", EngineFor(c, "hevc"))
|
||||
}
|
||||
// HDR gets tone-mapped on both paths
|
||||
hdr := src("hevc", 3840, 2160, 60)
|
||||
hdr.Video.ColorTransfer = "smpte2084"
|
||||
r = recipes(c, "h264", hdr, 1920, 1080, 10000)
|
||||
if !strings.HasPrefix(r[0].VF, "tonemap_vaapi") || !strings.Contains(r[len(r)-1].VF, "tonemap=hable") {
|
||||
t.Fatalf("hdr: %q … %q", r[0].VF, r[len(r)-1].VF)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHLSArgs(t *testing.T) {
|
||||
in := src("h264", 1920, 1080, 60)
|
||||
r := recipes(gridCaps(), "h264", in, 1920, 1080, 10000)[0]
|
||||
a := strings.Join(hlsArgs("/d", "/v.mkv", in, r, false), " ")
|
||||
for _, want := range []string{"-hwaccel vaapi -hwaccel_device /dev/dri/renderD129", "-i /v.mkv", "-map 0:v:0 -map 0:a:0?", "-vf scale_vaapi=w=1920:h=1080:format=nv12", "-c:v h264_vaapi -b:v 10000k -maxrate 12500k", "-c:a aac -b:a 160k -ac 2", "-g 240 -force_key_frames expr:gte(t,n_forced*4)", "-hls_segment_type fmp4", "-hls_segment_filename /d/seg%05d.m4s", "/d/index.m3u8"} {
|
||||
if !strings.Contains(a, want) {
|
||||
t.Errorf("missing %q in\n%s", want, a)
|
||||
}
|
||||
}
|
||||
// aac stays as it is; no audio means none; copy sets no keyframes
|
||||
in.Audio.Codec = "aac"
|
||||
if a := strings.Join(hlsArgs("/d", "/v", in, r, false), " "); !strings.Contains(a, "-c:a copy") {
|
||||
t.Error("aac should be copied")
|
||||
}
|
||||
in.Audio = nil
|
||||
if a := strings.Join(hlsArgs("/d", "/v", in, r, false), " "); !strings.Contains(a, " -an ") {
|
||||
t.Error("no audio stream → -an")
|
||||
}
|
||||
cp := recipes(gridCaps(), "copy", in, 1920, 1080, 0)[0]
|
||||
if a := strings.Join(hlsArgs("/d", "/v", in, cp, true), " "); strings.Contains(a, "force_key_frames") || strings.Contains(a, "-vf") {
|
||||
t.Error("copy must not ask for keyframes or filter")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseProgress(t *testing.T) {
|
||||
b := []byte("frame=10\nout_time_us=1000000\nspeed=2.5x\nprogress=continue\nframe=20\nout_time_us=2500000\nspeed=3.1x\nprogress=continue\n")
|
||||
s, sp := parseProgress(b)
|
||||
if s != 2.5 || sp != 3.1 {
|
||||
t.Fatalf("got %v %v", s, sp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidChoice(t *testing.T) {
|
||||
if err := ValidChoice("h264", "720p"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ValidChoice("vp8", "720p") == nil || ValidChoice("h264", "9p") == nil || ValidChoice("", "") == nil {
|
||||
t.Fatal("bad choices must be refused")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
package media
|
||||
|
||||
// Running one long ffmpeg: as the daemon's user directly, or as root through this binary's
|
||||
// run-ffmpeg helper when the file needs root to read. The helper is told to finish by its
|
||||
// standard input closing, which also happens by itself if the daemon goes away — so a
|
||||
// transcode never outlives the console that asked for it.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type ffproc struct {
|
||||
cmd *exec.Cmd
|
||||
in io.WriteCloser // the helper's stdin, root mode only
|
||||
done chan struct{}
|
||||
err error
|
||||
tail *tailBuf
|
||||
}
|
||||
|
||||
// startFF launches ffmpeg with args, under nice where there is one.
|
||||
func startFF(c Caps, self string, root bool, args []string) (*ffproc, error) {
|
||||
var argv []string
|
||||
if root {
|
||||
argv = priv.Wrap(append([]string{self, "run-ffmpeg"}, args...)...)
|
||||
} else {
|
||||
argv = append([]string{c.FFmpeg}, args...)
|
||||
if c.Nice != "" {
|
||||
argv = append([]string{c.Nice, "-n", "10"}, argv...)
|
||||
}
|
||||
}
|
||||
cmd := exec.Command(argv[0], argv[1:]...)
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
p := &ffproc{cmd: cmd, done: make(chan struct{}), tail: &tailBuf{}}
|
||||
cmd.Stderr = p.tail
|
||||
cmd.Stdout = io.Discard
|
||||
if root {
|
||||
in, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
p.in = in
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
go func() {
|
||||
p.err = cmd.Wait()
|
||||
close(p.done)
|
||||
}()
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// exited reports whether ffmpeg has finished, and how.
|
||||
func (p *ffproc) exited() (bool, error) {
|
||||
select {
|
||||
case <-p.done:
|
||||
return true, p.err
|
||||
default:
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
// end asks ffmpeg to finish and waits a little for it.
|
||||
func (p *ffproc) end() {
|
||||
if done, _ := p.exited(); done {
|
||||
return
|
||||
}
|
||||
if p.in != nil {
|
||||
p.in.Close()
|
||||
} else {
|
||||
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGTERM)
|
||||
}
|
||||
select {
|
||||
case <-p.done:
|
||||
case <-time.After(5 * time.Second):
|
||||
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGKILL) // a root helper ignores this; it ends on its own within its own grace
|
||||
select {
|
||||
case <-p.done:
|
||||
case <-time.After(10 * time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// lastWords is what ffmpeg said last, for the record.
|
||||
func (p *ffproc) lastWords() string { return p.tail.last() }
|
||||
|
||||
// tailBuf keeps the last few KB written to it.
|
||||
type tailBuf struct {
|
||||
mu sync.Mutex
|
||||
b []byte
|
||||
}
|
||||
|
||||
const tailKeep = 4096
|
||||
|
||||
func (t *tailBuf) Write(b []byte) (int, error) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.b = append(t.b, b...)
|
||||
if len(t.b) > tailKeep {
|
||||
t.b = t.b[len(t.b)-tailKeep:]
|
||||
}
|
||||
return len(b), nil
|
||||
}
|
||||
|
||||
func (t *tailBuf) last() string {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
return lastLine(string(bytes.TrimSpace(t.b)))
|
||||
}
|
||||
|
||||
// RunFFmpegHelper implements `hamlab run-ffmpeg <args…>`: ffmpeg as this user (root, when
|
||||
// the daemon asked through sudo), finishing when stdin closes. Returns ffmpeg's exit code.
|
||||
func RunFFmpegHelper(args []string, stdin io.Reader, stderr io.Writer) int {
|
||||
ffmpeg, err := exec.LookPath("ffmpeg")
|
||||
if err != nil {
|
||||
io.WriteString(stderr, "ffmpeg is not installed\n")
|
||||
return 1
|
||||
}
|
||||
argv := []string{ffmpeg}
|
||||
if nice, err := exec.LookPath("nice"); err == nil {
|
||||
argv = []string{nice, "-n", "10", ffmpeg}
|
||||
}
|
||||
cmd := exec.Command(argv[0], append(argv[1:], args...)...)
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Stderr = stderr
|
||||
cmd.Stdout = io.Discard
|
||||
if err := cmd.Start(); err != nil {
|
||||
io.WriteString(stderr, err.Error()+"\n")
|
||||
return 1
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
go func() {
|
||||
_, _ = io.Copy(io.Discard, stdin) // returns when the daemon lets go
|
||||
cancel()
|
||||
}()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- cmd.Wait() }()
|
||||
var werr error
|
||||
select {
|
||||
case werr = <-done:
|
||||
case <-ctx.Done():
|
||||
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM)
|
||||
select {
|
||||
case werr = <-done:
|
||||
case <-time.After(5 * time.Second):
|
||||
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||
werr = <-done
|
||||
}
|
||||
}
|
||||
var ee *exec.ExitError
|
||||
if errors.As(werr, &ee) {
|
||||
return ee.ExitCode()
|
||||
}
|
||||
if werr != nil {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// clearDir empties a folder without removing it.
|
||||
func clearDir(dir string) error {
|
||||
ents, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, e := range ents {
|
||||
if err := os.RemoveAll(dir + "/" + e.Name()); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func hasSegments(playlist string) bool {
|
||||
b, err := os.ReadFile(playlist)
|
||||
return err == nil && strings.Contains(string(b), "#EXTINF")
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
package media
|
||||
|
||||
// Playing what the browser cannot play as it is: an HLS stream, re-encoded on demand by
|
||||
// ffmpeg on the machine's own GPU where it has one it can use, on its CPU otherwise. A
|
||||
// stream is one file at one codec and size, in a folder of its own under
|
||||
// <data dir>/transcode/: ffmpeg writes an event playlist and fragmented-MP4 segments there
|
||||
// as fast as it can, and the player reads them as they land, so the range it can seek in
|
||||
// grows until the whole file is through — on a GPU, several times faster than watching.
|
||||
// Nothing is kept: a stream nobody has touched for a while is ended and its folder
|
||||
// removed, and the folder is emptied when the daemon starts.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Stream is one re-encode, in progress or finished.
|
||||
type Stream struct {
|
||||
ID string `json:"id"`
|
||||
Path string `json:"path"`
|
||||
Codec string `json:"codec"`
|
||||
Quality string `json:"quality"`
|
||||
Engine string `json:"engine"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
Kbps int `json:"kbps"`
|
||||
Source Info `json:"source"`
|
||||
Started time.Time `json:"started"`
|
||||
Root bool `json:"root"`
|
||||
|
||||
key string
|
||||
dir string
|
||||
mu sync.Mutex
|
||||
used time.Time
|
||||
proc *ffproc
|
||||
finished bool
|
||||
failure string
|
||||
gone bool // forgotten by the Streamer
|
||||
}
|
||||
|
||||
// StreamStatus is what the player asks for while the file is still going through.
|
||||
type StreamStatus struct {
|
||||
Running bool `json:"running"`
|
||||
Done bool `json:"done"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Progress float64 `json:"progress"` // seconds of output written
|
||||
Speed float64 `json:"speed"` // × realtime
|
||||
Segments int `json:"segments"`
|
||||
}
|
||||
|
||||
// Streamer keeps one machine's streams.
|
||||
type Streamer struct {
|
||||
dir string
|
||||
self string
|
||||
limit int
|
||||
mu sync.Mutex
|
||||
streams map[string]*Stream
|
||||
}
|
||||
|
||||
const streamIdle = 10 * time.Minute
|
||||
const firstSegmentWait = 40 * time.Second
|
||||
|
||||
var streamFile = regexp.MustCompile(`^(index\.m3u8|init\.mp4|seg\d{5}\.m4s)$`)
|
||||
|
||||
// NewStreamer empties the folder — a daemon restart orphans every stream — and starts the
|
||||
// sweeper.
|
||||
func NewStreamer(dataDir, self string, limit int) *Streamer {
|
||||
m := &Streamer{dir: filepath.Join(dataDir, "transcode"), self: self, limit: limit, streams: map[string]*Stream{}}
|
||||
_ = os.MkdirAll(m.dir, 0o755)
|
||||
_ = clearDir(m.dir)
|
||||
go func() {
|
||||
for range time.Tick(30 * time.Second) {
|
||||
m.Sweep(streamIdle)
|
||||
}
|
||||
}()
|
||||
return m
|
||||
}
|
||||
|
||||
// Open starts a stream, or hands back the one already serving the same request. It
|
||||
// returns once the first segment is on disk, so the player's first fetch is answered.
|
||||
func (m *Streamer) Open(ctx context.Context, c Caps, root bool, path, codec, quality string, src Info) (*Stream, error) {
|
||||
if c.Error != "" {
|
||||
return nil, errors.New(c.Error)
|
||||
}
|
||||
if err := ValidChoice(codec, quality); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
q, _ := qualityByID(quality)
|
||||
if src.Video == nil {
|
||||
return nil, &DecodeError{Msg: "no video stream"}
|
||||
}
|
||||
if src.Video.Width <= 0 || src.Video.Height <= 0 {
|
||||
return nil, &DecodeError{Msg: "the picture's size is unknown"}
|
||||
}
|
||||
key := fmt.Sprintf("%s|%d|%.0f|%s|%s|%v", path, src.Size, src.Duration, codec, quality, root)
|
||||
m.mu.Lock()
|
||||
for _, s := range m.streams {
|
||||
if s.key == key {
|
||||
if s.failed() { // not the stream to hand back: let it go and make a fresh one
|
||||
delete(m.streams, s.ID)
|
||||
_ = os.RemoveAll(s.dir)
|
||||
break
|
||||
}
|
||||
m.mu.Unlock()
|
||||
s.touch()
|
||||
return s, nil
|
||||
}
|
||||
}
|
||||
running := 0
|
||||
for _, s := range m.streams {
|
||||
if s.running() {
|
||||
running++
|
||||
}
|
||||
}
|
||||
if running >= m.limit {
|
||||
m.mu.Unlock()
|
||||
return nil, fmt.Errorf("%d videos are already being re-encoded on this machine; one must finish or be closed first", running)
|
||||
}
|
||||
s := &Stream{ID: newID(), Path: path, Codec: codec, Quality: quality, Source: src, Started: time.Now(), Root: root, key: key, used: time.Now()}
|
||||
s.dir = filepath.Join(m.dir, s.ID)
|
||||
m.streams[s.ID] = s
|
||||
m.mu.Unlock()
|
||||
|
||||
s.Width, s.Height, s.Kbps = fit(q, src, codec)
|
||||
if codec == "copy" {
|
||||
s.Width, s.Height, s.Kbps = src.Video.Width, src.Video.Height, int(src.BitRate/1000)
|
||||
}
|
||||
rs := recipes(c, codec, src, s.Width, s.Height, s.Kbps)
|
||||
if len(rs) == 0 {
|
||||
m.forget(s)
|
||||
return nil, errors.New("nothing on this machine encodes " + codec)
|
||||
}
|
||||
var last string
|
||||
for _, r := range rs {
|
||||
if err := os.MkdirAll(s.dir, 0o755); err != nil { // the root helper's ffmpeg writes here too
|
||||
m.forget(s)
|
||||
return nil, err
|
||||
}
|
||||
_ = clearDir(s.dir)
|
||||
args := hlsArgs(s.dir, path, src, r, codec == "copy")
|
||||
p, err := startFF(c, m.self, root, args)
|
||||
if err != nil {
|
||||
last = r.Engine + ": " + err.Error()
|
||||
continue
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.proc, s.Engine = p, r.Engine
|
||||
s.mu.Unlock()
|
||||
msg, ok := s.waitFirstSegment(ctx)
|
||||
if ok {
|
||||
go s.watch()
|
||||
log.Printf("media: stream %s: %s → %s %s %dx%d %dk on %s", s.ID, filepath.Base(path), codec, quality, s.Width, s.Height, s.Kbps, r.Engine)
|
||||
return s, nil
|
||||
}
|
||||
p.end()
|
||||
if ctx.Err() != nil || s.closed() { // the caller left, or somebody closed it meanwhile: not the recipe's fault
|
||||
m.forget(s)
|
||||
return nil, errors.New("cancelled")
|
||||
}
|
||||
last = r.Engine + ": " + msg
|
||||
log.Printf("media: stream %s on %s: %s", filepath.Base(path), r.Engine, msg)
|
||||
}
|
||||
m.forget(s)
|
||||
return nil, &DecodeError{Msg: last}
|
||||
}
|
||||
|
||||
// waitFirstSegment watches for the playlist's first segment, or ffmpeg giving up.
|
||||
func (s *Stream) waitFirstSegment(ctx context.Context) (string, bool) {
|
||||
deadline := time.Now().Add(firstSegmentWait)
|
||||
playlist := filepath.Join(s.dir, "index.m3u8")
|
||||
for {
|
||||
if hasSegments(playlist) {
|
||||
return "", true
|
||||
}
|
||||
if done, err := s.proc.exited(); done {
|
||||
msg := s.proc.lastWords()
|
||||
if msg == "" && err != nil {
|
||||
msg = err.Error()
|
||||
}
|
||||
if msg == "" {
|
||||
msg = "ffmpeg ended before writing a segment"
|
||||
}
|
||||
return msg, false
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return "no segment after " + firstSegmentWait.String(), false
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err().Error(), false
|
||||
case <-time.After(250 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// watch records how the encode ended.
|
||||
func (s *Stream) watch() {
|
||||
<-s.proc.done
|
||||
s.mu.Lock()
|
||||
s.finished = true
|
||||
if s.proc.err != nil {
|
||||
s.failure = s.proc.lastWords()
|
||||
if s.failure == "" {
|
||||
s.failure = s.proc.err.Error()
|
||||
}
|
||||
}
|
||||
s.mu.Unlock()
|
||||
if s.failure != "" {
|
||||
log.Printf("media: stream %s ended early: %s", s.ID, s.failure)
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Streamer) Find(id string) (*Stream, bool) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
s, ok := m.streams[id]
|
||||
return s, ok
|
||||
}
|
||||
|
||||
// Close ends a stream and removes its folder.
|
||||
func (m *Streamer) Close(id string) {
|
||||
m.mu.Lock()
|
||||
s, ok := m.streams[id]
|
||||
m.mu.Unlock()
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
p := s.proc
|
||||
s.mu.Unlock()
|
||||
if p != nil {
|
||||
p.end()
|
||||
}
|
||||
m.forget(s)
|
||||
}
|
||||
|
||||
func (m *Streamer) forget(s *Stream) {
|
||||
m.mu.Lock()
|
||||
delete(m.streams, s.ID)
|
||||
m.mu.Unlock()
|
||||
s.mu.Lock()
|
||||
s.gone = true
|
||||
s.mu.Unlock()
|
||||
_ = os.RemoveAll(s.dir)
|
||||
}
|
||||
|
||||
func (s *Stream) closed() bool { s.mu.Lock(); defer s.mu.Unlock(); return s.gone }
|
||||
|
||||
func (s *Stream) failed() bool {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.finished && s.failure != ""
|
||||
}
|
||||
|
||||
// Sweep closes what nobody has touched for a while.
|
||||
func (m *Streamer) Sweep(idle time.Duration) {
|
||||
m.mu.Lock()
|
||||
var old []string
|
||||
for id, s := range m.streams {
|
||||
s.mu.Lock()
|
||||
if time.Since(s.used) > idle {
|
||||
old = append(old, id)
|
||||
}
|
||||
s.mu.Unlock()
|
||||
}
|
||||
m.mu.Unlock()
|
||||
for _, id := range old {
|
||||
log.Printf("media: stream %s idle, closed", id)
|
||||
m.Close(id)
|
||||
}
|
||||
}
|
||||
|
||||
// File is the on-disk path of one of the stream's files — the playlist, the init segment
|
||||
// or a media segment — and nothing else.
|
||||
func (s *Stream) File(name string) (string, bool) {
|
||||
if !streamFile.MatchString(name) {
|
||||
return "", false
|
||||
}
|
||||
s.touch()
|
||||
return filepath.Join(s.dir, name), true
|
||||
}
|
||||
|
||||
func (s *Stream) touch() { s.mu.Lock(); s.used = time.Now(); s.mu.Unlock() }
|
||||
|
||||
func (s *Stream) running() bool {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.proc != nil && !s.finished
|
||||
}
|
||||
|
||||
// Status is read from ffmpeg's own progress file and the playlist.
|
||||
func (s *Stream) Status() StreamStatus {
|
||||
s.mu.Lock()
|
||||
st := StreamStatus{Running: s.proc != nil && !s.finished, Done: s.finished && s.failure == "", Error: s.failure}
|
||||
s.mu.Unlock()
|
||||
if b, err := os.ReadFile(filepath.Join(s.dir, "progress")); err == nil {
|
||||
st.Progress, st.Speed = parseProgress(b)
|
||||
}
|
||||
if b, err := os.ReadFile(filepath.Join(s.dir, "index.m3u8")); err == nil {
|
||||
st.Segments = strings.Count(string(b), "#EXTINF")
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// parseProgress takes the last out_time_us and speed from ffmpeg's -progress output.
|
||||
func parseProgress(b []byte) (seconds, speed float64) {
|
||||
sc := bufio.NewScanner(strings.NewReader(string(b)))
|
||||
for sc.Scan() {
|
||||
k, v, ok := strings.Cut(sc.Text(), "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
v = strings.TrimSpace(v)
|
||||
switch k {
|
||||
case "out_time_us":
|
||||
if us, err := strconv.ParseInt(v, 10, 64); err == nil && us >= 0 {
|
||||
seconds = float64(us) / 1e6
|
||||
}
|
||||
case "speed":
|
||||
if x, err := strconv.ParseFloat(strings.TrimSuffix(v, "x"), 64); err == nil {
|
||||
speed = x
|
||||
}
|
||||
}
|
||||
}
|
||||
return seconds, speed
|
||||
}
|
||||
|
||||
func newID() string {
|
||||
b := make([]byte, 8)
|
||||
_, _ = rand.Read(b)
|
||||
return hex.EncodeToString(b)
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The real thing, when ffmpeg is here: a generated clip goes through as H.264 and comes out
|
||||
// as a playlist of segments, the status follows it to the end, and closing removes it all.
|
||||
func TestStreamWithFFmpeg(t *testing.T) {
|
||||
if _, err := exec.LookPath("ffmpeg"); err != nil {
|
||||
t.Skip("no ffmpeg")
|
||||
}
|
||||
c := Probe(context.Background())
|
||||
if c.Error != "" {
|
||||
t.Skip(c.Error)
|
||||
}
|
||||
if EngineFor(c, "h264") == "" {
|
||||
t.Skip("nothing here encodes h264")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
clip := filepath.Join(dir, "clip.mp4")
|
||||
out, err := exec.Command("ffmpeg", "-nostdin", "-v", "error", "-f", "lavfi", "-i", "testsrc=size=640x360:rate=25:duration=9", "-f", "lavfi", "-i", "sine=frequency=440:duration=9",
|
||||
"-c:v", "mpeg4", "-q:v", "5", "-c:a", "aac", "-shortest", clip).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Skipf("could not make a clip: %s", out)
|
||||
}
|
||||
src, err := Inspect(context.Background(), c, clip, false)
|
||||
if err != nil || src.Video == nil || src.Audio == nil || src.Video.Codec != "mpeg4" || src.Duration < 8 {
|
||||
t.Fatalf("inspect: %+v %v", src, err)
|
||||
}
|
||||
m := NewStreamer(dir, "hamlab", 2)
|
||||
s, err := m.Open(context.Background(), c, false, clip, "h264", "480p", src)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if s.Width != 640 || s.Height != 360 || s.Engine == "" {
|
||||
t.Fatalf("stream: %+v", s)
|
||||
}
|
||||
if p, ok := s.File("index.m3u8"); !ok {
|
||||
t.Fatal("no playlist")
|
||||
} else if !hasSegments(p) {
|
||||
t.Fatal("Open returned before the first segment")
|
||||
}
|
||||
for _, bad := range []string{"../x", "progress", "seg1.m4s", "index.m3u8/", "init.mp4.tmp"} {
|
||||
if _, ok := s.File(bad); ok {
|
||||
t.Fatalf("%q must not be served", bad)
|
||||
}
|
||||
}
|
||||
// the same request is the same stream
|
||||
if again, err := m.Open(context.Background(), c, false, clip, "h264", "480p", src); err != nil || again.ID != s.ID {
|
||||
t.Fatal("not reused")
|
||||
}
|
||||
var st StreamStatus
|
||||
for i := 0; i < 120; i++ {
|
||||
st = s.Status()
|
||||
if st.Done || st.Error != "" {
|
||||
break
|
||||
}
|
||||
time.Sleep(250 * time.Millisecond)
|
||||
}
|
||||
if !st.Done || st.Error != "" || st.Segments < 2 || st.Progress < 8 {
|
||||
t.Fatalf("status at the end: %+v", st)
|
||||
}
|
||||
if init, _ := s.File("init.mp4"); !exists(init) {
|
||||
t.Fatal("no init segment")
|
||||
}
|
||||
m.Close(s.ID)
|
||||
if _, ok := m.Find(s.ID); ok || exists(s.dir) {
|
||||
t.Fatal("close must forget and remove")
|
||||
}
|
||||
// an unplayable source is a DecodeError, and leaves nothing behind
|
||||
bad := filepath.Join(dir, "bad.mkv")
|
||||
_ = os.WriteFile(bad, []byte("not a video"), 0o644)
|
||||
if _, err := Inspect(context.Background(), c, bad, false); err == nil {
|
||||
t.Fatal("inspect of garbage")
|
||||
}
|
||||
if left, _ := os.ReadDir(m.dir); len(left) != 0 {
|
||||
t.Fatalf("left behind: %v", left)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// Track is one stream of a media file, as ffprobe reports it.
|
||||
type Track struct {
|
||||
Index int `json:"index"`
|
||||
Type string `json:"type"` // video | audio
|
||||
Codec string `json:"codec"`
|
||||
Profile string `json:"profile,omitempty"`
|
||||
PixFmt string `json:"pix_fmt,omitempty"`
|
||||
Width int `json:"width,omitempty"`
|
||||
Height int `json:"height,omitempty"`
|
||||
FPS float64 `json:"fps,omitempty"`
|
||||
ColorTransfer string `json:"color_transfer,omitempty"` // smpte2084 / arib-std-b67 mean HDR
|
||||
Channels int `json:"channels,omitempty"`
|
||||
AttachedPic bool `json:"attached_pic,omitempty"`
|
||||
}
|
||||
|
||||
// Info is what a file holds: the picture to take, the sound to go with it, and the whole.
|
||||
type Info struct {
|
||||
Container string `json:"container"` // "matroska,webm", "mov,mp4,m4a,3gp,3g2,mj2"
|
||||
Duration float64 `json:"duration"`
|
||||
BitRate int64 `json:"bit_rate,omitempty"` // bits per second, whole file
|
||||
Size int64 `json:"size,omitempty"`
|
||||
Video *Track `json:"video,omitempty"` // the first real picture stream; cover art is not it
|
||||
Audio *Track `json:"audio,omitempty"` // the first sound stream
|
||||
}
|
||||
|
||||
// HDR reports a picture that needs tone mapping to look right on an SDR screen.
|
||||
func (i Info) HDR() bool {
|
||||
return i.Video != nil && (i.Video.ColorTransfer == "smpte2084" || i.Video.ColorTransfer == "arib-std-b67")
|
||||
}
|
||||
|
||||
type probeJSON struct {
|
||||
Streams []struct {
|
||||
Index int `json:"index"`
|
||||
CodecType string `json:"codec_type"`
|
||||
CodecName string `json:"codec_name"`
|
||||
Profile string `json:"profile"`
|
||||
PixFmt string `json:"pix_fmt"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
RFrameRate string `json:"r_frame_rate"`
|
||||
AvgFrameRate string `json:"avg_frame_rate"`
|
||||
ColorTransfer string `json:"color_transfer"`
|
||||
Channels int `json:"channels"`
|
||||
Disposition struct {
|
||||
AttachedPic int `json:"attached_pic"`
|
||||
} `json:"disposition"`
|
||||
} `json:"streams"`
|
||||
Format struct {
|
||||
FormatName string `json:"format_name"`
|
||||
Duration string `json:"duration"`
|
||||
BitRate string `json:"bit_rate"`
|
||||
Size string `json:"size"`
|
||||
} `json:"format"`
|
||||
}
|
||||
|
||||
// Inspect asks ffprobe what a file is. As root through sudo when asked, for a file only
|
||||
// root may read. An error about the file itself is a DecodeError.
|
||||
func Inspect(ctx context.Context, c Caps, path string, root bool) (Info, error) {
|
||||
if c.FFprobe == "" {
|
||||
return Info{}, errors.New("ffprobe is not installed")
|
||||
}
|
||||
argv := []string{c.FFprobe, "-v", "error", "-show_entries",
|
||||
"stream=index,codec_type,codec_name,profile,pix_fmt,width,height,r_frame_rate,avg_frame_rate,color_transfer,channels,disposition:format=format_name,duration,bit_rate,size",
|
||||
"-of", "json", path}
|
||||
if root {
|
||||
argv = priv.Wrap(argv...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return Info{}, ctx.Err()
|
||||
}
|
||||
msg := lastLine(errb.String())
|
||||
if msg == "" {
|
||||
msg = "ffprobe: " + err.Error()
|
||||
}
|
||||
return Info{}, &DecodeError{Msg: msg}
|
||||
}
|
||||
var pj probeJSON
|
||||
if err := json.Unmarshal(out, &pj); err != nil {
|
||||
return Info{}, &DecodeError{Msg: "ffprobe: unreadable answer"}
|
||||
}
|
||||
in := Info{Container: pj.Format.FormatName}
|
||||
in.Duration, _ = strconv.ParseFloat(pj.Format.Duration, 64)
|
||||
in.BitRate, _ = strconv.ParseInt(pj.Format.BitRate, 10, 64)
|
||||
in.Size, _ = strconv.ParseInt(pj.Format.Size, 10, 64)
|
||||
var art *Track
|
||||
for _, s := range pj.Streams {
|
||||
st := &Track{Index: s.Index, Type: s.CodecType, Codec: s.CodecName, Profile: s.Profile, PixFmt: s.PixFmt, Width: s.Width, Height: s.Height,
|
||||
ColorTransfer: s.ColorTransfer, Channels: s.Channels, AttachedPic: s.Disposition.AttachedPic != 0}
|
||||
st.FPS = fps(s.AvgFrameRate)
|
||||
if st.FPS == 0 {
|
||||
st.FPS = fps(s.RFrameRate)
|
||||
}
|
||||
switch {
|
||||
case s.CodecType == "video" && st.AttachedPic:
|
||||
if art == nil {
|
||||
art = st
|
||||
}
|
||||
case s.CodecType == "video" && in.Video == nil:
|
||||
in.Video = st
|
||||
case s.CodecType == "audio" && in.Audio == nil:
|
||||
in.Audio = st
|
||||
}
|
||||
}
|
||||
if in.Video == nil {
|
||||
in.Video = art // cover art is the only picture there is
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
||||
// fps parses ffprobe's "60000/1001".
|
||||
func fps(r string) float64 {
|
||||
n, d, ok := strings.Cut(r, "/")
|
||||
if !ok {
|
||||
v, _ := strconv.ParseFloat(r, 64)
|
||||
return v
|
||||
}
|
||||
nf, _ := strconv.ParseFloat(n, 64)
|
||||
df, _ := strconv.ParseFloat(d, 64)
|
||||
if df == 0 {
|
||||
return 0
|
||||
}
|
||||
return nf / df
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/user"
|
||||
"strconv"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// ownerOf is "root:render 0660" for a device node, the thing to say when it cannot be opened.
|
||||
func ownerOf(st os.FileInfo) string {
|
||||
sys, ok := st.Sys().(*syscall.Stat_t)
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
u := strconv.Itoa(int(sys.Uid))
|
||||
if x, err := user.LookupId(u); err == nil {
|
||||
u = x.Username
|
||||
}
|
||||
g := strconv.Itoa(int(sys.Gid))
|
||||
if x, err := user.LookupGroupId(g); err == nil {
|
||||
g = x.Name
|
||||
}
|
||||
return fmt.Sprintf("%s:%s %04o", u, g, st.Mode().Perm())
|
||||
}
|
||||
@@ -0,0 +1,656 @@
|
||||
// Package thumbs makes the Files tab's previews: one small JPEG for an image or a video,
|
||||
// by ffmpeg, decoded on a GPU where the machine has one it can use and on the CPU
|
||||
// otherwise.
|
||||
//
|
||||
// Nothing about the machine is assumed. A probe looks ffmpeg and ffprobe up, opens every
|
||||
// /dev/dri/renderD* and exercises it (a VAAPI upload and download of a synthetic frame),
|
||||
// so a node the daemon's user cannot open, or a driver that does not speak VAAPI, is
|
||||
// reported once rather than tried per file. vainfo, where installed, says which codecs
|
||||
// each node decodes, and a video goes to the first node that lists its codec; without
|
||||
// vainfo the first working node is tried and ffmpeg falls back to software on its own when
|
||||
// the stream turns out unsupported. A hardware run that fails outright is retried on the
|
||||
// CPU before the file is called undecodable. Images are always decoded on the CPU: one
|
||||
// frame of a JPEG at a reduced DCT size costs less than the trip through a GPU.
|
||||
package media
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/metrics"
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// GPU is one hardware decoder as the probe found it.
|
||||
type GPU struct {
|
||||
API string `json:"api"` // vaapi | cuda
|
||||
Device string `json:"device"` // /dev/dri/renderD128; "0" for cuda (its device index)
|
||||
Name string `json:"name,omitempty"` // "Intel Arc A380"
|
||||
Codecs []string `json:"codecs,omitempty"` // ffmpeg codec names it decodes; nil when nothing could say
|
||||
Encodes []string `json:"encodes,omitempty"` // ffmpeg codec names it encodes, from vainfo
|
||||
Integrated bool `json:"integrated,omitempty"` // on the CPU die; a card on the bus is preferred for encoding
|
||||
Reason string `json:"reason,omitempty"` // Unusable only: why
|
||||
Permission bool `json:"permission,omitempty"` // Unusable only: the node exists but this user may not open it
|
||||
}
|
||||
|
||||
// Caps is what the machine can make previews with, as probed.
|
||||
type Caps struct {
|
||||
FFmpeg string `json:"ffmpeg,omitempty"`
|
||||
FFprobe string `json:"ffprobe,omitempty"`
|
||||
Nice string `json:"nice,omitempty"`
|
||||
Version string `json:"version,omitempty"`
|
||||
GPUs []GPU `json:"gpus"` // usable, in node order
|
||||
Encoders []string `json:"encoders,omitempty"` // the encoders ffmpeg itself was built with that matter here
|
||||
Unusable []GPU `json:"unusable,omitempty"` // render nodes that exist but cannot be used, with the reason
|
||||
Error string `json:"error,omitempty"` // why nothing can be made at all
|
||||
ProbedAt time.Time `json:"probed_at"`
|
||||
}
|
||||
|
||||
// Opts shapes one preview.
|
||||
type Opts struct {
|
||||
Size int // the long edge in pixels; 256 when zero
|
||||
Timeout time.Duration // for the whole job, ffprobe and every ffmpeg attempt; 30 s when zero
|
||||
}
|
||||
|
||||
func (o Opts) defaults() Opts {
|
||||
if o.Size <= 0 {
|
||||
o.Size = 256
|
||||
}
|
||||
if o.Timeout <= 0 {
|
||||
o.Timeout = 30 * time.Second
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// Result is a finished preview.
|
||||
type Result struct {
|
||||
JPEG []byte `json:"-"`
|
||||
Engine string `json:"engine"` // "cpu", or "vaapi:/dev/dri/renderD129"
|
||||
Codec string `json:"codec"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
}
|
||||
|
||||
// ErrUnsupported: not an image or a video by its name.
|
||||
var ErrUnsupported = errors.New("not an image or video")
|
||||
|
||||
// A DecodeError is ffmpeg's verdict on the file itself. It will not change until the
|
||||
// file does, so the caller may remember it.
|
||||
type DecodeError struct{ Msg string }
|
||||
|
||||
func (e *DecodeError) Error() string { return e.Msg }
|
||||
|
||||
// ---- what gets a preview ----
|
||||
|
||||
var imageExt = set("png jpg jpeg jpe jfif gif webp bmp tif tiff avif heic heif hif jxl svg ico psd tga thm")
|
||||
var videoExt = set("mp4 m4v mkv webm mov avi wmv flv mpg mpeg mpe m2v ts m2ts mts 3gp 3g2 ogv ogm mxf vob asf rm rmvb divx f4v mjpeg mjpg 360 osv insv lrf")
|
||||
var audioExt = set("mp3 flac m4a m4b aac ogg oga opus wav wma aiff aif alac ape wv mka")
|
||||
|
||||
func set(s string) map[string]bool {
|
||||
m := map[string]bool{}
|
||||
for _, w := range strings.Fields(s) {
|
||||
m[w] = true
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Kind is "image", "video", "audio" or "" for a file name. An audio file's preview is its
|
||||
// cover art, when it carries any.
|
||||
func Kind(name string) string {
|
||||
ext := strings.ToLower(strings.TrimPrefix(filepath.Ext(name), "."))
|
||||
switch {
|
||||
case imageExt[ext]:
|
||||
return "image"
|
||||
case videoExt[ext]:
|
||||
return "video"
|
||||
case audioExt[ext]:
|
||||
return "audio"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- the probe ----
|
||||
|
||||
// Probe finds out what this process can make previews with. It runs ffmpeg against every
|
||||
// render node, so it takes a few hundred milliseconds; callers keep the answer.
|
||||
func Probe(ctx context.Context) Caps {
|
||||
c := Caps{ProbedAt: time.Now(), GPUs: []GPU{}}
|
||||
c.FFmpeg, _ = exec.LookPath("ffmpeg")
|
||||
c.FFprobe, _ = exec.LookPath("ffprobe")
|
||||
c.Nice, _ = exec.LookPath("nice")
|
||||
switch {
|
||||
case c.FFmpeg == "" && c.FFprobe == "":
|
||||
c.Error = "ffmpeg is not installed"
|
||||
case c.FFmpeg == "":
|
||||
c.Error = "ffmpeg is not installed (ffprobe is)"
|
||||
case c.FFprobe == "":
|
||||
c.Error = "ffprobe is not installed (ffmpeg is)"
|
||||
}
|
||||
if c.Error != "" {
|
||||
return c
|
||||
}
|
||||
c.Version = ffmpegVersion(ctx, c.FFmpeg)
|
||||
c.Encoders = encoders(ctx, c.FFmpeg)
|
||||
accels := hwaccels(ctx, c.FFmpeg)
|
||||
if accels["vaapi"] {
|
||||
nodes, _ := filepath.Glob("/dev/dri/renderD*")
|
||||
sort.Strings(nodes)
|
||||
vainfo, _ := exec.LookPath("vainfo")
|
||||
for _, dev := range nodes {
|
||||
g := GPU{API: "vaapi", Device: dev, Name: metrics.RenderNodeName(dev), Integrated: metrics.RenderNodeIntegrated(dev)}
|
||||
if err := openable(dev); err != nil {
|
||||
g.Reason, g.Permission = err.Error(), errors.Is(err, os.ErrPermission)
|
||||
c.Unusable = append(c.Unusable, g)
|
||||
continue
|
||||
}
|
||||
if msg := hwWorks(ctx, c.FFmpeg, "vaapi=va:"+dev); msg != "" {
|
||||
g.Reason = msg
|
||||
c.Unusable = append(c.Unusable, g)
|
||||
continue
|
||||
}
|
||||
if vainfo != "" {
|
||||
g.Codecs, g.Encodes = vainfoCodecs(ctx, vainfo, dev)
|
||||
}
|
||||
c.GPUs = append(c.GPUs, g)
|
||||
}
|
||||
}
|
||||
if accels["cuda"] && exists("/dev/nvidiactl") {
|
||||
g := GPU{API: "cuda", Device: "0", Name: "NVIDIA GPU"}
|
||||
if msg := hwWorks(ctx, c.FFmpeg, "cuda=cu:0"); msg != "" {
|
||||
g.Reason = msg
|
||||
c.Unusable = append(c.Unusable, g)
|
||||
} else {
|
||||
c.GPUs = append(c.GPUs, g)
|
||||
}
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// openable: the node can be opened read-write, which is what a VAAPI driver needs. The
|
||||
// usual reason it can't be is the owning group: a render node is root:render 0660 on
|
||||
// most distributions, and a daemon running as an ordinary user is not in render.
|
||||
func openable(dev string) error {
|
||||
f, err := os.OpenFile(dev, os.O_RDWR, 0)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrPermission) {
|
||||
owner := ""
|
||||
if st, serr := os.Stat(dev); serr == nil {
|
||||
owner = ownerOf(st)
|
||||
}
|
||||
if owner != "" {
|
||||
return fmt.Errorf("%w: %s is %s", os.ErrPermission, dev, owner)
|
||||
}
|
||||
return fmt.Errorf("%w: cannot open %s", os.ErrPermission, dev)
|
||||
}
|
||||
return err
|
||||
}
|
||||
f.Close()
|
||||
return nil
|
||||
}
|
||||
|
||||
// hwWorks initialises a hardware device in ffmpeg and moves one synthetic frame through
|
||||
// it. "" when it works, else the last thing ffmpeg said.
|
||||
func hwWorks(ctx context.Context, ffmpeg, device string) string {
|
||||
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
name, rest, _ := strings.Cut(device, "=") // "vaapi", "va:/dev/…"
|
||||
label, _, _ := strings.Cut(rest, ":") // "va"
|
||||
cmd := exec.CommandContext(ctx, ffmpeg, "-nostdin", "-hide_banner", "-v", "error", "-init_hw_device", device, "-filter_hw_device", label,
|
||||
"-f", "lavfi", "-i", "color=c=black:s=64x64:r=1:d=1", "-vf", "format=nv12,hwupload,hwdownload,format=nv12", "-frames:v", "1", "-f", "null", "-")
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := lastLine(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return name + ": " + msg
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func ffmpegVersion(ctx context.Context, ffmpeg string) string {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
out, _ := exec.CommandContext(ctx, ffmpeg, "-version").Output()
|
||||
f := strings.Fields(string(out)) // "ffmpeg version 7.1.5-0+deb13u1 Copyright …"
|
||||
if len(f) >= 3 && f[1] == "version" {
|
||||
v := f[2]
|
||||
if i := strings.IndexAny(v, "-+~"); i > 0 {
|
||||
v = v[:i]
|
||||
}
|
||||
return v
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func hwaccels(ctx context.Context, ffmpeg string) map[string]bool {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
out, _ := exec.CommandContext(ctx, ffmpeg, "-hide_banner", "-hwaccels").Output()
|
||||
m := map[string]bool{}
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
l = strings.TrimSpace(l)
|
||||
if l != "" && !strings.HasPrefix(l, "Hardware acceleration methods") {
|
||||
m[l] = true
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
var vaProfileRe = regexp.MustCompile(`VAProfile([A-Za-z0-9]+?)(?:Simple|Main|High|Profile|Baseline|Constrained|Scc|Advanced|Version|Stereo|Multiview)\w*\s*:\s*VAEntrypoint(VLD|EncSlice|EncSliceLP)\b`)
|
||||
|
||||
// wantedEncoders are the encoders this package may ask ffmpeg for.
|
||||
var wantedEncoders = set("libx264 libx265 libsvtav1 libaom-av1 libopenh264 h264_vaapi hevc_vaapi av1_vaapi aac")
|
||||
|
||||
// encoders lists the wanted encoders this ffmpeg was built with.
|
||||
func encoders(ctx context.Context, ffmpeg string) []string {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
out, _ := exec.CommandContext(ctx, ffmpeg, "-hide_banner", "-encoders").Output()
|
||||
var found []string
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
f := strings.Fields(l)
|
||||
if len(f) >= 2 && wantedEncoders[f[1]] {
|
||||
found = append(found, f[1])
|
||||
}
|
||||
}
|
||||
sort.Strings(found)
|
||||
return found
|
||||
}
|
||||
|
||||
// vaCodec maps a VAProfile family to the ffmpeg codec it decodes.
|
||||
var vaCodec = map[string]string{"H264": "h264", "HEVC": "hevc", "VP9": "vp9", "AV1": "av1", "MPEG2": "mpeg2video", "VC1": "vc1", "VP8": "vp8", "JPEG": "mjpeg", "MPEG4": "mpeg4", "H263": "h263"}
|
||||
|
||||
func vainfoCodecs(ctx context.Context, vainfo, dev string) (decodes, encodes []string) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
out, _ := exec.CommandContext(ctx, vainfo, "--display", "drm", "--device", dev).Output()
|
||||
return parseVainfo(string(out))
|
||||
}
|
||||
|
||||
// parseVainfo names the codecs the decode (VLD) and encode (EncSlice) profiles cover,
|
||||
// the way ffmpeg names them.
|
||||
func parseVainfo(text string) (decodes, encodes []string) {
|
||||
dec, enc := map[string]bool{}, map[string]bool{}
|
||||
for _, m := range vaProfileRe.FindAllStringSubmatch(text, -1) {
|
||||
c, ok := vaCodec[m[1]]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if m[2] == "VLD" {
|
||||
dec[c] = true
|
||||
} else {
|
||||
enc[c] = true
|
||||
}
|
||||
}
|
||||
return keys(dec), keys(enc)
|
||||
}
|
||||
|
||||
func keys(m map[string]bool) []string {
|
||||
if len(m) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(m))
|
||||
for c := range m {
|
||||
out = append(out, c)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// pick is the GPU for a codec: the first that lists it, else the first that could not
|
||||
// say (ffmpeg will fall back to software by itself if that guess is wrong), else none.
|
||||
func (c Caps) pick(codec string) *GPU {
|
||||
var unknown *GPU
|
||||
for i := range c.GPUs {
|
||||
g := &c.GPUs[i]
|
||||
if g.Codecs == nil {
|
||||
if unknown == nil {
|
||||
unknown = g
|
||||
}
|
||||
continue
|
||||
}
|
||||
for _, x := range g.Codecs {
|
||||
if x == codec {
|
||||
return g
|
||||
}
|
||||
}
|
||||
}
|
||||
return unknown
|
||||
}
|
||||
|
||||
// ---- making one ----
|
||||
|
||||
type fileInfo struct {
|
||||
index string // the stream to take
|
||||
codec string
|
||||
w, h int
|
||||
duration float64
|
||||
}
|
||||
|
||||
type attempt struct {
|
||||
gpu *GPU
|
||||
lowres int
|
||||
seek float64
|
||||
}
|
||||
|
||||
func (a attempt) engine() string {
|
||||
if a.gpu == nil {
|
||||
return "cpu"
|
||||
}
|
||||
return a.gpu.API + ":" + a.gpu.Device
|
||||
}
|
||||
|
||||
// Make produces the preview of one file.
|
||||
func Make(ctx context.Context, c Caps, path string, o Opts) (Result, error) {
|
||||
kind := Kind(path)
|
||||
if kind == "" {
|
||||
return Result{}, ErrUnsupported
|
||||
}
|
||||
if c.Error != "" {
|
||||
return Result{}, errors.New(c.Error)
|
||||
}
|
||||
o = o.defaults()
|
||||
ctx, cancel := context.WithTimeout(ctx, o.Timeout)
|
||||
defer cancel()
|
||||
fi, err := probeFile(ctx, c, path)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
var last string
|
||||
for _, eng := range engines(c, kind, fi) {
|
||||
for _, a := range attempts(eng, kind, fi, o.Size) {
|
||||
data, msg, err := run(ctx, c, a, path, fi, o.Size)
|
||||
if err == nil && len(data) > 0 {
|
||||
return Result{JPEG: data, Engine: a.engine(), Codec: fi.codec, Width: fi.w, Height: fi.h}, nil
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return Result{}, ctx.Err() // the clock ran out: not the file's fault
|
||||
}
|
||||
if err != nil { // the decoder refused: try the next engine, not another seek
|
||||
last = msg
|
||||
break
|
||||
}
|
||||
last = "no frame at " + strconv.FormatFloat(a.seek, 'f', 1, 64) + " s"
|
||||
}
|
||||
}
|
||||
if last == "" {
|
||||
last = "could not decode"
|
||||
}
|
||||
return Result{}, &DecodeError{Msg: last}
|
||||
}
|
||||
|
||||
// engines is the order to try: the fitting GPU first for a video, then the CPU.
|
||||
func engines(c Caps, kind string, fi fileInfo) []*GPU {
|
||||
if kind == "video" {
|
||||
if g := c.pick(fi.codec); g != nil {
|
||||
return []*GPU{g, nil}
|
||||
}
|
||||
}
|
||||
return []*GPU{nil}
|
||||
}
|
||||
|
||||
// attempts for one engine: a seek into a video, then from its start if that found no
|
||||
// frame (a wrong duration in the header); a reduced-size decode of a JPEG, then full size.
|
||||
func attempts(gpu *GPU, kind string, fi fileInfo, size int) []attempt {
|
||||
if kind == "video" {
|
||||
s := seekFor(fi.duration)
|
||||
if s > 0 {
|
||||
return []attempt{{gpu: gpu, seek: s}, {gpu: gpu}}
|
||||
}
|
||||
return []attempt{{gpu: gpu}}
|
||||
}
|
||||
if fi.codec == "mjpeg" && gpu == nil {
|
||||
if l := lowresFor(fi.w, fi.h, size); l > 0 {
|
||||
return []attempt{{lowres: l}, {}}
|
||||
}
|
||||
}
|
||||
return []attempt{{gpu: gpu}}
|
||||
}
|
||||
|
||||
// seekFor: a tenth of the way in, never more than 30 s, so a title card or a black lead-in
|
||||
// is skipped and a long file still seeks to a keyframe near its start.
|
||||
func seekFor(duration float64) float64 {
|
||||
if duration <= 0 {
|
||||
return 0
|
||||
}
|
||||
s := duration / 10
|
||||
if s > 30 {
|
||||
s = 30
|
||||
}
|
||||
if s < 0.5 {
|
||||
return 0
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// lowresFor: how many halvings a JPEG decoder may take while the picture stays at least
|
||||
// twice the preview size. libjpeg does this in the DCT, so a 24 MP photo decodes at 1.5 MP.
|
||||
func lowresFor(w, h, size int) int {
|
||||
m := w
|
||||
if h < m {
|
||||
m = h
|
||||
}
|
||||
l := 0
|
||||
for l < 3 && m>>(l+1) >= size*2 {
|
||||
l++
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
type probeOut struct {
|
||||
Streams []struct {
|
||||
Index int `json:"index"`
|
||||
Codec string `json:"codec_name"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
Disposition struct {
|
||||
AttachedPic int `json:"attached_pic"`
|
||||
} `json:"disposition"`
|
||||
} `json:"streams"`
|
||||
Format struct {
|
||||
Duration string `json:"duration"`
|
||||
} `json:"format"`
|
||||
}
|
||||
|
||||
// probeFile asks ffprobe which stream to take and what is in it.
|
||||
func probeFile(ctx context.Context, c Caps, path string) (fileInfo, error) {
|
||||
cmd := exec.CommandContext(ctx, c.FFprobe, "-v", "error", "-select_streams", "v", "-show_entries", "stream=index,codec_name,width,height,disposition:format=duration", "-of", "json", path)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return fileInfo{}, ctx.Err()
|
||||
}
|
||||
msg := lastLine(errb.String())
|
||||
if msg == "" {
|
||||
msg = "ffprobe: " + err.Error()
|
||||
}
|
||||
return fileInfo{}, &DecodeError{Msg: msg}
|
||||
}
|
||||
var po probeOut
|
||||
if err := json.Unmarshal(out, &po); err != nil {
|
||||
return fileInfo{}, &DecodeError{Msg: "ffprobe: unreadable answer"}
|
||||
}
|
||||
if len(po.Streams) == 0 {
|
||||
return fileInfo{}, &DecodeError{Msg: "no video stream"}
|
||||
}
|
||||
s := po.Streams[0]
|
||||
for _, x := range po.Streams { // cover art rides along as a video stream; the picture is not the film
|
||||
if x.Disposition.AttachedPic == 0 {
|
||||
s = x
|
||||
break
|
||||
}
|
||||
}
|
||||
fi := fileInfo{index: strconv.Itoa(s.Index), codec: s.Codec, w: s.Width, h: s.Height}
|
||||
fi.duration, _ = strconv.ParseFloat(po.Format.Duration, 64)
|
||||
return fi, nil
|
||||
}
|
||||
|
||||
// run is one ffmpeg: the JPEG on success; on failure what ffmpeg said and the error.
|
||||
// Empty output with a clean exit means the seek found nothing.
|
||||
func run(ctx context.Context, c Caps, a attempt, path string, fi fileInfo, size int) ([]byte, string, error) {
|
||||
args := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
|
||||
if a.gpu != nil {
|
||||
args = append(args, "-hwaccel", a.gpu.API, "-hwaccel_device", a.gpu.Device)
|
||||
}
|
||||
if a.lowres > 0 {
|
||||
args = append(args, "-lowres", strconv.Itoa(a.lowres))
|
||||
}
|
||||
if a.seek > 0 {
|
||||
args = append(args, "-noaccurate_seek", "-ss", strconv.FormatFloat(a.seek, 'f', 3, 64))
|
||||
}
|
||||
args = append(args, "-i", path, "-map", "0:"+fi.index, "-frames:v", "1",
|
||||
"-vf", fmt.Sprintf("scale=w=%d:h=%d:force_original_aspect_ratio=decrease:flags=bicubic", size, size),
|
||||
"-f", "image2pipe", "-c:v", "mjpeg", "-q:v", "4", "pipe:1")
|
||||
name, argv := c.FFmpeg, args
|
||||
if c.Nice != "" { // a burst of previews must not crowd out what the machine is for
|
||||
name, argv = c.Nice, append([]string{"-n", "10", c.FFmpeg}, args...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, name, argv...)
|
||||
var out, errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := lastLine(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return nil, msg, err
|
||||
}
|
||||
return out.Bytes(), "", nil
|
||||
}
|
||||
|
||||
// ---- the same, as root ----
|
||||
|
||||
// ProbeRoot is Probe run as root through this binary, for the Files tab's root mode: root
|
||||
// opens every render node, whatever group the daemon's user is in.
|
||||
func ProbeRoot(ctx context.Context, self string) (Caps, error) {
|
||||
full := priv.Wrap(self, "thumb", "--probe")
|
||||
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
if msg := lastLine(errb.String()); msg != "" {
|
||||
return Caps{}, errors.New(msg)
|
||||
}
|
||||
return Caps{}, err
|
||||
}
|
||||
var c Caps
|
||||
if err := json.Unmarshal(out, &c); err != nil {
|
||||
return Caps{}, err
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
type helperOut struct {
|
||||
Engine string `json:"engine,omitempty"`
|
||||
Codec string `json:"codec,omitempty"`
|
||||
Width int `json:"width,omitempty"`
|
||||
Height int `json:"height,omitempty"`
|
||||
JPEG []byte `json:"jpeg,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Permanent bool `json:"permanent,omitempty"` // a DecodeError: the file, not the moment
|
||||
}
|
||||
|
||||
// MakeRoot is Make run as root through this binary, with the caps ProbeRoot found.
|
||||
func MakeRoot(ctx context.Context, self string, c Caps, path string, o Opts) (Result, error) {
|
||||
o = o.defaults()
|
||||
caps, _ := json.Marshal(c)
|
||||
full := priv.Wrap(self, "thumb", "--size", strconv.Itoa(o.Size), "--timeout", o.Timeout.String(), path)
|
||||
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
|
||||
cmd.Stdin = bytes.NewReader(caps)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return Result{}, ctx.Err()
|
||||
}
|
||||
if msg := lastLine(errb.String()); msg != "" {
|
||||
return Result{}, errors.New(msg)
|
||||
}
|
||||
return Result{}, err
|
||||
}
|
||||
var h helperOut
|
||||
if err := json.Unmarshal(out, &h); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if h.Error != "" {
|
||||
if h.Permanent {
|
||||
return Result{}, &DecodeError{Msg: h.Error}
|
||||
}
|
||||
return Result{}, errors.New(h.Error)
|
||||
}
|
||||
return Result{JPEG: h.JPEG, Engine: h.Engine, Codec: h.Codec, Width: h.Width, Height: h.Height}, nil
|
||||
}
|
||||
|
||||
// Helper implements `hamlab thumb …`, the root side of the above. `--probe` prints Caps;
|
||||
// `[--size N] [--timeout D] <path>` reads Caps from stdin and prints the preview as JSON.
|
||||
// Returns the exit code.
|
||||
func Helper(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
|
||||
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
|
||||
ctx := context.Background()
|
||||
o := Opts{}
|
||||
path := ""
|
||||
for i := 0; i < len(args); i++ {
|
||||
switch a := args[i]; {
|
||||
case a == "--probe":
|
||||
return emit(stdout, Probe(ctx), stderr)
|
||||
case a == "--size" && i+1 < len(args):
|
||||
o.Size, _ = strconv.Atoi(args[i+1])
|
||||
i++
|
||||
case a == "--timeout" && i+1 < len(args):
|
||||
o.Timeout, _ = time.ParseDuration(args[i+1])
|
||||
i++
|
||||
case strings.HasPrefix(a, "-"):
|
||||
return fail(errors.New("usage: hamlab thumb --probe | [--size N] [--timeout D] <path>"))
|
||||
default:
|
||||
path = a
|
||||
}
|
||||
}
|
||||
if path == "" || !filepath.IsAbs(path) {
|
||||
return fail(errors.New("usage: hamlab thumb --probe | [--size N] [--timeout D] <absolute path>"))
|
||||
}
|
||||
var c Caps
|
||||
if err := json.NewDecoder(io.LimitReader(stdin, 1<<20)).Decode(&c); err != nil {
|
||||
return fail(fmt.Errorf("caps on stdin: %w", err))
|
||||
}
|
||||
res, err := Make(ctx, c, path, o)
|
||||
if err != nil {
|
||||
var de *DecodeError
|
||||
return emit(stdout, helperOut{Error: err.Error(), Permanent: errors.As(err, &de)}, stderr)
|
||||
}
|
||||
return emit(stdout, helperOut{Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}, stderr)
|
||||
}
|
||||
|
||||
func emit(w io.Writer, v any, stderr io.Writer) int {
|
||||
if err := json.NewEncoder(w).Encode(v); err != nil {
|
||||
fmt.Fprintln(stderr, err.Error())
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ---- small helpers ----
|
||||
|
||||
func lastLine(s string) string {
|
||||
lines := strings.Split(strings.TrimSpace(s), "\n")
|
||||
return strings.TrimSpace(lines[len(lines)-1])
|
||||
}
|
||||
|
||||
func exists(p string) bool { _, err := os.Stat(p); return err == nil }
|
||||
@@ -0,0 +1,253 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/jpeg"
|
||||
"image/png"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestKind(t *testing.T) {
|
||||
for name, want := range map[string]string{"a.JPG": "image", "b.heic": "image", "c.mkv": "video", "GH010098.MP4": "video", "x.OSV": "video", "d.txt": "", "e": "", ".jpg": "image"} {
|
||||
if got := Kind(name); got != want {
|
||||
t.Errorf("Kind(%q) = %q, want %q", name, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestKeyFollowsTheFile(t *testing.T) {
|
||||
a := Key("/x/a.jpg", 10, 20)
|
||||
if a != Key("/x/a.jpg", 10, 20) || len(a) != 64 {
|
||||
t.Fatal("key is not stable")
|
||||
}
|
||||
for _, b := range []string{Key("/x/b.jpg", 10, 20), Key("/x/a.jpg", 11, 20), Key("/x/a.jpg", 10, 21)} {
|
||||
if b == a {
|
||||
t.Fatal("key ignores part of the file's identity")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseVainfo(t *testing.T) {
|
||||
text := `vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 25.2.3 ()
|
||||
VAProfileMPEG2Simple : VAEntrypointVLD
|
||||
VAProfileH264Main : VAEntrypointVLD
|
||||
VAProfileH264High : VAEntrypointEncSliceLP
|
||||
VAProfileJPEGBaseline : VAEntrypointVLD
|
||||
VAProfileHEVCMain10 : VAEntrypointVLD
|
||||
VAProfileHEVCMain10 : VAEntrypointEncSlice
|
||||
VAProfileHEVCMain444_12 : VAEntrypointVLD
|
||||
VAProfileVP9Profile2 : VAEntrypointVLD
|
||||
VAProfileAV1Profile0 : VAEntrypointEncSliceLP
|
||||
`
|
||||
dec, enc := parseVainfo(text)
|
||||
same := func(got, want []string) bool {
|
||||
if len(got) != len(want) {
|
||||
return false
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
if !same(dec, []string{"h264", "hevc", "mjpeg", "mpeg2video", "vp9"}) {
|
||||
t.Fatalf("decodes = %v", dec)
|
||||
}
|
||||
if !same(enc, []string{"av1", "h264", "hevc"}) {
|
||||
t.Fatalf("encodes = %v", enc)
|
||||
}
|
||||
if d, e := parseVainfo(""); d != nil || e != nil {
|
||||
t.Fatal("nothing from nothing")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPickAndEngines(t *testing.T) {
|
||||
igpu := GPU{API: "vaapi", Device: "/dev/dri/renderD128", Codecs: []string{"h264", "hevc"}}
|
||||
arc := GPU{API: "vaapi", Device: "/dev/dri/renderD129", Codecs: []string{"av1", "h264", "hevc"}}
|
||||
c := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", GPUs: []GPU{igpu, arc}}
|
||||
if g := c.pick("av1"); g == nil || g.Device != arc.Device {
|
||||
t.Fatal("av1 should go to the card that lists it")
|
||||
}
|
||||
if g := c.pick("h264"); g == nil || g.Device != igpu.Device {
|
||||
t.Fatal("h264 should go to the first card that lists it")
|
||||
}
|
||||
if g := c.pick("vc1"); g != nil {
|
||||
t.Fatal("a codec nobody lists is software")
|
||||
}
|
||||
// a card nothing could describe is tried, after the ones that answered
|
||||
c.GPUs = append(c.GPUs, GPU{API: "cuda", Device: "0"})
|
||||
if g := c.pick("vc1"); g == nil || g.API != "cuda" {
|
||||
t.Fatal("an undescribed card is the guess of last resort")
|
||||
}
|
||||
// videos: GPU then CPU; images: CPU only
|
||||
if e := engines(c, "video", fileInfo{codec: "hevc"}); len(e) != 2 || e[0] == nil || e[1] != nil {
|
||||
t.Fatal("a video tries the GPU, then the CPU")
|
||||
}
|
||||
if e := engines(c, "image", fileInfo{codec: "mjpeg"}); len(e) != 1 || e[0] != nil {
|
||||
t.Fatal("an image is CPU only")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAttempts(t *testing.T) {
|
||||
a := attempts(nil, "video", fileInfo{duration: 120}, 256)
|
||||
if len(a) != 2 || a[0].seek != 12 || a[1].seek != 0 {
|
||||
t.Fatalf("video attempts = %+v", a)
|
||||
}
|
||||
if a := attempts(nil, "video", fileInfo{duration: 7200}, 256); a[0].seek != 30 {
|
||||
t.Fatalf("seek capped at 30 s, got %v", a[0].seek)
|
||||
}
|
||||
if a := attempts(nil, "video", fileInfo{duration: 2}, 256); len(a) != 1 || a[0].seek != 0 {
|
||||
t.Fatalf("a short clip is taken from its start, got %+v", a)
|
||||
}
|
||||
if a := attempts(nil, "image", fileInfo{codec: "mjpeg", w: 6000, h: 4000}, 256); len(a) != 2 || a[0].lowres != 2 || a[1].lowres != 0 {
|
||||
t.Fatalf("jpeg attempts = %+v", a)
|
||||
}
|
||||
if a := attempts(nil, "image", fileInfo{codec: "mjpeg", w: 800, h: 600}, 256); len(a) != 1 || a[0].lowres != 0 {
|
||||
t.Fatalf("a small jpeg decodes whole, got %+v", a)
|
||||
}
|
||||
if a := attempts(nil, "image", fileInfo{codec: "png", w: 6000, h: 4000}, 256); len(a) != 1 || a[0].lowres != 0 {
|
||||
t.Fatalf("only jpeg has lowres, got %+v", a)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache(t *testing.T) {
|
||||
c := NewCache(filepath.Join(t.TempDir(), "thumbs.db"))
|
||||
k := Key("/a", 1, 2)
|
||||
if _, found, err := c.Get(k); found || err != nil {
|
||||
t.Fatal("an empty cache found something, or complained")
|
||||
}
|
||||
if _, err := os.Stat(c.Path); err == nil {
|
||||
t.Fatal("a miss must not create the database")
|
||||
}
|
||||
if n, b := c.Stats(); n != 0 || b != 0 {
|
||||
t.Fatal("stats of nothing")
|
||||
}
|
||||
if err := c.Put(k, Entry{Path: "/a", Engine: "cpu", Codec: "png", Width: 640, Height: 360, JPEG: []byte("jpeg")}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e, found, err := c.Get(k)
|
||||
if err != nil || !found || string(e.JPEG) != "jpeg" || e.Engine != "cpu" || e.Width != 640 || e.Path != "/a" || e.Made.IsZero() {
|
||||
t.Fatalf("put then get: %+v %v %v", e, found, err)
|
||||
}
|
||||
k2 := Key("/b", 1, 2)
|
||||
if err := c.Put(k2, Entry{Path: "/b", Failure: "no video stream"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if e, found, _ := c.Get(k2); !found || e.JPEG != nil || e.Failure != "no video stream" {
|
||||
t.Fatal("a failure is remembered with its reason")
|
||||
}
|
||||
if err := c.Put(k, Entry{}); err == nil {
|
||||
t.Fatal("an empty preview must not be kept")
|
||||
}
|
||||
// replacing an entry does not count it twice
|
||||
if err := c.Put(k, Entry{Path: "/a", JPEG: []byte("jpeg2")}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
n, b := c.Stats()
|
||||
if n != 2 || b == 0 {
|
||||
t.Fatalf("stats = %d entries, %d bytes", n, b)
|
||||
}
|
||||
// survives a reopen
|
||||
if err := c.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c2 := NewCache(c.Path)
|
||||
if e, found, _ := c2.Get(k); !found || string(e.JPEG) != "jpeg2" {
|
||||
t.Fatal("not there after reopening")
|
||||
}
|
||||
if n, _ := c2.Stats(); n != 2 {
|
||||
t.Fatal("count lost on reopen")
|
||||
}
|
||||
c2.Close()
|
||||
}
|
||||
|
||||
// The real thing, when ffmpeg is here: a PNG and a generated video each come back as a
|
||||
// JPEG no larger than the asked size, with the aspect kept.
|
||||
func TestMakeWithFFmpeg(t *testing.T) {
|
||||
if _, err := exec.LookPath("ffmpeg"); err != nil {
|
||||
t.Skip("no ffmpeg")
|
||||
}
|
||||
c := Probe(context.Background())
|
||||
if c.Error != "" {
|
||||
t.Skip(c.Error)
|
||||
}
|
||||
dir := t.TempDir()
|
||||
img := image.NewRGBA(image.Rect(0, 0, 640, 360))
|
||||
for y := 0; y < 360; y++ {
|
||||
for x := 0; x < 640; x++ {
|
||||
img.Set(x, y, color.RGBA{uint8(x / 3), uint8(y), 90, 255})
|
||||
}
|
||||
}
|
||||
pngPath := filepath.Join(dir, "a.png")
|
||||
f, _ := os.Create(pngPath)
|
||||
_ = png.Encode(f, img)
|
||||
f.Close()
|
||||
res, err := Make(context.Background(), c, pngPath, Opts{Size: 200})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
check(t, res, 200, 112, "cpu")
|
||||
|
||||
// a two-second test pattern, with the always-built-in mpeg4 encoder
|
||||
vid := filepath.Join(dir, "v.mp4")
|
||||
out, err := exec.Command("ffmpeg", "-nostdin", "-v", "error", "-f", "lavfi", "-i", "testsrc=size=320x180:rate=10:duration=2", "-c:v", "mpeg4", "-q:v", "5", vid).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Skipf("could not make a test video: %s", out)
|
||||
}
|
||||
res, err = Make(context.Background(), c, vid, Opts{Size: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
check(t, res, 100, 56, "")
|
||||
if res.Codec != "mpeg4" || res.Width != 320 {
|
||||
t.Fatalf("probe said %s %dx%d", res.Codec, res.Width, res.Height)
|
||||
}
|
||||
|
||||
// not media, and media that isn't
|
||||
if _, err := Make(context.Background(), c, filepath.Join(dir, "x.txt"), Opts{}); err != ErrUnsupported {
|
||||
t.Fatalf("a .txt is unsupported, got %v", err)
|
||||
}
|
||||
bad := filepath.Join(dir, "bad.mp4")
|
||||
_ = os.WriteFile(bad, []byte("not a video at all"), 0o644)
|
||||
_, err = Make(context.Background(), c, bad, Opts{})
|
||||
var de *DecodeError
|
||||
if err == nil || !isDecodeErr(err, &de) {
|
||||
t.Fatalf("garbage should be a DecodeError, got %v", err)
|
||||
}
|
||||
// the clock: a cancelled context is not the file's fault
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
|
||||
defer cancel()
|
||||
if _, err := Make(ctx, c, pngPath, Opts{}); err == nil || isDecodeErr(err, &de) {
|
||||
t.Fatalf("a timeout must not read as a decode failure, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func check(t *testing.T, res Result, w, h int, engine string) {
|
||||
t.Helper()
|
||||
im, err := jpeg.Decode(bytes.NewReader(res.JPEG))
|
||||
if err != nil {
|
||||
t.Fatalf("not a jpeg: %v", err)
|
||||
}
|
||||
if b := im.Bounds(); b.Dx() != w || b.Dy() < h-1 || b.Dy() > h+1 { // the short side rounds either way
|
||||
t.Fatalf("got %dx%d, want %dx%d", b.Dx(), b.Dy(), w, h)
|
||||
}
|
||||
if engine != "" && res.Engine != engine {
|
||||
t.Fatalf("engine %q, want %q", res.Engine, engine)
|
||||
}
|
||||
}
|
||||
|
||||
func isDecodeErr(err error, de **DecodeError) bool {
|
||||
x, ok := err.(*DecodeError)
|
||||
if ok {
|
||||
*de = x
|
||||
}
|
||||
return ok
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package metrics
|
||||
|
||||
// The ZFS ARC — the read cache in RAM — read live from /proc/spl/kstat/zfs/arcstats on a
|
||||
// machine that has the module loaded, and left out of the sample everywhere else. The kstat
|
||||
// file is counters since boot; the hit rate here is over the sampling interval, which is the
|
||||
// number that says whether the cache is doing its job *now*. The lifetime figures and the
|
||||
// pool's caching tiers (L2ARC, SLOG, special vdevs) are the storage package's business.
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const arcstatsPath = "/proc/spl/kstat/zfs/arcstats"
|
||||
|
||||
// ARC is the read cache as one sample sees it. Sizes are bytes.
|
||||
type ARC struct {
|
||||
Size uint64 `json:"size"`
|
||||
Target uint64 `json:"target"` // c: what the ARC is currently allowed to hold
|
||||
Min uint64 `json:"min"` // c_min: the floor it will not shrink below
|
||||
Max uint64 `json:"max"` // c_max: the ceiling (zfs_arc_max, or half of RAM)
|
||||
// what is in it
|
||||
MRU uint64 `json:"mru"` // recently used once
|
||||
MFU uint64 `json:"mfu"` // used more than once
|
||||
Anon uint64 `json:"anon"` // in flight: dirty, not yet on disk
|
||||
Data uint64 `json:"data"` // file contents
|
||||
Metadata uint64 `json:"metadata"` // dnodes, indirect blocks, dbufs, bonus buffers
|
||||
MetaUsed uint64 `json:"meta_used"`
|
||||
Headers uint64 `json:"headers"`
|
||||
Compressed uint64 `json:"compressed"`
|
||||
Uncompressed uint64 `json:"uncompressed"`
|
||||
MRUGhost uint64 `json:"mru_ghost"` // what it recently evicted and still remembers
|
||||
MFUGhost uint64 `json:"mfu_ghost"`
|
||||
// L2ARC, the cache devices (0 when there are none)
|
||||
L2Size uint64 `json:"l2_size"`
|
||||
L2ASize uint64 `json:"l2_asize"` // as stored on the device, after compression
|
||||
// over this interval
|
||||
ReadsPerSec float64 `json:"reads_per_sec"`
|
||||
MissesPerSec float64 `json:"misses_per_sec"`
|
||||
HitPct float64 `json:"hit_pct"` // of the interval's reads; an idle interval keeps the last one
|
||||
HitPctKnown bool `json:"hit_pct_known"` // false until an interval has had any reads
|
||||
L2HitPct float64 `json:"l2_hit_pct"`
|
||||
// since boot
|
||||
Hits uint64 `json:"hits"`
|
||||
Misses uint64 `json:"misses"`
|
||||
LifetimeHitPct float64 `json:"lifetime_hit_pct"`
|
||||
// pressure
|
||||
NoGrow bool `json:"no_grow"` // arc_no_grow: memory pressure is stopping it growing
|
||||
Throttles uint64 `json:"throttles"` // memory_throttle_count since boot
|
||||
Prunes uint64 `json:"prunes"` // arc_prune: metadata evictions asked of the filesystems
|
||||
EvictSkip uint64 `json:"evict_skip"`
|
||||
}
|
||||
|
||||
type arcPrev struct {
|
||||
hits, misses, l2hits, l2misses uint64
|
||||
hitPct float64
|
||||
known bool
|
||||
at time.Time
|
||||
}
|
||||
|
||||
// readKstat parses a kstat table ("name type data" rows after two header lines).
|
||||
func readKstat(p string) map[string]uint64 {
|
||||
s := readStr(p)
|
||||
if s == "" {
|
||||
return nil
|
||||
}
|
||||
out := map[string]uint64{}
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) != 3 || f[1] != "4" { // type 4 is KSTAT_DATA_UINT64
|
||||
continue
|
||||
}
|
||||
v, err := strconv.ParseUint(f[2], 10, 64)
|
||||
if err == nil {
|
||||
out[f[0]] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// arc reads the ARC; nil when this machine has no ZFS module loaded.
|
||||
func (c *Collector) arc(now time.Time) *ARC {
|
||||
k := readKstat(arcstatsPath)
|
||||
if k == nil {
|
||||
return nil
|
||||
}
|
||||
a := &ARC{Size: k["size"], Target: k["c"], Min: k["c_min"], Max: k["c_max"],
|
||||
MRU: k["mru_size"], MFU: k["mfu_size"], Anon: k["anon_size"], Data: k["data_size"], Metadata: k["metadata_size"],
|
||||
MetaUsed: k["arc_meta_used"], Headers: k["hdr_size"], Compressed: k["compressed_size"], Uncompressed: k["uncompressed_size"],
|
||||
MRUGhost: k["mru_ghost_size"], MFUGhost: k["mfu_ghost_size"], L2Size: k["l2_size"], L2ASize: k["l2_asize"],
|
||||
Hits: k["hits"], Misses: k["misses"], NoGrow: k["arc_no_grow"] != 0, Throttles: k["memory_throttle_count"], Prunes: k["arc_prune"], EvictSkip: k["evict_skip"]}
|
||||
if n := a.Hits + a.Misses; n > 0 {
|
||||
a.LifetimeHitPct = round1(float64(a.Hits) / float64(n) * 100)
|
||||
}
|
||||
p := c.prevARC
|
||||
if !p.at.IsZero() && a.Hits >= p.hits && a.Misses >= p.misses {
|
||||
if dt := now.Sub(p.at).Seconds(); dt > 0 {
|
||||
hits, misses := a.Hits-p.hits, a.Misses-p.misses
|
||||
a.ReadsPerSec = round1(float64(hits+misses) / dt)
|
||||
a.MissesPerSec = round1(float64(misses) / dt)
|
||||
if hits+misses > 0 {
|
||||
p.hitPct, p.known = round1(float64(hits)/float64(hits+misses)*100), true
|
||||
}
|
||||
if l2 := (k["l2_hits"] - p.l2hits) + (k["l2_misses"] - p.l2misses); l2 > 0 && k["l2_hits"] >= p.l2hits {
|
||||
a.L2HitPct = round1(float64(k["l2_hits"]-p.l2hits) / float64(l2) * 100)
|
||||
}
|
||||
}
|
||||
}
|
||||
a.HitPct, a.HitPctKnown = p.hitPct, p.known
|
||||
c.prevARC = arcPrev{hits: a.Hits, misses: a.Misses, l2hits: k["l2_hits"], l2misses: k["l2_misses"], hitPct: p.hitPct, known: p.known, at: now}
|
||||
return a
|
||||
}
|
||||
@@ -0,0 +1,259 @@
|
||||
package metrics
|
||||
|
||||
// CPU power, from the RAPL energy counters the kernel keeps under /sys/class/powercap: Intel
|
||||
// since Sandy Bridge, and AMD Zen through the same driver. Each counter is a running total in
|
||||
// µJ, so power is the difference between two readings over the time between them. Every
|
||||
// package is summed (`package-0`, and `package-1` on a two-socket board). The other top-level
|
||||
// domains are left out. `psys` claims the whole platform but is only as good as the firmware's
|
||||
// calibration, and on one machine here it reads lower than the package inside it. The MMIO
|
||||
// copy of a package (`intel-rapl-mmio:0`) measures the same thing again. This is the CPU's own
|
||||
// figure, shown beside its clock: cores, built-in graphics and memory controller, not disks,
|
||||
// fans or the power supply.
|
||||
//
|
||||
// The counters are root-only since the PLATYPUS side channel (2020). So, like the GPU client
|
||||
// walk, they are read in-process when the daemon is root, and otherwise by a root helper,
|
||||
// `hamlab cpu-energy`, started once through sudo or doas and asked over a pipe at every
|
||||
// sample: one privilege escalation per daemon, not one a second in the auth log. The helper
|
||||
// takes nothing from the daemon but the newline that asks; it finds and reads the counters
|
||||
// itself. Without root there is no reading, and nothing is shown.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
const raplRoot = "/sys/class/powercap"
|
||||
|
||||
var raplPkgRe = regexp.MustCompile(`^package-\d+$`)
|
||||
|
||||
// raplCount is one package's counter at one moment.
|
||||
type raplCount struct {
|
||||
Dir string `json:"d"`
|
||||
UJ uint64 `json:"uj"`
|
||||
}
|
||||
|
||||
type cpuPowerState struct {
|
||||
probed bool
|
||||
dirs []string // the package domains, found at the first sample
|
||||
max map[string]uint64 // each counter's range: where it wraps to zero
|
||||
useHelper bool // the counters are not readable in-process
|
||||
helper *energyHelper
|
||||
prev []raplCount
|
||||
prevAt time.Time
|
||||
}
|
||||
|
||||
// raplPackages lists the package domains under root, sorted. intel-rapl:0 is a package;
|
||||
// intel-rapl:0:0 is a part of one (core, uncore, dram).
|
||||
func raplPackages(root string) []string {
|
||||
ents, err := os.ReadDir(root)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, e := range ents {
|
||||
n := e.Name()
|
||||
if !strings.HasPrefix(n, "intel-rapl:") || strings.Count(n, ":") != 1 {
|
||||
continue
|
||||
}
|
||||
if raplPkgRe.MatchString(strings.TrimSpace(readStr(filepath.Join(root, n, "name")))) {
|
||||
out = append(out, n)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func readRAPL(root string, dirs []string) ([]raplCount, bool) {
|
||||
out := make([]raplCount, 0, len(dirs))
|
||||
for _, d := range dirs {
|
||||
b, err := os.ReadFile(filepath.Join(root, d, "energy_uj"))
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
v, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64)
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
out = append(out, raplCount{d, v})
|
||||
}
|
||||
return out, len(out) > 0
|
||||
}
|
||||
|
||||
// raplWatts is the power between two readings of the same counters. A counter that wrapped
|
||||
// is carried across its range. One that went backwards with no known range, or a set of
|
||||
// counters that changed between readings, gives no answer rather than a wrong one.
|
||||
func raplWatts(prev, cur []raplCount, max map[string]uint64, dt float64) (float64, bool) {
|
||||
if dt <= 0 || len(prev) == 0 || len(prev) != len(cur) {
|
||||
return 0, false
|
||||
}
|
||||
var uj float64
|
||||
for i := range cur {
|
||||
if cur[i].Dir != prev[i].Dir {
|
||||
return 0, false
|
||||
}
|
||||
if cur[i].UJ >= prev[i].UJ {
|
||||
uj += float64(cur[i].UJ - prev[i].UJ)
|
||||
continue
|
||||
}
|
||||
m := max[cur[i].Dir]
|
||||
if m == 0 || prev[i].UJ > m {
|
||||
return 0, false
|
||||
}
|
||||
uj += float64(m - prev[i].UJ + cur[i].UJ)
|
||||
}
|
||||
return uj / 1e6 / dt, true
|
||||
}
|
||||
|
||||
// cpuPower is the CPU packages' power since the previous sample, in watts; 0 when this
|
||||
// machine has no counters or no way to read them.
|
||||
func (c *Collector) cpuPower(now time.Time) float64 {
|
||||
p := &c.power
|
||||
if !p.probed {
|
||||
p.probed = true
|
||||
p.dirs = raplPackages(raplRoot)
|
||||
p.max = map[string]uint64{}
|
||||
for _, d := range p.dirs {
|
||||
if v, ok := readInt(filepath.Join(raplRoot, d, "max_energy_range_uj")); ok && v > 0 {
|
||||
p.max[d] = uint64(v)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(p.dirs) == 0 {
|
||||
return 0
|
||||
}
|
||||
var cur []raplCount
|
||||
ok := false
|
||||
if !p.useHelper {
|
||||
if cur, ok = readRAPL(raplRoot, p.dirs); !ok && priv.Mode() != "root" && priv.Available() {
|
||||
p.useHelper = true
|
||||
}
|
||||
}
|
||||
if p.useHelper {
|
||||
if p.helper == nil {
|
||||
p.helper = &energyHelper{}
|
||||
}
|
||||
cur, ok = p.helper.read()
|
||||
}
|
||||
if !ok {
|
||||
p.prev = nil
|
||||
return 0
|
||||
}
|
||||
w, good := raplWatts(p.prev, cur, p.max, now.Sub(p.prevAt).Seconds())
|
||||
p.prev, p.prevAt = cur, now
|
||||
if !good {
|
||||
return 0
|
||||
}
|
||||
return round1(w)
|
||||
}
|
||||
|
||||
// energyHelper is the root child that answers each newline with the package counters, one
|
||||
// JSON line. The same shape as the GPU client walk's helper, kept separate from it.
|
||||
type energyHelper struct {
|
||||
mu sync.Mutex
|
||||
cmd *exec.Cmd
|
||||
in io.WriteCloser
|
||||
out *bufio.Reader
|
||||
failedAt time.Time
|
||||
}
|
||||
|
||||
func (h *energyHelper) read() ([]raplCount, bool) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
if h.cmd == nil {
|
||||
if !h.failedAt.IsZero() && time.Since(h.failedAt) < helperRetry {
|
||||
return nil, false
|
||||
}
|
||||
if !h.start() {
|
||||
h.failedAt = time.Now()
|
||||
return nil, false
|
||||
}
|
||||
}
|
||||
// a stuck child must not stall the sampler: it is killed, and tried again later
|
||||
cmd := h.cmd
|
||||
stop := time.AfterFunc(5*time.Second, func() { cmd.Process.Kill() })
|
||||
defer stop.Stop()
|
||||
if _, err := io.WriteString(h.in, "\n"); err != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
line, err := h.out.ReadBytes('\n')
|
||||
if err != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
var cs []raplCount
|
||||
if json.Unmarshal(line, &cs) != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
return cs, len(cs) > 0
|
||||
}
|
||||
|
||||
func (h *energyHelper) start() bool {
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
full := priv.Wrap(self, "cpu-energy")
|
||||
cmd := exec.Command(full[0], full[1:]...)
|
||||
in, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return false
|
||||
}
|
||||
h.cmd, h.in, h.out = cmd, in, bufio.NewReader(out)
|
||||
return true
|
||||
}
|
||||
|
||||
func (h *energyHelper) fail() {
|
||||
if h.cmd != nil {
|
||||
h.in.Close()
|
||||
h.cmd.Process.Kill()
|
||||
go h.cmd.Wait()
|
||||
}
|
||||
h.cmd, h.in, h.out = nil, nil, nil
|
||||
h.failedAt = time.Now()
|
||||
}
|
||||
|
||||
// CPUEnergyHelper is `hamlab cpu-energy`: for every line read on stdin, one JSON line of the
|
||||
// package counters on stdout. Ends when stdin does — when the daemon that started it has gone.
|
||||
func CPUEnergyHelper(stdin io.Reader, stdout io.Writer) int {
|
||||
return cpuEnergyHelper(raplRoot, stdin, stdout)
|
||||
}
|
||||
|
||||
func cpuEnergyHelper(root string, stdin io.Reader, stdout io.Writer) int {
|
||||
dirs := raplPackages(root)
|
||||
sc := bufio.NewScanner(stdin)
|
||||
w := bufio.NewWriter(stdout)
|
||||
for sc.Scan() {
|
||||
cs, ok := readRAPL(root, dirs)
|
||||
if !ok {
|
||||
cs = []raplCount{}
|
||||
}
|
||||
b, _ := json.Marshal(cs)
|
||||
w.Write(b)
|
||||
w.WriteByte('\n')
|
||||
if w.Flush() != nil {
|
||||
return 1
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A fake /sys/class/powercap: a package with its parts, the platform domain, and the MMIO
|
||||
// copy of the same package. Only the package counts.
|
||||
func fakeRAPL(t *testing.T, domains map[string][2]string) string {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
for dir, v := range domains {
|
||||
d := filepath.Join(root, dir)
|
||||
if err := os.MkdirAll(d, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
os.WriteFile(filepath.Join(d, "name"), []byte(v[0]+"\n"), 0o644)
|
||||
os.WriteFile(filepath.Join(d, "energy_uj"), []byte(v[1]+"\n"), 0o644)
|
||||
}
|
||||
return root
|
||||
}
|
||||
|
||||
func TestRAPLPackagesOnlyCountsPackages(t *testing.T) {
|
||||
root := fakeRAPL(t, map[string][2]string{
|
||||
"intel-rapl:0": {"package-0", "1000"},
|
||||
"intel-rapl:0:0": {"core", "600"},
|
||||
"intel-rapl:0:1": {"uncore", "10"},
|
||||
"intel-rapl:1": {"psys", "300"},
|
||||
"intel-rapl-mmio:0": {"package-0", "1000"},
|
||||
})
|
||||
if got := raplPackages(root); len(got) != 1 || got[0] != "intel-rapl:0" {
|
||||
t.Fatalf("packages = %v, want [intel-rapl:0]", got)
|
||||
}
|
||||
|
||||
two := fakeRAPL(t, map[string][2]string{"intel-rapl:0": {"package-0", "5"}, "intel-rapl:1": {"package-1", "7"}})
|
||||
got := raplPackages(two)
|
||||
if strings.Join(got, ",") != "intel-rapl:0,intel-rapl:1" {
|
||||
t.Fatalf("two sockets: packages = %v", got)
|
||||
}
|
||||
cs, ok := readRAPL(two, got)
|
||||
if !ok || len(cs) != 2 || cs[0].UJ != 5 || cs[1].UJ != 7 {
|
||||
t.Fatalf("readRAPL = %+v, %v", cs, ok)
|
||||
}
|
||||
if got := raplPackages(filepath.Join(two, "nope")); len(got) != 0 {
|
||||
t.Fatalf("no powercap: packages = %v, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRAPLWatts(t *testing.T) {
|
||||
c := func(d string, uj uint64) raplCount { return raplCount{d, uj} }
|
||||
max := map[string]uint64{"intel-rapl:0": 1_000_000_000}
|
||||
cases := []struct {
|
||||
name string
|
||||
prev, cur []raplCount
|
||||
dt float64
|
||||
want float64
|
||||
ok bool
|
||||
}{
|
||||
{"steady", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:0", 10_000_000)}, 2, 5, true},
|
||||
{"two packages add", []raplCount{c("intel-rapl:0", 0), c("intel-rapl:1", 0)}, []raplCount{c("intel-rapl:0", 4_000_000), c("intel-rapl:1", 6_000_000)}, 1, 10, true},
|
||||
{"wrapped", []raplCount{c("intel-rapl:0", 999_000_000)}, []raplCount{c("intel-rapl:0", 1_000_000)}, 1, 2, true},
|
||||
{"backwards, range unknown", []raplCount{c("intel-rapl:1", 5_000_000)}, []raplCount{c("intel-rapl:1", 1)}, 1, 0, false},
|
||||
{"counters changed", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:1", 9)}, 1, 0, false},
|
||||
{"first reading", nil, []raplCount{c("intel-rapl:0", 9)}, 1, 0, false},
|
||||
{"no time passed", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:0", 9)}, 0, 0, false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
got, ok := raplWatts(tc.prev, tc.cur, max, tc.dt)
|
||||
if ok != tc.ok || (ok && (got < tc.want-1e-9 || got > tc.want+1e-9)) {
|
||||
t.Errorf("%s: got %v, %v; want %v, %v", tc.name, got, ok, tc.want, tc.ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The helper answers every newline with the counters it found itself, and stops at EOF.
|
||||
func TestCPUEnergyHelper(t *testing.T) {
|
||||
root := fakeRAPL(t, map[string][2]string{"intel-rapl:0": {"package-0", "42"}, "intel-rapl:0:0": {"core", "7"}})
|
||||
var out bytes.Buffer
|
||||
if code := cpuEnergyHelper(root, strings.NewReader("\n\n"), &out); code != 0 {
|
||||
t.Fatalf("exit %d", code)
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(out.String()), "\n")
|
||||
if len(lines) != 2 {
|
||||
t.Fatalf("got %d lines, want 2: %q", len(lines), out.String())
|
||||
}
|
||||
var cs []raplCount
|
||||
if err := json.Unmarshal([]byte(lines[1]), &cs); err != nil || len(cs) != 1 || cs[0] != (raplCount{"intel-rapl:0", 42}) {
|
||||
t.Fatalf("line = %s (%v)", lines[1], err)
|
||||
}
|
||||
}
|
||||
|
||||
// History keeps the average of the samples that had a reading.
|
||||
func TestFoldCPUPower(t *testing.T) {
|
||||
b := []Sample{{T: 1, CPUPower: 4}, {T: 2}, {T: 3, CPUPower: 8}}
|
||||
if got := fold(b).CPUPower; got != 6 {
|
||||
t.Fatalf("folded CPU power = %v, want 6", got)
|
||||
}
|
||||
if got := fold([]Sample{{T: 1}, {T: 2}}).CPUPower; got != 0 {
|
||||
t.Fatalf("no readings folded to %v, want 0", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
package metrics
|
||||
|
||||
// One kept history point is the average of the samples behind it. At the default ten-minute
|
||||
// window that bucket is a single sample and nothing is averaged; at a day it is five minutes
|
||||
// of them, and the difference matters — a graph of the last day drawn from whichever instant
|
||||
// landed on each five-minute mark would show spikes that never happened and miss the ones
|
||||
// that did.
|
||||
//
|
||||
// The newest sample of the bucket is the template: its shape, labels, totals and slow
|
||||
// sections (filesystems, UPS) are the current ones. Only the moving numbers are averaged,
|
||||
// and only where an entry is present in that sample — a disk plugged in halfway through the
|
||||
// bucket is averaged over the samples that have it.
|
||||
|
||||
func fold(b []Sample) Sample {
|
||||
out := b[len(b)-1]
|
||||
if len(b) == 1 {
|
||||
return out
|
||||
}
|
||||
n := float64(len(b))
|
||||
sum := func(get func(Sample) float64) float64 {
|
||||
var t float64
|
||||
for _, s := range b {
|
||||
t += get(s)
|
||||
}
|
||||
return t / n
|
||||
}
|
||||
sumU := func(get func(Sample) uint64) uint64 {
|
||||
var t uint64
|
||||
for _, s := range b {
|
||||
t += get(s)
|
||||
}
|
||||
return t / uint64(len(b))
|
||||
}
|
||||
out.CPU = sum(func(s Sample) float64 { return s.CPU })
|
||||
// CPU power: over the samples that have a reading, so a helper restart is not a dip to zero
|
||||
var pw, pn float64
|
||||
for _, s := range b {
|
||||
if s.CPUPower > 0 {
|
||||
pw, pn = pw+s.CPUPower, pn+1
|
||||
}
|
||||
}
|
||||
out.CPUPower = 0
|
||||
if pn > 0 {
|
||||
out.CPUPower = round1(pw / pn)
|
||||
}
|
||||
out.Load[0] = sum(func(s Sample) float64 { return s.Load[0] })
|
||||
out.Load[1] = sum(func(s Sample) float64 { return s.Load[1] })
|
||||
out.Load[2] = sum(func(s Sample) float64 { return s.Load[2] })
|
||||
|
||||
out.Cores = append([]float64(nil), out.Cores...)
|
||||
for i := range out.Cores {
|
||||
var t, c float64
|
||||
for _, s := range b {
|
||||
if i < len(s.Cores) {
|
||||
t, c = t+s.Cores[i], c+1
|
||||
}
|
||||
}
|
||||
if c > 0 {
|
||||
out.Cores[i] = t / c
|
||||
}
|
||||
}
|
||||
|
||||
out.Mem.Used = sumU(func(s Sample) uint64 { return s.Mem.Used })
|
||||
out.Mem.Avail = sumU(func(s Sample) uint64 { return s.Mem.Avail })
|
||||
out.Mem.Free = sumU(func(s Sample) uint64 { return s.Mem.Free })
|
||||
out.Mem.Cached = sumU(func(s Sample) uint64 { return s.Mem.Cached })
|
||||
out.Swap.Used = sumU(func(s Sample) uint64 { return s.Swap.Used })
|
||||
|
||||
// the rest are lists whose entries are matched by name across the bucket
|
||||
temps := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Temps))
|
||||
for _, x := range s.Temps {
|
||||
m[x.Chip+"/"+x.Label] = []float64{x.C}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Temps = append([]Temp(nil), out.Temps...)
|
||||
for i, x := range out.Temps {
|
||||
if v, ok := temps[x.Chip+"/"+x.Label]; ok {
|
||||
out.Temps[i].C = v[0]
|
||||
}
|
||||
}
|
||||
|
||||
fans := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Fans))
|
||||
for _, x := range s.Fans {
|
||||
m[x.Chip+"/"+x.Label] = []float64{float64(x.RPM)}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Fans = append([]Fan(nil), out.Fans...)
|
||||
for i, x := range out.Fans {
|
||||
if v, ok := fans[x.Chip+"/"+x.Label]; ok {
|
||||
out.Fans[i].RPM = int(v[0] + .5)
|
||||
}
|
||||
}
|
||||
|
||||
nets := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Net))
|
||||
for _, x := range s.Net {
|
||||
m[x.Iface] = []float64{x.RxBps, x.TxBps}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Net = append([]Net(nil), out.Net...)
|
||||
for i, x := range out.Net {
|
||||
if v, ok := nets[x.Iface]; ok {
|
||||
out.Net[i].RxBps, out.Net[i].TxBps = v[0], v[1]
|
||||
}
|
||||
}
|
||||
|
||||
disks := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Disks))
|
||||
for _, x := range s.Disks {
|
||||
m[x.Name] = []float64{x.ReadBps, x.WriteBps, x.UtilPct}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Disks = append([]Disk(nil), out.Disks...)
|
||||
for i, x := range out.Disks {
|
||||
if v, ok := disks[x.Name]; ok {
|
||||
out.Disks[i].ReadBps, out.Disks[i].WriteBps, out.Disks[i].UtilPct = v[0], v[1], v[2]
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// avgBy averages each sample's named numbers across the bucket, per name.
|
||||
func avgBy(b []Sample, of func(Sample) map[string][]float64) map[string][]float64 {
|
||||
sums := map[string][]float64{}
|
||||
count := map[string]float64{}
|
||||
for _, s := range b {
|
||||
for k, vs := range of(s) {
|
||||
acc := sums[k]
|
||||
if acc == nil {
|
||||
acc = make([]float64, len(vs))
|
||||
}
|
||||
for i, v := range vs {
|
||||
if i < len(acc) {
|
||||
acc[i] += v
|
||||
}
|
||||
}
|
||||
sums[k], count[k] = acc, count[k]+1
|
||||
}
|
||||
}
|
||||
for k, acc := range sums {
|
||||
for i := range acc {
|
||||
acc[i] /= count[k]
|
||||
}
|
||||
}
|
||||
return sums
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestFoldAverages(t *testing.T) {
|
||||
b := []Sample{
|
||||
{T: 1, CPU: 10, Cores: []float64{10, 0}, Mem: Mem{Total: 100, Used: 10},
|
||||
Temps: []Temp{{Chip: "c", Label: "l", C: 40}}, Net: []Net{{Iface: "eth0", RxBps: 100}},
|
||||
Disks: []Disk{{Name: "sda", ReadBps: 10, UtilPct: 5}}},
|
||||
{T: 2, CPU: 30, Cores: []float64{30, 100}, Mem: Mem{Total: 100, Used: 30},
|
||||
Temps: []Temp{{Chip: "c", Label: "l", C: 60}}, Net: []Net{{Iface: "eth0", RxBps: 300}, {Iface: "new", RxBps: 7}},
|
||||
Disks: []Disk{{Name: "sda", ReadBps: 30, UtilPct: 15}}},
|
||||
}
|
||||
f := fold(b)
|
||||
if f.T != 2 {
|
||||
t.Errorf("a folded point is stamped with the newest sample in it, got %d", f.T)
|
||||
}
|
||||
if f.CPU != 20 || f.Cores[0] != 20 || f.Cores[1] != 50 {
|
||||
t.Errorf("cpu averaged: %v %v", f.CPU, f.Cores)
|
||||
}
|
||||
if f.Mem.Used != 20 || f.Mem.Total != 100 {
|
||||
t.Errorf("memory: used averaged, total taken from the newest, got %+v", f.Mem)
|
||||
}
|
||||
if f.Temps[0].C != 50 || f.Net[0].RxBps != 200 || f.Disks[0].ReadBps != 20 || f.Disks[0].UtilPct != 10 {
|
||||
t.Errorf("lists matched by name: %v %v %v", f.Temps, f.Net, f.Disks)
|
||||
}
|
||||
if len(f.Net) != 2 || f.Net[1].RxBps != 7 {
|
||||
t.Errorf("an interface that appeared inside the bucket is averaged over the samples that have it: %v", f.Net)
|
||||
}
|
||||
// the template's own slices must not be touched: Latest() shares them
|
||||
if b[1].Cores[0] != 30 || b[1].Temps[0].C != 60 {
|
||||
t.Error("fold wrote through to the sample it templated from")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStrideAndTrim(t *testing.T) {
|
||||
c := &Collector{interval: 2 * time.Second, window: DefaultWindow}
|
||||
if got := c.stride(); got != 2*time.Second {
|
||||
t.Errorf("a window of %v over %d points is finer than the interval, so the interval wins; got %v", DefaultWindow, History, got)
|
||||
}
|
||||
c.window = 24 * time.Hour
|
||||
if got := c.stride(); got != 24*time.Hour/History {
|
||||
t.Errorf("stride at a day: %v", got)
|
||||
}
|
||||
// a full day of samples at the day stride keeps exactly the budget
|
||||
c.interval, c.window = time.Second, time.Hour
|
||||
stride := c.stride().Milliseconds()
|
||||
for i := 0; i < 5*History; i++ {
|
||||
c.keep(Sample{T: int64(i) * stride})
|
||||
}
|
||||
if len(c.ring) != History {
|
||||
t.Errorf("the point count is the budget, got %d", len(c.ring))
|
||||
}
|
||||
if span := c.ring[len(c.ring)-1].T - c.ring[0].T; span > c.window.Milliseconds() {
|
||||
t.Errorf("the kept history reaches back further than the window: %d ms", span)
|
||||
}
|
||||
// narrowing the window drops what is now too old, at once
|
||||
c.SetWindow(MinWindow)
|
||||
if span := c.ring[len(c.ring)-1].T - c.ring[0].T; span > MinWindow.Milliseconds() {
|
||||
t.Errorf("narrowing the window left %d ms of history", span)
|
||||
}
|
||||
c.SetWindow(365 * 24 * time.Hour)
|
||||
if c.Window() != MaxWindow {
|
||||
t.Errorf("a window past the maximum is the maximum, got %v", c.Window())
|
||||
}
|
||||
}
|
||||
|
||||
// Points are one stride apart — not a stride and a sample — and jitter doesn't change that.
|
||||
func TestKeepSpacing(t *testing.T) {
|
||||
c := &Collector{interval: 2 * time.Second, window: time.Hour} // stride 12 s: six samples a point
|
||||
at := int64(1_000_000)
|
||||
for i := 0; i < 6*40; i++ {
|
||||
jitter := int64((i*37)%41 - 20) // ±20 ms, deterministic
|
||||
c.keep(Sample{T: at + jitter, CPU: float64(i % 6)})
|
||||
at += 2000
|
||||
}
|
||||
if len(c.ring) != 40 {
|
||||
t.Fatalf("240 samples at six a point are 40 points, got %d", len(c.ring))
|
||||
}
|
||||
for i := 1; i < len(c.ring); i++ {
|
||||
if gap := c.ring[i].T - c.ring[i-1].T; gap < 11900 || gap > 12100 {
|
||||
t.Fatalf("point %d is %d ms after the one before; a stride is 12000", i, gap)
|
||||
}
|
||||
}
|
||||
if c.ring[3].CPU != 2.5 {
|
||||
t.Errorf("each point is the mean of its own six samples (0…5 → 2.5), got %v", c.ring[3].CPU)
|
||||
}
|
||||
// and when the stride is the interval, every sample is a point
|
||||
c = &Collector{interval: 2 * time.Second, window: DefaultWindow}
|
||||
for i := 0; i < 50; i++ {
|
||||
c.keep(Sample{T: int64(i)*2000 + int64(i%3) - 1})
|
||||
}
|
||||
if len(c.ring) != 50 {
|
||||
t.Errorf("at the default window every sample is kept, got %d of 50", len(c.ring))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFoldNVMe(t *testing.T) {
|
||||
// an Apple SSD as the kernel lists it: one controller, three namespaces
|
||||
got := foldNVMe([]Disk{
|
||||
{Name: "nvme0n1", Model: "APPLE SSD AP0512Z", Size: 500 << 30, ReadBps: 100, WriteBps: 200, UtilPct: 12, Temp: 37},
|
||||
{Name: "nvme0n2", Model: "APPLE SSD AP0512Z", Size: 3 << 20, ReadBps: 10, WriteBps: 20, UtilPct: 40},
|
||||
{Name: "nvme0n3", Model: "APPLE SSD AP0512Z", Size: 128 << 20, ReadBps: 1, WriteBps: 2, UtilPct: 5},
|
||||
{Name: "sda", Size: 100 << 30, ReadBps: 7, UtilPct: 3},
|
||||
{Name: "nvme1n1", Size: 200 << 30, ReadBps: 5},
|
||||
})
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("three namespaces of one controller are one drive: %v", names(got))
|
||||
}
|
||||
d := got[0]
|
||||
if d.Name != "nvme0" {
|
||||
t.Errorf("the drive is the controller, got %q", d.Name)
|
||||
}
|
||||
if d.Size != 500<<30+3<<20+128<<20 {
|
||||
t.Errorf("its size is all of its namespaces, got %d", d.Size)
|
||||
}
|
||||
if d.ReadBps != 111 || d.WriteBps != 222 {
|
||||
t.Errorf("rates add up, got %v/%v", d.ReadBps, d.WriteBps)
|
||||
}
|
||||
if d.UtilPct != 40 {
|
||||
t.Errorf("busy is the longest of them, not the sum: got %v", d.UtilPct)
|
||||
}
|
||||
if d.Temp != 37 {
|
||||
t.Errorf("a temperature from any namespace is the drive's, got %v", d.Temp)
|
||||
}
|
||||
if got[1].Name != "sda" || got[1].UtilPct != 3 || got[2].Name != "nvme1" {
|
||||
t.Errorf("everything else is left alone: %v", names(got))
|
||||
}
|
||||
if n := len(foldNVMe(nil)); n != 0 {
|
||||
t.Errorf("no disks, no drives, got %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func names(d []Disk) []string {
|
||||
out := make([]string, len(d))
|
||||
for i, x := range d {
|
||||
out[i] = x.Name
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,705 @@
|
||||
package metrics
|
||||
|
||||
// GPU telemetry, the way btop and intel_gpu_top get it without a vendor library: the
|
||||
// device from sysfs, and who is using it from DRM fdinfo.
|
||||
//
|
||||
// - The device: /sys/class/drm/card<n> with a driver we know to be a GPU. Clock from the
|
||||
// driver's frequency files, temperature/power/fan from the hwmon under the device, and
|
||||
// how busy it is from the one counter every Intel GT exports without root — RC6 idle
|
||||
// residency, so busy = 100 − idle. amdgpu says it outright (gpu_busy_percent).
|
||||
// - The clients: every process holding /dev/dri/* open has a drm-engine-<name> running
|
||||
// total (nanoseconds) in /proc/<pid>/fdinfo/<fd>; the delta between two reads is that
|
||||
// process's share of each engine, and the sum across clients is the engine's load. Other
|
||||
// users' fdinfo is unreadable, so when the daemon isn't root it keeps one root child
|
||||
// (`hamlab drm-clients`) that does the walk, asked over a pipe — one sudo per daemon
|
||||
// start, not one per second.
|
||||
//
|
||||
// A GPU with no counter of either kind (Asahi, today) is still listed: a name is better
|
||||
// than silence about hardware the machine has.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// GPU is one graphics device as a Sample reports it.
|
||||
type GPU struct {
|
||||
Name string `json:"name"`
|
||||
Driver string `json:"driver"`
|
||||
Bus string `json:"bus,omitempty"` // PCI address; empty for a platform device
|
||||
Integrated bool `json:"integrated,omitempty"`
|
||||
Usage float64 `json:"usage"` // % busy
|
||||
UsageSrc string `json:"usage_src,omitempty"` // "idle" (100 − idle residency), "busy" (the driver's own %), "clients" (the busiest engine); "" when nothing measures it
|
||||
Engines map[string]float64 `json:"engines,omitempty"` // render/video/copy/enhance… % from the clients, normalised by engine count
|
||||
Temp float64 `json:"temp,omitempty"`
|
||||
TempSrc string `json:"temp_src,omitempty"` // "gpu" from the device's own sensor, "package" the CPU die an integrated GPU shares
|
||||
PowerW float64 `json:"power_w,omitempty"`
|
||||
PowerCapW float64 `json:"power_cap_w,omitempty"`
|
||||
FanRPM int `json:"fan_rpm,omitempty"`
|
||||
MHz float64 `json:"mhz,omitempty"` // actual clock (0 when power-gated)
|
||||
ReqMHz float64 `json:"req_mhz,omitempty"` // what the driver asked for
|
||||
MaxMHz float64 `json:"max_mhz,omitempty"`
|
||||
MemUsed uint64 `json:"mem_used,omitempty"` // VRAM used, or the clients' resident bytes where the driver has no VRAM counter
|
||||
MemTotal uint64 `json:"mem_total,omitempty"`
|
||||
Clients []GPUClient `json:"clients"`
|
||||
ClientsOK bool `json:"clients_ok"` // false when fdinfo of other users' processes could not be read
|
||||
Stalled bool `json:"stalled,omitempty"` // a sysfs read of this card never returned; its readings are paused until restart
|
||||
}
|
||||
|
||||
// GPUClient is one process with the device open.
|
||||
type GPUClient struct {
|
||||
PID int `json:"pid"`
|
||||
Name string `json:"name"`
|
||||
Usage float64 `json:"usage"` // its busiest engine, %
|
||||
Engines map[string]float64 `json:"engines,omitempty"`
|
||||
Mem uint64 `json:"mem,omitempty"` // resident bytes
|
||||
}
|
||||
|
||||
// gpuDrivers are DRM drivers that drive a GPU, not just a scanout (simpledrm, ast, the
|
||||
// Apple display coprocessor) — those have no engines and nothing to say.
|
||||
var gpuDrivers = map[string]bool{"i915": true, "xe": true, "amdgpu": true, "radeon": true, "nouveau": true, "nvidia-drm": true,
|
||||
"asahi": true, "panfrost": true, "panthor": true, "v3d": true, "msm": true, "etnaviv": true, "lima": true, "powervr": true, "virtio_gpu": true}
|
||||
|
||||
type gpuDev struct {
|
||||
card, dev, key, bus, driver, name, hwmon string
|
||||
integrated, stalled bool
|
||||
prevIdle, prevEnergy float64
|
||||
prevAt time.Time
|
||||
}
|
||||
|
||||
type gpuState struct {
|
||||
devs []*gpuDev
|
||||
listedAt time.Time
|
||||
clients map[string]clientPrev // device key + client id → engine totals at the previous scan
|
||||
last []drmFD // the newest client walk, reused between walks
|
||||
scanAt time.Time
|
||||
helper *drmHelper
|
||||
ok bool // the last walk saw every process
|
||||
}
|
||||
|
||||
type clientPrev struct {
|
||||
ns map[string]uint64
|
||||
at time.Time
|
||||
}
|
||||
|
||||
const gpuRelist = time.Minute // how often the card list is re-read: hotplug is rare, the readdir is cheap
|
||||
const gpuScanEvery = time.Second // engine shares over less than this are noise
|
||||
|
||||
// gpus is the GPU section of a sample. cpuTemp stands in for an integrated GPU's
|
||||
// temperature: it is the same die.
|
||||
func (c *Collector) gpus(now time.Time, cpuTemp float64) []GPU {
|
||||
g := &c.gpu
|
||||
if g.devs == nil || now.Sub(g.listedAt) >= gpuRelist {
|
||||
g.devs, g.listedAt = listGPUs(g.devs), now
|
||||
}
|
||||
if len(g.devs) == 0 {
|
||||
return []GPU{}
|
||||
}
|
||||
if g.clients == nil {
|
||||
g.clients = map[string]clientPrev{}
|
||||
}
|
||||
if g.scanAt.IsZero() || now.Sub(g.scanAt) >= gpuScanEvery {
|
||||
g.last, g.ok = c.scanClients()
|
||||
g.scanAt = now
|
||||
}
|
||||
out := make([]GPU, 0, len(g.devs))
|
||||
live := map[string]bool{}
|
||||
for _, d := range g.devs {
|
||||
x := GPU{Name: d.name, Driver: d.driver, Bus: d.bus, Integrated: d.integrated, Clients: []GPUClient{}, ClientsOK: g.ok}
|
||||
d.readWithin(&x, now)
|
||||
if x.Temp == 0 && d.integrated && cpuTemp > 0 {
|
||||
x.Temp, x.TempSrc = cpuTemp, "package"
|
||||
}
|
||||
// the clients: one entry per client id, under the first pid that holds it (a child
|
||||
// that inherited the fd shares the client)
|
||||
byID := map[string]*GPUClient{}
|
||||
var order []string
|
||||
engines := map[string]float64{}
|
||||
var resident uint64
|
||||
for _, f := range g.last {
|
||||
fi := parseFdinfo(f.Info)
|
||||
if fi.pdev != "" && fi.pdev != d.bus || fi.pdev == "" && fi.driver != d.driver {
|
||||
continue
|
||||
}
|
||||
key := d.key + "/" + fi.client
|
||||
live[key] = true
|
||||
if byID[fi.client] != nil {
|
||||
continue
|
||||
}
|
||||
cl := &GPUClient{PID: f.PID, Name: f.Comm, Mem: fi.mem}
|
||||
byID[fi.client] = cl
|
||||
order = append(order, fi.client)
|
||||
resident += fi.mem
|
||||
if prev, ok := g.clients[key]; ok {
|
||||
if dt := now.Sub(prev.at).Nanoseconds(); dt > 0 {
|
||||
for e, ns := range fi.engines {
|
||||
p, ok := prev.ns[e]
|
||||
if !ok || ns < p {
|
||||
continue
|
||||
}
|
||||
pct := float64(ns-p) / float64(dt) * 100
|
||||
if n := fi.capacity[e]; n > 1 {
|
||||
pct /= float64(n)
|
||||
}
|
||||
pct = round1(pct)
|
||||
if cl.Engines == nil {
|
||||
cl.Engines = map[string]float64{}
|
||||
}
|
||||
cl.Engines[e] = pct
|
||||
engines[e] += pct
|
||||
if pct > cl.Usage {
|
||||
cl.Usage = pct
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
g.clients[key] = clientPrev{ns: fi.engines, at: now}
|
||||
}
|
||||
for _, id := range order {
|
||||
x.Clients = append(x.Clients, *byID[id])
|
||||
}
|
||||
sort.SliceStable(x.Clients, func(i, j int) bool { return x.Clients[i].Usage > x.Clients[j].Usage })
|
||||
if len(engines) > 0 {
|
||||
x.Engines = map[string]float64{}
|
||||
for e, v := range engines {
|
||||
if v > 100 {
|
||||
v = 100
|
||||
}
|
||||
x.Engines[e] = round1(v)
|
||||
}
|
||||
}
|
||||
if x.UsageSrc == "" && len(x.Engines) > 0 {
|
||||
for _, v := range x.Engines {
|
||||
if v > x.Usage {
|
||||
x.Usage = v
|
||||
}
|
||||
}
|
||||
x.UsageSrc = "clients"
|
||||
}
|
||||
if x.MemTotal == 0 && x.MemUsed == 0 {
|
||||
x.MemUsed = resident
|
||||
}
|
||||
out = append(out, x)
|
||||
}
|
||||
for k := range g.clients { // a client that closed the device
|
||||
if !live[k] {
|
||||
delete(g.clients, k)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// listGPUs enumerates the cards, keeping the counters of those already known.
|
||||
func listGPUs(known []*gpuDev) []*gpuDev {
|
||||
ents, _ := os.ReadDir("/sys/class/drm")
|
||||
old := map[string]*gpuDev{}
|
||||
for _, d := range known {
|
||||
old[d.card] = d
|
||||
}
|
||||
out := []*gpuDev{}
|
||||
for _, e := range ents {
|
||||
n := e.Name()
|
||||
if !strings.HasPrefix(n, "card") || strings.Contains(n, "-") {
|
||||
continue
|
||||
}
|
||||
card := "/sys/class/drm/" + n
|
||||
dev, err := filepath.EvalSymlinks(card + "/device")
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
driver := ueventValue(dev+"/uevent", "DRIVER")
|
||||
if driver == "" { // some drivers hang the DRIVER key off the parent
|
||||
driver = filepath.Base(readLink(dev + "/driver"))
|
||||
}
|
||||
if !gpuDrivers[driver] {
|
||||
continue
|
||||
}
|
||||
if d, ok := old[card]; ok && d.dev == dev {
|
||||
out = append(out, d)
|
||||
continue
|
||||
}
|
||||
d := &gpuDev{card: card, dev: dev, driver: driver}
|
||||
if pci := ueventValue(dev+"/uevent", "PCI_SLOT_NAME"); pci != "" {
|
||||
d.bus, d.key = pci, pci
|
||||
// an on-die GPU hangs off root bus 0 (Intel's 00:02.0, an AMD APU's 00:08.0 bridge)
|
||||
d.integrated = strings.HasPrefix(pci, "0000:00:")
|
||||
} else {
|
||||
d.key = "driver:" + driver
|
||||
d.integrated = true
|
||||
}
|
||||
if hw, _ := filepath.Glob(dev + "/hwmon/hwmon*"); len(hw) > 0 {
|
||||
d.hwmon = hw[0]
|
||||
}
|
||||
d.name = gpuName(dev, d.bus, driver)
|
||||
out = append(out, d)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].card < out[j].card })
|
||||
return out
|
||||
}
|
||||
|
||||
// RenderNodeName is the GPU behind a render node ("/dev/dri/renderD128" → "Intel Arc A380"),
|
||||
// named the same way the Processes tab names it; "" when sysfs has nothing for it.
|
||||
func RenderNodeName(node string) string {
|
||||
dev, err := filepath.EvalSymlinks("/sys/class/drm/" + filepath.Base(node) + "/device")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
driver := ueventValue(dev+"/uevent", "DRIVER")
|
||||
if driver == "" {
|
||||
driver = filepath.Base(readLink(dev + "/driver"))
|
||||
}
|
||||
return gpuName(dev, ueventValue(dev+"/uevent", "PCI_SLOT_NAME"), driver)
|
||||
}
|
||||
|
||||
// RenderNodeIntegrated reports a render node whose GPU sits on the CPU die (PCI root bus 0,
|
||||
// or no PCI bus at all), as opposed to a card on the bus.
|
||||
func RenderNodeIntegrated(node string) bool {
|
||||
dev, err := filepath.EvalSymlinks("/sys/class/drm/" + filepath.Base(node) + "/device")
|
||||
if err != nil {
|
||||
return true
|
||||
}
|
||||
pci := ueventValue(dev+"/uevent", "PCI_SLOT_NAME")
|
||||
return pci == "" || strings.HasPrefix(pci, "0000:00:")
|
||||
}
|
||||
|
||||
var gpuVendors = map[string]string{"0x8086": "Intel", "0x1002": "AMD", "0x10de": "NVIDIA", "0x1af4": "Virtio", "0x106b": "Apple"}
|
||||
|
||||
var (
|
||||
revRe = regexp.MustCompile(`\s*\(rev [^)]*\)`)
|
||||
bracketRe = regexp.MustCompile(`\[([^\]]+)\]`)
|
||||
)
|
||||
|
||||
// gpuName is "Intel Arc A380": from the embedded PCI ID table first, so the name is the
|
||||
// same on every machine; from lspci for a card newer than the table; and failing both,
|
||||
// the vendor and the raw device ID.
|
||||
func gpuName(dev, bus, driver string) string {
|
||||
if bus != "" {
|
||||
vid := strings.TrimPrefix(strings.TrimSpace(readStr(dev+"/vendor")), "0x")
|
||||
did := strings.TrimPrefix(strings.TrimSpace(readStr(dev+"/device")), "0x")
|
||||
if vn, dn := pciName(vid, did); dn != "" {
|
||||
return lspciName(vn + " " + dn)
|
||||
}
|
||||
if out, ok := runOut("lspci", "-s", bus); ok {
|
||||
if _, desc, found := strings.Cut(out, ": "); found {
|
||||
return lspciName(desc)
|
||||
}
|
||||
}
|
||||
}
|
||||
vendor := gpuVendors[strings.TrimSpace(readStr(dev+"/vendor"))]
|
||||
if compat := readStr(dev + "/of_node/compatible"); compat != "" { // "apple,agx-t8103\0apple,agx-g13g"
|
||||
first := strings.SplitN(compat, "\x00", 2)[0]
|
||||
if v, model, ok := strings.Cut(first, ","); ok {
|
||||
if vendor == "" && v != "" {
|
||||
vendor = strings.ToUpper(v[:1]) + v[1:]
|
||||
}
|
||||
return vendor + " GPU (" + model + ")"
|
||||
}
|
||||
}
|
||||
if vendor == "" && driver != "" {
|
||||
vendor = strings.ToUpper(driver[:1]) + driver[1:]
|
||||
}
|
||||
if id := strings.TrimPrefix(strings.TrimSpace(readStr(dev+"/device")), "0x"); id != "" {
|
||||
return strings.TrimSpace(vendor + " GPU " + id)
|
||||
}
|
||||
return strings.TrimSpace(vendor + " GPU")
|
||||
}
|
||||
|
||||
// lspciName turns "Intel Corporation DG2 [Arc A380] (rev 05)" into "Intel Arc A380".
|
||||
func lspciName(desc string) string {
|
||||
desc = revRe.ReplaceAllString(strings.TrimSpace(desc), "")
|
||||
vendor := ""
|
||||
for _, v := range []struct{ long, short string }{{"Intel Corporation", "Intel"}, {"Advanced Micro Devices, Inc. [AMD/ATI]", "AMD"}, {"Advanced Micro Devices, Inc. [AMD]", "AMD"}, {"NVIDIA Corporation", "NVIDIA"}, {"Red Hat, Inc.", "Virtio"}} {
|
||||
if strings.HasPrefix(desc, v.long) {
|
||||
vendor = v.short
|
||||
desc = strings.TrimSpace(desc[len(v.long):])
|
||||
break
|
||||
}
|
||||
}
|
||||
if vendor == "" { // vendor not recognised: its first word
|
||||
if i := strings.IndexByte(desc, ' '); i > 0 {
|
||||
vendor, desc = desc[:i], strings.TrimSpace(desc[i+1:])
|
||||
}
|
||||
}
|
||||
if m := bracketRe.FindStringSubmatch(desc); m != nil {
|
||||
desc = m[1]
|
||||
}
|
||||
return strings.TrimSpace(vendor + " " + desc)
|
||||
}
|
||||
|
||||
// readWithin is read with a deadline. A sysfs read that does not come back must not take
|
||||
// the whole sampler with it — every graph on every page of this machine stops — so the
|
||||
// device is marked stalled, its goroutine abandoned, and the rest of the sample goes on.
|
||||
const gpuReadWithin = 3 * time.Second
|
||||
|
||||
func (d *gpuDev) readWithin(x *GPU, now time.Time) {
|
||||
if d.stalled {
|
||||
x.Stalled = true
|
||||
return
|
||||
}
|
||||
done := make(chan GPU, 1)
|
||||
go func() {
|
||||
y := *x
|
||||
d.read(&y, now)
|
||||
done <- y
|
||||
}()
|
||||
select {
|
||||
case y := <-done:
|
||||
*x = y
|
||||
case <-time.After(gpuReadWithin):
|
||||
d.stalled, x.Stalled = true, true
|
||||
log.Printf("gpu: %s (%s): a sysfs read did not return within %s; its readings are paused until restart", d.name, d.card, gpuReadWithin)
|
||||
}
|
||||
}
|
||||
|
||||
// read fills the device-level numbers: clock, idle/busy, hwmon.
|
||||
func (d *gpuDev) read(x *GPU, now time.Time) {
|
||||
dt := now.Sub(d.prevAt).Seconds()
|
||||
switch d.driver {
|
||||
case "i915":
|
||||
x.MHz = readF(d.card + "/gt_act_freq_mhz")
|
||||
x.ReqMHz = readF(d.card + "/gt_cur_freq_mhz")
|
||||
x.MaxMHz = readF(d.card + "/gt_max_freq_mhz")
|
||||
idle, ok := readInt(d.card + "/gt/gt0/rc6_residency_ms")
|
||||
if !ok {
|
||||
idle, ok = readInt(d.card + "/power/rc6_residency_ms")
|
||||
}
|
||||
if ok {
|
||||
d.busyFromIdle(x, float64(idle), dt)
|
||||
}
|
||||
case "xe":
|
||||
gt := d.dev + "/tile0/gt0"
|
||||
x.MHz = readF(gt + "/freq0/act_freq")
|
||||
x.ReqMHz = readF(gt + "/freq0/cur_freq")
|
||||
x.MaxMHz = readF(gt + "/freq0/max_freq")
|
||||
if idle, ok := readInt(gt + "/gtidle/idle_residency_ms"); ok {
|
||||
d.busyFromIdle(x, float64(idle), dt)
|
||||
}
|
||||
case "amdgpu":
|
||||
if v, ok := readInt(d.dev + "/gpu_busy_percent"); ok {
|
||||
x.Usage, x.UsageSrc = float64(v), "busy"
|
||||
}
|
||||
x.MemUsed = uint64(readF(d.dev + "/mem_info_vram_used"))
|
||||
x.MemTotal = uint64(readF(d.dev + "/mem_info_vram_total"))
|
||||
// "1: 1800Mhz *" — the starred level is the current one, the last the highest
|
||||
for _, line := range strings.Split(readStr(d.dev+"/pp_dpm_sclk"), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) < 2 {
|
||||
continue
|
||||
}
|
||||
mhz := leadingNum(f[1])
|
||||
x.MaxMHz = mhz
|
||||
if strings.HasSuffix(line, "*") {
|
||||
x.MHz = mhz
|
||||
}
|
||||
}
|
||||
}
|
||||
if d.hwmon != "" {
|
||||
if t := readF(d.hwmon + "/temp1_input"); t > 0 {
|
||||
x.Temp, x.TempSrc = t/1000, "gpu"
|
||||
}
|
||||
if p := readF(d.hwmon + "/power1_input"); p > 0 {
|
||||
x.PowerW = round1(p / 1e6)
|
||||
} else if p := readF(d.hwmon + "/power1_average"); p > 0 {
|
||||
x.PowerW = round1(p / 1e6)
|
||||
} else if e := readF(d.hwmon + "/energy1_input"); e > 0 { // µJ, cumulative
|
||||
if d.prevEnergy > 0 && e >= d.prevEnergy && dt > 0 && !d.prevAt.IsZero() {
|
||||
x.PowerW = round1((e - d.prevEnergy) / 1e6 / dt)
|
||||
}
|
||||
d.prevEnergy = e
|
||||
}
|
||||
if p := readF(d.hwmon + "/power1_cap"); p > 0 {
|
||||
x.PowerCapW = p / 1e6
|
||||
} else if p := readF(d.hwmon + "/power1_max"); p > 0 {
|
||||
x.PowerCapW = p / 1e6
|
||||
}
|
||||
x.FanRPM = int(readF(d.hwmon + "/fan1_input"))
|
||||
}
|
||||
d.prevAt = now
|
||||
}
|
||||
|
||||
// busyFromIdle: the GT's idle residency is a running ms total; the share of the interval
|
||||
// it did not cover is how busy the device was.
|
||||
func (d *gpuDev) busyFromIdle(x *GPU, idle, dt float64) {
|
||||
x.UsageSrc = "idle"
|
||||
if d.prevAt.IsZero() || dt <= 0 || idle < d.prevIdle {
|
||||
d.prevIdle = idle
|
||||
return
|
||||
}
|
||||
busy := 100 - (idle-d.prevIdle)/(dt*1000)*100
|
||||
if busy < 0 {
|
||||
busy = 0
|
||||
}
|
||||
if busy > 100 {
|
||||
busy = 100
|
||||
}
|
||||
x.Usage = round1(busy)
|
||||
d.prevIdle = idle
|
||||
}
|
||||
|
||||
// ---- DRM fdinfo: who has the device open ----
|
||||
|
||||
// drmFD is one client of a DRM device: the fdinfo text of one open fd, with its owner.
|
||||
type drmFD struct {
|
||||
PID int `json:"pid"`
|
||||
Comm string `json:"comm"`
|
||||
Info string `json:"info"`
|
||||
}
|
||||
|
||||
type fdinfo struct {
|
||||
driver, pdev, client string
|
||||
engines map[string]uint64 // ns, cumulative
|
||||
capacity map[string]int
|
||||
mem uint64 // bytes
|
||||
}
|
||||
|
||||
func parseFdinfo(s string) fdinfo {
|
||||
fi := fdinfo{engines: map[string]uint64{}, capacity: map[string]int{}}
|
||||
var resident, memory uint64
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
k, v, ok := strings.Cut(line, ":")
|
||||
if !ok || !strings.HasPrefix(k, "drm-") {
|
||||
continue
|
||||
}
|
||||
v = strings.TrimSpace(v)
|
||||
switch {
|
||||
case k == "drm-driver":
|
||||
fi.driver = v
|
||||
case k == "drm-pdev":
|
||||
fi.pdev = v
|
||||
case k == "drm-client-id":
|
||||
fi.client = v
|
||||
case strings.HasPrefix(k, "drm-engine-capacity-"):
|
||||
fi.capacity[k[len("drm-engine-capacity-"):]], _ = strconv.Atoi(v)
|
||||
case strings.HasPrefix(k, "drm-engine-"):
|
||||
n, _ := strconv.ParseUint(strings.TrimSuffix(v, " ns"), 10, 64)
|
||||
fi.engines[k[len("drm-engine-"):]] = n
|
||||
case strings.HasPrefix(k, "drm-resident-"):
|
||||
resident += kib(v)
|
||||
case strings.HasPrefix(k, "drm-memory-"):
|
||||
memory += kib(v)
|
||||
}
|
||||
}
|
||||
fi.mem = resident
|
||||
if fi.mem == 0 {
|
||||
fi.mem = memory
|
||||
}
|
||||
return fi
|
||||
}
|
||||
|
||||
// kib parses "52624 KiB" (or MiB, or a bare count of bytes) to bytes.
|
||||
func kib(v string) uint64 {
|
||||
f := strings.Fields(v)
|
||||
if len(f) == 0 {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.ParseUint(f[0], 10, 64)
|
||||
if len(f) > 1 {
|
||||
switch f[1] {
|
||||
case "KiB":
|
||||
n <<= 10
|
||||
case "MiB":
|
||||
n <<= 20
|
||||
case "GiB":
|
||||
n <<= 30
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// drmScan walks /proc for every fd open on /dev/dri/*, one entry per client per process.
|
||||
// Run as root it sees everyone; as a user, only that user's processes.
|
||||
func drmScan() (out []drmFD, complete bool) {
|
||||
ents, err := os.ReadDir("/proc")
|
||||
if err != nil {
|
||||
return []drmFD{}, false
|
||||
}
|
||||
complete = true
|
||||
out = []drmFD{}
|
||||
for _, e := range ents {
|
||||
pid, err := strconv.Atoi(e.Name())
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
fds, err := os.ReadDir("/proc/" + e.Name() + "/fd")
|
||||
if err != nil {
|
||||
if !os.IsNotExist(err) {
|
||||
complete = false
|
||||
}
|
||||
continue
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
comm := ""
|
||||
for _, fd := range fds {
|
||||
target, err := os.Readlink("/proc/" + e.Name() + "/fd/" + fd.Name())
|
||||
if err != nil || !strings.HasPrefix(target, "/dev/dri/") {
|
||||
continue
|
||||
}
|
||||
info := readStr("/proc/" + e.Name() + "/fdinfo/" + fd.Name())
|
||||
_, rest, ok := strings.Cut(info, "drm-client-id:")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
id := strings.TrimSpace(strings.SplitN(rest, "\n", 2)[0])
|
||||
if seen[id] {
|
||||
continue
|
||||
}
|
||||
seen[id] = true
|
||||
if comm == "" {
|
||||
comm = strings.TrimSpace(readStr("/proc/" + e.Name() + "/comm"))
|
||||
}
|
||||
out = append(out, drmFD{PID: pid, Comm: comm, Info: info})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].PID < out[j].PID })
|
||||
return out, complete
|
||||
}
|
||||
|
||||
// scanClients walks in-process when that sees everything (root), else through the helper.
|
||||
func (c *Collector) scanClients() ([]drmFD, bool) {
|
||||
if priv.Mode() == "root" || !priv.Available() {
|
||||
return drmScan()
|
||||
}
|
||||
if c.gpu.helper == nil {
|
||||
c.gpu.helper = &drmHelper{}
|
||||
}
|
||||
if fds, ok := c.gpu.helper.scan(); ok {
|
||||
return fds, true
|
||||
}
|
||||
return drmScan() // the helper is down: our own processes are better than nothing
|
||||
}
|
||||
|
||||
// drmHelper is the root child that answers "scan" requests. One line in, one JSON line out.
|
||||
type drmHelper struct {
|
||||
mu sync.Mutex
|
||||
cmd *exec.Cmd
|
||||
in io.WriteCloser
|
||||
out *bufio.Reader
|
||||
failedAt time.Time
|
||||
}
|
||||
|
||||
const helperRetry = 30 * time.Second
|
||||
|
||||
func (h *drmHelper) scan() ([]drmFD, bool) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
if h.cmd == nil {
|
||||
if !h.failedAt.IsZero() && time.Since(h.failedAt) < helperRetry {
|
||||
return nil, false
|
||||
}
|
||||
if !h.start() {
|
||||
h.failedAt = time.Now()
|
||||
return nil, false
|
||||
}
|
||||
}
|
||||
// a stuck child must not stall the sampler: it is killed, and tried again later
|
||||
cmd := h.cmd
|
||||
stop := time.AfterFunc(5*time.Second, func() { cmd.Process.Kill() })
|
||||
defer stop.Stop()
|
||||
if _, err := io.WriteString(h.in, "\n"); err != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
line, err := h.out.ReadBytes('\n')
|
||||
if err != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
var fds []drmFD
|
||||
if json.Unmarshal(line, &fds) != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
if fds == nil {
|
||||
fds = []drmFD{}
|
||||
}
|
||||
return fds, true
|
||||
}
|
||||
|
||||
func (h *drmHelper) start() bool {
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
full := priv.Wrap(self, "drm-clients")
|
||||
cmd := exec.Command(full[0], full[1:]...)
|
||||
in, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return false
|
||||
}
|
||||
h.cmd, h.in, h.out = cmd, in, bufio.NewReaderSize(out, 1<<20)
|
||||
return true
|
||||
}
|
||||
|
||||
func (h *drmHelper) fail() {
|
||||
if h.cmd != nil {
|
||||
h.in.Close()
|
||||
h.cmd.Process.Kill()
|
||||
go h.cmd.Wait()
|
||||
}
|
||||
h.cmd, h.in, h.out = nil, nil, nil
|
||||
h.failedAt = time.Now()
|
||||
}
|
||||
|
||||
// DRMClientsHelper is `hamlab drm-clients`: for every line read on stdin, one JSON line of
|
||||
// drmFD on stdout. Ends when stdin does — when the daemon that started it has gone.
|
||||
func DRMClientsHelper(stdin io.Reader, stdout io.Writer) int {
|
||||
sc := bufio.NewScanner(stdin)
|
||||
w := bufio.NewWriter(stdout)
|
||||
for sc.Scan() {
|
||||
fds, _ := drmScan()
|
||||
b, _ := json.Marshal(fds)
|
||||
w.Write(b)
|
||||
w.WriteByte('\n')
|
||||
if w.Flush() != nil {
|
||||
return 1
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ---- small helpers ----
|
||||
|
||||
func readF(p string) float64 {
|
||||
v, err := strconv.ParseFloat(strings.TrimSpace(readStr(p)), 64)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func readLink(p string) string {
|
||||
t, err := os.Readlink(p)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func ueventValue(p, key string) string {
|
||||
for _, line := range strings.Split(readStr(p), "\n") {
|
||||
if k, v, ok := strings.Cut(line, "="); ok && k == key {
|
||||
return strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const fdinfoA = `pos: 0
|
||||
flags: 02100002
|
||||
drm-driver: i915
|
||||
drm-client-id: 544
|
||||
drm-pdev: 0000:00:02.0
|
||||
drm-total-system0: 52624 KiB
|
||||
drm-resident-system0: 50140 KiB
|
||||
drm-resident-stolen-system0: 32 KiB
|
||||
drm-engine-render: 1000000000 ns
|
||||
drm-engine-copy: 0 ns
|
||||
drm-engine-video: 2000000000 ns
|
||||
drm-engine-capacity-video: 2
|
||||
drm-engine-video-enhance: 0 ns
|
||||
`
|
||||
|
||||
func TestParseFdinfo(t *testing.T) {
|
||||
fi := parseFdinfo(fdinfoA)
|
||||
if fi.driver != "i915" || fi.pdev != "0000:00:02.0" || fi.client != "544" {
|
||||
t.Fatalf("identity: %+v", fi)
|
||||
}
|
||||
if fi.engines["video"] != 2000000000 || fi.capacity["video"] != 2 {
|
||||
t.Errorf("video engine: %+v", fi)
|
||||
}
|
||||
if fi.mem != (50140+32)<<10 {
|
||||
t.Errorf("resident = %d", fi.mem)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLspciName(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"Intel Corporation DG2 [Arc A380] (rev 05)": "Intel Arc A380",
|
||||
"Intel Corporation RocketLake-S GT1 [UHD Graphics 750] (rev 04)": "Intel UHD Graphics 750",
|
||||
"Intel Corporation Alder Lake-N [UHD Graphics]": "Intel UHD Graphics",
|
||||
"Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c7)": "AMD Radeon RX 6600/6600 XT/6600M",
|
||||
"NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)": "NVIDIA GeForce RTX 3060 Lite Hash Rate",
|
||||
"Moore Threads Technology Co.,Ltd MTT S80": "Moore Threads Technology Co.,Ltd MTT S80",
|
||||
} {
|
||||
if got := lspciName(in); got != want {
|
||||
t.Errorf("%q → %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Two scans a second apart: a client that ran its render engine for half of it and both
|
||||
// video engines flat out reads 50% render, 100% video, and the device's engines follow.
|
||||
func TestGPUClientsShare(t *testing.T) {
|
||||
c := NewCollector()
|
||||
d := &gpuDev{card: "/nonexistent/card9", dev: "/nonexistent", key: "0000:00:02.0", bus: "0000:00:02.0", driver: "i915", name: "Test GPU", integrated: true}
|
||||
c.gpu.devs = []*gpuDev{d}
|
||||
c.gpu.listedAt = time.Now().Add(time.Hour) // keep the fake list
|
||||
c.gpu.clients = map[string]clientPrev{}
|
||||
t0 := time.Now()
|
||||
c.gpu.last = []drmFD{{PID: 7, Comm: "ffmpeg", Info: fdinfoA}}
|
||||
c.gpu.scanAt = t0 // no walk: the fake list stands
|
||||
first := c.gpus(t0, 40)
|
||||
if len(first) != 1 || len(first[0].Clients) != 1 || first[0].Clients[0].Usage != 0 {
|
||||
t.Fatalf("first pass: %+v", first)
|
||||
}
|
||||
if first[0].Temp != 40 || first[0].TempSrc != "package" {
|
||||
t.Errorf("an integrated GPU wears the package temperature: %+v", first[0])
|
||||
}
|
||||
later := fdinfoA
|
||||
later = replace(later, "drm-engine-render: 1000000000 ns", "drm-engine-render: 1500000000 ns")
|
||||
later = replace(later, "drm-engine-video: 2000000000 ns", "drm-engine-video: 4000000000 ns")
|
||||
c.gpu.last = []drmFD{{PID: 7, Comm: "ffmpeg", Info: later}}
|
||||
c.gpu.scanAt = t0.Add(time.Second)
|
||||
got := c.gpus(t0.Add(time.Second), 40)
|
||||
cl := got[0].Clients[0]
|
||||
if cl.Engines["render"] != 50 || cl.Engines["video"] != 100 || cl.Usage != 100 {
|
||||
t.Errorf("client share: %+v", cl)
|
||||
}
|
||||
if got[0].Engines["video"] != 100 || got[0].Usage != 100 || got[0].UsageSrc != "clients" {
|
||||
t.Errorf("device from clients: %+v", got[0])
|
||||
}
|
||||
if got[0].MemUsed != cl.Mem {
|
||||
t.Errorf("resident memory should be the clients' sum: %d vs %d", got[0].MemUsed, cl.Mem)
|
||||
}
|
||||
// the Processes table joins the client to its row
|
||||
c.ring = []Sample{{T: 1, GPUs: got}}
|
||||
c.prevProc = nil
|
||||
table := c.Processes()
|
||||
if len(table.GPUs) != 1 {
|
||||
t.Errorf("table carries the GPUs: %+v", table.GPUs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFlattenGPU(t *testing.T) {
|
||||
s := Sample{T: 5, GPUs: []GPU{{Name: "Intel Arc A380", Driver: "i915", Bus: "0000:03:00.0", Usage: 12, UsageSrc: "idle", Engines: map[string]float64{"video": 30}, PowerW: 9.5, PowerCapW: 55, MHz: 2450, MaxMHz: 2450, Clients: []GPUClient{{PID: 1}}}}}
|
||||
got := map[string]any{}
|
||||
for _, x := range flatten(s, nil) {
|
||||
got[x.ID] = x.Value
|
||||
}
|
||||
for id, want := range map[string]any{"gpu.intel_arc_a380.usage": 12.0, "gpu.intel_arc_a380.video": 30.0, "gpu.intel_arc_a380.power": 9.5, "gpu.intel_arc_a380.freq": 2450.0, "gpu.intel_arc_a380.clients": 1.0} {
|
||||
if got[id] != want {
|
||||
t.Errorf("%s = %v, want %v", id, got[id], want)
|
||||
}
|
||||
}
|
||||
if _, ok := got["gpu.intel_arc_a380.temp"]; ok {
|
||||
t.Error("no temperature sensor, no temperature")
|
||||
}
|
||||
}
|
||||
|
||||
func replace(s, old, new string) string {
|
||||
for i := 0; i+len(old) <= len(s); i++ {
|
||||
if s[i:i+len(old)] == old {
|
||||
return s[:i] + new + s[i+len(old):]
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// TestLiveGPU reads this machine's GPUs for real: HAMLAB_LIVE=1 go test -run TestLiveGPU -v ./internal/metrics
|
||||
func TestLiveGPU(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_LIVE") == "" {
|
||||
t.Skip("set HAMLAB_LIVE=1 to read the real hardware")
|
||||
}
|
||||
c := NewCollector()
|
||||
c.sample()
|
||||
time.Sleep(1100 * time.Millisecond)
|
||||
c.sample()
|
||||
s, _ := c.Latest()
|
||||
b, _ := json.MarshalIndent(s.GPUs, "", " ")
|
||||
t.Logf("%s", b)
|
||||
for _, x := range flatten(s, func(id string) bool { return len(id) > 4 && id[:4] == "gpu." }) {
|
||||
t.Logf("%s = %v %s", x.ID, x.Value, x.Unit)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLiveARC: HAMLAB_LIVE=1 go test -run TestLiveARC -v ./internal/metrics
|
||||
func TestLiveARC(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_LIVE") == "" {
|
||||
t.Skip("set HAMLAB_LIVE=1 to read the real machine")
|
||||
}
|
||||
c := NewCollector()
|
||||
c.sample()
|
||||
time.Sleep(1100 * time.Millisecond)
|
||||
c.sample()
|
||||
s, _ := c.Latest()
|
||||
b, _ := json.MarshalIndent(s.ARC, "", " ")
|
||||
t.Logf("%s", b)
|
||||
for _, x := range flatten(s, func(id string) bool { return strings.HasPrefix(id, "zfs.arc.") }) {
|
||||
t.Logf("%s = %v %s", x.ID, x.Value, x.Unit)
|
||||
}
|
||||
}
|
||||
|
||||
// The embedded table names the lab's four GPUs the way lspci would, so a machine without
|
||||
// pciutils (the router) agrees with one that has it.
|
||||
func TestPCIName(t *testing.T) {
|
||||
for id, want := range map[string]string{"8086:3185": "Intel UHD Graphics 600", "8086:56a5": "Intel Arc A380", "8086:46d1": "Intel UHD Graphics", "8086:4c8a": "Intel UHD Graphics 750", "10de:2684": "NVIDIA GeForce RTX 4090"} {
|
||||
v, d := pciName(id[:4], id[5:])
|
||||
if d == "" {
|
||||
t.Errorf("%s: not in the table", id)
|
||||
continue
|
||||
}
|
||||
if got := lspciName(v + " " + d); got != want {
|
||||
t.Errorf("%s → %q, want %q", id, got, want)
|
||||
}
|
||||
}
|
||||
if _, d := pciName("8086", "ffff"); d != "" {
|
||||
t.Errorf("an unknown device should be empty, got %q", d)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,47 @@
|
||||
package metrics
|
||||
|
||||
// A trimmed PCI ID database, embedded, so a GPU is named the same on every machine —
|
||||
// "Intel UHD Graphics 600" — whether or not that machine has pciutils. OpenWrt has neither
|
||||
// lspci nor pci.ids, and the router's GPU came up as "Intel GPU 3185" beside its twin's
|
||||
// "Intel UHD Graphics 600" until this. scripts/pci-ids-gpu.sh regenerates the file.
|
||||
|
||||
import (
|
||||
_ "embed"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
//go:embed pciids_gpu.txt
|
||||
var pciIDsGPU string
|
||||
|
||||
var pciTable struct {
|
||||
once sync.Once
|
||||
vendors map[string]string // "8086" → "Intel Corporation"
|
||||
devices map[string]string // "8086:3185" → "GeminiLake [UHD Graphics 600]"
|
||||
}
|
||||
|
||||
// pciName looks a vendor and device ID up (hex, lower case, no 0x): the vendor's name and
|
||||
// the device's; empty where unknown.
|
||||
func pciName(vendor, device string) (string, string) {
|
||||
pciTable.once.Do(func() {
|
||||
pciTable.vendors, pciTable.devices = map[string]string{}, map[string]string{}
|
||||
cur := ""
|
||||
for _, line := range strings.Split(pciIDsGPU, "\n") {
|
||||
switch {
|
||||
case line == "" || line[0] == '#':
|
||||
case line[0] == '\t':
|
||||
if cur == "" || len(line) < 7 || line[1] == '\t' {
|
||||
continue // a subsystem, or a device before any vendor
|
||||
}
|
||||
pciTable.devices[cur+":"+line[1:5]] = strings.TrimSpace(line[5:])
|
||||
default:
|
||||
if len(line) >= 6 {
|
||||
cur = line[:4]
|
||||
pciTable.vendors[cur] = strings.TrimSpace(line[4:])
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
vendor, device = strings.ToLower(vendor), strings.ToLower(device)
|
||||
return pciTable.vendors[vendor], pciTable.devices[vendor+":"+device]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,259 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
|
||||
"hamlab/internal/sensors"
|
||||
)
|
||||
|
||||
// The collector as a sensors.Provider: every number in a Sample, one sensor each. The
|
||||
// IDs are built from names the hardware reports (chip, label, interface, device, mount),
|
||||
// never from enumeration order like hwmon3, which moves between boots.
|
||||
|
||||
// flattener turns one Sample into sensors. With want set it builds only those, so walking
|
||||
// the whole ring for a history costs string compares, not allocations.
|
||||
type flattener struct {
|
||||
want func(id string) bool
|
||||
seen map[string]int
|
||||
out []sensors.Sensor
|
||||
}
|
||||
|
||||
type bounds struct{ min, max, warn, crit *float64 }
|
||||
|
||||
var pct = bounds{min: sensors.F(0), max: sensors.F(100)}
|
||||
|
||||
func upTo(max float64) bounds { return bounds{min: sensors.F(0), max: sensors.F(max)} }
|
||||
|
||||
func (f *flattener) add(id, name, kind, unit string, v any, t int64, b bounds, labels ...string) {
|
||||
f.seen[id]++
|
||||
if n := f.seen[id]; n > 1 { // two identical chips: the second is coretemp_2
|
||||
id = fmt.Sprintf("%s_%d", id, n)
|
||||
}
|
||||
if f.want != nil && !f.want(id) {
|
||||
return
|
||||
}
|
||||
s := sensors.Sensor{ID: id, Name: name, Kind: kind, Unit: unit, Value: v, T: t, Min: b.min, Max: b.max, Warn: b.warn, Crit: b.crit}
|
||||
if len(labels) > 0 {
|
||||
s.Labels = make(map[string]string, len(labels)/2)
|
||||
for i := 0; i+1 < len(labels); i += 2 {
|
||||
s.Labels[labels[i]] = labels[i+1]
|
||||
}
|
||||
}
|
||||
f.out = append(f.out, s)
|
||||
}
|
||||
|
||||
func ratio(part, whole uint64) float64 {
|
||||
if whole == 0 {
|
||||
return 0
|
||||
}
|
||||
return round1(float64(part) / float64(whole) * 100)
|
||||
}
|
||||
|
||||
func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
|
||||
f := &flattener{want: want, seen: map[string]int{}}
|
||||
t, none := s.T, bounds{}
|
||||
|
||||
f.add("cpu.usage", "CPU", "usage", "%", s.CPU, t, pct)
|
||||
for i, c := range s.Cores {
|
||||
n := strconv.Itoa(i)
|
||||
f.add("cpu.core."+n+".usage", "Core "+n, "usage", "%", c, t, pct, "core", n)
|
||||
}
|
||||
if c := cpuTemp(s.Temps); c > 0 {
|
||||
f.add("cpu.temp", "CPU temperature", "temperature", "°C", c, t, none)
|
||||
}
|
||||
if s.CPUPower > 0 {
|
||||
f.add("cpu.power", "CPU power", "power", "W", s.CPUPower, t, none)
|
||||
}
|
||||
for i, w := range []string{"1", "5", "15"} {
|
||||
f.add("load."+w, "Load "+w+" min", "load", "", s.Load[i], t, none)
|
||||
}
|
||||
f.add("uptime", "Uptime", "duration", "s", s.Uptime, t, none)
|
||||
f.add("hamlab.sample_interval", "Sampling interval", "duration", "ms", float64(s.Interval), t, none)
|
||||
|
||||
for _, m := range []struct {
|
||||
id, name string
|
||||
v Mem
|
||||
}{{"mem", "Memory", s.Mem}, {"swap", "Swap", s.Swap}} {
|
||||
f.add(m.id+".total", m.name+" total", "bytes", "B", float64(m.v.Total), t, none)
|
||||
f.add(m.id+".used", m.name+" used", "bytes", "B", float64(m.v.Used), t, upTo(float64(m.v.Total)))
|
||||
f.add(m.id+".avail", m.name+" available", "bytes", "B", float64(m.v.Avail), t, upTo(float64(m.v.Total)))
|
||||
f.add(m.id+".used_pct", m.name+" used", "usage", "%", ratio(m.v.Used, m.v.Total), t, pct)
|
||||
}
|
||||
f.add("mem.free", "Memory free", "bytes", "B", float64(s.Mem.Free), t, upTo(float64(s.Mem.Total)))
|
||||
f.add("mem.cached", "Memory cached", "bytes", "B", float64(s.Mem.Cached), t, upTo(float64(s.Mem.Total)))
|
||||
f.add("mem.buffers", "Memory buffers", "bytes", "B", float64(s.Mem.Buffers), t, upTo(float64(s.Mem.Total)))
|
||||
|
||||
for _, x := range s.Temps {
|
||||
b := bounds{}
|
||||
if x.Max > 0 {
|
||||
b.warn = sensors.F(x.Max)
|
||||
}
|
||||
if x.Crit > 0 {
|
||||
b.crit = sensors.F(x.Crit)
|
||||
}
|
||||
f.add(sensors.ID("temp", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "temperature", "°C", x.C, t, b, "chip", x.Chip, "label", x.Label)
|
||||
}
|
||||
for _, x := range s.Fans {
|
||||
f.add(sensors.ID("fan", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "fan", "rpm", float64(x.RPM), t, none, "chip", x.Chip, "label", x.Label)
|
||||
}
|
||||
|
||||
for _, x := range s.Net {
|
||||
p := "net." + sensors.Seg(x.Iface)
|
||||
f.add(p+".rx", x.Iface+" receive", "rate", "B/s", x.RxBps, t, none, "iface", x.Iface)
|
||||
f.add(p+".tx", x.Iface+" transmit", "rate", "B/s", x.TxBps, t, none, "iface", x.Iface)
|
||||
f.add(p+".rx_total", x.Iface+" received", "bytes", "B", float64(x.RxTotal), t, none, "iface", x.Iface)
|
||||
f.add(p+".tx_total", x.Iface+" transmitted", "bytes", "B", float64(x.TxTotal), t, none, "iface", x.Iface)
|
||||
f.add(p+".up", x.Iface+" link", "state", "", x.Up, t, none, "iface", x.Iface)
|
||||
}
|
||||
|
||||
for _, x := range s.Disks {
|
||||
p := "disk." + sensors.Seg(x.Name)
|
||||
f.add(p+".read", x.Name+" read", "rate", "B/s", x.ReadBps, t, none, "device", x.Name, "model", x.Model)
|
||||
f.add(p+".write", x.Name+" write", "rate", "B/s", x.WriteBps, t, none, "device", x.Name, "model", x.Model)
|
||||
f.add(p+".util", x.Name+" busy", "usage", "%", x.UtilPct, t, pct, "device", x.Name, "model", x.Model)
|
||||
f.add(p+".size", x.Name+" size", "bytes", "B", float64(x.Size), t, none, "device", x.Name, "model", x.Model)
|
||||
if x.Temp > 0 {
|
||||
f.add(p+".temp", x.Name+" temperature", "temperature", "°C", x.Temp, t, none, "device", x.Name, "model", x.Model)
|
||||
}
|
||||
}
|
||||
|
||||
for _, x := range s.FS {
|
||||
p := "fs." + sensors.Seg(x.Mount)
|
||||
// remote says the bytes are on another machine (a CIFS or NFS share), so a view of
|
||||
// this machine's hardware can leave it out without losing the sensor
|
||||
lbl := []string{"mount", x.Mount, "fstype", x.Type}
|
||||
if x.Remote {
|
||||
lbl = append(lbl, "remote", "1")
|
||||
}
|
||||
if x.Disk != "" {
|
||||
lbl = append(lbl, "disk", x.Disk)
|
||||
}
|
||||
f.add(p+".size", x.Mount+" size", "bytes", "B", float64(x.Size), s.fsT, none, lbl...)
|
||||
f.add(p+".used", x.Mount+" used", "bytes", "B", float64(x.Used), s.fsT, upTo(float64(x.Size)), lbl...)
|
||||
f.add(p+".used_pct", x.Mount+" used", "usage", "%", ratio(x.Used, x.Size), s.fsT, pct, lbl...)
|
||||
}
|
||||
|
||||
for _, g := range s.GPUs {
|
||||
p := "gpu." + sensors.Seg(g.Name)
|
||||
lbl := []string{"name", g.Name, "driver", g.Driver}
|
||||
if g.Bus != "" {
|
||||
lbl = append(lbl, "bus", g.Bus)
|
||||
}
|
||||
if g.Integrated {
|
||||
lbl = append(lbl, "integrated", "1")
|
||||
}
|
||||
if g.UsageSrc != "" {
|
||||
f.add(p+".usage", g.Name, "usage", "%", g.Usage, t, pct, lbl...)
|
||||
}
|
||||
for _, e := range []string{"render", "video", "copy", "video-enhance", "compute"} {
|
||||
if v, ok := g.Engines[e]; ok {
|
||||
f.add(p+"."+sensors.Seg(e), g.Name+" "+e, "usage", "%", v, t, pct, lbl...)
|
||||
}
|
||||
}
|
||||
if g.Temp > 0 {
|
||||
f.add(p+".temp", g.Name+" temperature", "temperature", "°C", g.Temp, t, none, append(lbl, "source", g.TempSrc)...)
|
||||
}
|
||||
if g.PowerW > 0 || g.PowerCapW > 0 {
|
||||
b := none
|
||||
if g.PowerCapW > 0 {
|
||||
b = upTo(g.PowerCapW)
|
||||
}
|
||||
f.add(p+".power", g.Name+" power", "power", "W", g.PowerW, t, b, lbl...)
|
||||
}
|
||||
if g.FanRPM > 0 {
|
||||
f.add(p+".fan", g.Name+" fan", "fan", "rpm", float64(g.FanRPM), t, none, lbl...)
|
||||
}
|
||||
if g.MaxMHz > 0 {
|
||||
f.add(p+".freq", g.Name+" clock", "frequency", "MHz", g.MHz, t, upTo(g.MaxMHz), lbl...)
|
||||
}
|
||||
if g.MemTotal > 0 {
|
||||
f.add(p+".mem_used", g.Name+" memory used", "bytes", "B", float64(g.MemUsed), t, upTo(float64(g.MemTotal)), lbl...)
|
||||
f.add(p+".mem_total", g.Name+" memory", "bytes", "B", float64(g.MemTotal), t, none, lbl...)
|
||||
} else if g.MemUsed > 0 {
|
||||
f.add(p+".mem_used", g.Name+" memory used", "bytes", "B", float64(g.MemUsed), t, none, lbl...)
|
||||
}
|
||||
f.add(p+".clients", g.Name+" clients", "count", "", float64(len(g.Clients)), t, none, lbl...)
|
||||
}
|
||||
|
||||
if a := s.ARC; a != nil {
|
||||
p := "zfs.arc."
|
||||
cap := upTo(float64(a.Max))
|
||||
f.add(p+"size", "ARC size", "bytes", "B", float64(a.Size), t, cap)
|
||||
f.add(p+"target", "ARC target", "bytes", "B", float64(a.Target), t, cap)
|
||||
f.add(p+"min", "ARC floor", "bytes", "B", float64(a.Min), t, none)
|
||||
f.add(p+"max", "ARC ceiling", "bytes", "B", float64(a.Max), t, none)
|
||||
f.add(p+"mru", "ARC recently used", "bytes", "B", float64(a.MRU), t, cap)
|
||||
f.add(p+"mfu", "ARC frequently used", "bytes", "B", float64(a.MFU), t, cap)
|
||||
f.add(p+"metadata", "ARC metadata", "bytes", "B", float64(a.Metadata), t, cap)
|
||||
f.add(p+"data", "ARC data", "bytes", "B", float64(a.Data), t, cap)
|
||||
if a.HitPctKnown {
|
||||
f.add(p+"hit_pct", "ARC hit rate", "usage", "%", a.HitPct, t, pct)
|
||||
}
|
||||
f.add(p+"reads", "ARC reads", "rate", "/s", a.ReadsPerSec, t, none)
|
||||
f.add(p+"misses", "ARC misses", "rate", "/s", a.MissesPerSec, t, none)
|
||||
f.add(p+"lifetime_hit_pct", "ARC hit rate since boot", "usage", "%", a.LifetimeHitPct, t, pct)
|
||||
f.add(p+"no_grow", "ARC held by memory pressure", "state", "", a.NoGrow, t, none)
|
||||
if a.Compressed > 0 {
|
||||
f.add(p+"compress_ratio", "ARC compression", "ratio", "x", round1(float64(a.Uncompressed)/float64(a.Compressed)*10)/10, t, none)
|
||||
}
|
||||
if a.L2Size > 0 {
|
||||
f.add(p+"l2_size", "L2ARC size", "bytes", "B", float64(a.L2Size), t, none)
|
||||
f.add(p+"l2_hit_pct", "L2ARC hit rate", "usage", "%", a.L2HitPct, t, pct)
|
||||
}
|
||||
}
|
||||
|
||||
if u := s.UPS; u != nil {
|
||||
ut := s.upsT
|
||||
f.add("ups.battery", "UPS battery", "usage", "%", u.BatteryPct, ut, pct, "model", u.Model)
|
||||
f.add("ups.runtime", "UPS runtime", "duration", "min", u.RuntimeMin, ut, none, "model", u.Model)
|
||||
f.add("ups.load", "UPS load", "power", "W", u.LoadW, ut, none, "model", u.Model)
|
||||
f.add("ups.load_pct", "UPS load", "usage", "%", u.LoadPct, ut, pct, "model", u.Model)
|
||||
f.add("ups.in_v", "UPS input", "voltage", "V", u.InV, ut, none, "model", u.Model)
|
||||
f.add("ups.out_v", "UPS output", "voltage", "V", u.OutV, ut, none, "model", u.Model)
|
||||
f.add("ups.on_battery", "UPS on battery", "state", "", u.OnBattery, ut, none, "model", u.Model)
|
||||
f.add("ups.state", "UPS state", "state", "", u.State, ut, none, "model", u.Model)
|
||||
f.add("ups.source", "UPS power source", "state", "", u.Source, ut, none, "model", u.Model)
|
||||
f.add("ups.last_event", "UPS last event", "state", "", u.LastEvent, ut, none, "model", u.Model, "driver", u.Driver)
|
||||
}
|
||||
return f.out
|
||||
}
|
||||
|
||||
// Sensors implements sensors.Provider.
|
||||
func (c *Collector) Sensors() []sensors.Sensor {
|
||||
s, ok := c.Latest()
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
return flatten(s, nil)
|
||||
}
|
||||
|
||||
// SensorHistories implements sensors.Historian: the ring, seen through the sensors that
|
||||
// match. A sensor that didn't exist in a sample (a disk plugged in since) simply has a
|
||||
// shorter series; nothing is zero-filled. States have no curve and are left out.
|
||||
func (c *Collector) SensorHistories(match func(id string) bool) []sensors.Series {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
by := map[string]*sensors.Series{}
|
||||
var order []string
|
||||
for _, smp := range c.ring {
|
||||
for _, x := range flatten(smp, match) {
|
||||
v, ok := x.Value.(float64)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
s := by[x.ID]
|
||||
if s == nil {
|
||||
s = &sensors.Series{ID: x.ID, Unit: x.Unit, T: []int64{}, V: []float64{}}
|
||||
by[x.ID] = s
|
||||
order = append(order, x.ID)
|
||||
}
|
||||
s.T, s.V = append(s.T, smp.T), append(s.V, v)
|
||||
}
|
||||
}
|
||||
out := make([]sensors.Series, 0, len(order))
|
||||
for _, id := range order {
|
||||
out = append(out, *by[id])
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package metrics
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestFlatten(t *testing.T) {
|
||||
s := Sample{T: 1000, CPU: 10, Cores: []float64{5, 15}, Mem: Mem{Total: 200, Used: 50},
|
||||
Temps: []Temp{{Chip: "nvme", Label: "Composite", C: 40, Crit: 85}, {Chip: "nvme", Label: "Composite", C: 45}},
|
||||
Fans: []Fan{{Chip: "nct6775", Label: "fan2", RPM: 1100}},
|
||||
FS: []FS{{Mount: "/", Size: 100, Used: 25}}, fsT: 900}
|
||||
got := map[string]any{}
|
||||
for _, x := range flatten(s, nil) {
|
||||
if _, dup := got[x.ID]; dup {
|
||||
t.Errorf("duplicate id %s", x.ID)
|
||||
}
|
||||
got[x.ID] = x.Value
|
||||
}
|
||||
for id, want := range map[string]any{"cpu.usage": 10.0, "cpu.core.1.usage": 15.0, "mem.used_pct": 25.0, "temp.nvme.composite": 40.0,
|
||||
"temp.nvme.composite_2": 45.0, "fan.nct6775.fan2": 1100.0, "fs.root.used_pct": 25.0} {
|
||||
if got[id] != want {
|
||||
t.Errorf("%s = %v, want %v", id, got[id], want)
|
||||
}
|
||||
}
|
||||
one := flatten(s, func(id string) bool { return id == "temp.nvme.composite_2" })
|
||||
if len(one) != 1 || one[0].Value != 45.0 {
|
||||
t.Errorf("only: got %+v", one)
|
||||
}
|
||||
for _, x := range flatten(s, func(id string) bool { return id == "fs.root.used" }) {
|
||||
if x.T != 900 {
|
||||
t.Errorf("fs sensor should carry the fs read time, got %d", x.T)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSensorHistories(t *testing.T) {
|
||||
c := NewCollector()
|
||||
c.ring = []Sample{{T: 1, CPU: 10}, {T: 2, CPU: 20, Net: []Net{{Iface: "eth0", Up: true, RxBps: 5}}}}
|
||||
got := map[string][]float64{}
|
||||
for _, h := range c.SensorHistories(func(id string) bool { return id == "cpu.usage" || id == "net.eth0.rx" || id == "net.eth0.up" }) {
|
||||
if len(h.T) != len(h.V) {
|
||||
t.Errorf("%s: t and v differ in length", h.ID)
|
||||
}
|
||||
got[h.ID] = h.V
|
||||
}
|
||||
if len(got["cpu.usage"]) != 2 || got["cpu.usage"][1] != 20 {
|
||||
t.Errorf("cpu: %v", got["cpu.usage"])
|
||||
}
|
||||
if len(got["net.eth0.rx"]) != 1 {
|
||||
t.Errorf("a sensor that appeared later has a shorter history, got %v", got["net.eth0.rx"])
|
||||
}
|
||||
if _, ok := got["net.eth0.up"]; ok {
|
||||
t.Error("a state has no history")
|
||||
}
|
||||
if len(c.SensorHistories(func(string) bool { return false })) != 0 {
|
||||
t.Error("nothing matched, nothing returned")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,360 @@
|
||||
// Package network is what the Network tab shows: the machine's interfaces as the kernel has
|
||||
// them, how the internet looks from out there, and a speed test that measures the line.
|
||||
//
|
||||
// Everything here is read from /proc and /sys, or measured against Cloudflare's public speed
|
||||
// endpoints — no tool to install, on any of the machines. That matters for the router, where
|
||||
// flash is small and `opkg install` is a decision.
|
||||
package network
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Addr is one address on an interface.
|
||||
type Addr struct {
|
||||
IP string `json:"ip"`
|
||||
Prefix int `json:"prefix"` // network bits, as in /24
|
||||
V6 bool `json:"v6,omitempty"`
|
||||
}
|
||||
|
||||
// Iface is one network interface: what it is, whether it is up, and what it has moved.
|
||||
type Iface struct {
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"` // ethernet | wifi | bridge | tunnel | loopback | virtual
|
||||
MAC string `json:"mac,omitempty"`
|
||||
MTU int `json:"mtu"`
|
||||
// Up is the administrative state, Carrier whether a cable (or an association) is actually
|
||||
// there. Up without Carrier is the usual "it's plugged into nothing".
|
||||
Up bool `json:"up"`
|
||||
Carrier bool `json:"carrier"`
|
||||
// SpeedMbps is what the link negotiated, where the driver says: 0 for wireless, virtual
|
||||
// and tunnel devices, which have no such number.
|
||||
SpeedMbps int `json:"speed_mbps,omitempty"`
|
||||
Duplex string `json:"duplex,omitempty"`
|
||||
Driver string `json:"driver,omitempty"`
|
||||
SignalDbm *float64 `json:"signal_dbm,omitempty"` // wireless only, from /proc/net/wireless
|
||||
Addrs []Addr `json:"addrs"`
|
||||
RxTotal uint64 `json:"rx_total"`
|
||||
TxTotal uint64 `json:"tx_total"`
|
||||
// Default marks the interface the default route leaves by — the one that matters when
|
||||
// the question is about the internet.
|
||||
Default bool `json:"default,omitempty"`
|
||||
}
|
||||
|
||||
// Report is the machine's network as it stands.
|
||||
type Report struct {
|
||||
At time.Time `json:"at"`
|
||||
Host string `json:"host,omitempty"`
|
||||
Ifaces []Iface `json:"ifaces"`
|
||||
Gateway string `json:"gateway,omitempty"`
|
||||
GatewayVia string `json:"gateway_via,omitempty"` // the interface it is reached through
|
||||
DNS []string `json:"dns,omitempty"`
|
||||
Search []string `json:"search,omitempty"`
|
||||
}
|
||||
|
||||
func readStr(path string) string {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
func readInt(path string) int {
|
||||
n, err := strconv.Atoi(readStr(path))
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// kindOf names what sort of device this is, from what the kernel exposes about it rather
|
||||
// than from its name — a bridge is a bridge whatever it is called.
|
||||
func kindOf(name string, flags net.Flags) string {
|
||||
sys := "/sys/class/net/" + name + "/"
|
||||
switch {
|
||||
case flags&net.FlagLoopback != 0:
|
||||
return "loopback"
|
||||
case dirExists(sys + "wireless"), dirExists(sys + "phy80211"):
|
||||
return "wifi"
|
||||
case dirExists(sys + "bridge"):
|
||||
return "bridge"
|
||||
case dirExists(sys + "bonding"):
|
||||
return "bond"
|
||||
}
|
||||
// DEVTYPE says vlan, bridge, wlan… for the devices that have one
|
||||
for _, line := range strings.Split(readStr(sys+"uevent"), "\n") {
|
||||
if v, ok := strings.CutPrefix(line, "DEVTYPE="); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
if !dirExists(sys + "device") { // no bus device behind it: veth, tun, wireguard, docker's own
|
||||
return "virtual"
|
||||
}
|
||||
return "ethernet"
|
||||
}
|
||||
|
||||
func dirExists(p string) bool {
|
||||
st, err := os.Stat(p)
|
||||
return err == nil && st.IsDir()
|
||||
}
|
||||
|
||||
// driverOf is the module behind an interface, which is what names the hardware in one word.
|
||||
func driverOf(name string) string {
|
||||
l, err := os.Readlink("/sys/class/net/" + name + "/device/driver")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return l[strings.LastIndexByte(l, '/')+1:]
|
||||
}
|
||||
|
||||
// wirelessSignal is the signal level of each wireless interface, in dBm.
|
||||
func wirelessSignal() map[string]float64 {
|
||||
out := map[string]float64{}
|
||||
f, err := os.Open("/proc/net/wireless")
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
name, rest, ok := strings.Cut(sc.Text(), ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
fs := strings.Fields(rest)
|
||||
if len(fs) < 3 {
|
||||
continue
|
||||
}
|
||||
if v, err := strconv.ParseFloat(strings.TrimSuffix(fs[2], "."), 64); err == nil {
|
||||
out[strings.TrimSpace(name)] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// counters is every interface's byte totals, from the same file the sampler reads.
|
||||
func counters() map[string][2]uint64 {
|
||||
out := map[string][2]uint64{}
|
||||
f, err := os.Open("/proc/net/dev")
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
name, rest, ok := strings.Cut(sc.Text(), ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
fs := strings.Fields(rest)
|
||||
if len(fs) < 16 {
|
||||
continue
|
||||
}
|
||||
rx, _ := strconv.ParseUint(fs[0], 10, 64)
|
||||
tx, _ := strconv.ParseUint(fs[8], 10, 64)
|
||||
out[strings.TrimSpace(name)] = [2]uint64{rx, tx}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// defaultRoute is the gateway the machine sends the internet to, and the interface it
|
||||
// leaves by. Read from /proc/net/route, whose addresses are little-endian hex.
|
||||
func defaultRoute() (gw, via string) {
|
||||
f, err := os.Open("/proc/net/route")
|
||||
if err != nil {
|
||||
return "", ""
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
best := -1
|
||||
for sc.Scan() {
|
||||
fs := strings.Fields(sc.Text())
|
||||
if len(fs) < 8 || fs[0] == "Iface" {
|
||||
continue
|
||||
}
|
||||
if fs[1] != "00000000" { // not a default route
|
||||
continue
|
||||
}
|
||||
metric, _ := strconv.Atoi(fs[6])
|
||||
if best >= 0 && metric >= best {
|
||||
continue
|
||||
}
|
||||
ip, err := hexIP(fs[2])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
best, gw, via = metric, ip, fs[0]
|
||||
}
|
||||
return gw, via
|
||||
}
|
||||
|
||||
// hexIP reads one address out of /proc/net/route, where each is printed as a 32-bit word in
|
||||
// the host's byte order: 10.0.0.1 appears as "0100000A" on a little-endian machine. So the
|
||||
// text is parsed as a plain number, then written back out in native order to recover the four
|
||||
// bytes — which is right on either sort of machine, without asking which this is.
|
||||
func hexIP(h string) (string, error) {
|
||||
v, err := strconv.ParseUint(h, 16, 32)
|
||||
if err != nil || len(h) != 8 {
|
||||
return "", fmt.Errorf("not an address: %q", h)
|
||||
}
|
||||
var b [4]byte
|
||||
binary.NativeEndian.PutUint32(b[:], uint32(v))
|
||||
return net.IPv4(b[0], b[1], b[2], b[3]).String(), nil
|
||||
}
|
||||
|
||||
// resolvers is what /etc/resolv.conf points at.
|
||||
func resolvers() (servers, search []string) {
|
||||
f, err := os.Open("/etc/resolv.conf")
|
||||
if err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
fs := strings.Fields(sc.Text())
|
||||
if len(fs) < 2 || strings.HasPrefix(fs[0], "#") {
|
||||
continue
|
||||
}
|
||||
switch fs[0] {
|
||||
case "nameserver":
|
||||
servers = append(servers, fs[1])
|
||||
case "search", "domain":
|
||||
search = append(search, fs[1:]...)
|
||||
}
|
||||
}
|
||||
return servers, search
|
||||
}
|
||||
|
||||
// Interfaces is every interface the kernel has, with its addresses and totals.
|
||||
func Interfaces() []Iface {
|
||||
list, err := net.Interfaces()
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
_, via := defaultRoute()
|
||||
sig, cnt := wirelessSignal(), counters()
|
||||
out := make([]Iface, 0, len(list))
|
||||
for _, in := range list {
|
||||
sys := "/sys/class/net/" + in.Name + "/"
|
||||
f := Iface{
|
||||
Name: in.Name, Kind: kindOf(in.Name, in.Flags), MTU: in.MTU,
|
||||
MAC: in.HardwareAddr.String(), Up: in.Flags&net.FlagUp != 0,
|
||||
Driver: driverOf(in.Name), Default: in.Name == via, Addrs: []Addr{},
|
||||
}
|
||||
f.Carrier = readStr(sys+"carrier") == "1"
|
||||
if f.Kind == "loopback" {
|
||||
f.Carrier = f.Up
|
||||
}
|
||||
if sp := readInt(sys + "speed"); sp > 0 { // -1 when the driver doesn't know
|
||||
f.SpeedMbps = sp
|
||||
}
|
||||
if d := readStr(sys + "duplex"); d != "" && d != "unknown" {
|
||||
f.Duplex = d
|
||||
}
|
||||
if v, ok := sig[in.Name]; ok {
|
||||
f.SignalDbm = &v
|
||||
}
|
||||
if c, ok := cnt[in.Name]; ok {
|
||||
f.RxTotal, f.TxTotal = c[0], c[1]
|
||||
}
|
||||
addrs, _ := in.Addrs()
|
||||
for _, a := range addrs {
|
||||
n, ok := a.(*net.IPNet)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
ones, _ := n.Mask.Size()
|
||||
f.Addrs = append(f.Addrs, Addr{IP: n.IP.String(), Prefix: ones, V6: n.IP.To4() == nil})
|
||||
}
|
||||
out = append(out, f)
|
||||
}
|
||||
// the interface the internet leaves by first, then the ones carrying addresses, then the rest
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
a, b := out[i], out[j]
|
||||
if a.Default != b.Default {
|
||||
return a.Default
|
||||
}
|
||||
ai, bi := len(a.Addrs) > 0 && a.Kind != "loopback", len(b.Addrs) > 0 && b.Kind != "loopback"
|
||||
if ai != bi {
|
||||
return ai
|
||||
}
|
||||
return a.Name < b.Name
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// Look is the machine's network now. Cheap: a handful of small files.
|
||||
func Look(host string) Report {
|
||||
gw, via := defaultRoute()
|
||||
dns, search := resolvers()
|
||||
return Report{At: time.Now(), Host: host, Ifaces: Interfaces(), Gateway: gw, GatewayVia: via, DNS: dns, Search: search}
|
||||
}
|
||||
|
||||
// ---- how the internet sees this machine ----
|
||||
|
||||
// Outside is what the far end of the line says about this connection: the address the
|
||||
// internet sees, and where it came ashore. Refreshed on its own clock, because the public
|
||||
// address here is a DHCP lease and moves without warning.
|
||||
type Outside struct {
|
||||
At time.Time `json:"at"`
|
||||
IP string `json:"ip,omitempty"`
|
||||
Colo string `json:"colo,omitempty"` // the Cloudflare edge that answered — "SEA"
|
||||
Country string `json:"country,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Watcher keeps the outside view fresh and holds the speed tester.
|
||||
type Watcher struct {
|
||||
mu sync.RWMutex
|
||||
out Outside
|
||||
Test *Tester
|
||||
}
|
||||
|
||||
const outsideEvery = 15 * time.Minute
|
||||
|
||||
func NewWatcher(onResult func(Result)) *Watcher {
|
||||
w := &Watcher{Test: NewTester(onResult)}
|
||||
go func() {
|
||||
for {
|
||||
w.refresh()
|
||||
time.Sleep(outsideEvery)
|
||||
}
|
||||
}()
|
||||
return w
|
||||
}
|
||||
|
||||
func (w *Watcher) refresh() {
|
||||
o := probeOutside()
|
||||
w.mu.Lock()
|
||||
// a failed probe does not throw away a good answer; it is noted beside it
|
||||
if o.IP == "" && w.out.IP != "" {
|
||||
w.out.Error, w.out.At = o.Error, o.At
|
||||
} else {
|
||||
w.out = o
|
||||
}
|
||||
w.mu.Unlock()
|
||||
}
|
||||
|
||||
// Outside is the last known view from outside, asking again if it has gone stale.
|
||||
func (w *Watcher) Outside() Outside {
|
||||
w.mu.RLock()
|
||||
o := w.out
|
||||
w.mu.RUnlock()
|
||||
if time.Since(o.At) > outsideEvery {
|
||||
w.refresh()
|
||||
w.mu.RLock()
|
||||
o = w.out
|
||||
w.mu.RUnlock()
|
||||
}
|
||||
return o
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
package network
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestHexIP(t *testing.T) {
|
||||
// /proc/net/route prints each address as a host-order word: on a little-endian machine
|
||||
// 10.0.0.1 is written "0100000A". These cases are that machine, which is all of ours.
|
||||
for hex, want := range map[string]string{
|
||||
"0100000A": "10.0.0.1",
|
||||
"00000000": "0.0.0.0",
|
||||
"0101A8C0": "192.168.1.1",
|
||||
} {
|
||||
got, err := hexIP(hex)
|
||||
if err != nil || got != want {
|
||||
t.Errorf("hexIP(%q) = %q, %v; want %q", hex, got, err, want)
|
||||
}
|
||||
}
|
||||
if _, err := hexIP("nonsense"); err == nil {
|
||||
t.Error("hexIP accepted nonsense")
|
||||
}
|
||||
}
|
||||
|
||||
func TestServerTimeIsNotLatency(t *testing.T) {
|
||||
h := http.Header{}
|
||||
h.Add("Server-Timing", "cfSpeedEdge;dur=7, cfSpeedWorker;dur=36")
|
||||
h.Add("Server-Timing", `cfL4;desc="?proto=TCP&rtt=3505&min_rtt=2856"`)
|
||||
if got := serverMs(h); got != 43 {
|
||||
t.Errorf("serverMs = %v, want 43 (7+36, and nothing from the cfL4 description)", got)
|
||||
}
|
||||
if got := serverMs(http.Header{}); got != 0 {
|
||||
t.Errorf("serverMs with no header = %v, want 0", got)
|
||||
}
|
||||
h2 := http.Header{"Cf-Ray": {"a41f6720f966a36c-SEA"}, "Cf-Meta-Ip": {"50.47.252.221"}}
|
||||
if m := readMeta(h2); m.Colo != "SEA" || m.IP != "50.47.252.221" {
|
||||
t.Errorf("readMeta = %+v, want the edge SEA and the address", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRates(t *testing.T) {
|
||||
// a gibibyte in eight seconds is 1073.7 Mbps
|
||||
if got := mbps(1<<30, 8); math.Abs(got-1073.7) > 0.1 {
|
||||
t.Errorf("mbps(1GiB, 8s) = %v, want ~1073.7", got)
|
||||
}
|
||||
if got := best([]float64{12.5, 9.2, 40, 9.9}); got != 9.2 {
|
||||
t.Errorf("best = %v, want 9.2 (the least of them)", got)
|
||||
}
|
||||
if got := jitter([]float64{10, 10, 10}); got != 0 {
|
||||
t.Errorf("jitter of a steady line = %v, want 0", got)
|
||||
}
|
||||
if got := jitter([]float64{10}); got != 0 {
|
||||
t.Errorf("jitter of one sample = %v, want 0", got)
|
||||
}
|
||||
if got := jitter([]float64{5, 50, 5, 50}); got != 45 {
|
||||
t.Errorf("jitter of a line alternating by 45 = %v, want 45", got)
|
||||
}
|
||||
// one slow probe among steady ones moves it a little, not enormously
|
||||
if got := jitter([]float64{10, 10, 10, 60, 10, 10}); got > 25 {
|
||||
t.Errorf("jitter with a single outlier = %v, want it not to run away", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The body an upload sends is a fixed size, counts itself, and costs nothing to make.
|
||||
func TestZerosBodyCountsItself(t *testing.T) {
|
||||
var n atomic.Int64
|
||||
z := &zeros{left: 100, n: &n}
|
||||
buf := make([]byte, 30)
|
||||
total := 0
|
||||
for {
|
||||
got, err := z.Read(buf)
|
||||
total += got
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
if total != 100 || n.Load() != 100 {
|
||||
t.Errorf("read %d bytes and counted %d, want 100 and 100", total, n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
// Interfaces has to work on whatever machine this runs on, and always name a loopback.
|
||||
func TestInterfacesHasLoopback(t *testing.T) {
|
||||
ifs := Interfaces()
|
||||
if len(ifs) == 0 {
|
||||
t.Fatal("no interfaces at all")
|
||||
}
|
||||
lo := false
|
||||
for _, f := range ifs {
|
||||
if f.Kind == "loopback" {
|
||||
lo = true
|
||||
if !f.Up || len(f.Addrs) == 0 {
|
||||
t.Errorf("loopback %s looks wrong: up=%v addrs=%v", f.Name, f.Up, f.Addrs)
|
||||
}
|
||||
}
|
||||
if f.Name == "" {
|
||||
t.Error("an interface with no name")
|
||||
}
|
||||
}
|
||||
if !lo {
|
||||
t.Error("no loopback among the interfaces")
|
||||
}
|
||||
}
|
||||
|
||||
// The real thing, against the real endpoint: skipped by default, because `make test` must
|
||||
// not need the internet. HAMLAB_NET_TEST=1 go test ./internal/network/ -run Live -v
|
||||
func TestLiveSpeedTest(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_NET_TEST") == "" {
|
||||
t.Skip("set HAMLAB_NET_TEST=1 to measure the actual line")
|
||||
}
|
||||
tr := NewTester(nil)
|
||||
if _, err := tr.Start(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tr.Start(); err == nil {
|
||||
t.Error("a second test started while the first was running")
|
||||
}
|
||||
deadline := time.Now().Add(90 * time.Second)
|
||||
for tr.Running() && time.Now().Before(deadline) {
|
||||
time.Sleep(250 * time.Millisecond)
|
||||
}
|
||||
st := tr.State()
|
||||
if st.Result == nil {
|
||||
t.Fatal("no result")
|
||||
}
|
||||
r := *st.Result
|
||||
t.Logf("down %s up %s latency %.1f ms jitter %.1f ms via %s ip %s %.0f MB moved in %.1fs",
|
||||
Mbits(r.DownMbps), Mbits(r.UpMbps), r.LatencyMs, r.JitterMs, r.Colo, r.IP,
|
||||
float64(r.DownBytes+r.UpBytes)/1e6, r.TookSec)
|
||||
if r.Error != "" {
|
||||
t.Fatalf("test failed: %s", r.Error)
|
||||
}
|
||||
if r.DownMbps <= 0 || r.UpMbps <= 0 || r.LatencyMs <= 0 {
|
||||
t.Errorf("a figure came back zero: %+v", r)
|
||||
}
|
||||
if r.IP == "" {
|
||||
t.Error("no public address reported")
|
||||
}
|
||||
if len(tr.History()) != 1 {
|
||||
t.Errorf("history has %d results, want 1", len(tr.History()))
|
||||
}
|
||||
}
|
||||
|
||||
// A phase measured against a local server, at a rate the test controls. This is the part of
|
||||
// the speed test with arithmetic in it — the window after the ramp, the byte cap that ends it
|
||||
// early, and the rate that comes out — and none of it needs the internet to check.
|
||||
func TestPhaseMeasuresTheWindow(t *testing.T) {
|
||||
const perTick = 1 << 20 // a mebibyte every 5 ms ≈ 1678 Mbps, whatever the machine is doing
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/up" {
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
w.WriteHeader(204)
|
||||
return
|
||||
}
|
||||
buf := make([]byte, perTick)
|
||||
fl, _ := w.(http.Flusher)
|
||||
for {
|
||||
if _, err := w.Write(buf); err != nil {
|
||||
return
|
||||
}
|
||||
if fl != nil {
|
||||
fl.Flush()
|
||||
}
|
||||
select {
|
||||
case <-time.After(5 * time.Millisecond):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
defer func(up string, w, d, mw time.Duration, m int64) {
|
||||
upURL, warmUp, downFor, minWindow, measureBytes = up, w, d, mw, m
|
||||
}(upURL, warmUp, downFor, minWindow, measureBytes)
|
||||
upURL = srv.URL + "/up"
|
||||
warmUp, downFor, minWindow, measureBytes = 100*time.Millisecond, 3*time.Second, 200*time.Millisecond, 8<<20
|
||||
here := Source{"local", func(int) string { return srv.URL + "/down" }}
|
||||
|
||||
tr := NewTester(nil)
|
||||
began := time.Now()
|
||||
mbps, moved, note, err := tr.phase(context.Background(), srv.Client(), "download", 2, downFor, 0, 1, here)
|
||||
took := time.Since(began)
|
||||
if err != nil {
|
||||
t.Fatalf("phase: %v", err)
|
||||
}
|
||||
if note != "" {
|
||||
t.Errorf("note = %q, want none — nothing went wrong", note)
|
||||
}
|
||||
// The cap must have ended it early rather than the clock: the phase was allowed three
|
||||
// seconds and should be done in well under one, having moved at least the cap.
|
||||
if moved < measureBytes {
|
||||
t.Errorf("moved %d bytes, want at least the %d-byte cap", moved, measureBytes)
|
||||
}
|
||||
if took > time.Second {
|
||||
t.Errorf("phase took %v of the %v it was allowed; the byte cap should have ended it", took.Round(time.Millisecond), downFor)
|
||||
}
|
||||
// The server hands out a mebibyte per 5 ms per stream, so the rate is in the hundreds of
|
||||
// Mbps. The point is that it is a real quotient of what moved over how long, not that it
|
||||
// hits a particular figure on a loaded test machine.
|
||||
if mbps < 50 || mbps > 20000 {
|
||||
t.Errorf("mbps = %v, which is not a believable quotient", mbps)
|
||||
}
|
||||
t.Logf("measured %.0f Mbps over %.1f MiB", mbps, float64(moved)/(1<<20))
|
||||
|
||||
// A phase whose server refuses outright fails, and says what it was told.
|
||||
bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(429) }))
|
||||
defer bad.Close()
|
||||
refuser := Source{"refuser", func(int) string { return bad.URL + "/down" }}
|
||||
_, _, _, err = tr.phase(context.Background(), bad.Client(), "download", 2, 300*time.Millisecond, 0, 1, refuser)
|
||||
if err == nil {
|
||||
t.Error("a phase against a server that refuses everything should fail")
|
||||
} else if !refused(err) {
|
||||
t.Errorf("error = %q, want the refusal that makes the fallback kick in", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Stopping a test mid-flight is reported as cancelled, not as whatever the streams said on
|
||||
// their way out.
|
||||
func TestCancelIsReportedAsCancelled(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
buf := make([]byte, 1<<16)
|
||||
for {
|
||||
if _, err := w.Write(buf); err != nil {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-time.After(time.Millisecond):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
defer func(w, d time.Duration) { warmUp, downFor = w, d }(warmUp, downFor)
|
||||
warmUp, downFor = 10*time.Millisecond, 10*time.Second
|
||||
here := Source{"local", func(int) string { return srv.URL + "/down" }}
|
||||
|
||||
tr := NewTester(nil)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
go func() { time.Sleep(200 * time.Millisecond); cancel() }()
|
||||
_, _, _, err := tr.phase(ctx, srv.Client(), "download", 1, downFor, 0, 1, here)
|
||||
// Cancelled after bytes moved: the figure stands, and the phase does not claim an error.
|
||||
if err != nil && !errors.Is(err, context.Canceled) {
|
||||
t.Errorf("err = %v, want nil or a cancellation", err)
|
||||
}
|
||||
}
|
||||
|
||||
// When the first source refuses, the download is taken from the next one and the result says
|
||||
// so. This is the whole reason there is more than one.
|
||||
func TestDownloadFallsBackWhenRefused(t *testing.T) {
|
||||
serving := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/up" {
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
w.WriteHeader(204)
|
||||
return
|
||||
}
|
||||
// paced, so the phase is governed by its window and not by a loopback moving a
|
||||
// gibibyte before the first tick
|
||||
buf := make([]byte, 1<<20)
|
||||
fl, _ := w.(http.Flusher)
|
||||
for i := 0; i < 64; i++ {
|
||||
if _, err := w.Write(buf); err != nil {
|
||||
return
|
||||
}
|
||||
if fl != nil {
|
||||
fl.Flush()
|
||||
}
|
||||
select {
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer serving.Close()
|
||||
refusing := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(429) }))
|
||||
defer refusing.Close()
|
||||
|
||||
defer func(srcs []Source, up string, w, d, mw time.Duration, m int64) {
|
||||
downSources, upURL, warmUp, downFor, minWindow, measureBytes = srcs, up, w, d, mw, m
|
||||
}(downSources, upURL, warmUp, downFor, minWindow, measureBytes)
|
||||
downSources = []Source{
|
||||
{"refuser", func(int) string { return refusing.URL + "/down" }},
|
||||
{"the-other-one", func(int) string { return serving.URL + "/down" }},
|
||||
}
|
||||
upURL = serving.URL + "/up"
|
||||
warmUp, downFor, upFor, minWindow, measureBytes = 50*time.Millisecond, time.Second, 300*time.Millisecond, 100*time.Millisecond, 4<<20
|
||||
|
||||
tr := NewTester(nil)
|
||||
if _, err := tr.Start(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
for tr.Running() && time.Now().Before(deadline) {
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
res := tr.State().Result
|
||||
if res == nil {
|
||||
t.Fatal("no result")
|
||||
}
|
||||
if res.Source != "the-other-one" {
|
||||
t.Errorf("source = %q, want the second one — the first refused", res.Source)
|
||||
}
|
||||
if res.DownMbps <= 0 {
|
||||
t.Errorf("download = %v, want a figure from the source that did serve: %+v", res.DownMbps, res)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,687 @@
|
||||
package network
|
||||
|
||||
// The speed test. It measures against Cloudflare's public speed endpoints, the same ones
|
||||
// speed.cloudflare.com uses: a plain GET of N bytes down, a plain POST of N bytes up, and a
|
||||
// zero-byte GET for the round trip. No account, no API key, nothing to install, and an edge
|
||||
// close to every one of these machines.
|
||||
//
|
||||
// The method, since a number without one means little — it is stated in the UI too:
|
||||
//
|
||||
// - Latency is the round trip of a zero-byte response on a connection that is already open,
|
||||
// less the time the far end says it spent on the request (its Server-Timing header). That
|
||||
// subtraction is the difference between measuring the line and measuring Cloudflare's
|
||||
// thinking time: a zero-byte response is almost entirely the latter — tens of
|
||||
// milliseconds of worker, against a few of actual wire. The figure reported is the best of
|
||||
// several probes; jitter is the average change from one to the next.
|
||||
// - Download and upload each run several streams at once for a fixed few seconds. One
|
||||
// stream measures one TCP connection's congestion window as much as the line, which is
|
||||
// why every speed test ever made uses more than one.
|
||||
// - The first second of each phase is thrown away. That is the ramp: counting it reports a
|
||||
// number nobody's connection ever actually did.
|
||||
// - Each phase is capped in bytes as well as seconds, so a very fast line cannot run away
|
||||
// with the meter.
|
||||
//
|
||||
// The one thing to know about the far end: it rate-limits by volume, and a test moves a few
|
||||
// hundred megabytes. Several in a row will start earning 429s, which is why there is a
|
||||
// cooldown between tests and why a phase that gets refused says so instead of reporting the
|
||||
// decayed figure it would otherwise compute.
|
||||
//
|
||||
// HTTP/2 is deliberately off: over one HTTP/2 connection the streams would share a single
|
||||
// congestion window, which measures the window, not the line.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"math"
|
||||
"net"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The endpoints, and the shape of a phase. Variables rather than constants only so the tests
|
||||
// can point them at a local server and shrink the windows; nothing else rebinds them.
|
||||
// Source is somewhere to pull bytes from. Cloudflare's endpoint is the first choice because
|
||||
// it is anycast — near wherever Ham Lab is running — and because the same host answers for
|
||||
// upload and for the trace. The fallback exists because that endpoint rate-limits by volume
|
||||
// and will refuse a connection that has been tested hard: a plain 100 MB file on a public
|
||||
// mirror has no such opinion. It is in Seattle, so on a machine far from there it will read
|
||||
// low, which is exactly why every result names the source it used.
|
||||
type Source struct {
|
||||
Name string
|
||||
// URL takes the number of bytes wanted, where the source can be asked; a fixed file
|
||||
// ignores it and is fetched again each time round.
|
||||
URL func(bytes int) string
|
||||
}
|
||||
|
||||
var downSources = []Source{
|
||||
{"cloudflare", func(b int) string { return fmt.Sprintf(cfDownURL, b) }},
|
||||
{"linode-seattle", func(int) string { return "https://speedtest.seattle.linode.com/100MB-seattle.bin" }},
|
||||
}
|
||||
|
||||
var (
|
||||
cfDownURL = "https://speed.cloudflare.com/__down?bytes=%d"
|
||||
upURL = "https://speed.cloudflare.com/__up"
|
||||
// traceURL is Cloudflare's standard trace: a few lines of key=value naming the address it
|
||||
// sees, the edge that answered and the country. A few hundred bytes.
|
||||
traceURL = "https://speed.cloudflare.com/cdn-cgi/trace"
|
||||
|
||||
warmUp = time.Second
|
||||
downFor = 4 * time.Second
|
||||
upFor = 4 * time.Second
|
||||
// measureBytes ends a phase early once this much has moved inside the window. A fast line
|
||||
// says what it is in well under the time allowed, and stopping there is what keeps a test
|
||||
// down to a couple of hundred megabytes — which is the difference between the far end
|
||||
// serving this happily and rate-limiting it. A slow line never reaches it and is timed out
|
||||
// by downFor/upFor instead, which is the honest way round.
|
||||
measureBytes = int64(64) << 20
|
||||
// minWindow is how long the window must last whatever the byte cap says. Without it a very
|
||||
// fast line fills the cap in a few hundredths of a second and the phase would be over
|
||||
// before there was anything to divide by.
|
||||
minWindow = time.Second
|
||||
)
|
||||
|
||||
const (
|
||||
latencyProbes = 8
|
||||
downStreams = 4
|
||||
upStreams = 3
|
||||
downChunk = 25 << 20 // bytes per download request; the far end refuses much more
|
||||
upChunk = 25 << 20 // and per upload request
|
||||
// phaseBudget is the runaway guard, counting the ramp too: nothing measures forever.
|
||||
phaseBudget = int64(512) << 20
|
||||
httpTimeout = 90 * time.Second
|
||||
// Cooldown is how long the line is left alone between tests. The far end rate-limits by
|
||||
// volume, and two back-to-back tests are the surest way to be refused by it.
|
||||
Cooldown = 20 * time.Second
|
||||
|
||||
// Method is what the UI tells the reader about the numbers it is showing.
|
||||
Method = "Cloudflare's speed endpoints · 4 streams down, 3 up · a 1 s ramp, then up to 4 s or 64 MiB each way"
|
||||
)
|
||||
|
||||
// Result is one finished speed test.
|
||||
type Result struct {
|
||||
At time.Time `json:"at"`
|
||||
DownMbps float64 `json:"down_mbps"`
|
||||
UpMbps float64 `json:"up_mbps"`
|
||||
LatencyMs float64 `json:"latency_ms"`
|
||||
JitterMs float64 `json:"jitter_ms"`
|
||||
DownBytes int64 `json:"down_bytes"`
|
||||
UpBytes int64 `json:"up_bytes"`
|
||||
TookSec float64 `json:"took_sec"`
|
||||
// Colo is the Cloudflare edge that answered — "SEA" — and IP is this connection's public
|
||||
// address as that edge saw it.
|
||||
Colo string `json:"colo,omitempty"`
|
||||
IP string `json:"ip,omitempty"`
|
||||
// Source is where the download came from, which matters because the fallback is not in the
|
||||
// same place as the anycast one. Empty means the usual.
|
||||
Source string `json:"source,omitempty"`
|
||||
// Note is set when the figures are real but something went wrong along the way — a stream
|
||||
// that was refused part-way, most often. The number is what was actually seen; the note is
|
||||
// why it may be low.
|
||||
Note string `json:"note,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Run is a test in progress, or the last one to have finished.
|
||||
type Run struct {
|
||||
Started time.Time `json:"started"`
|
||||
Running bool `json:"running"`
|
||||
// Phase is where it has got to: latency, download, upload, done.
|
||||
Phase string `json:"phase"`
|
||||
// Done is how far through the whole test, 0–1, so one bar can show all of it.
|
||||
Done float64 `json:"done"`
|
||||
// Mbps is the live figure for the phase under way — what the needle shows.
|
||||
Mbps float64 `json:"mbps"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Result *Result `json:"result,omitempty"`
|
||||
}
|
||||
|
||||
// Tester runs one speed test at a time and remembers what the last ones said.
|
||||
type Tester struct {
|
||||
mu sync.RWMutex
|
||||
run Run
|
||||
hist []Result
|
||||
// lastTry is when a test was last *started*, which is what the cooldown counts from — a
|
||||
// test that was refused still used the line, and hammering retries at an endpoint that has
|
||||
// just rate-limited us is the one thing guaranteed not to help.
|
||||
lastTry time.Time
|
||||
cancel context.CancelFunc
|
||||
onDone func(Result)
|
||||
maxHist int
|
||||
}
|
||||
|
||||
func NewTester(onDone func(Result)) *Tester {
|
||||
return &Tester{onDone: onDone, maxHist: 50}
|
||||
}
|
||||
|
||||
// Load takes the results kept from before a restart, oldest first.
|
||||
func (t *Tester) Load(rs []Result) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.hist = append(t.hist[:0], rs...)
|
||||
if n := len(t.hist); n > 0 {
|
||||
last := t.hist[n-1]
|
||||
t.run = Run{Started: last.At, Phase: "done", Done: 1, Result: &last}
|
||||
t.lastTry = last.At
|
||||
}
|
||||
}
|
||||
|
||||
func (t *Tester) State() Run {
|
||||
t.mu.RLock()
|
||||
defer t.mu.RUnlock()
|
||||
return t.run
|
||||
}
|
||||
|
||||
func (t *Tester) History() []Result {
|
||||
t.mu.RLock()
|
||||
defer t.mu.RUnlock()
|
||||
return append([]Result{}, t.hist...)
|
||||
}
|
||||
|
||||
func (t *Tester) Running() bool {
|
||||
t.mu.RLock()
|
||||
defer t.mu.RUnlock()
|
||||
return t.run.Running
|
||||
}
|
||||
|
||||
// Cancel stops a test in progress. Reports whether there was one.
|
||||
func (t *Tester) Cancel() bool {
|
||||
t.mu.Lock()
|
||||
c := t.cancel
|
||||
t.mu.Unlock()
|
||||
if c == nil {
|
||||
return false
|
||||
}
|
||||
c()
|
||||
return true
|
||||
}
|
||||
|
||||
var ErrBusy = errors.New("a speed test is already running")
|
||||
|
||||
// Start begins a test in the background and returns its opening state.
|
||||
func (t *Tester) Start() (Run, error) {
|
||||
t.mu.Lock()
|
||||
if t.run.Running {
|
||||
r := t.run
|
||||
t.mu.Unlock()
|
||||
return r, ErrBusy
|
||||
}
|
||||
if since := time.Since(t.lastTry); since < Cooldown && !t.lastTry.IsZero() {
|
||||
r := t.run
|
||||
t.mu.Unlock()
|
||||
return r, fmt.Errorf("give the line %d more seconds — testing back to back is what gets this connection rate-limited", int((Cooldown-since).Seconds())+1)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
t.cancel = cancel
|
||||
t.lastTry = time.Now()
|
||||
t.run = Run{Started: t.lastTry, Running: true, Phase: "latency"}
|
||||
r := t.run
|
||||
t.mu.Unlock()
|
||||
go t.measure(ctx, cancel)
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (t *Tester) set(fn func(*Run)) {
|
||||
t.mu.Lock()
|
||||
fn(&t.run)
|
||||
t.mu.Unlock()
|
||||
}
|
||||
|
||||
func (t *Tester) measure(ctx context.Context, cancel context.CancelFunc) {
|
||||
defer cancel()
|
||||
started := time.Now()
|
||||
res := Result{At: started}
|
||||
|
||||
tr := newTransport(downStreams)
|
||||
defer tr.CloseIdleConnections()
|
||||
client := &http.Client{Transport: tr, Timeout: httpTimeout}
|
||||
|
||||
// latency, and the meta the same responses carry
|
||||
lat, meta, err := t.latency(ctx, client)
|
||||
if err == nil {
|
||||
res.LatencyMs, res.JitterMs = best(lat), jitter(lat)
|
||||
res.Colo, res.IP = meta.Colo, meta.IP
|
||||
}
|
||||
var note string
|
||||
if err == nil {
|
||||
t.set(func(r *Run) { r.Phase, r.Done, r.Mbps = "download", 0.1, 0 })
|
||||
// Each source in turn: a refusal from the first is exactly what the second is for, and
|
||||
// anything else (no route, no DNS) would fail the same way twice, so it stops there.
|
||||
for i, src := range downSources {
|
||||
res.Source = src.Name
|
||||
res.DownMbps, res.DownBytes, note, err = t.phase(ctx, client, "download", downStreams, downFor, 0.1, 0.6, src)
|
||||
if err == nil || !refused(err) || i == len(downSources)-1 {
|
||||
break
|
||||
}
|
||||
log.Printf("speed test: %s refused the download, trying %s", src.Name, downSources[i+1].Name)
|
||||
}
|
||||
res.Note = join(res.Note, note)
|
||||
}
|
||||
if err == nil {
|
||||
t.set(func(r *Run) { r.Phase, r.Done, r.Mbps = "upload", 0.7, 0 })
|
||||
res.UpMbps, res.UpBytes, note, err = t.phase(ctx, client, "upload", upStreams, upFor, 0.7, 1.0, Source{Name: "cloudflare"})
|
||||
res.Note = join(res.Note, note)
|
||||
}
|
||||
res.TookSec = round(time.Since(started).Seconds(), 1)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
res.Error = "cancelled"
|
||||
} else {
|
||||
res.Error = err.Error()
|
||||
}
|
||||
}
|
||||
|
||||
t.mu.Lock()
|
||||
t.cancel = nil
|
||||
t.run = Run{Started: started, Phase: "done", Done: 1, Result: &res, Bytes: res.DownBytes + res.UpBytes}
|
||||
if res.Error == "" || res.DownMbps > 0 { // a cancelled test that got a number is still a reading
|
||||
t.hist = append(t.hist, res)
|
||||
if len(t.hist) > t.maxHist {
|
||||
t.hist = t.hist[len(t.hist)-t.maxHist:]
|
||||
}
|
||||
}
|
||||
done, out := t.onDone, res
|
||||
t.mu.Unlock()
|
||||
if done != nil {
|
||||
done(out)
|
||||
}
|
||||
}
|
||||
|
||||
// meta is what the edge says about the connection. The address comes from a header of the
|
||||
// speed endpoint's own; the edge's name is the tail of the ray ID, which every Cloudflare
|
||||
// response carries ("a41f6720f966a36c-SEA").
|
||||
type meta struct{ IP, Colo string }
|
||||
|
||||
func readMeta(h http.Header) meta {
|
||||
m := meta{IP: h.Get("cf-meta-ip")}
|
||||
if ray := h.Get("cf-ray"); ray != "" {
|
||||
if i := strings.LastIndexByte(ray, '-'); i >= 0 {
|
||||
m.Colo = ray[i+1:]
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// serverMs is how long the far end says it spent on this request, from Server-Timing. The
|
||||
// cfL4 entry carries no dur= and contributes nothing, which is what we want — it is a
|
||||
// description of the connection, not time spent.
|
||||
func serverMs(h http.Header) float64 {
|
||||
var total float64
|
||||
for _, v := range h.Values("Server-Timing") {
|
||||
for _, part := range strings.Split(v, ",") {
|
||||
_, after, ok := strings.Cut(part, "dur=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
end := 0
|
||||
for end < len(after) && (after[end] == '.' || (after[end] >= '0' && after[end] <= '9')) {
|
||||
end++
|
||||
}
|
||||
if f, err := strconv.ParseFloat(after[:end], 64); err == nil {
|
||||
total += f
|
||||
}
|
||||
}
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// latency times several zero-byte responses on an already-open connection.
|
||||
func (t *Tester) latency(ctx context.Context, client *http.Client) ([]float64, meta, error) {
|
||||
var m meta
|
||||
var out []float64
|
||||
for i := 0; i <= latencyProbes; i++ { // the first is the handshake, and is not counted
|
||||
start := time.Now()
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", fmt.Sprintf(cfDownURL, 0), nil)
|
||||
if err != nil {
|
||||
return nil, m, err
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, m, err
|
||||
}
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
return nil, m, fmt.Errorf("speed endpoint answered %s", resp.Status)
|
||||
}
|
||||
if i == 0 {
|
||||
m = readMeta(resp.Header)
|
||||
continue
|
||||
}
|
||||
rtt := float64(time.Since(start).Microseconds())/1000 - serverMs(resp.Header)
|
||||
if rtt < 0.01 { // the far end claimed more time than the whole request took
|
||||
rtt = 0.01
|
||||
}
|
||||
out = append(out, rtt)
|
||||
t.set(func(r *Run) { r.Done = 0.02 + 0.08*float64(i)/latencyProbes })
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, m, errors.New("no reply from the speed endpoint")
|
||||
}
|
||||
return out, m, nil
|
||||
}
|
||||
|
||||
// phase pushes traffic with several streams for a fixed spell and reports the rate over the
|
||||
// window after the ramp. from and to are where this phase sits in the whole test's progress.
|
||||
//
|
||||
// The window ends when the traffic does, not when the clock runs out: if the far end starts
|
||||
// refusing part-way, the figure is what was actually moved while it was moving, and the note
|
||||
// says what happened. Coasting to the end of the clock with no bytes arriving would report a
|
||||
// number a third of the truth.
|
||||
func (t *Tester) phase(parent context.Context, client *http.Client, dir string, streams int, forHow time.Duration, from, to float64, src Source) (float64, int64, string, error) {
|
||||
// parent is the test's own context, which is cancelled when somebody presses Stop. ctx is
|
||||
// this phase's, which *this function* cancels to shut the streams down — so the two are
|
||||
// kept apart: asking ctx whether the test was cancelled would always say yes by then, and
|
||||
// every failure would be reported as "context canceled" whatever really happened.
|
||||
ctx, stop := context.WithCancel(parent)
|
||||
defer stop()
|
||||
|
||||
var moved atomic.Int64
|
||||
var firstErr error
|
||||
var errOnce sync.Once
|
||||
var alive atomic.Int32
|
||||
alive.Store(int32(streams))
|
||||
dead := make(chan struct{})
|
||||
var deadOnce sync.Once
|
||||
|
||||
for i := 0; i < streams; i++ {
|
||||
go func() {
|
||||
defer func() {
|
||||
if alive.Add(-1) == 0 {
|
||||
deadOnce.Do(func() { close(dead) })
|
||||
}
|
||||
}()
|
||||
tries := 0
|
||||
for ctx.Err() == nil && moved.Load() < phaseBudget {
|
||||
var err error
|
||||
if dir == "download" {
|
||||
err = stream(ctx, client, src, &moved)
|
||||
} else {
|
||||
err = upstream(ctx, client, &moved)
|
||||
}
|
||||
if err == nil {
|
||||
tries = 0
|
||||
continue
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
errOnce.Do(func() { firstErr = err })
|
||||
if tries++; tries > 2 { // refused three times: this stream is finished
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-time.After(400 * time.Millisecond):
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// the ramp, measured and thrown away
|
||||
select {
|
||||
case <-time.After(warmUp):
|
||||
case <-dead:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
mark, at := moved.Load(), time.Now()
|
||||
last, lastAt := mark, at // the last time bytes were still arriving
|
||||
deadline := at.Add(forHow)
|
||||
tick := time.NewTicker(200 * time.Millisecond)
|
||||
defer tick.Stop()
|
||||
|
||||
for time.Now().Before(deadline) {
|
||||
select {
|
||||
case <-tick.C:
|
||||
case <-dead:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
if parent.Err() != nil {
|
||||
break
|
||||
}
|
||||
now := moved.Load()
|
||||
if now > last {
|
||||
last, lastAt = now, time.Now()
|
||||
}
|
||||
// enough has moved to know, and it has been long enough to mean it
|
||||
if (now-mark >= measureBytes && time.Since(at) >= minWindow) || now >= phaseBudget {
|
||||
break
|
||||
}
|
||||
if secs := time.Since(at).Seconds(); secs > 0 {
|
||||
live := mbps(now-mark, secs)
|
||||
part := math.Min(1, secs/forHow.Seconds())
|
||||
t.set(func(r *Run) { r.Mbps, r.Bytes, r.Done = live, now, from+(to-from)*part })
|
||||
}
|
||||
// every stream has given up and nothing more is coming
|
||||
if alive.Load() == 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
total, secs := last-mark, lastAt.Sub(at).Seconds()
|
||||
stop()
|
||||
|
||||
if total <= 0 {
|
||||
switch {
|
||||
case firstErr != nil: // what the far end said explains it better than anything else
|
||||
return 0, moved.Load(), "", firstErr
|
||||
case parent.Err() != nil:
|
||||
return 0, moved.Load(), "", parent.Err()
|
||||
}
|
||||
return 0, moved.Load(), "", fmt.Errorf("nothing moved in the %s phase", dir)
|
||||
}
|
||||
if secs < 0.02 { // too short a window to divide by; the guard is against dividing by zero,
|
||||
// not a judgement about how long a good measurement takes
|
||||
return 0, moved.Load(), "", fmt.Errorf("the %s phase was over before it could be measured", dir)
|
||||
}
|
||||
note := ""
|
||||
if firstErr != nil {
|
||||
note = fmt.Sprintf("the %s phase was cut short: %s", dir, firstErr)
|
||||
}
|
||||
return mbps(total, secs), moved.Load(), note, nil
|
||||
}
|
||||
|
||||
// join puts two notes together without leaving an empty one dangling.
|
||||
func join(a, b string) string {
|
||||
switch {
|
||||
case a == "":
|
||||
return b
|
||||
case b == "":
|
||||
return a
|
||||
}
|
||||
return a + "; " + b
|
||||
}
|
||||
|
||||
// stream fetches one chunk from src, counting bytes as they arrive.
|
||||
func stream(ctx context.Context, client *http.Client, src Source, n *atomic.Int64) error {
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", src.URL(downChunk), nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
return answered(resp.StatusCode, resp.Status)
|
||||
}
|
||||
buf := make([]byte, 64<<10)
|
||||
for {
|
||||
got, err := resp.Body.Read(buf)
|
||||
if got > 0 {
|
||||
n.Add(int64(got))
|
||||
}
|
||||
if err != nil {
|
||||
if err == io.EOF {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// upstream posts one chunk of nothing in particular, counting bytes as they go out.
|
||||
func upstream(ctx context.Context, client *http.Client, n *atomic.Int64) error {
|
||||
body := &zeros{left: upChunk, n: n}
|
||||
req, err := http.NewRequestWithContext(ctx, "POST", upURL, body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.ContentLength = upChunk
|
||||
req.Header.Set("Content-Type", "application/octet-stream")
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != 200 && resp.StatusCode != 204 {
|
||||
return answered(resp.StatusCode, resp.Status)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// answered turns the far end's refusal into something worth reading. It rate-limits by
|
||||
// volume: enough testing in a short while and it stops serving the big transfers, which is
|
||||
// the only failure of this test anyone here is likely to meet.
|
||||
func answered(code int, status string) error {
|
||||
if code == 429 || code == 403 {
|
||||
return errRefused
|
||||
}
|
||||
return fmt.Errorf("the speed endpoint answered %s", status)
|
||||
}
|
||||
|
||||
// errRefused is the far end rate-limiting this connection, which is the only failure of this
|
||||
// test anyone here is likely to meet: it limits by volume, and a test moves a few hundred
|
||||
// megabytes. It is recognisable so the download phase can try its other source.
|
||||
var errRefused = errors.New("the speed endpoint is rate-limiting this connection — it does that after a lot of testing; leave it a few minutes")
|
||||
|
||||
func refused(err error) bool { return errors.Is(err, errRefused) }
|
||||
|
||||
// zeros is a request body of a fixed size that costs nothing to produce, counting itself as
|
||||
// it is read out onto the wire.
|
||||
type zeros struct {
|
||||
left int
|
||||
n *atomic.Int64
|
||||
}
|
||||
|
||||
func (z *zeros) Read(p []byte) (int, error) {
|
||||
if z.left <= 0 {
|
||||
return 0, io.EOF
|
||||
}
|
||||
if len(p) > z.left {
|
||||
p = p[:z.left]
|
||||
}
|
||||
for i := range p {
|
||||
p[i] = 0
|
||||
}
|
||||
z.left -= len(p)
|
||||
z.n.Add(int64(len(p)))
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func newTransport(streams int) *http.Transport {
|
||||
return &http.Transport{
|
||||
Proxy: http.ProxyFromEnvironment,
|
||||
DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext,
|
||||
TLSHandshakeTimeout: 10 * time.Second,
|
||||
MaxIdleConns: streams * 2,
|
||||
MaxIdleConnsPerHost: streams * 2,
|
||||
// An empty map turns HTTP/2 off: see the note at the top of this file.
|
||||
TLSNextProto: map[string]func(string, *tls.Conn) http.RoundTripper{},
|
||||
}
|
||||
}
|
||||
|
||||
// probeOutside asks the far end who it thinks we are. One small request.
|
||||
func probeOutside() Outside {
|
||||
tr := newTransport(1)
|
||||
defer tr.CloseIdleConnections()
|
||||
client := &http.Client{Transport: tr, Timeout: 15 * time.Second}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", traceURL, nil)
|
||||
if err != nil {
|
||||
return Outside{At: time.Now(), Error: err.Error()}
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return Outside{At: time.Now(), Error: shortErr(err)}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
|
||||
if resp.StatusCode != 200 {
|
||||
return Outside{At: time.Now(), Error: "the trace endpoint answered " + resp.Status}
|
||||
}
|
||||
o := Outside{At: time.Now()}
|
||||
for _, line := range strings.Split(string(body), "\n") {
|
||||
k, v, ok := strings.Cut(line, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
switch k {
|
||||
case "ip":
|
||||
o.IP = v
|
||||
case "colo":
|
||||
o.Colo = v
|
||||
case "loc":
|
||||
o.Country = v
|
||||
}
|
||||
}
|
||||
if o.IP == "" {
|
||||
o.Error = "the trace endpoint said nothing about this connection"
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// shortErr keeps the part of a network error a person can act on.
|
||||
func shortErr(err error) string {
|
||||
s := err.Error()
|
||||
if i := strings.LastIndex(s, ": "); i > 0 && len(s)-i < 60 {
|
||||
return s[i+2:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func mbps(bytes int64, secs float64) float64 {
|
||||
return round(float64(bytes)*8/secs/1e6, 1)
|
||||
}
|
||||
|
||||
func best(v []float64) float64 {
|
||||
m := v[0]
|
||||
for _, x := range v {
|
||||
if x < m {
|
||||
m = x
|
||||
}
|
||||
}
|
||||
return round(m, 1)
|
||||
}
|
||||
|
||||
// jitter is how much the line wobbles: the average change from one round trip to the next,
|
||||
// which is the usual definition and, unlike a standard deviation, is not run away with by a
|
||||
// single slow probe.
|
||||
func jitter(v []float64) float64 {
|
||||
if len(v) < 2 {
|
||||
return 0
|
||||
}
|
||||
var sum float64
|
||||
for i := 1; i < len(v); i++ {
|
||||
sum += math.Abs(v[i] - v[i-1])
|
||||
}
|
||||
return round(sum/float64(len(v)-1), 1)
|
||||
}
|
||||
|
||||
func round(v float64, places int) float64 {
|
||||
p := math.Pow(10, float64(places))
|
||||
return math.Round(v*p) / p
|
||||
}
|
||||
|
||||
// Mbits renders a rate the way the line is sold, for a log line.
|
||||
func Mbits(v float64) string { return strconv.FormatFloat(v, 'f', 1, 64) + " Mbps" }
|
||||
@@ -0,0 +1,464 @@
|
||||
// Package nodes is the hub side of multi-host: the list of managed Ham Labs, their
|
||||
// health, adding one (pair with its password, receive a token, pin its certificate),
|
||||
// removing one (revoke the token), and the reverse proxy that turns
|
||||
// /api/nodes/<name>/… into /api/… on that node, WebSockets included.
|
||||
package nodes
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/tlsauto"
|
||||
)
|
||||
|
||||
var nameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,31}$`)
|
||||
|
||||
type Status struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
Fingerprint string `json:"fingerprint,omitempty"`
|
||||
Added time.Time `json:"added"`
|
||||
Online bool `json:"online"`
|
||||
Host string `json:"host,omitempty"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Load string `json:"load,omitempty"`
|
||||
Uptime float64 `json:"uptime,omitempty"`
|
||||
RebootNeed bool `json:"reboot_needed,omitempty"`
|
||||
LastSeen time.Time `json:"last_seen,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
cfg *config.Config
|
||||
hubName string
|
||||
mu sync.Mutex
|
||||
status map[string]*Status
|
||||
proxies map[string]*httputil.ReverseProxy
|
||||
clients map[string]*http.Client
|
||||
}
|
||||
|
||||
func New(cfg *config.Config, hubName string) *Manager {
|
||||
return &Manager{cfg: cfg, hubName: hubName, status: map[string]*Status{}, proxies: map[string]*httputil.ReverseProxy{}, clients: map[string]*http.Client{}}
|
||||
}
|
||||
|
||||
func transportFor(n config.Node) *http.Transport {
|
||||
t := &http.Transport{
|
||||
DialContext: (&net.Dialer{Timeout: 5 * time.Second}).DialContext,
|
||||
TLSHandshakeTimeout: 5 * time.Second,
|
||||
ResponseHeaderTimeout: 60 * time.Second,
|
||||
MaxIdleConnsPerHost: 8,
|
||||
}
|
||||
if strings.HasPrefix(n.URL, "https://") {
|
||||
if n.Fingerprint != "" {
|
||||
t.TLSClientConfig = tlsauto.PinnedConfig(n.Fingerprint)
|
||||
} else {
|
||||
t.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
|
||||
}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func (m *Manager) client(n config.Node) *http.Client {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if c, ok := m.clients[n.Name]; ok {
|
||||
return c
|
||||
}
|
||||
c := &http.Client{Transport: transportFor(n), Timeout: 15 * time.Second}
|
||||
m.clients[n.Name] = c
|
||||
return c
|
||||
}
|
||||
|
||||
func (m *Manager) node(name string) (config.Node, bool) {
|
||||
var found config.Node
|
||||
ok := false
|
||||
m.cfg.View(func(c *config.Config) {
|
||||
for _, n := range c.Nodes {
|
||||
if n.Name == name {
|
||||
found, ok = n, true
|
||||
}
|
||||
}
|
||||
})
|
||||
return found, ok
|
||||
}
|
||||
|
||||
// ---- probing ----
|
||||
|
||||
func (m *Manager) probe(ctx context.Context, n config.Node) *Status {
|
||||
st := &Status{Name: n.Name, URL: n.URL, Fingerprint: n.Fingerprint, Added: n.Added}
|
||||
req, _ := http.NewRequestWithContext(ctx, "GET", strings.TrimRight(n.URL, "/")+"/api/me", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
resp, err := m.client(n).Do(req)
|
||||
if err != nil {
|
||||
st.Error = shortErr(err)
|
||||
return st
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
st.Error = fmt.Sprintf("HTTP %d", resp.StatusCode)
|
||||
if resp.StatusCode == 401 {
|
||||
st.Error = "token rejected (re-add the machine)"
|
||||
}
|
||||
return st
|
||||
}
|
||||
var me struct {
|
||||
Host, Version, Load string
|
||||
Uptime float64
|
||||
RebootNeeded bool `json:"reboot_needed"`
|
||||
}
|
||||
if json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&me) != nil {
|
||||
st.Error = "bad reply"
|
||||
return st
|
||||
}
|
||||
st.Online, st.Host, st.Version, st.Load, st.Uptime, st.RebootNeed, st.LastSeen = true, me.Host, me.Version, me.Load, me.Uptime, me.RebootNeeded, time.Now()
|
||||
return st
|
||||
}
|
||||
|
||||
func shortErr(err error) string {
|
||||
s := err.Error()
|
||||
if i := strings.LastIndex(s, ": "); i >= 0 {
|
||||
s = s[i+2:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Run refreshes every node's status periodically.
|
||||
func (m *Manager) Run(ctx context.Context) {
|
||||
m.refresh(ctx)
|
||||
t := time.NewTicker(15 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
m.refresh(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Manager) refresh(ctx context.Context) {
|
||||
var list []config.Node
|
||||
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
|
||||
var wg sync.WaitGroup
|
||||
for _, n := range list {
|
||||
wg.Add(1)
|
||||
go func(n config.Node) {
|
||||
defer wg.Done()
|
||||
pctx, cancel := context.WithTimeout(ctx, 8*time.Second)
|
||||
defer cancel()
|
||||
st := m.probe(pctx, n)
|
||||
m.mu.Lock()
|
||||
if prev, ok := m.status[n.Name]; ok && !st.Online && prev.Online {
|
||||
st.LastSeen = prev.LastSeen // keep the last good sighting
|
||||
}
|
||||
m.status[n.Name] = st
|
||||
m.mu.Unlock()
|
||||
}(n)
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// List returns every managed node with its last known status.
|
||||
func (m *Manager) List() []Status {
|
||||
var list []config.Node
|
||||
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
out := make([]Status, 0, len(list))
|
||||
for _, n := range list {
|
||||
if st, ok := m.status[n.Name]; ok {
|
||||
out = append(out, *st)
|
||||
} else {
|
||||
out = append(out, Status{Name: n.Name, URL: n.URL, Fingerprint: n.Fingerprint, Added: n.Added, Error: "not probed yet"})
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- add / remove ----
|
||||
|
||||
// Add pairs with a node: pin its certificate (trust on first use), trade the node's
|
||||
// password for a token named after this hub, and store the node. The password is
|
||||
// used once and never kept.
|
||||
func (m *Manager) Add(ctx context.Context, name, rawURL, username, password string) (Status, error) {
|
||||
if !nameRe.MatchString(name) {
|
||||
return Status{}, errors.New("name: lowercase letters, digits and dashes only")
|
||||
}
|
||||
if _, exists := m.node(name); exists {
|
||||
return Status{}, errors.New("a machine with that name already exists")
|
||||
}
|
||||
u, err := url.Parse(strings.TrimSpace(rawURL))
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
return Status{}, errors.New("address must be http(s)://host:port")
|
||||
}
|
||||
n := config.Node{Name: name, URL: u.Scheme + "://" + u.Host, Added: time.Now()}
|
||||
if u.Scheme == "https" {
|
||||
host := u.Host
|
||||
if _, _, err := net.SplitHostPort(host); err != nil {
|
||||
host += ":443"
|
||||
}
|
||||
fp, err := tlsauto.Peek(host)
|
||||
if err != nil {
|
||||
return Status{}, fmt.Errorf("connect: %s", shortErr(err))
|
||||
}
|
||||
n.Fingerprint = fp
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"username": username, "password": password, "name": "hub:" + m.hubName})
|
||||
req, _ := http.NewRequestWithContext(ctx, "POST", n.URL+"/api/tokens", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := (&http.Client{Transport: transportFor(n), Timeout: 15 * time.Second}).Do(req)
|
||||
if err != nil {
|
||||
return Status{}, fmt.Errorf("connect: %s", shortErr(err))
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
var reply struct {
|
||||
Token string `json:"token"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&reply)
|
||||
if resp.StatusCode != 200 || reply.Token == "" {
|
||||
if reply.Error == "" {
|
||||
reply.Error = fmt.Sprintf("HTTP %d", resp.StatusCode)
|
||||
}
|
||||
return Status{}, errors.New("pairing refused: " + reply.Error)
|
||||
}
|
||||
n.Token = reply.Token
|
||||
if err := m.cfg.Update(func(c *config.Config) { c.Nodes = append(c.Nodes, n) }); err != nil {
|
||||
return Status{}, err
|
||||
}
|
||||
st := m.probe(ctx, n)
|
||||
m.mu.Lock()
|
||||
m.status[name] = st
|
||||
m.mu.Unlock()
|
||||
return *st, nil
|
||||
}
|
||||
|
||||
// Remove revokes our token on the node (best effort) and forgets it.
|
||||
func (m *Manager) Remove(ctx context.Context, name string) error {
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
return errors.New("no such machine")
|
||||
}
|
||||
req, _ := http.NewRequestWithContext(ctx, "DELETE", n.URL+"/api/tokens/"+url.PathEscape("hub:"+m.hubName), nil)
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
if resp, err := m.client(n).Do(req); err == nil {
|
||||
resp.Body.Close()
|
||||
}
|
||||
err := m.cfg.Update(func(c *config.Config) {
|
||||
kept := c.Nodes[:0]
|
||||
for _, x := range c.Nodes {
|
||||
if x.Name != name {
|
||||
kept = append(kept, x)
|
||||
}
|
||||
}
|
||||
c.Nodes = kept
|
||||
})
|
||||
m.mu.Lock()
|
||||
delete(m.status, name)
|
||||
delete(m.proxies, name)
|
||||
delete(m.clients, name)
|
||||
m.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
|
||||
// ---- update from hub ----
|
||||
|
||||
type PushResult struct {
|
||||
Name string `json:"name"`
|
||||
OK bool `json:"ok"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Push uploads binPath (normally the hub's own executable) to the node's self-update
|
||||
// endpoint. The node verifies it, installs it and restarts itself. Refused when the
|
||||
// node's architecture differs from ours.
|
||||
func (m *Manager) Push(ctx context.Context, name, binPath, ourArch string, force bool) PushResult {
|
||||
res := PushResult{Name: name}
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
res.Error = "no such machine"
|
||||
return res
|
||||
}
|
||||
// arch check via /api/me
|
||||
req, _ := http.NewRequestWithContext(ctx, "GET", n.URL+"/api/me", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
resp, err := m.client(n).Do(req)
|
||||
if err != nil {
|
||||
res.Error = "unreachable: " + shortErr(err)
|
||||
return res
|
||||
}
|
||||
var me struct {
|
||||
Arch string `json:"arch"`
|
||||
Version string `json:"version"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&me)
|
||||
resp.Body.Close()
|
||||
if me.Arch != "" && ourArch != "" && me.Arch != ourArch {
|
||||
// look for a sibling build: /usr/local/bin/hamlab-arm64 next to /usr/local/bin/hamlab
|
||||
alt := binPath + "-" + me.Arch
|
||||
if _, err := os.Stat(alt); err != nil {
|
||||
res.Error = fmt.Sprintf("node is %s but only a %s build is installed here (expected %s; run make deploy, which builds it)", me.Arch, ourArch, alt)
|
||||
return res
|
||||
}
|
||||
binPath = alt
|
||||
}
|
||||
f, err := os.Open(binPath)
|
||||
if err != nil {
|
||||
res.Error = err.Error()
|
||||
return res
|
||||
}
|
||||
defer f.Close()
|
||||
st, _ := f.Stat()
|
||||
target := n.URL + "/api/self-update"
|
||||
if force {
|
||||
target += "?force=1"
|
||||
}
|
||||
up, _ := http.NewRequestWithContext(ctx, "POST", target, f)
|
||||
up.ContentLength = st.Size()
|
||||
up.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
up.Header.Set("Content-Type", "application/octet-stream")
|
||||
c := &http.Client{Transport: transportFor(n), Timeout: 5 * time.Minute}
|
||||
r2, err := c.Do(up)
|
||||
if err != nil {
|
||||
res.Error = "upload: " + shortErr(err)
|
||||
return res
|
||||
}
|
||||
defer r2.Body.Close()
|
||||
var reply struct {
|
||||
Version string `json:"version"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r2.Body, 64<<10)).Decode(&reply)
|
||||
if r2.StatusCode != 200 {
|
||||
if reply.Error == "" {
|
||||
reply.Error = fmt.Sprintf("HTTP %d", r2.StatusCode)
|
||||
}
|
||||
res.Error = reply.Error
|
||||
return res
|
||||
}
|
||||
res.OK, res.Version = true, reply.Version
|
||||
return res
|
||||
}
|
||||
|
||||
// PushAll updates every node, in parallel.
|
||||
func (m *Manager) PushAll(ctx context.Context, binPath, ourArch string, force bool) []PushResult {
|
||||
var list []config.Node
|
||||
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
|
||||
out := make([]PushResult, len(list))
|
||||
var wg sync.WaitGroup
|
||||
for i, n := range list {
|
||||
wg.Add(1)
|
||||
go func(i int, name string) {
|
||||
defer wg.Done()
|
||||
out[i] = m.Push(ctx, name, binPath, ourArch, force)
|
||||
}(i, n.Name)
|
||||
}
|
||||
wg.Wait()
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- proxy ----
|
||||
|
||||
// Proxy returns the handler for /api/nodes/<name>/<rest>: it rewrites to <node>/api/<rest>,
|
||||
// swaps the hub's cookie for the node's bearer token, and drops Origin so the node's
|
||||
// WebSocket origin check (meant for browsers) doesn't reject the hub.
|
||||
// Do makes one request to a managed machine as this hub and returns the whole response.
|
||||
func (m *Manager) Do(ctx context.Context, name, method, path string, body []byte, hdr http.Header) (int, []byte, error) {
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
return 0, nil, errors.New("no such machine")
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(n.URL, "/")+path, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
for k, v := range hdr {
|
||||
req.Header[k] = v
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
resp, err := m.client(n).Do(req)
|
||||
if err != nil {
|
||||
return 0, nil, errors.New(shortErr(err))
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
||||
return resp.StatusCode, b, err
|
||||
}
|
||||
|
||||
// Names lists the managed machines.
|
||||
func (m *Manager) Names() []string {
|
||||
var out []string
|
||||
m.cfg.View(func(c *config.Config) {
|
||||
for _, n := range c.Nodes {
|
||||
out = append(out, n.Name)
|
||||
}
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) Proxy(name string) (http.Handler, bool) {
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
m.mu.Lock()
|
||||
if p, ok := m.proxies[name]; ok {
|
||||
m.mu.Unlock()
|
||||
return p, true
|
||||
}
|
||||
m.mu.Unlock()
|
||||
target, _ := url.Parse(n.URL)
|
||||
prefix := "/api/nodes/" + name
|
||||
p := &httputil.ReverseProxy{
|
||||
Director: func(r *http.Request) {
|
||||
r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
|
||||
r.URL.Path = "/api" + strings.TrimPrefix(r.URL.Path, prefix)
|
||||
r.URL.RawPath = ""
|
||||
r.Host = target.Host
|
||||
r.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
r.Header.Del("Cookie")
|
||||
r.Header.Del("Origin")
|
||||
r.Header.Del("X-Forwarded-For")
|
||||
r.Header.Del("X-Forwarded-Proto")
|
||||
},
|
||||
Transport: transportFor(n),
|
||||
FlushInterval: -1, // stream task output / log tails immediately
|
||||
// The node answers with its own policy headers and this hub adds the same ones to
|
||||
// everything it serves; relayed twice, Access-Control-Allow-Origin is invalid and a
|
||||
// browser on another origin refuses the response. The hub's copy is the one that counts.
|
||||
ModifyResponse: func(resp *http.Response) error {
|
||||
for _, h := range []string{"Access-Control-Allow-Origin", "X-Frame-Options", "Content-Security-Policy", "X-Content-Type-Options", "Referrer-Policy"} {
|
||||
resp.Header.Del(h)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": name + " unreachable: " + shortErr(err)})
|
||||
},
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.proxies[name] = p
|
||||
m.mu.Unlock()
|
||||
return p, true
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
// Package priv is the one place that knows how this daemon gets root: it already is root,
|
||||
// or it has passwordless sudo, or passwordless doas (Alpine, some BSD-minded setups), or
|
||||
// it has nothing and the privileged features stay off.
|
||||
package priv
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
mode string // "root" | "sudo" | "doas" | ""
|
||||
prefix []string
|
||||
probed time.Time
|
||||
reprobe = 30 * time.Second
|
||||
)
|
||||
|
||||
// detect settles how root is reached. Finding a way is remembered for good; finding none
|
||||
// is only believed for a while, because the probe can fail for reasons that pass — sudo
|
||||
// taking longer than the timeout on a machine that is still booting left a daemon
|
||||
// convinced it had no root until its next restart.
|
||||
func detect() {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if mode != "" || (!probed.IsZero() && time.Since(probed) < reprobe) {
|
||||
return
|
||||
}
|
||||
probed = time.Now()
|
||||
if os.Geteuid() == 0 {
|
||||
mode = "root"
|
||||
return
|
||||
}
|
||||
for _, tool := range []string{"sudo", "doas"} {
|
||||
if _, err := exec.LookPath(tool); err != nil {
|
||||
continue
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
err := exec.CommandContext(ctx, tool, "-n", "true").Run()
|
||||
cancel()
|
||||
if err == nil {
|
||||
mode, prefix = tool, []string{tool, "-n"}
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Mode reports how root is reached: "root", "sudo", "doas", or "" when it can't be.
|
||||
func Mode() string { detect(); mu.Lock(); defer mu.Unlock(); return mode }
|
||||
|
||||
// Available reports whether commands can be run as root without a prompt.
|
||||
func Available() bool { return Mode() != "" }
|
||||
|
||||
// Wrap returns args prefixed so they run as root. Without privilege it returns args
|
||||
// unchanged; callers gate on Available() where that matters.
|
||||
func Wrap(args ...string) []string {
|
||||
detect()
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return append(append([]string{}, prefix...), args...)
|
||||
}
|
||||
|
||||
// AsUser returns args prefixed so they run as another user with that user's HOME.
|
||||
func AsUser(user string, args ...string) []string {
|
||||
switch Mode() {
|
||||
case "sudo":
|
||||
return append([]string{"sudo", "-n", "-u", user, "-H"}, args...)
|
||||
case "doas":
|
||||
return append([]string{"doas", "-n", "-u", user}, args...)
|
||||
case "root":
|
||||
if user == "root" {
|
||||
return args
|
||||
}
|
||||
// su exists everywhere (util-linux, shadow, busybox), needs no password from root
|
||||
// and sets HOME for the target user.
|
||||
return []string{"su", "-s", "/bin/sh", user, "-c", Quote(args)}
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
// Quote joins args into one POSIX shell command line.
|
||||
func Quote(args []string) string {
|
||||
q := make([]string, len(args))
|
||||
for i, a := range args {
|
||||
q[i] = "'" + strings.ReplaceAll(a, "'", `'\''`) + "'"
|
||||
}
|
||||
return strings.Join(q, " ")
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
// Package selfupdate finds out whether a newer Ham Lab has been published and fetches it.
|
||||
//
|
||||
// A source is one URL. Two shapes are understood, told apart by the URL itself:
|
||||
//
|
||||
// - a Gitea/Forgejo repository API base — ".../api/v1/repos/<owner>/<repo>" — whose
|
||||
// newest release names the version and carries the binaries as assets;
|
||||
// - any other URL, treated as a static directory holding VERSION, SHA256SUMS and
|
||||
// hamlab-linux-<arch>, which is exactly what `make release` writes and what
|
||||
// install.sh already downloads.
|
||||
//
|
||||
// Either way the binary is checked against SHA256SUMS before it is allowed near the
|
||||
// running one. A token is sent when the source is behind a login, so a private Gitea
|
||||
// serves its own fleet without the instance being opened to the world.
|
||||
package selfupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// DefaultCheckEvery is how often a daemon asks, when it is asking at all.
|
||||
const DefaultCheckEvery = 24 * time.Hour
|
||||
|
||||
// DefaultURL is where a build comes from when the config does not say: the project's own
|
||||
// releases. An install elsewhere points `update.url` at its own, or at "off".
|
||||
const DefaultURL = "https://git.bigham.dev/api/v1/repos/mitchell/hamlab"
|
||||
|
||||
// maxBinary is the most we will download; a hamlab binary is ~20 MB.
|
||||
const maxBinary = 512 << 20
|
||||
|
||||
// Source is where updates come from.
|
||||
type Source struct {
|
||||
URL string // API base or static directory
|
||||
Token string // optional; for a source behind a login
|
||||
}
|
||||
|
||||
// Release is one published build.
|
||||
type Release struct {
|
||||
Version string `json:"version"`
|
||||
Notes string `json:"notes,omitempty"`
|
||||
Published time.Time `json:"published,omitempty"`
|
||||
// Binary and Sums are where to get this build for the architecture that was asked for.
|
||||
Binary string `json:"-"`
|
||||
Sums string `json:"-"`
|
||||
}
|
||||
|
||||
// AssetName is what a build for an architecture is called, in a release and on disk.
|
||||
func AssetName(arch string) string { return "hamlab-linux-" + arch }
|
||||
|
||||
func (s Source) api() bool { return strings.Contains(s.URL, "/api/v1/repos/") }
|
||||
|
||||
func (s Source) get(ctx context.Context, url string) (*http.Response, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.Token != "" {
|
||||
req.Header.Set("Authorization", "token "+s.Token)
|
||||
}
|
||||
req.Header.Set("User-Agent", "hamlab-selfupdate")
|
||||
resp, err := (&http.Client{Timeout: 10 * time.Minute}).Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resp.StatusCode != 200 {
|
||||
resp.Body.Close()
|
||||
switch resp.StatusCode {
|
||||
case 401, 403:
|
||||
return nil, fmt.Errorf("%s: the update source needs a login (HTTP %d); set update.token", url, resp.StatusCode)
|
||||
case 404:
|
||||
return nil, fmt.Errorf("%s: nothing published there (HTTP 404)", url)
|
||||
}
|
||||
return nil, fmt.Errorf("%s: HTTP %d", url, resp.StatusCode)
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
func (s Source) getAll(ctx context.Context, url string, limit int64) ([]byte, error) {
|
||||
resp, err := s.get(ctx, url)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
return io.ReadAll(io.LimitReader(resp.Body, limit))
|
||||
}
|
||||
|
||||
// Latest is the newest published build, and where to get it for arch.
|
||||
func (s Source) Latest(ctx context.Context, arch string) (Release, error) {
|
||||
if s.URL == "" {
|
||||
return Release{}, errors.New("no update source configured")
|
||||
}
|
||||
base := strings.TrimSuffix(s.URL, "/")
|
||||
if !s.api() {
|
||||
b, err := s.getAll(ctx, base+"/VERSION", 4096)
|
||||
if err != nil {
|
||||
return Release{}, err
|
||||
}
|
||||
v := strings.TrimSpace(strings.SplitN(string(b), "\n", 2)[0])
|
||||
if v == "" {
|
||||
return Release{}, errors.New("the source published an empty VERSION")
|
||||
}
|
||||
return Release{Version: v, Binary: base + "/" + AssetName(arch), Sums: base + "/SHA256SUMS"}, nil
|
||||
}
|
||||
b, err := s.getAll(ctx, base+"/releases/latest", 4<<20)
|
||||
if err != nil {
|
||||
return Release{}, err
|
||||
}
|
||||
var rel struct {
|
||||
TagName string `json:"tag_name"`
|
||||
Body string `json:"body"`
|
||||
PublishedAt time.Time `json:"published_at"`
|
||||
Assets []struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"browser_download_url"`
|
||||
} `json:"assets"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &rel); err != nil {
|
||||
return Release{}, fmt.Errorf("the source did not answer with a release: %w", err)
|
||||
}
|
||||
if rel.TagName == "" {
|
||||
return Release{}, errors.New("the newest release has no tag")
|
||||
}
|
||||
out := Release{Version: rel.TagName, Notes: rel.Body, Published: rel.PublishedAt}
|
||||
want := AssetName(arch)
|
||||
for _, a := range rel.Assets {
|
||||
switch a.Name {
|
||||
case want:
|
||||
out.Binary = a.URL
|
||||
case "SHA256SUMS":
|
||||
out.Sums = a.URL
|
||||
}
|
||||
}
|
||||
if out.Binary == "" {
|
||||
return out, fmt.Errorf("release %s has no %s", rel.TagName, want)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Fetch downloads the build for arch to dest and checks it against SHA256SUMS. A release
|
||||
// without sums is refused: an unverified binary is not worth installing over a working one.
|
||||
func (s Source) Fetch(ctx context.Context, rel Release, arch, dest string) error {
|
||||
if rel.Binary == "" {
|
||||
return errors.New("that release has no binary for this machine")
|
||||
}
|
||||
if rel.Sums == "" {
|
||||
return errors.New("that release has no SHA256SUMS, so the download cannot be verified")
|
||||
}
|
||||
sums, err := s.getAll(ctx, rel.Sums, 1<<20)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
want := sumFor(string(sums), AssetName(arch))
|
||||
if want == "" {
|
||||
return fmt.Errorf("SHA256SUMS has no entry for %s", AssetName(arch))
|
||||
}
|
||||
resp, err := s.get(ctx, rel.Binary)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
f, err := os.OpenFile(dest, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, maxBinary))
|
||||
cerr := f.Close()
|
||||
if err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
os.Remove(dest)
|
||||
return err
|
||||
}
|
||||
if n < 1<<20 {
|
||||
os.Remove(dest)
|
||||
return fmt.Errorf("downloaded %d bytes: too small to be hamlab", n)
|
||||
}
|
||||
if got := hex.EncodeToString(h.Sum(nil)); got != want {
|
||||
os.Remove(dest)
|
||||
return fmt.Errorf("checksum mismatch: SHA256SUMS says %s, the download is %s", want[:16]+"…", got[:16]+"…")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// sumFor finds a file's checksum in SHA256SUMS ("<hex> <name>", or "*<name>" for binary mode).
|
||||
func sumFor(sums, name string) string {
|
||||
for _, line := range strings.Split(sums, "\n") {
|
||||
f := strings.Fields(strings.TrimSpace(line))
|
||||
if len(f) == 2 && (f[1] == name || f[1] == "*"+name) && len(f[0]) == 64 {
|
||||
return strings.ToLower(f[0])
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Compare orders two versions: -1 if a is older than b, 0 if the same, 1 if newer.
|
||||
// It is install.sh's vnum() rule, so the daemon and the installer never disagree about
|
||||
// which way round two builds go: v0.1.0-69-g9c4f9df → 0001000069, the commits since the
|
||||
// tag included, the hash ignored.
|
||||
func Compare(a, b string) int {
|
||||
na, oka := vnum(a)
|
||||
nb, okb := vnum(b)
|
||||
switch {
|
||||
case !oka && !okb:
|
||||
return strings.Compare(a, b)
|
||||
case !oka:
|
||||
return -1
|
||||
case !okb:
|
||||
return 1
|
||||
case na < nb:
|
||||
return -1
|
||||
case na > nb:
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func vnum(v string) (int64, bool) {
|
||||
v = strings.TrimSpace(v)
|
||||
v = strings.TrimPrefix(v, "v")
|
||||
if i := strings.LastIndex(v, "-g"); i >= 0 && isHex(v[i+2:]) {
|
||||
v = v[:i]
|
||||
}
|
||||
if v == "" {
|
||||
return 0, false
|
||||
}
|
||||
parts := strings.FieldsFunc(v, func(r rune) bool { return r == '.' || r == '-' })
|
||||
nums := make([]int64, 4)
|
||||
for i := 0; i < 4 && i < len(parts); i++ {
|
||||
n, err := strconv.ParseInt(parts[i], 10, 64)
|
||||
if err != nil {
|
||||
if i == 0 {
|
||||
return 0, false
|
||||
}
|
||||
break
|
||||
}
|
||||
nums[i] = n
|
||||
}
|
||||
// %d%03d%03d%05d, as the installer does it
|
||||
return ((nums[0]*1000+nums[1])*1000+nums[2])*100000 + nums[3], true
|
||||
}
|
||||
|
||||
func isHex(s string) bool {
|
||||
if s == "" {
|
||||
return false
|
||||
}
|
||||
for _, r := range s {
|
||||
if !strings.ContainsRune("0123456789abcdefABCDEF", r) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
package selfupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestCompareMatchesInstaller(t *testing.T) {
|
||||
older := []([2]string){
|
||||
{"v0.1.0-69-g9c4f9df", "v0.1.0-107-gad88a48"},
|
||||
{"v0.1.0", "v0.1.0-1-gabc1234"},
|
||||
{"v0.1.0-107", "v0.2.0"},
|
||||
{"v0.9.9-999", "v1.0.0"},
|
||||
{"v0.1.9", "v0.1.10"},
|
||||
}
|
||||
for _, p := range older {
|
||||
if got := Compare(p[0], p[1]); got != -1 {
|
||||
t.Errorf("Compare(%q, %q) = %d, want -1", p[0], p[1], got)
|
||||
}
|
||||
if got := Compare(p[1], p[0]); got != 1 {
|
||||
t.Errorf("Compare(%q, %q) = %d, want 1", p[1], p[0], got)
|
||||
}
|
||||
}
|
||||
same := []string{"v0.1.0-107-gad88a48", "v1.2.3", "dev"}
|
||||
for _, v := range same {
|
||||
if got := Compare(v, v); got != 0 {
|
||||
t.Errorf("Compare(%q, %q) = %d, want 0", v, v, got)
|
||||
}
|
||||
}
|
||||
// the hash is ignored: the same build described two ways is the same build
|
||||
if got := Compare("v0.1.0-107-gad88a48", "v0.1.0-107-gffffff1"); got != 0 {
|
||||
t.Errorf("hash should not count: %d", got)
|
||||
}
|
||||
// "dev" is not a version; anything numbered beats it
|
||||
if got := Compare("dev", "v0.1.0"); got != -1 {
|
||||
t.Errorf("Compare(dev, v0.1.0) = %d, want -1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// serve puts a release behind both shapes of source and returns the two base URLs.
|
||||
func serve(t *testing.T, version string, body []byte, token string) (static, api string, srv *httptest.Server) {
|
||||
t.Helper()
|
||||
sum := sha256.Sum256(body)
|
||||
sums := fmt.Sprintf("%s %s\n%s install.sh\n", hex.EncodeToString(sum[:]), AssetName("amd64"), strings.Repeat("0", 64))
|
||||
mux := http.NewServeMux()
|
||||
guard := func(h http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if token != "" && r.Header.Get("Authorization") != "token "+token {
|
||||
w.WriteHeader(401)
|
||||
return
|
||||
}
|
||||
h(w, r)
|
||||
}
|
||||
}
|
||||
mux.HandleFunc("/dl/VERSION", guard(func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, version) }))
|
||||
mux.HandleFunc("/dl/SHA256SUMS", guard(func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, sums) }))
|
||||
mux.HandleFunc("/dl/"+AssetName("amd64"), guard(func(w http.ResponseWriter, r *http.Request) { w.Write(body) }))
|
||||
mux.HandleFunc("/api/v1/repos/m/hamlab/releases/latest", guard(func(w http.ResponseWriter, r *http.Request) {
|
||||
base := "http://" + r.Host + "/dl/"
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"tag_name": version, "body": "notes here",
|
||||
"assets": []map[string]string{
|
||||
{"name": AssetName("amd64"), "browser_download_url": base + AssetName("amd64")},
|
||||
{"name": "SHA256SUMS", "browser_download_url": base + "SHA256SUMS"},
|
||||
},
|
||||
})
|
||||
}))
|
||||
srv = httptest.NewServer(mux)
|
||||
return srv.URL + "/dl", srv.URL + "/api/v1/repos/m/hamlab", srv
|
||||
}
|
||||
|
||||
func TestLatestAndFetch(t *testing.T) {
|
||||
body := make([]byte, 2<<20)
|
||||
copy(body, "ELF fake hamlab")
|
||||
static, api, srv := serve(t, "v0.9.0-3-gdeadbee", body, "")
|
||||
defer srv.Close()
|
||||
|
||||
for _, url := range []string{static, api} {
|
||||
s := Source{URL: url}
|
||||
rel, err := s.Latest(context.Background(), "amd64")
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", url, err)
|
||||
}
|
||||
if rel.Version != "v0.9.0-3-gdeadbee" {
|
||||
t.Fatalf("%s: version %q", url, rel.Version)
|
||||
}
|
||||
dest := filepath.Join(t.TempDir(), "hamlab.new")
|
||||
if err := s.Fetch(context.Background(), rel, "amd64", dest); err != nil {
|
||||
t.Fatalf("%s: fetch: %v", url, err)
|
||||
}
|
||||
got, _ := os.ReadFile(dest)
|
||||
if len(got) != len(body) {
|
||||
t.Fatalf("%s: got %d bytes, want %d", url, len(got), len(body))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchRefusesBadChecksum(t *testing.T) {
|
||||
body := make([]byte, 2<<20)
|
||||
static, _, srv := serve(t, "v1.0.0", body, "")
|
||||
defer srv.Close()
|
||||
s := Source{URL: static}
|
||||
rel, err := s.Latest(context.Background(), "amd64")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rel.Sums = static + "/VERSION" // sums that name nothing
|
||||
dest := filepath.Join(t.TempDir(), "hamlab.new")
|
||||
if err := s.Fetch(context.Background(), rel, "amd64", dest); err == nil {
|
||||
t.Fatal("a release whose sums do not name the binary must be refused")
|
||||
}
|
||||
if _, err := os.Stat(dest); !os.IsNotExist(err) {
|
||||
t.Fatal("a refused download must not be left on disk")
|
||||
}
|
||||
// and an arch nobody published
|
||||
if _, err := s.Latest(context.Background(), "riscv64"); err == nil {
|
||||
t.Log("static sources cannot know an arch is missing until the fetch; that is fine")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTokenIsSentAndMissingTokenSaysSo(t *testing.T) {
|
||||
body := make([]byte, 2<<20)
|
||||
_, api, srv := serve(t, "v1.0.0", body, "s3cret")
|
||||
defer srv.Close()
|
||||
if _, err := (Source{URL: api}).Latest(context.Background(), "amd64"); err == nil || !strings.Contains(err.Error(), "update.token") {
|
||||
t.Fatalf("a walled source should say a token is needed, got %v", err)
|
||||
}
|
||||
rel, err := (Source{URL: api, Token: "s3cret"}).Latest(context.Background(), "amd64")
|
||||
if err != nil || rel.Version != "v1.0.0" {
|
||||
t.Fatalf("with the token: %v %+v", err, rel)
|
||||
}
|
||||
dest := filepath.Join(t.TempDir(), "hamlab.new")
|
||||
if err := (Source{URL: api, Token: "s3cret"}).Fetch(context.Background(), rel, "amd64", dest); err != nil {
|
||||
t.Fatalf("fetch with token: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
package sensors
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// promFamily names the metric family for a sensor and scales its value to the base unit
|
||||
// Prometheus convention wants (seconds, not minutes).
|
||||
func promFamily(s Sensor) (name string, scale float64) {
|
||||
scale = 1
|
||||
unit := map[string]string{"%": "percent", "°C": "celsius", "rpm": "rpm", "B/s": "bytes_per_second", "B": "bytes",
|
||||
"s": "seconds", "V": "volts", "W": "watts", "MHz": "megahertz"}[s.Unit]
|
||||
if s.Unit == "min" {
|
||||
unit, scale = "seconds", 60
|
||||
}
|
||||
if s.Kind == "count" { // hamlab_count{id=…} would say nothing; name it after the thing counted
|
||||
return "hamlab_" + strings.ReplaceAll(Seg(strings.ReplaceAll(s.ID, ".", "_")), "-", "_"), 1
|
||||
}
|
||||
name = "hamlab_" + s.Kind
|
||||
if unit != "" {
|
||||
name += "_" + unit
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
var promEsc = strings.NewReplacer(`\`, `\\`, `"`, `\"`, "\n", `\n`)
|
||||
|
||||
// WriteProm writes sensors in the Prometheus text exposition format: one family per kind
|
||||
// of measurement, the sensor's ID and its real-world labels as labels. Booleans are 0/1;
|
||||
// a text state is a 1 carrying the text as a label, the usual way to export an enum.
|
||||
func WriteProm(w io.Writer, all []Sensor) {
|
||||
fams := map[string][]string{}
|
||||
for _, s := range all {
|
||||
name, scale := promFamily(s)
|
||||
labels := []string{`id="` + promEsc.Replace(s.ID) + `"`}
|
||||
keys := make([]string, 0, len(s.Labels))
|
||||
for k := range s.Labels {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
if s.Labels[k] != "" {
|
||||
labels = append(labels, Seg(k)+`="`+promEsc.Replace(s.Labels[k])+`"`)
|
||||
}
|
||||
}
|
||||
var v float64
|
||||
switch x := s.Value.(type) {
|
||||
case float64:
|
||||
v = x * scale
|
||||
case bool:
|
||||
if x {
|
||||
v = 1
|
||||
}
|
||||
case string:
|
||||
v = 1
|
||||
labels = append(labels, `value="`+promEsc.Replace(x)+`"`)
|
||||
default:
|
||||
continue
|
||||
}
|
||||
fams[name] = append(fams[name], fmt.Sprintf("%s{%s} %v", name, strings.Join(labels, ","), v))
|
||||
}
|
||||
names := make([]string, 0, len(fams))
|
||||
for n := range fams {
|
||||
names = append(names, n)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, n := range names {
|
||||
fmt.Fprintf(w, "# TYPE %s gauge\n", n)
|
||||
for _, line := range fams[n] {
|
||||
fmt.Fprintln(w, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,275 @@
|
||||
// Package sensors is the one standard way anything reads what Ham Lab measures.
|
||||
//
|
||||
// Every measurable thing is a Sensor: a stable dotted ID ("cpu.core.3.usage",
|
||||
// "fan.nct6775.fan2", "fs.root.used_pct"), a little metadata, and a current value. A
|
||||
// consumer discovers them from the catalog, then asks for exactly the ones it wants —
|
||||
// one by ID, many by glob, or as a shaped query where the client sends the object it
|
||||
// wants and gets it back with the values filled in. See SENSORS.md.
|
||||
//
|
||||
// Sources of sensors are Providers. The registry only joins them; it measures nothing.
|
||||
package sensors
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Sensor is one measurable thing and its reading. Value is a float64, bool or string.
|
||||
type Sensor struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"` // human label
|
||||
Kind string `json:"kind"` // usage | load | temperature | fan | rate | bytes | duration | voltage | power | state
|
||||
Unit string `json:"unit,omitempty"` // % · °C · rpm · B/s · B · s · min · V · W · MHz
|
||||
Value any `json:"value"`
|
||||
T int64 `json:"t"` // unix ms the value was read — slow sensors are older than the tick
|
||||
Min *float64 `json:"min,omitempty"`
|
||||
Max *float64 `json:"max,omitempty"`
|
||||
Warn *float64 `json:"warn,omitempty"` // the hardware's own thresholds, where it states them
|
||||
Crit *float64 `json:"crit,omitempty"`
|
||||
Labels map[string]string `json:"labels,omitempty"` // the real names behind the ID: chip, iface, mount…
|
||||
}
|
||||
|
||||
// Series is a numeric sensor's recent past.
|
||||
type Series struct {
|
||||
ID string `json:"id"`
|
||||
Unit string `json:"unit,omitempty"`
|
||||
T []int64 `json:"t"`
|
||||
V []float64 `json:"v"`
|
||||
}
|
||||
|
||||
// Provider is a source of sensors. Sensors returns every sensor it has, with current values.
|
||||
type Provider interface {
|
||||
Sensors() []Sensor
|
||||
}
|
||||
|
||||
// Historian is a Provider that also remembers: the recent past of every numeric sensor
|
||||
// of its own that match accepts.
|
||||
type Historian interface {
|
||||
SensorHistories(match func(id string) bool) []Series
|
||||
}
|
||||
|
||||
// Ticker is a Provider that can say when its values have changed.
|
||||
type Ticker interface {
|
||||
Subscribe() (tick <-chan struct{}, cancel func())
|
||||
}
|
||||
|
||||
// Subscribe follows the first provider that ticks. ok is false when none does.
|
||||
func (r *Registry) Subscribe() (tick <-chan struct{}, cancel func(), ok bool) {
|
||||
for _, p := range r.providers {
|
||||
if t, is := p.(Ticker); is {
|
||||
tick, cancel = t.Subscribe()
|
||||
return tick, cancel, true
|
||||
}
|
||||
}
|
||||
return nil, nil, false
|
||||
}
|
||||
|
||||
// F is a pointer to a float, for the optional bounds.
|
||||
func F(v float64) *float64 { return &v }
|
||||
|
||||
// Seg makes one ID segment from a real-world name: lower case, [a-z0-9_-] only, so IDs
|
||||
// survive URLs, query strings and templating languages untouched. "Package id 0" →
|
||||
// "package_id_0", "/mnt/tank" → "mnt_tank", "/" → "root".
|
||||
func Seg(name string) string {
|
||||
if name == "/" {
|
||||
return "root"
|
||||
}
|
||||
var b strings.Builder
|
||||
gap := false
|
||||
for _, r := range strings.ToLower(strings.TrimSpace(name)) {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z', r >= '0' && r <= '9', r == '-':
|
||||
if gap && b.Len() > 0 {
|
||||
b.WriteByte('_')
|
||||
}
|
||||
gap = false
|
||||
b.WriteRune(r)
|
||||
default:
|
||||
gap = true
|
||||
}
|
||||
}
|
||||
if b.Len() == 0 {
|
||||
return "unnamed"
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// ID joins already-safe segments.
|
||||
func ID(segs ...string) string { return strings.Join(segs, ".") }
|
||||
|
||||
// Registry joins providers into one namespace.
|
||||
type Registry struct{ providers []Provider }
|
||||
|
||||
func NewRegistry(p ...Provider) *Registry { return &Registry{providers: p} }
|
||||
|
||||
func (r *Registry) Add(p Provider) { r.providers = append(r.providers, p) }
|
||||
|
||||
// All returns every sensor, sorted by ID. Two sensors that would share an ID (two
|
||||
// identical chips) get "_2", "_3" on the colliding one, in the provider's order.
|
||||
func (r *Registry) All() []Sensor {
|
||||
var out []Sensor
|
||||
seen := map[string]int{}
|
||||
for _, p := range r.providers {
|
||||
for _, s := range p.Sensors() {
|
||||
seen[s.ID]++
|
||||
if n := seen[s.ID]; n > 1 {
|
||||
s.ID = fmt.Sprintf("%s_%d", s.ID, n)
|
||||
}
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||
return out
|
||||
}
|
||||
|
||||
// History is one sensor's recent past. ok is false when nothing remembers it.
|
||||
func (r *Registry) History(id string) (Series, bool) {
|
||||
got := r.Histories(func(x string) bool { return x == id })
|
||||
if len(got) == 0 {
|
||||
return Series{}, false
|
||||
}
|
||||
return got[0], true
|
||||
}
|
||||
|
||||
// Histories is the recent past of every numeric sensor match accepts, sorted by ID.
|
||||
func (r *Registry) Histories(match func(id string) bool) []Series {
|
||||
out := []Series{}
|
||||
for _, p := range r.providers {
|
||||
if h, ok := p.(Historian); ok {
|
||||
out = append(out, h.SensorHistories(match)...)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||
return out
|
||||
}
|
||||
|
||||
// Matcher turns comma-separated selectors (IDs and globs) into a match function.
|
||||
func Matcher(selectors string) (func(id string) bool, error) {
|
||||
var sels []string
|
||||
for _, sel := range strings.Split(selectors, ",") {
|
||||
if sel = strings.TrimSpace(sel); sel == "" {
|
||||
continue
|
||||
}
|
||||
if _, err := path.Match(sel, ""); err != nil {
|
||||
return nil, fmt.Errorf("bad pattern %q", sel)
|
||||
}
|
||||
sels = append(sels, sel)
|
||||
}
|
||||
return func(id string) bool {
|
||||
for _, sel := range sels {
|
||||
if ok, _ := path.Match(sel, id); ok {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}, nil
|
||||
}
|
||||
|
||||
// IsGlob reports whether a selector names many sensors rather than one.
|
||||
func IsGlob(sel string) bool { return strings.ContainsAny(sel, "*?[") }
|
||||
|
||||
// Select returns the sensors a selector names: an exact ID, or a glob where * crosses
|
||||
// dots ("fan.*" is every fan, "cpu.core.*.usage" every core).
|
||||
func Select(all []Sensor, sel string) ([]Sensor, error) {
|
||||
if !IsGlob(sel) {
|
||||
for _, s := range all {
|
||||
if s.ID == sel {
|
||||
return []Sensor{s}, nil
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
if _, err := path.Match(sel, ""); err != nil {
|
||||
return nil, fmt.Errorf("bad pattern %q", sel)
|
||||
}
|
||||
var out []Sensor
|
||||
for _, s := range all {
|
||||
if ok, _ := path.Match(sel, s.ID); ok {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Shape answers a shaped query: the client sends the object it wants, with selectors
|
||||
// where the values go, and gets the same object back with them filled in.
|
||||
//
|
||||
// {"cpu": "cpu.usage", "fans": "fan.*", "box": {"up": "uptime"}}
|
||||
// → {"cpu": 12.4, "fans": {"fan.nct6775.fan1": 820}, "box": {"up": 80520}}
|
||||
//
|
||||
// An exact ID becomes its value (null if there is no such sensor here — machines differ,
|
||||
// and a dashboard shouldn't fail because one has no fans). A glob becomes an object keyed
|
||||
// by ID. Objects and arrays are walked. With full, every value is the whole Sensor.
|
||||
func Shape(all []Sensor, query json.RawMessage, full bool) (any, error) {
|
||||
var q any
|
||||
if err := json.Unmarshal(query, &q); err != nil {
|
||||
return nil, errors.New("query is not valid JSON")
|
||||
}
|
||||
budget := MaxValues
|
||||
return shape(all, q, full, 0, &budget)
|
||||
}
|
||||
|
||||
// MaxValues is how many sensor values one query may return. A query is a few bytes that
|
||||
// can ask for everything many times over ({"a":"*","b":"*",…}); without a ceiling 10 KB of
|
||||
// request buys 27 MB of response. Ten thousand is thirty times a whole machine.
|
||||
const MaxValues = 10000
|
||||
|
||||
func shape(all []Sensor, q any, full bool, depth int, budget *int) (any, error) {
|
||||
if depth > 16 {
|
||||
return nil, errors.New("query nested too deep")
|
||||
}
|
||||
leaf := func(s Sensor) any {
|
||||
if full {
|
||||
return s
|
||||
}
|
||||
return s.Value
|
||||
}
|
||||
switch v := q.(type) {
|
||||
case string:
|
||||
got, err := Select(all, v)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !IsGlob(v) {
|
||||
if len(got) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
*budget--
|
||||
return leaf(got[0]), nil
|
||||
}
|
||||
if *budget -= len(got); *budget < 0 {
|
||||
return nil, fmt.Errorf("query returns more than %d values; ask for less", MaxValues)
|
||||
}
|
||||
m := make(map[string]any, len(got))
|
||||
for _, s := range got {
|
||||
m[s.ID] = leaf(s)
|
||||
}
|
||||
return m, nil
|
||||
case map[string]any:
|
||||
m := make(map[string]any, len(v))
|
||||
for k, sub := range v {
|
||||
r, err := shape(all, sub, full, depth+1, budget)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
m[k] = r
|
||||
}
|
||||
return m, nil
|
||||
case []any:
|
||||
a := make([]any, len(v))
|
||||
for i, sub := range v {
|
||||
r, err := shape(all, sub, full, depth+1, budget)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a[i] = r
|
||||
}
|
||||
return a, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("a query holds sensor IDs, objects and arrays; got %v", v)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package sensors
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type fake []Sensor
|
||||
|
||||
func (f fake) Sensors() []Sensor { return f }
|
||||
|
||||
func TestSeg(t *testing.T) {
|
||||
for in, want := range map[string]string{"/": "root", "/mnt/tank": "mnt_tank", "Package id 0": "package_id_0", "enp3s0": "enp3s0", " ": "unnamed", "Tctl+": "tctl", "nvme0n1": "nvme0n1"} {
|
||||
if got := Seg(in); got != want {
|
||||
t.Errorf("Seg(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryDedupes(t *testing.T) {
|
||||
r := NewRegistry(fake{{ID: "temp.nvme.composite", Value: 40.0}}, fake{{ID: "temp.nvme.composite", Value: 50.0}})
|
||||
all := r.All()
|
||||
if len(all) != 2 || all[0].ID != "temp.nvme.composite" || all[1].ID != "temp.nvme.composite_2" {
|
||||
t.Fatalf("got %+v", all)
|
||||
}
|
||||
}
|
||||
|
||||
func TestShape(t *testing.T) {
|
||||
all := []Sensor{{ID: "cpu.usage", Value: 12.5, Unit: "%"}, {ID: "fan.a.fan1", Value: 800.0}, {ID: "fan.a.fan2", Value: 900.0}, {ID: "net.eth0.up", Value: true}}
|
||||
got, err := Shape(all, json.RawMessage(`{"cpu":"cpu.usage","fans":"fan.*","box":{"link":"net.eth0.up","gone":"ups.battery"},"list":["cpu.usage"]}`), false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[string]any{
|
||||
"cpu": 12.5,
|
||||
"fans": map[string]any{"fan.a.fan1": 800.0, "fan.a.fan2": 900.0},
|
||||
"box": map[string]any{"link": true, "gone": nil},
|
||||
"list": []any{12.5},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("got %#v\nwant %#v", got, want)
|
||||
}
|
||||
full, _ := Shape(all, json.RawMessage(`"cpu.usage"`), true)
|
||||
if s, ok := full.(Sensor); !ok || s.Unit != "%" {
|
||||
t.Fatalf("full: got %#v", full)
|
||||
}
|
||||
for _, bad := range []string{`{"x": 5}`, `{"x": "fan.["}`, `not json`} {
|
||||
if _, err := Shape(all, json.RawMessage(bad), false); err == nil {
|
||||
t.Errorf("Shape(%s) should fail", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteProm(t *testing.T) {
|
||||
var b strings.Builder
|
||||
WriteProm(&b, []Sensor{
|
||||
{ID: "cpu.usage", Kind: "usage", Unit: "%", Value: 12.5},
|
||||
{ID: "ups.runtime", Kind: "duration", Unit: "min", Value: 2.0},
|
||||
{ID: "net.eth0.up", Kind: "state", Value: true, Labels: map[string]string{"iface": "eth0"}},
|
||||
{ID: "ups.state", Kind: "state", Value: `Nor"mal`},
|
||||
{ID: "updates.pending", Kind: "count", Value: 3.0},
|
||||
})
|
||||
got := b.String()
|
||||
for _, want := range []string{
|
||||
"# TYPE hamlab_usage_percent gauge\n" + `hamlab_usage_percent{id="cpu.usage"} 12.5`,
|
||||
`hamlab_duration_seconds{id="ups.runtime"} 120`,
|
||||
`hamlab_state{id="net.eth0.up",iface="eth0"} 1`,
|
||||
`hamlab_state{id="ups.state",value="Nor\"mal"} 1`,
|
||||
`hamlab_updates_pending{id="updates.pending"} 3`,
|
||||
} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("missing %q in:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestShapeBudget(t *testing.T) {
|
||||
all := make([]Sensor, 300)
|
||||
for i := range all {
|
||||
all[i] = Sensor{ID: "s." + strings.Repeat("x", i%7) + string(rune('a'+i%26)) + strings.Repeat("y", i/26), Value: 1.0}
|
||||
}
|
||||
q := map[string]string{}
|
||||
for i := 0; i < 40; i++ { // 40 × 300 = 12 000 values
|
||||
q["k"+strings.Repeat("z", i)] = "*"
|
||||
}
|
||||
b, _ := json.Marshal(q)
|
||||
if _, err := Shape(all, b, false); err == nil {
|
||||
t.Fatal("a query returning more than MaxValues must be refused")
|
||||
}
|
||||
if _, err := Shape(all, json.RawMessage(`{"a":"*","b":"*"}`), false); err != nil {
|
||||
t.Fatalf("an ordinary query must pass: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,368 @@
|
||||
// Package services wraps systemctl and journalctl for the Services tab: list units,
|
||||
// show one, read its unit file, read and tail its journal, and start/stop/etc.
|
||||
//
|
||||
// Reads run as the daemon's user (systemd exposes state to everyone; the journal needs
|
||||
// the systemd-journal or adm group, or sudo). Actions go through passwordless sudo when
|
||||
// it's available, otherwise plain systemctl so polkit gets its say.
|
||||
package services
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type Unit struct {
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
Load string `json:"load"`
|
||||
Active string `json:"active"`
|
||||
Sub string `json:"sub"`
|
||||
Enabled string `json:"enabled"` // enabled | disabled | static | masked | … | "" (unknown)
|
||||
Failed bool `json:"failed"`
|
||||
}
|
||||
|
||||
type Detail struct {
|
||||
Unit
|
||||
Props map[string]string `json:"props"`
|
||||
}
|
||||
|
||||
type Entry struct {
|
||||
T int64 `json:"t"` // unix ms
|
||||
Priority int `json:"p"` // syslog priority 0-7
|
||||
Ident string `json:"ident"`
|
||||
PID string `json:"pid,omitempty"`
|
||||
Msg string `json:"msg"`
|
||||
}
|
||||
|
||||
var unitRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9@._:\\-]{0,255}\.service$`)
|
||||
|
||||
func Valid(unit string) bool { return unitRe.MatchString(unit) }
|
||||
|
||||
var Actions = map[string]bool{"start": true, "stop": true, "restart": true, "reload": true, "enable": true, "disable": true, "reset-failed": true}
|
||||
|
||||
// Properties shown on the detail page, in display order.
|
||||
var showProps = []string{"Id", "Description", "LoadState", "ActiveState", "SubState", "UnitFileState", "FragmentPath",
|
||||
"MainPID", "ExecMainStartTimestamp", "ActiveEnterTimestamp", "InactiveEnterTimestamp", "NRestarts", "Result",
|
||||
"MemoryCurrent", "MemoryPeak", "CPUUsageNSec", "TasksCurrent", "ExecStart", "User", "Restart"}
|
||||
|
||||
type Manager struct {
|
||||
sudo bool // actions via sudo -n
|
||||
journalSudo bool // journal needs sudo (user isn't in systemd-journal/adm)
|
||||
mu sync.Mutex
|
||||
cache []Unit
|
||||
cacheAt time.Time
|
||||
}
|
||||
|
||||
// ErrNoSystemd is what the Services tab says on OpenRC, runit and friends.
|
||||
var ErrNoSystemd = errors.New("this host doesn't run systemd, which the Services tab needs")
|
||||
|
||||
// HasSystemd reports whether systemd is the running init (the test sd_booted() uses).
|
||||
func HasSystemd() bool {
|
||||
_, err := os.Stat("/run/systemd/system")
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func New() *Manager {
|
||||
m := &Manager{}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
m.sudo = priv.Available()
|
||||
// Prefer reading the journal as ourselves: a plain child can be killed cleanly when a
|
||||
// tail ends, whereas a root journalctl under sudo can only be reached with sudo kill.
|
||||
var errb bytes.Buffer
|
||||
probe := exec.CommandContext(ctx, "journalctl", "--system", "-n", "1", "-q", "-o", "cat")
|
||||
probe.Stderr = &errb
|
||||
if err := probe.Run(); err != nil || strings.Contains(strings.ToLower(errb.String()), "permission") {
|
||||
m.journalSudo = m.sudo
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Sudo reports whether actions will run through passwordless sudo.
|
||||
func (m *Manager) Sudo() bool { return m.sudo }
|
||||
|
||||
// JournalSudo reports whether reading the system journal needs sudo for this user.
|
||||
func (m *Manager) JournalSudo() bool { return m.journalSudo }
|
||||
|
||||
func run(ctx context.Context, args ...string) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
var out, errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := strings.TrimSpace(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return out.String(), errors.New(msg)
|
||||
}
|
||||
return out.String(), nil
|
||||
}
|
||||
|
||||
// List merges `list-units --all` (runtime state) with `list-unit-files` (enablement).
|
||||
// Cached for a few seconds because the UI polls it.
|
||||
func (m *Manager) List(ctx context.Context) ([]Unit, error) {
|
||||
if !HasSystemd() {
|
||||
return nil, ErrNoSystemd
|
||||
}
|
||||
m.mu.Lock()
|
||||
if time.Since(m.cacheAt) < 200*time.Millisecond && m.cache != nil { // coalesce simultaneous clients only
|
||||
c := m.cache
|
||||
m.mu.Unlock()
|
||||
return c, nil
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
var out, files string
|
||||
var err error
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
files, _ = run(ctx, "systemctl", "list-unit-files", "--type=service", "--output=json", "--no-pager")
|
||||
close(done)
|
||||
}()
|
||||
out, err = run(ctx, "systemctl", "list-units", "--type=service", "--all", "--output=json", "--no-pager", "--plain")
|
||||
<-done
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var units []listedUnit
|
||||
if err := json.Unmarshal([]byte(out), &units); err != nil {
|
||||
// systemd before 246 (RHEL 8, Ubuntu 20.04) ignores --output=json here and prints
|
||||
// the table; the columns have been the same since the beginning.
|
||||
if units = parseUnitTable(out); len(units) == 0 {
|
||||
return nil, fmt.Errorf("parse list-units: %w", err)
|
||||
}
|
||||
}
|
||||
var uf []listedFile
|
||||
if json.Unmarshal([]byte(files), &uf) != nil {
|
||||
uf = parseFileTable(files)
|
||||
}
|
||||
enabled := map[string]string{}
|
||||
for _, f := range uf {
|
||||
enabled[f.UnitFile] = f.State
|
||||
}
|
||||
|
||||
seen := map[string]bool{}
|
||||
list := make([]Unit, 0, len(units)+len(uf))
|
||||
for _, u := range units {
|
||||
if strings.Contains(u.Unit, "@.") { // uninstantiated template
|
||||
continue
|
||||
}
|
||||
seen[u.Unit] = true
|
||||
list = append(list, Unit{Name: u.Unit, Description: u.Description, Load: u.Load, Active: u.Active, Sub: u.Sub,
|
||||
Enabled: enabled[u.Unit], Failed: u.Active == "failed"})
|
||||
}
|
||||
// Unit files that exist but were never loaded this boot (disabled and never started).
|
||||
for _, f := range uf {
|
||||
if seen[f.UnitFile] || strings.Contains(f.UnitFile, "@.") || f.State == "masked" || f.State == "alias" {
|
||||
continue
|
||||
}
|
||||
list = append(list, Unit{Name: f.UnitFile, Load: "not-loaded", Active: "inactive", Sub: "dead", Enabled: f.State})
|
||||
}
|
||||
sort.Slice(list, func(i, j int) bool { return list[i].Name < list[j].Name })
|
||||
|
||||
m.mu.Lock()
|
||||
m.cache, m.cacheAt = list, time.Now()
|
||||
m.mu.Unlock()
|
||||
return list, nil
|
||||
}
|
||||
|
||||
type listedUnit struct{ Unit, Load, Active, Sub, Description string }
|
||||
|
||||
type listedFile struct {
|
||||
UnitFile string `json:"unit_file"`
|
||||
State string `json:"state"`
|
||||
}
|
||||
|
||||
// parseUnitTable reads the plain `systemctl list-units` table: UNIT LOAD ACTIVE SUB DESCRIPTION…
|
||||
func parseUnitTable(text string) []listedUnit {
|
||||
var units []listedUnit
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
f := strings.Fields(strings.TrimLeft(line, "●*× "))
|
||||
if len(f) < 4 || !strings.HasSuffix(f[0], ".service") {
|
||||
continue
|
||||
}
|
||||
units = append(units, listedUnit{Unit: f[0], Load: f[1], Active: f[2], Sub: f[3], Description: strings.Join(f[4:], " ")})
|
||||
}
|
||||
return units
|
||||
}
|
||||
|
||||
// parseFileTable reads the plain `systemctl list-unit-files` table: UNIT FILE, STATE[, PRESET]
|
||||
func parseFileTable(text string) []listedFile {
|
||||
var uf []listedFile
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
if f := strings.Fields(line); len(f) >= 2 && strings.HasSuffix(f[0], ".service") {
|
||||
uf = append(uf, listedFile{UnitFile: f[0], State: f[1]})
|
||||
}
|
||||
}
|
||||
return uf
|
||||
}
|
||||
|
||||
func (m *Manager) invalidate() {
|
||||
m.mu.Lock()
|
||||
m.cacheAt = time.Time{}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
func (m *Manager) Show(ctx context.Context, unit string) (Detail, error) {
|
||||
if !Valid(unit) {
|
||||
return Detail{}, errors.New("invalid unit name")
|
||||
}
|
||||
out, err := run(ctx, "systemctl", "show", unit, "-p", strings.Join(showProps, ","), "--no-pager")
|
||||
if err != nil {
|
||||
return Detail{}, err
|
||||
}
|
||||
props := map[string]string{}
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if k, v, ok := strings.Cut(line, "="); ok && v != "" && v != "[not set]" && v != "infinity" {
|
||||
props[k] = v
|
||||
}
|
||||
}
|
||||
if props["LoadState"] == "not-found" {
|
||||
return Detail{}, errors.New("no such unit")
|
||||
}
|
||||
d := Detail{Props: props}
|
||||
d.Name, d.Description = unit, props["Description"]
|
||||
d.Load, d.Active, d.Sub, d.Enabled = props["LoadState"], props["ActiveState"], props["SubState"], props["UnitFileState"]
|
||||
d.Failed = d.Active == "failed"
|
||||
return d, nil
|
||||
}
|
||||
|
||||
func (m *Manager) Cat(ctx context.Context, unit string) (string, error) {
|
||||
if !Valid(unit) {
|
||||
return "", errors.New("invalid unit name")
|
||||
}
|
||||
return run(ctx, "systemctl", "cat", unit, "--no-pager")
|
||||
}
|
||||
|
||||
func parseEntry(line []byte) (Entry, bool) {
|
||||
var raw map[string]any
|
||||
if json.Unmarshal(line, &raw) != nil {
|
||||
return Entry{}, false
|
||||
}
|
||||
e := Entry{Priority: 6}
|
||||
switch v := raw["MESSAGE"].(type) {
|
||||
case string:
|
||||
e.Msg = v
|
||||
case []any: // non-UTF-8 payload comes as a byte array
|
||||
b := make([]byte, 0, len(v))
|
||||
for _, x := range v {
|
||||
if f, ok := x.(float64); ok {
|
||||
b = append(b, byte(f))
|
||||
}
|
||||
}
|
||||
e.Msg = string(b)
|
||||
default:
|
||||
e.Msg = ""
|
||||
}
|
||||
if p, ok := raw["PRIORITY"].(string); ok {
|
||||
e.Priority, _ = strconv.Atoi(p)
|
||||
}
|
||||
if ts, ok := raw["__REALTIME_TIMESTAMP"].(string); ok {
|
||||
us, _ := strconv.ParseInt(ts, 10, 64)
|
||||
e.T = us / 1000
|
||||
}
|
||||
if id, ok := raw["SYSLOG_IDENTIFIER"].(string); ok {
|
||||
e.Ident = id
|
||||
} else if c, ok := raw["_COMM"].(string); ok {
|
||||
e.Ident = c
|
||||
}
|
||||
if pid, ok := raw["_PID"].(string); ok {
|
||||
e.PID = pid
|
||||
}
|
||||
return e, true
|
||||
}
|
||||
|
||||
func (m *Manager) journalArgs(unit string, n int) []string {
|
||||
args := []string{"journalctl", "-u", unit, "-n", strconv.Itoa(n), "-o", "json", "--no-pager", "--all"}
|
||||
if m.journalSudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
// Journal returns the last n entries.
|
||||
func (m *Manager) Journal(ctx context.Context, unit string, n int) ([]Entry, error) {
|
||||
if !Valid(unit) {
|
||||
return nil, errors.New("invalid unit name")
|
||||
}
|
||||
out, err := run(ctx, m.journalArgs(unit, n)...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries := []Entry{}
|
||||
for _, line := range bytes.Split([]byte(out), []byte("\n")) {
|
||||
if e, ok := parseEntry(line); ok {
|
||||
entries = append(entries, e)
|
||||
}
|
||||
}
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
// Tail streams entries (last n, then follow) to fn until ctx is cancelled.
|
||||
func (m *Manager) Tail(ctx context.Context, unit string, n int, fn func(Entry)) error {
|
||||
if !Valid(unit) {
|
||||
return errors.New("invalid unit name")
|
||||
}
|
||||
args := append(m.journalArgs(unit, n), "-f")
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
// Own process group so the kill reaches journalctl itself, not just a sudo wrapper.
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error {
|
||||
pg := "-" + strconv.Itoa(cmd.Process.Pid)
|
||||
if m.journalSudo { // children are root's; only sudo can signal them
|
||||
k := priv.Wrap("kill", "-9", "--", pg)
|
||||
return exec.Command(k[0], k[1:]...).Run()
|
||||
}
|
||||
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||
}
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
}
|
||||
sc := bufio.NewScanner(stdout)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
|
||||
for sc.Scan() {
|
||||
if e, ok := parseEntry(sc.Bytes()); ok {
|
||||
fn(e)
|
||||
}
|
||||
}
|
||||
_ = cmd.Wait()
|
||||
return nil
|
||||
}
|
||||
|
||||
// Action runs start/stop/restart/reload/enable/disable/reset-failed.
|
||||
func (m *Manager) Action(ctx context.Context, action, unit string) error {
|
||||
if !Valid(unit) {
|
||||
return errors.New("invalid unit name")
|
||||
}
|
||||
if !Actions[action] {
|
||||
return errors.New("unknown action")
|
||||
}
|
||||
args := []string{"systemctl", action, unit, "--no-pager"}
|
||||
if m.sudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
_, err := run(ctx, args...)
|
||||
m.invalidate()
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
package services
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestTables(t *testing.T) {
|
||||
u := parseUnitTable(" UNIT LOAD ACTIVE SUB DESCRIPTION\n● bad.service loaded failed failed Bad thing here\n ssh.service loaded active running OpenBSD Secure Shell server\n\nLOAD = x\n12 loaded units listed.\n")
|
||||
if len(u) != 2 || u[0].Unit != "bad.service" || u[0].Active != "failed" || u[1].Description != "OpenBSD Secure Shell server" {
|
||||
t.Fatalf("%+v", u)
|
||||
}
|
||||
f := parseFileTable("UNIT FILE STATE VENDOR PRESET\nssh.service enabled enabled\nfoo@.service static -\n\n2 unit files listed.\n")
|
||||
if len(f) != 2 || f[0].State != "enabled" {
|
||||
t.Fatalf("%+v", f)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
#!/sbin/openrc-run
|
||||
# Ham Lab web console (written by hamlab install)
|
||||
|
||||
name="hamlab"
|
||||
description="Ham Lab web console"
|
||||
|
||||
export HAMLAB_CONFIG="@CONFIG@"
|
||||
export HOME="@HOME@"
|
||||
export USER="@USER@"
|
||||
|
||||
supervisor="supervise-daemon"
|
||||
command="@BIN@"
|
||||
command_args="serve"
|
||||
command_user="@USER@:@GROUP@"
|
||||
directory="@HOME@"
|
||||
respawn_delay=3
|
||||
respawn_max=0
|
||||
output_log="/var/log/hamlab.log"
|
||||
error_log="/var/log/hamlab.log"
|
||||
|
||||
depend() {
|
||||
need localmount
|
||||
after net firewall
|
||||
}
|
||||
|
||||
start_pre() {
|
||||
checkpath -f -o "@USER@:@GROUP@" -m 0640 /var/log/hamlab.log
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
#!/bin/sh /etc/rc.common
|
||||
# Ham Lab web console (written by hamlab install)
|
||||
|
||||
USE_PROCD=1
|
||||
# after network and firewall, before the tail of the boot
|
||||
START=95
|
||||
STOP=10
|
||||
|
||||
start_service() {
|
||||
procd_open_instance hamlab
|
||||
procd_set_param command "@BIN@" serve
|
||||
procd_set_param env HAMLAB_CONFIG="@CONFIG@" HOME="@HOME@" USER="@USER@"
|
||||
procd_set_param user "@USER@"
|
||||
procd_set_param group "@GROUP@"
|
||||
procd_set_param pidfile /var/run/hamlab.pid
|
||||
# respawn: settle 3s, 5 retries inside that window, then give up counting
|
||||
procd_set_param respawn 3600 3 0
|
||||
procd_set_param stdout 1
|
||||
procd_set_param stderr 1
|
||||
procd_close_instance
|
||||
}
|
||||
|
||||
service_triggers() {
|
||||
procd_add_reload_trigger network
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
[Unit]
|
||||
Description=Ham Lab web console
|
||||
After=network.target
|
||||
# never give up restarting
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=@USER@
|
||||
Group=@GROUP@
|
||||
# read the system journal without sudo (keeps the daemon's own log clean)
|
||||
SupplementaryGroups=@GROUPS@
|
||||
WorkingDirectory=-@HOME@
|
||||
Environment=HAMLAB_CONFIG=@CONFIG@
|
||||
ExecStart=@BIN@ serve
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
KillMode=control-group
|
||||
TimeoutStopSec=10
|
||||
LimitNOFILE=65536
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,718 @@
|
||||
// Package setup is `hamlab install` and `hamlab uninstall`: the binary installs itself, so
|
||||
// a new machine needs nothing but this one file. It copies itself into place, writes the
|
||||
// config, sets the login password and registers a service with whichever init the host
|
||||
// runs (systemd or OpenRC), as root or through sudo/doas.
|
||||
package setup
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"crypto/rand"
|
||||
"crypto/tls"
|
||||
_ "embed"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"golang.org/x/term"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
)
|
||||
|
||||
//go:embed hamlab.service
|
||||
var systemdUnit string
|
||||
|
||||
//go:embed hamlab.openrc
|
||||
var openrcScript string
|
||||
|
||||
//go:embed hamlab.procd
|
||||
var procdScript string
|
||||
|
||||
const (
|
||||
unitPath = "/etc/systemd/system/hamlab.service"
|
||||
openrcPath = "/etc/init.d/hamlab"
|
||||
// procd (OpenWrt) uses the same path as OpenRC; the two are told apart by content.
|
||||
procdPath = "/etc/init.d/hamlab"
|
||||
sudoersPath = "/etc/sudoers.d/hamlab-quiet"
|
||||
)
|
||||
|
||||
type options struct {
|
||||
user, listen, tls, prefix, data string
|
||||
passwordStdin, openFirewall bool
|
||||
noStart bool
|
||||
}
|
||||
|
||||
func say(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) }
|
||||
|
||||
// elevate re-runs this command as root through sudo or doas (which may prompt), naming
|
||||
// the invoking user so the service still runs as them.
|
||||
func elevate(sub string, args []string, keepUser bool) error {
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, tool := range []string{"sudo", "doas"} {
|
||||
path, err := exec.LookPath(tool)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
full := []string{tool, self, sub}
|
||||
if u, err := user.Current(); err == nil && keepUser && !hasFlag(args, "user") {
|
||||
full = append(full, "--user", u.Username)
|
||||
}
|
||||
full = append(full, args...)
|
||||
return syscall.Exec(path, full, os.Environ())
|
||||
}
|
||||
return errors.New("this needs root, and neither sudo nor doas is installed: run it as root")
|
||||
}
|
||||
|
||||
func hasFlag(args []string, name string) bool {
|
||||
for _, a := range args {
|
||||
a = strings.TrimLeft(a, "-")
|
||||
if a == name || strings.HasPrefix(a, name+"=") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Install is `hamlab install [flags]`.
|
||||
func Install(args []string, version string) error {
|
||||
var o options
|
||||
fs := flag.NewFlagSet("install", flag.ContinueOnError)
|
||||
fs.StringVar(&o.user, "user", "", "user the console runs as; terminals are this user's shell (default: you)")
|
||||
fs.StringVar(&o.listen, "listen", "", "listen address, e.g. 0.0.0.0:8100 (default: keep, else 127.0.0.1:8100)")
|
||||
fs.StringVar(&o.tls, "tls", "", `"auto" serves HTTPS with a self-signed certificate, "off" plain HTTP (default: keep)`)
|
||||
fs.StringVar(&o.prefix, "prefix", "/usr/local/bin", "directory the binary is installed into")
|
||||
fs.StringVar(&o.data, "data", "", "config and state directory (default: ~/.config/hamlab of the user)")
|
||||
fs.BoolVar(&o.passwordStdin, "password-stdin", false, "read the login password from stdin instead of prompting")
|
||||
fs.BoolVar(&o.openFirewall, "open-firewall", false, "open the listen port in firewalld/ufw when one is active")
|
||||
fs.BoolVar(&o.noStart, "no-start", false, "install and enable, but don't start the service")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if os.Geteuid() != 0 {
|
||||
return elevate("install", args, true)
|
||||
}
|
||||
sbinPath()
|
||||
|
||||
// who
|
||||
name := o.user
|
||||
for _, env := range []string{"SUDO_USER", "DOAS_USER"} {
|
||||
if name == "" {
|
||||
name = os.Getenv(env)
|
||||
}
|
||||
}
|
||||
if name == "" {
|
||||
name = "root"
|
||||
}
|
||||
acct, err := lookupUser(name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// where
|
||||
dataDir := o.data
|
||||
if dataDir == "" {
|
||||
dataDir = filepath.Dir(existingConfig(acct.home))
|
||||
}
|
||||
cfgPath := filepath.Join(dataDir, "config.json")
|
||||
|
||||
// binary
|
||||
dest := filepath.Join(o.prefix, "hamlab")
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
self, _ = filepath.EvalSymlinks(self)
|
||||
if err := os.MkdirAll(o.prefix, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if self != dest {
|
||||
if err := CopyAtomic(self, dest); err != nil {
|
||||
return fmt.Errorf("install binary: %w", err)
|
||||
}
|
||||
// builds for other architectures sit beside it so a hub can push them to matching machines
|
||||
sibs, _ := filepath.Glob(filepath.Join(filepath.Dir(self), "hamlab-linux-*"))
|
||||
for _, sib := range sibs {
|
||||
arch := strings.TrimPrefix(filepath.Base(sib), "hamlab-linux-")
|
||||
if sib == self || strings.Contains(arch, ".") {
|
||||
continue
|
||||
}
|
||||
if err := CopyAtomic(sib, dest+"-"+arch); err != nil {
|
||||
say("warning: %s: %v", sib, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
restorecon(dest)
|
||||
say("binary %s (%s)", dest, version)
|
||||
|
||||
// config + password
|
||||
cfg, err := config.Load(cfgPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.listen != "" {
|
||||
if _, _, err := net.SplitHostPort(o.listen); err != nil {
|
||||
return fmt.Errorf("--listen %q: want host:port", o.listen)
|
||||
}
|
||||
cfg.Listen = o.listen
|
||||
}
|
||||
switch o.tls {
|
||||
case "":
|
||||
case "auto":
|
||||
cfg.TLS = "auto"
|
||||
case "off", "none":
|
||||
cfg.TLS = ""
|
||||
default:
|
||||
return fmt.Errorf("--tls %q: want auto or off", o.tls)
|
||||
}
|
||||
generated := ""
|
||||
cfg.MigrateUsers(acct.name)
|
||||
if len(cfg.Users) == 0 || o.passwordStdin {
|
||||
pw, gen, err := choosePassword(o.passwordStdin)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h, err := auth.HashPassword(pw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// the first account is named after the OS user the service runs as; --password-stdin
|
||||
// on an existing install resets that first account
|
||||
if len(cfg.Users) == 0 {
|
||||
cfg.Users = []config.User{{Name: acct.name, Hash: h, Created: time.Now()}}
|
||||
} else {
|
||||
cfg.Users[0].Hash = h
|
||||
}
|
||||
if gen {
|
||||
generated = pw
|
||||
}
|
||||
}
|
||||
made := firstMissing(dataDir) // e.g. ~/.config, when this is the first thing to use it
|
||||
if err := cfg.Save(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := chownData(dataDir, made, acct); err != nil {
|
||||
return err
|
||||
}
|
||||
say("config %s", cfgPath)
|
||||
|
||||
// service
|
||||
vars := map[string]string{"@USER@": acct.name, "@GROUP@": acct.group, "@HOME@": acct.home, "@BIN@": dest, "@CONFIG@": cfgPath,
|
||||
"@GROUPS@": journalGroups()}
|
||||
initSys := ""
|
||||
switch {
|
||||
case isDir("/run/systemd/system"):
|
||||
initSys = "systemd"
|
||||
unit := fill(systemdUnit, vars)
|
||||
if vars["@GROUPS@"] == "" {
|
||||
unit = dropLines(unit, "SupplementaryGroups=")
|
||||
}
|
||||
if err := os.WriteFile(unitPath, []byte(unit), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
restorecon(unitPath)
|
||||
if err := runAll([][]string{{"systemctl", "daemon-reload"}, {"systemctl", "enable", "hamlab.service"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
if !o.noStart {
|
||||
if err := run("systemctl", "restart", "hamlab.service"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
case exists("/sbin/procd") && exists("/etc/rc.common"):
|
||||
initSys = "procd"
|
||||
if err := os.WriteFile(procdPath, []byte(fill(procdScript, vars)), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := run(procdPath, "enable"); err != nil {
|
||||
say("warning: %s enable: %v", procdPath, err)
|
||||
}
|
||||
if !o.noStart {
|
||||
if err := run(procdPath, "restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
case exists("/sbin/openrc-run") || exists("/usr/sbin/openrc-run") || have("rc-service"):
|
||||
initSys = "OpenRC"
|
||||
if err := os.WriteFile(openrcPath, []byte(fill(openrcScript, vars)), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := run("rc-update", "add", "hamlab", "default"); err != nil {
|
||||
say("warning: rc-update: %v", err)
|
||||
}
|
||||
if !o.noStart {
|
||||
if err := run("rc-service", "hamlab", "restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if initSys != "" {
|
||||
say("service hamlab (%s), runs as %s, starts at boot", initSys, acct.name)
|
||||
}
|
||||
if acct.name != "root" {
|
||||
quietSudoers()
|
||||
}
|
||||
|
||||
port := firewall(cfg.Listen, o.openFirewall)
|
||||
|
||||
scheme := "http"
|
||||
if cfg.TLS == "auto" {
|
||||
scheme = "https"
|
||||
}
|
||||
url := scheme + "://" + browseAddr(cfg.Listen)
|
||||
switch {
|
||||
case initSys == "":
|
||||
say("\nNo systemd, procd or OpenRC here, so nothing starts it at boot.\nRun it under your own supervisor, as %s:\n HAMLAB_CONFIG=%s %s serve", acct.name, cfgPath, dest)
|
||||
case o.noStart:
|
||||
case healthy(cfg.Listen, cfg.TLS == "auto"):
|
||||
say("\nHam Lab is up: %s", url)
|
||||
default:
|
||||
say("\nInstalled, but %s isn't answering yet. Check: %s", url, map[string]string{"systemd": "journalctl -u hamlab -n 30", "OpenRC": "tail /var/log/hamlab.log"}[initSys])
|
||||
}
|
||||
if generated != "" {
|
||||
say("Login: user %s, password (shown once; change it in Settings): %s", cfg.Users[0].Name, generated)
|
||||
}
|
||||
if host, _, _ := net.SplitHostPort(cfg.Listen); isLoopback(host) {
|
||||
say("It only listens on this machine. For LAN access: hamlab install --listen 0.0.0.0:%s", port)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Uninstall is `hamlab uninstall [--purge]`.
|
||||
func Uninstall(args []string) error {
|
||||
fs := flag.NewFlagSet("uninstall", flag.ContinueOnError)
|
||||
purge := fs.Bool("purge", false, "also delete the config and state directory")
|
||||
prefix := fs.String("prefix", "/usr/local/bin", "directory the binary was installed into")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if os.Geteuid() != 0 {
|
||||
return elevate("uninstall", args, false)
|
||||
}
|
||||
sbinPath()
|
||||
dataDir := ""
|
||||
if b, err := os.ReadFile(unitPath); err == nil {
|
||||
dataDir = configDirFrom(string(b), "Environment=HAMLAB_CONFIG=")
|
||||
_ = run("systemctl", "disable", "--now", "hamlab.service")
|
||||
os.Remove(unitPath)
|
||||
_ = run("systemctl", "daemon-reload")
|
||||
}
|
||||
if b, err := os.ReadFile(openrcPath); err == nil {
|
||||
if strings.Contains(string(b), "USE_PROCD=1") {
|
||||
dataDir = configDirFrom(string(b), "HAMLAB_CONFIG=")
|
||||
_ = run(procdPath, "stop")
|
||||
_ = run(procdPath, "disable")
|
||||
} else {
|
||||
dataDir = configDirFrom(string(b), "export HAMLAB_CONFIG=")
|
||||
_ = run("rc-service", "hamlab", "stop")
|
||||
_ = run("rc-update", "del", "hamlab", "default")
|
||||
}
|
||||
os.Remove(openrcPath)
|
||||
}
|
||||
os.Remove(sudoersPath)
|
||||
bins, _ := filepath.Glob(filepath.Join(*prefix, "hamlab-*"))
|
||||
for _, b := range append(bins, filepath.Join(*prefix, "hamlab")) {
|
||||
os.Remove(b)
|
||||
}
|
||||
say("Ham Lab removed.")
|
||||
switch {
|
||||
case dataDir == "":
|
||||
case *purge:
|
||||
if err := purgeData(dataDir); err != nil {
|
||||
return err
|
||||
}
|
||||
default:
|
||||
say("Config and state kept in %s (uninstall --purge deletes them).", dataDir)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// sbinPath makes sure the admin tools resolve: a root shell reached with plain `su` keeps
|
||||
// the user's PATH, which on Debian has no sbin in it.
|
||||
func sbinPath() {
|
||||
os.Setenv("PATH", os.Getenv("PATH")+":/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin")
|
||||
}
|
||||
|
||||
// ownFiles is everything Ham Lab writes into its data directory.
|
||||
var ownFiles = []string{"config.json", "config.json.tmp", "state.json", "state.json.tmp", "state.json.corrupt", "cert.pem", "key.pem", "hamlab.prev", "hamlab.new"}
|
||||
|
||||
// purgeData deletes Ham Lab's data. The whole directory goes only when it is unmistakably
|
||||
// ours (named "hamlab"); a --data pointed at a shared directory such as a home loses
|
||||
// just the files Ham Lab wrote.
|
||||
func purgeData(dir string) error {
|
||||
if filepath.Base(dir) == "hamlab" {
|
||||
say("Deleted %s.", dir)
|
||||
return os.RemoveAll(dir)
|
||||
}
|
||||
for _, f := range ownFiles {
|
||||
os.Remove(filepath.Join(dir, f))
|
||||
}
|
||||
say("Deleted Ham Lab's files in %s (the directory isn't Ham Lab's alone, so it stays).", dir)
|
||||
return nil
|
||||
}
|
||||
|
||||
func configDirFrom(text, key string) string {
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
if v, ok := strings.CutPrefix(strings.TrimSpace(line), key); ok {
|
||||
if v = strings.Trim(v, `"'`); strings.HasSuffix(v, "/config.json") && filepath.Dir(v) != "/" {
|
||||
return filepath.Dir(v)
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- pieces ----
|
||||
|
||||
type account struct {
|
||||
name, group, home string
|
||||
uid, gid int
|
||||
}
|
||||
|
||||
func lookupUser(name string) (account, error) {
|
||||
a := account{name: name}
|
||||
if u, err := user.Lookup(name); err == nil {
|
||||
a.uid, _ = strconv.Atoi(u.Uid)
|
||||
a.gid, _ = strconv.Atoi(u.Gid)
|
||||
a.home = u.HomeDir
|
||||
} else if out, err2 := exec.Command("getent", "passwd", name).Output(); err2 == nil { // LDAP/SSSD accounts
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if len(f) < 6 {
|
||||
return a, fmt.Errorf("no such user: %s", name)
|
||||
}
|
||||
a.uid, _ = strconv.Atoi(f[2])
|
||||
a.gid, _ = strconv.Atoi(f[3])
|
||||
a.home = f[5]
|
||||
} else {
|
||||
return a, fmt.Errorf("no such user: %s", name)
|
||||
}
|
||||
if a.home == "" || !isDir(a.home) {
|
||||
return a, fmt.Errorf("user %s has no home directory (%q); pick another with --user", name, a.home)
|
||||
}
|
||||
a.group = strconv.Itoa(a.gid)
|
||||
if g, err := user.LookupGroupId(a.group); err == nil {
|
||||
a.group = g.Name
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// existingConfig returns the config path for a home directory: whichever already exists,
|
||||
// else the default for a new install.
|
||||
func existingConfig(home string) string {
|
||||
for _, p := range config.Candidates(home) {
|
||||
if exists(p) {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return config.Candidates(home)[0]
|
||||
}
|
||||
|
||||
// journalGroups lists the groups that can read the system journal and exist here. A unit
|
||||
// naming a group the host doesn't have fails to start, and "adm" is not universal.
|
||||
func journalGroups() string {
|
||||
var have []string
|
||||
for _, g := range []string{"systemd-journal", "adm"} {
|
||||
if _, err := user.LookupGroup(g); err == nil {
|
||||
have = append(have, g)
|
||||
}
|
||||
}
|
||||
return strings.Join(have, " ")
|
||||
}
|
||||
|
||||
func choosePassword(fromStdin bool) (pw string, generated bool, err error) {
|
||||
switch {
|
||||
case fromStdin:
|
||||
line, _ := bufio.NewReader(os.Stdin).ReadString('\n')
|
||||
pw = strings.TrimRight(line, "\r\n")
|
||||
case term.IsTerminal(int(os.Stdin.Fd())):
|
||||
fmt.Fprint(os.Stderr, "Login password for the web console (12+ characters, empty to generate one): ")
|
||||
p1, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
if len(p1) > 0 {
|
||||
fmt.Fprint(os.Stderr, "Again: ")
|
||||
p2, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
if !bytes.Equal(p1, p2) {
|
||||
return "", false, errors.New("passwords do not match")
|
||||
}
|
||||
}
|
||||
pw = string(p1)
|
||||
}
|
||||
if pw == "" && !fromStdin {
|
||||
return randomPassword(), true, nil
|
||||
}
|
||||
if len(pw) < 12 {
|
||||
return "", false, errors.New("password must be at least 12 characters")
|
||||
}
|
||||
return pw, false, nil
|
||||
}
|
||||
|
||||
func randomPassword() string {
|
||||
const alphabet = "abcdefghijkmnpqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ23456789" // no look-alikes
|
||||
b := make([]byte, 20)
|
||||
_, _ = rand.Read(b)
|
||||
for i := range b {
|
||||
b[i] = alphabet[int(b[i])%len(alphabet)]
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// CopyAtomic puts src at dst with mode 0755 by writing beside it and renaming, so dst is
|
||||
// always either the old file or the whole new one, and a running dst is never written to.
|
||||
func CopyAtomic(src, dst string) error {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer in.Close()
|
||||
tmp := filepath.Join(filepath.Dir(dst), "."+filepath.Base(dst)+".new")
|
||||
out, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = io.Copy(out, in)
|
||||
if err == nil {
|
||||
err = out.Sync()
|
||||
}
|
||||
if cerr := out.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err == nil {
|
||||
err = os.Chmod(tmp, 0o755)
|
||||
}
|
||||
if err == nil {
|
||||
err = os.Rename(tmp, dst)
|
||||
}
|
||||
if err != nil {
|
||||
os.Remove(tmp)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// firstMissing returns the topmost directory that creating dir would have to make, or "".
|
||||
func firstMissing(dir string) string {
|
||||
missing := ""
|
||||
for d := filepath.Clean(dir); d != "/" && d != "." && !exists(d); d = filepath.Dir(d) {
|
||||
missing = d
|
||||
}
|
||||
return missing
|
||||
}
|
||||
|
||||
// chownData hands Ham Lab's data to the account that runs it. Directories this install
|
||||
// created are ours to hand over whole. In one that already existed (a --data pointed at
|
||||
// something shared), only the files Ham Lab writes change owner: a recursive chown of
|
||||
// somebody else's tree can't be undone.
|
||||
func chownData(dataDir, made string, a account) error {
|
||||
if made != "" {
|
||||
return chownTree(made, a)
|
||||
}
|
||||
if filepath.Base(dataDir) == "hamlab" {
|
||||
return chownTree(dataDir, a)
|
||||
}
|
||||
for _, f := range ownFiles {
|
||||
if p := filepath.Join(dataDir, f); exists(p) {
|
||||
if err := os.Lchown(p, a.uid, a.gid); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chownTree(dir string, a account) error {
|
||||
return filepath.Walk(dir, func(p string, _ os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Lchown(p, a.uid, a.gid)
|
||||
})
|
||||
}
|
||||
|
||||
func fill(tmpl string, vars map[string]string) string {
|
||||
for k, v := range vars {
|
||||
tmpl = strings.ReplaceAll(tmpl, k, v)
|
||||
}
|
||||
return tmpl
|
||||
}
|
||||
|
||||
func dropLines(text, prefix string) string {
|
||||
var keep []string
|
||||
for _, l := range strings.Split(text, "\n") {
|
||||
if !strings.HasPrefix(l, prefix) {
|
||||
keep = append(keep, l)
|
||||
}
|
||||
}
|
||||
return strings.Join(keep, "\n")
|
||||
}
|
||||
|
||||
// quietSudoers keeps the daemon's read-only polling (SMART, UPS, journal) out of the auth
|
||||
// log: at 250 ms refresh rates the sudo session lines would otherwise dominate it.
|
||||
func quietSudoers() {
|
||||
if !isDir("/etc/sudoers.d") || !have("visudo") {
|
||||
return
|
||||
}
|
||||
var cmds []string
|
||||
for _, bin := range []string{"smartctl", "pwrstat", "journalctl", "tail", "cat", "ls", "crontab"} {
|
||||
for _, dir := range []string{"/usr/sbin", "/usr/bin", "/sbin", "/bin"} {
|
||||
if exists(dir + "/" + bin) {
|
||||
cmds = append(cmds, dir+"/"+bin)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, bin := range []string{"zpool", "zfs"} { // listing only: scrubs and snapshots still log
|
||||
for _, dir := range []string{"/usr/sbin", "/sbin", "/usr/bin", "/bin"} {
|
||||
if exists(dir + "/" + bin) {
|
||||
cmds = append(cmds, dir+"/"+bin+" list *", dir+"/"+bin+" status *")
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(cmds) == 0 {
|
||||
return
|
||||
}
|
||||
text := "# Ham Lab: don't log the daemon's read-only polling commands (written by hamlab install)\n" +
|
||||
"Cmnd_Alias HAMLAB_QUIET = " + strings.Join(cmds, ", ") + "\n" +
|
||||
"Defaults!HAMLAB_QUIET !syslog, !pam_session, !pam_setcred\n"
|
||||
tmp := sudoersPath + ".new"
|
||||
if os.WriteFile(tmp, []byte(text), 0o440) != nil {
|
||||
return
|
||||
}
|
||||
if exec.Command("visudo", "-cf", tmp).Run() != nil {
|
||||
os.Remove(tmp)
|
||||
say("warning: sudoers snippet failed validation; skipped")
|
||||
return
|
||||
}
|
||||
_ = os.Rename(tmp, sudoersPath)
|
||||
}
|
||||
|
||||
// firewall reports (and with open, fixes) an active host firewall in front of a
|
||||
// non-loopback listen address. It returns the port.
|
||||
func firewall(listen string, open bool) string {
|
||||
host, port, _ := net.SplitHostPort(listen)
|
||||
if isLoopback(host) {
|
||||
return port
|
||||
}
|
||||
switch {
|
||||
case have("firewall-cmd") && exec.Command("firewall-cmd", "--state").Run() == nil:
|
||||
if open {
|
||||
if err := runAll([][]string{{"firewall-cmd", "--quiet", "--permanent", "--add-port=" + port + "/tcp"}, {"firewall-cmd", "--quiet", "--reload"}}); err != nil {
|
||||
say("warning: firewalld: %v", err)
|
||||
} else {
|
||||
say("firewall firewalld: opened %s/tcp", port)
|
||||
}
|
||||
} else {
|
||||
say("firewall firewalld is active and will block port %s. Open it: hamlab install --open-firewall", port)
|
||||
}
|
||||
case have("ufw"):
|
||||
out, _ := exec.Command("ufw", "status").Output()
|
||||
if !bytes.Contains(out, []byte("Status: active")) {
|
||||
break
|
||||
}
|
||||
if open {
|
||||
if err := run("ufw", "allow", port+"/tcp"); err != nil {
|
||||
say("warning: ufw: %v", err)
|
||||
} else {
|
||||
say("firewall ufw: opened %s/tcp", port)
|
||||
}
|
||||
} else {
|
||||
say("firewall ufw is active and will block port %s. Open it: hamlab install --open-firewall", port)
|
||||
}
|
||||
}
|
||||
return port
|
||||
}
|
||||
|
||||
func isLoopback(host string) bool {
|
||||
ip := net.ParseIP(host)
|
||||
return host == "localhost" || (ip != nil && ip.IsLoopback())
|
||||
}
|
||||
|
||||
// browseAddr turns a listen address into one a browser can open.
|
||||
func browseAddr(listen string) string {
|
||||
host, port, err := net.SplitHostPort(listen)
|
||||
if err != nil {
|
||||
return listen
|
||||
}
|
||||
if ip := net.ParseIP(host); host == "" || (ip != nil && ip.IsUnspecified()) {
|
||||
host = "localhost"
|
||||
if addrs, err := net.InterfaceAddrs(); err == nil {
|
||||
for _, a := range addrs {
|
||||
if n, ok := a.(*net.IPNet); ok && n.IP.To4() != nil && !n.IP.IsLoopback() {
|
||||
host = n.IP.String()
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return net.JoinHostPort(host, port)
|
||||
}
|
||||
|
||||
// healthy polls /healthz for up to 10 s.
|
||||
func healthy(listen string, https bool) bool {
|
||||
host, port, _ := net.SplitHostPort(listen)
|
||||
if ip := net.ParseIP(host); host == "" || (ip != nil && ip.IsUnspecified()) {
|
||||
host = "127.0.0.1"
|
||||
}
|
||||
scheme := "http"
|
||||
if https {
|
||||
scheme = "https"
|
||||
}
|
||||
c := &http.Client{Timeout: 2 * time.Second, Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}} // self-signed; reachability only
|
||||
for i := 0; i < 20; i++ {
|
||||
if resp, err := c.Get(scheme + "://" + net.JoinHostPort(host, port) + "/healthz"); err == nil {
|
||||
resp.Body.Close()
|
||||
return resp.StatusCode < 500
|
||||
}
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// restorecon fixes the SELinux label of a file we created, where SELinux exists.
|
||||
func restorecon(path string) {
|
||||
if have("restorecon") {
|
||||
_ = exec.Command("restorecon", "-F", path).Run()
|
||||
}
|
||||
}
|
||||
|
||||
func run(args ...string) error {
|
||||
out, err := exec.Command(args[0], args[1:]...).CombinedOutput()
|
||||
if err != nil {
|
||||
if msg := strings.TrimSpace(string(out)); msg != "" {
|
||||
return fmt.Errorf("%s: %s", strings.Join(args, " "), msg)
|
||||
}
|
||||
return fmt.Errorf("%s: %w", strings.Join(args, " "), err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func runAll(cmds [][]string) error {
|
||||
for _, c := range cmds {
|
||||
if err := run(c...); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func have(bin string) bool { _, err := exec.LookPath(bin); return err == nil }
|
||||
func exists(p string) bool { _, err := os.Stat(p); return err == nil }
|
||||
func isDir(p string) bool { st, err := os.Stat(p); return err == nil && st.IsDir() }
|
||||
@@ -0,0 +1,73 @@
|
||||
package setup
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestUnitWithoutJournalGroups(t *testing.T) {
|
||||
vars := map[string]string{"@USER@": "ham", "@GROUP@": "ham", "@HOME@": "/home/ham", "@BIN@": "/usr/local/bin/hamlab",
|
||||
"@CONFIG@": "/home/ham/.config/hamlab/config.json", "@GROUPS@": ""}
|
||||
unit := dropLines(fill(systemdUnit, vars), "SupplementaryGroups=")
|
||||
if strings.Contains(unit, "SupplementaryGroups") || strings.Contains(unit, "@") {
|
||||
t.Fatalf("unit still has a groups line or an unfilled placeholder:\n%s", unit)
|
||||
}
|
||||
if !strings.Contains(unit, "ExecStart=/usr/local/bin/hamlab serve") {
|
||||
t.Fatalf("ExecStart missing:\n%s", unit)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigDirFrom(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"Environment=HAMLAB_CONFIG=/home/ham/.config/hamlab/config.json": "/home/ham/.config/hamlab",
|
||||
`export HAMLAB_CONFIG="/root/.config/hamlab/config.json"`: "/root/.config/hamlab",
|
||||
"Environment=HAMLAB_CONFIG=/config.json": "",
|
||||
"Environment=HAMLAB_CONFIG=/etc/passwd": "",
|
||||
} {
|
||||
key := "Environment=HAMLAB_CONFIG="
|
||||
if strings.HasPrefix(in, "export") {
|
||||
key = "export HAMLAB_CONFIG="
|
||||
}
|
||||
if got := configDirFrom(in, key); got != want {
|
||||
t.Errorf("%q: got %q want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --purge must never take a shared directory with it.
|
||||
func TestPurgeSparesSharedDirectory(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
os.WriteFile(filepath.Join(home, "config.json"), []byte("{}"), 0o600)
|
||||
os.WriteFile(filepath.Join(home, "thesis.txt"), []byte("irreplaceable"), 0o600)
|
||||
if err := purgeData(home); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(home, "thesis.txt")); err != nil {
|
||||
t.Fatal("purge deleted a file that wasn't Ham Lab's")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(home, "config.json")); err == nil {
|
||||
t.Fatal("purge left config.json behind")
|
||||
}
|
||||
|
||||
own := filepath.Join(t.TempDir(), "hamlab")
|
||||
os.MkdirAll(own, 0o700)
|
||||
os.WriteFile(filepath.Join(own, "config.json"), []byte("{}"), 0o600)
|
||||
if err := purgeData(own); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(own); err == nil {
|
||||
t.Fatal("purge left Ham Lab's own directory behind")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFirstMissing(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
if got := firstMissing(filepath.Join(base, "a", "b", "c")); got != filepath.Join(base, "a") {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
if got := firstMissing(base); got != "" {
|
||||
t.Errorf("existing dir: got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
package space
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"io"
|
||||
)
|
||||
|
||||
// The tree crosses from the helper process to the daemon as a preorder stream of varints:
|
||||
// a few bytes per node, decoded without reflection, so a million nodes take well under a
|
||||
// second either way.
|
||||
|
||||
const codecMagic = "HLSPACE2"
|
||||
|
||||
func Encode(w io.Writer, root *Node) error {
|
||||
bw := bufio.NewWriterSize(w, 1<<20)
|
||||
if _, err := bw.WriteString(codecMagic); err != nil {
|
||||
return err
|
||||
}
|
||||
var buf [binary.MaxVarintLen64]byte
|
||||
putU := func(v uint64) error { n := binary.PutUvarint(buf[:], v); _, err := bw.Write(buf[:n]); return err }
|
||||
putI := func(v int64) error { n := binary.PutVarint(buf[:], v); _, err := bw.Write(buf[:n]); return err }
|
||||
var enc func(n *Node) error
|
||||
enc = func(n *Node) error {
|
||||
if err := putU(uint64(len(n.Name))); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := bw.WriteString(n.Name); err != nil {
|
||||
return err
|
||||
}
|
||||
flags := uint64(0)
|
||||
if n.Dir {
|
||||
flags |= 1
|
||||
}
|
||||
if n.Mount {
|
||||
flags |= 2
|
||||
}
|
||||
if n.Denied {
|
||||
flags |= 4
|
||||
}
|
||||
if n.Partial {
|
||||
flags |= 8
|
||||
}
|
||||
for _, f := range []func() error{
|
||||
func() error { return putU(flags) },
|
||||
func() error { return putI(n.Size) },
|
||||
func() error { return putI(n.Apparent) },
|
||||
func() error { return putI(n.MTime) },
|
||||
func() error { return putI(n.MTimeNs) },
|
||||
func() error { return putU(uint64(n.Files)) },
|
||||
func() error { return putU(uint64(n.Dirs)) },
|
||||
func() error { return putU(uint64(n.SmallN)) },
|
||||
func() error { return putI(n.SmallSize) },
|
||||
func() error { return putU(uint64(len(n.Children))) },
|
||||
} {
|
||||
if err := f(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for _, c := range n.Children {
|
||||
if err := enc(c); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if err := enc(root); err != nil {
|
||||
return err
|
||||
}
|
||||
return bw.Flush()
|
||||
}
|
||||
|
||||
func Decode(r io.Reader) (*Node, error) {
|
||||
br := bufio.NewReaderSize(r, 1<<20)
|
||||
magic := make([]byte, len(codecMagic))
|
||||
if _, err := io.ReadFull(br, magic); err != nil || string(magic) != codecMagic {
|
||||
return nil, errors.New("space: not a scan stream")
|
||||
}
|
||||
var dec func() (*Node, error)
|
||||
dec = func() (*Node, error) {
|
||||
l, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if l > 1<<16 {
|
||||
return nil, errors.New("space: bad name length")
|
||||
}
|
||||
name := make([]byte, l)
|
||||
if _, err := io.ReadFull(br, name); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n := &Node{Name: string(name)}
|
||||
flags, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Dir, n.Mount, n.Denied, n.Partial = flags&1 != 0, flags&2 != 0, flags&4 != 0, flags&8 != 0
|
||||
if n.Size, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.Apparent, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.MTime, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.MTimeNs, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
u, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Files = uint32(u)
|
||||
if u, err = binary.ReadUvarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Dirs = uint32(u)
|
||||
if u, err = binary.ReadUvarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.SmallN = uint32(u)
|
||||
if n.SmallSize, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cnt, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cnt > 1<<24 {
|
||||
return nil, errors.New("space: bad child count")
|
||||
}
|
||||
if cnt > 0 {
|
||||
n.Children = make([]*Node, 0, cnt)
|
||||
for i := uint64(0); i < cnt; i++ {
|
||||
c, err := dec()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Children = append(n.Children, c)
|
||||
}
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
return dec()
|
||||
}
|
||||
@@ -0,0 +1,735 @@
|
||||
package space
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// ---- the helper: `hamlab space-walk <root>…`, run by the daemon (as root when it can) ----
|
||||
//
|
||||
// The walk runs in its own process so that root can read every directory, so that the
|
||||
// memory of the walk is given back when it ends, and so that a runaway scan can be killed
|
||||
// without touching the console. The daemon writes a header line to the helper's stdin
|
||||
// (how each root is to be walked, and what changed) followed by the last scan's tree when
|
||||
// there is one, then holds stdin open: EOF means the daemon has gone or wants the walk
|
||||
// stopped. Progress goes to stderr as JSON lines; a result line and the tree to stdout.
|
||||
|
||||
type header struct {
|
||||
Prev bool `json:"prev"`
|
||||
Modes map[string]Mode `json:"modes,omitempty"`
|
||||
Dirty []string `json:"dirty,omitempty"`
|
||||
Touched []string `json:"touched,omitempty"`
|
||||
}
|
||||
|
||||
type resultLine struct {
|
||||
Complete bool `json:"complete"`
|
||||
Partial int64 `json:"partial"`
|
||||
Reused int64 `json:"reused"`
|
||||
Read int64 `json:"read"`
|
||||
}
|
||||
|
||||
func Helper(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
|
||||
if len(args) == 0 {
|
||||
fmt.Fprintln(stderr, "usage: hamlab space-walk <mountpoint>...")
|
||||
return 2
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
sig := make(chan os.Signal, 1)
|
||||
signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP)
|
||||
go func() { <-sig; cancel() }()
|
||||
in := bufio.NewReaderSize(stdin, 1<<20)
|
||||
line, err := in.ReadBytes('\n')
|
||||
if err != nil {
|
||||
fmt.Fprintln(stderr, "error: no header:", err)
|
||||
return 2
|
||||
}
|
||||
var h header
|
||||
if err := json.Unmarshal(line, &h); err != nil {
|
||||
fmt.Fprintln(stderr, "error: bad header:", err)
|
||||
return 2
|
||||
}
|
||||
opts := Options{Modes: h.Modes, Dirty: set(h.Dirty), Touched: set(h.Touched)}
|
||||
if h.Prev {
|
||||
if opts.Prev, err = Decode(in); err != nil {
|
||||
fmt.Fprintln(stderr, "error: last tree:", err)
|
||||
return 2
|
||||
}
|
||||
}
|
||||
go func() { _, _ = io.Copy(io.Discard, in); cancel() }() // the daemon went away, or asked us to stop
|
||||
enc := json.NewEncoder(stderr)
|
||||
res, err := Walk(ctx, args, opts, func(p Progress) { _ = enc.Encode(p) })
|
||||
if err != nil && !errors.Is(err, context.Canceled) {
|
||||
fmt.Fprintln(stderr, "error:", err)
|
||||
return 1
|
||||
}
|
||||
rl, _ := json.Marshal(resultLine{Complete: res.Complete, Partial: res.Partial, Reused: res.Reused, Read: res.Read})
|
||||
if _, err := stdout.Write(append(rl, '\n')); err != nil {
|
||||
return 1
|
||||
}
|
||||
if err := Encode(stdout, res.Root); err != nil {
|
||||
fmt.Fprintln(stderr, "error:", err)
|
||||
return 1
|
||||
}
|
||||
if !res.Complete {
|
||||
return 3
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func set(list []string) map[string]bool {
|
||||
if len(list) == 0 {
|
||||
return nil
|
||||
}
|
||||
m := make(map[string]bool, len(list))
|
||||
for _, s := range list {
|
||||
m[s] = true
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ---- the daemon side: one scan per target, the last result kept, on disk too ----
|
||||
|
||||
type Scan struct {
|
||||
Disk string `json:"disk"`
|
||||
Roots []string `json:"roots"`
|
||||
Started time.Time `json:"started"`
|
||||
Finished time.Time `json:"finished,omitempty"`
|
||||
Running bool `json:"running"`
|
||||
Loading bool `json:"loading,omitempty"` // read back from disk after a restart, not yet decoded
|
||||
Error string `json:"error,omitempty"`
|
||||
Progress Progress `json:"progress"`
|
||||
AsRoot bool `json:"as_root"`
|
||||
// How this scan was done: "full", or "refresh" / "resume" from the one before, and by what
|
||||
// means per root ("zfs diff" or "mtime").
|
||||
Method string `json:"method,omitempty"`
|
||||
ByRoot map[string]string `json:"by_root,omitempty"`
|
||||
Partial bool `json:"partial,omitempty"` // cancelled before it finished; Scan again resumes it
|
||||
// once finished:
|
||||
Files uint32 `json:"files"`
|
||||
Dirs uint32 `json:"dirs"`
|
||||
Size int64 `json:"size"`
|
||||
Denied int64 `json:"denied"`
|
||||
Read int64 `json:"read"` // directories read this time
|
||||
Reused int64 `json:"reused"` // directories kept from the scan before
|
||||
PartialDirs int64 `json:"partial_dirs"` // directories not read (cancelled)
|
||||
// Bases is, per root that is a ZFS dataset, the newest snapshot that existed when the
|
||||
// scan started: the next scan asks `zfs diff` what changed since it.
|
||||
Bases map[string]string `json:"bases,omitempty"`
|
||||
|
||||
root *Node
|
||||
largest []File
|
||||
cancel context.CancelFunc
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
self string // this binary, for the helper
|
||||
dir string // where results are kept between restarts
|
||||
mu sync.Mutex
|
||||
scans map[string]*Scan
|
||||
}
|
||||
|
||||
// New makes a manager that keeps results under dir and reads back the ones there.
|
||||
func New(self, dir string) *Manager {
|
||||
m := &Manager{self: self, dir: dir, scans: map[string]*Scan{}}
|
||||
m.loadAll()
|
||||
return m
|
||||
}
|
||||
|
||||
func (m *Manager) fileFor(id, ext string) string {
|
||||
safe := strings.NewReplacer(":", "_", "/", "_", "\\", "_").Replace(id)
|
||||
return filepath.Join(m.dir, safe+ext)
|
||||
}
|
||||
|
||||
// loadAll reads the saved scans' metadata now and their trees in the background, one at a
|
||||
// time, so a restart does not double the memory of the last scans all at once.
|
||||
func (m *Manager) loadAll() {
|
||||
metas, _ := filepath.Glob(filepath.Join(m.dir, "*.json"))
|
||||
var pending []*Scan
|
||||
for _, mp := range metas {
|
||||
b, err := os.ReadFile(mp)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
s := &Scan{}
|
||||
if json.Unmarshal(b, s) != nil || s.Disk == "" {
|
||||
continue
|
||||
}
|
||||
s.Running, s.cancel = false, nil
|
||||
s.Loading = true
|
||||
m.scans[s.Disk] = s
|
||||
pending = append(pending, s)
|
||||
}
|
||||
if len(pending) == 0 {
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
for _, s := range pending {
|
||||
root, err := m.readTree(s.Disk)
|
||||
m.mu.Lock()
|
||||
if err != nil {
|
||||
log.Printf("space: saved scan of %s unreadable (%v); dropped", s.Disk, err)
|
||||
if m.scans[s.Disk] == s {
|
||||
delete(m.scans, s.Disk)
|
||||
}
|
||||
} else if m.scans[s.Disk] == s {
|
||||
s.root, s.Loading = root, false
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func (m *Manager) readTree(id string) (*Node, error) {
|
||||
f, err := os.Open(m.fileFor(id, ".scan"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
br := bufio.NewReaderSize(f, 1<<20)
|
||||
if _, err := br.ReadBytes('\n'); err != nil { // the result line
|
||||
return nil, err
|
||||
}
|
||||
return Decode(br)
|
||||
}
|
||||
|
||||
func (m *Manager) saveMeta(s *Scan) {
|
||||
if m.dir == "" {
|
||||
return
|
||||
}
|
||||
_ = os.MkdirAll(m.dir, 0o700)
|
||||
b, err := json.MarshalIndent(s, "", " ")
|
||||
if err == nil {
|
||||
err = writeAtomic(m.fileFor(s.Disk, ".json"), b)
|
||||
}
|
||||
if err != nil {
|
||||
log.Printf("space: save %s: %v", s.Disk, err)
|
||||
}
|
||||
}
|
||||
|
||||
func writeAtomic(path string, b []byte) error {
|
||||
tmp := path + ".tmp"
|
||||
if err := os.WriteFile(tmp, b, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// Status is the scan of id (a copy without the tree), or nil.
|
||||
func (m *Manager) Status(id string) *Scan {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.copyOf(m.scans[id])
|
||||
}
|
||||
|
||||
func (m *Manager) copyOf(s *Scan) *Scan {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
c := *s
|
||||
c.root, c.largest, c.cancel = nil, nil, nil
|
||||
return &c
|
||||
}
|
||||
|
||||
func (m *Manager) All() map[string]*Scan {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
out := map[string]*Scan{}
|
||||
for k, s := range m.scans {
|
||||
out[k] = m.copyOf(s)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Start scans the roots of id in the background, from the last scan of it when there is
|
||||
// one. datasets names the ZFS dataset behind each root that has one.
|
||||
func (m *Manager) Start(id string, roots []string, datasets map[string]string) (*Scan, error) {
|
||||
if len(roots) == 0 {
|
||||
return nil, errors.New("nothing is mounted from this disk")
|
||||
}
|
||||
m.mu.Lock()
|
||||
prev := m.scans[id]
|
||||
if prev != nil && prev.Running {
|
||||
m.mu.Unlock()
|
||||
return nil, errors.New("a scan of this disk is already running")
|
||||
}
|
||||
if prev != nil && (prev.root == nil || prev.Loading) {
|
||||
prev = nil // nothing to start from
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s := &Scan{Disk: id, Roots: roots, Started: time.Now(), Running: true, AsRoot: priv.Available(), cancel: cancel, Bases: map[string]string{}}
|
||||
if prev != nil {
|
||||
// the new scan takes the old one's place in the list while it runs, but keeps its
|
||||
// tree, so the pages still answer until the new one is in
|
||||
s.root, s.largest = prev.root, prev.largest
|
||||
s.Files, s.Dirs, s.Size = prev.Files, prev.Dirs, prev.Size
|
||||
}
|
||||
m.scans[id] = s
|
||||
m.mu.Unlock()
|
||||
go m.run(ctx, s, prev, datasets)
|
||||
return m.Status(id), nil
|
||||
}
|
||||
|
||||
// plan decides how each root is walked this time, given the last scan.
|
||||
func (m *Manager) plan(ctx context.Context, s, prev *Scan, datasets map[string]string) (header, Options) {
|
||||
h := header{Modes: map[string]Mode{}}
|
||||
opts := Options{Modes: h.Modes}
|
||||
if prev == nil || prev.root == nil {
|
||||
s.Method = "full"
|
||||
return h, opts
|
||||
}
|
||||
h.Prev = true
|
||||
opts.Prev = prev.root
|
||||
s.ByRoot = map[string]string{}
|
||||
dirty, touched := map[string]bool{}, map[string]bool{}
|
||||
// every dataset is asked at once what changed; the answers come back in root order
|
||||
type diffed struct {
|
||||
changed []change
|
||||
err error
|
||||
took time.Duration
|
||||
}
|
||||
answers := map[string]chan diffed{}
|
||||
for _, r := range s.Roots {
|
||||
if base := prev.Bases[r]; base != "" && datasets[r] != "" {
|
||||
ch := make(chan diffed, 1)
|
||||
answers[r] = ch
|
||||
go func(r, base string) {
|
||||
t0 := time.Now()
|
||||
c, err := zfsDiff(ctx, base, r)
|
||||
ch <- diffed{changed: c, err: err, took: time.Since(t0)}
|
||||
}(r, base)
|
||||
}
|
||||
}
|
||||
for _, r := range s.Roots {
|
||||
mode := ModeMTime
|
||||
how := "mtime"
|
||||
if ch, ok := answers[r]; ok {
|
||||
m.mu.Lock()
|
||||
s.Progress.Current = "asking zfs what changed in " + r
|
||||
m.mu.Unlock()
|
||||
d := <-ch
|
||||
if d.err == nil {
|
||||
mode, how = ModeDiff, fmt.Sprintf("zfs diff since %s: %d changes in %s", strings.TrimPrefix(prev.Bases[r], datasets[r]), len(d.changed), d.took.Round(time.Millisecond))
|
||||
for _, c := range d.changed {
|
||||
dirty[c.parent] = true
|
||||
if c.dir {
|
||||
dirty[c.path] = true
|
||||
}
|
||||
}
|
||||
} else {
|
||||
how = "mtime (zfs diff: " + d.err.Error() + ")"
|
||||
}
|
||||
}
|
||||
if Find(prev.root, r) == nil {
|
||||
mode, how = ModeFull, "full (new root)"
|
||||
}
|
||||
h.Modes[r] = mode
|
||||
s.ByRoot[r] = how
|
||||
}
|
||||
m.mu.Lock()
|
||||
s.Progress.Current = ""
|
||||
m.mu.Unlock()
|
||||
// what the last walk never got to is read whatever the change list says
|
||||
for _, p := range partialPaths(prev.root, "/") {
|
||||
dirty[p] = true
|
||||
}
|
||||
for p := range dirty {
|
||||
for a := filepath.Dir(p); ; a = filepath.Dir(a) {
|
||||
touched[a] = true
|
||||
if a == "/" {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
for p := range dirty {
|
||||
h.Dirty = append(h.Dirty, p)
|
||||
}
|
||||
for p := range touched {
|
||||
h.Touched = append(h.Touched, p)
|
||||
}
|
||||
sort.Strings(h.Dirty)
|
||||
sort.Strings(h.Touched)
|
||||
opts.Dirty, opts.Touched = dirty, touched
|
||||
if prev.Partial {
|
||||
s.Method = "resume"
|
||||
} else {
|
||||
s.Method = "refresh"
|
||||
}
|
||||
return h, opts
|
||||
}
|
||||
|
||||
func partialPaths(n *Node, path string) []string {
|
||||
var out []string
|
||||
var walk func(n *Node, path string)
|
||||
walk = func(n *Node, path string) {
|
||||
if n.Partial {
|
||||
out = append(out, path)
|
||||
}
|
||||
for _, c := range n.Children {
|
||||
if c.Dir {
|
||||
walk(c, join(path, c.Name))
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(n, path)
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) run(ctx context.Context, s *Scan, prev *Scan, datasets map[string]string) {
|
||||
finish := func(err error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
s.Running = false
|
||||
s.Finished = time.Now()
|
||||
if err != nil {
|
||||
s.Error = err.Error()
|
||||
}
|
||||
m.saveMeta(s)
|
||||
}
|
||||
// the newest snapshot of each dataset, so the next scan can ask what changed since
|
||||
for _, r := range s.Roots {
|
||||
if ds := datasets[r]; ds != "" {
|
||||
if snap := newestSnapshot(ctx, ds); snap != "" {
|
||||
s.Bases[r] = snap
|
||||
}
|
||||
}
|
||||
}
|
||||
h, _ := m.plan(ctx, s, prev, datasets)
|
||||
if ctx.Err() != nil { // cancelled while zfs was being asked
|
||||
finish(errors.New("cancelled"))
|
||||
return
|
||||
}
|
||||
args := append([]string{m.self, "space-walk"}, s.Roots...)
|
||||
if s.AsRoot {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
cmd := exec.Command(args[0], args[1:]...)
|
||||
stdin, _ := cmd.StdinPipe() // held open: closing it tells the helper to stop
|
||||
stdout, _ := cmd.StdoutPipe()
|
||||
stderr, _ := cmd.StderrPipe()
|
||||
if err := cmd.Start(); err != nil {
|
||||
finish(err)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
hb, _ := json.Marshal(h)
|
||||
_, err := stdin.Write(append(hb, '\n'))
|
||||
if err == nil && h.Prev {
|
||||
err = Encode(stdin, prev.root)
|
||||
}
|
||||
if err != nil {
|
||||
log.Printf("space: %s: sending the last tree: %v", s.Disk, err)
|
||||
}
|
||||
<-ctx.Done()
|
||||
_ = cmd.Process.Signal(syscall.SIGTERM)
|
||||
stdin.Close()
|
||||
}()
|
||||
var lastErr string
|
||||
go func() {
|
||||
sc := bufio.NewScanner(stderr)
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
var p Progress
|
||||
if strings.HasPrefix(line, "{") && json.Unmarshal([]byte(line), &p) == nil {
|
||||
m.mu.Lock()
|
||||
s.Progress = p
|
||||
m.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
if line != "" {
|
||||
lastErr = line
|
||||
}
|
||||
}
|
||||
}()
|
||||
// the result streams to disk as it is decoded, so a restart finds it
|
||||
_ = os.MkdirAll(m.dir, 0o700)
|
||||
tmp := m.fileFor(s.Disk, ".scan.tmp")
|
||||
out, ferr := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
|
||||
var src io.Reader = stdout
|
||||
if ferr == nil {
|
||||
src = io.TeeReader(stdout, out)
|
||||
}
|
||||
br := bufio.NewReaderSize(src, 1<<20)
|
||||
var rl resultLine
|
||||
line, rerr := br.ReadBytes('\n')
|
||||
if rerr == nil {
|
||||
rerr = json.Unmarshal(line, &rl)
|
||||
}
|
||||
var root *Node
|
||||
var decErr error
|
||||
if rerr == nil {
|
||||
root, decErr = Decode(br)
|
||||
} else {
|
||||
decErr = rerr
|
||||
_, _ = io.Copy(io.Discard, br)
|
||||
}
|
||||
if ferr == nil {
|
||||
out.Close()
|
||||
}
|
||||
waitErr := cmd.Wait()
|
||||
cancelled := ctx.Err() != nil
|
||||
if root == nil {
|
||||
_ = os.Remove(tmp)
|
||||
switch {
|
||||
case cancelled:
|
||||
finish(errors.New("cancelled"))
|
||||
case waitErr != nil && lastErr != "":
|
||||
finish(errors.New(strings.TrimPrefix(lastErr, "error: ")))
|
||||
case waitErr != nil:
|
||||
finish(waitErr)
|
||||
default:
|
||||
finish(decErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if ferr == nil {
|
||||
_ = os.Rename(tmp, m.fileFor(s.Disk, ".scan"))
|
||||
}
|
||||
m.mu.Lock()
|
||||
s.root, s.largest = root, nil
|
||||
s.Files, s.Dirs, s.Size, s.Denied = root.Files, root.Dirs, root.Size, s.Progress.Denied
|
||||
s.Read, s.Reused, s.PartialDirs = rl.Read, rl.Reused, rl.Partial
|
||||
s.Partial = !rl.Complete
|
||||
s.Running = false
|
||||
s.Finished = time.Now()
|
||||
if s.Partial {
|
||||
s.Error = "cancelled"
|
||||
}
|
||||
m.saveMeta(s)
|
||||
m.mu.Unlock()
|
||||
what := "scanned"
|
||||
if s.Partial {
|
||||
what = "partly scanned (cancelled)"
|
||||
}
|
||||
log.Printf("space: %s %s: %d files, %d dirs, %d bytes in %s (%s; %d dirs read, %d kept%s)", s.Disk, what, root.Files, root.Dirs, root.Size, s.Finished.Sub(s.Started).Round(time.Millisecond), s.Method, rl.Read, rl.Reused, map[bool]string{true: ", as root", false: ""}[s.AsRoot])
|
||||
}
|
||||
|
||||
func (m *Manager) Cancel(id string) bool {
|
||||
m.mu.Lock()
|
||||
s := m.scans[id]
|
||||
m.mu.Unlock()
|
||||
if s == nil || !s.Running || s.cancel == nil {
|
||||
return false
|
||||
}
|
||||
s.cancel()
|
||||
return true
|
||||
}
|
||||
|
||||
// Forget drops the scan of id, in memory and on disk.
|
||||
func (m *Manager) Forget(id string) {
|
||||
m.mu.Lock()
|
||||
if s := m.scans[id]; s != nil && s.cancel != nil {
|
||||
s.cancel()
|
||||
}
|
||||
delete(m.scans, id)
|
||||
m.mu.Unlock()
|
||||
_ = os.Remove(m.fileFor(id, ".scan"))
|
||||
_ = os.Remove(m.fileFor(id, ".json"))
|
||||
}
|
||||
|
||||
// ---- ZFS: what changed since the last scan ----
|
||||
|
||||
type change struct {
|
||||
path, parent string
|
||||
dir bool
|
||||
}
|
||||
|
||||
// newestSnapshot is the most recent snapshot of a dataset, "" when it has none.
|
||||
func newestSnapshot(ctx context.Context, dataset string) string {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-t", "snapshot", "-o", "name", "-s", "creation", "-d", "1", dataset).Output()
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
last := lines[len(lines)-1]
|
||||
if !strings.Contains(last, "@") {
|
||||
return ""
|
||||
}
|
||||
return last
|
||||
}
|
||||
|
||||
// zfsDiff lists what changed in a dataset since a snapshot: every changed, added, removed
|
||||
// or renamed path, with the directory whose listing that touches.
|
||||
func zfsDiff(ctx context.Context, snapshot, mount string) ([]change, error) {
|
||||
// A diff slower than this is not worth waiting for: the walk by mtime is bounded. The
|
||||
// clock runs inside sudo — `timeout` is zfs's parent and kills it as root — because sudo
|
||||
// on Debian gives the command a session of its own, out of reach of our process group;
|
||||
// the context is only a backstop.
|
||||
const patience = 3 * time.Minute
|
||||
ctx, cancel := context.WithTimeout(ctx, patience+30*time.Second)
|
||||
defer cancel()
|
||||
args := priv.Wrap("timeout", "-s", "KILL", strconv.Itoa(int(patience.Seconds())), "zfs", "diff", "-H", "-F", snapshot)
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
|
||||
cmd.WaitDelay = 5 * time.Second
|
||||
t0 := time.Now()
|
||||
var stderr strings.Builder
|
||||
cmd.Stderr = &stderr
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var changes []change
|
||||
sc := bufio.NewScanner(out)
|
||||
sc.Buffer(make([]byte, 64<<10), 4<<20)
|
||||
for sc.Scan() {
|
||||
f := strings.Split(sc.Text(), "\t")
|
||||
if len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
isDir := f[1] == "/"
|
||||
for _, p := range f[2:] { // a rename has the old and the new path
|
||||
p = unescape(p)
|
||||
if !strings.HasPrefix(p, mount) {
|
||||
continue
|
||||
}
|
||||
changes = append(changes, change{path: p, parent: filepath.Dir(p), dir: isDir})
|
||||
}
|
||||
}
|
||||
if err := cmd.Wait(); err != nil {
|
||||
msg := strings.TrimSpace(stderr.String())
|
||||
if errors.Is(ctx.Err(), context.DeadlineExceeded) || time.Since(t0) >= patience {
|
||||
msg = "gave up after " + patience.String()
|
||||
}
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
if i := strings.IndexByte(msg, '\n'); i > 0 {
|
||||
msg = msg[:i]
|
||||
}
|
||||
return nil, errors.New(msg)
|
||||
}
|
||||
return changes, nil
|
||||
}
|
||||
|
||||
// unescape undoes zfs diff's \ooo escapes for spaces and odd bytes in names.
|
||||
func unescape(s string) string {
|
||||
if !strings.Contains(s, "\\") {
|
||||
return s
|
||||
}
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(s); i++ {
|
||||
if s[i] == '\\' && i+3 < len(s) {
|
||||
if v, err := strconv.ParseUint(s[i+1:i+4], 8, 8); err == nil {
|
||||
b.WriteByte(byte(v))
|
||||
i += 3
|
||||
continue
|
||||
}
|
||||
}
|
||||
b.WriteByte(s[i])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// Tree is the subtree at path, cut to depth levels of at most limit children each, leaving
|
||||
// out children smaller than min bytes; what is left out is summed into MoreN/MoreSize. A
|
||||
// treemap asks deep with a min the size of the smallest tile it could draw; a folder list
|
||||
// asks one level with no min.
|
||||
type Tree struct {
|
||||
Node
|
||||
Path string `json:"path"`
|
||||
Children []*Tree `json:"children,omitempty"`
|
||||
MoreN int `json:"more_n,omitempty"`
|
||||
MoreSize int64 `json:"more_size,omitempty"`
|
||||
Total int64 `json:"total"` // the whole scan's size, for percentages
|
||||
}
|
||||
|
||||
func (m *Manager) Tree(id, path string, depth, limit int, min int64) (*Tree, error) {
|
||||
m.mu.Lock()
|
||||
s := m.scans[id]
|
||||
var root *Node
|
||||
if s != nil {
|
||||
root = s.root
|
||||
}
|
||||
m.mu.Unlock()
|
||||
if root == nil {
|
||||
if s != nil && s.Loading {
|
||||
return nil, errors.New("the last scan is still being read back")
|
||||
}
|
||||
return nil, errors.New("no scan of this disk yet")
|
||||
}
|
||||
n := Find(root, path)
|
||||
if n == nil {
|
||||
return nil, errors.New("no such path in the scan")
|
||||
}
|
||||
if path == "" {
|
||||
path = "/"
|
||||
}
|
||||
t := cut(n, path, depth, limit, min)
|
||||
t.Total = root.Size
|
||||
return t, nil
|
||||
}
|
||||
|
||||
func cut(n *Node, path string, depth, limit int, min int64) *Tree {
|
||||
t := &Tree{Node: *n, Path: path}
|
||||
t.Node.Children = nil
|
||||
if !n.Dir {
|
||||
return t
|
||||
}
|
||||
if depth <= 0 { // as deep as was asked: what is inside is one lump, not nothing
|
||||
if len(n.Children) > 0 {
|
||||
t.MoreN, t.MoreSize = len(n.Children), n.Size-n.SmallSize
|
||||
}
|
||||
return t
|
||||
}
|
||||
for i, c := range n.Children {
|
||||
if i >= limit || (c.Size < min && i > 0) { // children are sorted largest first: the first is always kept
|
||||
t.MoreN++
|
||||
t.MoreSize += c.Size
|
||||
continue
|
||||
}
|
||||
t.Children = append(t.Children, cut(c, join(path, c.Name), depth-1, limit, min))
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// Largest is the n biggest files of the last scan of id.
|
||||
func (m *Manager) Largest(id string, n int) ([]File, error) {
|
||||
m.mu.Lock()
|
||||
s := m.scans[id]
|
||||
var root *Node
|
||||
var have []File
|
||||
if s != nil {
|
||||
root, have = s.root, s.largest
|
||||
}
|
||||
m.mu.Unlock()
|
||||
if root == nil {
|
||||
return nil, errors.New("no scan of this disk yet")
|
||||
}
|
||||
if len(have) < n {
|
||||
have = Largest(root, n) // the tree is never changed once made, so no lock while walking it
|
||||
m.mu.Lock()
|
||||
if s.root == root {
|
||||
s.largest = have
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
if len(have) > n {
|
||||
return have[:n], nil
|
||||
}
|
||||
return have, nil
|
||||
}
|
||||
@@ -0,0 +1,516 @@
|
||||
// Package space measures what is using a disk — it is what the UI calls the **Usage** tab
|
||||
// (renamed from "Space" on 2026-09-24; the package and its /api/space endpoints kept the
|
||||
// old name rather than churn a working API): a parallel walk of the filesystems a disk
|
||||
// backs, kept as a tree of directories and the files worth seeing, the way WizTree does it.
|
||||
//
|
||||
// The walk never crosses a filesystem boundary by itself — /proc, /sys, another disk's
|
||||
// mount — but every mountpoint the disk backs is a root of the same scan, and the roots
|
||||
// are grafted into one tree at their paths, so a ZFS pool of many datasets, or a disk with
|
||||
// / and /home on separate partitions, reads as one picture.
|
||||
//
|
||||
// Files below KeepMin are not kept one by one: a directory of a million small files is
|
||||
// one "smaller files" entry with their count and total, which is all a treemap can show
|
||||
// of them anyway, and what keeps a scan of a few million files within a few hundred MB.
|
||||
//
|
||||
// A walk can start from the last one. Given the previous tree, a directory is read again
|
||||
// only when something says it changed: on ZFS, `zfs diff` against a snapshot older than
|
||||
// the last scan names every path that changed, and directories it does not name are kept
|
||||
// as they were without so much as a stat; elsewhere a directory whose mtime is what it was
|
||||
// keeps its list and only its kept files are stat'ed again, since a file can grow without
|
||||
// its directory noticing. A directory the last walk did not finish (it was cancelled) is
|
||||
// read again whatever else is true, which is what makes a cancelled scan resumable.
|
||||
package space
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// KeepMin is the smallest file kept as its own node.
|
||||
const KeepMin = 128 << 10
|
||||
|
||||
// Node is a directory or a file. Sizes are bytes allocated on disk (what fills the disk);
|
||||
// Apparent is the length. For a directory both include everything beneath it.
|
||||
type Node struct {
|
||||
Name string `json:"name"`
|
||||
Dir bool `json:"dir"`
|
||||
Size int64 `json:"size"`
|
||||
Apparent int64 `json:"apparent"`
|
||||
MTime int64 `json:"mtime"` // unix seconds
|
||||
MTimeNs int64 `json:"-"` // directories: the same to the nanosecond, so a change in the same second is not missed
|
||||
Files uint32 `json:"files"` // beneath a directory (kept and small alike)
|
||||
Dirs uint32 `json:"dirs"`
|
||||
SmallN uint32 `json:"small_n,omitempty"` // files here below KeepMin, not listed
|
||||
SmallSize int64 `json:"small_size,omitempty"` // their allocated bytes
|
||||
Mount bool `json:"mount,omitempty"` // another filesystem starts here (not this disk's, or a root of its own)
|
||||
Denied bool `json:"denied,omitempty"` // could not be read
|
||||
Partial bool `json:"partial,omitempty"` // the walk was cancelled before this directory was read
|
||||
Children []*Node `json:"children,omitempty"` // directories: sorted by Size, largest first
|
||||
}
|
||||
|
||||
// Progress is how far a running walk has got.
|
||||
type Progress struct {
|
||||
Files int64 `json:"files"`
|
||||
Dirs int64 `json:"dirs"` // directories read
|
||||
Reused int64 `json:"reused"` // directories kept from the last scan without reading
|
||||
Bytes int64 `json:"bytes"`
|
||||
Denied int64 `json:"denied"`
|
||||
Current string `json:"current"`
|
||||
}
|
||||
|
||||
// Mode is how a root is walked when there is a previous tree.
|
||||
type Mode int
|
||||
|
||||
const (
|
||||
ModeFull Mode = iota // read everything
|
||||
ModeMTime // read a directory again only if its mtime changed; stat its kept files
|
||||
ModeDiff // read only the directories a change list names
|
||||
)
|
||||
|
||||
// Options is what a walk starts from.
|
||||
type Options struct {
|
||||
Prev *Node // the last scan's tree, or nil
|
||||
// Modes says how each root is walked (by root path); a root not listed is ModeFull.
|
||||
Modes map[string]Mode
|
||||
// Dirty is every directory a change list named (ModeDiff roots): those are read again.
|
||||
// Touched is every ancestor of a dirty directory: those are kept but looked into.
|
||||
Dirty, Touched map[string]bool
|
||||
}
|
||||
|
||||
type walker struct {
|
||||
ctx context.Context
|
||||
opts Options
|
||||
mu sync.Mutex
|
||||
queue []job
|
||||
cond *sync.Cond
|
||||
pending int // dirs queued or being read
|
||||
files atomic.Int64
|
||||
dirs atomic.Int64
|
||||
reused atomic.Int64
|
||||
bytes atomic.Int64
|
||||
denied atomic.Int64
|
||||
partial atomic.Int64
|
||||
curMu sync.Mutex
|
||||
current string
|
||||
stopping bool
|
||||
}
|
||||
|
||||
type job struct {
|
||||
path string
|
||||
node *Node
|
||||
prev *Node // the same directory in the last scan, if it had one
|
||||
dev uint64 // the filesystem this root is on; a child on another is a mountpoint, not descended
|
||||
mode Mode
|
||||
}
|
||||
|
||||
// Result is what a walk returns besides the tree.
|
||||
type Result struct {
|
||||
Root *Node
|
||||
Partial int64 // directories not read because the walk was cancelled
|
||||
Reused int64
|
||||
Read int64
|
||||
Complete bool
|
||||
}
|
||||
|
||||
// Walk measures roots (mountpoints, absolute) and returns them grafted into one tree whose
|
||||
// root is "/". progress, if set, is called about four times a second. A cancelled walk
|
||||
// still returns the tree, with the directories it did not get to marked Partial.
|
||||
func Walk(ctx context.Context, roots []string, opts Options, progress func(Progress)) (Result, error) {
|
||||
w := &walker{ctx: ctx, opts: opts}
|
||||
w.cond = sync.NewCond(&w.mu)
|
||||
root := &Node{Name: "/", Dir: true}
|
||||
sort.Strings(roots)
|
||||
trees := map[string]*Node{}
|
||||
for _, r := range roots {
|
||||
r = filepath.Clean(r)
|
||||
if !filepath.IsAbs(r) {
|
||||
continue
|
||||
}
|
||||
var st syscall.Stat_t
|
||||
if err := syscall.Lstat(r, &st); err != nil || st.Mode&syscall.S_IFMT != syscall.S_IFDIR {
|
||||
continue
|
||||
}
|
||||
n := &Node{Name: filepath.Base(r), Dir: true, MTime: int64(st.Mtim.Sec), MTimeNs: st.Mtim.Nano()}
|
||||
if r == "/" {
|
||||
n = root
|
||||
}
|
||||
trees[r] = n
|
||||
var prev *Node
|
||||
if opts.Prev != nil {
|
||||
prev = Find(opts.Prev, r)
|
||||
}
|
||||
mode := opts.Modes[r]
|
||||
if prev == nil {
|
||||
mode = ModeFull
|
||||
}
|
||||
w.push(job{path: r, node: n, prev: prev, dev: uint64(st.Dev), mode: mode})
|
||||
}
|
||||
if len(trees) == 0 {
|
||||
return Result{Root: root}, os.ErrNotExist
|
||||
}
|
||||
workers := 32
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < workers; i++ {
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); w.run() }()
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() { wg.Wait(); close(done) }()
|
||||
tick := time.NewTicker(250 * time.Millisecond)
|
||||
defer tick.Stop()
|
||||
cancelled := false
|
||||
for waiting := true; waiting; {
|
||||
select {
|
||||
case <-done:
|
||||
waiting = false
|
||||
case <-tick.C:
|
||||
if progress != nil {
|
||||
progress(w.snapshot())
|
||||
}
|
||||
case <-ctx.Done():
|
||||
cancelled = true
|
||||
w.mu.Lock()
|
||||
w.stopping = true
|
||||
for _, j := range w.queue { // never read: say so, so the next walk reads them
|
||||
j.node.Partial = true
|
||||
w.partial.Add(1)
|
||||
}
|
||||
w.queue = nil
|
||||
w.cond.Broadcast()
|
||||
w.mu.Unlock()
|
||||
<-done
|
||||
waiting = false
|
||||
}
|
||||
}
|
||||
if progress != nil {
|
||||
progress(w.snapshot())
|
||||
}
|
||||
// graft each root at its path; deepest last so parents exist first
|
||||
for _, r := range roots {
|
||||
n, ok := trees[filepath.Clean(r)]
|
||||
if !ok || n == root {
|
||||
continue
|
||||
}
|
||||
graft(root, filepath.Clean(r), n)
|
||||
}
|
||||
total(root)
|
||||
res := Result{Root: root, Partial: w.partial.Load(), Reused: w.reused.Load(), Read: w.dirs.Load(), Complete: !cancelled}
|
||||
if cancelled {
|
||||
return res, ctx.Err()
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
func (w *walker) snapshot() Progress {
|
||||
w.curMu.Lock()
|
||||
cur := w.current
|
||||
w.curMu.Unlock()
|
||||
return Progress{Files: w.files.Load(), Dirs: w.dirs.Load(), Reused: w.reused.Load(), Bytes: w.bytes.Load(), Denied: w.denied.Load(), Current: cur}
|
||||
}
|
||||
|
||||
func (w *walker) push(j job) {
|
||||
w.mu.Lock()
|
||||
if w.stopping {
|
||||
j.node.Partial = true
|
||||
w.partial.Add(1)
|
||||
w.mu.Unlock()
|
||||
return
|
||||
}
|
||||
w.queue = append(w.queue, j)
|
||||
w.pending++
|
||||
w.cond.Signal()
|
||||
w.mu.Unlock()
|
||||
}
|
||||
|
||||
// run takes directories off the queue until there are none left and none being read.
|
||||
func (w *walker) run() {
|
||||
for {
|
||||
w.mu.Lock()
|
||||
for len(w.queue) == 0 && w.pending > 0 && !w.stopping {
|
||||
w.cond.Wait()
|
||||
}
|
||||
if w.stopping || (len(w.queue) == 0 && w.pending == 0) {
|
||||
w.cond.Broadcast()
|
||||
w.mu.Unlock()
|
||||
return
|
||||
}
|
||||
j := w.queue[len(w.queue)-1] // depth first: keeps the queue short and the disk head local
|
||||
w.queue = w.queue[:len(w.queue)-1]
|
||||
w.mu.Unlock()
|
||||
w.visit(j)
|
||||
w.mu.Lock()
|
||||
w.pending--
|
||||
if w.pending == 0 {
|
||||
w.cond.Broadcast()
|
||||
}
|
||||
w.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
func join(dir, name string) string {
|
||||
if dir == "/" {
|
||||
return "/" + name
|
||||
}
|
||||
return dir + "/" + name
|
||||
}
|
||||
|
||||
// visit decides whether a directory must be read or can be taken from the last scan.
|
||||
func (w *walker) visit(j job) {
|
||||
if j.prev == nil || j.prev.Partial || j.prev.Denied || j.mode == ModeFull {
|
||||
w.readDir(j)
|
||||
return
|
||||
}
|
||||
switch j.mode {
|
||||
case ModeDiff:
|
||||
if w.opts.Dirty[j.path] {
|
||||
w.readDir(j)
|
||||
return
|
||||
}
|
||||
if !w.opts.Touched[j.path] {
|
||||
w.reuse(j, false) // nothing under here changed: keep the whole subtree, no stat at all
|
||||
return
|
||||
}
|
||||
w.reuse(j, true) // something deeper changed: keep this list, look into the children
|
||||
case ModeMTime:
|
||||
var st syscall.Stat_t
|
||||
if err := syscall.Lstat(j.path, &st); err != nil {
|
||||
w.readDir(j)
|
||||
return
|
||||
}
|
||||
if st.Mtim.Nano() != j.prev.MTimeNs {
|
||||
w.readDir(j) // an entry was added, removed or renamed
|
||||
return
|
||||
}
|
||||
w.reuse(j, true) // same entries; files may have grown, and subdirectories are their own question
|
||||
}
|
||||
}
|
||||
|
||||
// reuse keeps a directory as the last scan saw it. With descend, its subdirectories are
|
||||
// visited (each decides for itself) and, in mtime mode, its kept files are stat'ed again.
|
||||
func (w *walker) reuse(j job, descend bool) {
|
||||
w.reused.Add(1)
|
||||
n, p := j.node, j.prev
|
||||
n.MTime, n.MTimeNs, n.SmallN, n.SmallSize, n.Denied, n.Mount = p.MTime, p.MTimeNs, p.SmallN, p.SmallSize, p.Denied, p.Mount
|
||||
if !descend {
|
||||
n.Children = p.Children
|
||||
n.Apparent = p.Apparent - childApparent(p) // total() adds the children back
|
||||
w.files.Add(int64(p.Files))
|
||||
w.bytes.Add(p.Size)
|
||||
w.reused.Add(int64(p.Dirs))
|
||||
return
|
||||
}
|
||||
w.files.Add(int64(p.SmallN))
|
||||
w.bytes.Add(p.SmallSize)
|
||||
n.Apparent = p.Apparent - childApparent(p) // the small files' lengths; total() adds the children back
|
||||
var st syscall.Stat_t
|
||||
for _, c := range p.Children {
|
||||
if c.Dir {
|
||||
d := &Node{Name: c.Name, Dir: true, MTime: c.MTime, MTimeNs: c.MTimeNs, Mount: c.Mount}
|
||||
n.Children = append(n.Children, d)
|
||||
if c.Mount {
|
||||
continue
|
||||
}
|
||||
w.push(job{path: join(j.path, c.Name), node: d, prev: c, dev: j.dev, mode: j.mode})
|
||||
continue
|
||||
}
|
||||
if j.mode == ModeMTime {
|
||||
if err := syscall.Lstat(join(j.path, c.Name), &st); err != nil {
|
||||
continue // gone since (the directory's mtime would have said so, but be safe)
|
||||
}
|
||||
c = &Node{Name: c.Name, Size: int64(st.Blocks) * 512, Apparent: int64(st.Size), MTime: int64(st.Mtim.Sec)}
|
||||
}
|
||||
n.Children = append(n.Children, c)
|
||||
w.files.Add(1)
|
||||
w.bytes.Add(c.Size)
|
||||
}
|
||||
}
|
||||
|
||||
func childApparent(p *Node) int64 {
|
||||
var a int64
|
||||
for _, c := range p.Children {
|
||||
a += c.Apparent
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
func (w *walker) readDir(j job) {
|
||||
w.curMu.Lock()
|
||||
w.current = j.path
|
||||
w.curMu.Unlock()
|
||||
w.dirs.Add(1)
|
||||
f, err := os.Open(j.path)
|
||||
if err != nil {
|
||||
j.node.Denied = true
|
||||
w.denied.Add(1)
|
||||
return
|
||||
}
|
||||
names, err := f.Readdirnames(-1)
|
||||
f.Close()
|
||||
if err != nil {
|
||||
j.node.Denied = true
|
||||
w.denied.Add(1)
|
||||
}
|
||||
var prevDirs map[string]*Node
|
||||
if j.prev != nil {
|
||||
prevDirs = map[string]*Node{}
|
||||
for _, c := range j.prev.Children {
|
||||
if c.Dir {
|
||||
prevDirs[c.Name] = c
|
||||
}
|
||||
}
|
||||
}
|
||||
var st syscall.Stat_t
|
||||
for _, name := range names {
|
||||
p := join(j.path, name)
|
||||
if err := syscall.Lstat(p, &st); err != nil {
|
||||
continue
|
||||
}
|
||||
switch st.Mode & syscall.S_IFMT {
|
||||
case syscall.S_IFDIR:
|
||||
c := &Node{Name: name, Dir: true, MTime: int64(st.Mtim.Sec), MTimeNs: st.Mtim.Nano()}
|
||||
j.node.Children = append(j.node.Children, c)
|
||||
if uint64(st.Dev) != j.dev {
|
||||
c.Mount = true // something else is mounted here; if it is this disk's too, it is a root of its own
|
||||
continue
|
||||
}
|
||||
w.push(job{path: p, node: c, prev: prevDirs[name], dev: j.dev, mode: j.mode})
|
||||
case syscall.S_IFREG, syscall.S_IFLNK:
|
||||
alloc := int64(st.Blocks) * 512
|
||||
w.files.Add(1)
|
||||
w.bytes.Add(alloc)
|
||||
if alloc >= KeepMin || int64(st.Size) >= KeepMin {
|
||||
j.node.Children = append(j.node.Children, &Node{Name: name, Size: alloc, Apparent: int64(st.Size), MTime: int64(st.Mtim.Sec)})
|
||||
} else {
|
||||
j.node.SmallN++
|
||||
j.node.SmallSize += alloc
|
||||
j.node.Apparent += int64(st.Size) // added to the directory's own apparent total below
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// graft puts a root's tree at path under root, making any directories in between.
|
||||
func graft(root *Node, path string, tree *Node) {
|
||||
parts := strings.Split(strings.Trim(path, "/"), "/")
|
||||
cur := root
|
||||
for i, part := range parts {
|
||||
var next *Node
|
||||
for _, c := range cur.Children {
|
||||
if c.Name == part && c.Dir {
|
||||
next = c
|
||||
break
|
||||
}
|
||||
}
|
||||
if i == len(parts)-1 {
|
||||
if next == nil {
|
||||
cur.Children = append(cur.Children, tree)
|
||||
} else {
|
||||
// the parent's walk saw this directory as a mountpoint; the walk of the mount fills it
|
||||
next.Children, next.SmallN, next.SmallSize, next.Apparent, next.Denied, next.Partial = tree.Children, tree.SmallN, tree.SmallSize, tree.Apparent, tree.Denied, tree.Partial
|
||||
next.Mount = true
|
||||
}
|
||||
return
|
||||
}
|
||||
if next == nil {
|
||||
next = &Node{Name: part, Dir: true}
|
||||
cur.Children = append(cur.Children, next)
|
||||
}
|
||||
cur = next
|
||||
}
|
||||
}
|
||||
|
||||
// total sums every directory from its children and sorts them largest first.
|
||||
func total(n *Node) {
|
||||
if !n.Dir {
|
||||
return
|
||||
}
|
||||
n.Size = n.SmallSize
|
||||
n.Files = n.SmallN
|
||||
n.Dirs = 0
|
||||
apparent := n.Apparent // the small files' lengths
|
||||
for _, c := range n.Children {
|
||||
total(c)
|
||||
n.Size += c.Size
|
||||
apparent += c.Apparent
|
||||
if c.Dir {
|
||||
n.Dirs += 1 + c.Dirs
|
||||
n.Files += c.Files
|
||||
} else {
|
||||
n.Files++
|
||||
}
|
||||
}
|
||||
n.Apparent = apparent
|
||||
sort.Slice(n.Children, func(i, j int) bool { return n.Children[i].Size > n.Children[j].Size })
|
||||
}
|
||||
|
||||
// Find follows path (absolute, "/" for the root) down the tree.
|
||||
func Find(root *Node, path string) *Node {
|
||||
cur := root
|
||||
for _, part := range strings.Split(strings.Trim(filepath.Clean(path), "/"), "/") {
|
||||
if part == "" {
|
||||
continue
|
||||
}
|
||||
var next *Node
|
||||
for _, c := range cur.Children {
|
||||
if c.Name == part {
|
||||
next = c
|
||||
break
|
||||
}
|
||||
}
|
||||
if next == nil {
|
||||
return nil
|
||||
}
|
||||
cur = next
|
||||
}
|
||||
return cur
|
||||
}
|
||||
|
||||
// File is one of the largest files, with its full path.
|
||||
type File struct {
|
||||
Path string `json:"path"`
|
||||
Size int64 `json:"size"`
|
||||
Apparent int64 `json:"apparent"`
|
||||
MTime int64 `json:"mtime"`
|
||||
}
|
||||
|
||||
// Largest is the n biggest files in the tree, largest first.
|
||||
func Largest(root *Node, n int) []File {
|
||||
var out []File
|
||||
minIdx := -1
|
||||
var walk func(n *Node, path string)
|
||||
walk = func(node *Node, path string) {
|
||||
for _, c := range node.Children {
|
||||
p := join(path, c.Name)
|
||||
if c.Dir {
|
||||
if c.Size > 0 && (len(out) < n || c.Size > out[minIdx].Size) {
|
||||
walk(c, p)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if len(out) < n {
|
||||
out = append(out, File{Path: p, Size: c.Size, Apparent: c.Apparent, MTime: c.MTime})
|
||||
if minIdx < 0 || c.Size < out[minIdx].Size {
|
||||
minIdx = len(out) - 1
|
||||
}
|
||||
} else if c.Size > out[minIdx].Size {
|
||||
out[minIdx] = File{Path: p, Size: c.Size, Apparent: c.Apparent, MTime: c.MTime}
|
||||
minIdx = 0
|
||||
for i := range out {
|
||||
if out[i].Size < out[minIdx].Size {
|
||||
minIdx = i
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(root, "/")
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Size > out[j].Size })
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package space
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func mk(t *testing.T, dir string, files map[string]int) {
|
||||
for p, n := range files {
|
||||
full := filepath.Join(dir, p)
|
||||
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(full, make([]byte, n), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefreshAndResume(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
mk(t, dir, map[string]int{"a/big1": 300 << 10, "a/small": 100, "b/c/big2": 500 << 10, "b/d/big3": 200 << 10})
|
||||
res, err := Walk(context.Background(), []string{dir}, Options{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
root := Find(res.Root, dir)
|
||||
if root == nil || root.Files != 4 {
|
||||
t.Fatalf("full walk: %+v", root)
|
||||
}
|
||||
want := root.Size
|
||||
// grow a file in place (its directory's mtime does not change) and add a file elsewhere
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
if err := os.WriteFile(filepath.Join(dir, "a/big1"), make([]byte, 600<<10), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mk(t, dir, map[string]int{"b/d/big4": 400 << 10})
|
||||
res2, err := Walk(context.Background(), []string{dir}, Options{Prev: res.Root, Modes: map[string]Mode{dir: ModeMTime}}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r2 := Find(res2.Root, dir)
|
||||
if r2.Files != 5 || r2.Size != want+(300<<10)+(400<<10) {
|
||||
t.Fatalf("mtime refresh: files %d size %d want %d (reused %d read %d)", r2.Files, r2.Size, want+(300<<10)+(400<<10), res2.Reused, res2.Read)
|
||||
}
|
||||
if res2.Reused == 0 || res2.Read == 0 {
|
||||
t.Fatalf("mtime refresh should reuse some and read some: reused %d read %d", res2.Reused, res2.Read)
|
||||
}
|
||||
// diff mode: only b/d named dirty; a's growth must NOT be seen (that is the contract), b/d's new file must
|
||||
dirty := map[string]bool{filepath.Join(dir, "b/d"): true}
|
||||
touched := map[string]bool{dir: true, filepath.Join(dir, "b"): true}
|
||||
res3, err := Walk(context.Background(), []string{dir}, Options{Prev: res.Root, Modes: map[string]Mode{dir: ModeDiff}, Dirty: dirty, Touched: touched}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r3 := Find(res3.Root, dir)
|
||||
if r3.Files != 5 || r3.Size != want+(400<<10) || res3.Read != 1 {
|
||||
t.Fatalf("diff refresh: files %d size %d want %d read %d", r3.Files, r3.Size, want+(400<<10), res3.Read)
|
||||
}
|
||||
// a cancelled walk marks what it did not read; resuming reads those
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
res4, err := Walk(ctx, []string{dir}, Options{}, nil)
|
||||
if err == nil || res4.Complete {
|
||||
t.Fatal("cancelled walk should say so")
|
||||
}
|
||||
if len(partialPaths(res4.Root, "/")) == 0 {
|
||||
t.Fatal("cancelled walk should leave partial directories")
|
||||
}
|
||||
res5, err := Walk(context.Background(), []string{dir}, Options{Prev: res4.Root, Modes: map[string]Mode{dir: ModeMTime}}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r5 := Find(res5.Root, dir); r5.Files != 5 || len(partialPaths(res5.Root, "/")) != 0 {
|
||||
t.Fatalf("resume: files %d partial %v", r5.Files, partialPaths(res5.Root, "/"))
|
||||
}
|
||||
// the codec round-trips the partial flag
|
||||
pr, pw, _ := os.Pipe()
|
||||
go func() { _ = Encode(pw, res4.Root); pw.Close() }()
|
||||
back, err := Decode(pr)
|
||||
if err != nil || len(partialPaths(back, "/")) != len(partialPaths(res4.Root, "/")) {
|
||||
t.Fatalf("codec: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
package storage
|
||||
|
||||
// The ZFS caching tiers, for the ZFS tab: what the ARC has done since boot and how it is
|
||||
// tuned, and for each pool which cache devices it has — L2ARC, a separate log (SLOG), a
|
||||
// special allocation class, a dedup table — or, as is common, none of them, which the tab
|
||||
// should say plainly rather than leave a blank.
|
||||
//
|
||||
// The live numbers (size, the hit rate over the last interval) come from the sampler in
|
||||
// internal/metrics, which also publishes them as zfs.arc.* sensors; this is the slower,
|
||||
// fuller picture, read when the tab asks.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// CacheDev is one member of a pool's cache, log, special, dedup or spare class.
|
||||
type CacheDev struct {
|
||||
Name string `json:"name"`
|
||||
Size uint64 `json:"size,omitempty"`
|
||||
Alloc uint64 `json:"alloc,omitempty"`
|
||||
Health string `json:"health,omitempty"`
|
||||
}
|
||||
|
||||
// PoolCaching is what one pool has beyond its data vdevs, and the properties that steer
|
||||
// what is cached.
|
||||
type PoolCaching struct {
|
||||
Pool string `json:"pool"`
|
||||
Cache []CacheDev `json:"cache"` // L2ARC
|
||||
Log []CacheDev `json:"log"` // SLOG
|
||||
Special []CacheDev `json:"special"` // metadata / small blocks
|
||||
Dedup []CacheDev `json:"dedup"` // dedup table vdevs
|
||||
Spare []CacheDev `json:"spare"`
|
||||
// dataset properties at the pool root
|
||||
PrimaryCache string `json:"primarycache"`
|
||||
SecondaryCache string `json:"secondarycache"`
|
||||
Sync string `json:"sync"`
|
||||
LogBias string `json:"logbias"`
|
||||
DedupProp string `json:"dedup_prop"`
|
||||
Compression string `json:"compression"`
|
||||
RecordSize string `json:"recordsize"`
|
||||
DedupRatio string `json:"dedup_ratio"` // zpool's, e.g. "1.00x"
|
||||
}
|
||||
|
||||
// ARCDetail is the full picture.
|
||||
type ARCDetail struct {
|
||||
Available bool `json:"available"`
|
||||
MemTotal uint64 `json:"mem_total"`
|
||||
Tunables map[string]string `json:"tunables"` // zfs module parameters that shape the cache; "" when unset
|
||||
Lifetime struct {
|
||||
Hits, Misses uint64 `json:",omitempty"`
|
||||
DemandDataHits, DemandDataMisses uint64
|
||||
DemandMetaHits, DemandMetaMisses uint64
|
||||
PrefetchDataHits, PrefetchDataMisses uint64
|
||||
PrefetchMetaHits, PrefetchMetaMisses uint64
|
||||
L2Hits, L2Misses, L2ReadBytes, L2WriteBytes uint64
|
||||
L2Feeds, L2RebuildSize uint64
|
||||
EvictL2Eligible uint64 // bytes evicted from RAM that an L2ARC could have caught
|
||||
Deleted, EvictSkip, MutexMiss, Prune, Throttle uint64
|
||||
} `json:"lifetime"`
|
||||
Zfetch struct{ Hits, Misses, IOIssued uint64 } `json:"zfetch"` // the prefetcher
|
||||
ZIL struct {
|
||||
Commits uint64 `json:"commits"`
|
||||
NormalBytes, NormalWrite uint64 // written to the pool's own vdevs
|
||||
SlogBytes, SlogWrite uint64 // written to a separate log device
|
||||
} `json:"zil"`
|
||||
DBuf struct{ Size, Target, Hits, Misses uint64 } `json:"dbuf"` // the dbuf cache in front of the ARC
|
||||
Pools []PoolCaching `json:"pools"`
|
||||
}
|
||||
|
||||
const kstatDir = "/proc/spl/kstat/zfs/"
|
||||
|
||||
func readKstat(p string) map[string]uint64 {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
out := map[string]uint64{}
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) != 3 || f[1] != "4" {
|
||||
continue
|
||||
}
|
||||
if v, err := strconv.ParseUint(f[2], 10, 64); err == nil {
|
||||
out[f[0]] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
var arcTunables = []string{"zfs_arc_max", "zfs_arc_min", "zfs_arc_sys_free", "zfs_arc_meta_balance", "zfs_prefetch_disable", "l2arc_write_max", "l2arc_write_boost", "l2arc_noprefetch", "l2arc_rebuild_enabled", "zfs_dirty_data_max"}
|
||||
|
||||
const poolCachingEvery = 30 * time.Second // zpool list -v and zfs get are an exec each
|
||||
|
||||
// ARC reads the kstats now and the pool layout from a short cache.
|
||||
func (m *Manager) ARC(ctx context.Context) ARCDetail {
|
||||
d := ARCDetail{Tunables: map[string]string{}, Pools: []PoolCaching{}}
|
||||
k := readKstat(kstatDir + "arcstats")
|
||||
if k == nil {
|
||||
return d
|
||||
}
|
||||
d.Available = true
|
||||
l := &d.Lifetime
|
||||
l.Hits, l.Misses = k["hits"], k["misses"]
|
||||
l.DemandDataHits, l.DemandDataMisses = k["demand_data_hits"], k["demand_data_misses"]
|
||||
l.DemandMetaHits, l.DemandMetaMisses = k["demand_metadata_hits"], k["demand_metadata_misses"]
|
||||
l.PrefetchDataHits, l.PrefetchDataMisses = k["prefetch_data_hits"], k["prefetch_data_misses"]
|
||||
l.PrefetchMetaHits, l.PrefetchMetaMisses = k["prefetch_metadata_hits"], k["prefetch_metadata_misses"]
|
||||
l.L2Hits, l.L2Misses, l.L2ReadBytes, l.L2WriteBytes = k["l2_hits"], k["l2_misses"], k["l2_read_bytes"], k["l2_write_bytes"]
|
||||
l.L2Feeds, l.L2RebuildSize, l.EvictL2Eligible = k["l2_feeds"], k["l2_rebuild_size"], k["evict_l2_eligible"]
|
||||
l.Deleted, l.EvictSkip, l.MutexMiss, l.Prune, l.Throttle = k["deleted"], k["evict_skip"], k["mutex_miss"], k["arc_prune"], k["memory_throttle_count"]
|
||||
if z := readKstat(kstatDir + "zfetchstats"); z != nil {
|
||||
d.Zfetch.Hits, d.Zfetch.Misses, d.Zfetch.IOIssued = z["hits"], z["misses"], z["io_issued"]
|
||||
}
|
||||
if z := readKstat(kstatDir + "zil"); z != nil {
|
||||
d.ZIL.Commits = z["zil_commit_count"]
|
||||
d.ZIL.NormalBytes, d.ZIL.NormalWrite = z["zil_itx_metaslab_normal_bytes"], z["zil_itx_metaslab_normal_write"]
|
||||
d.ZIL.SlogBytes, d.ZIL.SlogWrite = z["zil_itx_metaslab_slog_bytes"], z["zil_itx_metaslab_slog_write"]
|
||||
}
|
||||
if z := readKstat(kstatDir + "dbufstats"); z != nil {
|
||||
d.DBuf.Size, d.DBuf.Target, d.DBuf.Hits, d.DBuf.Misses = z["cache_size_bytes"], z["cache_target_bytes"], z["hash_hits"], z["hash_misses"]
|
||||
}
|
||||
for _, t := range arcTunables {
|
||||
if b, err := os.ReadFile("/sys/module/zfs/parameters/" + t); err == nil {
|
||||
d.Tunables[t] = strings.TrimSpace(string(b))
|
||||
}
|
||||
}
|
||||
if b, err := os.ReadFile("/proc/meminfo"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if strings.HasPrefix(line, "MemTotal:") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) >= 2 {
|
||||
d.MemTotal = u64(f[1]) << 10
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
d.Pools = m.poolCaching(ctx)
|
||||
return d
|
||||
}
|
||||
|
||||
var cachingMu sync.Mutex
|
||||
var cachingAt time.Time
|
||||
var cachingLast []PoolCaching
|
||||
|
||||
func (m *Manager) poolCaching(ctx context.Context) []PoolCaching {
|
||||
cachingMu.Lock()
|
||||
defer cachingMu.Unlock()
|
||||
if cachingLast != nil && time.Since(cachingAt) < poolCachingEvery {
|
||||
return cachingLast
|
||||
}
|
||||
out := []PoolCaching{}
|
||||
bin, ok := lookPath("zpool")
|
||||
if !ok {
|
||||
return out
|
||||
}
|
||||
// zpool list -v needs no root; the vdev tree under each pool, with the class headers
|
||||
raw, code, err := m.runPlain(ctx, bin, "list", "-Hpv", "-o", "name,size,alloc,free,health")
|
||||
if err != nil || code != 0 {
|
||||
return out
|
||||
}
|
||||
out = parsePoolCaching(string(raw))
|
||||
if zfs, ok := lookPath("zfs"); ok {
|
||||
for i := range out {
|
||||
p := &out[i]
|
||||
if props, code, err := m.runPlain(ctx, zfs, "get", "-Hp", "-o", "property,value", "primarycache,secondarycache,sync,logbias,dedup,compression,recordsize", p.Pool); err == nil && code == 0 {
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(props)), "\n") {
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) < 2 {
|
||||
continue
|
||||
}
|
||||
switch f[0] {
|
||||
case "primarycache":
|
||||
p.PrimaryCache = f[1]
|
||||
case "secondarycache":
|
||||
p.SecondaryCache = f[1]
|
||||
case "sync":
|
||||
p.Sync = f[1]
|
||||
case "logbias":
|
||||
p.LogBias = f[1]
|
||||
case "dedup":
|
||||
p.DedupProp = f[1]
|
||||
case "compression":
|
||||
p.Compression = f[1]
|
||||
case "recordsize":
|
||||
p.RecordSize = humanSize(u64(f[1]))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, pool := range parsePools(func() []byte {
|
||||
b, _, _ := m.runPlain(ctx, bin, "list", "-Hp", "-o", "name,size,alloc,free,frag,cap,health,dedup")
|
||||
return b
|
||||
}()) {
|
||||
for i := range out {
|
||||
if out[i].Pool == pool.Name {
|
||||
out[i].DedupRatio = pool.Dedup
|
||||
}
|
||||
}
|
||||
}
|
||||
cachingLast, cachingAt = out, time.Now()
|
||||
return out
|
||||
}
|
||||
|
||||
// runPlain runs a command as ourselves, never through sudo: these are read-only zpool/zfs
|
||||
// queries that need no root on Linux, and a poll must not write to the auth log.
|
||||
func (m *Manager) runPlain(ctx context.Context, bin string, args ...string) ([]byte, int, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, bin, args...).Output()
|
||||
if ee, ok := err.(*exec.ExitError); ok {
|
||||
return out, ee.ExitCode(), nil
|
||||
}
|
||||
return out, 0, err
|
||||
}
|
||||
|
||||
var cacheClasses = map[string]bool{"logs": true, "log": true, "cache": true, "special": true, "dedup": true, "spares": true, "spare": true}
|
||||
|
||||
// parsePoolCaching reads `zpool list -Hpv -o name,size,alloc,free,health`: a pool line, then
|
||||
// its vdevs indented, with a bare class name ("logs", "cache", "special", "dedup", "spares")
|
||||
// introducing each extra class. Only those classes' members are kept; the data vdevs are the
|
||||
// pool and are shown elsewhere.
|
||||
func parsePoolCaching(text string) []PoolCaching {
|
||||
var out []PoolCaching
|
||||
var cur *PoolCaching
|
||||
class := ""
|
||||
for _, raw := range strings.Split(text, "\n") {
|
||||
if strings.TrimSpace(raw) == "" {
|
||||
continue
|
||||
}
|
||||
indented := strings.HasPrefix(raw, " ") || strings.HasPrefix(raw, "\t")
|
||||
f := strings.Split(strings.TrimSpace(raw), "\t")
|
||||
name := strings.TrimSpace(f[0])
|
||||
if !indented {
|
||||
out = append(out, PoolCaching{Pool: name, Cache: []CacheDev{}, Log: []CacheDev{}, Special: []CacheDev{}, Dedup: []CacheDev{}, Spare: []CacheDev{}})
|
||||
cur, class = &out[len(out)-1], ""
|
||||
continue
|
||||
}
|
||||
if cur == nil {
|
||||
continue
|
||||
}
|
||||
if cacheClasses[name] && (len(f) < 2 || f[1] == "-" || f[1] == "") {
|
||||
class = name
|
||||
continue
|
||||
}
|
||||
if class == "" {
|
||||
continue // a data vdev or one of its disks
|
||||
}
|
||||
// a mirror or raidz inside a class lists its members below it; keep them all, the
|
||||
// grouping is visible in the vdev tree on the pool card
|
||||
dev := CacheDev{Name: name}
|
||||
if len(f) >= 5 {
|
||||
dev.Size, dev.Alloc, dev.Health = u64(f[1]), u64(f[2]), strings.TrimSpace(f[4])
|
||||
if f[1] == "-" {
|
||||
dev.Size = 0
|
||||
}
|
||||
}
|
||||
switch class {
|
||||
case "logs", "log":
|
||||
cur.Log = append(cur.Log, dev)
|
||||
case "cache":
|
||||
cur.Cache = append(cur.Cache, dev)
|
||||
case "special":
|
||||
cur.Special = append(cur.Special, dev)
|
||||
case "dedup":
|
||||
cur.Dedup = append(cur.Dedup, dev)
|
||||
case "spares", "spare":
|
||||
cur.Spare = append(cur.Spare, dev)
|
||||
}
|
||||
}
|
||||
if out == nil {
|
||||
out = []PoolCaching{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func humanSize(b uint64) string {
|
||||
switch {
|
||||
case b >= 1<<20 && b%(1<<20) == 0:
|
||||
return strconv.FormatUint(b>>20, 10) + "M"
|
||||
case b >= 1<<10 && b%(1<<10) == 0:
|
||||
return strconv.FormatUint(b>>10, 10) + "K"
|
||||
}
|
||||
return strconv.FormatUint(b, 10)
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// zpool list -Hpv on a pool with every extra class; the data raidz and its disks are skipped
|
||||
const listV = "tank\t32000000000000\t23000000000000\t9000000000000\tONLINE\n" +
|
||||
"\traidz2-0\t32000000000000\t23000000000000\t9000000000000\tONLINE\n" +
|
||||
"\t\twwn-0x1\t4000000000000\t-\t-\tONLINE\n" +
|
||||
"\t\twwn-0x2\t4000000000000\t-\t-\tONLINE\n" +
|
||||
"\tspecial\t-\t-\t-\t-\n" +
|
||||
"\t\tmirror-1\t500000000000\t120000000000\t380000000000\tONLINE\n" +
|
||||
"\t\t\tnvme0n1p1\t500000000000\t-\t-\tONLINE\n" +
|
||||
"\t\t\tnvme1n1p1\t500000000000\t-\t-\tONLINE\n" +
|
||||
"\tlogs\t-\t-\t-\t-\n" +
|
||||
"\t\tnvme2n1p1\t16000000000\t4000000\t15996000000\tONLINE\n" +
|
||||
"\tcache\t-\t-\t-\t-\n" +
|
||||
"\t\tnvme2n1p2\t400000000000\t210000000000\t190000000000\tONLINE\n" +
|
||||
"\tspares\t-\t-\t-\t-\n" +
|
||||
"\t\twwn-0x9\t4000000000000\t-\t-\tAVAIL\n" +
|
||||
"scratch\t1000000000000\t1000000000\t999000000000\tONLINE\n" +
|
||||
"\tsda\t1000000000000\t1000000000\t999000000000\tONLINE\n"
|
||||
|
||||
func TestParsePoolCaching(t *testing.T) {
|
||||
got := parsePoolCaching(listV)
|
||||
if len(got) != 2 || got[0].Pool != "tank" || got[1].Pool != "scratch" {
|
||||
t.Fatalf("pools: %+v", got)
|
||||
}
|
||||
tank := got[0]
|
||||
if len(tank.Cache) != 1 || tank.Cache[0].Name != "nvme2n1p2" || tank.Cache[0].Alloc != 210000000000 {
|
||||
t.Errorf("cache: %+v", tank.Cache)
|
||||
}
|
||||
if len(tank.Log) != 1 || tank.Log[0].Name != "nvme2n1p1" {
|
||||
t.Errorf("log: %+v", tank.Log)
|
||||
}
|
||||
if len(tank.Special) != 3 || tank.Special[0].Name != "mirror-1" || tank.Special[0].Alloc != 120000000000 {
|
||||
t.Errorf("special: %+v", tank.Special)
|
||||
}
|
||||
if len(tank.Spare) != 1 || tank.Spare[0].Health != "AVAIL" {
|
||||
t.Errorf("spares: %+v", tank.Spare)
|
||||
}
|
||||
if len(tank.Dedup) != 0 {
|
||||
t.Errorf("no dedup class: %+v", tank.Dedup)
|
||||
}
|
||||
s := got[1]
|
||||
if len(s.Cache)+len(s.Log)+len(s.Special)+len(s.Spare) != 0 {
|
||||
t.Errorf("a plain pool has no extra classes: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHumanSize(t *testing.T) {
|
||||
for in, want := range map[uint64]string{131072: "128K", 1048576: "1M", 16384: "16K", 1000: "1000"} {
|
||||
if got := humanSize(in); got != want {
|
||||
t.Errorf("%d → %s, want %s", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestLiveARC reads this machine's ZFS for real: HAMLAB_LIVE=1 go test -run TestLiveARC -v ./internal/storage
|
||||
func TestLiveARC(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_LIVE") == "" {
|
||||
t.Skip("set HAMLAB_LIVE=1 to read the real machine")
|
||||
}
|
||||
d := New(false).ARC(context.Background())
|
||||
b, _ := json.MarshalIndent(d, "", " ")
|
||||
t.Logf("%s", b)
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user