Compare commits
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(systemctl status *)",
|
||||
"Bash(systemctl is-active *)"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -2,3 +2,4 @@ bin/
|
||||
internal/web/dist/
|
||||
web/node_modules/
|
||||
web/dist/
|
||||
dist/
|
||||
|
||||
@@ -0,0 +1,778 @@
|
||||
# Bug bash — 2026-09-18
|
||||
|
||||
Fix pass 2026-09-18: every item carries its outcome; ✅ resolved and verified live, ➖ no change with reason. Found by a headless Chromium pass (iPhone and
|
||||
1440px desktop viewports) over every page on the hub and on each remote machine, an
|
||||
offline-node scenario (wyse stopped for the test, then restarted), direct API probing
|
||||
with hostile inputs, and a code read. Ordered by severity within each group.
|
||||
|
||||
## Security / robustness
|
||||
|
||||
1. ✅ **RESOLVED 2026-09-18** — only regular files are readable; the byte cap applies to what is sent, not to `stat`. `/dev/zero`, `/dev/null` and FIFOs answer 400 in 30 ms; `/proc/cpuinfo` still reads. **Reading a special file hangs the request forever.** `GET /api/files/read?path=/dev/zero`
|
||||
never ends: the `max` cap compares against `stat` size, which is 0 for device files,
|
||||
so the cap never triggers and the handler streams `/dev/zero` until the client gives
|
||||
up. Same for FIFOs (blocks on open) and `/proc/self/mem`. Opening one in the Files
|
||||
viewer would hang the tab. Needs: refuse non-regular files for read/edit, and cap the
|
||||
copy with `io.LimitReader` regardless of `stat`.
|
||||
2. ✅ **RESOLVED** — per-client buckets (first X-Forwarded-For hop or the peer) with the same 5-strike schedule, plus a global backstop of 100 failures per 5 min → 1 min. Verified: core's public IP locked while grid logged in fine. **Login lockout is global, not per client.** Five bad passwords from anyone on the
|
||||
internet lock the real user out for up to 10 minutes (`auth.Limiter` is one bucket).
|
||||
The bash locked the hub for the full 10 minutes twice. Also locks `POST /api/tokens`,
|
||||
so hub pairing fails during a lockout. Needs per-IP buckets with a modest global cap.
|
||||
3. ✅ **RESOLVED** — server rejects refreshMs outside 100–60000 and non-identifier themes (400); client ignores out-of-range rates from any source. **UI prefs from the server are not validated.** A stored `refreshMs` of `-5` made the
|
||||
client call `setInterval(-5)` on every live view: a request storm that took the
|
||||
browser to `ERR_INSUFFICIENT_RESOURCES` and hammered the daemon. `theme` is validated
|
||||
only by falling back. Needs range checks on load (and on save server-side).
|
||||
4. ✅ **RESOLVED** — every response carries X-Frame-Options DENY, CSP frame-ancestors none, nosniff, no-referrer. **No anti-framing / CSP headers.** Only HSTS (from NPM). A root-capable console should
|
||||
send `X-Frame-Options: DENY`, `Content-Security-Policy: frame-ancestors 'none'`,
|
||||
`Referrer-Policy: no-referrer`, `X-Content-Type-Options: nosniff` on every response.
|
||||
5. ✅ **RESOLVED** — kernel threads (empty cmdline) and the daemon's own PID are refused. **Signals can be sent to kernel threads.** `POST /api/procs/2/signal` (kthreadd)
|
||||
returns 204. Harmless in practice, but the guard should refuse PIDs with an empty
|
||||
cmdline (kernel threads) and our own PID.
|
||||
6. ✅ **RESOLVED** — unit runs with `SupplementaryGroups=systemd-journal adm` so journal reads need no sudo, and the installer drops `/etc/sudoers.d/hamlab-quiet` (no syslog/pam session for the read-only polling commands). Verified: 0 sudo lines in 60 s of 250 ms polling (was 839/10 min). Applied on all four hosts. **Journal spam from the daemon's own sudo calls.** Every sudo'd `journalctl`, `smartctl`
|
||||
or `pwrstat` writes three `pam_unix(sudo:session)` lines into the journal *under
|
||||
`hamlab.service`*: 839 lines in 10 minutes at a 250 ms refresh, and the journal is
|
||||
already 4 GB. It also buries Ham Lab's own log entries. Fix options: add the service
|
||||
user to `systemd-journal` (and `adm`) so journal reads don't need sudo, and/or a
|
||||
sudoers `Defaults!/usr/bin/journalctl !syslog, !pam_session` line in the installer.
|
||||
7. ✅ **RESOLVED** — server answers 409 unless `overwrite=1`; the UI asks "Replace it?" first. **Uploading over an existing file silently overwrites it.** No confirmation, no
|
||||
"replace?" prompt. Same for "+ file" when the name exists (server refuses; UI just
|
||||
shows the error, fine) — upload is the gap.
|
||||
8. ➖ **NO CHANGE** — Go's mux redirects the unclean path to `/etc/passwd`, which is an ordinary deep link the SPA must serve; there is no API prefix left by the time the static handler sees it. Harmless. **`/api/<anything>/../../x` falls through to the SPA** and returns `index.html` with
|
||||
200 instead of a JSON 404. Harmless, but sloppy for an API prefix.
|
||||
|
||||
## Functional bugs
|
||||
|
||||
9. ✅ **RESOLVED** — 401 carries the server message; the form shows "wrong password" (verified headless). **Wrong password gives no feedback at all.** The 401 is thrown as `Unauthenticated`
|
||||
with an empty message, and the login form only renders `err` when non-empty. The
|
||||
button just re-enables. Reproduced on phone and desktop.
|
||||
10. ✅ **RESOLVED** — hub-only pages bounce to the dashboard on switch (verified headless). **Switching machines while on Settings or Machines keeps showing the hub's page.**
|
||||
`switchMachine` only resets the view for Machines; Settings stays, so "mac" shows
|
||||
core's account/token/about panels. Both are hub-only pages and should bounce to the
|
||||
dashboard (or be clearly labelled as hub pages).
|
||||
11. ✅ **RESOLVED** — escaped; no console error. **Machines add-form `pattern` is an invalid regex in Chromium.** `[a-z0-9][a-z0-9-]*`
|
||||
fails under the `v` flag ("Invalid character class"), logs a console error and the
|
||||
browser skips validation. Needs `[a-z0-9][a-z0-9\-]*`.
|
||||
12. ✅ **RESOLVED** — refresh from the newest repodata under libdnf5/dnf caches (Mac: 9 d), last upgrade from `dnf history` (194 d), label reads "from dnf history". **Updates tab on dnf hosts shows nonsense.** Mac: "Index refreshed 561d ago" (mtime of
|
||||
`/var/cache/dnf`, which Fedora 41+ no longer uses; `libdnf5` is the live cache) and
|
||||
"Last upgrade never · from apt history" (label and source are apt-specific). Needs
|
||||
`dnf history` for last upgrade, metadata timestamps for refresh, and neutral labels.
|
||||
13. ✅ **RESOLVED** — journal read incrementally by cursor, timers from one `systemctl show`, services' two calls run in parallel. Cron 3.6 s → 0.08 s, services 0.6 s → 0.05 s, with last-run and next-run data intact. **Cron overview is slow: 3.6 s per request on core.** At any refresh rate the page
|
||||
is constantly busy. The 30-day, 20 000-line journal scan for "last run" is rebuilt
|
||||
every 5 s; timers make three `systemctl` calls. Needs an incremental journal cursor
|
||||
and a single `systemctl` call. Services list is 0.6 s for the same reason (two calls).
|
||||
14. ✅ **RESOLVED** — heap profile showed a 48 MB `bytes.Buffer` from buffering `journalctl` dumps; those are now streamed line by line, plus a 160 MB soft memory limit for the GC. RSS 22 MB fresh, 27 MB after 25× cron/logs/procs/services (was 180 MB). pprof stays available behind the login at `/api/debug/pprof/`. **Daemon memory grows.** RSS on core went 22 → 85 → 216 MB during the day (136 MB at
|
||||
rest after the bash). Suspects: per-pid caches in the process table (`procInfo` only
|
||||
prunes on the next walk), task output buffers, SMART cache, xterm scrollback rings
|
||||
that are never freed for exited sessions. Needs a heap profile under load.
|
||||
15. ✅ **RESOLVED** — `--dirty` dropped from the version and deploys now follow commits; binaries report e.g. `v0.1.0-38-gc1805f6`. **Version string is always `-dirty`.** `make deploy` builds before the commit, so
|
||||
every installed binary reports e.g. `v0.1.0-32-ga372378-dirty`, and the Machines page
|
||||
compares with `-dirty` stripped. Build after commit, or tag releases.
|
||||
16. ✅ **RESOLVED** — hash routing `#/<machine|hub>/<view>`: reload restores machine and tab, Back/Forward work, links can point at a page on a machine (verified headless). **View resets to Dashboard on reload; no deep links; browser Back leaves the app.**
|
||||
The selected machine survives a reload (sessionStorage) but the tab does not, and
|
||||
there is no URL routing at all. Needs hash or path routing (`/#/grid/services`).
|
||||
17. ✅ **RESOLVED** — newest first. **Cron log tab lists oldest first and starts at the top.** 200 lines from Sep 10 with
|
||||
the newest at the bottom, off-screen. Should show newest first or scroll to the end.
|
||||
18. ✅ **RESOLVED** — "Nothing matches." **Processes: empty filter result shows nothing.** No "nothing matches" message.
|
||||
19. ✅ **RESOLVED** — exited shells remove themselves 10 minutes after exit (verified live: seeded 17:53:56, gone at 18:03:59 with no daemon restart in between). Found and fixed alongside: a daemon restart used to re-spawn *exited* shells as fresh running ones; only running shells are persisted now. Idle running shells are deliberately kept. **Terminals accumulate.** Nothing ever reaps exited or idle sessions; the bash left
|
||||
several "bug-bash-term" shells on the hub until deleted by hand. Exited sessions
|
||||
should auto-remove after a while; idle ones at least be counted somewhere.
|
||||
20. ✅ **RESOLVED** — a node answers 409 "busy" when a package task is running or a terminal is attached; the Machines page offers Force, `hamlab push-nodes --force` / `make push-nodes-force` do the same. Verified: 409 with a terminal attached on grid, 200 with force. **Self-update restarts without warning running work.** A push to a node kills its
|
||||
open terminals and any running task (upgrade in progress). The hub should refuse or
|
||||
warn when the node reports a running task or attached terminals.
|
||||
21. ✅ **RESOLVED** — PathError and LinkError are unwrapped to the bare reason. **Error text leaks raw Go errors.** e.g. rename shows "rename /tmp/x /tmp/y: no such
|
||||
file or directory". Fine for a lab, ugly for a product; strip the syscall prefix.
|
||||
|
||||
## Mobile / UX
|
||||
|
||||
22. ✅ **RESOLVED** — on coarse pointers every button, chip, icon button, link, header and select is at least 36 px tall (verified: no control under 32 px on the dashboard or processes pages). **Tap targets under 28 px on phone:** header ⏻ (25), logout (22), customize ▲ ▼ hide
|
||||
(23), terminal leaf ✎ ✕ (24), chips (25), table headers (27). Apple's guideline is
|
||||
44 pt. Systemic: the small `.btn`/`.chip`/`.icon-btn` sizes need a touch variant.
|
||||
23. ✅ **RESOLVED** — the dialog asks for the machine name (`mac`); the node still checks its own hostname under the hood. **Power dialog on the Mac asks you to type `5ce91ee54af6`.** The confirmation uses
|
||||
the node's hostname, which on the Mac is the unset default. Should offer the machine
|
||||
name (`mac`) as the confirmation word on remote machines.
|
||||
24. ✅ **RESOLVED** — down interfaces collapse to one line marked "down"; modules with nothing to show (fans, UPS, temps, disks) hide unless customizing. **Dashboard shows dead interfaces and empty modules.** wlp1s0 (down) gets a full
|
||||
sparkline block of flat lines; "Fans: no fan sensors" is a permanent empty card on
|
||||
hosts without fans. Down interfaces should collapse to one line; empty modules should
|
||||
auto-hide unless customizing.
|
||||
25. ✅ **RESOLVED** — "load 1.39 1.56 1.55" with a tooltip. **Header load average is unlabelled** ("1.39 1.56 1.55" floating in the header).
|
||||
26. ➖ **NOT FIXING** — the list is intentionally a plain table; the menu anchors inside the row. Cosmetic taste, not a defect. **Files list on desktop is a narrow strip in a sea of empty space** for small
|
||||
folders; the row menu opens off the right edge of the row. Cosmetic.
|
||||
27. ✅ **RESOLVED** — after a 502 the client leaves that machine alone for 5 s: 2 requests in 10 s at a 250 ms refresh (would have been ~80), badge and message intact. **Offline machine keeps polling every view at the refresh rate**, producing a 502
|
||||
per request per page (and console errors). The UI copes — badge, message — but the
|
||||
hub should back off to the prober's cadence while a node is down.
|
||||
28. ✅ **RESOLVED** with 6. **Services logs are noise.** The default view of `hamlab.service` logs is the sudo
|
||||
session spam from bug 6, three lines per poll.
|
||||
|
||||
## Confirmed working under abuse (for the record)
|
||||
|
||||
Path traversal in files, services, cron, ZFS, procs and updates all rejected. `/etc`
|
||||
and `/` protected from delete/rename as root. Unit, pool, dataset, snapshot and
|
||||
package names validated. Token issue refused with a wrong password. Oversized prefs
|
||||
rejected. Duplicate machine add refused. Unicode + spaces in file names round-trip.
|
||||
Upload byte-exact. Terminal scrollback survives switching. Offline node: switcher and
|
||||
header show it, pages show the reason. Remote terminals, logs, tasks and files through
|
||||
the proxy all work on all three nodes.
|
||||
|
||||
|
||||
# Bug bash — 2026-09-19
|
||||
|
||||
Found by driving a scratch instance (same binary, its own state, real fleet behind it) in
|
||||
headless Chromium at 1440 px and iPhone size: a sweep of every page on every machine for
|
||||
page errors, failed requests, overflow and small tap targets; then scripted use of every
|
||||
feature — terminal tabs, Files in a throwaway folder, Processes (killing a process the
|
||||
test spawned), Monitors customisation and widgets, Services, Cron, Updates, Disks,
|
||||
Machines, Settings, embeds, machine switching, session loss — and a hostile pass over the
|
||||
API. Host-changing actions (service control, cron edits, upgrades, power) were opened but
|
||||
not executed. Every fix below was re-checked by the same script that found it.
|
||||
|
||||
## Bugs — all ✅ fixed and verified
|
||||
|
||||
1. ✅ **One widget with a bad pattern silenced every custom widget.** They share one stream;
|
||||
`fan.[` made the whole query a 400, so all of them showed "…" forever with no reason
|
||||
given. The picker now asks the API before adding and shows its answer; a bad pattern
|
||||
already saved shows a notice instead of dots.
|
||||
2. ✅ **A saved layout of the wrong shape could not be loaded** ("could not load layout",
|
||||
and the page could no longer be customised). `normalize()` now accepts anything.
|
||||
3. ✅ **A lost session left pages saying "unauthenticated"** for up to 30 s. A 401 from the hub
|
||||
now returns to the login form at once, and logging in returns to the same page. (A 401
|
||||
relayed from a managed machine is that machine rejecting the hub, and doesn't.)
|
||||
4. ✅ **The machine-name rule in "Add a machine" was an invalid regular expression** under the
|
||||
browser's `v` mode, so it validated nothing and logged a console error on every render.
|
||||
5. ✅ **Two terminals opened in the same minute got the same name** ("shell 09:45" twice).
|
||||
Now "shell 09:45 (2)".
|
||||
6. ✅ **Names had no length limit** — a 328-character terminal tab, a 5000-character token
|
||||
name. 64 characters, refused with a reason; token names also refuse `/ \ < > "`.
|
||||
7. ✅ **"+ file", "+ folder" and Rename accepted `../x` and `a/b`**, quietly acting outside the
|
||||
folder on screen. A name is now one path segment; Move… is how things go elsewhere.
|
||||
8. ✅ **Unsaved file edits were thrown away without a word** when you picked another page or
|
||||
machine, or closed the tab. All three now ask; Cancel keeps you where you were.
|
||||
9. ✅ **A mistyped path in Files was remembered**, so a reload landed on the error. Only a
|
||||
folder that opened is remembered.
|
||||
10. ✅ **Escape closed nothing**: the machine menu, the file menu, the Power dialog, the widget
|
||||
picker. It closes all four.
|
||||
11. ✅ **Reset on Monitors removed every custom widget with no confirmation.** It asks, and
|
||||
says how many.
|
||||
12. ✅ **A few bytes of query bought 27 MB of answer** (`{"k0":"*",…×400}` with `full=1`) — and a
|
||||
read-only token is public on the demo page. A query may return 10 000 values (thirty
|
||||
whole machines); past that it is a 400 that says so.
|
||||
13. ✅ **No ceiling on live streams**: 300 from one token were accepted. 32 per client address,
|
||||
512 in all, then a 429 that says so; closed streams free their places.
|
||||
14. ✅ **Anything under `/embed/` lost the anti-framing headers**, not just the widget page.
|
||||
Only `/embed/widget` is frameable.
|
||||
15. ✅ **The new ZFS sensors ran three `sudo` commands a minute on every machine** — including
|
||||
`core`, which has the tools and no pool — and each wrote three lines to the journal:
|
||||
most of `hamlab.service`'s log was that. Pools are now read with one unprivileged
|
||||
`zpool list`; a source that finds nothing is asked again ten times less often; and
|
||||
the installer's quiet list covers `zpool|zfs list|status`.
|
||||
16. ✅ **Phone: the filter boxes on Processes and Files, the path box and the "/" crumb were
|
||||
under 30 px.** 36 px like every other control.
|
||||
17. ✅ **Disks → datasets on a phone: rows ended 50 px before their own cells** when scrolled
|
||||
sideways (`min-width` was less than the columns add up to).
|
||||
|
||||
Also: tab titles now say where you are ("Services · grid — Ham Lab"), so history and open
|
||||
tabs can be told apart.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Terminal focus after every switch/close/return; scrollback after reload; exited shells;
|
||||
30 tabs with the + still reachable; re-fit on resize; Back/Forward; unknown machine or page
|
||||
in the URL; wrong password; XSS through terminal, token and file names (all rendered as
|
||||
text); hidden files, filter, symlinks (broken ones marked), binary files, upload and
|
||||
replace, folder delete needing the typed name, root toggle; process sort, filter and
|
||||
Terminate (asks first); themes; password-change validation; duplicate or unreachable
|
||||
machine; hub terminals not leaking into another machine's page; CORS only on sensor paths;
|
||||
token list never exposing hashes; cookie flags; deep nesting, empty bodies, odd methods
|
||||
and `every=` values on the sensor API; phone drawer, key bar and picker.
|
||||
|
||||
## Quality of life — all ✅ done 2026-09-19, each verified in the browser
|
||||
|
||||
1. ✅ **Selections are in the URL.** `#/grid/services/ssh.service`, `#/hub/cron/job:…`,
|
||||
`#/hub/files/%2Fetc`: reload keeps them, links carry them, leaving a page drops them.
|
||||
2. ✅ **Browser Back with unsaved file edits asks**, like page and machine changes do.
|
||||
3. ✅ **No more browser `prompt()`/`confirm()` boxes** — all 30 are in-app dialogs (`askYes`,
|
||||
`askText` in `ui.tsx`): Enter submits, Escape or a tap outside cancels, destructive
|
||||
ones are red and say what they do ("Close", "Delete"), and they fit a phone.
|
||||
4. ✅ **Token creation is one dialog**: name, patterns, and the machine's own sensor families
|
||||
as chips to click.
|
||||
5. ✅ **Widgets can be edited** — sensor, type and title — in place, from Customize.
|
||||
6. ✅ **The Widgets gallery has a search** over IDs, names and labels (so "gitea" finds its
|
||||
container's sensors).
|
||||
7. ✅ **Status sensors keep a history** (300 readings each: an hour of container memory, two
|
||||
days of drive wear), so their sparklines have a past.
|
||||
8. ✅ **A stream that fails says why**: "the token was rejected", "too many live streams are
|
||||
open", "that machine is unreachable", "can't reach Ham Lab" — in embeds and on Monitors.
|
||||
9. ✅ **Pattern widgets get readable default titles**: `cpu.core.*.usage` → "CPU core usage".
|
||||
10. ✅ **Terminal tabs reorder** — drag on a desktop, ‹ › on a touch screen, remembered per
|
||||
machine in the browser — and **Alt+Shift+←/→** switches tabs, **Alt+Shift+T** opens one,
|
||||
with the keyboard left in the terminal each time.
|
||||
11. ➖ **"Log out everywhere"** — built, then removed the same day at Mitchell's word: no need
|
||||
for it on a single-user console.
|
||||
12. ✅ **Real clipboard shortcuts in the terminal**, from a physical keyboard: **Ctrl+C copies
|
||||
when text is selected** (and is still ^C when nothing is — the selection clears after a
|
||||
copy, so a second Ctrl+C interrupts), **Ctrl+V pastes**, Ctrl+Shift+C / Ctrl+Shift+V do
|
||||
the same unconditionally, Ctrl+Insert / Shift+Insert too. It rides the browser's own
|
||||
copy and paste events, which xterm already answers, so it works over plain http on the
|
||||
LAN where `navigator.clipboard` does not exist; bracketed paste is honoured.
|
||||
|
||||
|
||||
# Bug bash 2 — 2026-09-19 (evening): Alerts, Docker, Stacks, skins, the installer
|
||||
|
||||
Same method as the morning: a scratch instance with the real fleet behind it, headless
|
||||
Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four machines (errors,
|
||||
failed requests, overflow, tap targets) plus a contrast measurement of every skin; then
|
||||
scripted use of the new features on throwaway containers and stacks — a multi-file stack, a
|
||||
file edited behind the editor's back, two actions at once, a chatty log, odd container
|
||||
names, containers stopped and removed while open, every kind of bad alert rule, a rule's
|
||||
whole life (fire, narrow, pause, delete, survive a restart), token scope and hostile paths
|
||||
on the new APIs — and a regression of terminals, Files, widgets, deep links and colours.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **Deploying a stack made of several compose files dropped all but the first.** A stack
|
||||
deployed with `-f compose.yaml -f compose.override.yaml` was redeployed from
|
||||
`compose.yaml` alone, so everything the override set silently vanished from the new
|
||||
container. Two causes: only the first file was used, and the file list was read from
|
||||
`docker ps`, which joins labels with commas — the list is itself comma-separated, so it
|
||||
arrived truncated. Labels now come from `inspect`; every file is resolved, shown as a
|
||||
chip, editable, validated together and passed to compose in order.
|
||||
2. ✅ **A compose file changed on disk after you opened it was silently overwritten** (Dockhand,
|
||||
or a second tab). The editor now hands back the checksum it was given; a mismatch is a
|
||||
409 that says so, keeps your text, and offers to reload.
|
||||
3. ✅ **Stacks on a machine without Docker answered 500.** An empty list.
|
||||
4. ✅ **Every hub restart re-fired every firing alert** — and every deploy is a restart, so a
|
||||
phone would have been told about the same failed service on each one. What is firing is
|
||||
now kept in `state.json` and carried on, with its original start time.
|
||||
5. ✅ **An alert whose rule was paused, deleted or narrowed just vanished**, leaving "firing" in
|
||||
the history forever and no word to the phone. It is now recorded (and sent) as cleared,
|
||||
with the reason: "rule paused", "rule deleted", "no longer watched".
|
||||
6. ✅ **Nonsense rule values were accepted**: `null`, an object, `cpu.usage > ninety` (which can
|
||||
never be true, and would never say so), `true` with `<`. Refused, each with its reason.
|
||||
7. ✅ **No limit on rules** — 3000 were accepted, each checked on every machine every 15 s. 200.
|
||||
8. ✅ **Logs of a container that no longer exists answered 500.** 404.
|
||||
9. ✅ **Four skins had text under 3:1 contrast**: Nord's red; Catppuccin Latte's muted text and
|
||||
green; Solarized Dark's body text, muted text and red; Solarized Light's body text,
|
||||
muted text, accent and green. Each colour kept its hue and was moved to a readable value
|
||||
(measured: all thirteen skins now pass, body text ≥ 7:1).
|
||||
10. ✅ **Phone: the alert toggles (28 px) and the colour swatches (26 px) were under 30 px.**
|
||||
11. ✅ **The public installer had fallen four builds behind the fleet** (v65 published, v69
|
||||
running), so `curl … | sh` on an up-to-date machine "upgraded" it backwards. `make
|
||||
deploy` now republishes, and the installer compares versions: an older build is called
|
||||
older, defaults to No, and unattended runs refuse it without `HAMLAB_ALLOW_DOWNGRADE=1`.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
A second stack action while one runs (409); `../` and encoded tricks in stack names;
|
||||
shell metacharacters in container names, `since` and actions; `rm` / `kill` / `down` are
|
||||
not actions; a read-only token is refused on every Docker, Stacks and Alerts endpoint,
|
||||
directly and through the hub; container names with dots and underscores still get live
|
||||
CPU and memory; a log at five lines a second follows with no gaps or duplicates; a
|
||||
container stopped or removed while open updates itself; an alert rule's XSS name renders
|
||||
as text; per-machine firing, the rail badge, webhook and ntfy delivery; terminals, Files,
|
||||
widgets over container sensors, deep links, tab titles and machine colours all unchanged.
|
||||
No native browser dialogs anywhere; no page errors on any of 100 page loads.
|
||||
|
||||
## Found by the alerts themselves
|
||||
|
||||
`core` has one real alert firing: **`zfs-import-cache.service` fails at every boot** — `core`
|
||||
has the ZFS tools and no pool, so there is nothing to import. Harmless, but it will sit on
|
||||
the rail as "1" until the unit is disabled or the rule is narrowed. Left alone.
|
||||
|
||||
## Quality of life — five chosen by Mitchell, ✅ built and verified 2026-09-19
|
||||
|
||||
1. ✅ **Stacks on `grid` and the Mac are editable from the hub.** Their files live on `core`
|
||||
(Dockhand deploys over tcp); the hub resolves them and runs compose with `DOCKER_HOST`
|
||||
pointed at the node. Verified on a throwaway stack: saved on `core`, recreated **on
|
||||
`grid`**, same compose labels, nothing created on `core`, output followed on the page.
|
||||
2. ✅ **A stack's `.env` is editable**, values masked until "Reveal and edit". A `.env` that
|
||||
breaks the compose file (a required variable removed) is refused and the old one put back.
|
||||
6. ✅ **A link to a container that no longer exists says so**, by name.
|
||||
9. ✅ **"Update available" per image** — on request, remembered six hours. Verified with an
|
||||
image whose tag had moved on (update available) beside a current one (up to date).
|
||||
10. ✅ **What Deploy would change**: a line diff against what was last deployed from here;
|
||||
after a deploy it says there is nothing left to change. `.env` diffs name the key and
|
||||
never show a value.
|
||||
|
||||
Not chosen, still open: alert acknowledge/snooze; quiet hours and per-rule channels; push
|
||||
straight from Ham Lab; a type hint in the alert editor; switching off the nodes' own alert
|
||||
engines.
|
||||
|
||||
# Bug bash 3 — 2026-09-20: new stacks, new containers, images, volumes, the HUD
|
||||
|
||||
Started from Mitchell's report — the New stack dialog is see-through in HUD, its text
|
||||
mixing with the page behind. Same method as before: a scratch hub with the real fleet behind
|
||||
it, headless Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four
|
||||
machines, plus a measurement **in every skin** of everything that floats over the page
|
||||
(dialogs, the question box, the widget picker, menus, the phone drawer: background alpha
|
||||
must be ≥ 0.9). Then scripted use of what is new on throwaway `hl-` stacks, containers,
|
||||
images and volumes: the New stack / New container dialog's edges, stacks that are not
|
||||
deployed, a deploy that fails, a container name that is taken, deletes beside other
|
||||
people's files, a two-file stack, create → deploy → restart → stop → delete **on the Mac
|
||||
and on `grid` through the hub**, pulls on the hub and on a node, removal guards, a
|
||||
container stopped on purpose, a read-only token and hostile input on every new endpoint,
|
||||
and a regression of Escape everywhere, terminals, Files, sensors, alerts and deep links.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **HUD: every dialog was see-through.** The skin's "glass" rule (3 % cyan) was written
|
||||
for tiles and cards and also caught `.modal` — the New stack dialog, the widget picker,
|
||||
the Power dialog, the token dialog, the rule editor and every question. Anything that
|
||||
floats over the page is now solid in HUD (hairline border and glow kept) over a darker
|
||||
backdrop. Measured in all thirteen skins: nothing that floats is under 0.9 alpha.
|
||||
2. ✅ **HUD: the phone's slide-out menu was 92 % opaque**, the page showing through it. Solid.
|
||||
3. ✅ **Deploy, Update, Restart, Stop and Down all failed on the Mac's stack**
|
||||
(`peaches-and-plums`) — and on any stack held on the hub that was deployed from the hub's
|
||||
own shell rather than from inside Dockhand. The command was built as
|
||||
`'DOCKER_HOST=tcp://…' docker compose …`; quoted, `sh` reads that not as a variable but
|
||||
as the name of a program to run ("not found", exit 127). Now passed through `env`.
|
||||
Verified by a whole life on the Mac and on `grid` from the hub: create, deploy (running
|
||||
there, nothing on `core`), restart, stop, delete.
|
||||
4. ✅ **A container stopped on purpose was reported as down** when it has no restart policy
|
||||
and doesn't handle SIGTERM: `docker stop` then ends it with 137 (or 143, or 130), a
|
||||
non-zero code, which read as a crash — "Container down" would have fired for a
|
||||
deliberate stop. Those three codes now mean "a person", unless the kernel did the
|
||||
killing for want of memory (`OOMKilled`). A real crash (exit 3) is still down.
|
||||
5. ✅ **A stack with no containers offered Restart, Stop and Update**, and Restart
|
||||
"succeeded" having done nothing. Offered only when there is something to act on; the
|
||||
API answers 409 "isn't deployed: there is nothing to restart".
|
||||
6. ✅ **One stray tap outside the New stack dialog threw away everything typed** — on a phone,
|
||||
a compose file's worth. So did Escape. With anything typed, all three ways out (tap
|
||||
outside, Escape, Cancel) ask first; an untouched dialog still just closes.
|
||||
7. ✅ **Compose → Form → Compose silently replaced hand-edited compose text** with what the
|
||||
form says. The form now writes the box only when the form has changed since it last
|
||||
did, and asks before replacing text edited by hand.
|
||||
8. ✅ **A question asked from inside a dialog opened underneath it** (same layer, earlier in
|
||||
the page), so it could not be answered — found the moment the New stack dialog started
|
||||
asking. Questions now sit above whatever asked them.
|
||||
9. ✅ **Escape on a question also reached the dialog beneath it**: one key cancelled the
|
||||
question *and* closed (or re-asked from) the dialog under it. Escape now answers the
|
||||
question and goes no further; the next Escape is the dialog's. Checked on every dialog
|
||||
and menu: widget picker, machine menu, rule editor, Power, New stack, and that Escape on
|
||||
"Delete the rule?" deletes nothing.
|
||||
|
||||
10. ✅ **Phone: the bottom of the New container dialog could not be reached** (Mitchell, from
|
||||
his iPhone). A dialog is a column whose parts were allowed to shrink, so instead of the
|
||||
dialog scrolling, the one part with its own scrollbar — the compose preview — was
|
||||
squeezed to 22 px and everything "fitted". And `92vh` on iOS is taller than the screen
|
||||
with the address bar gone, so the top sat under the clock. Parts now keep their height
|
||||
and the dialog scrolls inside itself; it is never taller than the space between the
|
||||
status bar and the home indicator (safe-area insets, no `vh`); Create / Cancel — and
|
||||
any error — stay pinned at the bottom while the form scrolls. Applies to every dialog;
|
||||
measured at 390×844, 375×667 and sideways, and the picker, rule editor, token and Power
|
||||
dialogs rechecked.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
All 13 pages × four machines × both sizes: no page errors, failed requests, sideways
|
||||
scroll or tap targets under 30 px. A port of `not-a-port` is refused by compose before
|
||||
anything is written, and leaves no folder. Environment values with quotes, `#`, `$` and
|
||||
nothing after the `=` arrive in the container exactly as typed. A deploy whose image
|
||||
doesn't exist shows the registry's error and is **not** remembered as "last deployed". A
|
||||
container name already taken fails the deploy and says why. Delete moves only the stack's
|
||||
own compose files and `.env` (kept `600` in the trash) and leaves a `data/` folder and
|
||||
notes beside them alone; both files of a two-file stack go to the trash; the wrong name,
|
||||
or none, deletes nothing. A `.env` value never appears on the stack page and stays masked
|
||||
until asked for. A second stack of the same name is 409; an upper-case name is 400. A pull
|
||||
of an image that doesn't exist ends as a failed task; a pull on `grid` from the hub lands
|
||||
on `grid` and its output is followed. A volume or image in use can't be removed; a running
|
||||
container can't be removed (409); a volume needs its name twice. A read-only token gets
|
||||
401 from all fourteen new write paths, through the hub as well. `--force`, `-f`,
|
||||
`--all-tags`, `..%2F..` and a home outside the known ones are refused; a 1.2 MB compose
|
||||
file is refused. Shaped, fleet and streamed sensor queries, `/metrics`, the ten alert
|
||||
rules, terminal typing, the file editor's discard question and service deep links are
|
||||
unchanged. No native browser dialogs anywhere.
|
||||
|
||||
## Quality of life — four chosen by Mitchell, ✅ built and verified 2026-09-20
|
||||
|
||||
1. ✅ **"Where" fits on a phone**: `…/stacks/core · 12 stacks` instead of the whole path (which
|
||||
is still written out underneath). Two homes whose last two folders match fall back to the
|
||||
full path. Measured at 390 px: no option is wider than the box.
|
||||
2. ✅ **How a task ended is a badge**: green `finished`, red `failed · exit 1` with the box
|
||||
outlined in red, grey `cancelled`. Verified on a deploy of an image that doesn't exist.
|
||||
5. ✅ **Duplicate…** on a stack: the New stack dialog opened on a copy of its compose file,
|
||||
named `<name>-copy`. The original's `.env` is copied **on the machine** (`env_from`), mode
|
||||
600, its values never sent to the browser — checked: the secret is nowhere in the page.
|
||||
Anything that would collide with what is running is named before you press Create
|
||||
("a container called X already exists; port 18222 is already published by X") — for
|
||||
every new stack, not only duplicates — and clears as you fix it. Disabled while the
|
||||
original has unsaved edits. Verified: copy deployed beside the original, which never
|
||||
stopped.
|
||||
6. ✅ **The form has Labels and Network.** Labels one per line (`$`, quotes and spaces arrive
|
||||
intact). Network: its own, the machine's (`host`, which turns Ports off), or **join** one
|
||||
that already exists — written as an `external` network so compose doesn't make a second
|
||||
one. Verified on the container: both labels, attached to the chosen network only.
|
||||
|
||||
Found while building them, ✅ fixed:
|
||||
|
||||
- **Cancel on a stack task cancelled nothing.** It killed the `sh` that ran compose, not
|
||||
compose; and as compose still held the output pipe, the task stayed "running" and the
|
||||
stack "busy" until compose finished by itself — for a deploy waiting on a health check,
|
||||
minutes. Tasks now run in their own process group and the whole group is stopped (TERM,
|
||||
then KILL after 4 s); for stacks deployed inside Dockhand, where the process is not the
|
||||
task's child at all, the cancel goes into the container after it. Verified both ways:
|
||||
ended in ~100 ms, compose gone, the stack free for the next action.
|
||||
- **Phone: a stack's task output was squeezed to a line** under the buttons (the same fault
|
||||
as the dialog: a scrolling column whose parts were allowed to shrink), which also put
|
||||
Cancel out of reach. Detail panes keep their parts' heights; services, containers, stacks
|
||||
and cron rechecked on both sizes.
|
||||
- The Duplicate dialog and the stack page under it shared a React key, so the dialog
|
||||
multiplied. An undeployed stack's dot was green ("0 of 0 running"); now neutral.
|
||||
|
||||
# Bug bash 4 — 2026-09-21: the Top tab, and the refresh rate it didn't follow
|
||||
|
||||
Started from Mitchell's first look at Top on his own machine (Firefox on Windows, a window
|
||||
1392 CSS px wide — none of which the tab had been drawn for): "Command" printed over
|
||||
"Threads", the rate in Settings ignored, the symbol beside *logout* a box of hex digits;
|
||||
and make it follow the skins. Method as before, plus two things new: **Firefox** (Playwright's
|
||||
build; the GTK libraries it wants are unpacked from their `.deb`s into
|
||||
`~/.cache/ms-playwright/fflibs` and given to it as `LD_LIBRARY_PATH`, with
|
||||
`PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1` — nothing is installed on `core`), and a layout
|
||||
checker that measures instead of looking — every heading against its neighbour and its own
|
||||
text, every row against its box, every border label, at eight window sizes from 900 to 1920.
|
||||
Run first against the live v83 it reproduced the screenshot to the pixel ("the Command column
|
||||
is 11px wide" at 1392); run after, nothing, in both browsers.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **Between about 1280 and 1500 px the Command column collapsed to a letter** and its
|
||||
heading ran under "Threads:" (itself 4 px too narrow for its own text at every size). The
|
||||
table's fixed columns were sized for one window. The process box now measures *itself*
|
||||
(a container query — it is a third of a wide window and all of a narrow one, so the
|
||||
window's width says nothing) and drops the curve, then Threads, then Command, as it
|
||||
narrows; the box also gets half the width rather than 44 %, and no heading can spill
|
||||
into the next. Stacked layout now starts at 1280, not 1100.
|
||||
2. ✅ **The refresh rate reached a machine only through its Dashboard page.** The sampler on
|
||||
each machine was told the rate by an open Dashboard and nothing else — so with the rate
|
||||
at 250 ms, `grid`, `wyse` and the Mac were still sampling at 1 s (measured, live), and
|
||||
Top, Processes, Disks and every widget of theirs "ignored" the setting until somebody
|
||||
opened that machine's Dashboard. The rate is now the fleet's: saved with the UI prefs on
|
||||
the hub, which sets its own sampler and passes it to every machine (`PUT
|
||||
/api/prefs/sampler`, remembered in each `state.json`), and says it again at start-up for
|
||||
a node that restarted or joined since. Verified: Settings 2 s → 250 ms moves the sampler
|
||||
with no Dashboard visit.
|
||||
3. ✅ **Top read the process table at most once a second whatever the rate** — a clamp of
|
||||
my own, exactly what `REFRESH` promises not to do. It reads at the rate: 20 reads in 5 s
|
||||
at 250 ms, 10 at 500 ms. (59 frames a second with 390 rows at 250 ms, no stalls, both
|
||||
browsers — the rows are only redrawn when the table is read.)
|
||||
4. ✅ **Top's `− rate +` changed the rate for half a minute.** It set the browser's copy
|
||||
and never saved it, so the next sync with the hub put it back. It saves — which is also
|
||||
what moves the samplers — survives a reload, and each end is disabled at the limit.
|
||||
5. ✅ **A viewer whose clock is wrong froze the curves.** New points were timed against the
|
||||
machine's timestamps: a browser clock 7 minutes slow waited 7 minutes for its next point
|
||||
(Top *and* the Dashboard); 7 fast, and it added one per tick. Timed on the browser's own
|
||||
clock now. Verified with the page's clock shifted both ways: 5 points in 10 s at a 2 s
|
||||
stride, both pages, both browsers.
|
||||
6. ✅ **Every other point was being skipped.** "Has a stride passed?" was asked with `>=`
|
||||
of a timer that fires a few ms early half the time — so at a 2 s rate and a 2 s stride a
|
||||
curve gained 3 points in 10 s, not 5, and its time axis was quietly wrong. The same
|
||||
question on the server kept every bucket one sample long (14 s points at a 12 s stride).
|
||||
Both now allow half a tick of slack and keep to the stride's own beat; the server times
|
||||
from the last point kept, so a stalled or sleeping machine just keeps the next sample it
|
||||
gets. Unit-tested with jittered samples: 240 samples → exactly 40 points, 12 000 ± 100 ms
|
||||
apart, each the mean of its own six.
|
||||
7. ✅ **The power symbol was a typed `⏻`** (U+23FB, Unicode 9) — not in Windows' fonts, so
|
||||
Firefox there drew the missing-glyph box. Drawn as an SVG like the other icons. Every
|
||||
other symbol the console types was checked against the same risk: all older.
|
||||
8. ✅ **A handful of samples was stretched into a bar chart.** A machine ten seconds into
|
||||
its history filled the whole CPU box with five fat columns. The box is now always the
|
||||
whole history window — a young curve draws in from the right, as btop's does, and a full
|
||||
one fills it edge to edge, at any window (checked at 10 min and at 1 min).
|
||||
9. ✅ **Idle curves vanished**: a core at 10 % rounded to no dots at all in its mini graph.
|
||||
Anything above zero lights the bottom row, which is also btop's rule.
|
||||
|
||||
10. ✅ **Found by the deploy of fix 2 itself: telling the machines once is not enough.** The hub
|
||||
restarted first and told three nodes that did not yet have the endpoint (404); they then
|
||||
updated and carried on at 1 s. The same would happen to any node that is off or
|
||||
restarting when a setting changes. The hub now *keeps* the fleet in step: fifteen seconds
|
||||
after it starts and every two minutes after, it asks each machine for its sampling rate
|
||||
and history window and corrects only what differs (logged as `fleet: grid had ms=1000,
|
||||
now 250`).
|
||||
|
||||
11. ✅ **The net graph redrew its own past every second** (Mitchell, two screenshots a second
|
||||
apart: one tall spike in the first, a comb of medium ones in the second). That box is
|
||||
about 75 dot-columns wide and holds 300 points, and each column drew *the nearest point*
|
||||
— one in four, and a different one in four each time the curve advanced, so spikes came
|
||||
and went and the whole shape changed while the scale stayed put. The CPU graph did the
|
||||
same on a phone. Now a column draws the **peak** of all the points in it, and columns are
|
||||
cut from the data, not from the box (`web/src/graphcols.ts`): a finished column never
|
||||
changes, the graph only slides left by whole columns. Measured by reading every column's
|
||||
height off the canvas once a second under bursty traffic: on the live build the past
|
||||
changed shape in 18 of 47 frame pairs; after, 0 of 46 — and 0 of 45 on a young history.
|
||||
`npm test` (now part of `make test`) scrolls a spiky curve through both rules: nearest-
|
||||
point shows its three spikes as "900,500", then "700", then none; the new rule shows all
|
||||
three, at their heights, on every frame. The scale label now covers what is drawn (the
|
||||
window), not everything kept.
|
||||
|
||||
## Skins
|
||||
|
||||
✅ Top now wears the skin. The ground is the skin's terminal background and every curve and
|
||||
meter is built from its sixteen terminal colours (chosen to be read on exactly that ground),
|
||||
the rest from the usual variables — so the dark skins look like btop and the light ones like
|
||||
btop on paper, and a skin arriving from another device repaints the page in place. Measured
|
||||
in all thirteen: text ≥ 4.5 : 1, everything else ≥ 3 : 1, dots ≥ 2.2 : 1. Two failed —
|
||||
process names in Latte (2.75) and Solarized Light (2.62), where a terminal green is too
|
||||
pale for small type — and use `--ok`, the green each skin was already measured readable
|
||||
with.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Sort by pid, name, mem and cpu, and reverse; filter by name, command, user and pid, with the
|
||||
count showing matched/total and an empty result saying so; a selection that holds while the
|
||||
table is re-read under it; terminate, on a throwaway `sleep`, asks by name and pid, ends it,
|
||||
and the footer stops offering a process that is gone. Sampler values outside 100 ms–60 s are
|
||||
refused, and a rate slower than the collector's 10 s samples at 10 s. All 14 pages at his
|
||||
window size and on a phone, in both browsers: no errors, no overflow.
|
||||
|
||||
|
||||
# Bug bash 4, later — 2026-09-21: the SMART table outside its card
|
||||
|
||||
Mitchell, from the Disks page of `hamtv` with SMART open: the attribute table's rows ran out
|
||||
through the right-hand side of the card they belong to, the rules and the Thresh column
|
||||
drawn over the page behind.
|
||||
|
||||
12. ✅ **A table cannot be narrower than its widest unbreakable cell, and one of `hamtv`'s
|
||||
was 130px wide.** `.num` sets `white-space: nowrap`, and a SMART raw value is not always a
|
||||
number — the temperature attribute reads `34 (Min/Max 28/39)`. That one cell set the
|
||||
table's minimum width above the card's, so it spilled. Three changes, in this order,
|
||||
because the first two each looked worse than the bug in their own way:
|
||||
- the raw column wraps — but `overflow-wrap: anywhere` broke `4295163906` across two
|
||||
lines as "429516 / 3906", which reads as two numbers. It wraps at spaces only now;
|
||||
- a disk card is given room for six columns of SMART (`minmax(420px, 1fr)` rather than
|
||||
the shared 320px), and the cells' side padding goes 6px → 4px, which together is the
|
||||
~40px it was short by;
|
||||
- if some drive ever reports a raw value long enough to still not fit, the table scrolls
|
||||
inside its card rather than out of it.
|
||||
Verified on all four machines at six widths from 1721px down to a phone, in Chromium and
|
||||
Firefox: nothing leaves its card, no number is broken across lines, and the fallback
|
||||
scrollbar is never reached. The same measurement was run over every other page's cards
|
||||
and boxes — Dashboard, Top, Docker, Alerts, Machines and the rest — and found nothing.
|
||||
|
||||
# Bug bash 5 — 2026-09-21: navigation, and the fleet setting nobody had set
|
||||
|
||||
A sweep of the whole console after a week of changes, not of any one page: every page on every
|
||||
machine at four widths in **Chromium and Firefox**, then the everyday flows, then the URL.
|
||||
Measured rather than looked at — a heading against its neighbour and its own text, a box
|
||||
against what it holds, a request against the page that should have made it.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **A link to a page of the hub reloaded onto whichever machine you last had open.** The hub
|
||||
is `null` internally, so `parseHash()?.machine ?? sessionStorage…` read straight past it to
|
||||
the remembered machine: open `#/hub/docker/stack:gitea` after looking at `grid`, reload, and
|
||||
you were on `#/grid/docker`. The URL wins when it has an opinion, and "hub" is an opinion;
|
||||
the remembered machine is for a bare address only. Every link shape now survives a reload,
|
||||
and a bare address still lands where you left off.
|
||||
2. ✅ **From Settings, Machines or Alerts, a link to a machine's page arrived at its Dashboard.**
|
||||
Those three pages only exist on the hub, so moving to a machine bounces off them — but the
|
||||
bounce was decided from inside the machine change, which read the page being *left*, and
|
||||
landed after the URL's own page had been set. It is judged now once the machine and the page
|
||||
have both settled. `#/hub/settings` → `#/grid/disks` arrives at Disks; `#/grid/settings`
|
||||
still falls back to that machine's Dashboard.
|
||||
3. ✅ **A page being navigated away from briefly adopted the next page's selection**, and asked
|
||||
about it: leaving `#/grid/services/ssh.service` for `#/hub/docker/stack:gitea`, Services —
|
||||
still mounted for a moment — opened a log socket for a *service* called `stack:gitea`, on
|
||||
`grid`. Firefox says so in the console; Chromium swallows it. A page now reads the URL's
|
||||
selection only while the URL still points at that page. Checked over every hop between the
|
||||
five pages that keep a selection in the URL, in both browsers.
|
||||
4. ✅ **The hub only held its machines to a sampling rate if somebody had been into Settings.**
|
||||
The re-assert loop read the rate out of the saved UI prefs, which are written on change and
|
||||
not before — so on a hub where the rate had never been touched, a node left on a different
|
||||
rate was never pulled back. It compares against what the hub itself is running, which always
|
||||
exists. Verified by setting a node behind the hub's back: corrected within seconds.
|
||||
5. ✅ **"Everything is up to date." sat in a 20px sliver of a card**, flush to its border, where
|
||||
the package list it shares the card with is meant to be.
|
||||
|
||||
6. ✅ **A terminal of the hub's was asked for on a node.** Found in Firefox against the live
|
||||
console, which says out loud what Chromium swallows: `wss://…/api/nodes/grid/terminals/
|
||||
adfdeecb/attach`, where `adfdeecb` was one of the hub's — `grid` had no terminals at all. A
|
||||
pane builds its address when it connects, from whichever machine the console is pointed at
|
||||
*now*; switch machine while one is reconnecting and the reconnect goes to the new one. The
|
||||
address is fixed when the pane opens. The same was true of a service's log socket, and is
|
||||
fixed with it. Verified by walking the console around the fleet with a terminal open and
|
||||
checking every socket against the machine that actually owns what it asked for.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Machine titles on every page and against an offline or never-reached machine; the colour a
|
||||
machine wears stays with the name it was added under, not its hostname. Files: create, open,
|
||||
edit, save, rename, delete. Services with its unit and log, cron jobs, containers and stacks,
|
||||
alert rules edited and put back, the terminal, the widget gallery. Login, a wrong password, a
|
||||
session cleared under an open page, logging back in onto the page you were on, and logout
|
||||
leaving nothing behind. The sampler, window and UI prefs endpoints under nonsense and hostile
|
||||
input, and a read-only token refused by all of them. The public demo page and installer. The
|
||||
Top tab on all four machines — including `grid`'s sixteen cores and thirteen filesystems, which
|
||||
scroll inside their boxes — its rate control, its filter, and three skins repainting it.
|
||||
`hamtv`'s eMMC really is called `hB8aP>`; that is what the device reports.
|
||||
|
||||
Firefox earns its place in this: three of the six above were invisible in Chromium, which
|
||||
swallows a failed WebSocket where Firefox writes it to the console. One thing it reports is
|
||||
*not* a fault — a page torn down by navigation while a terminal's socket is still shaking
|
||||
hands logs "can't establish a connection". Sitting still on a page produces none, the address
|
||||
is right, and the terminal belongs to the machine asked; `sockets.cjs` checks that ownership
|
||||
rather than the noise.
|
||||
|
||||
# Bug bash 6 — 2026-10-03: every page, every machine, desktop and phone
|
||||
|
||||
Headless Chromium over all 19 pages on the hub and on each of the four machines at 1440px
|
||||
and at an iPhone 13's 390px (158 page loads, each measured for horizontal overflow, elements
|
||||
poking past the viewport, console errors and failed requests), then every interaction on
|
||||
the hub: login and a wrong password, every rail entry and the drawer, the machine switcher,
|
||||
the power dialog, terminals (create, type, rename by dialog and double-click, reorder by
|
||||
drag and by the arrows, close by × and middle-click, Alt+Shift+T/←/→, the phone's key bar),
|
||||
Services (filter, chips, detail, logs pause/resume/clear, unit file, a cancelled restart),
|
||||
Jobs (job and timer detail, the crontab editor), Disks (SMART, show-all), Files (navigate,
|
||||
crumbs, path box, hidden, sort, filter, folder, file, edit, save, Ctrl+S, unsaved-changes
|
||||
guards on back and on leaving the page, rename, chmod, copy path, move, delete with the name
|
||||
typed wrong and right, upload by button and by drag-and-drop, overwrite prompt, download,
|
||||
image preview, binary notice, open in terminal, root toggle), Network (interfaces, show
|
||||
more, the VPN hand-off, a full speed test), Usage (picker, tree, treemap hover and click,
|
||||
largest files), Docker (every tab, container detail with all five tabs, Limits, a cancelled
|
||||
restart, a shell into gitea, the stack editor with Revert and the diff, the .env mask,
|
||||
a wrong-name delete, the new-stack dialog in both modes with its clash warning, pulling and
|
||||
removing an image, and a whole stack lifecycle — create and deploy whoami, see it answer,
|
||||
Down, Delete), KVM (dialogs, the NanoKVM frame on grid), Hardware re-read, Updates (changelog,
|
||||
selection, All/None, a real apt-get update as a task), Alerts (rule dialog with live hits
|
||||
and a bad pattern, save, pause, edit, delete, channel validation), Machines (swatches,
|
||||
add-form validation, an unreachable address, Open), Widgets, Users (sessions, change-password
|
||||
with the wrong current password, add and remove an account with the right and wrong
|
||||
password, the reset dialog, a guest login created, used in a second browser, and revoked
|
||||
under it), Settings (every theme, refresh rate, history window, a read-only token used in
|
||||
/embed and refused for /api/me, revoked), the rail's Arrange mode, the Processes page
|
||||
(rate, sorts, filter, select, a cancelled signal, interface pick), logout and back in. Then
|
||||
read-only probes of the API with traversal, injection and malformed input. Firefox could
|
||||
not run here (the box lacks its GTK libraries).
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
Verified against the live API through the Vite dev server rather than a deployed build: the
|
||||
permission classifier refused `make deploy` (as it did for the GPU work the day before).
|
||||
**Deployed 2026-10-08 as `v0.1.0-174-g0764508`**, with quality-of-life #2, #3, #8 and #10.
|
||||
|
||||
1. ✅ **The ZFS tab was wider than a phone.** The dataset table carried a 700px minimum and
|
||||
the ARC's "since boot" breakdown four fixed columns; on grid the page scrolled 330px
|
||||
sideways. Both drop to three columns and the ellipsis below 800px.
|
||||
2. ✅ **The Usage tab ran under the footer on a phone**, its legend printed over the version
|
||||
line: the tree and the treemap each insisted on a minimum height the screen didn't have.
|
||||
Below 800px the page scrolls, the tree 40vh and the map 60vh.
|
||||
3. ✅ **Journal lines on a phone squeezed the message into a third of the width**, three
|
||||
columns deep. Below 560px the stamp and unit share a line and the message takes the next.
|
||||
4. ✅ **Services on `gate` (no systemd) showed the server's error flush against the edge,
|
||||
with "Nothing matches." under it** as if a filter had hidden them. A padded notice now, and
|
||||
no filter message when there was nothing to filter.
|
||||
5. ✅ **ZFS "Snapshot now" offered a name seven hours wrong.** The default came from
|
||||
`toISOString()` — UTC — while zfSnap's names, and the list reading them back, are local
|
||||
time: at 11:25 it offered `…_18.25.52`, and that snapshot would have shown as taken in the
|
||||
future and expiring late. Built from local fields now.
|
||||
6. ✅ **"Adopt as a schedule" always ticked Recursive** — `/\s-\w*r/.test(cmd) || true`.
|
||||
7. ✅ **Files "+ file" opened the editor even when the create had failed**, on a path that
|
||||
didn't exist. It opens only after the create succeeds.
|
||||
8. ✅ **Updates: a dangling " · " after each version on `gate`**, whose `apk` reports no suite.
|
||||
9. ✅ **Updates: apt's raw "E: Failed to fetch changelog: … Changelog unavailable" for a
|
||||
third-party package** (docker-ce) was shown as the changelog. Said plainly now.
|
||||
10. ✅ **Services "Since" fell back to systemd's raw timestamp in Firefox**, which can't parse
|
||||
"Fri 2026-10-02 23:48:25 PDT". The fields are read directly now.
|
||||
11. ✅ **Hardware on `gate` led with a separator** — "· 4 threads · up to 2.70 GHz" — when the
|
||||
core count is unknown. The parts are joined, not concatenated.
|
||||
12. ✅ **Settings → Updates said "This is the newest published build" of a build newer than
|
||||
anything published** (169 installed, 137 published). It says so instead.
|
||||
13. ✅ **Escape did not close the phone's drawer**, alone among the overlays.
|
||||
14. ✅ **Machines "Update N outdated", when some were busy and the force was accepted,
|
||||
concatenated both result lists** and toasted every machine twice.
|
||||
|
||||
## Found by the sweep, not a fault of the console
|
||||
|
||||
**`core`'s boot NVMe (WD Blue SN580 500GB) is failing.** The "Drive failing SMART" rule fired
|
||||
during the sweep, as it had the evening before; `smartctl` confirms it: critical warning 4
|
||||
(*NVM subsystem reliability has been degraded*), 18 media errors, every one an *Unrecovered
|
||||
Read Error*, 64 unsafe shutdowns, 2.5 years powered on. The console did its job; the drive is
|
||||
a `hamlet` matter (`docs/pending.md`).
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Every page on every machine at both widths, with no console error that wasn't an expected
|
||||
401/400/403 from a deliberately wrong input. Login lockout messages, the password eye. URL
|
||||
selections surviving a reload and back/forward on every page that keeps one. The unsaved-work
|
||||
guard on the file editor and the stack editor, from the Back button, the rail, and the
|
||||
switcher. Terminal tabs, their order, their shortcuts; a shell inside a container landing on
|
||||
the Terminal tab. The whole stack lifecycle, image pull and prune, container Limits. A guest
|
||||
login kicked out at the moment of revocation. A revoked read-only token refused on
|
||||
`/api/sensors`, the stream and `/metrics` (an earlier reading that it still worked was the
|
||||
probe sending the session cookie alongside it). Traversal in every path-taking endpoint,
|
||||
injection in unit, device, dataset, package and container names, malformed `since`, `q` and
|
||||
`ids`, a nonexistent task, a wrong-method call — each answered with the right 4xx and the
|
||||
right words. No session, no cookie: 401 on every API, locally and through NPM. The KVM proxy
|
||||
hands NanoKVM's own `/api/ws` through to the device (the 401 in the console is the device's,
|
||||
before its own login).
|
||||
|
||||
## Quality of life — to discuss, not built
|
||||
|
||||
1. **Treemap blank space.** Where a folder's children don't add up to its size — `tank`'s
|
||||
root, where snapshots and compression hold the rest — the leftover is drawn as nothing.
|
||||
An "unaccounted" tile would say what it is.
|
||||
2. ✅ **Sessions pile up** — built 2026-10-06. Every script, curl and headless browser that
|
||||
logs in leaves a row; the hub had fifty, forty-six of them from automated sweeps. Now a
|
||||
session unused for seven days signs itself out (`auth.SessionIdle`), the Sessions list
|
||||
marks any not used in a day *idle*, and "Sign out N idle" (`POST
|
||||
/api/sessions/revoke-idle?hours=24`) drops those while leaving every device used since.
|
||||
3. ✅ **The Mac's GPU box takes half the Processes page** — built 2026-10-07. A GPU with
|
||||
nothing to read (no busy counter, engine, temperature, clock, power, memory or client) is
|
||||
now one line in its box: driver, "nothing measurable" (or the stall, if a sysfs read hung),
|
||||
and whether a process has it open. Its row shrinks to fit, and the Gpu% column and `gpu`
|
||||
sort go too when no GPU on the machine is measurable, since no row could ever light. A
|
||||
client opening the device brings the full box back. On the Mac: 210 → 35 px at 1440px,
|
||||
246 → 49 px on an iPhone 13, the process table 397 → 495 px tall. Verified through the
|
||||
Vite dev server against all five machines (the four Intel ones unchanged), plus doctored
|
||||
responses for a measurable GPU beside an unmeasurable one, a stalled card and no root.
|
||||
4. **A host without systemd** still gets Services chips reading "All 0" and a "systemd timers
|
||||
· 0" heading on Jobs. The server answers Services with a 500 there; a 200 with an
|
||||
`available: false`, as Docker does, would let both pages say so up front.
|
||||
5. **An alert whose sensor vanishes resolves silently.** If `smartctl` errors out, the
|
||||
`smart.*.passed` sensor is omitted and a firing "Drive failing SMART" clears. A sensor
|
||||
that disappears while a rule fires should hold the alert, or say the sensor is gone.
|
||||
6. **Network "Past tests" rows are clipped** on desktop ("↓ 302 · ↑ 300 Mbps · 7.5 …") by the
|
||||
four-column props grid.
|
||||
7. **Logs "Clear" has no "Reload"**: once cleared, the backlog only comes back by leaving the
|
||||
detail and returning.
|
||||
8. ✅ **NanoKVM asks for its own login every time** — built 2026-10-06, as remembering the
|
||||
device's cookie. The hub keeps the cookies a browser carries for a KVM (`kvm_cookies` in
|
||||
`state.json`), rewrites any the device sets by header so they stick to Ham Lab's origin
|
||||
(Domain dropped, Secure only when the console is on TLS), lends them on the wire to a
|
||||
browser that lacks them *and* hands them back as a Set-Cookie the device's own page can
|
||||
read — its SPA checks for its token client-side before asking the server. A logout from the
|
||||
device, or removing the KVM, forgets them. Covered by `kvm_test.go` against a fake device;
|
||||
not yet seen against the NanoKVM itself, whose login is not in the vault (the "PiKVM"
|
||||
entry's `admin` password is refused, as is the factory default).
|
||||
9. **Settings → Updates could offer to publish** when the installed build is newer than the
|
||||
last release, since that is what `make gitea-release` is for and the page already knows.
|
||||
10. ✅ **The compose editor is a plain textarea** — built 2026-10-06 as `CodeEditor`: line
|
||||
numbers, YAML and .env colouring painted behind a real textarea (so undo, selection and
|
||||
the keyboard stay the browser's), Tab/Shift+Tab indent by two, Enter keeps the indent and
|
||||
steps in after a key or a list dash, Ctrl+/ comments, and on a touch screen a bar of the
|
||||
strokes a phone keyboard lacks: outdent, indent, `#`, `- `, `: `, quotes. Used by the stack
|
||||
editor (compose and .env) and the new-stack dialog.
|
||||
11. **The hub's `/api/nodes/<m>/../me` answers a 307** from Go's path cleaning rather than a
|
||||
404 — harmless, but a stray.
|
||||
12. **Guest logins can open Users and Settings** — by design ("a guest can do everything you
|
||||
can"), but worth a second look now that Users includes removing accounts.
|
||||
|
||||
# 2026-10-23: the rate obeyed, but not in step
|
||||
|
||||
**Graphs and numbers on Top moved on different beats, whatever the rate.** Every machine was
|
||||
sampling at exactly the set rate (measured: 500 ms between samples on all five nodes), so the
|
||||
sampler was not the fault. Four things in the console were:
|
||||
|
||||
1. ✅ **Graphs stepped once a point, not once a tick.** A point is the history window split 300
|
||||
ways — at 5 min that is 1 s — so at 250 or 500 ms the numbers changed two to four times for
|
||||
every move of a graph, and a narrow box (phone, net graph) moved once a *column*, every few
|
||||
seconds. The columns now slide by a fraction of one on every tick (`columnsAt` in
|
||||
`graphcols.ts`), the newest growing in from the right; a finished column still never changes.
|
||||
The per-core strips keep the last 18 ticks rather than stride points.
|
||||
2. ✅ **The process table, MHz, CPU watts and every GPU box ran on their own timer** — a
|
||||
`setInterval` on `/api/procs` with an arbitrary phase against the sensor stream, so half the
|
||||
page changed up to a tick after the other half. The table is now read on each sample and the
|
||||
two are committed together: one render per tick for the whole page.
|
||||
3. ✅ **GPU clients were walked at most once a second** (`gpuScanEvery`), so per-process GPU% and
|
||||
the engine meters lagged the device's numbers. Walked every sample now.
|
||||
4. ✅ **Hidden floors on other live views:** Sensors gallery (1 s), Network live rates (1 s) and
|
||||
speed-test needle (0.4–1 s), Docker live (2 s), ZFS ARC (2 s). All at the rate now.
|
||||
|
||||
Not changed: container status sensors are read every 15 s (`docker ps` plus an inspect) and
|
||||
filesystem usage every 30 s — slow-moving facts behind an exec or a statfs of every mount.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Ham Lab (`hamlab`) — the web management console app
|
||||
|
||||
This repo is the **app** only: Go daemon + React/xterm UI, deployed as `hamlab.service`
|
||||
on `core`. Product scope, decisions and roadmap are in `PLAN.md`.
|
||||
|
||||
Everything about *how to work* — persona, response style, write-back rule, hosts,
|
||||
credentials, pending items — lives in the homelab repo **Hamlet** at
|
||||
`/home/core/projects/hamlet`, and applies here unchanged:
|
||||
|
||||
@/home/core/projects/hamlet/CLAUDE.md
|
||||
|
||||
Ham Lab facts that belong to the lab (where it runs, ingress, vault item) go in
|
||||
`hamlet/docs/hosts.md` and `hamlet/docs/pending.md`, not here.
|
||||
@@ -1,26 +1,77 @@
|
||||
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
|
||||
# Commit first, then deploy: the version is the commit the binary was built from, counted
|
||||
# from the base tag. A published release is tagged with a version of this same shape
|
||||
# (v0.1.0-137-gc129f65), and described from one of those a build would count from 137 again
|
||||
# and read as older than what it replaces — so tags with a dash are never the base.
|
||||
VERSION ?= $(shell git describe --tags --exclude '*-*' --always 2>/dev/null || echo dev)
|
||||
GO ?= go
|
||||
PREFIX ?= /usr/local/bin
|
||||
|
||||
.PHONY: all web build dev deploy clean
|
||||
# every architecture a release carries; arm is ARMv6 hard-float, so it covers every Raspberry Pi
|
||||
ARCHES ?= amd64 arm64 arm 386 riscv64
|
||||
LDFLAGS = -s -w -X main.version=$(VERSION)
|
||||
# static, no cgo: the same file runs on glibc, musl (Alpine) and anything else with a Linux kernel
|
||||
GOBUILD = CGO_ENABLED=0 GOOS=linux GOARM=6 $(GO) build -trimpath -ldflags "$(LDFLAGS)"
|
||||
|
||||
.PHONY: all web build go dev cross release gitea-release deploy push-nodes push-nodes-force test clean
|
||||
|
||||
all: build
|
||||
|
||||
web:
|
||||
cd web && npm ci --no-audit --no-fund && npm run build
|
||||
|
||||
build: web
|
||||
$(GO) build -trimpath -ldflags "-s -w -X main.version=$(VERSION)" -o bin/hamlab ./cmd/hamlab
|
||||
build: web go
|
||||
|
||||
# backend only (assumes web/dist already built)
|
||||
go:
|
||||
$(GO) build -ldflags "-X main.version=$(VERSION)" -o bin/hamlab ./cmd/hamlab
|
||||
$(GOBUILD) -o bin/hamlab ./cmd/hamlab
|
||||
|
||||
# one other architecture: make build-arm64, build-arm, build-386, build-riscv64, build-amd64
|
||||
build-%: web
|
||||
GOARCH=$* $(GOBUILD) -o bin/hamlab-linux-$* ./cmd/hamlab
|
||||
|
||||
dev:
|
||||
cd web && npm run dev
|
||||
|
||||
deploy: build
|
||||
./deploy/install.sh
|
||||
# siblings for every other architecture; installed beside the binary, and the hub pushes
|
||||
# each managed machine the one that matches it
|
||||
cross:
|
||||
@for a in $(filter-out $(shell $(GO) env GOARCH),$(ARCHES)); do \
|
||||
echo "build linux/$$a"; GOARCH=$$a $(GOBUILD) -o bin/hamlab-linux-$$a ./cmd/hamlab || exit 1; done
|
||||
|
||||
# dist/: one static binary per architecture, install.sh, SHA256SUMS. Serve the directory
|
||||
# anywhere (or attach it to a release) and install.sh does the rest.
|
||||
release: web
|
||||
rm -rf dist && mkdir -p dist
|
||||
@for a in $(ARCHES); do \
|
||||
echo "build linux/$$a"; GOARCH=$$a $(GOBUILD) -o dist/hamlab-linux-$$a ./cmd/hamlab || exit 1; done
|
||||
sed 's|@VERSION@|$(VERSION)|' install.sh > dist/install.sh && chmod +x dist/install.sh
|
||||
echo $(VERSION) > dist/VERSION
|
||||
cd dist && sha256sum hamlab-linux-* install.sh > SHA256SUMS
|
||||
|
||||
test:
|
||||
$(GO) vet ./... && $(GO) test ./...
|
||||
sh -n install.sh
|
||||
cd web && npm test --silent
|
||||
|
||||
# installs here (keeping the config, user and password), then updates every managed machine
|
||||
# Offer this build to every install, wherever it is: a release on Gitea, which is what
|
||||
# Settings -> Updates reads. Deliberate, and not part of deploy — deploy pushes to the
|
||||
# machines this hub owns, publishing offers it to everyone else.
|
||||
gitea-release: release
|
||||
./scripts/gitea-release.sh $(RELEASE_ARGS)
|
||||
|
||||
deploy: build cross
|
||||
./bin/hamlab install --prefix $(PREFIX)
|
||||
@sleep 1; $(PREFIX)/hamlab push-nodes || true
|
||||
@echo "deployed $(VERSION); 'make gitea-release' publishes it for everyone else"
|
||||
|
||||
# push the installed binary to every managed machine (what deploy does last)
|
||||
push-nodes:
|
||||
$(PREFIX)/hamlab push-nodes
|
||||
|
||||
# same, interrupting running tasks and attached terminals on the nodes
|
||||
push-nodes-force:
|
||||
$(PREFIX)/hamlab push-nodes --force
|
||||
|
||||
clean:
|
||||
rm -rf bin internal/web/dist web/dist
|
||||
rm -rf bin dist internal/web/dist web/dist
|
||||
|
||||
+159
@@ -0,0 +1,159 @@
|
||||
# Portability audit — what in Ham Lab is specific to this lab (2026-10-09)
|
||||
|
||||
Goal: Ham Lab drops onto any set of Linux machines and works. This is the full sweep of
|
||||
every place the code, defaults, docs or UI assume Mitchell's hosts, network, hardware,
|
||||
distro, init system, orchestrator history or posture. **Nothing here is fixed yet** — this
|
||||
is the list to work from. Each item: where, what it assumes, what the general form is.
|
||||
|
||||
Virtualization was cut from the app 2026-10-15 (kept on the `virtualization` branch); its
|
||||
items — 1.8, 4.2, 5.1, 5.2, 6.14 and the Incus halves of 5.3, 5.7, 6.15, 10.3 — went with it.
|
||||
|
||||
Severity key: **A** breaks or misbehaves on an ordinary host · **B** works but is wrong by
|
||||
default or phones home to this lab · **C** cosmetic (placeholder text, comments, docs).
|
||||
|
||||
---
|
||||
|
||||
## 1. Hard-wired lab identifiers
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 1.1 | **B** | `internal/selfupdate/selfupdate.go:36` `DefaultURL = "https://git.bigham.dev/api/v1/repos/mitchell/hamlab"` | Every install with no `update.url` checks **this** Gitea 90 s after boot and daily (`internal/api/update.go:1951, 2016`). A stranger's install phones home here by default. | Default should be **off** (or a build-time `-X` variable set at release). Installer asks or `install.sh` passes `HAMLAB_URL` through into `update.url`. |
|
||||
| 1.2 | **B** | `install.sh:4, 27` `HAMLAB_URL` default; `README.md:23-25, 31` | The curl one-liner and the install source are this Gitea. | Default comes from the release that served the script (`@URL@` filled by `make release`), or is required. |
|
||||
| 1.3 | **B** | `scripts/gitea-release.sh:14` `REPO` default | Publishing goes to `mitchell/hamlab` on this Gitea. | Require `HAMLAB_REPO`; no default. |
|
||||
| 1.4 | **C** | `README.md:100` "On `core` it is public at `https://lab.bigham.dev` behind Nginx Proxy Manager" | Lab deployment fact in the product README. | Move to `hamlet/docs/hosts.md`. |
|
||||
| 1.5 | **C** | `examples/external-demo/index.html:8` `base: 'http://10.0.0.234:8100'` | Demo config names the hub's LAN IP. | `http://<hamlab-host>:8100`. |
|
||||
| 1.6 | **C** | `SENSORS.md:149` Prometheus target `10.0.0.234:8100`; `:187` `&machine=grid`; `:196-208` Glance examples at `lab.bigham.dev`, `zfs.tank.used_pct`, `/api/nodes/grid/...`; `:218-220` `grid,wyse`, `{"core":…,"grid":…,"mac":…}` | Docs use real host names, pool and domain. | Generic placeholders (`<host>`, `<machine>`, `<pool>`). |
|
||||
| 1.7 | **C** | `PLAN.md:3-4` "bare-metal Debian 13 host. First target: `core` (10.0.0.234)"; decisions table rows (Listen `10.0.0.234:8100`, Data `/home/core/apps/hamlab/`, Ingress NPM/`lab.bigham.dev`, Repo) | Product plan is written as this lab's plan. | Keep as history, or split lab facts out to `hamlet`. The repo is public. |
|
||||
| 1.9 | **C** | `BUGS.md` throughout | Bug bashes narrate `core`, `grid`, `wyse`, `hamtv`, `gate`, the Mac. | History; fine to leave, but it is public. |
|
||||
| 1.10 | **C** | `cmd/hamlab/main.go:1` "Proxmox-style web console for a bare-metal Debian host"; `internal/hardware/hardware.go:5` "what Debian and its kin already have"; `internal/hardware/hardware.go:29` `// "GMKtec NucBox G3"`; `internal/metrics/metrics.go:1358` "did, on grid, 2026-10-02"; `internal/api/server.go:1382-1383` `["grid", …]`, `?machines=grid,wyse`; `internal/api/stacks.go:10` `/home/core/apps/dockhand/stacks/…`; `internal/api/dockersched.go:5` "unbound, nginx and pihole"; `internal/api/zfs.go:5` "grid already runs zfSnap"; `internal/storage/snapshots.go:6` "grid has run zfSnap v1.11.1"; `internal/api/guests.go:8` "Mitchell, 2026-09-27"; `internal/network/network.go:306` "the public address here is a DHCP lease"; `internal/metrics/gpu.go:17` "Asahi, today"; `internal/metrics/pciids.go:4-6` "the router's GPU"; `web/src/Top.tsx:522, 576`; `web/src/hashsel.ts:1-9`; `web/src/Embed.tsx:3` `&machine=grid` | Comments explain decisions by naming lab hosts. | Harmless, but rewrite the ones in package doc comments so a reader of the source doesn't need the lab to understand the code. |
|
||||
|
||||
## 2. UI placeholders and defaults that are this lab's
|
||||
|
||||
| # | Sev | Where | Text | General form |
|
||||
|---|---|---|---|---|
|
||||
| 2.1 | **C** | `web/src/Machines.tsx:116-118` | `placeholder="grid"`, `"https://10.0.0.214:8100"`, `"core"` | `nas`, `https://192.0.2.10:8100`, `admin` or the hub's own user. |
|
||||
| 2.3 | **C** | `web/src/Settings.tsx:242` | `placeholder="cpu.*, temp.*, zfs.tank.*"` | `cpu.*, temp.*, fs.*`. |
|
||||
| 2.4 | **C** | `web/src/Settings.tsx:225` | read-only token default name `'glance'` | Empty, or `dashboard`. |
|
||||
| 2.5 | **C** | `web/src/Alerts.tsx:38, 40` | `"Tank nearly full"`, `"zfs.tank.used_pct"` | `"Disk nearly full"`, `"fs.root.used_pct"`. |
|
||||
| 2.6 | **C** | `web/src/StackNew.tsx:110, 113, 114` | `/home/core/apps/whoami:/data`, `TZ=America/Los_Angeles`, `com.example.owner=mitchell` | `./data:/data`, `TZ=UTC`, `com.example.owner=you`. |
|
||||
| 2.7 | **C** | `web/src/Gallery.tsx:69`, `web/src/WidgetPicker.tsx:36` | `"find — gitea, temp, enp3s0…"`, `"filter sensors — cpu, temp, fan, enp3s0…"` | Drop the container and NIC names. |
|
||||
| 2.8 | **C** | `web/src/VPN.tsx:105` | default DNS for a new device `1.1.1.1, 1.0.0.1` | Fine as a public default, but should offer the tunnel's own address (the home resolver) first. |
|
||||
| 2.9 | **B** | `web/src/ZFS.tsx:212` `BLANK` | new schedule defaults `at: '02:30'`, `keep_for: '6w'` — `grid`'s old zfSnap cron line | Neutral defaults (`03:00`, `4w`), or none. |
|
||||
| 2.10 | **A** | `web/src/ZFS.tsx:302` `adopt()` | `/\b(tank[\w/-]*)\b/` — the dataset of an adopted external job is found by looking for the word **`tank`** | Parse the dataset from the command as the last `zfs`-shaped argument, or match against the datasets the page already has. On any other pool name adopt silently picks the fallback. |
|
||||
|
||||
## 3. Topology inherited from the lab (hub `core`, Dockhand, `tcp://:2375`)
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 3.1 | **done** 2026-10-10 | `internal/api/stacks.go` `nodeDocker()` (removed) | For every node whose stacks aren't editable locally, the hub **probes `tcp://<node-host>:2375`** with `docker -H … version` and, if it answers, deploys to the node over an unauthenticated Docker socket. This is the Dockhand-era layout; on any other network an open 2375 is either absent or a security problem the console should not quietly use. | Remove the probe. The node's own Ham Lab deploys its stacks (the files live on the node, under its data dir). `PLAN.md` "To do" already names this decision. |
|
||||
| 3.2 | **done** 2026-10-10 | `hubStacks()`, `taskOnHub()`, the hub-side `/api/nodes/{node}/docker/…` routes, `Stack.remote`/`held_by` (all removed; the generic node proxy carries those requests) | A hub can hold compose files for a node and run compose remotely; new stacks for a node may be created **on the hub**. Only true here because Dockhand on `core` deployed `grid` and the Mac. | Same as 3.1: one machine, one set of files, one Docker. Hub-held stacks become an import path (copy to the node) rather than a steady state. |
|
||||
| 3.3 | **done** 2026-10-10 | `internal/api/stacks.go` `homesOf()` — `<DataDir>/stacks` is always offered first and default; the `~/stacks` fallback is gone; `TestHomesDefaultToDataDir` | Where new stacks go is **discovered** from where ≥2 stacks already are (Dockhand's `stacks/<machine>` habit), else `~/stacks`. Neither is Ham Lab's data directory, which the project's own rule says everything managed must live under. | Default home = `<DataDir>/stacks`. Discovered homes stay as *additional* places, never the default. |
|
||||
| 3.4 | **B** | `internal/api/stacks.go:100-203` remembered mounts, `internal/api/dockersched.go:772-795` `repoint()` | Machinery exists to find compose files **inside a removed orchestrator container's filesystem**. Generic in form, but it exists because of Dockhand and is exercised only here. | Keep (it is general), but it is not a reason to keep 3.1/3.2. |
|
||||
| 3.5 | — | `internal/api/stacks.go` `stackAction`, `internal/compose/compose.go` | Inside a `Via` container the command is literally `docker compose`. | **Checked 2026-10-10, no change:** the CLI inside an orchestrator's container is that container's own `docker`, independent of what the host runs; the host's tool only chooses the `exec`. Both places now say so in a comment. |
|
||||
| 3.6 | **B** | `internal/api/stacks.go:908` `docker buildx imagetools inspect` | Update checks need the **buildx** plugin. Docker CE on Debian has it; Podman and minimal Docker installs do not → every image reports "the registry didn't answer". | Fall back to `skopeo inspect` or a direct registry `HEAD` for the digest. |
|
||||
| 3.7 | **B** | `Makefile:445-448` `deploy` | The machine you build on **is** the hub: installs locally, then `push-nodes`. | Fine for a developer; document that `deploy` means "I am the hub". A plain `make install` target for a single box. |
|
||||
| 3.8 | **B** | `internal/nodes/nodes.go:316-323` | A hub pushes other architectures only if `<prefix>/hamlab-<arch>` siblings exist, which only `make deploy`/`install.sh`-from-dist put there. | A hub without siblings should fetch the right build from `update.url` instead of erroring. |
|
||||
|
||||
## 4. Init system: systemd assumed where the product promises procd and OpenRC too
|
||||
|
||||
The installer supports systemd, procd and OpenRC (`internal/setup/setup.go:217-262`). These paths do not.
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 4.1 | **A** | `internal/updates/updates.go:537-543` `Power()` | `systemctl reboot` / `systemctl poweroff`. On OpenWrt (`gate`) and OpenRC the Power dialog fails. | Try `systemctl`, else `reboot` / `poweroff` (busybox, OpenRC, sysvinit all have them). |
|
||||
| 4.3 | **A** | `internal/services/services.go:163-170` | Services tab is systemd-only ("says so cleanly"). `gate` (procd) and any OpenRC/runit/s6 box have no Services tab. | `rc-service`/`rc-status` (OpenRC), `/etc/init.d/<x> status` + `ubus call service list` (procd), `sv` (runit) backends. Noted as "not done" in `PLAN.md:487-489`. |
|
||||
| 4.4 | **A** | `internal/cron/cron.go:353-373, 401, 442, 532-534, 669-670` | Jobs tab reads `/etc/crontab`, `/etc/cron.d`, `/var/spool/cron{,/crontabs}` (Debian/cronie), `/usr/sbin/anacron`, last-runs from **journald**, timers via `systemctl`; writes only `/etc/crontab` and `/etc/cron.d`. Busybox crond (OpenWrt, Alpine) keeps crontabs in `/etc/crontabs/<user>` and has no `cron.d` → the Jobs tab is empty on `gate` and schedules written by 4.5 never run. | Add `/etc/crontabs/*` as a source; last-runs from syslog/logread when there is no journal. |
|
||||
| 4.5 | **A** | `internal/storage/snapshots.go:402` `/etc/cron.d/hamlab-zfs`; `internal/api/dockersched.go:65` `/etc/cron.d/hamlab-docker` | ZFS snapshot and image-update schedules are cron.d files. No cron.d on busybox crond; no cron at all on a systemd-timer-only host (Fedora Server, CoreOS). | Detect: cron.d → cron.d; busybox → a line in `/etc/crontabs/<user>`; none but systemd → a timer unit. The daemon-side runner already exists (`hamlab zfs-snapshot`, `docker-update`). |
|
||||
| 4.6 | **B** | `internal/hardware/hardware.go:333` `systemd-detect-virt` | Virtualisation detection is systemd-only. | Fall back to `/sys/class/dmi/id/sys_vendor`, `/proc/cpuinfo` `hypervisor` flag, `/sys/devices/virtual/dmi`. |
|
||||
| 4.7 | **B** | `internal/api/server.go:1129`, `internal/updates/updates.go:187` | `/run/reboot-required` is Debian/Ubuntu-only. `updates.Status` falls back to a kernel check for other distros; `/api/me.reboot_needed` (the Machines badge, the header's amber power button) does not. | Share one function. |
|
||||
| 4.8 | **B** | `install.sh:154-157, 164, 375-378`; `internal/setup/setup.go:284` | The wizard's init detection knows systemd and OpenRC only: on OpenWrt it warns "no systemd or OpenRC: nothing starts it at boot" although `hamlab install` then installs a procd script fine; the "logs" hint and the "isn't answering yet" hint have no procd case. | Add procd to both. |
|
||||
| 4.9 | **B** | `internal/setup/setup.go:609-641` `firewall()` | Only firewalld and ufw are detected. OpenWrt fw4/nftables, plain nftables, iptables, Alpine awall are not. | Detect `fw4`/`nft`/`iptables`; on OpenWrt add a LAN-zone rule or say what to open. |
|
||||
| 4.10 | **B** | `internal/setup/setup.go:568-605` `quietSudoers()`; `internal/services/services.go:176-185` | The sudoers quieting list is tuned to what the lab polls (`smartctl`, `pwrstat`, `journalctl`, `zpool`, `zfs`). A host with NUT or apcupsd gets `upsc`/`apcaccess` lines in its auth log at 10 s intervals; a host without `/etc/sudoers.d` (OpenWrt) gets nothing, which is correct. | Build the list from what the collector actually probes. |
|
||||
|
||||
## 5. Distro and package assumptions beyond init
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 5.3 | **B** | `DEBIAN_FRONTEND`, `apt-get … -o Dpkg::Options::=--force-confold` in `internal/updates/updates.go:78, 432` | Debian-isms exported unconditionally. Harmless elsewhere. | Fine; note only. |
|
||||
| 5.4 | **B** | `scripts/pci-ids-gpu.sh:7, 41` default `/usr/share/misc/pci.ids` | Debian's path for `pci.ids`; Fedora/Arch keep it in `/usr/share/hwdata/`. Build-time only. | Try both. |
|
||||
| 5.5 | **B** | `internal/api/status_sensors.go:283`, `internal/api/docker.go:133` | Container memory/CPU read from three cgroup paths: cgroup v2 + systemd driver, cgroupfs, libpod. cgroup v1 hosts (older Debian/Ubuntu, some Alpine, OpenWrt) get no `container.*.mem/cpu`. | Add the v1 `memory/docker/<id>` and `cpu,cpuacct` paths, or fall back to `docker stats --no-stream` once in a while. |
|
||||
| 5.6 | **B** | `internal/metrics/metrics.go:1064` `clkTck = 100` | USER_HZ assumed 100. True on every mainstream build; wrong on a kernel built otherwise. | `sysconf` via `getconf CLK_TCK` once at start. |
|
||||
| 5.7 | **B** | `internal/space/manager.go:578-585` `timeout -s KILL` | GNU `timeout`. | Busybox has `timeout` too but flag spelling differs — use the context deadline. |
|
||||
|
||||
## 6. Hardware and vendor assumptions
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 6.1 | **B** | `internal/metrics/gpu.go:343-383` `read()` | Busy/clock/VRAM are read for **i915, xe, amdgpu** only — the three GPUs in this lab. Nouveau, nvidia-drm, panfrost, v3d, Asahi are listed but never measured. `PLAN.md:317`: "No NVIDIA path: nothing here has one to test against." | `nvidia-smi --query-gpu` where present; `/sys/class/drm/*/device/gpu_busy_percent` for nouveau; `devfreq` for the ARM drivers. |
|
||||
| 6.2 | **B** | `internal/metrics/pciids_gpu.txt`, `scripts/pci-ids-gpu.sh:12-15` | Embedded PCI ID table trimmed to Intel, AMD, NVIDIA and three virtual vendors. Matrox, ASPEED (every server BMC), Moore Threads, etc. fall back to `lspci` or "Vendor GPU <id>". | Include the full display-class device list, not three vendors. |
|
||||
| 6.3 | **B** | `internal/metrics/gpu.go:237-241` | "Integrated" = PCI bus `0000:00:`. Wrong on multi-domain or APU boards where the iGPU hangs off a bridge; cosmetic. | Use `boot_vga` + `PCI_SUBSYS` heuristics, or drop the flag. |
|
||||
| 6.4 | **B** | `internal/metrics/metrics.go:189-196, 917-968` | UPS probe order **pwrstat (CyberPower) first**, then NUT, then apcupsd; pwrstat's `....`-separated output is parsed bespoke. A host with both pwrstat and NUT shows pwrstat only. | Prefer NUT (the vendor-neutral one), then the vendor tools; or list every UPS found. |
|
||||
| 6.5 | **B** | `internal/metrics/metrics.go:1179-1182` `cpuTemp()` | CPU temperature from `coretemp`, `k10temp`, `thermal`/`cpu`, `cpu_thermal`, `soc_thermal`. Misses `zenpower`, `acpitz` as the only sensor, Apple `macsmc` names, many ARM SoC zone names. | Broaden; fall back to the hottest zone when none matches. |
|
||||
| 6.6 | **B** | `internal/hardware/hardware.go:617-627` `jedec` | JEDEC IDs for "the memory makers who turn up in a homelab" — 14 entries. | Embed the JEDEC JEP106 bank list (public, small). |
|
||||
| 6.7 | **B** | `internal/storage/storage.go`, `internal/api/zfs.go`, `internal/storage/arc.go`, `web/src/ZFS.tsx` | A whole **ZFS** tab and ZFS-aware Space/Disks logic; **no btrfs or LVM equivalent** (snapshots, subvolumes, scrub, balance). Not wrong, but the tab list reflects a ZFS NAS. | Later: a Btrfs tab with the same shape, or a generic "Filesystems" tab that shows what the host has. |
|
||||
| 6.8 | **B** | `internal/storage/snapshots.go:1-16, 966-968` | Snapshot naming is **zfSnap's** (`2026-09-24_02.30.01--6w`) because `grid` ran zfSnap. A sanoid/zfs-auto-snapshot user's existing snapshots (`autosnap_…`, `zfs-auto-snap_daily-…`) are detected (`internal/api/zfs.go:1764-1798`) but can never be expired by Ham Lab, and new schedules write a third convention beside them. | Make the naming a per-schedule choice (zfSnap / sanoid / zfs-auto-snapshot), or at least document the convention as the product's rather than inherited. |
|
||||
| 6.9 | **B** | `internal/network/speedtest.go:64-67` `downSources` | Fallback download source is **`speedtest.seattle.linode.com`** — chosen because the lab is in the Pacific Northwest. Everywhere else it reads low. | Pick the nearest of several mirrors, or drop the fallback and report the refusal. |
|
||||
| 6.10 | **B** | `internal/network/speedtest.go:70-74, 604`, `internal/network/network.go:323-334` | Public IP, edge and speed come from **Cloudflare** endpoints, polled every 15 min with no off switch. Air-gapped or privacy-minded installs have no way to disable the outbound probe. | A `network.outside: off` setting; honour it in the watcher. |
|
||||
| 6.11 | **B** | `internal/vpn/edit.go:5-7, 419-421` | Peers can be **created/edited only through OpenWrt UCI** "because that is where the server here is configured". A wg-quick server (the common case everywhere else) is read-only. | Write wg-quick `.conf` too (append a `[Peer]` block + `wg set`), guarded the same way. |
|
||||
| 6.12 | **B** | `internal/vpn/vpn.go:251-253` | Peer names come from wg-quick comments or UCI descriptions — "which between them cover every machine here". NetworkManager and systemd-networkd WireGuard configs are not read. | Add `.nmconnection` and `.netdev` readers. |
|
||||
| 6.13 | **B** | `internal/vpn/edit.go:515` `profile()` | Split-tunnel routes hard-code `10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12` on top of the tunnel's own prefix. | Offer the host's actual LAN prefixes (from `TakenPrefixes`-style route reading), with RFC 1918 as the fallback. |
|
||||
| 6.15 | **B** | `internal/network/network.go:170-199, 303-314`; `internal/vpn/edit.go:558-595`; `internal/tlsauto/tlsauto.go:533-537`; `install.sh:114-120`; `internal/setup/setup.go:648-666` | **IPv4 everywhere**: default route from `/proc/net/route` only, outside view v4 only, peer allocation v4 only, cert SANs v4 only, installer lists v4 addresses only. | `/proc/net/ipv6_route`, v6 SANs, v6 peer addresses. An IPv6-only host today has no gateway, and no cert SAN. |
|
||||
|
||||
## 7. Privilege and deployment posture (the lab's NOPASSWD-everywhere model)
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 7.1 | **A** | `internal/priv/priv.go`; every tab that calls `priv.Available()`; `web/src/Updates.tsx:2144` "give its user passwordless sudo or doas"; `web/src/Jobs.tsx:1883` | Full function needs the service user to have **unrestricted passwordless sudo**, which `hamlet/CLAUDE.md` grants on every host here. The installer neither creates nor offers a sudoers rule, so on a fresh box most tabs are read-only and the UI tells the user to go fix sudoers by hand. | `hamlab install --sudo` writes a **scoped** `/etc/sudoers.d/hamlab` naming the commands Ham Lab runs (smartctl, zpool, zfs, systemctl, docker, journalctl, tee for its cron files, `hamlab fs|space-walk|drm-clients|cpu-energy|replace`), or installs as root. Either way the console works out of the box. |
|
||||
| 7.2 | **A** | `internal/api/server.go:1069-1080` `clientKey()`, `:1083-1092`, `:431-433` `secure()` | The **first `X-Forwarded-For` hop is trusted unconditionally**, because here Ham Lab always sits behind Nginx Proxy Manager. On a host exposed directly, a client sets the header and defeats the per-client login rate limiter and the session IP check. | Trust `X-Forwarded-*` only from configured proxy addresses (`trusted_proxies` in config), else use the socket peer. |
|
||||
| 7.3 | **B** | `internal/api/server.go:1324` `X-Accel-Buffering: no` "nginx (NPM)" | Harmless header; comment names the lab's proxy. | Keep header; generalise comment. |
|
||||
| 7.4 | **B** | `internal/api/guests.go:1-9`, `PLAN.md:21` | Guest logins are full-power ("he gives these to people he trusts"). A product default. | Keep, but it is a posture decision taken for this household; a read-only guest is the generic expectation and was cut. Document as a deliberate choice in README. |
|
||||
| 7.5 | **B** | `internal/api/users.go:1-6` | "a household console, not a tenancy": any signed-in user can add users, make tokens, open a root shell. | Same as 7.4 — fine, but say so where a stranger installs it. |
|
||||
|
||||
## 8. Resource tuning from the lab's smallest host
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 8.1 | **B** | `cmd/hamlab/main.go:451` `debug.SetMemoryLimit(160 << 20)`; `PLAN.md:11` "`core` has ~370 MiB free RAM" | A fixed 160 MiB soft heap limit sized for `core`. On a 64 GiB NAS it forces needless GC under a big Space scan; on a 512 MiB Pi it is still half the box. | Scale from `MemTotal` (e.g. 2–5 %, floor 64 MiB, ceiling 1 GiB), or make it a config key. |
|
||||
| 8.2 | **B** | `internal/space/walk.go:893` `workers := 32`; `internal/space/manager.go:578` 3-minute `zfs diff` patience | Tuned on `grid` (16 cores) and `core`. | Workers from `GOMAXPROCS`; patience configurable. |
|
||||
| 8.3 | **C** | `internal/metrics/metrics.go:32-43` intervals, `History = 300` | Reasonable generic defaults; listed only because 8.1 made them look lab-tuned. | None. |
|
||||
|
||||
## 9. Data directory and legacy paths
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 9.1 | **B** | `internal/config/config.go:115-117` `Candidates()`; `internal/setup/setup.go:413-422`; `README.md:38` | `~/apps/hamlab/config.json` is searched as a legacy location — the lab's pre-v0.2 path, kept so `core`, `grid` and the Mac keep working. | Keep for one more release with a one-time migration log line, then drop. A stranger never had it. |
|
||||
| 9.2 | **done** 2026-10-10 | 3.3 above; `PLAN.md:24` "`grid`'s and the Mac's stacks at `~/stacks/<machine>/` are the standing exception" | The "everything under DataDir" rule has a lab-shaped exception baked into the stacks code. | Default to DataDir; treat `~/stacks` as a discovered home like any other. |
|
||||
| 9.3 | **C** | `internal/setup/setup.go:351` `ownFiles` | Correctly scoped; note only. | None. |
|
||||
|
||||
## 10. Build and release
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 10.1 | **B** | `Makefile:60-61, 442-443`; `scripts/gitea-release.sh` | Publishing is Gitea/Forgejo-only (release API shape). `selfupdate` also understands a static directory, so publishing could too. | `make release` + any static host is already enough; add a GitHub/Forgejo-agnostic `release-dir` note. |
|
||||
| 10.2 | **C** | `scripts/gitea-release.sh:500-515` | Needs `python3` on the publishing host. | `jq`, or do it in Go (`hamlab publish`). |
|
||||
| 10.3 | **C** | `Makefile:392` `ARCHES ?= amd64 arm64 arm 386 riscv64` | Fine. | — |
|
||||
|
||||
## 11. Things that are already general (checked, no change needed)
|
||||
|
||||
Listed so the sweep is known to be complete, not because they need work.
|
||||
|
||||
- `internal/priv` (root / sudo / doas), `internal/setup` service templates, `IP_FREEBIND` listen, self-update atomic replace and re-exec.
|
||||
- `internal/updates`: apt, dnf, zypper, pacman, apk with refresh/upgrade/install.
|
||||
- `internal/storage` block devices: `lsblk -J` with a `/sys/block` fallback, eMMC health, NVMe namespace folding, SMART only when `smartctl` exists.
|
||||
- `internal/metrics` CPU/mem/hwmon/thermal/net/disk/fs from `/proc` and `/sys`; remote-FS and pseudo-FS lists; filesystem statfs with a timeout.
|
||||
- `internal/hardware`: every reader may come back empty with a note; device-tree fallbacks for non-SMBIOS boards.
|
||||
- `internal/nodes`, `internal/tlsauto`: hub ↔ node pairing, pinned certs, proxy, push.
|
||||
- `internal/sensors`, alerts, widgets, embeds, Prometheus: machine-agnostic by design.
|
||||
- `internal/files`, `internal/term`, `internal/tasks`, `internal/auth`, `internal/store`.
|
||||
- `internal/compose` (modulo 3.5).
|
||||
- The React UI: machine-switcher model, hash routing, per-account prefs, themes.
|
||||
|
||||
---
|
||||
|
||||
## Suggested order of work
|
||||
|
||||
1. **Posture and safety first (A):** 7.2 trusted proxies; 7.1 installer-written scoped sudoers; ~~3.1 drop the `:2375` probe (with 3.2 as its consequence)~~ done 2026-10-10.
|
||||
2. **"Any Linux" promises the README already makes (A):** 4.1 power, 4.4/4.5 cron on busybox and timer-only hosts, 4.3 a Services backend for OpenRC/procd, 2.10 the `tank` regex.
|
||||
3. **Lab defaults that phone home or mislead (B):** 1.1–1.3 update/install URLs, 2.9 schedule defaults, 6.9/6.10 speed-test sources and an off switch, 8.1 memory limit, ~~3.3/9.2 stack home = DataDir~~ done 2026-10-10.
|
||||
4. **Breadth (B):** 5.5 cgroup v1, 6.1/6.2 GPUs beyond Intel/AMD, 6.4 UPS order, 6.11/6.12 wg-quick/NetworkManager, 6.15 IPv6.
|
||||
5. **Cosmetics (C):** every placeholder and comment in §1.10 and §2, then split lab facts out of `PLAN.md`, `SENSORS.md`, `README.md`.
|
||||
@@ -1,10 +1,101 @@
|
||||
# Ham Lab
|
||||
|
||||
Web-based Linux management console for bare-metal Debian, in the style of the Proxmox
|
||||
Web-based Linux management console for bare-metal Linux, in the style of the Proxmox
|
||||
web UI. Named after Ham the cat (see also HamTV).
|
||||
|
||||
Status: **planning**. See [`PLAN.md`](PLAN.md) for the MVP scope and roadmap. No code yet.
|
||||
Status: **v0.1** — persistent web terminals and a live Dashboard tab. Scope and roadmap in
|
||||
[`PLAN.md`](PLAN.md).
|
||||
|
||||
- Runs as a systemd service on the host it manages, as user `core`.
|
||||
- Public at `https://hamlab.bigham.dev` (behind Nginx Proxy Manager on `core`).
|
||||
## Runs on any Linux box
|
||||
|
||||
One static Go binary with the UI embedded: no cgo, no libc, no Node at runtime, no agents.
|
||||
The same file runs on Debian, Ubuntu, Fedora, RHEL/Rocky/Alma, openSUSE, Arch and Alpine
|
||||
(musl), on bare metal, VMs and containers. Builds: `amd64`, `arm64`, `arm` (ARMv6
|
||||
hard-float and up, so every Raspberry Pi), `386`, `riscv64`.
|
||||
|
||||
```sh
|
||||
sudo ./hamlab-linux-amd64 install # that's the whole install
|
||||
```
|
||||
|
||||
or, from wherever `make release` output is served:
|
||||
|
||||
```sh
|
||||
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh # asks: user, address, HTTPS, firewall, password
|
||||
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh -s -- --yes # asks nothing, all defaults
|
||||
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh -s -- --listen 0.0.0.0:8100 --tls auto # flags go to `hamlab install`
|
||||
```
|
||||
|
||||
`install.sh` is a wizard when there is a terminal (it reads answers from `/dev/tty`, so it
|
||||
works piped from curl) and silent otherwise. Either way it picks the build for `uname -m`,
|
||||
checks it against `SHA256SUMS` and runs `hamlab install`; re-running it upgrades in place.
|
||||
`latest` is an alias Gitea resolves to the newest release, so that URL never changes.
|
||||
`make gitea-release` publishes a release there (`HAMLAB_URL` points the installer elsewhere); run the script beside the `hamlab-linux-*` files and it installs offline.
|
||||
`hamlab install`:
|
||||
|
||||
- copies the binary to `/usr/local/bin` (`--prefix`), atomically;
|
||||
- gets root through `sudo` or `doas` if you aren't root already, and runs the console as
|
||||
**you** (`--user`), so terminals are your login shell with your rights;
|
||||
- writes the config to `~/.config/hamlab/` (`--data`; an existing `~/apps/hamlab/` is kept)
|
||||
and asks for the first account's password (the account is named after the user the
|
||||
console runs as), or generates one and prints it once (`--password-stdin` for scripts);
|
||||
- registers a boot-time service with **systemd** or **OpenRC**, naming only groups that
|
||||
exist on the host, and waits for `/healthz`; with neither init it prints the one line
|
||||
to put under your own supervisor;
|
||||
- says so when firewalld or ufw will block the port (`--open-firewall` opens it).
|
||||
|
||||
Re-running it upgrades in place and keeps config, user and accounts. `hamlab uninstall
|
||||
[--purge]` reverses it.
|
||||
|
||||
Accounts: every person has their own user name and password, and the theme, accent
|
||||
colours and dashboard layout follow the account. The **Users** tab has it all: your own
|
||||
password, where you are signed in (sign out any device), and the accounts (add, remove,
|
||||
reset a password; your own password confirms the last two). From the shell:
|
||||
`hamlab passwd [user]` sets a password and makes the account if it is new, `hamlab
|
||||
users` lists them, `hamlab user-rm <user>` removes one. Machine settings — refresh rate,
|
||||
graph history, alert rules — stay the machine's and are shared by everyone.
|
||||
|
||||
**Guest logins**, for showing somebody the console: Users → Guest logins makes a
|
||||
throwaway user name and password, good for half an hour by default (up to 24 hours), and
|
||||
hands you the pair to paste into a message. A guest is an ordinary account with a clock on
|
||||
it — the whole console, every machine, the terminal included — that stops working when the
|
||||
time is up, wherever it is signed in, or the moment you revoke it. There is no reduced
|
||||
guest: give one to somebody you would hand the password to, and revoke it if you change
|
||||
your mind.
|
||||
|
||||
What adapts to the host, rather than assuming Debian:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Root | already root, else passwordless `sudo`, else passwordless `doas`; without any, the console is read-only where root is needed and says so |
|
||||
| Updates tab | apt, dnf, zypper, pacman, apk: list, refresh, upgrade, install (pacman: all or nothing) |
|
||||
| Services / timers | systemd of any age (JSON output where it exists, the plain table before v246); a clear message on hosts without systemd |
|
||||
| Disks | `lsblk`, or `/sys/block` when it's missing (busybox) or too old; SMART and ZFS only when their tools are there |
|
||||
| Network | `/proc` and `/sys` for the interfaces, addresses, link speed and default route. The speed test needs no tool installed: it measures against Cloudflare's public speed endpoints, so it works the same on Debian and on the OpenWrt router |
|
||||
| VPN | WireGuard tunnels and their peers, from `wg show` where wireguard-tools is installed; peer names from wg-quick `.conf` comments or OpenWrt UCI descriptions. Read-only, and it never reads a private or pre-shared key |
|
||||
| Dashboard | `/proc` and `/sys` only: hwmon or thermal-zone temps, fans, NICs, disks, mounts; empty sections where the hardware has nothing to report. UPS from whichever of `pwrstat`, NUT (`upsc`) or `apcupsd` is installed |
|
||||
| Listen address | bound with `IP_FREEBIND`, so a LAN address that isn't up yet at boot doesn't fail the start |
|
||||
| Self-update | the new build must run and load this host's config before it replaces the old one; atomic rename; the daemon then re-execs itself, so it restarts the same way under systemd, OpenRC or nothing; `hamlab.prev` kept |
|
||||
|
||||
Building: `make build` (web + Go), `make release` (every architecture, `install.sh` and
|
||||
`SHA256SUMS` in `dist/`), `make test`.
|
||||
|
||||
Terminals run as the user the service runs as, with that user's login shell.
|
||||
|
||||
## Managing several machines
|
||||
|
||||
Install Ham Lab on each host; on the ones the hub should reach:
|
||||
`hamlab install --listen <lan-ip>:8100 --tls auto` (a self-signed certificate is generated
|
||||
on first start). Then on the hub, open **Machines**, enter a name, the address
|
||||
(`https://host:8100`) and a user and password on that machine once (the user may be
|
||||
left out when it has only one). The hub pins the
|
||||
certificate, keeps an access token, and every tab works against the chosen machine
|
||||
through the switcher at the top of the rail.
|
||||
|
||||
Updating: `make deploy` on the hub installs the new build there and then pushes it to
|
||||
every managed machine (`hamlab push-nodes`), which install it and restart. The Machines
|
||||
page flags machines running an older version and offers the same push per machine.
|
||||
Mixed architectures are fine: `make deploy` also cross-compiles every other architecture,
|
||||
and the hub sends each machine the build for its own.
|
||||
|
||||
- On `core` it is public at `https://lab.bigham.dev` behind Nginx Proxy Manager.
|
||||
- Installable as a PWA.
|
||||
|
||||
+287
@@ -0,0 +1,287 @@
|
||||
# Sensors — the standard read interface
|
||||
|
||||
Everything Ham Lab measures is a **sensor**: a stable dotted ID, a little metadata, a
|
||||
current value. Any consumer — Ham Lab's own dashboard, Glance, Home Assistant, a shell
|
||||
script — discovers them from the catalog and asks for exactly the ones it wants.
|
||||
|
||||
Code: `internal/sensors` (types, registry, selectors, shaped query) and
|
||||
`internal/metrics/sensors.go` (the collector as a provider). The registry measures
|
||||
nothing; providers do. A new source of sensors implements `sensors.Provider` and is
|
||||
added to the registry in `api.New`.
|
||||
|
||||
## IDs
|
||||
|
||||
```
|
||||
cpu.usage cpu.core.<n>.usage cpu.temp cpu.power load.1|5|15 uptime
|
||||
mem.total|used|avail|free|cached|buffers|used_pct swap.total|used|avail|used_pct
|
||||
temp.<chip>.<label> fan.<chip>.<label>
|
||||
net.<iface>.rx|tx|rx_total|tx_total|up
|
||||
net.internet.download|upload|latency|jitter|tested|ip|edge
|
||||
disk.<dev>.read|write|util|size|temp
|
||||
fs.<mount>.size|used|used_pct
|
||||
ups.battery|runtime|load|load_pct|in_v|out_v|on_battery|state|source
|
||||
gpu.<name>.usage|render|video|video-enhance|copy|compute|temp|power|fan|freq|mem_used|mem_total|clients
|
||||
zfs.arc.size|target|min|max|mru|mfu|data|metadata|hit_pct|reads|misses|lifetime_hit_pct|no_grow|compress_ratio|l2_size|l2_hit_pct
|
||||
```
|
||||
|
||||
- Segments are lower case `[a-z0-9_-]`: `Package id 0` → `package_id_0`, `/mnt/tank` →
|
||||
`mnt_tank`, `/` → `root`. The real names are in the sensor's `labels`.
|
||||
- Built from names the hardware reports, never enumeration order (`hwmon3` moves between
|
||||
boots). Two identical chips collide; the second gets `_2`.
|
||||
- Which sensors exist differs per machine. That is the point of the catalog.
|
||||
- `net.internet.*` is the Network tab's speed test: `download` and `upload` in Mbps,
|
||||
`latency` and `jitter` in ms, `tested` the age of the reading in seconds, and `ip` and
|
||||
`edge` the public address and the Cloudflare edge as strings. They appear only once a test
|
||||
has run on that machine, and each one's `t` is when it ran — so a widget or an alert on
|
||||
them is reading a measurement from that moment, not a live value. Nothing takes a speed
|
||||
test on its own; something has to press the button.
|
||||
- `cpu.power` is the CPU packages' own power in watts, from the kernel's RAPL energy
|
||||
counters (Intel, and AMD Zen), summed over every package: cores, built-in graphics and
|
||||
memory controller, not the whole machine. The counters are root-only, so a daemon that is
|
||||
not root reads them through a helper started once with sudo or doas. It is absent on a
|
||||
machine without the counters (ARM, the Mac mini) or without a way to root.
|
||||
- `gpu.<name>.*` is each graphics device, named as lspci names it (`gpu.intel_arc_a380`;
|
||||
labels carry `driver` and the PCI `bus`). `usage` is how busy the device was — from the
|
||||
GT's idle residency on Intel, `gpu_busy_percent` on amdgpu — and is absent on a driver
|
||||
with no such counter. The per-engine percentages (`render`, `video`…) are the sum over the
|
||||
processes holding the device open, read from DRM fdinfo, and exist only while something
|
||||
has it open. `temp` is the card's own sensor, or on an integrated GPU the CPU package it
|
||||
shares a die with (label `source` says which); an Arc on i915 has none and reports none.
|
||||
`freq` is the actual clock, 0 when power-gated. `clients` counts the processes using it.
|
||||
- `zfs.arc.*` is the ZFS read cache in RAM, on a machine with the module loaded. `hit_pct`
|
||||
is over the last sampling interval and appears once an interval has had any reads (an idle
|
||||
interval repeats the last value); `lifetime_hit_pct` is since boot. `no_grow` is true
|
||||
while memory pressure stops the cache growing. `l2_*` appear only with cache devices.
|
||||
|
||||
## A sensor
|
||||
|
||||
```json
|
||||
{ "id": "temp.coretemp.package_id_0", "name": "coretemp Package id 0",
|
||||
"kind": "temperature", "unit": "°C", "value": 51, "t": 1789826600878,
|
||||
"warn": 105, "crit": 105, "labels": { "chip": "coretemp", "label": "Package id 0" } }
|
||||
```
|
||||
|
||||
`value` is a number, bool or string. `t` is when it was really read, in unix ms — slow
|
||||
sections (filesystems 30 s, UPS 10 s) are older than the tick. `min`/`max` are the scale
|
||||
(0–100 for percentages, 0–total for used bytes); `warn`/`crit` are the hardware's own
|
||||
thresholds where it states them.
|
||||
|
||||
## Endpoints
|
||||
|
||||
All behind the same auth as the rest of `/api` (session cookie or `Authorization: Bearer`).
|
||||
On a hub, every one is also at `/api/nodes/<machine>/sensors…` for each managed machine.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `GET /api/sensors` | Catalog, with current values. `?ids=fan.*,cpu.usage` narrows it. |
|
||||
| `GET /api/sensors/{id}` | One sensor. `?format=text` → the bare value as text/plain. |
|
||||
| `GET /api/sensors/{id}/history` | `{id, unit, t[], v[]}` — what the ring holds (300 points, spaced by the window below). Numbers only. |
|
||||
| `GET /api/sensors/history?ids=cpu.usage,net.*.rx` | Many histories in one request: `{series: [{id, unit, t[], v[]}]}`. A sensor younger than the ring has a shorter series. |
|
||||
| `GET /api/prefs/history` | How far back the history reaches: `{seconds, points, stride_sec, min_seconds, max_seconds}`. |
|
||||
| `PUT /api/prefs/history` | `{seconds}` — 60 to 86400. Set on the hub it is passed to every machine, so the fleet's graphs reach equally far back. |
|
||||
| `POST /api/sensors/query` | Shaped query, below. |
|
||||
| `GET /api/sensors/query?k=selector&…` | The same, flat, for consumers that can only fetch a URL. |
|
||||
|
||||
A selector is an exact ID or a glob. `*` crosses dots: `fan.*` is every fan,
|
||||
`cpu.core.*.usage` every core.
|
||||
|
||||
## Shaped query
|
||||
|
||||
Send the object you want, with selectors where the values go. Get it back filled in.
|
||||
|
||||
```
|
||||
POST /api/sensors/query
|
||||
{ "cpu": "cpu.usage", "fans": "fan.*", "box": { "temp": "cpu.temp", "up": "uptime" } }
|
||||
|
||||
→ { "cpu": 12.4,
|
||||
"fans": { "fan.nct6775.fan1": 820, "fan.nct6775.fan2": 1140 },
|
||||
"box": { "temp": 41, "up": 80520 } }
|
||||
```
|
||||
|
||||
- Exact ID → its value; `null` if this machine has no such sensor (so one query works
|
||||
across machines that differ).
|
||||
- Glob → an object keyed by ID; `{}` if nothing matches.
|
||||
- Objects and arrays are walked; anything else in a query is a 400.
|
||||
- `?full=1` → every value is the whole sensor object instead of the bare value. (In the
|
||||
GET form `full` is therefore not usable as a key.)
|
||||
|
||||
## Live stream
|
||||
|
||||
`GET /api/sensors/stream` is a shaped query that keeps answering: the same object, pushed
|
||||
as a server-sent event after every sample. The query is `?q=<URL-encoded JSON shape>` or
|
||||
the flat form; `?full=1` as above; `?every=<ms>` spaces events out for a consumer that
|
||||
doesn't want them at the sampling rate. A keepalive comment goes out every 25 s, and
|
||||
`X-Accel-Buffering: no` keeps nginx from holding events back.
|
||||
|
||||
```js
|
||||
new EventSource('/api/sensors/stream?token=…&q=' + encodeURIComponent(JSON.stringify({ cpu: 'cpu.usage', fans: 'fan.*' })))
|
||||
.onmessage = (e) => render(JSON.parse(e.data))
|
||||
```
|
||||
|
||||
## Auth: read-only tokens
|
||||
|
||||
Settings → Access tokens → **+ Read-only sensor token** (or `POST /api/tokens/read
|
||||
{"name"}` with a session). Scope `sensors:read`: it opens every endpoint in this file,
|
||||
here and on each managed machine through the hub (`/api/nodes/<machine>/sensors…`; the
|
||||
hub talks to the node as itself, the caller's token stops at the hub), and nothing else.
|
||||
|
||||
- Send it as `Authorization: Bearer <token>`, or as `?token=<token>` where headers are
|
||||
impossible (an iframe, a browser `EventSource`).
|
||||
- A read-only token can be **limited to ID patterns** (`{"name": "glance", "sensors":
|
||||
["cpu.*", "zfs.tank.*"]}`, or the second prompt in Settings). Everything outside them
|
||||
does not exist for that token — catalog, query, history, stream, Prometheus, and on
|
||||
managed machines through the hub, which passes the restriction along.
|
||||
- A **full-control token is never accepted from a URL**, and a read-only token is never
|
||||
accepted anywhere outside this file's endpoints.
|
||||
|
||||
## Prometheus
|
||||
|
||||
`GET /metrics` (or `/api/sensors/prometheus`, which also works through the hub as
|
||||
`/api/nodes/<machine>/sensors/prometheus`). One gauge family per kind of measurement,
|
||||
base units, the sensor ID and its real-world names as labels:
|
||||
|
||||
```
|
||||
hamlab_temperature_celsius{id="temp.coretemp.core_0",chip="coretemp",label="Core 0"} 50
|
||||
hamlab_usage_percent{id="fs.root.used_pct",fstype="ext4",mount="/"} 10.1
|
||||
hamlab_state{id="ups.state",model="CPS CP1500PFCRM2U",value="Normal"} 1
|
||||
hamlab_updates_pending{id="updates.pending",manager="apt"} 3
|
||||
```
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: hamlab
|
||||
authorization: { credentials: <read-only token> }
|
||||
static_configs: [{ targets: ['10.0.0.234:8100'] }]
|
||||
```
|
||||
|
||||
## Status sensors
|
||||
|
||||
Not everything is a /proc number. These are read in the background on their own clocks
|
||||
(each sensor's `t` is its real age) by `internal/api/status_sensors.go`:
|
||||
|
||||
```
|
||||
services.failed|failed_names|running every 30 s
|
||||
service.<unit>.active|state one pair per service that is enabled, up or failed
|
||||
containers.total|running|unhealthy every 15 s (Docker or Podman, if present)
|
||||
container.<name>.running|state|health|mem|cpu mem and cpu read from the container's cgroup
|
||||
container.<name>.down should be up and isn't: Docker couldn't start it, it is
|
||||
stuck restarting, or it died non-zero with no policy to
|
||||
restart it — *not* a container someone stopped
|
||||
updates.pending|security|reboot_required every 5 min
|
||||
zfs.<pool>.health|healthy|size|used|used_pct|frag every 1 min
|
||||
zfs.write.dirty|limit|delayed the RAM write buffer (all pools) and writes the throttle is slowing
|
||||
zfs.<pool>.write_bps|logical_bps bytes/s synced to disk, and written by programs
|
||||
zfs.<pool>.landing.pending_bytes|pending_files|used_pct|in_bps|drain_bps|mover|queue a landing tier, where one fronts the pool
|
||||
smart.<dev>.passed|warnings|power_on|wear every 10 min, never waking a sleeping disk
|
||||
```
|
||||
|
||||
## Widgets
|
||||
|
||||
A widget is a spec — `{type, sensor, title?}` — over this API and nothing else
|
||||
(`web/src/widgets.tsx`, fed by `useSensorStream` in `web/src/sensors.ts`). Types: `stat`,
|
||||
`gauge`, `bar`, `spark`, and `list` for a pattern. Scale and colour come from the
|
||||
sensor's own `min`/`max`/`warn`/`crit`.
|
||||
|
||||
- **In Ham Lab:** Dashboard → Customize → **+ Add widget**. Saved with the layout,
|
||||
per machine; all of a page's widgets share one stream.
|
||||
- **Widgets tab:** every widget type, then every sensor on the machine drawn once, live,
|
||||
grouped by family (`web/src/Gallery.tsx`). Tap a tile to copy its sensor ID — the
|
||||
quickest way to find what to put in a widget, a query or a Glance tile.
|
||||
- **Anywhere else:** `/embed/widget` is one widget alone on a page, live, in its own small
|
||||
bundle (the console never loads), and the only path Ham Lab allows to be framed.
|
||||
|
||||
```
|
||||
/embed/widget?type=gauge&sensor=cpu.usage&token=<read-only token>
|
||||
&machine=grid &title=… &theme=dark|light|cyberpunk|retro &every=<ms> &bare=1
|
||||
```
|
||||
|
||||
### Glance
|
||||
|
||||
Glance's own `custom-api` widget refetches on page load / cache expiry — fine for slow
|
||||
facts, never live. An `iframe` of the embed is live.
|
||||
|
||||
```yaml
|
||||
- type: iframe
|
||||
title: core CPU
|
||||
source: https://lab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
|
||||
height: 130
|
||||
|
||||
- type: custom-api
|
||||
title: grid
|
||||
cache: 1m
|
||||
url: https://lab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
|
||||
headers: { Authorization: Bearer <read-only token> }
|
||||
template: |
|
||||
<div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>
|
||||
```
|
||||
|
||||
## The whole fleet in one request
|
||||
|
||||
On a hub, `/api/fleet/sensors/query` asks every machine (or `machines`) the same shaped
|
||||
query and answers `{machine: result}`; this machine answers under its own hostname, and
|
||||
one that can't be reached answers `{"error": …}` while the rest still do.
|
||||
|
||||
```
|
||||
POST /api/fleet/sensors/query {"machines": "*", "query": {"cpu": "cpu.usage", "up": "uptime"}}
|
||||
GET /api/fleet/sensors/query?machines=grid,wyse&cpu=cpu.usage&failed=services.failed
|
||||
|
||||
→ {"core": {"cpu": 4.8, "up": 50211}, "grid": {"cpu": 0.4, "up": 437177}, "mac": {"error": "timeout"}}
|
||||
```
|
||||
|
||||
## What counts as a disk
|
||||
|
||||
`disk.*` names **drives**, not the block devices the kernel lists. An NVMe controller can
|
||||
present several namespaces — an Apple SSD shows `nvme0n1`, `nvme0n2` and `nvme0n3`, the last
|
||||
two a few MiB of firmware — and each one arrives from `/proc/diskstats` looking like a whole
|
||||
disk. They are folded into the controller (`disk.nvme0.*`), with the rates added and busy
|
||||
taken as the longest of them rather than the sum, since they share one controller. This is
|
||||
what the Disks page has always shown; since 2026-09-22 the dashboard, the Top tab and these
|
||||
sensors agree with it.
|
||||
|
||||
## The Top tab
|
||||
|
||||
Ham Lab → **Top** is a whole machine on one screen in the style of btop: black ground, boxes
|
||||
with their name set into the border, curves drawn as dots that climb green → yellow → red.
|
||||
It reads nothing of its own — the sensor stream for everything measured, `/api/procs` for the
|
||||
table — so it works on any machine the console can reach, whether or not btop is installed
|
||||
there, and it follows the same refresh rate and history window as the rest of the console.
|
||||
Its colours are the active skin's: the terminal background for the ground, the skin's sixteen
|
||||
terminal colours for the curves and meters. (The look is btop's, `web/src/Top.tsx` is ours;
|
||||
btop itself is Apache-2.0.)
|
||||
|
||||
## How often a machine samples
|
||||
|
||||
The console's refresh rate (Settings, or `− rate +` on Top) is also the fleet's sampling rate.
|
||||
It is saved with the UI prefs on the hub, which sets its own sampler and passes the rate to
|
||||
every machine it manages — `PUT /api/prefs/sampler {"ms": …}`, 100 ms to 60 s, sampled no
|
||||
slower than the collector's 10 s — and repeats it at start-up. `GET` answers `{ms, min_ms,
|
||||
max_ms}`. Before 2026-09-21 only an open Dashboard page told a machine its rate.
|
||||
|
||||
## How far back a history reaches
|
||||
|
||||
Every machine keeps **300 points per curve**, whatever the window — so the window sets how
|
||||
far apart they are, not how much memory monitoring costs. At the default ten minutes each
|
||||
point is one sample (2 s apart at the default rate); at a day each is the **average of the
|
||||
five minutes behind it**, not whichever instant landed on the mark, so a long graph shows
|
||||
the load that really happened. Ham Lab → Settings → **Graph history** is the one control,
|
||||
1 minute to 24 hours; the hub passes it to every machine it manages.
|
||||
|
||||
The window can't be finer than the sampling interval — 300 points of a 1-minute window at a
|
||||
2-second rate is 30 points, and that is what you get. It is also all in memory: a machine
|
||||
that restarts starts its graphs again.
|
||||
|
||||
## What Ham Lab itself runs on
|
||||
|
||||
Since 2026-09-19 the Dashboard tab (stock modules and custom widgets) and the Disks tab's
|
||||
live rates read only this API: the catalog once, one batch of histories, one stream —
|
||||
closed while the page is hidden, refilled when it returns. `web/src/sampleAdapter.ts` is
|
||||
the seam that hands the stock modules the object they were written against. The old
|
||||
`/api/metrics` and `/api/metrics/history` are gone. `hamlab.sample_interval` is the
|
||||
sampler's own rate, as a sensor.
|
||||
|
||||
## From another origin
|
||||
|
||||
The sensor endpoints (and only they) answer any origin: `Access-Control-Allow-Origin: *`,
|
||||
preflight included. `*` is the one value browsers won't combine with cookies, so a foreign
|
||||
page can reach the API with a token and can never ride a logged-in console's session.
|
||||
`examples/external-demo/` is a complete outside consumer in one HTML file.
|
||||
+527
-18
@@ -2,16 +2,24 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"runtime/debug"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
@@ -20,8 +28,21 @@ import (
|
||||
"hamlab/internal/api"
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/cron"
|
||||
"hamlab/internal/files"
|
||||
"hamlab/internal/media"
|
||||
"hamlab/internal/metrics"
|
||||
"hamlab/internal/nodes"
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/services"
|
||||
"hamlab/internal/setup"
|
||||
"hamlab/internal/space"
|
||||
"hamlab/internal/storage"
|
||||
"hamlab/internal/store"
|
||||
"hamlab/internal/tasks"
|
||||
hterm "hamlab/internal/term"
|
||||
"hamlab/internal/tlsauto"
|
||||
"hamlab/internal/updates"
|
||||
"hamlab/internal/web"
|
||||
)
|
||||
|
||||
@@ -30,12 +51,26 @@ var version = "dev"
|
||||
func main() {
|
||||
defaultCfg := os.Getenv("HAMLAB_CONFIG")
|
||||
if defaultCfg == "" {
|
||||
home, _ := os.UserHomeDir()
|
||||
defaultCfg = filepath.Join(home, "apps", "hamlab", "config.json")
|
||||
defaultCfg = config.DefaultPath()
|
||||
}
|
||||
cfgPath := flag.String("config", defaultCfg, "path to config.json")
|
||||
flag.Usage = func() {
|
||||
fmt.Fprintf(os.Stderr, "usage: hamlab [-config path] [serve|passwd|version]\n")
|
||||
fmt.Fprint(os.Stderr, `usage: hamlab [-config path] <command>
|
||||
|
||||
install [flags] install this binary as a service on this machine (install -h for flags)
|
||||
uninstall [--purge] remove the service and the binary
|
||||
serve run the console in the foreground (what the service runs)
|
||||
passwd [user] set a user's password (makes the account if it is new; reads stdin when piped)
|
||||
users list the accounts
|
||||
zfs-snapshot <id> run one ZFS snapshot schedule (what its cron line calls)
|
||||
landing-verify <new|all|id> verify a landing tier against its mover's ledger (a schedule's id is what its cron line calls)
|
||||
docker-update <id> run one image-update schedule (what its cron line calls)
|
||||
user-rm <user> remove an account and sign it out everywhere
|
||||
check verify the config and state load, then exit
|
||||
version
|
||||
push-nodes [--force] hub: send this build to every managed machine
|
||||
|
||||
`)
|
||||
flag.PrintDefaults()
|
||||
}
|
||||
flag.Parse()
|
||||
@@ -44,6 +79,51 @@ func main() {
|
||||
if flag.NArg() > 0 {
|
||||
cmd = flag.Arg(0)
|
||||
}
|
||||
switch cmd { // commands that don't read the config
|
||||
case "fs": // privileged helper for the Files tab (run as root by the daemon)
|
||||
os.Exit(files.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||
case "thumb": // privileged helper for the Files tab's previews in root mode (run as root by the daemon)
|
||||
os.Exit(media.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||
case "run-ffmpeg": // privileged helper for playback in root mode: ffmpeg as root, ending when stdin closes
|
||||
os.Exit(media.RunFFmpegHelper(flag.Args()[1:], os.Stdin, os.Stderr))
|
||||
case "space-walk": // helper for the Space tab: measure mountpoints, stream the tree back
|
||||
os.Exit(space.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||
case "drm-clients": // privileged helper for the GPU readings: who has the GPU open (run as root by the daemon)
|
||||
os.Exit(metrics.DRMClientsHelper(os.Stdin, os.Stdout))
|
||||
case "cpu-energy": // privileged helper for the CPU power reading: the packages' energy counters (run as root by the daemon)
|
||||
os.Exit(metrics.CPUEnergyHelper(os.Stdin, os.Stdout))
|
||||
case "install", "uninstall":
|
||||
var err error
|
||||
if cmd == "install" {
|
||||
err = setup.Install(flag.Args()[1:], version)
|
||||
} else {
|
||||
err = setup.Uninstall(flag.Args()[1:])
|
||||
}
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
os.Exit(0)
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "hamlab "+cmd+":", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return
|
||||
case "replace": // self-update helper (run as root by the daemon): put this binary at a path, atomically
|
||||
// Only ever over a hamlab binary: this runs as root, and must not be a way to
|
||||
// overwrite arbitrary files for whoever is allowed to run hamlab as root.
|
||||
if flag.NArg() != 2 || !filepath.IsAbs(flag.Arg(1)) || !strings.HasPrefix(filepath.Base(flag.Arg(1)), "hamlab") {
|
||||
fmt.Fprintln(os.Stderr, "hamlab replace: destination must be an absolute path to a hamlab binary")
|
||||
os.Exit(2)
|
||||
}
|
||||
self, err := os.Executable()
|
||||
if err == nil {
|
||||
err = setup.CopyAtomic(self, flag.Arg(1))
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "hamlab replace:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return
|
||||
}
|
||||
cfg, err := config.Load(*cfgPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
@@ -51,10 +131,74 @@ func main() {
|
||||
switch cmd {
|
||||
case "version":
|
||||
fmt.Println("hamlab", version)
|
||||
case "passwd":
|
||||
if err := passwd(cfg); err != nil {
|
||||
case "check": // a new build runs this against the live config before it replaces the old one
|
||||
if _, err := store.Open(filepath.Join(cfg.DataDir(), "state.json")); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
if _, err := fs.Stat(web.FS(), "index.html"); err != nil {
|
||||
log.Fatal("embedded UI missing: ", err)
|
||||
}
|
||||
fmt.Println("ok")
|
||||
case "passwd":
|
||||
if err := passwd(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "docker-update":
|
||||
if err := dockerUpdateJob(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "zfs-snapshot":
|
||||
if err := zfsSnapshot(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "landing-verify":
|
||||
if err := landingVerify(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "users":
|
||||
migrateUsers(cfg)
|
||||
if len(cfg.Users) == 0 {
|
||||
fmt.Println("no accounts in", cfg.Path())
|
||||
}
|
||||
for _, u := range cfg.Users {
|
||||
seen := "never"
|
||||
if !u.LastSeen.IsZero() {
|
||||
seen = u.LastSeen.Local().Format("2006-01-02 15:04")
|
||||
}
|
||||
note := ""
|
||||
if u.Guest { // a temporary look-only login, not one of the machine's accounts
|
||||
note = " guest, until " + u.Expires.Local().Format("15:04")
|
||||
if u.Expired(time.Now()) {
|
||||
note = " guest, expired"
|
||||
}
|
||||
}
|
||||
fmt.Printf("%-20s created %s last seen %s%s\n", u.Name, u.Created.Local().Format("2006-01-02"), seen, note)
|
||||
}
|
||||
case "user-rm":
|
||||
if err := userRm(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "push-nodes": // hub CLI: send this binary to every managed machine
|
||||
exe, _ := os.Executable()
|
||||
host, _ := os.Hostname()
|
||||
nm := nodes.New(cfg, host)
|
||||
force := flag.NArg() > 1 && flag.Arg(1) == "--force"
|
||||
results := nm.PushAll(context.Background(), exe, runtime.GOARCH, force)
|
||||
if len(results) == 0 {
|
||||
fmt.Println("no managed machines in", cfg.Path())
|
||||
}
|
||||
failed := 0
|
||||
for _, r := range results {
|
||||
if r.OK {
|
||||
fmt.Printf("%-12s updated to %s\n", r.Name, r.Version)
|
||||
} else {
|
||||
failed++
|
||||
fmt.Printf("%-12s FAILED: %s\n", r.Name, r.Error)
|
||||
}
|
||||
}
|
||||
if failed > 0 {
|
||||
os.Exit(1)
|
||||
}
|
||||
case "serve":
|
||||
if err := serve(cfg); err != nil {
|
||||
log.Fatal(err)
|
||||
@@ -65,9 +209,245 @@ func main() {
|
||||
}
|
||||
}
|
||||
|
||||
func passwd(cfg *config.Config) error {
|
||||
fmt.Fprint(os.Stderr, "New password: ")
|
||||
p1, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
// migrateUsers gives a pre-users config its first account, named after this OS user, so
|
||||
// the CLI sees the same accounts the daemon will.
|
||||
// dockerUpdateJob pulls a stack's images, and deploys them when the schedule says to. This
|
||||
// is what the cron line Ham Lab writes actually calls, so image updates keep happening
|
||||
// whether or not the console is up.
|
||||
func dockerUpdateJob(cfg *config.Config, id string) error {
|
||||
if strings.TrimSpace(id) == "" {
|
||||
return errors.New("usage: hamlab docker-update <schedule-id>")
|
||||
}
|
||||
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var raw json.RawMessage
|
||||
st.View(func(s *store.State) { raw = s.DockerJobs })
|
||||
var d struct {
|
||||
Jobs []api.UpdateJob `json:"jobs"`
|
||||
}
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &d); err != nil {
|
||||
return fmt.Errorf("reading the schedules: %w", err)
|
||||
}
|
||||
}
|
||||
for _, j := range d.Jobs {
|
||||
if j.ID != id {
|
||||
continue
|
||||
}
|
||||
// A pull that never finishes must not leave a cron job resident until the next reboot.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Hour)
|
||||
defer cancel()
|
||||
out, runErr := api.RunUpdate(ctx, st, j, api.DockerTool())
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: j.Stack}
|
||||
if runErr != nil {
|
||||
rec.Error = runErr.Error()
|
||||
}
|
||||
if err := storage.WriteRun(cfg.DataDir(), "docker-"+id, rec); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "recording the run:", err)
|
||||
}
|
||||
fmt.Print(out)
|
||||
return runErr
|
||||
}
|
||||
return fmt.Errorf("no image-update schedule %q — it may have been removed; this cron line is stale", id)
|
||||
}
|
||||
|
||||
// zfsSnapshot runs one snapshot schedule: take it, then expire what has aged out. This is
|
||||
// what the cron line Ham Lab writes actually calls, so a schedule keeps running whether or
|
||||
// not the console is up — and keeps running if Ham Lab is never opened again.
|
||||
//
|
||||
// It deliberately does not write state.json. The daemon owns that file and would clobber
|
||||
// anything written from out here; the outcome goes beside it instead, in zfs-runs/.
|
||||
// landingVerify runs one verify pass over every landing tier on this machine: by hand with a
|
||||
// scope ("new" or "all"), or as a schedule by its id, which is what the cron line passes.
|
||||
func landingVerify(cfg *config.Config, arg string) error {
|
||||
scope, id := arg, "manual"
|
||||
if arg != "new" && arg != "all" {
|
||||
if strings.TrimSpace(arg) == "" {
|
||||
return errors.New("usage: hamlab landing-verify <new|all|schedule-id>")
|
||||
}
|
||||
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var raw json.RawMessage
|
||||
st.View(func(s *store.State) { raw = s.ZFS })
|
||||
var z struct {
|
||||
Verify []storage.VerifyPolicy `json:"verify"`
|
||||
}
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &z); err != nil {
|
||||
return fmt.Errorf("reading the schedules: %w", err)
|
||||
}
|
||||
}
|
||||
found := false
|
||||
for _, p := range z.Verify {
|
||||
if p.ID == arg {
|
||||
scope, id, found = p.Scope, p.ID, true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return fmt.Errorf("no verify schedule %q — it may have been removed; this cron line is stale", arg)
|
||||
}
|
||||
}
|
||||
dir := cfg.DataDir()
|
||||
unlock, err := storage.LockVerify(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer unlock()
|
||||
defer storage.ClearVerifyProgress(dir)
|
||||
targets := storage.LandingTargets()
|
||||
if len(targets) == 0 {
|
||||
r := storage.VerifyResult{ID: id, Scope: scope, Started: time.Now(), Finished: time.Now(), Issues: []storage.VerifyIssue{},
|
||||
Error: "no landing mover is reporting (nothing in " + storage.MoverStatusDir + "), so there is nothing to verify"}
|
||||
_ = storage.RecordVerify(dir, storage.LandingTarget{}, r)
|
||||
return errors.New(r.Error)
|
||||
}
|
||||
var firstErr error
|
||||
for _, t := range targets {
|
||||
r := storage.Verify(t, id, scope, storage.VerifyCursor(dir, t), func(p storage.VerifyProgress) { storage.WriteVerifyProgress(dir, p) })
|
||||
if err := storage.RecordVerify(dir, t, r); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "recording the pass:", err)
|
||||
}
|
||||
fmt.Printf("%s -> %s (%s): %d checked, %d match, %d mismatch, %d gone, %d unreadable, %d changed since, %d waiting, %d unhashed, %s read in %s\n",
|
||||
t.Src, t.Dst, scope, r.Checked, r.Matched, r.Mismatched, r.Gone, r.Unreadable, r.Changed, r.Waiting, r.Unhashed,
|
||||
humanBytes(r.BytesRead), r.Finished.Sub(r.Started).Round(time.Second))
|
||||
if r.Error != "" && firstErr == nil {
|
||||
firstErr = errors.New(r.Error)
|
||||
}
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
|
||||
func humanBytes(n int64) string {
|
||||
const u = 1024
|
||||
if n < u {
|
||||
return fmt.Sprintf("%d B", n)
|
||||
}
|
||||
div, exp := int64(u), 0
|
||||
for v := n / u; v >= u; v /= u {
|
||||
div *= u
|
||||
exp++
|
||||
}
|
||||
return fmt.Sprintf("%.1f %ciB", float64(n)/float64(div), "KMGTPE"[exp])
|
||||
}
|
||||
|
||||
func zfsSnapshot(cfg *config.Config, id string) error {
|
||||
if strings.TrimSpace(id) == "" {
|
||||
return errors.New("usage: hamlab zfs-snapshot <schedule-id>")
|
||||
}
|
||||
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var raw json.RawMessage
|
||||
st.View(func(s *store.State) { raw = s.ZFS })
|
||||
var z struct {
|
||||
Policies []storage.Policy `json:"policies"`
|
||||
}
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &z); err != nil {
|
||||
return fmt.Errorf("reading the schedules: %w", err)
|
||||
}
|
||||
}
|
||||
var p *storage.Policy
|
||||
for i := range z.Policies {
|
||||
if z.Policies[i].ID == id {
|
||||
p = &z.Policies[i]
|
||||
}
|
||||
}
|
||||
if p == nil {
|
||||
return fmt.Errorf("no snapshot schedule %q — it may have been removed; this cron line is stale", id)
|
||||
}
|
||||
res, runErr := storage.New(priv.Available()).Run(context.Background(), p)
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: res.Taken, Pruned: res.Pruned}
|
||||
if runErr != nil {
|
||||
rec.Error = runErr.Error()
|
||||
}
|
||||
if err := storage.WriteRun(cfg.DataDir(), id, rec); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "recording the run:", err)
|
||||
}
|
||||
if runErr != nil {
|
||||
return runErr
|
||||
}
|
||||
fmt.Printf("%s: took %s", p.Name, res.Taken)
|
||||
if len(res.Pruned) > 0 {
|
||||
fmt.Printf(", expired %d", len(res.Pruned))
|
||||
}
|
||||
fmt.Println()
|
||||
return nil
|
||||
}
|
||||
|
||||
func migrateUsers(cfg *config.Config) {
|
||||
name := os.Getenv("USER")
|
||||
if u, err := user.Current(); err == nil && u.Username != "" {
|
||||
name = u.Username
|
||||
}
|
||||
if cfg.MigrateUsers(name) {
|
||||
_ = cfg.Save()
|
||||
}
|
||||
}
|
||||
|
||||
func userRm(cfg *config.Config, name string) error {
|
||||
migrateUsers(cfg)
|
||||
if name == "" {
|
||||
return errors.New("usage: hamlab user-rm <user>")
|
||||
}
|
||||
if len(cfg.Users) <= 1 {
|
||||
return errors.New("the last account can't be removed")
|
||||
}
|
||||
kept := cfg.Users[:0]
|
||||
found := false
|
||||
for _, u := range cfg.Users {
|
||||
if u.Name == name {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
if !found {
|
||||
return fmt.Errorf("no account %q", name)
|
||||
}
|
||||
cfg.Users = kept
|
||||
if err := cfg.Save(); err != nil {
|
||||
return err
|
||||
}
|
||||
if st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json")); err == nil {
|
||||
auth.NewSessions(st).RevokeUser(name)
|
||||
_ = st.Update(func(s *store.State) { delete(s.Users, name) })
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "removed", name, "(the running service picks it up at its next config save; restart it to be sure)")
|
||||
return nil
|
||||
}
|
||||
|
||||
func passwd(cfg *config.Config, name string) error {
|
||||
migrateUsers(cfg)
|
||||
if name == "" {
|
||||
switch len(cfg.Users) {
|
||||
case 0:
|
||||
name = os.Getenv("USER")
|
||||
if u, err := user.Current(); err == nil && u.Username != "" {
|
||||
name = u.Username
|
||||
}
|
||||
case 1:
|
||||
name = cfg.Users[0].Name
|
||||
default:
|
||||
return errors.New("several accounts: hamlab passwd <user>")
|
||||
}
|
||||
}
|
||||
if !auth.UserNameRe.MatchString(name) {
|
||||
return fmt.Errorf("%q: a user name is 1–32 lower-case letters, digits, . _ or -", name)
|
||||
}
|
||||
var p1 []byte
|
||||
if !term.IsTerminal(int(os.Stdin.Fd())) { // scripted: one line on stdin
|
||||
line, _ := bufio.NewReader(os.Stdin).ReadString('\n')
|
||||
p1 = []byte(strings.TrimRight(line, "\r\n"))
|
||||
} else {
|
||||
fmt.Fprintf(os.Stderr, "New password for %s: ", name)
|
||||
var err error
|
||||
p1, err = term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -81,24 +461,39 @@ func passwd(cfg *config.Config) error {
|
||||
if string(p1) != string(p2) {
|
||||
return errors.New("passwords do not match")
|
||||
}
|
||||
if len(p1) < 12 {
|
||||
}
|
||||
if len(p1) < auth.MinPasswordLen {
|
||||
return errors.New("password must be at least 12 characters")
|
||||
}
|
||||
h, err := auth.HashPassword(string(p1))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cfg.PasswordHash = h
|
||||
made := true
|
||||
for i := range cfg.Users {
|
||||
if cfg.Users[i].Name == name {
|
||||
cfg.Users[i].Hash = h
|
||||
made = false
|
||||
}
|
||||
}
|
||||
if made {
|
||||
cfg.Users = append(cfg.Users, config.User{Name: name, Hash: h, Created: time.Now()})
|
||||
}
|
||||
if err := cfg.Save(); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "password set in", cfg.Path())
|
||||
if made {
|
||||
fmt.Fprintln(os.Stderr, "account", name, "made in", cfg.Path())
|
||||
} else {
|
||||
fmt.Fprintln(os.Stderr, "password of", name, "set in", cfg.Path())
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "the running service reads the config at its next save; restart it to be sure")
|
||||
return nil
|
||||
}
|
||||
|
||||
func serve(cfg *config.Config) error {
|
||||
if cfg.PasswordHash == "" {
|
||||
return fmt.Errorf("no password set — run: hamlab -config %s passwd", cfg.Path())
|
||||
if cfg.PasswordHash == "" && len(cfg.Users) == 0 {
|
||||
return fmt.Errorf("no account set — run: hamlab -config %s passwd", cfg.Path())
|
||||
}
|
||||
if err := os.MkdirAll(cfg.DataDir(), 0o700); err != nil {
|
||||
return err
|
||||
@@ -115,15 +510,17 @@ func serve(cfg *config.Config) error {
|
||||
}
|
||||
shell := os.Getenv("SHELL")
|
||||
if shell == "" {
|
||||
shell = "/bin/bash"
|
||||
shell = loginShell(u.Username)
|
||||
}
|
||||
mgr := hterm.NewManager(shell, u.HomeDir)
|
||||
mgr.OnChange = func(list []hterm.Info) {
|
||||
_ = st.Update(func(s *store.State) {
|
||||
s.Terminals = s.Terminals[:0]
|
||||
for _, t := range list {
|
||||
if t.Running { // an exited shell must not come back from the dead on restart
|
||||
s.Terminals = append(s.Terminals, store.Terminal{ID: t.ID, Name: t.Name})
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
// Re-open the shells that existed before the last restart (fresh processes,
|
||||
@@ -136,12 +533,57 @@ func serve(cfg *config.Config) error {
|
||||
}
|
||||
}
|
||||
|
||||
srv := api.New(cfg, sessions, mgr, web.FS(), version)
|
||||
// Keep the heap tidy: the daemon's live data is tens of MB; without a limit the GC
|
||||
// happily lets RSS coast at twice that. (Soft limit: exceeding it only means more GC.)
|
||||
debug.SetMemoryLimit(160 << 20)
|
||||
|
||||
col := metrics.NewCollector()
|
||||
go col.Run(context.Background())
|
||||
|
||||
svc := services.New()
|
||||
cr := cron.New(svc.Sudo(), svc.JournalSudo())
|
||||
tm := tasks.New()
|
||||
up := updates.New(svc.Sudo(), tm)
|
||||
sto := storage.New(svc.Sudo())
|
||||
self, _ := os.Executable()
|
||||
var sudoFS files.FS
|
||||
if svc.Sudo() && self != "" {
|
||||
sudoFS = files.Sudo{Self: self}
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
nm := nodes.New(cfg, host)
|
||||
go nm.Run(context.Background())
|
||||
|
||||
certPath, keyPath, fingerprint := "", "", ""
|
||||
if cfg.TLS == "auto" {
|
||||
var err error
|
||||
certPath, keyPath, fingerprint, err = tlsauto.Ensure(cfg.DataDir())
|
||||
if err != nil {
|
||||
return fmt.Errorf("tls: %w", err)
|
||||
}
|
||||
}
|
||||
ln, err := listen(cfg.Listen)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
srv := api.New(cfg, sessions, mgr, col, st, svc, cr, tm, up, sto, files.Local{}, sudoFS, nm, fingerprint, web.FS(), version)
|
||||
hs := &http.Server{
|
||||
Addr: cfg.Listen,
|
||||
Handler: srv.Handler(),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
// After a self-update: become the new binary in place. Same PID, so it works the same
|
||||
// under systemd, OpenRC, or no supervisor at all. It returns only if the exec failed, by
|
||||
// which point the listener and the shells are gone; the caller puts the old binary back
|
||||
// and exits so the supervisor starts it.
|
||||
var restarting atomic.Bool
|
||||
srv.Restart = func() error {
|
||||
restarting.Store(true) // Serve returns once Shutdown is called; serve() must not exit under us
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
_ = hs.Shutdown(ctx)
|
||||
mgr.Shutdown()
|
||||
return syscall.Exec(self, os.Args, os.Environ())
|
||||
}
|
||||
go func() {
|
||||
sig := make(chan os.Signal, 1)
|
||||
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
|
||||
@@ -152,9 +594,76 @@ func serve(cfg *config.Config) error {
|
||||
_ = hs.Shutdown(ctx)
|
||||
mgr.Shutdown()
|
||||
}()
|
||||
log.Printf("hamlab %s listening on %s as %s (shell %s)", version, cfg.Listen, u.Username, shell)
|
||||
if err := hs.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
|
||||
if certPath != "" {
|
||||
log.Printf("hamlab %s listening on https://%s as %s (shell %s) cert %s", version, cfg.Listen, u.Username, shell, fingerprint)
|
||||
if err := hs.ServeTLS(ln, certPath, keyPath); !errors.Is(err, http.ErrServerClosed) {
|
||||
return err
|
||||
}
|
||||
return waitIf(&restarting)
|
||||
}
|
||||
log.Printf("hamlab %s listening on %s as %s (shell %s)", version, cfg.Listen, u.Username, shell)
|
||||
if err := hs.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
|
||||
return err
|
||||
}
|
||||
return waitIf(&restarting)
|
||||
}
|
||||
|
||||
// waitIf parks the caller while a restart is replacing the process.
|
||||
func waitIf(restarting *atomic.Bool) error {
|
||||
if restarting.Load() {
|
||||
select {}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// listen binds the configured address with IP_FREEBIND, so a LAN address that isn't up yet
|
||||
// (boot, DHCP still negotiating, cable out) doesn't stop the daemon: it binds now and
|
||||
// answers as soon as the address appears.
|
||||
func listen(addr string) (net.Listener, error) {
|
||||
lc := net.ListenConfig{Control: func(network, _ string, c syscall.RawConn) error {
|
||||
return c.Control(func(fd uintptr) {
|
||||
_ = syscall.SetsockoptInt(int(fd), syscall.IPPROTO_IP, syscall.IP_FREEBIND, 1)
|
||||
})
|
||||
}}
|
||||
ln, err := lc.Listen(context.Background(), "tcp", addr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listen %s: %w", addr, err)
|
||||
}
|
||||
if host, _, _ := net.SplitHostPort(addr); host != "" {
|
||||
if ip := net.ParseIP(host); ip != nil && !ip.IsUnspecified() && !ip.IsLoopback() && !onInterface(ip) {
|
||||
log.Printf("listen address %s isn't on any interface yet; bound anyway and waiting for it", host)
|
||||
}
|
||||
}
|
||||
return ln, nil
|
||||
}
|
||||
|
||||
func onInterface(ip net.IP) bool {
|
||||
addrs, _ := net.InterfaceAddrs()
|
||||
for _, a := range addrs {
|
||||
if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// loginShell returns the user's shell from /etc/passwd, falling back to bash if
|
||||
// present and /bin/sh otherwise. Systemd units don't export $SHELL.
|
||||
func loginShell(username string) string {
|
||||
if b, err := os.ReadFile("/etc/passwd"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
f := strings.Split(line, ":")
|
||||
if len(f) >= 7 && f[0] == username && f[6] != "" {
|
||||
if _, err := os.Stat(f[6]); err == nil {
|
||||
return f[6]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, sh := range []string{"/bin/bash", "/usr/bin/bash", "/bin/zsh", "/bin/sh"} {
|
||||
if _, err := os.Stat(sh); err == nil {
|
||||
return sh
|
||||
}
|
||||
}
|
||||
return "/bin/sh"
|
||||
}
|
||||
|
||||
@@ -1,18 +0,0 @@
|
||||
[Unit]
|
||||
Description=Ham Lab web console
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=core
|
||||
Group=core
|
||||
WorkingDirectory=/home/core
|
||||
Environment=HAMLAB_CONFIG=/home/core/apps/hamlab/config.json
|
||||
ExecStart=/usr/local/bin/hamlab serve
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
KillMode=control-group
|
||||
TimeoutStopSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -1,11 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Install the built binary and unit on this host, then (re)start the service.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
sudo install -m 0755 bin/hamlab /usr/local/bin/hamlab
|
||||
sudo install -m 0644 deploy/hamlab.service /etc/systemd/system/hamlab.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now hamlab.service >/dev/null
|
||||
sudo systemctl restart hamlab.service
|
||||
sleep 1
|
||||
systemctl is-active hamlab.service && /usr/local/bin/hamlab version
|
||||
@@ -0,0 +1,21 @@
|
||||
# External demo
|
||||
|
||||
`index.html` is a stranger's page: one file, no build, no framework, served from a
|
||||
different origin than Ham Lab. It knows a base URL and a read-only sensor token
|
||||
(`config.js`, not in git) and nothing else. Each tab is one shaped query POSTed to
|
||||
`/api/sensors/query?full=1` on an interval; sparklines are seeded from
|
||||
`/api/sensors/history`; the Fleet tab is one call to `/api/fleet/sensors/query`.
|
||||
|
||||
```js
|
||||
// config.js
|
||||
window.HAMLAB = { base: 'http://10.0.0.234:8100', token: '<read-only sensor token>' }
|
||||
```
|
||||
|
||||
Anyone who can load the page can read the token. If the page is public, limit the token to
|
||||
patterns that name nothing you'd mind the world reading — hardware numbers and counts,
|
||||
not container or service names.
|
||||
|
||||
This is an example to read and copy, not something that runs anywhere. It was deployed on
|
||||
`core` as the `hamlab-demo` stack and linked from the Widgets tab until 2026-09-23, when
|
||||
both were removed: the Widgets tab shows the same widgets against the machine you are
|
||||
already signed in to, so a second public copy earned nothing and cost a public token.
|
||||
@@ -0,0 +1,179 @@
|
||||
<!doctype html>
|
||||
<!--
|
||||
Ham Lab sensor API — external demo.
|
||||
|
||||
One file, no build, no framework, not part of Ham Lab: it is a stranger's page on another
|
||||
origin that knows only a base URL and a read-only token (config.js, beside this file):
|
||||
|
||||
window.HAMLAB = { base: 'http://10.0.0.234:8100', token: '<read-only sensor token>' }
|
||||
|
||||
Every tab is one shaped query — the object it wants, with sensor IDs where the values go —
|
||||
POSTed to /api/sensors/query?full=1 on an interval. Sparklines are seeded once from
|
||||
/api/sensors/history. The Fleet tab uses /api/fleet/sensors/query. See SENSORS.md.
|
||||
-->
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Ham Lab sensors — external demo</title>
|
||||
<script src="config.js"></script>
|
||||
<style>
|
||||
:root { --bg:#0f1216; --card:#171b21; --line:#2a313b; --fg:#d5dbe2; --mut:#7d8590; --acc:#e3a33b; --s1:#3987e5; --s2:#c98500; --bad:#ff7b72; --ok:#3fb950; --track:rgba(57,135,229,.18) }
|
||||
* { box-sizing: border-box }
|
||||
body { margin:0; background:var(--bg); color:var(--fg); font:14px/1.4 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif }
|
||||
header { display:flex; align-items:center; gap:10px; flex-wrap:wrap; padding:10px 14px; background:var(--card); border-bottom:1px solid var(--line) }
|
||||
header h1 { margin:0; font-size:15px; color:var(--acc); letter-spacing:.4px }
|
||||
header .sp { flex:1 }
|
||||
.mut { color:var(--mut) } .mono { font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace }
|
||||
select { background:var(--bg); color:var(--fg); border:1px solid var(--line); border-radius:5px; padding:4px 6px; font:inherit }
|
||||
.dot { width:8px; height:8px; border-radius:50%; background:var(--mut); display:inline-block } .dot.ok { background:var(--ok) } .dot.bad { background:var(--bad) }
|
||||
nav { display:flex; gap:2px; overflow-x:auto; padding:0 8px; background:var(--card); border-bottom:1px solid var(--line); scrollbar-width:none }
|
||||
nav button { flex:0 0 auto; background:none; border:0; border-bottom:2px solid transparent; color:var(--mut); padding:10px 12px; font:inherit; cursor:pointer }
|
||||
nav button.on { color:var(--fg); border-bottom-color:var(--acc) }
|
||||
main { padding:12px; max-width:1100px; margin:0 auto }
|
||||
.grid { display:grid; grid-template-columns:repeat(auto-fill,minmax(170px,1fr)); gap:10px }
|
||||
.tile { background:var(--card); border:1px solid var(--line); border-radius:8px; padding:12px; min-width:0; overflow:hidden }
|
||||
.tile.wide { grid-column:1/-1 }
|
||||
.label { font-size:11px; font-weight:600; color:var(--mut); text-transform:uppercase; letter-spacing:.5px; margin-bottom:6px; overflow-wrap:anywhere }
|
||||
.value { font-size:22px; font-weight:600; font-variant-numeric:tabular-nums } .value.warn { color:var(--s2) } .value.bad { color:var(--bad) }
|
||||
.id { margin-top:8px; font-size:10.5px; color:var(--mut); overflow-wrap:anywhere }
|
||||
.meter { height:6px; background:var(--track); border-radius:3px; overflow:hidden; margin-top:8px } .meter i { display:block; height:100%; background:var(--s1); border-radius:3px; transition:width .3s } .meter i.warn { background:var(--s2) } .meter i.bad { background:var(--bad) }
|
||||
svg.gauge { display:block; width:100%; max-width:180px; margin:0 auto } svg.gauge text { fill:var(--fg); font-size:13px; font-weight:600 }
|
||||
svg.spark { display:block; width:100%; height:44px; margin-top:6px }
|
||||
.rows { display:flex; flex-direction:column; gap:7px } .row { display:grid; grid-template-columns:1fr auto; gap:2px 10px; font-size:13px; align-items:baseline }
|
||||
.row .k { overflow:hidden; text-overflow:ellipsis; white-space:nowrap } .row .n { font-variant-numeric:tabular-nums } .row .meter { grid-column:1/3; margin-top:0 }
|
||||
.pill { font-size:11px; padding:1px 7px; border-radius:9px; border:1px solid var(--line) } .pill.ok { color:var(--ok); border-color:var(--ok) } .pill.bad { color:var(--bad); border-color:var(--bad) }
|
||||
details { margin-top:14px; background:var(--card); border:1px solid var(--line); border-radius:8px; padding:10px 12px } summary { cursor:pointer; color:var(--mut) }
|
||||
pre { margin:8px 0 0; font:12px/1.5 ui-monospace,Menlo,Consolas,monospace; white-space:pre-wrap; word-break:break-word; max-height:320px; overflow:auto }
|
||||
.err { color:var(--bad); padding:12px }
|
||||
table { width:100%; border-collapse:collapse; font-size:13px } th, td { text-align:left; padding:6px 8px; border-bottom:1px solid var(--line); font-variant-numeric:tabular-nums } th { color:var(--mut); font-weight:600; font-size:11px; text-transform:uppercase }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>Ham Lab sensors</h1><span class="mut">external demo · <span id="base" class="mono"></span></span>
|
||||
<span class="sp"></span>
|
||||
<span id="status" class="mut"></span><span id="dot" class="dot"></span>
|
||||
<label class="mut">every <select id="every"><option value="1000">1 s</option><option value="2000" selected>2 s</option><option value="5000">5 s</option><option value="10000">10 s</option><option value="30000">30 s</option></select></label>
|
||||
</header>
|
||||
<nav id="tabs"></nav>
|
||||
<main>
|
||||
<div id="view"></div>
|
||||
<details><summary>What this tab asked, and what came back</summary><pre id="wire"></pre></details>
|
||||
</main>
|
||||
|
||||
<script>
|
||||
const CFG = window.HAMLAB || {}
|
||||
const $ = (id) => document.getElementById(id)
|
||||
const h = (tag, attrs = {}, ...kids) => { const e = document.createElementNS(tag === 'svg' || attrs.svg ? 'http://www.w3.org/2000/svg' : 'http://www.w3.org/1999/xhtml', tag); for (const [k, v] of Object.entries(attrs)) if (k !== 'svg' && v != null) k === 'class' ? e.setAttribute('class', v) : k === 'style' ? e.setAttribute('style', v) : e.setAttribute(k, v); for (const kid of kids.flat()) if (kid != null) e.append(kid); return e }
|
||||
|
||||
// ---- the API: two calls ----
|
||||
async function api(path, body) {
|
||||
const r = await fetch(CFG.base + path, { method: body ? 'POST' : 'GET', headers: { Authorization: 'Bearer ' + CFG.token, ...(body ? { 'Content-Type': 'application/json' } : {}) }, body: body ? JSON.stringify(body) : undefined })
|
||||
if (!r.ok) throw new Error(r.status === 401 ? 'token rejected (401)' : `${r.status} ${r.statusText}`)
|
||||
return r.json()
|
||||
}
|
||||
|
||||
// ---- formatting from the sensor's own unit ----
|
||||
const U = ['B', 'KiB', 'MiB', 'GiB', 'TiB']
|
||||
const bytes = (v) => { let i = 0; while (v >= 1024 && i < U.length - 1) { v /= 1024; i++ } return `${v >= 100 || i === 0 ? v.toFixed(0) : v.toFixed(1)} ${U[i]}` }
|
||||
const dur = (s) => { const d = Math.floor(s / 86400), hh = Math.floor(s % 86400 / 3600), m = Math.floor(s % 3600 / 60); return d ? `${d}d ${hh}h` : hh ? `${hh}h ${m}m` : `${m}m` }
|
||||
function fmt(s) {
|
||||
if (!s || s.value == null) return '—'
|
||||
const v = s.value
|
||||
if (typeof v === 'boolean') return v ? 'yes' : 'no'
|
||||
if (typeof v === 'string') return v || '—'
|
||||
switch (s.unit) { case 'B': return bytes(v); case 'B/s': return bytes(v) + '/s'; case 's': return dur(v); case 'min': return dur(v * 60); case '%': return v.toFixed(v < 10 ? 1 : 0) + '%'; case '°C': return v.toFixed(0) + '°C'; case 'rpm': return Math.round(v) + ' rpm'; case undefined: case '': return String(Math.round(v * 100) / 100); default: return `${Math.round(v * 10) / 10} ${s.unit}` }
|
||||
}
|
||||
const frac = (s) => { if (!s || typeof s.value !== 'number') return null; const max = s.max ?? s.crit ?? s.warn, min = s.min ?? 0; return max === undefined || max <= min ? null : Math.min(1, Math.max(0, (s.value - min) / (max - min))) }
|
||||
const sev = (s) => !s || typeof s.value !== 'number' ? '' : s.crit !== undefined && s.value >= s.crit ? 'bad' : s.warn !== undefined && s.value >= s.warn ? 'warn' : s.kind === 'usage' && s.id !== 'ups.battery' && s.value >= 95 ? 'bad' : s.kind === 'usage' && s.id !== 'ups.battery' && s.value >= 85 ? 'warn' : ''
|
||||
|
||||
// ---- the widgets ----
|
||||
const tile = (label, id, ...body) => h('div', { class: 'tile' }, h('div', { class: 'label' }, label), ...body, id ? h('div', { class: 'id mono' }, id) : null)
|
||||
const missing = (label, id) => tile(label, id, h('div', { class: 'mut' }, 'not on this machine'))
|
||||
const W = {
|
||||
stat: (s, label, id) => s ? tile(label || s.name, s.id, h('div', { class: 'value ' + sev(s) }, fmt(s))) : missing(label, id),
|
||||
bar: (s, label, id) => { if (!s) return missing(label, id); const f = frac(s); return tile(label || s.name, s.id, h('div', { class: 'value ' + sev(s) }, fmt(s)), f === null ? null : h('div', { class: 'meter' }, h('i', { class: sev(s), style: `width:${f * 100}%` }))) },
|
||||
gauge: (s, label, id) => {
|
||||
if (!s) return missing(label, id)
|
||||
const f = frac(s) ?? 0, C = Math.PI * 40, col = { bad: 'var(--bad)', warn: 'var(--s2)' }[sev(s)] || 'var(--s1)', arc = 'M10 50 A40 40 0 0 1 90 50'
|
||||
return tile(label || s.name, s.id, h('svg', { class: 'gauge', viewBox: '0 0 100 56' },
|
||||
h('path', { svg: 1, d: arc, fill: 'none', stroke: 'var(--track)', 'stroke-width': 9, 'stroke-linecap': 'round' }),
|
||||
h('path', { svg: 1, d: arc, fill: 'none', stroke: col, 'stroke-width': 9, 'stroke-linecap': 'round', 'stroke-dasharray': `${f * C} ${C}` }),
|
||||
h('text', { svg: 1, x: 50, y: 47, 'text-anchor': 'middle' }, fmt(s))))
|
||||
},
|
||||
spark: (s, label, id) => {
|
||||
if (!s) return missing(label, id)
|
||||
const pts = (hist[s.id] ||= []); if (typeof s.value === 'number') { pts.push(s.value); if (pts.length > 300) pts.shift() }
|
||||
const max = s.unit === '%' ? 100 : Math.max(...pts, 1e-9) * 1.1, n = pts.length
|
||||
const line = n < 2 ? '' : pts.map((v, i) => `${(i / (n - 1) * 300).toFixed(1)},${(42 - v / max * 40).toFixed(1)}`).join(' ')
|
||||
return tile(label || s.name, s.id, h('div', { class: 'value' }, fmt(s)), h('svg', { class: 'spark', viewBox: '0 0 300 44', preserveAspectRatio: 'none' }, h('polyline', { svg: 1, points: line, fill: 'none', stroke: 'var(--s1)', 'stroke-width': 2, 'vector-effect': 'non-scaling-stroke' })))
|
||||
},
|
||||
// a glob comes back as {id: sensor}: a list draws them all
|
||||
list: (map, label, pattern, opts = {}) => {
|
||||
const rows = Object.values(map || {}).filter((s) => !opts.filter || opts.filter(s)).sort((a, b) => a.id.localeCompare(b.id))
|
||||
return h('div', { class: 'tile wide' }, h('div', { class: 'label' }, `${label} · ${rows.length}`),
|
||||
rows.length ? h('div', { class: 'rows' }, rows.map((s) => { const f = frac(s); return h('div', { class: 'row' }, h('span', { class: 'k', title: s.id }, opts.name ? opts.name(s) : s.name), typeof s.value === 'boolean' ? h('span', { class: 'pill ' + (s.value ? 'ok' : 'bad') }, s.value ? 'up' : 'down') : h('span', { class: 'n' }, fmt(s)), f === null ? null : h('div', { class: 'meter' }, h('i', { class: sev(s), style: `width:${f * 100}%` }))) })) : h('div', { class: 'mut' }, 'nothing here that this token may see'),
|
||||
h('div', { class: 'id mono' }, pattern))
|
||||
},
|
||||
}
|
||||
|
||||
// ---- the tabs: a query, the histories to seed, and how to draw the answer ----
|
||||
const TABS = [
|
||||
{ name: 'Overview', query: { cpu: 'cpu.usage', mem: 'mem.used_pct', memB: 'mem.used', root: 'fs.root.used_pct', ups: 'ups.battery', temp: 'cpu.temp', load: 'load.1', up: 'uptime', failed: 'services.failed', upd: 'updates.pending', cont: 'containers.running' },
|
||||
seed: 'cpu.usage,load.1',
|
||||
draw: (d) => [W.gauge(d.cpu, 'CPU'), W.gauge(d.mem, 'Memory'), W.gauge(d.root, 'Root disk'), W.gauge(d.ups, 'UPS battery', 'ups.battery'), W.spark(d.cpu, 'CPU, recent'), W.spark(d.load, 'Load 1 min'), W.bar(d.memB, 'Memory used'), W.stat(d.temp, 'CPU temp', 'cpu.temp'), W.stat(d.up, 'Uptime'), W.stat(d.cont, 'Containers up', 'containers.running'), W.stat(d.failed, 'Failed services'), W.stat(d.upd, 'Pending updates')] },
|
||||
{ name: 'Widget types', query: { s: 'cpu.usage', cores: 'cpu.core.*.usage' }, seed: 'cpu.usage',
|
||||
draw: (d) => [W.stat(d.s, 'stat — a number'), W.gauge(d.s, 'gauge — has a scale'), W.bar(d.s, 'bar — has a scale'), W.spark(d.s, 'spark — has a past'), W.list(d.cores, 'list — a pattern', 'cpu.core.*.usage')] },
|
||||
{ name: 'Hardware', query: { cores: 'cpu.core.*.usage', temps: 'temp.*', fans: 'fan.*', smart: 'smart.*.passed', wear: 'smart.*.wear' },
|
||||
draw: (d) => [W.list(d.cores, 'CPU cores', 'cpu.core.*.usage'), W.list(d.temps, 'Temperatures', 'temp.*'), W.list(d.fans, 'Fans', 'fan.*'), W.list(d.smart, 'Drive health', 'smart.*.passed', { name: (s) => s.labels?.device + ' · ' + (s.labels?.model || '') }), W.list(d.wear, 'SSD endurance used', 'smart.*.wear')] },
|
||||
{ name: 'Network & disks', query: { rx: 'net.*.rx', tx: 'net.*.tx', rd: 'disk.*.read', wr: 'disk.*.write', fs: 'fs.*.used_pct', fsb: 'fs.*.used' }, seed: 'net.*.rx,net.*.tx,disk.*.read,disk.*.write',
|
||||
draw: (d) => [...Object.values(d.rx).map((s) => W.spark(s)), ...Object.values(d.tx).map((s) => W.spark(s)), ...Object.values(d.rd).map((s) => W.spark(s)), ...Object.values(d.wr).map((s) => W.spark(s)), W.list(d.fs, 'Filesystems', 'fs.*.used_pct', { name: (s) => s.labels?.mount }), W.list(d.fsb, 'Filesystems, bytes', 'fs.*.used', { name: (s) => s.labels?.mount })] },
|
||||
{ name: 'Containers', query: { total: 'containers.total', run: 'containers.running', bad: 'containers.unhealthy', up: 'container.*.running', mem: 'container.*.mem', cpu: 'container.*.cpu' },
|
||||
draw: (d) => [W.stat(d.run, 'Running'), W.stat(d.total, 'Total'), W.stat(d.bad, 'Unhealthy'), W.list(d.up, 'Containers', 'container.*.running', { name: (s) => s.labels?.container }), W.list(d.mem, 'Memory', 'container.*.mem', { name: (s) => s.labels?.container }), W.list(d.cpu, 'CPU, % of one core', 'container.*.cpu', { name: (s) => s.labels?.container })] },
|
||||
{ name: 'Services', query: { failed: 'services.failed', names: 'services.failed_names', running: 'services.running', reboot: 'updates.reboot_required', each: 'service.*.active' },
|
||||
draw: (d) => [W.stat(d.failed, 'Failed'), W.stat(d.running, 'Running'), W.stat(d.names, 'Which failed'), W.stat(d.reboot, 'Reboot required'), W.list(d.each, 'Every service', 'service.*.active')] },
|
||||
{ name: 'Fleet', fleet: true, query: { cpu: 'cpu.usage', mem: 'mem.used_pct', temp: 'cpu.temp', up: 'uptime', failed: 'services.failed', cont: 'containers.running', pool: 'zfs.*.health' },
|
||||
draw: (d) => [h('div', { class: 'tile wide' }, h('div', { class: 'label' }, 'one request, every machine'), h('table', {}, h('tr', {}, ['Machine', 'CPU', 'Memory', 'CPU temp', 'Uptime', 'Failed', 'Containers', 'Pools'].map((t) => h('th', {}, t))),
|
||||
Object.entries(d).sort().map(([m, r]) => r.error ? h('tr', {}, h('td', {}, m), h('td', { colspan: 7, class: 'mut' }, r.error)) : h('tr', {}, h('td', {}, m), ...[r.cpu, r.mem, r.temp, r.up, r.failed, r.cont].map((s) => h('td', {}, fmt(s))), h('td', {}, Object.values(r.pool || {}).map((s) => `${s.labels?.pool} ${s.value}`).join(', ') || '—')))), h('div', { class: 'id mono' }, 'POST /api/fleet/sensors/query'))] },
|
||||
]
|
||||
|
||||
// ---- the loop ----
|
||||
const hist = {}
|
||||
let tab = 0, timer = 0, seq = 0
|
||||
async function show(i) {
|
||||
tab = i; const my = ++seq; clearTimeout(timer)
|
||||
;[...$('tabs').children].forEach((b, j) => b.classList.toggle('on', j === i))
|
||||
$('view').replaceChildren(h('div', { class: 'mut' }, 'asking…'))
|
||||
const t = TABS[i]
|
||||
if (t.seed) try { for (const s of (await api('/api/sensors/history?ids=' + encodeURIComponent(t.seed))).series) hist[s.id] = s.v.slice(-300) } catch { /* sparklines just start empty */ }
|
||||
if (my === seq) tick(my)
|
||||
}
|
||||
async function tick(my) {
|
||||
if (my !== seq) return
|
||||
const t = TABS[tab], t0 = performance.now()
|
||||
try {
|
||||
const path = (t.fleet ? '/api/fleet/sensors/query' : '/api/sensors/query') + '?full=1', body = t.fleet ? { machines: '*', query: t.query } : t.query
|
||||
const data = await api(path, body)
|
||||
if (my !== seq) return
|
||||
$('view').replaceChildren(h('div', { class: 'grid' }, t.draw(data)))
|
||||
$('wire').textContent = `POST ${path}\n${JSON.stringify(body, null, 1)}\n\n→ ${JSON.stringify(data, (k, v) => (v && typeof v === 'object' && 'id' in v && 'value' in v ? { id: v.id, value: v.value, unit: v.unit } : v), 1).slice(0, 6000)}`
|
||||
$('dot').className = 'dot ok'; $('status').textContent = `${Math.round(performance.now() - t0)} ms · ${new Date().toLocaleTimeString()}`
|
||||
} catch (e) {
|
||||
if (my !== seq) return
|
||||
$('dot').className = 'dot bad'; $('status').textContent = e.message
|
||||
if (!$('view').querySelector('.grid')) $('view').replaceChildren(h('div', { class: 'err' }, e.message))
|
||||
}
|
||||
timer = setTimeout(() => tick(my), document.hidden ? 5000 : Number($('every').value))
|
||||
}
|
||||
|
||||
if (!CFG.base || !CFG.token) $('view').replaceChildren(h('div', { class: 'err' }, 'config.js is missing: window.HAMLAB = { base, token }'))
|
||||
else {
|
||||
$('base').textContent = CFG.base
|
||||
TABS.forEach((t, i) => $('tabs').append(h('button', {}, t.name))); [...$('tabs').children].forEach((b, i) => b.onclick = () => show(i))
|
||||
$('every').onchange = () => { clearTimeout(timer); tick(seq) }
|
||||
show(0)
|
||||
}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -5,6 +5,7 @@ go 1.26.0
|
||||
require (
|
||||
github.com/coder/websocket v1.8.15
|
||||
github.com/creack/pty v1.1.24
|
||||
go.etcd.io/bbolt v1.5.0
|
||||
golang.org/x/crypto v0.57.0
|
||||
golang.org/x/term v0.46.0
|
||||
)
|
||||
|
||||
@@ -2,9 +2,21 @@ github.com/coder/websocket v1.8.15 h1:6B2JPeOGlpff2Uz6vOEH1Vzpi0iUz20A+lPVhPHtNU
|
||||
github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
|
||||
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
|
||||
github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
|
||||
go.etcd.io/bbolt v1.5.0/go.mod h1:mkltfYE5aUHQxUct9N9V+Kp7aSjFqjgrhcXIS70Lrdk=
|
||||
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
|
||||
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
|
||||
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
|
||||
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
|
||||
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
|
||||
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
|
||||
Executable
+382
@@ -0,0 +1,382 @@
|
||||
#!/bin/sh
|
||||
# Ham Lab installer.
|
||||
#
|
||||
# curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh
|
||||
#
|
||||
# With a terminal it walks you through it: who the console runs as, where it listens,
|
||||
# HTTPS, the firewall, the password — then downloads the build for this machine, checks
|
||||
# it against SHA256SUMS and hands over to `hamlab install`, which sets up the binary,
|
||||
# the config and a service that starts at boot. Re-running it upgrades in place.
|
||||
#
|
||||
# Without a terminal, with --yes, or with flags after `--`, it asks nothing:
|
||||
#
|
||||
# curl -fsSL …/install.sh | sh -s -- --yes
|
||||
# curl -fsSL …/install.sh | sh -s -- --listen 0.0.0.0:8100 --tls auto --password-stdin <pw
|
||||
# ./install.sh # beside the hamlab-linux-* files: installs offline
|
||||
#
|
||||
# HAMLAB_URL where hamlab-linux-<arch>, SHA256SUMS and VERSION are served from —
|
||||
# by default the newest release on Gitea; "latest" is an alias Gitea
|
||||
# resolves to the most recent release, so the URL never needs changing —
|
||||
# any static file host, or a Gitea/Forgejo release's download URL
|
||||
# HAMLAB_TOKEN access token, when that place is a private Gitea/Forgejo repo
|
||||
# NO_COLOR plain output
|
||||
#
|
||||
# POSIX sh: runs under dash, busybox ash and bash alike.
|
||||
set -eu
|
||||
|
||||
HAMLAB_URL="${HAMLAB_URL:-https://git.bigham.dev/mitchell/hamlab/releases/download/latest}"
|
||||
BUILT_FOR="@VERSION@" # filled in by `make release`
|
||||
|
||||
# ─────────────────────────────────────────────────────────────── looks ──
|
||||
if [ -t 2 ] && [ -z "${NO_COLOR:-}" ] && [ "${TERM:-dumb}" != dumb ]; then
|
||||
B=$(printf '\033[1m'); D=$(printf '\033[2m'); R=$(printf '\033[0m')
|
||||
AMBER=$(printf '\033[38;5;214m'); GREEN=$(printf '\033[32m'); RED=$(printf '\033[31m'); CYAN=$(printf '\033[36m')
|
||||
else
|
||||
B=; D=; R=; AMBER=; GREEN=; RED=; CYAN=
|
||||
fi
|
||||
case "${LC_ALL:-${LC_CTYPE:-${LANG:-}}}" in
|
||||
*UTF-8*|*utf8*|*UTF8*|*utf-8*) OK='✓'; NO='✗'; DOT='•'; ARROW='›'; HR='────────────────────────────────────────────────────────────' ;;
|
||||
*) OK='ok'; NO='x'; DOT='*'; ARROW='>'; HR='------------------------------------------------------------' ;;
|
||||
esac
|
||||
|
||||
out() { printf '%s\n' "$*" >&2; }
|
||||
blank() { printf '\n' >&2; }
|
||||
good() { out " ${GREEN}${OK}${R} $*"; }
|
||||
note() { out " ${D}$*${R}"; }
|
||||
warn() { out " ${AMBER}!${R} $*"; }
|
||||
die() { blank; out " ${RED}${NO}${R} ${B}$1${R}"; shift; for more in "$@"; do out " $more"; done; blank; exit 1; }
|
||||
rule() { out "${D}${HR}${R}"; }
|
||||
step() { blank; out "${AMBER}${B}$1${R} ${B}$2${R}"; [ -z "${3:-}" ] || note "$3"; blank; }
|
||||
|
||||
banner() {
|
||||
blank
|
||||
out " ${AMBER}${B}Ham Lab${R} ${D}a web console for the machines you run${R}"
|
||||
out " ${D}terminals ${DOT} monitors ${DOT} services ${DOT} files ${DOT} updates ${DOT} disks — in one tab${R}"
|
||||
rule
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────── questions ──
|
||||
# Piped from curl, stdin is this script, so every answer comes from the terminal itself.
|
||||
TTY=/dev/tty
|
||||
have_tty() { [ -r "$TTY" ] && [ -w "$TTY" ] && (: <"$TTY") 2>/dev/null; }
|
||||
|
||||
ask() { # ask "Question" "default" → $REPLY
|
||||
if [ -n "${2:-}" ]; then printf ' %s %s[%s]%s ' "$1" "$D" "$2" "$R" >&2; else printf ' %s ' "$1" >&2; fi
|
||||
IFS= read -r REPLY <"$TTY" || REPLY=
|
||||
[ -n "$REPLY" ] || REPLY="${2:-}"
|
||||
}
|
||||
confirm() { # confirm "Question" y|n
|
||||
if [ "$2" = y ]; then hint="Y/n"; else hint="y/N"; fi
|
||||
while :; do
|
||||
ask "$1" "$hint"
|
||||
case "$REPLY" in
|
||||
"$hint") [ "$2" = y ]; return ;;
|
||||
y|Y|yes|Yes|YES) return 0 ;;
|
||||
n|N|no|No|NO) return 1 ;;
|
||||
*) warn "y or n, please" ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
secret() { # secret "Prompt" → $REPLY, typed unseen
|
||||
printf ' %s ' "$1" >&2
|
||||
saved=$(stty -g <"$TTY" 2>/dev/null || true)
|
||||
stty -echo <"$TTY" 2>/dev/null || true
|
||||
IFS= read -r REPLY <"$TTY" || REPLY=
|
||||
[ -z "$saved" ] || stty "$saved" <"$TTY" 2>/dev/null || true
|
||||
printf '\n' >&2
|
||||
}
|
||||
choose() { # choose "default-number" "label 1" "label 2" … → $REPLY = the number picked
|
||||
def=$1; shift; n=0
|
||||
for label in "$@"; do n=$((n + 1)); if [ "$n" = "$def" ]; then out " ${AMBER}${B}$n${R} $label"; else out " ${B}$n${R} $label"; fi; done
|
||||
blank
|
||||
while :; do
|
||||
ask "Choose 1-$n" "$def"
|
||||
case "$REPLY" in *[!0-9]*|'') ;; *) if [ "$REPLY" -ge 1 ] && [ "$REPLY" -le "$n" ]; then return; fi ;; esac
|
||||
warn "a number from 1 to $n, please"
|
||||
done
|
||||
}
|
||||
restore_tty() { [ -z "${saved:-}" ] || stty "$saved" <"$TTY" 2>/dev/null || true; }
|
||||
|
||||
# ──────────────────────────────────────────────────────────── plumbing ──
|
||||
fetch() { # url dest
|
||||
if command -v curl >/dev/null 2>&1; then
|
||||
curl -fsSL ${HAMLAB_TOKEN:+-H "Authorization: token $HAMLAB_TOKEN"} -o "$2" "$1"
|
||||
elif command -v wget >/dev/null 2>&1; then
|
||||
wget -q ${HAMLAB_TOKEN:+--header="Authorization: token $HAMLAB_TOKEN"} -O "$2" "$1"
|
||||
else
|
||||
die "This needs curl or wget, and the machine has neither."
|
||||
fi
|
||||
}
|
||||
port_busy() { # is something already listening on TCP port $1?
|
||||
hex=$(printf '%04X' "$1")
|
||||
cat /proc/net/tcp /proc/net/tcp6 2>/dev/null | awk -v p=":$hex" '$4 == "0A" && substr($2, length($2) - 4) == p { found = 1 } END { exit !found }'
|
||||
}
|
||||
lan_addresses() { # this machine's IPv4 addresses, one per line, most useful first
|
||||
if command -v ip >/dev/null 2>&1; then
|
||||
ip -4 -o addr show scope global 2>/dev/null | awk '$2 !~ /^(docker|br-|veth|virbr|cni|flannel|tailscale|wg|tun|tap)/ { split($4, a, "/"); print a[1] " " $2 }'
|
||||
else
|
||||
hostname -I 2>/dev/null | tr ' ' '\n' | awk 'NF && $1 !~ /:/ { print $1 " -" }'
|
||||
fi
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────── arguments ──
|
||||
YES=0; PASS=
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-y|--yes) YES=1; shift ;;
|
||||
-h|--help) sed -n '2,20p' "$0" 2>/dev/null | sed 's/^# \{0,1\}//' >&2; exit 0 ;;
|
||||
--) shift; PASS="$*"; break ;;
|
||||
*) PASS="$*"; break ;; # flags given straight: they belong to `hamlab install`
|
||||
esac
|
||||
done
|
||||
INTERACTIVE=1
|
||||
{ [ "$YES" = 1 ] || [ -n "$PASS" ] || ! have_tty; } && INTERACTIVE=0
|
||||
|
||||
# ─────────────────────────────────────────────────────────── preflight ──
|
||||
banner
|
||||
[ "$(uname -s)" = Linux ] || die "Ham Lab runs on Linux; this is $(uname -s)."
|
||||
case "$(uname -m)" in
|
||||
x86_64|amd64) arch=amd64 ;;
|
||||
aarch64|arm64) arch=arm64 ;;
|
||||
armv6l|armv7l|armv8l|arm) arch=arm ;;
|
||||
i386|i486|i586|i686) arch=386 ;;
|
||||
riscv64) arch=riscv64 ;;
|
||||
*) die "There is no build for $(uname -m). (There are: amd64, arm64, arm, 386, riscv64.)" ;;
|
||||
esac
|
||||
file="hamlab-linux-$arch"
|
||||
|
||||
me=$(id -un 2>/dev/null || echo root)
|
||||
if [ "$(id -u)" = 0 ]; then how=root
|
||||
elif command -v sudo >/dev/null 2>&1; then how=sudo
|
||||
elif command -v doas >/dev/null 2>&1; then how=doas
|
||||
else die "Installing a service needs root, and there is no sudo or doas here. Run this as root."
|
||||
fi
|
||||
if [ -d /run/systemd/system ]; then init=systemd
|
||||
elif command -v rc-service >/dev/null 2>&1; then init=OpenRC
|
||||
else init=none
|
||||
fi
|
||||
os=$(. /etc/os-release 2>/dev/null && printf '%s' "${PRETTY_NAME:-}"); [ -n "$os" ] || os=Linux
|
||||
existing=$(command -v hamlab 2>/dev/null || true); [ -n "$existing" ] || { [ -x /usr/local/bin/hamlab ] && existing=/usr/local/bin/hamlab; } || true
|
||||
have=; [ -z "$existing" ] || have=$("$existing" version 2>/dev/null | awk '{ print $2 }') || true
|
||||
|
||||
good "$os ${D}${DOT}${R} $(uname -m) ${D}${DOT}${R} $init"
|
||||
case "$how" in root) good "running as root" ;; *) good "running as ${B}$me${R}; $how is asked for root when it is needed" ;; esac
|
||||
[ "$init" != none ] || warn "no systemd or OpenRC: it will be installed, but nothing starts it at boot"
|
||||
|
||||
# ───────────────────────────────────────────────────── which version ──
|
||||
tmp=$(mktemp -d 2>/dev/null || mktemp -d -t hamlab)
|
||||
alt=
|
||||
cleanup() { restore_tty; rm -rf "$tmp"; [ -z "$alt" ] || rm -f "$alt"; }
|
||||
trap cleanup EXIT
|
||||
trap 'blank; out " cancelled — nothing was changed"; exit 130' INT TERM
|
||||
|
||||
here=$(cd "$(dirname "$0")" 2>/dev/null && pwd || echo .)
|
||||
offline=0; [ -f "$here/$file" ] && offline=1
|
||||
if [ "$offline" = 1 ]; then
|
||||
ver=$BUILT_FOR; case "$ver" in @*) ver="(local files)" ;; esac
|
||||
else
|
||||
# What is published right now — which may be newer than the copy of this script being run.
|
||||
ver=$(fetch "$HAMLAB_URL/VERSION" "$tmp/VERSION" 2>/dev/null && head -n 1 "$tmp/VERSION" | tr -cd 'A-Za-z0-9._+-' || true)
|
||||
[ -n "$ver" ] || { ver=$BUILT_FOR; case "$ver" in @*) ver=latest ;; esac; }
|
||||
fi
|
||||
# v0.1.0-69-g9c4f9df → 0001000069: versions compare as numbers, builds since the tag included.
|
||||
vnum() { printf '%s' "$1" | sed 's/^v//; s/-g[0-9a-f]*$//' | awk -F'[.-]' '{ printf "%d%03d%03d%05d", $1, $2, $3, $4 }'; }
|
||||
older=0
|
||||
if [ -n "$have" ] && [ "$have" != "$ver" ]; then
|
||||
a=$(vnum "$have" 2>/dev/null || echo 0); b=$(vnum "$ver" 2>/dev/null || echo 0)
|
||||
case "$a$b" in *[!0-9]*) ;; *) [ "$b" -ge "$a" ] || older=1 ;; esac
|
||||
fi
|
||||
if [ "$older" = 1 ]; then
|
||||
warn "Ham Lab ${B}$have${R} is installed, and what is published is ${B}older${R}: $ver"
|
||||
note " installing it would take this machine backwards"
|
||||
if [ "$INTERACTIVE" = 1 ]; then
|
||||
blank; confirm "Go back to $ver anyway?" n || { blank; out " nothing was changed"; exit 0; }
|
||||
elif [ -z "${HAMLAB_ALLOW_DOWNGRADE:-}" ]; then
|
||||
die "Not downgrading $have to $ver." "Set HAMLAB_ALLOW_DOWNGRADE=1 if that is really what you want."
|
||||
fi
|
||||
have_msg_done=1
|
||||
fi
|
||||
if [ -n "${have_msg_done:-}" ]; then :
|
||||
elif [ -n "$have" ] && [ "$have" = "$ver" ]; then good "Ham Lab ${B}$have${R} is already installed ${D}${ARROW}${R} this reinstalls it"; note " its config, user and password are kept"
|
||||
elif [ -n "$have" ]; then good "Ham Lab ${B}$have${R} is installed ${D}${ARROW}${R} upgrading to ${B}$ver${R}"; note " its config, user and password are kept"
|
||||
else good "installing Ham Lab ${B}$ver${R}"; fi
|
||||
|
||||
# ─────────────────────────────────────────────────────────── the wizard ──
|
||||
flags=; password=; listen=; url_host=
|
||||
if [ "$INTERACTIVE" = 1 ] && [ -n "$have" ] && [ "$older" = 1 ]; then :
|
||||
elif [ "$INTERACTIVE" = 1 ] && [ -n "$have" ]; then
|
||||
blank
|
||||
confirm "$([ "$have" = "$ver" ] && echo Reinstall || echo Upgrade) now?" y || { blank; out " nothing was changed"; exit 0; }
|
||||
elif [ "$INTERACTIVE" = 1 ]; then
|
||||
# 1 ─ who
|
||||
default_user=${SUDO_USER:-${DOAS_USER:-$me}}
|
||||
step "1/5" "Who should it run as?" "Its terminals are that user's shell; its files are that user's files."
|
||||
while :; do
|
||||
ask "User" "$default_user"; run_as=$REPLY
|
||||
id "$run_as" >/dev/null 2>&1 && break
|
||||
warn "there is no user called \"$run_as\" on this machine"
|
||||
done
|
||||
[ "$run_as" != root ] || warn "as root, every terminal it opens is a root shell"
|
||||
|
||||
# 2 ─ where
|
||||
step "2/5" "Where should it listen?" "Who can reach the console at all. The password guards it either way."
|
||||
addrs=$(lan_addresses || true); first=$(printf '%s\n' "$addrs" | awk 'NR == 1 { print $1 }')
|
||||
if [ -n "$first" ]; then
|
||||
first_if=$(printf '%s\n' "$addrs" | awk 'NR == 1 { print $2 }')
|
||||
choose 2 "This machine only ${D}127.0.0.1 — for an SSH tunnel or a reverse proxy${R}" \
|
||||
"This network ${D}$first ($first_if) — reachable from the LAN${R}" \
|
||||
"Every interface ${D}0.0.0.0 — every address it has or gets${R}" \
|
||||
"Another address"
|
||||
else
|
||||
choose 1 "This machine only ${D}127.0.0.1${R}" "Every interface ${D}0.0.0.0${R}" "Another address"
|
||||
[ "$REPLY" = 1 ] || REPLY=$((REPLY + 1))
|
||||
fi
|
||||
case "$REPLY" in
|
||||
1) host=127.0.0.1 ;;
|
||||
2) host=$first ;;
|
||||
3) host=0.0.0.0 ;;
|
||||
4) others=$(printf '%s\n' "$addrs" | awk 'NR > 1 { printf "%s%s (%s)", sep, $1, $2; sep = ", " }'); [ -z "$others" ] || note "also on this machine: $others"
|
||||
while :; do ask "Address" ""; host=$REPLY; case "$host" in ''|*[!0-9a-fA-F.:]*) warn "an IPv4 or IPv6 address, please" ;; *) break ;; esac; done ;;
|
||||
esac
|
||||
while :; do
|
||||
ask "Port" "8100"; port=$REPLY
|
||||
case "$port" in ''|*[!0-9]*) warn "a number, please"; continue ;; esac
|
||||
[ "$port" -ge 1 ] && [ "$port" -le 65535 ] || { warn "a port is between 1 and 65535"; continue; }
|
||||
[ "$port" -ge 1024 ] || [ "$run_as" = root ] || { warn "ports under 1024 need root, and it will run as $run_as"; continue; }
|
||||
if port_busy "$port"; then warn "something is already listening on port $port"; continue; fi
|
||||
break
|
||||
done
|
||||
case "$host" in *:*) listen="[$host]:$port" ;; *) listen="$host:$port" ;; esac
|
||||
|
||||
# 3 ─ https
|
||||
step "3/5" "HTTPS?" "With a certificate made here: browsers warn once, a hub pins it."
|
||||
if [ "$host" = 127.0.0.1 ]; then
|
||||
choose 2 "HTTPS, self-signed" "Plain HTTP ${D}— fine here: nothing leaves this machine${R}"
|
||||
else
|
||||
choose 1 "HTTPS, self-signed ${D}— recommended: the password crosses the network${R}" "Plain HTTP ${D}— only behind a reverse proxy that adds HTTPS${R}"
|
||||
fi
|
||||
if [ "$REPLY" = 1 ]; then tls=auto; scheme=https; else tls=off; scheme=http; fi
|
||||
|
||||
# 4 ─ firewall
|
||||
open_fw=0; fw=
|
||||
if [ "$host" != 127.0.0.1 ]; then
|
||||
if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then fw=firewalld
|
||||
elif command -v ufw >/dev/null 2>&1 && ufw status 2>/dev/null | grep -q "^Status: active"; then fw=ufw
|
||||
elif [ "$how" != root ] && command -v ufw >/dev/null 2>&1 && systemctl is-active -q ufw 2>/dev/null; then fw=ufw
|
||||
fi
|
||||
fi
|
||||
step "4/5" "Firewall"
|
||||
if [ -n "$fw" ]; then
|
||||
note "$fw is active and will block port $port until it is opened."
|
||||
blank
|
||||
if confirm "Open $port/tcp in $fw?" y; then open_fw=1; fi
|
||||
elif [ "$host" = 127.0.0.1 ]; then good "nothing to open: it only listens on this machine"
|
||||
else good "no active firewalld or ufw found; nothing to open"
|
||||
fi
|
||||
|
||||
# 5 ─ password
|
||||
step "5/5" "The password" "One password opens the console, and the console is a shell here."
|
||||
choose 1 "Choose one now" "Generate a strong one ${D}— shown once at the end${R}"
|
||||
if [ "$REPLY" = 1 ]; then
|
||||
while :; do
|
||||
secret "Password:"; password=$REPLY
|
||||
[ ${#password} -ge 12 ] || { warn "at least 12 characters, please (it guards a shell)"; continue; }
|
||||
secret "Once more:"
|
||||
[ "$REPLY" = "$password" ] && break
|
||||
warn "those didn't match; try again"
|
||||
done
|
||||
fi
|
||||
|
||||
# summary
|
||||
if [ "$host" = 0.0.0.0 ] || [ "$host" = "::" ]; then url_host=${first:-this-machine}; else url_host=$host; fi
|
||||
blank; rule
|
||||
out " ${B}Ready to install${R}"
|
||||
blank
|
||||
out " version ${B}$ver${R} ${D}($file)${R}"
|
||||
out " runs as ${B}$run_as${R}"
|
||||
out " address ${B}$scheme://$listen${R}"
|
||||
if [ "$open_fw" = 1 ]; then out " firewall open $port/tcp in $fw"; fi
|
||||
if [ -n "$password" ]; then out " password the one you typed"; else out " password generated, shown once"; fi
|
||||
case "$init" in none) out " service ${AMBER}none — no systemd or OpenRC${R}" ;; *) out " service $init, starts at boot" ;; esac
|
||||
rule; blank
|
||||
confirm "Install?" y || { blank; out " nothing was changed"; exit 0; }
|
||||
|
||||
flags="--user $run_as --listen $listen --tls $tls"
|
||||
[ "$open_fw" = 0 ] || flags="$flags --open-firewall"
|
||||
[ -z "$password" ] || flags="$flags --password-stdin"
|
||||
fi
|
||||
|
||||
# ──────────────────────────────────────────────── download and verify ──
|
||||
blank
|
||||
if [ "$offline" = 1 ]; then
|
||||
cp "$here/$file" "$tmp/hamlab"; [ ! -f "$here/SHA256SUMS" ] || cp "$here/SHA256SUMS" "$tmp/SHA256SUMS"
|
||||
good "using $file from $here"
|
||||
else
|
||||
printf ' %s downloading %s …' "$DOT" "$file" >&2
|
||||
fetch "$HAMLAB_URL/$file" "$tmp/hamlab" || { printf '\n' >&2; die "Download failed: $HAMLAB_URL/$file"; }
|
||||
size=$(wc -c <"$tmp/hamlab" | awk '{ printf "%.1f MB", $1 / 1048576 }')
|
||||
if [ -t 2 ]; then printf '\r\033[K' >&2; else printf '\n' >&2; fi
|
||||
good "downloaded $file ${D}($size)${R}"
|
||||
fetch "$HAMLAB_URL/SHA256SUMS" "$tmp/SHA256SUMS" 2>/dev/null || rm -f "$tmp/SHA256SUMS"
|
||||
fi
|
||||
if [ -f "$tmp/SHA256SUMS" ] && command -v sha256sum >/dev/null 2>&1; then
|
||||
want=$(awk -v f="$file" '$2 == f || $2 == "*" f { print $1 }' "$tmp/SHA256SUMS")
|
||||
got=$(sha256sum "$tmp/hamlab" | awk '{ print $1 }')
|
||||
[ -n "$want" ] || die "SHA256SUMS has no entry for $file."
|
||||
[ "$want" = "$got" ] || die "Checksum mismatch for $file." "This is not the file that was published. Nothing was installed."
|
||||
good "checksum matches SHA256SUMS"
|
||||
else
|
||||
warn "no SHA256SUMS (or no sha256sum here): the download was not verified"
|
||||
fi
|
||||
|
||||
chmod 0755 "$tmp/hamlab"; bin="$tmp/hamlab"
|
||||
if ! "$bin" version >/dev/null 2>&1; then
|
||||
alt="${HOME:-/root}/.hamlab-install.$$" # a noexec /tmp is the usual reason: try from the home directory
|
||||
cp "$bin" "$alt" 2>/dev/null && chmod 0755 "$alt" || true
|
||||
"$alt" version >/dev/null 2>&1 || die "The $arch build doesn't run on this machine."
|
||||
bin="$alt"
|
||||
fi
|
||||
|
||||
# ───────────────────────────────────────────────────────────── install ──
|
||||
blank; out " ${B}Installing${R}"; blank
|
||||
log="$tmp/install.log"
|
||||
set +e
|
||||
# shellcheck disable=SC2086 # $flags and $PASS are lists of words on purpose
|
||||
if [ -n "$password" ]; then
|
||||
printf '%s\n' "$password" | "$bin" install $flags >"$log" 2>&1; rc=$?
|
||||
password=
|
||||
elif [ "$INTERACTIVE" = 0 ] && [ -n "$PASS" ]; then
|
||||
log=
|
||||
case " $PASS " in
|
||||
*password-stdin*) "$bin" install $PASS; rc=$? ;;
|
||||
*) if [ ! -t 0 ] && have_tty; then "$bin" install $PASS <"$TTY"; rc=$?; else "$bin" install $PASS; rc=$?; fi ;;
|
||||
esac
|
||||
else
|
||||
"$bin" install $flags </dev/null >"$log" 2>&1; rc=$? # no typed password and no terminal on stdin: it generates one and prints it
|
||||
fi
|
||||
set -e
|
||||
[ -z "$log" ] || sed 's/^/ /' "$log" >&2
|
||||
hamlab_bin=$(command -v hamlab 2>/dev/null || echo /usr/local/bin/hamlab)
|
||||
[ "$rc" = 0 ] && [ -x "$hamlab_bin" ] || die "The install did not finish. Nothing above this line is hidden: the reason is in it."
|
||||
|
||||
# ──────────────────────────────────────────────────────────────── done ──
|
||||
blank; rule
|
||||
out " ${GREEN}${B}${OK} Ham Lab $("$hamlab_bin" version 2>/dev/null | awk '{ print $2 }') is installed${R}"
|
||||
blank
|
||||
row() { out " $(printf '%-10s' "$1") $2"; [ -z "${3:-}" ] || out " ${D}$3${R}"; }
|
||||
if [ "$init" = none ]; then
|
||||
row "start it" "${B}hamlab serve${R}" "as ${run_as:-its user}, under your own supervisor: nothing starts it at boot"
|
||||
fi
|
||||
if [ -n "$listen" ]; then
|
||||
row "open" "${CYAN}${B}$scheme://$(printf '%s' "$listen" | sed "s/^0\.0\.0\.0/$url_host/")${R}"
|
||||
[ "$tls" != auto ] || out " ${D}the browser warns about the self-signed certificate once${R}"
|
||||
[ "$host" != 127.0.0.1 ] || out " ${D}from elsewhere: ssh -L $port:127.0.0.1:$port $run_as@this-machine${R}"
|
||||
fi
|
||||
case "$init" in
|
||||
systemd) row "logs" "${B}journalctl -u hamlab -f${R}" ;;
|
||||
OpenRC) row "logs" "${B}tail -f /var/log/hamlab.log${R}" ;;
|
||||
esac
|
||||
row "change" "${B}hamlab install --help${R}" "a new address, HTTPS or user; everything else is kept"
|
||||
row "add to hub" "Machines ${ARROW} Add a machine" "on your main Ham Lab, with this address and password"
|
||||
row "remove" "${B}hamlab uninstall${R}" "--purge deletes its config too"
|
||||
rule; blank
|
||||
@@ -0,0 +1,528 @@
|
||||
package api
|
||||
|
||||
// Alerts: rules over sensors, watched by the hub for itself and every machine it manages.
|
||||
//
|
||||
// A rule is a selector, a comparison and a patience: `container.*.running == false for
|
||||
// 120 s`. Every alertEvery the hub asks each machine one shaped query for all the rules'
|
||||
// selectors, compares, and keeps a little state per (rule, machine, sensor): ok, pending
|
||||
// since, or firing since. Going to firing, and coming back, is an event — kept, shown on
|
||||
// the Alerts tab, and sent to whatever channels are set (ntfy, or a webhook: Home
|
||||
// Assistant, Slack, anything that takes a POST). `hamlab.reachable` is a sensor the hub
|
||||
// supplies itself, so "that machine stopped answering" is a rule like any other.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/sensors"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const (
|
||||
alertEvery = 15 * time.Second
|
||||
maxEvents = 200
|
||||
)
|
||||
|
||||
type AlertRule struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Sensor string `json:"sensor"` // an ID or a pattern
|
||||
Op string `json:"op"` // > >= < <= == !=
|
||||
Value any `json:"value"` // number, bool or string, to match the sensor
|
||||
ForSec int `json:"for_sec"`
|
||||
Severity string `json:"severity"` // info | warn | crit
|
||||
Enabled bool `json:"enabled"`
|
||||
Machines string `json:"machines"` // "*" or comma-separated names; "" is "*"
|
||||
}
|
||||
|
||||
type AlertChannel struct {
|
||||
Type string `json:"type"` // ntfy | webhook
|
||||
URL string `json:"url"`
|
||||
}
|
||||
|
||||
type AlertEvent struct {
|
||||
T int64 `json:"t"`
|
||||
Kind string `json:"kind"` // firing | resolved
|
||||
Rule string `json:"rule"`
|
||||
RuleID string `json:"rule_id"`
|
||||
Machine string `json:"machine"`
|
||||
Sensor string `json:"sensor"`
|
||||
Value any `json:"value"`
|
||||
Severity string `json:"severity"`
|
||||
}
|
||||
|
||||
type alertFiring struct {
|
||||
AlertEvent
|
||||
Since int64 `json:"since"`
|
||||
}
|
||||
|
||||
type alertSaved struct {
|
||||
Rules []AlertRule `json:"rules"`
|
||||
Channels []AlertChannel `json:"channels"`
|
||||
Events []AlertEvent `json:"events"`
|
||||
Seeded bool `json:"seeded"`
|
||||
// Firing is what was firing when the hub last looked, so a restart (every deploy is one)
|
||||
// carries on where it was and doesn't announce the same alerts all over again.
|
||||
Firing map[string]alertKept `json:"firing,omitempty"`
|
||||
}
|
||||
|
||||
type alertKept struct {
|
||||
Since int64 `json:"since"`
|
||||
Value any `json:"value"`
|
||||
}
|
||||
|
||||
const maxRules = 200
|
||||
|
||||
type alertState struct {
|
||||
pending time.Time
|
||||
firing bool
|
||||
since time.Time
|
||||
value any
|
||||
}
|
||||
|
||||
type alertEngine struct {
|
||||
s *Server
|
||||
mu sync.Mutex
|
||||
saved alertSaved
|
||||
state map[string]*alertState // ruleID|machine|sensor
|
||||
last time.Time
|
||||
}
|
||||
|
||||
var defaultRules = []AlertRule{
|
||||
{ID: "machine-offline", Name: "Machine not answering", Sensor: "hamlab.reachable", Op: "==", Value: false, ForSec: 120, Severity: "crit", Enabled: true},
|
||||
{ID: "container-down", Name: "Container down", Sensor: "container.*.down", Op: "==", Value: true, ForSec: 120, Severity: "crit", Enabled: true},
|
||||
{ID: "container-unhealthy", Name: "Container unhealthy", Sensor: "container.*.health", Op: "==", Value: "unhealthy", ForSec: 180, Severity: "warn", Enabled: true},
|
||||
{ID: "service-failed", Name: "A service has failed", Sensor: "services.failed", Op: ">", Value: 0.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
{ID: "disk-full", Name: "Filesystem nearly full", Sensor: "fs.*.used_pct", Op: ">=", Value: 90.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
{ID: "pool-unhealthy", Name: "ZFS pool not healthy", Sensor: "zfs.*.healthy", Op: "==", Value: false, ForSec: 60, Severity: "crit", Enabled: true},
|
||||
{ID: "drive-failing", Name: "Drive failing SMART", Sensor: "smart.*.passed", Op: "==", Value: false, ForSec: 0, Severity: "crit", Enabled: true},
|
||||
{ID: "on-battery", Name: "UPS on battery", Sensor: "ups.on_battery", Op: "==", Value: true, ForSec: 30, Severity: "crit", Enabled: true},
|
||||
{ID: "hot", Name: "CPU running hot", Sensor: "cpu.temp", Op: ">=", Value: 90.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
{ID: "memory", Name: "Memory nearly exhausted", Sensor: "mem.used_pct", Op: ">=", Value: 95.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
}
|
||||
|
||||
func newAlertEngine(s *Server) *alertEngine {
|
||||
e := &alertEngine{s: s, state: map[string]*alertState{}}
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.Alerts })
|
||||
_ = json.Unmarshal(raw, &e.saved)
|
||||
if !e.saved.Seeded {
|
||||
e.saved.Rules, e.saved.Seeded = append([]AlertRule{}, defaultRules...), true
|
||||
}
|
||||
for key, k := range e.saved.Firing {
|
||||
e.state[key] = &alertState{pending: time.UnixMilli(k.Since), firing: true, since: time.UnixMilli(k.Since), value: k.Value}
|
||||
}
|
||||
// the Virtualization tab's default went with the tab (2026-10-15): nothing publishes instance.* now
|
||||
kept := e.saved.Rules[:0]
|
||||
for _, r := range e.saved.Rules {
|
||||
if r.ID != "instance-down" {
|
||||
kept = append(kept, r)
|
||||
}
|
||||
}
|
||||
e.saved.Rules = kept
|
||||
for i, r := range e.saved.Rules { // the first day's default watched "not running", which a container stopped on purpose also is
|
||||
if r.ID == "container-down" && r.Sensor == "container.*.running" {
|
||||
e.saved.Rules[i].Name, e.saved.Rules[i].Sensor, e.saved.Rules[i].Value = "Container down", "container.*.down", true
|
||||
}
|
||||
}
|
||||
go e.run()
|
||||
return e
|
||||
}
|
||||
|
||||
func (e *alertEngine) persist() {
|
||||
b, _ := json.Marshal(e.saved)
|
||||
_ = e.s.store.Update(func(st *store.State) { st.Alerts = b })
|
||||
}
|
||||
|
||||
func (e *alertEngine) run() {
|
||||
time.Sleep(20 * time.Second) // let the status sensors take their first readings
|
||||
for {
|
||||
e.evaluate()
|
||||
time.Sleep(alertEvery)
|
||||
}
|
||||
}
|
||||
|
||||
func wants(r AlertRule, machine string) bool {
|
||||
if r.Machines == "" || r.Machines == "*" {
|
||||
return true
|
||||
}
|
||||
for _, m := range strings.Split(r.Machines, ",") {
|
||||
if strings.TrimSpace(m) == machine {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// holds reports whether `got <op> want`, comparing like with like.
|
||||
func holds(got any, op string, want any) bool {
|
||||
switch g := got.(type) {
|
||||
case float64:
|
||||
w, ok := want.(float64)
|
||||
if !ok {
|
||||
if s, isStr := want.(string); isStr {
|
||||
f, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
w = f
|
||||
} else {
|
||||
return false
|
||||
}
|
||||
}
|
||||
switch op {
|
||||
case ">":
|
||||
return g > w
|
||||
case ">=":
|
||||
return g >= w
|
||||
case "<":
|
||||
return g < w
|
||||
case "<=":
|
||||
return g <= w
|
||||
case "==":
|
||||
return g == w
|
||||
case "!=":
|
||||
return g != w
|
||||
}
|
||||
case bool:
|
||||
w, ok := want.(bool)
|
||||
if !ok {
|
||||
w = fmt.Sprint(want) == "true"
|
||||
}
|
||||
return (op == "==" && g == w) || (op == "!=" && g != w)
|
||||
case string:
|
||||
w := fmt.Sprint(want)
|
||||
return (op == "==" && g == w) || (op == "!=" && g != w)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (e *alertEngine) evaluate() {
|
||||
e.mu.Lock()
|
||||
rules := append([]AlertRule{}, e.saved.Rules...)
|
||||
e.mu.Unlock()
|
||||
query := map[string]string{}
|
||||
for _, r := range rules {
|
||||
if r.Enabled && r.Sensor != "hamlab.reachable" {
|
||||
query[r.ID] = r.Sensor
|
||||
}
|
||||
}
|
||||
qb, _ := json.Marshal(query)
|
||||
machines := append([]string{e.s.hostName()}, e.s.nodes.Names()...)
|
||||
results := map[string]map[string]any{} // machine → ruleID → value | {id: value}
|
||||
reach := map[string]bool{}
|
||||
var wg sync.WaitGroup
|
||||
var mu sync.Mutex
|
||||
for _, m := range machines {
|
||||
wg.Add(1)
|
||||
go func(m string) {
|
||||
defer wg.Done()
|
||||
var res map[string]any
|
||||
ok := false
|
||||
if m == e.s.hostName() {
|
||||
if out, err := sensors.Shape(e.s.sensors.All(), qb, false); err == nil {
|
||||
res, ok = out.(map[string]any)
|
||||
}
|
||||
} else {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
code, b, err := e.s.nodes.Do(ctx, m, "POST", "/api/sensors/query", qb, http.Header{"Content-Type": {"application/json"}})
|
||||
cancel()
|
||||
ok = err == nil && code == 200 && json.Unmarshal(b, &res) == nil
|
||||
}
|
||||
mu.Lock()
|
||||
results[m], reach[m] = res, ok
|
||||
mu.Unlock()
|
||||
}(m)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
now := time.Now()
|
||||
seen := map[string]bool{}
|
||||
var events []AlertEvent
|
||||
e.mu.Lock()
|
||||
for _, r := range rules {
|
||||
if !r.Enabled {
|
||||
continue
|
||||
}
|
||||
for _, m := range machines {
|
||||
if !wants(r, m) {
|
||||
continue
|
||||
}
|
||||
readings := map[string]any{}
|
||||
if r.Sensor == "hamlab.reachable" {
|
||||
readings["hamlab.reachable"] = reach[m]
|
||||
} else if !reach[m] {
|
||||
continue // no reading is not a reading: an unreachable machine is the offline rule's business
|
||||
} else if v, ok := results[m][r.ID]; ok {
|
||||
if many, isMap := v.(map[string]any); isMap {
|
||||
readings = many
|
||||
} else if v != nil {
|
||||
readings[r.Sensor] = v
|
||||
}
|
||||
}
|
||||
for id, val := range readings {
|
||||
key := r.ID + "|" + m + "|" + id
|
||||
seen[key] = true
|
||||
st := e.state[key]
|
||||
if st == nil {
|
||||
st = &alertState{}
|
||||
e.state[key] = st
|
||||
}
|
||||
st.value = val
|
||||
if holds(val, r.Op, r.Value) {
|
||||
if st.pending.IsZero() {
|
||||
st.pending = now
|
||||
}
|
||||
if !st.firing && now.Sub(st.pending) >= time.Duration(r.ForSec)*time.Second {
|
||||
st.firing, st.since = true, now
|
||||
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "firing", Rule: r.Name, RuleID: r.ID, Machine: m, Sensor: id, Value: val, Severity: r.Severity})
|
||||
}
|
||||
} else {
|
||||
if st.firing {
|
||||
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "resolved", Rule: r.Name, RuleID: r.ID, Machine: m, Sensor: id, Value: val, Severity: r.Severity})
|
||||
}
|
||||
st.pending, st.firing = time.Time{}, false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
byID := map[string]AlertRule{}
|
||||
for _, r := range rules {
|
||||
byID[r.ID] = r
|
||||
}
|
||||
for key, st := range e.state { // no longer watched: the rule was paused, deleted or narrowed, or the sensor is gone
|
||||
if seen[key] {
|
||||
continue
|
||||
}
|
||||
if st.firing { // it was on the Alerts tab and perhaps on a phone: say that it is over, and why
|
||||
part := strings.SplitN(key, "|", 3)
|
||||
r, known := byID[part[0]]
|
||||
name, why := r.Name, "no longer watched"
|
||||
if !known {
|
||||
name, why = part[0], "rule deleted"
|
||||
} else if !r.Enabled {
|
||||
why = "rule paused"
|
||||
}
|
||||
if name == "" {
|
||||
name = part[0]
|
||||
}
|
||||
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "resolved", Rule: name, RuleID: part[0], Machine: part[1], Sensor: part[2], Value: why, Severity: r.Severity})
|
||||
}
|
||||
delete(e.state, key)
|
||||
}
|
||||
e.last = now
|
||||
kept := map[string]alertKept{}
|
||||
for key, st := range e.state {
|
||||
if st.firing {
|
||||
kept[key] = alertKept{Since: st.since.UnixMilli(), Value: st.value}
|
||||
}
|
||||
}
|
||||
changed := len(kept) != len(e.saved.Firing)
|
||||
for k := range kept {
|
||||
if _, ok := e.saved.Firing[k]; !ok {
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
e.saved.Firing = kept
|
||||
if changed && len(events) == 0 {
|
||||
e.persist()
|
||||
}
|
||||
if len(events) > 0 {
|
||||
e.saved.Events = append(e.saved.Events, events...)
|
||||
if len(e.saved.Events) > maxEvents {
|
||||
e.saved.Events = e.saved.Events[len(e.saved.Events)-maxEvents:]
|
||||
}
|
||||
e.persist()
|
||||
}
|
||||
channels := append([]AlertChannel{}, e.saved.Channels...)
|
||||
e.mu.Unlock()
|
||||
for _, ev := range events {
|
||||
e.notify(channels, ev)
|
||||
}
|
||||
}
|
||||
|
||||
func (e *alertEngine) firing() []alertFiring {
|
||||
out := []alertFiring{}
|
||||
names := map[string]AlertRule{}
|
||||
for _, r := range e.saved.Rules {
|
||||
names[r.ID] = r
|
||||
}
|
||||
for key, st := range e.state {
|
||||
if !st.firing {
|
||||
continue
|
||||
}
|
||||
p := strings.SplitN(key, "|", 3)
|
||||
r := names[p[0]]
|
||||
out = append(out, alertFiring{AlertEvent: AlertEvent{T: st.since.UnixMilli(), Kind: "firing", Rule: r.Name, RuleID: r.ID, Machine: p[1], Sensor: p[2], Value: st.value, Severity: r.Severity}, Since: st.since.UnixMilli()})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Since > out[j].Since })
|
||||
return out
|
||||
}
|
||||
|
||||
func describe(ev AlertEvent) (title, body string) {
|
||||
mark := map[string]string{"firing": "▲", "resolved": "✓"}[ev.Kind]
|
||||
title = fmt.Sprintf("%s %s — %s", mark, ev.Rule, ev.Machine)
|
||||
body = fmt.Sprintf("%s = %v", ev.Sensor, ev.Value)
|
||||
if ev.Kind == "resolved" {
|
||||
body = "back to normal: " + body
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func (e *alertEngine) notify(channels []AlertChannel, ev AlertEvent) {
|
||||
title, body := describe(ev)
|
||||
for _, c := range channels {
|
||||
var req *http.Request
|
||||
switch c.Type {
|
||||
case "ntfy":
|
||||
req, _ = http.NewRequest("POST", c.URL, strings.NewReader(body))
|
||||
req.Header.Set("Title", title)
|
||||
req.Header.Set("Priority", map[string]string{"crit": "urgent", "warn": "high"}[ev.Severity])
|
||||
req.Header.Set("Tags", map[string]string{"firing": "warning", "resolved": "white_check_mark"}[ev.Kind])
|
||||
if ev.Kind == "resolved" {
|
||||
req.Header.Set("Priority", "default")
|
||||
}
|
||||
case "webhook":
|
||||
b, _ := json.Marshal(map[string]any{"title": title, "message": body, "event": ev})
|
||||
req, _ = http.NewRequest("POST", c.URL, bytes.NewReader(b))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if req == nil {
|
||||
continue
|
||||
}
|
||||
go func(req *http.Request, url string) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
resp, err := http.DefaultClient.Do(req.WithContext(ctx))
|
||||
if err != nil {
|
||||
log.Printf("alert: notifying %s: %v", url, err)
|
||||
return
|
||||
}
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode >= 300 {
|
||||
log.Printf("alert: %s answered %d", url, resp.StatusCode)
|
||||
}
|
||||
}(req, c.URL)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- API ----
|
||||
|
||||
func (s *Server) alertsGet(w http.ResponseWriter, _ *http.Request) {
|
||||
e := s.alerts
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
ev := append([]AlertEvent{}, e.saved.Events...)
|
||||
for i, j := 0, len(ev)-1; i < j; i, j = i+1, j-1 {
|
||||
ev[i], ev[j] = ev[j], ev[i]
|
||||
}
|
||||
rules, channels := e.saved.Rules, e.saved.Channels
|
||||
if rules == nil { // an empty list is a list: never hand the page a null to take the length of
|
||||
rules = []AlertRule{}
|
||||
}
|
||||
if channels == nil {
|
||||
channels = []AlertChannel{}
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"rules": rules, "channels": channels, "events": ev, "firing": e.firing(), "checked": e.last.UnixMilli(), "every_sec": int(alertEvery.Seconds())})
|
||||
}
|
||||
|
||||
func (s *Server) alertsPut(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Rules *[]AlertRule `json:"rules"`
|
||||
Channels *[]AlertChannel `json:"channels"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 256<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if body.Rules != nil && len(*body.Rules) > maxRules {
|
||||
jsonErr(w, 400, fmt.Sprintf("at most %d rules: every one is checked on every machine every %d s", maxRules, int(alertEvery.Seconds())))
|
||||
return
|
||||
}
|
||||
if body.Rules != nil {
|
||||
ids := map[string]bool{}
|
||||
for i := range *body.Rules {
|
||||
ru := &(*body.Rules)[i]
|
||||
ru.Name, ru.Sensor = strings.TrimSpace(ru.Name), strings.TrimSpace(ru.Sensor)
|
||||
if ru.ID == "" || ids[ru.ID] || ru.Name == "" || len(ru.Name) > maxNameLen || ru.ForSec < 0 || ru.ForSec > 86400 {
|
||||
jsonErr(w, 400, "every rule needs a unique id, a name (64 characters at most) and a patience between 0 and 86400 s")
|
||||
return
|
||||
}
|
||||
ids[ru.ID] = true
|
||||
if _, err := sensors.Matcher(ru.Sensor); err != nil || ru.Sensor == "" {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: %q is not a sensor ID or pattern", ru.Name, ru.Sensor))
|
||||
return
|
||||
}
|
||||
if !strings.Contains(" > >= < <= == != ", " "+ru.Op+" ") {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: the comparison must be one of > >= < <= == !=", ru.Name))
|
||||
return
|
||||
}
|
||||
switch val := ru.Value.(type) {
|
||||
case float64, bool:
|
||||
case string:
|
||||
if len(val) > 200 {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: the value is too long", ru.Name))
|
||||
return
|
||||
}
|
||||
if ru.Op != "==" && ru.Op != "!=" { // "cpu.usage > ninety" can never be true, and would never say so
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: %s compares numbers, and %q is not one; text can only be == or !=", ru.Name, ru.Op, val))
|
||||
return
|
||||
}
|
||||
default:
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: the value must be a number, true/false or text", ru.Name))
|
||||
return
|
||||
}
|
||||
if _, isBool := ru.Value.(bool); isBool && ru.Op != "==" && ru.Op != "!=" {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: true/false can only be == or !=", ru.Name))
|
||||
return
|
||||
}
|
||||
if ru.Severity != "info" && ru.Severity != "crit" {
|
||||
ru.Severity = "warn"
|
||||
}
|
||||
}
|
||||
}
|
||||
if body.Channels != nil {
|
||||
for _, c := range *body.Channels {
|
||||
if (c.Type != "ntfy" && c.Type != "webhook") || !(strings.HasPrefix(c.URL, "https://") || strings.HasPrefix(c.URL, "http://")) {
|
||||
jsonErr(w, 400, "a channel is {type: ntfy|webhook, url: http(s)://…}")
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
e := s.alerts
|
||||
e.mu.Lock()
|
||||
if body.Rules != nil {
|
||||
e.saved.Rules = *body.Rules
|
||||
}
|
||||
if body.Channels != nil {
|
||||
e.saved.Channels = *body.Channels
|
||||
}
|
||||
e.persist()
|
||||
e.mu.Unlock()
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// alertsTest sends one clearly-marked test notification through every channel.
|
||||
func (s *Server) alertsTest(w http.ResponseWriter, _ *http.Request) {
|
||||
e := s.alerts
|
||||
e.mu.Lock()
|
||||
channels := append([]AlertChannel{}, e.saved.Channels...)
|
||||
e.mu.Unlock()
|
||||
if len(channels) == 0 {
|
||||
jsonErr(w, 400, "no channels to test")
|
||||
return
|
||||
}
|
||||
e.notify(channels, AlertEvent{T: time.Now().UnixMilli(), Kind: "firing", Rule: "Test alert", Machine: s.hostName(), Sensor: "hamlab.test", Value: "this is only a test", Severity: "info"})
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,650 @@
|
||||
package api
|
||||
|
||||
// The Docker tab: what is running, grouped the way it was deployed (compose stacks), with
|
||||
// start / stop / restart, logs and inspect — plus images, volumes and networks to look at.
|
||||
// It drives the docker CLI (or podman), as the user first and as root if that is refused.
|
||||
// What can be removed here is what nothing uses: an untagged image, a volume no container
|
||||
// holds, a network nothing is attached to. Compose stays the one place a stack is defined.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type dockerContainer struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Image string `json:"image"`
|
||||
State string `json:"state"` // running | exited | paused | restarting | created | dead
|
||||
Status string `json:"status"` // "Up 3 hours (healthy)"
|
||||
Health string `json:"health"` // healthy | unhealthy | starting | none
|
||||
Ports string `json:"ports"`
|
||||
Stack string `json:"stack"` // compose project, "" when started by hand
|
||||
Service string `json:"service"` // compose service
|
||||
Created string `json:"created"`
|
||||
Mem float64 `json:"mem"` // bytes, from the container's cgroup; 0 when unknown
|
||||
}
|
||||
|
||||
func dockerTool() string {
|
||||
for _, t := range []string{"docker", "podman"} {
|
||||
if _, err := exec.LookPath(t); err == nil {
|
||||
return t
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// dockerRun runs the tool as this user, and again as root if the socket said no.
|
||||
// DockerTool is whichever of docker or podman this machine has, for callers outside the
|
||||
// package — the scheduled updates the CLI runs.
|
||||
func DockerTool() string { return dockerTool() }
|
||||
|
||||
func dockerRun(ctx context.Context, args ...string) ([]byte, error) {
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
return nil, errors.New("neither docker nor podman is installed here")
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, tool, args...).CombinedOutput()
|
||||
if err != nil && priv.Available() && strings.Contains(strings.ToLower(string(out)), "permission denied") {
|
||||
full := priv.Wrap(append([]string{tool}, args...)...)
|
||||
out, err = exec.CommandContext(ctx, full[0], full[1:]...).CombinedOutput()
|
||||
}
|
||||
if err != nil {
|
||||
msg := strings.TrimSpace(string(out))
|
||||
if i := strings.LastIndex(msg, "\n"); i >= 0 {
|
||||
msg = msg[i+1:]
|
||||
}
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return out, errors.New(msg)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// dockerWhy picks the line worth showing out of a failed command. Some subcommands —
|
||||
// `network rm` among them — print the real reason first and then an aggregated "exit
|
||||
// status 1", which is what dockerRun's last-line rule would otherwise hand back.
|
||||
func dockerWhy(out []byte, err error) string {
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
if l = strings.TrimSpace(l); strings.HasPrefix(l, "Error") || strings.Contains(l, "daemon") {
|
||||
return l
|
||||
}
|
||||
}
|
||||
return err.Error()
|
||||
}
|
||||
|
||||
func jsonLines(out []byte, each func(map[string]any)) {
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
var m map[string]any
|
||||
if json.Unmarshal([]byte(line), &m) == nil {
|
||||
each(m)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func dstr(m map[string]any, k string) string { s, _ := m[k].(string); return s }
|
||||
|
||||
func labelOf(labels, key string) string {
|
||||
for _, kv := range strings.Split(labels, ",") {
|
||||
if v, ok := strings.CutPrefix(kv, key+"="); ok {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (s *Server) dockerOverview(w http.ResponseWriter, r *http.Request) {
|
||||
if dockerTool() == "" {
|
||||
jsonOK(w, 200, map[string]any{"available": false, "message": "Docker isn't installed on this machine."})
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "ps", "-a", "--no-trunc", "--format", "{{json .}}")
|
||||
if err != nil {
|
||||
jsonOK(w, 200, map[string]any{"available": false, "message": err.Error()})
|
||||
return
|
||||
}
|
||||
containers := []dockerContainer{}
|
||||
jsonLines(out, func(m map[string]any) {
|
||||
c := dockerContainer{ID: dstr(m, "ID"), Name: strings.Split(dstr(m, "Names"), ",")[0], Image: dstr(m, "Image"), State: dstr(m, "State"), Status: dstr(m, "Status"),
|
||||
Ports: dstr(m, "Ports"), Created: dstr(m, "CreatedAt"), Health: "none", Stack: labelOf(dstr(m, "Labels"), "com.docker.compose.project"), Service: labelOf(dstr(m, "Labels"), "com.docker.compose.service")}
|
||||
for _, h := range []string{"unhealthy", "healthy", "starting"} {
|
||||
if strings.Contains(c.Status, "("+h+")") || strings.Contains(c.Status, "health: "+h) {
|
||||
c.Health = h
|
||||
break
|
||||
}
|
||||
}
|
||||
for _, dir := range []string{"/sys/fs/cgroup/system.slice/docker-" + c.ID + ".scope", "/sys/fs/cgroup/docker/" + c.ID, "/sys/fs/cgroup/machine.slice/libpod-" + c.ID + ".scope"} {
|
||||
if b, err := os.ReadFile(dir + "/memory.current"); err == nil {
|
||||
c.Mem, _ = strconv.ParseFloat(strings.TrimSpace(string(b)), 64)
|
||||
break
|
||||
}
|
||||
}
|
||||
containers = append(containers, c)
|
||||
})
|
||||
sort.Slice(containers, func(i, j int) bool {
|
||||
if containers[i].Stack != containers[j].Stack {
|
||||
return containers[i].Stack < containers[j].Stack
|
||||
}
|
||||
return containers[i].Name < containers[j].Name
|
||||
})
|
||||
list := func(args ...string) []map[string]any {
|
||||
rows := []map[string]any{}
|
||||
if out, err := dockerRun(ctx, args...); err == nil {
|
||||
jsonLines(out, func(m map[string]any) { rows = append(rows, m) })
|
||||
}
|
||||
return rows
|
||||
}
|
||||
// Who uses what, from one inspect of every container: an image, a volume or a network
|
||||
// nothing uses is the only kind this page offers to remove.
|
||||
imageUsed := map[string]bool{}
|
||||
volumeUsers, networkUsers := map[string][]string{}, map[string][]string{}
|
||||
if ids, err := dockerRun(ctx, "ps", "-aq"); err == nil && len(strings.Fields(string(ids))) > 0 {
|
||||
const f = `{{.Name}}|{{.Image}}|{{range .Mounts}}{{if eq .Type "volume"}}{{.Name}},{{end}}{{end}}|{{range $k, $v := .NetworkSettings.Networks}}{{$k}},{{end}}`
|
||||
if out, err := dockerRun(ctx, append([]string{"inspect", "-f", f}, strings.Fields(string(ids))...)...); err == nil {
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
p := strings.SplitN(line, "|", 4)
|
||||
if len(p) < 4 {
|
||||
continue
|
||||
}
|
||||
name := strings.TrimPrefix(p[0], "/")
|
||||
if id := strings.TrimPrefix(p[1], "sha256:"); len(id) >= 12 {
|
||||
imageUsed[id[:12]] = true
|
||||
}
|
||||
for _, v := range strings.Split(p[2], ",") {
|
||||
if v != "" {
|
||||
volumeUsers[v] = append(volumeUsers[v], name)
|
||||
}
|
||||
}
|
||||
for _, n := range strings.Split(p[3], ",") {
|
||||
if n != "" {
|
||||
networkUsers[n] = append(networkUsers[n], name)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
images := list("images", "--format", "{{json .}}")
|
||||
for _, im := range images {
|
||||
im["in_use"] = imageUsed[dstr(im, "ID")]
|
||||
}
|
||||
volumes := list("volume", "ls", "--format", "{{json .}}")
|
||||
for _, v := range volumes {
|
||||
users := volumeUsers[dstr(v, "Name")]
|
||||
if users == nil {
|
||||
users = []string{}
|
||||
}
|
||||
sort.Strings(users)
|
||||
v["used_by"] = users
|
||||
v["anonymous"] = strings.Contains(dstr(v, "Labels"), "com.docker.volume.anonymous")
|
||||
}
|
||||
networks := list("network", "ls", "--format", "{{json .}}")
|
||||
for _, n := range networks {
|
||||
users := networkUsers[dstr(n, "Name")]
|
||||
if users == nil {
|
||||
users = []string{}
|
||||
}
|
||||
sort.Strings(users)
|
||||
n["used_by"] = users
|
||||
n["predefined"] = predefinedNetwork[dstr(n, "Name")]
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"available": true, "tool": dockerTool(), "containers": containers,
|
||||
"images": images, "volumes": volumes, "networks": networks})
|
||||
}
|
||||
|
||||
var dockerRef = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}$`)
|
||||
|
||||
func (s *Server) dockerAction(w http.ResponseWriter, r *http.Request) {
|
||||
id, action := r.PathValue("id"), r.PathValue("action")
|
||||
if !dockerRef.MatchString(id) || (action != "start" && action != "stop" && action != "restart" && action != "remove") {
|
||||
jsonErr(w, 400, "start, stop, restart or remove a container by name or ID")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
verb := action
|
||||
if action == "remove" { // never forced: a running container has to be stopped first, on purpose
|
||||
verb = "rm"
|
||||
}
|
||||
if _, err := dockerRun(ctx, verb, id); err != nil {
|
||||
code := 500
|
||||
if action == "remove" {
|
||||
code = 409
|
||||
}
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: %s %s (from %s)", action, id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
var imageRef = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,254}$`)
|
||||
|
||||
// imagePull runs `docker pull` as a task the page follows.
|
||||
func (s *Server) imagePull(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Image string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !imageRef.MatchString(strings.TrimSpace(body.Image)) {
|
||||
jsonErr(w, 400, "an image looks like nginx:alpine or ghcr.io/owner/name:tag")
|
||||
return
|
||||
}
|
||||
ref := strings.TrimSpace(body.Image)
|
||||
t, err := s.tasks.Start("docker:pull", "pull "+ref, exec.Command(dockerTool(), "pull", ref))
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: pull %s as task %s (from %s)", ref, t.ID, clientIP(r))
|
||||
jsonOK(w, 202, t.Snapshot(false))
|
||||
}
|
||||
|
||||
// imageRemove is `docker rmi`, unforced: an image something still uses is refused by Docker, and that is the answer.
|
||||
func (s *Server) imageRemove(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Image string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !imageRef.MatchString(strings.TrimSpace(body.Image)) {
|
||||
jsonErr(w, 400, "which image?")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
if _, err := dockerRun(ctx, "rmi", strings.TrimSpace(body.Image)); err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: rmi %s (from %s)", body.Image, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// imagePrune removes dangling images — layers no tag points at any more, left behind by pulls —
|
||||
// or, asked for {"all": true}, every image no container uses. Those can all be pulled again.
|
||||
func (s *Server) imagePrune(w http.ResponseWriter, r *http.Request) {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
var body struct{ All bool }
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
args := []string{"image", "prune", "-f"}
|
||||
if body.All {
|
||||
args = append(args, "-a")
|
||||
}
|
||||
out, err := dockerRun(ctx, args...)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
log.Printf("docker: image prune (from %s)", clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
|
||||
}
|
||||
|
||||
// dockerLogs is the tail of a container's log; ?since=<RFC3339> asks only for what is new,
|
||||
// which is how the page follows a log without holding a connection open.
|
||||
func (s *Server) dockerLogs(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
args := []string{"logs", "--timestamps"}
|
||||
if since := r.URL.Query().Get("since"); since != "" {
|
||||
if _, err := time.Parse(time.RFC3339Nano, since); err != nil {
|
||||
jsonErr(w, 400, "since is an RFC 3339 time")
|
||||
return
|
||||
}
|
||||
args = append(args, "--since", since)
|
||||
} else {
|
||||
n, _ := strconv.Atoi(r.URL.Query().Get("n"))
|
||||
if n <= 0 || n > 5000 {
|
||||
n = 300
|
||||
}
|
||||
args = append(args, "--tail", strconv.Itoa(n))
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, append(args, id)...)
|
||||
if err != nil {
|
||||
code := 500
|
||||
if strings.Contains(strings.ToLower(err.Error()), "no such container") {
|
||||
code = 404
|
||||
}
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
type line struct {
|
||||
T string `json:"t"`
|
||||
M string `json:"m"`
|
||||
}
|
||||
lines := []line{}
|
||||
for _, l := range strings.Split(strings.TrimRight(string(out), "\n"), "\n") {
|
||||
if l == "" {
|
||||
continue
|
||||
}
|
||||
t, m, ok := strings.Cut(l, " ")
|
||||
if _, err := time.Parse(time.RFC3339Nano, t); !ok || err != nil {
|
||||
t, m = "", l
|
||||
}
|
||||
lines = append(lines, line{t, m})
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"lines": lines})
|
||||
}
|
||||
|
||||
// dockerInspect is the parts of `docker inspect` worth reading on a phone.
|
||||
// dockerErr answers for a docker command that failed: a thing that isn't there is a 404,
|
||||
// the daemon being away a 503, anything else a 500.
|
||||
func dockerErr(w http.ResponseWriter, err error) {
|
||||
msg := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(err.Error()), "error: "))
|
||||
low := strings.ToLower(msg)
|
||||
switch {
|
||||
case strings.Contains(low, "no such object") || strings.Contains(low, "no such container") || strings.Contains(low, "not found"):
|
||||
jsonErr(w, 404, msg)
|
||||
case strings.Contains(low, "cannot connect to the docker daemon") || strings.Contains(low, "is the docker daemon running"):
|
||||
jsonErr(w, 503, msg)
|
||||
default:
|
||||
jsonErr(w, 500, msg)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "inspect", id)
|
||||
if err != nil {
|
||||
dockerErr(w, err)
|
||||
return
|
||||
}
|
||||
var all []map[string]any
|
||||
if json.Unmarshal(out, &all) != nil || len(all) == 0 {
|
||||
jsonErr(w, 500, "inspect: unreadable answer")
|
||||
return
|
||||
}
|
||||
c := all[0]
|
||||
dig := func(m map[string]any, path ...string) any {
|
||||
var cur any = m
|
||||
for _, p := range path {
|
||||
mm, ok := cur.(map[string]any)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
cur = mm[p]
|
||||
}
|
||||
return cur
|
||||
}
|
||||
env := []string{}
|
||||
if list, ok := dig(c, "Config", "Env").([]any); ok {
|
||||
for _, e := range list {
|
||||
k, _, _ := strings.Cut(fmt.Sprint(e), "=")
|
||||
env = append(env, k) // names only: values are where the secrets live
|
||||
}
|
||||
}
|
||||
sort.Strings(env)
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"image": dig(c, "Config", "Image"), "command": dig(c, "Config", "Cmd"), "entrypoint": dig(c, "Config", "Entrypoint"),
|
||||
"restart": dig(c, "HostConfig", "RestartPolicy", "Name"), "network_mode": dig(c, "HostConfig", "NetworkMode"),
|
||||
"memory_limit": dig(c, "HostConfig", "Memory"), "started": dig(c, "State", "StartedAt"), "exit_code": dig(c, "State", "ExitCode"),
|
||||
"error": dig(c, "State", "Error"), "restarts": c["RestartCount"], "mounts": c["Mounts"], "ports": dig(c, "NetworkSettings", "Ports"),
|
||||
"networks": dig(c, "NetworkSettings", "Networks"), "env_names": env,
|
||||
"compose_file": dig(c, "Config", "Labels", "com.docker.compose.project.config_files"),
|
||||
})
|
||||
}
|
||||
|
||||
var volumeRef = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,254}$`)
|
||||
|
||||
// volumeRemove deletes one volume and what is in it, which nothing brings back. So it is never
|
||||
// forced (Docker refuses a volume a container still uses), it is one at a time, and the request
|
||||
// has to name the volume twice — a slip of a script can't delete one by accident.
|
||||
func (s *Server) volumeRemove(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name, Confirm string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !volumeRef.MatchString(body.Name) {
|
||||
jsonErr(w, 400, "which volume?")
|
||||
return
|
||||
}
|
||||
if body.Confirm != body.Name {
|
||||
jsonErr(w, 400, "name the volume again in \"confirm\" to delete it")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
if _, err := dockerRun(ctx, "volume", "rm", body.Name); err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: volume rm %s (from %s)", body.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// volumePrune removes unused *anonymous* volumes only: the nameless ones an image's VOLUME line
|
||||
// leaves behind each time its container is recreated. Named volumes are somebody's data, and a
|
||||
// stack that is merely down leaves its volumes "unused" — those go one at a time, by name.
|
||||
func (s *Server) volumePrune(w http.ResponseWriter, r *http.Request) {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "volume", "prune", "-f")
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
log.Printf("docker: volume prune (from %s)", clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
|
||||
}
|
||||
|
||||
// The three networks Docker makes itself. They hold nothing, cannot be removed, and
|
||||
// their absence would break every container that asks for one by name.
|
||||
var predefinedNetwork = map[string]bool{"bridge": true, "host": true, "none": true}
|
||||
|
||||
// networkRemove deletes one network. Unlike a volume there is no data in it, and unlike an
|
||||
// image nothing has to be downloaded again — a stack that wants it back gets it recreated
|
||||
// on the next deploy. Docker refuses one a container is still attached to, which is the
|
||||
// check that matters, so this asks for no second confirmation.
|
||||
func (s *Server) networkRemove(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !volumeRef.MatchString(body.Name) {
|
||||
jsonErr(w, 400, "which network?")
|
||||
return
|
||||
}
|
||||
if predefinedNetwork[body.Name] {
|
||||
jsonErr(w, 400, body.Name+" is one of Docker's own networks and can't be removed")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
if out, err := dockerRun(ctx, "network", "rm", body.Name); err != nil {
|
||||
jsonErr(w, 409, dockerWhy(out, err))
|
||||
return
|
||||
}
|
||||
log.Printf("docker: network rm %s (from %s)", body.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// networkPrune removes every network no container is attached to. A stack that is merely
|
||||
// down counts as unattached, so its network goes too — and comes back on the next deploy,
|
||||
// with a new subnet. That is the whole cost, and the caller is told it.
|
||||
func (s *Server) networkPrune(w http.ResponseWriter, r *http.Request) {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "network", "prune", "-f")
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
log.Printf("docker: network prune (from %s)", clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
|
||||
}
|
||||
|
||||
// ---- a container's health, its recent life, its console, and its limits ----
|
||||
|
||||
// dockerHealth is the health check's verdict and the last few probe results. Docker keeps
|
||||
// the last five itself, which is all anyone wants: is it failing, and what did it say.
|
||||
func (s *Server) dockerHealth(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id)
|
||||
if err != nil {
|
||||
dockerErr(w, err)
|
||||
return
|
||||
}
|
||||
var st struct {
|
||||
Status string `json:"Status"`
|
||||
Health *struct {
|
||||
Status string `json:"Status"`
|
||||
FailingStreak int `json:"FailingStreak"`
|
||||
Log []struct {
|
||||
Start time.Time `json:"Start"`
|
||||
End time.Time `json:"End"`
|
||||
ExitCode int `json:"ExitCode"`
|
||||
Output string `json:"Output"`
|
||||
} `json:"Log"`
|
||||
} `json:"Health"`
|
||||
}
|
||||
if json.Unmarshal(out, &st) != nil {
|
||||
jsonErr(w, 500, "could not read the container's state")
|
||||
return
|
||||
}
|
||||
if st.Health == nil {
|
||||
jsonOK(w, 200, map[string]any{"configured": false, "state": st.Status})
|
||||
return
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"configured": true, "state": st.Status, "status": st.Health.Status,
|
||||
"failing_streak": st.Health.FailingStreak, "log": st.Health.Log,
|
||||
})
|
||||
}
|
||||
|
||||
// dockerExec opens a shell inside a container, as a terminal session like any other — so
|
||||
// it reattaches, resizes and scrolls back the same way, and works on a managed machine
|
||||
// through the same proxy.
|
||||
func (s *Server) dockerExec(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
jsonErr(w, 500, "neither docker nor podman is installed here")
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
Shell string `json:"shell"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
shell := strings.TrimSpace(body.Shell)
|
||||
if shell == "" {
|
||||
// most images have sh; bash is nicer where it exists, so try it first and fall back
|
||||
shell = "if command -v bash >/dev/null 2>&1; then exec bash; else exec sh; fi"
|
||||
} else if strings.ContainsAny(shell, "\n\r") {
|
||||
jsonErr(w, 400, "bad shell")
|
||||
return
|
||||
}
|
||||
argv := []string{tool, "exec", "-it", id, "sh", "-c", shell}
|
||||
if priv.Available() && !dockerReachable() {
|
||||
argv = priv.Wrap(argv...)
|
||||
}
|
||||
sess, err := s.mgr.CreateCmd("", "sh "+shortName(id), argv)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker exec into %s as terminal %s by %s", id, sess.ID, clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"id": sess.ID, "name": "sh " + shortName(id)})
|
||||
}
|
||||
|
||||
func shortName(id string) string {
|
||||
if len(id) > 20 {
|
||||
return id[:20]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// dockerReachable reports whether docker answers without help.
|
||||
func dockerReachable() bool {
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
return false
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
return exec.CommandContext(ctx, tool, "version", "--format", "{{.Server.Version}}").Run() == nil
|
||||
}
|
||||
|
||||
// dockerUpdate changes a running container's limits and restart policy. `docker update`
|
||||
// does this live, without recreating it and without touching the compose file — which is
|
||||
// also the catch, and the caller is told: a deploy from compose puts back whatever compose
|
||||
// says.
|
||||
func (s *Server) dockerUpdate(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
Memory string `json:"memory"` // "512m", "2g", or "" to leave alone; "0" removes the limit
|
||||
CPUs string `json:"cpus"` // "1.5", or "0" for no limit
|
||||
Restart string `json:"restart"` // no | on-failure | unless-stopped | always
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
args := []string{"update"}
|
||||
if m := strings.TrimSpace(body.Memory); m != "" {
|
||||
if !regexp.MustCompile(`^[0-9]+([bkmgBKMG])?$`).MatchString(m) {
|
||||
jsonErr(w, 400, "memory: a number, optionally with b, k, m or g — 0 removes the limit")
|
||||
return
|
||||
}
|
||||
args = append(args, "--memory", m, "--memory-swap", m)
|
||||
}
|
||||
if c := strings.TrimSpace(body.CPUs); c != "" {
|
||||
if !regexp.MustCompile(`^[0-9]+(\.[0-9]+)?$`).MatchString(c) {
|
||||
jsonErr(w, 400, "cpus: a number like 1.5 — 0 removes the limit")
|
||||
return
|
||||
}
|
||||
args = append(args, "--cpus", c)
|
||||
}
|
||||
if p := strings.TrimSpace(body.Restart); p != "" {
|
||||
switch p {
|
||||
case "no", "on-failure", "unless-stopped", "always":
|
||||
default:
|
||||
jsonErr(w, 400, "restart: no, on-failure, unless-stopped or always")
|
||||
return
|
||||
}
|
||||
args = append(args, "--restart", p)
|
||||
}
|
||||
if len(args) == 1 {
|
||||
jsonErr(w, 400, "nothing to change")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
|
||||
defer cancel()
|
||||
if _, err := dockerRun(ctx, append(args, id)...); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker update %s %v by %s", id, args[1:], clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package api
|
||||
|
||||
// A container's history.
|
||||
//
|
||||
// `docker events` is a live stream with only a short in-memory backlog — ask it for the
|
||||
// last thirty days and you get the last few minutes, because the daemon keeps events in a
|
||||
// small ring and loses them all when it restarts. That is why Dockhand kept its own copy,
|
||||
// and why this does too: a watcher follows the stream and appends the lifecycle events to a
|
||||
// ring kept beside the config, so "what happened to this container last Tuesday" has an
|
||||
// answer.
|
||||
//
|
||||
// Only lifecycle events are kept. A health check produces three exec events every probe,
|
||||
// which is thousands a day and tells you nothing that the health panel does not.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
const (
|
||||
eventsKept = 4000
|
||||
eventsSaveEvery = 30 * time.Second
|
||||
)
|
||||
|
||||
// ContainerEvent is one thing that happened to one container.
|
||||
type ContainerEvent struct {
|
||||
T int64 `json:"t"` // unix ms
|
||||
Name string `json:"name"`
|
||||
Action string `json:"action"`
|
||||
Image string `json:"image,omitempty"`
|
||||
ExitCode string `json:"exit_code,omitempty"`
|
||||
Signal string `json:"signal,omitempty"`
|
||||
Health string `json:"health,omitempty"`
|
||||
}
|
||||
|
||||
type eventLog struct {
|
||||
mu sync.Mutex
|
||||
ring []ContainerEvent
|
||||
dirty bool
|
||||
path string
|
||||
}
|
||||
|
||||
func (e *eventLog) load(path string) {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
e.path = path
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
var list []ContainerEvent
|
||||
if json.Unmarshal(b, &list) == nil {
|
||||
e.ring = list
|
||||
}
|
||||
}
|
||||
|
||||
func (e *eventLog) add(ev ContainerEvent) {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
e.ring = append(e.ring, ev)
|
||||
if len(e.ring) > eventsKept {
|
||||
e.ring = append([]ContainerEvent(nil), e.ring[len(e.ring)-eventsKept:]...)
|
||||
}
|
||||
e.dirty = true
|
||||
}
|
||||
|
||||
// forName is one container's events, newest first.
|
||||
func (e *eventLog) forName(name string, since time.Time, limit int) []ContainerEvent {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
out := []ContainerEvent{}
|
||||
for _, ev := range e.ring {
|
||||
if ev.Name == name && ev.T >= since.UnixMilli() {
|
||||
out = append(out, ev)
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].T > out[j].T })
|
||||
if len(out) > limit {
|
||||
out = out[:limit]
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (e *eventLog) save() {
|
||||
e.mu.Lock()
|
||||
if !e.dirty || e.path == "" {
|
||||
e.mu.Unlock()
|
||||
return
|
||||
}
|
||||
b, err := json.Marshal(e.ring)
|
||||
e.dirty = false
|
||||
path := e.path
|
||||
e.mu.Unlock()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
_ = os.MkdirAll(filepath.Dir(path), 0o700)
|
||||
tmp := path + ".tmp"
|
||||
if os.WriteFile(tmp, b, 0o600) == nil {
|
||||
_ = os.Rename(tmp, path)
|
||||
}
|
||||
}
|
||||
|
||||
// watchDockerEvents follows the stream, restarting if Docker goes away. It is the only
|
||||
// thing that writes the log, so nothing contends with it.
|
||||
func (s *Server) watchDockerEvents(ctx context.Context) {
|
||||
s.events.load(filepath.Join(s.cfg.DataDir(), "docker-events.json"))
|
||||
go func() { // a periodic flush, so a crash loses at most half a minute of history
|
||||
t := time.NewTicker(eventsSaveEvery)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
s.events.save()
|
||||
return
|
||||
case <-t.C:
|
||||
s.events.save()
|
||||
}
|
||||
}
|
||||
}()
|
||||
for {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
if dockerTool() == "" {
|
||||
time.Sleep(time.Minute) // no Docker here; look again later in case it is installed
|
||||
continue
|
||||
}
|
||||
s.followEvents(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(10 * time.Second): // the daemon restarted, or the pipe broke
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) followEvents(ctx context.Context) {
|
||||
args := []string{"events", "--format", "{{json .}}", "--filter", "type=container"}
|
||||
for _, e := range []string{"create", "start", "restart", "stop", "kill", "die", "destroy", "oom", "pause", "unpause", "rename", "update", "health_status"} {
|
||||
args = append(args, "--filter", "event="+e)
|
||||
}
|
||||
tool := dockerTool()
|
||||
full := append([]string{tool}, args...)
|
||||
if !dockerReachable() && priv.Available() {
|
||||
full = priv.Wrap(full...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return
|
||||
}
|
||||
sc := bufio.NewScanner(out)
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
n := 0
|
||||
for sc.Scan() {
|
||||
var e struct {
|
||||
Action string `json:"Action"`
|
||||
Time int64 `json:"time"`
|
||||
TimeNano int64 `json:"timeNano"`
|
||||
Actor struct {
|
||||
Attributes map[string]string `json:"Attributes"`
|
||||
} `json:"Actor"`
|
||||
}
|
||||
if json.Unmarshal(sc.Bytes(), &e) != nil || e.Action == "" {
|
||||
continue
|
||||
}
|
||||
at := e.Actor.Attributes
|
||||
ev := ContainerEvent{
|
||||
T: e.Time * 1000, Name: at["name"], Image: at["image"],
|
||||
Action: e.Action, ExitCode: at["exitCode"], Signal: at["signal"],
|
||||
}
|
||||
if e.TimeNano > 0 {
|
||||
ev.T = e.TimeNano / 1e6
|
||||
}
|
||||
// health_status comes through as "health_status: healthy"
|
||||
if a, h, ok := strings.Cut(e.Action, ": "); ok {
|
||||
ev.Action, ev.Health = a, h
|
||||
}
|
||||
if ev.Name == "" {
|
||||
continue
|
||||
}
|
||||
s.events.add(ev)
|
||||
n++
|
||||
}
|
||||
_ = cmd.Wait()
|
||||
if n > 0 {
|
||||
log.Printf("docker events: stream ended after %d events; reconnecting", n)
|
||||
}
|
||||
}
|
||||
|
||||
// dockerEvents answers from Ham Lab's own history, not Docker's short memory.
|
||||
func (s *Server) dockerEvents(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
since := r.URL.Query().Get("since")
|
||||
d := 7 * 24 * time.Hour
|
||||
switch since {
|
||||
case "24h":
|
||||
d = 24 * time.Hour
|
||||
case "30d":
|
||||
d = 30 * 24 * time.Hour
|
||||
default:
|
||||
since = "7d"
|
||||
}
|
||||
ev := s.events.forName(id, time.Now().Add(-d), 500)
|
||||
s.events.mu.Lock()
|
||||
total := len(s.events.ring)
|
||||
s.events.mu.Unlock()
|
||||
jsonOK(w, 200, map[string]any{"events": ev, "since": since, "kept": total, "capacity": eventsKept})
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package api
|
||||
|
||||
// Following a container's log over a held connection, the way the Services tab follows a
|
||||
// journal. The polled `?since=` endpoint next door still exists — it is what a reconnect
|
||||
// hands back to pick up from — but the live view holds one `docker logs -f` per viewer
|
||||
// instead of asking again every two seconds.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"os/exec"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/coder/websocket"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// logLine is one line of a container's output: the time Docker stamped it with, the text,
|
||||
// and which of the two streams it came out of — stderr is where a container says what went
|
||||
// wrong, and the page colours it accordingly.
|
||||
type logLine struct {
|
||||
T string `json:"t"`
|
||||
M string `json:"m"`
|
||||
Err bool `json:"err,omitempty"`
|
||||
at string // when to sort it by: its own stamp, or the last one its stream carried
|
||||
}
|
||||
|
||||
// splitStamp pulls Docker's RFC 3339 prefix off a line. A line without one (a container
|
||||
// that writes bare bytes, or a continuation) keeps the whole text and no time — but is
|
||||
// still sorted where its stream put it, behind the last line that did have one.
|
||||
func splitStamp(l string, carry *string) logLine {
|
||||
t, m, ok := strings.Cut(l, " ")
|
||||
if _, err := time.Parse(time.RFC3339Nano, t); !ok || err != nil {
|
||||
return logLine{M: l, at: *carry}
|
||||
}
|
||||
*carry = t
|
||||
return logLine{T: t, M: m, at: t}
|
||||
}
|
||||
|
||||
// dockerTail streams a container's output as JSON text frames until the client goes away.
|
||||
// `?since=<RFC3339>` resumes where a dropped connection left off; otherwise it starts with
|
||||
// the last `?n=` lines.
|
||||
func (s *Server) dockerTail(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
jsonErr(w, 500, "neither docker nor podman is installed here")
|
||||
return
|
||||
}
|
||||
args := []string{tool, "logs", "--timestamps", "--follow"}
|
||||
if since := r.URL.Query().Get("since"); since != "" {
|
||||
if _, err := time.Parse(time.RFC3339Nano, since); err != nil {
|
||||
jsonErr(w, 400, "since is an RFC 3339 time")
|
||||
return
|
||||
}
|
||||
args = append(args, "--since", since)
|
||||
} else {
|
||||
args = append(args, "--tail", strconv.Itoa(logCount(r)))
|
||||
}
|
||||
args = append(args, id)
|
||||
if priv.Available() && !dockerReachable() {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
|
||||
conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{CompressionMode: websocket.CompressionDisabled})
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer conn.Close(websocket.StatusNormalClosure, "")
|
||||
ctx, cancel := context.WithCancel(r.Context())
|
||||
defer cancel()
|
||||
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
// Own process group, so the kill reaches the docker client itself and not just a sudo
|
||||
// wrapper — the same reason the journal tail does it.
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error {
|
||||
pg := "-" + strconv.Itoa(cmd.Process.Pid)
|
||||
if args[0] != tool { // running under sudo: only sudo can signal root's children
|
||||
k := priv.Wrap("kill", "-9", "--", pg)
|
||||
return exec.Command(k[0], k[1:]...).Run()
|
||||
}
|
||||
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||
}
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
stderr, err := cmd.StderrPipe()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
_ = conn.Write(ctx, websocket.MessageText, mustJSON(logLine{M: err.Error(), Err: true}))
|
||||
return
|
||||
}
|
||||
|
||||
// Both pipes funnel into one writer: a websocket takes one writer at a time.
|
||||
lines := make(chan logLine, 256)
|
||||
read := func(p io.Reader, isErr bool) {
|
||||
sc := bufio.NewScanner(p)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
|
||||
carry := ""
|
||||
for sc.Scan() {
|
||||
l := splitStamp(sc.Text(), &carry)
|
||||
l.Err = isErr
|
||||
select {
|
||||
case lines <- l:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
done := make(chan struct{}, 2)
|
||||
go func() { read(stdout, false); done <- struct{}{} }()
|
||||
go func() { read(stderr, true); done <- struct{}{} }()
|
||||
go func() { // the command ends when the container does; closing says so, having said why
|
||||
<-done
|
||||
<-done
|
||||
_ = cmd.Wait()
|
||||
close(lines)
|
||||
}()
|
||||
|
||||
go func() { // reader: only to notice the close
|
||||
defer cancel()
|
||||
for {
|
||||
if _, _, err := conn.Read(ctx); err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
ping := time.NewTicker(25 * time.Second)
|
||||
defer ping.Stop()
|
||||
// stdout and stderr are two pipes, and what Docker replays down them arrives in no
|
||||
// particular order. So the opening burst is held for a moment and sorted by the time
|
||||
// Docker stamped each line with; after that the log is live and arrives in order anyway.
|
||||
const burstFor = 300 * time.Millisecond
|
||||
warm := time.NewTimer(burstFor)
|
||||
defer warm.Stop()
|
||||
warmC := warm.C // nil once the burst has been flushed: a nil channel never fires
|
||||
var burst []logLine
|
||||
write := func(l logLine) bool {
|
||||
return conn.Write(ctx, websocket.MessageText, mustJSON(l)) == nil
|
||||
}
|
||||
flush := func() bool {
|
||||
sort.SliceStable(burst, func(i, j int) bool { return burst[i].at < burst[j].at })
|
||||
for _, l := range burst {
|
||||
if !write(l) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
burst = nil
|
||||
return true
|
||||
}
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case l, ok := <-lines:
|
||||
if !ok { // the container stopped, or never existed: say the last of it and go
|
||||
flush()
|
||||
return
|
||||
}
|
||||
if warmC != nil {
|
||||
burst = append(burst, l)
|
||||
if len(burst) < 10000 { // a container that never stops talking still gets read
|
||||
continue
|
||||
}
|
||||
if !flush() {
|
||||
return
|
||||
}
|
||||
continue
|
||||
}
|
||||
if !write(l) {
|
||||
return
|
||||
}
|
||||
case <-warmC:
|
||||
warmC = nil
|
||||
if !flush() {
|
||||
return
|
||||
}
|
||||
case <-ping.C:
|
||||
pctx, pc := context.WithTimeout(ctx, 10*time.Second)
|
||||
err := conn.Ping(pctx)
|
||||
pc()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func mustJSON(v any) []byte { b, _ := json.Marshal(v); return b }
|
||||
@@ -0,0 +1,356 @@
|
||||
package api
|
||||
|
||||
// Scheduled image updates.
|
||||
//
|
||||
// Dockhand kept three containers up to date nightly — unbound, nginx and pihole, which is
|
||||
// DNS and ingress — and that is the one thing it does that Ham Lab did not. This replaces
|
||||
// it, and deliberately in the same shape as the ZFS snapshot schedules: a real line in a
|
||||
// cron file Ham Lab owns, calling the binary, rather than a ticker inside the daemon.
|
||||
// Updating an image is not something that should stop happening because a web console is
|
||||
// down, and a cron line is visible to anyone looking at the machine the normal way.
|
||||
//
|
||||
// A schedule stores everything the run needs — project, compose files, how compose has to
|
||||
// be invoked for that stack — so the cron job is self-contained and does not have to
|
||||
// rediscover the world before it can act.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/compose"
|
||||
"hamlab/internal/cron"
|
||||
"hamlab/internal/storage"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// UpdateJob is a standing instruction to pull a stack's images, and optionally deploy them.
|
||||
type UpdateJob struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Stack string `json:"stack"`
|
||||
Machine string `json:"machine,omitempty"` // for display; the job runs where it was made
|
||||
// Services narrows it to some of the stack's services; empty is all of them.
|
||||
Services []string `json:"services,omitempty"`
|
||||
Cron string `json:"cron"`
|
||||
// Apply says whether a newer image is deployed or only fetched. Check-only still pulls,
|
||||
// so the Docker tab shows what is behind without a registry round trip.
|
||||
Apply bool `json:"apply"`
|
||||
Enabled bool `json:"enabled"`
|
||||
|
||||
// How to run compose for this stack, captured when the schedule was made.
|
||||
Spec composeSpec `json:"spec"`
|
||||
}
|
||||
|
||||
type composeSpec struct {
|
||||
Project string `json:"project"`
|
||||
Files []string `json:"files"`
|
||||
Via string `json:"via,omitempty"`
|
||||
// Remote is read but never written: schedules made before v0.2 could point compose at
|
||||
// another machine's Docker over tcp. Such a job is refused rather than run against this
|
||||
// machine's Docker, which is where it would land without the variable.
|
||||
Remote string `json:"remote,omitempty"`
|
||||
}
|
||||
|
||||
func (c composeSpec) toSpec(tool string) compose.Spec {
|
||||
return compose.Spec{Project: c.Project, Files: c.Files, Via: c.Via, Tool: tool}
|
||||
}
|
||||
|
||||
const dockerCronFile = "/etc/cron.d/hamlab-docker"
|
||||
|
||||
type dockerSchedState struct {
|
||||
Jobs []UpdateJob `json:"jobs"`
|
||||
}
|
||||
|
||||
func (s *Server) dsLoad() dockerSchedState {
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.DockerJobs })
|
||||
var d dockerSchedState
|
||||
if len(raw) > 0 {
|
||||
_ = json.Unmarshal(raw, &d)
|
||||
}
|
||||
if d.Jobs == nil {
|
||||
d.Jobs = []UpdateJob{}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func (s *Server) dsSave(d dockerSchedState) error {
|
||||
b, err := json.Marshal(d)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.store.Update(func(st *store.State) { st.DockerJobs = b })
|
||||
}
|
||||
|
||||
// dsCronText renders the schedules as cron lines. Same shape as the ZFS file: Ham Lab owns
|
||||
// it whole and rewrites it on every change.
|
||||
func dsCronText(jobs []UpdateJob, user, exe, config string) (string, error) {
|
||||
var b strings.Builder
|
||||
b.WriteString("# Ham Lab image-update schedules — written by the Docker tab, rewritten whole on every change.\n")
|
||||
b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
|
||||
b.WriteString("SHELL=/bin/sh\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
|
||||
for i := range jobs {
|
||||
j := &jobs[i]
|
||||
if !j.Enabled {
|
||||
b.WriteString("# " + j.Name + " — paused\n")
|
||||
continue
|
||||
}
|
||||
expr := strings.Join(strings.Fields(j.Cron), " ")
|
||||
if err := cron.ValidSchedule(expr); err != nil {
|
||||
return "", err
|
||||
}
|
||||
what := "pull only"
|
||||
if j.Apply {
|
||||
what = "pull and deploy"
|
||||
}
|
||||
b.WriteString("# " + j.Name + " — " + j.Stack + ", " + what + "\n")
|
||||
b.WriteString(expr + " " + user + " " + exe + " -config " + config + " docker-update " + j.ID + "\n\n")
|
||||
}
|
||||
return b.String(), nil
|
||||
}
|
||||
|
||||
func (s *Server) dsSyncCron(ctx context.Context, d dockerSchedState) error {
|
||||
live := false
|
||||
for i := range d.Jobs {
|
||||
if d.Jobs[i].Enabled {
|
||||
live = true
|
||||
}
|
||||
}
|
||||
if !live {
|
||||
return s.storage.RemoveRootFile(ctx, dockerCronFile)
|
||||
}
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
exe, _ = filepath.EvalSymlinks(exe)
|
||||
text, err := dsCronText(d.Jobs, s.user, exe, s.cfg.Path())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.storage.WriteRootFile(ctx, dockerCronFile, text)
|
||||
}
|
||||
|
||||
// repoint keeps a schedule working when what deployed its stack has been removed.
|
||||
//
|
||||
// A schedule captures how to invoke compose at the moment it is made, so the cron job never
|
||||
// has to rediscover the world at 3am. But "run compose inside this container" stops being
|
||||
// true the day that container is deleted — which is exactly what happens when an
|
||||
// orchestrator is retired, and would otherwise break every schedule made before it went.
|
||||
// So: if the container named is not running, the files are looked up again, through live
|
||||
// mounts and remembered ones both, and compose runs on the host instead.
|
||||
func repoint(ctx context.Context, db *store.Store, spec compose.Spec) compose.Spec {
|
||||
if spec.Via == "" || db == nil {
|
||||
return spec
|
||||
}
|
||||
if out, err := dockerRun(ctx, "inspect", "-f", "{{.State.Running}}", spec.Via); err == nil && strings.TrimSpace(string(out)) == "true" {
|
||||
return spec
|
||||
}
|
||||
st := stack{Name: spec.Project, File: spec.Files[0], Files: spec.Files}
|
||||
st.resolve(mountsSeen(ctx, db)) // read-only: this also runs from cron, where the daemon owns state.json
|
||||
if !st.Editable || st.Via != "" {
|
||||
return spec // still only reachable through a container, or not reachable at all
|
||||
}
|
||||
log.Printf("docker schedule %s: %s is gone, running compose here against %s", spec.Project, spec.Via, st.HostFiles[0])
|
||||
spec.Files, spec.Via = st.HostFiles, ""
|
||||
return spec
|
||||
}
|
||||
|
||||
// RunUpdate pulls, and deploys when the schedule says to. Shared with the CLI so the button
|
||||
// and the cron line cannot drift apart.
|
||||
func RunUpdate(ctx context.Context, db *store.Store, j UpdateJob, tool string) (string, error) {
|
||||
if len(j.Spec.Files) == 0 {
|
||||
return "", errors.New("this schedule has no compose file recorded — open it in the Docker tab and save it again")
|
||||
}
|
||||
if j.Spec.Remote != "" {
|
||||
return "", errors.New("this schedule was made to deploy to another machine over tcp, which this console no longer does — make it again on that machine")
|
||||
}
|
||||
spec := repoint(ctx, db, j.Spec.toSpec(tool))
|
||||
var log strings.Builder
|
||||
run := func(args ...string) error {
|
||||
cmd := compose.Cmd(ctx, spec, args...)
|
||||
out, err := cmd.CombinedOutput()
|
||||
log.Write(out)
|
||||
return err
|
||||
}
|
||||
args := append([]string{"pull"}, j.Services...)
|
||||
if err := run(args...); err != nil {
|
||||
return log.String(), err
|
||||
}
|
||||
if !j.Apply {
|
||||
return log.String(), nil
|
||||
}
|
||||
up := append([]string{"up", "-d"}, j.Services...)
|
||||
if err := run(up...); err != nil {
|
||||
return log.String(), err
|
||||
}
|
||||
return log.String(), nil
|
||||
}
|
||||
|
||||
// ---- endpoints ----
|
||||
|
||||
func (s *Server) dsList(w http.ResponseWriter, _ *http.Request) {
|
||||
d := s.dsLoad()
|
||||
now := time.Now()
|
||||
out := make([]map[string]any, 0, len(d.Jobs))
|
||||
for _, j := range d.Jobs {
|
||||
m := map[string]any{"job": j, "last": storage.ReadRun(s.cfg.DataDir(), "docker-"+j.ID)}
|
||||
if j.Enabled {
|
||||
if n, ok := cron.NextRun(strings.Join(strings.Fields(j.Cron), " "), now); ok {
|
||||
m["next"] = n
|
||||
}
|
||||
}
|
||||
out = append(out, m)
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"jobs": out, "file": dockerCronFile})
|
||||
}
|
||||
|
||||
func (s *Server) dsPut(w http.ResponseWriter, r *http.Request) {
|
||||
var j UpdateJob
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 32<<10)).Decode(&j); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
j.Cron = strings.Join(strings.Fields(j.Cron), " ")
|
||||
if err := cron.ValidSchedule(j.Cron); err != nil {
|
||||
jsonErr(w, 400, "schedule: "+err.Error())
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(j.Stack) == "" {
|
||||
jsonErr(w, 400, "which stack?")
|
||||
return
|
||||
}
|
||||
// resolve the stack now, so the cron job never has to
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
stacks, err := s.findStacks(ctx)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
found := false
|
||||
for _, st := range stacks {
|
||||
if st.Name == j.Stack {
|
||||
// seenFiles is the list as compose itself sees it: host paths normally, the
|
||||
// orchestrator container's own paths when compose runs inside one. Getting this
|
||||
// backwards gives a cron line that works from the console and fails at 3am.
|
||||
j.Spec = composeSpec{Project: st.Name, Files: seenFiles(st, -1, ""), Via: st.Via}
|
||||
if len(j.Spec.Files) == 0 {
|
||||
jsonErr(w, 409, "that stack's compose file isn't anywhere this console can reach")
|
||||
return
|
||||
}
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no stack called "+j.Stack+" on this machine")
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(j.Name) == "" {
|
||||
j.Name = j.Stack
|
||||
}
|
||||
d := s.dsLoad()
|
||||
replaced := false
|
||||
for i := range d.Jobs {
|
||||
if d.Jobs[i].ID == j.ID && j.ID != "" {
|
||||
d.Jobs[i] = j
|
||||
replaced = true
|
||||
}
|
||||
}
|
||||
if !replaced {
|
||||
b := make([]byte, 6)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
j.ID = hex.EncodeToString(b)
|
||||
d.Jobs = append(d.Jobs, j)
|
||||
}
|
||||
if err := s.dsSave(d); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.dsSyncCron(r.Context(), d); err != nil {
|
||||
jsonErr(w, 500, "saved, but writing "+dockerCronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker schedule %q saved by %s (%s, %q, %s)", j.Name, clientIP(r), j.Stack, j.Cron, map[bool]string{true: "pull and deploy", false: "pull only"}[j.Apply])
|
||||
jsonOK(w, 200, j)
|
||||
}
|
||||
|
||||
func (s *Server) dsDelete(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
d := s.dsLoad()
|
||||
kept := d.Jobs[:0]
|
||||
found := false
|
||||
for _, j := range d.Jobs {
|
||||
if j.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, j)
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
d.Jobs = kept
|
||||
if err := s.dsSave(d); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.dsSyncCron(r.Context(), d); err != nil {
|
||||
jsonErr(w, 500, "removed, but writing "+dockerCronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
storage.ForgetRun(s.cfg.DataDir(), "docker-"+id)
|
||||
log.Printf("docker schedule %s removed by %s", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) dsRun(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
var job UpdateJob
|
||||
found := false
|
||||
for _, j := range s.dsLoad().Jobs {
|
||||
if j.ID == id {
|
||||
job, found = j, true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
// Outliving the request: a pull the browser gave up on still finishes, but not forever.
|
||||
ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 30*time.Minute)
|
||||
defer cancel()
|
||||
out, err := RunUpdate(ctx, s.store, job, dockerTool())
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: job.Stack}
|
||||
if err != nil {
|
||||
rec.Error = err.Error()
|
||||
}
|
||||
_ = storage.WriteRun(s.cfg.DataDir(), "docker-"+id, rec)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error()+"\n"+tail(out, 2000))
|
||||
return
|
||||
}
|
||||
log.Printf("docker schedule %q run by %s", job.Name, clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"output": tail(out, 4000)})
|
||||
}
|
||||
|
||||
func tail(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return "…" + s[len(s)-n:]
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"hamlab/internal/files"
|
||||
)
|
||||
|
||||
// A file read answers Range requests — what a <video> needs to seek — and HEAD, through
|
||||
// the same handler that streams everything else.
|
||||
func TestFilesReadRanges(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "clip.mp4")
|
||||
data := make([]byte, 10000)
|
||||
for i := range data {
|
||||
data[i] = byte(i % 251)
|
||||
}
|
||||
if err := os.WriteFile(p, data, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{fsLocal: files.Local{}}
|
||||
get := func(method, extra string, hdr map[string]string) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest(method, "/api/files/read?path="+url.QueryEscape(p)+extra, nil)
|
||||
for k, v := range hdr {
|
||||
r.Header.Set(k, v)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
s.filesRead(w, r)
|
||||
return w
|
||||
}
|
||||
w := get("GET", "", nil)
|
||||
if w.Code != 200 || w.Body.Len() != 10000 || w.Header().Get("Content-Type") != "video/mp4" || w.Header().Get("Accept-Ranges") != "bytes" {
|
||||
t.Fatalf("whole: %d %d %q %q", w.Code, w.Body.Len(), w.Header().Get("Content-Type"), w.Header().Get("Accept-Ranges"))
|
||||
}
|
||||
w = get("GET", "", map[string]string{"Range": "bytes=100-199"})
|
||||
if w.Code != 206 || w.Header().Get("Content-Range") != "bytes 100-199/10000" || w.Body.Len() != 100 || w.Body.Bytes()[0] != data[100] {
|
||||
t.Fatalf("range: %d %q %d", w.Code, w.Header().Get("Content-Range"), w.Body.Len())
|
||||
}
|
||||
w = get("GET", "", map[string]string{"Range": "bytes=9990-"})
|
||||
if w.Code != 206 || w.Body.Len() != 10 || w.Body.Bytes()[9] != data[9999] {
|
||||
t.Fatalf("tail range: %d %d", w.Code, w.Body.Len())
|
||||
}
|
||||
w = get("HEAD", "", nil)
|
||||
if w.Code != 200 || w.Body.Len() != 0 || w.Header().Get("Content-Length") != "10000" {
|
||||
t.Fatalf("head: %d %d %q", w.Code, w.Body.Len(), w.Header().Get("Content-Length"))
|
||||
}
|
||||
if w := get("GET", "&max=50", nil); w.Code != 413 {
|
||||
t.Fatalf("the editor's cap: %d", w.Code)
|
||||
}
|
||||
if w := get("GET", "", map[string]string{"Range": "bytes=20000-"}); w.Code != 416 {
|
||||
t.Fatalf("a range past the end: %d", w.Code)
|
||||
}
|
||||
if err := os.Chmod(p, 0); err == nil && os.Geteuid() != 0 {
|
||||
if w := get("GET", "", nil); w.Code != 403 {
|
||||
t.Fatalf("unreadable: %d", w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A write over the cap is refused whole: the file is not truncated first.
|
||||
func TestFilesWriteCap(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "f.txt")
|
||||
if err := os.WriteFile(p, []byte("keep me"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{fsLocal: files.Local{}}
|
||||
body := make([]byte, 8<<20+1)
|
||||
r := httptest.NewRequest("PUT", "/api/files/write?path="+url.QueryEscape(p), bytes.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
s.filesWrite(w, r)
|
||||
if w.Code != 413 {
|
||||
t.Fatalf("got %d", w.Code)
|
||||
}
|
||||
if b, _ := os.ReadFile(p); string(b) != "keep me" {
|
||||
t.Fatalf("the file was touched: %q", b)
|
||||
}
|
||||
r = httptest.NewRequest("PUT", "/api/files/write?path="+url.QueryEscape(p), bytes.NewReader([]byte("new")))
|
||||
w = httptest.NewRecorder()
|
||||
s.filesWrite(w, r)
|
||||
if b, _ := os.ReadFile(p); w.Code != 204 || string(b) != "new" {
|
||||
t.Fatalf("a write within the cap: %d %q", w.Code, b)
|
||||
}
|
||||
}
|
||||
|
||||
// A link to nothing is a 404, not an empty 200.
|
||||
func TestFilesReadDanglingLink(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
l := filepath.Join(dir, "broken")
|
||||
if err := os.Symlink("nowhere", l); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{fsLocal: files.Local{}}
|
||||
r := httptest.NewRequest("GET", "/api/files/read?path="+url.QueryEscape(l), nil)
|
||||
w := httptest.NewRecorder()
|
||||
s.filesRead(w, r)
|
||||
if w.Code != 404 {
|
||||
t.Fatalf("got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// dockerErr tells a thing that isn't there from a daemon that isn't up from anything else.
|
||||
func TestDockerErr(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
msg string
|
||||
want int
|
||||
}{
|
||||
{"error: No such object: zzz", 404}, {"Error response from daemon: No such container: x", 404},
|
||||
{"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?", 503},
|
||||
{"something else entirely", 500},
|
||||
} {
|
||||
w := httptest.NewRecorder()
|
||||
dockerErr(w, errors.New(c.msg))
|
||||
if w.Code != c.want {
|
||||
t.Errorf("%q → %d, want %d", c.msg, w.Code, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
package api
|
||||
|
||||
// Guest logins: a temporary account for handing to somebody you want to show around. It is
|
||||
// an ordinary account with a clock on it — every tab, every action, on this machine and on
|
||||
// every machine this hub manages — that stops working at its expiry (half an hour by
|
||||
// default) or the moment it is revoked.
|
||||
//
|
||||
// Deliberately not a reduced account: Mitchell, 2026-09-27, replacing a read-only version
|
||||
// built the same day. He gives these to people he trusts and revokes them otherwise, and a
|
||||
// console with half its buttons missing is not the console he wanted to show off. What that
|
||||
// means in practice: a guest has the Terminal tab, which is a shell as the user the daemon
|
||||
// runs as — root on every host here — and can make an account or a token of its own that
|
||||
// outlives the guest login.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const (
|
||||
guestPrefix = "guest-"
|
||||
guestTTLDefault = 30 * time.Minute
|
||||
guestTTLMax = 24 * time.Hour
|
||||
guestReapEvery = time.Minute
|
||||
)
|
||||
|
||||
// ---- the accounts themselves ----
|
||||
|
||||
func (s *Server) listGuests(w http.ResponseWriter, r *http.Request) {
|
||||
now := time.Now()
|
||||
counts := s.sessions.Count()
|
||||
out := []map[string]any{}
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, u := range c.Users {
|
||||
if !u.Guest || u.Expired(now) {
|
||||
continue
|
||||
}
|
||||
out = append(out, map[string]any{
|
||||
"name": u.Name, "created": u.Created, "expires": u.Expires,
|
||||
"issued_by": u.IssuedBy, "sessions": counts[u.Name], "used": !u.LastSeen.IsZero(),
|
||||
})
|
||||
}
|
||||
})
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
// addGuest makes a guest login and hands back its password — the only time it is ever
|
||||
// readable, here as everywhere else a password is generated.
|
||||
func (s *Server) addGuest(w http.ResponseWriter, r *http.Request) {
|
||||
me := actorOf(r)
|
||||
var body struct {
|
||||
Minutes int `json:"minutes"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
ttl := guestTTLDefault
|
||||
if body.Minutes > 0 {
|
||||
ttl = time.Duration(body.Minutes) * time.Minute
|
||||
}
|
||||
if ttl > guestTTLMax {
|
||||
jsonErr(w, 400, "a guest login lasts at most 24 hours")
|
||||
return
|
||||
}
|
||||
pw, err := auth.GuestPassword()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
hash, err := auth.HashPassword(pw)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
var name string
|
||||
for try := 0; try < 8 && name == ""; try++ {
|
||||
suffix, err := auth.GuestSuffix()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
candidate := guestPrefix + suffix
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
if _, taken := c.User(candidate); !taken {
|
||||
name = candidate
|
||||
}
|
||||
})
|
||||
}
|
||||
if name == "" {
|
||||
jsonErr(w, 500, "could not find a free guest name")
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
expires := now.Add(ttl)
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
c.Users = append(c.Users, config.User{Name: name, Hash: hash, Created: now, Guest: true, Expires: expires, IssuedBy: me.User})
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("guest login %q issued until %s (by %s)", name, expires.Format(time.RFC3339), clientIP(r))
|
||||
jsonOK(w, 200, map[string]any{"name": name, "password": pw, "expires": expires, "minutes": int(ttl.Minutes())})
|
||||
}
|
||||
|
||||
// revokeGuest ends a guest login now. No password: it only ever takes access away, and the
|
||||
// whole point of the thing is that ending it is one tap.
|
||||
func (s *Server) revokeGuest(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
found := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.Users[:0]
|
||||
for _, u := range c.Users {
|
||||
if u.Name == name && u.Guest {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
c.Users = kept
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such guest login")
|
||||
return
|
||||
}
|
||||
s.forgetGuest(name)
|
||||
log.Printf("guest login %q revoked (by %s)", name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// forgetGuest signs a guest out everywhere and drops the look they left behind.
|
||||
func (s *Server) forgetGuest(name string) {
|
||||
s.sessions.RevokeUser(name)
|
||||
_ = s.store.Update(func(st *store.State) { delete(st.Users, name) })
|
||||
}
|
||||
|
||||
// watchGuests clears out guest logins whose time is up. Expiry is enforced on every request
|
||||
// as well — this is what keeps the config file from collecting them, and what signs a guest
|
||||
// out of a console they left open rather than waiting for their next click.
|
||||
func (s *Server) watchGuests() {
|
||||
for {
|
||||
s.reapGuests()
|
||||
time.Sleep(guestReapEvery)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) reapGuests() {
|
||||
now := time.Now()
|
||||
var gone []string
|
||||
var have bool
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, u := range c.Users {
|
||||
if u.Guest {
|
||||
have = true
|
||||
if u.Expired(now) {
|
||||
gone = append(gone, u.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
if !have || len(gone) == 0 {
|
||||
return
|
||||
}
|
||||
expired := map[string]bool{}
|
||||
for _, n := range gone {
|
||||
expired[n] = true
|
||||
}
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.Users[:0]
|
||||
for _, u := range c.Users {
|
||||
if expired[u.Name] {
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
c.Users = kept
|
||||
}); err != nil {
|
||||
log.Printf("guests: %v", err)
|
||||
return
|
||||
}
|
||||
for _, n := range gone {
|
||||
s.forgetGuest(n)
|
||||
}
|
||||
log.Printf("guest login(s) expired: %s", strings.Join(gone, ", "))
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// A guest login is an ordinary account with a clock on it, so the clock is the thing worth
|
||||
// testing: one whose time is up is stopped at the door, whether or not the minute-by-minute
|
||||
// sweep has reached their account yet, and their session is dropped on the way — so the
|
||||
// console they left open goes back to the login form rather than carrying on.
|
||||
func TestExpiredGuestIsRefusedAndSignedOut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
expires time.Time
|
||||
wantOK bool
|
||||
}{
|
||||
{"still has time", time.Now().Add(10 * time.Minute), true},
|
||||
{"time is up", time.Now().Add(-time.Second), false},
|
||||
{"no expiry at all", time.Time{}, false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cfg := config.Default(filepath.Join(dir, "config.json"))
|
||||
if err := cfg.Update(func(c *config.Config) {
|
||||
c.Users = []config.User{{Name: "guest-abcd", Hash: "x", Guest: true, Expires: tc.expires}}
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err := store.Open(filepath.Join(dir, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{cfg: cfg, store: st, sessions: auth.NewSessions(st)}
|
||||
tok, err := s.sessions.Create("guest-abcd", "10.0.0.5", "curl")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := httptest.NewRequest("GET", "/api/me", nil)
|
||||
r.AddCookie(&http.Cookie{Name: cookieName, Value: tok})
|
||||
w := httptest.NewRecorder()
|
||||
a, ok, answered := s.cookieActor(w, r)
|
||||
if ok != tc.wantOK {
|
||||
t.Fatalf("ok = %v, want %v", ok, tc.wantOK)
|
||||
}
|
||||
if tc.wantOK {
|
||||
if !a.Guest || a.User != "guest-abcd" {
|
||||
t.Errorf("actor = %+v, want the guest account marked as one", a)
|
||||
}
|
||||
if answered {
|
||||
t.Error("answered the caller for a guest that still has time")
|
||||
}
|
||||
return
|
||||
}
|
||||
if !answered || w.Code != 401 {
|
||||
t.Errorf("answered = %v, code = %d, want true and 401", answered, w.Code)
|
||||
}
|
||||
if _, live := s.sessions.Lookup(tok, "10.0.0.5"); live {
|
||||
t.Error("the session of an expired guest is still live")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package api
|
||||
|
||||
// The Hardware tab: what the machine is made of. Read once and remembered — none of it
|
||||
// changes while the kernel runs — with ?refresh=1 for the one thing that can, a drive
|
||||
// plugged in or pulled out.
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
)
|
||||
|
||||
func (s *Server) hardwareReport(w http.ResponseWriter, r *http.Request) {
|
||||
jsonOK(w, 200, s.hardware.Report(r.Context(), r.URL.Query().Get("refresh") == "1"))
|
||||
}
|
||||
@@ -0,0 +1,216 @@
|
||||
package api
|
||||
|
||||
// The ZFS tab's landing-tier checks: what has waited too long on a landing branch, and verify
|
||||
// passes that read the pool's copy of every moved file back against the mover's ledger —
|
||||
// run by hand or on a schedule written to /etc/cron.d/hamlab-landing, just as snapshot
|
||||
// schedules are. A pass runs as its own process (`hamlab landing-verify`), the same one cron
|
||||
// starts, so it survives the console being restarted or upgraded mid-pass.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
func (s *Server) syncVerifyCron(ctx context.Context, z zfsState) error {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
exe, _ = filepath.EvalSymlinks(exe)
|
||||
return s.storage.WriteVerifyCron(ctx, z.Verify, s.user, exe, s.cfg.Path())
|
||||
}
|
||||
|
||||
// startVerify launches a pass in its own session: arg is "new", "all" or a schedule id.
|
||||
func (s *Server) startVerify(arg string) error {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
exe, _ = filepath.EvalSymlinks(exe)
|
||||
dir := filepath.Join(s.cfg.DataDir(), "landing-verify")
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := os.OpenFile(filepath.Join(dir, "last.log"), os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cmd := exec.Command(exe, "-config", s.cfg.Path(), "landing-verify", arg)
|
||||
cmd.Stdout, cmd.Stderr = out, out
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
|
||||
if err := cmd.Start(); err != nil {
|
||||
out.Close()
|
||||
return err
|
||||
}
|
||||
go func() { _ = cmd.Wait(); out.Close() }()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Server) landingOverview(w http.ResponseWriter, r *http.Request) {
|
||||
dir := s.cfg.DataDir()
|
||||
now := time.Now()
|
||||
type tier struct {
|
||||
storage.LandingTarget
|
||||
Cursor int64 `json:"cursor"`
|
||||
StaleCount int `json:"stale_count"`
|
||||
StaleBytes int64 `json:"stale_bytes"`
|
||||
Stale []storage.StaleFile `json:"stale"`
|
||||
}
|
||||
tiers := []tier{}
|
||||
for _, t := range storage.LandingTargets() {
|
||||
files, n, b := storage.StaleFiles(t.Src, 100, now)
|
||||
if files == nil {
|
||||
files = []storage.StaleFile{}
|
||||
}
|
||||
tiers = append(tiers, tier{LandingTarget: t, Cursor: storage.VerifyCursor(dir, t), StaleCount: n, StaleBytes: b, Stale: files})
|
||||
}
|
||||
z := s.zfsLoad()
|
||||
pols := make([]map[string]any, 0, len(z.Verify))
|
||||
for i := range z.Verify {
|
||||
p := z.Verify[i]
|
||||
expr, _ := p.CronExpr()
|
||||
m := map[string]any{"policy": p, "cron": expr, "last": storage.LastVerify(dir, p.ID)}
|
||||
if p.Enabled {
|
||||
if n := p.NextRun(now); !n.IsZero() {
|
||||
m["next"] = n
|
||||
}
|
||||
}
|
||||
pols = append(pols, m)
|
||||
}
|
||||
hist := storage.VerifyHistory(dir)
|
||||
if len(hist) > 10 {
|
||||
hist = hist[:10]
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"tiers": tiers, "stale_after_sec": int(storage.StaleAfter.Seconds()),
|
||||
"running": storage.RunningVerify(dir), "history": hist, "policies": pols,
|
||||
"cron_file": storage.VerifyCronFile,
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) landingVerifyNow(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Scope string `json:"scope"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1024)).Decode(&body); err != nil || (body.Scope != "new" && body.Scope != "all") {
|
||||
jsonErr(w, 400, "scope: new or all")
|
||||
return
|
||||
}
|
||||
if storage.RunningVerify(s.cfg.DataDir()) != nil {
|
||||
jsonErr(w, 409, "a verify pass is already running")
|
||||
return
|
||||
}
|
||||
if len(storage.LandingTargets()) == 0 {
|
||||
jsonErr(w, 409, "no landing mover is reporting on this machine, so there is nothing to verify")
|
||||
return
|
||||
}
|
||||
if err := s.startVerify(body.Scope); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("landing verify (%s) started by %s", body.Scope, clientIP(r))
|
||||
jsonOK(w, 202, map[string]string{"started": body.Scope})
|
||||
}
|
||||
|
||||
func (s *Server) landingPolicyPut(w http.ResponseWriter, r *http.Request) {
|
||||
var p storage.VerifyPolicy
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 16<<10)).Decode(&p); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if err := p.Validate(); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
z := s.zfsLoad()
|
||||
found := false
|
||||
for i := range z.Verify {
|
||||
if p.ID != "" && z.Verify[i].ID == p.ID {
|
||||
z.Verify[i], found = p, true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
if p.ID == "" {
|
||||
id, err := randomID()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
p.ID = id
|
||||
}
|
||||
z.Verify = append(z.Verify, p)
|
||||
}
|
||||
if err := s.zfsSave(z); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.syncVerifyCron(r.Context(), z); err != nil {
|
||||
jsonErr(w, 500, "saved, but writing "+storage.VerifyCronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
expr, _ := p.CronExpr()
|
||||
log.Printf("landing verify schedule %q saved by %s (%q, %s)", p.Name, clientIP(r), expr, p.Scope)
|
||||
jsonOK(w, 200, p)
|
||||
}
|
||||
|
||||
func (s *Server) landingPolicyDelete(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
z := s.zfsLoad()
|
||||
kept := z.Verify[:0]
|
||||
found := false
|
||||
for _, p := range z.Verify {
|
||||
if p.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, p)
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
z.Verify = kept
|
||||
if err := s.zfsSave(z); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.syncVerifyCron(r.Context(), z); err != nil {
|
||||
jsonErr(w, 500, "removed, but writing "+storage.VerifyCronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
storage.ForgetVerify(s.cfg.DataDir(), id)
|
||||
log.Printf("landing verify schedule %s removed by %s", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) landingPolicyRun(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
found := false
|
||||
for _, p := range s.zfsLoad().Verify {
|
||||
found = found || p.ID == id
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
if storage.RunningVerify(s.cfg.DataDir()) != nil {
|
||||
jsonErr(w, 409, "a verify pass is already running")
|
||||
return
|
||||
}
|
||||
if err := s.startVerify(id); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("landing verify schedule %s run by %s", id, clientIP(r))
|
||||
jsonOK(w, 202, map[string]string{"started": id})
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package api
|
||||
|
||||
// The Network tab: the machine's interfaces, how the internet looks from here, and the speed
|
||||
// test. One machine at a time, like every other tab — the test runs on the machine in the
|
||||
// switcher, so the number is that machine's line and not the hub's.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/network"
|
||||
"hamlab/internal/sensors"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// networkOverview is everything the tab draws at once: the interfaces, the outside view, the
|
||||
// last test and the ones before it.
|
||||
func (s *Server) networkOverview(w http.ResponseWriter, _ *http.Request) {
|
||||
run := s.net.Test.State()
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"report": network.Look(s.hostName()),
|
||||
"outside": s.net.Outside(),
|
||||
"run": run,
|
||||
"history": s.net.Test.History(),
|
||||
"method": network.Method,
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) speedtestState(w http.ResponseWriter, _ *http.Request) {
|
||||
jsonOK(w, 200, s.net.Test.State())
|
||||
}
|
||||
|
||||
func (s *Server) speedtestStart(w http.ResponseWriter, r *http.Request) {
|
||||
run, err := s.net.Test.Start()
|
||||
if err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("speed test started on %s (by %s)", s.hostName(), clientIP(r))
|
||||
jsonOK(w, 200, run)
|
||||
}
|
||||
|
||||
func (s *Server) speedtestCancel(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.net.Test.Cancel() {
|
||||
jsonErr(w, 404, "no speed test is running")
|
||||
return
|
||||
}
|
||||
log.Printf("speed test cancelled on %s (by %s)", s.hostName(), clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// ---- keeping the results ----
|
||||
|
||||
// savedNetwork is what survives a restart: the recent results, so the history is a history
|
||||
// rather than whatever has happened since the daemon last started.
|
||||
type savedNetwork struct {
|
||||
Results []network.Result `json:"results,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Server) loadNetwork() []network.Result {
|
||||
var saved savedNetwork
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.Network })
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &saved); err != nil {
|
||||
log.Printf("network: kept results unreadable (%v); starting a fresh history", err)
|
||||
}
|
||||
}
|
||||
return saved.Results
|
||||
}
|
||||
|
||||
// keepResult is called when a test finishes: append it to what is on disk.
|
||||
func (s *Server) keepResult(r network.Result) {
|
||||
if r.Error != "" {
|
||||
log.Printf("speed test on %s failed: %s", s.hostName(), r.Error)
|
||||
} else {
|
||||
log.Printf("speed test on %s: down %s, up %s, latency %.1f ms via %s",
|
||||
s.hostName(), network.Mbits(r.DownMbps), network.Mbits(r.UpMbps), r.LatencyMs, r.Colo)
|
||||
}
|
||||
results := s.net.Test.History()
|
||||
b, err := json.Marshal(savedNetwork{Results: results})
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := s.store.Update(func(st *store.State) { st.Network = b }); err != nil {
|
||||
log.Printf("network: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- the speed test as sensors ----
|
||||
|
||||
// netSensors publishes the last speed test as ordinary sensors, so the line can be a widget,
|
||||
// a graph, an alert or a Prometheus series like anything else Ham Lab measures. Each one's
|
||||
// timestamp is when the test ran, which is how a stale reading says so.
|
||||
type netSensors struct{ w *network.Watcher }
|
||||
|
||||
func (n netSensors) Sensors() []sensors.Sensor {
|
||||
var out []sensors.Sensor
|
||||
if o := n.w.Outside(); o.IP != "" {
|
||||
at := o.At.UnixMilli()
|
||||
out = append(out, sensors.Sensor{ID: "net.internet.ip", Name: "Public address", Kind: "state", Value: o.IP, T: at})
|
||||
if o.Colo != "" {
|
||||
out = append(out, sensors.Sensor{ID: "net.internet.edge", Name: "Internet edge", Kind: "state", Value: o.Colo, T: at})
|
||||
}
|
||||
}
|
||||
run := n.w.Test.State()
|
||||
if run.Result == nil || run.Result.Error != "" {
|
||||
return out
|
||||
}
|
||||
r := run.Result
|
||||
at := r.At.UnixMilli()
|
||||
return append(out,
|
||||
sensors.Sensor{ID: "net.internet.download", Name: "Download", Kind: "rate", Unit: "Mbps", Value: r.DownMbps, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.upload", Name: "Upload", Kind: "rate", Unit: "Mbps", Value: r.UpMbps, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.latency", Name: "Latency", Kind: "duration", Unit: "ms", Value: r.LatencyMs, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.jitter", Name: "Jitter", Kind: "duration", Unit: "ms", Value: r.JitterMs, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.tested", Name: "Speed test age", Kind: "duration", Unit: "s", Value: round1(time.Since(r.At).Seconds()), T: at, Min: sensors.F(0)},
|
||||
)
|
||||
}
|
||||
|
||||
func round1(v float64) float64 { return float64(int64(v*10+0.5)) / 10 }
|
||||
@@ -0,0 +1,563 @@
|
||||
package api
|
||||
|
||||
// Playing a video in the Files viewer, YouTube's way: Auto (the player reads the connection),
|
||||
// any quality by hand, or the file itself. The qualities are made on the fly as HLS by
|
||||
// internal/media, on the device the Transcode device setting names — which may be on another
|
||||
// machine than the file.
|
||||
//
|
||||
// On every machine:
|
||||
// POST /api/files/play[?sudo=1] {path | url, name, device} open a play
|
||||
// GET /api/files/play/{id}/master.m3u8, /{rung}.m3u8, /{rung}/{n}.ts
|
||||
// GET /api/files/play/{id} what it is doing
|
||||
// DELETE /api/files/play/{id} (or POST …/close, for a beacon) end it
|
||||
// GET /api/files/play-devices what it can encode on
|
||||
// On the hub:
|
||||
// POST /api/play {node, path, sudo} open a play of a file on node, wherever the setting
|
||||
// says, answering which machine the play is on
|
||||
// GET|PUT /api/transcode the setting, and every machine's devices
|
||||
// GET /api/play-src/{token} a file, for a machine transcoding it, by a token
|
||||
// that names that one file; ranges pass through
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/files"
|
||||
"hamlab/internal/media"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// playRuns is how many plays may be encoding at once on one machine; a fourth ends the one
|
||||
// asked of least lately. The A380 makes about two 8K-sourced streams in real time.
|
||||
const playRuns = 3
|
||||
|
||||
type playReq struct {
|
||||
Path string `json:"path,omitempty"`
|
||||
URL string `json:"url,omitempty"` // a file on another machine, as the hub serves it
|
||||
Name string `json:"name,omitempty"`
|
||||
Device string `json:"device,omitempty"`
|
||||
}
|
||||
|
||||
type playOut struct {
|
||||
Node string `json:"node"` // the machine the play is on: "" for the hub; filled in by the hub
|
||||
Play *media.Play `json:"play"`
|
||||
Master string `json:"master"` // the playlist, under that machine's /api
|
||||
Engine string `json:"engine"`
|
||||
Host string `json:"host"` // that machine's name, for the menu
|
||||
Note string `json:"note,omitempty"`
|
||||
Status media.PlayStatus `json:"status"`
|
||||
}
|
||||
|
||||
// playDevices: GET /api/files/play-devices — what this machine could make the player's H.264
|
||||
// on, probed as the daemon's own user (a play from another machine never runs as root).
|
||||
func (s *Server) playDevices(w http.ResponseWriter, r *http.Request) {
|
||||
caps, err := s.thumbs.capsFor(r.Context(), false, false)
|
||||
if err != nil {
|
||||
caps.Error = err.Error()
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"host": s.hostName(), "devices": media.Devices(caps)})
|
||||
}
|
||||
|
||||
// playStart: POST /api/files/play[?sudo=1] — open a play of a file here, or of a URL the hub
|
||||
// serves a file on another machine at.
|
||||
func (s *Server) playStart(w http.ResponseWriter, r *http.Request) {
|
||||
var body playReq
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
root := r.URL.Query().Get("sudo") == "1"
|
||||
if root && s.fsSudo == nil {
|
||||
jsonErr(w, 403, "root access needs passwordless sudo on the host")
|
||||
return
|
||||
}
|
||||
out, code, err := s.openPlay(r.Context(), body, root)
|
||||
if err != nil {
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) openPlay(ctx context.Context, body playReq, root bool) (playOut, int, error) {
|
||||
if root && s.fsSudo == nil && body.URL == "" {
|
||||
return playOut{}, 403, errors.New("root access needs passwordless sudo on the host")
|
||||
}
|
||||
in, name := body.Path, body.Name
|
||||
if body.URL != "" {
|
||||
u, err := url.Parse(body.URL)
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || body.Path != "" {
|
||||
return playOut{}, 400, errors.New("a play is of a path or of an http URL")
|
||||
}
|
||||
in, root = body.URL, false
|
||||
} else {
|
||||
p, err := files.Clean(body.Path)
|
||||
if err != nil {
|
||||
return playOut{}, 400, err
|
||||
}
|
||||
if media.Kind(p) != "video" {
|
||||
return playOut{}, 415, errors.New("not a video")
|
||||
}
|
||||
fsys := s.fsLocal
|
||||
if root {
|
||||
fsys = s.fsSudo
|
||||
}
|
||||
if _, err := fsys.Stat(ctx, p); err != nil {
|
||||
return playOut{}, 404, err
|
||||
}
|
||||
in = p
|
||||
if name == "" {
|
||||
name = filepath.Base(p)
|
||||
}
|
||||
}
|
||||
caps, err := s.thumbs.capsFor(ctx, root, false)
|
||||
if err == nil && caps.Error != "" {
|
||||
err = errors.New(caps.Error)
|
||||
}
|
||||
if err != nil {
|
||||
return playOut{}, 503, err
|
||||
}
|
||||
ictx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
src, err := media.Inspect(ictx, caps, in, root)
|
||||
cancel()
|
||||
if err != nil {
|
||||
var de *media.DecodeError
|
||||
if errors.As(err, &de) {
|
||||
return playOut{}, 422, errors.New(de.Msg)
|
||||
}
|
||||
return playOut{}, 504, err
|
||||
}
|
||||
p, err := s.plays.Open(caps, root, in, name, src, body.Device)
|
||||
if err != nil {
|
||||
return playOut{}, 422, err
|
||||
}
|
||||
log.Printf("files: play %s on %s (%d qualities, %s)", name, p.Engine(), len(p.Rungs), map[bool]string{true: "from another machine", false: "here"}[p.Remote])
|
||||
return playOut{Play: p, Master: "/api/files/play/" + p.ID + "/master.m3u8", Engine: p.Engine(), Host: s.hostName(), Status: p.Status()}, 200, nil
|
||||
}
|
||||
|
||||
// playStatus: GET /api/files/play/{id}
|
||||
func (s *Server) playStatus(w http.ResponseWriter, r *http.Request) {
|
||||
p, ok := s.plays.Find(r.PathValue("id"))
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such play")
|
||||
return
|
||||
}
|
||||
jsonOK(w, 200, p.Status())
|
||||
}
|
||||
|
||||
// playStop: DELETE /api/files/play/{id}, or POST …/close — the form a page can send on its
|
||||
// way out (navigator.sendBeacon), so a phone closing the tab releases the machine at once.
|
||||
func (s *Server) playStop(w http.ResponseWriter, r *http.Request) {
|
||||
s.plays.Close(r.PathValue("id"))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// playPlaylist: GET /api/files/play/{id}/{file} — master.m3u8, or one quality's <name>.m3u8.
|
||||
func (s *Server) playPlaylist(w http.ResponseWriter, r *http.Request) {
|
||||
p, ok := s.plays.Find(r.PathValue("id"))
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such play")
|
||||
return
|
||||
}
|
||||
f := r.PathValue("file")
|
||||
var b []byte
|
||||
if f == "master.m3u8" {
|
||||
b = p.Master()
|
||||
} else if name, isList := strings.CutSuffix(f, ".m3u8"); isList {
|
||||
if b, ok = p.Media(name); !ok {
|
||||
jsonErr(w, 404, "no such quality")
|
||||
return
|
||||
}
|
||||
} else {
|
||||
jsonErr(w, 404, "not a play file")
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
_, _ = w.Write(b)
|
||||
}
|
||||
|
||||
// playPiece: GET /api/files/play/{id}/{rung}/{seg} — a piece, made first if need be.
|
||||
func (s *Server) playPiece(w http.ResponseWriter, r *http.Request) {
|
||||
p, ok := s.plays.Find(r.PathValue("id"))
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such play")
|
||||
return
|
||||
}
|
||||
ns, isTS := strings.CutSuffix(r.PathValue("seg"), ".ts")
|
||||
n, err := strconv.Atoi(ns)
|
||||
if !isTS || err != nil || strconv.Itoa(n) != ns {
|
||||
jsonErr(w, 404, "not a piece")
|
||||
return
|
||||
}
|
||||
file, err := p.Segment(r.Context().Done(), r.PathValue("rung"), n)
|
||||
switch {
|
||||
case errors.Is(err, media.ErrNotFound):
|
||||
jsonErr(w, 404, err.Error())
|
||||
return
|
||||
case r.Context().Err() != nil:
|
||||
return // the player moved on
|
||||
case err != nil:
|
||||
jsonErr(w, 503, err.Error())
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "video/mp2t")
|
||||
w.Header().Set("Cache-Control", "private, max-age=3600")
|
||||
http.ServeFile(w, r, file)
|
||||
}
|
||||
|
||||
// ---- on the hub: where plays are made ----
|
||||
|
||||
// transcodeChoice is the setting; nil is the machine the file is on.
|
||||
func (s *Server) transcodeChoice() *store.Transcode {
|
||||
var t *store.Transcode
|
||||
if s.store == nil {
|
||||
return nil
|
||||
}
|
||||
s.store.View(func(st *store.State) {
|
||||
if st.Transcode != nil {
|
||||
c := *st.Transcode
|
||||
t = &c
|
||||
}
|
||||
})
|
||||
return t
|
||||
}
|
||||
|
||||
type transcodeMachine struct {
|
||||
Node string `json:"node"` // "" is the hub
|
||||
Host string `json:"host"`
|
||||
Devices []media.Device `json:"devices"`
|
||||
Error string `json:"error,omitempty"` // the machine didn't answer
|
||||
}
|
||||
|
||||
// getTranscode: GET /api/transcode — the setting, and what every machine can encode on,
|
||||
// asked of each now. ?only=selected asks only the chosen machine, for the device's name.
|
||||
func (s *Server) getTranscode(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Query().Get("only") == "selected" {
|
||||
c := s.transcodeChoice()
|
||||
out := map[string]any{"selected": c}
|
||||
if c != nil {
|
||||
name := c.Device
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
|
||||
defer cancel()
|
||||
var devs []media.Device
|
||||
if c.Node == "" {
|
||||
caps, _ := s.thumbs.capsFor(ctx, false, false)
|
||||
devs = media.Devices(caps)
|
||||
} else if code, b, err := s.nodes.Do(ctx, c.Node, "GET", "/api/files/play-devices", nil, nil); err == nil && code == 200 {
|
||||
var d struct{ Devices []media.Device }
|
||||
_ = json.Unmarshal(b, &d)
|
||||
devs = d.Devices
|
||||
}
|
||||
for _, d := range devs {
|
||||
if d.ID == c.Device {
|
||||
name = d.Name
|
||||
}
|
||||
}
|
||||
host := c.Node
|
||||
if host == "" {
|
||||
host = s.hostName()
|
||||
}
|
||||
out["host"], out["name"] = host, name
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
return
|
||||
}
|
||||
names := []string{""}
|
||||
if s.nodes != nil {
|
||||
names = append(names, s.nodes.Names()...)
|
||||
}
|
||||
out := make([]transcodeMachine, len(names))
|
||||
var wg sync.WaitGroup
|
||||
for i, n := range names {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
m := transcodeMachine{Node: n, Host: n}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 8*time.Second)
|
||||
defer cancel()
|
||||
if n == "" {
|
||||
caps, err := s.thumbs.capsFor(ctx, false, false)
|
||||
if err != nil {
|
||||
caps.Error = err.Error()
|
||||
}
|
||||
m.Host, m.Devices = s.hostName(), media.Devices(caps)
|
||||
} else {
|
||||
code, b, err := s.nodes.Do(ctx, n, "GET", "/api/files/play-devices", nil, nil)
|
||||
var d struct {
|
||||
Host string `json:"host"`
|
||||
Devices []media.Device `json:"devices"`
|
||||
}
|
||||
switch {
|
||||
case err != nil:
|
||||
m.Error = err.Error()
|
||||
case code == 404:
|
||||
m.Error = "runs an older Ham Lab"
|
||||
case code != 200 || json.Unmarshal(b, &d) != nil:
|
||||
m.Error = fmt.Sprintf("answered %d", code)
|
||||
default:
|
||||
m.Devices = d.Devices
|
||||
}
|
||||
}
|
||||
out[i] = m
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
jsonOK(w, 200, map[string]any{"selected": s.transcodeChoice(), "machines": out})
|
||||
}
|
||||
|
||||
// putTranscode: PUT /api/transcode {node, device}, or {} for the machine the file is on.
|
||||
func (s *Server) putTranscode(w http.ResponseWriter, r *http.Request) {
|
||||
var body *store.Transcode
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "expected {\"node\": …, \"device\": …}")
|
||||
return
|
||||
}
|
||||
if body != nil && body.Device == "" {
|
||||
body = nil
|
||||
}
|
||||
if body != nil {
|
||||
known := body.Node == ""
|
||||
if s.nodes != nil {
|
||||
for _, n := range s.nodes.Names() {
|
||||
known = known || n == body.Node
|
||||
}
|
||||
}
|
||||
if !known {
|
||||
jsonErr(w, 400, "no such machine")
|
||||
return
|
||||
}
|
||||
if body.Device != media.CPU && !strings.HasPrefix(body.Device, "/dev/dri/") {
|
||||
jsonErr(w, 400, "a device is a render node or \"cpu\"")
|
||||
return
|
||||
}
|
||||
}
|
||||
if err := s.store.Update(func(st *store.State) { st.Transcode = body }); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
s.getTranscode(w, r)
|
||||
}
|
||||
|
||||
// hubPlay: POST /api/play {node, path, sudo} — open a play of a file on node ("" the hub) on
|
||||
// the device the setting names. When that is on another machine, that machine reads the file
|
||||
// through this hub, by a token good for that one file. If it can't be reached, the file's own
|
||||
// machine makes it instead, and the answer says so.
|
||||
func (s *Server) hubPlay(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Node string `json:"node"`
|
||||
Path string `json:"path"`
|
||||
Sudo bool `json:"sudo"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
p, err := files.Clean(body.Path)
|
||||
if err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
if body.Node != "" && (s.nodes == nil || !has(s.nodes.Names(), body.Node)) {
|
||||
jsonErr(w, 404, "no such machine")
|
||||
return
|
||||
}
|
||||
choice := s.transcodeChoice()
|
||||
var note string
|
||||
if choice != nil && choice.Node != body.Node {
|
||||
if s.nodes != nil && choice.Node != "" && !has(s.nodes.Names(), choice.Node) {
|
||||
note = "The transcode machine " + choice.Node + " is no longer managed here; made where the file is."
|
||||
} else {
|
||||
src, err := s.srcURL(choice.Node, body.Node, p, body.Sudo)
|
||||
if err == nil {
|
||||
req := playReq{URL: src, Name: filepath.Base(p), Device: choice.Device}
|
||||
out, code, err := s.playOn(r.Context(), choice.Node, req, false)
|
||||
if err == nil {
|
||||
jsonOK(w, 200, out)
|
||||
return
|
||||
}
|
||||
if code == 422 || code == 415 {
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
log.Printf("files: play on %s failed (%v); making it where the file is", nodeLabel(choice.Node), err)
|
||||
note = "Couldn't make it on " + nodeLabel(choice.Node) + " (" + err.Error() + "), so it is made where the file is."
|
||||
}
|
||||
}
|
||||
device := ""
|
||||
if choice != nil && choice.Node == body.Node {
|
||||
device = choice.Device
|
||||
}
|
||||
out, code, err := s.playOn(r.Context(), body.Node, playReq{Path: p, Device: device}, body.Sudo)
|
||||
if err != nil {
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
out.Note = note
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func nodeLabel(n string) string {
|
||||
if n == "" {
|
||||
return "the hub"
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func has(list []string, x string) bool {
|
||||
for _, v := range list {
|
||||
if v == x {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// playOn opens a play on a machine: here, or through that node's API.
|
||||
func (s *Server) playOn(ctx context.Context, node string, req playReq, sudo bool) (playOut, int, error) {
|
||||
if node == "" {
|
||||
out, code, err := s.openPlay(ctx, req, sudo)
|
||||
return out, code, err
|
||||
}
|
||||
b, _ := json.Marshal(req)
|
||||
path := "/api/files/play"
|
||||
if sudo {
|
||||
path += "?sudo=1"
|
||||
}
|
||||
code, resp, err := s.nodes.Do(ctx, node, "POST", path, b, http.Header{"Content-Type": []string{"application/json"}})
|
||||
if err != nil {
|
||||
return playOut{}, 502, err
|
||||
}
|
||||
if code != 200 {
|
||||
var e struct{ Error string }
|
||||
_ = json.Unmarshal(resp, &e)
|
||||
if e.Error == "" {
|
||||
e.Error = fmt.Sprintf("%s answered %d", node, code)
|
||||
}
|
||||
return playOut{}, code, errors.New(e.Error)
|
||||
}
|
||||
var out playOut
|
||||
if err := json.Unmarshal(resp, &out); err != nil {
|
||||
return playOut{}, 502, err
|
||||
}
|
||||
out.Node = node
|
||||
return out, 200, nil
|
||||
}
|
||||
|
||||
// ---- the file, for the machine transcoding it ----
|
||||
|
||||
type srcGrant struct {
|
||||
node, path string
|
||||
sudo bool
|
||||
until time.Time
|
||||
}
|
||||
|
||||
type srcGrants struct {
|
||||
mu sync.Mutex
|
||||
m map[string]*srcGrant
|
||||
}
|
||||
|
||||
const srcLife = 12 * time.Hour // from its last use: a film paused overnight still plays
|
||||
|
||||
var grants = &srcGrants{m: map[string]*srcGrant{}}
|
||||
|
||||
func (g *srcGrants) mint(node, path string, sudo bool) string {
|
||||
b := make([]byte, 24)
|
||||
_, _ = rand.Read(b)
|
||||
tok := hex.EncodeToString(b)
|
||||
now := time.Now()
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
for k, v := range g.m {
|
||||
if now.After(v.until) {
|
||||
delete(g.m, k)
|
||||
}
|
||||
}
|
||||
g.m[tok] = &srcGrant{node: node, path: path, sudo: sudo, until: now.Add(srcLife)}
|
||||
return tok
|
||||
}
|
||||
|
||||
func (g *srcGrants) use(tok string) (srcGrant, bool) {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
v, ok := g.m[tok]
|
||||
if !ok || time.Now().After(v.until) {
|
||||
return srcGrant{}, false
|
||||
}
|
||||
v.until = time.Now().Add(srcLife)
|
||||
return *v, true
|
||||
}
|
||||
|
||||
// srcURL is where the machine `to` can read a file on machine `from` through this hub: this
|
||||
// hub's own address as `to` reaches it, and a token for that one file.
|
||||
func (s *Server) srcURL(to, from, path string, sudo bool) (string, error) {
|
||||
host, port, err := net.SplitHostPort(s.cfg.Listen)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
ip := net.ParseIP(host)
|
||||
if host == "" || ip == nil || ip.IsUnspecified() || ip.IsLoopback() {
|
||||
// Listening everywhere: the address this machine reaches `to` from is one it answers on.
|
||||
target := "127.0.0.1:1"
|
||||
if to != "" {
|
||||
if u, err := url.Parse(s.nodes.URL(to)); err == nil && u.Host != "" {
|
||||
target = u.Host
|
||||
}
|
||||
}
|
||||
c, err := net.Dial("udp", target)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
host = c.LocalAddr().(*net.UDPAddr).IP.String()
|
||||
c.Close()
|
||||
}
|
||||
scheme := "http"
|
||||
if s.fingerprint != "" {
|
||||
scheme = "https"
|
||||
}
|
||||
return scheme + "://" + net.JoinHostPort(host, port) + "/api/play-src/" + grants.mint(from, path, sudo), nil
|
||||
}
|
||||
|
||||
// playSrc: GET /api/play-src/{token} — the one file a token names, ranges and all, read from
|
||||
// its machine. The token is the only key: it is minted for a machine transcoding that file.
|
||||
func (s *Server) playSrc(w http.ResponseWriter, r *http.Request) {
|
||||
g, ok := grants.use(r.PathValue("token"))
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such file")
|
||||
return
|
||||
}
|
||||
q := url.Values{"path": {g.path}}
|
||||
if g.sudo {
|
||||
q.Set("sudo", "1")
|
||||
}
|
||||
r2 := r.Clone(r.Context())
|
||||
r2.URL.RawQuery = q.Encode()
|
||||
r2.Header.Del("Cookie")
|
||||
if g.node == "" {
|
||||
r2.URL.Path = "/api/files/read"
|
||||
s.filesRead(w, r2)
|
||||
return
|
||||
}
|
||||
p, ok := s.nodes.Proxy(g.node)
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such machine")
|
||||
return
|
||||
}
|
||||
r2.URL.Path = "/api/nodes/" + g.node + "/files/read"
|
||||
p.ServeHTTP(w, r2)
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/files"
|
||||
)
|
||||
|
||||
// The file a machine transcodes from another is read through the hub by a token naming that
|
||||
// one file: ranges pass through, the query can't point it anywhere else, and an unknown token
|
||||
// is nothing.
|
||||
func TestPlaySrc(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "film.mp4")
|
||||
if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "other.mp4"), []byte("secret"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{fsLocal: files.Local{}, cfg: &config.Config{Listen: "10.0.0.234:8100"}}
|
||||
u, err := s.srcURL("", "", p, false)
|
||||
if err != nil || !strings.HasPrefix(u, "http://10.0.0.234:8100/api/play-src/") {
|
||||
t.Fatalf("url %q %v", u, err)
|
||||
}
|
||||
tok := strings.TrimPrefix(u, "http://10.0.0.234:8100/api/play-src/")
|
||||
get := func(tok, query, rng string) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("GET", "/api/play-src/"+tok+query, nil)
|
||||
r.SetPathValue("token", tok)
|
||||
if rng != "" {
|
||||
r.Header.Set("Range", rng)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
s.playSrc(w, r)
|
||||
return w
|
||||
}
|
||||
if w := get(tok, "", "bytes=2-4"); w.Code != 206 || w.Body.String() != "234" {
|
||||
t.Fatalf("range: %d %q", w.Code, w.Body)
|
||||
}
|
||||
if w := get(tok, "?path="+filepath.Join(dir, "other.mp4"), ""); w.Body.String() != "0123456789" {
|
||||
t.Fatalf("the query moved it: %q", w.Body)
|
||||
}
|
||||
if w := get("nope", "", ""); w.Code != 404 {
|
||||
t.Fatalf("unknown token: %d", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// A preview made on another machine is kept by the file's own: only while the file is still
|
||||
// the one it was made of, and a failure is kept as a failure.
|
||||
func TestThumbPut(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "film.mp4")
|
||||
if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fi, _ := os.Stat(p)
|
||||
s := &Server{fsLocal: files.Local{}, thumbs: newThumbSvc(dir, "")}
|
||||
v := fmt.Sprintf("%d-%d", fi.ModTime().UnixMilli(), fi.Size())
|
||||
put := func(v string, body string, hdr map[string]string) int {
|
||||
r := httptest.NewRequest("PUT", "/api/files/thumb?path="+url.QueryEscape(p)+"&v="+v, strings.NewReader(body))
|
||||
for k, x := range hdr {
|
||||
r.Header.Set(k, x)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
s.thumbPut(w, r)
|
||||
return w.Code
|
||||
}
|
||||
get := func() *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("GET", "/api/files/thumb?cached=1&path="+url.QueryEscape(p)+"&v="+v, nil)
|
||||
w := httptest.NewRecorder()
|
||||
s.filesThumb(w, r)
|
||||
return w
|
||||
}
|
||||
if w := get(); w.Code != 404 {
|
||||
t.Fatalf("cached=1 before anything: %d", w.Code)
|
||||
}
|
||||
if c := put("1-10", "jpeg", nil); c != 409 {
|
||||
t.Fatalf("a changed file: %d", c)
|
||||
}
|
||||
if c := put(v, "jpeg-bytes", map[string]string{"X-Hamlab-Thumb": "grid vaapi:/dev/dri/renderD129"}); c != 204 {
|
||||
t.Fatalf("put: %d", c)
|
||||
}
|
||||
if w := get(); w.Code != 200 || w.Body.String() != "jpeg-bytes" || !strings.Contains(w.Header().Get("X-Hamlab-Thumb"), "grid") {
|
||||
t.Fatalf("kept: %d %q %q", w.Code, w.Body, w.Header().Get("X-Hamlab-Thumb"))
|
||||
}
|
||||
}
|
||||
+2347
-31
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,224 @@
|
||||
package api
|
||||
|
||||
// The Space tab: what is using a disk or a pool. The unit is a whole volume, never a
|
||||
// partition: a disk on its own is scanned through every filesystem it backs, and a ZFS pool
|
||||
// through every mounted dataset, as one thing — its member disks are not scanned apart.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"net/http"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/space"
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
// spaceTarget is one thing a scan can measure: a disk on its own, or a ZFS pool as a whole.
|
||||
// A disk that is a member of a pool is not a target of its own — its share of the pool means
|
||||
// nothing by itself — it is listed under the pool. (A disk with a pool partition *and* a
|
||||
// filesystem of its own, root on one partition and a pool on another, is both.)
|
||||
type spaceTarget struct {
|
||||
ID string `json:"id"` // "nvme0" or "pool:tank"
|
||||
Kind string `json:"kind"` // disk | pool
|
||||
Name string `json:"name"`
|
||||
Model string `json:"model,omitempty"` // disk: its model
|
||||
Layout string `json:"layout,omitempty"` // pool: raidz2, mirror, …
|
||||
Health string `json:"health,omitempty"` // pool: ONLINE, DEGRADED, …
|
||||
Size uint64 `json:"size"`
|
||||
Used uint64 `json:"used,omitempty"` // pool: allocated
|
||||
Disks []string `json:"disks,omitempty"` // pool: its member disks
|
||||
Roots []string `json:"roots"` // the mountpoints a scan walks
|
||||
Scan *space.Scan `json:"scan,omitempty"` // the last scan, if any
|
||||
|
||||
datasets map[string]string // root → ZFS dataset, for the ones that are
|
||||
}
|
||||
|
||||
var layoutRe = regexp.MustCompile(`\b(draid\d*|raidz[123]?|mirror)\b`)
|
||||
|
||||
// spaceTargets lists the disks and pools a scan can measure.
|
||||
func (s *Server) spaceTargets(ctx context.Context) []spaceTarget {
|
||||
devs, _ := storage.Devices(ctx)
|
||||
var out []spaceTarget
|
||||
pools := map[string]*spaceTarget{}
|
||||
var poolOrder []string
|
||||
for _, d := range devs {
|
||||
disk := spaceTarget{ID: d.Name, Kind: "disk", Name: d.Name, Model: d.Model, Size: d.Size, Roots: []string{}}
|
||||
seen := map[string]bool{}
|
||||
inPool := false
|
||||
for _, p := range d.Parts {
|
||||
if p.FSType == "zfs_member" && p.Label != "" {
|
||||
inPool = true
|
||||
pt := pools[p.Label]
|
||||
if pt == nil {
|
||||
pt = &spaceTarget{ID: "pool:" + p.Label, Kind: "pool", Name: p.Label, Roots: []string{}, datasets: map[string]string{}}
|
||||
for _, zm := range zfsMounts(ctx, p.Label) {
|
||||
pt.Roots = append(pt.Roots, zm.mount)
|
||||
pt.datasets[zm.mount] = zm.dataset
|
||||
}
|
||||
pools[p.Label] = pt
|
||||
poolOrder = append(poolOrder, p.Label)
|
||||
}
|
||||
pt.Disks = append(pt.Disks, d.Name)
|
||||
continue
|
||||
}
|
||||
if p.Mount != "" && p.Mount != "[SWAP]" && !seen[p.Mount] {
|
||||
seen[p.Mount] = true
|
||||
disk.Roots = append(disk.Roots, p.Mount)
|
||||
}
|
||||
}
|
||||
if !inPool || len(disk.Roots) > 0 {
|
||||
sort.Strings(disk.Roots)
|
||||
out = append(out, disk)
|
||||
}
|
||||
}
|
||||
if len(pools) > 0 {
|
||||
// size, health and layout from zpool; the vdev tree names the layout
|
||||
for _, p := range s.storage.ZFS(ctx).Pools {
|
||||
if pt := pools[p.Name]; pt != nil {
|
||||
pt.Size, pt.Used, pt.Health = p.Size, p.Alloc, p.Health
|
||||
if m := layoutRe.FindString(p.Config); m != "" {
|
||||
pt.Layout = m
|
||||
} else if len(pt.Disks) == 1 {
|
||||
pt.Layout = "single disk"
|
||||
} else if len(pt.Disks) > 1 {
|
||||
pt.Layout = "stripe"
|
||||
}
|
||||
}
|
||||
}
|
||||
var ps []spaceTarget
|
||||
for _, name := range poolOrder {
|
||||
sort.Strings(pools[name].Roots)
|
||||
ps = append(ps, *pools[name])
|
||||
}
|
||||
out = append(ps, out...) // pools first: they are what a NAS is
|
||||
}
|
||||
if out == nil {
|
||||
out = []spaceTarget{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type zfsMount struct{ mount, dataset string }
|
||||
|
||||
// zfsMounts is every mounted filesystem of a pool, with its dataset.
|
||||
func zfsMounts(ctx context.Context, pool string) []zfsMount {
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-t", "filesystem", "-r", "-o", "name,mountpoint,mounted", pool).Output()
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var mounts []zfsMount
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) == 3 && f[2] == "yes" && strings.HasPrefix(f[1], "/") {
|
||||
mounts = append(mounts, zfsMount{mount: f[1], dataset: f[0]})
|
||||
}
|
||||
}
|
||||
return mounts
|
||||
}
|
||||
|
||||
func (s *Server) spaceOverview(w http.ResponseWriter, r *http.Request) {
|
||||
targets := s.spaceTargets(r.Context())
|
||||
scans := s.space.All()
|
||||
for i := range targets {
|
||||
targets[i].Scan = scans[targets[i].ID]
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"targets": targets})
|
||||
}
|
||||
|
||||
func (s *Server) spaceTarget(w http.ResponseWriter, r *http.Request, id string) (spaceTarget, bool) {
|
||||
for _, t := range s.spaceTargets(r.Context()) {
|
||||
if t.ID == id {
|
||||
return t, true
|
||||
}
|
||||
}
|
||||
jsonErr(w, 404, "no such disk or pool")
|
||||
return spaceTarget{}, false
|
||||
}
|
||||
|
||||
func (s *Server) spaceStatus(w http.ResponseWriter, r *http.Request) {
|
||||
st := s.space.Status(r.PathValue("disk"))
|
||||
if st == nil {
|
||||
jsonErr(w, 404, "not scanned")
|
||||
return
|
||||
}
|
||||
jsonOK(w, 200, st)
|
||||
}
|
||||
|
||||
func (s *Server) spaceScan(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.spaceTarget(w, r, r.PathValue("disk"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
st, err := s.space.Start(t.ID, t.Roots, t.datasets)
|
||||
if err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("space: scan of %s (%s) started by %s", t.ID, strings.Join(t.Roots, " "), clientIP(r))
|
||||
jsonOK(w, 202, st)
|
||||
}
|
||||
|
||||
func (s *Server) spaceForget(w http.ResponseWriter, r *http.Request) {
|
||||
s.space.Forget(r.PathValue("disk"))
|
||||
log.Printf("space: scan of %s forgotten by %s", r.PathValue("disk"), clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) spaceCancel(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.space.Cancel(r.PathValue("disk")) {
|
||||
jsonErr(w, 404, "no scan running")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) spaceTree(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query()
|
||||
path := q.Get("path")
|
||||
if path == "" {
|
||||
path = "/"
|
||||
}
|
||||
depth, _ := strconv.Atoi(q.Get("depth"))
|
||||
if depth < 1 || depth > 32 {
|
||||
depth = 1
|
||||
}
|
||||
limit, _ := strconv.Atoi(q.Get("limit"))
|
||||
if limit < 1 || limit > 2000 {
|
||||
limit = 300
|
||||
}
|
||||
min, _ := strconv.ParseInt(q.Get("min"), 10, 64)
|
||||
if min < 0 {
|
||||
min = 0
|
||||
}
|
||||
t, err := s.space.Tree(r.PathValue("disk"), path, depth, limit, min)
|
||||
if err != nil {
|
||||
jsonErr(w, 404, err.Error())
|
||||
return
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
jsonOK(w, 200, t)
|
||||
}
|
||||
|
||||
func (s *Server) spaceLargest(w http.ResponseWriter, r *http.Request) {
|
||||
n, _ := strconv.Atoi(r.URL.Query().Get("n"))
|
||||
if n < 1 || n > 5000 {
|
||||
n = 500
|
||||
}
|
||||
files, err := s.space.Largest(r.PathValue("disk"), n)
|
||||
if err != nil {
|
||||
jsonErr(w, 404, err.Error())
|
||||
return
|
||||
}
|
||||
if files == nil {
|
||||
files = []space.File{}
|
||||
}
|
||||
jsonOK(w, 200, files)
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,156 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"hamlab/internal/compose"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// A stack deployed by an orchestrator running in a container records the path its compose
|
||||
// file had inside that container. Resolving it has to work both while the orchestrator is
|
||||
// running — compose is then run inside it — and after it has been removed, where the same
|
||||
// mount is only remembered and compose has to run on the host instead.
|
||||
func TestResolveThroughMounts(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "orch", "stacks", "app")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
file := filepath.Join(dir, "compose.yaml")
|
||||
if err := os.WriteFile(file, []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inner := "/app/data/stacks/app/compose.yaml"
|
||||
|
||||
for _, tc := range []struct {
|
||||
name, container, wantVia string
|
||||
}{
|
||||
{"orchestrator running", "orch", "orch"},
|
||||
{"orchestrator removed", "", ""},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
st := stack{Name: "app", File: inner, Files: []string{inner}}
|
||||
st.resolve([]mountMap{{tc.container, "/app/data", filepath.Join(root, "orch")}})
|
||||
if !st.Editable {
|
||||
t.Fatalf("not editable: %s", st.Note)
|
||||
}
|
||||
if st.HostFile != file {
|
||||
t.Errorf("host file = %q, want %q", st.HostFile, file)
|
||||
}
|
||||
if st.Via != tc.wantVia {
|
||||
t.Errorf("via = %q, want %q", st.Via, tc.wantVia)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Nothing explains the path: the stack is listed, but read-only and says why.
|
||||
st := stack{Name: "app", File: inner, Files: []string{inner}}
|
||||
st.resolve(nil)
|
||||
if st.Editable || st.Note == "" {
|
||||
t.Errorf("unresolvable stack should be read-only with a note, got editable=%v note=%q", st.Editable, st.Note)
|
||||
}
|
||||
}
|
||||
|
||||
// Every file of a multi-file stack must resolve through the same place. Redeploying from
|
||||
// the first file alone would silently drop whatever the override set.
|
||||
func TestResolveRefusesSplitStack(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "orch", "stacks", "app")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "compose.yaml"), []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st := stack{Name: "app", File: "/app/data/stacks/app/compose.yaml", Files: []string{
|
||||
"/app/data/stacks/app/compose.yaml",
|
||||
"/app/data/stacks/app/override.yaml", // never written: not on disk
|
||||
}}
|
||||
st.resolve([]mountMap{{"orch", "/app/data", filepath.Join(root, "orch")}})
|
||||
if st.Editable {
|
||||
t.Error("a stack with a file that cannot be found should not be editable")
|
||||
}
|
||||
}
|
||||
|
||||
// A schedule made while an orchestrator was running says "run compose inside that
|
||||
// container". When the container is gone the schedule must still work, against the same
|
||||
// files on the host — otherwise retiring the orchestrator silently breaks every scheduled
|
||||
// update made before it went.
|
||||
func TestRepointSurvivesAMissingContainer(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "orch", "stacks", "app")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
host := filepath.Join(dir, "compose.yaml")
|
||||
if err := os.WriteFile(host, []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := store.Open(filepath.Join(root, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.Update(func(st *store.State) {
|
||||
st.Mounts = []store.RememberedMount{{Container: "gone-orchestrator", Dest: "/app/data", Src: filepath.Join(root, "orch")}}
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
in := compose.Spec{Project: "app", Files: []string{"/app/data/stacks/app/compose.yaml"}, Via: "gone-orchestrator", Tool: "docker"}
|
||||
got := repoint(context.Background(), db, in)
|
||||
if got.Via != "" {
|
||||
t.Errorf("via = %q, want it dropped", got.Via)
|
||||
}
|
||||
if len(got.Files) != 1 || got.Files[0] != host {
|
||||
t.Errorf("files = %v, want [%s]", got.Files, host)
|
||||
}
|
||||
|
||||
// Nothing remembered: the schedule is left exactly as it was rather than pointed at a
|
||||
// guess. It will fail loudly, which is better than deploying the wrong file.
|
||||
empty, err := store.Open(filepath.Join(root, "empty.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := repoint(context.Background(), empty, in); got.Via != in.Via || len(got.Files) != 1 || got.Files[0] != in.Files[0] {
|
||||
t.Errorf("unresolvable spec was changed: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Where new stacks go: Ham Lab's own <data dir>/stacks is always offered, first and as the
|
||||
// default, whether or not anything is in it yet. A folder this machine already keeps stacks
|
||||
// in — two or more deployed from it — is offered too, but never ahead of Ham Lab's own.
|
||||
func TestHomesDefaultToDataDir(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
cfg := config.Default(filepath.Join(root, "data", "config.json"))
|
||||
s := &Server{cfg: cfg}
|
||||
own := filepath.Join(root, "data", "stacks")
|
||||
legacy := filepath.Join(root, "home", "stacks", "box")
|
||||
deployed := func(dir, name string) stack {
|
||||
f := filepath.Join(dir, name, "compose.yaml")
|
||||
return stack{Name: name, Containers: []string{name + "-1"}, Files: []string{f}, File: f, HostFiles: []string{f}, HostFile: f, Editable: true}
|
||||
}
|
||||
|
||||
// Nothing deployed anywhere: one home, Ham Lab's own.
|
||||
homes := s.homesOf(nil)
|
||||
if len(homes) != 1 || homes[0].Dir != own || !homes[0].Default {
|
||||
t.Fatalf("empty machine: homes = %+v, want just %s as default", homes, own)
|
||||
}
|
||||
|
||||
// Two stacks deployed from a pre-existing folder: it is offered, after Ham Lab's own.
|
||||
homes = s.homesOf([]stack{deployed(legacy, "a"), deployed(legacy, "b")})
|
||||
if len(homes) != 2 || homes[0].Dir != own || !homes[0].Default || homes[1].Dir != legacy || homes[1].Default || homes[1].Stacks != 2 {
|
||||
t.Fatalf("legacy folder: homes = %+v, want [%s (default), %s]", homes, own, legacy)
|
||||
}
|
||||
|
||||
// One lone stack elsewhere says nothing about where the next should go; one in Ham Lab's
|
||||
// own folder is already home.
|
||||
homes = s.homesOf([]stack{deployed(filepath.Join(root, "home", "apps"), "orchestrator"), deployed(own, "c")})
|
||||
if len(homes) != 1 || homes[0].Dir != own || homes[0].Stacks != 1 || !homes[0].Default {
|
||||
t.Fatalf("lone stacks: homes = %+v, want just %s", homes, own)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,375 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/sensors"
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
// statusSensors is the sensors.Provider for what isn't a /proc number: failed services,
|
||||
// pending updates, pool health, drive health. Each source costs an exec or several, so
|
||||
// they are read in the background on their own clocks and served from memory; every
|
||||
// sensor's t says how old its reading really is.
|
||||
type statusSensors struct {
|
||||
s *Server
|
||||
mu sync.RWMutex
|
||||
by map[string][]sensors.Sensor // source → its sensors
|
||||
|
||||
cpuPrev map[string]cpuMark // container ID → CPU time at the previous read
|
||||
|
||||
hist map[string]*sensors.Series // numeric sensors only, statusHistory points each
|
||||
acc map[string]*bucket // the point being built for each, when the window is wider than the read rate
|
||||
}
|
||||
|
||||
// statusHistory is how many readings each status sensor keeps: at 15 s that is over an
|
||||
// hour of container memory; at 10 min, two days of drive wear.
|
||||
const statusHistory = 300
|
||||
|
||||
// A point being averaged, the same idea as the sampler's: these sources are read every 15 to
|
||||
// 300 seconds, so a wide window keeps one point per stride rather than one per read.
|
||||
type bucket struct {
|
||||
last int64 // when this sensor's last point was kept
|
||||
sum float64
|
||||
n float64
|
||||
}
|
||||
|
||||
const (
|
||||
servicesEvery = 30 * time.Second
|
||||
containersEvery = 15 * time.Second
|
||||
updatesEvery = 5 * time.Minute
|
||||
zfsEvery = time.Minute
|
||||
smartEvery = 10 * time.Minute // storage.Manager caches SMART that long anyway, and never wakes a sleeping disk
|
||||
)
|
||||
|
||||
func newStatusSensors(s *Server) *statusSensors {
|
||||
p := &statusSensors{s: s, by: map[string][]sensors.Sensor{}, hist: map[string]*sensors.Series{}, acc: map[string]*bucket{}}
|
||||
go p.loop("services", servicesEvery, p.services)
|
||||
go p.loop("containers", containersEvery, p.containers)
|
||||
go p.loop("updates", updatesEvery, p.updates)
|
||||
go p.loop("zfs", zfsEvery, p.zfs)
|
||||
go p.loop("smart", smartEvery, p.smart)
|
||||
return p
|
||||
}
|
||||
|
||||
func (p *statusSensors) loop(name string, every time.Duration, read func(context.Context, int64) []sensors.Sensor) {
|
||||
for {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
got := read(ctx, time.Now().UnixMilli())
|
||||
cancel()
|
||||
p.mu.Lock()
|
||||
p.by[name] = got
|
||||
window := p.s.metrics.Window()
|
||||
stride := window.Milliseconds() / statusHistory
|
||||
live := map[string]bool{}
|
||||
for _, x := range got {
|
||||
v, ok := x.Value.(float64)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
live[x.ID] = true
|
||||
h := p.hist[x.ID]
|
||||
if h == nil {
|
||||
h = &sensors.Series{ID: x.ID, Unit: x.Unit}
|
||||
p.hist[x.ID] = h
|
||||
}
|
||||
b := p.acc[x.ID]
|
||||
if b == nil {
|
||||
b = &bucket{}
|
||||
p.acc[x.ID] = b
|
||||
}
|
||||
b.sum, b.n = b.sum+v, b.n+1
|
||||
// due a stride after the last point kept, with half a read of slack for a late timer
|
||||
if b.last != 0 && x.T-b.last < stride-every.Milliseconds()/2 {
|
||||
continue // still filling this point
|
||||
}
|
||||
h.T, h.V = append(h.T, x.T), append(h.V, b.sum/b.n)
|
||||
*b = bucket{last: x.T}
|
||||
if len(h.T) > statusHistory {
|
||||
h.T, h.V = h.T[len(h.T)-statusHistory:], h.V[len(h.V)-statusHistory:]
|
||||
}
|
||||
if cut := x.T - window.Milliseconds(); h.T[0] < cut { // and nothing older than the window
|
||||
i := 0
|
||||
for i < len(h.T) && h.T[i] < cut {
|
||||
i++
|
||||
}
|
||||
h.T, h.V = h.T[i:], h.V[i:]
|
||||
}
|
||||
}
|
||||
for id := range p.hist { // a container that is gone takes its history with it
|
||||
if strings.HasPrefix(id, prefixOf[name]) && !live[id] && prefixOf[name] != "" {
|
||||
delete(p.hist, id)
|
||||
delete(p.acc, id)
|
||||
}
|
||||
}
|
||||
p.mu.Unlock()
|
||||
if len(got) == 0 { // no pools, no Docker, no SMART: look again, but rarely
|
||||
time.Sleep(10 * every)
|
||||
continue
|
||||
}
|
||||
time.Sleep(every)
|
||||
}
|
||||
}
|
||||
|
||||
// prefixOf says which IDs a source owns, so its loop forgets only its own departed sensors.
|
||||
var prefixOf = map[string]string{"containers": "container.", "smart": "smart.", "zfs": "zfs."}
|
||||
|
||||
// SensorHistories implements sensors.Historian.
|
||||
func (p *statusSensors) SensorHistories(match func(id string) bool) []sensors.Series {
|
||||
p.mu.RLock()
|
||||
defer p.mu.RUnlock()
|
||||
var out []sensors.Series
|
||||
for id, h := range p.hist {
|
||||
if match(id) {
|
||||
out = append(out, sensors.Series{ID: h.ID, Unit: h.Unit, T: append([]int64{}, h.T...), V: append([]float64{}, h.V...)})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (p *statusSensors) Sensors() []sensors.Sensor {
|
||||
p.mu.RLock()
|
||||
defer p.mu.RUnlock()
|
||||
var out []sensors.Sensor
|
||||
for _, v := range p.by {
|
||||
out = append(out, v...)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (p *statusSensors) services(ctx context.Context, t int64) []sensors.Sensor {
|
||||
units, err := p.s.svc.List(ctx)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var failed, running float64
|
||||
var names []string
|
||||
var per []sensors.Sensor
|
||||
for _, u := range units {
|
||||
if u.Failed {
|
||||
failed++
|
||||
names = append(names, u.Name)
|
||||
}
|
||||
if u.Sub == "running" {
|
||||
running++
|
||||
}
|
||||
// One pair per service that is meant to be doing something: enabled, up, or broken.
|
||||
// (Every static and one-shot unit on the box would triple the catalog for nothing.)
|
||||
short, isService := strings.CutSuffix(u.Name, ".service")
|
||||
if !isService || !(u.Enabled == "enabled" || u.Active == "active" || u.Failed) {
|
||||
continue
|
||||
}
|
||||
id, l := "service."+sensors.Seg(short), map[string]string{"unit": u.Name, "description": u.Description}
|
||||
per = append(per,
|
||||
sensors.Sensor{ID: id + ".active", Name: short, Kind: "state", Value: u.Active == "active", T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".state", Name: short + " state", Kind: "state", Value: u.Sub, T: t, Labels: l})
|
||||
}
|
||||
return append(per, []sensors.Sensor{
|
||||
{ID: "services.failed", Name: "Failed services", Kind: "count", Value: failed, T: t, Warn: sensors.F(1)},
|
||||
{ID: "services.failed_names", Name: "Failed services", Kind: "state", Value: strings.Join(names, ", "), T: t},
|
||||
{ID: "services.running", Name: "Running services", Kind: "count", Value: running, T: t},
|
||||
}...)
|
||||
}
|
||||
|
||||
// containers reads Docker (or Podman) with one `ps`, then each running container's memory
|
||||
// and CPU straight from its cgroup — `docker stats` takes seconds and a daemon round-trip
|
||||
// per call; two small files per container take microseconds.
|
||||
func (p *statusSensors) containers(ctx context.Context, t int64) []sensors.Sensor {
|
||||
var out []byte
|
||||
var err error
|
||||
tool := ""
|
||||
for _, t := range []string{"docker", "podman"} {
|
||||
if _, e := exec.LookPath(t); e != nil {
|
||||
continue
|
||||
}
|
||||
tool = t
|
||||
args := []string{tool, "ps", "-a", "--no-trunc", "--format", "{{json .}}"}
|
||||
if out, err = exec.CommandContext(ctx, args[0], args[1:]...).Output(); err != nil && priv.Available() {
|
||||
full := priv.Wrap(args...) // not in the docker group: ask as root
|
||||
out, err = exec.CommandContext(ctx, full[0], full[1:]...).Output()
|
||||
}
|
||||
break
|
||||
}
|
||||
if err != nil || len(out) == 0 {
|
||||
return nil
|
||||
}
|
||||
// "Not running" is not "down": servarr stopped by hand in June is off, not broken. What is
|
||||
// broken is a container that should be up and isn't — one inspect of the stopped ones tells
|
||||
// them apart: Docker refused to start it (State.Error: yesterday's Gitea), it is stuck
|
||||
// restarting, or it died with a non-zero code that no restart policy picked up. Under
|
||||
// `always` / `unless-stopped` a crash is restarted by Docker, so "exited" there means a
|
||||
// person stopped it. With no restart policy, `docker stop` leaves 143 (the process took the
|
||||
// SIGTERM), 137 (it didn't, and Docker killed it after the grace period) or 130 (SIGINT):
|
||||
// also a person, unless the kernel did the killing for want of memory.
|
||||
type stopped struct {
|
||||
Name string
|
||||
State struct {
|
||||
Status string
|
||||
ExitCode int
|
||||
Error string
|
||||
OOMKilled bool
|
||||
}
|
||||
HostConfig struct{ RestartPolicy struct{ Name string } }
|
||||
}
|
||||
down := map[string]bool{}
|
||||
var ids []string
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
var c struct{ ID, State string }
|
||||
if json.Unmarshal([]byte(line), &c) == nil && c.ID != "" && c.State != "running" {
|
||||
ids = append(ids, c.ID)
|
||||
}
|
||||
}
|
||||
if len(ids) > 0 {
|
||||
args := append([]string{"inspect"}, ids...)
|
||||
insp, ierr := exec.CommandContext(ctx, tool, args...).Output()
|
||||
if ierr != nil && priv.Available() {
|
||||
full := priv.Wrap(append([]string{tool}, args...)...)
|
||||
insp, _ = exec.CommandContext(ctx, full[0], full[1:]...).Output()
|
||||
}
|
||||
var list []stopped
|
||||
if json.Unmarshal(insp, &list) == nil {
|
||||
for _, c := range list {
|
||||
pol := c.HostConfig.RestartPolicy.Name
|
||||
managed := pol == "always" || pol == "unless-stopped"
|
||||
code := c.State.ExitCode
|
||||
byHand := (code == 130 || code == 137 || code == 143) && !c.State.OOMKilled
|
||||
down[strings.TrimPrefix(c.Name, "/")] = c.State.Status == "restarting" || c.State.Status == "dead" || c.State.Error != "" ||
|
||||
(!managed && c.State.Status == "exited" && code != 0 && !byHand)
|
||||
}
|
||||
}
|
||||
}
|
||||
var res []sensors.Sensor
|
||||
var running, unhealthy, total float64
|
||||
seenCPU := map[string]cpuMark{}
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
var c struct{ ID, Names, Image, State, Status string }
|
||||
if json.Unmarshal([]byte(line), &c) != nil || c.Names == "" {
|
||||
continue
|
||||
}
|
||||
total++
|
||||
name := strings.Split(c.Names, ",")[0]
|
||||
id, l := "container."+sensors.Seg(name), map[string]string{"container": name, "image": c.Image}
|
||||
up := c.State == "running"
|
||||
health := "none"
|
||||
for _, h := range []string{"unhealthy", "healthy", "starting"} { // "unhealthy" first: it contains "healthy"
|
||||
if strings.Contains(c.Status, "("+h+")") || strings.Contains(c.Status, "(health: "+h+")") {
|
||||
health = h
|
||||
break
|
||||
}
|
||||
}
|
||||
if up {
|
||||
running++
|
||||
}
|
||||
if health == "unhealthy" {
|
||||
unhealthy++
|
||||
}
|
||||
res = append(res,
|
||||
sensors.Sensor{ID: id + ".running", Name: name, Kind: "state", Value: up, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".down", Name: name + " down", Kind: "state", Value: down[name], T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".state", Name: name + " state", Kind: "state", Value: c.State, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".health", Name: name + " health", Kind: "state", Value: health, T: t, Labels: l})
|
||||
if !up {
|
||||
continue
|
||||
}
|
||||
cg := ""
|
||||
for _, dir := range []string{"/sys/fs/cgroup/system.slice/docker-" + c.ID + ".scope", "/sys/fs/cgroup/docker/" + c.ID, "/sys/fs/cgroup/machine.slice/libpod-" + c.ID + ".scope"} {
|
||||
if _, e := os.Stat(dir + "/memory.current"); e == nil {
|
||||
cg = dir
|
||||
break
|
||||
}
|
||||
}
|
||||
if cg == "" {
|
||||
continue
|
||||
}
|
||||
if b, e := os.ReadFile(cg + "/memory.current"); e == nil {
|
||||
if v, e := strconv.ParseFloat(strings.TrimSpace(string(b)), 64); e == nil {
|
||||
res = append(res, sensors.Sensor{ID: id + ".mem", Name: name + " memory", Kind: "bytes", Unit: "B", Value: v, T: t, Labels: l})
|
||||
}
|
||||
}
|
||||
if b, e := os.ReadFile(cg + "/cpu.stat"); e == nil {
|
||||
for _, ln := range strings.Split(string(b), "\n") {
|
||||
if f := strings.Fields(ln); len(f) == 2 && f[0] == "usage_usec" {
|
||||
usec, _ := strconv.ParseFloat(f[1], 64)
|
||||
seenCPU[c.ID] = cpuMark{usec, t}
|
||||
if prev, ok := p.cpuPrev[c.ID]; ok && t > prev.t && usec >= prev.usec {
|
||||
// percent of one core, like the Processes tab
|
||||
pctv := (usec - prev.usec) / 1000 / float64(t-prev.t) * 100
|
||||
res = append(res, sensors.Sensor{ID: id + ".cpu", Name: name + " CPU", Kind: "usage", Unit: "%", Value: float64(int(pctv*10+0.5)) / 10, T: t, Min: sensors.F(0), Labels: l})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
p.cpuPrev = seenCPU // only this loop touches it
|
||||
return append(res,
|
||||
sensors.Sensor{ID: "containers.total", Name: "Containers", Kind: "count", Value: total, T: t},
|
||||
sensors.Sensor{ID: "containers.running", Name: "Running containers", Kind: "count", Value: running, T: t},
|
||||
sensors.Sensor{ID: "containers.unhealthy", Name: "Unhealthy containers", Kind: "count", Value: unhealthy, T: t, Warn: sensors.F(1)})
|
||||
}
|
||||
|
||||
type cpuMark struct {
|
||||
usec float64
|
||||
t int64
|
||||
}
|
||||
|
||||
func (p *statusSensors) updates(ctx context.Context, t int64) []sensors.Sensor {
|
||||
st := p.s.updates.Status(ctx)
|
||||
if st.Manager == "" {
|
||||
return nil
|
||||
}
|
||||
return []sensors.Sensor{
|
||||
{ID: "updates.pending", Name: "Pending updates", Kind: "count", Value: float64(len(st.Packages)), T: t, Labels: map[string]string{"manager": st.Manager}},
|
||||
{ID: "updates.security", Name: "Pending security updates", Kind: "count", Value: float64(st.SecurityCount), T: t, Warn: sensors.F(1)},
|
||||
{ID: "updates.reboot_required", Name: "Reboot required", Kind: "state", Value: st.RebootNeeded, T: t},
|
||||
}
|
||||
}
|
||||
|
||||
func (p *statusSensors) zfs(ctx context.Context, t int64) []sensors.Sensor {
|
||||
var out []sensors.Sensor
|
||||
for _, pool := range p.s.storage.Pools(ctx) {
|
||||
id, l := "zfs."+sensors.Seg(pool.Name), map[string]string{"pool": pool.Name}
|
||||
out = append(out,
|
||||
sensors.Sensor{ID: id + ".health", Name: pool.Name + " health", Kind: "state", Value: pool.Health, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".healthy", Name: pool.Name + " healthy", Kind: "state", Value: pool.Health == "ONLINE", T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".size", Name: pool.Name + " size", Kind: "bytes", Unit: "B", Value: float64(pool.Size), T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".used", Name: pool.Name + " used", Kind: "bytes", Unit: "B", Value: float64(pool.Alloc), T: t, Min: sensors.F(0), Max: sensors.F(float64(pool.Size)), Labels: l},
|
||||
sensors.Sensor{ID: id + ".used_pct", Name: pool.Name + " used", Kind: "usage", Unit: "%", Value: float64(pool.Cap), T: t, Min: sensors.F(0), Max: sensors.F(100), Labels: l},
|
||||
sensors.Sensor{ID: id + ".frag", Name: pool.Name + " fragmentation", Kind: "fragmentation", Unit: "%", Value: float64(pool.Frag), T: t, Min: sensors.F(0), Max: sensors.F(100), Labels: l},
|
||||
)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (p *statusSensors) smart(ctx context.Context, t int64) []sensors.Sensor {
|
||||
devs, err := storage.Devices(ctx)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []sensors.Sensor
|
||||
for _, d := range devs {
|
||||
sm := p.s.storage.Smart(ctx, d.Name, false)
|
||||
if sm.Passed == nil || sm.Standby || sm.Error != "" {
|
||||
continue // no verdict to report: no smartctl, asleep, or not a SMART device
|
||||
}
|
||||
id, l := "smart."+sensors.Seg(d.Name), map[string]string{"device": d.Name, "model": d.Model}
|
||||
out = append(out,
|
||||
sensors.Sensor{ID: id + ".passed", Name: d.Name + " health", Kind: "state", Value: *sm.Passed, T: sm.FetchedAt, Labels: l},
|
||||
sensors.Sensor{ID: id + ".warnings", Name: d.Name + " warnings", Kind: "count", Value: float64(len(sm.Warnings)), T: sm.FetchedAt, Warn: sensors.F(1), Labels: l},
|
||||
)
|
||||
if sm.PowerOnHours > 0 {
|
||||
out = append(out, sensors.Sensor{ID: id + ".power_on", Name: d.Name + " powered on", Kind: "duration", Unit: "s", Value: float64(sm.PowerOnHours) * 3600, T: sm.FetchedAt, Labels: l})
|
||||
}
|
||||
if sm.NVMe != nil {
|
||||
out = append(out, sensors.Sensor{ID: id + ".wear", Name: d.Name + " endurance used", Kind: "usage", Unit: "%", Value: float64(sm.NVMe.PercentageUsed), T: sm.FetchedAt, Min: sensors.F(0), Max: sensors.F(100), Labels: l})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package api
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestStreamGate(t *testing.T) {
|
||||
var g streamGate
|
||||
for i := 0; i < maxStreamsPerClient; i++ {
|
||||
if !g.enter("a") {
|
||||
t.Fatalf("stream %d for one client refused early", i)
|
||||
}
|
||||
}
|
||||
if g.enter("a") {
|
||||
t.Fatal("one client went past its ceiling")
|
||||
}
|
||||
if !g.enter("b") {
|
||||
t.Fatal("another client must still get in")
|
||||
}
|
||||
g.leave("a")
|
||||
if !g.enter("a") {
|
||||
t.Fatal("a closed stream must free its place")
|
||||
}
|
||||
for i := 0; i < maxStreamsPerClient+1; i++ {
|
||||
g.leave("a")
|
||||
}
|
||||
g.leave("b")
|
||||
if g.total != -1 && g.total != 0 { // leave() past zero must not wedge the gate open or shut
|
||||
t.Logf("total after over-leave: %d", g.total)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,650 @@
|
||||
package api
|
||||
|
||||
// Previews for the Files tab: GET /api/files/thumb hands back one small JPEG of an image or
|
||||
// a video, made by internal/thumbs and kept in <data dir>/thumbs.db. The work is bounded
|
||||
// two ways — a handful of ffmpegs at once, and one per file however many rows ask for it —
|
||||
// because a folder of a thousand photos asks for every row the moment previews go on.
|
||||
//
|
||||
// Root mode (?sudo=1) makes the preview through the same root helper the file reads use,
|
||||
// so a file only root can read gets a preview too, and the probe runs as root as well: root
|
||||
// opens every render node, whatever groups the daemon's user has.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/media"
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const thumbSize = 256
|
||||
const thumbTimeout = 30 * time.Second
|
||||
const capsTTL = 10 * time.Minute // how long a probe is believed; a refresh after an install or a group change does not wait
|
||||
|
||||
type thumbSvc struct {
|
||||
cache *media.Cache
|
||||
self string
|
||||
sem chan struct{}
|
||||
mu sync.Mutex
|
||||
caps map[bool]*capsEntry // keyed by root
|
||||
calls map[string]*thumbCall
|
||||
}
|
||||
|
||||
type capsEntry struct {
|
||||
mu sync.Mutex
|
||||
caps media.Caps
|
||||
err error
|
||||
at time.Time
|
||||
}
|
||||
|
||||
type thumbCall struct {
|
||||
done chan struct{}
|
||||
res media.Result
|
||||
err error
|
||||
}
|
||||
|
||||
func newThumbSvc(dataDir, self string) *thumbSvc {
|
||||
n := runtime.NumCPU() / 2 // each ffmpeg is itself threaded; previews must not take the whole machine
|
||||
if n < 2 {
|
||||
n = 2
|
||||
}
|
||||
if n > 4 {
|
||||
n = 4
|
||||
}
|
||||
return &thumbSvc{cache: media.NewCache(filepath.Join(dataDir, "thumbs.db")), self: self, sem: make(chan struct{}, n), caps: map[bool]*capsEntry{}, calls: map[string]*thumbCall{}}
|
||||
}
|
||||
|
||||
// capsFor is what previews can be made with, probed once and kept for a while.
|
||||
func (t *thumbSvc) capsFor(ctx context.Context, root, refresh bool) (media.Caps, error) {
|
||||
t.mu.Lock()
|
||||
e := t.caps[root]
|
||||
if e == nil {
|
||||
e = &capsEntry{}
|
||||
t.caps[root] = e
|
||||
}
|
||||
t.mu.Unlock()
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
if !refresh && !e.at.IsZero() && time.Since(e.at) < capsTTL {
|
||||
return e.caps, e.err
|
||||
}
|
||||
if root {
|
||||
e.caps, e.err = media.ProbeRoot(ctx, t.self)
|
||||
} else {
|
||||
e.caps, e.err = media.Probe(ctx), nil
|
||||
}
|
||||
e.at = time.Now()
|
||||
mode := "as " + map[bool]string{false: "the daemon", true: "root"}[root]
|
||||
switch {
|
||||
case e.err != nil:
|
||||
log.Printf("thumbs: probe %s failed: %v", mode, e.err)
|
||||
case e.caps.Error != "":
|
||||
log.Printf("thumbs: %s: %s", mode, e.caps.Error)
|
||||
default:
|
||||
log.Printf("thumbs: %s: ffmpeg %s, %s", mode, e.caps.Version, describeCaps(e.caps))
|
||||
}
|
||||
return e.caps, e.err
|
||||
}
|
||||
|
||||
// describeCaps is the one line that says how previews get made here.
|
||||
func describeCaps(c media.Caps) string {
|
||||
if len(c.GPUs) == 0 {
|
||||
return "CPU only"
|
||||
}
|
||||
parts := make([]string, 0, len(c.GPUs))
|
||||
for _, g := range c.GPUs {
|
||||
name := g.Name
|
||||
if name == "" {
|
||||
name = g.Device
|
||||
}
|
||||
p := name + " (" + g.API
|
||||
if len(g.Codecs) > 0 {
|
||||
p += ": " + strings.Join(g.Codecs, ", ")
|
||||
}
|
||||
parts = append(parts, p+")")
|
||||
}
|
||||
return "GPU " + strings.Join(parts, ", ")
|
||||
}
|
||||
|
||||
// make produces a preview, sharing the work when the same file is asked for twice at once.
|
||||
// in is a path, or a URL the hub serves another machine's file at (kind then says what it
|
||||
// is); device is the one to decode on ("" any, "cpu" none).
|
||||
func (t *thumbSvc) make(ctx context.Context, root bool, key, in, kind, device string) (media.Result, error) {
|
||||
t.mu.Lock()
|
||||
if c, ok := t.calls[key]; ok {
|
||||
t.mu.Unlock()
|
||||
select {
|
||||
case <-c.done:
|
||||
return c.res, c.err
|
||||
case <-ctx.Done():
|
||||
return media.Result{}, ctx.Err()
|
||||
}
|
||||
}
|
||||
c := &thumbCall{done: make(chan struct{})}
|
||||
t.calls[key] = c
|
||||
t.mu.Unlock()
|
||||
defer func() {
|
||||
t.mu.Lock()
|
||||
delete(t.calls, key)
|
||||
t.mu.Unlock()
|
||||
close(c.done)
|
||||
}()
|
||||
// The work itself runs on a context of its own: the first requester going away must not
|
||||
// cancel a preview three other rows are waiting for.
|
||||
wctx, cancel := context.WithTimeout(context.Background(), thumbTimeout+5*time.Second)
|
||||
defer cancel()
|
||||
select {
|
||||
case t.sem <- struct{}{}:
|
||||
defer func() { <-t.sem }()
|
||||
case <-ctx.Done():
|
||||
c.err = ctx.Err()
|
||||
return c.res, c.err
|
||||
}
|
||||
caps, err := t.capsFor(wctx, root, false)
|
||||
if err != nil {
|
||||
c.err = err
|
||||
return c.res, c.err
|
||||
}
|
||||
o := media.Opts{Size: thumbSize, Timeout: thumbTimeout, Kind: kind}
|
||||
caps = caps.Only(device)
|
||||
if root {
|
||||
c.res, c.err = media.MakeRoot(wctx, t.self, caps, in, o)
|
||||
} else {
|
||||
c.res, c.err = media.Make(wctx, caps, in, o)
|
||||
}
|
||||
return c.res, c.err
|
||||
}
|
||||
|
||||
// ---- handlers ----
|
||||
|
||||
// thumbDeviceHeader carries the device a hub wants a video preview made on: the Transcode
|
||||
// device setting, when it is on the machine the file is on.
|
||||
const thumbDeviceHeader = "X-Hamlab-Device"
|
||||
|
||||
// filesThumb: GET /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1][&cached=1]. v is the
|
||||
// client's idea of the file, from its listing; it makes a cache hit free of a stat, and a
|
||||
// changed file a new URL. cached=1 answers from the cache or 404, and makes nothing: the hub
|
||||
// asks that before having the preview made on another machine.
|
||||
//
|
||||
// A video preview is made on the Transcode device: on the hub, by the setting (another
|
||||
// machine's device reads the file through the hub); on a node, by the hub's header.
|
||||
func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
|
||||
fsys, ok := s.fsFor(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
p, ok := cleanQuery(w, r, "path")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if media.Kind(p) == "" {
|
||||
jsonErr(w, 415, "no preview for this kind of file")
|
||||
return
|
||||
}
|
||||
root := r.URL.Query().Get("sudo") == "1"
|
||||
if mt, sz, ok := parseV(r.URL.Query().Get("v")); ok {
|
||||
key := media.Key(p, sz, mt)
|
||||
if r.Header.Get("If-None-Match") == `"`+key+`"` {
|
||||
w.WriteHeader(304)
|
||||
return
|
||||
}
|
||||
if s.serveCachedThumb(w, key) {
|
||||
return
|
||||
}
|
||||
}
|
||||
in, err := fsys.Stat(r.Context(), p)
|
||||
if err != nil {
|
||||
fsErr(w, err)
|
||||
return
|
||||
}
|
||||
if in.Type == "dir" || in.Type == "other" || in.Broken {
|
||||
jsonErr(w, 400, "not a regular file")
|
||||
return
|
||||
}
|
||||
key := media.Key(p, in.Size, in.Mtime)
|
||||
if s.serveCachedThumb(w, key) {
|
||||
return
|
||||
}
|
||||
if r.URL.Query().Get("cached") == "1" {
|
||||
jsonErr(w, 404, "not made yet")
|
||||
return
|
||||
}
|
||||
device := r.Header.Get(thumbDeviceHeader)
|
||||
if c := s.transcodeChoice(); c != nil && device == "" && media.Kind(p) == "video" {
|
||||
if c.Node != "" {
|
||||
// The hub's own file, previewed on another machine.
|
||||
if res, ok := s.thumbElsewhere(r.Context(), c, "", p, root, key); ok {
|
||||
if res.Failure != "" {
|
||||
_ = s.thumbs.cache.Put(key, media.Entry{Path: p, Failure: res.Failure})
|
||||
jsonErr(w, 422, res.Failure)
|
||||
return
|
||||
}
|
||||
s.keepThumb(w, key, p, res)
|
||||
return
|
||||
}
|
||||
} else {
|
||||
device = c.Device
|
||||
}
|
||||
}
|
||||
res, err := s.thumbs.make(r.Context(), root, key, p, "", device)
|
||||
if err != nil {
|
||||
var de *media.DecodeError
|
||||
switch {
|
||||
case errors.As(err, &de):
|
||||
if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Failure: de.Msg}); err != nil {
|
||||
log.Printf("thumbs: cache: %v", err)
|
||||
}
|
||||
log.Printf("thumbs: %s: %s", p, de.Msg)
|
||||
jsonErr(w, 422, de.Msg)
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
log.Printf("thumbs: %s: timed out", p)
|
||||
jsonErr(w, 504, "preview timed out")
|
||||
case errors.Is(err, context.Canceled):
|
||||
jsonErr(w, 499, "cancelled")
|
||||
default:
|
||||
jsonErr(w, 503, err.Error())
|
||||
}
|
||||
return
|
||||
}
|
||||
s.keepThumb(w, key, p, res)
|
||||
}
|
||||
|
||||
// keepThumb puts a preview in the cache and answers with it.
|
||||
func (s *Server) keepThumb(w http.ResponseWriter, key, p string, res media.Result) {
|
||||
if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}); err != nil {
|
||||
log.Printf("thumbs: cache: %v", err)
|
||||
}
|
||||
writeThumb(w, key, res.JPEG, res.Engine)
|
||||
}
|
||||
|
||||
// ---- previews made on another machine (the Transcode device) ----
|
||||
|
||||
// thumbFrom: POST /api/files/thumb-from {url, kind, key, device} — a preview of a file on
|
||||
// another machine, read through the hub at url, made here on device. Answers the JPEG (the
|
||||
// engine in X-Hamlab-Thumb), or 422 with why the file can't be previewed.
|
||||
func (s *Server) thumbFrom(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ URL, Kind, Key, Device string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 8192)).Decode(&body); err != nil || !strings.HasPrefix(body.URL, "http") || body.Key == "" {
|
||||
jsonErr(w, 400, "expected {url, kind, key, device}")
|
||||
return
|
||||
}
|
||||
res, err := s.thumbs.make(r.Context(), false, "from:"+body.Key, body.URL, body.Kind, body.Device)
|
||||
if err != nil {
|
||||
var de *media.DecodeError
|
||||
if errors.As(err, &de) {
|
||||
jsonErr(w, 422, de.Msg)
|
||||
} else {
|
||||
jsonErr(w, 503, err.Error())
|
||||
}
|
||||
return
|
||||
}
|
||||
w.Header().Set("X-Hamlab-Thumb-Codec", res.Codec)
|
||||
w.Header().Set("X-Hamlab-Thumb-Size", fmt.Sprintf("%dx%d", res.Width, res.Height))
|
||||
writeThumb(w, body.Key, res.JPEG, res.Engine)
|
||||
}
|
||||
|
||||
// thumbPut: PUT /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1] — a preview another machine
|
||||
// made of this file, kept here as if made here. The body is the JPEG, or nothing with
|
||||
// X-Hamlab-Thumb-Failure saying why the file can't be previewed. Refused if the file has
|
||||
// changed since v.
|
||||
func (s *Server) thumbPut(w http.ResponseWriter, r *http.Request) {
|
||||
fsys, ok := s.fsFor(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
p, ok := cleanQuery(w, r, "path")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
mt, sz, ok := parseV(r.URL.Query().Get("v"))
|
||||
if !ok {
|
||||
jsonErr(w, 400, "v=<mtime>-<size> is required")
|
||||
return
|
||||
}
|
||||
in, err := fsys.Stat(r.Context(), p)
|
||||
if err != nil {
|
||||
fsErr(w, err)
|
||||
return
|
||||
}
|
||||
if in.Size != sz || in.Mtime != mt {
|
||||
jsonErr(w, 409, "the file has changed")
|
||||
return
|
||||
}
|
||||
key := media.Key(p, sz, mt)
|
||||
e := media.Entry{Path: p, Engine: r.Header.Get("X-Hamlab-Thumb"), Codec: r.Header.Get("X-Hamlab-Thumb-Codec"), Failure: r.Header.Get("X-Hamlab-Thumb-Failure")}
|
||||
fmt.Sscanf(r.Header.Get("X-Hamlab-Thumb-Size"), "%dx%d", &e.Width, &e.Height)
|
||||
if e.Failure == "" {
|
||||
if e.JPEG, err = io.ReadAll(io.LimitReader(r.Body, 4<<20)); err != nil || len(e.JPEG) == 0 {
|
||||
jsonErr(w, 400, "no picture")
|
||||
return
|
||||
}
|
||||
}
|
||||
if err := s.thumbs.cache.Put(key, e); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// thumbElsewhere has a video preview made on the Transcode device's machine c.Node, of a
|
||||
// file on machine node ("" the hub), and answers it. ok is false when that machine can't
|
||||
// (unreachable, an older Ham Lab): the caller makes it where the file is instead. A file
|
||||
// that can't be previewed is a result too, with the failure in it.
|
||||
func (s *Server) thumbElsewhere(ctx context.Context, c *store.Transcode, node, p string, root bool, key string) (media.Result, bool) {
|
||||
src, err := s.srcURL(c.Node, node, p, root)
|
||||
if err != nil {
|
||||
return media.Result{}, false
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"url": src, "kind": "video", "key": key, "device": c.Device})
|
||||
ctx, cancel := context.WithTimeout(ctx, thumbTimeout+10*time.Second)
|
||||
defer cancel()
|
||||
var code int
|
||||
var b []byte
|
||||
var hdr http.Header
|
||||
if c.Node == "" {
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest("POST", "/api/files/thumb-from", bytes.NewReader(body)).WithContext(ctx)
|
||||
s.thumbFrom(rec, req)
|
||||
code, b, hdr = rec.Code, rec.Body.Bytes(), rec.Header()
|
||||
} else {
|
||||
code, b, hdr, err = s.nodes.DoHeaders(ctx, c.Node, "POST", "/api/files/thumb-from", body, http.Header{"Content-Type": []string{"application/json"}})
|
||||
if err != nil {
|
||||
log.Printf("thumbs: %s on %s: %v; made where it is", p, c.Node, err)
|
||||
return media.Result{}, false
|
||||
}
|
||||
}
|
||||
switch code {
|
||||
case 200:
|
||||
res := media.Result{JPEG: b, Engine: c.Node + " " + hdr.Get("X-Hamlab-Thumb"), Codec: hdr.Get("X-Hamlab-Thumb-Codec")}
|
||||
fmt.Sscanf(hdr.Get("X-Hamlab-Thumb-Size"), "%dx%d", &res.Width, &res.Height)
|
||||
if c.Node == "" {
|
||||
res.Engine = s.hostName() + " " + hdr.Get("X-Hamlab-Thumb")
|
||||
}
|
||||
return res, true
|
||||
case 422:
|
||||
var e struct{ Error string }
|
||||
_ = json.Unmarshal(b, &e)
|
||||
return media.Result{Failure: e.Error}, true
|
||||
}
|
||||
log.Printf("thumbs: %s on %s answered %d; made where it is", p, nodeLabel(c.Node), code)
|
||||
return media.Result{}, false
|
||||
}
|
||||
|
||||
// hubThumb: GET /api/nodes/{name}/files/thumb — a node's preview, through the hub. Without a
|
||||
// Transcode device setting, or for an image or a sound file's cover, it goes to the node as
|
||||
// any request does. A video is made on the setting's device: the node's own (told by header),
|
||||
// or another machine's, which reads the file through the hub; the node keeps the result in
|
||||
// its own cache either way.
|
||||
func (s *Server) hubThumb(w http.ResponseWriter, r *http.Request) {
|
||||
node := r.PathValue("name")
|
||||
proxy, ok := s.nodes.Proxy(node)
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such machine")
|
||||
return
|
||||
}
|
||||
c := s.transcodeChoice()
|
||||
q := r.URL.Query()
|
||||
p := q.Get("path")
|
||||
if c == nil || media.Kind(p) != "video" {
|
||||
proxy.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
if c.Node == node {
|
||||
r.Header.Set(thumbDeviceHeader, c.Device)
|
||||
proxy.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
mt, sz, okV := parseV(q.Get("v"))
|
||||
if !okV {
|
||||
proxy.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
// The node's own cache first.
|
||||
cq := url.Values{"path": {p}, "v": {q.Get("v")}, "cached": {"1"}}
|
||||
if q.Get("sudo") == "1" {
|
||||
cq.Set("sudo", "1")
|
||||
}
|
||||
hdr := http.Header{}
|
||||
if inm := r.Header.Get("If-None-Match"); inm != "" {
|
||||
hdr.Set("If-None-Match", inm)
|
||||
}
|
||||
code, b, rh, err := s.nodes.DoHeaders(r.Context(), node, "GET", "/api/files/thumb?"+cq.Encode(), nil, hdr)
|
||||
if err == nil && code != 404 {
|
||||
for _, k := range []string{"Content-Type", "Cache-Control", "ETag", "X-Hamlab-Thumb"} {
|
||||
if v := rh.Get(k); v != "" {
|
||||
w.Header().Set(k, v)
|
||||
}
|
||||
}
|
||||
w.WriteHeader(code)
|
||||
_, _ = w.Write(b)
|
||||
return
|
||||
}
|
||||
key := media.Key(p, sz, mt)
|
||||
res, made := s.thumbElsewhere(r.Context(), c, node, p, q.Get("sudo") == "1", key)
|
||||
if !made {
|
||||
proxy.ServeHTTP(w, r) // the node makes it itself
|
||||
return
|
||||
}
|
||||
// Kept where the file is, so it is made once.
|
||||
ph := http.Header{"X-Hamlab-Thumb": {res.Engine}, "X-Hamlab-Thumb-Codec": {res.Codec}, "X-Hamlab-Thumb-Size": {fmt.Sprintf("%dx%d", res.Width, res.Height)}}
|
||||
if res.Failure != "" {
|
||||
ph.Set("X-Hamlab-Thumb-Failure", res.Failure)
|
||||
}
|
||||
pq := url.Values{"path": {p}, "v": {q.Get("v")}}
|
||||
if q.Get("sudo") == "1" {
|
||||
pq.Set("sudo", "1")
|
||||
}
|
||||
if code, _, _, err := s.nodes.DoHeaders(r.Context(), node, "PUT", "/api/files/thumb?"+pq.Encode(), res.JPEG, ph); err != nil || code != 204 {
|
||||
log.Printf("thumbs: keeping %s's preview of %s: %d %v", node, p, code, err)
|
||||
}
|
||||
if res.Failure != "" {
|
||||
jsonErr(w, 422, res.Failure)
|
||||
return
|
||||
}
|
||||
writeThumb(w, key, res.JPEG, res.Engine)
|
||||
}
|
||||
|
||||
// serveCachedThumb answers from the cache when it can. A cache that cannot be read is a
|
||||
// miss: the preview is made again rather than refused.
|
||||
func (s *Server) serveCachedThumb(w http.ResponseWriter, key string) bool {
|
||||
e, found, err := s.thumbs.cache.Get(key)
|
||||
if err != nil {
|
||||
log.Printf("thumbs: cache: %v", err)
|
||||
}
|
||||
switch {
|
||||
case !found:
|
||||
return false
|
||||
case e.Failure != "":
|
||||
jsonErr(w, 422, e.Failure)
|
||||
default:
|
||||
writeThumb(w, key, e.JPEG, "cache ("+e.Engine+")")
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func writeThumb(w http.ResponseWriter, key string, jpeg []byte, engine string) {
|
||||
w.Header().Set("Content-Type", "image/jpeg")
|
||||
w.Header().Set("Content-Length", strconv.Itoa(len(jpeg)))
|
||||
w.Header().Set("Cache-Control", "private, max-age=604800")
|
||||
w.Header().Set("ETag", `"`+key+`"`)
|
||||
w.Header().Set("X-Hamlab-Thumb", engine)
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
_, _ = w.Write(jpeg)
|
||||
}
|
||||
|
||||
func parseV(v string) (mtime, size int64, ok bool) {
|
||||
a, b, found := strings.Cut(v, "-")
|
||||
if !found {
|
||||
return 0, 0, false
|
||||
}
|
||||
mtime, err1 := strconv.ParseInt(a, 10, 64)
|
||||
size, err2 := strconv.ParseInt(b, 10, 64)
|
||||
return mtime, size, err1 == nil && err2 == nil
|
||||
}
|
||||
|
||||
type thumbStatus struct {
|
||||
OK bool `json:"ok"`
|
||||
Engine string `json:"engine"` // gpu | cpu | ""
|
||||
Summary string `json:"summary"`
|
||||
FFmpeg string `json:"ffmpeg,omitempty"`
|
||||
GPUs []media.GPU `json:"gpus"`
|
||||
Unusable []media.GPU `json:"unusable,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Install *thumbInstall `json:"install,omitempty"` // ffmpeg is missing and this can put it on
|
||||
GPUFix bool `json:"gpu_fix,omitempty"` // a render node is there but closed to the daemon's user, and this can open it
|
||||
User string `json:"user"`
|
||||
Cache thumbCache `json:"cache"`
|
||||
}
|
||||
|
||||
type thumbCache struct {
|
||||
Path string `json:"path"`
|
||||
Entries int64 `json:"entries"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
|
||||
type thumbInstall struct {
|
||||
PM string `json:"pm"`
|
||||
Package string `json:"package"`
|
||||
}
|
||||
|
||||
// thumbsStatus: GET /api/files/thumbs[?sudo=1][&refresh=1] — how previews get made on this machine.
|
||||
func (s *Server) thumbsStatus(w http.ResponseWriter, r *http.Request) {
|
||||
root := r.URL.Query().Get("sudo") == "1"
|
||||
if root && s.fsSudo == nil {
|
||||
jsonErr(w, 403, "root access needs passwordless sudo on the host")
|
||||
return
|
||||
}
|
||||
caps, err := s.thumbs.capsFor(r.Context(), root, r.URL.Query().Get("refresh") == "1")
|
||||
st := thumbStatus{GPUs: []media.GPU{}, User: s.user, Cache: thumbCache{Path: s.thumbs.cache.Path}}
|
||||
st.Cache.Entries, st.Cache.Bytes = s.thumbs.cache.Stats()
|
||||
if root {
|
||||
st.User = "root"
|
||||
}
|
||||
if err != nil {
|
||||
st.Error = err.Error()
|
||||
jsonOK(w, 200, st)
|
||||
return
|
||||
}
|
||||
st.FFmpeg, st.GPUs, st.Unusable, st.Error = caps.Version, caps.GPUs, caps.Unusable, caps.Error
|
||||
if st.GPUs == nil {
|
||||
st.GPUs = []media.GPU{}
|
||||
}
|
||||
st.OK = caps.Error == ""
|
||||
switch {
|
||||
case !st.OK:
|
||||
if pkg := ffmpegPackage(s.updates.PM()); pkg != "" && priv.Available() {
|
||||
st.Install = &thumbInstall{PM: s.updates.PM(), Package: pkg}
|
||||
}
|
||||
st.Summary = st.Error
|
||||
case len(caps.GPUs) > 0:
|
||||
st.Engine = "gpu"
|
||||
st.Summary = describeCaps(caps) + " · images on the CPU"
|
||||
default:
|
||||
st.Engine = "cpu"
|
||||
st.Summary = "CPU only"
|
||||
}
|
||||
if !root {
|
||||
for _, g := range caps.Unusable {
|
||||
if g.Permission && gpuFixPossible() {
|
||||
st.GPUFix = true
|
||||
}
|
||||
}
|
||||
}
|
||||
jsonOK(w, 200, st)
|
||||
}
|
||||
|
||||
// ffmpegPackage is what to install for ffmpeg on a package manager. Fedora ships it as
|
||||
// ffmpeg-free; the others under its own name.
|
||||
func ffmpegPackage(pm string) string {
|
||||
switch pm {
|
||||
case "":
|
||||
return ""
|
||||
case "dnf":
|
||||
if b, _ := os.ReadFile("/etc/os-release"); strings.Contains(string(b), "ID=fedora") {
|
||||
return "ffmpeg-free"
|
||||
}
|
||||
}
|
||||
return "ffmpeg"
|
||||
}
|
||||
|
||||
const hamlabUnit = "/etc/systemd/system/hamlab.service"
|
||||
|
||||
// gpuFixPossible: the daemon is an ordinary user under systemd, with root to hand.
|
||||
func gpuFixPossible() bool {
|
||||
if os.Geteuid() == 0 || !priv.Available() {
|
||||
return false
|
||||
}
|
||||
_, err := os.Stat(hamlabUnit)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// thumbsInstall: POST /api/files/thumbs/install — put ffmpeg on with the machine's package manager.
|
||||
func (s *Server) thumbsInstall(w http.ResponseWriter, r *http.Request) {
|
||||
pkg := ffmpegPackage(s.updates.PM())
|
||||
if pkg == "" {
|
||||
jsonErr(w, 400, "no supported package manager on this machine")
|
||||
return
|
||||
}
|
||||
t, err := s.updates.Install([]string{pkg})
|
||||
if err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("thumbs: installing %s as task %s (from %s)", pkg, t.ID, clientIP(r))
|
||||
jsonOK(w, 202, t.Snapshot(false))
|
||||
}
|
||||
|
||||
// gpuAccessScript gives hamlab.service the groups that own the render nodes, by a drop-in,
|
||||
// then restarts it: a running process cannot gain a group.
|
||||
const gpuAccessScript = `set -e
|
||||
unit=` + hamlabUnit + `
|
||||
[ -f "$unit" ] || { echo "no $unit: this machine does not run Ham Lab under systemd"; exit 1; }
|
||||
groups=""
|
||||
for g in render video; do
|
||||
if getent group "$g" >/dev/null 2>&1; then groups="$groups $g"; fi
|
||||
done
|
||||
[ -n "$groups" ] || { echo "neither a render nor a video group exists here"; exit 1; }
|
||||
mkdir -p "$unit.d"
|
||||
printf '[Service]\nSupplementaryGroups=%s\n' "${groups# }" > "$unit.d/gpu.conf"
|
||||
echo "hamlab.service: supplementary groups${groups} ($unit.d/gpu.conf)"
|
||||
systemctl daemon-reload
|
||||
echo 'Ham Lab restarts in 3 seconds to pick up its new groups; the page reconnects by itself.'
|
||||
systemd-run --quiet --on-active=3 --unit=hamlab-gpu-restart systemctl restart hamlab.service
|
||||
`
|
||||
|
||||
// thumbsGPUAccess: POST /api/files/thumbs/gpu-access — open the render nodes to the daemon.
|
||||
func (s *Server) thumbsGPUAccess(w http.ResponseWriter, r *http.Request) {
|
||||
if !gpuFixPossible() {
|
||||
jsonErr(w, 400, "nothing to do: Ham Lab is root, has no root, or is not under systemd here")
|
||||
return
|
||||
}
|
||||
if t, busy := s.tasks.RunningOf("thumbs:gpu"); busy { // a second run would only collide with the first's restart timer
|
||||
jsonErr(w, 409, "already being done (task "+t.ID+")")
|
||||
return
|
||||
}
|
||||
full := priv.Wrap("sh", "-c", gpuAccessScript)
|
||||
t, err := s.tasks.Start("thumbs:gpu", "give Ham Lab access to the GPU", exec.Command(full[0], full[1:]...))
|
||||
if err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("thumbs: GPU access drop-in as task %s, restart follows (from %s)", t.ID, clientIP(r))
|
||||
jsonOK(w, 202, t.Snapshot(false))
|
||||
}
|
||||
@@ -0,0 +1,217 @@
|
||||
package api
|
||||
|
||||
// Updating Ham Lab itself: is there a newer build published, and install it on request.
|
||||
// The check is a plain HTTP GET against whatever `update.url` names — this project's Gitea
|
||||
// releases by default — so a fleet, a stranger's install and an air-gapped copy pointed at
|
||||
// a directory on a NAS all work the same way. Nothing is ever installed unasked: an update
|
||||
// restarts the daemon and drops open terminals, which is not a thing to do to somebody.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/selfupdate"
|
||||
)
|
||||
|
||||
type updateState struct {
|
||||
mu sync.Mutex
|
||||
checking bool
|
||||
installing bool
|
||||
checkedAt time.Time
|
||||
latest selfupdate.Release
|
||||
err string
|
||||
}
|
||||
|
||||
// source is the update source as configured, and whether looking is switched off.
|
||||
func (s *Server) source() (selfupdate.Source, bool) {
|
||||
var u config.UpdateSource
|
||||
s.cfg.View(func(c *config.Config) { u = c.Updates })
|
||||
url := strings.TrimSpace(u.URL)
|
||||
if url == "" {
|
||||
url = selfupdate.DefaultURL
|
||||
}
|
||||
if strings.EqualFold(url, "off") || strings.EqualFold(url, "none") {
|
||||
return selfupdate.Source{}, false
|
||||
}
|
||||
return selfupdate.Source{URL: url, Token: u.Token}, true
|
||||
}
|
||||
|
||||
func (s *Server) checksEnabled() bool {
|
||||
var u config.UpdateSource
|
||||
s.cfg.View(func(c *config.Config) { u = c.Updates })
|
||||
if strings.EqualFold(u.Check, "never") {
|
||||
return false
|
||||
}
|
||||
_, on := s.source()
|
||||
return on
|
||||
}
|
||||
|
||||
// Behind reports whether a newer build than this one has been seen, and which.
|
||||
func (s *Server) behind() (string, bool) {
|
||||
s.update.mu.Lock()
|
||||
defer s.update.mu.Unlock()
|
||||
v := s.update.latest.Version
|
||||
if v == "" || selfupdate.Compare(s.version, v) >= 0 {
|
||||
return "", false
|
||||
}
|
||||
return v, true
|
||||
}
|
||||
|
||||
// checkNow asks the source what the newest build is. One at a time.
|
||||
func (s *Server) checkNow(ctx context.Context) {
|
||||
src, on := s.source()
|
||||
if !on {
|
||||
return
|
||||
}
|
||||
s.update.mu.Lock()
|
||||
if s.update.checking {
|
||||
s.update.mu.Unlock()
|
||||
return
|
||||
}
|
||||
s.update.checking = true
|
||||
s.update.mu.Unlock()
|
||||
ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
|
||||
defer cancel()
|
||||
rel, err := src.Latest(ctx, runtime.GOARCH)
|
||||
s.update.mu.Lock()
|
||||
s.update.checking = false
|
||||
s.update.checkedAt = time.Now()
|
||||
if err != nil {
|
||||
s.update.err = err.Error()
|
||||
} else {
|
||||
s.update.err, s.update.latest = "", rel
|
||||
}
|
||||
s.update.mu.Unlock()
|
||||
if err != nil {
|
||||
log.Printf("update check: %v", err)
|
||||
return
|
||||
}
|
||||
if selfupdate.Compare(s.version, rel.Version) < 0 {
|
||||
log.Printf("update check: %s is published; this is %s", rel.Version, s.version)
|
||||
}
|
||||
}
|
||||
|
||||
// watchForUpdates asks once shortly after boot and then daily.
|
||||
func (s *Server) watchForUpdates(ctx context.Context) {
|
||||
t := time.NewTimer(90 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
if s.checksEnabled() {
|
||||
s.checkNow(ctx)
|
||||
}
|
||||
t.Reset(selfupdate.DefaultCheckEvery)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) updateStatus(w http.ResponseWriter, _ *http.Request) {
|
||||
src, on := s.source()
|
||||
s.update.mu.Lock()
|
||||
defer s.update.mu.Unlock()
|
||||
latest := s.update.latest.Version
|
||||
out := map[string]any{
|
||||
"current": s.version, "latest": latest, "arch": runtime.GOARCH,
|
||||
"behind": latest != "" && selfupdate.Compare(s.version, latest) < 0,
|
||||
"checking": s.update.checking,
|
||||
"installing": s.update.installing,
|
||||
"enabled": on, "auto": s.checksEnabled(), "url": src.URL,
|
||||
"notes": s.update.latest.Notes, "published": s.update.latest.Published,
|
||||
}
|
||||
if !s.update.checkedAt.IsZero() {
|
||||
out["checked"] = s.update.checkedAt
|
||||
}
|
||||
if s.update.err != "" {
|
||||
out["error"] = s.update.err
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) updateCheck(w http.ResponseWriter, r *http.Request) {
|
||||
if _, on := s.source(); !on {
|
||||
jsonErr(w, 400, "update checks are switched off (update.url is \"off\")")
|
||||
return
|
||||
}
|
||||
s.checkNow(r.Context())
|
||||
s.updateStatus(w, r)
|
||||
}
|
||||
|
||||
// updateInstall fetches the published build for this machine and puts it in place.
|
||||
func (s *Server) updateInstall(w http.ResponseWriter, r *http.Request) {
|
||||
src, on := s.source()
|
||||
if !on {
|
||||
jsonErr(w, 400, "update checks are switched off (update.url is \"off\")")
|
||||
return
|
||||
}
|
||||
force := r.URL.Query().Get("force") == "1"
|
||||
if err := s.updateAllowed(force); err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
Version string `json:"version"` // optional: refuse if what is published is not this
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
|
||||
s.update.mu.Lock()
|
||||
if s.update.installing {
|
||||
s.update.mu.Unlock()
|
||||
jsonErr(w, 409, "an update is already being installed")
|
||||
return
|
||||
}
|
||||
rel := s.update.latest
|
||||
s.update.installing = true
|
||||
s.update.mu.Unlock()
|
||||
done := func() { s.update.mu.Lock(); s.update.installing = false; s.update.mu.Unlock() }
|
||||
|
||||
ctx := context.WithoutCancel(r.Context()) // a closed browser tab must not abort an install
|
||||
if rel.Version == "" || body.Version != "" && body.Version != rel.Version {
|
||||
var err error
|
||||
if rel, err = src.Latest(ctx, runtime.GOARCH); err != nil {
|
||||
done()
|
||||
jsonErr(w, 502, err.Error())
|
||||
return
|
||||
}
|
||||
s.update.mu.Lock()
|
||||
s.update.latest, s.update.checkedAt, s.update.err = rel, time.Now(), ""
|
||||
s.update.mu.Unlock()
|
||||
}
|
||||
if body.Version != "" && body.Version != rel.Version {
|
||||
done()
|
||||
jsonErr(w, 409, "what is published is now "+rel.Version+", not "+body.Version+"; look again")
|
||||
return
|
||||
}
|
||||
if !force && selfupdate.Compare(s.version, rel.Version) >= 0 {
|
||||
done()
|
||||
jsonErr(w, 409, "this machine is already running "+s.version+"; nothing newer is published")
|
||||
return
|
||||
}
|
||||
tmp := filepath.Join(s.cfg.DataDir(), "hamlab.new")
|
||||
if err := src.Fetch(ctx, rel, runtime.GOARCH, tmp); err != nil {
|
||||
done()
|
||||
jsonErr(w, 502, err.Error())
|
||||
return
|
||||
}
|
||||
newVersion, err := s.installBinary(ctx, tmp)
|
||||
if err != nil {
|
||||
done()
|
||||
jsonErr(w, codeOf(err, 500), err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("update: installed %s (was %s), asked for by %s; restarting", newVersion, s.version, clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"version": newVersion, "previous": s.version})
|
||||
s.restartAfterUpdate()
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakeDocker puts a `docker` on PATH whose `image inspect` answers with the given RepoDigests.
|
||||
func fakeDocker(t *testing.T, digests string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
script := "#!/bin/sh\necho '" + digests + "'\n"
|
||||
if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(script), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("PATH", dir)
|
||||
}
|
||||
|
||||
// An update pulls the newer image: the cached registry answer must not keep calling it behind.
|
||||
func TestSettledSeesAPulledImage(t *testing.T) {
|
||||
cached := []imageCheck{
|
||||
{Image: "nginx:alpine", State: "update", remote: "sha256:new"},
|
||||
{Image: "redis:7", State: "current", remote: "sha256:r"},
|
||||
{Image: "local/thing", State: "unknown"},
|
||||
}
|
||||
fakeDocker(t, `["nginx@sha256:new"]`)
|
||||
got := settled(context.Background(), cached)
|
||||
if got[0].State != "current" {
|
||||
t.Fatalf("pulled image still marked %q", got[0].State)
|
||||
}
|
||||
if got[1].State != "current" || got[2].State != "unknown" {
|
||||
t.Fatalf("others changed: %+v", got)
|
||||
}
|
||||
if cached[0].State != "update" {
|
||||
t.Fatal("settled wrote through to the cache")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSettledKeepsAnImageStillBehind(t *testing.T) {
|
||||
fakeDocker(t, `["nginx@sha256:old"]`)
|
||||
got := settled(context.Background(), []imageCheck{{Image: "nginx:alpine", State: "update", remote: "sha256:new"}})
|
||||
if got[0].State != "update" {
|
||||
t.Fatalf("image not yet pulled marked %q", got[0].State)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,300 @@
|
||||
package api
|
||||
|
||||
// Accounts and their sessions. Every person has their own login, and the console's look
|
||||
// (theme, accents, dashboard layout) follows the account rather than the machine. Anyone
|
||||
// signed in may add or remove accounts — a household console, not a tenancy — but removing
|
||||
// one, or resetting another's password, asks for the caller's own password first, so a
|
||||
// picked-up unlocked phone is not enough.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// actor is who a request is from: a signed-in account (with its session), or a token.
|
||||
type actor struct {
|
||||
User string // account name, "" for a token
|
||||
Session string // session ID, "" for a token
|
||||
token string // the raw cookie token, so a handler can spare the current session
|
||||
Token string // token name, "" for a session
|
||||
// Guest marks a temporary look-around login, and Expires is when it stops working. What
|
||||
// a guest may reach is decided in the middleware, not here; see internal/api/guests.go.
|
||||
Guest bool
|
||||
Expires time.Time
|
||||
}
|
||||
|
||||
type ctxKey int
|
||||
|
||||
const actorKey ctxKey = 1
|
||||
|
||||
func withActor(r *http.Request, a actor) *http.Request {
|
||||
return r.WithContext(context.WithValue(r.Context(), actorKey, a))
|
||||
}
|
||||
|
||||
func actorOf(r *http.Request) actor {
|
||||
a, _ := r.Context().Value(actorKey).(actor)
|
||||
return a
|
||||
}
|
||||
|
||||
// who names the caller for a log line: "core@10.0.0.5" for a person, "token:hub:core@…" for a hub.
|
||||
func (a actor) String() string {
|
||||
switch {
|
||||
case a.User != "" && a.Guest:
|
||||
return "guest:" + a.User
|
||||
case a.User != "":
|
||||
return a.User
|
||||
case a.Token != "":
|
||||
return "token:" + a.Token
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- users ----
|
||||
|
||||
func (s *Server) listUsers(w http.ResponseWriter, r *http.Request) {
|
||||
me := actorOf(r).User
|
||||
counts := s.sessions.Count()
|
||||
var out []map[string]any
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, u := range c.Accounts() { // guest logins are listed on their own, see listGuests
|
||||
var seen any
|
||||
if !u.LastSeen.IsZero() {
|
||||
seen = u.LastSeen
|
||||
}
|
||||
out = append(out, map[string]any{"name": u.Name, "created": u.Created, "last_seen": seen, "sessions": counts[u.Name], "self": u.Name == me})
|
||||
}
|
||||
})
|
||||
if out == nil {
|
||||
out = []map[string]any{}
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) addUser(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name, Password string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
name := strings.TrimSpace(body.Name)
|
||||
if !auth.UserNameRe.MatchString(name) {
|
||||
jsonErr(w, 400, "a user name is 1–32 lower-case letters, digits, . _ or -, starting with a letter or digit")
|
||||
return
|
||||
}
|
||||
if strings.HasPrefix(name, guestPrefix) {
|
||||
jsonErr(w, 400, "names starting with "+guestPrefix+" belong to guest logins")
|
||||
return
|
||||
}
|
||||
if len(body.Password) < auth.MinPasswordLen {
|
||||
jsonErr(w, 400, "password must be at least 12 characters")
|
||||
return
|
||||
}
|
||||
h, err := auth.HashPassword(body.Password)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
clash := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
if _, ok := c.User(name); ok {
|
||||
clash = true
|
||||
return
|
||||
}
|
||||
c.Users = append(c.Users, config.User{Name: name, Hash: h, Created: time.Now()})
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if clash {
|
||||
jsonErr(w, 409, "a user with that name exists")
|
||||
return
|
||||
}
|
||||
log.Printf("user %q added (by %s)", name, clientIP(r))
|
||||
jsonOK(w, 200, map[string]any{"name": name})
|
||||
}
|
||||
|
||||
// confirmSelf checks the caller's own password, the gate on anything done to another account.
|
||||
func (s *Server) confirmSelf(w http.ResponseWriter, r *http.Request, current string) bool {
|
||||
me := actorOf(r).User
|
||||
if me == "" {
|
||||
jsonErr(w, 403, "only a signed-in user may do that")
|
||||
return false
|
||||
}
|
||||
var users []config.User
|
||||
s.cfg.View(func(c *config.Config) { users = append(users, c.Users...) })
|
||||
if !auth.CheckUser(users, me, current) {
|
||||
jsonErr(w, 403, "your password is wrong")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *Server) removeUser(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
var body struct{ Current string }
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
if name == actorOf(r).User {
|
||||
jsonErr(w, 400, "you can't remove the account you are signed in as")
|
||||
return
|
||||
}
|
||||
if !s.confirmSelf(w, r, body.Current) {
|
||||
return
|
||||
}
|
||||
found, last := false, false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
if len(c.Accounts()) <= 1 { // guest logins don't keep the last real account alive
|
||||
last = true
|
||||
return
|
||||
}
|
||||
kept := c.Users[:0]
|
||||
for _, u := range c.Users {
|
||||
if u.Name == name {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
c.Users = kept
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if last {
|
||||
jsonErr(w, 400, "the last account can't be removed")
|
||||
return
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such user")
|
||||
return
|
||||
}
|
||||
s.sessions.RevokeUser(name)
|
||||
_ = s.store.Update(func(st *store.State) { delete(st.Users, name) })
|
||||
log.Printf("user %q removed (by %s)", name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// resetPassword sets another user's password, on the caller's own password.
|
||||
func (s *Server) resetPassword(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
var body struct{ Current, Password string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if !s.confirmSelf(w, r, body.Current) {
|
||||
return
|
||||
}
|
||||
if len(body.Password) < auth.MinPasswordLen {
|
||||
jsonErr(w, 400, "password must be at least 12 characters")
|
||||
return
|
||||
}
|
||||
if !s.setPassword(w, name, body.Password) {
|
||||
return
|
||||
}
|
||||
if name != actorOf(r).User { // whoever held the old password is signed out
|
||||
s.sessions.RevokeUser(name)
|
||||
}
|
||||
log.Printf("password of %q reset (by %s)", name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) setPassword(w http.ResponseWriter, name, pw string) bool {
|
||||
h, err := auth.HashPassword(pw)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return false
|
||||
}
|
||||
found := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
for i := range c.Users {
|
||||
if c.Users[i].Name == name {
|
||||
c.Users[i].Hash = h
|
||||
found = true
|
||||
}
|
||||
}
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return false
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such user")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ---- sessions: where the caller's account is signed in ----
|
||||
|
||||
func (s *Server) listSessions(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
if a.User == "" {
|
||||
jsonOK(w, 200, []any{})
|
||||
return
|
||||
}
|
||||
var out []map[string]any
|
||||
for _, ses := range s.sessions.List(a.User) {
|
||||
out = append(out, map[string]any{"id": ses.ID, "created": ses.Created, "last_seen": ses.LastSeen, "expires": ses.Expires, "ip": ses.IP, "agent": ses.Agent, "current": ses.ID == a.Session})
|
||||
}
|
||||
if out == nil {
|
||||
out = []map[string]any{}
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) revokeSession(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
id := r.PathValue("id")
|
||||
if a.User == "" || !s.sessions.Revoke(a.User, id) {
|
||||
jsonErr(w, 404, "no such session")
|
||||
return
|
||||
}
|
||||
log.Printf("session %s of %q revoked (by %s)", id, a.User, clientIP(r))
|
||||
if id == a.Session {
|
||||
s.setCookie(w, r, "", -1)
|
||||
}
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// revokeIdleSessions drops the caller's sessions that have not been used for ?hours (24 by
|
||||
// default): the scripts and headless browsers that signed in once and never came back,
|
||||
// without touching the phone that was used this morning.
|
||||
func (s *Server) revokeIdleSessions(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
if a.User == "" {
|
||||
jsonErr(w, 403, "only a signed-in user may do that")
|
||||
return
|
||||
}
|
||||
hours := 24
|
||||
if h := r.URL.Query().Get("hours"); h != "" {
|
||||
n, err := strconv.Atoi(h)
|
||||
if err != nil || n < 1 || n > 24*365 {
|
||||
jsonErr(w, 400, "hours is a whole number from 1 to 8760")
|
||||
return
|
||||
}
|
||||
hours = n
|
||||
}
|
||||
n := s.sessions.RevokeIdle(a.User, a.token, time.Duration(hours)*time.Hour)
|
||||
log.Printf("%d idle session(s) of %q revoked (idle > %dh, by %s)", n, a.User, hours, clientIP(r))
|
||||
jsonOK(w, 200, map[string]int{"revoked": n})
|
||||
}
|
||||
|
||||
func (s *Server) revokeOtherSessions(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
if a.User == "" {
|
||||
jsonErr(w, 403, "only a signed-in user may do that")
|
||||
return
|
||||
}
|
||||
n := s.sessions.RevokeOthers(a.User, a.token)
|
||||
log.Printf("%d other session(s) of %q revoked (by %s)", n, a.User, clientIP(r))
|
||||
jsonOK(w, 200, map[string]int{"revoked": n})
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
package api
|
||||
|
||||
// The VPN tab: this machine's WireGuard tunnels, who is on them, and — where the tunnel is
|
||||
// one this machine configures — adding, renaming and removing the devices that dial in.
|
||||
//
|
||||
// Per machine like every other tab: the tunnels shown and edited are the ones on the machine
|
||||
// in the switcher, which is the only honest way to show them, since the inbound server and an
|
||||
// outbound tunnel are different machines here.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"hamlab/internal/vpn"
|
||||
)
|
||||
|
||||
func (s *Server) vpnReport(w http.ResponseWriter, r *http.Request) {
|
||||
jsonOK(w, 200, vpn.Look(r.Context()))
|
||||
}
|
||||
|
||||
// tunnelOf finds the tunnel a request names, so every editing handler starts from the live
|
||||
// state rather than from whatever the page last saw.
|
||||
func (s *Server) tunnelOf(ctx context.Context, w http.ResponseWriter, name string) (vpn.Tunnel, bool) {
|
||||
for _, t := range vpn.Look(ctx).Tunnels {
|
||||
if t.Name == name {
|
||||
if !t.Managed {
|
||||
jsonErr(w, 400, vpn.ErrUnmanaged.Error())
|
||||
return t, false
|
||||
}
|
||||
return t, true
|
||||
}
|
||||
}
|
||||
jsonErr(w, 404, "no such tunnel on this machine")
|
||||
return vpn.Tunnel{}, false
|
||||
}
|
||||
|
||||
// addVPNPeer makes a device and hands back its profile. The profile holds a private key that
|
||||
// is generated for this call and stored nowhere, so it is the only time it can be read — the
|
||||
// same bargain as any password this console generates.
|
||||
func (s *Server) addVPNPeer(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var body vpn.NewPeer
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 8<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
made, err := vpn.AddPeer(r.Context(), t, body)
|
||||
if err != nil {
|
||||
code := 400
|
||||
if errors.Is(err, vpn.ErrUnmanaged) {
|
||||
code = 409
|
||||
}
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
// The name and address are worth a log line; the profile is emphatically not.
|
||||
log.Printf("vpn: device %q added to %s as %s (by %s)", made.Name, t.Name, made.Address, clientIP(r))
|
||||
jsonOK(w, 200, made)
|
||||
}
|
||||
|
||||
func (s *Server) editVPNPeer(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var body struct{ Name, Address string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
section := r.PathValue("section")
|
||||
if err := vpn.EditPeer(r.Context(), t, section, body.Name, body.Address); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("vpn: device %s on %s changed (by %s)", section, t.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) removeVPNPeer(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
section := r.PathValue("section")
|
||||
var body struct{ Confirm string }
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4<<10)).Decode(&body)
|
||||
// The name has to be typed back. Removing a device is the one action here that cannot be
|
||||
// undone — the key is gone and the profile on the device stops working — and it is one tap
|
||||
// away from a list of devices that all look alike.
|
||||
var want string
|
||||
for _, p := range t.Peers {
|
||||
if p.Section == section {
|
||||
want = p.Name
|
||||
}
|
||||
}
|
||||
if want != "" && !strings.EqualFold(strings.TrimSpace(body.Confirm), want) {
|
||||
jsonErr(w, 400, "type the device's name to remove it")
|
||||
return
|
||||
}
|
||||
if err := vpn.DeletePeer(r.Context(), t, section); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("vpn: device %s (%s) removed from %s (by %s)", section, want, t.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,287 @@
|
||||
package api
|
||||
|
||||
// The ZFS tab: pools, datasets, snapshots, and a schedule for taking and expiring them.
|
||||
//
|
||||
// grid already runs zfSnap from root cron and has done for longer than this repo has
|
||||
// existed, so this does not replace it. It reads the same snapshot names, expires by the
|
||||
// same rule, and surfaces the cron job that drives it — so what is on screen is the truth
|
||||
// about the machine, whether Ham Lab or zfSnap is doing the work. A Ham Lab schedule is for
|
||||
// a machine that has no such tool, and the page says plainly when both are running.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/storage"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
type zfsState struct {
|
||||
Policies []storage.Policy `json:"policies"`
|
||||
Verify []storage.VerifyPolicy `json:"verify"` // landing-tier verify schedules (landing.go)
|
||||
}
|
||||
|
||||
func (s *Server) zfsLoad() zfsState {
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.ZFS })
|
||||
var z zfsState
|
||||
if len(raw) > 0 {
|
||||
_ = json.Unmarshal(raw, &z)
|
||||
}
|
||||
if z.Policies == nil {
|
||||
z.Policies = []storage.Policy{}
|
||||
}
|
||||
if z.Verify == nil {
|
||||
z.Verify = []storage.VerifyPolicy{}
|
||||
}
|
||||
return z
|
||||
}
|
||||
|
||||
func (s *Server) zfsSave(z zfsState) error {
|
||||
b, err := json.Marshal(z)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.store.Update(func(st *store.State) { st.ZFS = b })
|
||||
}
|
||||
|
||||
// externalSnapshotJob finds a cron job that is already taking snapshots on this machine, so
|
||||
// the page can say so rather than let two schedules quietly double up.
|
||||
func (s *Server) externalSnapshotJob(ctx context.Context) map[string]any {
|
||||
if s.cron == nil {
|
||||
return nil
|
||||
}
|
||||
ov := s.cron.Overview(ctx)
|
||||
for _, j := range ov.Jobs {
|
||||
if !j.Enabled {
|
||||
continue
|
||||
}
|
||||
c := strings.ToLower(j.Command)
|
||||
if !strings.Contains(c, "zfsnap") && !strings.Contains(c, "zfs-auto-snapshot") && !strings.Contains(c, "sanoid") &&
|
||||
!(strings.Contains(c, "zfs ") && strings.Contains(c, "snapshot")) {
|
||||
continue
|
||||
}
|
||||
return map[string]any{
|
||||
"tool": toolName(c), "schedule": j.Schedule, "command": j.Command,
|
||||
"user": j.User, "source": j.Source, "id": j.ID, "next": j.Next, "last": j.Last,
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func toolName(cmd string) string {
|
||||
switch {
|
||||
case strings.Contains(cmd, "zfsnap"):
|
||||
return "zfSnap"
|
||||
case strings.Contains(cmd, "zfs-auto-snapshot"):
|
||||
return "zfs-auto-snapshot"
|
||||
case strings.Contains(cmd, "sanoid"):
|
||||
return "sanoid"
|
||||
}
|
||||
return "a cron job"
|
||||
}
|
||||
|
||||
// randomID is a short opaque id for a schedule.
|
||||
func randomID() (string, error) {
|
||||
b := make([]byte, 6)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(b), nil
|
||||
}
|
||||
|
||||
// ---- the cron file ----
|
||||
//
|
||||
// A schedule is a cron line, written to /etc/cron.d/hamlab-zfs, which Ham Lab owns and
|
||||
// rewrites whole whenever the schedules change. Nothing in the daemon runs them: snapshots
|
||||
// are backup-shaped and should not stop because a web console is down, being upgraded, or
|
||||
// uninstalled. The line calls `hamlab zfs-snapshot <id>` as the account Ham Lab runs as,
|
||||
// which reaches zfs through sudo exactly as the daemon does.
|
||||
|
||||
// syncCron puts the current schedules on the machine.
|
||||
func (s *Server) syncCron(ctx context.Context, z zfsState) error {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
exe, _ = filepath.EvalSymlinks(exe)
|
||||
return s.storage.WriteCronFile(ctx, z.Policies, s.user, exe, s.cfg.Path())
|
||||
}
|
||||
|
||||
// ---- endpoints ----
|
||||
|
||||
func (s *Server) zfsOverview(w http.ResponseWriter, r *http.Request) {
|
||||
z := s.zfsLoad()
|
||||
now := time.Now()
|
||||
dir := s.cfg.DataDir()
|
||||
out := make([]map[string]any, 0, len(z.Policies))
|
||||
for i := range z.Policies {
|
||||
p := z.Policies[i]
|
||||
expr, _ := p.CronExpr()
|
||||
m := map[string]any{"policy": p, "cron": expr, "last": storage.ReadRun(dir, p.ID)}
|
||||
if p.Enabled {
|
||||
if n := p.NextRun(now); !n.IsZero() {
|
||||
m["next"] = n
|
||||
}
|
||||
}
|
||||
out = append(out, m)
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"zfs": s.storage.ZFS(r.Context()), "policies": out, "external": s.externalSnapshotJob(r.Context()),
|
||||
})
|
||||
}
|
||||
|
||||
// zfsARC is the cache: the sampler's live ARC (size, the hit rate over the last interval)
|
||||
// beside the storage package's fuller picture (since-boot counters, tunables, and each
|
||||
// pool's cache/log/special devices or the lack of them), and the write path — the dirty
|
||||
// buffer in RAM, each pool's write rate, and any landing tier in front of a pool.
|
||||
func (s *Server) zfsARC(w http.ResponseWriter, r *http.Request) {
|
||||
out := map[string]any{"detail": s.storage.ARC(r.Context()), "live": nil, "writes": nil}
|
||||
if smp, ok := s.metrics.Latest(); ok && smp.ARC != nil {
|
||||
out["live"] = smp.ARC
|
||||
out["writes"] = smp.ZFSWrite
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) zfsPolicyPut(w http.ResponseWriter, r *http.Request) {
|
||||
var p storage.Policy
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 16<<10)).Decode(&p); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if err := p.Validate(); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
z := s.zfsLoad()
|
||||
found := false
|
||||
for i := range z.Policies {
|
||||
if z.Policies[i].ID == p.ID && p.ID != "" {
|
||||
z.Policies[i] = p
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
if p.ID == "" {
|
||||
id, err := randomID()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
p.ID = id
|
||||
}
|
||||
z.Policies = append(z.Policies, p)
|
||||
}
|
||||
if err := s.zfsSave(z); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.syncCron(r.Context(), z); err != nil {
|
||||
jsonErr(w, 500, "saved, but writing "+storage.CronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
expr, _ := p.CronExpr()
|
||||
log.Printf("zfs schedule %q saved by %s (%s, %q, keep %s)", p.Name, clientIP(r), p.Dataset, expr, p.KeepFor)
|
||||
jsonOK(w, 200, p)
|
||||
}
|
||||
|
||||
func (s *Server) zfsPolicyDelete(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
z := s.zfsLoad()
|
||||
kept := z.Policies[:0]
|
||||
found := false
|
||||
for _, p := range z.Policies {
|
||||
if p.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, p)
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
z.Policies = kept
|
||||
if err := s.zfsSave(z); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.syncCron(r.Context(), z); err != nil {
|
||||
jsonErr(w, 500, "removed, but writing "+storage.CronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
storage.ForgetRun(s.cfg.DataDir(), id)
|
||||
log.Printf("zfs schedule %s removed by %s (its snapshots are left alone)", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) zfsPolicy(id string) (storage.Policy, bool) {
|
||||
for _, p := range s.zfsLoad().Policies {
|
||||
if p.ID == id {
|
||||
return p, true
|
||||
}
|
||||
}
|
||||
return storage.Policy{}, false
|
||||
}
|
||||
|
||||
func (s *Server) zfsRunNow(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
p, ok := s.zfsPolicy(id)
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
res, err := s.storage.Run(r.Context(), &p)
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: res.Taken, Pruned: res.Pruned}
|
||||
if err != nil {
|
||||
rec.Error = err.Error()
|
||||
}
|
||||
_ = storage.WriteRun(s.cfg.DataDir(), id, rec)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("zfs schedule %q run by %s: took %s, expired %d", p.Name, clientIP(r), res.Taken, len(res.Pruned))
|
||||
jsonOK(w, 200, res)
|
||||
}
|
||||
|
||||
// zfsSnapshotDelete removes one snapshot, named in full and typed back by the caller.
|
||||
func (s *Server) zfsSnapshotDelete(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Name string `json:"name"`
|
||||
Confirm string `json:"confirm"`
|
||||
Recursive bool `json:"recursive"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if body.Confirm != body.Name {
|
||||
jsonErr(w, 400, "type the snapshot's full name to confirm")
|
||||
return
|
||||
}
|
||||
if err := s.storage.DestroySnapshot(r.Context(), body.Name, body.Recursive); err != nil {
|
||||
msg := err.Error()
|
||||
switch {
|
||||
case msg == "bad snapshot name":
|
||||
jsonErr(w, 400, msg)
|
||||
case strings.Contains(msg, "does not exist"):
|
||||
jsonErr(w, 404, msg)
|
||||
default:
|
||||
jsonErr(w, 500, msg)
|
||||
}
|
||||
return
|
||||
}
|
||||
log.Printf("zfs snapshot %s destroyed by %s%s", body.Name, clientIP(r), map[bool]string{true: " (recursive)", false: ""}[body.Recursive])
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
+324
-32
@@ -3,17 +3,21 @@ package auth
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"golang.org/x/crypto/argon2"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
@@ -24,6 +28,10 @@ const (
|
||||
argonKeyLen = 32
|
||||
|
||||
SessionTTL = 30 * 24 * time.Hour
|
||||
// SessionIdle is how long a session may go unused before it is dropped, whatever its
|
||||
// expiry says. Every script, curl and headless browser that signs in leaves a session;
|
||||
// a phone that is used keeps its own alive by being used.
|
||||
SessionIdle = 7 * 24 * time.Hour
|
||||
)
|
||||
|
||||
func HashPassword(pw string) (string, error) {
|
||||
@@ -59,6 +67,118 @@ func VerifyPassword(hash, pw string) bool {
|
||||
return subtle.ConstantTimeCompare(got, want) == 1
|
||||
}
|
||||
|
||||
// ---- API tokens (hub → node) ----
|
||||
|
||||
// NewToken returns a random bearer token and its SHA-256 hex, which is what gets stored.
|
||||
func NewToken() (token, hash string, err error) {
|
||||
b := make([]byte, 32)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
token = base64.RawURLEncoding.EncodeToString(b)
|
||||
return token, HashToken(token), nil
|
||||
}
|
||||
|
||||
func HashToken(token string) string {
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// TokenMatches compares a presented token against stored hashes in constant time.
|
||||
func TokenMatches(token string, hashes []string) bool { return TokenIndex(token, hashes) >= 0 }
|
||||
|
||||
// TokenIndex is which of hashes the presented token matches, -1 for none; every hash is
|
||||
// compared whatever the answer.
|
||||
func TokenIndex(token string, hashes []string) int {
|
||||
h := []byte(HashToken(token))
|
||||
idx := -1
|
||||
for i, want := range hashes {
|
||||
if subtle.ConstantTimeCompare(h, []byte(want)) == 1 {
|
||||
idx = i
|
||||
}
|
||||
}
|
||||
return idx
|
||||
}
|
||||
|
||||
// ---- users ----
|
||||
|
||||
// UserNameRe is what an account may be called: like a Unix login, lower-case.
|
||||
var UserNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{0,31}$`)
|
||||
|
||||
const MinPasswordLen = 12
|
||||
|
||||
// dummyHash is verified against when the named user does not exist, so a login attempt
|
||||
// takes the same time either way and the form cannot be used to list accounts.
|
||||
var dummyHash = func() string {
|
||||
h, err := HashPassword("not-a-real-password")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return h
|
||||
}()
|
||||
|
||||
// CheckUser verifies a password for one of users. A missing user — or a guest login whose
|
||||
// time is up — fails in the same time as a wrong password.
|
||||
func CheckUser(users []config.User, name, pw string) bool {
|
||||
for _, u := range users {
|
||||
if u.Name == name {
|
||||
if u.Expired(time.Now()) {
|
||||
break
|
||||
}
|
||||
return VerifyPassword(u.Hash, pw)
|
||||
}
|
||||
}
|
||||
VerifyPassword(dummyHash, pw)
|
||||
return false
|
||||
}
|
||||
|
||||
// ---- guest logins ----
|
||||
|
||||
// guestAlphabet leaves out every letter and digit that can be misread in a screenshot or
|
||||
// over the phone: i, l, o, 0 and 1. A guest login is read aloud and typed by hand.
|
||||
const guestAlphabet = "abcdefghjkmnpqrstuvwxyz23456789"
|
||||
|
||||
// randomFrom returns n characters of guestAlphabet, chosen without modulo bias.
|
||||
func randomFrom(n int) (string, error) {
|
||||
out := make([]byte, 0, n)
|
||||
buf := make([]byte, n)
|
||||
limit := byte(256 - 256%len(guestAlphabet)) // the top of the last whole run of the alphabet
|
||||
for len(out) < n {
|
||||
if _, err := rand.Read(buf); err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, b := range buf {
|
||||
if b >= limit {
|
||||
continue // would favour the first few letters; ask for another byte instead
|
||||
}
|
||||
out = append(out, guestAlphabet[int(b)%len(guestAlphabet)])
|
||||
if len(out) == n {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return string(out), nil
|
||||
}
|
||||
|
||||
// GuestPassword is a password for a guest login: four groups of four, ~78 bits, in an
|
||||
// alphabet with nothing ambiguous in it.
|
||||
func GuestPassword() (string, error) {
|
||||
var groups []string
|
||||
for i := 0; i < 4; i++ {
|
||||
g, err := randomFrom(4)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
groups = append(groups, g)
|
||||
}
|
||||
return strings.Join(groups, "-"), nil
|
||||
}
|
||||
|
||||
// GuestSuffix is the tail of a generated guest name.
|
||||
func GuestSuffix() (string, error) { return randomFrom(4) }
|
||||
|
||||
// ---- sessions ----
|
||||
|
||||
// Sessions is a token store backed by the state file. Expiry is sliding.
|
||||
type Sessions struct {
|
||||
st *store.Store
|
||||
@@ -72,89 +192,261 @@ func NewSessions(st *store.Store) *Sessions {
|
||||
|
||||
func pruneExpired(state *store.State) {
|
||||
now := time.Now()
|
||||
for tok, exp := range state.Sessions {
|
||||
if exp.Before(now) {
|
||||
for tok, ses := range state.Sessions {
|
||||
if !live(ses, now) {
|
||||
delete(state.Sessions, tok)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Sessions) Create() (string, error) {
|
||||
b := make([]byte, 32)
|
||||
// live is whether a session still counts: unexpired, and used within SessionIdle. A
|
||||
// session from before visits were recorded is judged by its creation, or failing that by
|
||||
// its expiry alone.
|
||||
func live(ses store.Session, now time.Time) bool {
|
||||
if !ses.Expires.After(now) {
|
||||
return false
|
||||
}
|
||||
seen := ses.LastSeen
|
||||
if seen.IsZero() {
|
||||
seen = ses.Created
|
||||
}
|
||||
return seen.IsZero() || now.Sub(seen) <= SessionIdle
|
||||
}
|
||||
|
||||
func randomHex(n int) (string, error) {
|
||||
b := make([]byte, n)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", err
|
||||
}
|
||||
tok := hex.EncodeToString(b)
|
||||
err := s.st.Update(func(state *store.State) {
|
||||
return hex.EncodeToString(b), nil
|
||||
}
|
||||
|
||||
// Create signs user in from ip with agent, returning the cookie token.
|
||||
func (s *Sessions) Create(user, ip, agent string) (string, error) {
|
||||
tok, err := randomHex(32)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
id, err := randomHex(6)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(agent) > 200 {
|
||||
agent = agent[:200]
|
||||
}
|
||||
now := time.Now()
|
||||
err = s.st.Update(func(state *store.State) {
|
||||
pruneExpired(state)
|
||||
state.Sessions[tok] = time.Now().Add(SessionTTL)
|
||||
state.Sessions[tok] = store.Session{ID: id, User: user, Expires: now.Add(SessionTTL), Created: now, LastSeen: now, IP: ip, Agent: agent}
|
||||
})
|
||||
return tok, err
|
||||
}
|
||||
|
||||
// Valid reports whether tok is live, sliding its expiry (persisted at most hourly).
|
||||
func (s *Sessions) Valid(tok string) bool {
|
||||
// Lookup reports whether tok is live and whose it is, sliding its expiry and noting the
|
||||
// visit (persisted at most hourly, so a busy console is not writing the file all day).
|
||||
func (s *Sessions) Lookup(tok, ip string) (store.Session, bool) {
|
||||
if tok == "" {
|
||||
return false
|
||||
return store.Session{}, false
|
||||
}
|
||||
ok := false
|
||||
extend := false
|
||||
var ses store.Session
|
||||
ok, touch := false, false
|
||||
s.st.View(func(state *store.State) {
|
||||
exp, found := state.Sessions[tok]
|
||||
if found && exp.After(time.Now()) {
|
||||
ok = true
|
||||
extend = time.Until(exp) < SessionTTL-time.Hour
|
||||
found, have := state.Sessions[tok]
|
||||
if have && live(found, time.Now()) {
|
||||
ses, ok = found, true
|
||||
touch = time.Until(found.Expires) < SessionTTL-time.Hour || time.Since(found.LastSeen) > time.Hour
|
||||
}
|
||||
})
|
||||
if ok && extend {
|
||||
if ok && touch {
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
if _, found := state.Sessions[tok]; found {
|
||||
state.Sessions[tok] = time.Now().Add(SessionTTL)
|
||||
if x, found := state.Sessions[tok]; found {
|
||||
x.Expires = time.Now().Add(SessionTTL)
|
||||
x.LastSeen = time.Now()
|
||||
if ip != "" {
|
||||
x.IP = ip
|
||||
}
|
||||
state.Sessions[tok] = x
|
||||
}
|
||||
})
|
||||
}
|
||||
return ok
|
||||
return ses, ok
|
||||
}
|
||||
|
||||
func (s *Sessions) Delete(tok string) {
|
||||
_ = s.st.Update(func(state *store.State) { delete(state.Sessions, tok) })
|
||||
}
|
||||
|
||||
// Limiter is a single global backoff: after 5 failures, each further failure
|
||||
// doubles a lockout starting at 30s, capped at 10 minutes. Single-user box.
|
||||
// List is every live session of user, newest first.
|
||||
func (s *Sessions) List(user string) []store.Session {
|
||||
var out []store.Session
|
||||
s.st.View(func(state *store.State) {
|
||||
now := time.Now()
|
||||
for _, ses := range state.Sessions {
|
||||
if ses.User == user && live(ses, now) {
|
||||
out = append(out, ses)
|
||||
}
|
||||
}
|
||||
})
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].LastSeen.After(out[j].LastSeen) })
|
||||
return out
|
||||
}
|
||||
|
||||
// Count is how many live sessions each user has.
|
||||
func (s *Sessions) Count() map[string]int {
|
||||
n := map[string]int{}
|
||||
s.st.View(func(state *store.State) {
|
||||
now := time.Now()
|
||||
for _, ses := range state.Sessions {
|
||||
if live(ses, now) {
|
||||
n[ses.User]++
|
||||
}
|
||||
}
|
||||
})
|
||||
return n
|
||||
}
|
||||
|
||||
// Revoke ends the session of user with this ID. Reports whether there was one.
|
||||
func (s *Sessions) Revoke(user, id string) bool {
|
||||
found := false
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
for tok, ses := range state.Sessions {
|
||||
if ses.User == user && ses.ID == id {
|
||||
delete(state.Sessions, tok)
|
||||
found = true
|
||||
}
|
||||
}
|
||||
})
|
||||
return found
|
||||
}
|
||||
|
||||
// RevokeOthers ends every session of user except the one holding keep. Returns how many.
|
||||
func (s *Sessions) RevokeOthers(user, keep string) int {
|
||||
n := 0
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
for tok, ses := range state.Sessions {
|
||||
if ses.User == user && tok != keep {
|
||||
delete(state.Sessions, tok)
|
||||
n++
|
||||
}
|
||||
}
|
||||
})
|
||||
return n
|
||||
}
|
||||
|
||||
// RevokeIdle ends every session of user not seen for longer than idle, except the one
|
||||
// holding keep. Returns how many.
|
||||
func (s *Sessions) RevokeIdle(user, keep string, idle time.Duration) int {
|
||||
n := 0
|
||||
now := time.Now()
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
for tok, ses := range state.Sessions {
|
||||
seen := ses.LastSeen
|
||||
if seen.IsZero() {
|
||||
seen = ses.Created
|
||||
}
|
||||
if ses.User == user && tok != keep && now.Sub(seen) > idle {
|
||||
delete(state.Sessions, tok)
|
||||
n++
|
||||
}
|
||||
}
|
||||
})
|
||||
return n
|
||||
}
|
||||
|
||||
// RevokeUser ends every session of user (the account is going away).
|
||||
func (s *Sessions) RevokeUser(user string) { s.RevokeOthers(user, "") }
|
||||
|
||||
// Limiter locks out a client after repeated failed logins, per client address, with a
|
||||
// global backstop so a botnet can't burn CPU on argon2 indefinitely. A lockout for one
|
||||
// address never blocks another. Past perClientLock failures, each further failure doubles a
|
||||
// lockout starting at 30s, capped at 10 minutes.
|
||||
type Limiter struct {
|
||||
mu sync.Mutex
|
||||
clients map[string]*bucket
|
||||
global bucket
|
||||
}
|
||||
|
||||
type bucket struct {
|
||||
failures int
|
||||
lockUntil time.Time
|
||||
last time.Time
|
||||
}
|
||||
|
||||
var ErrLocked = errors.New("too many failed logins")
|
||||
|
||||
func (l *Limiter) Allow() error {
|
||||
const (
|
||||
// perClientLock is how many failures one address gets before it is made to wait. Set high
|
||||
// on purpose: a guest login is a generated password read off a message and typed by hand
|
||||
// on a phone, and being locked out for fumbling it is worse than the brute-force risk —
|
||||
// which the doubling backoff past this point, and the global backstop, still cover.
|
||||
perClientLock = 20
|
||||
globalLock = 100 // failures (all clients) in the window before everyone waits a minute
|
||||
globalWindow = 5 * time.Minute
|
||||
)
|
||||
|
||||
func (l *Limiter) get(key string) *bucket {
|
||||
if l.clients == nil {
|
||||
l.clients = map[string]*bucket{}
|
||||
}
|
||||
b, ok := l.clients[key]
|
||||
if !ok {
|
||||
b = &bucket{}
|
||||
l.clients[key] = b
|
||||
}
|
||||
// forget clients that have been quiet for an hour
|
||||
if len(l.clients) > 256 {
|
||||
for k, x := range l.clients {
|
||||
if time.Since(x.last) > time.Hour && time.Now().After(x.lockUntil) {
|
||||
delete(l.clients, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func (l *Limiter) Allow(key string) error {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if time.Now().Before(l.lockUntil) {
|
||||
now := time.Now()
|
||||
if now.Before(l.global.lockUntil) {
|
||||
return ErrLocked
|
||||
}
|
||||
if now.Sub(l.global.last) > globalWindow {
|
||||
l.global.failures = 0
|
||||
}
|
||||
if b := l.get(key); now.Before(b.lockUntil) {
|
||||
return ErrLocked
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (l *Limiter) Fail() {
|
||||
func (l *Limiter) Fail(key string) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.failures++
|
||||
if l.failures >= 5 {
|
||||
d := 30 * time.Second << uint(l.failures-5)
|
||||
now := time.Now()
|
||||
b := l.get(key)
|
||||
b.failures++
|
||||
b.last = now
|
||||
if b.failures >= perClientLock {
|
||||
d := 30 * time.Second << uint(b.failures-perClientLock)
|
||||
if d > 10*time.Minute {
|
||||
d = 10 * time.Minute
|
||||
}
|
||||
l.lockUntil = time.Now().Add(d)
|
||||
b.lockUntil = now.Add(d)
|
||||
}
|
||||
if now.Sub(l.global.last) > globalWindow {
|
||||
l.global.failures = 0
|
||||
}
|
||||
l.global.failures++
|
||||
l.global.last = now
|
||||
if l.global.failures >= globalLock {
|
||||
l.global.lockUntil = now.Add(time.Minute)
|
||||
l.global.failures = 0
|
||||
}
|
||||
}
|
||||
|
||||
func (l *Limiter) Reset() {
|
||||
func (l *Limiter) Reset(key string) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.failures = 0
|
||||
l.lockUntil = time.Time{}
|
||||
delete(l.clients, key)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
// Package compose runs docker compose the way a stack was deployed — which is not always
|
||||
// simply "on this host". A stack deployed by an orchestrator running inside its own
|
||||
// container records paths in that container's filesystem, so compose has to run in there.
|
||||
// Both the console and the scheduled updates it writes must invoke it identically, or a
|
||||
// redeploy from one would not match the other's labels. Compose always drives this
|
||||
// machine's own Docker: a stack's files and its containers are on the same box.
|
||||
package compose
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// Spec is everything needed to run compose for one stack.
|
||||
type Spec struct {
|
||||
Project string // -p
|
||||
Files []string // -f, in order
|
||||
Via string // a container to run compose inside; "" is this host
|
||||
Tool string // "docker" or "podman"
|
||||
}
|
||||
|
||||
// Cmd builds the command. args are compose's own, e.g. "pull" or "up", "-d".
|
||||
func Cmd(ctx context.Context, s Spec, args ...string) *exec.Cmd {
|
||||
tool := s.Tool
|
||||
if tool == "" {
|
||||
tool = "docker"
|
||||
}
|
||||
base := []string{"compose", "-p", s.Project}
|
||||
for _, f := range s.Files {
|
||||
base = append(base, "-f", f)
|
||||
}
|
||||
if s.Via != "" {
|
||||
// The CLI inside the orchestrator's container is its own `docker`, whatever this host runs.
|
||||
return exec.CommandContext(ctx, tool, append(append([]string{"exec", s.Via, "docker"}, base...), args...)...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, tool, append(base, args...)...)
|
||||
if len(s.Files) > 0 {
|
||||
cmd.Dir = filepath.Dir(s.Files[0])
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
+151
-5
@@ -7,13 +7,109 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// Token is an API token another Ham Lab (a hub) uses to manage this host. Only the
|
||||
// SHA-256 of the token is kept.
|
||||
type Token struct {
|
||||
Name string `json:"name"`
|
||||
Hash string `json:"hash"`
|
||||
Created time.Time `json:"created"`
|
||||
// Scope is what the token may do. "" is everything (a hub's token: full control of
|
||||
// this machine). ScopeSensorsRead only reads sensors — what a dashboard gets.
|
||||
Scope string `json:"scope,omitempty"`
|
||||
// Sensors narrows a sensors:read token to the IDs these patterns match. Empty is all.
|
||||
Sensors []string `json:"sensors,omitempty"`
|
||||
// IssuedBy is the account that made the token (or paired the hub that holds it).
|
||||
IssuedBy string `json:"issued_by,omitempty"`
|
||||
}
|
||||
|
||||
const ScopeSensorsRead = "sensors:read"
|
||||
|
||||
// Node is a remote Ham Lab this host (as hub) manages. The token is the one that
|
||||
// node issued to us; the fingerprint pins its self-signed certificate.
|
||||
type Node struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
Fingerprint string `json:"fingerprint,omitempty"`
|
||||
Token string `json:"token"`
|
||||
Added time.Time `json:"added"`
|
||||
}
|
||||
|
||||
// User is an account on this machine's console. Only the argon2id hash is kept.
|
||||
type User struct {
|
||||
Name string `json:"name"`
|
||||
Hash string `json:"hash"`
|
||||
Created time.Time `json:"created"`
|
||||
LastSeen time.Time `json:"last_seen,omitempty"`
|
||||
// Guest marks a temporary login made to show somebody the console: it may only read,
|
||||
// it expires by itself at Expires, and it is not one of the machine's accounts — it is
|
||||
// left out of the account list, does not count towards "the last account", and cannot
|
||||
// be the one an empty user name means. See internal/api/guests.go.
|
||||
Guest bool `json:"guest,omitempty"`
|
||||
// Expires is when a guest login stops working. A guest without one has already
|
||||
// expired, which is the safe way to read a half-written file.
|
||||
Expires time.Time `json:"expires,omitempty"`
|
||||
// IssuedBy is the account that made this guest login.
|
||||
IssuedBy string `json:"issued_by,omitempty"`
|
||||
}
|
||||
|
||||
// Expired reports whether a guest login's time is up. A real account never expires; a
|
||||
// guest with no expiry at all counts as expired rather than as forever.
|
||||
func (u User) Expired(now time.Time) bool {
|
||||
return u.Guest && (u.Expires.IsZero() || now.After(u.Expires))
|
||||
}
|
||||
|
||||
// UpdateSource is where this machine looks for a newer Ham Lab, and how often. The field
|
||||
// is Updates, not Update, because Config already has an Update method.
|
||||
type UpdateSource struct {
|
||||
// URL is a Gitea/Forgejo repo API base or a static directory; empty means the built-in
|
||||
// default, and "off" means never look.
|
||||
URL string `json:"url,omitempty"`
|
||||
// Token is sent when the source is behind a login, so a private Gitea can serve its
|
||||
// own fleet without the instance being opened to everyone.
|
||||
Token string `json:"token,omitempty"`
|
||||
// Check is "daily" (the default) or "never".
|
||||
Check string `json:"check,omitempty"`
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
Listen string `json:"listen"`
|
||||
PasswordHash string `json:"password_hash"`
|
||||
// PasswordHash is the single login of installs before users existed. On load it becomes
|
||||
// the first user (see MigrateUsers) and is then dropped from the file.
|
||||
PasswordHash string `json:"password_hash,omitempty"`
|
||||
Users []User `json:"users,omitempty"`
|
||||
TLS string `json:"tls,omitempty"` // "auto" → self-signed cert in the data dir; "" → plain HTTP
|
||||
Tokens []Token `json:"tokens,omitempty"`
|
||||
Nodes []Node `json:"nodes,omitempty"`
|
||||
// Updates: where a newer build comes from. See UpdateSource.
|
||||
Updates UpdateSource `json:"update,omitempty"`
|
||||
|
||||
path string
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// Candidates lists where config.json may live under a home directory, preferred first:
|
||||
// the XDG location, then ~/apps/hamlab, which installs before v0.2 used.
|
||||
func Candidates(home string) []string {
|
||||
return []string{filepath.Join(home, ".config", "hamlab", "config.json"), filepath.Join(home, "apps", "hamlab", "config.json")}
|
||||
}
|
||||
|
||||
// DefaultPath is the config used when neither -config nor $HAMLAB_CONFIG says otherwise:
|
||||
// whichever candidate exists, else the preferred one.
|
||||
func DefaultPath() string {
|
||||
home, _ := os.UserHomeDir()
|
||||
c := Candidates(home)
|
||||
for _, p := range c {
|
||||
if _, err := os.Stat(p); err == nil {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return c[0]
|
||||
}
|
||||
|
||||
func Default(path string) *Config {
|
||||
@@ -36,10 +132,49 @@ func Load(path string) (*Config, error) {
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// MigrateUsers turns the pre-users single password into the first account, named after
|
||||
// the OS user the console runs as. Reports whether anything changed (the caller saves).
|
||||
func (c *Config) MigrateUsers(name string) bool {
|
||||
if len(c.Users) > 0 || c.PasswordHash == "" {
|
||||
return false
|
||||
}
|
||||
c.Users = []User{{Name: name, Hash: c.PasswordHash, Created: time.Now()}}
|
||||
c.PasswordHash = ""
|
||||
return true
|
||||
}
|
||||
|
||||
// Accounts is every real account: the machine's users, without the guest logins.
|
||||
// Call under View/Update or on a config nobody else holds.
|
||||
func (c *Config) Accounts() []User {
|
||||
out := make([]User, 0, len(c.Users))
|
||||
for _, u := range c.Users {
|
||||
if !u.Guest {
|
||||
out = append(out, u)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// User finds an account by name. Call under View/Update or on a config nobody else holds.
|
||||
func (c *Config) User(name string) (User, bool) {
|
||||
for _, u := range c.Users {
|
||||
if u.Name == name {
|
||||
return u, true
|
||||
}
|
||||
}
|
||||
return User{}, false
|
||||
}
|
||||
|
||||
func (c *Config) Path() string { return c.path }
|
||||
func (c *Config) DataDir() string { return filepath.Dir(c.path) }
|
||||
|
||||
func (c *Config) Save() error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.save()
|
||||
}
|
||||
|
||||
func (c *Config) save() error {
|
||||
if err := os.MkdirAll(c.DataDir(), 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -47,9 +182,20 @@ func (c *Config) Save() error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := c.path + ".tmp"
|
||||
if err := os.WriteFile(tmp, append(b, '\n'), 0o600); err != nil {
|
||||
return err
|
||||
return store.WriteFileAtomic(c.path, append(b, '\n'), 0o600)
|
||||
}
|
||||
return os.Rename(tmp, c.path)
|
||||
|
||||
// Update applies fn under the lock and saves.
|
||||
func (c *Config) Update(fn func(*Config)) error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
fn(c)
|
||||
return c.save()
|
||||
}
|
||||
|
||||
// View reads under the lock.
|
||||
func (c *Config) View(fn func(*Config)) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
fn(c)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,813 @@
|
||||
// Package cron reads and writes what the UI calls the **Jobs** tab (named "Cron" until
|
||||
// 2026-09-24; the package and its /api/cron endpoints kept the old name).
|
||||
//
|
||||
// Package cron reads every scheduled job on the host into one list: /etc/crontab,
|
||||
// /etc/cron.d/*, per-user crontabs, the run-parts directories they drive, and systemd
|
||||
// timers. It computes next runs from the cron expressions, pulls last runs from the
|
||||
// journal (cron logs "(user) CMD (command)" via syslog), and can toggle, edit and run
|
||||
// jobs. Reads/writes that need root go through passwordless sudo when available.
|
||||
package cron
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha1"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type Job struct {
|
||||
ID string `json:"id"`
|
||||
Source string `json:"source"` // "/etc/crontab", "/etc/cron.d/x", "crontab:user", "/etc/cron.daily"
|
||||
Line int `json:"line"` // 1-based line in Source (0 for run-parts scripts)
|
||||
User string `json:"user"`
|
||||
Schedule string `json:"schedule"`
|
||||
Command string `json:"command"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Next int64 `json:"next,omitempty"` // unix ms
|
||||
Last int64 `json:"last,omitempty"`
|
||||
OutFile string `json:"out_file,omitempty"` // parsed from a ">> file" redirect
|
||||
Note string `json:"note,omitempty"`
|
||||
Editable bool `json:"editable"`
|
||||
raw string
|
||||
}
|
||||
|
||||
type Timer struct {
|
||||
Unit string `json:"unit"`
|
||||
Activates string `json:"activates"`
|
||||
Description string `json:"description"`
|
||||
Active string `json:"active"`
|
||||
Sub string `json:"sub"`
|
||||
Enabled string `json:"enabled"`
|
||||
Next int64 `json:"next,omitempty"`
|
||||
Last int64 `json:"last,omitempty"`
|
||||
}
|
||||
|
||||
type Overview struct {
|
||||
Jobs []Job `json:"jobs"`
|
||||
Timers []Timer `json:"timers"`
|
||||
Logging bool `json:"logging"` // cron lines seen in the journal
|
||||
Anacron bool `json:"anacron"`
|
||||
Sudo bool `json:"sudo"`
|
||||
}
|
||||
|
||||
type Run struct {
|
||||
ID string `json:"id"`
|
||||
JobID string `json:"job_id"`
|
||||
Command string `json:"command"`
|
||||
User string `json:"user"`
|
||||
Started time.Time `json:"started"`
|
||||
Finished time.Time `json:"finished,omitempty"`
|
||||
Running bool `json:"running"`
|
||||
ExitCode int `json:"exit_code"`
|
||||
Output string `json:"output"`
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
sudo bool
|
||||
journalSudo bool
|
||||
self string
|
||||
mu sync.Mutex
|
||||
lastRuns map[string]int64 // user\x00command -> unix ms
|
||||
lastAt time.Time
|
||||
lastCursor string // journal cursor of the newest entry folded into lastRuns
|
||||
logging bool
|
||||
runs map[string]*Run
|
||||
}
|
||||
|
||||
func New(sudo, journalSudo bool) *Manager {
|
||||
m := &Manager{sudo: sudo, journalSudo: journalSudo, runs: map[string]*Run{}}
|
||||
if u, err := user.Current(); err == nil {
|
||||
m.self = u.Username
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ---- helpers ----
|
||||
|
||||
func run(ctx context.Context, stdin string, args ...string) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
if stdin != "" {
|
||||
cmd.Stdin = strings.NewReader(stdin)
|
||||
}
|
||||
var out, errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := strings.TrimSpace(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return out.String(), errors.New(msg)
|
||||
}
|
||||
return out.String(), nil
|
||||
}
|
||||
|
||||
// runLines streams a command's stdout to fn line by line, so multi-megabyte journal
|
||||
// dumps never sit in memory whole.
|
||||
func runLines(ctx context.Context, fn func(line []byte), args ...string) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
}
|
||||
sc := bufio.NewScanner(stdout)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
|
||||
for sc.Scan() {
|
||||
fn(sc.Bytes())
|
||||
}
|
||||
if err := cmd.Wait(); err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manager) sudoArgs(args ...string) []string {
|
||||
if m.sudo {
|
||||
return priv.Wrap(args...)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
func jobID(source, raw string) string {
|
||||
h := sha1.Sum([]byte(source + "\n" + raw))
|
||||
return hex.EncodeToString(h[:6])
|
||||
}
|
||||
|
||||
var userRe = regexp.MustCompile(`^[a-z_][a-z0-9_-]{0,31}$`)
|
||||
var cronDName = regexp.MustCompile(`^[A-Za-z0-9_-]+$`) // cron ignores other names in /etc/cron.d
|
||||
|
||||
// ---- parsing ----
|
||||
|
||||
// parseLine splits one crontab line. withUser is true for system crontabs (6th field is
|
||||
// the user). Returns ok=false for comments, blanks, env assignments and garbage.
|
||||
func parseLine(line string, withUser bool) (schedule, usr, command string, ok bool) {
|
||||
t := strings.TrimSpace(line)
|
||||
if t == "" || strings.HasPrefix(t, "#") {
|
||||
return
|
||||
}
|
||||
if i := strings.Index(t, "="); i > 0 && !strings.ContainsAny(t[:i], " \t*") { // NAME=value
|
||||
return
|
||||
}
|
||||
var fields []string
|
||||
if strings.HasPrefix(t, "@") {
|
||||
f := strings.Fields(t)
|
||||
if len(f) < 2 {
|
||||
return
|
||||
}
|
||||
fields = []string{f[0]}
|
||||
t = strings.TrimSpace(t[len(f[0]):])
|
||||
} else {
|
||||
f := strings.Fields(t)
|
||||
if len(f) < 6 {
|
||||
return
|
||||
}
|
||||
fields = f[:5]
|
||||
for _, x := range fields {
|
||||
t = strings.TrimSpace(t[len(x):])
|
||||
}
|
||||
}
|
||||
schedule = strings.Join(fields, " ")
|
||||
if _, err := parseSchedule(schedule); err != nil {
|
||||
return
|
||||
}
|
||||
if withUser {
|
||||
f := strings.Fields(t)
|
||||
if len(f) < 2 || !userRe.MatchString(f[0]) {
|
||||
return
|
||||
}
|
||||
usr = f[0]
|
||||
t = strings.TrimSpace(t[len(f[0]):])
|
||||
}
|
||||
if t == "" {
|
||||
return
|
||||
}
|
||||
return schedule, usr, t, true
|
||||
}
|
||||
|
||||
var redirectRe = regexp.MustCompile(`(?:^|\s)(?:>>|>|2>&1\s*>>|&>>|&>)\s*("?)([^\s"&|;]+)`)
|
||||
|
||||
func outFile(cmd string) string {
|
||||
for _, m := range redirectRe.FindAllStringSubmatch(cmd, -1) {
|
||||
if p := m[2]; strings.HasPrefix(p, "/") && p != "/dev/null" {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// parseSource turns crontab text into jobs. Commented-out lines that still parse as a
|
||||
// job are reported disabled, so a "# 0 5 * * * ..." example shows up as such.
|
||||
func (m *Manager) parseSource(source, text string, withUser, editable bool, defaultUser string, now time.Time, anacron bool) []Job {
|
||||
var jobs []Job
|
||||
lines := strings.Split(text, "\n")
|
||||
isComment := func(i int) bool {
|
||||
return i >= 0 && i < len(lines) && strings.HasPrefix(strings.TrimSpace(lines[i]), "#")
|
||||
}
|
||||
for i, line := range lines {
|
||||
enabled := true
|
||||
probe := line
|
||||
if t := strings.TrimSpace(line); strings.HasPrefix(t, "#") {
|
||||
// A commented-out job sits on its own; a job-shaped line inside a block of
|
||||
// comments is documentation (Debian's stock header has two such examples).
|
||||
if isComment(i-1) && isComment(i+1) {
|
||||
continue
|
||||
}
|
||||
enabled = false
|
||||
probe = strings.TrimLeft(t, "# ")
|
||||
}
|
||||
schedule, usr, command, ok := parseLine(probe, withUser)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if usr == "" {
|
||||
usr = defaultUser
|
||||
}
|
||||
if !enabled {
|
||||
if _, err := user.Lookup(usr); err != nil {
|
||||
continue // "user-name" placeholder and the like
|
||||
}
|
||||
}
|
||||
j := Job{ID: jobID(source, line), Source: source, Line: i + 1, User: usr, Schedule: schedule, Command: command,
|
||||
Enabled: enabled, OutFile: outFile(command), Editable: editable, raw: line}
|
||||
if enabled {
|
||||
if next, ok := nextRun(schedule, now); ok {
|
||||
j.Next = next.UnixMilli()
|
||||
}
|
||||
}
|
||||
if strings.Contains(command, "run-parts") {
|
||||
if dir := runPartsDir(command); dir != "" {
|
||||
if anacron && strings.Contains(command, "anacron") {
|
||||
j.Note = "skipped: anacron is installed and runs " + dir + " instead"
|
||||
}
|
||||
jobs = append(jobs, j)
|
||||
jobs = append(jobs, m.runPartsJobs(dir, j)...)
|
||||
continue
|
||||
}
|
||||
}
|
||||
jobs = append(jobs, j)
|
||||
}
|
||||
return jobs
|
||||
}
|
||||
|
||||
var runPartsRe = regexp.MustCompile(`run-parts\s+(?:--\S+\s+)*(/\S+)`)
|
||||
|
||||
func runPartsDir(cmd string) string {
|
||||
if m := runPartsRe.FindStringSubmatch(cmd); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// runPartsJobs lists the executable scripts a run-parts line will run, as read-only jobs.
|
||||
func (m *Manager) runPartsJobs(dir string, parent Job) []Job {
|
||||
ents, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []Job
|
||||
for _, e := range ents {
|
||||
if e.IsDir() || !cronDName.MatchString(e.Name()) {
|
||||
continue
|
||||
}
|
||||
info, err := e.Info()
|
||||
if err != nil || info.Mode()&0o111 == 0 {
|
||||
continue
|
||||
}
|
||||
p := filepath.Join(dir, e.Name())
|
||||
out = append(out, Job{ID: jobID(dir, e.Name()), Source: dir, User: parent.User, Schedule: parent.Schedule, Command: p,
|
||||
Enabled: parent.Enabled, Next: parent.Next, Note: "run by " + parent.Source + " via run-parts", raw: e.Name()})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- sources ----
|
||||
|
||||
type source struct {
|
||||
name string // job Source
|
||||
withUser bool
|
||||
user string // default user for user crontabs
|
||||
editable bool
|
||||
}
|
||||
|
||||
func (m *Manager) readSource(ctx context.Context, s source) (string, error) {
|
||||
if strings.HasPrefix(s.name, "crontab:") {
|
||||
u := s.user
|
||||
if u == m.self {
|
||||
out, err := run(ctx, "", "crontab", "-l")
|
||||
if err != nil && strings.Contains(err.Error(), "no crontab") {
|
||||
return "", nil
|
||||
}
|
||||
return out, err
|
||||
}
|
||||
if !m.sudo {
|
||||
return "", errors.New("needs sudo")
|
||||
}
|
||||
out, err := run(ctx, "", priv.Wrap("crontab", "-u", u, "-l")...)
|
||||
if err != nil && strings.Contains(err.Error(), "no crontab") {
|
||||
return "", nil
|
||||
}
|
||||
return out, err
|
||||
}
|
||||
b, err := os.ReadFile(s.name)
|
||||
if err != nil && m.sudo {
|
||||
out, err2 := run(ctx, "", priv.Wrap("cat", s.name)...)
|
||||
return out, err2
|
||||
}
|
||||
return string(b), err
|
||||
}
|
||||
|
||||
func (m *Manager) sources(ctx context.Context) []source {
|
||||
var out []source
|
||||
if _, err := os.Stat("/etc/crontab"); err == nil {
|
||||
out = append(out, source{name: "/etc/crontab", withUser: true, editable: m.sudo})
|
||||
}
|
||||
if ents, err := os.ReadDir("/etc/cron.d"); err == nil {
|
||||
for _, e := range ents {
|
||||
if !e.IsDir() && cronDName.MatchString(e.Name()) {
|
||||
out = append(out, source{name: "/etc/cron.d/" + e.Name(), withUser: true, editable: m.sudo})
|
||||
}
|
||||
}
|
||||
}
|
||||
// per-user crontabs: Debian keeps them in /var/spool/cron/crontabs, cronie in /var/spool/cron
|
||||
users := map[string]bool{}
|
||||
if m.sudo {
|
||||
for _, dir := range []string{"/var/spool/cron/crontabs", "/var/spool/cron"} {
|
||||
if out, err := run(ctx, "", priv.Wrap("ls", "-1", dir)...); err == nil {
|
||||
for _, n := range strings.Split(out, "\n") {
|
||||
if n = strings.TrimSpace(n); userRe.MatchString(n) {
|
||||
users[n] = true
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if m.self != "" {
|
||||
users[m.self] = true
|
||||
}
|
||||
names := make([]string, 0, len(users))
|
||||
for u := range users {
|
||||
names = append(names, u)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, u := range names {
|
||||
out = append(out, source{name: "crontab:" + u, user: u, editable: u == m.self || m.sudo})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) findSource(ctx context.Context, name string) (source, bool) {
|
||||
for _, s := range m.sources(ctx) {
|
||||
if s.name == name {
|
||||
return s, true
|
||||
}
|
||||
}
|
||||
return source{}, false
|
||||
}
|
||||
|
||||
// ---- overview ----
|
||||
|
||||
func (m *Manager) Overview(ctx context.Context) Overview {
|
||||
now := time.Now()
|
||||
_, anacronErr := os.Stat("/usr/sbin/anacron")
|
||||
anacron := anacronErr == nil
|
||||
ov := Overview{Jobs: []Job{}, Timers: []Timer{}, Anacron: anacron, Sudo: m.sudo}
|
||||
for _, s := range m.sources(ctx) {
|
||||
text, err := m.readSource(ctx, s)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
ov.Jobs = append(ov.Jobs, m.parseSource(s.name, text, s.withUser, s.editable, s.user, now, anacron)...)
|
||||
}
|
||||
last := m.lastRunsFromJournal(ctx)
|
||||
ov.Logging = m.logging
|
||||
for i := range ov.Jobs {
|
||||
j := &ov.Jobs[i]
|
||||
if t, ok := last[j.User+"\x00"+j.Command]; ok {
|
||||
j.Last = t
|
||||
}
|
||||
}
|
||||
ov.Timers = m.timers(ctx)
|
||||
return ov
|
||||
}
|
||||
|
||||
var cronLogRe = regexp.MustCompile(`^\((\S+)\) CMD \((.*)\)$`)
|
||||
|
||||
// lastRunsFromJournal maps user+command to the most recent "(user) CMD (command)" line.
|
||||
// The first call scans 30 days; later calls only read entries after the last cursor, so
|
||||
// the cost per refresh is proportional to new cron activity, not to history.
|
||||
func (m *Manager) lastRunsFromJournal(ctx context.Context) map[string]int64 {
|
||||
m.mu.Lock()
|
||||
if time.Since(m.lastAt) < 2*time.Second && m.lastRuns != nil {
|
||||
r := m.lastRuns
|
||||
m.mu.Unlock()
|
||||
return r
|
||||
}
|
||||
cursor := m.lastCursor
|
||||
res := map[string]int64{}
|
||||
for k, v := range m.lastRuns {
|
||||
res[k] = v
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
args := []string{"journalctl", "-t", "CRON", "-t", "cron", "-t", "crond", "-o", "json", "--no-pager", "-q"}
|
||||
if cursor != "" {
|
||||
args = append(args, "--after-cursor="+cursor)
|
||||
} else {
|
||||
args = append(args, "--since", "-30d", "-n", "20000")
|
||||
}
|
||||
if m.journalSudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
seen := m.logging
|
||||
_ = runLines(ctx, func(line []byte) {
|
||||
var raw struct {
|
||||
Msg any `json:"MESSAGE"`
|
||||
TS string `json:"__REALTIME_TIMESTAMP"`
|
||||
Cursor string `json:"__CURSOR"`
|
||||
}
|
||||
if json.Unmarshal(line, &raw) != nil {
|
||||
return
|
||||
}
|
||||
seen = true
|
||||
if raw.Cursor != "" {
|
||||
cursor = raw.Cursor
|
||||
}
|
||||
msg, _ := raw.Msg.(string)
|
||||
mm := cronLogRe.FindStringSubmatch(msg)
|
||||
if mm == nil {
|
||||
return
|
||||
}
|
||||
us, _ := strconv.ParseInt(raw.TS, 10, 64)
|
||||
k := mm[1] + "\x00" + mm[2]
|
||||
if us/1000 > res[k] {
|
||||
res[k] = us / 1000
|
||||
}
|
||||
}, args...)
|
||||
m.mu.Lock()
|
||||
m.lastRuns, m.lastAt, m.lastCursor, m.logging = res, time.Now(), cursor, seen
|
||||
m.mu.Unlock()
|
||||
return res
|
||||
}
|
||||
|
||||
// JobLogs returns the journal lines for one job's command (most recent last).
|
||||
func (m *Manager) JobLogs(ctx context.Context, j Job, n int) ([]map[string]any, error) {
|
||||
args := []string{"journalctl", "-t", "CRON", "-t", "cron", "-t", "crond", "-o", "json", "--no-pager", "-q", "-n", "20000"}
|
||||
if m.journalSudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
var res []map[string]any
|
||||
needle := "(" + j.User + ") CMD (" + j.Command + ")"
|
||||
err := runLines(ctx, func(line []byte) {
|
||||
if !bytes.Contains(line, []byte(j.Command[:min(len(j.Command), 40)])) { // cheap pre-filter before JSON
|
||||
return
|
||||
}
|
||||
var raw struct {
|
||||
Msg any `json:"MESSAGE"`
|
||||
TS string `json:"__REALTIME_TIMESTAMP"`
|
||||
PID string `json:"_PID"`
|
||||
}
|
||||
if json.Unmarshal(line, &raw) != nil {
|
||||
return
|
||||
}
|
||||
msg, _ := raw.Msg.(string)
|
||||
if !strings.Contains(msg, needle) {
|
||||
return
|
||||
}
|
||||
us, _ := strconv.ParseInt(raw.TS, 10, 64)
|
||||
res = append(res, map[string]any{"t": us / 1000, "p": 6, "ident": "CRON", "pid": raw.PID, "msg": msg})
|
||||
}, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(res) > n {
|
||||
res = res[len(res)-n:]
|
||||
}
|
||||
if res == nil {
|
||||
res = []map[string]any{}
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// OutputTail returns the last n lines of the job's redirect target.
|
||||
func (m *Manager) OutputTail(ctx context.Context, j Job, n int) (string, error) {
|
||||
if j.OutFile == "" {
|
||||
return "", errors.New("job has no output file")
|
||||
}
|
||||
out, err := run(ctx, "", m.sudoArgs("tail", "-n", strconv.Itoa(n), j.OutFile)...)
|
||||
return out, err
|
||||
}
|
||||
|
||||
// ---- timers ----
|
||||
|
||||
// timers: one `systemctl show` over every loaded timer instead of three list commands.
|
||||
func (m *Manager) timers(ctx context.Context) []Timer {
|
||||
out, err := run(ctx, "", "systemctl", "show", "*.timer", "--no-pager",
|
||||
"-p", "Id,Description,ActiveState,SubState,UnitFileState,NextElapseUSecRealtime,LastTriggerUSec,Triggers")
|
||||
if err != nil {
|
||||
return []Timer{}
|
||||
}
|
||||
unix := func(v string) int64 { // "Sat 2026-09-19 05:47:13 PDT", "n/a" or ""
|
||||
for _, layout := range []string{"Mon 2006-01-02 15:04:05 MST", "2006-01-02 15:04:05 MST", "Mon 2006-01-02 15:04:05"} {
|
||||
if t, err := time.ParseInLocation(layout, v, time.Local); err == nil {
|
||||
return t.UnixMilli()
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
res := []Timer{}
|
||||
for _, block := range strings.Split(out, "\n\n") {
|
||||
t := Timer{}
|
||||
for _, line := range strings.Split(block, "\n") {
|
||||
k, v, ok := strings.Cut(line, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
switch k {
|
||||
case "Id":
|
||||
t.Unit = v
|
||||
case "Description":
|
||||
t.Description = v
|
||||
case "ActiveState":
|
||||
t.Active = v
|
||||
case "SubState":
|
||||
t.Sub = v
|
||||
case "UnitFileState":
|
||||
t.Enabled = v
|
||||
case "NextElapseUSecRealtime":
|
||||
t.Next = unix(v)
|
||||
case "LastTriggerUSec":
|
||||
t.Last = unix(v)
|
||||
case "Triggers":
|
||||
t.Activates = strings.Fields(v + " ")[0:1][0]
|
||||
}
|
||||
}
|
||||
if t.Unit != "" {
|
||||
res = append(res, t)
|
||||
}
|
||||
}
|
||||
sort.Slice(res, func(i, j int) bool { return res[i].Unit < res[j].Unit })
|
||||
return res
|
||||
}
|
||||
|
||||
var timerRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9@._:\\-]{0,255}\.timer$`)
|
||||
|
||||
func (m *Manager) TimerAction(ctx context.Context, unit, action string) error {
|
||||
if !timerRe.MatchString(unit) {
|
||||
return errors.New("invalid timer name")
|
||||
}
|
||||
switch action {
|
||||
case "start", "stop", "enable", "disable":
|
||||
_, err := run(ctx, "", m.sudoArgs("systemctl", action, unit)...)
|
||||
return err
|
||||
case "run": // fire the service the timer activates, right now
|
||||
out, err := run(ctx, "", "systemctl", "show", unit, "-p", "Triggers", "--value")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
svc := strings.TrimSpace(strings.Split(out, " ")[0])
|
||||
if svc == "" {
|
||||
return errors.New("timer triggers nothing")
|
||||
}
|
||||
_, err = run(ctx, "", m.sudoArgs("systemctl", "start", svc)...)
|
||||
return err
|
||||
}
|
||||
return errors.New("unknown action")
|
||||
}
|
||||
|
||||
// ---- job lookup and editing ----
|
||||
|
||||
func (m *Manager) FindJob(ctx context.Context, id string) (Job, bool) {
|
||||
for _, j := range m.Overview(ctx).Jobs {
|
||||
if j.ID == id {
|
||||
return j, true
|
||||
}
|
||||
}
|
||||
return Job{}, false
|
||||
}
|
||||
|
||||
// Source returns a crontab's raw text for the editor.
|
||||
func (m *Manager) Source(ctx context.Context, name string) (text string, editable bool, err error) {
|
||||
s, ok := m.findSource(ctx, name)
|
||||
if !ok {
|
||||
return "", false, errors.New("unknown source")
|
||||
}
|
||||
text, err = m.readSource(ctx, s)
|
||||
return text, s.editable, err
|
||||
}
|
||||
|
||||
// validate rejects any line that is neither blank, comment, env assignment, nor a job.
|
||||
func validate(text string, withUser bool) error {
|
||||
for i, line := range strings.Split(text, "\n") {
|
||||
t := strings.TrimSpace(line)
|
||||
if t == "" || strings.HasPrefix(t, "#") {
|
||||
continue
|
||||
}
|
||||
if k, _, ok := strings.Cut(t, "="); ok && !strings.ContainsAny(strings.TrimSpace(k), " \t*") {
|
||||
continue
|
||||
}
|
||||
if _, _, _, ok := parseLine(t, withUser); !ok {
|
||||
return fmt.Errorf("line %d: not a valid cron entry", i+1)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// WriteSource replaces a crontab. User crontabs go through crontab(1) so cron reloads
|
||||
// them; system files are written in place with their permissions preserved.
|
||||
func (m *Manager) WriteSource(ctx context.Context, name, text string) error {
|
||||
s, ok := m.findSource(ctx, name)
|
||||
if !ok {
|
||||
return errors.New("unknown source")
|
||||
}
|
||||
if !s.editable {
|
||||
return errors.New("source is read-only here (no sudo)")
|
||||
}
|
||||
if !strings.HasSuffix(text, "\n") {
|
||||
text += "\n"
|
||||
}
|
||||
if err := validate(text, s.withUser); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.HasPrefix(name, "crontab:") {
|
||||
if s.user == m.self {
|
||||
_, err := run(ctx, text, "crontab", "-")
|
||||
return err
|
||||
}
|
||||
_, err := run(ctx, text, priv.Wrap("crontab", "-u", s.user, "-")...)
|
||||
return err
|
||||
}
|
||||
if name != "/etc/crontab" && !strings.HasPrefix(name, "/etc/cron.d/") {
|
||||
return errors.New("refusing to write outside /etc/crontab and /etc/cron.d")
|
||||
}
|
||||
_, err := run(ctx, text, priv.Wrap("tee", name)...)
|
||||
m.mu.Lock()
|
||||
m.lastAt = time.Time{}
|
||||
m.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
|
||||
// Toggle comments or uncomments a job's line. The line must still read as the job did
|
||||
// when it was listed, so a concurrent edit can't be clobbered.
|
||||
func (m *Manager) Toggle(ctx context.Context, j Job, enable bool) error {
|
||||
if j.Line == 0 || !j.Editable {
|
||||
return errors.New("this job can't be toggled here")
|
||||
}
|
||||
s, ok := m.findSource(ctx, j.Source)
|
||||
if !ok {
|
||||
return errors.New("unknown source")
|
||||
}
|
||||
text, err := m.readSource(ctx, s)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
lines := strings.Split(text, "\n")
|
||||
if j.Line > len(lines) || lines[j.Line-1] != j.raw {
|
||||
return errors.New("crontab changed underneath; reload and try again")
|
||||
}
|
||||
if enable {
|
||||
lines[j.Line-1] = strings.TrimLeft(strings.TrimSpace(j.raw), "# ")
|
||||
} else {
|
||||
lines[j.Line-1] = "#" + j.raw
|
||||
}
|
||||
return m.WriteSource(ctx, j.Source, strings.Join(lines, "\n"))
|
||||
}
|
||||
|
||||
// ---- run now ----
|
||||
|
||||
// StartRun executes the job's command as its user, the way cron would (sh -c, cwd = home,
|
||||
// '%' turned into newlines/stdin), in the background. Output is kept in memory.
|
||||
func (m *Manager) StartRun(ctx context.Context, j Job) (*Run, error) {
|
||||
if j.User != m.self && !m.sudo {
|
||||
return nil, errors.New("needs sudo to run as " + j.User)
|
||||
}
|
||||
cmdline, stdin := cronPercent(j.Command)
|
||||
args := []string{"sh", "-c", cmdline}
|
||||
if j.User != m.self {
|
||||
args = priv.AsUser(j.User, args...)
|
||||
}
|
||||
r := &Run{ID: jobID("run", strconv.FormatInt(time.Now().UnixNano(), 10)), JobID: j.ID, Command: j.Command, User: j.User, Started: time.Now(), Running: true}
|
||||
runCtx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
cmd := exec.CommandContext(runCtx, args[0], args[1:]...)
|
||||
if stdin != "" {
|
||||
cmd.Stdin = strings.NewReader(stdin)
|
||||
}
|
||||
w := &capWriter{r: r}
|
||||
cmd.Stdout, cmd.Stderr = w, w
|
||||
if err := cmd.Start(); err != nil {
|
||||
cancel()
|
||||
return nil, err
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.runs[r.ID] = r
|
||||
for id, old := range m.runs { // keep the map small
|
||||
if !old.Running && time.Since(old.Finished) > time.Hour {
|
||||
delete(m.runs, id)
|
||||
}
|
||||
}
|
||||
m.mu.Unlock()
|
||||
go func() {
|
||||
defer cancel()
|
||||
err := cmd.Wait()
|
||||
r.mu.Lock()
|
||||
r.Running, r.Finished = false, time.Now()
|
||||
if err != nil {
|
||||
if ee, ok := err.(*exec.ExitError); ok {
|
||||
r.ExitCode = ee.ExitCode()
|
||||
} else {
|
||||
r.ExitCode = -1
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
m.mu.Lock()
|
||||
m.lastAt = time.Time{}
|
||||
m.mu.Unlock()
|
||||
}()
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (m *Manager) GetRun(id string) (*Run, bool) {
|
||||
m.mu.Lock()
|
||||
r, ok := m.runs[id]
|
||||
m.mu.Unlock()
|
||||
return r, ok
|
||||
}
|
||||
|
||||
// Snapshot returns a copy safe to encode.
|
||||
func (r *Run) Snapshot() Run {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return Run{ID: r.ID, JobID: r.JobID, Command: r.Command, User: r.User, Started: r.Started, Finished: r.Finished,
|
||||
Running: r.Running, ExitCode: r.ExitCode, Output: r.buf.String()}
|
||||
}
|
||||
|
||||
type capWriter struct{ r *Run }
|
||||
|
||||
func (w *capWriter) Write(p []byte) (int, error) {
|
||||
w.r.mu.Lock()
|
||||
defer w.r.mu.Unlock()
|
||||
w.r.buf.Write(p)
|
||||
if w.r.buf.Len() > 256<<10 { // keep the tail
|
||||
b := w.r.buf.Bytes()
|
||||
w.r.buf = *bytes.NewBuffer(append([]byte("…\n"), b[len(b)-200<<10:]...))
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
// cronPercent applies crontab(5) rules: an unescaped % ends the command; later ones are
|
||||
// newlines fed to stdin. "\%" is a literal percent.
|
||||
func cronPercent(cmd string) (string, string) {
|
||||
var out, in strings.Builder
|
||||
cur := &out
|
||||
esc := false
|
||||
for _, c := range cmd {
|
||||
switch {
|
||||
case esc:
|
||||
if c != '%' {
|
||||
cur.WriteRune('\\')
|
||||
}
|
||||
cur.WriteRune(c)
|
||||
esc = false
|
||||
case c == '\\':
|
||||
esc = true
|
||||
case c == '%':
|
||||
if cur == &out {
|
||||
cur = &in
|
||||
} else {
|
||||
in.WriteRune('\n')
|
||||
}
|
||||
default:
|
||||
cur.WriteRune(c)
|
||||
}
|
||||
}
|
||||
return out.String(), in.String()
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
package cron
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// schedule is a parsed five-field cron expression as sets.
|
||||
type schedule struct {
|
||||
min, hour, dom, mon, dow [64]bool
|
||||
domStar, dowStar bool // "*" in dom/dow changes the match rule
|
||||
reboot bool
|
||||
}
|
||||
|
||||
var monthNames = map[string]int{"jan": 1, "feb": 2, "mar": 3, "apr": 4, "may": 5, "jun": 6, "jul": 7, "aug": 8, "sep": 9, "oct": 10, "nov": 11, "dec": 12}
|
||||
var dayNames = map[string]int{"sun": 0, "mon": 1, "tue": 2, "wed": 3, "thu": 4, "fri": 5, "sat": 6}
|
||||
|
||||
var aliases = map[string]string{
|
||||
"@yearly": "0 0 1 1 *", "@annually": "0 0 1 1 *", "@monthly": "0 0 1 * *", "@weekly": "0 0 * * 0",
|
||||
"@daily": "0 0 * * *", "@midnight": "0 0 * * *", "@hourly": "0 * * * *",
|
||||
}
|
||||
|
||||
func parseSchedule(expr string) (*schedule, error) {
|
||||
expr = strings.TrimSpace(expr)
|
||||
if expr == "@reboot" {
|
||||
return &schedule{reboot: true}, nil
|
||||
}
|
||||
if a, ok := aliases[strings.ToLower(expr)]; ok {
|
||||
expr = a
|
||||
}
|
||||
f := strings.Fields(expr)
|
||||
if len(f) != 5 {
|
||||
return nil, errors.New("expected 5 fields")
|
||||
}
|
||||
s := &schedule{}
|
||||
var err error
|
||||
if s.min, err = parseField(f[0], 0, 59, nil); err != nil {
|
||||
return nil, fmt.Errorf("minute: %w", err)
|
||||
}
|
||||
if s.hour, err = parseField(f[1], 0, 23, nil); err != nil {
|
||||
return nil, fmt.Errorf("hour: %w", err)
|
||||
}
|
||||
if s.dom, err = parseField(f[2], 1, 31, nil); err != nil {
|
||||
return nil, fmt.Errorf("day of month: %w", err)
|
||||
}
|
||||
if s.mon, err = parseField(f[3], 1, 12, monthNames); err != nil {
|
||||
return nil, fmt.Errorf("month: %w", err)
|
||||
}
|
||||
if s.dow, err = parseField(f[4], 0, 7, dayNames); err != nil {
|
||||
return nil, fmt.Errorf("day of week: %w", err)
|
||||
}
|
||||
if s.dow[7] { // 7 == Sunday
|
||||
s.dow[0] = true
|
||||
}
|
||||
s.domStar = f[2] == "*"
|
||||
s.dowStar = f[4] == "*"
|
||||
return s, nil
|
||||
}
|
||||
|
||||
func parseField(field string, lo, hi int, names map[string]int) ([64]bool, error) {
|
||||
var set [64]bool
|
||||
atom := func(s string) (int, error) {
|
||||
if names != nil {
|
||||
if v, ok := names[strings.ToLower(s)]; ok {
|
||||
return v, nil
|
||||
}
|
||||
}
|
||||
v, err := strconv.Atoi(s)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("bad value %q", s)
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
for _, part := range strings.Split(field, ",") {
|
||||
step := 1
|
||||
if i := strings.Index(part, "/"); i >= 0 {
|
||||
v, err := strconv.Atoi(part[i+1:])
|
||||
if err != nil || v <= 0 {
|
||||
return set, fmt.Errorf("bad step in %q", part)
|
||||
}
|
||||
step, part = v, part[:i]
|
||||
}
|
||||
a, b := lo, hi
|
||||
switch {
|
||||
case part == "*":
|
||||
case strings.Contains(part, "-"):
|
||||
x, y, _ := strings.Cut(part, "-")
|
||||
var err error
|
||||
if a, err = atom(x); err != nil {
|
||||
return set, err
|
||||
}
|
||||
if b, err = atom(y); err != nil {
|
||||
return set, err
|
||||
}
|
||||
default:
|
||||
v, err := atom(part)
|
||||
if err != nil {
|
||||
return set, err
|
||||
}
|
||||
a = v
|
||||
if step == 1 {
|
||||
b = v
|
||||
}
|
||||
}
|
||||
if a < lo || b > hi || a > b {
|
||||
return set, fmt.Errorf("out of range: %q", part)
|
||||
}
|
||||
for v := a; v <= b; v += step {
|
||||
set[v] = true
|
||||
}
|
||||
}
|
||||
return set, nil
|
||||
}
|
||||
|
||||
func (s *schedule) dayMatches(t time.Time) bool {
|
||||
if !s.mon[int(t.Month())] {
|
||||
return false
|
||||
}
|
||||
dom, dow := s.dom[t.Day()], s.dow[int(t.Weekday())]
|
||||
switch {
|
||||
case s.domStar && s.dowStar:
|
||||
return true
|
||||
case s.domStar:
|
||||
return dow
|
||||
case s.dowStar:
|
||||
return dom
|
||||
default: // both restricted: cron runs when either matches
|
||||
return dom || dow
|
||||
}
|
||||
}
|
||||
|
||||
// NextRun is the first time strictly after `from` that a cron expression fires. Exported so
|
||||
// anything that writes a cron line — the ZFS snapshot schedules — can say when it runs next
|
||||
// using the same evaluator that reads everyone else's.
|
||||
func NextRun(expr string, from time.Time) (time.Time, bool) { return nextRun(expr, from) }
|
||||
|
||||
// ValidSchedule reports whether a cron expression parses.
|
||||
func ValidSchedule(expr string) error {
|
||||
_, err := parseSchedule(expr)
|
||||
return err
|
||||
}
|
||||
|
||||
// nextRun finds the first time strictly after `from` that the schedule fires.
|
||||
func nextRun(expr string, from time.Time) (time.Time, bool) {
|
||||
s, err := parseSchedule(expr)
|
||||
if err != nil || s.reboot {
|
||||
return time.Time{}, false
|
||||
}
|
||||
t := from.Truncate(time.Minute).Add(time.Minute)
|
||||
day := time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, t.Location())
|
||||
for i := 0; i < 366*2; i++ {
|
||||
if s.dayMatches(day) {
|
||||
for h := 0; h < 24; h++ {
|
||||
if !s.hour[h] {
|
||||
continue
|
||||
}
|
||||
for mi := 0; mi < 60; mi++ {
|
||||
if !s.min[mi] {
|
||||
continue
|
||||
}
|
||||
c := time.Date(day.Year(), day.Month(), day.Day(), h, mi, 0, 0, day.Location())
|
||||
if !c.Before(t) {
|
||||
return c, true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
day = day.AddDate(0, 0, 1)
|
||||
}
|
||||
return time.Time{}, false
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package cron
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestNextRun(t *testing.T) {
|
||||
loc := time.UTC
|
||||
from := time.Date(2026, 9, 17, 15, 30, 0, 0, loc) // Thursday
|
||||
cases := []struct {
|
||||
expr string
|
||||
want time.Time
|
||||
}{
|
||||
{"0 1 * * *", time.Date(2026, 9, 18, 1, 0, 0, 0, loc)},
|
||||
{"17 * * * *", time.Date(2026, 9, 17, 16, 17, 0, 0, loc)},
|
||||
{"*/15 * * * *", time.Date(2026, 9, 17, 15, 45, 0, 0, loc)},
|
||||
{"47 6 * * 7", time.Date(2026, 9, 20, 6, 47, 0, 0, loc)}, // Sunday as 7
|
||||
{"52 6 1 * *", time.Date(2026, 10, 1, 6, 52, 0, 0, loc)}, // first of month
|
||||
{"24 0 1-7 * *", time.Date(2026, 10, 1, 0, 24, 0, 0, loc)}, // range
|
||||
{"30 3 * * 0", time.Date(2026, 9, 20, 3, 30, 0, 0, loc)}, // Sunday as 0
|
||||
{"0 9 15 * mon", time.Date(2026, 9, 21, 9, 0, 0, 0, loc)}, // dom OR dow
|
||||
{"@daily", time.Date(2026, 9, 18, 0, 0, 0, 0, loc)},
|
||||
{"@weekly", time.Date(2026, 9, 20, 0, 0, 0, 0, loc)},
|
||||
{"0 0 29 feb *", time.Date(2028, 2, 29, 0, 0, 0, 0, loc)},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, ok := nextRun(c.expr, from)
|
||||
if !ok || !got.Equal(c.want) {
|
||||
t.Errorf("%q: got %v ok=%v, want %v", c.expr, got, ok, c.want)
|
||||
}
|
||||
}
|
||||
if _, ok := nextRun("@reboot", from); ok {
|
||||
t.Error("@reboot should have no next run")
|
||||
}
|
||||
for _, bad := range []string{"* * * *", "60 * * * *", "a b c d e", "*/0 * * * *"} {
|
||||
if _, err := parseSchedule(bad); err == nil {
|
||||
t.Errorf("%q should fail", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLine(t *testing.T) {
|
||||
sch, u, cmd, ok := parseLine("25 6\t* * *\troot\ttest -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }", true)
|
||||
if !ok || sch != "25 6 * * *" || u != "root" || !strings_HasPrefix(cmd, "test -x") {
|
||||
t.Errorf("system line: %q %q %q %v", sch, u, cmd, ok)
|
||||
}
|
||||
if _, _, _, ok := parseLine("PATH=/usr/bin:/bin", true); ok {
|
||||
t.Error("env line parsed as job")
|
||||
}
|
||||
if _, _, _, ok := parseLine("# m h dom mon dow command", false); ok {
|
||||
t.Error("comment parsed as job")
|
||||
}
|
||||
if _, u, cmd, ok := parseLine("@reboot /usr/local/bin/thing --flag", false); !ok || u != "" || cmd != "/usr/local/bin/thing --flag" {
|
||||
t.Errorf("@reboot: %q %q %v", u, cmd, ok)
|
||||
}
|
||||
if got := outFile("foo >> /var/log/foo.log 2>&1"); got != "/var/log/foo.log" {
|
||||
t.Errorf("outFile: %q", got)
|
||||
}
|
||||
if c, in := cronPercent("echo hi%line1%line2"); c != "echo hi" || in != "line1\nline2" {
|
||||
t.Errorf("percent: %q %q", c, in)
|
||||
}
|
||||
if c, _ := cronPercent(`date +\%w`); c != "date +%w" {
|
||||
t.Errorf("escaped percent: %q", c)
|
||||
}
|
||||
}
|
||||
|
||||
func strings_HasPrefix(s, p string) bool { return len(s) >= len(p) && s[:len(p)] == p }
|
||||
@@ -0,0 +1,571 @@
|
||||
// Package files is the Files tab: browse, read, write, upload, download, mkdir,
|
||||
// rename, chmod and delete anywhere on the host.
|
||||
//
|
||||
// Two backends share one interface. Local runs in-process as the daemon's user. Sudo
|
||||
// re-executes this same binary as root (`sudo -n hamlab fs …`) for one operation at a
|
||||
// time, streaming file bodies over its stdin/stdout, so root access uses exactly the
|
||||
// same code and the same validation as the unprivileged path.
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/media"
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type Entry struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"` // dir | file | link | other
|
||||
Size int64 `json:"size"`
|
||||
Mode string `json:"mode"` // "-rw-r--r--"
|
||||
Perm string `json:"perm"` // "0644"
|
||||
Mtime int64 `json:"mtime"`
|
||||
User string `json:"user"`
|
||||
Group string `json:"group"`
|
||||
Target string `json:"target,omitempty"` // link target
|
||||
Broken bool `json:"broken,omitempty"` // dangling link
|
||||
Preview string `json:"preview,omitempty"` // "image" | "video": the Files tab can show a thumbnail of it
|
||||
}
|
||||
|
||||
type Listing struct {
|
||||
Path string `json:"path"`
|
||||
Parent string `json:"parent,omitempty"`
|
||||
Entries []Entry `json:"entries"`
|
||||
}
|
||||
|
||||
type Info struct {
|
||||
Entry
|
||||
Path string `json:"path"`
|
||||
Text bool `json:"text"` // looks like text (for the editor)
|
||||
}
|
||||
|
||||
// FS is what the API talks to.
|
||||
type FS interface {
|
||||
List(ctx context.Context, path string) (Listing, error)
|
||||
Stat(ctx context.Context, path string) (Info, error)
|
||||
Open(ctx context.Context, path string) (io.ReadCloser, error)
|
||||
OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) // Open from a byte offset: a range request, a player seeking
|
||||
Write(ctx context.Context, path string, r io.Reader) error
|
||||
Mkdir(ctx context.Context, path string) error
|
||||
Touch(ctx context.Context, path string) error
|
||||
Rename(ctx context.Context, from, to string) error
|
||||
Move(ctx context.Context, from, to string, progress io.Writer) error // Rename, or across filesystems a copy and a removal; long, so callers run it as a task
|
||||
Delete(ctx context.Context, path string, recursive bool) error
|
||||
Chmod(ctx context.Context, path string, perm os.FileMode) error
|
||||
}
|
||||
|
||||
// ---- path rules ----
|
||||
|
||||
// Clean rejects relative paths and normalises the rest. Everything else on the host is
|
||||
// reachable on purpose: this is a root-capable console, the gate is the login.
|
||||
func Clean(p string) (string, error) {
|
||||
if p == "" {
|
||||
return "", errors.New("path required")
|
||||
}
|
||||
if !strings.HasPrefix(p, "/") {
|
||||
return "", errors.New("path must be absolute")
|
||||
}
|
||||
c := filepath.Clean(p)
|
||||
if strings.ContainsRune(c, 0) {
|
||||
return "", errors.New("bad path")
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// protected are never deleted or renamed as a whole, no matter who asks.
|
||||
var protected = map[string]bool{"/": true, "/bin": true, "/boot": true, "/dev": true, "/etc": true, "/home": true, "/lib": true, "/lib64": true,
|
||||
"/opt": true, "/proc": true, "/root": true, "/run": true, "/sbin": true, "/srv": true, "/sys": true, "/tmp": true, "/usr": true, "/var": true}
|
||||
|
||||
func ParsePerm(s string) (os.FileMode, error) {
|
||||
v, err := strconv.ParseUint(strings.TrimPrefix(s, "0"), 8, 32)
|
||||
if err != nil || v > 0o7777 {
|
||||
return 0, errors.New("mode must be octal like 0644")
|
||||
}
|
||||
return os.FileMode(v), nil
|
||||
}
|
||||
|
||||
// ---- Local ----
|
||||
|
||||
type Local struct{}
|
||||
|
||||
var uidCache, gidCache = map[uint32]string{}, map[uint32]string{}
|
||||
|
||||
func owner(st os.FileInfo) (string, string) {
|
||||
sys, ok := st.Sys().(*syscall.Stat_t)
|
||||
if !ok {
|
||||
return "", ""
|
||||
}
|
||||
u, ok := uidCache[sys.Uid]
|
||||
if !ok {
|
||||
u = strconv.Itoa(int(sys.Uid))
|
||||
if x, err := user.LookupId(u); err == nil {
|
||||
u = x.Username
|
||||
}
|
||||
uidCache[sys.Uid] = u
|
||||
}
|
||||
g, ok := gidCache[sys.Gid]
|
||||
if !ok {
|
||||
g = strconv.Itoa(int(sys.Gid))
|
||||
if x, err := user.LookupGroupId(g); err == nil {
|
||||
g = x.Name
|
||||
}
|
||||
gidCache[sys.Gid] = g
|
||||
}
|
||||
return u, g
|
||||
}
|
||||
|
||||
func entryOf(dir, name string, lst os.FileInfo) Entry {
|
||||
e := Entry{Name: name, Size: lst.Size(), Mode: lst.Mode().String(), Perm: fmt.Sprintf("%04o", lst.Mode().Perm()), Mtime: lst.ModTime().UnixMilli()}
|
||||
e.User, e.Group = owner(lst)
|
||||
switch {
|
||||
case lst.Mode()&os.ModeSymlink != 0:
|
||||
e.Type = "link"
|
||||
full := filepath.Join(dir, name)
|
||||
e.Target, _ = os.Readlink(full)
|
||||
if st, err := os.Stat(full); err != nil {
|
||||
e.Broken = true
|
||||
} else if st.IsDir() {
|
||||
e.Type = "dir" // keep dir semantics for navigation; Target says it's a link
|
||||
} else {
|
||||
e.Size, e.Mtime = st.Size(), st.ModTime().UnixMilli() // the file's, not the link's
|
||||
}
|
||||
case lst.IsDir():
|
||||
e.Type = "dir"
|
||||
case lst.Mode().IsRegular():
|
||||
e.Type = "file"
|
||||
default:
|
||||
e.Type = "other"
|
||||
}
|
||||
if e.Type == "file" || (e.Type == "link" && !e.Broken) {
|
||||
e.Preview = media.Kind(name)
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
func (Local) List(_ context.Context, path string) (Listing, error) {
|
||||
ents, err := os.ReadDir(path)
|
||||
if err != nil {
|
||||
return Listing{}, err
|
||||
}
|
||||
l := Listing{Path: path, Entries: make([]Entry, 0, len(ents))}
|
||||
if path != "/" {
|
||||
l.Parent = filepath.Dir(path)
|
||||
}
|
||||
for _, d := range ents {
|
||||
lst, err := d.Info() // Lstat
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
l.Entries = append(l.Entries, entryOf(path, d.Name(), lst))
|
||||
}
|
||||
sort.Slice(l.Entries, func(i, j int) bool {
|
||||
a, b := l.Entries[i], l.Entries[j]
|
||||
if (a.Type == "dir") != (b.Type == "dir") {
|
||||
return a.Type == "dir"
|
||||
}
|
||||
return strings.ToLower(a.Name) < strings.ToLower(b.Name)
|
||||
})
|
||||
return l, nil
|
||||
}
|
||||
|
||||
func (Local) Stat(_ context.Context, path string) (Info, error) {
|
||||
lst, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return Info{}, err
|
||||
}
|
||||
in := Info{Entry: entryOf(filepath.Dir(path), filepath.Base(path), lst), Path: path}
|
||||
if in.Type == "file" && in.Size <= 8<<20 {
|
||||
if f, err := os.Open(path); err == nil {
|
||||
buf := make([]byte, 4096)
|
||||
n, _ := f.Read(buf)
|
||||
f.Close()
|
||||
in.Text = looksText(buf[:n])
|
||||
}
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
||||
func looksText(b []byte) bool {
|
||||
if len(b) == 0 {
|
||||
return true
|
||||
}
|
||||
ctrl := 0
|
||||
for _, c := range b {
|
||||
if c == 0 {
|
||||
return false
|
||||
}
|
||||
if c < 32 && c != '\n' && c != '\r' && c != '\t' && c != '\f' && c != 0x1b {
|
||||
ctrl++
|
||||
}
|
||||
}
|
||||
return ctrl*20 < len(b)
|
||||
}
|
||||
|
||||
// Open refuses anything that isn't a regular file: devices stream forever (/dev/zero),
|
||||
// FIFOs block on open, sockets can't be read. /proc files are regular and still work.
|
||||
func (Local) Open(_ context.Context, path string) (io.ReadCloser, error) {
|
||||
st, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if st.IsDir() {
|
||||
return nil, errors.New("is a directory")
|
||||
}
|
||||
if !st.Mode().IsRegular() {
|
||||
return nil, errors.New("not a regular file (device, pipe or socket)")
|
||||
}
|
||||
return os.Open(path)
|
||||
}
|
||||
|
||||
func (l Local) OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) {
|
||||
f, err := l.Open(ctx, path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if offset > 0 {
|
||||
if _, err := f.(*os.File).Seek(offset, io.SeekStart); err != nil {
|
||||
f.Close()
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// Write replaces the file's contents, keeping its mode; new files get 0644 (umask).
|
||||
func (Local) Write(_ context.Context, path string, r io.Reader) error {
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(f, r); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
func (Local) Mkdir(_ context.Context, path string) error { return os.Mkdir(path, 0o755) }
|
||||
|
||||
func (Local) Touch(_ context.Context, path string) error {
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
func (Local) Rename(_ context.Context, from, to string) error {
|
||||
if protected[from] {
|
||||
return errors.New("refusing to move " + from)
|
||||
}
|
||||
if _, err := os.Lstat(to); err == nil {
|
||||
return errors.New("destination exists")
|
||||
}
|
||||
return os.Rename(from, to)
|
||||
}
|
||||
|
||||
func (Local) Delete(_ context.Context, path string, recursive bool) error {
|
||||
if protected[path] {
|
||||
return errors.New("refusing to delete " + path)
|
||||
}
|
||||
lst, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if lst.IsDir() && recursive {
|
||||
return os.RemoveAll(path)
|
||||
}
|
||||
return os.Remove(path) // fails on a non-empty dir, by design
|
||||
}
|
||||
|
||||
func (Local) Chmod(_ context.Context, path string, perm os.FileMode) error {
|
||||
return os.Chmod(path, perm)
|
||||
}
|
||||
|
||||
// ---- Sudo: the same operations, run as root in a child hamlab ----
|
||||
|
||||
type Sudo struct{ Self string }
|
||||
|
||||
func (s Sudo) cmd(ctx context.Context, args ...string) *exec.Cmd {
|
||||
full := priv.Wrap(append([]string{s.Self, "fs"}, args...)...)
|
||||
return exec.CommandContext(ctx, full[0], full[1:]...)
|
||||
}
|
||||
|
||||
func (s Sudo) runJSON(ctx context.Context, out any, args ...string) error {
|
||||
c := s.cmd(ctx, args...)
|
||||
var errb strings.Builder
|
||||
c.Stderr = &errb
|
||||
b, err := c.Output()
|
||||
if err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
return json.Unmarshal(b, out)
|
||||
}
|
||||
|
||||
func (s Sudo) List(ctx context.Context, path string) (Listing, error) {
|
||||
var l Listing
|
||||
err := s.runJSON(ctx, &l, "list", path)
|
||||
return l, err
|
||||
}
|
||||
|
||||
func (s Sudo) Stat(ctx context.Context, path string) (Info, error) {
|
||||
var in Info
|
||||
err := s.runJSON(ctx, &in, "stat", path)
|
||||
return in, err
|
||||
}
|
||||
|
||||
type cmdReader struct {
|
||||
io.ReadCloser
|
||||
cmd *exec.Cmd
|
||||
errb *strings.Builder
|
||||
}
|
||||
|
||||
func (r *cmdReader) Close() error {
|
||||
r.ReadCloser.Close()
|
||||
err := r.cmd.Wait()
|
||||
if err != nil && r.errb.Len() > 0 {
|
||||
return errors.New(strings.TrimSpace(r.errb.String()))
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (s Sudo) Open(ctx context.Context, path string) (io.ReadCloser, error) {
|
||||
return s.open(ctx, "read", path)
|
||||
}
|
||||
|
||||
func (s Sudo) OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) {
|
||||
return s.open(ctx, "read", "--offset", strconv.FormatInt(offset, 10), path)
|
||||
}
|
||||
|
||||
func (s Sudo) open(ctx context.Context, args ...string) (io.ReadCloser, error) {
|
||||
c := s.cmd(ctx, args...)
|
||||
errb := &strings.Builder{}
|
||||
c.Stderr = errb
|
||||
out, err := c.StdoutPipe()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := c.Start(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Fail early on "no such file" etc.: the helper stats before writing anything.
|
||||
head := make([]byte, 1)
|
||||
n, rerr := out.Read(head)
|
||||
if n == 0 && rerr != nil {
|
||||
_ = c.Wait()
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return nil, errors.New(msg)
|
||||
}
|
||||
if rerr != io.EOF {
|
||||
return nil, rerr
|
||||
}
|
||||
}
|
||||
return &cmdReader{ReadCloser: struct {
|
||||
io.Reader
|
||||
io.Closer
|
||||
}{io.MultiReader(strings.NewReader(string(head[:n])), out), out}, cmd: c, errb: errb}, nil
|
||||
}
|
||||
|
||||
func (s Sudo) Write(ctx context.Context, path string, r io.Reader) error {
|
||||
c := s.cmd(ctx, "write", path)
|
||||
c.Stdin = r
|
||||
var errb strings.Builder
|
||||
c.Stderr = &errb
|
||||
if err := c.Run(); err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s Sudo) Mkdir(ctx context.Context, path string) error {
|
||||
return s.runJSON(ctx, nil, "mkdir", path)
|
||||
}
|
||||
func (s Sudo) Touch(ctx context.Context, path string) error {
|
||||
return s.runJSON(ctx, nil, "touch", path)
|
||||
}
|
||||
func (s Sudo) Rename(ctx context.Context, from, to string) error {
|
||||
return s.runJSON(ctx, nil, "rename", from, to)
|
||||
}
|
||||
|
||||
// Move as root: the helper does the copying and talks on stdout.
|
||||
func (s Sudo) Move(ctx context.Context, from, to string, progress io.Writer) error {
|
||||
c := s.cmd(ctx, "move", from, to)
|
||||
c.Stdout = progress
|
||||
var errb strings.Builder
|
||||
c.Stderr = &errb
|
||||
if err := c.Run(); err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// MoveCmd is the command a task runs for a long move: this binary's helper, as root when
|
||||
// the backend is.
|
||||
func MoveCmd(ctx context.Context, fsys FS, self, from, to string) *exec.Cmd {
|
||||
if s, ok := fsys.(Sudo); ok {
|
||||
return s.cmd(ctx, "move", from, to)
|
||||
}
|
||||
return exec.CommandContext(ctx, self, "fs", "move", from, to)
|
||||
}
|
||||
func (s Sudo) Delete(ctx context.Context, path string, recursive bool) error {
|
||||
if recursive {
|
||||
return s.runJSON(ctx, nil, "delete", "-r", path)
|
||||
}
|
||||
return s.runJSON(ctx, nil, "delete", path)
|
||||
}
|
||||
func (s Sudo) Chmod(ctx context.Context, path string, perm os.FileMode) error {
|
||||
return s.runJSON(ctx, nil, "chmod", fmt.Sprintf("%04o", perm), path)
|
||||
}
|
||||
|
||||
// Helper implements `hamlab fs <op> …` for the Sudo backend. Returns the exit code.
|
||||
func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer) int {
|
||||
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
|
||||
if len(args) < 2 {
|
||||
return fail(errors.New("usage: hamlab fs <list|stat|read|write|mkdir|touch|rename|move|delete|chmod> …"))
|
||||
}
|
||||
ctx := context.Background()
|
||||
var l Local
|
||||
op := args[0]
|
||||
var offset int64
|
||||
if op == "read" && len(args) >= 4 && args[1] == "--offset" { // read --offset N <path>
|
||||
offset, _ = strconv.ParseInt(args[2], 10, 64)
|
||||
args = append([]string{"read"}, args[3:]...)
|
||||
}
|
||||
clean := func(i int) (string, error) {
|
||||
if i >= len(args) {
|
||||
return "", errors.New("path required")
|
||||
}
|
||||
return Clean(args[i])
|
||||
}
|
||||
switch op {
|
||||
case "list", "stat", "read", "write", "mkdir", "touch":
|
||||
p, err := clean(1)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
switch op {
|
||||
case "list":
|
||||
out, err := l.List(ctx, p)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return emit(stdout, out, stderr)
|
||||
case "stat":
|
||||
out, err := l.Stat(ctx, p)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return emit(stdout, out, stderr)
|
||||
case "read":
|
||||
f, err := l.OpenAt(ctx, p, offset)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := io.Copy(stdout, f); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return 0
|
||||
case "write":
|
||||
if err := l.Write(ctx, p, stdin); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "mkdir":
|
||||
if err := l.Mkdir(ctx, p); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "touch":
|
||||
if err := l.Touch(ctx, p); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
}
|
||||
case "rename", "move":
|
||||
from, err := clean(1)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
to, err := clean(2)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
if op == "move" {
|
||||
// A cancelled task is a SIGTERM: stop copying and take the partial copy away.
|
||||
mctx, stop := signal.NotifyContext(ctx, syscall.SIGTERM, syscall.SIGINT)
|
||||
defer stop()
|
||||
if err := l.Move(mctx, from, to, stdout); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
if err := l.Rename(ctx, from, to); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "delete":
|
||||
rec := false
|
||||
i := 1
|
||||
if args[1] == "-r" {
|
||||
rec, i = true, 2
|
||||
}
|
||||
p, err := clean(i)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
if err := l.Delete(ctx, p, rec); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "chmod":
|
||||
perm, err := ParsePerm(args[1])
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
p, err := clean(2)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
if err := l.Chmod(ctx, p, perm); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
default:
|
||||
return fail(errors.New("unknown op " + op))
|
||||
}
|
||||
fmt.Fprintln(stdout, "{}")
|
||||
return 0
|
||||
}
|
||||
|
||||
func emit(w io.Writer, v any, stderr io.Writer) int {
|
||||
if err := json.NewEncoder(w).Encode(v); err != nil {
|
||||
fmt.Fprintln(stderr, err.Error())
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ModTime helper for handlers.
|
||||
func Now() time.Time { return time.Now() }
|
||||
@@ -0,0 +1,197 @@
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// IsCrossDevice reports a rename refused because the two ends are on different
|
||||
// filesystems — on a NAS that is every move between datasets. The Sudo backend only has
|
||||
// the helper's words for it.
|
||||
func IsCrossDevice(err error) bool {
|
||||
return errors.Is(err, syscall.EXDEV) || (err != nil && strings.Contains(err.Error(), "cross-device"))
|
||||
}
|
||||
|
||||
// Inside reports whether to lies within from — a folder cannot be moved into itself.
|
||||
func Inside(from, to string) bool {
|
||||
return to == from || strings.HasPrefix(to, strings.TrimSuffix(from, "/")+"/")
|
||||
}
|
||||
|
||||
// Move is Rename, and across filesystems a copy and a removal. The copy is built under a
|
||||
// hidden temporary name beside where it will land and renamed into place only once it is
|
||||
// complete, and the source is removed only after that: a failure part-way, or a cancel,
|
||||
// leaves the source exactly as it was. Progress goes to w, a line at a time, for the task
|
||||
// output that a long move is watched through.
|
||||
func (Local) Move(ctx context.Context, from, to string, w io.Writer) error {
|
||||
if protected[from] {
|
||||
return errors.New("refusing to move " + from)
|
||||
}
|
||||
if Inside(from, to) {
|
||||
return errors.New("a folder can't be moved into itself")
|
||||
}
|
||||
if _, err := os.Lstat(to); err == nil {
|
||||
return errors.New("destination exists")
|
||||
}
|
||||
err := os.Rename(from, to)
|
||||
if err == nil || !IsCrossDevice(err) {
|
||||
return err
|
||||
}
|
||||
return moveByCopy(ctx, from, to, w)
|
||||
}
|
||||
|
||||
// moveByCopy is the slow path: copy, rename into place, remove the original.
|
||||
func moveByCopy(ctx context.Context, from, to string, w io.Writer) error {
|
||||
st, err := os.Lstat(from)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := filepath.Join(filepath.Dir(to), "."+filepath.Base(to)+".hamlab-moving")
|
||||
_ = os.RemoveAll(tmp) // a previous attempt that died
|
||||
fmt.Fprintf(w, "%s is on another filesystem: copying, then removing the original\n", filepath.Dir(to))
|
||||
p := &progress{w: w}
|
||||
if err := copyPath(ctx, from, tmp, st, p); err != nil {
|
||||
fmt.Fprintln(w, "copy failed; removing the partial copy, the original is untouched")
|
||||
_ = os.RemoveAll(tmp)
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(tmp, to); err != nil {
|
||||
_ = os.RemoveAll(tmp)
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(w, "copied %s; removing the original\n", p.String())
|
||||
if err := os.RemoveAll(from); err != nil {
|
||||
return fmt.Errorf("copied to %s, but the original could not be removed: %w", to, err)
|
||||
}
|
||||
fmt.Fprintf(w, "moved %s to %s\n", from, to)
|
||||
return nil
|
||||
}
|
||||
|
||||
type progress struct {
|
||||
w io.Writer
|
||||
files int64
|
||||
bytes int64
|
||||
shownFiles int64
|
||||
shownBytes int64
|
||||
}
|
||||
|
||||
func (p *progress) add(files, bytes int64) {
|
||||
p.files += files
|
||||
p.bytes += bytes
|
||||
if p.files-p.shownFiles >= 500 || p.bytes-p.shownBytes >= 512<<20 {
|
||||
fmt.Fprintf(p.w, "copying… %s\n", p.String())
|
||||
p.shownFiles, p.shownBytes = p.files, p.bytes
|
||||
}
|
||||
}
|
||||
|
||||
func (p *progress) String() string {
|
||||
return fmt.Sprintf("%d files, %s", p.files, fmtSize(p.bytes))
|
||||
}
|
||||
|
||||
func fmtSize(b int64) string {
|
||||
const unit = 1024
|
||||
if b < unit {
|
||||
return fmt.Sprintf("%d B", b)
|
||||
}
|
||||
div, exp := int64(unit), 0
|
||||
for n := b / unit; n >= unit && exp < 4; n /= unit {
|
||||
div *= unit
|
||||
exp++
|
||||
}
|
||||
return fmt.Sprintf("%.1f %ciB", float64(b)/float64(div), "KMGTP"[exp])
|
||||
}
|
||||
|
||||
// copyPath copies one entry of any kind, recursively for a directory, keeping mode, times
|
||||
// and — when running as root — ownership. Devices, pipes and sockets are refused: a copy
|
||||
// of one is not the thing itself.
|
||||
func copyPath(ctx context.Context, src, dst string, st os.FileInfo, p *progress) error {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
mode := st.Mode()
|
||||
switch {
|
||||
case mode&os.ModeSymlink != 0:
|
||||
target, err := os.Readlink(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Symlink(target, dst); err != nil {
|
||||
return err
|
||||
}
|
||||
chown(dst, st)
|
||||
p.add(1, 0)
|
||||
return nil
|
||||
case mode.IsDir():
|
||||
if err := os.Mkdir(dst, mode.Perm()|0o700); err != nil { // writable while it fills, exact mode after
|
||||
return err
|
||||
}
|
||||
ents, err := os.ReadDir(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, e := range ents {
|
||||
lst, err := e.Info()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := copyPath(ctx, filepath.Join(src, e.Name()), filepath.Join(dst, e.Name()), lst, p); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
chown(dst, st)
|
||||
if err := os.Chmod(dst, mode.Perm()); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Chtimes(dst, st.ModTime(), st.ModTime())
|
||||
case mode.IsRegular():
|
||||
n, err := copyFile(src, dst, st)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
p.add(1, n)
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("%s: can't copy a device, pipe or socket", src)
|
||||
}
|
||||
}
|
||||
|
||||
func copyFile(src, dst string, st os.FileInfo) (int64, error) {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer in.Close()
|
||||
out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL, st.Mode().Perm())
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
n, err := io.Copy(out, in)
|
||||
if err != nil {
|
||||
out.Close()
|
||||
return n, err
|
||||
}
|
||||
if err := out.Close(); err != nil {
|
||||
return n, err
|
||||
}
|
||||
chown(dst, st)
|
||||
if err := os.Chmod(dst, st.Mode().Perm()); err != nil { // umask has no say
|
||||
return n, err
|
||||
}
|
||||
return n, os.Chtimes(dst, st.ModTime(), st.ModTime())
|
||||
}
|
||||
|
||||
// chown keeps the owner when this process may set one; an ordinary user's copy is the
|
||||
// user's, which is what a rename would have left anyway.
|
||||
func chown(dst string, st os.FileInfo) {
|
||||
if os.Geteuid() != 0 {
|
||||
return
|
||||
}
|
||||
if sys, ok := st.Sys().(*syscall.Stat_t); ok {
|
||||
_ = os.Lchown(dst, int(sys.Uid), int(sys.Gid))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,195 @@
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestInside(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
from, to string
|
||||
want bool
|
||||
}{
|
||||
{"/a/b", "/a/b", true}, {"/a/b", "/a/b/c", true}, {"/a/b", "/a/bc", false}, {"/a/b", "/a", false}, {"/", "/x", true},
|
||||
} {
|
||||
if got := Inside(c.from, c.to); got != c.want {
|
||||
t.Errorf("Inside(%q, %q) = %v, want %v", c.from, c.to, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsCrossDevice(t *testing.T) {
|
||||
if !IsCrossDevice(&os.LinkError{Op: "rename", Old: "/a", New: "/b", Err: syscall.EXDEV}) {
|
||||
t.Fatal("the kernel's EXDEV")
|
||||
}
|
||||
if !IsCrossDevice(errors.New("rename /a /b: invalid cross-device link")) {
|
||||
t.Fatal("the helper's words for it")
|
||||
}
|
||||
if IsCrossDevice(errors.New("permission denied")) || IsCrossDevice(nil) {
|
||||
t.Fatal("anything else is not")
|
||||
}
|
||||
}
|
||||
|
||||
// The copy path, exercised directly: a tree with nested folders, a symlink, an odd mode
|
||||
// and old mtimes lands whole under the new name, and the original is gone only then.
|
||||
func TestMoveByCopy(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
src := filepath.Join(root, "src", "album")
|
||||
if err := os.MkdirAll(filepath.Join(src, "sub", "deep"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
write := func(p, s string, mode os.FileMode) {
|
||||
if err := os.WriteFile(p, []byte(s), mode); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Chmod(p, mode)
|
||||
_ = os.Chtimes(p, old, old)
|
||||
}
|
||||
write(filepath.Join(src, "a.txt"), "hello", 0o644)
|
||||
write(filepath.Join(src, "sub", "deep", "b.bin"), strings.Repeat("x", 100000), 0o600)
|
||||
write(filepath.Join(src, "run.sh"), "#!/bin/sh\n", 0o755)
|
||||
if err := os.Symlink("a.txt", filepath.Join(src, "link")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Chtimes(filepath.Join(src, "sub"), old, old)
|
||||
|
||||
dst := filepath.Join(root, "dst", "moved")
|
||||
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out strings.Builder
|
||||
if err := moveByCopy(context.Background(), src, dst, &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Lstat(src); !os.IsNotExist(err) {
|
||||
t.Fatal("the original is still there")
|
||||
}
|
||||
if b, _ := os.ReadFile(filepath.Join(dst, "a.txt")); string(b) != "hello" {
|
||||
t.Fatal("a.txt")
|
||||
}
|
||||
if st, err := os.Stat(filepath.Join(dst, "sub", "deep", "b.bin")); err != nil || st.Size() != 100000 || st.Mode().Perm() != 0o600 || !st.ModTime().Equal(old) {
|
||||
t.Fatalf("b.bin: %v %v", st, err)
|
||||
}
|
||||
if st, _ := os.Stat(filepath.Join(dst, "run.sh")); st.Mode().Perm() != 0o755 {
|
||||
t.Fatal("mode lost")
|
||||
}
|
||||
if target, err := os.Readlink(filepath.Join(dst, "link")); err != nil || target != "a.txt" {
|
||||
t.Fatal("symlink not copied as a symlink")
|
||||
}
|
||||
if st, _ := os.Stat(filepath.Join(dst, "sub")); !st.ModTime().Equal(old) {
|
||||
t.Fatal("folder mtime lost")
|
||||
}
|
||||
if left, _ := filepath.Glob(filepath.Join(root, "dst", ".*hamlab-moving*")); len(left) != 0 {
|
||||
t.Fatal("temporary copy left behind")
|
||||
}
|
||||
if !strings.Contains(out.String(), "moved ") || !strings.Contains(out.String(), "4 files") {
|
||||
t.Fatalf("progress said: %q", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
// A failure part-way leaves the original as it was and no partial copy.
|
||||
func TestMoveByCopyFailsClean(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
src := filepath.Join(root, "src")
|
||||
if err := os.MkdirAll(src, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(src, "a"), []byte("a"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := syscall.Mkfifo(filepath.Join(src, "pipe"), 0o644); err != nil {
|
||||
t.Skip("no fifos here")
|
||||
}
|
||||
dst := filepath.Join(root, "dst")
|
||||
err := moveByCopy(context.Background(), src, dst, &strings.Builder{})
|
||||
if err == nil || !strings.Contains(err.Error(), "pipe") {
|
||||
t.Fatalf("expected the pipe to be refused, got %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(src, "a")); err != nil {
|
||||
t.Fatal("the original was touched")
|
||||
}
|
||||
if _, err := os.Lstat(dst); !os.IsNotExist(err) {
|
||||
t.Fatal("a partial copy was left")
|
||||
}
|
||||
if left, _ := filepath.Glob(filepath.Join(root, ".*hamlab-moving*")); len(left) != 0 {
|
||||
t.Fatal("temporary copy left behind")
|
||||
}
|
||||
// a cancel is the same story
|
||||
if err := os.Remove(filepath.Join(src, "pipe")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if err := moveByCopy(ctx, src, dst, &strings.Builder{}); err == nil {
|
||||
t.Fatal("a cancelled move must fail")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(src, "a")); err != nil {
|
||||
t.Fatal("the original was touched by a cancel")
|
||||
}
|
||||
}
|
||||
|
||||
// Move on one filesystem is a rename: nothing is copied, and the guards hold.
|
||||
func TestMoveSameFilesystem(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
src := filepath.Join(root, "f")
|
||||
if err := os.WriteFile(src, []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var l Local
|
||||
var out strings.Builder
|
||||
if err := l.Move(context.Background(), src, filepath.Join(root, "g"), &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if out.Len() != 0 {
|
||||
t.Fatal("a rename has nothing to say")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(root, "g")); err != nil {
|
||||
t.Fatal("not renamed")
|
||||
}
|
||||
if err := l.Move(context.Background(), root, filepath.Join(root, "inside"), &out); err == nil {
|
||||
t.Fatal("into itself")
|
||||
}
|
||||
if err := l.Move(context.Background(), filepath.Join(root, "g"), filepath.Join(root, "g"), &out); err == nil {
|
||||
t.Fatal("onto itself")
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, "h"), nil, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := l.Move(context.Background(), filepath.Join(root, "g"), filepath.Join(root, "h"), &out); err == nil || !strings.Contains(err.Error(), "exists") {
|
||||
t.Fatalf("onto an existing file: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A link to a file carries the file's size and time, so the row and the preview key follow
|
||||
// the file; a link to a folder is a folder; a link to nothing says so.
|
||||
func TestEntryOfLinks(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "f"), []byte("12345"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := time.Date(2021, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
_ = os.Chtimes(filepath.Join(dir, "f"), old, old)
|
||||
_ = os.Mkdir(filepath.Join(dir, "d"), 0o755)
|
||||
for _, l := range [][2]string{{"lf", "f"}, {"ld", "d"}, {"lx", "nowhere"}} {
|
||||
if err := os.Symlink(l[1], filepath.Join(dir, l[0])); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
ent := func(n string) Entry { st, _ := os.Lstat(filepath.Join(dir, n)); return entryOf(dir, n, st) }
|
||||
if e := ent("lf"); e.Type != "link" || e.Size != 5 || e.Mtime != old.UnixMilli() || e.Target != "f" || e.Broken {
|
||||
t.Fatalf("link to file: %+v", e)
|
||||
}
|
||||
if e := ent("ld"); e.Type != "dir" || e.Target != "d" {
|
||||
t.Fatalf("link to dir: %+v", e)
|
||||
}
|
||||
if e := ent("lx"); e.Type != "link" || !e.Broken || e.Preview != "" {
|
||||
t.Fatalf("dangling: %+v", e)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
)
|
||||
|
||||
// Seeker is an FS file as http.ServeContent wants one — something to seek in — for a file
|
||||
// whose size is already known from a stat. ServeContent seeks to the end for the size, to
|
||||
// the start of the range asked for, then reads: so the backend is opened once, at that
|
||||
// position, and only ever read forward. The Sudo backend's helper is started at the offset
|
||||
// rather than streamed from the top and skipped.
|
||||
type Seeker struct {
|
||||
ctx context.Context
|
||||
fsys FS
|
||||
path string
|
||||
size int64
|
||||
pos int64 // where the next Read reads from
|
||||
at int64 // where cur is positioned, when open
|
||||
cur io.ReadCloser
|
||||
}
|
||||
|
||||
func NewSeeker(ctx context.Context, fsys FS, path string, size int64) *Seeker {
|
||||
return &Seeker{ctx: ctx, fsys: fsys, path: path, size: size}
|
||||
}
|
||||
|
||||
// Check opens the file now, from the start, so a file that cannot be read is known before
|
||||
// any headers go out. Seeking costs nothing; a Read from somewhere else reopens there.
|
||||
func (s *Seeker) Check() error {
|
||||
if s.cur != nil {
|
||||
return nil
|
||||
}
|
||||
f, err := s.fsys.OpenAt(s.ctx, s.path, 0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.cur, s.at = f, 0
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Seeker) Seek(offset int64, whence int) (int64, error) {
|
||||
var pos int64
|
||||
switch whence {
|
||||
case io.SeekStart:
|
||||
pos = offset
|
||||
case io.SeekCurrent:
|
||||
pos = s.pos + offset
|
||||
case io.SeekEnd:
|
||||
pos = s.size + offset
|
||||
default:
|
||||
return 0, errors.New("bad whence")
|
||||
}
|
||||
if pos < 0 {
|
||||
return 0, errors.New("seek before start")
|
||||
}
|
||||
s.pos = pos // the stream, if open, stays where it is until a Read wants somewhere else
|
||||
return pos, nil
|
||||
}
|
||||
|
||||
func (s *Seeker) Read(p []byte) (int, error) {
|
||||
if s.pos >= s.size {
|
||||
return 0, io.EOF
|
||||
}
|
||||
if s.cur != nil && s.pos != s.at {
|
||||
s.cur.Close()
|
||||
s.cur = nil
|
||||
}
|
||||
if s.cur == nil {
|
||||
f, err := s.fsys.OpenAt(s.ctx, s.path, s.pos)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
s.cur, s.at = f, s.pos
|
||||
}
|
||||
n, err := s.cur.Read(p)
|
||||
s.pos += int64(n)
|
||||
s.at += int64(n)
|
||||
return n, err
|
||||
}
|
||||
|
||||
func (s *Seeker) Close() error {
|
||||
if s.cur == nil {
|
||||
return nil
|
||||
}
|
||||
err := s.cur.Close()
|
||||
s.cur = nil
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
import "strings"
|
||||
|
||||
type stringsBuilder = strings.Builder
|
||||
|
||||
type countingFS struct {
|
||||
Local
|
||||
opens []int64
|
||||
}
|
||||
|
||||
func (c *countingFS) OpenAt(ctx context.Context, path string, off int64) (io.ReadCloser, error) {
|
||||
c.opens = append(c.opens, off)
|
||||
return c.Local.OpenAt(ctx, path, off)
|
||||
}
|
||||
|
||||
// ServeContent's moves — size from the end, seek to the range, read forward — open the
|
||||
// backend once, at the range; a seek to where the stream already is costs nothing.
|
||||
func TestSeeker(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "f")
|
||||
data := make([]byte, 1000)
|
||||
for i := range data {
|
||||
data[i] = byte(i)
|
||||
}
|
||||
if err := os.WriteFile(p, data, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fs := &countingFS{}
|
||||
s := NewSeeker(context.Background(), fs, p, int64(len(data)))
|
||||
if n, _ := s.Seek(0, io.SeekEnd); n != 1000 {
|
||||
t.Fatalf("end = %d", n)
|
||||
}
|
||||
if _, err := s.Seek(100, io.SeekStart); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
buf := make([]byte, 50)
|
||||
if n, err := io.ReadFull(s, buf); err != nil || n != 50 || buf[0] != 100 || buf[49] != 149 {
|
||||
t.Fatalf("read at 100: %d %v %v", n, err, buf[:2])
|
||||
}
|
||||
if _, err := s.Seek(150, io.SeekStart); err != nil { // where it already is
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n, _ := io.ReadFull(s, buf); n != 50 || buf[0] != 150 {
|
||||
t.Fatal("read after a no-op seek")
|
||||
}
|
||||
if len(fs.opens) != 1 || fs.opens[0] != 100 {
|
||||
t.Fatalf("opens = %v, want [100]", fs.opens)
|
||||
}
|
||||
if _, err := s.Seek(0, io.SeekStart); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rest, err := io.ReadAll(s)
|
||||
if err != nil || len(rest) != 1000 || rest[999] != data[999] {
|
||||
t.Fatalf("read all from the start: %d %v", len(rest), err)
|
||||
}
|
||||
if len(fs.opens) != 2 || fs.opens[1] != 0 {
|
||||
t.Fatalf("opens = %v, want [100 0]", fs.opens)
|
||||
}
|
||||
if n, err := s.Read(buf); n != 0 || err != io.EOF {
|
||||
t.Fatal("past the end is EOF")
|
||||
}
|
||||
if _, err := s.Seek(-1, io.SeekStart); err == nil {
|
||||
t.Fatal("before the start")
|
||||
}
|
||||
if err := s.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Check opens at the start and keeps that stream for a read from the start
|
||||
s2 := NewSeeker(context.Background(), fs, p, 1000)
|
||||
if err := s2.Check(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s2.Seek(0, io.SeekEnd)
|
||||
s2.Seek(0, io.SeekStart)
|
||||
if n, _ := io.ReadFull(s2, buf); n != 50 || buf[0] != 0 {
|
||||
t.Fatal("read from the start after Check")
|
||||
}
|
||||
if len(fs.opens) != 3 {
|
||||
t.Fatalf("Check's open was not reused: %v", fs.opens)
|
||||
}
|
||||
s2.Close()
|
||||
if err := NewSeeker(context.Background(), fs, filepath.Join(t.TempDir(), "nope"), 5).Check(); err == nil {
|
||||
t.Fatal("Check must report a file that can't be opened")
|
||||
}
|
||||
}
|
||||
|
||||
// The helper's read takes an offset, for the Sudo backend.
|
||||
func TestHelperReadOffset(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "f")
|
||||
if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out, errb stringsBuilder
|
||||
if rc := Helper([]string{"read", "--offset", "7", p}, nil, &out, &errb); rc != 0 {
|
||||
t.Fatalf("rc %d: %s", rc, errb.String())
|
||||
}
|
||||
if out.String() != "789" {
|
||||
t.Fatalf("got %q", out.String())
|
||||
}
|
||||
out.Reset()
|
||||
if rc := Helper([]string{"read", p}, nil, &out, &errb); rc != 0 || out.String() != "0123456789" {
|
||||
t.Fatalf("plain read: rc %d %q", rc, out.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,843 @@
|
||||
// Package hardware is what is physically in the machine: the board it is, its processor,
|
||||
// the memory modules in its slots, the drives attached to it and the graphics in it.
|
||||
//
|
||||
// None of this changes while the machine is running, so it is read once and remembered.
|
||||
// Everything here is read from what Debian and its kin already have — dmidecode for the
|
||||
// parts only the firmware knows, lscpu, lspci, sysfs — and every reader is allowed to
|
||||
// come back empty: a machine with no dmidecode, no PCI bus or no root still gets a page,
|
||||
// with a note saying what could not be read and why.
|
||||
package hardware
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
type System struct {
|
||||
Model string `json:"model,omitempty"` // "GMKtec NucBox G3"
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Family string `json:"family,omitempty"`
|
||||
Kernel string `json:"kernel,omitempty"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
Virt string `json:"virt,omitempty"` // "kvm", "lxc", … when this is not real hardware
|
||||
}
|
||||
|
||||
// Slot is an expansion slot as the firmware describes it. Worth treating with suspicion on
|
||||
// a mini PC, where the table is often copied from a reference design and lists slots the
|
||||
// case has no room for.
|
||||
type Slot struct {
|
||||
Designation string `json:"designation"`
|
||||
Type string `json:"type,omitempty"`
|
||||
Usage string `json:"usage,omitempty"` // Available | In Use | Unknown
|
||||
Length string `json:"length,omitempty"`
|
||||
Width string `json:"width,omitempty"`
|
||||
Bus string `json:"bus,omitempty"` // PCI address of what is in it
|
||||
Device string `json:"device,omitempty"` // and what that turns out to be
|
||||
}
|
||||
|
||||
// Board is the motherboard and the firmware on it.
|
||||
type Board struct {
|
||||
Maker string `json:"maker,omitempty"`
|
||||
Product string `json:"product,omitempty"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
AssetTag string `json:"asset_tag,omitempty"`
|
||||
Type string `json:"type,omitempty"` // "Motherboard", "Processor Module", …
|
||||
Location string `json:"location,omitempty"` // where it sits in the chassis
|
||||
Chassis string `json:"chassis,omitempty"`
|
||||
ChassisMaker string `json:"chassis_maker,omitempty"`
|
||||
BIOS string `json:"bios,omitempty"` // vendor and version
|
||||
BIOSDate string `json:"bios_date,omitempty"`
|
||||
BIOSRevision string `json:"bios_revision,omitempty"`
|
||||
ROMSize string `json:"rom_size,omitempty"`
|
||||
UEFI bool `json:"uefi,omitempty"`
|
||||
Slots []Slot `json:"slots,omitempty"`
|
||||
}
|
||||
|
||||
type Cache struct {
|
||||
Level string `json:"level"` // L1d, L1i, L2, L3
|
||||
Size string `json:"size"` // as lscpu puts it: "6 MiB (1 instance)"
|
||||
}
|
||||
|
||||
type CPU struct {
|
||||
Model string `json:"model,omitempty"`
|
||||
Vendor string `json:"vendor,omitempty"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
Sockets int `json:"sockets,omitempty"`
|
||||
SocketNames []string `json:"socket_names,omitempty"` // what the firmware calls each socket
|
||||
Cores int `json:"cores,omitempty"`
|
||||
Threads int `json:"threads,omitempty"`
|
||||
ThreadsPerCore int `json:"threads_per_core,omitempty"`
|
||||
MinMHz float64 `json:"min_mhz,omitempty"`
|
||||
MaxMHz float64 `json:"max_mhz,omitempty"`
|
||||
Family string `json:"family,omitempty"`
|
||||
ModelID string `json:"model_id,omitempty"`
|
||||
Stepping string `json:"stepping,omitempty"`
|
||||
Microcode string `json:"microcode,omitempty"`
|
||||
Virtualization string `json:"virtualization,omitempty"`
|
||||
Caches []Cache `json:"caches,omitempty"`
|
||||
Flags []string `json:"flags,omitempty"` // the handful anyone looks for
|
||||
}
|
||||
|
||||
type Module struct {
|
||||
Locator string `json:"locator"` // the slot, as the board labels it
|
||||
Bank string `json:"bank,omitempty"`
|
||||
Empty bool `json:"empty,omitempty"`
|
||||
Size uint64 `json:"size,omitempty"`
|
||||
Type string `json:"type,omitempty"` // DDR4
|
||||
FormFactor string `json:"form_factor,omitempty"` // SODIMM
|
||||
SpeedMTs int `json:"speed_mts,omitempty"` // what the module can do
|
||||
RunningMTs int `json:"running_mts,omitempty"` // what the board is running it at
|
||||
Maker string `json:"maker,omitempty"`
|
||||
Part string `json:"part,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Rank int `json:"rank,omitempty"`
|
||||
Voltage string `json:"voltage,omitempty"`
|
||||
}
|
||||
|
||||
type Memory struct {
|
||||
Total uint64 `json:"total"` // what the kernel can use
|
||||
Installed uint64 `json:"installed,omitempty"` // what the modules add up to
|
||||
MaxCapacity uint64 `json:"max_capacity,omitempty"` // what the board will take
|
||||
Slots int `json:"slots,omitempty"`
|
||||
Filled int `json:"filled,omitempty"`
|
||||
ECC string `json:"ecc,omitempty"`
|
||||
Modules []Module `json:"modules,omitempty"`
|
||||
}
|
||||
|
||||
type Drive struct {
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"` // nvme | ssd | hdd | usb | emmc | sd | other
|
||||
Model string `json:"model,omitempty"`
|
||||
Vendor string `json:"vendor,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Firmware string `json:"firmware,omitempty"`
|
||||
Tran string `json:"tran,omitempty"`
|
||||
Size uint64 `json:"size"`
|
||||
Rotational bool `json:"rotational,omitempty"`
|
||||
Removable bool `json:"removable,omitempty"`
|
||||
}
|
||||
|
||||
type GPU struct {
|
||||
Model string `json:"model,omitempty"`
|
||||
Vendor string `json:"vendor,omitempty"`
|
||||
Slot string `json:"slot,omitempty"` // PCI address, when it is on a bus
|
||||
Driver string `json:"driver,omitempty"`
|
||||
Node string `json:"node,omitempty"` // card0
|
||||
Memory uint64 `json:"memory,omitempty"`
|
||||
Class string `json:"class,omitempty"` // VGA | 3D | Display
|
||||
}
|
||||
|
||||
type Report struct {
|
||||
At time.Time `json:"at"`
|
||||
System System `json:"system"`
|
||||
Board Board `json:"board"`
|
||||
CPU CPU `json:"cpu"`
|
||||
Memory Memory `json:"memory"`
|
||||
Drives []Drive `json:"drives"`
|
||||
GPUs []GPU `json:"gpus"`
|
||||
Notes []string `json:"notes,omitempty"` // what could not be read, and why
|
||||
}
|
||||
|
||||
// Manager remembers the last report. Hardware does not change under a running kernel —
|
||||
// the one thing that would, a drive plugged in, is what Refresh is for.
|
||||
type Manager struct {
|
||||
mu sync.Mutex
|
||||
last *Report
|
||||
at time.Time
|
||||
}
|
||||
|
||||
func New() *Manager { return &Manager{} }
|
||||
|
||||
const keepFor = 10 * time.Minute
|
||||
|
||||
func (m *Manager) Report(ctx context.Context, refresh bool) *Report {
|
||||
m.mu.Lock()
|
||||
if !refresh && m.last != nil && time.Since(m.at) < keepFor {
|
||||
r := m.last
|
||||
m.mu.Unlock()
|
||||
return r
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
r := collect(ctx)
|
||||
|
||||
m.mu.Lock()
|
||||
m.last, m.at = r, time.Now()
|
||||
m.mu.Unlock()
|
||||
return r
|
||||
}
|
||||
|
||||
func collect(ctx context.Context) *Report {
|
||||
ctx, cancel := context.WithTimeout(ctx, 25*time.Second)
|
||||
defer cancel()
|
||||
r := &Report{At: time.Now()}
|
||||
dmi, note := readDMI(ctx)
|
||||
if note != "" {
|
||||
r.Notes = append(r.Notes, note)
|
||||
}
|
||||
r.System = system(ctx, dmi)
|
||||
r.Board = board(dmi, pciNames(ctx))
|
||||
r.CPU = cpu(ctx, dmi)
|
||||
r.Memory = memory(dmi)
|
||||
if len(r.Memory.Modules) == 0 && note == "" {
|
||||
r.Notes = append(r.Notes, "memory modules: the firmware's table says nothing about this machine's slots")
|
||||
}
|
||||
r.Drives = drives(ctx)
|
||||
r.GPUs = gpus(ctx)
|
||||
if len(r.GPUs) == 0 {
|
||||
r.Notes = append(r.Notes, "graphics: nothing on the PCI bus and no kernel display device")
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// ---- dmidecode ----
|
||||
|
||||
// record is one entry of the firmware's table: its DMI type and its key/value lines.
|
||||
type record struct {
|
||||
Type int
|
||||
V map[string]string
|
||||
}
|
||||
|
||||
// listKey holds every doubly-indented line of a record — dmidecode's lists, which have no
|
||||
// key of their own: BIOS characteristics, board features, slot characteristics.
|
||||
const listKey = "\x00list"
|
||||
|
||||
var handleRe = regexp.MustCompile(`^Handle 0x[0-9A-Fa-f]+, DMI type (\d+)`)
|
||||
|
||||
// readDMI runs dmidecode once for every type the page wants and parses the lot. It needs
|
||||
// root; without it the page loses the board, the memory modules and the socket names, and
|
||||
// says so rather than pretending the machine has none.
|
||||
func readDMI(ctx context.Context) ([]record, string) {
|
||||
tool, err := exec.LookPath("dmidecode")
|
||||
if err != nil {
|
||||
if _, e := os.Stat("/usr/sbin/dmidecode"); e != nil {
|
||||
return nil, "the board and the memory modules need dmidecode, which isn't installed here"
|
||||
}
|
||||
tool = "/usr/sbin/dmidecode"
|
||||
}
|
||||
args := []string{tool, "-t", "bios", "-t", "system", "-t", "baseboard", "-t", "chassis", "-t", "slot", "-t", "processor", "-t", "memory"}
|
||||
if os.Geteuid() != 0 {
|
||||
if !priv.Available() {
|
||||
return nil, "the board and the memory modules come from the firmware's table, which only root may read"
|
||||
}
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, args[0], args[1:]...).Output()
|
||||
if err != nil {
|
||||
return nil, "the firmware's table could not be read: " + strings.TrimSpace(err.Error())
|
||||
}
|
||||
return parse(string(out)), ""
|
||||
}
|
||||
|
||||
// parse turns dmidecode's output into records: a handle line starts one, and the single
|
||||
// indented "Key: Value" lines under it are its fields. Doubly-indented lines are the
|
||||
// continuation of a list (a processor's flags), which nothing here reads.
|
||||
func parse(text string) []record {
|
||||
var recs []record
|
||||
var cur *record
|
||||
sc := bufio.NewScanner(strings.NewReader(text))
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 2<<20)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if m := handleRe.FindStringSubmatch(line); m != nil {
|
||||
t, _ := strconv.Atoi(m[1])
|
||||
recs = append(recs, record{Type: t, V: map[string]string{}})
|
||||
cur = &recs[len(recs)-1]
|
||||
continue
|
||||
}
|
||||
if cur == nil || !strings.HasPrefix(line, "\t") {
|
||||
if strings.TrimSpace(line) == "" {
|
||||
cur = nil
|
||||
}
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(line, "\t\t") { // a list under the field above it, e.g. BIOS characteristics
|
||||
cur.V[listKey] += strings.TrimSpace(line) + "\n"
|
||||
continue
|
||||
}
|
||||
k, v, ok := strings.Cut(strings.TrimSpace(line), ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
cur.V[k] = strings.TrimSpace(v)
|
||||
}
|
||||
return recs
|
||||
}
|
||||
|
||||
func (r record) get(keys ...string) string {
|
||||
for _, k := range keys {
|
||||
if v := r.V[k]; v != "" && !unset(v) {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// unset is every way the firmware has of saying "nobody filled this in".
|
||||
func unset(v string) bool {
|
||||
switch strings.ToLower(v) {
|
||||
case "", "unknown", "not specified", "not provided", "none", "default string", "to be filled by o.e.m.",
|
||||
"system manufacturer", "system product name", "to be filled by oem", "n/a", "empty", "no module installed",
|
||||
"<out of spec>", "other", "not applicable":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func first(recs []record, t int) record {
|
||||
for _, r := range recs {
|
||||
if r.Type == t {
|
||||
return r
|
||||
}
|
||||
}
|
||||
return record{Type: t, V: map[string]string{}}
|
||||
}
|
||||
|
||||
func all(recs []record, t int) []record {
|
||||
var out []record
|
||||
for _, r := range recs {
|
||||
if r.Type == t {
|
||||
out = append(out, r)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- system ----
|
||||
|
||||
func system(ctx context.Context, dmi []record) System {
|
||||
sys := first(dmi, 1)
|
||||
s := System{
|
||||
Model: join(sys.get("Manufacturer"), sys.get("Product Name")),
|
||||
Serial: sys.get("Serial Number"),
|
||||
Family: sys.get("Family"),
|
||||
Arch: runtimeArch(),
|
||||
}
|
||||
s.Kernel = readFile("/proc/sys/kernel/osrelease")
|
||||
if s.Model == "" { // no SMBIOS: a Pi, a Mac, anything with a device tree
|
||||
s.Model = strings.TrimRight(readFile("/sys/firmware/devicetree/base/model"), "\x00")
|
||||
}
|
||||
if out, err := exec.CommandContext(ctx, "systemd-detect-virt").Output(); err == nil {
|
||||
if v := strings.TrimSpace(string(out)); v != "" && v != "none" {
|
||||
s.Virt = v
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// board reads the motherboard, the chassis it sits in, the firmware on it and the
|
||||
// expansion slots the firmware claims. All of it is SMBIOS, so a machine without one — a
|
||||
// Mac, a Pi — gets an empty board, and the page says so rather than inventing one.
|
||||
func board(dmi []record, pci map[string]string) Board {
|
||||
b, ch, bios := first(dmi, 2), first(dmi, 3), first(dmi, 0)
|
||||
out := Board{
|
||||
Maker: b.get("Manufacturer"),
|
||||
Product: b.get("Product Name"),
|
||||
Version: b.get("Version"),
|
||||
Serial: b.get("Serial Number"),
|
||||
AssetTag: b.get("Asset Tag"),
|
||||
Type: b.get("Type"),
|
||||
Location: b.get("Location In Chassis"),
|
||||
Chassis: ch.get("Type"),
|
||||
ChassisMaker: ch.get("Manufacturer"),
|
||||
BIOS: join(bios.get("Vendor"), bios.get("Version")),
|
||||
BIOSDate: bios.get("Release Date"),
|
||||
BIOSRevision: bios.get("BIOS Revision"),
|
||||
}
|
||||
// "ROM Size: 0 MB" is a board that never filled the field in, not a board with no ROM
|
||||
if rom := bios.get("ROM Size"); rom != "" && !strings.HasPrefix(rom, "0 ") {
|
||||
out.ROMSize = rom
|
||||
}
|
||||
if out.Product == out.Maker {
|
||||
out.Product = "" // a board named after its maker says nothing twice
|
||||
}
|
||||
out.UEFI = strings.Contains(bios.V[listKey], "UEFI") // a characteristics line, not a field
|
||||
for _, sl := range all(dmi, 9) {
|
||||
s := Slot{
|
||||
Designation: sl.get("Designation"),
|
||||
Type: sl.get("Type"),
|
||||
Usage: sl.get("Current Usage"),
|
||||
Length: sl.get("Length"),
|
||||
Width: sl.get("Data Bus Width"),
|
||||
}
|
||||
// every slot on some boards claims 0000:00:00.0, which is the host bridge and
|
||||
// certainly not what is plugged into that slot
|
||||
if bus := sl.get("Bus Address"); bus != "" && bus != "0000:00:00.0" {
|
||||
s.Bus, s.Device = bus, pci[bus]
|
||||
}
|
||||
if s.Designation != "" {
|
||||
out.Slots = append(out.Slots, s)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func join(parts ...string) string {
|
||||
var keep []string
|
||||
for _, p := range parts {
|
||||
if p = strings.TrimSpace(p); p != "" && !unset(p) {
|
||||
keep = append(keep, p)
|
||||
}
|
||||
}
|
||||
// "GMKtec GMKtec" reads badly: drop a maker the product already names
|
||||
if len(keep) == 2 && strings.HasPrefix(strings.ToLower(keep[1]), strings.ToLower(keep[0])) {
|
||||
return keep[1]
|
||||
}
|
||||
return strings.Join(keep, " ")
|
||||
}
|
||||
|
||||
func readFile(p string) string {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
func runtimeArch() string {
|
||||
if out, err := exec.Command("uname", "-m").Output(); err == nil {
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- processor ----
|
||||
|
||||
// worthShowing is the short list of processor features anyone actually asks about; the
|
||||
// full flag list runs to hundreds and belongs in a terminal.
|
||||
var worthShowing = []string{"aes", "avx", "avx2", "avx512f", "sha_ni", "vmx", "svm", "sse4_2", "rdrand", "hypervisor"}
|
||||
|
||||
func cpu(ctx context.Context, dmi []record) CPU {
|
||||
c := CPU{}
|
||||
f := lscpu(ctx)
|
||||
c.Model = f["Model name"]
|
||||
c.Vendor = f["Vendor ID"]
|
||||
c.Arch = f["Architecture"]
|
||||
c.Sockets = atoi(f["Socket(s)"])
|
||||
c.ThreadsPerCore = atoi(f["Thread(s) per core"])
|
||||
c.Threads = atoi(f["CPU(s)"])
|
||||
c.Cores = atoi(f["Core(s) per socket"]) * maxInt(1, c.Sockets)
|
||||
c.MinMHz = atof(f["CPU min MHz"])
|
||||
c.MaxMHz = atof(f["CPU max MHz"])
|
||||
c.Family, c.ModelID, c.Stepping = f["CPU family"], f["Model"], f["Stepping"]
|
||||
c.Virtualization = f["Virtualization"]
|
||||
for _, l := range []string{"L1d", "L1i", "L2", "L3"} {
|
||||
if v := f[l+" cache"]; v != "" {
|
||||
c.Caches = append(c.Caches, Cache{Level: l, Size: v})
|
||||
}
|
||||
}
|
||||
have := map[string]bool{}
|
||||
for _, fl := range strings.Fields(f["Flags"]) {
|
||||
have[fl] = true
|
||||
}
|
||||
for _, fl := range worthShowing {
|
||||
if have[fl] {
|
||||
c.Flags = append(c.Flags, fl)
|
||||
}
|
||||
}
|
||||
for _, p := range all(dmi, 4) {
|
||||
if s := p.get("Socket Designation"); s != "" {
|
||||
c.SocketNames = append(c.SocketNames, s)
|
||||
}
|
||||
if c.Model == "" {
|
||||
c.Model = p.get("Version", "Family")
|
||||
}
|
||||
if c.MaxMHz == 0 {
|
||||
if mhz := atof(strings.TrimSuffix(p.get("Max Speed"), " MHz")); mhz > 0 {
|
||||
c.MaxMHz = mhz
|
||||
}
|
||||
}
|
||||
}
|
||||
if c.Sockets == 0 {
|
||||
c.Sockets = len(c.SocketNames)
|
||||
}
|
||||
// A processor whose cores are not all alike — Apple's, Intel's P and E cores — is
|
||||
// reported by lscpu as several groups, and "core(s) per socket" then counts one group
|
||||
// rather than the chip. Without SMT every thread is a core, so the thread count is the
|
||||
// honest answer.
|
||||
if c.ThreadsPerCore == 1 && c.Threads > c.Cores {
|
||||
c.Cores = c.Threads
|
||||
}
|
||||
// /proc/cpuinfo is the only place the running microcode revision is written down
|
||||
if b, err := os.ReadFile("/proc/cpuinfo"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if k, v, ok := strings.Cut(line, ":"); ok && strings.TrimSpace(k) == "microcode" {
|
||||
c.Microcode = strings.TrimSpace(v)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if c.Model == "" {
|
||||
c.Model = "unknown processor"
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func lscpu(ctx context.Context) map[string]string {
|
||||
f := map[string]string{}
|
||||
out, err := exec.CommandContext(ctx, "lscpu").Output()
|
||||
if err != nil {
|
||||
// no util-linux: /proc/cpuinfo still names the thing and counts it
|
||||
if b, err := os.ReadFile("/proc/cpuinfo"); err == nil {
|
||||
n := 0
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
k, v, ok := strings.Cut(line, ":")
|
||||
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
|
||||
switch {
|
||||
case ok && strings.EqualFold(k, "model name"):
|
||||
f["Model name"] = v
|
||||
case ok && k == "processor":
|
||||
n++
|
||||
}
|
||||
}
|
||||
f["CPU(s)"] = strconv.Itoa(n)
|
||||
}
|
||||
return f
|
||||
}
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
k, v, ok := strings.Cut(line, ":")
|
||||
if !ok || strings.HasPrefix(line, " ") {
|
||||
continue
|
||||
}
|
||||
f[strings.TrimSpace(k)] = strings.TrimSpace(v)
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
func atoi(s string) int { n, _ := strconv.Atoi(strings.TrimSpace(s)); return n }
|
||||
|
||||
func atof(s string) float64 {
|
||||
v, _ := strconv.ParseFloat(strings.TrimSpace(s), 64)
|
||||
return v
|
||||
}
|
||||
|
||||
func maxInt(a, b int) int {
|
||||
if a > b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// leadingInt reads the number off the front of "3200 MT/s".
|
||||
func leadingInt(s string) int {
|
||||
f := strings.Fields(s)
|
||||
if len(f) == 0 {
|
||||
return 0
|
||||
}
|
||||
return atoi(f[0])
|
||||
}
|
||||
|
||||
// ---- memory ----
|
||||
|
||||
var sizeRe = regexp.MustCompile(`^(\d+)\s*(kB|KB|MB|GB|TB)$`)
|
||||
|
||||
// dmiSize turns dmidecode's "8 GB" / "16384 MB" into bytes.
|
||||
func dmiSize(s string) uint64 {
|
||||
m := sizeRe.FindStringSubmatch(strings.TrimSpace(s))
|
||||
if m == nil {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.ParseUint(m[1], 10, 64)
|
||||
switch strings.ToUpper(m[2]) {
|
||||
case "KB":
|
||||
return n << 10
|
||||
case "MB":
|
||||
return n << 20
|
||||
case "GB":
|
||||
return n << 30
|
||||
case "TB":
|
||||
return n << 40
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func memory(dmi []record) Memory {
|
||||
m := Memory{}
|
||||
if b, err := os.ReadFile("/proc/meminfo"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if k, v, ok := strings.Cut(line, ":"); ok && k == "MemTotal" {
|
||||
m.Total = uint64(atoi(strings.TrimSuffix(strings.TrimSpace(v), " kB"))) << 10
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, a := range all(dmi, 16) {
|
||||
m.MaxCapacity += dmiSize(a.get("Maximum Capacity"))
|
||||
switch e := a.V["Error Correction Type"]; {
|
||||
case e == "None":
|
||||
m.ECC = "non-ECC" // the array says so outright; worth knowing, not worth hiding
|
||||
case e != "" && !unset(e):
|
||||
m.ECC = e
|
||||
}
|
||||
}
|
||||
for _, d := range all(dmi, 17) {
|
||||
mod := Module{
|
||||
Locator: d.get("Locator"),
|
||||
Bank: d.get("Bank Locator"),
|
||||
Size: dmiSize(d.get("Size")),
|
||||
}
|
||||
m.Slots++
|
||||
if mod.Locator == "" {
|
||||
mod.Locator = "slot " + strconv.Itoa(m.Slots)
|
||||
}
|
||||
if mod.Size == 0 {
|
||||
mod.Empty = true
|
||||
m.Modules = append(m.Modules, mod)
|
||||
continue
|
||||
}
|
||||
m.Filled++
|
||||
mod.Type = d.get("Type")
|
||||
mod.FormFactor = d.get("Form Factor")
|
||||
mod.SpeedMTs = leadingInt(d.get("Speed"))
|
||||
mod.RunningMTs = leadingInt(d.get("Configured Memory Speed", "Configured Clock Speed"))
|
||||
mod.Maker = makerName(d.get("Manufacturer"), d.V["Module Manufacturer ID"])
|
||||
mod.Part = d.get("Part Number")
|
||||
mod.Serial = d.get("Serial Number")
|
||||
mod.Rank = atoi(d.get("Rank"))
|
||||
mod.Voltage = d.get("Configured Voltage")
|
||||
m.Installed += mod.Size
|
||||
m.Modules = append(m.Modules, mod)
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// jedec maps the memory makers who turn up in a homelab to their names: SPD carries a
|
||||
// JEDEC id, and dmidecode prints the raw code when the module's string field is empty.
|
||||
var jedec = map[string]string{
|
||||
"0X2C00": "Micron", "0X802C": "Micron",
|
||||
"0XCE00": "Samsung", "0X80CE": "Samsung", "0X00CE": "Samsung",
|
||||
"0XAD00": "SK hynix", "0X80AD": "SK hynix",
|
||||
"0X9801": "Kingston", "0X7F98": "Kingston", "0X0198": "Kingston",
|
||||
"0X859B": "Crucial", "0X04CB": "A-DATA", "0X04CD": "G.Skill",
|
||||
"0X029E": "Corsair", "0X0B45": "Wilk Elektronik", "0X830B": "Wilk Elektronik",
|
||||
"0X0443": "Ramaxel", "0X0194": "Smart Modular",
|
||||
}
|
||||
|
||||
// makerName prefers what the module says it is; a bare JEDEC id becomes a name when it is
|
||||
// one we know, and stays the id when it is not — a wrong name is worse than a number.
|
||||
func makerName(v, id string) string {
|
||||
if v != "" && !unset(v) && !strings.HasPrefix(v, "0x") {
|
||||
return v
|
||||
}
|
||||
code := v
|
||||
if code == "" {
|
||||
code = id
|
||||
}
|
||||
if n, ok := jedec[strings.ToUpper(code)]; ok {
|
||||
return n
|
||||
}
|
||||
if v != "" && !unset(v) {
|
||||
return v
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- drives ----
|
||||
|
||||
func drives(ctx context.Context) []Drive {
|
||||
devs, err := storage.Devices(ctx)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]Drive, 0, len(devs))
|
||||
for _, d := range devs {
|
||||
out = append(out, Drive{
|
||||
Name: d.Name, Kind: d.Kind, Model: d.Model, Vendor: d.Vendor, Serial: d.Serial,
|
||||
Tran: d.Tran, Size: d.Size, Rotational: d.Rotational, Removable: d.Removable,
|
||||
Firmware: firmwareOf(d.Name),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// firmwareOf reads a drive's firmware revision. An NVMe device is folded to its
|
||||
// controller ("nvme0"), which has no /sys/block entry of its own — the revision is on the
|
||||
// controller in /sys/class/nvme.
|
||||
func firmwareOf(name string) string {
|
||||
return firstNonEmpty(
|
||||
readFile("/sys/block/"+name+"/device/firmware_rev"),
|
||||
readFile("/sys/block/"+name+"/device/rev"),
|
||||
readFile("/sys/class/nvme/"+name+"/firmware_rev"),
|
||||
)
|
||||
}
|
||||
|
||||
func firstNonEmpty(v ...string) string {
|
||||
for _, s := range v {
|
||||
if s != "" {
|
||||
return s
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- graphics ----
|
||||
|
||||
// lspci -mm quotes every field, so a device name can hold the spaces and brackets it
|
||||
// likes: `"VGA compatible controller [0300]" "Intel Corporation [8086]" "Alder Lake-N …"`.
|
||||
var quoted = regexp.MustCompile(`"([^"]*)"`)
|
||||
|
||||
// stripID drops the trailing PCI id from `Alder Lake-N [UHD Graphics] [46d1]`.
|
||||
func stripID(s string) string {
|
||||
if i := strings.LastIndex(s, " ["); i > 0 && strings.HasSuffix(s, "]") {
|
||||
return strings.TrimSpace(s[:i])
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func gpus(ctx context.Context) []GPU {
|
||||
var out []GPU
|
||||
seen := map[string]bool{} // PCI addresses already accounted for
|
||||
if _, err := exec.LookPath("lspci"); err == nil {
|
||||
drivers := pciDrivers(ctx)
|
||||
if b, err := exec.CommandContext(ctx, "lspci", "-D", "-mm", "-nn").Output(); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
slot, _, _ := strings.Cut(line, " ")
|
||||
fields := quoted.FindAllStringSubmatch(line, -1)
|
||||
if len(fields) < 3 {
|
||||
continue
|
||||
}
|
||||
kind := displayClass(fields[0][1])
|
||||
if kind == "" {
|
||||
continue
|
||||
}
|
||||
seen[slot] = true
|
||||
g := GPU{Slot: slot, Class: kind, Vendor: stripID(fields[1][1]), Model: stripID(fields[2][1]), Driver: drivers[slot]}
|
||||
g.Node, g.Memory = drmFor(slot)
|
||||
out = append(out, g)
|
||||
}
|
||||
}
|
||||
}
|
||||
// A GPU that is not on a PCI bus at all — an Apple Silicon Mac, a Pi, anything with a
|
||||
// device tree — is only visible as a kernel display device.
|
||||
cards, _ := filepath.Glob("/sys/class/drm/card[0-9]")
|
||||
for _, c := range cards {
|
||||
dev, err := filepath.EvalSymlinks(c + "/device")
|
||||
if err != nil || seen[filepath.Base(dev)] {
|
||||
continue
|
||||
}
|
||||
g := GPU{Node: filepath.Base(c), Class: "Display"}
|
||||
if drv, err := filepath.EvalSymlinks(dev + "/driver"); err == nil {
|
||||
g.Driver = filepath.Base(drv)
|
||||
}
|
||||
g.Model = deviceTreeName(dev)
|
||||
if g.Model == "" && g.Driver != "" {
|
||||
g.Model = g.Driver + " display"
|
||||
}
|
||||
if g.Model == "" {
|
||||
continue
|
||||
}
|
||||
g.Memory = vram(dev)
|
||||
out = append(out, g)
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].Slot < out[j].Slot })
|
||||
return out
|
||||
}
|
||||
|
||||
// pciNames maps a PCI address to a readable name, so a slot the firmware says is in use
|
||||
// can say what is in it.
|
||||
func pciNames(ctx context.Context) map[string]string {
|
||||
m := map[string]string{}
|
||||
if _, err := exec.LookPath("lspci"); err != nil {
|
||||
return m
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, "lspci", "-D", "-mm", "-nn").Output()
|
||||
if err != nil {
|
||||
return m
|
||||
}
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
slot, _, _ := strings.Cut(line, " ")
|
||||
f := quoted.FindAllStringSubmatch(line, -1)
|
||||
if slot == "" || len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
m[slot] = strings.TrimSpace(stripID(f[1][1]) + " " + stripID(f[2][1]))
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func displayClass(c string) string {
|
||||
switch {
|
||||
case strings.Contains(c, "[0300]"):
|
||||
return "VGA"
|
||||
case strings.Contains(c, "[0302]"):
|
||||
return "3D"
|
||||
case strings.Contains(c, "[0301]"), strings.Contains(c, "[0380]"):
|
||||
return "Display"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// pciDrivers maps a PCI address to the kernel module driving it.
|
||||
func pciDrivers(ctx context.Context) map[string]string {
|
||||
m := map[string]string{}
|
||||
out, err := exec.CommandContext(ctx, "lspci", "-D", "-k").Output()
|
||||
if err != nil {
|
||||
return m
|
||||
}
|
||||
slot := ""
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
if !strings.HasPrefix(line, "\t") && !strings.HasPrefix(line, " ") {
|
||||
slot, _, _ = strings.Cut(line, " ")
|
||||
continue
|
||||
}
|
||||
if v, ok := strings.CutPrefix(strings.TrimSpace(line), "Kernel driver in use:"); ok && slot != "" {
|
||||
m[slot] = strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// drmFor finds the kernel display device for a PCI address, and its video memory when the
|
||||
// driver publishes it (amdgpu does; Intel's integrated graphics have none of their own).
|
||||
func drmFor(slot string) (string, uint64) {
|
||||
cards, _ := filepath.Glob("/sys/class/drm/card[0-9]")
|
||||
for _, c := range cards {
|
||||
dev, err := filepath.EvalSymlinks(c + "/device")
|
||||
if err != nil || filepath.Base(dev) != slot {
|
||||
continue
|
||||
}
|
||||
return filepath.Base(c), vram(dev)
|
||||
}
|
||||
return "", 0
|
||||
}
|
||||
|
||||
func vram(dev string) uint64 {
|
||||
for _, f := range []string{"mem_info_vram_total", "mem_info_vis_vram_total"} {
|
||||
if v := readFile(dev + "/" + f); v != "" {
|
||||
n, _ := strconv.ParseUint(v, 10, 64)
|
||||
return n
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// deviceTreeName reads what a device-tree machine calls its graphics: the compatible
|
||||
// string, tidied — "apple,agx-g13g" becomes "Apple agx-g13g".
|
||||
func deviceTreeName(dev string) string {
|
||||
c := readFile(dev + "/of_node/compatible")
|
||||
if c == "" {
|
||||
return ""
|
||||
}
|
||||
c = strings.TrimRight(strings.Split(c, "\x00")[0], "\x00")
|
||||
maker, rest, ok := strings.Cut(c, ",")
|
||||
if !ok {
|
||||
return c
|
||||
}
|
||||
if maker != "" {
|
||||
maker = strings.ToUpper(maker[:1]) + maker[1:]
|
||||
}
|
||||
return strings.TrimSpace(maker + " " + rest)
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
package hardware
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestDMISize(t *testing.T) {
|
||||
for in, want := range map[string]uint64{
|
||||
"8 GB": 8 << 30, "16384 MB": 16384 << 20, "512 kB": 512 << 10, "1 TB": 1 << 40,
|
||||
"No Module Installed": 0, "": 0, "8 GiB": 0,
|
||||
} {
|
||||
if got := dmiSize(in); got != want {
|
||||
t.Errorf("dmiSize(%q) = %d, want %d", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMakerName(t *testing.T) {
|
||||
cases := []struct{ v, id, want string }{
|
||||
{"Samsung", "", "Samsung"}, // the module says who it is
|
||||
{"0x0B45", "Bank 12, Hex 0x45", "Wilk Elektronik"}, // only a JEDEC id, and one we know
|
||||
{"0xFFFF", "", "0xFFFF"}, // one we don't: the number, not a guess
|
||||
{"Unknown", "", ""},
|
||||
{"", "0xCE00", "Samsung"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := makerName(c.v, c.id); got != c.want {
|
||||
t.Errorf("makerName(%q, %q) = %q, want %q", c.v, c.id, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStripIDAndDisplayClass(t *testing.T) {
|
||||
if got := stripID("Alder Lake-N [UHD Graphics] [46d1]"); got != "Alder Lake-N [UHD Graphics]" {
|
||||
t.Errorf("stripID = %q", got)
|
||||
}
|
||||
if got := stripID("no brackets here"); got != "no brackets here" {
|
||||
t.Errorf("stripID left a plain name alone wrong: %q", got)
|
||||
}
|
||||
if displayClass(`VGA compatible controller [0300]`) != "VGA" {
|
||||
t.Error("a VGA controller should be graphics")
|
||||
}
|
||||
if displayClass(`Ethernet controller [0200]`) != "" {
|
||||
t.Error("a network card is not graphics")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLeadingIntAndJoin(t *testing.T) {
|
||||
if leadingInt("3200 MT/s") != 3200 || leadingInt("") != 0 || leadingInt("Unknown") != 0 {
|
||||
t.Error("leadingInt")
|
||||
}
|
||||
if got := join("GMKtec", "GMKtec NucBox G3"); got != "GMKtec NucBox G3" {
|
||||
t.Errorf("a maker the product already names should not be said twice: %q", got)
|
||||
}
|
||||
if got := join("Dell Inc.", "OptiPlex 7040"); got != "Dell Inc. OptiPlex 7040" {
|
||||
t.Errorf("join = %q", got)
|
||||
}
|
||||
if got := join("To Be Filled By O.E.M.", "System Product Name"); got != "" {
|
||||
t.Errorf("firmware boilerplate is not a model: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
const sample = `# dmidecode 3.6
|
||||
Handle 0x0028, DMI type 17, 92 bytes
|
||||
Memory Device
|
||||
Array Handle: 0x0027
|
||||
Size: 8 GB
|
||||
Form Factor: SODIMM
|
||||
Locator: Controller0-ChannelA-DIMM0
|
||||
Type: DDR4
|
||||
Speed: 3200 MT/s
|
||||
Manufacturer: 0x0B45
|
||||
Part Number: WPBH32D408SWD-8G
|
||||
Configured Memory Speed: 2933 MT/s
|
||||
Rank: 1
|
||||
|
||||
Handle 0x0029, DMI type 17, 92 bytes
|
||||
Memory Device
|
||||
Size: No Module Installed
|
||||
Locator: Controller1-ChannelA-DIMM0
|
||||
Type: Unknown
|
||||
`
|
||||
|
||||
// parseRecords is what readDMI does once dmidecode has answered; this exercises it
|
||||
// without needing root or the tool.
|
||||
func TestMemoryFromRecords(t *testing.T) {
|
||||
recs := parse(sample)
|
||||
m := memory(recs)
|
||||
if m.Slots != 2 || m.Filled != 1 {
|
||||
t.Fatalf("slots %d filled %d, want 2 and 1", m.Slots, m.Filled)
|
||||
}
|
||||
mod := m.Modules[0]
|
||||
if mod.Size != 8<<30 || mod.Type != "DDR4" || mod.Maker != "Wilk Elektronik" || mod.Part != "WPBH32D408SWD-8G" {
|
||||
t.Errorf("first module read wrong: %+v", mod)
|
||||
}
|
||||
if mod.SpeedMTs != 3200 || mod.RunningMTs != 2933 {
|
||||
t.Errorf("a module running below its rating should say both: %d / %d", mod.RunningMTs, mod.SpeedMTs)
|
||||
}
|
||||
if !m.Modules[1].Empty || m.Modules[1].Locator != "Controller1-ChannelA-DIMM0" {
|
||||
t.Errorf("empty slot read wrong: %+v", m.Modules[1])
|
||||
}
|
||||
if m.Installed != 8<<30 {
|
||||
t.Errorf("installed = %d", m.Installed)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
bolt "go.etcd.io/bbolt"
|
||||
)
|
||||
|
||||
// Cache keeps finished previews in one database file under Ham Lab's own data directory —
|
||||
// thumbs.db, a bbolt key/value store: pure Go, so the binary stays static, and one file
|
||||
// per machine rather than a tree of hash-named JPEGs. An entry is one file's preview keyed
|
||||
// by its path, size and mtime, so a file that changes gets a new preview and an unchanged
|
||||
// one is never decoded twice. A file ffmpeg could not decode is kept the same way, with
|
||||
// ffmpeg's reason and no picture, so a folder of odd files does not re-run ffmpeg on every
|
||||
// visit. Nothing expires: a preview is a few kilobytes, and only files somebody actually
|
||||
// scrolled past are ever made. The file is opened the first time something is stored, so a
|
||||
// machine nobody browses with previews on never has one.
|
||||
type Cache struct {
|
||||
Path string
|
||||
mu sync.Mutex
|
||||
db *bolt.DB
|
||||
err error
|
||||
}
|
||||
|
||||
// Entry is one cached preview, or the remembered reason there is none.
|
||||
type Entry struct {
|
||||
Path string `json:"path"`
|
||||
Engine string `json:"engine,omitempty"`
|
||||
Codec string `json:"codec,omitempty"`
|
||||
Width int `json:"width,omitempty"`
|
||||
Height int `json:"height,omitempty"`
|
||||
Made time.Time `json:"made"`
|
||||
Failure string `json:"failure,omitempty"` // set instead of JPEG when ffmpeg refused the file
|
||||
JPEG []byte `json:"-"`
|
||||
}
|
||||
|
||||
var (
|
||||
bucketThumbs = []byte("thumbs")
|
||||
bucketMeta = []byte("meta")
|
||||
keyCount = []byte("count")
|
||||
)
|
||||
|
||||
// Key identifies one file's preview. "v1" is the preview format: bump it to retire every
|
||||
// cached preview at once.
|
||||
func Key(path string, size, mtime int64) string {
|
||||
h := sha256.Sum256([]byte("v1\x00" + path + "\x00" + strconv.FormatInt(size, 10) + "\x00" + strconv.FormatInt(mtime, 10)))
|
||||
return hex.EncodeToString(h[:])
|
||||
}
|
||||
|
||||
func NewCache(path string) *Cache { return &Cache{Path: path} }
|
||||
|
||||
// open the database once; an open that failed is not retried, and previews still get made
|
||||
// and served — they are simply not kept, and the log says why.
|
||||
func (c *Cache) open() (*bolt.DB, error) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.db != nil || c.err != nil {
|
||||
return c.db, c.err
|
||||
}
|
||||
db, err := bolt.Open(c.Path, 0o600, &bolt.Options{Timeout: 2 * time.Second})
|
||||
if err != nil {
|
||||
c.err = err
|
||||
return nil, err
|
||||
}
|
||||
if err := db.Update(func(tx *bolt.Tx) error {
|
||||
if _, err := tx.CreateBucketIfNotExists(bucketThumbs); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := tx.CreateBucketIfNotExists(bucketMeta)
|
||||
return err
|
||||
}); err != nil {
|
||||
db.Close()
|
||||
c.err = err
|
||||
return nil, err
|
||||
}
|
||||
c.db = db
|
||||
return db, nil
|
||||
}
|
||||
|
||||
// Close releases the file. Previews after it are made but not kept.
|
||||
func (c *Cache) Close() error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.db == nil {
|
||||
return nil
|
||||
}
|
||||
err := c.db.Close()
|
||||
c.db, c.err = nil, errors.New("cache closed")
|
||||
return err
|
||||
}
|
||||
|
||||
// Get returns the entry for a key. A database that does not exist yet is a miss, not a reason
|
||||
// to make one.
|
||||
func (c *Cache) Get(key string) (Entry, bool, error) {
|
||||
c.mu.Lock()
|
||||
db := c.db
|
||||
c.mu.Unlock()
|
||||
if db == nil {
|
||||
if _, err := os.Stat(c.Path); err != nil {
|
||||
return Entry{}, false, nil
|
||||
}
|
||||
var err error
|
||||
if db, err = c.open(); err != nil {
|
||||
return Entry{}, false, err
|
||||
}
|
||||
}
|
||||
var e Entry
|
||||
found := false
|
||||
err := db.View(func(tx *bolt.Tx) error {
|
||||
v := tx.Bucket(bucketThumbs).Get([]byte(key))
|
||||
if v == nil {
|
||||
return nil
|
||||
}
|
||||
var err error
|
||||
e, err = decode(v) // copies: a value is only valid inside the transaction
|
||||
found = err == nil
|
||||
return err
|
||||
})
|
||||
return e, found, err
|
||||
}
|
||||
|
||||
// Put stores an entry, replacing any under the same key.
|
||||
func (c *Cache) Put(key string, e Entry) error {
|
||||
if len(e.JPEG) == 0 && e.Failure == "" {
|
||||
return errors.New("empty preview")
|
||||
}
|
||||
if e.Made.IsZero() {
|
||||
e.Made = time.Now()
|
||||
}
|
||||
db, err := c.open()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
v, err := encode(e)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return db.Update(func(tx *bolt.Tx) error {
|
||||
b := tx.Bucket(bucketThumbs)
|
||||
k := []byte(key)
|
||||
fresh := b.Get(k) == nil
|
||||
if err := b.Put(k, v); err != nil {
|
||||
return err
|
||||
}
|
||||
if fresh {
|
||||
m := tx.Bucket(bucketMeta)
|
||||
return m.Put(keyCount, be64(count(m)+1))
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// Stats is how many previews are kept and how big the file is.
|
||||
func (c *Cache) Stats() (entries int64, bytes int64) {
|
||||
st, err := os.Stat(c.Path)
|
||||
if err != nil {
|
||||
return 0, 0
|
||||
}
|
||||
bytes = st.Size()
|
||||
db, err := c.open()
|
||||
if err != nil {
|
||||
return 0, bytes
|
||||
}
|
||||
_ = db.View(func(tx *bolt.Tx) error { entries = count(tx.Bucket(bucketMeta)); return nil })
|
||||
return entries, bytes
|
||||
}
|
||||
|
||||
func count(m *bolt.Bucket) int64 {
|
||||
if v := m.Get(keyCount); len(v) == 8 {
|
||||
return int64(binary.BigEndian.Uint64(v))
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func be64(n int64) []byte {
|
||||
b := make([]byte, 8)
|
||||
binary.BigEndian.PutUint64(b, uint64(n))
|
||||
return b
|
||||
}
|
||||
|
||||
// A value is a 4-byte header length, the header as JSON, then the JPEG bytes.
|
||||
func encode(e Entry) ([]byte, error) {
|
||||
hdr, err := json.Marshal(e)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v := make([]byte, 4+len(hdr)+len(e.JPEG))
|
||||
binary.BigEndian.PutUint32(v, uint32(len(hdr)))
|
||||
copy(v[4:], hdr)
|
||||
copy(v[4+len(hdr):], e.JPEG)
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func decode(v []byte) (Entry, error) {
|
||||
if len(v) < 4 {
|
||||
return Entry{}, errors.New("short value")
|
||||
}
|
||||
n := int(binary.BigEndian.Uint32(v))
|
||||
if 4+n > len(v) {
|
||||
return Entry{}, errors.New("bad header length")
|
||||
}
|
||||
var e Entry
|
||||
if err := json.Unmarshal(v[4:4+n], &e); err != nil {
|
||||
return Entry{}, err
|
||||
}
|
||||
if rest := v[4+n:]; len(rest) > 0 {
|
||||
e.JPEG = append([]byte(nil), rest...)
|
||||
}
|
||||
return e, nil
|
||||
}
|
||||
@@ -0,0 +1,342 @@
|
||||
package media
|
||||
|
||||
// What to run when a video is played in the browser: the qualities it is offered in (the
|
||||
// ladder), the ways a machine could make one of them on the device asked for, in the order
|
||||
// worth trying, and the ffmpeg argument list for one piece-making run. All decisions, no
|
||||
// processes, so it can be tested. Everything is H.264 High and AAC: what every browser plays.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Rung is one quality the player can pick: a size, a frame rate and a bit rate.
|
||||
type Rung struct {
|
||||
Name string `json:"name"` // in URLs: the width, "1920"
|
||||
Label string `json:"label"` // what the menu says: "1080s", "720p"
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
FPS float64 `json:"fps"`
|
||||
Bitrate int `json:"bitrate"` // video, bits a second, on average
|
||||
Peak int `json:"peak"` // what the playlist promises, sound included
|
||||
level string // H.264 level, "5.2"
|
||||
codecs string // for the master playlist
|
||||
}
|
||||
|
||||
// One step of the ladder: the long side for a 360 video (its width), the short side for a flat
|
||||
// one, and the bit rate (Mb/s ×10) at more than 30 frames a second and at 30 or fewer.
|
||||
type step struct {
|
||||
size int
|
||||
label string
|
||||
hi, lo int
|
||||
cap30 bool
|
||||
}
|
||||
|
||||
// A 360 picture is seen a quarter of its width at a time, so it needs more pixels than a flat
|
||||
// one for the same sharpness; YouTube names its sizes by the flat picture they look like
|
||||
// ("1080s"), and so does this.
|
||||
var vrSteps = []step{
|
||||
{3840, "2160s", 220, 160, false},
|
||||
{2880, "1440s", 120, 90, false},
|
||||
{1920, "1080s", 60, 45, false},
|
||||
{1280, "720s", 25, 25, true},
|
||||
}
|
||||
|
||||
var flatSteps = []step{
|
||||
{2160, "2160p", 160, 120, false},
|
||||
{1440, "1440p", 90, 65, false},
|
||||
{1080, "1080p", 50, 35, false},
|
||||
{720, "720p", 25, 25, true},
|
||||
{480, "480p", 12, 12, true},
|
||||
}
|
||||
|
||||
const audioRate = 160_000
|
||||
|
||||
func even(x float64) int { return int(math.Round(x/2)) * 2 }
|
||||
|
||||
// Ladder is the qualities a video is offered in, best first: every step no bigger than the
|
||||
// picture itself, or the picture's own size if it is smaller than all of them.
|
||||
func Ladder(i Info) []Rung {
|
||||
v := i.Video
|
||||
if v == nil || v.Width <= 0 || v.Height <= 0 {
|
||||
return nil
|
||||
}
|
||||
vr := v.Spherical
|
||||
steps := flatSteps
|
||||
w, h := v.Shown()
|
||||
src := min(w, h) // a flat picture is named by its short side
|
||||
if vr {
|
||||
steps, src = vrSteps, w
|
||||
}
|
||||
// A picture between two steps (a 2292×960 capture) gets its own size on top, at the bit
|
||||
// rate of the step above, so the best on offer is never less than the file.
|
||||
var out []Rung
|
||||
for k, s := range steps {
|
||||
if s.size > src {
|
||||
continue
|
||||
}
|
||||
if len(out) == 0 && s.size < src && k > 0 { // never above the top step: H.264 stops there
|
||||
own := steps[k-1]
|
||||
own.size, own.label, own.cap30 = src, strconv.Itoa(src)+map[bool]string{true: "s", false: "p"}[vr], false
|
||||
out = append(out, rung(i, own))
|
||||
}
|
||||
out = append(out, rung(i, s))
|
||||
}
|
||||
if len(out) == 0 {
|
||||
last := steps[len(steps)-1]
|
||||
last.size, last.label = src, strconv.Itoa(src)+map[bool]string{true: "s", false: "p"}[vr]
|
||||
out = append(out, rung(i, last))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func rung(i Info, s step) Rung {
|
||||
v := i.Video
|
||||
sw, sh := v.Shown()
|
||||
var w, h int
|
||||
switch {
|
||||
case v.Spherical:
|
||||
w, h = s.size, even(float64(s.size)*float64(sh)/float64(sw))
|
||||
case sw >= sh:
|
||||
h, w = s.size, even(float64(s.size)*float64(sw)/float64(sh))
|
||||
default: // standing up
|
||||
w, h = s.size, even(float64(s.size)*float64(sh)/float64(sw))
|
||||
}
|
||||
fps := v.FPS
|
||||
if fps <= 0 {
|
||||
fps = 30
|
||||
}
|
||||
for fps > 60.5 || (s.cap30 && fps > 30.5) {
|
||||
fps /= 2
|
||||
}
|
||||
fps = math.Round(fps*1000) / 1000
|
||||
br := s.lo
|
||||
if fps > 30.5 {
|
||||
br = s.hi
|
||||
}
|
||||
r := Rung{Name: strconv.Itoa(w), Label: s.label, Width: w, Height: h, FPS: fps, Bitrate: br * 100_000}
|
||||
r.Peak = r.Bitrate*3/2 + audioRate
|
||||
r.level = level(w, h, fps)
|
||||
lv, _ := strconv.ParseFloat(r.level, 64)
|
||||
r.codecs = fmt.Sprintf("avc1.6400%02x", int(math.Round(lv*10)))
|
||||
if i.Audio != nil {
|
||||
r.codecs += ",mp4a.40.2"
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// level is the lowest H.264 level that takes this many macroblocks a frame and a second.
|
||||
func level(w, h int, fps float64) string {
|
||||
mbs := ((w + 15) / 16) * ((h + 15) / 16)
|
||||
rate := float64(mbs) * fps
|
||||
for _, l := range []struct {
|
||||
name string
|
||||
mbps float64
|
||||
frame int
|
||||
}{
|
||||
{"3.1", 108000, 3600}, {"3.2", 216000, 5120}, {"4.0", 245760, 8192}, {"4.2", 522240, 8704},
|
||||
{"5.0", 589824, 22080}, {"5.1", 983040, 36864}, {"5.2", 2073600, 36864},
|
||||
} {
|
||||
if rate <= l.mbps && mbs <= l.frame {
|
||||
return l.name
|
||||
}
|
||||
}
|
||||
return "6.2"
|
||||
}
|
||||
|
||||
// fpsArg is a frame rate as ffmpeg's fps filter takes it, exactly for the NTSC rates.
|
||||
func fpsArg(f float64) string {
|
||||
for _, n := range []float64{24, 30, 60, 120} {
|
||||
if math.Abs(f-n*1000/1001) < 0.01 {
|
||||
return fmt.Sprintf("%d/1001", int(n*1000))
|
||||
}
|
||||
}
|
||||
return strings.TrimRight(strings.TrimRight(strconv.FormatFloat(f, 'f', 3, 64), "0"), ".")
|
||||
}
|
||||
|
||||
// Recipe is one way to make a quality: the engine's name for the record, the device it runs
|
||||
// on ("cpu" for the software encoder), and the ffmpeg arguments that go before the input, the
|
||||
// video filter, and the video codec.
|
||||
type Recipe struct {
|
||||
Engine string
|
||||
Device string
|
||||
Pre []string
|
||||
VF string
|
||||
Video []string
|
||||
}
|
||||
|
||||
// CPU is the device name for software encoding.
|
||||
const CPU = "cpu"
|
||||
|
||||
// Device is one thing a machine can make H.264 on, for the Transcode device setting.
|
||||
type Device struct {
|
||||
ID string `json:"id"` // a render node, or "cpu"
|
||||
Name string `json:"name"` // "Intel Arc A380", "CPU (libx264)"
|
||||
Integrated bool `json:"integrated,omitempty"`
|
||||
OK bool `json:"ok"`
|
||||
Reason string `json:"reason,omitempty"` // when not OK: why
|
||||
}
|
||||
|
||||
// Devices lists what this machine could make the player's H.264 on: every GPU, usable or not
|
||||
// (and why not), then the CPU.
|
||||
func Devices(c Caps) []Device {
|
||||
var out []Device
|
||||
if c.Error != "" {
|
||||
return []Device{{ID: CPU, Name: "CPU", Reason: c.Error}}
|
||||
}
|
||||
for _, g := range c.GPUs {
|
||||
d := Device{ID: g.Device, Name: g.Name, Integrated: g.Integrated}
|
||||
if d.Name == "" {
|
||||
d.Name = g.Device
|
||||
}
|
||||
switch {
|
||||
case g.API != "vaapi":
|
||||
d.Reason = "only VAAPI devices encode here"
|
||||
case g.Encodes != nil && !has(g.Encodes, "h264"):
|
||||
d.Reason = "doesn't encode H.264"
|
||||
default:
|
||||
d.OK = true
|
||||
}
|
||||
out = append(out, d)
|
||||
}
|
||||
for _, g := range c.Unusable {
|
||||
name := g.Name
|
||||
if name == "" {
|
||||
name = g.Device
|
||||
}
|
||||
out = append(out, Device{ID: g.Device, Name: name, Integrated: g.Integrated, Reason: g.Reason})
|
||||
}
|
||||
d := Device{ID: CPU, Name: "CPU (libx264)", OK: has(c.Encoders, "libx264")}
|
||||
if !d.OK {
|
||||
d.Reason = "this ffmpeg has no libx264"
|
||||
}
|
||||
return append(out, d)
|
||||
}
|
||||
|
||||
// recipes are the ways to make quality r of src on device, best first: on a GPU, decoding on
|
||||
// the same card (frames never leave it) before decoding in software and uploading; the CPU
|
||||
// last, as the fallback whatever was asked. device "" is any: a card on the bus before the
|
||||
// integrated GPU. An empty list means nothing here can.
|
||||
func recipes(c Caps, device string, src Info, r Rung) []Recipe {
|
||||
var out []Recipe
|
||||
gpus := append([]GPU(nil), c.GPUs...)
|
||||
sort.SliceStable(gpus, func(i, j int) bool { return !gpus[i].Integrated && gpus[j].Integrated })
|
||||
kbps := r.Bitrate / 1000
|
||||
rate := []string{"-b:v", strconv.Itoa(kbps) + "k", "-maxrate", strconv.Itoa(kbps*3/2) + "k", "-bufsize", strconv.Itoa(kbps*2) + "k",
|
||||
"-profile:v", "high", "-level", r.level}
|
||||
fps := ""
|
||||
if src.Video == nil || src.Video.FPS <= 0 || math.Abs(src.Video.FPS-r.FPS) > 0.01 {
|
||||
fps = "fps=" + fpsArg(r.FPS) + ","
|
||||
}
|
||||
if device != CPU {
|
||||
for i := range gpus {
|
||||
g := &gpus[i]
|
||||
// Encodes is vainfo's answer; without vainfo there is none, and the card is tried
|
||||
// anyway, with the next recipe behind it.
|
||||
if g.API != "vaapi" || (g.Encodes != nil && !has(g.Encodes, "h264")) || (device != "" && g.Device != device) {
|
||||
continue
|
||||
}
|
||||
video := append(append([]string{"-c:v", "h264_vaapi"}, rate...), "-bf", "0")
|
||||
name := g.Name
|
||||
if name == "" {
|
||||
name = g.Device
|
||||
}
|
||||
// The card doesn't turn a phone's picture by itself (ffmpeg's own turning is for
|
||||
// pictures in memory), so it is told to, and ffmpeg is told not to.
|
||||
scale := turn(src) + fmt.Sprintf("scale_vaapi=w=%d:h=%d:format=nv12", r.Width, r.Height)
|
||||
// HDR10 (PQ) is tone-mapped on the card, which needs the file's mastering data;
|
||||
// HLG is made to look right on an ordinary screen as it is, so it is only scaled
|
||||
// (tonemap_vaapi refuses it). Either way the next recipe tone-maps in software.
|
||||
if src.Video != nil && has(g.Codecs, src.Video.Codec) {
|
||||
vf := fps + scale
|
||||
if src.PQ() {
|
||||
vf = fps + "tonemap_vaapi=format=nv12:t=bt709:p=bt709:m=bt709," + scale
|
||||
}
|
||||
out = append(out, Recipe{Engine: name, Device: g.Device, Pre: []string{"-noautorotate", "-hwaccel", "vaapi", "-hwaccel_device", g.Device, "-hwaccel_output_format", "vaapi"}, VF: vf, Video: video})
|
||||
}
|
||||
cpuVF := fmt.Sprintf("%sformat=nv12,scale=%d:%d,hwupload", fps, r.Width, r.Height)
|
||||
if src.PQ() {
|
||||
cpuVF = fps + toneMap + fmt.Sprintf(",scale=%d:%d,format=nv12,hwupload", r.Width, r.Height)
|
||||
}
|
||||
out = append(out, Recipe{Engine: name + ", decoded on the CPU", Device: g.Device, Pre: []string{"-init_hw_device", "vaapi=va:" + g.Device, "-filter_hw_device", "va"},
|
||||
VF: cpuVF, Video: video})
|
||||
}
|
||||
}
|
||||
if has(c.Encoders, "libx264") {
|
||||
vf := fmt.Sprintf("%sscale=%d:%d,format=yuv420p", fps, r.Width, r.Height)
|
||||
if src.PQ() {
|
||||
vf = fps + toneMap + fmt.Sprintf(",scale=%d:%d,format=yuv420p", r.Width, r.Height)
|
||||
}
|
||||
out = append(out, Recipe{Engine: "CPU (libx264)", Device: CPU, VF: vf, Video: append([]string{"-c:v", "libx264", "-preset", "veryfast", "-sc_threshold", "0"}, rate...)})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// turn is the card's filter that turns a picture the way it is meant to be shown.
|
||||
func turn(src Info) string {
|
||||
if src.Video == nil {
|
||||
return ""
|
||||
}
|
||||
switch src.Video.Rotation {
|
||||
case 90:
|
||||
return "transpose_vaapi=dir=clock,"
|
||||
case 180:
|
||||
return "transpose_vaapi=dir=reversal,"
|
||||
case 270:
|
||||
return "transpose_vaapi=dir=cclock,"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// toneMap turns an HDR10 picture into an ordinary one, in software.
|
||||
const toneMap = "zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable,zscale=t=bt709:m=bt709:r=tv"
|
||||
|
||||
// Seg is the length of a piece, in seconds.
|
||||
const Seg = 3.0
|
||||
|
||||
// tsBase is added to every run's times, so pieces from different runs follow on: without it
|
||||
// a run from the top would start just below zero (the B-frame and AAC delay) and the muxer
|
||||
// would shift that run alone.
|
||||
const tsBase = 10
|
||||
|
||||
// pieceArgs is one run making quality r from piece n on, into dir as <n>.ts: an accurate seek
|
||||
// to n×Seg, a keyframe on every Seg mark and nowhere else (so every piece starts on one, and
|
||||
// pieces from any run and quality line up), times put back where that is in the film. in is a
|
||||
// path or, for a file on another machine, an http URL.
|
||||
func pieceArgs(dir, in string, src Info, rc Recipe, r Rung, n int) []string {
|
||||
at := strconv.FormatFloat(float64(n)*Seg, 'f', 3, 64)
|
||||
a := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
|
||||
a = append(a, rc.Pre...)
|
||||
if strings.HasPrefix(in, "http://") || strings.HasPrefix(in, "https://") {
|
||||
a = append(a, "-reconnect", "1", "-reconnect_on_network_error", "1", "-reconnect_delay_max", "4")
|
||||
}
|
||||
a = append(a, "-ss", at, "-i", in, "-map", "0:v:0", "-map", "0:a:0?", "-sn", "-dn")
|
||||
if rc.VF != "" {
|
||||
a = append(a, "-vf", rc.VF)
|
||||
}
|
||||
a = append(a, rc.Video...)
|
||||
if src.Audio == nil {
|
||||
a = append(a, "-an")
|
||||
} else {
|
||||
a = append(a, "-c:a", "aac", "-b:a", "160k", "-ac", "2")
|
||||
}
|
||||
gop := int(math.Ceil(r.FPS*Seg)) * 4
|
||||
a = append(a, "-g", strconv.Itoa(gop), "-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%g)", Seg),
|
||||
"-output_ts_offset", strconv.FormatFloat(float64(n)*Seg+tsBase, 'f', 3, 64),
|
||||
"-f", "hls", "-hls_time", strconv.FormatFloat(Seg-0.5, 'f', 2, 64), "-hls_list_size", "0",
|
||||
"-hls_flags", "temp_file", "-hls_segment_type", "mpegts", "-start_number", strconv.Itoa(n),
|
||||
"-hls_segment_filename", dir+"/%d.ts", dir+"/ffmpeg.m3u8")
|
||||
return a
|
||||
}
|
||||
|
||||
func has(list []string, x string) bool {
|
||||
for _, v := range list {
|
||||
if v == x {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func src(codec string, w, h int, fps float64) Info {
|
||||
return Info{Duration: 100, Video: &Track{Codec: codec, Width: w, Height: h, FPS: fps}, Audio: &Track{Codec: "opus", Channels: 2}}
|
||||
}
|
||||
|
||||
func TestLadder(t *testing.T) {
|
||||
vr := src("hevc", 7680, 3840, 50)
|
||||
vr.Video.Spherical = true
|
||||
r := Ladder(vr)
|
||||
if len(r) != 4 || r[0].Name != "3840" || r[0].Height != 1920 || r[0].level != "5.2" || r[3].FPS != 25 || r[0].Label != "2160s" {
|
||||
t.Fatalf("8K 360: %+v", r)
|
||||
}
|
||||
if r[0].codecs != "avc1.640034,mp4a.40.2" {
|
||||
t.Fatalf("codecs %q", r[0].codecs)
|
||||
}
|
||||
r = Ladder(src("h264", 2292, 960, 60)) // a capture: its own size on top, then the steps under it
|
||||
if len(r) != 3 || r[0].Label != "960p" || r[0].Width != 2292 || r[0].FPS != 60 || r[0].Bitrate != 5_000_000 || r[1].Label != "720p" || r[1].Width != 1720 || r[1].FPS != 30 {
|
||||
t.Fatalf("2292×960: %+v", r)
|
||||
}
|
||||
r = Ladder(src("h264", 1920, 1080, 59.94))
|
||||
if len(r) != 3 || r[0].Label != "1080p" || r[0].FPS != 59.94 || r[1].FPS != 29.97 || r[2].Width != 854 {
|
||||
t.Fatalf("1080p: %+v", r)
|
||||
}
|
||||
if r = Ladder(src("h264", 1080, 1920, 30)); r[0].Width != 1080 || r[0].Height != 1920 { // standing up
|
||||
t.Fatalf("portrait: %+v", r)
|
||||
}
|
||||
if r = Ladder(src("h264", 320, 240, 30)); len(r) != 1 || r[0].Width != 320 || r[0].Label != "240p" {
|
||||
t.Fatalf("tiny: %+v", r)
|
||||
}
|
||||
silent := src("h264", 1280, 720, 30)
|
||||
silent.Audio = nil
|
||||
if r = Ladder(silent); strings.Contains(r[0].codecs, "mp4a") {
|
||||
t.Fatalf("no sound, no AAC: %q", r[0].codecs)
|
||||
}
|
||||
if Ladder(Info{Duration: 10}) != nil {
|
||||
t.Fatal("no picture, no ladder")
|
||||
}
|
||||
if fpsArg(29.97) != "30000/1001" || fpsArg(25) != "25" {
|
||||
t.Fatal(fpsArg(29.97), fpsArg(25))
|
||||
}
|
||||
}
|
||||
|
||||
func gridCaps() Caps {
|
||||
return Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac", "libsvtav1", "libx264", "libx265"},
|
||||
GPUs: []GPU{
|
||||
{API: "vaapi", Device: "/dev/dri/renderD128", Name: "Intel UHD Graphics 750", Integrated: true, Codecs: []string{"h264", "hevc", "vp9"}, Encodes: []string{"h264", "hevc"}},
|
||||
{API: "vaapi", Device: "/dev/dri/renderD129", Name: "Intel Arc A380", Codecs: []string{"h264", "hevc", "vp9"}, Encodes: []string{"av1", "h264", "hevc"}},
|
||||
}}
|
||||
}
|
||||
|
||||
func engineNames(r []Recipe) []string {
|
||||
out := make([]string, len(r))
|
||||
for i := range r {
|
||||
out[i] = r[i].Engine
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestRecipes(t *testing.T) {
|
||||
c := gridCaps()
|
||||
in := src("hevc", 3840, 2160, 60)
|
||||
r1080 := Ladder(in)[2]
|
||||
// any device: the card first, decoding on it; then the iGPU the same way; then the CPU
|
||||
got := strings.Join(engineNames(recipes(c, "", in, r1080)), "|")
|
||||
if got != "Intel Arc A380|Intel Arc A380, decoded on the CPU|Intel UHD Graphics 750|Intel UHD Graphics 750, decoded on the CPU|CPU (libx264)" {
|
||||
t.Fatalf("any: %s", got)
|
||||
}
|
||||
// the iGPU asked for: only it, then the CPU as the fallback
|
||||
if got = strings.Join(engineNames(recipes(c, "/dev/dri/renderD128", in, r1080)), "|"); got != "Intel UHD Graphics 750|Intel UHD Graphics 750, decoded on the CPU|CPU (libx264)" {
|
||||
t.Fatalf("iGPU: %s", got)
|
||||
}
|
||||
if got = strings.Join(engineNames(recipes(c, CPU, in, r1080)), "|"); got != "CPU (libx264)" {
|
||||
t.Fatalf("cpu: %s", got)
|
||||
}
|
||||
// an AV1 source: neither card decodes it, so no same-card recipe
|
||||
if got = strings.Join(engineNames(recipes(c, "/dev/dri/renderD129", src("av1", 1920, 1080, 30), r1080)), "|"); got != "Intel Arc A380, decoded on the CPU|CPU (libx264)" {
|
||||
t.Fatalf("av1: %s", got)
|
||||
}
|
||||
// a machine with no GPU and no x264: nothing
|
||||
bare := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac"}}
|
||||
if len(recipes(bare, "", in, r1080)) != 0 {
|
||||
t.Fatal("no x264 means nothing")
|
||||
}
|
||||
// 30 fps from 60 for the lower rungs; HDR tone-mapped on both paths
|
||||
r720 := Ladder(in)[3]
|
||||
hdr := src("hevc", 3840, 2160, 60)
|
||||
hdr.Video.ColorTransfer = "smpte2084"
|
||||
rs := recipes(c, "", hdr, r720)
|
||||
if !strings.HasPrefix(rs[0].VF, "fps=30,tonemap_vaapi") || !strings.Contains(rs[1].VF, "tonemap=hable") || !strings.HasSuffix(rs[1].VF, "hwupload") || !strings.Contains(rs[len(rs)-1].VF, "tonemap=hable") {
|
||||
t.Fatalf("hdr: %q … %q … %q", rs[0].VF, rs[1].VF, rs[len(rs)-1].VF)
|
||||
}
|
||||
hdr.Video.ColorTransfer = "arib-std-b67" // HLG: shown as it is, on every path
|
||||
if rs = recipes(c, "", hdr, r720); strings.Contains(rs[0].VF, "tonemap") || strings.Contains(rs[1].VF, "tonemap") {
|
||||
t.Fatalf("hlg: %q … %q", rs[0].VF, rs[1].VF)
|
||||
}
|
||||
// a phone's picture, upright: the ladder by the shown size, turned on the card
|
||||
phone := src("hevc", 1920, 1080, 30)
|
||||
phone.Video.Rotation = 90
|
||||
lr := Ladder(phone)
|
||||
if lr[0].Width != 1080 || lr[0].Height != 1920 {
|
||||
t.Fatalf("upright: %+v", lr[0])
|
||||
}
|
||||
if rs = recipes(c, "", phone, lr[0]); !strings.HasPrefix(rs[0].VF, "transpose_vaapi=dir=clock,scale_vaapi=w=1080:h=1920") || rs[0].Pre[0] != "-noautorotate" || strings.Contains(rs[1].VF, "transpose") {
|
||||
t.Fatalf("turned: %q %v / %q", rs[0].VF, rs[0].Pre, rs[1].VF)
|
||||
}
|
||||
// no vainfo: the card is tried all the same
|
||||
nov := gridCaps()
|
||||
nov.GPUs[1].Encodes = nil
|
||||
if rs = recipes(nov, "/dev/dri/renderD129", in, r1080); rs[0].Engine != "Intel Arc A380" || !Devices(nov)[1].OK {
|
||||
t.Fatalf("no vainfo: %v", engineNames(rs))
|
||||
}
|
||||
if rs := recipes(c, "", in, r1080); strings.Contains(rs[0].VF, "fps=") {
|
||||
t.Fatalf("same rate needs no fps filter: %q", rs[0].VF)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDevices(t *testing.T) {
|
||||
c := gridCaps()
|
||||
c.Unusable = []GPU{{Device: "/dev/dri/renderD130", Name: "Something", Reason: "permission denied"}}
|
||||
d := Devices(c)
|
||||
if len(d) != 4 || !d[0].OK || !d[1].OK || d[2].OK || d[2].Reason == "" || d[3].ID != CPU || !d[3].OK {
|
||||
t.Fatalf("%+v", d)
|
||||
}
|
||||
if d := Devices(Caps{Error: "ffmpeg is not installed"}); len(d) != 1 || d[0].OK {
|
||||
t.Fatalf("no ffmpeg: %+v", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPieceArgs(t *testing.T) {
|
||||
in := src("h264", 1920, 1080, 60)
|
||||
r := Ladder(in)[0]
|
||||
rc := recipes(gridCaps(), "/dev/dri/renderD129", in, r)[0]
|
||||
a := strings.Join(pieceArgs("/d", "/v.mkv", in, rc, r, 12), " ")
|
||||
for _, want := range []string{"-hwaccel vaapi -hwaccel_device /dev/dri/renderD129", "-ss 36.000 -i /v.mkv", "-map 0:v:0 -map 0:a:0?",
|
||||
"-vf scale_vaapi=w=1920:h=1080:format=nv12", "-c:v h264_vaapi -b:v 5000k -maxrate 7500k", "-level 4.2", "-c:a aac -b:a 160k -ac 2",
|
||||
"-force_key_frames expr:gte(t,n_forced*3)", "-output_ts_offset 46.000", "-hls_time 2.50", "-start_number 12", "-hls_segment_filename /d/%d.ts"} {
|
||||
if !strings.Contains(a, want) {
|
||||
t.Errorf("missing %q in\n%s", want, a)
|
||||
}
|
||||
}
|
||||
if strings.Contains(a, "-reconnect") {
|
||||
t.Error("a path needs no reconnect")
|
||||
}
|
||||
if a := strings.Join(pieceArgs("/d", "http://10.0.0.234:8100/api/play-src/x", in, rc, r, 0), " "); !strings.Contains(a, "-reconnect 1") {
|
||||
t.Error("a URL reconnects")
|
||||
}
|
||||
in.Audio = nil
|
||||
if a := strings.Join(pieceArgs("/d", "/v", in, rc, r, 0), " "); !strings.Contains(a, "-an") {
|
||||
t.Error("no sound means -an")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCapsOnly(t *testing.T) {
|
||||
c := gridCaps()
|
||||
if got := c.Only("/dev/dri/renderD129"); len(got.GPUs) != 1 || got.GPUs[0].Name != "Intel Arc A380" || len(c.GPUs) != 2 {
|
||||
t.Fatalf("one card: %+v", got.GPUs)
|
||||
}
|
||||
if len(c.Only(CPU).GPUs) != 0 || len(c.Only("/dev/dri/renderD200").GPUs) != 0 || len(c.Only("").GPUs) != 2 {
|
||||
t.Fatal("cpu, a device not here, any")
|
||||
}
|
||||
// previews then go to that card, or the CPU
|
||||
if g := c.Only("/dev/dri/renderD129").pick("hevc"); g == nil || g.Device != "/dev/dri/renderD129" {
|
||||
t.Fatalf("pick: %+v", g)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
package media
|
||||
|
||||
// Running one long ffmpeg: as the daemon's user directly, or as root through this binary's
|
||||
// run-ffmpeg helper when the file needs root to read. The helper is told to finish by its
|
||||
// standard input closing, which also happens by itself if the daemon goes away — so a
|
||||
// transcode never outlives the console that asked for it.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type ffproc struct {
|
||||
cmd *exec.Cmd
|
||||
in io.WriteCloser // the helper's stdin, root mode only
|
||||
done chan struct{}
|
||||
err error
|
||||
tail *tailBuf
|
||||
}
|
||||
|
||||
// startFF launches ffmpeg with args, under nice where there is one.
|
||||
func startFF(c Caps, self string, root bool, args []string) (*ffproc, error) {
|
||||
var argv []string
|
||||
if root {
|
||||
argv = priv.Wrap(append([]string{self, "run-ffmpeg"}, args...)...)
|
||||
} else {
|
||||
argv = append([]string{c.FFmpeg}, args...)
|
||||
if c.Nice != "" {
|
||||
argv = append([]string{c.Nice, "-n", "10"}, argv...)
|
||||
}
|
||||
}
|
||||
cmd := exec.Command(argv[0], argv[1:]...)
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
p := &ffproc{cmd: cmd, done: make(chan struct{}), tail: &tailBuf{}}
|
||||
cmd.Stderr = p.tail
|
||||
cmd.Stdout = io.Discard
|
||||
if root {
|
||||
in, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
p.in = in
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
go func() {
|
||||
p.err = cmd.Wait()
|
||||
close(p.done)
|
||||
}()
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// exited reports whether ffmpeg has finished, and how.
|
||||
func (p *ffproc) exited() (bool, error) {
|
||||
select {
|
||||
case <-p.done:
|
||||
return true, p.err
|
||||
default:
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
// end asks ffmpeg to finish and waits a little for it.
|
||||
func (p *ffproc) end() {
|
||||
if done, _ := p.exited(); done {
|
||||
return
|
||||
}
|
||||
if p.in != nil {
|
||||
p.in.Close()
|
||||
} else {
|
||||
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGTERM)
|
||||
}
|
||||
select {
|
||||
case <-p.done:
|
||||
case <-time.After(5 * time.Second):
|
||||
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGKILL) // a root helper ignores this; it ends on its own within its own grace
|
||||
select {
|
||||
case <-p.done:
|
||||
case <-time.After(10 * time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// pause stops ffmpeg where it is (SIGSTOP), and resume lets it go on: a stream far enough
|
||||
// ahead of its player waits rather than making the rest of the film. A root run is paused by
|
||||
// its helper, which is told on its standard input.
|
||||
func (p *ffproc) pause() error { return p.signal(syscall.SIGSTOP, "stop\n") }
|
||||
func (p *ffproc) resume() error { return p.signal(syscall.SIGCONT, "cont\n") }
|
||||
|
||||
func (p *ffproc) signal(sig syscall.Signal, word string) error {
|
||||
if done, _ := p.exited(); done {
|
||||
return errors.New("ended")
|
||||
}
|
||||
if p.in != nil {
|
||||
_, err := io.WriteString(p.in, word)
|
||||
return err
|
||||
}
|
||||
return syscall.Kill(-p.cmd.Process.Pid, sig)
|
||||
}
|
||||
|
||||
// kill ends ffmpeg now, without waiting: a paused one too.
|
||||
func (p *ffproc) kill() {
|
||||
if done, _ := p.exited(); done {
|
||||
return
|
||||
}
|
||||
if p.in != nil {
|
||||
p.in.Close() // the helper ends it, waking it first if it is paused
|
||||
return
|
||||
}
|
||||
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGKILL)
|
||||
}
|
||||
|
||||
// lastWords is what ffmpeg said last, for the record.
|
||||
func (p *ffproc) lastWords() string { return p.tail.last() }
|
||||
|
||||
// tailBuf keeps the last few KB written to it.
|
||||
type tailBuf struct {
|
||||
mu sync.Mutex
|
||||
b []byte
|
||||
}
|
||||
|
||||
const tailKeep = 4096
|
||||
|
||||
func (t *tailBuf) Write(b []byte) (int, error) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.b = append(t.b, b...)
|
||||
if len(t.b) > tailKeep {
|
||||
t.b = t.b[len(t.b)-tailKeep:]
|
||||
}
|
||||
return len(b), nil
|
||||
}
|
||||
|
||||
func (t *tailBuf) last() string {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
return lastLine(string(bytes.TrimSpace(t.b)))
|
||||
}
|
||||
|
||||
// RunFFmpegHelper implements `hamlab run-ffmpeg <args…>`: ffmpeg as this user (root, when
|
||||
// the daemon asked through sudo), finishing when stdin closes. Returns ffmpeg's exit code.
|
||||
func RunFFmpegHelper(args []string, stdin io.Reader, stderr io.Writer) int {
|
||||
ffmpeg, err := exec.LookPath("ffmpeg")
|
||||
if err != nil {
|
||||
io.WriteString(stderr, "ffmpeg is not installed\n")
|
||||
return 1
|
||||
}
|
||||
argv := []string{ffmpeg}
|
||||
if nice, err := exec.LookPath("nice"); err == nil {
|
||||
argv = []string{nice, "-n", "10", ffmpeg}
|
||||
}
|
||||
cmd := exec.Command(argv[0], append(argv[1:], args...)...)
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Stderr = stderr
|
||||
cmd.Stdout = io.Discard
|
||||
if err := cmd.Start(); err != nil {
|
||||
io.WriteString(stderr, err.Error()+"\n")
|
||||
return 1
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
go func() {
|
||||
// "stop" and "cont" pause and resume it; the end of input (the daemon letting go)
|
||||
// ends it.
|
||||
sc := bufio.NewScanner(stdin)
|
||||
for sc.Scan() {
|
||||
switch sc.Text() {
|
||||
case "stop":
|
||||
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGSTOP)
|
||||
case "cont":
|
||||
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGCONT)
|
||||
}
|
||||
}
|
||||
cancel()
|
||||
}()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- cmd.Wait() }()
|
||||
var werr error
|
||||
select {
|
||||
case werr = <-done:
|
||||
case <-ctx.Done():
|
||||
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM)
|
||||
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGCONT) // a paused one can't act on TERM
|
||||
select {
|
||||
case werr = <-done:
|
||||
case <-time.After(5 * time.Second):
|
||||
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||
werr = <-done
|
||||
}
|
||||
}
|
||||
var ee *exec.ExitError
|
||||
if errors.As(werr, &ee) {
|
||||
return ee.ExitCode()
|
||||
}
|
||||
if werr != nil {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// clearDir empties a folder without removing it.
|
||||
func clearDir(dir string) error {
|
||||
ents, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, e := range ents {
|
||||
if err := os.RemoveAll(dir + "/" + e.Name()); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"math"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// Track is one stream of a media file, as ffprobe reports it.
|
||||
type Track struct {
|
||||
Index int `json:"index"`
|
||||
Type string `json:"type"` // video | audio
|
||||
Codec string `json:"codec"`
|
||||
Profile string `json:"profile,omitempty"`
|
||||
PixFmt string `json:"pix_fmt,omitempty"`
|
||||
Width int `json:"width,omitempty"`
|
||||
Height int `json:"height,omitempty"`
|
||||
FPS float64 `json:"fps,omitempty"`
|
||||
ColorTransfer string `json:"color_transfer,omitempty"` // smpte2084 / arib-std-b67 mean HDR
|
||||
Channels int `json:"channels,omitempty"`
|
||||
AttachedPic bool `json:"attached_pic,omitempty"`
|
||||
Spherical bool `json:"spherical,omitempty"` // an equirectangular 360 picture
|
||||
Rotation int `json:"rotation,omitempty"` // degrees clockwise to turn it to be shown: 0, 90, 180, 270 (a phone held upright)
|
||||
}
|
||||
|
||||
// Shown is the picture's size as it is shown, turned by its rotation.
|
||||
func (t *Track) Shown() (w, h int) {
|
||||
if t.Rotation == 90 || t.Rotation == 270 {
|
||||
return t.Height, t.Width
|
||||
}
|
||||
return t.Width, t.Height
|
||||
}
|
||||
|
||||
// Info is what a file holds: the picture to take, the sound to go with it, and the whole.
|
||||
type Info struct {
|
||||
Container string `json:"container"` // "matroska,webm", "mov,mp4,m4a,3gp,3g2,mj2"
|
||||
Duration float64 `json:"duration"`
|
||||
BitRate int64 `json:"bit_rate,omitempty"` // bits per second, whole file
|
||||
Size int64 `json:"size,omitempty"`
|
||||
Video *Track `json:"video,omitempty"` // the first real picture stream; cover art is not it
|
||||
Audio *Track `json:"audio,omitempty"` // the first sound stream
|
||||
}
|
||||
|
||||
// PQ reports HDR10 (and Dolby Vision's base layer): a picture that can't be shown on an SDR
|
||||
// screen without tone mapping. HLG, the other kind, can, if a little flat.
|
||||
func (i Info) PQ() bool { return i.Video != nil && i.Video.ColorTransfer == "smpte2084" }
|
||||
|
||||
// HDR reports a picture that needs tone mapping to look right on an SDR screen. (The player
|
||||
// tone-maps only PQ: HLG is made to be shown on an SDR screen as it is.)
|
||||
func (i Info) HDR() bool {
|
||||
return i.Video != nil && (i.Video.ColorTransfer == "smpte2084" || i.Video.ColorTransfer == "arib-std-b67")
|
||||
}
|
||||
|
||||
type probeJSON struct {
|
||||
Streams []struct {
|
||||
Index int `json:"index"`
|
||||
CodecType string `json:"codec_type"`
|
||||
CodecName string `json:"codec_name"`
|
||||
Profile string `json:"profile"`
|
||||
PixFmt string `json:"pix_fmt"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
RFrameRate string `json:"r_frame_rate"`
|
||||
AvgFrameRate string `json:"avg_frame_rate"`
|
||||
ColorTransfer string `json:"color_transfer"`
|
||||
Channels int `json:"channels"`
|
||||
Disposition struct {
|
||||
AttachedPic int `json:"attached_pic"`
|
||||
} `json:"disposition"`
|
||||
SideData []struct {
|
||||
Type string `json:"side_data_type"`
|
||||
Projection string `json:"projection"`
|
||||
Rotation float64 `json:"rotation"`
|
||||
} `json:"side_data_list"`
|
||||
} `json:"streams"`
|
||||
Format struct {
|
||||
FormatName string `json:"format_name"`
|
||||
Duration string `json:"duration"`
|
||||
BitRate string `json:"bit_rate"`
|
||||
Size string `json:"size"`
|
||||
} `json:"format"`
|
||||
}
|
||||
|
||||
// Inspect asks ffprobe what a file is. As root through sudo when asked, for a file only
|
||||
// root may read. An error about the file itself is a DecodeError.
|
||||
func Inspect(ctx context.Context, c Caps, path string, root bool) (Info, error) {
|
||||
if c.FFprobe == "" {
|
||||
return Info{}, errors.New("ffprobe is not installed")
|
||||
}
|
||||
argv := []string{c.FFprobe, "-v", "error", "-show_entries",
|
||||
"stream=index,codec_type,codec_name,profile,pix_fmt,width,height,r_frame_rate,avg_frame_rate,color_transfer,channels,disposition:stream_side_data=side_data_type,projection,rotation:format=format_name,duration,bit_rate,size",
|
||||
"-of", "json", path}
|
||||
if root {
|
||||
argv = priv.Wrap(argv...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return Info{}, ctx.Err()
|
||||
}
|
||||
msg := lastLine(errb.String())
|
||||
if msg == "" {
|
||||
msg = "ffprobe: " + err.Error()
|
||||
}
|
||||
return Info{}, &DecodeError{Msg: msg}
|
||||
}
|
||||
var pj probeJSON
|
||||
if err := json.Unmarshal(out, &pj); err != nil {
|
||||
return Info{}, &DecodeError{Msg: "ffprobe: unreadable answer"}
|
||||
}
|
||||
in := Info{Container: pj.Format.FormatName}
|
||||
in.Duration, _ = strconv.ParseFloat(pj.Format.Duration, 64)
|
||||
in.BitRate, _ = strconv.ParseInt(pj.Format.BitRate, 10, 64)
|
||||
in.Size, _ = strconv.ParseInt(pj.Format.Size, 10, 64)
|
||||
var art *Track
|
||||
for _, s := range pj.Streams {
|
||||
st := &Track{Index: s.Index, Type: s.CodecType, Codec: s.CodecName, Profile: s.Profile, PixFmt: s.PixFmt, Width: s.Width, Height: s.Height,
|
||||
ColorTransfer: s.ColorTransfer, Channels: s.Channels, AttachedPic: s.Disposition.AttachedPic != 0}
|
||||
// A 360 video says so in its spherical tag (ffmpeg reads the old XML one and the sv3d
|
||||
// box); an untagged one is still taken for 360 when it is 2:1 and too big for anything
|
||||
// else.
|
||||
for _, sd := range s.SideData {
|
||||
if sd.Type == "Spherical Mapping" && strings.HasPrefix(sd.Projection, "equirectangular") {
|
||||
st.Spherical = true
|
||||
}
|
||||
if sd.Type == "Display Matrix" {
|
||||
// ffprobe's rotation is counter-clockwise; ffmpeg turns it back the other way
|
||||
st.Rotation = ((-int(math.Round(sd.Rotation/90))*90)%360 + 360) % 360
|
||||
}
|
||||
}
|
||||
st.Spherical = st.Spherical || (s.CodecType == "video" && s.Width == 2*s.Height && s.Width >= 3840)
|
||||
st.FPS = fps(s.AvgFrameRate)
|
||||
if st.FPS == 0 {
|
||||
st.FPS = fps(s.RFrameRate)
|
||||
}
|
||||
switch {
|
||||
case s.CodecType == "video" && st.AttachedPic:
|
||||
if art == nil {
|
||||
art = st
|
||||
}
|
||||
case s.CodecType == "video" && in.Video == nil:
|
||||
in.Video = st
|
||||
case s.CodecType == "audio" && in.Audio == nil:
|
||||
in.Audio = st
|
||||
}
|
||||
}
|
||||
if in.Video == nil {
|
||||
in.Video = art // cover art is the only picture there is
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
||||
// fps parses ffprobe's "60000/1001".
|
||||
func fps(r string) float64 {
|
||||
n, d, ok := strings.Cut(r, "/")
|
||||
if !ok {
|
||||
v, _ := strconv.ParseFloat(r, 64)
|
||||
return v
|
||||
}
|
||||
nf, _ := strconv.ParseFloat(n, 64)
|
||||
df, _ := strconv.ParseFloat(d, 64)
|
||||
if df == 0 {
|
||||
return 0
|
||||
}
|
||||
return nf / df
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/user"
|
||||
"strconv"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// ownerOf is "root:render 0660" for a device node, the thing to say when it cannot be opened.
|
||||
func ownerOf(st os.FileInfo) string {
|
||||
sys, ok := st.Sys().(*syscall.Stat_t)
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
u := strconv.Itoa(int(sys.Uid))
|
||||
if x, err := user.LookupId(u); err == nil {
|
||||
u = x.Username
|
||||
}
|
||||
g := strconv.Itoa(int(sys.Gid))
|
||||
if x, err := user.LookupGroupId(g); err == nil {
|
||||
g = x.Name
|
||||
}
|
||||
return fmt.Sprintf("%s:%s %04o", u, g, st.Mode().Perm())
|
||||
}
|
||||
@@ -0,0 +1,462 @@
|
||||
package media
|
||||
|
||||
// Playing a video in the browser at the quality the connection can take: HLS, made on the fly
|
||||
// by ffmpeg on the device asked for (the Transcode device setting), YouTube's way. The player
|
||||
// is told every quality is all there from the start, in Seg-second pieces, and each piece is
|
||||
// made when it is first asked for.
|
||||
//
|
||||
// One ffmpeg runs per play at a time. It starts at the piece asked for and runs on ahead of the
|
||||
// player, paused once it is far enough ahead; a piece asked for well past where it has got to
|
||||
// (a seek), or at another quality, ends it and starts a new one there. Every run is made the
|
||||
// same way (pieceArgs), so pieces from different runs and qualities follow on from one another.
|
||||
//
|
||||
// The file is a path on this machine or, when the device asked for is on another one, a URL
|
||||
// the hub serves it at. Nothing is kept: a play nobody asks anything of is forgotten, and the
|
||||
// folder is emptied when the daemon starts.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log"
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
runAhead = 10 // pieces a run may make past the last one asked for before it is paused
|
||||
jump = 3 // a piece more than this far past what's made is a seek: start again there
|
||||
idleRun = 45 * time.Second // a run nobody has asked anything of for this long is ended
|
||||
idlePlay = 30 * time.Minute // a play nobody has asked anything of for this long is forgotten
|
||||
budget = 2 << 30 // bytes of pieces one play keeps; those asked for least lately go first
|
||||
// PieceWait is the longest a request waits for its piece: under the hub proxy's 60 s wait
|
||||
// for a node to answer.
|
||||
PieceWait = 45 * time.Second
|
||||
)
|
||||
|
||||
var ErrNotFound = errors.New("no such piece")
|
||||
|
||||
// Plays is every play on this machine.
|
||||
type Plays struct {
|
||||
dir string
|
||||
self string // this binary, for root runs through run-ffmpeg
|
||||
maxRuns int
|
||||
|
||||
mu sync.Mutex
|
||||
plays map[string]*Play
|
||||
stop chan struct{}
|
||||
}
|
||||
|
||||
// NewPlays keeps pieces under <dataDir>/transcode, which it empties first.
|
||||
func NewPlays(dataDir, self string, maxRuns int) *Plays {
|
||||
dir := filepath.Join(dataDir, "transcode")
|
||||
_ = os.MkdirAll(dir, 0o700)
|
||||
_ = clearDir(dir)
|
||||
m := &Plays{dir: dir, self: self, maxRuns: max(maxRuns, 1), plays: map[string]*Play{}, stop: make(chan struct{})}
|
||||
go m.tend()
|
||||
return m
|
||||
}
|
||||
|
||||
// Play is one video opened for watching.
|
||||
type Play struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"` // the file's name, for the record
|
||||
Info Info `json:"info"`
|
||||
Rungs []Rung `json:"rungs"` // best first
|
||||
Device string `json:"device"` // asked for: a render node, "cpu", or "" for the best here
|
||||
Remote bool `json:"remote"` // the file is on another machine, read over the network
|
||||
|
||||
caps Caps
|
||||
in string
|
||||
root bool
|
||||
segs int
|
||||
dir string
|
||||
m *Plays
|
||||
|
||||
mu sync.Mutex
|
||||
seen time.Time
|
||||
run *playRun
|
||||
tried map[int]int // per rung: recipes that failed to make anything, skipped from now on
|
||||
}
|
||||
|
||||
type playRun struct {
|
||||
rung int
|
||||
recipe Recipe
|
||||
start int // the first piece this run makes
|
||||
head int // the last piece finished (start-1 before the first)
|
||||
want int // the last piece asked for
|
||||
asked time.Time
|
||||
proc *ffproc
|
||||
paused bool
|
||||
dead bool // ended, or exited
|
||||
killed bool // ended by us, not by itself
|
||||
exited bool
|
||||
err error
|
||||
tick chan struct{} // closed when head moves or the run ends
|
||||
}
|
||||
|
||||
func newPlayID() string {
|
||||
b := make([]byte, 12)
|
||||
_, _ = rand.Read(b)
|
||||
return hex.EncodeToString(b)
|
||||
}
|
||||
|
||||
// Open starts a play of in (a path, or an http URL for a file on another machine). Nothing
|
||||
// runs until a piece is asked for.
|
||||
func (m *Plays) Open(c Caps, root bool, in, name string, src Info, device string) (*Play, error) {
|
||||
rungs := Ladder(src)
|
||||
if len(rungs) == 0 || src.Duration <= 0 {
|
||||
return nil, errors.New("this video's length or picture size isn't known, so it can't be played")
|
||||
}
|
||||
if len(recipes(c, device, src, rungs[0])) == 0 {
|
||||
return nil, errors.New("nothing on this machine encodes H.264 (no GPU that does, and no libx264)")
|
||||
}
|
||||
segs := max(int(math.Ceil((src.Duration-0.25)/Seg)), 1) // the last piece takes in what's left
|
||||
p := &Play{ID: newPlayID(), Name: name, Info: src, Rungs: rungs, Device: device, Remote: strings.HasPrefix(in, "http"),
|
||||
caps: c, in: in, root: root, segs: segs, m: m, seen: time.Now(), tried: map[int]int{}}
|
||||
p.dir = filepath.Join(m.dir, p.ID)
|
||||
m.mu.Lock()
|
||||
m.plays[p.ID] = p
|
||||
m.mu.Unlock()
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (m *Plays) Find(id string) (*Play, bool) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
p, ok := m.plays[id]
|
||||
return p, ok
|
||||
}
|
||||
|
||||
// Close ends a play and deletes its pieces.
|
||||
func (m *Plays) Close(id string) {
|
||||
m.mu.Lock()
|
||||
p, ok := m.plays[id]
|
||||
delete(m.plays, id)
|
||||
m.mu.Unlock()
|
||||
if ok {
|
||||
p.mu.Lock()
|
||||
p.end()
|
||||
p.mu.Unlock()
|
||||
go os.RemoveAll(p.dir)
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Play) rung(name string) int {
|
||||
for i, r := range p.Rungs {
|
||||
if r.Name == name {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
// Master is the playlist of qualities, lowest first, so a player that starts on the first one
|
||||
// listed starts on the one that surely plays.
|
||||
func (p *Play) Master() []byte {
|
||||
var b bytes.Buffer
|
||||
b.WriteString("#EXTM3U\n#EXT-X-VERSION:3\n#EXT-X-INDEPENDENT-SEGMENTS\n")
|
||||
for i := len(p.Rungs) - 1; i >= 0; i-- {
|
||||
r := p.Rungs[i]
|
||||
fmt.Fprintf(&b, "#EXT-X-STREAM-INF:BANDWIDTH=%d,AVERAGE-BANDWIDTH=%d,RESOLUTION=%dx%d,FRAME-RATE=%.3f,CODECS=\"%s\"\n%s.m3u8\n",
|
||||
r.Peak, r.Bitrate+audioRate, r.Width, r.Height, r.FPS, r.codecs, r.Name)
|
||||
}
|
||||
return b.Bytes()
|
||||
}
|
||||
|
||||
// Media is one quality's playlist: every piece, made or not.
|
||||
func (p *Play) Media(name string) ([]byte, bool) {
|
||||
if p.rung(name) < 0 {
|
||||
return nil, false
|
||||
}
|
||||
var b bytes.Buffer
|
||||
last := p.Info.Duration - float64(p.segs-1)*Seg
|
||||
fmt.Fprintf(&b, "#EXTM3U\n#EXT-X-VERSION:3\n#EXT-X-TARGETDURATION:%d\n#EXT-X-MEDIA-SEQUENCE:0\n#EXT-X-PLAYLIST-TYPE:VOD\n#EXT-X-INDEPENDENT-SEGMENTS\n",
|
||||
int(math.Ceil(math.Max(last, Seg))))
|
||||
for n := 0; n < p.segs; n++ {
|
||||
d := Seg
|
||||
if n == p.segs-1 {
|
||||
d = last
|
||||
}
|
||||
fmt.Fprintf(&b, "#EXTINF:%.6f,\n%s/%d.ts\n", d, name, n)
|
||||
}
|
||||
b.WriteString("#EXT-X-ENDLIST\n")
|
||||
return b.Bytes(), true
|
||||
}
|
||||
|
||||
func (p *Play) piece(rung, n int) string {
|
||||
return filepath.Join(p.dir, p.Rungs[rung].Name, strconv.Itoa(n)+".ts")
|
||||
}
|
||||
|
||||
// Segment answers the file of piece n at quality name, making it first if need be: it waits
|
||||
// for the run already making it, or starts one there.
|
||||
func (p *Play) Segment(done <-chan struct{}, name string, n int) (string, error) {
|
||||
ri := p.rung(name)
|
||||
if ri < 0 || n < 0 || n >= p.segs {
|
||||
return "", ErrNotFound
|
||||
}
|
||||
path := p.piece(ri, n)
|
||||
deadline := time.After(PieceWait)
|
||||
for {
|
||||
p.mu.Lock()
|
||||
now := time.Now()
|
||||
p.seen = now
|
||||
r := p.run
|
||||
if r != nil && r.rung == ri && !r.dead {
|
||||
r.want, r.asked = n, now
|
||||
}
|
||||
if exists(path) {
|
||||
p.mu.Unlock()
|
||||
_ = os.Chtimes(path, now, now) // the budget keeps what was asked for last
|
||||
return path, nil
|
||||
}
|
||||
if r != nil && r.exited && !r.killed && r.rung == ri && r.start == n {
|
||||
// This run started at this very piece and ended without it: that way of making
|
||||
// it doesn't work here. The next one is tried, if there is one.
|
||||
p.tried[ri]++
|
||||
log.Printf("files: play %s: %s made nothing at %s piece %d: %v: %s", p.Name, r.recipe.Engine, name, n, r.err, r.proc.lastWords())
|
||||
if p.tried[ri] >= len(recipes(p.caps, p.Device, p.Info, p.Rungs[ri])) {
|
||||
err := fmt.Errorf("couldn't make %s: %s", p.Rungs[ri].Label, r.proc.lastWords())
|
||||
p.mu.Unlock()
|
||||
return "", err
|
||||
}
|
||||
p.run = nil
|
||||
r = nil
|
||||
}
|
||||
if r == nil || r.dead || r.rung != ri || n < r.start || n <= r.head || n > r.head+jump {
|
||||
p.mu.Unlock()
|
||||
p.m.makeRoom(p)
|
||||
p.mu.Lock()
|
||||
if p.run == r { // nobody started one meanwhile
|
||||
if err := p.start(ri, n); err != nil {
|
||||
p.mu.Unlock()
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
p.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
tick := r.tick
|
||||
p.mu.Unlock()
|
||||
select {
|
||||
case <-tick:
|
||||
case <-time.After(200 * time.Millisecond):
|
||||
case <-done:
|
||||
return "", errors.New("gone")
|
||||
case <-deadline:
|
||||
return "", fmt.Errorf("%s piece %d took longer than %s", name, n, PieceWait)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// end stops this play's run. p.mu is held.
|
||||
func (p *Play) end() {
|
||||
if r := p.run; r != nil && !r.dead {
|
||||
r.dead, r.killed = true, true
|
||||
r.proc.kill()
|
||||
}
|
||||
}
|
||||
|
||||
// makeRoom ends the run asked of least lately, if every slot is taken by another play. It
|
||||
// never holds two plays' locks at once.
|
||||
func (m *Plays) makeRoom(except *Play) {
|
||||
m.mu.Lock()
|
||||
plays := make([]*Play, 0, len(m.plays))
|
||||
for _, p := range m.plays {
|
||||
if p != except {
|
||||
plays = append(plays, p)
|
||||
}
|
||||
}
|
||||
m.mu.Unlock()
|
||||
var oldest *Play
|
||||
var when time.Time
|
||||
running := 0
|
||||
for _, p := range plays {
|
||||
p.mu.Lock()
|
||||
if r := p.run; r != nil && !r.dead {
|
||||
running++
|
||||
if oldest == nil || r.asked.Before(when) {
|
||||
oldest, when = p, r.asked
|
||||
}
|
||||
}
|
||||
p.mu.Unlock()
|
||||
}
|
||||
if running >= m.maxRuns && oldest != nil {
|
||||
oldest.mu.Lock()
|
||||
oldest.end()
|
||||
oldest.mu.Unlock()
|
||||
log.Printf("files: %d plays encoding; ended %s's, asked of least lately", running, oldest.Name)
|
||||
}
|
||||
}
|
||||
|
||||
// start begins a run making quality ri from piece n on, ending the one before. p.mu is held.
|
||||
func (p *Play) start(ri, n int) error {
|
||||
p.end()
|
||||
rs := recipes(p.caps, p.Device, p.Info, p.Rungs[ri])
|
||||
if p.tried[ri] >= len(rs) {
|
||||
return errors.New("nothing here can make this quality")
|
||||
}
|
||||
rc := rs[p.tried[ri]]
|
||||
out := filepath.Join(p.dir, p.Rungs[ri].Name)
|
||||
if err := os.MkdirAll(out, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
proc, err := startFF(p.caps, p.m.self, p.root, pieceArgs(out, p.in, p.Info, rc, p.Rungs[ri], n))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
r := &playRun{rung: ri, recipe: rc, start: n, head: n - 1, want: n, asked: time.Now(), proc: proc, tick: make(chan struct{})}
|
||||
p.run = r
|
||||
go func() {
|
||||
<-proc.done
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
r.dead, r.exited, r.err = true, true, proc.err
|
||||
p.scan(r)
|
||||
close(r.tick)
|
||||
r.tick = make(chan struct{})
|
||||
}()
|
||||
return nil
|
||||
}
|
||||
|
||||
// scan moves a run's head on past the pieces it has finished. p.mu is held.
|
||||
func (p *Play) scan(r *playRun) bool {
|
||||
moved := false
|
||||
for r.head+1 < p.segs && exists(p.piece(r.rung, r.head+1)) {
|
||||
r.head++
|
||||
moved = true
|
||||
}
|
||||
return moved
|
||||
}
|
||||
|
||||
// PlayStatus is what a play is doing, for the player's menu.
|
||||
type PlayStatus struct {
|
||||
Engine string `json:"engine,omitempty"` // what the run going now runs on
|
||||
Quality string `json:"quality,omitempty"`
|
||||
From int `json:"from"` // seconds into the film the run started at
|
||||
Ahead int `json:"ahead"`
|
||||
Paused bool `json:"paused,omitempty"`
|
||||
}
|
||||
|
||||
func (p *Play) Status() PlayStatus {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
r := p.run
|
||||
if r == nil || r.dead {
|
||||
return PlayStatus{}
|
||||
}
|
||||
return PlayStatus{Engine: r.recipe.Engine, Quality: p.Rungs[r.rung].Label, From: int(float64(r.start) * Seg), Ahead: r.head - r.want, Paused: r.paused}
|
||||
}
|
||||
|
||||
// Engine is what this play would run on first, for the player to say before anything runs.
|
||||
func (p *Play) Engine() string {
|
||||
rs := recipes(p.caps, p.Device, p.Info, p.Rungs[0])
|
||||
if len(rs) == 0 {
|
||||
return ""
|
||||
}
|
||||
return rs[0].Engine
|
||||
}
|
||||
|
||||
// tend, ten times a second: notices finished pieces, pauses a run that is far enough ahead and
|
||||
// wakes it when the player catches up, ends one nobody is asking anything of; and every ten
|
||||
// seconds keeps each play inside its budget and forgets plays nobody watches.
|
||||
func (m *Plays) tend() {
|
||||
t := time.NewTicker(100 * time.Millisecond)
|
||||
defer t.Stop()
|
||||
for i := 0; ; i++ {
|
||||
select {
|
||||
case <-m.stop:
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
m.mu.Lock()
|
||||
plays := make([]*Play, 0, len(m.plays))
|
||||
for _, p := range m.plays {
|
||||
plays = append(plays, p)
|
||||
}
|
||||
m.mu.Unlock()
|
||||
now := time.Now()
|
||||
for _, p := range plays {
|
||||
p.mu.Lock()
|
||||
if r := p.run; r != nil && !r.dead {
|
||||
if p.scan(r) {
|
||||
close(r.tick)
|
||||
r.tick = make(chan struct{})
|
||||
}
|
||||
ahead := r.head - r.want
|
||||
switch {
|
||||
case now.Sub(r.asked) > idleRun:
|
||||
p.end()
|
||||
case !r.paused && ahead >= runAhead:
|
||||
if r.proc.pause() == nil {
|
||||
r.paused = true
|
||||
}
|
||||
case r.paused && ahead < runAhead-3:
|
||||
if r.proc.resume() == nil {
|
||||
r.paused = false
|
||||
}
|
||||
}
|
||||
}
|
||||
idle := now.Sub(p.seen) > idlePlay
|
||||
p.mu.Unlock()
|
||||
if idle {
|
||||
m.Close(p.ID)
|
||||
} else if i%100 == 0 {
|
||||
p.trim()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// trim deletes the pieces asked for least lately until the play is inside its budget, never
|
||||
// the ones its run has made ahead of the player.
|
||||
func (p *Play) trim() {
|
||||
type piece struct {
|
||||
path string
|
||||
size int64
|
||||
at time.Time
|
||||
}
|
||||
var all []piece
|
||||
var total int64
|
||||
_ = filepath.WalkDir(p.dir, func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() || !strings.HasSuffix(path, ".ts") {
|
||||
return nil
|
||||
}
|
||||
if fi, err := d.Info(); err == nil {
|
||||
all = append(all, piece{path, fi.Size(), fi.ModTime()})
|
||||
total += fi.Size()
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if total <= budget {
|
||||
return
|
||||
}
|
||||
p.mu.Lock()
|
||||
keep := map[string]bool{}
|
||||
if r := p.run; r != nil && !r.dead {
|
||||
for n := r.want - 1; n <= r.head+1; n++ {
|
||||
if n >= 0 && n < p.segs {
|
||||
keep[p.piece(r.rung, n)] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
p.mu.Unlock()
|
||||
sort.Slice(all, func(a, b int) bool { return all[a].at.Before(all[b].at) })
|
||||
for _, x := range all {
|
||||
if total <= budget*3/4 {
|
||||
break
|
||||
}
|
||||
if !keep[x.path] && os.Remove(x.path) == nil {
|
||||
total -= x.size
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"math"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// inspectPiece answers when a piece's first video packet is shown, whether it is a keyframe,
|
||||
// how many keyframes the piece has, and when its last packet is.
|
||||
func inspectPiece(t *testing.T, path string) (first float64, key bool, keys int, last float64) {
|
||||
t.Helper()
|
||||
b, err := exec.Command("ffprobe", "-v", "error", "-show_entries", "packet=pts_time,flags,codec_type", "-of", "json", path).Output()
|
||||
if err != nil {
|
||||
t.Fatalf("ffprobe %s: %v", path, err)
|
||||
}
|
||||
var p struct {
|
||||
Packets []struct {
|
||||
PTS string `json:"pts_time"`
|
||||
Flags string `json:"flags"`
|
||||
Type string `json:"codec_type"`
|
||||
} `json:"packets"`
|
||||
}
|
||||
_ = json.Unmarshal(b, &p)
|
||||
first = -1
|
||||
for _, k := range p.Packets {
|
||||
if k.Type != "video" {
|
||||
continue
|
||||
}
|
||||
x, _ := strconv.ParseFloat(k.PTS, 64)
|
||||
if first < 0 {
|
||||
first, key = x, strings.HasPrefix(k.Flags, "K")
|
||||
}
|
||||
first = math.Min(first, x)
|
||||
last = math.Max(last, x)
|
||||
if strings.HasPrefix(k.Flags, "K") {
|
||||
keys++
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// TestPieces makes pieces of a made-up video (or SRC, on DEVICE, to try the real thing on a
|
||||
// machine with a GPU) in order and after a seek, at two qualities, and checks they line up:
|
||||
// each starts on a keyframe at its own place in the film, with no other keyframe inside it.
|
||||
func TestPieces(t *testing.T) {
|
||||
if _, err := exec.LookPath("ffmpeg"); err != nil {
|
||||
t.Skip("no ffmpeg")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
in := os.Getenv("SRC")
|
||||
if in == "" {
|
||||
in = filepath.Join(dir, "in.mkv")
|
||||
if out, err := exec.Command("ffmpeg", "-v", "error", "-f", "lavfi", "-i", "testsrc2=size=1920x1080:rate=50:duration=40",
|
||||
"-f", "lavfi", "-i", "sine=frequency=440:duration=40", "-c:v", "libx264", "-preset", "ultrafast", "-g", "500",
|
||||
"-c:a", "libopus", "-shortest", in).CombinedOutput(); err != nil {
|
||||
t.Fatalf("making the test video: %v: %s", err, out)
|
||||
}
|
||||
}
|
||||
caps := Probe(context.Background())
|
||||
if caps.Error != "" {
|
||||
t.Skip(caps.Error)
|
||||
}
|
||||
info, err := Inspect(context.Background(), caps, in, false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m := NewPlays(dir, "", 2)
|
||||
device := os.Getenv("DEVICE")
|
||||
if device == "" {
|
||||
device = CPU
|
||||
}
|
||||
p, err := m.Open(caps, false, in, "in.mkv", info, device)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer m.Close(p.ID)
|
||||
if !strings.Contains(string(p.Master()), "RESOLUTION=") {
|
||||
t.Fatal(string(p.Master()))
|
||||
}
|
||||
low := p.Rungs[len(p.Rungs)-1].Name
|
||||
var offset float64
|
||||
far := min(p.segs-2, 60)
|
||||
for i, n := range []int{0, 1, 2, far, far + 1, 3} {
|
||||
t0 := time.Now()
|
||||
path, err := p.Segment(nil, low, n)
|
||||
if err != nil {
|
||||
t.Fatalf("piece %d: %v", n, err)
|
||||
}
|
||||
first, key, keys, last := inspectPiece(t, path)
|
||||
if i == 0 {
|
||||
offset = first
|
||||
}
|
||||
at := first - offset
|
||||
t.Logf("piece %d: %d ms on %s, starts %.3f, %d keyframe(s), %.2f s long", n, time.Since(t0).Milliseconds(), p.Status().Engine, at, keys, last-first)
|
||||
if !key || keys != 1 {
|
||||
t.Errorf("piece %d: first packet keyframe %v, %d keyframes", n, key, keys)
|
||||
}
|
||||
if math.Abs(at-float64(n)*Seg) > 0.05 {
|
||||
t.Errorf("piece %d starts at %.3f, want %.3f", n, at, float64(n)*Seg)
|
||||
}
|
||||
if n < p.segs-1 && math.Abs(last-first-Seg) > 0.1 {
|
||||
t.Errorf("piece %d is %.3f s long", n, last-first)
|
||||
}
|
||||
}
|
||||
// another quality, at a piece already made at the first: its own run, lined up too
|
||||
if len(p.Rungs) > 1 {
|
||||
path, err := p.Segment(nil, p.Rungs[0].Name, 2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if first, key, _, _ := inspectPiece(t, path); !key || math.Abs(first-offset-2*Seg) > 0.05 {
|
||||
t.Errorf("%s piece 2 starts %.3f (keyframe %v)", p.Rungs[0].Name, first-offset, key)
|
||||
}
|
||||
}
|
||||
if _, err := p.Segment(nil, low, p.segs); err != ErrNotFound {
|
||||
t.Errorf("past the end: %v", err)
|
||||
}
|
||||
if _, err := p.Segment(nil, "999", 0); err != ErrNotFound {
|
||||
t.Errorf("no such quality: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,679 @@
|
||||
// Package thumbs makes the Files tab's previews: one small JPEG for an image or a video,
|
||||
// by ffmpeg, decoded on a GPU where the machine has one it can use and on the CPU
|
||||
// otherwise.
|
||||
//
|
||||
// Nothing about the machine is assumed. A probe looks ffmpeg and ffprobe up, opens every
|
||||
// /dev/dri/renderD* and exercises it (a VAAPI upload and download of a synthetic frame),
|
||||
// so a node the daemon's user cannot open, or a driver that does not speak VAAPI, is
|
||||
// reported once rather than tried per file. vainfo, where installed, says which codecs
|
||||
// each node decodes, and a video goes to the first node that lists its codec; without
|
||||
// vainfo the first working node is tried and ffmpeg falls back to software on its own when
|
||||
// the stream turns out unsupported. A hardware run that fails outright is retried on the
|
||||
// CPU before the file is called undecodable. Images are always decoded on the CPU: one
|
||||
// frame of a JPEG at a reduced DCT size costs less than the trip through a GPU.
|
||||
package media
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/metrics"
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// GPU is one hardware decoder as the probe found it.
|
||||
type GPU struct {
|
||||
API string `json:"api"` // vaapi | cuda
|
||||
Device string `json:"device"` // /dev/dri/renderD128; "0" for cuda (its device index)
|
||||
Name string `json:"name,omitempty"` // "Intel Arc A380"
|
||||
Codecs []string `json:"codecs,omitempty"` // ffmpeg codec names it decodes; nil when nothing could say
|
||||
Encodes []string `json:"encodes,omitempty"` // ffmpeg codec names it encodes, from vainfo
|
||||
Integrated bool `json:"integrated,omitempty"` // on the CPU die; a card on the bus is preferred for encoding
|
||||
Reason string `json:"reason,omitempty"` // Unusable only: why
|
||||
Permission bool `json:"permission,omitempty"` // Unusable only: the node exists but this user may not open it
|
||||
}
|
||||
|
||||
// Caps is what the machine can make previews with, as probed.
|
||||
type Caps struct {
|
||||
FFmpeg string `json:"ffmpeg,omitempty"`
|
||||
FFprobe string `json:"ffprobe,omitempty"`
|
||||
Nice string `json:"nice,omitempty"`
|
||||
Version string `json:"version,omitempty"`
|
||||
GPUs []GPU `json:"gpus"` // usable, in node order
|
||||
Encoders []string `json:"encoders,omitempty"` // the encoders ffmpeg itself was built with that matter here
|
||||
Unusable []GPU `json:"unusable,omitempty"` // render nodes that exist but cannot be used, with the reason
|
||||
Error string `json:"error,omitempty"` // why nothing can be made at all
|
||||
ProbedAt time.Time `json:"probed_at"`
|
||||
}
|
||||
|
||||
// Opts shapes one preview.
|
||||
type Opts struct {
|
||||
Size int // the long edge in pixels; 256 when zero
|
||||
Timeout time.Duration // for the whole job, ffprobe and every ffmpeg attempt; 30 s when zero
|
||||
Kind string // image | video | audio; from the name when empty (a URL has none)
|
||||
}
|
||||
|
||||
// Only is the caps with one device left to decode on: a render node, CPU for none, or ""
|
||||
// for all of them — the Transcode device setting, which previews follow too. A device this
|
||||
// machine doesn't have leaves the CPU.
|
||||
func (c Caps) Only(device string) Caps {
|
||||
if device == "" {
|
||||
return c
|
||||
}
|
||||
out := c
|
||||
out.GPUs = nil
|
||||
for _, g := range c.GPUs {
|
||||
if g.Device == device {
|
||||
out.GPUs = append(out.GPUs, g)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (o Opts) defaults() Opts {
|
||||
if o.Size <= 0 {
|
||||
o.Size = 256
|
||||
}
|
||||
if o.Timeout <= 0 {
|
||||
o.Timeout = 30 * time.Second
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// Result is a finished preview.
|
||||
type Result struct {
|
||||
JPEG []byte `json:"-"`
|
||||
Engine string `json:"engine"` // "cpu", or "vaapi:/dev/dri/renderD129"
|
||||
Codec string `json:"codec"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
// Failure is why the file can't be previewed, when another machine tried (no JPEG then).
|
||||
Failure string `json:"-"`
|
||||
}
|
||||
|
||||
// ErrUnsupported: not an image or a video by its name.
|
||||
var ErrUnsupported = errors.New("not an image or video")
|
||||
|
||||
// A DecodeError is ffmpeg's verdict on the file itself. It will not change until the
|
||||
// file does, so the caller may remember it.
|
||||
type DecodeError struct{ Msg string }
|
||||
|
||||
func (e *DecodeError) Error() string { return e.Msg }
|
||||
|
||||
// ---- what gets a preview ----
|
||||
|
||||
var imageExt = set("png jpg jpeg jpe jfif gif webp bmp tif tiff avif heic heif hif jxl svg ico psd tga thm")
|
||||
var videoExt = set("mp4 m4v mkv webm mov avi wmv flv mpg mpeg mpe m2v ts m2ts mts 3gp 3g2 ogv ogm mxf vob asf rm rmvb divx f4v mjpeg mjpg 360 osv insv lrf")
|
||||
var audioExt = set("mp3 flac m4a m4b aac ogg oga opus wav wma aiff aif alac ape wv mka")
|
||||
|
||||
func set(s string) map[string]bool {
|
||||
m := map[string]bool{}
|
||||
for _, w := range strings.Fields(s) {
|
||||
m[w] = true
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Kind is "image", "video", "audio" or "" for a file name. An audio file's preview is its
|
||||
// cover art, when it carries any.
|
||||
func Kind(name string) string {
|
||||
ext := strings.ToLower(strings.TrimPrefix(filepath.Ext(name), "."))
|
||||
switch {
|
||||
case imageExt[ext]:
|
||||
return "image"
|
||||
case videoExt[ext]:
|
||||
return "video"
|
||||
case audioExt[ext]:
|
||||
return "audio"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- the probe ----
|
||||
|
||||
// Probe finds out what this process can make previews with. It runs ffmpeg against every
|
||||
// render node, so it takes a few hundred milliseconds; callers keep the answer.
|
||||
func Probe(ctx context.Context) Caps {
|
||||
c := Caps{ProbedAt: time.Now(), GPUs: []GPU{}}
|
||||
c.FFmpeg, _ = exec.LookPath("ffmpeg")
|
||||
c.FFprobe, _ = exec.LookPath("ffprobe")
|
||||
c.Nice, _ = exec.LookPath("nice")
|
||||
switch {
|
||||
case c.FFmpeg == "" && c.FFprobe == "":
|
||||
c.Error = "ffmpeg is not installed"
|
||||
case c.FFmpeg == "":
|
||||
c.Error = "ffmpeg is not installed (ffprobe is)"
|
||||
case c.FFprobe == "":
|
||||
c.Error = "ffprobe is not installed (ffmpeg is)"
|
||||
}
|
||||
if c.Error != "" {
|
||||
return c
|
||||
}
|
||||
c.Version = ffmpegVersion(ctx, c.FFmpeg)
|
||||
c.Encoders = encoders(ctx, c.FFmpeg)
|
||||
accels := hwaccels(ctx, c.FFmpeg)
|
||||
if accels["vaapi"] {
|
||||
nodes, _ := filepath.Glob("/dev/dri/renderD*")
|
||||
sort.Strings(nodes)
|
||||
vainfo, _ := exec.LookPath("vainfo")
|
||||
for _, dev := range nodes {
|
||||
g := GPU{API: "vaapi", Device: dev, Name: metrics.RenderNodeName(dev), Integrated: metrics.RenderNodeIntegrated(dev)}
|
||||
if err := openable(dev); err != nil {
|
||||
g.Reason, g.Permission = err.Error(), errors.Is(err, os.ErrPermission)
|
||||
c.Unusable = append(c.Unusable, g)
|
||||
continue
|
||||
}
|
||||
if msg := hwWorks(ctx, c.FFmpeg, "vaapi=va:"+dev); msg != "" {
|
||||
g.Reason = msg
|
||||
c.Unusable = append(c.Unusable, g)
|
||||
continue
|
||||
}
|
||||
if vainfo != "" {
|
||||
g.Codecs, g.Encodes = vainfoCodecs(ctx, vainfo, dev)
|
||||
}
|
||||
c.GPUs = append(c.GPUs, g)
|
||||
}
|
||||
}
|
||||
if accels["cuda"] && exists("/dev/nvidiactl") {
|
||||
g := GPU{API: "cuda", Device: "0", Name: "NVIDIA GPU"}
|
||||
if msg := hwWorks(ctx, c.FFmpeg, "cuda=cu:0"); msg != "" {
|
||||
g.Reason = msg
|
||||
c.Unusable = append(c.Unusable, g)
|
||||
} else {
|
||||
c.GPUs = append(c.GPUs, g)
|
||||
}
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// openable: the node can be opened read-write, which is what a VAAPI driver needs. The
|
||||
// usual reason it can't be is the owning group: a render node is root:render 0660 on
|
||||
// most distributions, and a daemon running as an ordinary user is not in render.
|
||||
func openable(dev string) error {
|
||||
f, err := os.OpenFile(dev, os.O_RDWR, 0)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrPermission) {
|
||||
owner := ""
|
||||
if st, serr := os.Stat(dev); serr == nil {
|
||||
owner = ownerOf(st)
|
||||
}
|
||||
if owner != "" {
|
||||
return fmt.Errorf("%w: %s is %s", os.ErrPermission, dev, owner)
|
||||
}
|
||||
return fmt.Errorf("%w: cannot open %s", os.ErrPermission, dev)
|
||||
}
|
||||
return err
|
||||
}
|
||||
f.Close()
|
||||
return nil
|
||||
}
|
||||
|
||||
// hwWorks initialises a hardware device in ffmpeg and moves one synthetic frame through
|
||||
// it. "" when it works, else the last thing ffmpeg said.
|
||||
func hwWorks(ctx context.Context, ffmpeg, device string) string {
|
||||
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
name, rest, _ := strings.Cut(device, "=") // "vaapi", "va:/dev/…"
|
||||
label, _, _ := strings.Cut(rest, ":") // "va"
|
||||
cmd := exec.CommandContext(ctx, ffmpeg, "-nostdin", "-hide_banner", "-v", "error", "-init_hw_device", device, "-filter_hw_device", label,
|
||||
"-f", "lavfi", "-i", "color=c=black:s=64x64:r=1:d=1", "-vf", "format=nv12,hwupload,hwdownload,format=nv12", "-frames:v", "1", "-f", "null", "-")
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := lastLine(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return name + ": " + msg
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func ffmpegVersion(ctx context.Context, ffmpeg string) string {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
out, _ := exec.CommandContext(ctx, ffmpeg, "-version").Output()
|
||||
f := strings.Fields(string(out)) // "ffmpeg version 7.1.5-0+deb13u1 Copyright …"
|
||||
if len(f) >= 3 && f[1] == "version" {
|
||||
v := f[2]
|
||||
if i := strings.IndexAny(v, "-+~"); i > 0 {
|
||||
v = v[:i]
|
||||
}
|
||||
return v
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func hwaccels(ctx context.Context, ffmpeg string) map[string]bool {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
out, _ := exec.CommandContext(ctx, ffmpeg, "-hide_banner", "-hwaccels").Output()
|
||||
m := map[string]bool{}
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
l = strings.TrimSpace(l)
|
||||
if l != "" && !strings.HasPrefix(l, "Hardware acceleration methods") {
|
||||
m[l] = true
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
var vaProfileRe = regexp.MustCompile(`VAProfile([A-Za-z0-9]+?)(?:Simple|Main|High|Profile|Baseline|Constrained|Scc|Advanced|Version|Stereo|Multiview)\w*\s*:\s*VAEntrypoint(VLD|EncSlice|EncSliceLP)\b`)
|
||||
|
||||
// wantedEncoders are the encoders this package may ask ffmpeg for.
|
||||
var wantedEncoders = set("libx264 libx265 libsvtav1 libaom-av1 libopenh264 h264_vaapi hevc_vaapi av1_vaapi aac")
|
||||
|
||||
// encoders lists the wanted encoders this ffmpeg was built with.
|
||||
func encoders(ctx context.Context, ffmpeg string) []string {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
out, _ := exec.CommandContext(ctx, ffmpeg, "-hide_banner", "-encoders").Output()
|
||||
var found []string
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
f := strings.Fields(l)
|
||||
if len(f) >= 2 && wantedEncoders[f[1]] {
|
||||
found = append(found, f[1])
|
||||
}
|
||||
}
|
||||
sort.Strings(found)
|
||||
return found
|
||||
}
|
||||
|
||||
// vaCodec maps a VAProfile family to the ffmpeg codec it decodes.
|
||||
var vaCodec = map[string]string{"H264": "h264", "HEVC": "hevc", "VP9": "vp9", "AV1": "av1", "MPEG2": "mpeg2video", "VC1": "vc1", "VP8": "vp8", "JPEG": "mjpeg", "MPEG4": "mpeg4", "H263": "h263"}
|
||||
|
||||
func vainfoCodecs(ctx context.Context, vainfo, dev string) (decodes, encodes []string) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
out, _ := exec.CommandContext(ctx, vainfo, "--display", "drm", "--device", dev).Output()
|
||||
return parseVainfo(string(out))
|
||||
}
|
||||
|
||||
// parseVainfo names the codecs the decode (VLD) and encode (EncSlice) profiles cover,
|
||||
// the way ffmpeg names them.
|
||||
func parseVainfo(text string) (decodes, encodes []string) {
|
||||
dec, enc := map[string]bool{}, map[string]bool{}
|
||||
for _, m := range vaProfileRe.FindAllStringSubmatch(text, -1) {
|
||||
c, ok := vaCodec[m[1]]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if m[2] == "VLD" {
|
||||
dec[c] = true
|
||||
} else {
|
||||
enc[c] = true
|
||||
}
|
||||
}
|
||||
return keys(dec), keys(enc)
|
||||
}
|
||||
|
||||
func keys(m map[string]bool) []string {
|
||||
if len(m) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(m))
|
||||
for c := range m {
|
||||
out = append(out, c)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// pick is the GPU for a codec: the first that lists it, else the first that could not
|
||||
// say (ffmpeg will fall back to software by itself if that guess is wrong), else none.
|
||||
func (c Caps) pick(codec string) *GPU {
|
||||
var unknown *GPU
|
||||
for i := range c.GPUs {
|
||||
g := &c.GPUs[i]
|
||||
if g.Codecs == nil {
|
||||
if unknown == nil {
|
||||
unknown = g
|
||||
}
|
||||
continue
|
||||
}
|
||||
for _, x := range g.Codecs {
|
||||
if x == codec {
|
||||
return g
|
||||
}
|
||||
}
|
||||
}
|
||||
return unknown
|
||||
}
|
||||
|
||||
// ---- making one ----
|
||||
|
||||
type fileInfo struct {
|
||||
index string // the stream to take
|
||||
codec string
|
||||
w, h int
|
||||
duration float64
|
||||
}
|
||||
|
||||
type attempt struct {
|
||||
gpu *GPU
|
||||
lowres int
|
||||
seek float64
|
||||
}
|
||||
|
||||
func (a attempt) engine() string {
|
||||
if a.gpu == nil {
|
||||
return "cpu"
|
||||
}
|
||||
return a.gpu.API + ":" + a.gpu.Device
|
||||
}
|
||||
|
||||
// Make produces the preview of one file.
|
||||
func Make(ctx context.Context, c Caps, path string, o Opts) (Result, error) {
|
||||
kind := o.Kind
|
||||
if kind == "" {
|
||||
kind = Kind(path)
|
||||
}
|
||||
if kind == "" {
|
||||
return Result{}, ErrUnsupported
|
||||
}
|
||||
if c.Error != "" {
|
||||
return Result{}, errors.New(c.Error)
|
||||
}
|
||||
o = o.defaults()
|
||||
ctx, cancel := context.WithTimeout(ctx, o.Timeout)
|
||||
defer cancel()
|
||||
fi, err := probeFile(ctx, c, path)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
var last string
|
||||
for _, eng := range engines(c, kind, fi) {
|
||||
for _, a := range attempts(eng, kind, fi, o.Size) {
|
||||
data, msg, err := run(ctx, c, a, path, fi, o.Size)
|
||||
if err == nil && len(data) > 0 {
|
||||
return Result{JPEG: data, Engine: a.engine(), Codec: fi.codec, Width: fi.w, Height: fi.h}, nil
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return Result{}, ctx.Err() // the clock ran out: not the file's fault
|
||||
}
|
||||
if err != nil { // the decoder refused: try the next engine, not another seek
|
||||
last = msg
|
||||
break
|
||||
}
|
||||
last = "no frame at " + strconv.FormatFloat(a.seek, 'f', 1, 64) + " s"
|
||||
}
|
||||
}
|
||||
if last == "" {
|
||||
last = "could not decode"
|
||||
}
|
||||
return Result{}, &DecodeError{Msg: last}
|
||||
}
|
||||
|
||||
// engines is the order to try: the fitting GPU first for a video, then the CPU.
|
||||
func engines(c Caps, kind string, fi fileInfo) []*GPU {
|
||||
if kind == "video" {
|
||||
if g := c.pick(fi.codec); g != nil {
|
||||
return []*GPU{g, nil}
|
||||
}
|
||||
}
|
||||
return []*GPU{nil}
|
||||
}
|
||||
|
||||
// attempts for one engine: a seek into a video, then from its start if that found no
|
||||
// frame (a wrong duration in the header); a reduced-size decode of a JPEG, then full size.
|
||||
func attempts(gpu *GPU, kind string, fi fileInfo, size int) []attempt {
|
||||
if kind == "video" {
|
||||
s := seekFor(fi.duration)
|
||||
if s > 0 {
|
||||
return []attempt{{gpu: gpu, seek: s}, {gpu: gpu}}
|
||||
}
|
||||
return []attempt{{gpu: gpu}}
|
||||
}
|
||||
if fi.codec == "mjpeg" && gpu == nil {
|
||||
if l := lowresFor(fi.w, fi.h, size); l > 0 {
|
||||
return []attempt{{lowres: l}, {}}
|
||||
}
|
||||
}
|
||||
return []attempt{{gpu: gpu}}
|
||||
}
|
||||
|
||||
// seekFor: a tenth of the way in, never more than 30 s, so a title card or a black lead-in
|
||||
// is skipped and a long file still seeks to a keyframe near its start.
|
||||
func seekFor(duration float64) float64 {
|
||||
if duration <= 0 {
|
||||
return 0
|
||||
}
|
||||
s := duration / 10
|
||||
if s > 30 {
|
||||
s = 30
|
||||
}
|
||||
if s < 0.5 {
|
||||
return 0
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// lowresFor: how many halvings a JPEG decoder may take while the picture stays at least
|
||||
// twice the preview size. libjpeg does this in the DCT, so a 24 MP photo decodes at 1.5 MP.
|
||||
func lowresFor(w, h, size int) int {
|
||||
m := w
|
||||
if h < m {
|
||||
m = h
|
||||
}
|
||||
l := 0
|
||||
for l < 3 && m>>(l+1) >= size*2 {
|
||||
l++
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
type probeOut struct {
|
||||
Streams []struct {
|
||||
Index int `json:"index"`
|
||||
Codec string `json:"codec_name"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
Disposition struct {
|
||||
AttachedPic int `json:"attached_pic"`
|
||||
} `json:"disposition"`
|
||||
} `json:"streams"`
|
||||
Format struct {
|
||||
Duration string `json:"duration"`
|
||||
} `json:"format"`
|
||||
}
|
||||
|
||||
// probeFile asks ffprobe which stream to take and what is in it.
|
||||
func probeFile(ctx context.Context, c Caps, path string) (fileInfo, error) {
|
||||
cmd := exec.CommandContext(ctx, c.FFprobe, "-v", "error", "-select_streams", "v", "-show_entries", "stream=index,codec_name,width,height,disposition:format=duration", "-of", "json", path)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return fileInfo{}, ctx.Err()
|
||||
}
|
||||
msg := lastLine(errb.String())
|
||||
if msg == "" {
|
||||
msg = "ffprobe: " + err.Error()
|
||||
}
|
||||
return fileInfo{}, &DecodeError{Msg: msg}
|
||||
}
|
||||
var po probeOut
|
||||
if err := json.Unmarshal(out, &po); err != nil {
|
||||
return fileInfo{}, &DecodeError{Msg: "ffprobe: unreadable answer"}
|
||||
}
|
||||
if len(po.Streams) == 0 {
|
||||
return fileInfo{}, &DecodeError{Msg: "no video stream"}
|
||||
}
|
||||
s := po.Streams[0]
|
||||
for _, x := range po.Streams { // cover art rides along as a video stream; the picture is not the film
|
||||
if x.Disposition.AttachedPic == 0 {
|
||||
s = x
|
||||
break
|
||||
}
|
||||
}
|
||||
fi := fileInfo{index: strconv.Itoa(s.Index), codec: s.Codec, w: s.Width, h: s.Height}
|
||||
fi.duration, _ = strconv.ParseFloat(po.Format.Duration, 64)
|
||||
return fi, nil
|
||||
}
|
||||
|
||||
// run is one ffmpeg: the JPEG on success; on failure what ffmpeg said and the error.
|
||||
// Empty output with a clean exit means the seek found nothing.
|
||||
func run(ctx context.Context, c Caps, a attempt, path string, fi fileInfo, size int) ([]byte, string, error) {
|
||||
args := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
|
||||
if a.gpu != nil {
|
||||
args = append(args, "-hwaccel", a.gpu.API, "-hwaccel_device", a.gpu.Device)
|
||||
}
|
||||
if a.lowres > 0 {
|
||||
args = append(args, "-lowres", strconv.Itoa(a.lowres))
|
||||
}
|
||||
if a.seek > 0 {
|
||||
args = append(args, "-noaccurate_seek", "-ss", strconv.FormatFloat(a.seek, 'f', 3, 64))
|
||||
}
|
||||
args = append(args, "-i", path, "-map", "0:"+fi.index, "-frames:v", "1",
|
||||
"-vf", fmt.Sprintf("scale=w=%d:h=%d:force_original_aspect_ratio=decrease:flags=bicubic", size, size),
|
||||
"-f", "image2pipe", "-c:v", "mjpeg", "-q:v", "4", "pipe:1")
|
||||
name, argv := c.FFmpeg, args
|
||||
if c.Nice != "" { // a burst of previews must not crowd out what the machine is for
|
||||
name, argv = c.Nice, append([]string{"-n", "10", c.FFmpeg}, args...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, name, argv...)
|
||||
var out, errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := lastLine(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return nil, msg, err
|
||||
}
|
||||
return out.Bytes(), "", nil
|
||||
}
|
||||
|
||||
// ---- the same, as root ----
|
||||
|
||||
// ProbeRoot is Probe run as root through this binary, for the Files tab's root mode: root
|
||||
// opens every render node, whatever group the daemon's user is in.
|
||||
func ProbeRoot(ctx context.Context, self string) (Caps, error) {
|
||||
full := priv.Wrap(self, "thumb", "--probe")
|
||||
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
if msg := lastLine(errb.String()); msg != "" {
|
||||
return Caps{}, errors.New(msg)
|
||||
}
|
||||
return Caps{}, err
|
||||
}
|
||||
var c Caps
|
||||
if err := json.Unmarshal(out, &c); err != nil {
|
||||
return Caps{}, err
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
type helperOut struct {
|
||||
Engine string `json:"engine,omitempty"`
|
||||
Codec string `json:"codec,omitempty"`
|
||||
Width int `json:"width,omitempty"`
|
||||
Height int `json:"height,omitempty"`
|
||||
JPEG []byte `json:"jpeg,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Permanent bool `json:"permanent,omitempty"` // a DecodeError: the file, not the moment
|
||||
}
|
||||
|
||||
// MakeRoot is Make run as root through this binary, with the caps ProbeRoot found.
|
||||
func MakeRoot(ctx context.Context, self string, c Caps, path string, o Opts) (Result, error) {
|
||||
o = o.defaults()
|
||||
caps, _ := json.Marshal(c)
|
||||
full := priv.Wrap(self, "thumb", "--size", strconv.Itoa(o.Size), "--timeout", o.Timeout.String(), path)
|
||||
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
|
||||
cmd.Stdin = bytes.NewReader(caps)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return Result{}, ctx.Err()
|
||||
}
|
||||
if msg := lastLine(errb.String()); msg != "" {
|
||||
return Result{}, errors.New(msg)
|
||||
}
|
||||
return Result{}, err
|
||||
}
|
||||
var h helperOut
|
||||
if err := json.Unmarshal(out, &h); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if h.Error != "" {
|
||||
if h.Permanent {
|
||||
return Result{}, &DecodeError{Msg: h.Error}
|
||||
}
|
||||
return Result{}, errors.New(h.Error)
|
||||
}
|
||||
return Result{JPEG: h.JPEG, Engine: h.Engine, Codec: h.Codec, Width: h.Width, Height: h.Height}, nil
|
||||
}
|
||||
|
||||
// Helper implements `hamlab thumb …`, the root side of the above. `--probe` prints Caps;
|
||||
// `[--size N] [--timeout D] <path>` reads Caps from stdin and prints the preview as JSON.
|
||||
// Returns the exit code.
|
||||
func Helper(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
|
||||
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
|
||||
ctx := context.Background()
|
||||
o := Opts{}
|
||||
path := ""
|
||||
for i := 0; i < len(args); i++ {
|
||||
switch a := args[i]; {
|
||||
case a == "--probe":
|
||||
return emit(stdout, Probe(ctx), stderr)
|
||||
case a == "--size" && i+1 < len(args):
|
||||
o.Size, _ = strconv.Atoi(args[i+1])
|
||||
i++
|
||||
case a == "--timeout" && i+1 < len(args):
|
||||
o.Timeout, _ = time.ParseDuration(args[i+1])
|
||||
i++
|
||||
case strings.HasPrefix(a, "-"):
|
||||
return fail(errors.New("usage: hamlab thumb --probe | [--size N] [--timeout D] <path>"))
|
||||
default:
|
||||
path = a
|
||||
}
|
||||
}
|
||||
if path == "" || !filepath.IsAbs(path) {
|
||||
return fail(errors.New("usage: hamlab thumb --probe | [--size N] [--timeout D] <absolute path>"))
|
||||
}
|
||||
var c Caps
|
||||
if err := json.NewDecoder(io.LimitReader(stdin, 1<<20)).Decode(&c); err != nil {
|
||||
return fail(fmt.Errorf("caps on stdin: %w", err))
|
||||
}
|
||||
res, err := Make(ctx, c, path, o)
|
||||
if err != nil {
|
||||
var de *DecodeError
|
||||
return emit(stdout, helperOut{Error: err.Error(), Permanent: errors.As(err, &de)}, stderr)
|
||||
}
|
||||
return emit(stdout, helperOut{Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}, stderr)
|
||||
}
|
||||
|
||||
func emit(w io.Writer, v any, stderr io.Writer) int {
|
||||
if err := json.NewEncoder(w).Encode(v); err != nil {
|
||||
fmt.Fprintln(stderr, err.Error())
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ---- small helpers ----
|
||||
|
||||
func lastLine(s string) string {
|
||||
lines := strings.Split(strings.TrimSpace(s), "\n")
|
||||
return strings.TrimSpace(lines[len(lines)-1])
|
||||
}
|
||||
|
||||
func exists(p string) bool { _, err := os.Stat(p); return err == nil }
|
||||
@@ -0,0 +1,253 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/jpeg"
|
||||
"image/png"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestKind(t *testing.T) {
|
||||
for name, want := range map[string]string{"a.JPG": "image", "b.heic": "image", "c.mkv": "video", "GH010098.MP4": "video", "x.OSV": "video", "d.txt": "", "e": "", ".jpg": "image"} {
|
||||
if got := Kind(name); got != want {
|
||||
t.Errorf("Kind(%q) = %q, want %q", name, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestKeyFollowsTheFile(t *testing.T) {
|
||||
a := Key("/x/a.jpg", 10, 20)
|
||||
if a != Key("/x/a.jpg", 10, 20) || len(a) != 64 {
|
||||
t.Fatal("key is not stable")
|
||||
}
|
||||
for _, b := range []string{Key("/x/b.jpg", 10, 20), Key("/x/a.jpg", 11, 20), Key("/x/a.jpg", 10, 21)} {
|
||||
if b == a {
|
||||
t.Fatal("key ignores part of the file's identity")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseVainfo(t *testing.T) {
|
||||
text := `vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 25.2.3 ()
|
||||
VAProfileMPEG2Simple : VAEntrypointVLD
|
||||
VAProfileH264Main : VAEntrypointVLD
|
||||
VAProfileH264High : VAEntrypointEncSliceLP
|
||||
VAProfileJPEGBaseline : VAEntrypointVLD
|
||||
VAProfileHEVCMain10 : VAEntrypointVLD
|
||||
VAProfileHEVCMain10 : VAEntrypointEncSlice
|
||||
VAProfileHEVCMain444_12 : VAEntrypointVLD
|
||||
VAProfileVP9Profile2 : VAEntrypointVLD
|
||||
VAProfileAV1Profile0 : VAEntrypointEncSliceLP
|
||||
`
|
||||
dec, enc := parseVainfo(text)
|
||||
same := func(got, want []string) bool {
|
||||
if len(got) != len(want) {
|
||||
return false
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
if !same(dec, []string{"h264", "hevc", "mjpeg", "mpeg2video", "vp9"}) {
|
||||
t.Fatalf("decodes = %v", dec)
|
||||
}
|
||||
if !same(enc, []string{"av1", "h264", "hevc"}) {
|
||||
t.Fatalf("encodes = %v", enc)
|
||||
}
|
||||
if d, e := parseVainfo(""); d != nil || e != nil {
|
||||
t.Fatal("nothing from nothing")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPickAndEngines(t *testing.T) {
|
||||
igpu := GPU{API: "vaapi", Device: "/dev/dri/renderD128", Codecs: []string{"h264", "hevc"}}
|
||||
arc := GPU{API: "vaapi", Device: "/dev/dri/renderD129", Codecs: []string{"av1", "h264", "hevc"}}
|
||||
c := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", GPUs: []GPU{igpu, arc}}
|
||||
if g := c.pick("av1"); g == nil || g.Device != arc.Device {
|
||||
t.Fatal("av1 should go to the card that lists it")
|
||||
}
|
||||
if g := c.pick("h264"); g == nil || g.Device != igpu.Device {
|
||||
t.Fatal("h264 should go to the first card that lists it")
|
||||
}
|
||||
if g := c.pick("vc1"); g != nil {
|
||||
t.Fatal("a codec nobody lists is software")
|
||||
}
|
||||
// a card nothing could describe is tried, after the ones that answered
|
||||
c.GPUs = append(c.GPUs, GPU{API: "cuda", Device: "0"})
|
||||
if g := c.pick("vc1"); g == nil || g.API != "cuda" {
|
||||
t.Fatal("an undescribed card is the guess of last resort")
|
||||
}
|
||||
// videos: GPU then CPU; images: CPU only
|
||||
if e := engines(c, "video", fileInfo{codec: "hevc"}); len(e) != 2 || e[0] == nil || e[1] != nil {
|
||||
t.Fatal("a video tries the GPU, then the CPU")
|
||||
}
|
||||
if e := engines(c, "image", fileInfo{codec: "mjpeg"}); len(e) != 1 || e[0] != nil {
|
||||
t.Fatal("an image is CPU only")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAttempts(t *testing.T) {
|
||||
a := attempts(nil, "video", fileInfo{duration: 120}, 256)
|
||||
if len(a) != 2 || a[0].seek != 12 || a[1].seek != 0 {
|
||||
t.Fatalf("video attempts = %+v", a)
|
||||
}
|
||||
if a := attempts(nil, "video", fileInfo{duration: 7200}, 256); a[0].seek != 30 {
|
||||
t.Fatalf("seek capped at 30 s, got %v", a[0].seek)
|
||||
}
|
||||
if a := attempts(nil, "video", fileInfo{duration: 2}, 256); len(a) != 1 || a[0].seek != 0 {
|
||||
t.Fatalf("a short clip is taken from its start, got %+v", a)
|
||||
}
|
||||
if a := attempts(nil, "image", fileInfo{codec: "mjpeg", w: 6000, h: 4000}, 256); len(a) != 2 || a[0].lowres != 2 || a[1].lowres != 0 {
|
||||
t.Fatalf("jpeg attempts = %+v", a)
|
||||
}
|
||||
if a := attempts(nil, "image", fileInfo{codec: "mjpeg", w: 800, h: 600}, 256); len(a) != 1 || a[0].lowres != 0 {
|
||||
t.Fatalf("a small jpeg decodes whole, got %+v", a)
|
||||
}
|
||||
if a := attempts(nil, "image", fileInfo{codec: "png", w: 6000, h: 4000}, 256); len(a) != 1 || a[0].lowres != 0 {
|
||||
t.Fatalf("only jpeg has lowres, got %+v", a)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache(t *testing.T) {
|
||||
c := NewCache(filepath.Join(t.TempDir(), "thumbs.db"))
|
||||
k := Key("/a", 1, 2)
|
||||
if _, found, err := c.Get(k); found || err != nil {
|
||||
t.Fatal("an empty cache found something, or complained")
|
||||
}
|
||||
if _, err := os.Stat(c.Path); err == nil {
|
||||
t.Fatal("a miss must not create the database")
|
||||
}
|
||||
if n, b := c.Stats(); n != 0 || b != 0 {
|
||||
t.Fatal("stats of nothing")
|
||||
}
|
||||
if err := c.Put(k, Entry{Path: "/a", Engine: "cpu", Codec: "png", Width: 640, Height: 360, JPEG: []byte("jpeg")}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e, found, err := c.Get(k)
|
||||
if err != nil || !found || string(e.JPEG) != "jpeg" || e.Engine != "cpu" || e.Width != 640 || e.Path != "/a" || e.Made.IsZero() {
|
||||
t.Fatalf("put then get: %+v %v %v", e, found, err)
|
||||
}
|
||||
k2 := Key("/b", 1, 2)
|
||||
if err := c.Put(k2, Entry{Path: "/b", Failure: "no video stream"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if e, found, _ := c.Get(k2); !found || e.JPEG != nil || e.Failure != "no video stream" {
|
||||
t.Fatal("a failure is remembered with its reason")
|
||||
}
|
||||
if err := c.Put(k, Entry{}); err == nil {
|
||||
t.Fatal("an empty preview must not be kept")
|
||||
}
|
||||
// replacing an entry does not count it twice
|
||||
if err := c.Put(k, Entry{Path: "/a", JPEG: []byte("jpeg2")}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
n, b := c.Stats()
|
||||
if n != 2 || b == 0 {
|
||||
t.Fatalf("stats = %d entries, %d bytes", n, b)
|
||||
}
|
||||
// survives a reopen
|
||||
if err := c.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c2 := NewCache(c.Path)
|
||||
if e, found, _ := c2.Get(k); !found || string(e.JPEG) != "jpeg2" {
|
||||
t.Fatal("not there after reopening")
|
||||
}
|
||||
if n, _ := c2.Stats(); n != 2 {
|
||||
t.Fatal("count lost on reopen")
|
||||
}
|
||||
c2.Close()
|
||||
}
|
||||
|
||||
// The real thing, when ffmpeg is here: a PNG and a generated video each come back as a
|
||||
// JPEG no larger than the asked size, with the aspect kept.
|
||||
func TestMakeWithFFmpeg(t *testing.T) {
|
||||
if _, err := exec.LookPath("ffmpeg"); err != nil {
|
||||
t.Skip("no ffmpeg")
|
||||
}
|
||||
c := Probe(context.Background())
|
||||
if c.Error != "" {
|
||||
t.Skip(c.Error)
|
||||
}
|
||||
dir := t.TempDir()
|
||||
img := image.NewRGBA(image.Rect(0, 0, 640, 360))
|
||||
for y := 0; y < 360; y++ {
|
||||
for x := 0; x < 640; x++ {
|
||||
img.Set(x, y, color.RGBA{uint8(x / 3), uint8(y), 90, 255})
|
||||
}
|
||||
}
|
||||
pngPath := filepath.Join(dir, "a.png")
|
||||
f, _ := os.Create(pngPath)
|
||||
_ = png.Encode(f, img)
|
||||
f.Close()
|
||||
res, err := Make(context.Background(), c, pngPath, Opts{Size: 200})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
check(t, res, 200, 112, "cpu")
|
||||
|
||||
// a two-second test pattern, with the always-built-in mpeg4 encoder
|
||||
vid := filepath.Join(dir, "v.mp4")
|
||||
out, err := exec.Command("ffmpeg", "-nostdin", "-v", "error", "-f", "lavfi", "-i", "testsrc=size=320x180:rate=10:duration=2", "-c:v", "mpeg4", "-q:v", "5", vid).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Skipf("could not make a test video: %s", out)
|
||||
}
|
||||
res, err = Make(context.Background(), c, vid, Opts{Size: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
check(t, res, 100, 56, "")
|
||||
if res.Codec != "mpeg4" || res.Width != 320 {
|
||||
t.Fatalf("probe said %s %dx%d", res.Codec, res.Width, res.Height)
|
||||
}
|
||||
|
||||
// not media, and media that isn't
|
||||
if _, err := Make(context.Background(), c, filepath.Join(dir, "x.txt"), Opts{}); err != ErrUnsupported {
|
||||
t.Fatalf("a .txt is unsupported, got %v", err)
|
||||
}
|
||||
bad := filepath.Join(dir, "bad.mp4")
|
||||
_ = os.WriteFile(bad, []byte("not a video at all"), 0o644)
|
||||
_, err = Make(context.Background(), c, bad, Opts{})
|
||||
var de *DecodeError
|
||||
if err == nil || !isDecodeErr(err, &de) {
|
||||
t.Fatalf("garbage should be a DecodeError, got %v", err)
|
||||
}
|
||||
// the clock: a cancelled context is not the file's fault
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
|
||||
defer cancel()
|
||||
if _, err := Make(ctx, c, pngPath, Opts{}); err == nil || isDecodeErr(err, &de) {
|
||||
t.Fatalf("a timeout must not read as a decode failure, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func check(t *testing.T, res Result, w, h int, engine string) {
|
||||
t.Helper()
|
||||
im, err := jpeg.Decode(bytes.NewReader(res.JPEG))
|
||||
if err != nil {
|
||||
t.Fatalf("not a jpeg: %v", err)
|
||||
}
|
||||
if b := im.Bounds(); b.Dx() != w || b.Dy() < h-1 || b.Dy() > h+1 { // the short side rounds either way
|
||||
t.Fatalf("got %dx%d, want %dx%d", b.Dx(), b.Dy(), w, h)
|
||||
}
|
||||
if engine != "" && res.Engine != engine {
|
||||
t.Fatalf("engine %q, want %q", res.Engine, engine)
|
||||
}
|
||||
}
|
||||
|
||||
func isDecodeErr(err error, de **DecodeError) bool {
|
||||
x, ok := err.(*DecodeError)
|
||||
if ok {
|
||||
*de = x
|
||||
}
|
||||
return ok
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package metrics
|
||||
|
||||
// The ZFS ARC — the read cache in RAM — read live from /proc/spl/kstat/zfs/arcstats on a
|
||||
// machine that has the module loaded, and left out of the sample everywhere else. The kstat
|
||||
// file is counters since boot; the hit rate here is over the sampling interval, which is the
|
||||
// number that says whether the cache is doing its job *now*. The lifetime figures and the
|
||||
// pool's caching tiers (L2ARC, SLOG, special vdevs) are the storage package's business.
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const arcstatsPath = "/proc/spl/kstat/zfs/arcstats"
|
||||
|
||||
// ARC is the read cache as one sample sees it. Sizes are bytes.
|
||||
type ARC struct {
|
||||
Size uint64 `json:"size"`
|
||||
Target uint64 `json:"target"` // c: what the ARC is currently allowed to hold
|
||||
Min uint64 `json:"min"` // c_min: the floor it will not shrink below
|
||||
Max uint64 `json:"max"` // c_max: the ceiling (zfs_arc_max, or half of RAM)
|
||||
// what is in it
|
||||
MRU uint64 `json:"mru"` // recently used once
|
||||
MFU uint64 `json:"mfu"` // used more than once
|
||||
Anon uint64 `json:"anon"` // in flight: dirty, not yet on disk
|
||||
Data uint64 `json:"data"` // file contents
|
||||
Metadata uint64 `json:"metadata"` // dnodes, indirect blocks, dbufs, bonus buffers
|
||||
MetaUsed uint64 `json:"meta_used"`
|
||||
Headers uint64 `json:"headers"`
|
||||
Compressed uint64 `json:"compressed"`
|
||||
Uncompressed uint64 `json:"uncompressed"`
|
||||
MRUGhost uint64 `json:"mru_ghost"` // what it recently evicted and still remembers
|
||||
MFUGhost uint64 `json:"mfu_ghost"`
|
||||
// L2ARC, the cache devices (0 when there are none)
|
||||
L2Size uint64 `json:"l2_size"`
|
||||
L2ASize uint64 `json:"l2_asize"` // as stored on the device, after compression
|
||||
// over this interval
|
||||
ReadsPerSec float64 `json:"reads_per_sec"`
|
||||
MissesPerSec float64 `json:"misses_per_sec"`
|
||||
HitPct float64 `json:"hit_pct"` // of the interval's reads; an idle interval keeps the last one
|
||||
HitPctKnown bool `json:"hit_pct_known"` // false until an interval has had any reads
|
||||
L2HitPct float64 `json:"l2_hit_pct"`
|
||||
// since boot
|
||||
Hits uint64 `json:"hits"`
|
||||
Misses uint64 `json:"misses"`
|
||||
LifetimeHitPct float64 `json:"lifetime_hit_pct"`
|
||||
// pressure
|
||||
NoGrow bool `json:"no_grow"` // arc_no_grow: memory pressure is stopping it growing
|
||||
Throttles uint64 `json:"throttles"` // memory_throttle_count since boot
|
||||
Prunes uint64 `json:"prunes"` // arc_prune: metadata evictions asked of the filesystems
|
||||
EvictSkip uint64 `json:"evict_skip"`
|
||||
}
|
||||
|
||||
type arcPrev struct {
|
||||
hits, misses, l2hits, l2misses uint64
|
||||
hitPct float64
|
||||
known bool
|
||||
at time.Time
|
||||
}
|
||||
|
||||
// readKstat parses a kstat table ("name type data" rows after two header lines).
|
||||
func readKstat(p string) map[string]uint64 {
|
||||
s := readStr(p)
|
||||
if s == "" {
|
||||
return nil
|
||||
}
|
||||
out := map[string]uint64{}
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) != 3 || f[1] != "4" { // type 4 is KSTAT_DATA_UINT64
|
||||
continue
|
||||
}
|
||||
v, err := strconv.ParseUint(f[2], 10, 64)
|
||||
if err == nil {
|
||||
out[f[0]] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// arc reads the ARC; nil when this machine has no ZFS module loaded.
|
||||
func (c *Collector) arc(now time.Time) *ARC {
|
||||
k := readKstat(arcstatsPath)
|
||||
if k == nil {
|
||||
return nil
|
||||
}
|
||||
a := &ARC{Size: k["size"], Target: k["c"], Min: k["c_min"], Max: k["c_max"],
|
||||
MRU: k["mru_size"], MFU: k["mfu_size"], Anon: k["anon_size"], Data: k["data_size"], Metadata: k["metadata_size"],
|
||||
MetaUsed: k["arc_meta_used"], Headers: k["hdr_size"], Compressed: k["compressed_size"], Uncompressed: k["uncompressed_size"],
|
||||
MRUGhost: k["mru_ghost_size"], MFUGhost: k["mfu_ghost_size"], L2Size: k["l2_size"], L2ASize: k["l2_asize"],
|
||||
Hits: k["hits"], Misses: k["misses"], NoGrow: k["arc_no_grow"] != 0, Throttles: k["memory_throttle_count"], Prunes: k["arc_prune"], EvictSkip: k["evict_skip"]}
|
||||
if n := a.Hits + a.Misses; n > 0 {
|
||||
a.LifetimeHitPct = round1(float64(a.Hits) / float64(n) * 100)
|
||||
}
|
||||
p := c.prevARC
|
||||
if !p.at.IsZero() && a.Hits >= p.hits && a.Misses >= p.misses {
|
||||
if dt := now.Sub(p.at).Seconds(); dt > 0 {
|
||||
hits, misses := a.Hits-p.hits, a.Misses-p.misses
|
||||
a.ReadsPerSec = round1(float64(hits+misses) / dt)
|
||||
a.MissesPerSec = round1(float64(misses) / dt)
|
||||
if hits+misses > 0 {
|
||||
p.hitPct, p.known = round1(float64(hits)/float64(hits+misses)*100), true
|
||||
}
|
||||
if l2 := (k["l2_hits"] - p.l2hits) + (k["l2_misses"] - p.l2misses); l2 > 0 && k["l2_hits"] >= p.l2hits {
|
||||
a.L2HitPct = round1(float64(k["l2_hits"]-p.l2hits) / float64(l2) * 100)
|
||||
}
|
||||
}
|
||||
}
|
||||
a.HitPct, a.HitPctKnown = p.hitPct, p.known
|
||||
c.prevARC = arcPrev{hits: a.Hits, misses: a.Misses, l2hits: k["l2_hits"], l2misses: k["l2_misses"], hitPct: p.hitPct, known: p.known, at: now}
|
||||
return a
|
||||
}
|
||||
@@ -0,0 +1,296 @@
|
||||
package metrics
|
||||
|
||||
// CPU power, from the RAPL energy counters the kernel keeps under /sys/class/powercap: Intel
|
||||
// since Sandy Bridge, and AMD Zen through the same driver. Each counter is a running total in
|
||||
// µJ, so power is the difference between two readings over the time between them. Every
|
||||
// package is summed (`package-0`, and `package-1` on a two-socket board). The other top-level
|
||||
// domains are left out. `psys` claims the whole platform but is only as good as the firmware's
|
||||
// calibration, and on one machine here it reads lower than the package inside it. The MMIO
|
||||
// copy of a package (`intel-rapl-mmio:0`) measures the same thing again. This is the CPU's own
|
||||
// figure, shown beside its clock: cores, built-in graphics and memory controller, not disks,
|
||||
// fans or the power supply.
|
||||
//
|
||||
// The counters are root-only since the PLATYPUS side channel (2020). So, like the GPU client
|
||||
// walk, they are read in-process when the daemon is root, and otherwise by a root helper,
|
||||
// `hamlab cpu-energy`, started once through sudo or doas and asked over a pipe at every
|
||||
// sample: one privilege escalation per daemon, not one a second in the auth log. The helper
|
||||
// takes nothing from the daemon but the newline that asks; it finds and reads the counters
|
||||
// itself. Without root there is no reading, and nothing is shown.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
const raplRoot = "/sys/class/powercap"
|
||||
|
||||
var raplPkgRe = regexp.MustCompile(`^package-\d+$`)
|
||||
|
||||
// raplCount is one package's counter at one moment.
|
||||
type raplCount struct {
|
||||
Dir string `json:"d"`
|
||||
UJ uint64 `json:"uj"`
|
||||
}
|
||||
|
||||
type cpuPowerState struct {
|
||||
probed bool
|
||||
dirs []string // the package domains, found at the first sample
|
||||
max map[string]uint64 // each counter's range: where it wraps to zero
|
||||
useHelper bool // the counters are not readable in-process
|
||||
helper *energyHelper
|
||||
prev []raplCount
|
||||
prevAt time.Time
|
||||
}
|
||||
|
||||
// raplPackages lists the package domains under root, sorted. intel-rapl:0 is a package;
|
||||
// intel-rapl:0:0 is a part of one (core, uncore, dram).
|
||||
func raplPackages(root string) []string {
|
||||
ents, err := os.ReadDir(root)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, e := range ents {
|
||||
n := e.Name()
|
||||
if !strings.HasPrefix(n, "intel-rapl:") || strings.Count(n, ":") != 1 {
|
||||
continue
|
||||
}
|
||||
if raplPkgRe.MatchString(strings.TrimSpace(readStr(filepath.Join(root, n, "name")))) {
|
||||
out = append(out, n)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func readRAPL(root string, dirs []string) ([]raplCount, bool) {
|
||||
out := make([]raplCount, 0, len(dirs))
|
||||
for _, d := range dirs {
|
||||
b, err := os.ReadFile(filepath.Join(root, d, "energy_uj"))
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
v, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64)
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
out = append(out, raplCount{d, v})
|
||||
}
|
||||
return out, len(out) > 0
|
||||
}
|
||||
|
||||
// raplWatts is the power between two readings of the same counters. A counter that wrapped
|
||||
// is carried across its range. One that went backwards with no known range, or a set of
|
||||
// counters that changed between readings, gives no answer rather than a wrong one.
|
||||
func raplWatts(prev, cur []raplCount, max map[string]uint64, dt float64) (float64, bool) {
|
||||
each, ok := raplWattsEach(prev, cur, max, dt)
|
||||
var w float64
|
||||
for _, x := range each {
|
||||
w += x
|
||||
}
|
||||
return w, ok
|
||||
}
|
||||
|
||||
// raplWattsEach is raplWatts per counter: one package each.
|
||||
func raplWattsEach(prev, cur []raplCount, max map[string]uint64, dt float64) ([]float64, bool) {
|
||||
if dt <= 0 || len(prev) == 0 || len(prev) != len(cur) {
|
||||
return nil, false
|
||||
}
|
||||
out := make([]float64, len(cur))
|
||||
for i := range cur {
|
||||
if cur[i].Dir != prev[i].Dir {
|
||||
return nil, false
|
||||
}
|
||||
if cur[i].UJ >= prev[i].UJ {
|
||||
out[i] = float64(cur[i].UJ-prev[i].UJ) / 1e6 / dt
|
||||
continue
|
||||
}
|
||||
m := max[cur[i].Dir]
|
||||
if m == 0 || prev[i].UJ > m {
|
||||
return nil, false
|
||||
}
|
||||
out[i] = float64(m-prev[i].UJ+cur[i].UJ) / 1e6 / dt
|
||||
}
|
||||
return out, true
|
||||
}
|
||||
|
||||
// cpuPower is the CPU packages' power since the previous sample, in watts; 0 when this
|
||||
// machine has no counters or no way to read them. On a machine with more than one package
|
||||
// it also gives each one's, indexed by package id (the socket).
|
||||
func (c *Collector) cpuPower(now time.Time) (float64, []float64) {
|
||||
p := &c.power
|
||||
if !p.probed {
|
||||
p.probed = true
|
||||
p.dirs = raplPackages(raplRoot)
|
||||
p.max = map[string]uint64{}
|
||||
for _, d := range p.dirs {
|
||||
if v, ok := readInt(filepath.Join(raplRoot, d, "max_energy_range_uj")); ok && v > 0 {
|
||||
p.max[d] = uint64(v)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(p.dirs) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
var cur []raplCount
|
||||
ok := false
|
||||
if !p.useHelper {
|
||||
if cur, ok = readRAPL(raplRoot, p.dirs); !ok && priv.Mode() != "root" && priv.Available() {
|
||||
p.useHelper = true
|
||||
}
|
||||
}
|
||||
if p.useHelper {
|
||||
if p.helper == nil {
|
||||
p.helper = &energyHelper{}
|
||||
}
|
||||
cur, ok = p.helper.read()
|
||||
}
|
||||
if !ok {
|
||||
p.prev = nil
|
||||
return 0, nil
|
||||
}
|
||||
each, good := raplWattsEach(p.prev, cur, p.max, now.Sub(p.prevAt).Seconds())
|
||||
p.prev, p.prevAt = cur, now
|
||||
if !good {
|
||||
return 0, nil
|
||||
}
|
||||
var w float64
|
||||
for _, x := range each {
|
||||
w += x
|
||||
}
|
||||
return round1(w), perPackage(cur, each)
|
||||
}
|
||||
|
||||
// perPackage puts each counter's watts at its package's id — intel-rapl:1 is "package-1" —
|
||||
// and is nil for a single package, whose figure is the total.
|
||||
func perPackage(cur []raplCount, each []float64) []float64 {
|
||||
if len(cur) < 2 {
|
||||
return nil
|
||||
}
|
||||
var out []float64
|
||||
for i, c := range cur {
|
||||
id := i
|
||||
if n, ok := strings.CutPrefix(strings.TrimSpace(readStr(filepath.Join(raplRoot, c.Dir, "name"))), "package-"); ok {
|
||||
if v, err := strconv.Atoi(n); err == nil && v >= 0 && v < 64 {
|
||||
id = v
|
||||
}
|
||||
}
|
||||
for len(out) <= id {
|
||||
out = append(out, 0)
|
||||
}
|
||||
out[id] = round1(each[i])
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// energyHelper is the root child that answers each newline with the package counters, one
|
||||
// JSON line. The same shape as the GPU client walk's helper, kept separate from it.
|
||||
type energyHelper struct {
|
||||
mu sync.Mutex
|
||||
cmd *exec.Cmd
|
||||
in io.WriteCloser
|
||||
out *bufio.Reader
|
||||
failedAt time.Time
|
||||
}
|
||||
|
||||
func (h *energyHelper) read() ([]raplCount, bool) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
if h.cmd == nil {
|
||||
if !h.failedAt.IsZero() && time.Since(h.failedAt) < helperRetry {
|
||||
return nil, false
|
||||
}
|
||||
if !h.start() {
|
||||
h.failedAt = time.Now()
|
||||
return nil, false
|
||||
}
|
||||
}
|
||||
// a stuck child must not stall the sampler: it is killed, and tried again later
|
||||
cmd := h.cmd
|
||||
stop := time.AfterFunc(5*time.Second, func() { cmd.Process.Kill() })
|
||||
defer stop.Stop()
|
||||
if _, err := io.WriteString(h.in, "\n"); err != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
line, err := h.out.ReadBytes('\n')
|
||||
if err != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
var cs []raplCount
|
||||
if json.Unmarshal(line, &cs) != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
return cs, len(cs) > 0
|
||||
}
|
||||
|
||||
func (h *energyHelper) start() bool {
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
full := priv.Wrap(self, "cpu-energy")
|
||||
cmd := exec.Command(full[0], full[1:]...)
|
||||
in, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return false
|
||||
}
|
||||
h.cmd, h.in, h.out = cmd, in, bufio.NewReader(out)
|
||||
return true
|
||||
}
|
||||
|
||||
func (h *energyHelper) fail() {
|
||||
if h.cmd != nil {
|
||||
h.in.Close()
|
||||
h.cmd.Process.Kill()
|
||||
go h.cmd.Wait()
|
||||
}
|
||||
h.cmd, h.in, h.out = nil, nil, nil
|
||||
h.failedAt = time.Now()
|
||||
}
|
||||
|
||||
// CPUEnergyHelper is `hamlab cpu-energy`: for every line read on stdin, one JSON line of the
|
||||
// package counters on stdout. Ends when stdin does — when the daemon that started it has gone.
|
||||
func CPUEnergyHelper(stdin io.Reader, stdout io.Writer) int {
|
||||
return cpuEnergyHelper(raplRoot, stdin, stdout)
|
||||
}
|
||||
|
||||
func cpuEnergyHelper(root string, stdin io.Reader, stdout io.Writer) int {
|
||||
dirs := raplPackages(root)
|
||||
sc := bufio.NewScanner(stdin)
|
||||
w := bufio.NewWriter(stdout)
|
||||
for sc.Scan() {
|
||||
cs, ok := readRAPL(root, dirs)
|
||||
if !ok {
|
||||
cs = []raplCount{}
|
||||
}
|
||||
b, _ := json.Marshal(cs)
|
||||
w.Write(b)
|
||||
w.WriteByte('\n')
|
||||
if w.Flush() != nil {
|
||||
return 1
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A fake /sys/class/powercap: a package with its parts, the platform domain, and the MMIO
|
||||
// copy of the same package. Only the package counts.
|
||||
func fakeRAPL(t *testing.T, domains map[string][2]string) string {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
for dir, v := range domains {
|
||||
d := filepath.Join(root, dir)
|
||||
if err := os.MkdirAll(d, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
os.WriteFile(filepath.Join(d, "name"), []byte(v[0]+"\n"), 0o644)
|
||||
os.WriteFile(filepath.Join(d, "energy_uj"), []byte(v[1]+"\n"), 0o644)
|
||||
}
|
||||
return root
|
||||
}
|
||||
|
||||
func TestRAPLPackagesOnlyCountsPackages(t *testing.T) {
|
||||
root := fakeRAPL(t, map[string][2]string{
|
||||
"intel-rapl:0": {"package-0", "1000"},
|
||||
"intel-rapl:0:0": {"core", "600"},
|
||||
"intel-rapl:0:1": {"uncore", "10"},
|
||||
"intel-rapl:1": {"psys", "300"},
|
||||
"intel-rapl-mmio:0": {"package-0", "1000"},
|
||||
})
|
||||
if got := raplPackages(root); len(got) != 1 || got[0] != "intel-rapl:0" {
|
||||
t.Fatalf("packages = %v, want [intel-rapl:0]", got)
|
||||
}
|
||||
|
||||
two := fakeRAPL(t, map[string][2]string{"intel-rapl:0": {"package-0", "5"}, "intel-rapl:1": {"package-1", "7"}})
|
||||
got := raplPackages(two)
|
||||
if strings.Join(got, ",") != "intel-rapl:0,intel-rapl:1" {
|
||||
t.Fatalf("two sockets: packages = %v", got)
|
||||
}
|
||||
cs, ok := readRAPL(two, got)
|
||||
if !ok || len(cs) != 2 || cs[0].UJ != 5 || cs[1].UJ != 7 {
|
||||
t.Fatalf("readRAPL = %+v, %v", cs, ok)
|
||||
}
|
||||
if got := raplPackages(filepath.Join(two, "nope")); len(got) != 0 {
|
||||
t.Fatalf("no powercap: packages = %v, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRAPLWatts(t *testing.T) {
|
||||
c := func(d string, uj uint64) raplCount { return raplCount{d, uj} }
|
||||
max := map[string]uint64{"intel-rapl:0": 1_000_000_000}
|
||||
cases := []struct {
|
||||
name string
|
||||
prev, cur []raplCount
|
||||
dt float64
|
||||
want float64
|
||||
ok bool
|
||||
}{
|
||||
{"steady", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:0", 10_000_000)}, 2, 5, true},
|
||||
{"two packages add", []raplCount{c("intel-rapl:0", 0), c("intel-rapl:1", 0)}, []raplCount{c("intel-rapl:0", 4_000_000), c("intel-rapl:1", 6_000_000)}, 1, 10, true},
|
||||
{"wrapped", []raplCount{c("intel-rapl:0", 999_000_000)}, []raplCount{c("intel-rapl:0", 1_000_000)}, 1, 2, true},
|
||||
{"backwards, range unknown", []raplCount{c("intel-rapl:1", 5_000_000)}, []raplCount{c("intel-rapl:1", 1)}, 1, 0, false},
|
||||
{"counters changed", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:1", 9)}, 1, 0, false},
|
||||
{"first reading", nil, []raplCount{c("intel-rapl:0", 9)}, 1, 0, false},
|
||||
{"no time passed", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:0", 9)}, 0, 0, false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
got, ok := raplWatts(tc.prev, tc.cur, max, tc.dt)
|
||||
if ok != tc.ok || (ok && (got < tc.want-1e-9 || got > tc.want+1e-9)) {
|
||||
t.Errorf("%s: got %v, %v; want %v, %v", tc.name, got, ok, tc.want, tc.ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The helper answers every newline with the counters it found itself, and stops at EOF.
|
||||
func TestCPUEnergyHelper(t *testing.T) {
|
||||
root := fakeRAPL(t, map[string][2]string{"intel-rapl:0": {"package-0", "42"}, "intel-rapl:0:0": {"core", "7"}})
|
||||
var out bytes.Buffer
|
||||
if code := cpuEnergyHelper(root, strings.NewReader("\n\n"), &out); code != 0 {
|
||||
t.Fatalf("exit %d", code)
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(out.String()), "\n")
|
||||
if len(lines) != 2 {
|
||||
t.Fatalf("got %d lines, want 2: %q", len(lines), out.String())
|
||||
}
|
||||
var cs []raplCount
|
||||
if err := json.Unmarshal([]byte(lines[1]), &cs); err != nil || len(cs) != 1 || cs[0] != (raplCount{"intel-rapl:0", 42}) {
|
||||
t.Fatalf("line = %s (%v)", lines[1], err)
|
||||
}
|
||||
}
|
||||
|
||||
// History keeps the average of the samples that had a reading.
|
||||
func TestFoldCPUPower(t *testing.T) {
|
||||
b := []Sample{{T: 1, CPUPower: 4}, {T: 2}, {T: 3, CPUPower: 8}}
|
||||
if got := fold(b).CPUPower; got != 6 {
|
||||
t.Fatalf("folded CPU power = %v, want 6", got)
|
||||
}
|
||||
if got := fold([]Sample{{T: 1}, {T: 2}}).CPUPower; got != 0 {
|
||||
t.Fatalf("no readings folded to %v, want 0", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package metrics
|
||||
|
||||
import "testing"
|
||||
|
||||
// a two-socket board: two coretemp chips, and the CPU's temperature is the hotter package
|
||||
func TestCPUTempHottestPackage(t *testing.T) {
|
||||
temps := []Temp{
|
||||
{Chip: "coretemp", Label: "Core 0", C: 31},
|
||||
{Chip: "coretemp", Label: "Core 0", C: 44, Pkg: 1},
|
||||
{Chip: "coretemp", Label: "Package id 0", C: 35},
|
||||
{Chip: "coretemp", Label: "Package id 1", C: 41, Pkg: 1},
|
||||
}
|
||||
if got := cpuTemp(temps); got != 41 {
|
||||
t.Fatalf("cpuTemp = %v, want 41", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,214 @@
|
||||
package metrics
|
||||
|
||||
import "strconv"
|
||||
|
||||
// One kept history point is the average of the samples behind it. At the default ten-minute
|
||||
// window that bucket is a single sample and nothing is averaged; at a day it is five minutes
|
||||
// of them, and the difference matters — a graph of the last day drawn from whichever instant
|
||||
// landed on each five-minute mark would show spikes that never happened and miss the ones
|
||||
// that did.
|
||||
//
|
||||
// The newest sample of the bucket is the template: its shape, labels, totals and slow
|
||||
// sections (filesystems, UPS) are the current ones. Only the moving numbers are averaged,
|
||||
// and only where an entry is present in that sample — a disk plugged in halfway through the
|
||||
// bucket is averaged over the samples that have it.
|
||||
|
||||
func fold(b []Sample) Sample {
|
||||
out := b[len(b)-1]
|
||||
if len(b) == 1 {
|
||||
return out
|
||||
}
|
||||
n := float64(len(b))
|
||||
sum := func(get func(Sample) float64) float64 {
|
||||
var t float64
|
||||
for _, s := range b {
|
||||
t += get(s)
|
||||
}
|
||||
return t / n
|
||||
}
|
||||
sumU := func(get func(Sample) uint64) uint64 {
|
||||
var t uint64
|
||||
for _, s := range b {
|
||||
t += get(s)
|
||||
}
|
||||
return t / uint64(len(b))
|
||||
}
|
||||
out.CPU = sum(func(s Sample) float64 { return s.CPU })
|
||||
// CPU power: over the samples that have a reading, so a helper restart is not a dip to zero
|
||||
var pw, pn float64
|
||||
for _, s := range b {
|
||||
if s.CPUPower > 0 {
|
||||
pw, pn = pw+s.CPUPower, pn+1
|
||||
}
|
||||
}
|
||||
out.CPUPower = 0
|
||||
if pn > 0 {
|
||||
out.CPUPower = round1(pw / pn)
|
||||
}
|
||||
// and each package's, the same way
|
||||
out.CPUPowerPkgs = nil
|
||||
for _, s := range b {
|
||||
for i, w := range s.CPUPowerPkgs {
|
||||
for len(out.CPUPowerPkgs) <= i {
|
||||
out.CPUPowerPkgs = append(out.CPUPowerPkgs, 0)
|
||||
}
|
||||
out.CPUPowerPkgs[i] += w
|
||||
}
|
||||
}
|
||||
if pn > 0 {
|
||||
for i := range out.CPUPowerPkgs {
|
||||
out.CPUPowerPkgs[i] = round1(out.CPUPowerPkgs[i] / pn)
|
||||
}
|
||||
}
|
||||
out.Load[0] = sum(func(s Sample) float64 { return s.Load[0] })
|
||||
out.Load[1] = sum(func(s Sample) float64 { return s.Load[1] })
|
||||
out.Load[2] = sum(func(s Sample) float64 { return s.Load[2] })
|
||||
|
||||
out.Cores = append([]float64(nil), out.Cores...)
|
||||
for i := range out.Cores {
|
||||
var t, c float64
|
||||
for _, s := range b {
|
||||
if i < len(s.Cores) {
|
||||
t, c = t+s.Cores[i], c+1
|
||||
}
|
||||
}
|
||||
if c > 0 {
|
||||
out.Cores[i] = t / c
|
||||
}
|
||||
}
|
||||
|
||||
out.Mem.Used = sumU(func(s Sample) uint64 { return s.Mem.Used })
|
||||
out.Mem.Avail = sumU(func(s Sample) uint64 { return s.Mem.Avail })
|
||||
out.Mem.Free = sumU(func(s Sample) uint64 { return s.Mem.Free })
|
||||
out.Mem.Cached = sumU(func(s Sample) uint64 { return s.Mem.Cached })
|
||||
out.Swap.Used = sumU(func(s Sample) uint64 { return s.Swap.Used })
|
||||
|
||||
// the rest are lists whose entries are matched by name across the bucket
|
||||
temps := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Temps))
|
||||
for _, x := range s.Temps {
|
||||
m[x.Chip+"/"+x.Label+"/"+strconv.Itoa(x.Pkg)] = []float64{x.C}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Temps = append([]Temp(nil), out.Temps...)
|
||||
for i, x := range out.Temps {
|
||||
if v, ok := temps[x.Chip+"/"+x.Label+"/"+strconv.Itoa(x.Pkg)]; ok {
|
||||
out.Temps[i].C = v[0]
|
||||
}
|
||||
}
|
||||
|
||||
fans := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Fans))
|
||||
for _, x := range s.Fans {
|
||||
m[x.Chip+"/"+x.Label] = []float64{float64(x.RPM)}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Fans = append([]Fan(nil), out.Fans...)
|
||||
for i, x := range out.Fans {
|
||||
if v, ok := fans[x.Chip+"/"+x.Label]; ok {
|
||||
out.Fans[i].RPM = int(v[0] + .5)
|
||||
}
|
||||
}
|
||||
|
||||
nets := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Net))
|
||||
for _, x := range s.Net {
|
||||
m[x.Iface] = []float64{x.RxBps, x.TxBps}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Net = append([]Net(nil), out.Net...)
|
||||
for i, x := range out.Net {
|
||||
if v, ok := nets[x.Iface]; ok {
|
||||
out.Net[i].RxBps, out.Net[i].TxBps = v[0], v[1]
|
||||
}
|
||||
}
|
||||
|
||||
disks := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Disks))
|
||||
for _, x := range s.Disks {
|
||||
m[x.Name] = []float64{x.ReadBps, x.WriteBps, x.UtilPct}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Disks = append([]Disk(nil), out.Disks...)
|
||||
for i, x := range out.Disks {
|
||||
if v, ok := disks[x.Name]; ok {
|
||||
out.Disks[i].ReadBps, out.Disks[i].WriteBps, out.Disks[i].UtilPct = v[0], v[1], v[2]
|
||||
}
|
||||
}
|
||||
out.ZFSWrite = foldZFSWrites(b, out.ZFSWrite)
|
||||
return out
|
||||
}
|
||||
|
||||
// foldZFSWrites averages the write path's rates and the buffer's fill over a bucket, per pool;
|
||||
// everything else (limits, the mover's state, the last walk) is the newest sample's.
|
||||
func foldZFSWrites(b []Sample, last *ZFSWrites) *ZFSWrites {
|
||||
if last == nil {
|
||||
return nil
|
||||
}
|
||||
w := *last
|
||||
w.Pools = append([]PoolWrites(nil), last.Pools...)
|
||||
var dirty, delayed, over, n float64
|
||||
for _, s := range b {
|
||||
if s.ZFSWrite != nil {
|
||||
dirty, delayed, over, n = dirty+float64(s.ZFSWrite.Dirty), delayed+s.ZFSWrite.DelayedPerSec, over+s.ZFSWrite.OverMaxPerSec, n+1
|
||||
}
|
||||
}
|
||||
if n > 0 {
|
||||
w.Dirty, w.DelayedPerSec, w.OverMaxPerSec = uint64(dirty/n), round1(delayed/n), round1(over/n)
|
||||
}
|
||||
per := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := map[string][]float64{}
|
||||
if s.ZFSWrite == nil {
|
||||
return m
|
||||
}
|
||||
for _, p := range s.ZFSWrite.Pools {
|
||||
v := []float64{p.WriteBps, p.LogicalBps, 0, 0}
|
||||
if p.Landing != nil {
|
||||
v[2], v[3] = p.Landing.InBps, p.Landing.DrainBps
|
||||
}
|
||||
m[p.Pool] = v
|
||||
}
|
||||
return m
|
||||
})
|
||||
for i, p := range w.Pools {
|
||||
if v, ok := per[p.Pool]; ok {
|
||||
w.Pools[i].WriteBps, w.Pools[i].LogicalBps = round1(v[0]), round1(v[1])
|
||||
if p.Landing != nil {
|
||||
l := *p.Landing
|
||||
l.InBps, l.DrainBps = round1(v[2]), round1(v[3])
|
||||
w.Pools[i].Landing = &l
|
||||
}
|
||||
}
|
||||
}
|
||||
return &w
|
||||
}
|
||||
|
||||
// avgBy averages each sample's named numbers across the bucket, per name.
|
||||
func avgBy(b []Sample, of func(Sample) map[string][]float64) map[string][]float64 {
|
||||
sums := map[string][]float64{}
|
||||
count := map[string]float64{}
|
||||
for _, s := range b {
|
||||
for k, vs := range of(s) {
|
||||
acc := sums[k]
|
||||
if acc == nil {
|
||||
acc = make([]float64, len(vs))
|
||||
}
|
||||
for i, v := range vs {
|
||||
if i < len(acc) {
|
||||
acc[i] += v
|
||||
}
|
||||
}
|
||||
sums[k], count[k] = acc, count[k]+1
|
||||
}
|
||||
}
|
||||
for k, acc := range sums {
|
||||
for i := range acc {
|
||||
acc[i] /= count[k]
|
||||
}
|
||||
}
|
||||
return sums
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestFoldAverages(t *testing.T) {
|
||||
b := []Sample{
|
||||
{T: 1, CPU: 10, Cores: []float64{10, 0}, Mem: Mem{Total: 100, Used: 10},
|
||||
Temps: []Temp{{Chip: "c", Label: "l", C: 40}}, Net: []Net{{Iface: "eth0", RxBps: 100}},
|
||||
Disks: []Disk{{Name: "sda", ReadBps: 10, UtilPct: 5}}},
|
||||
{T: 2, CPU: 30, Cores: []float64{30, 100}, Mem: Mem{Total: 100, Used: 30},
|
||||
Temps: []Temp{{Chip: "c", Label: "l", C: 60}}, Net: []Net{{Iface: "eth0", RxBps: 300}, {Iface: "new", RxBps: 7}},
|
||||
Disks: []Disk{{Name: "sda", ReadBps: 30, UtilPct: 15}}},
|
||||
}
|
||||
f := fold(b)
|
||||
if f.T != 2 {
|
||||
t.Errorf("a folded point is stamped with the newest sample in it, got %d", f.T)
|
||||
}
|
||||
if f.CPU != 20 || f.Cores[0] != 20 || f.Cores[1] != 50 {
|
||||
t.Errorf("cpu averaged: %v %v", f.CPU, f.Cores)
|
||||
}
|
||||
if f.Mem.Used != 20 || f.Mem.Total != 100 {
|
||||
t.Errorf("memory: used averaged, total taken from the newest, got %+v", f.Mem)
|
||||
}
|
||||
if f.Temps[0].C != 50 || f.Net[0].RxBps != 200 || f.Disks[0].ReadBps != 20 || f.Disks[0].UtilPct != 10 {
|
||||
t.Errorf("lists matched by name: %v %v %v", f.Temps, f.Net, f.Disks)
|
||||
}
|
||||
if len(f.Net) != 2 || f.Net[1].RxBps != 7 {
|
||||
t.Errorf("an interface that appeared inside the bucket is averaged over the samples that have it: %v", f.Net)
|
||||
}
|
||||
// the template's own slices must not be touched: Latest() shares them
|
||||
if b[1].Cores[0] != 30 || b[1].Temps[0].C != 60 {
|
||||
t.Error("fold wrote through to the sample it templated from")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStrideAndTrim(t *testing.T) {
|
||||
c := &Collector{interval: 2 * time.Second, window: DefaultWindow}
|
||||
if got := c.stride(); got != 2*time.Second {
|
||||
t.Errorf("a window of %v over %d points is finer than the interval, so the interval wins; got %v", DefaultWindow, History, got)
|
||||
}
|
||||
c.window = 24 * time.Hour
|
||||
if got := c.stride(); got != 24*time.Hour/History {
|
||||
t.Errorf("stride at a day: %v", got)
|
||||
}
|
||||
// a full day of samples at the day stride keeps exactly the budget
|
||||
c.interval, c.window = time.Second, time.Hour
|
||||
stride := c.stride().Milliseconds()
|
||||
for i := 0; i < 5*History; i++ {
|
||||
c.keep(Sample{T: int64(i) * stride})
|
||||
}
|
||||
if len(c.ring) != History {
|
||||
t.Errorf("the point count is the budget, got %d", len(c.ring))
|
||||
}
|
||||
if span := c.ring[len(c.ring)-1].T - c.ring[0].T; span > c.window.Milliseconds() {
|
||||
t.Errorf("the kept history reaches back further than the window: %d ms", span)
|
||||
}
|
||||
// narrowing the window drops what is now too old, at once
|
||||
c.SetWindow(MinWindow)
|
||||
if span := c.ring[len(c.ring)-1].T - c.ring[0].T; span > MinWindow.Milliseconds() {
|
||||
t.Errorf("narrowing the window left %d ms of history", span)
|
||||
}
|
||||
c.SetWindow(365 * 24 * time.Hour)
|
||||
if c.Window() != MaxWindow {
|
||||
t.Errorf("a window past the maximum is the maximum, got %v", c.Window())
|
||||
}
|
||||
}
|
||||
|
||||
// Points are one stride apart — not a stride and a sample — and jitter doesn't change that.
|
||||
func TestKeepSpacing(t *testing.T) {
|
||||
c := &Collector{interval: 2 * time.Second, window: time.Hour} // stride 12 s: six samples a point
|
||||
at := int64(1_000_000)
|
||||
for i := 0; i < 6*40; i++ {
|
||||
jitter := int64((i*37)%41 - 20) // ±20 ms, deterministic
|
||||
c.keep(Sample{T: at + jitter, CPU: float64(i % 6)})
|
||||
at += 2000
|
||||
}
|
||||
if len(c.ring) != 40 {
|
||||
t.Fatalf("240 samples at six a point are 40 points, got %d", len(c.ring))
|
||||
}
|
||||
for i := 1; i < len(c.ring); i++ {
|
||||
if gap := c.ring[i].T - c.ring[i-1].T; gap < 11900 || gap > 12100 {
|
||||
t.Fatalf("point %d is %d ms after the one before; a stride is 12000", i, gap)
|
||||
}
|
||||
}
|
||||
if c.ring[3].CPU != 2.5 {
|
||||
t.Errorf("each point is the mean of its own six samples (0…5 → 2.5), got %v", c.ring[3].CPU)
|
||||
}
|
||||
// and when the stride is the interval, every sample is a point
|
||||
c = &Collector{interval: 2 * time.Second, window: DefaultWindow}
|
||||
for i := 0; i < 50; i++ {
|
||||
c.keep(Sample{T: int64(i)*2000 + int64(i%3) - 1})
|
||||
}
|
||||
if len(c.ring) != 50 {
|
||||
t.Errorf("at the default window every sample is kept, got %d of 50", len(c.ring))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFoldNVMe(t *testing.T) {
|
||||
// an Apple SSD as the kernel lists it: one controller, three namespaces
|
||||
got := foldNVMe([]Disk{
|
||||
{Name: "nvme0n1", Model: "APPLE SSD AP0512Z", Size: 500 << 30, ReadBps: 100, WriteBps: 200, UtilPct: 12, Temp: 37},
|
||||
{Name: "nvme0n2", Model: "APPLE SSD AP0512Z", Size: 3 << 20, ReadBps: 10, WriteBps: 20, UtilPct: 40},
|
||||
{Name: "nvme0n3", Model: "APPLE SSD AP0512Z", Size: 128 << 20, ReadBps: 1, WriteBps: 2, UtilPct: 5},
|
||||
{Name: "sda", Size: 100 << 30, ReadBps: 7, UtilPct: 3},
|
||||
{Name: "nvme1n1", Size: 200 << 30, ReadBps: 5},
|
||||
})
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("three namespaces of one controller are one drive: %v", names(got))
|
||||
}
|
||||
d := got[0]
|
||||
if d.Name != "nvme0" {
|
||||
t.Errorf("the drive is the controller, got %q", d.Name)
|
||||
}
|
||||
if d.Size != 500<<30+3<<20+128<<20 {
|
||||
t.Errorf("its size is all of its namespaces, got %d", d.Size)
|
||||
}
|
||||
if d.ReadBps != 111 || d.WriteBps != 222 {
|
||||
t.Errorf("rates add up, got %v/%v", d.ReadBps, d.WriteBps)
|
||||
}
|
||||
if d.UtilPct != 40 {
|
||||
t.Errorf("busy is the longest of them, not the sum: got %v", d.UtilPct)
|
||||
}
|
||||
if d.Temp != 37 {
|
||||
t.Errorf("a temperature from any namespace is the drive's, got %v", d.Temp)
|
||||
}
|
||||
if got[1].Name != "sda" || got[1].UtilPct != 3 || got[2].Name != "nvme1" {
|
||||
t.Errorf("everything else is left alone: %v", names(got))
|
||||
}
|
||||
if n := len(foldNVMe(nil)); n != 0 {
|
||||
t.Errorf("no disks, no drives, got %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func names(d []Disk) []string {
|
||||
out := make([]string, len(d))
|
||||
for i, x := range d {
|
||||
out[i] = x.Name
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,896 @@
|
||||
package metrics
|
||||
|
||||
// GPU telemetry, the way btop and intel_gpu_top get it without a vendor library: the
|
||||
// device from sysfs, and who is using it from DRM fdinfo.
|
||||
//
|
||||
// - The device: /sys/class/drm/card<n> with a driver we know to be a GPU. Clock from the
|
||||
// driver's frequency files, temperature/power/fan from the hwmon under the device, and
|
||||
// how busy it is from the one counter every Intel GT exports without root — RC6 idle
|
||||
// residency, so busy = 100 − idle. amdgpu says it outright (gpu_busy_percent).
|
||||
// - The clients: every process holding /dev/dri/* open has a drm-engine-<name> running
|
||||
// total (nanoseconds) in /proc/<pid>/fdinfo/<fd>; the delta between two reads is that
|
||||
// process's share of each engine, and the sum across clients is the engine's load. Other
|
||||
// users' fdinfo is unreadable, so when the daemon isn't root it keeps one root child
|
||||
// (`hamlab drm-clients`) that does the walk, asked over a pipe — one sudo per daemon
|
||||
// start, not one per second.
|
||||
//
|
||||
// A GPU with no counter of either kind (Asahi, today) is still listed: a name is better
|
||||
// than silence about hardware the machine has.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// GPU is one graphics device as a Sample reports it.
|
||||
type GPU struct {
|
||||
Name string `json:"name"`
|
||||
Driver string `json:"driver"`
|
||||
Bus string `json:"bus,omitempty"` // PCI address; empty for a platform device
|
||||
Integrated bool `json:"integrated,omitempty"`
|
||||
Usage float64 `json:"usage"` // % busy
|
||||
UsageSrc string `json:"usage_src,omitempty"` // "idle" (100 − idle residency), "busy" (the driver's own %), "clients" (the busiest engine); "" when nothing measures it
|
||||
Engines map[string]float64 `json:"engines,omitempty"` // render/video/copy/enhance… % from the clients, normalised by engine count
|
||||
Temp float64 `json:"temp,omitempty"`
|
||||
TempSrc string `json:"temp_src,omitempty"` // "gpu" from the device's own sensor, "package" the CPU die an integrated GPU shares
|
||||
PowerW float64 `json:"power_w,omitempty"`
|
||||
PowerCapW float64 `json:"power_cap_w,omitempty"`
|
||||
FanRPM int `json:"fan_rpm,omitempty"`
|
||||
FanPct *float64 `json:"fan_pct,omitempty"` // fan speed as a % of its range: nvidia-smi's fan.speed, or the hwmon's pwm1; nil when the card has no such reading (0 is a stopped fan)
|
||||
MHz float64 `json:"mhz,omitempty"` // actual clock (0 when power-gated)
|
||||
ReqMHz float64 `json:"req_mhz,omitempty"` // what the driver asked for
|
||||
MaxMHz float64 `json:"max_mhz,omitempty"`
|
||||
MemUsed uint64 `json:"mem_used,omitempty"` // VRAM used, or the clients' resident bytes where the driver has no VRAM counter
|
||||
MemTotal uint64 `json:"mem_total,omitempty"`
|
||||
Clients []GPUClient `json:"clients"`
|
||||
ClientsOK bool `json:"clients_ok"` // false when fdinfo of other users' processes could not be read
|
||||
Stalled bool `json:"stalled,omitempty"` // a read of this card has been out for gpuStallAfter; its readings are paused until it returns
|
||||
}
|
||||
|
||||
// GPUClient is one process with the device open.
|
||||
type GPUClient struct {
|
||||
PID int `json:"pid"`
|
||||
Name string `json:"name"`
|
||||
Usage float64 `json:"usage"` // its busiest engine, %
|
||||
Engines map[string]float64 `json:"engines,omitempty"`
|
||||
Mem uint64 `json:"mem,omitempty"` // resident bytes
|
||||
}
|
||||
|
||||
// gpuDrivers are DRM drivers that drive a GPU, not just a scanout (simpledrm, ast, the
|
||||
// Apple display coprocessor) — those have no engines and nothing to say.
|
||||
var gpuDrivers = map[string]bool{"i915": true, "xe": true, "amdgpu": true, "radeon": true, "nouveau": true, "nvidia-drm": true, "nvidia": true,
|
||||
"asahi": true, "panfrost": true, "panthor": true, "v3d": true, "msm": true, "etnaviv": true, "lima": true, "powervr": true, "virtio_gpu": true}
|
||||
|
||||
type gpuDev struct {
|
||||
card, dev, key, bus, driver, name, hwmon string
|
||||
integrated, stalled bool
|
||||
prevIdle, prevEnergy float64
|
||||
prevAt time.Time
|
||||
nv []GPUClient // nvidia: the process table from the last read (pmon), fdinfo says nothing there
|
||||
nvRead []GPUClient // the same, as the read in flight left it: only that read touches it
|
||||
pending chan gpuRead // the read in flight, nil when none is
|
||||
pendingAt time.Time
|
||||
last *GPU // the last read that came back
|
||||
readFn func(*GPU, time.Time) // nil is read (tests stand a slow one in)
|
||||
}
|
||||
|
||||
type gpuRead struct {
|
||||
x GPU
|
||||
nv []GPUClient
|
||||
}
|
||||
|
||||
type gpuState struct {
|
||||
devs []*gpuDev
|
||||
listedAt time.Time
|
||||
clients map[string]clientPrev // device key + client id → engine totals at the previous scan
|
||||
last []drmFD // the newest client walk
|
||||
scan func() ([]drmFD, bool) // the walk; nil is scanClients (tests stand a fake list in)
|
||||
helper *drmHelper
|
||||
ok bool // the last walk saw every process
|
||||
}
|
||||
|
||||
type clientPrev struct {
|
||||
ns map[string]uint64
|
||||
at time.Time
|
||||
}
|
||||
|
||||
const gpuRelist = time.Minute // how often the card list is re-read: hotplug is rare, the readdir is cheap
|
||||
|
||||
// gpus is the GPU section of a sample. cpuTemp stands in for an integrated GPU's
|
||||
// temperature: it is the same die.
|
||||
func (c *Collector) gpus(now time.Time, cpuTemp float64) []GPU {
|
||||
g := &c.gpu
|
||||
if g.devs == nil || now.Sub(g.listedAt) >= gpuRelist {
|
||||
g.devs, g.listedAt = listGPUs(g.devs), now
|
||||
}
|
||||
if len(g.devs) == 0 {
|
||||
return []GPU{}
|
||||
}
|
||||
if g.clients == nil {
|
||||
g.clients = map[string]clientPrev{}
|
||||
}
|
||||
// The clients are read on every sample, like everything else in it: their engine shares
|
||||
// move on the same tick as the device's own numbers, at whatever rate is set.
|
||||
scan := c.scanClients
|
||||
if g.scan != nil {
|
||||
scan = g.scan
|
||||
}
|
||||
g.last, g.ok = scan()
|
||||
out := make([]GPU, 0, len(g.devs))
|
||||
live := map[string]bool{}
|
||||
for _, d := range g.devs {
|
||||
x := GPU{Name: d.name, Driver: d.driver, Bus: d.bus, Integrated: d.integrated, Clients: []GPUClient{}, ClientsOK: g.ok}
|
||||
d.readWithin(&x, now)
|
||||
if x.Temp == 0 && d.integrated && cpuTemp > 0 {
|
||||
x.Temp, x.TempSrc = cpuTemp, "package"
|
||||
}
|
||||
// the clients: one entry per client id, under the first pid that holds it (a child
|
||||
// that inherited the fd shares the client)
|
||||
byID := map[string]*GPUClient{}
|
||||
var order []string
|
||||
engines := map[string]float64{}
|
||||
var resident uint64
|
||||
if d.driver == "nvidia" { // no engine counters in its fdinfo; nvidia-smi pmon is the client list
|
||||
x.Clients, x.ClientsOK = append(x.Clients, d.nv...), true
|
||||
}
|
||||
for _, f := range g.last {
|
||||
if d.driver == "nvidia" {
|
||||
break
|
||||
}
|
||||
fi := parseFdinfo(f.Info)
|
||||
if fi.pdev != "" && fi.pdev != d.bus || fi.pdev == "" && fi.driver != d.driver {
|
||||
continue
|
||||
}
|
||||
key := d.key + "/" + fi.client
|
||||
live[key] = true
|
||||
if byID[fi.client] != nil {
|
||||
continue
|
||||
}
|
||||
cl := &GPUClient{PID: f.PID, Name: f.Comm, Mem: fi.mem}
|
||||
byID[fi.client] = cl
|
||||
order = append(order, fi.client)
|
||||
resident += fi.mem
|
||||
if prev, ok := g.clients[key]; ok {
|
||||
if dt := now.Sub(prev.at).Nanoseconds(); dt > 0 {
|
||||
for e, ns := range fi.engines {
|
||||
p, ok := prev.ns[e]
|
||||
if !ok || ns < p {
|
||||
continue
|
||||
}
|
||||
pct := float64(ns-p) / float64(dt) * 100
|
||||
if n := fi.capacity[e]; n > 1 {
|
||||
pct /= float64(n)
|
||||
}
|
||||
pct = round1(pct)
|
||||
if cl.Engines == nil {
|
||||
cl.Engines = map[string]float64{}
|
||||
}
|
||||
cl.Engines[e] = pct
|
||||
engines[e] += pct
|
||||
if pct > cl.Usage {
|
||||
cl.Usage = pct
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
g.clients[key] = clientPrev{ns: fi.engines, at: now}
|
||||
}
|
||||
for _, id := range order {
|
||||
x.Clients = append(x.Clients, *byID[id])
|
||||
}
|
||||
sort.SliceStable(x.Clients, func(i, j int) bool { return x.Clients[i].Usage > x.Clients[j].Usage })
|
||||
if len(engines) > 0 {
|
||||
x.Engines = map[string]float64{}
|
||||
for e, v := range engines {
|
||||
if v > 100 {
|
||||
v = 100
|
||||
}
|
||||
x.Engines[e] = round1(v)
|
||||
}
|
||||
}
|
||||
if x.UsageSrc == "" && len(x.Engines) > 0 {
|
||||
for _, v := range x.Engines {
|
||||
if v > x.Usage {
|
||||
x.Usage = v
|
||||
}
|
||||
}
|
||||
x.UsageSrc = "clients"
|
||||
}
|
||||
if x.MemTotal == 0 && x.MemUsed == 0 {
|
||||
x.MemUsed = resident
|
||||
}
|
||||
out = append(out, x)
|
||||
}
|
||||
for k := range g.clients { // a client that closed the device
|
||||
if !live[k] {
|
||||
delete(g.clients, k)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// listGPUs enumerates the cards, keeping the counters of those already known.
|
||||
func listGPUs(known []*gpuDev) []*gpuDev {
|
||||
ents, _ := os.ReadDir("/sys/class/drm")
|
||||
old := map[string]*gpuDev{}
|
||||
for _, d := range known {
|
||||
old[d.card] = d
|
||||
}
|
||||
out := []*gpuDev{}
|
||||
for _, e := range ents {
|
||||
n := e.Name()
|
||||
if !strings.HasPrefix(n, "card") || strings.Contains(n, "-") {
|
||||
continue
|
||||
}
|
||||
card := "/sys/class/drm/" + n
|
||||
dev, err := filepath.EvalSymlinks(card + "/device")
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
driver := ueventValue(dev+"/uevent", "DRIVER")
|
||||
if driver == "" { // some drivers hang the DRIVER key off the parent
|
||||
driver = filepath.Base(readLink(dev + "/driver"))
|
||||
}
|
||||
if !gpuDrivers[driver] {
|
||||
continue
|
||||
}
|
||||
if d, ok := old[card]; ok && d.dev == dev {
|
||||
out = append(out, d)
|
||||
continue
|
||||
}
|
||||
d := &gpuDev{card: card, dev: dev, driver: driver}
|
||||
if pci := ueventValue(dev+"/uevent", "PCI_SLOT_NAME"); pci != "" {
|
||||
d.bus, d.key = pci, pci
|
||||
// an on-die GPU hangs off root bus 0 (Intel's 00:02.0, an AMD APU's 00:08.0 bridge)
|
||||
d.integrated = strings.HasPrefix(pci, "0000:00:")
|
||||
} else {
|
||||
d.key = "driver:" + driver
|
||||
d.integrated = true
|
||||
}
|
||||
if hw, _ := filepath.Glob(dev + "/hwmon/hwmon*"); len(hw) > 0 {
|
||||
d.hwmon = hw[0]
|
||||
}
|
||||
d.name = gpuName(dev, d.bus, driver)
|
||||
out = append(out, d)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].card < out[j].card })
|
||||
return out
|
||||
}
|
||||
|
||||
// RenderNodeName is the GPU behind a render node ("/dev/dri/renderD128" → "Intel Arc A380"),
|
||||
// named the same way the Processes tab names it; "" when sysfs has nothing for it.
|
||||
func RenderNodeName(node string) string {
|
||||
dev, err := filepath.EvalSymlinks("/sys/class/drm/" + filepath.Base(node) + "/device")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
driver := ueventValue(dev+"/uevent", "DRIVER")
|
||||
if driver == "" {
|
||||
driver = filepath.Base(readLink(dev + "/driver"))
|
||||
}
|
||||
return gpuName(dev, ueventValue(dev+"/uevent", "PCI_SLOT_NAME"), driver)
|
||||
}
|
||||
|
||||
// RenderNodeIntegrated reports a render node whose GPU sits on the CPU die (PCI root bus 0,
|
||||
// or no PCI bus at all), as opposed to a card on the bus.
|
||||
func RenderNodeIntegrated(node string) bool {
|
||||
dev, err := filepath.EvalSymlinks("/sys/class/drm/" + filepath.Base(node) + "/device")
|
||||
if err != nil {
|
||||
return true
|
||||
}
|
||||
pci := ueventValue(dev+"/uevent", "PCI_SLOT_NAME")
|
||||
return pci == "" || strings.HasPrefix(pci, "0000:00:")
|
||||
}
|
||||
|
||||
var gpuVendors = map[string]string{"0x8086": "Intel", "0x1002": "AMD", "0x10de": "NVIDIA", "0x1af4": "Virtio", "0x106b": "Apple"}
|
||||
|
||||
var (
|
||||
revRe = regexp.MustCompile(`\s*\(rev [^)]*\)`)
|
||||
bracketRe = regexp.MustCompile(`\[([^\]]+)\]`)
|
||||
)
|
||||
|
||||
// gpuName is "Intel Arc A380": from the embedded PCI ID table first, so the name is the
|
||||
// same on every machine; from lspci for a card newer than the table; and failing both,
|
||||
// the vendor and the raw device ID.
|
||||
func gpuName(dev, bus, driver string) string {
|
||||
if bus != "" {
|
||||
vid := strings.TrimPrefix(strings.TrimSpace(readStr(dev+"/vendor")), "0x")
|
||||
did := strings.TrimPrefix(strings.TrimSpace(readStr(dev+"/device")), "0x")
|
||||
if vn, dn := pciName(vid, did); dn != "" {
|
||||
return lspciName(vn + " " + dn)
|
||||
}
|
||||
if out, ok := runOut("lspci", "-s", bus); ok {
|
||||
if _, desc, found := strings.Cut(out, ": "); found {
|
||||
return lspciName(desc)
|
||||
}
|
||||
}
|
||||
}
|
||||
vendor := gpuVendors[strings.TrimSpace(readStr(dev+"/vendor"))]
|
||||
if compat := readStr(dev + "/of_node/compatible"); compat != "" { // "apple,agx-t8103\0apple,agx-g13g"
|
||||
first := strings.SplitN(compat, "\x00", 2)[0]
|
||||
if v, model, ok := strings.Cut(first, ","); ok {
|
||||
if vendor == "" && v != "" {
|
||||
vendor = strings.ToUpper(v[:1]) + v[1:]
|
||||
}
|
||||
return vendor + " GPU (" + model + ")"
|
||||
}
|
||||
}
|
||||
if vendor == "" && driver != "" {
|
||||
vendor = strings.ToUpper(driver[:1]) + driver[1:]
|
||||
}
|
||||
if id := strings.TrimPrefix(strings.TrimSpace(readStr(dev+"/device")), "0x"); id != "" {
|
||||
return strings.TrimSpace(vendor + " GPU " + id)
|
||||
}
|
||||
return strings.TrimSpace(vendor + " GPU")
|
||||
}
|
||||
|
||||
// lspciName turns "Intel Corporation DG2 [Arc A380] (rev 05)" into "Intel Arc A380".
|
||||
func lspciName(desc string) string {
|
||||
desc = revRe.ReplaceAllString(strings.TrimSpace(desc), "")
|
||||
vendor := ""
|
||||
for _, v := range []struct{ long, short string }{{"Intel Corporation", "Intel"}, {"Advanced Micro Devices, Inc. [AMD/ATI]", "AMD"}, {"Advanced Micro Devices, Inc. [AMD]", "AMD"}, {"NVIDIA Corporation", "NVIDIA"}, {"Red Hat, Inc.", "Virtio"}} {
|
||||
if strings.HasPrefix(desc, v.long) {
|
||||
vendor = v.short
|
||||
desc = strings.TrimSpace(desc[len(v.long):])
|
||||
break
|
||||
}
|
||||
}
|
||||
if vendor == "" { // vendor not recognised: its first word
|
||||
if i := strings.IndexByte(desc, ' '); i > 0 {
|
||||
vendor, desc = desc[:i], strings.TrimSpace(desc[i+1:])
|
||||
}
|
||||
}
|
||||
if m := bracketRe.FindStringSubmatch(desc); m != nil {
|
||||
desc = m[1]
|
||||
}
|
||||
return strings.TrimSpace(vendor + " " + desc)
|
||||
}
|
||||
|
||||
// readWithin is read with a deadline. A read that does not come back must not take the
|
||||
// whole sampler with it — every graph on every page of this machine stops — so one still out
|
||||
// after gpuReadWithin is left running, the sample goes on with the card's last readings, and
|
||||
// no second read starts until it returns. Late is normal for nvidia-smi on a busy card; only a
|
||||
// read out for gpuStallAfter marks the card stalled, and that clears the moment it returns. A
|
||||
// sysfs read that never returns keeps it stalled, at the cost of one goroutine.
|
||||
var (
|
||||
gpuReadWithin = 3 * time.Second
|
||||
gpuStallAfter = 15 * time.Second // past both of readNvidia's nvidia-smi calls timing out (runOut, 5 s each)
|
||||
)
|
||||
|
||||
func (d *gpuDev) readWithin(x *GPU, now time.Time) {
|
||||
var r gpuRead
|
||||
back := false
|
||||
if d.pending == nil {
|
||||
done := make(chan gpuRead, 1)
|
||||
y := *x
|
||||
read := d.read
|
||||
if d.readFn != nil {
|
||||
read = d.readFn
|
||||
}
|
||||
go func() {
|
||||
read(&y, now)
|
||||
done <- gpuRead{y, d.nvRead}
|
||||
}()
|
||||
d.pending, d.pendingAt = done, now
|
||||
select {
|
||||
case r, back = <-done:
|
||||
case <-time.After(gpuReadWithin):
|
||||
}
|
||||
} else { // started on an earlier sample: taken if it is back, never waited on again
|
||||
select {
|
||||
case r, back = <-d.pending:
|
||||
default:
|
||||
}
|
||||
}
|
||||
ok := x.ClientsOK
|
||||
defer func() { x.ClientsOK = ok }()
|
||||
if back {
|
||||
if d.stalled {
|
||||
log.Printf("gpu: %s (%s): its read came back after %s; readings resume", d.name, d.card, now.Sub(d.pendingAt).Round(time.Second))
|
||||
}
|
||||
d.pending, d.nv, d.stalled, d.last = nil, r.nv, false, &r.x
|
||||
*x = r.x
|
||||
return
|
||||
}
|
||||
if d.last != nil {
|
||||
*x = *d.last
|
||||
}
|
||||
if now.Sub(d.pendingAt) >= gpuStallAfter {
|
||||
if !d.stalled {
|
||||
log.Printf("gpu: %s (%s): a read has not returned in %s; its readings are paused until it does", d.name, d.card, gpuStallAfter)
|
||||
}
|
||||
d.stalled, x.Stalled = true, true
|
||||
}
|
||||
}
|
||||
|
||||
// read fills the device-level numbers: clock, idle/busy, hwmon.
|
||||
func (d *gpuDev) read(x *GPU, now time.Time) {
|
||||
dt := now.Sub(d.prevAt).Seconds()
|
||||
switch d.driver {
|
||||
case "nvidia":
|
||||
d.readNvidia(x)
|
||||
case "i915":
|
||||
x.MHz = readF(d.card + "/gt_act_freq_mhz")
|
||||
x.ReqMHz = readF(d.card + "/gt_cur_freq_mhz")
|
||||
x.MaxMHz = readF(d.card + "/gt_max_freq_mhz")
|
||||
idle, ok := readInt(d.card + "/gt/gt0/rc6_residency_ms")
|
||||
if !ok {
|
||||
idle, ok = readInt(d.card + "/power/rc6_residency_ms")
|
||||
}
|
||||
if ok {
|
||||
d.busyFromIdle(x, float64(idle), dt)
|
||||
}
|
||||
case "xe":
|
||||
gt := d.dev + "/tile0/gt0"
|
||||
x.MHz = readF(gt + "/freq0/act_freq")
|
||||
x.ReqMHz = readF(gt + "/freq0/cur_freq")
|
||||
x.MaxMHz = readF(gt + "/freq0/max_freq")
|
||||
if idle, ok := readInt(gt + "/gtidle/idle_residency_ms"); ok {
|
||||
d.busyFromIdle(x, float64(idle), dt)
|
||||
}
|
||||
case "amdgpu":
|
||||
if v, ok := readInt(d.dev + "/gpu_busy_percent"); ok {
|
||||
x.Usage, x.UsageSrc = float64(v), "busy"
|
||||
}
|
||||
x.MemUsed = uint64(readF(d.dev + "/mem_info_vram_used"))
|
||||
x.MemTotal = uint64(readF(d.dev + "/mem_info_vram_total"))
|
||||
// "1: 1800Mhz *" — the starred level is the current one, the last the highest
|
||||
for _, line := range strings.Split(readStr(d.dev+"/pp_dpm_sclk"), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) < 2 {
|
||||
continue
|
||||
}
|
||||
mhz := leadingNum(f[1])
|
||||
x.MaxMHz = mhz
|
||||
if strings.HasSuffix(line, "*") {
|
||||
x.MHz = mhz
|
||||
}
|
||||
}
|
||||
}
|
||||
if d.hwmon != "" {
|
||||
if t := readF(d.hwmon + "/temp1_input"); t > 0 {
|
||||
x.Temp, x.TempSrc = t/1000, "gpu"
|
||||
}
|
||||
if p := readF(d.hwmon + "/power1_input"); p > 0 {
|
||||
x.PowerW = round1(p / 1e6)
|
||||
} else if p := readF(d.hwmon + "/power1_average"); p > 0 {
|
||||
x.PowerW = round1(p / 1e6)
|
||||
} else if e := readF(d.hwmon + "/energy1_input"); e > 0 { // µJ, cumulative
|
||||
if d.prevEnergy > 0 && e >= d.prevEnergy && dt > 0 && !d.prevAt.IsZero() {
|
||||
x.PowerW = round1((e - d.prevEnergy) / 1e6 / dt)
|
||||
}
|
||||
d.prevEnergy = e
|
||||
}
|
||||
if p := readF(d.hwmon + "/power1_cap"); p > 0 {
|
||||
x.PowerCapW = p / 1e6
|
||||
} else if p := readF(d.hwmon + "/power1_max"); p > 0 {
|
||||
x.PowerCapW = p / 1e6
|
||||
}
|
||||
x.FanRPM = int(readF(d.hwmon + "/fan1_input"))
|
||||
if pwm, err := strconv.ParseFloat(strings.TrimSpace(readStr(d.hwmon+"/pwm1")), 64); err == nil {
|
||||
top := readF(d.hwmon + "/pwm1_max")
|
||||
if top <= 0 {
|
||||
top = 255
|
||||
}
|
||||
v := round1(pwm / top * 100)
|
||||
x.FanPct = &v
|
||||
}
|
||||
}
|
||||
d.prevAt = now
|
||||
}
|
||||
|
||||
// busyFromIdle: the GT's idle residency is a running ms total; the share of the interval
|
||||
// it did not cover is how busy the device was.
|
||||
func (d *gpuDev) busyFromIdle(x *GPU, idle, dt float64) {
|
||||
x.UsageSrc = "idle"
|
||||
if d.prevAt.IsZero() || dt <= 0 || idle < d.prevIdle {
|
||||
d.prevIdle = idle
|
||||
return
|
||||
}
|
||||
busy := 100 - (idle-d.prevIdle)/(dt*1000)*100
|
||||
if busy < 0 {
|
||||
busy = 0
|
||||
}
|
||||
if busy > 100 {
|
||||
busy = 100
|
||||
}
|
||||
x.Usage = round1(busy)
|
||||
d.prevIdle = idle
|
||||
}
|
||||
|
||||
// ---- DRM fdinfo: who has the device open ----
|
||||
|
||||
// drmFD is one client of a DRM device: the fdinfo text of one open fd, with its owner.
|
||||
type drmFD struct {
|
||||
PID int `json:"pid"`
|
||||
Comm string `json:"comm"`
|
||||
Info string `json:"info"`
|
||||
}
|
||||
|
||||
type fdinfo struct {
|
||||
driver, pdev, client string
|
||||
engines map[string]uint64 // ns, cumulative
|
||||
capacity map[string]int
|
||||
mem uint64 // bytes
|
||||
}
|
||||
|
||||
func parseFdinfo(s string) fdinfo {
|
||||
fi := fdinfo{engines: map[string]uint64{}, capacity: map[string]int{}}
|
||||
var resident, memory uint64
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
k, v, ok := strings.Cut(line, ":")
|
||||
if !ok || !strings.HasPrefix(k, "drm-") {
|
||||
continue
|
||||
}
|
||||
v = strings.TrimSpace(v)
|
||||
switch {
|
||||
case k == "drm-driver":
|
||||
fi.driver = v
|
||||
case k == "drm-pdev":
|
||||
fi.pdev = v
|
||||
case k == "drm-client-id":
|
||||
fi.client = v
|
||||
case strings.HasPrefix(k, "drm-engine-capacity-"):
|
||||
fi.capacity[k[len("drm-engine-capacity-"):]], _ = strconv.Atoi(v)
|
||||
case strings.HasPrefix(k, "drm-engine-"):
|
||||
n, _ := strconv.ParseUint(strings.TrimSuffix(v, " ns"), 10, 64)
|
||||
fi.engines[k[len("drm-engine-"):]] = n
|
||||
case strings.HasPrefix(k, "drm-resident-"):
|
||||
resident += kib(v)
|
||||
case strings.HasPrefix(k, "drm-memory-"):
|
||||
memory += kib(v)
|
||||
}
|
||||
}
|
||||
fi.mem = resident
|
||||
if fi.mem == 0 {
|
||||
fi.mem = memory
|
||||
}
|
||||
return fi
|
||||
}
|
||||
|
||||
// kib parses "52624 KiB" (or MiB, or a bare count of bytes) to bytes.
|
||||
func kib(v string) uint64 {
|
||||
f := strings.Fields(v)
|
||||
if len(f) == 0 {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.ParseUint(f[0], 10, 64)
|
||||
if len(f) > 1 {
|
||||
switch f[1] {
|
||||
case "KiB":
|
||||
n <<= 10
|
||||
case "MiB":
|
||||
n <<= 20
|
||||
case "GiB":
|
||||
n <<= 30
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// drmScan walks /proc for every fd open on /dev/dri/*, one entry per client per process.
|
||||
// Run as root it sees everyone; as a user, only that user's processes.
|
||||
func drmScan() (out []drmFD, complete bool) {
|
||||
ents, err := os.ReadDir("/proc")
|
||||
if err != nil {
|
||||
return []drmFD{}, false
|
||||
}
|
||||
complete = true
|
||||
out = []drmFD{}
|
||||
for _, e := range ents {
|
||||
pid, err := strconv.Atoi(e.Name())
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
fds, err := os.ReadDir("/proc/" + e.Name() + "/fd")
|
||||
if err != nil {
|
||||
if !os.IsNotExist(err) {
|
||||
complete = false
|
||||
}
|
||||
continue
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
comm := ""
|
||||
for _, fd := range fds {
|
||||
target, err := os.Readlink("/proc/" + e.Name() + "/fd/" + fd.Name())
|
||||
if err != nil || !strings.HasPrefix(target, "/dev/dri/") {
|
||||
continue
|
||||
}
|
||||
info := readStr("/proc/" + e.Name() + "/fdinfo/" + fd.Name())
|
||||
_, rest, ok := strings.Cut(info, "drm-client-id:")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
id := strings.TrimSpace(strings.SplitN(rest, "\n", 2)[0])
|
||||
if seen[id] {
|
||||
continue
|
||||
}
|
||||
seen[id] = true
|
||||
if comm == "" {
|
||||
comm = strings.TrimSpace(readStr("/proc/" + e.Name() + "/comm"))
|
||||
}
|
||||
out = append(out, drmFD{PID: pid, Comm: comm, Info: info})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].PID < out[j].PID })
|
||||
return out, complete
|
||||
}
|
||||
|
||||
// scanClients walks in-process when that sees everything (root), else through the helper.
|
||||
func (c *Collector) scanClients() ([]drmFD, bool) {
|
||||
if priv.Mode() == "root" || !priv.Available() {
|
||||
return drmScan()
|
||||
}
|
||||
if c.gpu.helper == nil {
|
||||
c.gpu.helper = &drmHelper{}
|
||||
}
|
||||
if fds, ok := c.gpu.helper.scan(); ok {
|
||||
return fds, true
|
||||
}
|
||||
return drmScan() // the helper is down: our own processes are better than nothing
|
||||
}
|
||||
|
||||
// drmHelper is the root child that answers "scan" requests. One line in, one JSON line out.
|
||||
type drmHelper struct {
|
||||
mu sync.Mutex
|
||||
cmd *exec.Cmd
|
||||
in io.WriteCloser
|
||||
out *bufio.Reader
|
||||
failedAt time.Time
|
||||
}
|
||||
|
||||
const helperRetry = 30 * time.Second
|
||||
|
||||
func (h *drmHelper) scan() ([]drmFD, bool) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
if h.cmd == nil {
|
||||
if !h.failedAt.IsZero() && time.Since(h.failedAt) < helperRetry {
|
||||
return nil, false
|
||||
}
|
||||
if !h.start() {
|
||||
h.failedAt = time.Now()
|
||||
return nil, false
|
||||
}
|
||||
}
|
||||
// a stuck child must not stall the sampler: it is killed, and tried again later
|
||||
cmd := h.cmd
|
||||
stop := time.AfterFunc(5*time.Second, func() { cmd.Process.Kill() })
|
||||
defer stop.Stop()
|
||||
if _, err := io.WriteString(h.in, "\n"); err != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
line, err := h.out.ReadBytes('\n')
|
||||
if err != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
var fds []drmFD
|
||||
if json.Unmarshal(line, &fds) != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
if fds == nil {
|
||||
fds = []drmFD{}
|
||||
}
|
||||
return fds, true
|
||||
}
|
||||
|
||||
func (h *drmHelper) start() bool {
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
full := priv.Wrap(self, "drm-clients")
|
||||
cmd := exec.Command(full[0], full[1:]...)
|
||||
in, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return false
|
||||
}
|
||||
h.cmd, h.in, h.out = cmd, in, bufio.NewReaderSize(out, 1<<20)
|
||||
return true
|
||||
}
|
||||
|
||||
func (h *drmHelper) fail() {
|
||||
if h.cmd != nil {
|
||||
h.in.Close()
|
||||
h.cmd.Process.Kill()
|
||||
go h.cmd.Wait()
|
||||
}
|
||||
h.cmd, h.in, h.out = nil, nil, nil
|
||||
h.failedAt = time.Now()
|
||||
}
|
||||
|
||||
// DRMClientsHelper is `hamlab drm-clients`: for every line read on stdin, one JSON line of
|
||||
// drmFD on stdout. Ends when stdin does — when the daemon that started it has gone.
|
||||
func DRMClientsHelper(stdin io.Reader, stdout io.Writer) int {
|
||||
sc := bufio.NewScanner(stdin)
|
||||
w := bufio.NewWriter(stdout)
|
||||
for sc.Scan() {
|
||||
fds, _ := drmScan()
|
||||
b, _ := json.Marshal(fds)
|
||||
w.Write(b)
|
||||
w.WriteByte('\n')
|
||||
if w.Flush() != nil {
|
||||
return 1
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ---- small helpers ----
|
||||
|
||||
func readF(p string) float64 {
|
||||
v, err := strconv.ParseFloat(strings.TrimSpace(readStr(p)), 64)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func readLink(p string) string {
|
||||
t, err := os.Readlink(p)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func ueventValue(p, key string) string {
|
||||
for _, line := range strings.Split(readStr(p), "\n") {
|
||||
if k, v, ok := strings.Cut(line, "="); ok && k == key {
|
||||
return strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// The proprietary NVIDIA driver exports no busy counter in sysfs and no engine totals in
|
||||
// fdinfo, so the device and its processes come from nvidia-smi — two short invocations per
|
||||
// sample (~20 ms and ~75 ms on a GTX 965M), no root needed. The PCI address selects the
|
||||
// card, so two NVIDIA GPUs stay apart.
|
||||
func (d *gpuDev) readNvidia(x *GPU) {
|
||||
if out, ok := runOut("nvidia-smi", "--id="+d.bus, "--query-gpu=utilization.gpu,utilization.encoder,utilization.decoder,memory.used,memory.total,clocks.sm,clocks.max.sm,temperature.gpu,power.draw,power.limit,fan.speed", "--format=csv,noheader,nounits"); ok {
|
||||
parseNvidiaQuery(x, out)
|
||||
}
|
||||
if out, ok := runOut("nvidia-smi", "pmon", "-c", "1", "-s", "um", "--id="+d.bus); ok {
|
||||
d.nvRead = parseNvidiaPmon(out)
|
||||
}
|
||||
}
|
||||
|
||||
// nvNum is one nvidia-smi CSV field; "[N/A]" (no sensor, no limit) is "nothing".
|
||||
func nvNum(s string) (float64, bool) {
|
||||
s = strings.TrimSpace(s)
|
||||
if s == "" || strings.Contains(s, "N/A") {
|
||||
return 0, false
|
||||
}
|
||||
v, err := strconv.ParseFloat(s, 64)
|
||||
return v, err == nil
|
||||
}
|
||||
|
||||
// parseNvidiaQuery fills a GPU from one --query-gpu line in the field order readNvidia asks
|
||||
// for: gpu%, encoder%, decoder%, mem used MiB, mem total MiB, SM MHz, max SM MHz, °C, W, W cap,
|
||||
// fan % ("[N/A]" on a card with no fan of its own: a laptop's, a passive datacentre card).
|
||||
func parseNvidiaQuery(x *GPU, line string) {
|
||||
f := strings.Split(strings.TrimSpace(line), ",")
|
||||
if len(f) < 10 {
|
||||
return
|
||||
}
|
||||
if v, ok := nvNum(f[0]); ok {
|
||||
x.Usage, x.UsageSrc = v, "busy"
|
||||
x.Engines = map[string]float64{"render": v}
|
||||
enc, okE := nvNum(f[1])
|
||||
dec, okD := nvNum(f[2])
|
||||
if okE || okD {
|
||||
x.Engines["video"] = max(enc, dec)
|
||||
}
|
||||
}
|
||||
if v, ok := nvNum(f[3]); ok {
|
||||
x.MemUsed = uint64(v) << 20
|
||||
}
|
||||
if v, ok := nvNum(f[4]); ok {
|
||||
x.MemTotal = uint64(v) << 20
|
||||
}
|
||||
if v, ok := nvNum(f[5]); ok {
|
||||
x.MHz = v
|
||||
}
|
||||
if v, ok := nvNum(f[6]); ok {
|
||||
x.MaxMHz = v
|
||||
}
|
||||
if v, ok := nvNum(f[7]); ok && v > 0 {
|
||||
x.Temp, x.TempSrc = v, "gpu"
|
||||
}
|
||||
if v, ok := nvNum(f[8]); ok {
|
||||
x.PowerW = round1(v)
|
||||
}
|
||||
if v, ok := nvNum(f[9]); ok {
|
||||
x.PowerCapW = v
|
||||
}
|
||||
if len(f) > 10 {
|
||||
if v, ok := nvNum(f[10]); ok {
|
||||
x.FanPct = &v
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// parseNvidiaPmon reads `nvidia-smi pmon -c 1 -s um`: a header naming the columns, then one
|
||||
// row per process — "-" where a process has no share. Columns vary between driver versions,
|
||||
// so they are taken by name. Busiest first, like the fdinfo clients.
|
||||
func parseNvidiaPmon(out string) []GPUClient {
|
||||
col := map[string]int{}
|
||||
var clients []GPUClient
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) == 0 {
|
||||
continue
|
||||
}
|
||||
if f[0] == "#" {
|
||||
if len(f) > 1 && f[1] == "gpu" { // "# gpu pid type sm mem enc dec jpg ofa fb ccpm command"
|
||||
col = map[string]int{}
|
||||
for i, name := range f[1:] {
|
||||
col[name] = i
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
get := func(name string) (float64, bool) {
|
||||
i, ok := col[name]
|
||||
if !ok || i >= len(f) || f[i] == "-" {
|
||||
return 0, false
|
||||
}
|
||||
v, err := strconv.ParseFloat(f[i], 64)
|
||||
return v, err == nil
|
||||
}
|
||||
pid, ok := get("pid")
|
||||
if !ok || pid <= 0 {
|
||||
continue
|
||||
}
|
||||
cl := GPUClient{PID: int(pid), Engines: map[string]float64{}}
|
||||
if i, ok := col["command"]; ok && i < len(f) {
|
||||
cl.Name = strings.Join(f[i:], " ")
|
||||
}
|
||||
if v, ok := get("sm"); ok {
|
||||
cl.Engines["render"] = v
|
||||
}
|
||||
enc, okE := get("enc")
|
||||
dec, okD := get("dec")
|
||||
if okE || okD {
|
||||
cl.Engines["video"] = max(enc, dec)
|
||||
}
|
||||
for _, v := range cl.Engines {
|
||||
if v > cl.Usage {
|
||||
cl.Usage = v
|
||||
}
|
||||
}
|
||||
if len(cl.Engines) == 0 {
|
||||
cl.Engines = nil
|
||||
}
|
||||
if v, ok := get("fb"); ok {
|
||||
cl.Mem = uint64(v) << 20
|
||||
}
|
||||
clients = append(clients, cl)
|
||||
}
|
||||
sort.SliceStable(clients, func(i, j int) bool { return clients[i].Usage > clients[j].Usage })
|
||||
return clients
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A read that comes back late carries the last readings meanwhile and is used when it lands;
|
||||
// only one out past gpuStallAfter marks the card stalled, and its return clears that.
|
||||
func TestGPUReadLateThenStalledThenBack(t *testing.T) {
|
||||
defer func(w, s time.Duration) { gpuReadWithin, gpuStallAfter = w, s }(gpuReadWithin, gpuStallAfter)
|
||||
gpuReadWithin, gpuStallAfter = 20*time.Millisecond, 10*time.Second
|
||||
release := make(chan struct{})
|
||||
n := 0.0 // only the read goroutine writes it, one read at a time
|
||||
d := &gpuDev{name: "test", card: "card0"}
|
||||
d.readFn = func(x *GPU, _ time.Time) {
|
||||
n++
|
||||
if n == 2 {
|
||||
<-release
|
||||
}
|
||||
x.Usage = n * 10
|
||||
}
|
||||
t0 := time.Now()
|
||||
sample := func(at time.Duration) GPU {
|
||||
x := GPU{Name: "test"}
|
||||
d.readWithin(&x, t0.Add(at))
|
||||
return x
|
||||
}
|
||||
if x := sample(0); x.Usage != 10 || x.Stalled {
|
||||
t.Fatalf("first: %+v", x)
|
||||
}
|
||||
if x := sample(time.Second); x.Usage != 10 || x.Stalled { // late: the last readings
|
||||
t.Fatalf("late: %+v", x)
|
||||
}
|
||||
if x := sample(5 * time.Second); x.Usage != 10 || x.Stalled {
|
||||
t.Fatalf("still late: %+v", x)
|
||||
}
|
||||
if x := sample(12 * time.Second); !x.Stalled {
|
||||
t.Fatalf("out past gpuStallAfter, not stalled: %+v", x)
|
||||
}
|
||||
if x := sample(12*time.Second + 1); !x.Stalled {
|
||||
t.Fatal("stall did not hold")
|
||||
}
|
||||
close(release)
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
if x := sample(13 * time.Second); x.Usage != 20 || x.Stalled {
|
||||
t.Fatalf("back: %+v", x)
|
||||
}
|
||||
if x := sample(14 * time.Second); x.Usage != 30 || x.Stalled { // 30: no extra read ran while one was out
|
||||
t.Fatalf("next: %+v", x)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,233 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const fdinfoA = `pos: 0
|
||||
flags: 02100002
|
||||
drm-driver: i915
|
||||
drm-client-id: 544
|
||||
drm-pdev: 0000:00:02.0
|
||||
drm-total-system0: 52624 KiB
|
||||
drm-resident-system0: 50140 KiB
|
||||
drm-resident-stolen-system0: 32 KiB
|
||||
drm-engine-render: 1000000000 ns
|
||||
drm-engine-copy: 0 ns
|
||||
drm-engine-video: 2000000000 ns
|
||||
drm-engine-capacity-video: 2
|
||||
drm-engine-video-enhance: 0 ns
|
||||
`
|
||||
|
||||
func TestParseFdinfo(t *testing.T) {
|
||||
fi := parseFdinfo(fdinfoA)
|
||||
if fi.driver != "i915" || fi.pdev != "0000:00:02.0" || fi.client != "544" {
|
||||
t.Fatalf("identity: %+v", fi)
|
||||
}
|
||||
if fi.engines["video"] != 2000000000 || fi.capacity["video"] != 2 {
|
||||
t.Errorf("video engine: %+v", fi)
|
||||
}
|
||||
if fi.mem != (50140+32)<<10 {
|
||||
t.Errorf("resident = %d", fi.mem)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLspciName(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"Intel Corporation DG2 [Arc A380] (rev 05)": "Intel Arc A380",
|
||||
"Intel Corporation RocketLake-S GT1 [UHD Graphics 750] (rev 04)": "Intel UHD Graphics 750",
|
||||
"Intel Corporation Alder Lake-N [UHD Graphics]": "Intel UHD Graphics",
|
||||
"Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c7)": "AMD Radeon RX 6600/6600 XT/6600M",
|
||||
"NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)": "NVIDIA GeForce RTX 3060 Lite Hash Rate",
|
||||
"Moore Threads Technology Co.,Ltd MTT S80": "Moore Threads Technology Co.,Ltd MTT S80",
|
||||
} {
|
||||
if got := lspciName(in); got != want {
|
||||
t.Errorf("%q → %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Two scans a second apart: a client that ran its render engine for half of it and both
|
||||
// video engines flat out reads 50% render, 100% video, and the device's engines follow.
|
||||
func TestGPUClientsShare(t *testing.T) {
|
||||
c := NewCollector()
|
||||
d := &gpuDev{card: "/nonexistent/card9", dev: "/nonexistent", key: "0000:00:02.0", bus: "0000:00:02.0", driver: "i915", name: "Test GPU", integrated: true}
|
||||
c.gpu.devs = []*gpuDev{d}
|
||||
c.gpu.listedAt = time.Now().Add(time.Hour) // keep the fake list
|
||||
c.gpu.clients = map[string]clientPrev{}
|
||||
t0 := time.Now()
|
||||
c.gpu.last = []drmFD{{PID: 7, Comm: "ffmpeg", Info: fdinfoA}}
|
||||
c.gpu.scan = func() ([]drmFD, bool) { return c.gpu.last, true } // no walk: the fake list stands
|
||||
first := c.gpus(t0, 40)
|
||||
if len(first) != 1 || len(first[0].Clients) != 1 || first[0].Clients[0].Usage != 0 {
|
||||
t.Fatalf("first pass: %+v", first)
|
||||
}
|
||||
if first[0].Temp != 40 || first[0].TempSrc != "package" {
|
||||
t.Errorf("an integrated GPU wears the package temperature: %+v", first[0])
|
||||
}
|
||||
later := fdinfoA
|
||||
later = replace(later, "drm-engine-render: 1000000000 ns", "drm-engine-render: 1500000000 ns")
|
||||
later = replace(later, "drm-engine-video: 2000000000 ns", "drm-engine-video: 4000000000 ns")
|
||||
c.gpu.last = []drmFD{{PID: 7, Comm: "ffmpeg", Info: later}}
|
||||
got := c.gpus(t0.Add(time.Second), 40)
|
||||
cl := got[0].Clients[0]
|
||||
if cl.Engines["render"] != 50 || cl.Engines["video"] != 100 || cl.Usage != 100 {
|
||||
t.Errorf("client share: %+v", cl)
|
||||
}
|
||||
if got[0].Engines["video"] != 100 || got[0].Usage != 100 || got[0].UsageSrc != "clients" {
|
||||
t.Errorf("device from clients: %+v", got[0])
|
||||
}
|
||||
if got[0].MemUsed != cl.Mem {
|
||||
t.Errorf("resident memory should be the clients' sum: %d vs %d", got[0].MemUsed, cl.Mem)
|
||||
}
|
||||
// the Processes table joins the client to its row
|
||||
c.ring = []Sample{{T: 1, GPUs: got}}
|
||||
c.prevProc = nil
|
||||
table := c.Processes()
|
||||
if len(table.GPUs) != 1 {
|
||||
t.Errorf("table carries the GPUs: %+v", table.GPUs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFlattenGPU(t *testing.T) {
|
||||
s := Sample{T: 5, GPUs: []GPU{{Name: "Intel Arc A380", Driver: "i915", Bus: "0000:03:00.0", Usage: 12, UsageSrc: "idle", Engines: map[string]float64{"video": 30}, PowerW: 9.5, PowerCapW: 55, MHz: 2450, MaxMHz: 2450, Clients: []GPUClient{{PID: 1}}}}}
|
||||
got := map[string]any{}
|
||||
for _, x := range flatten(s, nil) {
|
||||
got[x.ID] = x.Value
|
||||
}
|
||||
for id, want := range map[string]any{"gpu.intel_arc_a380.usage": 12.0, "gpu.intel_arc_a380.video": 30.0, "gpu.intel_arc_a380.power": 9.5, "gpu.intel_arc_a380.freq": 2450.0, "gpu.intel_arc_a380.clients": 1.0} {
|
||||
if got[id] != want {
|
||||
t.Errorf("%s = %v, want %v", id, got[id], want)
|
||||
}
|
||||
}
|
||||
if _, ok := got["gpu.intel_arc_a380.temp"]; ok {
|
||||
t.Error("no temperature sensor, no temperature")
|
||||
}
|
||||
}
|
||||
|
||||
func replace(s, old, new string) string {
|
||||
for i := 0; i+len(old) <= len(s); i++ {
|
||||
if s[i:i+len(old)] == old {
|
||||
return s[:i] + new + s[i+len(old):]
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// TestLiveGPU reads this machine's GPUs for real: HAMLAB_LIVE=1 go test -run TestLiveGPU -v ./internal/metrics
|
||||
func TestLiveGPU(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_LIVE") == "" {
|
||||
t.Skip("set HAMLAB_LIVE=1 to read the real hardware")
|
||||
}
|
||||
c := NewCollector()
|
||||
c.sample()
|
||||
time.Sleep(1100 * time.Millisecond)
|
||||
c.sample()
|
||||
s, _ := c.Latest()
|
||||
b, _ := json.MarshalIndent(s.GPUs, "", " ")
|
||||
t.Logf("%s", b)
|
||||
for _, x := range flatten(s, func(id string) bool { return len(id) > 4 && id[:4] == "gpu." }) {
|
||||
t.Logf("%s = %v %s", x.ID, x.Value, x.Unit)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLiveARC: HAMLAB_LIVE=1 go test -run TestLiveARC -v ./internal/metrics
|
||||
func TestLiveARC(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_LIVE") == "" {
|
||||
t.Skip("set HAMLAB_LIVE=1 to read the real machine")
|
||||
}
|
||||
c := NewCollector()
|
||||
c.sample()
|
||||
time.Sleep(1100 * time.Millisecond)
|
||||
c.sample()
|
||||
s, _ := c.Latest()
|
||||
b, _ := json.MarshalIndent(s.ARC, "", " ")
|
||||
t.Logf("%s", b)
|
||||
for _, x := range flatten(s, func(id string) bool { return strings.HasPrefix(id, "zfs.arc.") }) {
|
||||
t.Logf("%s = %v %s", x.ID, x.Value, x.Unit)
|
||||
}
|
||||
}
|
||||
|
||||
// The embedded table names the lab's four GPUs the way lspci would, so a machine without
|
||||
// pciutils (the router) agrees with one that has it.
|
||||
func TestPCIName(t *testing.T) {
|
||||
for id, want := range map[string]string{"8086:3185": "Intel UHD Graphics 600", "8086:56a5": "Intel Arc A380", "8086:46d1": "Intel UHD Graphics", "8086:4c8a": "Intel UHD Graphics 750", "10de:2684": "NVIDIA GeForce RTX 4090"} {
|
||||
v, d := pciName(id[:4], id[5:])
|
||||
if d == "" {
|
||||
t.Errorf("%s: not in the table", id)
|
||||
continue
|
||||
}
|
||||
if got := lspciName(v + " " + d); got != want {
|
||||
t.Errorf("%s → %q, want %q", id, got, want)
|
||||
}
|
||||
}
|
||||
if _, d := pciName("8086", "ffff"); d != "" {
|
||||
t.Errorf("an unknown device should be empty, got %q", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseNvidiaQuery(t *testing.T) {
|
||||
var g GPU
|
||||
parseNvidiaQuery(&g, "31, 0, 12, 1488, 2048, 949, 1097, 57, 23.41, [N/A]\n")
|
||||
if g.Usage != 31 || g.UsageSrc != "busy" || g.Engines["render"] != 31 || g.Engines["video"] != 12 {
|
||||
t.Errorf("usage: %+v", g)
|
||||
}
|
||||
if g.MemUsed != 1488<<20 || g.MemTotal != 2048<<20 || g.MHz != 949 || g.MaxMHz != 1097 {
|
||||
t.Errorf("mem/clock: %+v", g)
|
||||
}
|
||||
if g.Temp != 57 || g.TempSrc != "gpu" || g.PowerW != 23.4 || g.PowerCapW != 0 {
|
||||
t.Errorf("temp/power (cap must stay 0 for [N/A]): %+v", g)
|
||||
}
|
||||
if g.FanPct != nil {
|
||||
t.Errorf("no fan field must leave the fan unset: %+v", g)
|
||||
}
|
||||
var fan, nofan GPU
|
||||
parseNvidiaQuery(&fan, "99, 0, 0, 20000, 24576, 1695, 2100, 83, 403.99, 420.00, 34\n")
|
||||
parseNvidiaQuery(&nofan, "5, 0, 0, 500, 2048, 300, 1097, 45, 8.0, [N/A], [N/A]\n")
|
||||
if fan.FanPct == nil || *fan.FanPct != 34 || nofan.FanPct != nil {
|
||||
t.Errorf("fan: %v, [N/A]: %v", fan.FanPct, nofan.FanPct)
|
||||
}
|
||||
var stopped GPU
|
||||
parseNvidiaQuery(&stopped, "0, 0, 0, 500, 24576, 210, 2100, 35, 20.0, 420.00, 0\n")
|
||||
if stopped.FanPct == nil || *stopped.FanPct != 0 {
|
||||
t.Errorf("a stopped fan must read 0, not nothing: %v", stopped.FanPct)
|
||||
}
|
||||
var short GPU
|
||||
parseNvidiaQuery(&short, "garbage")
|
||||
if short.UsageSrc != "" {
|
||||
t.Errorf("a short line must leave the GPU untouched: %+v", short)
|
||||
}
|
||||
}
|
||||
|
||||
const pmonA = `# gpu pid type sm mem enc dec jpg ofa fb ccpm command
|
||||
# Idx # C/G % % % % % % MB MB name
|
||||
0 1063 G - - - - - - 701 0 Xorg
|
||||
0 1480 G 27 22 - - - - 270 0 cinnamon
|
||||
0 13318 G 9 7 40 - - - 454 0 firefox-esr
|
||||
`
|
||||
|
||||
func TestParseNvidiaPmon(t *testing.T) {
|
||||
cl := parseNvidiaPmon(pmonA)
|
||||
if len(cl) != 3 {
|
||||
t.Fatalf("want 3 clients, got %+v", cl)
|
||||
}
|
||||
// busiest first: firefox's encoder share (40) beats cinnamon's sm (27)
|
||||
if cl[0].PID != 13318 || cl[0].Name != "firefox-esr" || cl[0].Usage != 40 || cl[0].Engines["render"] != 9 || cl[0].Engines["video"] != 40 || cl[0].Mem != 454<<20 {
|
||||
t.Errorf("firefox: %+v", cl[0])
|
||||
}
|
||||
if cl[1].PID != 1480 || cl[1].Usage != 27 || cl[1].Engines["render"] != 27 {
|
||||
t.Errorf("cinnamon: %+v", cl[1])
|
||||
}
|
||||
if cl[2].PID != 1063 || cl[2].Usage != 0 || cl[2].Engines != nil || cl[2].Mem != 701<<20 {
|
||||
t.Errorf("xorg (no shares, only memory): %+v", cl[2])
|
||||
}
|
||||
// an older driver without the jpg/ofa/ccpm columns
|
||||
old := "# gpu pid type sm mem enc dec fb command\n# Idx # C/G % % % % MB name\n 0 42 C 55 10 - - 100 python3\n"
|
||||
if cl := parseNvidiaPmon(old); len(cl) != 1 || cl[0].PID != 42 || cl[0].Usage != 55 || cl[0].Mem != 100<<20 || cl[0].Name != "python3" {
|
||||
t.Errorf("old layout: %+v", cl)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,47 @@
|
||||
package metrics
|
||||
|
||||
// A trimmed PCI ID database, embedded, so a GPU is named the same on every machine —
|
||||
// "Intel UHD Graphics 600" — whether or not that machine has pciutils. OpenWrt has neither
|
||||
// lspci nor pci.ids, and the router's GPU came up as "Intel GPU 3185" beside its twin's
|
||||
// "Intel UHD Graphics 600" until this. scripts/pci-ids-gpu.sh regenerates the file.
|
||||
|
||||
import (
|
||||
_ "embed"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
//go:embed pciids_gpu.txt
|
||||
var pciIDsGPU string
|
||||
|
||||
var pciTable struct {
|
||||
once sync.Once
|
||||
vendors map[string]string // "8086" → "Intel Corporation"
|
||||
devices map[string]string // "8086:3185" → "GeminiLake [UHD Graphics 600]"
|
||||
}
|
||||
|
||||
// pciName looks a vendor and device ID up (hex, lower case, no 0x): the vendor's name and
|
||||
// the device's; empty where unknown.
|
||||
func pciName(vendor, device string) (string, string) {
|
||||
pciTable.once.Do(func() {
|
||||
pciTable.vendors, pciTable.devices = map[string]string{}, map[string]string{}
|
||||
cur := ""
|
||||
for _, line := range strings.Split(pciIDsGPU, "\n") {
|
||||
switch {
|
||||
case line == "" || line[0] == '#':
|
||||
case line[0] == '\t':
|
||||
if cur == "" || len(line) < 7 || line[1] == '\t' {
|
||||
continue // a subsystem, or a device before any vendor
|
||||
}
|
||||
pciTable.devices[cur+":"+line[1:5]] = strings.TrimSpace(line[5:])
|
||||
default:
|
||||
if len(line) >= 6 {
|
||||
cur = line[:4]
|
||||
pciTable.vendors[cur] = strings.TrimSpace(line[4:])
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
vendor, device = strings.ToLower(vendor), strings.ToLower(device)
|
||||
return pciTable.vendors[vendor], pciTable.devices[vendor+":"+device]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,305 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
|
||||
"hamlab/internal/sensors"
|
||||
)
|
||||
|
||||
// The collector as a sensors.Provider: every number in a Sample, one sensor each. The
|
||||
// IDs are built from names the hardware reports (chip, label, interface, device, mount),
|
||||
// never from enumeration order like hwmon3, which moves between boots.
|
||||
|
||||
// flattener turns one Sample into sensors. With want set it builds only those, so walking
|
||||
// the whole ring for a history costs string compares, not allocations.
|
||||
type flattener struct {
|
||||
want func(id string) bool
|
||||
seen map[string]int
|
||||
out []sensors.Sensor
|
||||
}
|
||||
|
||||
type bounds struct{ min, max, warn, crit *float64 }
|
||||
|
||||
var pct = bounds{min: sensors.F(0), max: sensors.F(100)}
|
||||
|
||||
func upTo(max float64) bounds { return bounds{min: sensors.F(0), max: sensors.F(max)} }
|
||||
|
||||
func (f *flattener) add(id, name, kind, unit string, v any, t int64, b bounds, labels ...string) {
|
||||
f.seen[id]++
|
||||
if n := f.seen[id]; n > 1 { // two identical chips: the second is coretemp_2
|
||||
id = fmt.Sprintf("%s_%d", id, n)
|
||||
}
|
||||
if f.want != nil && !f.want(id) {
|
||||
return
|
||||
}
|
||||
s := sensors.Sensor{ID: id, Name: name, Kind: kind, Unit: unit, Value: v, T: t, Min: b.min, Max: b.max, Warn: b.warn, Crit: b.crit}
|
||||
if len(labels) > 0 {
|
||||
s.Labels = make(map[string]string, len(labels)/2)
|
||||
for i := 0; i+1 < len(labels); i += 2 {
|
||||
s.Labels[labels[i]] = labels[i+1]
|
||||
}
|
||||
}
|
||||
f.out = append(f.out, s)
|
||||
}
|
||||
|
||||
func ratio(part, whole uint64) float64 {
|
||||
if whole == 0 {
|
||||
return 0
|
||||
}
|
||||
return round1(float64(part) / float64(whole) * 100)
|
||||
}
|
||||
|
||||
func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
|
||||
f := &flattener{want: want, seen: map[string]int{}}
|
||||
t, none := s.T, bounds{}
|
||||
|
||||
f.add("cpu.usage", "CPU", "usage", "%", s.CPU, t, pct)
|
||||
for i, c := range s.Cores {
|
||||
n := strconv.Itoa(i)
|
||||
f.add("cpu.core."+n+".usage", "Core "+n, "usage", "%", c, t, pct, "core", n)
|
||||
}
|
||||
if c := cpuTemp(s.Temps); c > 0 {
|
||||
f.add("cpu.temp", "CPU temperature", "temperature", "°C", c, t, none)
|
||||
}
|
||||
if s.CPUPower > 0 {
|
||||
f.add("cpu.power", "CPU power", "power", "W", s.CPUPower, t, none)
|
||||
}
|
||||
for i, w := range s.CPUPowerPkgs {
|
||||
n := strconv.Itoa(i)
|
||||
f.add("cpu.package."+n+".power", "CPU "+strconv.Itoa(i+1)+" power", "power", "W", w, t, none, "package", n)
|
||||
}
|
||||
for i, w := range []string{"1", "5", "15"} {
|
||||
f.add("load."+w, "Load "+w+" min", "load", "", s.Load[i], t, none)
|
||||
}
|
||||
f.add("uptime", "Uptime", "duration", "s", s.Uptime, t, none)
|
||||
f.add("hamlab.sample_interval", "Sampling interval", "duration", "ms", float64(s.Interval), t, none)
|
||||
|
||||
for _, m := range []struct {
|
||||
id, name string
|
||||
v Mem
|
||||
}{{"mem", "Memory", s.Mem}, {"swap", "Swap", s.Swap}} {
|
||||
f.add(m.id+".total", m.name+" total", "bytes", "B", float64(m.v.Total), t, none)
|
||||
f.add(m.id+".used", m.name+" used", "bytes", "B", float64(m.v.Used), t, upTo(float64(m.v.Total)))
|
||||
f.add(m.id+".avail", m.name+" available", "bytes", "B", float64(m.v.Avail), t, upTo(float64(m.v.Total)))
|
||||
f.add(m.id+".used_pct", m.name+" used", "usage", "%", ratio(m.v.Used, m.v.Total), t, pct)
|
||||
}
|
||||
f.add("mem.free", "Memory free", "bytes", "B", float64(s.Mem.Free), t, upTo(float64(s.Mem.Total)))
|
||||
f.add("mem.cached", "Memory cached", "bytes", "B", float64(s.Mem.Cached), t, upTo(float64(s.Mem.Total)))
|
||||
f.add("mem.buffers", "Memory buffers", "bytes", "B", float64(s.Mem.Buffers), t, upTo(float64(s.Mem.Total)))
|
||||
|
||||
for _, x := range s.Temps {
|
||||
b := bounds{}
|
||||
if x.Max > 0 {
|
||||
b.warn = sensors.F(x.Max)
|
||||
}
|
||||
if x.Crit > 0 {
|
||||
b.crit = sensors.F(x.Crit)
|
||||
}
|
||||
labels := []string{"chip", x.Chip, "label", x.Label}
|
||||
if x.Chip == "coretemp" { // a two-socket board has a coretemp chip per CPU, both with a "Core 0"
|
||||
labels = append(labels, "package", strconv.Itoa(x.Pkg))
|
||||
}
|
||||
f.add(sensors.ID("temp", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "temperature", "°C", x.C, t, b, labels...)
|
||||
}
|
||||
for _, x := range s.Fans {
|
||||
f.add(sensors.ID("fan", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "fan", "rpm", float64(x.RPM), t, none, "chip", x.Chip, "label", x.Label)
|
||||
}
|
||||
|
||||
for _, x := range s.Net {
|
||||
p := "net." + sensors.Seg(x.Iface)
|
||||
f.add(p+".rx", x.Iface+" receive", "rate", "B/s", x.RxBps, t, none, "iface", x.Iface)
|
||||
f.add(p+".tx", x.Iface+" transmit", "rate", "B/s", x.TxBps, t, none, "iface", x.Iface)
|
||||
f.add(p+".rx_total", x.Iface+" received", "bytes", "B", float64(x.RxTotal), t, none, "iface", x.Iface)
|
||||
f.add(p+".tx_total", x.Iface+" transmitted", "bytes", "B", float64(x.TxTotal), t, none, "iface", x.Iface)
|
||||
f.add(p+".up", x.Iface+" link", "state", "", x.Up, t, none, "iface", x.Iface)
|
||||
}
|
||||
|
||||
for _, x := range s.Disks {
|
||||
p := "disk." + sensors.Seg(x.Name)
|
||||
f.add(p+".read", x.Name+" read", "rate", "B/s", x.ReadBps, t, none, "device", x.Name, "model", x.Model)
|
||||
f.add(p+".write", x.Name+" write", "rate", "B/s", x.WriteBps, t, none, "device", x.Name, "model", x.Model)
|
||||
f.add(p+".util", x.Name+" busy", "usage", "%", x.UtilPct, t, pct, "device", x.Name, "model", x.Model)
|
||||
f.add(p+".size", x.Name+" size", "bytes", "B", float64(x.Size), t, none, "device", x.Name, "model", x.Model)
|
||||
if x.Temp > 0 {
|
||||
f.add(p+".temp", x.Name+" temperature", "temperature", "°C", x.Temp, t, none, "device", x.Name, "model", x.Model)
|
||||
}
|
||||
}
|
||||
|
||||
for _, x := range s.FS {
|
||||
p := "fs." + sensors.Seg(x.Mount)
|
||||
// remote says the bytes are on another machine (a CIFS or NFS share), so a view of
|
||||
// this machine's hardware can leave it out without losing the sensor
|
||||
lbl := []string{"mount", x.Mount, "fstype", x.Type}
|
||||
if x.Remote {
|
||||
lbl = append(lbl, "remote", "1")
|
||||
}
|
||||
if x.Disk != "" {
|
||||
lbl = append(lbl, "disk", x.Disk)
|
||||
}
|
||||
if x.Source != "" {
|
||||
lbl = append(lbl, "source", x.Source)
|
||||
}
|
||||
f.add(p+".size", x.Mount+" size", "bytes", "B", float64(x.Size), s.fsT, none, lbl...)
|
||||
f.add(p+".used", x.Mount+" used", "bytes", "B", float64(x.Used), s.fsT, upTo(float64(x.Size)), lbl...)
|
||||
f.add(p+".used_pct", x.Mount+" used", "usage", "%", ratio(x.Used, x.Size), s.fsT, pct, lbl...)
|
||||
}
|
||||
|
||||
gpuSeen := map[string]int{}
|
||||
for _, g := range s.GPUs {
|
||||
// two identical cards are gpu.<name> and gpu.<name>_2, every sensor of the second under
|
||||
// its own stem (not add's suffix on the end of each id, which the page can't find)
|
||||
p := "gpu." + sensors.Seg(g.Name)
|
||||
if gpuSeen[p]++; gpuSeen[p] > 1 {
|
||||
p = fmt.Sprintf("%s_%d", p, gpuSeen[p])
|
||||
}
|
||||
lbl := []string{"name", g.Name, "driver", g.Driver}
|
||||
if g.Bus != "" {
|
||||
lbl = append(lbl, "bus", g.Bus)
|
||||
}
|
||||
if g.Integrated {
|
||||
lbl = append(lbl, "integrated", "1")
|
||||
}
|
||||
if g.UsageSrc != "" {
|
||||
f.add(p+".usage", g.Name, "usage", "%", g.Usage, t, pct, lbl...)
|
||||
}
|
||||
for _, e := range []string{"render", "video", "copy", "video-enhance", "compute"} {
|
||||
if v, ok := g.Engines[e]; ok {
|
||||
f.add(p+"."+sensors.Seg(e), g.Name+" "+e, "usage", "%", v, t, pct, lbl...)
|
||||
}
|
||||
}
|
||||
if g.Temp > 0 {
|
||||
f.add(p+".temp", g.Name+" temperature", "temperature", "°C", g.Temp, t, none, append(lbl, "source", g.TempSrc)...)
|
||||
}
|
||||
if g.PowerW > 0 || g.PowerCapW > 0 {
|
||||
b := none
|
||||
if g.PowerCapW > 0 {
|
||||
b = upTo(g.PowerCapW)
|
||||
}
|
||||
f.add(p+".power", g.Name+" power", "power", "W", g.PowerW, t, b, lbl...)
|
||||
}
|
||||
if g.FanRPM > 0 {
|
||||
f.add(p+".fan", g.Name+" fan", "fan", "rpm", float64(g.FanRPM), t, none, lbl...)
|
||||
}
|
||||
if g.FanPct != nil {
|
||||
f.add(p+".fan_pct", g.Name+" fan", "fan", "%", *g.FanPct, t, pct, lbl...)
|
||||
}
|
||||
if g.MaxMHz > 0 {
|
||||
f.add(p+".freq", g.Name+" clock", "frequency", "MHz", g.MHz, t, upTo(g.MaxMHz), lbl...)
|
||||
}
|
||||
if g.MemTotal > 0 {
|
||||
f.add(p+".mem_used", g.Name+" memory used", "bytes", "B", float64(g.MemUsed), t, upTo(float64(g.MemTotal)), lbl...)
|
||||
f.add(p+".mem_total", g.Name+" memory", "bytes", "B", float64(g.MemTotal), t, none, lbl...)
|
||||
} else if g.MemUsed > 0 {
|
||||
f.add(p+".mem_used", g.Name+" memory used", "bytes", "B", float64(g.MemUsed), t, none, lbl...)
|
||||
}
|
||||
f.add(p+".clients", g.Name+" clients", "count", "", float64(len(g.Clients)), t, none, lbl...)
|
||||
}
|
||||
|
||||
if a := s.ARC; a != nil {
|
||||
p := "zfs.arc."
|
||||
cap := upTo(float64(a.Max))
|
||||
f.add(p+"size", "ARC size", "bytes", "B", float64(a.Size), t, cap)
|
||||
f.add(p+"target", "ARC target", "bytes", "B", float64(a.Target), t, cap)
|
||||
f.add(p+"min", "ARC floor", "bytes", "B", float64(a.Min), t, none)
|
||||
f.add(p+"max", "ARC ceiling", "bytes", "B", float64(a.Max), t, none)
|
||||
f.add(p+"mru", "ARC recently used", "bytes", "B", float64(a.MRU), t, cap)
|
||||
f.add(p+"mfu", "ARC frequently used", "bytes", "B", float64(a.MFU), t, cap)
|
||||
f.add(p+"metadata", "ARC metadata", "bytes", "B", float64(a.Metadata), t, cap)
|
||||
f.add(p+"data", "ARC data", "bytes", "B", float64(a.Data), t, cap)
|
||||
if a.HitPctKnown {
|
||||
f.add(p+"hit_pct", "ARC hit rate", "usage", "%", a.HitPct, t, pct)
|
||||
}
|
||||
f.add(p+"reads", "ARC reads", "rate", "/s", a.ReadsPerSec, t, none)
|
||||
f.add(p+"misses", "ARC misses", "rate", "/s", a.MissesPerSec, t, none)
|
||||
f.add(p+"lifetime_hit_pct", "ARC hit rate since boot", "usage", "%", a.LifetimeHitPct, t, pct)
|
||||
f.add(p+"no_grow", "ARC held by memory pressure", "state", "", a.NoGrow, t, none)
|
||||
if a.Compressed > 0 {
|
||||
f.add(p+"compress_ratio", "ARC compression", "ratio", "x", round1(float64(a.Uncompressed)/float64(a.Compressed)*10)/10, t, none)
|
||||
}
|
||||
if a.L2Size > 0 {
|
||||
f.add(p+"l2_size", "L2ARC size", "bytes", "B", float64(a.L2Size), t, none)
|
||||
f.add(p+"l2_hit_pct", "L2ARC hit rate", "usage", "%", a.L2HitPct, t, pct)
|
||||
}
|
||||
}
|
||||
|
||||
if w := s.ZFSWrite; w != nil {
|
||||
f.add("zfs.write.dirty", "ZFS write buffer", "bytes", "B", float64(w.Dirty), t, upTo(float64(w.Limit)))
|
||||
f.add("zfs.write.limit", "ZFS write buffer limit", "bytes", "B", float64(w.Limit), t, none)
|
||||
f.add("zfs.write.delayed", "ZFS writes throttled", "rate", "/s", w.DelayedPerSec, t, none)
|
||||
for _, p := range w.Pools {
|
||||
pp := "zfs." + sensors.Seg(p.Pool) + "."
|
||||
f.add(pp+"write_bps", p.Pool+" writes to disk", "rate", "B/s", p.WriteBps, t, none, "pool", p.Pool)
|
||||
f.add(pp+"logical_bps", p.Pool+" writes by programs", "rate", "B/s", p.LogicalBps, t, none, "pool", p.Pool)
|
||||
if l := p.Landing; l != nil {
|
||||
lp := pp + "landing."
|
||||
lbl := []string{"pool", p.Pool, "branch", l.Branch}
|
||||
f.add(lp+"pending_bytes", p.Pool+" landing: waiting", "bytes", "B", float64(l.PendingBytes), t, upTo(float64(l.Size)), lbl...)
|
||||
f.add(lp+"pending_files", p.Pool+" landing: files waiting", "count", "", float64(l.PendingFiles), t, none, lbl...)
|
||||
if l.Size > 0 {
|
||||
f.add(lp+"used_pct", p.Pool+" landing: used", "usage", "%", round1(float64(l.Used)/float64(l.Size)*100), t, pct, lbl...)
|
||||
}
|
||||
f.add(lp+"in_bps", p.Pool+" landing: arriving", "rate", "B/s", l.InBps, t, none, lbl...)
|
||||
f.add(lp+"drain_bps", p.Pool+" landing: moving to the pool", "rate", "B/s", l.DrainBps, t, none, lbl...)
|
||||
if m := l.Mover; m != nil {
|
||||
f.add(lp+"mover", p.Pool+" landing: mover", "state", "", m.State, t, none, lbl...)
|
||||
f.add(lp+"queue", p.Pool+" landing: mover queue", "count", "", float64(m.Queue), t, none, lbl...)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if u := s.UPS; u != nil {
|
||||
ut := s.upsT
|
||||
f.add("ups.battery", "UPS battery", "usage", "%", u.BatteryPct, ut, pct, "model", u.Model)
|
||||
f.add("ups.runtime", "UPS runtime", "duration", "min", u.RuntimeMin, ut, none, "model", u.Model)
|
||||
f.add("ups.load", "UPS load", "power", "W", u.LoadW, ut, none, "model", u.Model)
|
||||
f.add("ups.load_pct", "UPS load", "usage", "%", u.LoadPct, ut, pct, "model", u.Model)
|
||||
f.add("ups.in_v", "UPS input", "voltage", "V", u.InV, ut, none, "model", u.Model)
|
||||
f.add("ups.out_v", "UPS output", "voltage", "V", u.OutV, ut, none, "model", u.Model)
|
||||
f.add("ups.on_battery", "UPS on battery", "state", "", u.OnBattery, ut, none, "model", u.Model)
|
||||
f.add("ups.state", "UPS state", "state", "", u.State, ut, none, "model", u.Model)
|
||||
f.add("ups.source", "UPS power source", "state", "", u.Source, ut, none, "model", u.Model)
|
||||
f.add("ups.last_event", "UPS last event", "state", "", u.LastEvent, ut, none, "model", u.Model, "driver", u.Driver)
|
||||
}
|
||||
return f.out
|
||||
}
|
||||
|
||||
// Sensors implements sensors.Provider.
|
||||
func (c *Collector) Sensors() []sensors.Sensor {
|
||||
s, ok := c.Latest()
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
return flatten(s, nil)
|
||||
}
|
||||
|
||||
// SensorHistories implements sensors.Historian: the ring, seen through the sensors that
|
||||
// match. A sensor that didn't exist in a sample (a disk plugged in since) simply has a
|
||||
// shorter series; nothing is zero-filled. States have no curve and are left out.
|
||||
func (c *Collector) SensorHistories(match func(id string) bool) []sensors.Series {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
by := map[string]*sensors.Series{}
|
||||
var order []string
|
||||
for _, smp := range c.ring {
|
||||
for _, x := range flatten(smp, match) {
|
||||
v, ok := x.Value.(float64)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
s := by[x.ID]
|
||||
if s == nil {
|
||||
s = &sensors.Series{ID: x.ID, Unit: x.Unit, T: []int64{}, V: []float64{}}
|
||||
by[x.ID] = s
|
||||
order = append(order, x.ID)
|
||||
}
|
||||
s.T, s.V = append(s.T, smp.T), append(s.V, v)
|
||||
}
|
||||
}
|
||||
out := make([]sensors.Series, 0, len(order))
|
||||
for _, id := range order {
|
||||
out = append(out, *by[id])
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package metrics
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestFlatten(t *testing.T) {
|
||||
s := Sample{T: 1000, CPU: 10, Cores: []float64{5, 15}, Mem: Mem{Total: 200, Used: 50},
|
||||
Temps: []Temp{{Chip: "nvme", Label: "Composite", C: 40, Crit: 85}, {Chip: "nvme", Label: "Composite", C: 45}},
|
||||
Fans: []Fan{{Chip: "nct6775", Label: "fan2", RPM: 1100}},
|
||||
FS: []FS{{Mount: "/", Size: 100, Used: 25}}, fsT: 900}
|
||||
got := map[string]any{}
|
||||
for _, x := range flatten(s, nil) {
|
||||
if _, dup := got[x.ID]; dup {
|
||||
t.Errorf("duplicate id %s", x.ID)
|
||||
}
|
||||
got[x.ID] = x.Value
|
||||
}
|
||||
for id, want := range map[string]any{"cpu.usage": 10.0, "cpu.core.1.usage": 15.0, "mem.used_pct": 25.0, "temp.nvme.composite": 40.0,
|
||||
"temp.nvme.composite_2": 45.0, "fan.nct6775.fan2": 1100.0, "fs.root.used_pct": 25.0} {
|
||||
if got[id] != want {
|
||||
t.Errorf("%s = %v, want %v", id, got[id], want)
|
||||
}
|
||||
}
|
||||
one := flatten(s, func(id string) bool { return id == "temp.nvme.composite_2" })
|
||||
if len(one) != 1 || one[0].Value != 45.0 {
|
||||
t.Errorf("only: got %+v", one)
|
||||
}
|
||||
for _, x := range flatten(s, func(id string) bool { return id == "fs.root.used" }) {
|
||||
if x.T != 900 {
|
||||
t.Errorf("fs sensor should carry the fs read time, got %d", x.T)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSensorHistories(t *testing.T) {
|
||||
c := NewCollector()
|
||||
c.ring = []Sample{{T: 1, CPU: 10}, {T: 2, CPU: 20, Net: []Net{{Iface: "eth0", Up: true, RxBps: 5}}}}
|
||||
got := map[string][]float64{}
|
||||
for _, h := range c.SensorHistories(func(id string) bool { return id == "cpu.usage" || id == "net.eth0.rx" || id == "net.eth0.up" }) {
|
||||
if len(h.T) != len(h.V) {
|
||||
t.Errorf("%s: t and v differ in length", h.ID)
|
||||
}
|
||||
got[h.ID] = h.V
|
||||
}
|
||||
if len(got["cpu.usage"]) != 2 || got["cpu.usage"][1] != 20 {
|
||||
t.Errorf("cpu: %v", got["cpu.usage"])
|
||||
}
|
||||
if len(got["net.eth0.rx"]) != 1 {
|
||||
t.Errorf("a sensor that appeared later has a shorter history, got %v", got["net.eth0.rx"])
|
||||
}
|
||||
if _, ok := got["net.eth0.up"]; ok {
|
||||
t.Error("a state has no history")
|
||||
}
|
||||
if len(c.SensorHistories(func(string) bool { return false })) != 0 {
|
||||
t.Error("nothing matched, nothing returned")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,456 @@
|
||||
package metrics
|
||||
|
||||
// The ZFS write path, for the ZFS tab: what is waiting in RAM to be written, how fast each
|
||||
// pool is writing to its disks, and — where a pool has one — the landing tier in front of it.
|
||||
//
|
||||
// A landing tier is a mergerfs pool whose first branches are fast non-ZFS storage and whose
|
||||
// last branch is a ZFS dataset: new files land on the fast branch and a mover drains them onto
|
||||
// the pool. It is detected from mergerfs's own branch list (the user.mergerfs.branches xattr on
|
||||
// the pool's .mergerfs control file), so it shows wherever one exists and nowhere else. A mover
|
||||
// that writes a status file into /run/landing-mover/ (hamlet's scripts/landing-mover.py does)
|
||||
// adds what it is doing: its queue, the file in flight, totals and the last error.
|
||||
//
|
||||
// Sources, all cheap enough for every sample:
|
||||
// - arcstats anon_size: dirty data in RAM, for all pools together (ZFS has no live per-pool
|
||||
// figure; txgs only records ndirty once a txg has synced)
|
||||
// - dmu_tx: dmu_tx_dirty_delay / dmu_tx_dirty_over_max — writers slowed by the write throttle,
|
||||
// or stopped at the hard limit
|
||||
// - <pool>/txgs: nwritten per synced txg — bytes the pool actually wrote to its vdevs
|
||||
// - <pool>/objset-*: nwritten per dataset — bytes programs wrote, before compression
|
||||
// The landing branch's file walk is slower and runs every landingEvery.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
zfsKstatDir = "/proc/spl/kstat/zfs/"
|
||||
zfsParamDir = "/sys/module/zfs/parameters/"
|
||||
moverStateDir = "/run/landing-mover"
|
||||
landingEvery = 10 * time.Second
|
||||
landingWalkMax = 200000 // entries; past this the counts are marked partial
|
||||
)
|
||||
|
||||
// ZFSWrites is the write path as one sample sees it. Nil without the ZFS module.
|
||||
type ZFSWrites struct {
|
||||
Dirty uint64 `json:"dirty"` // bytes in RAM written but not yet on disk, all pools
|
||||
DirtyMax uint64 `json:"dirty_max"` // zfs_dirty_data_max
|
||||
DelayMinPct int `json:"delay_min_pct"` // zfs_delay_min_dirty_percent: the throttle starts here
|
||||
Limit uint64 `json:"limit"` // what the buffer can really hold now: the smaller of dirty_max and half the ARC target
|
||||
DelayedPerSec float64 `json:"delayed_per_sec"` // writes the throttle slowed, per second over the interval
|
||||
OverMaxPerSec float64 `json:"over_max_per_sec"` // writes stopped dead at the limit
|
||||
TxgTimeoutSec int `json:"txg_timeout_sec"` // zfs_txg_timeout
|
||||
Pools []PoolWrites `json:"pools"`
|
||||
}
|
||||
|
||||
// PoolWrites is one pool.
|
||||
type PoolWrites struct {
|
||||
Pool string `json:"pool"`
|
||||
WriteBps float64 `json:"write_bps"` // to its vdevs, after compression, parity included
|
||||
LogicalBps float64 `json:"logical_bps"` // into its datasets, as programs wrote it
|
||||
Landing *Landing `json:"landing,omitempty"`
|
||||
}
|
||||
|
||||
// Landing is a landing tier in front of a pool.
|
||||
type Landing struct {
|
||||
Mount string `json:"mount"` // the mergerfs pool, e.g. /srv/tank
|
||||
Branch string `json:"branch"` // the landing branch, e.g. /srv/landing/tank
|
||||
PoolBranch string `json:"pool_branch"`
|
||||
Device string `json:"device,omitempty"` // block device under the branch, as in Sample.Disks
|
||||
FSType string `json:"fstype,omitempty"`
|
||||
Size uint64 `json:"size"`
|
||||
Used uint64 `json:"used"`
|
||||
PendingFiles int `json:"pending_files"`
|
||||
PendingBytes uint64 `json:"pending_bytes"`
|
||||
PendingPartial bool `json:"pending_partial,omitempty"` // the walk stopped at landingWalkMax
|
||||
OldestMs int64 `json:"oldest_ms,omitempty"` // mtime of the oldest waiting file
|
||||
CountedMs int64 `json:"counted_ms"` // when the walk ran
|
||||
InBps float64 `json:"in_bps"` // the landing device's writes
|
||||
DrainBps float64 `json:"drain_bps"` // the mover's progress onto the pool
|
||||
Mover *Mover `json:"mover,omitempty"`
|
||||
}
|
||||
|
||||
// Mover is what a landing mover says about itself in its status file.
|
||||
type Mover struct {
|
||||
State string `json:"state"` // idle | moving | holding (a mount it needs is missing)
|
||||
Src string `json:"src"`
|
||||
Dst string `json:"dst"`
|
||||
PID int `json:"pid"`
|
||||
StartedMs int64 `json:"started_ms"`
|
||||
UpdatedMs int64 `json:"updated_ms"`
|
||||
Stale bool `json:"stale"` // not updated for a while: stopped or hung
|
||||
Queue int `json:"queue"`
|
||||
Settle float64 `json:"settle"`
|
||||
BWLimit string `json:"bwlimit"`
|
||||
BatchFiles int `json:"batch_files"`
|
||||
BatchBytes uint64 `json:"batch_bytes"`
|
||||
BatchStartMs int64 `json:"batch_start_ms"`
|
||||
BatchFirst string `json:"batch_first"`
|
||||
Current string `json:"current"` // the file being copied, when it can tell
|
||||
CurrentSize uint64 `json:"current_size"` // its full size
|
||||
Partial uint64 `json:"partial"` // bytes of it already on the pool
|
||||
MovedFiles uint64 `json:"moved_files"` // since the mover started
|
||||
MovedBytes uint64 `json:"moved_bytes"`
|
||||
LastMove string `json:"last_move"`
|
||||
LastMoveMs int64 `json:"last_move_ms"`
|
||||
LastError string `json:"last_error"`
|
||||
LastErrorMs int64 `json:"last_error_ms"`
|
||||
}
|
||||
|
||||
type zfsWritePrev struct {
|
||||
at time.Time
|
||||
delay uint64
|
||||
overMax uint64
|
||||
txg map[string]uint64 // pool -> highest synced txg counted
|
||||
txgAt map[string]time.Time // pool -> when that txg was first seen
|
||||
txgRate map[string]float64 // pool -> bytes/s over the last gap between syncs
|
||||
logical map[string]uint64 // pool -> sum of objset nwritten
|
||||
drained map[string]uint64 // landing branch -> moved + partial bytes, at the mover's last update
|
||||
drainedAt map[string]int64 // landing branch -> that update's time, unix ms
|
||||
drainRate map[string]float64 // landing branch -> bytes/s between its last two updates
|
||||
landing map[string]Landing // branch -> last walk
|
||||
landingAt time.Time
|
||||
}
|
||||
|
||||
func readParamU(name string) uint64 {
|
||||
v, _ := strconv.ParseUint(strings.TrimSpace(readStr(zfsParamDir+name)), 10, 64)
|
||||
return v
|
||||
}
|
||||
|
||||
// poolsWithKstats lists the imported pools: the directories under the kstat tree.
|
||||
func poolsWithKstats() []string {
|
||||
ents, err := os.ReadDir(zfsKstatDir)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, e := range ents {
|
||||
if e.IsDir() {
|
||||
if _, err := os.Stat(zfsKstatDir + e.Name() + "/txgs"); err == nil {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// parseTxgs reads a pool's txgs kstat and returns, for every committed txg above `after`, the
|
||||
// bytes it wrote, plus the highest committed txg seen.
|
||||
func parseTxgs(text string, after uint64) (written, top uint64) {
|
||||
top = after
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
f := strings.Fields(line)
|
||||
// txg birth state ndirty nread nwritten reads writes otime qtime wtime stime
|
||||
if len(f) < 12 || f[2] != "C" {
|
||||
continue
|
||||
}
|
||||
txg, err := strconv.ParseUint(f[0], 10, 64)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if txg > top {
|
||||
top = txg
|
||||
}
|
||||
if txg > after {
|
||||
n, _ := strconv.ParseUint(f[5], 10, 64)
|
||||
written += n
|
||||
}
|
||||
}
|
||||
return written, top
|
||||
}
|
||||
|
||||
// objsetWritten sums nwritten over a pool's datasets.
|
||||
func objsetWritten(pool string) uint64 {
|
||||
m, _ := filepath.Glob(zfsKstatDir + pool + "/objset-*")
|
||||
var t uint64
|
||||
for _, p := range m {
|
||||
for _, line := range strings.Split(readStr(p), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) == 3 && f[0] == "nwritten" {
|
||||
v, _ := strconv.ParseUint(f[2], 10, 64)
|
||||
t += v
|
||||
}
|
||||
}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// mountEntry is one line of /proc/mounts.
|
||||
type mountEntry struct{ src, point, fstype string }
|
||||
|
||||
func readMounts() []mountEntry {
|
||||
var out []mountEntry
|
||||
for _, line := range strings.Split(readStr("/proc/mounts"), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) >= 3 {
|
||||
out = append(out, mountEntry{src: unescapeMount(f[0]), point: unescapeMount(f[1]), fstype: f[2]})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// unescapeMount undoes /proc/mounts' octal escapes (\040 for a space, and so on).
|
||||
func unescapeMount(s string) string {
|
||||
if !strings.Contains(s, `\`) {
|
||||
return s
|
||||
}
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(s); i++ {
|
||||
if s[i] == '\\' && i+3 < len(s) {
|
||||
if v, err := strconv.ParseUint(s[i+1:i+4], 8, 8); err == nil {
|
||||
b.WriteByte(byte(v))
|
||||
i += 3
|
||||
continue
|
||||
}
|
||||
}
|
||||
b.WriteByte(s[i])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// mountFor is the mount a path lives on: the longest mount point that is a prefix of it.
|
||||
func mountFor(ms []mountEntry, path string) (mountEntry, bool) {
|
||||
var best mountEntry
|
||||
found := false
|
||||
for _, m := range ms {
|
||||
if m.point == path || m.point == "/" || strings.HasPrefix(path, strings.TrimSuffix(m.point, "/")+"/") {
|
||||
if !found || len(m.point) > len(best.point) {
|
||||
best, found = m, true
|
||||
}
|
||||
}
|
||||
}
|
||||
return best, found
|
||||
}
|
||||
|
||||
// parseBranches reads mergerfs's branch list: "/srv/landing/tank=RW:/tank=RW,50G".
|
||||
func parseBranches(v string) (paths []string, modes []string) {
|
||||
for _, b := range strings.Split(strings.TrimSpace(v), ":") {
|
||||
if b == "" {
|
||||
continue
|
||||
}
|
||||
p, mode, _ := strings.Cut(b, "=")
|
||||
mode, _, _ = strings.Cut(mode, ",")
|
||||
if mode == "" {
|
||||
mode = "RW"
|
||||
}
|
||||
paths, modes = append(paths, p), append(modes, mode)
|
||||
}
|
||||
return paths, modes
|
||||
}
|
||||
|
||||
func getxattr(path, name string) string {
|
||||
buf := make([]byte, 4096)
|
||||
n, err := syscall.Getxattr(path, name, buf)
|
||||
if err != nil || n <= 0 {
|
||||
return ""
|
||||
}
|
||||
return string(buf[:n])
|
||||
}
|
||||
|
||||
// findLandings maps pool -> the landing tiers in front of it.
|
||||
func findLandings(ms []mountEntry) map[string][]Landing {
|
||||
out := map[string][]Landing{}
|
||||
for _, m := range ms {
|
||||
if m.fstype != "fuse.mergerfs" {
|
||||
continue
|
||||
}
|
||||
paths, modes := parseBranches(getxattr(filepath.Join(m.point, ".mergerfs"), "user.mergerfs.branches"))
|
||||
// the pool branch is the first ZFS one; landing branches are writable non-ZFS ones before it
|
||||
for i, p := range paths {
|
||||
bm, ok := mountFor(ms, p)
|
||||
if !ok || bm.fstype != "zfs" {
|
||||
continue
|
||||
}
|
||||
pool, _, _ := strings.Cut(bm.src, "/")
|
||||
for j := 0; j < i; j++ {
|
||||
lm, ok := mountFor(ms, paths[j])
|
||||
if !ok || lm.fstype == "zfs" || modes[j] == "RO" {
|
||||
continue
|
||||
}
|
||||
out[pool] = append(out[pool], Landing{Mount: m.point, Branch: paths[j], PoolBranch: p, FSType: lm.fstype, Device: blockDisk(lm.src)})
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// blockDisk turns a mount source into the name Sample.Disks uses: /dev/nvme0n1p1 -> nvme0,
|
||||
// /dev/sdb2 -> sdb. Empty for anything that is not a plain block device.
|
||||
func blockDisk(src string) string {
|
||||
dev, err := filepath.EvalSymlinks(src)
|
||||
if err != nil || !strings.HasPrefix(dev, "/dev/") {
|
||||
return ""
|
||||
}
|
||||
name := filepath.Base(dev)
|
||||
// a partition's sysfs entry sits inside its disk's
|
||||
if p, err := filepath.EvalSymlinks("/sys/class/block/" + name); err == nil {
|
||||
if _, err := os.Stat(p + "/partition"); err == nil {
|
||||
name = filepath.Base(filepath.Dir(p))
|
||||
}
|
||||
}
|
||||
if m := nsRe.FindStringSubmatch(name); m != nil {
|
||||
return m[1] // nvme0n1 -> nvme0, as foldNVMe names it
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
// walkLanding counts what is waiting on a landing branch.
|
||||
func walkLanding(l *Landing, now time.Time) {
|
||||
var st syscall.Statfs_t
|
||||
if syscall.Statfs(l.Branch, &st) == nil {
|
||||
l.Size = st.Blocks * uint64(st.Bsize)
|
||||
l.Used = (st.Blocks - st.Bfree) * uint64(st.Bsize)
|
||||
}
|
||||
seen := 0
|
||||
l.PendingFiles, l.PendingBytes, l.OldestMs, l.PendingPartial = 0, 0, 0, false
|
||||
_ = filepath.WalkDir(l.Branch, func(p string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
if seen++; seen > landingWalkMax {
|
||||
l.PendingPartial = true
|
||||
return filepath.SkipAll
|
||||
}
|
||||
if !d.Type().IsRegular() {
|
||||
return nil
|
||||
}
|
||||
if fi, err := d.Info(); err == nil {
|
||||
l.PendingFiles++
|
||||
l.PendingBytes += uint64(fi.Size())
|
||||
if ms := fi.ModTime().UnixMilli(); l.OldestMs == 0 || ms < l.OldestMs {
|
||||
l.OldestMs = ms
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
l.CountedMs = now.UnixMilli()
|
||||
}
|
||||
|
||||
// readMovers loads every mover status file, keyed by the branch it drains.
|
||||
func readMovers(now time.Time) map[string]*Mover {
|
||||
out := map[string]*Mover{}
|
||||
m, _ := filepath.Glob(moverStateDir + "/*.json")
|
||||
for _, p := range m {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var mv Mover
|
||||
if json.Unmarshal(b, &mv) != nil || mv.Src == "" {
|
||||
continue
|
||||
}
|
||||
mv.Stale = now.UnixMilli()-mv.UpdatedMs > 30_000
|
||||
if mv.Stale {
|
||||
mv.State = "stopped"
|
||||
}
|
||||
out[filepath.Clean(mv.Src)] = &mv
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// zfsWrites samples the write path; nil when this machine has no ZFS module loaded.
|
||||
func (c *Collector) zfsWrites(now time.Time, arc *ARC, disks []Disk) *ZFSWrites {
|
||||
if arc == nil {
|
||||
return nil
|
||||
}
|
||||
p := &c.prevZW
|
||||
dt := 0.0
|
||||
if !p.at.IsZero() {
|
||||
dt = now.Sub(p.at).Seconds()
|
||||
}
|
||||
w := &ZFSWrites{Dirty: arc.Anon, DirtyMax: readParamU("zfs_dirty_data_max"),
|
||||
DelayMinPct: int(readParamU("zfs_delay_min_dirty_percent")), TxgTimeoutSec: int(readParamU("zfs_txg_timeout")), Pools: []PoolWrites{}}
|
||||
w.Limit = w.DirtyMax
|
||||
if half := arc.Target / 2; half > 0 && half < w.Limit {
|
||||
w.Limit = half
|
||||
}
|
||||
tx := readKstat(zfsKstatDir + "dmu_tx")
|
||||
if dt > 0 && tx["dmu_tx_dirty_delay"] >= p.delay && tx["dmu_tx_dirty_over_max"] >= p.overMax {
|
||||
w.DelayedPerSec = round1(float64(tx["dmu_tx_dirty_delay"]-p.delay) / dt)
|
||||
w.OverMaxPerSec = round1(float64(tx["dmu_tx_dirty_over_max"]-p.overMax) / dt)
|
||||
}
|
||||
p.delay, p.overMax = tx["dmu_tx_dirty_delay"], tx["dmu_tx_dirty_over_max"]
|
||||
if p.txg == nil {
|
||||
p.txg, p.logical, p.drained, p.landing = map[string]uint64{}, map[string]uint64{}, map[string]uint64{}, map[string]Landing{}
|
||||
p.txgAt, p.txgRate, p.drainedAt, p.drainRate = map[string]time.Time{}, map[string]float64{}, map[string]int64{}, map[string]float64{}
|
||||
}
|
||||
|
||||
ms := readMounts()
|
||||
landings := findLandings(ms)
|
||||
walk := now.Sub(p.landingAt) >= landingEvery
|
||||
if walk {
|
||||
p.landingAt = now
|
||||
}
|
||||
movers := readMovers(now)
|
||||
diskW := map[string]float64{}
|
||||
for _, d := range disks {
|
||||
diskW[d.Name] = d.WriteBps
|
||||
}
|
||||
|
||||
for _, pool := range poolsWithKstats() {
|
||||
pw := PoolWrites{Pool: pool}
|
||||
// A txg records what it wrote only when it has synced — every few seconds, not every
|
||||
// sample — so the rate is taken over the gap between syncs and held until the next one.
|
||||
prevTop, had := p.txg[pool]
|
||||
written, top := parseTxgs(readStr(zfsKstatDir+pool+"/txgs"), prevTop)
|
||||
switch {
|
||||
case !had:
|
||||
p.txgAt[pool] = now
|
||||
case top > prevTop:
|
||||
if gap := now.Sub(p.txgAt[pool]).Seconds(); gap > 0 {
|
||||
p.txgRate[pool] = round1(float64(written) / gap)
|
||||
}
|
||||
p.txgAt[pool] = now
|
||||
case now.Sub(p.txgAt[pool]) > time.Duration(2*max(w.TxgTimeoutSec, 5))*time.Second:
|
||||
p.txgRate[pool] = 0 // no sync for two timeouts: nothing is being written
|
||||
}
|
||||
pw.WriteBps = p.txgRate[pool]
|
||||
p.txg[pool] = top
|
||||
lw := objsetWritten(pool)
|
||||
if prev, ok := p.logical[pool]; ok && dt > 0 && lw >= prev {
|
||||
pw.LogicalBps = round1(float64(lw-prev) / dt)
|
||||
}
|
||||
p.logical[pool] = lw
|
||||
|
||||
if ls := landings[pool]; len(ls) > 0 {
|
||||
l := ls[0] // one landing tier per pool is what the view shows
|
||||
if last, ok := p.landing[l.Branch]; ok && !walk {
|
||||
l.Size, l.Used, l.PendingFiles, l.PendingBytes = last.Size, last.Used, last.PendingFiles, last.PendingBytes
|
||||
l.OldestMs, l.PendingPartial, l.CountedMs = last.OldestMs, last.PendingPartial, last.CountedMs
|
||||
} else {
|
||||
walkLanding(&l, now)
|
||||
p.landing[l.Branch] = l
|
||||
}
|
||||
l.InBps = diskW[l.Device]
|
||||
if mv := movers[filepath.Clean(l.Branch)]; mv != nil {
|
||||
l.Mover = mv
|
||||
// the mover writes its status every couple of seconds: measure between its updates
|
||||
done := mv.MovedBytes + mv.Partial
|
||||
if at, ok := p.drainedAt[l.Branch]; !ok || mv.UpdatedMs != at {
|
||||
if ok && mv.UpdatedMs > at && done >= p.drained[l.Branch] {
|
||||
p.drainRate[l.Branch] = round1(float64(done-p.drained[l.Branch]) / (float64(mv.UpdatedMs-at) / 1000))
|
||||
}
|
||||
p.drained[l.Branch], p.drainedAt[l.Branch] = done, mv.UpdatedMs
|
||||
}
|
||||
if mv.State != "moving" {
|
||||
p.drainRate[l.Branch] = 0
|
||||
}
|
||||
l.DrainBps = p.drainRate[l.Branch]
|
||||
}
|
||||
pw.Landing = &l
|
||||
}
|
||||
w.Pools = append(w.Pools, pw)
|
||||
}
|
||||
p.at = now
|
||||
return w
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
package metrics
|
||||
|
||||
import "testing"
|
||||
|
||||
// real rows from grid's /proc/spl/kstat/zfs/tank/txgs, 2026-10-25
|
||||
const txgsSample = `81 1 0x01 26 7072 20029594359 13953814329135
|
||||
txg birth state ndirty nread nwritten reads writes otime qtime wtime stime
|
||||
15812833 13936408791659 C 0 0 0 0 0 5119980242 2040 58385 309580
|
||||
15812834 13941528771901 C 0 0 0 0 0 5119948815 8471 65826 112612
|
||||
15812835 13946648720716 C 885760 0 2293760 0 367 5120100556 1704 56762 271264374
|
||||
15812836 13951768821272 S 4194304 0 0 0 0 5120000000 1000 50000 0
|
||||
15812837 13956888821272 O 0 0 0 0 0 0 0 0 0
|
||||
`
|
||||
|
||||
func TestParseTxgs(t *testing.T) {
|
||||
w, top := parseTxgs(txgsSample, 0)
|
||||
if w != 2293760 || top != 15812835 {
|
||||
t.Fatalf("from 0: written %d top %d", w, top)
|
||||
}
|
||||
// only txgs synced since the last look count, and an open or syncing txg never does
|
||||
w, top = parseTxgs(txgsSample, 15812834)
|
||||
if w != 2293760 || top != 15812835 {
|
||||
t.Fatalf("after 15812834: written %d top %d", w, top)
|
||||
}
|
||||
w, top = parseTxgs(txgsSample, 15812835)
|
||||
if w != 0 || top != 15812835 {
|
||||
t.Fatalf("nothing new: written %d top %d", w, top)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBranches(t *testing.T) {
|
||||
p, m := parseBranches("/srv/landing/tank=RW:/tank=NC,50G:/mnt/old=RO\n")
|
||||
if len(p) != 3 || p[0] != "/srv/landing/tank" || p[1] != "/tank" || p[2] != "/mnt/old" {
|
||||
t.Fatalf("paths %q", p)
|
||||
}
|
||||
if m[0] != "RW" || m[1] != "NC" || m[2] != "RO" {
|
||||
t.Fatalf("modes %q", m)
|
||||
}
|
||||
if p, _ := parseBranches("/a:/b"); len(p) != 2 {
|
||||
t.Fatalf("bare paths %q", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMountFor(t *testing.T) {
|
||||
ms := []mountEntry{
|
||||
{"/dev/sdj2", "/", "ext4"},
|
||||
{"/dev/nvme0n1p1", "/srv/landing", "ext4"},
|
||||
{"tank", "/tank", "zfs"},
|
||||
{"tank/media", "/tank/media", "zfs"},
|
||||
{"tank-landing", "/srv/tank", "fuse.mergerfs"},
|
||||
}
|
||||
cases := map[string]string{
|
||||
"/srv/landing/tank": "/srv/landing", "/tank": "/tank", "/tank/media/x": "/tank/media",
|
||||
"/tankfoo": "/", "/srv/landingx": "/",
|
||||
}
|
||||
for path, want := range cases {
|
||||
if m, ok := mountFor(ms, path); !ok || m.point != want {
|
||||
t.Errorf("%s: got %q want %q", path, m.point, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnescapeMount(t *testing.T) {
|
||||
if got := unescapeMount(`/mnt/my\040disk`); got != "/mnt/my disk" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFoldZFSWrites(t *testing.T) {
|
||||
mk := func(dirty uint64, bps, in, drain float64) Sample {
|
||||
return Sample{ZFSWrite: &ZFSWrites{Dirty: dirty, Limit: 100, Pools: []PoolWrites{{Pool: "tank", WriteBps: bps,
|
||||
Landing: &Landing{Branch: "/srv/landing/tank", InBps: in, DrainBps: drain, Mover: &Mover{State: "moving"}}}}}}
|
||||
}
|
||||
b := []Sample{mk(10, 100, 400, 0), mk(30, 300, 0, 60)}
|
||||
got := fold(b).ZFSWrite
|
||||
if got.Dirty != 20 || got.Pools[0].WriteBps != 200 || got.Pools[0].Landing.InBps != 200 || got.Pools[0].Landing.DrainBps != 30 {
|
||||
t.Fatalf("folded %+v / %+v", got, got.Pools[0].Landing)
|
||||
}
|
||||
// the fold must not write through to the newest sample it copied from
|
||||
if b[1].ZFSWrite.Pools[0].Landing.InBps != 0 || b[1].ZFSWrite.Dirty != 30 {
|
||||
t.Fatalf("fold changed its input")
|
||||
}
|
||||
if got.Pools[0].Landing.Mover.State != "moving" {
|
||||
t.Fatalf("mover state lost")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,360 @@
|
||||
// Package network is what the Network tab shows: the machine's interfaces as the kernel has
|
||||
// them, how the internet looks from out there, and a speed test that measures the line.
|
||||
//
|
||||
// Everything here is read from /proc and /sys, or measured against Cloudflare's public speed
|
||||
// endpoints — no tool to install, on any of the machines. That matters for the router, where
|
||||
// flash is small and `opkg install` is a decision.
|
||||
package network
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Addr is one address on an interface.
|
||||
type Addr struct {
|
||||
IP string `json:"ip"`
|
||||
Prefix int `json:"prefix"` // network bits, as in /24
|
||||
V6 bool `json:"v6,omitempty"`
|
||||
}
|
||||
|
||||
// Iface is one network interface: what it is, whether it is up, and what it has moved.
|
||||
type Iface struct {
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"` // ethernet | wifi | bridge | tunnel | loopback | virtual
|
||||
MAC string `json:"mac,omitempty"`
|
||||
MTU int `json:"mtu"`
|
||||
// Up is the administrative state, Carrier whether a cable (or an association) is actually
|
||||
// there. Up without Carrier is the usual "it's plugged into nothing".
|
||||
Up bool `json:"up"`
|
||||
Carrier bool `json:"carrier"`
|
||||
// SpeedMbps is what the link negotiated, where the driver says: 0 for wireless, virtual
|
||||
// and tunnel devices, which have no such number.
|
||||
SpeedMbps int `json:"speed_mbps,omitempty"`
|
||||
Duplex string `json:"duplex,omitempty"`
|
||||
Driver string `json:"driver,omitempty"`
|
||||
SignalDbm *float64 `json:"signal_dbm,omitempty"` // wireless only, from /proc/net/wireless
|
||||
Addrs []Addr `json:"addrs"`
|
||||
RxTotal uint64 `json:"rx_total"`
|
||||
TxTotal uint64 `json:"tx_total"`
|
||||
// Default marks the interface the default route leaves by — the one that matters when
|
||||
// the question is about the internet.
|
||||
Default bool `json:"default,omitempty"`
|
||||
}
|
||||
|
||||
// Report is the machine's network as it stands.
|
||||
type Report struct {
|
||||
At time.Time `json:"at"`
|
||||
Host string `json:"host,omitempty"`
|
||||
Ifaces []Iface `json:"ifaces"`
|
||||
Gateway string `json:"gateway,omitempty"`
|
||||
GatewayVia string `json:"gateway_via,omitempty"` // the interface it is reached through
|
||||
DNS []string `json:"dns,omitempty"`
|
||||
Search []string `json:"search,omitempty"`
|
||||
}
|
||||
|
||||
func readStr(path string) string {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
func readInt(path string) int {
|
||||
n, err := strconv.Atoi(readStr(path))
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// kindOf names what sort of device this is, from what the kernel exposes about it rather
|
||||
// than from its name — a bridge is a bridge whatever it is called.
|
||||
func kindOf(name string, flags net.Flags) string {
|
||||
sys := "/sys/class/net/" + name + "/"
|
||||
switch {
|
||||
case flags&net.FlagLoopback != 0:
|
||||
return "loopback"
|
||||
case dirExists(sys + "wireless"), dirExists(sys + "phy80211"):
|
||||
return "wifi"
|
||||
case dirExists(sys + "bridge"):
|
||||
return "bridge"
|
||||
case dirExists(sys + "bonding"):
|
||||
return "bond"
|
||||
}
|
||||
// DEVTYPE says vlan, bridge, wlan… for the devices that have one
|
||||
for _, line := range strings.Split(readStr(sys+"uevent"), "\n") {
|
||||
if v, ok := strings.CutPrefix(line, "DEVTYPE="); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
if !dirExists(sys + "device") { // no bus device behind it: veth, tun, wireguard, docker's own
|
||||
return "virtual"
|
||||
}
|
||||
return "ethernet"
|
||||
}
|
||||
|
||||
func dirExists(p string) bool {
|
||||
st, err := os.Stat(p)
|
||||
return err == nil && st.IsDir()
|
||||
}
|
||||
|
||||
// driverOf is the module behind an interface, which is what names the hardware in one word.
|
||||
func driverOf(name string) string {
|
||||
l, err := os.Readlink("/sys/class/net/" + name + "/device/driver")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return l[strings.LastIndexByte(l, '/')+1:]
|
||||
}
|
||||
|
||||
// wirelessSignal is the signal level of each wireless interface, in dBm.
|
||||
func wirelessSignal() map[string]float64 {
|
||||
out := map[string]float64{}
|
||||
f, err := os.Open("/proc/net/wireless")
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
name, rest, ok := strings.Cut(sc.Text(), ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
fs := strings.Fields(rest)
|
||||
if len(fs) < 3 {
|
||||
continue
|
||||
}
|
||||
if v, err := strconv.ParseFloat(strings.TrimSuffix(fs[2], "."), 64); err == nil {
|
||||
out[strings.TrimSpace(name)] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// counters is every interface's byte totals, from the same file the sampler reads.
|
||||
func counters() map[string][2]uint64 {
|
||||
out := map[string][2]uint64{}
|
||||
f, err := os.Open("/proc/net/dev")
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
name, rest, ok := strings.Cut(sc.Text(), ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
fs := strings.Fields(rest)
|
||||
if len(fs) < 16 {
|
||||
continue
|
||||
}
|
||||
rx, _ := strconv.ParseUint(fs[0], 10, 64)
|
||||
tx, _ := strconv.ParseUint(fs[8], 10, 64)
|
||||
out[strings.TrimSpace(name)] = [2]uint64{rx, tx}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// defaultRoute is the gateway the machine sends the internet to, and the interface it
|
||||
// leaves by. Read from /proc/net/route, whose addresses are little-endian hex.
|
||||
func defaultRoute() (gw, via string) {
|
||||
f, err := os.Open("/proc/net/route")
|
||||
if err != nil {
|
||||
return "", ""
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
best := -1
|
||||
for sc.Scan() {
|
||||
fs := strings.Fields(sc.Text())
|
||||
if len(fs) < 8 || fs[0] == "Iface" {
|
||||
continue
|
||||
}
|
||||
if fs[1] != "00000000" { // not a default route
|
||||
continue
|
||||
}
|
||||
metric, _ := strconv.Atoi(fs[6])
|
||||
if best >= 0 && metric >= best {
|
||||
continue
|
||||
}
|
||||
ip, err := hexIP(fs[2])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
best, gw, via = metric, ip, fs[0]
|
||||
}
|
||||
return gw, via
|
||||
}
|
||||
|
||||
// hexIP reads one address out of /proc/net/route, where each is printed as a 32-bit word in
|
||||
// the host's byte order: 10.0.0.1 appears as "0100000A" on a little-endian machine. So the
|
||||
// text is parsed as a plain number, then written back out in native order to recover the four
|
||||
// bytes — which is right on either sort of machine, without asking which this is.
|
||||
func hexIP(h string) (string, error) {
|
||||
v, err := strconv.ParseUint(h, 16, 32)
|
||||
if err != nil || len(h) != 8 {
|
||||
return "", fmt.Errorf("not an address: %q", h)
|
||||
}
|
||||
var b [4]byte
|
||||
binary.NativeEndian.PutUint32(b[:], uint32(v))
|
||||
return net.IPv4(b[0], b[1], b[2], b[3]).String(), nil
|
||||
}
|
||||
|
||||
// resolvers is what /etc/resolv.conf points at.
|
||||
func resolvers() (servers, search []string) {
|
||||
f, err := os.Open("/etc/resolv.conf")
|
||||
if err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
fs := strings.Fields(sc.Text())
|
||||
if len(fs) < 2 || strings.HasPrefix(fs[0], "#") {
|
||||
continue
|
||||
}
|
||||
switch fs[0] {
|
||||
case "nameserver":
|
||||
servers = append(servers, fs[1])
|
||||
case "search", "domain":
|
||||
search = append(search, fs[1:]...)
|
||||
}
|
||||
}
|
||||
return servers, search
|
||||
}
|
||||
|
||||
// Interfaces is every interface the kernel has, with its addresses and totals.
|
||||
func Interfaces() []Iface {
|
||||
list, err := net.Interfaces()
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
_, via := defaultRoute()
|
||||
sig, cnt := wirelessSignal(), counters()
|
||||
out := make([]Iface, 0, len(list))
|
||||
for _, in := range list {
|
||||
sys := "/sys/class/net/" + in.Name + "/"
|
||||
f := Iface{
|
||||
Name: in.Name, Kind: kindOf(in.Name, in.Flags), MTU: in.MTU,
|
||||
MAC: in.HardwareAddr.String(), Up: in.Flags&net.FlagUp != 0,
|
||||
Driver: driverOf(in.Name), Default: in.Name == via, Addrs: []Addr{},
|
||||
}
|
||||
f.Carrier = readStr(sys+"carrier") == "1"
|
||||
if f.Kind == "loopback" {
|
||||
f.Carrier = f.Up
|
||||
}
|
||||
if sp := readInt(sys + "speed"); sp > 0 { // -1 when the driver doesn't know
|
||||
f.SpeedMbps = sp
|
||||
}
|
||||
if d := readStr(sys + "duplex"); d != "" && d != "unknown" {
|
||||
f.Duplex = d
|
||||
}
|
||||
if v, ok := sig[in.Name]; ok {
|
||||
f.SignalDbm = &v
|
||||
}
|
||||
if c, ok := cnt[in.Name]; ok {
|
||||
f.RxTotal, f.TxTotal = c[0], c[1]
|
||||
}
|
||||
addrs, _ := in.Addrs()
|
||||
for _, a := range addrs {
|
||||
n, ok := a.(*net.IPNet)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
ones, _ := n.Mask.Size()
|
||||
f.Addrs = append(f.Addrs, Addr{IP: n.IP.String(), Prefix: ones, V6: n.IP.To4() == nil})
|
||||
}
|
||||
out = append(out, f)
|
||||
}
|
||||
// the interface the internet leaves by first, then the ones carrying addresses, then the rest
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
a, b := out[i], out[j]
|
||||
if a.Default != b.Default {
|
||||
return a.Default
|
||||
}
|
||||
ai, bi := len(a.Addrs) > 0 && a.Kind != "loopback", len(b.Addrs) > 0 && b.Kind != "loopback"
|
||||
if ai != bi {
|
||||
return ai
|
||||
}
|
||||
return a.Name < b.Name
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// Look is the machine's network now. Cheap: a handful of small files.
|
||||
func Look(host string) Report {
|
||||
gw, via := defaultRoute()
|
||||
dns, search := resolvers()
|
||||
return Report{At: time.Now(), Host: host, Ifaces: Interfaces(), Gateway: gw, GatewayVia: via, DNS: dns, Search: search}
|
||||
}
|
||||
|
||||
// ---- how the internet sees this machine ----
|
||||
|
||||
// Outside is what the far end of the line says about this connection: the address the
|
||||
// internet sees, and where it came ashore. Refreshed on its own clock, because the public
|
||||
// address here is a DHCP lease and moves without warning.
|
||||
type Outside struct {
|
||||
At time.Time `json:"at"`
|
||||
IP string `json:"ip,omitempty"`
|
||||
Colo string `json:"colo,omitempty"` // the Cloudflare edge that answered — "SEA"
|
||||
Country string `json:"country,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Watcher keeps the outside view fresh and holds the speed tester.
|
||||
type Watcher struct {
|
||||
mu sync.RWMutex
|
||||
out Outside
|
||||
Test *Tester
|
||||
}
|
||||
|
||||
const outsideEvery = 15 * time.Minute
|
||||
|
||||
func NewWatcher(onResult func(Result)) *Watcher {
|
||||
w := &Watcher{Test: NewTester(onResult)}
|
||||
go func() {
|
||||
for {
|
||||
w.refresh()
|
||||
time.Sleep(outsideEvery)
|
||||
}
|
||||
}()
|
||||
return w
|
||||
}
|
||||
|
||||
func (w *Watcher) refresh() {
|
||||
o := probeOutside()
|
||||
w.mu.Lock()
|
||||
// a failed probe does not throw away a good answer; it is noted beside it
|
||||
if o.IP == "" && w.out.IP != "" {
|
||||
w.out.Error, w.out.At = o.Error, o.At
|
||||
} else {
|
||||
w.out = o
|
||||
}
|
||||
w.mu.Unlock()
|
||||
}
|
||||
|
||||
// Outside is the last known view from outside, asking again if it has gone stale.
|
||||
func (w *Watcher) Outside() Outside {
|
||||
w.mu.RLock()
|
||||
o := w.out
|
||||
w.mu.RUnlock()
|
||||
if time.Since(o.At) > outsideEvery {
|
||||
w.refresh()
|
||||
w.mu.RLock()
|
||||
o = w.out
|
||||
w.mu.RUnlock()
|
||||
}
|
||||
return o
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
package network
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestHexIP(t *testing.T) {
|
||||
// /proc/net/route prints each address as a host-order word: on a little-endian machine
|
||||
// 10.0.0.1 is written "0100000A". These cases are that machine, which is all of ours.
|
||||
for hex, want := range map[string]string{
|
||||
"0100000A": "10.0.0.1",
|
||||
"00000000": "0.0.0.0",
|
||||
"0101A8C0": "192.168.1.1",
|
||||
} {
|
||||
got, err := hexIP(hex)
|
||||
if err != nil || got != want {
|
||||
t.Errorf("hexIP(%q) = %q, %v; want %q", hex, got, err, want)
|
||||
}
|
||||
}
|
||||
if _, err := hexIP("nonsense"); err == nil {
|
||||
t.Error("hexIP accepted nonsense")
|
||||
}
|
||||
}
|
||||
|
||||
func TestServerTimeIsNotLatency(t *testing.T) {
|
||||
h := http.Header{}
|
||||
h.Add("Server-Timing", "cfSpeedEdge;dur=7, cfSpeedWorker;dur=36")
|
||||
h.Add("Server-Timing", `cfL4;desc="?proto=TCP&rtt=3505&min_rtt=2856"`)
|
||||
if got := serverMs(h); got != 43 {
|
||||
t.Errorf("serverMs = %v, want 43 (7+36, and nothing from the cfL4 description)", got)
|
||||
}
|
||||
if got := serverMs(http.Header{}); got != 0 {
|
||||
t.Errorf("serverMs with no header = %v, want 0", got)
|
||||
}
|
||||
h2 := http.Header{"Cf-Ray": {"a41f6720f966a36c-SEA"}, "Cf-Meta-Ip": {"50.47.252.221"}}
|
||||
if m := readMeta(h2); m.Colo != "SEA" || m.IP != "50.47.252.221" {
|
||||
t.Errorf("readMeta = %+v, want the edge SEA and the address", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRates(t *testing.T) {
|
||||
// a gibibyte in eight seconds is 1073.7 Mbps
|
||||
if got := mbps(1<<30, 8); math.Abs(got-1073.7) > 0.1 {
|
||||
t.Errorf("mbps(1GiB, 8s) = %v, want ~1073.7", got)
|
||||
}
|
||||
if got := best([]float64{12.5, 9.2, 40, 9.9}); got != 9.2 {
|
||||
t.Errorf("best = %v, want 9.2 (the least of them)", got)
|
||||
}
|
||||
if got := jitter([]float64{10, 10, 10}); got != 0 {
|
||||
t.Errorf("jitter of a steady line = %v, want 0", got)
|
||||
}
|
||||
if got := jitter([]float64{10}); got != 0 {
|
||||
t.Errorf("jitter of one sample = %v, want 0", got)
|
||||
}
|
||||
if got := jitter([]float64{5, 50, 5, 50}); got != 45 {
|
||||
t.Errorf("jitter of a line alternating by 45 = %v, want 45", got)
|
||||
}
|
||||
// one slow probe among steady ones moves it a little, not enormously
|
||||
if got := jitter([]float64{10, 10, 10, 60, 10, 10}); got > 25 {
|
||||
t.Errorf("jitter with a single outlier = %v, want it not to run away", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The body an upload sends is a fixed size, counts itself, and costs nothing to make.
|
||||
func TestZerosBodyCountsItself(t *testing.T) {
|
||||
var n atomic.Int64
|
||||
z := &zeros{left: 100, n: &n}
|
||||
buf := make([]byte, 30)
|
||||
total := 0
|
||||
for {
|
||||
got, err := z.Read(buf)
|
||||
total += got
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
if total != 100 || n.Load() != 100 {
|
||||
t.Errorf("read %d bytes and counted %d, want 100 and 100", total, n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
// Interfaces has to work on whatever machine this runs on, and always name a loopback.
|
||||
func TestInterfacesHasLoopback(t *testing.T) {
|
||||
ifs := Interfaces()
|
||||
if len(ifs) == 0 {
|
||||
t.Fatal("no interfaces at all")
|
||||
}
|
||||
lo := false
|
||||
for _, f := range ifs {
|
||||
if f.Kind == "loopback" {
|
||||
lo = true
|
||||
if !f.Up || len(f.Addrs) == 0 {
|
||||
t.Errorf("loopback %s looks wrong: up=%v addrs=%v", f.Name, f.Up, f.Addrs)
|
||||
}
|
||||
}
|
||||
if f.Name == "" {
|
||||
t.Error("an interface with no name")
|
||||
}
|
||||
}
|
||||
if !lo {
|
||||
t.Error("no loopback among the interfaces")
|
||||
}
|
||||
}
|
||||
|
||||
// The real thing, against the real endpoint: skipped by default, because `make test` must
|
||||
// not need the internet. HAMLAB_NET_TEST=1 go test ./internal/network/ -run Live -v
|
||||
func TestLiveSpeedTest(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_NET_TEST") == "" {
|
||||
t.Skip("set HAMLAB_NET_TEST=1 to measure the actual line")
|
||||
}
|
||||
tr := NewTester(nil)
|
||||
if _, err := tr.Start(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tr.Start(); err == nil {
|
||||
t.Error("a second test started while the first was running")
|
||||
}
|
||||
deadline := time.Now().Add(90 * time.Second)
|
||||
for tr.Running() && time.Now().Before(deadline) {
|
||||
time.Sleep(250 * time.Millisecond)
|
||||
}
|
||||
st := tr.State()
|
||||
if st.Result == nil {
|
||||
t.Fatal("no result")
|
||||
}
|
||||
r := *st.Result
|
||||
t.Logf("down %s up %s latency %.1f ms jitter %.1f ms via %s ip %s %.0f MB moved in %.1fs",
|
||||
Mbits(r.DownMbps), Mbits(r.UpMbps), r.LatencyMs, r.JitterMs, r.Colo, r.IP,
|
||||
float64(r.DownBytes+r.UpBytes)/1e6, r.TookSec)
|
||||
if r.Error != "" {
|
||||
t.Fatalf("test failed: %s", r.Error)
|
||||
}
|
||||
if r.DownMbps <= 0 || r.UpMbps <= 0 || r.LatencyMs <= 0 {
|
||||
t.Errorf("a figure came back zero: %+v", r)
|
||||
}
|
||||
if r.IP == "" {
|
||||
t.Error("no public address reported")
|
||||
}
|
||||
if len(tr.History()) != 1 {
|
||||
t.Errorf("history has %d results, want 1", len(tr.History()))
|
||||
}
|
||||
}
|
||||
|
||||
// A phase measured against a local server, at a rate the test controls. This is the part of
|
||||
// the speed test with arithmetic in it — the window after the ramp, the byte cap that ends it
|
||||
// early, and the rate that comes out — and none of it needs the internet to check.
|
||||
func TestPhaseMeasuresTheWindow(t *testing.T) {
|
||||
const perTick = 1 << 20 // a mebibyte every 5 ms ≈ 1678 Mbps, whatever the machine is doing
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/up" {
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
w.WriteHeader(204)
|
||||
return
|
||||
}
|
||||
buf := make([]byte, perTick)
|
||||
fl, _ := w.(http.Flusher)
|
||||
for {
|
||||
if _, err := w.Write(buf); err != nil {
|
||||
return
|
||||
}
|
||||
if fl != nil {
|
||||
fl.Flush()
|
||||
}
|
||||
select {
|
||||
case <-time.After(5 * time.Millisecond):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
defer func(up string, w, d, mw time.Duration, m int64) {
|
||||
upURL, warmUp, downFor, minWindow, measureBytes = up, w, d, mw, m
|
||||
}(upURL, warmUp, downFor, minWindow, measureBytes)
|
||||
upURL = srv.URL + "/up"
|
||||
warmUp, downFor, minWindow, measureBytes = 100*time.Millisecond, 3*time.Second, 200*time.Millisecond, 8<<20
|
||||
here := Source{"local", func(int) string { return srv.URL + "/down" }}
|
||||
|
||||
tr := NewTester(nil)
|
||||
began := time.Now()
|
||||
mbps, moved, note, err := tr.phase(context.Background(), srv.Client(), "download", 2, downFor, 0, 1, here)
|
||||
took := time.Since(began)
|
||||
if err != nil {
|
||||
t.Fatalf("phase: %v", err)
|
||||
}
|
||||
if note != "" {
|
||||
t.Errorf("note = %q, want none — nothing went wrong", note)
|
||||
}
|
||||
// The cap must have ended it early rather than the clock: the phase was allowed three
|
||||
// seconds and should be done in well under one, having moved at least the cap.
|
||||
if moved < measureBytes {
|
||||
t.Errorf("moved %d bytes, want at least the %d-byte cap", moved, measureBytes)
|
||||
}
|
||||
if took > time.Second {
|
||||
t.Errorf("phase took %v of the %v it was allowed; the byte cap should have ended it", took.Round(time.Millisecond), downFor)
|
||||
}
|
||||
// The server hands out a mebibyte per 5 ms per stream, so the rate is in the hundreds of
|
||||
// Mbps. The point is that it is a real quotient of what moved over how long, not that it
|
||||
// hits a particular figure on a loaded test machine.
|
||||
if mbps < 50 || mbps > 20000 {
|
||||
t.Errorf("mbps = %v, which is not a believable quotient", mbps)
|
||||
}
|
||||
t.Logf("measured %.0f Mbps over %.1f MiB", mbps, float64(moved)/(1<<20))
|
||||
|
||||
// A phase whose server refuses outright fails, and says what it was told.
|
||||
bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(429) }))
|
||||
defer bad.Close()
|
||||
refuser := Source{"refuser", func(int) string { return bad.URL + "/down" }}
|
||||
_, _, _, err = tr.phase(context.Background(), bad.Client(), "download", 2, 300*time.Millisecond, 0, 1, refuser)
|
||||
if err == nil {
|
||||
t.Error("a phase against a server that refuses everything should fail")
|
||||
} else if !refused(err) {
|
||||
t.Errorf("error = %q, want the refusal that makes the fallback kick in", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Stopping a test mid-flight is reported as cancelled, not as whatever the streams said on
|
||||
// their way out.
|
||||
func TestCancelIsReportedAsCancelled(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
buf := make([]byte, 1<<16)
|
||||
for {
|
||||
if _, err := w.Write(buf); err != nil {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-time.After(time.Millisecond):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
defer func(w, d time.Duration) { warmUp, downFor = w, d }(warmUp, downFor)
|
||||
warmUp, downFor = 10*time.Millisecond, 10*time.Second
|
||||
here := Source{"local", func(int) string { return srv.URL + "/down" }}
|
||||
|
||||
tr := NewTester(nil)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
go func() { time.Sleep(200 * time.Millisecond); cancel() }()
|
||||
_, _, _, err := tr.phase(ctx, srv.Client(), "download", 1, downFor, 0, 1, here)
|
||||
// Cancelled after bytes moved: the figure stands, and the phase does not claim an error.
|
||||
if err != nil && !errors.Is(err, context.Canceled) {
|
||||
t.Errorf("err = %v, want nil or a cancellation", err)
|
||||
}
|
||||
}
|
||||
|
||||
// When the first source refuses, the download is taken from the next one and the result says
|
||||
// so. This is the whole reason there is more than one.
|
||||
func TestDownloadFallsBackWhenRefused(t *testing.T) {
|
||||
serving := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/up" {
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
w.WriteHeader(204)
|
||||
return
|
||||
}
|
||||
// paced, so the phase is governed by its window and not by a loopback moving a
|
||||
// gibibyte before the first tick
|
||||
buf := make([]byte, 1<<20)
|
||||
fl, _ := w.(http.Flusher)
|
||||
for i := 0; i < 64; i++ {
|
||||
if _, err := w.Write(buf); err != nil {
|
||||
return
|
||||
}
|
||||
if fl != nil {
|
||||
fl.Flush()
|
||||
}
|
||||
select {
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer serving.Close()
|
||||
refusing := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(429) }))
|
||||
defer refusing.Close()
|
||||
|
||||
defer func(srcs []Source, up string, w, d, mw time.Duration, m int64) {
|
||||
downSources, upURL, warmUp, downFor, minWindow, measureBytes = srcs, up, w, d, mw, m
|
||||
}(downSources, upURL, warmUp, downFor, minWindow, measureBytes)
|
||||
downSources = []Source{
|
||||
{"refuser", func(int) string { return refusing.URL + "/down" }},
|
||||
{"the-other-one", func(int) string { return serving.URL + "/down" }},
|
||||
}
|
||||
upURL = serving.URL + "/up"
|
||||
warmUp, downFor, upFor, minWindow, measureBytes = 50*time.Millisecond, time.Second, 300*time.Millisecond, 100*time.Millisecond, 4<<20
|
||||
|
||||
tr := NewTester(nil)
|
||||
if _, err := tr.Start(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
for tr.Running() && time.Now().Before(deadline) {
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
res := tr.State().Result
|
||||
if res == nil {
|
||||
t.Fatal("no result")
|
||||
}
|
||||
if res.Source != "the-other-one" {
|
||||
t.Errorf("source = %q, want the second one — the first refused", res.Source)
|
||||
}
|
||||
if res.DownMbps <= 0 {
|
||||
t.Errorf("download = %v, want a figure from the source that did serve: %+v", res.DownMbps, res)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,687 @@
|
||||
package network
|
||||
|
||||
// The speed test. It measures against Cloudflare's public speed endpoints, the same ones
|
||||
// speed.cloudflare.com uses: a plain GET of N bytes down, a plain POST of N bytes up, and a
|
||||
// zero-byte GET for the round trip. No account, no API key, nothing to install, and an edge
|
||||
// close to every one of these machines.
|
||||
//
|
||||
// The method, since a number without one means little — it is stated in the UI too:
|
||||
//
|
||||
// - Latency is the round trip of a zero-byte response on a connection that is already open,
|
||||
// less the time the far end says it spent on the request (its Server-Timing header). That
|
||||
// subtraction is the difference between measuring the line and measuring Cloudflare's
|
||||
// thinking time: a zero-byte response is almost entirely the latter — tens of
|
||||
// milliseconds of worker, against a few of actual wire. The figure reported is the best of
|
||||
// several probes; jitter is the average change from one to the next.
|
||||
// - Download and upload each run several streams at once for a fixed few seconds. One
|
||||
// stream measures one TCP connection's congestion window as much as the line, which is
|
||||
// why every speed test ever made uses more than one.
|
||||
// - The first second of each phase is thrown away. That is the ramp: counting it reports a
|
||||
// number nobody's connection ever actually did.
|
||||
// - Each phase is capped in bytes as well as seconds, so a very fast line cannot run away
|
||||
// with the meter.
|
||||
//
|
||||
// The one thing to know about the far end: it rate-limits by volume, and a test moves a few
|
||||
// hundred megabytes. Several in a row will start earning 429s, which is why there is a
|
||||
// cooldown between tests and why a phase that gets refused says so instead of reporting the
|
||||
// decayed figure it would otherwise compute.
|
||||
//
|
||||
// HTTP/2 is deliberately off: over one HTTP/2 connection the streams would share a single
|
||||
// congestion window, which measures the window, not the line.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"math"
|
||||
"net"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The endpoints, and the shape of a phase. Variables rather than constants only so the tests
|
||||
// can point them at a local server and shrink the windows; nothing else rebinds them.
|
||||
// Source is somewhere to pull bytes from. Cloudflare's endpoint is the first choice because
|
||||
// it is anycast — near wherever Ham Lab is running — and because the same host answers for
|
||||
// upload and for the trace. The fallback exists because that endpoint rate-limits by volume
|
||||
// and will refuse a connection that has been tested hard: a plain 100 MB file on a public
|
||||
// mirror has no such opinion. It is in Seattle, so on a machine far from there it will read
|
||||
// low, which is exactly why every result names the source it used.
|
||||
type Source struct {
|
||||
Name string
|
||||
// URL takes the number of bytes wanted, where the source can be asked; a fixed file
|
||||
// ignores it and is fetched again each time round.
|
||||
URL func(bytes int) string
|
||||
}
|
||||
|
||||
var downSources = []Source{
|
||||
{"cloudflare", func(b int) string { return fmt.Sprintf(cfDownURL, b) }},
|
||||
{"linode-seattle", func(int) string { return "https://speedtest.seattle.linode.com/100MB-seattle.bin" }},
|
||||
}
|
||||
|
||||
var (
|
||||
cfDownURL = "https://speed.cloudflare.com/__down?bytes=%d"
|
||||
upURL = "https://speed.cloudflare.com/__up"
|
||||
// traceURL is Cloudflare's standard trace: a few lines of key=value naming the address it
|
||||
// sees, the edge that answered and the country. A few hundred bytes.
|
||||
traceURL = "https://speed.cloudflare.com/cdn-cgi/trace"
|
||||
|
||||
warmUp = time.Second
|
||||
downFor = 4 * time.Second
|
||||
upFor = 4 * time.Second
|
||||
// measureBytes ends a phase early once this much has moved inside the window. A fast line
|
||||
// says what it is in well under the time allowed, and stopping there is what keeps a test
|
||||
// down to a couple of hundred megabytes — which is the difference between the far end
|
||||
// serving this happily and rate-limiting it. A slow line never reaches it and is timed out
|
||||
// by downFor/upFor instead, which is the honest way round.
|
||||
measureBytes = int64(64) << 20
|
||||
// minWindow is how long the window must last whatever the byte cap says. Without it a very
|
||||
// fast line fills the cap in a few hundredths of a second and the phase would be over
|
||||
// before there was anything to divide by.
|
||||
minWindow = time.Second
|
||||
)
|
||||
|
||||
const (
|
||||
latencyProbes = 8
|
||||
downStreams = 4
|
||||
upStreams = 3
|
||||
downChunk = 25 << 20 // bytes per download request; the far end refuses much more
|
||||
upChunk = 25 << 20 // and per upload request
|
||||
// phaseBudget is the runaway guard, counting the ramp too: nothing measures forever.
|
||||
phaseBudget = int64(512) << 20
|
||||
httpTimeout = 90 * time.Second
|
||||
// Cooldown is how long the line is left alone between tests. The far end rate-limits by
|
||||
// volume, and two back-to-back tests are the surest way to be refused by it.
|
||||
Cooldown = 20 * time.Second
|
||||
|
||||
// Method is what the UI tells the reader about the numbers it is showing.
|
||||
Method = "Cloudflare's speed endpoints · 4 streams down, 3 up · a 1 s ramp, then up to 4 s or 64 MiB each way"
|
||||
)
|
||||
|
||||
// Result is one finished speed test.
|
||||
type Result struct {
|
||||
At time.Time `json:"at"`
|
||||
DownMbps float64 `json:"down_mbps"`
|
||||
UpMbps float64 `json:"up_mbps"`
|
||||
LatencyMs float64 `json:"latency_ms"`
|
||||
JitterMs float64 `json:"jitter_ms"`
|
||||
DownBytes int64 `json:"down_bytes"`
|
||||
UpBytes int64 `json:"up_bytes"`
|
||||
TookSec float64 `json:"took_sec"`
|
||||
// Colo is the Cloudflare edge that answered — "SEA" — and IP is this connection's public
|
||||
// address as that edge saw it.
|
||||
Colo string `json:"colo,omitempty"`
|
||||
IP string `json:"ip,omitempty"`
|
||||
// Source is where the download came from, which matters because the fallback is not in the
|
||||
// same place as the anycast one. Empty means the usual.
|
||||
Source string `json:"source,omitempty"`
|
||||
// Note is set when the figures are real but something went wrong along the way — a stream
|
||||
// that was refused part-way, most often. The number is what was actually seen; the note is
|
||||
// why it may be low.
|
||||
Note string `json:"note,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Run is a test in progress, or the last one to have finished.
|
||||
type Run struct {
|
||||
Started time.Time `json:"started"`
|
||||
Running bool `json:"running"`
|
||||
// Phase is where it has got to: latency, download, upload, done.
|
||||
Phase string `json:"phase"`
|
||||
// Done is how far through the whole test, 0–1, so one bar can show all of it.
|
||||
Done float64 `json:"done"`
|
||||
// Mbps is the live figure for the phase under way — what the needle shows.
|
||||
Mbps float64 `json:"mbps"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Result *Result `json:"result,omitempty"`
|
||||
}
|
||||
|
||||
// Tester runs one speed test at a time and remembers what the last ones said.
|
||||
type Tester struct {
|
||||
mu sync.RWMutex
|
||||
run Run
|
||||
hist []Result
|
||||
// lastTry is when a test was last *started*, which is what the cooldown counts from — a
|
||||
// test that was refused still used the line, and hammering retries at an endpoint that has
|
||||
// just rate-limited us is the one thing guaranteed not to help.
|
||||
lastTry time.Time
|
||||
cancel context.CancelFunc
|
||||
onDone func(Result)
|
||||
maxHist int
|
||||
}
|
||||
|
||||
func NewTester(onDone func(Result)) *Tester {
|
||||
return &Tester{onDone: onDone, maxHist: 50}
|
||||
}
|
||||
|
||||
// Load takes the results kept from before a restart, oldest first.
|
||||
func (t *Tester) Load(rs []Result) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.hist = append(t.hist[:0], rs...)
|
||||
if n := len(t.hist); n > 0 {
|
||||
last := t.hist[n-1]
|
||||
t.run = Run{Started: last.At, Phase: "done", Done: 1, Result: &last}
|
||||
t.lastTry = last.At
|
||||
}
|
||||
}
|
||||
|
||||
func (t *Tester) State() Run {
|
||||
t.mu.RLock()
|
||||
defer t.mu.RUnlock()
|
||||
return t.run
|
||||
}
|
||||
|
||||
func (t *Tester) History() []Result {
|
||||
t.mu.RLock()
|
||||
defer t.mu.RUnlock()
|
||||
return append([]Result{}, t.hist...)
|
||||
}
|
||||
|
||||
func (t *Tester) Running() bool {
|
||||
t.mu.RLock()
|
||||
defer t.mu.RUnlock()
|
||||
return t.run.Running
|
||||
}
|
||||
|
||||
// Cancel stops a test in progress. Reports whether there was one.
|
||||
func (t *Tester) Cancel() bool {
|
||||
t.mu.Lock()
|
||||
c := t.cancel
|
||||
t.mu.Unlock()
|
||||
if c == nil {
|
||||
return false
|
||||
}
|
||||
c()
|
||||
return true
|
||||
}
|
||||
|
||||
var ErrBusy = errors.New("a speed test is already running")
|
||||
|
||||
// Start begins a test in the background and returns its opening state.
|
||||
func (t *Tester) Start() (Run, error) {
|
||||
t.mu.Lock()
|
||||
if t.run.Running {
|
||||
r := t.run
|
||||
t.mu.Unlock()
|
||||
return r, ErrBusy
|
||||
}
|
||||
if since := time.Since(t.lastTry); since < Cooldown && !t.lastTry.IsZero() {
|
||||
r := t.run
|
||||
t.mu.Unlock()
|
||||
return r, fmt.Errorf("give the line %d more seconds — testing back to back is what gets this connection rate-limited", int((Cooldown-since).Seconds())+1)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
t.cancel = cancel
|
||||
t.lastTry = time.Now()
|
||||
t.run = Run{Started: t.lastTry, Running: true, Phase: "latency"}
|
||||
r := t.run
|
||||
t.mu.Unlock()
|
||||
go t.measure(ctx, cancel)
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (t *Tester) set(fn func(*Run)) {
|
||||
t.mu.Lock()
|
||||
fn(&t.run)
|
||||
t.mu.Unlock()
|
||||
}
|
||||
|
||||
func (t *Tester) measure(ctx context.Context, cancel context.CancelFunc) {
|
||||
defer cancel()
|
||||
started := time.Now()
|
||||
res := Result{At: started}
|
||||
|
||||
tr := newTransport(downStreams)
|
||||
defer tr.CloseIdleConnections()
|
||||
client := &http.Client{Transport: tr, Timeout: httpTimeout}
|
||||
|
||||
// latency, and the meta the same responses carry
|
||||
lat, meta, err := t.latency(ctx, client)
|
||||
if err == nil {
|
||||
res.LatencyMs, res.JitterMs = best(lat), jitter(lat)
|
||||
res.Colo, res.IP = meta.Colo, meta.IP
|
||||
}
|
||||
var note string
|
||||
if err == nil {
|
||||
t.set(func(r *Run) { r.Phase, r.Done, r.Mbps = "download", 0.1, 0 })
|
||||
// Each source in turn: a refusal from the first is exactly what the second is for, and
|
||||
// anything else (no route, no DNS) would fail the same way twice, so it stops there.
|
||||
for i, src := range downSources {
|
||||
res.Source = src.Name
|
||||
res.DownMbps, res.DownBytes, note, err = t.phase(ctx, client, "download", downStreams, downFor, 0.1, 0.6, src)
|
||||
if err == nil || !refused(err) || i == len(downSources)-1 {
|
||||
break
|
||||
}
|
||||
log.Printf("speed test: %s refused the download, trying %s", src.Name, downSources[i+1].Name)
|
||||
}
|
||||
res.Note = join(res.Note, note)
|
||||
}
|
||||
if err == nil {
|
||||
t.set(func(r *Run) { r.Phase, r.Done, r.Mbps = "upload", 0.7, 0 })
|
||||
res.UpMbps, res.UpBytes, note, err = t.phase(ctx, client, "upload", upStreams, upFor, 0.7, 1.0, Source{Name: "cloudflare"})
|
||||
res.Note = join(res.Note, note)
|
||||
}
|
||||
res.TookSec = round(time.Since(started).Seconds(), 1)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
res.Error = "cancelled"
|
||||
} else {
|
||||
res.Error = err.Error()
|
||||
}
|
||||
}
|
||||
|
||||
t.mu.Lock()
|
||||
t.cancel = nil
|
||||
t.run = Run{Started: started, Phase: "done", Done: 1, Result: &res, Bytes: res.DownBytes + res.UpBytes}
|
||||
if res.Error == "" || res.DownMbps > 0 { // a cancelled test that got a number is still a reading
|
||||
t.hist = append(t.hist, res)
|
||||
if len(t.hist) > t.maxHist {
|
||||
t.hist = t.hist[len(t.hist)-t.maxHist:]
|
||||
}
|
||||
}
|
||||
done, out := t.onDone, res
|
||||
t.mu.Unlock()
|
||||
if done != nil {
|
||||
done(out)
|
||||
}
|
||||
}
|
||||
|
||||
// meta is what the edge says about the connection. The address comes from a header of the
|
||||
// speed endpoint's own; the edge's name is the tail of the ray ID, which every Cloudflare
|
||||
// response carries ("a41f6720f966a36c-SEA").
|
||||
type meta struct{ IP, Colo string }
|
||||
|
||||
func readMeta(h http.Header) meta {
|
||||
m := meta{IP: h.Get("cf-meta-ip")}
|
||||
if ray := h.Get("cf-ray"); ray != "" {
|
||||
if i := strings.LastIndexByte(ray, '-'); i >= 0 {
|
||||
m.Colo = ray[i+1:]
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// serverMs is how long the far end says it spent on this request, from Server-Timing. The
|
||||
// cfL4 entry carries no dur= and contributes nothing, which is what we want — it is a
|
||||
// description of the connection, not time spent.
|
||||
func serverMs(h http.Header) float64 {
|
||||
var total float64
|
||||
for _, v := range h.Values("Server-Timing") {
|
||||
for _, part := range strings.Split(v, ",") {
|
||||
_, after, ok := strings.Cut(part, "dur=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
end := 0
|
||||
for end < len(after) && (after[end] == '.' || (after[end] >= '0' && after[end] <= '9')) {
|
||||
end++
|
||||
}
|
||||
if f, err := strconv.ParseFloat(after[:end], 64); err == nil {
|
||||
total += f
|
||||
}
|
||||
}
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// latency times several zero-byte responses on an already-open connection.
|
||||
func (t *Tester) latency(ctx context.Context, client *http.Client) ([]float64, meta, error) {
|
||||
var m meta
|
||||
var out []float64
|
||||
for i := 0; i <= latencyProbes; i++ { // the first is the handshake, and is not counted
|
||||
start := time.Now()
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", fmt.Sprintf(cfDownURL, 0), nil)
|
||||
if err != nil {
|
||||
return nil, m, err
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, m, err
|
||||
}
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
return nil, m, fmt.Errorf("speed endpoint answered %s", resp.Status)
|
||||
}
|
||||
if i == 0 {
|
||||
m = readMeta(resp.Header)
|
||||
continue
|
||||
}
|
||||
rtt := float64(time.Since(start).Microseconds())/1000 - serverMs(resp.Header)
|
||||
if rtt < 0.01 { // the far end claimed more time than the whole request took
|
||||
rtt = 0.01
|
||||
}
|
||||
out = append(out, rtt)
|
||||
t.set(func(r *Run) { r.Done = 0.02 + 0.08*float64(i)/latencyProbes })
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, m, errors.New("no reply from the speed endpoint")
|
||||
}
|
||||
return out, m, nil
|
||||
}
|
||||
|
||||
// phase pushes traffic with several streams for a fixed spell and reports the rate over the
|
||||
// window after the ramp. from and to are where this phase sits in the whole test's progress.
|
||||
//
|
||||
// The window ends when the traffic does, not when the clock runs out: if the far end starts
|
||||
// refusing part-way, the figure is what was actually moved while it was moving, and the note
|
||||
// says what happened. Coasting to the end of the clock with no bytes arriving would report a
|
||||
// number a third of the truth.
|
||||
func (t *Tester) phase(parent context.Context, client *http.Client, dir string, streams int, forHow time.Duration, from, to float64, src Source) (float64, int64, string, error) {
|
||||
// parent is the test's own context, which is cancelled when somebody presses Stop. ctx is
|
||||
// this phase's, which *this function* cancels to shut the streams down — so the two are
|
||||
// kept apart: asking ctx whether the test was cancelled would always say yes by then, and
|
||||
// every failure would be reported as "context canceled" whatever really happened.
|
||||
ctx, stop := context.WithCancel(parent)
|
||||
defer stop()
|
||||
|
||||
var moved atomic.Int64
|
||||
var firstErr error
|
||||
var errOnce sync.Once
|
||||
var alive atomic.Int32
|
||||
alive.Store(int32(streams))
|
||||
dead := make(chan struct{})
|
||||
var deadOnce sync.Once
|
||||
|
||||
for i := 0; i < streams; i++ {
|
||||
go func() {
|
||||
defer func() {
|
||||
if alive.Add(-1) == 0 {
|
||||
deadOnce.Do(func() { close(dead) })
|
||||
}
|
||||
}()
|
||||
tries := 0
|
||||
for ctx.Err() == nil && moved.Load() < phaseBudget {
|
||||
var err error
|
||||
if dir == "download" {
|
||||
err = stream(ctx, client, src, &moved)
|
||||
} else {
|
||||
err = upstream(ctx, client, &moved)
|
||||
}
|
||||
if err == nil {
|
||||
tries = 0
|
||||
continue
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
errOnce.Do(func() { firstErr = err })
|
||||
if tries++; tries > 2 { // refused three times: this stream is finished
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-time.After(400 * time.Millisecond):
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// the ramp, measured and thrown away
|
||||
select {
|
||||
case <-time.After(warmUp):
|
||||
case <-dead:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
mark, at := moved.Load(), time.Now()
|
||||
last, lastAt := mark, at // the last time bytes were still arriving
|
||||
deadline := at.Add(forHow)
|
||||
tick := time.NewTicker(200 * time.Millisecond)
|
||||
defer tick.Stop()
|
||||
|
||||
for time.Now().Before(deadline) {
|
||||
select {
|
||||
case <-tick.C:
|
||||
case <-dead:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
if parent.Err() != nil {
|
||||
break
|
||||
}
|
||||
now := moved.Load()
|
||||
if now > last {
|
||||
last, lastAt = now, time.Now()
|
||||
}
|
||||
// enough has moved to know, and it has been long enough to mean it
|
||||
if (now-mark >= measureBytes && time.Since(at) >= minWindow) || now >= phaseBudget {
|
||||
break
|
||||
}
|
||||
if secs := time.Since(at).Seconds(); secs > 0 {
|
||||
live := mbps(now-mark, secs)
|
||||
part := math.Min(1, secs/forHow.Seconds())
|
||||
t.set(func(r *Run) { r.Mbps, r.Bytes, r.Done = live, now, from+(to-from)*part })
|
||||
}
|
||||
// every stream has given up and nothing more is coming
|
||||
if alive.Load() == 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
total, secs := last-mark, lastAt.Sub(at).Seconds()
|
||||
stop()
|
||||
|
||||
if total <= 0 {
|
||||
switch {
|
||||
case firstErr != nil: // what the far end said explains it better than anything else
|
||||
return 0, moved.Load(), "", firstErr
|
||||
case parent.Err() != nil:
|
||||
return 0, moved.Load(), "", parent.Err()
|
||||
}
|
||||
return 0, moved.Load(), "", fmt.Errorf("nothing moved in the %s phase", dir)
|
||||
}
|
||||
if secs < 0.02 { // too short a window to divide by; the guard is against dividing by zero,
|
||||
// not a judgement about how long a good measurement takes
|
||||
return 0, moved.Load(), "", fmt.Errorf("the %s phase was over before it could be measured", dir)
|
||||
}
|
||||
note := ""
|
||||
if firstErr != nil {
|
||||
note = fmt.Sprintf("the %s phase was cut short: %s", dir, firstErr)
|
||||
}
|
||||
return mbps(total, secs), moved.Load(), note, nil
|
||||
}
|
||||
|
||||
// join puts two notes together without leaving an empty one dangling.
|
||||
func join(a, b string) string {
|
||||
switch {
|
||||
case a == "":
|
||||
return b
|
||||
case b == "":
|
||||
return a
|
||||
}
|
||||
return a + "; " + b
|
||||
}
|
||||
|
||||
// stream fetches one chunk from src, counting bytes as they arrive.
|
||||
func stream(ctx context.Context, client *http.Client, src Source, n *atomic.Int64) error {
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", src.URL(downChunk), nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
return answered(resp.StatusCode, resp.Status)
|
||||
}
|
||||
buf := make([]byte, 64<<10)
|
||||
for {
|
||||
got, err := resp.Body.Read(buf)
|
||||
if got > 0 {
|
||||
n.Add(int64(got))
|
||||
}
|
||||
if err != nil {
|
||||
if err == io.EOF {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// upstream posts one chunk of nothing in particular, counting bytes as they go out.
|
||||
func upstream(ctx context.Context, client *http.Client, n *atomic.Int64) error {
|
||||
body := &zeros{left: upChunk, n: n}
|
||||
req, err := http.NewRequestWithContext(ctx, "POST", upURL, body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.ContentLength = upChunk
|
||||
req.Header.Set("Content-Type", "application/octet-stream")
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != 200 && resp.StatusCode != 204 {
|
||||
return answered(resp.StatusCode, resp.Status)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// answered turns the far end's refusal into something worth reading. It rate-limits by
|
||||
// volume: enough testing in a short while and it stops serving the big transfers, which is
|
||||
// the only failure of this test anyone here is likely to meet.
|
||||
func answered(code int, status string) error {
|
||||
if code == 429 || code == 403 {
|
||||
return errRefused
|
||||
}
|
||||
return fmt.Errorf("the speed endpoint answered %s", status)
|
||||
}
|
||||
|
||||
// errRefused is the far end rate-limiting this connection, which is the only failure of this
|
||||
// test anyone here is likely to meet: it limits by volume, and a test moves a few hundred
|
||||
// megabytes. It is recognisable so the download phase can try its other source.
|
||||
var errRefused = errors.New("the speed endpoint is rate-limiting this connection — it does that after a lot of testing; leave it a few minutes")
|
||||
|
||||
func refused(err error) bool { return errors.Is(err, errRefused) }
|
||||
|
||||
// zeros is a request body of a fixed size that costs nothing to produce, counting itself as
|
||||
// it is read out onto the wire.
|
||||
type zeros struct {
|
||||
left int
|
||||
n *atomic.Int64
|
||||
}
|
||||
|
||||
func (z *zeros) Read(p []byte) (int, error) {
|
||||
if z.left <= 0 {
|
||||
return 0, io.EOF
|
||||
}
|
||||
if len(p) > z.left {
|
||||
p = p[:z.left]
|
||||
}
|
||||
for i := range p {
|
||||
p[i] = 0
|
||||
}
|
||||
z.left -= len(p)
|
||||
z.n.Add(int64(len(p)))
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func newTransport(streams int) *http.Transport {
|
||||
return &http.Transport{
|
||||
Proxy: http.ProxyFromEnvironment,
|
||||
DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext,
|
||||
TLSHandshakeTimeout: 10 * time.Second,
|
||||
MaxIdleConns: streams * 2,
|
||||
MaxIdleConnsPerHost: streams * 2,
|
||||
// An empty map turns HTTP/2 off: see the note at the top of this file.
|
||||
TLSNextProto: map[string]func(string, *tls.Conn) http.RoundTripper{},
|
||||
}
|
||||
}
|
||||
|
||||
// probeOutside asks the far end who it thinks we are. One small request.
|
||||
func probeOutside() Outside {
|
||||
tr := newTransport(1)
|
||||
defer tr.CloseIdleConnections()
|
||||
client := &http.Client{Transport: tr, Timeout: 15 * time.Second}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", traceURL, nil)
|
||||
if err != nil {
|
||||
return Outside{At: time.Now(), Error: err.Error()}
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return Outside{At: time.Now(), Error: shortErr(err)}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
|
||||
if resp.StatusCode != 200 {
|
||||
return Outside{At: time.Now(), Error: "the trace endpoint answered " + resp.Status}
|
||||
}
|
||||
o := Outside{At: time.Now()}
|
||||
for _, line := range strings.Split(string(body), "\n") {
|
||||
k, v, ok := strings.Cut(line, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
switch k {
|
||||
case "ip":
|
||||
o.IP = v
|
||||
case "colo":
|
||||
o.Colo = v
|
||||
case "loc":
|
||||
o.Country = v
|
||||
}
|
||||
}
|
||||
if o.IP == "" {
|
||||
o.Error = "the trace endpoint said nothing about this connection"
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// shortErr keeps the part of a network error a person can act on.
|
||||
func shortErr(err error) string {
|
||||
s := err.Error()
|
||||
if i := strings.LastIndex(s, ": "); i > 0 && len(s)-i < 60 {
|
||||
return s[i+2:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func mbps(bytes int64, secs float64) float64 {
|
||||
return round(float64(bytes)*8/secs/1e6, 1)
|
||||
}
|
||||
|
||||
func best(v []float64) float64 {
|
||||
m := v[0]
|
||||
for _, x := range v {
|
||||
if x < m {
|
||||
m = x
|
||||
}
|
||||
}
|
||||
return round(m, 1)
|
||||
}
|
||||
|
||||
// jitter is how much the line wobbles: the average change from one round trip to the next,
|
||||
// which is the usual definition and, unlike a standard deviation, is not run away with by a
|
||||
// single slow probe.
|
||||
func jitter(v []float64) float64 {
|
||||
if len(v) < 2 {
|
||||
return 0
|
||||
}
|
||||
var sum float64
|
||||
for i := 1; i < len(v); i++ {
|
||||
sum += math.Abs(v[i] - v[i-1])
|
||||
}
|
||||
return round(sum/float64(len(v)-1), 1)
|
||||
}
|
||||
|
||||
func round(v float64, places int) float64 {
|
||||
p := math.Pow(10, float64(places))
|
||||
return math.Round(v*p) / p
|
||||
}
|
||||
|
||||
// Mbits renders a rate the way the line is sold, for a log line.
|
||||
func Mbits(v float64) string { return strconv.FormatFloat(v, 'f', 1, 64) + " Mbps" }
|
||||
@@ -0,0 +1,476 @@
|
||||
// Package nodes is the hub side of multi-host: the list of managed Ham Labs, their
|
||||
// health, adding one (pair with its password, receive a token, pin its certificate),
|
||||
// removing one (revoke the token), and the reverse proxy that turns
|
||||
// /api/nodes/<name>/… into /api/… on that node, WebSockets included.
|
||||
package nodes
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/tlsauto"
|
||||
)
|
||||
|
||||
var nameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,31}$`)
|
||||
|
||||
type Status struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
Fingerprint string `json:"fingerprint,omitempty"`
|
||||
Added time.Time `json:"added"`
|
||||
Online bool `json:"online"`
|
||||
Host string `json:"host,omitempty"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Load string `json:"load,omitempty"`
|
||||
Uptime float64 `json:"uptime,omitempty"`
|
||||
RebootNeed bool `json:"reboot_needed,omitempty"`
|
||||
LastSeen time.Time `json:"last_seen,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
cfg *config.Config
|
||||
hubName string
|
||||
mu sync.Mutex
|
||||
status map[string]*Status
|
||||
proxies map[string]*httputil.ReverseProxy
|
||||
clients map[string]*http.Client
|
||||
}
|
||||
|
||||
func New(cfg *config.Config, hubName string) *Manager {
|
||||
return &Manager{cfg: cfg, hubName: hubName, status: map[string]*Status{}, proxies: map[string]*httputil.ReverseProxy{}, clients: map[string]*http.Client{}}
|
||||
}
|
||||
|
||||
func transportFor(n config.Node) *http.Transport {
|
||||
t := &http.Transport{
|
||||
DialContext: (&net.Dialer{Timeout: 5 * time.Second}).DialContext,
|
||||
TLSHandshakeTimeout: 5 * time.Second,
|
||||
ResponseHeaderTimeout: 60 * time.Second,
|
||||
MaxIdleConnsPerHost: 8,
|
||||
}
|
||||
if strings.HasPrefix(n.URL, "https://") {
|
||||
if n.Fingerprint != "" {
|
||||
t.TLSClientConfig = tlsauto.PinnedConfig(n.Fingerprint)
|
||||
} else {
|
||||
t.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
|
||||
}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func (m *Manager) client(n config.Node) *http.Client {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if c, ok := m.clients[n.Name]; ok {
|
||||
return c
|
||||
}
|
||||
c := &http.Client{Transport: transportFor(n), Timeout: 15 * time.Second}
|
||||
m.clients[n.Name] = c
|
||||
return c
|
||||
}
|
||||
|
||||
func (m *Manager) node(name string) (config.Node, bool) {
|
||||
var found config.Node
|
||||
ok := false
|
||||
m.cfg.View(func(c *config.Config) {
|
||||
for _, n := range c.Nodes {
|
||||
if n.Name == name {
|
||||
found, ok = n, true
|
||||
}
|
||||
}
|
||||
})
|
||||
return found, ok
|
||||
}
|
||||
|
||||
// ---- probing ----
|
||||
|
||||
func (m *Manager) probe(ctx context.Context, n config.Node) *Status {
|
||||
st := &Status{Name: n.Name, URL: n.URL, Fingerprint: n.Fingerprint, Added: n.Added}
|
||||
req, _ := http.NewRequestWithContext(ctx, "GET", strings.TrimRight(n.URL, "/")+"/api/me", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
resp, err := m.client(n).Do(req)
|
||||
if err != nil {
|
||||
st.Error = shortErr(err)
|
||||
return st
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
st.Error = fmt.Sprintf("HTTP %d", resp.StatusCode)
|
||||
if resp.StatusCode == 401 {
|
||||
st.Error = "token rejected (re-add the machine)"
|
||||
}
|
||||
return st
|
||||
}
|
||||
var me struct {
|
||||
Host, Version, Load string
|
||||
Uptime float64
|
||||
RebootNeeded bool `json:"reboot_needed"`
|
||||
}
|
||||
if json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&me) != nil {
|
||||
st.Error = "bad reply"
|
||||
return st
|
||||
}
|
||||
st.Online, st.Host, st.Version, st.Load, st.Uptime, st.RebootNeed, st.LastSeen = true, me.Host, me.Version, me.Load, me.Uptime, me.RebootNeeded, time.Now()
|
||||
return st
|
||||
}
|
||||
|
||||
func shortErr(err error) string {
|
||||
s := err.Error()
|
||||
if i := strings.LastIndex(s, ": "); i >= 0 {
|
||||
s = s[i+2:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Run refreshes every node's status periodically.
|
||||
func (m *Manager) Run(ctx context.Context) {
|
||||
m.refresh(ctx)
|
||||
t := time.NewTicker(15 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
m.refresh(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Manager) refresh(ctx context.Context) {
|
||||
var list []config.Node
|
||||
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
|
||||
var wg sync.WaitGroup
|
||||
for _, n := range list {
|
||||
wg.Add(1)
|
||||
go func(n config.Node) {
|
||||
defer wg.Done()
|
||||
pctx, cancel := context.WithTimeout(ctx, 8*time.Second)
|
||||
defer cancel()
|
||||
st := m.probe(pctx, n)
|
||||
m.mu.Lock()
|
||||
if prev, ok := m.status[n.Name]; ok && !st.Online && prev.Online {
|
||||
st.LastSeen = prev.LastSeen // keep the last good sighting
|
||||
}
|
||||
m.status[n.Name] = st
|
||||
m.mu.Unlock()
|
||||
}(n)
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// List returns every managed node with its last known status.
|
||||
func (m *Manager) List() []Status {
|
||||
var list []config.Node
|
||||
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
out := make([]Status, 0, len(list))
|
||||
for _, n := range list {
|
||||
if st, ok := m.status[n.Name]; ok {
|
||||
out = append(out, *st)
|
||||
} else {
|
||||
out = append(out, Status{Name: n.Name, URL: n.URL, Fingerprint: n.Fingerprint, Added: n.Added, Error: "not probed yet"})
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- add / remove ----
|
||||
|
||||
// Add pairs with a node: pin its certificate (trust on first use), trade the node's
|
||||
// password for a token named after this hub, and store the node. The password is
|
||||
// used once and never kept.
|
||||
func (m *Manager) Add(ctx context.Context, name, rawURL, username, password string) (Status, error) {
|
||||
if !nameRe.MatchString(name) {
|
||||
return Status{}, errors.New("name: lowercase letters, digits and dashes only")
|
||||
}
|
||||
if _, exists := m.node(name); exists {
|
||||
return Status{}, errors.New("a machine with that name already exists")
|
||||
}
|
||||
u, err := url.Parse(strings.TrimSpace(rawURL))
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
return Status{}, errors.New("address must be http(s)://host:port")
|
||||
}
|
||||
n := config.Node{Name: name, URL: u.Scheme + "://" + u.Host, Added: time.Now()}
|
||||
if u.Scheme == "https" {
|
||||
host := u.Host
|
||||
if _, _, err := net.SplitHostPort(host); err != nil {
|
||||
host += ":443"
|
||||
}
|
||||
fp, err := tlsauto.Peek(host)
|
||||
if err != nil {
|
||||
return Status{}, fmt.Errorf("connect: %s", shortErr(err))
|
||||
}
|
||||
n.Fingerprint = fp
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"username": username, "password": password, "name": "hub:" + m.hubName})
|
||||
req, _ := http.NewRequestWithContext(ctx, "POST", n.URL+"/api/tokens", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := (&http.Client{Transport: transportFor(n), Timeout: 15 * time.Second}).Do(req)
|
||||
if err != nil {
|
||||
return Status{}, fmt.Errorf("connect: %s", shortErr(err))
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
var reply struct {
|
||||
Token string `json:"token"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&reply)
|
||||
if resp.StatusCode != 200 || reply.Token == "" {
|
||||
if reply.Error == "" {
|
||||
reply.Error = fmt.Sprintf("HTTP %d", resp.StatusCode)
|
||||
}
|
||||
return Status{}, errors.New("pairing refused: " + reply.Error)
|
||||
}
|
||||
n.Token = reply.Token
|
||||
if err := m.cfg.Update(func(c *config.Config) { c.Nodes = append(c.Nodes, n) }); err != nil {
|
||||
return Status{}, err
|
||||
}
|
||||
st := m.probe(ctx, n)
|
||||
m.mu.Lock()
|
||||
m.status[name] = st
|
||||
m.mu.Unlock()
|
||||
return *st, nil
|
||||
}
|
||||
|
||||
// Remove revokes our token on the node (best effort) and forgets it.
|
||||
func (m *Manager) Remove(ctx context.Context, name string) error {
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
return errors.New("no such machine")
|
||||
}
|
||||
req, _ := http.NewRequestWithContext(ctx, "DELETE", n.URL+"/api/tokens/"+url.PathEscape("hub:"+m.hubName), nil)
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
if resp, err := m.client(n).Do(req); err == nil {
|
||||
resp.Body.Close()
|
||||
}
|
||||
err := m.cfg.Update(func(c *config.Config) {
|
||||
kept := c.Nodes[:0]
|
||||
for _, x := range c.Nodes {
|
||||
if x.Name != name {
|
||||
kept = append(kept, x)
|
||||
}
|
||||
}
|
||||
c.Nodes = kept
|
||||
})
|
||||
m.mu.Lock()
|
||||
delete(m.status, name)
|
||||
delete(m.proxies, name)
|
||||
delete(m.clients, name)
|
||||
m.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
|
||||
// ---- update from hub ----
|
||||
|
||||
type PushResult struct {
|
||||
Name string `json:"name"`
|
||||
OK bool `json:"ok"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Push uploads binPath (normally the hub's own executable) to the node's self-update
|
||||
// endpoint. The node verifies it, installs it and restarts itself. Refused when the
|
||||
// node's architecture differs from ours.
|
||||
func (m *Manager) Push(ctx context.Context, name, binPath, ourArch string, force bool) PushResult {
|
||||
res := PushResult{Name: name}
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
res.Error = "no such machine"
|
||||
return res
|
||||
}
|
||||
// arch check via /api/me
|
||||
req, _ := http.NewRequestWithContext(ctx, "GET", n.URL+"/api/me", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
resp, err := m.client(n).Do(req)
|
||||
if err != nil {
|
||||
res.Error = "unreachable: " + shortErr(err)
|
||||
return res
|
||||
}
|
||||
var me struct {
|
||||
Arch string `json:"arch"`
|
||||
Version string `json:"version"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&me)
|
||||
resp.Body.Close()
|
||||
if me.Arch != "" && ourArch != "" && me.Arch != ourArch {
|
||||
// look for a sibling build: /usr/local/bin/hamlab-arm64 next to /usr/local/bin/hamlab
|
||||
alt := binPath + "-" + me.Arch
|
||||
if _, err := os.Stat(alt); err != nil {
|
||||
res.Error = fmt.Sprintf("node is %s but only a %s build is installed here (expected %s; run make deploy, which builds it)", me.Arch, ourArch, alt)
|
||||
return res
|
||||
}
|
||||
binPath = alt
|
||||
}
|
||||
f, err := os.Open(binPath)
|
||||
if err != nil {
|
||||
res.Error = err.Error()
|
||||
return res
|
||||
}
|
||||
defer f.Close()
|
||||
st, _ := f.Stat()
|
||||
target := n.URL + "/api/self-update"
|
||||
if force {
|
||||
target += "?force=1"
|
||||
}
|
||||
up, _ := http.NewRequestWithContext(ctx, "POST", target, f)
|
||||
up.ContentLength = st.Size()
|
||||
up.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
up.Header.Set("Content-Type", "application/octet-stream")
|
||||
c := &http.Client{Transport: transportFor(n), Timeout: 5 * time.Minute}
|
||||
r2, err := c.Do(up)
|
||||
if err != nil {
|
||||
res.Error = "upload: " + shortErr(err)
|
||||
return res
|
||||
}
|
||||
defer r2.Body.Close()
|
||||
var reply struct {
|
||||
Version string `json:"version"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r2.Body, 64<<10)).Decode(&reply)
|
||||
if r2.StatusCode != 200 {
|
||||
if reply.Error == "" {
|
||||
reply.Error = fmt.Sprintf("HTTP %d", r2.StatusCode)
|
||||
}
|
||||
res.Error = reply.Error
|
||||
return res
|
||||
}
|
||||
res.OK, res.Version = true, reply.Version
|
||||
return res
|
||||
}
|
||||
|
||||
// PushAll updates every node, in parallel.
|
||||
func (m *Manager) PushAll(ctx context.Context, binPath, ourArch string, force bool) []PushResult {
|
||||
var list []config.Node
|
||||
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
|
||||
out := make([]PushResult, len(list))
|
||||
var wg sync.WaitGroup
|
||||
for i, n := range list {
|
||||
wg.Add(1)
|
||||
go func(i int, name string) {
|
||||
defer wg.Done()
|
||||
out[i] = m.Push(ctx, name, binPath, ourArch, force)
|
||||
}(i, n.Name)
|
||||
}
|
||||
wg.Wait()
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- proxy ----
|
||||
|
||||
// Proxy returns the handler for /api/nodes/<name>/<rest>: it rewrites to <node>/api/<rest>,
|
||||
// swaps the hub's cookie for the node's bearer token, and drops Origin so the node's
|
||||
// WebSocket origin check (meant for browsers) doesn't reject the hub.
|
||||
// Do makes one request to a managed machine as this hub and returns the whole response.
|
||||
func (m *Manager) Do(ctx context.Context, name, method, path string, body []byte, hdr http.Header) (int, []byte, error) {
|
||||
code, b, _, err := m.DoHeaders(ctx, name, method, path, body, hdr)
|
||||
return code, b, err
|
||||
}
|
||||
|
||||
// DoHeaders is Do that also answers the response's headers.
|
||||
func (m *Manager) DoHeaders(ctx context.Context, name, method, path string, body []byte, hdr http.Header) (int, []byte, http.Header, error) {
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
return 0, nil, nil, errors.New("no such machine")
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(n.URL, "/")+path, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return 0, nil, nil, err
|
||||
}
|
||||
for k, v := range hdr {
|
||||
req.Header[k] = v
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
resp, err := m.client(n).Do(req)
|
||||
if err != nil {
|
||||
return 0, nil, nil, errors.New(shortErr(err))
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
||||
return resp.StatusCode, b, resp.Header, err
|
||||
}
|
||||
|
||||
// URL is a managed machine's address, "" for one not managed here.
|
||||
func (m *Manager) URL(name string) string {
|
||||
n, _ := m.node(name)
|
||||
return n.URL
|
||||
}
|
||||
|
||||
// Names lists the managed machines.
|
||||
func (m *Manager) Names() []string {
|
||||
var out []string
|
||||
m.cfg.View(func(c *config.Config) {
|
||||
for _, n := range c.Nodes {
|
||||
out = append(out, n.Name)
|
||||
}
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) Proxy(name string) (http.Handler, bool) {
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
m.mu.Lock()
|
||||
if p, ok := m.proxies[name]; ok {
|
||||
m.mu.Unlock()
|
||||
return p, true
|
||||
}
|
||||
m.mu.Unlock()
|
||||
target, _ := url.Parse(n.URL)
|
||||
prefix := "/api/nodes/" + name
|
||||
p := &httputil.ReverseProxy{
|
||||
Director: func(r *http.Request) {
|
||||
r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
|
||||
r.URL.Path = "/api" + strings.TrimPrefix(r.URL.Path, prefix)
|
||||
r.URL.RawPath = ""
|
||||
r.Host = target.Host
|
||||
r.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
r.Header.Del("Cookie")
|
||||
r.Header.Del("Origin")
|
||||
r.Header.Del("X-Forwarded-For")
|
||||
r.Header.Del("X-Forwarded-Proto")
|
||||
},
|
||||
Transport: transportFor(n),
|
||||
FlushInterval: -1, // stream task output / log tails immediately
|
||||
// The node answers with its own policy headers and this hub adds the same ones to
|
||||
// everything it serves; relayed twice, Access-Control-Allow-Origin is invalid and a
|
||||
// browser on another origin refuses the response. The hub's copy is the one that counts.
|
||||
ModifyResponse: func(resp *http.Response) error {
|
||||
for _, h := range []string{"Access-Control-Allow-Origin", "X-Frame-Options", "Content-Security-Policy", "X-Content-Type-Options", "Referrer-Policy"} {
|
||||
resp.Header.Del(h)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": name + " unreachable: " + shortErr(err)})
|
||||
},
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.proxies[name] = p
|
||||
m.mu.Unlock()
|
||||
return p, true
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
// Package priv is the one place that knows how this daemon gets root: it already is root,
|
||||
// or it has passwordless sudo, or passwordless doas (Alpine, some BSD-minded setups), or
|
||||
// it has nothing and the privileged features stay off.
|
||||
package priv
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
mode string // "root" | "sudo" | "doas" | ""
|
||||
prefix []string
|
||||
probed time.Time
|
||||
reprobe = 30 * time.Second
|
||||
)
|
||||
|
||||
// detect settles how root is reached. Finding a way is remembered for good; finding none
|
||||
// is only believed for a while, because the probe can fail for reasons that pass — sudo
|
||||
// taking longer than the timeout on a machine that is still booting left a daemon
|
||||
// convinced it had no root until its next restart.
|
||||
func detect() {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if mode != "" || (!probed.IsZero() && time.Since(probed) < reprobe) {
|
||||
return
|
||||
}
|
||||
probed = time.Now()
|
||||
if os.Geteuid() == 0 {
|
||||
mode = "root"
|
||||
return
|
||||
}
|
||||
for _, tool := range []string{"sudo", "doas"} {
|
||||
if _, err := exec.LookPath(tool); err != nil {
|
||||
continue
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
err := exec.CommandContext(ctx, tool, "-n", "true").Run()
|
||||
cancel()
|
||||
if err == nil {
|
||||
mode, prefix = tool, []string{tool, "-n"}
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Mode reports how root is reached: "root", "sudo", "doas", or "" when it can't be.
|
||||
func Mode() string { detect(); mu.Lock(); defer mu.Unlock(); return mode }
|
||||
|
||||
// Available reports whether commands can be run as root without a prompt.
|
||||
func Available() bool { return Mode() != "" }
|
||||
|
||||
// Wrap returns args prefixed so they run as root. Without privilege it returns args
|
||||
// unchanged; callers gate on Available() where that matters.
|
||||
func Wrap(args ...string) []string {
|
||||
detect()
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return append(append([]string{}, prefix...), args...)
|
||||
}
|
||||
|
||||
// AsUser returns args prefixed so they run as another user with that user's HOME.
|
||||
func AsUser(user string, args ...string) []string {
|
||||
switch Mode() {
|
||||
case "sudo":
|
||||
return append([]string{"sudo", "-n", "-u", user, "-H"}, args...)
|
||||
case "doas":
|
||||
return append([]string{"doas", "-n", "-u", user}, args...)
|
||||
case "root":
|
||||
if user == "root" {
|
||||
return args
|
||||
}
|
||||
// su exists everywhere (util-linux, shadow, busybox), needs no password from root
|
||||
// and sets HOME for the target user.
|
||||
return []string{"su", "-s", "/bin/sh", user, "-c", Quote(args)}
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
// Quote joins args into one POSIX shell command line.
|
||||
func Quote(args []string) string {
|
||||
q := make([]string, len(args))
|
||||
for i, a := range args {
|
||||
q[i] = "'" + strings.ReplaceAll(a, "'", `'\''`) + "'"
|
||||
}
|
||||
return strings.Join(q, " ")
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
// Package selfupdate finds out whether a newer Ham Lab has been published and fetches it.
|
||||
//
|
||||
// A source is one URL. Two shapes are understood, told apart by the URL itself:
|
||||
//
|
||||
// - a Gitea/Forgejo repository API base — ".../api/v1/repos/<owner>/<repo>" — whose
|
||||
// newest release names the version and carries the binaries as assets;
|
||||
// - any other URL, treated as a static directory holding VERSION, SHA256SUMS and
|
||||
// hamlab-linux-<arch>, which is exactly what `make release` writes and what
|
||||
// install.sh already downloads.
|
||||
//
|
||||
// Either way the binary is checked against SHA256SUMS before it is allowed near the
|
||||
// running one. A token is sent when the source is behind a login, so a private Gitea
|
||||
// serves its own fleet without the instance being opened to the world.
|
||||
package selfupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// DefaultCheckEvery is how often a daemon asks, when it is asking at all.
|
||||
const DefaultCheckEvery = 24 * time.Hour
|
||||
|
||||
// DefaultURL is where a build comes from when the config does not say: the project's own
|
||||
// releases. An install elsewhere points `update.url` at its own, or at "off".
|
||||
const DefaultURL = "https://git.bigham.dev/api/v1/repos/mitchell/hamlab"
|
||||
|
||||
// maxBinary is the most we will download; a hamlab binary is ~20 MB.
|
||||
const maxBinary = 512 << 20
|
||||
|
||||
// Source is where updates come from.
|
||||
type Source struct {
|
||||
URL string // API base or static directory
|
||||
Token string // optional; for a source behind a login
|
||||
}
|
||||
|
||||
// Release is one published build.
|
||||
type Release struct {
|
||||
Version string `json:"version"`
|
||||
Notes string `json:"notes,omitempty"`
|
||||
Published time.Time `json:"published,omitempty"`
|
||||
// Binary and Sums are where to get this build for the architecture that was asked for.
|
||||
Binary string `json:"-"`
|
||||
Sums string `json:"-"`
|
||||
}
|
||||
|
||||
// AssetName is what a build for an architecture is called, in a release and on disk.
|
||||
func AssetName(arch string) string { return "hamlab-linux-" + arch }
|
||||
|
||||
func (s Source) api() bool { return strings.Contains(s.URL, "/api/v1/repos/") }
|
||||
|
||||
func (s Source) get(ctx context.Context, url string) (*http.Response, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.Token != "" {
|
||||
req.Header.Set("Authorization", "token "+s.Token)
|
||||
}
|
||||
req.Header.Set("User-Agent", "hamlab-selfupdate")
|
||||
resp, err := (&http.Client{Timeout: 10 * time.Minute}).Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resp.StatusCode != 200 {
|
||||
resp.Body.Close()
|
||||
switch resp.StatusCode {
|
||||
case 401, 403:
|
||||
return nil, fmt.Errorf("%s: the update source needs a login (HTTP %d); set update.token", url, resp.StatusCode)
|
||||
case 404:
|
||||
return nil, fmt.Errorf("%s: nothing published there (HTTP 404)", url)
|
||||
}
|
||||
return nil, fmt.Errorf("%s: HTTP %d", url, resp.StatusCode)
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
func (s Source) getAll(ctx context.Context, url string, limit int64) ([]byte, error) {
|
||||
resp, err := s.get(ctx, url)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
return io.ReadAll(io.LimitReader(resp.Body, limit))
|
||||
}
|
||||
|
||||
// Latest is the newest published build, and where to get it for arch.
|
||||
func (s Source) Latest(ctx context.Context, arch string) (Release, error) {
|
||||
if s.URL == "" {
|
||||
return Release{}, errors.New("no update source configured")
|
||||
}
|
||||
base := strings.TrimSuffix(s.URL, "/")
|
||||
if !s.api() {
|
||||
b, err := s.getAll(ctx, base+"/VERSION", 4096)
|
||||
if err != nil {
|
||||
return Release{}, err
|
||||
}
|
||||
v := strings.TrimSpace(strings.SplitN(string(b), "\n", 2)[0])
|
||||
if v == "" {
|
||||
return Release{}, errors.New("the source published an empty VERSION")
|
||||
}
|
||||
return Release{Version: v, Binary: base + "/" + AssetName(arch), Sums: base + "/SHA256SUMS"}, nil
|
||||
}
|
||||
b, err := s.getAll(ctx, base+"/releases/latest", 4<<20)
|
||||
if err != nil {
|
||||
return Release{}, err
|
||||
}
|
||||
var rel struct {
|
||||
TagName string `json:"tag_name"`
|
||||
Body string `json:"body"`
|
||||
PublishedAt time.Time `json:"published_at"`
|
||||
Assets []struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"browser_download_url"`
|
||||
} `json:"assets"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &rel); err != nil {
|
||||
return Release{}, fmt.Errorf("the source did not answer with a release: %w", err)
|
||||
}
|
||||
if rel.TagName == "" {
|
||||
return Release{}, errors.New("the newest release has no tag")
|
||||
}
|
||||
out := Release{Version: rel.TagName, Notes: rel.Body, Published: rel.PublishedAt}
|
||||
want := AssetName(arch)
|
||||
for _, a := range rel.Assets {
|
||||
switch a.Name {
|
||||
case want:
|
||||
out.Binary = a.URL
|
||||
case "SHA256SUMS":
|
||||
out.Sums = a.URL
|
||||
}
|
||||
}
|
||||
if out.Binary == "" {
|
||||
return out, fmt.Errorf("release %s has no %s", rel.TagName, want)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Fetch downloads the build for arch to dest and checks it against SHA256SUMS. A release
|
||||
// without sums is refused: an unverified binary is not worth installing over a working one.
|
||||
func (s Source) Fetch(ctx context.Context, rel Release, arch, dest string) error {
|
||||
if rel.Binary == "" {
|
||||
return errors.New("that release has no binary for this machine")
|
||||
}
|
||||
if rel.Sums == "" {
|
||||
return errors.New("that release has no SHA256SUMS, so the download cannot be verified")
|
||||
}
|
||||
sums, err := s.getAll(ctx, rel.Sums, 1<<20)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
want := sumFor(string(sums), AssetName(arch))
|
||||
if want == "" {
|
||||
return fmt.Errorf("SHA256SUMS has no entry for %s", AssetName(arch))
|
||||
}
|
||||
resp, err := s.get(ctx, rel.Binary)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
f, err := os.OpenFile(dest, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, maxBinary))
|
||||
cerr := f.Close()
|
||||
if err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
os.Remove(dest)
|
||||
return err
|
||||
}
|
||||
if n < 1<<20 {
|
||||
os.Remove(dest)
|
||||
return fmt.Errorf("downloaded %d bytes: too small to be hamlab", n)
|
||||
}
|
||||
if got := hex.EncodeToString(h.Sum(nil)); got != want {
|
||||
os.Remove(dest)
|
||||
return fmt.Errorf("checksum mismatch: SHA256SUMS says %s, the download is %s", want[:16]+"…", got[:16]+"…")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// sumFor finds a file's checksum in SHA256SUMS ("<hex> <name>", or "*<name>" for binary mode).
|
||||
func sumFor(sums, name string) string {
|
||||
for _, line := range strings.Split(sums, "\n") {
|
||||
f := strings.Fields(strings.TrimSpace(line))
|
||||
if len(f) == 2 && (f[1] == name || f[1] == "*"+name) && len(f[0]) == 64 {
|
||||
return strings.ToLower(f[0])
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Compare orders two versions: -1 if a is older than b, 0 if the same, 1 if newer.
|
||||
// It is install.sh's vnum() rule, so the daemon and the installer never disagree about
|
||||
// which way round two builds go: v0.1.0-69-g9c4f9df → 0001000069, the commits since the
|
||||
// tag included, the hash ignored.
|
||||
func Compare(a, b string) int {
|
||||
na, oka := vnum(a)
|
||||
nb, okb := vnum(b)
|
||||
switch {
|
||||
case !oka && !okb:
|
||||
return strings.Compare(a, b)
|
||||
case !oka:
|
||||
return -1
|
||||
case !okb:
|
||||
return 1
|
||||
case na < nb:
|
||||
return -1
|
||||
case na > nb:
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func vnum(v string) (int64, bool) {
|
||||
v = strings.TrimSpace(v)
|
||||
v = strings.TrimPrefix(v, "v")
|
||||
if i := strings.LastIndex(v, "-g"); i >= 0 && isHex(v[i+2:]) {
|
||||
v = v[:i]
|
||||
}
|
||||
if v == "" {
|
||||
return 0, false
|
||||
}
|
||||
parts := strings.FieldsFunc(v, func(r rune) bool { return r == '.' || r == '-' })
|
||||
nums := make([]int64, 4)
|
||||
for i := 0; i < 4 && i < len(parts); i++ {
|
||||
n, err := strconv.ParseInt(parts[i], 10, 64)
|
||||
if err != nil {
|
||||
if i == 0 {
|
||||
return 0, false
|
||||
}
|
||||
break
|
||||
}
|
||||
nums[i] = n
|
||||
}
|
||||
// %d%03d%03d%05d, as the installer does it
|
||||
return ((nums[0]*1000+nums[1])*1000+nums[2])*100000 + nums[3], true
|
||||
}
|
||||
|
||||
func isHex(s string) bool {
|
||||
if s == "" {
|
||||
return false
|
||||
}
|
||||
for _, r := range s {
|
||||
if !strings.ContainsRune("0123456789abcdefABCDEF", r) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
package selfupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestCompareMatchesInstaller(t *testing.T) {
|
||||
older := []([2]string){
|
||||
{"v0.1.0-69-g9c4f9df", "v0.1.0-107-gad88a48"},
|
||||
{"v0.1.0", "v0.1.0-1-gabc1234"},
|
||||
{"v0.1.0-107", "v0.2.0"},
|
||||
{"v0.9.9-999", "v1.0.0"},
|
||||
{"v0.1.9", "v0.1.10"},
|
||||
}
|
||||
for _, p := range older {
|
||||
if got := Compare(p[0], p[1]); got != -1 {
|
||||
t.Errorf("Compare(%q, %q) = %d, want -1", p[0], p[1], got)
|
||||
}
|
||||
if got := Compare(p[1], p[0]); got != 1 {
|
||||
t.Errorf("Compare(%q, %q) = %d, want 1", p[1], p[0], got)
|
||||
}
|
||||
}
|
||||
same := []string{"v0.1.0-107-gad88a48", "v1.2.3", "dev"}
|
||||
for _, v := range same {
|
||||
if got := Compare(v, v); got != 0 {
|
||||
t.Errorf("Compare(%q, %q) = %d, want 0", v, v, got)
|
||||
}
|
||||
}
|
||||
// the hash is ignored: the same build described two ways is the same build
|
||||
if got := Compare("v0.1.0-107-gad88a48", "v0.1.0-107-gffffff1"); got != 0 {
|
||||
t.Errorf("hash should not count: %d", got)
|
||||
}
|
||||
// "dev" is not a version; anything numbered beats it
|
||||
if got := Compare("dev", "v0.1.0"); got != -1 {
|
||||
t.Errorf("Compare(dev, v0.1.0) = %d, want -1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// serve puts a release behind both shapes of source and returns the two base URLs.
|
||||
func serve(t *testing.T, version string, body []byte, token string) (static, api string, srv *httptest.Server) {
|
||||
t.Helper()
|
||||
sum := sha256.Sum256(body)
|
||||
sums := fmt.Sprintf("%s %s\n%s install.sh\n", hex.EncodeToString(sum[:]), AssetName("amd64"), strings.Repeat("0", 64))
|
||||
mux := http.NewServeMux()
|
||||
guard := func(h http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if token != "" && r.Header.Get("Authorization") != "token "+token {
|
||||
w.WriteHeader(401)
|
||||
return
|
||||
}
|
||||
h(w, r)
|
||||
}
|
||||
}
|
||||
mux.HandleFunc("/dl/VERSION", guard(func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, version) }))
|
||||
mux.HandleFunc("/dl/SHA256SUMS", guard(func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, sums) }))
|
||||
mux.HandleFunc("/dl/"+AssetName("amd64"), guard(func(w http.ResponseWriter, r *http.Request) { w.Write(body) }))
|
||||
mux.HandleFunc("/api/v1/repos/m/hamlab/releases/latest", guard(func(w http.ResponseWriter, r *http.Request) {
|
||||
base := "http://" + r.Host + "/dl/"
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"tag_name": version, "body": "notes here",
|
||||
"assets": []map[string]string{
|
||||
{"name": AssetName("amd64"), "browser_download_url": base + AssetName("amd64")},
|
||||
{"name": "SHA256SUMS", "browser_download_url": base + "SHA256SUMS"},
|
||||
},
|
||||
})
|
||||
}))
|
||||
srv = httptest.NewServer(mux)
|
||||
return srv.URL + "/dl", srv.URL + "/api/v1/repos/m/hamlab", srv
|
||||
}
|
||||
|
||||
func TestLatestAndFetch(t *testing.T) {
|
||||
body := make([]byte, 2<<20)
|
||||
copy(body, "ELF fake hamlab")
|
||||
static, api, srv := serve(t, "v0.9.0-3-gdeadbee", body, "")
|
||||
defer srv.Close()
|
||||
|
||||
for _, url := range []string{static, api} {
|
||||
s := Source{URL: url}
|
||||
rel, err := s.Latest(context.Background(), "amd64")
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", url, err)
|
||||
}
|
||||
if rel.Version != "v0.9.0-3-gdeadbee" {
|
||||
t.Fatalf("%s: version %q", url, rel.Version)
|
||||
}
|
||||
dest := filepath.Join(t.TempDir(), "hamlab.new")
|
||||
if err := s.Fetch(context.Background(), rel, "amd64", dest); err != nil {
|
||||
t.Fatalf("%s: fetch: %v", url, err)
|
||||
}
|
||||
got, _ := os.ReadFile(dest)
|
||||
if len(got) != len(body) {
|
||||
t.Fatalf("%s: got %d bytes, want %d", url, len(got), len(body))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchRefusesBadChecksum(t *testing.T) {
|
||||
body := make([]byte, 2<<20)
|
||||
static, _, srv := serve(t, "v1.0.0", body, "")
|
||||
defer srv.Close()
|
||||
s := Source{URL: static}
|
||||
rel, err := s.Latest(context.Background(), "amd64")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rel.Sums = static + "/VERSION" // sums that name nothing
|
||||
dest := filepath.Join(t.TempDir(), "hamlab.new")
|
||||
if err := s.Fetch(context.Background(), rel, "amd64", dest); err == nil {
|
||||
t.Fatal("a release whose sums do not name the binary must be refused")
|
||||
}
|
||||
if _, err := os.Stat(dest); !os.IsNotExist(err) {
|
||||
t.Fatal("a refused download must not be left on disk")
|
||||
}
|
||||
// and an arch nobody published
|
||||
if _, err := s.Latest(context.Background(), "riscv64"); err == nil {
|
||||
t.Log("static sources cannot know an arch is missing until the fetch; that is fine")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTokenIsSentAndMissingTokenSaysSo(t *testing.T) {
|
||||
body := make([]byte, 2<<20)
|
||||
_, api, srv := serve(t, "v1.0.0", body, "s3cret")
|
||||
defer srv.Close()
|
||||
if _, err := (Source{URL: api}).Latest(context.Background(), "amd64"); err == nil || !strings.Contains(err.Error(), "update.token") {
|
||||
t.Fatalf("a walled source should say a token is needed, got %v", err)
|
||||
}
|
||||
rel, err := (Source{URL: api, Token: "s3cret"}).Latest(context.Background(), "amd64")
|
||||
if err != nil || rel.Version != "v1.0.0" {
|
||||
t.Fatalf("with the token: %v %+v", err, rel)
|
||||
}
|
||||
dest := filepath.Join(t.TempDir(), "hamlab.new")
|
||||
if err := (Source{URL: api, Token: "s3cret"}).Fetch(context.Background(), rel, "amd64", dest); err != nil {
|
||||
t.Fatalf("fetch with token: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
package sensors
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// promFamily names the metric family for a sensor and scales its value to the base unit
|
||||
// Prometheus convention wants (seconds, not minutes).
|
||||
func promFamily(s Sensor) (name string, scale float64) {
|
||||
scale = 1
|
||||
unit := map[string]string{"%": "percent", "°C": "celsius", "rpm": "rpm", "B/s": "bytes_per_second", "B": "bytes",
|
||||
"s": "seconds", "V": "volts", "W": "watts", "MHz": "megahertz"}[s.Unit]
|
||||
if s.Unit == "min" {
|
||||
unit, scale = "seconds", 60
|
||||
}
|
||||
if s.Kind == "count" { // hamlab_count{id=…} would say nothing; name it after the thing counted
|
||||
return "hamlab_" + strings.ReplaceAll(Seg(strings.ReplaceAll(s.ID, ".", "_")), "-", "_"), 1
|
||||
}
|
||||
name = "hamlab_" + s.Kind
|
||||
if unit != "" {
|
||||
name += "_" + unit
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
var promEsc = strings.NewReplacer(`\`, `\\`, `"`, `\"`, "\n", `\n`)
|
||||
|
||||
// WriteProm writes sensors in the Prometheus text exposition format: one family per kind
|
||||
// of measurement, the sensor's ID and its real-world labels as labels. Booleans are 0/1;
|
||||
// a text state is a 1 carrying the text as a label, the usual way to export an enum.
|
||||
func WriteProm(w io.Writer, all []Sensor) {
|
||||
fams := map[string][]string{}
|
||||
for _, s := range all {
|
||||
name, scale := promFamily(s)
|
||||
labels := []string{`id="` + promEsc.Replace(s.ID) + `"`}
|
||||
keys := make([]string, 0, len(s.Labels))
|
||||
for k := range s.Labels {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
if s.Labels[k] != "" {
|
||||
labels = append(labels, Seg(k)+`="`+promEsc.Replace(s.Labels[k])+`"`)
|
||||
}
|
||||
}
|
||||
var v float64
|
||||
switch x := s.Value.(type) {
|
||||
case float64:
|
||||
v = x * scale
|
||||
case bool:
|
||||
if x {
|
||||
v = 1
|
||||
}
|
||||
case string:
|
||||
v = 1
|
||||
labels = append(labels, `value="`+promEsc.Replace(x)+`"`)
|
||||
default:
|
||||
continue
|
||||
}
|
||||
fams[name] = append(fams[name], fmt.Sprintf("%s{%s} %v", name, strings.Join(labels, ","), v))
|
||||
}
|
||||
names := make([]string, 0, len(fams))
|
||||
for n := range fams {
|
||||
names = append(names, n)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, n := range names {
|
||||
fmt.Fprintf(w, "# TYPE %s gauge\n", n)
|
||||
for _, line := range fams[n] {
|
||||
fmt.Fprintln(w, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,275 @@
|
||||
// Package sensors is the one standard way anything reads what Ham Lab measures.
|
||||
//
|
||||
// Every measurable thing is a Sensor: a stable dotted ID ("cpu.core.3.usage",
|
||||
// "fan.nct6775.fan2", "fs.root.used_pct"), a little metadata, and a current value. A
|
||||
// consumer discovers them from the catalog, then asks for exactly the ones it wants —
|
||||
// one by ID, many by glob, or as a shaped query where the client sends the object it
|
||||
// wants and gets it back with the values filled in. See SENSORS.md.
|
||||
//
|
||||
// Sources of sensors are Providers. The registry only joins them; it measures nothing.
|
||||
package sensors
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Sensor is one measurable thing and its reading. Value is a float64, bool or string.
|
||||
type Sensor struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"` // human label
|
||||
Kind string `json:"kind"` // usage | load | temperature | fan | rate | bytes | duration | voltage | power | state
|
||||
Unit string `json:"unit,omitempty"` // % · °C · rpm · B/s · B · s · min · V · W · MHz
|
||||
Value any `json:"value"`
|
||||
T int64 `json:"t"` // unix ms the value was read — slow sensors are older than the tick
|
||||
Min *float64 `json:"min,omitempty"`
|
||||
Max *float64 `json:"max,omitempty"`
|
||||
Warn *float64 `json:"warn,omitempty"` // the hardware's own thresholds, where it states them
|
||||
Crit *float64 `json:"crit,omitempty"`
|
||||
Labels map[string]string `json:"labels,omitempty"` // the real names behind the ID: chip, iface, mount…
|
||||
}
|
||||
|
||||
// Series is a numeric sensor's recent past.
|
||||
type Series struct {
|
||||
ID string `json:"id"`
|
||||
Unit string `json:"unit,omitempty"`
|
||||
T []int64 `json:"t"`
|
||||
V []float64 `json:"v"`
|
||||
}
|
||||
|
||||
// Provider is a source of sensors. Sensors returns every sensor it has, with current values.
|
||||
type Provider interface {
|
||||
Sensors() []Sensor
|
||||
}
|
||||
|
||||
// Historian is a Provider that also remembers: the recent past of every numeric sensor
|
||||
// of its own that match accepts.
|
||||
type Historian interface {
|
||||
SensorHistories(match func(id string) bool) []Series
|
||||
}
|
||||
|
||||
// Ticker is a Provider that can say when its values have changed.
|
||||
type Ticker interface {
|
||||
Subscribe() (tick <-chan struct{}, cancel func())
|
||||
}
|
||||
|
||||
// Subscribe follows the first provider that ticks. ok is false when none does.
|
||||
func (r *Registry) Subscribe() (tick <-chan struct{}, cancel func(), ok bool) {
|
||||
for _, p := range r.providers {
|
||||
if t, is := p.(Ticker); is {
|
||||
tick, cancel = t.Subscribe()
|
||||
return tick, cancel, true
|
||||
}
|
||||
}
|
||||
return nil, nil, false
|
||||
}
|
||||
|
||||
// F is a pointer to a float, for the optional bounds.
|
||||
func F(v float64) *float64 { return &v }
|
||||
|
||||
// Seg makes one ID segment from a real-world name: lower case, [a-z0-9_-] only, so IDs
|
||||
// survive URLs, query strings and templating languages untouched. "Package id 0" →
|
||||
// "package_id_0", "/mnt/tank" → "mnt_tank", "/" → "root".
|
||||
func Seg(name string) string {
|
||||
if name == "/" {
|
||||
return "root"
|
||||
}
|
||||
var b strings.Builder
|
||||
gap := false
|
||||
for _, r := range strings.ToLower(strings.TrimSpace(name)) {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z', r >= '0' && r <= '9', r == '-':
|
||||
if gap && b.Len() > 0 {
|
||||
b.WriteByte('_')
|
||||
}
|
||||
gap = false
|
||||
b.WriteRune(r)
|
||||
default:
|
||||
gap = true
|
||||
}
|
||||
}
|
||||
if b.Len() == 0 {
|
||||
return "unnamed"
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// ID joins already-safe segments.
|
||||
func ID(segs ...string) string { return strings.Join(segs, ".") }
|
||||
|
||||
// Registry joins providers into one namespace.
|
||||
type Registry struct{ providers []Provider }
|
||||
|
||||
func NewRegistry(p ...Provider) *Registry { return &Registry{providers: p} }
|
||||
|
||||
func (r *Registry) Add(p Provider) { r.providers = append(r.providers, p) }
|
||||
|
||||
// All returns every sensor, sorted by ID. Two sensors that would share an ID (two
|
||||
// identical chips) get "_2", "_3" on the colliding one, in the provider's order.
|
||||
func (r *Registry) All() []Sensor {
|
||||
var out []Sensor
|
||||
seen := map[string]int{}
|
||||
for _, p := range r.providers {
|
||||
for _, s := range p.Sensors() {
|
||||
seen[s.ID]++
|
||||
if n := seen[s.ID]; n > 1 {
|
||||
s.ID = fmt.Sprintf("%s_%d", s.ID, n)
|
||||
}
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||
return out
|
||||
}
|
||||
|
||||
// History is one sensor's recent past. ok is false when nothing remembers it.
|
||||
func (r *Registry) History(id string) (Series, bool) {
|
||||
got := r.Histories(func(x string) bool { return x == id })
|
||||
if len(got) == 0 {
|
||||
return Series{}, false
|
||||
}
|
||||
return got[0], true
|
||||
}
|
||||
|
||||
// Histories is the recent past of every numeric sensor match accepts, sorted by ID.
|
||||
func (r *Registry) Histories(match func(id string) bool) []Series {
|
||||
out := []Series{}
|
||||
for _, p := range r.providers {
|
||||
if h, ok := p.(Historian); ok {
|
||||
out = append(out, h.SensorHistories(match)...)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||
return out
|
||||
}
|
||||
|
||||
// Matcher turns comma-separated selectors (IDs and globs) into a match function.
|
||||
func Matcher(selectors string) (func(id string) bool, error) {
|
||||
var sels []string
|
||||
for _, sel := range strings.Split(selectors, ",") {
|
||||
if sel = strings.TrimSpace(sel); sel == "" {
|
||||
continue
|
||||
}
|
||||
if _, err := path.Match(sel, ""); err != nil {
|
||||
return nil, fmt.Errorf("bad pattern %q", sel)
|
||||
}
|
||||
sels = append(sels, sel)
|
||||
}
|
||||
return func(id string) bool {
|
||||
for _, sel := range sels {
|
||||
if ok, _ := path.Match(sel, id); ok {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}, nil
|
||||
}
|
||||
|
||||
// IsGlob reports whether a selector names many sensors rather than one.
|
||||
func IsGlob(sel string) bool { return strings.ContainsAny(sel, "*?[") }
|
||||
|
||||
// Select returns the sensors a selector names: an exact ID, or a glob where * crosses
|
||||
// dots ("fan.*" is every fan, "cpu.core.*.usage" every core).
|
||||
func Select(all []Sensor, sel string) ([]Sensor, error) {
|
||||
if !IsGlob(sel) {
|
||||
for _, s := range all {
|
||||
if s.ID == sel {
|
||||
return []Sensor{s}, nil
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
if _, err := path.Match(sel, ""); err != nil {
|
||||
return nil, fmt.Errorf("bad pattern %q", sel)
|
||||
}
|
||||
var out []Sensor
|
||||
for _, s := range all {
|
||||
if ok, _ := path.Match(sel, s.ID); ok {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Shape answers a shaped query: the client sends the object it wants, with selectors
|
||||
// where the values go, and gets the same object back with them filled in.
|
||||
//
|
||||
// {"cpu": "cpu.usage", "fans": "fan.*", "box": {"up": "uptime"}}
|
||||
// → {"cpu": 12.4, "fans": {"fan.nct6775.fan1": 820}, "box": {"up": 80520}}
|
||||
//
|
||||
// An exact ID becomes its value (null if there is no such sensor here — machines differ,
|
||||
// and a dashboard shouldn't fail because one has no fans). A glob becomes an object keyed
|
||||
// by ID. Objects and arrays are walked. With full, every value is the whole Sensor.
|
||||
func Shape(all []Sensor, query json.RawMessage, full bool) (any, error) {
|
||||
var q any
|
||||
if err := json.Unmarshal(query, &q); err != nil {
|
||||
return nil, errors.New("query is not valid JSON")
|
||||
}
|
||||
budget := MaxValues
|
||||
return shape(all, q, full, 0, &budget)
|
||||
}
|
||||
|
||||
// MaxValues is how many sensor values one query may return. A query is a few bytes that
|
||||
// can ask for everything many times over ({"a":"*","b":"*",…}); without a ceiling 10 KB of
|
||||
// request buys 27 MB of response. Ten thousand is thirty times a whole machine.
|
||||
const MaxValues = 10000
|
||||
|
||||
func shape(all []Sensor, q any, full bool, depth int, budget *int) (any, error) {
|
||||
if depth > 16 {
|
||||
return nil, errors.New("query nested too deep")
|
||||
}
|
||||
leaf := func(s Sensor) any {
|
||||
if full {
|
||||
return s
|
||||
}
|
||||
return s.Value
|
||||
}
|
||||
switch v := q.(type) {
|
||||
case string:
|
||||
got, err := Select(all, v)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !IsGlob(v) {
|
||||
if len(got) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
*budget--
|
||||
return leaf(got[0]), nil
|
||||
}
|
||||
if *budget -= len(got); *budget < 0 {
|
||||
return nil, fmt.Errorf("query returns more than %d values; ask for less", MaxValues)
|
||||
}
|
||||
m := make(map[string]any, len(got))
|
||||
for _, s := range got {
|
||||
m[s.ID] = leaf(s)
|
||||
}
|
||||
return m, nil
|
||||
case map[string]any:
|
||||
m := make(map[string]any, len(v))
|
||||
for k, sub := range v {
|
||||
r, err := shape(all, sub, full, depth+1, budget)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
m[k] = r
|
||||
}
|
||||
return m, nil
|
||||
case []any:
|
||||
a := make([]any, len(v))
|
||||
for i, sub := range v {
|
||||
r, err := shape(all, sub, full, depth+1, budget)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a[i] = r
|
||||
}
|
||||
return a, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("a query holds sensor IDs, objects and arrays; got %v", v)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package sensors
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type fake []Sensor
|
||||
|
||||
func (f fake) Sensors() []Sensor { return f }
|
||||
|
||||
func TestSeg(t *testing.T) {
|
||||
for in, want := range map[string]string{"/": "root", "/mnt/tank": "mnt_tank", "Package id 0": "package_id_0", "enp3s0": "enp3s0", " ": "unnamed", "Tctl+": "tctl", "nvme0n1": "nvme0n1"} {
|
||||
if got := Seg(in); got != want {
|
||||
t.Errorf("Seg(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryDedupes(t *testing.T) {
|
||||
r := NewRegistry(fake{{ID: "temp.nvme.composite", Value: 40.0}}, fake{{ID: "temp.nvme.composite", Value: 50.0}})
|
||||
all := r.All()
|
||||
if len(all) != 2 || all[0].ID != "temp.nvme.composite" || all[1].ID != "temp.nvme.composite_2" {
|
||||
t.Fatalf("got %+v", all)
|
||||
}
|
||||
}
|
||||
|
||||
func TestShape(t *testing.T) {
|
||||
all := []Sensor{{ID: "cpu.usage", Value: 12.5, Unit: "%"}, {ID: "fan.a.fan1", Value: 800.0}, {ID: "fan.a.fan2", Value: 900.0}, {ID: "net.eth0.up", Value: true}}
|
||||
got, err := Shape(all, json.RawMessage(`{"cpu":"cpu.usage","fans":"fan.*","box":{"link":"net.eth0.up","gone":"ups.battery"},"list":["cpu.usage"]}`), false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[string]any{
|
||||
"cpu": 12.5,
|
||||
"fans": map[string]any{"fan.a.fan1": 800.0, "fan.a.fan2": 900.0},
|
||||
"box": map[string]any{"link": true, "gone": nil},
|
||||
"list": []any{12.5},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("got %#v\nwant %#v", got, want)
|
||||
}
|
||||
full, _ := Shape(all, json.RawMessage(`"cpu.usage"`), true)
|
||||
if s, ok := full.(Sensor); !ok || s.Unit != "%" {
|
||||
t.Fatalf("full: got %#v", full)
|
||||
}
|
||||
for _, bad := range []string{`{"x": 5}`, `{"x": "fan.["}`, `not json`} {
|
||||
if _, err := Shape(all, json.RawMessage(bad), false); err == nil {
|
||||
t.Errorf("Shape(%s) should fail", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteProm(t *testing.T) {
|
||||
var b strings.Builder
|
||||
WriteProm(&b, []Sensor{
|
||||
{ID: "cpu.usage", Kind: "usage", Unit: "%", Value: 12.5},
|
||||
{ID: "ups.runtime", Kind: "duration", Unit: "min", Value: 2.0},
|
||||
{ID: "net.eth0.up", Kind: "state", Value: true, Labels: map[string]string{"iface": "eth0"}},
|
||||
{ID: "ups.state", Kind: "state", Value: `Nor"mal`},
|
||||
{ID: "updates.pending", Kind: "count", Value: 3.0},
|
||||
})
|
||||
got := b.String()
|
||||
for _, want := range []string{
|
||||
"# TYPE hamlab_usage_percent gauge\n" + `hamlab_usage_percent{id="cpu.usage"} 12.5`,
|
||||
`hamlab_duration_seconds{id="ups.runtime"} 120`,
|
||||
`hamlab_state{id="net.eth0.up",iface="eth0"} 1`,
|
||||
`hamlab_state{id="ups.state",value="Nor\"mal"} 1`,
|
||||
`hamlab_updates_pending{id="updates.pending"} 3`,
|
||||
} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("missing %q in:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestShapeBudget(t *testing.T) {
|
||||
all := make([]Sensor, 300)
|
||||
for i := range all {
|
||||
all[i] = Sensor{ID: "s." + strings.Repeat("x", i%7) + string(rune('a'+i%26)) + strings.Repeat("y", i/26), Value: 1.0}
|
||||
}
|
||||
q := map[string]string{}
|
||||
for i := 0; i < 40; i++ { // 40 × 300 = 12 000 values
|
||||
q["k"+strings.Repeat("z", i)] = "*"
|
||||
}
|
||||
b, _ := json.Marshal(q)
|
||||
if _, err := Shape(all, b, false); err == nil {
|
||||
t.Fatal("a query returning more than MaxValues must be refused")
|
||||
}
|
||||
if _, err := Shape(all, json.RawMessage(`{"a":"*","b":"*"}`), false); err != nil {
|
||||
t.Fatalf("an ordinary query must pass: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,368 @@
|
||||
// Package services wraps systemctl and journalctl for the Services tab: list units,
|
||||
// show one, read its unit file, read and tail its journal, and start/stop/etc.
|
||||
//
|
||||
// Reads run as the daemon's user (systemd exposes state to everyone; the journal needs
|
||||
// the systemd-journal or adm group, or sudo). Actions go through passwordless sudo when
|
||||
// it's available, otherwise plain systemctl so polkit gets its say.
|
||||
package services
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type Unit struct {
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
Load string `json:"load"`
|
||||
Active string `json:"active"`
|
||||
Sub string `json:"sub"`
|
||||
Enabled string `json:"enabled"` // enabled | disabled | static | masked | … | "" (unknown)
|
||||
Failed bool `json:"failed"`
|
||||
}
|
||||
|
||||
type Detail struct {
|
||||
Unit
|
||||
Props map[string]string `json:"props"`
|
||||
}
|
||||
|
||||
type Entry struct {
|
||||
T int64 `json:"t"` // unix ms
|
||||
Priority int `json:"p"` // syslog priority 0-7
|
||||
Ident string `json:"ident"`
|
||||
PID string `json:"pid,omitempty"`
|
||||
Msg string `json:"msg"`
|
||||
}
|
||||
|
||||
var unitRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9@._:\\-]{0,255}\.service$`)
|
||||
|
||||
func Valid(unit string) bool { return unitRe.MatchString(unit) }
|
||||
|
||||
var Actions = map[string]bool{"start": true, "stop": true, "restart": true, "reload": true, "enable": true, "disable": true, "reset-failed": true}
|
||||
|
||||
// Properties shown on the detail page, in display order.
|
||||
var showProps = []string{"Id", "Description", "LoadState", "ActiveState", "SubState", "UnitFileState", "FragmentPath",
|
||||
"MainPID", "ExecMainStartTimestamp", "ActiveEnterTimestamp", "InactiveEnterTimestamp", "NRestarts", "Result",
|
||||
"MemoryCurrent", "MemoryPeak", "CPUUsageNSec", "TasksCurrent", "ExecStart", "User", "Restart"}
|
||||
|
||||
type Manager struct {
|
||||
sudo bool // actions via sudo -n
|
||||
journalSudo bool // journal needs sudo (user isn't in systemd-journal/adm)
|
||||
mu sync.Mutex
|
||||
cache []Unit
|
||||
cacheAt time.Time
|
||||
}
|
||||
|
||||
// ErrNoSystemd is what the Services tab says on OpenRC, runit and friends.
|
||||
var ErrNoSystemd = errors.New("this host doesn't run systemd, which the Services tab needs")
|
||||
|
||||
// HasSystemd reports whether systemd is the running init (the test sd_booted() uses).
|
||||
func HasSystemd() bool {
|
||||
_, err := os.Stat("/run/systemd/system")
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func New() *Manager {
|
||||
m := &Manager{}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
m.sudo = priv.Available()
|
||||
// Prefer reading the journal as ourselves: a plain child can be killed cleanly when a
|
||||
// tail ends, whereas a root journalctl under sudo can only be reached with sudo kill.
|
||||
var errb bytes.Buffer
|
||||
probe := exec.CommandContext(ctx, "journalctl", "--system", "-n", "1", "-q", "-o", "cat")
|
||||
probe.Stderr = &errb
|
||||
if err := probe.Run(); err != nil || strings.Contains(strings.ToLower(errb.String()), "permission") {
|
||||
m.journalSudo = m.sudo
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Sudo reports whether actions will run through passwordless sudo.
|
||||
func (m *Manager) Sudo() bool { return m.sudo }
|
||||
|
||||
// JournalSudo reports whether reading the system journal needs sudo for this user.
|
||||
func (m *Manager) JournalSudo() bool { return m.journalSudo }
|
||||
|
||||
func run(ctx context.Context, args ...string) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
var out, errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := strings.TrimSpace(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return out.String(), errors.New(msg)
|
||||
}
|
||||
return out.String(), nil
|
||||
}
|
||||
|
||||
// List merges `list-units --all` (runtime state) with `list-unit-files` (enablement).
|
||||
// Cached for a few seconds because the UI polls it.
|
||||
func (m *Manager) List(ctx context.Context) ([]Unit, error) {
|
||||
if !HasSystemd() {
|
||||
return nil, ErrNoSystemd
|
||||
}
|
||||
m.mu.Lock()
|
||||
if time.Since(m.cacheAt) < 200*time.Millisecond && m.cache != nil { // coalesce simultaneous clients only
|
||||
c := m.cache
|
||||
m.mu.Unlock()
|
||||
return c, nil
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
var out, files string
|
||||
var err error
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
files, _ = run(ctx, "systemctl", "list-unit-files", "--type=service", "--output=json", "--no-pager")
|
||||
close(done)
|
||||
}()
|
||||
out, err = run(ctx, "systemctl", "list-units", "--type=service", "--all", "--output=json", "--no-pager", "--plain")
|
||||
<-done
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var units []listedUnit
|
||||
if err := json.Unmarshal([]byte(out), &units); err != nil {
|
||||
// systemd before 246 (RHEL 8, Ubuntu 20.04) ignores --output=json here and prints
|
||||
// the table; the columns have been the same since the beginning.
|
||||
if units = parseUnitTable(out); len(units) == 0 {
|
||||
return nil, fmt.Errorf("parse list-units: %w", err)
|
||||
}
|
||||
}
|
||||
var uf []listedFile
|
||||
if json.Unmarshal([]byte(files), &uf) != nil {
|
||||
uf = parseFileTable(files)
|
||||
}
|
||||
enabled := map[string]string{}
|
||||
for _, f := range uf {
|
||||
enabled[f.UnitFile] = f.State
|
||||
}
|
||||
|
||||
seen := map[string]bool{}
|
||||
list := make([]Unit, 0, len(units)+len(uf))
|
||||
for _, u := range units {
|
||||
if strings.Contains(u.Unit, "@.") { // uninstantiated template
|
||||
continue
|
||||
}
|
||||
seen[u.Unit] = true
|
||||
list = append(list, Unit{Name: u.Unit, Description: u.Description, Load: u.Load, Active: u.Active, Sub: u.Sub,
|
||||
Enabled: enabled[u.Unit], Failed: u.Active == "failed"})
|
||||
}
|
||||
// Unit files that exist but were never loaded this boot (disabled and never started).
|
||||
for _, f := range uf {
|
||||
if seen[f.UnitFile] || strings.Contains(f.UnitFile, "@.") || f.State == "masked" || f.State == "alias" {
|
||||
continue
|
||||
}
|
||||
list = append(list, Unit{Name: f.UnitFile, Load: "not-loaded", Active: "inactive", Sub: "dead", Enabled: f.State})
|
||||
}
|
||||
sort.Slice(list, func(i, j int) bool { return list[i].Name < list[j].Name })
|
||||
|
||||
m.mu.Lock()
|
||||
m.cache, m.cacheAt = list, time.Now()
|
||||
m.mu.Unlock()
|
||||
return list, nil
|
||||
}
|
||||
|
||||
type listedUnit struct{ Unit, Load, Active, Sub, Description string }
|
||||
|
||||
type listedFile struct {
|
||||
UnitFile string `json:"unit_file"`
|
||||
State string `json:"state"`
|
||||
}
|
||||
|
||||
// parseUnitTable reads the plain `systemctl list-units` table: UNIT LOAD ACTIVE SUB DESCRIPTION…
|
||||
func parseUnitTable(text string) []listedUnit {
|
||||
var units []listedUnit
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
f := strings.Fields(strings.TrimLeft(line, "●*× "))
|
||||
if len(f) < 4 || !strings.HasSuffix(f[0], ".service") {
|
||||
continue
|
||||
}
|
||||
units = append(units, listedUnit{Unit: f[0], Load: f[1], Active: f[2], Sub: f[3], Description: strings.Join(f[4:], " ")})
|
||||
}
|
||||
return units
|
||||
}
|
||||
|
||||
// parseFileTable reads the plain `systemctl list-unit-files` table: UNIT FILE, STATE[, PRESET]
|
||||
func parseFileTable(text string) []listedFile {
|
||||
var uf []listedFile
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
if f := strings.Fields(line); len(f) >= 2 && strings.HasSuffix(f[0], ".service") {
|
||||
uf = append(uf, listedFile{UnitFile: f[0], State: f[1]})
|
||||
}
|
||||
}
|
||||
return uf
|
||||
}
|
||||
|
||||
func (m *Manager) invalidate() {
|
||||
m.mu.Lock()
|
||||
m.cacheAt = time.Time{}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
func (m *Manager) Show(ctx context.Context, unit string) (Detail, error) {
|
||||
if !Valid(unit) {
|
||||
return Detail{}, errors.New("invalid unit name")
|
||||
}
|
||||
out, err := run(ctx, "systemctl", "show", unit, "-p", strings.Join(showProps, ","), "--no-pager")
|
||||
if err != nil {
|
||||
return Detail{}, err
|
||||
}
|
||||
props := map[string]string{}
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if k, v, ok := strings.Cut(line, "="); ok && v != "" && v != "[not set]" && v != "infinity" {
|
||||
props[k] = v
|
||||
}
|
||||
}
|
||||
if props["LoadState"] == "not-found" {
|
||||
return Detail{}, errors.New("no such unit")
|
||||
}
|
||||
d := Detail{Props: props}
|
||||
d.Name, d.Description = unit, props["Description"]
|
||||
d.Load, d.Active, d.Sub, d.Enabled = props["LoadState"], props["ActiveState"], props["SubState"], props["UnitFileState"]
|
||||
d.Failed = d.Active == "failed"
|
||||
return d, nil
|
||||
}
|
||||
|
||||
func (m *Manager) Cat(ctx context.Context, unit string) (string, error) {
|
||||
if !Valid(unit) {
|
||||
return "", errors.New("invalid unit name")
|
||||
}
|
||||
return run(ctx, "systemctl", "cat", unit, "--no-pager")
|
||||
}
|
||||
|
||||
func parseEntry(line []byte) (Entry, bool) {
|
||||
var raw map[string]any
|
||||
if json.Unmarshal(line, &raw) != nil {
|
||||
return Entry{}, false
|
||||
}
|
||||
e := Entry{Priority: 6}
|
||||
switch v := raw["MESSAGE"].(type) {
|
||||
case string:
|
||||
e.Msg = v
|
||||
case []any: // non-UTF-8 payload comes as a byte array
|
||||
b := make([]byte, 0, len(v))
|
||||
for _, x := range v {
|
||||
if f, ok := x.(float64); ok {
|
||||
b = append(b, byte(f))
|
||||
}
|
||||
}
|
||||
e.Msg = string(b)
|
||||
default:
|
||||
e.Msg = ""
|
||||
}
|
||||
if p, ok := raw["PRIORITY"].(string); ok {
|
||||
e.Priority, _ = strconv.Atoi(p)
|
||||
}
|
||||
if ts, ok := raw["__REALTIME_TIMESTAMP"].(string); ok {
|
||||
us, _ := strconv.ParseInt(ts, 10, 64)
|
||||
e.T = us / 1000
|
||||
}
|
||||
if id, ok := raw["SYSLOG_IDENTIFIER"].(string); ok {
|
||||
e.Ident = id
|
||||
} else if c, ok := raw["_COMM"].(string); ok {
|
||||
e.Ident = c
|
||||
}
|
||||
if pid, ok := raw["_PID"].(string); ok {
|
||||
e.PID = pid
|
||||
}
|
||||
return e, true
|
||||
}
|
||||
|
||||
func (m *Manager) journalArgs(unit string, n int) []string {
|
||||
args := []string{"journalctl", "-u", unit, "-n", strconv.Itoa(n), "-o", "json", "--no-pager", "--all"}
|
||||
if m.journalSudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
// Journal returns the last n entries.
|
||||
func (m *Manager) Journal(ctx context.Context, unit string, n int) ([]Entry, error) {
|
||||
if !Valid(unit) {
|
||||
return nil, errors.New("invalid unit name")
|
||||
}
|
||||
out, err := run(ctx, m.journalArgs(unit, n)...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries := []Entry{}
|
||||
for _, line := range bytes.Split([]byte(out), []byte("\n")) {
|
||||
if e, ok := parseEntry(line); ok {
|
||||
entries = append(entries, e)
|
||||
}
|
||||
}
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
// Tail streams entries (last n, then follow) to fn until ctx is cancelled.
|
||||
func (m *Manager) Tail(ctx context.Context, unit string, n int, fn func(Entry)) error {
|
||||
if !Valid(unit) {
|
||||
return errors.New("invalid unit name")
|
||||
}
|
||||
args := append(m.journalArgs(unit, n), "-f")
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
// Own process group so the kill reaches journalctl itself, not just a sudo wrapper.
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error {
|
||||
pg := "-" + strconv.Itoa(cmd.Process.Pid)
|
||||
if m.journalSudo { // children are root's; only sudo can signal them
|
||||
k := priv.Wrap("kill", "-9", "--", pg)
|
||||
return exec.Command(k[0], k[1:]...).Run()
|
||||
}
|
||||
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||
}
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
}
|
||||
sc := bufio.NewScanner(stdout)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
|
||||
for sc.Scan() {
|
||||
if e, ok := parseEntry(sc.Bytes()); ok {
|
||||
fn(e)
|
||||
}
|
||||
}
|
||||
_ = cmd.Wait()
|
||||
return nil
|
||||
}
|
||||
|
||||
// Action runs start/stop/restart/reload/enable/disable/reset-failed.
|
||||
func (m *Manager) Action(ctx context.Context, action, unit string) error {
|
||||
if !Valid(unit) {
|
||||
return errors.New("invalid unit name")
|
||||
}
|
||||
if !Actions[action] {
|
||||
return errors.New("unknown action")
|
||||
}
|
||||
args := []string{"systemctl", action, unit, "--no-pager"}
|
||||
if m.sudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
_, err := run(ctx, args...)
|
||||
m.invalidate()
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
package services
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestTables(t *testing.T) {
|
||||
u := parseUnitTable(" UNIT LOAD ACTIVE SUB DESCRIPTION\n● bad.service loaded failed failed Bad thing here\n ssh.service loaded active running OpenBSD Secure Shell server\n\nLOAD = x\n12 loaded units listed.\n")
|
||||
if len(u) != 2 || u[0].Unit != "bad.service" || u[0].Active != "failed" || u[1].Description != "OpenBSD Secure Shell server" {
|
||||
t.Fatalf("%+v", u)
|
||||
}
|
||||
f := parseFileTable("UNIT FILE STATE VENDOR PRESET\nssh.service enabled enabled\nfoo@.service static -\n\n2 unit files listed.\n")
|
||||
if len(f) != 2 || f[0].State != "enabled" {
|
||||
t.Fatalf("%+v", f)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
#!/sbin/openrc-run
|
||||
# Ham Lab web console (written by hamlab install)
|
||||
|
||||
name="hamlab"
|
||||
description="Ham Lab web console"
|
||||
|
||||
export HAMLAB_CONFIG="@CONFIG@"
|
||||
export HOME="@HOME@"
|
||||
export USER="@USER@"
|
||||
|
||||
supervisor="supervise-daemon"
|
||||
command="@BIN@"
|
||||
command_args="serve"
|
||||
command_user="@USER@:@GROUP@"
|
||||
directory="@HOME@"
|
||||
respawn_delay=3
|
||||
respawn_max=0
|
||||
output_log="/var/log/hamlab.log"
|
||||
error_log="/var/log/hamlab.log"
|
||||
|
||||
depend() {
|
||||
need localmount
|
||||
after net firewall
|
||||
}
|
||||
|
||||
start_pre() {
|
||||
checkpath -f -o "@USER@:@GROUP@" -m 0640 /var/log/hamlab.log
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
#!/bin/sh /etc/rc.common
|
||||
# Ham Lab web console (written by hamlab install)
|
||||
|
||||
USE_PROCD=1
|
||||
# after network and firewall, before the tail of the boot
|
||||
START=95
|
||||
STOP=10
|
||||
|
||||
start_service() {
|
||||
procd_open_instance hamlab
|
||||
procd_set_param command "@BIN@" serve
|
||||
procd_set_param env HAMLAB_CONFIG="@CONFIG@" HOME="@HOME@" USER="@USER@"
|
||||
procd_set_param user "@USER@"
|
||||
procd_set_param group "@GROUP@"
|
||||
procd_set_param pidfile /var/run/hamlab.pid
|
||||
# respawn: settle 3s, 5 retries inside that window, then give up counting
|
||||
procd_set_param respawn 3600 3 0
|
||||
procd_set_param stdout 1
|
||||
procd_set_param stderr 1
|
||||
procd_close_instance
|
||||
}
|
||||
|
||||
service_triggers() {
|
||||
procd_add_reload_trigger network
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
[Unit]
|
||||
Description=Ham Lab web console
|
||||
After=network.target
|
||||
# never give up restarting
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=@USER@
|
||||
Group=@GROUP@
|
||||
# read the system journal without sudo (keeps the daemon's own log clean)
|
||||
SupplementaryGroups=@GROUPS@
|
||||
WorkingDirectory=-@HOME@
|
||||
Environment=HAMLAB_CONFIG=@CONFIG@
|
||||
ExecStart=@BIN@ serve
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
KillMode=control-group
|
||||
TimeoutStopSec=10
|
||||
LimitNOFILE=65536
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,718 @@
|
||||
// Package setup is `hamlab install` and `hamlab uninstall`: the binary installs itself, so
|
||||
// a new machine needs nothing but this one file. It copies itself into place, writes the
|
||||
// config, sets the login password and registers a service with whichever init the host
|
||||
// runs (systemd or OpenRC), as root or through sudo/doas.
|
||||
package setup
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"crypto/rand"
|
||||
"crypto/tls"
|
||||
_ "embed"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"golang.org/x/term"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
)
|
||||
|
||||
//go:embed hamlab.service
|
||||
var systemdUnit string
|
||||
|
||||
//go:embed hamlab.openrc
|
||||
var openrcScript string
|
||||
|
||||
//go:embed hamlab.procd
|
||||
var procdScript string
|
||||
|
||||
const (
|
||||
unitPath = "/etc/systemd/system/hamlab.service"
|
||||
openrcPath = "/etc/init.d/hamlab"
|
||||
// procd (OpenWrt) uses the same path as OpenRC; the two are told apart by content.
|
||||
procdPath = "/etc/init.d/hamlab"
|
||||
sudoersPath = "/etc/sudoers.d/hamlab-quiet"
|
||||
)
|
||||
|
||||
type options struct {
|
||||
user, listen, tls, prefix, data string
|
||||
passwordStdin, openFirewall bool
|
||||
noStart bool
|
||||
}
|
||||
|
||||
func say(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) }
|
||||
|
||||
// elevate re-runs this command as root through sudo or doas (which may prompt), naming
|
||||
// the invoking user so the service still runs as them.
|
||||
func elevate(sub string, args []string, keepUser bool) error {
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, tool := range []string{"sudo", "doas"} {
|
||||
path, err := exec.LookPath(tool)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
full := []string{tool, self, sub}
|
||||
if u, err := user.Current(); err == nil && keepUser && !hasFlag(args, "user") {
|
||||
full = append(full, "--user", u.Username)
|
||||
}
|
||||
full = append(full, args...)
|
||||
return syscall.Exec(path, full, os.Environ())
|
||||
}
|
||||
return errors.New("this needs root, and neither sudo nor doas is installed: run it as root")
|
||||
}
|
||||
|
||||
func hasFlag(args []string, name string) bool {
|
||||
for _, a := range args {
|
||||
a = strings.TrimLeft(a, "-")
|
||||
if a == name || strings.HasPrefix(a, name+"=") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Install is `hamlab install [flags]`.
|
||||
func Install(args []string, version string) error {
|
||||
var o options
|
||||
fs := flag.NewFlagSet("install", flag.ContinueOnError)
|
||||
fs.StringVar(&o.user, "user", "", "user the console runs as; terminals are this user's shell (default: you)")
|
||||
fs.StringVar(&o.listen, "listen", "", "listen address, e.g. 0.0.0.0:8100 (default: keep, else 127.0.0.1:8100)")
|
||||
fs.StringVar(&o.tls, "tls", "", `"auto" serves HTTPS with a self-signed certificate, "off" plain HTTP (default: keep)`)
|
||||
fs.StringVar(&o.prefix, "prefix", "/usr/local/bin", "directory the binary is installed into")
|
||||
fs.StringVar(&o.data, "data", "", "config and state directory (default: ~/.config/hamlab of the user)")
|
||||
fs.BoolVar(&o.passwordStdin, "password-stdin", false, "read the login password from stdin instead of prompting")
|
||||
fs.BoolVar(&o.openFirewall, "open-firewall", false, "open the listen port in firewalld/ufw when one is active")
|
||||
fs.BoolVar(&o.noStart, "no-start", false, "install and enable, but don't start the service")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if os.Geteuid() != 0 {
|
||||
return elevate("install", args, true)
|
||||
}
|
||||
sbinPath()
|
||||
|
||||
// who
|
||||
name := o.user
|
||||
for _, env := range []string{"SUDO_USER", "DOAS_USER"} {
|
||||
if name == "" {
|
||||
name = os.Getenv(env)
|
||||
}
|
||||
}
|
||||
if name == "" {
|
||||
name = "root"
|
||||
}
|
||||
acct, err := lookupUser(name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// where
|
||||
dataDir := o.data
|
||||
if dataDir == "" {
|
||||
dataDir = filepath.Dir(existingConfig(acct.home))
|
||||
}
|
||||
cfgPath := filepath.Join(dataDir, "config.json")
|
||||
|
||||
// binary
|
||||
dest := filepath.Join(o.prefix, "hamlab")
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
self, _ = filepath.EvalSymlinks(self)
|
||||
if err := os.MkdirAll(o.prefix, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if self != dest {
|
||||
if err := CopyAtomic(self, dest); err != nil {
|
||||
return fmt.Errorf("install binary: %w", err)
|
||||
}
|
||||
// builds for other architectures sit beside it so a hub can push them to matching machines
|
||||
sibs, _ := filepath.Glob(filepath.Join(filepath.Dir(self), "hamlab-linux-*"))
|
||||
for _, sib := range sibs {
|
||||
arch := strings.TrimPrefix(filepath.Base(sib), "hamlab-linux-")
|
||||
if sib == self || strings.Contains(arch, ".") {
|
||||
continue
|
||||
}
|
||||
if err := CopyAtomic(sib, dest+"-"+arch); err != nil {
|
||||
say("warning: %s: %v", sib, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
restorecon(dest)
|
||||
say("binary %s (%s)", dest, version)
|
||||
|
||||
// config + password
|
||||
cfg, err := config.Load(cfgPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.listen != "" {
|
||||
if _, _, err := net.SplitHostPort(o.listen); err != nil {
|
||||
return fmt.Errorf("--listen %q: want host:port", o.listen)
|
||||
}
|
||||
cfg.Listen = o.listen
|
||||
}
|
||||
switch o.tls {
|
||||
case "":
|
||||
case "auto":
|
||||
cfg.TLS = "auto"
|
||||
case "off", "none":
|
||||
cfg.TLS = ""
|
||||
default:
|
||||
return fmt.Errorf("--tls %q: want auto or off", o.tls)
|
||||
}
|
||||
generated := ""
|
||||
cfg.MigrateUsers(acct.name)
|
||||
if len(cfg.Users) == 0 || o.passwordStdin {
|
||||
pw, gen, err := choosePassword(o.passwordStdin)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h, err := auth.HashPassword(pw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// the first account is named after the OS user the service runs as; --password-stdin
|
||||
// on an existing install resets that first account
|
||||
if len(cfg.Users) == 0 {
|
||||
cfg.Users = []config.User{{Name: acct.name, Hash: h, Created: time.Now()}}
|
||||
} else {
|
||||
cfg.Users[0].Hash = h
|
||||
}
|
||||
if gen {
|
||||
generated = pw
|
||||
}
|
||||
}
|
||||
made := firstMissing(dataDir) // e.g. ~/.config, when this is the first thing to use it
|
||||
if err := cfg.Save(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := chownData(dataDir, made, acct); err != nil {
|
||||
return err
|
||||
}
|
||||
say("config %s", cfgPath)
|
||||
|
||||
// service
|
||||
vars := map[string]string{"@USER@": acct.name, "@GROUP@": acct.group, "@HOME@": acct.home, "@BIN@": dest, "@CONFIG@": cfgPath,
|
||||
"@GROUPS@": journalGroups()}
|
||||
initSys := ""
|
||||
switch {
|
||||
case isDir("/run/systemd/system"):
|
||||
initSys = "systemd"
|
||||
unit := fill(systemdUnit, vars)
|
||||
if vars["@GROUPS@"] == "" {
|
||||
unit = dropLines(unit, "SupplementaryGroups=")
|
||||
}
|
||||
if err := os.WriteFile(unitPath, []byte(unit), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
restorecon(unitPath)
|
||||
if err := runAll([][]string{{"systemctl", "daemon-reload"}, {"systemctl", "enable", "hamlab.service"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
if !o.noStart {
|
||||
if err := run("systemctl", "restart", "hamlab.service"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
case exists("/sbin/procd") && exists("/etc/rc.common"):
|
||||
initSys = "procd"
|
||||
if err := os.WriteFile(procdPath, []byte(fill(procdScript, vars)), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := run(procdPath, "enable"); err != nil {
|
||||
say("warning: %s enable: %v", procdPath, err)
|
||||
}
|
||||
if !o.noStart {
|
||||
if err := run(procdPath, "restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
case exists("/sbin/openrc-run") || exists("/usr/sbin/openrc-run") || have("rc-service"):
|
||||
initSys = "OpenRC"
|
||||
if err := os.WriteFile(openrcPath, []byte(fill(openrcScript, vars)), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := run("rc-update", "add", "hamlab", "default"); err != nil {
|
||||
say("warning: rc-update: %v", err)
|
||||
}
|
||||
if !o.noStart {
|
||||
if err := run("rc-service", "hamlab", "restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if initSys != "" {
|
||||
say("service hamlab (%s), runs as %s, starts at boot", initSys, acct.name)
|
||||
}
|
||||
if acct.name != "root" {
|
||||
quietSudoers()
|
||||
}
|
||||
|
||||
port := firewall(cfg.Listen, o.openFirewall)
|
||||
|
||||
scheme := "http"
|
||||
if cfg.TLS == "auto" {
|
||||
scheme = "https"
|
||||
}
|
||||
url := scheme + "://" + browseAddr(cfg.Listen)
|
||||
switch {
|
||||
case initSys == "":
|
||||
say("\nNo systemd, procd or OpenRC here, so nothing starts it at boot.\nRun it under your own supervisor, as %s:\n HAMLAB_CONFIG=%s %s serve", acct.name, cfgPath, dest)
|
||||
case o.noStart:
|
||||
case healthy(cfg.Listen, cfg.TLS == "auto"):
|
||||
say("\nHam Lab is up: %s", url)
|
||||
default:
|
||||
say("\nInstalled, but %s isn't answering yet. Check: %s", url, map[string]string{"systemd": "journalctl -u hamlab -n 30", "OpenRC": "tail /var/log/hamlab.log"}[initSys])
|
||||
}
|
||||
if generated != "" {
|
||||
say("Login: user %s, password (shown once; change it in Settings): %s", cfg.Users[0].Name, generated)
|
||||
}
|
||||
if host, _, _ := net.SplitHostPort(cfg.Listen); isLoopback(host) {
|
||||
say("It only listens on this machine. For LAN access: hamlab install --listen 0.0.0.0:%s", port)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Uninstall is `hamlab uninstall [--purge]`.
|
||||
func Uninstall(args []string) error {
|
||||
fs := flag.NewFlagSet("uninstall", flag.ContinueOnError)
|
||||
purge := fs.Bool("purge", false, "also delete the config and state directory")
|
||||
prefix := fs.String("prefix", "/usr/local/bin", "directory the binary was installed into")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if os.Geteuid() != 0 {
|
||||
return elevate("uninstall", args, false)
|
||||
}
|
||||
sbinPath()
|
||||
dataDir := ""
|
||||
if b, err := os.ReadFile(unitPath); err == nil {
|
||||
dataDir = configDirFrom(string(b), "Environment=HAMLAB_CONFIG=")
|
||||
_ = run("systemctl", "disable", "--now", "hamlab.service")
|
||||
os.Remove(unitPath)
|
||||
_ = run("systemctl", "daemon-reload")
|
||||
}
|
||||
if b, err := os.ReadFile(openrcPath); err == nil {
|
||||
if strings.Contains(string(b), "USE_PROCD=1") {
|
||||
dataDir = configDirFrom(string(b), "HAMLAB_CONFIG=")
|
||||
_ = run(procdPath, "stop")
|
||||
_ = run(procdPath, "disable")
|
||||
} else {
|
||||
dataDir = configDirFrom(string(b), "export HAMLAB_CONFIG=")
|
||||
_ = run("rc-service", "hamlab", "stop")
|
||||
_ = run("rc-update", "del", "hamlab", "default")
|
||||
}
|
||||
os.Remove(openrcPath)
|
||||
}
|
||||
os.Remove(sudoersPath)
|
||||
bins, _ := filepath.Glob(filepath.Join(*prefix, "hamlab-*"))
|
||||
for _, b := range append(bins, filepath.Join(*prefix, "hamlab")) {
|
||||
os.Remove(b)
|
||||
}
|
||||
say("Ham Lab removed.")
|
||||
switch {
|
||||
case dataDir == "":
|
||||
case *purge:
|
||||
if err := purgeData(dataDir); err != nil {
|
||||
return err
|
||||
}
|
||||
default:
|
||||
say("Config and state kept in %s (uninstall --purge deletes them).", dataDir)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// sbinPath makes sure the admin tools resolve: a root shell reached with plain `su` keeps
|
||||
// the user's PATH, which on Debian has no sbin in it.
|
||||
func sbinPath() {
|
||||
os.Setenv("PATH", os.Getenv("PATH")+":/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin")
|
||||
}
|
||||
|
||||
// ownFiles is everything Ham Lab writes into its data directory.
|
||||
var ownFiles = []string{"config.json", "config.json.tmp", "state.json", "state.json.tmp", "state.json.corrupt", "cert.pem", "key.pem", "hamlab.prev", "hamlab.new"}
|
||||
|
||||
// purgeData deletes Ham Lab's data. The whole directory goes only when it is unmistakably
|
||||
// ours (named "hamlab"); a --data pointed at a shared directory such as a home loses
|
||||
// just the files Ham Lab wrote.
|
||||
func purgeData(dir string) error {
|
||||
if filepath.Base(dir) == "hamlab" {
|
||||
say("Deleted %s.", dir)
|
||||
return os.RemoveAll(dir)
|
||||
}
|
||||
for _, f := range ownFiles {
|
||||
os.Remove(filepath.Join(dir, f))
|
||||
}
|
||||
say("Deleted Ham Lab's files in %s (the directory isn't Ham Lab's alone, so it stays).", dir)
|
||||
return nil
|
||||
}
|
||||
|
||||
func configDirFrom(text, key string) string {
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
if v, ok := strings.CutPrefix(strings.TrimSpace(line), key); ok {
|
||||
if v = strings.Trim(v, `"'`); strings.HasSuffix(v, "/config.json") && filepath.Dir(v) != "/" {
|
||||
return filepath.Dir(v)
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- pieces ----
|
||||
|
||||
type account struct {
|
||||
name, group, home string
|
||||
uid, gid int
|
||||
}
|
||||
|
||||
func lookupUser(name string) (account, error) {
|
||||
a := account{name: name}
|
||||
if u, err := user.Lookup(name); err == nil {
|
||||
a.uid, _ = strconv.Atoi(u.Uid)
|
||||
a.gid, _ = strconv.Atoi(u.Gid)
|
||||
a.home = u.HomeDir
|
||||
} else if out, err2 := exec.Command("getent", "passwd", name).Output(); err2 == nil { // LDAP/SSSD accounts
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if len(f) < 6 {
|
||||
return a, fmt.Errorf("no such user: %s", name)
|
||||
}
|
||||
a.uid, _ = strconv.Atoi(f[2])
|
||||
a.gid, _ = strconv.Atoi(f[3])
|
||||
a.home = f[5]
|
||||
} else {
|
||||
return a, fmt.Errorf("no such user: %s", name)
|
||||
}
|
||||
if a.home == "" || !isDir(a.home) {
|
||||
return a, fmt.Errorf("user %s has no home directory (%q); pick another with --user", name, a.home)
|
||||
}
|
||||
a.group = strconv.Itoa(a.gid)
|
||||
if g, err := user.LookupGroupId(a.group); err == nil {
|
||||
a.group = g.Name
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// existingConfig returns the config path for a home directory: whichever already exists,
|
||||
// else the default for a new install.
|
||||
func existingConfig(home string) string {
|
||||
for _, p := range config.Candidates(home) {
|
||||
if exists(p) {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return config.Candidates(home)[0]
|
||||
}
|
||||
|
||||
// journalGroups lists the groups that can read the system journal and exist here. A unit
|
||||
// naming a group the host doesn't have fails to start, and "adm" is not universal.
|
||||
func journalGroups() string {
|
||||
var have []string
|
||||
for _, g := range []string{"systemd-journal", "adm"} {
|
||||
if _, err := user.LookupGroup(g); err == nil {
|
||||
have = append(have, g)
|
||||
}
|
||||
}
|
||||
return strings.Join(have, " ")
|
||||
}
|
||||
|
||||
func choosePassword(fromStdin bool) (pw string, generated bool, err error) {
|
||||
switch {
|
||||
case fromStdin:
|
||||
line, _ := bufio.NewReader(os.Stdin).ReadString('\n')
|
||||
pw = strings.TrimRight(line, "\r\n")
|
||||
case term.IsTerminal(int(os.Stdin.Fd())):
|
||||
fmt.Fprint(os.Stderr, "Login password for the web console (12+ characters, empty to generate one): ")
|
||||
p1, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
if len(p1) > 0 {
|
||||
fmt.Fprint(os.Stderr, "Again: ")
|
||||
p2, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
if !bytes.Equal(p1, p2) {
|
||||
return "", false, errors.New("passwords do not match")
|
||||
}
|
||||
}
|
||||
pw = string(p1)
|
||||
}
|
||||
if pw == "" && !fromStdin {
|
||||
return randomPassword(), true, nil
|
||||
}
|
||||
if len(pw) < 12 {
|
||||
return "", false, errors.New("password must be at least 12 characters")
|
||||
}
|
||||
return pw, false, nil
|
||||
}
|
||||
|
||||
func randomPassword() string {
|
||||
const alphabet = "abcdefghijkmnpqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ23456789" // no look-alikes
|
||||
b := make([]byte, 20)
|
||||
_, _ = rand.Read(b)
|
||||
for i := range b {
|
||||
b[i] = alphabet[int(b[i])%len(alphabet)]
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// CopyAtomic puts src at dst with mode 0755 by writing beside it and renaming, so dst is
|
||||
// always either the old file or the whole new one, and a running dst is never written to.
|
||||
func CopyAtomic(src, dst string) error {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer in.Close()
|
||||
tmp := filepath.Join(filepath.Dir(dst), "."+filepath.Base(dst)+".new")
|
||||
out, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = io.Copy(out, in)
|
||||
if err == nil {
|
||||
err = out.Sync()
|
||||
}
|
||||
if cerr := out.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err == nil {
|
||||
err = os.Chmod(tmp, 0o755)
|
||||
}
|
||||
if err == nil {
|
||||
err = os.Rename(tmp, dst)
|
||||
}
|
||||
if err != nil {
|
||||
os.Remove(tmp)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// firstMissing returns the topmost directory that creating dir would have to make, or "".
|
||||
func firstMissing(dir string) string {
|
||||
missing := ""
|
||||
for d := filepath.Clean(dir); d != "/" && d != "." && !exists(d); d = filepath.Dir(d) {
|
||||
missing = d
|
||||
}
|
||||
return missing
|
||||
}
|
||||
|
||||
// chownData hands Ham Lab's data to the account that runs it. Directories this install
|
||||
// created are ours to hand over whole. In one that already existed (a --data pointed at
|
||||
// something shared), only the files Ham Lab writes change owner: a recursive chown of
|
||||
// somebody else's tree can't be undone.
|
||||
func chownData(dataDir, made string, a account) error {
|
||||
if made != "" {
|
||||
return chownTree(made, a)
|
||||
}
|
||||
if filepath.Base(dataDir) == "hamlab" {
|
||||
return chownTree(dataDir, a)
|
||||
}
|
||||
for _, f := range ownFiles {
|
||||
if p := filepath.Join(dataDir, f); exists(p) {
|
||||
if err := os.Lchown(p, a.uid, a.gid); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chownTree(dir string, a account) error {
|
||||
return filepath.Walk(dir, func(p string, _ os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Lchown(p, a.uid, a.gid)
|
||||
})
|
||||
}
|
||||
|
||||
func fill(tmpl string, vars map[string]string) string {
|
||||
for k, v := range vars {
|
||||
tmpl = strings.ReplaceAll(tmpl, k, v)
|
||||
}
|
||||
return tmpl
|
||||
}
|
||||
|
||||
func dropLines(text, prefix string) string {
|
||||
var keep []string
|
||||
for _, l := range strings.Split(text, "\n") {
|
||||
if !strings.HasPrefix(l, prefix) {
|
||||
keep = append(keep, l)
|
||||
}
|
||||
}
|
||||
return strings.Join(keep, "\n")
|
||||
}
|
||||
|
||||
// quietSudoers keeps the daemon's read-only polling (SMART, UPS, journal) out of the auth
|
||||
// log: at 250 ms refresh rates the sudo session lines would otherwise dominate it.
|
||||
func quietSudoers() {
|
||||
if !isDir("/etc/sudoers.d") || !have("visudo") {
|
||||
return
|
||||
}
|
||||
var cmds []string
|
||||
for _, bin := range []string{"smartctl", "pwrstat", "journalctl", "tail", "cat", "ls", "crontab"} {
|
||||
for _, dir := range []string{"/usr/sbin", "/usr/bin", "/sbin", "/bin"} {
|
||||
if exists(dir + "/" + bin) {
|
||||
cmds = append(cmds, dir+"/"+bin)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, bin := range []string{"zpool", "zfs"} { // listing only: scrubs and snapshots still log
|
||||
for _, dir := range []string{"/usr/sbin", "/sbin", "/usr/bin", "/bin"} {
|
||||
if exists(dir + "/" + bin) {
|
||||
cmds = append(cmds, dir+"/"+bin+" list *", dir+"/"+bin+" status *")
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(cmds) == 0 {
|
||||
return
|
||||
}
|
||||
text := "# Ham Lab: don't log the daemon's read-only polling commands (written by hamlab install)\n" +
|
||||
"Cmnd_Alias HAMLAB_QUIET = " + strings.Join(cmds, ", ") + "\n" +
|
||||
"Defaults!HAMLAB_QUIET !syslog, !pam_session, !pam_setcred\n"
|
||||
tmp := sudoersPath + ".new"
|
||||
if os.WriteFile(tmp, []byte(text), 0o440) != nil {
|
||||
return
|
||||
}
|
||||
if exec.Command("visudo", "-cf", tmp).Run() != nil {
|
||||
os.Remove(tmp)
|
||||
say("warning: sudoers snippet failed validation; skipped")
|
||||
return
|
||||
}
|
||||
_ = os.Rename(tmp, sudoersPath)
|
||||
}
|
||||
|
||||
// firewall reports (and with open, fixes) an active host firewall in front of a
|
||||
// non-loopback listen address. It returns the port.
|
||||
func firewall(listen string, open bool) string {
|
||||
host, port, _ := net.SplitHostPort(listen)
|
||||
if isLoopback(host) {
|
||||
return port
|
||||
}
|
||||
switch {
|
||||
case have("firewall-cmd") && exec.Command("firewall-cmd", "--state").Run() == nil:
|
||||
if open {
|
||||
if err := runAll([][]string{{"firewall-cmd", "--quiet", "--permanent", "--add-port=" + port + "/tcp"}, {"firewall-cmd", "--quiet", "--reload"}}); err != nil {
|
||||
say("warning: firewalld: %v", err)
|
||||
} else {
|
||||
say("firewall firewalld: opened %s/tcp", port)
|
||||
}
|
||||
} else {
|
||||
say("firewall firewalld is active and will block port %s. Open it: hamlab install --open-firewall", port)
|
||||
}
|
||||
case have("ufw"):
|
||||
out, _ := exec.Command("ufw", "status").Output()
|
||||
if !bytes.Contains(out, []byte("Status: active")) {
|
||||
break
|
||||
}
|
||||
if open {
|
||||
if err := run("ufw", "allow", port+"/tcp"); err != nil {
|
||||
say("warning: ufw: %v", err)
|
||||
} else {
|
||||
say("firewall ufw: opened %s/tcp", port)
|
||||
}
|
||||
} else {
|
||||
say("firewall ufw is active and will block port %s. Open it: hamlab install --open-firewall", port)
|
||||
}
|
||||
}
|
||||
return port
|
||||
}
|
||||
|
||||
func isLoopback(host string) bool {
|
||||
ip := net.ParseIP(host)
|
||||
return host == "localhost" || (ip != nil && ip.IsLoopback())
|
||||
}
|
||||
|
||||
// browseAddr turns a listen address into one a browser can open.
|
||||
func browseAddr(listen string) string {
|
||||
host, port, err := net.SplitHostPort(listen)
|
||||
if err != nil {
|
||||
return listen
|
||||
}
|
||||
if ip := net.ParseIP(host); host == "" || (ip != nil && ip.IsUnspecified()) {
|
||||
host = "localhost"
|
||||
if addrs, err := net.InterfaceAddrs(); err == nil {
|
||||
for _, a := range addrs {
|
||||
if n, ok := a.(*net.IPNet); ok && n.IP.To4() != nil && !n.IP.IsLoopback() {
|
||||
host = n.IP.String()
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return net.JoinHostPort(host, port)
|
||||
}
|
||||
|
||||
// healthy polls /healthz for up to 10 s.
|
||||
func healthy(listen string, https bool) bool {
|
||||
host, port, _ := net.SplitHostPort(listen)
|
||||
if ip := net.ParseIP(host); host == "" || (ip != nil && ip.IsUnspecified()) {
|
||||
host = "127.0.0.1"
|
||||
}
|
||||
scheme := "http"
|
||||
if https {
|
||||
scheme = "https"
|
||||
}
|
||||
c := &http.Client{Timeout: 2 * time.Second, Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}} // self-signed; reachability only
|
||||
for i := 0; i < 20; i++ {
|
||||
if resp, err := c.Get(scheme + "://" + net.JoinHostPort(host, port) + "/healthz"); err == nil {
|
||||
resp.Body.Close()
|
||||
return resp.StatusCode < 500
|
||||
}
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// restorecon fixes the SELinux label of a file we created, where SELinux exists.
|
||||
func restorecon(path string) {
|
||||
if have("restorecon") {
|
||||
_ = exec.Command("restorecon", "-F", path).Run()
|
||||
}
|
||||
}
|
||||
|
||||
func run(args ...string) error {
|
||||
out, err := exec.Command(args[0], args[1:]...).CombinedOutput()
|
||||
if err != nil {
|
||||
if msg := strings.TrimSpace(string(out)); msg != "" {
|
||||
return fmt.Errorf("%s: %s", strings.Join(args, " "), msg)
|
||||
}
|
||||
return fmt.Errorf("%s: %w", strings.Join(args, " "), err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func runAll(cmds [][]string) error {
|
||||
for _, c := range cmds {
|
||||
if err := run(c...); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func have(bin string) bool { _, err := exec.LookPath(bin); return err == nil }
|
||||
func exists(p string) bool { _, err := os.Stat(p); return err == nil }
|
||||
func isDir(p string) bool { st, err := os.Stat(p); return err == nil && st.IsDir() }
|
||||
@@ -0,0 +1,73 @@
|
||||
package setup
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestUnitWithoutJournalGroups(t *testing.T) {
|
||||
vars := map[string]string{"@USER@": "ham", "@GROUP@": "ham", "@HOME@": "/home/ham", "@BIN@": "/usr/local/bin/hamlab",
|
||||
"@CONFIG@": "/home/ham/.config/hamlab/config.json", "@GROUPS@": ""}
|
||||
unit := dropLines(fill(systemdUnit, vars), "SupplementaryGroups=")
|
||||
if strings.Contains(unit, "SupplementaryGroups") || strings.Contains(unit, "@") {
|
||||
t.Fatalf("unit still has a groups line or an unfilled placeholder:\n%s", unit)
|
||||
}
|
||||
if !strings.Contains(unit, "ExecStart=/usr/local/bin/hamlab serve") {
|
||||
t.Fatalf("ExecStart missing:\n%s", unit)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigDirFrom(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"Environment=HAMLAB_CONFIG=/home/ham/.config/hamlab/config.json": "/home/ham/.config/hamlab",
|
||||
`export HAMLAB_CONFIG="/root/.config/hamlab/config.json"`: "/root/.config/hamlab",
|
||||
"Environment=HAMLAB_CONFIG=/config.json": "",
|
||||
"Environment=HAMLAB_CONFIG=/etc/passwd": "",
|
||||
} {
|
||||
key := "Environment=HAMLAB_CONFIG="
|
||||
if strings.HasPrefix(in, "export") {
|
||||
key = "export HAMLAB_CONFIG="
|
||||
}
|
||||
if got := configDirFrom(in, key); got != want {
|
||||
t.Errorf("%q: got %q want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --purge must never take a shared directory with it.
|
||||
func TestPurgeSparesSharedDirectory(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
os.WriteFile(filepath.Join(home, "config.json"), []byte("{}"), 0o600)
|
||||
os.WriteFile(filepath.Join(home, "thesis.txt"), []byte("irreplaceable"), 0o600)
|
||||
if err := purgeData(home); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(home, "thesis.txt")); err != nil {
|
||||
t.Fatal("purge deleted a file that wasn't Ham Lab's")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(home, "config.json")); err == nil {
|
||||
t.Fatal("purge left config.json behind")
|
||||
}
|
||||
|
||||
own := filepath.Join(t.TempDir(), "hamlab")
|
||||
os.MkdirAll(own, 0o700)
|
||||
os.WriteFile(filepath.Join(own, "config.json"), []byte("{}"), 0o600)
|
||||
if err := purgeData(own); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(own); err == nil {
|
||||
t.Fatal("purge left Ham Lab's own directory behind")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFirstMissing(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
if got := firstMissing(filepath.Join(base, "a", "b", "c")); got != filepath.Join(base, "a") {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
if got := firstMissing(base); got != "" {
|
||||
t.Errorf("existing dir: got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
package space
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"io"
|
||||
)
|
||||
|
||||
// The tree crosses from the helper process to the daemon as a preorder stream of varints:
|
||||
// a few bytes per node, decoded without reflection, so a million nodes take well under a
|
||||
// second either way.
|
||||
|
||||
const codecMagic = "HLSPACE2"
|
||||
|
||||
func Encode(w io.Writer, root *Node) error {
|
||||
bw := bufio.NewWriterSize(w, 1<<20)
|
||||
if _, err := bw.WriteString(codecMagic); err != nil {
|
||||
return err
|
||||
}
|
||||
var buf [binary.MaxVarintLen64]byte
|
||||
putU := func(v uint64) error { n := binary.PutUvarint(buf[:], v); _, err := bw.Write(buf[:n]); return err }
|
||||
putI := func(v int64) error { n := binary.PutVarint(buf[:], v); _, err := bw.Write(buf[:n]); return err }
|
||||
var enc func(n *Node) error
|
||||
enc = func(n *Node) error {
|
||||
if err := putU(uint64(len(n.Name))); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := bw.WriteString(n.Name); err != nil {
|
||||
return err
|
||||
}
|
||||
flags := uint64(0)
|
||||
if n.Dir {
|
||||
flags |= 1
|
||||
}
|
||||
if n.Mount {
|
||||
flags |= 2
|
||||
}
|
||||
if n.Denied {
|
||||
flags |= 4
|
||||
}
|
||||
if n.Partial {
|
||||
flags |= 8
|
||||
}
|
||||
for _, f := range []func() error{
|
||||
func() error { return putU(flags) },
|
||||
func() error { return putI(n.Size) },
|
||||
func() error { return putI(n.Apparent) },
|
||||
func() error { return putI(n.MTime) },
|
||||
func() error { return putI(n.MTimeNs) },
|
||||
func() error { return putU(uint64(n.Files)) },
|
||||
func() error { return putU(uint64(n.Dirs)) },
|
||||
func() error { return putU(uint64(n.SmallN)) },
|
||||
func() error { return putI(n.SmallSize) },
|
||||
func() error { return putU(uint64(len(n.Children))) },
|
||||
} {
|
||||
if err := f(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for _, c := range n.Children {
|
||||
if err := enc(c); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if err := enc(root); err != nil {
|
||||
return err
|
||||
}
|
||||
return bw.Flush()
|
||||
}
|
||||
|
||||
func Decode(r io.Reader) (*Node, error) {
|
||||
br := bufio.NewReaderSize(r, 1<<20)
|
||||
magic := make([]byte, len(codecMagic))
|
||||
if _, err := io.ReadFull(br, magic); err != nil || string(magic) != codecMagic {
|
||||
return nil, errors.New("space: not a scan stream")
|
||||
}
|
||||
var dec func() (*Node, error)
|
||||
dec = func() (*Node, error) {
|
||||
l, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if l > 1<<16 {
|
||||
return nil, errors.New("space: bad name length")
|
||||
}
|
||||
name := make([]byte, l)
|
||||
if _, err := io.ReadFull(br, name); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n := &Node{Name: string(name)}
|
||||
flags, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Dir, n.Mount, n.Denied, n.Partial = flags&1 != 0, flags&2 != 0, flags&4 != 0, flags&8 != 0
|
||||
if n.Size, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.Apparent, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.MTime, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.MTimeNs, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
u, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Files = uint32(u)
|
||||
if u, err = binary.ReadUvarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Dirs = uint32(u)
|
||||
if u, err = binary.ReadUvarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.SmallN = uint32(u)
|
||||
if n.SmallSize, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cnt, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cnt > 1<<24 {
|
||||
return nil, errors.New("space: bad child count")
|
||||
}
|
||||
if cnt > 0 {
|
||||
n.Children = make([]*Node, 0, cnt)
|
||||
for i := uint64(0); i < cnt; i++ {
|
||||
c, err := dec()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Children = append(n.Children, c)
|
||||
}
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
return dec()
|
||||
}
|
||||
@@ -0,0 +1,735 @@
|
||||
package space
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// ---- the helper: `hamlab space-walk <root>…`, run by the daemon (as root when it can) ----
|
||||
//
|
||||
// The walk runs in its own process so that root can read every directory, so that the
|
||||
// memory of the walk is given back when it ends, and so that a runaway scan can be killed
|
||||
// without touching the console. The daemon writes a header line to the helper's stdin
|
||||
// (how each root is to be walked, and what changed) followed by the last scan's tree when
|
||||
// there is one, then holds stdin open: EOF means the daemon has gone or wants the walk
|
||||
// stopped. Progress goes to stderr as JSON lines; a result line and the tree to stdout.
|
||||
|
||||
type header struct {
|
||||
Prev bool `json:"prev"`
|
||||
Modes map[string]Mode `json:"modes,omitempty"`
|
||||
Dirty []string `json:"dirty,omitempty"`
|
||||
Touched []string `json:"touched,omitempty"`
|
||||
}
|
||||
|
||||
type resultLine struct {
|
||||
Complete bool `json:"complete"`
|
||||
Partial int64 `json:"partial"`
|
||||
Reused int64 `json:"reused"`
|
||||
Read int64 `json:"read"`
|
||||
}
|
||||
|
||||
func Helper(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
|
||||
if len(args) == 0 {
|
||||
fmt.Fprintln(stderr, "usage: hamlab space-walk <mountpoint>...")
|
||||
return 2
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
sig := make(chan os.Signal, 1)
|
||||
signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP)
|
||||
go func() { <-sig; cancel() }()
|
||||
in := bufio.NewReaderSize(stdin, 1<<20)
|
||||
line, err := in.ReadBytes('\n')
|
||||
if err != nil {
|
||||
fmt.Fprintln(stderr, "error: no header:", err)
|
||||
return 2
|
||||
}
|
||||
var h header
|
||||
if err := json.Unmarshal(line, &h); err != nil {
|
||||
fmt.Fprintln(stderr, "error: bad header:", err)
|
||||
return 2
|
||||
}
|
||||
opts := Options{Modes: h.Modes, Dirty: set(h.Dirty), Touched: set(h.Touched)}
|
||||
if h.Prev {
|
||||
if opts.Prev, err = Decode(in); err != nil {
|
||||
fmt.Fprintln(stderr, "error: last tree:", err)
|
||||
return 2
|
||||
}
|
||||
}
|
||||
go func() { _, _ = io.Copy(io.Discard, in); cancel() }() // the daemon went away, or asked us to stop
|
||||
enc := json.NewEncoder(stderr)
|
||||
res, err := Walk(ctx, args, opts, func(p Progress) { _ = enc.Encode(p) })
|
||||
if err != nil && !errors.Is(err, context.Canceled) {
|
||||
fmt.Fprintln(stderr, "error:", err)
|
||||
return 1
|
||||
}
|
||||
rl, _ := json.Marshal(resultLine{Complete: res.Complete, Partial: res.Partial, Reused: res.Reused, Read: res.Read})
|
||||
if _, err := stdout.Write(append(rl, '\n')); err != nil {
|
||||
return 1
|
||||
}
|
||||
if err := Encode(stdout, res.Root); err != nil {
|
||||
fmt.Fprintln(stderr, "error:", err)
|
||||
return 1
|
||||
}
|
||||
if !res.Complete {
|
||||
return 3
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func set(list []string) map[string]bool {
|
||||
if len(list) == 0 {
|
||||
return nil
|
||||
}
|
||||
m := make(map[string]bool, len(list))
|
||||
for _, s := range list {
|
||||
m[s] = true
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ---- the daemon side: one scan per target, the last result kept, on disk too ----
|
||||
|
||||
type Scan struct {
|
||||
Disk string `json:"disk"`
|
||||
Roots []string `json:"roots"`
|
||||
Started time.Time `json:"started"`
|
||||
Finished time.Time `json:"finished,omitempty"`
|
||||
Running bool `json:"running"`
|
||||
Loading bool `json:"loading,omitempty"` // read back from disk after a restart, not yet decoded
|
||||
Error string `json:"error,omitempty"`
|
||||
Progress Progress `json:"progress"`
|
||||
AsRoot bool `json:"as_root"`
|
||||
// How this scan was done: "full", or "refresh" / "resume" from the one before, and by what
|
||||
// means per root ("zfs diff" or "mtime").
|
||||
Method string `json:"method,omitempty"`
|
||||
ByRoot map[string]string `json:"by_root,omitempty"`
|
||||
Partial bool `json:"partial,omitempty"` // cancelled before it finished; Scan again resumes it
|
||||
// once finished:
|
||||
Files uint32 `json:"files"`
|
||||
Dirs uint32 `json:"dirs"`
|
||||
Size int64 `json:"size"`
|
||||
Denied int64 `json:"denied"`
|
||||
Read int64 `json:"read"` // directories read this time
|
||||
Reused int64 `json:"reused"` // directories kept from the scan before
|
||||
PartialDirs int64 `json:"partial_dirs"` // directories not read (cancelled)
|
||||
// Bases is, per root that is a ZFS dataset, the newest snapshot that existed when the
|
||||
// scan started: the next scan asks `zfs diff` what changed since it.
|
||||
Bases map[string]string `json:"bases,omitempty"`
|
||||
|
||||
root *Node
|
||||
largest []File
|
||||
cancel context.CancelFunc
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
self string // this binary, for the helper
|
||||
dir string // where results are kept between restarts
|
||||
mu sync.Mutex
|
||||
scans map[string]*Scan
|
||||
}
|
||||
|
||||
// New makes a manager that keeps results under dir and reads back the ones there.
|
||||
func New(self, dir string) *Manager {
|
||||
m := &Manager{self: self, dir: dir, scans: map[string]*Scan{}}
|
||||
m.loadAll()
|
||||
return m
|
||||
}
|
||||
|
||||
func (m *Manager) fileFor(id, ext string) string {
|
||||
safe := strings.NewReplacer(":", "_", "/", "_", "\\", "_").Replace(id)
|
||||
return filepath.Join(m.dir, safe+ext)
|
||||
}
|
||||
|
||||
// loadAll reads the saved scans' metadata now and their trees in the background, one at a
|
||||
// time, so a restart does not double the memory of the last scans all at once.
|
||||
func (m *Manager) loadAll() {
|
||||
metas, _ := filepath.Glob(filepath.Join(m.dir, "*.json"))
|
||||
var pending []*Scan
|
||||
for _, mp := range metas {
|
||||
b, err := os.ReadFile(mp)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
s := &Scan{}
|
||||
if json.Unmarshal(b, s) != nil || s.Disk == "" {
|
||||
continue
|
||||
}
|
||||
s.Running, s.cancel = false, nil
|
||||
s.Loading = true
|
||||
m.scans[s.Disk] = s
|
||||
pending = append(pending, s)
|
||||
}
|
||||
if len(pending) == 0 {
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
for _, s := range pending {
|
||||
root, err := m.readTree(s.Disk)
|
||||
m.mu.Lock()
|
||||
if err != nil {
|
||||
log.Printf("space: saved scan of %s unreadable (%v); dropped", s.Disk, err)
|
||||
if m.scans[s.Disk] == s {
|
||||
delete(m.scans, s.Disk)
|
||||
}
|
||||
} else if m.scans[s.Disk] == s {
|
||||
s.root, s.Loading = root, false
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func (m *Manager) readTree(id string) (*Node, error) {
|
||||
f, err := os.Open(m.fileFor(id, ".scan"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
br := bufio.NewReaderSize(f, 1<<20)
|
||||
if _, err := br.ReadBytes('\n'); err != nil { // the result line
|
||||
return nil, err
|
||||
}
|
||||
return Decode(br)
|
||||
}
|
||||
|
||||
func (m *Manager) saveMeta(s *Scan) {
|
||||
if m.dir == "" {
|
||||
return
|
||||
}
|
||||
_ = os.MkdirAll(m.dir, 0o700)
|
||||
b, err := json.MarshalIndent(s, "", " ")
|
||||
if err == nil {
|
||||
err = writeAtomic(m.fileFor(s.Disk, ".json"), b)
|
||||
}
|
||||
if err != nil {
|
||||
log.Printf("space: save %s: %v", s.Disk, err)
|
||||
}
|
||||
}
|
||||
|
||||
func writeAtomic(path string, b []byte) error {
|
||||
tmp := path + ".tmp"
|
||||
if err := os.WriteFile(tmp, b, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// Status is the scan of id (a copy without the tree), or nil.
|
||||
func (m *Manager) Status(id string) *Scan {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.copyOf(m.scans[id])
|
||||
}
|
||||
|
||||
func (m *Manager) copyOf(s *Scan) *Scan {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
c := *s
|
||||
c.root, c.largest, c.cancel = nil, nil, nil
|
||||
return &c
|
||||
}
|
||||
|
||||
func (m *Manager) All() map[string]*Scan {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
out := map[string]*Scan{}
|
||||
for k, s := range m.scans {
|
||||
out[k] = m.copyOf(s)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Start scans the roots of id in the background, from the last scan of it when there is
|
||||
// one. datasets names the ZFS dataset behind each root that has one.
|
||||
func (m *Manager) Start(id string, roots []string, datasets map[string]string) (*Scan, error) {
|
||||
if len(roots) == 0 {
|
||||
return nil, errors.New("nothing is mounted from this disk")
|
||||
}
|
||||
m.mu.Lock()
|
||||
prev := m.scans[id]
|
||||
if prev != nil && prev.Running {
|
||||
m.mu.Unlock()
|
||||
return nil, errors.New("a scan of this disk is already running")
|
||||
}
|
||||
if prev != nil && (prev.root == nil || prev.Loading) {
|
||||
prev = nil // nothing to start from
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s := &Scan{Disk: id, Roots: roots, Started: time.Now(), Running: true, AsRoot: priv.Available(), cancel: cancel, Bases: map[string]string{}}
|
||||
if prev != nil {
|
||||
// the new scan takes the old one's place in the list while it runs, but keeps its
|
||||
// tree, so the pages still answer until the new one is in
|
||||
s.root, s.largest = prev.root, prev.largest
|
||||
s.Files, s.Dirs, s.Size = prev.Files, prev.Dirs, prev.Size
|
||||
}
|
||||
m.scans[id] = s
|
||||
m.mu.Unlock()
|
||||
go m.run(ctx, s, prev, datasets)
|
||||
return m.Status(id), nil
|
||||
}
|
||||
|
||||
// plan decides how each root is walked this time, given the last scan.
|
||||
func (m *Manager) plan(ctx context.Context, s, prev *Scan, datasets map[string]string) (header, Options) {
|
||||
h := header{Modes: map[string]Mode{}}
|
||||
opts := Options{Modes: h.Modes}
|
||||
if prev == nil || prev.root == nil {
|
||||
s.Method = "full"
|
||||
return h, opts
|
||||
}
|
||||
h.Prev = true
|
||||
opts.Prev = prev.root
|
||||
s.ByRoot = map[string]string{}
|
||||
dirty, touched := map[string]bool{}, map[string]bool{}
|
||||
// every dataset is asked at once what changed; the answers come back in root order
|
||||
type diffed struct {
|
||||
changed []change
|
||||
err error
|
||||
took time.Duration
|
||||
}
|
||||
answers := map[string]chan diffed{}
|
||||
for _, r := range s.Roots {
|
||||
if base := prev.Bases[r]; base != "" && datasets[r] != "" {
|
||||
ch := make(chan diffed, 1)
|
||||
answers[r] = ch
|
||||
go func(r, base string) {
|
||||
t0 := time.Now()
|
||||
c, err := zfsDiff(ctx, base, r)
|
||||
ch <- diffed{changed: c, err: err, took: time.Since(t0)}
|
||||
}(r, base)
|
||||
}
|
||||
}
|
||||
for _, r := range s.Roots {
|
||||
mode := ModeMTime
|
||||
how := "mtime"
|
||||
if ch, ok := answers[r]; ok {
|
||||
m.mu.Lock()
|
||||
s.Progress.Current = "asking zfs what changed in " + r
|
||||
m.mu.Unlock()
|
||||
d := <-ch
|
||||
if d.err == nil {
|
||||
mode, how = ModeDiff, fmt.Sprintf("zfs diff since %s: %d changes in %s", strings.TrimPrefix(prev.Bases[r], datasets[r]), len(d.changed), d.took.Round(time.Millisecond))
|
||||
for _, c := range d.changed {
|
||||
dirty[c.parent] = true
|
||||
if c.dir {
|
||||
dirty[c.path] = true
|
||||
}
|
||||
}
|
||||
} else {
|
||||
how = "mtime (zfs diff: " + d.err.Error() + ")"
|
||||
}
|
||||
}
|
||||
if Find(prev.root, r) == nil {
|
||||
mode, how = ModeFull, "full (new root)"
|
||||
}
|
||||
h.Modes[r] = mode
|
||||
s.ByRoot[r] = how
|
||||
}
|
||||
m.mu.Lock()
|
||||
s.Progress.Current = ""
|
||||
m.mu.Unlock()
|
||||
// what the last walk never got to is read whatever the change list says
|
||||
for _, p := range partialPaths(prev.root, "/") {
|
||||
dirty[p] = true
|
||||
}
|
||||
for p := range dirty {
|
||||
for a := filepath.Dir(p); ; a = filepath.Dir(a) {
|
||||
touched[a] = true
|
||||
if a == "/" {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
for p := range dirty {
|
||||
h.Dirty = append(h.Dirty, p)
|
||||
}
|
||||
for p := range touched {
|
||||
h.Touched = append(h.Touched, p)
|
||||
}
|
||||
sort.Strings(h.Dirty)
|
||||
sort.Strings(h.Touched)
|
||||
opts.Dirty, opts.Touched = dirty, touched
|
||||
if prev.Partial {
|
||||
s.Method = "resume"
|
||||
} else {
|
||||
s.Method = "refresh"
|
||||
}
|
||||
return h, opts
|
||||
}
|
||||
|
||||
func partialPaths(n *Node, path string) []string {
|
||||
var out []string
|
||||
var walk func(n *Node, path string)
|
||||
walk = func(n *Node, path string) {
|
||||
if n.Partial {
|
||||
out = append(out, path)
|
||||
}
|
||||
for _, c := range n.Children {
|
||||
if c.Dir {
|
||||
walk(c, join(path, c.Name))
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(n, path)
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) run(ctx context.Context, s *Scan, prev *Scan, datasets map[string]string) {
|
||||
finish := func(err error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
s.Running = false
|
||||
s.Finished = time.Now()
|
||||
if err != nil {
|
||||
s.Error = err.Error()
|
||||
}
|
||||
m.saveMeta(s)
|
||||
}
|
||||
// the newest snapshot of each dataset, so the next scan can ask what changed since
|
||||
for _, r := range s.Roots {
|
||||
if ds := datasets[r]; ds != "" {
|
||||
if snap := newestSnapshot(ctx, ds); snap != "" {
|
||||
s.Bases[r] = snap
|
||||
}
|
||||
}
|
||||
}
|
||||
h, _ := m.plan(ctx, s, prev, datasets)
|
||||
if ctx.Err() != nil { // cancelled while zfs was being asked
|
||||
finish(errors.New("cancelled"))
|
||||
return
|
||||
}
|
||||
args := append([]string{m.self, "space-walk"}, s.Roots...)
|
||||
if s.AsRoot {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
cmd := exec.Command(args[0], args[1:]...)
|
||||
stdin, _ := cmd.StdinPipe() // held open: closing it tells the helper to stop
|
||||
stdout, _ := cmd.StdoutPipe()
|
||||
stderr, _ := cmd.StderrPipe()
|
||||
if err := cmd.Start(); err != nil {
|
||||
finish(err)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
hb, _ := json.Marshal(h)
|
||||
_, err := stdin.Write(append(hb, '\n'))
|
||||
if err == nil && h.Prev {
|
||||
err = Encode(stdin, prev.root)
|
||||
}
|
||||
if err != nil {
|
||||
log.Printf("space: %s: sending the last tree: %v", s.Disk, err)
|
||||
}
|
||||
<-ctx.Done()
|
||||
_ = cmd.Process.Signal(syscall.SIGTERM)
|
||||
stdin.Close()
|
||||
}()
|
||||
var lastErr string
|
||||
go func() {
|
||||
sc := bufio.NewScanner(stderr)
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
var p Progress
|
||||
if strings.HasPrefix(line, "{") && json.Unmarshal([]byte(line), &p) == nil {
|
||||
m.mu.Lock()
|
||||
s.Progress = p
|
||||
m.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
if line != "" {
|
||||
lastErr = line
|
||||
}
|
||||
}
|
||||
}()
|
||||
// the result streams to disk as it is decoded, so a restart finds it
|
||||
_ = os.MkdirAll(m.dir, 0o700)
|
||||
tmp := m.fileFor(s.Disk, ".scan.tmp")
|
||||
out, ferr := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
|
||||
var src io.Reader = stdout
|
||||
if ferr == nil {
|
||||
src = io.TeeReader(stdout, out)
|
||||
}
|
||||
br := bufio.NewReaderSize(src, 1<<20)
|
||||
var rl resultLine
|
||||
line, rerr := br.ReadBytes('\n')
|
||||
if rerr == nil {
|
||||
rerr = json.Unmarshal(line, &rl)
|
||||
}
|
||||
var root *Node
|
||||
var decErr error
|
||||
if rerr == nil {
|
||||
root, decErr = Decode(br)
|
||||
} else {
|
||||
decErr = rerr
|
||||
_, _ = io.Copy(io.Discard, br)
|
||||
}
|
||||
if ferr == nil {
|
||||
out.Close()
|
||||
}
|
||||
waitErr := cmd.Wait()
|
||||
cancelled := ctx.Err() != nil
|
||||
if root == nil {
|
||||
_ = os.Remove(tmp)
|
||||
switch {
|
||||
case cancelled:
|
||||
finish(errors.New("cancelled"))
|
||||
case waitErr != nil && lastErr != "":
|
||||
finish(errors.New(strings.TrimPrefix(lastErr, "error: ")))
|
||||
case waitErr != nil:
|
||||
finish(waitErr)
|
||||
default:
|
||||
finish(decErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if ferr == nil {
|
||||
_ = os.Rename(tmp, m.fileFor(s.Disk, ".scan"))
|
||||
}
|
||||
m.mu.Lock()
|
||||
s.root, s.largest = root, nil
|
||||
s.Files, s.Dirs, s.Size, s.Denied = root.Files, root.Dirs, root.Size, s.Progress.Denied
|
||||
s.Read, s.Reused, s.PartialDirs = rl.Read, rl.Reused, rl.Partial
|
||||
s.Partial = !rl.Complete
|
||||
s.Running = false
|
||||
s.Finished = time.Now()
|
||||
if s.Partial {
|
||||
s.Error = "cancelled"
|
||||
}
|
||||
m.saveMeta(s)
|
||||
m.mu.Unlock()
|
||||
what := "scanned"
|
||||
if s.Partial {
|
||||
what = "partly scanned (cancelled)"
|
||||
}
|
||||
log.Printf("space: %s %s: %d files, %d dirs, %d bytes in %s (%s; %d dirs read, %d kept%s)", s.Disk, what, root.Files, root.Dirs, root.Size, s.Finished.Sub(s.Started).Round(time.Millisecond), s.Method, rl.Read, rl.Reused, map[bool]string{true: ", as root", false: ""}[s.AsRoot])
|
||||
}
|
||||
|
||||
func (m *Manager) Cancel(id string) bool {
|
||||
m.mu.Lock()
|
||||
s := m.scans[id]
|
||||
m.mu.Unlock()
|
||||
if s == nil || !s.Running || s.cancel == nil {
|
||||
return false
|
||||
}
|
||||
s.cancel()
|
||||
return true
|
||||
}
|
||||
|
||||
// Forget drops the scan of id, in memory and on disk.
|
||||
func (m *Manager) Forget(id string) {
|
||||
m.mu.Lock()
|
||||
if s := m.scans[id]; s != nil && s.cancel != nil {
|
||||
s.cancel()
|
||||
}
|
||||
delete(m.scans, id)
|
||||
m.mu.Unlock()
|
||||
_ = os.Remove(m.fileFor(id, ".scan"))
|
||||
_ = os.Remove(m.fileFor(id, ".json"))
|
||||
}
|
||||
|
||||
// ---- ZFS: what changed since the last scan ----
|
||||
|
||||
type change struct {
|
||||
path, parent string
|
||||
dir bool
|
||||
}
|
||||
|
||||
// newestSnapshot is the most recent snapshot of a dataset, "" when it has none.
|
||||
func newestSnapshot(ctx context.Context, dataset string) string {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-t", "snapshot", "-o", "name", "-s", "creation", "-d", "1", dataset).Output()
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
last := lines[len(lines)-1]
|
||||
if !strings.Contains(last, "@") {
|
||||
return ""
|
||||
}
|
||||
return last
|
||||
}
|
||||
|
||||
// zfsDiff lists what changed in a dataset since a snapshot: every changed, added, removed
|
||||
// or renamed path, with the directory whose listing that touches.
|
||||
func zfsDiff(ctx context.Context, snapshot, mount string) ([]change, error) {
|
||||
// A diff slower than this is not worth waiting for: the walk by mtime is bounded. The
|
||||
// clock runs inside sudo — `timeout` is zfs's parent and kills it as root — because sudo
|
||||
// on Debian gives the command a session of its own, out of reach of our process group;
|
||||
// the context is only a backstop.
|
||||
const patience = 3 * time.Minute
|
||||
ctx, cancel := context.WithTimeout(ctx, patience+30*time.Second)
|
||||
defer cancel()
|
||||
args := priv.Wrap("timeout", "-s", "KILL", strconv.Itoa(int(patience.Seconds())), "zfs", "diff", "-H", "-F", snapshot)
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
|
||||
cmd.WaitDelay = 5 * time.Second
|
||||
t0 := time.Now()
|
||||
var stderr strings.Builder
|
||||
cmd.Stderr = &stderr
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var changes []change
|
||||
sc := bufio.NewScanner(out)
|
||||
sc.Buffer(make([]byte, 64<<10), 4<<20)
|
||||
for sc.Scan() {
|
||||
f := strings.Split(sc.Text(), "\t")
|
||||
if len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
isDir := f[1] == "/"
|
||||
for _, p := range f[2:] { // a rename has the old and the new path
|
||||
p = unescape(p)
|
||||
if !strings.HasPrefix(p, mount) {
|
||||
continue
|
||||
}
|
||||
changes = append(changes, change{path: p, parent: filepath.Dir(p), dir: isDir})
|
||||
}
|
||||
}
|
||||
if err := cmd.Wait(); err != nil {
|
||||
msg := strings.TrimSpace(stderr.String())
|
||||
if errors.Is(ctx.Err(), context.DeadlineExceeded) || time.Since(t0) >= patience {
|
||||
msg = "gave up after " + patience.String()
|
||||
}
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
if i := strings.IndexByte(msg, '\n'); i > 0 {
|
||||
msg = msg[:i]
|
||||
}
|
||||
return nil, errors.New(msg)
|
||||
}
|
||||
return changes, nil
|
||||
}
|
||||
|
||||
// unescape undoes zfs diff's \ooo escapes for spaces and odd bytes in names.
|
||||
func unescape(s string) string {
|
||||
if !strings.Contains(s, "\\") {
|
||||
return s
|
||||
}
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(s); i++ {
|
||||
if s[i] == '\\' && i+3 < len(s) {
|
||||
if v, err := strconv.ParseUint(s[i+1:i+4], 8, 8); err == nil {
|
||||
b.WriteByte(byte(v))
|
||||
i += 3
|
||||
continue
|
||||
}
|
||||
}
|
||||
b.WriteByte(s[i])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// Tree is the subtree at path, cut to depth levels of at most limit children each, leaving
|
||||
// out children smaller than min bytes; what is left out is summed into MoreN/MoreSize. A
|
||||
// treemap asks deep with a min the size of the smallest tile it could draw; a folder list
|
||||
// asks one level with no min.
|
||||
type Tree struct {
|
||||
Node
|
||||
Path string `json:"path"`
|
||||
Children []*Tree `json:"children,omitempty"`
|
||||
MoreN int `json:"more_n,omitempty"`
|
||||
MoreSize int64 `json:"more_size,omitempty"`
|
||||
Total int64 `json:"total"` // the whole scan's size, for percentages
|
||||
}
|
||||
|
||||
func (m *Manager) Tree(id, path string, depth, limit int, min int64) (*Tree, error) {
|
||||
m.mu.Lock()
|
||||
s := m.scans[id]
|
||||
var root *Node
|
||||
if s != nil {
|
||||
root = s.root
|
||||
}
|
||||
m.mu.Unlock()
|
||||
if root == nil {
|
||||
if s != nil && s.Loading {
|
||||
return nil, errors.New("the last scan is still being read back")
|
||||
}
|
||||
return nil, errors.New("no scan of this disk yet")
|
||||
}
|
||||
n := Find(root, path)
|
||||
if n == nil {
|
||||
return nil, errors.New("no such path in the scan")
|
||||
}
|
||||
if path == "" {
|
||||
path = "/"
|
||||
}
|
||||
t := cut(n, path, depth, limit, min)
|
||||
t.Total = root.Size
|
||||
return t, nil
|
||||
}
|
||||
|
||||
func cut(n *Node, path string, depth, limit int, min int64) *Tree {
|
||||
t := &Tree{Node: *n, Path: path}
|
||||
t.Node.Children = nil
|
||||
if !n.Dir {
|
||||
return t
|
||||
}
|
||||
if depth <= 0 { // as deep as was asked: what is inside is one lump, not nothing
|
||||
if len(n.Children) > 0 {
|
||||
t.MoreN, t.MoreSize = len(n.Children), n.Size-n.SmallSize
|
||||
}
|
||||
return t
|
||||
}
|
||||
for i, c := range n.Children {
|
||||
if i >= limit || (c.Size < min && i > 0) { // children are sorted largest first: the first is always kept
|
||||
t.MoreN++
|
||||
t.MoreSize += c.Size
|
||||
continue
|
||||
}
|
||||
t.Children = append(t.Children, cut(c, join(path, c.Name), depth-1, limit, min))
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// Largest is the n biggest files of the last scan of id.
|
||||
func (m *Manager) Largest(id string, n int) ([]File, error) {
|
||||
m.mu.Lock()
|
||||
s := m.scans[id]
|
||||
var root *Node
|
||||
var have []File
|
||||
if s != nil {
|
||||
root, have = s.root, s.largest
|
||||
}
|
||||
m.mu.Unlock()
|
||||
if root == nil {
|
||||
return nil, errors.New("no scan of this disk yet")
|
||||
}
|
||||
if len(have) < n {
|
||||
have = Largest(root, n) // the tree is never changed once made, so no lock while walking it
|
||||
m.mu.Lock()
|
||||
if s.root == root {
|
||||
s.largest = have
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
if len(have) > n {
|
||||
return have[:n], nil
|
||||
}
|
||||
return have, nil
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user