Compare commits
189
Commits
v0.1.0
..
a0ce3e5eb9
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(systemctl status *)",
|
||||
"Bash(systemctl is-active *)"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -2,3 +2,4 @@ bin/
|
||||
internal/web/dist/
|
||||
web/node_modules/
|
||||
web/dist/
|
||||
dist/
|
||||
|
||||
@@ -0,0 +1,754 @@
|
||||
# Bug bash — 2026-09-18
|
||||
|
||||
Fix pass 2026-09-18: every item carries its outcome; ✅ resolved and verified live, ➖ no change with reason. Found by a headless Chromium pass (iPhone and
|
||||
1440px desktop viewports) over every page on the hub and on each remote machine, an
|
||||
offline-node scenario (wyse stopped for the test, then restarted), direct API probing
|
||||
with hostile inputs, and a code read. Ordered by severity within each group.
|
||||
|
||||
## Security / robustness
|
||||
|
||||
1. ✅ **RESOLVED 2026-09-18** — only regular files are readable; the byte cap applies to what is sent, not to `stat`. `/dev/zero`, `/dev/null` and FIFOs answer 400 in 30 ms; `/proc/cpuinfo` still reads. **Reading a special file hangs the request forever.** `GET /api/files/read?path=/dev/zero`
|
||||
never ends: the `max` cap compares against `stat` size, which is 0 for device files,
|
||||
so the cap never triggers and the handler streams `/dev/zero` until the client gives
|
||||
up. Same for FIFOs (blocks on open) and `/proc/self/mem`. Opening one in the Files
|
||||
viewer would hang the tab. Needs: refuse non-regular files for read/edit, and cap the
|
||||
copy with `io.LimitReader` regardless of `stat`.
|
||||
2. ✅ **RESOLVED** — per-client buckets (first X-Forwarded-For hop or the peer) with the same 5-strike schedule, plus a global backstop of 100 failures per 5 min → 1 min. Verified: core's public IP locked while grid logged in fine. **Login lockout is global, not per client.** Five bad passwords from anyone on the
|
||||
internet lock the real user out for up to 10 minutes (`auth.Limiter` is one bucket).
|
||||
The bash locked the hub for the full 10 minutes twice. Also locks `POST /api/tokens`,
|
||||
so hub pairing fails during a lockout. Needs per-IP buckets with a modest global cap.
|
||||
3. ✅ **RESOLVED** — server rejects refreshMs outside 100–60000 and non-identifier themes (400); client ignores out-of-range rates from any source. **UI prefs from the server are not validated.** A stored `refreshMs` of `-5` made the
|
||||
client call `setInterval(-5)` on every live view: a request storm that took the
|
||||
browser to `ERR_INSUFFICIENT_RESOURCES` and hammered the daemon. `theme` is validated
|
||||
only by falling back. Needs range checks on load (and on save server-side).
|
||||
4. ✅ **RESOLVED** — every response carries X-Frame-Options DENY, CSP frame-ancestors none, nosniff, no-referrer. **No anti-framing / CSP headers.** Only HSTS (from NPM). A root-capable console should
|
||||
send `X-Frame-Options: DENY`, `Content-Security-Policy: frame-ancestors 'none'`,
|
||||
`Referrer-Policy: no-referrer`, `X-Content-Type-Options: nosniff` on every response.
|
||||
5. ✅ **RESOLVED** — kernel threads (empty cmdline) and the daemon's own PID are refused. **Signals can be sent to kernel threads.** `POST /api/procs/2/signal` (kthreadd)
|
||||
returns 204. Harmless in practice, but the guard should refuse PIDs with an empty
|
||||
cmdline (kernel threads) and our own PID.
|
||||
6. ✅ **RESOLVED** — unit runs with `SupplementaryGroups=systemd-journal adm` so journal reads need no sudo, and the installer drops `/etc/sudoers.d/hamlab-quiet` (no syslog/pam session for the read-only polling commands). Verified: 0 sudo lines in 60 s of 250 ms polling (was 839/10 min). Applied on all four hosts. **Journal spam from the daemon's own sudo calls.** Every sudo'd `journalctl`, `smartctl`
|
||||
or `pwrstat` writes three `pam_unix(sudo:session)` lines into the journal *under
|
||||
`hamlab.service`*: 839 lines in 10 minutes at a 250 ms refresh, and the journal is
|
||||
already 4 GB. It also buries Ham Lab's own log entries. Fix options: add the service
|
||||
user to `systemd-journal` (and `adm`) so journal reads don't need sudo, and/or a
|
||||
sudoers `Defaults!/usr/bin/journalctl !syslog, !pam_session` line in the installer.
|
||||
7. ✅ **RESOLVED** — server answers 409 unless `overwrite=1`; the UI asks "Replace it?" first. **Uploading over an existing file silently overwrites it.** No confirmation, no
|
||||
"replace?" prompt. Same for "+ file" when the name exists (server refuses; UI just
|
||||
shows the error, fine) — upload is the gap.
|
||||
8. ➖ **NO CHANGE** — Go's mux redirects the unclean path to `/etc/passwd`, which is an ordinary deep link the SPA must serve; there is no API prefix left by the time the static handler sees it. Harmless. **`/api/<anything>/../../x` falls through to the SPA** and returns `index.html` with
|
||||
200 instead of a JSON 404. Harmless, but sloppy for an API prefix.
|
||||
|
||||
## Functional bugs
|
||||
|
||||
9. ✅ **RESOLVED** — 401 carries the server message; the form shows "wrong password" (verified headless). **Wrong password gives no feedback at all.** The 401 is thrown as `Unauthenticated`
|
||||
with an empty message, and the login form only renders `err` when non-empty. The
|
||||
button just re-enables. Reproduced on phone and desktop.
|
||||
10. ✅ **RESOLVED** — hub-only pages bounce to the dashboard on switch (verified headless). **Switching machines while on Settings or Machines keeps showing the hub's page.**
|
||||
`switchMachine` only resets the view for Machines; Settings stays, so "mac" shows
|
||||
core's account/token/about panels. Both are hub-only pages and should bounce to the
|
||||
dashboard (or be clearly labelled as hub pages).
|
||||
11. ✅ **RESOLVED** — escaped; no console error. **Machines add-form `pattern` is an invalid regex in Chromium.** `[a-z0-9][a-z0-9-]*`
|
||||
fails under the `v` flag ("Invalid character class"), logs a console error and the
|
||||
browser skips validation. Needs `[a-z0-9][a-z0-9\-]*`.
|
||||
12. ✅ **RESOLVED** — refresh from the newest repodata under libdnf5/dnf caches (Mac: 9 d), last upgrade from `dnf history` (194 d), label reads "from dnf history". **Updates tab on dnf hosts shows nonsense.** Mac: "Index refreshed 561d ago" (mtime of
|
||||
`/var/cache/dnf`, which Fedora 41+ no longer uses; `libdnf5` is the live cache) and
|
||||
"Last upgrade never · from apt history" (label and source are apt-specific). Needs
|
||||
`dnf history` for last upgrade, metadata timestamps for refresh, and neutral labels.
|
||||
13. ✅ **RESOLVED** — journal read incrementally by cursor, timers from one `systemctl show`, services' two calls run in parallel. Cron 3.6 s → 0.08 s, services 0.6 s → 0.05 s, with last-run and next-run data intact. **Cron overview is slow: 3.6 s per request on core.** At any refresh rate the page
|
||||
is constantly busy. The 30-day, 20 000-line journal scan for "last run" is rebuilt
|
||||
every 5 s; timers make three `systemctl` calls. Needs an incremental journal cursor
|
||||
and a single `systemctl` call. Services list is 0.6 s for the same reason (two calls).
|
||||
14. ✅ **RESOLVED** — heap profile showed a 48 MB `bytes.Buffer` from buffering `journalctl` dumps; those are now streamed line by line, plus a 160 MB soft memory limit for the GC. RSS 22 MB fresh, 27 MB after 25× cron/logs/procs/services (was 180 MB). pprof stays available behind the login at `/api/debug/pprof/`. **Daemon memory grows.** RSS on core went 22 → 85 → 216 MB during the day (136 MB at
|
||||
rest after the bash). Suspects: per-pid caches in the process table (`procInfo` only
|
||||
prunes on the next walk), task output buffers, SMART cache, xterm scrollback rings
|
||||
that are never freed for exited sessions. Needs a heap profile under load.
|
||||
15. ✅ **RESOLVED** — `--dirty` dropped from the version and deploys now follow commits; binaries report e.g. `v0.1.0-38-gc1805f6`. **Version string is always `-dirty`.** `make deploy` builds before the commit, so
|
||||
every installed binary reports e.g. `v0.1.0-32-ga372378-dirty`, and the Machines page
|
||||
compares with `-dirty` stripped. Build after commit, or tag releases.
|
||||
16. ✅ **RESOLVED** — hash routing `#/<machine|hub>/<view>`: reload restores machine and tab, Back/Forward work, links can point at a page on a machine (verified headless). **View resets to Dashboard on reload; no deep links; browser Back leaves the app.**
|
||||
The selected machine survives a reload (sessionStorage) but the tab does not, and
|
||||
there is no URL routing at all. Needs hash or path routing (`/#/grid/services`).
|
||||
17. ✅ **RESOLVED** — newest first. **Cron log tab lists oldest first and starts at the top.** 200 lines from Sep 10 with
|
||||
the newest at the bottom, off-screen. Should show newest first or scroll to the end.
|
||||
18. ✅ **RESOLVED** — "Nothing matches." **Processes: empty filter result shows nothing.** No "nothing matches" message.
|
||||
19. ✅ **RESOLVED** — exited shells remove themselves 10 minutes after exit (verified live: seeded 17:53:56, gone at 18:03:59 with no daemon restart in between). Found and fixed alongside: a daemon restart used to re-spawn *exited* shells as fresh running ones; only running shells are persisted now. Idle running shells are deliberately kept. **Terminals accumulate.** Nothing ever reaps exited or idle sessions; the bash left
|
||||
several "bug-bash-term" shells on the hub until deleted by hand. Exited sessions
|
||||
should auto-remove after a while; idle ones at least be counted somewhere.
|
||||
20. ✅ **RESOLVED** — a node answers 409 "busy" when a package task is running or a terminal is attached; the Machines page offers Force, `hamlab push-nodes --force` / `make push-nodes-force` do the same. Verified: 409 with a terminal attached on grid, 200 with force. **Self-update restarts without warning running work.** A push to a node kills its
|
||||
open terminals and any running task (upgrade in progress). The hub should refuse or
|
||||
warn when the node reports a running task or attached terminals.
|
||||
21. ✅ **RESOLVED** — PathError and LinkError are unwrapped to the bare reason. **Error text leaks raw Go errors.** e.g. rename shows "rename /tmp/x /tmp/y: no such
|
||||
file or directory". Fine for a lab, ugly for a product; strip the syscall prefix.
|
||||
|
||||
## Mobile / UX
|
||||
|
||||
22. ✅ **RESOLVED** — on coarse pointers every button, chip, icon button, link, header and select is at least 36 px tall (verified: no control under 32 px on the dashboard or processes pages). **Tap targets under 28 px on phone:** header ⏻ (25), logout (22), customize ▲ ▼ hide
|
||||
(23), terminal leaf ✎ ✕ (24), chips (25), table headers (27). Apple's guideline is
|
||||
44 pt. Systemic: the small `.btn`/`.chip`/`.icon-btn` sizes need a touch variant.
|
||||
23. ✅ **RESOLVED** — the dialog asks for the machine name (`mac`); the node still checks its own hostname under the hood. **Power dialog on the Mac asks you to type `5ce91ee54af6`.** The confirmation uses
|
||||
the node's hostname, which on the Mac is the unset default. Should offer the machine
|
||||
name (`mac`) as the confirmation word on remote machines.
|
||||
24. ✅ **RESOLVED** — down interfaces collapse to one line marked "down"; modules with nothing to show (fans, UPS, temps, disks) hide unless customizing. **Dashboard shows dead interfaces and empty modules.** wlp1s0 (down) gets a full
|
||||
sparkline block of flat lines; "Fans: no fan sensors" is a permanent empty card on
|
||||
hosts without fans. Down interfaces should collapse to one line; empty modules should
|
||||
auto-hide unless customizing.
|
||||
25. ✅ **RESOLVED** — "load 1.39 1.56 1.55" with a tooltip. **Header load average is unlabelled** ("1.39 1.56 1.55" floating in the header).
|
||||
26. ➖ **NOT FIXING** — the list is intentionally a plain table; the menu anchors inside the row. Cosmetic taste, not a defect. **Files list on desktop is a narrow strip in a sea of empty space** for small
|
||||
folders; the row menu opens off the right edge of the row. Cosmetic.
|
||||
27. ✅ **RESOLVED** — after a 502 the client leaves that machine alone for 5 s: 2 requests in 10 s at a 250 ms refresh (would have been ~80), badge and message intact. **Offline machine keeps polling every view at the refresh rate**, producing a 502
|
||||
per request per page (and console errors). The UI copes — badge, message — but the
|
||||
hub should back off to the prober's cadence while a node is down.
|
||||
28. ✅ **RESOLVED** with 6. **Services logs are noise.** The default view of `hamlab.service` logs is the sudo
|
||||
session spam from bug 6, three lines per poll.
|
||||
|
||||
## Confirmed working under abuse (for the record)
|
||||
|
||||
Path traversal in files, services, cron, ZFS, procs and updates all rejected. `/etc`
|
||||
and `/` protected from delete/rename as root. Unit, pool, dataset, snapshot and
|
||||
package names validated. Token issue refused with a wrong password. Oversized prefs
|
||||
rejected. Duplicate machine add refused. Unicode + spaces in file names round-trip.
|
||||
Upload byte-exact. Terminal scrollback survives switching. Offline node: switcher and
|
||||
header show it, pages show the reason. Remote terminals, logs, tasks and files through
|
||||
the proxy all work on all three nodes.
|
||||
|
||||
|
||||
# Bug bash — 2026-09-19
|
||||
|
||||
Found by driving a scratch instance (same binary, its own state, real fleet behind it) in
|
||||
headless Chromium at 1440 px and iPhone size: a sweep of every page on every machine for
|
||||
page errors, failed requests, overflow and small tap targets; then scripted use of every
|
||||
feature — terminal tabs, Files in a throwaway folder, Processes (killing a process the
|
||||
test spawned), Monitors customisation and widgets, Services, Cron, Updates, Disks,
|
||||
Machines, Settings, embeds, machine switching, session loss — and a hostile pass over the
|
||||
API. Host-changing actions (service control, cron edits, upgrades, power) were opened but
|
||||
not executed. Every fix below was re-checked by the same script that found it.
|
||||
|
||||
## Bugs — all ✅ fixed and verified
|
||||
|
||||
1. ✅ **One widget with a bad pattern silenced every custom widget.** They share one stream;
|
||||
`fan.[` made the whole query a 400, so all of them showed "…" forever with no reason
|
||||
given. The picker now asks the API before adding and shows its answer; a bad pattern
|
||||
already saved shows a notice instead of dots.
|
||||
2. ✅ **A saved layout of the wrong shape could not be loaded** ("could not load layout",
|
||||
and the page could no longer be customised). `normalize()` now accepts anything.
|
||||
3. ✅ **A lost session left pages saying "unauthenticated"** for up to 30 s. A 401 from the hub
|
||||
now returns to the login form at once, and logging in returns to the same page. (A 401
|
||||
relayed from a managed machine is that machine rejecting the hub, and doesn't.)
|
||||
4. ✅ **The machine-name rule in "Add a machine" was an invalid regular expression** under the
|
||||
browser's `v` mode, so it validated nothing and logged a console error on every render.
|
||||
5. ✅ **Two terminals opened in the same minute got the same name** ("shell 09:45" twice).
|
||||
Now "shell 09:45 (2)".
|
||||
6. ✅ **Names had no length limit** — a 328-character terminal tab, a 5000-character token
|
||||
name. 64 characters, refused with a reason; token names also refuse `/ \ < > "`.
|
||||
7. ✅ **"+ file", "+ folder" and Rename accepted `../x` and `a/b`**, quietly acting outside the
|
||||
folder on screen. A name is now one path segment; Move… is how things go elsewhere.
|
||||
8. ✅ **Unsaved file edits were thrown away without a word** when you picked another page or
|
||||
machine, or closed the tab. All three now ask; Cancel keeps you where you were.
|
||||
9. ✅ **A mistyped path in Files was remembered**, so a reload landed on the error. Only a
|
||||
folder that opened is remembered.
|
||||
10. ✅ **Escape closed nothing**: the machine menu, the file menu, the Power dialog, the widget
|
||||
picker. It closes all four.
|
||||
11. ✅ **Reset on Monitors removed every custom widget with no confirmation.** It asks, and
|
||||
says how many.
|
||||
12. ✅ **A few bytes of query bought 27 MB of answer** (`{"k0":"*",…×400}` with `full=1`) — and a
|
||||
read-only token is public on the demo page. A query may return 10 000 values (thirty
|
||||
whole machines); past that it is a 400 that says so.
|
||||
13. ✅ **No ceiling on live streams**: 300 from one token were accepted. 32 per client address,
|
||||
512 in all, then a 429 that says so; closed streams free their places.
|
||||
14. ✅ **Anything under `/embed/` lost the anti-framing headers**, not just the widget page.
|
||||
Only `/embed/widget` is frameable.
|
||||
15. ✅ **The new ZFS sensors ran three `sudo` commands a minute on every machine** — including
|
||||
`core`, which has the tools and no pool — and each wrote three lines to the journal:
|
||||
most of `hamlab.service`'s log was that. Pools are now read with one unprivileged
|
||||
`zpool list`; a source that finds nothing is asked again ten times less often; and
|
||||
the installer's quiet list covers `zpool|zfs list|status`.
|
||||
16. ✅ **Phone: the filter boxes on Processes and Files, the path box and the "/" crumb were
|
||||
under 30 px.** 36 px like every other control.
|
||||
17. ✅ **Disks → datasets on a phone: rows ended 50 px before their own cells** when scrolled
|
||||
sideways (`min-width` was less than the columns add up to).
|
||||
|
||||
Also: tab titles now say where you are ("Services · grid — Ham Lab"), so history and open
|
||||
tabs can be told apart.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Terminal focus after every switch/close/return; scrollback after reload; exited shells;
|
||||
30 tabs with the + still reachable; re-fit on resize; Back/Forward; unknown machine or page
|
||||
in the URL; wrong password; XSS through terminal, token and file names (all rendered as
|
||||
text); hidden files, filter, symlinks (broken ones marked), binary files, upload and
|
||||
replace, folder delete needing the typed name, root toggle; process sort, filter and
|
||||
Terminate (asks first); themes; password-change validation; duplicate or unreachable
|
||||
machine; hub terminals not leaking into another machine's page; CORS only on sensor paths;
|
||||
token list never exposing hashes; cookie flags; deep nesting, empty bodies, odd methods
|
||||
and `every=` values on the sensor API; phone drawer, key bar and picker.
|
||||
|
||||
## Quality of life — all ✅ done 2026-09-19, each verified in the browser
|
||||
|
||||
1. ✅ **Selections are in the URL.** `#/grid/services/ssh.service`, `#/hub/cron/job:…`,
|
||||
`#/hub/files/%2Fetc`: reload keeps them, links carry them, leaving a page drops them.
|
||||
2. ✅ **Browser Back with unsaved file edits asks**, like page and machine changes do.
|
||||
3. ✅ **No more browser `prompt()`/`confirm()` boxes** — all 30 are in-app dialogs (`askYes`,
|
||||
`askText` in `ui.tsx`): Enter submits, Escape or a tap outside cancels, destructive
|
||||
ones are red and say what they do ("Close", "Delete"), and they fit a phone.
|
||||
4. ✅ **Token creation is one dialog**: name, patterns, and the machine's own sensor families
|
||||
as chips to click.
|
||||
5. ✅ **Widgets can be edited** — sensor, type and title — in place, from Customize.
|
||||
6. ✅ **The Widgets gallery has a search** over IDs, names and labels (so "gitea" finds its
|
||||
container's sensors).
|
||||
7. ✅ **Status sensors keep a history** (300 readings each: an hour of container memory, two
|
||||
days of drive wear), so their sparklines have a past.
|
||||
8. ✅ **A stream that fails says why**: "the token was rejected", "too many live streams are
|
||||
open", "that machine is unreachable", "can't reach Ham Lab" — in embeds and on Monitors.
|
||||
9. ✅ **Pattern widgets get readable default titles**: `cpu.core.*.usage` → "CPU core usage".
|
||||
10. ✅ **Terminal tabs reorder** — drag on a desktop, ‹ › on a touch screen, remembered per
|
||||
machine in the browser — and **Alt+Shift+←/→** switches tabs, **Alt+Shift+T** opens one,
|
||||
with the keyboard left in the terminal each time.
|
||||
11. ➖ **"Log out everywhere"** — built, then removed the same day at Mitchell's word: no need
|
||||
for it on a single-user console.
|
||||
12. ✅ **Real clipboard shortcuts in the terminal**, from a physical keyboard: **Ctrl+C copies
|
||||
when text is selected** (and is still ^C when nothing is — the selection clears after a
|
||||
copy, so a second Ctrl+C interrupts), **Ctrl+V pastes**, Ctrl+Shift+C / Ctrl+Shift+V do
|
||||
the same unconditionally, Ctrl+Insert / Shift+Insert too. It rides the browser's own
|
||||
copy and paste events, which xterm already answers, so it works over plain http on the
|
||||
LAN where `navigator.clipboard` does not exist; bracketed paste is honoured.
|
||||
|
||||
|
||||
# Bug bash 2 — 2026-09-19 (evening): Alerts, Docker, Stacks, skins, the installer
|
||||
|
||||
Same method as the morning: a scratch instance with the real fleet behind it, headless
|
||||
Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four machines (errors,
|
||||
failed requests, overflow, tap targets) plus a contrast measurement of every skin; then
|
||||
scripted use of the new features on throwaway containers and stacks — a multi-file stack, a
|
||||
file edited behind the editor's back, two actions at once, a chatty log, odd container
|
||||
names, containers stopped and removed while open, every kind of bad alert rule, a rule's
|
||||
whole life (fire, narrow, pause, delete, survive a restart), token scope and hostile paths
|
||||
on the new APIs — and a regression of terminals, Files, widgets, deep links and colours.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **Deploying a stack made of several compose files dropped all but the first.** A stack
|
||||
deployed with `-f compose.yaml -f compose.override.yaml` was redeployed from
|
||||
`compose.yaml` alone, so everything the override set silently vanished from the new
|
||||
container. Two causes: only the first file was used, and the file list was read from
|
||||
`docker ps`, which joins labels with commas — the list is itself comma-separated, so it
|
||||
arrived truncated. Labels now come from `inspect`; every file is resolved, shown as a
|
||||
chip, editable, validated together and passed to compose in order.
|
||||
2. ✅ **A compose file changed on disk after you opened it was silently overwritten** (Dockhand,
|
||||
or a second tab). The editor now hands back the checksum it was given; a mismatch is a
|
||||
409 that says so, keeps your text, and offers to reload.
|
||||
3. ✅ **Stacks on a machine without Docker answered 500.** An empty list.
|
||||
4. ✅ **Every hub restart re-fired every firing alert** — and every deploy is a restart, so a
|
||||
phone would have been told about the same failed service on each one. What is firing is
|
||||
now kept in `state.json` and carried on, with its original start time.
|
||||
5. ✅ **An alert whose rule was paused, deleted or narrowed just vanished**, leaving "firing" in
|
||||
the history forever and no word to the phone. It is now recorded (and sent) as cleared,
|
||||
with the reason: "rule paused", "rule deleted", "no longer watched".
|
||||
6. ✅ **Nonsense rule values were accepted**: `null`, an object, `cpu.usage > ninety` (which can
|
||||
never be true, and would never say so), `true` with `<`. Refused, each with its reason.
|
||||
7. ✅ **No limit on rules** — 3000 were accepted, each checked on every machine every 15 s. 200.
|
||||
8. ✅ **Logs of a container that no longer exists answered 500.** 404.
|
||||
9. ✅ **Four skins had text under 3:1 contrast**: Nord's red; Catppuccin Latte's muted text and
|
||||
green; Solarized Dark's body text, muted text and red; Solarized Light's body text,
|
||||
muted text, accent and green. Each colour kept its hue and was moved to a readable value
|
||||
(measured: all thirteen skins now pass, body text ≥ 7:1).
|
||||
10. ✅ **Phone: the alert toggles (28 px) and the colour swatches (26 px) were under 30 px.**
|
||||
11. ✅ **The public installer had fallen four builds behind the fleet** (v65 published, v69
|
||||
running), so `curl … | sh` on an up-to-date machine "upgraded" it backwards. `make
|
||||
deploy` now republishes, and the installer compares versions: an older build is called
|
||||
older, defaults to No, and unattended runs refuse it without `HAMLAB_ALLOW_DOWNGRADE=1`.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
A second stack action while one runs (409); `../` and encoded tricks in stack names;
|
||||
shell metacharacters in container names, `since` and actions; `rm` / `kill` / `down` are
|
||||
not actions; a read-only token is refused on every Docker, Stacks and Alerts endpoint,
|
||||
directly and through the hub; container names with dots and underscores still get live
|
||||
CPU and memory; a log at five lines a second follows with no gaps or duplicates; a
|
||||
container stopped or removed while open updates itself; an alert rule's XSS name renders
|
||||
as text; per-machine firing, the rail badge, webhook and ntfy delivery; terminals, Files,
|
||||
widgets over container sensors, deep links, tab titles and machine colours all unchanged.
|
||||
No native browser dialogs anywhere; no page errors on any of 100 page loads.
|
||||
|
||||
## Found by the alerts themselves
|
||||
|
||||
`core` has one real alert firing: **`zfs-import-cache.service` fails at every boot** — `core`
|
||||
has the ZFS tools and no pool, so there is nothing to import. Harmless, but it will sit on
|
||||
the rail as "1" until the unit is disabled or the rule is narrowed. Left alone.
|
||||
|
||||
## Quality of life — five chosen by Mitchell, ✅ built and verified 2026-09-19
|
||||
|
||||
1. ✅ **Stacks on `grid` and the Mac are editable from the hub.** Their files live on `core`
|
||||
(Dockhand deploys over tcp); the hub resolves them and runs compose with `DOCKER_HOST`
|
||||
pointed at the node. Verified on a throwaway stack: saved on `core`, recreated **on
|
||||
`grid`**, same compose labels, nothing created on `core`, output followed on the page.
|
||||
2. ✅ **A stack's `.env` is editable**, values masked until "Reveal and edit". A `.env` that
|
||||
breaks the compose file (a required variable removed) is refused and the old one put back.
|
||||
6. ✅ **A link to a container that no longer exists says so**, by name.
|
||||
9. ✅ **"Update available" per image** — on request, remembered six hours. Verified with an
|
||||
image whose tag had moved on (update available) beside a current one (up to date).
|
||||
10. ✅ **What Deploy would change**: a line diff against what was last deployed from here;
|
||||
after a deploy it says there is nothing left to change. `.env` diffs name the key and
|
||||
never show a value.
|
||||
|
||||
Not chosen, still open: alert acknowledge/snooze; quiet hours and per-rule channels; push
|
||||
straight from Ham Lab; a type hint in the alert editor; switching off the nodes' own alert
|
||||
engines.
|
||||
|
||||
# Bug bash 3 — 2026-09-20: new stacks, new containers, images, volumes, the HUD
|
||||
|
||||
Started from Mitchell's report — the New stack dialog is see-through in HUD, its text
|
||||
mixing with the page behind. Same method as before: a scratch hub with the real fleet behind
|
||||
it, headless Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four
|
||||
machines, plus a measurement **in every skin** of everything that floats over the page
|
||||
(dialogs, the question box, the widget picker, menus, the phone drawer: background alpha
|
||||
must be ≥ 0.9). Then scripted use of what is new on throwaway `hl-` stacks, containers,
|
||||
images and volumes: the New stack / New container dialog's edges, stacks that are not
|
||||
deployed, a deploy that fails, a container name that is taken, deletes beside other
|
||||
people's files, a two-file stack, create → deploy → restart → stop → delete **on the Mac
|
||||
and on `grid` through the hub**, pulls on the hub and on a node, removal guards, a
|
||||
container stopped on purpose, a read-only token and hostile input on every new endpoint,
|
||||
and a regression of Escape everywhere, terminals, Files, sensors, alerts and deep links.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **HUD: every dialog was see-through.** The skin's "glass" rule (3 % cyan) was written
|
||||
for tiles and cards and also caught `.modal` — the New stack dialog, the widget picker,
|
||||
the Power dialog, the token dialog, the rule editor and every question. Anything that
|
||||
floats over the page is now solid in HUD (hairline border and glow kept) over a darker
|
||||
backdrop. Measured in all thirteen skins: nothing that floats is under 0.9 alpha.
|
||||
2. ✅ **HUD: the phone's slide-out menu was 92 % opaque**, the page showing through it. Solid.
|
||||
3. ✅ **Deploy, Update, Restart, Stop and Down all failed on the Mac's stack**
|
||||
(`peaches-and-plums`) — and on any stack held on the hub that was deployed from the hub's
|
||||
own shell rather than from inside Dockhand. The command was built as
|
||||
`'DOCKER_HOST=tcp://…' docker compose …`; quoted, `sh` reads that not as a variable but
|
||||
as the name of a program to run ("not found", exit 127). Now passed through `env`.
|
||||
Verified by a whole life on the Mac and on `grid` from the hub: create, deploy (running
|
||||
there, nothing on `core`), restart, stop, delete.
|
||||
4. ✅ **A container stopped on purpose was reported as down** when it has no restart policy
|
||||
and doesn't handle SIGTERM: `docker stop` then ends it with 137 (or 143, or 130), a
|
||||
non-zero code, which read as a crash — "Container down" would have fired for a
|
||||
deliberate stop. Those three codes now mean "a person", unless the kernel did the
|
||||
killing for want of memory (`OOMKilled`). A real crash (exit 3) is still down.
|
||||
5. ✅ **A stack with no containers offered Restart, Stop and Update**, and Restart
|
||||
"succeeded" having done nothing. Offered only when there is something to act on; the
|
||||
API answers 409 "isn't deployed: there is nothing to restart".
|
||||
6. ✅ **One stray tap outside the New stack dialog threw away everything typed** — on a phone,
|
||||
a compose file's worth. So did Escape. With anything typed, all three ways out (tap
|
||||
outside, Escape, Cancel) ask first; an untouched dialog still just closes.
|
||||
7. ✅ **Compose → Form → Compose silently replaced hand-edited compose text** with what the
|
||||
form says. The form now writes the box only when the form has changed since it last
|
||||
did, and asks before replacing text edited by hand.
|
||||
8. ✅ **A question asked from inside a dialog opened underneath it** (same layer, earlier in
|
||||
the page), so it could not be answered — found the moment the New stack dialog started
|
||||
asking. Questions now sit above whatever asked them.
|
||||
9. ✅ **Escape on a question also reached the dialog beneath it**: one key cancelled the
|
||||
question *and* closed (or re-asked from) the dialog under it. Escape now answers the
|
||||
question and goes no further; the next Escape is the dialog's. Checked on every dialog
|
||||
and menu: widget picker, machine menu, rule editor, Power, New stack, and that Escape on
|
||||
"Delete the rule?" deletes nothing.
|
||||
|
||||
10. ✅ **Phone: the bottom of the New container dialog could not be reached** (Mitchell, from
|
||||
his iPhone). A dialog is a column whose parts were allowed to shrink, so instead of the
|
||||
dialog scrolling, the one part with its own scrollbar — the compose preview — was
|
||||
squeezed to 22 px and everything "fitted". And `92vh` on iOS is taller than the screen
|
||||
with the address bar gone, so the top sat under the clock. Parts now keep their height
|
||||
and the dialog scrolls inside itself; it is never taller than the space between the
|
||||
status bar and the home indicator (safe-area insets, no `vh`); Create / Cancel — and
|
||||
any error — stay pinned at the bottom while the form scrolls. Applies to every dialog;
|
||||
measured at 390×844, 375×667 and sideways, and the picker, rule editor, token and Power
|
||||
dialogs rechecked.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
All 13 pages × four machines × both sizes: no page errors, failed requests, sideways
|
||||
scroll or tap targets under 30 px. A port of `not-a-port` is refused by compose before
|
||||
anything is written, and leaves no folder. Environment values with quotes, `#`, `$` and
|
||||
nothing after the `=` arrive in the container exactly as typed. A deploy whose image
|
||||
doesn't exist shows the registry's error and is **not** remembered as "last deployed". A
|
||||
container name already taken fails the deploy and says why. Delete moves only the stack's
|
||||
own compose files and `.env` (kept `600` in the trash) and leaves a `data/` folder and
|
||||
notes beside them alone; both files of a two-file stack go to the trash; the wrong name,
|
||||
or none, deletes nothing. A `.env` value never appears on the stack page and stays masked
|
||||
until asked for. A second stack of the same name is 409; an upper-case name is 400. A pull
|
||||
of an image that doesn't exist ends as a failed task; a pull on `grid` from the hub lands
|
||||
on `grid` and its output is followed. A volume or image in use can't be removed; a running
|
||||
container can't be removed (409); a volume needs its name twice. A read-only token gets
|
||||
401 from all fourteen new write paths, through the hub as well. `--force`, `-f`,
|
||||
`--all-tags`, `..%2F..` and a home outside the known ones are refused; a 1.2 MB compose
|
||||
file is refused. Shaped, fleet and streamed sensor queries, `/metrics`, the ten alert
|
||||
rules, terminal typing, the file editor's discard question and service deep links are
|
||||
unchanged. No native browser dialogs anywhere.
|
||||
|
||||
## Quality of life — four chosen by Mitchell, ✅ built and verified 2026-09-20
|
||||
|
||||
1. ✅ **"Where" fits on a phone**: `…/stacks/core · 12 stacks` instead of the whole path (which
|
||||
is still written out underneath). Two homes whose last two folders match fall back to the
|
||||
full path. Measured at 390 px: no option is wider than the box.
|
||||
2. ✅ **How a task ended is a badge**: green `finished`, red `failed · exit 1` with the box
|
||||
outlined in red, grey `cancelled`. Verified on a deploy of an image that doesn't exist.
|
||||
5. ✅ **Duplicate…** on a stack: the New stack dialog opened on a copy of its compose file,
|
||||
named `<name>-copy`. The original's `.env` is copied **on the machine** (`env_from`), mode
|
||||
600, its values never sent to the browser — checked: the secret is nowhere in the page.
|
||||
Anything that would collide with what is running is named before you press Create
|
||||
("a container called X already exists; port 18222 is already published by X") — for
|
||||
every new stack, not only duplicates — and clears as you fix it. Disabled while the
|
||||
original has unsaved edits. Verified: copy deployed beside the original, which never
|
||||
stopped.
|
||||
6. ✅ **The form has Labels and Network.** Labels one per line (`$`, quotes and spaces arrive
|
||||
intact). Network: its own, the machine's (`host`, which turns Ports off), or **join** one
|
||||
that already exists — written as an `external` network so compose doesn't make a second
|
||||
one. Verified on the container: both labels, attached to the chosen network only.
|
||||
|
||||
Found while building them, ✅ fixed:
|
||||
|
||||
- **Cancel on a stack task cancelled nothing.** It killed the `sh` that ran compose, not
|
||||
compose; and as compose still held the output pipe, the task stayed "running" and the
|
||||
stack "busy" until compose finished by itself — for a deploy waiting on a health check,
|
||||
minutes. Tasks now run in their own process group and the whole group is stopped (TERM,
|
||||
then KILL after 4 s); for stacks deployed inside Dockhand, where the process is not the
|
||||
task's child at all, the cancel goes into the container after it. Verified both ways:
|
||||
ended in ~100 ms, compose gone, the stack free for the next action.
|
||||
- **Phone: a stack's task output was squeezed to a line** under the buttons (the same fault
|
||||
as the dialog: a scrolling column whose parts were allowed to shrink), which also put
|
||||
Cancel out of reach. Detail panes keep their parts' heights; services, containers, stacks
|
||||
and cron rechecked on both sizes.
|
||||
- The Duplicate dialog and the stack page under it shared a React key, so the dialog
|
||||
multiplied. An undeployed stack's dot was green ("0 of 0 running"); now neutral.
|
||||
|
||||
# Bug bash 4 — 2026-09-21: the Top tab, and the refresh rate it didn't follow
|
||||
|
||||
Started from Mitchell's first look at Top on his own machine (Firefox on Windows, a window
|
||||
1392 CSS px wide — none of which the tab had been drawn for): "Command" printed over
|
||||
"Threads", the rate in Settings ignored, the symbol beside *logout* a box of hex digits;
|
||||
and make it follow the skins. Method as before, plus two things new: **Firefox** (Playwright's
|
||||
build; the GTK libraries it wants are unpacked from their `.deb`s into
|
||||
`~/.cache/ms-playwright/fflibs` and given to it as `LD_LIBRARY_PATH`, with
|
||||
`PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1` — nothing is installed on `core`), and a layout
|
||||
checker that measures instead of looking — every heading against its neighbour and its own
|
||||
text, every row against its box, every border label, at eight window sizes from 900 to 1920.
|
||||
Run first against the live v83 it reproduced the screenshot to the pixel ("the Command column
|
||||
is 11px wide" at 1392); run after, nothing, in both browsers.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **Between about 1280 and 1500 px the Command column collapsed to a letter** and its
|
||||
heading ran under "Threads:" (itself 4 px too narrow for its own text at every size). The
|
||||
table's fixed columns were sized for one window. The process box now measures *itself*
|
||||
(a container query — it is a third of a wide window and all of a narrow one, so the
|
||||
window's width says nothing) and drops the curve, then Threads, then Command, as it
|
||||
narrows; the box also gets half the width rather than 44 %, and no heading can spill
|
||||
into the next. Stacked layout now starts at 1280, not 1100.
|
||||
2. ✅ **The refresh rate reached a machine only through its Dashboard page.** The sampler on
|
||||
each machine was told the rate by an open Dashboard and nothing else — so with the rate
|
||||
at 250 ms, `grid`, `wyse` and the Mac were still sampling at 1 s (measured, live), and
|
||||
Top, Processes, Disks and every widget of theirs "ignored" the setting until somebody
|
||||
opened that machine's Dashboard. The rate is now the fleet's: saved with the UI prefs on
|
||||
the hub, which sets its own sampler and passes it to every machine (`PUT
|
||||
/api/prefs/sampler`, remembered in each `state.json`), and says it again at start-up for
|
||||
a node that restarted or joined since. Verified: Settings 2 s → 250 ms moves the sampler
|
||||
with no Dashboard visit.
|
||||
3. ✅ **Top read the process table at most once a second whatever the rate** — a clamp of
|
||||
my own, exactly what `REFRESH` promises not to do. It reads at the rate: 20 reads in 5 s
|
||||
at 250 ms, 10 at 500 ms. (59 frames a second with 390 rows at 250 ms, no stalls, both
|
||||
browsers — the rows are only redrawn when the table is read.)
|
||||
4. ✅ **Top's `− rate +` changed the rate for half a minute.** It set the browser's copy
|
||||
and never saved it, so the next sync with the hub put it back. It saves — which is also
|
||||
what moves the samplers — survives a reload, and each end is disabled at the limit.
|
||||
5. ✅ **A viewer whose clock is wrong froze the curves.** New points were timed against the
|
||||
machine's timestamps: a browser clock 7 minutes slow waited 7 minutes for its next point
|
||||
(Top *and* the Dashboard); 7 fast, and it added one per tick. Timed on the browser's own
|
||||
clock now. Verified with the page's clock shifted both ways: 5 points in 10 s at a 2 s
|
||||
stride, both pages, both browsers.
|
||||
6. ✅ **Every other point was being skipped.** "Has a stride passed?" was asked with `>=`
|
||||
of a timer that fires a few ms early half the time — so at a 2 s rate and a 2 s stride a
|
||||
curve gained 3 points in 10 s, not 5, and its time axis was quietly wrong. The same
|
||||
question on the server kept every bucket one sample long (14 s points at a 12 s stride).
|
||||
Both now allow half a tick of slack and keep to the stride's own beat; the server times
|
||||
from the last point kept, so a stalled or sleeping machine just keeps the next sample it
|
||||
gets. Unit-tested with jittered samples: 240 samples → exactly 40 points, 12 000 ± 100 ms
|
||||
apart, each the mean of its own six.
|
||||
7. ✅ **The power symbol was a typed `⏻`** (U+23FB, Unicode 9) — not in Windows' fonts, so
|
||||
Firefox there drew the missing-glyph box. Drawn as an SVG like the other icons. Every
|
||||
other symbol the console types was checked against the same risk: all older.
|
||||
8. ✅ **A handful of samples was stretched into a bar chart.** A machine ten seconds into
|
||||
its history filled the whole CPU box with five fat columns. The box is now always the
|
||||
whole history window — a young curve draws in from the right, as btop's does, and a full
|
||||
one fills it edge to edge, at any window (checked at 10 min and at 1 min).
|
||||
9. ✅ **Idle curves vanished**: a core at 10 % rounded to no dots at all in its mini graph.
|
||||
Anything above zero lights the bottom row, which is also btop's rule.
|
||||
|
||||
10. ✅ **Found by the deploy of fix 2 itself: telling the machines once is not enough.** The hub
|
||||
restarted first and told three nodes that did not yet have the endpoint (404); they then
|
||||
updated and carried on at 1 s. The same would happen to any node that is off or
|
||||
restarting when a setting changes. The hub now *keeps* the fleet in step: fifteen seconds
|
||||
after it starts and every two minutes after, it asks each machine for its sampling rate
|
||||
and history window and corrects only what differs (logged as `fleet: grid had ms=1000,
|
||||
now 250`).
|
||||
|
||||
11. ✅ **The net graph redrew its own past every second** (Mitchell, two screenshots a second
|
||||
apart: one tall spike in the first, a comb of medium ones in the second). That box is
|
||||
about 75 dot-columns wide and holds 300 points, and each column drew *the nearest point*
|
||||
— one in four, and a different one in four each time the curve advanced, so spikes came
|
||||
and went and the whole shape changed while the scale stayed put. The CPU graph did the
|
||||
same on a phone. Now a column draws the **peak** of all the points in it, and columns are
|
||||
cut from the data, not from the box (`web/src/graphcols.ts`): a finished column never
|
||||
changes, the graph only slides left by whole columns. Measured by reading every column's
|
||||
height off the canvas once a second under bursty traffic: on the live build the past
|
||||
changed shape in 18 of 47 frame pairs; after, 0 of 46 — and 0 of 45 on a young history.
|
||||
`npm test` (now part of `make test`) scrolls a spiky curve through both rules: nearest-
|
||||
point shows its three spikes as "900,500", then "700", then none; the new rule shows all
|
||||
three, at their heights, on every frame. The scale label now covers what is drawn (the
|
||||
window), not everything kept.
|
||||
|
||||
## Skins
|
||||
|
||||
✅ Top now wears the skin. The ground is the skin's terminal background and every curve and
|
||||
meter is built from its sixteen terminal colours (chosen to be read on exactly that ground),
|
||||
the rest from the usual variables — so the dark skins look like btop and the light ones like
|
||||
btop on paper, and a skin arriving from another device repaints the page in place. Measured
|
||||
in all thirteen: text ≥ 4.5 : 1, everything else ≥ 3 : 1, dots ≥ 2.2 : 1. Two failed —
|
||||
process names in Latte (2.75) and Solarized Light (2.62), where a terminal green is too
|
||||
pale for small type — and use `--ok`, the green each skin was already measured readable
|
||||
with.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Sort by pid, name, mem and cpu, and reverse; filter by name, command, user and pid, with the
|
||||
count showing matched/total and an empty result saying so; a selection that holds while the
|
||||
table is re-read under it; terminate, on a throwaway `sleep`, asks by name and pid, ends it,
|
||||
and the footer stops offering a process that is gone. Sampler values outside 100 ms–60 s are
|
||||
refused, and a rate slower than the collector's 10 s samples at 10 s. All 14 pages at his
|
||||
window size and on a phone, in both browsers: no errors, no overflow.
|
||||
|
||||
|
||||
# Bug bash 4, later — 2026-09-21: the SMART table outside its card
|
||||
|
||||
Mitchell, from the Disks page of `hamtv` with SMART open: the attribute table's rows ran out
|
||||
through the right-hand side of the card they belong to, the rules and the Thresh column
|
||||
drawn over the page behind.
|
||||
|
||||
12. ✅ **A table cannot be narrower than its widest unbreakable cell, and one of `hamtv`'s
|
||||
was 130px wide.** `.num` sets `white-space: nowrap`, and a SMART raw value is not always a
|
||||
number — the temperature attribute reads `34 (Min/Max 28/39)`. That one cell set the
|
||||
table's minimum width above the card's, so it spilled. Three changes, in this order,
|
||||
because the first two each looked worse than the bug in their own way:
|
||||
- the raw column wraps — but `overflow-wrap: anywhere` broke `4295163906` across two
|
||||
lines as "429516 / 3906", which reads as two numbers. It wraps at spaces only now;
|
||||
- a disk card is given room for six columns of SMART (`minmax(420px, 1fr)` rather than
|
||||
the shared 320px), and the cells' side padding goes 6px → 4px, which together is the
|
||||
~40px it was short by;
|
||||
- if some drive ever reports a raw value long enough to still not fit, the table scrolls
|
||||
inside its card rather than out of it.
|
||||
Verified on all four machines at six widths from 1721px down to a phone, in Chromium and
|
||||
Firefox: nothing leaves its card, no number is broken across lines, and the fallback
|
||||
scrollbar is never reached. The same measurement was run over every other page's cards
|
||||
and boxes — Dashboard, Top, Docker, Alerts, Machines and the rest — and found nothing.
|
||||
|
||||
# Bug bash 5 — 2026-09-21: navigation, and the fleet setting nobody had set
|
||||
|
||||
A sweep of the whole console after a week of changes, not of any one page: every page on every
|
||||
machine at four widths in **Chromium and Firefox**, then the everyday flows, then the URL.
|
||||
Measured rather than looked at — a heading against its neighbour and its own text, a box
|
||||
against what it holds, a request against the page that should have made it.
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
1. ✅ **A link to a page of the hub reloaded onto whichever machine you last had open.** The hub
|
||||
is `null` internally, so `parseHash()?.machine ?? sessionStorage…` read straight past it to
|
||||
the remembered machine: open `#/hub/docker/stack:gitea` after looking at `grid`, reload, and
|
||||
you were on `#/grid/docker`. The URL wins when it has an opinion, and "hub" is an opinion;
|
||||
the remembered machine is for a bare address only. Every link shape now survives a reload,
|
||||
and a bare address still lands where you left off.
|
||||
2. ✅ **From Settings, Machines or Alerts, a link to a machine's page arrived at its Dashboard.**
|
||||
Those three pages only exist on the hub, so moving to a machine bounces off them — but the
|
||||
bounce was decided from inside the machine change, which read the page being *left*, and
|
||||
landed after the URL's own page had been set. It is judged now once the machine and the page
|
||||
have both settled. `#/hub/settings` → `#/grid/disks` arrives at Disks; `#/grid/settings`
|
||||
still falls back to that machine's Dashboard.
|
||||
3. ✅ **A page being navigated away from briefly adopted the next page's selection**, and asked
|
||||
about it: leaving `#/grid/services/ssh.service` for `#/hub/docker/stack:gitea`, Services —
|
||||
still mounted for a moment — opened a log socket for a *service* called `stack:gitea`, on
|
||||
`grid`. Firefox says so in the console; Chromium swallows it. A page now reads the URL's
|
||||
selection only while the URL still points at that page. Checked over every hop between the
|
||||
five pages that keep a selection in the URL, in both browsers.
|
||||
4. ✅ **The hub only held its machines to a sampling rate if somebody had been into Settings.**
|
||||
The re-assert loop read the rate out of the saved UI prefs, which are written on change and
|
||||
not before — so on a hub where the rate had never been touched, a node left on a different
|
||||
rate was never pulled back. It compares against what the hub itself is running, which always
|
||||
exists. Verified by setting a node behind the hub's back: corrected within seconds.
|
||||
5. ✅ **"Everything is up to date." sat in a 20px sliver of a card**, flush to its border, where
|
||||
the package list it shares the card with is meant to be.
|
||||
|
||||
6. ✅ **A terminal of the hub's was asked for on a node.** Found in Firefox against the live
|
||||
console, which says out loud what Chromium swallows: `wss://…/api/nodes/grid/terminals/
|
||||
adfdeecb/attach`, where `adfdeecb` was one of the hub's — `grid` had no terminals at all. A
|
||||
pane builds its address when it connects, from whichever machine the console is pointed at
|
||||
*now*; switch machine while one is reconnecting and the reconnect goes to the new one. The
|
||||
address is fixed when the pane opens. The same was true of a service's log socket, and is
|
||||
fixed with it. Verified by walking the console around the fleet with a terminal open and
|
||||
checking every socket against the machine that actually owns what it asked for.
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Machine titles on every page and against an offline or never-reached machine; the colour a
|
||||
machine wears stays with the name it was added under, not its hostname. Files: create, open,
|
||||
edit, save, rename, delete. Services with its unit and log, cron jobs, containers and stacks,
|
||||
alert rules edited and put back, the terminal, the widget gallery. Login, a wrong password, a
|
||||
session cleared under an open page, logging back in onto the page you were on, and logout
|
||||
leaving nothing behind. The sampler, window and UI prefs endpoints under nonsense and hostile
|
||||
input, and a read-only token refused by all of them. The public demo page and installer. The
|
||||
Top tab on all four machines — including `grid`'s sixteen cores and thirteen filesystems, which
|
||||
scroll inside their boxes — its rate control, its filter, and three skins repainting it.
|
||||
`hamtv`'s eMMC really is called `hB8aP>`; that is what the device reports.
|
||||
|
||||
Firefox earns its place in this: three of the six above were invisible in Chromium, which
|
||||
swallows a failed WebSocket where Firefox writes it to the console. One thing it reports is
|
||||
*not* a fault — a page torn down by navigation while a terminal's socket is still shaking
|
||||
hands logs "can't establish a connection". Sitting still on a page produces none, the address
|
||||
is right, and the terminal belongs to the machine asked; `sockets.cjs` checks that ownership
|
||||
rather than the noise.
|
||||
|
||||
# Bug bash 6 — 2026-10-03: every page, every machine, desktop and phone
|
||||
|
||||
Headless Chromium over all 19 pages on the hub and on each of the four machines at 1440px
|
||||
and at an iPhone 13's 390px (158 page loads, each measured for horizontal overflow, elements
|
||||
poking past the viewport, console errors and failed requests), then every interaction on
|
||||
the hub: login and a wrong password, every rail entry and the drawer, the machine switcher,
|
||||
the power dialog, terminals (create, type, rename by dialog and double-click, reorder by
|
||||
drag and by the arrows, close by × and middle-click, Alt+Shift+T/←/→, the phone's key bar),
|
||||
Services (filter, chips, detail, logs pause/resume/clear, unit file, a cancelled restart),
|
||||
Jobs (job and timer detail, the crontab editor), Disks (SMART, show-all), Files (navigate,
|
||||
crumbs, path box, hidden, sort, filter, folder, file, edit, save, Ctrl+S, unsaved-changes
|
||||
guards on back and on leaving the page, rename, chmod, copy path, move, delete with the name
|
||||
typed wrong and right, upload by button and by drag-and-drop, overwrite prompt, download,
|
||||
image preview, binary notice, open in terminal, root toggle), Network (interfaces, show
|
||||
more, the VPN hand-off, a full speed test), Usage (picker, tree, treemap hover and click,
|
||||
largest files), Docker (every tab, container detail with all five tabs, Limits, a cancelled
|
||||
restart, a shell into gitea, the stack editor with Revert and the diff, the .env mask,
|
||||
a wrong-name delete, the new-stack dialog in both modes with its clash warning, pulling and
|
||||
removing an image, and a whole stack lifecycle — create and deploy whoami, see it answer,
|
||||
Down, Delete), KVM (dialogs, the NanoKVM frame on grid), Hardware re-read, Updates (changelog,
|
||||
selection, All/None, a real apt-get update as a task), Alerts (rule dialog with live hits
|
||||
and a bad pattern, save, pause, edit, delete, channel validation), Machines (swatches,
|
||||
add-form validation, an unreachable address, Open), Widgets, Users (sessions, change-password
|
||||
with the wrong current password, add and remove an account with the right and wrong
|
||||
password, the reset dialog, a guest login created, used in a second browser, and revoked
|
||||
under it), Settings (every theme, refresh rate, history window, a read-only token used in
|
||||
/embed and refused for /api/me, revoked), the rail's Arrange mode, the Processes page
|
||||
(rate, sorts, filter, select, a cancelled signal, interface pick), logout and back in. Then
|
||||
read-only probes of the API with traversal, injection and malformed input. Firefox could
|
||||
not run here (the box lacks its GTK libraries).
|
||||
|
||||
## Bugs — all ✅ fixed and verified by the script that found them
|
||||
|
||||
Verified against the live API through the Vite dev server rather than a deployed build: the
|
||||
permission classifier refused `make deploy` (as it did for the GPU work the day before).
|
||||
**Deployed 2026-10-08 as `v0.1.0-174-g0764508`**, with quality-of-life #2, #3, #8 and #10.
|
||||
|
||||
1. ✅ **The ZFS tab was wider than a phone.** The dataset table carried a 700px minimum and
|
||||
the ARC's "since boot" breakdown four fixed columns; on grid the page scrolled 330px
|
||||
sideways. Both drop to three columns and the ellipsis below 800px.
|
||||
2. ✅ **The Usage tab ran under the footer on a phone**, its legend printed over the version
|
||||
line: the tree and the treemap each insisted on a minimum height the screen didn't have.
|
||||
Below 800px the page scrolls, the tree 40vh and the map 60vh.
|
||||
3. ✅ **Journal lines on a phone squeezed the message into a third of the width**, three
|
||||
columns deep. Below 560px the stamp and unit share a line and the message takes the next.
|
||||
4. ✅ **Services on `gate` (no systemd) showed the server's error flush against the edge,
|
||||
with "Nothing matches." under it** as if a filter had hidden them. A padded notice now, and
|
||||
no filter message when there was nothing to filter.
|
||||
5. ✅ **ZFS "Snapshot now" offered a name seven hours wrong.** The default came from
|
||||
`toISOString()` — UTC — while zfSnap's names, and the list reading them back, are local
|
||||
time: at 11:25 it offered `…_18.25.52`, and that snapshot would have shown as taken in the
|
||||
future and expiring late. Built from local fields now.
|
||||
6. ✅ **"Adopt as a schedule" always ticked Recursive** — `/\s-\w*r/.test(cmd) || true`.
|
||||
7. ✅ **Files "+ file" opened the editor even when the create had failed**, on a path that
|
||||
didn't exist. It opens only after the create succeeds.
|
||||
8. ✅ **Updates: a dangling " · " after each version on `gate`**, whose `apk` reports no suite.
|
||||
9. ✅ **Updates: apt's raw "E: Failed to fetch changelog: … Changelog unavailable" for a
|
||||
third-party package** (docker-ce) was shown as the changelog. Said plainly now.
|
||||
10. ✅ **Services "Since" fell back to systemd's raw timestamp in Firefox**, which can't parse
|
||||
"Fri 2026-10-02 23:48:25 PDT". The fields are read directly now.
|
||||
11. ✅ **Hardware on `gate` led with a separator** — "· 4 threads · up to 2.70 GHz" — when the
|
||||
core count is unknown. The parts are joined, not concatenated.
|
||||
12. ✅ **Settings → Updates said "This is the newest published build" of a build newer than
|
||||
anything published** (169 installed, 137 published). It says so instead.
|
||||
13. ✅ **Escape did not close the phone's drawer**, alone among the overlays.
|
||||
14. ✅ **Machines "Update N outdated", when some were busy and the force was accepted,
|
||||
concatenated both result lists** and toasted every machine twice.
|
||||
|
||||
## Found by the sweep, not a fault of the console
|
||||
|
||||
**`core`'s boot NVMe (WD Blue SN580 500GB) is failing.** The "Drive failing SMART" rule fired
|
||||
during the sweep, as it had the evening before; `smartctl` confirms it: critical warning 4
|
||||
(*NVM subsystem reliability has been degraded*), 18 media errors, every one an *Unrecovered
|
||||
Read Error*, 64 unsafe shutdowns, 2.5 years powered on. The console did its job; the drive is
|
||||
a `hamlet` matter (`docs/pending.md`).
|
||||
|
||||
## Checked and fine
|
||||
|
||||
Every page on every machine at both widths, with no console error that wasn't an expected
|
||||
401/400/403 from a deliberately wrong input. Login lockout messages, the password eye. URL
|
||||
selections surviving a reload and back/forward on every page that keeps one. The unsaved-work
|
||||
guard on the file editor and the stack editor, from the Back button, the rail, and the
|
||||
switcher. Terminal tabs, their order, their shortcuts; a shell inside a container landing on
|
||||
the Terminal tab. The whole stack lifecycle, image pull and prune, container Limits. A guest
|
||||
login kicked out at the moment of revocation. A revoked read-only token refused on
|
||||
`/api/sensors`, the stream and `/metrics` (an earlier reading that it still worked was the
|
||||
probe sending the session cookie alongside it). Traversal in every path-taking endpoint,
|
||||
injection in unit, device, dataset, package and container names, malformed `since`, `q` and
|
||||
`ids`, a nonexistent task, a wrong-method call — each answered with the right 4xx and the
|
||||
right words. No session, no cookie: 401 on every API, locally and through NPM. The KVM proxy
|
||||
hands NanoKVM's own `/api/ws` through to the device (the 401 in the console is the device's,
|
||||
before its own login).
|
||||
|
||||
## Quality of life — to discuss, not built
|
||||
|
||||
1. **Treemap blank space.** Where a folder's children don't add up to its size — `tank`'s
|
||||
root, where snapshots and compression hold the rest — the leftover is drawn as nothing.
|
||||
An "unaccounted" tile would say what it is.
|
||||
2. ✅ **Sessions pile up** — built 2026-10-06. Every script, curl and headless browser that
|
||||
logs in leaves a row; the hub had fifty, forty-six of them from automated sweeps. Now a
|
||||
session unused for seven days signs itself out (`auth.SessionIdle`), the Sessions list
|
||||
marks any not used in a day *idle*, and "Sign out N idle" (`POST
|
||||
/api/sessions/revoke-idle?hours=24`) drops those while leaving every device used since.
|
||||
3. ✅ **The Mac's GPU box takes half the Processes page** — built 2026-10-07. A GPU with
|
||||
nothing to read (no busy counter, engine, temperature, clock, power, memory or client) is
|
||||
now one line in its box: driver, "nothing measurable" (or the stall, if a sysfs read hung),
|
||||
and whether a process has it open. Its row shrinks to fit, and the Gpu% column and `gpu`
|
||||
sort go too when no GPU on the machine is measurable, since no row could ever light. A
|
||||
client opening the device brings the full box back. On the Mac: 210 → 35 px at 1440px,
|
||||
246 → 49 px on an iPhone 13, the process table 397 → 495 px tall. Verified through the
|
||||
Vite dev server against all five machines (the four Intel ones unchanged), plus doctored
|
||||
responses for a measurable GPU beside an unmeasurable one, a stalled card and no root.
|
||||
4. **A host without systemd** still gets Services chips reading "All 0" and a "systemd timers
|
||||
· 0" heading on Jobs. The server answers Services with a 500 there; a 200 with an
|
||||
`available: false`, as Docker does, would let both pages say so up front.
|
||||
5. **An alert whose sensor vanishes resolves silently.** If `smartctl` errors out, the
|
||||
`smart.*.passed` sensor is omitted and a firing "Drive failing SMART" clears. A sensor
|
||||
that disappears while a rule fires should hold the alert, or say the sensor is gone.
|
||||
6. **Network "Past tests" rows are clipped** on desktop ("↓ 302 · ↑ 300 Mbps · 7.5 …") by the
|
||||
four-column props grid.
|
||||
7. **Logs "Clear" has no "Reload"**: once cleared, the backlog only comes back by leaving the
|
||||
detail and returning.
|
||||
8. ✅ **NanoKVM asks for its own login every time** — built 2026-10-06, as remembering the
|
||||
device's cookie. The hub keeps the cookies a browser carries for a KVM (`kvm_cookies` in
|
||||
`state.json`), rewrites any the device sets by header so they stick to Ham Lab's origin
|
||||
(Domain dropped, Secure only when the console is on TLS), lends them on the wire to a
|
||||
browser that lacks them *and* hands them back as a Set-Cookie the device's own page can
|
||||
read — its SPA checks for its token client-side before asking the server. A logout from the
|
||||
device, or removing the KVM, forgets them. Covered by `kvm_test.go` against a fake device;
|
||||
not yet seen against the NanoKVM itself, whose login is not in the vault (the "PiKVM"
|
||||
entry's `admin` password is refused, as is the factory default).
|
||||
9. **Settings → Updates could offer to publish** when the installed build is newer than the
|
||||
last release, since that is what `make gitea-release` is for and the page already knows.
|
||||
10. ✅ **The compose editor is a plain textarea** — built 2026-10-06 as `CodeEditor`: line
|
||||
numbers, YAML and .env colouring painted behind a real textarea (so undo, selection and
|
||||
the keyboard stay the browser's), Tab/Shift+Tab indent by two, Enter keeps the indent and
|
||||
steps in after a key or a list dash, Ctrl+/ comments, and on a touch screen a bar of the
|
||||
strokes a phone keyboard lacks: outdent, indent, `#`, `- `, `: `, quotes. Used by the stack
|
||||
editor (compose and .env) and the new-stack dialog.
|
||||
11. **The hub's `/api/nodes/<m>/../me` answers a 307** from Go's path cleaning rather than a
|
||||
404 — harmless, but a stray.
|
||||
12. **Guest logins can open Users and Settings** — by design ("a guest can do everything you
|
||||
can"), but worth a second look now that Users includes removing accounts.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Ham Lab (`hamlab`) — the web management console app
|
||||
|
||||
This repo is the **app** only: Go daemon + React/xterm UI, deployed as `hamlab.service`
|
||||
on `core`. Product scope, decisions and roadmap are in `PLAN.md`.
|
||||
|
||||
Everything about *how to work* — persona, response style, write-back rule, hosts,
|
||||
credentials, pending items — lives in the homelab repo **Hamlet** at
|
||||
`/home/core/projects/hamlet`, and applies here unchanged:
|
||||
|
||||
@/home/core/projects/hamlet/CLAUDE.md
|
||||
|
||||
Ham Lab facts that belong to the lab (where it runs, ingress, vault item) go in
|
||||
`hamlet/docs/hosts.md` and `hamlet/docs/pending.md`, not here.
|
||||
@@ -1,26 +1,77 @@
|
||||
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
|
||||
# Commit first, then deploy: the version is the commit the binary was built from, counted
|
||||
# from the base tag. A published release is tagged with a version of this same shape
|
||||
# (v0.1.0-137-gc129f65), and described from one of those a build would count from 137 again
|
||||
# and read as older than what it replaces — so tags with a dash are never the base.
|
||||
VERSION ?= $(shell git describe --tags --exclude '*-*' --always 2>/dev/null || echo dev)
|
||||
GO ?= go
|
||||
PREFIX ?= /usr/local/bin
|
||||
|
||||
.PHONY: all web build dev deploy clean
|
||||
# every architecture a release carries; arm is ARMv6 hard-float, so it covers every Raspberry Pi
|
||||
ARCHES ?= amd64 arm64 arm 386 riscv64
|
||||
LDFLAGS = -s -w -X main.version=$(VERSION)
|
||||
# static, no cgo: the same file runs on glibc, musl (Alpine) and anything else with a Linux kernel
|
||||
GOBUILD = CGO_ENABLED=0 GOOS=linux GOARM=6 $(GO) build -trimpath -ldflags "$(LDFLAGS)"
|
||||
|
||||
.PHONY: all web build go dev cross release gitea-release deploy push-nodes push-nodes-force test clean
|
||||
|
||||
all: build
|
||||
|
||||
web:
|
||||
cd web && npm ci --no-audit --no-fund && npm run build
|
||||
|
||||
build: web
|
||||
$(GO) build -trimpath -ldflags "-s -w -X main.version=$(VERSION)" -o bin/hamlab ./cmd/hamlab
|
||||
build: web go
|
||||
|
||||
# backend only (assumes web/dist already built)
|
||||
go:
|
||||
$(GO) build -ldflags "-X main.version=$(VERSION)" -o bin/hamlab ./cmd/hamlab
|
||||
$(GOBUILD) -o bin/hamlab ./cmd/hamlab
|
||||
|
||||
# one other architecture: make build-arm64, build-arm, build-386, build-riscv64, build-amd64
|
||||
build-%: web
|
||||
GOARCH=$* $(GOBUILD) -o bin/hamlab-linux-$* ./cmd/hamlab
|
||||
|
||||
dev:
|
||||
cd web && npm run dev
|
||||
|
||||
deploy: build
|
||||
./deploy/install.sh
|
||||
# siblings for every other architecture; installed beside the binary, and the hub pushes
|
||||
# each managed machine the one that matches it
|
||||
cross:
|
||||
@for a in $(filter-out $(shell $(GO) env GOARCH),$(ARCHES)); do \
|
||||
echo "build linux/$$a"; GOARCH=$$a $(GOBUILD) -o bin/hamlab-linux-$$a ./cmd/hamlab || exit 1; done
|
||||
|
||||
# dist/: one static binary per architecture, install.sh, SHA256SUMS. Serve the directory
|
||||
# anywhere (or attach it to a release) and install.sh does the rest.
|
||||
release: web
|
||||
rm -rf dist && mkdir -p dist
|
||||
@for a in $(ARCHES); do \
|
||||
echo "build linux/$$a"; GOARCH=$$a $(GOBUILD) -o dist/hamlab-linux-$$a ./cmd/hamlab || exit 1; done
|
||||
sed 's|@VERSION@|$(VERSION)|' install.sh > dist/install.sh && chmod +x dist/install.sh
|
||||
echo $(VERSION) > dist/VERSION
|
||||
cd dist && sha256sum hamlab-linux-* install.sh > SHA256SUMS
|
||||
|
||||
test:
|
||||
$(GO) vet ./... && $(GO) test ./...
|
||||
sh -n install.sh
|
||||
cd web && npm test --silent
|
||||
|
||||
# installs here (keeping the config, user and password), then updates every managed machine
|
||||
# Offer this build to every install, wherever it is: a release on Gitea, which is what
|
||||
# Settings -> Updates reads. Deliberate, and not part of deploy — deploy pushes to the
|
||||
# machines this hub owns, publishing offers it to everyone else.
|
||||
gitea-release: release
|
||||
./scripts/gitea-release.sh $(RELEASE_ARGS)
|
||||
|
||||
deploy: build cross
|
||||
./bin/hamlab install --prefix $(PREFIX)
|
||||
@sleep 1; $(PREFIX)/hamlab push-nodes || true
|
||||
@echo "deployed $(VERSION); 'make gitea-release' publishes it for everyone else"
|
||||
|
||||
# push the installed binary to every managed machine (what deploy does last)
|
||||
push-nodes:
|
||||
$(PREFIX)/hamlab push-nodes
|
||||
|
||||
# same, interrupting running tasks and attached terminals on the nodes
|
||||
push-nodes-force:
|
||||
$(PREFIX)/hamlab push-nodes --force
|
||||
|
||||
clean:
|
||||
rm -rf bin internal/web/dist web/dist
|
||||
rm -rf bin dist internal/web/dist web/dist
|
||||
|
||||
@@ -18,6 +18,10 @@ stub tab or a roadmap item.
|
||||
| Data | `/home/core/apps/hamlab/` — `config.yaml`, `state.db` (SQLite) | `apps/` is the nightly rsync target to `grid`, so it's backed up for free. |
|
||||
| Ingress | NPM proxy host `hamlab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
|
||||
| Repo | Gitea `mitchell/hamlab`, private, `main` | Created 2026-09-17. |
|
||||
| Guest logins | A **guest** is an account like any other, with `guest: true` and an expiry: `login`, sessions and prefs need no special case, and **it may do everything an account may do**. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. | Built 2026-09-27. A read-only version was built first and **cut the same day**: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is *an account*: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to. |
|
||||
| Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
|
||||
| Capability, not policy | **A tab offers what the selected machine can do, detected live — never what a rule says should run there.** Where the software is missing but installable, Ham Lab installs it; where it cannot be installed or the hardware lacks it, the tab says so in a sentence and offers what does work. Every action lands on the machine in the switcher. | Mitchell, 2026-10-09, after the Virtualization plan's first draft excluded the Mac mini and `hamtv` on the strength of `hamlet/CLAUDE.md`'s deployment rules: *"I want the option to do it anywhere I want, unless the hardware / software literally doesn't support it. Ham Lab needs to be built in a general-purpose way that can run on any Linux system."* Those rules bind the agent's unasked deployments, not the product. |
|
||||
| Ham Lab's data lives in Ham Lab's folder | **Everything a tab manages sits under the machine's data directory (`config.DataDir()`), never at a convenient path elsewhere.** Existing material is imported by copying. The engine's runtime state (`/var/lib/docker`, `/var/lib/incus`) is the engine's; Ham Lab keeps what would rebuild it. | Mitchell, 2026-10-09, when the Virtualization plan pointed its ISO library at `/tank/software/isos/`: *"ALL data needs to be under a HamLab folder, because it manages its own data."* The same reason `core`'s compose files moved under `apps/hamlab/stacks/` on 2026-10-05. `grid`'s and the Mac's stacks at `~/stacks/<machine>/` are the standing exception. |
|
||||
|
||||
## Layout
|
||||
|
||||
@@ -25,22 +29,22 @@ stub tab or a roadmap item.
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ header: Ham Lab · core · load 0.42 · ● connected · [logout] │
|
||||
├──────────────┬───────────────────────────────────────────────┤
|
||||
│ ▾ Terminal │ │
|
||||
│ shell 1 ● │ │
|
||||
│ backups │ main content (xterm) │
|
||||
│ + New │ │
|
||||
│ ▸ Services │ │
|
||||
│ ▸ Cron │ │
|
||||
│ ▸ ZFS │ │
|
||||
│ ▸ Dashboard │ │
|
||||
│ Dashboard │ shell 1 ● │ backups │ + │
|
||||
│ Processes ├───────────────────────────────────────────────┤
|
||||
│ Terminal │ │
|
||||
│ Services │ main content (xterm) │
|
||||
│ Cron │ │
|
||||
│ … │ │
|
||||
├──────────────┴───────────────────────────────────────────────┤
|
||||
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- Left rail: collapsible groups, one expanded at a time. Active leaf highlighted.
|
||||
- Terminal group: one leaf per session, `● ` when a process is running, "+ New
|
||||
Terminal" pinned at the bottom. Right-click / long-press → rename, close.
|
||||
- Left rail: flat list of pages, active one highlighted. No nested leaves.
|
||||
- Terminal page: tab strip across the top of the pane, one tab per session, `● ` when
|
||||
a process is running, `+` pinned at the right edge. Active tab carries rename and
|
||||
close; double-click renames, middle-click closes. (Until 2026-09-18 the sessions
|
||||
were leaves under a collapsible Terminal group in the rail.)
|
||||
- Non-MVP groups render as disabled leaves with "coming soon" — the rail shape is
|
||||
the product; stubs keep it honest about scope.
|
||||
- **Mobile**: rail becomes a slide-in drawer; xterm fits the viewport; a sticky key
|
||||
@@ -119,31 +123,680 @@ something long in one, switch away and back, refresh — it's all still there.
|
||||
|
||||
## Roadmap (not MVP — notes only)
|
||||
|
||||
- **Updates tab** — **shipped 2026-09-17** (`internal/updates`, `internal/tasks`,
|
||||
`Updates.tsx`): upgradable packages with current → candidate, suite, security badge
|
||||
and description; index age, last upgrade (from `/var/log/apt/history.log`),
|
||||
unattended-upgrades on/off, reboot-required banner with the packages that caused it;
|
||||
Refresh index, Upgrade all (dist-upgrade), Upgrade selected; changelog per package on
|
||||
demand. Runs go through the new **tasks** package: background command, output kept,
|
||||
followed live over `/api/tasks/{id}/stream`, cancellable, listed at `/api/tasks` (the
|
||||
seed of a Task History tab). apt is fully supported; dnf/pacman list only.
|
||||
- **Power** — header ⏻ button (amber when a reboot is pending), dialog that requires the
|
||||
hostname typed back, then `systemctl reboot` / `poweroff`. `/api/me` carries
|
||||
`reboot_needed` so the header knows.
|
||||
|
||||
- **tmux-backed sessions** so shells survive a daemon restart / redeploy. Would spawn
|
||||
`tmux new -A -s hamlab-<id>`; needs `mouse on` and status-bar handling in xterm.
|
||||
- **Services tab** — `systemctl list-units`, per-unit page: status, last 200 journal
|
||||
lines (live-tail via WS), start/stop/restart/enable buttons, unit file viewer.
|
||||
- **Cron tab** — parse `/etc/crontab`, `/etc/cron.d/*`, user crontabs, and systemd
|
||||
timers into one table with next-run times; edit via `crontab -` round-trip.
|
||||
- **ZFS tab** — pools, datasets (`used/avail/refer/usedbysnapshots`), snapshot list,
|
||||
scrub status, and the `freeing` counter. `grid` is the target, which implies
|
||||
multi-host.
|
||||
- **Dashboard** — CPU/mem/load/net, disks and SMART (`smartctl -j`), temps and fans
|
||||
(`sensors -j`), UPS from `pwrstat`. Ham Lab collects on a timer into a SQLite
|
||||
ring (e.g. 10 s resolution, 7 days) and draws sparklines. This is where the
|
||||
"collects its own data" requirement lands; no Prometheus dependency.
|
||||
- **Multi-host** — same binary as a headless *agent* on `grid` / Mac mini /
|
||||
`wyse-5070-1`, `core`'s instance is the UI and fans out over mTLS. The rail grows a
|
||||
host level at the top, exactly like Proxmox's datacenter → node tree. Design the
|
||||
MVP URL space so `host` can be inserted later without breaking clients.
|
||||
- **Auth hardening** — WebAuthn passkeys (phone Face ID), TOTP fallback, session
|
||||
list + revoke, optional NPM access list restricting to WireGuard/LAN.
|
||||
- **Files** — browse/upload/download/edit under `/`, with a sudo toggle.
|
||||
- **Cron tab** — **shipped 2026-09-17** (`internal/cron`, `Cron.tsx`): one list built
|
||||
from `/etc/crontab`, `/etc/cron.d/*`, every user crontab (via sudo), the scripts in
|
||||
the `run-parts` directories those drive, plus systemd timers. Next run computed by an
|
||||
in-house cron-expression evaluator (tested: ranges, steps, names, dom-OR-dow rule,
|
||||
`@` aliases); last run from the journal's `(user) CMD (command)` lines (cron logs via
|
||||
syslog to journald; reading needs sudo/adm). Commented-out jobs show as disabled;
|
||||
job-shaped lines inside comment blocks (Debian's header examples) are ignored. Per
|
||||
job: Run now (background, as the job's user, `sh -c` with crontab `%` rules, output
|
||||
captured), Enable/Disable (comments the line in place, refuses if the file changed
|
||||
underneath), Edit source (textarea; validated line-by-line; user crontabs installed
|
||||
with `crontab -`, system files written via `sudo tee`), Cron log, and Output file
|
||||
when the command redirects to one. Timers: Run now (starts the activated service),
|
||||
start/stop/enable/disable, live logs of the service via the Services tab's tail.
|
||||
- **Disks tab** — **shipped 2026-09-17** (`internal/storage`, `Disks.tsx`), replacing
|
||||
the planned ZFS tab: block devices from `lsblk -J` (kind, model, size, temperature
|
||||
from the device's hwmon node, partitions with filesystem usage, live read/write/busy
|
||||
from the metrics sampler); SMART via `smartctl -j` when smartmontools is present (one
|
||||
button installs it as a task otherwise): health, temperature, power-on time, self-test
|
||||
status, ATA attribute table with the failure-predicting IDs flagged, NVMe endurance and
|
||||
error counters, short/long self-tests. Spinning disks are read with `-n standby` so
|
||||
the tab never wakes a sleeping drive; "Wake and read" is explicit. ZFS section when
|
||||
the tools and module are there: pools with health, capacity, fragmentation, last scan
|
||||
and vdev tree; scrub start/pause/stop; datasets tree with used/refer/in-snapshots/
|
||||
snapshot count/compressratio/mount; per-dataset snapshot list and "Snapshot now".
|
||||
Verified against `grid` (raidz2 `tank`, 10 datasets, 515 snapshots, 8 WD40EZAZ + 970
|
||||
EVO Plus SMART). Snapshot destroy and dataset property editing are deliberately not
|
||||
there yet.
|
||||
- **Dashboard** — **first cut shipped 2026-09-17** (`internal/metrics`, `Dashboard.tsx`):
|
||||
CPU (total + per core), memory/swap, load, uptime, hwmon temps and fans read straight
|
||||
from `/sys` (no lm-sensors), per-interface network rates, per-disk read/write/busy
|
||||
from `/proc/diskstats`, filesystem usage, UPS via `sudo -n pwrstat -status`. Sampled
|
||||
every 2 s into an in-memory ring of 10 min; the UI (the **Dashboard** tab
|
||||
since 2026-09-19) reads it all through the sensor API — see Sensors below. **Customizable**: a
|
||||
Customize mode reorders and hides modules (CPU, Memory, Load, Uptime, Temperatures,
|
||||
Fans, Network, Disks, Filesystems, UPS) and hides individual rows (an interface, a
|
||||
mount, a disk, a sensor); the layout is stored server-side in `state.json` via
|
||||
`GET/PUT /api/prefs/dashboard` so every device sees the same dashboard. The update
|
||||
interval (100 ms – 2 s) is part of the layout and drives both the browser's polling and
|
||||
the daemon's sampling clock; the chart window is always 300 samples, so 30 s at 100 ms
|
||||
and 10 min at 2 s. Still to do: persistent
|
||||
history (SQLite ring, e.g. 10 s × 7 days), SMART (`smartctl -j`), longer time ranges.
|
||||
- **Updating from the Updates tab** — **shipped 2026-09-26**: the "what is out of date" list
|
||||
was read-only, so seeing a newer image and doing something about it were different places.
|
||||
Each row now has **Update**, and the header has **Update all**. Updating is per stack —
|
||||
that is the unit compose deals in — so a row acts on whichever stacks use that image, and
|
||||
several stacks run through a queue one at a time rather than recreating half the machine at
|
||||
once, each with its task output on screen. A row that succeeds says "updated just now"
|
||||
instead of re-checking, since a check is a registry request.
|
||||
- **Recreate, and the Updates tab at full width** (2026-09-26): a stack's actions gained
|
||||
**Recreate** (`up -d --force-recreate`), the only way out of a container Docker refuses to
|
||||
start — one pinned to a network that has since been recreated answers "network … not
|
||||
found" to Deploy and Update alike, because as far as compose is concerned nothing about it
|
||||
changed. Found the hard way when a SiYuan update landed in exactly that state. The Updates
|
||||
tab also stopped rendering as a list-and-detail pair: it is a page, not a list, so on a
|
||||
wide window it now sits under the tab strip at full width instead of being squeezed into
|
||||
the right-hand half beside an empty column.
|
||||
- **Cron in English** (`crontext.ts`, 2026-09-26): schedules read "daily at 3:15am" rather
|
||||
than `15 3 * * *`, with the expression kept beside it. Handles lists, ranges, steps, day
|
||||
and month names, `@daily` and friends, weekdays/weekends, midnight and noon — and declines
|
||||
to describe anything it is not sure of, because a confident wrong sentence is worse than
|
||||
five fields. The Jobs tab had its own smaller version of this; it now uses this one.
|
||||
|
||||
- **Hardware tab** — **shipped 2026-09-25** (`internal/hardware`, `internal/api/hardware.go`,
|
||||
`Hardware.tsx`): what the machine is physically made of, per machine like every other page.
|
||||
Four headline tiles (processor, memory, storage, graphics) over cards for each: the
|
||||
processor with its cores, clock range, socket, caches, microcode and the handful of feature
|
||||
flags anyone asks about; **every memory slot** with size, type, form factor, rated and
|
||||
running speed, maker, part number, serial, rank and voltage, and the empty ones marked
|
||||
empty; each drive with model, size, bus, firmware and serial; each display adapter with its
|
||||
driver, PCI address and video memory; and the board itself — model, firmware and date,
|
||||
serial, kernel. Read from what Debian already has (dmidecode for what only the firmware
|
||||
knows, lscpu, lspci, sysfs) and cached for ten minutes, since none of it changes under a
|
||||
running kernel; "Re-read" is there for the one thing that does, a drive plugged in.
|
||||
Every reader may come back empty and the page says what it could not read and why: a
|
||||
machine with no dmidecode, no root, or no PCI bus still gets a page. Verified on `core`
|
||||
(SMBIOS, Intel graphics) and on the Mac mini (Asahi: no SMBIOS memory table, a
|
||||
device-tree GPU, heterogeneous cores — where lscpu's "cores per socket" counts one
|
||||
cluster, so without SMT the thread count is the honest core count).
|
||||
|
||||
- **Sensors** — **shipped 2026-09-19** (`internal/sensors`, `internal/metrics/sensors.go`,
|
||||
`internal/api/status_sensors.go`, `web/src/sensors.ts`, `widgets.tsx`, `Embed.tsx`;
|
||||
reference in `SENSORS.md`): everything measured is a sensor with a stable dotted ID;
|
||||
catalog, single value, history, a **shaped query** (the client sends the object it wants
|
||||
and gets it back filled in — GraphQL's idea without GraphQL, since sensors differ per
|
||||
machine and a static schema would collapse to `sensor(id:)`), the same query as a live
|
||||
SSE stream, Prometheus `/metrics`, read-only `sensors:read` tokens (never full tokens in
|
||||
a URL), status sensors (services, updates, ZFS, SMART), a widget library, custom widgets
|
||||
on the Dashboard tab (briefly renamed "Monitors" 2026-09-19, back to "Dashboard"
|
||||
2026-09-20 at Mitchell's request; the view id and URL were always `dashboard`), and a
|
||||
frameable `/embed/widget` for Glance and friends. (An external demo page was deployed at
|
||||
`hamlab-demo.bigham.dev` and linked from the Widgets tab 2026-09-19; **both removed
|
||||
2026-09-23** at Mitchell's request — the Widgets tab shows the same thing against the
|
||||
machine you are signed in to, so the second copy earned nothing and cost a public
|
||||
read-only token, which was revoked with it. `examples/external-demo/` stays as an example
|
||||
to read.) The stock Dashboard modules and the Disks
|
||||
tab read the sensor API too (`sampleAdapter.ts`); `/api/metrics` is retired.
|
||||
Also per-service and per-container sensors (memory and CPU from cgroups), tokens limited
|
||||
to ID patterns, and `/api/fleet/sensors/query` for the whole fleet in one request.
|
||||
- **Alerts** — **shipped 2026-09-19** (`internal/api/alerts.go`, `Alerts.tsx`): rules over
|
||||
sensors (`container.*.running == false for 120 s`), evaluated by the hub every 15 s for
|
||||
itself and every managed machine in one shaped query each; `hamlab.reachable` is supplied
|
||||
by the hub so "a machine stopped answering" is a rule like any other. Ten defaults
|
||||
(machine offline, container down/unhealthy, failed service, disk full, pool unhealthy,
|
||||
SMART failing, UPS on battery, CPU hot, memory). Firing and resolved are events: kept
|
||||
(last 200, in `state.json`), shown on the hub-only Alerts tab with a count on the rail,
|
||||
and POSTed to channels — **ntfy** (title/priority headers) or a **webhook** (JSON; Home
|
||||
Assistant, Slack). The rule editor shows what a selector matches across the fleet as
|
||||
you type. Not done: quiet hours, per-rule channels, Web Push straight to the PWA.
|
||||
- **Docker tab** — **shipped 2026-09-19** (`internal/api/docker.go`, `Docker.tsx`):
|
||||
containers grouped by compose stack with live memory and CPU from the container
|
||||
sensors, start / stop / restart (asked first), logs that follow (polled with `--since`,
|
||||
no held connection), inspect (env *names* only), and images / volumes / networks
|
||||
tables. Works on every machine through the hub; `#/grid/docker/immich_server` links to
|
||||
one. **Stacks** (`internal/api/stacks.go`): every compose project, found from its
|
||||
containers' labels, with its file to edit and Deploy (`up -d`), Update (pull, then
|
||||
deploy), Restart and Stop as followed tasks. The file is looked for where the label says
|
||||
*and through every container's mounts*, because an orchestrator that runs compose inside
|
||||
its own container (Dockhand here; Dockge, Portainer) records a path in its own
|
||||
filesystem; compose is then run **inside that container**, so `.env`, relative paths
|
||||
and the compose labels stay exactly as deployed (verified: labels identical after a
|
||||
redeploy). Save validates with `compose config` first, keeps the previous file as
|
||||
`<file>.hamlab-bak`, and never deploys by itself. A stack deployed from another host
|
||||
(Dockhand on `core` driving `grid` over tcp) has no file on that machine, so the node
|
||||
alone lists it read-only — but **the hub fills that in** (`hubStacks`): asked about a
|
||||
node, it resolves the files on itself and runs compose here with `DOCKER_HOST` pointed
|
||||
at the node's `tcp://…:2375`, the way it was deployed; the task runs on the hub and the
|
||||
node's page follows it. Also per stack: every file it was deployed from (chips), its
|
||||
**`.env`** (values masked until "Reveal and edit"; validated in place, put back if
|
||||
compose objects), **what Deploy would change** (a line diff against the copy kept at the
|
||||
last deploy from here, `<file>.hamlab-deployed`, else the copy the last save replaced;
|
||||
`.env` values never shown), and **image update checks** — asked for, never polled, since
|
||||
each is a registry request Docker Hub counts; remembered six hours. A save refuses a
|
||||
file changed on disk meanwhile.
|
||||
**Making and unmaking (2026-09-20):** "+ New stack" — a name, a folder, compose text and
|
||||
an optional `.env` (written root-only), checked by compose before it is kept, then
|
||||
"Create and deploy" or "Create only". "+ New container" is the same dialog opened on a
|
||||
form (image, ports, volumes, environment, restart, command, host networking) that writes
|
||||
the compose for you: a container made here is a one-service stack, so it stays editable
|
||||
and redeployable, where a bare `docker run` could only be thrown away and typed again.
|
||||
New stacks go in a **home** — a folder where two or more stacks are already deployed from
|
||||
(Dockhand's `stacks/core` here; `~/stacks` on a machine with none) — and are deployed the
|
||||
way their neighbours were (inside `dockhand`, or over tcp to a node from the hub). Homes
|
||||
are also scanned, so a stack with no containers ("not deployed": never deployed, or taken
|
||||
down) is still listed. **Down** removes a stack's containers; **Delete…** (type its name)
|
||||
takes it down and moves its own files to `<home>/.hamlab-trash/<name>-<time>/` — moved,
|
||||
not erased; volumes and bind-mounted data are never touched. A stopped container can be
|
||||
**removed** (a running one must be stopped first, on purpose). Images: **pull** (a followed
|
||||
task), **remove** (unforced: Docker refuses one still in use) and **prune** (dangling only).
|
||||
**Volumes (2026-09-20):** each says which containers use it; one that nothing uses can be
|
||||
deleted — a named one by typing its name (it may be the data of a stack that is only
|
||||
down), an anonymous one by confirming its full name — and "Prune anonymous" clears the
|
||||
nameless leftovers of recreated containers, never named volumes. Images are marked "in
|
||||
use", and "Remove unused" takes every image no container uses. Nothing is ever forced:
|
||||
Docker refuses what is still in use, and that refusal is the answer. Still absent:
|
||||
removing networks.
|
||||
- **Top merged into Processes** — **2026-09-23.** The two tabs answered the same question
|
||||
from the same data (Top already read `/api/procs`, the endpoint the Processes table was
|
||||
built on), so at Mitchell's instruction there is now one tab, **Processes**, showing what
|
||||
Top showed: the btop-style whole-machine screen. `Procs.tsx` is deleted; the `top` view
|
||||
id and its route are gone. A rail order saved before this keeps its shape — `railorder.ts`
|
||||
maps the retired `top` to `procs` and drops the duplicate, so the merged page sits where
|
||||
Top sat rather than falling to the bottom.
|
||||
- **GPUs on the Processes tab** — **shipped 2026-10-02** (`internal/metrics/gpu.go`, the
|
||||
`GPUBox` in `Top.tsx`, `gpu.*` sensors). Asked for as "GPU activity and temps, like btop
|
||||
has it, always on". Each DRM device with a real GPU driver gets a box under the cpu box:
|
||||
busy curve, busy meter, a meter per engine (render/video/copy/enhance), temperature, clock,
|
||||
power and fan where the card has them, and the processes that have it open with their
|
||||
share. The process table gains a `Gpu%` column and sort. **How it is measured, and the
|
||||
honest gaps:** device busy is Intel's RC6 idle residency inverted (the counter every GT
|
||||
exposes without root; `intel_gpu_top` shows the same number) or amdgpu's own percent;
|
||||
per-process and per-engine shares are DRM fdinfo deltas, the way nvtop reads them. Other
|
||||
users' fdinfo needs root, so a daemon running as `core` keeps **one** root child
|
||||
(`hamlab drm-clients`, over a pipe) rather than a sudo a second. The Arc A380 on i915 6.12
|
||||
exposes power, energy and fan but **no temperature**, so its box shows none; an integrated
|
||||
GPU shows the CPU package temperature marked `*`. Asahi (the Mac mini) has neither a busy
|
||||
counter nor fdinfo in 6.19, so it lists the device and says so. No NVIDIA path: nothing here
|
||||
has one to test against. **Names come from an embedded PCI ID table** (2026-10-02,
|
||||
`internal/metrics/pciids_gpu.txt`, regenerated by `scripts/pci-ids-gpu.sh` from the hub's
|
||||
`pci.ids`): `gate` has no `lspci` and showed "Intel GPU 3185" beside `hamtv`'s "Intel UHD
|
||||
Graphics 600" for the identical chip. The table is tried first everywhere, so the name — and
|
||||
the sensor ID built from it — does not depend on which distro's pciutils a machine has;
|
||||
`lspci` is the fallback for a card newer than the table. Two fixes the same evening:
|
||||
the gpu box's layout rules had been lost from the stylesheet (v0.1.0-166/167), and on
|
||||
`grid` the sampler hung for 42 minutes in Go's poller on the Arc's `fan1_input` —
|
||||
`readStr` now uses plain blocking syscalls and each GPU's sysfs reads run under a
|
||||
3-second deadline (v0.1.0-168; `hamlet/docs/gotchas.md` has the mechanism).
|
||||
- **The ARC on the ZFS tab** — **shipped 2026-10-02** (`internal/metrics/arc.go`,
|
||||
`internal/storage/arc.go`, `GET /api/zfs/arc`, the Cache section in `ZFS.tsx`, `zfs.arc.*`
|
||||
sensors). Asked for as "current ARC usage, including any types of caching services in the
|
||||
pool". One card for the ARC: size against its ceiling with the target and floor, sparklines
|
||||
of size and hit rate from the sampler's history, what it holds (MRU/MFU/dirty, data vs
|
||||
metadata, compression, the ghost lists), the prefetcher and dbuf cache, memory-pressure
|
||||
counters, and the since-boot breakdown by demand/prefetch × data/metadata. Then a card per
|
||||
pool naming every caching tier — L2ARC, SLOG, special vdev, dedup table, spares — with its
|
||||
devices and fill, **or saying plainly that there is none and what that means** (the ZIL on
|
||||
the pool's own vdevs, with how much it has written), plus the properties that steer caching
|
||||
(`primarycache`, `secondarycache`, `sync`, `logbias`). Read from `/proc/spl/kstat/zfs/*`
|
||||
and `/sys/module/zfs/parameters`, with `zpool list -Hpv` and `zfs get` run without sudo
|
||||
(they need none) every 30 s at most. The day it shipped it showed `tank`'s ARC pinned at
|
||||
its 2 GiB floor under `arc_no_grow`, with a 53 GiB python3 process holding the RAM.
|
||||
- **Dashboard retired** — **2026-09-23**, at Mitchell's instruction, the same reasoning that
|
||||
merged Top into Processes: the app repeated itself. The customisable module list is gone
|
||||
(`Dashboard.tsx` deleted), and with it `/api/prefs/dashboard`, `applyPrefs`, and the
|
||||
per-user `dashboard` blob in `state.json`. The Processes tab already shows those numbers
|
||||
on one screen. `fmtBytes`/`fmtRate` moved to `fmt.ts` (Disks, Files and Space use them);
|
||||
`MONITOR_HISTORY` and `seriesOf` went with the page. The default view after login, after
|
||||
switching machines, and when a hub-only page is left behind, is now **Processes**. A saved
|
||||
rail order naming `dashboard` degrades quietly — `orderBy` drops ids with no page.
|
||||
**Widgets stays and says what it is for**: a notice at the top explains these are for
|
||||
dashboards *outside* Ham Lab — Glance, Home Assistant, a wall display — served from
|
||||
`/embed/widget` with a read-only sensor token, not a second copy of the console's screens.
|
||||
**It also closed a real bug.** `applyPrefs` set the machine's sampling rate from
|
||||
`intervalMs` inside the dashboard layout, and its `else` branch reset the sampler to the
|
||||
default whenever that field was absent. Since layouts became per-user, one account saving
|
||||
a dashboard could silently reset the sampling rate for the whole machine, overriding the
|
||||
fleet refresh rate set in Settings. The sampler now has one source: `refreshMs` in the UI
|
||||
prefs, which the hub fans out, remembered per machine as `sampler_ms`.
|
||||
- **ZFS tab, with scheduled snapshots** — **shipped 2026-09-24** (`internal/storage/snapshots.go`,
|
||||
`internal/api/zfs.go`, `ZFS.tsx`). Pools, datasets and snapshots moved out of Disks —
|
||||
Disks is the hardware, ZFS is the filesystem on top of it — and the tab gained what it was
|
||||
split out for: **snapshot schedules**. Hourly, daily, weekly or monthly, per dataset,
|
||||
optionally recursive, with a **retention period** and a floor that keeps the newest few
|
||||
whatever their age. A missed window (the machine was off) runs once on return rather than
|
||||
being skipped.
|
||||
**The naming is zfSnap's, deliberately.** `grid` has run zfSnap v1.11.1 from root cron at
|
||||
02:30 with a 6-week TTL since before this repo existed, and it works. So rather than
|
||||
invent a second convention for two tools to disagree over, this writes and reads exactly
|
||||
what zfSnap does — `tank@2026-09-24_02.30.01--6w`, **retention encoded in the name**. Three
|
||||
consequences: Ham Lab shows and expires the snapshots zfSnap already took; the two can run
|
||||
side by side without fighting, since both only remove what has passed the expiry written
|
||||
into its own name; and changing a schedule never re-dates old snapshots, because each
|
||||
carries the rule it was taken under. The page **detects the existing cron job**
|
||||
(zfSnap, zfs-auto-snapshot, sanoid, or a bare `zfs snapshot`) and says so, warning if a Ham
|
||||
Lab schedule is running as well.
|
||||
**Nothing is destroyed unseen.** Only names that parse as a stamp and a TTL are ever
|
||||
candidates for expiry, so a snapshot taken by hand is invisible to a schedule; Run now
|
||||
says that expired snapshots go with it; and a snapshot deleted by hand is confirmed first
|
||||
and needs its full name echoed back by the caller. (A "what would it delete?" preview
|
||||
shipped and was removed the same day as redundant — Mitchell's call.)
|
||||
The datasets table is a **snapshot explorer**: each dataset shows how many snapshots it has
|
||||
and what they cost, and opens into the snapshots themselves — when each was taken, **when
|
||||
it expires** (read from the TTL in its own name, so no server call), its size, and Delete.
|
||||
Deleting takes the same-named snapshot on descendant datasets too, which is how a recursive
|
||||
schedule made it.
|
||||
**A schedule is a real cron job** (2026-09-24, Mitchell's call after the first cut ran it
|
||||
from a ticker inside the daemon). Ham Lab writes `/etc/cron.d/hamlab-zfs`, owns it, and
|
||||
rewrites it whole on every change; each line calls `hamlab zfs-snapshot <id>` as the
|
||||
account Ham Lab runs as, reaching zfs through sudo as the daemon does. Snapshots are
|
||||
backup-shaped and must not stop because a console is down, upgrading, or uninstalled — and
|
||||
a cron line is greppable by anyone looking at the machine the normal way. **A cron
|
||||
expression is the single description of when a schedule fires**: the four common shapes
|
||||
render to one, `custom` states one outright, and the next run is read back with the same
|
||||
evaluator the Jobs tab uses on everyone else's lines, so the page, the file and the
|
||||
prediction cannot disagree. A run's outcome is written to `zfs-runs/<id>.json` beside the
|
||||
config, never to `state.json`, because the cron job is a separate process and the daemon
|
||||
would clobber it. An existing external job can be **adopted** — its schedule, retention
|
||||
and dataset read out of the cron line into a new Ham Lab schedule.
|
||||
**zfSnap is retired on `grid`** as of that day: Ham Lab now takes the nightly `tank`
|
||||
snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it
|
||||
had taken left untouched and fully readable here — which is what adopting its naming
|
||||
bought. See `hamlet/docs/hosts.md`.
|
||||
- **KVM tab** — **shipped 2026-09-24** (`internal/api/kvm.go`, `KVM.tsx`, `config.KVM`).
|
||||
Paste a device's address; its own web page appears in the tab. **Nothing in it knows
|
||||
anything about any particular KVM** — the first device here is a Sipeed NanoKVM on `grid`
|
||||
at `10.0.0.217`, but none of that is in the code.
|
||||
**Why it proxies rather than framing the device directly.** Ham Lab is served over TLS and
|
||||
a KVM usually is not, so a bare `http://…` iframe is blocked as mixed content before it
|
||||
loads. Serving the device from Ham Lab's own origin (`/kvm/<id>/…`) fixes that, puts it
|
||||
behind **Ham Lab's** login rather than a second one, and makes the iframe same-origin.
|
||||
**The general problem, and the answer.** A device UI asks for its assets by absolute path
|
||||
— `/assets/app.js`, `/api/whatever` — which no path prefix can rewrite away, and rewriting
|
||||
bodies per device would not be general. So the prefix is only the entry point: any request
|
||||
Ham Lab does not serve itself is routed to whichever KVM the asking page belongs to, read
|
||||
from the **Referer**, falling back to a **cookie** when there is none (a WebSocket upgrade).
|
||||
Ham Lab's own routes always match first, so a device can never shadow the console, and the
|
||||
fallback is gated on a live session. The device never receives Ham Lab's session cookie,
|
||||
and its `X-Frame-Options`/CSP/HSTS are dropped on the way out since Ham Lab is the origin
|
||||
now. Redirects to the device's root are rewritten back through the prefix.
|
||||
**One catch found in testing:** Ham Lab's own `frame-ancestors 'none'` forbids framing even
|
||||
from the same origin, so `/kvm/` paths send `'self'` instead. Without that the iframe was
|
||||
refused before it loaded.
|
||||
**The keyboard** needs less than it sounds: a page cannot forward keys into an iframe and
|
||||
does not need to, since one with focus receives them directly. Fullscreen plus
|
||||
`navigator.keyboard.lock()` is what takes Escape, Tab and Ctrl-W back from the browser;
|
||||
that is Chromium-only, and the tab says which version you are getting rather than
|
||||
pretending.
|
||||
**One KVM per machine, and the tab follows the machine switcher** (corrected the same day:
|
||||
the first cut was a hub-side list you pointed at machines, which had it backwards). Switch
|
||||
to `grid` and you get grid's; adding a second for a machine replaces the first. The
|
||||
addresses are still kept **on the hub** rather than on each machine, because a KVM earns
|
||||
its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub
|
||||
proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at
|
||||
all. Not done: holding the device's own login on the hub, so its page does not ask for a
|
||||
second password.
|
||||
- **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
|
||||
Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
|
||||
type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
|
||||
each repaints the terminal too. **Each machine wears a colour** (`accents.ts`): a stripe
|
||||
across the header and its name in the switcher — picked from ten swatches or any colour
|
||||
on the Machines page, else derived from its name — saved with the theme on the hub.
|
||||
- **Multi-host** — **Phase 1 shipped 2026-09-18** (`internal/nodes`, `internal/tlsauto`,
|
||||
`MachineSwitcher.tsx`, `Machines.tsx`). One binary, hub and spoke: a host whose
|
||||
`config.json` lists `nodes` is a hub; any other is a standalone node with its full UI.
|
||||
Node side: `"tls": "auto"` makes a self-signed cert in the data dir and serves HTTPS;
|
||||
`POST /api/tokens {password,name}` trades the login password for a long-lived bearer
|
||||
token (SHA-256 stored in `tokens`), accepted by the auth middleware alongside the
|
||||
cookie; `DELETE /api/tokens/{name}` revokes. Hub side: `POST /api/nodes {name,url,
|
||||
password}` peeks the node's certificate (trust on first use, fingerprint pinned),
|
||||
obtains a token named `hub:<hostname>`, stores `{name,url,fingerprint,token}`; a
|
||||
15 s prober keeps each node's status; `/api/nodes/{name}/…` is a reverse proxy to the
|
||||
node's `/api/…` (bearer swapped in, cookie and Origin stripped, streaming flush) that
|
||||
carries WebSockets, so terminals, log tails and task output work remotely.
|
||||
Frontend: `setMachine()` rewrites every `/api/…` path; the rail has a machine
|
||||
switcher at the top and a Machines page to add/remove/open. Pages remount on switch;
|
||||
terminals live on the node. Nodes: `grid`, `wyse`. **Phase 3 (update from hub)
|
||||
shipped 2026-09-18**: node `POST /api/self-update` takes a binary, checks it answers
|
||||
`version`, keeps `hamlab.prev` in the data dir, installs over the executable via
|
||||
`sudo install`, replies, then restarts the service; the hub pushes its own executable
|
||||
with `POST /api/machines/{name}/update` or `/api/machines-update-all` (architecture
|
||||
checked first via `/api/me.arch`), the Machines page shows an **outdated** badge and
|
||||
Update buttons, and `hamlab push-nodes` does the same from the CLI — `make deploy` runs
|
||||
it last, so one deploy on the hub updates every machine. First install on a new host
|
||||
is still by hand (scp binary + `install.sh`, `hamlab passwd`, `tls auto`). Still to
|
||||
do: Phase 2 Datacenter overview; optional firewall rule limiting a node's port to
|
||||
the hub. **Cross-architecture** (2026-09-18): `make deploy` also builds
|
||||
`bin/hamlab-linux-arm64`, `install.sh` installs it beside the binary as
|
||||
`hamlab-arm64`, and the hub picks the sibling matching a node's `/api/me.arch` when
|
||||
pushing — the Mac mini (Fedora Asahi, aarch64, node `mac`) updates like the rest.
|
||||
- **Portability + install** — **shipped 2026-09-18** (`internal/setup`, `internal/priv`,
|
||||
`install.sh`, `make release`): the binary installs itself (`hamlab install` /
|
||||
`uninstall`; systemd or OpenRC; root, sudo or doas), all builds static on amd64, arm64,
|
||||
arm, 386 and riscv64, `install.sh` as the curl-able bootstrap with checksums. Every
|
||||
privileged call goes through `priv` instead of a literal `sudo -n`. Updates tab acts on
|
||||
dnf, zypper, pacman and apk as well as apt. Services list falls back to the plain
|
||||
table on systemd < 246; Disks falls back to `/sys/block` without `lsblk`. Listener
|
||||
uses `IP_FREEBIND`. Self-update: pre-flight `check` against the live config, atomic
|
||||
replace, exec in place. Config default moved to `~/.config/hamlab` (existing
|
||||
`~/apps/hamlab` still found by the installer and the CLI). Tested in
|
||||
containers: Debian 12, Ubuntu 22.04, Fedora 40, Rocky 8, openSUSE Leap 15.5, Alpine
|
||||
3.18/3.20 (OpenRC + doas, non-root), plus the four lab hosts. **procd (OpenWrt) added
|
||||
2026-09-26** (`internal/setup/hamlab.procd`): detected on `/sbin/procd` + `/etc/rc.common`
|
||||
ahead of the OpenRC branch, installed to the same `/etc/init.d/hamlab` path and told apart
|
||||
from OpenRC at uninstall by `USE_PROCD=1` in the file. Verified on the new router
|
||||
(`gate`, OpenWrt 25.12.5, running as root): `S95`/`K10` symlinks, procd supervising the
|
||||
instance, Docker tab and stacks working through the hub. The Updates tab needed no change
|
||||
— apk-tools 3 answers `apk version -l '<'` the same way Alpine's does. Not done: an
|
||||
OpenRC or procd backend for the Services tab (it is systemd-only and says so cleanly);
|
||||
runit/s6/sysvinit service files; public hosting for `install.sh` (the Gitea repo is
|
||||
private, so the one-liner needs `HAMLAB_TOKEN`).
|
||||
- **Users, sessions and settings that follow the account** — **shipped 2026-09-22**
|
||||
(`internal/api/users.go`, `config.User`, `store.Session`/`UserState`, `Login.tsx`,
|
||||
`Users.tsx` — a hub-only **Users** tab in the rail, added the same day at Mitchell's
|
||||
request, holds everything about accounts: your own password, your sessions, the
|
||||
accounts; Settings keeps appearance, rates, tokens and About). **Rail order is per account** too (2026-09-22,
|
||||
`railorder.ts`, `Rail.tsx`): "⇅ Arrange" at the foot of the rail puts a handle on every
|
||||
page; drag one up or down (pointer events, so a finger works), let go and the order is
|
||||
saved in the UI prefs with the theme; "Reset" restores the built-in order; pages added
|
||||
in a later build go after the ones the saved order names. The login is a user name and a password. `config.json` holds `users`
|
||||
(name, argon2id hash, created, last seen); the old single `password_hash` becomes the
|
||||
account named after the OS user the console runs as on first boot, its signed-in
|
||||
sessions become that account's, and the machine-wide theme/accents/dashboard become
|
||||
theirs — nothing to redo after the upgrade. A session carries its user, an ID, when it
|
||||
was made and last seen, the IP and the browser; **Users → Sessions** lists yours
|
||||
("Safari on iPhone · 10.0.0.5 · seen 2 min ago"), revokes one, or signs out everywhere
|
||||
else. **Users → Users** lists accounts with their session counts, adds one, removes
|
||||
one (their sessions and settings go too; never yourself, never the last), and resets
|
||||
another's password (they are signed out); removing and resetting take your own
|
||||
password first. Everyone signed in can do everything — a household console, not a
|
||||
tenancy — and each machine keeps its own accounts: a hub reaches a node with its token,
|
||||
so pairing (Machines → Add) now takes a user and password on the node, the user
|
||||
optional while it has one account. Per account: `ui` (theme, accents) and `dashboard`
|
||||
in `state.json` under `users.<name>`. Still the machine's: refresh rate (it is how
|
||||
often every machine samples — it rides in the UI prefs object for the client but is
|
||||
stored once, machine-wide), graph history, alert rules, terminals (the shells run as
|
||||
one OS user anyway). A login attempt for a missing user takes as long as a wrong
|
||||
password, so the form can't list accounts. Every write's log line now names the
|
||||
account or token (`core@10.0.0.5 via …`), and Access tokens say who issued them. CLI:
|
||||
`hamlab passwd [user]` (makes the account if new; reads stdin when piped), `hamlab
|
||||
users`, `hamlab user-rm <user>`; `hamlab install` names the first account after the
|
||||
service user. Not done: a read-only viewer role.
|
||||
- **Space tab** — **shipped 2026-09-22** (`internal/space`, `internal/api/space.go`,
|
||||
`Space.tsx`), asked for as "exactly WizTree": what is using a disk. The unit is a whole
|
||||
volume, picked from a dropdown at the top: a **disk on its own** (every filesystem its
|
||||
partitions back) or a **ZFS pool as a whole** (every mounted dataset) — a pool's member
|
||||
disks are listed under it with the layout (raidz2, 8 disks: sda …) and never offered
|
||||
apart, since one disk's share of a pool means nothing (Mitchell's call, same day, after a
|
||||
first cut that listed the disks as chips). The walk runs as a helper process, `hamlab space-walk <roots>`,
|
||||
as root through `priv` when the host allows (so `/var/lib/docker` and `/root` count), 32
|
||||
workers reading directories depth-first, never crossing a filesystem boundary by itself;
|
||||
each root is grafted into one tree at its path. Files under 128 KiB are not kept one by one
|
||||
but summed per folder as "N smaller files" — what keeps a scan of millions of files to tens
|
||||
of MB in the daemon; the tree crosses back as a varint stream (`codec.go`), and the helper's
|
||||
memory goes with it. core's root: 646k files in ~1.2 s. Progress (files, bytes, current
|
||||
folder) every 250 ms; Cancel sends SIGTERM. The page: picker → Scan/Rescan → **Folders**
|
||||
(a tree largest-first: size, share of parent as a bar, files, folders, modified; expand in
|
||||
place) over a **treemap** of the selected folder — squarified, nested to the file, tiles
|
||||
coloured by kind (video, images, audio, archives & disk images, documents, code & config,
|
||||
databases & logs, other; the validated categorical set, dark and light) with folders as
|
||||
labelled boxes; hover for path, size and share; click a folder to drill, breadcrumbs and ↑
|
||||
to come back. The map asks the server for the folder pruned to what a tile could show
|
||||
(`min` = 1/6000 of the folder, 24 deep; deeper folders come back as one lump), the list
|
||||
one level at a time. **Largest files**: the 1,000 biggest with a path filter. Works on
|
||||
every machine through the hub.
|
||||
**Smarter and faster (2026-09-23, at Mitchell's request):** a scan is *kept* — in memory
|
||||
and in `<data>/space/<target>.scan` (the same varint stream) + `.json`, read back after a
|
||||
restart — and the next scan *starts from it*: the helper is handed the last tree on stdin
|
||||
and reads a directory again only when something says it changed. On a ZFS pool with
|
||||
snapshots, the scan records the newest snapshot of each dataset at start (`bases`), and the
|
||||
next one runs `zfs diff -H -F <base>` per dataset: only the directories it names are
|
||||
read, their ancestors kept but looked into, everything else kept wholesale without a
|
||||
stat. The diffs run side by side and one is given three minutes — `timeout -s KILL`
|
||||
inside sudo, since Debian's sudo gives the command its own session and a process-group
|
||||
kill from outside never reached zfs — after which that dataset is walked by mtime
|
||||
instead. On `tank` a diff is metadata-bound like the walk: 8 s to 3 min per dataset for
|
||||
a handful of changes, and `immich`, which changes constantly, ran past twenty minutes
|
||||
the first time. A refresh of `tank` took 4 min 19 s against 7 min for the full scan. Elsewhere (ext4, or a dataset with no snapshot) a directory whose mtime — to the
|
||||
nanosecond — is unchanged keeps its list and only its kept files are stat'ed again, since
|
||||
a file can grow without its directory noticing. A **cancelled** scan keeps what it walked,
|
||||
marks the folders it never got to (`partial`, "not yet read" in the tree) and **Resume**
|
||||
reads just those. core: full 1.2 s → refresh 0.3 s (2 folders read, 84,755 kept). The
|
||||
status line says which (`method`, `by_root`, read/kept counts); **Forget** drops the saved
|
||||
result so the next scan is full. The old result stays browsable while a refresh runs.
|
||||
**Treemap rethought** the same day: every file a tile grouped by folder — folders are only
|
||||
a thin line and a name strip on the first level — instead of nested boxes with headers,
|
||||
which on Linux's single-child folder chains was a staircase of headers around nothing.
|
||||
Not done: delete from here, a size-over-time history, per-user column choices.
|
||||
- **Updater** — **shipped 2026-09-23** (`internal/selfupdate`, `internal/api/update.go`,
|
||||
the Updates card in `Settings.tsx`, `scripts/gitea-release.sh`). A Ham Lab asks one URL
|
||||
what the newest build is and installs it on request. `update.url` in `config.json` is
|
||||
either a Gitea/Forgejo repo API base (`…/api/v1/repos/<owner>/<repo>`, whose newest
|
||||
release names the version and carries the binaries as assets) or any static directory
|
||||
holding `VERSION`, `SHA256SUMS` and `hamlab-linux-<arch>` — which is exactly what
|
||||
`make release` writes and what `install.sh` already downloads, so the demo site, a
|
||||
folder on a NAS and a Gitea release are all the same thing to it. `"off"` never looks.
|
||||
`update.token` is sent when the source is behind a login, so a **private** Gitea serves
|
||||
its own fleet without the instance being opened to the world. The binary is verified
|
||||
against `SHA256SUMS` before it goes anywhere near the running one, then takes the same
|
||||
path `POST /api/self-update` always did (runs `version`, runs `check` against this
|
||||
machine's config, atomic replace keeping `hamlab.prev`, re-exec in place) — that path is
|
||||
now one shared method, `installBinary`. Version ordering is `install.sh`'s `vnum()` rule
|
||||
ported to Go and tested against it, so the daemon and the installer never disagree about
|
||||
which of two builds is newer; the commit count since the tag counts, the hash does not.
|
||||
Checked ~90 s after boot and daily (`update.check: "never"` switches that off), never
|
||||
installed unasked — an update restarts the daemon and drops open terminals, and the
|
||||
dialog says so. Settings → **Updates**: installed, published, when it last looked, the
|
||||
source, the release notes, Check for updates, Install; the rail badges Settings when a
|
||||
newer build is out (`update_available` rides on `/api/me`). Publishing is
|
||||
`make gitea-release` → `scripts/gitea-release.sh`, which creates the release and uploads
|
||||
the five architectures, `SHA256SUMS`, `install.sh` and `VERSION`; it is deliberately
|
||||
**not** part of `make deploy`, because deploy pushes this build to the machines this hub
|
||||
owns while publishing offers it to every install anywhere, and one tag per deploy would
|
||||
be noise. Re-publishing a version needs `--force`.
|
||||
**Public since 2026-09-23**, at Mitchell's instruction and after the trade was put to him
|
||||
twice: Gitea's instance-wide `REQUIRE_SIGNIN_VIEW` is off and `mitchell/hamlab` is marked
|
||||
public, so anyone can browse it, clone it and download a release with no account. The
|
||||
other five repos there are each marked private and are invisible — an anonymous request
|
||||
for one is a 404, not a login prompt — registration is still disabled, and new repos
|
||||
still default to private. No token is needed by anything any more (the read-only one was
|
||||
revoked; only `hamlab-publish`, which creates releases, remains). Gitea resolves
|
||||
`releases/download/latest/<asset>`, so `install.sh` and `HAMLAB_URL` now point straight
|
||||
at the newest release and never need changing. What is public is the source, the history,
|
||||
`BUGS.md` and this file — which describes the lab and names the console's hostname.
|
||||
- **Auth hardening** — WebAuthn passkeys (phone Face ID), TOTP fallback, optional NPM
|
||||
access list restricting to WireGuard/LAN.
|
||||
- **Files tab** — **shipped 2026-09-18** (`internal/files`, `Files.tsx`): browse from
|
||||
`/` with breadcrumbs, an editable path box, sort, hidden-file toggle and a filter;
|
||||
rows show type icon, size, modified, perms and owner, link targets and broken links.
|
||||
Text files (≤ 2 MiB, sniffed) open in an editor with Ctrl+S / Save; images preview;
|
||||
everything else downloads. Upload via drag-and-drop or picker with per-file progress
|
||||
(streamed, no size limit). New folder / file, rename, move, chmod, copy path, delete
|
||||
(folders need the name typed back), and "open in terminal", which `cd`s the active
|
||||
shell (or a fresh one) to the folder. **Sudo toggle**: root access re-executes the
|
||||
same binary as `hamlab fs <op>` under `sudo -n`, streaming bodies over stdin/stdout,
|
||||
so both modes share one implementation and one set of rules (absolute paths only;
|
||||
`/`, `/etc`, `/usr`, … are never deleted or moved whole). Every write is logged with
|
||||
the caller's IP.
|
||||
- **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth.
|
||||
- **Notifications** — push on service failure / SMART warning / UPS event.
|
||||
- **Terminal extras** — search, copy-mode, font/theme, split panes, upload via
|
||||
drag-drop into cwd.
|
||||
|
||||
- **Settings + themes** — **shipped 2026-09-18** (`Settings.tsx`, `themes.ts`): Settings
|
||||
page with Appearance (Dark, Light, Cyberpunk, Retro — CSS variables on `<html>` plus a
|
||||
matching xterm palette, applied before first paint from localStorage and synced hub-side
|
||||
via `GET/PUT /api/prefs/ui` so the choice follows the login), Account (change password,
|
||||
`POST /api/password`, current password required), Access tokens (list/revoke the tokens
|
||||
hubs hold), About (version, host, arch, uptime, certificate).
|
||||
|
||||
## Network tab (2026-09-27)
|
||||
|
||||
A **Network** tab, per machine like every other: the interfaces as the kernel has them
|
||||
(addresses, MAC, MTU, negotiated link speed, carrier, byte totals, live rates from the
|
||||
sampler's own sensors), the default route and resolvers, what the internet sees of this
|
||||
connection, and an **internet speed test**.
|
||||
|
||||
Decisions worth keeping:
|
||||
|
||||
- **Measured by the daemon, not the browser.** The test runs on the machine in the switcher,
|
||||
so the figure is that machine's line — the router's, if you switch to `gate` — and it
|
||||
carries on if the phone locks or the tab closes. The page polls a run it is not driving.
|
||||
- **No tool to install.** It measures against Cloudflare's public speed endpoints
|
||||
(`__down`, `__up`, `cdn-cgi/trace`), which is what makes it work identically on Debian and
|
||||
on OpenWrt, where `opkg install` costs flash. That is the feature's one external
|
||||
dependency, and it is named in `internal/network/speedtest.go`.
|
||||
- **Several streams, HTTP/2 off, and the ramp thrown away.** One stream over one connection
|
||||
measures a congestion window; HTTP/2 would put every stream back on one connection and do
|
||||
the same. The first second of each phase is measured and discarded.
|
||||
- **Latency is the round trip less the far end's own Server-Timing.** A zero-byte response
|
||||
is mostly Cloudflare's worker running — 20 to 45 ms of it against a few ms of wire. Left
|
||||
in, it reported 31 ms for a 7 ms line. Jitter is the mean change between consecutive
|
||||
probes, not a standard deviation, so one slow probe doesn't run away with it.
|
||||
- **A refused phase is not a number.** The far end rate-limits by volume, and a test moves a
|
||||
few hundred megabytes; several in a row earn 429s, and the block observed while building this
|
||||
lasted over twenty minutes. So: a 20-second cooldown counted from the last *attempt* (a
|
||||
refused test still used the line), a stream that retries twice before giving up, the rate
|
||||
computed over the window in which bytes were actually still arriving, and a plain sentence
|
||||
when it was cut short. Coasting to the end of the clock with nothing arriving reported a
|
||||
third of the truth.
|
||||
- **A second download source, for when the first refuses.** A plain 100 MB file on Linode's
|
||||
Seattle mirror, which has no opinion about volume. It is only reached after a refusal, and
|
||||
every result names the source it used — that mirror is in one place, so on a machine far from
|
||||
Seattle it would read low, and a number that quietly came from somewhere else would be a lie.
|
||||
Upload stays on Cloudflare, which has never been the one to refuse.
|
||||
- **The last result is published as sensors** (`net.internet.*`), so the line is a widget, a
|
||||
graph, an alert or a Prometheus series like anything else. Each carries the time the test
|
||||
ran, not the time it was asked for.
|
||||
- **Not scheduled.** Nothing takes a speed test on its own. A nightly one recorded to the
|
||||
history is the obvious next step and deliberately not built yet — the cron shape for it
|
||||
already exists twice over (ZFS snapshots, Docker image updates).
|
||||
|
||||
## VPN tab (2026-09-27)
|
||||
|
||||
A **VPN** tab beside Network, per machine: every WireGuard tunnel on the machine in the
|
||||
switcher, and the devices on it — name, address, last handshake, where it connects from, and
|
||||
what it has moved.
|
||||
|
||||
- **Per machine is the point, not a limitation.** The tunnel people dial into and a tunnel a
|
||||
machine dials out through are on different boxes here (`gate` and `hamtv`), and a combined
|
||||
view would invite exactly the question "which box is this on".
|
||||
- **The kernel is the truth, the config only supplies names.** `wg show all dump` has the
|
||||
handshakes and counters and knows nothing but public keys; names come from the `# comment`
|
||||
above a peer in a wg-quick file, or a UCI section's `description` on OpenWrt. Both are read
|
||||
only to label what the kernel already said.
|
||||
- **Server or client is worked out, not declared.** A peer allowed `0.0.0.0/0` that has an
|
||||
endpoint is a way out, so a tunnel holding one is this machine dialling out; a tunnel with a
|
||||
listen port and peers is one being dialled. Anything else says neither rather than guessing.
|
||||
- **"Online" is a handshake in the last three minutes**, and the tab says so. WireGuard is
|
||||
silent when idle; there is no connection to be in, and a tab claiming otherwise would be
|
||||
lying twice a day.
|
||||
- **No secrets are read.** Private keys are in the second column of the dump and are read
|
||||
past; pre-shared keys are noted as present and never read.
|
||||
- **Devices can be created, renamed and removed** (added 2026-09-27, asked for the same day),
|
||||
on a tunnel the machine configures — today OpenWrt's UCI, which is where the server here
|
||||
lives. A wg-quick host is reported unmanaged and shows no buttons: half-supporting a second
|
||||
configuration format is worse than an honest refusal.
|
||||
- **Nothing restarts the network.** `uci commit` writes the file and applies nothing; `wg
|
||||
set` changes one peer on the live interface. Proven on `gate` by the existing peers'
|
||||
transfer counters surviving a create and a delete — a reload would have zeroed them.
|
||||
- **The two halves are ordered so they cannot disagree quietly.** Adding writes the
|
||||
configuration first and rolls it back if the live tunnel refuses, because a device that
|
||||
works only after the next reboot is the worst of the three outcomes. Removing does the
|
||||
live tunnel first, since that is the moment access ends, and says so plainly if the
|
||||
configuration half then fails.
|
||||
- **The keypair is generated on the machine** and the profile, with its QR, is shown once.
|
||||
The private key is stored nowhere — the same bargain as a guest login's password.
|
||||
- **Removing asks for the name to be typed back**, at both ends. It cannot be undone and it
|
||||
sits one tap from a list of devices that all look alike.
|
||||
|
||||
## To do
|
||||
|
||||
- **Grow the Files tab until copyparty can go** — asked for 2026-09-23 as two items (Files
|
||||
polish, and deprecating copyparty), merged 2026-09-24: they are the same tab and the same
|
||||
work. `copyparty` on `grid` (`copyparty.service`, `:3923`, `/etc/copyparty.conf`, serving
|
||||
all of `/tank` as user `core` with one account behind NPM) overlaps **Files** almost
|
||||
entirely. The aim is to stop the service and keep only Ham Lab, with general ease-of-use
|
||||
polish along the way. What Files must gain first, in the order that matters:
|
||||
- **Multi-select and bulk actions.** copyparty selects many rows and moves, deletes or
|
||||
downloads them together; Files acts on one row at a time.
|
||||
- **Download a folder**, and a multi-file download, as a streamed zip or tar — the thing
|
||||
copyparty is reached for most.
|
||||
- **Resumable uploads.** copyparty's up2k chunks a file, survives a dropped connection
|
||||
and skips what is already there; Files streams one PUT, so a phone upload that drops at
|
||||
90% starts over.
|
||||
- **A gallery.** Thumbnails for a folder of images and video, and play audio/video in
|
||||
place — `/tank/media` and `/tank/immich` are most of what is browsed.
|
||||
- **Search** below the current folder by name, since `find` in a terminal is the fallback
|
||||
today.
|
||||
- *Decide when built:* whether share links (copyparty's public per-path links) are wanted
|
||||
at all — they are the one feature that hands out access without a login, and Ham Lab has
|
||||
no notion of that; if not, say so in the docs rather than leaving a gap unexplained.
|
||||
- *After:* `systemctl disable --now copyparty`, remove its NPM host, keep
|
||||
`/etc/copyparty.conf` until a rollback is clearly not needed, then update
|
||||
`hamlet`'s `docs/hosts.md` bare-metal list, `docs/gotchas.md` (the "`docker ps` is not
|
||||
the inventory" note names it) and the `grid` row in `CLAUDE.md`.
|
||||
|
||||
- **Grow the Docker tab until Dockhand can go** — asked for 2026-09-23 and widened
|
||||
2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the
|
||||
Files item: it is one tab, and the gaps that block retiring the old tool and the things
|
||||
that would simply make it better are the same body of work. The *homelab* half —
|
||||
the cutover itself, then moving every stack off Dockhand's data dir — is `hamlet`'s
|
||||
`docs/pending.md` → "Retire Dockhand; Ham Lab becomes the orchestrator", agreed
|
||||
2026-09-20. This entry is the *app* half.
|
||||
|
||||
**Done 2026-09-24 — the eight Mitchell picked from the inventory**, all in the Docker tab:
|
||||
per-container **Stats**, **Health** (with the last few probe outputs), **Events** (Ham Lab
|
||||
keeps its own 4000-entry ring, since `docker events` forgets after about a minute),
|
||||
**Shell** (`docker exec`, handed to the Terminal tab) and **Limits** (`docker update` for
|
||||
memory, CPU and restart policy); an **updates** tab giving the whole machine at once with
|
||||
a per-container indicator in the list; and **schedules**, which write
|
||||
`/etc/cron.d/hamlab-docker` and call
|
||||
`hamlab docker-update <id>` — the shape the ZFS schedules already use.
|
||||
Also done: Ham Lab keeps working when `dockhand` is removed. It remembers where a
|
||||
container's bind mounts pointed (`mounts` in `state.json`) and resolves a stack through
|
||||
that record when no running container explains its path, running compose on the host
|
||||
instead; `repoint` does the same for a schedule at run time. Without this, removing the
|
||||
container would have made 17 stacks on `core` and 5 on `grid` unfindable at once.
|
||||
|
||||
- **Left from the Dockhand inventory, both to decide rather than build.**
|
||||
- *Git-sourced stacks.* Dockhand's `stack_sources` table points a stack at a repo and
|
||||
pulls it. Its `git_repositories` and `git_stacks` tables are **empty**, so nothing
|
||||
here uses it — dropping it is the honest answer unless Mitchell says otherwise.
|
||||
- *Webhooks, OIDC, LDAP, vulnerability scans.* All four tables empty. Nothing to move.
|
||||
|
||||
**Done 2026-09-25 — the two gaps in the tab itself.**
|
||||
- *Logs are followed over a held connection*, the way the Services tab follows a journal
|
||||
(`internal/api/docker_logs.go`, `/api/docker/containers/{id}/logs/tail`): one
|
||||
`docker logs -f` per viewer, stdout and stderr both, stderr coloured as an error. A
|
||||
reconnect resumes from the last line's timestamp, so a container that restarts or a
|
||||
phone that slept doesn't replay what is already on screen; the client dedupes by line
|
||||
rather than by "later than", because `--since` is inclusive and two lines can share a
|
||||
timestamp. Docker replays the opening backlog down two pipes in no particular order, so
|
||||
the first 300 ms is held and sorted before it is sent. The polled `?since=` endpoint
|
||||
stays — it is what the resume is built on. Reconnects back off to 15 s, since a
|
||||
container that has stopped ends the stream immediately.
|
||||
- *Networks can be removed and pruned* — `POST /api/docker/networks/{remove,prune}`, the
|
||||
shape images and volumes already had. The overview says who is attached to each one, so
|
||||
the ✕ is only offered on a network nothing uses and never on Docker's own three. Prune
|
||||
is deliberately untested against this host: `core` has several networks belonging to
|
||||
stacks that are merely down (`pihole_default`, `transmission_default`, `gluetun_default`
|
||||
among them), and pruning would take them and hand them new subnets on the next deploy.
|
||||
|
||||
- **The `*:2375` question, which this is the moment to settle.** A stack deployed to
|
||||
`grid` or the Mac still goes over `tcp://<node>:2375` from the hub. Decide whether the
|
||||
hub should instead ask the node's own Ham Lab to deploy locally, which would end the
|
||||
unauthenticated Docker socket exposure — its own item in `docs/pending.md`, currently
|
||||
filed under "decided, no action", and this is the change that would make closing it
|
||||
free.
|
||||
|
||||
- **A Virtualization tab** — asked for 2026-09-23; **scoped, built and deployed 2026-10-09 — see
|
||||
`VIRTUALIZATION.md`** for what shipped (phases 1 and 2) and what is left. Incus 6.0 LTS as the engine; the tab detects per machine whether Incus is
|
||||
running, installable, or unavailable, and whether KVM is there, and acts on the machine in
|
||||
the switcher — never a default host. Containers and VMs in one list, console/shell through
|
||||
the terminal machinery, snapshots, exports, and the old Proxmox estate on `tank` as a later
|
||||
import. That file has the probe of every host, the per-machine layout Ham Lab computes
|
||||
(pool driver, a bridge subnet that avoids the LAN's /8 and Docker's pools, the Docker
|
||||
`FORWARD` fix) and the phased build.
|
||||
|
||||
|
||||
## Open questions
|
||||
|
||||
- Idle-session reaping policy (never / after N days) — leaning never; it's a
|
||||
|
||||
+164
@@ -0,0 +1,164 @@
|
||||
# Portability audit — what in Ham Lab is specific to this lab (2026-10-09)
|
||||
|
||||
Goal: Ham Lab drops onto any set of Linux machines and works. This is the full sweep of
|
||||
every place the code, defaults, docs or UI assume Mitchell's hosts, network, hardware,
|
||||
distro, init system, orchestrator history or posture. **Nothing here is fixed yet** — this
|
||||
is the list to work from. Each item: where, what it assumes, what the general form is.
|
||||
|
||||
Severity key: **A** breaks or misbehaves on an ordinary host · **B** works but is wrong by
|
||||
default or phones home to this lab · **C** cosmetic (placeholder text, comments, docs).
|
||||
|
||||
---
|
||||
|
||||
## 1. Hard-wired lab identifiers
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 1.1 | **B** | `internal/selfupdate/selfupdate.go:36` `DefaultURL = "https://git.bigham.dev/api/v1/repos/mitchell/hamlab"` | Every install with no `update.url` checks **this** Gitea 90 s after boot and daily (`internal/api/update.go:1951, 2016`). A stranger's install phones home here by default. | Default should be **off** (or a build-time `-X` variable set at release). Installer asks or `install.sh` passes `HAMLAB_URL` through into `update.url`. |
|
||||
| 1.2 | **B** | `install.sh:4, 27` `HAMLAB_URL` default; `README.md:23-25, 31` | The curl one-liner and the install source are this Gitea. | Default comes from the release that served the script (`@URL@` filled by `make release`), or is required. |
|
||||
| 1.3 | **B** | `scripts/gitea-release.sh:14` `REPO` default | Publishing goes to `mitchell/hamlab` on this Gitea. | Require `HAMLAB_REPO`; no default. |
|
||||
| 1.4 | **C** | `README.md:100` "On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager" | Lab deployment fact in the product README. | Move to `hamlet/docs/hosts.md`. |
|
||||
| 1.5 | **C** | `examples/external-demo/index.html:8` `base: 'http://10.0.0.234:8100'` | Demo config names the hub's LAN IP. | `http://<hamlab-host>:8100`. |
|
||||
| 1.6 | **C** | `SENSORS.md:149` Prometheus target `10.0.0.234:8100`; `:187` `&machine=grid`; `:196-208` Glance examples at `hamlab.bigham.dev`, `zfs.tank.used_pct`, `/api/nodes/grid/...`; `:218-220` `grid,wyse`, `{"core":…,"grid":…,"mac":…}` | Docs use real host names, pool and domain. | Generic placeholders (`<host>`, `<machine>`, `<pool>`). |
|
||||
| 1.7 | **C** | `PLAN.md:3-4` "bare-metal Debian 13 host. First target: `core` (10.0.0.234)"; decisions table rows (Listen `10.0.0.234:8100`, Data `/home/core/apps/hamlab/`, Ingress NPM/`hamlab.bigham.dev`, Repo) | Product plan is written as this lab's plan. | Keep as history, or split lab facts out to `hamlet`. The repo is public. |
|
||||
| 1.8 | **C** | `VIRTUALIZATION.md:339-353` per-machine probe table; `:412, 425-436` `~/apps/hamlab`, `/tank/...`, boot NVMe free space | Plan is written against the five lab hosts. | Same as 1.7. |
|
||||
| 1.9 | **C** | `BUGS.md` throughout | Bug bashes narrate `core`, `grid`, `wyse`, `hamtv`, `gate`, the Mac. | History; fine to leave, but it is public. |
|
||||
| 1.10 | **C** | `cmd/hamlab/main.go:1` "Proxmox-style web console for a bare-metal Debian host"; `internal/hardware/hardware.go:5` "what Debian and its kin already have"; `internal/hardware/hardware.go:29` `// "GMKtec NucBox G3"`; `internal/metrics/metrics.go:1358` "did, on grid, 2026-10-02"; `internal/api/server.go:1382-1383` `["grid", …]`, `?machines=grid,wyse`; `internal/api/stacks.go:10` `/home/core/apps/dockhand/stacks/…`; `internal/api/dockersched.go:5` "unbound, nginx and pihole"; `internal/api/zfs.go:5` "grid already runs zfSnap"; `internal/storage/snapshots.go:6` "grid has run zfSnap v1.11.1"; `internal/api/guests.go:8` "Mitchell, 2026-09-27"; `internal/network/network.go:306` "the public address here is a DHCP lease"; `internal/metrics/gpu.go:17` "Asahi, today"; `internal/metrics/pciids.go:4-6` "the router's GPU"; `web/src/Top.tsx:522, 576`; `web/src/hashsel.ts:1-9`; `web/src/Embed.tsx:3` `&machine=grid` | Comments explain decisions by naming lab hosts. | Harmless, but rewrite the ones in package doc comments so a reader of the source doesn't need the lab to understand the code. |
|
||||
|
||||
## 2. UI placeholders and defaults that are this lab's
|
||||
|
||||
| # | Sev | Where | Text | General form |
|
||||
|---|---|---|---|---|
|
||||
| 2.1 | **C** | `web/src/Machines.tsx:116-118` | `placeholder="grid"`, `"https://10.0.0.214:8100"`, `"core"` | `nas`, `https://192.0.2.10:8100`, `admin` or the hub's own user. |
|
||||
| 2.2 | **C** | `web/src/KVM.tsx:166` | `placeholder="10.0.0.217"` (the NanoKVM on `grid`) | `192.0.2.20` or `kvm.local`. |
|
||||
| 2.3 | **C** | `web/src/Settings.tsx:242` | `placeholder="cpu.*, temp.*, zfs.tank.*"` | `cpu.*, temp.*, fs.*`. |
|
||||
| 2.4 | **C** | `web/src/Settings.tsx:225` | read-only token default name `'glance'` | Empty, or `dashboard`. |
|
||||
| 2.5 | **C** | `web/src/Alerts.tsx:38, 40` | `"Tank nearly full"`, `"zfs.tank.used_pct"` | `"Disk nearly full"`, `"fs.root.used_pct"`. |
|
||||
| 2.6 | **C** | `web/src/StackNew.tsx:110, 113, 114` | `/home/core/apps/whoami:/data`, `TZ=America/Los_Angeles`, `com.example.owner=mitchell` | `./data:/data`, `TZ=UTC`, `com.example.owner=you`. |
|
||||
| 2.7 | **C** | `web/src/Gallery.tsx:69`, `web/src/WidgetPicker.tsx:36` | `"find — gitea, temp, enp3s0…"`, `"filter sensors — cpu, temp, fan, enp3s0…"` | Drop the container and NIC names. |
|
||||
| 2.8 | **C** | `web/src/VPN.tsx:105` | default DNS for a new device `1.1.1.1, 1.0.0.1` | Fine as a public default, but should offer the tunnel's own address (the home resolver) first. |
|
||||
| 2.9 | **B** | `web/src/ZFS.tsx:212` `BLANK` | new schedule defaults `at: '02:30'`, `keep_for: '6w'` — `grid`'s old zfSnap cron line | Neutral defaults (`03:00`, `4w`), or none. |
|
||||
| 2.10 | **A** | `web/src/ZFS.tsx:302` `adopt()` | `/\b(tank[\w/-]*)\b/` — the dataset of an adopted external job is found by looking for the word **`tank`** | Parse the dataset from the command as the last `zfs`-shaped argument, or match against the datasets the page already has. On any other pool name adopt silently picks the fallback. |
|
||||
|
||||
## 3. Topology inherited from the lab (hub `core`, Dockhand, `tcp://:2375`)
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 3.1 | **A** | `internal/api/stacks.go:483-509` `nodeDocker()` | For every node whose stacks aren't editable locally, the hub **probes `tcp://<node-host>:2375`** with `docker -H … version` and, if it answers, deploys to the node over an unauthenticated Docker socket. This is the Dockhand-era layout; on any other network an open 2375 is either absent or a security problem the console should not quietly use. | Remove the probe. The node's own Ham Lab deploys its stacks (the files live on the node, under its data dir). `PLAN.md` "To do" already names this decision. |
|
||||
| 3.2 | **A** | `internal/api/stacks.go:439-479` `hubStacks()`, `:1036-1074` `homesFor()`, `web/src/Docker.tsx` "kept on `<hub>`, deployed to this machine over `tcp://…`" | A hub can hold compose files for a node and run compose remotely; new stacks for a node may be created **on the hub**. Only true here because Dockhand on `core` deployed `grid` and the Mac. | Same as 3.1: one machine, one set of files, one Docker. Hub-held stacks become an import path (copy to the node) rather than a steady state. |
|
||||
| 3.3 | **B** | `internal/api/stacks.go:290-382` `homesOf()`; `:373-376` fallback `~/stacks` | Where new stacks go is **discovered** from where ≥2 stacks already are (Dockhand's `stacks/<machine>` habit), else `~/stacks`. Neither is Ham Lab's data directory, which the project's own rule says everything managed must live under. | Default home = `<DataDir>/stacks`. Discovered homes stay as *additional* places, never the default. |
|
||||
| 3.4 | **B** | `internal/api/stacks.go:100-203` remembered mounts, `internal/api/dockersched.go:772-795` `repoint()` | Machinery exists to find compose files **inside a removed orchestrator container's filesystem**. Generic in form, but it exists because of Dockhand and is exercised only here. | Keep (it is general), but it is not a reason to keep 3.1/3.2. |
|
||||
| 3.5 | **B** | `internal/api/stacks.go:801` | Inside a `Via` container the command is literally `docker compose`, even when `dockerTool()` is `podman`. | Use the detected tool. |
|
||||
| 3.6 | **B** | `internal/api/stacks.go:908` `docker buildx imagetools inspect` | Update checks need the **buildx** plugin. Docker CE on Debian has it; Podman and minimal Docker installs do not → every image reports "the registry didn't answer". | Fall back to `skopeo inspect` or a direct registry `HEAD` for the digest. |
|
||||
| 3.7 | **B** | `Makefile:445-448` `deploy` | The machine you build on **is** the hub: installs locally, then `push-nodes`. | Fine for a developer; document that `deploy` means "I am the hub". A plain `make install` target for a single box. |
|
||||
| 3.8 | **B** | `internal/nodes/nodes.go:316-323` | A hub pushes other architectures only if `<prefix>/hamlab-<arch>` siblings exist, which only `make deploy`/`install.sh`-from-dist put there. | A hub without siblings should fetch the right build from `update.url` instead of erroring. |
|
||||
|
||||
## 4. Init system: systemd assumed where the product promises procd and OpenRC too
|
||||
|
||||
The installer supports systemd, procd and OpenRC (`internal/setup/setup.go:217-262`). These paths do not.
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 4.1 | **A** | `internal/updates/updates.go:537-543` `Power()` | `systemctl reboot` / `systemctl poweroff`. On OpenWrt (`gate`) and OpenRC the Power dialog fails. | Try `systemctl`, else `reboot` / `poweroff` (busybox, OpenRC, sysvinit all have them). |
|
||||
| 4.2 | **A** | `internal/incus/incus.go:1017, 1044-1058, 1069, 1075` | The Incus install script writes `/etc/systemd/system/*` drop-ins and units, adds `SupplementaryGroups` to `hamlab.service`, restarts with `systemd-run`. On OpenRC/procd the daemon never joins `incus-admin` → the tab stays in "repair" forever; the Docker FORWARD fix is never persisted. `HamlabUnit = "hamlab.service"` (`:35`) is surfaced in the API (`:280`). | Branch on the detected init: OpenRC `rc-update`/`conf.d`, procd init script edits, or fall back to `usermod` + a daemon restart via the installer's own supervisor knowledge. Docker rules via a boot hook that is not a unit. |
|
||||
| 4.3 | **A** | `internal/services/services.go:163-170` | Services tab is systemd-only ("says so cleanly"). `gate` (procd) and any OpenRC/runit/s6 box have no Services tab. | `rc-service`/`rc-status` (OpenRC), `/etc/init.d/<x> status` + `ubus call service list` (procd), `sv` (runit) backends. Noted as "not done" in `PLAN.md:487-489`. |
|
||||
| 4.4 | **A** | `internal/cron/cron.go:353-373, 401, 442, 532-534, 669-670` | Jobs tab reads `/etc/crontab`, `/etc/cron.d`, `/var/spool/cron{,/crontabs}` (Debian/cronie), `/usr/sbin/anacron`, last-runs from **journald**, timers via `systemctl`; writes only `/etc/crontab` and `/etc/cron.d`. Busybox crond (OpenWrt, Alpine) keeps crontabs in `/etc/crontabs/<user>` and has no `cron.d` → the Jobs tab is empty on `gate` and schedules written by 4.5 never run. | Add `/etc/crontabs/*` as a source; last-runs from syslog/logread when there is no journal. |
|
||||
| 4.5 | **A** | `internal/storage/snapshots.go:402` `/etc/cron.d/hamlab-zfs`; `internal/api/dockersched.go:65` `/etc/cron.d/hamlab-docker` | ZFS snapshot and image-update schedules are cron.d files. No cron.d on busybox crond; no cron at all on a systemd-timer-only host (Fedora Server, CoreOS). | Detect: cron.d → cron.d; busybox → a line in `/etc/crontabs/<user>`; none but systemd → a timer unit. The daemon-side runner already exists (`hamlab zfs-snapshot`, `docker-update`). |
|
||||
| 4.6 | **B** | `internal/hardware/hardware.go:333` `systemd-detect-virt` | Virtualisation detection is systemd-only. | Fall back to `/sys/class/dmi/id/sys_vendor`, `/proc/cpuinfo` `hypervisor` flag, `/sys/devices/virtual/dmi`. |
|
||||
| 4.7 | **B** | `internal/api/server.go:1129`, `internal/updates/updates.go:187` | `/run/reboot-required` is Debian/Ubuntu-only. `updates.Status` falls back to a kernel check for other distros; `/api/me.reboot_needed` (the Machines badge, the header's amber power button) does not. | Share one function. |
|
||||
| 4.8 | **B** | `install.sh:154-157, 164, 375-378`; `internal/setup/setup.go:284` | The wizard's init detection knows systemd and OpenRC only: on OpenWrt it warns "no systemd or OpenRC: nothing starts it at boot" although `hamlab install` then installs a procd script fine; the "logs" hint and the "isn't answering yet" hint have no procd case. | Add procd to both. |
|
||||
| 4.9 | **B** | `internal/setup/setup.go:609-641` `firewall()` | Only firewalld and ufw are detected. OpenWrt fw4/nftables, plain nftables, iptables, Alpine awall are not. | Detect `fw4`/`nft`/`iptables`; on OpenWrt add a LAN-zone rule or say what to open. |
|
||||
| 4.10 | **B** | `internal/setup/setup.go:568-605` `quietSudoers()`; `internal/services/services.go:176-185` | The sudoers quieting list is tuned to what the lab polls (`smartctl`, `pwrstat`, `journalctl`, `zpool`, `zfs`). A host with NUT or apcupsd gets `upsc`/`apcaccess` lines in its auth log at 10 s intervals; a host without `/etc/sudoers.d` (OpenWrt) gets nothing, which is correct. | Build the list from what the collector actually probes. |
|
||||
|
||||
## 5. Distro and package assumptions beyond init
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 5.1 | **B** | `internal/incus/incus.go:213-250, 351-352` | Incus can be installed with apt, dnf, pacman, apk. **zypper is missing** (the Updates tab supports it), so an openSUSE host reads "No package manager Ham Lab knows how to drive". `opkg` is known only to say "not packaged for OpenWrt". | Add zypper; phrase the opkg case as "not packaged for this distribution". |
|
||||
| 5.2 | **B** | `internal/incus/incus.go:973-978` | dnf branch adds `qemu-system-x86-core`/`edk2-ovmf` or `qemu-system-aarch64-core`/`edk2-aarch64`; other arches (riscv64, ppc64le, which `IncusArch()` maps) get no QEMU. pacman and apk branches assume their package names never change. | Table per PM × arch, or let `incus` pull its own deps where the distro does that (Debian does). |
|
||||
| 5.3 | **B** | `internal/incus/incus.go:958, 970` `DEBIAN_FRONTEND`, `apt-get … -o Dpkg::Options::=--force-confold`; `internal/updates/updates.go:78, 432` | Debian-isms exported unconditionally. Harmless elsewhere. | Fine; note only. |
|
||||
| 5.4 | **B** | `scripts/pci-ids-gpu.sh:7, 41` default `/usr/share/misc/pci.ids` | Debian's path for `pci.ids`; Fedora/Arch keep it in `/usr/share/hwdata/`. Build-time only. | Try both. |
|
||||
| 5.5 | **B** | `internal/api/status_sensors.go:283`, `internal/api/docker.go:133` | Container memory/CPU read from three cgroup paths: cgroup v2 + systemd driver, cgroupfs, libpod. cgroup v1 hosts (older Debian/Ubuntu, some Alpine, OpenWrt) get no `container.*.mem/cpu`. | Add the v1 `memory/docker/<id>` and `cpu,cpuacct` paths, or fall back to `docker stats --no-stream` once in a while. |
|
||||
| 5.6 | **B** | `internal/metrics/metrics.go:1064` `clkTck = 100` | USER_HZ assumed 100. True on every mainstream build; wrong on a kernel built otherwise. | `sysconf` via `getconf CLK_TCK` once at start. |
|
||||
| 5.7 | **B** | `internal/incus/subnet.go:150-179` `ip -j`; `internal/space/manager.go:578-585` `timeout -s KILL` | iproute2 JSON output (busybox `ip` has no `-j`); GNU `timeout`. | Parse plain `ip route`/`ip addr`; busybox has `timeout` too but flag spelling differs — use the context deadline. |
|
||||
|
||||
## 6. Hardware and vendor assumptions
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 6.1 | **B** | `internal/metrics/gpu.go:343-383` `read()` | Busy/clock/VRAM are read for **i915, xe, amdgpu** only — the three GPUs in this lab. Nouveau, nvidia-drm, panfrost, v3d, Asahi are listed but never measured. `PLAN.md:317`: "No NVIDIA path: nothing here has one to test against." | `nvidia-smi --query-gpu` where present; `/sys/class/drm/*/device/gpu_busy_percent` for nouveau; `devfreq` for the ARM drivers. |
|
||||
| 6.2 | **B** | `internal/metrics/pciids_gpu.txt`, `scripts/pci-ids-gpu.sh:12-15` | Embedded PCI ID table trimmed to Intel, AMD, NVIDIA and three virtual vendors. Matrox, ASPEED (every server BMC), Moore Threads, etc. fall back to `lspci` or "Vendor GPU <id>". | Include the full display-class device list, not three vendors. |
|
||||
| 6.3 | **B** | `internal/metrics/gpu.go:237-241` | "Integrated" = PCI bus `0000:00:`. Wrong on multi-domain or APU boards where the iGPU hangs off a bridge; cosmetic. | Use `boot_vga` + `PCI_SUBSYS` heuristics, or drop the flag. |
|
||||
| 6.4 | **B** | `internal/metrics/metrics.go:189-196, 917-968` | UPS probe order **pwrstat (CyberPower) first**, then NUT, then apcupsd; pwrstat's `....`-separated output is parsed bespoke. A host with both pwrstat and NUT shows pwrstat only. | Prefer NUT (the vendor-neutral one), then the vendor tools; or list every UPS found. |
|
||||
| 6.5 | **B** | `internal/metrics/metrics.go:1179-1182` `cpuTemp()` | CPU temperature from `coretemp`, `k10temp`, `thermal`/`cpu`, `cpu_thermal`, `soc_thermal`. Misses `zenpower`, `acpitz` as the only sensor, Apple `macsmc` names, many ARM SoC zone names. | Broaden; fall back to the hottest zone when none matches. |
|
||||
| 6.6 | **B** | `internal/hardware/hardware.go:617-627` `jedec` | JEDEC IDs for "the memory makers who turn up in a homelab" — 14 entries. | Embed the JEDEC JEP106 bank list (public, small). |
|
||||
| 6.7 | **B** | `internal/storage/storage.go`, `internal/api/zfs.go`, `internal/storage/arc.go`, `web/src/ZFS.tsx` | A whole **ZFS** tab and ZFS-aware Space/Disks logic; **no btrfs or LVM equivalent** (snapshots, subvolumes, scrub, balance). Not wrong, but the tab list reflects a ZFS NAS. | Later: a Btrfs tab with the same shape, or a generic "Filesystems" tab that shows what the host has. |
|
||||
| 6.8 | **B** | `internal/storage/snapshots.go:1-16, 966-968` | Snapshot naming is **zfSnap's** (`2026-09-24_02.30.01--6w`) because `grid` ran zfSnap. A sanoid/zfs-auto-snapshot user's existing snapshots (`autosnap_…`, `zfs-auto-snap_daily-…`) are detected (`internal/api/zfs.go:1764-1798`) but can never be expired by Ham Lab, and new schedules write a third convention beside them. | Make the naming a per-schedule choice (zfSnap / sanoid / zfs-auto-snapshot), or at least document the convention as the product's rather than inherited. |
|
||||
| 6.9 | **B** | `internal/network/speedtest.go:64-67` `downSources` | Fallback download source is **`speedtest.seattle.linode.com`** — chosen because the lab is in the Pacific Northwest. Everywhere else it reads low. | Pick the nearest of several mirrors, or drop the fallback and report the refusal. |
|
||||
| 6.10 | **B** | `internal/network/speedtest.go:70-74, 604`, `internal/network/network.go:323-334` | Public IP, edge and speed come from **Cloudflare** endpoints, polled every 15 min with no off switch. Air-gapped or privacy-minded installs have no way to disable the outbound probe. | A `network.outside: off` setting; honour it in the watcher. |
|
||||
| 6.11 | **B** | `internal/vpn/edit.go:5-7, 419-421` | Peers can be **created/edited only through OpenWrt UCI** "because that is where the server here is configured". A wg-quick server (the common case everywhere else) is read-only. | Write wg-quick `.conf` too (append a `[Peer]` block + `wg set`), guarded the same way. |
|
||||
| 6.12 | **B** | `internal/vpn/vpn.go:251-253` | Peer names come from wg-quick comments or UCI descriptions — "which between them cover every machine here". NetworkManager and systemd-networkd WireGuard configs are not read. | Add `.nmconnection` and `.netdev` readers. |
|
||||
| 6.13 | **B** | `internal/vpn/edit.go:515` `profile()` | Split-tunnel routes hard-code `10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12` on top of the tunnel's own prefix. | Offer the host's actual LAN prefixes (from `TakenPrefixes`-style route reading), with RFC 1918 as the fallback. |
|
||||
| 6.14 | **B** | `internal/incus/subnet.go:1-9, 23-34, 55`; `web/src/Virtualization.tsx:133` | Bridge subnet candidates skip **all of 10/8** because "this LAN is 10.0.0.0/8"; order is 172.16.100–254, then 192.168.254–200, then 172.16.1–99; last-resort `172.16.100.0/24`. Works anywhere, but a lab on 172.16/12 gets 192.168.x first and a lab on 192.168/16 never gets its own range. | Read the host's routes (already done) and pick from all three RFC 1918 blocks in a neutral order. |
|
||||
| 6.15 | **B** | `internal/network/network.go:170-199, 303-314`; `internal/incus/subnet.go`; `internal/vpn/edit.go:558-595`; `internal/tlsauto/tlsauto.go:533-537`; `install.sh:114-120`; `internal/setup/setup.go:648-666` | **IPv4 everywhere**: default route from `/proc/net/route` only, outside view v4 only, bridge/peer allocation v4 only, cert SANs v4 only, installer lists v4 addresses only. | `/proc/net/ipv6_route`, v6 SANs, v6 peer addresses. An IPv6-only host today has no gateway, no cert SAN and no bridge. |
|
||||
|
||||
## 7. Privilege and deployment posture (the lab's NOPASSWD-everywhere model)
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 7.1 | **A** | `internal/priv/priv.go`; every tab that calls `priv.Available()`; `web/src/Updates.tsx:2144` "give its user passwordless sudo or doas"; `web/src/Jobs.tsx:1883` | Full function needs the service user to have **unrestricted passwordless sudo**, which `hamlet/CLAUDE.md` grants on every host here. The installer neither creates nor offers a sudoers rule, so on a fresh box most tabs are read-only and the UI tells the user to go fix sudoers by hand. | `hamlab install --sudo` writes a **scoped** `/etc/sudoers.d/hamlab` naming the commands Ham Lab runs (smartctl, zpool, zfs, systemctl, docker, journalctl, tee for its cron files, `hamlab fs|space-walk|drm-clients|replace`), or installs as root. Either way the console works out of the box. |
|
||||
| 7.2 | **A** | `internal/api/server.go:1069-1080` `clientKey()`, `:1083-1092`, `:431-433` `secure()` | The **first `X-Forwarded-For` hop is trusted unconditionally**, because here Ham Lab always sits behind Nginx Proxy Manager. On a host exposed directly, a client sets the header and defeats the per-client login rate limiter and the session IP check. | Trust `X-Forwarded-*` only from configured proxy addresses (`trusted_proxies` in config), else use the socket peer. |
|
||||
| 7.3 | **B** | `internal/api/server.go:1324` `X-Accel-Buffering: no` "nginx (NPM)" | Harmless header; comment names the lab's proxy. | Keep header; generalise comment. |
|
||||
| 7.4 | **B** | `internal/api/guests.go:1-9`, `PLAN.md:21` | Guest logins are full-power ("he gives these to people he trusts"). A product default. | Keep, but it is a posture decision taken for this household; a read-only guest is the generic expectation and was cut. Document as a deliberate choice in README. |
|
||||
| 7.5 | **B** | `internal/api/users.go:1-6` | "a household console, not a tenancy": any signed-in user can add users, make tokens, open a root shell. | Same as 7.4 — fine, but say so where a stranger installs it. |
|
||||
|
||||
## 8. Resource tuning from the lab's smallest host
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 8.1 | **B** | `cmd/hamlab/main.go:451` `debug.SetMemoryLimit(160 << 20)`; `PLAN.md:11` "`core` has ~370 MiB free RAM" | A fixed 160 MiB soft heap limit sized for `core`. On a 64 GiB NAS it forces needless GC under a big Space scan; on a 512 MiB Pi it is still half the box. | Scale from `MemTotal` (e.g. 2–5 %, floor 64 MiB, ceiling 1 GiB), or make it a config key. |
|
||||
| 8.2 | **B** | `internal/space/walk.go:893` `workers := 32`; `internal/space/manager.go:578` 3-minute `zfs diff` patience | Tuned on `grid` (16 cores) and `core`. | Workers from `GOMAXPROCS`; patience configurable. |
|
||||
| 8.3 | **C** | `internal/metrics/metrics.go:32-43` intervals, `History = 300` | Reasonable generic defaults; listed only because 8.1 made them look lab-tuned. | None. |
|
||||
|
||||
## 9. Data directory and legacy paths
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 9.1 | **B** | `internal/config/config.go:115-117` `Candidates()`; `internal/setup/setup.go:413-422`; `README.md:38` | `~/apps/hamlab/config.json` is searched as a legacy location — the lab's pre-v0.2 path, kept so `core`, `grid` and the Mac keep working. | Keep for one more release with a one-time migration log line, then drop. A stranger never had it. |
|
||||
| 9.2 | **B** | 3.3 above; `PLAN.md:24` "`grid`'s and the Mac's stacks at `~/stacks/<machine>/` are the standing exception" | The "everything under DataDir" rule has a lab-shaped exception baked into the stacks code. | Default to DataDir; treat `~/stacks` as a discovered home like any other. |
|
||||
| 9.3 | **C** | `internal/setup/setup.go:351` `ownFiles` | Correctly scoped; note only. | None. |
|
||||
|
||||
## 10. Build and release
|
||||
|
||||
| # | Sev | Where | Assumes | General form |
|
||||
|---|---|---|---|---|
|
||||
| 10.1 | **B** | `Makefile:60-61, 442-443`; `scripts/gitea-release.sh` | Publishing is Gitea/Forgejo-only (release API shape). `selfupdate` also understands a static directory, so publishing could too. | `make release` + any static host is already enough; add a GitHub/Forgejo-agnostic `release-dir` note. |
|
||||
| 10.2 | **C** | `scripts/gitea-release.sh:500-515` | Needs `python3` on the publishing host. | `jq`, or do it in Go (`hamlab publish`). |
|
||||
| 10.3 | **C** | `Makefile:392` `ARCHES ?= amd64 arm64 arm 386 riscv64` | Fine. `ppc64le`/`s390x` are mapped by `IncusArch()` but never built. | Add or drop them from `IncusArch`. |
|
||||
|
||||
## 11. Things that are already general (checked, no change needed)
|
||||
|
||||
Listed so the sweep is known to be complete, not because they need work.
|
||||
|
||||
- `internal/priv` (root / sudo / doas), `internal/setup` service templates, `IP_FREEBIND` listen, self-update atomic replace and re-exec.
|
||||
- `internal/updates`: apt, dnf, zypper, pacman, apk with refresh/upgrade/install.
|
||||
- `internal/storage` block devices: `lsblk -J` with a `/sys/block` fallback, eMMC health, NVMe namespace folding, SMART only when `smartctl` exists.
|
||||
- `internal/metrics` CPU/mem/hwmon/thermal/net/disk/fs from `/proc` and `/sys`; remote-FS and pseudo-FS lists; filesystem statfs with a timeout.
|
||||
- `internal/hardware`: every reader may come back empty with a note; device-tree fallbacks for non-SMBIOS boards.
|
||||
- `internal/incus` detection and the per-machine "ready / repair / installable / unavailable" model (modulo 4.2, 5.1, 5.2).
|
||||
- `internal/nodes`, `internal/tlsauto`: hub ↔ node pairing, pinned certs, proxy, push.
|
||||
- `internal/sensors`, alerts, widgets, embeds, Prometheus: machine-agnostic by design.
|
||||
- `internal/files`, `internal/term`, `internal/tasks`, `internal/auth`, `internal/store`.
|
||||
- `internal/api/kvm.go`: "nothing in it knows anything about any particular KVM" — verified.
|
||||
- `internal/compose` (modulo 3.5).
|
||||
- The React UI: machine-switcher model, hash routing, per-account prefs, themes.
|
||||
|
||||
---
|
||||
|
||||
## Suggested order of work
|
||||
|
||||
1. **Posture and safety first (A):** 7.2 trusted proxies; 7.1 installer-written scoped sudoers; 3.1 drop the `:2375` probe (with 3.2 as its consequence).
|
||||
2. **"Any Linux" promises the README already makes (A):** 4.1 power, 4.4/4.5 cron on busybox and timer-only hosts, 4.2 Incus install on non-systemd, 4.3 a Services backend for OpenRC/procd, 2.10 the `tank` regex.
|
||||
3. **Lab defaults that phone home or mislead (B):** 1.1–1.3 update/install URLs, 2.9 schedule defaults, 6.9/6.10 speed-test sources and an off switch, 8.1 memory limit, 3.3/9.2 stack home = DataDir.
|
||||
4. **Breadth (B):** 5.1 zypper for Incus, 5.5 cgroup v1, 6.1/6.2 GPUs beyond Intel/AMD, 6.4 UPS order, 6.11/6.12 wg-quick/NetworkManager, 6.15 IPv6.
|
||||
5. **Cosmetics (C):** every placeholder and comment in §1.10 and §2, then split lab facts out of `PLAN.md`, `VIRTUALIZATION.md`, `SENSORS.md`, `README.md`.
|
||||
@@ -1,10 +1,101 @@
|
||||
# Ham Lab
|
||||
|
||||
Web-based Linux management console for bare-metal Debian, in the style of the Proxmox
|
||||
Web-based Linux management console for bare-metal Linux, in the style of the Proxmox
|
||||
web UI. Named after Ham the cat (see also HamTV).
|
||||
|
||||
Status: **planning**. See [`PLAN.md`](PLAN.md) for the MVP scope and roadmap. No code yet.
|
||||
Status: **v0.1** — persistent web terminals and a live Dashboard tab. Scope and roadmap in
|
||||
[`PLAN.md`](PLAN.md).
|
||||
|
||||
- Runs as a systemd service on the host it manages, as user `core`.
|
||||
- Public at `https://hamlab.bigham.dev` (behind Nginx Proxy Manager on `core`).
|
||||
## Runs on any Linux box
|
||||
|
||||
One static Go binary with the UI embedded: no cgo, no libc, no Node at runtime, no agents.
|
||||
The same file runs on Debian, Ubuntu, Fedora, RHEL/Rocky/Alma, openSUSE, Arch and Alpine
|
||||
(musl), on bare metal, VMs and containers. Builds: `amd64`, `arm64`, `arm` (ARMv6
|
||||
hard-float and up, so every Raspberry Pi), `386`, `riscv64`.
|
||||
|
||||
```sh
|
||||
sudo ./hamlab-linux-amd64 install # that's the whole install
|
||||
```
|
||||
|
||||
or, from wherever `make release` output is served:
|
||||
|
||||
```sh
|
||||
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh # asks: user, address, HTTPS, firewall, password
|
||||
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh -s -- --yes # asks nothing, all defaults
|
||||
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh -s -- --listen 0.0.0.0:8100 --tls auto # flags go to `hamlab install`
|
||||
```
|
||||
|
||||
`install.sh` is a wizard when there is a terminal (it reads answers from `/dev/tty`, so it
|
||||
works piped from curl) and silent otherwise. Either way it picks the build for `uname -m`,
|
||||
checks it against `SHA256SUMS` and runs `hamlab install`; re-running it upgrades in place.
|
||||
`latest` is an alias Gitea resolves to the newest release, so that URL never changes.
|
||||
`make gitea-release` publishes a release there (`HAMLAB_URL` points the installer elsewhere); run the script beside the `hamlab-linux-*` files and it installs offline.
|
||||
`hamlab install`:
|
||||
|
||||
- copies the binary to `/usr/local/bin` (`--prefix`), atomically;
|
||||
- gets root through `sudo` or `doas` if you aren't root already, and runs the console as
|
||||
**you** (`--user`), so terminals are your login shell with your rights;
|
||||
- writes the config to `~/.config/hamlab/` (`--data`; an existing `~/apps/hamlab/` is kept)
|
||||
and asks for the first account's password (the account is named after the user the
|
||||
console runs as), or generates one and prints it once (`--password-stdin` for scripts);
|
||||
- registers a boot-time service with **systemd** or **OpenRC**, naming only groups that
|
||||
exist on the host, and waits for `/healthz`; with neither init it prints the one line
|
||||
to put under your own supervisor;
|
||||
- says so when firewalld or ufw will block the port (`--open-firewall` opens it).
|
||||
|
||||
Re-running it upgrades in place and keeps config, user and accounts. `hamlab uninstall
|
||||
[--purge]` reverses it.
|
||||
|
||||
Accounts: every person has their own user name and password, and the theme, accent
|
||||
colours and dashboard layout follow the account. The **Users** tab has it all: your own
|
||||
password, where you are signed in (sign out any device), and the accounts (add, remove,
|
||||
reset a password; your own password confirms the last two). From the shell:
|
||||
`hamlab passwd [user]` sets a password and makes the account if it is new, `hamlab
|
||||
users` lists them, `hamlab user-rm <user>` removes one. Machine settings — refresh rate,
|
||||
graph history, alert rules — stay the machine's and are shared by everyone.
|
||||
|
||||
**Guest logins**, for showing somebody the console: Users → Guest logins makes a
|
||||
throwaway user name and password, good for half an hour by default (up to 24 hours), and
|
||||
hands you the pair to paste into a message. A guest is an ordinary account with a clock on
|
||||
it — the whole console, every machine, the terminal included — that stops working when the
|
||||
time is up, wherever it is signed in, or the moment you revoke it. There is no reduced
|
||||
guest: give one to somebody you would hand the password to, and revoke it if you change
|
||||
your mind.
|
||||
|
||||
What adapts to the host, rather than assuming Debian:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Root | already root, else passwordless `sudo`, else passwordless `doas`; without any, the console is read-only where root is needed and says so |
|
||||
| Updates tab | apt, dnf, zypper, pacman, apk: list, refresh, upgrade, install (pacman: all or nothing) |
|
||||
| Services / timers | systemd of any age (JSON output where it exists, the plain table before v246); a clear message on hosts without systemd |
|
||||
| Disks | `lsblk`, or `/sys/block` when it's missing (busybox) or too old; SMART and ZFS only when their tools are there |
|
||||
| Network | `/proc` and `/sys` for the interfaces, addresses, link speed and default route. The speed test needs no tool installed: it measures against Cloudflare's public speed endpoints, so it works the same on Debian and on the OpenWrt router |
|
||||
| VPN | WireGuard tunnels and their peers, from `wg show` where wireguard-tools is installed; peer names from wg-quick `.conf` comments or OpenWrt UCI descriptions. Read-only, and it never reads a private or pre-shared key |
|
||||
| Dashboard | `/proc` and `/sys` only: hwmon or thermal-zone temps, fans, NICs, disks, mounts; empty sections where the hardware has nothing to report. UPS from whichever of `pwrstat`, NUT (`upsc`) or `apcupsd` is installed |
|
||||
| Listen address | bound with `IP_FREEBIND`, so a LAN address that isn't up yet at boot doesn't fail the start |
|
||||
| Self-update | the new build must run and load this host's config before it replaces the old one; atomic rename; the daemon then re-execs itself, so it restarts the same way under systemd, OpenRC or nothing; `hamlab.prev` kept |
|
||||
|
||||
Building: `make build` (web + Go), `make release` (every architecture, `install.sh` and
|
||||
`SHA256SUMS` in `dist/`), `make test`.
|
||||
|
||||
Terminals run as the user the service runs as, with that user's login shell.
|
||||
|
||||
## Managing several machines
|
||||
|
||||
Install Ham Lab on each host; on the ones the hub should reach:
|
||||
`hamlab install --listen <lan-ip>:8100 --tls auto` (a self-signed certificate is generated
|
||||
on first start). Then on the hub, open **Machines**, enter a name, the address
|
||||
(`https://host:8100`) and a user and password on that machine once (the user may be
|
||||
left out when it has only one). The hub pins the
|
||||
certificate, keeps an access token, and every tab works against the chosen machine
|
||||
through the switcher at the top of the rail.
|
||||
|
||||
Updating: `make deploy` on the hub installs the new build there and then pushes it to
|
||||
every managed machine (`hamlab push-nodes`), which install it and restart. The Machines
|
||||
page flags machines running an older version and offers the same push per machine.
|
||||
Mixed architectures are fine: `make deploy` also cross-compiles every other architecture,
|
||||
and the hub sends each machine the build for its own.
|
||||
|
||||
- On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager.
|
||||
- Installable as a PWA.
|
||||
|
||||
+279
@@ -0,0 +1,279 @@
|
||||
# Sensors — the standard read interface
|
||||
|
||||
Everything Ham Lab measures is a **sensor**: a stable dotted ID, a little metadata, a
|
||||
current value. Any consumer — Ham Lab's own dashboard, Glance, Home Assistant, a shell
|
||||
script — discovers them from the catalog and asks for exactly the ones it wants.
|
||||
|
||||
Code: `internal/sensors` (types, registry, selectors, shaped query) and
|
||||
`internal/metrics/sensors.go` (the collector as a provider). The registry measures
|
||||
nothing; providers do. A new source of sensors implements `sensors.Provider` and is
|
||||
added to the registry in `api.New`.
|
||||
|
||||
## IDs
|
||||
|
||||
```
|
||||
cpu.usage cpu.core.<n>.usage cpu.temp load.1|5|15 uptime
|
||||
mem.total|used|avail|free|cached|buffers|used_pct swap.total|used|avail|used_pct
|
||||
temp.<chip>.<label> fan.<chip>.<label>
|
||||
net.<iface>.rx|tx|rx_total|tx_total|up
|
||||
net.internet.download|upload|latency|jitter|tested|ip|edge
|
||||
disk.<dev>.read|write|util|size|temp
|
||||
fs.<mount>.size|used|used_pct
|
||||
ups.battery|runtime|load|load_pct|in_v|out_v|on_battery|state|source
|
||||
gpu.<name>.usage|render|video|video-enhance|copy|compute|temp|power|fan|freq|mem_used|mem_total|clients
|
||||
zfs.arc.size|target|min|max|mru|mfu|data|metadata|hit_pct|reads|misses|lifetime_hit_pct|no_grow|compress_ratio|l2_size|l2_hit_pct
|
||||
```
|
||||
|
||||
- Segments are lower case `[a-z0-9_-]`: `Package id 0` → `package_id_0`, `/mnt/tank` →
|
||||
`mnt_tank`, `/` → `root`. The real names are in the sensor's `labels`.
|
||||
- Built from names the hardware reports, never enumeration order (`hwmon3` moves between
|
||||
boots). Two identical chips collide; the second gets `_2`.
|
||||
- Which sensors exist differs per machine. That is the point of the catalog.
|
||||
- `net.internet.*` is the Network tab's speed test: `download` and `upload` in Mbps,
|
||||
`latency` and `jitter` in ms, `tested` the age of the reading in seconds, and `ip` and
|
||||
`edge` the public address and the Cloudflare edge as strings. They appear only once a test
|
||||
has run on that machine, and each one's `t` is when it ran — so a widget or an alert on
|
||||
them is reading a measurement from that moment, not a live value. Nothing takes a speed
|
||||
test on its own; something has to press the button.
|
||||
- `gpu.<name>.*` is each graphics device, named as lspci names it (`gpu.intel_arc_a380`;
|
||||
labels carry `driver` and the PCI `bus`). `usage` is how busy the device was — from the
|
||||
GT's idle residency on Intel, `gpu_busy_percent` on amdgpu — and is absent on a driver
|
||||
with no such counter. The per-engine percentages (`render`, `video`…) are the sum over the
|
||||
processes holding the device open, read from DRM fdinfo, and exist only while something
|
||||
has it open. `temp` is the card's own sensor, or on an integrated GPU the CPU package it
|
||||
shares a die with (label `source` says which); an Arc on i915 has none and reports none.
|
||||
`freq` is the actual clock, 0 when power-gated. `clients` counts the processes using it.
|
||||
- `zfs.arc.*` is the ZFS read cache in RAM, on a machine with the module loaded. `hit_pct`
|
||||
is over the last sampling interval and appears once an interval has had any reads (an idle
|
||||
interval repeats the last value); `lifetime_hit_pct` is since boot. `no_grow` is true
|
||||
while memory pressure stops the cache growing. `l2_*` appear only with cache devices.
|
||||
|
||||
## A sensor
|
||||
|
||||
```json
|
||||
{ "id": "temp.coretemp.package_id_0", "name": "coretemp Package id 0",
|
||||
"kind": "temperature", "unit": "°C", "value": 51, "t": 1789826600878,
|
||||
"warn": 105, "crit": 105, "labels": { "chip": "coretemp", "label": "Package id 0" } }
|
||||
```
|
||||
|
||||
`value` is a number, bool or string. `t` is when it was really read, in unix ms — slow
|
||||
sections (filesystems 30 s, UPS 10 s) are older than the tick. `min`/`max` are the scale
|
||||
(0–100 for percentages, 0–total for used bytes); `warn`/`crit` are the hardware's own
|
||||
thresholds where it states them.
|
||||
|
||||
## Endpoints
|
||||
|
||||
All behind the same auth as the rest of `/api` (session cookie or `Authorization: Bearer`).
|
||||
On a hub, every one is also at `/api/nodes/<machine>/sensors…` for each managed machine.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `GET /api/sensors` | Catalog, with current values. `?ids=fan.*,cpu.usage` narrows it. |
|
||||
| `GET /api/sensors/{id}` | One sensor. `?format=text` → the bare value as text/plain. |
|
||||
| `GET /api/sensors/{id}/history` | `{id, unit, t[], v[]}` — what the ring holds (300 points, spaced by the window below). Numbers only. |
|
||||
| `GET /api/sensors/history?ids=cpu.usage,net.*.rx` | Many histories in one request: `{series: [{id, unit, t[], v[]}]}`. A sensor younger than the ring has a shorter series. |
|
||||
| `GET /api/prefs/history` | How far back the history reaches: `{seconds, points, stride_sec, min_seconds, max_seconds}`. |
|
||||
| `PUT /api/prefs/history` | `{seconds}` — 60 to 86400. Set on the hub it is passed to every machine, so the fleet's graphs reach equally far back. |
|
||||
| `POST /api/sensors/query` | Shaped query, below. |
|
||||
| `GET /api/sensors/query?k=selector&…` | The same, flat, for consumers that can only fetch a URL. |
|
||||
|
||||
A selector is an exact ID or a glob. `*` crosses dots: `fan.*` is every fan,
|
||||
`cpu.core.*.usage` every core.
|
||||
|
||||
## Shaped query
|
||||
|
||||
Send the object you want, with selectors where the values go. Get it back filled in.
|
||||
|
||||
```
|
||||
POST /api/sensors/query
|
||||
{ "cpu": "cpu.usage", "fans": "fan.*", "box": { "temp": "cpu.temp", "up": "uptime" } }
|
||||
|
||||
→ { "cpu": 12.4,
|
||||
"fans": { "fan.nct6775.fan1": 820, "fan.nct6775.fan2": 1140 },
|
||||
"box": { "temp": 41, "up": 80520 } }
|
||||
```
|
||||
|
||||
- Exact ID → its value; `null` if this machine has no such sensor (so one query works
|
||||
across machines that differ).
|
||||
- Glob → an object keyed by ID; `{}` if nothing matches.
|
||||
- Objects and arrays are walked; anything else in a query is a 400.
|
||||
- `?full=1` → every value is the whole sensor object instead of the bare value. (In the
|
||||
GET form `full` is therefore not usable as a key.)
|
||||
|
||||
## Live stream
|
||||
|
||||
`GET /api/sensors/stream` is a shaped query that keeps answering: the same object, pushed
|
||||
as a server-sent event after every sample. The query is `?q=<URL-encoded JSON shape>` or
|
||||
the flat form; `?full=1` as above; `?every=<ms>` spaces events out for a consumer that
|
||||
doesn't want them at the sampling rate. A keepalive comment goes out every 25 s, and
|
||||
`X-Accel-Buffering: no` keeps nginx from holding events back.
|
||||
|
||||
```js
|
||||
new EventSource('/api/sensors/stream?token=…&q=' + encodeURIComponent(JSON.stringify({ cpu: 'cpu.usage', fans: 'fan.*' })))
|
||||
.onmessage = (e) => render(JSON.parse(e.data))
|
||||
```
|
||||
|
||||
## Auth: read-only tokens
|
||||
|
||||
Settings → Access tokens → **+ Read-only sensor token** (or `POST /api/tokens/read
|
||||
{"name"}` with a session). Scope `sensors:read`: it opens every endpoint in this file,
|
||||
here and on each managed machine through the hub (`/api/nodes/<machine>/sensors…`; the
|
||||
hub talks to the node as itself, the caller's token stops at the hub), and nothing else.
|
||||
|
||||
- Send it as `Authorization: Bearer <token>`, or as `?token=<token>` where headers are
|
||||
impossible (an iframe, a browser `EventSource`).
|
||||
- A read-only token can be **limited to ID patterns** (`{"name": "glance", "sensors":
|
||||
["cpu.*", "zfs.tank.*"]}`, or the second prompt in Settings). Everything outside them
|
||||
does not exist for that token — catalog, query, history, stream, Prometheus, and on
|
||||
managed machines through the hub, which passes the restriction along.
|
||||
- A **full-control token is never accepted from a URL**, and a read-only token is never
|
||||
accepted anywhere outside this file's endpoints.
|
||||
|
||||
## Prometheus
|
||||
|
||||
`GET /metrics` (or `/api/sensors/prometheus`, which also works through the hub as
|
||||
`/api/nodes/<machine>/sensors/prometheus`). One gauge family per kind of measurement,
|
||||
base units, the sensor ID and its real-world names as labels:
|
||||
|
||||
```
|
||||
hamlab_temperature_celsius{id="temp.coretemp.core_0",chip="coretemp",label="Core 0"} 50
|
||||
hamlab_usage_percent{id="fs.root.used_pct",fstype="ext4",mount="/"} 10.1
|
||||
hamlab_state{id="ups.state",model="CPS CP1500PFCRM2U",value="Normal"} 1
|
||||
hamlab_updates_pending{id="updates.pending",manager="apt"} 3
|
||||
```
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: hamlab
|
||||
authorization: { credentials: <read-only token> }
|
||||
static_configs: [{ targets: ['10.0.0.234:8100'] }]
|
||||
```
|
||||
|
||||
## Status sensors
|
||||
|
||||
Not everything is a /proc number. These are read in the background on their own clocks
|
||||
(each sensor's `t` is its real age) by `internal/api/status_sensors.go`:
|
||||
|
||||
```
|
||||
services.failed|failed_names|running every 30 s
|
||||
service.<unit>.active|state one pair per service that is enabled, up or failed
|
||||
containers.total|running|unhealthy every 15 s (Docker or Podman, if present)
|
||||
container.<name>.running|state|health|mem|cpu mem and cpu read from the container's cgroup
|
||||
container.<name>.down should be up and isn't: Docker couldn't start it, it is
|
||||
stuck restarting, or it died non-zero with no policy to
|
||||
restart it — *not* a container someone stopped
|
||||
updates.pending|security|reboot_required every 5 min
|
||||
zfs.<pool>.health|healthy|size|used|used_pct|frag every 1 min
|
||||
smart.<dev>.passed|warnings|power_on|wear every 10 min, never waking a sleeping disk
|
||||
```
|
||||
|
||||
## Widgets
|
||||
|
||||
A widget is a spec — `{type, sensor, title?}` — over this API and nothing else
|
||||
(`web/src/widgets.tsx`, fed by `useSensorStream` in `web/src/sensors.ts`). Types: `stat`,
|
||||
`gauge`, `bar`, `spark`, and `list` for a pattern. Scale and colour come from the
|
||||
sensor's own `min`/`max`/`warn`/`crit`.
|
||||
|
||||
- **In Ham Lab:** Dashboard → Customize → **+ Add widget**. Saved with the layout,
|
||||
per machine; all of a page's widgets share one stream.
|
||||
- **Widgets tab:** every widget type, then every sensor on the machine drawn once, live,
|
||||
grouped by family (`web/src/Gallery.tsx`). Tap a tile to copy its sensor ID — the
|
||||
quickest way to find what to put in a widget, a query or a Glance tile.
|
||||
- **Anywhere else:** `/embed/widget` is one widget alone on a page, live, in its own small
|
||||
bundle (the console never loads), and the only path Ham Lab allows to be framed.
|
||||
|
||||
```
|
||||
/embed/widget?type=gauge&sensor=cpu.usage&token=<read-only token>
|
||||
&machine=grid &title=… &theme=dark|light|cyberpunk|retro &every=<ms> &bare=1
|
||||
```
|
||||
|
||||
### Glance
|
||||
|
||||
Glance's own `custom-api` widget refetches on page load / cache expiry — fine for slow
|
||||
facts, never live. An `iframe` of the embed is live.
|
||||
|
||||
```yaml
|
||||
- type: iframe
|
||||
title: core CPU
|
||||
source: https://hamlab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
|
||||
height: 130
|
||||
|
||||
- type: custom-api
|
||||
title: grid
|
||||
cache: 1m
|
||||
url: https://hamlab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
|
||||
headers: { Authorization: Bearer <read-only token> }
|
||||
template: |
|
||||
<div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>
|
||||
```
|
||||
|
||||
## The whole fleet in one request
|
||||
|
||||
On a hub, `/api/fleet/sensors/query` asks every machine (or `machines`) the same shaped
|
||||
query and answers `{machine: result}`; this machine answers under its own hostname, and
|
||||
one that can't be reached answers `{"error": …}` while the rest still do.
|
||||
|
||||
```
|
||||
POST /api/fleet/sensors/query {"machines": "*", "query": {"cpu": "cpu.usage", "up": "uptime"}}
|
||||
GET /api/fleet/sensors/query?machines=grid,wyse&cpu=cpu.usage&failed=services.failed
|
||||
|
||||
→ {"core": {"cpu": 4.8, "up": 50211}, "grid": {"cpu": 0.4, "up": 437177}, "mac": {"error": "timeout"}}
|
||||
```
|
||||
|
||||
## What counts as a disk
|
||||
|
||||
`disk.*` names **drives**, not the block devices the kernel lists. An NVMe controller can
|
||||
present several namespaces — an Apple SSD shows `nvme0n1`, `nvme0n2` and `nvme0n3`, the last
|
||||
two a few MiB of firmware — and each one arrives from `/proc/diskstats` looking like a whole
|
||||
disk. They are folded into the controller (`disk.nvme0.*`), with the rates added and busy
|
||||
taken as the longest of them rather than the sum, since they share one controller. This is
|
||||
what the Disks page has always shown; since 2026-09-22 the dashboard, the Top tab and these
|
||||
sensors agree with it.
|
||||
|
||||
## The Top tab
|
||||
|
||||
Ham Lab → **Top** is a whole machine on one screen in the style of btop: black ground, boxes
|
||||
with their name set into the border, curves drawn as dots that climb green → yellow → red.
|
||||
It reads nothing of its own — the sensor stream for everything measured, `/api/procs` for the
|
||||
table — so it works on any machine the console can reach, whether or not btop is installed
|
||||
there, and it follows the same refresh rate and history window as the rest of the console.
|
||||
Its colours are the active skin's: the terminal background for the ground, the skin's sixteen
|
||||
terminal colours for the curves and meters. (The look is btop's, `web/src/Top.tsx` is ours;
|
||||
btop itself is Apache-2.0.)
|
||||
|
||||
## How often a machine samples
|
||||
|
||||
The console's refresh rate (Settings, or `− rate +` on Top) is also the fleet's sampling rate.
|
||||
It is saved with the UI prefs on the hub, which sets its own sampler and passes the rate to
|
||||
every machine it manages — `PUT /api/prefs/sampler {"ms": …}`, 100 ms to 60 s, sampled no
|
||||
slower than the collector's 10 s — and repeats it at start-up. `GET` answers `{ms, min_ms,
|
||||
max_ms}`. Before 2026-09-21 only an open Dashboard page told a machine its rate.
|
||||
|
||||
## How far back a history reaches
|
||||
|
||||
Every machine keeps **300 points per curve**, whatever the window — so the window sets how
|
||||
far apart they are, not how much memory monitoring costs. At the default ten minutes each
|
||||
point is one sample (2 s apart at the default rate); at a day each is the **average of the
|
||||
five minutes behind it**, not whichever instant landed on the mark, so a long graph shows
|
||||
the load that really happened. Ham Lab → Settings → **Graph history** is the one control,
|
||||
1 minute to 24 hours; the hub passes it to every machine it manages.
|
||||
|
||||
The window can't be finer than the sampling interval — 300 points of a 1-minute window at a
|
||||
2-second rate is 30 points, and that is what you get. It is also all in memory: a machine
|
||||
that restarts starts its graphs again.
|
||||
|
||||
## What Ham Lab itself runs on
|
||||
|
||||
Since 2026-09-19 the Dashboard tab (stock modules and custom widgets) and the Disks tab's
|
||||
live rates read only this API: the catalog once, one batch of histories, one stream —
|
||||
closed while the page is hidden, refilled when it returns. `web/src/sampleAdapter.ts` is
|
||||
the seam that hands the stock modules the object they were written against. The old
|
||||
`/api/metrics` and `/api/metrics/history` are gone. `hamlab.sample_interval` is the
|
||||
sampler's own rate, as a sensor.
|
||||
|
||||
## From another origin
|
||||
|
||||
The sensor endpoints (and only they) answer any origin: `Access-Control-Allow-Origin: *`,
|
||||
preflight included. `*` is the one value browsers won't combine with cookies, so a foreign
|
||||
page can reach the API with a token and can never ride a logged-in console's session.
|
||||
`examples/external-demo/` is a complete outside consumer in one HTML file.
|
||||
@@ -0,0 +1,355 @@
|
||||
# Virtualization tab — plan (2026-10-09; phase 1 built the same day)
|
||||
|
||||
**Status:** phases 1 and 2 are **deployed to all five machines as v0.1.0-188** (2026-10-09).
|
||||
Incus runs on `grid` (installed from the tab); the other machines show Install or, on `gate`,
|
||||
unavailable. Phase 2 verified live on `grid` through the hub: ISO fetched by URL with its
|
||||
sha256 checked, **New VM from ISO** booted Alpine with Secure Boot off, **the VGA console
|
||||
drew it in a real browser** (headless Chromium, 1280×800 canvas, keyboard and mouse live),
|
||||
a virtiofs mount appeared in a running VM, an export tarball landed in `virt/exports/` and the
|
||||
nightly cron line is written, `instance.*` sensors flow and the "VM or container down" alert
|
||||
default is in. Two things found and fixed on the way: spice-html5 wants the `binary`
|
||||
websocket subprotocol agreed; and it opens one websocket per SPICE channel, so the bridge
|
||||
holds **one console operation per VM** and gives each channel its own data websocket on it
|
||||
(Incus refuses a second operation as "already connected"). **Left from phase 2:** the LAN
|
||||
bridge step (`br0` on the uplink — needs a live test on a machine that can afford to drop
|
||||
off the network for a moment, which is a decision, not a build), cloud-init presets beyond
|
||||
"Use my SSH keys", and a backup line for `grid`'s `virt/exports/` (see `hamlet/docs/backups.md`).
|
||||
`t-alpine` on `grid` is the live example; delete it from the tab when done.
|
||||
|
||||
A tab for creating, watching, consoling into and deleting **LXC containers and KVM virtual
|
||||
machines**, the way Proxmox does with `pct` and `qm` — **per machine, like every other
|
||||
page.** This is the scoping PLAN.md asked for. Nothing here is built. Live state of the
|
||||
hosts stays in `hamlet/docs/hosts.md`.
|
||||
|
||||
## The governing rule: capability, not policy
|
||||
|
||||
Stated by Mitchell 2026-10-09, after the first draft of this plan excluded hosts on the
|
||||
strength of the homelab's deployment rules:
|
||||
|
||||
> Anything that's done when a different machine is selected SHOULD GO ON THAT machine. I
|
||||
> want the option to do it anywhere I want, unless the hardware / software literally
|
||||
> doesn't support it. Ham Lab needs to be built in a general-purpose way that can run on
|
||||
> any Linux system. If certain parts / libraries aren't available on that machine, those
|
||||
> tabs should just report that.
|
||||
|
||||
So:
|
||||
|
||||
- **Every action lands on the machine in the switcher.** There is no "VM host", no default
|
||||
machine, no cross-machine placement. `grid` is where he will most likely click first;
|
||||
the tab does not know or care.
|
||||
- **The tab is gated by what the machine can do, detected live** — never by a rule about
|
||||
what *should* run there. The `hamlet/CLAUDE.md` do-not-break rules (Mac mini for
|
||||
single-core workloads, `hamtv` not spare hardware) bind the *agent's* unasked
|
||||
deployments. They are not feature gates for the product, and the first draft was wrong
|
||||
to treat them as such.
|
||||
- **Ham Lab owns its data, and its data lives in its own folder.** Everything this tab
|
||||
manages — instance storage, ISOs, image tarballs, exports — sits under Ham Lab's data
|
||||
directory on *that* machine (`config.DataDir()`, `~/apps/hamlab` on `core`, `grid` and
|
||||
the Mac), in one `virt/` tree. Never a path elsewhere on the machine, however convenient
|
||||
the existing folder. Stated 2026-10-09 after the first Library draft pointed the ISO shelf
|
||||
at `/tank/software/isos/`: *"ALL data needs to be under a HamLab folder, because it
|
||||
manages its own data."* This is the rule that moved `core`'s compose files under
|
||||
`apps/hamlab/stacks/` on 2026-10-05; `grid`'s and the Mac's stacks, still at
|
||||
`~/stacks/<machine>/`, are the outstanding exception and are not touched by this plan.
|
||||
The engine's own runtime state (`/var/lib/incus`, like `/var/lib/docker`) is the engine's,
|
||||
not Ham Lab's data: Ham Lab keeps what it would need to rebuild it.
|
||||
- **Ham Lab installs or reports.** It detects what is missing, offers to install it where it
|
||||
knows how, and where it cannot or the hardware cannot do the job, says so in one sentence
|
||||
and offers what *does* work (containers without KVM, for instance).
|
||||
|
||||
## Decision: Incus 6.0 LTS as the engine
|
||||
|
||||
| Option | Verdict |
|
||||
|---|---|
|
||||
| **Incus** (LXD fork, linuxcontainers.org) | **Chosen.** One daemon, one CLI and one JSON API for both containers and VMs. Packaged by Debian 13 (`incus` 6.0.4 LTS — `apt install incus` pulls `qemu-system-x86`, `swtpm`, `virtiofsd`, `incus-agent`, so VMs work out of the box), Fedora (6.23), Arch, Alpine, Ubuntu (Zabbly repo); not by OpenWrt. Storage drivers for ZFS, btrfs, LVM and plain directories, so it fits a machine with or without ZFS. Built-in image server (`images:debian/13`, `images:ubuntu/24.04`, Alpine, Fedora, Arch… as container *or* VM, per architecture), snapshot schedules with expiry, `incus export` backups, per-instance CPU/memory limits, cloud-init, text and VGA console over websocket. Everything is reachable as `incus … --format json` or `incus query`, which is exactly how Ham Lab already drives Docker: `exec.Command`, no new Go dependency, no cgo. |
|
||||
| libvirt/KVM + `virsh`, plus LXC separately | Two daemons, two models, XML. libvirt's official Go binding needs cgo, which the static binary rules out (the pure-Go `go-libvirt` exists but then containers are a second project). Rejected. |
|
||||
| Proxmox | An OS, not a library. Rejected. |
|
||||
|
||||
## What each machine can do today
|
||||
|
||||
Probed 2026-10-09 (`/dev/kvm`, cpu flags, kvm modules, RAM, disk, package availability).
|
||||
These are **facts the tab will detect at runtime**, not a list of where instances are
|
||||
allowed. The last column is what the tab would show the day it ships.
|
||||
|
||||
| Machine | Arch | KVM | RAM (free) | Incus from the package manager | Pool driver available | Tab state on first open |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `grid` | x86-64 | yes — VT-x, `kvm_intel` loaded, nested on | 62 GiB (~25; ARC holds 23 of the rest, max 40) | apt: `incus` 6.0.4 | **zfs** (2.3.9, `tank`) | **Install** — containers and VMs |
|
||||
| `core` | x86-64 | yes — N100, `kvm_intel` loaded | 7.5 GiB (~4) | apt: `incus` 6.0.4 | btrfs or dir (ZFS was purged 2026-09-23) | **Install** — containers and VMs; the create form shows free memory so a VM's size is an informed choice |
|
||||
| Mac mini | aarch64 | yes — Asahi KVM, 16K pages | 15 GiB (~8) | dnf: `incus` 6.23, `qemu-system-aarch64-core`, `edk2-aarch64` | btrfs or dir | **Install** — containers and VMs, **arm64 images only** (the picker filters by host arch). Whether a stock 4K-page guest kernel boots under the 16K host is untested; `muvm` does exactly that, so it is expected to. |
|
||||
| `hamtv` | x86-64 | yes | 3.7 GiB (~2.9) | apt: `incus` 6.0.4 | btrfs or dir | **Install** — containers and small VMs |
|
||||
| `gate` | x86-64 | **no** `/dev/kvm` | 8 GiB (~6) | **none** — OpenWrt's opkg has no Incus | — | **Unavailable**: "Incus isn't packaged for OpenWrt, and this kernel has no KVM." Nothing to install. |
|
||||
|
||||
Nothing is installed anywhere today: no qemu, libvirt, lxc or incus on any host.
|
||||
|
||||
### Detection the tab runs, in order
|
||||
|
||||
1. `incus` on `PATH` and its socket answering → **ready**; read pools, networks, instances.
|
||||
2. Else: a package manager Ham Lab knows (apt, dnf, pacman, apk) that has an `incus`
|
||||
candidate → **installable**, with the Install dialog pre-filled (below).
|
||||
3. Else → **unavailable**, with the reason in one sentence (no package; no root; OpenWrt).
|
||||
4. Independently: `/dev/kvm` present and openable → VMs offered. Absent → **containers
|
||||
only**, said once on the page, and the create form has no VM option.
|
||||
5. Architecture from `uname -m`, so the image picker only shows images that can run here.
|
||||
6. Pool driver: `zfs` if the module and tools are present, else `btrfs` (installing
|
||||
`btrfs-progs` as part of Install), else `dir`. The dialog says which it picked and why.
|
||||
|
||||
## What the tab does (v1)
|
||||
|
||||
Mirrors the Docker tab's shape: a list, a detail page, actions that are followed tasks,
|
||||
destructive ones confirmed by typing the name.
|
||||
|
||||
- **Install** — one task that runs *on the selected machine*: install the package(s),
|
||||
`incus admin init --preseed` with the layout from the dialog, create the pool and bridge,
|
||||
add Ham Lab's user to `incus-admin` (socket access, the `docker`-group shape) and add it
|
||||
to the unit's `SupplementaryGroups`, apply the Docker fix where Docker is present (below),
|
||||
then restart Ham Lab. Idempotent, so it doubles as "repair". The dialog shows and lets him
|
||||
edit: pool driver (detected), pool size (proposed from free space), bridge subnet
|
||||
(computed), whether to apply the Docker rules. Where it all goes is not a choice: Ham
|
||||
Lab's folder.
|
||||
- **Instances** — containers and VMs in one list: type, state, IPv4, CPU and memory live,
|
||||
disk, snapshot count, uptime. Click for detail.
|
||||
- **Create** — container or VM (VM only where KVM is); image from the `images:` remote,
|
||||
filtered to this architecture, searchable, with the common ones pinned (Debian 13, Ubuntu
|
||||
24.04, Alpine, Fedora); name, CPUs, memory (with this machine's free memory beside it),
|
||||
root disk size; network (the NAT bridge, or a LAN bridge if this machine has one);
|
||||
optional cloud-init user-data (SSH key, packages, a first command); profile. Create, or
|
||||
create and start. Collisions (name taken) named before Create.
|
||||
- **Lifecycle** — start, stop, restart, force stop, rename, delete (type the name; the
|
||||
instance's snapshots go with it, and the dialog says so).
|
||||
- **Console** — two buttons, both open as tabs in the Terminal page through the existing PTY
|
||||
manager (`term.CreateCmd`, the `docker exec` path): **Console** (`incus console`, the
|
||||
serial console — works before the network is up, shows boot output) and **Shell**
|
||||
(`incus exec <n> -- login shell`). Containers and, via `incus-agent`, VMs.
|
||||
- **Snapshots** — take, restore (confirmed), delete; a **schedule** per instance with an
|
||||
expiry, which is Incus's own `snapshots.schedule`/`snapshots.expiry`, so no cron file of
|
||||
ours this time.
|
||||
- **Resources** — edit CPU and memory limits (live for containers; VMs take CPU hotplug and
|
||||
memory balloon live with the agent, else on restart), grow the root disk, add a bind mount
|
||||
(a host path into a container; virtiofs into a VM).
|
||||
- **Library** — ISOs for VMs and images (container templates and VM images). Its own
|
||||
section below.
|
||||
- **Storage and networks** — the pools with their fill, the bridges with their subnets.
|
||||
Read-only in v1 beyond what Create and Install need.
|
||||
- **Sensors** — `instance.<name>.cpu`, `.memory`, `.running` from Incus's state, like the
|
||||
`container.*` sensors, so Alerts gets an "instance down" default and widgets can show one.
|
||||
|
||||
## Where the data lives
|
||||
|
||||
One tree per machine, under Ham Lab's data directory, created by Install:
|
||||
|
||||
```
|
||||
~/apps/hamlab/virt/
|
||||
pool.img the instance pool's backing file (zfs or btrfs loop; a plain directory
|
||||
pool/ when the driver is dir) — every instance's disks live in here
|
||||
isos/ the ISO library: the files themselves
|
||||
images/ the image library: template tarballs and VM disk images as files, the
|
||||
way Proxmox keeps CT templates — Incus's image store is loaded from here
|
||||
exports/ nightly `incus export` tarballs, with retention
|
||||
virt.json what Ham Lab chose and when: driver, pool size, bridge subnet, ISO and
|
||||
image inventory, export schedule
|
||||
```
|
||||
|
||||
Consequences, stated once:
|
||||
|
||||
- The tree is on whatever disk holds Ham Lab's data directory — the boot NVMe on all three.
|
||||
Install proposes the pool size from that disk's free space. On `grid` that disk has
|
||||
790 GiB free; on `core`, 175 GiB.
|
||||
- Bulk data a guest needs is **bind-mounted in from where it already is** (`/tank/…` into an
|
||||
instance on `grid`), the same split Docker has: the instance is Ham Lab's, the media is
|
||||
not. That is a device on the instance, recorded in its config, not a second home.
|
||||
- On `core` the whole tree rides the existing nightly rsync to `grid` for free. On `grid`
|
||||
and the Mac it is as backed up as the rest of their Ham Lab folders — which is to say the
|
||||
exports are the backup, and the backup job will need to be told about them in phase 2.
|
||||
- Existing material elsewhere — the 105 GiB of ISOs at `/tank/software/isos/`, the Proxmox
|
||||
archives — is **imported into the tree by copying**, file by file, on request. The
|
||||
library never indexes a folder it does not own.
|
||||
|
||||
## Library: ISOs and images (asked for 2026-10-09)
|
||||
|
||||
Proxmox has a storage view with **ISO Images** and **CT Templates**. Incus has the same two
|
||||
things under other names, and the tab gives them one home, a **Library** sub-tab, per machine
|
||||
like everything else.
|
||||
|
||||
| Proxmox calls it | Incus calls it | What it is here |
|
||||
|---|---|---|
|
||||
| ISO Images | an ISO file attached to a VM as a `disk` device (added as a ROM device; VMs only), or a custom storage volume of type `iso` | **`virt/isos/` in Ham Lab's folder**, attached by path — no copy into the pool |
|
||||
| CT Templates | **images** — the `images:` remote (linuxcontainers.org), the local image store, anything imported or published | **`virt/images/` in Ham Lab's folder** — tarballs and disk images as files, loaded into Incus from there; the remote is the catalogue to pull from |
|
||||
|
||||
### ISOs
|
||||
|
||||
- **The ISO library is `virt/isos/` in Ham Lab's folder**, on every machine, full stop. The
|
||||
105 GiB at `/tank/software/isos/` on `grid` (`linux/`, `windows/`, `truenas/`,
|
||||
`virtio_drivers/`…) is a *source* to import from — pick the ones wanted, Ham Lab copies them
|
||||
in — not the library itself.
|
||||
- **Getting an ISO in:** upload from the browser (the Files tab's `PUT /api/files/upload`
|
||||
with its progress bar, reused), **download from a URL** as a followed task with an optional
|
||||
checksum (the usual way — an ISO starts as a link), or **import from a path** already on the
|
||||
machine — which **copies** the file into `virt/isos/` and says so. Rename, delete (refused while a VM has it attached, naming the VM).
|
||||
- **Using one:** attach to / detach from a VM as a ROM device by path, with `boot.priority`
|
||||
set so it boots first; **New VM from ISO** makes an empty VM (`incus init --empty --vm`)
|
||||
with the ISO attached, starts it and opens the console. After install, Detach is one click
|
||||
and boot order falls back to the disk.
|
||||
- **Honest limit:** an ISO install is an interactive graphical installer, so it needs the
|
||||
**VGA console in the browser** (`spice-html5` over Incus's VGA console websocket).
|
||||
Installers that run on a serial console (Debian with `console=ttyS0`, Alpine, Arch) work
|
||||
through the text console before that exists. Windows needs the virtio drivers ISO attached
|
||||
as a second ROM device as well — Incus's documented path is `distrobuilder repack-windows`
|
||||
to bake them in; the drivers are already in `virtio_drivers/` on `tank`. So **the ISO shelf
|
||||
and the VGA console ship together, in phase 2**, moved up from phase 3.
|
||||
- ISO *volumes* (`incus storage volume import <pool> <iso> <name> --type=iso`) are the
|
||||
alternative: read-only, attachable to several VMs at once, but they copy the file into the
|
||||
pool. Not used here — a file in `virt/isos/` is already Ham Lab's, and a volume would copy it
|
||||
a second time into the pool. Offered as an option only if a need shows up.
|
||||
|
||||
### Images (container templates and VM images)
|
||||
|
||||
- **The catalogue:** browse `images:` filtered to this machine's architecture, by type
|
||||
(container / VM), with the common ones pinned. **Download** copies one into the local
|
||||
store (`incus image copy images:debian/13 local: --alias debian-13`) so a later Create is
|
||||
instant and works without the internet. Images used by Create are cached by Incus anyway;
|
||||
this is just the explicit version, Proxmox's "Templates → Download".
|
||||
- **The local store:** every image here with alias, type, arch, size, origin (remote,
|
||||
published, imported), when it was last used, and **which instances were made from it**
|
||||
(Incus records the base image on each instance). Alias, delete (refused while an
|
||||
instance's base — Incus knows), "New instance from this image".
|
||||
- **Import:** a unified tarball or metadata + rootfs pair (a container template from
|
||||
anywhere, including the phase-3 Proxmox conversions), or a **qcow2 / raw disk with
|
||||
metadata Ham Lab writes** for a VM image — Home Assistant OS, OPNsense, a vendor cloud
|
||||
image — by upload, URL, or a path on the machine. Same three ways in as ISOs.
|
||||
- **Publish:** a stopped instance becomes an image with an alias — Proxmox's "convert to
|
||||
template", the one thing the Images section of v1 always had.
|
||||
- **Where they live:** as files in `virt/images/` — the downloaded or imported tarball, the
|
||||
published image exported to one (`incus image export`) — so the library is complete without
|
||||
Incus and can be reloaded into a fresh install. Incus's own image store under
|
||||
`/var/lib/incus/images/` is a cache derived from it, and the page says so.
|
||||
|
||||
## Not in v1, with reasons
|
||||
|
||||
| Thing | Why not yet |
|
||||
|---|---|
|
||||
| **VGA console in the browser** | Incus hands the VM's SPICE socket over a websocket; drawing it needs `spice-html5` in the UI. Doable (the Incus UI does it); text console + shell cover every Linux guest until then. **Phase 2, with the ISO shelf** — an ISO install is useless without it. |
|
||||
| **Windows VMs** | Incus runs them (ISO repacked with virtio drivers by `distrobuilder`), but without a VGA console in the browser they are unusable from the tab. After the above. |
|
||||
| **GPU passthrough to a VM** | Removes the card from the host and from whatever uses it there (Jellyfin on `grid`). Offered only with that warning, later. Sharing a GPU into a *container* (`gpu` device) leaves the host its card and is how `r720-jellyfin` used to run. Phase 3. |
|
||||
| **Clustering / live migration** | Incus clusters, but these machines differ in size and one is arm64, with no shared storage. Not planned. |
|
||||
| **A fleet-wide instance list** | One machine per view. The switcher is the machine. |
|
||||
| **A second engine (bare LXC on OpenWrt, libvirt-only hosts)** | Would double the backend for one router that has no KVM. The tab reports Incus unavailable there and stops. |
|
||||
|
||||
## Layout decisions Ham Lab makes per machine (overridable in the Install dialog)
|
||||
|
||||
1. **Storage.** The instance pool's backing lives in Ham Lab's folder: Install creates a
|
||||
sparse `virt/pool.img` and hands it to Incus as the pool's `source` (zfs or btrfs by the
|
||||
detection above), or `virt/pool/` as a `dir` pool where neither is available. Sized from
|
||||
the disk's free space, shown and editable. Bulk data is **bind-mounted in**, not stored in
|
||||
the pool. The driver, path and size are recorded in `virt.json` so the page can say where
|
||||
the data is and Install can rebuild the pool from the file after a reinstall.
|
||||
2. **Bridge subnet, computed.** Incus's `auto` picks a random `10.x.y.0/24`, and **this
|
||||
LAN is `10.0.0.0/8`** — every host has a /8 on its NIC, so any 10.x bridge collides.
|
||||
Ham Lab reads the machine's routes and picks the first free /24 from `192.168.0.0/16`,
|
||||
then `172.16.0.0/12`, that is neither on-link nor routed nor inside Docker's configured
|
||||
pool (`172.28.0.0/14` on `grid`; `172.19`–`172.29` and `192.168.80`–`143` live on
|
||||
`core`). Instances get `name.incus` DNS from Incus's dnsmasq.
|
||||
3. **LAN addresses are a later, explicit step.** A `br0` on the machine's uplink lets an
|
||||
instance take a lease from `gate` and a Pi-hole name, but it rewrites the host's network
|
||||
config and drops it off the network for seconds — never part of Install, always its own
|
||||
confirmed action (with the NanoKVM standing by on `grid`). macvlan was considered and
|
||||
rejected: no host change, but host and guests cannot talk, so nothing on the host could
|
||||
reach an instance.
|
||||
4. **Docker and Incus on one host.** Docker sets the `FORWARD` policy to `DROP`, which
|
||||
silently kills the Incus bridge's traffic. Fix is two `DOCKER-USER` accept rules for the
|
||||
bridge, persisted — applied by Install wherever Docker is detected (`grid`, `core`, the
|
||||
Mac, `hamtv` all run it).
|
||||
5. **Memory.** A VM's memory is a hard reservation; the create form shows the machine's
|
||||
free memory and warns past it, and never refuses. On a ZFS host Install offers to cap
|
||||
the ARC (on `grid`, 40 GiB → 16 GiB) so guest RAM is deterministic; his choice.
|
||||
6. **Privilege.** Ham Lab's user joins `incus-admin`; no sudo in the hot path. Install
|
||||
itself goes through `priv` like the other installers.
|
||||
7. **Backups.** Nightly `incus export --instance-only` of every instance to `virt/exports/`
|
||||
with a retention, as a cron line like the ZFS schedules (`hamlab incus-export`), from
|
||||
phase 2. On `core` that folder is inside the nightly rsync already; on `grid` and the Mac
|
||||
the backup job gets a line for it in phase 2.
|
||||
|
||||
## The old Proxmox estate (importable, phase 3)
|
||||
|
||||
`/tank/backups/proxmox/` holds the retired `r720` node: `/etc/pve` (12 LXC confs, 2 VM
|
||||
confs), 118 GiB of `vzdump` archives, and three raw VM disks. Latest of each:
|
||||
|
||||
| ID | Name | Latest archive | Size |
|
||||
|---|---|---|---|
|
||||
| 100 | r720-docker | 2026-01-11 | 20.7 GB |
|
||||
| 101 | r720-cockpit | 2026-01-11 | 439 MB |
|
||||
| 102 | r720-wordpress-alexblaskovich | 2026-01-11 | 1.2 GB |
|
||||
| 103 | r720-spoonterm | 2025-06-14 | 494 MB |
|
||||
| 104 | r720-ollama | 2026-01-11 | 11.3 GB |
|
||||
| 105 | r720-transmission | 2026-01-11 | 322 MB |
|
||||
| 106 | r720-apache | 2026-01-11 | 347 MB |
|
||||
| 107 | r720-nut | 2025-03-29 | 205 MB |
|
||||
| 108 | r720-filebrowser | 2026-01-11 | 262 MB |
|
||||
| 109 | r720-bookstack | 2025-04-27 | 516 MB |
|
||||
| 110 | r720-jellyfin | 2026-01-11 | 2.2 GB |
|
||||
| 111 | r720-wireguard | 2025-04-13 | 510 MB |
|
||||
| 112 | r720-eggwiki | 2026-01-11 | 634 MB |
|
||||
| VMs | steam (102), powerpanel (105/107/112) | `.vma.zst` 2024–2025; raw disks `vm-105`, `vm-112`, `base-104` | 19 GB, 1.9 GB ×6, 97 GiB |
|
||||
|
||||
- **LXC:** feasible and scriptable. Unpack the rootfs (`tar --zstd -xpf … --numeric-owner`),
|
||||
read `etc/vzdump/pct.conf` for arch/ostype, drop it, write `metadata.yaml`, pack, `incus
|
||||
image import`, launch. Unprivileged uid shifting is Incus's job after that. The result is
|
||||
an image in the Library, so a restored container is launched from there like any other
|
||||
template. An import action in the tab (pick an archive on this machine) is plausible once
|
||||
the script works by hand. x86-64 archives only run on x86-64 machines; the picker says so.
|
||||
- **VMs:** the raw disks import directly (`incus-migrate`, or an empty VM with the disk
|
||||
imported into its root volume). The `.vma.zst` archives need Proxmox's `vma` extractor (or
|
||||
the Python port) to become raw disks first. Most of these are the same `powerpanel` box
|
||||
backed up six times, and `tank-space-audit.md` already has them down for deletion — decide
|
||||
which, if any, are wanted before anything is restored.
|
||||
|
||||
## Build plan
|
||||
|
||||
| Phase | What | Effort |
|
||||
|---|---|---|
|
||||
| **1 — detection, Install, the tab** | `internal/incus` (detection, CLI wrapper, JSON types, `incus monitor` for live state, the preseed builder and subnet picker), `internal/api/virt.go` (routes below), `Virtualization.tsx` (the three page states; list, detail, create dialog, snapshot panel), rail entry `virt`, terminal integration, sensors, and the Library's **Images** shelf (browse and download from `images:`, local store, import, publish). **Tested by clicking Install on `grid` first, then `core`** — two different pool drivers, both with Docker present. Then the Mac for the arm64 path. `gate` for the unavailable state. | 2–3 sessions |
|
||||
| **2 — operations (done 2026-10-09, except the LAN bridge)** | The Library's **ISO** shelf (directory, upload/URL/path, attach, New VM from ISO) **with the VGA console** (`spice-html5`), snapshot schedules UI, nightly exports with retention and the per-machine export directory, limits editor, bind mounts, the LAN-bridge step, instance alert default, cloud-init presets. | 1–2 sessions |
|
||||
| **3 — the rest** | Proxmox import (archives become Library images), GPU device for containers, Windows via `distrobuilder repack-windows`, GPU passthrough to a VM with its warning. | as wanted |
|
||||
|
||||
### API sketch
|
||||
|
||||
```
|
||||
GET /api/virt {state: ready|installable|unavailable, reason, kvm, arch,
|
||||
pool_drivers[], pools[], networks[], instances[], free_memory}
|
||||
POST /api/virt/install {driver, size, path, subnet, docker_rules, arc_max?} task
|
||||
GET /api/virt/images?remote=images&q= local cache + remote search, filtered to this arch
|
||||
DELETE /api/virt/images/{fingerprint}
|
||||
POST /api/virt/instances {name, type, image, cpu, memory, disk, network, cloudinit, profile, start} task
|
||||
GET /api/virt/instances/{name} config, state, devices, snapshots
|
||||
PATCH /api/virt/instances/{name} limits, description, devices
|
||||
POST /api/virt/instances/{name}/{start|stop|restart|kill|delete|rename|publish|export}
|
||||
POST /api/virt/instances/{name}/console → terminal {id} (incus console)
|
||||
POST /api/virt/instances/{name}/exec → terminal {id} (incus exec -- sh -lc …)
|
||||
POST /api/virt/instances/{name}/snapshots {name}; POST …/{snap}/restore; DELETE …/{snap}
|
||||
PUT /api/virt/instances/{name}/schedule {cron, expiry}
|
||||
POST /api/virt/import {path} task (export tarball, or phase-3 Proxmox archive)
|
||||
|
||||
GET /api/virt/isos virt/isos/, scanned: name, size, attached-to
|
||||
POST /api/virt/isos/import {path} task: copy a file on this machine into virt/isos/
|
||||
POST /api/virt/isos/fetch {url, sha256?} task; uploads reuse PUT /api/files/upload
|
||||
DELETE /api/virt/isos/{name} refused while attached
|
||||
POST /api/virt/instances/{name}/iso {iso, boot_first} attach; DELETE …/iso detach
|
||||
POST /api/virt/instances/from-iso {name, iso, cpu, memory, disk, network} task → console
|
||||
GET /api/virt/images/remote?q=&type= the `images:` catalogue for this arch
|
||||
POST /api/virt/images/download {alias} task (incus image copy images:… local:)
|
||||
POST /api/virt/images/import {path | url, alias, type} task: copy into virt/images/, then load into Incus
|
||||
GET /api/virt/instances/{name}/vga websocket: Incus VGA console → spice-html5 (phase 2)
|
||||
```
|
||||
|
||||
Everything is per machine and reaches a node through the existing `/api/nodes/{name}/…`
|
||||
proxy with no hub-side code; consoles attach over the existing `/api/terminals/{id}/attach`.
|
||||
Tasks stream through `/api/tasks/{id}/stream` as Docker's do.
|
||||
|
||||
## Sources
|
||||
|
||||
- Incus on Debian: https://wiki.debian.org/Incus
|
||||
- Incus and Docker on one host (the `FORWARD` drop): https://linuxcontainers.org/incus/docs/main/howto/network_bridge_firewalld/
|
||||
- Bridge options and `ipv4.address: auto`: https://linuxcontainers.org/incus/docs/main/reference/network_bridge/
|
||||
- Consoles (text and VGA): https://linuxcontainers.org/incus/docs/main/howto/instances_console/
|
||||
- ZFS storage driver: https://linuxcontainers.org/incus/docs/main/reference/storage_zfs/
|
||||
- macvlan's host-isolation: https://linuxcontainers.org/incus/docs/main/reference/network_macvlan/
|
||||
- Image format for the Proxmox import: https://linuxcontainers.org/incus/docs/main/reference/image_format/
|
||||
+461
-18
@@ -2,16 +2,24 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"runtime/debug"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
@@ -20,8 +28,21 @@ import (
|
||||
"hamlab/internal/api"
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/cron"
|
||||
"hamlab/internal/files"
|
||||
"hamlab/internal/incus"
|
||||
"hamlab/internal/metrics"
|
||||
"hamlab/internal/nodes"
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/services"
|
||||
"hamlab/internal/setup"
|
||||
"hamlab/internal/space"
|
||||
"hamlab/internal/storage"
|
||||
"hamlab/internal/store"
|
||||
"hamlab/internal/tasks"
|
||||
hterm "hamlab/internal/term"
|
||||
"hamlab/internal/tlsauto"
|
||||
"hamlab/internal/updates"
|
||||
"hamlab/internal/web"
|
||||
)
|
||||
|
||||
@@ -30,12 +51,26 @@ var version = "dev"
|
||||
func main() {
|
||||
defaultCfg := os.Getenv("HAMLAB_CONFIG")
|
||||
if defaultCfg == "" {
|
||||
home, _ := os.UserHomeDir()
|
||||
defaultCfg = filepath.Join(home, "apps", "hamlab", "config.json")
|
||||
defaultCfg = config.DefaultPath()
|
||||
}
|
||||
cfgPath := flag.String("config", defaultCfg, "path to config.json")
|
||||
flag.Usage = func() {
|
||||
fmt.Fprintf(os.Stderr, "usage: hamlab [-config path] [serve|passwd|version]\n")
|
||||
fmt.Fprint(os.Stderr, `usage: hamlab [-config path] <command>
|
||||
|
||||
install [flags] install this binary as a service on this machine (install -h for flags)
|
||||
uninstall [--purge] remove the service and the binary
|
||||
serve run the console in the foreground (what the service runs)
|
||||
passwd [user] set a user's password (makes the account if it is new; reads stdin when piped)
|
||||
users list the accounts
|
||||
zfs-snapshot <id> run one ZFS snapshot schedule (what its cron line calls)
|
||||
virt-export [name…] export Incus instances to the library (what its cron line calls)
|
||||
docker-update <id> run one image-update schedule (what its cron line calls)
|
||||
user-rm <user> remove an account and sign it out everywhere
|
||||
check verify the config and state load, then exit
|
||||
version
|
||||
push-nodes [--force] hub: send this build to every managed machine
|
||||
|
||||
`)
|
||||
flag.PrintDefaults()
|
||||
}
|
||||
flag.Parse()
|
||||
@@ -44,6 +79,45 @@ func main() {
|
||||
if flag.NArg() > 0 {
|
||||
cmd = flag.Arg(0)
|
||||
}
|
||||
switch cmd { // commands that don't read the config
|
||||
case "fs": // privileged helper for the Files tab (run as root by the daemon)
|
||||
os.Exit(files.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||
case "space-walk": // helper for the Space tab: measure mountpoints, stream the tree back
|
||||
os.Exit(space.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||
case "drm-clients": // privileged helper for the GPU readings: who has the GPU open (run as root by the daemon)
|
||||
os.Exit(metrics.DRMClientsHelper(os.Stdin, os.Stdout))
|
||||
case "install", "uninstall":
|
||||
var err error
|
||||
if cmd == "install" {
|
||||
err = setup.Install(flag.Args()[1:], version)
|
||||
} else {
|
||||
err = setup.Uninstall(flag.Args()[1:])
|
||||
}
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
os.Exit(0)
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "hamlab "+cmd+":", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return
|
||||
case "replace": // self-update helper (run as root by the daemon): put this binary at a path, atomically
|
||||
// Only ever over a hamlab binary: this runs as root, and must not be a way to
|
||||
// overwrite arbitrary files for whoever is allowed to run hamlab as root.
|
||||
if flag.NArg() != 2 || !filepath.IsAbs(flag.Arg(1)) || !strings.HasPrefix(filepath.Base(flag.Arg(1)), "hamlab") {
|
||||
fmt.Fprintln(os.Stderr, "hamlab replace: destination must be an absolute path to a hamlab binary")
|
||||
os.Exit(2)
|
||||
}
|
||||
self, err := os.Executable()
|
||||
if err == nil {
|
||||
err = setup.CopyAtomic(self, flag.Arg(1))
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "hamlab replace:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return
|
||||
}
|
||||
cfg, err := config.Load(*cfgPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
@@ -51,10 +125,74 @@ func main() {
|
||||
switch cmd {
|
||||
case "version":
|
||||
fmt.Println("hamlab", version)
|
||||
case "passwd":
|
||||
if err := passwd(cfg); err != nil {
|
||||
case "check": // a new build runs this against the live config before it replaces the old one
|
||||
if _, err := store.Open(filepath.Join(cfg.DataDir(), "state.json")); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
if _, err := fs.Stat(web.FS(), "index.html"); err != nil {
|
||||
log.Fatal("embedded UI missing: ", err)
|
||||
}
|
||||
fmt.Println("ok")
|
||||
case "passwd":
|
||||
if err := passwd(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "docker-update":
|
||||
if err := dockerUpdateJob(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "zfs-snapshot":
|
||||
if err := zfsSnapshot(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "virt-export": // the nightly cron line from the Virtualization tab; optional instance names
|
||||
if err := virtExport(cfg, flag.Args()[1:]); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "users":
|
||||
migrateUsers(cfg)
|
||||
if len(cfg.Users) == 0 {
|
||||
fmt.Println("no accounts in", cfg.Path())
|
||||
}
|
||||
for _, u := range cfg.Users {
|
||||
seen := "never"
|
||||
if !u.LastSeen.IsZero() {
|
||||
seen = u.LastSeen.Local().Format("2006-01-02 15:04")
|
||||
}
|
||||
note := ""
|
||||
if u.Guest { // a temporary look-only login, not one of the machine's accounts
|
||||
note = " guest, until " + u.Expires.Local().Format("15:04")
|
||||
if u.Expired(time.Now()) {
|
||||
note = " guest, expired"
|
||||
}
|
||||
}
|
||||
fmt.Printf("%-20s created %s last seen %s%s\n", u.Name, u.Created.Local().Format("2006-01-02"), seen, note)
|
||||
}
|
||||
case "user-rm":
|
||||
if err := userRm(cfg, flag.Arg(1)); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "push-nodes": // hub CLI: send this binary to every managed machine
|
||||
exe, _ := os.Executable()
|
||||
host, _ := os.Hostname()
|
||||
nm := nodes.New(cfg, host)
|
||||
force := flag.NArg() > 1 && flag.Arg(1) == "--force"
|
||||
results := nm.PushAll(context.Background(), exe, runtime.GOARCH, force)
|
||||
if len(results) == 0 {
|
||||
fmt.Println("no managed machines in", cfg.Path())
|
||||
}
|
||||
failed := 0
|
||||
for _, r := range results {
|
||||
if r.OK {
|
||||
fmt.Printf("%-12s updated to %s\n", r.Name, r.Version)
|
||||
} else {
|
||||
failed++
|
||||
fmt.Printf("%-12s FAILED: %s\n", r.Name, r.Error)
|
||||
}
|
||||
}
|
||||
if failed > 0 {
|
||||
os.Exit(1)
|
||||
}
|
||||
case "serve":
|
||||
if err := serve(cfg); err != nil {
|
||||
log.Fatal(err)
|
||||
@@ -65,9 +203,185 @@ func main() {
|
||||
}
|
||||
}
|
||||
|
||||
func passwd(cfg *config.Config) error {
|
||||
fmt.Fprint(os.Stderr, "New password: ")
|
||||
p1, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
// migrateUsers gives a pre-users config its first account, named after this OS user, so
|
||||
// the CLI sees the same accounts the daemon will.
|
||||
// dockerUpdateJob pulls a stack's images, and deploys them when the schedule says to. This
|
||||
// is what the cron line Ham Lab writes actually calls, so image updates keep happening
|
||||
// whether or not the console is up.
|
||||
func dockerUpdateJob(cfg *config.Config, id string) error {
|
||||
if strings.TrimSpace(id) == "" {
|
||||
return errors.New("usage: hamlab docker-update <schedule-id>")
|
||||
}
|
||||
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var raw json.RawMessage
|
||||
st.View(func(s *store.State) { raw = s.DockerJobs })
|
||||
var d struct {
|
||||
Jobs []api.UpdateJob `json:"jobs"`
|
||||
}
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &d); err != nil {
|
||||
return fmt.Errorf("reading the schedules: %w", err)
|
||||
}
|
||||
}
|
||||
for _, j := range d.Jobs {
|
||||
if j.ID != id {
|
||||
continue
|
||||
}
|
||||
// A pull that never finishes must not leave a cron job resident until the next reboot.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Hour)
|
||||
defer cancel()
|
||||
out, runErr := api.RunUpdate(ctx, st, j, api.DockerTool())
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: j.Stack}
|
||||
if runErr != nil {
|
||||
rec.Error = runErr.Error()
|
||||
}
|
||||
if err := storage.WriteRun(cfg.DataDir(), "docker-"+id, rec); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "recording the run:", err)
|
||||
}
|
||||
fmt.Print(out)
|
||||
return runErr
|
||||
}
|
||||
return fmt.Errorf("no image-update schedule %q — it may have been removed; this cron line is stale", id)
|
||||
}
|
||||
|
||||
// zfsSnapshot runs one snapshot schedule: take it, then expire what has aged out. This is
|
||||
// what the cron line Ham Lab writes actually calls, so a schedule keeps running whether or
|
||||
// not the console is up — and keeps running if Ham Lab is never opened again.
|
||||
//
|
||||
// It deliberately does not write state.json. The daemon owns that file and would clobber
|
||||
// anything written from out here; the outcome goes beside it instead, in zfs-runs/.
|
||||
func zfsSnapshot(cfg *config.Config, id string) error {
|
||||
if strings.TrimSpace(id) == "" {
|
||||
return errors.New("usage: hamlab zfs-snapshot <schedule-id>")
|
||||
}
|
||||
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var raw json.RawMessage
|
||||
st.View(func(s *store.State) { raw = s.ZFS })
|
||||
var z struct {
|
||||
Policies []storage.Policy `json:"policies"`
|
||||
}
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &z); err != nil {
|
||||
return fmt.Errorf("reading the schedules: %w", err)
|
||||
}
|
||||
}
|
||||
var p *storage.Policy
|
||||
for i := range z.Policies {
|
||||
if z.Policies[i].ID == id {
|
||||
p = &z.Policies[i]
|
||||
}
|
||||
}
|
||||
if p == nil {
|
||||
return fmt.Errorf("no snapshot schedule %q — it may have been removed; this cron line is stale", id)
|
||||
}
|
||||
res, runErr := storage.New(priv.Available()).Run(context.Background(), p)
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: res.Taken, Pruned: res.Pruned}
|
||||
if runErr != nil {
|
||||
rec.Error = runErr.Error()
|
||||
}
|
||||
if err := storage.WriteRun(cfg.DataDir(), id, rec); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "recording the run:", err)
|
||||
}
|
||||
if runErr != nil {
|
||||
return runErr
|
||||
}
|
||||
fmt.Printf("%s: took %s", p.Name, res.Taken)
|
||||
if len(res.Pruned) > 0 {
|
||||
fmt.Printf(", expired %d", len(res.Pruned))
|
||||
}
|
||||
fmt.Println()
|
||||
return nil
|
||||
}
|
||||
|
||||
// virtExport runs the instance exports the Virtualization tab scheduled, as the account Ham
|
||||
// Lab runs as (which is in incus-admin), and records the run beside the data.
|
||||
func virtExport(cfg *config.Config, names []string) error {
|
||||
who := os.Getenv("USER")
|
||||
if u, err := user.Current(); err == nil && u.Username != "" {
|
||||
who = u.Username
|
||||
}
|
||||
run := incus.New(cfg.DataDir(), who, version).RunExports(context.Background(), names, func(l string) { fmt.Println(l) })
|
||||
fmt.Printf("exported %d, pruned %d, %.0f s\n", len(run.Exported), len(run.Pruned), run.Seconds)
|
||||
if len(run.Errors) > 0 {
|
||||
return errors.New(strings.Join(run.Errors, "; "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func migrateUsers(cfg *config.Config) {
|
||||
name := os.Getenv("USER")
|
||||
if u, err := user.Current(); err == nil && u.Username != "" {
|
||||
name = u.Username
|
||||
}
|
||||
if cfg.MigrateUsers(name) {
|
||||
_ = cfg.Save()
|
||||
}
|
||||
}
|
||||
|
||||
func userRm(cfg *config.Config, name string) error {
|
||||
migrateUsers(cfg)
|
||||
if name == "" {
|
||||
return errors.New("usage: hamlab user-rm <user>")
|
||||
}
|
||||
if len(cfg.Users) <= 1 {
|
||||
return errors.New("the last account can't be removed")
|
||||
}
|
||||
kept := cfg.Users[:0]
|
||||
found := false
|
||||
for _, u := range cfg.Users {
|
||||
if u.Name == name {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
if !found {
|
||||
return fmt.Errorf("no account %q", name)
|
||||
}
|
||||
cfg.Users = kept
|
||||
if err := cfg.Save(); err != nil {
|
||||
return err
|
||||
}
|
||||
if st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json")); err == nil {
|
||||
auth.NewSessions(st).RevokeUser(name)
|
||||
_ = st.Update(func(s *store.State) { delete(s.Users, name) })
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "removed", name, "(the running service picks it up at its next config save; restart it to be sure)")
|
||||
return nil
|
||||
}
|
||||
|
||||
func passwd(cfg *config.Config, name string) error {
|
||||
migrateUsers(cfg)
|
||||
if name == "" {
|
||||
switch len(cfg.Users) {
|
||||
case 0:
|
||||
name = os.Getenv("USER")
|
||||
if u, err := user.Current(); err == nil && u.Username != "" {
|
||||
name = u.Username
|
||||
}
|
||||
case 1:
|
||||
name = cfg.Users[0].Name
|
||||
default:
|
||||
return errors.New("several accounts: hamlab passwd <user>")
|
||||
}
|
||||
}
|
||||
if !auth.UserNameRe.MatchString(name) {
|
||||
return fmt.Errorf("%q: a user name is 1–32 lower-case letters, digits, . _ or -", name)
|
||||
}
|
||||
var p1 []byte
|
||||
if !term.IsTerminal(int(os.Stdin.Fd())) { // scripted: one line on stdin
|
||||
line, _ := bufio.NewReader(os.Stdin).ReadString('\n')
|
||||
p1 = []byte(strings.TrimRight(line, "\r\n"))
|
||||
} else {
|
||||
fmt.Fprintf(os.Stderr, "New password for %s: ", name)
|
||||
var err error
|
||||
p1, err = term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -81,24 +395,39 @@ func passwd(cfg *config.Config) error {
|
||||
if string(p1) != string(p2) {
|
||||
return errors.New("passwords do not match")
|
||||
}
|
||||
if len(p1) < 12 {
|
||||
}
|
||||
if len(p1) < auth.MinPasswordLen {
|
||||
return errors.New("password must be at least 12 characters")
|
||||
}
|
||||
h, err := auth.HashPassword(string(p1))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cfg.PasswordHash = h
|
||||
made := true
|
||||
for i := range cfg.Users {
|
||||
if cfg.Users[i].Name == name {
|
||||
cfg.Users[i].Hash = h
|
||||
made = false
|
||||
}
|
||||
}
|
||||
if made {
|
||||
cfg.Users = append(cfg.Users, config.User{Name: name, Hash: h, Created: time.Now()})
|
||||
}
|
||||
if err := cfg.Save(); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "password set in", cfg.Path())
|
||||
if made {
|
||||
fmt.Fprintln(os.Stderr, "account", name, "made in", cfg.Path())
|
||||
} else {
|
||||
fmt.Fprintln(os.Stderr, "password of", name, "set in", cfg.Path())
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "the running service reads the config at its next save; restart it to be sure")
|
||||
return nil
|
||||
}
|
||||
|
||||
func serve(cfg *config.Config) error {
|
||||
if cfg.PasswordHash == "" {
|
||||
return fmt.Errorf("no password set — run: hamlab -config %s passwd", cfg.Path())
|
||||
if cfg.PasswordHash == "" && len(cfg.Users) == 0 {
|
||||
return fmt.Errorf("no account set — run: hamlab -config %s passwd", cfg.Path())
|
||||
}
|
||||
if err := os.MkdirAll(cfg.DataDir(), 0o700); err != nil {
|
||||
return err
|
||||
@@ -115,15 +444,17 @@ func serve(cfg *config.Config) error {
|
||||
}
|
||||
shell := os.Getenv("SHELL")
|
||||
if shell == "" {
|
||||
shell = "/bin/bash"
|
||||
shell = loginShell(u.Username)
|
||||
}
|
||||
mgr := hterm.NewManager(shell, u.HomeDir)
|
||||
mgr.OnChange = func(list []hterm.Info) {
|
||||
_ = st.Update(func(s *store.State) {
|
||||
s.Terminals = s.Terminals[:0]
|
||||
for _, t := range list {
|
||||
if t.Running { // an exited shell must not come back from the dead on restart
|
||||
s.Terminals = append(s.Terminals, store.Terminal{ID: t.ID, Name: t.Name})
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
// Re-open the shells that existed before the last restart (fresh processes,
|
||||
@@ -136,12 +467,57 @@ func serve(cfg *config.Config) error {
|
||||
}
|
||||
}
|
||||
|
||||
srv := api.New(cfg, sessions, mgr, web.FS(), version)
|
||||
// Keep the heap tidy: the daemon's live data is tens of MB; without a limit the GC
|
||||
// happily lets RSS coast at twice that. (Soft limit: exceeding it only means more GC.)
|
||||
debug.SetMemoryLimit(160 << 20)
|
||||
|
||||
col := metrics.NewCollector()
|
||||
go col.Run(context.Background())
|
||||
|
||||
svc := services.New()
|
||||
cr := cron.New(svc.Sudo(), svc.JournalSudo())
|
||||
tm := tasks.New()
|
||||
up := updates.New(svc.Sudo(), tm)
|
||||
sto := storage.New(svc.Sudo())
|
||||
self, _ := os.Executable()
|
||||
var sudoFS files.FS
|
||||
if svc.Sudo() && self != "" {
|
||||
sudoFS = files.Sudo{Self: self}
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
nm := nodes.New(cfg, host)
|
||||
go nm.Run(context.Background())
|
||||
|
||||
certPath, keyPath, fingerprint := "", "", ""
|
||||
if cfg.TLS == "auto" {
|
||||
var err error
|
||||
certPath, keyPath, fingerprint, err = tlsauto.Ensure(cfg.DataDir())
|
||||
if err != nil {
|
||||
return fmt.Errorf("tls: %w", err)
|
||||
}
|
||||
}
|
||||
ln, err := listen(cfg.Listen)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
srv := api.New(cfg, sessions, mgr, col, st, svc, cr, tm, up, sto, files.Local{}, sudoFS, nm, fingerprint, web.FS(), version)
|
||||
hs := &http.Server{
|
||||
Addr: cfg.Listen,
|
||||
Handler: srv.Handler(),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
// After a self-update: become the new binary in place. Same PID, so it works the same
|
||||
// under systemd, OpenRC, or no supervisor at all. It returns only if the exec failed, by
|
||||
// which point the listener and the shells are gone; the caller puts the old binary back
|
||||
// and exits so the supervisor starts it.
|
||||
var restarting atomic.Bool
|
||||
srv.Restart = func() error {
|
||||
restarting.Store(true) // Serve returns once Shutdown is called; serve() must not exit under us
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
_ = hs.Shutdown(ctx)
|
||||
mgr.Shutdown()
|
||||
return syscall.Exec(self, os.Args, os.Environ())
|
||||
}
|
||||
go func() {
|
||||
sig := make(chan os.Signal, 1)
|
||||
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
|
||||
@@ -152,9 +528,76 @@ func serve(cfg *config.Config) error {
|
||||
_ = hs.Shutdown(ctx)
|
||||
mgr.Shutdown()
|
||||
}()
|
||||
log.Printf("hamlab %s listening on %s as %s (shell %s)", version, cfg.Listen, u.Username, shell)
|
||||
if err := hs.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
|
||||
if certPath != "" {
|
||||
log.Printf("hamlab %s listening on https://%s as %s (shell %s) cert %s", version, cfg.Listen, u.Username, shell, fingerprint)
|
||||
if err := hs.ServeTLS(ln, certPath, keyPath); !errors.Is(err, http.ErrServerClosed) {
|
||||
return err
|
||||
}
|
||||
return waitIf(&restarting)
|
||||
}
|
||||
log.Printf("hamlab %s listening on %s as %s (shell %s)", version, cfg.Listen, u.Username, shell)
|
||||
if err := hs.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
|
||||
return err
|
||||
}
|
||||
return waitIf(&restarting)
|
||||
}
|
||||
|
||||
// waitIf parks the caller while a restart is replacing the process.
|
||||
func waitIf(restarting *atomic.Bool) error {
|
||||
if restarting.Load() {
|
||||
select {}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// listen binds the configured address with IP_FREEBIND, so a LAN address that isn't up yet
|
||||
// (boot, DHCP still negotiating, cable out) doesn't stop the daemon: it binds now and
|
||||
// answers as soon as the address appears.
|
||||
func listen(addr string) (net.Listener, error) {
|
||||
lc := net.ListenConfig{Control: func(network, _ string, c syscall.RawConn) error {
|
||||
return c.Control(func(fd uintptr) {
|
||||
_ = syscall.SetsockoptInt(int(fd), syscall.IPPROTO_IP, syscall.IP_FREEBIND, 1)
|
||||
})
|
||||
}}
|
||||
ln, err := lc.Listen(context.Background(), "tcp", addr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listen %s: %w", addr, err)
|
||||
}
|
||||
if host, _, _ := net.SplitHostPort(addr); host != "" {
|
||||
if ip := net.ParseIP(host); ip != nil && !ip.IsUnspecified() && !ip.IsLoopback() && !onInterface(ip) {
|
||||
log.Printf("listen address %s isn't on any interface yet; bound anyway and waiting for it", host)
|
||||
}
|
||||
}
|
||||
return ln, nil
|
||||
}
|
||||
|
||||
func onInterface(ip net.IP) bool {
|
||||
addrs, _ := net.InterfaceAddrs()
|
||||
for _, a := range addrs {
|
||||
if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// loginShell returns the user's shell from /etc/passwd, falling back to bash if
|
||||
// present and /bin/sh otherwise. Systemd units don't export $SHELL.
|
||||
func loginShell(username string) string {
|
||||
if b, err := os.ReadFile("/etc/passwd"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
f := strings.Split(line, ":")
|
||||
if len(f) >= 7 && f[0] == username && f[6] != "" {
|
||||
if _, err := os.Stat(f[6]); err == nil {
|
||||
return f[6]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, sh := range []string{"/bin/bash", "/usr/bin/bash", "/bin/zsh", "/bin/sh"} {
|
||||
if _, err := os.Stat(sh); err == nil {
|
||||
return sh
|
||||
}
|
||||
}
|
||||
return "/bin/sh"
|
||||
}
|
||||
|
||||
@@ -1,18 +0,0 @@
|
||||
[Unit]
|
||||
Description=Ham Lab web console
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=core
|
||||
Group=core
|
||||
WorkingDirectory=/home/core
|
||||
Environment=HAMLAB_CONFIG=/home/core/apps/hamlab/config.json
|
||||
ExecStart=/usr/local/bin/hamlab serve
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
KillMode=control-group
|
||||
TimeoutStopSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -1,11 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Install the built binary and unit on this host, then (re)start the service.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
sudo install -m 0755 bin/hamlab /usr/local/bin/hamlab
|
||||
sudo install -m 0644 deploy/hamlab.service /etc/systemd/system/hamlab.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now hamlab.service >/dev/null
|
||||
sudo systemctl restart hamlab.service
|
||||
sleep 1
|
||||
systemctl is-active hamlab.service && /usr/local/bin/hamlab version
|
||||
@@ -0,0 +1,21 @@
|
||||
# External demo
|
||||
|
||||
`index.html` is a stranger's page: one file, no build, no framework, served from a
|
||||
different origin than Ham Lab. It knows a base URL and a read-only sensor token
|
||||
(`config.js`, not in git) and nothing else. Each tab is one shaped query POSTed to
|
||||
`/api/sensors/query?full=1` on an interval; sparklines are seeded from
|
||||
`/api/sensors/history`; the Fleet tab is one call to `/api/fleet/sensors/query`.
|
||||
|
||||
```js
|
||||
// config.js
|
||||
window.HAMLAB = { base: 'http://10.0.0.234:8100', token: '<read-only sensor token>' }
|
||||
```
|
||||
|
||||
Anyone who can load the page can read the token. If the page is public, limit the token to
|
||||
patterns that name nothing you'd mind the world reading — hardware numbers and counts,
|
||||
not container or service names.
|
||||
|
||||
This is an example to read and copy, not something that runs anywhere. It was deployed on
|
||||
`core` as the `hamlab-demo` stack and linked from the Widgets tab until 2026-09-23, when
|
||||
both were removed: the Widgets tab shows the same widgets against the machine you are
|
||||
already signed in to, so a second public copy earned nothing and cost a public token.
|
||||
@@ -0,0 +1,179 @@
|
||||
<!doctype html>
|
||||
<!--
|
||||
Ham Lab sensor API — external demo.
|
||||
|
||||
One file, no build, no framework, not part of Ham Lab: it is a stranger's page on another
|
||||
origin that knows only a base URL and a read-only token (config.js, beside this file):
|
||||
|
||||
window.HAMLAB = { base: 'http://10.0.0.234:8100', token: '<read-only sensor token>' }
|
||||
|
||||
Every tab is one shaped query — the object it wants, with sensor IDs where the values go —
|
||||
POSTed to /api/sensors/query?full=1 on an interval. Sparklines are seeded once from
|
||||
/api/sensors/history. The Fleet tab uses /api/fleet/sensors/query. See SENSORS.md.
|
||||
-->
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Ham Lab sensors — external demo</title>
|
||||
<script src="config.js"></script>
|
||||
<style>
|
||||
:root { --bg:#0f1216; --card:#171b21; --line:#2a313b; --fg:#d5dbe2; --mut:#7d8590; --acc:#e3a33b; --s1:#3987e5; --s2:#c98500; --bad:#ff7b72; --ok:#3fb950; --track:rgba(57,135,229,.18) }
|
||||
* { box-sizing: border-box }
|
||||
body { margin:0; background:var(--bg); color:var(--fg); font:14px/1.4 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif }
|
||||
header { display:flex; align-items:center; gap:10px; flex-wrap:wrap; padding:10px 14px; background:var(--card); border-bottom:1px solid var(--line) }
|
||||
header h1 { margin:0; font-size:15px; color:var(--acc); letter-spacing:.4px }
|
||||
header .sp { flex:1 }
|
||||
.mut { color:var(--mut) } .mono { font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace }
|
||||
select { background:var(--bg); color:var(--fg); border:1px solid var(--line); border-radius:5px; padding:4px 6px; font:inherit }
|
||||
.dot { width:8px; height:8px; border-radius:50%; background:var(--mut); display:inline-block } .dot.ok { background:var(--ok) } .dot.bad { background:var(--bad) }
|
||||
nav { display:flex; gap:2px; overflow-x:auto; padding:0 8px; background:var(--card); border-bottom:1px solid var(--line); scrollbar-width:none }
|
||||
nav button { flex:0 0 auto; background:none; border:0; border-bottom:2px solid transparent; color:var(--mut); padding:10px 12px; font:inherit; cursor:pointer }
|
||||
nav button.on { color:var(--fg); border-bottom-color:var(--acc) }
|
||||
main { padding:12px; max-width:1100px; margin:0 auto }
|
||||
.grid { display:grid; grid-template-columns:repeat(auto-fill,minmax(170px,1fr)); gap:10px }
|
||||
.tile { background:var(--card); border:1px solid var(--line); border-radius:8px; padding:12px; min-width:0; overflow:hidden }
|
||||
.tile.wide { grid-column:1/-1 }
|
||||
.label { font-size:11px; font-weight:600; color:var(--mut); text-transform:uppercase; letter-spacing:.5px; margin-bottom:6px; overflow-wrap:anywhere }
|
||||
.value { font-size:22px; font-weight:600; font-variant-numeric:tabular-nums } .value.warn { color:var(--s2) } .value.bad { color:var(--bad) }
|
||||
.id { margin-top:8px; font-size:10.5px; color:var(--mut); overflow-wrap:anywhere }
|
||||
.meter { height:6px; background:var(--track); border-radius:3px; overflow:hidden; margin-top:8px } .meter i { display:block; height:100%; background:var(--s1); border-radius:3px; transition:width .3s } .meter i.warn { background:var(--s2) } .meter i.bad { background:var(--bad) }
|
||||
svg.gauge { display:block; width:100%; max-width:180px; margin:0 auto } svg.gauge text { fill:var(--fg); font-size:13px; font-weight:600 }
|
||||
svg.spark { display:block; width:100%; height:44px; margin-top:6px }
|
||||
.rows { display:flex; flex-direction:column; gap:7px } .row { display:grid; grid-template-columns:1fr auto; gap:2px 10px; font-size:13px; align-items:baseline }
|
||||
.row .k { overflow:hidden; text-overflow:ellipsis; white-space:nowrap } .row .n { font-variant-numeric:tabular-nums } .row .meter { grid-column:1/3; margin-top:0 }
|
||||
.pill { font-size:11px; padding:1px 7px; border-radius:9px; border:1px solid var(--line) } .pill.ok { color:var(--ok); border-color:var(--ok) } .pill.bad { color:var(--bad); border-color:var(--bad) }
|
||||
details { margin-top:14px; background:var(--card); border:1px solid var(--line); border-radius:8px; padding:10px 12px } summary { cursor:pointer; color:var(--mut) }
|
||||
pre { margin:8px 0 0; font:12px/1.5 ui-monospace,Menlo,Consolas,monospace; white-space:pre-wrap; word-break:break-word; max-height:320px; overflow:auto }
|
||||
.err { color:var(--bad); padding:12px }
|
||||
table { width:100%; border-collapse:collapse; font-size:13px } th, td { text-align:left; padding:6px 8px; border-bottom:1px solid var(--line); font-variant-numeric:tabular-nums } th { color:var(--mut); font-weight:600; font-size:11px; text-transform:uppercase }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>Ham Lab sensors</h1><span class="mut">external demo · <span id="base" class="mono"></span></span>
|
||||
<span class="sp"></span>
|
||||
<span id="status" class="mut"></span><span id="dot" class="dot"></span>
|
||||
<label class="mut">every <select id="every"><option value="1000">1 s</option><option value="2000" selected>2 s</option><option value="5000">5 s</option><option value="10000">10 s</option><option value="30000">30 s</option></select></label>
|
||||
</header>
|
||||
<nav id="tabs"></nav>
|
||||
<main>
|
||||
<div id="view"></div>
|
||||
<details><summary>What this tab asked, and what came back</summary><pre id="wire"></pre></details>
|
||||
</main>
|
||||
|
||||
<script>
|
||||
const CFG = window.HAMLAB || {}
|
||||
const $ = (id) => document.getElementById(id)
|
||||
const h = (tag, attrs = {}, ...kids) => { const e = document.createElementNS(tag === 'svg' || attrs.svg ? 'http://www.w3.org/2000/svg' : 'http://www.w3.org/1999/xhtml', tag); for (const [k, v] of Object.entries(attrs)) if (k !== 'svg' && v != null) k === 'class' ? e.setAttribute('class', v) : k === 'style' ? e.setAttribute('style', v) : e.setAttribute(k, v); for (const kid of kids.flat()) if (kid != null) e.append(kid); return e }
|
||||
|
||||
// ---- the API: two calls ----
|
||||
async function api(path, body) {
|
||||
const r = await fetch(CFG.base + path, { method: body ? 'POST' : 'GET', headers: { Authorization: 'Bearer ' + CFG.token, ...(body ? { 'Content-Type': 'application/json' } : {}) }, body: body ? JSON.stringify(body) : undefined })
|
||||
if (!r.ok) throw new Error(r.status === 401 ? 'token rejected (401)' : `${r.status} ${r.statusText}`)
|
||||
return r.json()
|
||||
}
|
||||
|
||||
// ---- formatting from the sensor's own unit ----
|
||||
const U = ['B', 'KiB', 'MiB', 'GiB', 'TiB']
|
||||
const bytes = (v) => { let i = 0; while (v >= 1024 && i < U.length - 1) { v /= 1024; i++ } return `${v >= 100 || i === 0 ? v.toFixed(0) : v.toFixed(1)} ${U[i]}` }
|
||||
const dur = (s) => { const d = Math.floor(s / 86400), hh = Math.floor(s % 86400 / 3600), m = Math.floor(s % 3600 / 60); return d ? `${d}d ${hh}h` : hh ? `${hh}h ${m}m` : `${m}m` }
|
||||
function fmt(s) {
|
||||
if (!s || s.value == null) return '—'
|
||||
const v = s.value
|
||||
if (typeof v === 'boolean') return v ? 'yes' : 'no'
|
||||
if (typeof v === 'string') return v || '—'
|
||||
switch (s.unit) { case 'B': return bytes(v); case 'B/s': return bytes(v) + '/s'; case 's': return dur(v); case 'min': return dur(v * 60); case '%': return v.toFixed(v < 10 ? 1 : 0) + '%'; case '°C': return v.toFixed(0) + '°C'; case 'rpm': return Math.round(v) + ' rpm'; case undefined: case '': return String(Math.round(v * 100) / 100); default: return `${Math.round(v * 10) / 10} ${s.unit}` }
|
||||
}
|
||||
const frac = (s) => { if (!s || typeof s.value !== 'number') return null; const max = s.max ?? s.crit ?? s.warn, min = s.min ?? 0; return max === undefined || max <= min ? null : Math.min(1, Math.max(0, (s.value - min) / (max - min))) }
|
||||
const sev = (s) => !s || typeof s.value !== 'number' ? '' : s.crit !== undefined && s.value >= s.crit ? 'bad' : s.warn !== undefined && s.value >= s.warn ? 'warn' : s.kind === 'usage' && s.id !== 'ups.battery' && s.value >= 95 ? 'bad' : s.kind === 'usage' && s.id !== 'ups.battery' && s.value >= 85 ? 'warn' : ''
|
||||
|
||||
// ---- the widgets ----
|
||||
const tile = (label, id, ...body) => h('div', { class: 'tile' }, h('div', { class: 'label' }, label), ...body, id ? h('div', { class: 'id mono' }, id) : null)
|
||||
const missing = (label, id) => tile(label, id, h('div', { class: 'mut' }, 'not on this machine'))
|
||||
const W = {
|
||||
stat: (s, label, id) => s ? tile(label || s.name, s.id, h('div', { class: 'value ' + sev(s) }, fmt(s))) : missing(label, id),
|
||||
bar: (s, label, id) => { if (!s) return missing(label, id); const f = frac(s); return tile(label || s.name, s.id, h('div', { class: 'value ' + sev(s) }, fmt(s)), f === null ? null : h('div', { class: 'meter' }, h('i', { class: sev(s), style: `width:${f * 100}%` }))) },
|
||||
gauge: (s, label, id) => {
|
||||
if (!s) return missing(label, id)
|
||||
const f = frac(s) ?? 0, C = Math.PI * 40, col = { bad: 'var(--bad)', warn: 'var(--s2)' }[sev(s)] || 'var(--s1)', arc = 'M10 50 A40 40 0 0 1 90 50'
|
||||
return tile(label || s.name, s.id, h('svg', { class: 'gauge', viewBox: '0 0 100 56' },
|
||||
h('path', { svg: 1, d: arc, fill: 'none', stroke: 'var(--track)', 'stroke-width': 9, 'stroke-linecap': 'round' }),
|
||||
h('path', { svg: 1, d: arc, fill: 'none', stroke: col, 'stroke-width': 9, 'stroke-linecap': 'round', 'stroke-dasharray': `${f * C} ${C}` }),
|
||||
h('text', { svg: 1, x: 50, y: 47, 'text-anchor': 'middle' }, fmt(s))))
|
||||
},
|
||||
spark: (s, label, id) => {
|
||||
if (!s) return missing(label, id)
|
||||
const pts = (hist[s.id] ||= []); if (typeof s.value === 'number') { pts.push(s.value); if (pts.length > 300) pts.shift() }
|
||||
const max = s.unit === '%' ? 100 : Math.max(...pts, 1e-9) * 1.1, n = pts.length
|
||||
const line = n < 2 ? '' : pts.map((v, i) => `${(i / (n - 1) * 300).toFixed(1)},${(42 - v / max * 40).toFixed(1)}`).join(' ')
|
||||
return tile(label || s.name, s.id, h('div', { class: 'value' }, fmt(s)), h('svg', { class: 'spark', viewBox: '0 0 300 44', preserveAspectRatio: 'none' }, h('polyline', { svg: 1, points: line, fill: 'none', stroke: 'var(--s1)', 'stroke-width': 2, 'vector-effect': 'non-scaling-stroke' })))
|
||||
},
|
||||
// a glob comes back as {id: sensor}: a list draws them all
|
||||
list: (map, label, pattern, opts = {}) => {
|
||||
const rows = Object.values(map || {}).filter((s) => !opts.filter || opts.filter(s)).sort((a, b) => a.id.localeCompare(b.id))
|
||||
return h('div', { class: 'tile wide' }, h('div', { class: 'label' }, `${label} · ${rows.length}`),
|
||||
rows.length ? h('div', { class: 'rows' }, rows.map((s) => { const f = frac(s); return h('div', { class: 'row' }, h('span', { class: 'k', title: s.id }, opts.name ? opts.name(s) : s.name), typeof s.value === 'boolean' ? h('span', { class: 'pill ' + (s.value ? 'ok' : 'bad') }, s.value ? 'up' : 'down') : h('span', { class: 'n' }, fmt(s)), f === null ? null : h('div', { class: 'meter' }, h('i', { class: sev(s), style: `width:${f * 100}%` }))) })) : h('div', { class: 'mut' }, 'nothing here that this token may see'),
|
||||
h('div', { class: 'id mono' }, pattern))
|
||||
},
|
||||
}
|
||||
|
||||
// ---- the tabs: a query, the histories to seed, and how to draw the answer ----
|
||||
const TABS = [
|
||||
{ name: 'Overview', query: { cpu: 'cpu.usage', mem: 'mem.used_pct', memB: 'mem.used', root: 'fs.root.used_pct', ups: 'ups.battery', temp: 'cpu.temp', load: 'load.1', up: 'uptime', failed: 'services.failed', upd: 'updates.pending', cont: 'containers.running' },
|
||||
seed: 'cpu.usage,load.1',
|
||||
draw: (d) => [W.gauge(d.cpu, 'CPU'), W.gauge(d.mem, 'Memory'), W.gauge(d.root, 'Root disk'), W.gauge(d.ups, 'UPS battery', 'ups.battery'), W.spark(d.cpu, 'CPU, recent'), W.spark(d.load, 'Load 1 min'), W.bar(d.memB, 'Memory used'), W.stat(d.temp, 'CPU temp', 'cpu.temp'), W.stat(d.up, 'Uptime'), W.stat(d.cont, 'Containers up', 'containers.running'), W.stat(d.failed, 'Failed services'), W.stat(d.upd, 'Pending updates')] },
|
||||
{ name: 'Widget types', query: { s: 'cpu.usage', cores: 'cpu.core.*.usage' }, seed: 'cpu.usage',
|
||||
draw: (d) => [W.stat(d.s, 'stat — a number'), W.gauge(d.s, 'gauge — has a scale'), W.bar(d.s, 'bar — has a scale'), W.spark(d.s, 'spark — has a past'), W.list(d.cores, 'list — a pattern', 'cpu.core.*.usage')] },
|
||||
{ name: 'Hardware', query: { cores: 'cpu.core.*.usage', temps: 'temp.*', fans: 'fan.*', smart: 'smart.*.passed', wear: 'smart.*.wear' },
|
||||
draw: (d) => [W.list(d.cores, 'CPU cores', 'cpu.core.*.usage'), W.list(d.temps, 'Temperatures', 'temp.*'), W.list(d.fans, 'Fans', 'fan.*'), W.list(d.smart, 'Drive health', 'smart.*.passed', { name: (s) => s.labels?.device + ' · ' + (s.labels?.model || '') }), W.list(d.wear, 'SSD endurance used', 'smart.*.wear')] },
|
||||
{ name: 'Network & disks', query: { rx: 'net.*.rx', tx: 'net.*.tx', rd: 'disk.*.read', wr: 'disk.*.write', fs: 'fs.*.used_pct', fsb: 'fs.*.used' }, seed: 'net.*.rx,net.*.tx,disk.*.read,disk.*.write',
|
||||
draw: (d) => [...Object.values(d.rx).map((s) => W.spark(s)), ...Object.values(d.tx).map((s) => W.spark(s)), ...Object.values(d.rd).map((s) => W.spark(s)), ...Object.values(d.wr).map((s) => W.spark(s)), W.list(d.fs, 'Filesystems', 'fs.*.used_pct', { name: (s) => s.labels?.mount }), W.list(d.fsb, 'Filesystems, bytes', 'fs.*.used', { name: (s) => s.labels?.mount })] },
|
||||
{ name: 'Containers', query: { total: 'containers.total', run: 'containers.running', bad: 'containers.unhealthy', up: 'container.*.running', mem: 'container.*.mem', cpu: 'container.*.cpu' },
|
||||
draw: (d) => [W.stat(d.run, 'Running'), W.stat(d.total, 'Total'), W.stat(d.bad, 'Unhealthy'), W.list(d.up, 'Containers', 'container.*.running', { name: (s) => s.labels?.container }), W.list(d.mem, 'Memory', 'container.*.mem', { name: (s) => s.labels?.container }), W.list(d.cpu, 'CPU, % of one core', 'container.*.cpu', { name: (s) => s.labels?.container })] },
|
||||
{ name: 'Services', query: { failed: 'services.failed', names: 'services.failed_names', running: 'services.running', reboot: 'updates.reboot_required', each: 'service.*.active' },
|
||||
draw: (d) => [W.stat(d.failed, 'Failed'), W.stat(d.running, 'Running'), W.stat(d.names, 'Which failed'), W.stat(d.reboot, 'Reboot required'), W.list(d.each, 'Every service', 'service.*.active')] },
|
||||
{ name: 'Fleet', fleet: true, query: { cpu: 'cpu.usage', mem: 'mem.used_pct', temp: 'cpu.temp', up: 'uptime', failed: 'services.failed', cont: 'containers.running', pool: 'zfs.*.health' },
|
||||
draw: (d) => [h('div', { class: 'tile wide' }, h('div', { class: 'label' }, 'one request, every machine'), h('table', {}, h('tr', {}, ['Machine', 'CPU', 'Memory', 'CPU temp', 'Uptime', 'Failed', 'Containers', 'Pools'].map((t) => h('th', {}, t))),
|
||||
Object.entries(d).sort().map(([m, r]) => r.error ? h('tr', {}, h('td', {}, m), h('td', { colspan: 7, class: 'mut' }, r.error)) : h('tr', {}, h('td', {}, m), ...[r.cpu, r.mem, r.temp, r.up, r.failed, r.cont].map((s) => h('td', {}, fmt(s))), h('td', {}, Object.values(r.pool || {}).map((s) => `${s.labels?.pool} ${s.value}`).join(', ') || '—')))), h('div', { class: 'id mono' }, 'POST /api/fleet/sensors/query'))] },
|
||||
]
|
||||
|
||||
// ---- the loop ----
|
||||
const hist = {}
|
||||
let tab = 0, timer = 0, seq = 0
|
||||
async function show(i) {
|
||||
tab = i; const my = ++seq; clearTimeout(timer)
|
||||
;[...$('tabs').children].forEach((b, j) => b.classList.toggle('on', j === i))
|
||||
$('view').replaceChildren(h('div', { class: 'mut' }, 'asking…'))
|
||||
const t = TABS[i]
|
||||
if (t.seed) try { for (const s of (await api('/api/sensors/history?ids=' + encodeURIComponent(t.seed))).series) hist[s.id] = s.v.slice(-300) } catch { /* sparklines just start empty */ }
|
||||
if (my === seq) tick(my)
|
||||
}
|
||||
async function tick(my) {
|
||||
if (my !== seq) return
|
||||
const t = TABS[tab], t0 = performance.now()
|
||||
try {
|
||||
const path = (t.fleet ? '/api/fleet/sensors/query' : '/api/sensors/query') + '?full=1', body = t.fleet ? { machines: '*', query: t.query } : t.query
|
||||
const data = await api(path, body)
|
||||
if (my !== seq) return
|
||||
$('view').replaceChildren(h('div', { class: 'grid' }, t.draw(data)))
|
||||
$('wire').textContent = `POST ${path}\n${JSON.stringify(body, null, 1)}\n\n→ ${JSON.stringify(data, (k, v) => (v && typeof v === 'object' && 'id' in v && 'value' in v ? { id: v.id, value: v.value, unit: v.unit } : v), 1).slice(0, 6000)}`
|
||||
$('dot').className = 'dot ok'; $('status').textContent = `${Math.round(performance.now() - t0)} ms · ${new Date().toLocaleTimeString()}`
|
||||
} catch (e) {
|
||||
if (my !== seq) return
|
||||
$('dot').className = 'dot bad'; $('status').textContent = e.message
|
||||
if (!$('view').querySelector('.grid')) $('view').replaceChildren(h('div', { class: 'err' }, e.message))
|
||||
}
|
||||
timer = setTimeout(() => tick(my), document.hidden ? 5000 : Number($('every').value))
|
||||
}
|
||||
|
||||
if (!CFG.base || !CFG.token) $('view').replaceChildren(h('div', { class: 'err' }, 'config.js is missing: window.HAMLAB = { base, token }'))
|
||||
else {
|
||||
$('base').textContent = CFG.base
|
||||
TABS.forEach((t, i) => $('tabs').append(h('button', {}, t.name))); [...$('tabs').children].forEach((b, i) => b.onclick = () => show(i))
|
||||
$('every').onchange = () => { clearTimeout(timer); tick(seq) }
|
||||
show(0)
|
||||
}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Executable
+382
@@ -0,0 +1,382 @@
|
||||
#!/bin/sh
|
||||
# Ham Lab installer.
|
||||
#
|
||||
# curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh
|
||||
#
|
||||
# With a terminal it walks you through it: who the console runs as, where it listens,
|
||||
# HTTPS, the firewall, the password — then downloads the build for this machine, checks
|
||||
# it against SHA256SUMS and hands over to `hamlab install`, which sets up the binary,
|
||||
# the config and a service that starts at boot. Re-running it upgrades in place.
|
||||
#
|
||||
# Without a terminal, with --yes, or with flags after `--`, it asks nothing:
|
||||
#
|
||||
# curl -fsSL …/install.sh | sh -s -- --yes
|
||||
# curl -fsSL …/install.sh | sh -s -- --listen 0.0.0.0:8100 --tls auto --password-stdin <pw
|
||||
# ./install.sh # beside the hamlab-linux-* files: installs offline
|
||||
#
|
||||
# HAMLAB_URL where hamlab-linux-<arch>, SHA256SUMS and VERSION are served from —
|
||||
# by default the newest release on Gitea; "latest" is an alias Gitea
|
||||
# resolves to the most recent release, so the URL never needs changing —
|
||||
# any static file host, or a Gitea/Forgejo release's download URL
|
||||
# HAMLAB_TOKEN access token, when that place is a private Gitea/Forgejo repo
|
||||
# NO_COLOR plain output
|
||||
#
|
||||
# POSIX sh: runs under dash, busybox ash and bash alike.
|
||||
set -eu
|
||||
|
||||
HAMLAB_URL="${HAMLAB_URL:-https://git.bigham.dev/mitchell/hamlab/releases/download/latest}"
|
||||
BUILT_FOR="@VERSION@" # filled in by `make release`
|
||||
|
||||
# ─────────────────────────────────────────────────────────────── looks ──
|
||||
if [ -t 2 ] && [ -z "${NO_COLOR:-}" ] && [ "${TERM:-dumb}" != dumb ]; then
|
||||
B=$(printf '\033[1m'); D=$(printf '\033[2m'); R=$(printf '\033[0m')
|
||||
AMBER=$(printf '\033[38;5;214m'); GREEN=$(printf '\033[32m'); RED=$(printf '\033[31m'); CYAN=$(printf '\033[36m')
|
||||
else
|
||||
B=; D=; R=; AMBER=; GREEN=; RED=; CYAN=
|
||||
fi
|
||||
case "${LC_ALL:-${LC_CTYPE:-${LANG:-}}}" in
|
||||
*UTF-8*|*utf8*|*UTF8*|*utf-8*) OK='✓'; NO='✗'; DOT='•'; ARROW='›'; HR='────────────────────────────────────────────────────────────' ;;
|
||||
*) OK='ok'; NO='x'; DOT='*'; ARROW='>'; HR='------------------------------------------------------------' ;;
|
||||
esac
|
||||
|
||||
out() { printf '%s\n' "$*" >&2; }
|
||||
blank() { printf '\n' >&2; }
|
||||
good() { out " ${GREEN}${OK}${R} $*"; }
|
||||
note() { out " ${D}$*${R}"; }
|
||||
warn() { out " ${AMBER}!${R} $*"; }
|
||||
die() { blank; out " ${RED}${NO}${R} ${B}$1${R}"; shift; for more in "$@"; do out " $more"; done; blank; exit 1; }
|
||||
rule() { out "${D}${HR}${R}"; }
|
||||
step() { blank; out "${AMBER}${B}$1${R} ${B}$2${R}"; [ -z "${3:-}" ] || note "$3"; blank; }
|
||||
|
||||
banner() {
|
||||
blank
|
||||
out " ${AMBER}${B}Ham Lab${R} ${D}a web console for the machines you run${R}"
|
||||
out " ${D}terminals ${DOT} monitors ${DOT} services ${DOT} files ${DOT} updates ${DOT} disks — in one tab${R}"
|
||||
rule
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────── questions ──
|
||||
# Piped from curl, stdin is this script, so every answer comes from the terminal itself.
|
||||
TTY=/dev/tty
|
||||
have_tty() { [ -r "$TTY" ] && [ -w "$TTY" ] && (: <"$TTY") 2>/dev/null; }
|
||||
|
||||
ask() { # ask "Question" "default" → $REPLY
|
||||
if [ -n "${2:-}" ]; then printf ' %s %s[%s]%s ' "$1" "$D" "$2" "$R" >&2; else printf ' %s ' "$1" >&2; fi
|
||||
IFS= read -r REPLY <"$TTY" || REPLY=
|
||||
[ -n "$REPLY" ] || REPLY="${2:-}"
|
||||
}
|
||||
confirm() { # confirm "Question" y|n
|
||||
if [ "$2" = y ]; then hint="Y/n"; else hint="y/N"; fi
|
||||
while :; do
|
||||
ask "$1" "$hint"
|
||||
case "$REPLY" in
|
||||
"$hint") [ "$2" = y ]; return ;;
|
||||
y|Y|yes|Yes|YES) return 0 ;;
|
||||
n|N|no|No|NO) return 1 ;;
|
||||
*) warn "y or n, please" ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
secret() { # secret "Prompt" → $REPLY, typed unseen
|
||||
printf ' %s ' "$1" >&2
|
||||
saved=$(stty -g <"$TTY" 2>/dev/null || true)
|
||||
stty -echo <"$TTY" 2>/dev/null || true
|
||||
IFS= read -r REPLY <"$TTY" || REPLY=
|
||||
[ -z "$saved" ] || stty "$saved" <"$TTY" 2>/dev/null || true
|
||||
printf '\n' >&2
|
||||
}
|
||||
choose() { # choose "default-number" "label 1" "label 2" … → $REPLY = the number picked
|
||||
def=$1; shift; n=0
|
||||
for label in "$@"; do n=$((n + 1)); if [ "$n" = "$def" ]; then out " ${AMBER}${B}$n${R} $label"; else out " ${B}$n${R} $label"; fi; done
|
||||
blank
|
||||
while :; do
|
||||
ask "Choose 1-$n" "$def"
|
||||
case "$REPLY" in *[!0-9]*|'') ;; *) if [ "$REPLY" -ge 1 ] && [ "$REPLY" -le "$n" ]; then return; fi ;; esac
|
||||
warn "a number from 1 to $n, please"
|
||||
done
|
||||
}
|
||||
restore_tty() { [ -z "${saved:-}" ] || stty "$saved" <"$TTY" 2>/dev/null || true; }
|
||||
|
||||
# ──────────────────────────────────────────────────────────── plumbing ──
|
||||
fetch() { # url dest
|
||||
if command -v curl >/dev/null 2>&1; then
|
||||
curl -fsSL ${HAMLAB_TOKEN:+-H "Authorization: token $HAMLAB_TOKEN"} -o "$2" "$1"
|
||||
elif command -v wget >/dev/null 2>&1; then
|
||||
wget -q ${HAMLAB_TOKEN:+--header="Authorization: token $HAMLAB_TOKEN"} -O "$2" "$1"
|
||||
else
|
||||
die "This needs curl or wget, and the machine has neither."
|
||||
fi
|
||||
}
|
||||
port_busy() { # is something already listening on TCP port $1?
|
||||
hex=$(printf '%04X' "$1")
|
||||
cat /proc/net/tcp /proc/net/tcp6 2>/dev/null | awk -v p=":$hex" '$4 == "0A" && substr($2, length($2) - 4) == p { found = 1 } END { exit !found }'
|
||||
}
|
||||
lan_addresses() { # this machine's IPv4 addresses, one per line, most useful first
|
||||
if command -v ip >/dev/null 2>&1; then
|
||||
ip -4 -o addr show scope global 2>/dev/null | awk '$2 !~ /^(docker|br-|veth|virbr|cni|flannel|tailscale|wg|tun|tap)/ { split($4, a, "/"); print a[1] " " $2 }'
|
||||
else
|
||||
hostname -I 2>/dev/null | tr ' ' '\n' | awk 'NF && $1 !~ /:/ { print $1 " -" }'
|
||||
fi
|
||||
}
|
||||
|
||||
# ────────────────────────────────────────────────────────── arguments ──
|
||||
YES=0; PASS=
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-y|--yes) YES=1; shift ;;
|
||||
-h|--help) sed -n '2,20p' "$0" 2>/dev/null | sed 's/^# \{0,1\}//' >&2; exit 0 ;;
|
||||
--) shift; PASS="$*"; break ;;
|
||||
*) PASS="$*"; break ;; # flags given straight: they belong to `hamlab install`
|
||||
esac
|
||||
done
|
||||
INTERACTIVE=1
|
||||
{ [ "$YES" = 1 ] || [ -n "$PASS" ] || ! have_tty; } && INTERACTIVE=0
|
||||
|
||||
# ─────────────────────────────────────────────────────────── preflight ──
|
||||
banner
|
||||
[ "$(uname -s)" = Linux ] || die "Ham Lab runs on Linux; this is $(uname -s)."
|
||||
case "$(uname -m)" in
|
||||
x86_64|amd64) arch=amd64 ;;
|
||||
aarch64|arm64) arch=arm64 ;;
|
||||
armv6l|armv7l|armv8l|arm) arch=arm ;;
|
||||
i386|i486|i586|i686) arch=386 ;;
|
||||
riscv64) arch=riscv64 ;;
|
||||
*) die "There is no build for $(uname -m). (There are: amd64, arm64, arm, 386, riscv64.)" ;;
|
||||
esac
|
||||
file="hamlab-linux-$arch"
|
||||
|
||||
me=$(id -un 2>/dev/null || echo root)
|
||||
if [ "$(id -u)" = 0 ]; then how=root
|
||||
elif command -v sudo >/dev/null 2>&1; then how=sudo
|
||||
elif command -v doas >/dev/null 2>&1; then how=doas
|
||||
else die "Installing a service needs root, and there is no sudo or doas here. Run this as root."
|
||||
fi
|
||||
if [ -d /run/systemd/system ]; then init=systemd
|
||||
elif command -v rc-service >/dev/null 2>&1; then init=OpenRC
|
||||
else init=none
|
||||
fi
|
||||
os=$(. /etc/os-release 2>/dev/null && printf '%s' "${PRETTY_NAME:-}"); [ -n "$os" ] || os=Linux
|
||||
existing=$(command -v hamlab 2>/dev/null || true); [ -n "$existing" ] || { [ -x /usr/local/bin/hamlab ] && existing=/usr/local/bin/hamlab; } || true
|
||||
have=; [ -z "$existing" ] || have=$("$existing" version 2>/dev/null | awk '{ print $2 }') || true
|
||||
|
||||
good "$os ${D}${DOT}${R} $(uname -m) ${D}${DOT}${R} $init"
|
||||
case "$how" in root) good "running as root" ;; *) good "running as ${B}$me${R}; $how is asked for root when it is needed" ;; esac
|
||||
[ "$init" != none ] || warn "no systemd or OpenRC: it will be installed, but nothing starts it at boot"
|
||||
|
||||
# ───────────────────────────────────────────────────── which version ──
|
||||
tmp=$(mktemp -d 2>/dev/null || mktemp -d -t hamlab)
|
||||
alt=
|
||||
cleanup() { restore_tty; rm -rf "$tmp"; [ -z "$alt" ] || rm -f "$alt"; }
|
||||
trap cleanup EXIT
|
||||
trap 'blank; out " cancelled — nothing was changed"; exit 130' INT TERM
|
||||
|
||||
here=$(cd "$(dirname "$0")" 2>/dev/null && pwd || echo .)
|
||||
offline=0; [ -f "$here/$file" ] && offline=1
|
||||
if [ "$offline" = 1 ]; then
|
||||
ver=$BUILT_FOR; case "$ver" in @*) ver="(local files)" ;; esac
|
||||
else
|
||||
# What is published right now — which may be newer than the copy of this script being run.
|
||||
ver=$(fetch "$HAMLAB_URL/VERSION" "$tmp/VERSION" 2>/dev/null && head -n 1 "$tmp/VERSION" | tr -cd 'A-Za-z0-9._+-' || true)
|
||||
[ -n "$ver" ] || { ver=$BUILT_FOR; case "$ver" in @*) ver=latest ;; esac; }
|
||||
fi
|
||||
# v0.1.0-69-g9c4f9df → 0001000069: versions compare as numbers, builds since the tag included.
|
||||
vnum() { printf '%s' "$1" | sed 's/^v//; s/-g[0-9a-f]*$//' | awk -F'[.-]' '{ printf "%d%03d%03d%05d", $1, $2, $3, $4 }'; }
|
||||
older=0
|
||||
if [ -n "$have" ] && [ "$have" != "$ver" ]; then
|
||||
a=$(vnum "$have" 2>/dev/null || echo 0); b=$(vnum "$ver" 2>/dev/null || echo 0)
|
||||
case "$a$b" in *[!0-9]*) ;; *) [ "$b" -ge "$a" ] || older=1 ;; esac
|
||||
fi
|
||||
if [ "$older" = 1 ]; then
|
||||
warn "Ham Lab ${B}$have${R} is installed, and what is published is ${B}older${R}: $ver"
|
||||
note " installing it would take this machine backwards"
|
||||
if [ "$INTERACTIVE" = 1 ]; then
|
||||
blank; confirm "Go back to $ver anyway?" n || { blank; out " nothing was changed"; exit 0; }
|
||||
elif [ -z "${HAMLAB_ALLOW_DOWNGRADE:-}" ]; then
|
||||
die "Not downgrading $have to $ver." "Set HAMLAB_ALLOW_DOWNGRADE=1 if that is really what you want."
|
||||
fi
|
||||
have_msg_done=1
|
||||
fi
|
||||
if [ -n "${have_msg_done:-}" ]; then :
|
||||
elif [ -n "$have" ] && [ "$have" = "$ver" ]; then good "Ham Lab ${B}$have${R} is already installed ${D}${ARROW}${R} this reinstalls it"; note " its config, user and password are kept"
|
||||
elif [ -n "$have" ]; then good "Ham Lab ${B}$have${R} is installed ${D}${ARROW}${R} upgrading to ${B}$ver${R}"; note " its config, user and password are kept"
|
||||
else good "installing Ham Lab ${B}$ver${R}"; fi
|
||||
|
||||
# ─────────────────────────────────────────────────────────── the wizard ──
|
||||
flags=; password=; listen=; url_host=
|
||||
if [ "$INTERACTIVE" = 1 ] && [ -n "$have" ] && [ "$older" = 1 ]; then :
|
||||
elif [ "$INTERACTIVE" = 1 ] && [ -n "$have" ]; then
|
||||
blank
|
||||
confirm "$([ "$have" = "$ver" ] && echo Reinstall || echo Upgrade) now?" y || { blank; out " nothing was changed"; exit 0; }
|
||||
elif [ "$INTERACTIVE" = 1 ]; then
|
||||
# 1 ─ who
|
||||
default_user=${SUDO_USER:-${DOAS_USER:-$me}}
|
||||
step "1/5" "Who should it run as?" "Its terminals are that user's shell; its files are that user's files."
|
||||
while :; do
|
||||
ask "User" "$default_user"; run_as=$REPLY
|
||||
id "$run_as" >/dev/null 2>&1 && break
|
||||
warn "there is no user called \"$run_as\" on this machine"
|
||||
done
|
||||
[ "$run_as" != root ] || warn "as root, every terminal it opens is a root shell"
|
||||
|
||||
# 2 ─ where
|
||||
step "2/5" "Where should it listen?" "Who can reach the console at all. The password guards it either way."
|
||||
addrs=$(lan_addresses || true); first=$(printf '%s\n' "$addrs" | awk 'NR == 1 { print $1 }')
|
||||
if [ -n "$first" ]; then
|
||||
first_if=$(printf '%s\n' "$addrs" | awk 'NR == 1 { print $2 }')
|
||||
choose 2 "This machine only ${D}127.0.0.1 — for an SSH tunnel or a reverse proxy${R}" \
|
||||
"This network ${D}$first ($first_if) — reachable from the LAN${R}" \
|
||||
"Every interface ${D}0.0.0.0 — every address it has or gets${R}" \
|
||||
"Another address"
|
||||
else
|
||||
choose 1 "This machine only ${D}127.0.0.1${R}" "Every interface ${D}0.0.0.0${R}" "Another address"
|
||||
[ "$REPLY" = 1 ] || REPLY=$((REPLY + 1))
|
||||
fi
|
||||
case "$REPLY" in
|
||||
1) host=127.0.0.1 ;;
|
||||
2) host=$first ;;
|
||||
3) host=0.0.0.0 ;;
|
||||
4) others=$(printf '%s\n' "$addrs" | awk 'NR > 1 { printf "%s%s (%s)", sep, $1, $2; sep = ", " }'); [ -z "$others" ] || note "also on this machine: $others"
|
||||
while :; do ask "Address" ""; host=$REPLY; case "$host" in ''|*[!0-9a-fA-F.:]*) warn "an IPv4 or IPv6 address, please" ;; *) break ;; esac; done ;;
|
||||
esac
|
||||
while :; do
|
||||
ask "Port" "8100"; port=$REPLY
|
||||
case "$port" in ''|*[!0-9]*) warn "a number, please"; continue ;; esac
|
||||
[ "$port" -ge 1 ] && [ "$port" -le 65535 ] || { warn "a port is between 1 and 65535"; continue; }
|
||||
[ "$port" -ge 1024 ] || [ "$run_as" = root ] || { warn "ports under 1024 need root, and it will run as $run_as"; continue; }
|
||||
if port_busy "$port"; then warn "something is already listening on port $port"; continue; fi
|
||||
break
|
||||
done
|
||||
case "$host" in *:*) listen="[$host]:$port" ;; *) listen="$host:$port" ;; esac
|
||||
|
||||
# 3 ─ https
|
||||
step "3/5" "HTTPS?" "With a certificate made here: browsers warn once, a hub pins it."
|
||||
if [ "$host" = 127.0.0.1 ]; then
|
||||
choose 2 "HTTPS, self-signed" "Plain HTTP ${D}— fine here: nothing leaves this machine${R}"
|
||||
else
|
||||
choose 1 "HTTPS, self-signed ${D}— recommended: the password crosses the network${R}" "Plain HTTP ${D}— only behind a reverse proxy that adds HTTPS${R}"
|
||||
fi
|
||||
if [ "$REPLY" = 1 ]; then tls=auto; scheme=https; else tls=off; scheme=http; fi
|
||||
|
||||
# 4 ─ firewall
|
||||
open_fw=0; fw=
|
||||
if [ "$host" != 127.0.0.1 ]; then
|
||||
if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then fw=firewalld
|
||||
elif command -v ufw >/dev/null 2>&1 && ufw status 2>/dev/null | grep -q "^Status: active"; then fw=ufw
|
||||
elif [ "$how" != root ] && command -v ufw >/dev/null 2>&1 && systemctl is-active -q ufw 2>/dev/null; then fw=ufw
|
||||
fi
|
||||
fi
|
||||
step "4/5" "Firewall"
|
||||
if [ -n "$fw" ]; then
|
||||
note "$fw is active and will block port $port until it is opened."
|
||||
blank
|
||||
if confirm "Open $port/tcp in $fw?" y; then open_fw=1; fi
|
||||
elif [ "$host" = 127.0.0.1 ]; then good "nothing to open: it only listens on this machine"
|
||||
else good "no active firewalld or ufw found; nothing to open"
|
||||
fi
|
||||
|
||||
# 5 ─ password
|
||||
step "5/5" "The password" "One password opens the console, and the console is a shell here."
|
||||
choose 1 "Choose one now" "Generate a strong one ${D}— shown once at the end${R}"
|
||||
if [ "$REPLY" = 1 ]; then
|
||||
while :; do
|
||||
secret "Password:"; password=$REPLY
|
||||
[ ${#password} -ge 12 ] || { warn "at least 12 characters, please (it guards a shell)"; continue; }
|
||||
secret "Once more:"
|
||||
[ "$REPLY" = "$password" ] && break
|
||||
warn "those didn't match; try again"
|
||||
done
|
||||
fi
|
||||
|
||||
# summary
|
||||
if [ "$host" = 0.0.0.0 ] || [ "$host" = "::" ]; then url_host=${first:-this-machine}; else url_host=$host; fi
|
||||
blank; rule
|
||||
out " ${B}Ready to install${R}"
|
||||
blank
|
||||
out " version ${B}$ver${R} ${D}($file)${R}"
|
||||
out " runs as ${B}$run_as${R}"
|
||||
out " address ${B}$scheme://$listen${R}"
|
||||
if [ "$open_fw" = 1 ]; then out " firewall open $port/tcp in $fw"; fi
|
||||
if [ -n "$password" ]; then out " password the one you typed"; else out " password generated, shown once"; fi
|
||||
case "$init" in none) out " service ${AMBER}none — no systemd or OpenRC${R}" ;; *) out " service $init, starts at boot" ;; esac
|
||||
rule; blank
|
||||
confirm "Install?" y || { blank; out " nothing was changed"; exit 0; }
|
||||
|
||||
flags="--user $run_as --listen $listen --tls $tls"
|
||||
[ "$open_fw" = 0 ] || flags="$flags --open-firewall"
|
||||
[ -z "$password" ] || flags="$flags --password-stdin"
|
||||
fi
|
||||
|
||||
# ──────────────────────────────────────────────── download and verify ──
|
||||
blank
|
||||
if [ "$offline" = 1 ]; then
|
||||
cp "$here/$file" "$tmp/hamlab"; [ ! -f "$here/SHA256SUMS" ] || cp "$here/SHA256SUMS" "$tmp/SHA256SUMS"
|
||||
good "using $file from $here"
|
||||
else
|
||||
printf ' %s downloading %s …' "$DOT" "$file" >&2
|
||||
fetch "$HAMLAB_URL/$file" "$tmp/hamlab" || { printf '\n' >&2; die "Download failed: $HAMLAB_URL/$file"; }
|
||||
size=$(wc -c <"$tmp/hamlab" | awk '{ printf "%.1f MB", $1 / 1048576 }')
|
||||
if [ -t 2 ]; then printf '\r\033[K' >&2; else printf '\n' >&2; fi
|
||||
good "downloaded $file ${D}($size)${R}"
|
||||
fetch "$HAMLAB_URL/SHA256SUMS" "$tmp/SHA256SUMS" 2>/dev/null || rm -f "$tmp/SHA256SUMS"
|
||||
fi
|
||||
if [ -f "$tmp/SHA256SUMS" ] && command -v sha256sum >/dev/null 2>&1; then
|
||||
want=$(awk -v f="$file" '$2 == f || $2 == "*" f { print $1 }' "$tmp/SHA256SUMS")
|
||||
got=$(sha256sum "$tmp/hamlab" | awk '{ print $1 }')
|
||||
[ -n "$want" ] || die "SHA256SUMS has no entry for $file."
|
||||
[ "$want" = "$got" ] || die "Checksum mismatch for $file." "This is not the file that was published. Nothing was installed."
|
||||
good "checksum matches SHA256SUMS"
|
||||
else
|
||||
warn "no SHA256SUMS (or no sha256sum here): the download was not verified"
|
||||
fi
|
||||
|
||||
chmod 0755 "$tmp/hamlab"; bin="$tmp/hamlab"
|
||||
if ! "$bin" version >/dev/null 2>&1; then
|
||||
alt="${HOME:-/root}/.hamlab-install.$$" # a noexec /tmp is the usual reason: try from the home directory
|
||||
cp "$bin" "$alt" 2>/dev/null && chmod 0755 "$alt" || true
|
||||
"$alt" version >/dev/null 2>&1 || die "The $arch build doesn't run on this machine."
|
||||
bin="$alt"
|
||||
fi
|
||||
|
||||
# ───────────────────────────────────────────────────────────── install ──
|
||||
blank; out " ${B}Installing${R}"; blank
|
||||
log="$tmp/install.log"
|
||||
set +e
|
||||
# shellcheck disable=SC2086 # $flags and $PASS are lists of words on purpose
|
||||
if [ -n "$password" ]; then
|
||||
printf '%s\n' "$password" | "$bin" install $flags >"$log" 2>&1; rc=$?
|
||||
password=
|
||||
elif [ "$INTERACTIVE" = 0 ] && [ -n "$PASS" ]; then
|
||||
log=
|
||||
case " $PASS " in
|
||||
*password-stdin*) "$bin" install $PASS; rc=$? ;;
|
||||
*) if [ ! -t 0 ] && have_tty; then "$bin" install $PASS <"$TTY"; rc=$?; else "$bin" install $PASS; rc=$?; fi ;;
|
||||
esac
|
||||
else
|
||||
"$bin" install $flags </dev/null >"$log" 2>&1; rc=$? # no typed password and no terminal on stdin: it generates one and prints it
|
||||
fi
|
||||
set -e
|
||||
[ -z "$log" ] || sed 's/^/ /' "$log" >&2
|
||||
hamlab_bin=$(command -v hamlab 2>/dev/null || echo /usr/local/bin/hamlab)
|
||||
[ "$rc" = 0 ] && [ -x "$hamlab_bin" ] || die "The install did not finish. Nothing above this line is hidden: the reason is in it."
|
||||
|
||||
# ──────────────────────────────────────────────────────────────── done ──
|
||||
blank; rule
|
||||
out " ${GREEN}${B}${OK} Ham Lab $("$hamlab_bin" version 2>/dev/null | awk '{ print $2 }') is installed${R}"
|
||||
blank
|
||||
row() { out " $(printf '%-10s' "$1") $2"; [ -z "${3:-}" ] || out " ${D}$3${R}"; }
|
||||
if [ "$init" = none ]; then
|
||||
row "start it" "${B}hamlab serve${R}" "as ${run_as:-its user}, under your own supervisor: nothing starts it at boot"
|
||||
fi
|
||||
if [ -n "$listen" ]; then
|
||||
row "open" "${CYAN}${B}$scheme://$(printf '%s' "$listen" | sed "s/^0\.0\.0\.0/$url_host/")${R}"
|
||||
[ "$tls" != auto ] || out " ${D}the browser warns about the self-signed certificate once${R}"
|
||||
[ "$host" != 127.0.0.1 ] || out " ${D}from elsewhere: ssh -L $port:127.0.0.1:$port $run_as@this-machine${R}"
|
||||
fi
|
||||
case "$init" in
|
||||
systemd) row "logs" "${B}journalctl -u hamlab -f${R}" ;;
|
||||
OpenRC) row "logs" "${B}tail -f /var/log/hamlab.log${R}" ;;
|
||||
esac
|
||||
row "change" "${B}hamlab install --help${R}" "a new address, HTTPS or user; everything else is kept"
|
||||
row "add to hub" "Machines ${ARROW} Add a machine" "on your main Ham Lab, with this address and password"
|
||||
row "remove" "${B}hamlab uninstall${R}" "--purge deletes its config too"
|
||||
rule; blank
|
||||
@@ -0,0 +1,541 @@
|
||||
package api
|
||||
|
||||
// Alerts: rules over sensors, watched by the hub for itself and every machine it manages.
|
||||
//
|
||||
// A rule is a selector, a comparison and a patience: `container.*.running == false for
|
||||
// 120 s`. Every alertEvery the hub asks each machine one shaped query for all the rules'
|
||||
// selectors, compares, and keeps a little state per (rule, machine, sensor): ok, pending
|
||||
// since, or firing since. Going to firing, and coming back, is an event — kept, shown on
|
||||
// the Alerts tab, and sent to whatever channels are set (ntfy, or a webhook: Home
|
||||
// Assistant, Slack, anything that takes a POST). `hamlab.reachable` is a sensor the hub
|
||||
// supplies itself, so "that machine stopped answering" is a rule like any other.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/sensors"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const (
|
||||
alertEvery = 15 * time.Second
|
||||
maxEvents = 200
|
||||
)
|
||||
|
||||
type AlertRule struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Sensor string `json:"sensor"` // an ID or a pattern
|
||||
Op string `json:"op"` // > >= < <= == !=
|
||||
Value any `json:"value"` // number, bool or string, to match the sensor
|
||||
ForSec int `json:"for_sec"`
|
||||
Severity string `json:"severity"` // info | warn | crit
|
||||
Enabled bool `json:"enabled"`
|
||||
Machines string `json:"machines"` // "*" or comma-separated names; "" is "*"
|
||||
}
|
||||
|
||||
type AlertChannel struct {
|
||||
Type string `json:"type"` // ntfy | webhook
|
||||
URL string `json:"url"`
|
||||
}
|
||||
|
||||
type AlertEvent struct {
|
||||
T int64 `json:"t"`
|
||||
Kind string `json:"kind"` // firing | resolved
|
||||
Rule string `json:"rule"`
|
||||
RuleID string `json:"rule_id"`
|
||||
Machine string `json:"machine"`
|
||||
Sensor string `json:"sensor"`
|
||||
Value any `json:"value"`
|
||||
Severity string `json:"severity"`
|
||||
}
|
||||
|
||||
type alertFiring struct {
|
||||
AlertEvent
|
||||
Since int64 `json:"since"`
|
||||
}
|
||||
|
||||
type alertSaved struct {
|
||||
Rules []AlertRule `json:"rules"`
|
||||
Channels []AlertChannel `json:"channels"`
|
||||
Events []AlertEvent `json:"events"`
|
||||
Seeded bool `json:"seeded"`
|
||||
// SawInstanceRule marks that the instance-down default has been offered once; set so that
|
||||
// removing it stays removed.
|
||||
SawInstanceRule bool `json:"saw_instance_rule,omitempty"`
|
||||
// Firing is what was firing when the hub last looked, so a restart (every deploy is one)
|
||||
// carries on where it was and doesn't announce the same alerts all over again.
|
||||
Firing map[string]alertKept `json:"firing,omitempty"`
|
||||
}
|
||||
|
||||
type alertKept struct {
|
||||
Since int64 `json:"since"`
|
||||
Value any `json:"value"`
|
||||
}
|
||||
|
||||
const maxRules = 200
|
||||
|
||||
type alertState struct {
|
||||
pending time.Time
|
||||
firing bool
|
||||
since time.Time
|
||||
value any
|
||||
}
|
||||
|
||||
type alertEngine struct {
|
||||
s *Server
|
||||
mu sync.Mutex
|
||||
saved alertSaved
|
||||
state map[string]*alertState // ruleID|machine|sensor
|
||||
last time.Time
|
||||
}
|
||||
|
||||
var defaultRules = []AlertRule{
|
||||
{ID: "machine-offline", Name: "Machine not answering", Sensor: "hamlab.reachable", Op: "==", Value: false, ForSec: 120, Severity: "crit", Enabled: true},
|
||||
{ID: "container-down", Name: "Container down", Sensor: "container.*.down", Op: "==", Value: true, ForSec: 120, Severity: "crit", Enabled: true},
|
||||
{ID: "container-unhealthy", Name: "Container unhealthy", Sensor: "container.*.health", Op: "==", Value: "unhealthy", ForSec: 180, Severity: "warn", Enabled: true},
|
||||
{ID: "instance-down", Name: "VM or container down", Sensor: "instance.*.down", Op: "==", Value: true, ForSec: 120, Severity: "crit", Enabled: true},
|
||||
{ID: "service-failed", Name: "A service has failed", Sensor: "services.failed", Op: ">", Value: 0.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
{ID: "disk-full", Name: "Filesystem nearly full", Sensor: "fs.*.used_pct", Op: ">=", Value: 90.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
{ID: "pool-unhealthy", Name: "ZFS pool not healthy", Sensor: "zfs.*.healthy", Op: "==", Value: false, ForSec: 60, Severity: "crit", Enabled: true},
|
||||
{ID: "drive-failing", Name: "Drive failing SMART", Sensor: "smart.*.passed", Op: "==", Value: false, ForSec: 0, Severity: "crit", Enabled: true},
|
||||
{ID: "on-battery", Name: "UPS on battery", Sensor: "ups.on_battery", Op: "==", Value: true, ForSec: 30, Severity: "crit", Enabled: true},
|
||||
{ID: "hot", Name: "CPU running hot", Sensor: "cpu.temp", Op: ">=", Value: 90.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
{ID: "memory", Name: "Memory nearly exhausted", Sensor: "mem.used_pct", Op: ">=", Value: 95.0, ForSec: 300, Severity: "warn", Enabled: true},
|
||||
}
|
||||
|
||||
func newAlertEngine(s *Server) *alertEngine {
|
||||
e := &alertEngine{s: s, state: map[string]*alertState{}}
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.Alerts })
|
||||
_ = json.Unmarshal(raw, &e.saved)
|
||||
if !e.saved.Seeded {
|
||||
e.saved.Rules, e.saved.Seeded = append([]AlertRule{}, defaultRules...), true
|
||||
}
|
||||
for key, k := range e.saved.Firing {
|
||||
e.state[key] = &alertState{pending: time.UnixMilli(k.Since), firing: true, since: time.UnixMilli(k.Since), value: k.Value}
|
||||
}
|
||||
// a default added after an install was seeded joins its rules once; removing it afterwards sticks
|
||||
if e.saved.Seeded && !e.saved.SawInstanceRule {
|
||||
has := false
|
||||
for _, r := range e.saved.Rules {
|
||||
if r.ID == "instance-down" {
|
||||
has = true
|
||||
}
|
||||
}
|
||||
if !has {
|
||||
for _, d := range defaultRules {
|
||||
if d.ID == "instance-down" {
|
||||
e.saved.Rules = append(e.saved.Rules, d)
|
||||
}
|
||||
}
|
||||
}
|
||||
e.saved.SawInstanceRule = true
|
||||
}
|
||||
for i, r := range e.saved.Rules { // the first day's default watched "not running", which a container stopped on purpose also is
|
||||
if r.ID == "container-down" && r.Sensor == "container.*.running" {
|
||||
e.saved.Rules[i].Name, e.saved.Rules[i].Sensor, e.saved.Rules[i].Value = "Container down", "container.*.down", true
|
||||
}
|
||||
}
|
||||
go e.run()
|
||||
return e
|
||||
}
|
||||
|
||||
func (e *alertEngine) persist() {
|
||||
b, _ := json.Marshal(e.saved)
|
||||
_ = e.s.store.Update(func(st *store.State) { st.Alerts = b })
|
||||
}
|
||||
|
||||
func (e *alertEngine) run() {
|
||||
time.Sleep(20 * time.Second) // let the status sensors take their first readings
|
||||
for {
|
||||
e.evaluate()
|
||||
time.Sleep(alertEvery)
|
||||
}
|
||||
}
|
||||
|
||||
func wants(r AlertRule, machine string) bool {
|
||||
if r.Machines == "" || r.Machines == "*" {
|
||||
return true
|
||||
}
|
||||
for _, m := range strings.Split(r.Machines, ",") {
|
||||
if strings.TrimSpace(m) == machine {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// holds reports whether `got <op> want`, comparing like with like.
|
||||
func holds(got any, op string, want any) bool {
|
||||
switch g := got.(type) {
|
||||
case float64:
|
||||
w, ok := want.(float64)
|
||||
if !ok {
|
||||
if s, isStr := want.(string); isStr {
|
||||
f, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
w = f
|
||||
} else {
|
||||
return false
|
||||
}
|
||||
}
|
||||
switch op {
|
||||
case ">":
|
||||
return g > w
|
||||
case ">=":
|
||||
return g >= w
|
||||
case "<":
|
||||
return g < w
|
||||
case "<=":
|
||||
return g <= w
|
||||
case "==":
|
||||
return g == w
|
||||
case "!=":
|
||||
return g != w
|
||||
}
|
||||
case bool:
|
||||
w, ok := want.(bool)
|
||||
if !ok {
|
||||
w = fmt.Sprint(want) == "true"
|
||||
}
|
||||
return (op == "==" && g == w) || (op == "!=" && g != w)
|
||||
case string:
|
||||
w := fmt.Sprint(want)
|
||||
return (op == "==" && g == w) || (op == "!=" && g != w)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (e *alertEngine) evaluate() {
|
||||
e.mu.Lock()
|
||||
rules := append([]AlertRule{}, e.saved.Rules...)
|
||||
e.mu.Unlock()
|
||||
query := map[string]string{}
|
||||
for _, r := range rules {
|
||||
if r.Enabled && r.Sensor != "hamlab.reachable" {
|
||||
query[r.ID] = r.Sensor
|
||||
}
|
||||
}
|
||||
qb, _ := json.Marshal(query)
|
||||
machines := append([]string{e.s.hostName()}, e.s.nodes.Names()...)
|
||||
results := map[string]map[string]any{} // machine → ruleID → value | {id: value}
|
||||
reach := map[string]bool{}
|
||||
var wg sync.WaitGroup
|
||||
var mu sync.Mutex
|
||||
for _, m := range machines {
|
||||
wg.Add(1)
|
||||
go func(m string) {
|
||||
defer wg.Done()
|
||||
var res map[string]any
|
||||
ok := false
|
||||
if m == e.s.hostName() {
|
||||
if out, err := sensors.Shape(e.s.sensors.All(), qb, false); err == nil {
|
||||
res, ok = out.(map[string]any)
|
||||
}
|
||||
} else {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
code, b, err := e.s.nodes.Do(ctx, m, "POST", "/api/sensors/query", qb, http.Header{"Content-Type": {"application/json"}})
|
||||
cancel()
|
||||
ok = err == nil && code == 200 && json.Unmarshal(b, &res) == nil
|
||||
}
|
||||
mu.Lock()
|
||||
results[m], reach[m] = res, ok
|
||||
mu.Unlock()
|
||||
}(m)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
now := time.Now()
|
||||
seen := map[string]bool{}
|
||||
var events []AlertEvent
|
||||
e.mu.Lock()
|
||||
for _, r := range rules {
|
||||
if !r.Enabled {
|
||||
continue
|
||||
}
|
||||
for _, m := range machines {
|
||||
if !wants(r, m) {
|
||||
continue
|
||||
}
|
||||
readings := map[string]any{}
|
||||
if r.Sensor == "hamlab.reachable" {
|
||||
readings["hamlab.reachable"] = reach[m]
|
||||
} else if !reach[m] {
|
||||
continue // no reading is not a reading: an unreachable machine is the offline rule's business
|
||||
} else if v, ok := results[m][r.ID]; ok {
|
||||
if many, isMap := v.(map[string]any); isMap {
|
||||
readings = many
|
||||
} else if v != nil {
|
||||
readings[r.Sensor] = v
|
||||
}
|
||||
}
|
||||
for id, val := range readings {
|
||||
key := r.ID + "|" + m + "|" + id
|
||||
seen[key] = true
|
||||
st := e.state[key]
|
||||
if st == nil {
|
||||
st = &alertState{}
|
||||
e.state[key] = st
|
||||
}
|
||||
st.value = val
|
||||
if holds(val, r.Op, r.Value) {
|
||||
if st.pending.IsZero() {
|
||||
st.pending = now
|
||||
}
|
||||
if !st.firing && now.Sub(st.pending) >= time.Duration(r.ForSec)*time.Second {
|
||||
st.firing, st.since = true, now
|
||||
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "firing", Rule: r.Name, RuleID: r.ID, Machine: m, Sensor: id, Value: val, Severity: r.Severity})
|
||||
}
|
||||
} else {
|
||||
if st.firing {
|
||||
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "resolved", Rule: r.Name, RuleID: r.ID, Machine: m, Sensor: id, Value: val, Severity: r.Severity})
|
||||
}
|
||||
st.pending, st.firing = time.Time{}, false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
byID := map[string]AlertRule{}
|
||||
for _, r := range rules {
|
||||
byID[r.ID] = r
|
||||
}
|
||||
for key, st := range e.state { // no longer watched: the rule was paused, deleted or narrowed, or the sensor is gone
|
||||
if seen[key] {
|
||||
continue
|
||||
}
|
||||
if st.firing { // it was on the Alerts tab and perhaps on a phone: say that it is over, and why
|
||||
part := strings.SplitN(key, "|", 3)
|
||||
r, known := byID[part[0]]
|
||||
name, why := r.Name, "no longer watched"
|
||||
if !known {
|
||||
name, why = part[0], "rule deleted"
|
||||
} else if !r.Enabled {
|
||||
why = "rule paused"
|
||||
}
|
||||
if name == "" {
|
||||
name = part[0]
|
||||
}
|
||||
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "resolved", Rule: name, RuleID: part[0], Machine: part[1], Sensor: part[2], Value: why, Severity: r.Severity})
|
||||
}
|
||||
delete(e.state, key)
|
||||
}
|
||||
e.last = now
|
||||
kept := map[string]alertKept{}
|
||||
for key, st := range e.state {
|
||||
if st.firing {
|
||||
kept[key] = alertKept{Since: st.since.UnixMilli(), Value: st.value}
|
||||
}
|
||||
}
|
||||
changed := len(kept) != len(e.saved.Firing)
|
||||
for k := range kept {
|
||||
if _, ok := e.saved.Firing[k]; !ok {
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
e.saved.Firing = kept
|
||||
if changed && len(events) == 0 {
|
||||
e.persist()
|
||||
}
|
||||
if len(events) > 0 {
|
||||
e.saved.Events = append(e.saved.Events, events...)
|
||||
if len(e.saved.Events) > maxEvents {
|
||||
e.saved.Events = e.saved.Events[len(e.saved.Events)-maxEvents:]
|
||||
}
|
||||
e.persist()
|
||||
}
|
||||
channels := append([]AlertChannel{}, e.saved.Channels...)
|
||||
e.mu.Unlock()
|
||||
for _, ev := range events {
|
||||
e.notify(channels, ev)
|
||||
}
|
||||
}
|
||||
|
||||
func (e *alertEngine) firing() []alertFiring {
|
||||
out := []alertFiring{}
|
||||
names := map[string]AlertRule{}
|
||||
for _, r := range e.saved.Rules {
|
||||
names[r.ID] = r
|
||||
}
|
||||
for key, st := range e.state {
|
||||
if !st.firing {
|
||||
continue
|
||||
}
|
||||
p := strings.SplitN(key, "|", 3)
|
||||
r := names[p[0]]
|
||||
out = append(out, alertFiring{AlertEvent: AlertEvent{T: st.since.UnixMilli(), Kind: "firing", Rule: r.Name, RuleID: r.ID, Machine: p[1], Sensor: p[2], Value: st.value, Severity: r.Severity}, Since: st.since.UnixMilli()})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Since > out[j].Since })
|
||||
return out
|
||||
}
|
||||
|
||||
func describe(ev AlertEvent) (title, body string) {
|
||||
mark := map[string]string{"firing": "▲", "resolved": "✓"}[ev.Kind]
|
||||
title = fmt.Sprintf("%s %s — %s", mark, ev.Rule, ev.Machine)
|
||||
body = fmt.Sprintf("%s = %v", ev.Sensor, ev.Value)
|
||||
if ev.Kind == "resolved" {
|
||||
body = "back to normal: " + body
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func (e *alertEngine) notify(channels []AlertChannel, ev AlertEvent) {
|
||||
title, body := describe(ev)
|
||||
for _, c := range channels {
|
||||
var req *http.Request
|
||||
switch c.Type {
|
||||
case "ntfy":
|
||||
req, _ = http.NewRequest("POST", c.URL, strings.NewReader(body))
|
||||
req.Header.Set("Title", title)
|
||||
req.Header.Set("Priority", map[string]string{"crit": "urgent", "warn": "high"}[ev.Severity])
|
||||
req.Header.Set("Tags", map[string]string{"firing": "warning", "resolved": "white_check_mark"}[ev.Kind])
|
||||
if ev.Kind == "resolved" {
|
||||
req.Header.Set("Priority", "default")
|
||||
}
|
||||
case "webhook":
|
||||
b, _ := json.Marshal(map[string]any{"title": title, "message": body, "event": ev})
|
||||
req, _ = http.NewRequest("POST", c.URL, bytes.NewReader(b))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if req == nil {
|
||||
continue
|
||||
}
|
||||
go func(req *http.Request, url string) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
resp, err := http.DefaultClient.Do(req.WithContext(ctx))
|
||||
if err != nil {
|
||||
log.Printf("alert: notifying %s: %v", url, err)
|
||||
return
|
||||
}
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode >= 300 {
|
||||
log.Printf("alert: %s answered %d", url, resp.StatusCode)
|
||||
}
|
||||
}(req, c.URL)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- API ----
|
||||
|
||||
func (s *Server) alertsGet(w http.ResponseWriter, _ *http.Request) {
|
||||
e := s.alerts
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
ev := append([]AlertEvent{}, e.saved.Events...)
|
||||
for i, j := 0, len(ev)-1; i < j; i, j = i+1, j-1 {
|
||||
ev[i], ev[j] = ev[j], ev[i]
|
||||
}
|
||||
rules, channels := e.saved.Rules, e.saved.Channels
|
||||
if rules == nil { // an empty list is a list: never hand the page a null to take the length of
|
||||
rules = []AlertRule{}
|
||||
}
|
||||
if channels == nil {
|
||||
channels = []AlertChannel{}
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"rules": rules, "channels": channels, "events": ev, "firing": e.firing(), "checked": e.last.UnixMilli(), "every_sec": int(alertEvery.Seconds())})
|
||||
}
|
||||
|
||||
func (s *Server) alertsPut(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Rules *[]AlertRule `json:"rules"`
|
||||
Channels *[]AlertChannel `json:"channels"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 256<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if body.Rules != nil && len(*body.Rules) > maxRules {
|
||||
jsonErr(w, 400, fmt.Sprintf("at most %d rules: every one is checked on every machine every %d s", maxRules, int(alertEvery.Seconds())))
|
||||
return
|
||||
}
|
||||
if body.Rules != nil {
|
||||
ids := map[string]bool{}
|
||||
for i := range *body.Rules {
|
||||
ru := &(*body.Rules)[i]
|
||||
ru.Name, ru.Sensor = strings.TrimSpace(ru.Name), strings.TrimSpace(ru.Sensor)
|
||||
if ru.ID == "" || ids[ru.ID] || ru.Name == "" || len(ru.Name) > maxNameLen || ru.ForSec < 0 || ru.ForSec > 86400 {
|
||||
jsonErr(w, 400, "every rule needs a unique id, a name (64 characters at most) and a patience between 0 and 86400 s")
|
||||
return
|
||||
}
|
||||
ids[ru.ID] = true
|
||||
if _, err := sensors.Matcher(ru.Sensor); err != nil || ru.Sensor == "" {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: %q is not a sensor ID or pattern", ru.Name, ru.Sensor))
|
||||
return
|
||||
}
|
||||
if !strings.Contains(" > >= < <= == != ", " "+ru.Op+" ") {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: the comparison must be one of > >= < <= == !=", ru.Name))
|
||||
return
|
||||
}
|
||||
switch val := ru.Value.(type) {
|
||||
case float64, bool:
|
||||
case string:
|
||||
if len(val) > 200 {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: the value is too long", ru.Name))
|
||||
return
|
||||
}
|
||||
if ru.Op != "==" && ru.Op != "!=" { // "cpu.usage > ninety" can never be true, and would never say so
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: %s compares numbers, and %q is not one; text can only be == or !=", ru.Name, ru.Op, val))
|
||||
return
|
||||
}
|
||||
default:
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: the value must be a number, true/false or text", ru.Name))
|
||||
return
|
||||
}
|
||||
if _, isBool := ru.Value.(bool); isBool && ru.Op != "==" && ru.Op != "!=" {
|
||||
jsonErr(w, 400, fmt.Sprintf("rule %q: true/false can only be == or !=", ru.Name))
|
||||
return
|
||||
}
|
||||
if ru.Severity != "info" && ru.Severity != "crit" {
|
||||
ru.Severity = "warn"
|
||||
}
|
||||
}
|
||||
}
|
||||
if body.Channels != nil {
|
||||
for _, c := range *body.Channels {
|
||||
if (c.Type != "ntfy" && c.Type != "webhook") || !(strings.HasPrefix(c.URL, "https://") || strings.HasPrefix(c.URL, "http://")) {
|
||||
jsonErr(w, 400, "a channel is {type: ntfy|webhook, url: http(s)://…}")
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
e := s.alerts
|
||||
e.mu.Lock()
|
||||
if body.Rules != nil {
|
||||
e.saved.Rules = *body.Rules
|
||||
}
|
||||
if body.Channels != nil {
|
||||
e.saved.Channels = *body.Channels
|
||||
}
|
||||
e.persist()
|
||||
e.mu.Unlock()
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// alertsTest sends one clearly-marked test notification through every channel.
|
||||
func (s *Server) alertsTest(w http.ResponseWriter, _ *http.Request) {
|
||||
e := s.alerts
|
||||
e.mu.Lock()
|
||||
channels := append([]AlertChannel{}, e.saved.Channels...)
|
||||
e.mu.Unlock()
|
||||
if len(channels) == 0 {
|
||||
jsonErr(w, 400, "no channels to test")
|
||||
return
|
||||
}
|
||||
e.notify(channels, AlertEvent{T: time.Now().UnixMilli(), Kind: "firing", Rule: "Test alert", Machine: s.hostName(), Sensor: "hamlab.test", Value: "this is only a test", Severity: "info"})
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,631 @@
|
||||
package api
|
||||
|
||||
// The Docker tab: what is running, grouped the way it was deployed (compose stacks), with
|
||||
// start / stop / restart, logs and inspect — plus images, volumes and networks to look at.
|
||||
// It drives the docker CLI (or podman), as the user first and as root if that is refused.
|
||||
// What can be removed here is what nothing uses: an untagged image, a volume no container
|
||||
// holds, a network nothing is attached to. Compose stays the one place a stack is defined.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type dockerContainer struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Image string `json:"image"`
|
||||
State string `json:"state"` // running | exited | paused | restarting | created | dead
|
||||
Status string `json:"status"` // "Up 3 hours (healthy)"
|
||||
Health string `json:"health"` // healthy | unhealthy | starting | none
|
||||
Ports string `json:"ports"`
|
||||
Stack string `json:"stack"` // compose project, "" when started by hand
|
||||
Service string `json:"service"` // compose service
|
||||
Created string `json:"created"`
|
||||
Mem float64 `json:"mem"` // bytes, from the container's cgroup; 0 when unknown
|
||||
}
|
||||
|
||||
func dockerTool() string {
|
||||
for _, t := range []string{"docker", "podman"} {
|
||||
if _, err := exec.LookPath(t); err == nil {
|
||||
return t
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// dockerRun runs the tool as this user, and again as root if the socket said no.
|
||||
// DockerTool is whichever of docker or podman this machine has, for callers outside the
|
||||
// package — the scheduled updates the CLI runs.
|
||||
func DockerTool() string { return dockerTool() }
|
||||
|
||||
func dockerRun(ctx context.Context, args ...string) ([]byte, error) {
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
return nil, errors.New("neither docker nor podman is installed here")
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, tool, args...).CombinedOutput()
|
||||
if err != nil && priv.Available() && strings.Contains(strings.ToLower(string(out)), "permission denied") {
|
||||
full := priv.Wrap(append([]string{tool}, args...)...)
|
||||
out, err = exec.CommandContext(ctx, full[0], full[1:]...).CombinedOutput()
|
||||
}
|
||||
if err != nil {
|
||||
msg := strings.TrimSpace(string(out))
|
||||
if i := strings.LastIndex(msg, "\n"); i >= 0 {
|
||||
msg = msg[i+1:]
|
||||
}
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return out, errors.New(msg)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// dockerWhy picks the line worth showing out of a failed command. Some subcommands —
|
||||
// `network rm` among them — print the real reason first and then an aggregated "exit
|
||||
// status 1", which is what dockerRun's last-line rule would otherwise hand back.
|
||||
func dockerWhy(out []byte, err error) string {
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
if l = strings.TrimSpace(l); strings.HasPrefix(l, "Error") || strings.Contains(l, "daemon") {
|
||||
return l
|
||||
}
|
||||
}
|
||||
return err.Error()
|
||||
}
|
||||
|
||||
func jsonLines(out []byte, each func(map[string]any)) {
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
var m map[string]any
|
||||
if json.Unmarshal([]byte(line), &m) == nil {
|
||||
each(m)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func dstr(m map[string]any, k string) string { s, _ := m[k].(string); return s }
|
||||
|
||||
func labelOf(labels, key string) string {
|
||||
for _, kv := range strings.Split(labels, ",") {
|
||||
if v, ok := strings.CutPrefix(kv, key+"="); ok {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (s *Server) dockerOverview(w http.ResponseWriter, r *http.Request) {
|
||||
if dockerTool() == "" {
|
||||
jsonOK(w, 200, map[string]any{"available": false, "message": "Docker isn't installed on this machine."})
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "ps", "-a", "--no-trunc", "--format", "{{json .}}")
|
||||
if err != nil {
|
||||
jsonOK(w, 200, map[string]any{"available": false, "message": err.Error()})
|
||||
return
|
||||
}
|
||||
containers := []dockerContainer{}
|
||||
jsonLines(out, func(m map[string]any) {
|
||||
c := dockerContainer{ID: dstr(m, "ID"), Name: strings.Split(dstr(m, "Names"), ",")[0], Image: dstr(m, "Image"), State: dstr(m, "State"), Status: dstr(m, "Status"),
|
||||
Ports: dstr(m, "Ports"), Created: dstr(m, "CreatedAt"), Health: "none", Stack: labelOf(dstr(m, "Labels"), "com.docker.compose.project"), Service: labelOf(dstr(m, "Labels"), "com.docker.compose.service")}
|
||||
for _, h := range []string{"unhealthy", "healthy", "starting"} {
|
||||
if strings.Contains(c.Status, "("+h+")") || strings.Contains(c.Status, "health: "+h) {
|
||||
c.Health = h
|
||||
break
|
||||
}
|
||||
}
|
||||
for _, dir := range []string{"/sys/fs/cgroup/system.slice/docker-" + c.ID + ".scope", "/sys/fs/cgroup/docker/" + c.ID, "/sys/fs/cgroup/machine.slice/libpod-" + c.ID + ".scope"} {
|
||||
if b, err := os.ReadFile(dir + "/memory.current"); err == nil {
|
||||
c.Mem, _ = strconv.ParseFloat(strings.TrimSpace(string(b)), 64)
|
||||
break
|
||||
}
|
||||
}
|
||||
containers = append(containers, c)
|
||||
})
|
||||
sort.Slice(containers, func(i, j int) bool {
|
||||
if containers[i].Stack != containers[j].Stack {
|
||||
return containers[i].Stack < containers[j].Stack
|
||||
}
|
||||
return containers[i].Name < containers[j].Name
|
||||
})
|
||||
list := func(args ...string) []map[string]any {
|
||||
rows := []map[string]any{}
|
||||
if out, err := dockerRun(ctx, args...); err == nil {
|
||||
jsonLines(out, func(m map[string]any) { rows = append(rows, m) })
|
||||
}
|
||||
return rows
|
||||
}
|
||||
// Who uses what, from one inspect of every container: an image, a volume or a network
|
||||
// nothing uses is the only kind this page offers to remove.
|
||||
imageUsed := map[string]bool{}
|
||||
volumeUsers, networkUsers := map[string][]string{}, map[string][]string{}
|
||||
if ids, err := dockerRun(ctx, "ps", "-aq"); err == nil && len(strings.Fields(string(ids))) > 0 {
|
||||
const f = `{{.Name}}|{{.Image}}|{{range .Mounts}}{{if eq .Type "volume"}}{{.Name}},{{end}}{{end}}|{{range $k, $v := .NetworkSettings.Networks}}{{$k}},{{end}}`
|
||||
if out, err := dockerRun(ctx, append([]string{"inspect", "-f", f}, strings.Fields(string(ids))...)...); err == nil {
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
p := strings.SplitN(line, "|", 4)
|
||||
if len(p) < 4 {
|
||||
continue
|
||||
}
|
||||
name := strings.TrimPrefix(p[0], "/")
|
||||
if id := strings.TrimPrefix(p[1], "sha256:"); len(id) >= 12 {
|
||||
imageUsed[id[:12]] = true
|
||||
}
|
||||
for _, v := range strings.Split(p[2], ",") {
|
||||
if v != "" {
|
||||
volumeUsers[v] = append(volumeUsers[v], name)
|
||||
}
|
||||
}
|
||||
for _, n := range strings.Split(p[3], ",") {
|
||||
if n != "" {
|
||||
networkUsers[n] = append(networkUsers[n], name)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
images := list("images", "--format", "{{json .}}")
|
||||
for _, im := range images {
|
||||
im["in_use"] = imageUsed[dstr(im, "ID")]
|
||||
}
|
||||
volumes := list("volume", "ls", "--format", "{{json .}}")
|
||||
for _, v := range volumes {
|
||||
users := volumeUsers[dstr(v, "Name")]
|
||||
if users == nil {
|
||||
users = []string{}
|
||||
}
|
||||
sort.Strings(users)
|
||||
v["used_by"] = users
|
||||
v["anonymous"] = strings.Contains(dstr(v, "Labels"), "com.docker.volume.anonymous")
|
||||
}
|
||||
networks := list("network", "ls", "--format", "{{json .}}")
|
||||
for _, n := range networks {
|
||||
users := networkUsers[dstr(n, "Name")]
|
||||
if users == nil {
|
||||
users = []string{}
|
||||
}
|
||||
sort.Strings(users)
|
||||
n["used_by"] = users
|
||||
n["predefined"] = predefinedNetwork[dstr(n, "Name")]
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"available": true, "tool": dockerTool(), "containers": containers,
|
||||
"images": images, "volumes": volumes, "networks": networks})
|
||||
}
|
||||
|
||||
var dockerRef = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}$`)
|
||||
|
||||
func (s *Server) dockerAction(w http.ResponseWriter, r *http.Request) {
|
||||
id, action := r.PathValue("id"), r.PathValue("action")
|
||||
if !dockerRef.MatchString(id) || (action != "start" && action != "stop" && action != "restart" && action != "remove") {
|
||||
jsonErr(w, 400, "start, stop, restart or remove a container by name or ID")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
verb := action
|
||||
if action == "remove" { // never forced: a running container has to be stopped first, on purpose
|
||||
verb = "rm"
|
||||
}
|
||||
if _, err := dockerRun(ctx, verb, id); err != nil {
|
||||
code := 500
|
||||
if action == "remove" {
|
||||
code = 409
|
||||
}
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: %s %s (from %s)", action, id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
var imageRef = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,254}$`)
|
||||
|
||||
// imagePull runs `docker pull` as a task the page follows.
|
||||
func (s *Server) imagePull(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Image string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !imageRef.MatchString(strings.TrimSpace(body.Image)) {
|
||||
jsonErr(w, 400, "an image looks like nginx:alpine or ghcr.io/owner/name:tag")
|
||||
return
|
||||
}
|
||||
ref := strings.TrimSpace(body.Image)
|
||||
t, err := s.tasks.Start("docker:pull", "pull "+ref, exec.Command(dockerTool(), "pull", ref))
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: pull %s as task %s (from %s)", ref, t.ID, clientIP(r))
|
||||
jsonOK(w, 202, t.Snapshot(false))
|
||||
}
|
||||
|
||||
// imageRemove is `docker rmi`, unforced: an image something still uses is refused by Docker, and that is the answer.
|
||||
func (s *Server) imageRemove(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Image string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !imageRef.MatchString(strings.TrimSpace(body.Image)) {
|
||||
jsonErr(w, 400, "which image?")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
if _, err := dockerRun(ctx, "rmi", strings.TrimSpace(body.Image)); err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: rmi %s (from %s)", body.Image, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// imagePrune removes dangling images — layers no tag points at any more, left behind by pulls —
|
||||
// or, asked for {"all": true}, every image no container uses. Those can all be pulled again.
|
||||
func (s *Server) imagePrune(w http.ResponseWriter, r *http.Request) {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
var body struct{ All bool }
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
args := []string{"image", "prune", "-f"}
|
||||
if body.All {
|
||||
args = append(args, "-a")
|
||||
}
|
||||
out, err := dockerRun(ctx, args...)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
log.Printf("docker: image prune (from %s)", clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
|
||||
}
|
||||
|
||||
// dockerLogs is the tail of a container's log; ?since=<RFC3339> asks only for what is new,
|
||||
// which is how the page follows a log without holding a connection open.
|
||||
func (s *Server) dockerLogs(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
args := []string{"logs", "--timestamps"}
|
||||
if since := r.URL.Query().Get("since"); since != "" {
|
||||
if _, err := time.Parse(time.RFC3339Nano, since); err != nil {
|
||||
jsonErr(w, 400, "since is an RFC 3339 time")
|
||||
return
|
||||
}
|
||||
args = append(args, "--since", since)
|
||||
} else {
|
||||
n, _ := strconv.Atoi(r.URL.Query().Get("n"))
|
||||
if n <= 0 || n > 5000 {
|
||||
n = 300
|
||||
}
|
||||
args = append(args, "--tail", strconv.Itoa(n))
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, append(args, id)...)
|
||||
if err != nil {
|
||||
code := 500
|
||||
if strings.Contains(strings.ToLower(err.Error()), "no such container") {
|
||||
code = 404
|
||||
}
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
type line struct {
|
||||
T string `json:"t"`
|
||||
M string `json:"m"`
|
||||
}
|
||||
lines := []line{}
|
||||
for _, l := range strings.Split(strings.TrimRight(string(out), "\n"), "\n") {
|
||||
if l == "" {
|
||||
continue
|
||||
}
|
||||
t, m, ok := strings.Cut(l, " ")
|
||||
if _, err := time.Parse(time.RFC3339Nano, t); !ok || err != nil {
|
||||
t, m = "", l
|
||||
}
|
||||
lines = append(lines, line{t, m})
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"lines": lines})
|
||||
}
|
||||
|
||||
// dockerInspect is the parts of `docker inspect` worth reading on a phone.
|
||||
func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "inspect", id)
|
||||
var all []map[string]any
|
||||
if err != nil || json.Unmarshal(out, &all) != nil || len(all) == 0 {
|
||||
jsonErr(w, 500, fmt.Sprint("inspect failed: ", err))
|
||||
return
|
||||
}
|
||||
c := all[0]
|
||||
dig := func(m map[string]any, path ...string) any {
|
||||
var cur any = m
|
||||
for _, p := range path {
|
||||
mm, ok := cur.(map[string]any)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
cur = mm[p]
|
||||
}
|
||||
return cur
|
||||
}
|
||||
env := []string{}
|
||||
if list, ok := dig(c, "Config", "Env").([]any); ok {
|
||||
for _, e := range list {
|
||||
k, _, _ := strings.Cut(fmt.Sprint(e), "=")
|
||||
env = append(env, k) // names only: values are where the secrets live
|
||||
}
|
||||
}
|
||||
sort.Strings(env)
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"image": dig(c, "Config", "Image"), "command": dig(c, "Config", "Cmd"), "entrypoint": dig(c, "Config", "Entrypoint"),
|
||||
"restart": dig(c, "HostConfig", "RestartPolicy", "Name"), "network_mode": dig(c, "HostConfig", "NetworkMode"),
|
||||
"memory_limit": dig(c, "HostConfig", "Memory"), "started": dig(c, "State", "StartedAt"), "exit_code": dig(c, "State", "ExitCode"),
|
||||
"error": dig(c, "State", "Error"), "restarts": c["RestartCount"], "mounts": c["Mounts"], "ports": dig(c, "NetworkSettings", "Ports"),
|
||||
"networks": dig(c, "NetworkSettings", "Networks"), "env_names": env,
|
||||
"compose_file": dig(c, "Config", "Labels", "com.docker.compose.project.config_files"),
|
||||
})
|
||||
}
|
||||
|
||||
var volumeRef = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,254}$`)
|
||||
|
||||
// volumeRemove deletes one volume and what is in it, which nothing brings back. So it is never
|
||||
// forced (Docker refuses a volume a container still uses), it is one at a time, and the request
|
||||
// has to name the volume twice — a slip of a script can't delete one by accident.
|
||||
func (s *Server) volumeRemove(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name, Confirm string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !volumeRef.MatchString(body.Name) {
|
||||
jsonErr(w, 400, "which volume?")
|
||||
return
|
||||
}
|
||||
if body.Confirm != body.Name {
|
||||
jsonErr(w, 400, "name the volume again in \"confirm\" to delete it")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
if _, err := dockerRun(ctx, "volume", "rm", body.Name); err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker: volume rm %s (from %s)", body.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// volumePrune removes unused *anonymous* volumes only: the nameless ones an image's VOLUME line
|
||||
// leaves behind each time its container is recreated. Named volumes are somebody's data, and a
|
||||
// stack that is merely down leaves its volumes "unused" — those go one at a time, by name.
|
||||
func (s *Server) volumePrune(w http.ResponseWriter, r *http.Request) {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "volume", "prune", "-f")
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
log.Printf("docker: volume prune (from %s)", clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
|
||||
}
|
||||
|
||||
// The three networks Docker makes itself. They hold nothing, cannot be removed, and
|
||||
// their absence would break every container that asks for one by name.
|
||||
var predefinedNetwork = map[string]bool{"bridge": true, "host": true, "none": true}
|
||||
|
||||
// networkRemove deletes one network. Unlike a volume there is no data in it, and unlike an
|
||||
// image nothing has to be downloaded again — a stack that wants it back gets it recreated
|
||||
// on the next deploy. Docker refuses one a container is still attached to, which is the
|
||||
// check that matters, so this asks for no second confirmation.
|
||||
func (s *Server) networkRemove(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !volumeRef.MatchString(body.Name) {
|
||||
jsonErr(w, 400, "which network?")
|
||||
return
|
||||
}
|
||||
if predefinedNetwork[body.Name] {
|
||||
jsonErr(w, 400, body.Name+" is one of Docker's own networks and can't be removed")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||
defer cancel()
|
||||
if out, err := dockerRun(ctx, "network", "rm", body.Name); err != nil {
|
||||
jsonErr(w, 409, dockerWhy(out, err))
|
||||
return
|
||||
}
|
||||
log.Printf("docker: network rm %s (from %s)", body.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// networkPrune removes every network no container is attached to. A stack that is merely
|
||||
// down counts as unattached, so its network goes too — and comes back on the next deploy,
|
||||
// with a new subnet. That is the whole cost, and the caller is told it.
|
||||
func (s *Server) networkPrune(w http.ResponseWriter, r *http.Request) {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "network", "prune", "-f")
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
log.Printf("docker: network prune (from %s)", clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
|
||||
}
|
||||
|
||||
// ---- a container's health, its recent life, its console, and its limits ----
|
||||
|
||||
// dockerHealth is the health check's verdict and the last few probe results. Docker keeps
|
||||
// the last five itself, which is all anyone wants: is it failing, and what did it say.
|
||||
func (s *Server) dockerHealth(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
|
||||
defer cancel()
|
||||
out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
var st struct {
|
||||
Status string `json:"Status"`
|
||||
Health *struct {
|
||||
Status string `json:"Status"`
|
||||
FailingStreak int `json:"FailingStreak"`
|
||||
Log []struct {
|
||||
Start time.Time `json:"Start"`
|
||||
End time.Time `json:"End"`
|
||||
ExitCode int `json:"ExitCode"`
|
||||
Output string `json:"Output"`
|
||||
} `json:"Log"`
|
||||
} `json:"Health"`
|
||||
}
|
||||
if json.Unmarshal(out, &st) != nil {
|
||||
jsonErr(w, 500, "could not read the container's state")
|
||||
return
|
||||
}
|
||||
if st.Health == nil {
|
||||
jsonOK(w, 200, map[string]any{"configured": false, "state": st.Status})
|
||||
return
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"configured": true, "state": st.Status, "status": st.Health.Status,
|
||||
"failing_streak": st.Health.FailingStreak, "log": st.Health.Log,
|
||||
})
|
||||
}
|
||||
|
||||
// dockerExec opens a shell inside a container, as a terminal session like any other — so
|
||||
// it reattaches, resizes and scrolls back the same way, and works on a managed machine
|
||||
// through the same proxy.
|
||||
func (s *Server) dockerExec(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
jsonErr(w, 500, "neither docker nor podman is installed here")
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
Shell string `json:"shell"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
shell := strings.TrimSpace(body.Shell)
|
||||
if shell == "" {
|
||||
// most images have sh; bash is nicer where it exists, so try it first and fall back
|
||||
shell = "if command -v bash >/dev/null 2>&1; then exec bash; else exec sh; fi"
|
||||
} else if strings.ContainsAny(shell, "\n\r") {
|
||||
jsonErr(w, 400, "bad shell")
|
||||
return
|
||||
}
|
||||
argv := []string{tool, "exec", "-it", id, "sh", "-c", shell}
|
||||
if priv.Available() && !dockerReachable() {
|
||||
argv = priv.Wrap(argv...)
|
||||
}
|
||||
sess, err := s.mgr.CreateCmd("", "sh "+shortName(id), argv)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker exec into %s as terminal %s by %s", id, sess.ID, clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"id": sess.ID, "name": "sh " + shortName(id)})
|
||||
}
|
||||
|
||||
func shortName(id string) string {
|
||||
if len(id) > 20 {
|
||||
return id[:20]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// dockerReachable reports whether docker answers without help.
|
||||
func dockerReachable() bool {
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
return false
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
return exec.CommandContext(ctx, tool, "version", "--format", "{{.Server.Version}}").Run() == nil
|
||||
}
|
||||
|
||||
// dockerUpdate changes a running container's limits and restart policy. `docker update`
|
||||
// does this live, without recreating it and without touching the compose file — which is
|
||||
// also the catch, and the caller is told: a deploy from compose puts back whatever compose
|
||||
// says.
|
||||
func (s *Server) dockerUpdate(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
Memory string `json:"memory"` // "512m", "2g", or "" to leave alone; "0" removes the limit
|
||||
CPUs string `json:"cpus"` // "1.5", or "0" for no limit
|
||||
Restart string `json:"restart"` // no | on-failure | unless-stopped | always
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
args := []string{"update"}
|
||||
if m := strings.TrimSpace(body.Memory); m != "" {
|
||||
if !regexp.MustCompile(`^[0-9]+([bkmgBKMG])?$`).MatchString(m) {
|
||||
jsonErr(w, 400, "memory: a number, optionally with b, k, m or g — 0 removes the limit")
|
||||
return
|
||||
}
|
||||
args = append(args, "--memory", m, "--memory-swap", m)
|
||||
}
|
||||
if c := strings.TrimSpace(body.CPUs); c != "" {
|
||||
if !regexp.MustCompile(`^[0-9]+(\.[0-9]+)?$`).MatchString(c) {
|
||||
jsonErr(w, 400, "cpus: a number like 1.5 — 0 removes the limit")
|
||||
return
|
||||
}
|
||||
args = append(args, "--cpus", c)
|
||||
}
|
||||
if p := strings.TrimSpace(body.Restart); p != "" {
|
||||
switch p {
|
||||
case "no", "on-failure", "unless-stopped", "always":
|
||||
default:
|
||||
jsonErr(w, 400, "restart: no, on-failure, unless-stopped or always")
|
||||
return
|
||||
}
|
||||
args = append(args, "--restart", p)
|
||||
}
|
||||
if len(args) == 1 {
|
||||
jsonErr(w, 400, "nothing to change")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
|
||||
defer cancel()
|
||||
if _, err := dockerRun(ctx, append(args, id)...); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker update %s %v by %s", id, args[1:], clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package api
|
||||
|
||||
// A container's history.
|
||||
//
|
||||
// `docker events` is a live stream with only a short in-memory backlog — ask it for the
|
||||
// last thirty days and you get the last few minutes, because the daemon keeps events in a
|
||||
// small ring and loses them all when it restarts. That is why Dockhand kept its own copy,
|
||||
// and why this does too: a watcher follows the stream and appends the lifecycle events to a
|
||||
// ring kept beside the config, so "what happened to this container last Tuesday" has an
|
||||
// answer.
|
||||
//
|
||||
// Only lifecycle events are kept. A health check produces three exec events every probe,
|
||||
// which is thousands a day and tells you nothing that the health panel does not.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
const (
|
||||
eventsKept = 4000
|
||||
eventsSaveEvery = 30 * time.Second
|
||||
)
|
||||
|
||||
// ContainerEvent is one thing that happened to one container.
|
||||
type ContainerEvent struct {
|
||||
T int64 `json:"t"` // unix ms
|
||||
Name string `json:"name"`
|
||||
Action string `json:"action"`
|
||||
Image string `json:"image,omitempty"`
|
||||
ExitCode string `json:"exit_code,omitempty"`
|
||||
Signal string `json:"signal,omitempty"`
|
||||
Health string `json:"health,omitempty"`
|
||||
}
|
||||
|
||||
type eventLog struct {
|
||||
mu sync.Mutex
|
||||
ring []ContainerEvent
|
||||
dirty bool
|
||||
path string
|
||||
}
|
||||
|
||||
func (e *eventLog) load(path string) {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
e.path = path
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
var list []ContainerEvent
|
||||
if json.Unmarshal(b, &list) == nil {
|
||||
e.ring = list
|
||||
}
|
||||
}
|
||||
|
||||
func (e *eventLog) add(ev ContainerEvent) {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
e.ring = append(e.ring, ev)
|
||||
if len(e.ring) > eventsKept {
|
||||
e.ring = append([]ContainerEvent(nil), e.ring[len(e.ring)-eventsKept:]...)
|
||||
}
|
||||
e.dirty = true
|
||||
}
|
||||
|
||||
// forName is one container's events, newest first.
|
||||
func (e *eventLog) forName(name string, since time.Time, limit int) []ContainerEvent {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
out := []ContainerEvent{}
|
||||
for _, ev := range e.ring {
|
||||
if ev.Name == name && ev.T >= since.UnixMilli() {
|
||||
out = append(out, ev)
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].T > out[j].T })
|
||||
if len(out) > limit {
|
||||
out = out[:limit]
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (e *eventLog) save() {
|
||||
e.mu.Lock()
|
||||
if !e.dirty || e.path == "" {
|
||||
e.mu.Unlock()
|
||||
return
|
||||
}
|
||||
b, err := json.Marshal(e.ring)
|
||||
e.dirty = false
|
||||
path := e.path
|
||||
e.mu.Unlock()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
_ = os.MkdirAll(filepath.Dir(path), 0o700)
|
||||
tmp := path + ".tmp"
|
||||
if os.WriteFile(tmp, b, 0o600) == nil {
|
||||
_ = os.Rename(tmp, path)
|
||||
}
|
||||
}
|
||||
|
||||
// watchDockerEvents follows the stream, restarting if Docker goes away. It is the only
|
||||
// thing that writes the log, so nothing contends with it.
|
||||
func (s *Server) watchDockerEvents(ctx context.Context) {
|
||||
s.events.load(filepath.Join(s.cfg.DataDir(), "docker-events.json"))
|
||||
go func() { // a periodic flush, so a crash loses at most half a minute of history
|
||||
t := time.NewTicker(eventsSaveEvery)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
s.events.save()
|
||||
return
|
||||
case <-t.C:
|
||||
s.events.save()
|
||||
}
|
||||
}
|
||||
}()
|
||||
for {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
if dockerTool() == "" {
|
||||
time.Sleep(time.Minute) // no Docker here; look again later in case it is installed
|
||||
continue
|
||||
}
|
||||
s.followEvents(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(10 * time.Second): // the daemon restarted, or the pipe broke
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) followEvents(ctx context.Context) {
|
||||
args := []string{"events", "--format", "{{json .}}", "--filter", "type=container"}
|
||||
for _, e := range []string{"create", "start", "restart", "stop", "kill", "die", "destroy", "oom", "pause", "unpause", "rename", "update", "health_status"} {
|
||||
args = append(args, "--filter", "event="+e)
|
||||
}
|
||||
tool := dockerTool()
|
||||
full := append([]string{tool}, args...)
|
||||
if !dockerReachable() && priv.Available() {
|
||||
full = priv.Wrap(full...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return
|
||||
}
|
||||
sc := bufio.NewScanner(out)
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
n := 0
|
||||
for sc.Scan() {
|
||||
var e struct {
|
||||
Action string `json:"Action"`
|
||||
Time int64 `json:"time"`
|
||||
TimeNano int64 `json:"timeNano"`
|
||||
Actor struct {
|
||||
Attributes map[string]string `json:"Attributes"`
|
||||
} `json:"Actor"`
|
||||
}
|
||||
if json.Unmarshal(sc.Bytes(), &e) != nil || e.Action == "" {
|
||||
continue
|
||||
}
|
||||
at := e.Actor.Attributes
|
||||
ev := ContainerEvent{
|
||||
T: e.Time * 1000, Name: at["name"], Image: at["image"],
|
||||
Action: e.Action, ExitCode: at["exitCode"], Signal: at["signal"],
|
||||
}
|
||||
if e.TimeNano > 0 {
|
||||
ev.T = e.TimeNano / 1e6
|
||||
}
|
||||
// health_status comes through as "health_status: healthy"
|
||||
if a, h, ok := strings.Cut(e.Action, ": "); ok {
|
||||
ev.Action, ev.Health = a, h
|
||||
}
|
||||
if ev.Name == "" {
|
||||
continue
|
||||
}
|
||||
s.events.add(ev)
|
||||
n++
|
||||
}
|
||||
_ = cmd.Wait()
|
||||
if n > 0 {
|
||||
log.Printf("docker events: stream ended after %d events; reconnecting", n)
|
||||
}
|
||||
}
|
||||
|
||||
// dockerEvents answers from Ham Lab's own history, not Docker's short memory.
|
||||
func (s *Server) dockerEvents(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
since := r.URL.Query().Get("since")
|
||||
d := 7 * 24 * time.Hour
|
||||
switch since {
|
||||
case "24h":
|
||||
d = 24 * time.Hour
|
||||
case "30d":
|
||||
d = 30 * 24 * time.Hour
|
||||
default:
|
||||
since = "7d"
|
||||
}
|
||||
ev := s.events.forName(id, time.Now().Add(-d), 500)
|
||||
s.events.mu.Lock()
|
||||
total := len(s.events.ring)
|
||||
s.events.mu.Unlock()
|
||||
jsonOK(w, 200, map[string]any{"events": ev, "since": since, "kept": total, "capacity": eventsKept})
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package api
|
||||
|
||||
// Following a container's log over a held connection, the way the Services tab follows a
|
||||
// journal. The polled `?since=` endpoint next door still exists — it is what a reconnect
|
||||
// hands back to pick up from — but the live view holds one `docker logs -f` per viewer
|
||||
// instead of asking again every two seconds.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"os/exec"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/coder/websocket"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// logLine is one line of a container's output: the time Docker stamped it with, the text,
|
||||
// and which of the two streams it came out of — stderr is where a container says what went
|
||||
// wrong, and the page colours it accordingly.
|
||||
type logLine struct {
|
||||
T string `json:"t"`
|
||||
M string `json:"m"`
|
||||
Err bool `json:"err,omitempty"`
|
||||
at string // when to sort it by: its own stamp, or the last one its stream carried
|
||||
}
|
||||
|
||||
// splitStamp pulls Docker's RFC 3339 prefix off a line. A line without one (a container
|
||||
// that writes bare bytes, or a continuation) keeps the whole text and no time — but is
|
||||
// still sorted where its stream put it, behind the last line that did have one.
|
||||
func splitStamp(l string, carry *string) logLine {
|
||||
t, m, ok := strings.Cut(l, " ")
|
||||
if _, err := time.Parse(time.RFC3339Nano, t); !ok || err != nil {
|
||||
return logLine{M: l, at: *carry}
|
||||
}
|
||||
*carry = t
|
||||
return logLine{T: t, M: m, at: t}
|
||||
}
|
||||
|
||||
// dockerTail streams a container's output as JSON text frames until the client goes away.
|
||||
// `?since=<RFC3339>` resumes where a dropped connection left off; otherwise it starts with
|
||||
// the last `?n=` lines.
|
||||
func (s *Server) dockerTail(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
if !dockerRef.MatchString(id) {
|
||||
jsonErr(w, 400, "bad container")
|
||||
return
|
||||
}
|
||||
tool := dockerTool()
|
||||
if tool == "" {
|
||||
jsonErr(w, 500, "neither docker nor podman is installed here")
|
||||
return
|
||||
}
|
||||
args := []string{tool, "logs", "--timestamps", "--follow"}
|
||||
if since := r.URL.Query().Get("since"); since != "" {
|
||||
if _, err := time.Parse(time.RFC3339Nano, since); err != nil {
|
||||
jsonErr(w, 400, "since is an RFC 3339 time")
|
||||
return
|
||||
}
|
||||
args = append(args, "--since", since)
|
||||
} else {
|
||||
args = append(args, "--tail", strconv.Itoa(logCount(r)))
|
||||
}
|
||||
args = append(args, id)
|
||||
if priv.Available() && !dockerReachable() {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
|
||||
conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{CompressionMode: websocket.CompressionDisabled})
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer conn.Close(websocket.StatusNormalClosure, "")
|
||||
ctx, cancel := context.WithCancel(r.Context())
|
||||
defer cancel()
|
||||
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
// Own process group, so the kill reaches the docker client itself and not just a sudo
|
||||
// wrapper — the same reason the journal tail does it.
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error {
|
||||
pg := "-" + strconv.Itoa(cmd.Process.Pid)
|
||||
if args[0] != tool { // running under sudo: only sudo can signal root's children
|
||||
k := priv.Wrap("kill", "-9", "--", pg)
|
||||
return exec.Command(k[0], k[1:]...).Run()
|
||||
}
|
||||
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||
}
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
stderr, err := cmd.StderrPipe()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
_ = conn.Write(ctx, websocket.MessageText, mustJSON(logLine{M: err.Error(), Err: true}))
|
||||
return
|
||||
}
|
||||
|
||||
// Both pipes funnel into one writer: a websocket takes one writer at a time.
|
||||
lines := make(chan logLine, 256)
|
||||
read := func(p io.Reader, isErr bool) {
|
||||
sc := bufio.NewScanner(p)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
|
||||
carry := ""
|
||||
for sc.Scan() {
|
||||
l := splitStamp(sc.Text(), &carry)
|
||||
l.Err = isErr
|
||||
select {
|
||||
case lines <- l:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
done := make(chan struct{}, 2)
|
||||
go func() { read(stdout, false); done <- struct{}{} }()
|
||||
go func() { read(stderr, true); done <- struct{}{} }()
|
||||
go func() { // the command ends when the container does; closing says so, having said why
|
||||
<-done
|
||||
<-done
|
||||
_ = cmd.Wait()
|
||||
close(lines)
|
||||
}()
|
||||
|
||||
go func() { // reader: only to notice the close
|
||||
defer cancel()
|
||||
for {
|
||||
if _, _, err := conn.Read(ctx); err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
ping := time.NewTicker(25 * time.Second)
|
||||
defer ping.Stop()
|
||||
// stdout and stderr are two pipes, and what Docker replays down them arrives in no
|
||||
// particular order. So the opening burst is held for a moment and sorted by the time
|
||||
// Docker stamped each line with; after that the log is live and arrives in order anyway.
|
||||
const burstFor = 300 * time.Millisecond
|
||||
warm := time.NewTimer(burstFor)
|
||||
defer warm.Stop()
|
||||
warmC := warm.C // nil once the burst has been flushed: a nil channel never fires
|
||||
var burst []logLine
|
||||
write := func(l logLine) bool {
|
||||
return conn.Write(ctx, websocket.MessageText, mustJSON(l)) == nil
|
||||
}
|
||||
flush := func() bool {
|
||||
sort.SliceStable(burst, func(i, j int) bool { return burst[i].at < burst[j].at })
|
||||
for _, l := range burst {
|
||||
if !write(l) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
burst = nil
|
||||
return true
|
||||
}
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case l, ok := <-lines:
|
||||
if !ok { // the container stopped, or never existed: say the last of it and go
|
||||
flush()
|
||||
return
|
||||
}
|
||||
if warmC != nil {
|
||||
burst = append(burst, l)
|
||||
if len(burst) < 10000 { // a container that never stops talking still gets read
|
||||
continue
|
||||
}
|
||||
if !flush() {
|
||||
return
|
||||
}
|
||||
continue
|
||||
}
|
||||
if !write(l) {
|
||||
return
|
||||
}
|
||||
case <-warmC:
|
||||
warmC = nil
|
||||
if !flush() {
|
||||
return
|
||||
}
|
||||
case <-ping.C:
|
||||
pctx, pc := context.WithTimeout(ctx, 10*time.Second)
|
||||
err := conn.Ping(pctx)
|
||||
pc()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func mustJSON(v any) []byte { b, _ := json.Marshal(v); return b }
|
||||
@@ -0,0 +1,351 @@
|
||||
package api
|
||||
|
||||
// Scheduled image updates.
|
||||
//
|
||||
// Dockhand kept three containers up to date nightly — unbound, nginx and pihole, which is
|
||||
// DNS and ingress — and that is the one thing it does that Ham Lab did not. This replaces
|
||||
// it, and deliberately in the same shape as the ZFS snapshot schedules: a real line in a
|
||||
// cron file Ham Lab owns, calling the binary, rather than a ticker inside the daemon.
|
||||
// Updating an image is not something that should stop happening because a web console is
|
||||
// down, and a cron line is visible to anyone looking at the machine the normal way.
|
||||
//
|
||||
// A schedule stores everything the run needs — project, compose files, how compose has to
|
||||
// be invoked for that stack — so the cron job is self-contained and does not have to
|
||||
// rediscover the world before it can act.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/compose"
|
||||
"hamlab/internal/cron"
|
||||
"hamlab/internal/storage"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// UpdateJob is a standing instruction to pull a stack's images, and optionally deploy them.
|
||||
type UpdateJob struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Stack string `json:"stack"`
|
||||
Machine string `json:"machine,omitempty"` // for display; the job runs where it was made
|
||||
// Services narrows it to some of the stack's services; empty is all of them.
|
||||
Services []string `json:"services,omitempty"`
|
||||
Cron string `json:"cron"`
|
||||
// Apply says whether a newer image is deployed or only fetched. Check-only still pulls,
|
||||
// so the Docker tab shows what is behind without a registry round trip.
|
||||
Apply bool `json:"apply"`
|
||||
Enabled bool `json:"enabled"`
|
||||
|
||||
// How to run compose for this stack, captured when the schedule was made.
|
||||
Spec composeSpec `json:"spec"`
|
||||
}
|
||||
|
||||
type composeSpec struct {
|
||||
Project string `json:"project"`
|
||||
Files []string `json:"files"`
|
||||
Via string `json:"via,omitempty"`
|
||||
Remote string `json:"remote,omitempty"`
|
||||
}
|
||||
|
||||
func (c composeSpec) toSpec(tool string) compose.Spec {
|
||||
return compose.Spec{Project: c.Project, Files: c.Files, Via: c.Via, Remote: c.Remote, Tool: tool}
|
||||
}
|
||||
|
||||
const dockerCronFile = "/etc/cron.d/hamlab-docker"
|
||||
|
||||
type dockerSchedState struct {
|
||||
Jobs []UpdateJob `json:"jobs"`
|
||||
}
|
||||
|
||||
func (s *Server) dsLoad() dockerSchedState {
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.DockerJobs })
|
||||
var d dockerSchedState
|
||||
if len(raw) > 0 {
|
||||
_ = json.Unmarshal(raw, &d)
|
||||
}
|
||||
if d.Jobs == nil {
|
||||
d.Jobs = []UpdateJob{}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func (s *Server) dsSave(d dockerSchedState) error {
|
||||
b, err := json.Marshal(d)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.store.Update(func(st *store.State) { st.DockerJobs = b })
|
||||
}
|
||||
|
||||
// dsCronText renders the schedules as cron lines. Same shape as the ZFS file: Ham Lab owns
|
||||
// it whole and rewrites it on every change.
|
||||
func dsCronText(jobs []UpdateJob, user, exe, config string) (string, error) {
|
||||
var b strings.Builder
|
||||
b.WriteString("# Ham Lab image-update schedules — written by the Docker tab, rewritten whole on every change.\n")
|
||||
b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
|
||||
b.WriteString("SHELL=/bin/sh\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
|
||||
for i := range jobs {
|
||||
j := &jobs[i]
|
||||
if !j.Enabled {
|
||||
b.WriteString("# " + j.Name + " — paused\n")
|
||||
continue
|
||||
}
|
||||
expr := strings.Join(strings.Fields(j.Cron), " ")
|
||||
if err := cron.ValidSchedule(expr); err != nil {
|
||||
return "", err
|
||||
}
|
||||
what := "pull only"
|
||||
if j.Apply {
|
||||
what = "pull and deploy"
|
||||
}
|
||||
b.WriteString("# " + j.Name + " — " + j.Stack + ", " + what + "\n")
|
||||
b.WriteString(expr + " " + user + " " + exe + " -config " + config + " docker-update " + j.ID + "\n\n")
|
||||
}
|
||||
return b.String(), nil
|
||||
}
|
||||
|
||||
func (s *Server) dsSyncCron(ctx context.Context, d dockerSchedState) error {
|
||||
live := false
|
||||
for i := range d.Jobs {
|
||||
if d.Jobs[i].Enabled {
|
||||
live = true
|
||||
}
|
||||
}
|
||||
if !live {
|
||||
return s.storage.RemoveRootFile(ctx, dockerCronFile)
|
||||
}
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
exe, _ = filepath.EvalSymlinks(exe)
|
||||
text, err := dsCronText(d.Jobs, s.user, exe, s.cfg.Path())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.storage.WriteRootFile(ctx, dockerCronFile, text)
|
||||
}
|
||||
|
||||
// repoint keeps a schedule working when what deployed its stack has been removed.
|
||||
//
|
||||
// A schedule captures how to invoke compose at the moment it is made, so the cron job never
|
||||
// has to rediscover the world at 3am. But "run compose inside this container" stops being
|
||||
// true the day that container is deleted — which is exactly what happens when an
|
||||
// orchestrator is retired, and would otherwise break every schedule made before it went.
|
||||
// So: if the container named is not running, the files are looked up again, through live
|
||||
// mounts and remembered ones both, and compose runs on the host instead.
|
||||
func repoint(ctx context.Context, db *store.Store, spec compose.Spec) compose.Spec {
|
||||
if spec.Via == "" || db == nil {
|
||||
return spec
|
||||
}
|
||||
if out, err := dockerRun(ctx, "inspect", "-f", "{{.State.Running}}", spec.Via); err == nil && strings.TrimSpace(string(out)) == "true" {
|
||||
return spec
|
||||
}
|
||||
st := stack{Name: spec.Project, File: spec.Files[0], Files: spec.Files}
|
||||
st.resolve(mountsSeen(ctx, db)) // read-only: this also runs from cron, where the daemon owns state.json
|
||||
if !st.Editable || st.Via != "" {
|
||||
return spec // still only reachable through a container, or not reachable at all
|
||||
}
|
||||
log.Printf("docker schedule %s: %s is gone, running compose here against %s", spec.Project, spec.Via, st.HostFiles[0])
|
||||
spec.Files, spec.Via = st.HostFiles, ""
|
||||
return spec
|
||||
}
|
||||
|
||||
// RunUpdate pulls, and deploys when the schedule says to. Shared with the CLI so the button
|
||||
// and the cron line cannot drift apart.
|
||||
func RunUpdate(ctx context.Context, db *store.Store, j UpdateJob, tool string) (string, error) {
|
||||
if len(j.Spec.Files) == 0 {
|
||||
return "", errors.New("this schedule has no compose file recorded — open it in the Docker tab and save it again")
|
||||
}
|
||||
spec := repoint(ctx, db, j.Spec.toSpec(tool))
|
||||
var log strings.Builder
|
||||
run := func(args ...string) error {
|
||||
cmd := compose.Cmd(ctx, spec, args...)
|
||||
out, err := cmd.CombinedOutput()
|
||||
log.Write(out)
|
||||
return err
|
||||
}
|
||||
args := append([]string{"pull"}, j.Services...)
|
||||
if err := run(args...); err != nil {
|
||||
return log.String(), err
|
||||
}
|
||||
if !j.Apply {
|
||||
return log.String(), nil
|
||||
}
|
||||
up := append([]string{"up", "-d"}, j.Services...)
|
||||
if err := run(up...); err != nil {
|
||||
return log.String(), err
|
||||
}
|
||||
return log.String(), nil
|
||||
}
|
||||
|
||||
// ---- endpoints ----
|
||||
|
||||
func (s *Server) dsList(w http.ResponseWriter, _ *http.Request) {
|
||||
d := s.dsLoad()
|
||||
now := time.Now()
|
||||
out := make([]map[string]any, 0, len(d.Jobs))
|
||||
for _, j := range d.Jobs {
|
||||
m := map[string]any{"job": j, "last": storage.ReadRun(s.cfg.DataDir(), "docker-"+j.ID)}
|
||||
if j.Enabled {
|
||||
if n, ok := cron.NextRun(strings.Join(strings.Fields(j.Cron), " "), now); ok {
|
||||
m["next"] = n
|
||||
}
|
||||
}
|
||||
out = append(out, m)
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"jobs": out, "file": dockerCronFile})
|
||||
}
|
||||
|
||||
func (s *Server) dsPut(w http.ResponseWriter, r *http.Request) {
|
||||
var j UpdateJob
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 32<<10)).Decode(&j); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
j.Cron = strings.Join(strings.Fields(j.Cron), " ")
|
||||
if err := cron.ValidSchedule(j.Cron); err != nil {
|
||||
jsonErr(w, 400, "schedule: "+err.Error())
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(j.Stack) == "" {
|
||||
jsonErr(w, 400, "which stack?")
|
||||
return
|
||||
}
|
||||
// resolve the stack now, so the cron job never has to
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
stacks, err := s.findStacks(ctx)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
found := false
|
||||
for _, st := range stacks {
|
||||
if st.Name == j.Stack {
|
||||
// seenFiles is the list as compose itself sees it: host paths normally, the
|
||||
// orchestrator container's own paths when compose runs inside one. Getting this
|
||||
// backwards gives a cron line that works from the console and fails at 3am.
|
||||
j.Spec = composeSpec{Project: st.Name, Files: seenFiles(st, -1, ""), Via: st.Via, Remote: st.Remote}
|
||||
if len(j.Spec.Files) == 0 {
|
||||
jsonErr(w, 409, "that stack's compose file isn't anywhere this console can reach")
|
||||
return
|
||||
}
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no stack called "+j.Stack+" on this machine")
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(j.Name) == "" {
|
||||
j.Name = j.Stack
|
||||
}
|
||||
d := s.dsLoad()
|
||||
replaced := false
|
||||
for i := range d.Jobs {
|
||||
if d.Jobs[i].ID == j.ID && j.ID != "" {
|
||||
d.Jobs[i] = j
|
||||
replaced = true
|
||||
}
|
||||
}
|
||||
if !replaced {
|
||||
b := make([]byte, 6)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
j.ID = hex.EncodeToString(b)
|
||||
d.Jobs = append(d.Jobs, j)
|
||||
}
|
||||
if err := s.dsSave(d); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.dsSyncCron(r.Context(), d); err != nil {
|
||||
jsonErr(w, 500, "saved, but writing "+dockerCronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("docker schedule %q saved by %s (%s, %q, %s)", j.Name, clientIP(r), j.Stack, j.Cron, map[bool]string{true: "pull and deploy", false: "pull only"}[j.Apply])
|
||||
jsonOK(w, 200, j)
|
||||
}
|
||||
|
||||
func (s *Server) dsDelete(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
d := s.dsLoad()
|
||||
kept := d.Jobs[:0]
|
||||
found := false
|
||||
for _, j := range d.Jobs {
|
||||
if j.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, j)
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
d.Jobs = kept
|
||||
if err := s.dsSave(d); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.dsSyncCron(r.Context(), d); err != nil {
|
||||
jsonErr(w, 500, "removed, but writing "+dockerCronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
storage.ForgetRun(s.cfg.DataDir(), "docker-"+id)
|
||||
log.Printf("docker schedule %s removed by %s", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) dsRun(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
var job UpdateJob
|
||||
found := false
|
||||
for _, j := range s.dsLoad().Jobs {
|
||||
if j.ID == id {
|
||||
job, found = j, true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
// Outliving the request: a pull the browser gave up on still finishes, but not forever.
|
||||
ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 30*time.Minute)
|
||||
defer cancel()
|
||||
out, err := RunUpdate(ctx, s.store, job, dockerTool())
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: job.Stack}
|
||||
if err != nil {
|
||||
rec.Error = err.Error()
|
||||
}
|
||||
_ = storage.WriteRun(s.cfg.DataDir(), "docker-"+id, rec)
|
||||
stackUpdatesForget("/" + job.Stack)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error()+"\n"+tail(out, 2000))
|
||||
return
|
||||
}
|
||||
log.Printf("docker schedule %q run by %s", job.Name, clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"output": tail(out, 4000)})
|
||||
}
|
||||
|
||||
func tail(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return "…" + s[len(s)-n:]
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
package api
|
||||
|
||||
// Guest logins: a temporary account for handing to somebody you want to show around. It is
|
||||
// an ordinary account with a clock on it — every tab, every action, on this machine and on
|
||||
// every machine this hub manages — that stops working at its expiry (half an hour by
|
||||
// default) or the moment it is revoked.
|
||||
//
|
||||
// Deliberately not a reduced account: Mitchell, 2026-09-27, replacing a read-only version
|
||||
// built the same day. He gives these to people he trusts and revokes them otherwise, and a
|
||||
// console with half its buttons missing is not the console he wanted to show off. What that
|
||||
// means in practice: a guest has the Terminal tab, which is a shell as the user the daemon
|
||||
// runs as — root on every host here — and can make an account or a token of its own that
|
||||
// outlives the guest login.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const (
|
||||
guestPrefix = "guest-"
|
||||
guestTTLDefault = 30 * time.Minute
|
||||
guestTTLMax = 24 * time.Hour
|
||||
guestReapEvery = time.Minute
|
||||
)
|
||||
|
||||
// ---- the accounts themselves ----
|
||||
|
||||
func (s *Server) listGuests(w http.ResponseWriter, r *http.Request) {
|
||||
now := time.Now()
|
||||
counts := s.sessions.Count()
|
||||
out := []map[string]any{}
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, u := range c.Users {
|
||||
if !u.Guest || u.Expired(now) {
|
||||
continue
|
||||
}
|
||||
out = append(out, map[string]any{
|
||||
"name": u.Name, "created": u.Created, "expires": u.Expires,
|
||||
"issued_by": u.IssuedBy, "sessions": counts[u.Name], "used": !u.LastSeen.IsZero(),
|
||||
})
|
||||
}
|
||||
})
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
// addGuest makes a guest login and hands back its password — the only time it is ever
|
||||
// readable, here as everywhere else a password is generated.
|
||||
func (s *Server) addGuest(w http.ResponseWriter, r *http.Request) {
|
||||
me := actorOf(r)
|
||||
var body struct {
|
||||
Minutes int `json:"minutes"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
ttl := guestTTLDefault
|
||||
if body.Minutes > 0 {
|
||||
ttl = time.Duration(body.Minutes) * time.Minute
|
||||
}
|
||||
if ttl > guestTTLMax {
|
||||
jsonErr(w, 400, "a guest login lasts at most 24 hours")
|
||||
return
|
||||
}
|
||||
pw, err := auth.GuestPassword()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
hash, err := auth.HashPassword(pw)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
var name string
|
||||
for try := 0; try < 8 && name == ""; try++ {
|
||||
suffix, err := auth.GuestSuffix()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
candidate := guestPrefix + suffix
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
if _, taken := c.User(candidate); !taken {
|
||||
name = candidate
|
||||
}
|
||||
})
|
||||
}
|
||||
if name == "" {
|
||||
jsonErr(w, 500, "could not find a free guest name")
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
expires := now.Add(ttl)
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
c.Users = append(c.Users, config.User{Name: name, Hash: hash, Created: now, Guest: true, Expires: expires, IssuedBy: me.User})
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("guest login %q issued until %s (by %s)", name, expires.Format(time.RFC3339), clientIP(r))
|
||||
jsonOK(w, 200, map[string]any{"name": name, "password": pw, "expires": expires, "minutes": int(ttl.Minutes())})
|
||||
}
|
||||
|
||||
// revokeGuest ends a guest login now. No password: it only ever takes access away, and the
|
||||
// whole point of the thing is that ending it is one tap.
|
||||
func (s *Server) revokeGuest(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
found := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.Users[:0]
|
||||
for _, u := range c.Users {
|
||||
if u.Name == name && u.Guest {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
c.Users = kept
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such guest login")
|
||||
return
|
||||
}
|
||||
s.forgetGuest(name)
|
||||
log.Printf("guest login %q revoked (by %s)", name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// forgetGuest signs a guest out everywhere and drops the look they left behind.
|
||||
func (s *Server) forgetGuest(name string) {
|
||||
s.sessions.RevokeUser(name)
|
||||
_ = s.store.Update(func(st *store.State) { delete(st.Users, name) })
|
||||
}
|
||||
|
||||
// watchGuests clears out guest logins whose time is up. Expiry is enforced on every request
|
||||
// as well — this is what keeps the config file from collecting them, and what signs a guest
|
||||
// out of a console they left open rather than waiting for their next click.
|
||||
func (s *Server) watchGuests() {
|
||||
for {
|
||||
s.reapGuests()
|
||||
time.Sleep(guestReapEvery)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) reapGuests() {
|
||||
now := time.Now()
|
||||
var gone []string
|
||||
var have bool
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, u := range c.Users {
|
||||
if u.Guest {
|
||||
have = true
|
||||
if u.Expired(now) {
|
||||
gone = append(gone, u.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
if !have || len(gone) == 0 {
|
||||
return
|
||||
}
|
||||
expired := map[string]bool{}
|
||||
for _, n := range gone {
|
||||
expired[n] = true
|
||||
}
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.Users[:0]
|
||||
for _, u := range c.Users {
|
||||
if expired[u.Name] {
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
c.Users = kept
|
||||
}); err != nil {
|
||||
log.Printf("guests: %v", err)
|
||||
return
|
||||
}
|
||||
for _, n := range gone {
|
||||
s.forgetGuest(n)
|
||||
}
|
||||
log.Printf("guest login(s) expired: %s", strings.Join(gone, ", "))
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// A guest login is an ordinary account with a clock on it, so the clock is the thing worth
|
||||
// testing: one whose time is up is stopped at the door, whether or not the minute-by-minute
|
||||
// sweep has reached their account yet, and their session is dropped on the way — so the
|
||||
// console they left open goes back to the login form rather than carrying on.
|
||||
func TestExpiredGuestIsRefusedAndSignedOut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
expires time.Time
|
||||
wantOK bool
|
||||
}{
|
||||
{"still has time", time.Now().Add(10 * time.Minute), true},
|
||||
{"time is up", time.Now().Add(-time.Second), false},
|
||||
{"no expiry at all", time.Time{}, false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cfg := config.Default(filepath.Join(dir, "config.json"))
|
||||
if err := cfg.Update(func(c *config.Config) {
|
||||
c.Users = []config.User{{Name: "guest-abcd", Hash: "x", Guest: true, Expires: tc.expires}}
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err := store.Open(filepath.Join(dir, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{cfg: cfg, store: st, sessions: auth.NewSessions(st)}
|
||||
tok, err := s.sessions.Create("guest-abcd", "10.0.0.5", "curl")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := httptest.NewRequest("GET", "/api/me", nil)
|
||||
r.AddCookie(&http.Cookie{Name: cookieName, Value: tok})
|
||||
w := httptest.NewRecorder()
|
||||
a, ok, answered := s.cookieActor(w, r)
|
||||
if ok != tc.wantOK {
|
||||
t.Fatalf("ok = %v, want %v", ok, tc.wantOK)
|
||||
}
|
||||
if tc.wantOK {
|
||||
if !a.Guest || a.User != "guest-abcd" {
|
||||
t.Errorf("actor = %+v, want the guest account marked as one", a)
|
||||
}
|
||||
if answered {
|
||||
t.Error("answered the caller for a guest that still has time")
|
||||
}
|
||||
return
|
||||
}
|
||||
if !answered || w.Code != 401 {
|
||||
t.Errorf("answered = %v, code = %d, want true and 401", answered, w.Code)
|
||||
}
|
||||
if _, live := s.sessions.Lookup(tok, "10.0.0.5"); live {
|
||||
t.Error("the session of an expired guest is still live")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package api
|
||||
|
||||
// The Hardware tab: what the machine is made of. Read once and remembered — none of it
|
||||
// changes while the kernel runs — with ?refresh=1 for the one thing that can, a drive
|
||||
// plugged in or pulled out.
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
)
|
||||
|
||||
func (s *Server) hardwareReport(w http.ResponseWriter, r *http.Request) {
|
||||
jsonOK(w, 200, s.hardware.Report(r.Context(), r.URL.Query().Get("refresh") == "1"))
|
||||
}
|
||||
@@ -0,0 +1,434 @@
|
||||
package api
|
||||
|
||||
// IP-KVMs: a device's own web page, shown inside Ham Lab.
|
||||
//
|
||||
// The whole difficulty is that Ham Lab is served over TLS and a KVM usually is not, so a
|
||||
// browser refuses to put the device's page in an iframe — mixed content, blocked before
|
||||
// anything loads. The fix is to serve the device from Ham Lab's own origin, which also
|
||||
// means it inherits Ham Lab's login instead of needing a second one, and makes the iframe
|
||||
// same-origin.
|
||||
//
|
||||
// Proxying an arbitrary web app under a path prefix is the hard part, and nothing here
|
||||
// knows anything about any particular device. A device UI asks for its assets by absolute
|
||||
// path — "/assets/app.js", "/api/whatever" — which a prefix cannot rewrite away. So the
|
||||
// prefix is only the entry point: any request Ham Lab does not serve itself is routed to
|
||||
// whichever KVM the asking page belongs to, worked out from the Referer, and from a cookie
|
||||
// when there is no Referer (a WebSocket upgrade, say). Ham Lab's own routes always win, so
|
||||
// a device can never shadow the console.
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
const kvmPrefix = "/kvm/"
|
||||
const kvmCookie = "hamlab_kvm"
|
||||
|
||||
type kvmProxies struct {
|
||||
mu sync.Mutex
|
||||
m map[string]*httputil.ReverseProxy
|
||||
}
|
||||
|
||||
func (s *Server) kvm(id string) (config.KVM, bool) {
|
||||
var found config.KVM
|
||||
ok := false
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, k := range c.KVMs {
|
||||
if k.ID == id {
|
||||
found, ok = k, true
|
||||
}
|
||||
}
|
||||
})
|
||||
return found, ok
|
||||
}
|
||||
|
||||
// kvmProxy is the reverse proxy for one device, made once and kept.
|
||||
func (s *Server) kvmProxy(k config.KVM) (*httputil.ReverseProxy, error) {
|
||||
s.kvms.mu.Lock()
|
||||
defer s.kvms.mu.Unlock()
|
||||
if s.kvms.m == nil {
|
||||
s.kvms.m = map[string]*httputil.ReverseProxy{}
|
||||
}
|
||||
if p, ok := s.kvms.m[k.ID]; ok {
|
||||
return p, nil
|
||||
}
|
||||
target, err := url.Parse(k.URL)
|
||||
if err != nil || target.Host == "" {
|
||||
return nil, errors.New("this KVM's address is not a URL")
|
||||
}
|
||||
prefix := kvmPrefix + k.ID
|
||||
p := &httputil.ReverseProxy{
|
||||
Director: func(r *http.Request) {
|
||||
r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
|
||||
// Entry through the prefix keeps the path after it; anything else arrived by its
|
||||
// own absolute path and is passed through untouched.
|
||||
if rest, ok := strings.CutPrefix(r.URL.Path, prefix); ok {
|
||||
if rest == "" {
|
||||
rest = "/"
|
||||
}
|
||||
r.URL.Path = rest
|
||||
}
|
||||
r.URL.RawPath = ""
|
||||
r.Host = target.Host
|
||||
// Whether the console itself is on TLS: the response rewriter needs to know, and by
|
||||
// then only the outbound request is in hand.
|
||||
if secure(r) {
|
||||
r.Header.Set(kvmSecureHeader, "1")
|
||||
} else {
|
||||
r.Header.Del(kvmSecureHeader)
|
||||
}
|
||||
// The device gets its own host and none of Ham Lab's session.
|
||||
r.Header.Del("Origin")
|
||||
stripCookie(r, cookieName, kvmCookie)
|
||||
// What is left is the device's own login. A browser that has it teaches the hub; one
|
||||
// that has not is lent what the hub remembers, so the device sees a signed-in client.
|
||||
if own := r.Cookies(); len(own) > 0 {
|
||||
s.rememberKVMCookies(k.ID, own)
|
||||
} else {
|
||||
for name, val := range s.kvmCookies(k.ID) {
|
||||
r.AddCookie(&http.Cookie{Name: name, Value: val})
|
||||
}
|
||||
}
|
||||
},
|
||||
FlushInterval: -1, // a video stream must not be buffered
|
||||
ModifyResponse: func(resp *http.Response) error {
|
||||
// The device's own framing and transport policy would stop it being embedded, or
|
||||
// send a browser off to plain HTTP. Ham Lab is the origin now, and says what it
|
||||
// serves; its own headers are added on the way out.
|
||||
for _, h := range []string{"X-Frame-Options", "Content-Security-Policy", "Content-Security-Policy-Report-Only", "Strict-Transport-Security", "Referrer-Policy"} {
|
||||
resp.Header.Del(h)
|
||||
}
|
||||
// A cookie the device sets by header is kept for Ham Lab's origin — so its Domain
|
||||
// (the device's) goes, Secure follows the console rather than the device, and the
|
||||
// hub remembers it too.
|
||||
if set := resp.Cookies(); len(set) > 0 {
|
||||
https := resp.Request != nil && resp.Request.Header.Get(kvmSecureHeader) == "1"
|
||||
resp.Header.Del("Set-Cookie")
|
||||
for _, c := range set {
|
||||
c.Domain = ""
|
||||
if c.Path == "" {
|
||||
c.Path = "/"
|
||||
}
|
||||
c.Secure = c.Secure && https
|
||||
if v := c.String(); v != "" {
|
||||
resp.Header.Add("Set-Cookie", v)
|
||||
}
|
||||
}
|
||||
s.rememberKVMCookies(k.ID, set)
|
||||
}
|
||||
// A redirect to the device's own root has to come back through the prefix.
|
||||
if loc := resp.Header.Get("Location"); loc != "" {
|
||||
if u, err := url.Parse(loc); err == nil && (u.Host == "" || u.Host == target.Host) && strings.HasPrefix(u.Path, "/") && !strings.HasPrefix(u.Path, prefix) {
|
||||
u.Scheme, u.Host = "", ""
|
||||
u.Path = prefix + u.Path
|
||||
resp.Header.Set("Location", u.String())
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": k.Name + " unreachable: " + err.Error()})
|
||||
},
|
||||
}
|
||||
s.kvms.m[k.ID] = p
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// kvmSecureHeader rides on the outbound request to say whether the console was reached
|
||||
// over TLS; the device ignores it.
|
||||
const kvmSecureHeader = "X-Hamlab-Secure"
|
||||
|
||||
// kvmCookies is what the hub remembers of a device's own cookies — its login, in practice.
|
||||
func (s *Server) kvmCookies(id string) map[string]string {
|
||||
var out map[string]string
|
||||
s.store.View(func(st *store.State) {
|
||||
if m := st.KVMCookies[id]; len(m) > 0 {
|
||||
out = make(map[string]string, len(m))
|
||||
for k, v := range m {
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// rememberKVMCookies keeps the cookies a browser (or the device's own Set-Cookie) carries
|
||||
// for a KVM, so that a browser without them can be lent them. A cookie emptied or expired
|
||||
// by the device is forgotten. Written only when something changed: this runs on every
|
||||
// proxied request.
|
||||
func (s *Server) rememberKVMCookies(id string, cs []*http.Cookie) {
|
||||
have := s.kvmCookies(id)
|
||||
changed := false
|
||||
for _, c := range cs {
|
||||
if c.Name == "" || c.Name == cookieName || c.Name == kvmCookie {
|
||||
continue
|
||||
}
|
||||
gone := c.Value == "" || c.MaxAge < 0 || (!c.Expires.IsZero() && c.Expires.Before(time.Now()))
|
||||
if gone {
|
||||
if _, ok := have[c.Name]; ok {
|
||||
changed = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
if have[c.Name] != c.Value {
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
if !changed {
|
||||
return
|
||||
}
|
||||
_ = s.store.Update(func(st *store.State) {
|
||||
if st.KVMCookies == nil {
|
||||
st.KVMCookies = map[string]map[string]string{}
|
||||
}
|
||||
m := st.KVMCookies[id]
|
||||
if m == nil {
|
||||
m = map[string]string{}
|
||||
}
|
||||
for _, c := range cs {
|
||||
if c.Name == "" || c.Name == cookieName || c.Name == kvmCookie {
|
||||
continue
|
||||
}
|
||||
if c.Value == "" || c.MaxAge < 0 || (!c.Expires.IsZero() && c.Expires.Before(time.Now())) {
|
||||
delete(m, c.Name)
|
||||
} else {
|
||||
m[c.Name] = c.Value
|
||||
}
|
||||
}
|
||||
if len(m) == 0 {
|
||||
delete(st.KVMCookies, id)
|
||||
} else {
|
||||
st.KVMCookies[id] = m
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// forgetKVMCookies drops a device's remembered login — when the device is removed.
|
||||
func (s *Server) forgetKVMCookies(id string) {
|
||||
_ = s.store.Update(func(st *store.State) { delete(st.KVMCookies, id) })
|
||||
}
|
||||
|
||||
// lendKVMCookies hands the browser the device cookies the hub remembers and it lacks, as
|
||||
// the device's own page would have set them: readable by its scripts, for Ham Lab's whole
|
||||
// origin, kept for half a year. The device's SPA checks for its token client-side before it
|
||||
// ever asks the server, so lending it only on the wire would still land on the login form.
|
||||
func (s *Server) lendKVMCookies(w http.ResponseWriter, r *http.Request, id string) {
|
||||
for name, val := range s.kvmCookies(id) {
|
||||
if _, err := r.Cookie(name); err == nil {
|
||||
continue
|
||||
}
|
||||
http.SetCookie(w, &http.Cookie{Name: name, Value: val, Path: "/", MaxAge: 180 * 24 * 3600, Secure: secure(r), SameSite: http.SameSiteLaxMode})
|
||||
}
|
||||
}
|
||||
|
||||
// stripCookie removes named cookies from a request, so Ham Lab's session is never handed to
|
||||
// a device and the device never sees which KVM the browser is looking at.
|
||||
func stripCookie(r *http.Request, names ...string) {
|
||||
cs := r.Cookies()
|
||||
kept := cs[:0]
|
||||
for _, c := range cs {
|
||||
drop := false
|
||||
for _, n := range names {
|
||||
if c.Name == n {
|
||||
drop = true
|
||||
}
|
||||
}
|
||||
if !drop {
|
||||
kept = append(kept, c)
|
||||
}
|
||||
}
|
||||
r.Header.Del("Cookie")
|
||||
for _, c := range kept {
|
||||
r.AddCookie(c)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) forgetKVMProxy(id string) {
|
||||
s.kvms.mu.Lock()
|
||||
delete(s.kvms.m, id)
|
||||
s.kvms.mu.Unlock()
|
||||
}
|
||||
|
||||
// serveKVM is the entry point: /kvm/<id>/… . It also drops a cookie naming the device, so
|
||||
// that requests the page makes later by absolute path can be traced back to it.
|
||||
func (s *Server) serveKVM(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
k, ok := s.kvm(id)
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such KVM")
|
||||
return
|
||||
}
|
||||
p, err := s.kvmProxy(k)
|
||||
if err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
http.SetCookie(w, &http.Cookie{Name: kvmCookie, Value: id, Path: "/", HttpOnly: true, Secure: secure(r), SameSite: http.SameSiteLaxMode})
|
||||
s.lendKVMCookies(w, r, id)
|
||||
p.ServeHTTP(w, r)
|
||||
}
|
||||
|
||||
// kvmFor works out which device a request belongs to when it arrived by absolute path
|
||||
// rather than through the prefix: the Referer names the page that asked, and a cookie
|
||||
// covers what has no Referer at all. Nothing device-specific: it never has to know which
|
||||
// paths a given KVM uses.
|
||||
func (s *Server) kvmFor(r *http.Request) (config.KVM, bool) {
|
||||
if ref := r.Header.Get("Referer"); ref != "" {
|
||||
if u, err := url.Parse(ref); err == nil {
|
||||
if rest, ok := strings.CutPrefix(u.Path, kvmPrefix); ok {
|
||||
if id, _, _ := strings.Cut(rest, "/"); id != "" {
|
||||
if k, ok := s.kvm(id); ok {
|
||||
return k, true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// asked for by a page that is not a KVM: not ours to answer
|
||||
return config.KVM{}, false
|
||||
}
|
||||
if c, err := r.Cookie(kvmCookie); err == nil && c.Value != "" {
|
||||
return s.kvm(c.Value)
|
||||
}
|
||||
return config.KVM{}, false
|
||||
}
|
||||
|
||||
// kvmFallback answers a request Ham Lab does not serve itself, when it came from inside a
|
||||
// KVM's page. It returns false when the request has nothing to do with a KVM, and the
|
||||
// caller carries on with its own 404.
|
||||
func (s *Server) kvmFallback(w http.ResponseWriter, r *http.Request) bool {
|
||||
k, ok := s.kvmFor(r)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
p, err := s.kvmProxy(k)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
p.ServeHTTP(w, r)
|
||||
return true
|
||||
}
|
||||
|
||||
func machineLabel(m string) string {
|
||||
if m == "" {
|
||||
return "this host"
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ---- the list ----
|
||||
|
||||
func (s *Server) listKVMs(w http.ResponseWriter, _ *http.Request) {
|
||||
out := []config.KVM{}
|
||||
s.cfg.View(func(c *config.Config) { out = append(out, c.KVMs...) })
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
// kvmForMachine is the KVM attached to a machine, "" being the hub itself.
|
||||
func (s *Server) kvmForMachine(machine string) (config.KVM, bool) {
|
||||
var found config.KVM
|
||||
ok := false
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, k := range c.KVMs {
|
||||
if k.Machine == machine {
|
||||
found, ok = k, true
|
||||
}
|
||||
}
|
||||
})
|
||||
return found, ok
|
||||
}
|
||||
|
||||
func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name, URL, Machine string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
raw := strings.TrimSpace(body.URL)
|
||||
if raw != "" && !strings.Contains(raw, "://") {
|
||||
raw = "http://" + raw // what you would type: an address, not a scheme
|
||||
}
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
jsonErr(w, 400, "address: http(s)://host, or just the host")
|
||||
return
|
||||
}
|
||||
name := strings.TrimSpace(body.Name)
|
||||
if name == "" {
|
||||
name = u.Host
|
||||
}
|
||||
if len(name) > maxNameLen {
|
||||
jsonErr(w, 400, "that name is too long")
|
||||
return
|
||||
}
|
||||
b := make([]byte, 6)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
machine := strings.TrimSpace(body.Machine)
|
||||
k := config.KVM{ID: hex.EncodeToString(b), Name: name, URL: u.Scheme + "://" + u.Host + strings.TrimSuffix(u.Path, "/"), Machine: machine, Added: time.Now()}
|
||||
// One per machine: a second one for the same host replaces the first rather than sitting
|
||||
// beside it, since the tab shows the machine you are looking at.
|
||||
var replaced string
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.KVMs[:0]
|
||||
for _, old := range c.KVMs {
|
||||
if old.Machine == machine {
|
||||
replaced = old.ID
|
||||
continue
|
||||
}
|
||||
kept = append(kept, old)
|
||||
}
|
||||
c.KVMs = append(kept, k)
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if replaced != "" {
|
||||
s.forgetKVMProxy(replaced)
|
||||
}
|
||||
log.Printf("kvm %q added for %s (%s) by %s", k.Name, machineLabel(machine), k.URL, clientIP(r))
|
||||
jsonOK(w, 201, k)
|
||||
}
|
||||
|
||||
func (s *Server) removeKVM(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
found := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
kept := c.KVMs[:0]
|
||||
for _, k := range c.KVMs {
|
||||
if k.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, k)
|
||||
}
|
||||
c.KVMs = kept
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such KVM")
|
||||
return
|
||||
}
|
||||
s.forgetKVMProxy(id)
|
||||
s.forgetKVMCookies(id)
|
||||
log.Printf("kvm %s removed by %s", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// A KVM's own login is a cookie its page sets in one browser. The hub remembers it from the
|
||||
// requests that carry it, lends it to a browser that lacks it — both on the wire to the
|
||||
// device and back to the browser, since the device's page checks for it client-side — and
|
||||
// forgets it when the device says it is gone or the KVM is removed.
|
||||
func TestKVMCookiesAreRememberedAndLent(t *testing.T) {
|
||||
var seen []string // the Cookie header the fake device received, per request
|
||||
device := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
seen = append(seen, r.Header.Get("Cookie"))
|
||||
if r.URL.Path == "/api/auth/login" {
|
||||
// a device that sets its token by header, with its own domain and Secure
|
||||
http.SetCookie(w, &http.Cookie{Name: "nano-kvm-token", Value: "tok-1", Domain: "10.0.0.217", Secure: true})
|
||||
}
|
||||
if r.URL.Path == "/api/auth/logout" {
|
||||
http.SetCookie(w, &http.Cookie{Name: "nano-kvm-token", Value: "", MaxAge: -1})
|
||||
}
|
||||
w.WriteHeader(200)
|
||||
}))
|
||||
defer device.Close()
|
||||
|
||||
dir := t.TempDir()
|
||||
cfg := config.Default(filepath.Join(dir, "config.json"))
|
||||
if err := cfg.Update(func(c *config.Config) {
|
||||
c.KVMs = []config.KVM{{ID: "abc123", Name: "test", URL: device.URL, Machine: ""}}
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err := store.Open(filepath.Join(dir, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &Server{cfg: cfg, store: st, sessions: auth.NewSessions(st)}
|
||||
k, _ := s.kvm("abc123")
|
||||
p, err := s.kvmProxy(k)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ask := func(path string, cookies ...*http.Cookie) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("GET", path, nil)
|
||||
r.SetPathValue("id", "abc123")
|
||||
for _, c := range cookies {
|
||||
r.AddCookie(c)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
if strings.HasPrefix(path, kvmPrefix) {
|
||||
s.serveKVM(w, r)
|
||||
} else {
|
||||
p.ServeHTTP(w, r)
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// 1. the device sets its token by header: it reaches the browser without the device's
|
||||
// domain or Secure (plain-http console), and the hub remembers it
|
||||
w := ask("/api/auth/login", &http.Cookie{Name: cookieName, Value: "hamlab-session"})
|
||||
var set *http.Cookie
|
||||
for _, c := range w.Result().Cookies() {
|
||||
if c.Name == "nano-kvm-token" {
|
||||
set = c
|
||||
}
|
||||
}
|
||||
if set == nil || set.Domain != "" || set.Secure || set.Value != "tok-1" || set.Path != "/" {
|
||||
t.Fatalf("rewritten Set-Cookie = %+v, want nano-kvm-token=tok-1 for / without Domain or Secure", set)
|
||||
}
|
||||
if got := s.kvmCookies("abc123"); got["nano-kvm-token"] != "tok-1" {
|
||||
t.Fatalf("remembered = %v, want the token", got)
|
||||
}
|
||||
if strings.Contains(seen[0], "hamlab_session") {
|
||||
t.Fatalf("the device was handed Ham Lab's session: %q", seen[0])
|
||||
}
|
||||
|
||||
// 2. a browser that has the cookie teaches a newer value
|
||||
ask("/api/vm/info", &http.Cookie{Name: "nano-kvm-token", Value: "tok-2"}, &http.Cookie{Name: cookieName, Value: "x"})
|
||||
if got := s.kvmCookies("abc123"); got["nano-kvm-token"] != "tok-2" {
|
||||
t.Fatalf("remembered = %v, want tok-2", got)
|
||||
}
|
||||
if seen[1] != "nano-kvm-token=tok-2" {
|
||||
t.Fatalf("device saw %q, want the browser's own cookie only", seen[1])
|
||||
}
|
||||
|
||||
// 3. a browser without it is lent it: on the wire, and back as a Set-Cookie it can read
|
||||
w = ask(kvmPrefix+"abc123/", &http.Cookie{Name: cookieName, Value: "x"})
|
||||
if seen[2] != "nano-kvm-token=tok-2" {
|
||||
t.Fatalf("device saw %q from a browser with no cookie, want the remembered token", seen[2])
|
||||
}
|
||||
lent := false
|
||||
for _, c := range w.Result().Cookies() {
|
||||
if c.Name == "nano-kvm-token" && c.Value == "tok-2" && !c.HttpOnly && c.Path == "/" && c.MaxAge > 0 {
|
||||
lent = true
|
||||
}
|
||||
}
|
||||
if !lent {
|
||||
t.Fatalf("browser was not lent the token: %v", w.Result().Cookies())
|
||||
}
|
||||
|
||||
// 4. the device logging out forgets it
|
||||
ask("/api/auth/logout", &http.Cookie{Name: "nano-kvm-token", Value: "tok-2"})
|
||||
if got := s.kvmCookies("abc123"); got != nil {
|
||||
t.Fatalf("after logout remembered = %v, want nothing", got)
|
||||
}
|
||||
|
||||
// 5. removing the KVM clears whatever is left
|
||||
s.rememberKVMCookies("abc123", []*http.Cookie{{Name: "nano-kvm-token", Value: "tok-3"}})
|
||||
s.forgetKVMCookies("abc123")
|
||||
if got := s.kvmCookies("abc123"); got != nil {
|
||||
t.Fatalf("after remove remembered = %v, want nothing", got)
|
||||
}
|
||||
_ = url.Parse
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package api
|
||||
|
||||
// The Network tab: the machine's interfaces, how the internet looks from here, and the speed
|
||||
// test. One machine at a time, like every other tab — the test runs on the machine in the
|
||||
// switcher, so the number is that machine's line and not the hub's.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/network"
|
||||
"hamlab/internal/sensors"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// networkOverview is everything the tab draws at once: the interfaces, the outside view, the
|
||||
// last test and the ones before it.
|
||||
func (s *Server) networkOverview(w http.ResponseWriter, _ *http.Request) {
|
||||
run := s.net.Test.State()
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"report": network.Look(s.hostName()),
|
||||
"outside": s.net.Outside(),
|
||||
"run": run,
|
||||
"history": s.net.Test.History(),
|
||||
"method": network.Method,
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) speedtestState(w http.ResponseWriter, _ *http.Request) {
|
||||
jsonOK(w, 200, s.net.Test.State())
|
||||
}
|
||||
|
||||
func (s *Server) speedtestStart(w http.ResponseWriter, r *http.Request) {
|
||||
run, err := s.net.Test.Start()
|
||||
if err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("speed test started on %s (by %s)", s.hostName(), clientIP(r))
|
||||
jsonOK(w, 200, run)
|
||||
}
|
||||
|
||||
func (s *Server) speedtestCancel(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.net.Test.Cancel() {
|
||||
jsonErr(w, 404, "no speed test is running")
|
||||
return
|
||||
}
|
||||
log.Printf("speed test cancelled on %s (by %s)", s.hostName(), clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// ---- keeping the results ----
|
||||
|
||||
// savedNetwork is what survives a restart: the recent results, so the history is a history
|
||||
// rather than whatever has happened since the daemon last started.
|
||||
type savedNetwork struct {
|
||||
Results []network.Result `json:"results,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Server) loadNetwork() []network.Result {
|
||||
var saved savedNetwork
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.Network })
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &saved); err != nil {
|
||||
log.Printf("network: kept results unreadable (%v); starting a fresh history", err)
|
||||
}
|
||||
}
|
||||
return saved.Results
|
||||
}
|
||||
|
||||
// keepResult is called when a test finishes: append it to what is on disk.
|
||||
func (s *Server) keepResult(r network.Result) {
|
||||
if r.Error != "" {
|
||||
log.Printf("speed test on %s failed: %s", s.hostName(), r.Error)
|
||||
} else {
|
||||
log.Printf("speed test on %s: down %s, up %s, latency %.1f ms via %s",
|
||||
s.hostName(), network.Mbits(r.DownMbps), network.Mbits(r.UpMbps), r.LatencyMs, r.Colo)
|
||||
}
|
||||
results := s.net.Test.History()
|
||||
b, err := json.Marshal(savedNetwork{Results: results})
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := s.store.Update(func(st *store.State) { st.Network = b }); err != nil {
|
||||
log.Printf("network: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- the speed test as sensors ----
|
||||
|
||||
// netSensors publishes the last speed test as ordinary sensors, so the line can be a widget,
|
||||
// a graph, an alert or a Prometheus series like anything else Ham Lab measures. Each one's
|
||||
// timestamp is when the test ran, which is how a stale reading says so.
|
||||
type netSensors struct{ w *network.Watcher }
|
||||
|
||||
func (n netSensors) Sensors() []sensors.Sensor {
|
||||
var out []sensors.Sensor
|
||||
if o := n.w.Outside(); o.IP != "" {
|
||||
at := o.At.UnixMilli()
|
||||
out = append(out, sensors.Sensor{ID: "net.internet.ip", Name: "Public address", Kind: "state", Value: o.IP, T: at})
|
||||
if o.Colo != "" {
|
||||
out = append(out, sensors.Sensor{ID: "net.internet.edge", Name: "Internet edge", Kind: "state", Value: o.Colo, T: at})
|
||||
}
|
||||
}
|
||||
run := n.w.Test.State()
|
||||
if run.Result == nil || run.Result.Error != "" {
|
||||
return out
|
||||
}
|
||||
r := run.Result
|
||||
at := r.At.UnixMilli()
|
||||
return append(out,
|
||||
sensors.Sensor{ID: "net.internet.download", Name: "Download", Kind: "rate", Unit: "Mbps", Value: r.DownMbps, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.upload", Name: "Upload", Kind: "rate", Unit: "Mbps", Value: r.UpMbps, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.latency", Name: "Latency", Kind: "duration", Unit: "ms", Value: r.LatencyMs, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.jitter", Name: "Jitter", Kind: "duration", Unit: "ms", Value: r.JitterMs, T: at, Min: sensors.F(0)},
|
||||
sensors.Sensor{ID: "net.internet.tested", Name: "Speed test age", Kind: "duration", Unit: "s", Value: round1(time.Since(r.At).Seconds()), T: at, Min: sensors.F(0)},
|
||||
)
|
||||
}
|
||||
|
||||
func round1(v float64) float64 { return float64(int64(v*10+0.5)) / 10 }
|
||||
+2309
-31
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,224 @@
|
||||
package api
|
||||
|
||||
// The Space tab: what is using a disk or a pool. The unit is a whole volume, never a
|
||||
// partition: a disk on its own is scanned through every filesystem it backs, and a ZFS pool
|
||||
// through every mounted dataset, as one thing — its member disks are not scanned apart.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"net/http"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/space"
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
// spaceTarget is one thing a scan can measure: a disk on its own, or a ZFS pool as a whole.
|
||||
// A disk that is a member of a pool is not a target of its own — its share of the pool means
|
||||
// nothing by itself — it is listed under the pool. (A disk with a pool partition *and* a
|
||||
// filesystem of its own, root on one partition and a pool on another, is both.)
|
||||
type spaceTarget struct {
|
||||
ID string `json:"id"` // "nvme0" or "pool:tank"
|
||||
Kind string `json:"kind"` // disk | pool
|
||||
Name string `json:"name"`
|
||||
Model string `json:"model,omitempty"` // disk: its model
|
||||
Layout string `json:"layout,omitempty"` // pool: raidz2, mirror, …
|
||||
Health string `json:"health,omitempty"` // pool: ONLINE, DEGRADED, …
|
||||
Size uint64 `json:"size"`
|
||||
Used uint64 `json:"used,omitempty"` // pool: allocated
|
||||
Disks []string `json:"disks,omitempty"` // pool: its member disks
|
||||
Roots []string `json:"roots"` // the mountpoints a scan walks
|
||||
Scan *space.Scan `json:"scan,omitempty"` // the last scan, if any
|
||||
|
||||
datasets map[string]string // root → ZFS dataset, for the ones that are
|
||||
}
|
||||
|
||||
var layoutRe = regexp.MustCompile(`\b(draid\d*|raidz[123]?|mirror)\b`)
|
||||
|
||||
// spaceTargets lists the disks and pools a scan can measure.
|
||||
func (s *Server) spaceTargets(ctx context.Context) []spaceTarget {
|
||||
devs, _ := storage.Devices(ctx)
|
||||
var out []spaceTarget
|
||||
pools := map[string]*spaceTarget{}
|
||||
var poolOrder []string
|
||||
for _, d := range devs {
|
||||
disk := spaceTarget{ID: d.Name, Kind: "disk", Name: d.Name, Model: d.Model, Size: d.Size, Roots: []string{}}
|
||||
seen := map[string]bool{}
|
||||
inPool := false
|
||||
for _, p := range d.Parts {
|
||||
if p.FSType == "zfs_member" && p.Label != "" {
|
||||
inPool = true
|
||||
pt := pools[p.Label]
|
||||
if pt == nil {
|
||||
pt = &spaceTarget{ID: "pool:" + p.Label, Kind: "pool", Name: p.Label, Roots: []string{}, datasets: map[string]string{}}
|
||||
for _, zm := range zfsMounts(ctx, p.Label) {
|
||||
pt.Roots = append(pt.Roots, zm.mount)
|
||||
pt.datasets[zm.mount] = zm.dataset
|
||||
}
|
||||
pools[p.Label] = pt
|
||||
poolOrder = append(poolOrder, p.Label)
|
||||
}
|
||||
pt.Disks = append(pt.Disks, d.Name)
|
||||
continue
|
||||
}
|
||||
if p.Mount != "" && p.Mount != "[SWAP]" && !seen[p.Mount] {
|
||||
seen[p.Mount] = true
|
||||
disk.Roots = append(disk.Roots, p.Mount)
|
||||
}
|
||||
}
|
||||
if !inPool || len(disk.Roots) > 0 {
|
||||
sort.Strings(disk.Roots)
|
||||
out = append(out, disk)
|
||||
}
|
||||
}
|
||||
if len(pools) > 0 {
|
||||
// size, health and layout from zpool; the vdev tree names the layout
|
||||
for _, p := range s.storage.ZFS(ctx).Pools {
|
||||
if pt := pools[p.Name]; pt != nil {
|
||||
pt.Size, pt.Used, pt.Health = p.Size, p.Alloc, p.Health
|
||||
if m := layoutRe.FindString(p.Config); m != "" {
|
||||
pt.Layout = m
|
||||
} else if len(pt.Disks) == 1 {
|
||||
pt.Layout = "single disk"
|
||||
} else if len(pt.Disks) > 1 {
|
||||
pt.Layout = "stripe"
|
||||
}
|
||||
}
|
||||
}
|
||||
var ps []spaceTarget
|
||||
for _, name := range poolOrder {
|
||||
sort.Strings(pools[name].Roots)
|
||||
ps = append(ps, *pools[name])
|
||||
}
|
||||
out = append(ps, out...) // pools first: they are what a NAS is
|
||||
}
|
||||
if out == nil {
|
||||
out = []spaceTarget{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type zfsMount struct{ mount, dataset string }
|
||||
|
||||
// zfsMounts is every mounted filesystem of a pool, with its dataset.
|
||||
func zfsMounts(ctx context.Context, pool string) []zfsMount {
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-t", "filesystem", "-r", "-o", "name,mountpoint,mounted", pool).Output()
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var mounts []zfsMount
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) == 3 && f[2] == "yes" && strings.HasPrefix(f[1], "/") {
|
||||
mounts = append(mounts, zfsMount{mount: f[1], dataset: f[0]})
|
||||
}
|
||||
}
|
||||
return mounts
|
||||
}
|
||||
|
||||
func (s *Server) spaceOverview(w http.ResponseWriter, r *http.Request) {
|
||||
targets := s.spaceTargets(r.Context())
|
||||
scans := s.space.All()
|
||||
for i := range targets {
|
||||
targets[i].Scan = scans[targets[i].ID]
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{"targets": targets})
|
||||
}
|
||||
|
||||
func (s *Server) spaceTarget(w http.ResponseWriter, r *http.Request, id string) (spaceTarget, bool) {
|
||||
for _, t := range s.spaceTargets(r.Context()) {
|
||||
if t.ID == id {
|
||||
return t, true
|
||||
}
|
||||
}
|
||||
jsonErr(w, 404, "no such disk or pool")
|
||||
return spaceTarget{}, false
|
||||
}
|
||||
|
||||
func (s *Server) spaceStatus(w http.ResponseWriter, r *http.Request) {
|
||||
st := s.space.Status(r.PathValue("disk"))
|
||||
if st == nil {
|
||||
jsonErr(w, 404, "not scanned")
|
||||
return
|
||||
}
|
||||
jsonOK(w, 200, st)
|
||||
}
|
||||
|
||||
func (s *Server) spaceScan(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.spaceTarget(w, r, r.PathValue("disk"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
st, err := s.space.Start(t.ID, t.Roots, t.datasets)
|
||||
if err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("space: scan of %s (%s) started by %s", t.ID, strings.Join(t.Roots, " "), clientIP(r))
|
||||
jsonOK(w, 202, st)
|
||||
}
|
||||
|
||||
func (s *Server) spaceForget(w http.ResponseWriter, r *http.Request) {
|
||||
s.space.Forget(r.PathValue("disk"))
|
||||
log.Printf("space: scan of %s forgotten by %s", r.PathValue("disk"), clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) spaceCancel(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.space.Cancel(r.PathValue("disk")) {
|
||||
jsonErr(w, 404, "no scan running")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) spaceTree(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query()
|
||||
path := q.Get("path")
|
||||
if path == "" {
|
||||
path = "/"
|
||||
}
|
||||
depth, _ := strconv.Atoi(q.Get("depth"))
|
||||
if depth < 1 || depth > 32 {
|
||||
depth = 1
|
||||
}
|
||||
limit, _ := strconv.Atoi(q.Get("limit"))
|
||||
if limit < 1 || limit > 2000 {
|
||||
limit = 300
|
||||
}
|
||||
min, _ := strconv.ParseInt(q.Get("min"), 10, 64)
|
||||
if min < 0 {
|
||||
min = 0
|
||||
}
|
||||
t, err := s.space.Tree(r.PathValue("disk"), path, depth, limit, min)
|
||||
if err != nil {
|
||||
jsonErr(w, 404, err.Error())
|
||||
return
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
jsonOK(w, 200, t)
|
||||
}
|
||||
|
||||
func (s *Server) spaceLargest(w http.ResponseWriter, r *http.Request) {
|
||||
n, _ := strconv.Atoi(r.URL.Query().Get("n"))
|
||||
if n < 1 || n > 5000 {
|
||||
n = 500
|
||||
}
|
||||
files, err := s.space.Largest(r.PathValue("disk"), n)
|
||||
if err != nil {
|
||||
jsonErr(w, 404, err.Error())
|
||||
return
|
||||
}
|
||||
if files == nil {
|
||||
files = []space.File{}
|
||||
}
|
||||
jsonOK(w, 200, files)
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,121 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"hamlab/internal/compose"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// A stack deployed by an orchestrator running in a container records the path its compose
|
||||
// file had inside that container. Resolving it has to work both while the orchestrator is
|
||||
// running — compose is then run inside it — and after it has been removed, where the same
|
||||
// mount is only remembered and compose has to run on the host instead.
|
||||
func TestResolveThroughMounts(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "orch", "stacks", "app")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
file := filepath.Join(dir, "compose.yaml")
|
||||
if err := os.WriteFile(file, []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inner := "/app/data/stacks/app/compose.yaml"
|
||||
|
||||
for _, tc := range []struct {
|
||||
name, container, wantVia string
|
||||
}{
|
||||
{"orchestrator running", "orch", "orch"},
|
||||
{"orchestrator removed", "", ""},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
st := stack{Name: "app", File: inner, Files: []string{inner}}
|
||||
st.resolve([]mountMap{{tc.container, "/app/data", filepath.Join(root, "orch")}})
|
||||
if !st.Editable {
|
||||
t.Fatalf("not editable: %s", st.Note)
|
||||
}
|
||||
if st.HostFile != file {
|
||||
t.Errorf("host file = %q, want %q", st.HostFile, file)
|
||||
}
|
||||
if st.Via != tc.wantVia {
|
||||
t.Errorf("via = %q, want %q", st.Via, tc.wantVia)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Nothing explains the path: the stack is listed, but read-only and says why.
|
||||
st := stack{Name: "app", File: inner, Files: []string{inner}}
|
||||
st.resolve(nil)
|
||||
if st.Editable || st.Note == "" {
|
||||
t.Errorf("unresolvable stack should be read-only with a note, got editable=%v note=%q", st.Editable, st.Note)
|
||||
}
|
||||
}
|
||||
|
||||
// Every file of a multi-file stack must resolve through the same place. Redeploying from
|
||||
// the first file alone would silently drop whatever the override set.
|
||||
func TestResolveRefusesSplitStack(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "orch", "stacks", "app")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "compose.yaml"), []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st := stack{Name: "app", File: "/app/data/stacks/app/compose.yaml", Files: []string{
|
||||
"/app/data/stacks/app/compose.yaml",
|
||||
"/app/data/stacks/app/override.yaml", // never written: not on disk
|
||||
}}
|
||||
st.resolve([]mountMap{{"orch", "/app/data", filepath.Join(root, "orch")}})
|
||||
if st.Editable {
|
||||
t.Error("a stack with a file that cannot be found should not be editable")
|
||||
}
|
||||
}
|
||||
|
||||
// A schedule made while an orchestrator was running says "run compose inside that
|
||||
// container". When the container is gone the schedule must still work, against the same
|
||||
// files on the host — otherwise retiring the orchestrator silently breaks every scheduled
|
||||
// update made before it went.
|
||||
func TestRepointSurvivesAMissingContainer(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "orch", "stacks", "app")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
host := filepath.Join(dir, "compose.yaml")
|
||||
if err := os.WriteFile(host, []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := store.Open(filepath.Join(root, "state.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.Update(func(st *store.State) {
|
||||
st.Mounts = []store.RememberedMount{{Container: "gone-orchestrator", Dest: "/app/data", Src: filepath.Join(root, "orch")}}
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
in := compose.Spec{Project: "app", Files: []string{"/app/data/stacks/app/compose.yaml"}, Via: "gone-orchestrator", Tool: "docker"}
|
||||
got := repoint(context.Background(), db, in)
|
||||
if got.Via != "" {
|
||||
t.Errorf("via = %q, want it dropped", got.Via)
|
||||
}
|
||||
if len(got.Files) != 1 || got.Files[0] != host {
|
||||
t.Errorf("files = %v, want [%s]", got.Files, host)
|
||||
}
|
||||
|
||||
// Nothing remembered: the schedule is left exactly as it was rather than pointed at a
|
||||
// guess. It will fail loudly, which is better than deploying the wrong file.
|
||||
empty, err := store.Open(filepath.Join(root, "empty.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := repoint(context.Background(), empty, in); got.Via != in.Via || len(got.Files) != 1 || got.Files[0] != in.Files[0] {
|
||||
t.Errorf("unresolvable spec was changed: %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,414 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/incus"
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/sensors"
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
// statusSensors is the sensors.Provider for what isn't a /proc number: failed services,
|
||||
// pending updates, pool health, drive health. Each source costs an exec or several, so
|
||||
// they are read in the background on their own clocks and served from memory; every
|
||||
// sensor's t says how old its reading really is.
|
||||
type statusSensors struct {
|
||||
s *Server
|
||||
mu sync.RWMutex
|
||||
by map[string][]sensors.Sensor // source → its sensors
|
||||
|
||||
cpuPrev map[string]cpuMark // container ID → CPU time at the previous read
|
||||
|
||||
hist map[string]*sensors.Series // numeric sensors only, statusHistory points each
|
||||
acc map[string]*bucket // the point being built for each, when the window is wider than the read rate
|
||||
}
|
||||
|
||||
// statusHistory is how many readings each status sensor keeps: at 15 s that is over an
|
||||
// hour of container memory; at 10 min, two days of drive wear.
|
||||
const statusHistory = 300
|
||||
|
||||
// A point being averaged, the same idea as the sampler's: these sources are read every 15 to
|
||||
// 300 seconds, so a wide window keeps one point per stride rather than one per read.
|
||||
type bucket struct {
|
||||
last int64 // when this sensor's last point was kept
|
||||
sum float64
|
||||
n float64
|
||||
}
|
||||
|
||||
const (
|
||||
servicesEvery = 30 * time.Second
|
||||
containersEvery = 15 * time.Second
|
||||
updatesEvery = 5 * time.Minute
|
||||
zfsEvery = time.Minute
|
||||
smartEvery = 10 * time.Minute // storage.Manager caches SMART that long anyway, and never wakes a sleeping disk
|
||||
)
|
||||
|
||||
func newStatusSensors(s *Server) *statusSensors {
|
||||
p := &statusSensors{s: s, by: map[string][]sensors.Sensor{}, hist: map[string]*sensors.Series{}, acc: map[string]*bucket{}}
|
||||
go p.loop("services", servicesEvery, p.services)
|
||||
go p.loop("containers", containersEvery, p.containers)
|
||||
go p.loop("instances", containersEvery, p.instances)
|
||||
go p.loop("updates", updatesEvery, p.updates)
|
||||
go p.loop("zfs", zfsEvery, p.zfs)
|
||||
go p.loop("smart", smartEvery, p.smart)
|
||||
return p
|
||||
}
|
||||
|
||||
func (p *statusSensors) loop(name string, every time.Duration, read func(context.Context, int64) []sensors.Sensor) {
|
||||
for {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
got := read(ctx, time.Now().UnixMilli())
|
||||
cancel()
|
||||
p.mu.Lock()
|
||||
p.by[name] = got
|
||||
window := p.s.metrics.Window()
|
||||
stride := window.Milliseconds() / statusHistory
|
||||
live := map[string]bool{}
|
||||
for _, x := range got {
|
||||
v, ok := x.Value.(float64)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
live[x.ID] = true
|
||||
h := p.hist[x.ID]
|
||||
if h == nil {
|
||||
h = &sensors.Series{ID: x.ID, Unit: x.Unit}
|
||||
p.hist[x.ID] = h
|
||||
}
|
||||
b := p.acc[x.ID]
|
||||
if b == nil {
|
||||
b = &bucket{}
|
||||
p.acc[x.ID] = b
|
||||
}
|
||||
b.sum, b.n = b.sum+v, b.n+1
|
||||
// due a stride after the last point kept, with half a read of slack for a late timer
|
||||
if b.last != 0 && x.T-b.last < stride-every.Milliseconds()/2 {
|
||||
continue // still filling this point
|
||||
}
|
||||
h.T, h.V = append(h.T, x.T), append(h.V, b.sum/b.n)
|
||||
*b = bucket{last: x.T}
|
||||
if len(h.T) > statusHistory {
|
||||
h.T, h.V = h.T[len(h.T)-statusHistory:], h.V[len(h.V)-statusHistory:]
|
||||
}
|
||||
if cut := x.T - window.Milliseconds(); h.T[0] < cut { // and nothing older than the window
|
||||
i := 0
|
||||
for i < len(h.T) && h.T[i] < cut {
|
||||
i++
|
||||
}
|
||||
h.T, h.V = h.T[i:], h.V[i:]
|
||||
}
|
||||
}
|
||||
for id := range p.hist { // a container that is gone takes its history with it
|
||||
if strings.HasPrefix(id, prefixOf[name]) && !live[id] && prefixOf[name] != "" {
|
||||
delete(p.hist, id)
|
||||
delete(p.acc, id)
|
||||
}
|
||||
}
|
||||
p.mu.Unlock()
|
||||
if len(got) == 0 { // no pools, no Docker, no SMART: look again, but rarely
|
||||
time.Sleep(10 * every)
|
||||
continue
|
||||
}
|
||||
time.Sleep(every)
|
||||
}
|
||||
}
|
||||
|
||||
// prefixOf says which IDs a source owns, so its loop forgets only its own departed sensors.
|
||||
var prefixOf = map[string]string{"containers": "container.", "smart": "smart.", "zfs": "zfs."}
|
||||
|
||||
// SensorHistories implements sensors.Historian.
|
||||
func (p *statusSensors) SensorHistories(match func(id string) bool) []sensors.Series {
|
||||
p.mu.RLock()
|
||||
defer p.mu.RUnlock()
|
||||
var out []sensors.Series
|
||||
for id, h := range p.hist {
|
||||
if match(id) {
|
||||
out = append(out, sensors.Series{ID: h.ID, Unit: h.Unit, T: append([]int64{}, h.T...), V: append([]float64{}, h.V...)})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (p *statusSensors) Sensors() []sensors.Sensor {
|
||||
p.mu.RLock()
|
||||
defer p.mu.RUnlock()
|
||||
var out []sensors.Sensor
|
||||
for _, v := range p.by {
|
||||
out = append(out, v...)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (p *statusSensors) services(ctx context.Context, t int64) []sensors.Sensor {
|
||||
units, err := p.s.svc.List(ctx)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var failed, running float64
|
||||
var names []string
|
||||
var per []sensors.Sensor
|
||||
for _, u := range units {
|
||||
if u.Failed {
|
||||
failed++
|
||||
names = append(names, u.Name)
|
||||
}
|
||||
if u.Sub == "running" {
|
||||
running++
|
||||
}
|
||||
// One pair per service that is meant to be doing something: enabled, up, or broken.
|
||||
// (Every static and one-shot unit on the box would triple the catalog for nothing.)
|
||||
short, isService := strings.CutSuffix(u.Name, ".service")
|
||||
if !isService || !(u.Enabled == "enabled" || u.Active == "active" || u.Failed) {
|
||||
continue
|
||||
}
|
||||
id, l := "service."+sensors.Seg(short), map[string]string{"unit": u.Name, "description": u.Description}
|
||||
per = append(per,
|
||||
sensors.Sensor{ID: id + ".active", Name: short, Kind: "state", Value: u.Active == "active", T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".state", Name: short + " state", Kind: "state", Value: u.Sub, T: t, Labels: l})
|
||||
}
|
||||
return append(per, []sensors.Sensor{
|
||||
{ID: "services.failed", Name: "Failed services", Kind: "count", Value: failed, T: t, Warn: sensors.F(1)},
|
||||
{ID: "services.failed_names", Name: "Failed services", Kind: "state", Value: strings.Join(names, ", "), T: t},
|
||||
{ID: "services.running", Name: "Running services", Kind: "count", Value: running, T: t},
|
||||
}...)
|
||||
}
|
||||
|
||||
// containers reads Docker (or Podman) with one `ps`, then each running container's memory
|
||||
// and CPU straight from its cgroup — `docker stats` takes seconds and a daemon round-trip
|
||||
// per call; two small files per container take microseconds.
|
||||
func (p *statusSensors) containers(ctx context.Context, t int64) []sensors.Sensor {
|
||||
var out []byte
|
||||
var err error
|
||||
tool := ""
|
||||
for _, t := range []string{"docker", "podman"} {
|
||||
if _, e := exec.LookPath(t); e != nil {
|
||||
continue
|
||||
}
|
||||
tool = t
|
||||
args := []string{tool, "ps", "-a", "--no-trunc", "--format", "{{json .}}"}
|
||||
if out, err = exec.CommandContext(ctx, args[0], args[1:]...).Output(); err != nil && priv.Available() {
|
||||
full := priv.Wrap(args...) // not in the docker group: ask as root
|
||||
out, err = exec.CommandContext(ctx, full[0], full[1:]...).Output()
|
||||
}
|
||||
break
|
||||
}
|
||||
if err != nil || len(out) == 0 {
|
||||
return nil
|
||||
}
|
||||
// "Not running" is not "down": servarr stopped by hand in June is off, not broken. What is
|
||||
// broken is a container that should be up and isn't — one inspect of the stopped ones tells
|
||||
// them apart: Docker refused to start it (State.Error: yesterday's Gitea), it is stuck
|
||||
// restarting, or it died with a non-zero code that no restart policy picked up. Under
|
||||
// `always` / `unless-stopped` a crash is restarted by Docker, so "exited" there means a
|
||||
// person stopped it. With no restart policy, `docker stop` leaves 143 (the process took the
|
||||
// SIGTERM), 137 (it didn't, and Docker killed it after the grace period) or 130 (SIGINT):
|
||||
// also a person, unless the kernel did the killing for want of memory.
|
||||
type stopped struct {
|
||||
Name string
|
||||
State struct {
|
||||
Status string
|
||||
ExitCode int
|
||||
Error string
|
||||
OOMKilled bool
|
||||
}
|
||||
HostConfig struct{ RestartPolicy struct{ Name string } }
|
||||
}
|
||||
down := map[string]bool{}
|
||||
var ids []string
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
var c struct{ ID, State string }
|
||||
if json.Unmarshal([]byte(line), &c) == nil && c.ID != "" && c.State != "running" {
|
||||
ids = append(ids, c.ID)
|
||||
}
|
||||
}
|
||||
if len(ids) > 0 {
|
||||
args := append([]string{"inspect"}, ids...)
|
||||
insp, ierr := exec.CommandContext(ctx, tool, args...).Output()
|
||||
if ierr != nil && priv.Available() {
|
||||
full := priv.Wrap(append([]string{tool}, args...)...)
|
||||
insp, _ = exec.CommandContext(ctx, full[0], full[1:]...).Output()
|
||||
}
|
||||
var list []stopped
|
||||
if json.Unmarshal(insp, &list) == nil {
|
||||
for _, c := range list {
|
||||
pol := c.HostConfig.RestartPolicy.Name
|
||||
managed := pol == "always" || pol == "unless-stopped"
|
||||
code := c.State.ExitCode
|
||||
byHand := (code == 130 || code == 137 || code == 143) && !c.State.OOMKilled
|
||||
down[strings.TrimPrefix(c.Name, "/")] = c.State.Status == "restarting" || c.State.Status == "dead" || c.State.Error != "" ||
|
||||
(!managed && c.State.Status == "exited" && code != 0 && !byHand)
|
||||
}
|
||||
}
|
||||
}
|
||||
var res []sensors.Sensor
|
||||
var running, unhealthy, total float64
|
||||
seenCPU := map[string]cpuMark{}
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
var c struct{ ID, Names, Image, State, Status string }
|
||||
if json.Unmarshal([]byte(line), &c) != nil || c.Names == "" {
|
||||
continue
|
||||
}
|
||||
total++
|
||||
name := strings.Split(c.Names, ",")[0]
|
||||
id, l := "container."+sensors.Seg(name), map[string]string{"container": name, "image": c.Image}
|
||||
up := c.State == "running"
|
||||
health := "none"
|
||||
for _, h := range []string{"unhealthy", "healthy", "starting"} { // "unhealthy" first: it contains "healthy"
|
||||
if strings.Contains(c.Status, "("+h+")") || strings.Contains(c.Status, "(health: "+h+")") {
|
||||
health = h
|
||||
break
|
||||
}
|
||||
}
|
||||
if up {
|
||||
running++
|
||||
}
|
||||
if health == "unhealthy" {
|
||||
unhealthy++
|
||||
}
|
||||
res = append(res,
|
||||
sensors.Sensor{ID: id + ".running", Name: name, Kind: "state", Value: up, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".down", Name: name + " down", Kind: "state", Value: down[name], T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".state", Name: name + " state", Kind: "state", Value: c.State, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".health", Name: name + " health", Kind: "state", Value: health, T: t, Labels: l})
|
||||
if !up {
|
||||
continue
|
||||
}
|
||||
cg := ""
|
||||
for _, dir := range []string{"/sys/fs/cgroup/system.slice/docker-" + c.ID + ".scope", "/sys/fs/cgroup/docker/" + c.ID, "/sys/fs/cgroup/machine.slice/libpod-" + c.ID + ".scope"} {
|
||||
if _, e := os.Stat(dir + "/memory.current"); e == nil {
|
||||
cg = dir
|
||||
break
|
||||
}
|
||||
}
|
||||
if cg == "" {
|
||||
continue
|
||||
}
|
||||
if b, e := os.ReadFile(cg + "/memory.current"); e == nil {
|
||||
if v, e := strconv.ParseFloat(strings.TrimSpace(string(b)), 64); e == nil {
|
||||
res = append(res, sensors.Sensor{ID: id + ".mem", Name: name + " memory", Kind: "bytes", Unit: "B", Value: v, T: t, Labels: l})
|
||||
}
|
||||
}
|
||||
if b, e := os.ReadFile(cg + "/cpu.stat"); e == nil {
|
||||
for _, ln := range strings.Split(string(b), "\n") {
|
||||
if f := strings.Fields(ln); len(f) == 2 && f[0] == "usage_usec" {
|
||||
usec, _ := strconv.ParseFloat(f[1], 64)
|
||||
seenCPU[c.ID] = cpuMark{usec, t}
|
||||
if prev, ok := p.cpuPrev[c.ID]; ok && t > prev.t && usec >= prev.usec {
|
||||
// percent of one core, like the Processes tab
|
||||
pctv := (usec - prev.usec) / 1000 / float64(t-prev.t) * 100
|
||||
res = append(res, sensors.Sensor{ID: id + ".cpu", Name: name + " CPU", Kind: "usage", Unit: "%", Value: float64(int(pctv*10+0.5)) / 10, T: t, Min: sensors.F(0), Labels: l})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
p.cpuPrev = seenCPU // only this loop touches it
|
||||
return append(res,
|
||||
sensors.Sensor{ID: "containers.total", Name: "Containers", Kind: "count", Value: total, T: t},
|
||||
sensors.Sensor{ID: "containers.running", Name: "Running containers", Kind: "count", Value: running, T: t},
|
||||
sensors.Sensor{ID: "containers.unhealthy", Name: "Unhealthy containers", Kind: "count", Value: unhealthy, T: t, Warn: sensors.F(1)})
|
||||
}
|
||||
|
||||
type cpuMark struct {
|
||||
usec float64
|
||||
t int64
|
||||
}
|
||||
|
||||
// instances are Incus's containers and VMs. "Down" means it should be up and is not: an
|
||||
// instance marked to start with the machine that is stopped, or one in Error — a stopped
|
||||
// instance with no such mark was stopped on purpose, as with Docker containers.
|
||||
func (p *statusSensors) instances(ctx context.Context, t int64) []sensors.Sensor {
|
||||
if !incus.Installed() {
|
||||
return nil
|
||||
}
|
||||
list, err := p.s.virt.InstancesFor(ctx, "sensors")
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var res []sensors.Sensor
|
||||
var running, total float64
|
||||
for _, in := range list {
|
||||
total++
|
||||
id := "instance." + sensors.Seg(in.Name)
|
||||
l := map[string]string{"instance": in.Name, "type": in.Type, "image": in.Image}
|
||||
up := in.Status == "Running"
|
||||
if up {
|
||||
running++
|
||||
}
|
||||
down := in.Status == "Error" || (!up && in.Config["boot.autostart"] == "true")
|
||||
res = append(res,
|
||||
sensors.Sensor{ID: id + ".running", Name: in.Name, Kind: "state", Value: up, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".down", Name: in.Name + " down", Kind: "state", Value: down, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".state", Name: in.Name + " state", Kind: "state", Value: strings.ToLower(in.Status), T: t, Labels: l})
|
||||
if up {
|
||||
res = append(res,
|
||||
sensors.Sensor{ID: id + ".cpu", Name: in.Name + " CPU", Kind: "usage", Unit: "%", Value: float64(int(in.CPUPct*10+0.5)) / 10, T: t, Min: sensors.F(0), Labels: l},
|
||||
sensors.Sensor{ID: id + ".mem", Name: in.Name + " memory", Kind: "bytes", Unit: "B", Value: float64(in.MemUsage), T: t, Labels: l})
|
||||
}
|
||||
}
|
||||
return append(res,
|
||||
sensors.Sensor{ID: "instances.total", Name: "Instances", Kind: "count", Value: total, T: t},
|
||||
sensors.Sensor{ID: "instances.running", Name: "Running instances", Kind: "count", Value: running, T: t})
|
||||
}
|
||||
|
||||
func (p *statusSensors) updates(ctx context.Context, t int64) []sensors.Sensor {
|
||||
st := p.s.updates.Status(ctx)
|
||||
if st.Manager == "" {
|
||||
return nil
|
||||
}
|
||||
return []sensors.Sensor{
|
||||
{ID: "updates.pending", Name: "Pending updates", Kind: "count", Value: float64(len(st.Packages)), T: t, Labels: map[string]string{"manager": st.Manager}},
|
||||
{ID: "updates.security", Name: "Pending security updates", Kind: "count", Value: float64(st.SecurityCount), T: t, Warn: sensors.F(1)},
|
||||
{ID: "updates.reboot_required", Name: "Reboot required", Kind: "state", Value: st.RebootNeeded, T: t},
|
||||
}
|
||||
}
|
||||
|
||||
func (p *statusSensors) zfs(ctx context.Context, t int64) []sensors.Sensor {
|
||||
var out []sensors.Sensor
|
||||
for _, pool := range p.s.storage.Pools(ctx) {
|
||||
id, l := "zfs."+sensors.Seg(pool.Name), map[string]string{"pool": pool.Name}
|
||||
out = append(out,
|
||||
sensors.Sensor{ID: id + ".health", Name: pool.Name + " health", Kind: "state", Value: pool.Health, T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".healthy", Name: pool.Name + " healthy", Kind: "state", Value: pool.Health == "ONLINE", T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".size", Name: pool.Name + " size", Kind: "bytes", Unit: "B", Value: float64(pool.Size), T: t, Labels: l},
|
||||
sensors.Sensor{ID: id + ".used", Name: pool.Name + " used", Kind: "bytes", Unit: "B", Value: float64(pool.Alloc), T: t, Min: sensors.F(0), Max: sensors.F(float64(pool.Size)), Labels: l},
|
||||
sensors.Sensor{ID: id + ".used_pct", Name: pool.Name + " used", Kind: "usage", Unit: "%", Value: float64(pool.Cap), T: t, Min: sensors.F(0), Max: sensors.F(100), Labels: l},
|
||||
sensors.Sensor{ID: id + ".frag", Name: pool.Name + " fragmentation", Kind: "fragmentation", Unit: "%", Value: float64(pool.Frag), T: t, Min: sensors.F(0), Max: sensors.F(100), Labels: l},
|
||||
)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (p *statusSensors) smart(ctx context.Context, t int64) []sensors.Sensor {
|
||||
devs, err := storage.Devices(ctx)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []sensors.Sensor
|
||||
for _, d := range devs {
|
||||
sm := p.s.storage.Smart(ctx, d.Name, false)
|
||||
if sm.Passed == nil || sm.Standby || sm.Error != "" {
|
||||
continue // no verdict to report: no smartctl, asleep, or not a SMART device
|
||||
}
|
||||
id, l := "smart."+sensors.Seg(d.Name), map[string]string{"device": d.Name, "model": d.Model}
|
||||
out = append(out,
|
||||
sensors.Sensor{ID: id + ".passed", Name: d.Name + " health", Kind: "state", Value: *sm.Passed, T: sm.FetchedAt, Labels: l},
|
||||
sensors.Sensor{ID: id + ".warnings", Name: d.Name + " warnings", Kind: "count", Value: float64(len(sm.Warnings)), T: sm.FetchedAt, Warn: sensors.F(1), Labels: l},
|
||||
)
|
||||
if sm.PowerOnHours > 0 {
|
||||
out = append(out, sensors.Sensor{ID: id + ".power_on", Name: d.Name + " powered on", Kind: "duration", Unit: "s", Value: float64(sm.PowerOnHours) * 3600, T: sm.FetchedAt, Labels: l})
|
||||
}
|
||||
if sm.NVMe != nil {
|
||||
out = append(out, sensors.Sensor{ID: id + ".wear", Name: d.Name + " endurance used", Kind: "usage", Unit: "%", Value: float64(sm.NVMe.PercentageUsed), T: sm.FetchedAt, Min: sensors.F(0), Max: sensors.F(100), Labels: l})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package api
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestStreamGate(t *testing.T) {
|
||||
var g streamGate
|
||||
for i := 0; i < maxStreamsPerClient; i++ {
|
||||
if !g.enter("a") {
|
||||
t.Fatalf("stream %d for one client refused early", i)
|
||||
}
|
||||
}
|
||||
if g.enter("a") {
|
||||
t.Fatal("one client went past its ceiling")
|
||||
}
|
||||
if !g.enter("b") {
|
||||
t.Fatal("another client must still get in")
|
||||
}
|
||||
g.leave("a")
|
||||
if !g.enter("a") {
|
||||
t.Fatal("a closed stream must free its place")
|
||||
}
|
||||
for i := 0; i < maxStreamsPerClient+1; i++ {
|
||||
g.leave("a")
|
||||
}
|
||||
g.leave("b")
|
||||
if g.total != -1 && g.total != 0 { // leave() past zero must not wedge the gate open or shut
|
||||
t.Logf("total after over-leave: %d", g.total)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,217 @@
|
||||
package api
|
||||
|
||||
// Updating Ham Lab itself: is there a newer build published, and install it on request.
|
||||
// The check is a plain HTTP GET against whatever `update.url` names — this project's Gitea
|
||||
// releases by default — so a fleet, a stranger's install and an air-gapped copy pointed at
|
||||
// a directory on a NAS all work the same way. Nothing is ever installed unasked: an update
|
||||
// restarts the daemon and drops open terminals, which is not a thing to do to somebody.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/selfupdate"
|
||||
)
|
||||
|
||||
type updateState struct {
|
||||
mu sync.Mutex
|
||||
checking bool
|
||||
installing bool
|
||||
checkedAt time.Time
|
||||
latest selfupdate.Release
|
||||
err string
|
||||
}
|
||||
|
||||
// source is the update source as configured, and whether looking is switched off.
|
||||
func (s *Server) source() (selfupdate.Source, bool) {
|
||||
var u config.UpdateSource
|
||||
s.cfg.View(func(c *config.Config) { u = c.Updates })
|
||||
url := strings.TrimSpace(u.URL)
|
||||
if url == "" {
|
||||
url = selfupdate.DefaultURL
|
||||
}
|
||||
if strings.EqualFold(url, "off") || strings.EqualFold(url, "none") {
|
||||
return selfupdate.Source{}, false
|
||||
}
|
||||
return selfupdate.Source{URL: url, Token: u.Token}, true
|
||||
}
|
||||
|
||||
func (s *Server) checksEnabled() bool {
|
||||
var u config.UpdateSource
|
||||
s.cfg.View(func(c *config.Config) { u = c.Updates })
|
||||
if strings.EqualFold(u.Check, "never") {
|
||||
return false
|
||||
}
|
||||
_, on := s.source()
|
||||
return on
|
||||
}
|
||||
|
||||
// Behind reports whether a newer build than this one has been seen, and which.
|
||||
func (s *Server) behind() (string, bool) {
|
||||
s.update.mu.Lock()
|
||||
defer s.update.mu.Unlock()
|
||||
v := s.update.latest.Version
|
||||
if v == "" || selfupdate.Compare(s.version, v) >= 0 {
|
||||
return "", false
|
||||
}
|
||||
return v, true
|
||||
}
|
||||
|
||||
// checkNow asks the source what the newest build is. One at a time.
|
||||
func (s *Server) checkNow(ctx context.Context) {
|
||||
src, on := s.source()
|
||||
if !on {
|
||||
return
|
||||
}
|
||||
s.update.mu.Lock()
|
||||
if s.update.checking {
|
||||
s.update.mu.Unlock()
|
||||
return
|
||||
}
|
||||
s.update.checking = true
|
||||
s.update.mu.Unlock()
|
||||
ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
|
||||
defer cancel()
|
||||
rel, err := src.Latest(ctx, runtime.GOARCH)
|
||||
s.update.mu.Lock()
|
||||
s.update.checking = false
|
||||
s.update.checkedAt = time.Now()
|
||||
if err != nil {
|
||||
s.update.err = err.Error()
|
||||
} else {
|
||||
s.update.err, s.update.latest = "", rel
|
||||
}
|
||||
s.update.mu.Unlock()
|
||||
if err != nil {
|
||||
log.Printf("update check: %v", err)
|
||||
return
|
||||
}
|
||||
if selfupdate.Compare(s.version, rel.Version) < 0 {
|
||||
log.Printf("update check: %s is published; this is %s", rel.Version, s.version)
|
||||
}
|
||||
}
|
||||
|
||||
// watchForUpdates asks once shortly after boot and then daily.
|
||||
func (s *Server) watchForUpdates(ctx context.Context) {
|
||||
t := time.NewTimer(90 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
if s.checksEnabled() {
|
||||
s.checkNow(ctx)
|
||||
}
|
||||
t.Reset(selfupdate.DefaultCheckEvery)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) updateStatus(w http.ResponseWriter, _ *http.Request) {
|
||||
src, on := s.source()
|
||||
s.update.mu.Lock()
|
||||
defer s.update.mu.Unlock()
|
||||
latest := s.update.latest.Version
|
||||
out := map[string]any{
|
||||
"current": s.version, "latest": latest, "arch": runtime.GOARCH,
|
||||
"behind": latest != "" && selfupdate.Compare(s.version, latest) < 0,
|
||||
"checking": s.update.checking,
|
||||
"installing": s.update.installing,
|
||||
"enabled": on, "auto": s.checksEnabled(), "url": src.URL,
|
||||
"notes": s.update.latest.Notes, "published": s.update.latest.Published,
|
||||
}
|
||||
if !s.update.checkedAt.IsZero() {
|
||||
out["checked"] = s.update.checkedAt
|
||||
}
|
||||
if s.update.err != "" {
|
||||
out["error"] = s.update.err
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) updateCheck(w http.ResponseWriter, r *http.Request) {
|
||||
if _, on := s.source(); !on {
|
||||
jsonErr(w, 400, "update checks are switched off (update.url is \"off\")")
|
||||
return
|
||||
}
|
||||
s.checkNow(r.Context())
|
||||
s.updateStatus(w, r)
|
||||
}
|
||||
|
||||
// updateInstall fetches the published build for this machine and puts it in place.
|
||||
func (s *Server) updateInstall(w http.ResponseWriter, r *http.Request) {
|
||||
src, on := s.source()
|
||||
if !on {
|
||||
jsonErr(w, 400, "update checks are switched off (update.url is \"off\")")
|
||||
return
|
||||
}
|
||||
force := r.URL.Query().Get("force") == "1"
|
||||
if err := s.updateAllowed(force); err != nil {
|
||||
jsonErr(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
Version string `json:"version"` // optional: refuse if what is published is not this
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
|
||||
s.update.mu.Lock()
|
||||
if s.update.installing {
|
||||
s.update.mu.Unlock()
|
||||
jsonErr(w, 409, "an update is already being installed")
|
||||
return
|
||||
}
|
||||
rel := s.update.latest
|
||||
s.update.installing = true
|
||||
s.update.mu.Unlock()
|
||||
done := func() { s.update.mu.Lock(); s.update.installing = false; s.update.mu.Unlock() }
|
||||
|
||||
ctx := context.WithoutCancel(r.Context()) // a closed browser tab must not abort an install
|
||||
if rel.Version == "" || body.Version != "" && body.Version != rel.Version {
|
||||
var err error
|
||||
if rel, err = src.Latest(ctx, runtime.GOARCH); err != nil {
|
||||
done()
|
||||
jsonErr(w, 502, err.Error())
|
||||
return
|
||||
}
|
||||
s.update.mu.Lock()
|
||||
s.update.latest, s.update.checkedAt, s.update.err = rel, time.Now(), ""
|
||||
s.update.mu.Unlock()
|
||||
}
|
||||
if body.Version != "" && body.Version != rel.Version {
|
||||
done()
|
||||
jsonErr(w, 409, "what is published is now "+rel.Version+", not "+body.Version+"; look again")
|
||||
return
|
||||
}
|
||||
if !force && selfupdate.Compare(s.version, rel.Version) >= 0 {
|
||||
done()
|
||||
jsonErr(w, 409, "this machine is already running "+s.version+"; nothing newer is published")
|
||||
return
|
||||
}
|
||||
tmp := filepath.Join(s.cfg.DataDir(), "hamlab.new")
|
||||
if err := src.Fetch(ctx, rel, runtime.GOARCH, tmp); err != nil {
|
||||
done()
|
||||
jsonErr(w, 502, err.Error())
|
||||
return
|
||||
}
|
||||
newVersion, err := s.installBinary(ctx, tmp)
|
||||
if err != nil {
|
||||
done()
|
||||
jsonErr(w, codeOf(err, 500), err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("update: installed %s (was %s), asked for by %s; restarting", newVersion, s.version, clientIP(r))
|
||||
jsonOK(w, 200, map[string]string{"version": newVersion, "previous": s.version})
|
||||
s.restartAfterUpdate()
|
||||
}
|
||||
@@ -0,0 +1,300 @@
|
||||
package api
|
||||
|
||||
// Accounts and their sessions. Every person has their own login, and the console's look
|
||||
// (theme, accents, dashboard layout) follows the account rather than the machine. Anyone
|
||||
// signed in may add or remove accounts — a household console, not a tenancy — but removing
|
||||
// one, or resetting another's password, asks for the caller's own password first, so a
|
||||
// picked-up unlocked phone is not enough.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// actor is who a request is from: a signed-in account (with its session), or a token.
|
||||
type actor struct {
|
||||
User string // account name, "" for a token
|
||||
Session string // session ID, "" for a token
|
||||
token string // the raw cookie token, so a handler can spare the current session
|
||||
Token string // token name, "" for a session
|
||||
// Guest marks a temporary look-around login, and Expires is when it stops working. What
|
||||
// a guest may reach is decided in the middleware, not here; see internal/api/guests.go.
|
||||
Guest bool
|
||||
Expires time.Time
|
||||
}
|
||||
|
||||
type ctxKey int
|
||||
|
||||
const actorKey ctxKey = 1
|
||||
|
||||
func withActor(r *http.Request, a actor) *http.Request {
|
||||
return r.WithContext(context.WithValue(r.Context(), actorKey, a))
|
||||
}
|
||||
|
||||
func actorOf(r *http.Request) actor {
|
||||
a, _ := r.Context().Value(actorKey).(actor)
|
||||
return a
|
||||
}
|
||||
|
||||
// who names the caller for a log line: "core@10.0.0.5" for a person, "token:hub:core@…" for a hub.
|
||||
func (a actor) String() string {
|
||||
switch {
|
||||
case a.User != "" && a.Guest:
|
||||
return "guest:" + a.User
|
||||
case a.User != "":
|
||||
return a.User
|
||||
case a.Token != "":
|
||||
return "token:" + a.Token
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- users ----
|
||||
|
||||
func (s *Server) listUsers(w http.ResponseWriter, r *http.Request) {
|
||||
me := actorOf(r).User
|
||||
counts := s.sessions.Count()
|
||||
var out []map[string]any
|
||||
s.cfg.View(func(c *config.Config) {
|
||||
for _, u := range c.Accounts() { // guest logins are listed on their own, see listGuests
|
||||
var seen any
|
||||
if !u.LastSeen.IsZero() {
|
||||
seen = u.LastSeen
|
||||
}
|
||||
out = append(out, map[string]any{"name": u.Name, "created": u.Created, "last_seen": seen, "sessions": counts[u.Name], "self": u.Name == me})
|
||||
}
|
||||
})
|
||||
if out == nil {
|
||||
out = []map[string]any{}
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) addUser(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct{ Name, Password string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
name := strings.TrimSpace(body.Name)
|
||||
if !auth.UserNameRe.MatchString(name) {
|
||||
jsonErr(w, 400, "a user name is 1–32 lower-case letters, digits, . _ or -, starting with a letter or digit")
|
||||
return
|
||||
}
|
||||
if strings.HasPrefix(name, guestPrefix) {
|
||||
jsonErr(w, 400, "names starting with "+guestPrefix+" belong to guest logins")
|
||||
return
|
||||
}
|
||||
if len(body.Password) < auth.MinPasswordLen {
|
||||
jsonErr(w, 400, "password must be at least 12 characters")
|
||||
return
|
||||
}
|
||||
h, err := auth.HashPassword(body.Password)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
clash := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
if _, ok := c.User(name); ok {
|
||||
clash = true
|
||||
return
|
||||
}
|
||||
c.Users = append(c.Users, config.User{Name: name, Hash: h, Created: time.Now()})
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if clash {
|
||||
jsonErr(w, 409, "a user with that name exists")
|
||||
return
|
||||
}
|
||||
log.Printf("user %q added (by %s)", name, clientIP(r))
|
||||
jsonOK(w, 200, map[string]any{"name": name})
|
||||
}
|
||||
|
||||
// confirmSelf checks the caller's own password, the gate on anything done to another account.
|
||||
func (s *Server) confirmSelf(w http.ResponseWriter, r *http.Request, current string) bool {
|
||||
me := actorOf(r).User
|
||||
if me == "" {
|
||||
jsonErr(w, 403, "only a signed-in user may do that")
|
||||
return false
|
||||
}
|
||||
var users []config.User
|
||||
s.cfg.View(func(c *config.Config) { users = append(users, c.Users...) })
|
||||
if !auth.CheckUser(users, me, current) {
|
||||
jsonErr(w, 403, "your password is wrong")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *Server) removeUser(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
var body struct{ Current string }
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
if name == actorOf(r).User {
|
||||
jsonErr(w, 400, "you can't remove the account you are signed in as")
|
||||
return
|
||||
}
|
||||
if !s.confirmSelf(w, r, body.Current) {
|
||||
return
|
||||
}
|
||||
found, last := false, false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
if len(c.Accounts()) <= 1 { // guest logins don't keep the last real account alive
|
||||
last = true
|
||||
return
|
||||
}
|
||||
kept := c.Users[:0]
|
||||
for _, u := range c.Users {
|
||||
if u.Name == name {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, u)
|
||||
}
|
||||
c.Users = kept
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if last {
|
||||
jsonErr(w, 400, "the last account can't be removed")
|
||||
return
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such user")
|
||||
return
|
||||
}
|
||||
s.sessions.RevokeUser(name)
|
||||
_ = s.store.Update(func(st *store.State) { delete(st.Users, name) })
|
||||
log.Printf("user %q removed (by %s)", name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// resetPassword sets another user's password, on the caller's own password.
|
||||
func (s *Server) resetPassword(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
var body struct{ Current, Password string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if !s.confirmSelf(w, r, body.Current) {
|
||||
return
|
||||
}
|
||||
if len(body.Password) < auth.MinPasswordLen {
|
||||
jsonErr(w, 400, "password must be at least 12 characters")
|
||||
return
|
||||
}
|
||||
if !s.setPassword(w, name, body.Password) {
|
||||
return
|
||||
}
|
||||
if name != actorOf(r).User { // whoever held the old password is signed out
|
||||
s.sessions.RevokeUser(name)
|
||||
}
|
||||
log.Printf("password of %q reset (by %s)", name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) setPassword(w http.ResponseWriter, name, pw string) bool {
|
||||
h, err := auth.HashPassword(pw)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return false
|
||||
}
|
||||
found := false
|
||||
if err := s.cfg.Update(func(c *config.Config) {
|
||||
for i := range c.Users {
|
||||
if c.Users[i].Name == name {
|
||||
c.Users[i].Hash = h
|
||||
found = true
|
||||
}
|
||||
}
|
||||
}); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return false
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such user")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ---- sessions: where the caller's account is signed in ----
|
||||
|
||||
func (s *Server) listSessions(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
if a.User == "" {
|
||||
jsonOK(w, 200, []any{})
|
||||
return
|
||||
}
|
||||
var out []map[string]any
|
||||
for _, ses := range s.sessions.List(a.User) {
|
||||
out = append(out, map[string]any{"id": ses.ID, "created": ses.Created, "last_seen": ses.LastSeen, "expires": ses.Expires, "ip": ses.IP, "agent": ses.Agent, "current": ses.ID == a.Session})
|
||||
}
|
||||
if out == nil {
|
||||
out = []map[string]any{}
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) revokeSession(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
id := r.PathValue("id")
|
||||
if a.User == "" || !s.sessions.Revoke(a.User, id) {
|
||||
jsonErr(w, 404, "no such session")
|
||||
return
|
||||
}
|
||||
log.Printf("session %s of %q revoked (by %s)", id, a.User, clientIP(r))
|
||||
if id == a.Session {
|
||||
s.setCookie(w, r, "", -1)
|
||||
}
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
// revokeIdleSessions drops the caller's sessions that have not been used for ?hours (24 by
|
||||
// default): the scripts and headless browsers that signed in once and never came back,
|
||||
// without touching the phone that was used this morning.
|
||||
func (s *Server) revokeIdleSessions(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
if a.User == "" {
|
||||
jsonErr(w, 403, "only a signed-in user may do that")
|
||||
return
|
||||
}
|
||||
hours := 24
|
||||
if h := r.URL.Query().Get("hours"); h != "" {
|
||||
n, err := strconv.Atoi(h)
|
||||
if err != nil || n < 1 || n > 24*365 {
|
||||
jsonErr(w, 400, "hours is a whole number from 1 to 8760")
|
||||
return
|
||||
}
|
||||
hours = n
|
||||
}
|
||||
n := s.sessions.RevokeIdle(a.User, a.token, time.Duration(hours)*time.Hour)
|
||||
log.Printf("%d idle session(s) of %q revoked (idle > %dh, by %s)", n, a.User, hours, clientIP(r))
|
||||
jsonOK(w, 200, map[string]int{"revoked": n})
|
||||
}
|
||||
|
||||
func (s *Server) revokeOtherSessions(w http.ResponseWriter, r *http.Request) {
|
||||
a := actorOf(r)
|
||||
if a.User == "" {
|
||||
jsonErr(w, 403, "only a signed-in user may do that")
|
||||
return
|
||||
}
|
||||
n := s.sessions.RevokeOthers(a.User, a.token)
|
||||
log.Printf("%d other session(s) of %q revoked (by %s)", n, a.User, clientIP(r))
|
||||
jsonOK(w, 200, map[string]int{"revoked": n})
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,115 @@
|
||||
package api
|
||||
|
||||
// The VPN tab: this machine's WireGuard tunnels, who is on them, and — where the tunnel is
|
||||
// one this machine configures — adding, renaming and removing the devices that dial in.
|
||||
//
|
||||
// Per machine like every other tab: the tunnels shown and edited are the ones on the machine
|
||||
// in the switcher, which is the only honest way to show them, since the inbound server and an
|
||||
// outbound tunnel are different machines here.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"hamlab/internal/vpn"
|
||||
)
|
||||
|
||||
func (s *Server) vpnReport(w http.ResponseWriter, r *http.Request) {
|
||||
jsonOK(w, 200, vpn.Look(r.Context()))
|
||||
}
|
||||
|
||||
// tunnelOf finds the tunnel a request names, so every editing handler starts from the live
|
||||
// state rather than from whatever the page last saw.
|
||||
func (s *Server) tunnelOf(ctx context.Context, w http.ResponseWriter, name string) (vpn.Tunnel, bool) {
|
||||
for _, t := range vpn.Look(ctx).Tunnels {
|
||||
if t.Name == name {
|
||||
if !t.Managed {
|
||||
jsonErr(w, 400, vpn.ErrUnmanaged.Error())
|
||||
return t, false
|
||||
}
|
||||
return t, true
|
||||
}
|
||||
}
|
||||
jsonErr(w, 404, "no such tunnel on this machine")
|
||||
return vpn.Tunnel{}, false
|
||||
}
|
||||
|
||||
// addVPNPeer makes a device and hands back its profile. The profile holds a private key that
|
||||
// is generated for this call and stored nowhere, so it is the only time it can be read — the
|
||||
// same bargain as any password this console generates.
|
||||
func (s *Server) addVPNPeer(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var body vpn.NewPeer
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 8<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
made, err := vpn.AddPeer(r.Context(), t, body)
|
||||
if err != nil {
|
||||
code := 400
|
||||
if errors.Is(err, vpn.ErrUnmanaged) {
|
||||
code = 409
|
||||
}
|
||||
jsonErr(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
// The name and address are worth a log line; the profile is emphatically not.
|
||||
log.Printf("vpn: device %q added to %s as %s (by %s)", made.Name, t.Name, made.Address, clientIP(r))
|
||||
jsonOK(w, 200, made)
|
||||
}
|
||||
|
||||
func (s *Server) editVPNPeer(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var body struct{ Name, Address string }
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4<<10)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
section := r.PathValue("section")
|
||||
if err := vpn.EditPeer(r.Context(), t, section, body.Name, body.Address); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("vpn: device %s on %s changed (by %s)", section, t.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) removeVPNPeer(w http.ResponseWriter, r *http.Request) {
|
||||
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
section := r.PathValue("section")
|
||||
var body struct{ Confirm string }
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4<<10)).Decode(&body)
|
||||
// The name has to be typed back. Removing a device is the one action here that cannot be
|
||||
// undone — the key is gone and the profile on the device stops working — and it is one tap
|
||||
// away from a list of devices that all look alike.
|
||||
var want string
|
||||
for _, p := range t.Peers {
|
||||
if p.Section == section {
|
||||
want = p.Name
|
||||
}
|
||||
}
|
||||
if want != "" && !strings.EqualFold(strings.TrimSpace(body.Confirm), want) {
|
||||
jsonErr(w, 400, "type the device's name to remove it")
|
||||
return
|
||||
}
|
||||
if err := vpn.DeletePeer(r.Context(), t, section); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("vpn: device %s (%s) removed from %s (by %s)", section, want, t.Name, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
@@ -0,0 +1,273 @@
|
||||
package api
|
||||
|
||||
// The ZFS tab: pools, datasets, snapshots, and a schedule for taking and expiring them.
|
||||
//
|
||||
// grid already runs zfSnap from root cron and has done for longer than this repo has
|
||||
// existed, so this does not replace it. It reads the same snapshot names, expires by the
|
||||
// same rule, and surfaces the cron job that drives it — so what is on screen is the truth
|
||||
// about the machine, whether Ham Lab or zfSnap is doing the work. A Ham Lab schedule is for
|
||||
// a machine that has no such tool, and the page says plainly when both are running.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/storage"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
type zfsState struct {
|
||||
Policies []storage.Policy `json:"policies"`
|
||||
}
|
||||
|
||||
func (s *Server) zfsLoad() zfsState {
|
||||
var raw json.RawMessage
|
||||
s.store.View(func(st *store.State) { raw = st.ZFS })
|
||||
var z zfsState
|
||||
if len(raw) > 0 {
|
||||
_ = json.Unmarshal(raw, &z)
|
||||
}
|
||||
if z.Policies == nil {
|
||||
z.Policies = []storage.Policy{}
|
||||
}
|
||||
return z
|
||||
}
|
||||
|
||||
func (s *Server) zfsSave(z zfsState) error {
|
||||
b, err := json.Marshal(z)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.store.Update(func(st *store.State) { st.ZFS = b })
|
||||
}
|
||||
|
||||
// externalSnapshotJob finds a cron job that is already taking snapshots on this machine, so
|
||||
// the page can say so rather than let two schedules quietly double up.
|
||||
func (s *Server) externalSnapshotJob(ctx context.Context) map[string]any {
|
||||
if s.cron == nil {
|
||||
return nil
|
||||
}
|
||||
ov := s.cron.Overview(ctx)
|
||||
for _, j := range ov.Jobs {
|
||||
if !j.Enabled {
|
||||
continue
|
||||
}
|
||||
c := strings.ToLower(j.Command)
|
||||
if !strings.Contains(c, "zfsnap") && !strings.Contains(c, "zfs-auto-snapshot") && !strings.Contains(c, "sanoid") &&
|
||||
!(strings.Contains(c, "zfs ") && strings.Contains(c, "snapshot")) {
|
||||
continue
|
||||
}
|
||||
return map[string]any{
|
||||
"tool": toolName(c), "schedule": j.Schedule, "command": j.Command,
|
||||
"user": j.User, "source": j.Source, "id": j.ID, "next": j.Next, "last": j.Last,
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func toolName(cmd string) string {
|
||||
switch {
|
||||
case strings.Contains(cmd, "zfsnap"):
|
||||
return "zfSnap"
|
||||
case strings.Contains(cmd, "zfs-auto-snapshot"):
|
||||
return "zfs-auto-snapshot"
|
||||
case strings.Contains(cmd, "sanoid"):
|
||||
return "sanoid"
|
||||
}
|
||||
return "a cron job"
|
||||
}
|
||||
|
||||
// randomID is a short opaque id for a schedule.
|
||||
func randomID() (string, error) {
|
||||
b := make([]byte, 6)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(b), nil
|
||||
}
|
||||
|
||||
// ---- the cron file ----
|
||||
//
|
||||
// A schedule is a cron line, written to /etc/cron.d/hamlab-zfs, which Ham Lab owns and
|
||||
// rewrites whole whenever the schedules change. Nothing in the daemon runs them: snapshots
|
||||
// are backup-shaped and should not stop because a web console is down, being upgraded, or
|
||||
// uninstalled. The line calls `hamlab zfs-snapshot <id>` as the account Ham Lab runs as,
|
||||
// which reaches zfs through sudo exactly as the daemon does.
|
||||
|
||||
// syncCron puts the current schedules on the machine.
|
||||
func (s *Server) syncCron(ctx context.Context, z zfsState) error {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
exe, _ = filepath.EvalSymlinks(exe)
|
||||
return s.storage.WriteCronFile(ctx, z.Policies, s.user, exe, s.cfg.Path())
|
||||
}
|
||||
|
||||
// ---- endpoints ----
|
||||
|
||||
func (s *Server) zfsOverview(w http.ResponseWriter, r *http.Request) {
|
||||
z := s.zfsLoad()
|
||||
now := time.Now()
|
||||
dir := s.cfg.DataDir()
|
||||
out := make([]map[string]any, 0, len(z.Policies))
|
||||
for i := range z.Policies {
|
||||
p := z.Policies[i]
|
||||
expr, _ := p.CronExpr()
|
||||
m := map[string]any{"policy": p, "cron": expr, "last": storage.ReadRun(dir, p.ID)}
|
||||
if p.Enabled {
|
||||
if n := p.NextRun(now); !n.IsZero() {
|
||||
m["next"] = n
|
||||
}
|
||||
}
|
||||
out = append(out, m)
|
||||
}
|
||||
jsonOK(w, 200, map[string]any{
|
||||
"zfs": s.storage.ZFS(r.Context()), "policies": out, "external": s.externalSnapshotJob(r.Context()),
|
||||
})
|
||||
}
|
||||
|
||||
// zfsARC is the cache: the sampler's live ARC (size, the hit rate over the last interval)
|
||||
// beside the storage package's fuller picture (since-boot counters, tunables, and each
|
||||
// pool's cache/log/special devices or the lack of them).
|
||||
func (s *Server) zfsARC(w http.ResponseWriter, r *http.Request) {
|
||||
out := map[string]any{"detail": s.storage.ARC(r.Context()), "live": nil}
|
||||
if smp, ok := s.metrics.Latest(); ok && smp.ARC != nil {
|
||||
out["live"] = smp.ARC
|
||||
}
|
||||
jsonOK(w, 200, out)
|
||||
}
|
||||
|
||||
func (s *Server) zfsPolicyPut(w http.ResponseWriter, r *http.Request) {
|
||||
var p storage.Policy
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 16<<10)).Decode(&p); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if err := p.Validate(); err != nil {
|
||||
jsonErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
z := s.zfsLoad()
|
||||
found := false
|
||||
for i := range z.Policies {
|
||||
if z.Policies[i].ID == p.ID && p.ID != "" {
|
||||
z.Policies[i] = p
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
if p.ID == "" {
|
||||
id, err := randomID()
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
p.ID = id
|
||||
}
|
||||
z.Policies = append(z.Policies, p)
|
||||
}
|
||||
if err := s.zfsSave(z); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.syncCron(r.Context(), z); err != nil {
|
||||
jsonErr(w, 500, "saved, but writing "+storage.CronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
expr, _ := p.CronExpr()
|
||||
log.Printf("zfs schedule %q saved by %s (%s, %q, keep %s)", p.Name, clientIP(r), p.Dataset, expr, p.KeepFor)
|
||||
jsonOK(w, 200, p)
|
||||
}
|
||||
|
||||
func (s *Server) zfsPolicyDelete(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
z := s.zfsLoad()
|
||||
kept := z.Policies[:0]
|
||||
found := false
|
||||
for _, p := range z.Policies {
|
||||
if p.ID == id {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
kept = append(kept, p)
|
||||
}
|
||||
if !found {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
z.Policies = kept
|
||||
if err := s.zfsSave(z); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.syncCron(r.Context(), z); err != nil {
|
||||
jsonErr(w, 500, "removed, but writing "+storage.CronFile+" failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
storage.ForgetRun(s.cfg.DataDir(), id)
|
||||
log.Printf("zfs schedule %s removed by %s (its snapshots are left alone)", id, clientIP(r))
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
|
||||
func (s *Server) zfsPolicy(id string) (storage.Policy, bool) {
|
||||
for _, p := range s.zfsLoad().Policies {
|
||||
if p.ID == id {
|
||||
return p, true
|
||||
}
|
||||
}
|
||||
return storage.Policy{}, false
|
||||
}
|
||||
|
||||
func (s *Server) zfsRunNow(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
p, ok := s.zfsPolicy(id)
|
||||
if !ok {
|
||||
jsonErr(w, 404, "no such schedule")
|
||||
return
|
||||
}
|
||||
res, err := s.storage.Run(r.Context(), &p)
|
||||
rec := storage.RunRecord{At: time.Now(), Taken: res.Taken, Pruned: res.Pruned}
|
||||
if err != nil {
|
||||
rec.Error = err.Error()
|
||||
}
|
||||
_ = storage.WriteRun(s.cfg.DataDir(), id, rec)
|
||||
if err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("zfs schedule %q run by %s: took %s, expired %d", p.Name, clientIP(r), res.Taken, len(res.Pruned))
|
||||
jsonOK(w, 200, res)
|
||||
}
|
||||
|
||||
// zfsSnapshotDelete removes one snapshot, named in full and typed back by the caller.
|
||||
func (s *Server) zfsSnapshotDelete(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Name string `json:"name"`
|
||||
Confirm string `json:"confirm"`
|
||||
Recursive bool `json:"recursive"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||
jsonErr(w, 400, "bad request")
|
||||
return
|
||||
}
|
||||
if body.Confirm != body.Name {
|
||||
jsonErr(w, 400, "type the snapshot's full name to confirm")
|
||||
return
|
||||
}
|
||||
if err := s.storage.DestroySnapshot(r.Context(), body.Name, body.Recursive); err != nil {
|
||||
jsonErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
log.Printf("zfs snapshot %s destroyed by %s%s", body.Name, clientIP(r), map[bool]string{true: " (recursive)", false: ""}[body.Recursive])
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
+324
-32
@@ -3,17 +3,21 @@ package auth
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"golang.org/x/crypto/argon2"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
@@ -24,6 +28,10 @@ const (
|
||||
argonKeyLen = 32
|
||||
|
||||
SessionTTL = 30 * 24 * time.Hour
|
||||
// SessionIdle is how long a session may go unused before it is dropped, whatever its
|
||||
// expiry says. Every script, curl and headless browser that signs in leaves a session;
|
||||
// a phone that is used keeps its own alive by being used.
|
||||
SessionIdle = 7 * 24 * time.Hour
|
||||
)
|
||||
|
||||
func HashPassword(pw string) (string, error) {
|
||||
@@ -59,6 +67,118 @@ func VerifyPassword(hash, pw string) bool {
|
||||
return subtle.ConstantTimeCompare(got, want) == 1
|
||||
}
|
||||
|
||||
// ---- API tokens (hub → node) ----
|
||||
|
||||
// NewToken returns a random bearer token and its SHA-256 hex, which is what gets stored.
|
||||
func NewToken() (token, hash string, err error) {
|
||||
b := make([]byte, 32)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
token = base64.RawURLEncoding.EncodeToString(b)
|
||||
return token, HashToken(token), nil
|
||||
}
|
||||
|
||||
func HashToken(token string) string {
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// TokenMatches compares a presented token against stored hashes in constant time.
|
||||
func TokenMatches(token string, hashes []string) bool { return TokenIndex(token, hashes) >= 0 }
|
||||
|
||||
// TokenIndex is which of hashes the presented token matches, -1 for none; every hash is
|
||||
// compared whatever the answer.
|
||||
func TokenIndex(token string, hashes []string) int {
|
||||
h := []byte(HashToken(token))
|
||||
idx := -1
|
||||
for i, want := range hashes {
|
||||
if subtle.ConstantTimeCompare(h, []byte(want)) == 1 {
|
||||
idx = i
|
||||
}
|
||||
}
|
||||
return idx
|
||||
}
|
||||
|
||||
// ---- users ----
|
||||
|
||||
// UserNameRe is what an account may be called: like a Unix login, lower-case.
|
||||
var UserNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{0,31}$`)
|
||||
|
||||
const MinPasswordLen = 12
|
||||
|
||||
// dummyHash is verified against when the named user does not exist, so a login attempt
|
||||
// takes the same time either way and the form cannot be used to list accounts.
|
||||
var dummyHash = func() string {
|
||||
h, err := HashPassword("not-a-real-password")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return h
|
||||
}()
|
||||
|
||||
// CheckUser verifies a password for one of users. A missing user — or a guest login whose
|
||||
// time is up — fails in the same time as a wrong password.
|
||||
func CheckUser(users []config.User, name, pw string) bool {
|
||||
for _, u := range users {
|
||||
if u.Name == name {
|
||||
if u.Expired(time.Now()) {
|
||||
break
|
||||
}
|
||||
return VerifyPassword(u.Hash, pw)
|
||||
}
|
||||
}
|
||||
VerifyPassword(dummyHash, pw)
|
||||
return false
|
||||
}
|
||||
|
||||
// ---- guest logins ----
|
||||
|
||||
// guestAlphabet leaves out every letter and digit that can be misread in a screenshot or
|
||||
// over the phone: i, l, o, 0 and 1. A guest login is read aloud and typed by hand.
|
||||
const guestAlphabet = "abcdefghjkmnpqrstuvwxyz23456789"
|
||||
|
||||
// randomFrom returns n characters of guestAlphabet, chosen without modulo bias.
|
||||
func randomFrom(n int) (string, error) {
|
||||
out := make([]byte, 0, n)
|
||||
buf := make([]byte, n)
|
||||
limit := byte(256 - 256%len(guestAlphabet)) // the top of the last whole run of the alphabet
|
||||
for len(out) < n {
|
||||
if _, err := rand.Read(buf); err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, b := range buf {
|
||||
if b >= limit {
|
||||
continue // would favour the first few letters; ask for another byte instead
|
||||
}
|
||||
out = append(out, guestAlphabet[int(b)%len(guestAlphabet)])
|
||||
if len(out) == n {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return string(out), nil
|
||||
}
|
||||
|
||||
// GuestPassword is a password for a guest login: four groups of four, ~78 bits, in an
|
||||
// alphabet with nothing ambiguous in it.
|
||||
func GuestPassword() (string, error) {
|
||||
var groups []string
|
||||
for i := 0; i < 4; i++ {
|
||||
g, err := randomFrom(4)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
groups = append(groups, g)
|
||||
}
|
||||
return strings.Join(groups, "-"), nil
|
||||
}
|
||||
|
||||
// GuestSuffix is the tail of a generated guest name.
|
||||
func GuestSuffix() (string, error) { return randomFrom(4) }
|
||||
|
||||
// ---- sessions ----
|
||||
|
||||
// Sessions is a token store backed by the state file. Expiry is sliding.
|
||||
type Sessions struct {
|
||||
st *store.Store
|
||||
@@ -72,89 +192,261 @@ func NewSessions(st *store.Store) *Sessions {
|
||||
|
||||
func pruneExpired(state *store.State) {
|
||||
now := time.Now()
|
||||
for tok, exp := range state.Sessions {
|
||||
if exp.Before(now) {
|
||||
for tok, ses := range state.Sessions {
|
||||
if !live(ses, now) {
|
||||
delete(state.Sessions, tok)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Sessions) Create() (string, error) {
|
||||
b := make([]byte, 32)
|
||||
// live is whether a session still counts: unexpired, and used within SessionIdle. A
|
||||
// session from before visits were recorded is judged by its creation, or failing that by
|
||||
// its expiry alone.
|
||||
func live(ses store.Session, now time.Time) bool {
|
||||
if !ses.Expires.After(now) {
|
||||
return false
|
||||
}
|
||||
seen := ses.LastSeen
|
||||
if seen.IsZero() {
|
||||
seen = ses.Created
|
||||
}
|
||||
return seen.IsZero() || now.Sub(seen) <= SessionIdle
|
||||
}
|
||||
|
||||
func randomHex(n int) (string, error) {
|
||||
b := make([]byte, n)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", err
|
||||
}
|
||||
tok := hex.EncodeToString(b)
|
||||
err := s.st.Update(func(state *store.State) {
|
||||
return hex.EncodeToString(b), nil
|
||||
}
|
||||
|
||||
// Create signs user in from ip with agent, returning the cookie token.
|
||||
func (s *Sessions) Create(user, ip, agent string) (string, error) {
|
||||
tok, err := randomHex(32)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
id, err := randomHex(6)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(agent) > 200 {
|
||||
agent = agent[:200]
|
||||
}
|
||||
now := time.Now()
|
||||
err = s.st.Update(func(state *store.State) {
|
||||
pruneExpired(state)
|
||||
state.Sessions[tok] = time.Now().Add(SessionTTL)
|
||||
state.Sessions[tok] = store.Session{ID: id, User: user, Expires: now.Add(SessionTTL), Created: now, LastSeen: now, IP: ip, Agent: agent}
|
||||
})
|
||||
return tok, err
|
||||
}
|
||||
|
||||
// Valid reports whether tok is live, sliding its expiry (persisted at most hourly).
|
||||
func (s *Sessions) Valid(tok string) bool {
|
||||
// Lookup reports whether tok is live and whose it is, sliding its expiry and noting the
|
||||
// visit (persisted at most hourly, so a busy console is not writing the file all day).
|
||||
func (s *Sessions) Lookup(tok, ip string) (store.Session, bool) {
|
||||
if tok == "" {
|
||||
return false
|
||||
return store.Session{}, false
|
||||
}
|
||||
ok := false
|
||||
extend := false
|
||||
var ses store.Session
|
||||
ok, touch := false, false
|
||||
s.st.View(func(state *store.State) {
|
||||
exp, found := state.Sessions[tok]
|
||||
if found && exp.After(time.Now()) {
|
||||
ok = true
|
||||
extend = time.Until(exp) < SessionTTL-time.Hour
|
||||
found, have := state.Sessions[tok]
|
||||
if have && live(found, time.Now()) {
|
||||
ses, ok = found, true
|
||||
touch = time.Until(found.Expires) < SessionTTL-time.Hour || time.Since(found.LastSeen) > time.Hour
|
||||
}
|
||||
})
|
||||
if ok && extend {
|
||||
if ok && touch {
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
if _, found := state.Sessions[tok]; found {
|
||||
state.Sessions[tok] = time.Now().Add(SessionTTL)
|
||||
if x, found := state.Sessions[tok]; found {
|
||||
x.Expires = time.Now().Add(SessionTTL)
|
||||
x.LastSeen = time.Now()
|
||||
if ip != "" {
|
||||
x.IP = ip
|
||||
}
|
||||
state.Sessions[tok] = x
|
||||
}
|
||||
})
|
||||
}
|
||||
return ok
|
||||
return ses, ok
|
||||
}
|
||||
|
||||
func (s *Sessions) Delete(tok string) {
|
||||
_ = s.st.Update(func(state *store.State) { delete(state.Sessions, tok) })
|
||||
}
|
||||
|
||||
// Limiter is a single global backoff: after 5 failures, each further failure
|
||||
// doubles a lockout starting at 30s, capped at 10 minutes. Single-user box.
|
||||
// List is every live session of user, newest first.
|
||||
func (s *Sessions) List(user string) []store.Session {
|
||||
var out []store.Session
|
||||
s.st.View(func(state *store.State) {
|
||||
now := time.Now()
|
||||
for _, ses := range state.Sessions {
|
||||
if ses.User == user && live(ses, now) {
|
||||
out = append(out, ses)
|
||||
}
|
||||
}
|
||||
})
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].LastSeen.After(out[j].LastSeen) })
|
||||
return out
|
||||
}
|
||||
|
||||
// Count is how many live sessions each user has.
|
||||
func (s *Sessions) Count() map[string]int {
|
||||
n := map[string]int{}
|
||||
s.st.View(func(state *store.State) {
|
||||
now := time.Now()
|
||||
for _, ses := range state.Sessions {
|
||||
if live(ses, now) {
|
||||
n[ses.User]++
|
||||
}
|
||||
}
|
||||
})
|
||||
return n
|
||||
}
|
||||
|
||||
// Revoke ends the session of user with this ID. Reports whether there was one.
|
||||
func (s *Sessions) Revoke(user, id string) bool {
|
||||
found := false
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
for tok, ses := range state.Sessions {
|
||||
if ses.User == user && ses.ID == id {
|
||||
delete(state.Sessions, tok)
|
||||
found = true
|
||||
}
|
||||
}
|
||||
})
|
||||
return found
|
||||
}
|
||||
|
||||
// RevokeOthers ends every session of user except the one holding keep. Returns how many.
|
||||
func (s *Sessions) RevokeOthers(user, keep string) int {
|
||||
n := 0
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
for tok, ses := range state.Sessions {
|
||||
if ses.User == user && tok != keep {
|
||||
delete(state.Sessions, tok)
|
||||
n++
|
||||
}
|
||||
}
|
||||
})
|
||||
return n
|
||||
}
|
||||
|
||||
// RevokeIdle ends every session of user not seen for longer than idle, except the one
|
||||
// holding keep. Returns how many.
|
||||
func (s *Sessions) RevokeIdle(user, keep string, idle time.Duration) int {
|
||||
n := 0
|
||||
now := time.Now()
|
||||
_ = s.st.Update(func(state *store.State) {
|
||||
for tok, ses := range state.Sessions {
|
||||
seen := ses.LastSeen
|
||||
if seen.IsZero() {
|
||||
seen = ses.Created
|
||||
}
|
||||
if ses.User == user && tok != keep && now.Sub(seen) > idle {
|
||||
delete(state.Sessions, tok)
|
||||
n++
|
||||
}
|
||||
}
|
||||
})
|
||||
return n
|
||||
}
|
||||
|
||||
// RevokeUser ends every session of user (the account is going away).
|
||||
func (s *Sessions) RevokeUser(user string) { s.RevokeOthers(user, "") }
|
||||
|
||||
// Limiter locks out a client after repeated failed logins, per client address, with a
|
||||
// global backstop so a botnet can't burn CPU on argon2 indefinitely. A lockout for one
|
||||
// address never blocks another. Past perClientLock failures, each further failure doubles a
|
||||
// lockout starting at 30s, capped at 10 minutes.
|
||||
type Limiter struct {
|
||||
mu sync.Mutex
|
||||
clients map[string]*bucket
|
||||
global bucket
|
||||
}
|
||||
|
||||
type bucket struct {
|
||||
failures int
|
||||
lockUntil time.Time
|
||||
last time.Time
|
||||
}
|
||||
|
||||
var ErrLocked = errors.New("too many failed logins")
|
||||
|
||||
func (l *Limiter) Allow() error {
|
||||
const (
|
||||
// perClientLock is how many failures one address gets before it is made to wait. Set high
|
||||
// on purpose: a guest login is a generated password read off a message and typed by hand
|
||||
// on a phone, and being locked out for fumbling it is worse than the brute-force risk —
|
||||
// which the doubling backoff past this point, and the global backstop, still cover.
|
||||
perClientLock = 20
|
||||
globalLock = 100 // failures (all clients) in the window before everyone waits a minute
|
||||
globalWindow = 5 * time.Minute
|
||||
)
|
||||
|
||||
func (l *Limiter) get(key string) *bucket {
|
||||
if l.clients == nil {
|
||||
l.clients = map[string]*bucket{}
|
||||
}
|
||||
b, ok := l.clients[key]
|
||||
if !ok {
|
||||
b = &bucket{}
|
||||
l.clients[key] = b
|
||||
}
|
||||
// forget clients that have been quiet for an hour
|
||||
if len(l.clients) > 256 {
|
||||
for k, x := range l.clients {
|
||||
if time.Since(x.last) > time.Hour && time.Now().After(x.lockUntil) {
|
||||
delete(l.clients, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func (l *Limiter) Allow(key string) error {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if time.Now().Before(l.lockUntil) {
|
||||
now := time.Now()
|
||||
if now.Before(l.global.lockUntil) {
|
||||
return ErrLocked
|
||||
}
|
||||
if now.Sub(l.global.last) > globalWindow {
|
||||
l.global.failures = 0
|
||||
}
|
||||
if b := l.get(key); now.Before(b.lockUntil) {
|
||||
return ErrLocked
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (l *Limiter) Fail() {
|
||||
func (l *Limiter) Fail(key string) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.failures++
|
||||
if l.failures >= 5 {
|
||||
d := 30 * time.Second << uint(l.failures-5)
|
||||
now := time.Now()
|
||||
b := l.get(key)
|
||||
b.failures++
|
||||
b.last = now
|
||||
if b.failures >= perClientLock {
|
||||
d := 30 * time.Second << uint(b.failures-perClientLock)
|
||||
if d > 10*time.Minute {
|
||||
d = 10 * time.Minute
|
||||
}
|
||||
l.lockUntil = time.Now().Add(d)
|
||||
b.lockUntil = now.Add(d)
|
||||
}
|
||||
if now.Sub(l.global.last) > globalWindow {
|
||||
l.global.failures = 0
|
||||
}
|
||||
l.global.failures++
|
||||
l.global.last = now
|
||||
if l.global.failures >= globalLock {
|
||||
l.global.lockUntil = now.Add(time.Minute)
|
||||
l.global.failures = 0
|
||||
}
|
||||
}
|
||||
|
||||
func (l *Limiter) Reset() {
|
||||
func (l *Limiter) Reset(key string) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.failures = 0
|
||||
l.lockUntil = time.Time{}
|
||||
delete(l.clients, key)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
// Package compose runs docker compose the way a stack was deployed — which is not always
|
||||
// simply "on this host". A stack deployed by an orchestrator running inside its own
|
||||
// container records paths in that container's filesystem, so compose has to run in there;
|
||||
// a stack deployed to another machine over tcp needs DOCKER_HOST pointed at it. Both the
|
||||
// console and the scheduled updates it writes must invoke it identically, or a redeploy
|
||||
// from one would not match the other's labels.
|
||||
package compose
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// Spec is everything needed to run compose for one stack.
|
||||
type Spec struct {
|
||||
Project string // -p
|
||||
Files []string // -f, in order
|
||||
Via string // a container to run compose inside; "" is this host
|
||||
Remote string // a DOCKER_HOST; "" is this machine's Docker
|
||||
Tool string // "docker" or "podman"
|
||||
}
|
||||
|
||||
// Cmd builds the command. args are compose's own, e.g. "pull" or "up", "-d".
|
||||
func Cmd(ctx context.Context, s Spec, args ...string) *exec.Cmd {
|
||||
tool := s.Tool
|
||||
if tool == "" {
|
||||
tool = "docker"
|
||||
}
|
||||
base := []string{"compose", "-p", s.Project}
|
||||
for _, f := range s.Files {
|
||||
base = append(base, "-f", f)
|
||||
}
|
||||
if s.Via != "" {
|
||||
pre := []string{"exec"}
|
||||
if s.Remote != "" {
|
||||
pre = append(pre, "-e", "DOCKER_HOST="+s.Remote)
|
||||
}
|
||||
return exec.CommandContext(ctx, tool, append(append(append(pre, s.Via, "docker"), base...), args...)...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, tool, append(base, args...)...)
|
||||
if len(s.Files) > 0 {
|
||||
cmd.Dir = filepath.Dir(s.Files[0])
|
||||
}
|
||||
if s.Remote != "" {
|
||||
cmd.Env = append(os.Environ(), "DOCKER_HOST="+s.Remote)
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
+169
-5
@@ -7,13 +7,127 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/store"
|
||||
)
|
||||
|
||||
// Token is an API token another Ham Lab (a hub) uses to manage this host. Only the
|
||||
// SHA-256 of the token is kept.
|
||||
type Token struct {
|
||||
Name string `json:"name"`
|
||||
Hash string `json:"hash"`
|
||||
Created time.Time `json:"created"`
|
||||
// Scope is what the token may do. "" is everything (a hub's token: full control of
|
||||
// this machine). ScopeSensorsRead only reads sensors — what a dashboard gets.
|
||||
Scope string `json:"scope,omitempty"`
|
||||
// Sensors narrows a sensors:read token to the IDs these patterns match. Empty is all.
|
||||
Sensors []string `json:"sensors,omitempty"`
|
||||
// IssuedBy is the account that made the token (or paired the hub that holds it).
|
||||
IssuedBy string `json:"issued_by,omitempty"`
|
||||
}
|
||||
|
||||
const ScopeSensorsRead = "sensors:read"
|
||||
|
||||
// Node is a remote Ham Lab this host (as hub) manages. The token is the one that
|
||||
// node issued to us; the fingerprint pins its self-signed certificate.
|
||||
type Node struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
Fingerprint string `json:"fingerprint,omitempty"`
|
||||
Token string `json:"token"`
|
||||
Added time.Time `json:"added"`
|
||||
}
|
||||
|
||||
// User is an account on this machine's console. Only the argon2id hash is kept.
|
||||
type User struct {
|
||||
Name string `json:"name"`
|
||||
Hash string `json:"hash"`
|
||||
Created time.Time `json:"created"`
|
||||
LastSeen time.Time `json:"last_seen,omitempty"`
|
||||
// Guest marks a temporary login made to show somebody the console: it may only read,
|
||||
// it expires by itself at Expires, and it is not one of the machine's accounts — it is
|
||||
// left out of the account list, does not count towards "the last account", and cannot
|
||||
// be the one an empty user name means. See internal/api/guests.go.
|
||||
Guest bool `json:"guest,omitempty"`
|
||||
// Expires is when a guest login stops working. A guest without one has already
|
||||
// expired, which is the safe way to read a half-written file.
|
||||
Expires time.Time `json:"expires,omitempty"`
|
||||
// IssuedBy is the account that made this guest login.
|
||||
IssuedBy string `json:"issued_by,omitempty"`
|
||||
}
|
||||
|
||||
// Expired reports whether a guest login's time is up. A real account never expires; a
|
||||
// guest with no expiry at all counts as expired rather than as forever.
|
||||
func (u User) Expired(now time.Time) bool {
|
||||
return u.Guest && (u.Expires.IsZero() || now.After(u.Expires))
|
||||
}
|
||||
|
||||
// UpdateSource is where this machine looks for a newer Ham Lab, and how often. The field
|
||||
// is Updates, not Update, because Config already has an Update method.
|
||||
type UpdateSource struct {
|
||||
// URL is a Gitea/Forgejo repo API base or a static directory; empty means the built-in
|
||||
// default, and "off" means never look.
|
||||
URL string `json:"url,omitempty"`
|
||||
// Token is sent when the source is behind a login, so a private Gitea can serve its
|
||||
// own fleet without the instance being opened to everyone.
|
||||
Token string `json:"token,omitempty"`
|
||||
// Check is "daily" (the default) or "never".
|
||||
Check string `json:"check,omitempty"`
|
||||
}
|
||||
|
||||
// KVM is an IP-KVM: a device with a web page that gives screen, keyboard and mouse access
|
||||
// to a machine, including when that machine has no OS running. Ham Lab holds the list on
|
||||
// the hub and proxies each one, because a KVM matters exactly when the machine it is
|
||||
// attached to cannot answer for itself.
|
||||
type KVM struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
// URL is the device's own address, as you would type it into a browser on the LAN.
|
||||
URL string `json:"url"`
|
||||
// Machine is the host this KVM is attached to, and is the key: one KVM per machine, shown
|
||||
// on that machine's KVM tab. Empty means the hub itself. The list lives on the hub even
|
||||
// for a node's KVM, because a KVM earns its keep when the machine is down and cannot
|
||||
// answer for itself.
|
||||
Machine string `json:"machine"`
|
||||
Added time.Time `json:"added"`
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
Listen string `json:"listen"`
|
||||
PasswordHash string `json:"password_hash"`
|
||||
// PasswordHash is the single login of installs before users existed. On load it becomes
|
||||
// the first user (see MigrateUsers) and is then dropped from the file.
|
||||
PasswordHash string `json:"password_hash,omitempty"`
|
||||
Users []User `json:"users,omitempty"`
|
||||
TLS string `json:"tls,omitempty"` // "auto" → self-signed cert in the data dir; "" → plain HTTP
|
||||
Tokens []Token `json:"tokens,omitempty"`
|
||||
Nodes []Node `json:"nodes,omitempty"`
|
||||
KVMs []KVM `json:"kvms,omitempty"`
|
||||
// Updates: where a newer build comes from. See UpdateSource.
|
||||
Updates UpdateSource `json:"update,omitempty"`
|
||||
|
||||
path string
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// Candidates lists where config.json may live under a home directory, preferred first:
|
||||
// the XDG location, then ~/apps/hamlab, which installs before v0.2 used.
|
||||
func Candidates(home string) []string {
|
||||
return []string{filepath.Join(home, ".config", "hamlab", "config.json"), filepath.Join(home, "apps", "hamlab", "config.json")}
|
||||
}
|
||||
|
||||
// DefaultPath is the config used when neither -config nor $HAMLAB_CONFIG says otherwise:
|
||||
// whichever candidate exists, else the preferred one.
|
||||
func DefaultPath() string {
|
||||
home, _ := os.UserHomeDir()
|
||||
c := Candidates(home)
|
||||
for _, p := range c {
|
||||
if _, err := os.Stat(p); err == nil {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return c[0]
|
||||
}
|
||||
|
||||
func Default(path string) *Config {
|
||||
@@ -36,10 +150,49 @@ func Load(path string) (*Config, error) {
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// MigrateUsers turns the pre-users single password into the first account, named after
|
||||
// the OS user the console runs as. Reports whether anything changed (the caller saves).
|
||||
func (c *Config) MigrateUsers(name string) bool {
|
||||
if len(c.Users) > 0 || c.PasswordHash == "" {
|
||||
return false
|
||||
}
|
||||
c.Users = []User{{Name: name, Hash: c.PasswordHash, Created: time.Now()}}
|
||||
c.PasswordHash = ""
|
||||
return true
|
||||
}
|
||||
|
||||
// Accounts is every real account: the machine's users, without the guest logins.
|
||||
// Call under View/Update or on a config nobody else holds.
|
||||
func (c *Config) Accounts() []User {
|
||||
out := make([]User, 0, len(c.Users))
|
||||
for _, u := range c.Users {
|
||||
if !u.Guest {
|
||||
out = append(out, u)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// User finds an account by name. Call under View/Update or on a config nobody else holds.
|
||||
func (c *Config) User(name string) (User, bool) {
|
||||
for _, u := range c.Users {
|
||||
if u.Name == name {
|
||||
return u, true
|
||||
}
|
||||
}
|
||||
return User{}, false
|
||||
}
|
||||
|
||||
func (c *Config) Path() string { return c.path }
|
||||
func (c *Config) DataDir() string { return filepath.Dir(c.path) }
|
||||
|
||||
func (c *Config) Save() error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.save()
|
||||
}
|
||||
|
||||
func (c *Config) save() error {
|
||||
if err := os.MkdirAll(c.DataDir(), 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -47,9 +200,20 @@ func (c *Config) Save() error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := c.path + ".tmp"
|
||||
if err := os.WriteFile(tmp, append(b, '\n'), 0o600); err != nil {
|
||||
return err
|
||||
return store.WriteFileAtomic(c.path, append(b, '\n'), 0o600)
|
||||
}
|
||||
return os.Rename(tmp, c.path)
|
||||
|
||||
// Update applies fn under the lock and saves.
|
||||
func (c *Config) Update(fn func(*Config)) error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
fn(c)
|
||||
return c.save()
|
||||
}
|
||||
|
||||
// View reads under the lock.
|
||||
func (c *Config) View(fn func(*Config)) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
fn(c)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,813 @@
|
||||
// Package cron reads and writes what the UI calls the **Jobs** tab (named "Cron" until
|
||||
// 2026-09-24; the package and its /api/cron endpoints kept the old name).
|
||||
//
|
||||
// Package cron reads every scheduled job on the host into one list: /etc/crontab,
|
||||
// /etc/cron.d/*, per-user crontabs, the run-parts directories they drive, and systemd
|
||||
// timers. It computes next runs from the cron expressions, pulls last runs from the
|
||||
// journal (cron logs "(user) CMD (command)" via syslog), and can toggle, edit and run
|
||||
// jobs. Reads/writes that need root go through passwordless sudo when available.
|
||||
package cron
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha1"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type Job struct {
|
||||
ID string `json:"id"`
|
||||
Source string `json:"source"` // "/etc/crontab", "/etc/cron.d/x", "crontab:user", "/etc/cron.daily"
|
||||
Line int `json:"line"` // 1-based line in Source (0 for run-parts scripts)
|
||||
User string `json:"user"`
|
||||
Schedule string `json:"schedule"`
|
||||
Command string `json:"command"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Next int64 `json:"next,omitempty"` // unix ms
|
||||
Last int64 `json:"last,omitempty"`
|
||||
OutFile string `json:"out_file,omitempty"` // parsed from a ">> file" redirect
|
||||
Note string `json:"note,omitempty"`
|
||||
Editable bool `json:"editable"`
|
||||
raw string
|
||||
}
|
||||
|
||||
type Timer struct {
|
||||
Unit string `json:"unit"`
|
||||
Activates string `json:"activates"`
|
||||
Description string `json:"description"`
|
||||
Active string `json:"active"`
|
||||
Sub string `json:"sub"`
|
||||
Enabled string `json:"enabled"`
|
||||
Next int64 `json:"next,omitempty"`
|
||||
Last int64 `json:"last,omitempty"`
|
||||
}
|
||||
|
||||
type Overview struct {
|
||||
Jobs []Job `json:"jobs"`
|
||||
Timers []Timer `json:"timers"`
|
||||
Logging bool `json:"logging"` // cron lines seen in the journal
|
||||
Anacron bool `json:"anacron"`
|
||||
Sudo bool `json:"sudo"`
|
||||
}
|
||||
|
||||
type Run struct {
|
||||
ID string `json:"id"`
|
||||
JobID string `json:"job_id"`
|
||||
Command string `json:"command"`
|
||||
User string `json:"user"`
|
||||
Started time.Time `json:"started"`
|
||||
Finished time.Time `json:"finished,omitempty"`
|
||||
Running bool `json:"running"`
|
||||
ExitCode int `json:"exit_code"`
|
||||
Output string `json:"output"`
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
sudo bool
|
||||
journalSudo bool
|
||||
self string
|
||||
mu sync.Mutex
|
||||
lastRuns map[string]int64 // user\x00command -> unix ms
|
||||
lastAt time.Time
|
||||
lastCursor string // journal cursor of the newest entry folded into lastRuns
|
||||
logging bool
|
||||
runs map[string]*Run
|
||||
}
|
||||
|
||||
func New(sudo, journalSudo bool) *Manager {
|
||||
m := &Manager{sudo: sudo, journalSudo: journalSudo, runs: map[string]*Run{}}
|
||||
if u, err := user.Current(); err == nil {
|
||||
m.self = u.Username
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ---- helpers ----
|
||||
|
||||
func run(ctx context.Context, stdin string, args ...string) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
if stdin != "" {
|
||||
cmd.Stdin = strings.NewReader(stdin)
|
||||
}
|
||||
var out, errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := strings.TrimSpace(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return out.String(), errors.New(msg)
|
||||
}
|
||||
return out.String(), nil
|
||||
}
|
||||
|
||||
// runLines streams a command's stdout to fn line by line, so multi-megabyte journal
|
||||
// dumps never sit in memory whole.
|
||||
func runLines(ctx context.Context, fn func(line []byte), args ...string) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stderr = &errb
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
}
|
||||
sc := bufio.NewScanner(stdout)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
|
||||
for sc.Scan() {
|
||||
fn(sc.Bytes())
|
||||
}
|
||||
if err := cmd.Wait(); err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manager) sudoArgs(args ...string) []string {
|
||||
if m.sudo {
|
||||
return priv.Wrap(args...)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
func jobID(source, raw string) string {
|
||||
h := sha1.Sum([]byte(source + "\n" + raw))
|
||||
return hex.EncodeToString(h[:6])
|
||||
}
|
||||
|
||||
var userRe = regexp.MustCompile(`^[a-z_][a-z0-9_-]{0,31}$`)
|
||||
var cronDName = regexp.MustCompile(`^[A-Za-z0-9_-]+$`) // cron ignores other names in /etc/cron.d
|
||||
|
||||
// ---- parsing ----
|
||||
|
||||
// parseLine splits one crontab line. withUser is true for system crontabs (6th field is
|
||||
// the user). Returns ok=false for comments, blanks, env assignments and garbage.
|
||||
func parseLine(line string, withUser bool) (schedule, usr, command string, ok bool) {
|
||||
t := strings.TrimSpace(line)
|
||||
if t == "" || strings.HasPrefix(t, "#") {
|
||||
return
|
||||
}
|
||||
if i := strings.Index(t, "="); i > 0 && !strings.ContainsAny(t[:i], " \t*") { // NAME=value
|
||||
return
|
||||
}
|
||||
var fields []string
|
||||
if strings.HasPrefix(t, "@") {
|
||||
f := strings.Fields(t)
|
||||
if len(f) < 2 {
|
||||
return
|
||||
}
|
||||
fields = []string{f[0]}
|
||||
t = strings.TrimSpace(t[len(f[0]):])
|
||||
} else {
|
||||
f := strings.Fields(t)
|
||||
if len(f) < 6 {
|
||||
return
|
||||
}
|
||||
fields = f[:5]
|
||||
for _, x := range fields {
|
||||
t = strings.TrimSpace(t[len(x):])
|
||||
}
|
||||
}
|
||||
schedule = strings.Join(fields, " ")
|
||||
if _, err := parseSchedule(schedule); err != nil {
|
||||
return
|
||||
}
|
||||
if withUser {
|
||||
f := strings.Fields(t)
|
||||
if len(f) < 2 || !userRe.MatchString(f[0]) {
|
||||
return
|
||||
}
|
||||
usr = f[0]
|
||||
t = strings.TrimSpace(t[len(f[0]):])
|
||||
}
|
||||
if t == "" {
|
||||
return
|
||||
}
|
||||
return schedule, usr, t, true
|
||||
}
|
||||
|
||||
var redirectRe = regexp.MustCompile(`(?:^|\s)(?:>>|>|2>&1\s*>>|&>>|&>)\s*("?)([^\s"&|;]+)`)
|
||||
|
||||
func outFile(cmd string) string {
|
||||
for _, m := range redirectRe.FindAllStringSubmatch(cmd, -1) {
|
||||
if p := m[2]; strings.HasPrefix(p, "/") && p != "/dev/null" {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// parseSource turns crontab text into jobs. Commented-out lines that still parse as a
|
||||
// job are reported disabled, so a "# 0 5 * * * ..." example shows up as such.
|
||||
func (m *Manager) parseSource(source, text string, withUser, editable bool, defaultUser string, now time.Time, anacron bool) []Job {
|
||||
var jobs []Job
|
||||
lines := strings.Split(text, "\n")
|
||||
isComment := func(i int) bool {
|
||||
return i >= 0 && i < len(lines) && strings.HasPrefix(strings.TrimSpace(lines[i]), "#")
|
||||
}
|
||||
for i, line := range lines {
|
||||
enabled := true
|
||||
probe := line
|
||||
if t := strings.TrimSpace(line); strings.HasPrefix(t, "#") {
|
||||
// A commented-out job sits on its own; a job-shaped line inside a block of
|
||||
// comments is documentation (Debian's stock header has two such examples).
|
||||
if isComment(i-1) && isComment(i+1) {
|
||||
continue
|
||||
}
|
||||
enabled = false
|
||||
probe = strings.TrimLeft(t, "# ")
|
||||
}
|
||||
schedule, usr, command, ok := parseLine(probe, withUser)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if usr == "" {
|
||||
usr = defaultUser
|
||||
}
|
||||
if !enabled {
|
||||
if _, err := user.Lookup(usr); err != nil {
|
||||
continue // "user-name" placeholder and the like
|
||||
}
|
||||
}
|
||||
j := Job{ID: jobID(source, line), Source: source, Line: i + 1, User: usr, Schedule: schedule, Command: command,
|
||||
Enabled: enabled, OutFile: outFile(command), Editable: editable, raw: line}
|
||||
if enabled {
|
||||
if next, ok := nextRun(schedule, now); ok {
|
||||
j.Next = next.UnixMilli()
|
||||
}
|
||||
}
|
||||
if strings.Contains(command, "run-parts") {
|
||||
if dir := runPartsDir(command); dir != "" {
|
||||
if anacron && strings.Contains(command, "anacron") {
|
||||
j.Note = "skipped: anacron is installed and runs " + dir + " instead"
|
||||
}
|
||||
jobs = append(jobs, j)
|
||||
jobs = append(jobs, m.runPartsJobs(dir, j)...)
|
||||
continue
|
||||
}
|
||||
}
|
||||
jobs = append(jobs, j)
|
||||
}
|
||||
return jobs
|
||||
}
|
||||
|
||||
var runPartsRe = regexp.MustCompile(`run-parts\s+(?:--\S+\s+)*(/\S+)`)
|
||||
|
||||
func runPartsDir(cmd string) string {
|
||||
if m := runPartsRe.FindStringSubmatch(cmd); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// runPartsJobs lists the executable scripts a run-parts line will run, as read-only jobs.
|
||||
func (m *Manager) runPartsJobs(dir string, parent Job) []Job {
|
||||
ents, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []Job
|
||||
for _, e := range ents {
|
||||
if e.IsDir() || !cronDName.MatchString(e.Name()) {
|
||||
continue
|
||||
}
|
||||
info, err := e.Info()
|
||||
if err != nil || info.Mode()&0o111 == 0 {
|
||||
continue
|
||||
}
|
||||
p := filepath.Join(dir, e.Name())
|
||||
out = append(out, Job{ID: jobID(dir, e.Name()), Source: dir, User: parent.User, Schedule: parent.Schedule, Command: p,
|
||||
Enabled: parent.Enabled, Next: parent.Next, Note: "run by " + parent.Source + " via run-parts", raw: e.Name()})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- sources ----
|
||||
|
||||
type source struct {
|
||||
name string // job Source
|
||||
withUser bool
|
||||
user string // default user for user crontabs
|
||||
editable bool
|
||||
}
|
||||
|
||||
func (m *Manager) readSource(ctx context.Context, s source) (string, error) {
|
||||
if strings.HasPrefix(s.name, "crontab:") {
|
||||
u := s.user
|
||||
if u == m.self {
|
||||
out, err := run(ctx, "", "crontab", "-l")
|
||||
if err != nil && strings.Contains(err.Error(), "no crontab") {
|
||||
return "", nil
|
||||
}
|
||||
return out, err
|
||||
}
|
||||
if !m.sudo {
|
||||
return "", errors.New("needs sudo")
|
||||
}
|
||||
out, err := run(ctx, "", priv.Wrap("crontab", "-u", u, "-l")...)
|
||||
if err != nil && strings.Contains(err.Error(), "no crontab") {
|
||||
return "", nil
|
||||
}
|
||||
return out, err
|
||||
}
|
||||
b, err := os.ReadFile(s.name)
|
||||
if err != nil && m.sudo {
|
||||
out, err2 := run(ctx, "", priv.Wrap("cat", s.name)...)
|
||||
return out, err2
|
||||
}
|
||||
return string(b), err
|
||||
}
|
||||
|
||||
func (m *Manager) sources(ctx context.Context) []source {
|
||||
var out []source
|
||||
if _, err := os.Stat("/etc/crontab"); err == nil {
|
||||
out = append(out, source{name: "/etc/crontab", withUser: true, editable: m.sudo})
|
||||
}
|
||||
if ents, err := os.ReadDir("/etc/cron.d"); err == nil {
|
||||
for _, e := range ents {
|
||||
if !e.IsDir() && cronDName.MatchString(e.Name()) {
|
||||
out = append(out, source{name: "/etc/cron.d/" + e.Name(), withUser: true, editable: m.sudo})
|
||||
}
|
||||
}
|
||||
}
|
||||
// per-user crontabs: Debian keeps them in /var/spool/cron/crontabs, cronie in /var/spool/cron
|
||||
users := map[string]bool{}
|
||||
if m.sudo {
|
||||
for _, dir := range []string{"/var/spool/cron/crontabs", "/var/spool/cron"} {
|
||||
if out, err := run(ctx, "", priv.Wrap("ls", "-1", dir)...); err == nil {
|
||||
for _, n := range strings.Split(out, "\n") {
|
||||
if n = strings.TrimSpace(n); userRe.MatchString(n) {
|
||||
users[n] = true
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if m.self != "" {
|
||||
users[m.self] = true
|
||||
}
|
||||
names := make([]string, 0, len(users))
|
||||
for u := range users {
|
||||
names = append(names, u)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, u := range names {
|
||||
out = append(out, source{name: "crontab:" + u, user: u, editable: u == m.self || m.sudo})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) findSource(ctx context.Context, name string) (source, bool) {
|
||||
for _, s := range m.sources(ctx) {
|
||||
if s.name == name {
|
||||
return s, true
|
||||
}
|
||||
}
|
||||
return source{}, false
|
||||
}
|
||||
|
||||
// ---- overview ----
|
||||
|
||||
func (m *Manager) Overview(ctx context.Context) Overview {
|
||||
now := time.Now()
|
||||
_, anacronErr := os.Stat("/usr/sbin/anacron")
|
||||
anacron := anacronErr == nil
|
||||
ov := Overview{Jobs: []Job{}, Timers: []Timer{}, Anacron: anacron, Sudo: m.sudo}
|
||||
for _, s := range m.sources(ctx) {
|
||||
text, err := m.readSource(ctx, s)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
ov.Jobs = append(ov.Jobs, m.parseSource(s.name, text, s.withUser, s.editable, s.user, now, anacron)...)
|
||||
}
|
||||
last := m.lastRunsFromJournal(ctx)
|
||||
ov.Logging = m.logging
|
||||
for i := range ov.Jobs {
|
||||
j := &ov.Jobs[i]
|
||||
if t, ok := last[j.User+"\x00"+j.Command]; ok {
|
||||
j.Last = t
|
||||
}
|
||||
}
|
||||
ov.Timers = m.timers(ctx)
|
||||
return ov
|
||||
}
|
||||
|
||||
var cronLogRe = regexp.MustCompile(`^\((\S+)\) CMD \((.*)\)$`)
|
||||
|
||||
// lastRunsFromJournal maps user+command to the most recent "(user) CMD (command)" line.
|
||||
// The first call scans 30 days; later calls only read entries after the last cursor, so
|
||||
// the cost per refresh is proportional to new cron activity, not to history.
|
||||
func (m *Manager) lastRunsFromJournal(ctx context.Context) map[string]int64 {
|
||||
m.mu.Lock()
|
||||
if time.Since(m.lastAt) < 2*time.Second && m.lastRuns != nil {
|
||||
r := m.lastRuns
|
||||
m.mu.Unlock()
|
||||
return r
|
||||
}
|
||||
cursor := m.lastCursor
|
||||
res := map[string]int64{}
|
||||
for k, v := range m.lastRuns {
|
||||
res[k] = v
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
args := []string{"journalctl", "-t", "CRON", "-t", "cron", "-t", "crond", "-o", "json", "--no-pager", "-q"}
|
||||
if cursor != "" {
|
||||
args = append(args, "--after-cursor="+cursor)
|
||||
} else {
|
||||
args = append(args, "--since", "-30d", "-n", "20000")
|
||||
}
|
||||
if m.journalSudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
seen := m.logging
|
||||
_ = runLines(ctx, func(line []byte) {
|
||||
var raw struct {
|
||||
Msg any `json:"MESSAGE"`
|
||||
TS string `json:"__REALTIME_TIMESTAMP"`
|
||||
Cursor string `json:"__CURSOR"`
|
||||
}
|
||||
if json.Unmarshal(line, &raw) != nil {
|
||||
return
|
||||
}
|
||||
seen = true
|
||||
if raw.Cursor != "" {
|
||||
cursor = raw.Cursor
|
||||
}
|
||||
msg, _ := raw.Msg.(string)
|
||||
mm := cronLogRe.FindStringSubmatch(msg)
|
||||
if mm == nil {
|
||||
return
|
||||
}
|
||||
us, _ := strconv.ParseInt(raw.TS, 10, 64)
|
||||
k := mm[1] + "\x00" + mm[2]
|
||||
if us/1000 > res[k] {
|
||||
res[k] = us / 1000
|
||||
}
|
||||
}, args...)
|
||||
m.mu.Lock()
|
||||
m.lastRuns, m.lastAt, m.lastCursor, m.logging = res, time.Now(), cursor, seen
|
||||
m.mu.Unlock()
|
||||
return res
|
||||
}
|
||||
|
||||
// JobLogs returns the journal lines for one job's command (most recent last).
|
||||
func (m *Manager) JobLogs(ctx context.Context, j Job, n int) ([]map[string]any, error) {
|
||||
args := []string{"journalctl", "-t", "CRON", "-t", "cron", "-t", "crond", "-o", "json", "--no-pager", "-q", "-n", "20000"}
|
||||
if m.journalSudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
var res []map[string]any
|
||||
needle := "(" + j.User + ") CMD (" + j.Command + ")"
|
||||
err := runLines(ctx, func(line []byte) {
|
||||
if !bytes.Contains(line, []byte(j.Command[:min(len(j.Command), 40)])) { // cheap pre-filter before JSON
|
||||
return
|
||||
}
|
||||
var raw struct {
|
||||
Msg any `json:"MESSAGE"`
|
||||
TS string `json:"__REALTIME_TIMESTAMP"`
|
||||
PID string `json:"_PID"`
|
||||
}
|
||||
if json.Unmarshal(line, &raw) != nil {
|
||||
return
|
||||
}
|
||||
msg, _ := raw.Msg.(string)
|
||||
if !strings.Contains(msg, needle) {
|
||||
return
|
||||
}
|
||||
us, _ := strconv.ParseInt(raw.TS, 10, 64)
|
||||
res = append(res, map[string]any{"t": us / 1000, "p": 6, "ident": "CRON", "pid": raw.PID, "msg": msg})
|
||||
}, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(res) > n {
|
||||
res = res[len(res)-n:]
|
||||
}
|
||||
if res == nil {
|
||||
res = []map[string]any{}
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// OutputTail returns the last n lines of the job's redirect target.
|
||||
func (m *Manager) OutputTail(ctx context.Context, j Job, n int) (string, error) {
|
||||
if j.OutFile == "" {
|
||||
return "", errors.New("job has no output file")
|
||||
}
|
||||
out, err := run(ctx, "", m.sudoArgs("tail", "-n", strconv.Itoa(n), j.OutFile)...)
|
||||
return out, err
|
||||
}
|
||||
|
||||
// ---- timers ----
|
||||
|
||||
// timers: one `systemctl show` over every loaded timer instead of three list commands.
|
||||
func (m *Manager) timers(ctx context.Context) []Timer {
|
||||
out, err := run(ctx, "", "systemctl", "show", "*.timer", "--no-pager",
|
||||
"-p", "Id,Description,ActiveState,SubState,UnitFileState,NextElapseUSecRealtime,LastTriggerUSec,Triggers")
|
||||
if err != nil {
|
||||
return []Timer{}
|
||||
}
|
||||
unix := func(v string) int64 { // "Sat 2026-09-19 05:47:13 PDT", "n/a" or ""
|
||||
for _, layout := range []string{"Mon 2006-01-02 15:04:05 MST", "2006-01-02 15:04:05 MST", "Mon 2006-01-02 15:04:05"} {
|
||||
if t, err := time.ParseInLocation(layout, v, time.Local); err == nil {
|
||||
return t.UnixMilli()
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
res := []Timer{}
|
||||
for _, block := range strings.Split(out, "\n\n") {
|
||||
t := Timer{}
|
||||
for _, line := range strings.Split(block, "\n") {
|
||||
k, v, ok := strings.Cut(line, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
switch k {
|
||||
case "Id":
|
||||
t.Unit = v
|
||||
case "Description":
|
||||
t.Description = v
|
||||
case "ActiveState":
|
||||
t.Active = v
|
||||
case "SubState":
|
||||
t.Sub = v
|
||||
case "UnitFileState":
|
||||
t.Enabled = v
|
||||
case "NextElapseUSecRealtime":
|
||||
t.Next = unix(v)
|
||||
case "LastTriggerUSec":
|
||||
t.Last = unix(v)
|
||||
case "Triggers":
|
||||
t.Activates = strings.Fields(v + " ")[0:1][0]
|
||||
}
|
||||
}
|
||||
if t.Unit != "" {
|
||||
res = append(res, t)
|
||||
}
|
||||
}
|
||||
sort.Slice(res, func(i, j int) bool { return res[i].Unit < res[j].Unit })
|
||||
return res
|
||||
}
|
||||
|
||||
var timerRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9@._:\\-]{0,255}\.timer$`)
|
||||
|
||||
func (m *Manager) TimerAction(ctx context.Context, unit, action string) error {
|
||||
if !timerRe.MatchString(unit) {
|
||||
return errors.New("invalid timer name")
|
||||
}
|
||||
switch action {
|
||||
case "start", "stop", "enable", "disable":
|
||||
_, err := run(ctx, "", m.sudoArgs("systemctl", action, unit)...)
|
||||
return err
|
||||
case "run": // fire the service the timer activates, right now
|
||||
out, err := run(ctx, "", "systemctl", "show", unit, "-p", "Triggers", "--value")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
svc := strings.TrimSpace(strings.Split(out, " ")[0])
|
||||
if svc == "" {
|
||||
return errors.New("timer triggers nothing")
|
||||
}
|
||||
_, err = run(ctx, "", m.sudoArgs("systemctl", "start", svc)...)
|
||||
return err
|
||||
}
|
||||
return errors.New("unknown action")
|
||||
}
|
||||
|
||||
// ---- job lookup and editing ----
|
||||
|
||||
func (m *Manager) FindJob(ctx context.Context, id string) (Job, bool) {
|
||||
for _, j := range m.Overview(ctx).Jobs {
|
||||
if j.ID == id {
|
||||
return j, true
|
||||
}
|
||||
}
|
||||
return Job{}, false
|
||||
}
|
||||
|
||||
// Source returns a crontab's raw text for the editor.
|
||||
func (m *Manager) Source(ctx context.Context, name string) (text string, editable bool, err error) {
|
||||
s, ok := m.findSource(ctx, name)
|
||||
if !ok {
|
||||
return "", false, errors.New("unknown source")
|
||||
}
|
||||
text, err = m.readSource(ctx, s)
|
||||
return text, s.editable, err
|
||||
}
|
||||
|
||||
// validate rejects any line that is neither blank, comment, env assignment, nor a job.
|
||||
func validate(text string, withUser bool) error {
|
||||
for i, line := range strings.Split(text, "\n") {
|
||||
t := strings.TrimSpace(line)
|
||||
if t == "" || strings.HasPrefix(t, "#") {
|
||||
continue
|
||||
}
|
||||
if k, _, ok := strings.Cut(t, "="); ok && !strings.ContainsAny(strings.TrimSpace(k), " \t*") {
|
||||
continue
|
||||
}
|
||||
if _, _, _, ok := parseLine(t, withUser); !ok {
|
||||
return fmt.Errorf("line %d: not a valid cron entry", i+1)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// WriteSource replaces a crontab. User crontabs go through crontab(1) so cron reloads
|
||||
// them; system files are written in place with their permissions preserved.
|
||||
func (m *Manager) WriteSource(ctx context.Context, name, text string) error {
|
||||
s, ok := m.findSource(ctx, name)
|
||||
if !ok {
|
||||
return errors.New("unknown source")
|
||||
}
|
||||
if !s.editable {
|
||||
return errors.New("source is read-only here (no sudo)")
|
||||
}
|
||||
if !strings.HasSuffix(text, "\n") {
|
||||
text += "\n"
|
||||
}
|
||||
if err := validate(text, s.withUser); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.HasPrefix(name, "crontab:") {
|
||||
if s.user == m.self {
|
||||
_, err := run(ctx, text, "crontab", "-")
|
||||
return err
|
||||
}
|
||||
_, err := run(ctx, text, priv.Wrap("crontab", "-u", s.user, "-")...)
|
||||
return err
|
||||
}
|
||||
if name != "/etc/crontab" && !strings.HasPrefix(name, "/etc/cron.d/") {
|
||||
return errors.New("refusing to write outside /etc/crontab and /etc/cron.d")
|
||||
}
|
||||
_, err := run(ctx, text, priv.Wrap("tee", name)...)
|
||||
m.mu.Lock()
|
||||
m.lastAt = time.Time{}
|
||||
m.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
|
||||
// Toggle comments or uncomments a job's line. The line must still read as the job did
|
||||
// when it was listed, so a concurrent edit can't be clobbered.
|
||||
func (m *Manager) Toggle(ctx context.Context, j Job, enable bool) error {
|
||||
if j.Line == 0 || !j.Editable {
|
||||
return errors.New("this job can't be toggled here")
|
||||
}
|
||||
s, ok := m.findSource(ctx, j.Source)
|
||||
if !ok {
|
||||
return errors.New("unknown source")
|
||||
}
|
||||
text, err := m.readSource(ctx, s)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
lines := strings.Split(text, "\n")
|
||||
if j.Line > len(lines) || lines[j.Line-1] != j.raw {
|
||||
return errors.New("crontab changed underneath; reload and try again")
|
||||
}
|
||||
if enable {
|
||||
lines[j.Line-1] = strings.TrimLeft(strings.TrimSpace(j.raw), "# ")
|
||||
} else {
|
||||
lines[j.Line-1] = "#" + j.raw
|
||||
}
|
||||
return m.WriteSource(ctx, j.Source, strings.Join(lines, "\n"))
|
||||
}
|
||||
|
||||
// ---- run now ----
|
||||
|
||||
// StartRun executes the job's command as its user, the way cron would (sh -c, cwd = home,
|
||||
// '%' turned into newlines/stdin), in the background. Output is kept in memory.
|
||||
func (m *Manager) StartRun(ctx context.Context, j Job) (*Run, error) {
|
||||
if j.User != m.self && !m.sudo {
|
||||
return nil, errors.New("needs sudo to run as " + j.User)
|
||||
}
|
||||
cmdline, stdin := cronPercent(j.Command)
|
||||
args := []string{"sh", "-c", cmdline}
|
||||
if j.User != m.self {
|
||||
args = priv.AsUser(j.User, args...)
|
||||
}
|
||||
r := &Run{ID: jobID("run", strconv.FormatInt(time.Now().UnixNano(), 10)), JobID: j.ID, Command: j.Command, User: j.User, Started: time.Now(), Running: true}
|
||||
runCtx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
cmd := exec.CommandContext(runCtx, args[0], args[1:]...)
|
||||
if stdin != "" {
|
||||
cmd.Stdin = strings.NewReader(stdin)
|
||||
}
|
||||
w := &capWriter{r: r}
|
||||
cmd.Stdout, cmd.Stderr = w, w
|
||||
if err := cmd.Start(); err != nil {
|
||||
cancel()
|
||||
return nil, err
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.runs[r.ID] = r
|
||||
for id, old := range m.runs { // keep the map small
|
||||
if !old.Running && time.Since(old.Finished) > time.Hour {
|
||||
delete(m.runs, id)
|
||||
}
|
||||
}
|
||||
m.mu.Unlock()
|
||||
go func() {
|
||||
defer cancel()
|
||||
err := cmd.Wait()
|
||||
r.mu.Lock()
|
||||
r.Running, r.Finished = false, time.Now()
|
||||
if err != nil {
|
||||
if ee, ok := err.(*exec.ExitError); ok {
|
||||
r.ExitCode = ee.ExitCode()
|
||||
} else {
|
||||
r.ExitCode = -1
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
m.mu.Lock()
|
||||
m.lastAt = time.Time{}
|
||||
m.mu.Unlock()
|
||||
}()
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (m *Manager) GetRun(id string) (*Run, bool) {
|
||||
m.mu.Lock()
|
||||
r, ok := m.runs[id]
|
||||
m.mu.Unlock()
|
||||
return r, ok
|
||||
}
|
||||
|
||||
// Snapshot returns a copy safe to encode.
|
||||
func (r *Run) Snapshot() Run {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return Run{ID: r.ID, JobID: r.JobID, Command: r.Command, User: r.User, Started: r.Started, Finished: r.Finished,
|
||||
Running: r.Running, ExitCode: r.ExitCode, Output: r.buf.String()}
|
||||
}
|
||||
|
||||
type capWriter struct{ r *Run }
|
||||
|
||||
func (w *capWriter) Write(p []byte) (int, error) {
|
||||
w.r.mu.Lock()
|
||||
defer w.r.mu.Unlock()
|
||||
w.r.buf.Write(p)
|
||||
if w.r.buf.Len() > 256<<10 { // keep the tail
|
||||
b := w.r.buf.Bytes()
|
||||
w.r.buf = *bytes.NewBuffer(append([]byte("…\n"), b[len(b)-200<<10:]...))
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
// cronPercent applies crontab(5) rules: an unescaped % ends the command; later ones are
|
||||
// newlines fed to stdin. "\%" is a literal percent.
|
||||
func cronPercent(cmd string) (string, string) {
|
||||
var out, in strings.Builder
|
||||
cur := &out
|
||||
esc := false
|
||||
for _, c := range cmd {
|
||||
switch {
|
||||
case esc:
|
||||
if c != '%' {
|
||||
cur.WriteRune('\\')
|
||||
}
|
||||
cur.WriteRune(c)
|
||||
esc = false
|
||||
case c == '\\':
|
||||
esc = true
|
||||
case c == '%':
|
||||
if cur == &out {
|
||||
cur = &in
|
||||
} else {
|
||||
in.WriteRune('\n')
|
||||
}
|
||||
default:
|
||||
cur.WriteRune(c)
|
||||
}
|
||||
}
|
||||
return out.String(), in.String()
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
package cron
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// schedule is a parsed five-field cron expression as sets.
|
||||
type schedule struct {
|
||||
min, hour, dom, mon, dow [64]bool
|
||||
domStar, dowStar bool // "*" in dom/dow changes the match rule
|
||||
reboot bool
|
||||
}
|
||||
|
||||
var monthNames = map[string]int{"jan": 1, "feb": 2, "mar": 3, "apr": 4, "may": 5, "jun": 6, "jul": 7, "aug": 8, "sep": 9, "oct": 10, "nov": 11, "dec": 12}
|
||||
var dayNames = map[string]int{"sun": 0, "mon": 1, "tue": 2, "wed": 3, "thu": 4, "fri": 5, "sat": 6}
|
||||
|
||||
var aliases = map[string]string{
|
||||
"@yearly": "0 0 1 1 *", "@annually": "0 0 1 1 *", "@monthly": "0 0 1 * *", "@weekly": "0 0 * * 0",
|
||||
"@daily": "0 0 * * *", "@midnight": "0 0 * * *", "@hourly": "0 * * * *",
|
||||
}
|
||||
|
||||
func parseSchedule(expr string) (*schedule, error) {
|
||||
expr = strings.TrimSpace(expr)
|
||||
if expr == "@reboot" {
|
||||
return &schedule{reboot: true}, nil
|
||||
}
|
||||
if a, ok := aliases[strings.ToLower(expr)]; ok {
|
||||
expr = a
|
||||
}
|
||||
f := strings.Fields(expr)
|
||||
if len(f) != 5 {
|
||||
return nil, errors.New("expected 5 fields")
|
||||
}
|
||||
s := &schedule{}
|
||||
var err error
|
||||
if s.min, err = parseField(f[0], 0, 59, nil); err != nil {
|
||||
return nil, fmt.Errorf("minute: %w", err)
|
||||
}
|
||||
if s.hour, err = parseField(f[1], 0, 23, nil); err != nil {
|
||||
return nil, fmt.Errorf("hour: %w", err)
|
||||
}
|
||||
if s.dom, err = parseField(f[2], 1, 31, nil); err != nil {
|
||||
return nil, fmt.Errorf("day of month: %w", err)
|
||||
}
|
||||
if s.mon, err = parseField(f[3], 1, 12, monthNames); err != nil {
|
||||
return nil, fmt.Errorf("month: %w", err)
|
||||
}
|
||||
if s.dow, err = parseField(f[4], 0, 7, dayNames); err != nil {
|
||||
return nil, fmt.Errorf("day of week: %w", err)
|
||||
}
|
||||
if s.dow[7] { // 7 == Sunday
|
||||
s.dow[0] = true
|
||||
}
|
||||
s.domStar = f[2] == "*"
|
||||
s.dowStar = f[4] == "*"
|
||||
return s, nil
|
||||
}
|
||||
|
||||
func parseField(field string, lo, hi int, names map[string]int) ([64]bool, error) {
|
||||
var set [64]bool
|
||||
atom := func(s string) (int, error) {
|
||||
if names != nil {
|
||||
if v, ok := names[strings.ToLower(s)]; ok {
|
||||
return v, nil
|
||||
}
|
||||
}
|
||||
v, err := strconv.Atoi(s)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("bad value %q", s)
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
for _, part := range strings.Split(field, ",") {
|
||||
step := 1
|
||||
if i := strings.Index(part, "/"); i >= 0 {
|
||||
v, err := strconv.Atoi(part[i+1:])
|
||||
if err != nil || v <= 0 {
|
||||
return set, fmt.Errorf("bad step in %q", part)
|
||||
}
|
||||
step, part = v, part[:i]
|
||||
}
|
||||
a, b := lo, hi
|
||||
switch {
|
||||
case part == "*":
|
||||
case strings.Contains(part, "-"):
|
||||
x, y, _ := strings.Cut(part, "-")
|
||||
var err error
|
||||
if a, err = atom(x); err != nil {
|
||||
return set, err
|
||||
}
|
||||
if b, err = atom(y); err != nil {
|
||||
return set, err
|
||||
}
|
||||
default:
|
||||
v, err := atom(part)
|
||||
if err != nil {
|
||||
return set, err
|
||||
}
|
||||
a = v
|
||||
if step == 1 {
|
||||
b = v
|
||||
}
|
||||
}
|
||||
if a < lo || b > hi || a > b {
|
||||
return set, fmt.Errorf("out of range: %q", part)
|
||||
}
|
||||
for v := a; v <= b; v += step {
|
||||
set[v] = true
|
||||
}
|
||||
}
|
||||
return set, nil
|
||||
}
|
||||
|
||||
func (s *schedule) dayMatches(t time.Time) bool {
|
||||
if !s.mon[int(t.Month())] {
|
||||
return false
|
||||
}
|
||||
dom, dow := s.dom[t.Day()], s.dow[int(t.Weekday())]
|
||||
switch {
|
||||
case s.domStar && s.dowStar:
|
||||
return true
|
||||
case s.domStar:
|
||||
return dow
|
||||
case s.dowStar:
|
||||
return dom
|
||||
default: // both restricted: cron runs when either matches
|
||||
return dom || dow
|
||||
}
|
||||
}
|
||||
|
||||
// NextRun is the first time strictly after `from` that a cron expression fires. Exported so
|
||||
// anything that writes a cron line — the ZFS snapshot schedules — can say when it runs next
|
||||
// using the same evaluator that reads everyone else's.
|
||||
func NextRun(expr string, from time.Time) (time.Time, bool) { return nextRun(expr, from) }
|
||||
|
||||
// ValidSchedule reports whether a cron expression parses.
|
||||
func ValidSchedule(expr string) error {
|
||||
_, err := parseSchedule(expr)
|
||||
return err
|
||||
}
|
||||
|
||||
// nextRun finds the first time strictly after `from` that the schedule fires.
|
||||
func nextRun(expr string, from time.Time) (time.Time, bool) {
|
||||
s, err := parseSchedule(expr)
|
||||
if err != nil || s.reboot {
|
||||
return time.Time{}, false
|
||||
}
|
||||
t := from.Truncate(time.Minute).Add(time.Minute)
|
||||
day := time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, t.Location())
|
||||
for i := 0; i < 366*2; i++ {
|
||||
if s.dayMatches(day) {
|
||||
for h := 0; h < 24; h++ {
|
||||
if !s.hour[h] {
|
||||
continue
|
||||
}
|
||||
for mi := 0; mi < 60; mi++ {
|
||||
if !s.min[mi] {
|
||||
continue
|
||||
}
|
||||
c := time.Date(day.Year(), day.Month(), day.Day(), h, mi, 0, 0, day.Location())
|
||||
if !c.Before(t) {
|
||||
return c, true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
day = day.AddDate(0, 0, 1)
|
||||
}
|
||||
return time.Time{}, false
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package cron
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestNextRun(t *testing.T) {
|
||||
loc := time.UTC
|
||||
from := time.Date(2026, 9, 17, 15, 30, 0, 0, loc) // Thursday
|
||||
cases := []struct {
|
||||
expr string
|
||||
want time.Time
|
||||
}{
|
||||
{"0 1 * * *", time.Date(2026, 9, 18, 1, 0, 0, 0, loc)},
|
||||
{"17 * * * *", time.Date(2026, 9, 17, 16, 17, 0, 0, loc)},
|
||||
{"*/15 * * * *", time.Date(2026, 9, 17, 15, 45, 0, 0, loc)},
|
||||
{"47 6 * * 7", time.Date(2026, 9, 20, 6, 47, 0, 0, loc)}, // Sunday as 7
|
||||
{"52 6 1 * *", time.Date(2026, 10, 1, 6, 52, 0, 0, loc)}, // first of month
|
||||
{"24 0 1-7 * *", time.Date(2026, 10, 1, 0, 24, 0, 0, loc)}, // range
|
||||
{"30 3 * * 0", time.Date(2026, 9, 20, 3, 30, 0, 0, loc)}, // Sunday as 0
|
||||
{"0 9 15 * mon", time.Date(2026, 9, 21, 9, 0, 0, 0, loc)}, // dom OR dow
|
||||
{"@daily", time.Date(2026, 9, 18, 0, 0, 0, 0, loc)},
|
||||
{"@weekly", time.Date(2026, 9, 20, 0, 0, 0, 0, loc)},
|
||||
{"0 0 29 feb *", time.Date(2028, 2, 29, 0, 0, 0, 0, loc)},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, ok := nextRun(c.expr, from)
|
||||
if !ok || !got.Equal(c.want) {
|
||||
t.Errorf("%q: got %v ok=%v, want %v", c.expr, got, ok, c.want)
|
||||
}
|
||||
}
|
||||
if _, ok := nextRun("@reboot", from); ok {
|
||||
t.Error("@reboot should have no next run")
|
||||
}
|
||||
for _, bad := range []string{"* * * *", "60 * * * *", "a b c d e", "*/0 * * * *"} {
|
||||
if _, err := parseSchedule(bad); err == nil {
|
||||
t.Errorf("%q should fail", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLine(t *testing.T) {
|
||||
sch, u, cmd, ok := parseLine("25 6\t* * *\troot\ttest -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }", true)
|
||||
if !ok || sch != "25 6 * * *" || u != "root" || !strings_HasPrefix(cmd, "test -x") {
|
||||
t.Errorf("system line: %q %q %q %v", sch, u, cmd, ok)
|
||||
}
|
||||
if _, _, _, ok := parseLine("PATH=/usr/bin:/bin", true); ok {
|
||||
t.Error("env line parsed as job")
|
||||
}
|
||||
if _, _, _, ok := parseLine("# m h dom mon dow command", false); ok {
|
||||
t.Error("comment parsed as job")
|
||||
}
|
||||
if _, u, cmd, ok := parseLine("@reboot /usr/local/bin/thing --flag", false); !ok || u != "" || cmd != "/usr/local/bin/thing --flag" {
|
||||
t.Errorf("@reboot: %q %q %v", u, cmd, ok)
|
||||
}
|
||||
if got := outFile("foo >> /var/log/foo.log 2>&1"); got != "/var/log/foo.log" {
|
||||
t.Errorf("outFile: %q", got)
|
||||
}
|
||||
if c, in := cronPercent("echo hi%line1%line2"); c != "echo hi" || in != "line1\nline2" {
|
||||
t.Errorf("percent: %q %q", c, in)
|
||||
}
|
||||
if c, _ := cronPercent(`date +\%w`); c != "date +%w" {
|
||||
t.Errorf("escaped percent: %q", c)
|
||||
}
|
||||
}
|
||||
|
||||
func strings_HasPrefix(s, p string) bool { return len(s) >= len(p) && s[:len(p)] == p }
|
||||
@@ -0,0 +1,503 @@
|
||||
// Package files is the Files tab: browse, read, write, upload, download, mkdir,
|
||||
// rename, chmod and delete anywhere on the host.
|
||||
//
|
||||
// Two backends share one interface. Local runs in-process as the daemon's user. Sudo
|
||||
// re-executes this same binary as root (`sudo -n hamlab fs …`) for one operation at a
|
||||
// time, streaming file bodies over its stdin/stdout, so root access uses exactly the
|
||||
// same code and the same validation as the unprivileged path.
|
||||
package files
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type Entry struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"` // dir | file | link | other
|
||||
Size int64 `json:"size"`
|
||||
Mode string `json:"mode"` // "-rw-r--r--"
|
||||
Perm string `json:"perm"` // "0644"
|
||||
Mtime int64 `json:"mtime"`
|
||||
User string `json:"user"`
|
||||
Group string `json:"group"`
|
||||
Target string `json:"target,omitempty"` // link target
|
||||
Broken bool `json:"broken,omitempty"` // dangling link
|
||||
}
|
||||
|
||||
type Listing struct {
|
||||
Path string `json:"path"`
|
||||
Parent string `json:"parent,omitempty"`
|
||||
Entries []Entry `json:"entries"`
|
||||
}
|
||||
|
||||
type Info struct {
|
||||
Entry
|
||||
Path string `json:"path"`
|
||||
Text bool `json:"text"` // looks like text (for the editor)
|
||||
}
|
||||
|
||||
// FS is what the API talks to.
|
||||
type FS interface {
|
||||
List(ctx context.Context, path string) (Listing, error)
|
||||
Stat(ctx context.Context, path string) (Info, error)
|
||||
Open(ctx context.Context, path string) (io.ReadCloser, error)
|
||||
Write(ctx context.Context, path string, r io.Reader) error
|
||||
Mkdir(ctx context.Context, path string) error
|
||||
Touch(ctx context.Context, path string) error
|
||||
Rename(ctx context.Context, from, to string) error
|
||||
Delete(ctx context.Context, path string, recursive bool) error
|
||||
Chmod(ctx context.Context, path string, perm os.FileMode) error
|
||||
}
|
||||
|
||||
// ---- path rules ----
|
||||
|
||||
// Clean rejects relative paths and normalises the rest. Everything else on the host is
|
||||
// reachable on purpose: this is a root-capable console, the gate is the login.
|
||||
func Clean(p string) (string, error) {
|
||||
if p == "" {
|
||||
return "", errors.New("path required")
|
||||
}
|
||||
if !strings.HasPrefix(p, "/") {
|
||||
return "", errors.New("path must be absolute")
|
||||
}
|
||||
c := filepath.Clean(p)
|
||||
if strings.ContainsRune(c, 0) {
|
||||
return "", errors.New("bad path")
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// protected are never deleted or renamed as a whole, no matter who asks.
|
||||
var protected = map[string]bool{"/": true, "/bin": true, "/boot": true, "/dev": true, "/etc": true, "/home": true, "/lib": true, "/lib64": true,
|
||||
"/opt": true, "/proc": true, "/root": true, "/run": true, "/sbin": true, "/srv": true, "/sys": true, "/tmp": true, "/usr": true, "/var": true}
|
||||
|
||||
func ParsePerm(s string) (os.FileMode, error) {
|
||||
v, err := strconv.ParseUint(strings.TrimPrefix(s, "0"), 8, 32)
|
||||
if err != nil || v > 0o7777 {
|
||||
return 0, errors.New("mode must be octal like 0644")
|
||||
}
|
||||
return os.FileMode(v), nil
|
||||
}
|
||||
|
||||
// ---- Local ----
|
||||
|
||||
type Local struct{}
|
||||
|
||||
var uidCache, gidCache = map[uint32]string{}, map[uint32]string{}
|
||||
|
||||
func owner(st os.FileInfo) (string, string) {
|
||||
sys, ok := st.Sys().(*syscall.Stat_t)
|
||||
if !ok {
|
||||
return "", ""
|
||||
}
|
||||
u, ok := uidCache[sys.Uid]
|
||||
if !ok {
|
||||
u = strconv.Itoa(int(sys.Uid))
|
||||
if x, err := user.LookupId(u); err == nil {
|
||||
u = x.Username
|
||||
}
|
||||
uidCache[sys.Uid] = u
|
||||
}
|
||||
g, ok := gidCache[sys.Gid]
|
||||
if !ok {
|
||||
g = strconv.Itoa(int(sys.Gid))
|
||||
if x, err := user.LookupGroupId(g); err == nil {
|
||||
g = x.Name
|
||||
}
|
||||
gidCache[sys.Gid] = g
|
||||
}
|
||||
return u, g
|
||||
}
|
||||
|
||||
func entryOf(dir, name string, lst os.FileInfo) Entry {
|
||||
e := Entry{Name: name, Size: lst.Size(), Mode: lst.Mode().String(), Perm: fmt.Sprintf("%04o", lst.Mode().Perm()), Mtime: lst.ModTime().UnixMilli()}
|
||||
e.User, e.Group = owner(lst)
|
||||
switch {
|
||||
case lst.Mode()&os.ModeSymlink != 0:
|
||||
e.Type = "link"
|
||||
full := filepath.Join(dir, name)
|
||||
e.Target, _ = os.Readlink(full)
|
||||
if st, err := os.Stat(full); err != nil {
|
||||
e.Broken = true
|
||||
} else if st.IsDir() {
|
||||
e.Type = "dir" // keep dir semantics for navigation; Target says it's a link
|
||||
} else {
|
||||
e.Size = st.Size()
|
||||
}
|
||||
case lst.IsDir():
|
||||
e.Type = "dir"
|
||||
case lst.Mode().IsRegular():
|
||||
e.Type = "file"
|
||||
default:
|
||||
e.Type = "other"
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
func (Local) List(_ context.Context, path string) (Listing, error) {
|
||||
ents, err := os.ReadDir(path)
|
||||
if err != nil {
|
||||
return Listing{}, err
|
||||
}
|
||||
l := Listing{Path: path, Entries: make([]Entry, 0, len(ents))}
|
||||
if path != "/" {
|
||||
l.Parent = filepath.Dir(path)
|
||||
}
|
||||
for _, d := range ents {
|
||||
lst, err := d.Info() // Lstat
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
l.Entries = append(l.Entries, entryOf(path, d.Name(), lst))
|
||||
}
|
||||
sort.Slice(l.Entries, func(i, j int) bool {
|
||||
a, b := l.Entries[i], l.Entries[j]
|
||||
if (a.Type == "dir") != (b.Type == "dir") {
|
||||
return a.Type == "dir"
|
||||
}
|
||||
return strings.ToLower(a.Name) < strings.ToLower(b.Name)
|
||||
})
|
||||
return l, nil
|
||||
}
|
||||
|
||||
func (Local) Stat(_ context.Context, path string) (Info, error) {
|
||||
lst, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return Info{}, err
|
||||
}
|
||||
in := Info{Entry: entryOf(filepath.Dir(path), filepath.Base(path), lst), Path: path}
|
||||
if in.Type == "file" && in.Size <= 8<<20 {
|
||||
if f, err := os.Open(path); err == nil {
|
||||
buf := make([]byte, 4096)
|
||||
n, _ := f.Read(buf)
|
||||
f.Close()
|
||||
in.Text = looksText(buf[:n])
|
||||
}
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
||||
func looksText(b []byte) bool {
|
||||
if len(b) == 0 {
|
||||
return true
|
||||
}
|
||||
ctrl := 0
|
||||
for _, c := range b {
|
||||
if c == 0 {
|
||||
return false
|
||||
}
|
||||
if c < 32 && c != '\n' && c != '\r' && c != '\t' && c != '\f' && c != 0x1b {
|
||||
ctrl++
|
||||
}
|
||||
}
|
||||
return ctrl*20 < len(b)
|
||||
}
|
||||
|
||||
// Open refuses anything that isn't a regular file: devices stream forever (/dev/zero),
|
||||
// FIFOs block on open, sockets can't be read. /proc files are regular and still work.
|
||||
func (Local) Open(_ context.Context, path string) (io.ReadCloser, error) {
|
||||
st, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if st.IsDir() {
|
||||
return nil, errors.New("is a directory")
|
||||
}
|
||||
if !st.Mode().IsRegular() {
|
||||
return nil, errors.New("not a regular file (device, pipe or socket)")
|
||||
}
|
||||
return os.Open(path)
|
||||
}
|
||||
|
||||
// Write replaces the file's contents, keeping its mode; new files get 0644 (umask).
|
||||
func (Local) Write(_ context.Context, path string, r io.Reader) error {
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(f, r); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
func (Local) Mkdir(_ context.Context, path string) error { return os.Mkdir(path, 0o755) }
|
||||
|
||||
func (Local) Touch(_ context.Context, path string) error {
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
func (Local) Rename(_ context.Context, from, to string) error {
|
||||
if protected[from] {
|
||||
return errors.New("refusing to move " + from)
|
||||
}
|
||||
if _, err := os.Lstat(to); err == nil {
|
||||
return errors.New("destination exists")
|
||||
}
|
||||
return os.Rename(from, to)
|
||||
}
|
||||
|
||||
func (Local) Delete(_ context.Context, path string, recursive bool) error {
|
||||
if protected[path] {
|
||||
return errors.New("refusing to delete " + path)
|
||||
}
|
||||
lst, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if lst.IsDir() && recursive {
|
||||
return os.RemoveAll(path)
|
||||
}
|
||||
return os.Remove(path) // fails on a non-empty dir, by design
|
||||
}
|
||||
|
||||
func (Local) Chmod(_ context.Context, path string, perm os.FileMode) error {
|
||||
return os.Chmod(path, perm)
|
||||
}
|
||||
|
||||
// ---- Sudo: the same operations, run as root in a child hamlab ----
|
||||
|
||||
type Sudo struct{ Self string }
|
||||
|
||||
func (s Sudo) cmd(ctx context.Context, args ...string) *exec.Cmd {
|
||||
full := priv.Wrap(append([]string{s.Self, "fs"}, args...)...)
|
||||
return exec.CommandContext(ctx, full[0], full[1:]...)
|
||||
}
|
||||
|
||||
func (s Sudo) runJSON(ctx context.Context, out any, args ...string) error {
|
||||
c := s.cmd(ctx, args...)
|
||||
var errb strings.Builder
|
||||
c.Stderr = &errb
|
||||
b, err := c.Output()
|
||||
if err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
return json.Unmarshal(b, out)
|
||||
}
|
||||
|
||||
func (s Sudo) List(ctx context.Context, path string) (Listing, error) {
|
||||
var l Listing
|
||||
err := s.runJSON(ctx, &l, "list", path)
|
||||
return l, err
|
||||
}
|
||||
|
||||
func (s Sudo) Stat(ctx context.Context, path string) (Info, error) {
|
||||
var in Info
|
||||
err := s.runJSON(ctx, &in, "stat", path)
|
||||
return in, err
|
||||
}
|
||||
|
||||
type cmdReader struct {
|
||||
io.ReadCloser
|
||||
cmd *exec.Cmd
|
||||
errb *strings.Builder
|
||||
}
|
||||
|
||||
func (r *cmdReader) Close() error {
|
||||
r.ReadCloser.Close()
|
||||
err := r.cmd.Wait()
|
||||
if err != nil && r.errb.Len() > 0 {
|
||||
return errors.New(strings.TrimSpace(r.errb.String()))
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (s Sudo) Open(ctx context.Context, path string) (io.ReadCloser, error) {
|
||||
c := s.cmd(ctx, "read", path)
|
||||
errb := &strings.Builder{}
|
||||
c.Stderr = errb
|
||||
out, err := c.StdoutPipe()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := c.Start(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Fail early on "no such file" etc.: the helper stats before writing anything.
|
||||
head := make([]byte, 1)
|
||||
n, rerr := out.Read(head)
|
||||
if n == 0 && rerr != nil {
|
||||
_ = c.Wait()
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return nil, errors.New(msg)
|
||||
}
|
||||
if rerr != io.EOF {
|
||||
return nil, rerr
|
||||
}
|
||||
}
|
||||
return &cmdReader{ReadCloser: struct {
|
||||
io.Reader
|
||||
io.Closer
|
||||
}{io.MultiReader(strings.NewReader(string(head[:n])), out), out}, cmd: c, errb: errb}, nil
|
||||
}
|
||||
|
||||
func (s Sudo) Write(ctx context.Context, path string, r io.Reader) error {
|
||||
c := s.cmd(ctx, "write", path)
|
||||
c.Stdin = r
|
||||
var errb strings.Builder
|
||||
c.Stderr = &errb
|
||||
if err := c.Run(); err != nil {
|
||||
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||
return errors.New(msg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s Sudo) Mkdir(ctx context.Context, path string) error {
|
||||
return s.runJSON(ctx, nil, "mkdir", path)
|
||||
}
|
||||
func (s Sudo) Touch(ctx context.Context, path string) error {
|
||||
return s.runJSON(ctx, nil, "touch", path)
|
||||
}
|
||||
func (s Sudo) Rename(ctx context.Context, from, to string) error {
|
||||
return s.runJSON(ctx, nil, "rename", from, to)
|
||||
}
|
||||
func (s Sudo) Delete(ctx context.Context, path string, recursive bool) error {
|
||||
if recursive {
|
||||
return s.runJSON(ctx, nil, "delete", "-r", path)
|
||||
}
|
||||
return s.runJSON(ctx, nil, "delete", path)
|
||||
}
|
||||
func (s Sudo) Chmod(ctx context.Context, path string, perm os.FileMode) error {
|
||||
return s.runJSON(ctx, nil, "chmod", fmt.Sprintf("%04o", perm), path)
|
||||
}
|
||||
|
||||
// Helper implements `hamlab fs <op> …` for the Sudo backend. Returns the exit code.
|
||||
func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer) int {
|
||||
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
|
||||
if len(args) < 2 {
|
||||
return fail(errors.New("usage: hamlab fs <list|stat|read|write|mkdir|touch|rename|delete|chmod> …"))
|
||||
}
|
||||
ctx := context.Background()
|
||||
var l Local
|
||||
op := args[0]
|
||||
clean := func(i int) (string, error) {
|
||||
if i >= len(args) {
|
||||
return "", errors.New("path required")
|
||||
}
|
||||
return Clean(args[i])
|
||||
}
|
||||
switch op {
|
||||
case "list", "stat", "read", "write", "mkdir", "touch":
|
||||
p, err := clean(1)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
switch op {
|
||||
case "list":
|
||||
out, err := l.List(ctx, p)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return emit(stdout, out, stderr)
|
||||
case "stat":
|
||||
out, err := l.Stat(ctx, p)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return emit(stdout, out, stderr)
|
||||
case "read":
|
||||
f, err := l.Open(ctx, p)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := io.Copy(stdout, f); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
return 0
|
||||
case "write":
|
||||
if err := l.Write(ctx, p, stdin); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "mkdir":
|
||||
if err := l.Mkdir(ctx, p); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "touch":
|
||||
if err := l.Touch(ctx, p); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
}
|
||||
case "rename":
|
||||
from, err := clean(1)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
to, err := clean(2)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
if err := l.Rename(ctx, from, to); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "delete":
|
||||
rec := false
|
||||
i := 1
|
||||
if args[1] == "-r" {
|
||||
rec, i = true, 2
|
||||
}
|
||||
p, err := clean(i)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
if err := l.Delete(ctx, p, rec); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
case "chmod":
|
||||
perm, err := ParsePerm(args[1])
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
p, err := clean(2)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
if err := l.Chmod(ctx, p, perm); err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
default:
|
||||
return fail(errors.New("unknown op " + op))
|
||||
}
|
||||
fmt.Fprintln(stdout, "{}")
|
||||
return 0
|
||||
}
|
||||
|
||||
func emit(w io.Writer, v any, stderr io.Writer) int {
|
||||
if err := json.NewEncoder(w).Encode(v); err != nil {
|
||||
fmt.Fprintln(stderr, err.Error())
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ModTime helper for handlers.
|
||||
func Now() time.Time { return time.Now() }
|
||||
@@ -0,0 +1,843 @@
|
||||
// Package hardware is what is physically in the machine: the board it is, its processor,
|
||||
// the memory modules in its slots, the drives attached to it and the graphics in it.
|
||||
//
|
||||
// None of this changes while the machine is running, so it is read once and remembered.
|
||||
// Everything here is read from what Debian and its kin already have — dmidecode for the
|
||||
// parts only the firmware knows, lscpu, lspci, sysfs — and every reader is allowed to
|
||||
// come back empty: a machine with no dmidecode, no PCI bus or no root still gets a page,
|
||||
// with a note saying what could not be read and why.
|
||||
package hardware
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/storage"
|
||||
)
|
||||
|
||||
type System struct {
|
||||
Model string `json:"model,omitempty"` // "GMKtec NucBox G3"
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Family string `json:"family,omitempty"`
|
||||
Kernel string `json:"kernel,omitempty"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
Virt string `json:"virt,omitempty"` // "kvm", "lxc", … when this is not real hardware
|
||||
}
|
||||
|
||||
// Slot is an expansion slot as the firmware describes it. Worth treating with suspicion on
|
||||
// a mini PC, where the table is often copied from a reference design and lists slots the
|
||||
// case has no room for.
|
||||
type Slot struct {
|
||||
Designation string `json:"designation"`
|
||||
Type string `json:"type,omitempty"`
|
||||
Usage string `json:"usage,omitempty"` // Available | In Use | Unknown
|
||||
Length string `json:"length,omitempty"`
|
||||
Width string `json:"width,omitempty"`
|
||||
Bus string `json:"bus,omitempty"` // PCI address of what is in it
|
||||
Device string `json:"device,omitempty"` // and what that turns out to be
|
||||
}
|
||||
|
||||
// Board is the motherboard and the firmware on it.
|
||||
type Board struct {
|
||||
Maker string `json:"maker,omitempty"`
|
||||
Product string `json:"product,omitempty"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
AssetTag string `json:"asset_tag,omitempty"`
|
||||
Type string `json:"type,omitempty"` // "Motherboard", "Processor Module", …
|
||||
Location string `json:"location,omitempty"` // where it sits in the chassis
|
||||
Chassis string `json:"chassis,omitempty"`
|
||||
ChassisMaker string `json:"chassis_maker,omitempty"`
|
||||
BIOS string `json:"bios,omitempty"` // vendor and version
|
||||
BIOSDate string `json:"bios_date,omitempty"`
|
||||
BIOSRevision string `json:"bios_revision,omitempty"`
|
||||
ROMSize string `json:"rom_size,omitempty"`
|
||||
UEFI bool `json:"uefi,omitempty"`
|
||||
Slots []Slot `json:"slots,omitempty"`
|
||||
}
|
||||
|
||||
type Cache struct {
|
||||
Level string `json:"level"` // L1d, L1i, L2, L3
|
||||
Size string `json:"size"` // as lscpu puts it: "6 MiB (1 instance)"
|
||||
}
|
||||
|
||||
type CPU struct {
|
||||
Model string `json:"model,omitempty"`
|
||||
Vendor string `json:"vendor,omitempty"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
Sockets int `json:"sockets,omitempty"`
|
||||
SocketNames []string `json:"socket_names,omitempty"` // what the firmware calls each socket
|
||||
Cores int `json:"cores,omitempty"`
|
||||
Threads int `json:"threads,omitempty"`
|
||||
ThreadsPerCore int `json:"threads_per_core,omitempty"`
|
||||
MinMHz float64 `json:"min_mhz,omitempty"`
|
||||
MaxMHz float64 `json:"max_mhz,omitempty"`
|
||||
Family string `json:"family,omitempty"`
|
||||
ModelID string `json:"model_id,omitempty"`
|
||||
Stepping string `json:"stepping,omitempty"`
|
||||
Microcode string `json:"microcode,omitempty"`
|
||||
Virtualization string `json:"virtualization,omitempty"`
|
||||
Caches []Cache `json:"caches,omitempty"`
|
||||
Flags []string `json:"flags,omitempty"` // the handful anyone looks for
|
||||
}
|
||||
|
||||
type Module struct {
|
||||
Locator string `json:"locator"` // the slot, as the board labels it
|
||||
Bank string `json:"bank,omitempty"`
|
||||
Empty bool `json:"empty,omitempty"`
|
||||
Size uint64 `json:"size,omitempty"`
|
||||
Type string `json:"type,omitempty"` // DDR4
|
||||
FormFactor string `json:"form_factor,omitempty"` // SODIMM
|
||||
SpeedMTs int `json:"speed_mts,omitempty"` // what the module can do
|
||||
RunningMTs int `json:"running_mts,omitempty"` // what the board is running it at
|
||||
Maker string `json:"maker,omitempty"`
|
||||
Part string `json:"part,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Rank int `json:"rank,omitempty"`
|
||||
Voltage string `json:"voltage,omitempty"`
|
||||
}
|
||||
|
||||
type Memory struct {
|
||||
Total uint64 `json:"total"` // what the kernel can use
|
||||
Installed uint64 `json:"installed,omitempty"` // what the modules add up to
|
||||
MaxCapacity uint64 `json:"max_capacity,omitempty"` // what the board will take
|
||||
Slots int `json:"slots,omitempty"`
|
||||
Filled int `json:"filled,omitempty"`
|
||||
ECC string `json:"ecc,omitempty"`
|
||||
Modules []Module `json:"modules,omitempty"`
|
||||
}
|
||||
|
||||
type Drive struct {
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"` // nvme | ssd | hdd | usb | emmc | sd | other
|
||||
Model string `json:"model,omitempty"`
|
||||
Vendor string `json:"vendor,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Firmware string `json:"firmware,omitempty"`
|
||||
Tran string `json:"tran,omitempty"`
|
||||
Size uint64 `json:"size"`
|
||||
Rotational bool `json:"rotational,omitempty"`
|
||||
Removable bool `json:"removable,omitempty"`
|
||||
}
|
||||
|
||||
type GPU struct {
|
||||
Model string `json:"model,omitempty"`
|
||||
Vendor string `json:"vendor,omitempty"`
|
||||
Slot string `json:"slot,omitempty"` // PCI address, when it is on a bus
|
||||
Driver string `json:"driver,omitempty"`
|
||||
Node string `json:"node,omitempty"` // card0
|
||||
Memory uint64 `json:"memory,omitempty"`
|
||||
Class string `json:"class,omitempty"` // VGA | 3D | Display
|
||||
}
|
||||
|
||||
type Report struct {
|
||||
At time.Time `json:"at"`
|
||||
System System `json:"system"`
|
||||
Board Board `json:"board"`
|
||||
CPU CPU `json:"cpu"`
|
||||
Memory Memory `json:"memory"`
|
||||
Drives []Drive `json:"drives"`
|
||||
GPUs []GPU `json:"gpus"`
|
||||
Notes []string `json:"notes,omitempty"` // what could not be read, and why
|
||||
}
|
||||
|
||||
// Manager remembers the last report. Hardware does not change under a running kernel —
|
||||
// the one thing that would, a drive plugged in, is what Refresh is for.
|
||||
type Manager struct {
|
||||
mu sync.Mutex
|
||||
last *Report
|
||||
at time.Time
|
||||
}
|
||||
|
||||
func New() *Manager { return &Manager{} }
|
||||
|
||||
const keepFor = 10 * time.Minute
|
||||
|
||||
func (m *Manager) Report(ctx context.Context, refresh bool) *Report {
|
||||
m.mu.Lock()
|
||||
if !refresh && m.last != nil && time.Since(m.at) < keepFor {
|
||||
r := m.last
|
||||
m.mu.Unlock()
|
||||
return r
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
r := collect(ctx)
|
||||
|
||||
m.mu.Lock()
|
||||
m.last, m.at = r, time.Now()
|
||||
m.mu.Unlock()
|
||||
return r
|
||||
}
|
||||
|
||||
func collect(ctx context.Context) *Report {
|
||||
ctx, cancel := context.WithTimeout(ctx, 25*time.Second)
|
||||
defer cancel()
|
||||
r := &Report{At: time.Now()}
|
||||
dmi, note := readDMI(ctx)
|
||||
if note != "" {
|
||||
r.Notes = append(r.Notes, note)
|
||||
}
|
||||
r.System = system(ctx, dmi)
|
||||
r.Board = board(dmi, pciNames(ctx))
|
||||
r.CPU = cpu(ctx, dmi)
|
||||
r.Memory = memory(dmi)
|
||||
if len(r.Memory.Modules) == 0 && note == "" {
|
||||
r.Notes = append(r.Notes, "memory modules: the firmware's table says nothing about this machine's slots")
|
||||
}
|
||||
r.Drives = drives(ctx)
|
||||
r.GPUs = gpus(ctx)
|
||||
if len(r.GPUs) == 0 {
|
||||
r.Notes = append(r.Notes, "graphics: nothing on the PCI bus and no kernel display device")
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// ---- dmidecode ----
|
||||
|
||||
// record is one entry of the firmware's table: its DMI type and its key/value lines.
|
||||
type record struct {
|
||||
Type int
|
||||
V map[string]string
|
||||
}
|
||||
|
||||
// listKey holds every doubly-indented line of a record — dmidecode's lists, which have no
|
||||
// key of their own: BIOS characteristics, board features, slot characteristics.
|
||||
const listKey = "\x00list"
|
||||
|
||||
var handleRe = regexp.MustCompile(`^Handle 0x[0-9A-Fa-f]+, DMI type (\d+)`)
|
||||
|
||||
// readDMI runs dmidecode once for every type the page wants and parses the lot. It needs
|
||||
// root; without it the page loses the board, the memory modules and the socket names, and
|
||||
// says so rather than pretending the machine has none.
|
||||
func readDMI(ctx context.Context) ([]record, string) {
|
||||
tool, err := exec.LookPath("dmidecode")
|
||||
if err != nil {
|
||||
if _, e := os.Stat("/usr/sbin/dmidecode"); e != nil {
|
||||
return nil, "the board and the memory modules need dmidecode, which isn't installed here"
|
||||
}
|
||||
tool = "/usr/sbin/dmidecode"
|
||||
}
|
||||
args := []string{tool, "-t", "bios", "-t", "system", "-t", "baseboard", "-t", "chassis", "-t", "slot", "-t", "processor", "-t", "memory"}
|
||||
if os.Geteuid() != 0 {
|
||||
if !priv.Available() {
|
||||
return nil, "the board and the memory modules come from the firmware's table, which only root may read"
|
||||
}
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, args[0], args[1:]...).Output()
|
||||
if err != nil {
|
||||
return nil, "the firmware's table could not be read: " + strings.TrimSpace(err.Error())
|
||||
}
|
||||
return parse(string(out)), ""
|
||||
}
|
||||
|
||||
// parse turns dmidecode's output into records: a handle line starts one, and the single
|
||||
// indented "Key: Value" lines under it are its fields. Doubly-indented lines are the
|
||||
// continuation of a list (a processor's flags), which nothing here reads.
|
||||
func parse(text string) []record {
|
||||
var recs []record
|
||||
var cur *record
|
||||
sc := bufio.NewScanner(strings.NewReader(text))
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 2<<20)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if m := handleRe.FindStringSubmatch(line); m != nil {
|
||||
t, _ := strconv.Atoi(m[1])
|
||||
recs = append(recs, record{Type: t, V: map[string]string{}})
|
||||
cur = &recs[len(recs)-1]
|
||||
continue
|
||||
}
|
||||
if cur == nil || !strings.HasPrefix(line, "\t") {
|
||||
if strings.TrimSpace(line) == "" {
|
||||
cur = nil
|
||||
}
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(line, "\t\t") { // a list under the field above it, e.g. BIOS characteristics
|
||||
cur.V[listKey] += strings.TrimSpace(line) + "\n"
|
||||
continue
|
||||
}
|
||||
k, v, ok := strings.Cut(strings.TrimSpace(line), ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
cur.V[k] = strings.TrimSpace(v)
|
||||
}
|
||||
return recs
|
||||
}
|
||||
|
||||
func (r record) get(keys ...string) string {
|
||||
for _, k := range keys {
|
||||
if v := r.V[k]; v != "" && !unset(v) {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// unset is every way the firmware has of saying "nobody filled this in".
|
||||
func unset(v string) bool {
|
||||
switch strings.ToLower(v) {
|
||||
case "", "unknown", "not specified", "not provided", "none", "default string", "to be filled by o.e.m.",
|
||||
"system manufacturer", "system product name", "to be filled by oem", "n/a", "empty", "no module installed",
|
||||
"<out of spec>", "other", "not applicable":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func first(recs []record, t int) record {
|
||||
for _, r := range recs {
|
||||
if r.Type == t {
|
||||
return r
|
||||
}
|
||||
}
|
||||
return record{Type: t, V: map[string]string{}}
|
||||
}
|
||||
|
||||
func all(recs []record, t int) []record {
|
||||
var out []record
|
||||
for _, r := range recs {
|
||||
if r.Type == t {
|
||||
out = append(out, r)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- system ----
|
||||
|
||||
func system(ctx context.Context, dmi []record) System {
|
||||
sys := first(dmi, 1)
|
||||
s := System{
|
||||
Model: join(sys.get("Manufacturer"), sys.get("Product Name")),
|
||||
Serial: sys.get("Serial Number"),
|
||||
Family: sys.get("Family"),
|
||||
Arch: runtimeArch(),
|
||||
}
|
||||
s.Kernel = readFile("/proc/sys/kernel/osrelease")
|
||||
if s.Model == "" { // no SMBIOS: a Pi, a Mac, anything with a device tree
|
||||
s.Model = strings.TrimRight(readFile("/sys/firmware/devicetree/base/model"), "\x00")
|
||||
}
|
||||
if out, err := exec.CommandContext(ctx, "systemd-detect-virt").Output(); err == nil {
|
||||
if v := strings.TrimSpace(string(out)); v != "" && v != "none" {
|
||||
s.Virt = v
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// board reads the motherboard, the chassis it sits in, the firmware on it and the
|
||||
// expansion slots the firmware claims. All of it is SMBIOS, so a machine without one — a
|
||||
// Mac, a Pi — gets an empty board, and the page says so rather than inventing one.
|
||||
func board(dmi []record, pci map[string]string) Board {
|
||||
b, ch, bios := first(dmi, 2), first(dmi, 3), first(dmi, 0)
|
||||
out := Board{
|
||||
Maker: b.get("Manufacturer"),
|
||||
Product: b.get("Product Name"),
|
||||
Version: b.get("Version"),
|
||||
Serial: b.get("Serial Number"),
|
||||
AssetTag: b.get("Asset Tag"),
|
||||
Type: b.get("Type"),
|
||||
Location: b.get("Location In Chassis"),
|
||||
Chassis: ch.get("Type"),
|
||||
ChassisMaker: ch.get("Manufacturer"),
|
||||
BIOS: join(bios.get("Vendor"), bios.get("Version")),
|
||||
BIOSDate: bios.get("Release Date"),
|
||||
BIOSRevision: bios.get("BIOS Revision"),
|
||||
}
|
||||
// "ROM Size: 0 MB" is a board that never filled the field in, not a board with no ROM
|
||||
if rom := bios.get("ROM Size"); rom != "" && !strings.HasPrefix(rom, "0 ") {
|
||||
out.ROMSize = rom
|
||||
}
|
||||
if out.Product == out.Maker {
|
||||
out.Product = "" // a board named after its maker says nothing twice
|
||||
}
|
||||
out.UEFI = strings.Contains(bios.V[listKey], "UEFI") // a characteristics line, not a field
|
||||
for _, sl := range all(dmi, 9) {
|
||||
s := Slot{
|
||||
Designation: sl.get("Designation"),
|
||||
Type: sl.get("Type"),
|
||||
Usage: sl.get("Current Usage"),
|
||||
Length: sl.get("Length"),
|
||||
Width: sl.get("Data Bus Width"),
|
||||
}
|
||||
// every slot on some boards claims 0000:00:00.0, which is the host bridge and
|
||||
// certainly not what is plugged into that slot
|
||||
if bus := sl.get("Bus Address"); bus != "" && bus != "0000:00:00.0" {
|
||||
s.Bus, s.Device = bus, pci[bus]
|
||||
}
|
||||
if s.Designation != "" {
|
||||
out.Slots = append(out.Slots, s)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func join(parts ...string) string {
|
||||
var keep []string
|
||||
for _, p := range parts {
|
||||
if p = strings.TrimSpace(p); p != "" && !unset(p) {
|
||||
keep = append(keep, p)
|
||||
}
|
||||
}
|
||||
// "GMKtec GMKtec" reads badly: drop a maker the product already names
|
||||
if len(keep) == 2 && strings.HasPrefix(strings.ToLower(keep[1]), strings.ToLower(keep[0])) {
|
||||
return keep[1]
|
||||
}
|
||||
return strings.Join(keep, " ")
|
||||
}
|
||||
|
||||
func readFile(p string) string {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
func runtimeArch() string {
|
||||
if out, err := exec.Command("uname", "-m").Output(); err == nil {
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- processor ----
|
||||
|
||||
// worthShowing is the short list of processor features anyone actually asks about; the
|
||||
// full flag list runs to hundreds and belongs in a terminal.
|
||||
var worthShowing = []string{"aes", "avx", "avx2", "avx512f", "sha_ni", "vmx", "svm", "sse4_2", "rdrand", "hypervisor"}
|
||||
|
||||
func cpu(ctx context.Context, dmi []record) CPU {
|
||||
c := CPU{}
|
||||
f := lscpu(ctx)
|
||||
c.Model = f["Model name"]
|
||||
c.Vendor = f["Vendor ID"]
|
||||
c.Arch = f["Architecture"]
|
||||
c.Sockets = atoi(f["Socket(s)"])
|
||||
c.ThreadsPerCore = atoi(f["Thread(s) per core"])
|
||||
c.Threads = atoi(f["CPU(s)"])
|
||||
c.Cores = atoi(f["Core(s) per socket"]) * maxInt(1, c.Sockets)
|
||||
c.MinMHz = atof(f["CPU min MHz"])
|
||||
c.MaxMHz = atof(f["CPU max MHz"])
|
||||
c.Family, c.ModelID, c.Stepping = f["CPU family"], f["Model"], f["Stepping"]
|
||||
c.Virtualization = f["Virtualization"]
|
||||
for _, l := range []string{"L1d", "L1i", "L2", "L3"} {
|
||||
if v := f[l+" cache"]; v != "" {
|
||||
c.Caches = append(c.Caches, Cache{Level: l, Size: v})
|
||||
}
|
||||
}
|
||||
have := map[string]bool{}
|
||||
for _, fl := range strings.Fields(f["Flags"]) {
|
||||
have[fl] = true
|
||||
}
|
||||
for _, fl := range worthShowing {
|
||||
if have[fl] {
|
||||
c.Flags = append(c.Flags, fl)
|
||||
}
|
||||
}
|
||||
for _, p := range all(dmi, 4) {
|
||||
if s := p.get("Socket Designation"); s != "" {
|
||||
c.SocketNames = append(c.SocketNames, s)
|
||||
}
|
||||
if c.Model == "" {
|
||||
c.Model = p.get("Version", "Family")
|
||||
}
|
||||
if c.MaxMHz == 0 {
|
||||
if mhz := atof(strings.TrimSuffix(p.get("Max Speed"), " MHz")); mhz > 0 {
|
||||
c.MaxMHz = mhz
|
||||
}
|
||||
}
|
||||
}
|
||||
if c.Sockets == 0 {
|
||||
c.Sockets = len(c.SocketNames)
|
||||
}
|
||||
// A processor whose cores are not all alike — Apple's, Intel's P and E cores — is
|
||||
// reported by lscpu as several groups, and "core(s) per socket" then counts one group
|
||||
// rather than the chip. Without SMT every thread is a core, so the thread count is the
|
||||
// honest answer.
|
||||
if c.ThreadsPerCore == 1 && c.Threads > c.Cores {
|
||||
c.Cores = c.Threads
|
||||
}
|
||||
// /proc/cpuinfo is the only place the running microcode revision is written down
|
||||
if b, err := os.ReadFile("/proc/cpuinfo"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if k, v, ok := strings.Cut(line, ":"); ok && strings.TrimSpace(k) == "microcode" {
|
||||
c.Microcode = strings.TrimSpace(v)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if c.Model == "" {
|
||||
c.Model = "unknown processor"
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func lscpu(ctx context.Context) map[string]string {
|
||||
f := map[string]string{}
|
||||
out, err := exec.CommandContext(ctx, "lscpu").Output()
|
||||
if err != nil {
|
||||
// no util-linux: /proc/cpuinfo still names the thing and counts it
|
||||
if b, err := os.ReadFile("/proc/cpuinfo"); err == nil {
|
||||
n := 0
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
k, v, ok := strings.Cut(line, ":")
|
||||
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
|
||||
switch {
|
||||
case ok && strings.EqualFold(k, "model name"):
|
||||
f["Model name"] = v
|
||||
case ok && k == "processor":
|
||||
n++
|
||||
}
|
||||
}
|
||||
f["CPU(s)"] = strconv.Itoa(n)
|
||||
}
|
||||
return f
|
||||
}
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
k, v, ok := strings.Cut(line, ":")
|
||||
if !ok || strings.HasPrefix(line, " ") {
|
||||
continue
|
||||
}
|
||||
f[strings.TrimSpace(k)] = strings.TrimSpace(v)
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
func atoi(s string) int { n, _ := strconv.Atoi(strings.TrimSpace(s)); return n }
|
||||
|
||||
func atof(s string) float64 {
|
||||
v, _ := strconv.ParseFloat(strings.TrimSpace(s), 64)
|
||||
return v
|
||||
}
|
||||
|
||||
func maxInt(a, b int) int {
|
||||
if a > b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// leadingInt reads the number off the front of "3200 MT/s".
|
||||
func leadingInt(s string) int {
|
||||
f := strings.Fields(s)
|
||||
if len(f) == 0 {
|
||||
return 0
|
||||
}
|
||||
return atoi(f[0])
|
||||
}
|
||||
|
||||
// ---- memory ----
|
||||
|
||||
var sizeRe = regexp.MustCompile(`^(\d+)\s*(kB|KB|MB|GB|TB)$`)
|
||||
|
||||
// dmiSize turns dmidecode's "8 GB" / "16384 MB" into bytes.
|
||||
func dmiSize(s string) uint64 {
|
||||
m := sizeRe.FindStringSubmatch(strings.TrimSpace(s))
|
||||
if m == nil {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.ParseUint(m[1], 10, 64)
|
||||
switch strings.ToUpper(m[2]) {
|
||||
case "KB":
|
||||
return n << 10
|
||||
case "MB":
|
||||
return n << 20
|
||||
case "GB":
|
||||
return n << 30
|
||||
case "TB":
|
||||
return n << 40
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func memory(dmi []record) Memory {
|
||||
m := Memory{}
|
||||
if b, err := os.ReadFile("/proc/meminfo"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if k, v, ok := strings.Cut(line, ":"); ok && k == "MemTotal" {
|
||||
m.Total = uint64(atoi(strings.TrimSuffix(strings.TrimSpace(v), " kB"))) << 10
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, a := range all(dmi, 16) {
|
||||
m.MaxCapacity += dmiSize(a.get("Maximum Capacity"))
|
||||
switch e := a.V["Error Correction Type"]; {
|
||||
case e == "None":
|
||||
m.ECC = "non-ECC" // the array says so outright; worth knowing, not worth hiding
|
||||
case e != "" && !unset(e):
|
||||
m.ECC = e
|
||||
}
|
||||
}
|
||||
for _, d := range all(dmi, 17) {
|
||||
mod := Module{
|
||||
Locator: d.get("Locator"),
|
||||
Bank: d.get("Bank Locator"),
|
||||
Size: dmiSize(d.get("Size")),
|
||||
}
|
||||
m.Slots++
|
||||
if mod.Locator == "" {
|
||||
mod.Locator = "slot " + strconv.Itoa(m.Slots)
|
||||
}
|
||||
if mod.Size == 0 {
|
||||
mod.Empty = true
|
||||
m.Modules = append(m.Modules, mod)
|
||||
continue
|
||||
}
|
||||
m.Filled++
|
||||
mod.Type = d.get("Type")
|
||||
mod.FormFactor = d.get("Form Factor")
|
||||
mod.SpeedMTs = leadingInt(d.get("Speed"))
|
||||
mod.RunningMTs = leadingInt(d.get("Configured Memory Speed", "Configured Clock Speed"))
|
||||
mod.Maker = makerName(d.get("Manufacturer"), d.V["Module Manufacturer ID"])
|
||||
mod.Part = d.get("Part Number")
|
||||
mod.Serial = d.get("Serial Number")
|
||||
mod.Rank = atoi(d.get("Rank"))
|
||||
mod.Voltage = d.get("Configured Voltage")
|
||||
m.Installed += mod.Size
|
||||
m.Modules = append(m.Modules, mod)
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// jedec maps the memory makers who turn up in a homelab to their names: SPD carries a
|
||||
// JEDEC id, and dmidecode prints the raw code when the module's string field is empty.
|
||||
var jedec = map[string]string{
|
||||
"0X2C00": "Micron", "0X802C": "Micron",
|
||||
"0XCE00": "Samsung", "0X80CE": "Samsung", "0X00CE": "Samsung",
|
||||
"0XAD00": "SK hynix", "0X80AD": "SK hynix",
|
||||
"0X9801": "Kingston", "0X7F98": "Kingston", "0X0198": "Kingston",
|
||||
"0X859B": "Crucial", "0X04CB": "A-DATA", "0X04CD": "G.Skill",
|
||||
"0X029E": "Corsair", "0X0B45": "Wilk Elektronik", "0X830B": "Wilk Elektronik",
|
||||
"0X0443": "Ramaxel", "0X0194": "Smart Modular",
|
||||
}
|
||||
|
||||
// makerName prefers what the module says it is; a bare JEDEC id becomes a name when it is
|
||||
// one we know, and stays the id when it is not — a wrong name is worse than a number.
|
||||
func makerName(v, id string) string {
|
||||
if v != "" && !unset(v) && !strings.HasPrefix(v, "0x") {
|
||||
return v
|
||||
}
|
||||
code := v
|
||||
if code == "" {
|
||||
code = id
|
||||
}
|
||||
if n, ok := jedec[strings.ToUpper(code)]; ok {
|
||||
return n
|
||||
}
|
||||
if v != "" && !unset(v) {
|
||||
return v
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- drives ----
|
||||
|
||||
func drives(ctx context.Context) []Drive {
|
||||
devs, err := storage.Devices(ctx)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]Drive, 0, len(devs))
|
||||
for _, d := range devs {
|
||||
out = append(out, Drive{
|
||||
Name: d.Name, Kind: d.Kind, Model: d.Model, Vendor: d.Vendor, Serial: d.Serial,
|
||||
Tran: d.Tran, Size: d.Size, Rotational: d.Rotational, Removable: d.Removable,
|
||||
Firmware: firmwareOf(d.Name),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// firmwareOf reads a drive's firmware revision. An NVMe device is folded to its
|
||||
// controller ("nvme0"), which has no /sys/block entry of its own — the revision is on the
|
||||
// controller in /sys/class/nvme.
|
||||
func firmwareOf(name string) string {
|
||||
return firstNonEmpty(
|
||||
readFile("/sys/block/"+name+"/device/firmware_rev"),
|
||||
readFile("/sys/block/"+name+"/device/rev"),
|
||||
readFile("/sys/class/nvme/"+name+"/firmware_rev"),
|
||||
)
|
||||
}
|
||||
|
||||
func firstNonEmpty(v ...string) string {
|
||||
for _, s := range v {
|
||||
if s != "" {
|
||||
return s
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- graphics ----
|
||||
|
||||
// lspci -mm quotes every field, so a device name can hold the spaces and brackets it
|
||||
// likes: `"VGA compatible controller [0300]" "Intel Corporation [8086]" "Alder Lake-N …"`.
|
||||
var quoted = regexp.MustCompile(`"([^"]*)"`)
|
||||
|
||||
// stripID drops the trailing PCI id from `Alder Lake-N [UHD Graphics] [46d1]`.
|
||||
func stripID(s string) string {
|
||||
if i := strings.LastIndex(s, " ["); i > 0 && strings.HasSuffix(s, "]") {
|
||||
return strings.TrimSpace(s[:i])
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func gpus(ctx context.Context) []GPU {
|
||||
var out []GPU
|
||||
seen := map[string]bool{} // PCI addresses already accounted for
|
||||
if _, err := exec.LookPath("lspci"); err == nil {
|
||||
drivers := pciDrivers(ctx)
|
||||
if b, err := exec.CommandContext(ctx, "lspci", "-D", "-mm", "-nn").Output(); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
slot, _, _ := strings.Cut(line, " ")
|
||||
fields := quoted.FindAllStringSubmatch(line, -1)
|
||||
if len(fields) < 3 {
|
||||
continue
|
||||
}
|
||||
kind := displayClass(fields[0][1])
|
||||
if kind == "" {
|
||||
continue
|
||||
}
|
||||
seen[slot] = true
|
||||
g := GPU{Slot: slot, Class: kind, Vendor: stripID(fields[1][1]), Model: stripID(fields[2][1]), Driver: drivers[slot]}
|
||||
g.Node, g.Memory = drmFor(slot)
|
||||
out = append(out, g)
|
||||
}
|
||||
}
|
||||
}
|
||||
// A GPU that is not on a PCI bus at all — an Apple Silicon Mac, a Pi, anything with a
|
||||
// device tree — is only visible as a kernel display device.
|
||||
cards, _ := filepath.Glob("/sys/class/drm/card[0-9]")
|
||||
for _, c := range cards {
|
||||
dev, err := filepath.EvalSymlinks(c + "/device")
|
||||
if err != nil || seen[filepath.Base(dev)] {
|
||||
continue
|
||||
}
|
||||
g := GPU{Node: filepath.Base(c), Class: "Display"}
|
||||
if drv, err := filepath.EvalSymlinks(dev + "/driver"); err == nil {
|
||||
g.Driver = filepath.Base(drv)
|
||||
}
|
||||
g.Model = deviceTreeName(dev)
|
||||
if g.Model == "" && g.Driver != "" {
|
||||
g.Model = g.Driver + " display"
|
||||
}
|
||||
if g.Model == "" {
|
||||
continue
|
||||
}
|
||||
g.Memory = vram(dev)
|
||||
out = append(out, g)
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].Slot < out[j].Slot })
|
||||
return out
|
||||
}
|
||||
|
||||
// pciNames maps a PCI address to a readable name, so a slot the firmware says is in use
|
||||
// can say what is in it.
|
||||
func pciNames(ctx context.Context) map[string]string {
|
||||
m := map[string]string{}
|
||||
if _, err := exec.LookPath("lspci"); err != nil {
|
||||
return m
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, "lspci", "-D", "-mm", "-nn").Output()
|
||||
if err != nil {
|
||||
return m
|
||||
}
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
slot, _, _ := strings.Cut(line, " ")
|
||||
f := quoted.FindAllStringSubmatch(line, -1)
|
||||
if slot == "" || len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
m[slot] = strings.TrimSpace(stripID(f[1][1]) + " " + stripID(f[2][1]))
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func displayClass(c string) string {
|
||||
switch {
|
||||
case strings.Contains(c, "[0300]"):
|
||||
return "VGA"
|
||||
case strings.Contains(c, "[0302]"):
|
||||
return "3D"
|
||||
case strings.Contains(c, "[0301]"), strings.Contains(c, "[0380]"):
|
||||
return "Display"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// pciDrivers maps a PCI address to the kernel module driving it.
|
||||
func pciDrivers(ctx context.Context) map[string]string {
|
||||
m := map[string]string{}
|
||||
out, err := exec.CommandContext(ctx, "lspci", "-D", "-k").Output()
|
||||
if err != nil {
|
||||
return m
|
||||
}
|
||||
slot := ""
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
if !strings.HasPrefix(line, "\t") && !strings.HasPrefix(line, " ") {
|
||||
slot, _, _ = strings.Cut(line, " ")
|
||||
continue
|
||||
}
|
||||
if v, ok := strings.CutPrefix(strings.TrimSpace(line), "Kernel driver in use:"); ok && slot != "" {
|
||||
m[slot] = strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// drmFor finds the kernel display device for a PCI address, and its video memory when the
|
||||
// driver publishes it (amdgpu does; Intel's integrated graphics have none of their own).
|
||||
func drmFor(slot string) (string, uint64) {
|
||||
cards, _ := filepath.Glob("/sys/class/drm/card[0-9]")
|
||||
for _, c := range cards {
|
||||
dev, err := filepath.EvalSymlinks(c + "/device")
|
||||
if err != nil || filepath.Base(dev) != slot {
|
||||
continue
|
||||
}
|
||||
return filepath.Base(c), vram(dev)
|
||||
}
|
||||
return "", 0
|
||||
}
|
||||
|
||||
func vram(dev string) uint64 {
|
||||
for _, f := range []string{"mem_info_vram_total", "mem_info_vis_vram_total"} {
|
||||
if v := readFile(dev + "/" + f); v != "" {
|
||||
n, _ := strconv.ParseUint(v, 10, 64)
|
||||
return n
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// deviceTreeName reads what a device-tree machine calls its graphics: the compatible
|
||||
// string, tidied — "apple,agx-g13g" becomes "Apple agx-g13g".
|
||||
func deviceTreeName(dev string) string {
|
||||
c := readFile(dev + "/of_node/compatible")
|
||||
if c == "" {
|
||||
return ""
|
||||
}
|
||||
c = strings.TrimRight(strings.Split(c, "\x00")[0], "\x00")
|
||||
maker, rest, ok := strings.Cut(c, ",")
|
||||
if !ok {
|
||||
return c
|
||||
}
|
||||
if maker != "" {
|
||||
maker = strings.ToUpper(maker[:1]) + maker[1:]
|
||||
}
|
||||
return strings.TrimSpace(maker + " " + rest)
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
package hardware
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestDMISize(t *testing.T) {
|
||||
for in, want := range map[string]uint64{
|
||||
"8 GB": 8 << 30, "16384 MB": 16384 << 20, "512 kB": 512 << 10, "1 TB": 1 << 40,
|
||||
"No Module Installed": 0, "": 0, "8 GiB": 0,
|
||||
} {
|
||||
if got := dmiSize(in); got != want {
|
||||
t.Errorf("dmiSize(%q) = %d, want %d", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMakerName(t *testing.T) {
|
||||
cases := []struct{ v, id, want string }{
|
||||
{"Samsung", "", "Samsung"}, // the module says who it is
|
||||
{"0x0B45", "Bank 12, Hex 0x45", "Wilk Elektronik"}, // only a JEDEC id, and one we know
|
||||
{"0xFFFF", "", "0xFFFF"}, // one we don't: the number, not a guess
|
||||
{"Unknown", "", ""},
|
||||
{"", "0xCE00", "Samsung"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := makerName(c.v, c.id); got != c.want {
|
||||
t.Errorf("makerName(%q, %q) = %q, want %q", c.v, c.id, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStripIDAndDisplayClass(t *testing.T) {
|
||||
if got := stripID("Alder Lake-N [UHD Graphics] [46d1]"); got != "Alder Lake-N [UHD Graphics]" {
|
||||
t.Errorf("stripID = %q", got)
|
||||
}
|
||||
if got := stripID("no brackets here"); got != "no brackets here" {
|
||||
t.Errorf("stripID left a plain name alone wrong: %q", got)
|
||||
}
|
||||
if displayClass(`VGA compatible controller [0300]`) != "VGA" {
|
||||
t.Error("a VGA controller should be graphics")
|
||||
}
|
||||
if displayClass(`Ethernet controller [0200]`) != "" {
|
||||
t.Error("a network card is not graphics")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLeadingIntAndJoin(t *testing.T) {
|
||||
if leadingInt("3200 MT/s") != 3200 || leadingInt("") != 0 || leadingInt("Unknown") != 0 {
|
||||
t.Error("leadingInt")
|
||||
}
|
||||
if got := join("GMKtec", "GMKtec NucBox G3"); got != "GMKtec NucBox G3" {
|
||||
t.Errorf("a maker the product already names should not be said twice: %q", got)
|
||||
}
|
||||
if got := join("Dell Inc.", "OptiPlex 7040"); got != "Dell Inc. OptiPlex 7040" {
|
||||
t.Errorf("join = %q", got)
|
||||
}
|
||||
if got := join("To Be Filled By O.E.M.", "System Product Name"); got != "" {
|
||||
t.Errorf("firmware boilerplate is not a model: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
const sample = `# dmidecode 3.6
|
||||
Handle 0x0028, DMI type 17, 92 bytes
|
||||
Memory Device
|
||||
Array Handle: 0x0027
|
||||
Size: 8 GB
|
||||
Form Factor: SODIMM
|
||||
Locator: Controller0-ChannelA-DIMM0
|
||||
Type: DDR4
|
||||
Speed: 3200 MT/s
|
||||
Manufacturer: 0x0B45
|
||||
Part Number: WPBH32D408SWD-8G
|
||||
Configured Memory Speed: 2933 MT/s
|
||||
Rank: 1
|
||||
|
||||
Handle 0x0029, DMI type 17, 92 bytes
|
||||
Memory Device
|
||||
Size: No Module Installed
|
||||
Locator: Controller1-ChannelA-DIMM0
|
||||
Type: Unknown
|
||||
`
|
||||
|
||||
// parseRecords is what readDMI does once dmidecode has answered; this exercises it
|
||||
// without needing root or the tool.
|
||||
func TestMemoryFromRecords(t *testing.T) {
|
||||
recs := parse(sample)
|
||||
m := memory(recs)
|
||||
if m.Slots != 2 || m.Filled != 1 {
|
||||
t.Fatalf("slots %d filled %d, want 2 and 1", m.Slots, m.Filled)
|
||||
}
|
||||
mod := m.Modules[0]
|
||||
if mod.Size != 8<<30 || mod.Type != "DDR4" || mod.Maker != "Wilk Elektronik" || mod.Part != "WPBH32D408SWD-8G" {
|
||||
t.Errorf("first module read wrong: %+v", mod)
|
||||
}
|
||||
if mod.SpeedMTs != 3200 || mod.RunningMTs != 2933 {
|
||||
t.Errorf("a module running below its rating should say both: %d / %d", mod.RunningMTs, mod.SpeedMTs)
|
||||
}
|
||||
if !m.Modules[1].Empty || m.Modules[1].Locator != "Controller1-ChannelA-DIMM0" {
|
||||
t.Errorf("empty slot read wrong: %+v", m.Modules[1])
|
||||
}
|
||||
if m.Installed != 8<<30 {
|
||||
t.Errorf("installed = %d", m.Installed)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,231 @@
|
||||
package incus
|
||||
|
||||
// Nightly exports: every instance as a tarball in virt/exports/<name>/, kept to a count, as a
|
||||
// cron line like the ZFS schedules — `hamlab virt-export` in /etc/cron.d/hamlab-virt — so the
|
||||
// backup runs whether or not the console is up. The settings and the last run's record are
|
||||
// files beside the data, not state.json, because the cron job is a separate process.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const CronFile = "/etc/cron.d/hamlab-virt"
|
||||
|
||||
type ExportConfig struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Hour int `json:"hour"`
|
||||
Minute int `json:"minute"`
|
||||
Keep int `json:"keep"` // newest exports kept per instance
|
||||
InstanceOnly bool `json:"instance_only"` // leave the snapshots out of the tarball
|
||||
Instances []string `json:"instances,omitempty"` // empty: every instance
|
||||
Updated time.Time `json:"updated,omitempty"`
|
||||
}
|
||||
|
||||
func DefaultExportConfig() ExportConfig {
|
||||
return ExportConfig{Hour: 2, Minute: 15, Keep: 3, InstanceOnly: true}
|
||||
}
|
||||
|
||||
func (c ExportConfig) Validate() error {
|
||||
if c.Hour < 0 || c.Hour > 23 || c.Minute < 0 || c.Minute > 59 {
|
||||
return errors.New("time: hour 0–23, minute 0–59")
|
||||
}
|
||||
if c.Keep < 1 || c.Keep > 100 {
|
||||
return errors.New("keep: between 1 and 100")
|
||||
}
|
||||
for _, n := range c.Instances {
|
||||
if !ValidName(n) {
|
||||
return errors.New("instance: " + n)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c ExportConfig) CronExpr() string { return fmt.Sprintf("%d %d * * *", c.Minute, c.Hour) }
|
||||
|
||||
// NextRun is the next time the line fires, for the page.
|
||||
func (c ExportConfig) NextRun(now time.Time) time.Time {
|
||||
n := time.Date(now.Year(), now.Month(), now.Day(), c.Hour, c.Minute, 0, 0, now.Location())
|
||||
if !n.After(now) {
|
||||
n = n.Add(24 * time.Hour)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// CronText is the whole file; rewritten on every change, never edited.
|
||||
func (c ExportConfig) CronText(user, exe, config string) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("# Ham Lab instance exports — written by the Virtualization tab, rewritten whole on every change.\n")
|
||||
b.WriteString("# Edit it there, not here.\n")
|
||||
b.WriteString("SHELL=/bin/sh\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
|
||||
fmt.Fprintf(&b, "# every instance to virt/exports, keep %d each\n", c.Keep)
|
||||
fmt.Fprintf(&b, "%s %s %s -config %s virt-export\n", c.CronExpr(), user, exe, config)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func (m *Manager) exportConfigPath() string { return filepath.Join(m.dir, "exports.json") }
|
||||
func (m *Manager) exportRunPath() string { return filepath.Join(m.dir, "export-runs.json") }
|
||||
|
||||
func (m *Manager) ExportConfig() ExportConfig {
|
||||
c := DefaultExportConfig()
|
||||
if b, err := os.ReadFile(m.exportConfigPath()); err == nil {
|
||||
_ = json.Unmarshal(b, &c)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func (m *Manager) SaveExportConfig(c ExportConfig) error {
|
||||
if err := c.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := m.EnsureDirs(); err != nil {
|
||||
return err
|
||||
}
|
||||
c.Updated = time.Now()
|
||||
b, _ := json.MarshalIndent(c, "", " ")
|
||||
return os.WriteFile(m.exportConfigPath(), b, 0o644)
|
||||
}
|
||||
|
||||
type ExportRun struct {
|
||||
At time.Time `json:"at"`
|
||||
Exported []string `json:"exported"`
|
||||
Pruned []string `json:"pruned"`
|
||||
Errors []string `json:"errors,omitempty"`
|
||||
Seconds float64 `json:"seconds"`
|
||||
}
|
||||
|
||||
func (m *Manager) LastExportRun() *ExportRun {
|
||||
b, err := os.ReadFile(m.exportRunPath())
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var r ExportRun
|
||||
if json.Unmarshal(b, &r) != nil {
|
||||
return nil
|
||||
}
|
||||
return &r
|
||||
}
|
||||
|
||||
// ExportFile is one tarball in the exports tree.
|
||||
type ExportFile struct {
|
||||
Instance string `json:"instance"`
|
||||
Name string `json:"name"`
|
||||
Size int64 `json:"size"`
|
||||
Modified time.Time `json:"modified"`
|
||||
}
|
||||
|
||||
func (m *Manager) ExportFiles() []ExportFile {
|
||||
out := []ExportFile{}
|
||||
dirs, _ := os.ReadDir(m.ExportsDir())
|
||||
for _, d := range dirs {
|
||||
if !d.IsDir() {
|
||||
continue
|
||||
}
|
||||
files, _ := os.ReadDir(filepath.Join(m.ExportsDir(), d.Name()))
|
||||
for _, f := range files {
|
||||
if f.IsDir() || strings.HasSuffix(f.Name(), ".part") {
|
||||
continue
|
||||
}
|
||||
if info, err := f.Info(); err == nil {
|
||||
out = append(out, ExportFile{Instance: d.Name(), Name: f.Name(), Size: info.Size(), Modified: info.ModTime()})
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].Instance != out[j].Instance {
|
||||
return out[i].Instance < out[j].Instance
|
||||
}
|
||||
return out[i].Modified.After(out[j].Modified)
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// RunExports exports the named instances (or every one when names is empty) and prunes each
|
||||
// instance's folder to the configured count. Output goes to w line by line for a task to show.
|
||||
func (m *Manager) RunExports(ctx context.Context, names []string, w func(string)) ExportRun {
|
||||
start := time.Now()
|
||||
c := m.ExportConfig()
|
||||
run := ExportRun{At: start, Exported: []string{}, Pruned: []string{}}
|
||||
if err := m.EnsureDirs(); err != nil {
|
||||
run.Errors = append(run.Errors, err.Error())
|
||||
return run
|
||||
}
|
||||
all, err := m.Instances(ctx)
|
||||
if err != nil {
|
||||
run.Errors = append(run.Errors, "listing instances: "+err.Error())
|
||||
return run
|
||||
}
|
||||
want := map[string]bool{}
|
||||
for _, n := range names {
|
||||
want[n] = true
|
||||
}
|
||||
if len(names) == 0 && len(c.Instances) > 0 {
|
||||
for _, n := range c.Instances {
|
||||
want[n] = true
|
||||
}
|
||||
}
|
||||
for _, in := range all {
|
||||
if len(want) > 0 && !want[in.Name] {
|
||||
continue
|
||||
}
|
||||
dir := filepath.Join(m.ExportsDir(), in.Name)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
run.Errors = append(run.Errors, in.Name+": "+err.Error())
|
||||
continue
|
||||
}
|
||||
stamp := start.Format("2006-01-02_15.04")
|
||||
final := filepath.Join(dir, fmt.Sprintf("%s-%s.tar.gz", in.Name, stamp))
|
||||
part := final + ".part"
|
||||
args := []string{"export", in.Name, part}
|
||||
if c.InstanceOnly {
|
||||
args = append(args, "--instance-only")
|
||||
}
|
||||
w(fmt.Sprintf("exporting %s …", in.Name))
|
||||
cctx, cancel := context.WithTimeout(ctx, 2*time.Hour)
|
||||
out, err := exec.CommandContext(cctx, "incus", args...).CombinedOutput()
|
||||
cancel()
|
||||
if err != nil {
|
||||
_ = os.Remove(part)
|
||||
msg := strings.TrimSpace(string(out))
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
run.Errors = append(run.Errors, in.Name+": "+msg)
|
||||
w(" failed: " + msg)
|
||||
continue
|
||||
}
|
||||
if err := os.Rename(part, final); err != nil {
|
||||
run.Errors = append(run.Errors, in.Name+": "+err.Error())
|
||||
continue
|
||||
}
|
||||
if st, err := os.Stat(final); err == nil {
|
||||
w(fmt.Sprintf(" %s (%.1f MiB)", filepath.Base(final), float64(st.Size())/(1<<20)))
|
||||
}
|
||||
run.Exported = append(run.Exported, filepath.Base(final))
|
||||
// prune: newest first, keep c.Keep
|
||||
var files []ExportFile
|
||||
for _, f := range m.ExportFiles() {
|
||||
if f.Instance == in.Name {
|
||||
files = append(files, f)
|
||||
}
|
||||
}
|
||||
for i := c.Keep; i < len(files); i++ {
|
||||
if err := os.Remove(filepath.Join(dir, files[i].Name)); err == nil {
|
||||
run.Pruned = append(run.Pruned, files[i].Name)
|
||||
w(" pruned " + files[i].Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
run.Seconds = time.Since(start).Seconds()
|
||||
b, _ := json.MarshalIndent(run, "", " ")
|
||||
_ = os.WriteFile(m.exportRunPath(), b, 0o644)
|
||||
return run
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,90 @@
|
||||
package incus
|
||||
|
||||
import (
|
||||
"os/exec"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The install script is generated, so a quoting slip would only show on a live machine.
|
||||
// Build it for the shapes of machine we have and let sh check the syntax.
|
||||
func TestInstallScriptShapes(t *testing.T) {
|
||||
if _, err := exec.LookPath("sh"); err != nil {
|
||||
t.Skip("no sh")
|
||||
}
|
||||
m := New("/home/core/apps/hamlab", "core", "test")
|
||||
cases := []struct {
|
||||
name string
|
||||
st Status
|
||||
o InstallOpts
|
||||
want []string
|
||||
}{
|
||||
{"grid: zfs, apt, docker", Status{State: "installable", PM: "apt", Arch: "x86_64", Drivers: []string{"zfs", "btrfs", "dir"}, Docker: true, ZFSHost: true, DiskFree: 700 << 30, MemTotal: 62 << 30},
|
||||
InstallOpts{Driver: "zfs", PoolSizeGiB: 300, Subnet: "172.16.100.1/24", DockerRules: true, ARCMaxGiB: 16},
|
||||
[]string{"apt-get", "zpool create", "incus storage create default zfs source=hamlab-virt", "ipv4.address='172.16.100.1/24'", "DOCKER-USER", "zfs_arc_max", "usermod -aG incus-admin 'core'", "SupplementaryGroups=incus-admin", "systemd-run"}},
|
||||
{"core: btrfs, apt, docker, no tools yet", Status{State: "installable", PM: "apt", Arch: "x86_64", Drivers: []string{"btrfs", "dir"}, Docker: true, DiskFree: 170 << 30, MemTotal: 8 << 30},
|
||||
InstallOpts{Driver: "btrfs", PoolSizeGiB: 40, Subnet: "172.16.100.1/24", DockerRules: true},
|
||||
[]string{"btrfs-progs", "mkfs.btrfs", "/etc/fstab", "incus storage create default btrfs source="}},
|
||||
{"mac: dir, dnf, arm64", Status{State: "installable", PM: "dnf", Arch: "aarch64", Drivers: []string{"dir"}, DiskFree: 200 << 30, MemTotal: 15 << 30},
|
||||
InstallOpts{Driver: "dir", Subnet: "172.16.101.1/24"},
|
||||
[]string{"dnf -y install incus qemu-system-aarch64-core edk2-aarch64", "incus storage create default dir source="}},
|
||||
{"repair: already installed", Status{State: "repair", PM: "apt", Arch: "x86_64", Drivers: []string{"dir"}, DiskFree: 10 << 30, MemTotal: 4 << 30},
|
||||
InstallOpts{Driver: "dir", Subnet: "172.16.100.1/24"},
|
||||
[]string{"already installed"}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
script, lay, err := m.InstallScript(c.st, c.o)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", c.name, err)
|
||||
}
|
||||
for _, w := range c.want {
|
||||
if !strings.Contains(script, w) {
|
||||
t.Errorf("%s: script lacks %q", c.name, w)
|
||||
}
|
||||
}
|
||||
if lay.Driver != c.o.Driver || lay.Subnet != c.o.Subnet {
|
||||
t.Errorf("%s: layout %+v", c.name, lay)
|
||||
}
|
||||
cmd := exec.Command("sh", "-n")
|
||||
cmd.Stdin = strings.NewReader(script)
|
||||
if out, err := cmd.CombinedOutput(); err != nil {
|
||||
t.Errorf("%s: sh -n: %v\n%s", c.name, err, out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstallScriptRefusals(t *testing.T) {
|
||||
m := New("/x", "core", "test")
|
||||
st := Status{State: "installable", PM: "apt", Drivers: []string{"btrfs", "dir"}, DiskFree: 20 << 30, MemTotal: 8 << 30}
|
||||
for _, o := range []InstallOpts{
|
||||
{Driver: "zfs", PoolSizeGiB: 10, Subnet: "172.16.100.1/24"}, // not offered here
|
||||
{Driver: "btrfs", PoolSizeGiB: 50, Subnet: "172.16.100.1/24"}, // more than is free
|
||||
{Driver: "btrfs", PoolSizeGiB: 10, Subnet: "172.16.100.0/24"}, // the network address, not the bridge's
|
||||
{Driver: "btrfs", PoolSizeGiB: 10, Subnet: "10.0.0.1/8"}, // too wide
|
||||
{Driver: "btrfs", PoolSizeGiB: 10, Subnet: "172.16.100.1/24", ARCMaxGiB: 64}, // more RAM than there is
|
||||
} {
|
||||
if _, _, err := m.InstallScript(st, o); err == nil {
|
||||
t.Errorf("accepted %+v", o)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateArgs(t *testing.T) {
|
||||
args, err := CreateArgs(CreateSpec{Name: "web-1", Image: "images:debian/13", VM: true, CPU: "2", Memory: "2GiB", Disk: "20GiB", Network: "incusbr0", Start: true}, true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := "launch images:debian/13 web-1 --vm -c limits.cpu=2 -c limits.memory=2GiB -d root,size=20GiB -n incusbr0"
|
||||
if got := strings.Join(args, " "); got != want {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
if _, err := CreateArgs(CreateSpec{Name: "web-1", Image: "images:debian/13", VM: true}, false); err == nil {
|
||||
t.Fatal("a VM without KVM must be refused")
|
||||
}
|
||||
if _, err := CreateArgs(CreateSpec{Name: "1bad", Image: "images:debian/13"}, true); err == nil {
|
||||
t.Fatal("a name starting with a digit must be refused")
|
||||
}
|
||||
if _, err := CreateArgs(CreateSpec{Name: "ok", Image: "images:debian/13", Memory: "2 gigs"}, true); err == nil {
|
||||
t.Fatal("a malformed size must be refused")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
package incus
|
||||
|
||||
// The bridge subnet. Incus's own `auto` picks a random 10.x.y.0/24, and on a LAN that is
|
||||
// itself 10.0.0.0/8 (this one) every such pick is on-link and collides. So Ham Lab chooses:
|
||||
// the first /24 that nothing on the machine already routes, is not an address it holds, and
|
||||
// is outside Docker's configured pools — tried in an order that also stays clear of Docker's
|
||||
// *defaults* (172.17–172.31 and 192.168.0.0/16 carved into /20s), which a Docker that has no
|
||||
// daemon.json will hand out later without asking anyone.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// candidates in order of preference: 172.16.0.0/16 is the one RFC 1918 block Docker never
|
||||
// takes by default; then the top of 192.168/16, which Docker reaches last; then the rest.
|
||||
func candidates() []netip.Prefix {
|
||||
var out []netip.Prefix
|
||||
for i := 100; i <= 254; i++ {
|
||||
out = append(out, netip.MustParsePrefix(fmt.Sprintf("172.16.%d.0/24", i)))
|
||||
}
|
||||
for i := 254; i >= 200; i-- {
|
||||
out = append(out, netip.MustParsePrefix(fmt.Sprintf("192.168.%d.0/24", i)))
|
||||
}
|
||||
for i := 1; i < 100; i++ {
|
||||
out = append(out, netip.MustParsePrefix(fmt.Sprintf("172.16.%d.0/24", i)))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func overlaps(a, b netip.Prefix) bool {
|
||||
return a.Contains(b.Addr()) || b.Contains(a.Addr())
|
||||
}
|
||||
|
||||
// PickSubnet returns the first candidate /24 clear of every prefix in taken.
|
||||
func PickSubnet(taken []netip.Prefix) netip.Prefix {
|
||||
for _, c := range candidates() {
|
||||
clear := true
|
||||
for _, t := range taken {
|
||||
if overlaps(c, t) {
|
||||
clear = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if clear {
|
||||
return c
|
||||
}
|
||||
}
|
||||
return netip.MustParsePrefix("172.16.100.0/24") // every candidate taken: not a machine this will run on
|
||||
}
|
||||
|
||||
// Gateway is the bridge's own address in a subnet: its first host, as CIDR, the way
|
||||
// Incus's ipv4.address wants it.
|
||||
func Gateway(p netip.Prefix) string {
|
||||
return fmt.Sprintf("%s/%d", p.Masked().Addr().Next(), p.Bits())
|
||||
}
|
||||
|
||||
// TakenPrefixes reads what the machine already uses: its routes, its addresses, and the
|
||||
// pools Docker is configured to carve networks from.
|
||||
func TakenPrefixes(ctx context.Context) []netip.Prefix {
|
||||
var out []netip.Prefix
|
||||
add := func(s string) {
|
||||
if p, err := netip.ParsePrefix(s); err == nil && p.Addr().Is4() {
|
||||
out = append(out, p.Masked())
|
||||
} else if a, err := netip.ParseAddr(s); err == nil && a.Is4() {
|
||||
out = append(out, netip.PrefixFrom(a, 32))
|
||||
}
|
||||
}
|
||||
c, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
if b, err := exec.CommandContext(c, "ip", "-j", "-4", "route", "show").Output(); err == nil {
|
||||
var routes []struct{ Dst string }
|
||||
if json.Unmarshal(b, &routes) == nil {
|
||||
for _, r := range routes {
|
||||
if r.Dst != "" && r.Dst != "default" {
|
||||
add(r.Dst)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if b, err := exec.CommandContext(c, "ip", "-j", "-4", "addr", "show").Output(); err == nil {
|
||||
var links []struct {
|
||||
AddrInfo []struct {
|
||||
Local string `json:"local"`
|
||||
Prefixlen int `json:"prefixlen"`
|
||||
} `json:"addr_info"`
|
||||
}
|
||||
if json.Unmarshal(b, &links) == nil {
|
||||
for _, l := range links {
|
||||
for _, a := range l.AddrInfo {
|
||||
add(fmt.Sprintf("%s/%d", a.Local, a.Prefixlen))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, p := range dockerPools() {
|
||||
add(p)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// dockerPools reads default-address-pools from Docker's daemon.json, and falls back to
|
||||
// Docker's built-in defaults when the file names none.
|
||||
func dockerPools() []string {
|
||||
var cfg struct {
|
||||
Pools []struct {
|
||||
Base string `json:"base"`
|
||||
} `json:"default-address-pools"`
|
||||
}
|
||||
if b, err := os.ReadFile("/etc/docker/daemon.json"); err == nil && json.Unmarshal(b, &cfg) == nil && len(cfg.Pools) > 0 {
|
||||
var out []string
|
||||
for _, p := range cfg.Pools {
|
||||
out = append(out, p.Base)
|
||||
}
|
||||
return out
|
||||
}
|
||||
if _, err := exec.LookPath("docker"); err != nil {
|
||||
return nil
|
||||
}
|
||||
return []string{"172.17.0.0/12", "192.168.0.0/16"}
|
||||
}
|
||||
|
||||
// String form used in the layout file and on the page.
|
||||
func prefixString(p netip.Prefix) string { return strings.TrimSpace(p.Masked().String()) }
|
||||
@@ -0,0 +1,41 @@
|
||||
package incus
|
||||
|
||||
import (
|
||||
"net/netip"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPickSubnetAvoidsTaken(t *testing.T) {
|
||||
taken := []netip.Prefix{
|
||||
netip.MustParsePrefix("10.0.0.0/8"), // the LAN, a /8
|
||||
netip.MustParsePrefix("172.16.100.0/24"), // first choice gone
|
||||
netip.MustParsePrefix("172.16.101.0/24"), // and the second
|
||||
netip.MustParsePrefix("172.28.0.0/14"), // Docker's configured pool
|
||||
netip.MustParsePrefix("192.168.80.0/20"), // a Docker network
|
||||
}
|
||||
got := PickSubnet(taken)
|
||||
if got.String() != "172.16.102.0/24" {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
if Gateway(got) != "172.16.102.1/24" {
|
||||
t.Fatalf("gateway %s", Gateway(got))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPickSubnetWholeBlockTaken(t *testing.T) {
|
||||
// 172.16.0.0/12 covers every 172.16.x candidate; the picker must move to 192.168
|
||||
got := PickSubnet([]netip.Prefix{netip.MustParsePrefix("172.16.0.0/12")})
|
||||
if got.String() != "192.168.254.0/24" {
|
||||
t.Fatalf("got %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOverlapsEitherWay(t *testing.T) {
|
||||
a := netip.MustParsePrefix("172.16.5.0/24")
|
||||
if !overlaps(a, netip.MustParsePrefix("172.16.0.0/12")) || !overlaps(netip.MustParsePrefix("172.16.0.0/12"), a) {
|
||||
t.Fatal("a /24 inside a /12 overlaps both ways")
|
||||
}
|
||||
if overlaps(a, netip.MustParsePrefix("172.16.6.0/24")) {
|
||||
t.Fatal("neighbours do not overlap")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
package incus
|
||||
|
||||
// The graphical console. Incus exposes a VM's SPICE display as a websocket: POST to the
|
||||
// instance's console endpoint with type vga and it answers an operation with two secrets,
|
||||
// one for the display stream and one for a control channel. Ham Lab dials both over Incus's
|
||||
// unix socket and hands the display stream to the browser, where spice-html5 draws it.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/coder/websocket"
|
||||
)
|
||||
|
||||
// SocketPath is where Incus's API listens; the daemon's account reaches it by being in
|
||||
// incus-admin, which Install arranges.
|
||||
func SocketPath() string {
|
||||
if p := os.Getenv("INCUS_SOCKET"); p != "" {
|
||||
return p
|
||||
}
|
||||
for _, p := range []string{"/var/lib/incus/unix.socket", "/run/incus/unix.socket"} {
|
||||
if _, err := os.Stat(p); err == nil {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return "/var/lib/incus/unix.socket"
|
||||
}
|
||||
|
||||
func (m *Manager) apiClient() *http.Client {
|
||||
sock := SocketPath()
|
||||
return &http.Client{Transport: &http.Transport{
|
||||
DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
|
||||
var d net.Dialer
|
||||
return d.DialContext(ctx, "unix", sock)
|
||||
},
|
||||
ForceAttemptHTTP2: false,
|
||||
}}
|
||||
}
|
||||
|
||||
// A SPICE client opens one connection per channel — main, display, inputs, cursor, playback
|
||||
// — and spice-html5 does the same with websockets, all to one URL. Incus allows exactly one
|
||||
// console operation per VM at a time, but takes any number of data websockets on it. So a VGA
|
||||
// session is one operation, held for as long as any channel is open, and every browser
|
||||
// websocket becomes one more data websocket on it.
|
||||
type vgaSession struct {
|
||||
op string
|
||||
secret string
|
||||
control *websocket.Conn
|
||||
cancel context.CancelFunc
|
||||
refs int
|
||||
}
|
||||
|
||||
// VGAChannel returns a fresh data websocket on the VM's console operation, opening the
|
||||
// operation first when none is held. release must be called when the channel is done.
|
||||
func (m *Manager) VGAChannel(ctx context.Context, name string) (data *websocket.Conn, release func(), err error) {
|
||||
if !ValidName(name) {
|
||||
return nil, nil, errors.New("bad name")
|
||||
}
|
||||
client := m.apiClient()
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if m.vga == nil {
|
||||
m.vga = map[string]*vgaSession{}
|
||||
}
|
||||
dial := func(s *vgaSession) (*websocket.Conn, error) {
|
||||
c, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
u := fmt.Sprintf("ws://incus/1.0/operations/%s/websocket?secret=%s", s.op, s.secret)
|
||||
ws, _, err := websocket.Dial(c, u, &websocket.DialOptions{HTTPClient: client, CompressionMode: websocket.CompressionDisabled})
|
||||
return ws, err
|
||||
}
|
||||
s := m.vga[name]
|
||||
if s != nil {
|
||||
if ws, err := dial(s); err == nil {
|
||||
s.refs++
|
||||
ws.SetReadLimit(64 << 20)
|
||||
return ws, m.releaser(name, s), nil
|
||||
}
|
||||
// the operation is gone (the VM restarted, or Incus did): start over
|
||||
s.cancel()
|
||||
delete(m.vga, name)
|
||||
}
|
||||
s, err = m.openVGA(ctx, client, name, false)
|
||||
if err != nil && strings.Contains(err.Error(), "already connected") {
|
||||
// a session a previous Ham Lab process held and never let go of; take it over
|
||||
s, err = m.openVGA(ctx, client, name, true)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
ws, err := dial(s)
|
||||
if err != nil {
|
||||
s.cancel()
|
||||
return nil, nil, err
|
||||
}
|
||||
ws.SetReadLimit(64 << 20)
|
||||
s.refs = 1
|
||||
m.vga[name] = s
|
||||
return ws, m.releaser(name, s), nil
|
||||
}
|
||||
|
||||
func (m *Manager) releaser(name string, s *vgaSession) func() {
|
||||
return func() {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
s.refs--
|
||||
if s.refs <= 0 && m.vga[name] == s {
|
||||
s.cancel()
|
||||
delete(m.vga, name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// openVGA asks Incus for the console operation and holds its control websocket, which is
|
||||
// what keeps the operation alive.
|
||||
func (m *Manager) openVGA(ctx context.Context, client *http.Client, name string, force bool) (*vgaSession, error) {
|
||||
body, _ := json.Marshal(map[string]any{"type": "vga", "width": 0, "height": 0, "force": force})
|
||||
c, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
req, _ := http.NewRequestWithContext(c, "POST", "http://incus/1.0/instances/"+name+"/console", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
if isPermission(err) {
|
||||
return nil, errors.New("Ham Lab's account can't reach Incus's socket; Finish setup on the Virtualization tab puts it in incus-admin")
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
var op struct {
|
||||
Error string `json:"error"`
|
||||
Metadata struct {
|
||||
ID string `json:"id"`
|
||||
Metadata struct {
|
||||
FDs map[string]string `json:"fds"`
|
||||
} `json:"metadata"`
|
||||
} `json:"metadata"`
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&op); err != nil {
|
||||
return nil, errors.New("unreadable answer from incus")
|
||||
}
|
||||
if op.Error != "" {
|
||||
return nil, errors.New(op.Error)
|
||||
}
|
||||
secret, controlSecret := op.Metadata.Metadata.FDs["0"], op.Metadata.Metadata.FDs["control"]
|
||||
if op.Metadata.ID == "" || secret == "" {
|
||||
return nil, errors.New("incus gave no console; is the VM running?")
|
||||
}
|
||||
s := &vgaSession{op: op.Metadata.ID, secret: secret}
|
||||
// the control websocket lives outside any one request: it is the session
|
||||
sctx, scancel := context.WithCancel(context.Background())
|
||||
s.cancel = scancel
|
||||
if controlSecret != "" {
|
||||
u := fmt.Sprintf("ws://incus/1.0/operations/%s/websocket?secret=%s", s.op, controlSecret)
|
||||
control, _, err := websocket.Dial(sctx, u, &websocket.DialOptions{HTTPClient: client, CompressionMode: websocket.CompressionDisabled})
|
||||
if err != nil {
|
||||
scancel()
|
||||
return nil, err
|
||||
}
|
||||
s.control = control
|
||||
go func() { // drain it; when it ends, so has the operation
|
||||
for {
|
||||
if _, _, err := control.Read(sctx); err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
control.Close(websocket.StatusNormalClosure, "")
|
||||
}()
|
||||
go func() { <-sctx.Done(); control.Close(websocket.StatusNormalClosure, "") }()
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
func isPermission(err error) bool {
|
||||
var pe *os.PathError
|
||||
if errors.As(err, &pe) {
|
||||
return os.IsPermission(pe)
|
||||
}
|
||||
var oe *net.OpError
|
||||
if errors.As(err, &oe) {
|
||||
return os.IsPermission(oe.Err) || isPermission(oe.Err)
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package metrics
|
||||
|
||||
// The ZFS ARC — the read cache in RAM — read live from /proc/spl/kstat/zfs/arcstats on a
|
||||
// machine that has the module loaded, and left out of the sample everywhere else. The kstat
|
||||
// file is counters since boot; the hit rate here is over the sampling interval, which is the
|
||||
// number that says whether the cache is doing its job *now*. The lifetime figures and the
|
||||
// pool's caching tiers (L2ARC, SLOG, special vdevs) are the storage package's business.
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const arcstatsPath = "/proc/spl/kstat/zfs/arcstats"
|
||||
|
||||
// ARC is the read cache as one sample sees it. Sizes are bytes.
|
||||
type ARC struct {
|
||||
Size uint64 `json:"size"`
|
||||
Target uint64 `json:"target"` // c: what the ARC is currently allowed to hold
|
||||
Min uint64 `json:"min"` // c_min: the floor it will not shrink below
|
||||
Max uint64 `json:"max"` // c_max: the ceiling (zfs_arc_max, or half of RAM)
|
||||
// what is in it
|
||||
MRU uint64 `json:"mru"` // recently used once
|
||||
MFU uint64 `json:"mfu"` // used more than once
|
||||
Anon uint64 `json:"anon"` // in flight: dirty, not yet on disk
|
||||
Data uint64 `json:"data"` // file contents
|
||||
Metadata uint64 `json:"metadata"` // dnodes, indirect blocks, dbufs, bonus buffers
|
||||
MetaUsed uint64 `json:"meta_used"`
|
||||
Headers uint64 `json:"headers"`
|
||||
Compressed uint64 `json:"compressed"`
|
||||
Uncompressed uint64 `json:"uncompressed"`
|
||||
MRUGhost uint64 `json:"mru_ghost"` // what it recently evicted and still remembers
|
||||
MFUGhost uint64 `json:"mfu_ghost"`
|
||||
// L2ARC, the cache devices (0 when there are none)
|
||||
L2Size uint64 `json:"l2_size"`
|
||||
L2ASize uint64 `json:"l2_asize"` // as stored on the device, after compression
|
||||
// over this interval
|
||||
ReadsPerSec float64 `json:"reads_per_sec"`
|
||||
MissesPerSec float64 `json:"misses_per_sec"`
|
||||
HitPct float64 `json:"hit_pct"` // of the interval's reads; an idle interval keeps the last one
|
||||
HitPctKnown bool `json:"hit_pct_known"` // false until an interval has had any reads
|
||||
L2HitPct float64 `json:"l2_hit_pct"`
|
||||
// since boot
|
||||
Hits uint64 `json:"hits"`
|
||||
Misses uint64 `json:"misses"`
|
||||
LifetimeHitPct float64 `json:"lifetime_hit_pct"`
|
||||
// pressure
|
||||
NoGrow bool `json:"no_grow"` // arc_no_grow: memory pressure is stopping it growing
|
||||
Throttles uint64 `json:"throttles"` // memory_throttle_count since boot
|
||||
Prunes uint64 `json:"prunes"` // arc_prune: metadata evictions asked of the filesystems
|
||||
EvictSkip uint64 `json:"evict_skip"`
|
||||
}
|
||||
|
||||
type arcPrev struct {
|
||||
hits, misses, l2hits, l2misses uint64
|
||||
hitPct float64
|
||||
known bool
|
||||
at time.Time
|
||||
}
|
||||
|
||||
// readKstat parses a kstat table ("name type data" rows after two header lines).
|
||||
func readKstat(p string) map[string]uint64 {
|
||||
s := readStr(p)
|
||||
if s == "" {
|
||||
return nil
|
||||
}
|
||||
out := map[string]uint64{}
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) != 3 || f[1] != "4" { // type 4 is KSTAT_DATA_UINT64
|
||||
continue
|
||||
}
|
||||
v, err := strconv.ParseUint(f[2], 10, 64)
|
||||
if err == nil {
|
||||
out[f[0]] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// arc reads the ARC; nil when this machine has no ZFS module loaded.
|
||||
func (c *Collector) arc(now time.Time) *ARC {
|
||||
k := readKstat(arcstatsPath)
|
||||
if k == nil {
|
||||
return nil
|
||||
}
|
||||
a := &ARC{Size: k["size"], Target: k["c"], Min: k["c_min"], Max: k["c_max"],
|
||||
MRU: k["mru_size"], MFU: k["mfu_size"], Anon: k["anon_size"], Data: k["data_size"], Metadata: k["metadata_size"],
|
||||
MetaUsed: k["arc_meta_used"], Headers: k["hdr_size"], Compressed: k["compressed_size"], Uncompressed: k["uncompressed_size"],
|
||||
MRUGhost: k["mru_ghost_size"], MFUGhost: k["mfu_ghost_size"], L2Size: k["l2_size"], L2ASize: k["l2_asize"],
|
||||
Hits: k["hits"], Misses: k["misses"], NoGrow: k["arc_no_grow"] != 0, Throttles: k["memory_throttle_count"], Prunes: k["arc_prune"], EvictSkip: k["evict_skip"]}
|
||||
if n := a.Hits + a.Misses; n > 0 {
|
||||
a.LifetimeHitPct = round1(float64(a.Hits) / float64(n) * 100)
|
||||
}
|
||||
p := c.prevARC
|
||||
if !p.at.IsZero() && a.Hits >= p.hits && a.Misses >= p.misses {
|
||||
if dt := now.Sub(p.at).Seconds(); dt > 0 {
|
||||
hits, misses := a.Hits-p.hits, a.Misses-p.misses
|
||||
a.ReadsPerSec = round1(float64(hits+misses) / dt)
|
||||
a.MissesPerSec = round1(float64(misses) / dt)
|
||||
if hits+misses > 0 {
|
||||
p.hitPct, p.known = round1(float64(hits)/float64(hits+misses)*100), true
|
||||
}
|
||||
if l2 := (k["l2_hits"] - p.l2hits) + (k["l2_misses"] - p.l2misses); l2 > 0 && k["l2_hits"] >= p.l2hits {
|
||||
a.L2HitPct = round1(float64(k["l2_hits"]-p.l2hits) / float64(l2) * 100)
|
||||
}
|
||||
}
|
||||
}
|
||||
a.HitPct, a.HitPctKnown = p.hitPct, p.known
|
||||
c.prevARC = arcPrev{hits: a.Hits, misses: a.Misses, l2hits: k["l2_hits"], l2misses: k["l2_misses"], hitPct: p.hitPct, known: p.known, at: now}
|
||||
return a
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
package metrics
|
||||
|
||||
// One kept history point is the average of the samples behind it. At the default ten-minute
|
||||
// window that bucket is a single sample and nothing is averaged; at a day it is five minutes
|
||||
// of them, and the difference matters — a graph of the last day drawn from whichever instant
|
||||
// landed on each five-minute mark would show spikes that never happened and miss the ones
|
||||
// that did.
|
||||
//
|
||||
// The newest sample of the bucket is the template: its shape, labels, totals and slow
|
||||
// sections (filesystems, UPS) are the current ones. Only the moving numbers are averaged,
|
||||
// and only where an entry is present in that sample — a disk plugged in halfway through the
|
||||
// bucket is averaged over the samples that have it.
|
||||
|
||||
func fold(b []Sample) Sample {
|
||||
out := b[len(b)-1]
|
||||
if len(b) == 1 {
|
||||
return out
|
||||
}
|
||||
n := float64(len(b))
|
||||
sum := func(get func(Sample) float64) float64 {
|
||||
var t float64
|
||||
for _, s := range b {
|
||||
t += get(s)
|
||||
}
|
||||
return t / n
|
||||
}
|
||||
sumU := func(get func(Sample) uint64) uint64 {
|
||||
var t uint64
|
||||
for _, s := range b {
|
||||
t += get(s)
|
||||
}
|
||||
return t / uint64(len(b))
|
||||
}
|
||||
out.CPU = sum(func(s Sample) float64 { return s.CPU })
|
||||
out.Load[0] = sum(func(s Sample) float64 { return s.Load[0] })
|
||||
out.Load[1] = sum(func(s Sample) float64 { return s.Load[1] })
|
||||
out.Load[2] = sum(func(s Sample) float64 { return s.Load[2] })
|
||||
|
||||
out.Cores = append([]float64(nil), out.Cores...)
|
||||
for i := range out.Cores {
|
||||
var t, c float64
|
||||
for _, s := range b {
|
||||
if i < len(s.Cores) {
|
||||
t, c = t+s.Cores[i], c+1
|
||||
}
|
||||
}
|
||||
if c > 0 {
|
||||
out.Cores[i] = t / c
|
||||
}
|
||||
}
|
||||
|
||||
out.Mem.Used = sumU(func(s Sample) uint64 { return s.Mem.Used })
|
||||
out.Mem.Avail = sumU(func(s Sample) uint64 { return s.Mem.Avail })
|
||||
out.Mem.Free = sumU(func(s Sample) uint64 { return s.Mem.Free })
|
||||
out.Mem.Cached = sumU(func(s Sample) uint64 { return s.Mem.Cached })
|
||||
out.Swap.Used = sumU(func(s Sample) uint64 { return s.Swap.Used })
|
||||
|
||||
// the rest are lists whose entries are matched by name across the bucket
|
||||
temps := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Temps))
|
||||
for _, x := range s.Temps {
|
||||
m[x.Chip+"/"+x.Label] = []float64{x.C}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Temps = append([]Temp(nil), out.Temps...)
|
||||
for i, x := range out.Temps {
|
||||
if v, ok := temps[x.Chip+"/"+x.Label]; ok {
|
||||
out.Temps[i].C = v[0]
|
||||
}
|
||||
}
|
||||
|
||||
fans := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Fans))
|
||||
for _, x := range s.Fans {
|
||||
m[x.Chip+"/"+x.Label] = []float64{float64(x.RPM)}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Fans = append([]Fan(nil), out.Fans...)
|
||||
for i, x := range out.Fans {
|
||||
if v, ok := fans[x.Chip+"/"+x.Label]; ok {
|
||||
out.Fans[i].RPM = int(v[0] + .5)
|
||||
}
|
||||
}
|
||||
|
||||
nets := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Net))
|
||||
for _, x := range s.Net {
|
||||
m[x.Iface] = []float64{x.RxBps, x.TxBps}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Net = append([]Net(nil), out.Net...)
|
||||
for i, x := range out.Net {
|
||||
if v, ok := nets[x.Iface]; ok {
|
||||
out.Net[i].RxBps, out.Net[i].TxBps = v[0], v[1]
|
||||
}
|
||||
}
|
||||
|
||||
disks := avgBy(b, func(s Sample) map[string][]float64 {
|
||||
m := make(map[string][]float64, len(s.Disks))
|
||||
for _, x := range s.Disks {
|
||||
m[x.Name] = []float64{x.ReadBps, x.WriteBps, x.UtilPct}
|
||||
}
|
||||
return m
|
||||
})
|
||||
out.Disks = append([]Disk(nil), out.Disks...)
|
||||
for i, x := range out.Disks {
|
||||
if v, ok := disks[x.Name]; ok {
|
||||
out.Disks[i].ReadBps, out.Disks[i].WriteBps, out.Disks[i].UtilPct = v[0], v[1], v[2]
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// avgBy averages each sample's named numbers across the bucket, per name.
|
||||
func avgBy(b []Sample, of func(Sample) map[string][]float64) map[string][]float64 {
|
||||
sums := map[string][]float64{}
|
||||
count := map[string]float64{}
|
||||
for _, s := range b {
|
||||
for k, vs := range of(s) {
|
||||
acc := sums[k]
|
||||
if acc == nil {
|
||||
acc = make([]float64, len(vs))
|
||||
}
|
||||
for i, v := range vs {
|
||||
if i < len(acc) {
|
||||
acc[i] += v
|
||||
}
|
||||
}
|
||||
sums[k], count[k] = acc, count[k]+1
|
||||
}
|
||||
}
|
||||
for k, acc := range sums {
|
||||
for i := range acc {
|
||||
acc[i] /= count[k]
|
||||
}
|
||||
}
|
||||
return sums
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestFoldAverages(t *testing.T) {
|
||||
b := []Sample{
|
||||
{T: 1, CPU: 10, Cores: []float64{10, 0}, Mem: Mem{Total: 100, Used: 10},
|
||||
Temps: []Temp{{Chip: "c", Label: "l", C: 40}}, Net: []Net{{Iface: "eth0", RxBps: 100}},
|
||||
Disks: []Disk{{Name: "sda", ReadBps: 10, UtilPct: 5}}},
|
||||
{T: 2, CPU: 30, Cores: []float64{30, 100}, Mem: Mem{Total: 100, Used: 30},
|
||||
Temps: []Temp{{Chip: "c", Label: "l", C: 60}}, Net: []Net{{Iface: "eth0", RxBps: 300}, {Iface: "new", RxBps: 7}},
|
||||
Disks: []Disk{{Name: "sda", ReadBps: 30, UtilPct: 15}}},
|
||||
}
|
||||
f := fold(b)
|
||||
if f.T != 2 {
|
||||
t.Errorf("a folded point is stamped with the newest sample in it, got %d", f.T)
|
||||
}
|
||||
if f.CPU != 20 || f.Cores[0] != 20 || f.Cores[1] != 50 {
|
||||
t.Errorf("cpu averaged: %v %v", f.CPU, f.Cores)
|
||||
}
|
||||
if f.Mem.Used != 20 || f.Mem.Total != 100 {
|
||||
t.Errorf("memory: used averaged, total taken from the newest, got %+v", f.Mem)
|
||||
}
|
||||
if f.Temps[0].C != 50 || f.Net[0].RxBps != 200 || f.Disks[0].ReadBps != 20 || f.Disks[0].UtilPct != 10 {
|
||||
t.Errorf("lists matched by name: %v %v %v", f.Temps, f.Net, f.Disks)
|
||||
}
|
||||
if len(f.Net) != 2 || f.Net[1].RxBps != 7 {
|
||||
t.Errorf("an interface that appeared inside the bucket is averaged over the samples that have it: %v", f.Net)
|
||||
}
|
||||
// the template's own slices must not be touched: Latest() shares them
|
||||
if b[1].Cores[0] != 30 || b[1].Temps[0].C != 60 {
|
||||
t.Error("fold wrote through to the sample it templated from")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStrideAndTrim(t *testing.T) {
|
||||
c := &Collector{interval: 2 * time.Second, window: DefaultWindow}
|
||||
if got := c.stride(); got != 2*time.Second {
|
||||
t.Errorf("a window of %v over %d points is finer than the interval, so the interval wins; got %v", DefaultWindow, History, got)
|
||||
}
|
||||
c.window = 24 * time.Hour
|
||||
if got := c.stride(); got != 24*time.Hour/History {
|
||||
t.Errorf("stride at a day: %v", got)
|
||||
}
|
||||
// a full day of samples at the day stride keeps exactly the budget
|
||||
c.interval, c.window = time.Second, time.Hour
|
||||
stride := c.stride().Milliseconds()
|
||||
for i := 0; i < 5*History; i++ {
|
||||
c.keep(Sample{T: int64(i) * stride})
|
||||
}
|
||||
if len(c.ring) != History {
|
||||
t.Errorf("the point count is the budget, got %d", len(c.ring))
|
||||
}
|
||||
if span := c.ring[len(c.ring)-1].T - c.ring[0].T; span > c.window.Milliseconds() {
|
||||
t.Errorf("the kept history reaches back further than the window: %d ms", span)
|
||||
}
|
||||
// narrowing the window drops what is now too old, at once
|
||||
c.SetWindow(MinWindow)
|
||||
if span := c.ring[len(c.ring)-1].T - c.ring[0].T; span > MinWindow.Milliseconds() {
|
||||
t.Errorf("narrowing the window left %d ms of history", span)
|
||||
}
|
||||
c.SetWindow(365 * 24 * time.Hour)
|
||||
if c.Window() != MaxWindow {
|
||||
t.Errorf("a window past the maximum is the maximum, got %v", c.Window())
|
||||
}
|
||||
}
|
||||
|
||||
// Points are one stride apart — not a stride and a sample — and jitter doesn't change that.
|
||||
func TestKeepSpacing(t *testing.T) {
|
||||
c := &Collector{interval: 2 * time.Second, window: time.Hour} // stride 12 s: six samples a point
|
||||
at := int64(1_000_000)
|
||||
for i := 0; i < 6*40; i++ {
|
||||
jitter := int64((i*37)%41 - 20) // ±20 ms, deterministic
|
||||
c.keep(Sample{T: at + jitter, CPU: float64(i % 6)})
|
||||
at += 2000
|
||||
}
|
||||
if len(c.ring) != 40 {
|
||||
t.Fatalf("240 samples at six a point are 40 points, got %d", len(c.ring))
|
||||
}
|
||||
for i := 1; i < len(c.ring); i++ {
|
||||
if gap := c.ring[i].T - c.ring[i-1].T; gap < 11900 || gap > 12100 {
|
||||
t.Fatalf("point %d is %d ms after the one before; a stride is 12000", i, gap)
|
||||
}
|
||||
}
|
||||
if c.ring[3].CPU != 2.5 {
|
||||
t.Errorf("each point is the mean of its own six samples (0…5 → 2.5), got %v", c.ring[3].CPU)
|
||||
}
|
||||
// and when the stride is the interval, every sample is a point
|
||||
c = &Collector{interval: 2 * time.Second, window: DefaultWindow}
|
||||
for i := 0; i < 50; i++ {
|
||||
c.keep(Sample{T: int64(i)*2000 + int64(i%3) - 1})
|
||||
}
|
||||
if len(c.ring) != 50 {
|
||||
t.Errorf("at the default window every sample is kept, got %d of 50", len(c.ring))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFoldNVMe(t *testing.T) {
|
||||
// an Apple SSD as the kernel lists it: one controller, three namespaces
|
||||
got := foldNVMe([]Disk{
|
||||
{Name: "nvme0n1", Model: "APPLE SSD AP0512Z", Size: 500 << 30, ReadBps: 100, WriteBps: 200, UtilPct: 12, Temp: 37},
|
||||
{Name: "nvme0n2", Model: "APPLE SSD AP0512Z", Size: 3 << 20, ReadBps: 10, WriteBps: 20, UtilPct: 40},
|
||||
{Name: "nvme0n3", Model: "APPLE SSD AP0512Z", Size: 128 << 20, ReadBps: 1, WriteBps: 2, UtilPct: 5},
|
||||
{Name: "sda", Size: 100 << 30, ReadBps: 7, UtilPct: 3},
|
||||
{Name: "nvme1n1", Size: 200 << 30, ReadBps: 5},
|
||||
})
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("three namespaces of one controller are one drive: %v", names(got))
|
||||
}
|
||||
d := got[0]
|
||||
if d.Name != "nvme0" {
|
||||
t.Errorf("the drive is the controller, got %q", d.Name)
|
||||
}
|
||||
if d.Size != 500<<30+3<<20+128<<20 {
|
||||
t.Errorf("its size is all of its namespaces, got %d", d.Size)
|
||||
}
|
||||
if d.ReadBps != 111 || d.WriteBps != 222 {
|
||||
t.Errorf("rates add up, got %v/%v", d.ReadBps, d.WriteBps)
|
||||
}
|
||||
if d.UtilPct != 40 {
|
||||
t.Errorf("busy is the longest of them, not the sum: got %v", d.UtilPct)
|
||||
}
|
||||
if d.Temp != 37 {
|
||||
t.Errorf("a temperature from any namespace is the drive's, got %v", d.Temp)
|
||||
}
|
||||
if got[1].Name != "sda" || got[1].UtilPct != 3 || got[2].Name != "nvme1" {
|
||||
t.Errorf("everything else is left alone: %v", names(got))
|
||||
}
|
||||
if n := len(foldNVMe(nil)); n != 0 {
|
||||
t.Errorf("no disks, no drives, got %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func names(d []Disk) []string {
|
||||
out := make([]string, len(d))
|
||||
for i, x := range d {
|
||||
out[i] = x.Name
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,680 @@
|
||||
package metrics
|
||||
|
||||
// GPU telemetry, the way btop and intel_gpu_top get it without a vendor library: the
|
||||
// device from sysfs, and who is using it from DRM fdinfo.
|
||||
//
|
||||
// - The device: /sys/class/drm/card<n> with a driver we know to be a GPU. Clock from the
|
||||
// driver's frequency files, temperature/power/fan from the hwmon under the device, and
|
||||
// how busy it is from the one counter every Intel GT exports without root — RC6 idle
|
||||
// residency, so busy = 100 − idle. amdgpu says it outright (gpu_busy_percent).
|
||||
// - The clients: every process holding /dev/dri/* open has a drm-engine-<name> running
|
||||
// total (nanoseconds) in /proc/<pid>/fdinfo/<fd>; the delta between two reads is that
|
||||
// process's share of each engine, and the sum across clients is the engine's load. Other
|
||||
// users' fdinfo is unreadable, so when the daemon isn't root it keeps one root child
|
||||
// (`hamlab drm-clients`) that does the walk, asked over a pipe — one sudo per daemon
|
||||
// start, not one per second.
|
||||
//
|
||||
// A GPU with no counter of either kind (Asahi, today) is still listed: a name is better
|
||||
// than silence about hardware the machine has.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// GPU is one graphics device as a Sample reports it.
|
||||
type GPU struct {
|
||||
Name string `json:"name"`
|
||||
Driver string `json:"driver"`
|
||||
Bus string `json:"bus,omitempty"` // PCI address; empty for a platform device
|
||||
Integrated bool `json:"integrated,omitempty"`
|
||||
Usage float64 `json:"usage"` // % busy
|
||||
UsageSrc string `json:"usage_src,omitempty"` // "idle" (100 − idle residency), "busy" (the driver's own %), "clients" (the busiest engine); "" when nothing measures it
|
||||
Engines map[string]float64 `json:"engines,omitempty"` // render/video/copy/enhance… % from the clients, normalised by engine count
|
||||
Temp float64 `json:"temp,omitempty"`
|
||||
TempSrc string `json:"temp_src,omitempty"` // "gpu" from the device's own sensor, "package" the CPU die an integrated GPU shares
|
||||
PowerW float64 `json:"power_w,omitempty"`
|
||||
PowerCapW float64 `json:"power_cap_w,omitempty"`
|
||||
FanRPM int `json:"fan_rpm,omitempty"`
|
||||
MHz float64 `json:"mhz,omitempty"` // actual clock (0 when power-gated)
|
||||
ReqMHz float64 `json:"req_mhz,omitempty"` // what the driver asked for
|
||||
MaxMHz float64 `json:"max_mhz,omitempty"`
|
||||
MemUsed uint64 `json:"mem_used,omitempty"` // VRAM used, or the clients' resident bytes where the driver has no VRAM counter
|
||||
MemTotal uint64 `json:"mem_total,omitempty"`
|
||||
Clients []GPUClient `json:"clients"`
|
||||
ClientsOK bool `json:"clients_ok"` // false when fdinfo of other users' processes could not be read
|
||||
Stalled bool `json:"stalled,omitempty"` // a sysfs read of this card never returned; its readings are paused until restart
|
||||
}
|
||||
|
||||
// GPUClient is one process with the device open.
|
||||
type GPUClient struct {
|
||||
PID int `json:"pid"`
|
||||
Name string `json:"name"`
|
||||
Usage float64 `json:"usage"` // its busiest engine, %
|
||||
Engines map[string]float64 `json:"engines,omitempty"`
|
||||
Mem uint64 `json:"mem,omitempty"` // resident bytes
|
||||
}
|
||||
|
||||
// gpuDrivers are DRM drivers that drive a GPU, not just a scanout (simpledrm, ast, the
|
||||
// Apple display coprocessor) — those have no engines and nothing to say.
|
||||
var gpuDrivers = map[string]bool{"i915": true, "xe": true, "amdgpu": true, "radeon": true, "nouveau": true, "nvidia-drm": true,
|
||||
"asahi": true, "panfrost": true, "panthor": true, "v3d": true, "msm": true, "etnaviv": true, "lima": true, "powervr": true, "virtio_gpu": true}
|
||||
|
||||
type gpuDev struct {
|
||||
card, dev, key, bus, driver, name, hwmon string
|
||||
integrated, stalled bool
|
||||
prevIdle, prevEnergy float64
|
||||
prevAt time.Time
|
||||
}
|
||||
|
||||
type gpuState struct {
|
||||
devs []*gpuDev
|
||||
listedAt time.Time
|
||||
clients map[string]clientPrev // device key + client id → engine totals at the previous scan
|
||||
last []drmFD // the newest client walk, reused between walks
|
||||
scanAt time.Time
|
||||
helper *drmHelper
|
||||
ok bool // the last walk saw every process
|
||||
}
|
||||
|
||||
type clientPrev struct {
|
||||
ns map[string]uint64
|
||||
at time.Time
|
||||
}
|
||||
|
||||
const gpuRelist = time.Minute // how often the card list is re-read: hotplug is rare, the readdir is cheap
|
||||
const gpuScanEvery = time.Second // engine shares over less than this are noise
|
||||
|
||||
// gpus is the GPU section of a sample. cpuTemp stands in for an integrated GPU's
|
||||
// temperature: it is the same die.
|
||||
func (c *Collector) gpus(now time.Time, cpuTemp float64) []GPU {
|
||||
g := &c.gpu
|
||||
if g.devs == nil || now.Sub(g.listedAt) >= gpuRelist {
|
||||
g.devs, g.listedAt = listGPUs(g.devs), now
|
||||
}
|
||||
if len(g.devs) == 0 {
|
||||
return []GPU{}
|
||||
}
|
||||
if g.clients == nil {
|
||||
g.clients = map[string]clientPrev{}
|
||||
}
|
||||
if g.scanAt.IsZero() || now.Sub(g.scanAt) >= gpuScanEvery {
|
||||
g.last, g.ok = c.scanClients()
|
||||
g.scanAt = now
|
||||
}
|
||||
out := make([]GPU, 0, len(g.devs))
|
||||
live := map[string]bool{}
|
||||
for _, d := range g.devs {
|
||||
x := GPU{Name: d.name, Driver: d.driver, Bus: d.bus, Integrated: d.integrated, Clients: []GPUClient{}, ClientsOK: g.ok}
|
||||
d.readWithin(&x, now)
|
||||
if x.Temp == 0 && d.integrated && cpuTemp > 0 {
|
||||
x.Temp, x.TempSrc = cpuTemp, "package"
|
||||
}
|
||||
// the clients: one entry per client id, under the first pid that holds it (a child
|
||||
// that inherited the fd shares the client)
|
||||
byID := map[string]*GPUClient{}
|
||||
var order []string
|
||||
engines := map[string]float64{}
|
||||
var resident uint64
|
||||
for _, f := range g.last {
|
||||
fi := parseFdinfo(f.Info)
|
||||
if fi.pdev != "" && fi.pdev != d.bus || fi.pdev == "" && fi.driver != d.driver {
|
||||
continue
|
||||
}
|
||||
key := d.key + "/" + fi.client
|
||||
live[key] = true
|
||||
if byID[fi.client] != nil {
|
||||
continue
|
||||
}
|
||||
cl := &GPUClient{PID: f.PID, Name: f.Comm, Mem: fi.mem}
|
||||
byID[fi.client] = cl
|
||||
order = append(order, fi.client)
|
||||
resident += fi.mem
|
||||
if prev, ok := g.clients[key]; ok {
|
||||
if dt := now.Sub(prev.at).Nanoseconds(); dt > 0 {
|
||||
for e, ns := range fi.engines {
|
||||
p, ok := prev.ns[e]
|
||||
if !ok || ns < p {
|
||||
continue
|
||||
}
|
||||
pct := float64(ns-p) / float64(dt) * 100
|
||||
if n := fi.capacity[e]; n > 1 {
|
||||
pct /= float64(n)
|
||||
}
|
||||
pct = round1(pct)
|
||||
if cl.Engines == nil {
|
||||
cl.Engines = map[string]float64{}
|
||||
}
|
||||
cl.Engines[e] = pct
|
||||
engines[e] += pct
|
||||
if pct > cl.Usage {
|
||||
cl.Usage = pct
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
g.clients[key] = clientPrev{ns: fi.engines, at: now}
|
||||
}
|
||||
for _, id := range order {
|
||||
x.Clients = append(x.Clients, *byID[id])
|
||||
}
|
||||
sort.SliceStable(x.Clients, func(i, j int) bool { return x.Clients[i].Usage > x.Clients[j].Usage })
|
||||
if len(engines) > 0 {
|
||||
x.Engines = map[string]float64{}
|
||||
for e, v := range engines {
|
||||
if v > 100 {
|
||||
v = 100
|
||||
}
|
||||
x.Engines[e] = round1(v)
|
||||
}
|
||||
}
|
||||
if x.UsageSrc == "" && len(x.Engines) > 0 {
|
||||
for _, v := range x.Engines {
|
||||
if v > x.Usage {
|
||||
x.Usage = v
|
||||
}
|
||||
}
|
||||
x.UsageSrc = "clients"
|
||||
}
|
||||
if x.MemTotal == 0 && x.MemUsed == 0 {
|
||||
x.MemUsed = resident
|
||||
}
|
||||
out = append(out, x)
|
||||
}
|
||||
for k := range g.clients { // a client that closed the device
|
||||
if !live[k] {
|
||||
delete(g.clients, k)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// listGPUs enumerates the cards, keeping the counters of those already known.
|
||||
func listGPUs(known []*gpuDev) []*gpuDev {
|
||||
ents, _ := os.ReadDir("/sys/class/drm")
|
||||
old := map[string]*gpuDev{}
|
||||
for _, d := range known {
|
||||
old[d.card] = d
|
||||
}
|
||||
out := []*gpuDev{}
|
||||
for _, e := range ents {
|
||||
n := e.Name()
|
||||
if !strings.HasPrefix(n, "card") || strings.Contains(n, "-") {
|
||||
continue
|
||||
}
|
||||
card := "/sys/class/drm/" + n
|
||||
dev, err := filepath.EvalSymlinks(card + "/device")
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
driver := ueventValue(dev+"/uevent", "DRIVER")
|
||||
if driver == "" { // some drivers hang the DRIVER key off the parent
|
||||
driver = filepath.Base(readLink(dev + "/driver"))
|
||||
}
|
||||
if !gpuDrivers[driver] {
|
||||
continue
|
||||
}
|
||||
if d, ok := old[card]; ok && d.dev == dev {
|
||||
out = append(out, d)
|
||||
continue
|
||||
}
|
||||
d := &gpuDev{card: card, dev: dev, driver: driver}
|
||||
if pci := ueventValue(dev+"/uevent", "PCI_SLOT_NAME"); pci != "" {
|
||||
d.bus, d.key = pci, pci
|
||||
// an on-die GPU hangs off root bus 0 (Intel's 00:02.0, an AMD APU's 00:08.0 bridge)
|
||||
d.integrated = strings.HasPrefix(pci, "0000:00:")
|
||||
} else {
|
||||
d.key = "driver:" + driver
|
||||
d.integrated = true
|
||||
}
|
||||
if hw, _ := filepath.Glob(dev + "/hwmon/hwmon*"); len(hw) > 0 {
|
||||
d.hwmon = hw[0]
|
||||
}
|
||||
d.name = gpuName(dev, d.bus, driver)
|
||||
out = append(out, d)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].card < out[j].card })
|
||||
return out
|
||||
}
|
||||
|
||||
var gpuVendors = map[string]string{"0x8086": "Intel", "0x1002": "AMD", "0x10de": "NVIDIA", "0x1af4": "Virtio", "0x106b": "Apple"}
|
||||
|
||||
var (
|
||||
revRe = regexp.MustCompile(`\s*\(rev [^)]*\)`)
|
||||
bracketRe = regexp.MustCompile(`\[([^\]]+)\]`)
|
||||
)
|
||||
|
||||
// gpuName is "Intel Arc A380": from the embedded PCI ID table first, so the name is the
|
||||
// same on every machine; from lspci for a card newer than the table; and failing both,
|
||||
// the vendor and the raw device ID.
|
||||
func gpuName(dev, bus, driver string) string {
|
||||
if bus != "" {
|
||||
vid := strings.TrimPrefix(strings.TrimSpace(readStr(dev+"/vendor")), "0x")
|
||||
did := strings.TrimPrefix(strings.TrimSpace(readStr(dev+"/device")), "0x")
|
||||
if vn, dn := pciName(vid, did); dn != "" {
|
||||
return lspciName(vn + " " + dn)
|
||||
}
|
||||
if out, ok := runOut("lspci", "-s", bus); ok {
|
||||
if _, desc, found := strings.Cut(out, ": "); found {
|
||||
return lspciName(desc)
|
||||
}
|
||||
}
|
||||
}
|
||||
vendor := gpuVendors[strings.TrimSpace(readStr(dev+"/vendor"))]
|
||||
if compat := readStr(dev + "/of_node/compatible"); compat != "" { // "apple,agx-t8103\0apple,agx-g13g"
|
||||
first := strings.SplitN(compat, "\x00", 2)[0]
|
||||
if v, model, ok := strings.Cut(first, ","); ok {
|
||||
if vendor == "" && v != "" {
|
||||
vendor = strings.ToUpper(v[:1]) + v[1:]
|
||||
}
|
||||
return vendor + " GPU (" + model + ")"
|
||||
}
|
||||
}
|
||||
if vendor == "" && driver != "" {
|
||||
vendor = strings.ToUpper(driver[:1]) + driver[1:]
|
||||
}
|
||||
if id := strings.TrimPrefix(strings.TrimSpace(readStr(dev+"/device")), "0x"); id != "" {
|
||||
return strings.TrimSpace(vendor + " GPU " + id)
|
||||
}
|
||||
return strings.TrimSpace(vendor + " GPU")
|
||||
}
|
||||
|
||||
// lspciName turns "Intel Corporation DG2 [Arc A380] (rev 05)" into "Intel Arc A380".
|
||||
func lspciName(desc string) string {
|
||||
desc = revRe.ReplaceAllString(strings.TrimSpace(desc), "")
|
||||
vendor := ""
|
||||
for _, v := range []struct{ long, short string }{{"Intel Corporation", "Intel"}, {"Advanced Micro Devices, Inc. [AMD/ATI]", "AMD"}, {"Advanced Micro Devices, Inc. [AMD]", "AMD"}, {"NVIDIA Corporation", "NVIDIA"}, {"Red Hat, Inc.", "Virtio"}} {
|
||||
if strings.HasPrefix(desc, v.long) {
|
||||
vendor = v.short
|
||||
desc = strings.TrimSpace(desc[len(v.long):])
|
||||
break
|
||||
}
|
||||
}
|
||||
if vendor == "" { // vendor not recognised: its first word
|
||||
if i := strings.IndexByte(desc, ' '); i > 0 {
|
||||
vendor, desc = desc[:i], strings.TrimSpace(desc[i+1:])
|
||||
}
|
||||
}
|
||||
if m := bracketRe.FindStringSubmatch(desc); m != nil {
|
||||
desc = m[1]
|
||||
}
|
||||
return strings.TrimSpace(vendor + " " + desc)
|
||||
}
|
||||
|
||||
// readWithin is read with a deadline. A sysfs read that does not come back must not take
|
||||
// the whole sampler with it — every graph on every page of this machine stops — so the
|
||||
// device is marked stalled, its goroutine abandoned, and the rest of the sample goes on.
|
||||
const gpuReadWithin = 3 * time.Second
|
||||
|
||||
func (d *gpuDev) readWithin(x *GPU, now time.Time) {
|
||||
if d.stalled {
|
||||
x.Stalled = true
|
||||
return
|
||||
}
|
||||
done := make(chan GPU, 1)
|
||||
go func() {
|
||||
y := *x
|
||||
d.read(&y, now)
|
||||
done <- y
|
||||
}()
|
||||
select {
|
||||
case y := <-done:
|
||||
*x = y
|
||||
case <-time.After(gpuReadWithin):
|
||||
d.stalled, x.Stalled = true, true
|
||||
log.Printf("gpu: %s (%s): a sysfs read did not return within %s; its readings are paused until restart", d.name, d.card, gpuReadWithin)
|
||||
}
|
||||
}
|
||||
|
||||
// read fills the device-level numbers: clock, idle/busy, hwmon.
|
||||
func (d *gpuDev) read(x *GPU, now time.Time) {
|
||||
dt := now.Sub(d.prevAt).Seconds()
|
||||
switch d.driver {
|
||||
case "i915":
|
||||
x.MHz = readF(d.card + "/gt_act_freq_mhz")
|
||||
x.ReqMHz = readF(d.card + "/gt_cur_freq_mhz")
|
||||
x.MaxMHz = readF(d.card + "/gt_max_freq_mhz")
|
||||
idle, ok := readInt(d.card + "/gt/gt0/rc6_residency_ms")
|
||||
if !ok {
|
||||
idle, ok = readInt(d.card + "/power/rc6_residency_ms")
|
||||
}
|
||||
if ok {
|
||||
d.busyFromIdle(x, float64(idle), dt)
|
||||
}
|
||||
case "xe":
|
||||
gt := d.dev + "/tile0/gt0"
|
||||
x.MHz = readF(gt + "/freq0/act_freq")
|
||||
x.ReqMHz = readF(gt + "/freq0/cur_freq")
|
||||
x.MaxMHz = readF(gt + "/freq0/max_freq")
|
||||
if idle, ok := readInt(gt + "/gtidle/idle_residency_ms"); ok {
|
||||
d.busyFromIdle(x, float64(idle), dt)
|
||||
}
|
||||
case "amdgpu":
|
||||
if v, ok := readInt(d.dev + "/gpu_busy_percent"); ok {
|
||||
x.Usage, x.UsageSrc = float64(v), "busy"
|
||||
}
|
||||
x.MemUsed = uint64(readF(d.dev + "/mem_info_vram_used"))
|
||||
x.MemTotal = uint64(readF(d.dev + "/mem_info_vram_total"))
|
||||
// "1: 1800Mhz *" — the starred level is the current one, the last the highest
|
||||
for _, line := range strings.Split(readStr(d.dev+"/pp_dpm_sclk"), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) < 2 {
|
||||
continue
|
||||
}
|
||||
mhz := leadingNum(f[1])
|
||||
x.MaxMHz = mhz
|
||||
if strings.HasSuffix(line, "*") {
|
||||
x.MHz = mhz
|
||||
}
|
||||
}
|
||||
}
|
||||
if d.hwmon != "" {
|
||||
if t := readF(d.hwmon + "/temp1_input"); t > 0 {
|
||||
x.Temp, x.TempSrc = t/1000, "gpu"
|
||||
}
|
||||
if p := readF(d.hwmon + "/power1_input"); p > 0 {
|
||||
x.PowerW = round1(p / 1e6)
|
||||
} else if p := readF(d.hwmon + "/power1_average"); p > 0 {
|
||||
x.PowerW = round1(p / 1e6)
|
||||
} else if e := readF(d.hwmon + "/energy1_input"); e > 0 { // µJ, cumulative
|
||||
if d.prevEnergy > 0 && e >= d.prevEnergy && dt > 0 && !d.prevAt.IsZero() {
|
||||
x.PowerW = round1((e - d.prevEnergy) / 1e6 / dt)
|
||||
}
|
||||
d.prevEnergy = e
|
||||
}
|
||||
if p := readF(d.hwmon + "/power1_cap"); p > 0 {
|
||||
x.PowerCapW = p / 1e6
|
||||
} else if p := readF(d.hwmon + "/power1_max"); p > 0 {
|
||||
x.PowerCapW = p / 1e6
|
||||
}
|
||||
x.FanRPM = int(readF(d.hwmon + "/fan1_input"))
|
||||
}
|
||||
d.prevAt = now
|
||||
}
|
||||
|
||||
// busyFromIdle: the GT's idle residency is a running ms total; the share of the interval
|
||||
// it did not cover is how busy the device was.
|
||||
func (d *gpuDev) busyFromIdle(x *GPU, idle, dt float64) {
|
||||
x.UsageSrc = "idle"
|
||||
if d.prevAt.IsZero() || dt <= 0 || idle < d.prevIdle {
|
||||
d.prevIdle = idle
|
||||
return
|
||||
}
|
||||
busy := 100 - (idle-d.prevIdle)/(dt*1000)*100
|
||||
if busy < 0 {
|
||||
busy = 0
|
||||
}
|
||||
if busy > 100 {
|
||||
busy = 100
|
||||
}
|
||||
x.Usage = round1(busy)
|
||||
d.prevIdle = idle
|
||||
}
|
||||
|
||||
// ---- DRM fdinfo: who has the device open ----
|
||||
|
||||
// drmFD is one client of a DRM device: the fdinfo text of one open fd, with its owner.
|
||||
type drmFD struct {
|
||||
PID int `json:"pid"`
|
||||
Comm string `json:"comm"`
|
||||
Info string `json:"info"`
|
||||
}
|
||||
|
||||
type fdinfo struct {
|
||||
driver, pdev, client string
|
||||
engines map[string]uint64 // ns, cumulative
|
||||
capacity map[string]int
|
||||
mem uint64 // bytes
|
||||
}
|
||||
|
||||
func parseFdinfo(s string) fdinfo {
|
||||
fi := fdinfo{engines: map[string]uint64{}, capacity: map[string]int{}}
|
||||
var resident, memory uint64
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
k, v, ok := strings.Cut(line, ":")
|
||||
if !ok || !strings.HasPrefix(k, "drm-") {
|
||||
continue
|
||||
}
|
||||
v = strings.TrimSpace(v)
|
||||
switch {
|
||||
case k == "drm-driver":
|
||||
fi.driver = v
|
||||
case k == "drm-pdev":
|
||||
fi.pdev = v
|
||||
case k == "drm-client-id":
|
||||
fi.client = v
|
||||
case strings.HasPrefix(k, "drm-engine-capacity-"):
|
||||
fi.capacity[k[len("drm-engine-capacity-"):]], _ = strconv.Atoi(v)
|
||||
case strings.HasPrefix(k, "drm-engine-"):
|
||||
n, _ := strconv.ParseUint(strings.TrimSuffix(v, " ns"), 10, 64)
|
||||
fi.engines[k[len("drm-engine-"):]] = n
|
||||
case strings.HasPrefix(k, "drm-resident-"):
|
||||
resident += kib(v)
|
||||
case strings.HasPrefix(k, "drm-memory-"):
|
||||
memory += kib(v)
|
||||
}
|
||||
}
|
||||
fi.mem = resident
|
||||
if fi.mem == 0 {
|
||||
fi.mem = memory
|
||||
}
|
||||
return fi
|
||||
}
|
||||
|
||||
// kib parses "52624 KiB" (or MiB, or a bare count of bytes) to bytes.
|
||||
func kib(v string) uint64 {
|
||||
f := strings.Fields(v)
|
||||
if len(f) == 0 {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.ParseUint(f[0], 10, 64)
|
||||
if len(f) > 1 {
|
||||
switch f[1] {
|
||||
case "KiB":
|
||||
n <<= 10
|
||||
case "MiB":
|
||||
n <<= 20
|
||||
case "GiB":
|
||||
n <<= 30
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// drmScan walks /proc for every fd open on /dev/dri/*, one entry per client per process.
|
||||
// Run as root it sees everyone; as a user, only that user's processes.
|
||||
func drmScan() (out []drmFD, complete bool) {
|
||||
ents, err := os.ReadDir("/proc")
|
||||
if err != nil {
|
||||
return []drmFD{}, false
|
||||
}
|
||||
complete = true
|
||||
out = []drmFD{}
|
||||
for _, e := range ents {
|
||||
pid, err := strconv.Atoi(e.Name())
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
fds, err := os.ReadDir("/proc/" + e.Name() + "/fd")
|
||||
if err != nil {
|
||||
if !os.IsNotExist(err) {
|
||||
complete = false
|
||||
}
|
||||
continue
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
comm := ""
|
||||
for _, fd := range fds {
|
||||
target, err := os.Readlink("/proc/" + e.Name() + "/fd/" + fd.Name())
|
||||
if err != nil || !strings.HasPrefix(target, "/dev/dri/") {
|
||||
continue
|
||||
}
|
||||
info := readStr("/proc/" + e.Name() + "/fdinfo/" + fd.Name())
|
||||
_, rest, ok := strings.Cut(info, "drm-client-id:")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
id := strings.TrimSpace(strings.SplitN(rest, "\n", 2)[0])
|
||||
if seen[id] {
|
||||
continue
|
||||
}
|
||||
seen[id] = true
|
||||
if comm == "" {
|
||||
comm = strings.TrimSpace(readStr("/proc/" + e.Name() + "/comm"))
|
||||
}
|
||||
out = append(out, drmFD{PID: pid, Comm: comm, Info: info})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].PID < out[j].PID })
|
||||
return out, complete
|
||||
}
|
||||
|
||||
// scanClients walks in-process when that sees everything (root), else through the helper.
|
||||
func (c *Collector) scanClients() ([]drmFD, bool) {
|
||||
if priv.Mode() == "root" || !priv.Available() {
|
||||
return drmScan()
|
||||
}
|
||||
if c.gpu.helper == nil {
|
||||
c.gpu.helper = &drmHelper{}
|
||||
}
|
||||
if fds, ok := c.gpu.helper.scan(); ok {
|
||||
return fds, true
|
||||
}
|
||||
return drmScan() // the helper is down: our own processes are better than nothing
|
||||
}
|
||||
|
||||
// drmHelper is the root child that answers "scan" requests. One line in, one JSON line out.
|
||||
type drmHelper struct {
|
||||
mu sync.Mutex
|
||||
cmd *exec.Cmd
|
||||
in io.WriteCloser
|
||||
out *bufio.Reader
|
||||
failedAt time.Time
|
||||
}
|
||||
|
||||
const helperRetry = 30 * time.Second
|
||||
|
||||
func (h *drmHelper) scan() ([]drmFD, bool) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
if h.cmd == nil {
|
||||
if !h.failedAt.IsZero() && time.Since(h.failedAt) < helperRetry {
|
||||
return nil, false
|
||||
}
|
||||
if !h.start() {
|
||||
h.failedAt = time.Now()
|
||||
return nil, false
|
||||
}
|
||||
}
|
||||
// a stuck child must not stall the sampler: it is killed, and tried again later
|
||||
cmd := h.cmd
|
||||
stop := time.AfterFunc(5*time.Second, func() { cmd.Process.Kill() })
|
||||
defer stop.Stop()
|
||||
if _, err := io.WriteString(h.in, "\n"); err != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
line, err := h.out.ReadBytes('\n')
|
||||
if err != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
var fds []drmFD
|
||||
if json.Unmarshal(line, &fds) != nil {
|
||||
h.fail()
|
||||
return nil, false
|
||||
}
|
||||
if fds == nil {
|
||||
fds = []drmFD{}
|
||||
}
|
||||
return fds, true
|
||||
}
|
||||
|
||||
func (h *drmHelper) start() bool {
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
full := priv.Wrap(self, "drm-clients")
|
||||
cmd := exec.Command(full[0], full[1:]...)
|
||||
in, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return false
|
||||
}
|
||||
h.cmd, h.in, h.out = cmd, in, bufio.NewReaderSize(out, 1<<20)
|
||||
return true
|
||||
}
|
||||
|
||||
func (h *drmHelper) fail() {
|
||||
if h.cmd != nil {
|
||||
h.in.Close()
|
||||
h.cmd.Process.Kill()
|
||||
go h.cmd.Wait()
|
||||
}
|
||||
h.cmd, h.in, h.out = nil, nil, nil
|
||||
h.failedAt = time.Now()
|
||||
}
|
||||
|
||||
// DRMClientsHelper is `hamlab drm-clients`: for every line read on stdin, one JSON line of
|
||||
// drmFD on stdout. Ends when stdin does — when the daemon that started it has gone.
|
||||
func DRMClientsHelper(stdin io.Reader, stdout io.Writer) int {
|
||||
sc := bufio.NewScanner(stdin)
|
||||
w := bufio.NewWriter(stdout)
|
||||
for sc.Scan() {
|
||||
fds, _ := drmScan()
|
||||
b, _ := json.Marshal(fds)
|
||||
w.Write(b)
|
||||
w.WriteByte('\n')
|
||||
if w.Flush() != nil {
|
||||
return 1
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ---- small helpers ----
|
||||
|
||||
func readF(p string) float64 {
|
||||
v, err := strconv.ParseFloat(strings.TrimSpace(readStr(p)), 64)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func readLink(p string) string {
|
||||
t, err := os.Readlink(p)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func ueventValue(p, key string) string {
|
||||
for _, line := range strings.Split(readStr(p), "\n") {
|
||||
if k, v, ok := strings.Cut(line, "="); ok && k == key {
|
||||
return strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const fdinfoA = `pos: 0
|
||||
flags: 02100002
|
||||
drm-driver: i915
|
||||
drm-client-id: 544
|
||||
drm-pdev: 0000:00:02.0
|
||||
drm-total-system0: 52624 KiB
|
||||
drm-resident-system0: 50140 KiB
|
||||
drm-resident-stolen-system0: 32 KiB
|
||||
drm-engine-render: 1000000000 ns
|
||||
drm-engine-copy: 0 ns
|
||||
drm-engine-video: 2000000000 ns
|
||||
drm-engine-capacity-video: 2
|
||||
drm-engine-video-enhance: 0 ns
|
||||
`
|
||||
|
||||
func TestParseFdinfo(t *testing.T) {
|
||||
fi := parseFdinfo(fdinfoA)
|
||||
if fi.driver != "i915" || fi.pdev != "0000:00:02.0" || fi.client != "544" {
|
||||
t.Fatalf("identity: %+v", fi)
|
||||
}
|
||||
if fi.engines["video"] != 2000000000 || fi.capacity["video"] != 2 {
|
||||
t.Errorf("video engine: %+v", fi)
|
||||
}
|
||||
if fi.mem != (50140+32)<<10 {
|
||||
t.Errorf("resident = %d", fi.mem)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLspciName(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"Intel Corporation DG2 [Arc A380] (rev 05)": "Intel Arc A380",
|
||||
"Intel Corporation RocketLake-S GT1 [UHD Graphics 750] (rev 04)": "Intel UHD Graphics 750",
|
||||
"Intel Corporation Alder Lake-N [UHD Graphics]": "Intel UHD Graphics",
|
||||
"Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c7)": "AMD Radeon RX 6600/6600 XT/6600M",
|
||||
"NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)": "NVIDIA GeForce RTX 3060 Lite Hash Rate",
|
||||
"Moore Threads Technology Co.,Ltd MTT S80": "Moore Threads Technology Co.,Ltd MTT S80",
|
||||
} {
|
||||
if got := lspciName(in); got != want {
|
||||
t.Errorf("%q → %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Two scans a second apart: a client that ran its render engine for half of it and both
|
||||
// video engines flat out reads 50% render, 100% video, and the device's engines follow.
|
||||
func TestGPUClientsShare(t *testing.T) {
|
||||
c := NewCollector()
|
||||
d := &gpuDev{card: "/nonexistent/card9", dev: "/nonexistent", key: "0000:00:02.0", bus: "0000:00:02.0", driver: "i915", name: "Test GPU", integrated: true}
|
||||
c.gpu.devs = []*gpuDev{d}
|
||||
c.gpu.listedAt = time.Now().Add(time.Hour) // keep the fake list
|
||||
c.gpu.clients = map[string]clientPrev{}
|
||||
t0 := time.Now()
|
||||
c.gpu.last = []drmFD{{PID: 7, Comm: "ffmpeg", Info: fdinfoA}}
|
||||
c.gpu.scanAt = t0 // no walk: the fake list stands
|
||||
first := c.gpus(t0, 40)
|
||||
if len(first) != 1 || len(first[0].Clients) != 1 || first[0].Clients[0].Usage != 0 {
|
||||
t.Fatalf("first pass: %+v", first)
|
||||
}
|
||||
if first[0].Temp != 40 || first[0].TempSrc != "package" {
|
||||
t.Errorf("an integrated GPU wears the package temperature: %+v", first[0])
|
||||
}
|
||||
later := fdinfoA
|
||||
later = replace(later, "drm-engine-render: 1000000000 ns", "drm-engine-render: 1500000000 ns")
|
||||
later = replace(later, "drm-engine-video: 2000000000 ns", "drm-engine-video: 4000000000 ns")
|
||||
c.gpu.last = []drmFD{{PID: 7, Comm: "ffmpeg", Info: later}}
|
||||
c.gpu.scanAt = t0.Add(time.Second)
|
||||
got := c.gpus(t0.Add(time.Second), 40)
|
||||
cl := got[0].Clients[0]
|
||||
if cl.Engines["render"] != 50 || cl.Engines["video"] != 100 || cl.Usage != 100 {
|
||||
t.Errorf("client share: %+v", cl)
|
||||
}
|
||||
if got[0].Engines["video"] != 100 || got[0].Usage != 100 || got[0].UsageSrc != "clients" {
|
||||
t.Errorf("device from clients: %+v", got[0])
|
||||
}
|
||||
if got[0].MemUsed != cl.Mem {
|
||||
t.Errorf("resident memory should be the clients' sum: %d vs %d", got[0].MemUsed, cl.Mem)
|
||||
}
|
||||
// the Processes table joins the client to its row
|
||||
c.ring = []Sample{{T: 1, GPUs: got}}
|
||||
c.prevProc = nil
|
||||
table := c.Processes()
|
||||
if len(table.GPUs) != 1 {
|
||||
t.Errorf("table carries the GPUs: %+v", table.GPUs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFlattenGPU(t *testing.T) {
|
||||
s := Sample{T: 5, GPUs: []GPU{{Name: "Intel Arc A380", Driver: "i915", Bus: "0000:03:00.0", Usage: 12, UsageSrc: "idle", Engines: map[string]float64{"video": 30}, PowerW: 9.5, PowerCapW: 55, MHz: 2450, MaxMHz: 2450, Clients: []GPUClient{{PID: 1}}}}}
|
||||
got := map[string]any{}
|
||||
for _, x := range flatten(s, nil) {
|
||||
got[x.ID] = x.Value
|
||||
}
|
||||
for id, want := range map[string]any{"gpu.intel_arc_a380.usage": 12.0, "gpu.intel_arc_a380.video": 30.0, "gpu.intel_arc_a380.power": 9.5, "gpu.intel_arc_a380.freq": 2450.0, "gpu.intel_arc_a380.clients": 1.0} {
|
||||
if got[id] != want {
|
||||
t.Errorf("%s = %v, want %v", id, got[id], want)
|
||||
}
|
||||
}
|
||||
if _, ok := got["gpu.intel_arc_a380.temp"]; ok {
|
||||
t.Error("no temperature sensor, no temperature")
|
||||
}
|
||||
}
|
||||
|
||||
func replace(s, old, new string) string {
|
||||
for i := 0; i+len(old) <= len(s); i++ {
|
||||
if s[i:i+len(old)] == old {
|
||||
return s[:i] + new + s[i+len(old):]
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// TestLiveGPU reads this machine's GPUs for real: HAMLAB_LIVE=1 go test -run TestLiveGPU -v ./internal/metrics
|
||||
func TestLiveGPU(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_LIVE") == "" {
|
||||
t.Skip("set HAMLAB_LIVE=1 to read the real hardware")
|
||||
}
|
||||
c := NewCollector()
|
||||
c.sample()
|
||||
time.Sleep(1100 * time.Millisecond)
|
||||
c.sample()
|
||||
s, _ := c.Latest()
|
||||
b, _ := json.MarshalIndent(s.GPUs, "", " ")
|
||||
t.Logf("%s", b)
|
||||
for _, x := range flatten(s, func(id string) bool { return len(id) > 4 && id[:4] == "gpu." }) {
|
||||
t.Logf("%s = %v %s", x.ID, x.Value, x.Unit)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLiveARC: HAMLAB_LIVE=1 go test -run TestLiveARC -v ./internal/metrics
|
||||
func TestLiveARC(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_LIVE") == "" {
|
||||
t.Skip("set HAMLAB_LIVE=1 to read the real machine")
|
||||
}
|
||||
c := NewCollector()
|
||||
c.sample()
|
||||
time.Sleep(1100 * time.Millisecond)
|
||||
c.sample()
|
||||
s, _ := c.Latest()
|
||||
b, _ := json.MarshalIndent(s.ARC, "", " ")
|
||||
t.Logf("%s", b)
|
||||
for _, x := range flatten(s, func(id string) bool { return strings.HasPrefix(id, "zfs.arc.") }) {
|
||||
t.Logf("%s = %v %s", x.ID, x.Value, x.Unit)
|
||||
}
|
||||
}
|
||||
|
||||
// The embedded table names the lab's four GPUs the way lspci would, so a machine without
|
||||
// pciutils (the router) agrees with one that has it.
|
||||
func TestPCIName(t *testing.T) {
|
||||
for id, want := range map[string]string{"8086:3185": "Intel UHD Graphics 600", "8086:56a5": "Intel Arc A380", "8086:46d1": "Intel UHD Graphics", "8086:4c8a": "Intel UHD Graphics 750", "10de:2684": "NVIDIA GeForce RTX 4090"} {
|
||||
v, d := pciName(id[:4], id[5:])
|
||||
if d == "" {
|
||||
t.Errorf("%s: not in the table", id)
|
||||
continue
|
||||
}
|
||||
if got := lspciName(v + " " + d); got != want {
|
||||
t.Errorf("%s → %q, want %q", id, got, want)
|
||||
}
|
||||
}
|
||||
if _, d := pciName("8086", "ffff"); d != "" {
|
||||
t.Errorf("an unknown device should be empty, got %q", d)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,47 @@
|
||||
package metrics
|
||||
|
||||
// A trimmed PCI ID database, embedded, so a GPU is named the same on every machine —
|
||||
// "Intel UHD Graphics 600" — whether or not that machine has pciutils. OpenWrt has neither
|
||||
// lspci nor pci.ids, and the router's GPU came up as "Intel GPU 3185" beside its twin's
|
||||
// "Intel UHD Graphics 600" until this. scripts/pci-ids-gpu.sh regenerates the file.
|
||||
|
||||
import (
|
||||
_ "embed"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
//go:embed pciids_gpu.txt
|
||||
var pciIDsGPU string
|
||||
|
||||
var pciTable struct {
|
||||
once sync.Once
|
||||
vendors map[string]string // "8086" → "Intel Corporation"
|
||||
devices map[string]string // "8086:3185" → "GeminiLake [UHD Graphics 600]"
|
||||
}
|
||||
|
||||
// pciName looks a vendor and device ID up (hex, lower case, no 0x): the vendor's name and
|
||||
// the device's; empty where unknown.
|
||||
func pciName(vendor, device string) (string, string) {
|
||||
pciTable.once.Do(func() {
|
||||
pciTable.vendors, pciTable.devices = map[string]string{}, map[string]string{}
|
||||
cur := ""
|
||||
for _, line := range strings.Split(pciIDsGPU, "\n") {
|
||||
switch {
|
||||
case line == "" || line[0] == '#':
|
||||
case line[0] == '\t':
|
||||
if cur == "" || len(line) < 7 || line[1] == '\t' {
|
||||
continue // a subsystem, or a device before any vendor
|
||||
}
|
||||
pciTable.devices[cur+":"+line[1:5]] = strings.TrimSpace(line[5:])
|
||||
default:
|
||||
if len(line) >= 6 {
|
||||
cur = line[:4]
|
||||
pciTable.vendors[cur] = strings.TrimSpace(line[4:])
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
vendor, device = strings.ToLower(vendor), strings.ToLower(device)
|
||||
return pciTable.vendors[vendor], pciTable.devices[vendor+":"+device]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,256 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
|
||||
"hamlab/internal/sensors"
|
||||
)
|
||||
|
||||
// The collector as a sensors.Provider: every number in a Sample, one sensor each. The
|
||||
// IDs are built from names the hardware reports (chip, label, interface, device, mount),
|
||||
// never from enumeration order like hwmon3, which moves between boots.
|
||||
|
||||
// flattener turns one Sample into sensors. With want set it builds only those, so walking
|
||||
// the whole ring for a history costs string compares, not allocations.
|
||||
type flattener struct {
|
||||
want func(id string) bool
|
||||
seen map[string]int
|
||||
out []sensors.Sensor
|
||||
}
|
||||
|
||||
type bounds struct{ min, max, warn, crit *float64 }
|
||||
|
||||
var pct = bounds{min: sensors.F(0), max: sensors.F(100)}
|
||||
|
||||
func upTo(max float64) bounds { return bounds{min: sensors.F(0), max: sensors.F(max)} }
|
||||
|
||||
func (f *flattener) add(id, name, kind, unit string, v any, t int64, b bounds, labels ...string) {
|
||||
f.seen[id]++
|
||||
if n := f.seen[id]; n > 1 { // two identical chips: the second is coretemp_2
|
||||
id = fmt.Sprintf("%s_%d", id, n)
|
||||
}
|
||||
if f.want != nil && !f.want(id) {
|
||||
return
|
||||
}
|
||||
s := sensors.Sensor{ID: id, Name: name, Kind: kind, Unit: unit, Value: v, T: t, Min: b.min, Max: b.max, Warn: b.warn, Crit: b.crit}
|
||||
if len(labels) > 0 {
|
||||
s.Labels = make(map[string]string, len(labels)/2)
|
||||
for i := 0; i+1 < len(labels); i += 2 {
|
||||
s.Labels[labels[i]] = labels[i+1]
|
||||
}
|
||||
}
|
||||
f.out = append(f.out, s)
|
||||
}
|
||||
|
||||
func ratio(part, whole uint64) float64 {
|
||||
if whole == 0 {
|
||||
return 0
|
||||
}
|
||||
return round1(float64(part) / float64(whole) * 100)
|
||||
}
|
||||
|
||||
func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
|
||||
f := &flattener{want: want, seen: map[string]int{}}
|
||||
t, none := s.T, bounds{}
|
||||
|
||||
f.add("cpu.usage", "CPU", "usage", "%", s.CPU, t, pct)
|
||||
for i, c := range s.Cores {
|
||||
n := strconv.Itoa(i)
|
||||
f.add("cpu.core."+n+".usage", "Core "+n, "usage", "%", c, t, pct, "core", n)
|
||||
}
|
||||
if c := cpuTemp(s.Temps); c > 0 {
|
||||
f.add("cpu.temp", "CPU temperature", "temperature", "°C", c, t, none)
|
||||
}
|
||||
for i, w := range []string{"1", "5", "15"} {
|
||||
f.add("load."+w, "Load "+w+" min", "load", "", s.Load[i], t, none)
|
||||
}
|
||||
f.add("uptime", "Uptime", "duration", "s", s.Uptime, t, none)
|
||||
f.add("hamlab.sample_interval", "Sampling interval", "duration", "ms", float64(s.Interval), t, none)
|
||||
|
||||
for _, m := range []struct {
|
||||
id, name string
|
||||
v Mem
|
||||
}{{"mem", "Memory", s.Mem}, {"swap", "Swap", s.Swap}} {
|
||||
f.add(m.id+".total", m.name+" total", "bytes", "B", float64(m.v.Total), t, none)
|
||||
f.add(m.id+".used", m.name+" used", "bytes", "B", float64(m.v.Used), t, upTo(float64(m.v.Total)))
|
||||
f.add(m.id+".avail", m.name+" available", "bytes", "B", float64(m.v.Avail), t, upTo(float64(m.v.Total)))
|
||||
f.add(m.id+".used_pct", m.name+" used", "usage", "%", ratio(m.v.Used, m.v.Total), t, pct)
|
||||
}
|
||||
f.add("mem.free", "Memory free", "bytes", "B", float64(s.Mem.Free), t, upTo(float64(s.Mem.Total)))
|
||||
f.add("mem.cached", "Memory cached", "bytes", "B", float64(s.Mem.Cached), t, upTo(float64(s.Mem.Total)))
|
||||
f.add("mem.buffers", "Memory buffers", "bytes", "B", float64(s.Mem.Buffers), t, upTo(float64(s.Mem.Total)))
|
||||
|
||||
for _, x := range s.Temps {
|
||||
b := bounds{}
|
||||
if x.Max > 0 {
|
||||
b.warn = sensors.F(x.Max)
|
||||
}
|
||||
if x.Crit > 0 {
|
||||
b.crit = sensors.F(x.Crit)
|
||||
}
|
||||
f.add(sensors.ID("temp", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "temperature", "°C", x.C, t, b, "chip", x.Chip, "label", x.Label)
|
||||
}
|
||||
for _, x := range s.Fans {
|
||||
f.add(sensors.ID("fan", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "fan", "rpm", float64(x.RPM), t, none, "chip", x.Chip, "label", x.Label)
|
||||
}
|
||||
|
||||
for _, x := range s.Net {
|
||||
p := "net." + sensors.Seg(x.Iface)
|
||||
f.add(p+".rx", x.Iface+" receive", "rate", "B/s", x.RxBps, t, none, "iface", x.Iface)
|
||||
f.add(p+".tx", x.Iface+" transmit", "rate", "B/s", x.TxBps, t, none, "iface", x.Iface)
|
||||
f.add(p+".rx_total", x.Iface+" received", "bytes", "B", float64(x.RxTotal), t, none, "iface", x.Iface)
|
||||
f.add(p+".tx_total", x.Iface+" transmitted", "bytes", "B", float64(x.TxTotal), t, none, "iface", x.Iface)
|
||||
f.add(p+".up", x.Iface+" link", "state", "", x.Up, t, none, "iface", x.Iface)
|
||||
}
|
||||
|
||||
for _, x := range s.Disks {
|
||||
p := "disk." + sensors.Seg(x.Name)
|
||||
f.add(p+".read", x.Name+" read", "rate", "B/s", x.ReadBps, t, none, "device", x.Name, "model", x.Model)
|
||||
f.add(p+".write", x.Name+" write", "rate", "B/s", x.WriteBps, t, none, "device", x.Name, "model", x.Model)
|
||||
f.add(p+".util", x.Name+" busy", "usage", "%", x.UtilPct, t, pct, "device", x.Name, "model", x.Model)
|
||||
f.add(p+".size", x.Name+" size", "bytes", "B", float64(x.Size), t, none, "device", x.Name, "model", x.Model)
|
||||
if x.Temp > 0 {
|
||||
f.add(p+".temp", x.Name+" temperature", "temperature", "°C", x.Temp, t, none, "device", x.Name, "model", x.Model)
|
||||
}
|
||||
}
|
||||
|
||||
for _, x := range s.FS {
|
||||
p := "fs." + sensors.Seg(x.Mount)
|
||||
// remote says the bytes are on another machine (a CIFS or NFS share), so a view of
|
||||
// this machine's hardware can leave it out without losing the sensor
|
||||
lbl := []string{"mount", x.Mount, "fstype", x.Type}
|
||||
if x.Remote {
|
||||
lbl = append(lbl, "remote", "1")
|
||||
}
|
||||
if x.Disk != "" {
|
||||
lbl = append(lbl, "disk", x.Disk)
|
||||
}
|
||||
f.add(p+".size", x.Mount+" size", "bytes", "B", float64(x.Size), s.fsT, none, lbl...)
|
||||
f.add(p+".used", x.Mount+" used", "bytes", "B", float64(x.Used), s.fsT, upTo(float64(x.Size)), lbl...)
|
||||
f.add(p+".used_pct", x.Mount+" used", "usage", "%", ratio(x.Used, x.Size), s.fsT, pct, lbl...)
|
||||
}
|
||||
|
||||
for _, g := range s.GPUs {
|
||||
p := "gpu." + sensors.Seg(g.Name)
|
||||
lbl := []string{"name", g.Name, "driver", g.Driver}
|
||||
if g.Bus != "" {
|
||||
lbl = append(lbl, "bus", g.Bus)
|
||||
}
|
||||
if g.Integrated {
|
||||
lbl = append(lbl, "integrated", "1")
|
||||
}
|
||||
if g.UsageSrc != "" {
|
||||
f.add(p+".usage", g.Name, "usage", "%", g.Usage, t, pct, lbl...)
|
||||
}
|
||||
for _, e := range []string{"render", "video", "copy", "video-enhance", "compute"} {
|
||||
if v, ok := g.Engines[e]; ok {
|
||||
f.add(p+"."+sensors.Seg(e), g.Name+" "+e, "usage", "%", v, t, pct, lbl...)
|
||||
}
|
||||
}
|
||||
if g.Temp > 0 {
|
||||
f.add(p+".temp", g.Name+" temperature", "temperature", "°C", g.Temp, t, none, append(lbl, "source", g.TempSrc)...)
|
||||
}
|
||||
if g.PowerW > 0 || g.PowerCapW > 0 {
|
||||
b := none
|
||||
if g.PowerCapW > 0 {
|
||||
b = upTo(g.PowerCapW)
|
||||
}
|
||||
f.add(p+".power", g.Name+" power", "power", "W", g.PowerW, t, b, lbl...)
|
||||
}
|
||||
if g.FanRPM > 0 {
|
||||
f.add(p+".fan", g.Name+" fan", "fan", "rpm", float64(g.FanRPM), t, none, lbl...)
|
||||
}
|
||||
if g.MaxMHz > 0 {
|
||||
f.add(p+".freq", g.Name+" clock", "frequency", "MHz", g.MHz, t, upTo(g.MaxMHz), lbl...)
|
||||
}
|
||||
if g.MemTotal > 0 {
|
||||
f.add(p+".mem_used", g.Name+" memory used", "bytes", "B", float64(g.MemUsed), t, upTo(float64(g.MemTotal)), lbl...)
|
||||
f.add(p+".mem_total", g.Name+" memory", "bytes", "B", float64(g.MemTotal), t, none, lbl...)
|
||||
} else if g.MemUsed > 0 {
|
||||
f.add(p+".mem_used", g.Name+" memory used", "bytes", "B", float64(g.MemUsed), t, none, lbl...)
|
||||
}
|
||||
f.add(p+".clients", g.Name+" clients", "count", "", float64(len(g.Clients)), t, none, lbl...)
|
||||
}
|
||||
|
||||
if a := s.ARC; a != nil {
|
||||
p := "zfs.arc."
|
||||
cap := upTo(float64(a.Max))
|
||||
f.add(p+"size", "ARC size", "bytes", "B", float64(a.Size), t, cap)
|
||||
f.add(p+"target", "ARC target", "bytes", "B", float64(a.Target), t, cap)
|
||||
f.add(p+"min", "ARC floor", "bytes", "B", float64(a.Min), t, none)
|
||||
f.add(p+"max", "ARC ceiling", "bytes", "B", float64(a.Max), t, none)
|
||||
f.add(p+"mru", "ARC recently used", "bytes", "B", float64(a.MRU), t, cap)
|
||||
f.add(p+"mfu", "ARC frequently used", "bytes", "B", float64(a.MFU), t, cap)
|
||||
f.add(p+"metadata", "ARC metadata", "bytes", "B", float64(a.Metadata), t, cap)
|
||||
f.add(p+"data", "ARC data", "bytes", "B", float64(a.Data), t, cap)
|
||||
if a.HitPctKnown {
|
||||
f.add(p+"hit_pct", "ARC hit rate", "usage", "%", a.HitPct, t, pct)
|
||||
}
|
||||
f.add(p+"reads", "ARC reads", "rate", "/s", a.ReadsPerSec, t, none)
|
||||
f.add(p+"misses", "ARC misses", "rate", "/s", a.MissesPerSec, t, none)
|
||||
f.add(p+"lifetime_hit_pct", "ARC hit rate since boot", "usage", "%", a.LifetimeHitPct, t, pct)
|
||||
f.add(p+"no_grow", "ARC held by memory pressure", "state", "", a.NoGrow, t, none)
|
||||
if a.Compressed > 0 {
|
||||
f.add(p+"compress_ratio", "ARC compression", "ratio", "x", round1(float64(a.Uncompressed)/float64(a.Compressed)*10)/10, t, none)
|
||||
}
|
||||
if a.L2Size > 0 {
|
||||
f.add(p+"l2_size", "L2ARC size", "bytes", "B", float64(a.L2Size), t, none)
|
||||
f.add(p+"l2_hit_pct", "L2ARC hit rate", "usage", "%", a.L2HitPct, t, pct)
|
||||
}
|
||||
}
|
||||
|
||||
if u := s.UPS; u != nil {
|
||||
ut := s.upsT
|
||||
f.add("ups.battery", "UPS battery", "usage", "%", u.BatteryPct, ut, pct, "model", u.Model)
|
||||
f.add("ups.runtime", "UPS runtime", "duration", "min", u.RuntimeMin, ut, none, "model", u.Model)
|
||||
f.add("ups.load", "UPS load", "power", "W", u.LoadW, ut, none, "model", u.Model)
|
||||
f.add("ups.load_pct", "UPS load", "usage", "%", u.LoadPct, ut, pct, "model", u.Model)
|
||||
f.add("ups.in_v", "UPS input", "voltage", "V", u.InV, ut, none, "model", u.Model)
|
||||
f.add("ups.out_v", "UPS output", "voltage", "V", u.OutV, ut, none, "model", u.Model)
|
||||
f.add("ups.on_battery", "UPS on battery", "state", "", u.OnBattery, ut, none, "model", u.Model)
|
||||
f.add("ups.state", "UPS state", "state", "", u.State, ut, none, "model", u.Model)
|
||||
f.add("ups.source", "UPS power source", "state", "", u.Source, ut, none, "model", u.Model)
|
||||
f.add("ups.last_event", "UPS last event", "state", "", u.LastEvent, ut, none, "model", u.Model, "driver", u.Driver)
|
||||
}
|
||||
return f.out
|
||||
}
|
||||
|
||||
// Sensors implements sensors.Provider.
|
||||
func (c *Collector) Sensors() []sensors.Sensor {
|
||||
s, ok := c.Latest()
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
return flatten(s, nil)
|
||||
}
|
||||
|
||||
// SensorHistories implements sensors.Historian: the ring, seen through the sensors that
|
||||
// match. A sensor that didn't exist in a sample (a disk plugged in since) simply has a
|
||||
// shorter series; nothing is zero-filled. States have no curve and are left out.
|
||||
func (c *Collector) SensorHistories(match func(id string) bool) []sensors.Series {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
by := map[string]*sensors.Series{}
|
||||
var order []string
|
||||
for _, smp := range c.ring {
|
||||
for _, x := range flatten(smp, match) {
|
||||
v, ok := x.Value.(float64)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
s := by[x.ID]
|
||||
if s == nil {
|
||||
s = &sensors.Series{ID: x.ID, Unit: x.Unit, T: []int64{}, V: []float64{}}
|
||||
by[x.ID] = s
|
||||
order = append(order, x.ID)
|
||||
}
|
||||
s.T, s.V = append(s.T, smp.T), append(s.V, v)
|
||||
}
|
||||
}
|
||||
out := make([]sensors.Series, 0, len(order))
|
||||
for _, id := range order {
|
||||
out = append(out, *by[id])
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package metrics
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestFlatten(t *testing.T) {
|
||||
s := Sample{T: 1000, CPU: 10, Cores: []float64{5, 15}, Mem: Mem{Total: 200, Used: 50},
|
||||
Temps: []Temp{{Chip: "nvme", Label: "Composite", C: 40, Crit: 85}, {Chip: "nvme", Label: "Composite", C: 45}},
|
||||
Fans: []Fan{{Chip: "nct6775", Label: "fan2", RPM: 1100}},
|
||||
FS: []FS{{Mount: "/", Size: 100, Used: 25}}, fsT: 900}
|
||||
got := map[string]any{}
|
||||
for _, x := range flatten(s, nil) {
|
||||
if _, dup := got[x.ID]; dup {
|
||||
t.Errorf("duplicate id %s", x.ID)
|
||||
}
|
||||
got[x.ID] = x.Value
|
||||
}
|
||||
for id, want := range map[string]any{"cpu.usage": 10.0, "cpu.core.1.usage": 15.0, "mem.used_pct": 25.0, "temp.nvme.composite": 40.0,
|
||||
"temp.nvme.composite_2": 45.0, "fan.nct6775.fan2": 1100.0, "fs.root.used_pct": 25.0} {
|
||||
if got[id] != want {
|
||||
t.Errorf("%s = %v, want %v", id, got[id], want)
|
||||
}
|
||||
}
|
||||
one := flatten(s, func(id string) bool { return id == "temp.nvme.composite_2" })
|
||||
if len(one) != 1 || one[0].Value != 45.0 {
|
||||
t.Errorf("only: got %+v", one)
|
||||
}
|
||||
for _, x := range flatten(s, func(id string) bool { return id == "fs.root.used" }) {
|
||||
if x.T != 900 {
|
||||
t.Errorf("fs sensor should carry the fs read time, got %d", x.T)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSensorHistories(t *testing.T) {
|
||||
c := NewCollector()
|
||||
c.ring = []Sample{{T: 1, CPU: 10}, {T: 2, CPU: 20, Net: []Net{{Iface: "eth0", Up: true, RxBps: 5}}}}
|
||||
got := map[string][]float64{}
|
||||
for _, h := range c.SensorHistories(func(id string) bool { return id == "cpu.usage" || id == "net.eth0.rx" || id == "net.eth0.up" }) {
|
||||
if len(h.T) != len(h.V) {
|
||||
t.Errorf("%s: t and v differ in length", h.ID)
|
||||
}
|
||||
got[h.ID] = h.V
|
||||
}
|
||||
if len(got["cpu.usage"]) != 2 || got["cpu.usage"][1] != 20 {
|
||||
t.Errorf("cpu: %v", got["cpu.usage"])
|
||||
}
|
||||
if len(got["net.eth0.rx"]) != 1 {
|
||||
t.Errorf("a sensor that appeared later has a shorter history, got %v", got["net.eth0.rx"])
|
||||
}
|
||||
if _, ok := got["net.eth0.up"]; ok {
|
||||
t.Error("a state has no history")
|
||||
}
|
||||
if len(c.SensorHistories(func(string) bool { return false })) != 0 {
|
||||
t.Error("nothing matched, nothing returned")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,360 @@
|
||||
// Package network is what the Network tab shows: the machine's interfaces as the kernel has
|
||||
// them, how the internet looks from out there, and a speed test that measures the line.
|
||||
//
|
||||
// Everything here is read from /proc and /sys, or measured against Cloudflare's public speed
|
||||
// endpoints — no tool to install, on any of the machines. That matters for the router, where
|
||||
// flash is small and `opkg install` is a decision.
|
||||
package network
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Addr is one address on an interface.
|
||||
type Addr struct {
|
||||
IP string `json:"ip"`
|
||||
Prefix int `json:"prefix"` // network bits, as in /24
|
||||
V6 bool `json:"v6,omitempty"`
|
||||
}
|
||||
|
||||
// Iface is one network interface: what it is, whether it is up, and what it has moved.
|
||||
type Iface struct {
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"` // ethernet | wifi | bridge | tunnel | loopback | virtual
|
||||
MAC string `json:"mac,omitempty"`
|
||||
MTU int `json:"mtu"`
|
||||
// Up is the administrative state, Carrier whether a cable (or an association) is actually
|
||||
// there. Up without Carrier is the usual "it's plugged into nothing".
|
||||
Up bool `json:"up"`
|
||||
Carrier bool `json:"carrier"`
|
||||
// SpeedMbps is what the link negotiated, where the driver says: 0 for wireless, virtual
|
||||
// and tunnel devices, which have no such number.
|
||||
SpeedMbps int `json:"speed_mbps,omitempty"`
|
||||
Duplex string `json:"duplex,omitempty"`
|
||||
Driver string `json:"driver,omitempty"`
|
||||
SignalDbm *float64 `json:"signal_dbm,omitempty"` // wireless only, from /proc/net/wireless
|
||||
Addrs []Addr `json:"addrs"`
|
||||
RxTotal uint64 `json:"rx_total"`
|
||||
TxTotal uint64 `json:"tx_total"`
|
||||
// Default marks the interface the default route leaves by — the one that matters when
|
||||
// the question is about the internet.
|
||||
Default bool `json:"default,omitempty"`
|
||||
}
|
||||
|
||||
// Report is the machine's network as it stands.
|
||||
type Report struct {
|
||||
At time.Time `json:"at"`
|
||||
Host string `json:"host,omitempty"`
|
||||
Ifaces []Iface `json:"ifaces"`
|
||||
Gateway string `json:"gateway,omitempty"`
|
||||
GatewayVia string `json:"gateway_via,omitempty"` // the interface it is reached through
|
||||
DNS []string `json:"dns,omitempty"`
|
||||
Search []string `json:"search,omitempty"`
|
||||
}
|
||||
|
||||
func readStr(path string) string {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
func readInt(path string) int {
|
||||
n, err := strconv.Atoi(readStr(path))
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// kindOf names what sort of device this is, from what the kernel exposes about it rather
|
||||
// than from its name — a bridge is a bridge whatever it is called.
|
||||
func kindOf(name string, flags net.Flags) string {
|
||||
sys := "/sys/class/net/" + name + "/"
|
||||
switch {
|
||||
case flags&net.FlagLoopback != 0:
|
||||
return "loopback"
|
||||
case dirExists(sys + "wireless"), dirExists(sys + "phy80211"):
|
||||
return "wifi"
|
||||
case dirExists(sys + "bridge"):
|
||||
return "bridge"
|
||||
case dirExists(sys + "bonding"):
|
||||
return "bond"
|
||||
}
|
||||
// DEVTYPE says vlan, bridge, wlan… for the devices that have one
|
||||
for _, line := range strings.Split(readStr(sys+"uevent"), "\n") {
|
||||
if v, ok := strings.CutPrefix(line, "DEVTYPE="); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
if !dirExists(sys + "device") { // no bus device behind it: veth, tun, wireguard, docker's own
|
||||
return "virtual"
|
||||
}
|
||||
return "ethernet"
|
||||
}
|
||||
|
||||
func dirExists(p string) bool {
|
||||
st, err := os.Stat(p)
|
||||
return err == nil && st.IsDir()
|
||||
}
|
||||
|
||||
// driverOf is the module behind an interface, which is what names the hardware in one word.
|
||||
func driverOf(name string) string {
|
||||
l, err := os.Readlink("/sys/class/net/" + name + "/device/driver")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return l[strings.LastIndexByte(l, '/')+1:]
|
||||
}
|
||||
|
||||
// wirelessSignal is the signal level of each wireless interface, in dBm.
|
||||
func wirelessSignal() map[string]float64 {
|
||||
out := map[string]float64{}
|
||||
f, err := os.Open("/proc/net/wireless")
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
name, rest, ok := strings.Cut(sc.Text(), ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
fs := strings.Fields(rest)
|
||||
if len(fs) < 3 {
|
||||
continue
|
||||
}
|
||||
if v, err := strconv.ParseFloat(strings.TrimSuffix(fs[2], "."), 64); err == nil {
|
||||
out[strings.TrimSpace(name)] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// counters is every interface's byte totals, from the same file the sampler reads.
|
||||
func counters() map[string][2]uint64 {
|
||||
out := map[string][2]uint64{}
|
||||
f, err := os.Open("/proc/net/dev")
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
name, rest, ok := strings.Cut(sc.Text(), ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
fs := strings.Fields(rest)
|
||||
if len(fs) < 16 {
|
||||
continue
|
||||
}
|
||||
rx, _ := strconv.ParseUint(fs[0], 10, 64)
|
||||
tx, _ := strconv.ParseUint(fs[8], 10, 64)
|
||||
out[strings.TrimSpace(name)] = [2]uint64{rx, tx}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// defaultRoute is the gateway the machine sends the internet to, and the interface it
|
||||
// leaves by. Read from /proc/net/route, whose addresses are little-endian hex.
|
||||
func defaultRoute() (gw, via string) {
|
||||
f, err := os.Open("/proc/net/route")
|
||||
if err != nil {
|
||||
return "", ""
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
best := -1
|
||||
for sc.Scan() {
|
||||
fs := strings.Fields(sc.Text())
|
||||
if len(fs) < 8 || fs[0] == "Iface" {
|
||||
continue
|
||||
}
|
||||
if fs[1] != "00000000" { // not a default route
|
||||
continue
|
||||
}
|
||||
metric, _ := strconv.Atoi(fs[6])
|
||||
if best >= 0 && metric >= best {
|
||||
continue
|
||||
}
|
||||
ip, err := hexIP(fs[2])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
best, gw, via = metric, ip, fs[0]
|
||||
}
|
||||
return gw, via
|
||||
}
|
||||
|
||||
// hexIP reads one address out of /proc/net/route, where each is printed as a 32-bit word in
|
||||
// the host's byte order: 10.0.0.1 appears as "0100000A" on a little-endian machine. So the
|
||||
// text is parsed as a plain number, then written back out in native order to recover the four
|
||||
// bytes — which is right on either sort of machine, without asking which this is.
|
||||
func hexIP(h string) (string, error) {
|
||||
v, err := strconv.ParseUint(h, 16, 32)
|
||||
if err != nil || len(h) != 8 {
|
||||
return "", fmt.Errorf("not an address: %q", h)
|
||||
}
|
||||
var b [4]byte
|
||||
binary.NativeEndian.PutUint32(b[:], uint32(v))
|
||||
return net.IPv4(b[0], b[1], b[2], b[3]).String(), nil
|
||||
}
|
||||
|
||||
// resolvers is what /etc/resolv.conf points at.
|
||||
func resolvers() (servers, search []string) {
|
||||
f, err := os.Open("/etc/resolv.conf")
|
||||
if err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
fs := strings.Fields(sc.Text())
|
||||
if len(fs) < 2 || strings.HasPrefix(fs[0], "#") {
|
||||
continue
|
||||
}
|
||||
switch fs[0] {
|
||||
case "nameserver":
|
||||
servers = append(servers, fs[1])
|
||||
case "search", "domain":
|
||||
search = append(search, fs[1:]...)
|
||||
}
|
||||
}
|
||||
return servers, search
|
||||
}
|
||||
|
||||
// Interfaces is every interface the kernel has, with its addresses and totals.
|
||||
func Interfaces() []Iface {
|
||||
list, err := net.Interfaces()
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
_, via := defaultRoute()
|
||||
sig, cnt := wirelessSignal(), counters()
|
||||
out := make([]Iface, 0, len(list))
|
||||
for _, in := range list {
|
||||
sys := "/sys/class/net/" + in.Name + "/"
|
||||
f := Iface{
|
||||
Name: in.Name, Kind: kindOf(in.Name, in.Flags), MTU: in.MTU,
|
||||
MAC: in.HardwareAddr.String(), Up: in.Flags&net.FlagUp != 0,
|
||||
Driver: driverOf(in.Name), Default: in.Name == via, Addrs: []Addr{},
|
||||
}
|
||||
f.Carrier = readStr(sys+"carrier") == "1"
|
||||
if f.Kind == "loopback" {
|
||||
f.Carrier = f.Up
|
||||
}
|
||||
if sp := readInt(sys + "speed"); sp > 0 { // -1 when the driver doesn't know
|
||||
f.SpeedMbps = sp
|
||||
}
|
||||
if d := readStr(sys + "duplex"); d != "" && d != "unknown" {
|
||||
f.Duplex = d
|
||||
}
|
||||
if v, ok := sig[in.Name]; ok {
|
||||
f.SignalDbm = &v
|
||||
}
|
||||
if c, ok := cnt[in.Name]; ok {
|
||||
f.RxTotal, f.TxTotal = c[0], c[1]
|
||||
}
|
||||
addrs, _ := in.Addrs()
|
||||
for _, a := range addrs {
|
||||
n, ok := a.(*net.IPNet)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
ones, _ := n.Mask.Size()
|
||||
f.Addrs = append(f.Addrs, Addr{IP: n.IP.String(), Prefix: ones, V6: n.IP.To4() == nil})
|
||||
}
|
||||
out = append(out, f)
|
||||
}
|
||||
// the interface the internet leaves by first, then the ones carrying addresses, then the rest
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
a, b := out[i], out[j]
|
||||
if a.Default != b.Default {
|
||||
return a.Default
|
||||
}
|
||||
ai, bi := len(a.Addrs) > 0 && a.Kind != "loopback", len(b.Addrs) > 0 && b.Kind != "loopback"
|
||||
if ai != bi {
|
||||
return ai
|
||||
}
|
||||
return a.Name < b.Name
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// Look is the machine's network now. Cheap: a handful of small files.
|
||||
func Look(host string) Report {
|
||||
gw, via := defaultRoute()
|
||||
dns, search := resolvers()
|
||||
return Report{At: time.Now(), Host: host, Ifaces: Interfaces(), Gateway: gw, GatewayVia: via, DNS: dns, Search: search}
|
||||
}
|
||||
|
||||
// ---- how the internet sees this machine ----
|
||||
|
||||
// Outside is what the far end of the line says about this connection: the address the
|
||||
// internet sees, and where it came ashore. Refreshed on its own clock, because the public
|
||||
// address here is a DHCP lease and moves without warning.
|
||||
type Outside struct {
|
||||
At time.Time `json:"at"`
|
||||
IP string `json:"ip,omitempty"`
|
||||
Colo string `json:"colo,omitempty"` // the Cloudflare edge that answered — "SEA"
|
||||
Country string `json:"country,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Watcher keeps the outside view fresh and holds the speed tester.
|
||||
type Watcher struct {
|
||||
mu sync.RWMutex
|
||||
out Outside
|
||||
Test *Tester
|
||||
}
|
||||
|
||||
const outsideEvery = 15 * time.Minute
|
||||
|
||||
func NewWatcher(onResult func(Result)) *Watcher {
|
||||
w := &Watcher{Test: NewTester(onResult)}
|
||||
go func() {
|
||||
for {
|
||||
w.refresh()
|
||||
time.Sleep(outsideEvery)
|
||||
}
|
||||
}()
|
||||
return w
|
||||
}
|
||||
|
||||
func (w *Watcher) refresh() {
|
||||
o := probeOutside()
|
||||
w.mu.Lock()
|
||||
// a failed probe does not throw away a good answer; it is noted beside it
|
||||
if o.IP == "" && w.out.IP != "" {
|
||||
w.out.Error, w.out.At = o.Error, o.At
|
||||
} else {
|
||||
w.out = o
|
||||
}
|
||||
w.mu.Unlock()
|
||||
}
|
||||
|
||||
// Outside is the last known view from outside, asking again if it has gone stale.
|
||||
func (w *Watcher) Outside() Outside {
|
||||
w.mu.RLock()
|
||||
o := w.out
|
||||
w.mu.RUnlock()
|
||||
if time.Since(o.At) > outsideEvery {
|
||||
w.refresh()
|
||||
w.mu.RLock()
|
||||
o = w.out
|
||||
w.mu.RUnlock()
|
||||
}
|
||||
return o
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
package network
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestHexIP(t *testing.T) {
|
||||
// /proc/net/route prints each address as a host-order word: on a little-endian machine
|
||||
// 10.0.0.1 is written "0100000A". These cases are that machine, which is all of ours.
|
||||
for hex, want := range map[string]string{
|
||||
"0100000A": "10.0.0.1",
|
||||
"00000000": "0.0.0.0",
|
||||
"0101A8C0": "192.168.1.1",
|
||||
} {
|
||||
got, err := hexIP(hex)
|
||||
if err != nil || got != want {
|
||||
t.Errorf("hexIP(%q) = %q, %v; want %q", hex, got, err, want)
|
||||
}
|
||||
}
|
||||
if _, err := hexIP("nonsense"); err == nil {
|
||||
t.Error("hexIP accepted nonsense")
|
||||
}
|
||||
}
|
||||
|
||||
func TestServerTimeIsNotLatency(t *testing.T) {
|
||||
h := http.Header{}
|
||||
h.Add("Server-Timing", "cfSpeedEdge;dur=7, cfSpeedWorker;dur=36")
|
||||
h.Add("Server-Timing", `cfL4;desc="?proto=TCP&rtt=3505&min_rtt=2856"`)
|
||||
if got := serverMs(h); got != 43 {
|
||||
t.Errorf("serverMs = %v, want 43 (7+36, and nothing from the cfL4 description)", got)
|
||||
}
|
||||
if got := serverMs(http.Header{}); got != 0 {
|
||||
t.Errorf("serverMs with no header = %v, want 0", got)
|
||||
}
|
||||
h2 := http.Header{"Cf-Ray": {"a41f6720f966a36c-SEA"}, "Cf-Meta-Ip": {"50.47.252.221"}}
|
||||
if m := readMeta(h2); m.Colo != "SEA" || m.IP != "50.47.252.221" {
|
||||
t.Errorf("readMeta = %+v, want the edge SEA and the address", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRates(t *testing.T) {
|
||||
// a gibibyte in eight seconds is 1073.7 Mbps
|
||||
if got := mbps(1<<30, 8); math.Abs(got-1073.7) > 0.1 {
|
||||
t.Errorf("mbps(1GiB, 8s) = %v, want ~1073.7", got)
|
||||
}
|
||||
if got := best([]float64{12.5, 9.2, 40, 9.9}); got != 9.2 {
|
||||
t.Errorf("best = %v, want 9.2 (the least of them)", got)
|
||||
}
|
||||
if got := jitter([]float64{10, 10, 10}); got != 0 {
|
||||
t.Errorf("jitter of a steady line = %v, want 0", got)
|
||||
}
|
||||
if got := jitter([]float64{10}); got != 0 {
|
||||
t.Errorf("jitter of one sample = %v, want 0", got)
|
||||
}
|
||||
if got := jitter([]float64{5, 50, 5, 50}); got != 45 {
|
||||
t.Errorf("jitter of a line alternating by 45 = %v, want 45", got)
|
||||
}
|
||||
// one slow probe among steady ones moves it a little, not enormously
|
||||
if got := jitter([]float64{10, 10, 10, 60, 10, 10}); got > 25 {
|
||||
t.Errorf("jitter with a single outlier = %v, want it not to run away", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The body an upload sends is a fixed size, counts itself, and costs nothing to make.
|
||||
func TestZerosBodyCountsItself(t *testing.T) {
|
||||
var n atomic.Int64
|
||||
z := &zeros{left: 100, n: &n}
|
||||
buf := make([]byte, 30)
|
||||
total := 0
|
||||
for {
|
||||
got, err := z.Read(buf)
|
||||
total += got
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
if total != 100 || n.Load() != 100 {
|
||||
t.Errorf("read %d bytes and counted %d, want 100 and 100", total, n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
// Interfaces has to work on whatever machine this runs on, and always name a loopback.
|
||||
func TestInterfacesHasLoopback(t *testing.T) {
|
||||
ifs := Interfaces()
|
||||
if len(ifs) == 0 {
|
||||
t.Fatal("no interfaces at all")
|
||||
}
|
||||
lo := false
|
||||
for _, f := range ifs {
|
||||
if f.Kind == "loopback" {
|
||||
lo = true
|
||||
if !f.Up || len(f.Addrs) == 0 {
|
||||
t.Errorf("loopback %s looks wrong: up=%v addrs=%v", f.Name, f.Up, f.Addrs)
|
||||
}
|
||||
}
|
||||
if f.Name == "" {
|
||||
t.Error("an interface with no name")
|
||||
}
|
||||
}
|
||||
if !lo {
|
||||
t.Error("no loopback among the interfaces")
|
||||
}
|
||||
}
|
||||
|
||||
// The real thing, against the real endpoint: skipped by default, because `make test` must
|
||||
// not need the internet. HAMLAB_NET_TEST=1 go test ./internal/network/ -run Live -v
|
||||
func TestLiveSpeedTest(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_NET_TEST") == "" {
|
||||
t.Skip("set HAMLAB_NET_TEST=1 to measure the actual line")
|
||||
}
|
||||
tr := NewTester(nil)
|
||||
if _, err := tr.Start(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tr.Start(); err == nil {
|
||||
t.Error("a second test started while the first was running")
|
||||
}
|
||||
deadline := time.Now().Add(90 * time.Second)
|
||||
for tr.Running() && time.Now().Before(deadline) {
|
||||
time.Sleep(250 * time.Millisecond)
|
||||
}
|
||||
st := tr.State()
|
||||
if st.Result == nil {
|
||||
t.Fatal("no result")
|
||||
}
|
||||
r := *st.Result
|
||||
t.Logf("down %s up %s latency %.1f ms jitter %.1f ms via %s ip %s %.0f MB moved in %.1fs",
|
||||
Mbits(r.DownMbps), Mbits(r.UpMbps), r.LatencyMs, r.JitterMs, r.Colo, r.IP,
|
||||
float64(r.DownBytes+r.UpBytes)/1e6, r.TookSec)
|
||||
if r.Error != "" {
|
||||
t.Fatalf("test failed: %s", r.Error)
|
||||
}
|
||||
if r.DownMbps <= 0 || r.UpMbps <= 0 || r.LatencyMs <= 0 {
|
||||
t.Errorf("a figure came back zero: %+v", r)
|
||||
}
|
||||
if r.IP == "" {
|
||||
t.Error("no public address reported")
|
||||
}
|
||||
if len(tr.History()) != 1 {
|
||||
t.Errorf("history has %d results, want 1", len(tr.History()))
|
||||
}
|
||||
}
|
||||
|
||||
// A phase measured against a local server, at a rate the test controls. This is the part of
|
||||
// the speed test with arithmetic in it — the window after the ramp, the byte cap that ends it
|
||||
// early, and the rate that comes out — and none of it needs the internet to check.
|
||||
func TestPhaseMeasuresTheWindow(t *testing.T) {
|
||||
const perTick = 1 << 20 // a mebibyte every 5 ms ≈ 1678 Mbps, whatever the machine is doing
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/up" {
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
w.WriteHeader(204)
|
||||
return
|
||||
}
|
||||
buf := make([]byte, perTick)
|
||||
fl, _ := w.(http.Flusher)
|
||||
for {
|
||||
if _, err := w.Write(buf); err != nil {
|
||||
return
|
||||
}
|
||||
if fl != nil {
|
||||
fl.Flush()
|
||||
}
|
||||
select {
|
||||
case <-time.After(5 * time.Millisecond):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
defer func(up string, w, d, mw time.Duration, m int64) {
|
||||
upURL, warmUp, downFor, minWindow, measureBytes = up, w, d, mw, m
|
||||
}(upURL, warmUp, downFor, minWindow, measureBytes)
|
||||
upURL = srv.URL + "/up"
|
||||
warmUp, downFor, minWindow, measureBytes = 100*time.Millisecond, 3*time.Second, 200*time.Millisecond, 8<<20
|
||||
here := Source{"local", func(int) string { return srv.URL + "/down" }}
|
||||
|
||||
tr := NewTester(nil)
|
||||
began := time.Now()
|
||||
mbps, moved, note, err := tr.phase(context.Background(), srv.Client(), "download", 2, downFor, 0, 1, here)
|
||||
took := time.Since(began)
|
||||
if err != nil {
|
||||
t.Fatalf("phase: %v", err)
|
||||
}
|
||||
if note != "" {
|
||||
t.Errorf("note = %q, want none — nothing went wrong", note)
|
||||
}
|
||||
// The cap must have ended it early rather than the clock: the phase was allowed three
|
||||
// seconds and should be done in well under one, having moved at least the cap.
|
||||
if moved < measureBytes {
|
||||
t.Errorf("moved %d bytes, want at least the %d-byte cap", moved, measureBytes)
|
||||
}
|
||||
if took > time.Second {
|
||||
t.Errorf("phase took %v of the %v it was allowed; the byte cap should have ended it", took.Round(time.Millisecond), downFor)
|
||||
}
|
||||
// The server hands out a mebibyte per 5 ms per stream, so the rate is in the hundreds of
|
||||
// Mbps. The point is that it is a real quotient of what moved over how long, not that it
|
||||
// hits a particular figure on a loaded test machine.
|
||||
if mbps < 50 || mbps > 20000 {
|
||||
t.Errorf("mbps = %v, which is not a believable quotient", mbps)
|
||||
}
|
||||
t.Logf("measured %.0f Mbps over %.1f MiB", mbps, float64(moved)/(1<<20))
|
||||
|
||||
// A phase whose server refuses outright fails, and says what it was told.
|
||||
bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(429) }))
|
||||
defer bad.Close()
|
||||
refuser := Source{"refuser", func(int) string { return bad.URL + "/down" }}
|
||||
_, _, _, err = tr.phase(context.Background(), bad.Client(), "download", 2, 300*time.Millisecond, 0, 1, refuser)
|
||||
if err == nil {
|
||||
t.Error("a phase against a server that refuses everything should fail")
|
||||
} else if !refused(err) {
|
||||
t.Errorf("error = %q, want the refusal that makes the fallback kick in", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Stopping a test mid-flight is reported as cancelled, not as whatever the streams said on
|
||||
// their way out.
|
||||
func TestCancelIsReportedAsCancelled(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
buf := make([]byte, 1<<16)
|
||||
for {
|
||||
if _, err := w.Write(buf); err != nil {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-time.After(time.Millisecond):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
defer func(w, d time.Duration) { warmUp, downFor = w, d }(warmUp, downFor)
|
||||
warmUp, downFor = 10*time.Millisecond, 10*time.Second
|
||||
here := Source{"local", func(int) string { return srv.URL + "/down" }}
|
||||
|
||||
tr := NewTester(nil)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
go func() { time.Sleep(200 * time.Millisecond); cancel() }()
|
||||
_, _, _, err := tr.phase(ctx, srv.Client(), "download", 1, downFor, 0, 1, here)
|
||||
// Cancelled after bytes moved: the figure stands, and the phase does not claim an error.
|
||||
if err != nil && !errors.Is(err, context.Canceled) {
|
||||
t.Errorf("err = %v, want nil or a cancellation", err)
|
||||
}
|
||||
}
|
||||
|
||||
// When the first source refuses, the download is taken from the next one and the result says
|
||||
// so. This is the whole reason there is more than one.
|
||||
func TestDownloadFallsBackWhenRefused(t *testing.T) {
|
||||
serving := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/up" {
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
w.WriteHeader(204)
|
||||
return
|
||||
}
|
||||
// paced, so the phase is governed by its window and not by a loopback moving a
|
||||
// gibibyte before the first tick
|
||||
buf := make([]byte, 1<<20)
|
||||
fl, _ := w.(http.Flusher)
|
||||
for i := 0; i < 64; i++ {
|
||||
if _, err := w.Write(buf); err != nil {
|
||||
return
|
||||
}
|
||||
if fl != nil {
|
||||
fl.Flush()
|
||||
}
|
||||
select {
|
||||
case <-time.After(2 * time.Millisecond):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer serving.Close()
|
||||
refusing := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(429) }))
|
||||
defer refusing.Close()
|
||||
|
||||
defer func(srcs []Source, up string, w, d, mw time.Duration, m int64) {
|
||||
downSources, upURL, warmUp, downFor, minWindow, measureBytes = srcs, up, w, d, mw, m
|
||||
}(downSources, upURL, warmUp, downFor, minWindow, measureBytes)
|
||||
downSources = []Source{
|
||||
{"refuser", func(int) string { return refusing.URL + "/down" }},
|
||||
{"the-other-one", func(int) string { return serving.URL + "/down" }},
|
||||
}
|
||||
upURL = serving.URL + "/up"
|
||||
warmUp, downFor, upFor, minWindow, measureBytes = 50*time.Millisecond, time.Second, 300*time.Millisecond, 100*time.Millisecond, 4<<20
|
||||
|
||||
tr := NewTester(nil)
|
||||
if _, err := tr.Start(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
for tr.Running() && time.Now().Before(deadline) {
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
res := tr.State().Result
|
||||
if res == nil {
|
||||
t.Fatal("no result")
|
||||
}
|
||||
if res.Source != "the-other-one" {
|
||||
t.Errorf("source = %q, want the second one — the first refused", res.Source)
|
||||
}
|
||||
if res.DownMbps <= 0 {
|
||||
t.Errorf("download = %v, want a figure from the source that did serve: %+v", res.DownMbps, res)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,687 @@
|
||||
package network
|
||||
|
||||
// The speed test. It measures against Cloudflare's public speed endpoints, the same ones
|
||||
// speed.cloudflare.com uses: a plain GET of N bytes down, a plain POST of N bytes up, and a
|
||||
// zero-byte GET for the round trip. No account, no API key, nothing to install, and an edge
|
||||
// close to every one of these machines.
|
||||
//
|
||||
// The method, since a number without one means little — it is stated in the UI too:
|
||||
//
|
||||
// - Latency is the round trip of a zero-byte response on a connection that is already open,
|
||||
// less the time the far end says it spent on the request (its Server-Timing header). That
|
||||
// subtraction is the difference between measuring the line and measuring Cloudflare's
|
||||
// thinking time: a zero-byte response is almost entirely the latter — tens of
|
||||
// milliseconds of worker, against a few of actual wire. The figure reported is the best of
|
||||
// several probes; jitter is the average change from one to the next.
|
||||
// - Download and upload each run several streams at once for a fixed few seconds. One
|
||||
// stream measures one TCP connection's congestion window as much as the line, which is
|
||||
// why every speed test ever made uses more than one.
|
||||
// - The first second of each phase is thrown away. That is the ramp: counting it reports a
|
||||
// number nobody's connection ever actually did.
|
||||
// - Each phase is capped in bytes as well as seconds, so a very fast line cannot run away
|
||||
// with the meter.
|
||||
//
|
||||
// The one thing to know about the far end: it rate-limits by volume, and a test moves a few
|
||||
// hundred megabytes. Several in a row will start earning 429s, which is why there is a
|
||||
// cooldown between tests and why a phase that gets refused says so instead of reporting the
|
||||
// decayed figure it would otherwise compute.
|
||||
//
|
||||
// HTTP/2 is deliberately off: over one HTTP/2 connection the streams would share a single
|
||||
// congestion window, which measures the window, not the line.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"math"
|
||||
"net"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The endpoints, and the shape of a phase. Variables rather than constants only so the tests
|
||||
// can point them at a local server and shrink the windows; nothing else rebinds them.
|
||||
// Source is somewhere to pull bytes from. Cloudflare's endpoint is the first choice because
|
||||
// it is anycast — near wherever Ham Lab is running — and because the same host answers for
|
||||
// upload and for the trace. The fallback exists because that endpoint rate-limits by volume
|
||||
// and will refuse a connection that has been tested hard: a plain 100 MB file on a public
|
||||
// mirror has no such opinion. It is in Seattle, so on a machine far from there it will read
|
||||
// low, which is exactly why every result names the source it used.
|
||||
type Source struct {
|
||||
Name string
|
||||
// URL takes the number of bytes wanted, where the source can be asked; a fixed file
|
||||
// ignores it and is fetched again each time round.
|
||||
URL func(bytes int) string
|
||||
}
|
||||
|
||||
var downSources = []Source{
|
||||
{"cloudflare", func(b int) string { return fmt.Sprintf(cfDownURL, b) }},
|
||||
{"linode-seattle", func(int) string { return "https://speedtest.seattle.linode.com/100MB-seattle.bin" }},
|
||||
}
|
||||
|
||||
var (
|
||||
cfDownURL = "https://speed.cloudflare.com/__down?bytes=%d"
|
||||
upURL = "https://speed.cloudflare.com/__up"
|
||||
// traceURL is Cloudflare's standard trace: a few lines of key=value naming the address it
|
||||
// sees, the edge that answered and the country. A few hundred bytes.
|
||||
traceURL = "https://speed.cloudflare.com/cdn-cgi/trace"
|
||||
|
||||
warmUp = time.Second
|
||||
downFor = 4 * time.Second
|
||||
upFor = 4 * time.Second
|
||||
// measureBytes ends a phase early once this much has moved inside the window. A fast line
|
||||
// says what it is in well under the time allowed, and stopping there is what keeps a test
|
||||
// down to a couple of hundred megabytes — which is the difference between the far end
|
||||
// serving this happily and rate-limiting it. A slow line never reaches it and is timed out
|
||||
// by downFor/upFor instead, which is the honest way round.
|
||||
measureBytes = int64(64) << 20
|
||||
// minWindow is how long the window must last whatever the byte cap says. Without it a very
|
||||
// fast line fills the cap in a few hundredths of a second and the phase would be over
|
||||
// before there was anything to divide by.
|
||||
minWindow = time.Second
|
||||
)
|
||||
|
||||
const (
|
||||
latencyProbes = 8
|
||||
downStreams = 4
|
||||
upStreams = 3
|
||||
downChunk = 25 << 20 // bytes per download request; the far end refuses much more
|
||||
upChunk = 25 << 20 // and per upload request
|
||||
// phaseBudget is the runaway guard, counting the ramp too: nothing measures forever.
|
||||
phaseBudget = int64(512) << 20
|
||||
httpTimeout = 90 * time.Second
|
||||
// Cooldown is how long the line is left alone between tests. The far end rate-limits by
|
||||
// volume, and two back-to-back tests are the surest way to be refused by it.
|
||||
Cooldown = 20 * time.Second
|
||||
|
||||
// Method is what the UI tells the reader about the numbers it is showing.
|
||||
Method = "Cloudflare's speed endpoints · 4 streams down, 3 up · a 1 s ramp, then up to 4 s or 64 MiB each way"
|
||||
)
|
||||
|
||||
// Result is one finished speed test.
|
||||
type Result struct {
|
||||
At time.Time `json:"at"`
|
||||
DownMbps float64 `json:"down_mbps"`
|
||||
UpMbps float64 `json:"up_mbps"`
|
||||
LatencyMs float64 `json:"latency_ms"`
|
||||
JitterMs float64 `json:"jitter_ms"`
|
||||
DownBytes int64 `json:"down_bytes"`
|
||||
UpBytes int64 `json:"up_bytes"`
|
||||
TookSec float64 `json:"took_sec"`
|
||||
// Colo is the Cloudflare edge that answered — "SEA" — and IP is this connection's public
|
||||
// address as that edge saw it.
|
||||
Colo string `json:"colo,omitempty"`
|
||||
IP string `json:"ip,omitempty"`
|
||||
// Source is where the download came from, which matters because the fallback is not in the
|
||||
// same place as the anycast one. Empty means the usual.
|
||||
Source string `json:"source,omitempty"`
|
||||
// Note is set when the figures are real but something went wrong along the way — a stream
|
||||
// that was refused part-way, most often. The number is what was actually seen; the note is
|
||||
// why it may be low.
|
||||
Note string `json:"note,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Run is a test in progress, or the last one to have finished.
|
||||
type Run struct {
|
||||
Started time.Time `json:"started"`
|
||||
Running bool `json:"running"`
|
||||
// Phase is where it has got to: latency, download, upload, done.
|
||||
Phase string `json:"phase"`
|
||||
// Done is how far through the whole test, 0–1, so one bar can show all of it.
|
||||
Done float64 `json:"done"`
|
||||
// Mbps is the live figure for the phase under way — what the needle shows.
|
||||
Mbps float64 `json:"mbps"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Result *Result `json:"result,omitempty"`
|
||||
}
|
||||
|
||||
// Tester runs one speed test at a time and remembers what the last ones said.
|
||||
type Tester struct {
|
||||
mu sync.RWMutex
|
||||
run Run
|
||||
hist []Result
|
||||
// lastTry is when a test was last *started*, which is what the cooldown counts from — a
|
||||
// test that was refused still used the line, and hammering retries at an endpoint that has
|
||||
// just rate-limited us is the one thing guaranteed not to help.
|
||||
lastTry time.Time
|
||||
cancel context.CancelFunc
|
||||
onDone func(Result)
|
||||
maxHist int
|
||||
}
|
||||
|
||||
func NewTester(onDone func(Result)) *Tester {
|
||||
return &Tester{onDone: onDone, maxHist: 50}
|
||||
}
|
||||
|
||||
// Load takes the results kept from before a restart, oldest first.
|
||||
func (t *Tester) Load(rs []Result) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.hist = append(t.hist[:0], rs...)
|
||||
if n := len(t.hist); n > 0 {
|
||||
last := t.hist[n-1]
|
||||
t.run = Run{Started: last.At, Phase: "done", Done: 1, Result: &last}
|
||||
t.lastTry = last.At
|
||||
}
|
||||
}
|
||||
|
||||
func (t *Tester) State() Run {
|
||||
t.mu.RLock()
|
||||
defer t.mu.RUnlock()
|
||||
return t.run
|
||||
}
|
||||
|
||||
func (t *Tester) History() []Result {
|
||||
t.mu.RLock()
|
||||
defer t.mu.RUnlock()
|
||||
return append([]Result{}, t.hist...)
|
||||
}
|
||||
|
||||
func (t *Tester) Running() bool {
|
||||
t.mu.RLock()
|
||||
defer t.mu.RUnlock()
|
||||
return t.run.Running
|
||||
}
|
||||
|
||||
// Cancel stops a test in progress. Reports whether there was one.
|
||||
func (t *Tester) Cancel() bool {
|
||||
t.mu.Lock()
|
||||
c := t.cancel
|
||||
t.mu.Unlock()
|
||||
if c == nil {
|
||||
return false
|
||||
}
|
||||
c()
|
||||
return true
|
||||
}
|
||||
|
||||
var ErrBusy = errors.New("a speed test is already running")
|
||||
|
||||
// Start begins a test in the background and returns its opening state.
|
||||
func (t *Tester) Start() (Run, error) {
|
||||
t.mu.Lock()
|
||||
if t.run.Running {
|
||||
r := t.run
|
||||
t.mu.Unlock()
|
||||
return r, ErrBusy
|
||||
}
|
||||
if since := time.Since(t.lastTry); since < Cooldown && !t.lastTry.IsZero() {
|
||||
r := t.run
|
||||
t.mu.Unlock()
|
||||
return r, fmt.Errorf("give the line %d more seconds — testing back to back is what gets this connection rate-limited", int((Cooldown-since).Seconds())+1)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
t.cancel = cancel
|
||||
t.lastTry = time.Now()
|
||||
t.run = Run{Started: t.lastTry, Running: true, Phase: "latency"}
|
||||
r := t.run
|
||||
t.mu.Unlock()
|
||||
go t.measure(ctx, cancel)
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (t *Tester) set(fn func(*Run)) {
|
||||
t.mu.Lock()
|
||||
fn(&t.run)
|
||||
t.mu.Unlock()
|
||||
}
|
||||
|
||||
func (t *Tester) measure(ctx context.Context, cancel context.CancelFunc) {
|
||||
defer cancel()
|
||||
started := time.Now()
|
||||
res := Result{At: started}
|
||||
|
||||
tr := newTransport(downStreams)
|
||||
defer tr.CloseIdleConnections()
|
||||
client := &http.Client{Transport: tr, Timeout: httpTimeout}
|
||||
|
||||
// latency, and the meta the same responses carry
|
||||
lat, meta, err := t.latency(ctx, client)
|
||||
if err == nil {
|
||||
res.LatencyMs, res.JitterMs = best(lat), jitter(lat)
|
||||
res.Colo, res.IP = meta.Colo, meta.IP
|
||||
}
|
||||
var note string
|
||||
if err == nil {
|
||||
t.set(func(r *Run) { r.Phase, r.Done, r.Mbps = "download", 0.1, 0 })
|
||||
// Each source in turn: a refusal from the first is exactly what the second is for, and
|
||||
// anything else (no route, no DNS) would fail the same way twice, so it stops there.
|
||||
for i, src := range downSources {
|
||||
res.Source = src.Name
|
||||
res.DownMbps, res.DownBytes, note, err = t.phase(ctx, client, "download", downStreams, downFor, 0.1, 0.6, src)
|
||||
if err == nil || !refused(err) || i == len(downSources)-1 {
|
||||
break
|
||||
}
|
||||
log.Printf("speed test: %s refused the download, trying %s", src.Name, downSources[i+1].Name)
|
||||
}
|
||||
res.Note = join(res.Note, note)
|
||||
}
|
||||
if err == nil {
|
||||
t.set(func(r *Run) { r.Phase, r.Done, r.Mbps = "upload", 0.7, 0 })
|
||||
res.UpMbps, res.UpBytes, note, err = t.phase(ctx, client, "upload", upStreams, upFor, 0.7, 1.0, Source{Name: "cloudflare"})
|
||||
res.Note = join(res.Note, note)
|
||||
}
|
||||
res.TookSec = round(time.Since(started).Seconds(), 1)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
res.Error = "cancelled"
|
||||
} else {
|
||||
res.Error = err.Error()
|
||||
}
|
||||
}
|
||||
|
||||
t.mu.Lock()
|
||||
t.cancel = nil
|
||||
t.run = Run{Started: started, Phase: "done", Done: 1, Result: &res, Bytes: res.DownBytes + res.UpBytes}
|
||||
if res.Error == "" || res.DownMbps > 0 { // a cancelled test that got a number is still a reading
|
||||
t.hist = append(t.hist, res)
|
||||
if len(t.hist) > t.maxHist {
|
||||
t.hist = t.hist[len(t.hist)-t.maxHist:]
|
||||
}
|
||||
}
|
||||
done, out := t.onDone, res
|
||||
t.mu.Unlock()
|
||||
if done != nil {
|
||||
done(out)
|
||||
}
|
||||
}
|
||||
|
||||
// meta is what the edge says about the connection. The address comes from a header of the
|
||||
// speed endpoint's own; the edge's name is the tail of the ray ID, which every Cloudflare
|
||||
// response carries ("a41f6720f966a36c-SEA").
|
||||
type meta struct{ IP, Colo string }
|
||||
|
||||
func readMeta(h http.Header) meta {
|
||||
m := meta{IP: h.Get("cf-meta-ip")}
|
||||
if ray := h.Get("cf-ray"); ray != "" {
|
||||
if i := strings.LastIndexByte(ray, '-'); i >= 0 {
|
||||
m.Colo = ray[i+1:]
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// serverMs is how long the far end says it spent on this request, from Server-Timing. The
|
||||
// cfL4 entry carries no dur= and contributes nothing, which is what we want — it is a
|
||||
// description of the connection, not time spent.
|
||||
func serverMs(h http.Header) float64 {
|
||||
var total float64
|
||||
for _, v := range h.Values("Server-Timing") {
|
||||
for _, part := range strings.Split(v, ",") {
|
||||
_, after, ok := strings.Cut(part, "dur=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
end := 0
|
||||
for end < len(after) && (after[end] == '.' || (after[end] >= '0' && after[end] <= '9')) {
|
||||
end++
|
||||
}
|
||||
if f, err := strconv.ParseFloat(after[:end], 64); err == nil {
|
||||
total += f
|
||||
}
|
||||
}
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// latency times several zero-byte responses on an already-open connection.
|
||||
func (t *Tester) latency(ctx context.Context, client *http.Client) ([]float64, meta, error) {
|
||||
var m meta
|
||||
var out []float64
|
||||
for i := 0; i <= latencyProbes; i++ { // the first is the handshake, and is not counted
|
||||
start := time.Now()
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", fmt.Sprintf(cfDownURL, 0), nil)
|
||||
if err != nil {
|
||||
return nil, m, err
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, m, err
|
||||
}
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
return nil, m, fmt.Errorf("speed endpoint answered %s", resp.Status)
|
||||
}
|
||||
if i == 0 {
|
||||
m = readMeta(resp.Header)
|
||||
continue
|
||||
}
|
||||
rtt := float64(time.Since(start).Microseconds())/1000 - serverMs(resp.Header)
|
||||
if rtt < 0.01 { // the far end claimed more time than the whole request took
|
||||
rtt = 0.01
|
||||
}
|
||||
out = append(out, rtt)
|
||||
t.set(func(r *Run) { r.Done = 0.02 + 0.08*float64(i)/latencyProbes })
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, m, errors.New("no reply from the speed endpoint")
|
||||
}
|
||||
return out, m, nil
|
||||
}
|
||||
|
||||
// phase pushes traffic with several streams for a fixed spell and reports the rate over the
|
||||
// window after the ramp. from and to are where this phase sits in the whole test's progress.
|
||||
//
|
||||
// The window ends when the traffic does, not when the clock runs out: if the far end starts
|
||||
// refusing part-way, the figure is what was actually moved while it was moving, and the note
|
||||
// says what happened. Coasting to the end of the clock with no bytes arriving would report a
|
||||
// number a third of the truth.
|
||||
func (t *Tester) phase(parent context.Context, client *http.Client, dir string, streams int, forHow time.Duration, from, to float64, src Source) (float64, int64, string, error) {
|
||||
// parent is the test's own context, which is cancelled when somebody presses Stop. ctx is
|
||||
// this phase's, which *this function* cancels to shut the streams down — so the two are
|
||||
// kept apart: asking ctx whether the test was cancelled would always say yes by then, and
|
||||
// every failure would be reported as "context canceled" whatever really happened.
|
||||
ctx, stop := context.WithCancel(parent)
|
||||
defer stop()
|
||||
|
||||
var moved atomic.Int64
|
||||
var firstErr error
|
||||
var errOnce sync.Once
|
||||
var alive atomic.Int32
|
||||
alive.Store(int32(streams))
|
||||
dead := make(chan struct{})
|
||||
var deadOnce sync.Once
|
||||
|
||||
for i := 0; i < streams; i++ {
|
||||
go func() {
|
||||
defer func() {
|
||||
if alive.Add(-1) == 0 {
|
||||
deadOnce.Do(func() { close(dead) })
|
||||
}
|
||||
}()
|
||||
tries := 0
|
||||
for ctx.Err() == nil && moved.Load() < phaseBudget {
|
||||
var err error
|
||||
if dir == "download" {
|
||||
err = stream(ctx, client, src, &moved)
|
||||
} else {
|
||||
err = upstream(ctx, client, &moved)
|
||||
}
|
||||
if err == nil {
|
||||
tries = 0
|
||||
continue
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
errOnce.Do(func() { firstErr = err })
|
||||
if tries++; tries > 2 { // refused three times: this stream is finished
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-time.After(400 * time.Millisecond):
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// the ramp, measured and thrown away
|
||||
select {
|
||||
case <-time.After(warmUp):
|
||||
case <-dead:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
mark, at := moved.Load(), time.Now()
|
||||
last, lastAt := mark, at // the last time bytes were still arriving
|
||||
deadline := at.Add(forHow)
|
||||
tick := time.NewTicker(200 * time.Millisecond)
|
||||
defer tick.Stop()
|
||||
|
||||
for time.Now().Before(deadline) {
|
||||
select {
|
||||
case <-tick.C:
|
||||
case <-dead:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
if parent.Err() != nil {
|
||||
break
|
||||
}
|
||||
now := moved.Load()
|
||||
if now > last {
|
||||
last, lastAt = now, time.Now()
|
||||
}
|
||||
// enough has moved to know, and it has been long enough to mean it
|
||||
if (now-mark >= measureBytes && time.Since(at) >= minWindow) || now >= phaseBudget {
|
||||
break
|
||||
}
|
||||
if secs := time.Since(at).Seconds(); secs > 0 {
|
||||
live := mbps(now-mark, secs)
|
||||
part := math.Min(1, secs/forHow.Seconds())
|
||||
t.set(func(r *Run) { r.Mbps, r.Bytes, r.Done = live, now, from+(to-from)*part })
|
||||
}
|
||||
// every stream has given up and nothing more is coming
|
||||
if alive.Load() == 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
total, secs := last-mark, lastAt.Sub(at).Seconds()
|
||||
stop()
|
||||
|
||||
if total <= 0 {
|
||||
switch {
|
||||
case firstErr != nil: // what the far end said explains it better than anything else
|
||||
return 0, moved.Load(), "", firstErr
|
||||
case parent.Err() != nil:
|
||||
return 0, moved.Load(), "", parent.Err()
|
||||
}
|
||||
return 0, moved.Load(), "", fmt.Errorf("nothing moved in the %s phase", dir)
|
||||
}
|
||||
if secs < 0.02 { // too short a window to divide by; the guard is against dividing by zero,
|
||||
// not a judgement about how long a good measurement takes
|
||||
return 0, moved.Load(), "", fmt.Errorf("the %s phase was over before it could be measured", dir)
|
||||
}
|
||||
note := ""
|
||||
if firstErr != nil {
|
||||
note = fmt.Sprintf("the %s phase was cut short: %s", dir, firstErr)
|
||||
}
|
||||
return mbps(total, secs), moved.Load(), note, nil
|
||||
}
|
||||
|
||||
// join puts two notes together without leaving an empty one dangling.
|
||||
func join(a, b string) string {
|
||||
switch {
|
||||
case a == "":
|
||||
return b
|
||||
case b == "":
|
||||
return a
|
||||
}
|
||||
return a + "; " + b
|
||||
}
|
||||
|
||||
// stream fetches one chunk from src, counting bytes as they arrive.
|
||||
func stream(ctx context.Context, client *http.Client, src Source, n *atomic.Int64) error {
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", src.URL(downChunk), nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
return answered(resp.StatusCode, resp.Status)
|
||||
}
|
||||
buf := make([]byte, 64<<10)
|
||||
for {
|
||||
got, err := resp.Body.Read(buf)
|
||||
if got > 0 {
|
||||
n.Add(int64(got))
|
||||
}
|
||||
if err != nil {
|
||||
if err == io.EOF {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// upstream posts one chunk of nothing in particular, counting bytes as they go out.
|
||||
func upstream(ctx context.Context, client *http.Client, n *atomic.Int64) error {
|
||||
body := &zeros{left: upChunk, n: n}
|
||||
req, err := http.NewRequestWithContext(ctx, "POST", upURL, body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.ContentLength = upChunk
|
||||
req.Header.Set("Content-Type", "application/octet-stream")
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != 200 && resp.StatusCode != 204 {
|
||||
return answered(resp.StatusCode, resp.Status)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// answered turns the far end's refusal into something worth reading. It rate-limits by
|
||||
// volume: enough testing in a short while and it stops serving the big transfers, which is
|
||||
// the only failure of this test anyone here is likely to meet.
|
||||
func answered(code int, status string) error {
|
||||
if code == 429 || code == 403 {
|
||||
return errRefused
|
||||
}
|
||||
return fmt.Errorf("the speed endpoint answered %s", status)
|
||||
}
|
||||
|
||||
// errRefused is the far end rate-limiting this connection, which is the only failure of this
|
||||
// test anyone here is likely to meet: it limits by volume, and a test moves a few hundred
|
||||
// megabytes. It is recognisable so the download phase can try its other source.
|
||||
var errRefused = errors.New("the speed endpoint is rate-limiting this connection — it does that after a lot of testing; leave it a few minutes")
|
||||
|
||||
func refused(err error) bool { return errors.Is(err, errRefused) }
|
||||
|
||||
// zeros is a request body of a fixed size that costs nothing to produce, counting itself as
|
||||
// it is read out onto the wire.
|
||||
type zeros struct {
|
||||
left int
|
||||
n *atomic.Int64
|
||||
}
|
||||
|
||||
func (z *zeros) Read(p []byte) (int, error) {
|
||||
if z.left <= 0 {
|
||||
return 0, io.EOF
|
||||
}
|
||||
if len(p) > z.left {
|
||||
p = p[:z.left]
|
||||
}
|
||||
for i := range p {
|
||||
p[i] = 0
|
||||
}
|
||||
z.left -= len(p)
|
||||
z.n.Add(int64(len(p)))
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func newTransport(streams int) *http.Transport {
|
||||
return &http.Transport{
|
||||
Proxy: http.ProxyFromEnvironment,
|
||||
DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext,
|
||||
TLSHandshakeTimeout: 10 * time.Second,
|
||||
MaxIdleConns: streams * 2,
|
||||
MaxIdleConnsPerHost: streams * 2,
|
||||
// An empty map turns HTTP/2 off: see the note at the top of this file.
|
||||
TLSNextProto: map[string]func(string, *tls.Conn) http.RoundTripper{},
|
||||
}
|
||||
}
|
||||
|
||||
// probeOutside asks the far end who it thinks we are. One small request.
|
||||
func probeOutside() Outside {
|
||||
tr := newTransport(1)
|
||||
defer tr.CloseIdleConnections()
|
||||
client := &http.Client{Transport: tr, Timeout: 15 * time.Second}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", traceURL, nil)
|
||||
if err != nil {
|
||||
return Outside{At: time.Now(), Error: err.Error()}
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return Outside{At: time.Now(), Error: shortErr(err)}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
|
||||
if resp.StatusCode != 200 {
|
||||
return Outside{At: time.Now(), Error: "the trace endpoint answered " + resp.Status}
|
||||
}
|
||||
o := Outside{At: time.Now()}
|
||||
for _, line := range strings.Split(string(body), "\n") {
|
||||
k, v, ok := strings.Cut(line, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
switch k {
|
||||
case "ip":
|
||||
o.IP = v
|
||||
case "colo":
|
||||
o.Colo = v
|
||||
case "loc":
|
||||
o.Country = v
|
||||
}
|
||||
}
|
||||
if o.IP == "" {
|
||||
o.Error = "the trace endpoint said nothing about this connection"
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// shortErr keeps the part of a network error a person can act on.
|
||||
func shortErr(err error) string {
|
||||
s := err.Error()
|
||||
if i := strings.LastIndex(s, ": "); i > 0 && len(s)-i < 60 {
|
||||
return s[i+2:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func mbps(bytes int64, secs float64) float64 {
|
||||
return round(float64(bytes)*8/secs/1e6, 1)
|
||||
}
|
||||
|
||||
func best(v []float64) float64 {
|
||||
m := v[0]
|
||||
for _, x := range v {
|
||||
if x < m {
|
||||
m = x
|
||||
}
|
||||
}
|
||||
return round(m, 1)
|
||||
}
|
||||
|
||||
// jitter is how much the line wobbles: the average change from one round trip to the next,
|
||||
// which is the usual definition and, unlike a standard deviation, is not run away with by a
|
||||
// single slow probe.
|
||||
func jitter(v []float64) float64 {
|
||||
if len(v) < 2 {
|
||||
return 0
|
||||
}
|
||||
var sum float64
|
||||
for i := 1; i < len(v); i++ {
|
||||
sum += math.Abs(v[i] - v[i-1])
|
||||
}
|
||||
return round(sum/float64(len(v)-1), 1)
|
||||
}
|
||||
|
||||
func round(v float64, places int) float64 {
|
||||
p := math.Pow(10, float64(places))
|
||||
return math.Round(v*p) / p
|
||||
}
|
||||
|
||||
// Mbits renders a rate the way the line is sold, for a log line.
|
||||
func Mbits(v float64) string { return strconv.FormatFloat(v, 'f', 1, 64) + " Mbps" }
|
||||
@@ -0,0 +1,464 @@
|
||||
// Package nodes is the hub side of multi-host: the list of managed Ham Labs, their
|
||||
// health, adding one (pair with its password, receive a token, pin its certificate),
|
||||
// removing one (revoke the token), and the reverse proxy that turns
|
||||
// /api/nodes/<name>/… into /api/… on that node, WebSockets included.
|
||||
package nodes
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/config"
|
||||
"hamlab/internal/tlsauto"
|
||||
)
|
||||
|
||||
var nameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,31}$`)
|
||||
|
||||
type Status struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
Fingerprint string `json:"fingerprint,omitempty"`
|
||||
Added time.Time `json:"added"`
|
||||
Online bool `json:"online"`
|
||||
Host string `json:"host,omitempty"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Load string `json:"load,omitempty"`
|
||||
Uptime float64 `json:"uptime,omitempty"`
|
||||
RebootNeed bool `json:"reboot_needed,omitempty"`
|
||||
LastSeen time.Time `json:"last_seen,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
cfg *config.Config
|
||||
hubName string
|
||||
mu sync.Mutex
|
||||
status map[string]*Status
|
||||
proxies map[string]*httputil.ReverseProxy
|
||||
clients map[string]*http.Client
|
||||
}
|
||||
|
||||
func New(cfg *config.Config, hubName string) *Manager {
|
||||
return &Manager{cfg: cfg, hubName: hubName, status: map[string]*Status{}, proxies: map[string]*httputil.ReverseProxy{}, clients: map[string]*http.Client{}}
|
||||
}
|
||||
|
||||
func transportFor(n config.Node) *http.Transport {
|
||||
t := &http.Transport{
|
||||
DialContext: (&net.Dialer{Timeout: 5 * time.Second}).DialContext,
|
||||
TLSHandshakeTimeout: 5 * time.Second,
|
||||
ResponseHeaderTimeout: 60 * time.Second,
|
||||
MaxIdleConnsPerHost: 8,
|
||||
}
|
||||
if strings.HasPrefix(n.URL, "https://") {
|
||||
if n.Fingerprint != "" {
|
||||
t.TLSClientConfig = tlsauto.PinnedConfig(n.Fingerprint)
|
||||
} else {
|
||||
t.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
|
||||
}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func (m *Manager) client(n config.Node) *http.Client {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if c, ok := m.clients[n.Name]; ok {
|
||||
return c
|
||||
}
|
||||
c := &http.Client{Transport: transportFor(n), Timeout: 15 * time.Second}
|
||||
m.clients[n.Name] = c
|
||||
return c
|
||||
}
|
||||
|
||||
func (m *Manager) node(name string) (config.Node, bool) {
|
||||
var found config.Node
|
||||
ok := false
|
||||
m.cfg.View(func(c *config.Config) {
|
||||
for _, n := range c.Nodes {
|
||||
if n.Name == name {
|
||||
found, ok = n, true
|
||||
}
|
||||
}
|
||||
})
|
||||
return found, ok
|
||||
}
|
||||
|
||||
// ---- probing ----
|
||||
|
||||
func (m *Manager) probe(ctx context.Context, n config.Node) *Status {
|
||||
st := &Status{Name: n.Name, URL: n.URL, Fingerprint: n.Fingerprint, Added: n.Added}
|
||||
req, _ := http.NewRequestWithContext(ctx, "GET", strings.TrimRight(n.URL, "/")+"/api/me", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
resp, err := m.client(n).Do(req)
|
||||
if err != nil {
|
||||
st.Error = shortErr(err)
|
||||
return st
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
st.Error = fmt.Sprintf("HTTP %d", resp.StatusCode)
|
||||
if resp.StatusCode == 401 {
|
||||
st.Error = "token rejected (re-add the machine)"
|
||||
}
|
||||
return st
|
||||
}
|
||||
var me struct {
|
||||
Host, Version, Load string
|
||||
Uptime float64
|
||||
RebootNeeded bool `json:"reboot_needed"`
|
||||
}
|
||||
if json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&me) != nil {
|
||||
st.Error = "bad reply"
|
||||
return st
|
||||
}
|
||||
st.Online, st.Host, st.Version, st.Load, st.Uptime, st.RebootNeed, st.LastSeen = true, me.Host, me.Version, me.Load, me.Uptime, me.RebootNeeded, time.Now()
|
||||
return st
|
||||
}
|
||||
|
||||
func shortErr(err error) string {
|
||||
s := err.Error()
|
||||
if i := strings.LastIndex(s, ": "); i >= 0 {
|
||||
s = s[i+2:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Run refreshes every node's status periodically.
|
||||
func (m *Manager) Run(ctx context.Context) {
|
||||
m.refresh(ctx)
|
||||
t := time.NewTicker(15 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
m.refresh(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Manager) refresh(ctx context.Context) {
|
||||
var list []config.Node
|
||||
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
|
||||
var wg sync.WaitGroup
|
||||
for _, n := range list {
|
||||
wg.Add(1)
|
||||
go func(n config.Node) {
|
||||
defer wg.Done()
|
||||
pctx, cancel := context.WithTimeout(ctx, 8*time.Second)
|
||||
defer cancel()
|
||||
st := m.probe(pctx, n)
|
||||
m.mu.Lock()
|
||||
if prev, ok := m.status[n.Name]; ok && !st.Online && prev.Online {
|
||||
st.LastSeen = prev.LastSeen // keep the last good sighting
|
||||
}
|
||||
m.status[n.Name] = st
|
||||
m.mu.Unlock()
|
||||
}(n)
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// List returns every managed node with its last known status.
|
||||
func (m *Manager) List() []Status {
|
||||
var list []config.Node
|
||||
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
out := make([]Status, 0, len(list))
|
||||
for _, n := range list {
|
||||
if st, ok := m.status[n.Name]; ok {
|
||||
out = append(out, *st)
|
||||
} else {
|
||||
out = append(out, Status{Name: n.Name, URL: n.URL, Fingerprint: n.Fingerprint, Added: n.Added, Error: "not probed yet"})
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- add / remove ----
|
||||
|
||||
// Add pairs with a node: pin its certificate (trust on first use), trade the node's
|
||||
// password for a token named after this hub, and store the node. The password is
|
||||
// used once and never kept.
|
||||
func (m *Manager) Add(ctx context.Context, name, rawURL, username, password string) (Status, error) {
|
||||
if !nameRe.MatchString(name) {
|
||||
return Status{}, errors.New("name: lowercase letters, digits and dashes only")
|
||||
}
|
||||
if _, exists := m.node(name); exists {
|
||||
return Status{}, errors.New("a machine with that name already exists")
|
||||
}
|
||||
u, err := url.Parse(strings.TrimSpace(rawURL))
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
return Status{}, errors.New("address must be http(s)://host:port")
|
||||
}
|
||||
n := config.Node{Name: name, URL: u.Scheme + "://" + u.Host, Added: time.Now()}
|
||||
if u.Scheme == "https" {
|
||||
host := u.Host
|
||||
if _, _, err := net.SplitHostPort(host); err != nil {
|
||||
host += ":443"
|
||||
}
|
||||
fp, err := tlsauto.Peek(host)
|
||||
if err != nil {
|
||||
return Status{}, fmt.Errorf("connect: %s", shortErr(err))
|
||||
}
|
||||
n.Fingerprint = fp
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"username": username, "password": password, "name": "hub:" + m.hubName})
|
||||
req, _ := http.NewRequestWithContext(ctx, "POST", n.URL+"/api/tokens", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := (&http.Client{Transport: transportFor(n), Timeout: 15 * time.Second}).Do(req)
|
||||
if err != nil {
|
||||
return Status{}, fmt.Errorf("connect: %s", shortErr(err))
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
var reply struct {
|
||||
Token string `json:"token"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&reply)
|
||||
if resp.StatusCode != 200 || reply.Token == "" {
|
||||
if reply.Error == "" {
|
||||
reply.Error = fmt.Sprintf("HTTP %d", resp.StatusCode)
|
||||
}
|
||||
return Status{}, errors.New("pairing refused: " + reply.Error)
|
||||
}
|
||||
n.Token = reply.Token
|
||||
if err := m.cfg.Update(func(c *config.Config) { c.Nodes = append(c.Nodes, n) }); err != nil {
|
||||
return Status{}, err
|
||||
}
|
||||
st := m.probe(ctx, n)
|
||||
m.mu.Lock()
|
||||
m.status[name] = st
|
||||
m.mu.Unlock()
|
||||
return *st, nil
|
||||
}
|
||||
|
||||
// Remove revokes our token on the node (best effort) and forgets it.
|
||||
func (m *Manager) Remove(ctx context.Context, name string) error {
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
return errors.New("no such machine")
|
||||
}
|
||||
req, _ := http.NewRequestWithContext(ctx, "DELETE", n.URL+"/api/tokens/"+url.PathEscape("hub:"+m.hubName), nil)
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
if resp, err := m.client(n).Do(req); err == nil {
|
||||
resp.Body.Close()
|
||||
}
|
||||
err := m.cfg.Update(func(c *config.Config) {
|
||||
kept := c.Nodes[:0]
|
||||
for _, x := range c.Nodes {
|
||||
if x.Name != name {
|
||||
kept = append(kept, x)
|
||||
}
|
||||
}
|
||||
c.Nodes = kept
|
||||
})
|
||||
m.mu.Lock()
|
||||
delete(m.status, name)
|
||||
delete(m.proxies, name)
|
||||
delete(m.clients, name)
|
||||
m.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
|
||||
// ---- update from hub ----
|
||||
|
||||
type PushResult struct {
|
||||
Name string `json:"name"`
|
||||
OK bool `json:"ok"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Push uploads binPath (normally the hub's own executable) to the node's self-update
|
||||
// endpoint. The node verifies it, installs it and restarts itself. Refused when the
|
||||
// node's architecture differs from ours.
|
||||
func (m *Manager) Push(ctx context.Context, name, binPath, ourArch string, force bool) PushResult {
|
||||
res := PushResult{Name: name}
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
res.Error = "no such machine"
|
||||
return res
|
||||
}
|
||||
// arch check via /api/me
|
||||
req, _ := http.NewRequestWithContext(ctx, "GET", n.URL+"/api/me", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
resp, err := m.client(n).Do(req)
|
||||
if err != nil {
|
||||
res.Error = "unreachable: " + shortErr(err)
|
||||
return res
|
||||
}
|
||||
var me struct {
|
||||
Arch string `json:"arch"`
|
||||
Version string `json:"version"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&me)
|
||||
resp.Body.Close()
|
||||
if me.Arch != "" && ourArch != "" && me.Arch != ourArch {
|
||||
// look for a sibling build: /usr/local/bin/hamlab-arm64 next to /usr/local/bin/hamlab
|
||||
alt := binPath + "-" + me.Arch
|
||||
if _, err := os.Stat(alt); err != nil {
|
||||
res.Error = fmt.Sprintf("node is %s but only a %s build is installed here (expected %s; run make deploy, which builds it)", me.Arch, ourArch, alt)
|
||||
return res
|
||||
}
|
||||
binPath = alt
|
||||
}
|
||||
f, err := os.Open(binPath)
|
||||
if err != nil {
|
||||
res.Error = err.Error()
|
||||
return res
|
||||
}
|
||||
defer f.Close()
|
||||
st, _ := f.Stat()
|
||||
target := n.URL + "/api/self-update"
|
||||
if force {
|
||||
target += "?force=1"
|
||||
}
|
||||
up, _ := http.NewRequestWithContext(ctx, "POST", target, f)
|
||||
up.ContentLength = st.Size()
|
||||
up.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
up.Header.Set("Content-Type", "application/octet-stream")
|
||||
c := &http.Client{Transport: transportFor(n), Timeout: 5 * time.Minute}
|
||||
r2, err := c.Do(up)
|
||||
if err != nil {
|
||||
res.Error = "upload: " + shortErr(err)
|
||||
return res
|
||||
}
|
||||
defer r2.Body.Close()
|
||||
var reply struct {
|
||||
Version string `json:"version"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r2.Body, 64<<10)).Decode(&reply)
|
||||
if r2.StatusCode != 200 {
|
||||
if reply.Error == "" {
|
||||
reply.Error = fmt.Sprintf("HTTP %d", r2.StatusCode)
|
||||
}
|
||||
res.Error = reply.Error
|
||||
return res
|
||||
}
|
||||
res.OK, res.Version = true, reply.Version
|
||||
return res
|
||||
}
|
||||
|
||||
// PushAll updates every node, in parallel.
|
||||
func (m *Manager) PushAll(ctx context.Context, binPath, ourArch string, force bool) []PushResult {
|
||||
var list []config.Node
|
||||
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
|
||||
out := make([]PushResult, len(list))
|
||||
var wg sync.WaitGroup
|
||||
for i, n := range list {
|
||||
wg.Add(1)
|
||||
go func(i int, name string) {
|
||||
defer wg.Done()
|
||||
out[i] = m.Push(ctx, name, binPath, ourArch, force)
|
||||
}(i, n.Name)
|
||||
}
|
||||
wg.Wait()
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- proxy ----
|
||||
|
||||
// Proxy returns the handler for /api/nodes/<name>/<rest>: it rewrites to <node>/api/<rest>,
|
||||
// swaps the hub's cookie for the node's bearer token, and drops Origin so the node's
|
||||
// WebSocket origin check (meant for browsers) doesn't reject the hub.
|
||||
// Do makes one request to a managed machine as this hub and returns the whole response.
|
||||
func (m *Manager) Do(ctx context.Context, name, method, path string, body []byte, hdr http.Header) (int, []byte, error) {
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
return 0, nil, errors.New("no such machine")
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(n.URL, "/")+path, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
for k, v := range hdr {
|
||||
req.Header[k] = v
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
resp, err := m.client(n).Do(req)
|
||||
if err != nil {
|
||||
return 0, nil, errors.New(shortErr(err))
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
||||
return resp.StatusCode, b, err
|
||||
}
|
||||
|
||||
// Names lists the managed machines.
|
||||
func (m *Manager) Names() []string {
|
||||
var out []string
|
||||
m.cfg.View(func(c *config.Config) {
|
||||
for _, n := range c.Nodes {
|
||||
out = append(out, n.Name)
|
||||
}
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) Proxy(name string) (http.Handler, bool) {
|
||||
n, ok := m.node(name)
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
m.mu.Lock()
|
||||
if p, ok := m.proxies[name]; ok {
|
||||
m.mu.Unlock()
|
||||
return p, true
|
||||
}
|
||||
m.mu.Unlock()
|
||||
target, _ := url.Parse(n.URL)
|
||||
prefix := "/api/nodes/" + name
|
||||
p := &httputil.ReverseProxy{
|
||||
Director: func(r *http.Request) {
|
||||
r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
|
||||
r.URL.Path = "/api" + strings.TrimPrefix(r.URL.Path, prefix)
|
||||
r.URL.RawPath = ""
|
||||
r.Host = target.Host
|
||||
r.Header.Set("Authorization", "Bearer "+n.Token)
|
||||
r.Header.Del("Cookie")
|
||||
r.Header.Del("Origin")
|
||||
r.Header.Del("X-Forwarded-For")
|
||||
r.Header.Del("X-Forwarded-Proto")
|
||||
},
|
||||
Transport: transportFor(n),
|
||||
FlushInterval: -1, // stream task output / log tails immediately
|
||||
// The node answers with its own policy headers and this hub adds the same ones to
|
||||
// everything it serves; relayed twice, Access-Control-Allow-Origin is invalid and a
|
||||
// browser on another origin refuses the response. The hub's copy is the one that counts.
|
||||
ModifyResponse: func(resp *http.Response) error {
|
||||
for _, h := range []string{"Access-Control-Allow-Origin", "X-Frame-Options", "Content-Security-Policy", "X-Content-Type-Options", "Referrer-Policy"} {
|
||||
resp.Header.Del(h)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": name + " unreachable: " + shortErr(err)})
|
||||
},
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.proxies[name] = p
|
||||
m.mu.Unlock()
|
||||
return p, true
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
// Package priv is the one place that knows how this daemon gets root: it already is root,
|
||||
// or it has passwordless sudo, or passwordless doas (Alpine, some BSD-minded setups), or
|
||||
// it has nothing and the privileged features stay off.
|
||||
package priv
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
mode string // "root" | "sudo" | "doas" | ""
|
||||
prefix []string
|
||||
probed time.Time
|
||||
reprobe = 30 * time.Second
|
||||
)
|
||||
|
||||
// detect settles how root is reached. Finding a way is remembered for good; finding none
|
||||
// is only believed for a while, because the probe can fail for reasons that pass — sudo
|
||||
// taking longer than the timeout on a machine that is still booting left a daemon
|
||||
// convinced it had no root until its next restart.
|
||||
func detect() {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if mode != "" || (!probed.IsZero() && time.Since(probed) < reprobe) {
|
||||
return
|
||||
}
|
||||
probed = time.Now()
|
||||
if os.Geteuid() == 0 {
|
||||
mode = "root"
|
||||
return
|
||||
}
|
||||
for _, tool := range []string{"sudo", "doas"} {
|
||||
if _, err := exec.LookPath(tool); err != nil {
|
||||
continue
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
err := exec.CommandContext(ctx, tool, "-n", "true").Run()
|
||||
cancel()
|
||||
if err == nil {
|
||||
mode, prefix = tool, []string{tool, "-n"}
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Mode reports how root is reached: "root", "sudo", "doas", or "" when it can't be.
|
||||
func Mode() string { detect(); mu.Lock(); defer mu.Unlock(); return mode }
|
||||
|
||||
// Available reports whether commands can be run as root without a prompt.
|
||||
func Available() bool { return Mode() != "" }
|
||||
|
||||
// Wrap returns args prefixed so they run as root. Without privilege it returns args
|
||||
// unchanged; callers gate on Available() where that matters.
|
||||
func Wrap(args ...string) []string {
|
||||
detect()
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return append(append([]string{}, prefix...), args...)
|
||||
}
|
||||
|
||||
// AsUser returns args prefixed so they run as another user with that user's HOME.
|
||||
func AsUser(user string, args ...string) []string {
|
||||
switch Mode() {
|
||||
case "sudo":
|
||||
return append([]string{"sudo", "-n", "-u", user, "-H"}, args...)
|
||||
case "doas":
|
||||
return append([]string{"doas", "-n", "-u", user}, args...)
|
||||
case "root":
|
||||
if user == "root" {
|
||||
return args
|
||||
}
|
||||
// su exists everywhere (util-linux, shadow, busybox), needs no password from root
|
||||
// and sets HOME for the target user.
|
||||
return []string{"su", "-s", "/bin/sh", user, "-c", Quote(args)}
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
// Quote joins args into one POSIX shell command line.
|
||||
func Quote(args []string) string {
|
||||
q := make([]string, len(args))
|
||||
for i, a := range args {
|
||||
q[i] = "'" + strings.ReplaceAll(a, "'", `'\''`) + "'"
|
||||
}
|
||||
return strings.Join(q, " ")
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
// Package selfupdate finds out whether a newer Ham Lab has been published and fetches it.
|
||||
//
|
||||
// A source is one URL. Two shapes are understood, told apart by the URL itself:
|
||||
//
|
||||
// - a Gitea/Forgejo repository API base — ".../api/v1/repos/<owner>/<repo>" — whose
|
||||
// newest release names the version and carries the binaries as assets;
|
||||
// - any other URL, treated as a static directory holding VERSION, SHA256SUMS and
|
||||
// hamlab-linux-<arch>, which is exactly what `make release` writes and what
|
||||
// install.sh already downloads.
|
||||
//
|
||||
// Either way the binary is checked against SHA256SUMS before it is allowed near the
|
||||
// running one. A token is sent when the source is behind a login, so a private Gitea
|
||||
// serves its own fleet without the instance being opened to the world.
|
||||
package selfupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// DefaultCheckEvery is how often a daemon asks, when it is asking at all.
|
||||
const DefaultCheckEvery = 24 * time.Hour
|
||||
|
||||
// DefaultURL is where a build comes from when the config does not say: the project's own
|
||||
// releases. An install elsewhere points `update.url` at its own, or at "off".
|
||||
const DefaultURL = "https://git.bigham.dev/api/v1/repos/mitchell/hamlab"
|
||||
|
||||
// maxBinary is the most we will download; a hamlab binary is ~20 MB.
|
||||
const maxBinary = 512 << 20
|
||||
|
||||
// Source is where updates come from.
|
||||
type Source struct {
|
||||
URL string // API base or static directory
|
||||
Token string // optional; for a source behind a login
|
||||
}
|
||||
|
||||
// Release is one published build.
|
||||
type Release struct {
|
||||
Version string `json:"version"`
|
||||
Notes string `json:"notes,omitempty"`
|
||||
Published time.Time `json:"published,omitempty"`
|
||||
// Binary and Sums are where to get this build for the architecture that was asked for.
|
||||
Binary string `json:"-"`
|
||||
Sums string `json:"-"`
|
||||
}
|
||||
|
||||
// AssetName is what a build for an architecture is called, in a release and on disk.
|
||||
func AssetName(arch string) string { return "hamlab-linux-" + arch }
|
||||
|
||||
func (s Source) api() bool { return strings.Contains(s.URL, "/api/v1/repos/") }
|
||||
|
||||
func (s Source) get(ctx context.Context, url string) (*http.Response, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.Token != "" {
|
||||
req.Header.Set("Authorization", "token "+s.Token)
|
||||
}
|
||||
req.Header.Set("User-Agent", "hamlab-selfupdate")
|
||||
resp, err := (&http.Client{Timeout: 10 * time.Minute}).Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resp.StatusCode != 200 {
|
||||
resp.Body.Close()
|
||||
switch resp.StatusCode {
|
||||
case 401, 403:
|
||||
return nil, fmt.Errorf("%s: the update source needs a login (HTTP %d); set update.token", url, resp.StatusCode)
|
||||
case 404:
|
||||
return nil, fmt.Errorf("%s: nothing published there (HTTP 404)", url)
|
||||
}
|
||||
return nil, fmt.Errorf("%s: HTTP %d", url, resp.StatusCode)
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
func (s Source) getAll(ctx context.Context, url string, limit int64) ([]byte, error) {
|
||||
resp, err := s.get(ctx, url)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
return io.ReadAll(io.LimitReader(resp.Body, limit))
|
||||
}
|
||||
|
||||
// Latest is the newest published build, and where to get it for arch.
|
||||
func (s Source) Latest(ctx context.Context, arch string) (Release, error) {
|
||||
if s.URL == "" {
|
||||
return Release{}, errors.New("no update source configured")
|
||||
}
|
||||
base := strings.TrimSuffix(s.URL, "/")
|
||||
if !s.api() {
|
||||
b, err := s.getAll(ctx, base+"/VERSION", 4096)
|
||||
if err != nil {
|
||||
return Release{}, err
|
||||
}
|
||||
v := strings.TrimSpace(strings.SplitN(string(b), "\n", 2)[0])
|
||||
if v == "" {
|
||||
return Release{}, errors.New("the source published an empty VERSION")
|
||||
}
|
||||
return Release{Version: v, Binary: base + "/" + AssetName(arch), Sums: base + "/SHA256SUMS"}, nil
|
||||
}
|
||||
b, err := s.getAll(ctx, base+"/releases/latest", 4<<20)
|
||||
if err != nil {
|
||||
return Release{}, err
|
||||
}
|
||||
var rel struct {
|
||||
TagName string `json:"tag_name"`
|
||||
Body string `json:"body"`
|
||||
PublishedAt time.Time `json:"published_at"`
|
||||
Assets []struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"browser_download_url"`
|
||||
} `json:"assets"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &rel); err != nil {
|
||||
return Release{}, fmt.Errorf("the source did not answer with a release: %w", err)
|
||||
}
|
||||
if rel.TagName == "" {
|
||||
return Release{}, errors.New("the newest release has no tag")
|
||||
}
|
||||
out := Release{Version: rel.TagName, Notes: rel.Body, Published: rel.PublishedAt}
|
||||
want := AssetName(arch)
|
||||
for _, a := range rel.Assets {
|
||||
switch a.Name {
|
||||
case want:
|
||||
out.Binary = a.URL
|
||||
case "SHA256SUMS":
|
||||
out.Sums = a.URL
|
||||
}
|
||||
}
|
||||
if out.Binary == "" {
|
||||
return out, fmt.Errorf("release %s has no %s", rel.TagName, want)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Fetch downloads the build for arch to dest and checks it against SHA256SUMS. A release
|
||||
// without sums is refused: an unverified binary is not worth installing over a working one.
|
||||
func (s Source) Fetch(ctx context.Context, rel Release, arch, dest string) error {
|
||||
if rel.Binary == "" {
|
||||
return errors.New("that release has no binary for this machine")
|
||||
}
|
||||
if rel.Sums == "" {
|
||||
return errors.New("that release has no SHA256SUMS, so the download cannot be verified")
|
||||
}
|
||||
sums, err := s.getAll(ctx, rel.Sums, 1<<20)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
want := sumFor(string(sums), AssetName(arch))
|
||||
if want == "" {
|
||||
return fmt.Errorf("SHA256SUMS has no entry for %s", AssetName(arch))
|
||||
}
|
||||
resp, err := s.get(ctx, rel.Binary)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
f, err := os.OpenFile(dest, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, maxBinary))
|
||||
cerr := f.Close()
|
||||
if err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
os.Remove(dest)
|
||||
return err
|
||||
}
|
||||
if n < 1<<20 {
|
||||
os.Remove(dest)
|
||||
return fmt.Errorf("downloaded %d bytes: too small to be hamlab", n)
|
||||
}
|
||||
if got := hex.EncodeToString(h.Sum(nil)); got != want {
|
||||
os.Remove(dest)
|
||||
return fmt.Errorf("checksum mismatch: SHA256SUMS says %s, the download is %s", want[:16]+"…", got[:16]+"…")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// sumFor finds a file's checksum in SHA256SUMS ("<hex> <name>", or "*<name>" for binary mode).
|
||||
func sumFor(sums, name string) string {
|
||||
for _, line := range strings.Split(sums, "\n") {
|
||||
f := strings.Fields(strings.TrimSpace(line))
|
||||
if len(f) == 2 && (f[1] == name || f[1] == "*"+name) && len(f[0]) == 64 {
|
||||
return strings.ToLower(f[0])
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Compare orders two versions: -1 if a is older than b, 0 if the same, 1 if newer.
|
||||
// It is install.sh's vnum() rule, so the daemon and the installer never disagree about
|
||||
// which way round two builds go: v0.1.0-69-g9c4f9df → 0001000069, the commits since the
|
||||
// tag included, the hash ignored.
|
||||
func Compare(a, b string) int {
|
||||
na, oka := vnum(a)
|
||||
nb, okb := vnum(b)
|
||||
switch {
|
||||
case !oka && !okb:
|
||||
return strings.Compare(a, b)
|
||||
case !oka:
|
||||
return -1
|
||||
case !okb:
|
||||
return 1
|
||||
case na < nb:
|
||||
return -1
|
||||
case na > nb:
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func vnum(v string) (int64, bool) {
|
||||
v = strings.TrimSpace(v)
|
||||
v = strings.TrimPrefix(v, "v")
|
||||
if i := strings.LastIndex(v, "-g"); i >= 0 && isHex(v[i+2:]) {
|
||||
v = v[:i]
|
||||
}
|
||||
if v == "" {
|
||||
return 0, false
|
||||
}
|
||||
parts := strings.FieldsFunc(v, func(r rune) bool { return r == '.' || r == '-' })
|
||||
nums := make([]int64, 4)
|
||||
for i := 0; i < 4 && i < len(parts); i++ {
|
||||
n, err := strconv.ParseInt(parts[i], 10, 64)
|
||||
if err != nil {
|
||||
if i == 0 {
|
||||
return 0, false
|
||||
}
|
||||
break
|
||||
}
|
||||
nums[i] = n
|
||||
}
|
||||
// %d%03d%03d%05d, as the installer does it
|
||||
return ((nums[0]*1000+nums[1])*1000+nums[2])*100000 + nums[3], true
|
||||
}
|
||||
|
||||
func isHex(s string) bool {
|
||||
if s == "" {
|
||||
return false
|
||||
}
|
||||
for _, r := range s {
|
||||
if !strings.ContainsRune("0123456789abcdefABCDEF", r) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
package selfupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestCompareMatchesInstaller(t *testing.T) {
|
||||
older := []([2]string){
|
||||
{"v0.1.0-69-g9c4f9df", "v0.1.0-107-gad88a48"},
|
||||
{"v0.1.0", "v0.1.0-1-gabc1234"},
|
||||
{"v0.1.0-107", "v0.2.0"},
|
||||
{"v0.9.9-999", "v1.0.0"},
|
||||
{"v0.1.9", "v0.1.10"},
|
||||
}
|
||||
for _, p := range older {
|
||||
if got := Compare(p[0], p[1]); got != -1 {
|
||||
t.Errorf("Compare(%q, %q) = %d, want -1", p[0], p[1], got)
|
||||
}
|
||||
if got := Compare(p[1], p[0]); got != 1 {
|
||||
t.Errorf("Compare(%q, %q) = %d, want 1", p[1], p[0], got)
|
||||
}
|
||||
}
|
||||
same := []string{"v0.1.0-107-gad88a48", "v1.2.3", "dev"}
|
||||
for _, v := range same {
|
||||
if got := Compare(v, v); got != 0 {
|
||||
t.Errorf("Compare(%q, %q) = %d, want 0", v, v, got)
|
||||
}
|
||||
}
|
||||
// the hash is ignored: the same build described two ways is the same build
|
||||
if got := Compare("v0.1.0-107-gad88a48", "v0.1.0-107-gffffff1"); got != 0 {
|
||||
t.Errorf("hash should not count: %d", got)
|
||||
}
|
||||
// "dev" is not a version; anything numbered beats it
|
||||
if got := Compare("dev", "v0.1.0"); got != -1 {
|
||||
t.Errorf("Compare(dev, v0.1.0) = %d, want -1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// serve puts a release behind both shapes of source and returns the two base URLs.
|
||||
func serve(t *testing.T, version string, body []byte, token string) (static, api string, srv *httptest.Server) {
|
||||
t.Helper()
|
||||
sum := sha256.Sum256(body)
|
||||
sums := fmt.Sprintf("%s %s\n%s install.sh\n", hex.EncodeToString(sum[:]), AssetName("amd64"), strings.Repeat("0", 64))
|
||||
mux := http.NewServeMux()
|
||||
guard := func(h http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if token != "" && r.Header.Get("Authorization") != "token "+token {
|
||||
w.WriteHeader(401)
|
||||
return
|
||||
}
|
||||
h(w, r)
|
||||
}
|
||||
}
|
||||
mux.HandleFunc("/dl/VERSION", guard(func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, version) }))
|
||||
mux.HandleFunc("/dl/SHA256SUMS", guard(func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, sums) }))
|
||||
mux.HandleFunc("/dl/"+AssetName("amd64"), guard(func(w http.ResponseWriter, r *http.Request) { w.Write(body) }))
|
||||
mux.HandleFunc("/api/v1/repos/m/hamlab/releases/latest", guard(func(w http.ResponseWriter, r *http.Request) {
|
||||
base := "http://" + r.Host + "/dl/"
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"tag_name": version, "body": "notes here",
|
||||
"assets": []map[string]string{
|
||||
{"name": AssetName("amd64"), "browser_download_url": base + AssetName("amd64")},
|
||||
{"name": "SHA256SUMS", "browser_download_url": base + "SHA256SUMS"},
|
||||
},
|
||||
})
|
||||
}))
|
||||
srv = httptest.NewServer(mux)
|
||||
return srv.URL + "/dl", srv.URL + "/api/v1/repos/m/hamlab", srv
|
||||
}
|
||||
|
||||
func TestLatestAndFetch(t *testing.T) {
|
||||
body := make([]byte, 2<<20)
|
||||
copy(body, "ELF fake hamlab")
|
||||
static, api, srv := serve(t, "v0.9.0-3-gdeadbee", body, "")
|
||||
defer srv.Close()
|
||||
|
||||
for _, url := range []string{static, api} {
|
||||
s := Source{URL: url}
|
||||
rel, err := s.Latest(context.Background(), "amd64")
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", url, err)
|
||||
}
|
||||
if rel.Version != "v0.9.0-3-gdeadbee" {
|
||||
t.Fatalf("%s: version %q", url, rel.Version)
|
||||
}
|
||||
dest := filepath.Join(t.TempDir(), "hamlab.new")
|
||||
if err := s.Fetch(context.Background(), rel, "amd64", dest); err != nil {
|
||||
t.Fatalf("%s: fetch: %v", url, err)
|
||||
}
|
||||
got, _ := os.ReadFile(dest)
|
||||
if len(got) != len(body) {
|
||||
t.Fatalf("%s: got %d bytes, want %d", url, len(got), len(body))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchRefusesBadChecksum(t *testing.T) {
|
||||
body := make([]byte, 2<<20)
|
||||
static, _, srv := serve(t, "v1.0.0", body, "")
|
||||
defer srv.Close()
|
||||
s := Source{URL: static}
|
||||
rel, err := s.Latest(context.Background(), "amd64")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rel.Sums = static + "/VERSION" // sums that name nothing
|
||||
dest := filepath.Join(t.TempDir(), "hamlab.new")
|
||||
if err := s.Fetch(context.Background(), rel, "amd64", dest); err == nil {
|
||||
t.Fatal("a release whose sums do not name the binary must be refused")
|
||||
}
|
||||
if _, err := os.Stat(dest); !os.IsNotExist(err) {
|
||||
t.Fatal("a refused download must not be left on disk")
|
||||
}
|
||||
// and an arch nobody published
|
||||
if _, err := s.Latest(context.Background(), "riscv64"); err == nil {
|
||||
t.Log("static sources cannot know an arch is missing until the fetch; that is fine")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTokenIsSentAndMissingTokenSaysSo(t *testing.T) {
|
||||
body := make([]byte, 2<<20)
|
||||
_, api, srv := serve(t, "v1.0.0", body, "s3cret")
|
||||
defer srv.Close()
|
||||
if _, err := (Source{URL: api}).Latest(context.Background(), "amd64"); err == nil || !strings.Contains(err.Error(), "update.token") {
|
||||
t.Fatalf("a walled source should say a token is needed, got %v", err)
|
||||
}
|
||||
rel, err := (Source{URL: api, Token: "s3cret"}).Latest(context.Background(), "amd64")
|
||||
if err != nil || rel.Version != "v1.0.0" {
|
||||
t.Fatalf("with the token: %v %+v", err, rel)
|
||||
}
|
||||
dest := filepath.Join(t.TempDir(), "hamlab.new")
|
||||
if err := (Source{URL: api, Token: "s3cret"}).Fetch(context.Background(), rel, "amd64", dest); err != nil {
|
||||
t.Fatalf("fetch with token: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
package sensors
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// promFamily names the metric family for a sensor and scales its value to the base unit
|
||||
// Prometheus convention wants (seconds, not minutes).
|
||||
func promFamily(s Sensor) (name string, scale float64) {
|
||||
scale = 1
|
||||
unit := map[string]string{"%": "percent", "°C": "celsius", "rpm": "rpm", "B/s": "bytes_per_second", "B": "bytes",
|
||||
"s": "seconds", "V": "volts", "W": "watts", "MHz": "megahertz"}[s.Unit]
|
||||
if s.Unit == "min" {
|
||||
unit, scale = "seconds", 60
|
||||
}
|
||||
if s.Kind == "count" { // hamlab_count{id=…} would say nothing; name it after the thing counted
|
||||
return "hamlab_" + strings.ReplaceAll(Seg(strings.ReplaceAll(s.ID, ".", "_")), "-", "_"), 1
|
||||
}
|
||||
name = "hamlab_" + s.Kind
|
||||
if unit != "" {
|
||||
name += "_" + unit
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
var promEsc = strings.NewReplacer(`\`, `\\`, `"`, `\"`, "\n", `\n`)
|
||||
|
||||
// WriteProm writes sensors in the Prometheus text exposition format: one family per kind
|
||||
// of measurement, the sensor's ID and its real-world labels as labels. Booleans are 0/1;
|
||||
// a text state is a 1 carrying the text as a label, the usual way to export an enum.
|
||||
func WriteProm(w io.Writer, all []Sensor) {
|
||||
fams := map[string][]string{}
|
||||
for _, s := range all {
|
||||
name, scale := promFamily(s)
|
||||
labels := []string{`id="` + promEsc.Replace(s.ID) + `"`}
|
||||
keys := make([]string, 0, len(s.Labels))
|
||||
for k := range s.Labels {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
if s.Labels[k] != "" {
|
||||
labels = append(labels, Seg(k)+`="`+promEsc.Replace(s.Labels[k])+`"`)
|
||||
}
|
||||
}
|
||||
var v float64
|
||||
switch x := s.Value.(type) {
|
||||
case float64:
|
||||
v = x * scale
|
||||
case bool:
|
||||
if x {
|
||||
v = 1
|
||||
}
|
||||
case string:
|
||||
v = 1
|
||||
labels = append(labels, `value="`+promEsc.Replace(x)+`"`)
|
||||
default:
|
||||
continue
|
||||
}
|
||||
fams[name] = append(fams[name], fmt.Sprintf("%s{%s} %v", name, strings.Join(labels, ","), v))
|
||||
}
|
||||
names := make([]string, 0, len(fams))
|
||||
for n := range fams {
|
||||
names = append(names, n)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, n := range names {
|
||||
fmt.Fprintf(w, "# TYPE %s gauge\n", n)
|
||||
for _, line := range fams[n] {
|
||||
fmt.Fprintln(w, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,275 @@
|
||||
// Package sensors is the one standard way anything reads what Ham Lab measures.
|
||||
//
|
||||
// Every measurable thing is a Sensor: a stable dotted ID ("cpu.core.3.usage",
|
||||
// "fan.nct6775.fan2", "fs.root.used_pct"), a little metadata, and a current value. A
|
||||
// consumer discovers them from the catalog, then asks for exactly the ones it wants —
|
||||
// one by ID, many by glob, or as a shaped query where the client sends the object it
|
||||
// wants and gets it back with the values filled in. See SENSORS.md.
|
||||
//
|
||||
// Sources of sensors are Providers. The registry only joins them; it measures nothing.
|
||||
package sensors
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Sensor is one measurable thing and its reading. Value is a float64, bool or string.
|
||||
type Sensor struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"` // human label
|
||||
Kind string `json:"kind"` // usage | load | temperature | fan | rate | bytes | duration | voltage | power | state
|
||||
Unit string `json:"unit,omitempty"` // % · °C · rpm · B/s · B · s · min · V · W · MHz
|
||||
Value any `json:"value"`
|
||||
T int64 `json:"t"` // unix ms the value was read — slow sensors are older than the tick
|
||||
Min *float64 `json:"min,omitempty"`
|
||||
Max *float64 `json:"max,omitempty"`
|
||||
Warn *float64 `json:"warn,omitempty"` // the hardware's own thresholds, where it states them
|
||||
Crit *float64 `json:"crit,omitempty"`
|
||||
Labels map[string]string `json:"labels,omitempty"` // the real names behind the ID: chip, iface, mount…
|
||||
}
|
||||
|
||||
// Series is a numeric sensor's recent past.
|
||||
type Series struct {
|
||||
ID string `json:"id"`
|
||||
Unit string `json:"unit,omitempty"`
|
||||
T []int64 `json:"t"`
|
||||
V []float64 `json:"v"`
|
||||
}
|
||||
|
||||
// Provider is a source of sensors. Sensors returns every sensor it has, with current values.
|
||||
type Provider interface {
|
||||
Sensors() []Sensor
|
||||
}
|
||||
|
||||
// Historian is a Provider that also remembers: the recent past of every numeric sensor
|
||||
// of its own that match accepts.
|
||||
type Historian interface {
|
||||
SensorHistories(match func(id string) bool) []Series
|
||||
}
|
||||
|
||||
// Ticker is a Provider that can say when its values have changed.
|
||||
type Ticker interface {
|
||||
Subscribe() (tick <-chan struct{}, cancel func())
|
||||
}
|
||||
|
||||
// Subscribe follows the first provider that ticks. ok is false when none does.
|
||||
func (r *Registry) Subscribe() (tick <-chan struct{}, cancel func(), ok bool) {
|
||||
for _, p := range r.providers {
|
||||
if t, is := p.(Ticker); is {
|
||||
tick, cancel = t.Subscribe()
|
||||
return tick, cancel, true
|
||||
}
|
||||
}
|
||||
return nil, nil, false
|
||||
}
|
||||
|
||||
// F is a pointer to a float, for the optional bounds.
|
||||
func F(v float64) *float64 { return &v }
|
||||
|
||||
// Seg makes one ID segment from a real-world name: lower case, [a-z0-9_-] only, so IDs
|
||||
// survive URLs, query strings and templating languages untouched. "Package id 0" →
|
||||
// "package_id_0", "/mnt/tank" → "mnt_tank", "/" → "root".
|
||||
func Seg(name string) string {
|
||||
if name == "/" {
|
||||
return "root"
|
||||
}
|
||||
var b strings.Builder
|
||||
gap := false
|
||||
for _, r := range strings.ToLower(strings.TrimSpace(name)) {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z', r >= '0' && r <= '9', r == '-':
|
||||
if gap && b.Len() > 0 {
|
||||
b.WriteByte('_')
|
||||
}
|
||||
gap = false
|
||||
b.WriteRune(r)
|
||||
default:
|
||||
gap = true
|
||||
}
|
||||
}
|
||||
if b.Len() == 0 {
|
||||
return "unnamed"
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// ID joins already-safe segments.
|
||||
func ID(segs ...string) string { return strings.Join(segs, ".") }
|
||||
|
||||
// Registry joins providers into one namespace.
|
||||
type Registry struct{ providers []Provider }
|
||||
|
||||
func NewRegistry(p ...Provider) *Registry { return &Registry{providers: p} }
|
||||
|
||||
func (r *Registry) Add(p Provider) { r.providers = append(r.providers, p) }
|
||||
|
||||
// All returns every sensor, sorted by ID. Two sensors that would share an ID (two
|
||||
// identical chips) get "_2", "_3" on the colliding one, in the provider's order.
|
||||
func (r *Registry) All() []Sensor {
|
||||
var out []Sensor
|
||||
seen := map[string]int{}
|
||||
for _, p := range r.providers {
|
||||
for _, s := range p.Sensors() {
|
||||
seen[s.ID]++
|
||||
if n := seen[s.ID]; n > 1 {
|
||||
s.ID = fmt.Sprintf("%s_%d", s.ID, n)
|
||||
}
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||
return out
|
||||
}
|
||||
|
||||
// History is one sensor's recent past. ok is false when nothing remembers it.
|
||||
func (r *Registry) History(id string) (Series, bool) {
|
||||
got := r.Histories(func(x string) bool { return x == id })
|
||||
if len(got) == 0 {
|
||||
return Series{}, false
|
||||
}
|
||||
return got[0], true
|
||||
}
|
||||
|
||||
// Histories is the recent past of every numeric sensor match accepts, sorted by ID.
|
||||
func (r *Registry) Histories(match func(id string) bool) []Series {
|
||||
out := []Series{}
|
||||
for _, p := range r.providers {
|
||||
if h, ok := p.(Historian); ok {
|
||||
out = append(out, h.SensorHistories(match)...)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||
return out
|
||||
}
|
||||
|
||||
// Matcher turns comma-separated selectors (IDs and globs) into a match function.
|
||||
func Matcher(selectors string) (func(id string) bool, error) {
|
||||
var sels []string
|
||||
for _, sel := range strings.Split(selectors, ",") {
|
||||
if sel = strings.TrimSpace(sel); sel == "" {
|
||||
continue
|
||||
}
|
||||
if _, err := path.Match(sel, ""); err != nil {
|
||||
return nil, fmt.Errorf("bad pattern %q", sel)
|
||||
}
|
||||
sels = append(sels, sel)
|
||||
}
|
||||
return func(id string) bool {
|
||||
for _, sel := range sels {
|
||||
if ok, _ := path.Match(sel, id); ok {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}, nil
|
||||
}
|
||||
|
||||
// IsGlob reports whether a selector names many sensors rather than one.
|
||||
func IsGlob(sel string) bool { return strings.ContainsAny(sel, "*?[") }
|
||||
|
||||
// Select returns the sensors a selector names: an exact ID, or a glob where * crosses
|
||||
// dots ("fan.*" is every fan, "cpu.core.*.usage" every core).
|
||||
func Select(all []Sensor, sel string) ([]Sensor, error) {
|
||||
if !IsGlob(sel) {
|
||||
for _, s := range all {
|
||||
if s.ID == sel {
|
||||
return []Sensor{s}, nil
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
if _, err := path.Match(sel, ""); err != nil {
|
||||
return nil, fmt.Errorf("bad pattern %q", sel)
|
||||
}
|
||||
var out []Sensor
|
||||
for _, s := range all {
|
||||
if ok, _ := path.Match(sel, s.ID); ok {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Shape answers a shaped query: the client sends the object it wants, with selectors
|
||||
// where the values go, and gets the same object back with them filled in.
|
||||
//
|
||||
// {"cpu": "cpu.usage", "fans": "fan.*", "box": {"up": "uptime"}}
|
||||
// → {"cpu": 12.4, "fans": {"fan.nct6775.fan1": 820}, "box": {"up": 80520}}
|
||||
//
|
||||
// An exact ID becomes its value (null if there is no such sensor here — machines differ,
|
||||
// and a dashboard shouldn't fail because one has no fans). A glob becomes an object keyed
|
||||
// by ID. Objects and arrays are walked. With full, every value is the whole Sensor.
|
||||
func Shape(all []Sensor, query json.RawMessage, full bool) (any, error) {
|
||||
var q any
|
||||
if err := json.Unmarshal(query, &q); err != nil {
|
||||
return nil, errors.New("query is not valid JSON")
|
||||
}
|
||||
budget := MaxValues
|
||||
return shape(all, q, full, 0, &budget)
|
||||
}
|
||||
|
||||
// MaxValues is how many sensor values one query may return. A query is a few bytes that
|
||||
// can ask for everything many times over ({"a":"*","b":"*",…}); without a ceiling 10 KB of
|
||||
// request buys 27 MB of response. Ten thousand is thirty times a whole machine.
|
||||
const MaxValues = 10000
|
||||
|
||||
func shape(all []Sensor, q any, full bool, depth int, budget *int) (any, error) {
|
||||
if depth > 16 {
|
||||
return nil, errors.New("query nested too deep")
|
||||
}
|
||||
leaf := func(s Sensor) any {
|
||||
if full {
|
||||
return s
|
||||
}
|
||||
return s.Value
|
||||
}
|
||||
switch v := q.(type) {
|
||||
case string:
|
||||
got, err := Select(all, v)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !IsGlob(v) {
|
||||
if len(got) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
*budget--
|
||||
return leaf(got[0]), nil
|
||||
}
|
||||
if *budget -= len(got); *budget < 0 {
|
||||
return nil, fmt.Errorf("query returns more than %d values; ask for less", MaxValues)
|
||||
}
|
||||
m := make(map[string]any, len(got))
|
||||
for _, s := range got {
|
||||
m[s.ID] = leaf(s)
|
||||
}
|
||||
return m, nil
|
||||
case map[string]any:
|
||||
m := make(map[string]any, len(v))
|
||||
for k, sub := range v {
|
||||
r, err := shape(all, sub, full, depth+1, budget)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
m[k] = r
|
||||
}
|
||||
return m, nil
|
||||
case []any:
|
||||
a := make([]any, len(v))
|
||||
for i, sub := range v {
|
||||
r, err := shape(all, sub, full, depth+1, budget)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a[i] = r
|
||||
}
|
||||
return a, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("a query holds sensor IDs, objects and arrays; got %v", v)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package sensors
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type fake []Sensor
|
||||
|
||||
func (f fake) Sensors() []Sensor { return f }
|
||||
|
||||
func TestSeg(t *testing.T) {
|
||||
for in, want := range map[string]string{"/": "root", "/mnt/tank": "mnt_tank", "Package id 0": "package_id_0", "enp3s0": "enp3s0", " ": "unnamed", "Tctl+": "tctl", "nvme0n1": "nvme0n1"} {
|
||||
if got := Seg(in); got != want {
|
||||
t.Errorf("Seg(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryDedupes(t *testing.T) {
|
||||
r := NewRegistry(fake{{ID: "temp.nvme.composite", Value: 40.0}}, fake{{ID: "temp.nvme.composite", Value: 50.0}})
|
||||
all := r.All()
|
||||
if len(all) != 2 || all[0].ID != "temp.nvme.composite" || all[1].ID != "temp.nvme.composite_2" {
|
||||
t.Fatalf("got %+v", all)
|
||||
}
|
||||
}
|
||||
|
||||
func TestShape(t *testing.T) {
|
||||
all := []Sensor{{ID: "cpu.usage", Value: 12.5, Unit: "%"}, {ID: "fan.a.fan1", Value: 800.0}, {ID: "fan.a.fan2", Value: 900.0}, {ID: "net.eth0.up", Value: true}}
|
||||
got, err := Shape(all, json.RawMessage(`{"cpu":"cpu.usage","fans":"fan.*","box":{"link":"net.eth0.up","gone":"ups.battery"},"list":["cpu.usage"]}`), false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[string]any{
|
||||
"cpu": 12.5,
|
||||
"fans": map[string]any{"fan.a.fan1": 800.0, "fan.a.fan2": 900.0},
|
||||
"box": map[string]any{"link": true, "gone": nil},
|
||||
"list": []any{12.5},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("got %#v\nwant %#v", got, want)
|
||||
}
|
||||
full, _ := Shape(all, json.RawMessage(`"cpu.usage"`), true)
|
||||
if s, ok := full.(Sensor); !ok || s.Unit != "%" {
|
||||
t.Fatalf("full: got %#v", full)
|
||||
}
|
||||
for _, bad := range []string{`{"x": 5}`, `{"x": "fan.["}`, `not json`} {
|
||||
if _, err := Shape(all, json.RawMessage(bad), false); err == nil {
|
||||
t.Errorf("Shape(%s) should fail", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteProm(t *testing.T) {
|
||||
var b strings.Builder
|
||||
WriteProm(&b, []Sensor{
|
||||
{ID: "cpu.usage", Kind: "usage", Unit: "%", Value: 12.5},
|
||||
{ID: "ups.runtime", Kind: "duration", Unit: "min", Value: 2.0},
|
||||
{ID: "net.eth0.up", Kind: "state", Value: true, Labels: map[string]string{"iface": "eth0"}},
|
||||
{ID: "ups.state", Kind: "state", Value: `Nor"mal`},
|
||||
{ID: "updates.pending", Kind: "count", Value: 3.0},
|
||||
})
|
||||
got := b.String()
|
||||
for _, want := range []string{
|
||||
"# TYPE hamlab_usage_percent gauge\n" + `hamlab_usage_percent{id="cpu.usage"} 12.5`,
|
||||
`hamlab_duration_seconds{id="ups.runtime"} 120`,
|
||||
`hamlab_state{id="net.eth0.up",iface="eth0"} 1`,
|
||||
`hamlab_state{id="ups.state",value="Nor\"mal"} 1`,
|
||||
`hamlab_updates_pending{id="updates.pending"} 3`,
|
||||
} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("missing %q in:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestShapeBudget(t *testing.T) {
|
||||
all := make([]Sensor, 300)
|
||||
for i := range all {
|
||||
all[i] = Sensor{ID: "s." + strings.Repeat("x", i%7) + string(rune('a'+i%26)) + strings.Repeat("y", i/26), Value: 1.0}
|
||||
}
|
||||
q := map[string]string{}
|
||||
for i := 0; i < 40; i++ { // 40 × 300 = 12 000 values
|
||||
q["k"+strings.Repeat("z", i)] = "*"
|
||||
}
|
||||
b, _ := json.Marshal(q)
|
||||
if _, err := Shape(all, b, false); err == nil {
|
||||
t.Fatal("a query returning more than MaxValues must be refused")
|
||||
}
|
||||
if _, err := Shape(all, json.RawMessage(`{"a":"*","b":"*"}`), false); err != nil {
|
||||
t.Fatalf("an ordinary query must pass: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,368 @@
|
||||
// Package services wraps systemctl and journalctl for the Services tab: list units,
|
||||
// show one, read its unit file, read and tail its journal, and start/stop/etc.
|
||||
//
|
||||
// Reads run as the daemon's user (systemd exposes state to everyone; the journal needs
|
||||
// the systemd-journal or adm group, or sudo). Actions go through passwordless sudo when
|
||||
// it's available, otherwise plain systemctl so polkit gets its say.
|
||||
package services
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
type Unit struct {
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
Load string `json:"load"`
|
||||
Active string `json:"active"`
|
||||
Sub string `json:"sub"`
|
||||
Enabled string `json:"enabled"` // enabled | disabled | static | masked | … | "" (unknown)
|
||||
Failed bool `json:"failed"`
|
||||
}
|
||||
|
||||
type Detail struct {
|
||||
Unit
|
||||
Props map[string]string `json:"props"`
|
||||
}
|
||||
|
||||
type Entry struct {
|
||||
T int64 `json:"t"` // unix ms
|
||||
Priority int `json:"p"` // syslog priority 0-7
|
||||
Ident string `json:"ident"`
|
||||
PID string `json:"pid,omitempty"`
|
||||
Msg string `json:"msg"`
|
||||
}
|
||||
|
||||
var unitRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9@._:\\-]{0,255}\.service$`)
|
||||
|
||||
func Valid(unit string) bool { return unitRe.MatchString(unit) }
|
||||
|
||||
var Actions = map[string]bool{"start": true, "stop": true, "restart": true, "reload": true, "enable": true, "disable": true, "reset-failed": true}
|
||||
|
||||
// Properties shown on the detail page, in display order.
|
||||
var showProps = []string{"Id", "Description", "LoadState", "ActiveState", "SubState", "UnitFileState", "FragmentPath",
|
||||
"MainPID", "ExecMainStartTimestamp", "ActiveEnterTimestamp", "InactiveEnterTimestamp", "NRestarts", "Result",
|
||||
"MemoryCurrent", "MemoryPeak", "CPUUsageNSec", "TasksCurrent", "ExecStart", "User", "Restart"}
|
||||
|
||||
type Manager struct {
|
||||
sudo bool // actions via sudo -n
|
||||
journalSudo bool // journal needs sudo (user isn't in systemd-journal/adm)
|
||||
mu sync.Mutex
|
||||
cache []Unit
|
||||
cacheAt time.Time
|
||||
}
|
||||
|
||||
// ErrNoSystemd is what the Services tab says on OpenRC, runit and friends.
|
||||
var ErrNoSystemd = errors.New("this host doesn't run systemd, which the Services tab needs")
|
||||
|
||||
// HasSystemd reports whether systemd is the running init (the test sd_booted() uses).
|
||||
func HasSystemd() bool {
|
||||
_, err := os.Stat("/run/systemd/system")
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func New() *Manager {
|
||||
m := &Manager{}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
m.sudo = priv.Available()
|
||||
// Prefer reading the journal as ourselves: a plain child can be killed cleanly when a
|
||||
// tail ends, whereas a root journalctl under sudo can only be reached with sudo kill.
|
||||
var errb bytes.Buffer
|
||||
probe := exec.CommandContext(ctx, "journalctl", "--system", "-n", "1", "-q", "-o", "cat")
|
||||
probe.Stderr = &errb
|
||||
if err := probe.Run(); err != nil || strings.Contains(strings.ToLower(errb.String()), "permission") {
|
||||
m.journalSudo = m.sudo
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Sudo reports whether actions will run through passwordless sudo.
|
||||
func (m *Manager) Sudo() bool { return m.sudo }
|
||||
|
||||
// JournalSudo reports whether reading the system journal needs sudo for this user.
|
||||
func (m *Manager) JournalSudo() bool { return m.journalSudo }
|
||||
|
||||
func run(ctx context.Context, args ...string) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
var out, errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := strings.TrimSpace(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return out.String(), errors.New(msg)
|
||||
}
|
||||
return out.String(), nil
|
||||
}
|
||||
|
||||
// List merges `list-units --all` (runtime state) with `list-unit-files` (enablement).
|
||||
// Cached for a few seconds because the UI polls it.
|
||||
func (m *Manager) List(ctx context.Context) ([]Unit, error) {
|
||||
if !HasSystemd() {
|
||||
return nil, ErrNoSystemd
|
||||
}
|
||||
m.mu.Lock()
|
||||
if time.Since(m.cacheAt) < 200*time.Millisecond && m.cache != nil { // coalesce simultaneous clients only
|
||||
c := m.cache
|
||||
m.mu.Unlock()
|
||||
return c, nil
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
var out, files string
|
||||
var err error
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
files, _ = run(ctx, "systemctl", "list-unit-files", "--type=service", "--output=json", "--no-pager")
|
||||
close(done)
|
||||
}()
|
||||
out, err = run(ctx, "systemctl", "list-units", "--type=service", "--all", "--output=json", "--no-pager", "--plain")
|
||||
<-done
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var units []listedUnit
|
||||
if err := json.Unmarshal([]byte(out), &units); err != nil {
|
||||
// systemd before 246 (RHEL 8, Ubuntu 20.04) ignores --output=json here and prints
|
||||
// the table; the columns have been the same since the beginning.
|
||||
if units = parseUnitTable(out); len(units) == 0 {
|
||||
return nil, fmt.Errorf("parse list-units: %w", err)
|
||||
}
|
||||
}
|
||||
var uf []listedFile
|
||||
if json.Unmarshal([]byte(files), &uf) != nil {
|
||||
uf = parseFileTable(files)
|
||||
}
|
||||
enabled := map[string]string{}
|
||||
for _, f := range uf {
|
||||
enabled[f.UnitFile] = f.State
|
||||
}
|
||||
|
||||
seen := map[string]bool{}
|
||||
list := make([]Unit, 0, len(units)+len(uf))
|
||||
for _, u := range units {
|
||||
if strings.Contains(u.Unit, "@.") { // uninstantiated template
|
||||
continue
|
||||
}
|
||||
seen[u.Unit] = true
|
||||
list = append(list, Unit{Name: u.Unit, Description: u.Description, Load: u.Load, Active: u.Active, Sub: u.Sub,
|
||||
Enabled: enabled[u.Unit], Failed: u.Active == "failed"})
|
||||
}
|
||||
// Unit files that exist but were never loaded this boot (disabled and never started).
|
||||
for _, f := range uf {
|
||||
if seen[f.UnitFile] || strings.Contains(f.UnitFile, "@.") || f.State == "masked" || f.State == "alias" {
|
||||
continue
|
||||
}
|
||||
list = append(list, Unit{Name: f.UnitFile, Load: "not-loaded", Active: "inactive", Sub: "dead", Enabled: f.State})
|
||||
}
|
||||
sort.Slice(list, func(i, j int) bool { return list[i].Name < list[j].Name })
|
||||
|
||||
m.mu.Lock()
|
||||
m.cache, m.cacheAt = list, time.Now()
|
||||
m.mu.Unlock()
|
||||
return list, nil
|
||||
}
|
||||
|
||||
type listedUnit struct{ Unit, Load, Active, Sub, Description string }
|
||||
|
||||
type listedFile struct {
|
||||
UnitFile string `json:"unit_file"`
|
||||
State string `json:"state"`
|
||||
}
|
||||
|
||||
// parseUnitTable reads the plain `systemctl list-units` table: UNIT LOAD ACTIVE SUB DESCRIPTION…
|
||||
func parseUnitTable(text string) []listedUnit {
|
||||
var units []listedUnit
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
f := strings.Fields(strings.TrimLeft(line, "●*× "))
|
||||
if len(f) < 4 || !strings.HasSuffix(f[0], ".service") {
|
||||
continue
|
||||
}
|
||||
units = append(units, listedUnit{Unit: f[0], Load: f[1], Active: f[2], Sub: f[3], Description: strings.Join(f[4:], " ")})
|
||||
}
|
||||
return units
|
||||
}
|
||||
|
||||
// parseFileTable reads the plain `systemctl list-unit-files` table: UNIT FILE, STATE[, PRESET]
|
||||
func parseFileTable(text string) []listedFile {
|
||||
var uf []listedFile
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
if f := strings.Fields(line); len(f) >= 2 && strings.HasSuffix(f[0], ".service") {
|
||||
uf = append(uf, listedFile{UnitFile: f[0], State: f[1]})
|
||||
}
|
||||
}
|
||||
return uf
|
||||
}
|
||||
|
||||
func (m *Manager) invalidate() {
|
||||
m.mu.Lock()
|
||||
m.cacheAt = time.Time{}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
func (m *Manager) Show(ctx context.Context, unit string) (Detail, error) {
|
||||
if !Valid(unit) {
|
||||
return Detail{}, errors.New("invalid unit name")
|
||||
}
|
||||
out, err := run(ctx, "systemctl", "show", unit, "-p", strings.Join(showProps, ","), "--no-pager")
|
||||
if err != nil {
|
||||
return Detail{}, err
|
||||
}
|
||||
props := map[string]string{}
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if k, v, ok := strings.Cut(line, "="); ok && v != "" && v != "[not set]" && v != "infinity" {
|
||||
props[k] = v
|
||||
}
|
||||
}
|
||||
if props["LoadState"] == "not-found" {
|
||||
return Detail{}, errors.New("no such unit")
|
||||
}
|
||||
d := Detail{Props: props}
|
||||
d.Name, d.Description = unit, props["Description"]
|
||||
d.Load, d.Active, d.Sub, d.Enabled = props["LoadState"], props["ActiveState"], props["SubState"], props["UnitFileState"]
|
||||
d.Failed = d.Active == "failed"
|
||||
return d, nil
|
||||
}
|
||||
|
||||
func (m *Manager) Cat(ctx context.Context, unit string) (string, error) {
|
||||
if !Valid(unit) {
|
||||
return "", errors.New("invalid unit name")
|
||||
}
|
||||
return run(ctx, "systemctl", "cat", unit, "--no-pager")
|
||||
}
|
||||
|
||||
func parseEntry(line []byte) (Entry, bool) {
|
||||
var raw map[string]any
|
||||
if json.Unmarshal(line, &raw) != nil {
|
||||
return Entry{}, false
|
||||
}
|
||||
e := Entry{Priority: 6}
|
||||
switch v := raw["MESSAGE"].(type) {
|
||||
case string:
|
||||
e.Msg = v
|
||||
case []any: // non-UTF-8 payload comes as a byte array
|
||||
b := make([]byte, 0, len(v))
|
||||
for _, x := range v {
|
||||
if f, ok := x.(float64); ok {
|
||||
b = append(b, byte(f))
|
||||
}
|
||||
}
|
||||
e.Msg = string(b)
|
||||
default:
|
||||
e.Msg = ""
|
||||
}
|
||||
if p, ok := raw["PRIORITY"].(string); ok {
|
||||
e.Priority, _ = strconv.Atoi(p)
|
||||
}
|
||||
if ts, ok := raw["__REALTIME_TIMESTAMP"].(string); ok {
|
||||
us, _ := strconv.ParseInt(ts, 10, 64)
|
||||
e.T = us / 1000
|
||||
}
|
||||
if id, ok := raw["SYSLOG_IDENTIFIER"].(string); ok {
|
||||
e.Ident = id
|
||||
} else if c, ok := raw["_COMM"].(string); ok {
|
||||
e.Ident = c
|
||||
}
|
||||
if pid, ok := raw["_PID"].(string); ok {
|
||||
e.PID = pid
|
||||
}
|
||||
return e, true
|
||||
}
|
||||
|
||||
func (m *Manager) journalArgs(unit string, n int) []string {
|
||||
args := []string{"journalctl", "-u", unit, "-n", strconv.Itoa(n), "-o", "json", "--no-pager", "--all"}
|
||||
if m.journalSudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
// Journal returns the last n entries.
|
||||
func (m *Manager) Journal(ctx context.Context, unit string, n int) ([]Entry, error) {
|
||||
if !Valid(unit) {
|
||||
return nil, errors.New("invalid unit name")
|
||||
}
|
||||
out, err := run(ctx, m.journalArgs(unit, n)...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries := []Entry{}
|
||||
for _, line := range bytes.Split([]byte(out), []byte("\n")) {
|
||||
if e, ok := parseEntry(line); ok {
|
||||
entries = append(entries, e)
|
||||
}
|
||||
}
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
// Tail streams entries (last n, then follow) to fn until ctx is cancelled.
|
||||
func (m *Manager) Tail(ctx context.Context, unit string, n int, fn func(Entry)) error {
|
||||
if !Valid(unit) {
|
||||
return errors.New("invalid unit name")
|
||||
}
|
||||
args := append(m.journalArgs(unit, n), "-f")
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
// Own process group so the kill reaches journalctl itself, not just a sudo wrapper.
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error {
|
||||
pg := "-" + strconv.Itoa(cmd.Process.Pid)
|
||||
if m.journalSudo { // children are root's; only sudo can signal them
|
||||
k := priv.Wrap("kill", "-9", "--", pg)
|
||||
return exec.Command(k[0], k[1:]...).Run()
|
||||
}
|
||||
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||
}
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
}
|
||||
sc := bufio.NewScanner(stdout)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
|
||||
for sc.Scan() {
|
||||
if e, ok := parseEntry(sc.Bytes()); ok {
|
||||
fn(e)
|
||||
}
|
||||
}
|
||||
_ = cmd.Wait()
|
||||
return nil
|
||||
}
|
||||
|
||||
// Action runs start/stop/restart/reload/enable/disable/reset-failed.
|
||||
func (m *Manager) Action(ctx context.Context, action, unit string) error {
|
||||
if !Valid(unit) {
|
||||
return errors.New("invalid unit name")
|
||||
}
|
||||
if !Actions[action] {
|
||||
return errors.New("unknown action")
|
||||
}
|
||||
args := []string{"systemctl", action, unit, "--no-pager"}
|
||||
if m.sudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
_, err := run(ctx, args...)
|
||||
m.invalidate()
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
package services
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestTables(t *testing.T) {
|
||||
u := parseUnitTable(" UNIT LOAD ACTIVE SUB DESCRIPTION\n● bad.service loaded failed failed Bad thing here\n ssh.service loaded active running OpenBSD Secure Shell server\n\nLOAD = x\n12 loaded units listed.\n")
|
||||
if len(u) != 2 || u[0].Unit != "bad.service" || u[0].Active != "failed" || u[1].Description != "OpenBSD Secure Shell server" {
|
||||
t.Fatalf("%+v", u)
|
||||
}
|
||||
f := parseFileTable("UNIT FILE STATE VENDOR PRESET\nssh.service enabled enabled\nfoo@.service static -\n\n2 unit files listed.\n")
|
||||
if len(f) != 2 || f[0].State != "enabled" {
|
||||
t.Fatalf("%+v", f)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
#!/sbin/openrc-run
|
||||
# Ham Lab web console (written by hamlab install)
|
||||
|
||||
name="hamlab"
|
||||
description="Ham Lab web console"
|
||||
|
||||
export HAMLAB_CONFIG="@CONFIG@"
|
||||
export HOME="@HOME@"
|
||||
export USER="@USER@"
|
||||
|
||||
supervisor="supervise-daemon"
|
||||
command="@BIN@"
|
||||
command_args="serve"
|
||||
command_user="@USER@:@GROUP@"
|
||||
directory="@HOME@"
|
||||
respawn_delay=3
|
||||
respawn_max=0
|
||||
output_log="/var/log/hamlab.log"
|
||||
error_log="/var/log/hamlab.log"
|
||||
|
||||
depend() {
|
||||
need localmount
|
||||
after net firewall
|
||||
}
|
||||
|
||||
start_pre() {
|
||||
checkpath -f -o "@USER@:@GROUP@" -m 0640 /var/log/hamlab.log
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
#!/bin/sh /etc/rc.common
|
||||
# Ham Lab web console (written by hamlab install)
|
||||
|
||||
USE_PROCD=1
|
||||
# after network and firewall, before the tail of the boot
|
||||
START=95
|
||||
STOP=10
|
||||
|
||||
start_service() {
|
||||
procd_open_instance hamlab
|
||||
procd_set_param command "@BIN@" serve
|
||||
procd_set_param env HAMLAB_CONFIG="@CONFIG@" HOME="@HOME@" USER="@USER@"
|
||||
procd_set_param user "@USER@"
|
||||
procd_set_param group "@GROUP@"
|
||||
procd_set_param pidfile /var/run/hamlab.pid
|
||||
# respawn: settle 3s, 5 retries inside that window, then give up counting
|
||||
procd_set_param respawn 3600 3 0
|
||||
procd_set_param stdout 1
|
||||
procd_set_param stderr 1
|
||||
procd_close_instance
|
||||
}
|
||||
|
||||
service_triggers() {
|
||||
procd_add_reload_trigger network
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
[Unit]
|
||||
Description=Ham Lab web console
|
||||
After=network.target
|
||||
# never give up restarting
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=@USER@
|
||||
Group=@GROUP@
|
||||
# read the system journal without sudo (keeps the daemon's own log clean)
|
||||
SupplementaryGroups=@GROUPS@
|
||||
WorkingDirectory=-@HOME@
|
||||
Environment=HAMLAB_CONFIG=@CONFIG@
|
||||
ExecStart=@BIN@ serve
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
KillMode=control-group
|
||||
TimeoutStopSec=10
|
||||
LimitNOFILE=65536
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,718 @@
|
||||
// Package setup is `hamlab install` and `hamlab uninstall`: the binary installs itself, so
|
||||
// a new machine needs nothing but this one file. It copies itself into place, writes the
|
||||
// config, sets the login password and registers a service with whichever init the host
|
||||
// runs (systemd or OpenRC), as root or through sudo/doas.
|
||||
package setup
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"crypto/rand"
|
||||
"crypto/tls"
|
||||
_ "embed"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"golang.org/x/term"
|
||||
|
||||
"hamlab/internal/auth"
|
||||
"hamlab/internal/config"
|
||||
)
|
||||
|
||||
//go:embed hamlab.service
|
||||
var systemdUnit string
|
||||
|
||||
//go:embed hamlab.openrc
|
||||
var openrcScript string
|
||||
|
||||
//go:embed hamlab.procd
|
||||
var procdScript string
|
||||
|
||||
const (
|
||||
unitPath = "/etc/systemd/system/hamlab.service"
|
||||
openrcPath = "/etc/init.d/hamlab"
|
||||
// procd (OpenWrt) uses the same path as OpenRC; the two are told apart by content.
|
||||
procdPath = "/etc/init.d/hamlab"
|
||||
sudoersPath = "/etc/sudoers.d/hamlab-quiet"
|
||||
)
|
||||
|
||||
type options struct {
|
||||
user, listen, tls, prefix, data string
|
||||
passwordStdin, openFirewall bool
|
||||
noStart bool
|
||||
}
|
||||
|
||||
func say(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) }
|
||||
|
||||
// elevate re-runs this command as root through sudo or doas (which may prompt), naming
|
||||
// the invoking user so the service still runs as them.
|
||||
func elevate(sub string, args []string, keepUser bool) error {
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, tool := range []string{"sudo", "doas"} {
|
||||
path, err := exec.LookPath(tool)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
full := []string{tool, self, sub}
|
||||
if u, err := user.Current(); err == nil && keepUser && !hasFlag(args, "user") {
|
||||
full = append(full, "--user", u.Username)
|
||||
}
|
||||
full = append(full, args...)
|
||||
return syscall.Exec(path, full, os.Environ())
|
||||
}
|
||||
return errors.New("this needs root, and neither sudo nor doas is installed: run it as root")
|
||||
}
|
||||
|
||||
func hasFlag(args []string, name string) bool {
|
||||
for _, a := range args {
|
||||
a = strings.TrimLeft(a, "-")
|
||||
if a == name || strings.HasPrefix(a, name+"=") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Install is `hamlab install [flags]`.
|
||||
func Install(args []string, version string) error {
|
||||
var o options
|
||||
fs := flag.NewFlagSet("install", flag.ContinueOnError)
|
||||
fs.StringVar(&o.user, "user", "", "user the console runs as; terminals are this user's shell (default: you)")
|
||||
fs.StringVar(&o.listen, "listen", "", "listen address, e.g. 0.0.0.0:8100 (default: keep, else 127.0.0.1:8100)")
|
||||
fs.StringVar(&o.tls, "tls", "", `"auto" serves HTTPS with a self-signed certificate, "off" plain HTTP (default: keep)`)
|
||||
fs.StringVar(&o.prefix, "prefix", "/usr/local/bin", "directory the binary is installed into")
|
||||
fs.StringVar(&o.data, "data", "", "config and state directory (default: ~/.config/hamlab of the user)")
|
||||
fs.BoolVar(&o.passwordStdin, "password-stdin", false, "read the login password from stdin instead of prompting")
|
||||
fs.BoolVar(&o.openFirewall, "open-firewall", false, "open the listen port in firewalld/ufw when one is active")
|
||||
fs.BoolVar(&o.noStart, "no-start", false, "install and enable, but don't start the service")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if os.Geteuid() != 0 {
|
||||
return elevate("install", args, true)
|
||||
}
|
||||
sbinPath()
|
||||
|
||||
// who
|
||||
name := o.user
|
||||
for _, env := range []string{"SUDO_USER", "DOAS_USER"} {
|
||||
if name == "" {
|
||||
name = os.Getenv(env)
|
||||
}
|
||||
}
|
||||
if name == "" {
|
||||
name = "root"
|
||||
}
|
||||
acct, err := lookupUser(name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// where
|
||||
dataDir := o.data
|
||||
if dataDir == "" {
|
||||
dataDir = filepath.Dir(existingConfig(acct.home))
|
||||
}
|
||||
cfgPath := filepath.Join(dataDir, "config.json")
|
||||
|
||||
// binary
|
||||
dest := filepath.Join(o.prefix, "hamlab")
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
self, _ = filepath.EvalSymlinks(self)
|
||||
if err := os.MkdirAll(o.prefix, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if self != dest {
|
||||
if err := CopyAtomic(self, dest); err != nil {
|
||||
return fmt.Errorf("install binary: %w", err)
|
||||
}
|
||||
// builds for other architectures sit beside it so a hub can push them to matching machines
|
||||
sibs, _ := filepath.Glob(filepath.Join(filepath.Dir(self), "hamlab-linux-*"))
|
||||
for _, sib := range sibs {
|
||||
arch := strings.TrimPrefix(filepath.Base(sib), "hamlab-linux-")
|
||||
if sib == self || strings.Contains(arch, ".") {
|
||||
continue
|
||||
}
|
||||
if err := CopyAtomic(sib, dest+"-"+arch); err != nil {
|
||||
say("warning: %s: %v", sib, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
restorecon(dest)
|
||||
say("binary %s (%s)", dest, version)
|
||||
|
||||
// config + password
|
||||
cfg, err := config.Load(cfgPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.listen != "" {
|
||||
if _, _, err := net.SplitHostPort(o.listen); err != nil {
|
||||
return fmt.Errorf("--listen %q: want host:port", o.listen)
|
||||
}
|
||||
cfg.Listen = o.listen
|
||||
}
|
||||
switch o.tls {
|
||||
case "":
|
||||
case "auto":
|
||||
cfg.TLS = "auto"
|
||||
case "off", "none":
|
||||
cfg.TLS = ""
|
||||
default:
|
||||
return fmt.Errorf("--tls %q: want auto or off", o.tls)
|
||||
}
|
||||
generated := ""
|
||||
cfg.MigrateUsers(acct.name)
|
||||
if len(cfg.Users) == 0 || o.passwordStdin {
|
||||
pw, gen, err := choosePassword(o.passwordStdin)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h, err := auth.HashPassword(pw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// the first account is named after the OS user the service runs as; --password-stdin
|
||||
// on an existing install resets that first account
|
||||
if len(cfg.Users) == 0 {
|
||||
cfg.Users = []config.User{{Name: acct.name, Hash: h, Created: time.Now()}}
|
||||
} else {
|
||||
cfg.Users[0].Hash = h
|
||||
}
|
||||
if gen {
|
||||
generated = pw
|
||||
}
|
||||
}
|
||||
made := firstMissing(dataDir) // e.g. ~/.config, when this is the first thing to use it
|
||||
if err := cfg.Save(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := chownData(dataDir, made, acct); err != nil {
|
||||
return err
|
||||
}
|
||||
say("config %s", cfgPath)
|
||||
|
||||
// service
|
||||
vars := map[string]string{"@USER@": acct.name, "@GROUP@": acct.group, "@HOME@": acct.home, "@BIN@": dest, "@CONFIG@": cfgPath,
|
||||
"@GROUPS@": journalGroups()}
|
||||
initSys := ""
|
||||
switch {
|
||||
case isDir("/run/systemd/system"):
|
||||
initSys = "systemd"
|
||||
unit := fill(systemdUnit, vars)
|
||||
if vars["@GROUPS@"] == "" {
|
||||
unit = dropLines(unit, "SupplementaryGroups=")
|
||||
}
|
||||
if err := os.WriteFile(unitPath, []byte(unit), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
restorecon(unitPath)
|
||||
if err := runAll([][]string{{"systemctl", "daemon-reload"}, {"systemctl", "enable", "hamlab.service"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
if !o.noStart {
|
||||
if err := run("systemctl", "restart", "hamlab.service"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
case exists("/sbin/procd") && exists("/etc/rc.common"):
|
||||
initSys = "procd"
|
||||
if err := os.WriteFile(procdPath, []byte(fill(procdScript, vars)), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := run(procdPath, "enable"); err != nil {
|
||||
say("warning: %s enable: %v", procdPath, err)
|
||||
}
|
||||
if !o.noStart {
|
||||
if err := run(procdPath, "restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
case exists("/sbin/openrc-run") || exists("/usr/sbin/openrc-run") || have("rc-service"):
|
||||
initSys = "OpenRC"
|
||||
if err := os.WriteFile(openrcPath, []byte(fill(openrcScript, vars)), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := run("rc-update", "add", "hamlab", "default"); err != nil {
|
||||
say("warning: rc-update: %v", err)
|
||||
}
|
||||
if !o.noStart {
|
||||
if err := run("rc-service", "hamlab", "restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if initSys != "" {
|
||||
say("service hamlab (%s), runs as %s, starts at boot", initSys, acct.name)
|
||||
}
|
||||
if acct.name != "root" {
|
||||
quietSudoers()
|
||||
}
|
||||
|
||||
port := firewall(cfg.Listen, o.openFirewall)
|
||||
|
||||
scheme := "http"
|
||||
if cfg.TLS == "auto" {
|
||||
scheme = "https"
|
||||
}
|
||||
url := scheme + "://" + browseAddr(cfg.Listen)
|
||||
switch {
|
||||
case initSys == "":
|
||||
say("\nNo systemd, procd or OpenRC here, so nothing starts it at boot.\nRun it under your own supervisor, as %s:\n HAMLAB_CONFIG=%s %s serve", acct.name, cfgPath, dest)
|
||||
case o.noStart:
|
||||
case healthy(cfg.Listen, cfg.TLS == "auto"):
|
||||
say("\nHam Lab is up: %s", url)
|
||||
default:
|
||||
say("\nInstalled, but %s isn't answering yet. Check: %s", url, map[string]string{"systemd": "journalctl -u hamlab -n 30", "OpenRC": "tail /var/log/hamlab.log"}[initSys])
|
||||
}
|
||||
if generated != "" {
|
||||
say("Login: user %s, password (shown once; change it in Settings): %s", cfg.Users[0].Name, generated)
|
||||
}
|
||||
if host, _, _ := net.SplitHostPort(cfg.Listen); isLoopback(host) {
|
||||
say("It only listens on this machine. For LAN access: hamlab install --listen 0.0.0.0:%s", port)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Uninstall is `hamlab uninstall [--purge]`.
|
||||
func Uninstall(args []string) error {
|
||||
fs := flag.NewFlagSet("uninstall", flag.ContinueOnError)
|
||||
purge := fs.Bool("purge", false, "also delete the config and state directory")
|
||||
prefix := fs.String("prefix", "/usr/local/bin", "directory the binary was installed into")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if os.Geteuid() != 0 {
|
||||
return elevate("uninstall", args, false)
|
||||
}
|
||||
sbinPath()
|
||||
dataDir := ""
|
||||
if b, err := os.ReadFile(unitPath); err == nil {
|
||||
dataDir = configDirFrom(string(b), "Environment=HAMLAB_CONFIG=")
|
||||
_ = run("systemctl", "disable", "--now", "hamlab.service")
|
||||
os.Remove(unitPath)
|
||||
_ = run("systemctl", "daemon-reload")
|
||||
}
|
||||
if b, err := os.ReadFile(openrcPath); err == nil {
|
||||
if strings.Contains(string(b), "USE_PROCD=1") {
|
||||
dataDir = configDirFrom(string(b), "HAMLAB_CONFIG=")
|
||||
_ = run(procdPath, "stop")
|
||||
_ = run(procdPath, "disable")
|
||||
} else {
|
||||
dataDir = configDirFrom(string(b), "export HAMLAB_CONFIG=")
|
||||
_ = run("rc-service", "hamlab", "stop")
|
||||
_ = run("rc-update", "del", "hamlab", "default")
|
||||
}
|
||||
os.Remove(openrcPath)
|
||||
}
|
||||
os.Remove(sudoersPath)
|
||||
bins, _ := filepath.Glob(filepath.Join(*prefix, "hamlab-*"))
|
||||
for _, b := range append(bins, filepath.Join(*prefix, "hamlab")) {
|
||||
os.Remove(b)
|
||||
}
|
||||
say("Ham Lab removed.")
|
||||
switch {
|
||||
case dataDir == "":
|
||||
case *purge:
|
||||
if err := purgeData(dataDir); err != nil {
|
||||
return err
|
||||
}
|
||||
default:
|
||||
say("Config and state kept in %s (uninstall --purge deletes them).", dataDir)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// sbinPath makes sure the admin tools resolve: a root shell reached with plain `su` keeps
|
||||
// the user's PATH, which on Debian has no sbin in it.
|
||||
func sbinPath() {
|
||||
os.Setenv("PATH", os.Getenv("PATH")+":/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin")
|
||||
}
|
||||
|
||||
// ownFiles is everything Ham Lab writes into its data directory.
|
||||
var ownFiles = []string{"config.json", "config.json.tmp", "state.json", "state.json.tmp", "state.json.corrupt", "cert.pem", "key.pem", "hamlab.prev", "hamlab.new"}
|
||||
|
||||
// purgeData deletes Ham Lab's data. The whole directory goes only when it is unmistakably
|
||||
// ours (named "hamlab"); a --data pointed at a shared directory such as a home loses
|
||||
// just the files Ham Lab wrote.
|
||||
func purgeData(dir string) error {
|
||||
if filepath.Base(dir) == "hamlab" {
|
||||
say("Deleted %s.", dir)
|
||||
return os.RemoveAll(dir)
|
||||
}
|
||||
for _, f := range ownFiles {
|
||||
os.Remove(filepath.Join(dir, f))
|
||||
}
|
||||
say("Deleted Ham Lab's files in %s (the directory isn't Ham Lab's alone, so it stays).", dir)
|
||||
return nil
|
||||
}
|
||||
|
||||
func configDirFrom(text, key string) string {
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
if v, ok := strings.CutPrefix(strings.TrimSpace(line), key); ok {
|
||||
if v = strings.Trim(v, `"'`); strings.HasSuffix(v, "/config.json") && filepath.Dir(v) != "/" {
|
||||
return filepath.Dir(v)
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ---- pieces ----
|
||||
|
||||
type account struct {
|
||||
name, group, home string
|
||||
uid, gid int
|
||||
}
|
||||
|
||||
func lookupUser(name string) (account, error) {
|
||||
a := account{name: name}
|
||||
if u, err := user.Lookup(name); err == nil {
|
||||
a.uid, _ = strconv.Atoi(u.Uid)
|
||||
a.gid, _ = strconv.Atoi(u.Gid)
|
||||
a.home = u.HomeDir
|
||||
} else if out, err2 := exec.Command("getent", "passwd", name).Output(); err2 == nil { // LDAP/SSSD accounts
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if len(f) < 6 {
|
||||
return a, fmt.Errorf("no such user: %s", name)
|
||||
}
|
||||
a.uid, _ = strconv.Atoi(f[2])
|
||||
a.gid, _ = strconv.Atoi(f[3])
|
||||
a.home = f[5]
|
||||
} else {
|
||||
return a, fmt.Errorf("no such user: %s", name)
|
||||
}
|
||||
if a.home == "" || !isDir(a.home) {
|
||||
return a, fmt.Errorf("user %s has no home directory (%q); pick another with --user", name, a.home)
|
||||
}
|
||||
a.group = strconv.Itoa(a.gid)
|
||||
if g, err := user.LookupGroupId(a.group); err == nil {
|
||||
a.group = g.Name
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// existingConfig returns the config path for a home directory: whichever already exists,
|
||||
// else the default for a new install.
|
||||
func existingConfig(home string) string {
|
||||
for _, p := range config.Candidates(home) {
|
||||
if exists(p) {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return config.Candidates(home)[0]
|
||||
}
|
||||
|
||||
// journalGroups lists the groups that can read the system journal and exist here. A unit
|
||||
// naming a group the host doesn't have fails to start, and "adm" is not universal.
|
||||
func journalGroups() string {
|
||||
var have []string
|
||||
for _, g := range []string{"systemd-journal", "adm"} {
|
||||
if _, err := user.LookupGroup(g); err == nil {
|
||||
have = append(have, g)
|
||||
}
|
||||
}
|
||||
return strings.Join(have, " ")
|
||||
}
|
||||
|
||||
func choosePassword(fromStdin bool) (pw string, generated bool, err error) {
|
||||
switch {
|
||||
case fromStdin:
|
||||
line, _ := bufio.NewReader(os.Stdin).ReadString('\n')
|
||||
pw = strings.TrimRight(line, "\r\n")
|
||||
case term.IsTerminal(int(os.Stdin.Fd())):
|
||||
fmt.Fprint(os.Stderr, "Login password for the web console (12+ characters, empty to generate one): ")
|
||||
p1, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
if len(p1) > 0 {
|
||||
fmt.Fprint(os.Stderr, "Again: ")
|
||||
p2, err := term.ReadPassword(int(os.Stdin.Fd()))
|
||||
fmt.Fprintln(os.Stderr)
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
if !bytes.Equal(p1, p2) {
|
||||
return "", false, errors.New("passwords do not match")
|
||||
}
|
||||
}
|
||||
pw = string(p1)
|
||||
}
|
||||
if pw == "" && !fromStdin {
|
||||
return randomPassword(), true, nil
|
||||
}
|
||||
if len(pw) < 12 {
|
||||
return "", false, errors.New("password must be at least 12 characters")
|
||||
}
|
||||
return pw, false, nil
|
||||
}
|
||||
|
||||
func randomPassword() string {
|
||||
const alphabet = "abcdefghijkmnpqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ23456789" // no look-alikes
|
||||
b := make([]byte, 20)
|
||||
_, _ = rand.Read(b)
|
||||
for i := range b {
|
||||
b[i] = alphabet[int(b[i])%len(alphabet)]
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// CopyAtomic puts src at dst with mode 0755 by writing beside it and renaming, so dst is
|
||||
// always either the old file or the whole new one, and a running dst is never written to.
|
||||
func CopyAtomic(src, dst string) error {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer in.Close()
|
||||
tmp := filepath.Join(filepath.Dir(dst), "."+filepath.Base(dst)+".new")
|
||||
out, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = io.Copy(out, in)
|
||||
if err == nil {
|
||||
err = out.Sync()
|
||||
}
|
||||
if cerr := out.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err == nil {
|
||||
err = os.Chmod(tmp, 0o755)
|
||||
}
|
||||
if err == nil {
|
||||
err = os.Rename(tmp, dst)
|
||||
}
|
||||
if err != nil {
|
||||
os.Remove(tmp)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// firstMissing returns the topmost directory that creating dir would have to make, or "".
|
||||
func firstMissing(dir string) string {
|
||||
missing := ""
|
||||
for d := filepath.Clean(dir); d != "/" && d != "." && !exists(d); d = filepath.Dir(d) {
|
||||
missing = d
|
||||
}
|
||||
return missing
|
||||
}
|
||||
|
||||
// chownData hands Ham Lab's data to the account that runs it. Directories this install
|
||||
// created are ours to hand over whole. In one that already existed (a --data pointed at
|
||||
// something shared), only the files Ham Lab writes change owner: a recursive chown of
|
||||
// somebody else's tree can't be undone.
|
||||
func chownData(dataDir, made string, a account) error {
|
||||
if made != "" {
|
||||
return chownTree(made, a)
|
||||
}
|
||||
if filepath.Base(dataDir) == "hamlab" {
|
||||
return chownTree(dataDir, a)
|
||||
}
|
||||
for _, f := range ownFiles {
|
||||
if p := filepath.Join(dataDir, f); exists(p) {
|
||||
if err := os.Lchown(p, a.uid, a.gid); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chownTree(dir string, a account) error {
|
||||
return filepath.Walk(dir, func(p string, _ os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Lchown(p, a.uid, a.gid)
|
||||
})
|
||||
}
|
||||
|
||||
func fill(tmpl string, vars map[string]string) string {
|
||||
for k, v := range vars {
|
||||
tmpl = strings.ReplaceAll(tmpl, k, v)
|
||||
}
|
||||
return tmpl
|
||||
}
|
||||
|
||||
func dropLines(text, prefix string) string {
|
||||
var keep []string
|
||||
for _, l := range strings.Split(text, "\n") {
|
||||
if !strings.HasPrefix(l, prefix) {
|
||||
keep = append(keep, l)
|
||||
}
|
||||
}
|
||||
return strings.Join(keep, "\n")
|
||||
}
|
||||
|
||||
// quietSudoers keeps the daemon's read-only polling (SMART, UPS, journal) out of the auth
|
||||
// log: at 250 ms refresh rates the sudo session lines would otherwise dominate it.
|
||||
func quietSudoers() {
|
||||
if !isDir("/etc/sudoers.d") || !have("visudo") {
|
||||
return
|
||||
}
|
||||
var cmds []string
|
||||
for _, bin := range []string{"smartctl", "pwrstat", "journalctl", "tail", "cat", "ls", "crontab"} {
|
||||
for _, dir := range []string{"/usr/sbin", "/usr/bin", "/sbin", "/bin"} {
|
||||
if exists(dir + "/" + bin) {
|
||||
cmds = append(cmds, dir+"/"+bin)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, bin := range []string{"zpool", "zfs"} { // listing only: scrubs and snapshots still log
|
||||
for _, dir := range []string{"/usr/sbin", "/sbin", "/usr/bin", "/bin"} {
|
||||
if exists(dir + "/" + bin) {
|
||||
cmds = append(cmds, dir+"/"+bin+" list *", dir+"/"+bin+" status *")
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(cmds) == 0 {
|
||||
return
|
||||
}
|
||||
text := "# Ham Lab: don't log the daemon's read-only polling commands (written by hamlab install)\n" +
|
||||
"Cmnd_Alias HAMLAB_QUIET = " + strings.Join(cmds, ", ") + "\n" +
|
||||
"Defaults!HAMLAB_QUIET !syslog, !pam_session, !pam_setcred\n"
|
||||
tmp := sudoersPath + ".new"
|
||||
if os.WriteFile(tmp, []byte(text), 0o440) != nil {
|
||||
return
|
||||
}
|
||||
if exec.Command("visudo", "-cf", tmp).Run() != nil {
|
||||
os.Remove(tmp)
|
||||
say("warning: sudoers snippet failed validation; skipped")
|
||||
return
|
||||
}
|
||||
_ = os.Rename(tmp, sudoersPath)
|
||||
}
|
||||
|
||||
// firewall reports (and with open, fixes) an active host firewall in front of a
|
||||
// non-loopback listen address. It returns the port.
|
||||
func firewall(listen string, open bool) string {
|
||||
host, port, _ := net.SplitHostPort(listen)
|
||||
if isLoopback(host) {
|
||||
return port
|
||||
}
|
||||
switch {
|
||||
case have("firewall-cmd") && exec.Command("firewall-cmd", "--state").Run() == nil:
|
||||
if open {
|
||||
if err := runAll([][]string{{"firewall-cmd", "--quiet", "--permanent", "--add-port=" + port + "/tcp"}, {"firewall-cmd", "--quiet", "--reload"}}); err != nil {
|
||||
say("warning: firewalld: %v", err)
|
||||
} else {
|
||||
say("firewall firewalld: opened %s/tcp", port)
|
||||
}
|
||||
} else {
|
||||
say("firewall firewalld is active and will block port %s. Open it: hamlab install --open-firewall", port)
|
||||
}
|
||||
case have("ufw"):
|
||||
out, _ := exec.Command("ufw", "status").Output()
|
||||
if !bytes.Contains(out, []byte("Status: active")) {
|
||||
break
|
||||
}
|
||||
if open {
|
||||
if err := run("ufw", "allow", port+"/tcp"); err != nil {
|
||||
say("warning: ufw: %v", err)
|
||||
} else {
|
||||
say("firewall ufw: opened %s/tcp", port)
|
||||
}
|
||||
} else {
|
||||
say("firewall ufw is active and will block port %s. Open it: hamlab install --open-firewall", port)
|
||||
}
|
||||
}
|
||||
return port
|
||||
}
|
||||
|
||||
func isLoopback(host string) bool {
|
||||
ip := net.ParseIP(host)
|
||||
return host == "localhost" || (ip != nil && ip.IsLoopback())
|
||||
}
|
||||
|
||||
// browseAddr turns a listen address into one a browser can open.
|
||||
func browseAddr(listen string) string {
|
||||
host, port, err := net.SplitHostPort(listen)
|
||||
if err != nil {
|
||||
return listen
|
||||
}
|
||||
if ip := net.ParseIP(host); host == "" || (ip != nil && ip.IsUnspecified()) {
|
||||
host = "localhost"
|
||||
if addrs, err := net.InterfaceAddrs(); err == nil {
|
||||
for _, a := range addrs {
|
||||
if n, ok := a.(*net.IPNet); ok && n.IP.To4() != nil && !n.IP.IsLoopback() {
|
||||
host = n.IP.String()
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return net.JoinHostPort(host, port)
|
||||
}
|
||||
|
||||
// healthy polls /healthz for up to 10 s.
|
||||
func healthy(listen string, https bool) bool {
|
||||
host, port, _ := net.SplitHostPort(listen)
|
||||
if ip := net.ParseIP(host); host == "" || (ip != nil && ip.IsUnspecified()) {
|
||||
host = "127.0.0.1"
|
||||
}
|
||||
scheme := "http"
|
||||
if https {
|
||||
scheme = "https"
|
||||
}
|
||||
c := &http.Client{Timeout: 2 * time.Second, Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}} // self-signed; reachability only
|
||||
for i := 0; i < 20; i++ {
|
||||
if resp, err := c.Get(scheme + "://" + net.JoinHostPort(host, port) + "/healthz"); err == nil {
|
||||
resp.Body.Close()
|
||||
return resp.StatusCode < 500
|
||||
}
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// restorecon fixes the SELinux label of a file we created, where SELinux exists.
|
||||
func restorecon(path string) {
|
||||
if have("restorecon") {
|
||||
_ = exec.Command("restorecon", "-F", path).Run()
|
||||
}
|
||||
}
|
||||
|
||||
func run(args ...string) error {
|
||||
out, err := exec.Command(args[0], args[1:]...).CombinedOutput()
|
||||
if err != nil {
|
||||
if msg := strings.TrimSpace(string(out)); msg != "" {
|
||||
return fmt.Errorf("%s: %s", strings.Join(args, " "), msg)
|
||||
}
|
||||
return fmt.Errorf("%s: %w", strings.Join(args, " "), err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func runAll(cmds [][]string) error {
|
||||
for _, c := range cmds {
|
||||
if err := run(c...); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func have(bin string) bool { _, err := exec.LookPath(bin); return err == nil }
|
||||
func exists(p string) bool { _, err := os.Stat(p); return err == nil }
|
||||
func isDir(p string) bool { st, err := os.Stat(p); return err == nil && st.IsDir() }
|
||||
@@ -0,0 +1,73 @@
|
||||
package setup
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestUnitWithoutJournalGroups(t *testing.T) {
|
||||
vars := map[string]string{"@USER@": "ham", "@GROUP@": "ham", "@HOME@": "/home/ham", "@BIN@": "/usr/local/bin/hamlab",
|
||||
"@CONFIG@": "/home/ham/.config/hamlab/config.json", "@GROUPS@": ""}
|
||||
unit := dropLines(fill(systemdUnit, vars), "SupplementaryGroups=")
|
||||
if strings.Contains(unit, "SupplementaryGroups") || strings.Contains(unit, "@") {
|
||||
t.Fatalf("unit still has a groups line or an unfilled placeholder:\n%s", unit)
|
||||
}
|
||||
if !strings.Contains(unit, "ExecStart=/usr/local/bin/hamlab serve") {
|
||||
t.Fatalf("ExecStart missing:\n%s", unit)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigDirFrom(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"Environment=HAMLAB_CONFIG=/home/ham/.config/hamlab/config.json": "/home/ham/.config/hamlab",
|
||||
`export HAMLAB_CONFIG="/root/.config/hamlab/config.json"`: "/root/.config/hamlab",
|
||||
"Environment=HAMLAB_CONFIG=/config.json": "",
|
||||
"Environment=HAMLAB_CONFIG=/etc/passwd": "",
|
||||
} {
|
||||
key := "Environment=HAMLAB_CONFIG="
|
||||
if strings.HasPrefix(in, "export") {
|
||||
key = "export HAMLAB_CONFIG="
|
||||
}
|
||||
if got := configDirFrom(in, key); got != want {
|
||||
t.Errorf("%q: got %q want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --purge must never take a shared directory with it.
|
||||
func TestPurgeSparesSharedDirectory(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
os.WriteFile(filepath.Join(home, "config.json"), []byte("{}"), 0o600)
|
||||
os.WriteFile(filepath.Join(home, "thesis.txt"), []byte("irreplaceable"), 0o600)
|
||||
if err := purgeData(home); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(home, "thesis.txt")); err != nil {
|
||||
t.Fatal("purge deleted a file that wasn't Ham Lab's")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(home, "config.json")); err == nil {
|
||||
t.Fatal("purge left config.json behind")
|
||||
}
|
||||
|
||||
own := filepath.Join(t.TempDir(), "hamlab")
|
||||
os.MkdirAll(own, 0o700)
|
||||
os.WriteFile(filepath.Join(own, "config.json"), []byte("{}"), 0o600)
|
||||
if err := purgeData(own); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(own); err == nil {
|
||||
t.Fatal("purge left Ham Lab's own directory behind")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFirstMissing(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
if got := firstMissing(filepath.Join(base, "a", "b", "c")); got != filepath.Join(base, "a") {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
if got := firstMissing(base); got != "" {
|
||||
t.Errorf("existing dir: got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
package space
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"io"
|
||||
)
|
||||
|
||||
// The tree crosses from the helper process to the daemon as a preorder stream of varints:
|
||||
// a few bytes per node, decoded without reflection, so a million nodes take well under a
|
||||
// second either way.
|
||||
|
||||
const codecMagic = "HLSPACE2"
|
||||
|
||||
func Encode(w io.Writer, root *Node) error {
|
||||
bw := bufio.NewWriterSize(w, 1<<20)
|
||||
if _, err := bw.WriteString(codecMagic); err != nil {
|
||||
return err
|
||||
}
|
||||
var buf [binary.MaxVarintLen64]byte
|
||||
putU := func(v uint64) error { n := binary.PutUvarint(buf[:], v); _, err := bw.Write(buf[:n]); return err }
|
||||
putI := func(v int64) error { n := binary.PutVarint(buf[:], v); _, err := bw.Write(buf[:n]); return err }
|
||||
var enc func(n *Node) error
|
||||
enc = func(n *Node) error {
|
||||
if err := putU(uint64(len(n.Name))); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := bw.WriteString(n.Name); err != nil {
|
||||
return err
|
||||
}
|
||||
flags := uint64(0)
|
||||
if n.Dir {
|
||||
flags |= 1
|
||||
}
|
||||
if n.Mount {
|
||||
flags |= 2
|
||||
}
|
||||
if n.Denied {
|
||||
flags |= 4
|
||||
}
|
||||
if n.Partial {
|
||||
flags |= 8
|
||||
}
|
||||
for _, f := range []func() error{
|
||||
func() error { return putU(flags) },
|
||||
func() error { return putI(n.Size) },
|
||||
func() error { return putI(n.Apparent) },
|
||||
func() error { return putI(n.MTime) },
|
||||
func() error { return putI(n.MTimeNs) },
|
||||
func() error { return putU(uint64(n.Files)) },
|
||||
func() error { return putU(uint64(n.Dirs)) },
|
||||
func() error { return putU(uint64(n.SmallN)) },
|
||||
func() error { return putI(n.SmallSize) },
|
||||
func() error { return putU(uint64(len(n.Children))) },
|
||||
} {
|
||||
if err := f(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for _, c := range n.Children {
|
||||
if err := enc(c); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if err := enc(root); err != nil {
|
||||
return err
|
||||
}
|
||||
return bw.Flush()
|
||||
}
|
||||
|
||||
func Decode(r io.Reader) (*Node, error) {
|
||||
br := bufio.NewReaderSize(r, 1<<20)
|
||||
magic := make([]byte, len(codecMagic))
|
||||
if _, err := io.ReadFull(br, magic); err != nil || string(magic) != codecMagic {
|
||||
return nil, errors.New("space: not a scan stream")
|
||||
}
|
||||
var dec func() (*Node, error)
|
||||
dec = func() (*Node, error) {
|
||||
l, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if l > 1<<16 {
|
||||
return nil, errors.New("space: bad name length")
|
||||
}
|
||||
name := make([]byte, l)
|
||||
if _, err := io.ReadFull(br, name); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n := &Node{Name: string(name)}
|
||||
flags, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Dir, n.Mount, n.Denied, n.Partial = flags&1 != 0, flags&2 != 0, flags&4 != 0, flags&8 != 0
|
||||
if n.Size, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.Apparent, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.MTime, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.MTimeNs, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
u, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Files = uint32(u)
|
||||
if u, err = binary.ReadUvarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Dirs = uint32(u)
|
||||
if u, err = binary.ReadUvarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.SmallN = uint32(u)
|
||||
if n.SmallSize, err = binary.ReadVarint(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cnt, err := binary.ReadUvarint(br)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cnt > 1<<24 {
|
||||
return nil, errors.New("space: bad child count")
|
||||
}
|
||||
if cnt > 0 {
|
||||
n.Children = make([]*Node, 0, cnt)
|
||||
for i := uint64(0); i < cnt; i++ {
|
||||
c, err := dec()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
n.Children = append(n.Children, c)
|
||||
}
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
return dec()
|
||||
}
|
||||
@@ -0,0 +1,735 @@
|
||||
package space
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// ---- the helper: `hamlab space-walk <root>…`, run by the daemon (as root when it can) ----
|
||||
//
|
||||
// The walk runs in its own process so that root can read every directory, so that the
|
||||
// memory of the walk is given back when it ends, and so that a runaway scan can be killed
|
||||
// without touching the console. The daemon writes a header line to the helper's stdin
|
||||
// (how each root is to be walked, and what changed) followed by the last scan's tree when
|
||||
// there is one, then holds stdin open: EOF means the daemon has gone or wants the walk
|
||||
// stopped. Progress goes to stderr as JSON lines; a result line and the tree to stdout.
|
||||
|
||||
type header struct {
|
||||
Prev bool `json:"prev"`
|
||||
Modes map[string]Mode `json:"modes,omitempty"`
|
||||
Dirty []string `json:"dirty,omitempty"`
|
||||
Touched []string `json:"touched,omitempty"`
|
||||
}
|
||||
|
||||
type resultLine struct {
|
||||
Complete bool `json:"complete"`
|
||||
Partial int64 `json:"partial"`
|
||||
Reused int64 `json:"reused"`
|
||||
Read int64 `json:"read"`
|
||||
}
|
||||
|
||||
func Helper(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
|
||||
if len(args) == 0 {
|
||||
fmt.Fprintln(stderr, "usage: hamlab space-walk <mountpoint>...")
|
||||
return 2
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
sig := make(chan os.Signal, 1)
|
||||
signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP)
|
||||
go func() { <-sig; cancel() }()
|
||||
in := bufio.NewReaderSize(stdin, 1<<20)
|
||||
line, err := in.ReadBytes('\n')
|
||||
if err != nil {
|
||||
fmt.Fprintln(stderr, "error: no header:", err)
|
||||
return 2
|
||||
}
|
||||
var h header
|
||||
if err := json.Unmarshal(line, &h); err != nil {
|
||||
fmt.Fprintln(stderr, "error: bad header:", err)
|
||||
return 2
|
||||
}
|
||||
opts := Options{Modes: h.Modes, Dirty: set(h.Dirty), Touched: set(h.Touched)}
|
||||
if h.Prev {
|
||||
if opts.Prev, err = Decode(in); err != nil {
|
||||
fmt.Fprintln(stderr, "error: last tree:", err)
|
||||
return 2
|
||||
}
|
||||
}
|
||||
go func() { _, _ = io.Copy(io.Discard, in); cancel() }() // the daemon went away, or asked us to stop
|
||||
enc := json.NewEncoder(stderr)
|
||||
res, err := Walk(ctx, args, opts, func(p Progress) { _ = enc.Encode(p) })
|
||||
if err != nil && !errors.Is(err, context.Canceled) {
|
||||
fmt.Fprintln(stderr, "error:", err)
|
||||
return 1
|
||||
}
|
||||
rl, _ := json.Marshal(resultLine{Complete: res.Complete, Partial: res.Partial, Reused: res.Reused, Read: res.Read})
|
||||
if _, err := stdout.Write(append(rl, '\n')); err != nil {
|
||||
return 1
|
||||
}
|
||||
if err := Encode(stdout, res.Root); err != nil {
|
||||
fmt.Fprintln(stderr, "error:", err)
|
||||
return 1
|
||||
}
|
||||
if !res.Complete {
|
||||
return 3
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func set(list []string) map[string]bool {
|
||||
if len(list) == 0 {
|
||||
return nil
|
||||
}
|
||||
m := make(map[string]bool, len(list))
|
||||
for _, s := range list {
|
||||
m[s] = true
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ---- the daemon side: one scan per target, the last result kept, on disk too ----
|
||||
|
||||
type Scan struct {
|
||||
Disk string `json:"disk"`
|
||||
Roots []string `json:"roots"`
|
||||
Started time.Time `json:"started"`
|
||||
Finished time.Time `json:"finished,omitempty"`
|
||||
Running bool `json:"running"`
|
||||
Loading bool `json:"loading,omitempty"` // read back from disk after a restart, not yet decoded
|
||||
Error string `json:"error,omitempty"`
|
||||
Progress Progress `json:"progress"`
|
||||
AsRoot bool `json:"as_root"`
|
||||
// How this scan was done: "full", or "refresh" / "resume" from the one before, and by what
|
||||
// means per root ("zfs diff" or "mtime").
|
||||
Method string `json:"method,omitempty"`
|
||||
ByRoot map[string]string `json:"by_root,omitempty"`
|
||||
Partial bool `json:"partial,omitempty"` // cancelled before it finished; Scan again resumes it
|
||||
// once finished:
|
||||
Files uint32 `json:"files"`
|
||||
Dirs uint32 `json:"dirs"`
|
||||
Size int64 `json:"size"`
|
||||
Denied int64 `json:"denied"`
|
||||
Read int64 `json:"read"` // directories read this time
|
||||
Reused int64 `json:"reused"` // directories kept from the scan before
|
||||
PartialDirs int64 `json:"partial_dirs"` // directories not read (cancelled)
|
||||
// Bases is, per root that is a ZFS dataset, the newest snapshot that existed when the
|
||||
// scan started: the next scan asks `zfs diff` what changed since it.
|
||||
Bases map[string]string `json:"bases,omitempty"`
|
||||
|
||||
root *Node
|
||||
largest []File
|
||||
cancel context.CancelFunc
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
self string // this binary, for the helper
|
||||
dir string // where results are kept between restarts
|
||||
mu sync.Mutex
|
||||
scans map[string]*Scan
|
||||
}
|
||||
|
||||
// New makes a manager that keeps results under dir and reads back the ones there.
|
||||
func New(self, dir string) *Manager {
|
||||
m := &Manager{self: self, dir: dir, scans: map[string]*Scan{}}
|
||||
m.loadAll()
|
||||
return m
|
||||
}
|
||||
|
||||
func (m *Manager) fileFor(id, ext string) string {
|
||||
safe := strings.NewReplacer(":", "_", "/", "_", "\\", "_").Replace(id)
|
||||
return filepath.Join(m.dir, safe+ext)
|
||||
}
|
||||
|
||||
// loadAll reads the saved scans' metadata now and their trees in the background, one at a
|
||||
// time, so a restart does not double the memory of the last scans all at once.
|
||||
func (m *Manager) loadAll() {
|
||||
metas, _ := filepath.Glob(filepath.Join(m.dir, "*.json"))
|
||||
var pending []*Scan
|
||||
for _, mp := range metas {
|
||||
b, err := os.ReadFile(mp)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
s := &Scan{}
|
||||
if json.Unmarshal(b, s) != nil || s.Disk == "" {
|
||||
continue
|
||||
}
|
||||
s.Running, s.cancel = false, nil
|
||||
s.Loading = true
|
||||
m.scans[s.Disk] = s
|
||||
pending = append(pending, s)
|
||||
}
|
||||
if len(pending) == 0 {
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
for _, s := range pending {
|
||||
root, err := m.readTree(s.Disk)
|
||||
m.mu.Lock()
|
||||
if err != nil {
|
||||
log.Printf("space: saved scan of %s unreadable (%v); dropped", s.Disk, err)
|
||||
if m.scans[s.Disk] == s {
|
||||
delete(m.scans, s.Disk)
|
||||
}
|
||||
} else if m.scans[s.Disk] == s {
|
||||
s.root, s.Loading = root, false
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func (m *Manager) readTree(id string) (*Node, error) {
|
||||
f, err := os.Open(m.fileFor(id, ".scan"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
br := bufio.NewReaderSize(f, 1<<20)
|
||||
if _, err := br.ReadBytes('\n'); err != nil { // the result line
|
||||
return nil, err
|
||||
}
|
||||
return Decode(br)
|
||||
}
|
||||
|
||||
func (m *Manager) saveMeta(s *Scan) {
|
||||
if m.dir == "" {
|
||||
return
|
||||
}
|
||||
_ = os.MkdirAll(m.dir, 0o700)
|
||||
b, err := json.MarshalIndent(s, "", " ")
|
||||
if err == nil {
|
||||
err = writeAtomic(m.fileFor(s.Disk, ".json"), b)
|
||||
}
|
||||
if err != nil {
|
||||
log.Printf("space: save %s: %v", s.Disk, err)
|
||||
}
|
||||
}
|
||||
|
||||
func writeAtomic(path string, b []byte) error {
|
||||
tmp := path + ".tmp"
|
||||
if err := os.WriteFile(tmp, b, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// Status is the scan of id (a copy without the tree), or nil.
|
||||
func (m *Manager) Status(id string) *Scan {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.copyOf(m.scans[id])
|
||||
}
|
||||
|
||||
func (m *Manager) copyOf(s *Scan) *Scan {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
c := *s
|
||||
c.root, c.largest, c.cancel = nil, nil, nil
|
||||
return &c
|
||||
}
|
||||
|
||||
func (m *Manager) All() map[string]*Scan {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
out := map[string]*Scan{}
|
||||
for k, s := range m.scans {
|
||||
out[k] = m.copyOf(s)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Start scans the roots of id in the background, from the last scan of it when there is
|
||||
// one. datasets names the ZFS dataset behind each root that has one.
|
||||
func (m *Manager) Start(id string, roots []string, datasets map[string]string) (*Scan, error) {
|
||||
if len(roots) == 0 {
|
||||
return nil, errors.New("nothing is mounted from this disk")
|
||||
}
|
||||
m.mu.Lock()
|
||||
prev := m.scans[id]
|
||||
if prev != nil && prev.Running {
|
||||
m.mu.Unlock()
|
||||
return nil, errors.New("a scan of this disk is already running")
|
||||
}
|
||||
if prev != nil && (prev.root == nil || prev.Loading) {
|
||||
prev = nil // nothing to start from
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s := &Scan{Disk: id, Roots: roots, Started: time.Now(), Running: true, AsRoot: priv.Available(), cancel: cancel, Bases: map[string]string{}}
|
||||
if prev != nil {
|
||||
// the new scan takes the old one's place in the list while it runs, but keeps its
|
||||
// tree, so the pages still answer until the new one is in
|
||||
s.root, s.largest = prev.root, prev.largest
|
||||
s.Files, s.Dirs, s.Size = prev.Files, prev.Dirs, prev.Size
|
||||
}
|
||||
m.scans[id] = s
|
||||
m.mu.Unlock()
|
||||
go m.run(ctx, s, prev, datasets)
|
||||
return m.Status(id), nil
|
||||
}
|
||||
|
||||
// plan decides how each root is walked this time, given the last scan.
|
||||
func (m *Manager) plan(ctx context.Context, s, prev *Scan, datasets map[string]string) (header, Options) {
|
||||
h := header{Modes: map[string]Mode{}}
|
||||
opts := Options{Modes: h.Modes}
|
||||
if prev == nil || prev.root == nil {
|
||||
s.Method = "full"
|
||||
return h, opts
|
||||
}
|
||||
h.Prev = true
|
||||
opts.Prev = prev.root
|
||||
s.ByRoot = map[string]string{}
|
||||
dirty, touched := map[string]bool{}, map[string]bool{}
|
||||
// every dataset is asked at once what changed; the answers come back in root order
|
||||
type diffed struct {
|
||||
changed []change
|
||||
err error
|
||||
took time.Duration
|
||||
}
|
||||
answers := map[string]chan diffed{}
|
||||
for _, r := range s.Roots {
|
||||
if base := prev.Bases[r]; base != "" && datasets[r] != "" {
|
||||
ch := make(chan diffed, 1)
|
||||
answers[r] = ch
|
||||
go func(r, base string) {
|
||||
t0 := time.Now()
|
||||
c, err := zfsDiff(ctx, base, r)
|
||||
ch <- diffed{changed: c, err: err, took: time.Since(t0)}
|
||||
}(r, base)
|
||||
}
|
||||
}
|
||||
for _, r := range s.Roots {
|
||||
mode := ModeMTime
|
||||
how := "mtime"
|
||||
if ch, ok := answers[r]; ok {
|
||||
m.mu.Lock()
|
||||
s.Progress.Current = "asking zfs what changed in " + r
|
||||
m.mu.Unlock()
|
||||
d := <-ch
|
||||
if d.err == nil {
|
||||
mode, how = ModeDiff, fmt.Sprintf("zfs diff since %s: %d changes in %s", strings.TrimPrefix(prev.Bases[r], datasets[r]), len(d.changed), d.took.Round(time.Millisecond))
|
||||
for _, c := range d.changed {
|
||||
dirty[c.parent] = true
|
||||
if c.dir {
|
||||
dirty[c.path] = true
|
||||
}
|
||||
}
|
||||
} else {
|
||||
how = "mtime (zfs diff: " + d.err.Error() + ")"
|
||||
}
|
||||
}
|
||||
if Find(prev.root, r) == nil {
|
||||
mode, how = ModeFull, "full (new root)"
|
||||
}
|
||||
h.Modes[r] = mode
|
||||
s.ByRoot[r] = how
|
||||
}
|
||||
m.mu.Lock()
|
||||
s.Progress.Current = ""
|
||||
m.mu.Unlock()
|
||||
// what the last walk never got to is read whatever the change list says
|
||||
for _, p := range partialPaths(prev.root, "/") {
|
||||
dirty[p] = true
|
||||
}
|
||||
for p := range dirty {
|
||||
for a := filepath.Dir(p); ; a = filepath.Dir(a) {
|
||||
touched[a] = true
|
||||
if a == "/" {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
for p := range dirty {
|
||||
h.Dirty = append(h.Dirty, p)
|
||||
}
|
||||
for p := range touched {
|
||||
h.Touched = append(h.Touched, p)
|
||||
}
|
||||
sort.Strings(h.Dirty)
|
||||
sort.Strings(h.Touched)
|
||||
opts.Dirty, opts.Touched = dirty, touched
|
||||
if prev.Partial {
|
||||
s.Method = "resume"
|
||||
} else {
|
||||
s.Method = "refresh"
|
||||
}
|
||||
return h, opts
|
||||
}
|
||||
|
||||
func partialPaths(n *Node, path string) []string {
|
||||
var out []string
|
||||
var walk func(n *Node, path string)
|
||||
walk = func(n *Node, path string) {
|
||||
if n.Partial {
|
||||
out = append(out, path)
|
||||
}
|
||||
for _, c := range n.Children {
|
||||
if c.Dir {
|
||||
walk(c, join(path, c.Name))
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(n, path)
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) run(ctx context.Context, s *Scan, prev *Scan, datasets map[string]string) {
|
||||
finish := func(err error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
s.Running = false
|
||||
s.Finished = time.Now()
|
||||
if err != nil {
|
||||
s.Error = err.Error()
|
||||
}
|
||||
m.saveMeta(s)
|
||||
}
|
||||
// the newest snapshot of each dataset, so the next scan can ask what changed since
|
||||
for _, r := range s.Roots {
|
||||
if ds := datasets[r]; ds != "" {
|
||||
if snap := newestSnapshot(ctx, ds); snap != "" {
|
||||
s.Bases[r] = snap
|
||||
}
|
||||
}
|
||||
}
|
||||
h, _ := m.plan(ctx, s, prev, datasets)
|
||||
if ctx.Err() != nil { // cancelled while zfs was being asked
|
||||
finish(errors.New("cancelled"))
|
||||
return
|
||||
}
|
||||
args := append([]string{m.self, "space-walk"}, s.Roots...)
|
||||
if s.AsRoot {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
cmd := exec.Command(args[0], args[1:]...)
|
||||
stdin, _ := cmd.StdinPipe() // held open: closing it tells the helper to stop
|
||||
stdout, _ := cmd.StdoutPipe()
|
||||
stderr, _ := cmd.StderrPipe()
|
||||
if err := cmd.Start(); err != nil {
|
||||
finish(err)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
hb, _ := json.Marshal(h)
|
||||
_, err := stdin.Write(append(hb, '\n'))
|
||||
if err == nil && h.Prev {
|
||||
err = Encode(stdin, prev.root)
|
||||
}
|
||||
if err != nil {
|
||||
log.Printf("space: %s: sending the last tree: %v", s.Disk, err)
|
||||
}
|
||||
<-ctx.Done()
|
||||
_ = cmd.Process.Signal(syscall.SIGTERM)
|
||||
stdin.Close()
|
||||
}()
|
||||
var lastErr string
|
||||
go func() {
|
||||
sc := bufio.NewScanner(stderr)
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
var p Progress
|
||||
if strings.HasPrefix(line, "{") && json.Unmarshal([]byte(line), &p) == nil {
|
||||
m.mu.Lock()
|
||||
s.Progress = p
|
||||
m.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
if line != "" {
|
||||
lastErr = line
|
||||
}
|
||||
}
|
||||
}()
|
||||
// the result streams to disk as it is decoded, so a restart finds it
|
||||
_ = os.MkdirAll(m.dir, 0o700)
|
||||
tmp := m.fileFor(s.Disk, ".scan.tmp")
|
||||
out, ferr := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
|
||||
var src io.Reader = stdout
|
||||
if ferr == nil {
|
||||
src = io.TeeReader(stdout, out)
|
||||
}
|
||||
br := bufio.NewReaderSize(src, 1<<20)
|
||||
var rl resultLine
|
||||
line, rerr := br.ReadBytes('\n')
|
||||
if rerr == nil {
|
||||
rerr = json.Unmarshal(line, &rl)
|
||||
}
|
||||
var root *Node
|
||||
var decErr error
|
||||
if rerr == nil {
|
||||
root, decErr = Decode(br)
|
||||
} else {
|
||||
decErr = rerr
|
||||
_, _ = io.Copy(io.Discard, br)
|
||||
}
|
||||
if ferr == nil {
|
||||
out.Close()
|
||||
}
|
||||
waitErr := cmd.Wait()
|
||||
cancelled := ctx.Err() != nil
|
||||
if root == nil {
|
||||
_ = os.Remove(tmp)
|
||||
switch {
|
||||
case cancelled:
|
||||
finish(errors.New("cancelled"))
|
||||
case waitErr != nil && lastErr != "":
|
||||
finish(errors.New(strings.TrimPrefix(lastErr, "error: ")))
|
||||
case waitErr != nil:
|
||||
finish(waitErr)
|
||||
default:
|
||||
finish(decErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if ferr == nil {
|
||||
_ = os.Rename(tmp, m.fileFor(s.Disk, ".scan"))
|
||||
}
|
||||
m.mu.Lock()
|
||||
s.root, s.largest = root, nil
|
||||
s.Files, s.Dirs, s.Size, s.Denied = root.Files, root.Dirs, root.Size, s.Progress.Denied
|
||||
s.Read, s.Reused, s.PartialDirs = rl.Read, rl.Reused, rl.Partial
|
||||
s.Partial = !rl.Complete
|
||||
s.Running = false
|
||||
s.Finished = time.Now()
|
||||
if s.Partial {
|
||||
s.Error = "cancelled"
|
||||
}
|
||||
m.saveMeta(s)
|
||||
m.mu.Unlock()
|
||||
what := "scanned"
|
||||
if s.Partial {
|
||||
what = "partly scanned (cancelled)"
|
||||
}
|
||||
log.Printf("space: %s %s: %d files, %d dirs, %d bytes in %s (%s; %d dirs read, %d kept%s)", s.Disk, what, root.Files, root.Dirs, root.Size, s.Finished.Sub(s.Started).Round(time.Millisecond), s.Method, rl.Read, rl.Reused, map[bool]string{true: ", as root", false: ""}[s.AsRoot])
|
||||
}
|
||||
|
||||
func (m *Manager) Cancel(id string) bool {
|
||||
m.mu.Lock()
|
||||
s := m.scans[id]
|
||||
m.mu.Unlock()
|
||||
if s == nil || !s.Running || s.cancel == nil {
|
||||
return false
|
||||
}
|
||||
s.cancel()
|
||||
return true
|
||||
}
|
||||
|
||||
// Forget drops the scan of id, in memory and on disk.
|
||||
func (m *Manager) Forget(id string) {
|
||||
m.mu.Lock()
|
||||
if s := m.scans[id]; s != nil && s.cancel != nil {
|
||||
s.cancel()
|
||||
}
|
||||
delete(m.scans, id)
|
||||
m.mu.Unlock()
|
||||
_ = os.Remove(m.fileFor(id, ".scan"))
|
||||
_ = os.Remove(m.fileFor(id, ".json"))
|
||||
}
|
||||
|
||||
// ---- ZFS: what changed since the last scan ----
|
||||
|
||||
type change struct {
|
||||
path, parent string
|
||||
dir bool
|
||||
}
|
||||
|
||||
// newestSnapshot is the most recent snapshot of a dataset, "" when it has none.
|
||||
func newestSnapshot(ctx context.Context, dataset string) string {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-t", "snapshot", "-o", "name", "-s", "creation", "-d", "1", dataset).Output()
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
|
||||
last := lines[len(lines)-1]
|
||||
if !strings.Contains(last, "@") {
|
||||
return ""
|
||||
}
|
||||
return last
|
||||
}
|
||||
|
||||
// zfsDiff lists what changed in a dataset since a snapshot: every changed, added, removed
|
||||
// or renamed path, with the directory whose listing that touches.
|
||||
func zfsDiff(ctx context.Context, snapshot, mount string) ([]change, error) {
|
||||
// A diff slower than this is not worth waiting for: the walk by mtime is bounded. The
|
||||
// clock runs inside sudo — `timeout` is zfs's parent and kills it as root — because sudo
|
||||
// on Debian gives the command a session of its own, out of reach of our process group;
|
||||
// the context is only a backstop.
|
||||
const patience = 3 * time.Minute
|
||||
ctx, cancel := context.WithTimeout(ctx, patience+30*time.Second)
|
||||
defer cancel()
|
||||
args := priv.Wrap("timeout", "-s", "KILL", strconv.Itoa(int(patience.Seconds())), "zfs", "diff", "-H", "-F", snapshot)
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
|
||||
cmd.WaitDelay = 5 * time.Second
|
||||
t0 := time.Now()
|
||||
var stderr strings.Builder
|
||||
cmd.Stderr = &stderr
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var changes []change
|
||||
sc := bufio.NewScanner(out)
|
||||
sc.Buffer(make([]byte, 64<<10), 4<<20)
|
||||
for sc.Scan() {
|
||||
f := strings.Split(sc.Text(), "\t")
|
||||
if len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
isDir := f[1] == "/"
|
||||
for _, p := range f[2:] { // a rename has the old and the new path
|
||||
p = unescape(p)
|
||||
if !strings.HasPrefix(p, mount) {
|
||||
continue
|
||||
}
|
||||
changes = append(changes, change{path: p, parent: filepath.Dir(p), dir: isDir})
|
||||
}
|
||||
}
|
||||
if err := cmd.Wait(); err != nil {
|
||||
msg := strings.TrimSpace(stderr.String())
|
||||
if errors.Is(ctx.Err(), context.DeadlineExceeded) || time.Since(t0) >= patience {
|
||||
msg = "gave up after " + patience.String()
|
||||
}
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
if i := strings.IndexByte(msg, '\n'); i > 0 {
|
||||
msg = msg[:i]
|
||||
}
|
||||
return nil, errors.New(msg)
|
||||
}
|
||||
return changes, nil
|
||||
}
|
||||
|
||||
// unescape undoes zfs diff's \ooo escapes for spaces and odd bytes in names.
|
||||
func unescape(s string) string {
|
||||
if !strings.Contains(s, "\\") {
|
||||
return s
|
||||
}
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(s); i++ {
|
||||
if s[i] == '\\' && i+3 < len(s) {
|
||||
if v, err := strconv.ParseUint(s[i+1:i+4], 8, 8); err == nil {
|
||||
b.WriteByte(byte(v))
|
||||
i += 3
|
||||
continue
|
||||
}
|
||||
}
|
||||
b.WriteByte(s[i])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// Tree is the subtree at path, cut to depth levels of at most limit children each, leaving
|
||||
// out children smaller than min bytes; what is left out is summed into MoreN/MoreSize. A
|
||||
// treemap asks deep with a min the size of the smallest tile it could draw; a folder list
|
||||
// asks one level with no min.
|
||||
type Tree struct {
|
||||
Node
|
||||
Path string `json:"path"`
|
||||
Children []*Tree `json:"children,omitempty"`
|
||||
MoreN int `json:"more_n,omitempty"`
|
||||
MoreSize int64 `json:"more_size,omitempty"`
|
||||
Total int64 `json:"total"` // the whole scan's size, for percentages
|
||||
}
|
||||
|
||||
func (m *Manager) Tree(id, path string, depth, limit int, min int64) (*Tree, error) {
|
||||
m.mu.Lock()
|
||||
s := m.scans[id]
|
||||
var root *Node
|
||||
if s != nil {
|
||||
root = s.root
|
||||
}
|
||||
m.mu.Unlock()
|
||||
if root == nil {
|
||||
if s != nil && s.Loading {
|
||||
return nil, errors.New("the last scan is still being read back")
|
||||
}
|
||||
return nil, errors.New("no scan of this disk yet")
|
||||
}
|
||||
n := Find(root, path)
|
||||
if n == nil {
|
||||
return nil, errors.New("no such path in the scan")
|
||||
}
|
||||
if path == "" {
|
||||
path = "/"
|
||||
}
|
||||
t := cut(n, path, depth, limit, min)
|
||||
t.Total = root.Size
|
||||
return t, nil
|
||||
}
|
||||
|
||||
func cut(n *Node, path string, depth, limit int, min int64) *Tree {
|
||||
t := &Tree{Node: *n, Path: path}
|
||||
t.Node.Children = nil
|
||||
if !n.Dir {
|
||||
return t
|
||||
}
|
||||
if depth <= 0 { // as deep as was asked: what is inside is one lump, not nothing
|
||||
if len(n.Children) > 0 {
|
||||
t.MoreN, t.MoreSize = len(n.Children), n.Size-n.SmallSize
|
||||
}
|
||||
return t
|
||||
}
|
||||
for i, c := range n.Children {
|
||||
if i >= limit || (c.Size < min && i > 0) { // children are sorted largest first: the first is always kept
|
||||
t.MoreN++
|
||||
t.MoreSize += c.Size
|
||||
continue
|
||||
}
|
||||
t.Children = append(t.Children, cut(c, join(path, c.Name), depth-1, limit, min))
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// Largest is the n biggest files of the last scan of id.
|
||||
func (m *Manager) Largest(id string, n int) ([]File, error) {
|
||||
m.mu.Lock()
|
||||
s := m.scans[id]
|
||||
var root *Node
|
||||
var have []File
|
||||
if s != nil {
|
||||
root, have = s.root, s.largest
|
||||
}
|
||||
m.mu.Unlock()
|
||||
if root == nil {
|
||||
return nil, errors.New("no scan of this disk yet")
|
||||
}
|
||||
if len(have) < n {
|
||||
have = Largest(root, n) // the tree is never changed once made, so no lock while walking it
|
||||
m.mu.Lock()
|
||||
if s.root == root {
|
||||
s.largest = have
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
if len(have) > n {
|
||||
return have[:n], nil
|
||||
}
|
||||
return have, nil
|
||||
}
|
||||
@@ -0,0 +1,516 @@
|
||||
// Package space measures what is using a disk — it is what the UI calls the **Usage** tab
|
||||
// (renamed from "Space" on 2026-09-24; the package and its /api/space endpoints kept the
|
||||
// old name rather than churn a working API): a parallel walk of the filesystems a disk
|
||||
// backs, kept as a tree of directories and the files worth seeing, the way WizTree does it.
|
||||
//
|
||||
// The walk never crosses a filesystem boundary by itself — /proc, /sys, another disk's
|
||||
// mount — but every mountpoint the disk backs is a root of the same scan, and the roots
|
||||
// are grafted into one tree at their paths, so a ZFS pool of many datasets, or a disk with
|
||||
// / and /home on separate partitions, reads as one picture.
|
||||
//
|
||||
// Files below KeepMin are not kept one by one: a directory of a million small files is
|
||||
// one "smaller files" entry with their count and total, which is all a treemap can show
|
||||
// of them anyway, and what keeps a scan of a few million files within a few hundred MB.
|
||||
//
|
||||
// A walk can start from the last one. Given the previous tree, a directory is read again
|
||||
// only when something says it changed: on ZFS, `zfs diff` against a snapshot older than
|
||||
// the last scan names every path that changed, and directories it does not name are kept
|
||||
// as they were without so much as a stat; elsewhere a directory whose mtime is what it was
|
||||
// keeps its list and only its kept files are stat'ed again, since a file can grow without
|
||||
// its directory noticing. A directory the last walk did not finish (it was cancelled) is
|
||||
// read again whatever else is true, which is what makes a cancelled scan resumable.
|
||||
package space
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// KeepMin is the smallest file kept as its own node.
|
||||
const KeepMin = 128 << 10
|
||||
|
||||
// Node is a directory or a file. Sizes are bytes allocated on disk (what fills the disk);
|
||||
// Apparent is the length. For a directory both include everything beneath it.
|
||||
type Node struct {
|
||||
Name string `json:"name"`
|
||||
Dir bool `json:"dir"`
|
||||
Size int64 `json:"size"`
|
||||
Apparent int64 `json:"apparent"`
|
||||
MTime int64 `json:"mtime"` // unix seconds
|
||||
MTimeNs int64 `json:"-"` // directories: the same to the nanosecond, so a change in the same second is not missed
|
||||
Files uint32 `json:"files"` // beneath a directory (kept and small alike)
|
||||
Dirs uint32 `json:"dirs"`
|
||||
SmallN uint32 `json:"small_n,omitempty"` // files here below KeepMin, not listed
|
||||
SmallSize int64 `json:"small_size,omitempty"` // their allocated bytes
|
||||
Mount bool `json:"mount,omitempty"` // another filesystem starts here (not this disk's, or a root of its own)
|
||||
Denied bool `json:"denied,omitempty"` // could not be read
|
||||
Partial bool `json:"partial,omitempty"` // the walk was cancelled before this directory was read
|
||||
Children []*Node `json:"children,omitempty"` // directories: sorted by Size, largest first
|
||||
}
|
||||
|
||||
// Progress is how far a running walk has got.
|
||||
type Progress struct {
|
||||
Files int64 `json:"files"`
|
||||
Dirs int64 `json:"dirs"` // directories read
|
||||
Reused int64 `json:"reused"` // directories kept from the last scan without reading
|
||||
Bytes int64 `json:"bytes"`
|
||||
Denied int64 `json:"denied"`
|
||||
Current string `json:"current"`
|
||||
}
|
||||
|
||||
// Mode is how a root is walked when there is a previous tree.
|
||||
type Mode int
|
||||
|
||||
const (
|
||||
ModeFull Mode = iota // read everything
|
||||
ModeMTime // read a directory again only if its mtime changed; stat its kept files
|
||||
ModeDiff // read only the directories a change list names
|
||||
)
|
||||
|
||||
// Options is what a walk starts from.
|
||||
type Options struct {
|
||||
Prev *Node // the last scan's tree, or nil
|
||||
// Modes says how each root is walked (by root path); a root not listed is ModeFull.
|
||||
Modes map[string]Mode
|
||||
// Dirty is every directory a change list named (ModeDiff roots): those are read again.
|
||||
// Touched is every ancestor of a dirty directory: those are kept but looked into.
|
||||
Dirty, Touched map[string]bool
|
||||
}
|
||||
|
||||
type walker struct {
|
||||
ctx context.Context
|
||||
opts Options
|
||||
mu sync.Mutex
|
||||
queue []job
|
||||
cond *sync.Cond
|
||||
pending int // dirs queued or being read
|
||||
files atomic.Int64
|
||||
dirs atomic.Int64
|
||||
reused atomic.Int64
|
||||
bytes atomic.Int64
|
||||
denied atomic.Int64
|
||||
partial atomic.Int64
|
||||
curMu sync.Mutex
|
||||
current string
|
||||
stopping bool
|
||||
}
|
||||
|
||||
type job struct {
|
||||
path string
|
||||
node *Node
|
||||
prev *Node // the same directory in the last scan, if it had one
|
||||
dev uint64 // the filesystem this root is on; a child on another is a mountpoint, not descended
|
||||
mode Mode
|
||||
}
|
||||
|
||||
// Result is what a walk returns besides the tree.
|
||||
type Result struct {
|
||||
Root *Node
|
||||
Partial int64 // directories not read because the walk was cancelled
|
||||
Reused int64
|
||||
Read int64
|
||||
Complete bool
|
||||
}
|
||||
|
||||
// Walk measures roots (mountpoints, absolute) and returns them grafted into one tree whose
|
||||
// root is "/". progress, if set, is called about four times a second. A cancelled walk
|
||||
// still returns the tree, with the directories it did not get to marked Partial.
|
||||
func Walk(ctx context.Context, roots []string, opts Options, progress func(Progress)) (Result, error) {
|
||||
w := &walker{ctx: ctx, opts: opts}
|
||||
w.cond = sync.NewCond(&w.mu)
|
||||
root := &Node{Name: "/", Dir: true}
|
||||
sort.Strings(roots)
|
||||
trees := map[string]*Node{}
|
||||
for _, r := range roots {
|
||||
r = filepath.Clean(r)
|
||||
if !filepath.IsAbs(r) {
|
||||
continue
|
||||
}
|
||||
var st syscall.Stat_t
|
||||
if err := syscall.Lstat(r, &st); err != nil || st.Mode&syscall.S_IFMT != syscall.S_IFDIR {
|
||||
continue
|
||||
}
|
||||
n := &Node{Name: filepath.Base(r), Dir: true, MTime: int64(st.Mtim.Sec), MTimeNs: st.Mtim.Nano()}
|
||||
if r == "/" {
|
||||
n = root
|
||||
}
|
||||
trees[r] = n
|
||||
var prev *Node
|
||||
if opts.Prev != nil {
|
||||
prev = Find(opts.Prev, r)
|
||||
}
|
||||
mode := opts.Modes[r]
|
||||
if prev == nil {
|
||||
mode = ModeFull
|
||||
}
|
||||
w.push(job{path: r, node: n, prev: prev, dev: uint64(st.Dev), mode: mode})
|
||||
}
|
||||
if len(trees) == 0 {
|
||||
return Result{Root: root}, os.ErrNotExist
|
||||
}
|
||||
workers := 32
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < workers; i++ {
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); w.run() }()
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() { wg.Wait(); close(done) }()
|
||||
tick := time.NewTicker(250 * time.Millisecond)
|
||||
defer tick.Stop()
|
||||
cancelled := false
|
||||
for waiting := true; waiting; {
|
||||
select {
|
||||
case <-done:
|
||||
waiting = false
|
||||
case <-tick.C:
|
||||
if progress != nil {
|
||||
progress(w.snapshot())
|
||||
}
|
||||
case <-ctx.Done():
|
||||
cancelled = true
|
||||
w.mu.Lock()
|
||||
w.stopping = true
|
||||
for _, j := range w.queue { // never read: say so, so the next walk reads them
|
||||
j.node.Partial = true
|
||||
w.partial.Add(1)
|
||||
}
|
||||
w.queue = nil
|
||||
w.cond.Broadcast()
|
||||
w.mu.Unlock()
|
||||
<-done
|
||||
waiting = false
|
||||
}
|
||||
}
|
||||
if progress != nil {
|
||||
progress(w.snapshot())
|
||||
}
|
||||
// graft each root at its path; deepest last so parents exist first
|
||||
for _, r := range roots {
|
||||
n, ok := trees[filepath.Clean(r)]
|
||||
if !ok || n == root {
|
||||
continue
|
||||
}
|
||||
graft(root, filepath.Clean(r), n)
|
||||
}
|
||||
total(root)
|
||||
res := Result{Root: root, Partial: w.partial.Load(), Reused: w.reused.Load(), Read: w.dirs.Load(), Complete: !cancelled}
|
||||
if cancelled {
|
||||
return res, ctx.Err()
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
func (w *walker) snapshot() Progress {
|
||||
w.curMu.Lock()
|
||||
cur := w.current
|
||||
w.curMu.Unlock()
|
||||
return Progress{Files: w.files.Load(), Dirs: w.dirs.Load(), Reused: w.reused.Load(), Bytes: w.bytes.Load(), Denied: w.denied.Load(), Current: cur}
|
||||
}
|
||||
|
||||
func (w *walker) push(j job) {
|
||||
w.mu.Lock()
|
||||
if w.stopping {
|
||||
j.node.Partial = true
|
||||
w.partial.Add(1)
|
||||
w.mu.Unlock()
|
||||
return
|
||||
}
|
||||
w.queue = append(w.queue, j)
|
||||
w.pending++
|
||||
w.cond.Signal()
|
||||
w.mu.Unlock()
|
||||
}
|
||||
|
||||
// run takes directories off the queue until there are none left and none being read.
|
||||
func (w *walker) run() {
|
||||
for {
|
||||
w.mu.Lock()
|
||||
for len(w.queue) == 0 && w.pending > 0 && !w.stopping {
|
||||
w.cond.Wait()
|
||||
}
|
||||
if w.stopping || (len(w.queue) == 0 && w.pending == 0) {
|
||||
w.cond.Broadcast()
|
||||
w.mu.Unlock()
|
||||
return
|
||||
}
|
||||
j := w.queue[len(w.queue)-1] // depth first: keeps the queue short and the disk head local
|
||||
w.queue = w.queue[:len(w.queue)-1]
|
||||
w.mu.Unlock()
|
||||
w.visit(j)
|
||||
w.mu.Lock()
|
||||
w.pending--
|
||||
if w.pending == 0 {
|
||||
w.cond.Broadcast()
|
||||
}
|
||||
w.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
func join(dir, name string) string {
|
||||
if dir == "/" {
|
||||
return "/" + name
|
||||
}
|
||||
return dir + "/" + name
|
||||
}
|
||||
|
||||
// visit decides whether a directory must be read or can be taken from the last scan.
|
||||
func (w *walker) visit(j job) {
|
||||
if j.prev == nil || j.prev.Partial || j.prev.Denied || j.mode == ModeFull {
|
||||
w.readDir(j)
|
||||
return
|
||||
}
|
||||
switch j.mode {
|
||||
case ModeDiff:
|
||||
if w.opts.Dirty[j.path] {
|
||||
w.readDir(j)
|
||||
return
|
||||
}
|
||||
if !w.opts.Touched[j.path] {
|
||||
w.reuse(j, false) // nothing under here changed: keep the whole subtree, no stat at all
|
||||
return
|
||||
}
|
||||
w.reuse(j, true) // something deeper changed: keep this list, look into the children
|
||||
case ModeMTime:
|
||||
var st syscall.Stat_t
|
||||
if err := syscall.Lstat(j.path, &st); err != nil {
|
||||
w.readDir(j)
|
||||
return
|
||||
}
|
||||
if st.Mtim.Nano() != j.prev.MTimeNs {
|
||||
w.readDir(j) // an entry was added, removed or renamed
|
||||
return
|
||||
}
|
||||
w.reuse(j, true) // same entries; files may have grown, and subdirectories are their own question
|
||||
}
|
||||
}
|
||||
|
||||
// reuse keeps a directory as the last scan saw it. With descend, its subdirectories are
|
||||
// visited (each decides for itself) and, in mtime mode, its kept files are stat'ed again.
|
||||
func (w *walker) reuse(j job, descend bool) {
|
||||
w.reused.Add(1)
|
||||
n, p := j.node, j.prev
|
||||
n.MTime, n.MTimeNs, n.SmallN, n.SmallSize, n.Denied, n.Mount = p.MTime, p.MTimeNs, p.SmallN, p.SmallSize, p.Denied, p.Mount
|
||||
if !descend {
|
||||
n.Children = p.Children
|
||||
n.Apparent = p.Apparent - childApparent(p) // total() adds the children back
|
||||
w.files.Add(int64(p.Files))
|
||||
w.bytes.Add(p.Size)
|
||||
w.reused.Add(int64(p.Dirs))
|
||||
return
|
||||
}
|
||||
w.files.Add(int64(p.SmallN))
|
||||
w.bytes.Add(p.SmallSize)
|
||||
n.Apparent = p.Apparent - childApparent(p) // the small files' lengths; total() adds the children back
|
||||
var st syscall.Stat_t
|
||||
for _, c := range p.Children {
|
||||
if c.Dir {
|
||||
d := &Node{Name: c.Name, Dir: true, MTime: c.MTime, MTimeNs: c.MTimeNs, Mount: c.Mount}
|
||||
n.Children = append(n.Children, d)
|
||||
if c.Mount {
|
||||
continue
|
||||
}
|
||||
w.push(job{path: join(j.path, c.Name), node: d, prev: c, dev: j.dev, mode: j.mode})
|
||||
continue
|
||||
}
|
||||
if j.mode == ModeMTime {
|
||||
if err := syscall.Lstat(join(j.path, c.Name), &st); err != nil {
|
||||
continue // gone since (the directory's mtime would have said so, but be safe)
|
||||
}
|
||||
c = &Node{Name: c.Name, Size: int64(st.Blocks) * 512, Apparent: int64(st.Size), MTime: int64(st.Mtim.Sec)}
|
||||
}
|
||||
n.Children = append(n.Children, c)
|
||||
w.files.Add(1)
|
||||
w.bytes.Add(c.Size)
|
||||
}
|
||||
}
|
||||
|
||||
func childApparent(p *Node) int64 {
|
||||
var a int64
|
||||
for _, c := range p.Children {
|
||||
a += c.Apparent
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
func (w *walker) readDir(j job) {
|
||||
w.curMu.Lock()
|
||||
w.current = j.path
|
||||
w.curMu.Unlock()
|
||||
w.dirs.Add(1)
|
||||
f, err := os.Open(j.path)
|
||||
if err != nil {
|
||||
j.node.Denied = true
|
||||
w.denied.Add(1)
|
||||
return
|
||||
}
|
||||
names, err := f.Readdirnames(-1)
|
||||
f.Close()
|
||||
if err != nil {
|
||||
j.node.Denied = true
|
||||
w.denied.Add(1)
|
||||
}
|
||||
var prevDirs map[string]*Node
|
||||
if j.prev != nil {
|
||||
prevDirs = map[string]*Node{}
|
||||
for _, c := range j.prev.Children {
|
||||
if c.Dir {
|
||||
prevDirs[c.Name] = c
|
||||
}
|
||||
}
|
||||
}
|
||||
var st syscall.Stat_t
|
||||
for _, name := range names {
|
||||
p := join(j.path, name)
|
||||
if err := syscall.Lstat(p, &st); err != nil {
|
||||
continue
|
||||
}
|
||||
switch st.Mode & syscall.S_IFMT {
|
||||
case syscall.S_IFDIR:
|
||||
c := &Node{Name: name, Dir: true, MTime: int64(st.Mtim.Sec), MTimeNs: st.Mtim.Nano()}
|
||||
j.node.Children = append(j.node.Children, c)
|
||||
if uint64(st.Dev) != j.dev {
|
||||
c.Mount = true // something else is mounted here; if it is this disk's too, it is a root of its own
|
||||
continue
|
||||
}
|
||||
w.push(job{path: p, node: c, prev: prevDirs[name], dev: j.dev, mode: j.mode})
|
||||
case syscall.S_IFREG, syscall.S_IFLNK:
|
||||
alloc := int64(st.Blocks) * 512
|
||||
w.files.Add(1)
|
||||
w.bytes.Add(alloc)
|
||||
if alloc >= KeepMin || int64(st.Size) >= KeepMin {
|
||||
j.node.Children = append(j.node.Children, &Node{Name: name, Size: alloc, Apparent: int64(st.Size), MTime: int64(st.Mtim.Sec)})
|
||||
} else {
|
||||
j.node.SmallN++
|
||||
j.node.SmallSize += alloc
|
||||
j.node.Apparent += int64(st.Size) // added to the directory's own apparent total below
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// graft puts a root's tree at path under root, making any directories in between.
|
||||
func graft(root *Node, path string, tree *Node) {
|
||||
parts := strings.Split(strings.Trim(path, "/"), "/")
|
||||
cur := root
|
||||
for i, part := range parts {
|
||||
var next *Node
|
||||
for _, c := range cur.Children {
|
||||
if c.Name == part && c.Dir {
|
||||
next = c
|
||||
break
|
||||
}
|
||||
}
|
||||
if i == len(parts)-1 {
|
||||
if next == nil {
|
||||
cur.Children = append(cur.Children, tree)
|
||||
} else {
|
||||
// the parent's walk saw this directory as a mountpoint; the walk of the mount fills it
|
||||
next.Children, next.SmallN, next.SmallSize, next.Apparent, next.Denied, next.Partial = tree.Children, tree.SmallN, tree.SmallSize, tree.Apparent, tree.Denied, tree.Partial
|
||||
next.Mount = true
|
||||
}
|
||||
return
|
||||
}
|
||||
if next == nil {
|
||||
next = &Node{Name: part, Dir: true}
|
||||
cur.Children = append(cur.Children, next)
|
||||
}
|
||||
cur = next
|
||||
}
|
||||
}
|
||||
|
||||
// total sums every directory from its children and sorts them largest first.
|
||||
func total(n *Node) {
|
||||
if !n.Dir {
|
||||
return
|
||||
}
|
||||
n.Size = n.SmallSize
|
||||
n.Files = n.SmallN
|
||||
n.Dirs = 0
|
||||
apparent := n.Apparent // the small files' lengths
|
||||
for _, c := range n.Children {
|
||||
total(c)
|
||||
n.Size += c.Size
|
||||
apparent += c.Apparent
|
||||
if c.Dir {
|
||||
n.Dirs += 1 + c.Dirs
|
||||
n.Files += c.Files
|
||||
} else {
|
||||
n.Files++
|
||||
}
|
||||
}
|
||||
n.Apparent = apparent
|
||||
sort.Slice(n.Children, func(i, j int) bool { return n.Children[i].Size > n.Children[j].Size })
|
||||
}
|
||||
|
||||
// Find follows path (absolute, "/" for the root) down the tree.
|
||||
func Find(root *Node, path string) *Node {
|
||||
cur := root
|
||||
for _, part := range strings.Split(strings.Trim(filepath.Clean(path), "/"), "/") {
|
||||
if part == "" {
|
||||
continue
|
||||
}
|
||||
var next *Node
|
||||
for _, c := range cur.Children {
|
||||
if c.Name == part {
|
||||
next = c
|
||||
break
|
||||
}
|
||||
}
|
||||
if next == nil {
|
||||
return nil
|
||||
}
|
||||
cur = next
|
||||
}
|
||||
return cur
|
||||
}
|
||||
|
||||
// File is one of the largest files, with its full path.
|
||||
type File struct {
|
||||
Path string `json:"path"`
|
||||
Size int64 `json:"size"`
|
||||
Apparent int64 `json:"apparent"`
|
||||
MTime int64 `json:"mtime"`
|
||||
}
|
||||
|
||||
// Largest is the n biggest files in the tree, largest first.
|
||||
func Largest(root *Node, n int) []File {
|
||||
var out []File
|
||||
minIdx := -1
|
||||
var walk func(n *Node, path string)
|
||||
walk = func(node *Node, path string) {
|
||||
for _, c := range node.Children {
|
||||
p := join(path, c.Name)
|
||||
if c.Dir {
|
||||
if c.Size > 0 && (len(out) < n || c.Size > out[minIdx].Size) {
|
||||
walk(c, p)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if len(out) < n {
|
||||
out = append(out, File{Path: p, Size: c.Size, Apparent: c.Apparent, MTime: c.MTime})
|
||||
if minIdx < 0 || c.Size < out[minIdx].Size {
|
||||
minIdx = len(out) - 1
|
||||
}
|
||||
} else if c.Size > out[minIdx].Size {
|
||||
out[minIdx] = File{Path: p, Size: c.Size, Apparent: c.Apparent, MTime: c.MTime}
|
||||
minIdx = 0
|
||||
for i := range out {
|
||||
if out[i].Size < out[minIdx].Size {
|
||||
minIdx = i
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(root, "/")
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Size > out[j].Size })
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package space
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func mk(t *testing.T, dir string, files map[string]int) {
|
||||
for p, n := range files {
|
||||
full := filepath.Join(dir, p)
|
||||
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(full, make([]byte, n), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefreshAndResume(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
mk(t, dir, map[string]int{"a/big1": 300 << 10, "a/small": 100, "b/c/big2": 500 << 10, "b/d/big3": 200 << 10})
|
||||
res, err := Walk(context.Background(), []string{dir}, Options{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
root := Find(res.Root, dir)
|
||||
if root == nil || root.Files != 4 {
|
||||
t.Fatalf("full walk: %+v", root)
|
||||
}
|
||||
want := root.Size
|
||||
// grow a file in place (its directory's mtime does not change) and add a file elsewhere
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
if err := os.WriteFile(filepath.Join(dir, "a/big1"), make([]byte, 600<<10), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mk(t, dir, map[string]int{"b/d/big4": 400 << 10})
|
||||
res2, err := Walk(context.Background(), []string{dir}, Options{Prev: res.Root, Modes: map[string]Mode{dir: ModeMTime}}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r2 := Find(res2.Root, dir)
|
||||
if r2.Files != 5 || r2.Size != want+(300<<10)+(400<<10) {
|
||||
t.Fatalf("mtime refresh: files %d size %d want %d (reused %d read %d)", r2.Files, r2.Size, want+(300<<10)+(400<<10), res2.Reused, res2.Read)
|
||||
}
|
||||
if res2.Reused == 0 || res2.Read == 0 {
|
||||
t.Fatalf("mtime refresh should reuse some and read some: reused %d read %d", res2.Reused, res2.Read)
|
||||
}
|
||||
// diff mode: only b/d named dirty; a's growth must NOT be seen (that is the contract), b/d's new file must
|
||||
dirty := map[string]bool{filepath.Join(dir, "b/d"): true}
|
||||
touched := map[string]bool{dir: true, filepath.Join(dir, "b"): true}
|
||||
res3, err := Walk(context.Background(), []string{dir}, Options{Prev: res.Root, Modes: map[string]Mode{dir: ModeDiff}, Dirty: dirty, Touched: touched}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r3 := Find(res3.Root, dir)
|
||||
if r3.Files != 5 || r3.Size != want+(400<<10) || res3.Read != 1 {
|
||||
t.Fatalf("diff refresh: files %d size %d want %d read %d", r3.Files, r3.Size, want+(400<<10), res3.Read)
|
||||
}
|
||||
// a cancelled walk marks what it did not read; resuming reads those
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
res4, err := Walk(ctx, []string{dir}, Options{}, nil)
|
||||
if err == nil || res4.Complete {
|
||||
t.Fatal("cancelled walk should say so")
|
||||
}
|
||||
if len(partialPaths(res4.Root, "/")) == 0 {
|
||||
t.Fatal("cancelled walk should leave partial directories")
|
||||
}
|
||||
res5, err := Walk(context.Background(), []string{dir}, Options{Prev: res4.Root, Modes: map[string]Mode{dir: ModeMTime}}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r5 := Find(res5.Root, dir); r5.Files != 5 || len(partialPaths(res5.Root, "/")) != 0 {
|
||||
t.Fatalf("resume: files %d partial %v", r5.Files, partialPaths(res5.Root, "/"))
|
||||
}
|
||||
// the codec round-trips the partial flag
|
||||
pr, pw, _ := os.Pipe()
|
||||
go func() { _ = Encode(pw, res4.Root); pw.Close() }()
|
||||
back, err := Decode(pr)
|
||||
if err != nil || len(partialPaths(back, "/")) != len(partialPaths(res4.Root, "/")) {
|
||||
t.Fatalf("codec: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
package storage
|
||||
|
||||
// The ZFS caching tiers, for the ZFS tab: what the ARC has done since boot and how it is
|
||||
// tuned, and for each pool which cache devices it has — L2ARC, a separate log (SLOG), a
|
||||
// special allocation class, a dedup table — or, as is common, none of them, which the tab
|
||||
// should say plainly rather than leave a blank.
|
||||
//
|
||||
// The live numbers (size, the hit rate over the last interval) come from the sampler in
|
||||
// internal/metrics, which also publishes them as zfs.arc.* sensors; this is the slower,
|
||||
// fuller picture, read when the tab asks.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// CacheDev is one member of a pool's cache, log, special, dedup or spare class.
|
||||
type CacheDev struct {
|
||||
Name string `json:"name"`
|
||||
Size uint64 `json:"size,omitempty"`
|
||||
Alloc uint64 `json:"alloc,omitempty"`
|
||||
Health string `json:"health,omitempty"`
|
||||
}
|
||||
|
||||
// PoolCaching is what one pool has beyond its data vdevs, and the properties that steer
|
||||
// what is cached.
|
||||
type PoolCaching struct {
|
||||
Pool string `json:"pool"`
|
||||
Cache []CacheDev `json:"cache"` // L2ARC
|
||||
Log []CacheDev `json:"log"` // SLOG
|
||||
Special []CacheDev `json:"special"` // metadata / small blocks
|
||||
Dedup []CacheDev `json:"dedup"` // dedup table vdevs
|
||||
Spare []CacheDev `json:"spare"`
|
||||
// dataset properties at the pool root
|
||||
PrimaryCache string `json:"primarycache"`
|
||||
SecondaryCache string `json:"secondarycache"`
|
||||
Sync string `json:"sync"`
|
||||
LogBias string `json:"logbias"`
|
||||
DedupProp string `json:"dedup_prop"`
|
||||
Compression string `json:"compression"`
|
||||
RecordSize string `json:"recordsize"`
|
||||
DedupRatio string `json:"dedup_ratio"` // zpool's, e.g. "1.00x"
|
||||
}
|
||||
|
||||
// ARCDetail is the full picture.
|
||||
type ARCDetail struct {
|
||||
Available bool `json:"available"`
|
||||
MemTotal uint64 `json:"mem_total"`
|
||||
Tunables map[string]string `json:"tunables"` // zfs module parameters that shape the cache; "" when unset
|
||||
Lifetime struct {
|
||||
Hits, Misses uint64 `json:",omitempty"`
|
||||
DemandDataHits, DemandDataMisses uint64
|
||||
DemandMetaHits, DemandMetaMisses uint64
|
||||
PrefetchDataHits, PrefetchDataMisses uint64
|
||||
PrefetchMetaHits, PrefetchMetaMisses uint64
|
||||
L2Hits, L2Misses, L2ReadBytes, L2WriteBytes uint64
|
||||
L2Feeds, L2RebuildSize uint64
|
||||
EvictL2Eligible uint64 // bytes evicted from RAM that an L2ARC could have caught
|
||||
Deleted, EvictSkip, MutexMiss, Prune, Throttle uint64
|
||||
} `json:"lifetime"`
|
||||
Zfetch struct{ Hits, Misses, IOIssued uint64 } `json:"zfetch"` // the prefetcher
|
||||
ZIL struct {
|
||||
Commits uint64 `json:"commits"`
|
||||
NormalBytes, NormalWrite uint64 // written to the pool's own vdevs
|
||||
SlogBytes, SlogWrite uint64 // written to a separate log device
|
||||
} `json:"zil"`
|
||||
DBuf struct{ Size, Target, Hits, Misses uint64 } `json:"dbuf"` // the dbuf cache in front of the ARC
|
||||
Pools []PoolCaching `json:"pools"`
|
||||
}
|
||||
|
||||
const kstatDir = "/proc/spl/kstat/zfs/"
|
||||
|
||||
func readKstat(p string) map[string]uint64 {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
out := map[string]uint64{}
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) != 3 || f[1] != "4" {
|
||||
continue
|
||||
}
|
||||
if v, err := strconv.ParseUint(f[2], 10, 64); err == nil {
|
||||
out[f[0]] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
var arcTunables = []string{"zfs_arc_max", "zfs_arc_min", "zfs_arc_sys_free", "zfs_arc_meta_balance", "zfs_prefetch_disable", "l2arc_write_max", "l2arc_write_boost", "l2arc_noprefetch", "l2arc_rebuild_enabled", "zfs_dirty_data_max"}
|
||||
|
||||
const poolCachingEvery = 30 * time.Second // zpool list -v and zfs get are an exec each
|
||||
|
||||
// ARC reads the kstats now and the pool layout from a short cache.
|
||||
func (m *Manager) ARC(ctx context.Context) ARCDetail {
|
||||
d := ARCDetail{Tunables: map[string]string{}, Pools: []PoolCaching{}}
|
||||
k := readKstat(kstatDir + "arcstats")
|
||||
if k == nil {
|
||||
return d
|
||||
}
|
||||
d.Available = true
|
||||
l := &d.Lifetime
|
||||
l.Hits, l.Misses = k["hits"], k["misses"]
|
||||
l.DemandDataHits, l.DemandDataMisses = k["demand_data_hits"], k["demand_data_misses"]
|
||||
l.DemandMetaHits, l.DemandMetaMisses = k["demand_metadata_hits"], k["demand_metadata_misses"]
|
||||
l.PrefetchDataHits, l.PrefetchDataMisses = k["prefetch_data_hits"], k["prefetch_data_misses"]
|
||||
l.PrefetchMetaHits, l.PrefetchMetaMisses = k["prefetch_metadata_hits"], k["prefetch_metadata_misses"]
|
||||
l.L2Hits, l.L2Misses, l.L2ReadBytes, l.L2WriteBytes = k["l2_hits"], k["l2_misses"], k["l2_read_bytes"], k["l2_write_bytes"]
|
||||
l.L2Feeds, l.L2RebuildSize, l.EvictL2Eligible = k["l2_feeds"], k["l2_rebuild_size"], k["evict_l2_eligible"]
|
||||
l.Deleted, l.EvictSkip, l.MutexMiss, l.Prune, l.Throttle = k["deleted"], k["evict_skip"], k["mutex_miss"], k["arc_prune"], k["memory_throttle_count"]
|
||||
if z := readKstat(kstatDir + "zfetchstats"); z != nil {
|
||||
d.Zfetch.Hits, d.Zfetch.Misses, d.Zfetch.IOIssued = z["hits"], z["misses"], z["io_issued"]
|
||||
}
|
||||
if z := readKstat(kstatDir + "zil"); z != nil {
|
||||
d.ZIL.Commits = z["zil_commit_count"]
|
||||
d.ZIL.NormalBytes, d.ZIL.NormalWrite = z["zil_itx_metaslab_normal_bytes"], z["zil_itx_metaslab_normal_write"]
|
||||
d.ZIL.SlogBytes, d.ZIL.SlogWrite = z["zil_itx_metaslab_slog_bytes"], z["zil_itx_metaslab_slog_write"]
|
||||
}
|
||||
if z := readKstat(kstatDir + "dbufstats"); z != nil {
|
||||
d.DBuf.Size, d.DBuf.Target, d.DBuf.Hits, d.DBuf.Misses = z["cache_size_bytes"], z["cache_target_bytes"], z["hash_hits"], z["hash_misses"]
|
||||
}
|
||||
for _, t := range arcTunables {
|
||||
if b, err := os.ReadFile("/sys/module/zfs/parameters/" + t); err == nil {
|
||||
d.Tunables[t] = strings.TrimSpace(string(b))
|
||||
}
|
||||
}
|
||||
if b, err := os.ReadFile("/proc/meminfo"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if strings.HasPrefix(line, "MemTotal:") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) >= 2 {
|
||||
d.MemTotal = u64(f[1]) << 10
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
d.Pools = m.poolCaching(ctx)
|
||||
return d
|
||||
}
|
||||
|
||||
var cachingMu sync.Mutex
|
||||
var cachingAt time.Time
|
||||
var cachingLast []PoolCaching
|
||||
|
||||
func (m *Manager) poolCaching(ctx context.Context) []PoolCaching {
|
||||
cachingMu.Lock()
|
||||
defer cachingMu.Unlock()
|
||||
if cachingLast != nil && time.Since(cachingAt) < poolCachingEvery {
|
||||
return cachingLast
|
||||
}
|
||||
out := []PoolCaching{}
|
||||
bin, ok := lookPath("zpool")
|
||||
if !ok {
|
||||
return out
|
||||
}
|
||||
// zpool list -v needs no root; the vdev tree under each pool, with the class headers
|
||||
raw, code, err := m.runPlain(ctx, bin, "list", "-Hpv", "-o", "name,size,alloc,free,health")
|
||||
if err != nil || code != 0 {
|
||||
return out
|
||||
}
|
||||
out = parsePoolCaching(string(raw))
|
||||
if zfs, ok := lookPath("zfs"); ok {
|
||||
for i := range out {
|
||||
p := &out[i]
|
||||
if props, code, err := m.runPlain(ctx, zfs, "get", "-Hp", "-o", "property,value", "primarycache,secondarycache,sync,logbias,dedup,compression,recordsize", p.Pool); err == nil && code == 0 {
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(props)), "\n") {
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) < 2 {
|
||||
continue
|
||||
}
|
||||
switch f[0] {
|
||||
case "primarycache":
|
||||
p.PrimaryCache = f[1]
|
||||
case "secondarycache":
|
||||
p.SecondaryCache = f[1]
|
||||
case "sync":
|
||||
p.Sync = f[1]
|
||||
case "logbias":
|
||||
p.LogBias = f[1]
|
||||
case "dedup":
|
||||
p.DedupProp = f[1]
|
||||
case "compression":
|
||||
p.Compression = f[1]
|
||||
case "recordsize":
|
||||
p.RecordSize = humanSize(u64(f[1]))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, pool := range parsePools(func() []byte {
|
||||
b, _, _ := m.runPlain(ctx, bin, "list", "-Hp", "-o", "name,size,alloc,free,frag,cap,health,dedup")
|
||||
return b
|
||||
}()) {
|
||||
for i := range out {
|
||||
if out[i].Pool == pool.Name {
|
||||
out[i].DedupRatio = pool.Dedup
|
||||
}
|
||||
}
|
||||
}
|
||||
cachingLast, cachingAt = out, time.Now()
|
||||
return out
|
||||
}
|
||||
|
||||
// runPlain runs a command as ourselves, never through sudo: these are read-only zpool/zfs
|
||||
// queries that need no root on Linux, and a poll must not write to the auth log.
|
||||
func (m *Manager) runPlain(ctx context.Context, bin string, args ...string) ([]byte, int, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, bin, args...).Output()
|
||||
if ee, ok := err.(*exec.ExitError); ok {
|
||||
return out, ee.ExitCode(), nil
|
||||
}
|
||||
return out, 0, err
|
||||
}
|
||||
|
||||
var cacheClasses = map[string]bool{"logs": true, "log": true, "cache": true, "special": true, "dedup": true, "spares": true, "spare": true}
|
||||
|
||||
// parsePoolCaching reads `zpool list -Hpv -o name,size,alloc,free,health`: a pool line, then
|
||||
// its vdevs indented, with a bare class name ("logs", "cache", "special", "dedup", "spares")
|
||||
// introducing each extra class. Only those classes' members are kept; the data vdevs are the
|
||||
// pool and are shown elsewhere.
|
||||
func parsePoolCaching(text string) []PoolCaching {
|
||||
var out []PoolCaching
|
||||
var cur *PoolCaching
|
||||
class := ""
|
||||
for _, raw := range strings.Split(text, "\n") {
|
||||
if strings.TrimSpace(raw) == "" {
|
||||
continue
|
||||
}
|
||||
indented := strings.HasPrefix(raw, " ") || strings.HasPrefix(raw, "\t")
|
||||
f := strings.Split(strings.TrimSpace(raw), "\t")
|
||||
name := strings.TrimSpace(f[0])
|
||||
if !indented {
|
||||
out = append(out, PoolCaching{Pool: name, Cache: []CacheDev{}, Log: []CacheDev{}, Special: []CacheDev{}, Dedup: []CacheDev{}, Spare: []CacheDev{}})
|
||||
cur, class = &out[len(out)-1], ""
|
||||
continue
|
||||
}
|
||||
if cur == nil {
|
||||
continue
|
||||
}
|
||||
if cacheClasses[name] && (len(f) < 2 || f[1] == "-" || f[1] == "") {
|
||||
class = name
|
||||
continue
|
||||
}
|
||||
if class == "" {
|
||||
continue // a data vdev or one of its disks
|
||||
}
|
||||
// a mirror or raidz inside a class lists its members below it; keep them all, the
|
||||
// grouping is visible in the vdev tree on the pool card
|
||||
dev := CacheDev{Name: name}
|
||||
if len(f) >= 5 {
|
||||
dev.Size, dev.Alloc, dev.Health = u64(f[1]), u64(f[2]), strings.TrimSpace(f[4])
|
||||
if f[1] == "-" {
|
||||
dev.Size = 0
|
||||
}
|
||||
}
|
||||
switch class {
|
||||
case "logs", "log":
|
||||
cur.Log = append(cur.Log, dev)
|
||||
case "cache":
|
||||
cur.Cache = append(cur.Cache, dev)
|
||||
case "special":
|
||||
cur.Special = append(cur.Special, dev)
|
||||
case "dedup":
|
||||
cur.Dedup = append(cur.Dedup, dev)
|
||||
case "spares", "spare":
|
||||
cur.Spare = append(cur.Spare, dev)
|
||||
}
|
||||
}
|
||||
if out == nil {
|
||||
out = []PoolCaching{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func humanSize(b uint64) string {
|
||||
switch {
|
||||
case b >= 1<<20 && b%(1<<20) == 0:
|
||||
return strconv.FormatUint(b>>20, 10) + "M"
|
||||
case b >= 1<<10 && b%(1<<10) == 0:
|
||||
return strconv.FormatUint(b>>10, 10) + "K"
|
||||
}
|
||||
return strconv.FormatUint(b, 10)
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// zpool list -Hpv on a pool with every extra class; the data raidz and its disks are skipped
|
||||
const listV = "tank\t32000000000000\t23000000000000\t9000000000000\tONLINE\n" +
|
||||
"\traidz2-0\t32000000000000\t23000000000000\t9000000000000\tONLINE\n" +
|
||||
"\t\twwn-0x1\t4000000000000\t-\t-\tONLINE\n" +
|
||||
"\t\twwn-0x2\t4000000000000\t-\t-\tONLINE\n" +
|
||||
"\tspecial\t-\t-\t-\t-\n" +
|
||||
"\t\tmirror-1\t500000000000\t120000000000\t380000000000\tONLINE\n" +
|
||||
"\t\t\tnvme0n1p1\t500000000000\t-\t-\tONLINE\n" +
|
||||
"\t\t\tnvme1n1p1\t500000000000\t-\t-\tONLINE\n" +
|
||||
"\tlogs\t-\t-\t-\t-\n" +
|
||||
"\t\tnvme2n1p1\t16000000000\t4000000\t15996000000\tONLINE\n" +
|
||||
"\tcache\t-\t-\t-\t-\n" +
|
||||
"\t\tnvme2n1p2\t400000000000\t210000000000\t190000000000\tONLINE\n" +
|
||||
"\tspares\t-\t-\t-\t-\n" +
|
||||
"\t\twwn-0x9\t4000000000000\t-\t-\tAVAIL\n" +
|
||||
"scratch\t1000000000000\t1000000000\t999000000000\tONLINE\n" +
|
||||
"\tsda\t1000000000000\t1000000000\t999000000000\tONLINE\n"
|
||||
|
||||
func TestParsePoolCaching(t *testing.T) {
|
||||
got := parsePoolCaching(listV)
|
||||
if len(got) != 2 || got[0].Pool != "tank" || got[1].Pool != "scratch" {
|
||||
t.Fatalf("pools: %+v", got)
|
||||
}
|
||||
tank := got[0]
|
||||
if len(tank.Cache) != 1 || tank.Cache[0].Name != "nvme2n1p2" || tank.Cache[0].Alloc != 210000000000 {
|
||||
t.Errorf("cache: %+v", tank.Cache)
|
||||
}
|
||||
if len(tank.Log) != 1 || tank.Log[0].Name != "nvme2n1p1" {
|
||||
t.Errorf("log: %+v", tank.Log)
|
||||
}
|
||||
if len(tank.Special) != 3 || tank.Special[0].Name != "mirror-1" || tank.Special[0].Alloc != 120000000000 {
|
||||
t.Errorf("special: %+v", tank.Special)
|
||||
}
|
||||
if len(tank.Spare) != 1 || tank.Spare[0].Health != "AVAIL" {
|
||||
t.Errorf("spares: %+v", tank.Spare)
|
||||
}
|
||||
if len(tank.Dedup) != 0 {
|
||||
t.Errorf("no dedup class: %+v", tank.Dedup)
|
||||
}
|
||||
s := got[1]
|
||||
if len(s.Cache)+len(s.Log)+len(s.Special)+len(s.Spare) != 0 {
|
||||
t.Errorf("a plain pool has no extra classes: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHumanSize(t *testing.T) {
|
||||
for in, want := range map[uint64]string{131072: "128K", 1048576: "1M", 16384: "16K", 1000: "1000"} {
|
||||
if got := humanSize(in); got != want {
|
||||
t.Errorf("%d → %s, want %s", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestLiveARC reads this machine's ZFS for real: HAMLAB_LIVE=1 go test -run TestLiveARC -v ./internal/storage
|
||||
func TestLiveARC(t *testing.T) {
|
||||
if os.Getenv("HAMLAB_LIVE") == "" {
|
||||
t.Skip("set HAMLAB_LIVE=1 to read the real machine")
|
||||
}
|
||||
d := New(false).ARC(context.Background())
|
||||
b, _ := json.MarshalIndent(d, "", " ")
|
||||
t.Logf("%s", b)
|
||||
}
|
||||
@@ -0,0 +1,493 @@
|
||||
package storage
|
||||
|
||||
// Scheduled snapshots with a retention period: take a snapshot of a dataset on a schedule,
|
||||
// and drop the ones that have aged out.
|
||||
//
|
||||
// **The naming is zfSnap's, deliberately.** grid has run zfSnap v1.11.1 from root cron since
|
||||
// before this repo existed and it works well, so rather than invent a second convention and
|
||||
// have two tools disagree about what a snapshot is, this writes and reads exactly what
|
||||
// zfSnap does: `<dataset>@2026-09-24_02.30.01--6w`, with the retention encoded in the name.
|
||||
// That has three consequences worth knowing. Ham Lab can show and manage the snapshots
|
||||
// zfSnap already took. The two can run side by side without fighting, because both only
|
||||
// ever destroy snapshots that have passed the expiry written into their own name. And
|
||||
// changing a policy's retention does not retroactively re-date old snapshots — each one
|
||||
// carries the rule it was taken under, which is the behaviour you want from a backup.
|
||||
//
|
||||
// Two rules make this safe to run unattended. Only a snapshot whose name parses as a stamp
|
||||
// and a TTL is ever a candidate — anything taken by hand or named anything else is invisible
|
||||
// to it. And every policy can be asked what it would delete, before it deletes anything.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/cron"
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// stampLayout is zfSnap's timestamp: 2026-09-24_02.30.01.
|
||||
const stampLayout = "2006-01-02_15.04.05"
|
||||
|
||||
// ttlSep separates the stamp from the retention in a snapshot name, as zfSnap writes it.
|
||||
const ttlSep = "--"
|
||||
|
||||
// Policy is a standing instruction: snapshot this dataset on this schedule, keep the
|
||||
// snapshots this long.
|
||||
type Policy struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Dataset string `json:"dataset"`
|
||||
// Recursive takes one atomic snapshot of the dataset and everything under it (zfs -r).
|
||||
Recursive bool `json:"recursive"`
|
||||
// Every is hourly | daily | weekly | monthly | custom. At is "HH:MM" (the minute alone
|
||||
// is used for hourly). Weekday is 0=Sunday, Day is 1–28. With "custom", Cron is a plain
|
||||
// cron expression and the rest is ignored — the escape hatch for anything the four
|
||||
// common shapes cannot say.
|
||||
Every string `json:"every"`
|
||||
At string `json:"at"`
|
||||
Weekday int `json:"weekday"`
|
||||
Day int `json:"day"`
|
||||
Cron string `json:"cron,omitempty"`
|
||||
// KeepFor is how long a snapshot lives: "6w", "30d", "12h". It is written into the
|
||||
// snapshot's own name, zfSnap-style, so each one carries the rule it was taken under.
|
||||
// KeepMin is a floor — that many newest snapshots survive however old they are, so a
|
||||
// dataset is never left with none.
|
||||
KeepFor string `json:"keep_for"`
|
||||
KeepMin int `json:"keep_min"`
|
||||
Prefix string `json:"prefix"`
|
||||
Enabled bool `json:"enabled"`
|
||||
|
||||
}
|
||||
|
||||
func (p *Policy) prefix() string { return strings.TrimSpace(p.Prefix) }
|
||||
|
||||
// Validate reports what is wrong with a policy, or nil.
|
||||
func (p *Policy) Validate() error {
|
||||
if !datasetRe.MatchString(p.Dataset) {
|
||||
return errors.New("dataset: a pool or pool/dataset name")
|
||||
}
|
||||
if pre := p.prefix(); pre != "" && (!snapRe.MatchString(pre) || strings.Contains(pre, ttlSep)) {
|
||||
return errors.New("prefix: letters, digits, . _ : and - only, and no --")
|
||||
}
|
||||
switch p.Every {
|
||||
case "hourly", "daily", "weekly", "monthly":
|
||||
case "custom":
|
||||
if strings.TrimSpace(p.Cron) == "" {
|
||||
return errors.New("a custom schedule needs a cron expression")
|
||||
}
|
||||
default:
|
||||
return errors.New("every: hourly, daily, weekly, monthly or custom")
|
||||
}
|
||||
if _, err := p.CronExpr(); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, _, err := p.clock(); err != nil {
|
||||
return err
|
||||
}
|
||||
if p.Every == "weekly" && (p.Weekday < 0 || p.Weekday > 6) {
|
||||
return errors.New("weekday: 0 (Sunday) to 6")
|
||||
}
|
||||
if p.Every == "monthly" && (p.Day < 1 || p.Day > 28) {
|
||||
return errors.New("day: 1 to 28, so every month has one")
|
||||
}
|
||||
if p.KeepFor != "" {
|
||||
if _, err := ParseAge(p.KeepFor); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if p.KeepMin < 0 {
|
||||
return errors.New("keep at least: 0 or more")
|
||||
}
|
||||
if p.KeepFor == "" {
|
||||
return errors.New("a retention period is required — it is written into each snapshot's name, and is what lets it be cleaned up later")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *Policy) clock() (hour, min int, err error) {
|
||||
at := strings.TrimSpace(p.At)
|
||||
if at == "" {
|
||||
return 0, 0, nil
|
||||
}
|
||||
h, m, ok := strings.Cut(at, ":")
|
||||
if !ok {
|
||||
return 0, 0, errors.New("at: HH:MM")
|
||||
}
|
||||
hour, err = strconv.Atoi(h)
|
||||
if err == nil {
|
||||
min, err = strconv.Atoi(m)
|
||||
}
|
||||
if err != nil || hour < 0 || hour > 23 || min < 0 || min > 59 {
|
||||
return 0, 0, errors.New("at: HH:MM, 00:00 to 23:59")
|
||||
}
|
||||
return hour, min, nil
|
||||
}
|
||||
|
||||
// ParseAge reads "6w", "30d", "12h", "90m" as a duration. Weeks and days are what a
|
||||
// retention period is actually written in, and time.ParseDuration knows neither.
|
||||
func ParseAge(s string) (time.Duration, error) {
|
||||
s = strings.TrimSpace(strings.ToLower(s))
|
||||
if s == "" {
|
||||
return 0, errors.New("empty period")
|
||||
}
|
||||
unit := s[len(s)-1]
|
||||
n, err := strconv.Atoi(s[:len(s)-1])
|
||||
if err != nil || n <= 0 {
|
||||
return 0, fmt.Errorf("%q: a number and a unit, like 6w, 30d or 12h", s)
|
||||
}
|
||||
switch unit {
|
||||
case 'm':
|
||||
return time.Duration(n) * time.Minute, nil
|
||||
case 'h':
|
||||
return time.Duration(n) * time.Hour, nil
|
||||
case 'd':
|
||||
return time.Duration(n) * 24 * time.Hour, nil
|
||||
case 'w':
|
||||
return time.Duration(n) * 7 * 24 * time.Hour, nil
|
||||
}
|
||||
return 0, fmt.Errorf("%q: the unit is m, h, d or w", s)
|
||||
}
|
||||
|
||||
// CronExpr renders the schedule as a cron expression — which is what actually runs it. The
|
||||
// four common shapes are sugar over this; a custom policy states it outright. One expression
|
||||
// is the only description of when a schedule fires, so the page, the cron file and the next
|
||||
// run can never disagree.
|
||||
func (p *Policy) CronExpr() (string, error) {
|
||||
if p.Every == "custom" {
|
||||
expr := strings.Join(strings.Fields(p.Cron), " ")
|
||||
if err := cron.ValidSchedule(expr); err != nil {
|
||||
return "", fmt.Errorf("cron expression: %w", err)
|
||||
}
|
||||
if strings.HasPrefix(expr, "@reboot") {
|
||||
return "", errors.New("@reboot is not a schedule for snapshots")
|
||||
}
|
||||
return expr, nil
|
||||
}
|
||||
hour, min, err := p.clock()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
switch p.Every {
|
||||
case "hourly":
|
||||
return fmt.Sprintf("%d * * * *", min), nil
|
||||
case "daily":
|
||||
return fmt.Sprintf("%d %d * * *", min, hour), nil
|
||||
case "weekly":
|
||||
return fmt.Sprintf("%d %d * * %d", min, hour, p.Weekday), nil
|
||||
case "monthly":
|
||||
return fmt.Sprintf("%d %d %d * *", min, hour, p.Day), nil
|
||||
}
|
||||
return "", errors.New("unknown schedule")
|
||||
}
|
||||
|
||||
// NextRun is when cron will next fire this schedule, read with the same evaluator the Jobs
|
||||
// tab uses on everyone else's lines.
|
||||
func (p *Policy) NextRun(now time.Time) time.Time {
|
||||
expr, err := p.CronExpr()
|
||||
if err != nil {
|
||||
return time.Time{}
|
||||
}
|
||||
t, ok := cron.NextRun(expr, now)
|
||||
if !ok {
|
||||
return time.Time{}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// SnapTime reads a snapshot's name in zfSnap's convention and returns when it was taken,
|
||||
// when it expires, and whether the name parsed at all. A name without a TTL never expires.
|
||||
// An optional prefix in front of the stamp is allowed, so a policy can label its own.
|
||||
func SnapTime(name string) (taken time.Time, expires time.Time, ok bool) {
|
||||
_, snap, cut := strings.Cut(name, "@")
|
||||
if !cut {
|
||||
return
|
||||
}
|
||||
body, ttl, hasTTL := strings.Cut(snap, ttlSep)
|
||||
// an optional "<prefix>-" before the stamp
|
||||
stamp := body
|
||||
if len(body) > len(stampLayout) {
|
||||
stamp = body[len(body)-len(stampLayout):]
|
||||
}
|
||||
t, err := time.ParseInLocation(stampLayout, stamp, time.Local)
|
||||
if err != nil {
|
||||
return time.Time{}, time.Time{}, false
|
||||
}
|
||||
taken, ok = t, true
|
||||
if hasTTL {
|
||||
if d, err := ParseAge(ttl); err == nil {
|
||||
expires = t.Add(d)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Doomed is which of this dataset's snapshots have passed the expiry written into their own
|
||||
// names, oldest first — exactly what zfSnap -d would remove. This is what the UI shows
|
||||
// before anything is deleted, and what Run then acts on. KeepMin holds back that many of the
|
||||
// newest whatever their names say, so a dataset is never left with none.
|
||||
func (m *Manager) Doomed(ctx context.Context, p *Policy) ([]Snapshot, error) {
|
||||
if err := p.Validate(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
all, err := m.Snapshots(ctx, p.Dataset)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var dated []Snapshot
|
||||
expiry := map[string]time.Time{}
|
||||
for _, s := range all {
|
||||
taken, exp, ok := SnapTime(s.Name)
|
||||
if !ok || exp.IsZero() {
|
||||
continue // hand-made, or kept forever: not ours to remove
|
||||
}
|
||||
_ = taken
|
||||
dated = append(dated, s)
|
||||
expiry[s.Name] = exp
|
||||
}
|
||||
sort.Slice(dated, func(i, j int) bool { return dated[i].Created > dated[j].Created }) // newest first
|
||||
now := time.Now()
|
||||
var doomed []Snapshot
|
||||
for i, s := range dated {
|
||||
if i < p.KeepMin {
|
||||
continue // the floor: these survive however old they are
|
||||
}
|
||||
if expiry[s.Name].Before(now) {
|
||||
doomed = append(doomed, s)
|
||||
}
|
||||
}
|
||||
sort.Slice(doomed, func(i, j int) bool { return doomed[i].Created < doomed[j].Created }) // oldest first
|
||||
return doomed, nil
|
||||
}
|
||||
|
||||
// DestroySnapshot removes one snapshot. recursive also removes the same-named snapshot on
|
||||
// every dataset beneath it, which is what a recursive policy created.
|
||||
func (m *Manager) DestroySnapshot(ctx context.Context, full string, recursive bool) error {
|
||||
dataset, snap, ok := strings.Cut(full, "@")
|
||||
if !ok || !datasetRe.MatchString(dataset) || !snapRe.MatchString(snap) {
|
||||
return errors.New("bad snapshot name")
|
||||
}
|
||||
args := []string{"zfs", "destroy"}
|
||||
if recursive {
|
||||
args = append(args, "-r")
|
||||
}
|
||||
out, code, err := m.run(ctx, append(args, full)...)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if code != 0 {
|
||||
return errors.New(strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SnapshotRecursive takes dataset@name, optionally of everything beneath it too.
|
||||
func (m *Manager) SnapshotRecursive(ctx context.Context, dataset, name string, recursive bool) error {
|
||||
if !datasetRe.MatchString(dataset) || !snapRe.MatchString(name) {
|
||||
return errors.New("bad snapshot name")
|
||||
}
|
||||
args := []string{"zfs", "snapshot"}
|
||||
if recursive {
|
||||
args = append(args, "-r")
|
||||
}
|
||||
out, code, err := m.run(ctx, append(args, dataset+"@"+name)...)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if code != 0 {
|
||||
return errors.New(strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RunResult is what one pass of a policy did.
|
||||
type RunResult struct {
|
||||
Taken string `json:"taken"`
|
||||
Pruned []string `json:"pruned"`
|
||||
Skipped int `json:"skipped"` // snapshots that are not this policy's, left alone
|
||||
}
|
||||
|
||||
// Run takes the snapshot and prunes what has aged out. It is the same code the scheduler
|
||||
// calls, so "Run now" and the timer cannot drift apart.
|
||||
func (m *Manager) Run(ctx context.Context, p *Policy) (RunResult, error) {
|
||||
var res RunResult
|
||||
if err := p.Validate(); err != nil {
|
||||
return res, err
|
||||
}
|
||||
name := time.Now().Format(stampLayout)
|
||||
if pre := strings.TrimSpace(p.Prefix); pre != "" {
|
||||
name = pre + "-" + name
|
||||
}
|
||||
if p.KeepFor != "" {
|
||||
name += ttlSep + strings.TrimSpace(strings.ToLower(p.KeepFor)) // zfSnap's convention: the retention lives in the name
|
||||
}
|
||||
if err := m.SnapshotRecursive(ctx, p.Dataset, name, p.Recursive); err != nil {
|
||||
return res, err
|
||||
}
|
||||
res.Taken = p.Dataset + "@" + name
|
||||
doomed, err := m.Doomed(ctx, p)
|
||||
if err != nil {
|
||||
return res, err // the snapshot was taken; the prune is what failed
|
||||
}
|
||||
for _, s := range doomed {
|
||||
if err := m.DestroySnapshot(ctx, s.Name, p.Recursive); err != nil {
|
||||
return res, fmt.Errorf("destroying %s: %w", s.Name, err)
|
||||
}
|
||||
res.Pruned = append(res.Pruned, s.Name)
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// ---- what happened when it ran ----
|
||||
//
|
||||
// The policies are a definition the daemon owns; what a run did is written by whatever ran
|
||||
// it, which is a cron job in its own process. Keeping the two apart means the CLI never has
|
||||
// to write state.json behind the daemon's back — the last writer would win and one of them
|
||||
// would lose its work.
|
||||
|
||||
// RunRecord is the outcome of one run of one schedule.
|
||||
type RunRecord struct {
|
||||
At time.Time `json:"at"`
|
||||
Taken string `json:"taken,omitempty"`
|
||||
Pruned []string `json:"pruned,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
func runPath(dir, id string) string { return filepath.Join(dir, "zfs-runs", id+".json") }
|
||||
|
||||
// WriteRun records what a run did, beside the config.
|
||||
func WriteRun(dir, id string, r RunRecord) error {
|
||||
if err := os.MkdirAll(filepath.Join(dir, "zfs-runs"), 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := json.Marshal(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
p := runPath(dir, id)
|
||||
tmp := p + ".tmp"
|
||||
if err := os.WriteFile(tmp, b, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, p)
|
||||
}
|
||||
|
||||
// ReadRun is the last thing a schedule did, or the zero value if it has never run.
|
||||
func ReadRun(dir, id string) RunRecord {
|
||||
var r RunRecord
|
||||
b, err := os.ReadFile(runPath(dir, id))
|
||||
if err == nil {
|
||||
_ = json.Unmarshal(b, &r)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// ForgetRun drops a schedule's record, for when the schedule itself goes.
|
||||
func ForgetRun(dir, id string) { _ = os.Remove(runPath(dir, id)) }
|
||||
|
||||
// ---- the cron file ----
|
||||
|
||||
// CronFile is where Ham Lab writes the snapshot schedules. One file it owns entirely and
|
||||
// rewrites whole, so there is never a half-updated set of lines, and so a schedule removed
|
||||
// in the UI is removed from the machine.
|
||||
const CronFile = "/etc/cron.d/hamlab-zfs"
|
||||
|
||||
// CronFileText renders every schedule as cron lines. user is who they run as — the account
|
||||
// Ham Lab itself runs as, which reaches zfs through sudo exactly as the daemon does, so the
|
||||
// files it writes stay owned by that account rather than by root.
|
||||
func CronFileText(policies []Policy, user, exe, config string) (string, error) {
|
||||
var b strings.Builder
|
||||
b.WriteString("# Ham Lab snapshot schedules — written by the ZFS tab, rewritten whole on every change.\n")
|
||||
b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
|
||||
b.WriteString("SHELL=/bin/sh\n")
|
||||
b.WriteString("PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
|
||||
for i := range policies {
|
||||
p := &policies[i]
|
||||
if !p.Enabled {
|
||||
b.WriteString("# " + p.Name + " — paused\n")
|
||||
continue
|
||||
}
|
||||
expr, err := p.CronExpr()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%s: %w", p.Name, err)
|
||||
}
|
||||
name := strings.ReplaceAll(p.Name, "\n", " ")
|
||||
fmt.Fprintf(&b, "# %s — %s, keep %s\n", name, p.Dataset, p.KeepFor)
|
||||
fmt.Fprintf(&b, "%s %s %s -config %s zfs-snapshot %s\n\n", expr, user, exe, config, p.ID)
|
||||
}
|
||||
return b.String(), nil
|
||||
}
|
||||
|
||||
// WriteCronFile puts the schedules on the machine, or removes the file when there are none.
|
||||
func (m *Manager) WriteCronFile(ctx context.Context, policies []Policy, user, exe, config string) error {
|
||||
live := false
|
||||
for i := range policies {
|
||||
if policies[i].Enabled {
|
||||
live = true
|
||||
}
|
||||
}
|
||||
if !live {
|
||||
out, code, err := m.run(ctx, "rm", "-f", CronFile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if code != 0 {
|
||||
return errors.New(strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
text, err := CronFileText(policies, user, exe, config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return m.writeRoot(ctx, CronFile, text)
|
||||
}
|
||||
|
||||
// WriteRootFile puts text at a root-owned path, through the same privilege escalation every
|
||||
// other write here uses. Shared by the snapshot schedules and the image-update schedules,
|
||||
// which both own a file in /etc/cron.d.
|
||||
func (m *Manager) WriteRootFile(ctx context.Context, path, text string) error {
|
||||
return m.writeRoot(ctx, path, text)
|
||||
}
|
||||
|
||||
// RemoveRootFile deletes one, for when the last schedule of a kind goes away.
|
||||
func (m *Manager) RemoveRootFile(ctx context.Context, path string) error {
|
||||
out, code, err := m.run(ctx, "rm", "-f", path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if code != 0 {
|
||||
return errors.New(strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manager) writeRoot(ctx context.Context, path, text string) error {
|
||||
args := []string{"tee", path}
|
||||
if m.sudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
cmd.Stdin = strings.NewReader(text)
|
||||
var errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = nil, &errb
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := strings.TrimSpace(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return errors.New(msg)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/cron"
|
||||
)
|
||||
|
||||
func TestSnapTimeReadsZfSnapNames(t *testing.T) {
|
||||
// the real thing off grid
|
||||
taken, exp, ok := SnapTime("tank@2026-09-24_02.30.01--6w")
|
||||
if !ok {
|
||||
t.Fatal("zfSnap's own name must parse")
|
||||
}
|
||||
if taken.Format("2006-01-02 15:04:05") != "2026-09-24 02:30:01" {
|
||||
t.Fatalf("taken = %v", taken)
|
||||
}
|
||||
if got := exp.Sub(taken); got != 6*7*24*time.Hour {
|
||||
t.Fatalf("ttl = %v, want 6w", got)
|
||||
}
|
||||
// a labelled one from a policy with a prefix
|
||||
if _, e, ok := SnapTime("tank/apps@nightly-2026-09-24_02.30.01--30d"); !ok || e.IsZero() {
|
||||
t.Fatal("a prefixed name must parse and expire")
|
||||
}
|
||||
// no TTL: never expires
|
||||
if _, e, ok := SnapTime("tank@2026-09-24_02.30.01"); !ok || !e.IsZero() {
|
||||
t.Fatal("a name without a TTL must parse and never expire")
|
||||
}
|
||||
// things that are not ours
|
||||
for _, n := range []string{"tank@before-upgrade", "tank@manual", "tank", "tank@2026-13-45_99.99.99--6w"} {
|
||||
if _, _, ok := SnapTime(n); ok {
|
||||
t.Errorf("%q must not parse as a scheduled snapshot", n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCronExprAndNextRun(t *testing.T) {
|
||||
now := time.Date(2026, 9, 24, 9, 15, 0, 0, time.Local)
|
||||
for _, c := range []struct {
|
||||
p Policy
|
||||
want string
|
||||
}{
|
||||
{Policy{Every: "daily", At: "02:30"}, "30 2 * * *"},
|
||||
{Policy{Every: "hourly", At: "00:30"}, "30 * * * *"},
|
||||
{Policy{Every: "weekly", At: "04:00", Weekday: 0}, "0 4 * * 0"},
|
||||
{Policy{Every: "monthly", At: "05:15", Day: 12}, "15 5 12 * *"},
|
||||
{Policy{Every: "custom", Cron: "*/15 9-17 * * 1-5"}, "*/15 9-17 * * 1-5"},
|
||||
} {
|
||||
got, err := c.p.CronExpr()
|
||||
if err != nil || got != c.want {
|
||||
t.Errorf("CronExpr(%+v) = %q, %v; want %q", c.p.Every, got, err, c.want)
|
||||
}
|
||||
}
|
||||
// the next run is read with the same evaluator the Jobs tab uses
|
||||
daily := Policy{Every: "daily", At: "02:30"}
|
||||
if n := daily.NextRun(now); n.Hour() != 2 || n.Minute() != 30 || n.Day() != 25 {
|
||||
t.Fatalf("NextRun = %v, want 25th 02:30", n)
|
||||
}
|
||||
// a custom expression that does not parse is refused, not silently ignored
|
||||
if _, err := (&Policy{Every: "custom", Cron: "not a schedule"}).CronExpr(); err == nil {
|
||||
t.Fatal("a bad cron expression must be refused")
|
||||
}
|
||||
if _, err := (&Policy{Every: "custom", Cron: "@reboot"}).CronExpr(); err == nil {
|
||||
t.Fatal("@reboot is not a snapshot schedule")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCronFileText(t *testing.T) {
|
||||
ps := []Policy{
|
||||
{ID: "abc", Name: "tank nightly", Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6w", Enabled: true},
|
||||
{ID: "def", Name: "paused one", Dataset: "tank/apps", Every: "hourly", At: "00:05", KeepFor: "2d"},
|
||||
}
|
||||
text, err := CronFileText(ps, "core", "/usr/local/bin/hamlab", "/home/core/apps/hamlab/config.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(text, "30 2 * * * core /usr/local/bin/hamlab -config /home/core/apps/hamlab/config.json zfs-snapshot abc") {
|
||||
t.Fatalf("the enabled schedule is missing or malformed:\n%s", text)
|
||||
}
|
||||
if strings.Contains(text, "zfs-snapshot def") {
|
||||
t.Fatalf("a paused schedule must not be scheduled:\n%s", text)
|
||||
}
|
||||
// every line cron will read must parse as a schedule
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") || strings.Contains(line, "=") {
|
||||
continue
|
||||
}
|
||||
f := strings.Fields(line)
|
||||
if err := cron.ValidSchedule(strings.Join(f[:5], " ")); err != nil {
|
||||
t.Errorf("line %q: %v", line, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRefusesNonsense(t *testing.T) {
|
||||
ok := Policy{Dataset: "tank/apps", Every: "daily", At: "02:30", KeepFor: "6w"}
|
||||
if err := ok.Validate(); err != nil {
|
||||
t.Fatalf("a sane policy: %v", err)
|
||||
}
|
||||
bad := []Policy{
|
||||
{Dataset: "", Every: "daily", At: "02:30", KeepFor: "6w"},
|
||||
{Dataset: "tank", Every: "yearly", At: "02:30", KeepFor: "6w"},
|
||||
{Dataset: "tank", Every: "custom", KeepFor: "6w"},
|
||||
{Dataset: "tank", Every: "custom", Cron: "not a schedule", KeepFor: "6w"},
|
||||
{Dataset: "tank", Every: "daily", At: "25:00", KeepFor: "6w"},
|
||||
{Dataset: "tank", Every: "daily", At: "02:30"}, // no retention
|
||||
{Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6 weeks"}, // unparseable
|
||||
{Dataset: "tank", Every: "monthly", At: "02:30", Day: 31, KeepFor: "6w"}, // not every month has one
|
||||
{Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6w", Prefix: "a--b"},
|
||||
}
|
||||
for i, p := range bad {
|
||||
if err := p.Validate(); err == nil {
|
||||
t.Errorf("bad policy %d was accepted: %+v", i, p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseAge(t *testing.T) {
|
||||
for s, want := range map[string]time.Duration{
|
||||
"6w": 6 * 7 * 24 * time.Hour, "30d": 30 * 24 * time.Hour,
|
||||
"12h": 12 * time.Hour, "90m": 90 * time.Minute,
|
||||
} {
|
||||
if got, err := ParseAge(s); err != nil || got != want {
|
||||
t.Errorf("ParseAge(%q) = %v, %v; want %v", s, got, err, want)
|
||||
}
|
||||
}
|
||||
for _, s := range []string{"", "6", "w", "-1d", "0w", "6y", "six weeks"} {
|
||||
if _, err := ParseAge(s); err == nil {
|
||||
t.Errorf("ParseAge(%q) should fail", s)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,926 @@
|
||||
// Package storage is the Disks tab: block devices from lsblk, SMART health via
|
||||
// smartctl (optional; the tab explains how to get it), and ZFS pools/datasets/snapshots
|
||||
// when the zfs tools and module are present. Spinning disks are never woken just to
|
||||
// read SMART (-n standby).
|
||||
package storage
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// ---- block devices ----
|
||||
|
||||
type Part struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"` // part | crypt | lvm | md | …
|
||||
FSType string `json:"fstype,omitempty"`
|
||||
Mount string `json:"mount,omitempty"`
|
||||
Label string `json:"label,omitempty"`
|
||||
Size uint64 `json:"size"`
|
||||
Used uint64 `json:"used,omitempty"`
|
||||
FSSize uint64 `json:"fs_size,omitempty"`
|
||||
}
|
||||
|
||||
type Device struct {
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"` // nvme | ssd | hdd | usb | emmc | sd | other
|
||||
Model string `json:"model,omitempty"`
|
||||
Vendor string `json:"vendor,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Tran string `json:"tran,omitempty"`
|
||||
Size uint64 `json:"size"`
|
||||
Rotational bool `json:"rotational"`
|
||||
Removable bool `json:"removable"`
|
||||
Temp float64 `json:"temp,omitempty"`
|
||||
Parts []Part `json:"parts"`
|
||||
Namespaces []string `json:"namespaces,omitempty"` // NVMe: the block devices behind this controller
|
||||
}
|
||||
|
||||
type lsblkNode struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Size uint64 `json:"size"`
|
||||
Model *string `json:"model"`
|
||||
Serial *string `json:"serial"`
|
||||
Vendor *string `json:"vendor"`
|
||||
Rota bool `json:"rota"`
|
||||
Tran *string `json:"tran"`
|
||||
FSType *string `json:"fstype"`
|
||||
Mountpoint *string `json:"mountpoint"`
|
||||
Label *string `json:"label"`
|
||||
RM bool `json:"rm"`
|
||||
Children []lsblkNode `json:"children"`
|
||||
}
|
||||
|
||||
func str(p *string) string {
|
||||
if p == nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(*p)
|
||||
}
|
||||
|
||||
func Devices(ctx context.Context) ([]Device, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
var root struct {
|
||||
Blockdevices []lsblkNode `json:"blockdevices"`
|
||||
}
|
||||
// lsblk knows labels and stacked devices (LVM, LUKS, md); without it (Alpine and other
|
||||
// busybox systems don't ship it) or with one too old for JSON, /sys has the essentials.
|
||||
out, err := exec.CommandContext(ctx, "lsblk", "-J", "-b", "-o", "NAME,TYPE,SIZE,MODEL,SERIAL,VENDOR,ROTA,TRAN,FSTYPE,MOUNTPOINT,LABEL,RM").Output()
|
||||
if err != nil || json.Unmarshal(out, &root) != nil {
|
||||
root.Blockdevices = sysBlockDevices()
|
||||
}
|
||||
devs := []Device{}
|
||||
for _, n := range root.Blockdevices {
|
||||
if n.Type != "disk" || n.Size == 0 || strings.HasPrefix(n.Name, "zd") || mmcAreaRe.MatchString(n.Name) {
|
||||
continue
|
||||
}
|
||||
d := Device{Name: n.Name, Model: str(n.Model), Vendor: str(n.Vendor), Serial: str(n.Serial), Tran: str(n.Tran),
|
||||
Size: n.Size, Rotational: n.Rota, Removable: n.RM, Temp: blockTemp(n.Name), Parts: []Part{}}
|
||||
switch {
|
||||
case strings.HasPrefix(n.Name, "mmcblk"):
|
||||
d.Kind = mmcKind(n.Name)
|
||||
if d.Model == "" {
|
||||
d.Model = strings.TrimSpace(readStrFile("/sys/block/" + n.Name + "/device/name"))
|
||||
}
|
||||
case d.Tran == "usb":
|
||||
d.Kind = "usb"
|
||||
case d.Tran == "nvme" || strings.HasPrefix(n.Name, "nvme"):
|
||||
d.Kind = "nvme"
|
||||
case n.Rota:
|
||||
d.Kind = "hdd"
|
||||
case d.Tran == "sata" || d.Tran == "sas" || strings.HasPrefix(n.Name, "sd"):
|
||||
d.Kind = "ssd"
|
||||
default:
|
||||
d.Kind = "other"
|
||||
}
|
||||
var walk func(nodes []lsblkNode)
|
||||
walk = func(nodes []lsblkNode) {
|
||||
for _, c := range nodes {
|
||||
p := Part{Name: c.Name, Type: c.Type, FSType: str(c.FSType), Mount: str(c.Mountpoint), Label: str(c.Label), Size: c.Size}
|
||||
if p.Mount != "" && p.Mount != "[SWAP]" {
|
||||
var st syscall.Statfs_t
|
||||
if syscall.Statfs(p.Mount, &st) == nil {
|
||||
bs := uint64(st.Bsize)
|
||||
p.FSSize, p.Used = st.Blocks*bs, (st.Blocks-st.Bfree)*bs
|
||||
}
|
||||
}
|
||||
d.Parts = append(d.Parts, p)
|
||||
walk(c.Children)
|
||||
}
|
||||
}
|
||||
walk(n.Children)
|
||||
devs = append(devs, d)
|
||||
}
|
||||
devs = foldNVMe(devs)
|
||||
sort.Slice(devs, func(i, j int) bool { return devs[i].Name < devs[j].Name })
|
||||
return devs, nil
|
||||
}
|
||||
|
||||
// sysBlockDevices builds the same tree lsblk would from /sys/block and /proc/mounts.
|
||||
func sysBlockDevices() []lsblkNode {
|
||||
type mnt struct{ point, fstype string }
|
||||
mounts := map[string]mnt{}
|
||||
if b, err := os.ReadFile("/proc/mounts"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if f := strings.Fields(line); len(f) >= 3 && strings.HasPrefix(f[0], "/dev/") {
|
||||
dev := f[0]
|
||||
if real, err := filepath.EvalSymlinks(dev); err == nil { // /dev/disk/by-uuid/…, /dev/mapper/…
|
||||
dev = real
|
||||
}
|
||||
if _, seen := mounts[filepath.Base(dev)]; !seen {
|
||||
mounts[filepath.Base(dev)] = mnt{strings.ReplaceAll(f[1], `\040`, " "), f[2]}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if b, err := os.ReadFile("/proc/swaps"); err == nil {
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if f := strings.Fields(line); len(f) > 0 && strings.HasPrefix(f[0], "/dev/") {
|
||||
mounts[filepath.Base(f[0])] = mnt{"[SWAP]", "swap"}
|
||||
}
|
||||
}
|
||||
}
|
||||
sp := func(v string) *string {
|
||||
if v = strings.TrimSpace(v); v == "" {
|
||||
return nil
|
||||
}
|
||||
return &v
|
||||
}
|
||||
node := func(dir, name, typ string) lsblkNode {
|
||||
n := lsblkNode{Name: name, Type: typ, Size: u64(strings.TrimSpace(readStrFile(dir+"/size"))) * 512}
|
||||
if m, ok := mounts[name]; ok {
|
||||
n.Mountpoint, n.FSType = sp(m.point), sp(m.fstype)
|
||||
}
|
||||
return n
|
||||
}
|
||||
var out []lsblkNode
|
||||
ents, _ := os.ReadDir("/sys/block")
|
||||
for _, e := range ents {
|
||||
name := e.Name()
|
||||
dir := "/sys/block/" + name
|
||||
if strings.HasPrefix(name, "loop") || strings.HasPrefix(name, "ram") || strings.HasPrefix(name, "zram") ||
|
||||
strings.HasPrefix(name, "dm-") || strings.HasPrefix(name, "md") || strings.HasPrefix(name, "sr") || strings.HasPrefix(name, "nbd") {
|
||||
continue
|
||||
}
|
||||
n := node(dir, name, "disk")
|
||||
n.Model, n.Vendor = sp(readStrFile(dir+"/device/model")), sp(readStrFile(dir+"/device/vendor"))
|
||||
n.Serial = sp(readStrFile(dir + "/device/serial"))
|
||||
n.Rota = strings.TrimSpace(readStrFile(dir+"/queue/rotational")) == "1"
|
||||
n.RM = strings.TrimSpace(readStrFile(dir+"/removable")) == "1"
|
||||
real, _ := filepath.EvalSymlinks(dir)
|
||||
switch {
|
||||
case strings.HasPrefix(name, "nvme"):
|
||||
n.Tran = sp("nvme")
|
||||
case strings.Contains(real, "/usb"):
|
||||
n.Tran = sp("usb")
|
||||
case strings.Contains(real, "/ata"):
|
||||
n.Tran = sp("sata")
|
||||
}
|
||||
subs, _ := os.ReadDir(dir)
|
||||
for _, sub := range subs {
|
||||
if _, err := os.Stat(dir + "/" + sub.Name() + "/partition"); err == nil {
|
||||
n.Children = append(n.Children, node(dir+"/"+sub.Name(), sub.Name(), "part"))
|
||||
}
|
||||
}
|
||||
out = append(out, n)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// An eMMC chip's boot and RPMB areas are hardware regions of the same chip, not drives;
|
||||
// the kernel lists them as disks of their own (mmcblk0boot0, mmcblk0boot1, mmcblk0rpmb).
|
||||
var mmcAreaRe = regexp.MustCompile(`^mmcblk\d+(boot\d+|rpmb|gp\d+)$`)
|
||||
|
||||
func mmcKind(name string) string {
|
||||
if strings.TrimSpace(readStrFile("/sys/block/"+name+"/device/type")) == "MMC" {
|
||||
return "emmc"
|
||||
}
|
||||
return "sd"
|
||||
}
|
||||
|
||||
// mmcHealth is the eMMC answer to SMART: the chip's own wear and end-of-life registers,
|
||||
// which the kernel publishes in /sys. SD cards have neither; smartctl speaks to neither.
|
||||
func mmcHealth(s Smart) Smart {
|
||||
dir := "/sys/block/" + s.Device + "/device/"
|
||||
s.Available, s.Supported = true, true
|
||||
s.SelfTestNote = "not a feature of eMMC or SD"
|
||||
s.Serial = strings.TrimSpace(readStrFile(dir + "serial"))
|
||||
s.Firmware = strings.TrimSpace(readStrFile(dir + "fwrev"))
|
||||
// fwrev is eight raw bytes in hex; vendors usually put ASCII in them ("0001")
|
||||
if b, err := hex.DecodeString(strings.TrimPrefix(s.Firmware, "0x")); err == nil {
|
||||
if t := strings.TrimRight(string(b), "\x00 "); t != "" && strings.IndexFunc(t, func(r rune) bool { return r < 0x20 || r > 0x7e }) < 0 {
|
||||
s.Firmware = t
|
||||
}
|
||||
}
|
||||
s.Model = strings.TrimSpace(readStrFile(dir + "name"))
|
||||
hx := func(v string) int { n, _ := strconv.ParseInt(strings.TrimPrefix(v, "0x"), 16, 32); return int(n) }
|
||||
life := strings.Fields(readStrFile(dir + "life_time"))
|
||||
eol := strings.TrimSpace(readStrFile(dir + "pre_eol_info"))
|
||||
if len(life) != 2 || eol == "" {
|
||||
s.Error = "This card reports no health data (SD cards don't; eMMC 5.0 and later does)."
|
||||
return s
|
||||
}
|
||||
m := &MMCHealth{LifeA: hx(life[0]), LifeB: hx(life[1]), PreEOL: hx(eol)}
|
||||
s.MMC = m
|
||||
ok := m.PreEOL <= 1 && m.LifeA < 10 && m.LifeB < 10
|
||||
s.Passed = &ok
|
||||
if m.PreEOL == 2 {
|
||||
s.Warnings = append(s.Warnings, "80% of reserved blocks consumed")
|
||||
} else if m.PreEOL >= 3 {
|
||||
s.Warnings = append(s.Warnings, "90% of reserved blocks consumed — replace soon")
|
||||
}
|
||||
if m.LifeA >= 10 || m.LifeB >= 10 {
|
||||
s.Warnings = append(s.Warnings, "estimated lifetime reached")
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
var nsRe = regexp.MustCompile(`^(nvme\d+)n\d+$`)
|
||||
|
||||
// foldNVMe merges nvme0n1, nvme0n2, … into one physical device "nvme0": one SSD, one
|
||||
// SMART record, however many namespaces the controller presents (Apple exposes three).
|
||||
func foldNVMe(devs []Device) []Device {
|
||||
byCtrl := map[string]*Device{}
|
||||
var out []Device
|
||||
for _, d := range devs {
|
||||
m := nsRe.FindStringSubmatch(d.Name)
|
||||
if m == nil {
|
||||
out = append(out, d)
|
||||
continue
|
||||
}
|
||||
ctrl := m[1]
|
||||
c, ok := byCtrl[ctrl]
|
||||
if !ok {
|
||||
nd := d
|
||||
nd.Name = ctrl
|
||||
nd.Parts = []Part{}
|
||||
nd.Namespaces = nil
|
||||
if mdl := strings.TrimSpace(readStrFile("/sys/class/nvme/" + ctrl + "/model")); mdl != "" {
|
||||
nd.Model = mdl
|
||||
}
|
||||
if sn := strings.TrimSpace(readStrFile("/sys/class/nvme/" + ctrl + "/serial")); sn != "" {
|
||||
nd.Serial = sn
|
||||
}
|
||||
if t := ctrlTemp(ctrl); t > 0 {
|
||||
nd.Temp = t
|
||||
}
|
||||
nd.Size = 0
|
||||
c = &nd
|
||||
byCtrl[ctrl] = c
|
||||
out = append(out, Device{}) // placeholder keeps order; filled below
|
||||
c.Namespaces = []string{}
|
||||
}
|
||||
c.Size += d.Size
|
||||
c.Namespaces = append(c.Namespaces, d.Name)
|
||||
c.Parts = append(c.Parts, d.Parts...)
|
||||
}
|
||||
// replace placeholders with the folded controllers, in first-seen order
|
||||
res := out[:0]
|
||||
seen := map[string]bool{}
|
||||
for _, d := range out {
|
||||
if d.Name != "" {
|
||||
res = append(res, d)
|
||||
continue
|
||||
}
|
||||
for _, c := range byCtrl {
|
||||
if !seen[c.Name] {
|
||||
seen[c.Name] = true
|
||||
res = append(res, *c)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
func readStrFile(p string) string {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func ctrlTemp(ctrl string) float64 {
|
||||
dirs, _ := filepath.Glob("/sys/class/nvme/" + ctrl + "/hwmon*")
|
||||
for _, d := range dirs {
|
||||
if b, err := os.ReadFile(filepath.Join(d, "temp1_input")); err == nil {
|
||||
if v, err := strconv.Atoi(strings.TrimSpace(string(b))); err == nil {
|
||||
return float64(v) / 1000
|
||||
}
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func blockTemp(name string) float64 {
|
||||
for _, pat := range []string{"/sys/block/" + name + "/device/hwmon*", "/sys/block/" + name + "/device/hwmon/hwmon*"} {
|
||||
dirs, _ := filepath.Glob(pat)
|
||||
for _, d := range dirs {
|
||||
if b, err := os.ReadFile(filepath.Join(d, "temp1_input")); err == nil {
|
||||
if v, err := strconv.Atoi(strings.TrimSpace(string(b))); err == nil {
|
||||
return float64(v) / 1000
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
var devRe = regexp.MustCompile(`^[a-z][a-z0-9]*$`)
|
||||
|
||||
func validDevice(name string) bool {
|
||||
if !devRe.MatchString(name) {
|
||||
return false
|
||||
}
|
||||
if _, err := os.Stat("/sys/block/" + name); err == nil {
|
||||
return true
|
||||
}
|
||||
_, err := os.Stat("/sys/class/nvme/" + name) // NVMe controller (nvme0), which is what smartctl wants anyway
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// ---- SMART ----
|
||||
|
||||
type Attr struct {
|
||||
ID int `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Value int `json:"value"`
|
||||
Worst int `json:"worst"`
|
||||
Thresh int `json:"thresh"`
|
||||
Raw string `json:"raw"`
|
||||
Failed string `json:"failed,omitempty"`
|
||||
Warn bool `json:"warn"`
|
||||
}
|
||||
|
||||
type Smart struct {
|
||||
Device string `json:"device"`
|
||||
Available bool `json:"available"` // smartctl installed
|
||||
Supported bool `json:"supported"` // device answered
|
||||
Standby bool `json:"standby"` // spun down; not read
|
||||
Passed *bool `json:"passed"`
|
||||
Temp int `json:"temp,omitempty"`
|
||||
PowerOnHours int `json:"power_on_hours,omitempty"`
|
||||
PowerCycles int `json:"power_cycles,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Firmware string `json:"firmware,omitempty"`
|
||||
Attrs []Attr `json:"attrs"`
|
||||
NVMe *NVMeLog `json:"nvme,omitempty"`
|
||||
MMC *MMCHealth `json:"mmc,omitempty"`
|
||||
SelfTest string `json:"self_test,omitempty"`
|
||||
SelfTestPct int `json:"self_test_remaining,omitempty"`
|
||||
SelfTestOK bool `json:"self_test_supported"`
|
||||
SelfTestNote string `json:"self_test_note,omitempty"`
|
||||
Warnings []string `json:"warnings"`
|
||||
Error string `json:"error,omitempty"`
|
||||
FetchedAt int64 `json:"fetched_at"`
|
||||
}
|
||||
|
||||
// MMCHealth is the eMMC EXT_CSD wear report. Life values are tenths: 1 = 0–10% of rated
|
||||
// life used, 10 = 90–100%, 11 = exceeded, 0 = not reported. A is the SLC area, B the MLC.
|
||||
type MMCHealth struct {
|
||||
LifeA int `json:"life_a"`
|
||||
LifeB int `json:"life_b"`
|
||||
PreEOL int `json:"pre_eol"` // 1 normal · 2 warning · 3 urgent
|
||||
}
|
||||
|
||||
type NVMeLog struct {
|
||||
CriticalWarning int `json:"critical_warning"`
|
||||
PercentageUsed int `json:"percentage_used"`
|
||||
AvailableSpare int `json:"available_spare"`
|
||||
SpareThreshold int `json:"spare_threshold"`
|
||||
MediaErrors int64 `json:"media_errors"`
|
||||
ErrorLogEntries int64 `json:"error_log_entries"`
|
||||
DataUnitsRead uint64 `json:"data_read_bytes"`
|
||||
DataUnitsWritten uint64 `json:"data_written_bytes"`
|
||||
UnsafeShutdowns int64 `json:"unsafe_shutdowns"`
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
sudo bool
|
||||
smartctl bool
|
||||
mu sync.Mutex
|
||||
cache map[string]Smart
|
||||
}
|
||||
|
||||
// lookPath also checks the sbin directories, which a user's PATH often lacks.
|
||||
func lookPath(bin string) (string, bool) {
|
||||
if p, err := exec.LookPath(bin); err == nil {
|
||||
return p, true
|
||||
}
|
||||
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin"} {
|
||||
if _, err := os.Stat(d + "/" + bin); err == nil {
|
||||
return d + "/" + bin, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
func New(sudo bool) *Manager {
|
||||
_, ok := lookPath("smartctl")
|
||||
return &Manager{sudo: sudo, smartctl: ok, cache: map[string]Smart{}}
|
||||
}
|
||||
|
||||
// HasSmartctl re-probes when it was missing at startup, so an install from the Disks
|
||||
// tab takes effect without a daemon restart.
|
||||
func (m *Manager) HasSmartctl() bool {
|
||||
if !m.smartctl {
|
||||
_, m.smartctl = lookPath("smartctl")
|
||||
}
|
||||
return m.smartctl
|
||||
}
|
||||
|
||||
func (m *Manager) run(ctx context.Context, args ...string) ([]byte, int, error) {
|
||||
if p, ok := lookPath(args[0]); ok {
|
||||
args[0] = p
|
||||
}
|
||||
if m.sudo {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
var out, errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||
err := cmd.Run()
|
||||
code := 0
|
||||
var ee *exec.ExitError
|
||||
if errors.As(err, &ee) {
|
||||
code = ee.ExitCode()
|
||||
err = nil // smartctl and zpool use exit codes as status; the caller decides
|
||||
if len(out.Bytes()) == 0 && errb.Len() > 0 {
|
||||
err = errors.New(strings.TrimSpace(strings.Split(errb.String(), "\n")[0])) // nothing usable came back; say why
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
msg := strings.TrimSpace(errb.String())
|
||||
if msg == "" {
|
||||
msg = err.Error()
|
||||
}
|
||||
return out.Bytes(), code, errors.New(msg)
|
||||
}
|
||||
return out.Bytes(), code, nil
|
||||
}
|
||||
|
||||
func num(v any) int64 {
|
||||
switch x := v.(type) {
|
||||
case float64:
|
||||
return int64(x)
|
||||
case json.Number:
|
||||
n, _ := x.Int64()
|
||||
return n
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// Smart reads SMART for one device. Cached for 10 minutes; spinning disks in standby are
|
||||
// left asleep and reported as such. force re-reads (and will wake the disk).
|
||||
func (m *Manager) Smart(ctx context.Context, dev string, force bool) Smart {
|
||||
s := Smart{Device: dev, Available: m.smartctl, Attrs: []Attr{}, Warnings: []string{}, FetchedAt: time.Now().UnixMilli()}
|
||||
if !validDevice(dev) {
|
||||
s.Error = "unknown device"
|
||||
return s
|
||||
}
|
||||
if strings.HasPrefix(dev, "mmcblk") {
|
||||
return mmcHealth(s)
|
||||
}
|
||||
if !m.HasSmartctl() {
|
||||
return s
|
||||
}
|
||||
m.mu.Lock()
|
||||
if c, ok := m.cache[dev]; ok && !force && time.Since(time.UnixMilli(c.FetchedAt)) < 10*time.Minute {
|
||||
m.mu.Unlock()
|
||||
return c
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
args := []string{"smartctl", "-j", "-a"}
|
||||
if !force {
|
||||
args = append(args, "-n", "standby")
|
||||
}
|
||||
args = append(args, "/dev/"+dev)
|
||||
out, code, err := m.run(ctx, args...)
|
||||
if err != nil && len(out) == 0 {
|
||||
s.Error = err.Error()
|
||||
return s
|
||||
}
|
||||
var raw map[string]any
|
||||
if json.Unmarshal(out, &raw) != nil {
|
||||
s.Error = "smartctl: unreadable output"
|
||||
return s
|
||||
}
|
||||
if code&2 != 0 { // -n standby: device is asleep (also "open failed", but JSON will say)
|
||||
if msgs, ok := raw["smartctl"].(map[string]any); ok {
|
||||
for _, mm := range asList(msgs["messages"]) {
|
||||
if strings.Contains(strings.ToLower(asString(mm["string"])), "standby") {
|
||||
s.Standby = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if s.Standby {
|
||||
m.mu.Lock()
|
||||
m.cache[dev] = s
|
||||
m.mu.Unlock()
|
||||
return s
|
||||
}
|
||||
}
|
||||
if st, ok := raw["smart_status"].(map[string]any); ok {
|
||||
p := st["passed"] == true
|
||||
s.Passed = &p
|
||||
s.Supported = true
|
||||
}
|
||||
s.Model = asString(raw["model_name"])
|
||||
s.Serial = asString(raw["serial_number"])
|
||||
s.Firmware = asString(raw["firmware_version"])
|
||||
if t, ok := raw["temperature"].(map[string]any); ok {
|
||||
s.Temp = int(num(t["current"]))
|
||||
}
|
||||
if t, ok := raw["power_on_time"].(map[string]any); ok {
|
||||
s.PowerOnHours = int(num(t["hours"]))
|
||||
}
|
||||
s.PowerCycles = int(num(raw["power_cycle_count"]))
|
||||
|
||||
if a, ok := raw["ata_smart_attributes"].(map[string]any); ok {
|
||||
for _, row := range asList(a["table"]) {
|
||||
at := Attr{ID: int(num(row["id"])), Name: asString(row["name"]), Value: int(num(row["value"])), Worst: int(num(row["worst"])), Thresh: int(num(row["thresh"])), Failed: asString(row["when_failed"])}
|
||||
if rw, ok := row["raw"].(map[string]any); ok {
|
||||
at.Raw = asString(rw["string"])
|
||||
rv := num(rw["value"])
|
||||
switch at.ID {
|
||||
case 5, 187, 188, 196, 197, 198:
|
||||
at.Warn = rv > 0
|
||||
case 199:
|
||||
at.Warn = rv > 0
|
||||
case 10, 11:
|
||||
at.Warn = rv > 100
|
||||
}
|
||||
if at.Warn {
|
||||
s.Warnings = append(s.Warnings, at.Name+" = "+at.Raw)
|
||||
}
|
||||
}
|
||||
if at.Failed != "" {
|
||||
at.Warn = true
|
||||
s.Warnings = append(s.Warnings, at.Name+" failed ("+at.Failed+")")
|
||||
}
|
||||
s.Attrs = append(s.Attrs, at)
|
||||
}
|
||||
}
|
||||
if n, ok := raw["nvme_smart_health_information_log"].(map[string]any); ok {
|
||||
l := &NVMeLog{CriticalWarning: int(num(n["critical_warning"])), PercentageUsed: int(num(n["percentage_used"])),
|
||||
AvailableSpare: int(num(n["available_spare"])), SpareThreshold: int(num(n["available_spare_threshold"])),
|
||||
MediaErrors: num(n["media_errors"]), ErrorLogEntries: num(n["num_err_log_entries"]), UnsafeShutdowns: num(n["unsafe_shutdowns"]),
|
||||
DataUnitsRead: uint64(num(n["data_units_read"])) * 512000, DataUnitsWritten: uint64(num(n["data_units_written"])) * 512000}
|
||||
s.NVMe = l
|
||||
s.Supported = true
|
||||
if l.CriticalWarning != 0 {
|
||||
s.Warnings = append(s.Warnings, "critical warning flags set")
|
||||
}
|
||||
if l.MediaErrors > 0 {
|
||||
s.Warnings = append(s.Warnings, "media errors: "+strconv.FormatInt(l.MediaErrors, 10))
|
||||
}
|
||||
if l.SpareThreshold > 0 && l.AvailableSpare < l.SpareThreshold {
|
||||
s.Warnings = append(s.Warnings, "available spare below threshold")
|
||||
}
|
||||
if l.PercentageUsed >= 90 {
|
||||
s.Warnings = append(s.Warnings, "endurance "+strconv.Itoa(l.PercentageUsed)+"% used")
|
||||
}
|
||||
}
|
||||
// Can this drive run self-tests from here? ATA says so outright; NVMe drives prove it
|
||||
// by exposing the self-test log, and some (WD SN580 among them) advertise the command
|
||||
// but reject the log page, which makes the whole feature unusable.
|
||||
if d, ok := raw["ata_smart_data"].(map[string]any); ok {
|
||||
if caps, ok := d["capabilities"].(map[string]any); ok {
|
||||
s.SelfTestOK = caps["self_tests_supported"] == true
|
||||
if !s.SelfTestOK {
|
||||
s.SelfTestNote = "drive reports no self-test support"
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, ok := raw["nvme_self_test_log"]; ok {
|
||||
s.SelfTestOK = true
|
||||
} else if s.NVMe != nil {
|
||||
s.SelfTestNote = "this drive's firmware doesn't expose the NVMe self-test log, so tests can't be run or read from here"
|
||||
}
|
||||
if msgs, ok := raw["smartctl"].(map[string]any); ok {
|
||||
for _, mm := range asList(msgs["messages"]) {
|
||||
if strings.Contains(asString(mm["string"]), "Self-test Log failed") {
|
||||
s.SelfTestOK = false
|
||||
s.SelfTestNote = "firmware rejects the self-test log page (" + asString(mm["string"]) + ")"
|
||||
}
|
||||
}
|
||||
}
|
||||
if d, ok := raw["ata_smart_data"].(map[string]any); ok {
|
||||
if st, ok := d["self_test"].(map[string]any); ok {
|
||||
if ss, ok := st["status"].(map[string]any); ok {
|
||||
s.SelfTest = asString(ss["string"])
|
||||
s.SelfTestPct = int(num(ss["remaining_percent"]))
|
||||
}
|
||||
}
|
||||
}
|
||||
if s.Passed != nil && !*s.Passed {
|
||||
s.Warnings = append([]string{"SMART overall health: FAILED"}, s.Warnings...)
|
||||
}
|
||||
if !s.Supported && s.Error == "" {
|
||||
s.Error = "device does not report SMART"
|
||||
if msgs, ok := raw["smartctl"].(map[string]any); ok {
|
||||
for _, mm := range asList(msgs["messages"]) {
|
||||
if asString(mm["severity"]) == "error" {
|
||||
s.Error = asString(mm["string"])
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.cache[dev] = s
|
||||
m.mu.Unlock()
|
||||
return s
|
||||
}
|
||||
|
||||
func asList(v any) []map[string]any {
|
||||
var out []map[string]any
|
||||
if l, ok := v.([]any); ok {
|
||||
for _, x := range l {
|
||||
if mm, ok := x.(map[string]any); ok {
|
||||
out = append(out, mm)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func asString(v any) string {
|
||||
if s, ok := v.(string); ok {
|
||||
return s
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// SelfTest starts a short or long self-test.
|
||||
func (m *Manager) SelfTest(ctx context.Context, dev, kind string) error {
|
||||
if !validDevice(dev) {
|
||||
return errors.New("unknown device")
|
||||
}
|
||||
if kind != "short" && kind != "long" {
|
||||
return errors.New("test must be short or long")
|
||||
}
|
||||
if !m.smartctl {
|
||||
return errors.New("smartctl is not installed")
|
||||
}
|
||||
if sm := m.Smart(ctx, dev, false); sm.Supported && !sm.SelfTestOK {
|
||||
return errors.New("self-tests aren't available on this drive: " + sm.SelfTestNote)
|
||||
}
|
||||
out, code, err := m.run(ctx, "smartctl", "-t", kind, "/dev/"+dev)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if code != 0 {
|
||||
return errors.New(strings.TrimSpace(lastLine(string(out))))
|
||||
}
|
||||
m.mu.Lock()
|
||||
delete(m.cache, dev)
|
||||
m.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
func lastLine(s string) string {
|
||||
lines := strings.Split(strings.TrimSpace(s), "\n")
|
||||
return lines[len(lines)-1]
|
||||
}
|
||||
|
||||
// ---- ZFS ----
|
||||
|
||||
type Pool struct {
|
||||
Name string `json:"name"`
|
||||
Size uint64 `json:"size"`
|
||||
Alloc uint64 `json:"alloc"`
|
||||
Free uint64 `json:"free"`
|
||||
Frag int `json:"frag"`
|
||||
Cap int `json:"cap"`
|
||||
Health string `json:"health"`
|
||||
Dedup string `json:"dedup"`
|
||||
Scan string `json:"scan,omitempty"`
|
||||
Errors string `json:"errors,omitempty"`
|
||||
Config string `json:"config,omitempty"` // vdev tree, verbatim
|
||||
}
|
||||
|
||||
type Dataset struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Used uint64 `json:"used"`
|
||||
Avail uint64 `json:"avail"`
|
||||
Refer uint64 `json:"refer"`
|
||||
UsedBySnap uint64 `json:"used_by_snapshots"`
|
||||
Mount string `json:"mount,omitempty"`
|
||||
Compress string `json:"compressratio"`
|
||||
Snapshots int `json:"snapshots"`
|
||||
}
|
||||
|
||||
type Snapshot struct {
|
||||
Name string `json:"name"`
|
||||
Used uint64 `json:"used"`
|
||||
Created int64 `json:"created"` // unix ms
|
||||
}
|
||||
|
||||
type ZFS struct {
|
||||
Available bool `json:"available"`
|
||||
Message string `json:"message,omitempty"`
|
||||
Pools []Pool `json:"pools"`
|
||||
Datasets []Dataset `json:"datasets"`
|
||||
}
|
||||
|
||||
var poolRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_.-]*$`)
|
||||
var datasetRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_.:/-]*$`)
|
||||
var snapRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.:-]*$`)
|
||||
|
||||
func u64(s string) uint64 { v, _ := strconv.ParseUint(s, 10, 64); return v }
|
||||
func atoi(s string) int { v, _ := strconv.Atoi(strings.TrimSuffix(s, "%")); return v }
|
||||
|
||||
// Pools is the pool list alone, for callers that poll: one `zpool list`, tried without
|
||||
// root first (it needs none on Linux) so polling writes nothing to the auth log.
|
||||
func (m *Manager) Pools(ctx context.Context) []Pool {
|
||||
bin, ok := lookPath("zpool")
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
args := []string{"list", "-Hp", "-o", "name,size,alloc,free,frag,cap,health,dedup"}
|
||||
out, err := exec.CommandContext(ctx, bin, args...).Output()
|
||||
if err != nil {
|
||||
var code int
|
||||
if out, code, err = m.run(ctx, append([]string{"zpool"}, args...)...); err != nil || code != 0 {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return parsePools(out)
|
||||
}
|
||||
|
||||
func parsePools(out []byte) []Pool {
|
||||
pools := []Pool{}
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) < 8 {
|
||||
continue
|
||||
}
|
||||
pools = append(pools, Pool{Name: f[0], Size: u64(f[1]), Alloc: u64(f[2]), Free: u64(f[3]), Frag: atoi(f[4]), Cap: atoi(f[5]), Health: f[6], Dedup: f[7]})
|
||||
}
|
||||
return pools
|
||||
}
|
||||
|
||||
func (m *Manager) ZFS(ctx context.Context) ZFS {
|
||||
z := ZFS{Pools: []Pool{}, Datasets: []Dataset{}}
|
||||
if _, ok := lookPath("zpool"); !ok {
|
||||
z.Message = "ZFS tools not installed"
|
||||
return z
|
||||
}
|
||||
out, code, err := m.run(ctx, "zpool", "list", "-Hp", "-o", "name,size,alloc,free,frag,cap,health,dedup")
|
||||
if err != nil || code != 0 {
|
||||
msg := "zpool list failed"
|
||||
if err != nil {
|
||||
msg = err.Error()
|
||||
}
|
||||
z.Message = msg
|
||||
return z
|
||||
}
|
||||
z.Available = true
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) < 8 {
|
||||
continue
|
||||
}
|
||||
p := Pool{Name: f[0], Size: u64(f[1]), Alloc: u64(f[2]), Free: u64(f[3]), Frag: atoi(f[4]), Cap: atoi(f[5]), Health: f[6], Dedup: f[7]}
|
||||
if st, _, err := m.run(ctx, "zpool", "status", p.Name); err == nil {
|
||||
p.Scan, p.Errors, p.Config = parseStatus(string(st))
|
||||
}
|
||||
z.Pools = append(z.Pools, p)
|
||||
}
|
||||
snapCount := map[string]int{}
|
||||
if so, _, err := m.run(ctx, "zfs", "list", "-t", "snapshot", "-H", "-o", "name"); err == nil {
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(so)), "\n") {
|
||||
if ds, _, ok := strings.Cut(line, "@"); ok {
|
||||
snapCount[ds]++
|
||||
}
|
||||
}
|
||||
}
|
||||
if do, _, err := m.run(ctx, "zfs", "list", "-Hp", "-t", "filesystem,volume", "-o", "name,type,used,avail,refer,usedbysnapshots,mountpoint,compressratio"); err == nil {
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(do)), "\n") {
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) < 8 {
|
||||
continue
|
||||
}
|
||||
d := Dataset{Name: f[0], Type: f[1], Used: u64(f[2]), Avail: u64(f[3]), Refer: u64(f[4]), UsedBySnap: u64(f[5]), Compress: f[7], Snapshots: snapCount[f[0]]}
|
||||
if f[6] != "-" && f[6] != "none" && f[6] != "legacy" {
|
||||
d.Mount = f[6]
|
||||
}
|
||||
z.Datasets = append(z.Datasets, d)
|
||||
}
|
||||
}
|
||||
return z
|
||||
}
|
||||
|
||||
// parseStatus pulls the scan line, the errors line and the vdev table out of zpool status.
|
||||
func parseStatus(text string) (scan, errs, config string) {
|
||||
lines := strings.Split(text, "\n")
|
||||
inConfig := false
|
||||
var cfg []string
|
||||
for i := 0; i < len(lines); i++ {
|
||||
l := lines[i]
|
||||
t := strings.TrimSpace(l)
|
||||
switch {
|
||||
case strings.HasPrefix(t, "scan:"):
|
||||
scan = strings.TrimSpace(strings.TrimPrefix(t, "scan:"))
|
||||
for i+1 < len(lines) && strings.HasPrefix(lines[i+1], "\t") && !strings.Contains(lines[i+1], ":") {
|
||||
i++
|
||||
scan += " " + strings.TrimSpace(lines[i])
|
||||
}
|
||||
case strings.HasPrefix(t, "errors:"):
|
||||
errs = strings.TrimSpace(strings.TrimPrefix(t, "errors:"))
|
||||
inConfig = false
|
||||
case strings.HasPrefix(t, "config:"):
|
||||
inConfig = true
|
||||
case inConfig && t != "":
|
||||
cfg = append(cfg, strings.TrimPrefix(l, "\t"))
|
||||
}
|
||||
}
|
||||
return scan, errs, strings.Join(cfg, "\n")
|
||||
}
|
||||
|
||||
func (m *Manager) Snapshots(ctx context.Context, dataset string) ([]Snapshot, error) {
|
||||
if !datasetRe.MatchString(dataset) {
|
||||
return nil, errors.New("bad dataset name")
|
||||
}
|
||||
out, code, err := m.run(ctx, "zfs", "list", "-t", "snapshot", "-Hp", "-d", "1", "-o", "name,used,creation", "-s", "creation", dataset)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if code != 0 {
|
||||
return nil, errors.New("zfs list failed")
|
||||
}
|
||||
snaps := []Snapshot{}
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
c, _ := strconv.ParseInt(f[2], 10, 64)
|
||||
snaps = append(snaps, Snapshot{Name: f[0], Used: u64(f[1]), Created: c * 1000})
|
||||
}
|
||||
return snaps, nil
|
||||
}
|
||||
|
||||
// Scrub starts, pauses or stops a scrub. action: start | pause | stop.
|
||||
func (m *Manager) Scrub(ctx context.Context, pool, action string) error {
|
||||
if !poolRe.MatchString(pool) {
|
||||
return errors.New("bad pool name")
|
||||
}
|
||||
args := []string{"zpool", "scrub"}
|
||||
switch action {
|
||||
case "start":
|
||||
case "pause":
|
||||
args = append(args, "-p")
|
||||
case "stop":
|
||||
args = append(args, "-s")
|
||||
default:
|
||||
return errors.New("unknown scrub action")
|
||||
}
|
||||
out, code, err := m.run(ctx, append(args, pool)...)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if code != 0 {
|
||||
return errors.New(strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreateSnapshot takes dataset@name.
|
||||
func (m *Manager) CreateSnapshot(ctx context.Context, dataset, name string) error {
|
||||
if !datasetRe.MatchString(dataset) || !snapRe.MatchString(name) {
|
||||
return errors.New("bad snapshot name")
|
||||
}
|
||||
out, code, err := m.run(ctx, "zfs", "snapshot", dataset+"@"+name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if code != 0 {
|
||||
return errors.New(strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The /sys fallback must find the same disks and partitions lsblk does.
|
||||
func TestSysBlockDevices(t *testing.T) {
|
||||
if _, err := os.Stat("/sys/block"); err != nil {
|
||||
t.Skip("no /sys/block")
|
||||
}
|
||||
for _, d := range sysBlockDevices() {
|
||||
if d.Name == "" || d.Type != "disk" {
|
||||
t.Fatalf("bad node %+v", d)
|
||||
}
|
||||
t.Logf("%s size=%d tran=%s rota=%v parts=%d", d.Name, d.Size, str(d.Tran), d.Rota, len(d.Children))
|
||||
for _, c := range d.Children {
|
||||
t.Logf(" %s size=%d fs=%s mount=%s", c.Name, c.Size, str(c.FSType), str(c.Mountpoint))
|
||||
}
|
||||
}
|
||||
}
|
||||
+98
-6
@@ -5,6 +5,7 @@ package store
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log"
|
||||
"os"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -15,9 +16,78 @@ type Terminal struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// Session is one signed-in browser. The map key is the cookie's token; ID is what the
|
||||
// UI sees and revokes by, so a session list never shows a usable token.
|
||||
type Session struct {
|
||||
ID string `json:"id"`
|
||||
User string `json:"user"`
|
||||
Expires time.Time `json:"expires"`
|
||||
Created time.Time `json:"created,omitempty"`
|
||||
LastSeen time.Time `json:"last_seen,omitempty"`
|
||||
IP string `json:"ip,omitempty"`
|
||||
Agent string `json:"agent,omitempty"`
|
||||
}
|
||||
|
||||
// UnmarshalJSON also reads the pre-users form, a bare expiry time: such a session has no
|
||||
// user yet; the server gives it to the migrated first account.
|
||||
func (s *Session) UnmarshalJSON(b []byte) error {
|
||||
if len(b) > 0 && b[0] == '"' {
|
||||
return json.Unmarshal(b, &s.Expires)
|
||||
}
|
||||
type plain Session
|
||||
return json.Unmarshal(b, (*plain)(s))
|
||||
}
|
||||
|
||||
// UserState is what follows an account rather than the machine: how the console looks
|
||||
// to that person and how their dashboard is laid out.
|
||||
type UserState struct {
|
||||
UI json.RawMessage `json:"ui,omitempty"` // theme, accents; opaque to the server
|
||||
}
|
||||
|
||||
// RememberedMount is one bind mount of a container that was running when Ham Lab last
|
||||
// looked. A compose stack records the path its file had *inside* whatever deployed it, so
|
||||
// when that container is gone the path is the only way back to the file on disk.
|
||||
type RememberedMount struct {
|
||||
Container string `json:"container"`
|
||||
Dest string `json:"dest"` // the path inside the container
|
||||
Src string `json:"src"` // the path on the machine
|
||||
Seen time.Time `json:"seen"`
|
||||
}
|
||||
|
||||
type State struct {
|
||||
Terminals []Terminal `json:"terminals"`
|
||||
Sessions map[string]time.Time `json:"sessions"` // token -> expiry
|
||||
Sessions map[string]Session `json:"sessions"` // token -> session
|
||||
Users map[string]*UserState `json:"users,omitempty"`
|
||||
// UI is the pre-users machine-wide look; on first boot with users it moves to the first
|
||||
// account. Kept so an old state file still reads.
|
||||
UI json.RawMessage `json:"ui,omitempty"`
|
||||
Alerts json.RawMessage `json:"alerts,omitempty"` // rules, channels and recent events (internal/api/alerts.go)
|
||||
ZFS json.RawMessage `json:"zfs,omitempty"` // snapshot schedules (internal/api/zfs.go)
|
||||
DockerJobs json.RawMessage `json:"docker_jobs,omitempty"` // image-update schedules (internal/api/dockersched.go)
|
||||
Network json.RawMessage `json:"network,omitempty"` // speed test results (internal/api/network.go)
|
||||
// Mounts remembers where a container's bind mounts pointed, so a stack deployed by an
|
||||
// orchestrator that has since been removed can still be found on disk. See
|
||||
// internal/api/stacks.go.
|
||||
Mounts []RememberedMount `json:"mounts,omitempty"`
|
||||
// KVMCookies is each KVM's own login, as the cookies its page set in a browser: kept so
|
||||
// the next browser — or this one after it has forgotten them — is signed into the device
|
||||
// already. See internal/api/kvm.go.
|
||||
KVMCookies map[string]map[string]string `json:"kvm_cookies,omitempty"`
|
||||
SamplerMs int `json:"sampler_ms,omitempty"` // how often this machine samples; the hub keeps every machine on the console's refresh rate
|
||||
HistorySec int `json:"history_sec,omitempty"` // how far back the graphs reach; the hub keeps every machine on the same one
|
||||
}
|
||||
|
||||
// UserState returns the prefs of an account, making an empty one if needed.
|
||||
func (st *State) UserState(name string) *UserState {
|
||||
if st.Users == nil {
|
||||
st.Users = map[string]*UserState{}
|
||||
}
|
||||
u := st.Users[name]
|
||||
if u == nil {
|
||||
u = &UserState{}
|
||||
st.Users[name] = u
|
||||
}
|
||||
return u
|
||||
}
|
||||
|
||||
type Store struct {
|
||||
@@ -33,12 +103,16 @@ func Open(path string) (*Store, error) {
|
||||
return nil, err
|
||||
}
|
||||
if err == nil {
|
||||
// State is conveniences (sessions, terminal names, layout). A file mangled by a power
|
||||
// cut must not keep the console down: set it aside and start clean.
|
||||
if err := json.Unmarshal(b, &s.state); err != nil {
|
||||
return nil, err
|
||||
s.state = State{}
|
||||
_ = os.Rename(path, path+".corrupt")
|
||||
log.Printf("state: %s unreadable (%v); kept as %s.corrupt, starting fresh", path, err, path)
|
||||
}
|
||||
}
|
||||
if s.state.Sessions == nil {
|
||||
s.state.Sessions = map[string]time.Time{}
|
||||
s.state.Sessions = map[string]Session{}
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
@@ -63,9 +137,27 @@ func (s *Store) save() error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := s.path + ".tmp"
|
||||
if err := os.WriteFile(tmp, b, 0o600); err != nil {
|
||||
return WriteFileAtomic(s.path, b, 0o600)
|
||||
}
|
||||
|
||||
// WriteFileAtomic writes beside path, syncs, then renames: after a crash the file is the
|
||||
// old version or the new one, never empty or half-written.
|
||||
func WriteFileAtomic(path string, data []byte, perm os.FileMode) error {
|
||||
tmp := path + ".tmp"
|
||||
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, perm)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, s.path)
|
||||
_, err = f.Write(data)
|
||||
if err == nil {
|
||||
err = f.Sync()
|
||||
}
|
||||
if cerr := f.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,260 @@
|
||||
// Package tasks runs long commands (apt upgrades, and later anything else the UI kicks
|
||||
// off) in the background, keeps their output, and lets any number of clients follow it
|
||||
// live. It is also the task history: what ran, when, and how it ended.
|
||||
package tasks
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"os/exec"
|
||||
"sort"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
maxOutput = 1 << 20 // bytes kept per task
|
||||
maxHistory = 200
|
||||
)
|
||||
|
||||
type Task struct {
|
||||
ID string
|
||||
Kind string // "updates", "cron", …
|
||||
Title string
|
||||
Command string
|
||||
Started time.Time
|
||||
Finished time.Time
|
||||
Running bool
|
||||
ExitCode int
|
||||
Err string
|
||||
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
subs map[chan []byte]struct{}
|
||||
cancel context.CancelFunc
|
||||
onCancel func()
|
||||
}
|
||||
|
||||
type Snapshot struct {
|
||||
ID string `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Title string `json:"title"`
|
||||
Command string `json:"command"`
|
||||
Started time.Time `json:"started"`
|
||||
Finished *time.Time `json:"finished,omitempty"`
|
||||
Running bool `json:"running"`
|
||||
ExitCode int `json:"exit_code"`
|
||||
Err string `json:"error,omitempty"`
|
||||
Output string `json:"output,omitempty"`
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
mu sync.Mutex
|
||||
tasks map[string]*Task
|
||||
}
|
||||
|
||||
func New() *Manager { return &Manager{tasks: map[string]*Task{}} }
|
||||
|
||||
func newID() string {
|
||||
b := make([]byte, 6)
|
||||
_, _ = rand.Read(b)
|
||||
return hex.EncodeToString(b)
|
||||
}
|
||||
|
||||
// Start launches cmd with its output captured into a new task. cmd must not have
|
||||
// Stdout/Stderr set. The task ends when the process exits or Cancel is called.
|
||||
func (m *Manager) Start(kind, title string, cmd *exec.Cmd) (*Task, error) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
t := &Task{ID: newID(), Kind: kind, Title: title, Command: cmd.String(), Started: time.Now(), Running: true,
|
||||
subs: map[chan []byte]struct{}{}, cancel: cancel}
|
||||
w := &writer{t: t}
|
||||
cmd.Stdout, cmd.Stderr = w, w
|
||||
// Its own process group: a task is usually `sh -c "a && b"`, and cancelling must reach a and
|
||||
// b, not just the shell — killing only the shell left compose running and, because compose
|
||||
// still held the output pipe, left the task "running" too. WaitDelay is the backstop for
|
||||
// anything that escapes the group and keeps the pipe open.
|
||||
if cmd.SysProcAttr == nil {
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{}
|
||||
}
|
||||
cmd.SysProcAttr.Setpgid = true
|
||||
cmd.WaitDelay = 5 * time.Second
|
||||
if err := cmd.Start(); err != nil {
|
||||
cancel()
|
||||
return nil, err
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.tasks[t.ID] = t
|
||||
m.prune()
|
||||
m.mu.Unlock()
|
||||
|
||||
go func() {
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- cmd.Wait() }()
|
||||
var err error
|
||||
select {
|
||||
case err = <-done:
|
||||
case <-ctx.Done():
|
||||
t.mu.Lock()
|
||||
hook := t.onCancel
|
||||
t.mu.Unlock()
|
||||
if hook != nil {
|
||||
go hook()
|
||||
}
|
||||
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM) // asked first: compose and apt both stop cleanly
|
||||
select {
|
||||
case err = <-done:
|
||||
case <-time.After(4 * time.Second):
|
||||
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||
_ = cmd.Process.Kill()
|
||||
err = <-done
|
||||
}
|
||||
t.mu.Lock()
|
||||
t.Err = "cancelled"
|
||||
t.mu.Unlock()
|
||||
}
|
||||
t.mu.Lock()
|
||||
t.Running, t.Finished = false, time.Now()
|
||||
if err != nil {
|
||||
var ee *exec.ExitError
|
||||
if errors.As(err, &ee) {
|
||||
t.ExitCode = ee.ExitCode()
|
||||
} else {
|
||||
t.ExitCode = -1
|
||||
if t.Err == "" {
|
||||
t.Err = err.Error()
|
||||
}
|
||||
}
|
||||
}
|
||||
for ch := range t.subs {
|
||||
close(ch)
|
||||
}
|
||||
t.subs = map[chan []byte]struct{}{}
|
||||
t.mu.Unlock()
|
||||
cancel()
|
||||
}()
|
||||
return t, nil
|
||||
}
|
||||
|
||||
// prune keeps the newest maxHistory finished tasks. Caller holds m.mu.
|
||||
func (m *Manager) prune() {
|
||||
if len(m.tasks) <= maxHistory {
|
||||
return
|
||||
}
|
||||
var finished []*Task
|
||||
for _, t := range m.tasks {
|
||||
if !t.Running {
|
||||
finished = append(finished, t)
|
||||
}
|
||||
}
|
||||
sort.Slice(finished, func(i, j int) bool { return finished[i].Started.Before(finished[j].Started) })
|
||||
for i := 0; i < len(finished) && len(m.tasks) > maxHistory; i++ {
|
||||
delete(m.tasks, finished[i].ID)
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Manager) Get(id string) (*Task, bool) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
t, ok := m.tasks[id]
|
||||
return t, ok
|
||||
}
|
||||
|
||||
// List returns every task newest first, without output.
|
||||
func (m *Manager) List() []Snapshot {
|
||||
m.mu.Lock()
|
||||
all := make([]*Task, 0, len(m.tasks))
|
||||
for _, t := range m.tasks {
|
||||
all = append(all, t)
|
||||
}
|
||||
m.mu.Unlock()
|
||||
sort.Slice(all, func(i, j int) bool { return all[i].Started.After(all[j].Started) })
|
||||
out := make([]Snapshot, 0, len(all))
|
||||
for _, t := range all {
|
||||
out = append(out, t.Snapshot(false))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// RunningOf reports whether a task of the given kind is in flight.
|
||||
func (m *Manager) RunningOf(kind string) (*Task, bool) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
for _, t := range m.tasks {
|
||||
if t.Kind == kind && t.Running {
|
||||
return t, true
|
||||
}
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
func (t *Task) Snapshot(withOutput bool) Snapshot {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
s := Snapshot{ID: t.ID, Kind: t.Kind, Title: t.Title, Command: t.Command, Started: t.Started, Running: t.Running, ExitCode: t.ExitCode, Err: t.Err}
|
||||
if !t.Running {
|
||||
f := t.Finished
|
||||
s.Finished = &f
|
||||
}
|
||||
if withOutput {
|
||||
s.Output = t.buf.String()
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Subscribe returns the output so far plus a channel of further chunks, closed when the
|
||||
// task ends. Call unsub when done following.
|
||||
func (t *Task) Subscribe() (backlog []byte, ch chan []byte, unsub func()) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
backlog = append([]byte(nil), t.buf.Bytes()...)
|
||||
if !t.Running {
|
||||
c := make(chan []byte)
|
||||
close(c)
|
||||
return backlog, c, func() {}
|
||||
}
|
||||
ch = make(chan []byte, 256)
|
||||
t.subs[ch] = struct{}{}
|
||||
return backlog, ch, func() {
|
||||
t.mu.Lock()
|
||||
if _, ok := t.subs[ch]; ok {
|
||||
delete(t.subs, ch)
|
||||
close(ch)
|
||||
}
|
||||
t.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
func (t *Task) Cancel() { t.cancel() }
|
||||
|
||||
// OnCancel sets something to do, besides stopping the task's own processes, when the task is
|
||||
// cancelled: whatever it started that isn't its child (a command inside a container).
|
||||
func (t *Task) OnCancel(fn func()) {
|
||||
t.mu.Lock()
|
||||
t.onCancel = fn
|
||||
t.mu.Unlock()
|
||||
}
|
||||
|
||||
type writer struct{ t *Task }
|
||||
|
||||
func (w *writer) Write(p []byte) (int, error) {
|
||||
t := w.t
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.buf.Write(p)
|
||||
if t.buf.Len() > maxOutput {
|
||||
b := t.buf.Bytes()
|
||||
t.buf = *bytes.NewBuffer(append([]byte("…\n"), b[len(b)-maxOutput*3/4:]...))
|
||||
}
|
||||
chunk := append([]byte(nil), p...)
|
||||
for ch := range t.subs {
|
||||
select {
|
||||
case ch <- chunk:
|
||||
default: // slow follower: drop the chunk rather than stall the process
|
||||
}
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"os/exec"
|
||||
"sort"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
@@ -68,6 +69,7 @@ type Manager struct {
|
||||
shell string
|
||||
home string
|
||||
OnChange func([]Info) // called (without locks held) when the list changes
|
||||
closing atomic.Bool
|
||||
}
|
||||
|
||||
func NewManager(shell, home string) *Manager {
|
||||
@@ -82,10 +84,21 @@ func newID() string {
|
||||
|
||||
// Create starts a new login shell. id may be empty.
|
||||
func (m *Manager) Create(id, name string) (*Session, error) {
|
||||
return m.CreateCmd(id, name, nil)
|
||||
}
|
||||
|
||||
// CreateCmd is Create with something other than a login shell: argv nil is the shell, and
|
||||
// anything else runs that instead — `docker exec -it …` for a container's console. Every
|
||||
// session is the same thing afterwards, so attach, resize, the scrollback ring and the tab
|
||||
// strip work without knowing which it is.
|
||||
func (m *Manager) CreateCmd(id, name string, argv []string) (*Session, error) {
|
||||
if id == "" {
|
||||
id = newID()
|
||||
}
|
||||
cmd := exec.Command(m.shell, "-l")
|
||||
if len(argv) > 0 {
|
||||
cmd = exec.Command(argv[0], argv[1:]...)
|
||||
}
|
||||
cmd.Dir = m.home
|
||||
cmd.Env = append(os.Environ(), "TERM=xterm-256color", "COLORTERM=truecolor", "HAMLAB=1")
|
||||
ptmx, err := pty.StartWithSize(cmd, &pty.Winsize{Cols: defaultCol, Rows: defaultRow})
|
||||
@@ -130,6 +143,15 @@ func (m *Manager) pump(s *Session) {
|
||||
s.mu.Unlock()
|
||||
_ = s.ptmx.Close()
|
||||
m.changed()
|
||||
// An exited shell stays listed for a while so the exit is visible, then reaps itself.
|
||||
time.AfterFunc(10*time.Minute, func() {
|
||||
m.mu.Lock()
|
||||
cur, ok := m.sessions[s.ID]
|
||||
m.mu.Unlock()
|
||||
if ok && cur == s && !s.running {
|
||||
_ = m.Close(s.ID)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// broadcast requires s.mu. Slow clients are dropped rather than blocking the PTY.
|
||||
@@ -274,13 +296,15 @@ func (m *Manager) Close(id string) error {
|
||||
}
|
||||
|
||||
func (m *Manager) changed() {
|
||||
if m.OnChange != nil {
|
||||
if m.OnChange != nil && !m.closing.Load() {
|
||||
m.OnChange(m.List())
|
||||
}
|
||||
}
|
||||
|
||||
// Shutdown hangs up every shell. Called on daemon exit.
|
||||
// Shutdown hangs up every shell. Called on daemon exit. The saved list is left as it
|
||||
// was, because that list is what the next start re-opens.
|
||||
func (m *Manager) Shutdown() {
|
||||
m.closing.Store(true)
|
||||
for _, s := range m.List() {
|
||||
_ = m.Close(s.ID)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
// Package tlsauto makes a self-signed certificate for a node so hub↔node traffic is
|
||||
// encrypted without any CA. The hub pins the certificate's fingerprint.
|
||||
package tlsauto
|
||||
|
||||
import (
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/hex"
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"math/big"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Fingerprint formats a certificate's SHA-256 the way the UI shows it.
|
||||
func Fingerprint(der []byte) string {
|
||||
sum := sha256.Sum256(der)
|
||||
h := hex.EncodeToString(sum[:])
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(h); i += 2 {
|
||||
if i > 0 {
|
||||
b.WriteByte(':')
|
||||
}
|
||||
b.WriteString(strings.ToUpper(h[i : i+2]))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// Ensure returns the cert/key paths and fingerprint, generating them on first use.
|
||||
func Ensure(dir string) (certPath, keyPath, fingerprint string, err error) {
|
||||
certPath, keyPath = filepath.Join(dir, "cert.pem"), filepath.Join(dir, "key.pem")
|
||||
if _, err := os.Stat(certPath); err == nil {
|
||||
if _, err := os.Stat(keyPath); err == nil {
|
||||
c, err := tls.LoadX509KeyPair(certPath, keyPath)
|
||||
if err == nil && len(c.Certificate) > 0 {
|
||||
return certPath, keyPath, Fingerprint(c.Certificate[0]), nil
|
||||
}
|
||||
}
|
||||
}
|
||||
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 127))
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
tmpl := &x509.Certificate{
|
||||
SerialNumber: serial,
|
||||
Subject: pkix.Name{CommonName: "hamlab@" + host, Organization: []string{"Ham Lab"}},
|
||||
NotBefore: time.Now().Add(-time.Hour),
|
||||
NotAfter: time.Now().AddDate(10, 0, 0),
|
||||
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
|
||||
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
|
||||
BasicConstraintsValid: true,
|
||||
DNSNames: []string{host, "localhost"},
|
||||
IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
|
||||
}
|
||||
if addrs, err := net.InterfaceAddrs(); err == nil {
|
||||
for _, a := range addrs {
|
||||
if ipn, ok := a.(*net.IPNet); ok && !ipn.IP.IsLoopback() && ipn.IP.To4() != nil {
|
||||
tmpl.IPAddresses = append(tmpl.IPAddresses, ipn.IP)
|
||||
}
|
||||
}
|
||||
}
|
||||
der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &priv.PublicKey, priv)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
keyDER, err := x509.MarshalECPrivateKey(priv)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
if err := os.WriteFile(certPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0o644); err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
if err := os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER}), 0o600); err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
return certPath, keyPath, Fingerprint(der), nil
|
||||
}
|
||||
|
||||
// PinnedConfig returns a client TLS config that accepts exactly one certificate.
|
||||
func PinnedConfig(fingerprint string) *tls.Config {
|
||||
return &tls.Config{
|
||||
InsecureSkipVerify: true, // we verify by fingerprint below, not by CA
|
||||
VerifyPeerCertificate: func(raw [][]byte, _ [][]*x509.Certificate) error {
|
||||
if len(raw) == 0 {
|
||||
return errors.New("no certificate presented")
|
||||
}
|
||||
if Fingerprint(raw[0]) != fingerprint {
|
||||
return errors.New("certificate fingerprint changed: " + Fingerprint(raw[0]))
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// Peek connects and returns the fingerprint the server presents (trust on first use).
|
||||
func Peek(addr string) (string, error) {
|
||||
d := &net.Dialer{Timeout: 5 * time.Second}
|
||||
conn, err := tls.DialWithDialer(d, "tcp", addr, &tls.Config{InsecureSkipVerify: true})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer conn.Close()
|
||||
certs := conn.ConnectionState().PeerCertificates
|
||||
if len(certs) == 0 {
|
||||
return "", errors.New("no certificate presented")
|
||||
}
|
||||
return Fingerprint(certs[0].Raw), nil
|
||||
}
|
||||
@@ -0,0 +1,544 @@
|
||||
// Package updates is the package-manager view: what can be upgraded, when the index
|
||||
// was last refreshed, whether a reboot is pending, and the refresh/upgrade actions,
|
||||
// which run as background tasks so their output can be followed live.
|
||||
//
|
||||
// Supported: apt (Debian, Ubuntu), dnf (Fedora, RHEL family), zypper (openSUSE, SLES),
|
||||
// pacman (Arch) and apk (Alpine). All of them list, refresh, upgrade and install; pacman
|
||||
// upgrades everything or nothing, because Arch doesn't support partial upgrades.
|
||||
package updates
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
"hamlab/internal/tasks"
|
||||
)
|
||||
|
||||
type Package struct {
|
||||
Name string `json:"name"`
|
||||
Current string `json:"current"`
|
||||
Candidate string `json:"candidate"`
|
||||
Suite string `json:"suite,omitempty"`
|
||||
Security bool `json:"security"`
|
||||
Description string `json:"description,omitempty"`
|
||||
}
|
||||
|
||||
type Status struct {
|
||||
Manager string `json:"manager"` // apt | dnf | zypper | pacman | apk | ""
|
||||
CanUpgrade bool `json:"can_upgrade"`
|
||||
Selective bool `json:"selective"` // individual packages can be upgraded
|
||||
Packages []Package `json:"packages"`
|
||||
SecurityCount int `json:"security_count"`
|
||||
LastRefresh int64 `json:"last_refresh,omitempty"` // unix ms
|
||||
LastUpgrade int64 `json:"last_upgrade,omitempty"`
|
||||
RebootNeeded bool `json:"reboot_needed"`
|
||||
RebootPkgs []string `json:"reboot_pkgs"`
|
||||
AutoUpgrades bool `json:"auto_upgrades"`
|
||||
RunningTask string `json:"running_task,omitempty"`
|
||||
Sudo bool `json:"sudo"`
|
||||
}
|
||||
|
||||
type Manager struct {
|
||||
sudo bool
|
||||
tasks *tasks.Manager
|
||||
pm string
|
||||
}
|
||||
|
||||
func New(sudo bool, tm *tasks.Manager) *Manager {
|
||||
m := &Manager{sudo: sudo, tasks: tm}
|
||||
for _, pm := range []string{"apt-get", "dnf", "zypper", "pacman", "apk"} {
|
||||
if _, err := exec.LookPath(pm); err == nil {
|
||||
m.pm = strings.TrimSuffix(pm, "-get")
|
||||
break
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func (m *Manager) sudoArgs(args ...string) []string {
|
||||
if m.sudo {
|
||||
return priv.Wrap(args...)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
func out(ctx context.Context, args ...string) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
cmd.Env = append(os.Environ(), "LC_ALL=C", "DEBIAN_FRONTEND=noninteractive")
|
||||
b, err := cmd.Output()
|
||||
if err != nil {
|
||||
var ee *exec.ExitError
|
||||
if errors.As(err, &ee) && len(ee.Stderr) > 0 {
|
||||
return string(b), errors.New(strings.TrimSpace(string(ee.Stderr)))
|
||||
}
|
||||
return string(b), err
|
||||
}
|
||||
return string(b), nil
|
||||
}
|
||||
|
||||
// newestMtime: the newest mtime among the paths matching any of the globs.
|
||||
func newestMtime(globs ...string) int64 {
|
||||
var best int64
|
||||
for _, g := range globs {
|
||||
matches, _ := filepath.Glob(g)
|
||||
for _, m := range matches {
|
||||
if t := mtimeMs(m); t > best {
|
||||
best = t
|
||||
}
|
||||
}
|
||||
}
|
||||
return best
|
||||
}
|
||||
|
||||
// dnfLastUpgrade: newest transaction in `dnf history` whose action includes an upgrade,
|
||||
// else the newest transaction of any kind.
|
||||
func dnfLastUpgrade(ctx context.Context) int64 {
|
||||
text, err := out(ctx, "dnf", "history", "list")
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
re := regexp.MustCompile(`(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+(\S+)`)
|
||||
var newest, upgrade int64
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
m := re.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
t, err := time.ParseInLocation("2006-01-02 15:04:05", m[1], time.Local)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
ms := t.UnixMilli()
|
||||
if ms > newest {
|
||||
newest = ms
|
||||
}
|
||||
if strings.Contains(m[2], "U") && ms > upgrade {
|
||||
upgrade = ms
|
||||
}
|
||||
}
|
||||
if upgrade > 0 {
|
||||
return upgrade
|
||||
}
|
||||
return newest
|
||||
}
|
||||
|
||||
func mtimeMs(p string) int64 {
|
||||
st, err := os.Stat(p)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return st.ModTime().UnixMilli()
|
||||
}
|
||||
|
||||
// ---- status ----
|
||||
|
||||
func (m *Manager) Status(ctx context.Context) Status {
|
||||
s := Status{Manager: m.pm, Packages: []Package{}, RebootPkgs: []string{}, Sudo: m.sudo}
|
||||
if t, ok := m.tasks.RunningOf("updates"); ok {
|
||||
s.RunningTask = t.ID
|
||||
}
|
||||
s.CanUpgrade = m.sudo && m.pm != ""
|
||||
s.Selective = m.pm != "pacman"
|
||||
switch m.pm {
|
||||
case "apt":
|
||||
s.Packages = aptUpgradable(ctx)
|
||||
s.LastRefresh = mtimeMs("/var/lib/apt/periodic/update-success-stamp")
|
||||
if s.LastRefresh == 0 {
|
||||
s.LastRefresh = mtimeMs("/var/lib/apt/lists/partial")
|
||||
}
|
||||
if s.LastRefresh == 0 {
|
||||
s.LastRefresh = mtimeMs("/var/lib/apt/lists")
|
||||
}
|
||||
s.LastUpgrade = aptLastUpgrade()
|
||||
s.AutoUpgrades = aptAutoUpgrades(ctx)
|
||||
case "dnf":
|
||||
s.Packages = dnfUpgradable(ctx)
|
||||
s.LastRefresh = newestMtime("/var/cache/libdnf5/*/repodata", "/var/cache/dnf/*/repodata", "/var/cache/libdnf5", "/var/cache/dnf")
|
||||
s.LastUpgrade = dnfLastUpgrade(ctx)
|
||||
case "zypper":
|
||||
s.Packages = zypperUpgradable(ctx)
|
||||
s.LastRefresh = newestMtime("/var/cache/zypp/raw/*", "/var/cache/zypp/raw")
|
||||
s.LastUpgrade = mtimeMs("/var/log/zypp/history")
|
||||
case "pacman":
|
||||
s.Packages = pacmanUpgradable(ctx)
|
||||
s.LastRefresh = newestMtime("/var/lib/pacman/sync/*.db", "/var/lib/pacman/sync")
|
||||
s.LastUpgrade = pacmanLastUpgrade()
|
||||
case "apk":
|
||||
s.Packages = apkUpgradable(ctx)
|
||||
s.LastRefresh = newestMtime("/var/cache/apk/APKINDEX.*", "/var/lib/apk/APKINDEX.*")
|
||||
s.LastUpgrade = mtimeMs("/lib/apk/db/installed")
|
||||
}
|
||||
for _, p := range s.Packages {
|
||||
if p.Security {
|
||||
s.SecurityCount++
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat("/run/reboot-required"); err == nil {
|
||||
s.RebootNeeded = true
|
||||
if b, err := os.ReadFile("/run/reboot-required.pkgs"); err == nil {
|
||||
for _, l := range strings.Split(strings.TrimSpace(string(b)), "\n") {
|
||||
if l = strings.TrimSpace(l); l != "" {
|
||||
s.RebootPkgs = append(s.RebootPkgs, l)
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if m.pm != "apt" {
|
||||
s.RebootNeeded = kernelChanged()
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// kernelChanged: the running kernel isn't the newest installed one (dnf/pacman hosts).
|
||||
func kernelChanged() bool {
|
||||
b, err := os.ReadFile("/proc/sys/kernel/osrelease")
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
running := strings.TrimSpace(string(b))
|
||||
ents, err := os.ReadDir("/lib/modules")
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
newest := ""
|
||||
for _, e := range ents {
|
||||
if e.Name() > newest {
|
||||
newest = e.Name()
|
||||
}
|
||||
}
|
||||
return newest != "" && newest != running
|
||||
}
|
||||
|
||||
// ---- apt ----
|
||||
|
||||
var aptListRe = regexp.MustCompile(`^(\S+)/(\S+)\s+(\S+)\s+\S+\s+\[upgradable from: (\S+)\]`)
|
||||
|
||||
func aptUpgradable(ctx context.Context) []Package {
|
||||
text, _ := out(ctx, "apt", "list", "--upgradable")
|
||||
var pkgs []Package
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
m := aptListRe.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
pkgs = append(pkgs, Package{Name: m[1], Suite: m[2], Candidate: m[3], Current: m[4],
|
||||
Security: strings.Contains(m[2], "security")})
|
||||
}
|
||||
if len(pkgs) == 0 {
|
||||
return []Package{}
|
||||
}
|
||||
// One apt-cache call for all the descriptions.
|
||||
args := append([]string{"apt-cache", "show", "--no-all-versions"}, names(pkgs)...)
|
||||
if text, err := out(ctx, args...); err == nil {
|
||||
desc := map[string]string{}
|
||||
cur := ""
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
if strings.HasPrefix(line, "Package: ") {
|
||||
cur = strings.TrimPrefix(line, "Package: ")
|
||||
} else if strings.HasPrefix(line, "Description-en: ") || strings.HasPrefix(line, "Description: ") {
|
||||
if _, ok := desc[cur]; !ok {
|
||||
_, v, _ := strings.Cut(line, ": ")
|
||||
desc[cur] = v
|
||||
}
|
||||
}
|
||||
}
|
||||
for i := range pkgs {
|
||||
pkgs[i].Description = desc[pkgs[i].Name]
|
||||
}
|
||||
}
|
||||
sort.Slice(pkgs, func(i, j int) bool { return pkgs[i].Name < pkgs[j].Name })
|
||||
return pkgs
|
||||
}
|
||||
|
||||
func names(pkgs []Package) []string {
|
||||
n := make([]string, len(pkgs))
|
||||
for i, p := range pkgs {
|
||||
n[i] = p.Name
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// aptLastUpgrade: the Start-Date of the last history.log entry that upgraded something.
|
||||
func aptLastUpgrade() int64 {
|
||||
f, err := os.Open("/var/log/apt/history.log")
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
defer f.Close()
|
||||
var last, start string
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 8<<20)
|
||||
for sc.Scan() {
|
||||
l := sc.Text()
|
||||
switch {
|
||||
case strings.HasPrefix(l, "Start-Date: "):
|
||||
start = strings.TrimPrefix(l, "Start-Date: ")
|
||||
case strings.HasPrefix(l, "Upgrade: "):
|
||||
last = start
|
||||
}
|
||||
}
|
||||
if last == "" {
|
||||
return 0
|
||||
}
|
||||
t, err := time.ParseInLocation("2006-01-02 15:04:05", last, time.Local)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return t.UnixMilli()
|
||||
}
|
||||
|
||||
func aptAutoUpgrades(ctx context.Context) bool {
|
||||
text, _ := out(ctx, "apt-config", "dump", "APT::Periodic::Unattended-Upgrade")
|
||||
return strings.Contains(text, `"1"`)
|
||||
}
|
||||
|
||||
// ---- dnf / zypper / pacman / apk ----
|
||||
|
||||
func dnfUpgradable(ctx context.Context) []Package {
|
||||
text, _ := out(ctx, "dnf", "-q", "check-update") // exit 100 when updates exist; output still parses
|
||||
var pkgs []Package
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) != 3 || !strings.Contains(f[0], ".") {
|
||||
continue
|
||||
}
|
||||
name := f[0][:strings.LastIndex(f[0], ".")]
|
||||
pkgs = append(pkgs, Package{Name: name, Candidate: f[1], Suite: f[2], Security: strings.Contains(f[2], "security")})
|
||||
}
|
||||
if pkgs == nil {
|
||||
return []Package{}
|
||||
}
|
||||
// check-update only names the candidate; one rpm query fills in what's installed.
|
||||
args := append([]string{"rpm", "-q", "--qf", `%{NAME} %|EPOCH?{%{EPOCH}:}:{}|%{VERSION}-%{RELEASE}\n`}, names(pkgs)...)
|
||||
if text, _ := out(ctx, args...); text != "" { // exits non-zero when any name isn't installed (new dependencies)
|
||||
cur := map[string]string{}
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
if f := strings.Fields(line); len(f) == 2 {
|
||||
cur[f[0]] = f[1] // several installed (kernels): the last listed is the newest
|
||||
}
|
||||
}
|
||||
for i := range pkgs {
|
||||
pkgs[i].Current = cur[pkgs[i].Name]
|
||||
}
|
||||
}
|
||||
return pkgs
|
||||
}
|
||||
|
||||
func pacmanUpgradable(ctx context.Context) []Package {
|
||||
text, _ := out(ctx, "pacman", "-Qu")
|
||||
var pkgs []Package
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
f := strings.Fields(line) // name old -> new
|
||||
if len(f) == 4 && f[2] == "->" {
|
||||
pkgs = append(pkgs, Package{Name: f[0], Current: f[1], Candidate: f[3]})
|
||||
}
|
||||
}
|
||||
if pkgs == nil {
|
||||
return []Package{}
|
||||
}
|
||||
return pkgs
|
||||
}
|
||||
|
||||
// zypperUpgradable parses the table `zypper list-updates` prints:
|
||||
// S | Repository | Name | Current Version | Available Version | Arch
|
||||
func zypperUpgradable(ctx context.Context) []Package {
|
||||
text, _ := out(ctx, "zypper", "--non-interactive", "--quiet", "list-updates")
|
||||
pkgs := []Package{}
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
f := strings.Split(line, "|")
|
||||
if len(f) < 5 || strings.TrimSpace(f[0]) != "v" {
|
||||
continue
|
||||
}
|
||||
repo := strings.TrimSpace(f[1])
|
||||
pkgs = append(pkgs, Package{Name: strings.TrimSpace(f[2]), Current: strings.TrimSpace(f[3]), Candidate: strings.TrimSpace(f[4]),
|
||||
Suite: repo, Security: strings.Contains(strings.ToLower(repo), "security")})
|
||||
}
|
||||
return pkgs
|
||||
}
|
||||
|
||||
var apkVerRe = regexp.MustCompile(`^(.+)-([^-]+-r\d+)\s+<\s+(\S+)`)
|
||||
|
||||
// apkUpgradable parses `apk version -l '<'`: "musl-1.2.4-r2 < 1.2.5-r0".
|
||||
func apkUpgradable(ctx context.Context) []Package {
|
||||
text, _ := out(ctx, "apk", "version", "-l", "<")
|
||||
pkgs := []Package{}
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
if m := apkVerRe.FindStringSubmatch(strings.TrimSpace(line)); m != nil {
|
||||
pkgs = append(pkgs, Package{Name: m[1], Current: m[2], Candidate: m[3]})
|
||||
}
|
||||
}
|
||||
return pkgs
|
||||
}
|
||||
|
||||
// pacmanLastUpgrade: the timestamp of the last "upgraded" line in pacman.log.
|
||||
func pacmanLastUpgrade() int64 {
|
||||
f, err := os.Open("/var/log/pacman.log")
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
defer f.Close()
|
||||
last := ""
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 1<<20)
|
||||
for sc.Scan() {
|
||||
if l := sc.Text(); strings.Contains(l, "[ALPM] upgraded ") && strings.HasPrefix(l, "[") {
|
||||
if end := strings.IndexByte(l, ']'); end > 1 {
|
||||
last = l[1:end]
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, layout := range []string{"2006-01-02T15:04:05-0700", "2006-01-02 15:04"} {
|
||||
if t, err := time.ParseInLocation(layout, last, time.Local); err == nil {
|
||||
return t.UnixMilli()
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ---- actions ----
|
||||
|
||||
// Never starts with '-', so a name can't be read as an option by any of the managers.
|
||||
var pkgNameRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9+._-]{0,127}$`)
|
||||
|
||||
func checkNames(pkgs []string) error {
|
||||
for _, p := range pkgs {
|
||||
if !pkgNameRe.MatchString(p) {
|
||||
return errors.New("bad package name: " + p)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manager) start(args ...string) (*tasks.Task, error) {
|
||||
if m.pm == "" {
|
||||
return nil, errors.New("no supported package manager on this host")
|
||||
}
|
||||
if !m.sudo {
|
||||
return nil, errors.New("needs root: run Ham Lab as root, or give its user passwordless sudo or doas")
|
||||
}
|
||||
if t, ok := m.tasks.RunningOf("updates"); ok {
|
||||
return nil, errors.New("another package task is running: " + t.Title)
|
||||
}
|
||||
full := priv.Wrap(append([]string{"env", "DEBIAN_FRONTEND=noninteractive", "APT_LISTCHANGES_FRONTEND=none", "LC_ALL=C"}, args...)...)
|
||||
cmd := exec.Command(full[0], full[1:]...)
|
||||
return m.tasks.Start("updates", strings.Join(args, " "), cmd)
|
||||
}
|
||||
|
||||
func (m *Manager) Refresh() (*tasks.Task, error) {
|
||||
switch m.pm {
|
||||
case "apt":
|
||||
return m.start("apt-get", "update")
|
||||
case "dnf":
|
||||
return m.start("dnf", "-y", "makecache", "--refresh")
|
||||
case "zypper":
|
||||
return m.start("zypper", "--non-interactive", "refresh")
|
||||
case "pacman":
|
||||
return m.start("pacman", "-Sy", "--noconfirm")
|
||||
case "apk":
|
||||
return m.start("apk", "update")
|
||||
}
|
||||
return m.start()
|
||||
}
|
||||
|
||||
// rolling reports a distro whose supported upgrade path is a full distribution upgrade.
|
||||
func rolling() bool {
|
||||
b, _ := os.ReadFile("/etc/os-release")
|
||||
return strings.Contains(string(b), "tumbleweed") || strings.Contains(string(b), "microos")
|
||||
}
|
||||
|
||||
// Upgrade upgrades everything, or only the named packages. Nothing here can block on a
|
||||
// question: apt keeps the current config file on a conflict, the rest run non-interactive.
|
||||
func (m *Manager) Upgrade(pkgs []string) (*tasks.Task, error) {
|
||||
if err := checkNames(pkgs); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
all := len(pkgs) == 0
|
||||
var args []string
|
||||
switch m.pm {
|
||||
case "apt":
|
||||
args = []string{"apt-get", "-y", "-o", "Dpkg::Options::=--force-confdef", "-o", "Dpkg::Options::=--force-confold"}
|
||||
if all {
|
||||
args = append(args, "dist-upgrade")
|
||||
} else {
|
||||
args = append(args, "install", "--only-upgrade")
|
||||
}
|
||||
case "dnf":
|
||||
args = []string{"dnf", "-y", "upgrade"}
|
||||
case "zypper":
|
||||
args = []string{"zypper", "--non-interactive", "update"}
|
||||
if all && rolling() {
|
||||
args = []string{"zypper", "--non-interactive", "dist-upgrade"}
|
||||
}
|
||||
case "pacman":
|
||||
if !all {
|
||||
return nil, errors.New("pacman upgrades everything or nothing: Arch doesn't support partial upgrades")
|
||||
}
|
||||
args = []string{"pacman", "-Syu", "--noconfirm"}
|
||||
case "apk":
|
||||
args = []string{"apk", "upgrade"}
|
||||
if all {
|
||||
args = []string{"apk", "upgrade", "--update-cache"}
|
||||
}
|
||||
}
|
||||
return m.start(append(args, pkgs...)...)
|
||||
}
|
||||
|
||||
// Install installs packages (used by other tabs for optional tooling like smartmontools).
|
||||
func (m *Manager) Install(pkgs []string) (*tasks.Task, error) {
|
||||
if len(pkgs) == 0 {
|
||||
return nil, errors.New("no packages")
|
||||
}
|
||||
if err := checkNames(pkgs); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var args []string
|
||||
switch m.pm {
|
||||
case "apt":
|
||||
args = []string{"apt-get", "-y", "--no-install-recommends", "install"}
|
||||
case "dnf":
|
||||
args = []string{"dnf", "-y", "install"}
|
||||
case "zypper":
|
||||
args = []string{"zypper", "--non-interactive", "install", "--no-recommends"}
|
||||
case "pacman":
|
||||
args = []string{"pacman", "-S", "--noconfirm", "--needed"}
|
||||
case "apk":
|
||||
args = []string{"apk", "add"}
|
||||
}
|
||||
return m.start(append(args, pkgs...)...)
|
||||
}
|
||||
|
||||
func (m *Manager) Changelog(ctx context.Context, pkg string) (string, error) {
|
||||
if !pkgNameRe.MatchString(pkg) {
|
||||
return "", errors.New("bad package name")
|
||||
}
|
||||
switch m.pm {
|
||||
case "apt":
|
||||
text, err := out(ctx, "apt-get", "changelog", "-qq", pkg)
|
||||
if len(text) > 64<<10 {
|
||||
text = text[:64<<10] + "\n…"
|
||||
}
|
||||
return text, err
|
||||
case "dnf":
|
||||
return out(ctx, "dnf", "-q", "changelog", pkg)
|
||||
}
|
||||
return "", errors.New("changelogs not available for " + m.pm)
|
||||
}
|
||||
|
||||
// Power runs systemctl reboot / poweroff.
|
||||
func (m *Manager) Power(ctx context.Context, action string) error {
|
||||
if action != "reboot" && action != "poweroff" {
|
||||
return errors.New("unknown power action")
|
||||
}
|
||||
_, err := out(ctx, m.sudoArgs("systemctl", action)...)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,575 @@
|
||||
package vpn
|
||||
|
||||
// Adding, renaming and removing the devices that dial in.
|
||||
//
|
||||
// Only OpenWrt's UCI is written, because that is where the server here is configured and
|
||||
// because a half-supported second path is worse than an honest refusal. A wg-quick host is
|
||||
// reported as unmanaged and the buttons do not appear.
|
||||
//
|
||||
// Two things happen for every change, in the order that fails safest:
|
||||
//
|
||||
// - Adding: the configuration is written first and committed, then the peer is applied to
|
||||
// the running tunnel. If applying fails the configuration is rolled back, so the two never
|
||||
// disagree silently — a peer that works only after the next reboot is the worst outcome
|
||||
// of the three.
|
||||
// - Removing: the running tunnel first, because that is the moment access actually ends,
|
||||
// then the configuration. If the second half fails the caller is told plainly that the
|
||||
// device is off now but would return on a reboot.
|
||||
//
|
||||
// Nothing here restarts the network. `wg set` changes one peer on a live interface and
|
||||
// touches nothing else, which matters rather a lot when the interface belongs to the router
|
||||
// the house is behind.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/netip"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// NameRe is what a device may be called: a short slug, because it also names the config
|
||||
// section it is stored in.
|
||||
var NameRe = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9 ._-]{0,30}$`)
|
||||
|
||||
// sectionRe guards the identifier that goes into a uci path.
|
||||
var sectionRe = regexp.MustCompile(`^[a-zA-Z0-9_]{1,48}$`)
|
||||
|
||||
const sectionPrefix = "wgpeer_"
|
||||
|
||||
var (
|
||||
ErrUnmanaged = errors.New("this tunnel is not configured through UCI, so Ham Lab will not edit it here")
|
||||
ErrNoRoom = errors.New("no free address is left in this tunnel's range")
|
||||
)
|
||||
|
||||
// NewPeer is what the caller asks for.
|
||||
type NewPeer struct {
|
||||
Iface string `json:"-"`
|
||||
Name string `json:"name"`
|
||||
// Address is the tunnel address to give the device, with or without a prefix. Empty means
|
||||
// the next free one.
|
||||
Address string `json:"address"`
|
||||
// Endpoint is what the device will dial: the house's name or address, and the port.
|
||||
Endpoint string `json:"endpoint"`
|
||||
// DNS is what the device should use while connected. Empty leaves the line out.
|
||||
DNS string `json:"dns"`
|
||||
// Split routes only the home networks through the tunnel instead of everything.
|
||||
Split bool `json:"split"`
|
||||
}
|
||||
|
||||
// Created is a new device, and the one chance to read its configuration.
|
||||
type Created struct {
|
||||
Section string `json:"section"`
|
||||
Name string `json:"name"`
|
||||
PublicKey string `json:"public_key"`
|
||||
Address string `json:"address"`
|
||||
// Config is the complete client profile. It holds the private key, which exists nowhere
|
||||
// else: it is generated here, handed over once and never stored.
|
||||
Config string `json:"config"`
|
||||
}
|
||||
|
||||
// uciPeer is one peer section as UCI holds it.
|
||||
type uciPeer struct {
|
||||
Section string
|
||||
Iface string // the wireguard_<iface> type it belongs to
|
||||
Desc string
|
||||
PublicKey string
|
||||
AllowedIPs []string
|
||||
}
|
||||
|
||||
// uciPeers reads every peer section. The type of a peer section names its interface
|
||||
// ("wireguard_wg0"), which is how a peer is tied to a tunnel.
|
||||
func uciPeers(ctx context.Context) ([]uciPeer, error) {
|
||||
if _, err := exec.LookPath("uci"); err != nil {
|
||||
return nil, ErrUnmanaged
|
||||
}
|
||||
b, err := exec.CommandContext(ctx, "uci", "-q", "show", "network").Output()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
by := map[string]*uciPeer{}
|
||||
var order []string
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
lhs, rhs, ok := strings.Cut(line, "=")
|
||||
if !ok || !strings.HasPrefix(lhs, "network.") {
|
||||
continue
|
||||
}
|
||||
parts := strings.Split(strings.TrimPrefix(lhs, "network."), ".")
|
||||
v := strings.Trim(rhs, "'")
|
||||
section := parts[0]
|
||||
if len(parts) == 1 { // the section's own type
|
||||
if !strings.HasPrefix(v, "wireguard_") {
|
||||
continue
|
||||
}
|
||||
by[section] = &uciPeer{Section: section, Iface: strings.TrimPrefix(v, "wireguard_")}
|
||||
order = append(order, section)
|
||||
continue
|
||||
}
|
||||
p := by[section]
|
||||
if p == nil {
|
||||
continue
|
||||
}
|
||||
switch parts[1] {
|
||||
case "description":
|
||||
p.Desc = v
|
||||
case "public_key":
|
||||
p.PublicKey = v
|
||||
case "allowed_ips":
|
||||
p.AllowedIPs = strings.Fields(strings.ReplaceAll(v, ",", " "))
|
||||
}
|
||||
}
|
||||
out := make([]uciPeer, 0, len(order))
|
||||
for _, s := range order {
|
||||
out = append(out, *by[s])
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Managed reports whether this tunnel's peers can be edited here, and labels each peer with
|
||||
// the section that holds it so the caller has a safe identifier to act on.
|
||||
func Managed(ctx context.Context, t *Tunnel) bool {
|
||||
peers, err := uciPeers(ctx)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
found := false
|
||||
byKey := map[string]string{}
|
||||
for _, p := range peers {
|
||||
if p.Iface == t.Name {
|
||||
byKey[p.PublicKey] = p.Section
|
||||
found = true
|
||||
}
|
||||
}
|
||||
for i := range t.Peers {
|
||||
t.Peers[i].Section = byKey[t.Peers[i].PublicKey]
|
||||
}
|
||||
// A tunnel with no peers at all is still manageable if UCI is the thing configuring it.
|
||||
if !found {
|
||||
for _, line := range uciInterfaces(ctx) {
|
||||
if line == t.Name {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return found
|
||||
}
|
||||
|
||||
// uciInterfaces is every interface UCI configures as wireguard.
|
||||
func uciInterfaces(ctx context.Context) []string {
|
||||
var out []string
|
||||
if _, err := exec.LookPath("uci"); err != nil {
|
||||
return out
|
||||
}
|
||||
b, err := exec.CommandContext(ctx, "uci", "-q", "show", "network").Output()
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
lhs, rhs, ok := strings.Cut(line, "=")
|
||||
if !ok || strings.Trim(rhs, "'") != "wireguard" {
|
||||
continue
|
||||
}
|
||||
parts := strings.Split(strings.TrimPrefix(lhs, "network."), ".")
|
||||
if len(parts) == 2 && parts[1] == "proto" {
|
||||
out = append(out, parts[0])
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// NextAddress is the lowest address in the tunnel's range that nothing is using. The
|
||||
// interface's own address is taken, as is every address any peer is allowed.
|
||||
func NextAddress(t Tunnel) (string, error) {
|
||||
var base netip.Prefix
|
||||
for _, a := range t.Addrs {
|
||||
p, err := netip.ParsePrefix(a)
|
||||
if err != nil || !p.Addr().Is4() {
|
||||
continue
|
||||
}
|
||||
base = p
|
||||
break
|
||||
}
|
||||
if !base.IsValid() {
|
||||
return "", errors.New("this tunnel has no IPv4 range to allocate from")
|
||||
}
|
||||
taken := map[netip.Addr]bool{base.Addr(): true}
|
||||
for _, p := range t.Peers {
|
||||
for _, a := range p.AllowedIPs {
|
||||
if ip, err := netip.ParsePrefix(strings.TrimSpace(a)); err == nil {
|
||||
taken[ip.Addr()] = true
|
||||
} else if ip, err := netip.ParseAddr(strings.TrimSpace(a)); err == nil {
|
||||
taken[ip] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
ip := base.Masked().Addr().Next() // .0 is the network
|
||||
for base.Contains(ip) {
|
||||
next := ip.Next()
|
||||
if !base.Contains(next) { // the last address in the range is the broadcast
|
||||
break
|
||||
}
|
||||
if !taken[ip] {
|
||||
return ip.String(), nil
|
||||
}
|
||||
ip = next
|
||||
}
|
||||
return "", ErrNoRoom
|
||||
}
|
||||
|
||||
// slug turns a device name into a section name, which may hold rather less.
|
||||
func slug(name string) string {
|
||||
var b strings.Builder
|
||||
for _, r := range strings.ToLower(name) {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z', r >= '0' && r <= '9':
|
||||
b.WriteRune(r)
|
||||
case r == ' ', r == '-', r == '.', r == '_':
|
||||
b.WriteByte('_')
|
||||
}
|
||||
}
|
||||
s := strings.Trim(b.String(), "_")
|
||||
if s == "" {
|
||||
s = "peer"
|
||||
}
|
||||
if len(s) > 24 {
|
||||
s = s[:24]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// genKeys makes a private key, its public half and a pre-shared key. The private key is
|
||||
// returned to be handed over once and is never written anywhere by this daemon.
|
||||
func genKeys(ctx context.Context) (priv, pub, psk string, err error) {
|
||||
run := func(name string, stdin string, args ...string) (string, error) {
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
if stdin != "" {
|
||||
cmd.Stdin = strings.NewReader(stdin)
|
||||
}
|
||||
out, err := cmd.Output()
|
||||
return strings.TrimSpace(string(out)), err
|
||||
}
|
||||
if priv, err = run("wg", "", "genkey"); err != nil {
|
||||
return "", "", "", fmt.Errorf("generating a key: %w", err)
|
||||
}
|
||||
if pub, err = run("wg", priv+"\n", "pubkey"); err != nil {
|
||||
return "", "", "", fmt.Errorf("deriving the public key: %w", err)
|
||||
}
|
||||
if psk, err = run("wg", "", "genpsk"); err != nil {
|
||||
return "", "", "", fmt.Errorf("generating a pre-shared key: %w", err)
|
||||
}
|
||||
return priv, pub, psk, nil
|
||||
}
|
||||
|
||||
// AddPeer creates a device: it writes the peer into the configuration, applies it to the
|
||||
// running tunnel, and returns the client profile once.
|
||||
func AddPeer(ctx context.Context, t Tunnel, n NewPeer) (*Created, error) {
|
||||
if !NameRe.MatchString(strings.TrimSpace(n.Name)) {
|
||||
return nil, errors.New("a device name is 1–31 letters, digits, spaces, dots, dashes or underscores")
|
||||
}
|
||||
name := strings.TrimSpace(n.Name)
|
||||
if strings.TrimSpace(n.Endpoint) == "" {
|
||||
return nil, errors.New("the device needs an endpoint to dial: the house's address or name, and the port")
|
||||
}
|
||||
endpoint, err := checkEndpoint(n.Endpoint)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
addr := strings.TrimSpace(n.Address)
|
||||
if addr == "" {
|
||||
if addr, err = NextAddress(t); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
host, err := hostOf(addr)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, p := range t.Peers {
|
||||
for _, a := range p.AllowedIPs {
|
||||
if strings.HasPrefix(strings.TrimSpace(a), host+"/") {
|
||||
return nil, fmt.Errorf("%s is already used by %s", host, firstNonEmpty(p.Name, "another device"))
|
||||
}
|
||||
}
|
||||
}
|
||||
section := sectionPrefix + slug(name)
|
||||
existing, err := uciPeers(ctx)
|
||||
if err != nil {
|
||||
return nil, ErrUnmanaged
|
||||
}
|
||||
taken := map[string]bool{}
|
||||
for _, p := range existing {
|
||||
taken[p.Section] = true
|
||||
}
|
||||
for i := 2; taken[section]; i++ {
|
||||
section = fmt.Sprintf("%s%s%d", sectionPrefix, slug(name), i)
|
||||
}
|
||||
if !sectionRe.MatchString(section) {
|
||||
return nil, errors.New("that name cannot be turned into a configuration section")
|
||||
}
|
||||
|
||||
priv, pub, psk, err := genKeys(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
allowed := host + "/32"
|
||||
|
||||
// the configuration first, so a peer can never be live but unrecorded
|
||||
set := [][]string{
|
||||
{"set", "network." + section + "=wireguard_" + t.Name},
|
||||
{"set", "network." + section + ".description=" + name},
|
||||
{"set", "network." + section + ".public_key=" + pub},
|
||||
{"set", "network." + section + ".preshared_key=" + psk},
|
||||
{"set", "network." + section + ".allowed_ips=" + allowed},
|
||||
}
|
||||
for _, args := range set {
|
||||
if err := uci(ctx, args...); err != nil {
|
||||
_ = uci(ctx, "revert", "network")
|
||||
return nil, fmt.Errorf("writing the configuration: %w", err)
|
||||
}
|
||||
}
|
||||
if err := uci(ctx, "commit", "network"); err != nil {
|
||||
_ = uci(ctx, "revert", "network")
|
||||
return nil, fmt.Errorf("saving the configuration: %w", err)
|
||||
}
|
||||
|
||||
// then the running tunnel; if this fails the configuration goes back as it was
|
||||
if err := wgSet(ctx, t.Name, pub, psk, allowed); err != nil {
|
||||
_ = uci(ctx, "delete", "network."+section)
|
||||
_ = uci(ctx, "commit", "network")
|
||||
return nil, fmt.Errorf("applying it to the running tunnel: %w", err)
|
||||
}
|
||||
|
||||
return &Created{
|
||||
Section: section, Name: name, PublicKey: pub, Address: host,
|
||||
Config: profile(priv, psk, t, host, endpoint, n.DNS, n.Split),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// EditPeer changes a device's label or the address it is allowed. The section keeps the name
|
||||
// it was created with; the description is what is shown.
|
||||
func EditPeer(ctx context.Context, t Tunnel, section, name, address string) error {
|
||||
if !sectionRe.MatchString(section) {
|
||||
return errors.New("that is not a device this can edit")
|
||||
}
|
||||
peers, err := uciPeers(ctx)
|
||||
if err != nil {
|
||||
return ErrUnmanaged
|
||||
}
|
||||
var me *uciPeer
|
||||
for i := range peers {
|
||||
if peers[i].Section == section && peers[i].Iface == t.Name {
|
||||
me = &peers[i]
|
||||
}
|
||||
}
|
||||
if me == nil {
|
||||
return errors.New("no such device on this tunnel")
|
||||
}
|
||||
if name = strings.TrimSpace(name); name != "" {
|
||||
if !NameRe.MatchString(name) {
|
||||
return errors.New("a device name is 1–31 letters, digits, spaces, dots, dashes or underscores")
|
||||
}
|
||||
if err := uci(ctx, "set", "network."+section+".description="+name); err != nil {
|
||||
_ = uci(ctx, "revert", "network")
|
||||
return err
|
||||
}
|
||||
}
|
||||
allowed := ""
|
||||
if address = strings.TrimSpace(address); address != "" {
|
||||
host, err := hostOf(address)
|
||||
if err != nil {
|
||||
_ = uci(ctx, "revert", "network")
|
||||
return err
|
||||
}
|
||||
for _, p := range t.Peers {
|
||||
if p.Section == section {
|
||||
continue
|
||||
}
|
||||
for _, a := range p.AllowedIPs {
|
||||
if strings.HasPrefix(strings.TrimSpace(a), host+"/") {
|
||||
_ = uci(ctx, "revert", "network")
|
||||
return fmt.Errorf("%s is already used by %s", host, firstNonEmpty(p.Name, "another device"))
|
||||
}
|
||||
}
|
||||
}
|
||||
allowed = host + "/32"
|
||||
if err := uci(ctx, "set", "network."+section+".allowed_ips="+allowed); err != nil {
|
||||
_ = uci(ctx, "revert", "network")
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := uci(ctx, "commit", "network"); err != nil {
|
||||
_ = uci(ctx, "revert", "network")
|
||||
return err
|
||||
}
|
||||
if allowed != "" { // the running tunnel has to be told, or the device keeps the old address
|
||||
if err := wgSet(ctx, t.Name, me.PublicKey, "", allowed); err != nil {
|
||||
return fmt.Errorf("saved, but the running tunnel refused the new address: %w", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeletePeer removes a device. The running tunnel goes first, because that is the moment the
|
||||
// device actually loses its way in.
|
||||
func DeletePeer(ctx context.Context, t Tunnel, section string) error {
|
||||
if !sectionRe.MatchString(section) {
|
||||
return errors.New("that is not a device this can remove")
|
||||
}
|
||||
peers, err := uciPeers(ctx)
|
||||
if err != nil {
|
||||
return ErrUnmanaged
|
||||
}
|
||||
var me *uciPeer
|
||||
for i := range peers {
|
||||
if peers[i].Section == section && peers[i].Iface == t.Name {
|
||||
me = &peers[i]
|
||||
}
|
||||
}
|
||||
if me == nil {
|
||||
return errors.New("no such device on this tunnel")
|
||||
}
|
||||
if err := wgRemove(ctx, t.Name, me.PublicKey); err != nil {
|
||||
return fmt.Errorf("removing it from the running tunnel: %w", err)
|
||||
}
|
||||
if err := uci(ctx, "delete", "network."+section); err != nil {
|
||||
return fmt.Errorf("the device is disconnected, but removing it from the configuration failed, so it would come back on a reboot: %w", err)
|
||||
}
|
||||
if err := uci(ctx, "commit", "network"); err != nil {
|
||||
return fmt.Errorf("the device is disconnected, but saving the configuration failed, so it would come back on a reboot: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ---- the small dangerous bits, each in one place ----
|
||||
|
||||
func uci(ctx context.Context, args ...string) error {
|
||||
out, err := exec.CommandContext(ctx, "uci", args...).CombinedOutput()
|
||||
if err != nil {
|
||||
return fmt.Errorf("uci %s: %s", strings.Join(args, " "), strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// wgSet applies a peer to the running interface. The pre-shared key goes in through a file
|
||||
// because that is the only way wg takes one, so the file is made readable by nobody else and
|
||||
// removed immediately.
|
||||
func wgSet(ctx context.Context, iface, pub, psk, allowed string) error {
|
||||
args := []string{"set", iface, "peer", pub}
|
||||
if psk != "" {
|
||||
f, err := os.CreateTemp("", "hamlab-psk")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.Remove(f.Name())
|
||||
if err := f.Chmod(0o600); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if _, err := f.WriteString(psk + "\n"); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
f.Close()
|
||||
args = append(args, "preshared-key", f.Name())
|
||||
}
|
||||
args = append(args, "allowed-ips", allowed)
|
||||
out, err := exec.CommandContext(ctx, "wg", args...).CombinedOutput()
|
||||
if err != nil {
|
||||
return fmt.Errorf("wg set: %s", strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func wgRemove(ctx context.Context, iface, pub string) error {
|
||||
out, err := exec.CommandContext(ctx, "wg", "set", iface, "peer", pub, "remove").CombinedOutput()
|
||||
if err != nil {
|
||||
return fmt.Errorf("wg set: %s", strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// profile is the client's own configuration file, which is the only place its private key
|
||||
// ever appears.
|
||||
func profile(priv, psk string, t Tunnel, addr, endpoint, dns string, split bool) string {
|
||||
prefix := 24
|
||||
for _, a := range t.Addrs {
|
||||
if p, err := netip.ParsePrefix(a); err == nil && p.Addr().Is4() {
|
||||
prefix = p.Bits()
|
||||
break
|
||||
}
|
||||
}
|
||||
allowed := "0.0.0.0/0"
|
||||
if split {
|
||||
nets := []string{}
|
||||
for _, a := range t.Addrs {
|
||||
if p, err := netip.ParsePrefix(a); err == nil && p.Addr().Is4() {
|
||||
nets = append(nets, p.Masked().String())
|
||||
}
|
||||
}
|
||||
nets = append(nets, "10.0.0.0/8", "192.168.0.0/16", "172.16.0.0/12")
|
||||
allowed = strings.Join(dedupe(nets), ", ")
|
||||
}
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "[Interface]\nPrivateKey = %s\nAddress = %s/%d\n", priv, addr, prefix)
|
||||
if dns = strings.TrimSpace(dns); dns != "" {
|
||||
fmt.Fprintf(&b, "DNS = %s\n", dns)
|
||||
}
|
||||
fmt.Fprintf(&b, "\n[Peer]\nPublicKey = %s\n", t.PublicKey)
|
||||
if psk != "" {
|
||||
fmt.Fprintf(&b, "PresharedKey = %s\n", psk)
|
||||
}
|
||||
fmt.Fprintf(&b, "Endpoint = %s\nAllowedIPs = %s\n", endpoint, allowed)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func dedupe(in []string) []string {
|
||||
seen := map[string]bool{}
|
||||
out := in[:0:0]
|
||||
for _, s := range in {
|
||||
if !seen[s] {
|
||||
seen[s] = true
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// hostOf takes an address with or without a prefix and returns the bare address.
|
||||
func hostOf(s string) (string, error) {
|
||||
s = strings.TrimSpace(s)
|
||||
if p, err := netip.ParsePrefix(s); err == nil {
|
||||
return p.Addr().String(), nil
|
||||
}
|
||||
a, err := netip.ParseAddr(s)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%q is not an address", s)
|
||||
}
|
||||
return a.String(), nil
|
||||
}
|
||||
|
||||
// checkEndpoint makes sure what the device will dial looks like a host and a port.
|
||||
func checkEndpoint(s string) (string, error) {
|
||||
s = strings.TrimSpace(s)
|
||||
host, port, err := net.SplitHostPort(s)
|
||||
if err != nil || host == "" {
|
||||
return "", fmt.Errorf("%q should be a name or address and a port, like home.example.com:51820", s)
|
||||
}
|
||||
if n, err := strconv.Atoi(port); err != nil || n < 1 || n > 65535 {
|
||||
return "", fmt.Errorf("%q is not a port", port)
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
func firstNonEmpty(a, b string) string {
|
||||
if a != "" {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
package vpn
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func tunnel(addrs []string, peers ...string) Tunnel {
|
||||
t := Tunnel{Name: "wg0", PublicKey: "SERVERKEY=", Addrs: addrs}
|
||||
for _, p := range peers {
|
||||
t.Peers = append(t.Peers, Peer{PublicKey: p + "KEY=", AllowedIPs: []string{p}})
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func TestNextAddress(t *testing.T) {
|
||||
// .1 is the interface itself and .2 and .3 are taken, so the next free one is .4
|
||||
got, err := NextAddress(tunnel([]string{"172.21.46.1/24"}, "172.21.46.3/32", "172.21.46.2/32"))
|
||||
if err != nil || got != "172.21.46.4" {
|
||||
t.Errorf("NextAddress = %q, %v; want 172.21.46.4", got, err)
|
||||
}
|
||||
// a gap is filled before the end is reached
|
||||
got, _ = NextAddress(tunnel([]string{"172.21.46.1/24"}, "172.21.46.2/32", "172.21.46.4/32"))
|
||||
if got != "172.21.46.3" {
|
||||
t.Errorf("NextAddress = %q, want the gap at .3", got)
|
||||
}
|
||||
// an empty tunnel starts after the server
|
||||
got, _ = NextAddress(tunnel([]string{"10.9.0.1/24"}))
|
||||
if got != "10.9.0.2" {
|
||||
t.Errorf("NextAddress = %q, want 10.9.0.2", got)
|
||||
}
|
||||
// a full one says so rather than handing out a duplicate
|
||||
full := tunnel([]string{"10.9.0.1/30"}, "10.9.0.2/32")
|
||||
if _, err := NextAddress(full); err != ErrNoRoom {
|
||||
t.Errorf("err = %v, want ErrNoRoom on a full range", err)
|
||||
}
|
||||
// IPv6-only, or nothing at all, is refused rather than guessed
|
||||
if _, err := NextAddress(tunnel([]string{"fc00::1/64"})); err == nil {
|
||||
t.Error("want an error when there is no IPv4 range")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlug(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"ip12": "ip12",
|
||||
"Mitchell's iPad": "mitchells_ipad",
|
||||
"my-phone": "my_phone",
|
||||
" spaced out ": "spaced__out",
|
||||
"!!!": "peer",
|
||||
strings.Repeat("a", 40): strings.Repeat("a", 24),
|
||||
} {
|
||||
if got := slug(in); got != want {
|
||||
t.Errorf("slug(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfile(t *testing.T) {
|
||||
tun := tunnel([]string{"172.21.46.1/24"})
|
||||
got := profile("PRIV=", "PSK=", tun, "172.21.46.4", "home.example.com:51820", "1.1.1.1, 1.0.0.1", false)
|
||||
for _, want := range []string{
|
||||
"PrivateKey = PRIV=",
|
||||
"Address = 172.21.46.4/24", // the interface's own prefix, not a bare /32
|
||||
"DNS = 1.1.1.1, 1.0.0.1",
|
||||
"PublicKey = SERVERKEY=",
|
||||
"PresharedKey = PSK=",
|
||||
"Endpoint = home.example.com:51820",
|
||||
"AllowedIPs = 0.0.0.0/0",
|
||||
} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("profile is missing %q:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
// no DNS line at all when none was asked for, rather than an empty one
|
||||
if strings.Contains(profile("P=", "", tun, "172.21.46.4", "h:1", "", false), "DNS") {
|
||||
t.Error("an empty DNS should leave the line out")
|
||||
}
|
||||
// split routes the private ranges instead of everything
|
||||
split := profile("P=", "", tun, "172.21.46.4", "h:1", "", true)
|
||||
if strings.Contains(split, "0.0.0.0/0") {
|
||||
t.Errorf("a split tunnel must not route everything:\n%s", split)
|
||||
}
|
||||
if !strings.Contains(split, "172.21.46.0/24") {
|
||||
t.Errorf("a split tunnel should include the tunnel's own range:\n%s", split)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckEndpoint(t *testing.T) {
|
||||
for _, ok := range []string{"bigham.dev:51820", "1.2.3.4:51820", "[2001:db8::1]:51820"} {
|
||||
if _, err := checkEndpoint(ok); err != nil {
|
||||
t.Errorf("checkEndpoint(%q) = %v, want it accepted", ok, err)
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{"", "bigham.dev", ":51820", "bigham.dev:0", "bigham.dev:notaport", "bigham.dev:99999"} {
|
||||
if _, err := checkEndpoint(bad); err == nil {
|
||||
t.Errorf("checkEndpoint(%q) was accepted, want refused", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHostOf(t *testing.T) {
|
||||
for in, want := range map[string]string{"172.21.46.4": "172.21.46.4", "172.21.46.4/32": "172.21.46.4", " 10.0.0.9/24 ": "10.0.0.9"} {
|
||||
got, err := hostOf(in)
|
||||
if err != nil || got != want {
|
||||
t.Errorf("hostOf(%q) = %q, %v; want %q", in, got, err, want)
|
||||
}
|
||||
}
|
||||
if _, err := hostOf("not-an-address"); err == nil {
|
||||
t.Error("hostOf accepted nonsense")
|
||||
}
|
||||
}
|
||||
|
||||
// The name is what becomes a configuration section, so it is the input worth being strict
|
||||
// about; everything refused here is refused before anything is written.
|
||||
func TestNameRe(t *testing.T) {
|
||||
for _, ok := range []string{"ip12", "My Phone", "a", "work-laptop.2"} {
|
||||
if !NameRe.MatchString(ok) {
|
||||
t.Errorf("%q should be a valid name", ok)
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{"", " leading", "semi;colon", "quote'd", "back`tick", "new\nline", "$(whoami)", strings.Repeat("a", 32)} {
|
||||
if NameRe.MatchString(bad) {
|
||||
t.Errorf("%q should be refused as a name", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,372 @@
|
||||
// Package vpn reads this machine's WireGuard tunnels: the interfaces, and the peers on them
|
||||
// with names put back on.
|
||||
//
|
||||
// The live state comes from `wg show all dump`, which is the kernel's own answer and the only
|
||||
// place handshakes and byte counts exist. The kernel knows nothing about names, though — a
|
||||
// peer there is a public key — so the configuration is read alongside it purely to label
|
||||
// them: `# comments` in a wg-quick .conf, or the `description` of a UCI peer section on
|
||||
// OpenWrt. Everything here is read-only, and no private or pre-shared key is ever read, let
|
||||
// alone returned.
|
||||
package vpn
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"hamlab/internal/priv"
|
||||
)
|
||||
|
||||
// Peer is one end of a tunnel: a device that dials in, or the server this machine dials out
|
||||
// to. Which it is depends on the tunnel; see Tunnel.Kind.
|
||||
type Peer struct {
|
||||
// Name is what the configuration calls it, where it says at all. The kernel has no idea.
|
||||
Name string `json:"name,omitempty"`
|
||||
PublicKey string `json:"public_key"`
|
||||
// Endpoint is where its packets come from, empty until it has been heard from.
|
||||
Endpoint string `json:"endpoint,omitempty"`
|
||||
AllowedIPs []string `json:"allowed_ips"`
|
||||
// Handshake is the last completed handshake, absent when there has never been one — a peer
|
||||
// that is configured but has not connected since this tunnel came up. A pointer because
|
||||
// omitempty does nothing for a zero time.Time, and a peer that has never been heard from
|
||||
// would otherwise be reported as having last spoken in the year one.
|
||||
Handshake *time.Time `json:"handshake,omitempty"`
|
||||
// Online is a handshake inside the last three minutes. WireGuard is silent when idle, so
|
||||
// this means "was talking recently", not "is connected" — there is no such thing here.
|
||||
Online bool `json:"online"`
|
||||
Rx int64 `json:"rx"`
|
||||
Tx int64 `json:"tx"`
|
||||
Keepalive int `json:"keepalive,omitempty"`
|
||||
// PSK notes that a pre-shared key is in use, without the key itself going anywhere.
|
||||
PSK bool `json:"psk,omitempty"`
|
||||
// Section is the configuration section holding this peer, and the identifier anything that
|
||||
// edits it uses. Empty when the peer is not in a configuration this can edit.
|
||||
Section string `json:"section,omitempty"`
|
||||
}
|
||||
|
||||
// Tunnel is one WireGuard interface.
|
||||
type Tunnel struct {
|
||||
Name string `json:"name"`
|
||||
// Managed is whether devices on this tunnel can be added and removed from here, which
|
||||
// today means its peers are configured through OpenWrt's UCI.
|
||||
Managed bool `json:"managed"`
|
||||
PublicKey string `json:"public_key"`
|
||||
// Kind is "server" when peers dial in to this machine, "client" when this machine dials
|
||||
// out through a peer, and "" when it is too new or too odd to tell. It is worked out from
|
||||
// the shape of the peers, not from anything WireGuard states: a peer that is routed
|
||||
// everything and has an endpoint of its own is an exit, and a tunnel with one of those is
|
||||
// this machine going out through it.
|
||||
Kind string `json:"kind"`
|
||||
ListenPort int `json:"listen_port,omitempty"`
|
||||
FwMark string `json:"fwmark,omitempty"`
|
||||
Addrs []string `json:"addrs"`
|
||||
MTU int `json:"mtu,omitempty"`
|
||||
Peers []Peer `json:"peers"`
|
||||
// Exit is the peer this machine goes out through, for a client tunnel.
|
||||
Exit string `json:"exit,omitempty"`
|
||||
}
|
||||
|
||||
// Report is every tunnel on this machine, and why there are none when there are none.
|
||||
type Report struct {
|
||||
At time.Time `json:"at"`
|
||||
Tunnels []Tunnel `json:"tunnels"`
|
||||
// Tool is false when wireguard-tools is not installed, Root false when the daemon cannot
|
||||
// reach root — either one means the peers cannot be read, and the tab should say which.
|
||||
Tool bool `json:"tool"`
|
||||
Root bool `json:"root"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
const onlineWithin = 3 * time.Minute
|
||||
|
||||
// Look reads the tunnels on this machine.
|
||||
func Look(ctx context.Context) Report {
|
||||
r := Report{At: time.Now(), Tunnels: []Tunnel{}, Root: priv.Available()}
|
||||
if _, err := exec.LookPath("wg"); err != nil {
|
||||
r.Note = "wireguard-tools is not installed here, so there is nothing to read"
|
||||
return r
|
||||
}
|
||||
r.Tool = true
|
||||
args := []string{"wg", "show", "all", "dump"}
|
||||
if priv.Available() {
|
||||
args = priv.Wrap(args...)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
if !r.Root {
|
||||
r.Note = "reading WireGuard needs root, and this daemon has no way to it"
|
||||
} else {
|
||||
r.Note = "wg show failed: " + strings.TrimSpace(err.Error())
|
||||
}
|
||||
return r
|
||||
}
|
||||
r.Tunnels = parseDump(string(out))
|
||||
names := peerNames()
|
||||
for i := range r.Tunnels {
|
||||
t := &r.Tunnels[i]
|
||||
t.Addrs, t.MTU = ifaceAddrs(t.Name)
|
||||
for j := range t.Peers {
|
||||
p := &t.Peers[j]
|
||||
p.Name = names[p.PublicKey]
|
||||
p.Online = p.Handshake != nil && time.Since(*p.Handshake) < onlineWithin
|
||||
}
|
||||
t.Kind, t.Exit = classify(*t)
|
||||
t.Managed = Managed(ctx, t)
|
||||
sort.SliceStable(t.Peers, func(a, b int) bool {
|
||||
pa, pb := t.Peers[a], t.Peers[b]
|
||||
if pa.Online != pb.Online {
|
||||
return pa.Online
|
||||
}
|
||||
if ha, hb := shookAt(pa), shookAt(pb); !ha.Equal(hb) {
|
||||
return ha.After(hb)
|
||||
}
|
||||
return pa.Name < pb.Name
|
||||
})
|
||||
}
|
||||
if len(r.Tunnels) == 0 {
|
||||
r.Note = "no WireGuard interfaces are up on this machine"
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// parseDump reads `wg show all dump`. The first line of each interface is the interface
|
||||
// itself, the rest are its peers; every line is tab-separated and positional.
|
||||
//
|
||||
// interface: name private-key public-key listen-port fwmark
|
||||
// peer: name public-key psk endpoint allowed-ips handshake rx tx keepalive
|
||||
//
|
||||
// The private key is in the second column and is read past without being kept.
|
||||
func parseDump(out string) []Tunnel {
|
||||
// An empty list, never nil: this is JSON to a browser that will take its length, and a
|
||||
// null there is a blank page rather than "no tunnels".
|
||||
tunnels := []Tunnel{}
|
||||
byName := map[string]int{}
|
||||
sc := bufio.NewScanner(strings.NewReader(out))
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 1<<20)
|
||||
for sc.Scan() {
|
||||
f := strings.Split(sc.Text(), "\t")
|
||||
if len(f) < 5 {
|
||||
continue
|
||||
}
|
||||
name := f[0]
|
||||
if len(f) == 5 { // an interface line
|
||||
port, _ := strconv.Atoi(f[3])
|
||||
byName[name] = len(tunnels)
|
||||
tunnels = append(tunnels, Tunnel{Name: name, PublicKey: f[2], ListenPort: port, FwMark: off(f[4]), Peers: []Peer{}})
|
||||
continue
|
||||
}
|
||||
if len(f) < 9 {
|
||||
continue
|
||||
}
|
||||
i, ok := byName[name]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
hs, _ := strconv.ParseInt(f[5], 10, 64)
|
||||
rx, _ := strconv.ParseInt(f[6], 10, 64)
|
||||
tx, _ := strconv.ParseInt(f[7], 10, 64)
|
||||
ka, _ := strconv.Atoi(f[8])
|
||||
p := Peer{PublicKey: f[1], PSK: f[2] != "(none)", Endpoint: off(f[3]), Rx: rx, Tx: tx, Keepalive: ka}
|
||||
if f[4] != "(none)" {
|
||||
p.AllowedIPs = strings.Split(f[4], ",")
|
||||
}
|
||||
if hs > 0 {
|
||||
at := time.Unix(hs, 0)
|
||||
p.Handshake = &at
|
||||
}
|
||||
tunnels[i].Peers = append(tunnels[i].Peers, p)
|
||||
}
|
||||
return tunnels
|
||||
}
|
||||
|
||||
// shookAt is a peer's last handshake, or the zero time when there has never been one.
|
||||
func shookAt(p Peer) time.Time {
|
||||
if p.Handshake == nil {
|
||||
return time.Time{}
|
||||
}
|
||||
return *p.Handshake
|
||||
}
|
||||
|
||||
// off turns wg's placeholders into an empty string.
|
||||
func off(v string) string {
|
||||
if v == "(none)" || v == "off" {
|
||||
return ""
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// classify decides whether this machine is the one being dialled or the one dialling. A peer
|
||||
// that is allowed every address and has an endpoint is a way out to the internet; a tunnel
|
||||
// with one of those is this machine using it. Otherwise, if anything can reach us here, it is
|
||||
// a server.
|
||||
func classify(t Tunnel) (kind, exit string) {
|
||||
for _, p := range t.Peers {
|
||||
for _, a := range p.AllowedIPs {
|
||||
if a == "0.0.0.0/0" || a == "::/0" {
|
||||
if p.Endpoint != "" {
|
||||
name := p.Name
|
||||
if name == "" {
|
||||
name = p.Endpoint
|
||||
}
|
||||
return "client", name
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if t.ListenPort > 0 && len(t.Peers) > 0 {
|
||||
return "server", ""
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// ifaceAddrs is what the interface carries, and its MTU.
|
||||
func ifaceAddrs(name string) ([]string, int) {
|
||||
addrs := []string{}
|
||||
mtu := 0
|
||||
ifi, err := net.InterfaceByName(name)
|
||||
if err != nil {
|
||||
return addrs, mtu
|
||||
}
|
||||
mtu = ifi.MTU
|
||||
got, err := ifi.Addrs()
|
||||
if err != nil {
|
||||
return addrs, mtu
|
||||
}
|
||||
for _, a := range got {
|
||||
addrs = append(addrs, a.String())
|
||||
}
|
||||
return addrs, mtu
|
||||
}
|
||||
|
||||
// ---- names ----
|
||||
|
||||
// peerNames maps a public key to what the configuration calls that peer. Two shapes are
|
||||
// understood, which between them cover every machine here: wg-quick's .conf files, where the
|
||||
// name is a comment, and OpenWrt's UCI, where it is the peer section's description.
|
||||
func peerNames() map[string]string {
|
||||
out := map[string]string{}
|
||||
for k, v := range confNames() {
|
||||
out[k] = v
|
||||
}
|
||||
for k, v := range uciNames() { // UCI wins: on OpenWrt it is the live configuration
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// confNames reads /etc/wireguard/*.conf. In a wg-quick file a peer is named by the comment
|
||||
// that introduces it, which is the convention every tool that writes these files follows:
|
||||
//
|
||||
// # phone
|
||||
// [Peer]
|
||||
// PublicKey = …
|
||||
//
|
||||
// confDir is where wg-quick keeps its files; a variable so the test can point it elsewhere.
|
||||
var confDir = "/etc/wireguard"
|
||||
|
||||
func confNames() map[string]string {
|
||||
out := map[string]string{}
|
||||
files, _ := filepath.Glob(filepath.Join(confDir, "*.conf"))
|
||||
for _, path := range files {
|
||||
text, err := readMaybeRoot(path)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
sc := bufio.NewScanner(strings.NewReader(text))
|
||||
// last is the most recent comment line; when a [Peer] block opens, that comment becomes
|
||||
// its pending name. Keeping the two apart is what lets each block take the comment that
|
||||
// introduces it and ignore any comment written inside it.
|
||||
last, pending := "", ""
|
||||
inPeer := false
|
||||
for sc.Scan() {
|
||||
line := strings.TrimSpace(sc.Text())
|
||||
switch {
|
||||
case strings.EqualFold(line, "[Peer]"):
|
||||
inPeer, pending, last = true, last, ""
|
||||
case strings.HasPrefix(line, "["):
|
||||
inPeer, pending, last = false, "", ""
|
||||
case strings.HasPrefix(line, "#"):
|
||||
if c := strings.TrimSpace(strings.TrimLeft(line, "#")); c != "" {
|
||||
last = c
|
||||
}
|
||||
case inPeer && strings.HasPrefix(line, "PublicKey"):
|
||||
// The separator is the first '='; the key's own base64 padding is the rest of
|
||||
// the line, so everything after that first one is the key.
|
||||
if _, v, ok := strings.Cut(line, "="); ok && pending != "" {
|
||||
out[strings.TrimSpace(v)] = pending
|
||||
}
|
||||
pending = ""
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// readMaybeRoot reads a file, falling back to root when it is not ours to read — which
|
||||
// /etc/wireguard always is, since it holds private keys. Only the comments and public keys in
|
||||
// it are looked at; nothing else in the file is kept or returned.
|
||||
func readMaybeRoot(path string) (string, error) {
|
||||
b, err := os.ReadFile(path)
|
||||
if err == nil {
|
||||
return string(b), nil
|
||||
}
|
||||
if !os.IsPermission(err) || !priv.Available() {
|
||||
return "", err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
args := priv.Wrap("cat", path)
|
||||
out, cerr := exec.CommandContext(ctx, args[0], args[1:]...).Output()
|
||||
if cerr != nil {
|
||||
return "", cerr
|
||||
}
|
||||
return string(out), nil
|
||||
}
|
||||
|
||||
// uciNames reads OpenWrt's configuration: each peer is a section with a description and a
|
||||
// public key. `uci show` is used rather than the file, so includes and overrides are applied.
|
||||
func uciNames() map[string]string {
|
||||
out := map[string]string{}
|
||||
if _, err := exec.LookPath("uci"); err != nil {
|
||||
return out
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
b, err := exec.CommandContext(ctx, "uci", "-q", "show", "network").Output()
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
desc, key := map[string]string{}, map[string]string{}
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
lhs, rhs, ok := strings.Cut(line, "=")
|
||||
if !ok || !strings.HasPrefix(lhs, "network.") {
|
||||
continue
|
||||
}
|
||||
parts := strings.Split(strings.TrimPrefix(lhs, "network."), ".")
|
||||
if len(parts) != 2 {
|
||||
continue
|
||||
}
|
||||
section, option := parts[0], parts[1]
|
||||
v := strings.Trim(rhs, "'")
|
||||
switch option {
|
||||
case "description":
|
||||
desc[section] = v
|
||||
case "public_key":
|
||||
key[section] = v
|
||||
}
|
||||
}
|
||||
for section, k := range key {
|
||||
if d := desc[section]; d != "" {
|
||||
out[k] = d
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
package vpn
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Real output of `wg show all dump`, from the two machines here: gate, where devices dial in,
|
||||
// and hamtv, which dials out through Mullvad. Keys are the real public ones, which are public.
|
||||
const gateDump = "wg0\tPRIVATEKEYWOULDBEHERE=\t4VwqgzHyelmVmNY+EANRWQggSSxvYYguJQMvUty9Sns=\t51820\toff\n" +
|
||||
"wg0\tNeWxo9wfqgipBQLuP676WFcMlW5RDrOcanc9y8sT7kU=\t3KWsu0mdVb2l5B/j0qHE03yE9LXi3FJ1WAjWMuxaLAw=\t10.0.0.33:63573\t172.21.46.3/32\t1790565744\t134192164\t415977404\toff\n" +
|
||||
"wg0\twUnRD990c6aglU0TCREtmfuUTPzqSmHrd3qTORpRJBg=\t4Pd5u2t6U9EdHlFlneg+hMr0G2v+IvoE/EqGfFdDPI8=\t(none)\t172.21.46.2/32\t0\t0\t0\toff\n"
|
||||
|
||||
const hamtvDump = "wg0\tPRIVATEKEYWOULDBEHERE=\tF/rgcHi+ZiaWPoSQ9iPcldIUbbMotE1hwSB/74Y9Qi4=\t39281\toff\n" +
|
||||
"wg0\tzqsfGglzJPY657WMRxf/S4omG7+ZkSDIpDq+ggbc9yo=\t(none)\t23.234.72.2:51820\t0.0.0.0/0,::/0\t1790571246\t18206675520\t2687837240\t25\n"
|
||||
|
||||
func TestParseServerDump(t *testing.T) {
|
||||
ts := parseDump(gateDump)
|
||||
if len(ts) != 1 {
|
||||
t.Fatalf("got %d tunnels, want 1", len(ts))
|
||||
}
|
||||
w := ts[0]
|
||||
if w.Name != "wg0" || w.ListenPort != 51820 {
|
||||
t.Errorf("interface = %s:%d, want wg0:51820", w.Name, w.ListenPort)
|
||||
}
|
||||
if w.PublicKey != "4VwqgzHyelmVmNY+EANRWQggSSxvYYguJQMvUty9Sns=" {
|
||||
t.Errorf("public key = %q; the third column is the public one", w.PublicKey)
|
||||
}
|
||||
// The private key is column two of the interface line and must not appear anywhere.
|
||||
if w.PublicKey == "PRIVATEKEYWOULDBEHERE=" || w.FwMark == "PRIVATEKEYWOULDBEHERE=" {
|
||||
t.Error("the private key was kept")
|
||||
}
|
||||
if len(w.Peers) != 2 {
|
||||
t.Fatalf("got %d peers, want 2", len(w.Peers))
|
||||
}
|
||||
p := w.Peers[0]
|
||||
if p.PublicKey != "NeWxo9wfqgipBQLuP676WFcMlW5RDrOcanc9y8sT7kU=" || p.Endpoint != "10.0.0.33:63573" {
|
||||
t.Errorf("first peer = %+v", p)
|
||||
}
|
||||
if !p.PSK {
|
||||
t.Error("first peer has a pre-shared key and should say so")
|
||||
}
|
||||
if got := p.AllowedIPs; len(got) != 1 || got[0] != "172.21.46.3/32" {
|
||||
t.Errorf("allowed ips = %v", got)
|
||||
}
|
||||
if p.Rx != 134192164 || p.Tx != 415977404 {
|
||||
t.Errorf("counters = %d/%d", p.Rx, p.Tx)
|
||||
}
|
||||
if p.Handshake == nil || p.Handshake.Unix() != 1790565744 {
|
||||
t.Errorf("handshake = %v", p.Handshake)
|
||||
}
|
||||
// The second peer has never connected: no endpoint, no handshake, nothing moved.
|
||||
if q := w.Peers[1]; q.Endpoint != "" || q.Handshake != nil || q.Rx != 0 {
|
||||
t.Errorf("second peer = %+v, want one that has never been heard from", q)
|
||||
}
|
||||
if kind, exit := classify(w); kind != "server" || exit != "" {
|
||||
t.Errorf("classify = %q/%q, want server", kind, exit)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseClientDump(t *testing.T) {
|
||||
ts := parseDump(hamtvDump)
|
||||
if len(ts) != 1 || len(ts[0].Peers) != 1 {
|
||||
t.Fatalf("got %d tunnels", len(ts))
|
||||
}
|
||||
w := ts[0]
|
||||
p := w.Peers[0]
|
||||
if len(p.AllowedIPs) != 2 || p.AllowedIPs[0] != "0.0.0.0/0" {
|
||||
t.Errorf("allowed ips = %v, want everything", p.AllowedIPs)
|
||||
}
|
||||
if p.Keepalive != 25 {
|
||||
t.Errorf("keepalive = %d, want 25", p.Keepalive)
|
||||
}
|
||||
if p.PSK {
|
||||
t.Error("this peer has no pre-shared key")
|
||||
}
|
||||
// It routes everything and has an endpoint, so this machine goes out through it.
|
||||
kind, exit := classify(w)
|
||||
if kind != "client" || exit != "23.234.72.2:51820" {
|
||||
t.Errorf("classify = %q/%q, want client through the endpoint", kind, exit)
|
||||
}
|
||||
}
|
||||
|
||||
// A tunnel that is up but has no peers at all is neither, and must not be guessed at.
|
||||
func TestClassifyUndecided(t *testing.T) {
|
||||
if kind, _ := classify(Tunnel{Name: "wg0", ListenPort: 51820}); kind != "" {
|
||||
t.Errorf("classify with no peers = %q, want no opinion", kind)
|
||||
}
|
||||
}
|
||||
|
||||
// Peers are named by the comment above them in a wg-quick file. The public key's own base64
|
||||
// padding is an '=' too, which is the thing to get wrong.
|
||||
func TestConfNames(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
conf := `[Interface]
|
||||
Address = 172.21.46.1/24
|
||||
PrivateKey = SECRETSECRETSECRETSECRETSECRETSECRETSECRET=
|
||||
ListenPort = 51820
|
||||
|
||||
# phone
|
||||
[Peer]
|
||||
PublicKey = NeWxo9wfqgipBQLuP676WFcMlW5RDrOcanc9y8sT7kU=
|
||||
AllowedIPs = 172.21.46.3/32
|
||||
|
||||
# the laptop
|
||||
[Peer]
|
||||
PublicKey = wUnRD990c6aglU0TCREtmfuUTPzqSmHrd3qTORpRJBg=
|
||||
AllowedIPs = 172.21.46.2/32
|
||||
`
|
||||
if err := os.WriteFile(filepath.Join(dir, "wg0.conf"), []byte(conf), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := confDir
|
||||
confDir = dir
|
||||
defer func() { confDir = old }()
|
||||
|
||||
names := confNames()
|
||||
if got := names["NeWxo9wfqgipBQLuP676WFcMlW5RDrOcanc9y8sT7kU="]; got != "phone" {
|
||||
t.Errorf("first peer named %q, want phone", got)
|
||||
}
|
||||
if got := names["wUnRD990c6aglU0TCREtmfuUTPzqSmHrd3qTORpRJBg="]; got != "the laptop" {
|
||||
t.Errorf("second peer named %q, want the laptop", got)
|
||||
}
|
||||
for k := range names {
|
||||
if k == "SECRETSECRETSECRETSECRETSECRETSECRETSECRET=" {
|
||||
t.Error("the private key was read as if it were a peer")
|
||||
}
|
||||
}
|
||||
if len(names) != 2 {
|
||||
t.Errorf("%d names, want 2: %v", len(names), names)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlineIsRecentHandshake(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
hs time.Time
|
||||
want bool
|
||||
}{
|
||||
{"just now", time.Now().Add(-30 * time.Second), true},
|
||||
{"a while back", time.Now().Add(-10 * time.Minute), false},
|
||||
{"never", time.Time{}, false},
|
||||
} {
|
||||
got := !tc.hs.IsZero() && time.Since(tc.hs) < onlineWithin
|
||||
if got != tc.want {
|
||||
t.Errorf("%s: online = %v, want %v", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A peer that has never handshaken must carry no time at all, not the zero one. omitempty
|
||||
// does nothing for a time.Time, so this is the difference between the page saying "never
|
||||
// connected" and saying the device was last seen in the year one.
|
||||
func TestNeverConnectedHasNoTimestamp(t *testing.T) {
|
||||
w := parseDump(gateDump)[0]
|
||||
b, err := json.Marshal(w.Peers[1])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(string(b), "handshake") {
|
||||
t.Errorf("json = %s, want no handshake field at all", b)
|
||||
}
|
||||
if strings.Contains(string(b), "0001-01-01") {
|
||||
t.Errorf("json = %s, want no year-one timestamp", b)
|
||||
}
|
||||
// and the one that has connected still carries its time
|
||||
if b, _ := json.Marshal(w.Peers[0]); !strings.Contains(string(b), "handshake") {
|
||||
t.Errorf("json = %s, want the handshake of a peer that has one", b)
|
||||
}
|
||||
}
|
||||
|
||||
// A machine with wireguard-tools installed but no tunnel up produces an empty dump. That has
|
||||
// to marshal as [] and not as null: the page takes the length of it, and null blanks the
|
||||
// whole console rather than saying "no tunnels".
|
||||
func TestEmptyDumpIsAnEmptyList(t *testing.T) {
|
||||
for _, dump := range []string{"", "\n", " \n\n"} {
|
||||
got := parseDump(dump)
|
||||
if got == nil {
|
||||
t.Fatalf("parseDump(%q) = nil, want an empty list", dump)
|
||||
}
|
||||
b, err := json.Marshal(got)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(b) != "[]" {
|
||||
t.Errorf("parseDump(%q) marshals to %s, want []", dump, b)
|
||||
}
|
||||
}
|
||||
}
|
||||
Executable
+69
@@ -0,0 +1,69 @@
|
||||
#!/bin/sh
|
||||
# Publish what `make release` built as a release on Gitea, which is where every Ham Lab
|
||||
# looks for a newer build (Settings → Updates, and `update.url` in config.json).
|
||||
#
|
||||
# make release && scripts/gitea-release.sh [--force]
|
||||
#
|
||||
# The token needs write:repository and is read from $HAMLAB_PUBLISH_TOKEN, else from
|
||||
# ~/.config/hamlab-publish-token. It is never printed.
|
||||
#
|
||||
# A release is a deliberate act, not part of `make deploy`: deploying pushes this build to
|
||||
# the machines this hub owns, while publishing offers it to every install everywhere.
|
||||
set -eu
|
||||
|
||||
REPO=${HAMLAB_REPO:-https://git.bigham.dev/api/v1/repos/mitchell/hamlab}
|
||||
DIST=${DIST:-dist}
|
||||
FORCE=0
|
||||
[ "${1:-}" != "--force" ] || FORCE=1
|
||||
|
||||
die() { printf '%s\n' "$*" >&2; exit 1; }
|
||||
|
||||
[ -f "$DIST/VERSION" ] || die "no $DIST/VERSION — run: make release"
|
||||
VERSION=$(head -n 1 "$DIST/VERSION" | tr -cd 'A-Za-z0-9._+-')
|
||||
[ -n "$VERSION" ] || die "$DIST/VERSION is empty"
|
||||
|
||||
TOKEN=${HAMLAB_PUBLISH_TOKEN:-}
|
||||
if [ -z "$TOKEN" ] && [ -r "$HOME/.config/hamlab-publish-token" ]; then
|
||||
TOKEN=$(cat "$HOME/.config/hamlab-publish-token")
|
||||
fi
|
||||
[ -n "$TOKEN" ] || die "no publish token: set HAMLAB_PUBLISH_TOKEN or write ~/.config/hamlab-publish-token (Gitea → Settings → Applications, scope write:repository)"
|
||||
|
||||
api() { # method path [curl args...]
|
||||
m=$1; p=$2; shift 2
|
||||
curl -sS -X "$m" -H "Authorization: token $TOKEN" "$REPO$p" "$@"
|
||||
}
|
||||
|
||||
SHA=$(git rev-parse HEAD)
|
||||
echo "publishing $VERSION ($(echo "$SHA" | cut -c1-8)) to $REPO"
|
||||
|
||||
# An existing release for this version is replaced only when asked: re-publishing a version
|
||||
# with different binaries is how an install ends up with something nobody can reproduce.
|
||||
existing=$(api GET "/releases/tags/$VERSION" | python3 -c 'import json,sys
|
||||
try: d = json.load(sys.stdin)
|
||||
except Exception: d = {}
|
||||
print(d.get("id") or "")' 2>/dev/null || true)
|
||||
if [ -n "$existing" ]; then
|
||||
[ "$FORCE" = 1 ] || die "$VERSION is already published (release $existing). Re-publish with --force."
|
||||
echo " replacing release $existing"
|
||||
api DELETE "/releases/$existing" -o /dev/null
|
||||
fi
|
||||
|
||||
notes=$(git log -1 --pretty=%B | head -20)
|
||||
payload=$(python3 -c 'import json,sys; print(json.dumps({"tag_name": sys.argv[1], "target": sys.argv[2], "name": sys.argv[1], "body": sys.argv[3], "draft": False, "prerelease": False}))' "$VERSION" "$SHA" "$notes")
|
||||
id=$(printf '%s' "$payload" | api POST /releases -H 'Content-Type: application/json' -d @- | python3 -c 'import json,sys
|
||||
d = json.load(sys.stdin)
|
||||
if not d.get("id"): sys.exit("gitea refused the release: %s" % d.get("message", d))
|
||||
print(d["id"])')
|
||||
[ -n "$id" ] || die "could not create the release"
|
||||
echo " release $id created"
|
||||
|
||||
n=0
|
||||
for f in "$DIST"/hamlab-linux-* "$DIST/SHA256SUMS" "$DIST/install.sh" "$DIST/VERSION"; do
|
||||
[ -f "$f" ] || continue
|
||||
name=$(basename "$f")
|
||||
api POST "/releases/$id/assets?name=$name" -F "attachment=@$f" -o /dev/null
|
||||
n=$((n + 1))
|
||||
echo " + $name"
|
||||
done
|
||||
echo "published $VERSION with $n assets"
|
||||
echo " every Ham Lab pointed at this repo will offer it in Settings → Updates"
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user