mitchell/hamlab
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 14 Wiki Activity

Compare commits

base: mitchell/hamlab:v0.1.0
Branches Tags
mitchell/hamlab:main mitchell/hamlab:virtualization
mitchell/hamlab:v0.1.0-137-gc129f65 mitchell/hamlab:v0.1.0-136-gd8bf5df mitchell/hamlab:v0.1.0-134-gf3441ae mitchell/hamlab:v0.1.0-132-g823c721 mitchell/hamlab:v0.1.0-127-gdaf33a4 mitchell/hamlab:v0.1.0-125-g09b7558 mitchell/hamlab:v0.1.0-123-g7d0ec5e mitchell/hamlab:v0.1.0-118-g2e4f3a7 mitchell/hamlab:v0.1.0-116-gbf0bdd5 mitchell/hamlab:v0.1.0-115-g9bf60de mitchell/hamlab:v0.1.0-114-g1eb59fd mitchell/hamlab:v0.1.0-112-g363ff02 mitchell/hamlab:v0.1.0-111-gb5454c1 mitchell/hamlab:v0.1.0-110-g1599ac6 mitchell/hamlab:v0.1.0
..
compare: mitchell/hamlab:0e8271907089f845c3d418aa32742d3ff9912f78
Branches Tags
mitchell/hamlab:main mitchell/hamlab:virtualization
mitchell/hamlab:v0.1.0-137-gc129f65 mitchell/hamlab:v0.1.0-136-gd8bf5df mitchell/hamlab:v0.1.0-134-gf3441ae mitchell/hamlab:v0.1.0-132-g823c721 mitchell/hamlab:v0.1.0-127-gdaf33a4 mitchell/hamlab:v0.1.0-125-g09b7558 mitchell/hamlab:v0.1.0-123-g7d0ec5e mitchell/hamlab:v0.1.0-118-g2e4f3a7 mitchell/hamlab:v0.1.0-116-gbf0bdd5 mitchell/hamlab:v0.1.0-115-g9bf60de mitchell/hamlab:v0.1.0-114-g1eb59fd mitchell/hamlab:v0.1.0-112-g363ff02 mitchell/hamlab:v0.1.0-111-gb5454c1 mitchell/hamlab:v0.1.0-110-g1599ac6 mitchell/hamlab:v0.1.0
163 Commits
v0.1.0 .. 0e82719070
Author SHA1 Message Date
Mitchell-JohnsonandClaude Opus 5 0e82719070 plan: the VPN tab edits devices, and what that cost in care 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 23:22:13 -07:00
Mitchell-JohnsonandClaude Opus 5 8fb21ee95e Create, rename and remove the devices that dial in 
On a tunnel this machine configures — today that means OpenWrt's UCI, which is
where the server here lives — the VPN tab can add a device, rename it and take
it away. A wg-quick host is reported as unmanaged and shows no buttons, because
half-supporting a second configuration format is worse than an honest refusal.

Adding generates the keypair and a pre-shared key on the machine, picks the
lowest free address in the tunnel's range, and hands back the complete client
profile with a QR code for the WireGuard apps. The private key in it is
generated for that one call and stored nowhere, so the profile is shown until
dismissed and then gone — the same bargain as any password this console makes.

Nothing restarts the network, which matters when the interface belongs to the
router the house is behind. `uci commit` writes the file and applies nothing;
`wg set` changes one peer on the live interface and touches nothing else.

The order of the two is chosen so they cannot disagree quietly. Adding writes
the configuration first and rolls it back if the live tunnel refuses, because a
device that works only after the next reboot is the worst of the three outcomes.
Removing does the live tunnel first, since that is the moment access actually
ends, and says plainly if the configuration half then fails.

Removing asks for the device's name to be typed back, at both ends. It is the
one action here that cannot be undone and it sits one tap from a list of devices
that all look alike.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 23:20:41 -07:00
Mitchell-JohnsonandClaude Opus 5 32fdbf169a Fix the blank VPN tab, and stop one page taking the console with it 
On a machine with wireguard-tools installed but no tunnel up — core, which is
the machine the console opens on — the tunnel list was a nil slice, which is
null in JSON, and the page took its length. That throws, React unmounts the
whole tree, and what is left is a blank window with nothing to go on.

Two fixes, because either alone leaves the failure possible. The parse returns
an empty list rather than nil, with a test that pins the marshalled shape; and
the page treats a missing list as an empty one, so a node still running an older
build cannot blank the console either.

And the reason a small mistake had such a large blast radius: nothing caught it.
Each page now renders inside a boundary, so a fault costs that page and says
what it was, while the rail, the switcher and every other tab carry on. The
terminals are deliberately left outside it — they hold live WebSockets, and
remounting them to recover from somebody else's error would drop the sessions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 22:18:59 -07:00
Mitchell-JohnsonandClaude Opus 5 acb83d3a4a A peer that never connected has no timestamp, and names are readable 
Two faults in the VPN tab, both found by running it against the real machines.

omitempty does nothing for a zero time.Time, so a peer that has never handshaken
was serialised as the year one and the page would have shown mbp14 as last seen
on 1 January 0001 instead of "never connected". The field is a pointer now, and
a test pins the JSON.

And peer names on a wg-quick host were never going to appear: /etc/wireguard is
root-only because it holds private keys, and the file was being opened as the
daemon's own user. It now falls back to reading it as root, the same way the
kernel state beside it is already read. Only comments and public keys are taken
from it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 22:01:13 -07:00
Mitchell-JohnsonandClaude Opus 5 6f634bbe11 A VPN tab: the tunnels, and who is on them 
Beside Network, per machine. Every WireGuard interface on the machine in the
switcher, with its address, listen port and public key, and under it the peers:
name, allowed addresses, last handshake, where they connect from, and what they
have moved.

Per machine is the point rather than a limitation. The tunnel devices dial into
and a tunnel a machine dials out through are different boxes here, and putting
them on one page would invite the question of which box you were looking at.

The kernel is the truth and the configuration only supplies names: wg show all
dump has the handshakes and the counters and knows nothing but public keys, so
the # comment above a peer in a wg-quick file, or a UCI section's description on
OpenWrt, is read alongside purely to label them. Whether a tunnel is dialled or
dialling is worked out from the shape of its peers, and says neither when it
cannot tell.

Online means a handshake in the last three minutes, and the tab says so in as
many words: WireGuard is silent when idle, so there is no connection to be in.

Read-only, and no secrets leave: the private key is the second column of the
dump and is read past, pre-shared keys are noted as present and never read.

The Network tab keeps listing tunnels as the interfaces they are, and hands off
to here for anything about them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 21:59:30 -07:00
Mitchell-JohnsonandClaude Opus 5 9c7848907b Network sits under Files by default 
Where Mitchell had already dragged it. An account with its own rail order is
unaffected either way; this is the slot a fresh one gets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 20:20:49 -07:00
Mitchell-JohnsonandClaude Opus 5 48d3887514 A second download source, for when the first one refuses 
Cloudflare rate-limits by volume, and the block it handed out while this was
being built lasted over twenty minutes — too fragile to be the only thing behind
a button. So the download phase tries its sources in order: the anycast endpoint
first, then a plain 100 MB file on Linode's Seattle mirror, which has no opinion
about how much has been pulled today.

Only a refusal moves to the next one; anything else would fail the same way
twice. Every result names the source it used, and the page says "from
linode-seattle" rather than "via SEA" when it was the fallback — that mirror is
in one fixed place, so somewhere else it reads low, and a figure that quietly
came from a different continent would be a lie.

Upload stays on Cloudflare, which has never been the one to refuse.

Also: the too-short-window guard was refusing to divide 512 MiB by ten
milliseconds on a link fast enough to hit the runaway budget before the first
tick. It now guards against dividing by zero and nothing more.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 20:15:33 -07:00
Mitchell-JohnsonandClaude Opus 5 2f59a54678 A Network tab, with an internet speed test 
Per machine like every other tab: the interfaces as the kernel has them
(addresses, MAC, MTU, negotiated link speed, carrier, totals, live rates from
the sampler's own sensors), the default route and resolvers, what the internet
sees of this connection, and a speed test with the last fifty results kept.

The test runs in the daemon, not the browser, so the figure is the line of the
machine in the switcher — the router's, if you switch to gate — and it carries
on when the phone locks. It measures against Cloudflare's public speed
endpoints, so there is nothing to install on any host, which is what makes it
work the same on Debian and on OpenWrt where flash is scarce.

What the measurement does and does not count:

  - Several streams, HTTP/2 deliberately off. One stream over one connection
    measures a congestion window, and HTTP/2 would put every stream back onto
    one connection and measure the same thing.
  - The first second of each phase is thrown away as the ramp.
  - A phase ends on 64 MiB as well as on four seconds, so a fast line says what
    it is without moving a gigabyte. A minimum window keeps a very fast line
    from filling the cap before there is anything to divide by.
  - Latency is the round trip less the far end's own Server-Timing. A zero-byte
    response is mostly Cloudflare's worker thinking: left in, it reported 31 ms
    for a line the edge itself measures at 3.5. Jitter is the mean change
    between consecutive probes, so one slow probe cannot run away with it.
  - The far end rate-limits by volume. So: a 20 s cooldown counted from the last
    attempt rather than the last success, streams that retry twice, the rate
    computed over the window in which bytes were actually still arriving, and a
    plain sentence when it was refused. Coasting to the end of the clock with
    nothing arriving reported a third of the truth.

The last result is published as net.internet.* sensors, each stamped with when
the test ran, so the line can be a widget, a graph, an alert or a Prometheus
series like anything else. The public address is refreshed on its own quarter
hour, which is worth having here: it is a DHCP lease and it moves.

Tests are offline — the window, the byte cap, the refusal and the cancellation
run against a local server; the route parser's byte order is pinned by a case
that would have caught the bug it did. The live measurement is behind
HAMLAB_NET_TEST=1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 20:11:03 -07:00
Mitchell-JohnsonandClaude Opus 5 a0987c940f Twenty login tries before the backoff starts, not five 
A guest password is typed by hand off a message on a phone. Five tries is not
many when a hyphen lands in the wrong place, and being told to wait is worse
than the brute-force risk here: past twenty the doubling lockout still applies,
the global backstop is unchanged, and the password is ~78 bits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 18:04:23 -07:00
Mitchell-JohnsonandClaude Opus 5 ed64aca63b An eye on the login form, to see what is being typed 
A guest password is read off a message and typed by hand, and from the login
form a mistyped character and a wrong password look identical. Revealed, the
field is a text field, so it also turns off capitalisation and autocorrect,
which a phone keyboard would otherwise offer for it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 18:03:26 -07:00
Mitchell-JohnsonandClaude Opus 5 179f342a2c A guest login is a full account with a clock on it 
Mitchell, on the read-only version built earlier today: scratch it. He hands
these to people he trusts and revokes them otherwise, so a console with half
its tabs missing solved a problem he doesn't have.

So the allow-list, the middleware check and every "not for a guest" branch in
the UI are gone. What is left is the part that was the point: a throwaway
account that expires by itself, is checked on every request, swept once a
minute, and can be ended in one tap. A guest now gets the whole console on
every machine, terminal included.

Said once, since it is his call and he has made it: that means a guest has a
root-capable shell on every host, and can make an account or a token of its own
that outlives the guest login.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 17:53:55 -07:00
Mitchell-JohnsonandClaude Opus 5 8a7167bf03 A guest login: temporary, look-only, revocable 
Users -> Guest logins makes a throwaway user name and password, good for half
an hour by default, to hand to somebody who wants to see the console. It is a
normal account with `guest: true` and an expiry, so login, sessions and the
saved look need no special case, and it is left out of the account list, out of
"the last account can't be removed", and out of the one account an empty user
name means.

What a guest may do is one allow-list, checked in the auth middleware, rather
than a flag ~190 handlers have to remember. The list is matched by asking the
router which handler a request would reach, so it cannot drift from the routes
it names, and it is checked against the scoped path — /api/nodes/grid/procs is
/api/procs on grid — because the hub proxies to a node with a token that is
full control of that node. The node cannot tell a guest from its owner, so the
only place it can be stopped is here, before the request leaves.

A guest gets the dashboard, containers, stacks, disks, hardware, ZFS and usage,
on this machine and on any managed one, and nothing else: no terminal (attaching
is a GET, and it is a root shell), no files, no container logs or environments,
no stack files, no alert channels, no cron, no tokens, no settings, no power.

Expiry is enforced on every request and swept once a minute, so a console left
open on a visitor's phone stops within the minute rather than at their next
click; revoking signs them out at once. The console hides what a guest cannot
use and says how long is left where the power button would be.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-27 17:23:16 -07:00
Mitchell-JohnsonandClaude Opus 5 4450f1285d Install under procd, so OpenWrt counts as a host too 
The binary was already static and musl-clean; the only thing standing between
Ham Lab and an OpenWrt box was that setup.go knew systemd and OpenRC and
nothing else, so an install there ended at "nothing starts it at boot."

Adds an embedded procd init script, detected on /sbin/procd + /etc/rc.common
ahead of the OpenRC branch. Both backends write /etc/init.d/hamlab, so
uninstall tells them apart by USE_PROCD=1 rather than by path.

Verified on the new router: procd supervises the instance, S95/K10 symlinks
land, and the hub drives its Docker and stacks like any other node. The
Updates tab needed nothing — apk-tools 3 answers the same query Alpine's does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-26 17:39:04 -07:00
Mitchell-JohnsonandClaude Opus 5 4e3c817b2a A Recreate action, and the Updates tab uses the whole window 
Recreate is `up -d --force-recreate`. It exists because Deploy and Update
both leave a container alone when its definition has not changed, which is
precisely the state a container is in when Docker will not start it — pinned
to a network that has since been recreated, answering "network … not found"
to everything. siyuan landed there today and there was no way out from the
console.

The Updates tab is a page, not a list with a detail pane, so on a wide window
it was being squeezed into the right-hand half with an empty list column
taking the left — which read as a phone layout on a desktop. It now sits
under the tab strip at full width. The list-and-detail tabs are unchanged.
The rule needed :not(.has-sel) to outrank the wide-window rule that grows the
list to fill the row.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-26 13:14:18 -07:00
Mitchell-JohnsonandClaude Opus 5 94800b8207 Update from the list that says what is out of date 
The Updates tab could tell you eleven images were behind and offer nothing to
do about it; updating meant knowing which stack each image belonged to and
going to find it. Now each row has Update and the header has Update all.

Updating is per stack, because that is the unit compose deals in, so a row
acts on whichever stacks use that image — postgres:15-alpine is wikijs's,
postgres:17 is vikunja's — and Update all walks the affected stacks through a
queue one at a time, with each task's output on screen, rather than
recreating half the machine at once. A stack that finishes says "updated just
now" rather than re-checking, since every check is a registry request and
Docker Hub counts them.

Also: cron expressions read as English. "daily at 3:15am", with 15 3 * * *
kept beside it. Jobs.tsx had a smaller version of this for its own list; it
now uses the shared one, which handles lists, ranges, steps, names,
weekdays/weekends, midnight and noon, and says nothing at all when it is not
certain — the caller falls back to the expression.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-26 12:49:32 -07:00
Mitchell-JohnsonandClaude Opus 5 03e0761f3e Hardware: a Motherboard card 
The board, the chassis and the firmware on it, which the System card was
reducing to two lines. Maker, model, version, serial and asset tag; chassis
and where the board sits in it; the firmware's vendor, version, revision,
date, ROM size and whether the machine boots UEFI — that last one is a line
in SMBIOS's characteristics list rather than a field, so the parser now keeps
those lists too.

Expansion slots come with it, with the device found at each occupied slot's
PCI address rather than the address alone. They are only shown when the
firmware tracks occupancy at all: core's NucBox lists five PCIe x1 slots it
does not physically have, all "Available", copied out of a reference design.
grid's board reports real addresses and real occupancy, which is the case
worth printing.

A machine with no SMBIOS gets a card that says so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-25 22:21:27 -07:00
Mitchell-JohnsonandClaude Opus 5 38039d1adc Hardware sits under KVM in the rail 
Mitchell's placement. The rail order is the account's anyway — this is only
where a page starts for someone who has never dragged it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-25 22:02:56 -07:00
Mitchell-JohnsonandClaude Opus 5 776bcb236d A Hardware tab: what the machine is made of 
Per machine, like every other page. Four tiles — processor, memory, storage,
graphics — over a card for each.

Memory is the part that had nowhere to live before: every slot, with the
module's size, type, form factor, rated and running speed, maker, part
number, serial, rank and voltage, and the empty slots marked empty. A module
whose maker is only a JEDEC id gets a name when it is one we know and keeps
the number when it is not, because a wrong name is worse than a code.

Everything is read from what Debian already has: dmidecode for the parts only
the firmware knows, lscpu, lspci, sysfs. None of it changes under a running
kernel, so it is read once and kept for ten minutes; Re-read is for the one
thing that does change, a drive plugged in.

Every reader is allowed to come back empty. A machine with no dmidecode, no
root to read it with, or no PCI bus at all still gets a page, with a note
saying what was missing and why. Checked against the Mac mini, which is all
three cases at once: no SMBIOS memory table, a GPU only the device tree knows
about, and cores that are not all alike — lscpu counts one cluster as "cores
per socket" there, so without SMT the thread count is the honest answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-25 21:41:40 -07:00
Mitchell-JohnsonandClaude Opus 5 09dc846730 docker: logs follow a held connection, and networks can be removed 
Two gaps the Docker tab had against Dockhand and against its own Services
tab next door.

Logs: one `docker logs -f` per viewer over a WebSocket, the shape the
Services tab's journal tail already uses, instead of asking again every two
seconds. stdout and stderr both, with stderr marked so the page can colour
what went wrong. Docker replays the backlog down two pipes in no particular
order, so the opening 300 ms is held and sorted by the stamp Docker wrote;
after that the log is live and in order. A reconnect resumes from the last
line rather than from the top — `--since` is inclusive and two lines can
share a nanosecond, so the client drops what it has already shown by line,
not by "later than". A container that has stopped ends the stream at once,
so reconnects back off to 15 s.

Networks: remove and prune, which images and volumes have had all along.
The overview now says which containers are attached to each network, so the
✕ appears only on one nothing uses, and never on bridge, host or none.
`network rm` prints the real reason and then an aggregate "exit status 1",
so the message shown is the first line, not the last.

Prune is not exercised against `core`: several networks here belong to
stacks that are merely down, and pruning would give them new subnets on
their next deploy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-25 21:17:37 -07:00
Mitchell-JohnsonandClaude Opus 5 b8c654b61b docker: drop the fleet-wide updates list 
A view shows one machine: the one in the switcher. Seeing another machine's
state means switching to it, not a hub-side page that reaches across — and
this was one. Recorded as a decision so it does not get rebuilt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-25 00:19:12 -07:00
Mitchell-JohnsonandClaude Opus 5 608f0b7005 docker: stack Inspect one property per row 
Inspect's values are long paths, network names and env lists. In a two-column
grid on one line each they were half truncated and read as columns of
unrelated things. One per row, label above value, wrapping instead of an
ellipsis. Mounts are one block rather than a repeated "Mount" label.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-25 00:17:09 -07:00
Mitchell-JohnsonandClaude Opus 5 639807ff8a plan: the Docker tab's Dockhand gap is closed; the cutover is homelab work 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 20:21:06 -07:00
Mitchell-JohnsonandClaude Opus 5 f17a365ea1 stacks: remember where a container's bind mounts pointed 
Every stack on core and grid is labelled with a path inside Dockhand's
filesystem — /app/data/stacks/… — because Dockhand ran compose from inside its
own container. Ham Lab resolves those by walking the bind mounts of running
containers, which works right up until the container doing the mounting is
removed. Retiring Dockhand would, in one step, make seventeen stacks on core
and five on grid unfindable.

So the mounts are written down as they are seen, and a path no live container
explains is looked for in what was seen before. A remembered mount resolves
with no container to run in, which is the right answer: compose runs on the
host, against the host's own copy of the file.

The same problem bites a scheduled update, which captures how to invoke
compose when it is made so the cron job never has to rediscover the world at
3am. repoint() checks the named container is still running and falls back the
same way if it is not — and pointedly leaves the spec alone if nothing
explains the path, so it fails loudly rather than deploying a guess.

Out in cron the record is read and never added to: the daemon owns state.json
and would clobber anything written from there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 20:19:54 -07:00
Mitchell-JohnsonandClaude Opus 5 192504ead3 docker: the eight things Dockhand did that Ham Lab did not 
Dockhand is being retired, so the Docker tab has to cover what it actually
earned its keep doing. Eight pieces, all per container or per machine:

- Stats: CPU, memory and IO over time, from sensors already collected.
- Health: the container's own healthcheck, with the last few probe outputs —
  which is where a failing check says why.
- Events: `docker events` keeps a minute of backlog and nothing more, so Ham
  Lab follows the stream and keeps a ring of its own. A restart at 4am is
  still there at 9.
- Shell: `docker exec` into a container, in the terminal already here.
- Limits: memory, CPU and restart policy changed live, without hand-editing
  the compose file for a number you want to try.
- Updates: what the registry has that this machine does not, for every image
  at once rather than one stack at a time.
- The same across every machine, asked of the hub — which is the only thing
  that can reach them all.
- Schedules: a real line in /etc/cron.d/hamlab-docker calling the binary,
  in the same shape as the ZFS snapshot schedules. Dockhand updated DNS and
  ingress nightly from a ticker inside a web app; a cron line keeps running
  when the console does not, and is visible to anyone reading the machine the
  normal way.

internal/compose is new and shared, so a deploy from the console and a deploy
from cron invoke compose identically — including the case where compose has to
run inside the orchestrator's own container, where the paths are that
container's and not the host's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 20:11:36 -07:00
Mitchell-JohnsonandClaude Opus 5 a70f3e2ed2 Container events are Ham Lab's own record, not Docker's short memory 
docker events is a live stream with a small in-memory backlog: ask for thirty days and you
get a few minutes, and a daemon restart loses the lot. That is why Dockhand kept its own
copy. A watcher now follows the stream and appends lifecycle events to a ring beside the
config, flushed every thirty seconds, so the question of what happened to a container last
week has an answer. Health-check exec events are excluded — three per probe, thousands a
day, and the health panel says it better.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 19:55:36 -07:00
Mitchell-JohnsonandClaude Opus 5 b3e1c495f0 Docker: a container's stats, health, events, shell and limits 
Five things the tab could not do, all on the container detail.

Stats draws CPU and memory over time from the container sensors the sampler already keeps,
so there is nothing new to collect.

Health shows the check's verdict, the failing streak and what the last five probes actually
printed, and says plainly when an image defines no health check at all rather than leaving
a blank panel.

Events asks Docker what has happened to the container over the last day, week or month —
started, died, killed, health going bad, with exit codes — rather than keeping a copy that
would only go stale.

Shell opens a session inside the container as an ordinary terminal, so it reattaches,
resizes and scrolls back like the rest and works on a managed machine through the same
proxy. The terminal manager grew a CreateCmd for it; every session is the same thing
afterwards.

Limits changes memory, CPUs and restart policy on the running container with docker update,
live and without recreating it. That is also the catch, and the dialog says so: it does not
edit the compose file, so a deploy puts back whatever the file says.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 19:52:00 -07:00
Mitchell-JohnsonandClaude Opus 5 c129f655a7 Reorder the default rail: the machine, then its storage, then what runs on it 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 19:12:14 -07:00
Mitchell-JohnsonandClaude Opus 5 d8bf5df89f Fullscreen on Safari, and a fallback where there is none 
Safari on a Mac wants the webkit prefix; Safari on iOS has no fullscreen for an ordinary
element at all, only for a <video>, so requestFullscreen was simply undefined and the tab
threw. It now uses whichever spelling the browser has, and where there is none the frame
fills the viewport in-page instead — the same result minus the browser's own chrome — with
a Done button and Escape to come back. The button reads Expand rather than Fullscreen when
that is what it will do.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 17:51:41 -07:00
Mitchell-JohnsonandClaude Opus 5 e64b0a1727 PLAN: the KVM tab is per machine 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 17:01:05 -07:00
Mitchell-JohnsonandClaude Opus 5 f3441ae718 The KVM tab is per machine, and follows the machine switcher 
It was a hub-side list you pointed at machines, which had it backwards. A machine has one
KVM; switch to grid and you get grid's. The address is still kept on the hub rather than on
the machine, because a KVM earns its keep exactly when that machine is down and cannot
answer for itself — so the tab reaches the device either way. Adding a second KVM for a
machine replaces the first rather than sitting beside it.

The tab is no longer hub-only and now sits beside Machines in the rail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 16:58:08 -07:00
Mitchell-JohnsonandClaude Opus 5 70fcf86283 PLAN: the KVM tab shipped 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 16:43:44 -07:00
Mitchell-JohnsonandClaude Opus 5 823c72122c A proxied KVM may be framed by the console that proxies it 
frame-ancestors 'none' forbids framing even from the same origin, so /kvm/ paths say
'self' instead. Without this the iframe was refused before it loaded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 16:42:51 -07:00
Mitchell-JohnsonandClaude Opus 5 fd88c4ff78 A KVM tab: paste an address, get the device's own page 
Ham Lab proxies each KVM from its own origin, which is what makes the simple version
possible. A KVM that speaks only plain HTTP would otherwise be blocked outright as mixed
content in a TLS console; served from here it loads, it is behind this login rather than a
second one, and the iframe is same-origin.

Nothing in it knows anything about any particular device. The prefix /kvm/<id>/ is only the
entry point — a device UI asks for its assets by absolute path, which no prefix can rewrite
away, so any request Ham Lab does not serve itself is routed back to whichever KVM the
asking page belongs to, worked out from the Referer and from a cookie when there is no
Referer, as on a WebSocket upgrade. Ham Lab's own routes always match first, so a device can
never shadow the console, and the fallback is gated on a live session.

The list is hub-side on purpose: a KVM earns its keep when the machine is down, which is
when that machine's own Ham Lab cannot answer.

On the keyboard, a page cannot forward keys into an iframe and does not need to — one with
focus receives them directly. Fullscreen plus navigator.keyboard.lock() is what takes
Escape, Tab and Ctrl-W back from the browser; that is Chromium-only and the tab says so
rather than pretending otherwise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 16:39:27 -07:00
Mitchell-JohnsonandClaude Opus 5 e6625a540c PLAN: the Docker tab's polish and retiring Dockhand are one item 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 16:15:00 -07:00
Mitchell-JohnsonandClaude Opus 5 66c1ed7d13 PLAN: the Files polish and retiring copyparty are one item 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 16:13:45 -07:00
Mitchell-JohnsonandClaude Opus 5 384aea79ca PLAN: the snapshot explorer, and the preview is gone 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 13:17:25 -07:00
Mitchell-JohnsonandClaude Opus 5 daf33a44b2 The datasets table is a snapshot explorer, and snapshots can be deleted 
Renamed Datasets to Snapshots and turned the columns around to match: how many each
dataset has and what they cost come first, the dataset's own size after. Opening one lists
its snapshots with when it was taken, when it expires — read from the TTL in its own name,
so no server call is needed — its size, and a Delete button. Deleting takes the same-named
snapshot on descendant datasets too, which is how a recursive schedule made it, and says so
before it does anything.

Dropped "What would it delete?" from the schedule cards, and the endpoint behind it. Run
now says plainly that expired snapshots go at the same time, which is the whole of what the
preview was telling anyone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 13:16:13 -07:00
Mitchell-JohnsonandClaude Opus 5 c3854bafda PLAN: schedules are cron jobs, and zfSnap is retired 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 13:12:17 -07:00
Mitchell-JohnsonandClaude Opus 5 09b7558d44 Snapshot schedules are real cron jobs, written and owned by Ham Lab 
A schedule is now a line in /etc/cron.d/hamlab-zfs, which Ham Lab owns and rewrites whole
whenever the schedules change; the line calls `hamlab zfs-snapshot <id>` as the account Ham
Lab runs as, reaching zfs through sudo exactly as the daemon does. The daemon's own ticker
is gone. Snapshots are backup-shaped and should not stop because a web console is down,
being upgraded, or uninstalled — and a cron line is greppable by anyone looking at the
machine the normal way, which a schedule hidden inside a daemon is not.

A cron expression is now the single description of when a schedule fires: the four common
shapes render to one, a custom schedule states it outright, and the next run is read back
with the same evaluator the Jobs tab uses on everyone else's lines. So the page, the file
and the prediction cannot disagree.

Schedules gain a custom cron expression and an optional snapshot name prefix. What a run
did moves out of the policy into zfs-runs/<id>.json beside the config, because the cron job
is a separate process and must never write state.json behind the daemon's back.

The detected external job can be adopted: its schedule, retention and dataset are read out
of the cron line into a new Ham Lab schedule, ready to check rather than retype.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 13:09:43 -07:00
Mitchell-JohnsonandClaude Opus 5 4dc25fdd13 PLAN: Usage, Jobs and the ZFS tab are done 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 13:03:38 -07:00
Mitchell-JohnsonandClaude Opus 5 7d0ec5e018 ZFS gets its own tab, with scheduled snapshots and a retention period 
Pools, datasets and snapshots move out of Disks — Disks is the hardware, this is the
filesystem on top of it. New: snapshot schedules, hourly to monthly, with a retention
period and a floor that keeps the newest few whatever their age.

The naming is zfSnap's, deliberately. grid has run zfSnap from root cron since before this
repo existed and it works, so rather than invent a second convention this writes and reads
exactly what zfSnap does: 2026-09-24_02.30.01--6w, with the retention in the name. Ham Lab
can therefore show and expire the snapshots zfSnap already took, the two cannot disagree
about what a snapshot is, and changing a schedule never re-dates old ones — each carries
the rule it was taken under. An existing cron job is detected and shown on the page, with a
warning if a Ham Lab schedule is running as well.

Nothing is destroyed unseen. Every schedule answers what it would delete, with the list,
before Run now touches anything, and only names that parse as a stamp and a TTL are ever
candidates. A snapshot taken by hand is invisible to it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 13:01:53 -07:00
Mitchell-JohnsonandClaude Opus 5 a07a444170 The Space tab is Usage; the Cron tab is Jobs 
Both are named for what they do rather than the tool underneath, the way Services is.
Old links and saved rail orders still land in the right place. The Go packages and their
endpoints keep their names, with a pointer in each to what the UI calls them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 12:54:23 -07:00
Mitchell-JohnsonandClaude Opus 5 dfa9e88f5b PLAN: the KVM tab, as simply as it can actually be built 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 12:41:58 -07:00
Mitchell-JohnsonandClaude Opus 5 260d5d4880 PLAN: the KVM is a Sipeed NanoKVM at 10.0.0.217 — framing is fine, HTTP is the blocker 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 12:39:51 -07:00
Mitchell-JohnsonandClaude Opus 5 72c7f45e19 PLAN: to do — a KVM tab, the keyboard-video-mouse kind 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 12:37:25 -07:00
Mitchell-JohnsonandClaude Opus 5 2e4f3a7fe6 A drive in a pool says which pool, not that nothing is on it 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 12:11:39 -07:00
Mitchell-JohnsonandClaude Opus 5 e0625be684 The disks box is drives; the net box lets you pick the interface 
Disks: one row per physical drive with its model, size, temperature and IO, and a usage
bar built from the filesystems the sampler can attribute to it. Attribution is new — a
mount's source device is resolved through sysfs to the drive behind it, following one
layer of device-mapper or md and folding NVMe namespaces the way the sampler does. A pool
member gets no usage bar, because one drive's share of a raidz pool is not a real number.

Net: every interface that is up is listed and tappable, defaulting to the one that has
actually carried traffic rather than whichever the kernel listed first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-24 12:09:39 -07:00
Mitchell-JohnsonandClaude Opus 5 bf0bdd54af The disks box is this machine's own storage, not a network share 
core mounts grid's tank over CIFS, and it was listed beside root and efi as though it
were a disk in the box. Filesystems whose storage is on another machine are now marked
remote — a label on the fs sensors, so the sensor stays for anyone who wants it — and the
Processes tab's disks box leaves them out.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-23 23:16:17 -07:00
Mitchell-JohnsonandClaude Opus 5 9bf60deffc Retire the Dashboard tab; Processes is the front page 
The app repeated itself. The module list, /api/prefs/dashboard, applyPrefs and the
per-user dashboard blob are gone. Widgets stays and now says plainly that it is for
dashboards outside Ham Lab.

This also closes a bug: applyPrefs set the sampling rate from intervalMs inside the
dashboard layout, and reset it to the default when that field was absent — so with
per-user layouts, one account saving a dashboard could reset the sampling rate for the
whole machine. The sampler now has one source.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-23 23:03:49 -07:00
Mitchell-JohnsonandClaude Opus 5 1eb59fd08c One tab called Processes, showing what Top showed 
The two read the same data and answered the same question. Procs.tsx is gone; a rail
order saved before this keeps the merged page in Top's slot. The mouse back-button item
is dropped from the plan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-23 22:58:06 -07:00
Mitchell-JohnsonandClaude Opus 5 e184b4a149 PLAN: seven more to do — tab overlap, two renames, ZFS scheduling, virtualization, Files polish 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-23 21:38:19 -07:00
Mitchell-JohnsonandClaude Opus 5 363ff02bda Drop the external demo: the Widgets tab is the demo 
The hamlab-demo site, its Widgets-tab button, the demo_url config key and the make publish
target that fed it are all gone, along with the public read-only token it needed. The
example stays in examples/external-demo as something to read. The installer one-liner
already comes from Gitea releases.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-23 20:17:22 -07:00
Mitchell-JohnsonandClaude Opus 5 b5454c1022 Public downloads: the installer and the updater come from Gitea releases, no token 
REQUIRE_SIGNIN_VIEW is off and mitchell/hamlab is public, so anyone can clone it or pull
a release. The other five repos are individually private and 404 to anonymous. Gitea
resolves releases/download/latest/<asset>, so the install one-liner and HAMLAB_URL point
at the newest release and never need changing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-23 18:59:42 -07:00
Mitchell-JohnsonandClaude Opus 5 1599ac65ef PLAN: the updater shipped; what is left is where a public download lives 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-23 14:33:20 -07:00
Mitchell-JohnsonandClaude Opus 5 68bf3bb218 An updater: what is published on Gitea, and installing it from Settings 
A new selfupdate package asks one URL what the newest build is — a Gitea/Forgejo repo
API base, or any directory holding VERSION, SHA256SUMS and hamlab-linux-<arch>, which is
what make release already writes and what install.sh already downloads. The binary is
checked against SHA256SUMS before it goes near the running one, and a token is sent when
the source is behind a login, so a private Gitea serves its own fleet without opening the
instance to everyone. Version ordering is install.sh's vnum rule, so the daemon and the
installer never disagree about which build is newer.

Settings gains an Updates card: installed, published, when it last looked, the source,
release notes, Check for updates, and Install. The rail badges Settings when a newer
build is out. Nothing installs unasked — it restarts the daemon and drops terminals.

scripts/gitea-release.sh (make gitea-release) publishes dist/ as a Gitea release.
Deliberate, and not part of deploy: deploy pushes to the machines this hub owns,
publishing offers the build to every install anywhere.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-23 14:31:32 -07:00
Mitchell-JohnsonandClaude Opus 5 98a8de8091 PLAN: to do — deprecate copyparty, deprecate Dockhand 
The Dockhand entry is the app half only; migrating the stacks stays in hamlet's
docs/pending.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-23 11:32:52 -07:00
Mitchell-JohnsonandClaude Fable 5.1 ad88a48c01 space: the diff's clock runs inside sudo, so zfs is killed too 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-23 01:19:19 -07:00
Mitchell-JohnsonandClaude Fable 5.1 314a021a3e space: zfs diffs run side by side, give up after four minutes, and die properly 
The timeout killed sudo and left zfs holding the pipe, so the refresh of tank sat on
immich's diff indefinitely. The process group is killed now, and a slow dataset is walked
by mtime instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-23 01:13:05 -07:00
Mitchell-JohnsonandClaude Fable 5.1 53d1da08a5 Space keeps its scans and refreshes them: zfs diff, folder mtimes, resume after cancel 
A scan is saved beside the config and read back after a restart. The next scan starts
from it: on a pool with snapshots zfs diff names what changed and only those folders are
read; elsewhere a folder whose mtime is unchanged keeps its list and only its kept files
are stat'ed again. A cancelled scan keeps what it walked and Resume reads the rest.
Forget drops the saved result. The treemap is every file a tile, grouped by folder, with
no nested boxes: the folder chains of a Linux system made a staircase of empty headers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-23 00:43:30 -07:00
Mitchell-JohnsonandClaude Fable 5.1 c96d6facad PLAN: Space scans a disk or a pool, from a dropdown 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-22 20:28:00 -07:00
Mitchell-JohnsonandClaude Fable 5.1 8f77cc23e1 Space: the picker itself — pools with layout and members, then disks on their own 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-22 20:27:20 -07:00
Mitchell-JohnsonandClaude Fable 5.1 e9fa21768e Space: the unit is a disk on its own or a ZFS pool as a whole, picked from a dropdown 
A pool's member disks are listed under it, not offered apart; the picker shows the
layout, members, size and scan state.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-22 20:26:03 -07:00
Mitchell-JohnsonandClaude Fable 5.1 4280b2226a Top: a hyper-thread wears its core's temperature, and the columns line up 
The process table carries each logical CPU's core id from sysfs topology; a row with
no reading still has the temperature cells, so the % column is one column.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-22 20:22:54 -07:00
Mitchell-JohnsonandClaude Fable 5.1 41e63f74e9 space: 32-bit builds — stat fields are int32 there 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-22 20:19:19 -07:00
Mitchell-JohnsonandClaude Fable 5.1 fc5236471e Space: what is using a disk, WizTree-style 
Pick a disk, not a partition; the scan walks every filesystem it backs, as root when
the host allows, in a helper process that streams the tree back. A tree of folders
largest-first over a treemap of the selected folder, nested to the file and coloured by
kind, plus the thousand largest files. The rail no longer offers the hub-only pages
while a remote machine is being looked at.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-22 20:18:36 -07:00
Mitchell-JohnsonandClaude Fable 5.1 7e3e916f25 Arrange the rail: drag pages into your own order, kept with the account 
The close cross on a detail pane is drawn, not a glyph, and sits in no box.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-22 19:59:25 -07:00
Mitchell-JohnsonandClaude Fable 5.1 b8ea7abcc7 A × closes a detail pane on a wide screen: Docker, Services, Cron 
On a phone the list gives way to the pane and Back returns; on a desktop the list stays
beside it and there was no way to put the pane away.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-22 19:52:35 -07:00
Mitchell-JohnsonandClaude Fable 5.1 027a2a575c A Users tab: own password, sessions and accounts in one place 
Settings keeps appearance, rates, tokens and About.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-22 19:45:01 -07:00
Mitchell-JohnsonandClaude Fable 5.1 2b54a06821 Users, with their own settings, and a list of where each is signed in 
The login is a user name and a password. The old single password becomes the account
named after the OS user on first boot, along with its sessions and the machine-wide look.
Settings gains Users (add, remove, reset a password — your own password confirms) and
Sessions (device, IP, last seen; revoke one or sign out everywhere else). Theme, accents
and the dashboard layout follow the account; the refresh rate, graph history and alerts
stay the machine's. Pairing a machine takes a user on it. Log lines name the actor.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-22 18:20:30 -07:00
Mitchell-JohnsonandClaude Opus 5 db214ebc2b PLAN: to do — users, with their own settings 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-22 17:36:53 -07:00
Mitchell-JohnsonandClaude Opus 5 e558ca7ace PLAN: to do — a public download page and an updater that uses it 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-22 16:47:34 -07:00
Mitchell-JohnsonandClaude Opus 5 4b255fe63f A drive is a drive: NVMe namespaces fold into their controller 
An Apple SSD presents three namespaces and the kernel lists each as a whole disk, so the
dashboard was showing one drive as three. The Disks page has always folded them; now the
dashboard, the Top tab and the disk.* sensors do too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-22 09:11:03 -07:00
Mitchell-JohnsonandClaude Opus 5 5761ee09db BUGS: note the one Firefox console line that is navigation, not a fault 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-22 00:11:37 -07:00
Mitchell-JohnsonandClaude Opus 5 5412e316de A live socket stays with the machine it was opened for 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-21 23:53:18 -07:00
Mitchell-JohnsonandClaude Opus 5 3c7e46104c Bug bash 5: the URL, the fleet loop, and an empty card 
A link to a hub page no longer reloads onto the last machine; a link from a hub-only page to
a machine's page arrives at that page; a page being left no longer asks about the next page's
selection; the hub holds its machines to the rate it is itself running.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-21 23:26:36 -07:00
Mitchell-JohnsonandClaude Opus 5 a7b8319974 Disks: the SMART table stays inside its card 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-21 21:09:43 -07:00
Mitchell-JohnsonandClaude Opus 5 dbfe163a75 A machine is titled by its own hostname, and follows a rename 
The name a machine was added under stays its identity — the URL, its accent, what the hub's
API calls it — and is what the Machines page shows as 'Added as'. The hostname is read live
rather than at start-up, so renaming a machine reaches the console without a restart.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-21 17:04:17 -07:00
Mitchell-JohnsonandClaude Fable 5.1 ec94cb8bff Top: a graph's past holds still — columns draw the peak of their own fixed slice of history 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-21 10:54:30 -07:00
Mitchell-JohnsonandClaude Fable 5.1 be0f2a5aaf The hub keeps every machine's sampling rate and history window in step 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-21 10:14:23 -07:00
Mitchell-JohnsonandClaude Fable 5.1 668e716601 Top wears the skin; the refresh rate reaches every machine; bug bash 4 
The process table's columns fit the box they are in, the rate control saves, curves are
timed on the viewer's clock and keep to the stride, the power icon is drawn, and a young
history draws in from the right instead of being stretched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-21 10:12:45 -07:00
Mitchell-JohnsonandClaude Opus 5 ef3dbdf543 Top: a long CPU model keeps to its line 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-20 23:54:33 -07:00
Mitchell-JohnsonandClaude Opus 5 0587f50142 A Top tab: the whole machine on one screen, in btop's style 
Boxes with their name in the border, dot curves that climb green to red, per-core meters and
temperatures, memory and filesystems, per-interface traffic and the process table. Drawn from
the sensor stream and /api/procs, so it works on every machine the console can see.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-20 23:52:29 -07:00
Mitchell-JohnsonandClaude Opus 5 b8eb4f5d40 Services, Docker and Cron fill the window when nothing is selected 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-20 20:08:09 -07:00
Mitchell-JohnsonandClaude Opus 5 04dd98e2cd Settings: one window for how far back every graph reaches, 1 minute to 24 hours 
Each machine keeps 300 points per curve whatever the window, so memory doesn't follow it;
a wider window means wider points, each the average of its bucket rather than one instant
out of it. The hub passes the setting to every machine it manages.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-20 19:07:55 -07:00
Mitchell-JohnsonandClaude Opus 5 885b050d02 The Monitors tab is called Dashboard again 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-20 18:29:09 -07:00
Mitchell-JohnsonandClaude Fable 5.1 1e992ca1eb Stacks: Duplicate, Labels and Network in the form, task verdict badges, a short Where; Cancel really cancels 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 13:12:54 -07:00
Mitchell-JohnsonandClaude Fable 5.1 a229f2b597 Dialogs scroll inside themselves on a phone: nothing squeezed, clear of the status bar, buttons pinned 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 12:43:23 -07:00
Mitchell-JohnsonandClaude Fable 5.1 a446a03b7d Bug bash 3: solid HUD dialogs, remote stacks deployed from the hub's shell, stopped-on-purpose is not down, the New stack dialog keeps what you typed 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 12:30:15 -07:00
Mitchell-JohnsonandClaude Fable 5.1 634e286796 Docker: delete volumes nothing uses, prune anonymous ones; images marked in use, remove unused 
A named volume is deleted by typing its name, one at a time; the API wants it named
twice; nothing is forced. Prune touches anonymous volumes only.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 08:01:57 -07:00
Mitchell-JohnsonandClaude Fable 5.1 c017b422d5 Docker: new stacks and containers, stack down/delete, container remove, image pull/remove/prune 
A new stack is a folder in a home (where stacks are already kept: Dockhand's group here,
~/stacks elsewhere) with compose.yaml and an optional root-only .env, validated by compose
before it is kept and deployed the way its neighbours were — inside dockhand, or to a node
over tcp from the hub. New container is the same dialog on a form that writes the compose.
Homes are scanned, so undeployed stacks stay listed. Delete moves a stack's files to
.hamlab-trash; nothing is forced, and volumes and data are never touched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 07:50:43 -07:00
Mitchell-JohnsonandClaude Fable 5.1 df6c1dcebf Login: no title under the logo — the logo has the name (it is the image's alt text now) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 19:43:44 -07:00
Mitchell-JohnsonandClaude Fable 5.1 3a882e2d84 New app icon and login logo: the HamLab penguin 
Every icon is rendered from web/scripts/logo-source.png by scripts/icons.mjs: trimmed to
the artwork's own edge, transparent corners for the browser icons, a dark ground from the
artwork's palette for iOS (which paints transparency black) and for Android's maskable
icon, where the whole artwork sits inside the 80% safe circle. The login screen no longer
crops the logo to a circle or repeats the wordmark. Service worker cache bumped so
installed apps fetch the new files.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 19:41:47 -07:00
Mitchell-JohnsonandClaude Fable 5.1 1d48267d90 Stacks: remote stacks editable from the hub, .env editing, deploy diff, image update checks; a gone container says so 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 16:42:54 -07:00
Mitchell-JohnsonandClaude Fable 5.1 eddba4250b Bug bash 2: 11 bugs in Stacks, Alerts, Docker, skins and the installer, fixed and verified (BUGS.md) 
Stacks: deploy with every compose file (labels from inspect, not comma-joined ps);
refuse a stale overwrite; empty list without Docker. Alerts: firing state survives a
restart without re-announcing; vanished alerts are recorded as cleared with the reason;
rule values validated; 200 rules. Docker: 404 for a removed container's logs. Skins: four
palettes lifted to readable contrast. Phone: bigger toggles and swatches. Installer:
deploy republishes; an older published build is never installed silently.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 15:59:51 -07:00
Mitchell-JohnsonandClaude Fable 5.1 9c4f9dfa9f Docker: a Stacks section — edit a compose file, validate, deploy; fix the sideways scroll on phones 
Stacks are found from container labels; the file is resolved on the host or through an
orchestrator's mounts, and compose runs where the file makes sense (inside Dockhand here),
so a redeploy is identical to how the stack was deployed. Save validates with compose and
keeps a .hamlab-bak; Deploy / Update / Restart / Stop run as followed tasks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 15:23:50 -07:00
Mitchell-JohnsonandClaude Fable 5.1 08fb9d6e38 container.<name>.down: should be up and isn't — not merely stopped; the default alert watches that 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 15:10:17 -07:00
Mitchell-JohnsonandClaude Fable 5.1 cb1617c2b8 Alerts tab, Docker tab, nine new skins including HUD, a colour for every machine 
Alerts: rules over sensors, watched fleet-wide from the hub; ntfy and webhook channels;
firing count on the rail. Docker: containers by stack with live use, start/stop/restart,
following logs, inspect, images/volumes/networks. Skins: HUD, Nord, Gruvbox, Catppuccin
(+Latte), Dracula, Tokyo Night, Solarized (+Light). Accents: per-machine colour.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 15:07:15 -07:00
Mitchell-JohnsonandClaude Fable 5.1 63fcba6077 Rail: an outline clock for Cron instead of the ⏱ emoji 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 12:06:15 -07:00
Mitchell-JohnsonandClaude Fable 5.1 4542fa6c99 installer messages fit 80 columns 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 11:14:16 -07:00
Mitchell-JohnsonandClaude Fable 5.1 d9470bbfe1 install.sh: every line fits 80 columns; the closing card is a tidy table 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 11:13:27 -07:00
Mitchell-JohnsonandClaude Fable 5.1 3c0c504e35 README: the public install one-liner 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 10:57:31 -07:00
Mitchell-JohnsonandClaude Fable 5.1 e75fa0d759 install.sh: an interactive installer — user, address, HTTPS, firewall, password — that still runs unattended; make publish 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 10:56:42 -07:00
Mitchell-JohnsonandClaude Fable 5.1 a838adcb27 Machine menu shows each machine's address instead of its load 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 10:45:24 -07:00
Mitchell-JohnsonandClaude Fable 5.1 d3f2a83249 Terminal: Ctrl+C copies a selection (else ^C), Ctrl+V pastes, over http too; drop Log out everywhere 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 10:44:17 -07:00
Mitchell-JohnsonandClaude Fable 5.1 d7ed4d8549 Quality of life: in-app dialogs, selections in the URL, editable widgets, token dialog, gallery search, status history, stream errors that say why, tab reordering and shortcuts, log out everywhere 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 10:34:59 -07:00
Mitchell-JohnsonandClaude Fable 5.1 8ee2a2d293 Bug bash 2026-09-19: 17 bugs fixed and verified (BUGS.md) 
Widgets: a bad pattern no longer silences every widget; malformed layouts load.
Shell: lost session returns to login at once; tab titles; Escape closes menus and dialogs.
Files: names are one path segment; unsaved edits ask before page/machine/tab changes;
only folders that opened are remembered. Terminals: unique default names; 64-char names.
API: queries capped at 10 000 values; 32 streams per client, 512 in all; only
/embed/widget is frameable. ZFS sensors: one unprivileged zpool list, no sudo journal spam.
Phone: 36 px inputs; dataset rows as wide as their cells. Machines: valid name pattern.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 10:18:23 -07:00
Mitchell-JohnsonandClaude Fable 5.1 36de4ad4a1 external demo: public-token guidance 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 09:14:30 -07:00
Mitchell-JohnsonandClaude Fable 5.1 47b21a6eca Hub proxy: don't relay a node's policy headers on top of the hub's own 
Access-Control-Allow-Origin arrived twice on /api/nodes/<m>/sensors, which a
browser on another origin refuses. Also: demo lists say when the token hides them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 09:11:35 -07:00
Mitchell-JohnsonandClaude Fable 5.1 62605d57eb Widgets tab links the external demo when config.json names one (demo_url) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 08:58:36 -07:00
Mitchell-JohnsonandClaude Fable 5.1 65a9888784 Sensor API answers other origins (token only); external demo page 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 08:49:59 -07:00
Mitchell-JohnsonandClaude Fable 5.1 0b0b2897cb Rail: Widgets sits under Machines 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 08:37:52 -07:00
Mitchell-JohnsonandClaude Fable 5.1 b1c1404181 Widgets tab: every widget type and every sensor, live, one of each 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 08:29:15 -07:00
Mitchell-JohnsonandClaude Fable 5.1 4c2eb57ea6 Sensors: per-service and container sensors, pattern-limited tokens, fleet query, steady sampler 
- service.<unit>.active|state for services that are enabled, up or failed
- containers.* and container.<name>.running|state|health|mem|cpu (cgroup reads, no docker stats)
- sensors:read tokens can be limited to ID patterns, enforced on every endpoint and,
  through the hub's filter header, on managed machines; the header can only narrow
- /api/fleet/sensors/query: one shaped query, every machine, one request
- the sampler waits what's left of the interval, so 250 ms means 4 a second
- Monitors asks for the families it draws instead of *
- unpowered hwmon sentinels (-93 °C) are not temperatures

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 08:08:44 -07:00
Mitchell-JohnsonandClaude Fable 5.1 4ddf10be51 The full switch: Monitors and Disks read only the sensor API; /api/metrics retired 
- stock Monitors modules fed by one sensor stream, the catalog and one batch of
  histories (GET /api/sensors/history?ids=…), through sampleAdapter.ts
- the stream closes while the page is hidden and refills its curves on return
- Disks tab live rates from the stream
- new sensors: hamlab.sample_interval, ups.last_event
- stream: asking for about the sampling rate means every sample
- NUT runtime rounded at the source

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 07:52:14 -07:00
Mitchell-JohnsonandClaude Fable 5.1 f01175ceca Embed: bare=1 is really transparent 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 07:40:36 -07:00
Mitchell-JohnsonandClaude Fable 5.1 b05a053920 Sensors: read-only tokens, live stream, Prometheus, status sensors, widgets, embeds 
- sensors:read tokens open the sensor endpoints (here and on managed machines
  through the hub) and nothing else; full tokens are never accepted from a URL
- /api/sensors/stream: a shaped query that keeps answering, over SSE
- /metrics: Prometheus text format
- services, updates, ZFS and SMART as background status sensors
- widget library over the sensor API; custom widgets on the dashboard
- /embed/widget: one live widget in its own bundle, the only frameable path
- Dashboard tab is now Monitors

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 07:37:10 -07:00
Mitchell-JohnsonandClaude Fable 5.1 468f45251d priv: a failed root probe is retried, not cached for the life of the daemon 
The Mac mini's daemon started while the machine was still booting, sudo -n true
took longer than the 3 s timeout, and sync.Once kept the answer: no root, so the
next self-update could not replace the binary.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 07:05:32 -07:00
Mitchell-JohnsonandClaude Fable 5.1 43bf787696 Sensors: one standard read interface — catalog, single value, history, shaped query 
Every number the collector samples becomes a sensor with a stable dotted ID
(cpu.core.3.usage, fan.nct6775.fan2, fs.root.used_pct). Clients discover them
from the catalog and ask for exactly what they want, including as a shaped
query: send the object you want, get it back with the values filled in.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 07:04:03 -07:00
Mitchell-JohnsonandClaude Fable 5.1 01aeb226ea Disks: show eMMC firmware revision as text when it is text 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 04:50:05 -07:00
Mitchell-JohnsonandClaude Fable 5.1 232163ff4c Disks: eMMC is one device, not three, and reports wear from its own registers 
mmcblkNboot0/boot1/rpmb are hardware areas of the chip; hide them on Disks and
Dashboard. Badge eMMC/SD instead of SSD. smartctl cannot talk to MMC, so read
life_time and pre_eol_info from /sys instead of showing its error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 04:48:52 -07:00
Mitchell-JohnsonandClaude Fable 5.1 488d929733 Portable install: the binary installs itself; static builds on five architectures; root via sudo, doas or uid 0; every package manager; older systemd; no lsblk 
- hamlab install / uninstall (internal/setup): systemd or OpenRC, elevates through
  sudo or doas, runs as the invoking user, sets or generates the password, names only
  journal groups that exist, waits for /healthz, reports an active firewalld/ufw.
  --purge and the data chown never touch a directory that isn't Ham Lab's alone.
- install.sh: curl-able bootstrap (arch detection, SHA256SUMS, offline beside the
  binaries, noexec /tmp fallback). make release builds dist/ for amd64, arm64, arm
  (ARMv6), 386, riscv64. Every build is CGO_ENABLED=0; the native one used to link glibc.
- internal/priv replaces every literal `sudo -n`: uid 0, sudo or doas.
- Updates acts on dnf, zypper, pacman and apk as well as apt; dnf shows installed versions.
- Services falls back to the plain systemctl table before systemd 246 and says so on
  hosts without systemd. Disks falls back to /sys/block without lsblk.
- Listener binds with IP_FREEBIND, so a LAN address that isn't up at boot is no failure.
- Self-update: the new build must pass `check` against the live config, is put in
  place by atomic rename, and the daemon re-execs itself (same under any supervisor).
- State and config writes are fsynced; a corrupt state.json is set aside, not fatal.
- Terminal shutdown no longer empties the saved list, so names come back after a
  restart by design rather than by winning a race.
- Config default is ~/.config/hamlab; an existing ~/apps/hamlab is still found.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 23:00:20 -07:00
Mitchell-JohnsonandClaude Fable 5.1 91b7f4f009 Terminal sessions move from the rail into a tab strip on the terminal page 
The rail is now a flat list of pages; Terminal is one of them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 22:10:22 -07:00
Mitchell-JohnsonandClaude Fable 5.1 7eb00d089f BUGS.md: accurate reap verification; note the resurrect fix 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 18:04:20 -07:00
Mitchell-JohnsonandClaude Fable 5.1 1ee4673d7c Don't resurrect exited terminals on daemon restart 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 17:53:29 -07:00
Mitchell-JohnsonandClaude Fable 5.1 44e85b768f BUGS.md: fix pass outcomes 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 17:53:05 -07:00
Mitchell-JohnsonandClaude Fable 5.1 c1805f6ad5 Bug 14: stream journal reads instead of buffering them; soft memory limit for the GC 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 17:51:09 -07:00
Mitchell-JohnsonandClaude Fable 5.1 03be65ee99 pprof behind the login for memory investigation 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 17:47:16 -07:00
Mitchell-JohnsonandClaude Fable 5.1 f093b322ee Bug bash fixes: hash routing + back button, hub-only pages reset on machine switch, exited terminals reap after 10 min, self-update refuses while busy unless forced, touch-sized controls, power confirm by machine name, down interfaces collapsed and empty modules hidden, load label, offline backoff, cron log newest first, procs empty message, timer timestamp parsing 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 17:46:16 -07:00
Mitchell-JohnsonandClaude Fable 5.1 27f9665606 Bug bash fixes: special-file reads, per-client lockout, prefs validation, security headers, kernel-thread guard, login feedback, error text; plus journal-quiet sudoers + journal groups, upload overwrite guard, machines pattern, dnf freshness, cron/services speed, clean version 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 17:41:52 -07:00
Mitchell-JohnsonandClaude Fable 5.1 57c58ebffb Bug bash 2026-09-18: findings list (discovery only) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 17:34:55 -07:00
Mitchell-JohnsonandClaude Fable 5.1 0af24f41c3 Disks: one card per physical drive (NVMe namespaces folded under their controller), partitions summarised on one line instead of listed 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 14:18:20 -07:00
Mitchell-JohnsonandClaude Fable 5.1 a372378098 Mac mini support: arm64 cross-build installed beside the binary and pushed by arch; ARM machine name from the device tree 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 14:13:09 -07:00
Mitchell-JohnsonandClaude Fable 5.1 b6502ba362 Disks: one toggle opens SMART for every drive, remembered per device 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 13:41:38 -07:00
Mitchell-JohnsonandClaude Fable 5.1 4b8a2b15ed Refresh rate is honoured exactly everywhere: no list floor, sampler up to 10 s, server caches reduced to coalescing only 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 13:32:36 -07:00
Mitchell-JohnsonandClaude Fable 5.1 bbf0f2580f Global refresh rate in Settings drives every live view and each machine's sampler; per-page pickers removed 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 13:27:13 -07:00
Mitchell-JohnsonandClaude Fable 5.1 a4f39859b6 Settings page: themes (Dark, Light, Cyberpunk, Retro) with xterm palettes and hub-synced choice, change password, access tokens, about 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 13:14:01 -07:00
Mitchell-JohnsonandClaude Fable 5.1 254e0cdd58 Update from hub: node self-update endpoint, hub push per machine / all, push-nodes CLI run by make deploy, outdated badges + Update buttons on Machines 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 12:04:31 -07:00
Mitchell-JohnsonandClaude Fable 5.1 e9be94e32b SMART: detect drives whose firmware rejects the self-test log (WD SN580) and hide the test buttons with an explanation 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 11:37:49 -07:00
Mitchell-JohnsonandClaude Fable 5.1 340232b724 Header dot: don't go red on machine switch; reflect remote reachability 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 11:36:04 -07:00
Mitchell-JohnsonandClaude Fable 5.1 7dcdb5dc98 Multi-host phase 1: API tokens, auto TLS with pinned fingerprints, hub-side node manager + reverse proxy (WebSockets included), machine switcher in the rail, Machines page 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 11:35:29 -07:00
Mitchell-JohnsonandClaude Fable 5.1 fc42675e9f Processes: show current / max clock so idle scaling isn't mistaken for the rated speed 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 06:13:24 -07:00
Mitchell-JohnsonandClaude Fable 5.1 bd475eaafd Processes tab: whole page scrolls, tools + column header pinned together, detail sheet sticks to the bottom, metrics boxes collapsible 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 06:12:29 -07:00
Mitchell-JohnsonandClaude Fable 5.1 b802dcf313 Processes tab: btop-style cpu/mem/net boxes with labels, gradient bars, block graph, CPU model/GHz/temp, memory breakdown, per-interface rates, and a refresh interval control 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 06:05:09 -07:00
Mitchell-JohnsonandClaude Fable 5.1 e4df878ac0 Processes tab (btop-style): every process with CPU%, memory, cpu time, start, sortable + filter, per-core and memory bars, detail pane with terminate/kill; module removed from the dashboard; sampling only while the tab is open 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 06:01:12 -07:00
Mitchell-JohnsonandClaude Fable 5.1 ff922bda74 Dashboard: Processes module (top by CPU/memory, PID, user, threads, SIGTERM), sampled at most every 2 s 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 05:58:11 -07:00
Mitchell-JohnsonandClaude Fable 5.1 4986af5ac9 Drop the top activity bar 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 05:55:11 -07:00
Mitchell-JohnsonandClaude Fable 5.1 116f4c72f6 Rail: proper folder icon for Files (inline SVG, matches the monochrome glyphs) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 05:54:20 -07:00
Mitchell-JohnsonandClaude Fable 5.1 2d23ba8582 Files rail icon: use a glyph iOS renders 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 05:48:33 -07:00
Mitchell-JohnsonandClaude Fable 5.1 63b020d556 Files tab: browser with breadcrumbs/sort/filter, text editor, image preview, streamed upload with progress, download, mkdir/touch/rename/move/chmod/delete, open-in-terminal; root toggle via sudo-reexecuted 'hamlab fs' helper 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 05:39:57 -07:00
Mitchell-JohnsonandClaude Fable 5.1 38d4ee09ae storage: re-probe smartctl lazily so an install from the Disks tab works without a restart 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 22:53:32 -07:00
Mitchell-JohnsonandClaude Fable 5.1 c704e87065 Disks tab: lsblk devices with partitions/usage/live IO, SMART via smartctl (standby-aware, self-tests, install button), ZFS pools/datasets/snapshots with scrub and snapshot-now 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 22:48:08 -07:00
Mitchell-JohnsonandClaude Fable 5.1 724f30c02b Feedback sweep: spinners on every async button, toasts for action outcomes, top activity bar, loading states on every page, terminal connecting overlay, dashboard save indicator, task pane pinned above the package list with auto-scroll and completion toast 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 18:04:45 -07:00
Mitchell-JohnsonandClaude Fable 5.1 d08b246b5f Land on the Dashboard by default 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 17:44:08 -07:00
Mitchell-JohnsonandClaude Fable 5.1 acb11193b9 Updates tab (apt list/refresh/upgrade/changelog, reboot flag) on a new background tasks package with live WS streaming; header power button with hostname confirmation 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 17:40:29 -07:00
Mitchell-JohnsonandClaude Fable 5.1 41c2f05535 Cron tab: crontabs, cron.d, user crontabs, run-parts dirs and systemd timers in one list; next/last run, run now, enable/disable, in-place editing, cron log and output file 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 16:21:35 -07:00
Mitchell-JohnsonandClaude Fable 5.1 bc891b1042 Services tab: systemd unit list/filter, detail with start/stop/restart/reload/enable/disable, live journal tail over WS, unit file viewer 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 15:35:03 -07:00
Mitchell-JohnsonandClaude Fable 5.1 cd3979efce Dashboard: stop flex children shrinking (bar text overflowed), wrap the bar, gate saves on loaded layout, fix double v in footer 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 14:07:56 -07:00
Mitchell-JohnsonandClaude Fable 5.1 1751afc974 Configurable update interval (100 ms – 2 s): drives daemon sampling and UI polling, saved with the dashboard layout 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 13:54:48 -07:00
Mitchell-JohnsonandClaude Fable 5.1 eb70daf129 Dashboard customize mode: reorder/hide modules and individual rows, layout saved server-side (GET/PUT /api/prefs/dashboard) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 13:46:41 -07:00
Mitchell-JohnsonandClaude Fable 5.1 7053cce7fe Portability: sysfs-only sensors with thermal-zone fallback, /sys/block disk discovery, fs blacklist, pwrstat/NUT/apcupsd UPS, login shell from passwd, templated unit + install for any user, cross-compile targets 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 13:43:44 -07:00
Mitchell-JohnsonandClaude Fable 5.1 3c6f4d4639 Dashboard tab: CPU/mem/load, hwmon temps+fans, net, disks, filesystems, UPS with 10-min sparklines 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 13:37:39 -07:00
Mitchell-JohnsonandClaude Fable 5.1 24d64245ec Ham Lab logo: PWA icon set + login screen 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 13:14:30 -07:00
Mitchell-JohnsonandClaude Fable 5.1 18111c089b CLAUDE.md: import Hamlet's rules; project settings 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 12:57:04 -07:00
141 changed files with 32948 additions and 341 deletions
Expand all files Collapse all files
.claude/settings.json
+8
View File
@@ -0,0 +1,8 @@
{
"permissions": {
"allow": [
"Bash(systemctl status *)",
"Bash(systemctl is-active *)"
]
}
}
.gitignore
+1
View File
@@ -2,3 +2,4 @@ bin/
internal/web/dist/
web/node_modules/
web/dist/
dist/
BUGS.md
+610
View File
@@ -0,0 +1,610 @@
# Bug bash — 2026-09-18
Fix pass 2026-09-18: every item carries its outcome; ✅ resolved and verified live, ➖ no change with reason. Found by a headless Chromium pass (iPhone and
1440px desktop viewports) over every page on the hub and on each remote machine, an
offline-node scenario (wyse stopped for the test, then restarted), direct API probing
with hostile inputs, and a code read. Ordered by severity within each group.
## Security / robustness
1. ✅ **RESOLVED 2026-09-18** — only regular files are readable; the byte cap applies to what is sent, not to `stat`. `/dev/zero`, `/dev/null` and FIFOs answer 400 in 30 ms; `/proc/cpuinfo` still reads. **Reading a special file hangs the request forever.** `GET /api/files/read?path=/dev/zero`
never ends: the `max` cap compares against `stat` size, which is 0 for device files,
so the cap never triggers and the handler streams `/dev/zero` until the client gives
up. Same for FIFOs (blocks on open) and `/proc/self/mem`. Opening one in the Files
viewer would hang the tab. Needs: refuse non-regular files for read/edit, and cap the
copy with `io.LimitReader` regardless of `stat`.
2. ✅ **RESOLVED** — per-client buckets (first X-Forwarded-For hop or the peer) with the same 5-strike schedule, plus a global backstop of 100 failures per 5 min → 1 min. Verified: core's public IP locked while grid logged in fine. **Login lockout is global, not per client.** Five bad passwords from anyone on the
internet lock the real user out for up to 10 minutes (`auth.Limiter` is one bucket).
The bash locked the hub for the full 10 minutes twice. Also locks `POST /api/tokens`,
so hub pairing fails during a lockout. Needs per-IP buckets with a modest global cap.
3. ✅ **RESOLVED** — server rejects refreshMs outside 100–60000 and non-identifier themes (400); client ignores out-of-range rates from any source. **UI prefs from the server are not validated.** A stored `refreshMs` of `-5` made the
client call `setInterval(-5)` on every live view: a request storm that took the
browser to `ERR_INSUFFICIENT_RESOURCES` and hammered the daemon. `theme` is validated
only by falling back. Needs range checks on load (and on save server-side).
4. ✅ **RESOLVED** — every response carries X-Frame-Options DENY, CSP frame-ancestors none, nosniff, no-referrer. **No anti-framing / CSP headers.** Only HSTS (from NPM). A root-capable console should
send `X-Frame-Options: DENY`, `Content-Security-Policy: frame-ancestors 'none'`,
`Referrer-Policy: no-referrer`, `X-Content-Type-Options: nosniff` on every response.
5. ✅ **RESOLVED** — kernel threads (empty cmdline) and the daemon's own PID are refused. **Signals can be sent to kernel threads.** `POST /api/procs/2/signal` (kthreadd)
returns 204. Harmless in practice, but the guard should refuse PIDs with an empty
cmdline (kernel threads) and our own PID.
6. ✅ **RESOLVED** — unit runs with `SupplementaryGroups=systemd-journal adm` so journal reads need no sudo, and the installer drops `/etc/sudoers.d/hamlab-quiet` (no syslog/pam session for the read-only polling commands). Verified: 0 sudo lines in 60 s of 250 ms polling (was 839/10 min). Applied on all four hosts. **Journal spam from the daemon's own sudo calls.** Every sudo'd `journalctl`, `smartctl`
or `pwrstat` writes three `pam_unix(sudo:session)` lines into the journal *under
`hamlab.service`*: 839 lines in 10 minutes at a 250 ms refresh, and the journal is
already 4 GB. It also buries Ham Lab's own log entries. Fix options: add the service
user to `systemd-journal` (and `adm`) so journal reads don't need sudo, and/or a
sudoers `Defaults!/usr/bin/journalctl !syslog, !pam_session` line in the installer.
7. ✅ **RESOLVED** — server answers 409 unless `overwrite=1`; the UI asks "Replace it?" first. **Uploading over an existing file silently overwrites it.** No confirmation, no
"replace?" prompt. Same for "+ file" when the name exists (server refuses; UI just
shows the error, fine) — upload is the gap.
8. ➖ **NO CHANGE** — Go's mux redirects the unclean path to `/etc/passwd`, which is an ordinary deep link the SPA must serve; there is no API prefix left by the time the static handler sees it. Harmless. **`/api/<anything>/../../x` falls through to the SPA** and returns `index.html` with
200 instead of a JSON 404. Harmless, but sloppy for an API prefix.
## Functional bugs
9. ✅ **RESOLVED** — 401 carries the server message; the form shows "wrong password" (verified headless). **Wrong password gives no feedback at all.** The 401 is thrown as `Unauthenticated`
with an empty message, and the login form only renders `err` when non-empty. The
button just re-enables. Reproduced on phone and desktop.
10. ✅ **RESOLVED** — hub-only pages bounce to the dashboard on switch (verified headless). **Switching machines while on Settings or Machines keeps showing the hub's page.**
`switchMachine` only resets the view for Machines; Settings stays, so "mac" shows
core's account/token/about panels. Both are hub-only pages and should bounce to the
dashboard (or be clearly labelled as hub pages).
11. ✅ **RESOLVED** — escaped; no console error. **Machines add-form `pattern` is an invalid regex in Chromium.** `[a-z0-9][a-z0-9-]*`
fails under the `v` flag ("Invalid character class"), logs a console error and the
browser skips validation. Needs `[a-z0-9][a-z0-9\-]*`.
12. ✅ **RESOLVED** — refresh from the newest repodata under libdnf5/dnf caches (Mac: 9 d), last upgrade from `dnf history` (194 d), label reads "from dnf history". **Updates tab on dnf hosts shows nonsense.** Mac: "Index refreshed 561d ago" (mtime of
`/var/cache/dnf`, which Fedora 41+ no longer uses; `libdnf5` is the live cache) and
"Last upgrade never · from apt history" (label and source are apt-specific). Needs
`dnf history` for last upgrade, metadata timestamps for refresh, and neutral labels.
13. ✅ **RESOLVED** — journal read incrementally by cursor, timers from one `systemctl show`, services' two calls run in parallel. Cron 3.6 s → 0.08 s, services 0.6 s → 0.05 s, with last-run and next-run data intact. **Cron overview is slow: 3.6 s per request on core.** At any refresh rate the page
is constantly busy. The 30-day, 20 000-line journal scan for "last run" is rebuilt
every 5 s; timers make three `systemctl` calls. Needs an incremental journal cursor
and a single `systemctl` call. Services list is 0.6 s for the same reason (two calls).
14. ✅ **RESOLVED** — heap profile showed a 48 MB `bytes.Buffer` from buffering `journalctl` dumps; those are now streamed line by line, plus a 160 MB soft memory limit for the GC. RSS 22 MB fresh, 27 MB after 25× cron/logs/procs/services (was 180 MB). pprof stays available behind the login at `/api/debug/pprof/`. **Daemon memory grows.** RSS on core went 22 → 85 → 216 MB during the day (136 MB at
rest after the bash). Suspects: per-pid caches in the process table (`procInfo` only
prunes on the next walk), task output buffers, SMART cache, xterm scrollback rings
that are never freed for exited sessions. Needs a heap profile under load.
15. ✅ **RESOLVED** — `--dirty` dropped from the version and deploys now follow commits; binaries report e.g. `v0.1.0-38-gc1805f6`. **Version string is always `-dirty`.** `make deploy` builds before the commit, so
every installed binary reports e.g. `v0.1.0-32-ga372378-dirty`, and the Machines page
compares with `-dirty` stripped. Build after commit, or tag releases.
16. ✅ **RESOLVED** — hash routing `#/<machine|hub>/<view>`: reload restores machine and tab, Back/Forward work, links can point at a page on a machine (verified headless). **View resets to Dashboard on reload; no deep links; browser Back leaves the app.**
The selected machine survives a reload (sessionStorage) but the tab does not, and
there is no URL routing at all. Needs hash or path routing (`/#/grid/services`).
17. ✅ **RESOLVED** — newest first. **Cron log tab lists oldest first and starts at the top.** 200 lines from Sep 10 with
the newest at the bottom, off-screen. Should show newest first or scroll to the end.
18. ✅ **RESOLVED** — "Nothing matches." **Processes: empty filter result shows nothing.** No "nothing matches" message.
19. ✅ **RESOLVED** — exited shells remove themselves 10 minutes after exit (verified live: seeded 17:53:56, gone at 18:03:59 with no daemon restart in between). Found and fixed alongside: a daemon restart used to re-spawn *exited* shells as fresh running ones; only running shells are persisted now. Idle running shells are deliberately kept. **Terminals accumulate.** Nothing ever reaps exited or idle sessions; the bash left
several "bug-bash-term" shells on the hub until deleted by hand. Exited sessions
should auto-remove after a while; idle ones at least be counted somewhere.
20. ✅ **RESOLVED** — a node answers 409 "busy" when a package task is running or a terminal is attached; the Machines page offers Force, `hamlab push-nodes --force` / `make push-nodes-force` do the same. Verified: 409 with a terminal attached on grid, 200 with force. **Self-update restarts without warning running work.** A push to a node kills its
open terminals and any running task (upgrade in progress). The hub should refuse or
warn when the node reports a running task or attached terminals.
21. ✅ **RESOLVED** — PathError and LinkError are unwrapped to the bare reason. **Error text leaks raw Go errors.** e.g. rename shows "rename /tmp/x /tmp/y: no such
file or directory". Fine for a lab, ugly for a product; strip the syscall prefix.
## Mobile / UX
22. ✅ **RESOLVED** — on coarse pointers every button, chip, icon button, link, header and select is at least 36 px tall (verified: no control under 32 px on the dashboard or processes pages). **Tap targets under 28 px on phone:** header ⏻ (25), logout (22), customize ▲ ▼ hide
(23), terminal leaf ✎ ✕ (24), chips (25), table headers (27). Apple's guideline is
44 pt. Systemic: the small `.btn`/`.chip`/`.icon-btn` sizes need a touch variant.
23. ✅ **RESOLVED** — the dialog asks for the machine name (`mac`); the node still checks its own hostname under the hood. **Power dialog on the Mac asks you to type `5ce91ee54af6`.** The confirmation uses
the node's hostname, which on the Mac is the unset default. Should offer the machine
name (`mac`) as the confirmation word on remote machines.
24. ✅ **RESOLVED** — down interfaces collapse to one line marked "down"; modules with nothing to show (fans, UPS, temps, disks) hide unless customizing. **Dashboard shows dead interfaces and empty modules.** wlp1s0 (down) gets a full
sparkline block of flat lines; "Fans: no fan sensors" is a permanent empty card on
hosts without fans. Down interfaces should collapse to one line; empty modules should
auto-hide unless customizing.
25. ✅ **RESOLVED** — "load 1.39 1.56 1.55" with a tooltip. **Header load average is unlabelled** ("1.39 1.56 1.55" floating in the header).
26. ➖ **NOT FIXING** — the list is intentionally a plain table; the menu anchors inside the row. Cosmetic taste, not a defect. **Files list on desktop is a narrow strip in a sea of empty space** for small
folders; the row menu opens off the right edge of the row. Cosmetic.
27. ✅ **RESOLVED** — after a 502 the client leaves that machine alone for 5 s: 2 requests in 10 s at a 250 ms refresh (would have been ~80), badge and message intact. **Offline machine keeps polling every view at the refresh rate**, producing a 502
per request per page (and console errors). The UI copes — badge, message — but the
hub should back off to the prober's cadence while a node is down.
28. ✅ **RESOLVED** with 6. **Services logs are noise.** The default view of `hamlab.service` logs is the sudo
session spam from bug 6, three lines per poll.
## Confirmed working under abuse (for the record)
Path traversal in files, services, cron, ZFS, procs and updates all rejected. `/etc`
and `/` protected from delete/rename as root. Unit, pool, dataset, snapshot and
package names validated. Token issue refused with a wrong password. Oversized prefs
rejected. Duplicate machine add refused. Unicode + spaces in file names round-trip.
Upload byte-exact. Terminal scrollback survives switching. Offline node: switcher and
header show it, pages show the reason. Remote terminals, logs, tasks and files through
the proxy all work on all three nodes.
# Bug bash — 2026-09-19
Found by driving a scratch instance (same binary, its own state, real fleet behind it) in
headless Chromium at 1440 px and iPhone size: a sweep of every page on every machine for
page errors, failed requests, overflow and small tap targets; then scripted use of every
feature — terminal tabs, Files in a throwaway folder, Processes (killing a process the
test spawned), Monitors customisation and widgets, Services, Cron, Updates, Disks,
Machines, Settings, embeds, machine switching, session loss — and a hostile pass over the
API. Host-changing actions (service control, cron edits, upgrades, power) were opened but
not executed. Every fix below was re-checked by the same script that found it.
## Bugs — all ✅ fixed and verified
1. ✅ **One widget with a bad pattern silenced every custom widget.** They share one stream;
`fan.[` made the whole query a 400, so all of them showed "…" forever with no reason
given. The picker now asks the API before adding and shows its answer; a bad pattern
already saved shows a notice instead of dots.
2. ✅ **A saved layout of the wrong shape could not be loaded** ("could not load layout",
and the page could no longer be customised). `normalize()` now accepts anything.
3. ✅ **A lost session left pages saying "unauthenticated"** for up to 30 s. A 401 from the hub
now returns to the login form at once, and logging in returns to the same page. (A 401
relayed from a managed machine is that machine rejecting the hub, and doesn't.)
4. ✅ **The machine-name rule in "Add a machine" was an invalid regular expression** under the
browser's `v` mode, so it validated nothing and logged a console error on every render.
5. ✅ **Two terminals opened in the same minute got the same name** ("shell 09:45" twice).
Now "shell 09:45 (2)".
6. ✅ **Names had no length limit** — a 328-character terminal tab, a 5000-character token
name. 64 characters, refused with a reason; token names also refuse `/ \ < > "`.
7. ✅ **"+ file", "+ folder" and Rename accepted `../x` and `a/b`**, quietly acting outside the
folder on screen. A name is now one path segment; Move… is how things go elsewhere.
8. ✅ **Unsaved file edits were thrown away without a word** when you picked another page or
machine, or closed the tab. All three now ask; Cancel keeps you where you were.
9. ✅ **A mistyped path in Files was remembered**, so a reload landed on the error. Only a
folder that opened is remembered.
10. ✅ **Escape closed nothing**: the machine menu, the file menu, the Power dialog, the widget
picker. It closes all four.
11. ✅ **Reset on Monitors removed every custom widget with no confirmation.** It asks, and
says how many.
12. ✅ **A few bytes of query bought 27 MB of answer** (`{"k0":"*",…×400}` with `full=1`) — and a
read-only token is public on the demo page. A query may return 10 000 values (thirty
whole machines); past that it is a 400 that says so.
13. ✅ **No ceiling on live streams**: 300 from one token were accepted. 32 per client address,
512 in all, then a 429 that says so; closed streams free their places.
14. ✅ **Anything under `/embed/` lost the anti-framing headers**, not just the widget page.
Only `/embed/widget` is frameable.
15. ✅ **The new ZFS sensors ran three `sudo` commands a minute on every machine** — including
`core`, which has the tools and no pool — and each wrote three lines to the journal:
most of `hamlab.service`'s log was that. Pools are now read with one unprivileged
`zpool list`; a source that finds nothing is asked again ten times less often; and
the installer's quiet list covers `zpool|zfs list|status`.
16. ✅ **Phone: the filter boxes on Processes and Files, the path box and the "/" crumb were
under 30 px.** 36 px like every other control.
17. ✅ **Disks → datasets on a phone: rows ended 50 px before their own cells** when scrolled
sideways (`min-width` was less than the columns add up to).
Also: tab titles now say where you are ("Services · grid — Ham Lab"), so history and open
tabs can be told apart.
## Checked and fine
Terminal focus after every switch/close/return; scrollback after reload; exited shells;
30 tabs with the + still reachable; re-fit on resize; Back/Forward; unknown machine or page
in the URL; wrong password; XSS through terminal, token and file names (all rendered as
text); hidden files, filter, symlinks (broken ones marked), binary files, upload and
replace, folder delete needing the typed name, root toggle; process sort, filter and
Terminate (asks first); themes; password-change validation; duplicate or unreachable
machine; hub terminals not leaking into another machine's page; CORS only on sensor paths;
token list never exposing hashes; cookie flags; deep nesting, empty bodies, odd methods
and `every=` values on the sensor API; phone drawer, key bar and picker.
## Quality of life — all ✅ done 2026-09-19, each verified in the browser
1. ✅ **Selections are in the URL.** `#/grid/services/ssh.service`, `#/hub/cron/job:…`,
`#/hub/files/%2Fetc`: reload keeps them, links carry them, leaving a page drops them.
2. ✅ **Browser Back with unsaved file edits asks**, like page and machine changes do.
3. ✅ **No more browser `prompt()`/`confirm()` boxes** — all 30 are in-app dialogs (`askYes`,
`askText` in `ui.tsx`): Enter submits, Escape or a tap outside cancels, destructive
ones are red and say what they do ("Close", "Delete"), and they fit a phone.
4. ✅ **Token creation is one dialog**: name, patterns, and the machine's own sensor families
as chips to click.
5. ✅ **Widgets can be edited** — sensor, type and title — in place, from Customize.
6. ✅ **The Widgets gallery has a search** over IDs, names and labels (so "gitea" finds its
container's sensors).
7. ✅ **Status sensors keep a history** (300 readings each: an hour of container memory, two
days of drive wear), so their sparklines have a past.
8. ✅ **A stream that fails says why**: "the token was rejected", "too many live streams are
open", "that machine is unreachable", "can't reach Ham Lab" — in embeds and on Monitors.
9. ✅ **Pattern widgets get readable default titles**: `cpu.core.*.usage` → "CPU core usage".
10. ✅ **Terminal tabs reorder** — drag on a desktop, ‹ › on a touch screen, remembered per
machine in the browser — and **Alt+Shift+←/→** switches tabs, **Alt+Shift+T** opens one,
with the keyboard left in the terminal each time.
11. ➖ **"Log out everywhere"** — built, then removed the same day at Mitchell's word: no need
for it on a single-user console.
12. ✅ **Real clipboard shortcuts in the terminal**, from a physical keyboard: **Ctrl+C copies
when text is selected** (and is still ^C when nothing is — the selection clears after a
copy, so a second Ctrl+C interrupts), **Ctrl+V pastes**, Ctrl+Shift+C / Ctrl+Shift+V do
the same unconditionally, Ctrl+Insert / Shift+Insert too. It rides the browser's own
copy and paste events, which xterm already answers, so it works over plain http on the
LAN where `navigator.clipboard` does not exist; bracketed paste is honoured.
# Bug bash 2 — 2026-09-19 (evening): Alerts, Docker, Stacks, skins, the installer
Same method as the morning: a scratch instance with the real fleet behind it, headless
Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four machines (errors,
failed requests, overflow, tap targets) plus a contrast measurement of every skin; then
scripted use of the new features on throwaway containers and stacks — a multi-file stack, a
file edited behind the editor's back, two actions at once, a chatty log, odd container
names, containers stopped and removed while open, every kind of bad alert rule, a rule's
whole life (fire, narrow, pause, delete, survive a restart), token scope and hostile paths
on the new APIs — and a regression of terminals, Files, widgets, deep links and colours.
## Bugs — all ✅ fixed and verified by the script that found them
1. ✅ **Deploying a stack made of several compose files dropped all but the first.** A stack
deployed with `-f compose.yaml -f compose.override.yaml` was redeployed from
`compose.yaml` alone, so everything the override set silently vanished from the new
container. Two causes: only the first file was used, and the file list was read from
`docker ps`, which joins labels with commas — the list is itself comma-separated, so it
arrived truncated. Labels now come from `inspect`; every file is resolved, shown as a
chip, editable, validated together and passed to compose in order.
2. ✅ **A compose file changed on disk after you opened it was silently overwritten** (Dockhand,
or a second tab). The editor now hands back the checksum it was given; a mismatch is a
409 that says so, keeps your text, and offers to reload.
3. ✅ **Stacks on a machine without Docker answered 500.** An empty list.
4. ✅ **Every hub restart re-fired every firing alert** — and every deploy is a restart, so a
phone would have been told about the same failed service on each one. What is firing is
now kept in `state.json` and carried on, with its original start time.
5. ✅ **An alert whose rule was paused, deleted or narrowed just vanished**, leaving "firing" in
the history forever and no word to the phone. It is now recorded (and sent) as cleared,
with the reason: "rule paused", "rule deleted", "no longer watched".
6. ✅ **Nonsense rule values were accepted**: `null`, an object, `cpu.usage > ninety` (which can
never be true, and would never say so), `true` with `<`. Refused, each with its reason.
7. ✅ **No limit on rules** — 3000 were accepted, each checked on every machine every 15 s. 200.
8. ✅ **Logs of a container that no longer exists answered 500.** 404.
9. ✅ **Four skins had text under 3:1 contrast**: Nord's red; Catppuccin Latte's muted text and
green; Solarized Dark's body text, muted text and red; Solarized Light's body text,
muted text, accent and green. Each colour kept its hue and was moved to a readable value
(measured: all thirteen skins now pass, body text ≥ 7:1).
10. ✅ **Phone: the alert toggles (28 px) and the colour swatches (26 px) were under 30 px.**
11. ✅ **The public installer had fallen four builds behind the fleet** (v65 published, v69
running), so `curl … | sh` on an up-to-date machine "upgraded" it backwards. `make
deploy` now republishes, and the installer compares versions: an older build is called
older, defaults to No, and unattended runs refuse it without `HAMLAB_ALLOW_DOWNGRADE=1`.
## Checked and fine
A second stack action while one runs (409); `../` and encoded tricks in stack names;
shell metacharacters in container names, `since` and actions; `rm` / `kill` / `down` are
not actions; a read-only token is refused on every Docker, Stacks and Alerts endpoint,
directly and through the hub; container names with dots and underscores still get live
CPU and memory; a log at five lines a second follows with no gaps or duplicates; a
container stopped or removed while open updates itself; an alert rule's XSS name renders
as text; per-machine firing, the rail badge, webhook and ntfy delivery; terminals, Files,
widgets over container sensors, deep links, tab titles and machine colours all unchanged.
No native browser dialogs anywhere; no page errors on any of 100 page loads.
## Found by the alerts themselves
`core` has one real alert firing: **`zfs-import-cache.service` fails at every boot** — `core`
has the ZFS tools and no pool, so there is nothing to import. Harmless, but it will sit on
the rail as "1" until the unit is disabled or the rule is narrowed. Left alone.
## Quality of life — five chosen by Mitchell, ✅ built and verified 2026-09-19
1. ✅ **Stacks on `grid` and the Mac are editable from the hub.** Their files live on `core`
(Dockhand deploys over tcp); the hub resolves them and runs compose with `DOCKER_HOST`
pointed at the node. Verified on a throwaway stack: saved on `core`, recreated **on
`grid`**, same compose labels, nothing created on `core`, output followed on the page.
2. ✅ **A stack's `.env` is editable**, values masked until "Reveal and edit". A `.env` that
breaks the compose file (a required variable removed) is refused and the old one put back.
6. ✅ **A link to a container that no longer exists says so**, by name.
9. ✅ **"Update available" per image** — on request, remembered six hours. Verified with an
image whose tag had moved on (update available) beside a current one (up to date).
10. ✅ **What Deploy would change**: a line diff against what was last deployed from here;
after a deploy it says there is nothing left to change. `.env` diffs name the key and
never show a value.
Not chosen, still open: alert acknowledge/snooze; quiet hours and per-rule channels; push
straight from Ham Lab; a type hint in the alert editor; switching off the nodes' own alert
engines.
# Bug bash 3 — 2026-09-20: new stacks, new containers, images, volumes, the HUD
Started from Mitchell's report — the New stack dialog is see-through in HUD, its text
mixing with the page behind. Same method as before: a scratch hub with the real fleet behind
it, headless Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four
machines, plus a measurement **in every skin** of everything that floats over the page
(dialogs, the question box, the widget picker, menus, the phone drawer: background alpha
must be ≥ 0.9). Then scripted use of what is new on throwaway `hl-` stacks, containers,
images and volumes: the New stack / New container dialog's edges, stacks that are not
deployed, a deploy that fails, a container name that is taken, deletes beside other
people's files, a two-file stack, create → deploy → restart → stop → delete **on the Mac
and on `grid` through the hub**, pulls on the hub and on a node, removal guards, a
container stopped on purpose, a read-only token and hostile input on every new endpoint,
and a regression of Escape everywhere, terminals, Files, sensors, alerts and deep links.
## Bugs — all ✅ fixed and verified by the script that found them
1. ✅ **HUD: every dialog was see-through.** The skin's "glass" rule (3 % cyan) was written
for tiles and cards and also caught `.modal` — the New stack dialog, the widget picker,
the Power dialog, the token dialog, the rule editor and every question. Anything that
floats over the page is now solid in HUD (hairline border and glow kept) over a darker
backdrop. Measured in all thirteen skins: nothing that floats is under 0.9 alpha.
2. ✅ **HUD: the phone's slide-out menu was 92 % opaque**, the page showing through it. Solid.
3. ✅ **Deploy, Update, Restart, Stop and Down all failed on the Mac's stack**
(`peaches-and-plums`) — and on any stack held on the hub that was deployed from the hub's
own shell rather than from inside Dockhand. The command was built as
`'DOCKER_HOST=tcp://…' docker compose …`; quoted, `sh` reads that not as a variable but
as the name of a program to run ("not found", exit 127). Now passed through `env`.
Verified by a whole life on the Mac and on `grid` from the hub: create, deploy (running
there, nothing on `core`), restart, stop, delete.
4. ✅ **A container stopped on purpose was reported as down** when it has no restart policy
and doesn't handle SIGTERM: `docker stop` then ends it with 137 (or 143, or 130), a
non-zero code, which read as a crash — "Container down" would have fired for a
deliberate stop. Those three codes now mean "a person", unless the kernel did the
killing for want of memory (`OOMKilled`). A real crash (exit 3) is still down.
5. ✅ **A stack with no containers offered Restart, Stop and Update**, and Restart
"succeeded" having done nothing. Offered only when there is something to act on; the
API answers 409 "isn't deployed: there is nothing to restart".
6. ✅ **One stray tap outside the New stack dialog threw away everything typed** — on a phone,
a compose file's worth. So did Escape. With anything typed, all three ways out (tap
outside, Escape, Cancel) ask first; an untouched dialog still just closes.
7. ✅ **Compose → Form → Compose silently replaced hand-edited compose text** with what the
form says. The form now writes the box only when the form has changed since it last
did, and asks before replacing text edited by hand.
8. ✅ **A question asked from inside a dialog opened underneath it** (same layer, earlier in
the page), so it could not be answered — found the moment the New stack dialog started
asking. Questions now sit above whatever asked them.
9. ✅ **Escape on a question also reached the dialog beneath it**: one key cancelled the
question *and* closed (or re-asked from) the dialog under it. Escape now answers the
question and goes no further; the next Escape is the dialog's. Checked on every dialog
and menu: widget picker, machine menu, rule editor, Power, New stack, and that Escape on
"Delete the rule?" deletes nothing.
10. ✅ **Phone: the bottom of the New container dialog could not be reached** (Mitchell, from
his iPhone). A dialog is a column whose parts were allowed to shrink, so instead of the
dialog scrolling, the one part with its own scrollbar — the compose preview — was
squeezed to 22 px and everything "fitted". And `92vh` on iOS is taller than the screen
with the address bar gone, so the top sat under the clock. Parts now keep their height
and the dialog scrolls inside itself; it is never taller than the space between the
status bar and the home indicator (safe-area insets, no `vh`); Create / Cancel — and
any error — stay pinned at the bottom while the form scrolls. Applies to every dialog;
measured at 390×844, 375×667 and sideways, and the picker, rule editor, token and Power
dialogs rechecked.
## Checked and fine
All 13 pages × four machines × both sizes: no page errors, failed requests, sideways
scroll or tap targets under 30 px. A port of `not-a-port` is refused by compose before
anything is written, and leaves no folder. Environment values with quotes, `#`, `$` and
nothing after the `=` arrive in the container exactly as typed. A deploy whose image
doesn't exist shows the registry's error and is **not** remembered as "last deployed". A
container name already taken fails the deploy and says why. Delete moves only the stack's
own compose files and `.env` (kept `600` in the trash) and leaves a `data/` folder and
notes beside them alone; both files of a two-file stack go to the trash; the wrong name,
or none, deletes nothing. A `.env` value never appears on the stack page and stays masked
until asked for. A second stack of the same name is 409; an upper-case name is 400. A pull
of an image that doesn't exist ends as a failed task; a pull on `grid` from the hub lands
on `grid` and its output is followed. A volume or image in use can't be removed; a running
container can't be removed (409); a volume needs its name twice. A read-only token gets
401 from all fourteen new write paths, through the hub as well. `--force`, `-f`,
`--all-tags`, `..%2F..` and a home outside the known ones are refused; a 1.2 MB compose
file is refused. Shaped, fleet and streamed sensor queries, `/metrics`, the ten alert
rules, terminal typing, the file editor's discard question and service deep links are
unchanged. No native browser dialogs anywhere.
## Quality of life — four chosen by Mitchell, ✅ built and verified 2026-09-20
1. ✅ **"Where" fits on a phone**: `…/stacks/core · 12 stacks` instead of the whole path (which
is still written out underneath). Two homes whose last two folders match fall back to the
full path. Measured at 390 px: no option is wider than the box.
2. ✅ **How a task ended is a badge**: green `finished`, red `failed · exit 1` with the box
outlined in red, grey `cancelled`. Verified on a deploy of an image that doesn't exist.
5. ✅ **Duplicate…** on a stack: the New stack dialog opened on a copy of its compose file,
named `<name>-copy`. The original's `.env` is copied **on the machine** (`env_from`), mode
600, its values never sent to the browser — checked: the secret is nowhere in the page.
Anything that would collide with what is running is named before you press Create
("a container called X already exists; port 18222 is already published by X") — for
every new stack, not only duplicates — and clears as you fix it. Disabled while the
original has unsaved edits. Verified: copy deployed beside the original, which never
stopped.
6. ✅ **The form has Labels and Network.** Labels one per line (`$`, quotes and spaces arrive
intact). Network: its own, the machine's (`host`, which turns Ports off), or **join** one
that already exists — written as an `external` network so compose doesn't make a second
one. Verified on the container: both labels, attached to the chosen network only.
Found while building them, ✅ fixed:
- **Cancel on a stack task cancelled nothing.** It killed the `sh` that ran compose, not
compose; and as compose still held the output pipe, the task stayed "running" and the
stack "busy" until compose finished by itself — for a deploy waiting on a health check,
minutes. Tasks now run in their own process group and the whole group is stopped (TERM,
then KILL after 4 s); for stacks deployed inside Dockhand, where the process is not the
task's child at all, the cancel goes into the container after it. Verified both ways:
ended in ~100 ms, compose gone, the stack free for the next action.
- **Phone: a stack's task output was squeezed to a line** under the buttons (the same fault
as the dialog: a scrolling column whose parts were allowed to shrink), which also put
Cancel out of reach. Detail panes keep their parts' heights; services, containers, stacks
and cron rechecked on both sizes.
- The Duplicate dialog and the stack page under it shared a React key, so the dialog
multiplied. An undeployed stack's dot was green ("0 of 0 running"); now neutral.
# Bug bash 4 — 2026-09-21: the Top tab, and the refresh rate it didn't follow
Started from Mitchell's first look at Top on his own machine (Firefox on Windows, a window
1392 CSS px wide — none of which the tab had been drawn for): "Command" printed over
"Threads", the rate in Settings ignored, the symbol beside *logout* a box of hex digits;
and make it follow the skins. Method as before, plus two things new: **Firefox** (Playwright's
build; the GTK libraries it wants are unpacked from their `.deb`s into
`~/.cache/ms-playwright/fflibs` and given to it as `LD_LIBRARY_PATH`, with
`PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1` — nothing is installed on `core`), and a layout
checker that measures instead of looking — every heading against its neighbour and its own
text, every row against its box, every border label, at eight window sizes from 900 to 1920.
Run first against the live v83 it reproduced the screenshot to the pixel ("the Command column
is 11px wide" at 1392); run after, nothing, in both browsers.
## Bugs — all ✅ fixed and verified by the script that found them
1. ✅ **Between about 1280 and 1500 px the Command column collapsed to a letter** and its
heading ran under "Threads:" (itself 4 px too narrow for its own text at every size). The
table's fixed columns were sized for one window. The process box now measures *itself*
(a container query — it is a third of a wide window and all of a narrow one, so the
window's width says nothing) and drops the curve, then Threads, then Command, as it
narrows; the box also gets half the width rather than 44 %, and no heading can spill
into the next. Stacked layout now starts at 1280, not 1100.
2. ✅ **The refresh rate reached a machine only through its Dashboard page.** The sampler on
each machine was told the rate by an open Dashboard and nothing else — so with the rate
at 250 ms, `grid`, `wyse` and the Mac were still sampling at 1 s (measured, live), and
Top, Processes, Disks and every widget of theirs "ignored" the setting until somebody
opened that machine's Dashboard. The rate is now the fleet's: saved with the UI prefs on
the hub, which sets its own sampler and passes it to every machine (`PUT
/api/prefs/sampler`, remembered in each `state.json`), and says it again at start-up for
a node that restarted or joined since. Verified: Settings 2 s → 250 ms moves the sampler
with no Dashboard visit.
3. ✅ **Top read the process table at most once a second whatever the rate** — a clamp of
my own, exactly what `REFRESH` promises not to do. It reads at the rate: 20 reads in 5 s
at 250 ms, 10 at 500 ms. (59 frames a second with 390 rows at 250 ms, no stalls, both
browsers — the rows are only redrawn when the table is read.)
4. ✅ **Top's `− rate +` changed the rate for half a minute.** It set the browser's copy
and never saved it, so the next sync with the hub put it back. It saves — which is also
what moves the samplers — survives a reload, and each end is disabled at the limit.
5. ✅ **A viewer whose clock is wrong froze the curves.** New points were timed against the
machine's timestamps: a browser clock 7 minutes slow waited 7 minutes for its next point
(Top *and* the Dashboard); 7 fast, and it added one per tick. Timed on the browser's own
clock now. Verified with the page's clock shifted both ways: 5 points in 10 s at a 2 s
stride, both pages, both browsers.
6. ✅ **Every other point was being skipped.** "Has a stride passed?" was asked with `>=`
of a timer that fires a few ms early half the time — so at a 2 s rate and a 2 s stride a
curve gained 3 points in 10 s, not 5, and its time axis was quietly wrong. The same
question on the server kept every bucket one sample long (14 s points at a 12 s stride).
Both now allow half a tick of slack and keep to the stride's own beat; the server times
from the last point kept, so a stalled or sleeping machine just keeps the next sample it
gets. Unit-tested with jittered samples: 240 samples → exactly 40 points, 12 000 ± 100 ms
apart, each the mean of its own six.
7. ✅ **The power symbol was a typed `⏻`** (U+23FB, Unicode 9) — not in Windows' fonts, so
Firefox there drew the missing-glyph box. Drawn as an SVG like the other icons. Every
other symbol the console types was checked against the same risk: all older.
8. ✅ **A handful of samples was stretched into a bar chart.** A machine ten seconds into
its history filled the whole CPU box with five fat columns. The box is now always the
whole history window — a young curve draws in from the right, as btop's does, and a full
one fills it edge to edge, at any window (checked at 10 min and at 1 min).
9. ✅ **Idle curves vanished**: a core at 10 % rounded to no dots at all in its mini graph.
Anything above zero lights the bottom row, which is also btop's rule.
10. ✅ **Found by the deploy of fix 2 itself: telling the machines once is not enough.** The hub
restarted first and told three nodes that did not yet have the endpoint (404); they then
updated and carried on at 1 s. The same would happen to any node that is off or
restarting when a setting changes. The hub now *keeps* the fleet in step: fifteen seconds
after it starts and every two minutes after, it asks each machine for its sampling rate
and history window and corrects only what differs (logged as `fleet: grid had ms=1000,
now 250`).
11. ✅ **The net graph redrew its own past every second** (Mitchell, two screenshots a second
apart: one tall spike in the first, a comb of medium ones in the second). That box is
about 75 dot-columns wide and holds 300 points, and each column drew *the nearest point*
— one in four, and a different one in four each time the curve advanced, so spikes came
and went and the whole shape changed while the scale stayed put. The CPU graph did the
same on a phone. Now a column draws the **peak** of all the points in it, and columns are
cut from the data, not from the box (`web/src/graphcols.ts`): a finished column never
changes, the graph only slides left by whole columns. Measured by reading every column's
height off the canvas once a second under bursty traffic: on the live build the past
changed shape in 18 of 47 frame pairs; after, 0 of 46 — and 0 of 45 on a young history.
`npm test` (now part of `make test`) scrolls a spiky curve through both rules: nearest-
point shows its three spikes as "900,500", then "700", then none; the new rule shows all
three, at their heights, on every frame. The scale label now covers what is drawn (the
window), not everything kept.
## Skins
✅ Top now wears the skin. The ground is the skin's terminal background and every curve and
meter is built from its sixteen terminal colours (chosen to be read on exactly that ground),
the rest from the usual variables — so the dark skins look like btop and the light ones like
btop on paper, and a skin arriving from another device repaints the page in place. Measured
in all thirteen: text ≥ 4.5 : 1, everything else ≥ 3 : 1, dots ≥ 2.2 : 1. Two failed —
process names in Latte (2.75) and Solarized Light (2.62), where a terminal green is too
pale for small type — and use `--ok`, the green each skin was already measured readable
with.
## Checked and fine
Sort by pid, name, mem and cpu, and reverse; filter by name, command, user and pid, with the
count showing matched/total and an empty result saying so; a selection that holds while the
table is re-read under it; terminate, on a throwaway `sleep`, asks by name and pid, ends it,
and the footer stops offering a process that is gone. Sampler values outside 100 ms–60 s are
refused, and a rate slower than the collector's 10 s samples at 10 s. All 14 pages at his
window size and on a phone, in both browsers: no errors, no overflow.
# Bug bash 4, later — 2026-09-21: the SMART table outside its card
Mitchell, from the Disks page of `hamtv` with SMART open: the attribute table's rows ran out
through the right-hand side of the card they belong to, the rules and the Thresh column
drawn over the page behind.
12. ✅ **A table cannot be narrower than its widest unbreakable cell, and one of `hamtv`'s
was 130px wide.** `.num` sets `white-space: nowrap`, and a SMART raw value is not always a
number — the temperature attribute reads `34 (Min/Max 28/39)`. That one cell set the
table's minimum width above the card's, so it spilled. Three changes, in this order,
because the first two each looked worse than the bug in their own way:
- the raw column wraps — but `overflow-wrap: anywhere` broke `4295163906` across two
lines as "429516 / 3906", which reads as two numbers. It wraps at spaces only now;
- a disk card is given room for six columns of SMART (`minmax(420px, 1fr)` rather than
the shared 320px), and the cells' side padding goes 6px → 4px, which together is the
~40px it was short by;
- if some drive ever reports a raw value long enough to still not fit, the table scrolls
inside its card rather than out of it.
Verified on all four machines at six widths from 1721px down to a phone, in Chromium and
Firefox: nothing leaves its card, no number is broken across lines, and the fallback
scrollbar is never reached. The same measurement was run over every other page's cards
and boxes — Dashboard, Top, Docker, Alerts, Machines and the rest — and found nothing.
# Bug bash 5 — 2026-09-21: navigation, and the fleet setting nobody had set
A sweep of the whole console after a week of changes, not of any one page: every page on every
machine at four widths in **Chromium and Firefox**, then the everyday flows, then the URL.
Measured rather than looked at — a heading against its neighbour and its own text, a box
against what it holds, a request against the page that should have made it.
## Bugs — all ✅ fixed and verified by the script that found them
1. ✅ **A link to a page of the hub reloaded onto whichever machine you last had open.** The hub
is `null` internally, so `parseHash()?.machine ?? sessionStorage…` read straight past it to
the remembered machine: open `#/hub/docker/stack:gitea` after looking at `grid`, reload, and
you were on `#/grid/docker`. The URL wins when it has an opinion, and "hub" is an opinion;
the remembered machine is for a bare address only. Every link shape now survives a reload,
and a bare address still lands where you left off.
2. ✅ **From Settings, Machines or Alerts, a link to a machine's page arrived at its Dashboard.**
Those three pages only exist on the hub, so moving to a machine bounces off them — but the
bounce was decided from inside the machine change, which read the page being *left*, and
landed after the URL's own page had been set. It is judged now once the machine and the page
have both settled. `#/hub/settings` → `#/grid/disks` arrives at Disks; `#/grid/settings`
still falls back to that machine's Dashboard.
3. ✅ **A page being navigated away from briefly adopted the next page's selection**, and asked
about it: leaving `#/grid/services/ssh.service` for `#/hub/docker/stack:gitea`, Services —
still mounted for a moment — opened a log socket for a *service* called `stack:gitea`, on
`grid`. Firefox says so in the console; Chromium swallows it. A page now reads the URL's
selection only while the URL still points at that page. Checked over every hop between the
five pages that keep a selection in the URL, in both browsers.
4. ✅ **The hub only held its machines to a sampling rate if somebody had been into Settings.**
The re-assert loop read the rate out of the saved UI prefs, which are written on change and
not before — so on a hub where the rate had never been touched, a node left on a different
rate was never pulled back. It compares against what the hub itself is running, which always
exists. Verified by setting a node behind the hub's back: corrected within seconds.
5. ✅ **"Everything is up to date." sat in a 20px sliver of a card**, flush to its border, where
the package list it shares the card with is meant to be.
6. ✅ **A terminal of the hub's was asked for on a node.** Found in Firefox against the live
console, which says out loud what Chromium swallows: `wss://…/api/nodes/grid/terminals/
adfdeecb/attach`, where `adfdeecb` was one of the hub's — `grid` had no terminals at all. A
pane builds its address when it connects, from whichever machine the console is pointed at
*now*; switch machine while one is reconnecting and the reconnect goes to the new one. The
address is fixed when the pane opens. The same was true of a service's log socket, and is
fixed with it. Verified by walking the console around the fleet with a terminal open and
checking every socket against the machine that actually owns what it asked for.
## Checked and fine
Machine titles on every page and against an offline or never-reached machine; the colour a
machine wears stays with the name it was added under, not its hostname. Files: create, open,
edit, save, rename, delete. Services with its unit and log, cron jobs, containers and stacks,
alert rules edited and put back, the terminal, the widget gallery. Login, a wrong password, a
session cleared under an open page, logging back in onto the page you were on, and logout
leaving nothing behind. The sampler, window and UI prefs endpoints under nonsense and hostile
input, and a read-only token refused by all of them. The public demo page and installer. The
Top tab on all four machines — including `grid`'s sixteen cores and thirteen filesystems, which
scroll inside their boxes — its rate control, its filter, and three skins repainting it.
`hamtv`'s eMMC really is called `hB8aP>`; that is what the device reports.
Firefox earns its place in this: three of the six above were invisible in Chromium, which
swallows a failed WebSocket where Firefox writes it to the console. One thing it reports is
*not* a fault — a page torn down by navigation while a terminal's socket is still shaking
hands logs "can't establish a connection". Sitting still on a page produces none, the address
is right, and the terminal belongs to the machine asked; `sockets.cjs` checks that ownership
rather than the noise.
CLAUDE.md
+13
View File
@@ -0,0 +1,13 @@
# Ham Lab (`hamlab`) — the web management console app
This repo is the **app** only: Go daemon + React/xterm UI, deployed as `hamlab.service`
on `core`. Product scope, decisions and roadmap are in `PLAN.md`.
Everything about *how to work* — persona, response style, write-back rule, hosts,
credentials, pending items — lives in the homelab repo **Hamlet** at
`/home/core/projects/hamlet`, and applies here unchanged:
@/home/core/projects/hamlet/CLAUDE.md
Ham Lab facts that belong to the lab (where it runs, ingress, vault item) go in
`hamlet/docs/hosts.md` and `hamlet/docs/pending.md`, not here.
Makefile
+56 -8
View File
@@ -1,26 +1,74 @@
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
# Commit first, then deploy: the version is the commit the binary was built from.
VERSION ?= $(shell git describe --tags --always 2>/dev/null || echo dev)
GO ?= go
PREFIX ?= /usr/local/bin
.PHONY: all web build dev deploy clean
# every architecture a release carries; arm is ARMv6 hard-float, so it covers every Raspberry Pi
ARCHES ?= amd64 arm64 arm 386 riscv64
LDFLAGS = -s -w -X main.version=$(VERSION)
# static, no cgo: the same file runs on glibc, musl (Alpine) and anything else with a Linux kernel
GOBUILD = CGO_ENABLED=0 GOOS=linux GOARM=6 $(GO) build -trimpath -ldflags "$(LDFLAGS)"
.PHONY: all web build go dev cross release gitea-release deploy push-nodes push-nodes-force test clean
all: build
web:
cd web && npm ci --no-audit --no-fund && npm run build
build: web
$(GO) build -trimpath -ldflags "-s -w -X main.version=$(VERSION)" -o bin/hamlab ./cmd/hamlab
build: web go
# backend only (assumes web/dist already built)
go:
$(GO) build -ldflags "-X main.version=$(VERSION)" -o bin/hamlab ./cmd/hamlab
$(GOBUILD) -o bin/hamlab ./cmd/hamlab
# one other architecture: make build-arm64, build-arm, build-386, build-riscv64, build-amd64
build-%: web
GOARCH=$* $(GOBUILD) -o bin/hamlab-linux-$* ./cmd/hamlab
dev:
cd web && npm run dev
deploy: build
./deploy/install.sh
# siblings for every other architecture; installed beside the binary, and the hub pushes
# each managed machine the one that matches it
cross:
@for a in $(filter-out $(shell $(GO) env GOARCH),$(ARCHES)); do \
echo "build linux/$$a"; GOARCH=$$a $(GOBUILD) -o bin/hamlab-linux-$$a ./cmd/hamlab || exit 1; done
# dist/: one static binary per architecture, install.sh, SHA256SUMS. Serve the directory
# anywhere (or attach it to a release) and install.sh does the rest.
release: web
rm -rf dist && mkdir -p dist
@for a in $(ARCHES); do \
echo "build linux/$$a"; GOARCH=$$a $(GOBUILD) -o dist/hamlab-linux-$$a ./cmd/hamlab || exit 1; done
sed 's|@VERSION@|$(VERSION)|' install.sh > dist/install.sh && chmod +x dist/install.sh
echo $(VERSION) > dist/VERSION
cd dist && sha256sum hamlab-linux-* install.sh > SHA256SUMS
test:
$(GO) vet ./... && $(GO) test ./...
sh -n install.sh
cd web && npm test --silent
# installs here (keeping the config, user and password), then updates every managed machine
# Offer this build to every install, wherever it is: a release on Gitea, which is what
# Settings -> Updates reads. Deliberate, and not part of deploy — deploy pushes to the
# machines this hub owns, publishing offers it to everyone else.
gitea-release: release
./scripts/gitea-release.sh $(RELEASE_ARGS)
deploy: build cross
./bin/hamlab install --prefix $(PREFIX)
@sleep 1; $(PREFIX)/hamlab push-nodes || true
@echo "deployed $(VERSION); 'make gitea-release' publishes it for everyone else"
# push the installed binary to every managed machine (what deploy does last)
push-nodes:
$(PREFIX)/hamlab push-nodes
# same, interrupting running tasks and attached terminals on the nodes
push-nodes-force:
$(PREFIX)/hamlab push-nodes --force
clean:
rm -rf bin internal/web/dist web/dist
rm -rf bin dist internal/web/dist web/dist
PLAN.md
+642 -27
View File
@@ -18,6 +18,8 @@ stub tab or a roadmap item.
| Data | `/home/core/apps/hamlab/` — `config.yaml`, `state.db` (SQLite) | `apps/` is the nightly rsync target to `grid`, so it's backed up for free. |
| Ingress | NPM proxy host `hamlab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
| Repo | Gitea `mitchell/hamlab`, private, `main` | Created 2026-09-17. |
| Guest logins | A **guest** is an account like any other, with `guest: true` and an expiry: `login`, sessions and prefs need no special case, and **it may do everything an account may do**. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. | Built 2026-09-27. A read-only version was built first and **cut the same day**: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is *an account*: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to. |
| Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
## Layout
@@ -25,22 +27,22 @@ stub tab or a roadmap item.
┌──────────────────────────────────────────────────────────────┐
│ header: Ham Lab · core · load 0.42 · ● connected · [logout] │
├──────────────┬───────────────────────────────────────────────┤
│ ▾ Terminal │ │
│ shell 1 ● │ │
│ backups │ main content (xterm) │
│ + New │ │
│ ▸ Services │ │
│ ▸ Cron │ │
│ ▸ ZFS │ │
│ ▸ Dashboard │ │
│ Dashboard │ shell 1 ● │ backups │ + │
│ Processes ├───────────────────────────────────────────────┤
│ Terminal │ │
│ Services │ main content (xterm) │
│ Cron │ │
│ … │ │
├──────────────┴───────────────────────────────────────────────┤
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev │
└──────────────────────────────────────────────────────────────┘
```
- Left rail: collapsible groups, one expanded at a time. Active leaf highlighted.
- Terminal group: one leaf per session, `● ` when a process is running, "+ New
Terminal" pinned at the bottom. Right-click / long-press → rename, close.
- Left rail: flat list of pages, active one highlighted. No nested leaves.
- Terminal page: tab strip across the top of the pane, one tab per session, `● ` when
a process is running, `+` pinned at the right edge. Active tab carries rename and
close; double-click renames, middle-click closes. (Until 2026-09-18 the sessions
were leaves under a collapsible Terminal group in the rail.)
- Non-MVP groups render as disabled leaves with "coming soon" — the rail shape is
the product; stubs keep it honest about scope.
- **Mobile**: rail becomes a slide-in drawer; xterm fits the viewport; a sticky key
@@ -119,31 +121,644 @@ something long in one, switch away and back, refresh — it's all still there.
## Roadmap (not MVP — notes only)
- **Updates tab** — **shipped 2026-09-17** (`internal/updates`, `internal/tasks`,
`Updates.tsx`): upgradable packages with current → candidate, suite, security badge
and description; index age, last upgrade (from `/var/log/apt/history.log`),
unattended-upgrades on/off, reboot-required banner with the packages that caused it;
Refresh index, Upgrade all (dist-upgrade), Upgrade selected; changelog per package on
demand. Runs go through the new **tasks** package: background command, output kept,
followed live over `/api/tasks/{id}/stream`, cancellable, listed at `/api/tasks` (the
seed of a Task History tab). apt is fully supported; dnf/pacman list only.
- **Power** — header ⏻ button (amber when a reboot is pending), dialog that requires the
hostname typed back, then `systemctl reboot` / `poweroff`. `/api/me` carries
`reboot_needed` so the header knows.
- **tmux-backed sessions** so shells survive a daemon restart / redeploy. Would spawn
`tmux new -A -s hamlab-<id>`; needs `mouse on` and status-bar handling in xterm.
- **Services tab** — `systemctl list-units`, per-unit page: status, last 200 journal
lines (live-tail via WS), start/stop/restart/enable buttons, unit file viewer.
- **Cron tab** — parse `/etc/crontab`, `/etc/cron.d/*`, user crontabs, and systemd
timers into one table with next-run times; edit via `crontab -` round-trip.
- **ZFS tab** — pools, datasets (`used/avail/refer/usedbysnapshots`), snapshot list,
scrub status, and the `freeing` counter. `grid` is the target, which implies
multi-host.
- **Dashboard** — CPU/mem/load/net, disks and SMART (`smartctl -j`), temps and fans
(`sensors -j`), UPS from `pwrstat`. Ham Lab collects on a timer into a SQLite
ring (e.g. 10 s resolution, 7 days) and draws sparklines. This is where the
"collects its own data" requirement lands; no Prometheus dependency.
- **Multi-host** — same binary as a headless *agent* on `grid` / Mac mini /
`wyse-5070-1`, `core`'s instance is the UI and fans out over mTLS. The rail grows a
host level at the top, exactly like Proxmox's datacenter → node tree. Design the
MVP URL space so `host` can be inserted later without breaking clients.
- **Auth hardening** — WebAuthn passkeys (phone Face ID), TOTP fallback, session
list + revoke, optional NPM access list restricting to WireGuard/LAN.
- **Files** — browse/upload/download/edit under `/`, with a sudo toggle.
- **Cron tab** — **shipped 2026-09-17** (`internal/cron`, `Cron.tsx`): one list built
from `/etc/crontab`, `/etc/cron.d/*`, every user crontab (via sudo), the scripts in
the `run-parts` directories those drive, plus systemd timers. Next run computed by an
in-house cron-expression evaluator (tested: ranges, steps, names, dom-OR-dow rule,
`@` aliases); last run from the journal's `(user) CMD (command)` lines (cron logs via
syslog to journald; reading needs sudo/adm). Commented-out jobs show as disabled;
job-shaped lines inside comment blocks (Debian's header examples) are ignored. Per
job: Run now (background, as the job's user, `sh -c` with crontab `%` rules, output
captured), Enable/Disable (comments the line in place, refuses if the file changed
underneath), Edit source (textarea; validated line-by-line; user crontabs installed
with `crontab -`, system files written via `sudo tee`), Cron log, and Output file
when the command redirects to one. Timers: Run now (starts the activated service),
start/stop/enable/disable, live logs of the service via the Services tab's tail.
- **Disks tab** — **shipped 2026-09-17** (`internal/storage`, `Disks.tsx`), replacing
the planned ZFS tab: block devices from `lsblk -J` (kind, model, size, temperature
from the device's hwmon node, partitions with filesystem usage, live read/write/busy
from the metrics sampler); SMART via `smartctl -j` when smartmontools is present (one
button installs it as a task otherwise): health, temperature, power-on time, self-test
status, ATA attribute table with the failure-predicting IDs flagged, NVMe endurance and
error counters, short/long self-tests. Spinning disks are read with `-n standby` so
the tab never wakes a sleeping drive; "Wake and read" is explicit. ZFS section when
the tools and module are there: pools with health, capacity, fragmentation, last scan
and vdev tree; scrub start/pause/stop; datasets tree with used/refer/in-snapshots/
snapshot count/compressratio/mount; per-dataset snapshot list and "Snapshot now".
Verified against `grid` (raidz2 `tank`, 10 datasets, 515 snapshots, 8 WD40EZAZ + 970
EVO Plus SMART). Snapshot destroy and dataset property editing are deliberately not
there yet.
- **Dashboard** — **first cut shipped 2026-09-17** (`internal/metrics`, `Dashboard.tsx`):
CPU (total + per core), memory/swap, load, uptime, hwmon temps and fans read straight
from `/sys` (no lm-sensors), per-interface network rates, per-disk read/write/busy
from `/proc/diskstats`, filesystem usage, UPS via `sudo -n pwrstat -status`. Sampled
every 2 s into an in-memory ring of 10 min; the UI (the **Dashboard** tab
since 2026-09-19) reads it all through the sensor API — see Sensors below. **Customizable**: a
Customize mode reorders and hides modules (CPU, Memory, Load, Uptime, Temperatures,
Fans, Network, Disks, Filesystems, UPS) and hides individual rows (an interface, a
mount, a disk, a sensor); the layout is stored server-side in `state.json` via
`GET/PUT /api/prefs/dashboard` so every device sees the same dashboard. The update
interval (100 ms – 2 s) is part of the layout and drives both the browser's polling and
the daemon's sampling clock; the chart window is always 300 samples, so 30 s at 100 ms
and 10 min at 2 s. Still to do: persistent
history (SQLite ring, e.g. 10 s × 7 days), SMART (`smartctl -j`), longer time ranges.
- **Updating from the Updates tab** — **shipped 2026-09-26**: the "what is out of date" list
was read-only, so seeing a newer image and doing something about it were different places.
Each row now has **Update**, and the header has **Update all**. Updating is per stack —
that is the unit compose deals in — so a row acts on whichever stacks use that image, and
several stacks run through a queue one at a time rather than recreating half the machine at
once, each with its task output on screen. A row that succeeds says "updated just now"
instead of re-checking, since a check is a registry request.
- **Recreate, and the Updates tab at full width** (2026-09-26): a stack's actions gained
**Recreate** (`up -d --force-recreate`), the only way out of a container Docker refuses to
start — one pinned to a network that has since been recreated answers "network … not
found" to Deploy and Update alike, because as far as compose is concerned nothing about it
changed. Found the hard way when a SiYuan update landed in exactly that state. The Updates
tab also stopped rendering as a list-and-detail pair: it is a page, not a list, so on a
wide window it now sits under the tab strip at full width instead of being squeezed into
the right-hand half beside an empty column.
- **Cron in English** (`crontext.ts`, 2026-09-26): schedules read "daily at 3:15am" rather
than `15 3 * * *`, with the expression kept beside it. Handles lists, ranges, steps, day
and month names, `@daily` and friends, weekdays/weekends, midnight and noon — and declines
to describe anything it is not sure of, because a confident wrong sentence is worse than
five fields. The Jobs tab had its own smaller version of this; it now uses this one.
- **Hardware tab** — **shipped 2026-09-25** (`internal/hardware`, `internal/api/hardware.go`,
`Hardware.tsx`): what the machine is physically made of, per machine like every other page.
Four headline tiles (processor, memory, storage, graphics) over cards for each: the
processor with its cores, clock range, socket, caches, microcode and the handful of feature
flags anyone asks about; **every memory slot** with size, type, form factor, rated and
running speed, maker, part number, serial, rank and voltage, and the empty ones marked
empty; each drive with model, size, bus, firmware and serial; each display adapter with its
driver, PCI address and video memory; and the board itself — model, firmware and date,
serial, kernel. Read from what Debian already has (dmidecode for what only the firmware
knows, lscpu, lspci, sysfs) and cached for ten minutes, since none of it changes under a
running kernel; "Re-read" is there for the one thing that does, a drive plugged in.
Every reader may come back empty and the page says what it could not read and why: a
machine with no dmidecode, no root, or no PCI bus still gets a page. Verified on `core`
(SMBIOS, Intel graphics) and on the Mac mini (Asahi: no SMBIOS memory table, a
device-tree GPU, heterogeneous cores — where lscpu's "cores per socket" counts one
cluster, so without SMT the thread count is the honest core count).
- **Sensors** — **shipped 2026-09-19** (`internal/sensors`, `internal/metrics/sensors.go`,
`internal/api/status_sensors.go`, `web/src/sensors.ts`, `widgets.tsx`, `Embed.tsx`;
reference in `SENSORS.md`): everything measured is a sensor with a stable dotted ID;
catalog, single value, history, a **shaped query** (the client sends the object it wants
and gets it back filled in — GraphQL's idea without GraphQL, since sensors differ per
machine and a static schema would collapse to `sensor(id:)`), the same query as a live
SSE stream, Prometheus `/metrics`, read-only `sensors:read` tokens (never full tokens in
a URL), status sensors (services, updates, ZFS, SMART), a widget library, custom widgets
on the Dashboard tab (briefly renamed "Monitors" 2026-09-19, back to "Dashboard"
2026-09-20 at Mitchell's request; the view id and URL were always `dashboard`), and a
frameable `/embed/widget` for Glance and friends. (An external demo page was deployed at
`hamlab-demo.bigham.dev` and linked from the Widgets tab 2026-09-19; **both removed
2026-09-23** at Mitchell's request — the Widgets tab shows the same thing against the
machine you are signed in to, so the second copy earned nothing and cost a public
read-only token, which was revoked with it. `examples/external-demo/` stays as an example
to read.) The stock Dashboard modules and the Disks
tab read the sensor API too (`sampleAdapter.ts`); `/api/metrics` is retired.
Also per-service and per-container sensors (memory and CPU from cgroups), tokens limited
to ID patterns, and `/api/fleet/sensors/query` for the whole fleet in one request.
- **Alerts** — **shipped 2026-09-19** (`internal/api/alerts.go`, `Alerts.tsx`): rules over
sensors (`container.*.running == false for 120 s`), evaluated by the hub every 15 s for
itself and every managed machine in one shaped query each; `hamlab.reachable` is supplied
by the hub so "a machine stopped answering" is a rule like any other. Ten defaults
(machine offline, container down/unhealthy, failed service, disk full, pool unhealthy,
SMART failing, UPS on battery, CPU hot, memory). Firing and resolved are events: kept
(last 200, in `state.json`), shown on the hub-only Alerts tab with a count on the rail,
and POSTed to channels — **ntfy** (title/priority headers) or a **webhook** (JSON; Home
Assistant, Slack). The rule editor shows what a selector matches across the fleet as
you type. Not done: quiet hours, per-rule channels, Web Push straight to the PWA.
- **Docker tab** — **shipped 2026-09-19** (`internal/api/docker.go`, `Docker.tsx`):
containers grouped by compose stack with live memory and CPU from the container
sensors, start / stop / restart (asked first), logs that follow (polled with `--since`,
no held connection), inspect (env *names* only), and images / volumes / networks
tables. Works on every machine through the hub; `#/grid/docker/immich_server` links to
one. **Stacks** (`internal/api/stacks.go`): every compose project, found from its
containers' labels, with its file to edit and Deploy (`up -d`), Update (pull, then
deploy), Restart and Stop as followed tasks. The file is looked for where the label says
*and through every container's mounts*, because an orchestrator that runs compose inside
its own container (Dockhand here; Dockge, Portainer) records a path in its own
filesystem; compose is then run **inside that container**, so `.env`, relative paths
and the compose labels stay exactly as deployed (verified: labels identical after a
redeploy). Save validates with `compose config` first, keeps the previous file as
`<file>.hamlab-bak`, and never deploys by itself. A stack deployed from another host
(Dockhand on `core` driving `grid` over tcp) has no file on that machine, so the node
alone lists it read-only — but **the hub fills that in** (`hubStacks`): asked about a
node, it resolves the files on itself and runs compose here with `DOCKER_HOST` pointed
at the node's `tcp://…:2375`, the way it was deployed; the task runs on the hub and the
node's page follows it. Also per stack: every file it was deployed from (chips), its
**`.env`** (values masked until "Reveal and edit"; validated in place, put back if
compose objects), **what Deploy would change** (a line diff against the copy kept at the
last deploy from here, `<file>.hamlab-deployed`, else the copy the last save replaced;
`.env` values never shown), and **image update checks** — asked for, never polled, since
each is a registry request Docker Hub counts; remembered six hours. A save refuses a
file changed on disk meanwhile.
**Making and unmaking (2026-09-20):** "+ New stack" — a name, a folder, compose text and
an optional `.env` (written root-only), checked by compose before it is kept, then
"Create and deploy" or "Create only". "+ New container" is the same dialog opened on a
form (image, ports, volumes, environment, restart, command, host networking) that writes
the compose for you: a container made here is a one-service stack, so it stays editable
and redeployable, where a bare `docker run` could only be thrown away and typed again.
New stacks go in a **home** — a folder where two or more stacks are already deployed from
(Dockhand's `stacks/core` here; `~/stacks` on a machine with none) — and are deployed the
way their neighbours were (inside `dockhand`, or over tcp to a node from the hub). Homes
are also scanned, so a stack with no containers ("not deployed": never deployed, or taken
down) is still listed. **Down** removes a stack's containers; **Delete…** (type its name)
takes it down and moves its own files to `<home>/.hamlab-trash/<name>-<time>/` — moved,
not erased; volumes and bind-mounted data are never touched. A stopped container can be
**removed** (a running one must be stopped first, on purpose). Images: **pull** (a followed
task), **remove** (unforced: Docker refuses one still in use) and **prune** (dangling only).
**Volumes (2026-09-20):** each says which containers use it; one that nothing uses can be
deleted — a named one by typing its name (it may be the data of a stack that is only
down), an anonymous one by confirming its full name — and "Prune anonymous" clears the
nameless leftovers of recreated containers, never named volumes. Images are marked "in
use", and "Remove unused" takes every image no container uses. Nothing is ever forced:
Docker refuses what is still in use, and that refusal is the answer. Still absent:
removing networks.
- **Top merged into Processes** — **2026-09-23.** The two tabs answered the same question
from the same data (Top already read `/api/procs`, the endpoint the Processes table was
built on), so at Mitchell's instruction there is now one tab, **Processes**, showing what
Top showed: the btop-style whole-machine screen. `Procs.tsx` is deleted; the `top` view
id and its route are gone. A rail order saved before this keeps its shape — `railorder.ts`
maps the retired `top` to `procs` and drops the duplicate, so the merged page sits where
Top sat rather than falling to the bottom.
- **Dashboard retired** — **2026-09-23**, at Mitchell's instruction, the same reasoning that
merged Top into Processes: the app repeated itself. The customisable module list is gone
(`Dashboard.tsx` deleted), and with it `/api/prefs/dashboard`, `applyPrefs`, and the
per-user `dashboard` blob in `state.json`. The Processes tab already shows those numbers
on one screen. `fmtBytes`/`fmtRate` moved to `fmt.ts` (Disks, Files and Space use them);
`MONITOR_HISTORY` and `seriesOf` went with the page. The default view after login, after
switching machines, and when a hub-only page is left behind, is now **Processes**. A saved
rail order naming `dashboard` degrades quietly — `orderBy` drops ids with no page.
**Widgets stays and says what it is for**: a notice at the top explains these are for
dashboards *outside* Ham Lab — Glance, Home Assistant, a wall display — served from
`/embed/widget` with a read-only sensor token, not a second copy of the console's screens.
**It also closed a real bug.** `applyPrefs` set the machine's sampling rate from
`intervalMs` inside the dashboard layout, and its `else` branch reset the sampler to the
default whenever that field was absent. Since layouts became per-user, one account saving
a dashboard could silently reset the sampling rate for the whole machine, overriding the
fleet refresh rate set in Settings. The sampler now has one source: `refreshMs` in the UI
prefs, which the hub fans out, remembered per machine as `sampler_ms`.
- **ZFS tab, with scheduled snapshots** — **shipped 2026-09-24** (`internal/storage/snapshots.go`,
`internal/api/zfs.go`, `ZFS.tsx`). Pools, datasets and snapshots moved out of Disks —
Disks is the hardware, ZFS is the filesystem on top of it — and the tab gained what it was
split out for: **snapshot schedules**. Hourly, daily, weekly or monthly, per dataset,
optionally recursive, with a **retention period** and a floor that keeps the newest few
whatever their age. A missed window (the machine was off) runs once on return rather than
being skipped.
**The naming is zfSnap's, deliberately.** `grid` has run zfSnap v1.11.1 from root cron at
02:30 with a 6-week TTL since before this repo existed, and it works. So rather than
invent a second convention for two tools to disagree over, this writes and reads exactly
what zfSnap does — `tank@2026-09-24_02.30.01--6w`, **retention encoded in the name**. Three
consequences: Ham Lab shows and expires the snapshots zfSnap already took; the two can run
side by side without fighting, since both only remove what has passed the expiry written
into its own name; and changing a schedule never re-dates old snapshots, because each
carries the rule it was taken under. The page **detects the existing cron job**
(zfSnap, zfs-auto-snapshot, sanoid, or a bare `zfs snapshot`) and says so, warning if a Ham
Lab schedule is running as well.
**Nothing is destroyed unseen.** Only names that parse as a stamp and a TTL are ever
candidates for expiry, so a snapshot taken by hand is invisible to a schedule; Run now
says that expired snapshots go with it; and a snapshot deleted by hand is confirmed first
and needs its full name echoed back by the caller. (A "what would it delete?" preview
shipped and was removed the same day as redundant — Mitchell's call.)
The datasets table is a **snapshot explorer**: each dataset shows how many snapshots it has
and what they cost, and opens into the snapshots themselves — when each was taken, **when
it expires** (read from the TTL in its own name, so no server call), its size, and Delete.
Deleting takes the same-named snapshot on descendant datasets too, which is how a recursive
schedule made it.
**A schedule is a real cron job** (2026-09-24, Mitchell's call after the first cut ran it
from a ticker inside the daemon). Ham Lab writes `/etc/cron.d/hamlab-zfs`, owns it, and
rewrites it whole on every change; each line calls `hamlab zfs-snapshot <id>` as the
account Ham Lab runs as, reaching zfs through sudo as the daemon does. Snapshots are
backup-shaped and must not stop because a console is down, upgrading, or uninstalled — and
a cron line is greppable by anyone looking at the machine the normal way. **A cron
expression is the single description of when a schedule fires**: the four common shapes
render to one, `custom` states one outright, and the next run is read back with the same
evaluator the Jobs tab uses on everyone else's lines, so the page, the file and the
prediction cannot disagree. A run's outcome is written to `zfs-runs/<id>.json` beside the
config, never to `state.json`, because the cron job is a separate process and the daemon
would clobber it. An existing external job can be **adopted** — its schedule, retention
and dataset read out of the cron line into a new Ham Lab schedule.
**zfSnap is retired on `grid`** as of that day: Ham Lab now takes the nightly `tank`
snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it
had taken left untouched and fully readable here — which is what adopting its naming
bought. See `hamlet/docs/hosts.md`.
- **KVM tab** — **shipped 2026-09-24** (`internal/api/kvm.go`, `KVM.tsx`, `config.KVM`).
Paste a device's address; its own web page appears in the tab. **Nothing in it knows
anything about any particular KVM** — the first device here is a Sipeed NanoKVM on `grid`
at `10.0.0.217`, but none of that is in the code.
**Why it proxies rather than framing the device directly.** Ham Lab is served over TLS and
a KVM usually is not, so a bare `http://…` iframe is blocked as mixed content before it
loads. Serving the device from Ham Lab's own origin (`/kvm/<id>/…`) fixes that, puts it
behind **Ham Lab's** login rather than a second one, and makes the iframe same-origin.
**The general problem, and the answer.** A device UI asks for its assets by absolute path
— `/assets/app.js`, `/api/whatever` — which no path prefix can rewrite away, and rewriting
bodies per device would not be general. So the prefix is only the entry point: any request
Ham Lab does not serve itself is routed to whichever KVM the asking page belongs to, read
from the **Referer**, falling back to a **cookie** when there is none (a WebSocket upgrade).
Ham Lab's own routes always match first, so a device can never shadow the console, and the
fallback is gated on a live session. The device never receives Ham Lab's session cookie,
and its `X-Frame-Options`/CSP/HSTS are dropped on the way out since Ham Lab is the origin
now. Redirects to the device's root are rewritten back through the prefix.
**One catch found in testing:** Ham Lab's own `frame-ancestors 'none'` forbids framing even
from the same origin, so `/kvm/` paths send `'self'` instead. Without that the iframe was
refused before it loaded.
**The keyboard** needs less than it sounds: a page cannot forward keys into an iframe and
does not need to, since one with focus receives them directly. Fullscreen plus
`navigator.keyboard.lock()` is what takes Escape, Tab and Ctrl-W back from the browser;
that is Chromium-only, and the tab says which version you are getting rather than
pretending.
**One KVM per machine, and the tab follows the machine switcher** (corrected the same day:
the first cut was a hub-side list you pointed at machines, which had it backwards). Switch
to `grid` and you get grid's; adding a second for a machine replaces the first. The
addresses are still kept **on the hub** rather than on each machine, because a KVM earns
its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub
proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at
all. Not done: holding the device's own login on the hub, so its page does not ask for a
second password.
- **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
each repaints the terminal too. **Each machine wears a colour** (`accents.ts`): a stripe
across the header and its name in the switcher — picked from ten swatches or any colour
on the Machines page, else derived from its name — saved with the theme on the hub.
- **Multi-host** — **Phase 1 shipped 2026-09-18** (`internal/nodes`, `internal/tlsauto`,
`MachineSwitcher.tsx`, `Machines.tsx`). One binary, hub and spoke: a host whose
`config.json` lists `nodes` is a hub; any other is a standalone node with its full UI.
Node side: `"tls": "auto"` makes a self-signed cert in the data dir and serves HTTPS;
`POST /api/tokens {password,name}` trades the login password for a long-lived bearer
token (SHA-256 stored in `tokens`), accepted by the auth middleware alongside the
cookie; `DELETE /api/tokens/{name}` revokes. Hub side: `POST /api/nodes {name,url,
password}` peeks the node's certificate (trust on first use, fingerprint pinned),
obtains a token named `hub:<hostname>`, stores `{name,url,fingerprint,token}`; a
15 s prober keeps each node's status; `/api/nodes/{name}/…` is a reverse proxy to the
node's `/api/…` (bearer swapped in, cookie and Origin stripped, streaming flush) that
carries WebSockets, so terminals, log tails and task output work remotely.
Frontend: `setMachine()` rewrites every `/api/…` path; the rail has a machine
switcher at the top and a Machines page to add/remove/open. Pages remount on switch;
terminals live on the node. Nodes: `grid`, `wyse`. **Phase 3 (update from hub)
shipped 2026-09-18**: node `POST /api/self-update` takes a binary, checks it answers
`version`, keeps `hamlab.prev` in the data dir, installs over the executable via
`sudo install`, replies, then restarts the service; the hub pushes its own executable
with `POST /api/machines/{name}/update` or `/api/machines-update-all` (architecture
checked first via `/api/me.arch`), the Machines page shows an **outdated** badge and
Update buttons, and `hamlab push-nodes` does the same from the CLI — `make deploy` runs
it last, so one deploy on the hub updates every machine. First install on a new host
is still by hand (scp binary + `install.sh`, `hamlab passwd`, `tls auto`). Still to
do: Phase 2 Datacenter overview; optional firewall rule limiting a node's port to
the hub. **Cross-architecture** (2026-09-18): `make deploy` also builds
`bin/hamlab-linux-arm64`, `install.sh` installs it beside the binary as
`hamlab-arm64`, and the hub picks the sibling matching a node's `/api/me.arch` when
pushing — the Mac mini (Fedora Asahi, aarch64, node `mac`) updates like the rest.
- **Portability + install** — **shipped 2026-09-18** (`internal/setup`, `internal/priv`,
`install.sh`, `make release`): the binary installs itself (`hamlab install` /
`uninstall`; systemd or OpenRC; root, sudo or doas), all builds static on amd64, arm64,
arm, 386 and riscv64, `install.sh` as the curl-able bootstrap with checksums. Every
privileged call goes through `priv` instead of a literal `sudo -n`. Updates tab acts on
dnf, zypper, pacman and apk as well as apt. Services list falls back to the plain
table on systemd < 246; Disks falls back to `/sys/block` without `lsblk`. Listener
uses `IP_FREEBIND`. Self-update: pre-flight `check` against the live config, atomic
replace, exec in place. Config default moved to `~/.config/hamlab` (existing
`~/apps/hamlab` still found by the installer and the CLI). Tested in
containers: Debian 12, Ubuntu 22.04, Fedora 40, Rocky 8, openSUSE Leap 15.5, Alpine
3.18/3.20 (OpenRC + doas, non-root), plus the four lab hosts. **procd (OpenWrt) added
2026-09-26** (`internal/setup/hamlab.procd`): detected on `/sbin/procd` + `/etc/rc.common`
ahead of the OpenRC branch, installed to the same `/etc/init.d/hamlab` path and told apart
from OpenRC at uninstall by `USE_PROCD=1` in the file. Verified on the new router
(`gate`, OpenWrt 25.12.5, running as root): `S95`/`K10` symlinks, procd supervising the
instance, Docker tab and stacks working through the hub. The Updates tab needed no change
— apk-tools 3 answers `apk version -l '<'` the same way Alpine's does. Not done: an
OpenRC or procd backend for the Services tab (it is systemd-only and says so cleanly);
runit/s6/sysvinit service files; public hosting for `install.sh` (the Gitea repo is
private, so the one-liner needs `HAMLAB_TOKEN`).
- **Users, sessions and settings that follow the account** — **shipped 2026-09-22**
(`internal/api/users.go`, `config.User`, `store.Session`/`UserState`, `Login.tsx`,
`Users.tsx` — a hub-only **Users** tab in the rail, added the same day at Mitchell's
request, holds everything about accounts: your own password, your sessions, the
accounts; Settings keeps appearance, rates, tokens and About). **Rail order is per account** too (2026-09-22,
`railorder.ts`, `Rail.tsx`): "⇅ Arrange" at the foot of the rail puts a handle on every
page; drag one up or down (pointer events, so a finger works), let go and the order is
saved in the UI prefs with the theme; "Reset" restores the built-in order; pages added
in a later build go after the ones the saved order names. The login is a user name and a password. `config.json` holds `users`
(name, argon2id hash, created, last seen); the old single `password_hash` becomes the
account named after the OS user the console runs as on first boot, its signed-in
sessions become that account's, and the machine-wide theme/accents/dashboard become
theirs — nothing to redo after the upgrade. A session carries its user, an ID, when it
was made and last seen, the IP and the browser; **Users → Sessions** lists yours
("Safari on iPhone · 10.0.0.5 · seen 2 min ago"), revokes one, or signs out everywhere
else. **Users → Users** lists accounts with their session counts, adds one, removes
one (their sessions and settings go too; never yourself, never the last), and resets
another's password (they are signed out); removing and resetting take your own
password first. Everyone signed in can do everything — a household console, not a
tenancy — and each machine keeps its own accounts: a hub reaches a node with its token,
so pairing (Machines → Add) now takes a user and password on the node, the user
optional while it has one account. Per account: `ui` (theme, accents) and `dashboard`
in `state.json` under `users.<name>`. Still the machine's: refresh rate (it is how
often every machine samples — it rides in the UI prefs object for the client but is
stored once, machine-wide), graph history, alert rules, terminals (the shells run as
one OS user anyway). A login attempt for a missing user takes as long as a wrong
password, so the form can't list accounts. Every write's log line now names the
account or token (`core@10.0.0.5 via …`), and Access tokens say who issued them. CLI:
`hamlab passwd [user]` (makes the account if new; reads stdin when piped), `hamlab
users`, `hamlab user-rm <user>`; `hamlab install` names the first account after the
service user. Not done: a read-only viewer role.
- **Space tab** — **shipped 2026-09-22** (`internal/space`, `internal/api/space.go`,
`Space.tsx`), asked for as "exactly WizTree": what is using a disk. The unit is a whole
volume, picked from a dropdown at the top: a **disk on its own** (every filesystem its
partitions back) or a **ZFS pool as a whole** (every mounted dataset) — a pool's member
disks are listed under it with the layout (raidz2, 8 disks: sda …) and never offered
apart, since one disk's share of a pool means nothing (Mitchell's call, same day, after a
first cut that listed the disks as chips). The walk runs as a helper process, `hamlab space-walk <roots>`,
as root through `priv` when the host allows (so `/var/lib/docker` and `/root` count), 32
workers reading directories depth-first, never crossing a filesystem boundary by itself;
each root is grafted into one tree at its path. Files under 128 KiB are not kept one by one
but summed per folder as "N smaller files" — what keeps a scan of millions of files to tens
of MB in the daemon; the tree crosses back as a varint stream (`codec.go`), and the helper's
memory goes with it. core's root: 646k files in ~1.2 s. Progress (files, bytes, current
folder) every 250 ms; Cancel sends SIGTERM. The page: picker → Scan/Rescan → **Folders**
(a tree largest-first: size, share of parent as a bar, files, folders, modified; expand in
place) over a **treemap** of the selected folder — squarified, nested to the file, tiles
coloured by kind (video, images, audio, archives & disk images, documents, code & config,
databases & logs, other; the validated categorical set, dark and light) with folders as
labelled boxes; hover for path, size and share; click a folder to drill, breadcrumbs and ↑
to come back. The map asks the server for the folder pruned to what a tile could show
(`min` = 1/6000 of the folder, 24 deep; deeper folders come back as one lump), the list
one level at a time. **Largest files**: the 1,000 biggest with a path filter. Works on
every machine through the hub.
**Smarter and faster (2026-09-23, at Mitchell's request):** a scan is *kept* — in memory
and in `<data>/space/<target>.scan` (the same varint stream) + `.json`, read back after a
restart — and the next scan *starts from it*: the helper is handed the last tree on stdin
and reads a directory again only when something says it changed. On a ZFS pool with
snapshots, the scan records the newest snapshot of each dataset at start (`bases`), and the
next one runs `zfs diff -H -F <base>` per dataset: only the directories it names are
read, their ancestors kept but looked into, everything else kept wholesale without a
stat. The diffs run side by side and one is given three minutes — `timeout -s KILL`
inside sudo, since Debian's sudo gives the command its own session and a process-group
kill from outside never reached zfs — after which that dataset is walked by mtime
instead. On `tank` a diff is metadata-bound like the walk: 8 s to 3 min per dataset for
a handful of changes, and `immich`, which changes constantly, ran past twenty minutes
the first time. A refresh of `tank` took 4 min 19 s against 7 min for the full scan. Elsewhere (ext4, or a dataset with no snapshot) a directory whose mtime — to the
nanosecond — is unchanged keeps its list and only its kept files are stat'ed again, since
a file can grow without its directory noticing. A **cancelled** scan keeps what it walked,
marks the folders it never got to (`partial`, "not yet read" in the tree) and **Resume**
reads just those. core: full 1.2 s → refresh 0.3 s (2 folders read, 84,755 kept). The
status line says which (`method`, `by_root`, read/kept counts); **Forget** drops the saved
result so the next scan is full. The old result stays browsable while a refresh runs.
**Treemap rethought** the same day: every file a tile grouped by folder — folders are only
a thin line and a name strip on the first level — instead of nested boxes with headers,
which on Linux's single-child folder chains was a staircase of headers around nothing.
Not done: delete from here, a size-over-time history, per-user column choices.
- **Updater** — **shipped 2026-09-23** (`internal/selfupdate`, `internal/api/update.go`,
the Updates card in `Settings.tsx`, `scripts/gitea-release.sh`). A Ham Lab asks one URL
what the newest build is and installs it on request. `update.url` in `config.json` is
either a Gitea/Forgejo repo API base (`…/api/v1/repos/<owner>/<repo>`, whose newest
release names the version and carries the binaries as assets) or any static directory
holding `VERSION`, `SHA256SUMS` and `hamlab-linux-<arch>` — which is exactly what
`make release` writes and what `install.sh` already downloads, so the demo site, a
folder on a NAS and a Gitea release are all the same thing to it. `"off"` never looks.
`update.token` is sent when the source is behind a login, so a **private** Gitea serves
its own fleet without the instance being opened to the world. The binary is verified
against `SHA256SUMS` before it goes anywhere near the running one, then takes the same
path `POST /api/self-update` always did (runs `version`, runs `check` against this
machine's config, atomic replace keeping `hamlab.prev`, re-exec in place) — that path is
now one shared method, `installBinary`. Version ordering is `install.sh`'s `vnum()` rule
ported to Go and tested against it, so the daemon and the installer never disagree about
which of two builds is newer; the commit count since the tag counts, the hash does not.
Checked ~90 s after boot and daily (`update.check: "never"` switches that off), never
installed unasked — an update restarts the daemon and drops open terminals, and the
dialog says so. Settings → **Updates**: installed, published, when it last looked, the
source, the release notes, Check for updates, Install; the rail badges Settings when a
newer build is out (`update_available` rides on `/api/me`). Publishing is
`make gitea-release` → `scripts/gitea-release.sh`, which creates the release and uploads
the five architectures, `SHA256SUMS`, `install.sh` and `VERSION`; it is deliberately
**not** part of `make deploy`, because deploy pushes this build to the machines this hub
owns while publishing offers it to every install anywhere, and one tag per deploy would
be noise. Re-publishing a version needs `--force`.
**Public since 2026-09-23**, at Mitchell's instruction and after the trade was put to him
twice: Gitea's instance-wide `REQUIRE_SIGNIN_VIEW` is off and `mitchell/hamlab` is marked
public, so anyone can browse it, clone it and download a release with no account. The
other five repos there are each marked private and are invisible — an anonymous request
for one is a 404, not a login prompt — registration is still disabled, and new repos
still default to private. No token is needed by anything any more (the read-only one was
revoked; only `hamlab-publish`, which creates releases, remains). Gitea resolves
`releases/download/latest/<asset>`, so `install.sh` and `HAMLAB_URL` now point straight
at the newest release and never need changing. What is public is the source, the history,
`BUGS.md` and this file — which describes the lab and names the console's hostname.
- **Auth hardening** — WebAuthn passkeys (phone Face ID), TOTP fallback, optional NPM
access list restricting to WireGuard/LAN.
- **Files tab** — **shipped 2026-09-18** (`internal/files`, `Files.tsx`): browse from
`/` with breadcrumbs, an editable path box, sort, hidden-file toggle and a filter;
rows show type icon, size, modified, perms and owner, link targets and broken links.
Text files (≤ 2 MiB, sniffed) open in an editor with Ctrl+S / Save; images preview;
everything else downloads. Upload via drag-and-drop or picker with per-file progress
(streamed, no size limit). New folder / file, rename, move, chmod, copy path, delete
(folders need the name typed back), and "open in terminal", which `cd`s the active
shell (or a fresh one) to the folder. **Sudo toggle**: root access re-executes the
same binary as `hamlab fs <op>` under `sudo -n`, streaming bodies over stdin/stdout,
so both modes share one implementation and one set of rules (absolute paths only;
`/`, `/etc`, `/usr`, … are never deleted or moved whole). Every write is logged with
the caller's IP.
- **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth.
- **Notifications** — push on service failure / SMART warning / UPS event.
- **Terminal extras** — search, copy-mode, font/theme, split panes, upload via
drag-drop into cwd.
- **Settings + themes** — **shipped 2026-09-18** (`Settings.tsx`, `themes.ts`): Settings
page with Appearance (Dark, Light, Cyberpunk, Retro — CSS variables on `<html>` plus a
matching xterm palette, applied before first paint from localStorage and synced hub-side
via `GET/PUT /api/prefs/ui` so the choice follows the login), Account (change password,
`POST /api/password`, current password required), Access tokens (list/revoke the tokens
hubs hold), About (version, host, arch, uptime, certificate).
## Network tab (2026-09-27)
A **Network** tab, per machine like every other: the interfaces as the kernel has them
(addresses, MAC, MTU, negotiated link speed, carrier, byte totals, live rates from the
sampler's own sensors), the default route and resolvers, what the internet sees of this
connection, and an **internet speed test**.
Decisions worth keeping:
- **Measured by the daemon, not the browser.** The test runs on the machine in the switcher,
so the figure is that machine's line — the router's, if you switch to `gate` — and it
carries on if the phone locks or the tab closes. The page polls a run it is not driving.
- **No tool to install.** It measures against Cloudflare's public speed endpoints
(`__down`, `__up`, `cdn-cgi/trace`), which is what makes it work identically on Debian and
on OpenWrt, where `opkg install` costs flash. That is the feature's one external
dependency, and it is named in `internal/network/speedtest.go`.
- **Several streams, HTTP/2 off, and the ramp thrown away.** One stream over one connection
measures a congestion window; HTTP/2 would put every stream back on one connection and do
the same. The first second of each phase is measured and discarded.
- **Latency is the round trip less the far end's own Server-Timing.** A zero-byte response
is mostly Cloudflare's worker running — 20 to 45 ms of it against a few ms of wire. Left
in, it reported 31 ms for a 7 ms line. Jitter is the mean change between consecutive
probes, not a standard deviation, so one slow probe doesn't run away with it.
- **A refused phase is not a number.** The far end rate-limits by volume, and a test moves a
few hundred megabytes; several in a row earn 429s, and the block observed while building this
lasted over twenty minutes. So: a 20-second cooldown counted from the last *attempt* (a
refused test still used the line), a stream that retries twice before giving up, the rate
computed over the window in which bytes were actually still arriving, and a plain sentence
when it was cut short. Coasting to the end of the clock with nothing arriving reported a
third of the truth.
- **A second download source, for when the first refuses.** A plain 100 MB file on Linode's
Seattle mirror, which has no opinion about volume. It is only reached after a refusal, and
every result names the source it used — that mirror is in one place, so on a machine far from
Seattle it would read low, and a number that quietly came from somewhere else would be a lie.
Upload stays on Cloudflare, which has never been the one to refuse.
- **The last result is published as sensors** (`net.internet.*`), so the line is a widget, a
graph, an alert or a Prometheus series like anything else. Each carries the time the test
ran, not the time it was asked for.
- **Not scheduled.** Nothing takes a speed test on its own. A nightly one recorded to the
history is the obvious next step and deliberately not built yet — the cron shape for it
already exists twice over (ZFS snapshots, Docker image updates).
## VPN tab (2026-09-27)
A **VPN** tab beside Network, per machine: every WireGuard tunnel on the machine in the
switcher, and the devices on it — name, address, last handshake, where it connects from, and
what it has moved.
- **Per machine is the point, not a limitation.** The tunnel people dial into and a tunnel a
machine dials out through are on different boxes here (`gate` and `hamtv`), and a combined
view would invite exactly the question "which box is this on".
- **The kernel is the truth, the config only supplies names.** `wg show all dump` has the
handshakes and counters and knows nothing but public keys; names come from the `# comment`
above a peer in a wg-quick file, or a UCI section's `description` on OpenWrt. Both are read
only to label what the kernel already said.
- **Server or client is worked out, not declared.** A peer allowed `0.0.0.0/0` that has an
endpoint is a way out, so a tunnel holding one is this machine dialling out; a tunnel with a
listen port and peers is one being dialled. Anything else says neither rather than guessing.
- **"Online" is a handshake in the last three minutes**, and the tab says so. WireGuard is
silent when idle; there is no connection to be in, and a tab claiming otherwise would be
lying twice a day.
- **No secrets are read.** Private keys are in the second column of the dump and are read
past; pre-shared keys are noted as present and never read.
- **Devices can be created, renamed and removed** (added 2026-09-27, asked for the same day),
on a tunnel the machine configures — today OpenWrt's UCI, which is where the server here
lives. A wg-quick host is reported unmanaged and shows no buttons: half-supporting a second
configuration format is worse than an honest refusal.
- **Nothing restarts the network.** `uci commit` writes the file and applies nothing; `wg
set` changes one peer on the live interface. Proven on `gate` by the existing peers'
transfer counters surviving a create and a delete — a reload would have zeroed them.
- **The two halves are ordered so they cannot disagree quietly.** Adding writes the
configuration first and rolls it back if the live tunnel refuses, because a device that
works only after the next reboot is the worst of the three outcomes. Removing does the
live tunnel first, since that is the moment access ends, and says so plainly if the
configuration half then fails.
- **The keypair is generated on the machine** and the profile, with its QR, is shown once.
The private key is stored nowhere — the same bargain as a guest login's password.
- **Removing asks for the name to be typed back**, at both ends. It cannot be undone and it
sits one tap from a list of devices that all look alike.
## To do
- **Grow the Files tab until copyparty can go** — asked for 2026-09-23 as two items (Files
polish, and deprecating copyparty), merged 2026-09-24: they are the same tab and the same
work. `copyparty` on `grid` (`copyparty.service`, `:3923`, `/etc/copyparty.conf`, serving
all of `/tank` as user `core` with one account behind NPM) overlaps **Files** almost
entirely. The aim is to stop the service and keep only Ham Lab, with general ease-of-use
polish along the way. What Files must gain first, in the order that matters:
- **Multi-select and bulk actions.** copyparty selects many rows and moves, deletes or
downloads them together; Files acts on one row at a time.
- **Download a folder**, and a multi-file download, as a streamed zip or tar — the thing
copyparty is reached for most.
- **Resumable uploads.** copyparty's up2k chunks a file, survives a dropped connection
and skips what is already there; Files streams one PUT, so a phone upload that drops at
90% starts over.
- **A gallery.** Thumbnails for a folder of images and video, and play audio/video in
place — `/tank/media` and `/tank/immich` are most of what is browsed.
- **Search** below the current folder by name, since `find` in a terminal is the fallback
today.
- *Decide when built:* whether share links (copyparty's public per-path links) are wanted
at all — they are the one feature that hands out access without a login, and Ham Lab has
no notion of that; if not, say so in the docs rather than leaving a gap unexplained.
- *After:* `systemctl disable --now copyparty`, remove its NPM host, keep
`/etc/copyparty.conf` until a rollback is clearly not needed, then update
`hamlet`'s `docs/hosts.md` bare-metal list, `docs/gotchas.md` (the "`docker ps` is not
the inventory" note names it) and the `grid` row in `CLAUDE.md`.
- **Grow the Docker tab until Dockhand can go** — asked for 2026-09-23 and widened
2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the
Files item: it is one tab, and the gaps that block retiring the old tool and the things
that would simply make it better are the same body of work. The *homelab* half —
the cutover itself, then moving every stack off Dockhand's data dir — is `hamlet`'s
`docs/pending.md` → "Retire Dockhand; Ham Lab becomes the orchestrator", agreed
2026-09-20. This entry is the *app* half.
**Done 2026-09-24 — the eight Mitchell picked from the inventory**, all in the Docker tab:
per-container **Stats**, **Health** (with the last few probe outputs), **Events** (Ham Lab
keeps its own 4000-entry ring, since `docker events` forgets after about a minute),
**Shell** (`docker exec`, handed to the Terminal tab) and **Limits** (`docker update` for
memory, CPU and restart policy); an **updates** tab giving the whole machine at once with
a per-container indicator in the list; and **schedules**, which write
`/etc/cron.d/hamlab-docker` and call
`hamlab docker-update <id>` — the shape the ZFS schedules already use.
Also done: Ham Lab keeps working when `dockhand` is removed. It remembers where a
container's bind mounts pointed (`mounts` in `state.json`) and resolves a stack through
that record when no running container explains its path, running compose on the host
instead; `repoint` does the same for a schedule at run time. Without this, removing the
container would have made 17 stacks on `core` and 5 on `grid` unfindable at once.
- **Left from the Dockhand inventory, both to decide rather than build.**
- *Git-sourced stacks.* Dockhand's `stack_sources` table points a stack at a repo and
pulls it. Its `git_repositories` and `git_stacks` tables are **empty**, so nothing
here uses it — dropping it is the honest answer unless Mitchell says otherwise.
- *Webhooks, OIDC, LDAP, vulnerability scans.* All four tables empty. Nothing to move.
**Done 2026-09-25 — the two gaps in the tab itself.**
- *Logs are followed over a held connection*, the way the Services tab follows a journal
(`internal/api/docker_logs.go`, `/api/docker/containers/{id}/logs/tail`): one
`docker logs -f` per viewer, stdout and stderr both, stderr coloured as an error. A
reconnect resumes from the last line's timestamp, so a container that restarts or a
phone that slept doesn't replay what is already on screen; the client dedupes by line
rather than by "later than", because `--since` is inclusive and two lines can share a
timestamp. Docker replays the opening backlog down two pipes in no particular order, so
the first 300 ms is held and sorted before it is sent. The polled `?since=` endpoint
stays — it is what the resume is built on. Reconnects back off to 15 s, since a
container that has stopped ends the stream immediately.
- *Networks can be removed and pruned* — `POST /api/docker/networks/{remove,prune}`, the
shape images and volumes already had. The overview says who is attached to each one, so
the ✕ is only offered on a network nothing uses and never on Docker's own three. Prune
is deliberately untested against this host: `core` has several networks belonging to
stacks that are merely down (`pihole_default`, `transmission_default`, `gluetun_default`
among them), and pruning would take them and hand them new subnets on the next deploy.
- **The `*:2375` question, which this is the moment to settle.** A stack deployed to
`grid` or the Mac still goes over `tcp://<node>:2375` from the hub. Decide whether the
hub should instead ask the node's own Ham Lab to deploy locally, which would end the
unauthenticated Docker socket exposure — its own item in `docs/pending.md`, currently
filed under "decided, no action", and this is the change that would make closing it
free.
- **A Virtualization tab** — asked for 2026-09-23, not started, and **large**. Run and
manage **LXC containers and virtual machines** from the console, the way Proxmox does —
which is the comparison Ham Lab has invited since its first line. Nothing exists yet.
What a future session has to settle before writing code: which stack (libvirt/KVM +
`virsh` is the obvious one for VMs; LXC or Incus for containers, and Incus covers both);
whether it manages what is already installed or installs it; where disk images live
(`tank` on `grid`, local NVMe elsewhere); console access (SPICE/VNC proxied through the
daemon, or a serial console over the existing terminal machinery, which would reuse
`internal/term`); snapshots and clones; and what it does on a host with no virtualisation
at all, which is every host here today. Worth scoping as its own plan before starting.
## Open questions
- Idle-session reaping policy (never / after N days) — leaning never; it's a
README.md
+95 -4
View File
@@ -1,10 +1,101 @@
# Ham Lab
Web-based Linux management console for bare-metal Debian, in the style of the Proxmox
Web-based Linux management console for bare-metal Linux, in the style of the Proxmox
web UI. Named after Ham the cat (see also HamTV).
Status: **planning**. See [`PLAN.md`](PLAN.md) for the MVP scope and roadmap. No code yet.
Status: **v0.1** — persistent web terminals and a live Dashboard tab. Scope and roadmap in
[`PLAN.md`](PLAN.md).
- Runs as a systemd service on the host it manages, as user `core`.
- Public at `https://hamlab.bigham.dev` (behind Nginx Proxy Manager on `core`).
## Runs on any Linux box
One static Go binary with the UI embedded: no cgo, no libc, no Node at runtime, no agents.
The same file runs on Debian, Ubuntu, Fedora, RHEL/Rocky/Alma, openSUSE, Arch and Alpine
(musl), on bare metal, VMs and containers. Builds: `amd64`, `arm64`, `arm` (ARMv6
hard-float and up, so every Raspberry Pi), `386`, `riscv64`.
```sh
sudo ./hamlab-linux-amd64 install # that's the whole install
```
or, from wherever `make release` output is served:
```sh
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh # asks: user, address, HTTPS, firewall, password
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh -s -- --yes # asks nothing, all defaults
curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh -s -- --listen 0.0.0.0:8100 --tls auto # flags go to `hamlab install`
```
`install.sh` is a wizard when there is a terminal (it reads answers from `/dev/tty`, so it
works piped from curl) and silent otherwise. Either way it picks the build for `uname -m`,
checks it against `SHA256SUMS` and runs `hamlab install`; re-running it upgrades in place.
`latest` is an alias Gitea resolves to the newest release, so that URL never changes.
`make gitea-release` publishes a release there (`HAMLAB_URL` points the installer elsewhere); run the script beside the `hamlab-linux-*` files and it installs offline.
`hamlab install`:
- copies the binary to `/usr/local/bin` (`--prefix`), atomically;
- gets root through `sudo` or `doas` if you aren't root already, and runs the console as
**you** (`--user`), so terminals are your login shell with your rights;
- writes the config to `~/.config/hamlab/` (`--data`; an existing `~/apps/hamlab/` is kept)
and asks for the first account's password (the account is named after the user the
console runs as), or generates one and prints it once (`--password-stdin` for scripts);
- registers a boot-time service with **systemd** or **OpenRC**, naming only groups that
exist on the host, and waits for `/healthz`; with neither init it prints the one line
to put under your own supervisor;
- says so when firewalld or ufw will block the port (`--open-firewall` opens it).
Re-running it upgrades in place and keeps config, user and accounts. `hamlab uninstall
[--purge]` reverses it.
Accounts: every person has their own user name and password, and the theme, accent
colours and dashboard layout follow the account. The **Users** tab has it all: your own
password, where you are signed in (sign out any device), and the accounts (add, remove,
reset a password; your own password confirms the last two). From the shell:
`hamlab passwd [user]` sets a password and makes the account if it is new, `hamlab
users` lists them, `hamlab user-rm <user>` removes one. Machine settings — refresh rate,
graph history, alert rules — stay the machine's and are shared by everyone.
**Guest logins**, for showing somebody the console: Users → Guest logins makes a
throwaway user name and password, good for half an hour by default (up to 24 hours), and
hands you the pair to paste into a message. A guest is an ordinary account with a clock on
it — the whole console, every machine, the terminal included — that stops working when the
time is up, wherever it is signed in, or the moment you revoke it. There is no reduced
guest: give one to somebody you would hand the password to, and revoke it if you change
your mind.
What adapts to the host, rather than assuming Debian:
| | |
|---|---|
| Root | already root, else passwordless `sudo`, else passwordless `doas`; without any, the console is read-only where root is needed and says so |
| Updates tab | apt, dnf, zypper, pacman, apk: list, refresh, upgrade, install (pacman: all or nothing) |
| Services / timers | systemd of any age (JSON output where it exists, the plain table before v246); a clear message on hosts without systemd |
| Disks | `lsblk`, or `/sys/block` when it's missing (busybox) or too old; SMART and ZFS only when their tools are there |
| Network | `/proc` and `/sys` for the interfaces, addresses, link speed and default route. The speed test needs no tool installed: it measures against Cloudflare's public speed endpoints, so it works the same on Debian and on the OpenWrt router |
| VPN | WireGuard tunnels and their peers, from `wg show` where wireguard-tools is installed; peer names from wg-quick `.conf` comments or OpenWrt UCI descriptions. Read-only, and it never reads a private or pre-shared key |
| Dashboard | `/proc` and `/sys` only: hwmon or thermal-zone temps, fans, NICs, disks, mounts; empty sections where the hardware has nothing to report. UPS from whichever of `pwrstat`, NUT (`upsc`) or `apcupsd` is installed |
| Listen address | bound with `IP_FREEBIND`, so a LAN address that isn't up yet at boot doesn't fail the start |
| Self-update | the new build must run and load this host's config before it replaces the old one; atomic rename; the daemon then re-execs itself, so it restarts the same way under systemd, OpenRC or nothing; `hamlab.prev` kept |
Building: `make build` (web + Go), `make release` (every architecture, `install.sh` and
`SHA256SUMS` in `dist/`), `make test`.
Terminals run as the user the service runs as, with that user's login shell.
## Managing several machines
Install Ham Lab on each host; on the ones the hub should reach:
`hamlab install --listen <lan-ip>:8100 --tls auto` (a self-signed certificate is generated
on first start). Then on the hub, open **Machines**, enter a name, the address
(`https://host:8100`) and a user and password on that machine once (the user may be
left out when it has only one). The hub pins the
certificate, keeps an access token, and every tab works against the chosen machine
through the switcher at the top of the rail.
Updating: `make deploy` on the hub installs the new build there and then pushes it to
every managed machine (`hamlab push-nodes`), which install it and restart. The Machines
page flags machines running an older version and offers the same push per machine.
Mixed architectures are fine: `make deploy` also cross-compiles every other architecture,
and the hub sends each machine the build for its own.
- On `core` it is public at `https://hamlab.bigham.dev` behind Nginx Proxy Manager.
- Installable as a PWA.
SENSORS.md
+265
View File
@@ -0,0 +1,265 @@
# Sensors — the standard read interface
Everything Ham Lab measures is a **sensor**: a stable dotted ID, a little metadata, a
current value. Any consumer — Ham Lab's own dashboard, Glance, Home Assistant, a shell
script — discovers them from the catalog and asks for exactly the ones it wants.
Code: `internal/sensors` (types, registry, selectors, shaped query) and
`internal/metrics/sensors.go` (the collector as a provider). The registry measures
nothing; providers do. A new source of sensors implements `sensors.Provider` and is
added to the registry in `api.New`.
## IDs
```
cpu.usage cpu.core.<n>.usage cpu.temp load.1|5|15 uptime
mem.total|used|avail|free|cached|buffers|used_pct swap.total|used|avail|used_pct
temp.<chip>.<label> fan.<chip>.<label>
net.<iface>.rx|tx|rx_total|tx_total|up
net.internet.download|upload|latency|jitter|tested|ip|edge
disk.<dev>.read|write|util|size|temp
fs.<mount>.size|used|used_pct
ups.battery|runtime|load|load_pct|in_v|out_v|on_battery|state|source
```
- Segments are lower case `[a-z0-9_-]`: `Package id 0` → `package_id_0`, `/mnt/tank` →
`mnt_tank`, `/` → `root`. The real names are in the sensor's `labels`.
- Built from names the hardware reports, never enumeration order (`hwmon3` moves between
boots). Two identical chips collide; the second gets `_2`.
- Which sensors exist differs per machine. That is the point of the catalog.
- `net.internet.*` is the Network tab's speed test: `download` and `upload` in Mbps,
`latency` and `jitter` in ms, `tested` the age of the reading in seconds, and `ip` and
`edge` the public address and the Cloudflare edge as strings. They appear only once a test
has run on that machine, and each one's `t` is when it ran — so a widget or an alert on
them is reading a measurement from that moment, not a live value. Nothing takes a speed
test on its own; something has to press the button.
## A sensor
```json
{ "id": "temp.coretemp.package_id_0", "name": "coretemp Package id 0",
"kind": "temperature", "unit": "°C", "value": 51, "t": 1789826600878,
"warn": 105, "crit": 105, "labels": { "chip": "coretemp", "label": "Package id 0" } }
```
`value` is a number, bool or string. `t` is when it was really read, in unix ms — slow
sections (filesystems 30 s, UPS 10 s) are older than the tick. `min`/`max` are the scale
(0–100 for percentages, 0–total for used bytes); `warn`/`crit` are the hardware's own
thresholds where it states them.
## Endpoints
All behind the same auth as the rest of `/api` (session cookie or `Authorization: Bearer`).
On a hub, every one is also at `/api/nodes/<machine>/sensors…` for each managed machine.
| | |
|---|---|
| `GET /api/sensors` | Catalog, with current values. `?ids=fan.*,cpu.usage` narrows it. |
| `GET /api/sensors/{id}` | One sensor. `?format=text` → the bare value as text/plain. |
| `GET /api/sensors/{id}/history` | `{id, unit, t[], v[]}` — what the ring holds (300 points, spaced by the window below). Numbers only. |
| `GET /api/sensors/history?ids=cpu.usage,net.*.rx` | Many histories in one request: `{series: [{id, unit, t[], v[]}]}`. A sensor younger than the ring has a shorter series. |
| `GET /api/prefs/history` | How far back the history reaches: `{seconds, points, stride_sec, min_seconds, max_seconds}`. |
| `PUT /api/prefs/history` | `{seconds}` — 60 to 86400. Set on the hub it is passed to every machine, so the fleet's graphs reach equally far back. |
| `POST /api/sensors/query` | Shaped query, below. |
| `GET /api/sensors/query?k=selector&…` | The same, flat, for consumers that can only fetch a URL. |
A selector is an exact ID or a glob. `*` crosses dots: `fan.*` is every fan,
`cpu.core.*.usage` every core.
## Shaped query
Send the object you want, with selectors where the values go. Get it back filled in.
```
POST /api/sensors/query
{ "cpu": "cpu.usage", "fans": "fan.*", "box": { "temp": "cpu.temp", "up": "uptime" } }
→ { "cpu": 12.4,
"fans": { "fan.nct6775.fan1": 820, "fan.nct6775.fan2": 1140 },
"box": { "temp": 41, "up": 80520 } }
```
- Exact ID → its value; `null` if this machine has no such sensor (so one query works
across machines that differ).
- Glob → an object keyed by ID; `{}` if nothing matches.
- Objects and arrays are walked; anything else in a query is a 400.
- `?full=1` → every value is the whole sensor object instead of the bare value. (In the
GET form `full` is therefore not usable as a key.)
## Live stream
`GET /api/sensors/stream` is a shaped query that keeps answering: the same object, pushed
as a server-sent event after every sample. The query is `?q=<URL-encoded JSON shape>` or
the flat form; `?full=1` as above; `?every=<ms>` spaces events out for a consumer that
doesn't want them at the sampling rate. A keepalive comment goes out every 25 s, and
`X-Accel-Buffering: no` keeps nginx from holding events back.
```js
new EventSource('/api/sensors/stream?token=…&q=' + encodeURIComponent(JSON.stringify({ cpu: 'cpu.usage', fans: 'fan.*' })))
.onmessage = (e) => render(JSON.parse(e.data))
```
## Auth: read-only tokens
Settings → Access tokens → **+ Read-only sensor token** (or `POST /api/tokens/read
{"name"}` with a session). Scope `sensors:read`: it opens every endpoint in this file,
here and on each managed machine through the hub (`/api/nodes/<machine>/sensors…`; the
hub talks to the node as itself, the caller's token stops at the hub), and nothing else.
- Send it as `Authorization: Bearer <token>`, or as `?token=<token>` where headers are
impossible (an iframe, a browser `EventSource`).
- A read-only token can be **limited to ID patterns** (`{"name": "glance", "sensors":
["cpu.*", "zfs.tank.*"]}`, or the second prompt in Settings). Everything outside them
does not exist for that token — catalog, query, history, stream, Prometheus, and on
managed machines through the hub, which passes the restriction along.
- A **full-control token is never accepted from a URL**, and a read-only token is never
accepted anywhere outside this file's endpoints.
## Prometheus
`GET /metrics` (or `/api/sensors/prometheus`, which also works through the hub as
`/api/nodes/<machine>/sensors/prometheus`). One gauge family per kind of measurement,
base units, the sensor ID and its real-world names as labels:
```
hamlab_temperature_celsius{id="temp.coretemp.core_0",chip="coretemp",label="Core 0"} 50
hamlab_usage_percent{id="fs.root.used_pct",fstype="ext4",mount="/"} 10.1
hamlab_state{id="ups.state",model="CPS CP1500PFCRM2U",value="Normal"} 1
hamlab_updates_pending{id="updates.pending",manager="apt"} 3
```
```yaml
scrape_configs:
- job_name: hamlab
authorization: { credentials: <read-only token> }
static_configs: [{ targets: ['10.0.0.234:8100'] }]
```
## Status sensors
Not everything is a /proc number. These are read in the background on their own clocks
(each sensor's `t` is its real age) by `internal/api/status_sensors.go`:
```
services.failed|failed_names|running every 30 s
service.<unit>.active|state one pair per service that is enabled, up or failed
containers.total|running|unhealthy every 15 s (Docker or Podman, if present)
container.<name>.running|state|health|mem|cpu mem and cpu read from the container's cgroup
container.<name>.down should be up and isn't: Docker couldn't start it, it is
stuck restarting, or it died non-zero with no policy to
restart it — *not* a container someone stopped
updates.pending|security|reboot_required every 5 min
zfs.<pool>.health|healthy|size|used|used_pct|frag every 1 min
smart.<dev>.passed|warnings|power_on|wear every 10 min, never waking a sleeping disk
```
## Widgets
A widget is a spec — `{type, sensor, title?}` — over this API and nothing else
(`web/src/widgets.tsx`, fed by `useSensorStream` in `web/src/sensors.ts`). Types: `stat`,
`gauge`, `bar`, `spark`, and `list` for a pattern. Scale and colour come from the
sensor's own `min`/`max`/`warn`/`crit`.
- **In Ham Lab:** Dashboard → Customize → **+ Add widget**. Saved with the layout,
per machine; all of a page's widgets share one stream.
- **Widgets tab:** every widget type, then every sensor on the machine drawn once, live,
grouped by family (`web/src/Gallery.tsx`). Tap a tile to copy its sensor ID — the
quickest way to find what to put in a widget, a query or a Glance tile.
- **Anywhere else:** `/embed/widget` is one widget alone on a page, live, in its own small
bundle (the console never loads), and the only path Ham Lab allows to be framed.
```
/embed/widget?type=gauge&sensor=cpu.usage&token=<read-only token>
&machine=grid &title=… &theme=dark|light|cyberpunk|retro &every=<ms> &bare=1
```
### Glance
Glance's own `custom-api` widget refetches on page load / cache expiry — fine for slow
facts, never live. An `iframe` of the embed is live.
```yaml
- type: iframe
title: core CPU
source: https://hamlab.bigham.dev/embed/widget?type=gauge&sensor=cpu.usage&bare=1&token=<read-only token>
height: 130
- type: custom-api
title: grid
cache: 1m
url: https://hamlab.bigham.dev/api/nodes/grid/sensors/query?pool=zfs.tank.used_pct&failed=services.failed&updates=updates.pending
headers: { Authorization: Bearer <read-only token> }
template: |
<div>tank {{ .JSON.Float "pool" }}% · {{ .JSON.Int "failed" }} failed · {{ .JSON.Int "updates" }} updates</div>
```
## The whole fleet in one request
On a hub, `/api/fleet/sensors/query` asks every machine (or `machines`) the same shaped
query and answers `{machine: result}`; this machine answers under its own hostname, and
one that can't be reached answers `{"error": …}` while the rest still do.
```
POST /api/fleet/sensors/query {"machines": "*", "query": {"cpu": "cpu.usage", "up": "uptime"}}
GET /api/fleet/sensors/query?machines=grid,wyse&cpu=cpu.usage&failed=services.failed
→ {"core": {"cpu": 4.8, "up": 50211}, "grid": {"cpu": 0.4, "up": 437177}, "mac": {"error": "timeout"}}
```
## What counts as a disk
`disk.*` names **drives**, not the block devices the kernel lists. An NVMe controller can
present several namespaces — an Apple SSD shows `nvme0n1`, `nvme0n2` and `nvme0n3`, the last
two a few MiB of firmware — and each one arrives from `/proc/diskstats` looking like a whole
disk. They are folded into the controller (`disk.nvme0.*`), with the rates added and busy
taken as the longest of them rather than the sum, since they share one controller. This is
what the Disks page has always shown; since 2026-09-22 the dashboard, the Top tab and these
sensors agree with it.
## The Top tab
Ham Lab → **Top** is a whole machine on one screen in the style of btop: black ground, boxes
with their name set into the border, curves drawn as dots that climb green → yellow → red.
It reads nothing of its own — the sensor stream for everything measured, `/api/procs` for the
table — so it works on any machine the console can reach, whether or not btop is installed
there, and it follows the same refresh rate and history window as the rest of the console.
Its colours are the active skin's: the terminal background for the ground, the skin's sixteen
terminal colours for the curves and meters. (The look is btop's, `web/src/Top.tsx` is ours;
btop itself is Apache-2.0.)
## How often a machine samples
The console's refresh rate (Settings, or `− rate +` on Top) is also the fleet's sampling rate.
It is saved with the UI prefs on the hub, which sets its own sampler and passes the rate to
every machine it manages — `PUT /api/prefs/sampler {"ms": …}`, 100 ms to 60 s, sampled no
slower than the collector's 10 s — and repeats it at start-up. `GET` answers `{ms, min_ms,
max_ms}`. Before 2026-09-21 only an open Dashboard page told a machine its rate.
## How far back a history reaches
Every machine keeps **300 points per curve**, whatever the window — so the window sets how
far apart they are, not how much memory monitoring costs. At the default ten minutes each
point is one sample (2 s apart at the default rate); at a day each is the **average of the
five minutes behind it**, not whichever instant landed on the mark, so a long graph shows
the load that really happened. Ham Lab → Settings → **Graph history** is the one control,
1 minute to 24 hours; the hub passes it to every machine it manages.
The window can't be finer than the sampling interval — 300 points of a 1-minute window at a
2-second rate is 30 points, and that is what you get. It is also all in memory: a machine
that restarts starts its graphs again.
## What Ham Lab itself runs on
Since 2026-09-19 the Dashboard tab (stock modules and custom widgets) and the Disks tab's
live rates read only this API: the catalog once, one batch of histories, one stream —
closed while the page is hidden, refilled when it returns. `web/src/sampleAdapter.ts` is
the seam that hands the stock modules the object they were written against. The old
`/api/metrics` and `/api/metrics/history` are gone. `hamlab.sample_interval` is the
sampler's own rate, as a sensor.
## From another origin
The sensor endpoints (and only they) answer any origin: `Access-Control-Allow-Origin: *`,
preflight included. `*` is the one value browsers won't combine with cookies, so a foreign
page can reach the API with a token and can never ride a logged-in console's session.
`examples/external-demo/` is a complete outside consumer in one HTML file.
cmd/hamlab/main.go
+444 -24
View File
@@ -2,16 +2,24 @@
package main
import (
"bufio"
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"io/fs"
"log"
"net"
"net/http"
"os"
"os/signal"
"os/user"
"path/filepath"
"runtime"
"runtime/debug"
"strings"
"sync/atomic"
"syscall"
"time"
@@ -20,8 +28,20 @@ import (
"hamlab/internal/api"
"hamlab/internal/auth"
"hamlab/internal/config"
"hamlab/internal/cron"
"hamlab/internal/files"
"hamlab/internal/metrics"
"hamlab/internal/nodes"
"hamlab/internal/priv"
"hamlab/internal/services"
"hamlab/internal/setup"
"hamlab/internal/space"
"hamlab/internal/storage"
"hamlab/internal/store"
"hamlab/internal/tasks"
hterm "hamlab/internal/term"
"hamlab/internal/tlsauto"
"hamlab/internal/updates"
"hamlab/internal/web"
)
@@ -30,12 +50,25 @@ var version = "dev"
func main() {
defaultCfg := os.Getenv("HAMLAB_CONFIG")
if defaultCfg == "" {
home, _ := os.UserHomeDir()
defaultCfg = filepath.Join(home, "apps", "hamlab", "config.json")
defaultCfg = config.DefaultPath()
}
cfgPath := flag.String("config", defaultCfg, "path to config.json")
flag.Usage = func() {
fmt.Fprintf(os.Stderr, "usage: hamlab [-config path] [serve|passwd|version]\n")
fmt.Fprint(os.Stderr, `usage: hamlab [-config path] <command>
install [flags] install this binary as a service on this machine (install -h for flags)
uninstall [--purge] remove the service and the binary
serve run the console in the foreground (what the service runs)
passwd [user] set a user's password (makes the account if it is new; reads stdin when piped)
users list the accounts
zfs-snapshot <id> run one ZFS snapshot schedule (what its cron line calls)
docker-update <id> run one image-update schedule (what its cron line calls)
user-rm <user> remove an account and sign it out everywhere
check verify the config and state load, then exit
version
push-nodes [--force] hub: send this build to every managed machine
`)
flag.PrintDefaults()
}
flag.Parse()
@@ -44,6 +77,43 @@ func main() {
if flag.NArg() > 0 {
cmd = flag.Arg(0)
}
switch cmd { // commands that don't read the config
case "fs": // privileged helper for the Files tab (run as root by the daemon)
os.Exit(files.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
case "space-walk": // helper for the Space tab: measure mountpoints, stream the tree back
os.Exit(space.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
case "install", "uninstall":
var err error
if cmd == "install" {
err = setup.Install(flag.Args()[1:], version)
} else {
err = setup.Uninstall(flag.Args()[1:])
}
if errors.Is(err, flag.ErrHelp) {
os.Exit(0)
}
if err != nil {
fmt.Fprintln(os.Stderr, "hamlab "+cmd+":", err)
os.Exit(1)
}
return
case "replace": // self-update helper (run as root by the daemon): put this binary at a path, atomically
// Only ever over a hamlab binary: this runs as root, and must not be a way to
// overwrite arbitrary files for whoever is allowed to run hamlab as root.
if flag.NArg() != 2 || !filepath.IsAbs(flag.Arg(1)) || !strings.HasPrefix(filepath.Base(flag.Arg(1)), "hamlab") {
fmt.Fprintln(os.Stderr, "hamlab replace: destination must be an absolute path to a hamlab binary")
os.Exit(2)
}
self, err := os.Executable()
if err == nil {
err = setup.CopyAtomic(self, flag.Arg(1))
}
if err != nil {
fmt.Fprintln(os.Stderr, "hamlab replace:", err)
os.Exit(1)
}
return
}
cfg, err := config.Load(*cfgPath)
if err != nil {
log.Fatal(err)
@@ -51,10 +121,70 @@ func main() {
switch cmd {
case "version":
fmt.Println("hamlab", version)
case "passwd":
if err := passwd(cfg); err != nil {
case "check": // a new build runs this against the live config before it replaces the old one
if _, err := store.Open(filepath.Join(cfg.DataDir(), "state.json")); err != nil {
log.Fatal(err)
}
if _, err := fs.Stat(web.FS(), "index.html"); err != nil {
log.Fatal("embedded UI missing: ", err)
}
fmt.Println("ok")
case "passwd":
if err := passwd(cfg, flag.Arg(1)); err != nil {
log.Fatal(err)
}
case "docker-update":
if err := dockerUpdateJob(cfg, flag.Arg(1)); err != nil {
log.Fatal(err)
}
case "zfs-snapshot":
if err := zfsSnapshot(cfg, flag.Arg(1)); err != nil {
log.Fatal(err)
}
case "users":
migrateUsers(cfg)
if len(cfg.Users) == 0 {
fmt.Println("no accounts in", cfg.Path())
}
for _, u := range cfg.Users {
seen := "never"
if !u.LastSeen.IsZero() {
seen = u.LastSeen.Local().Format("2006-01-02 15:04")
}
note := ""
if u.Guest { // a temporary look-only login, not one of the machine's accounts
note = " guest, until " + u.Expires.Local().Format("15:04")
if u.Expired(time.Now()) {
note = " guest, expired"
}
}
fmt.Printf("%-20s created %s last seen %s%s\n", u.Name, u.Created.Local().Format("2006-01-02"), seen, note)
}
case "user-rm":
if err := userRm(cfg, flag.Arg(1)); err != nil {
log.Fatal(err)
}
case "push-nodes": // hub CLI: send this binary to every managed machine
exe, _ := os.Executable()
host, _ := os.Hostname()
nm := nodes.New(cfg, host)
force := flag.NArg() > 1 && flag.Arg(1) == "--force"
results := nm.PushAll(context.Background(), exe, runtime.GOARCH, force)
if len(results) == 0 {
fmt.Println("no managed machines in", cfg.Path())
}
failed := 0
for _, r := range results {
if r.OK {
fmt.Printf("%-12s updated to %s\n", r.Name, r.Version)
} else {
failed++
fmt.Printf("%-12s FAILED: %s\n", r.Name, r.Error)
}
}
if failed > 0 {
os.Exit(1)
}
case "serve":
if err := serve(cfg); err != nil {
log.Fatal(err)
@@ -65,40 +195,216 @@ func main() {
}
}
func passwd(cfg *config.Config) error {
fmt.Fprint(os.Stderr, "New password: ")
p1, err := term.ReadPassword(int(os.Stdin.Fd()))
fmt.Fprintln(os.Stderr)
// migrateUsers gives a pre-users config its first account, named after this OS user, so
// the CLI sees the same accounts the daemon will.
// dockerUpdateJob pulls a stack's images, and deploys them when the schedule says to. This
// is what the cron line Ham Lab writes actually calls, so image updates keep happening
// whether or not the console is up.
func dockerUpdateJob(cfg *config.Config, id string) error {
if strings.TrimSpace(id) == "" {
return errors.New("usage: hamlab docker-update <schedule-id>")
}
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
if err != nil {
return err
}
fmt.Fprint(os.Stderr, "Again: ")
p2, err := term.ReadPassword(int(os.Stdin.Fd()))
fmt.Fprintln(os.Stderr)
var raw json.RawMessage
st.View(func(s *store.State) { raw = s.DockerJobs })
var d struct {
Jobs []api.UpdateJob `json:"jobs"`
}
if len(raw) > 0 {
if err := json.Unmarshal(raw, &d); err != nil {
return fmt.Errorf("reading the schedules: %w", err)
}
}
for _, j := range d.Jobs {
if j.ID != id {
continue
}
// A pull that never finishes must not leave a cron job resident until the next reboot.
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Hour)
defer cancel()
out, runErr := api.RunUpdate(ctx, st, j, api.DockerTool())
rec := storage.RunRecord{At: time.Now(), Taken: j.Stack}
if runErr != nil {
rec.Error = runErr.Error()
}
if err := storage.WriteRun(cfg.DataDir(), "docker-"+id, rec); err != nil {
fmt.Fprintln(os.Stderr, "recording the run:", err)
}
fmt.Print(out)
return runErr
}
return fmt.Errorf("no image-update schedule %q — it may have been removed; this cron line is stale", id)
}
// zfsSnapshot runs one snapshot schedule: take it, then expire what has aged out. This is
// what the cron line Ham Lab writes actually calls, so a schedule keeps running whether or
// not the console is up — and keeps running if Ham Lab is never opened again.
//
// It deliberately does not write state.json. The daemon owns that file and would clobber
// anything written from out here; the outcome goes beside it instead, in zfs-runs/.
func zfsSnapshot(cfg *config.Config, id string) error {
if strings.TrimSpace(id) == "" {
return errors.New("usage: hamlab zfs-snapshot <schedule-id>")
}
st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
if err != nil {
return err
}
if string(p1) != string(p2) {
return errors.New("passwords do not match")
var raw json.RawMessage
st.View(func(s *store.State) { raw = s.ZFS })
var z struct {
Policies []storage.Policy `json:"policies"`
}
if len(p1) < 12 {
if len(raw) > 0 {
if err := json.Unmarshal(raw, &z); err != nil {
return fmt.Errorf("reading the schedules: %w", err)
}
}
var p *storage.Policy
for i := range z.Policies {
if z.Policies[i].ID == id {
p = &z.Policies[i]
}
}
if p == nil {
return fmt.Errorf("no snapshot schedule %q — it may have been removed; this cron line is stale", id)
}
res, runErr := storage.New(priv.Available()).Run(context.Background(), p)
rec := storage.RunRecord{At: time.Now(), Taken: res.Taken, Pruned: res.Pruned}
if runErr != nil {
rec.Error = runErr.Error()
}
if err := storage.WriteRun(cfg.DataDir(), id, rec); err != nil {
fmt.Fprintln(os.Stderr, "recording the run:", err)
}
if runErr != nil {
return runErr
}
fmt.Printf("%s: took %s", p.Name, res.Taken)
if len(res.Pruned) > 0 {
fmt.Printf(", expired %d", len(res.Pruned))
}
fmt.Println()
return nil
}
func migrateUsers(cfg *config.Config) {
name := os.Getenv("USER")
if u, err := user.Current(); err == nil && u.Username != "" {
name = u.Username
}
if cfg.MigrateUsers(name) {
_ = cfg.Save()
}
}
func userRm(cfg *config.Config, name string) error {
migrateUsers(cfg)
if name == "" {
return errors.New("usage: hamlab user-rm <user>")
}
if len(cfg.Users) <= 1 {
return errors.New("the last account can't be removed")
}
kept := cfg.Users[:0]
found := false
for _, u := range cfg.Users {
if u.Name == name {
found = true
continue
}
kept = append(kept, u)
}
if !found {
return fmt.Errorf("no account %q", name)
}
cfg.Users = kept
if err := cfg.Save(); err != nil {
return err
}
if st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json")); err == nil {
auth.NewSessions(st).RevokeUser(name)
_ = st.Update(func(s *store.State) { delete(s.Users, name) })
}
fmt.Fprintln(os.Stderr, "removed", name, "(the running service picks it up at its next config save; restart it to be sure)")
return nil
}
func passwd(cfg *config.Config, name string) error {
migrateUsers(cfg)
if name == "" {
switch len(cfg.Users) {
case 0:
name = os.Getenv("USER")
if u, err := user.Current(); err == nil && u.Username != "" {
name = u.Username
}
case 1:
name = cfg.Users[0].Name
default:
return errors.New("several accounts: hamlab passwd <user>")
}
}
if !auth.UserNameRe.MatchString(name) {
return fmt.Errorf("%q: a user name is 1–32 lower-case letters, digits, . _ or -", name)
}
var p1 []byte
if !term.IsTerminal(int(os.Stdin.Fd())) { // scripted: one line on stdin
line, _ := bufio.NewReader(os.Stdin).ReadString('\n')
p1 = []byte(strings.TrimRight(line, "\r\n"))
} else {
fmt.Fprintf(os.Stderr, "New password for %s: ", name)
var err error
p1, err = term.ReadPassword(int(os.Stdin.Fd()))
fmt.Fprintln(os.Stderr)
if err != nil {
return err
}
fmt.Fprint(os.Stderr, "Again: ")
p2, err := term.ReadPassword(int(os.Stdin.Fd()))
fmt.Fprintln(os.Stderr)
if err != nil {
return err
}
if string(p1) != string(p2) {
return errors.New("passwords do not match")
}
}
if len(p1) < auth.MinPasswordLen {
return errors.New("password must be at least 12 characters")
}
h, err := auth.HashPassword(string(p1))
if err != nil {
return err
}
cfg.PasswordHash = h
made := true
for i := range cfg.Users {
if cfg.Users[i].Name == name {
cfg.Users[i].Hash = h
made = false
}
}
if made {
cfg.Users = append(cfg.Users, config.User{Name: name, Hash: h, Created: time.Now()})
}
if err := cfg.Save(); err != nil {
return err
}
fmt.Fprintln(os.Stderr, "password set in", cfg.Path())
if made {
fmt.Fprintln(os.Stderr, "account", name, "made in", cfg.Path())
} else {
fmt.Fprintln(os.Stderr, "password of", name, "set in", cfg.Path())
}
fmt.Fprintln(os.Stderr, "the running service reads the config at its next save; restart it to be sure")
return nil
}
func serve(cfg *config.Config) error {
if cfg.PasswordHash == "" {
return fmt.Errorf("no password set — run: hamlab -config %s passwd", cfg.Path())
if cfg.PasswordHash == "" && len(cfg.Users) == 0 {
return fmt.Errorf("no account set — run: hamlab -config %s passwd", cfg.Path())
}
if err := os.MkdirAll(cfg.DataDir(), 0o700); err != nil {
return err
@@ -115,14 +421,16 @@ func serve(cfg *config.Config) error {
}
shell := os.Getenv("SHELL")
if shell == "" {
shell = "/bin/bash"
shell = loginShell(u.Username)
}
mgr := hterm.NewManager(shell, u.HomeDir)
mgr.OnChange = func(list []hterm.Info) {
_ = st.Update(func(s *store.State) {
s.Terminals = s.Terminals[:0]
for _, t := range list {
s.Terminals = append(s.Terminals, store.Terminal{ID: t.ID, Name: t.Name})
if t.Running { // an exited shell must not come back from the dead on restart
s.Terminals = append(s.Terminals, store.Terminal{ID: t.ID, Name: t.Name})
}
}
})
}
@@ -136,12 +444,57 @@ func serve(cfg *config.Config) error {
}
}
srv := api.New(cfg, sessions, mgr, web.FS(), version)
// Keep the heap tidy: the daemon's live data is tens of MB; without a limit the GC
// happily lets RSS coast at twice that. (Soft limit: exceeding it only means more GC.)
debug.SetMemoryLimit(160 << 20)
col := metrics.NewCollector()
go col.Run(context.Background())
svc := services.New()
cr := cron.New(svc.Sudo(), svc.JournalSudo())
tm := tasks.New()
up := updates.New(svc.Sudo(), tm)
sto := storage.New(svc.Sudo())
self, _ := os.Executable()
var sudoFS files.FS
if svc.Sudo() && self != "" {
sudoFS = files.Sudo{Self: self}
}
host, _ := os.Hostname()
nm := nodes.New(cfg, host)
go nm.Run(context.Background())
certPath, keyPath, fingerprint := "", "", ""
if cfg.TLS == "auto" {
var err error
certPath, keyPath, fingerprint, err = tlsauto.Ensure(cfg.DataDir())
if err != nil {
return fmt.Errorf("tls: %w", err)
}
}
ln, err := listen(cfg.Listen)
if err != nil {
return err
}
srv := api.New(cfg, sessions, mgr, col, st, svc, cr, tm, up, sto, files.Local{}, sudoFS, nm, fingerprint, web.FS(), version)
hs := &http.Server{
Addr: cfg.Listen,
Handler: srv.Handler(),
ReadHeaderTimeout: 10 * time.Second,
}
// After a self-update: become the new binary in place. Same PID, so it works the same
// under systemd, OpenRC, or no supervisor at all. It returns only if the exec failed, by
// which point the listener and the shells are gone; the caller puts the old binary back
// and exits so the supervisor starts it.
var restarting atomic.Bool
srv.Restart = func() error {
restarting.Store(true) // Serve returns once Shutdown is called; serve() must not exit under us
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
_ = hs.Shutdown(ctx)
mgr.Shutdown()
return syscall.Exec(self, os.Args, os.Environ())
}
go func() {
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
@@ -152,9 +505,76 @@ func serve(cfg *config.Config) error {
_ = hs.Shutdown(ctx)
mgr.Shutdown()
}()
if certPath != "" {
log.Printf("hamlab %s listening on https://%s as %s (shell %s) cert %s", version, cfg.Listen, u.Username, shell, fingerprint)
if err := hs.ServeTLS(ln, certPath, keyPath); !errors.Is(err, http.ErrServerClosed) {
return err
}
return waitIf(&restarting)
}
log.Printf("hamlab %s listening on %s as %s (shell %s)", version, cfg.Listen, u.Username, shell)
if err := hs.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
if err := hs.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
return err
}
return waitIf(&restarting)
}
// waitIf parks the caller while a restart is replacing the process.
func waitIf(restarting *atomic.Bool) error {
if restarting.Load() {
select {}
}
return nil
}
// listen binds the configured address with IP_FREEBIND, so a LAN address that isn't up yet
// (boot, DHCP still negotiating, cable out) doesn't stop the daemon: it binds now and
// answers as soon as the address appears.
func listen(addr string) (net.Listener, error) {
lc := net.ListenConfig{Control: func(network, _ string, c syscall.RawConn) error {
return c.Control(func(fd uintptr) {
_ = syscall.SetsockoptInt(int(fd), syscall.IPPROTO_IP, syscall.IP_FREEBIND, 1)
})
}}
ln, err := lc.Listen(context.Background(), "tcp", addr)
if err != nil {
return nil, fmt.Errorf("listen %s: %w", addr, err)
}
if host, _, _ := net.SplitHostPort(addr); host != "" {
if ip := net.ParseIP(host); ip != nil && !ip.IsUnspecified() && !ip.IsLoopback() && !onInterface(ip) {
log.Printf("listen address %s isn't on any interface yet; bound anyway and waiting for it", host)
}
}
return ln, nil
}
func onInterface(ip net.IP) bool {
addrs, _ := net.InterfaceAddrs()
for _, a := range addrs {
if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
return true
}
}
return false
}
// loginShell returns the user's shell from /etc/passwd, falling back to bash if
// present and /bin/sh otherwise. Systemd units don't export $SHELL.
func loginShell(username string) string {
if b, err := os.ReadFile("/etc/passwd"); err == nil {
for _, line := range strings.Split(string(b), "\n") {
f := strings.Split(line, ":")
if len(f) >= 7 && f[0] == username && f[6] != "" {
if _, err := os.Stat(f[6]); err == nil {
return f[6]
}
}
}
}
for _, sh := range []string{"/bin/bash", "/usr/bin/bash", "/bin/zsh", "/bin/sh"} {
if _, err := os.Stat(sh); err == nil {
return sh
}
}
return "/bin/sh"
}
deploy/hamlab.service
-18
View File
@@ -1,18 +0,0 @@
[Unit]
Description=Ham Lab web console
After=network.target
[Service]
Type=simple
User=core
Group=core
WorkingDirectory=/home/core
Environment=HAMLAB_CONFIG=/home/core/apps/hamlab/config.json
ExecStart=/usr/local/bin/hamlab serve
Restart=always
RestartSec=3
KillMode=control-group
TimeoutStopSec=10
[Install]
WantedBy=multi-user.target
deploy/install.sh
-11
View File
@@ -1,11 +0,0 @@
#!/bin/sh
# Install the built binary and unit on this host, then (re)start the service.
set -eu
cd "$(dirname "$0")/.."
sudo install -m 0755 bin/hamlab /usr/local/bin/hamlab
sudo install -m 0644 deploy/hamlab.service /etc/systemd/system/hamlab.service
sudo systemctl daemon-reload
sudo systemctl enable --now hamlab.service >/dev/null
sudo systemctl restart hamlab.service
sleep 1
systemctl is-active hamlab.service && /usr/local/bin/hamlab version
examples/external-demo/README.md
+21
View File
@@ -0,0 +1,21 @@
# External demo
`index.html` is a stranger's page: one file, no build, no framework, served from a
different origin than Ham Lab. It knows a base URL and a read-only sensor token
(`config.js`, not in git) and nothing else. Each tab is one shaped query POSTed to
`/api/sensors/query?full=1` on an interval; sparklines are seeded from
`/api/sensors/history`; the Fleet tab is one call to `/api/fleet/sensors/query`.
```js
// config.js
window.HAMLAB = { base: 'http://10.0.0.234:8100', token: '<read-only sensor token>' }
```
Anyone who can load the page can read the token. If the page is public, limit the token to
patterns that name nothing you'd mind the world reading — hardware numbers and counts,
not container or service names.
This is an example to read and copy, not something that runs anywhere. It was deployed on
`core` as the `hamlab-demo` stack and linked from the Widgets tab until 2026-09-23, when
both were removed: the Widgets tab shows the same widgets against the machine you are
already signed in to, so a second public copy earned nothing and cost a public token.
examples/external-demo/index.html
+179
View File
@@ -0,0 +1,179 @@
<!doctype html>
<!--
Ham Lab sensor API — external demo.
One file, no build, no framework, not part of Ham Lab: it is a stranger's page on another
origin that knows only a base URL and a read-only token (config.js, beside this file):
window.HAMLAB = { base: 'http://10.0.0.234:8100', token: '<read-only sensor token>' }
Every tab is one shaped query — the object it wants, with sensor IDs where the values go —
POSTed to /api/sensors/query?full=1 on an interval. Sparklines are seeded once from
/api/sensors/history. The Fleet tab uses /api/fleet/sensors/query. See SENSORS.md.
-->
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Ham Lab sensors — external demo</title>
<script src="config.js"></script>
<style>
:root { --bg:#0f1216; --card:#171b21; --line:#2a313b; --fg:#d5dbe2; --mut:#7d8590; --acc:#e3a33b; --s1:#3987e5; --s2:#c98500; --bad:#ff7b72; --ok:#3fb950; --track:rgba(57,135,229,.18) }
* { box-sizing: border-box }
body { margin:0; background:var(--bg); color:var(--fg); font:14px/1.4 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif }
header { display:flex; align-items:center; gap:10px; flex-wrap:wrap; padding:10px 14px; background:var(--card); border-bottom:1px solid var(--line) }
header h1 { margin:0; font-size:15px; color:var(--acc); letter-spacing:.4px }
header .sp { flex:1 }
.mut { color:var(--mut) } .mono { font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace }
select { background:var(--bg); color:var(--fg); border:1px solid var(--line); border-radius:5px; padding:4px 6px; font:inherit }
.dot { width:8px; height:8px; border-radius:50%; background:var(--mut); display:inline-block } .dot.ok { background:var(--ok) } .dot.bad { background:var(--bad) }
nav { display:flex; gap:2px; overflow-x:auto; padding:0 8px; background:var(--card); border-bottom:1px solid var(--line); scrollbar-width:none }
nav button { flex:0 0 auto; background:none; border:0; border-bottom:2px solid transparent; color:var(--mut); padding:10px 12px; font:inherit; cursor:pointer }
nav button.on { color:var(--fg); border-bottom-color:var(--acc) }
main { padding:12px; max-width:1100px; margin:0 auto }
.grid { display:grid; grid-template-columns:repeat(auto-fill,minmax(170px,1fr)); gap:10px }
.tile { background:var(--card); border:1px solid var(--line); border-radius:8px; padding:12px; min-width:0; overflow:hidden }
.tile.wide { grid-column:1/-1 }
.label { font-size:11px; font-weight:600; color:var(--mut); text-transform:uppercase; letter-spacing:.5px; margin-bottom:6px; overflow-wrap:anywhere }
.value { font-size:22px; font-weight:600; font-variant-numeric:tabular-nums } .value.warn { color:var(--s2) } .value.bad { color:var(--bad) }
.id { margin-top:8px; font-size:10.5px; color:var(--mut); overflow-wrap:anywhere }
.meter { height:6px; background:var(--track); border-radius:3px; overflow:hidden; margin-top:8px } .meter i { display:block; height:100%; background:var(--s1); border-radius:3px; transition:width .3s } .meter i.warn { background:var(--s2) } .meter i.bad { background:var(--bad) }
svg.gauge { display:block; width:100%; max-width:180px; margin:0 auto } svg.gauge text { fill:var(--fg); font-size:13px; font-weight:600 }
svg.spark { display:block; width:100%; height:44px; margin-top:6px }
.rows { display:flex; flex-direction:column; gap:7px } .row { display:grid; grid-template-columns:1fr auto; gap:2px 10px; font-size:13px; align-items:baseline }
.row .k { overflow:hidden; text-overflow:ellipsis; white-space:nowrap } .row .n { font-variant-numeric:tabular-nums } .row .meter { grid-column:1/3; margin-top:0 }
.pill { font-size:11px; padding:1px 7px; border-radius:9px; border:1px solid var(--line) } .pill.ok { color:var(--ok); border-color:var(--ok) } .pill.bad { color:var(--bad); border-color:var(--bad) }
details { margin-top:14px; background:var(--card); border:1px solid var(--line); border-radius:8px; padding:10px 12px } summary { cursor:pointer; color:var(--mut) }
pre { margin:8px 0 0; font:12px/1.5 ui-monospace,Menlo,Consolas,monospace; white-space:pre-wrap; word-break:break-word; max-height:320px; overflow:auto }
.err { color:var(--bad); padding:12px }
table { width:100%; border-collapse:collapse; font-size:13px } th, td { text-align:left; padding:6px 8px; border-bottom:1px solid var(--line); font-variant-numeric:tabular-nums } th { color:var(--mut); font-weight:600; font-size:11px; text-transform:uppercase }
</style>
</head>
<body>
<header>
<h1>Ham Lab sensors</h1><span class="mut">external demo · <span id="base" class="mono"></span></span>
<span class="sp"></span>
<span id="status" class="mut"></span><span id="dot" class="dot"></span>
<label class="mut">every <select id="every"><option value="1000">1 s</option><option value="2000" selected>2 s</option><option value="5000">5 s</option><option value="10000">10 s</option><option value="30000">30 s</option></select></label>
</header>
<nav id="tabs"></nav>
<main>
<div id="view"></div>
<details><summary>What this tab asked, and what came back</summary><pre id="wire"></pre></details>
</main>
<script>
const CFG = window.HAMLAB || {}
const $ = (id) => document.getElementById(id)
const h = (tag, attrs = {}, ...kids) => { const e = document.createElementNS(tag === 'svg' || attrs.svg ? 'http://www.w3.org/2000/svg' : 'http://www.w3.org/1999/xhtml', tag); for (const [k, v] of Object.entries(attrs)) if (k !== 'svg' && v != null) k === 'class' ? e.setAttribute('class', v) : k === 'style' ? e.setAttribute('style', v) : e.setAttribute(k, v); for (const kid of kids.flat()) if (kid != null) e.append(kid); return e }
// ---- the API: two calls ----
async function api(path, body) {
const r = await fetch(CFG.base + path, { method: body ? 'POST' : 'GET', headers: { Authorization: 'Bearer ' + CFG.token, ...(body ? { 'Content-Type': 'application/json' } : {}) }, body: body ? JSON.stringify(body) : undefined })
if (!r.ok) throw new Error(r.status === 401 ? 'token rejected (401)' : `${r.status} ${r.statusText}`)
return r.json()
}
// ---- formatting from the sensor's own unit ----
const U = ['B', 'KiB', 'MiB', 'GiB', 'TiB']
const bytes = (v) => { let i = 0; while (v >= 1024 && i < U.length - 1) { v /= 1024; i++ } return `${v >= 100 || i === 0 ? v.toFixed(0) : v.toFixed(1)} ${U[i]}` }
const dur = (s) => { const d = Math.floor(s / 86400), hh = Math.floor(s % 86400 / 3600), m = Math.floor(s % 3600 / 60); return d ? `${d}d ${hh}h` : hh ? `${hh}h ${m}m` : `${m}m` }
function fmt(s) {
if (!s || s.value == null) return '—'
const v = s.value
if (typeof v === 'boolean') return v ? 'yes' : 'no'
if (typeof v === 'string') return v || '—'
switch (s.unit) { case 'B': return bytes(v); case 'B/s': return bytes(v) + '/s'; case 's': return dur(v); case 'min': return dur(v * 60); case '%': return v.toFixed(v < 10 ? 1 : 0) + '%'; case '°C': return v.toFixed(0) + '°C'; case 'rpm': return Math.round(v) + ' rpm'; case undefined: case '': return String(Math.round(v * 100) / 100); default: return `${Math.round(v * 10) / 10} ${s.unit}` }
}
const frac = (s) => { if (!s || typeof s.value !== 'number') return null; const max = s.max ?? s.crit ?? s.warn, min = s.min ?? 0; return max === undefined || max <= min ? null : Math.min(1, Math.max(0, (s.value - min) / (max - min))) }
const sev = (s) => !s || typeof s.value !== 'number' ? '' : s.crit !== undefined && s.value >= s.crit ? 'bad' : s.warn !== undefined && s.value >= s.warn ? 'warn' : s.kind === 'usage' && s.id !== 'ups.battery' && s.value >= 95 ? 'bad' : s.kind === 'usage' && s.id !== 'ups.battery' && s.value >= 85 ? 'warn' : ''
// ---- the widgets ----
const tile = (label, id, ...body) => h('div', { class: 'tile' }, h('div', { class: 'label' }, label), ...body, id ? h('div', { class: 'id mono' }, id) : null)
const missing = (label, id) => tile(label, id, h('div', { class: 'mut' }, 'not on this machine'))
const W = {
stat: (s, label, id) => s ? tile(label || s.name, s.id, h('div', { class: 'value ' + sev(s) }, fmt(s))) : missing(label, id),
bar: (s, label, id) => { if (!s) return missing(label, id); const f = frac(s); return tile(label || s.name, s.id, h('div', { class: 'value ' + sev(s) }, fmt(s)), f === null ? null : h('div', { class: 'meter' }, h('i', { class: sev(s), style: `width:${f * 100}%` }))) },
gauge: (s, label, id) => {
if (!s) return missing(label, id)
const f = frac(s) ?? 0, C = Math.PI * 40, col = { bad: 'var(--bad)', warn: 'var(--s2)' }[sev(s)] || 'var(--s1)', arc = 'M10 50 A40 40 0 0 1 90 50'
return tile(label || s.name, s.id, h('svg', { class: 'gauge', viewBox: '0 0 100 56' },
h('path', { svg: 1, d: arc, fill: 'none', stroke: 'var(--track)', 'stroke-width': 9, 'stroke-linecap': 'round' }),
h('path', { svg: 1, d: arc, fill: 'none', stroke: col, 'stroke-width': 9, 'stroke-linecap': 'round', 'stroke-dasharray': `${f * C} ${C}` }),
h('text', { svg: 1, x: 50, y: 47, 'text-anchor': 'middle' }, fmt(s))))
},
spark: (s, label, id) => {
if (!s) return missing(label, id)
const pts = (hist[s.id] ||= []); if (typeof s.value === 'number') { pts.push(s.value); if (pts.length > 300) pts.shift() }
const max = s.unit === '%' ? 100 : Math.max(...pts, 1e-9) * 1.1, n = pts.length
const line = n < 2 ? '' : pts.map((v, i) => `${(i / (n - 1) * 300).toFixed(1)},${(42 - v / max * 40).toFixed(1)}`).join(' ')
return tile(label || s.name, s.id, h('div', { class: 'value' }, fmt(s)), h('svg', { class: 'spark', viewBox: '0 0 300 44', preserveAspectRatio: 'none' }, h('polyline', { svg: 1, points: line, fill: 'none', stroke: 'var(--s1)', 'stroke-width': 2, 'vector-effect': 'non-scaling-stroke' })))
},
// a glob comes back as {id: sensor}: a list draws them all
list: (map, label, pattern, opts = {}) => {
const rows = Object.values(map || {}).filter((s) => !opts.filter || opts.filter(s)).sort((a, b) => a.id.localeCompare(b.id))
return h('div', { class: 'tile wide' }, h('div', { class: 'label' }, `${label} · ${rows.length}`),
rows.length ? h('div', { class: 'rows' }, rows.map((s) => { const f = frac(s); return h('div', { class: 'row' }, h('span', { class: 'k', title: s.id }, opts.name ? opts.name(s) : s.name), typeof s.value === 'boolean' ? h('span', { class: 'pill ' + (s.value ? 'ok' : 'bad') }, s.value ? 'up' : 'down') : h('span', { class: 'n' }, fmt(s)), f === null ? null : h('div', { class: 'meter' }, h('i', { class: sev(s), style: `width:${f * 100}%` }))) })) : h('div', { class: 'mut' }, 'nothing here that this token may see'),
h('div', { class: 'id mono' }, pattern))
},
}
// ---- the tabs: a query, the histories to seed, and how to draw the answer ----
const TABS = [
{ name: 'Overview', query: { cpu: 'cpu.usage', mem: 'mem.used_pct', memB: 'mem.used', root: 'fs.root.used_pct', ups: 'ups.battery', temp: 'cpu.temp', load: 'load.1', up: 'uptime', failed: 'services.failed', upd: 'updates.pending', cont: 'containers.running' },
seed: 'cpu.usage,load.1',
draw: (d) => [W.gauge(d.cpu, 'CPU'), W.gauge(d.mem, 'Memory'), W.gauge(d.root, 'Root disk'), W.gauge(d.ups, 'UPS battery', 'ups.battery'), W.spark(d.cpu, 'CPU, recent'), W.spark(d.load, 'Load 1 min'), W.bar(d.memB, 'Memory used'), W.stat(d.temp, 'CPU temp', 'cpu.temp'), W.stat(d.up, 'Uptime'), W.stat(d.cont, 'Containers up', 'containers.running'), W.stat(d.failed, 'Failed services'), W.stat(d.upd, 'Pending updates')] },
{ name: 'Widget types', query: { s: 'cpu.usage', cores: 'cpu.core.*.usage' }, seed: 'cpu.usage',
draw: (d) => [W.stat(d.s, 'stat — a number'), W.gauge(d.s, 'gauge — has a scale'), W.bar(d.s, 'bar — has a scale'), W.spark(d.s, 'spark — has a past'), W.list(d.cores, 'list — a pattern', 'cpu.core.*.usage')] },
{ name: 'Hardware', query: { cores: 'cpu.core.*.usage', temps: 'temp.*', fans: 'fan.*', smart: 'smart.*.passed', wear: 'smart.*.wear' },
draw: (d) => [W.list(d.cores, 'CPU cores', 'cpu.core.*.usage'), W.list(d.temps, 'Temperatures', 'temp.*'), W.list(d.fans, 'Fans', 'fan.*'), W.list(d.smart, 'Drive health', 'smart.*.passed', { name: (s) => s.labels?.device + ' · ' + (s.labels?.model || '') }), W.list(d.wear, 'SSD endurance used', 'smart.*.wear')] },
{ name: 'Network & disks', query: { rx: 'net.*.rx', tx: 'net.*.tx', rd: 'disk.*.read', wr: 'disk.*.write', fs: 'fs.*.used_pct', fsb: 'fs.*.used' }, seed: 'net.*.rx,net.*.tx,disk.*.read,disk.*.write',
draw: (d) => [...Object.values(d.rx).map((s) => W.spark(s)), ...Object.values(d.tx).map((s) => W.spark(s)), ...Object.values(d.rd).map((s) => W.spark(s)), ...Object.values(d.wr).map((s) => W.spark(s)), W.list(d.fs, 'Filesystems', 'fs.*.used_pct', { name: (s) => s.labels?.mount }), W.list(d.fsb, 'Filesystems, bytes', 'fs.*.used', { name: (s) => s.labels?.mount })] },
{ name: 'Containers', query: { total: 'containers.total', run: 'containers.running', bad: 'containers.unhealthy', up: 'container.*.running', mem: 'container.*.mem', cpu: 'container.*.cpu' },
draw: (d) => [W.stat(d.run, 'Running'), W.stat(d.total, 'Total'), W.stat(d.bad, 'Unhealthy'), W.list(d.up, 'Containers', 'container.*.running', { name: (s) => s.labels?.container }), W.list(d.mem, 'Memory', 'container.*.mem', { name: (s) => s.labels?.container }), W.list(d.cpu, 'CPU, % of one core', 'container.*.cpu', { name: (s) => s.labels?.container })] },
{ name: 'Services', query: { failed: 'services.failed', names: 'services.failed_names', running: 'services.running', reboot: 'updates.reboot_required', each: 'service.*.active' },
draw: (d) => [W.stat(d.failed, 'Failed'), W.stat(d.running, 'Running'), W.stat(d.names, 'Which failed'), W.stat(d.reboot, 'Reboot required'), W.list(d.each, 'Every service', 'service.*.active')] },
{ name: 'Fleet', fleet: true, query: { cpu: 'cpu.usage', mem: 'mem.used_pct', temp: 'cpu.temp', up: 'uptime', failed: 'services.failed', cont: 'containers.running', pool: 'zfs.*.health' },
draw: (d) => [h('div', { class: 'tile wide' }, h('div', { class: 'label' }, 'one request, every machine'), h('table', {}, h('tr', {}, ['Machine', 'CPU', 'Memory', 'CPU temp', 'Uptime', 'Failed', 'Containers', 'Pools'].map((t) => h('th', {}, t))),
Object.entries(d).sort().map(([m, r]) => r.error ? h('tr', {}, h('td', {}, m), h('td', { colspan: 7, class: 'mut' }, r.error)) : h('tr', {}, h('td', {}, m), ...[r.cpu, r.mem, r.temp, r.up, r.failed, r.cont].map((s) => h('td', {}, fmt(s))), h('td', {}, Object.values(r.pool || {}).map((s) => `${s.labels?.pool} ${s.value}`).join(', ') || '—')))), h('div', { class: 'id mono' }, 'POST /api/fleet/sensors/query'))] },
]
// ---- the loop ----
const hist = {}
let tab = 0, timer = 0, seq = 0
async function show(i) {
tab = i; const my = ++seq; clearTimeout(timer)
;[...$('tabs').children].forEach((b, j) => b.classList.toggle('on', j === i))
$('view').replaceChildren(h('div', { class: 'mut' }, 'asking…'))
const t = TABS[i]
if (t.seed) try { for (const s of (await api('/api/sensors/history?ids=' + encodeURIComponent(t.seed))).series) hist[s.id] = s.v.slice(-300) } catch { /* sparklines just start empty */ }
if (my === seq) tick(my)
}
async function tick(my) {
if (my !== seq) return
const t = TABS[tab], t0 = performance.now()
try {
const path = (t.fleet ? '/api/fleet/sensors/query' : '/api/sensors/query') + '?full=1', body = t.fleet ? { machines: '*', query: t.query } : t.query
const data = await api(path, body)
if (my !== seq) return
$('view').replaceChildren(h('div', { class: 'grid' }, t.draw(data)))
$('wire').textContent = `POST ${path}\n${JSON.stringify(body, null, 1)}\n\n→ ${JSON.stringify(data, (k, v) => (v && typeof v === 'object' && 'id' in v && 'value' in v ? { id: v.id, value: v.value, unit: v.unit } : v), 1).slice(0, 6000)}`
$('dot').className = 'dot ok'; $('status').textContent = `${Math.round(performance.now() - t0)} ms · ${new Date().toLocaleTimeString()}`
} catch (e) {
if (my !== seq) return
$('dot').className = 'dot bad'; $('status').textContent = e.message
if (!$('view').querySelector('.grid')) $('view').replaceChildren(h('div', { class: 'err' }, e.message))
}
timer = setTimeout(() => tick(my), document.hidden ? 5000 : Number($('every').value))
}
if (!CFG.base || !CFG.token) $('view').replaceChildren(h('div', { class: 'err' }, 'config.js is missing: window.HAMLAB = { base, token }'))
else {
$('base').textContent = CFG.base
TABS.forEach((t, i) => $('tabs').append(h('button', {}, t.name))); [...$('tabs').children].forEach((b, i) => b.onclick = () => show(i))
$('every').onchange = () => { clearTimeout(timer); tick(seq) }
show(0)
}
</script>
</body>
</html>
install.sh
Executable
+382
View File
@@ -0,0 +1,382 @@
#!/bin/sh
# Ham Lab installer.
#
# curl -fsSL https://git.bigham.dev/mitchell/hamlab/releases/download/latest/install.sh | sh
#
# With a terminal it walks you through it: who the console runs as, where it listens,
# HTTPS, the firewall, the password — then downloads the build for this machine, checks
# it against SHA256SUMS and hands over to `hamlab install`, which sets up the binary,
# the config and a service that starts at boot. Re-running it upgrades in place.
#
# Without a terminal, with --yes, or with flags after `--`, it asks nothing:
#
# curl -fsSL …/install.sh | sh -s -- --yes
# curl -fsSL …/install.sh | sh -s -- --listen 0.0.0.0:8100 --tls auto --password-stdin <pw
# ./install.sh # beside the hamlab-linux-* files: installs offline
#
# HAMLAB_URL where hamlab-linux-<arch>, SHA256SUMS and VERSION are served from —
# by default the newest release on Gitea; "latest" is an alias Gitea
# resolves to the most recent release, so the URL never needs changing —
# any static file host, or a Gitea/Forgejo release's download URL
# HAMLAB_TOKEN access token, when that place is a private Gitea/Forgejo repo
# NO_COLOR plain output
#
# POSIX sh: runs under dash, busybox ash and bash alike.
set -eu
HAMLAB_URL="${HAMLAB_URL:-https://git.bigham.dev/mitchell/hamlab/releases/download/latest}"
BUILT_FOR="@VERSION@" # filled in by `make release`
# ─────────────────────────────────────────────────────────────── looks ──
if [ -t 2 ] && [ -z "${NO_COLOR:-}" ] && [ "${TERM:-dumb}" != dumb ]; then
B=$(printf '\033[1m'); D=$(printf '\033[2m'); R=$(printf '\033[0m')
AMBER=$(printf '\033[38;5;214m'); GREEN=$(printf '\033[32m'); RED=$(printf '\033[31m'); CYAN=$(printf '\033[36m')
else
B=; D=; R=; AMBER=; GREEN=; RED=; CYAN=
fi
case "${LC_ALL:-${LC_CTYPE:-${LANG:-}}}" in
*UTF-8*|*utf8*|*UTF8*|*utf-8*) OK='✓'; NO='✗'; DOT='•'; ARROW='›'; HR='────────────────────────────────────────────────────────────' ;;
*) OK='ok'; NO='x'; DOT='*'; ARROW='>'; HR='------------------------------------------------------------' ;;
esac
out() { printf '%s\n' "$*" >&2; }
blank() { printf '\n' >&2; }
good() { out " ${GREEN}${OK}${R} $*"; }
note() { out " ${D}$*${R}"; }
warn() { out " ${AMBER}!${R} $*"; }
die() { blank; out " ${RED}${NO}${R} ${B}$1${R}"; shift; for more in "$@"; do out " $more"; done; blank; exit 1; }
rule() { out "${D}${HR}${R}"; }
step() { blank; out "${AMBER}${B}$1${R} ${B}$2${R}"; [ -z "${3:-}" ] || note "$3"; blank; }
banner() {
blank
out " ${AMBER}${B}Ham Lab${R} ${D}a web console for the machines you run${R}"
out " ${D}terminals ${DOT} monitors ${DOT} services ${DOT} files ${DOT} updates ${DOT} disks — in one tab${R}"
rule
}
# ─────────────────────────────────────────────────────────── questions ──
# Piped from curl, stdin is this script, so every answer comes from the terminal itself.
TTY=/dev/tty
have_tty() { [ -r "$TTY" ] && [ -w "$TTY" ] && (: <"$TTY") 2>/dev/null; }
ask() { # ask "Question" "default" → $REPLY
if [ -n "${2:-}" ]; then printf ' %s %s[%s]%s ' "$1" "$D" "$2" "$R" >&2; else printf ' %s ' "$1" >&2; fi
IFS= read -r REPLY <"$TTY" || REPLY=
[ -n "$REPLY" ] || REPLY="${2:-}"
}
confirm() { # confirm "Question" y|n
if [ "$2" = y ]; then hint="Y/n"; else hint="y/N"; fi
while :; do
ask "$1" "$hint"
case "$REPLY" in
"$hint") [ "$2" = y ]; return ;;
y|Y|yes|Yes|YES) return 0 ;;
n|N|no|No|NO) return 1 ;;
*) warn "y or n, please" ;;
esac
done
}
secret() { # secret "Prompt" → $REPLY, typed unseen
printf ' %s ' "$1" >&2
saved=$(stty -g <"$TTY" 2>/dev/null || true)
stty -echo <"$TTY" 2>/dev/null || true
IFS= read -r REPLY <"$TTY" || REPLY=
[ -z "$saved" ] || stty "$saved" <"$TTY" 2>/dev/null || true
printf '\n' >&2
}
choose() { # choose "default-number" "label 1" "label 2" … → $REPLY = the number picked
def=$1; shift; n=0
for label in "$@"; do n=$((n + 1)); if [ "$n" = "$def" ]; then out " ${AMBER}${B}$n${R} $label"; else out " ${B}$n${R} $label"; fi; done
blank
while :; do
ask "Choose 1-$n" "$def"
case "$REPLY" in *[!0-9]*|'') ;; *) if [ "$REPLY" -ge 1 ] && [ "$REPLY" -le "$n" ]; then return; fi ;; esac
warn "a number from 1 to $n, please"
done
}
restore_tty() { [ -z "${saved:-}" ] || stty "$saved" <"$TTY" 2>/dev/null || true; }
# ──────────────────────────────────────────────────────────── plumbing ──
fetch() { # url dest
if command -v curl >/dev/null 2>&1; then
curl -fsSL ${HAMLAB_TOKEN:+-H "Authorization: token $HAMLAB_TOKEN"} -o "$2" "$1"
elif command -v wget >/dev/null 2>&1; then
wget -q ${HAMLAB_TOKEN:+--header="Authorization: token $HAMLAB_TOKEN"} -O "$2" "$1"
else
die "This needs curl or wget, and the machine has neither."
fi
}
port_busy() { # is something already listening on TCP port $1?
hex=$(printf '%04X' "$1")
cat /proc/net/tcp /proc/net/tcp6 2>/dev/null | awk -v p=":$hex" '$4 == "0A" && substr($2, length($2) - 4) == p { found = 1 } END { exit !found }'
}
lan_addresses() { # this machine's IPv4 addresses, one per line, most useful first
if command -v ip >/dev/null 2>&1; then
ip -4 -o addr show scope global 2>/dev/null | awk '$2 !~ /^(docker|br-|veth|virbr|cni|flannel|tailscale|wg|tun|tap)/ { split($4, a, "/"); print a[1] " " $2 }'
else
hostname -I 2>/dev/null | tr ' ' '\n' | awk 'NF && $1 !~ /:/ { print $1 " -" }'
fi
}
# ────────────────────────────────────────────────────────── arguments ──
YES=0; PASS=
while [ $# -gt 0 ]; do
case "$1" in
-y|--yes) YES=1; shift ;;
-h|--help) sed -n '2,20p' "$0" 2>/dev/null | sed 's/^# \{0,1\}//' >&2; exit 0 ;;
--) shift; PASS="$*"; break ;;
*) PASS="$*"; break ;; # flags given straight: they belong to `hamlab install`
esac
done
INTERACTIVE=1
{ [ "$YES" = 1 ] || [ -n "$PASS" ] || ! have_tty; } && INTERACTIVE=0
# ─────────────────────────────────────────────────────────── preflight ──
banner
[ "$(uname -s)" = Linux ] || die "Ham Lab runs on Linux; this is $(uname -s)."
case "$(uname -m)" in
x86_64|amd64) arch=amd64 ;;
aarch64|arm64) arch=arm64 ;;
armv6l|armv7l|armv8l|arm) arch=arm ;;
i386|i486|i586|i686) arch=386 ;;
riscv64) arch=riscv64 ;;
*) die "There is no build for $(uname -m). (There are: amd64, arm64, arm, 386, riscv64.)" ;;
esac
file="hamlab-linux-$arch"
me=$(id -un 2>/dev/null || echo root)
if [ "$(id -u)" = 0 ]; then how=root
elif command -v sudo >/dev/null 2>&1; then how=sudo
elif command -v doas >/dev/null 2>&1; then how=doas
else die "Installing a service needs root, and there is no sudo or doas here. Run this as root."
fi
if [ -d /run/systemd/system ]; then init=systemd
elif command -v rc-service >/dev/null 2>&1; then init=OpenRC
else init=none
fi
os=$(. /etc/os-release 2>/dev/null && printf '%s' "${PRETTY_NAME:-}"); [ -n "$os" ] || os=Linux
existing=$(command -v hamlab 2>/dev/null || true); [ -n "$existing" ] || { [ -x /usr/local/bin/hamlab ] && existing=/usr/local/bin/hamlab; } || true
have=; [ -z "$existing" ] || have=$("$existing" version 2>/dev/null | awk '{ print $2 }') || true
good "$os ${D}${DOT}${R} $(uname -m) ${D}${DOT}${R} $init"
case "$how" in root) good "running as root" ;; *) good "running as ${B}$me${R}; $how is asked for root when it is needed" ;; esac
[ "$init" != none ] || warn "no systemd or OpenRC: it will be installed, but nothing starts it at boot"
# ───────────────────────────────────────────────────── which version ──
tmp=$(mktemp -d 2>/dev/null || mktemp -d -t hamlab)
alt=
cleanup() { restore_tty; rm -rf "$tmp"; [ -z "$alt" ] || rm -f "$alt"; }
trap cleanup EXIT
trap 'blank; out " cancelled — nothing was changed"; exit 130' INT TERM
here=$(cd "$(dirname "$0")" 2>/dev/null && pwd || echo .)
offline=0; [ -f "$here/$file" ] && offline=1
if [ "$offline" = 1 ]; then
ver=$BUILT_FOR; case "$ver" in @*) ver="(local files)" ;; esac
else
# What is published right now — which may be newer than the copy of this script being run.
ver=$(fetch "$HAMLAB_URL/VERSION" "$tmp/VERSION" 2>/dev/null && head -n 1 "$tmp/VERSION" | tr -cd 'A-Za-z0-9._+-' || true)
[ -n "$ver" ] || { ver=$BUILT_FOR; case "$ver" in @*) ver=latest ;; esac; }
fi
# v0.1.0-69-g9c4f9df → 0001000069: versions compare as numbers, builds since the tag included.
vnum() { printf '%s' "$1" | sed 's/^v//; s/-g[0-9a-f]*$//' | awk -F'[.-]' '{ printf "%d%03d%03d%05d", $1, $2, $3, $4 }'; }
older=0
if [ -n "$have" ] && [ "$have" != "$ver" ]; then
a=$(vnum "$have" 2>/dev/null || echo 0); b=$(vnum "$ver" 2>/dev/null || echo 0)
case "$a$b" in *[!0-9]*) ;; *) [ "$b" -ge "$a" ] || older=1 ;; esac
fi
if [ "$older" = 1 ]; then
warn "Ham Lab ${B}$have${R} is installed, and what is published is ${B}older${R}: $ver"
note " installing it would take this machine backwards"
if [ "$INTERACTIVE" = 1 ]; then
blank; confirm "Go back to $ver anyway?" n || { blank; out " nothing was changed"; exit 0; }
elif [ -z "${HAMLAB_ALLOW_DOWNGRADE:-}" ]; then
die "Not downgrading $have to $ver." "Set HAMLAB_ALLOW_DOWNGRADE=1 if that is really what you want."
fi
have_msg_done=1
fi
if [ -n "${have_msg_done:-}" ]; then :
elif [ -n "$have" ] && [ "$have" = "$ver" ]; then good "Ham Lab ${B}$have${R} is already installed ${D}${ARROW}${R} this reinstalls it"; note " its config, user and password are kept"
elif [ -n "$have" ]; then good "Ham Lab ${B}$have${R} is installed ${D}${ARROW}${R} upgrading to ${B}$ver${R}"; note " its config, user and password are kept"
else good "installing Ham Lab ${B}$ver${R}"; fi
# ─────────────────────────────────────────────────────────── the wizard ──
flags=; password=; listen=; url_host=
if [ "$INTERACTIVE" = 1 ] && [ -n "$have" ] && [ "$older" = 1 ]; then :
elif [ "$INTERACTIVE" = 1 ] && [ -n "$have" ]; then
blank
confirm "$([ "$have" = "$ver" ] && echo Reinstall || echo Upgrade) now?" y || { blank; out " nothing was changed"; exit 0; }
elif [ "$INTERACTIVE" = 1 ]; then
# 1 ─ who
default_user=${SUDO_USER:-${DOAS_USER:-$me}}
step "1/5" "Who should it run as?" "Its terminals are that user's shell; its files are that user's files."
while :; do
ask "User" "$default_user"; run_as=$REPLY
id "$run_as" >/dev/null 2>&1 && break
warn "there is no user called \"$run_as\" on this machine"
done
[ "$run_as" != root ] || warn "as root, every terminal it opens is a root shell"
# 2 ─ where
step "2/5" "Where should it listen?" "Who can reach the console at all. The password guards it either way."
addrs=$(lan_addresses || true); first=$(printf '%s\n' "$addrs" | awk 'NR == 1 { print $1 }')
if [ -n "$first" ]; then
first_if=$(printf '%s\n' "$addrs" | awk 'NR == 1 { print $2 }')
choose 2 "This machine only ${D}127.0.0.1 — for an SSH tunnel or a reverse proxy${R}" \
"This network ${D}$first ($first_if) — reachable from the LAN${R}" \
"Every interface ${D}0.0.0.0 — every address it has or gets${R}" \
"Another address"
else
choose 1 "This machine only ${D}127.0.0.1${R}" "Every interface ${D}0.0.0.0${R}" "Another address"
[ "$REPLY" = 1 ] || REPLY=$((REPLY + 1))
fi
case "$REPLY" in
1) host=127.0.0.1 ;;
2) host=$first ;;
3) host=0.0.0.0 ;;
4) others=$(printf '%s\n' "$addrs" | awk 'NR > 1 { printf "%s%s (%s)", sep, $1, $2; sep = ", " }'); [ -z "$others" ] || note "also on this machine: $others"
while :; do ask "Address" ""; host=$REPLY; case "$host" in ''|*[!0-9a-fA-F.:]*) warn "an IPv4 or IPv6 address, please" ;; *) break ;; esac; done ;;
esac
while :; do
ask "Port" "8100"; port=$REPLY
case "$port" in ''|*[!0-9]*) warn "a number, please"; continue ;; esac
[ "$port" -ge 1 ] && [ "$port" -le 65535 ] || { warn "a port is between 1 and 65535"; continue; }
[ "$port" -ge 1024 ] || [ "$run_as" = root ] || { warn "ports under 1024 need root, and it will run as $run_as"; continue; }
if port_busy "$port"; then warn "something is already listening on port $port"; continue; fi
break
done
case "$host" in *:*) listen="[$host]:$port" ;; *) listen="$host:$port" ;; esac
# 3 ─ https
step "3/5" "HTTPS?" "With a certificate made here: browsers warn once, a hub pins it."
if [ "$host" = 127.0.0.1 ]; then
choose 2 "HTTPS, self-signed" "Plain HTTP ${D}— fine here: nothing leaves this machine${R}"
else
choose 1 "HTTPS, self-signed ${D}— recommended: the password crosses the network${R}" "Plain HTTP ${D}— only behind a reverse proxy that adds HTTPS${R}"
fi
if [ "$REPLY" = 1 ]; then tls=auto; scheme=https; else tls=off; scheme=http; fi
# 4 ─ firewall
open_fw=0; fw=
if [ "$host" != 127.0.0.1 ]; then
if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then fw=firewalld
elif command -v ufw >/dev/null 2>&1 && ufw status 2>/dev/null | grep -q "^Status: active"; then fw=ufw
elif [ "$how" != root ] && command -v ufw >/dev/null 2>&1 && systemctl is-active -q ufw 2>/dev/null; then fw=ufw
fi
fi
step "4/5" "Firewall"
if [ -n "$fw" ]; then
note "$fw is active and will block port $port until it is opened."
blank
if confirm "Open $port/tcp in $fw?" y; then open_fw=1; fi
elif [ "$host" = 127.0.0.1 ]; then good "nothing to open: it only listens on this machine"
else good "no active firewalld or ufw found; nothing to open"
fi
# 5 ─ password
step "5/5" "The password" "One password opens the console, and the console is a shell here."
choose 1 "Choose one now" "Generate a strong one ${D}— shown once at the end${R}"
if [ "$REPLY" = 1 ]; then
while :; do
secret "Password:"; password=$REPLY
[ ${#password} -ge 12 ] || { warn "at least 12 characters, please (it guards a shell)"; continue; }
secret "Once more:"
[ "$REPLY" = "$password" ] && break
warn "those didn't match; try again"
done
fi
# summary
if [ "$host" = 0.0.0.0 ] || [ "$host" = "::" ]; then url_host=${first:-this-machine}; else url_host=$host; fi
blank; rule
out " ${B}Ready to install${R}"
blank
out " version ${B}$ver${R} ${D}($file)${R}"
out " runs as ${B}$run_as${R}"
out " address ${B}$scheme://$listen${R}"
if [ "$open_fw" = 1 ]; then out " firewall open $port/tcp in $fw"; fi
if [ -n "$password" ]; then out " password the one you typed"; else out " password generated, shown once"; fi
case "$init" in none) out " service ${AMBER}none — no systemd or OpenRC${R}" ;; *) out " service $init, starts at boot" ;; esac
rule; blank
confirm "Install?" y || { blank; out " nothing was changed"; exit 0; }
flags="--user $run_as --listen $listen --tls $tls"
[ "$open_fw" = 0 ] || flags="$flags --open-firewall"
[ -z "$password" ] || flags="$flags --password-stdin"
fi
# ──────────────────────────────────────────────── download and verify ──
blank
if [ "$offline" = 1 ]; then
cp "$here/$file" "$tmp/hamlab"; [ ! -f "$here/SHA256SUMS" ] || cp "$here/SHA256SUMS" "$tmp/SHA256SUMS"
good "using $file from $here"
else
printf ' %s downloading %s …' "$DOT" "$file" >&2
fetch "$HAMLAB_URL/$file" "$tmp/hamlab" || { printf '\n' >&2; die "Download failed: $HAMLAB_URL/$file"; }
size=$(wc -c <"$tmp/hamlab" | awk '{ printf "%.1f MB", $1 / 1048576 }')
if [ -t 2 ]; then printf '\r\033[K' >&2; else printf '\n' >&2; fi
good "downloaded $file ${D}($size)${R}"
fetch "$HAMLAB_URL/SHA256SUMS" "$tmp/SHA256SUMS" 2>/dev/null || rm -f "$tmp/SHA256SUMS"
fi
if [ -f "$tmp/SHA256SUMS" ] && command -v sha256sum >/dev/null 2>&1; then
want=$(awk -v f="$file" '$2 == f || $2 == "*" f { print $1 }' "$tmp/SHA256SUMS")
got=$(sha256sum "$tmp/hamlab" | awk '{ print $1 }')
[ -n "$want" ] || die "SHA256SUMS has no entry for $file."
[ "$want" = "$got" ] || die "Checksum mismatch for $file." "This is not the file that was published. Nothing was installed."
good "checksum matches SHA256SUMS"
else
warn "no SHA256SUMS (or no sha256sum here): the download was not verified"
fi
chmod 0755 "$tmp/hamlab"; bin="$tmp/hamlab"
if ! "$bin" version >/dev/null 2>&1; then
alt="${HOME:-/root}/.hamlab-install.$$" # a noexec /tmp is the usual reason: try from the home directory
cp "$bin" "$alt" 2>/dev/null && chmod 0755 "$alt" || true
"$alt" version >/dev/null 2>&1 || die "The $arch build doesn't run on this machine."
bin="$alt"
fi
# ───────────────────────────────────────────────────────────── install ──
blank; out " ${B}Installing${R}"; blank
log="$tmp/install.log"
set +e
# shellcheck disable=SC2086 # $flags and $PASS are lists of words on purpose
if [ -n "$password" ]; then
printf '%s\n' "$password" | "$bin" install $flags >"$log" 2>&1; rc=$?
password=
elif [ "$INTERACTIVE" = 0 ] && [ -n "$PASS" ]; then
log=
case " $PASS " in
*password-stdin*) "$bin" install $PASS; rc=$? ;;
*) if [ ! -t 0 ] && have_tty; then "$bin" install $PASS <"$TTY"; rc=$?; else "$bin" install $PASS; rc=$?; fi ;;
esac
else
"$bin" install $flags </dev/null >"$log" 2>&1; rc=$? # no typed password and no terminal on stdin: it generates one and prints it
fi
set -e
[ -z "$log" ] || sed 's/^/ /' "$log" >&2
hamlab_bin=$(command -v hamlab 2>/dev/null || echo /usr/local/bin/hamlab)
[ "$rc" = 0 ] && [ -x "$hamlab_bin" ] || die "The install did not finish. Nothing above this line is hidden: the reason is in it."
# ──────────────────────────────────────────────────────────────── done ──
blank; rule
out " ${GREEN}${B}${OK} Ham Lab $("$hamlab_bin" version 2>/dev/null | awk '{ print $2 }') is installed${R}"
blank
row() { out " $(printf '%-10s' "$1") $2"; [ -z "${3:-}" ] || out " ${D}$3${R}"; }
if [ "$init" = none ]; then
row "start it" "${B}hamlab serve${R}" "as ${run_as:-its user}, under your own supervisor: nothing starts it at boot"
fi
if [ -n "$listen" ]; then
row "open" "${CYAN}${B}$scheme://$(printf '%s' "$listen" | sed "s/^0\.0\.0\.0/$url_host/")${R}"
[ "$tls" != auto ] || out " ${D}the browser warns about the self-signed certificate once${R}"
[ "$host" != 127.0.0.1 ] || out " ${D}from elsewhere: ssh -L $port:127.0.0.1:$port $run_as@this-machine${R}"
fi
case "$init" in
systemd) row "logs" "${B}journalctl -u hamlab -f${R}" ;;
OpenRC) row "logs" "${B}tail -f /var/log/hamlab.log${R}" ;;
esac
row "change" "${B}hamlab install --help${R}" "a new address, HTTPS or user; everything else is kept"
row "add to hub" "Machines ${ARROW} Add a machine" "on your main Ham Lab, with this address and password"
row "remove" "${B}hamlab uninstall${R}" "--purge deletes its config too"
rule; blank
internal/api/alerts.go
+520
View File
@@ -0,0 +1,520 @@
package api
// Alerts: rules over sensors, watched by the hub for itself and every machine it manages.
//
// A rule is a selector, a comparison and a patience: `container.*.running == false for
// 120 s`. Every alertEvery the hub asks each machine one shaped query for all the rules'
// selectors, compares, and keeps a little state per (rule, machine, sensor): ok, pending
// since, or firing since. Going to firing, and coming back, is an event — kept, shown on
// the Alerts tab, and sent to whatever channels are set (ntfy, or a webhook: Home
// Assistant, Slack, anything that takes a POST). `hamlab.reachable` is a sensor the hub
// supplies itself, so "that machine stopped answering" is a rule like any other.
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"sort"
"strconv"
"strings"
"sync"
"time"
"hamlab/internal/sensors"
"hamlab/internal/store"
)
const (
alertEvery = 15 * time.Second
maxEvents = 200
)
type AlertRule struct {
ID string `json:"id"`
Name string `json:"name"`
Sensor string `json:"sensor"` // an ID or a pattern
Op string `json:"op"` // > >= < <= == !=
Value any `json:"value"` // number, bool or string, to match the sensor
ForSec int `json:"for_sec"`
Severity string `json:"severity"` // info | warn | crit
Enabled bool `json:"enabled"`
Machines string `json:"machines"` // "*" or comma-separated names; "" is "*"
}
type AlertChannel struct {
Type string `json:"type"` // ntfy | webhook
URL string `json:"url"`
}
type AlertEvent struct {
T int64 `json:"t"`
Kind string `json:"kind"` // firing | resolved
Rule string `json:"rule"`
RuleID string `json:"rule_id"`
Machine string `json:"machine"`
Sensor string `json:"sensor"`
Value any `json:"value"`
Severity string `json:"severity"`
}
type alertFiring struct {
AlertEvent
Since int64 `json:"since"`
}
type alertSaved struct {
Rules []AlertRule `json:"rules"`
Channels []AlertChannel `json:"channels"`
Events []AlertEvent `json:"events"`
Seeded bool `json:"seeded"`
// Firing is what was firing when the hub last looked, so a restart (every deploy is one)
// carries on where it was and doesn't announce the same alerts all over again.
Firing map[string]alertKept `json:"firing,omitempty"`
}
type alertKept struct {
Since int64 `json:"since"`
Value any `json:"value"`
}
const maxRules = 200
type alertState struct {
pending time.Time
firing bool
since time.Time
value any
}
type alertEngine struct {
s *Server
mu sync.Mutex
saved alertSaved
state map[string]*alertState // ruleID|machine|sensor
last time.Time
}
var defaultRules = []AlertRule{
{ID: "machine-offline", Name: "Machine not answering", Sensor: "hamlab.reachable", Op: "==", Value: false, ForSec: 120, Severity: "crit", Enabled: true},
{ID: "container-down", Name: "Container down", Sensor: "container.*.down", Op: "==", Value: true, ForSec: 120, Severity: "crit", Enabled: true},
{ID: "container-unhealthy", Name: "Container unhealthy", Sensor: "container.*.health", Op: "==", Value: "unhealthy", ForSec: 180, Severity: "warn", Enabled: true},
{ID: "service-failed", Name: "A service has failed", Sensor: "services.failed", Op: ">", Value: 0.0, ForSec: 300, Severity: "warn", Enabled: true},
{ID: "disk-full", Name: "Filesystem nearly full", Sensor: "fs.*.used_pct", Op: ">=", Value: 90.0, ForSec: 300, Severity: "warn", Enabled: true},
{ID: "pool-unhealthy", Name: "ZFS pool not healthy", Sensor: "zfs.*.healthy", Op: "==", Value: false, ForSec: 60, Severity: "crit", Enabled: true},
{ID: "drive-failing", Name: "Drive failing SMART", Sensor: "smart.*.passed", Op: "==", Value: false, ForSec: 0, Severity: "crit", Enabled: true},
{ID: "on-battery", Name: "UPS on battery", Sensor: "ups.on_battery", Op: "==", Value: true, ForSec: 30, Severity: "crit", Enabled: true},
{ID: "hot", Name: "CPU running hot", Sensor: "cpu.temp", Op: ">=", Value: 90.0, ForSec: 300, Severity: "warn", Enabled: true},
{ID: "memory", Name: "Memory nearly exhausted", Sensor: "mem.used_pct", Op: ">=", Value: 95.0, ForSec: 300, Severity: "warn", Enabled: true},
}
func newAlertEngine(s *Server) *alertEngine {
e := &alertEngine{s: s, state: map[string]*alertState{}}
var raw json.RawMessage
s.store.View(func(st *store.State) { raw = st.Alerts })
_ = json.Unmarshal(raw, &e.saved)
if !e.saved.Seeded {
e.saved.Rules, e.saved.Seeded = append([]AlertRule{}, defaultRules...), true
}
for key, k := range e.saved.Firing {
e.state[key] = &alertState{pending: time.UnixMilli(k.Since), firing: true, since: time.UnixMilli(k.Since), value: k.Value}
}
for i, r := range e.saved.Rules { // the first day's default watched "not running", which a container stopped on purpose also is
if r.ID == "container-down" && r.Sensor == "container.*.running" {
e.saved.Rules[i].Name, e.saved.Rules[i].Sensor, e.saved.Rules[i].Value = "Container down", "container.*.down", true
}
}
go e.run()
return e
}
func (e *alertEngine) persist() {
b, _ := json.Marshal(e.saved)
_ = e.s.store.Update(func(st *store.State) { st.Alerts = b })
}
func (e *alertEngine) run() {
time.Sleep(20 * time.Second) // let the status sensors take their first readings
for {
e.evaluate()
time.Sleep(alertEvery)
}
}
func wants(r AlertRule, machine string) bool {
if r.Machines == "" || r.Machines == "*" {
return true
}
for _, m := range strings.Split(r.Machines, ",") {
if strings.TrimSpace(m) == machine {
return true
}
}
return false
}
// holds reports whether `got <op> want`, comparing like with like.
func holds(got any, op string, want any) bool {
switch g := got.(type) {
case float64:
w, ok := want.(float64)
if !ok {
if s, isStr := want.(string); isStr {
f, err := strconv.ParseFloat(s, 64)
if err != nil {
return false
}
w = f
} else {
return false
}
}
switch op {
case ">":
return g > w
case ">=":
return g >= w
case "<":
return g < w
case "<=":
return g <= w
case "==":
return g == w
case "!=":
return g != w
}
case bool:
w, ok := want.(bool)
if !ok {
w = fmt.Sprint(want) == "true"
}
return (op == "==" && g == w) || (op == "!=" && g != w)
case string:
w := fmt.Sprint(want)
return (op == "==" && g == w) || (op == "!=" && g != w)
}
return false
}
func (e *alertEngine) evaluate() {
e.mu.Lock()
rules := append([]AlertRule{}, e.saved.Rules...)
e.mu.Unlock()
query := map[string]string{}
for _, r := range rules {
if r.Enabled && r.Sensor != "hamlab.reachable" {
query[r.ID] = r.Sensor
}
}
qb, _ := json.Marshal(query)
machines := append([]string{e.s.hostName()}, e.s.nodes.Names()...)
results := map[string]map[string]any{} // machine → ruleID → value | {id: value}
reach := map[string]bool{}
var wg sync.WaitGroup
var mu sync.Mutex
for _, m := range machines {
wg.Add(1)
go func(m string) {
defer wg.Done()
var res map[string]any
ok := false
if m == e.s.hostName() {
if out, err := sensors.Shape(e.s.sensors.All(), qb, false); err == nil {
res, ok = out.(map[string]any)
}
} else {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
code, b, err := e.s.nodes.Do(ctx, m, "POST", "/api/sensors/query", qb, http.Header{"Content-Type": {"application/json"}})
cancel()
ok = err == nil && code == 200 && json.Unmarshal(b, &res) == nil
}
mu.Lock()
results[m], reach[m] = res, ok
mu.Unlock()
}(m)
}
wg.Wait()
now := time.Now()
seen := map[string]bool{}
var events []AlertEvent
e.mu.Lock()
for _, r := range rules {
if !r.Enabled {
continue
}
for _, m := range machines {
if !wants(r, m) {
continue
}
readings := map[string]any{}
if r.Sensor == "hamlab.reachable" {
readings["hamlab.reachable"] = reach[m]
} else if !reach[m] {
continue // no reading is not a reading: an unreachable machine is the offline rule's business
} else if v, ok := results[m][r.ID]; ok {
if many, isMap := v.(map[string]any); isMap {
readings = many
} else if v != nil {
readings[r.Sensor] = v
}
}
for id, val := range readings {
key := r.ID + "|" + m + "|" + id
seen[key] = true
st := e.state[key]
if st == nil {
st = &alertState{}
e.state[key] = st
}
st.value = val
if holds(val, r.Op, r.Value) {
if st.pending.IsZero() {
st.pending = now
}
if !st.firing && now.Sub(st.pending) >= time.Duration(r.ForSec)*time.Second {
st.firing, st.since = true, now
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "firing", Rule: r.Name, RuleID: r.ID, Machine: m, Sensor: id, Value: val, Severity: r.Severity})
}
} else {
if st.firing {
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "resolved", Rule: r.Name, RuleID: r.ID, Machine: m, Sensor: id, Value: val, Severity: r.Severity})
}
st.pending, st.firing = time.Time{}, false
}
}
}
}
byID := map[string]AlertRule{}
for _, r := range rules {
byID[r.ID] = r
}
for key, st := range e.state { // no longer watched: the rule was paused, deleted or narrowed, or the sensor is gone
if seen[key] {
continue
}
if st.firing { // it was on the Alerts tab and perhaps on a phone: say that it is over, and why
part := strings.SplitN(key, "|", 3)
r, known := byID[part[0]]
name, why := r.Name, "no longer watched"
if !known {
name, why = part[0], "rule deleted"
} else if !r.Enabled {
why = "rule paused"
}
if name == "" {
name = part[0]
}
events = append(events, AlertEvent{T: now.UnixMilli(), Kind: "resolved", Rule: name, RuleID: part[0], Machine: part[1], Sensor: part[2], Value: why, Severity: r.Severity})
}
delete(e.state, key)
}
e.last = now
kept := map[string]alertKept{}
for key, st := range e.state {
if st.firing {
kept[key] = alertKept{Since: st.since.UnixMilli(), Value: st.value}
}
}
changed := len(kept) != len(e.saved.Firing)
for k := range kept {
if _, ok := e.saved.Firing[k]; !ok {
changed = true
}
}
e.saved.Firing = kept
if changed && len(events) == 0 {
e.persist()
}
if len(events) > 0 {
e.saved.Events = append(e.saved.Events, events...)
if len(e.saved.Events) > maxEvents {
e.saved.Events = e.saved.Events[len(e.saved.Events)-maxEvents:]
}
e.persist()
}
channels := append([]AlertChannel{}, e.saved.Channels...)
e.mu.Unlock()
for _, ev := range events {
e.notify(channels, ev)
}
}
func (e *alertEngine) firing() []alertFiring {
out := []alertFiring{}
names := map[string]AlertRule{}
for _, r := range e.saved.Rules {
names[r.ID] = r
}
for key, st := range e.state {
if !st.firing {
continue
}
p := strings.SplitN(key, "|", 3)
r := names[p[0]]
out = append(out, alertFiring{AlertEvent: AlertEvent{T: st.since.UnixMilli(), Kind: "firing", Rule: r.Name, RuleID: r.ID, Machine: p[1], Sensor: p[2], Value: st.value, Severity: r.Severity}, Since: st.since.UnixMilli()})
}
sort.Slice(out, func(i, j int) bool { return out[i].Since > out[j].Since })
return out
}
func describe(ev AlertEvent) (title, body string) {
mark := map[string]string{"firing": "▲", "resolved": "✓"}[ev.Kind]
title = fmt.Sprintf("%s %s — %s", mark, ev.Rule, ev.Machine)
body = fmt.Sprintf("%s = %v", ev.Sensor, ev.Value)
if ev.Kind == "resolved" {
body = "back to normal: " + body
}
return
}
func (e *alertEngine) notify(channels []AlertChannel, ev AlertEvent) {
title, body := describe(ev)
for _, c := range channels {
var req *http.Request
switch c.Type {
case "ntfy":
req, _ = http.NewRequest("POST", c.URL, strings.NewReader(body))
req.Header.Set("Title", title)
req.Header.Set("Priority", map[string]string{"crit": "urgent", "warn": "high"}[ev.Severity])
req.Header.Set("Tags", map[string]string{"firing": "warning", "resolved": "white_check_mark"}[ev.Kind])
if ev.Kind == "resolved" {
req.Header.Set("Priority", "default")
}
case "webhook":
b, _ := json.Marshal(map[string]any{"title": title, "message": body, "event": ev})
req, _ = http.NewRequest("POST", c.URL, bytes.NewReader(b))
req.Header.Set("Content-Type", "application/json")
}
if req == nil {
continue
}
go func(req *http.Request, url string) {
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
resp, err := http.DefaultClient.Do(req.WithContext(ctx))
if err != nil {
log.Printf("alert: notifying %s: %v", url, err)
return
}
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
if resp.StatusCode >= 300 {
log.Printf("alert: %s answered %d", url, resp.StatusCode)
}
}(req, c.URL)
}
}
// ---- API ----
func (s *Server) alertsGet(w http.ResponseWriter, _ *http.Request) {
e := s.alerts
e.mu.Lock()
defer e.mu.Unlock()
ev := append([]AlertEvent{}, e.saved.Events...)
for i, j := 0, len(ev)-1; i < j; i, j = i+1, j-1 {
ev[i], ev[j] = ev[j], ev[i]
}
rules, channels := e.saved.Rules, e.saved.Channels
if rules == nil { // an empty list is a list: never hand the page a null to take the length of
rules = []AlertRule{}
}
if channels == nil {
channels = []AlertChannel{}
}
jsonOK(w, 200, map[string]any{"rules": rules, "channels": channels, "events": ev, "firing": e.firing(), "checked": e.last.UnixMilli(), "every_sec": int(alertEvery.Seconds())})
}
func (s *Server) alertsPut(w http.ResponseWriter, r *http.Request) {
var body struct {
Rules *[]AlertRule `json:"rules"`
Channels *[]AlertChannel `json:"channels"`
}
if err := json.NewDecoder(io.LimitReader(r.Body, 256<<10)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
if body.Rules != nil && len(*body.Rules) > maxRules {
jsonErr(w, 400, fmt.Sprintf("at most %d rules: every one is checked on every machine every %d s", maxRules, int(alertEvery.Seconds())))
return
}
if body.Rules != nil {
ids := map[string]bool{}
for i := range *body.Rules {
ru := &(*body.Rules)[i]
ru.Name, ru.Sensor = strings.TrimSpace(ru.Name), strings.TrimSpace(ru.Sensor)
if ru.ID == "" || ids[ru.ID] || ru.Name == "" || len(ru.Name) > maxNameLen || ru.ForSec < 0 || ru.ForSec > 86400 {
jsonErr(w, 400, "every rule needs a unique id, a name (64 characters at most) and a patience between 0 and 86400 s")
return
}
ids[ru.ID] = true
if _, err := sensors.Matcher(ru.Sensor); err != nil || ru.Sensor == "" {
jsonErr(w, 400, fmt.Sprintf("rule %q: %q is not a sensor ID or pattern", ru.Name, ru.Sensor))
return
}
if !strings.Contains(" > >= < <= == != ", " "+ru.Op+" ") {
jsonErr(w, 400, fmt.Sprintf("rule %q: the comparison must be one of > >= < <= == !=", ru.Name))
return
}
switch val := ru.Value.(type) {
case float64, bool:
case string:
if len(val) > 200 {
jsonErr(w, 400, fmt.Sprintf("rule %q: the value is too long", ru.Name))
return
}
if ru.Op != "==" && ru.Op != "!=" { // "cpu.usage > ninety" can never be true, and would never say so
jsonErr(w, 400, fmt.Sprintf("rule %q: %s compares numbers, and %q is not one; text can only be == or !=", ru.Name, ru.Op, val))
return
}
default:
jsonErr(w, 400, fmt.Sprintf("rule %q: the value must be a number, true/false or text", ru.Name))
return
}
if _, isBool := ru.Value.(bool); isBool && ru.Op != "==" && ru.Op != "!=" {
jsonErr(w, 400, fmt.Sprintf("rule %q: true/false can only be == or !=", ru.Name))
return
}
if ru.Severity != "info" && ru.Severity != "crit" {
ru.Severity = "warn"
}
}
}
if body.Channels != nil {
for _, c := range *body.Channels {
if (c.Type != "ntfy" && c.Type != "webhook") || !(strings.HasPrefix(c.URL, "https://") || strings.HasPrefix(c.URL, "http://")) {
jsonErr(w, 400, "a channel is {type: ntfy|webhook, url: http(s)://…}")
return
}
}
}
e := s.alerts
e.mu.Lock()
if body.Rules != nil {
e.saved.Rules = *body.Rules
}
if body.Channels != nil {
e.saved.Channels = *body.Channels
}
e.persist()
e.mu.Unlock()
w.WriteHeader(204)
}
// alertsTest sends one clearly-marked test notification through every channel.
func (s *Server) alertsTest(w http.ResponseWriter, _ *http.Request) {
e := s.alerts
e.mu.Lock()
channels := append([]AlertChannel{}, e.saved.Channels...)
e.mu.Unlock()
if len(channels) == 0 {
jsonErr(w, 400, "no channels to test")
return
}
e.notify(channels, AlertEvent{T: time.Now().UnixMilli(), Kind: "firing", Rule: "Test alert", Machine: s.hostName(), Sensor: "hamlab.test", Value: "this is only a test", Severity: "info"})
w.WriteHeader(204)
}
internal/api/docker.go
+631
View File
@@ -0,0 +1,631 @@
package api
// The Docker tab: what is running, grouped the way it was deployed (compose stacks), with
// start / stop / restart, logs and inspect — plus images, volumes and networks to look at.
// It drives the docker CLI (or podman), as the user first and as root if that is refused.
// What can be removed here is what nothing uses: an untagged image, a volume no container
// holds, a network nothing is attached to. Compose stays the one place a stack is defined.
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"os"
"os/exec"
"regexp"
"sort"
"strconv"
"strings"
"time"
"hamlab/internal/priv"
)
type dockerContainer struct {
ID string `json:"id"`
Name string `json:"name"`
Image string `json:"image"`
State string `json:"state"` // running | exited | paused | restarting | created | dead
Status string `json:"status"` // "Up 3 hours (healthy)"
Health string `json:"health"` // healthy | unhealthy | starting | none
Ports string `json:"ports"`
Stack string `json:"stack"` // compose project, "" when started by hand
Service string `json:"service"` // compose service
Created string `json:"created"`
Mem float64 `json:"mem"` // bytes, from the container's cgroup; 0 when unknown
}
func dockerTool() string {
for _, t := range []string{"docker", "podman"} {
if _, err := exec.LookPath(t); err == nil {
return t
}
}
return ""
}
// dockerRun runs the tool as this user, and again as root if the socket said no.
// DockerTool is whichever of docker or podman this machine has, for callers outside the
// package — the scheduled updates the CLI runs.
func DockerTool() string { return dockerTool() }
func dockerRun(ctx context.Context, args ...string) ([]byte, error) {
tool := dockerTool()
if tool == "" {
return nil, errors.New("neither docker nor podman is installed here")
}
out, err := exec.CommandContext(ctx, tool, args...).CombinedOutput()
if err != nil && priv.Available() && strings.Contains(strings.ToLower(string(out)), "permission denied") {
full := priv.Wrap(append([]string{tool}, args...)...)
out, err = exec.CommandContext(ctx, full[0], full[1:]...).CombinedOutput()
}
if err != nil {
msg := strings.TrimSpace(string(out))
if i := strings.LastIndex(msg, "\n"); i >= 0 {
msg = msg[i+1:]
}
if msg == "" {
msg = err.Error()
}
return out, errors.New(msg)
}
return out, nil
}
// dockerWhy picks the line worth showing out of a failed command. Some subcommands —
// `network rm` among them — print the real reason first and then an aggregated "exit
// status 1", which is what dockerRun's last-line rule would otherwise hand back.
func dockerWhy(out []byte, err error) string {
for _, l := range strings.Split(string(out), "\n") {
if l = strings.TrimSpace(l); strings.HasPrefix(l, "Error") || strings.Contains(l, "daemon") {
return l
}
}
return err.Error()
}
func jsonLines(out []byte, each func(map[string]any)) {
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
var m map[string]any
if json.Unmarshal([]byte(line), &m) == nil {
each(m)
}
}
}
func dstr(m map[string]any, k string) string { s, _ := m[k].(string); return s }
func labelOf(labels, key string) string {
for _, kv := range strings.Split(labels, ",") {
if v, ok := strings.CutPrefix(kv, key+"="); ok {
return v
}
}
return ""
}
func (s *Server) dockerOverview(w http.ResponseWriter, r *http.Request) {
if dockerTool() == "" {
jsonOK(w, 200, map[string]any{"available": false, "message": "Docker isn't installed on this machine."})
return
}
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
defer cancel()
out, err := dockerRun(ctx, "ps", "-a", "--no-trunc", "--format", "{{json .}}")
if err != nil {
jsonOK(w, 200, map[string]any{"available": false, "message": err.Error()})
return
}
containers := []dockerContainer{}
jsonLines(out, func(m map[string]any) {
c := dockerContainer{ID: dstr(m, "ID"), Name: strings.Split(dstr(m, "Names"), ",")[0], Image: dstr(m, "Image"), State: dstr(m, "State"), Status: dstr(m, "Status"),
Ports: dstr(m, "Ports"), Created: dstr(m, "CreatedAt"), Health: "none", Stack: labelOf(dstr(m, "Labels"), "com.docker.compose.project"), Service: labelOf(dstr(m, "Labels"), "com.docker.compose.service")}
for _, h := range []string{"unhealthy", "healthy", "starting"} {
if strings.Contains(c.Status, "("+h+")") || strings.Contains(c.Status, "health: "+h) {
c.Health = h
break
}
}
for _, dir := range []string{"/sys/fs/cgroup/system.slice/docker-" + c.ID + ".scope", "/sys/fs/cgroup/docker/" + c.ID, "/sys/fs/cgroup/machine.slice/libpod-" + c.ID + ".scope"} {
if b, err := os.ReadFile(dir + "/memory.current"); err == nil {
c.Mem, _ = strconv.ParseFloat(strings.TrimSpace(string(b)), 64)
break
}
}
containers = append(containers, c)
})
sort.Slice(containers, func(i, j int) bool {
if containers[i].Stack != containers[j].Stack {
return containers[i].Stack < containers[j].Stack
}
return containers[i].Name < containers[j].Name
})
list := func(args ...string) []map[string]any {
rows := []map[string]any{}
if out, err := dockerRun(ctx, args...); err == nil {
jsonLines(out, func(m map[string]any) { rows = append(rows, m) })
}
return rows
}
// Who uses what, from one inspect of every container: an image, a volume or a network
// nothing uses is the only kind this page offers to remove.
imageUsed := map[string]bool{}
volumeUsers, networkUsers := map[string][]string{}, map[string][]string{}
if ids, err := dockerRun(ctx, "ps", "-aq"); err == nil && len(strings.Fields(string(ids))) > 0 {
const f = `{{.Name}}|{{.Image}}|{{range .Mounts}}{{if eq .Type "volume"}}{{.Name}},{{end}}{{end}}|{{range $k, $v := .NetworkSettings.Networks}}{{$k}},{{end}}`
if out, err := dockerRun(ctx, append([]string{"inspect", "-f", f}, strings.Fields(string(ids))...)...); err == nil {
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
p := strings.SplitN(line, "|", 4)
if len(p) < 4 {
continue
}
name := strings.TrimPrefix(p[0], "/")
if id := strings.TrimPrefix(p[1], "sha256:"); len(id) >= 12 {
imageUsed[id[:12]] = true
}
for _, v := range strings.Split(p[2], ",") {
if v != "" {
volumeUsers[v] = append(volumeUsers[v], name)
}
}
for _, n := range strings.Split(p[3], ",") {
if n != "" {
networkUsers[n] = append(networkUsers[n], name)
}
}
}
}
}
images := list("images", "--format", "{{json .}}")
for _, im := range images {
im["in_use"] = imageUsed[dstr(im, "ID")]
}
volumes := list("volume", "ls", "--format", "{{json .}}")
for _, v := range volumes {
users := volumeUsers[dstr(v, "Name")]
if users == nil {
users = []string{}
}
sort.Strings(users)
v["used_by"] = users
v["anonymous"] = strings.Contains(dstr(v, "Labels"), "com.docker.volume.anonymous")
}
networks := list("network", "ls", "--format", "{{json .}}")
for _, n := range networks {
users := networkUsers[dstr(n, "Name")]
if users == nil {
users = []string{}
}
sort.Strings(users)
n["used_by"] = users
n["predefined"] = predefinedNetwork[dstr(n, "Name")]
}
jsonOK(w, 200, map[string]any{"available": true, "tool": dockerTool(), "containers": containers,
"images": images, "volumes": volumes, "networks": networks})
}
var dockerRef = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}$`)
func (s *Server) dockerAction(w http.ResponseWriter, r *http.Request) {
id, action := r.PathValue("id"), r.PathValue("action")
if !dockerRef.MatchString(id) || (action != "start" && action != "stop" && action != "restart" && action != "remove") {
jsonErr(w, 400, "start, stop, restart or remove a container by name or ID")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
defer cancel()
verb := action
if action == "remove" { // never forced: a running container has to be stopped first, on purpose
verb = "rm"
}
if _, err := dockerRun(ctx, verb, id); err != nil {
code := 500
if action == "remove" {
code = 409
}
jsonErr(w, code, err.Error())
return
}
log.Printf("docker: %s %s (from %s)", action, id, clientIP(r))
w.WriteHeader(204)
}
var imageRef = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,254}$`)
// imagePull runs `docker pull` as a task the page follows.
func (s *Server) imagePull(w http.ResponseWriter, r *http.Request) {
var body struct{ Image string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !imageRef.MatchString(strings.TrimSpace(body.Image)) {
jsonErr(w, 400, "an image looks like nginx:alpine or ghcr.io/owner/name:tag")
return
}
ref := strings.TrimSpace(body.Image)
t, err := s.tasks.Start("docker:pull", "pull "+ref, exec.Command(dockerTool(), "pull", ref))
if err != nil {
jsonErr(w, 500, err.Error())
return
}
log.Printf("docker: pull %s as task %s (from %s)", ref, t.ID, clientIP(r))
jsonOK(w, 202, t.Snapshot(false))
}
// imageRemove is `docker rmi`, unforced: an image something still uses is refused by Docker, and that is the answer.
func (s *Server) imageRemove(w http.ResponseWriter, r *http.Request) {
var body struct{ Image string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !imageRef.MatchString(strings.TrimSpace(body.Image)) {
jsonErr(w, 400, "which image?")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
defer cancel()
if _, err := dockerRun(ctx, "rmi", strings.TrimSpace(body.Image)); err != nil {
jsonErr(w, 409, err.Error())
return
}
log.Printf("docker: rmi %s (from %s)", body.Image, clientIP(r))
w.WriteHeader(204)
}
// imagePrune removes dangling images — layers no tag points at any more, left behind by pulls —
// or, asked for {"all": true}, every image no container uses. Those can all be pulled again.
func (s *Server) imagePrune(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
defer cancel()
var body struct{ All bool }
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
args := []string{"image", "prune", "-f"}
if body.All {
args = append(args, "-a")
}
out, err := dockerRun(ctx, args...)
if err != nil {
jsonErr(w, 500, err.Error())
return
}
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
log.Printf("docker: image prune (from %s)", clientIP(r))
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
}
// dockerLogs is the tail of a container's log; ?since=<RFC3339> asks only for what is new,
// which is how the page follows a log without holding a connection open.
func (s *Server) dockerLogs(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
if !dockerRef.MatchString(id) {
jsonErr(w, 400, "bad container")
return
}
args := []string{"logs", "--timestamps"}
if since := r.URL.Query().Get("since"); since != "" {
if _, err := time.Parse(time.RFC3339Nano, since); err != nil {
jsonErr(w, 400, "since is an RFC 3339 time")
return
}
args = append(args, "--since", since)
} else {
n, _ := strconv.Atoi(r.URL.Query().Get("n"))
if n <= 0 || n > 5000 {
n = 300
}
args = append(args, "--tail", strconv.Itoa(n))
}
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
defer cancel()
out, err := dockerRun(ctx, append(args, id)...)
if err != nil {
code := 500
if strings.Contains(strings.ToLower(err.Error()), "no such container") {
code = 404
}
jsonErr(w, code, err.Error())
return
}
type line struct {
T string `json:"t"`
M string `json:"m"`
}
lines := []line{}
for _, l := range strings.Split(strings.TrimRight(string(out), "\n"), "\n") {
if l == "" {
continue
}
t, m, ok := strings.Cut(l, " ")
if _, err := time.Parse(time.RFC3339Nano, t); !ok || err != nil {
t, m = "", l
}
lines = append(lines, line{t, m})
}
jsonOK(w, 200, map[string]any{"lines": lines})
}
// dockerInspect is the parts of `docker inspect` worth reading on a phone.
func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
if !dockerRef.MatchString(id) {
jsonErr(w, 400, "bad container")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
defer cancel()
out, err := dockerRun(ctx, "inspect", id)
var all []map[string]any
if err != nil || json.Unmarshal(out, &all) != nil || len(all) == 0 {
jsonErr(w, 500, fmt.Sprint("inspect failed: ", err))
return
}
c := all[0]
dig := func(m map[string]any, path ...string) any {
var cur any = m
for _, p := range path {
mm, ok := cur.(map[string]any)
if !ok {
return nil
}
cur = mm[p]
}
return cur
}
env := []string{}
if list, ok := dig(c, "Config", "Env").([]any); ok {
for _, e := range list {
k, _, _ := strings.Cut(fmt.Sprint(e), "=")
env = append(env, k) // names only: values are where the secrets live
}
}
sort.Strings(env)
jsonOK(w, 200, map[string]any{
"image": dig(c, "Config", "Image"), "command": dig(c, "Config", "Cmd"), "entrypoint": dig(c, "Config", "Entrypoint"),
"restart": dig(c, "HostConfig", "RestartPolicy", "Name"), "network_mode": dig(c, "HostConfig", "NetworkMode"),
"memory_limit": dig(c, "HostConfig", "Memory"), "started": dig(c, "State", "StartedAt"), "exit_code": dig(c, "State", "ExitCode"),
"error": dig(c, "State", "Error"), "restarts": c["RestartCount"], "mounts": c["Mounts"], "ports": dig(c, "NetworkSettings", "Ports"),
"networks": dig(c, "NetworkSettings", "Networks"), "env_names": env,
"compose_file": dig(c, "Config", "Labels", "com.docker.compose.project.config_files"),
})
}
var volumeRef = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,254}$`)
// volumeRemove deletes one volume and what is in it, which nothing brings back. So it is never
// forced (Docker refuses a volume a container still uses), it is one at a time, and the request
// has to name the volume twice — a slip of a script can't delete one by accident.
func (s *Server) volumeRemove(w http.ResponseWriter, r *http.Request) {
var body struct{ Name, Confirm string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !volumeRef.MatchString(body.Name) {
jsonErr(w, 400, "which volume?")
return
}
if body.Confirm != body.Name {
jsonErr(w, 400, "name the volume again in \"confirm\" to delete it")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
defer cancel()
if _, err := dockerRun(ctx, "volume", "rm", body.Name); err != nil {
jsonErr(w, 409, err.Error())
return
}
log.Printf("docker: volume rm %s (from %s)", body.Name, clientIP(r))
w.WriteHeader(204)
}
// volumePrune removes unused *anonymous* volumes only: the nameless ones an image's VOLUME line
// leaves behind each time its container is recreated. Named volumes are somebody's data, and a
// stack that is merely down leaves its volumes "unused" — those go one at a time, by name.
func (s *Server) volumePrune(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
defer cancel()
out, err := dockerRun(ctx, "volume", "prune", "-f")
if err != nil {
jsonErr(w, 500, err.Error())
return
}
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
log.Printf("docker: volume prune (from %s)", clientIP(r))
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
}
// The three networks Docker makes itself. They hold nothing, cannot be removed, and
// their absence would break every container that asks for one by name.
var predefinedNetwork = map[string]bool{"bridge": true, "host": true, "none": true}
// networkRemove deletes one network. Unlike a volume there is no data in it, and unlike an
// image nothing has to be downloaded again — a stack that wants it back gets it recreated
// on the next deploy. Docker refuses one a container is still attached to, which is the
// check that matters, so this asks for no second confirmation.
func (s *Server) networkRemove(w http.ResponseWriter, r *http.Request) {
var body struct{ Name string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil || !volumeRef.MatchString(body.Name) {
jsonErr(w, 400, "which network?")
return
}
if predefinedNetwork[body.Name] {
jsonErr(w, 400, body.Name+" is one of Docker's own networks and can't be removed")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
defer cancel()
if out, err := dockerRun(ctx, "network", "rm", body.Name); err != nil {
jsonErr(w, 409, dockerWhy(out, err))
return
}
log.Printf("docker: network rm %s (from %s)", body.Name, clientIP(r))
w.WriteHeader(204)
}
// networkPrune removes every network no container is attached to. A stack that is merely
// down counts as unattached, so its network goes too — and comes back on the next deploy,
// with a new subnet. That is the whole cost, and the caller is told it.
func (s *Server) networkPrune(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
defer cancel()
out, err := dockerRun(ctx, "network", "prune", "-f")
if err != nil {
jsonErr(w, 500, err.Error())
return
}
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
log.Printf("docker: network prune (from %s)", clientIP(r))
jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
}
// ---- a container's health, its recent life, its console, and its limits ----
// dockerHealth is the health check's verdict and the last few probe results. Docker keeps
// the last five itself, which is all anyone wants: is it failing, and what did it say.
func (s *Server) dockerHealth(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
if !dockerRef.MatchString(id) {
jsonErr(w, 400, "bad container")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
defer cancel()
out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id)
if err != nil {
jsonErr(w, 500, err.Error())
return
}
var st struct {
Status string `json:"Status"`
Health *struct {
Status string `json:"Status"`
FailingStreak int `json:"FailingStreak"`
Log []struct {
Start time.Time `json:"Start"`
End time.Time `json:"End"`
ExitCode int `json:"ExitCode"`
Output string `json:"Output"`
} `json:"Log"`
} `json:"Health"`
}
if json.Unmarshal(out, &st) != nil {
jsonErr(w, 500, "could not read the container's state")
return
}
if st.Health == nil {
jsonOK(w, 200, map[string]any{"configured": false, "state": st.Status})
return
}
jsonOK(w, 200, map[string]any{
"configured": true, "state": st.Status, "status": st.Health.Status,
"failing_streak": st.Health.FailingStreak, "log": st.Health.Log,
})
}
// dockerExec opens a shell inside a container, as a terminal session like any other — so
// it reattaches, resizes and scrolls back the same way, and works on a managed machine
// through the same proxy.
func (s *Server) dockerExec(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
if !dockerRef.MatchString(id) {
jsonErr(w, 400, "bad container")
return
}
tool := dockerTool()
if tool == "" {
jsonErr(w, 500, "neither docker nor podman is installed here")
return
}
var body struct {
Shell string `json:"shell"`
}
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
shell := strings.TrimSpace(body.Shell)
if shell == "" {
// most images have sh; bash is nicer where it exists, so try it first and fall back
shell = "if command -v bash >/dev/null 2>&1; then exec bash; else exec sh; fi"
} else if strings.ContainsAny(shell, "\n\r") {
jsonErr(w, 400, "bad shell")
return
}
argv := []string{tool, "exec", "-it", id, "sh", "-c", shell}
if priv.Available() && !dockerReachable() {
argv = priv.Wrap(argv...)
}
sess, err := s.mgr.CreateCmd("", "sh "+shortName(id), argv)
if err != nil {
jsonErr(w, 500, err.Error())
return
}
log.Printf("docker exec into %s as terminal %s by %s", id, sess.ID, clientIP(r))
jsonOK(w, 200, map[string]string{"id": sess.ID, "name": "sh " + shortName(id)})
}
func shortName(id string) string {
if len(id) > 20 {
return id[:20]
}
return id
}
// dockerReachable reports whether docker answers without help.
func dockerReachable() bool {
tool := dockerTool()
if tool == "" {
return false
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
return exec.CommandContext(ctx, tool, "version", "--format", "{{.Server.Version}}").Run() == nil
}
// dockerUpdate changes a running container's limits and restart policy. `docker update`
// does this live, without recreating it and without touching the compose file — which is
// also the catch, and the caller is told: a deploy from compose puts back whatever compose
// says.
func (s *Server) dockerUpdate(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
if !dockerRef.MatchString(id) {
jsonErr(w, 400, "bad container")
return
}
var body struct {
Memory string `json:"memory"` // "512m", "2g", or "" to leave alone; "0" removes the limit
CPUs string `json:"cpus"` // "1.5", or "0" for no limit
Restart string `json:"restart"` // no | on-failure | unless-stopped | always
}
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
args := []string{"update"}
if m := strings.TrimSpace(body.Memory); m != "" {
if !regexp.MustCompile(`^[0-9]+([bkmgBKMG])?$`).MatchString(m) {
jsonErr(w, 400, "memory: a number, optionally with b, k, m or g — 0 removes the limit")
return
}
args = append(args, "--memory", m, "--memory-swap", m)
}
if c := strings.TrimSpace(body.CPUs); c != "" {
if !regexp.MustCompile(`^[0-9]+(\.[0-9]+)?$`).MatchString(c) {
jsonErr(w, 400, "cpus: a number like 1.5 — 0 removes the limit")
return
}
args = append(args, "--cpus", c)
}
if p := strings.TrimSpace(body.Restart); p != "" {
switch p {
case "no", "on-failure", "unless-stopped", "always":
default:
jsonErr(w, 400, "restart: no, on-failure, unless-stopped or always")
return
}
args = append(args, "--restart", p)
}
if len(args) == 1 {
jsonErr(w, 400, "nothing to change")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
defer cancel()
if _, err := dockerRun(ctx, append(args, id)...); err != nil {
jsonErr(w, 500, err.Error())
return
}
log.Printf("docker update %s %v by %s", id, args[1:], clientIP(r))
w.WriteHeader(204)
}
internal/api/docker_events.go
+229
View File
@@ -0,0 +1,229 @@
package api
// A container's history.
//
// `docker events` is a live stream with only a short in-memory backlog — ask it for the
// last thirty days and you get the last few minutes, because the daemon keeps events in a
// small ring and loses them all when it restarts. That is why Dockhand kept its own copy,
// and why this does too: a watcher follows the stream and appends the lifecycle events to a
// ring kept beside the config, so "what happened to this container last Tuesday" has an
// answer.
//
// Only lifecycle events are kept. A health check produces three exec events every probe,
// which is thousands a day and tells you nothing that the health panel does not.
import (
"bufio"
"context"
"encoding/json"
"log"
"net/http"
"os"
"os/exec"
"path/filepath"
"sort"
"strings"
"sync"
"time"
"hamlab/internal/priv"
)
const (
eventsKept = 4000
eventsSaveEvery = 30 * time.Second
)
// ContainerEvent is one thing that happened to one container.
type ContainerEvent struct {
T int64 `json:"t"` // unix ms
Name string `json:"name"`
Action string `json:"action"`
Image string `json:"image,omitempty"`
ExitCode string `json:"exit_code,omitempty"`
Signal string `json:"signal,omitempty"`
Health string `json:"health,omitempty"`
}
type eventLog struct {
mu sync.Mutex
ring []ContainerEvent
dirty bool
path string
}
func (e *eventLog) load(path string) {
e.mu.Lock()
defer e.mu.Unlock()
e.path = path
b, err := os.ReadFile(path)
if err != nil {
return
}
var list []ContainerEvent
if json.Unmarshal(b, &list) == nil {
e.ring = list
}
}
func (e *eventLog) add(ev ContainerEvent) {
e.mu.Lock()
defer e.mu.Unlock()
e.ring = append(e.ring, ev)
if len(e.ring) > eventsKept {
e.ring = append([]ContainerEvent(nil), e.ring[len(e.ring)-eventsKept:]...)
}
e.dirty = true
}
// forName is one container's events, newest first.
func (e *eventLog) forName(name string, since time.Time, limit int) []ContainerEvent {
e.mu.Lock()
defer e.mu.Unlock()
out := []ContainerEvent{}
for _, ev := range e.ring {
if ev.Name == name && ev.T >= since.UnixMilli() {
out = append(out, ev)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].T > out[j].T })
if len(out) > limit {
out = out[:limit]
}
return out
}
func (e *eventLog) save() {
e.mu.Lock()
if !e.dirty || e.path == "" {
e.mu.Unlock()
return
}
b, err := json.Marshal(e.ring)
e.dirty = false
path := e.path
e.mu.Unlock()
if err != nil {
return
}
_ = os.MkdirAll(filepath.Dir(path), 0o700)
tmp := path + ".tmp"
if os.WriteFile(tmp, b, 0o600) == nil {
_ = os.Rename(tmp, path)
}
}
// watchDockerEvents follows the stream, restarting if Docker goes away. It is the only
// thing that writes the log, so nothing contends with it.
func (s *Server) watchDockerEvents(ctx context.Context) {
s.events.load(filepath.Join(s.cfg.DataDir(), "docker-events.json"))
go func() { // a periodic flush, so a crash loses at most half a minute of history
t := time.NewTicker(eventsSaveEvery)
defer t.Stop()
for {
select {
case <-ctx.Done():
s.events.save()
return
case <-t.C:
s.events.save()
}
}
}()
for {
if ctx.Err() != nil {
return
}
if dockerTool() == "" {
time.Sleep(time.Minute) // no Docker here; look again later in case it is installed
continue
}
s.followEvents(ctx)
select {
case <-ctx.Done():
return
case <-time.After(10 * time.Second): // the daemon restarted, or the pipe broke
}
}
}
func (s *Server) followEvents(ctx context.Context) {
args := []string{"events", "--format", "{{json .}}", "--filter", "type=container"}
for _, e := range []string{"create", "start", "restart", "stop", "kill", "die", "destroy", "oom", "pause", "unpause", "rename", "update", "health_status"} {
args = append(args, "--filter", "event="+e)
}
tool := dockerTool()
full := append([]string{tool}, args...)
if !dockerReachable() && priv.Available() {
full = priv.Wrap(full...)
}
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
out, err := cmd.StdoutPipe()
if err != nil {
return
}
if err := cmd.Start(); err != nil {
return
}
sc := bufio.NewScanner(out)
sc.Buffer(make([]byte, 64<<10), 1<<20)
n := 0
for sc.Scan() {
var e struct {
Action string `json:"Action"`
Time int64 `json:"time"`
TimeNano int64 `json:"timeNano"`
Actor struct {
Attributes map[string]string `json:"Attributes"`
} `json:"Actor"`
}
if json.Unmarshal(sc.Bytes(), &e) != nil || e.Action == "" {
continue
}
at := e.Actor.Attributes
ev := ContainerEvent{
T: e.Time * 1000, Name: at["name"], Image: at["image"],
Action: e.Action, ExitCode: at["exitCode"], Signal: at["signal"],
}
if e.TimeNano > 0 {
ev.T = e.TimeNano / 1e6
}
// health_status comes through as "health_status: healthy"
if a, h, ok := strings.Cut(e.Action, ": "); ok {
ev.Action, ev.Health = a, h
}
if ev.Name == "" {
continue
}
s.events.add(ev)
n++
}
_ = cmd.Wait()
if n > 0 {
log.Printf("docker events: stream ended after %d events; reconnecting", n)
}
}
// dockerEvents answers from Ham Lab's own history, not Docker's short memory.
func (s *Server) dockerEvents(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
if !dockerRef.MatchString(id) {
jsonErr(w, 400, "bad container")
return
}
since := r.URL.Query().Get("since")
d := 7 * 24 * time.Hour
switch since {
case "24h":
d = 24 * time.Hour
case "30d":
d = 30 * 24 * time.Hour
default:
since = "7d"
}
ev := s.events.forName(id, time.Now().Add(-d), 500)
s.events.mu.Lock()
total := len(s.events.ring)
s.events.mu.Unlock()
jsonOK(w, 200, map[string]any{"events": ev, "since": since, "kept": total, "capacity": eventsKept})
}
internal/api/docker_logs.go
+206
View File
@@ -0,0 +1,206 @@
package api
// Following a container's log over a held connection, the way the Services tab follows a
// journal. The polled `?since=` endpoint next door still exists — it is what a reconnect
// hands back to pick up from — but the live view holds one `docker logs -f` per viewer
// instead of asking again every two seconds.
import (
"bufio"
"context"
"encoding/json"
"io"
"net/http"
"os/exec"
"sort"
"strconv"
"strings"
"syscall"
"time"
"github.com/coder/websocket"
"hamlab/internal/priv"
)
// logLine is one line of a container's output: the time Docker stamped it with, the text,
// and which of the two streams it came out of — stderr is where a container says what went
// wrong, and the page colours it accordingly.
type logLine struct {
T string `json:"t"`
M string `json:"m"`
Err bool `json:"err,omitempty"`
at string // when to sort it by: its own stamp, or the last one its stream carried
}
// splitStamp pulls Docker's RFC 3339 prefix off a line. A line without one (a container
// that writes bare bytes, or a continuation) keeps the whole text and no time — but is
// still sorted where its stream put it, behind the last line that did have one.
func splitStamp(l string, carry *string) logLine {
t, m, ok := strings.Cut(l, " ")
if _, err := time.Parse(time.RFC3339Nano, t); !ok || err != nil {
return logLine{M: l, at: *carry}
}
*carry = t
return logLine{T: t, M: m, at: t}
}
// dockerTail streams a container's output as JSON text frames until the client goes away.
// `?since=<RFC3339>` resumes where a dropped connection left off; otherwise it starts with
// the last `?n=` lines.
func (s *Server) dockerTail(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
if !dockerRef.MatchString(id) {
jsonErr(w, 400, "bad container")
return
}
tool := dockerTool()
if tool == "" {
jsonErr(w, 500, "neither docker nor podman is installed here")
return
}
args := []string{tool, "logs", "--timestamps", "--follow"}
if since := r.URL.Query().Get("since"); since != "" {
if _, err := time.Parse(time.RFC3339Nano, since); err != nil {
jsonErr(w, 400, "since is an RFC 3339 time")
return
}
args = append(args, "--since", since)
} else {
args = append(args, "--tail", strconv.Itoa(logCount(r)))
}
args = append(args, id)
if priv.Available() && !dockerReachable() {
args = priv.Wrap(args...)
}
conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{CompressionMode: websocket.CompressionDisabled})
if err != nil {
return
}
defer conn.Close(websocket.StatusNormalClosure, "")
ctx, cancel := context.WithCancel(r.Context())
defer cancel()
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
// Own process group, so the kill reaches the docker client itself and not just a sudo
// wrapper — the same reason the journal tail does it.
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error {
pg := "-" + strconv.Itoa(cmd.Process.Pid)
if args[0] != tool { // running under sudo: only sudo can signal root's children
k := priv.Wrap("kill", "-9", "--", pg)
return exec.Command(k[0], k[1:]...).Run()
}
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
}
stdout, err := cmd.StdoutPipe()
if err != nil {
return
}
stderr, err := cmd.StderrPipe()
if err != nil {
return
}
if err := cmd.Start(); err != nil {
_ = conn.Write(ctx, websocket.MessageText, mustJSON(logLine{M: err.Error(), Err: true}))
return
}
// Both pipes funnel into one writer: a websocket takes one writer at a time.
lines := make(chan logLine, 256)
read := func(p io.Reader, isErr bool) {
sc := bufio.NewScanner(p)
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
carry := ""
for sc.Scan() {
l := splitStamp(sc.Text(), &carry)
l.Err = isErr
select {
case lines <- l:
case <-ctx.Done():
return
}
}
}
done := make(chan struct{}, 2)
go func() { read(stdout, false); done <- struct{}{} }()
go func() { read(stderr, true); done <- struct{}{} }()
go func() { // the command ends when the container does; closing says so, having said why
<-done
<-done
_ = cmd.Wait()
close(lines)
}()
go func() { // reader: only to notice the close
defer cancel()
for {
if _, _, err := conn.Read(ctx); err != nil {
return
}
}
}()
ping := time.NewTicker(25 * time.Second)
defer ping.Stop()
// stdout and stderr are two pipes, and what Docker replays down them arrives in no
// particular order. So the opening burst is held for a moment and sorted by the time
// Docker stamped each line with; after that the log is live and arrives in order anyway.
const burstFor = 300 * time.Millisecond
warm := time.NewTimer(burstFor)
defer warm.Stop()
warmC := warm.C // nil once the burst has been flushed: a nil channel never fires
var burst []logLine
write := func(l logLine) bool {
return conn.Write(ctx, websocket.MessageText, mustJSON(l)) == nil
}
flush := func() bool {
sort.SliceStable(burst, func(i, j int) bool { return burst[i].at < burst[j].at })
for _, l := range burst {
if !write(l) {
return false
}
}
burst = nil
return true
}
for {
select {
case <-ctx.Done():
return
case l, ok := <-lines:
if !ok { // the container stopped, or never existed: say the last of it and go
flush()
return
}
if warmC != nil {
burst = append(burst, l)
if len(burst) < 10000 { // a container that never stops talking still gets read
continue
}
if !flush() {
return
}
continue
}
if !write(l) {
return
}
case <-warmC:
warmC = nil
if !flush() {
return
}
case <-ping.C:
pctx, pc := context.WithTimeout(ctx, 10*time.Second)
err := conn.Ping(pctx)
pc()
if err != nil {
return
}
}
}
}
func mustJSON(v any) []byte { b, _ := json.Marshal(v); return b }
internal/api/dockersched.go
+351
View File
@@ -0,0 +1,351 @@
package api
// Scheduled image updates.
//
// Dockhand kept three containers up to date nightly — unbound, nginx and pihole, which is
// DNS and ingress — and that is the one thing it does that Ham Lab did not. This replaces
// it, and deliberately in the same shape as the ZFS snapshot schedules: a real line in a
// cron file Ham Lab owns, calling the binary, rather than a ticker inside the daemon.
// Updating an image is not something that should stop happening because a web console is
// down, and a cron line is visible to anyone looking at the machine the normal way.
//
// A schedule stores everything the run needs — project, compose files, how compose has to
// be invoked for that stack — so the cron job is self-contained and does not have to
// rediscover the world before it can act.
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"io"
"log"
"net/http"
"os"
"path/filepath"
"strings"
"time"
"hamlab/internal/compose"
"hamlab/internal/cron"
"hamlab/internal/storage"
"hamlab/internal/store"
)
// UpdateJob is a standing instruction to pull a stack's images, and optionally deploy them.
type UpdateJob struct {
ID string `json:"id"`
Name string `json:"name"`
Stack string `json:"stack"`
Machine string `json:"machine,omitempty"` // for display; the job runs where it was made
// Services narrows it to some of the stack's services; empty is all of them.
Services []string `json:"services,omitempty"`
Cron string `json:"cron"`
// Apply says whether a newer image is deployed or only fetched. Check-only still pulls,
// so the Docker tab shows what is behind without a registry round trip.
Apply bool `json:"apply"`
Enabled bool `json:"enabled"`
// How to run compose for this stack, captured when the schedule was made.
Spec composeSpec `json:"spec"`
}
type composeSpec struct {
Project string `json:"project"`
Files []string `json:"files"`
Via string `json:"via,omitempty"`
Remote string `json:"remote,omitempty"`
}
func (c composeSpec) toSpec(tool string) compose.Spec {
return compose.Spec{Project: c.Project, Files: c.Files, Via: c.Via, Remote: c.Remote, Tool: tool}
}
const dockerCronFile = "/etc/cron.d/hamlab-docker"
type dockerSchedState struct {
Jobs []UpdateJob `json:"jobs"`
}
func (s *Server) dsLoad() dockerSchedState {
var raw json.RawMessage
s.store.View(func(st *store.State) { raw = st.DockerJobs })
var d dockerSchedState
if len(raw) > 0 {
_ = json.Unmarshal(raw, &d)
}
if d.Jobs == nil {
d.Jobs = []UpdateJob{}
}
return d
}
func (s *Server) dsSave(d dockerSchedState) error {
b, err := json.Marshal(d)
if err != nil {
return err
}
return s.store.Update(func(st *store.State) { st.DockerJobs = b })
}
// dsCronText renders the schedules as cron lines. Same shape as the ZFS file: Ham Lab owns
// it whole and rewrites it on every change.
func dsCronText(jobs []UpdateJob, user, exe, config string) (string, error) {
var b strings.Builder
b.WriteString("# Ham Lab image-update schedules — written by the Docker tab, rewritten whole on every change.\n")
b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
b.WriteString("SHELL=/bin/sh\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
for i := range jobs {
j := &jobs[i]
if !j.Enabled {
b.WriteString("# " + j.Name + " — paused\n")
continue
}
expr := strings.Join(strings.Fields(j.Cron), " ")
if err := cron.ValidSchedule(expr); err != nil {
return "", err
}
what := "pull only"
if j.Apply {
what = "pull and deploy"
}
b.WriteString("# " + j.Name + " — " + j.Stack + ", " + what + "\n")
b.WriteString(expr + " " + user + " " + exe + " -config " + config + " docker-update " + j.ID + "\n\n")
}
return b.String(), nil
}
func (s *Server) dsSyncCron(ctx context.Context, d dockerSchedState) error {
live := false
for i := range d.Jobs {
if d.Jobs[i].Enabled {
live = true
}
}
if !live {
return s.storage.RemoveRootFile(ctx, dockerCronFile)
}
exe, err := os.Executable()
if err != nil {
return err
}
exe, _ = filepath.EvalSymlinks(exe)
text, err := dsCronText(d.Jobs, s.user, exe, s.cfg.Path())
if err != nil {
return err
}
return s.storage.WriteRootFile(ctx, dockerCronFile, text)
}
// repoint keeps a schedule working when what deployed its stack has been removed.
//
// A schedule captures how to invoke compose at the moment it is made, so the cron job never
// has to rediscover the world at 3am. But "run compose inside this container" stops being
// true the day that container is deleted — which is exactly what happens when an
// orchestrator is retired, and would otherwise break every schedule made before it went.
// So: if the container named is not running, the files are looked up again, through live
// mounts and remembered ones both, and compose runs on the host instead.
func repoint(ctx context.Context, db *store.Store, spec compose.Spec) compose.Spec {
if spec.Via == "" || db == nil {
return spec
}
if out, err := dockerRun(ctx, "inspect", "-f", "{{.State.Running}}", spec.Via); err == nil && strings.TrimSpace(string(out)) == "true" {
return spec
}
st := stack{Name: spec.Project, File: spec.Files[0], Files: spec.Files}
st.resolve(mountsSeen(ctx, db)) // read-only: this also runs from cron, where the daemon owns state.json
if !st.Editable || st.Via != "" {
return spec // still only reachable through a container, or not reachable at all
}
log.Printf("docker schedule %s: %s is gone, running compose here against %s", spec.Project, spec.Via, st.HostFiles[0])
spec.Files, spec.Via = st.HostFiles, ""
return spec
}
// RunUpdate pulls, and deploys when the schedule says to. Shared with the CLI so the button
// and the cron line cannot drift apart.
func RunUpdate(ctx context.Context, db *store.Store, j UpdateJob, tool string) (string, error) {
if len(j.Spec.Files) == 0 {
return "", errors.New("this schedule has no compose file recorded — open it in the Docker tab and save it again")
}
spec := repoint(ctx, db, j.Spec.toSpec(tool))
var log strings.Builder
run := func(args ...string) error {
cmd := compose.Cmd(ctx, spec, args...)
out, err := cmd.CombinedOutput()
log.Write(out)
return err
}
args := append([]string{"pull"}, j.Services...)
if err := run(args...); err != nil {
return log.String(), err
}
if !j.Apply {
return log.String(), nil
}
up := append([]string{"up", "-d"}, j.Services...)
if err := run(up...); err != nil {
return log.String(), err
}
return log.String(), nil
}
// ---- endpoints ----
func (s *Server) dsList(w http.ResponseWriter, _ *http.Request) {
d := s.dsLoad()
now := time.Now()
out := make([]map[string]any, 0, len(d.Jobs))
for _, j := range d.Jobs {
m := map[string]any{"job": j, "last": storage.ReadRun(s.cfg.DataDir(), "docker-"+j.ID)}
if j.Enabled {
if n, ok := cron.NextRun(strings.Join(strings.Fields(j.Cron), " "), now); ok {
m["next"] = n
}
}
out = append(out, m)
}
jsonOK(w, 200, map[string]any{"jobs": out, "file": dockerCronFile})
}
func (s *Server) dsPut(w http.ResponseWriter, r *http.Request) {
var j UpdateJob
if err := json.NewDecoder(io.LimitReader(r.Body, 32<<10)).Decode(&j); err != nil {
jsonErr(w, 400, "bad request")
return
}
j.Cron = strings.Join(strings.Fields(j.Cron), " ")
if err := cron.ValidSchedule(j.Cron); err != nil {
jsonErr(w, 400, "schedule: "+err.Error())
return
}
if strings.TrimSpace(j.Stack) == "" {
jsonErr(w, 400, "which stack?")
return
}
// resolve the stack now, so the cron job never has to
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
defer cancel()
stacks, err := s.findStacks(ctx)
if err != nil {
jsonErr(w, 500, err.Error())
return
}
found := false
for _, st := range stacks {
if st.Name == j.Stack {
// seenFiles is the list as compose itself sees it: host paths normally, the
// orchestrator container's own paths when compose runs inside one. Getting this
// backwards gives a cron line that works from the console and fails at 3am.
j.Spec = composeSpec{Project: st.Name, Files: seenFiles(st, -1, ""), Via: st.Via, Remote: st.Remote}
if len(j.Spec.Files) == 0 {
jsonErr(w, 409, "that stack's compose file isn't anywhere this console can reach")
return
}
found = true
}
}
if !found {
jsonErr(w, 404, "no stack called "+j.Stack+" on this machine")
return
}
if strings.TrimSpace(j.Name) == "" {
j.Name = j.Stack
}
d := s.dsLoad()
replaced := false
for i := range d.Jobs {
if d.Jobs[i].ID == j.ID && j.ID != "" {
d.Jobs[i] = j
replaced = true
}
}
if !replaced {
b := make([]byte, 6)
if _, err := rand.Read(b); err != nil {
jsonErr(w, 500, err.Error())
return
}
j.ID = hex.EncodeToString(b)
d.Jobs = append(d.Jobs, j)
}
if err := s.dsSave(d); err != nil {
jsonErr(w, 500, err.Error())
return
}
if err := s.dsSyncCron(r.Context(), d); err != nil {
jsonErr(w, 500, "saved, but writing "+dockerCronFile+" failed: "+err.Error())
return
}
log.Printf("docker schedule %q saved by %s (%s, %q, %s)", j.Name, clientIP(r), j.Stack, j.Cron, map[bool]string{true: "pull and deploy", false: "pull only"}[j.Apply])
jsonOK(w, 200, j)
}
func (s *Server) dsDelete(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
d := s.dsLoad()
kept := d.Jobs[:0]
found := false
for _, j := range d.Jobs {
if j.ID == id {
found = true
continue
}
kept = append(kept, j)
}
if !found {
jsonErr(w, 404, "no such schedule")
return
}
d.Jobs = kept
if err := s.dsSave(d); err != nil {
jsonErr(w, 500, err.Error())
return
}
if err := s.dsSyncCron(r.Context(), d); err != nil {
jsonErr(w, 500, "removed, but writing "+dockerCronFile+" failed: "+err.Error())
return
}
storage.ForgetRun(s.cfg.DataDir(), "docker-"+id)
log.Printf("docker schedule %s removed by %s", id, clientIP(r))
w.WriteHeader(204)
}
func (s *Server) dsRun(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
var job UpdateJob
found := false
for _, j := range s.dsLoad().Jobs {
if j.ID == id {
job, found = j, true
}
}
if !found {
jsonErr(w, 404, "no such schedule")
return
}
// Outliving the request: a pull the browser gave up on still finishes, but not forever.
ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 30*time.Minute)
defer cancel()
out, err := RunUpdate(ctx, s.store, job, dockerTool())
rec := storage.RunRecord{At: time.Now(), Taken: job.Stack}
if err != nil {
rec.Error = err.Error()
}
_ = storage.WriteRun(s.cfg.DataDir(), "docker-"+id, rec)
stackUpdatesForget("/" + job.Stack)
if err != nil {
jsonErr(w, 500, err.Error()+"\n"+tail(out, 2000))
return
}
log.Printf("docker schedule %q run by %s", job.Name, clientIP(r))
jsonOK(w, 200, map[string]string{"output": tail(out, 4000)})
}
func tail(s string, n int) string {
if len(s) <= n {
return s
}
return "…" + s[len(s)-n:]
}
internal/api/guests.go
+193
View File
@@ -0,0 +1,193 @@
package api
// Guest logins: a temporary account for handing to somebody you want to show around. It is
// an ordinary account with a clock on it — every tab, every action, on this machine and on
// every machine this hub manages — that stops working at its expiry (half an hour by
// default) or the moment it is revoked.
//
// Deliberately not a reduced account: Mitchell, 2026-09-27, replacing a read-only version
// built the same day. He gives these to people he trusts and revokes them otherwise, and a
// console with half its buttons missing is not the console he wanted to show off. What that
// means in practice: a guest has the Terminal tab, which is a shell as the user the daemon
// runs as — root on every host here — and can make an account or a token of its own that
// outlives the guest login.
import (
"encoding/json"
"io"
"log"
"net/http"
"strings"
"time"
"hamlab/internal/auth"
"hamlab/internal/config"
"hamlab/internal/store"
)
const (
guestPrefix = "guest-"
guestTTLDefault = 30 * time.Minute
guestTTLMax = 24 * time.Hour
guestReapEvery = time.Minute
)
// ---- the accounts themselves ----
func (s *Server) listGuests(w http.ResponseWriter, r *http.Request) {
now := time.Now()
counts := s.sessions.Count()
out := []map[string]any{}
s.cfg.View(func(c *config.Config) {
for _, u := range c.Users {
if !u.Guest || u.Expired(now) {
continue
}
out = append(out, map[string]any{
"name": u.Name, "created": u.Created, "expires": u.Expires,
"issued_by": u.IssuedBy, "sessions": counts[u.Name], "used": !u.LastSeen.IsZero(),
})
}
})
jsonOK(w, 200, out)
}
// addGuest makes a guest login and hands back its password — the only time it is ever
// readable, here as everywhere else a password is generated.
func (s *Server) addGuest(w http.ResponseWriter, r *http.Request) {
me := actorOf(r)
var body struct {
Minutes int `json:"minutes"`
}
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
ttl := guestTTLDefault
if body.Minutes > 0 {
ttl = time.Duration(body.Minutes) * time.Minute
}
if ttl > guestTTLMax {
jsonErr(w, 400, "a guest login lasts at most 24 hours")
return
}
pw, err := auth.GuestPassword()
if err != nil {
jsonErr(w, 500, err.Error())
return
}
hash, err := auth.HashPassword(pw)
if err != nil {
jsonErr(w, 500, err.Error())
return
}
var name string
for try := 0; try < 8 && name == ""; try++ {
suffix, err := auth.GuestSuffix()
if err != nil {
jsonErr(w, 500, err.Error())
return
}
candidate := guestPrefix + suffix
s.cfg.View(func(c *config.Config) {
if _, taken := c.User(candidate); !taken {
name = candidate
}
})
}
if name == "" {
jsonErr(w, 500, "could not find a free guest name")
return
}
now := time.Now()
expires := now.Add(ttl)
if err := s.cfg.Update(func(c *config.Config) {
c.Users = append(c.Users, config.User{Name: name, Hash: hash, Created: now, Guest: true, Expires: expires, IssuedBy: me.User})
}); err != nil {
jsonErr(w, 500, err.Error())
return
}
log.Printf("guest login %q issued until %s (by %s)", name, expires.Format(time.RFC3339), clientIP(r))
jsonOK(w, 200, map[string]any{"name": name, "password": pw, "expires": expires, "minutes": int(ttl.Minutes())})
}
// revokeGuest ends a guest login now. No password: it only ever takes access away, and the
// whole point of the thing is that ending it is one tap.
func (s *Server) revokeGuest(w http.ResponseWriter, r *http.Request) {
name := r.PathValue("name")
found := false
if err := s.cfg.Update(func(c *config.Config) {
kept := c.Users[:0]
for _, u := range c.Users {
if u.Name == name && u.Guest {
found = true
continue
}
kept = append(kept, u)
}
c.Users = kept
}); err != nil {
jsonErr(w, 500, err.Error())
return
}
if !found {
jsonErr(w, 404, "no such guest login")
return
}
s.forgetGuest(name)
log.Printf("guest login %q revoked (by %s)", name, clientIP(r))
w.WriteHeader(204)
}
// forgetGuest signs a guest out everywhere and drops the look they left behind.
func (s *Server) forgetGuest(name string) {
s.sessions.RevokeUser(name)
_ = s.store.Update(func(st *store.State) { delete(st.Users, name) })
}
// watchGuests clears out guest logins whose time is up. Expiry is enforced on every request
// as well — this is what keeps the config file from collecting them, and what signs a guest
// out of a console they left open rather than waiting for their next click.
func (s *Server) watchGuests() {
for {
s.reapGuests()
time.Sleep(guestReapEvery)
}
}
func (s *Server) reapGuests() {
now := time.Now()
var gone []string
var have bool
s.cfg.View(func(c *config.Config) {
for _, u := range c.Users {
if u.Guest {
have = true
if u.Expired(now) {
gone = append(gone, u.Name)
}
}
}
})
if !have || len(gone) == 0 {
return
}
expired := map[string]bool{}
for _, n := range gone {
expired[n] = true
}
if err := s.cfg.Update(func(c *config.Config) {
kept := c.Users[:0]
for _, u := range c.Users {
if expired[u.Name] {
continue
}
kept = append(kept, u)
}
c.Users = kept
}); err != nil {
log.Printf("guests: %v", err)
return
}
for _, n := range gone {
s.forgetGuest(n)
}
log.Printf("guest login(s) expired: %s", strings.Join(gone, ", "))
}
internal/api/guests_test.go
+70
View File
@@ -0,0 +1,70 @@
package api
import (
"net/http"
"net/http/httptest"
"path/filepath"
"testing"
"time"
"hamlab/internal/auth"
"hamlab/internal/config"
"hamlab/internal/store"
)
// A guest login is an ordinary account with a clock on it, so the clock is the thing worth
// testing: one whose time is up is stopped at the door, whether or not the minute-by-minute
// sweep has reached their account yet, and their session is dropped on the way — so the
// console they left open goes back to the login form rather than carrying on.
func TestExpiredGuestIsRefusedAndSignedOut(t *testing.T) {
for _, tc := range []struct {
name string
expires time.Time
wantOK bool
}{
{"still has time", time.Now().Add(10 * time.Minute), true},
{"time is up", time.Now().Add(-time.Second), false},
{"no expiry at all", time.Time{}, false},
} {
t.Run(tc.name, func(t *testing.T) {
dir := t.TempDir()
cfg := config.Default(filepath.Join(dir, "config.json"))
if err := cfg.Update(func(c *config.Config) {
c.Users = []config.User{{Name: "guest-abcd", Hash: "x", Guest: true, Expires: tc.expires}}
}); err != nil {
t.Fatal(err)
}
st, err := store.Open(filepath.Join(dir, "state.json"))
if err != nil {
t.Fatal(err)
}
s := &Server{cfg: cfg, store: st, sessions: auth.NewSessions(st)}
tok, err := s.sessions.Create("guest-abcd", "10.0.0.5", "curl")
if err != nil {
t.Fatal(err)
}
r := httptest.NewRequest("GET", "/api/me", nil)
r.AddCookie(&http.Cookie{Name: cookieName, Value: tok})
w := httptest.NewRecorder()
a, ok, answered := s.cookieActor(w, r)
if ok != tc.wantOK {
t.Fatalf("ok = %v, want %v", ok, tc.wantOK)
}
if tc.wantOK {
if !a.Guest || a.User != "guest-abcd" {
t.Errorf("actor = %+v, want the guest account marked as one", a)
}
if answered {
t.Error("answered the caller for a guest that still has time")
}
return
}
if !answered || w.Code != 401 {
t.Errorf("answered = %v, code = %d, want true and 401", answered, w.Code)
}
if _, live := s.sessions.Lookup(tok, "10.0.0.5"); live {
t.Error("the session of an expired guest is still live")
}
})
}
}
internal/api/hardware.go
+13
View File
@@ -0,0 +1,13 @@
package api
// The Hardware tab: what the machine is made of. Read once and remembered — none of it
// changes while the kernel runs — with ?refresh=1 for the one thing that can, a drive
// plugged in or pulled out.
import (
"net/http"
)
func (s *Server) hardwareReport(w http.ResponseWriter, r *http.Request) {
jsonOK(w, 200, s.hardware.Report(r.Context(), r.URL.Query().Get("refresh") == "1"))
}
internal/api/kvm.go
+310
View File
@@ -0,0 +1,310 @@
package api
// IP-KVMs: a device's own web page, shown inside Ham Lab.
//
// The whole difficulty is that Ham Lab is served over TLS and a KVM usually is not, so a
// browser refuses to put the device's page in an iframe — mixed content, blocked before
// anything loads. The fix is to serve the device from Ham Lab's own origin, which also
// means it inherits Ham Lab's login instead of needing a second one, and makes the iframe
// same-origin.
//
// Proxying an arbitrary web app under a path prefix is the hard part, and nothing here
// knows anything about any particular device. A device UI asks for its assets by absolute
// path — "/assets/app.js", "/api/whatever" — which a prefix cannot rewrite away. So the
// prefix is only the entry point: any request Ham Lab does not serve itself is routed to
// whichever KVM the asking page belongs to, worked out from the Referer, and from a cookie
// when there is no Referer (a WebSocket upgrade, say). Ham Lab's own routes always win, so
// a device can never shadow the console.
import (
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"io"
"log"
"net/http"
"net/http/httputil"
"net/url"
"strings"
"sync"
"time"
"hamlab/internal/config"
)
const kvmPrefix = "/kvm/"
const kvmCookie = "hamlab_kvm"
type kvmProxies struct {
mu sync.Mutex
m map[string]*httputil.ReverseProxy
}
func (s *Server) kvm(id string) (config.KVM, bool) {
var found config.KVM
ok := false
s.cfg.View(func(c *config.Config) {
for _, k := range c.KVMs {
if k.ID == id {
found, ok = k, true
}
}
})
return found, ok
}
// kvmProxy is the reverse proxy for one device, made once and kept.
func (s *Server) kvmProxy(k config.KVM) (*httputil.ReverseProxy, error) {
s.kvms.mu.Lock()
defer s.kvms.mu.Unlock()
if s.kvms.m == nil {
s.kvms.m = map[string]*httputil.ReverseProxy{}
}
if p, ok := s.kvms.m[k.ID]; ok {
return p, nil
}
target, err := url.Parse(k.URL)
if err != nil || target.Host == "" {
return nil, errors.New("this KVM's address is not a URL")
}
prefix := kvmPrefix + k.ID
p := &httputil.ReverseProxy{
Director: func(r *http.Request) {
r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
// Entry through the prefix keeps the path after it; anything else arrived by its
// own absolute path and is passed through untouched.
if rest, ok := strings.CutPrefix(r.URL.Path, prefix); ok {
if rest == "" {
rest = "/"
}
r.URL.Path = rest
}
r.URL.RawPath = ""
r.Host = target.Host
// The device gets its own host and none of Ham Lab's session.
r.Header.Del("Origin")
stripCookie(r, cookieName, kvmCookie)
},
FlushInterval: -1, // a video stream must not be buffered
ModifyResponse: func(resp *http.Response) error {
// The device's own framing and transport policy would stop it being embedded, or
// send a browser off to plain HTTP. Ham Lab is the origin now, and says what it
// serves; its own headers are added on the way out.
for _, h := range []string{"X-Frame-Options", "Content-Security-Policy", "Content-Security-Policy-Report-Only", "Strict-Transport-Security", "Referrer-Policy"} {
resp.Header.Del(h)
}
// A redirect to the device's own root has to come back through the prefix.
if loc := resp.Header.Get("Location"); loc != "" {
if u, err := url.Parse(loc); err == nil && (u.Host == "" || u.Host == target.Host) && strings.HasPrefix(u.Path, "/") && !strings.HasPrefix(u.Path, prefix) {
u.Scheme, u.Host = "", ""
u.Path = prefix + u.Path
resp.Header.Set("Location", u.String())
}
}
return nil
},
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadGateway)
_ = json.NewEncoder(w).Encode(map[string]string{"error": k.Name + " unreachable: " + err.Error()})
},
}
s.kvms.m[k.ID] = p
return p, nil
}
// stripCookie removes named cookies from a request, so Ham Lab's session is never handed to
// a device and the device never sees which KVM the browser is looking at.
func stripCookie(r *http.Request, names ...string) {
cs := r.Cookies()
kept := cs[:0]
for _, c := range cs {
drop := false
for _, n := range names {
if c.Name == n {
drop = true
}
}
if !drop {
kept = append(kept, c)
}
}
r.Header.Del("Cookie")
for _, c := range kept {
r.AddCookie(c)
}
}
func (s *Server) forgetKVMProxy(id string) {
s.kvms.mu.Lock()
delete(s.kvms.m, id)
s.kvms.mu.Unlock()
}
// serveKVM is the entry point: /kvm/<id>/… . It also drops a cookie naming the device, so
// that requests the page makes later by absolute path can be traced back to it.
func (s *Server) serveKVM(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
k, ok := s.kvm(id)
if !ok {
jsonErr(w, 404, "no such KVM")
return
}
p, err := s.kvmProxy(k)
if err != nil {
jsonErr(w, 400, err.Error())
return
}
http.SetCookie(w, &http.Cookie{Name: kvmCookie, Value: id, Path: "/", HttpOnly: true, Secure: secure(r), SameSite: http.SameSiteLaxMode})
p.ServeHTTP(w, r)
}
// kvmFor works out which device a request belongs to when it arrived by absolute path
// rather than through the prefix: the Referer names the page that asked, and a cookie
// covers what has no Referer at all. Nothing device-specific: it never has to know which
// paths a given KVM uses.
func (s *Server) kvmFor(r *http.Request) (config.KVM, bool) {
if ref := r.Header.Get("Referer"); ref != "" {
if u, err := url.Parse(ref); err == nil {
if rest, ok := strings.CutPrefix(u.Path, kvmPrefix); ok {
if id, _, _ := strings.Cut(rest, "/"); id != "" {
if k, ok := s.kvm(id); ok {
return k, true
}
}
}
}
// asked for by a page that is not a KVM: not ours to answer
return config.KVM{}, false
}
if c, err := r.Cookie(kvmCookie); err == nil && c.Value != "" {
return s.kvm(c.Value)
}
return config.KVM{}, false
}
// kvmFallback answers a request Ham Lab does not serve itself, when it came from inside a
// KVM's page. It returns false when the request has nothing to do with a KVM, and the
// caller carries on with its own 404.
func (s *Server) kvmFallback(w http.ResponseWriter, r *http.Request) bool {
k, ok := s.kvmFor(r)
if !ok {
return false
}
p, err := s.kvmProxy(k)
if err != nil {
return false
}
p.ServeHTTP(w, r)
return true
}
func machineLabel(m string) string {
if m == "" {
return "this host"
}
return m
}
// ---- the list ----
func (s *Server) listKVMs(w http.ResponseWriter, _ *http.Request) {
out := []config.KVM{}
s.cfg.View(func(c *config.Config) { out = append(out, c.KVMs...) })
jsonOK(w, 200, out)
}
// kvmForMachine is the KVM attached to a machine, "" being the hub itself.
func (s *Server) kvmForMachine(machine string) (config.KVM, bool) {
var found config.KVM
ok := false
s.cfg.View(func(c *config.Config) {
for _, k := range c.KVMs {
if k.Machine == machine {
found, ok = k, true
}
}
})
return found, ok
}
func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
var body struct{ Name, URL, Machine string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
raw := strings.TrimSpace(body.URL)
if raw != "" && !strings.Contains(raw, "://") {
raw = "http://" + raw // what you would type: an address, not a scheme
}
u, err := url.Parse(raw)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
jsonErr(w, 400, "address: http(s)://host, or just the host")
return
}
name := strings.TrimSpace(body.Name)
if name == "" {
name = u.Host
}
if len(name) > maxNameLen {
jsonErr(w, 400, "that name is too long")
return
}
b := make([]byte, 6)
if _, err := rand.Read(b); err != nil {
jsonErr(w, 500, err.Error())
return
}
machine := strings.TrimSpace(body.Machine)
k := config.KVM{ID: hex.EncodeToString(b), Name: name, URL: u.Scheme + "://" + u.Host + strings.TrimSuffix(u.Path, "/"), Machine: machine, Added: time.Now()}
// One per machine: a second one for the same host replaces the first rather than sitting
// beside it, since the tab shows the machine you are looking at.
var replaced string
if err := s.cfg.Update(func(c *config.Config) {
kept := c.KVMs[:0]
for _, old := range c.KVMs {
if old.Machine == machine {
replaced = old.ID
continue
}
kept = append(kept, old)
}
c.KVMs = append(kept, k)
}); err != nil {
jsonErr(w, 500, err.Error())
return
}
if replaced != "" {
s.forgetKVMProxy(replaced)
}
log.Printf("kvm %q added for %s (%s) by %s", k.Name, machineLabel(machine), k.URL, clientIP(r))
jsonOK(w, 201, k)
}
func (s *Server) removeKVM(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
found := false
if err := s.cfg.Update(func(c *config.Config) {
kept := c.KVMs[:0]
for _, k := range c.KVMs {
if k.ID == id {
found = true
continue
}
kept = append(kept, k)
}
c.KVMs = kept
}); err != nil {
jsonErr(w, 500, err.Error())
return
}
if !found {
jsonErr(w, 404, "no such KVM")
return
}
s.forgetKVMProxy(id)
log.Printf("kvm %s removed by %s", id, clientIP(r))
w.WriteHeader(204)
}
internal/api/network.go
+123
View File
@@ -0,0 +1,123 @@
package api
// The Network tab: the machine's interfaces, how the internet looks from here, and the speed
// test. One machine at a time, like every other tab — the test runs on the machine in the
// switcher, so the number is that machine's line and not the hub's.
import (
"encoding/json"
"log"
"net/http"
"time"
"hamlab/internal/network"
"hamlab/internal/sensors"
"hamlab/internal/store"
)
// networkOverview is everything the tab draws at once: the interfaces, the outside view, the
// last test and the ones before it.
func (s *Server) networkOverview(w http.ResponseWriter, _ *http.Request) {
run := s.net.Test.State()
jsonOK(w, 200, map[string]any{
"report": network.Look(s.hostName()),
"outside": s.net.Outside(),
"run": run,
"history": s.net.Test.History(),
"method": network.Method,
})
}
func (s *Server) speedtestState(w http.ResponseWriter, _ *http.Request) {
jsonOK(w, 200, s.net.Test.State())
}
func (s *Server) speedtestStart(w http.ResponseWriter, r *http.Request) {
run, err := s.net.Test.Start()
if err != nil {
jsonErr(w, 409, err.Error())
return
}
log.Printf("speed test started on %s (by %s)", s.hostName(), clientIP(r))
jsonOK(w, 200, run)
}
func (s *Server) speedtestCancel(w http.ResponseWriter, r *http.Request) {
if !s.net.Test.Cancel() {
jsonErr(w, 404, "no speed test is running")
return
}
log.Printf("speed test cancelled on %s (by %s)", s.hostName(), clientIP(r))
w.WriteHeader(204)
}
// ---- keeping the results ----
// savedNetwork is what survives a restart: the recent results, so the history is a history
// rather than whatever has happened since the daemon last started.
type savedNetwork struct {
Results []network.Result `json:"results,omitempty"`
}
func (s *Server) loadNetwork() []network.Result {
var saved savedNetwork
var raw json.RawMessage
s.store.View(func(st *store.State) { raw = st.Network })
if len(raw) > 0 {
if err := json.Unmarshal(raw, &saved); err != nil {
log.Printf("network: kept results unreadable (%v); starting a fresh history", err)
}
}
return saved.Results
}
// keepResult is called when a test finishes: append it to what is on disk.
func (s *Server) keepResult(r network.Result) {
if r.Error != "" {
log.Printf("speed test on %s failed: %s", s.hostName(), r.Error)
} else {
log.Printf("speed test on %s: down %s, up %s, latency %.1f ms via %s",
s.hostName(), network.Mbits(r.DownMbps), network.Mbits(r.UpMbps), r.LatencyMs, r.Colo)
}
results := s.net.Test.History()
b, err := json.Marshal(savedNetwork{Results: results})
if err != nil {
return
}
if err := s.store.Update(func(st *store.State) { st.Network = b }); err != nil {
log.Printf("network: %v", err)
}
}
// ---- the speed test as sensors ----
// netSensors publishes the last speed test as ordinary sensors, so the line can be a widget,
// a graph, an alert or a Prometheus series like anything else Ham Lab measures. Each one's
// timestamp is when the test ran, which is how a stale reading says so.
type netSensors struct{ w *network.Watcher }
func (n netSensors) Sensors() []sensors.Sensor {
var out []sensors.Sensor
if o := n.w.Outside(); o.IP != "" {
at := o.At.UnixMilli()
out = append(out, sensors.Sensor{ID: "net.internet.ip", Name: "Public address", Kind: "state", Value: o.IP, T: at})
if o.Colo != "" {
out = append(out, sensors.Sensor{ID: "net.internet.edge", Name: "Internet edge", Kind: "state", Value: o.Colo, T: at})
}
}
run := n.w.Test.State()
if run.Result == nil || run.Result.Error != "" {
return out
}
r := run.Result
at := r.At.UnixMilli()
return append(out,
sensors.Sensor{ID: "net.internet.download", Name: "Download", Kind: "rate", Unit: "Mbps", Value: r.DownMbps, T: at, Min: sensors.F(0)},
sensors.Sensor{ID: "net.internet.upload", Name: "Upload", Kind: "rate", Unit: "Mbps", Value: r.UpMbps, T: at, Min: sensors.F(0)},
sensors.Sensor{ID: "net.internet.latency", Name: "Latency", Kind: "duration", Unit: "ms", Value: r.LatencyMs, T: at, Min: sensors.F(0)},
sensors.Sensor{ID: "net.internet.jitter", Name: "Jitter", Kind: "duration", Unit: "ms", Value: r.JitterMs, T: at, Min: sensors.F(0)},
sensors.Sensor{ID: "net.internet.tested", Name: "Speed test age", Kind: "duration", Unit: "s", Value: round1(time.Since(r.At).Seconds()), T: at, Min: sensors.F(0)},
)
}
func round1(v float64) float64 { return float64(int64(v*10+0.5)) / 10 }
internal/api/server.go
+2285 -41
View File
File diff suppressed because it is too large Load Diff
internal/api/space.go
+224
View File
@@ -0,0 +1,224 @@
package api
// The Space tab: what is using a disk or a pool. The unit is a whole volume, never a
// partition: a disk on its own is scanned through every filesystem it backs, and a ZFS pool
// through every mounted dataset, as one thing — its member disks are not scanned apart.
import (
"context"
"log"
"net/http"
"os/exec"
"regexp"
"sort"
"strconv"
"strings"
"time"
"hamlab/internal/space"
"hamlab/internal/storage"
)
// spaceTarget is one thing a scan can measure: a disk on its own, or a ZFS pool as a whole.
// A disk that is a member of a pool is not a target of its own — its share of the pool means
// nothing by itself — it is listed under the pool. (A disk with a pool partition *and* a
// filesystem of its own, root on one partition and a pool on another, is both.)
type spaceTarget struct {
ID string `json:"id"` // "nvme0" or "pool:tank"
Kind string `json:"kind"` // disk | pool
Name string `json:"name"`
Model string `json:"model,omitempty"` // disk: its model
Layout string `json:"layout,omitempty"` // pool: raidz2, mirror, …
Health string `json:"health,omitempty"` // pool: ONLINE, DEGRADED, …
Size uint64 `json:"size"`
Used uint64 `json:"used,omitempty"` // pool: allocated
Disks []string `json:"disks,omitempty"` // pool: its member disks
Roots []string `json:"roots"` // the mountpoints a scan walks
Scan *space.Scan `json:"scan,omitempty"` // the last scan, if any
datasets map[string]string // root → ZFS dataset, for the ones that are
}
var layoutRe = regexp.MustCompile(`\b(draid\d*|raidz[123]?|mirror)\b`)
// spaceTargets lists the disks and pools a scan can measure.
func (s *Server) spaceTargets(ctx context.Context) []spaceTarget {
devs, _ := storage.Devices(ctx)
var out []spaceTarget
pools := map[string]*spaceTarget{}
var poolOrder []string
for _, d := range devs {
disk := spaceTarget{ID: d.Name, Kind: "disk", Name: d.Name, Model: d.Model, Size: d.Size, Roots: []string{}}
seen := map[string]bool{}
inPool := false
for _, p := range d.Parts {
if p.FSType == "zfs_member" && p.Label != "" {
inPool = true
pt := pools[p.Label]
if pt == nil {
pt = &spaceTarget{ID: "pool:" + p.Label, Kind: "pool", Name: p.Label, Roots: []string{}, datasets: map[string]string{}}
for _, zm := range zfsMounts(ctx, p.Label) {
pt.Roots = append(pt.Roots, zm.mount)
pt.datasets[zm.mount] = zm.dataset
}
pools[p.Label] = pt
poolOrder = append(poolOrder, p.Label)
}
pt.Disks = append(pt.Disks, d.Name)
continue
}
if p.Mount != "" && p.Mount != "[SWAP]" && !seen[p.Mount] {
seen[p.Mount] = true
disk.Roots = append(disk.Roots, p.Mount)
}
}
if !inPool || len(disk.Roots) > 0 {
sort.Strings(disk.Roots)
out = append(out, disk)
}
}
if len(pools) > 0 {
// size, health and layout from zpool; the vdev tree names the layout
for _, p := range s.storage.ZFS(ctx).Pools {
if pt := pools[p.Name]; pt != nil {
pt.Size, pt.Used, pt.Health = p.Size, p.Alloc, p.Health
if m := layoutRe.FindString(p.Config); m != "" {
pt.Layout = m
} else if len(pt.Disks) == 1 {
pt.Layout = "single disk"
} else if len(pt.Disks) > 1 {
pt.Layout = "stripe"
}
}
}
var ps []spaceTarget
for _, name := range poolOrder {
sort.Strings(pools[name].Roots)
ps = append(ps, *pools[name])
}
out = append(ps, out...) // pools first: they are what a NAS is
}
if out == nil {
out = []spaceTarget{}
}
return out
}
type zfsMount struct{ mount, dataset string }
// zfsMounts is every mounted filesystem of a pool, with its dataset.
func zfsMounts(ctx context.Context, pool string) []zfsMount {
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-t", "filesystem", "-r", "-o", "name,mountpoint,mounted", pool).Output()
if err != nil {
return nil
}
var mounts []zfsMount
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
f := strings.Split(line, "\t")
if len(f) == 3 && f[2] == "yes" && strings.HasPrefix(f[1], "/") {
mounts = append(mounts, zfsMount{mount: f[1], dataset: f[0]})
}
}
return mounts
}
func (s *Server) spaceOverview(w http.ResponseWriter, r *http.Request) {
targets := s.spaceTargets(r.Context())
scans := s.space.All()
for i := range targets {
targets[i].Scan = scans[targets[i].ID]
}
jsonOK(w, 200, map[string]any{"targets": targets})
}
func (s *Server) spaceTarget(w http.ResponseWriter, r *http.Request, id string) (spaceTarget, bool) {
for _, t := range s.spaceTargets(r.Context()) {
if t.ID == id {
return t, true
}
}
jsonErr(w, 404, "no such disk or pool")
return spaceTarget{}, false
}
func (s *Server) spaceStatus(w http.ResponseWriter, r *http.Request) {
st := s.space.Status(r.PathValue("disk"))
if st == nil {
jsonErr(w, 404, "not scanned")
return
}
jsonOK(w, 200, st)
}
func (s *Server) spaceScan(w http.ResponseWriter, r *http.Request) {
t, ok := s.spaceTarget(w, r, r.PathValue("disk"))
if !ok {
return
}
st, err := s.space.Start(t.ID, t.Roots, t.datasets)
if err != nil {
jsonErr(w, 409, err.Error())
return
}
log.Printf("space: scan of %s (%s) started by %s", t.ID, strings.Join(t.Roots, " "), clientIP(r))
jsonOK(w, 202, st)
}
func (s *Server) spaceForget(w http.ResponseWriter, r *http.Request) {
s.space.Forget(r.PathValue("disk"))
log.Printf("space: scan of %s forgotten by %s", r.PathValue("disk"), clientIP(r))
w.WriteHeader(204)
}
func (s *Server) spaceCancel(w http.ResponseWriter, r *http.Request) {
if !s.space.Cancel(r.PathValue("disk")) {
jsonErr(w, 404, "no scan running")
return
}
w.WriteHeader(204)
}
func (s *Server) spaceTree(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
path := q.Get("path")
if path == "" {
path = "/"
}
depth, _ := strconv.Atoi(q.Get("depth"))
if depth < 1 || depth > 32 {
depth = 1
}
limit, _ := strconv.Atoi(q.Get("limit"))
if limit < 1 || limit > 2000 {
limit = 300
}
min, _ := strconv.ParseInt(q.Get("min"), 10, 64)
if min < 0 {
min = 0
}
t, err := s.space.Tree(r.PathValue("disk"), path, depth, limit, min)
if err != nil {
jsonErr(w, 404, err.Error())
return
}
w.Header().Set("Cache-Control", "no-store")
jsonOK(w, 200, t)
}
func (s *Server) spaceLargest(w http.ResponseWriter, r *http.Request) {
n, _ := strconv.Atoi(r.URL.Query().Get("n"))
if n < 1 || n > 5000 {
n = 500
}
files, err := s.space.Largest(r.PathValue("disk"), n)
if err != nil {
jsonErr(w, 404, err.Error())
return
}
if files == nil {
files = []space.File{}
}
jsonOK(w, 200, files)
}
internal/api/stacks.go
+1269
View File
File diff suppressed because it is too large Load Diff
internal/api/stacks_test.go
+121
View File
@@ -0,0 +1,121 @@
package api
import (
"context"
"os"
"path/filepath"
"testing"
"hamlab/internal/compose"
"hamlab/internal/store"
)
// A stack deployed by an orchestrator running in a container records the path its compose
// file had inside that container. Resolving it has to work both while the orchestrator is
// running — compose is then run inside it — and after it has been removed, where the same
// mount is only remembered and compose has to run on the host instead.
func TestResolveThroughMounts(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "orch", "stacks", "app")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
file := filepath.Join(dir, "compose.yaml")
if err := os.WriteFile(file, []byte("services: {}\n"), 0o644); err != nil {
t.Fatal(err)
}
inner := "/app/data/stacks/app/compose.yaml"
for _, tc := range []struct {
name, container, wantVia string
}{
{"orchestrator running", "orch", "orch"},
{"orchestrator removed", "", ""},
} {
t.Run(tc.name, func(t *testing.T) {
st := stack{Name: "app", File: inner, Files: []string{inner}}
st.resolve([]mountMap{{tc.container, "/app/data", filepath.Join(root, "orch")}})
if !st.Editable {
t.Fatalf("not editable: %s", st.Note)
}
if st.HostFile != file {
t.Errorf("host file = %q, want %q", st.HostFile, file)
}
if st.Via != tc.wantVia {
t.Errorf("via = %q, want %q", st.Via, tc.wantVia)
}
})
}
// Nothing explains the path: the stack is listed, but read-only and says why.
st := stack{Name: "app", File: inner, Files: []string{inner}}
st.resolve(nil)
if st.Editable || st.Note == "" {
t.Errorf("unresolvable stack should be read-only with a note, got editable=%v note=%q", st.Editable, st.Note)
}
}
// Every file of a multi-file stack must resolve through the same place. Redeploying from
// the first file alone would silently drop whatever the override set.
func TestResolveRefusesSplitStack(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "orch", "stacks", "app")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "compose.yaml"), []byte("services: {}\n"), 0o644); err != nil {
t.Fatal(err)
}
st := stack{Name: "app", File: "/app/data/stacks/app/compose.yaml", Files: []string{
"/app/data/stacks/app/compose.yaml",
"/app/data/stacks/app/override.yaml", // never written: not on disk
}}
st.resolve([]mountMap{{"orch", "/app/data", filepath.Join(root, "orch")}})
if st.Editable {
t.Error("a stack with a file that cannot be found should not be editable")
}
}
// A schedule made while an orchestrator was running says "run compose inside that
// container". When the container is gone the schedule must still work, against the same
// files on the host — otherwise retiring the orchestrator silently breaks every scheduled
// update made before it went.
func TestRepointSurvivesAMissingContainer(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "orch", "stacks", "app")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
host := filepath.Join(dir, "compose.yaml")
if err := os.WriteFile(host, []byte("services: {}\n"), 0o644); err != nil {
t.Fatal(err)
}
db, err := store.Open(filepath.Join(root, "state.json"))
if err != nil {
t.Fatal(err)
}
if err := db.Update(func(st *store.State) {
st.Mounts = []store.RememberedMount{{Container: "gone-orchestrator", Dest: "/app/data", Src: filepath.Join(root, "orch")}}
}); err != nil {
t.Fatal(err)
}
in := compose.Spec{Project: "app", Files: []string{"/app/data/stacks/app/compose.yaml"}, Via: "gone-orchestrator", Tool: "docker"}
got := repoint(context.Background(), db, in)
if got.Via != "" {
t.Errorf("via = %q, want it dropped", got.Via)
}
if len(got.Files) != 1 || got.Files[0] != host {
t.Errorf("files = %v, want [%s]", got.Files, host)
}
// Nothing remembered: the schedule is left exactly as it was rather than pointed at a
// guess. It will fail loudly, which is better than deploying the wrong file.
empty, err := store.Open(filepath.Join(root, "empty.json"))
if err != nil {
t.Fatal(err)
}
if got := repoint(context.Background(), empty, in); got.Via != in.Via || len(got.Files) != 1 || got.Files[0] != in.Files[0] {
t.Errorf("unresolvable spec was changed: %+v", got)
}
}
internal/api/status_sensors.go
+375
View File
@@ -0,0 +1,375 @@
package api
import (
"context"
"encoding/json"
"os"
"os/exec"
"strconv"
"strings"
"sync"
"time"
"hamlab/internal/priv"
"hamlab/internal/sensors"
"hamlab/internal/storage"
)
// statusSensors is the sensors.Provider for what isn't a /proc number: failed services,
// pending updates, pool health, drive health. Each source costs an exec or several, so
// they are read in the background on their own clocks and served from memory; every
// sensor's t says how old its reading really is.
type statusSensors struct {
s *Server
mu sync.RWMutex
by map[string][]sensors.Sensor // source → its sensors
cpuPrev map[string]cpuMark // container ID → CPU time at the previous read
hist map[string]*sensors.Series // numeric sensors only, statusHistory points each
acc map[string]*bucket // the point being built for each, when the window is wider than the read rate
}
// statusHistory is how many readings each status sensor keeps: at 15 s that is over an
// hour of container memory; at 10 min, two days of drive wear.
const statusHistory = 300
// A point being averaged, the same idea as the sampler's: these sources are read every 15 to
// 300 seconds, so a wide window keeps one point per stride rather than one per read.
type bucket struct {
last int64 // when this sensor's last point was kept
sum float64
n float64
}
const (
servicesEvery = 30 * time.Second
containersEvery = 15 * time.Second
updatesEvery = 5 * time.Minute
zfsEvery = time.Minute
smartEvery = 10 * time.Minute // storage.Manager caches SMART that long anyway, and never wakes a sleeping disk
)
func newStatusSensors(s *Server) *statusSensors {
p := &statusSensors{s: s, by: map[string][]sensors.Sensor{}, hist: map[string]*sensors.Series{}, acc: map[string]*bucket{}}
go p.loop("services", servicesEvery, p.services)
go p.loop("containers", containersEvery, p.containers)
go p.loop("updates", updatesEvery, p.updates)
go p.loop("zfs", zfsEvery, p.zfs)
go p.loop("smart", smartEvery, p.smart)
return p
}
func (p *statusSensors) loop(name string, every time.Duration, read func(context.Context, int64) []sensors.Sensor) {
for {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
got := read(ctx, time.Now().UnixMilli())
cancel()
p.mu.Lock()
p.by[name] = got
window := p.s.metrics.Window()
stride := window.Milliseconds() / statusHistory
live := map[string]bool{}
for _, x := range got {
v, ok := x.Value.(float64)
if !ok {
continue
}
live[x.ID] = true
h := p.hist[x.ID]
if h == nil {
h = &sensors.Series{ID: x.ID, Unit: x.Unit}
p.hist[x.ID] = h
}
b := p.acc[x.ID]
if b == nil {
b = &bucket{}
p.acc[x.ID] = b
}
b.sum, b.n = b.sum+v, b.n+1
// due a stride after the last point kept, with half a read of slack for a late timer
if b.last != 0 && x.T-b.last < stride-every.Milliseconds()/2 {
continue // still filling this point
}
h.T, h.V = append(h.T, x.T), append(h.V, b.sum/b.n)
*b = bucket{last: x.T}
if len(h.T) > statusHistory {
h.T, h.V = h.T[len(h.T)-statusHistory:], h.V[len(h.V)-statusHistory:]
}
if cut := x.T - window.Milliseconds(); h.T[0] < cut { // and nothing older than the window
i := 0
for i < len(h.T) && h.T[i] < cut {
i++
}
h.T, h.V = h.T[i:], h.V[i:]
}
}
for id := range p.hist { // a container that is gone takes its history with it
if strings.HasPrefix(id, prefixOf[name]) && !live[id] && prefixOf[name] != "" {
delete(p.hist, id)
delete(p.acc, id)
}
}
p.mu.Unlock()
if len(got) == 0 { // no pools, no Docker, no SMART: look again, but rarely
time.Sleep(10 * every)
continue
}
time.Sleep(every)
}
}
// prefixOf says which IDs a source owns, so its loop forgets only its own departed sensors.
var prefixOf = map[string]string{"containers": "container.", "smart": "smart.", "zfs": "zfs."}
// SensorHistories implements sensors.Historian.
func (p *statusSensors) SensorHistories(match func(id string) bool) []sensors.Series {
p.mu.RLock()
defer p.mu.RUnlock()
var out []sensors.Series
for id, h := range p.hist {
if match(id) {
out = append(out, sensors.Series{ID: h.ID, Unit: h.Unit, T: append([]int64{}, h.T...), V: append([]float64{}, h.V...)})
}
}
return out
}
func (p *statusSensors) Sensors() []sensors.Sensor {
p.mu.RLock()
defer p.mu.RUnlock()
var out []sensors.Sensor
for _, v := range p.by {
out = append(out, v...)
}
return out
}
func (p *statusSensors) services(ctx context.Context, t int64) []sensors.Sensor {
units, err := p.s.svc.List(ctx)
if err != nil {
return nil
}
var failed, running float64
var names []string
var per []sensors.Sensor
for _, u := range units {
if u.Failed {
failed++
names = append(names, u.Name)
}
if u.Sub == "running" {
running++
}
// One pair per service that is meant to be doing something: enabled, up, or broken.
// (Every static and one-shot unit on the box would triple the catalog for nothing.)
short, isService := strings.CutSuffix(u.Name, ".service")
if !isService || !(u.Enabled == "enabled" || u.Active == "active" || u.Failed) {
continue
}
id, l := "service."+sensors.Seg(short), map[string]string{"unit": u.Name, "description": u.Description}
per = append(per,
sensors.Sensor{ID: id + ".active", Name: short, Kind: "state", Value: u.Active == "active", T: t, Labels: l},
sensors.Sensor{ID: id + ".state", Name: short + " state", Kind: "state", Value: u.Sub, T: t, Labels: l})
}
return append(per, []sensors.Sensor{
{ID: "services.failed", Name: "Failed services", Kind: "count", Value: failed, T: t, Warn: sensors.F(1)},
{ID: "services.failed_names", Name: "Failed services", Kind: "state", Value: strings.Join(names, ", "), T: t},
{ID: "services.running", Name: "Running services", Kind: "count", Value: running, T: t},
}...)
}
// containers reads Docker (or Podman) with one `ps`, then each running container's memory
// and CPU straight from its cgroup — `docker stats` takes seconds and a daemon round-trip
// per call; two small files per container take microseconds.
func (p *statusSensors) containers(ctx context.Context, t int64) []sensors.Sensor {
var out []byte
var err error
tool := ""
for _, t := range []string{"docker", "podman"} {
if _, e := exec.LookPath(t); e != nil {
continue
}
tool = t
args := []string{tool, "ps", "-a", "--no-trunc", "--format", "{{json .}}"}
if out, err = exec.CommandContext(ctx, args[0], args[1:]...).Output(); err != nil && priv.Available() {
full := priv.Wrap(args...) // not in the docker group: ask as root
out, err = exec.CommandContext(ctx, full[0], full[1:]...).Output()
}
break
}
if err != nil || len(out) == 0 {
return nil
}
// "Not running" is not "down": servarr stopped by hand in June is off, not broken. What is
// broken is a container that should be up and isn't — one inspect of the stopped ones tells
// them apart: Docker refused to start it (State.Error: yesterday's Gitea), it is stuck
// restarting, or it died with a non-zero code that no restart policy picked up. Under
// `always` / `unless-stopped` a crash is restarted by Docker, so "exited" there means a
// person stopped it. With no restart policy, `docker stop` leaves 143 (the process took the
// SIGTERM), 137 (it didn't, and Docker killed it after the grace period) or 130 (SIGINT):
// also a person, unless the kernel did the killing for want of memory.
type stopped struct {
Name string
State struct {
Status string
ExitCode int
Error string
OOMKilled bool
}
HostConfig struct{ RestartPolicy struct{ Name string } }
}
down := map[string]bool{}
var ids []string
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
var c struct{ ID, State string }
if json.Unmarshal([]byte(line), &c) == nil && c.ID != "" && c.State != "running" {
ids = append(ids, c.ID)
}
}
if len(ids) > 0 {
args := append([]string{"inspect"}, ids...)
insp, ierr := exec.CommandContext(ctx, tool, args...).Output()
if ierr != nil && priv.Available() {
full := priv.Wrap(append([]string{tool}, args...)...)
insp, _ = exec.CommandContext(ctx, full[0], full[1:]...).Output()
}
var list []stopped
if json.Unmarshal(insp, &list) == nil {
for _, c := range list {
pol := c.HostConfig.RestartPolicy.Name
managed := pol == "always" || pol == "unless-stopped"
code := c.State.ExitCode
byHand := (code == 130 || code == 137 || code == 143) && !c.State.OOMKilled
down[strings.TrimPrefix(c.Name, "/")] = c.State.Status == "restarting" || c.State.Status == "dead" || c.State.Error != "" ||
(!managed && c.State.Status == "exited" && code != 0 && !byHand)
}
}
}
var res []sensors.Sensor
var running, unhealthy, total float64
seenCPU := map[string]cpuMark{}
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
var c struct{ ID, Names, Image, State, Status string }
if json.Unmarshal([]byte(line), &c) != nil || c.Names == "" {
continue
}
total++
name := strings.Split(c.Names, ",")[0]
id, l := "container."+sensors.Seg(name), map[string]string{"container": name, "image": c.Image}
up := c.State == "running"
health := "none"
for _, h := range []string{"unhealthy", "healthy", "starting"} { // "unhealthy" first: it contains "healthy"
if strings.Contains(c.Status, "("+h+")") || strings.Contains(c.Status, "(health: "+h+")") {
health = h
break
}
}
if up {
running++
}
if health == "unhealthy" {
unhealthy++
}
res = append(res,
sensors.Sensor{ID: id + ".running", Name: name, Kind: "state", Value: up, T: t, Labels: l},
sensors.Sensor{ID: id + ".down", Name: name + " down", Kind: "state", Value: down[name], T: t, Labels: l},
sensors.Sensor{ID: id + ".state", Name: name + " state", Kind: "state", Value: c.State, T: t, Labels: l},
sensors.Sensor{ID: id + ".health", Name: name + " health", Kind: "state", Value: health, T: t, Labels: l})
if !up {
continue
}
cg := ""
for _, dir := range []string{"/sys/fs/cgroup/system.slice/docker-" + c.ID + ".scope", "/sys/fs/cgroup/docker/" + c.ID, "/sys/fs/cgroup/machine.slice/libpod-" + c.ID + ".scope"} {
if _, e := os.Stat(dir + "/memory.current"); e == nil {
cg = dir
break
}
}
if cg == "" {
continue
}
if b, e := os.ReadFile(cg + "/memory.current"); e == nil {
if v, e := strconv.ParseFloat(strings.TrimSpace(string(b)), 64); e == nil {
res = append(res, sensors.Sensor{ID: id + ".mem", Name: name + " memory", Kind: "bytes", Unit: "B", Value: v, T: t, Labels: l})
}
}
if b, e := os.ReadFile(cg + "/cpu.stat"); e == nil {
for _, ln := range strings.Split(string(b), "\n") {
if f := strings.Fields(ln); len(f) == 2 && f[0] == "usage_usec" {
usec, _ := strconv.ParseFloat(f[1], 64)
seenCPU[c.ID] = cpuMark{usec, t}
if prev, ok := p.cpuPrev[c.ID]; ok && t > prev.t && usec >= prev.usec {
// percent of one core, like the Processes tab
pctv := (usec - prev.usec) / 1000 / float64(t-prev.t) * 100
res = append(res, sensors.Sensor{ID: id + ".cpu", Name: name + " CPU", Kind: "usage", Unit: "%", Value: float64(int(pctv*10+0.5)) / 10, T: t, Min: sensors.F(0), Labels: l})
}
}
}
}
}
p.cpuPrev = seenCPU // only this loop touches it
return append(res,
sensors.Sensor{ID: "containers.total", Name: "Containers", Kind: "count", Value: total, T: t},
sensors.Sensor{ID: "containers.running", Name: "Running containers", Kind: "count", Value: running, T: t},
sensors.Sensor{ID: "containers.unhealthy", Name: "Unhealthy containers", Kind: "count", Value: unhealthy, T: t, Warn: sensors.F(1)})
}
type cpuMark struct {
usec float64
t int64
}
func (p *statusSensors) updates(ctx context.Context, t int64) []sensors.Sensor {
st := p.s.updates.Status(ctx)
if st.Manager == "" {
return nil
}
return []sensors.Sensor{
{ID: "updates.pending", Name: "Pending updates", Kind: "count", Value: float64(len(st.Packages)), T: t, Labels: map[string]string{"manager": st.Manager}},
{ID: "updates.security", Name: "Pending security updates", Kind: "count", Value: float64(st.SecurityCount), T: t, Warn: sensors.F(1)},
{ID: "updates.reboot_required", Name: "Reboot required", Kind: "state", Value: st.RebootNeeded, T: t},
}
}
func (p *statusSensors) zfs(ctx context.Context, t int64) []sensors.Sensor {
var out []sensors.Sensor
for _, pool := range p.s.storage.Pools(ctx) {
id, l := "zfs."+sensors.Seg(pool.Name), map[string]string{"pool": pool.Name}
out = append(out,
sensors.Sensor{ID: id + ".health", Name: pool.Name + " health", Kind: "state", Value: pool.Health, T: t, Labels: l},
sensors.Sensor{ID: id + ".healthy", Name: pool.Name + " healthy", Kind: "state", Value: pool.Health == "ONLINE", T: t, Labels: l},
sensors.Sensor{ID: id + ".size", Name: pool.Name + " size", Kind: "bytes", Unit: "B", Value: float64(pool.Size), T: t, Labels: l},
sensors.Sensor{ID: id + ".used", Name: pool.Name + " used", Kind: "bytes", Unit: "B", Value: float64(pool.Alloc), T: t, Min: sensors.F(0), Max: sensors.F(float64(pool.Size)), Labels: l},
sensors.Sensor{ID: id + ".used_pct", Name: pool.Name + " used", Kind: "usage", Unit: "%", Value: float64(pool.Cap), T: t, Min: sensors.F(0), Max: sensors.F(100), Labels: l},
sensors.Sensor{ID: id + ".frag", Name: pool.Name + " fragmentation", Kind: "fragmentation", Unit: "%", Value: float64(pool.Frag), T: t, Min: sensors.F(0), Max: sensors.F(100), Labels: l},
)
}
return out
}
func (p *statusSensors) smart(ctx context.Context, t int64) []sensors.Sensor {
devs, err := storage.Devices(ctx)
if err != nil {
return nil
}
var out []sensors.Sensor
for _, d := range devs {
sm := p.s.storage.Smart(ctx, d.Name, false)
if sm.Passed == nil || sm.Standby || sm.Error != "" {
continue // no verdict to report: no smartctl, asleep, or not a SMART device
}
id, l := "smart."+sensors.Seg(d.Name), map[string]string{"device": d.Name, "model": d.Model}
out = append(out,
sensors.Sensor{ID: id + ".passed", Name: d.Name + " health", Kind: "state", Value: *sm.Passed, T: sm.FetchedAt, Labels: l},
sensors.Sensor{ID: id + ".warnings", Name: d.Name + " warnings", Kind: "count", Value: float64(len(sm.Warnings)), T: sm.FetchedAt, Warn: sensors.F(1), Labels: l},
)
if sm.PowerOnHours > 0 {
out = append(out, sensors.Sensor{ID: id + ".power_on", Name: d.Name + " powered on", Kind: "duration", Unit: "s", Value: float64(sm.PowerOnHours) * 3600, T: sm.FetchedAt, Labels: l})
}
if sm.NVMe != nil {
out = append(out, sensors.Sensor{ID: id + ".wear", Name: d.Name + " endurance used", Kind: "usage", Unit: "%", Value: float64(sm.NVMe.PercentageUsed), T: sm.FetchedAt, Min: sensors.F(0), Max: sensors.F(100), Labels: l})
}
}
return out
}
internal/api/streamgate_test.go
+29
View File
@@ -0,0 +1,29 @@
package api
import "testing"
func TestStreamGate(t *testing.T) {
var g streamGate
for i := 0; i < maxStreamsPerClient; i++ {
if !g.enter("a") {
t.Fatalf("stream %d for one client refused early", i)
}
}
if g.enter("a") {
t.Fatal("one client went past its ceiling")
}
if !g.enter("b") {
t.Fatal("another client must still get in")
}
g.leave("a")
if !g.enter("a") {
t.Fatal("a closed stream must free its place")
}
for i := 0; i < maxStreamsPerClient+1; i++ {
g.leave("a")
}
g.leave("b")
if g.total != -1 && g.total != 0 { // leave() past zero must not wedge the gate open or shut
t.Logf("total after over-leave: %d", g.total)
}
}
internal/api/update.go
+217
View File
@@ -0,0 +1,217 @@
package api
// Updating Ham Lab itself: is there a newer build published, and install it on request.
// The check is a plain HTTP GET against whatever `update.url` names — this project's Gitea
// releases by default — so a fleet, a stranger's install and an air-gapped copy pointed at
// a directory on a NAS all work the same way. Nothing is ever installed unasked: an update
// restarts the daemon and drops open terminals, which is not a thing to do to somebody.
import (
"context"
"encoding/json"
"io"
"log"
"net/http"
"path/filepath"
"runtime"
"strings"
"sync"
"time"
"hamlab/internal/config"
"hamlab/internal/selfupdate"
)
type updateState struct {
mu sync.Mutex
checking bool
installing bool
checkedAt time.Time
latest selfupdate.Release
err string
}
// source is the update source as configured, and whether looking is switched off.
func (s *Server) source() (selfupdate.Source, bool) {
var u config.UpdateSource
s.cfg.View(func(c *config.Config) { u = c.Updates })
url := strings.TrimSpace(u.URL)
if url == "" {
url = selfupdate.DefaultURL
}
if strings.EqualFold(url, "off") || strings.EqualFold(url, "none") {
return selfupdate.Source{}, false
}
return selfupdate.Source{URL: url, Token: u.Token}, true
}
func (s *Server) checksEnabled() bool {
var u config.UpdateSource
s.cfg.View(func(c *config.Config) { u = c.Updates })
if strings.EqualFold(u.Check, "never") {
return false
}
_, on := s.source()
return on
}
// Behind reports whether a newer build than this one has been seen, and which.
func (s *Server) behind() (string, bool) {
s.update.mu.Lock()
defer s.update.mu.Unlock()
v := s.update.latest.Version
if v == "" || selfupdate.Compare(s.version, v) >= 0 {
return "", false
}
return v, true
}
// checkNow asks the source what the newest build is. One at a time.
func (s *Server) checkNow(ctx context.Context) {
src, on := s.source()
if !on {
return
}
s.update.mu.Lock()
if s.update.checking {
s.update.mu.Unlock()
return
}
s.update.checking = true
s.update.mu.Unlock()
ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
rel, err := src.Latest(ctx, runtime.GOARCH)
s.update.mu.Lock()
s.update.checking = false
s.update.checkedAt = time.Now()
if err != nil {
s.update.err = err.Error()
} else {
s.update.err, s.update.latest = "", rel
}
s.update.mu.Unlock()
if err != nil {
log.Printf("update check: %v", err)
return
}
if selfupdate.Compare(s.version, rel.Version) < 0 {
log.Printf("update check: %s is published; this is %s", rel.Version, s.version)
}
}
// watchForUpdates asks once shortly after boot and then daily.
func (s *Server) watchForUpdates(ctx context.Context) {
t := time.NewTimer(90 * time.Second)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
}
if s.checksEnabled() {
s.checkNow(ctx)
}
t.Reset(selfupdate.DefaultCheckEvery)
}
}
func (s *Server) updateStatus(w http.ResponseWriter, _ *http.Request) {
src, on := s.source()
s.update.mu.Lock()
defer s.update.mu.Unlock()
latest := s.update.latest.Version
out := map[string]any{
"current": s.version, "latest": latest, "arch": runtime.GOARCH,
"behind": latest != "" && selfupdate.Compare(s.version, latest) < 0,
"checking": s.update.checking,
"installing": s.update.installing,
"enabled": on, "auto": s.checksEnabled(), "url": src.URL,
"notes": s.update.latest.Notes, "published": s.update.latest.Published,
}
if !s.update.checkedAt.IsZero() {
out["checked"] = s.update.checkedAt
}
if s.update.err != "" {
out["error"] = s.update.err
}
w.Header().Set("Cache-Control", "no-store")
jsonOK(w, 200, out)
}
func (s *Server) updateCheck(w http.ResponseWriter, r *http.Request) {
if _, on := s.source(); !on {
jsonErr(w, 400, "update checks are switched off (update.url is \"off\")")
return
}
s.checkNow(r.Context())
s.updateStatus(w, r)
}
// updateInstall fetches the published build for this machine and puts it in place.
func (s *Server) updateInstall(w http.ResponseWriter, r *http.Request) {
src, on := s.source()
if !on {
jsonErr(w, 400, "update checks are switched off (update.url is \"off\")")
return
}
force := r.URL.Query().Get("force") == "1"
if err := s.updateAllowed(force); err != nil {
jsonErr(w, 409, err.Error())
return
}
var body struct {
Version string `json:"version"` // optional: refuse if what is published is not this
}
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
s.update.mu.Lock()
if s.update.installing {
s.update.mu.Unlock()
jsonErr(w, 409, "an update is already being installed")
return
}
rel := s.update.latest
s.update.installing = true
s.update.mu.Unlock()
done := func() { s.update.mu.Lock(); s.update.installing = false; s.update.mu.Unlock() }
ctx := context.WithoutCancel(r.Context()) // a closed browser tab must not abort an install
if rel.Version == "" || body.Version != "" && body.Version != rel.Version {
var err error
if rel, err = src.Latest(ctx, runtime.GOARCH); err != nil {
done()
jsonErr(w, 502, err.Error())
return
}
s.update.mu.Lock()
s.update.latest, s.update.checkedAt, s.update.err = rel, time.Now(), ""
s.update.mu.Unlock()
}
if body.Version != "" && body.Version != rel.Version {
done()
jsonErr(w, 409, "what is published is now "+rel.Version+", not "+body.Version+"; look again")
return
}
if !force && selfupdate.Compare(s.version, rel.Version) >= 0 {
done()
jsonErr(w, 409, "this machine is already running "+s.version+"; nothing newer is published")
return
}
tmp := filepath.Join(s.cfg.DataDir(), "hamlab.new")
if err := src.Fetch(ctx, rel, runtime.GOARCH, tmp); err != nil {
done()
jsonErr(w, 502, err.Error())
return
}
newVersion, err := s.installBinary(ctx, tmp)
if err != nil {
done()
jsonErr(w, codeOf(err, 500), err.Error())
return
}
log.Printf("update: installed %s (was %s), asked for by %s; restarting", newVersion, s.version, clientIP(r))
jsonOK(w, 200, map[string]string{"version": newVersion, "previous": s.version})
s.restartAfterUpdate()
}
internal/api/users.go
+276
View File
@@ -0,0 +1,276 @@
package api
// Accounts and their sessions. Every person has their own login, and the console's look
// (theme, accents, dashboard layout) follows the account rather than the machine. Anyone
// signed in may add or remove accounts — a household console, not a tenancy — but removing
// one, or resetting another's password, asks for the caller's own password first, so a
// picked-up unlocked phone is not enough.
import (
"context"
"encoding/json"
"io"
"log"
"net/http"
"strings"
"time"
"hamlab/internal/auth"
"hamlab/internal/config"
"hamlab/internal/store"
)
// actor is who a request is from: a signed-in account (with its session), or a token.
type actor struct {
User string // account name, "" for a token
Session string // session ID, "" for a token
token string // the raw cookie token, so a handler can spare the current session
Token string // token name, "" for a session
// Guest marks a temporary look-around login, and Expires is when it stops working. What
// a guest may reach is decided in the middleware, not here; see internal/api/guests.go.
Guest bool
Expires time.Time
}
type ctxKey int
const actorKey ctxKey = 1
func withActor(r *http.Request, a actor) *http.Request {
return r.WithContext(context.WithValue(r.Context(), actorKey, a))
}
func actorOf(r *http.Request) actor {
a, _ := r.Context().Value(actorKey).(actor)
return a
}
// who names the caller for a log line: "core@10.0.0.5" for a person, "token:hub:core@…" for a hub.
func (a actor) String() string {
switch {
case a.User != "" && a.Guest:
return "guest:" + a.User
case a.User != "":
return a.User
case a.Token != "":
return "token:" + a.Token
}
return ""
}
// ---- users ----
func (s *Server) listUsers(w http.ResponseWriter, r *http.Request) {
me := actorOf(r).User
counts := s.sessions.Count()
var out []map[string]any
s.cfg.View(func(c *config.Config) {
for _, u := range c.Accounts() { // guest logins are listed on their own, see listGuests
var seen any
if !u.LastSeen.IsZero() {
seen = u.LastSeen
}
out = append(out, map[string]any{"name": u.Name, "created": u.Created, "last_seen": seen, "sessions": counts[u.Name], "self": u.Name == me})
}
})
if out == nil {
out = []map[string]any{}
}
jsonOK(w, 200, out)
}
func (s *Server) addUser(w http.ResponseWriter, r *http.Request) {
var body struct{ Name, Password string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
name := strings.TrimSpace(body.Name)
if !auth.UserNameRe.MatchString(name) {
jsonErr(w, 400, "a user name is 1–32 lower-case letters, digits, . _ or -, starting with a letter or digit")
return
}
if strings.HasPrefix(name, guestPrefix) {
jsonErr(w, 400, "names starting with "+guestPrefix+" belong to guest logins")
return
}
if len(body.Password) < auth.MinPasswordLen {
jsonErr(w, 400, "password must be at least 12 characters")
return
}
h, err := auth.HashPassword(body.Password)
if err != nil {
jsonErr(w, 500, err.Error())
return
}
clash := false
if err := s.cfg.Update(func(c *config.Config) {
if _, ok := c.User(name); ok {
clash = true
return
}
c.Users = append(c.Users, config.User{Name: name, Hash: h, Created: time.Now()})
}); err != nil {
jsonErr(w, 500, err.Error())
return
}
if clash {
jsonErr(w, 409, "a user with that name exists")
return
}
log.Printf("user %q added (by %s)", name, clientIP(r))
jsonOK(w, 200, map[string]any{"name": name})
}
// confirmSelf checks the caller's own password, the gate on anything done to another account.
func (s *Server) confirmSelf(w http.ResponseWriter, r *http.Request, current string) bool {
me := actorOf(r).User
if me == "" {
jsonErr(w, 403, "only a signed-in user may do that")
return false
}
var users []config.User
s.cfg.View(func(c *config.Config) { users = append(users, c.Users...) })
if !auth.CheckUser(users, me, current) {
jsonErr(w, 403, "your password is wrong")
return false
}
return true
}
func (s *Server) removeUser(w http.ResponseWriter, r *http.Request) {
name := r.PathValue("name")
var body struct{ Current string }
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
if name == actorOf(r).User {
jsonErr(w, 400, "you can't remove the account you are signed in as")
return
}
if !s.confirmSelf(w, r, body.Current) {
return
}
found, last := false, false
if err := s.cfg.Update(func(c *config.Config) {
if len(c.Accounts()) <= 1 { // guest logins don't keep the last real account alive
last = true
return
}
kept := c.Users[:0]
for _, u := range c.Users {
if u.Name == name {
found = true
continue
}
kept = append(kept, u)
}
c.Users = kept
}); err != nil {
jsonErr(w, 500, err.Error())
return
}
if last {
jsonErr(w, 400, "the last account can't be removed")
return
}
if !found {
jsonErr(w, 404, "no such user")
return
}
s.sessions.RevokeUser(name)
_ = s.store.Update(func(st *store.State) { delete(st.Users, name) })
log.Printf("user %q removed (by %s)", name, clientIP(r))
w.WriteHeader(204)
}
// resetPassword sets another user's password, on the caller's own password.
func (s *Server) resetPassword(w http.ResponseWriter, r *http.Request) {
name := r.PathValue("name")
var body struct{ Current, Password string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
if !s.confirmSelf(w, r, body.Current) {
return
}
if len(body.Password) < auth.MinPasswordLen {
jsonErr(w, 400, "password must be at least 12 characters")
return
}
if !s.setPassword(w, name, body.Password) {
return
}
if name != actorOf(r).User { // whoever held the old password is signed out
s.sessions.RevokeUser(name)
}
log.Printf("password of %q reset (by %s)", name, clientIP(r))
w.WriteHeader(204)
}
func (s *Server) setPassword(w http.ResponseWriter, name, pw string) bool {
h, err := auth.HashPassword(pw)
if err != nil {
jsonErr(w, 500, err.Error())
return false
}
found := false
if err := s.cfg.Update(func(c *config.Config) {
for i := range c.Users {
if c.Users[i].Name == name {
c.Users[i].Hash = h
found = true
}
}
}); err != nil {
jsonErr(w, 500, err.Error())
return false
}
if !found {
jsonErr(w, 404, "no such user")
return false
}
return true
}
// ---- sessions: where the caller's account is signed in ----
func (s *Server) listSessions(w http.ResponseWriter, r *http.Request) {
a := actorOf(r)
if a.User == "" {
jsonOK(w, 200, []any{})
return
}
var out []map[string]any
for _, ses := range s.sessions.List(a.User) {
out = append(out, map[string]any{"id": ses.ID, "created": ses.Created, "last_seen": ses.LastSeen, "expires": ses.Expires, "ip": ses.IP, "agent": ses.Agent, "current": ses.ID == a.Session})
}
if out == nil {
out = []map[string]any{}
}
jsonOK(w, 200, out)
}
func (s *Server) revokeSession(w http.ResponseWriter, r *http.Request) {
a := actorOf(r)
id := r.PathValue("id")
if a.User == "" || !s.sessions.Revoke(a.User, id) {
jsonErr(w, 404, "no such session")
return
}
log.Printf("session %s of %q revoked (by %s)", id, a.User, clientIP(r))
if id == a.Session {
s.setCookie(w, r, "", -1)
}
w.WriteHeader(204)
}
func (s *Server) revokeOtherSessions(w http.ResponseWriter, r *http.Request) {
a := actorOf(r)
if a.User == "" {
jsonErr(w, 403, "only a signed-in user may do that")
return
}
n := s.sessions.RevokeOthers(a.User, a.token)
log.Printf("%d other session(s) of %q revoked (by %s)", n, a.User, clientIP(r))
jsonOK(w, 200, map[string]int{"revoked": n})
}
internal/api/vpn.go
+115
View File
@@ -0,0 +1,115 @@
package api
// The VPN tab: this machine's WireGuard tunnels, who is on them, and — where the tunnel is
// one this machine configures — adding, renaming and removing the devices that dial in.
//
// Per machine like every other tab: the tunnels shown and edited are the ones on the machine
// in the switcher, which is the only honest way to show them, since the inbound server and an
// outbound tunnel are different machines here.
import (
"context"
"encoding/json"
"errors"
"io"
"log"
"net/http"
"strings"
"hamlab/internal/vpn"
)
func (s *Server) vpnReport(w http.ResponseWriter, r *http.Request) {
jsonOK(w, 200, vpn.Look(r.Context()))
}
// tunnelOf finds the tunnel a request names, so every editing handler starts from the live
// state rather than from whatever the page last saw.
func (s *Server) tunnelOf(ctx context.Context, w http.ResponseWriter, name string) (vpn.Tunnel, bool) {
for _, t := range vpn.Look(ctx).Tunnels {
if t.Name == name {
if !t.Managed {
jsonErr(w, 400, vpn.ErrUnmanaged.Error())
return t, false
}
return t, true
}
}
jsonErr(w, 404, "no such tunnel on this machine")
return vpn.Tunnel{}, false
}
// addVPNPeer makes a device and hands back its profile. The profile holds a private key that
// is generated for this call and stored nowhere, so it is the only time it can be read — the
// same bargain as any password this console generates.
func (s *Server) addVPNPeer(w http.ResponseWriter, r *http.Request) {
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
if !ok {
return
}
var body vpn.NewPeer
if err := json.NewDecoder(io.LimitReader(r.Body, 8<<10)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
made, err := vpn.AddPeer(r.Context(), t, body)
if err != nil {
code := 400
if errors.Is(err, vpn.ErrUnmanaged) {
code = 409
}
jsonErr(w, code, err.Error())
return
}
// The name and address are worth a log line; the profile is emphatically not.
log.Printf("vpn: device %q added to %s as %s (by %s)", made.Name, t.Name, made.Address, clientIP(r))
jsonOK(w, 200, made)
}
func (s *Server) editVPNPeer(w http.ResponseWriter, r *http.Request) {
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
if !ok {
return
}
var body struct{ Name, Address string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4<<10)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
section := r.PathValue("section")
if err := vpn.EditPeer(r.Context(), t, section, body.Name, body.Address); err != nil {
jsonErr(w, 400, err.Error())
return
}
log.Printf("vpn: device %s on %s changed (by %s)", section, t.Name, clientIP(r))
w.WriteHeader(204)
}
func (s *Server) removeVPNPeer(w http.ResponseWriter, r *http.Request) {
t, ok := s.tunnelOf(r.Context(), w, r.PathValue("iface"))
if !ok {
return
}
section := r.PathValue("section")
var body struct{ Confirm string }
_ = json.NewDecoder(io.LimitReader(r.Body, 4<<10)).Decode(&body)
// The name has to be typed back. Removing a device is the one action here that cannot be
// undone — the key is gone and the profile on the device stops working — and it is one tap
// away from a list of devices that all look alike.
var want string
for _, p := range t.Peers {
if p.Section == section {
want = p.Name
}
}
if want != "" && !strings.EqualFold(strings.TrimSpace(body.Confirm), want) {
jsonErr(w, 400, "type the device's name to remove it")
return
}
if err := vpn.DeletePeer(r.Context(), t, section); err != nil {
jsonErr(w, 400, err.Error())
return
}
log.Printf("vpn: device %s (%s) removed from %s (by %s)", section, want, t.Name, clientIP(r))
w.WriteHeader(204)
}
internal/api/zfs.go
+262
View File
@@ -0,0 +1,262 @@
package api
// The ZFS tab: pools, datasets, snapshots, and a schedule for taking and expiring them.
//
// grid already runs zfSnap from root cron and has done for longer than this repo has
// existed, so this does not replace it. It reads the same snapshot names, expires by the
// same rule, and surfaces the cron job that drives it — so what is on screen is the truth
// about the machine, whether Ham Lab or zfSnap is doing the work. A Ham Lab schedule is for
// a machine that has no such tool, and the page says plainly when both are running.
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"io"
"log"
"net/http"
"os"
"path/filepath"
"strings"
"time"
"hamlab/internal/storage"
"hamlab/internal/store"
)
type zfsState struct {
Policies []storage.Policy `json:"policies"`
}
func (s *Server) zfsLoad() zfsState {
var raw json.RawMessage
s.store.View(func(st *store.State) { raw = st.ZFS })
var z zfsState
if len(raw) > 0 {
_ = json.Unmarshal(raw, &z)
}
if z.Policies == nil {
z.Policies = []storage.Policy{}
}
return z
}
func (s *Server) zfsSave(z zfsState) error {
b, err := json.Marshal(z)
if err != nil {
return err
}
return s.store.Update(func(st *store.State) { st.ZFS = b })
}
// externalSnapshotJob finds a cron job that is already taking snapshots on this machine, so
// the page can say so rather than let two schedules quietly double up.
func (s *Server) externalSnapshotJob(ctx context.Context) map[string]any {
if s.cron == nil {
return nil
}
ov := s.cron.Overview(ctx)
for _, j := range ov.Jobs {
if !j.Enabled {
continue
}
c := strings.ToLower(j.Command)
if !strings.Contains(c, "zfsnap") && !strings.Contains(c, "zfs-auto-snapshot") && !strings.Contains(c, "sanoid") &&
!(strings.Contains(c, "zfs ") && strings.Contains(c, "snapshot")) {
continue
}
return map[string]any{
"tool": toolName(c), "schedule": j.Schedule, "command": j.Command,
"user": j.User, "source": j.Source, "id": j.ID, "next": j.Next, "last": j.Last,
}
}
return nil
}
func toolName(cmd string) string {
switch {
case strings.Contains(cmd, "zfsnap"):
return "zfSnap"
case strings.Contains(cmd, "zfs-auto-snapshot"):
return "zfs-auto-snapshot"
case strings.Contains(cmd, "sanoid"):
return "sanoid"
}
return "a cron job"
}
// randomID is a short opaque id for a schedule.
func randomID() (string, error) {
b := make([]byte, 6)
if _, err := rand.Read(b); err != nil {
return "", err
}
return hex.EncodeToString(b), nil
}
// ---- the cron file ----
//
// A schedule is a cron line, written to /etc/cron.d/hamlab-zfs, which Ham Lab owns and
// rewrites whole whenever the schedules change. Nothing in the daemon runs them: snapshots
// are backup-shaped and should not stop because a web console is down, being upgraded, or
// uninstalled. The line calls `hamlab zfs-snapshot <id>` as the account Ham Lab runs as,
// which reaches zfs through sudo exactly as the daemon does.
// syncCron puts the current schedules on the machine.
func (s *Server) syncCron(ctx context.Context, z zfsState) error {
exe, err := os.Executable()
if err != nil {
return err
}
exe, _ = filepath.EvalSymlinks(exe)
return s.storage.WriteCronFile(ctx, z.Policies, s.user, exe, s.cfg.Path())
}
// ---- endpoints ----
func (s *Server) zfsOverview(w http.ResponseWriter, r *http.Request) {
z := s.zfsLoad()
now := time.Now()
dir := s.cfg.DataDir()
out := make([]map[string]any, 0, len(z.Policies))
for i := range z.Policies {
p := z.Policies[i]
expr, _ := p.CronExpr()
m := map[string]any{"policy": p, "cron": expr, "last": storage.ReadRun(dir, p.ID)}
if p.Enabled {
if n := p.NextRun(now); !n.IsZero() {
m["next"] = n
}
}
out = append(out, m)
}
jsonOK(w, 200, map[string]any{
"zfs": s.storage.ZFS(r.Context()), "policies": out, "external": s.externalSnapshotJob(r.Context()),
})
}
func (s *Server) zfsPolicyPut(w http.ResponseWriter, r *http.Request) {
var p storage.Policy
if err := json.NewDecoder(io.LimitReader(r.Body, 16<<10)).Decode(&p); err != nil {
jsonErr(w, 400, "bad request")
return
}
if err := p.Validate(); err != nil {
jsonErr(w, 400, err.Error())
return
}
z := s.zfsLoad()
found := false
for i := range z.Policies {
if z.Policies[i].ID == p.ID && p.ID != "" {
z.Policies[i] = p
found = true
}
}
if !found {
if p.ID == "" {
id, err := randomID()
if err != nil {
jsonErr(w, 500, err.Error())
return
}
p.ID = id
}
z.Policies = append(z.Policies, p)
}
if err := s.zfsSave(z); err != nil {
jsonErr(w, 500, err.Error())
return
}
if err := s.syncCron(r.Context(), z); err != nil {
jsonErr(w, 500, "saved, but writing "+storage.CronFile+" failed: "+err.Error())
return
}
expr, _ := p.CronExpr()
log.Printf("zfs schedule %q saved by %s (%s, %q, keep %s)", p.Name, clientIP(r), p.Dataset, expr, p.KeepFor)
jsonOK(w, 200, p)
}
func (s *Server) zfsPolicyDelete(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
z := s.zfsLoad()
kept := z.Policies[:0]
found := false
for _, p := range z.Policies {
if p.ID == id {
found = true
continue
}
kept = append(kept, p)
}
if !found {
jsonErr(w, 404, "no such schedule")
return
}
z.Policies = kept
if err := s.zfsSave(z); err != nil {
jsonErr(w, 500, err.Error())
return
}
if err := s.syncCron(r.Context(), z); err != nil {
jsonErr(w, 500, "removed, but writing "+storage.CronFile+" failed: "+err.Error())
return
}
storage.ForgetRun(s.cfg.DataDir(), id)
log.Printf("zfs schedule %s removed by %s (its snapshots are left alone)", id, clientIP(r))
w.WriteHeader(204)
}
func (s *Server) zfsPolicy(id string) (storage.Policy, bool) {
for _, p := range s.zfsLoad().Policies {
if p.ID == id {
return p, true
}
}
return storage.Policy{}, false
}
func (s *Server) zfsRunNow(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
p, ok := s.zfsPolicy(id)
if !ok {
jsonErr(w, 404, "no such schedule")
return
}
res, err := s.storage.Run(r.Context(), &p)
rec := storage.RunRecord{At: time.Now(), Taken: res.Taken, Pruned: res.Pruned}
if err != nil {
rec.Error = err.Error()
}
_ = storage.WriteRun(s.cfg.DataDir(), id, rec)
if err != nil {
jsonErr(w, 500, err.Error())
return
}
log.Printf("zfs schedule %q run by %s: took %s, expired %d", p.Name, clientIP(r), res.Taken, len(res.Pruned))
jsonOK(w, 200, res)
}
// zfsSnapshotDelete removes one snapshot, named in full and typed back by the caller.
func (s *Server) zfsSnapshotDelete(w http.ResponseWriter, r *http.Request) {
var body struct {
Name string `json:"name"`
Confirm string `json:"confirm"`
Recursive bool `json:"recursive"`
}
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
if body.Confirm != body.Name {
jsonErr(w, 400, "type the snapshot's full name to confirm")
return
}
if err := s.storage.DestroySnapshot(r.Context(), body.Name, body.Recursive); err != nil {
jsonErr(w, 500, err.Error())
return
}
log.Printf("zfs snapshot %s destroyed by %s%s", body.Name, clientIP(r), map[bool]string{true: " (recursive)", false: ""}[body.Recursive])
w.WriteHeader(204)
}
internal/auth/auth.go
+287 -33
View File
@@ -3,17 +3,21 @@ package auth
import (
"crypto/rand"
"crypto/sha256"
"crypto/subtle"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"regexp"
"sort"
"strings"
"sync"
"time"
"golang.org/x/crypto/argon2"
"hamlab/internal/config"
"hamlab/internal/store"
)
@@ -59,6 +63,118 @@ func VerifyPassword(hash, pw string) bool {
return subtle.ConstantTimeCompare(got, want) == 1
}
// ---- API tokens (hub → node) ----
// NewToken returns a random bearer token and its SHA-256 hex, which is what gets stored.
func NewToken() (token, hash string, err error) {
b := make([]byte, 32)
if _, err := rand.Read(b); err != nil {
return "", "", err
}
token = base64.RawURLEncoding.EncodeToString(b)
return token, HashToken(token), nil
}
func HashToken(token string) string {
sum := sha256.Sum256([]byte(token))
return hex.EncodeToString(sum[:])
}
// TokenMatches compares a presented token against stored hashes in constant time.
func TokenMatches(token string, hashes []string) bool { return TokenIndex(token, hashes) >= 0 }
// TokenIndex is which of hashes the presented token matches, -1 for none; every hash is
// compared whatever the answer.
func TokenIndex(token string, hashes []string) int {
h := []byte(HashToken(token))
idx := -1
for i, want := range hashes {
if subtle.ConstantTimeCompare(h, []byte(want)) == 1 {
idx = i
}
}
return idx
}
// ---- users ----
// UserNameRe is what an account may be called: like a Unix login, lower-case.
var UserNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{0,31}$`)
const MinPasswordLen = 12
// dummyHash is verified against when the named user does not exist, so a login attempt
// takes the same time either way and the form cannot be used to list accounts.
var dummyHash = func() string {
h, err := HashPassword("not-a-real-password")
if err != nil {
return ""
}
return h
}()
// CheckUser verifies a password for one of users. A missing user — or a guest login whose
// time is up — fails in the same time as a wrong password.
func CheckUser(users []config.User, name, pw string) bool {
for _, u := range users {
if u.Name == name {
if u.Expired(time.Now()) {
break
}
return VerifyPassword(u.Hash, pw)
}
}
VerifyPassword(dummyHash, pw)
return false
}
// ---- guest logins ----
// guestAlphabet leaves out every letter and digit that can be misread in a screenshot or
// over the phone: i, l, o, 0 and 1. A guest login is read aloud and typed by hand.
const guestAlphabet = "abcdefghjkmnpqrstuvwxyz23456789"
// randomFrom returns n characters of guestAlphabet, chosen without modulo bias.
func randomFrom(n int) (string, error) {
out := make([]byte, 0, n)
buf := make([]byte, n)
limit := byte(256 - 256%len(guestAlphabet)) // the top of the last whole run of the alphabet
for len(out) < n {
if _, err := rand.Read(buf); err != nil {
return "", err
}
for _, b := range buf {
if b >= limit {
continue // would favour the first few letters; ask for another byte instead
}
out = append(out, guestAlphabet[int(b)%len(guestAlphabet)])
if len(out) == n {
break
}
}
}
return string(out), nil
}
// GuestPassword is a password for a guest login: four groups of four, ~78 bits, in an
// alphabet with nothing ambiguous in it.
func GuestPassword() (string, error) {
var groups []string
for i := 0; i < 4; i++ {
g, err := randomFrom(4)
if err != nil {
return "", err
}
groups = append(groups, g)
}
return strings.Join(groups, "-"), nil
}
// GuestSuffix is the tail of a generated guest name.
func GuestSuffix() (string, error) { return randomFrom(4) }
// ---- sessions ----
// Sessions is a token store backed by the state file. Expiry is sliding.
type Sessions struct {
st *store.Store
@@ -72,89 +188,227 @@ func NewSessions(st *store.Store) *Sessions {
func pruneExpired(state *store.State) {
now := time.Now()
for tok, exp := range state.Sessions {
if exp.Before(now) {
for tok, ses := range state.Sessions {
if ses.Expires.Before(now) {
delete(state.Sessions, tok)
}
}
}
func (s *Sessions) Create() (string, error) {
b := make([]byte, 32)
func randomHex(n int) (string, error) {
b := make([]byte, n)
if _, err := rand.Read(b); err != nil {
return "", err
}
tok := hex.EncodeToString(b)
err := s.st.Update(func(state *store.State) {
return hex.EncodeToString(b), nil
}
// Create signs user in from ip with agent, returning the cookie token.
func (s *Sessions) Create(user, ip, agent string) (string, error) {
tok, err := randomHex(32)
if err != nil {
return "", err
}
id, err := randomHex(6)
if err != nil {
return "", err
}
if len(agent) > 200 {
agent = agent[:200]
}
now := time.Now()
err = s.st.Update(func(state *store.State) {
pruneExpired(state)
state.Sessions[tok] = time.Now().Add(SessionTTL)
state.Sessions[tok] = store.Session{ID: id, User: user, Expires: now.Add(SessionTTL), Created: now, LastSeen: now, IP: ip, Agent: agent}
})
return tok, err
}
// Valid reports whether tok is live, sliding its expiry (persisted at most hourly).
func (s *Sessions) Valid(tok string) bool {
// Lookup reports whether tok is live and whose it is, sliding its expiry and noting the
// visit (persisted at most hourly, so a busy console is not writing the file all day).
func (s *Sessions) Lookup(tok, ip string) (store.Session, bool) {
if tok == "" {
return false
return store.Session{}, false
}
ok := false
extend := false
var ses store.Session
ok, touch := false, false
s.st.View(func(state *store.State) {
exp, found := state.Sessions[tok]
if found && exp.After(time.Now()) {
ok = true
extend = time.Until(exp) < SessionTTL-time.Hour
found, have := state.Sessions[tok]
if have && found.Expires.After(time.Now()) {
ses, ok = found, true
touch = time.Until(found.Expires) < SessionTTL-time.Hour || time.Since(found.LastSeen) > time.Hour
}
})
if ok && extend {
if ok && touch {
_ = s.st.Update(func(state *store.State) {
if _, found := state.Sessions[tok]; found {
state.Sessions[tok] = time.Now().Add(SessionTTL)
if x, found := state.Sessions[tok]; found {
x.Expires = time.Now().Add(SessionTTL)
x.LastSeen = time.Now()
if ip != "" {
x.IP = ip
}
state.Sessions[tok] = x
}
})
}
return ok
return ses, ok
}
func (s *Sessions) Delete(tok string) {
_ = s.st.Update(func(state *store.State) { delete(state.Sessions, tok) })
}
// Limiter is a single global backoff: after 5 failures, each further failure
// doubles a lockout starting at 30s, capped at 10 minutes. Single-user box.
// List is every live session of user, newest first.
func (s *Sessions) List(user string) []store.Session {
var out []store.Session
s.st.View(func(state *store.State) {
now := time.Now()
for _, ses := range state.Sessions {
if ses.User == user && ses.Expires.After(now) {
out = append(out, ses)
}
}
})
sort.Slice(out, func(i, j int) bool { return out[i].LastSeen.After(out[j].LastSeen) })
return out
}
// Count is how many live sessions each user has.
func (s *Sessions) Count() map[string]int {
n := map[string]int{}
s.st.View(func(state *store.State) {
now := time.Now()
for _, ses := range state.Sessions {
if ses.Expires.After(now) {
n[ses.User]++
}
}
})
return n
}
// Revoke ends the session of user with this ID. Reports whether there was one.
func (s *Sessions) Revoke(user, id string) bool {
found := false
_ = s.st.Update(func(state *store.State) {
for tok, ses := range state.Sessions {
if ses.User == user && ses.ID == id {
delete(state.Sessions, tok)
found = true
}
}
})
return found
}
// RevokeOthers ends every session of user except the one holding keep. Returns how many.
func (s *Sessions) RevokeOthers(user, keep string) int {
n := 0
_ = s.st.Update(func(state *store.State) {
for tok, ses := range state.Sessions {
if ses.User == user && tok != keep {
delete(state.Sessions, tok)
n++
}
}
})
return n
}
// RevokeUser ends every session of user (the account is going away).
func (s *Sessions) RevokeUser(user string) { s.RevokeOthers(user, "") }
// Limiter locks out a client after repeated failed logins, per client address, with a
// global backstop so a botnet can't burn CPU on argon2 indefinitely. A lockout for one
// address never blocks another. Past perClientLock failures, each further failure doubles a
// lockout starting at 30s, capped at 10 minutes.
type Limiter struct {
mu sync.Mutex
mu sync.Mutex
clients map[string]*bucket
global bucket
}
type bucket struct {
failures int
lockUntil time.Time
last time.Time
}
var ErrLocked = errors.New("too many failed logins")
func (l *Limiter) Allow() error {
const (
// perClientLock is how many failures one address gets before it is made to wait. Set high
// on purpose: a guest login is a generated password read off a message and typed by hand
// on a phone, and being locked out for fumbling it is worse than the brute-force risk —
// which the doubling backoff past this point, and the global backstop, still cover.
perClientLock = 20
globalLock = 100 // failures (all clients) in the window before everyone waits a minute
globalWindow = 5 * time.Minute
)
func (l *Limiter) get(key string) *bucket {
if l.clients == nil {
l.clients = map[string]*bucket{}
}
b, ok := l.clients[key]
if !ok {
b = &bucket{}
l.clients[key] = b
}
// forget clients that have been quiet for an hour
if len(l.clients) > 256 {
for k, x := range l.clients {
if time.Since(x.last) > time.Hour && time.Now().After(x.lockUntil) {
delete(l.clients, k)
}
}
}
return b
}
func (l *Limiter) Allow(key string) error {
l.mu.Lock()
defer l.mu.Unlock()
if time.Now().Before(l.lockUntil) {
now := time.Now()
if now.Before(l.global.lockUntil) {
return ErrLocked
}
if now.Sub(l.global.last) > globalWindow {
l.global.failures = 0
}
if b := l.get(key); now.Before(b.lockUntil) {
return ErrLocked
}
return nil
}
func (l *Limiter) Fail() {
func (l *Limiter) Fail(key string) {
l.mu.Lock()
defer l.mu.Unlock()
l.failures++
if l.failures >= 5 {
d := 30 * time.Second << uint(l.failures-5)
now := time.Now()
b := l.get(key)
b.failures++
b.last = now
if b.failures >= perClientLock {
d := 30 * time.Second << uint(b.failures-perClientLock)
if d > 10*time.Minute {
d = 10 * time.Minute
}
l.lockUntil = time.Now().Add(d)
b.lockUntil = now.Add(d)
}
if now.Sub(l.global.last) > globalWindow {
l.global.failures = 0
}
l.global.failures++
l.global.last = now
if l.global.failures >= globalLock {
l.global.lockUntil = now.Add(time.Minute)
l.global.failures = 0
}
}
func (l *Limiter) Reset() {
func (l *Limiter) Reset(key string) {
l.mu.Lock()
defer l.mu.Unlock()
l.failures = 0
l.lockUntil = time.Time{}
delete(l.clients, key)
}
internal/compose/compose.go
+50
View File
@@ -0,0 +1,50 @@
// Package compose runs docker compose the way a stack was deployed — which is not always
// simply "on this host". A stack deployed by an orchestrator running inside its own
// container records paths in that container's filesystem, so compose has to run in there;
// a stack deployed to another machine over tcp needs DOCKER_HOST pointed at it. Both the
// console and the scheduled updates it writes must invoke it identically, or a redeploy
// from one would not match the other's labels.
package compose
import (
"context"
"os"
"os/exec"
"path/filepath"
)
// Spec is everything needed to run compose for one stack.
type Spec struct {
Project string // -p
Files []string // -f, in order
Via string // a container to run compose inside; "" is this host
Remote string // a DOCKER_HOST; "" is this machine's Docker
Tool string // "docker" or "podman"
}
// Cmd builds the command. args are compose's own, e.g. "pull" or "up", "-d".
func Cmd(ctx context.Context, s Spec, args ...string) *exec.Cmd {
tool := s.Tool
if tool == "" {
tool = "docker"
}
base := []string{"compose", "-p", s.Project}
for _, f := range s.Files {
base = append(base, "-f", f)
}
if s.Via != "" {
pre := []string{"exec"}
if s.Remote != "" {
pre = append(pre, "-e", "DOCKER_HOST="+s.Remote)
}
return exec.CommandContext(ctx, tool, append(append(append(pre, s.Via, "docker"), base...), args...)...)
}
cmd := exec.CommandContext(ctx, tool, append(base, args...)...)
if len(s.Files) > 0 {
cmd.Dir = filepath.Dir(s.Files[0])
}
if s.Remote != "" {
cmd.Env = append(os.Environ(), "DOCKER_HOST="+s.Remote)
}
return cmd
}
internal/config/config.go
+171 -7
View File
@@ -7,13 +7,127 @@ import (
"fmt"
"os"
"path/filepath"
"sync"
"time"
"hamlab/internal/store"
)
// Token is an API token another Ham Lab (a hub) uses to manage this host. Only the
// SHA-256 of the token is kept.
type Token struct {
Name string `json:"name"`
Hash string `json:"hash"`
Created time.Time `json:"created"`
// Scope is what the token may do. "" is everything (a hub's token: full control of
// this machine). ScopeSensorsRead only reads sensors — what a dashboard gets.
Scope string `json:"scope,omitempty"`
// Sensors narrows a sensors:read token to the IDs these patterns match. Empty is all.
Sensors []string `json:"sensors,omitempty"`
// IssuedBy is the account that made the token (or paired the hub that holds it).
IssuedBy string `json:"issued_by,omitempty"`
}
const ScopeSensorsRead = "sensors:read"
// Node is a remote Ham Lab this host (as hub) manages. The token is the one that
// node issued to us; the fingerprint pins its self-signed certificate.
type Node struct {
Name string `json:"name"`
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Token string `json:"token"`
Added time.Time `json:"added"`
}
// User is an account on this machine's console. Only the argon2id hash is kept.
type User struct {
Name string `json:"name"`
Hash string `json:"hash"`
Created time.Time `json:"created"`
LastSeen time.Time `json:"last_seen,omitempty"`
// Guest marks a temporary login made to show somebody the console: it may only read,
// it expires by itself at Expires, and it is not one of the machine's accounts — it is
// left out of the account list, does not count towards "the last account", and cannot
// be the one an empty user name means. See internal/api/guests.go.
Guest bool `json:"guest,omitempty"`
// Expires is when a guest login stops working. A guest without one has already
// expired, which is the safe way to read a half-written file.
Expires time.Time `json:"expires,omitempty"`
// IssuedBy is the account that made this guest login.
IssuedBy string `json:"issued_by,omitempty"`
}
// Expired reports whether a guest login's time is up. A real account never expires; a
// guest with no expiry at all counts as expired rather than as forever.
func (u User) Expired(now time.Time) bool {
return u.Guest && (u.Expires.IsZero() || now.After(u.Expires))
}
// UpdateSource is where this machine looks for a newer Ham Lab, and how often. The field
// is Updates, not Update, because Config already has an Update method.
type UpdateSource struct {
// URL is a Gitea/Forgejo repo API base or a static directory; empty means the built-in
// default, and "off" means never look.
URL string `json:"url,omitempty"`
// Token is sent when the source is behind a login, so a private Gitea can serve its
// own fleet without the instance being opened to everyone.
Token string `json:"token,omitempty"`
// Check is "daily" (the default) or "never".
Check string `json:"check,omitempty"`
}
// KVM is an IP-KVM: a device with a web page that gives screen, keyboard and mouse access
// to a machine, including when that machine has no OS running. Ham Lab holds the list on
// the hub and proxies each one, because a KVM matters exactly when the machine it is
// attached to cannot answer for itself.
type KVM struct {
ID string `json:"id"`
Name string `json:"name"`
// URL is the device's own address, as you would type it into a browser on the LAN.
URL string `json:"url"`
// Machine is the host this KVM is attached to, and is the key: one KVM per machine, shown
// on that machine's KVM tab. Empty means the hub itself. The list lives on the hub even
// for a node's KVM, because a KVM earns its keep when the machine is down and cannot
// answer for itself.
Machine string `json:"machine"`
Added time.Time `json:"added"`
}
type Config struct {
Listen string `json:"listen"`
PasswordHash string `json:"password_hash"`
Listen string `json:"listen"`
// PasswordHash is the single login of installs before users existed. On load it becomes
// the first user (see MigrateUsers) and is then dropped from the file.
PasswordHash string `json:"password_hash,omitempty"`
Users []User `json:"users,omitempty"`
TLS string `json:"tls,omitempty"` // "auto" → self-signed cert in the data dir; "" → plain HTTP
Tokens []Token `json:"tokens,omitempty"`
Nodes []Node `json:"nodes,omitempty"`
KVMs []KVM `json:"kvms,omitempty"`
// Updates: where a newer build comes from. See UpdateSource.
Updates UpdateSource `json:"update,omitempty"`
path string
mu sync.Mutex
}
// Candidates lists where config.json may live under a home directory, preferred first:
// the XDG location, then ~/apps/hamlab, which installs before v0.2 used.
func Candidates(home string) []string {
return []string{filepath.Join(home, ".config", "hamlab", "config.json"), filepath.Join(home, "apps", "hamlab", "config.json")}
}
// DefaultPath is the config used when neither -config nor $HAMLAB_CONFIG says otherwise:
// whichever candidate exists, else the preferred one.
func DefaultPath() string {
home, _ := os.UserHomeDir()
c := Candidates(home)
for _, p := range c {
if _, err := os.Stat(p); err == nil {
return p
}
}
return c[0]
}
func Default(path string) *Config {
@@ -36,10 +150,49 @@ func Load(path string) (*Config, error) {
return c, nil
}
// MigrateUsers turns the pre-users single password into the first account, named after
// the OS user the console runs as. Reports whether anything changed (the caller saves).
func (c *Config) MigrateUsers(name string) bool {
if len(c.Users) > 0 || c.PasswordHash == "" {
return false
}
c.Users = []User{{Name: name, Hash: c.PasswordHash, Created: time.Now()}}
c.PasswordHash = ""
return true
}
// Accounts is every real account: the machine's users, without the guest logins.
// Call under View/Update or on a config nobody else holds.
func (c *Config) Accounts() []User {
out := make([]User, 0, len(c.Users))
for _, u := range c.Users {
if !u.Guest {
out = append(out, u)
}
}
return out
}
// User finds an account by name. Call under View/Update or on a config nobody else holds.
func (c *Config) User(name string) (User, bool) {
for _, u := range c.Users {
if u.Name == name {
return u, true
}
}
return User{}, false
}
func (c *Config) Path() string { return c.path }
func (c *Config) DataDir() string { return filepath.Dir(c.path) }
func (c *Config) Save() error {
c.mu.Lock()
defer c.mu.Unlock()
return c.save()
}
func (c *Config) save() error {
if err := os.MkdirAll(c.DataDir(), 0o700); err != nil {
return err
}
@@ -47,9 +200,20 @@ func (c *Config) Save() error {
if err != nil {
return err
}
tmp := c.path + ".tmp"
if err := os.WriteFile(tmp, append(b, '\n'), 0o600); err != nil {
return err
}
return os.Rename(tmp, c.path)
return store.WriteFileAtomic(c.path, append(b, '\n'), 0o600)
}
// Update applies fn under the lock and saves.
func (c *Config) Update(fn func(*Config)) error {
c.mu.Lock()
defer c.mu.Unlock()
fn(c)
return c.save()
}
// View reads under the lock.
func (c *Config) View(fn func(*Config)) {
c.mu.Lock()
defer c.mu.Unlock()
fn(c)
}
internal/cron/cron.go
+813
View File
@@ -0,0 +1,813 @@
// Package cron reads and writes what the UI calls the **Jobs** tab (named "Cron" until
// 2026-09-24; the package and its /api/cron endpoints kept the old name).
//
// Package cron reads every scheduled job on the host into one list: /etc/crontab,
// /etc/cron.d/*, per-user crontabs, the run-parts directories they drive, and systemd
// timers. It computes next runs from the cron expressions, pulls last runs from the
// journal (cron logs "(user) CMD (command)" via syslog), and can toggle, edit and run
// jobs. Reads/writes that need root go through passwordless sudo when available.
package cron
import (
"bufio"
"bytes"
"context"
"crypto/sha1"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"os/user"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
"hamlab/internal/priv"
)
type Job struct {
ID string `json:"id"`
Source string `json:"source"` // "/etc/crontab", "/etc/cron.d/x", "crontab:user", "/etc/cron.daily"
Line int `json:"line"` // 1-based line in Source (0 for run-parts scripts)
User string `json:"user"`
Schedule string `json:"schedule"`
Command string `json:"command"`
Enabled bool `json:"enabled"`
Next int64 `json:"next,omitempty"` // unix ms
Last int64 `json:"last,omitempty"`
OutFile string `json:"out_file,omitempty"` // parsed from a ">> file" redirect
Note string `json:"note,omitempty"`
Editable bool `json:"editable"`
raw string
}
type Timer struct {
Unit string `json:"unit"`
Activates string `json:"activates"`
Description string `json:"description"`
Active string `json:"active"`
Sub string `json:"sub"`
Enabled string `json:"enabled"`
Next int64 `json:"next,omitempty"`
Last int64 `json:"last,omitempty"`
}
type Overview struct {
Jobs []Job `json:"jobs"`
Timers []Timer `json:"timers"`
Logging bool `json:"logging"` // cron lines seen in the journal
Anacron bool `json:"anacron"`
Sudo bool `json:"sudo"`
}
type Run struct {
ID string `json:"id"`
JobID string `json:"job_id"`
Command string `json:"command"`
User string `json:"user"`
Started time.Time `json:"started"`
Finished time.Time `json:"finished,omitempty"`
Running bool `json:"running"`
ExitCode int `json:"exit_code"`
Output string `json:"output"`
mu sync.Mutex
buf bytes.Buffer
}
type Manager struct {
sudo bool
journalSudo bool
self string
mu sync.Mutex
lastRuns map[string]int64 // user\x00command -> unix ms
lastAt time.Time
lastCursor string // journal cursor of the newest entry folded into lastRuns
logging bool
runs map[string]*Run
}
func New(sudo, journalSudo bool) *Manager {
m := &Manager{sudo: sudo, journalSudo: journalSudo, runs: map[string]*Run{}}
if u, err := user.Current(); err == nil {
m.self = u.Username
}
return m
}
// ---- helpers ----
func run(ctx context.Context, stdin string, args ...string) (string, error) {
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
if stdin != "" {
cmd.Stdin = strings.NewReader(stdin)
}
var out, errb bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errb
if err := cmd.Run(); err != nil {
msg := strings.TrimSpace(errb.String())
if msg == "" {
msg = err.Error()
}
return out.String(), errors.New(msg)
}
return out.String(), nil
}
// runLines streams a command's stdout to fn line by line, so multi-megabyte journal
// dumps never sit in memory whole.
func runLines(ctx context.Context, fn func(line []byte), args ...string) error {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
var errb bytes.Buffer
cmd.Stderr = &errb
stdout, err := cmd.StdoutPipe()
if err != nil {
return err
}
if err := cmd.Start(); err != nil {
return err
}
sc := bufio.NewScanner(stdout)
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
for sc.Scan() {
fn(sc.Bytes())
}
if err := cmd.Wait(); err != nil {
if msg := strings.TrimSpace(errb.String()); msg != "" {
return errors.New(msg)
}
return err
}
return nil
}
func (m *Manager) sudoArgs(args ...string) []string {
if m.sudo {
return priv.Wrap(args...)
}
return args
}
func jobID(source, raw string) string {
h := sha1.Sum([]byte(source + "\n" + raw))
return hex.EncodeToString(h[:6])
}
var userRe = regexp.MustCompile(`^[a-z_][a-z0-9_-]{0,31}$`)
var cronDName = regexp.MustCompile(`^[A-Za-z0-9_-]+$`) // cron ignores other names in /etc/cron.d
// ---- parsing ----
// parseLine splits one crontab line. withUser is true for system crontabs (6th field is
// the user). Returns ok=false for comments, blanks, env assignments and garbage.
func parseLine(line string, withUser bool) (schedule, usr, command string, ok bool) {
t := strings.TrimSpace(line)
if t == "" || strings.HasPrefix(t, "#") {
return
}
if i := strings.Index(t, "="); i > 0 && !strings.ContainsAny(t[:i], " \t*") { // NAME=value
return
}
var fields []string
if strings.HasPrefix(t, "@") {
f := strings.Fields(t)
if len(f) < 2 {
return
}
fields = []string{f[0]}
t = strings.TrimSpace(t[len(f[0]):])
} else {
f := strings.Fields(t)
if len(f) < 6 {
return
}
fields = f[:5]
for _, x := range fields {
t = strings.TrimSpace(t[len(x):])
}
}
schedule = strings.Join(fields, " ")
if _, err := parseSchedule(schedule); err != nil {
return
}
if withUser {
f := strings.Fields(t)
if len(f) < 2 || !userRe.MatchString(f[0]) {
return
}
usr = f[0]
t = strings.TrimSpace(t[len(f[0]):])
}
if t == "" {
return
}
return schedule, usr, t, true
}
var redirectRe = regexp.MustCompile(`(?:^|\s)(?:>>|>|2>&1\s*>>|&>>|&>)\s*("?)([^\s"&|;]+)`)
func outFile(cmd string) string {
for _, m := range redirectRe.FindAllStringSubmatch(cmd, -1) {
if p := m[2]; strings.HasPrefix(p, "/") && p != "/dev/null" {
return p
}
}
return ""
}
// parseSource turns crontab text into jobs. Commented-out lines that still parse as a
// job are reported disabled, so a "# 0 5 * * * ..." example shows up as such.
func (m *Manager) parseSource(source, text string, withUser, editable bool, defaultUser string, now time.Time, anacron bool) []Job {
var jobs []Job
lines := strings.Split(text, "\n")
isComment := func(i int) bool {
return i >= 0 && i < len(lines) && strings.HasPrefix(strings.TrimSpace(lines[i]), "#")
}
for i, line := range lines {
enabled := true
probe := line
if t := strings.TrimSpace(line); strings.HasPrefix(t, "#") {
// A commented-out job sits on its own; a job-shaped line inside a block of
// comments is documentation (Debian's stock header has two such examples).
if isComment(i-1) && isComment(i+1) {
continue
}
enabled = false
probe = strings.TrimLeft(t, "# ")
}
schedule, usr, command, ok := parseLine(probe, withUser)
if !ok {
continue
}
if usr == "" {
usr = defaultUser
}
if !enabled {
if _, err := user.Lookup(usr); err != nil {
continue // "user-name" placeholder and the like
}
}
j := Job{ID: jobID(source, line), Source: source, Line: i + 1, User: usr, Schedule: schedule, Command: command,
Enabled: enabled, OutFile: outFile(command), Editable: editable, raw: line}
if enabled {
if next, ok := nextRun(schedule, now); ok {
j.Next = next.UnixMilli()
}
}
if strings.Contains(command, "run-parts") {
if dir := runPartsDir(command); dir != "" {
if anacron && strings.Contains(command, "anacron") {
j.Note = "skipped: anacron is installed and runs " + dir + " instead"
}
jobs = append(jobs, j)
jobs = append(jobs, m.runPartsJobs(dir, j)...)
continue
}
}
jobs = append(jobs, j)
}
return jobs
}
var runPartsRe = regexp.MustCompile(`run-parts\s+(?:--\S+\s+)*(/\S+)`)
func runPartsDir(cmd string) string {
if m := runPartsRe.FindStringSubmatch(cmd); m != nil {
return m[1]
}
return ""
}
// runPartsJobs lists the executable scripts a run-parts line will run, as read-only jobs.
func (m *Manager) runPartsJobs(dir string, parent Job) []Job {
ents, err := os.ReadDir(dir)
if err != nil {
return nil
}
var out []Job
for _, e := range ents {
if e.IsDir() || !cronDName.MatchString(e.Name()) {
continue
}
info, err := e.Info()
if err != nil || info.Mode()&0o111 == 0 {
continue
}
p := filepath.Join(dir, e.Name())
out = append(out, Job{ID: jobID(dir, e.Name()), Source: dir, User: parent.User, Schedule: parent.Schedule, Command: p,
Enabled: parent.Enabled, Next: parent.Next, Note: "run by " + parent.Source + " via run-parts", raw: e.Name()})
}
return out
}
// ---- sources ----
type source struct {
name string // job Source
withUser bool
user string // default user for user crontabs
editable bool
}
func (m *Manager) readSource(ctx context.Context, s source) (string, error) {
if strings.HasPrefix(s.name, "crontab:") {
u := s.user
if u == m.self {
out, err := run(ctx, "", "crontab", "-l")
if err != nil && strings.Contains(err.Error(), "no crontab") {
return "", nil
}
return out, err
}
if !m.sudo {
return "", errors.New("needs sudo")
}
out, err := run(ctx, "", priv.Wrap("crontab", "-u", u, "-l")...)
if err != nil && strings.Contains(err.Error(), "no crontab") {
return "", nil
}
return out, err
}
b, err := os.ReadFile(s.name)
if err != nil && m.sudo {
out, err2 := run(ctx, "", priv.Wrap("cat", s.name)...)
return out, err2
}
return string(b), err
}
func (m *Manager) sources(ctx context.Context) []source {
var out []source
if _, err := os.Stat("/etc/crontab"); err == nil {
out = append(out, source{name: "/etc/crontab", withUser: true, editable: m.sudo})
}
if ents, err := os.ReadDir("/etc/cron.d"); err == nil {
for _, e := range ents {
if !e.IsDir() && cronDName.MatchString(e.Name()) {
out = append(out, source{name: "/etc/cron.d/" + e.Name(), withUser: true, editable: m.sudo})
}
}
}
// per-user crontabs: Debian keeps them in /var/spool/cron/crontabs, cronie in /var/spool/cron
users := map[string]bool{}
if m.sudo {
for _, dir := range []string{"/var/spool/cron/crontabs", "/var/spool/cron"} {
if out, err := run(ctx, "", priv.Wrap("ls", "-1", dir)...); err == nil {
for _, n := range strings.Split(out, "\n") {
if n = strings.TrimSpace(n); userRe.MatchString(n) {
users[n] = true
}
}
break
}
}
}
if m.self != "" {
users[m.self] = true
}
names := make([]string, 0, len(users))
for u := range users {
names = append(names, u)
}
sort.Strings(names)
for _, u := range names {
out = append(out, source{name: "crontab:" + u, user: u, editable: u == m.self || m.sudo})
}
return out
}
func (m *Manager) findSource(ctx context.Context, name string) (source, bool) {
for _, s := range m.sources(ctx) {
if s.name == name {
return s, true
}
}
return source{}, false
}
// ---- overview ----
func (m *Manager) Overview(ctx context.Context) Overview {
now := time.Now()
_, anacronErr := os.Stat("/usr/sbin/anacron")
anacron := anacronErr == nil
ov := Overview{Jobs: []Job{}, Timers: []Timer{}, Anacron: anacron, Sudo: m.sudo}
for _, s := range m.sources(ctx) {
text, err := m.readSource(ctx, s)
if err != nil {
continue
}
ov.Jobs = append(ov.Jobs, m.parseSource(s.name, text, s.withUser, s.editable, s.user, now, anacron)...)
}
last := m.lastRunsFromJournal(ctx)
ov.Logging = m.logging
for i := range ov.Jobs {
j := &ov.Jobs[i]
if t, ok := last[j.User+"\x00"+j.Command]; ok {
j.Last = t
}
}
ov.Timers = m.timers(ctx)
return ov
}
var cronLogRe = regexp.MustCompile(`^\((\S+)\) CMD \((.*)\)$`)
// lastRunsFromJournal maps user+command to the most recent "(user) CMD (command)" line.
// The first call scans 30 days; later calls only read entries after the last cursor, so
// the cost per refresh is proportional to new cron activity, not to history.
func (m *Manager) lastRunsFromJournal(ctx context.Context) map[string]int64 {
m.mu.Lock()
if time.Since(m.lastAt) < 2*time.Second && m.lastRuns != nil {
r := m.lastRuns
m.mu.Unlock()
return r
}
cursor := m.lastCursor
res := map[string]int64{}
for k, v := range m.lastRuns {
res[k] = v
}
m.mu.Unlock()
args := []string{"journalctl", "-t", "CRON", "-t", "cron", "-t", "crond", "-o", "json", "--no-pager", "-q"}
if cursor != "" {
args = append(args, "--after-cursor="+cursor)
} else {
args = append(args, "--since", "-30d", "-n", "20000")
}
if m.journalSudo {
args = priv.Wrap(args...)
}
seen := m.logging
_ = runLines(ctx, func(line []byte) {
var raw struct {
Msg any `json:"MESSAGE"`
TS string `json:"__REALTIME_TIMESTAMP"`
Cursor string `json:"__CURSOR"`
}
if json.Unmarshal(line, &raw) != nil {
return
}
seen = true
if raw.Cursor != "" {
cursor = raw.Cursor
}
msg, _ := raw.Msg.(string)
mm := cronLogRe.FindStringSubmatch(msg)
if mm == nil {
return
}
us, _ := strconv.ParseInt(raw.TS, 10, 64)
k := mm[1] + "\x00" + mm[2]
if us/1000 > res[k] {
res[k] = us / 1000
}
}, args...)
m.mu.Lock()
m.lastRuns, m.lastAt, m.lastCursor, m.logging = res, time.Now(), cursor, seen
m.mu.Unlock()
return res
}
// JobLogs returns the journal lines for one job's command (most recent last).
func (m *Manager) JobLogs(ctx context.Context, j Job, n int) ([]map[string]any, error) {
args := []string{"journalctl", "-t", "CRON", "-t", "cron", "-t", "crond", "-o", "json", "--no-pager", "-q", "-n", "20000"}
if m.journalSudo {
args = priv.Wrap(args...)
}
var res []map[string]any
needle := "(" + j.User + ") CMD (" + j.Command + ")"
err := runLines(ctx, func(line []byte) {
if !bytes.Contains(line, []byte(j.Command[:min(len(j.Command), 40)])) { // cheap pre-filter before JSON
return
}
var raw struct {
Msg any `json:"MESSAGE"`
TS string `json:"__REALTIME_TIMESTAMP"`
PID string `json:"_PID"`
}
if json.Unmarshal(line, &raw) != nil {
return
}
msg, _ := raw.Msg.(string)
if !strings.Contains(msg, needle) {
return
}
us, _ := strconv.ParseInt(raw.TS, 10, 64)
res = append(res, map[string]any{"t": us / 1000, "p": 6, "ident": "CRON", "pid": raw.PID, "msg": msg})
}, args...)
if err != nil {
return nil, err
}
if len(res) > n {
res = res[len(res)-n:]
}
if res == nil {
res = []map[string]any{}
}
return res, nil
}
// OutputTail returns the last n lines of the job's redirect target.
func (m *Manager) OutputTail(ctx context.Context, j Job, n int) (string, error) {
if j.OutFile == "" {
return "", errors.New("job has no output file")
}
out, err := run(ctx, "", m.sudoArgs("tail", "-n", strconv.Itoa(n), j.OutFile)...)
return out, err
}
// ---- timers ----
// timers: one `systemctl show` over every loaded timer instead of three list commands.
func (m *Manager) timers(ctx context.Context) []Timer {
out, err := run(ctx, "", "systemctl", "show", "*.timer", "--no-pager",
"-p", "Id,Description,ActiveState,SubState,UnitFileState,NextElapseUSecRealtime,LastTriggerUSec,Triggers")
if err != nil {
return []Timer{}
}
unix := func(v string) int64 { // "Sat 2026-09-19 05:47:13 PDT", "n/a" or ""
for _, layout := range []string{"Mon 2006-01-02 15:04:05 MST", "2006-01-02 15:04:05 MST", "Mon 2006-01-02 15:04:05"} {
if t, err := time.ParseInLocation(layout, v, time.Local); err == nil {
return t.UnixMilli()
}
}
return 0
}
res := []Timer{}
for _, block := range strings.Split(out, "\n\n") {
t := Timer{}
for _, line := range strings.Split(block, "\n") {
k, v, ok := strings.Cut(line, "=")
if !ok {
continue
}
switch k {
case "Id":
t.Unit = v
case "Description":
t.Description = v
case "ActiveState":
t.Active = v
case "SubState":
t.Sub = v
case "UnitFileState":
t.Enabled = v
case "NextElapseUSecRealtime":
t.Next = unix(v)
case "LastTriggerUSec":
t.Last = unix(v)
case "Triggers":
t.Activates = strings.Fields(v + " ")[0:1][0]
}
}
if t.Unit != "" {
res = append(res, t)
}
}
sort.Slice(res, func(i, j int) bool { return res[i].Unit < res[j].Unit })
return res
}
var timerRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9@._:\\-]{0,255}\.timer$`)
func (m *Manager) TimerAction(ctx context.Context, unit, action string) error {
if !timerRe.MatchString(unit) {
return errors.New("invalid timer name")
}
switch action {
case "start", "stop", "enable", "disable":
_, err := run(ctx, "", m.sudoArgs("systemctl", action, unit)...)
return err
case "run": // fire the service the timer activates, right now
out, err := run(ctx, "", "systemctl", "show", unit, "-p", "Triggers", "--value")
if err != nil {
return err
}
svc := strings.TrimSpace(strings.Split(out, " ")[0])
if svc == "" {
return errors.New("timer triggers nothing")
}
_, err = run(ctx, "", m.sudoArgs("systemctl", "start", svc)...)
return err
}
return errors.New("unknown action")
}
// ---- job lookup and editing ----
func (m *Manager) FindJob(ctx context.Context, id string) (Job, bool) {
for _, j := range m.Overview(ctx).Jobs {
if j.ID == id {
return j, true
}
}
return Job{}, false
}
// Source returns a crontab's raw text for the editor.
func (m *Manager) Source(ctx context.Context, name string) (text string, editable bool, err error) {
s, ok := m.findSource(ctx, name)
if !ok {
return "", false, errors.New("unknown source")
}
text, err = m.readSource(ctx, s)
return text, s.editable, err
}
// validate rejects any line that is neither blank, comment, env assignment, nor a job.
func validate(text string, withUser bool) error {
for i, line := range strings.Split(text, "\n") {
t := strings.TrimSpace(line)
if t == "" || strings.HasPrefix(t, "#") {
continue
}
if k, _, ok := strings.Cut(t, "="); ok && !strings.ContainsAny(strings.TrimSpace(k), " \t*") {
continue
}
if _, _, _, ok := parseLine(t, withUser); !ok {
return fmt.Errorf("line %d: not a valid cron entry", i+1)
}
}
return nil
}
// WriteSource replaces a crontab. User crontabs go through crontab(1) so cron reloads
// them; system files are written in place with their permissions preserved.
func (m *Manager) WriteSource(ctx context.Context, name, text string) error {
s, ok := m.findSource(ctx, name)
if !ok {
return errors.New("unknown source")
}
if !s.editable {
return errors.New("source is read-only here (no sudo)")
}
if !strings.HasSuffix(text, "\n") {
text += "\n"
}
if err := validate(text, s.withUser); err != nil {
return err
}
if strings.HasPrefix(name, "crontab:") {
if s.user == m.self {
_, err := run(ctx, text, "crontab", "-")
return err
}
_, err := run(ctx, text, priv.Wrap("crontab", "-u", s.user, "-")...)
return err
}
if name != "/etc/crontab" && !strings.HasPrefix(name, "/etc/cron.d/") {
return errors.New("refusing to write outside /etc/crontab and /etc/cron.d")
}
_, err := run(ctx, text, priv.Wrap("tee", name)...)
m.mu.Lock()
m.lastAt = time.Time{}
m.mu.Unlock()
return err
}
// Toggle comments or uncomments a job's line. The line must still read as the job did
// when it was listed, so a concurrent edit can't be clobbered.
func (m *Manager) Toggle(ctx context.Context, j Job, enable bool) error {
if j.Line == 0 || !j.Editable {
return errors.New("this job can't be toggled here")
}
s, ok := m.findSource(ctx, j.Source)
if !ok {
return errors.New("unknown source")
}
text, err := m.readSource(ctx, s)
if err != nil {
return err
}
lines := strings.Split(text, "\n")
if j.Line > len(lines) || lines[j.Line-1] != j.raw {
return errors.New("crontab changed underneath; reload and try again")
}
if enable {
lines[j.Line-1] = strings.TrimLeft(strings.TrimSpace(j.raw), "# ")
} else {
lines[j.Line-1] = "#" + j.raw
}
return m.WriteSource(ctx, j.Source, strings.Join(lines, "\n"))
}
// ---- run now ----
// StartRun executes the job's command as its user, the way cron would (sh -c, cwd = home,
// '%' turned into newlines/stdin), in the background. Output is kept in memory.
func (m *Manager) StartRun(ctx context.Context, j Job) (*Run, error) {
if j.User != m.self && !m.sudo {
return nil, errors.New("needs sudo to run as " + j.User)
}
cmdline, stdin := cronPercent(j.Command)
args := []string{"sh", "-c", cmdline}
if j.User != m.self {
args = priv.AsUser(j.User, args...)
}
r := &Run{ID: jobID("run", strconv.FormatInt(time.Now().UnixNano(), 10)), JobID: j.ID, Command: j.Command, User: j.User, Started: time.Now(), Running: true}
runCtx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
cmd := exec.CommandContext(runCtx, args[0], args[1:]...)
if stdin != "" {
cmd.Stdin = strings.NewReader(stdin)
}
w := &capWriter{r: r}
cmd.Stdout, cmd.Stderr = w, w
if err := cmd.Start(); err != nil {
cancel()
return nil, err
}
m.mu.Lock()
m.runs[r.ID] = r
for id, old := range m.runs { // keep the map small
if !old.Running && time.Since(old.Finished) > time.Hour {
delete(m.runs, id)
}
}
m.mu.Unlock()
go func() {
defer cancel()
err := cmd.Wait()
r.mu.Lock()
r.Running, r.Finished = false, time.Now()
if err != nil {
if ee, ok := err.(*exec.ExitError); ok {
r.ExitCode = ee.ExitCode()
} else {
r.ExitCode = -1
}
}
r.mu.Unlock()
m.mu.Lock()
m.lastAt = time.Time{}
m.mu.Unlock()
}()
return r, nil
}
func (m *Manager) GetRun(id string) (*Run, bool) {
m.mu.Lock()
r, ok := m.runs[id]
m.mu.Unlock()
return r, ok
}
// Snapshot returns a copy safe to encode.
func (r *Run) Snapshot() Run {
r.mu.Lock()
defer r.mu.Unlock()
return Run{ID: r.ID, JobID: r.JobID, Command: r.Command, User: r.User, Started: r.Started, Finished: r.Finished,
Running: r.Running, ExitCode: r.ExitCode, Output: r.buf.String()}
}
type capWriter struct{ r *Run }
func (w *capWriter) Write(p []byte) (int, error) {
w.r.mu.Lock()
defer w.r.mu.Unlock()
w.r.buf.Write(p)
if w.r.buf.Len() > 256<<10 { // keep the tail
b := w.r.buf.Bytes()
w.r.buf = *bytes.NewBuffer(append([]byte("…\n"), b[len(b)-200<<10:]...))
}
return len(p), nil
}
// cronPercent applies crontab(5) rules: an unescaped % ends the command; later ones are
// newlines fed to stdin. "\%" is a literal percent.
func cronPercent(cmd string) (string, string) {
var out, in strings.Builder
cur := &out
esc := false
for _, c := range cmd {
switch {
case esc:
if c != '%' {
cur.WriteRune('\\')
}
cur.WriteRune(c)
esc = false
case c == '\\':
esc = true
case c == '%':
if cur == &out {
cur = &in
} else {
in.WriteRune('\n')
}
default:
cur.WriteRune(c)
}
}
return out.String(), in.String()
}
internal/cron/schedule.go
+174
View File
@@ -0,0 +1,174 @@
package cron
import (
"errors"
"fmt"
"strconv"
"strings"
"time"
)
// schedule is a parsed five-field cron expression as sets.
type schedule struct {
min, hour, dom, mon, dow [64]bool
domStar, dowStar bool // "*" in dom/dow changes the match rule
reboot bool
}
var monthNames = map[string]int{"jan": 1, "feb": 2, "mar": 3, "apr": 4, "may": 5, "jun": 6, "jul": 7, "aug": 8, "sep": 9, "oct": 10, "nov": 11, "dec": 12}
var dayNames = map[string]int{"sun": 0, "mon": 1, "tue": 2, "wed": 3, "thu": 4, "fri": 5, "sat": 6}
var aliases = map[string]string{
"@yearly": "0 0 1 1 *", "@annually": "0 0 1 1 *", "@monthly": "0 0 1 * *", "@weekly": "0 0 * * 0",
"@daily": "0 0 * * *", "@midnight": "0 0 * * *", "@hourly": "0 * * * *",
}
func parseSchedule(expr string) (*schedule, error) {
expr = strings.TrimSpace(expr)
if expr == "@reboot" {
return &schedule{reboot: true}, nil
}
if a, ok := aliases[strings.ToLower(expr)]; ok {
expr = a
}
f := strings.Fields(expr)
if len(f) != 5 {
return nil, errors.New("expected 5 fields")
}
s := &schedule{}
var err error
if s.min, err = parseField(f[0], 0, 59, nil); err != nil {
return nil, fmt.Errorf("minute: %w", err)
}
if s.hour, err = parseField(f[1], 0, 23, nil); err != nil {
return nil, fmt.Errorf("hour: %w", err)
}
if s.dom, err = parseField(f[2], 1, 31, nil); err != nil {
return nil, fmt.Errorf("day of month: %w", err)
}
if s.mon, err = parseField(f[3], 1, 12, monthNames); err != nil {
return nil, fmt.Errorf("month: %w", err)
}
if s.dow, err = parseField(f[4], 0, 7, dayNames); err != nil {
return nil, fmt.Errorf("day of week: %w", err)
}
if s.dow[7] { // 7 == Sunday
s.dow[0] = true
}
s.domStar = f[2] == "*"
s.dowStar = f[4] == "*"
return s, nil
}
func parseField(field string, lo, hi int, names map[string]int) ([64]bool, error) {
var set [64]bool
atom := func(s string) (int, error) {
if names != nil {
if v, ok := names[strings.ToLower(s)]; ok {
return v, nil
}
}
v, err := strconv.Atoi(s)
if err != nil {
return 0, fmt.Errorf("bad value %q", s)
}
return v, nil
}
for _, part := range strings.Split(field, ",") {
step := 1
if i := strings.Index(part, "/"); i >= 0 {
v, err := strconv.Atoi(part[i+1:])
if err != nil || v <= 0 {
return set, fmt.Errorf("bad step in %q", part)
}
step, part = v, part[:i]
}
a, b := lo, hi
switch {
case part == "*":
case strings.Contains(part, "-"):
x, y, _ := strings.Cut(part, "-")
var err error
if a, err = atom(x); err != nil {
return set, err
}
if b, err = atom(y); err != nil {
return set, err
}
default:
v, err := atom(part)
if err != nil {
return set, err
}
a = v
if step == 1 {
b = v
}
}
if a < lo || b > hi || a > b {
return set, fmt.Errorf("out of range: %q", part)
}
for v := a; v <= b; v += step {
set[v] = true
}
}
return set, nil
}
func (s *schedule) dayMatches(t time.Time) bool {
if !s.mon[int(t.Month())] {
return false
}
dom, dow := s.dom[t.Day()], s.dow[int(t.Weekday())]
switch {
case s.domStar && s.dowStar:
return true
case s.domStar:
return dow
case s.dowStar:
return dom
default: // both restricted: cron runs when either matches
return dom || dow
}
}
// NextRun is the first time strictly after `from` that a cron expression fires. Exported so
// anything that writes a cron line — the ZFS snapshot schedules — can say when it runs next
// using the same evaluator that reads everyone else's.
func NextRun(expr string, from time.Time) (time.Time, bool) { return nextRun(expr, from) }
// ValidSchedule reports whether a cron expression parses.
func ValidSchedule(expr string) error {
_, err := parseSchedule(expr)
return err
}
// nextRun finds the first time strictly after `from` that the schedule fires.
func nextRun(expr string, from time.Time) (time.Time, bool) {
s, err := parseSchedule(expr)
if err != nil || s.reboot {
return time.Time{}, false
}
t := from.Truncate(time.Minute).Add(time.Minute)
day := time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, t.Location())
for i := 0; i < 366*2; i++ {
if s.dayMatches(day) {
for h := 0; h < 24; h++ {
if !s.hour[h] {
continue
}
for mi := 0; mi < 60; mi++ {
if !s.min[mi] {
continue
}
c := time.Date(day.Year(), day.Month(), day.Day(), h, mi, 0, 0, day.Location())
if !c.Before(t) {
return c, true
}
}
}
}
day = day.AddDate(0, 0, 1)
}
return time.Time{}, false
}
internal/cron/schedule_test.go
+68
View File
@@ -0,0 +1,68 @@
package cron
import (
"testing"
"time"
)
func TestNextRun(t *testing.T) {
loc := time.UTC
from := time.Date(2026, 9, 17, 15, 30, 0, 0, loc) // Thursday
cases := []struct {
expr string
want time.Time
}{
{"0 1 * * *", time.Date(2026, 9, 18, 1, 0, 0, 0, loc)},
{"17 * * * *", time.Date(2026, 9, 17, 16, 17, 0, 0, loc)},
{"*/15 * * * *", time.Date(2026, 9, 17, 15, 45, 0, 0, loc)},
{"47 6 * * 7", time.Date(2026, 9, 20, 6, 47, 0, 0, loc)}, // Sunday as 7
{"52 6 1 * *", time.Date(2026, 10, 1, 6, 52, 0, 0, loc)}, // first of month
{"24 0 1-7 * *", time.Date(2026, 10, 1, 0, 24, 0, 0, loc)}, // range
{"30 3 * * 0", time.Date(2026, 9, 20, 3, 30, 0, 0, loc)}, // Sunday as 0
{"0 9 15 * mon", time.Date(2026, 9, 21, 9, 0, 0, 0, loc)}, // dom OR dow
{"@daily", time.Date(2026, 9, 18, 0, 0, 0, 0, loc)},
{"@weekly", time.Date(2026, 9, 20, 0, 0, 0, 0, loc)},
{"0 0 29 feb *", time.Date(2028, 2, 29, 0, 0, 0, 0, loc)},
}
for _, c := range cases {
got, ok := nextRun(c.expr, from)
if !ok || !got.Equal(c.want) {
t.Errorf("%q: got %v ok=%v, want %v", c.expr, got, ok, c.want)
}
}
if _, ok := nextRun("@reboot", from); ok {
t.Error("@reboot should have no next run")
}
for _, bad := range []string{"* * * *", "60 * * * *", "a b c d e", "*/0 * * * *"} {
if _, err := parseSchedule(bad); err == nil {
t.Errorf("%q should fail", bad)
}
}
}
func TestParseLine(t *testing.T) {
sch, u, cmd, ok := parseLine("25 6\t* * *\troot\ttest -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }", true)
if !ok || sch != "25 6 * * *" || u != "root" || !strings_HasPrefix(cmd, "test -x") {
t.Errorf("system line: %q %q %q %v", sch, u, cmd, ok)
}
if _, _, _, ok := parseLine("PATH=/usr/bin:/bin", true); ok {
t.Error("env line parsed as job")
}
if _, _, _, ok := parseLine("# m h dom mon dow command", false); ok {
t.Error("comment parsed as job")
}
if _, u, cmd, ok := parseLine("@reboot /usr/local/bin/thing --flag", false); !ok || u != "" || cmd != "/usr/local/bin/thing --flag" {
t.Errorf("@reboot: %q %q %v", u, cmd, ok)
}
if got := outFile("foo >> /var/log/foo.log 2>&1"); got != "/var/log/foo.log" {
t.Errorf("outFile: %q", got)
}
if c, in := cronPercent("echo hi%line1%line2"); c != "echo hi" || in != "line1\nline2" {
t.Errorf("percent: %q %q", c, in)
}
if c, _ := cronPercent(`date +\%w`); c != "date +%w" {
t.Errorf("escaped percent: %q", c)
}
}
func strings_HasPrefix(s, p string) bool { return len(s) >= len(p) && s[:len(p)] == p }
internal/files/files.go
+503
View File
@@ -0,0 +1,503 @@
// Package files is the Files tab: browse, read, write, upload, download, mkdir,
// rename, chmod and delete anywhere on the host.
//
// Two backends share one interface. Local runs in-process as the daemon's user. Sudo
// re-executes this same binary as root (`sudo -n hamlab fs …`) for one operation at a
// time, streaming file bodies over its stdin/stdout, so root access uses exactly the
// same code and the same validation as the unprivileged path.
package files
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"os/user"
"path/filepath"
"sort"
"strconv"
"strings"
"syscall"
"time"
"hamlab/internal/priv"
)
type Entry struct {
Name string `json:"name"`
Type string `json:"type"` // dir | file | link | other
Size int64 `json:"size"`
Mode string `json:"mode"` // "-rw-r--r--"
Perm string `json:"perm"` // "0644"
Mtime int64 `json:"mtime"`
User string `json:"user"`
Group string `json:"group"`
Target string `json:"target,omitempty"` // link target
Broken bool `json:"broken,omitempty"` // dangling link
}
type Listing struct {
Path string `json:"path"`
Parent string `json:"parent,omitempty"`
Entries []Entry `json:"entries"`
}
type Info struct {
Entry
Path string `json:"path"`
Text bool `json:"text"` // looks like text (for the editor)
}
// FS is what the API talks to.
type FS interface {
List(ctx context.Context, path string) (Listing, error)
Stat(ctx context.Context, path string) (Info, error)
Open(ctx context.Context, path string) (io.ReadCloser, error)
Write(ctx context.Context, path string, r io.Reader) error
Mkdir(ctx context.Context, path string) error
Touch(ctx context.Context, path string) error
Rename(ctx context.Context, from, to string) error
Delete(ctx context.Context, path string, recursive bool) error
Chmod(ctx context.Context, path string, perm os.FileMode) error
}
// ---- path rules ----
// Clean rejects relative paths and normalises the rest. Everything else on the host is
// reachable on purpose: this is a root-capable console, the gate is the login.
func Clean(p string) (string, error) {
if p == "" {
return "", errors.New("path required")
}
if !strings.HasPrefix(p, "/") {
return "", errors.New("path must be absolute")
}
c := filepath.Clean(p)
if strings.ContainsRune(c, 0) {
return "", errors.New("bad path")
}
return c, nil
}
// protected are never deleted or renamed as a whole, no matter who asks.
var protected = map[string]bool{"/": true, "/bin": true, "/boot": true, "/dev": true, "/etc": true, "/home": true, "/lib": true, "/lib64": true,
"/opt": true, "/proc": true, "/root": true, "/run": true, "/sbin": true, "/srv": true, "/sys": true, "/tmp": true, "/usr": true, "/var": true}
func ParsePerm(s string) (os.FileMode, error) {
v, err := strconv.ParseUint(strings.TrimPrefix(s, "0"), 8, 32)
if err != nil || v > 0o7777 {
return 0, errors.New("mode must be octal like 0644")
}
return os.FileMode(v), nil
}
// ---- Local ----
type Local struct{}
var uidCache, gidCache = map[uint32]string{}, map[uint32]string{}
func owner(st os.FileInfo) (string, string) {
sys, ok := st.Sys().(*syscall.Stat_t)
if !ok {
return "", ""
}
u, ok := uidCache[sys.Uid]
if !ok {
u = strconv.Itoa(int(sys.Uid))
if x, err := user.LookupId(u); err == nil {
u = x.Username
}
uidCache[sys.Uid] = u
}
g, ok := gidCache[sys.Gid]
if !ok {
g = strconv.Itoa(int(sys.Gid))
if x, err := user.LookupGroupId(g); err == nil {
g = x.Name
}
gidCache[sys.Gid] = g
}
return u, g
}
func entryOf(dir, name string, lst os.FileInfo) Entry {
e := Entry{Name: name, Size: lst.Size(), Mode: lst.Mode().String(), Perm: fmt.Sprintf("%04o", lst.Mode().Perm()), Mtime: lst.ModTime().UnixMilli()}
e.User, e.Group = owner(lst)
switch {
case lst.Mode()&os.ModeSymlink != 0:
e.Type = "link"
full := filepath.Join(dir, name)
e.Target, _ = os.Readlink(full)
if st, err := os.Stat(full); err != nil {
e.Broken = true
} else if st.IsDir() {
e.Type = "dir" // keep dir semantics for navigation; Target says it's a link
} else {
e.Size = st.Size()
}
case lst.IsDir():
e.Type = "dir"
case lst.Mode().IsRegular():
e.Type = "file"
default:
e.Type = "other"
}
return e
}
func (Local) List(_ context.Context, path string) (Listing, error) {
ents, err := os.ReadDir(path)
if err != nil {
return Listing{}, err
}
l := Listing{Path: path, Entries: make([]Entry, 0, len(ents))}
if path != "/" {
l.Parent = filepath.Dir(path)
}
for _, d := range ents {
lst, err := d.Info() // Lstat
if err != nil {
continue
}
l.Entries = append(l.Entries, entryOf(path, d.Name(), lst))
}
sort.Slice(l.Entries, func(i, j int) bool {
a, b := l.Entries[i], l.Entries[j]
if (a.Type == "dir") != (b.Type == "dir") {
return a.Type == "dir"
}
return strings.ToLower(a.Name) < strings.ToLower(b.Name)
})
return l, nil
}
func (Local) Stat(_ context.Context, path string) (Info, error) {
lst, err := os.Lstat(path)
if err != nil {
return Info{}, err
}
in := Info{Entry: entryOf(filepath.Dir(path), filepath.Base(path), lst), Path: path}
if in.Type == "file" && in.Size <= 8<<20 {
if f, err := os.Open(path); err == nil {
buf := make([]byte, 4096)
n, _ := f.Read(buf)
f.Close()
in.Text = looksText(buf[:n])
}
}
return in, nil
}
func looksText(b []byte) bool {
if len(b) == 0 {
return true
}
ctrl := 0
for _, c := range b {
if c == 0 {
return false
}
if c < 32 && c != '\n' && c != '\r' && c != '\t' && c != '\f' && c != 0x1b {
ctrl++
}
}
return ctrl*20 < len(b)
}
// Open refuses anything that isn't a regular file: devices stream forever (/dev/zero),
// FIFOs block on open, sockets can't be read. /proc files are regular and still work.
func (Local) Open(_ context.Context, path string) (io.ReadCloser, error) {
st, err := os.Stat(path)
if err != nil {
return nil, err
}
if st.IsDir() {
return nil, errors.New("is a directory")
}
if !st.Mode().IsRegular() {
return nil, errors.New("not a regular file (device, pipe or socket)")
}
return os.Open(path)
}
// Write replaces the file's contents, keeping its mode; new files get 0644 (umask).
func (Local) Write(_ context.Context, path string, r io.Reader) error {
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
if err != nil {
return err
}
if _, err := io.Copy(f, r); err != nil {
f.Close()
return err
}
return f.Close()
}
func (Local) Mkdir(_ context.Context, path string) error { return os.Mkdir(path, 0o755) }
func (Local) Touch(_ context.Context, path string) error {
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil {
return err
}
return f.Close()
}
func (Local) Rename(_ context.Context, from, to string) error {
if protected[from] {
return errors.New("refusing to move " + from)
}
if _, err := os.Lstat(to); err == nil {
return errors.New("destination exists")
}
return os.Rename(from, to)
}
func (Local) Delete(_ context.Context, path string, recursive bool) error {
if protected[path] {
return errors.New("refusing to delete " + path)
}
lst, err := os.Lstat(path)
if err != nil {
return err
}
if lst.IsDir() && recursive {
return os.RemoveAll(path)
}
return os.Remove(path) // fails on a non-empty dir, by design
}
func (Local) Chmod(_ context.Context, path string, perm os.FileMode) error {
return os.Chmod(path, perm)
}
// ---- Sudo: the same operations, run as root in a child hamlab ----
type Sudo struct{ Self string }
func (s Sudo) cmd(ctx context.Context, args ...string) *exec.Cmd {
full := priv.Wrap(append([]string{s.Self, "fs"}, args...)...)
return exec.CommandContext(ctx, full[0], full[1:]...)
}
func (s Sudo) runJSON(ctx context.Context, out any, args ...string) error {
c := s.cmd(ctx, args...)
var errb strings.Builder
c.Stderr = &errb
b, err := c.Output()
if err != nil {
if msg := strings.TrimSpace(errb.String()); msg != "" {
return errors.New(msg)
}
return err
}
if out == nil {
return nil
}
return json.Unmarshal(b, out)
}
func (s Sudo) List(ctx context.Context, path string) (Listing, error) {
var l Listing
err := s.runJSON(ctx, &l, "list", path)
return l, err
}
func (s Sudo) Stat(ctx context.Context, path string) (Info, error) {
var in Info
err := s.runJSON(ctx, &in, "stat", path)
return in, err
}
type cmdReader struct {
io.ReadCloser
cmd *exec.Cmd
errb *strings.Builder
}
func (r *cmdReader) Close() error {
r.ReadCloser.Close()
err := r.cmd.Wait()
if err != nil && r.errb.Len() > 0 {
return errors.New(strings.TrimSpace(r.errb.String()))
}
return err
}
func (s Sudo) Open(ctx context.Context, path string) (io.ReadCloser, error) {
c := s.cmd(ctx, "read", path)
errb := &strings.Builder{}
c.Stderr = errb
out, err := c.StdoutPipe()
if err != nil {
return nil, err
}
if err := c.Start(); err != nil {
return nil, err
}
// Fail early on "no such file" etc.: the helper stats before writing anything.
head := make([]byte, 1)
n, rerr := out.Read(head)
if n == 0 && rerr != nil {
_ = c.Wait()
if msg := strings.TrimSpace(errb.String()); msg != "" {
return nil, errors.New(msg)
}
if rerr != io.EOF {
return nil, rerr
}
}
return &cmdReader{ReadCloser: struct {
io.Reader
io.Closer
}{io.MultiReader(strings.NewReader(string(head[:n])), out), out}, cmd: c, errb: errb}, nil
}
func (s Sudo) Write(ctx context.Context, path string, r io.Reader) error {
c := s.cmd(ctx, "write", path)
c.Stdin = r
var errb strings.Builder
c.Stderr = &errb
if err := c.Run(); err != nil {
if msg := strings.TrimSpace(errb.String()); msg != "" {
return errors.New(msg)
}
return err
}
return nil
}
func (s Sudo) Mkdir(ctx context.Context, path string) error {
return s.runJSON(ctx, nil, "mkdir", path)
}
func (s Sudo) Touch(ctx context.Context, path string) error {
return s.runJSON(ctx, nil, "touch", path)
}
func (s Sudo) Rename(ctx context.Context, from, to string) error {
return s.runJSON(ctx, nil, "rename", from, to)
}
func (s Sudo) Delete(ctx context.Context, path string, recursive bool) error {
if recursive {
return s.runJSON(ctx, nil, "delete", "-r", path)
}
return s.runJSON(ctx, nil, "delete", path)
}
func (s Sudo) Chmod(ctx context.Context, path string, perm os.FileMode) error {
return s.runJSON(ctx, nil, "chmod", fmt.Sprintf("%04o", perm), path)
}
// Helper implements `hamlab fs <op> …` for the Sudo backend. Returns the exit code.
func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer) int {
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
if len(args) < 2 {
return fail(errors.New("usage: hamlab fs <list|stat|read|write|mkdir|touch|rename|delete|chmod> …"))
}
ctx := context.Background()
var l Local
op := args[0]
clean := func(i int) (string, error) {
if i >= len(args) {
return "", errors.New("path required")
}
return Clean(args[i])
}
switch op {
case "list", "stat", "read", "write", "mkdir", "touch":
p, err := clean(1)
if err != nil {
return fail(err)
}
switch op {
case "list":
out, err := l.List(ctx, p)
if err != nil {
return fail(err)
}
return emit(stdout, out, stderr)
case "stat":
out, err := l.Stat(ctx, p)
if err != nil {
return fail(err)
}
return emit(stdout, out, stderr)
case "read":
f, err := l.Open(ctx, p)
if err != nil {
return fail(err)
}
defer f.Close()
if _, err := io.Copy(stdout, f); err != nil {
return fail(err)
}
return 0
case "write":
if err := l.Write(ctx, p, stdin); err != nil {
return fail(err)
}
case "mkdir":
if err := l.Mkdir(ctx, p); err != nil {
return fail(err)
}
case "touch":
if err := l.Touch(ctx, p); err != nil {
return fail(err)
}
}
case "rename":
from, err := clean(1)
if err != nil {
return fail(err)
}
to, err := clean(2)
if err != nil {
return fail(err)
}
if err := l.Rename(ctx, from, to); err != nil {
return fail(err)
}
case "delete":
rec := false
i := 1
if args[1] == "-r" {
rec, i = true, 2
}
p, err := clean(i)
if err != nil {
return fail(err)
}
if err := l.Delete(ctx, p, rec); err != nil {
return fail(err)
}
case "chmod":
perm, err := ParsePerm(args[1])
if err != nil {
return fail(err)
}
p, err := clean(2)
if err != nil {
return fail(err)
}
if err := l.Chmod(ctx, p, perm); err != nil {
return fail(err)
}
default:
return fail(errors.New("unknown op " + op))
}
fmt.Fprintln(stdout, "{}")
return 0
}
func emit(w io.Writer, v any, stderr io.Writer) int {
if err := json.NewEncoder(w).Encode(v); err != nil {
fmt.Fprintln(stderr, err.Error())
return 1
}
return 0
}
// ModTime helper for handlers.
func Now() time.Time { return time.Now() }
internal/hardware/hardware.go
+843
View File
@@ -0,0 +1,843 @@
// Package hardware is what is physically in the machine: the board it is, its processor,
// the memory modules in its slots, the drives attached to it and the graphics in it.
//
// None of this changes while the machine is running, so it is read once and remembered.
// Everything here is read from what Debian and its kin already have — dmidecode for the
// parts only the firmware knows, lscpu, lspci, sysfs — and every reader is allowed to
// come back empty: a machine with no dmidecode, no PCI bus or no root still gets a page,
// with a note saying what could not be read and why.
package hardware
import (
"bufio"
"context"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
"hamlab/internal/priv"
"hamlab/internal/storage"
)
type System struct {
Model string `json:"model,omitempty"` // "GMKtec NucBox G3"
Serial string `json:"serial,omitempty"`
Family string `json:"family,omitempty"`
Kernel string `json:"kernel,omitempty"`
Arch string `json:"arch,omitempty"`
Virt string `json:"virt,omitempty"` // "kvm", "lxc", … when this is not real hardware
}
// Slot is an expansion slot as the firmware describes it. Worth treating with suspicion on
// a mini PC, where the table is often copied from a reference design and lists slots the
// case has no room for.
type Slot struct {
Designation string `json:"designation"`
Type string `json:"type,omitempty"`
Usage string `json:"usage,omitempty"` // Available | In Use | Unknown
Length string `json:"length,omitempty"`
Width string `json:"width,omitempty"`
Bus string `json:"bus,omitempty"` // PCI address of what is in it
Device string `json:"device,omitempty"` // and what that turns out to be
}
// Board is the motherboard and the firmware on it.
type Board struct {
Maker string `json:"maker,omitempty"`
Product string `json:"product,omitempty"`
Version string `json:"version,omitempty"`
Serial string `json:"serial,omitempty"`
AssetTag string `json:"asset_tag,omitempty"`
Type string `json:"type,omitempty"` // "Motherboard", "Processor Module", …
Location string `json:"location,omitempty"` // where it sits in the chassis
Chassis string `json:"chassis,omitempty"`
ChassisMaker string `json:"chassis_maker,omitempty"`
BIOS string `json:"bios,omitempty"` // vendor and version
BIOSDate string `json:"bios_date,omitempty"`
BIOSRevision string `json:"bios_revision,omitempty"`
ROMSize string `json:"rom_size,omitempty"`
UEFI bool `json:"uefi,omitempty"`
Slots []Slot `json:"slots,omitempty"`
}
type Cache struct {
Level string `json:"level"` // L1d, L1i, L2, L3
Size string `json:"size"` // as lscpu puts it: "6 MiB (1 instance)"
}
type CPU struct {
Model string `json:"model,omitempty"`
Vendor string `json:"vendor,omitempty"`
Arch string `json:"arch,omitempty"`
Sockets int `json:"sockets,omitempty"`
SocketNames []string `json:"socket_names,omitempty"` // what the firmware calls each socket
Cores int `json:"cores,omitempty"`
Threads int `json:"threads,omitempty"`
ThreadsPerCore int `json:"threads_per_core,omitempty"`
MinMHz float64 `json:"min_mhz,omitempty"`
MaxMHz float64 `json:"max_mhz,omitempty"`
Family string `json:"family,omitempty"`
ModelID string `json:"model_id,omitempty"`
Stepping string `json:"stepping,omitempty"`
Microcode string `json:"microcode,omitempty"`
Virtualization string `json:"virtualization,omitempty"`
Caches []Cache `json:"caches,omitempty"`
Flags []string `json:"flags,omitempty"` // the handful anyone looks for
}
type Module struct {
Locator string `json:"locator"` // the slot, as the board labels it
Bank string `json:"bank,omitempty"`
Empty bool `json:"empty,omitempty"`
Size uint64 `json:"size,omitempty"`
Type string `json:"type,omitempty"` // DDR4
FormFactor string `json:"form_factor,omitempty"` // SODIMM
SpeedMTs int `json:"speed_mts,omitempty"` // what the module can do
RunningMTs int `json:"running_mts,omitempty"` // what the board is running it at
Maker string `json:"maker,omitempty"`
Part string `json:"part,omitempty"`
Serial string `json:"serial,omitempty"`
Rank int `json:"rank,omitempty"`
Voltage string `json:"voltage,omitempty"`
}
type Memory struct {
Total uint64 `json:"total"` // what the kernel can use
Installed uint64 `json:"installed,omitempty"` // what the modules add up to
MaxCapacity uint64 `json:"max_capacity,omitempty"` // what the board will take
Slots int `json:"slots,omitempty"`
Filled int `json:"filled,omitempty"`
ECC string `json:"ecc,omitempty"`
Modules []Module `json:"modules,omitempty"`
}
type Drive struct {
Name string `json:"name"`
Kind string `json:"kind"` // nvme | ssd | hdd | usb | emmc | sd | other
Model string `json:"model,omitempty"`
Vendor string `json:"vendor,omitempty"`
Serial string `json:"serial,omitempty"`
Firmware string `json:"firmware,omitempty"`
Tran string `json:"tran,omitempty"`
Size uint64 `json:"size"`
Rotational bool `json:"rotational,omitempty"`
Removable bool `json:"removable,omitempty"`
}
type GPU struct {
Model string `json:"model,omitempty"`
Vendor string `json:"vendor,omitempty"`
Slot string `json:"slot,omitempty"` // PCI address, when it is on a bus
Driver string `json:"driver,omitempty"`
Node string `json:"node,omitempty"` // card0
Memory uint64 `json:"memory,omitempty"`
Class string `json:"class,omitempty"` // VGA | 3D | Display
}
type Report struct {
At time.Time `json:"at"`
System System `json:"system"`
Board Board `json:"board"`
CPU CPU `json:"cpu"`
Memory Memory `json:"memory"`
Drives []Drive `json:"drives"`
GPUs []GPU `json:"gpus"`
Notes []string `json:"notes,omitempty"` // what could not be read, and why
}
// Manager remembers the last report. Hardware does not change under a running kernel —
// the one thing that would, a drive plugged in, is what Refresh is for.
type Manager struct {
mu sync.Mutex
last *Report
at time.Time
}
func New() *Manager { return &Manager{} }
const keepFor = 10 * time.Minute
func (m *Manager) Report(ctx context.Context, refresh bool) *Report {
m.mu.Lock()
if !refresh && m.last != nil && time.Since(m.at) < keepFor {
r := m.last
m.mu.Unlock()
return r
}
m.mu.Unlock()
r := collect(ctx)
m.mu.Lock()
m.last, m.at = r, time.Now()
m.mu.Unlock()
return r
}
func collect(ctx context.Context) *Report {
ctx, cancel := context.WithTimeout(ctx, 25*time.Second)
defer cancel()
r := &Report{At: time.Now()}
dmi, note := readDMI(ctx)
if note != "" {
r.Notes = append(r.Notes, note)
}
r.System = system(ctx, dmi)
r.Board = board(dmi, pciNames(ctx))
r.CPU = cpu(ctx, dmi)
r.Memory = memory(dmi)
if len(r.Memory.Modules) == 0 && note == "" {
r.Notes = append(r.Notes, "memory modules: the firmware's table says nothing about this machine's slots")
}
r.Drives = drives(ctx)
r.GPUs = gpus(ctx)
if len(r.GPUs) == 0 {
r.Notes = append(r.Notes, "graphics: nothing on the PCI bus and no kernel display device")
}
return r
}
// ---- dmidecode ----
// record is one entry of the firmware's table: its DMI type and its key/value lines.
type record struct {
Type int
V map[string]string
}
// listKey holds every doubly-indented line of a record — dmidecode's lists, which have no
// key of their own: BIOS characteristics, board features, slot characteristics.
const listKey = "\x00list"
var handleRe = regexp.MustCompile(`^Handle 0x[0-9A-Fa-f]+, DMI type (\d+)`)
// readDMI runs dmidecode once for every type the page wants and parses the lot. It needs
// root; without it the page loses the board, the memory modules and the socket names, and
// says so rather than pretending the machine has none.
func readDMI(ctx context.Context) ([]record, string) {
tool, err := exec.LookPath("dmidecode")
if err != nil {
if _, e := os.Stat("/usr/sbin/dmidecode"); e != nil {
return nil, "the board and the memory modules need dmidecode, which isn't installed here"
}
tool = "/usr/sbin/dmidecode"
}
args := []string{tool, "-t", "bios", "-t", "system", "-t", "baseboard", "-t", "chassis", "-t", "slot", "-t", "processor", "-t", "memory"}
if os.Geteuid() != 0 {
if !priv.Available() {
return nil, "the board and the memory modules come from the firmware's table, which only root may read"
}
args = priv.Wrap(args...)
}
out, err := exec.CommandContext(ctx, args[0], args[1:]...).Output()
if err != nil {
return nil, "the firmware's table could not be read: " + strings.TrimSpace(err.Error())
}
return parse(string(out)), ""
}
// parse turns dmidecode's output into records: a handle line starts one, and the single
// indented "Key: Value" lines under it are its fields. Doubly-indented lines are the
// continuation of a list (a processor's flags), which nothing here reads.
func parse(text string) []record {
var recs []record
var cur *record
sc := bufio.NewScanner(strings.NewReader(text))
sc.Buffer(make([]byte, 0, 64<<10), 2<<20)
for sc.Scan() {
line := sc.Text()
if m := handleRe.FindStringSubmatch(line); m != nil {
t, _ := strconv.Atoi(m[1])
recs = append(recs, record{Type: t, V: map[string]string{}})
cur = &recs[len(recs)-1]
continue
}
if cur == nil || !strings.HasPrefix(line, "\t") {
if strings.TrimSpace(line) == "" {
cur = nil
}
continue
}
if strings.HasPrefix(line, "\t\t") { // a list under the field above it, e.g. BIOS characteristics
cur.V[listKey] += strings.TrimSpace(line) + "\n"
continue
}
k, v, ok := strings.Cut(strings.TrimSpace(line), ":")
if !ok {
continue
}
cur.V[k] = strings.TrimSpace(v)
}
return recs
}
func (r record) get(keys ...string) string {
for _, k := range keys {
if v := r.V[k]; v != "" && !unset(v) {
return v
}
}
return ""
}
// unset is every way the firmware has of saying "nobody filled this in".
func unset(v string) bool {
switch strings.ToLower(v) {
case "", "unknown", "not specified", "not provided", "none", "default string", "to be filled by o.e.m.",
"system manufacturer", "system product name", "to be filled by oem", "n/a", "empty", "no module installed",
"<out of spec>", "other", "not applicable":
return true
}
return false
}
func first(recs []record, t int) record {
for _, r := range recs {
if r.Type == t {
return r
}
}
return record{Type: t, V: map[string]string{}}
}
func all(recs []record, t int) []record {
var out []record
for _, r := range recs {
if r.Type == t {
out = append(out, r)
}
}
return out
}
// ---- system ----
func system(ctx context.Context, dmi []record) System {
sys := first(dmi, 1)
s := System{
Model: join(sys.get("Manufacturer"), sys.get("Product Name")),
Serial: sys.get("Serial Number"),
Family: sys.get("Family"),
Arch: runtimeArch(),
}
s.Kernel = readFile("/proc/sys/kernel/osrelease")
if s.Model == "" { // no SMBIOS: a Pi, a Mac, anything with a device tree
s.Model = strings.TrimRight(readFile("/sys/firmware/devicetree/base/model"), "\x00")
}
if out, err := exec.CommandContext(ctx, "systemd-detect-virt").Output(); err == nil {
if v := strings.TrimSpace(string(out)); v != "" && v != "none" {
s.Virt = v
}
}
return s
}
// board reads the motherboard, the chassis it sits in, the firmware on it and the
// expansion slots the firmware claims. All of it is SMBIOS, so a machine without one — a
// Mac, a Pi — gets an empty board, and the page says so rather than inventing one.
func board(dmi []record, pci map[string]string) Board {
b, ch, bios := first(dmi, 2), first(dmi, 3), first(dmi, 0)
out := Board{
Maker: b.get("Manufacturer"),
Product: b.get("Product Name"),
Version: b.get("Version"),
Serial: b.get("Serial Number"),
AssetTag: b.get("Asset Tag"),
Type: b.get("Type"),
Location: b.get("Location In Chassis"),
Chassis: ch.get("Type"),
ChassisMaker: ch.get("Manufacturer"),
BIOS: join(bios.get("Vendor"), bios.get("Version")),
BIOSDate: bios.get("Release Date"),
BIOSRevision: bios.get("BIOS Revision"),
}
// "ROM Size: 0 MB" is a board that never filled the field in, not a board with no ROM
if rom := bios.get("ROM Size"); rom != "" && !strings.HasPrefix(rom, "0 ") {
out.ROMSize = rom
}
if out.Product == out.Maker {
out.Product = "" // a board named after its maker says nothing twice
}
out.UEFI = strings.Contains(bios.V[listKey], "UEFI") // a characteristics line, not a field
for _, sl := range all(dmi, 9) {
s := Slot{
Designation: sl.get("Designation"),
Type: sl.get("Type"),
Usage: sl.get("Current Usage"),
Length: sl.get("Length"),
Width: sl.get("Data Bus Width"),
}
// every slot on some boards claims 0000:00:00.0, which is the host bridge and
// certainly not what is plugged into that slot
if bus := sl.get("Bus Address"); bus != "" && bus != "0000:00:00.0" {
s.Bus, s.Device = bus, pci[bus]
}
if s.Designation != "" {
out.Slots = append(out.Slots, s)
}
}
return out
}
func join(parts ...string) string {
var keep []string
for _, p := range parts {
if p = strings.TrimSpace(p); p != "" && !unset(p) {
keep = append(keep, p)
}
}
// "GMKtec GMKtec" reads badly: drop a maker the product already names
if len(keep) == 2 && strings.HasPrefix(strings.ToLower(keep[1]), strings.ToLower(keep[0])) {
return keep[1]
}
return strings.Join(keep, " ")
}
func readFile(p string) string {
b, err := os.ReadFile(p)
if err != nil {
return ""
}
return strings.TrimSpace(string(b))
}
func runtimeArch() string {
if out, err := exec.Command("uname", "-m").Output(); err == nil {
return strings.TrimSpace(string(out))
}
return ""
}
// ---- processor ----
// worthShowing is the short list of processor features anyone actually asks about; the
// full flag list runs to hundreds and belongs in a terminal.
var worthShowing = []string{"aes", "avx", "avx2", "avx512f", "sha_ni", "vmx", "svm", "sse4_2", "rdrand", "hypervisor"}
func cpu(ctx context.Context, dmi []record) CPU {
c := CPU{}
f := lscpu(ctx)
c.Model = f["Model name"]
c.Vendor = f["Vendor ID"]
c.Arch = f["Architecture"]
c.Sockets = atoi(f["Socket(s)"])
c.ThreadsPerCore = atoi(f["Thread(s) per core"])
c.Threads = atoi(f["CPU(s)"])
c.Cores = atoi(f["Core(s) per socket"]) * maxInt(1, c.Sockets)
c.MinMHz = atof(f["CPU min MHz"])
c.MaxMHz = atof(f["CPU max MHz"])
c.Family, c.ModelID, c.Stepping = f["CPU family"], f["Model"], f["Stepping"]
c.Virtualization = f["Virtualization"]
for _, l := range []string{"L1d", "L1i", "L2", "L3"} {
if v := f[l+" cache"]; v != "" {
c.Caches = append(c.Caches, Cache{Level: l, Size: v})
}
}
have := map[string]bool{}
for _, fl := range strings.Fields(f["Flags"]) {
have[fl] = true
}
for _, fl := range worthShowing {
if have[fl] {
c.Flags = append(c.Flags, fl)
}
}
for _, p := range all(dmi, 4) {
if s := p.get("Socket Designation"); s != "" {
c.SocketNames = append(c.SocketNames, s)
}
if c.Model == "" {
c.Model = p.get("Version", "Family")
}
if c.MaxMHz == 0 {
if mhz := atof(strings.TrimSuffix(p.get("Max Speed"), " MHz")); mhz > 0 {
c.MaxMHz = mhz
}
}
}
if c.Sockets == 0 {
c.Sockets = len(c.SocketNames)
}
// A processor whose cores are not all alike — Apple's, Intel's P and E cores — is
// reported by lscpu as several groups, and "core(s) per socket" then counts one group
// rather than the chip. Without SMT every thread is a core, so the thread count is the
// honest answer.
if c.ThreadsPerCore == 1 && c.Threads > c.Cores {
c.Cores = c.Threads
}
// /proc/cpuinfo is the only place the running microcode revision is written down
if b, err := os.ReadFile("/proc/cpuinfo"); err == nil {
for _, line := range strings.Split(string(b), "\n") {
if k, v, ok := strings.Cut(line, ":"); ok && strings.TrimSpace(k) == "microcode" {
c.Microcode = strings.TrimSpace(v)
break
}
}
}
if c.Model == "" {
c.Model = "unknown processor"
}
return c
}
func lscpu(ctx context.Context) map[string]string {
f := map[string]string{}
out, err := exec.CommandContext(ctx, "lscpu").Output()
if err != nil {
// no util-linux: /proc/cpuinfo still names the thing and counts it
if b, err := os.ReadFile("/proc/cpuinfo"); err == nil {
n := 0
for _, line := range strings.Split(string(b), "\n") {
k, v, ok := strings.Cut(line, ":")
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
switch {
case ok && strings.EqualFold(k, "model name"):
f["Model name"] = v
case ok && k == "processor":
n++
}
}
f["CPU(s)"] = strconv.Itoa(n)
}
return f
}
for _, line := range strings.Split(string(out), "\n") {
k, v, ok := strings.Cut(line, ":")
if !ok || strings.HasPrefix(line, " ") {
continue
}
f[strings.TrimSpace(k)] = strings.TrimSpace(v)
}
return f
}
func atoi(s string) int { n, _ := strconv.Atoi(strings.TrimSpace(s)); return n }
func atof(s string) float64 {
v, _ := strconv.ParseFloat(strings.TrimSpace(s), 64)
return v
}
func maxInt(a, b int) int {
if a > b {
return a
}
return b
}
// leadingInt reads the number off the front of "3200 MT/s".
func leadingInt(s string) int {
f := strings.Fields(s)
if len(f) == 0 {
return 0
}
return atoi(f[0])
}
// ---- memory ----
var sizeRe = regexp.MustCompile(`^(\d+)\s*(kB|KB|MB|GB|TB)$`)
// dmiSize turns dmidecode's "8 GB" / "16384 MB" into bytes.
func dmiSize(s string) uint64 {
m := sizeRe.FindStringSubmatch(strings.TrimSpace(s))
if m == nil {
return 0
}
n, _ := strconv.ParseUint(m[1], 10, 64)
switch strings.ToUpper(m[2]) {
case "KB":
return n << 10
case "MB":
return n << 20
case "GB":
return n << 30
case "TB":
return n << 40
}
return 0
}
func memory(dmi []record) Memory {
m := Memory{}
if b, err := os.ReadFile("/proc/meminfo"); err == nil {
for _, line := range strings.Split(string(b), "\n") {
if k, v, ok := strings.Cut(line, ":"); ok && k == "MemTotal" {
m.Total = uint64(atoi(strings.TrimSuffix(strings.TrimSpace(v), " kB"))) << 10
break
}
}
}
for _, a := range all(dmi, 16) {
m.MaxCapacity += dmiSize(a.get("Maximum Capacity"))
switch e := a.V["Error Correction Type"]; {
case e == "None":
m.ECC = "non-ECC" // the array says so outright; worth knowing, not worth hiding
case e != "" && !unset(e):
m.ECC = e
}
}
for _, d := range all(dmi, 17) {
mod := Module{
Locator: d.get("Locator"),
Bank: d.get("Bank Locator"),
Size: dmiSize(d.get("Size")),
}
m.Slots++
if mod.Locator == "" {
mod.Locator = "slot " + strconv.Itoa(m.Slots)
}
if mod.Size == 0 {
mod.Empty = true
m.Modules = append(m.Modules, mod)
continue
}
m.Filled++
mod.Type = d.get("Type")
mod.FormFactor = d.get("Form Factor")
mod.SpeedMTs = leadingInt(d.get("Speed"))
mod.RunningMTs = leadingInt(d.get("Configured Memory Speed", "Configured Clock Speed"))
mod.Maker = makerName(d.get("Manufacturer"), d.V["Module Manufacturer ID"])
mod.Part = d.get("Part Number")
mod.Serial = d.get("Serial Number")
mod.Rank = atoi(d.get("Rank"))
mod.Voltage = d.get("Configured Voltage")
m.Installed += mod.Size
m.Modules = append(m.Modules, mod)
}
return m
}
// jedec maps the memory makers who turn up in a homelab to their names: SPD carries a
// JEDEC id, and dmidecode prints the raw code when the module's string field is empty.
var jedec = map[string]string{
"0X2C00": "Micron", "0X802C": "Micron",
"0XCE00": "Samsung", "0X80CE": "Samsung", "0X00CE": "Samsung",
"0XAD00": "SK hynix", "0X80AD": "SK hynix",
"0X9801": "Kingston", "0X7F98": "Kingston", "0X0198": "Kingston",
"0X859B": "Crucial", "0X04CB": "A-DATA", "0X04CD": "G.Skill",
"0X029E": "Corsair", "0X0B45": "Wilk Elektronik", "0X830B": "Wilk Elektronik",
"0X0443": "Ramaxel", "0X0194": "Smart Modular",
}
// makerName prefers what the module says it is; a bare JEDEC id becomes a name when it is
// one we know, and stays the id when it is not — a wrong name is worse than a number.
func makerName(v, id string) string {
if v != "" && !unset(v) && !strings.HasPrefix(v, "0x") {
return v
}
code := v
if code == "" {
code = id
}
if n, ok := jedec[strings.ToUpper(code)]; ok {
return n
}
if v != "" && !unset(v) {
return v
}
return ""
}
// ---- drives ----
func drives(ctx context.Context) []Drive {
devs, err := storage.Devices(ctx)
if err != nil {
return nil
}
out := make([]Drive, 0, len(devs))
for _, d := range devs {
out = append(out, Drive{
Name: d.Name, Kind: d.Kind, Model: d.Model, Vendor: d.Vendor, Serial: d.Serial,
Tran: d.Tran, Size: d.Size, Rotational: d.Rotational, Removable: d.Removable,
Firmware: firmwareOf(d.Name),
})
}
return out
}
// firmwareOf reads a drive's firmware revision. An NVMe device is folded to its
// controller ("nvme0"), which has no /sys/block entry of its own — the revision is on the
// controller in /sys/class/nvme.
func firmwareOf(name string) string {
return firstNonEmpty(
readFile("/sys/block/"+name+"/device/firmware_rev"),
readFile("/sys/block/"+name+"/device/rev"),
readFile("/sys/class/nvme/"+name+"/firmware_rev"),
)
}
func firstNonEmpty(v ...string) string {
for _, s := range v {
if s != "" {
return s
}
}
return ""
}
// ---- graphics ----
// lspci -mm quotes every field, so a device name can hold the spaces and brackets it
// likes: `"VGA compatible controller [0300]" "Intel Corporation [8086]" "Alder Lake-N …"`.
var quoted = regexp.MustCompile(`"([^"]*)"`)
// stripID drops the trailing PCI id from `Alder Lake-N [UHD Graphics] [46d1]`.
func stripID(s string) string {
if i := strings.LastIndex(s, " ["); i > 0 && strings.HasSuffix(s, "]") {
return strings.TrimSpace(s[:i])
}
return s
}
func gpus(ctx context.Context) []GPU {
var out []GPU
seen := map[string]bool{} // PCI addresses already accounted for
if _, err := exec.LookPath("lspci"); err == nil {
drivers := pciDrivers(ctx)
if b, err := exec.CommandContext(ctx, "lspci", "-D", "-mm", "-nn").Output(); err == nil {
for _, line := range strings.Split(string(b), "\n") {
slot, _, _ := strings.Cut(line, " ")
fields := quoted.FindAllStringSubmatch(line, -1)
if len(fields) < 3 {
continue
}
kind := displayClass(fields[0][1])
if kind == "" {
continue
}
seen[slot] = true
g := GPU{Slot: slot, Class: kind, Vendor: stripID(fields[1][1]), Model: stripID(fields[2][1]), Driver: drivers[slot]}
g.Node, g.Memory = drmFor(slot)
out = append(out, g)
}
}
}
// A GPU that is not on a PCI bus at all — an Apple Silicon Mac, a Pi, anything with a
// device tree — is only visible as a kernel display device.
cards, _ := filepath.Glob("/sys/class/drm/card[0-9]")
for _, c := range cards {
dev, err := filepath.EvalSymlinks(c + "/device")
if err != nil || seen[filepath.Base(dev)] {
continue
}
g := GPU{Node: filepath.Base(c), Class: "Display"}
if drv, err := filepath.EvalSymlinks(dev + "/driver"); err == nil {
g.Driver = filepath.Base(drv)
}
g.Model = deviceTreeName(dev)
if g.Model == "" && g.Driver != "" {
g.Model = g.Driver + " display"
}
if g.Model == "" {
continue
}
g.Memory = vram(dev)
out = append(out, g)
}
sort.SliceStable(out, func(i, j int) bool { return out[i].Slot < out[j].Slot })
return out
}
// pciNames maps a PCI address to a readable name, so a slot the firmware says is in use
// can say what is in it.
func pciNames(ctx context.Context) map[string]string {
m := map[string]string{}
if _, err := exec.LookPath("lspci"); err != nil {
return m
}
out, err := exec.CommandContext(ctx, "lspci", "-D", "-mm", "-nn").Output()
if err != nil {
return m
}
for _, line := range strings.Split(string(out), "\n") {
slot, _, _ := strings.Cut(line, " ")
f := quoted.FindAllStringSubmatch(line, -1)
if slot == "" || len(f) < 3 {
continue
}
m[slot] = strings.TrimSpace(stripID(f[1][1]) + " " + stripID(f[2][1]))
}
return m
}
func displayClass(c string) string {
switch {
case strings.Contains(c, "[0300]"):
return "VGA"
case strings.Contains(c, "[0302]"):
return "3D"
case strings.Contains(c, "[0301]"), strings.Contains(c, "[0380]"):
return "Display"
}
return ""
}
// pciDrivers maps a PCI address to the kernel module driving it.
func pciDrivers(ctx context.Context) map[string]string {
m := map[string]string{}
out, err := exec.CommandContext(ctx, "lspci", "-D", "-k").Output()
if err != nil {
return m
}
slot := ""
for _, line := range strings.Split(string(out), "\n") {
if !strings.HasPrefix(line, "\t") && !strings.HasPrefix(line, " ") {
slot, _, _ = strings.Cut(line, " ")
continue
}
if v, ok := strings.CutPrefix(strings.TrimSpace(line), "Kernel driver in use:"); ok && slot != "" {
m[slot] = strings.TrimSpace(v)
}
}
return m
}
// drmFor finds the kernel display device for a PCI address, and its video memory when the
// driver publishes it (amdgpu does; Intel's integrated graphics have none of their own).
func drmFor(slot string) (string, uint64) {
cards, _ := filepath.Glob("/sys/class/drm/card[0-9]")
for _, c := range cards {
dev, err := filepath.EvalSymlinks(c + "/device")
if err != nil || filepath.Base(dev) != slot {
continue
}
return filepath.Base(c), vram(dev)
}
return "", 0
}
func vram(dev string) uint64 {
for _, f := range []string{"mem_info_vram_total", "mem_info_vis_vram_total"} {
if v := readFile(dev + "/" + f); v != "" {
n, _ := strconv.ParseUint(v, 10, 64)
return n
}
}
return 0
}
// deviceTreeName reads what a device-tree machine calls its graphics: the compatible
// string, tidied — "apple,agx-g13g" becomes "Apple agx-g13g".
func deviceTreeName(dev string) string {
c := readFile(dev + "/of_node/compatible")
if c == "" {
return ""
}
c = strings.TrimRight(strings.Split(c, "\x00")[0], "\x00")
maker, rest, ok := strings.Cut(c, ",")
if !ok {
return c
}
if maker != "" {
maker = strings.ToUpper(maker[:1]) + maker[1:]
}
return strings.TrimSpace(maker + " " + rest)
}
internal/hardware/hardware_test.go
+103
View File
@@ -0,0 +1,103 @@
package hardware
import "testing"
func TestDMISize(t *testing.T) {
for in, want := range map[string]uint64{
"8 GB": 8 << 30, "16384 MB": 16384 << 20, "512 kB": 512 << 10, "1 TB": 1 << 40,
"No Module Installed": 0, "": 0, "8 GiB": 0,
} {
if got := dmiSize(in); got != want {
t.Errorf("dmiSize(%q) = %d, want %d", in, got, want)
}
}
}
func TestMakerName(t *testing.T) {
cases := []struct{ v, id, want string }{
{"Samsung", "", "Samsung"}, // the module says who it is
{"0x0B45", "Bank 12, Hex 0x45", "Wilk Elektronik"}, // only a JEDEC id, and one we know
{"0xFFFF", "", "0xFFFF"}, // one we don't: the number, not a guess
{"Unknown", "", ""},
{"", "0xCE00", "Samsung"},
}
for _, c := range cases {
if got := makerName(c.v, c.id); got != c.want {
t.Errorf("makerName(%q, %q) = %q, want %q", c.v, c.id, got, c.want)
}
}
}
func TestStripIDAndDisplayClass(t *testing.T) {
if got := stripID("Alder Lake-N [UHD Graphics] [46d1]"); got != "Alder Lake-N [UHD Graphics]" {
t.Errorf("stripID = %q", got)
}
if got := stripID("no brackets here"); got != "no brackets here" {
t.Errorf("stripID left a plain name alone wrong: %q", got)
}
if displayClass(`VGA compatible controller [0300]`) != "VGA" {
t.Error("a VGA controller should be graphics")
}
if displayClass(`Ethernet controller [0200]`) != "" {
t.Error("a network card is not graphics")
}
}
func TestLeadingIntAndJoin(t *testing.T) {
if leadingInt("3200 MT/s") != 3200 || leadingInt("") != 0 || leadingInt("Unknown") != 0 {
t.Error("leadingInt")
}
if got := join("GMKtec", "GMKtec NucBox G3"); got != "GMKtec NucBox G3" {
t.Errorf("a maker the product already names should not be said twice: %q", got)
}
if got := join("Dell Inc.", "OptiPlex 7040"); got != "Dell Inc. OptiPlex 7040" {
t.Errorf("join = %q", got)
}
if got := join("To Be Filled By O.E.M.", "System Product Name"); got != "" {
t.Errorf("firmware boilerplate is not a model: %q", got)
}
}
const sample = `# dmidecode 3.6
Handle 0x0028, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0027
Size: 8 GB
Form Factor: SODIMM
Locator: Controller0-ChannelA-DIMM0
Type: DDR4
Speed: 3200 MT/s
Manufacturer: 0x0B45
Part Number: WPBH32D408SWD-8G
Configured Memory Speed: 2933 MT/s
Rank: 1
Handle 0x0029, DMI type 17, 92 bytes
Memory Device
Size: No Module Installed
Locator: Controller1-ChannelA-DIMM0
Type: Unknown
`
// parseRecords is what readDMI does once dmidecode has answered; this exercises it
// without needing root or the tool.
func TestMemoryFromRecords(t *testing.T) {
recs := parse(sample)
m := memory(recs)
if m.Slots != 2 || m.Filled != 1 {
t.Fatalf("slots %d filled %d, want 2 and 1", m.Slots, m.Filled)
}
mod := m.Modules[0]
if mod.Size != 8<<30 || mod.Type != "DDR4" || mod.Maker != "Wilk Elektronik" || mod.Part != "WPBH32D408SWD-8G" {
t.Errorf("first module read wrong: %+v", mod)
}
if mod.SpeedMTs != 3200 || mod.RunningMTs != 2933 {
t.Errorf("a module running below its rating should say both: %d / %d", mod.RunningMTs, mod.SpeedMTs)
}
if !m.Modules[1].Empty || m.Modules[1].Locator != "Controller1-ChannelA-DIMM0" {
t.Errorf("empty slot read wrong: %+v", m.Modules[1])
}
if m.Installed != 8<<30 {
t.Errorf("installed = %d", m.Installed)
}
}
internal/metrics/fold.go
+141
View File
@@ -0,0 +1,141 @@
package metrics
// One kept history point is the average of the samples behind it. At the default ten-minute
// window that bucket is a single sample and nothing is averaged; at a day it is five minutes
// of them, and the difference matters — a graph of the last day drawn from whichever instant
// landed on each five-minute mark would show spikes that never happened and miss the ones
// that did.
//
// The newest sample of the bucket is the template: its shape, labels, totals and slow
// sections (filesystems, UPS) are the current ones. Only the moving numbers are averaged,
// and only where an entry is present in that sample — a disk plugged in halfway through the
// bucket is averaged over the samples that have it.
func fold(b []Sample) Sample {
out := b[len(b)-1]
if len(b) == 1 {
return out
}
n := float64(len(b))
sum := func(get func(Sample) float64) float64 {
var t float64
for _, s := range b {
t += get(s)
}
return t / n
}
sumU := func(get func(Sample) uint64) uint64 {
var t uint64
for _, s := range b {
t += get(s)
}
return t / uint64(len(b))
}
out.CPU = sum(func(s Sample) float64 { return s.CPU })
out.Load[0] = sum(func(s Sample) float64 { return s.Load[0] })
out.Load[1] = sum(func(s Sample) float64 { return s.Load[1] })
out.Load[2] = sum(func(s Sample) float64 { return s.Load[2] })
out.Cores = append([]float64(nil), out.Cores...)
for i := range out.Cores {
var t, c float64
for _, s := range b {
if i < len(s.Cores) {
t, c = t+s.Cores[i], c+1
}
}
if c > 0 {
out.Cores[i] = t / c
}
}
out.Mem.Used = sumU(func(s Sample) uint64 { return s.Mem.Used })
out.Mem.Avail = sumU(func(s Sample) uint64 { return s.Mem.Avail })
out.Mem.Free = sumU(func(s Sample) uint64 { return s.Mem.Free })
out.Mem.Cached = sumU(func(s Sample) uint64 { return s.Mem.Cached })
out.Swap.Used = sumU(func(s Sample) uint64 { return s.Swap.Used })
// the rest are lists whose entries are matched by name across the bucket
temps := avgBy(b, func(s Sample) map[string][]float64 {
m := make(map[string][]float64, len(s.Temps))
for _, x := range s.Temps {
m[x.Chip+"/"+x.Label] = []float64{x.C}
}
return m
})
out.Temps = append([]Temp(nil), out.Temps...)
for i, x := range out.Temps {
if v, ok := temps[x.Chip+"/"+x.Label]; ok {
out.Temps[i].C = v[0]
}
}
fans := avgBy(b, func(s Sample) map[string][]float64 {
m := make(map[string][]float64, len(s.Fans))
for _, x := range s.Fans {
m[x.Chip+"/"+x.Label] = []float64{float64(x.RPM)}
}
return m
})
out.Fans = append([]Fan(nil), out.Fans...)
for i, x := range out.Fans {
if v, ok := fans[x.Chip+"/"+x.Label]; ok {
out.Fans[i].RPM = int(v[0] + .5)
}
}
nets := avgBy(b, func(s Sample) map[string][]float64 {
m := make(map[string][]float64, len(s.Net))
for _, x := range s.Net {
m[x.Iface] = []float64{x.RxBps, x.TxBps}
}
return m
})
out.Net = append([]Net(nil), out.Net...)
for i, x := range out.Net {
if v, ok := nets[x.Iface]; ok {
out.Net[i].RxBps, out.Net[i].TxBps = v[0], v[1]
}
}
disks := avgBy(b, func(s Sample) map[string][]float64 {
m := make(map[string][]float64, len(s.Disks))
for _, x := range s.Disks {
m[x.Name] = []float64{x.ReadBps, x.WriteBps, x.UtilPct}
}
return m
})
out.Disks = append([]Disk(nil), out.Disks...)
for i, x := range out.Disks {
if v, ok := disks[x.Name]; ok {
out.Disks[i].ReadBps, out.Disks[i].WriteBps, out.Disks[i].UtilPct = v[0], v[1], v[2]
}
}
return out
}
// avgBy averages each sample's named numbers across the bucket, per name.
func avgBy(b []Sample, of func(Sample) map[string][]float64) map[string][]float64 {
sums := map[string][]float64{}
count := map[string]float64{}
for _, s := range b {
for k, vs := range of(s) {
acc := sums[k]
if acc == nil {
acc = make([]float64, len(vs))
}
for i, v := range vs {
if i < len(acc) {
acc[i] += v
}
}
sums[k], count[k] = acc, count[k]+1
}
}
for k, acc := range sums {
for i := range acc {
acc[i] /= count[k]
}
}
return sums
}
internal/metrics/fold_test.go
+143
View File
@@ -0,0 +1,143 @@
package metrics
import (
"testing"
"time"
)
func TestFoldAverages(t *testing.T) {
b := []Sample{
{T: 1, CPU: 10, Cores: []float64{10, 0}, Mem: Mem{Total: 100, Used: 10},
Temps: []Temp{{Chip: "c", Label: "l", C: 40}}, Net: []Net{{Iface: "eth0", RxBps: 100}},
Disks: []Disk{{Name: "sda", ReadBps: 10, UtilPct: 5}}},
{T: 2, CPU: 30, Cores: []float64{30, 100}, Mem: Mem{Total: 100, Used: 30},
Temps: []Temp{{Chip: "c", Label: "l", C: 60}}, Net: []Net{{Iface: "eth0", RxBps: 300}, {Iface: "new", RxBps: 7}},
Disks: []Disk{{Name: "sda", ReadBps: 30, UtilPct: 15}}},
}
f := fold(b)
if f.T != 2 {
t.Errorf("a folded point is stamped with the newest sample in it, got %d", f.T)
}
if f.CPU != 20 || f.Cores[0] != 20 || f.Cores[1] != 50 {
t.Errorf("cpu averaged: %v %v", f.CPU, f.Cores)
}
if f.Mem.Used != 20 || f.Mem.Total != 100 {
t.Errorf("memory: used averaged, total taken from the newest, got %+v", f.Mem)
}
if f.Temps[0].C != 50 || f.Net[0].RxBps != 200 || f.Disks[0].ReadBps != 20 || f.Disks[0].UtilPct != 10 {
t.Errorf("lists matched by name: %v %v %v", f.Temps, f.Net, f.Disks)
}
if len(f.Net) != 2 || f.Net[1].RxBps != 7 {
t.Errorf("an interface that appeared inside the bucket is averaged over the samples that have it: %v", f.Net)
}
// the template's own slices must not be touched: Latest() shares them
if b[1].Cores[0] != 30 || b[1].Temps[0].C != 60 {
t.Error("fold wrote through to the sample it templated from")
}
}
func TestStrideAndTrim(t *testing.T) {
c := &Collector{interval: 2 * time.Second, window: DefaultWindow}
if got := c.stride(); got != 2*time.Second {
t.Errorf("a window of %v over %d points is finer than the interval, so the interval wins; got %v", DefaultWindow, History, got)
}
c.window = 24 * time.Hour
if got := c.stride(); got != 24*time.Hour/History {
t.Errorf("stride at a day: %v", got)
}
// a full day of samples at the day stride keeps exactly the budget
c.interval, c.window = time.Second, time.Hour
stride := c.stride().Milliseconds()
for i := 0; i < 5*History; i++ {
c.keep(Sample{T: int64(i) * stride})
}
if len(c.ring) != History {
t.Errorf("the point count is the budget, got %d", len(c.ring))
}
if span := c.ring[len(c.ring)-1].T - c.ring[0].T; span > c.window.Milliseconds() {
t.Errorf("the kept history reaches back further than the window: %d ms", span)
}
// narrowing the window drops what is now too old, at once
c.SetWindow(MinWindow)
if span := c.ring[len(c.ring)-1].T - c.ring[0].T; span > MinWindow.Milliseconds() {
t.Errorf("narrowing the window left %d ms of history", span)
}
c.SetWindow(365 * 24 * time.Hour)
if c.Window() != MaxWindow {
t.Errorf("a window past the maximum is the maximum, got %v", c.Window())
}
}
// Points are one stride apart — not a stride and a sample — and jitter doesn't change that.
func TestKeepSpacing(t *testing.T) {
c := &Collector{interval: 2 * time.Second, window: time.Hour} // stride 12 s: six samples a point
at := int64(1_000_000)
for i := 0; i < 6*40; i++ {
jitter := int64((i*37)%41 - 20) // ±20 ms, deterministic
c.keep(Sample{T: at + jitter, CPU: float64(i % 6)})
at += 2000
}
if len(c.ring) != 40 {
t.Fatalf("240 samples at six a point are 40 points, got %d", len(c.ring))
}
for i := 1; i < len(c.ring); i++ {
if gap := c.ring[i].T - c.ring[i-1].T; gap < 11900 || gap > 12100 {
t.Fatalf("point %d is %d ms after the one before; a stride is 12000", i, gap)
}
}
if c.ring[3].CPU != 2.5 {
t.Errorf("each point is the mean of its own six samples (0…5 → 2.5), got %v", c.ring[3].CPU)
}
// and when the stride is the interval, every sample is a point
c = &Collector{interval: 2 * time.Second, window: DefaultWindow}
for i := 0; i < 50; i++ {
c.keep(Sample{T: int64(i)*2000 + int64(i%3) - 1})
}
if len(c.ring) != 50 {
t.Errorf("at the default window every sample is kept, got %d of 50", len(c.ring))
}
}
func TestFoldNVMe(t *testing.T) {
// an Apple SSD as the kernel lists it: one controller, three namespaces
got := foldNVMe([]Disk{
{Name: "nvme0n1", Model: "APPLE SSD AP0512Z", Size: 500 << 30, ReadBps: 100, WriteBps: 200, UtilPct: 12, Temp: 37},
{Name: "nvme0n2", Model: "APPLE SSD AP0512Z", Size: 3 << 20, ReadBps: 10, WriteBps: 20, UtilPct: 40},
{Name: "nvme0n3", Model: "APPLE SSD AP0512Z", Size: 128 << 20, ReadBps: 1, WriteBps: 2, UtilPct: 5},
{Name: "sda", Size: 100 << 30, ReadBps: 7, UtilPct: 3},
{Name: "nvme1n1", Size: 200 << 30, ReadBps: 5},
})
if len(got) != 3 {
t.Fatalf("three namespaces of one controller are one drive: %v", names(got))
}
d := got[0]
if d.Name != "nvme0" {
t.Errorf("the drive is the controller, got %q", d.Name)
}
if d.Size != 500<<30+3<<20+128<<20 {
t.Errorf("its size is all of its namespaces, got %d", d.Size)
}
if d.ReadBps != 111 || d.WriteBps != 222 {
t.Errorf("rates add up, got %v/%v", d.ReadBps, d.WriteBps)
}
if d.UtilPct != 40 {
t.Errorf("busy is the longest of them, not the sum: got %v", d.UtilPct)
}
if d.Temp != 37 {
t.Errorf("a temperature from any namespace is the drive's, got %v", d.Temp)
}
if got[1].Name != "sda" || got[1].UtilPct != 3 || got[2].Name != "nvme1" {
t.Errorf("everything else is left alone: %v", names(got))
}
if n := len(foldNVMe(nil)); n != 0 {
t.Errorf("no disks, no drives, got %d", n)
}
}
func names(d []Disk) []string {
out := make([]string, len(d))
for i, x := range d {
out[i] = x.Name
}
return out
}
internal/metrics/metrics.go
+1354
View File
File diff suppressed because it is too large Load Diff
internal/metrics/sensors.go
+187
View File
@@ -0,0 +1,187 @@
package metrics
import (
"fmt"
"strconv"
"hamlab/internal/sensors"
)
// The collector as a sensors.Provider: every number in a Sample, one sensor each. The
// IDs are built from names the hardware reports (chip, label, interface, device, mount),
// never from enumeration order like hwmon3, which moves between boots.
// flattener turns one Sample into sensors. With want set it builds only those, so walking
// the whole ring for a history costs string compares, not allocations.
type flattener struct {
want func(id string) bool
seen map[string]int
out []sensors.Sensor
}
type bounds struct{ min, max, warn, crit *float64 }
var pct = bounds{min: sensors.F(0), max: sensors.F(100)}
func upTo(max float64) bounds { return bounds{min: sensors.F(0), max: sensors.F(max)} }
func (f *flattener) add(id, name, kind, unit string, v any, t int64, b bounds, labels ...string) {
f.seen[id]++
if n := f.seen[id]; n > 1 { // two identical chips: the second is coretemp_2
id = fmt.Sprintf("%s_%d", id, n)
}
if f.want != nil && !f.want(id) {
return
}
s := sensors.Sensor{ID: id, Name: name, Kind: kind, Unit: unit, Value: v, T: t, Min: b.min, Max: b.max, Warn: b.warn, Crit: b.crit}
if len(labels) > 0 {
s.Labels = make(map[string]string, len(labels)/2)
for i := 0; i+1 < len(labels); i += 2 {
s.Labels[labels[i]] = labels[i+1]
}
}
f.out = append(f.out, s)
}
func ratio(part, whole uint64) float64 {
if whole == 0 {
return 0
}
return round1(float64(part) / float64(whole) * 100)
}
func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
f := &flattener{want: want, seen: map[string]int{}}
t, none := s.T, bounds{}
f.add("cpu.usage", "CPU", "usage", "%", s.CPU, t, pct)
for i, c := range s.Cores {
n := strconv.Itoa(i)
f.add("cpu.core."+n+".usage", "Core "+n, "usage", "%", c, t, pct, "core", n)
}
if c := cpuTemp(s.Temps); c > 0 {
f.add("cpu.temp", "CPU temperature", "temperature", "°C", c, t, none)
}
for i, w := range []string{"1", "5", "15"} {
f.add("load."+w, "Load "+w+" min", "load", "", s.Load[i], t, none)
}
f.add("uptime", "Uptime", "duration", "s", s.Uptime, t, none)
f.add("hamlab.sample_interval", "Sampling interval", "duration", "ms", float64(s.Interval), t, none)
for _, m := range []struct {
id, name string
v Mem
}{{"mem", "Memory", s.Mem}, {"swap", "Swap", s.Swap}} {
f.add(m.id+".total", m.name+" total", "bytes", "B", float64(m.v.Total), t, none)
f.add(m.id+".used", m.name+" used", "bytes", "B", float64(m.v.Used), t, upTo(float64(m.v.Total)))
f.add(m.id+".avail", m.name+" available", "bytes", "B", float64(m.v.Avail), t, upTo(float64(m.v.Total)))
f.add(m.id+".used_pct", m.name+" used", "usage", "%", ratio(m.v.Used, m.v.Total), t, pct)
}
f.add("mem.free", "Memory free", "bytes", "B", float64(s.Mem.Free), t, upTo(float64(s.Mem.Total)))
f.add("mem.cached", "Memory cached", "bytes", "B", float64(s.Mem.Cached), t, upTo(float64(s.Mem.Total)))
f.add("mem.buffers", "Memory buffers", "bytes", "B", float64(s.Mem.Buffers), t, upTo(float64(s.Mem.Total)))
for _, x := range s.Temps {
b := bounds{}
if x.Max > 0 {
b.warn = sensors.F(x.Max)
}
if x.Crit > 0 {
b.crit = sensors.F(x.Crit)
}
f.add(sensors.ID("temp", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "temperature", "°C", x.C, t, b, "chip", x.Chip, "label", x.Label)
}
for _, x := range s.Fans {
f.add(sensors.ID("fan", sensors.Seg(x.Chip), sensors.Seg(x.Label)), x.Chip+" "+x.Label, "fan", "rpm", float64(x.RPM), t, none, "chip", x.Chip, "label", x.Label)
}
for _, x := range s.Net {
p := "net." + sensors.Seg(x.Iface)
f.add(p+".rx", x.Iface+" receive", "rate", "B/s", x.RxBps, t, none, "iface", x.Iface)
f.add(p+".tx", x.Iface+" transmit", "rate", "B/s", x.TxBps, t, none, "iface", x.Iface)
f.add(p+".rx_total", x.Iface+" received", "bytes", "B", float64(x.RxTotal), t, none, "iface", x.Iface)
f.add(p+".tx_total", x.Iface+" transmitted", "bytes", "B", float64(x.TxTotal), t, none, "iface", x.Iface)
f.add(p+".up", x.Iface+" link", "state", "", x.Up, t, none, "iface", x.Iface)
}
for _, x := range s.Disks {
p := "disk." + sensors.Seg(x.Name)
f.add(p+".read", x.Name+" read", "rate", "B/s", x.ReadBps, t, none, "device", x.Name, "model", x.Model)
f.add(p+".write", x.Name+" write", "rate", "B/s", x.WriteBps, t, none, "device", x.Name, "model", x.Model)
f.add(p+".util", x.Name+" busy", "usage", "%", x.UtilPct, t, pct, "device", x.Name, "model", x.Model)
f.add(p+".size", x.Name+" size", "bytes", "B", float64(x.Size), t, none, "device", x.Name, "model", x.Model)
if x.Temp > 0 {
f.add(p+".temp", x.Name+" temperature", "temperature", "°C", x.Temp, t, none, "device", x.Name, "model", x.Model)
}
}
for _, x := range s.FS {
p := "fs." + sensors.Seg(x.Mount)
// remote says the bytes are on another machine (a CIFS or NFS share), so a view of
// this machine's hardware can leave it out without losing the sensor
lbl := []string{"mount", x.Mount, "fstype", x.Type}
if x.Remote {
lbl = append(lbl, "remote", "1")
}
if x.Disk != "" {
lbl = append(lbl, "disk", x.Disk)
}
f.add(p+".size", x.Mount+" size", "bytes", "B", float64(x.Size), s.fsT, none, lbl...)
f.add(p+".used", x.Mount+" used", "bytes", "B", float64(x.Used), s.fsT, upTo(float64(x.Size)), lbl...)
f.add(p+".used_pct", x.Mount+" used", "usage", "%", ratio(x.Used, x.Size), s.fsT, pct, lbl...)
}
if u := s.UPS; u != nil {
ut := s.upsT
f.add("ups.battery", "UPS battery", "usage", "%", u.BatteryPct, ut, pct, "model", u.Model)
f.add("ups.runtime", "UPS runtime", "duration", "min", u.RuntimeMin, ut, none, "model", u.Model)
f.add("ups.load", "UPS load", "power", "W", u.LoadW, ut, none, "model", u.Model)
f.add("ups.load_pct", "UPS load", "usage", "%", u.LoadPct, ut, pct, "model", u.Model)
f.add("ups.in_v", "UPS input", "voltage", "V", u.InV, ut, none, "model", u.Model)
f.add("ups.out_v", "UPS output", "voltage", "V", u.OutV, ut, none, "model", u.Model)
f.add("ups.on_battery", "UPS on battery", "state", "", u.OnBattery, ut, none, "model", u.Model)
f.add("ups.state", "UPS state", "state", "", u.State, ut, none, "model", u.Model)
f.add("ups.source", "UPS power source", "state", "", u.Source, ut, none, "model", u.Model)
f.add("ups.last_event", "UPS last event", "state", "", u.LastEvent, ut, none, "model", u.Model, "driver", u.Driver)
}
return f.out
}
// Sensors implements sensors.Provider.
func (c *Collector) Sensors() []sensors.Sensor {
s, ok := c.Latest()
if !ok {
return nil
}
return flatten(s, nil)
}
// SensorHistories implements sensors.Historian: the ring, seen through the sensors that
// match. A sensor that didn't exist in a sample (a disk plugged in since) simply has a
// shorter series; nothing is zero-filled. States have no curve and are left out.
func (c *Collector) SensorHistories(match func(id string) bool) []sensors.Series {
c.mu.RLock()
defer c.mu.RUnlock()
by := map[string]*sensors.Series{}
var order []string
for _, smp := range c.ring {
for _, x := range flatten(smp, match) {
v, ok := x.Value.(float64)
if !ok {
continue
}
s := by[x.ID]
if s == nil {
s = &sensors.Series{ID: x.ID, Unit: x.Unit, T: []int64{}, V: []float64{}}
by[x.ID] = s
order = append(order, x.ID)
}
s.T, s.V = append(s.T, smp.T), append(s.V, v)
}
}
out := make([]sensors.Series, 0, len(order))
for _, id := range order {
out = append(out, *by[id])
}
return out
}
internal/metrics/sensors_test.go
+56
View File
@@ -0,0 +1,56 @@
package metrics
import "testing"
func TestFlatten(t *testing.T) {
s := Sample{T: 1000, CPU: 10, Cores: []float64{5, 15}, Mem: Mem{Total: 200, Used: 50},
Temps: []Temp{{Chip: "nvme", Label: "Composite", C: 40, Crit: 85}, {Chip: "nvme", Label: "Composite", C: 45}},
Fans: []Fan{{Chip: "nct6775", Label: "fan2", RPM: 1100}},
FS: []FS{{Mount: "/", Size: 100, Used: 25}}, fsT: 900}
got := map[string]any{}
for _, x := range flatten(s, nil) {
if _, dup := got[x.ID]; dup {
t.Errorf("duplicate id %s", x.ID)
}
got[x.ID] = x.Value
}
for id, want := range map[string]any{"cpu.usage": 10.0, "cpu.core.1.usage": 15.0, "mem.used_pct": 25.0, "temp.nvme.composite": 40.0,
"temp.nvme.composite_2": 45.0, "fan.nct6775.fan2": 1100.0, "fs.root.used_pct": 25.0} {
if got[id] != want {
t.Errorf("%s = %v, want %v", id, got[id], want)
}
}
one := flatten(s, func(id string) bool { return id == "temp.nvme.composite_2" })
if len(one) != 1 || one[0].Value != 45.0 {
t.Errorf("only: got %+v", one)
}
for _, x := range flatten(s, func(id string) bool { return id == "fs.root.used" }) {
if x.T != 900 {
t.Errorf("fs sensor should carry the fs read time, got %d", x.T)
}
}
}
func TestSensorHistories(t *testing.T) {
c := NewCollector()
c.ring = []Sample{{T: 1, CPU: 10}, {T: 2, CPU: 20, Net: []Net{{Iface: "eth0", Up: true, RxBps: 5}}}}
got := map[string][]float64{}
for _, h := range c.SensorHistories(func(id string) bool { return id == "cpu.usage" || id == "net.eth0.rx" || id == "net.eth0.up" }) {
if len(h.T) != len(h.V) {
t.Errorf("%s: t and v differ in length", h.ID)
}
got[h.ID] = h.V
}
if len(got["cpu.usage"]) != 2 || got["cpu.usage"][1] != 20 {
t.Errorf("cpu: %v", got["cpu.usage"])
}
if len(got["net.eth0.rx"]) != 1 {
t.Errorf("a sensor that appeared later has a shorter history, got %v", got["net.eth0.rx"])
}
if _, ok := got["net.eth0.up"]; ok {
t.Error("a state has no history")
}
if len(c.SensorHistories(func(string) bool { return false })) != 0 {
t.Error("nothing matched, nothing returned")
}
}
internal/network/network.go
+360
View File
@@ -0,0 +1,360 @@
// Package network is what the Network tab shows: the machine's interfaces as the kernel has
// them, how the internet looks from out there, and a speed test that measures the line.
//
// Everything here is read from /proc and /sys, or measured against Cloudflare's public speed
// endpoints — no tool to install, on any of the machines. That matters for the router, where
// flash is small and `opkg install` is a decision.
package network
import (
"bufio"
"encoding/binary"
"fmt"
"net"
"os"
"sort"
"strconv"
"strings"
"sync"
"time"
)
// Addr is one address on an interface.
type Addr struct {
IP string `json:"ip"`
Prefix int `json:"prefix"` // network bits, as in /24
V6 bool `json:"v6,omitempty"`
}
// Iface is one network interface: what it is, whether it is up, and what it has moved.
type Iface struct {
Name string `json:"name"`
Kind string `json:"kind"` // ethernet | wifi | bridge | tunnel | loopback | virtual
MAC string `json:"mac,omitempty"`
MTU int `json:"mtu"`
// Up is the administrative state, Carrier whether a cable (or an association) is actually
// there. Up without Carrier is the usual "it's plugged into nothing".
Up bool `json:"up"`
Carrier bool `json:"carrier"`
// SpeedMbps is what the link negotiated, where the driver says: 0 for wireless, virtual
// and tunnel devices, which have no such number.
SpeedMbps int `json:"speed_mbps,omitempty"`
Duplex string `json:"duplex,omitempty"`
Driver string `json:"driver,omitempty"`
SignalDbm *float64 `json:"signal_dbm,omitempty"` // wireless only, from /proc/net/wireless
Addrs []Addr `json:"addrs"`
RxTotal uint64 `json:"rx_total"`
TxTotal uint64 `json:"tx_total"`
// Default marks the interface the default route leaves by — the one that matters when
// the question is about the internet.
Default bool `json:"default,omitempty"`
}
// Report is the machine's network as it stands.
type Report struct {
At time.Time `json:"at"`
Host string `json:"host,omitempty"`
Ifaces []Iface `json:"ifaces"`
Gateway string `json:"gateway,omitempty"`
GatewayVia string `json:"gateway_via,omitempty"` // the interface it is reached through
DNS []string `json:"dns,omitempty"`
Search []string `json:"search,omitempty"`
}
func readStr(path string) string {
b, err := os.ReadFile(path)
if err != nil {
return ""
}
return strings.TrimSpace(string(b))
}
func readInt(path string) int {
n, err := strconv.Atoi(readStr(path))
if err != nil {
return 0
}
return n
}
// kindOf names what sort of device this is, from what the kernel exposes about it rather
// than from its name — a bridge is a bridge whatever it is called.
func kindOf(name string, flags net.Flags) string {
sys := "/sys/class/net/" + name + "/"
switch {
case flags&net.FlagLoopback != 0:
return "loopback"
case dirExists(sys + "wireless"), dirExists(sys + "phy80211"):
return "wifi"
case dirExists(sys + "bridge"):
return "bridge"
case dirExists(sys + "bonding"):
return "bond"
}
// DEVTYPE says vlan, bridge, wlan… for the devices that have one
for _, line := range strings.Split(readStr(sys+"uevent"), "\n") {
if v, ok := strings.CutPrefix(line, "DEVTYPE="); ok && v != "" {
return v
}
}
if !dirExists(sys + "device") { // no bus device behind it: veth, tun, wireguard, docker's own
return "virtual"
}
return "ethernet"
}
func dirExists(p string) bool {
st, err := os.Stat(p)
return err == nil && st.IsDir()
}
// driverOf is the module behind an interface, which is what names the hardware in one word.
func driverOf(name string) string {
l, err := os.Readlink("/sys/class/net/" + name + "/device/driver")
if err != nil {
return ""
}
return l[strings.LastIndexByte(l, '/')+1:]
}
// wirelessSignal is the signal level of each wireless interface, in dBm.
func wirelessSignal() map[string]float64 {
out := map[string]float64{}
f, err := os.Open("/proc/net/wireless")
if err != nil {
return out
}
defer f.Close()
sc := bufio.NewScanner(f)
for sc.Scan() {
name, rest, ok := strings.Cut(sc.Text(), ":")
if !ok {
continue
}
fs := strings.Fields(rest)
if len(fs) < 3 {
continue
}
if v, err := strconv.ParseFloat(strings.TrimSuffix(fs[2], "."), 64); err == nil {
out[strings.TrimSpace(name)] = v
}
}
return out
}
// counters is every interface's byte totals, from the same file the sampler reads.
func counters() map[string][2]uint64 {
out := map[string][2]uint64{}
f, err := os.Open("/proc/net/dev")
if err != nil {
return out
}
defer f.Close()
sc := bufio.NewScanner(f)
for sc.Scan() {
name, rest, ok := strings.Cut(sc.Text(), ":")
if !ok {
continue
}
fs := strings.Fields(rest)
if len(fs) < 16 {
continue
}
rx, _ := strconv.ParseUint(fs[0], 10, 64)
tx, _ := strconv.ParseUint(fs[8], 10, 64)
out[strings.TrimSpace(name)] = [2]uint64{rx, tx}
}
return out
}
// defaultRoute is the gateway the machine sends the internet to, and the interface it
// leaves by. Read from /proc/net/route, whose addresses are little-endian hex.
func defaultRoute() (gw, via string) {
f, err := os.Open("/proc/net/route")
if err != nil {
return "", ""
}
defer f.Close()
sc := bufio.NewScanner(f)
best := -1
for sc.Scan() {
fs := strings.Fields(sc.Text())
if len(fs) < 8 || fs[0] == "Iface" {
continue
}
if fs[1] != "00000000" { // not a default route
continue
}
metric, _ := strconv.Atoi(fs[6])
if best >= 0 && metric >= best {
continue
}
ip, err := hexIP(fs[2])
if err != nil {
continue
}
best, gw, via = metric, ip, fs[0]
}
return gw, via
}
// hexIP reads one address out of /proc/net/route, where each is printed as a 32-bit word in
// the host's byte order: 10.0.0.1 appears as "0100000A" on a little-endian machine. So the
// text is parsed as a plain number, then written back out in native order to recover the four
// bytes — which is right on either sort of machine, without asking which this is.
func hexIP(h string) (string, error) {
v, err := strconv.ParseUint(h, 16, 32)
if err != nil || len(h) != 8 {
return "", fmt.Errorf("not an address: %q", h)
}
var b [4]byte
binary.NativeEndian.PutUint32(b[:], uint32(v))
return net.IPv4(b[0], b[1], b[2], b[3]).String(), nil
}
// resolvers is what /etc/resolv.conf points at.
func resolvers() (servers, search []string) {
f, err := os.Open("/etc/resolv.conf")
if err != nil {
return nil, nil
}
defer f.Close()
sc := bufio.NewScanner(f)
for sc.Scan() {
fs := strings.Fields(sc.Text())
if len(fs) < 2 || strings.HasPrefix(fs[0], "#") {
continue
}
switch fs[0] {
case "nameserver":
servers = append(servers, fs[1])
case "search", "domain":
search = append(search, fs[1:]...)
}
}
return servers, search
}
// Interfaces is every interface the kernel has, with its addresses and totals.
func Interfaces() []Iface {
list, err := net.Interfaces()
if err != nil {
return nil
}
_, via := defaultRoute()
sig, cnt := wirelessSignal(), counters()
out := make([]Iface, 0, len(list))
for _, in := range list {
sys := "/sys/class/net/" + in.Name + "/"
f := Iface{
Name: in.Name, Kind: kindOf(in.Name, in.Flags), MTU: in.MTU,
MAC: in.HardwareAddr.String(), Up: in.Flags&net.FlagUp != 0,
Driver: driverOf(in.Name), Default: in.Name == via, Addrs: []Addr{},
}
f.Carrier = readStr(sys+"carrier") == "1"
if f.Kind == "loopback" {
f.Carrier = f.Up
}
if sp := readInt(sys + "speed"); sp > 0 { // -1 when the driver doesn't know
f.SpeedMbps = sp
}
if d := readStr(sys + "duplex"); d != "" && d != "unknown" {
f.Duplex = d
}
if v, ok := sig[in.Name]; ok {
f.SignalDbm = &v
}
if c, ok := cnt[in.Name]; ok {
f.RxTotal, f.TxTotal = c[0], c[1]
}
addrs, _ := in.Addrs()
for _, a := range addrs {
n, ok := a.(*net.IPNet)
if !ok {
continue
}
ones, _ := n.Mask.Size()
f.Addrs = append(f.Addrs, Addr{IP: n.IP.String(), Prefix: ones, V6: n.IP.To4() == nil})
}
out = append(out, f)
}
// the interface the internet leaves by first, then the ones carrying addresses, then the rest
sort.SliceStable(out, func(i, j int) bool {
a, b := out[i], out[j]
if a.Default != b.Default {
return a.Default
}
ai, bi := len(a.Addrs) > 0 && a.Kind != "loopback", len(b.Addrs) > 0 && b.Kind != "loopback"
if ai != bi {
return ai
}
return a.Name < b.Name
})
return out
}
// Look is the machine's network now. Cheap: a handful of small files.
func Look(host string) Report {
gw, via := defaultRoute()
dns, search := resolvers()
return Report{At: time.Now(), Host: host, Ifaces: Interfaces(), Gateway: gw, GatewayVia: via, DNS: dns, Search: search}
}
// ---- how the internet sees this machine ----
// Outside is what the far end of the line says about this connection: the address the
// internet sees, and where it came ashore. Refreshed on its own clock, because the public
// address here is a DHCP lease and moves without warning.
type Outside struct {
At time.Time `json:"at"`
IP string `json:"ip,omitempty"`
Colo string `json:"colo,omitempty"` // the Cloudflare edge that answered — "SEA"
Country string `json:"country,omitempty"`
Error string `json:"error,omitempty"`
}
// Watcher keeps the outside view fresh and holds the speed tester.
type Watcher struct {
mu sync.RWMutex
out Outside
Test *Tester
}
const outsideEvery = 15 * time.Minute
func NewWatcher(onResult func(Result)) *Watcher {
w := &Watcher{Test: NewTester(onResult)}
go func() {
for {
w.refresh()
time.Sleep(outsideEvery)
}
}()
return w
}
func (w *Watcher) refresh() {
o := probeOutside()
w.mu.Lock()
// a failed probe does not throw away a good answer; it is noted beside it
if o.IP == "" && w.out.IP != "" {
w.out.Error, w.out.At = o.Error, o.At
} else {
w.out = o
}
w.mu.Unlock()
}
// Outside is the last known view from outside, asking again if it has gone stale.
func (w *Watcher) Outside() Outside {
w.mu.RLock()
o := w.out
w.mu.RUnlock()
if time.Since(o.At) > outsideEvery {
w.refresh()
w.mu.RLock()
o = w.out
w.mu.RUnlock()
}
return o
}
internal/network/network_test.go
+317
View File
@@ -0,0 +1,317 @@
package network
import (
"context"
"errors"
"io"
"math"
"net/http"
"net/http/httptest"
"os"
"sync/atomic"
"testing"
"time"
)
func TestHexIP(t *testing.T) {
// /proc/net/route prints each address as a host-order word: on a little-endian machine
// 10.0.0.1 is written "0100000A". These cases are that machine, which is all of ours.
for hex, want := range map[string]string{
"0100000A": "10.0.0.1",
"00000000": "0.0.0.0",
"0101A8C0": "192.168.1.1",
} {
got, err := hexIP(hex)
if err != nil || got != want {
t.Errorf("hexIP(%q) = %q, %v; want %q", hex, got, err, want)
}
}
if _, err := hexIP("nonsense"); err == nil {
t.Error("hexIP accepted nonsense")
}
}
func TestServerTimeIsNotLatency(t *testing.T) {
h := http.Header{}
h.Add("Server-Timing", "cfSpeedEdge;dur=7, cfSpeedWorker;dur=36")
h.Add("Server-Timing", `cfL4;desc="?proto=TCP&rtt=3505&min_rtt=2856"`)
if got := serverMs(h); got != 43 {
t.Errorf("serverMs = %v, want 43 (7+36, and nothing from the cfL4 description)", got)
}
if got := serverMs(http.Header{}); got != 0 {
t.Errorf("serverMs with no header = %v, want 0", got)
}
h2 := http.Header{"Cf-Ray": {"a41f6720f966a36c-SEA"}, "Cf-Meta-Ip": {"50.47.252.221"}}
if m := readMeta(h2); m.Colo != "SEA" || m.IP != "50.47.252.221" {
t.Errorf("readMeta = %+v, want the edge SEA and the address", m)
}
}
func TestRates(t *testing.T) {
// a gibibyte in eight seconds is 1073.7 Mbps
if got := mbps(1<<30, 8); math.Abs(got-1073.7) > 0.1 {
t.Errorf("mbps(1GiB, 8s) = %v, want ~1073.7", got)
}
if got := best([]float64{12.5, 9.2, 40, 9.9}); got != 9.2 {
t.Errorf("best = %v, want 9.2 (the least of them)", got)
}
if got := jitter([]float64{10, 10, 10}); got != 0 {
t.Errorf("jitter of a steady line = %v, want 0", got)
}
if got := jitter([]float64{10}); got != 0 {
t.Errorf("jitter of one sample = %v, want 0", got)
}
if got := jitter([]float64{5, 50, 5, 50}); got != 45 {
t.Errorf("jitter of a line alternating by 45 = %v, want 45", got)
}
// one slow probe among steady ones moves it a little, not enormously
if got := jitter([]float64{10, 10, 10, 60, 10, 10}); got > 25 {
t.Errorf("jitter with a single outlier = %v, want it not to run away", got)
}
}
// The body an upload sends is a fixed size, counts itself, and costs nothing to make.
func TestZerosBodyCountsItself(t *testing.T) {
var n atomic.Int64
z := &zeros{left: 100, n: &n}
buf := make([]byte, 30)
total := 0
for {
got, err := z.Read(buf)
total += got
if err != nil {
break
}
}
if total != 100 || n.Load() != 100 {
t.Errorf("read %d bytes and counted %d, want 100 and 100", total, n.Load())
}
}
// Interfaces has to work on whatever machine this runs on, and always name a loopback.
func TestInterfacesHasLoopback(t *testing.T) {
ifs := Interfaces()
if len(ifs) == 0 {
t.Fatal("no interfaces at all")
}
lo := false
for _, f := range ifs {
if f.Kind == "loopback" {
lo = true
if !f.Up || len(f.Addrs) == 0 {
t.Errorf("loopback %s looks wrong: up=%v addrs=%v", f.Name, f.Up, f.Addrs)
}
}
if f.Name == "" {
t.Error("an interface with no name")
}
}
if !lo {
t.Error("no loopback among the interfaces")
}
}
// The real thing, against the real endpoint: skipped by default, because `make test` must
// not need the internet. HAMLAB_NET_TEST=1 go test ./internal/network/ -run Live -v
func TestLiveSpeedTest(t *testing.T) {
if os.Getenv("HAMLAB_NET_TEST") == "" {
t.Skip("set HAMLAB_NET_TEST=1 to measure the actual line")
}
tr := NewTester(nil)
if _, err := tr.Start(); err != nil {
t.Fatal(err)
}
if _, err := tr.Start(); err == nil {
t.Error("a second test started while the first was running")
}
deadline := time.Now().Add(90 * time.Second)
for tr.Running() && time.Now().Before(deadline) {
time.Sleep(250 * time.Millisecond)
}
st := tr.State()
if st.Result == nil {
t.Fatal("no result")
}
r := *st.Result
t.Logf("down %s up %s latency %.1f ms jitter %.1f ms via %s ip %s %.0f MB moved in %.1fs",
Mbits(r.DownMbps), Mbits(r.UpMbps), r.LatencyMs, r.JitterMs, r.Colo, r.IP,
float64(r.DownBytes+r.UpBytes)/1e6, r.TookSec)
if r.Error != "" {
t.Fatalf("test failed: %s", r.Error)
}
if r.DownMbps <= 0 || r.UpMbps <= 0 || r.LatencyMs <= 0 {
t.Errorf("a figure came back zero: %+v", r)
}
if r.IP == "" {
t.Error("no public address reported")
}
if len(tr.History()) != 1 {
t.Errorf("history has %d results, want 1", len(tr.History()))
}
}
// A phase measured against a local server, at a rate the test controls. This is the part of
// the speed test with arithmetic in it — the window after the ramp, the byte cap that ends it
// early, and the rate that comes out — and none of it needs the internet to check.
func TestPhaseMeasuresTheWindow(t *testing.T) {
const perTick = 1 << 20 // a mebibyte every 5 ms ≈ 1678 Mbps, whatever the machine is doing
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/up" {
_, _ = io.Copy(io.Discard, r.Body)
w.WriteHeader(204)
return
}
buf := make([]byte, perTick)
fl, _ := w.(http.Flusher)
for {
if _, err := w.Write(buf); err != nil {
return
}
if fl != nil {
fl.Flush()
}
select {
case <-time.After(5 * time.Millisecond):
case <-r.Context().Done():
return
}
}
}))
defer srv.Close()
defer func(up string, w, d, mw time.Duration, m int64) {
upURL, warmUp, downFor, minWindow, measureBytes = up, w, d, mw, m
}(upURL, warmUp, downFor, minWindow, measureBytes)
upURL = srv.URL + "/up"
warmUp, downFor, minWindow, measureBytes = 100*time.Millisecond, 3*time.Second, 200*time.Millisecond, 8<<20
here := Source{"local", func(int) string { return srv.URL + "/down" }}
tr := NewTester(nil)
began := time.Now()
mbps, moved, note, err := tr.phase(context.Background(), srv.Client(), "download", 2, downFor, 0, 1, here)
took := time.Since(began)
if err != nil {
t.Fatalf("phase: %v", err)
}
if note != "" {
t.Errorf("note = %q, want none — nothing went wrong", note)
}
// The cap must have ended it early rather than the clock: the phase was allowed three
// seconds and should be done in well under one, having moved at least the cap.
if moved < measureBytes {
t.Errorf("moved %d bytes, want at least the %d-byte cap", moved, measureBytes)
}
if took > time.Second {
t.Errorf("phase took %v of the %v it was allowed; the byte cap should have ended it", took.Round(time.Millisecond), downFor)
}
// The server hands out a mebibyte per 5 ms per stream, so the rate is in the hundreds of
// Mbps. The point is that it is a real quotient of what moved over how long, not that it
// hits a particular figure on a loaded test machine.
if mbps < 50 || mbps > 20000 {
t.Errorf("mbps = %v, which is not a believable quotient", mbps)
}
t.Logf("measured %.0f Mbps over %.1f MiB", mbps, float64(moved)/(1<<20))
// A phase whose server refuses outright fails, and says what it was told.
bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(429) }))
defer bad.Close()
refuser := Source{"refuser", func(int) string { return bad.URL + "/down" }}
_, _, _, err = tr.phase(context.Background(), bad.Client(), "download", 2, 300*time.Millisecond, 0, 1, refuser)
if err == nil {
t.Error("a phase against a server that refuses everything should fail")
} else if !refused(err) {
t.Errorf("error = %q, want the refusal that makes the fallback kick in", err)
}
}
// Stopping a test mid-flight is reported as cancelled, not as whatever the streams said on
// their way out.
func TestCancelIsReportedAsCancelled(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
buf := make([]byte, 1<<16)
for {
if _, err := w.Write(buf); err != nil {
return
}
select {
case <-time.After(time.Millisecond):
case <-r.Context().Done():
return
}
}
}))
defer srv.Close()
defer func(w, d time.Duration) { warmUp, downFor = w, d }(warmUp, downFor)
warmUp, downFor = 10*time.Millisecond, 10*time.Second
here := Source{"local", func(int) string { return srv.URL + "/down" }}
tr := NewTester(nil)
ctx, cancel := context.WithCancel(context.Background())
go func() { time.Sleep(200 * time.Millisecond); cancel() }()
_, _, _, err := tr.phase(ctx, srv.Client(), "download", 1, downFor, 0, 1, here)
// Cancelled after bytes moved: the figure stands, and the phase does not claim an error.
if err != nil && !errors.Is(err, context.Canceled) {
t.Errorf("err = %v, want nil or a cancellation", err)
}
}
// When the first source refuses, the download is taken from the next one and the result says
// so. This is the whole reason there is more than one.
func TestDownloadFallsBackWhenRefused(t *testing.T) {
serving := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/up" {
_, _ = io.Copy(io.Discard, r.Body)
w.WriteHeader(204)
return
}
// paced, so the phase is governed by its window and not by a loopback moving a
// gibibyte before the first tick
buf := make([]byte, 1<<20)
fl, _ := w.(http.Flusher)
for i := 0; i < 64; i++ {
if _, err := w.Write(buf); err != nil {
return
}
if fl != nil {
fl.Flush()
}
select {
case <-time.After(2 * time.Millisecond):
case <-r.Context().Done():
return
}
}
}))
defer serving.Close()
refusing := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(429) }))
defer refusing.Close()
defer func(srcs []Source, up string, w, d, mw time.Duration, m int64) {
downSources, upURL, warmUp, downFor, minWindow, measureBytes = srcs, up, w, d, mw, m
}(downSources, upURL, warmUp, downFor, minWindow, measureBytes)
downSources = []Source{
{"refuser", func(int) string { return refusing.URL + "/down" }},
{"the-other-one", func(int) string { return serving.URL + "/down" }},
}
upURL = serving.URL + "/up"
warmUp, downFor, upFor, minWindow, measureBytes = 50*time.Millisecond, time.Second, 300*time.Millisecond, 100*time.Millisecond, 4<<20
tr := NewTester(nil)
if _, err := tr.Start(); err != nil {
t.Fatal(err)
}
deadline := time.Now().Add(30 * time.Second)
for tr.Running() && time.Now().Before(deadline) {
time.Sleep(50 * time.Millisecond)
}
res := tr.State().Result
if res == nil {
t.Fatal("no result")
}
if res.Source != "the-other-one" {
t.Errorf("source = %q, want the second one — the first refused", res.Source)
}
if res.DownMbps <= 0 {
t.Errorf("download = %v, want a figure from the source that did serve: %+v", res.DownMbps, res)
}
}
internal/network/speedtest.go
+687
View File
@@ -0,0 +1,687 @@
package network
// The speed test. It measures against Cloudflare's public speed endpoints, the same ones
// speed.cloudflare.com uses: a plain GET of N bytes down, a plain POST of N bytes up, and a
// zero-byte GET for the round trip. No account, no API key, nothing to install, and an edge
// close to every one of these machines.
//
// The method, since a number without one means little — it is stated in the UI too:
//
// - Latency is the round trip of a zero-byte response on a connection that is already open,
// less the time the far end says it spent on the request (its Server-Timing header). That
// subtraction is the difference between measuring the line and measuring Cloudflare's
// thinking time: a zero-byte response is almost entirely the latter — tens of
// milliseconds of worker, against a few of actual wire. The figure reported is the best of
// several probes; jitter is the average change from one to the next.
// - Download and upload each run several streams at once for a fixed few seconds. One
// stream measures one TCP connection's congestion window as much as the line, which is
// why every speed test ever made uses more than one.
// - The first second of each phase is thrown away. That is the ramp: counting it reports a
// number nobody's connection ever actually did.
// - Each phase is capped in bytes as well as seconds, so a very fast line cannot run away
// with the meter.
//
// The one thing to know about the far end: it rate-limits by volume, and a test moves a few
// hundred megabytes. Several in a row will start earning 429s, which is why there is a
// cooldown between tests and why a phase that gets refused says so instead of reporting the
// decayed figure it would otherwise compute.
//
// HTTP/2 is deliberately off: over one HTTP/2 connection the streams would share a single
// congestion window, which measures the window, not the line.
import (
"context"
"crypto/tls"
"errors"
"fmt"
"io"
"log"
"math"
"net"
"net/http"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
)
// The endpoints, and the shape of a phase. Variables rather than constants only so the tests
// can point them at a local server and shrink the windows; nothing else rebinds them.
// Source is somewhere to pull bytes from. Cloudflare's endpoint is the first choice because
// it is anycast — near wherever Ham Lab is running — and because the same host answers for
// upload and for the trace. The fallback exists because that endpoint rate-limits by volume
// and will refuse a connection that has been tested hard: a plain 100 MB file on a public
// mirror has no such opinion. It is in Seattle, so on a machine far from there it will read
// low, which is exactly why every result names the source it used.
type Source struct {
Name string
// URL takes the number of bytes wanted, where the source can be asked; a fixed file
// ignores it and is fetched again each time round.
URL func(bytes int) string
}
var downSources = []Source{
{"cloudflare", func(b int) string { return fmt.Sprintf(cfDownURL, b) }},
{"linode-seattle", func(int) string { return "https://speedtest.seattle.linode.com/100MB-seattle.bin" }},
}
var (
cfDownURL = "https://speed.cloudflare.com/__down?bytes=%d"
upURL = "https://speed.cloudflare.com/__up"
// traceURL is Cloudflare's standard trace: a few lines of key=value naming the address it
// sees, the edge that answered and the country. A few hundred bytes.
traceURL = "https://speed.cloudflare.com/cdn-cgi/trace"
warmUp = time.Second
downFor = 4 * time.Second
upFor = 4 * time.Second
// measureBytes ends a phase early once this much has moved inside the window. A fast line
// says what it is in well under the time allowed, and stopping there is what keeps a test
// down to a couple of hundred megabytes — which is the difference between the far end
// serving this happily and rate-limiting it. A slow line never reaches it and is timed out
// by downFor/upFor instead, which is the honest way round.
measureBytes = int64(64) << 20
// minWindow is how long the window must last whatever the byte cap says. Without it a very
// fast line fills the cap in a few hundredths of a second and the phase would be over
// before there was anything to divide by.
minWindow = time.Second
)
const (
latencyProbes = 8
downStreams = 4
upStreams = 3
downChunk = 25 << 20 // bytes per download request; the far end refuses much more
upChunk = 25 << 20 // and per upload request
// phaseBudget is the runaway guard, counting the ramp too: nothing measures forever.
phaseBudget = int64(512) << 20
httpTimeout = 90 * time.Second
// Cooldown is how long the line is left alone between tests. The far end rate-limits by
// volume, and two back-to-back tests are the surest way to be refused by it.
Cooldown = 20 * time.Second
// Method is what the UI tells the reader about the numbers it is showing.
Method = "Cloudflare's speed endpoints · 4 streams down, 3 up · a 1 s ramp, then up to 4 s or 64 MiB each way"
)
// Result is one finished speed test.
type Result struct {
At time.Time `json:"at"`
DownMbps float64 `json:"down_mbps"`
UpMbps float64 `json:"up_mbps"`
LatencyMs float64 `json:"latency_ms"`
JitterMs float64 `json:"jitter_ms"`
DownBytes int64 `json:"down_bytes"`
UpBytes int64 `json:"up_bytes"`
TookSec float64 `json:"took_sec"`
// Colo is the Cloudflare edge that answered — "SEA" — and IP is this connection's public
// address as that edge saw it.
Colo string `json:"colo,omitempty"`
IP string `json:"ip,omitempty"`
// Source is where the download came from, which matters because the fallback is not in the
// same place as the anycast one. Empty means the usual.
Source string `json:"source,omitempty"`
// Note is set when the figures are real but something went wrong along the way — a stream
// that was refused part-way, most often. The number is what was actually seen; the note is
// why it may be low.
Note string `json:"note,omitempty"`
Error string `json:"error,omitempty"`
}
// Run is a test in progress, or the last one to have finished.
type Run struct {
Started time.Time `json:"started"`
Running bool `json:"running"`
// Phase is where it has got to: latency, download, upload, done.
Phase string `json:"phase"`
// Done is how far through the whole test, 0–1, so one bar can show all of it.
Done float64 `json:"done"`
// Mbps is the live figure for the phase under way — what the needle shows.
Mbps float64 `json:"mbps"`
Bytes int64 `json:"bytes"`
Result *Result `json:"result,omitempty"`
}
// Tester runs one speed test at a time and remembers what the last ones said.
type Tester struct {
mu sync.RWMutex
run Run
hist []Result
// lastTry is when a test was last *started*, which is what the cooldown counts from — a
// test that was refused still used the line, and hammering retries at an endpoint that has
// just rate-limited us is the one thing guaranteed not to help.
lastTry time.Time
cancel context.CancelFunc
onDone func(Result)
maxHist int
}
func NewTester(onDone func(Result)) *Tester {
return &Tester{onDone: onDone, maxHist: 50}
}
// Load takes the results kept from before a restart, oldest first.
func (t *Tester) Load(rs []Result) {
t.mu.Lock()
defer t.mu.Unlock()
t.hist = append(t.hist[:0], rs...)
if n := len(t.hist); n > 0 {
last := t.hist[n-1]
t.run = Run{Started: last.At, Phase: "done", Done: 1, Result: &last}
t.lastTry = last.At
}
}
func (t *Tester) State() Run {
t.mu.RLock()
defer t.mu.RUnlock()
return t.run
}
func (t *Tester) History() []Result {
t.mu.RLock()
defer t.mu.RUnlock()
return append([]Result{}, t.hist...)
}
func (t *Tester) Running() bool {
t.mu.RLock()
defer t.mu.RUnlock()
return t.run.Running
}
// Cancel stops a test in progress. Reports whether there was one.
func (t *Tester) Cancel() bool {
t.mu.Lock()
c := t.cancel
t.mu.Unlock()
if c == nil {
return false
}
c()
return true
}
var ErrBusy = errors.New("a speed test is already running")
// Start begins a test in the background and returns its opening state.
func (t *Tester) Start() (Run, error) {
t.mu.Lock()
if t.run.Running {
r := t.run
t.mu.Unlock()
return r, ErrBusy
}
if since := time.Since(t.lastTry); since < Cooldown && !t.lastTry.IsZero() {
r := t.run
t.mu.Unlock()
return r, fmt.Errorf("give the line %d more seconds — testing back to back is what gets this connection rate-limited", int((Cooldown-since).Seconds())+1)
}
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
t.cancel = cancel
t.lastTry = time.Now()
t.run = Run{Started: t.lastTry, Running: true, Phase: "latency"}
r := t.run
t.mu.Unlock()
go t.measure(ctx, cancel)
return r, nil
}
func (t *Tester) set(fn func(*Run)) {
t.mu.Lock()
fn(&t.run)
t.mu.Unlock()
}
func (t *Tester) measure(ctx context.Context, cancel context.CancelFunc) {
defer cancel()
started := time.Now()
res := Result{At: started}
tr := newTransport(downStreams)
defer tr.CloseIdleConnections()
client := &http.Client{Transport: tr, Timeout: httpTimeout}
// latency, and the meta the same responses carry
lat, meta, err := t.latency(ctx, client)
if err == nil {
res.LatencyMs, res.JitterMs = best(lat), jitter(lat)
res.Colo, res.IP = meta.Colo, meta.IP
}
var note string
if err == nil {
t.set(func(r *Run) { r.Phase, r.Done, r.Mbps = "download", 0.1, 0 })
// Each source in turn: a refusal from the first is exactly what the second is for, and
// anything else (no route, no DNS) would fail the same way twice, so it stops there.
for i, src := range downSources {
res.Source = src.Name
res.DownMbps, res.DownBytes, note, err = t.phase(ctx, client, "download", downStreams, downFor, 0.1, 0.6, src)
if err == nil || !refused(err) || i == len(downSources)-1 {
break
}
log.Printf("speed test: %s refused the download, trying %s", src.Name, downSources[i+1].Name)
}
res.Note = join(res.Note, note)
}
if err == nil {
t.set(func(r *Run) { r.Phase, r.Done, r.Mbps = "upload", 0.7, 0 })
res.UpMbps, res.UpBytes, note, err = t.phase(ctx, client, "upload", upStreams, upFor, 0.7, 1.0, Source{Name: "cloudflare"})
res.Note = join(res.Note, note)
}
res.TookSec = round(time.Since(started).Seconds(), 1)
if err != nil {
if ctx.Err() != nil {
res.Error = "cancelled"
} else {
res.Error = err.Error()
}
}
t.mu.Lock()
t.cancel = nil
t.run = Run{Started: started, Phase: "done", Done: 1, Result: &res, Bytes: res.DownBytes + res.UpBytes}
if res.Error == "" || res.DownMbps > 0 { // a cancelled test that got a number is still a reading
t.hist = append(t.hist, res)
if len(t.hist) > t.maxHist {
t.hist = t.hist[len(t.hist)-t.maxHist:]
}
}
done, out := t.onDone, res
t.mu.Unlock()
if done != nil {
done(out)
}
}
// meta is what the edge says about the connection. The address comes from a header of the
// speed endpoint's own; the edge's name is the tail of the ray ID, which every Cloudflare
// response carries ("a41f6720f966a36c-SEA").
type meta struct{ IP, Colo string }
func readMeta(h http.Header) meta {
m := meta{IP: h.Get("cf-meta-ip")}
if ray := h.Get("cf-ray"); ray != "" {
if i := strings.LastIndexByte(ray, '-'); i >= 0 {
m.Colo = ray[i+1:]
}
}
return m
}
// serverMs is how long the far end says it spent on this request, from Server-Timing. The
// cfL4 entry carries no dur= and contributes nothing, which is what we want — it is a
// description of the connection, not time spent.
func serverMs(h http.Header) float64 {
var total float64
for _, v := range h.Values("Server-Timing") {
for _, part := range strings.Split(v, ",") {
_, after, ok := strings.Cut(part, "dur=")
if !ok {
continue
}
end := 0
for end < len(after) && (after[end] == '.' || (after[end] >= '0' && after[end] <= '9')) {
end++
}
if f, err := strconv.ParseFloat(after[:end], 64); err == nil {
total += f
}
}
}
return total
}
// latency times several zero-byte responses on an already-open connection.
func (t *Tester) latency(ctx context.Context, client *http.Client) ([]float64, meta, error) {
var m meta
var out []float64
for i := 0; i <= latencyProbes; i++ { // the first is the handshake, and is not counted
start := time.Now()
req, err := http.NewRequestWithContext(ctx, "GET", fmt.Sprintf(cfDownURL, 0), nil)
if err != nil {
return nil, m, err
}
resp, err := client.Do(req)
if err != nil {
return nil, m, err
}
_, _ = io.Copy(io.Discard, resp.Body)
resp.Body.Close()
if resp.StatusCode != 200 {
return nil, m, fmt.Errorf("speed endpoint answered %s", resp.Status)
}
if i == 0 {
m = readMeta(resp.Header)
continue
}
rtt := float64(time.Since(start).Microseconds())/1000 - serverMs(resp.Header)
if rtt < 0.01 { // the far end claimed more time than the whole request took
rtt = 0.01
}
out = append(out, rtt)
t.set(func(r *Run) { r.Done = 0.02 + 0.08*float64(i)/latencyProbes })
}
if len(out) == 0 {
return nil, m, errors.New("no reply from the speed endpoint")
}
return out, m, nil
}
// phase pushes traffic with several streams for a fixed spell and reports the rate over the
// window after the ramp. from and to are where this phase sits in the whole test's progress.
//
// The window ends when the traffic does, not when the clock runs out: if the far end starts
// refusing part-way, the figure is what was actually moved while it was moving, and the note
// says what happened. Coasting to the end of the clock with no bytes arriving would report a
// number a third of the truth.
func (t *Tester) phase(parent context.Context, client *http.Client, dir string, streams int, forHow time.Duration, from, to float64, src Source) (float64, int64, string, error) {
// parent is the test's own context, which is cancelled when somebody presses Stop. ctx is
// this phase's, which *this function* cancels to shut the streams down — so the two are
// kept apart: asking ctx whether the test was cancelled would always say yes by then, and
// every failure would be reported as "context canceled" whatever really happened.
ctx, stop := context.WithCancel(parent)
defer stop()
var moved atomic.Int64
var firstErr error
var errOnce sync.Once
var alive atomic.Int32
alive.Store(int32(streams))
dead := make(chan struct{})
var deadOnce sync.Once
for i := 0; i < streams; i++ {
go func() {
defer func() {
if alive.Add(-1) == 0 {
deadOnce.Do(func() { close(dead) })
}
}()
tries := 0
for ctx.Err() == nil && moved.Load() < phaseBudget {
var err error
if dir == "download" {
err = stream(ctx, client, src, &moved)
} else {
err = upstream(ctx, client, &moved)
}
if err == nil {
tries = 0
continue
}
if ctx.Err() != nil {
return
}
errOnce.Do(func() { firstErr = err })
if tries++; tries > 2 { // refused three times: this stream is finished
return
}
select {
case <-time.After(400 * time.Millisecond):
case <-ctx.Done():
return
}
}
}()
}
// the ramp, measured and thrown away
select {
case <-time.After(warmUp):
case <-dead:
case <-ctx.Done():
}
mark, at := moved.Load(), time.Now()
last, lastAt := mark, at // the last time bytes were still arriving
deadline := at.Add(forHow)
tick := time.NewTicker(200 * time.Millisecond)
defer tick.Stop()
for time.Now().Before(deadline) {
select {
case <-tick.C:
case <-dead:
case <-ctx.Done():
}
if parent.Err() != nil {
break
}
now := moved.Load()
if now > last {
last, lastAt = now, time.Now()
}
// enough has moved to know, and it has been long enough to mean it
if (now-mark >= measureBytes && time.Since(at) >= minWindow) || now >= phaseBudget {
break
}
if secs := time.Since(at).Seconds(); secs > 0 {
live := mbps(now-mark, secs)
part := math.Min(1, secs/forHow.Seconds())
t.set(func(r *Run) { r.Mbps, r.Bytes, r.Done = live, now, from+(to-from)*part })
}
// every stream has given up and nothing more is coming
if alive.Load() == 0 {
break
}
}
total, secs := last-mark, lastAt.Sub(at).Seconds()
stop()
if total <= 0 {
switch {
case firstErr != nil: // what the far end said explains it better than anything else
return 0, moved.Load(), "", firstErr
case parent.Err() != nil:
return 0, moved.Load(), "", parent.Err()
}
return 0, moved.Load(), "", fmt.Errorf("nothing moved in the %s phase", dir)
}
if secs < 0.02 { // too short a window to divide by; the guard is against dividing by zero,
// not a judgement about how long a good measurement takes
return 0, moved.Load(), "", fmt.Errorf("the %s phase was over before it could be measured", dir)
}
note := ""
if firstErr != nil {
note = fmt.Sprintf("the %s phase was cut short: %s", dir, firstErr)
}
return mbps(total, secs), moved.Load(), note, nil
}
// join puts two notes together without leaving an empty one dangling.
func join(a, b string) string {
switch {
case a == "":
return b
case b == "":
return a
}
return a + "; " + b
}
// stream fetches one chunk from src, counting bytes as they arrive.
func stream(ctx context.Context, client *http.Client, src Source, n *atomic.Int64) error {
req, err := http.NewRequestWithContext(ctx, "GET", src.URL(downChunk), nil)
if err != nil {
return err
}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return answered(resp.StatusCode, resp.Status)
}
buf := make([]byte, 64<<10)
for {
got, err := resp.Body.Read(buf)
if got > 0 {
n.Add(int64(got))
}
if err != nil {
if err == io.EOF {
return nil
}
return err
}
}
}
// upstream posts one chunk of nothing in particular, counting bytes as they go out.
func upstream(ctx context.Context, client *http.Client, n *atomic.Int64) error {
body := &zeros{left: upChunk, n: n}
req, err := http.NewRequestWithContext(ctx, "POST", upURL, body)
if err != nil {
return err
}
req.ContentLength = upChunk
req.Header.Set("Content-Type", "application/octet-stream")
resp, err := client.Do(req)
if err != nil {
return err
}
_, _ = io.Copy(io.Discard, resp.Body)
resp.Body.Close()
if resp.StatusCode != 200 && resp.StatusCode != 204 {
return answered(resp.StatusCode, resp.Status)
}
return nil
}
// answered turns the far end's refusal into something worth reading. It rate-limits by
// volume: enough testing in a short while and it stops serving the big transfers, which is
// the only failure of this test anyone here is likely to meet.
func answered(code int, status string) error {
if code == 429 || code == 403 {
return errRefused
}
return fmt.Errorf("the speed endpoint answered %s", status)
}
// errRefused is the far end rate-limiting this connection, which is the only failure of this
// test anyone here is likely to meet: it limits by volume, and a test moves a few hundred
// megabytes. It is recognisable so the download phase can try its other source.
var errRefused = errors.New("the speed endpoint is rate-limiting this connection — it does that after a lot of testing; leave it a few minutes")
func refused(err error) bool { return errors.Is(err, errRefused) }
// zeros is a request body of a fixed size that costs nothing to produce, counting itself as
// it is read out onto the wire.
type zeros struct {
left int
n *atomic.Int64
}
func (z *zeros) Read(p []byte) (int, error) {
if z.left <= 0 {
return 0, io.EOF
}
if len(p) > z.left {
p = p[:z.left]
}
for i := range p {
p[i] = 0
}
z.left -= len(p)
z.n.Add(int64(len(p)))
return len(p), nil
}
func newTransport(streams int) *http.Transport {
return &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext,
TLSHandshakeTimeout: 10 * time.Second,
MaxIdleConns: streams * 2,
MaxIdleConnsPerHost: streams * 2,
// An empty map turns HTTP/2 off: see the note at the top of this file.
TLSNextProto: map[string]func(string, *tls.Conn) http.RoundTripper{},
}
}
// probeOutside asks the far end who it thinks we are. One small request.
func probeOutside() Outside {
tr := newTransport(1)
defer tr.CloseIdleConnections()
client := &http.Client{Transport: tr, Timeout: 15 * time.Second}
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, "GET", traceURL, nil)
if err != nil {
return Outside{At: time.Now(), Error: err.Error()}
}
resp, err := client.Do(req)
if err != nil {
return Outside{At: time.Now(), Error: shortErr(err)}
}
defer resp.Body.Close()
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
if resp.StatusCode != 200 {
return Outside{At: time.Now(), Error: "the trace endpoint answered " + resp.Status}
}
o := Outside{At: time.Now()}
for _, line := range strings.Split(string(body), "\n") {
k, v, ok := strings.Cut(line, "=")
if !ok {
continue
}
switch k {
case "ip":
o.IP = v
case "colo":
o.Colo = v
case "loc":
o.Country = v
}
}
if o.IP == "" {
o.Error = "the trace endpoint said nothing about this connection"
}
return o
}
// shortErr keeps the part of a network error a person can act on.
func shortErr(err error) string {
s := err.Error()
if i := strings.LastIndex(s, ": "); i > 0 && len(s)-i < 60 {
return s[i+2:]
}
return s
}
func mbps(bytes int64, secs float64) float64 {
return round(float64(bytes)*8/secs/1e6, 1)
}
func best(v []float64) float64 {
m := v[0]
for _, x := range v {
if x < m {
m = x
}
}
return round(m, 1)
}
// jitter is how much the line wobbles: the average change from one round trip to the next,
// which is the usual definition and, unlike a standard deviation, is not run away with by a
// single slow probe.
func jitter(v []float64) float64 {
if len(v) < 2 {
return 0
}
var sum float64
for i := 1; i < len(v); i++ {
sum += math.Abs(v[i] - v[i-1])
}
return round(sum/float64(len(v)-1), 1)
}
func round(v float64, places int) float64 {
p := math.Pow(10, float64(places))
return math.Round(v*p) / p
}
// Mbits renders a rate the way the line is sold, for a log line.
func Mbits(v float64) string { return strconv.FormatFloat(v, 'f', 1, 64) + " Mbps" }
internal/nodes/nodes.go
+464
View File
@@ -0,0 +1,464 @@
// Package nodes is the hub side of multi-host: the list of managed Ham Labs, their
// health, adding one (pair with its password, receive a token, pin its certificate),
// removing one (revoke the token), and the reverse proxy that turns
// /api/nodes/<name>/… into /api/… on that node, WebSockets included.
package nodes
import (
"bytes"
"context"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/http/httputil"
"net/url"
"os"
"regexp"
"sort"
"strings"
"sync"
"time"
"hamlab/internal/config"
"hamlab/internal/tlsauto"
)
var nameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,31}$`)
type Status struct {
Name string `json:"name"`
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Added time.Time `json:"added"`
Online bool `json:"online"`
Host string `json:"host,omitempty"`
Version string `json:"version,omitempty"`
Load string `json:"load,omitempty"`
Uptime float64 `json:"uptime,omitempty"`
RebootNeed bool `json:"reboot_needed,omitempty"`
LastSeen time.Time `json:"last_seen,omitempty"`
Error string `json:"error,omitempty"`
}
type Manager struct {
cfg *config.Config
hubName string
mu sync.Mutex
status map[string]*Status
proxies map[string]*httputil.ReverseProxy
clients map[string]*http.Client
}
func New(cfg *config.Config, hubName string) *Manager {
return &Manager{cfg: cfg, hubName: hubName, status: map[string]*Status{}, proxies: map[string]*httputil.ReverseProxy{}, clients: map[string]*http.Client{}}
}
func transportFor(n config.Node) *http.Transport {
t := &http.Transport{
DialContext: (&net.Dialer{Timeout: 5 * time.Second}).DialContext,
TLSHandshakeTimeout: 5 * time.Second,
ResponseHeaderTimeout: 60 * time.Second,
MaxIdleConnsPerHost: 8,
}
if strings.HasPrefix(n.URL, "https://") {
if n.Fingerprint != "" {
t.TLSClientConfig = tlsauto.PinnedConfig(n.Fingerprint)
} else {
t.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
}
}
return t
}
func (m *Manager) client(n config.Node) *http.Client {
m.mu.Lock()
defer m.mu.Unlock()
if c, ok := m.clients[n.Name]; ok {
return c
}
c := &http.Client{Transport: transportFor(n), Timeout: 15 * time.Second}
m.clients[n.Name] = c
return c
}
func (m *Manager) node(name string) (config.Node, bool) {
var found config.Node
ok := false
m.cfg.View(func(c *config.Config) {
for _, n := range c.Nodes {
if n.Name == name {
found, ok = n, true
}
}
})
return found, ok
}
// ---- probing ----
func (m *Manager) probe(ctx context.Context, n config.Node) *Status {
st := &Status{Name: n.Name, URL: n.URL, Fingerprint: n.Fingerprint, Added: n.Added}
req, _ := http.NewRequestWithContext(ctx, "GET", strings.TrimRight(n.URL, "/")+"/api/me", nil)
req.Header.Set("Authorization", "Bearer "+n.Token)
resp, err := m.client(n).Do(req)
if err != nil {
st.Error = shortErr(err)
return st
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
st.Error = fmt.Sprintf("HTTP %d", resp.StatusCode)
if resp.StatusCode == 401 {
st.Error = "token rejected (re-add the machine)"
}
return st
}
var me struct {
Host, Version, Load string
Uptime float64
RebootNeeded bool `json:"reboot_needed"`
}
if json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&me) != nil {
st.Error = "bad reply"
return st
}
st.Online, st.Host, st.Version, st.Load, st.Uptime, st.RebootNeed, st.LastSeen = true, me.Host, me.Version, me.Load, me.Uptime, me.RebootNeeded, time.Now()
return st
}
func shortErr(err error) string {
s := err.Error()
if i := strings.LastIndex(s, ": "); i >= 0 {
s = s[i+2:]
}
return s
}
// Run refreshes every node's status periodically.
func (m *Manager) Run(ctx context.Context) {
m.refresh(ctx)
t := time.NewTicker(15 * time.Second)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
m.refresh(ctx)
}
}
}
func (m *Manager) refresh(ctx context.Context) {
var list []config.Node
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
var wg sync.WaitGroup
for _, n := range list {
wg.Add(1)
go func(n config.Node) {
defer wg.Done()
pctx, cancel := context.WithTimeout(ctx, 8*time.Second)
defer cancel()
st := m.probe(pctx, n)
m.mu.Lock()
if prev, ok := m.status[n.Name]; ok && !st.Online && prev.Online {
st.LastSeen = prev.LastSeen // keep the last good sighting
}
m.status[n.Name] = st
m.mu.Unlock()
}(n)
}
wg.Wait()
}
// List returns every managed node with its last known status.
func (m *Manager) List() []Status {
var list []config.Node
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
m.mu.Lock()
defer m.mu.Unlock()
out := make([]Status, 0, len(list))
for _, n := range list {
if st, ok := m.status[n.Name]; ok {
out = append(out, *st)
} else {
out = append(out, Status{Name: n.Name, URL: n.URL, Fingerprint: n.Fingerprint, Added: n.Added, Error: "not probed yet"})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
return out
}
// ---- add / remove ----
// Add pairs with a node: pin its certificate (trust on first use), trade the node's
// password for a token named after this hub, and store the node. The password is
// used once and never kept.
func (m *Manager) Add(ctx context.Context, name, rawURL, username, password string) (Status, error) {
if !nameRe.MatchString(name) {
return Status{}, errors.New("name: lowercase letters, digits and dashes only")
}
if _, exists := m.node(name); exists {
return Status{}, errors.New("a machine with that name already exists")
}
u, err := url.Parse(strings.TrimSpace(rawURL))
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
return Status{}, errors.New("address must be http(s)://host:port")
}
n := config.Node{Name: name, URL: u.Scheme + "://" + u.Host, Added: time.Now()}
if u.Scheme == "https" {
host := u.Host
if _, _, err := net.SplitHostPort(host); err != nil {
host += ":443"
}
fp, err := tlsauto.Peek(host)
if err != nil {
return Status{}, fmt.Errorf("connect: %s", shortErr(err))
}
n.Fingerprint = fp
}
body, _ := json.Marshal(map[string]string{"username": username, "password": password, "name": "hub:" + m.hubName})
req, _ := http.NewRequestWithContext(ctx, "POST", n.URL+"/api/tokens", bytes.NewReader(body))
req.Header.Set("Content-Type", "application/json")
resp, err := (&http.Client{Transport: transportFor(n), Timeout: 15 * time.Second}).Do(req)
if err != nil {
return Status{}, fmt.Errorf("connect: %s", shortErr(err))
}
defer resp.Body.Close()
var reply struct {
Token string `json:"token"`
Error string `json:"error"`
}
_ = json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&reply)
if resp.StatusCode != 200 || reply.Token == "" {
if reply.Error == "" {
reply.Error = fmt.Sprintf("HTTP %d", resp.StatusCode)
}
return Status{}, errors.New("pairing refused: " + reply.Error)
}
n.Token = reply.Token
if err := m.cfg.Update(func(c *config.Config) { c.Nodes = append(c.Nodes, n) }); err != nil {
return Status{}, err
}
st := m.probe(ctx, n)
m.mu.Lock()
m.status[name] = st
m.mu.Unlock()
return *st, nil
}
// Remove revokes our token on the node (best effort) and forgets it.
func (m *Manager) Remove(ctx context.Context, name string) error {
n, ok := m.node(name)
if !ok {
return errors.New("no such machine")
}
req, _ := http.NewRequestWithContext(ctx, "DELETE", n.URL+"/api/tokens/"+url.PathEscape("hub:"+m.hubName), nil)
req.Header.Set("Authorization", "Bearer "+n.Token)
if resp, err := m.client(n).Do(req); err == nil {
resp.Body.Close()
}
err := m.cfg.Update(func(c *config.Config) {
kept := c.Nodes[:0]
for _, x := range c.Nodes {
if x.Name != name {
kept = append(kept, x)
}
}
c.Nodes = kept
})
m.mu.Lock()
delete(m.status, name)
delete(m.proxies, name)
delete(m.clients, name)
m.mu.Unlock()
return err
}
// ---- update from hub ----
type PushResult struct {
Name string `json:"name"`
OK bool `json:"ok"`
Version string `json:"version,omitempty"`
Error string `json:"error,omitempty"`
}
// Push uploads binPath (normally the hub's own executable) to the node's self-update
// endpoint. The node verifies it, installs it and restarts itself. Refused when the
// node's architecture differs from ours.
func (m *Manager) Push(ctx context.Context, name, binPath, ourArch string, force bool) PushResult {
res := PushResult{Name: name}
n, ok := m.node(name)
if !ok {
res.Error = "no such machine"
return res
}
// arch check via /api/me
req, _ := http.NewRequestWithContext(ctx, "GET", n.URL+"/api/me", nil)
req.Header.Set("Authorization", "Bearer "+n.Token)
resp, err := m.client(n).Do(req)
if err != nil {
res.Error = "unreachable: " + shortErr(err)
return res
}
var me struct {
Arch string `json:"arch"`
Version string `json:"version"`
}
_ = json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(&me)
resp.Body.Close()
if me.Arch != "" && ourArch != "" && me.Arch != ourArch {
// look for a sibling build: /usr/local/bin/hamlab-arm64 next to /usr/local/bin/hamlab
alt := binPath + "-" + me.Arch
if _, err := os.Stat(alt); err != nil {
res.Error = fmt.Sprintf("node is %s but only a %s build is installed here (expected %s; run make deploy, which builds it)", me.Arch, ourArch, alt)
return res
}
binPath = alt
}
f, err := os.Open(binPath)
if err != nil {
res.Error = err.Error()
return res
}
defer f.Close()
st, _ := f.Stat()
target := n.URL + "/api/self-update"
if force {
target += "?force=1"
}
up, _ := http.NewRequestWithContext(ctx, "POST", target, f)
up.ContentLength = st.Size()
up.Header.Set("Authorization", "Bearer "+n.Token)
up.Header.Set("Content-Type", "application/octet-stream")
c := &http.Client{Transport: transportFor(n), Timeout: 5 * time.Minute}
r2, err := c.Do(up)
if err != nil {
res.Error = "upload: " + shortErr(err)
return res
}
defer r2.Body.Close()
var reply struct {
Version string `json:"version"`
Error string `json:"error"`
}
_ = json.NewDecoder(io.LimitReader(r2.Body, 64<<10)).Decode(&reply)
if r2.StatusCode != 200 {
if reply.Error == "" {
reply.Error = fmt.Sprintf("HTTP %d", r2.StatusCode)
}
res.Error = reply.Error
return res
}
res.OK, res.Version = true, reply.Version
return res
}
// PushAll updates every node, in parallel.
func (m *Manager) PushAll(ctx context.Context, binPath, ourArch string, force bool) []PushResult {
var list []config.Node
m.cfg.View(func(c *config.Config) { list = append(list, c.Nodes...) })
out := make([]PushResult, len(list))
var wg sync.WaitGroup
for i, n := range list {
wg.Add(1)
go func(i int, name string) {
defer wg.Done()
out[i] = m.Push(ctx, name, binPath, ourArch, force)
}(i, n.Name)
}
wg.Wait()
return out
}
// ---- proxy ----
// Proxy returns the handler for /api/nodes/<name>/<rest>: it rewrites to <node>/api/<rest>,
// swaps the hub's cookie for the node's bearer token, and drops Origin so the node's
// WebSocket origin check (meant for browsers) doesn't reject the hub.
// Do makes one request to a managed machine as this hub and returns the whole response.
func (m *Manager) Do(ctx context.Context, name, method, path string, body []byte, hdr http.Header) (int, []byte, error) {
n, ok := m.node(name)
if !ok {
return 0, nil, errors.New("no such machine")
}
req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(n.URL, "/")+path, bytes.NewReader(body))
if err != nil {
return 0, nil, err
}
for k, v := range hdr {
req.Header[k] = v
}
req.Header.Set("Authorization", "Bearer "+n.Token)
resp, err := m.client(n).Do(req)
if err != nil {
return 0, nil, errors.New(shortErr(err))
}
defer resp.Body.Close()
b, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
return resp.StatusCode, b, err
}
// Names lists the managed machines.
func (m *Manager) Names() []string {
var out []string
m.cfg.View(func(c *config.Config) {
for _, n := range c.Nodes {
out = append(out, n.Name)
}
})
return out
}
func (m *Manager) Proxy(name string) (http.Handler, bool) {
n, ok := m.node(name)
if !ok {
return nil, false
}
m.mu.Lock()
if p, ok := m.proxies[name]; ok {
m.mu.Unlock()
return p, true
}
m.mu.Unlock()
target, _ := url.Parse(n.URL)
prefix := "/api/nodes/" + name
p := &httputil.ReverseProxy{
Director: func(r *http.Request) {
r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
r.URL.Path = "/api" + strings.TrimPrefix(r.URL.Path, prefix)
r.URL.RawPath = ""
r.Host = target.Host
r.Header.Set("Authorization", "Bearer "+n.Token)
r.Header.Del("Cookie")
r.Header.Del("Origin")
r.Header.Del("X-Forwarded-For")
r.Header.Del("X-Forwarded-Proto")
},
Transport: transportFor(n),
FlushInterval: -1, // stream task output / log tails immediately
// The node answers with its own policy headers and this hub adds the same ones to
// everything it serves; relayed twice, Access-Control-Allow-Origin is invalid and a
// browser on another origin refuses the response. The hub's copy is the one that counts.
ModifyResponse: func(resp *http.Response) error {
for _, h := range []string{"Access-Control-Allow-Origin", "X-Frame-Options", "Content-Security-Policy", "X-Content-Type-Options", "Referrer-Policy"} {
resp.Header.Del(h)
}
return nil
},
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadGateway)
_ = json.NewEncoder(w).Encode(map[string]string{"error": name + " unreachable: " + shortErr(err)})
},
}
m.mu.Lock()
m.proxies[name] = p
m.mu.Unlock()
return p, true
}
internal/priv/priv.go
+92
View File
@@ -0,0 +1,92 @@
// Package priv is the one place that knows how this daemon gets root: it already is root,
// or it has passwordless sudo, or passwordless doas (Alpine, some BSD-minded setups), or
// it has nothing and the privileged features stay off.
package priv
import (
"context"
"os"
"os/exec"
"strings"
"sync"
"time"
)
var (
mu sync.Mutex
mode string // "root" | "sudo" | "doas" | ""
prefix []string
probed time.Time
reprobe = 30 * time.Second
)
// detect settles how root is reached. Finding a way is remembered for good; finding none
// is only believed for a while, because the probe can fail for reasons that pass — sudo
// taking longer than the timeout on a machine that is still booting left a daemon
// convinced it had no root until its next restart.
func detect() {
mu.Lock()
defer mu.Unlock()
if mode != "" || (!probed.IsZero() && time.Since(probed) < reprobe) {
return
}
probed = time.Now()
if os.Geteuid() == 0 {
mode = "root"
return
}
for _, tool := range []string{"sudo", "doas"} {
if _, err := exec.LookPath(tool); err != nil {
continue
}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
err := exec.CommandContext(ctx, tool, "-n", "true").Run()
cancel()
if err == nil {
mode, prefix = tool, []string{tool, "-n"}
return
}
}
}
// Mode reports how root is reached: "root", "sudo", "doas", or "" when it can't be.
func Mode() string { detect(); mu.Lock(); defer mu.Unlock(); return mode }
// Available reports whether commands can be run as root without a prompt.
func Available() bool { return Mode() != "" }
// Wrap returns args prefixed so they run as root. Without privilege it returns args
// unchanged; callers gate on Available() where that matters.
func Wrap(args ...string) []string {
detect()
mu.Lock()
defer mu.Unlock()
return append(append([]string{}, prefix...), args...)
}
// AsUser returns args prefixed so they run as another user with that user's HOME.
func AsUser(user string, args ...string) []string {
switch Mode() {
case "sudo":
return append([]string{"sudo", "-n", "-u", user, "-H"}, args...)
case "doas":
return append([]string{"doas", "-n", "-u", user}, args...)
case "root":
if user == "root" {
return args
}
// su exists everywhere (util-linux, shadow, busybox), needs no password from root
// and sets HOME for the target user.
return []string{"su", "-s", "/bin/sh", user, "-c", Quote(args)}
}
return args
}
// Quote joins args into one POSIX shell command line.
func Quote(args []string) string {
q := make([]string, len(args))
for i, a := range args {
q[i] = "'" + strings.ReplaceAll(a, "'", `'\''`) + "'"
}
return strings.Join(q, " ")
}
internal/selfupdate/selfupdate.go
+264
View File
@@ -0,0 +1,264 @@
// Package selfupdate finds out whether a newer Ham Lab has been published and fetches it.
//
// A source is one URL. Two shapes are understood, told apart by the URL itself:
//
// - a Gitea/Forgejo repository API base — ".../api/v1/repos/<owner>/<repo>" — whose
// newest release names the version and carries the binaries as assets;
// - any other URL, treated as a static directory holding VERSION, SHA256SUMS and
// hamlab-linux-<arch>, which is exactly what `make release` writes and what
// install.sh already downloads.
//
// Either way the binary is checked against SHA256SUMS before it is allowed near the
// running one. A token is sent when the source is behind a login, so a private Gitea
// serves its own fleet without the instance being opened to the world.
package selfupdate
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
// DefaultCheckEvery is how often a daemon asks, when it is asking at all.
const DefaultCheckEvery = 24 * time.Hour
// DefaultURL is where a build comes from when the config does not say: the project's own
// releases. An install elsewhere points `update.url` at its own, or at "off".
const DefaultURL = "https://git.bigham.dev/api/v1/repos/mitchell/hamlab"
// maxBinary is the most we will download; a hamlab binary is ~20 MB.
const maxBinary = 512 << 20
// Source is where updates come from.
type Source struct {
URL string // API base or static directory
Token string // optional; for a source behind a login
}
// Release is one published build.
type Release struct {
Version string `json:"version"`
Notes string `json:"notes,omitempty"`
Published time.Time `json:"published,omitempty"`
// Binary and Sums are where to get this build for the architecture that was asked for.
Binary string `json:"-"`
Sums string `json:"-"`
}
// AssetName is what a build for an architecture is called, in a release and on disk.
func AssetName(arch string) string { return "hamlab-linux-" + arch }
func (s Source) api() bool { return strings.Contains(s.URL, "/api/v1/repos/") }
func (s Source) get(ctx context.Context, url string) (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return nil, err
}
if s.Token != "" {
req.Header.Set("Authorization", "token "+s.Token)
}
req.Header.Set("User-Agent", "hamlab-selfupdate")
resp, err := (&http.Client{Timeout: 10 * time.Minute}).Do(req)
if err != nil {
return nil, err
}
if resp.StatusCode != 200 {
resp.Body.Close()
switch resp.StatusCode {
case 401, 403:
return nil, fmt.Errorf("%s: the update source needs a login (HTTP %d); set update.token", url, resp.StatusCode)
case 404:
return nil, fmt.Errorf("%s: nothing published there (HTTP 404)", url)
}
return nil, fmt.Errorf("%s: HTTP %d", url, resp.StatusCode)
}
return resp, nil
}
func (s Source) getAll(ctx context.Context, url string, limit int64) ([]byte, error) {
resp, err := s.get(ctx, url)
if err != nil {
return nil, err
}
defer resp.Body.Close()
return io.ReadAll(io.LimitReader(resp.Body, limit))
}
// Latest is the newest published build, and where to get it for arch.
func (s Source) Latest(ctx context.Context, arch string) (Release, error) {
if s.URL == "" {
return Release{}, errors.New("no update source configured")
}
base := strings.TrimSuffix(s.URL, "/")
if !s.api() {
b, err := s.getAll(ctx, base+"/VERSION", 4096)
if err != nil {
return Release{}, err
}
v := strings.TrimSpace(strings.SplitN(string(b), "\n", 2)[0])
if v == "" {
return Release{}, errors.New("the source published an empty VERSION")
}
return Release{Version: v, Binary: base + "/" + AssetName(arch), Sums: base + "/SHA256SUMS"}, nil
}
b, err := s.getAll(ctx, base+"/releases/latest", 4<<20)
if err != nil {
return Release{}, err
}
var rel struct {
TagName string `json:"tag_name"`
Body string `json:"body"`
PublishedAt time.Time `json:"published_at"`
Assets []struct {
Name string `json:"name"`
URL string `json:"browser_download_url"`
} `json:"assets"`
}
if err := json.Unmarshal(b, &rel); err != nil {
return Release{}, fmt.Errorf("the source did not answer with a release: %w", err)
}
if rel.TagName == "" {
return Release{}, errors.New("the newest release has no tag")
}
out := Release{Version: rel.TagName, Notes: rel.Body, Published: rel.PublishedAt}
want := AssetName(arch)
for _, a := range rel.Assets {
switch a.Name {
case want:
out.Binary = a.URL
case "SHA256SUMS":
out.Sums = a.URL
}
}
if out.Binary == "" {
return out, fmt.Errorf("release %s has no %s", rel.TagName, want)
}
return out, nil
}
// Fetch downloads the build for arch to dest and checks it against SHA256SUMS. A release
// without sums is refused: an unverified binary is not worth installing over a working one.
func (s Source) Fetch(ctx context.Context, rel Release, arch, dest string) error {
if rel.Binary == "" {
return errors.New("that release has no binary for this machine")
}
if rel.Sums == "" {
return errors.New("that release has no SHA256SUMS, so the download cannot be verified")
}
sums, err := s.getAll(ctx, rel.Sums, 1<<20)
if err != nil {
return err
}
want := sumFor(string(sums), AssetName(arch))
if want == "" {
return fmt.Errorf("SHA256SUMS has no entry for %s", AssetName(arch))
}
resp, err := s.get(ctx, rel.Binary)
if err != nil {
return err
}
defer resp.Body.Close()
f, err := os.OpenFile(dest, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755)
if err != nil {
return err
}
h := sha256.New()
n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, maxBinary))
cerr := f.Close()
if err == nil {
err = cerr
}
if err != nil {
os.Remove(dest)
return err
}
if n < 1<<20 {
os.Remove(dest)
return fmt.Errorf("downloaded %d bytes: too small to be hamlab", n)
}
if got := hex.EncodeToString(h.Sum(nil)); got != want {
os.Remove(dest)
return fmt.Errorf("checksum mismatch: SHA256SUMS says %s, the download is %s", want[:16]+"…", got[:16]+"…")
}
return nil
}
// sumFor finds a file's checksum in SHA256SUMS ("<hex> <name>", or "*<name>" for binary mode).
func sumFor(sums, name string) string {
for _, line := range strings.Split(sums, "\n") {
f := strings.Fields(strings.TrimSpace(line))
if len(f) == 2 && (f[1] == name || f[1] == "*"+name) && len(f[0]) == 64 {
return strings.ToLower(f[0])
}
}
return ""
}
// Compare orders two versions: -1 if a is older than b, 0 if the same, 1 if newer.
// It is install.sh's vnum() rule, so the daemon and the installer never disagree about
// which way round two builds go: v0.1.0-69-g9c4f9df → 0001000069, the commits since the
// tag included, the hash ignored.
func Compare(a, b string) int {
na, oka := vnum(a)
nb, okb := vnum(b)
switch {
case !oka && !okb:
return strings.Compare(a, b)
case !oka:
return -1
case !okb:
return 1
case na < nb:
return -1
case na > nb:
return 1
}
return 0
}
func vnum(v string) (int64, bool) {
v = strings.TrimSpace(v)
v = strings.TrimPrefix(v, "v")
if i := strings.LastIndex(v, "-g"); i >= 0 && isHex(v[i+2:]) {
v = v[:i]
}
if v == "" {
return 0, false
}
parts := strings.FieldsFunc(v, func(r rune) bool { return r == '.' || r == '-' })
nums := make([]int64, 4)
for i := 0; i < 4 && i < len(parts); i++ {
n, err := strconv.ParseInt(parts[i], 10, 64)
if err != nil {
if i == 0 {
return 0, false
}
break
}
nums[i] = n
}
// %d%03d%03d%05d, as the installer does it
return ((nums[0]*1000+nums[1])*1000+nums[2])*100000 + nums[3], true
}
func isHex(s string) bool {
if s == "" {
return false
}
for _, r := range s {
if !strings.ContainsRune("0123456789abcdefABCDEF", r) {
return false
}
}
return true
}
internal/selfupdate/selfupdate_test.go
+145
View File
@@ -0,0 +1,145 @@
package selfupdate
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
)
func TestCompareMatchesInstaller(t *testing.T) {
older := []([2]string){
{"v0.1.0-69-g9c4f9df", "v0.1.0-107-gad88a48"},
{"v0.1.0", "v0.1.0-1-gabc1234"},
{"v0.1.0-107", "v0.2.0"},
{"v0.9.9-999", "v1.0.0"},
{"v0.1.9", "v0.1.10"},
}
for _, p := range older {
if got := Compare(p[0], p[1]); got != -1 {
t.Errorf("Compare(%q, %q) = %d, want -1", p[0], p[1], got)
}
if got := Compare(p[1], p[0]); got != 1 {
t.Errorf("Compare(%q, %q) = %d, want 1", p[1], p[0], got)
}
}
same := []string{"v0.1.0-107-gad88a48", "v1.2.3", "dev"}
for _, v := range same {
if got := Compare(v, v); got != 0 {
t.Errorf("Compare(%q, %q) = %d, want 0", v, v, got)
}
}
// the hash is ignored: the same build described two ways is the same build
if got := Compare("v0.1.0-107-gad88a48", "v0.1.0-107-gffffff1"); got != 0 {
t.Errorf("hash should not count: %d", got)
}
// "dev" is not a version; anything numbered beats it
if got := Compare("dev", "v0.1.0"); got != -1 {
t.Errorf("Compare(dev, v0.1.0) = %d, want -1", got)
}
}
// serve puts a release behind both shapes of source and returns the two base URLs.
func serve(t *testing.T, version string, body []byte, token string) (static, api string, srv *httptest.Server) {
t.Helper()
sum := sha256.Sum256(body)
sums := fmt.Sprintf("%s %s\n%s install.sh\n", hex.EncodeToString(sum[:]), AssetName("amd64"), strings.Repeat("0", 64))
mux := http.NewServeMux()
guard := func(h http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if token != "" && r.Header.Get("Authorization") != "token "+token {
w.WriteHeader(401)
return
}
h(w, r)
}
}
mux.HandleFunc("/dl/VERSION", guard(func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, version) }))
mux.HandleFunc("/dl/SHA256SUMS", guard(func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, sums) }))
mux.HandleFunc("/dl/"+AssetName("amd64"), guard(func(w http.ResponseWriter, r *http.Request) { w.Write(body) }))
mux.HandleFunc("/api/v1/repos/m/hamlab/releases/latest", guard(func(w http.ResponseWriter, r *http.Request) {
base := "http://" + r.Host + "/dl/"
_ = json.NewEncoder(w).Encode(map[string]any{
"tag_name": version, "body": "notes here",
"assets": []map[string]string{
{"name": AssetName("amd64"), "browser_download_url": base + AssetName("amd64")},
{"name": "SHA256SUMS", "browser_download_url": base + "SHA256SUMS"},
},
})
}))
srv = httptest.NewServer(mux)
return srv.URL + "/dl", srv.URL + "/api/v1/repos/m/hamlab", srv
}
func TestLatestAndFetch(t *testing.T) {
body := make([]byte, 2<<20)
copy(body, "ELF fake hamlab")
static, api, srv := serve(t, "v0.9.0-3-gdeadbee", body, "")
defer srv.Close()
for _, url := range []string{static, api} {
s := Source{URL: url}
rel, err := s.Latest(context.Background(), "amd64")
if err != nil {
t.Fatalf("%s: %v", url, err)
}
if rel.Version != "v0.9.0-3-gdeadbee" {
t.Fatalf("%s: version %q", url, rel.Version)
}
dest := filepath.Join(t.TempDir(), "hamlab.new")
if err := s.Fetch(context.Background(), rel, "amd64", dest); err != nil {
t.Fatalf("%s: fetch: %v", url, err)
}
got, _ := os.ReadFile(dest)
if len(got) != len(body) {
t.Fatalf("%s: got %d bytes, want %d", url, len(got), len(body))
}
}
}
func TestFetchRefusesBadChecksum(t *testing.T) {
body := make([]byte, 2<<20)
static, _, srv := serve(t, "v1.0.0", body, "")
defer srv.Close()
s := Source{URL: static}
rel, err := s.Latest(context.Background(), "amd64")
if err != nil {
t.Fatal(err)
}
rel.Sums = static + "/VERSION" // sums that name nothing
dest := filepath.Join(t.TempDir(), "hamlab.new")
if err := s.Fetch(context.Background(), rel, "amd64", dest); err == nil {
t.Fatal("a release whose sums do not name the binary must be refused")
}
if _, err := os.Stat(dest); !os.IsNotExist(err) {
t.Fatal("a refused download must not be left on disk")
}
// and an arch nobody published
if _, err := s.Latest(context.Background(), "riscv64"); err == nil {
t.Log("static sources cannot know an arch is missing until the fetch; that is fine")
}
}
func TestTokenIsSentAndMissingTokenSaysSo(t *testing.T) {
body := make([]byte, 2<<20)
_, api, srv := serve(t, "v1.0.0", body, "s3cret")
defer srv.Close()
if _, err := (Source{URL: api}).Latest(context.Background(), "amd64"); err == nil || !strings.Contains(err.Error(), "update.token") {
t.Fatalf("a walled source should say a token is needed, got %v", err)
}
rel, err := (Source{URL: api, Token: "s3cret"}).Latest(context.Background(), "amd64")
if err != nil || rel.Version != "v1.0.0" {
t.Fatalf("with the token: %v %+v", err, rel)
}
dest := filepath.Join(t.TempDir(), "hamlab.new")
if err := (Source{URL: api, Token: "s3cret"}).Fetch(context.Background(), rel, "amd64", dest); err != nil {
t.Fatalf("fetch with token: %v", err)
}
}
internal/sensors/prom.go
+76
View File
@@ -0,0 +1,76 @@
package sensors
import (
"fmt"
"io"
"sort"
"strings"
)
// promFamily names the metric family for a sensor and scales its value to the base unit
// Prometheus convention wants (seconds, not minutes).
func promFamily(s Sensor) (name string, scale float64) {
scale = 1
unit := map[string]string{"%": "percent", "°C": "celsius", "rpm": "rpm", "B/s": "bytes_per_second", "B": "bytes",
"s": "seconds", "V": "volts", "W": "watts", "MHz": "megahertz"}[s.Unit]
if s.Unit == "min" {
unit, scale = "seconds", 60
}
if s.Kind == "count" { // hamlab_count{id=…} would say nothing; name it after the thing counted
return "hamlab_" + strings.ReplaceAll(Seg(strings.ReplaceAll(s.ID, ".", "_")), "-", "_"), 1
}
name = "hamlab_" + s.Kind
if unit != "" {
name += "_" + unit
}
return
}
var promEsc = strings.NewReplacer(`\`, `\\`, `"`, `\"`, "\n", `\n`)
// WriteProm writes sensors in the Prometheus text exposition format: one family per kind
// of measurement, the sensor's ID and its real-world labels as labels. Booleans are 0/1;
// a text state is a 1 carrying the text as a label, the usual way to export an enum.
func WriteProm(w io.Writer, all []Sensor) {
fams := map[string][]string{}
for _, s := range all {
name, scale := promFamily(s)
labels := []string{`id="` + promEsc.Replace(s.ID) + `"`}
keys := make([]string, 0, len(s.Labels))
for k := range s.Labels {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
if s.Labels[k] != "" {
labels = append(labels, Seg(k)+`="`+promEsc.Replace(s.Labels[k])+`"`)
}
}
var v float64
switch x := s.Value.(type) {
case float64:
v = x * scale
case bool:
if x {
v = 1
}
case string:
v = 1
labels = append(labels, `value="`+promEsc.Replace(x)+`"`)
default:
continue
}
fams[name] = append(fams[name], fmt.Sprintf("%s{%s} %v", name, strings.Join(labels, ","), v))
}
names := make([]string, 0, len(fams))
for n := range fams {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
fmt.Fprintf(w, "# TYPE %s gauge\n", n)
for _, line := range fams[n] {
fmt.Fprintln(w, line)
}
}
}
internal/sensors/sensors.go
+275
View File
@@ -0,0 +1,275 @@
// Package sensors is the one standard way anything reads what Ham Lab measures.
//
// Every measurable thing is a Sensor: a stable dotted ID ("cpu.core.3.usage",
// "fan.nct6775.fan2", "fs.root.used_pct"), a little metadata, and a current value. A
// consumer discovers them from the catalog, then asks for exactly the ones it wants —
// one by ID, many by glob, or as a shaped query where the client sends the object it
// wants and gets it back with the values filled in. See SENSORS.md.
//
// Sources of sensors are Providers. The registry only joins them; it measures nothing.
package sensors
import (
"encoding/json"
"errors"
"fmt"
"path"
"sort"
"strings"
)
// Sensor is one measurable thing and its reading. Value is a float64, bool or string.
type Sensor struct {
ID string `json:"id"`
Name string `json:"name"` // human label
Kind string `json:"kind"` // usage | load | temperature | fan | rate | bytes | duration | voltage | power | state
Unit string `json:"unit,omitempty"` // % · °C · rpm · B/s · B · s · min · V · W · MHz
Value any `json:"value"`
T int64 `json:"t"` // unix ms the value was read — slow sensors are older than the tick
Min *float64 `json:"min,omitempty"`
Max *float64 `json:"max,omitempty"`
Warn *float64 `json:"warn,omitempty"` // the hardware's own thresholds, where it states them
Crit *float64 `json:"crit,omitempty"`
Labels map[string]string `json:"labels,omitempty"` // the real names behind the ID: chip, iface, mount…
}
// Series is a numeric sensor's recent past.
type Series struct {
ID string `json:"id"`
Unit string `json:"unit,omitempty"`
T []int64 `json:"t"`
V []float64 `json:"v"`
}
// Provider is a source of sensors. Sensors returns every sensor it has, with current values.
type Provider interface {
Sensors() []Sensor
}
// Historian is a Provider that also remembers: the recent past of every numeric sensor
// of its own that match accepts.
type Historian interface {
SensorHistories(match func(id string) bool) []Series
}
// Ticker is a Provider that can say when its values have changed.
type Ticker interface {
Subscribe() (tick <-chan struct{}, cancel func())
}
// Subscribe follows the first provider that ticks. ok is false when none does.
func (r *Registry) Subscribe() (tick <-chan struct{}, cancel func(), ok bool) {
for _, p := range r.providers {
if t, is := p.(Ticker); is {
tick, cancel = t.Subscribe()
return tick, cancel, true
}
}
return nil, nil, false
}
// F is a pointer to a float, for the optional bounds.
func F(v float64) *float64 { return &v }
// Seg makes one ID segment from a real-world name: lower case, [a-z0-9_-] only, so IDs
// survive URLs, query strings and templating languages untouched. "Package id 0" →
// "package_id_0", "/mnt/tank" → "mnt_tank", "/" → "root".
func Seg(name string) string {
if name == "/" {
return "root"
}
var b strings.Builder
gap := false
for _, r := range strings.ToLower(strings.TrimSpace(name)) {
switch {
case r >= 'a' && r <= 'z', r >= '0' && r <= '9', r == '-':
if gap && b.Len() > 0 {
b.WriteByte('_')
}
gap = false
b.WriteRune(r)
default:
gap = true
}
}
if b.Len() == 0 {
return "unnamed"
}
return b.String()
}
// ID joins already-safe segments.
func ID(segs ...string) string { return strings.Join(segs, ".") }
// Registry joins providers into one namespace.
type Registry struct{ providers []Provider }
func NewRegistry(p ...Provider) *Registry { return &Registry{providers: p} }
func (r *Registry) Add(p Provider) { r.providers = append(r.providers, p) }
// All returns every sensor, sorted by ID. Two sensors that would share an ID (two
// identical chips) get "_2", "_3" on the colliding one, in the provider's order.
func (r *Registry) All() []Sensor {
var out []Sensor
seen := map[string]int{}
for _, p := range r.providers {
for _, s := range p.Sensors() {
seen[s.ID]++
if n := seen[s.ID]; n > 1 {
s.ID = fmt.Sprintf("%s_%d", s.ID, n)
}
out = append(out, s)
}
}
sort.SliceStable(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return out
}
// History is one sensor's recent past. ok is false when nothing remembers it.
func (r *Registry) History(id string) (Series, bool) {
got := r.Histories(func(x string) bool { return x == id })
if len(got) == 0 {
return Series{}, false
}
return got[0], true
}
// Histories is the recent past of every numeric sensor match accepts, sorted by ID.
func (r *Registry) Histories(match func(id string) bool) []Series {
out := []Series{}
for _, p := range r.providers {
if h, ok := p.(Historian); ok {
out = append(out, h.SensorHistories(match)...)
}
}
sort.SliceStable(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return out
}
// Matcher turns comma-separated selectors (IDs and globs) into a match function.
func Matcher(selectors string) (func(id string) bool, error) {
var sels []string
for _, sel := range strings.Split(selectors, ",") {
if sel = strings.TrimSpace(sel); sel == "" {
continue
}
if _, err := path.Match(sel, ""); err != nil {
return nil, fmt.Errorf("bad pattern %q", sel)
}
sels = append(sels, sel)
}
return func(id string) bool {
for _, sel := range sels {
if ok, _ := path.Match(sel, id); ok {
return true
}
}
return false
}, nil
}
// IsGlob reports whether a selector names many sensors rather than one.
func IsGlob(sel string) bool { return strings.ContainsAny(sel, "*?[") }
// Select returns the sensors a selector names: an exact ID, or a glob where * crosses
// dots ("fan.*" is every fan, "cpu.core.*.usage" every core).
func Select(all []Sensor, sel string) ([]Sensor, error) {
if !IsGlob(sel) {
for _, s := range all {
if s.ID == sel {
return []Sensor{s}, nil
}
}
return nil, nil
}
if _, err := path.Match(sel, ""); err != nil {
return nil, fmt.Errorf("bad pattern %q", sel)
}
var out []Sensor
for _, s := range all {
if ok, _ := path.Match(sel, s.ID); ok {
out = append(out, s)
}
}
return out, nil
}
// Shape answers a shaped query: the client sends the object it wants, with selectors
// where the values go, and gets the same object back with them filled in.
//
// {"cpu": "cpu.usage", "fans": "fan.*", "box": {"up": "uptime"}}
// → {"cpu": 12.4, "fans": {"fan.nct6775.fan1": 820}, "box": {"up": 80520}}
//
// An exact ID becomes its value (null if there is no such sensor here — machines differ,
// and a dashboard shouldn't fail because one has no fans). A glob becomes an object keyed
// by ID. Objects and arrays are walked. With full, every value is the whole Sensor.
func Shape(all []Sensor, query json.RawMessage, full bool) (any, error) {
var q any
if err := json.Unmarshal(query, &q); err != nil {
return nil, errors.New("query is not valid JSON")
}
budget := MaxValues
return shape(all, q, full, 0, &budget)
}
// MaxValues is how many sensor values one query may return. A query is a few bytes that
// can ask for everything many times over ({"a":"*","b":"*",…}); without a ceiling 10 KB of
// request buys 27 MB of response. Ten thousand is thirty times a whole machine.
const MaxValues = 10000
func shape(all []Sensor, q any, full bool, depth int, budget *int) (any, error) {
if depth > 16 {
return nil, errors.New("query nested too deep")
}
leaf := func(s Sensor) any {
if full {
return s
}
return s.Value
}
switch v := q.(type) {
case string:
got, err := Select(all, v)
if err != nil {
return nil, err
}
if !IsGlob(v) {
if len(got) == 0 {
return nil, nil
}
*budget--
return leaf(got[0]), nil
}
if *budget -= len(got); *budget < 0 {
return nil, fmt.Errorf("query returns more than %d values; ask for less", MaxValues)
}
m := make(map[string]any, len(got))
for _, s := range got {
m[s.ID] = leaf(s)
}
return m, nil
case map[string]any:
m := make(map[string]any, len(v))
for k, sub := range v {
r, err := shape(all, sub, full, depth+1, budget)
if err != nil {
return nil, err
}
m[k] = r
}
return m, nil
case []any:
a := make([]any, len(v))
for i, sub := range v {
r, err := shape(all, sub, full, depth+1, budget)
if err != nil {
return nil, err
}
a[i] = r
}
return a, nil
default:
return nil, fmt.Errorf("a query holds sensor IDs, objects and arrays; got %v", v)
}
}
internal/sensors/sensors_test.go
+95
View File
@@ -0,0 +1,95 @@
package sensors
import (
"encoding/json"
"reflect"
"strings"
"testing"
)
type fake []Sensor
func (f fake) Sensors() []Sensor { return f }
func TestSeg(t *testing.T) {
for in, want := range map[string]string{"/": "root", "/mnt/tank": "mnt_tank", "Package id 0": "package_id_0", "enp3s0": "enp3s0", " ": "unnamed", "Tctl+": "tctl", "nvme0n1": "nvme0n1"} {
if got := Seg(in); got != want {
t.Errorf("Seg(%q) = %q, want %q", in, got, want)
}
}
}
func TestRegistryDedupes(t *testing.T) {
r := NewRegistry(fake{{ID: "temp.nvme.composite", Value: 40.0}}, fake{{ID: "temp.nvme.composite", Value: 50.0}})
all := r.All()
if len(all) != 2 || all[0].ID != "temp.nvme.composite" || all[1].ID != "temp.nvme.composite_2" {
t.Fatalf("got %+v", all)
}
}
func TestShape(t *testing.T) {
all := []Sensor{{ID: "cpu.usage", Value: 12.5, Unit: "%"}, {ID: "fan.a.fan1", Value: 800.0}, {ID: "fan.a.fan2", Value: 900.0}, {ID: "net.eth0.up", Value: true}}
got, err := Shape(all, json.RawMessage(`{"cpu":"cpu.usage","fans":"fan.*","box":{"link":"net.eth0.up","gone":"ups.battery"},"list":["cpu.usage"]}`), false)
if err != nil {
t.Fatal(err)
}
want := map[string]any{
"cpu": 12.5,
"fans": map[string]any{"fan.a.fan1": 800.0, "fan.a.fan2": 900.0},
"box": map[string]any{"link": true, "gone": nil},
"list": []any{12.5},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %#v\nwant %#v", got, want)
}
full, _ := Shape(all, json.RawMessage(`"cpu.usage"`), true)
if s, ok := full.(Sensor); !ok || s.Unit != "%" {
t.Fatalf("full: got %#v", full)
}
for _, bad := range []string{`{"x": 5}`, `{"x": "fan.["}`, `not json`} {
if _, err := Shape(all, json.RawMessage(bad), false); err == nil {
t.Errorf("Shape(%s) should fail", bad)
}
}
}
func TestWriteProm(t *testing.T) {
var b strings.Builder
WriteProm(&b, []Sensor{
{ID: "cpu.usage", Kind: "usage", Unit: "%", Value: 12.5},
{ID: "ups.runtime", Kind: "duration", Unit: "min", Value: 2.0},
{ID: "net.eth0.up", Kind: "state", Value: true, Labels: map[string]string{"iface": "eth0"}},
{ID: "ups.state", Kind: "state", Value: `Nor"mal`},
{ID: "updates.pending", Kind: "count", Value: 3.0},
})
got := b.String()
for _, want := range []string{
"# TYPE hamlab_usage_percent gauge\n" + `hamlab_usage_percent{id="cpu.usage"} 12.5`,
`hamlab_duration_seconds{id="ups.runtime"} 120`,
`hamlab_state{id="net.eth0.up",iface="eth0"} 1`,
`hamlab_state{id="ups.state",value="Nor\"mal"} 1`,
`hamlab_updates_pending{id="updates.pending"} 3`,
} {
if !strings.Contains(got, want) {
t.Errorf("missing %q in:\n%s", want, got)
}
}
}
func TestShapeBudget(t *testing.T) {
all := make([]Sensor, 300)
for i := range all {
all[i] = Sensor{ID: "s." + strings.Repeat("x", i%7) + string(rune('a'+i%26)) + strings.Repeat("y", i/26), Value: 1.0}
}
q := map[string]string{}
for i := 0; i < 40; i++ { // 40 × 300 = 12 000 values
q["k"+strings.Repeat("z", i)] = "*"
}
b, _ := json.Marshal(q)
if _, err := Shape(all, b, false); err == nil {
t.Fatal("a query returning more than MaxValues must be refused")
}
if _, err := Shape(all, json.RawMessage(`{"a":"*","b":"*"}`), false); err != nil {
t.Fatalf("an ordinary query must pass: %v", err)
}
}
internal/services/services.go
+368
View File
@@ -0,0 +1,368 @@
// Package services wraps systemctl and journalctl for the Services tab: list units,
// show one, read its unit file, read and tail its journal, and start/stop/etc.
//
// Reads run as the daemon's user (systemd exposes state to everyone; the journal needs
// the systemd-journal or adm group, or sudo). Actions go through passwordless sudo when
// it's available, otherwise plain systemctl so polkit gets its say.
package services
import (
"bufio"
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"syscall"
"time"
"hamlab/internal/priv"
)
type Unit struct {
Name string `json:"name"`
Description string `json:"description"`
Load string `json:"load"`
Active string `json:"active"`
Sub string `json:"sub"`
Enabled string `json:"enabled"` // enabled | disabled | static | masked | … | "" (unknown)
Failed bool `json:"failed"`
}
type Detail struct {
Unit
Props map[string]string `json:"props"`
}
type Entry struct {
T int64 `json:"t"` // unix ms
Priority int `json:"p"` // syslog priority 0-7
Ident string `json:"ident"`
PID string `json:"pid,omitempty"`
Msg string `json:"msg"`
}
var unitRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9@._:\\-]{0,255}\.service$`)
func Valid(unit string) bool { return unitRe.MatchString(unit) }
var Actions = map[string]bool{"start": true, "stop": true, "restart": true, "reload": true, "enable": true, "disable": true, "reset-failed": true}
// Properties shown on the detail page, in display order.
var showProps = []string{"Id", "Description", "LoadState", "ActiveState", "SubState", "UnitFileState", "FragmentPath",
"MainPID", "ExecMainStartTimestamp", "ActiveEnterTimestamp", "InactiveEnterTimestamp", "NRestarts", "Result",
"MemoryCurrent", "MemoryPeak", "CPUUsageNSec", "TasksCurrent", "ExecStart", "User", "Restart"}
type Manager struct {
sudo bool // actions via sudo -n
journalSudo bool // journal needs sudo (user isn't in systemd-journal/adm)
mu sync.Mutex
cache []Unit
cacheAt time.Time
}
// ErrNoSystemd is what the Services tab says on OpenRC, runit and friends.
var ErrNoSystemd = errors.New("this host doesn't run systemd, which the Services tab needs")
// HasSystemd reports whether systemd is the running init (the test sd_booted() uses).
func HasSystemd() bool {
_, err := os.Stat("/run/systemd/system")
return err == nil
}
func New() *Manager {
m := &Manager{}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
m.sudo = priv.Available()
// Prefer reading the journal as ourselves: a plain child can be killed cleanly when a
// tail ends, whereas a root journalctl under sudo can only be reached with sudo kill.
var errb bytes.Buffer
probe := exec.CommandContext(ctx, "journalctl", "--system", "-n", "1", "-q", "-o", "cat")
probe.Stderr = &errb
if err := probe.Run(); err != nil || strings.Contains(strings.ToLower(errb.String()), "permission") {
m.journalSudo = m.sudo
}
return m
}
// Sudo reports whether actions will run through passwordless sudo.
func (m *Manager) Sudo() bool { return m.sudo }
// JournalSudo reports whether reading the system journal needs sudo for this user.
func (m *Manager) JournalSudo() bool { return m.journalSudo }
func run(ctx context.Context, args ...string) (string, error) {
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
var out, errb bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errb
if err := cmd.Run(); err != nil {
msg := strings.TrimSpace(errb.String())
if msg == "" {
msg = err.Error()
}
return out.String(), errors.New(msg)
}
return out.String(), nil
}
// List merges `list-units --all` (runtime state) with `list-unit-files` (enablement).
// Cached for a few seconds because the UI polls it.
func (m *Manager) List(ctx context.Context) ([]Unit, error) {
if !HasSystemd() {
return nil, ErrNoSystemd
}
m.mu.Lock()
if time.Since(m.cacheAt) < 200*time.Millisecond && m.cache != nil { // coalesce simultaneous clients only
c := m.cache
m.mu.Unlock()
return c, nil
}
m.mu.Unlock()
var out, files string
var err error
done := make(chan struct{})
go func() {
files, _ = run(ctx, "systemctl", "list-unit-files", "--type=service", "--output=json", "--no-pager")
close(done)
}()
out, err = run(ctx, "systemctl", "list-units", "--type=service", "--all", "--output=json", "--no-pager", "--plain")
<-done
if err != nil {
return nil, err
}
var units []listedUnit
if err := json.Unmarshal([]byte(out), &units); err != nil {
// systemd before 246 (RHEL 8, Ubuntu 20.04) ignores --output=json here and prints
// the table; the columns have been the same since the beginning.
if units = parseUnitTable(out); len(units) == 0 {
return nil, fmt.Errorf("parse list-units: %w", err)
}
}
var uf []listedFile
if json.Unmarshal([]byte(files), &uf) != nil {
uf = parseFileTable(files)
}
enabled := map[string]string{}
for _, f := range uf {
enabled[f.UnitFile] = f.State
}
seen := map[string]bool{}
list := make([]Unit, 0, len(units)+len(uf))
for _, u := range units {
if strings.Contains(u.Unit, "@.") { // uninstantiated template
continue
}
seen[u.Unit] = true
list = append(list, Unit{Name: u.Unit, Description: u.Description, Load: u.Load, Active: u.Active, Sub: u.Sub,
Enabled: enabled[u.Unit], Failed: u.Active == "failed"})
}
// Unit files that exist but were never loaded this boot (disabled and never started).
for _, f := range uf {
if seen[f.UnitFile] || strings.Contains(f.UnitFile, "@.") || f.State == "masked" || f.State == "alias" {
continue
}
list = append(list, Unit{Name: f.UnitFile, Load: "not-loaded", Active: "inactive", Sub: "dead", Enabled: f.State})
}
sort.Slice(list, func(i, j int) bool { return list[i].Name < list[j].Name })
m.mu.Lock()
m.cache, m.cacheAt = list, time.Now()
m.mu.Unlock()
return list, nil
}
type listedUnit struct{ Unit, Load, Active, Sub, Description string }
type listedFile struct {
UnitFile string `json:"unit_file"`
State string `json:"state"`
}
// parseUnitTable reads the plain `systemctl list-units` table: UNIT LOAD ACTIVE SUB DESCRIPTION…
func parseUnitTable(text string) []listedUnit {
var units []listedUnit
for _, line := range strings.Split(text, "\n") {
f := strings.Fields(strings.TrimLeft(line, "●*× "))
if len(f) < 4 || !strings.HasSuffix(f[0], ".service") {
continue
}
units = append(units, listedUnit{Unit: f[0], Load: f[1], Active: f[2], Sub: f[3], Description: strings.Join(f[4:], " ")})
}
return units
}
// parseFileTable reads the plain `systemctl list-unit-files` table: UNIT FILE, STATE[, PRESET]
func parseFileTable(text string) []listedFile {
var uf []listedFile
for _, line := range strings.Split(text, "\n") {
if f := strings.Fields(line); len(f) >= 2 && strings.HasSuffix(f[0], ".service") {
uf = append(uf, listedFile{UnitFile: f[0], State: f[1]})
}
}
return uf
}
func (m *Manager) invalidate() {
m.mu.Lock()
m.cacheAt = time.Time{}
m.mu.Unlock()
}
func (m *Manager) Show(ctx context.Context, unit string) (Detail, error) {
if !Valid(unit) {
return Detail{}, errors.New("invalid unit name")
}
out, err := run(ctx, "systemctl", "show", unit, "-p", strings.Join(showProps, ","), "--no-pager")
if err != nil {
return Detail{}, err
}
props := map[string]string{}
for _, line := range strings.Split(out, "\n") {
if k, v, ok := strings.Cut(line, "="); ok && v != "" && v != "[not set]" && v != "infinity" {
props[k] = v
}
}
if props["LoadState"] == "not-found" {
return Detail{}, errors.New("no such unit")
}
d := Detail{Props: props}
d.Name, d.Description = unit, props["Description"]
d.Load, d.Active, d.Sub, d.Enabled = props["LoadState"], props["ActiveState"], props["SubState"], props["UnitFileState"]
d.Failed = d.Active == "failed"
return d, nil
}
func (m *Manager) Cat(ctx context.Context, unit string) (string, error) {
if !Valid(unit) {
return "", errors.New("invalid unit name")
}
return run(ctx, "systemctl", "cat", unit, "--no-pager")
}
func parseEntry(line []byte) (Entry, bool) {
var raw map[string]any
if json.Unmarshal(line, &raw) != nil {
return Entry{}, false
}
e := Entry{Priority: 6}
switch v := raw["MESSAGE"].(type) {
case string:
e.Msg = v
case []any: // non-UTF-8 payload comes as a byte array
b := make([]byte, 0, len(v))
for _, x := range v {
if f, ok := x.(float64); ok {
b = append(b, byte(f))
}
}
e.Msg = string(b)
default:
e.Msg = ""
}
if p, ok := raw["PRIORITY"].(string); ok {
e.Priority, _ = strconv.Atoi(p)
}
if ts, ok := raw["__REALTIME_TIMESTAMP"].(string); ok {
us, _ := strconv.ParseInt(ts, 10, 64)
e.T = us / 1000
}
if id, ok := raw["SYSLOG_IDENTIFIER"].(string); ok {
e.Ident = id
} else if c, ok := raw["_COMM"].(string); ok {
e.Ident = c
}
if pid, ok := raw["_PID"].(string); ok {
e.PID = pid
}
return e, true
}
func (m *Manager) journalArgs(unit string, n int) []string {
args := []string{"journalctl", "-u", unit, "-n", strconv.Itoa(n), "-o", "json", "--no-pager", "--all"}
if m.journalSudo {
args = priv.Wrap(args...)
}
return args
}
// Journal returns the last n entries.
func (m *Manager) Journal(ctx context.Context, unit string, n int) ([]Entry, error) {
if !Valid(unit) {
return nil, errors.New("invalid unit name")
}
out, err := run(ctx, m.journalArgs(unit, n)...)
if err != nil {
return nil, err
}
entries := []Entry{}
for _, line := range bytes.Split([]byte(out), []byte("\n")) {
if e, ok := parseEntry(line); ok {
entries = append(entries, e)
}
}
return entries, nil
}
// Tail streams entries (last n, then follow) to fn until ctx is cancelled.
func (m *Manager) Tail(ctx context.Context, unit string, n int, fn func(Entry)) error {
if !Valid(unit) {
return errors.New("invalid unit name")
}
args := append(m.journalArgs(unit, n), "-f")
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
// Own process group so the kill reaches journalctl itself, not just a sudo wrapper.
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error {
pg := "-" + strconv.Itoa(cmd.Process.Pid)
if m.journalSudo { // children are root's; only sudo can signal them
k := priv.Wrap("kill", "-9", "--", pg)
return exec.Command(k[0], k[1:]...).Run()
}
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
}
stdout, err := cmd.StdoutPipe()
if err != nil {
return err
}
if err := cmd.Start(); err != nil {
return err
}
sc := bufio.NewScanner(stdout)
sc.Buffer(make([]byte, 0, 64<<10), 4<<20)
for sc.Scan() {
if e, ok := parseEntry(sc.Bytes()); ok {
fn(e)
}
}
_ = cmd.Wait()
return nil
}
// Action runs start/stop/restart/reload/enable/disable/reset-failed.
func (m *Manager) Action(ctx context.Context, action, unit string) error {
if !Valid(unit) {
return errors.New("invalid unit name")
}
if !Actions[action] {
return errors.New("unknown action")
}
args := []string{"systemctl", action, unit, "--no-pager"}
if m.sudo {
args = priv.Wrap(args...)
}
_, err := run(ctx, args...)
m.invalidate()
return err
}
internal/services/tables_test.go
+14
View File
@@ -0,0 +1,14 @@
package services
import "testing"
func TestTables(t *testing.T) {
u := parseUnitTable(" UNIT LOAD ACTIVE SUB DESCRIPTION\n● bad.service loaded failed failed Bad thing here\n ssh.service loaded active running OpenBSD Secure Shell server\n\nLOAD = x\n12 loaded units listed.\n")
if len(u) != 2 || u[0].Unit != "bad.service" || u[0].Active != "failed" || u[1].Description != "OpenBSD Secure Shell server" {
t.Fatalf("%+v", u)
}
f := parseFileTable("UNIT FILE STATE VENDOR PRESET\nssh.service enabled enabled\nfoo@.service static -\n\n2 unit files listed.\n")
if len(f) != 2 || f[0].State != "enabled" {
t.Fatalf("%+v", f)
}
}
internal/setup/hamlab.openrc
+28
View File
@@ -0,0 +1,28 @@
#!/sbin/openrc-run
# Ham Lab web console (written by hamlab install)
name="hamlab"
description="Ham Lab web console"
export HAMLAB_CONFIG="@CONFIG@"
export HOME="@HOME@"
export USER="@USER@"
supervisor="supervise-daemon"
command="@BIN@"
command_args="serve"
command_user="@USER@:@GROUP@"
directory="@HOME@"
respawn_delay=3
respawn_max=0
output_log="/var/log/hamlab.log"
error_log="/var/log/hamlab.log"
depend() {
need localmount
after net firewall
}
start_pre() {
checkpath -f -o "@USER@:@GROUP@" -m 0640 /var/log/hamlab.log
}
internal/setup/hamlab.procd
+25
View File
@@ -0,0 +1,25 @@
#!/bin/sh /etc/rc.common
# Ham Lab web console (written by hamlab install)
USE_PROCD=1
# after network and firewall, before the tail of the boot
START=95
STOP=10
start_service() {
procd_open_instance hamlab
procd_set_param command "@BIN@" serve
procd_set_param env HAMLAB_CONFIG="@CONFIG@" HOME="@HOME@" USER="@USER@"
procd_set_param user "@USER@"
procd_set_param group "@GROUP@"
procd_set_param pidfile /var/run/hamlab.pid
# respawn: settle 3s, 5 retries inside that window, then give up counting
procd_set_param respawn 3600 3 0
procd_set_param stdout 1
procd_set_param stderr 1
procd_close_instance
}
service_triggers() {
procd_add_reload_trigger network
}
internal/setup/hamlab.service
+23
View File
@@ -0,0 +1,23 @@
[Unit]
Description=Ham Lab web console
After=network.target
# never give up restarting
StartLimitIntervalSec=0
[Service]
Type=simple
User=@USER@
Group=@GROUP@
# read the system journal without sudo (keeps the daemon's own log clean)
SupplementaryGroups=@GROUPS@
WorkingDirectory=-@HOME@
Environment=HAMLAB_CONFIG=@CONFIG@
ExecStart=@BIN@ serve
Restart=always
RestartSec=3
KillMode=control-group
TimeoutStopSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
internal/setup/setup.go
+718
View File
@@ -0,0 +1,718 @@
// Package setup is `hamlab install` and `hamlab uninstall`: the binary installs itself, so
// a new machine needs nothing but this one file. It copies itself into place, writes the
// config, sets the login password and registers a service with whichever init the host
// runs (systemd or OpenRC), as root or through sudo/doas.
package setup
import (
"bufio"
"bytes"
"crypto/rand"
"crypto/tls"
_ "embed"
"errors"
"flag"
"fmt"
"io"
"net"
"net/http"
"os"
"os/exec"
"os/user"
"path/filepath"
"strconv"
"strings"
"syscall"
"time"
"golang.org/x/term"
"hamlab/internal/auth"
"hamlab/internal/config"
)
//go:embed hamlab.service
var systemdUnit string
//go:embed hamlab.openrc
var openrcScript string
//go:embed hamlab.procd
var procdScript string
const (
unitPath = "/etc/systemd/system/hamlab.service"
openrcPath = "/etc/init.d/hamlab"
// procd (OpenWrt) uses the same path as OpenRC; the two are told apart by content.
procdPath = "/etc/init.d/hamlab"
sudoersPath = "/etc/sudoers.d/hamlab-quiet"
)
type options struct {
user, listen, tls, prefix, data string
passwordStdin, openFirewall bool
noStart bool
}
func say(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) }
// elevate re-runs this command as root through sudo or doas (which may prompt), naming
// the invoking user so the service still runs as them.
func elevate(sub string, args []string, keepUser bool) error {
self, err := os.Executable()
if err != nil {
return err
}
for _, tool := range []string{"sudo", "doas"} {
path, err := exec.LookPath(tool)
if err != nil {
continue
}
full := []string{tool, self, sub}
if u, err := user.Current(); err == nil && keepUser && !hasFlag(args, "user") {
full = append(full, "--user", u.Username)
}
full = append(full, args...)
return syscall.Exec(path, full, os.Environ())
}
return errors.New("this needs root, and neither sudo nor doas is installed: run it as root")
}
func hasFlag(args []string, name string) bool {
for _, a := range args {
a = strings.TrimLeft(a, "-")
if a == name || strings.HasPrefix(a, name+"=") {
return true
}
}
return false
}
// Install is `hamlab install [flags]`.
func Install(args []string, version string) error {
var o options
fs := flag.NewFlagSet("install", flag.ContinueOnError)
fs.StringVar(&o.user, "user", "", "user the console runs as; terminals are this user's shell (default: you)")
fs.StringVar(&o.listen, "listen", "", "listen address, e.g. 0.0.0.0:8100 (default: keep, else 127.0.0.1:8100)")
fs.StringVar(&o.tls, "tls", "", `"auto" serves HTTPS with a self-signed certificate, "off" plain HTTP (default: keep)`)
fs.StringVar(&o.prefix, "prefix", "/usr/local/bin", "directory the binary is installed into")
fs.StringVar(&o.data, "data", "", "config and state directory (default: ~/.config/hamlab of the user)")
fs.BoolVar(&o.passwordStdin, "password-stdin", false, "read the login password from stdin instead of prompting")
fs.BoolVar(&o.openFirewall, "open-firewall", false, "open the listen port in firewalld/ufw when one is active")
fs.BoolVar(&o.noStart, "no-start", false, "install and enable, but don't start the service")
if err := fs.Parse(args); err != nil {
return err
}
if os.Geteuid() != 0 {
return elevate("install", args, true)
}
sbinPath()
// who
name := o.user
for _, env := range []string{"SUDO_USER", "DOAS_USER"} {
if name == "" {
name = os.Getenv(env)
}
}
if name == "" {
name = "root"
}
acct, err := lookupUser(name)
if err != nil {
return err
}
// where
dataDir := o.data
if dataDir == "" {
dataDir = filepath.Dir(existingConfig(acct.home))
}
cfgPath := filepath.Join(dataDir, "config.json")
// binary
dest := filepath.Join(o.prefix, "hamlab")
self, err := os.Executable()
if err != nil {
return err
}
self, _ = filepath.EvalSymlinks(self)
if err := os.MkdirAll(o.prefix, 0o755); err != nil {
return err
}
if self != dest {
if err := CopyAtomic(self, dest); err != nil {
return fmt.Errorf("install binary: %w", err)
}
// builds for other architectures sit beside it so a hub can push them to matching machines
sibs, _ := filepath.Glob(filepath.Join(filepath.Dir(self), "hamlab-linux-*"))
for _, sib := range sibs {
arch := strings.TrimPrefix(filepath.Base(sib), "hamlab-linux-")
if sib == self || strings.Contains(arch, ".") {
continue
}
if err := CopyAtomic(sib, dest+"-"+arch); err != nil {
say("warning: %s: %v", sib, err)
}
}
}
restorecon(dest)
say("binary %s (%s)", dest, version)
// config + password
cfg, err := config.Load(cfgPath)
if err != nil {
return err
}
if o.listen != "" {
if _, _, err := net.SplitHostPort(o.listen); err != nil {
return fmt.Errorf("--listen %q: want host:port", o.listen)
}
cfg.Listen = o.listen
}
switch o.tls {
case "":
case "auto":
cfg.TLS = "auto"
case "off", "none":
cfg.TLS = ""
default:
return fmt.Errorf("--tls %q: want auto or off", o.tls)
}
generated := ""
cfg.MigrateUsers(acct.name)
if len(cfg.Users) == 0 || o.passwordStdin {
pw, gen, err := choosePassword(o.passwordStdin)
if err != nil {
return err
}
h, err := auth.HashPassword(pw)
if err != nil {
return err
}
// the first account is named after the OS user the service runs as; --password-stdin
// on an existing install resets that first account
if len(cfg.Users) == 0 {
cfg.Users = []config.User{{Name: acct.name, Hash: h, Created: time.Now()}}
} else {
cfg.Users[0].Hash = h
}
if gen {
generated = pw
}
}
made := firstMissing(dataDir) // e.g. ~/.config, when this is the first thing to use it
if err := cfg.Save(); err != nil {
return err
}
if err := chownData(dataDir, made, acct); err != nil {
return err
}
say("config %s", cfgPath)
// service
vars := map[string]string{"@USER@": acct.name, "@GROUP@": acct.group, "@HOME@": acct.home, "@BIN@": dest, "@CONFIG@": cfgPath,
"@GROUPS@": journalGroups()}
initSys := ""
switch {
case isDir("/run/systemd/system"):
initSys = "systemd"
unit := fill(systemdUnit, vars)
if vars["@GROUPS@"] == "" {
unit = dropLines(unit, "SupplementaryGroups=")
}
if err := os.WriteFile(unitPath, []byte(unit), 0o644); err != nil {
return err
}
restorecon(unitPath)
if err := runAll([][]string{{"systemctl", "daemon-reload"}, {"systemctl", "enable", "hamlab.service"}}); err != nil {
return err
}
if !o.noStart {
if err := run("systemctl", "restart", "hamlab.service"); err != nil {
return err
}
}
case exists("/sbin/procd") && exists("/etc/rc.common"):
initSys = "procd"
if err := os.WriteFile(procdPath, []byte(fill(procdScript, vars)), 0o755); err != nil {
return err
}
if err := run(procdPath, "enable"); err != nil {
say("warning: %s enable: %v", procdPath, err)
}
if !o.noStart {
if err := run(procdPath, "restart"); err != nil {
return err
}
}
case exists("/sbin/openrc-run") || exists("/usr/sbin/openrc-run") || have("rc-service"):
initSys = "OpenRC"
if err := os.WriteFile(openrcPath, []byte(fill(openrcScript, vars)), 0o755); err != nil {
return err
}
if err := run("rc-update", "add", "hamlab", "default"); err != nil {
say("warning: rc-update: %v", err)
}
if !o.noStart {
if err := run("rc-service", "hamlab", "restart"); err != nil {
return err
}
}
}
if initSys != "" {
say("service hamlab (%s), runs as %s, starts at boot", initSys, acct.name)
}
if acct.name != "root" {
quietSudoers()
}
port := firewall(cfg.Listen, o.openFirewall)
scheme := "http"
if cfg.TLS == "auto" {
scheme = "https"
}
url := scheme + "://" + browseAddr(cfg.Listen)
switch {
case initSys == "":
say("\nNo systemd, procd or OpenRC here, so nothing starts it at boot.\nRun it under your own supervisor, as %s:\n HAMLAB_CONFIG=%s %s serve", acct.name, cfgPath, dest)
case o.noStart:
case healthy(cfg.Listen, cfg.TLS == "auto"):
say("\nHam Lab is up: %s", url)
default:
say("\nInstalled, but %s isn't answering yet. Check: %s", url, map[string]string{"systemd": "journalctl -u hamlab -n 30", "OpenRC": "tail /var/log/hamlab.log"}[initSys])
}
if generated != "" {
say("Login: user %s, password (shown once; change it in Settings): %s", cfg.Users[0].Name, generated)
}
if host, _, _ := net.SplitHostPort(cfg.Listen); isLoopback(host) {
say("It only listens on this machine. For LAN access: hamlab install --listen 0.0.0.0:%s", port)
}
return nil
}
// Uninstall is `hamlab uninstall [--purge]`.
func Uninstall(args []string) error {
fs := flag.NewFlagSet("uninstall", flag.ContinueOnError)
purge := fs.Bool("purge", false, "also delete the config and state directory")
prefix := fs.String("prefix", "/usr/local/bin", "directory the binary was installed into")
if err := fs.Parse(args); err != nil {
return err
}
if os.Geteuid() != 0 {
return elevate("uninstall", args, false)
}
sbinPath()
dataDir := ""
if b, err := os.ReadFile(unitPath); err == nil {
dataDir = configDirFrom(string(b), "Environment=HAMLAB_CONFIG=")
_ = run("systemctl", "disable", "--now", "hamlab.service")
os.Remove(unitPath)
_ = run("systemctl", "daemon-reload")
}
if b, err := os.ReadFile(openrcPath); err == nil {
if strings.Contains(string(b), "USE_PROCD=1") {
dataDir = configDirFrom(string(b), "HAMLAB_CONFIG=")
_ = run(procdPath, "stop")
_ = run(procdPath, "disable")
} else {
dataDir = configDirFrom(string(b), "export HAMLAB_CONFIG=")
_ = run("rc-service", "hamlab", "stop")
_ = run("rc-update", "del", "hamlab", "default")
}
os.Remove(openrcPath)
}
os.Remove(sudoersPath)
bins, _ := filepath.Glob(filepath.Join(*prefix, "hamlab-*"))
for _, b := range append(bins, filepath.Join(*prefix, "hamlab")) {
os.Remove(b)
}
say("Ham Lab removed.")
switch {
case dataDir == "":
case *purge:
if err := purgeData(dataDir); err != nil {
return err
}
default:
say("Config and state kept in %s (uninstall --purge deletes them).", dataDir)
}
return nil
}
// sbinPath makes sure the admin tools resolve: a root shell reached with plain `su` keeps
// the user's PATH, which on Debian has no sbin in it.
func sbinPath() {
os.Setenv("PATH", os.Getenv("PATH")+":/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin")
}
// ownFiles is everything Ham Lab writes into its data directory.
var ownFiles = []string{"config.json", "config.json.tmp", "state.json", "state.json.tmp", "state.json.corrupt", "cert.pem", "key.pem", "hamlab.prev", "hamlab.new"}
// purgeData deletes Ham Lab's data. The whole directory goes only when it is unmistakably
// ours (named "hamlab"); a --data pointed at a shared directory such as a home loses
// just the files Ham Lab wrote.
func purgeData(dir string) error {
if filepath.Base(dir) == "hamlab" {
say("Deleted %s.", dir)
return os.RemoveAll(dir)
}
for _, f := range ownFiles {
os.Remove(filepath.Join(dir, f))
}
say("Deleted Ham Lab's files in %s (the directory isn't Ham Lab's alone, so it stays).", dir)
return nil
}
func configDirFrom(text, key string) string {
for _, line := range strings.Split(text, "\n") {
if v, ok := strings.CutPrefix(strings.TrimSpace(line), key); ok {
if v = strings.Trim(v, `"'`); strings.HasSuffix(v, "/config.json") && filepath.Dir(v) != "/" {
return filepath.Dir(v)
}
}
}
return ""
}
// ---- pieces ----
type account struct {
name, group, home string
uid, gid int
}
func lookupUser(name string) (account, error) {
a := account{name: name}
if u, err := user.Lookup(name); err == nil {
a.uid, _ = strconv.Atoi(u.Uid)
a.gid, _ = strconv.Atoi(u.Gid)
a.home = u.HomeDir
} else if out, err2 := exec.Command("getent", "passwd", name).Output(); err2 == nil { // LDAP/SSSD accounts
f := strings.Split(strings.TrimSpace(string(out)), ":")
if len(f) < 6 {
return a, fmt.Errorf("no such user: %s", name)
}
a.uid, _ = strconv.Atoi(f[2])
a.gid, _ = strconv.Atoi(f[3])
a.home = f[5]
} else {
return a, fmt.Errorf("no such user: %s", name)
}
if a.home == "" || !isDir(a.home) {
return a, fmt.Errorf("user %s has no home directory (%q); pick another with --user", name, a.home)
}
a.group = strconv.Itoa(a.gid)
if g, err := user.LookupGroupId(a.group); err == nil {
a.group = g.Name
}
return a, nil
}
// existingConfig returns the config path for a home directory: whichever already exists,
// else the default for a new install.
func existingConfig(home string) string {
for _, p := range config.Candidates(home) {
if exists(p) {
return p
}
}
return config.Candidates(home)[0]
}
// journalGroups lists the groups that can read the system journal and exist here. A unit
// naming a group the host doesn't have fails to start, and "adm" is not universal.
func journalGroups() string {
var have []string
for _, g := range []string{"systemd-journal", "adm"} {
if _, err := user.LookupGroup(g); err == nil {
have = append(have, g)
}
}
return strings.Join(have, " ")
}
func choosePassword(fromStdin bool) (pw string, generated bool, err error) {
switch {
case fromStdin:
line, _ := bufio.NewReader(os.Stdin).ReadString('\n')
pw = strings.TrimRight(line, "\r\n")
case term.IsTerminal(int(os.Stdin.Fd())):
fmt.Fprint(os.Stderr, "Login password for the web console (12+ characters, empty to generate one): ")
p1, err := term.ReadPassword(int(os.Stdin.Fd()))
fmt.Fprintln(os.Stderr)
if err != nil {
return "", false, err
}
if len(p1) > 0 {
fmt.Fprint(os.Stderr, "Again: ")
p2, err := term.ReadPassword(int(os.Stdin.Fd()))
fmt.Fprintln(os.Stderr)
if err != nil {
return "", false, err
}
if !bytes.Equal(p1, p2) {
return "", false, errors.New("passwords do not match")
}
}
pw = string(p1)
}
if pw == "" && !fromStdin {
return randomPassword(), true, nil
}
if len(pw) < 12 {
return "", false, errors.New("password must be at least 12 characters")
}
return pw, false, nil
}
func randomPassword() string {
const alphabet = "abcdefghijkmnpqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ23456789" // no look-alikes
b := make([]byte, 20)
_, _ = rand.Read(b)
for i := range b {
b[i] = alphabet[int(b[i])%len(alphabet)]
}
return string(b)
}
// CopyAtomic puts src at dst with mode 0755 by writing beside it and renaming, so dst is
// always either the old file or the whole new one, and a running dst is never written to.
func CopyAtomic(src, dst string) error {
in, err := os.Open(src)
if err != nil {
return err
}
defer in.Close()
tmp := filepath.Join(filepath.Dir(dst), "."+filepath.Base(dst)+".new")
out, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755)
if err != nil {
return err
}
_, err = io.Copy(out, in)
if err == nil {
err = out.Sync()
}
if cerr := out.Close(); err == nil {
err = cerr
}
if err == nil {
err = os.Chmod(tmp, 0o755)
}
if err == nil {
err = os.Rename(tmp, dst)
}
if err != nil {
os.Remove(tmp)
}
return err
}
// firstMissing returns the topmost directory that creating dir would have to make, or "".
func firstMissing(dir string) string {
missing := ""
for d := filepath.Clean(dir); d != "/" && d != "." && !exists(d); d = filepath.Dir(d) {
missing = d
}
return missing
}
// chownData hands Ham Lab's data to the account that runs it. Directories this install
// created are ours to hand over whole. In one that already existed (a --data pointed at
// something shared), only the files Ham Lab writes change owner: a recursive chown of
// somebody else's tree can't be undone.
func chownData(dataDir, made string, a account) error {
if made != "" {
return chownTree(made, a)
}
if filepath.Base(dataDir) == "hamlab" {
return chownTree(dataDir, a)
}
for _, f := range ownFiles {
if p := filepath.Join(dataDir, f); exists(p) {
if err := os.Lchown(p, a.uid, a.gid); err != nil {
return err
}
}
}
return nil
}
func chownTree(dir string, a account) error {
return filepath.Walk(dir, func(p string, _ os.FileInfo, err error) error {
if err != nil {
return err
}
return os.Lchown(p, a.uid, a.gid)
})
}
func fill(tmpl string, vars map[string]string) string {
for k, v := range vars {
tmpl = strings.ReplaceAll(tmpl, k, v)
}
return tmpl
}
func dropLines(text, prefix string) string {
var keep []string
for _, l := range strings.Split(text, "\n") {
if !strings.HasPrefix(l, prefix) {
keep = append(keep, l)
}
}
return strings.Join(keep, "\n")
}
// quietSudoers keeps the daemon's read-only polling (SMART, UPS, journal) out of the auth
// log: at 250 ms refresh rates the sudo session lines would otherwise dominate it.
func quietSudoers() {
if !isDir("/etc/sudoers.d") || !have("visudo") {
return
}
var cmds []string
for _, bin := range []string{"smartctl", "pwrstat", "journalctl", "tail", "cat", "ls", "crontab"} {
for _, dir := range []string{"/usr/sbin", "/usr/bin", "/sbin", "/bin"} {
if exists(dir + "/" + bin) {
cmds = append(cmds, dir+"/"+bin)
}
}
}
for _, bin := range []string{"zpool", "zfs"} { // listing only: scrubs and snapshots still log
for _, dir := range []string{"/usr/sbin", "/sbin", "/usr/bin", "/bin"} {
if exists(dir + "/" + bin) {
cmds = append(cmds, dir+"/"+bin+" list *", dir+"/"+bin+" status *")
}
}
}
if len(cmds) == 0 {
return
}
text := "# Ham Lab: don't log the daemon's read-only polling commands (written by hamlab install)\n" +
"Cmnd_Alias HAMLAB_QUIET = " + strings.Join(cmds, ", ") + "\n" +
"Defaults!HAMLAB_QUIET !syslog, !pam_session, !pam_setcred\n"
tmp := sudoersPath + ".new"
if os.WriteFile(tmp, []byte(text), 0o440) != nil {
return
}
if exec.Command("visudo", "-cf", tmp).Run() != nil {
os.Remove(tmp)
say("warning: sudoers snippet failed validation; skipped")
return
}
_ = os.Rename(tmp, sudoersPath)
}
// firewall reports (and with open, fixes) an active host firewall in front of a
// non-loopback listen address. It returns the port.
func firewall(listen string, open bool) string {
host, port, _ := net.SplitHostPort(listen)
if isLoopback(host) {
return port
}
switch {
case have("firewall-cmd") && exec.Command("firewall-cmd", "--state").Run() == nil:
if open {
if err := runAll([][]string{{"firewall-cmd", "--quiet", "--permanent", "--add-port=" + port + "/tcp"}, {"firewall-cmd", "--quiet", "--reload"}}); err != nil {
say("warning: firewalld: %v", err)
} else {
say("firewall firewalld: opened %s/tcp", port)
}
} else {
say("firewall firewalld is active and will block port %s. Open it: hamlab install --open-firewall", port)
}
case have("ufw"):
out, _ := exec.Command("ufw", "status").Output()
if !bytes.Contains(out, []byte("Status: active")) {
break
}
if open {
if err := run("ufw", "allow", port+"/tcp"); err != nil {
say("warning: ufw: %v", err)
} else {
say("firewall ufw: opened %s/tcp", port)
}
} else {
say("firewall ufw is active and will block port %s. Open it: hamlab install --open-firewall", port)
}
}
return port
}
func isLoopback(host string) bool {
ip := net.ParseIP(host)
return host == "localhost" || (ip != nil && ip.IsLoopback())
}
// browseAddr turns a listen address into one a browser can open.
func browseAddr(listen string) string {
host, port, err := net.SplitHostPort(listen)
if err != nil {
return listen
}
if ip := net.ParseIP(host); host == "" || (ip != nil && ip.IsUnspecified()) {
host = "localhost"
if addrs, err := net.InterfaceAddrs(); err == nil {
for _, a := range addrs {
if n, ok := a.(*net.IPNet); ok && n.IP.To4() != nil && !n.IP.IsLoopback() {
host = n.IP.String()
break
}
}
}
}
return net.JoinHostPort(host, port)
}
// healthy polls /healthz for up to 10 s.
func healthy(listen string, https bool) bool {
host, port, _ := net.SplitHostPort(listen)
if ip := net.ParseIP(host); host == "" || (ip != nil && ip.IsUnspecified()) {
host = "127.0.0.1"
}
scheme := "http"
if https {
scheme = "https"
}
c := &http.Client{Timeout: 2 * time.Second, Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}} // self-signed; reachability only
for i := 0; i < 20; i++ {
if resp, err := c.Get(scheme + "://" + net.JoinHostPort(host, port) + "/healthz"); err == nil {
resp.Body.Close()
return resp.StatusCode < 500
}
time.Sleep(500 * time.Millisecond)
}
return false
}
// restorecon fixes the SELinux label of a file we created, where SELinux exists.
func restorecon(path string) {
if have("restorecon") {
_ = exec.Command("restorecon", "-F", path).Run()
}
}
func run(args ...string) error {
out, err := exec.Command(args[0], args[1:]...).CombinedOutput()
if err != nil {
if msg := strings.TrimSpace(string(out)); msg != "" {
return fmt.Errorf("%s: %s", strings.Join(args, " "), msg)
}
return fmt.Errorf("%s: %w", strings.Join(args, " "), err)
}
return nil
}
func runAll(cmds [][]string) error {
for _, c := range cmds {
if err := run(c...); err != nil {
return err
}
}
return nil
}
func have(bin string) bool { _, err := exec.LookPath(bin); return err == nil }
func exists(p string) bool { _, err := os.Stat(p); return err == nil }
func isDir(p string) bool { st, err := os.Stat(p); return err == nil && st.IsDir() }
internal/setup/setup_test.go
+73
View File
@@ -0,0 +1,73 @@
package setup
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestUnitWithoutJournalGroups(t *testing.T) {
vars := map[string]string{"@USER@": "ham", "@GROUP@": "ham", "@HOME@": "/home/ham", "@BIN@": "/usr/local/bin/hamlab",
"@CONFIG@": "/home/ham/.config/hamlab/config.json", "@GROUPS@": ""}
unit := dropLines(fill(systemdUnit, vars), "SupplementaryGroups=")
if strings.Contains(unit, "SupplementaryGroups") || strings.Contains(unit, "@") {
t.Fatalf("unit still has a groups line or an unfilled placeholder:\n%s", unit)
}
if !strings.Contains(unit, "ExecStart=/usr/local/bin/hamlab serve") {
t.Fatalf("ExecStart missing:\n%s", unit)
}
}
func TestConfigDirFrom(t *testing.T) {
for in, want := range map[string]string{
"Environment=HAMLAB_CONFIG=/home/ham/.config/hamlab/config.json": "/home/ham/.config/hamlab",
`export HAMLAB_CONFIG="/root/.config/hamlab/config.json"`: "/root/.config/hamlab",
"Environment=HAMLAB_CONFIG=/config.json": "",
"Environment=HAMLAB_CONFIG=/etc/passwd": "",
} {
key := "Environment=HAMLAB_CONFIG="
if strings.HasPrefix(in, "export") {
key = "export HAMLAB_CONFIG="
}
if got := configDirFrom(in, key); got != want {
t.Errorf("%q: got %q want %q", in, got, want)
}
}
}
// --purge must never take a shared directory with it.
func TestPurgeSparesSharedDirectory(t *testing.T) {
home := t.TempDir()
os.WriteFile(filepath.Join(home, "config.json"), []byte("{}"), 0o600)
os.WriteFile(filepath.Join(home, "thesis.txt"), []byte("irreplaceable"), 0o600)
if err := purgeData(home); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(filepath.Join(home, "thesis.txt")); err != nil {
t.Fatal("purge deleted a file that wasn't Ham Lab's")
}
if _, err := os.Stat(filepath.Join(home, "config.json")); err == nil {
t.Fatal("purge left config.json behind")
}
own := filepath.Join(t.TempDir(), "hamlab")
os.MkdirAll(own, 0o700)
os.WriteFile(filepath.Join(own, "config.json"), []byte("{}"), 0o600)
if err := purgeData(own); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(own); err == nil {
t.Fatal("purge left Ham Lab's own directory behind")
}
}
func TestFirstMissing(t *testing.T) {
base := t.TempDir()
if got := firstMissing(filepath.Join(base, "a", "b", "c")); got != filepath.Join(base, "a") {
t.Errorf("got %q", got)
}
if got := firstMissing(base); got != "" {
t.Errorf("existing dir: got %q", got)
}
}
internal/space/codec.go
+147
View File
@@ -0,0 +1,147 @@
package space
import (
"bufio"
"encoding/binary"
"errors"
"io"
)
// The tree crosses from the helper process to the daemon as a preorder stream of varints:
// a few bytes per node, decoded without reflection, so a million nodes take well under a
// second either way.
const codecMagic = "HLSPACE2"
func Encode(w io.Writer, root *Node) error {
bw := bufio.NewWriterSize(w, 1<<20)
if _, err := bw.WriteString(codecMagic); err != nil {
return err
}
var buf [binary.MaxVarintLen64]byte
putU := func(v uint64) error { n := binary.PutUvarint(buf[:], v); _, err := bw.Write(buf[:n]); return err }
putI := func(v int64) error { n := binary.PutVarint(buf[:], v); _, err := bw.Write(buf[:n]); return err }
var enc func(n *Node) error
enc = func(n *Node) error {
if err := putU(uint64(len(n.Name))); err != nil {
return err
}
if _, err := bw.WriteString(n.Name); err != nil {
return err
}
flags := uint64(0)
if n.Dir {
flags |= 1
}
if n.Mount {
flags |= 2
}
if n.Denied {
flags |= 4
}
if n.Partial {
flags |= 8
}
for _, f := range []func() error{
func() error { return putU(flags) },
func() error { return putI(n.Size) },
func() error { return putI(n.Apparent) },
func() error { return putI(n.MTime) },
func() error { return putI(n.MTimeNs) },
func() error { return putU(uint64(n.Files)) },
func() error { return putU(uint64(n.Dirs)) },
func() error { return putU(uint64(n.SmallN)) },
func() error { return putI(n.SmallSize) },
func() error { return putU(uint64(len(n.Children))) },
} {
if err := f(); err != nil {
return err
}
}
for _, c := range n.Children {
if err := enc(c); err != nil {
return err
}
}
return nil
}
if err := enc(root); err != nil {
return err
}
return bw.Flush()
}
func Decode(r io.Reader) (*Node, error) {
br := bufio.NewReaderSize(r, 1<<20)
magic := make([]byte, len(codecMagic))
if _, err := io.ReadFull(br, magic); err != nil || string(magic) != codecMagic {
return nil, errors.New("space: not a scan stream")
}
var dec func() (*Node, error)
dec = func() (*Node, error) {
l, err := binary.ReadUvarint(br)
if err != nil {
return nil, err
}
if l > 1<<16 {
return nil, errors.New("space: bad name length")
}
name := make([]byte, l)
if _, err := io.ReadFull(br, name); err != nil {
return nil, err
}
n := &Node{Name: string(name)}
flags, err := binary.ReadUvarint(br)
if err != nil {
return nil, err
}
n.Dir, n.Mount, n.Denied, n.Partial = flags&1 != 0, flags&2 != 0, flags&4 != 0, flags&8 != 0
if n.Size, err = binary.ReadVarint(br); err != nil {
return nil, err
}
if n.Apparent, err = binary.ReadVarint(br); err != nil {
return nil, err
}
if n.MTime, err = binary.ReadVarint(br); err != nil {
return nil, err
}
if n.MTimeNs, err = binary.ReadVarint(br); err != nil {
return nil, err
}
u, err := binary.ReadUvarint(br)
if err != nil {
return nil, err
}
n.Files = uint32(u)
if u, err = binary.ReadUvarint(br); err != nil {
return nil, err
}
n.Dirs = uint32(u)
if u, err = binary.ReadUvarint(br); err != nil {
return nil, err
}
n.SmallN = uint32(u)
if n.SmallSize, err = binary.ReadVarint(br); err != nil {
return nil, err
}
cnt, err := binary.ReadUvarint(br)
if err != nil {
return nil, err
}
if cnt > 1<<24 {
return nil, errors.New("space: bad child count")
}
if cnt > 0 {
n.Children = make([]*Node, 0, cnt)
for i := uint64(0); i < cnt; i++ {
c, err := dec()
if err != nil {
return nil, err
}
n.Children = append(n.Children, c)
}
}
return n, nil
}
return dec()
}
internal/space/manager.go
+735
View File
@@ -0,0 +1,735 @@
package space
import (
"bufio"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"os"
"os/exec"
"os/signal"
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"syscall"
"time"
"hamlab/internal/priv"
)
// ---- the helper: `hamlab space-walk <root>…`, run by the daemon (as root when it can) ----
//
// The walk runs in its own process so that root can read every directory, so that the
// memory of the walk is given back when it ends, and so that a runaway scan can be killed
// without touching the console. The daemon writes a header line to the helper's stdin
// (how each root is to be walked, and what changed) followed by the last scan's tree when
// there is one, then holds stdin open: EOF means the daemon has gone or wants the walk
// stopped. Progress goes to stderr as JSON lines; a result line and the tree to stdout.
type header struct {
Prev bool `json:"prev"`
Modes map[string]Mode `json:"modes,omitempty"`
Dirty []string `json:"dirty,omitempty"`
Touched []string `json:"touched,omitempty"`
}
type resultLine struct {
Complete bool `json:"complete"`
Partial int64 `json:"partial"`
Reused int64 `json:"reused"`
Read int64 `json:"read"`
}
func Helper(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
if len(args) == 0 {
fmt.Fprintln(stderr, "usage: hamlab space-walk <mountpoint>...")
return 2
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP)
go func() { <-sig; cancel() }()
in := bufio.NewReaderSize(stdin, 1<<20)
line, err := in.ReadBytes('\n')
if err != nil {
fmt.Fprintln(stderr, "error: no header:", err)
return 2
}
var h header
if err := json.Unmarshal(line, &h); err != nil {
fmt.Fprintln(stderr, "error: bad header:", err)
return 2
}
opts := Options{Modes: h.Modes, Dirty: set(h.Dirty), Touched: set(h.Touched)}
if h.Prev {
if opts.Prev, err = Decode(in); err != nil {
fmt.Fprintln(stderr, "error: last tree:", err)
return 2
}
}
go func() { _, _ = io.Copy(io.Discard, in); cancel() }() // the daemon went away, or asked us to stop
enc := json.NewEncoder(stderr)
res, err := Walk(ctx, args, opts, func(p Progress) { _ = enc.Encode(p) })
if err != nil && !errors.Is(err, context.Canceled) {
fmt.Fprintln(stderr, "error:", err)
return 1
}
rl, _ := json.Marshal(resultLine{Complete: res.Complete, Partial: res.Partial, Reused: res.Reused, Read: res.Read})
if _, err := stdout.Write(append(rl, '\n')); err != nil {
return 1
}
if err := Encode(stdout, res.Root); err != nil {
fmt.Fprintln(stderr, "error:", err)
return 1
}
if !res.Complete {
return 3
}
return 0
}
func set(list []string) map[string]bool {
if len(list) == 0 {
return nil
}
m := make(map[string]bool, len(list))
for _, s := range list {
m[s] = true
}
return m
}
// ---- the daemon side: one scan per target, the last result kept, on disk too ----
type Scan struct {
Disk string `json:"disk"`
Roots []string `json:"roots"`
Started time.Time `json:"started"`
Finished time.Time `json:"finished,omitempty"`
Running bool `json:"running"`
Loading bool `json:"loading,omitempty"` // read back from disk after a restart, not yet decoded
Error string `json:"error,omitempty"`
Progress Progress `json:"progress"`
AsRoot bool `json:"as_root"`
// How this scan was done: "full", or "refresh" / "resume" from the one before, and by what
// means per root ("zfs diff" or "mtime").
Method string `json:"method,omitempty"`
ByRoot map[string]string `json:"by_root,omitempty"`
Partial bool `json:"partial,omitempty"` // cancelled before it finished; Scan again resumes it
// once finished:
Files uint32 `json:"files"`
Dirs uint32 `json:"dirs"`
Size int64 `json:"size"`
Denied int64 `json:"denied"`
Read int64 `json:"read"` // directories read this time
Reused int64 `json:"reused"` // directories kept from the scan before
PartialDirs int64 `json:"partial_dirs"` // directories not read (cancelled)
// Bases is, per root that is a ZFS dataset, the newest snapshot that existed when the
// scan started: the next scan asks `zfs diff` what changed since it.
Bases map[string]string `json:"bases,omitempty"`
root *Node
largest []File
cancel context.CancelFunc
}
type Manager struct {
self string // this binary, for the helper
dir string // where results are kept between restarts
mu sync.Mutex
scans map[string]*Scan
}
// New makes a manager that keeps results under dir and reads back the ones there.
func New(self, dir string) *Manager {
m := &Manager{self: self, dir: dir, scans: map[string]*Scan{}}
m.loadAll()
return m
}
func (m *Manager) fileFor(id, ext string) string {
safe := strings.NewReplacer(":", "_", "/", "_", "\\", "_").Replace(id)
return filepath.Join(m.dir, safe+ext)
}
// loadAll reads the saved scans' metadata now and their trees in the background, one at a
// time, so a restart does not double the memory of the last scans all at once.
func (m *Manager) loadAll() {
metas, _ := filepath.Glob(filepath.Join(m.dir, "*.json"))
var pending []*Scan
for _, mp := range metas {
b, err := os.ReadFile(mp)
if err != nil {
continue
}
s := &Scan{}
if json.Unmarshal(b, s) != nil || s.Disk == "" {
continue
}
s.Running, s.cancel = false, nil
s.Loading = true
m.scans[s.Disk] = s
pending = append(pending, s)
}
if len(pending) == 0 {
return
}
go func() {
for _, s := range pending {
root, err := m.readTree(s.Disk)
m.mu.Lock()
if err != nil {
log.Printf("space: saved scan of %s unreadable (%v); dropped", s.Disk, err)
if m.scans[s.Disk] == s {
delete(m.scans, s.Disk)
}
} else if m.scans[s.Disk] == s {
s.root, s.Loading = root, false
}
m.mu.Unlock()
}
}()
}
func (m *Manager) readTree(id string) (*Node, error) {
f, err := os.Open(m.fileFor(id, ".scan"))
if err != nil {
return nil, err
}
defer f.Close()
br := bufio.NewReaderSize(f, 1<<20)
if _, err := br.ReadBytes('\n'); err != nil { // the result line
return nil, err
}
return Decode(br)
}
func (m *Manager) saveMeta(s *Scan) {
if m.dir == "" {
return
}
_ = os.MkdirAll(m.dir, 0o700)
b, err := json.MarshalIndent(s, "", " ")
if err == nil {
err = writeAtomic(m.fileFor(s.Disk, ".json"), b)
}
if err != nil {
log.Printf("space: save %s: %v", s.Disk, err)
}
}
func writeAtomic(path string, b []byte) error {
tmp := path + ".tmp"
if err := os.WriteFile(tmp, b, 0o600); err != nil {
return err
}
return os.Rename(tmp, path)
}
// Status is the scan of id (a copy without the tree), or nil.
func (m *Manager) Status(id string) *Scan {
m.mu.Lock()
defer m.mu.Unlock()
return m.copyOf(m.scans[id])
}
func (m *Manager) copyOf(s *Scan) *Scan {
if s == nil {
return nil
}
c := *s
c.root, c.largest, c.cancel = nil, nil, nil
return &c
}
func (m *Manager) All() map[string]*Scan {
m.mu.Lock()
defer m.mu.Unlock()
out := map[string]*Scan{}
for k, s := range m.scans {
out[k] = m.copyOf(s)
}
return out
}
// Start scans the roots of id in the background, from the last scan of it when there is
// one. datasets names the ZFS dataset behind each root that has one.
func (m *Manager) Start(id string, roots []string, datasets map[string]string) (*Scan, error) {
if len(roots) == 0 {
return nil, errors.New("nothing is mounted from this disk")
}
m.mu.Lock()
prev := m.scans[id]
if prev != nil && prev.Running {
m.mu.Unlock()
return nil, errors.New("a scan of this disk is already running")
}
if prev != nil && (prev.root == nil || prev.Loading) {
prev = nil // nothing to start from
}
ctx, cancel := context.WithCancel(context.Background())
s := &Scan{Disk: id, Roots: roots, Started: time.Now(), Running: true, AsRoot: priv.Available(), cancel: cancel, Bases: map[string]string{}}
if prev != nil {
// the new scan takes the old one's place in the list while it runs, but keeps its
// tree, so the pages still answer until the new one is in
s.root, s.largest = prev.root, prev.largest
s.Files, s.Dirs, s.Size = prev.Files, prev.Dirs, prev.Size
}
m.scans[id] = s
m.mu.Unlock()
go m.run(ctx, s, prev, datasets)
return m.Status(id), nil
}
// plan decides how each root is walked this time, given the last scan.
func (m *Manager) plan(ctx context.Context, s, prev *Scan, datasets map[string]string) (header, Options) {
h := header{Modes: map[string]Mode{}}
opts := Options{Modes: h.Modes}
if prev == nil || prev.root == nil {
s.Method = "full"
return h, opts
}
h.Prev = true
opts.Prev = prev.root
s.ByRoot = map[string]string{}
dirty, touched := map[string]bool{}, map[string]bool{}
// every dataset is asked at once what changed; the answers come back in root order
type diffed struct {
changed []change
err error
took time.Duration
}
answers := map[string]chan diffed{}
for _, r := range s.Roots {
if base := prev.Bases[r]; base != "" && datasets[r] != "" {
ch := make(chan diffed, 1)
answers[r] = ch
go func(r, base string) {
t0 := time.Now()
c, err := zfsDiff(ctx, base, r)
ch <- diffed{changed: c, err: err, took: time.Since(t0)}
}(r, base)
}
}
for _, r := range s.Roots {
mode := ModeMTime
how := "mtime"
if ch, ok := answers[r]; ok {
m.mu.Lock()
s.Progress.Current = "asking zfs what changed in " + r
m.mu.Unlock()
d := <-ch
if d.err == nil {
mode, how = ModeDiff, fmt.Sprintf("zfs diff since %s: %d changes in %s", strings.TrimPrefix(prev.Bases[r], datasets[r]), len(d.changed), d.took.Round(time.Millisecond))
for _, c := range d.changed {
dirty[c.parent] = true
if c.dir {
dirty[c.path] = true
}
}
} else {
how = "mtime (zfs diff: " + d.err.Error() + ")"
}
}
if Find(prev.root, r) == nil {
mode, how = ModeFull, "full (new root)"
}
h.Modes[r] = mode
s.ByRoot[r] = how
}
m.mu.Lock()
s.Progress.Current = ""
m.mu.Unlock()
// what the last walk never got to is read whatever the change list says
for _, p := range partialPaths(prev.root, "/") {
dirty[p] = true
}
for p := range dirty {
for a := filepath.Dir(p); ; a = filepath.Dir(a) {
touched[a] = true
if a == "/" {
break
}
}
}
for p := range dirty {
h.Dirty = append(h.Dirty, p)
}
for p := range touched {
h.Touched = append(h.Touched, p)
}
sort.Strings(h.Dirty)
sort.Strings(h.Touched)
opts.Dirty, opts.Touched = dirty, touched
if prev.Partial {
s.Method = "resume"
} else {
s.Method = "refresh"
}
return h, opts
}
func partialPaths(n *Node, path string) []string {
var out []string
var walk func(n *Node, path string)
walk = func(n *Node, path string) {
if n.Partial {
out = append(out, path)
}
for _, c := range n.Children {
if c.Dir {
walk(c, join(path, c.Name))
}
}
}
walk(n, path)
return out
}
func (m *Manager) run(ctx context.Context, s *Scan, prev *Scan, datasets map[string]string) {
finish := func(err error) {
m.mu.Lock()
defer m.mu.Unlock()
s.Running = false
s.Finished = time.Now()
if err != nil {
s.Error = err.Error()
}
m.saveMeta(s)
}
// the newest snapshot of each dataset, so the next scan can ask what changed since
for _, r := range s.Roots {
if ds := datasets[r]; ds != "" {
if snap := newestSnapshot(ctx, ds); snap != "" {
s.Bases[r] = snap
}
}
}
h, _ := m.plan(ctx, s, prev, datasets)
if ctx.Err() != nil { // cancelled while zfs was being asked
finish(errors.New("cancelled"))
return
}
args := append([]string{m.self, "space-walk"}, s.Roots...)
if s.AsRoot {
args = priv.Wrap(args...)
}
cmd := exec.Command(args[0], args[1:]...)
stdin, _ := cmd.StdinPipe() // held open: closing it tells the helper to stop
stdout, _ := cmd.StdoutPipe()
stderr, _ := cmd.StderrPipe()
if err := cmd.Start(); err != nil {
finish(err)
return
}
go func() {
hb, _ := json.Marshal(h)
_, err := stdin.Write(append(hb, '\n'))
if err == nil && h.Prev {
err = Encode(stdin, prev.root)
}
if err != nil {
log.Printf("space: %s: sending the last tree: %v", s.Disk, err)
}
<-ctx.Done()
_ = cmd.Process.Signal(syscall.SIGTERM)
stdin.Close()
}()
var lastErr string
go func() {
sc := bufio.NewScanner(stderr)
sc.Buffer(make([]byte, 64<<10), 1<<20)
for sc.Scan() {
line := sc.Text()
var p Progress
if strings.HasPrefix(line, "{") && json.Unmarshal([]byte(line), &p) == nil {
m.mu.Lock()
s.Progress = p
m.mu.Unlock()
continue
}
if line != "" {
lastErr = line
}
}
}()
// the result streams to disk as it is decoded, so a restart finds it
_ = os.MkdirAll(m.dir, 0o700)
tmp := m.fileFor(s.Disk, ".scan.tmp")
out, ferr := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
var src io.Reader = stdout
if ferr == nil {
src = io.TeeReader(stdout, out)
}
br := bufio.NewReaderSize(src, 1<<20)
var rl resultLine
line, rerr := br.ReadBytes('\n')
if rerr == nil {
rerr = json.Unmarshal(line, &rl)
}
var root *Node
var decErr error
if rerr == nil {
root, decErr = Decode(br)
} else {
decErr = rerr
_, _ = io.Copy(io.Discard, br)
}
if ferr == nil {
out.Close()
}
waitErr := cmd.Wait()
cancelled := ctx.Err() != nil
if root == nil {
_ = os.Remove(tmp)
switch {
case cancelled:
finish(errors.New("cancelled"))
case waitErr != nil && lastErr != "":
finish(errors.New(strings.TrimPrefix(lastErr, "error: ")))
case waitErr != nil:
finish(waitErr)
default:
finish(decErr)
}
return
}
if ferr == nil {
_ = os.Rename(tmp, m.fileFor(s.Disk, ".scan"))
}
m.mu.Lock()
s.root, s.largest = root, nil
s.Files, s.Dirs, s.Size, s.Denied = root.Files, root.Dirs, root.Size, s.Progress.Denied
s.Read, s.Reused, s.PartialDirs = rl.Read, rl.Reused, rl.Partial
s.Partial = !rl.Complete
s.Running = false
s.Finished = time.Now()
if s.Partial {
s.Error = "cancelled"
}
m.saveMeta(s)
m.mu.Unlock()
what := "scanned"
if s.Partial {
what = "partly scanned (cancelled)"
}
log.Printf("space: %s %s: %d files, %d dirs, %d bytes in %s (%s; %d dirs read, %d kept%s)", s.Disk, what, root.Files, root.Dirs, root.Size, s.Finished.Sub(s.Started).Round(time.Millisecond), s.Method, rl.Read, rl.Reused, map[bool]string{true: ", as root", false: ""}[s.AsRoot])
}
func (m *Manager) Cancel(id string) bool {
m.mu.Lock()
s := m.scans[id]
m.mu.Unlock()
if s == nil || !s.Running || s.cancel == nil {
return false
}
s.cancel()
return true
}
// Forget drops the scan of id, in memory and on disk.
func (m *Manager) Forget(id string) {
m.mu.Lock()
if s := m.scans[id]; s != nil && s.cancel != nil {
s.cancel()
}
delete(m.scans, id)
m.mu.Unlock()
_ = os.Remove(m.fileFor(id, ".scan"))
_ = os.Remove(m.fileFor(id, ".json"))
}
// ---- ZFS: what changed since the last scan ----
type change struct {
path, parent string
dir bool
}
// newestSnapshot is the most recent snapshot of a dataset, "" when it has none.
func newestSnapshot(ctx context.Context, dataset string) string {
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-t", "snapshot", "-o", "name", "-s", "creation", "-d", "1", dataset).Output()
if err != nil {
return ""
}
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
last := lines[len(lines)-1]
if !strings.Contains(last, "@") {
return ""
}
return last
}
// zfsDiff lists what changed in a dataset since a snapshot: every changed, added, removed
// or renamed path, with the directory whose listing that touches.
func zfsDiff(ctx context.Context, snapshot, mount string) ([]change, error) {
// A diff slower than this is not worth waiting for: the walk by mtime is bounded. The
// clock runs inside sudo — `timeout` is zfs's parent and kills it as root — because sudo
// on Debian gives the command a session of its own, out of reach of our process group;
// the context is only a backstop.
const patience = 3 * time.Minute
ctx, cancel := context.WithTimeout(ctx, patience+30*time.Second)
defer cancel()
args := priv.Wrap("timeout", "-s", "KILL", strconv.Itoa(int(patience.Seconds())), "zfs", "diff", "-H", "-F", snapshot)
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
cmd.WaitDelay = 5 * time.Second
t0 := time.Now()
var stderr strings.Builder
cmd.Stderr = &stderr
out, err := cmd.StdoutPipe()
if err != nil {
return nil, err
}
if err := cmd.Start(); err != nil {
return nil, err
}
var changes []change
sc := bufio.NewScanner(out)
sc.Buffer(make([]byte, 64<<10), 4<<20)
for sc.Scan() {
f := strings.Split(sc.Text(), "\t")
if len(f) < 3 {
continue
}
isDir := f[1] == "/"
for _, p := range f[2:] { // a rename has the old and the new path
p = unescape(p)
if !strings.HasPrefix(p, mount) {
continue
}
changes = append(changes, change{path: p, parent: filepath.Dir(p), dir: isDir})
}
}
if err := cmd.Wait(); err != nil {
msg := strings.TrimSpace(stderr.String())
if errors.Is(ctx.Err(), context.DeadlineExceeded) || time.Since(t0) >= patience {
msg = "gave up after " + patience.String()
}
if msg == "" {
msg = err.Error()
}
if i := strings.IndexByte(msg, '\n'); i > 0 {
msg = msg[:i]
}
return nil, errors.New(msg)
}
return changes, nil
}
// unescape undoes zfs diff's \ooo escapes for spaces and odd bytes in names.
func unescape(s string) string {
if !strings.Contains(s, "\\") {
return s
}
var b strings.Builder
for i := 0; i < len(s); i++ {
if s[i] == '\\' && i+3 < len(s) {
if v, err := strconv.ParseUint(s[i+1:i+4], 8, 8); err == nil {
b.WriteByte(byte(v))
i += 3
continue
}
}
b.WriteByte(s[i])
}
return b.String()
}
// Tree is the subtree at path, cut to depth levels of at most limit children each, leaving
// out children smaller than min bytes; what is left out is summed into MoreN/MoreSize. A
// treemap asks deep with a min the size of the smallest tile it could draw; a folder list
// asks one level with no min.
type Tree struct {
Node
Path string `json:"path"`
Children []*Tree `json:"children,omitempty"`
MoreN int `json:"more_n,omitempty"`
MoreSize int64 `json:"more_size,omitempty"`
Total int64 `json:"total"` // the whole scan's size, for percentages
}
func (m *Manager) Tree(id, path string, depth, limit int, min int64) (*Tree, error) {
m.mu.Lock()
s := m.scans[id]
var root *Node
if s != nil {
root = s.root
}
m.mu.Unlock()
if root == nil {
if s != nil && s.Loading {
return nil, errors.New("the last scan is still being read back")
}
return nil, errors.New("no scan of this disk yet")
}
n := Find(root, path)
if n == nil {
return nil, errors.New("no such path in the scan")
}
if path == "" {
path = "/"
}
t := cut(n, path, depth, limit, min)
t.Total = root.Size
return t, nil
}
func cut(n *Node, path string, depth, limit int, min int64) *Tree {
t := &Tree{Node: *n, Path: path}
t.Node.Children = nil
if !n.Dir {
return t
}
if depth <= 0 { // as deep as was asked: what is inside is one lump, not nothing
if len(n.Children) > 0 {
t.MoreN, t.MoreSize = len(n.Children), n.Size-n.SmallSize
}
return t
}
for i, c := range n.Children {
if i >= limit || (c.Size < min && i > 0) { // children are sorted largest first: the first is always kept
t.MoreN++
t.MoreSize += c.Size
continue
}
t.Children = append(t.Children, cut(c, join(path, c.Name), depth-1, limit, min))
}
return t
}
// Largest is the n biggest files of the last scan of id.
func (m *Manager) Largest(id string, n int) ([]File, error) {
m.mu.Lock()
s := m.scans[id]
var root *Node
var have []File
if s != nil {
root, have = s.root, s.largest
}
m.mu.Unlock()
if root == nil {
return nil, errors.New("no scan of this disk yet")
}
if len(have) < n {
have = Largest(root, n) // the tree is never changed once made, so no lock while walking it
m.mu.Lock()
if s.root == root {
s.largest = have
}
m.mu.Unlock()
}
if len(have) > n {
return have[:n], nil
}
return have, nil
}
internal/space/walk.go
+516
View File
@@ -0,0 +1,516 @@
// Package space measures what is using a disk — it is what the UI calls the **Usage** tab
// (renamed from "Space" on 2026-09-24; the package and its /api/space endpoints kept the
// old name rather than churn a working API): a parallel walk of the filesystems a disk
// backs, kept as a tree of directories and the files worth seeing, the way WizTree does it.
//
// The walk never crosses a filesystem boundary by itself — /proc, /sys, another disk's
// mount — but every mountpoint the disk backs is a root of the same scan, and the roots
// are grafted into one tree at their paths, so a ZFS pool of many datasets, or a disk with
// / and /home on separate partitions, reads as one picture.
//
// Files below KeepMin are not kept one by one: a directory of a million small files is
// one "smaller files" entry with their count and total, which is all a treemap can show
// of them anyway, and what keeps a scan of a few million files within a few hundred MB.
//
// A walk can start from the last one. Given the previous tree, a directory is read again
// only when something says it changed: on ZFS, `zfs diff` against a snapshot older than
// the last scan names every path that changed, and directories it does not name are kept
// as they were without so much as a stat; elsewhere a directory whose mtime is what it was
// keeps its list and only its kept files are stat'ed again, since a file can grow without
// its directory noticing. A directory the last walk did not finish (it was cancelled) is
// read again whatever else is true, which is what makes a cancelled scan resumable.
package space
import (
"context"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
)
// KeepMin is the smallest file kept as its own node.
const KeepMin = 128 << 10
// Node is a directory or a file. Sizes are bytes allocated on disk (what fills the disk);
// Apparent is the length. For a directory both include everything beneath it.
type Node struct {
Name string `json:"name"`
Dir bool `json:"dir"`
Size int64 `json:"size"`
Apparent int64 `json:"apparent"`
MTime int64 `json:"mtime"` // unix seconds
MTimeNs int64 `json:"-"` // directories: the same to the nanosecond, so a change in the same second is not missed
Files uint32 `json:"files"` // beneath a directory (kept and small alike)
Dirs uint32 `json:"dirs"`
SmallN uint32 `json:"small_n,omitempty"` // files here below KeepMin, not listed
SmallSize int64 `json:"small_size,omitempty"` // their allocated bytes
Mount bool `json:"mount,omitempty"` // another filesystem starts here (not this disk's, or a root of its own)
Denied bool `json:"denied,omitempty"` // could not be read
Partial bool `json:"partial,omitempty"` // the walk was cancelled before this directory was read
Children []*Node `json:"children,omitempty"` // directories: sorted by Size, largest first
}
// Progress is how far a running walk has got.
type Progress struct {
Files int64 `json:"files"`
Dirs int64 `json:"dirs"` // directories read
Reused int64 `json:"reused"` // directories kept from the last scan without reading
Bytes int64 `json:"bytes"`
Denied int64 `json:"denied"`
Current string `json:"current"`
}
// Mode is how a root is walked when there is a previous tree.
type Mode int
const (
ModeFull Mode = iota // read everything
ModeMTime // read a directory again only if its mtime changed; stat its kept files
ModeDiff // read only the directories a change list names
)
// Options is what a walk starts from.
type Options struct {
Prev *Node // the last scan's tree, or nil
// Modes says how each root is walked (by root path); a root not listed is ModeFull.
Modes map[string]Mode
// Dirty is every directory a change list named (ModeDiff roots): those are read again.
// Touched is every ancestor of a dirty directory: those are kept but looked into.
Dirty, Touched map[string]bool
}
type walker struct {
ctx context.Context
opts Options
mu sync.Mutex
queue []job
cond *sync.Cond
pending int // dirs queued or being read
files atomic.Int64
dirs atomic.Int64
reused atomic.Int64
bytes atomic.Int64
denied atomic.Int64
partial atomic.Int64
curMu sync.Mutex
current string
stopping bool
}
type job struct {
path string
node *Node
prev *Node // the same directory in the last scan, if it had one
dev uint64 // the filesystem this root is on; a child on another is a mountpoint, not descended
mode Mode
}
// Result is what a walk returns besides the tree.
type Result struct {
Root *Node
Partial int64 // directories not read because the walk was cancelled
Reused int64
Read int64
Complete bool
}
// Walk measures roots (mountpoints, absolute) and returns them grafted into one tree whose
// root is "/". progress, if set, is called about four times a second. A cancelled walk
// still returns the tree, with the directories it did not get to marked Partial.
func Walk(ctx context.Context, roots []string, opts Options, progress func(Progress)) (Result, error) {
w := &walker{ctx: ctx, opts: opts}
w.cond = sync.NewCond(&w.mu)
root := &Node{Name: "/", Dir: true}
sort.Strings(roots)
trees := map[string]*Node{}
for _, r := range roots {
r = filepath.Clean(r)
if !filepath.IsAbs(r) {
continue
}
var st syscall.Stat_t
if err := syscall.Lstat(r, &st); err != nil || st.Mode&syscall.S_IFMT != syscall.S_IFDIR {
continue
}
n := &Node{Name: filepath.Base(r), Dir: true, MTime: int64(st.Mtim.Sec), MTimeNs: st.Mtim.Nano()}
if r == "/" {
n = root
}
trees[r] = n
var prev *Node
if opts.Prev != nil {
prev = Find(opts.Prev, r)
}
mode := opts.Modes[r]
if prev == nil {
mode = ModeFull
}
w.push(job{path: r, node: n, prev: prev, dev: uint64(st.Dev), mode: mode})
}
if len(trees) == 0 {
return Result{Root: root}, os.ErrNotExist
}
workers := 32
var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() { defer wg.Done(); w.run() }()
}
done := make(chan struct{})
go func() { wg.Wait(); close(done) }()
tick := time.NewTicker(250 * time.Millisecond)
defer tick.Stop()
cancelled := false
for waiting := true; waiting; {
select {
case <-done:
waiting = false
case <-tick.C:
if progress != nil {
progress(w.snapshot())
}
case <-ctx.Done():
cancelled = true
w.mu.Lock()
w.stopping = true
for _, j := range w.queue { // never read: say so, so the next walk reads them
j.node.Partial = true
w.partial.Add(1)
}
w.queue = nil
w.cond.Broadcast()
w.mu.Unlock()
<-done
waiting = false
}
}
if progress != nil {
progress(w.snapshot())
}
// graft each root at its path; deepest last so parents exist first
for _, r := range roots {
n, ok := trees[filepath.Clean(r)]
if !ok || n == root {
continue
}
graft(root, filepath.Clean(r), n)
}
total(root)
res := Result{Root: root, Partial: w.partial.Load(), Reused: w.reused.Load(), Read: w.dirs.Load(), Complete: !cancelled}
if cancelled {
return res, ctx.Err()
}
return res, nil
}
func (w *walker) snapshot() Progress {
w.curMu.Lock()
cur := w.current
w.curMu.Unlock()
return Progress{Files: w.files.Load(), Dirs: w.dirs.Load(), Reused: w.reused.Load(), Bytes: w.bytes.Load(), Denied: w.denied.Load(), Current: cur}
}
func (w *walker) push(j job) {
w.mu.Lock()
if w.stopping {
j.node.Partial = true
w.partial.Add(1)
w.mu.Unlock()
return
}
w.queue = append(w.queue, j)
w.pending++
w.cond.Signal()
w.mu.Unlock()
}
// run takes directories off the queue until there are none left and none being read.
func (w *walker) run() {
for {
w.mu.Lock()
for len(w.queue) == 0 && w.pending > 0 && !w.stopping {
w.cond.Wait()
}
if w.stopping || (len(w.queue) == 0 && w.pending == 0) {
w.cond.Broadcast()
w.mu.Unlock()
return
}
j := w.queue[len(w.queue)-1] // depth first: keeps the queue short and the disk head local
w.queue = w.queue[:len(w.queue)-1]
w.mu.Unlock()
w.visit(j)
w.mu.Lock()
w.pending--
if w.pending == 0 {
w.cond.Broadcast()
}
w.mu.Unlock()
}
}
func join(dir, name string) string {
if dir == "/" {
return "/" + name
}
return dir + "/" + name
}
// visit decides whether a directory must be read or can be taken from the last scan.
func (w *walker) visit(j job) {
if j.prev == nil || j.prev.Partial || j.prev.Denied || j.mode == ModeFull {
w.readDir(j)
return
}
switch j.mode {
case ModeDiff:
if w.opts.Dirty[j.path] {
w.readDir(j)
return
}
if !w.opts.Touched[j.path] {
w.reuse(j, false) // nothing under here changed: keep the whole subtree, no stat at all
return
}
w.reuse(j, true) // something deeper changed: keep this list, look into the children
case ModeMTime:
var st syscall.Stat_t
if err := syscall.Lstat(j.path, &st); err != nil {
w.readDir(j)
return
}
if st.Mtim.Nano() != j.prev.MTimeNs {
w.readDir(j) // an entry was added, removed or renamed
return
}
w.reuse(j, true) // same entries; files may have grown, and subdirectories are their own question
}
}
// reuse keeps a directory as the last scan saw it. With descend, its subdirectories are
// visited (each decides for itself) and, in mtime mode, its kept files are stat'ed again.
func (w *walker) reuse(j job, descend bool) {
w.reused.Add(1)
n, p := j.node, j.prev
n.MTime, n.MTimeNs, n.SmallN, n.SmallSize, n.Denied, n.Mount = p.MTime, p.MTimeNs, p.SmallN, p.SmallSize, p.Denied, p.Mount
if !descend {
n.Children = p.Children
n.Apparent = p.Apparent - childApparent(p) // total() adds the children back
w.files.Add(int64(p.Files))
w.bytes.Add(p.Size)
w.reused.Add(int64(p.Dirs))
return
}
w.files.Add(int64(p.SmallN))
w.bytes.Add(p.SmallSize)
n.Apparent = p.Apparent - childApparent(p) // the small files' lengths; total() adds the children back
var st syscall.Stat_t
for _, c := range p.Children {
if c.Dir {
d := &Node{Name: c.Name, Dir: true, MTime: c.MTime, MTimeNs: c.MTimeNs, Mount: c.Mount}
n.Children = append(n.Children, d)
if c.Mount {
continue
}
w.push(job{path: join(j.path, c.Name), node: d, prev: c, dev: j.dev, mode: j.mode})
continue
}
if j.mode == ModeMTime {
if err := syscall.Lstat(join(j.path, c.Name), &st); err != nil {
continue // gone since (the directory's mtime would have said so, but be safe)
}
c = &Node{Name: c.Name, Size: int64(st.Blocks) * 512, Apparent: int64(st.Size), MTime: int64(st.Mtim.Sec)}
}
n.Children = append(n.Children, c)
w.files.Add(1)
w.bytes.Add(c.Size)
}
}
func childApparent(p *Node) int64 {
var a int64
for _, c := range p.Children {
a += c.Apparent
}
return a
}
func (w *walker) readDir(j job) {
w.curMu.Lock()
w.current = j.path
w.curMu.Unlock()
w.dirs.Add(1)
f, err := os.Open(j.path)
if err != nil {
j.node.Denied = true
w.denied.Add(1)
return
}
names, err := f.Readdirnames(-1)
f.Close()
if err != nil {
j.node.Denied = true
w.denied.Add(1)
}
var prevDirs map[string]*Node
if j.prev != nil {
prevDirs = map[string]*Node{}
for _, c := range j.prev.Children {
if c.Dir {
prevDirs[c.Name] = c
}
}
}
var st syscall.Stat_t
for _, name := range names {
p := join(j.path, name)
if err := syscall.Lstat(p, &st); err != nil {
continue
}
switch st.Mode & syscall.S_IFMT {
case syscall.S_IFDIR:
c := &Node{Name: name, Dir: true, MTime: int64(st.Mtim.Sec), MTimeNs: st.Mtim.Nano()}
j.node.Children = append(j.node.Children, c)
if uint64(st.Dev) != j.dev {
c.Mount = true // something else is mounted here; if it is this disk's too, it is a root of its own
continue
}
w.push(job{path: p, node: c, prev: prevDirs[name], dev: j.dev, mode: j.mode})
case syscall.S_IFREG, syscall.S_IFLNK:
alloc := int64(st.Blocks) * 512
w.files.Add(1)
w.bytes.Add(alloc)
if alloc >= KeepMin || int64(st.Size) >= KeepMin {
j.node.Children = append(j.node.Children, &Node{Name: name, Size: alloc, Apparent: int64(st.Size), MTime: int64(st.Mtim.Sec)})
} else {
j.node.SmallN++
j.node.SmallSize += alloc
j.node.Apparent += int64(st.Size) // added to the directory's own apparent total below
}
}
}
}
// graft puts a root's tree at path under root, making any directories in between.
func graft(root *Node, path string, tree *Node) {
parts := strings.Split(strings.Trim(path, "/"), "/")
cur := root
for i, part := range parts {
var next *Node
for _, c := range cur.Children {
if c.Name == part && c.Dir {
next = c
break
}
}
if i == len(parts)-1 {
if next == nil {
cur.Children = append(cur.Children, tree)
} else {
// the parent's walk saw this directory as a mountpoint; the walk of the mount fills it
next.Children, next.SmallN, next.SmallSize, next.Apparent, next.Denied, next.Partial = tree.Children, tree.SmallN, tree.SmallSize, tree.Apparent, tree.Denied, tree.Partial
next.Mount = true
}
return
}
if next == nil {
next = &Node{Name: part, Dir: true}
cur.Children = append(cur.Children, next)
}
cur = next
}
}
// total sums every directory from its children and sorts them largest first.
func total(n *Node) {
if !n.Dir {
return
}
n.Size = n.SmallSize
n.Files = n.SmallN
n.Dirs = 0
apparent := n.Apparent // the small files' lengths
for _, c := range n.Children {
total(c)
n.Size += c.Size
apparent += c.Apparent
if c.Dir {
n.Dirs += 1 + c.Dirs
n.Files += c.Files
} else {
n.Files++
}
}
n.Apparent = apparent
sort.Slice(n.Children, func(i, j int) bool { return n.Children[i].Size > n.Children[j].Size })
}
// Find follows path (absolute, "/" for the root) down the tree.
func Find(root *Node, path string) *Node {
cur := root
for _, part := range strings.Split(strings.Trim(filepath.Clean(path), "/"), "/") {
if part == "" {
continue
}
var next *Node
for _, c := range cur.Children {
if c.Name == part {
next = c
break
}
}
if next == nil {
return nil
}
cur = next
}
return cur
}
// File is one of the largest files, with its full path.
type File struct {
Path string `json:"path"`
Size int64 `json:"size"`
Apparent int64 `json:"apparent"`
MTime int64 `json:"mtime"`
}
// Largest is the n biggest files in the tree, largest first.
func Largest(root *Node, n int) []File {
var out []File
minIdx := -1
var walk func(n *Node, path string)
walk = func(node *Node, path string) {
for _, c := range node.Children {
p := join(path, c.Name)
if c.Dir {
if c.Size > 0 && (len(out) < n || c.Size > out[minIdx].Size) {
walk(c, p)
}
continue
}
if len(out) < n {
out = append(out, File{Path: p, Size: c.Size, Apparent: c.Apparent, MTime: c.MTime})
if minIdx < 0 || c.Size < out[minIdx].Size {
minIdx = len(out) - 1
}
} else if c.Size > out[minIdx].Size {
out[minIdx] = File{Path: p, Size: c.Size, Apparent: c.Apparent, MTime: c.MTime}
minIdx = 0
for i := range out {
if out[i].Size < out[minIdx].Size {
minIdx = i
}
}
}
}
}
walk(root, "/")
sort.Slice(out, func(i, j int) bool { return out[i].Size > out[j].Size })
return out
}
internal/space/walk_test.go
+87
View File
@@ -0,0 +1,87 @@
package space
import (
"context"
"os"
"path/filepath"
"testing"
"time"
)
func mk(t *testing.T, dir string, files map[string]int) {
for p, n := range files {
full := filepath.Join(dir, p)
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(full, make([]byte, n), 0o644); err != nil {
t.Fatal(err)
}
}
}
func TestRefreshAndResume(t *testing.T) {
dir := t.TempDir()
mk(t, dir, map[string]int{"a/big1": 300 << 10, "a/small": 100, "b/c/big2": 500 << 10, "b/d/big3": 200 << 10})
res, err := Walk(context.Background(), []string{dir}, Options{}, nil)
if err != nil {
t.Fatal(err)
}
root := Find(res.Root, dir)
if root == nil || root.Files != 4 {
t.Fatalf("full walk: %+v", root)
}
want := root.Size
// grow a file in place (its directory's mtime does not change) and add a file elsewhere
time.Sleep(20 * time.Millisecond)
if err := os.WriteFile(filepath.Join(dir, "a/big1"), make([]byte, 600<<10), 0o644); err != nil {
t.Fatal(err)
}
mk(t, dir, map[string]int{"b/d/big4": 400 << 10})
res2, err := Walk(context.Background(), []string{dir}, Options{Prev: res.Root, Modes: map[string]Mode{dir: ModeMTime}}, nil)
if err != nil {
t.Fatal(err)
}
r2 := Find(res2.Root, dir)
if r2.Files != 5 || r2.Size != want+(300<<10)+(400<<10) {
t.Fatalf("mtime refresh: files %d size %d want %d (reused %d read %d)", r2.Files, r2.Size, want+(300<<10)+(400<<10), res2.Reused, res2.Read)
}
if res2.Reused == 0 || res2.Read == 0 {
t.Fatalf("mtime refresh should reuse some and read some: reused %d read %d", res2.Reused, res2.Read)
}
// diff mode: only b/d named dirty; a's growth must NOT be seen (that is the contract), b/d's new file must
dirty := map[string]bool{filepath.Join(dir, "b/d"): true}
touched := map[string]bool{dir: true, filepath.Join(dir, "b"): true}
res3, err := Walk(context.Background(), []string{dir}, Options{Prev: res.Root, Modes: map[string]Mode{dir: ModeDiff}, Dirty: dirty, Touched: touched}, nil)
if err != nil {
t.Fatal(err)
}
r3 := Find(res3.Root, dir)
if r3.Files != 5 || r3.Size != want+(400<<10) || res3.Read != 1 {
t.Fatalf("diff refresh: files %d size %d want %d read %d", r3.Files, r3.Size, want+(400<<10), res3.Read)
}
// a cancelled walk marks what it did not read; resuming reads those
ctx, cancel := context.WithCancel(context.Background())
cancel()
res4, err := Walk(ctx, []string{dir}, Options{}, nil)
if err == nil || res4.Complete {
t.Fatal("cancelled walk should say so")
}
if len(partialPaths(res4.Root, "/")) == 0 {
t.Fatal("cancelled walk should leave partial directories")
}
res5, err := Walk(context.Background(), []string{dir}, Options{Prev: res4.Root, Modes: map[string]Mode{dir: ModeMTime}}, nil)
if err != nil {
t.Fatal(err)
}
if r5 := Find(res5.Root, dir); r5.Files != 5 || len(partialPaths(res5.Root, "/")) != 0 {
t.Fatalf("resume: files %d partial %v", r5.Files, partialPaths(res5.Root, "/"))
}
// the codec round-trips the partial flag
pr, pw, _ := os.Pipe()
go func() { _ = Encode(pw, res4.Root); pw.Close() }()
back, err := Decode(pr)
if err != nil || len(partialPaths(back, "/")) != len(partialPaths(res4.Root, "/")) {
t.Fatalf("codec: %v", err)
}
}
internal/storage/snapshots.go
+493
View File
@@ -0,0 +1,493 @@
package storage
// Scheduled snapshots with a retention period: take a snapshot of a dataset on a schedule,
// and drop the ones that have aged out.
//
// **The naming is zfSnap's, deliberately.** grid has run zfSnap v1.11.1 from root cron since
// before this repo existed and it works well, so rather than invent a second convention and
// have two tools disagree about what a snapshot is, this writes and reads exactly what
// zfSnap does: `<dataset>@2026-09-24_02.30.01--6w`, with the retention encoded in the name.
// That has three consequences worth knowing. Ham Lab can show and manage the snapshots
// zfSnap already took. The two can run side by side without fighting, because both only
// ever destroy snapshots that have passed the expiry written into their own name. And
// changing a policy's retention does not retroactively re-date old snapshots — each one
// carries the rule it was taken under, which is the behaviour you want from a backup.
//
// Two rules make this safe to run unattended. Only a snapshot whose name parses as a stamp
// and a TTL is ever a candidate — anything taken by hand or named anything else is invisible
// to it. And every policy can be asked what it would delete, before it deletes anything.
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
"hamlab/internal/cron"
"hamlab/internal/priv"
)
// stampLayout is zfSnap's timestamp: 2026-09-24_02.30.01.
const stampLayout = "2006-01-02_15.04.05"
// ttlSep separates the stamp from the retention in a snapshot name, as zfSnap writes it.
const ttlSep = "--"
// Policy is a standing instruction: snapshot this dataset on this schedule, keep the
// snapshots this long.
type Policy struct {
ID string `json:"id"`
Name string `json:"name"`
Dataset string `json:"dataset"`
// Recursive takes one atomic snapshot of the dataset and everything under it (zfs -r).
Recursive bool `json:"recursive"`
// Every is hourly | daily | weekly | monthly | custom. At is "HH:MM" (the minute alone
// is used for hourly). Weekday is 0=Sunday, Day is 1–28. With "custom", Cron is a plain
// cron expression and the rest is ignored — the escape hatch for anything the four
// common shapes cannot say.
Every string `json:"every"`
At string `json:"at"`
Weekday int `json:"weekday"`
Day int `json:"day"`
Cron string `json:"cron,omitempty"`
// KeepFor is how long a snapshot lives: "6w", "30d", "12h". It is written into the
// snapshot's own name, zfSnap-style, so each one carries the rule it was taken under.
// KeepMin is a floor — that many newest snapshots survive however old they are, so a
// dataset is never left with none.
KeepFor string `json:"keep_for"`
KeepMin int `json:"keep_min"`
Prefix string `json:"prefix"`
Enabled bool `json:"enabled"`
}
func (p *Policy) prefix() string { return strings.TrimSpace(p.Prefix) }
// Validate reports what is wrong with a policy, or nil.
func (p *Policy) Validate() error {
if !datasetRe.MatchString(p.Dataset) {
return errors.New("dataset: a pool or pool/dataset name")
}
if pre := p.prefix(); pre != "" && (!snapRe.MatchString(pre) || strings.Contains(pre, ttlSep)) {
return errors.New("prefix: letters, digits, . _ : and - only, and no --")
}
switch p.Every {
case "hourly", "daily", "weekly", "monthly":
case "custom":
if strings.TrimSpace(p.Cron) == "" {
return errors.New("a custom schedule needs a cron expression")
}
default:
return errors.New("every: hourly, daily, weekly, monthly or custom")
}
if _, err := p.CronExpr(); err != nil {
return err
}
if _, _, err := p.clock(); err != nil {
return err
}
if p.Every == "weekly" && (p.Weekday < 0 || p.Weekday > 6) {
return errors.New("weekday: 0 (Sunday) to 6")
}
if p.Every == "monthly" && (p.Day < 1 || p.Day > 28) {
return errors.New("day: 1 to 28, so every month has one")
}
if p.KeepFor != "" {
if _, err := ParseAge(p.KeepFor); err != nil {
return err
}
}
if p.KeepMin < 0 {
return errors.New("keep at least: 0 or more")
}
if p.KeepFor == "" {
return errors.New("a retention period is required — it is written into each snapshot's name, and is what lets it be cleaned up later")
}
return nil
}
func (p *Policy) clock() (hour, min int, err error) {
at := strings.TrimSpace(p.At)
if at == "" {
return 0, 0, nil
}
h, m, ok := strings.Cut(at, ":")
if !ok {
return 0, 0, errors.New("at: HH:MM")
}
hour, err = strconv.Atoi(h)
if err == nil {
min, err = strconv.Atoi(m)
}
if err != nil || hour < 0 || hour > 23 || min < 0 || min > 59 {
return 0, 0, errors.New("at: HH:MM, 00:00 to 23:59")
}
return hour, min, nil
}
// ParseAge reads "6w", "30d", "12h", "90m" as a duration. Weeks and days are what a
// retention period is actually written in, and time.ParseDuration knows neither.
func ParseAge(s string) (time.Duration, error) {
s = strings.TrimSpace(strings.ToLower(s))
if s == "" {
return 0, errors.New("empty period")
}
unit := s[len(s)-1]
n, err := strconv.Atoi(s[:len(s)-1])
if err != nil || n <= 0 {
return 0, fmt.Errorf("%q: a number and a unit, like 6w, 30d or 12h", s)
}
switch unit {
case 'm':
return time.Duration(n) * time.Minute, nil
case 'h':
return time.Duration(n) * time.Hour, nil
case 'd':
return time.Duration(n) * 24 * time.Hour, nil
case 'w':
return time.Duration(n) * 7 * 24 * time.Hour, nil
}
return 0, fmt.Errorf("%q: the unit is m, h, d or w", s)
}
// CronExpr renders the schedule as a cron expression — which is what actually runs it. The
// four common shapes are sugar over this; a custom policy states it outright. One expression
// is the only description of when a schedule fires, so the page, the cron file and the next
// run can never disagree.
func (p *Policy) CronExpr() (string, error) {
if p.Every == "custom" {
expr := strings.Join(strings.Fields(p.Cron), " ")
if err := cron.ValidSchedule(expr); err != nil {
return "", fmt.Errorf("cron expression: %w", err)
}
if strings.HasPrefix(expr, "@reboot") {
return "", errors.New("@reboot is not a schedule for snapshots")
}
return expr, nil
}
hour, min, err := p.clock()
if err != nil {
return "", err
}
switch p.Every {
case "hourly":
return fmt.Sprintf("%d * * * *", min), nil
case "daily":
return fmt.Sprintf("%d %d * * *", min, hour), nil
case "weekly":
return fmt.Sprintf("%d %d * * %d", min, hour, p.Weekday), nil
case "monthly":
return fmt.Sprintf("%d %d %d * *", min, hour, p.Day), nil
}
return "", errors.New("unknown schedule")
}
// NextRun is when cron will next fire this schedule, read with the same evaluator the Jobs
// tab uses on everyone else's lines.
func (p *Policy) NextRun(now time.Time) time.Time {
expr, err := p.CronExpr()
if err != nil {
return time.Time{}
}
t, ok := cron.NextRun(expr, now)
if !ok {
return time.Time{}
}
return t
}
// SnapTime reads a snapshot's name in zfSnap's convention and returns when it was taken,
// when it expires, and whether the name parsed at all. A name without a TTL never expires.
// An optional prefix in front of the stamp is allowed, so a policy can label its own.
func SnapTime(name string) (taken time.Time, expires time.Time, ok bool) {
_, snap, cut := strings.Cut(name, "@")
if !cut {
return
}
body, ttl, hasTTL := strings.Cut(snap, ttlSep)
// an optional "<prefix>-" before the stamp
stamp := body
if len(body) > len(stampLayout) {
stamp = body[len(body)-len(stampLayout):]
}
t, err := time.ParseInLocation(stampLayout, stamp, time.Local)
if err != nil {
return time.Time{}, time.Time{}, false
}
taken, ok = t, true
if hasTTL {
if d, err := ParseAge(ttl); err == nil {
expires = t.Add(d)
}
}
return
}
// Doomed is which of this dataset's snapshots have passed the expiry written into their own
// names, oldest first — exactly what zfSnap -d would remove. This is what the UI shows
// before anything is deleted, and what Run then acts on. KeepMin holds back that many of the
// newest whatever their names say, so a dataset is never left with none.
func (m *Manager) Doomed(ctx context.Context, p *Policy) ([]Snapshot, error) {
if err := p.Validate(); err != nil {
return nil, err
}
all, err := m.Snapshots(ctx, p.Dataset)
if err != nil {
return nil, err
}
var dated []Snapshot
expiry := map[string]time.Time{}
for _, s := range all {
taken, exp, ok := SnapTime(s.Name)
if !ok || exp.IsZero() {
continue // hand-made, or kept forever: not ours to remove
}
_ = taken
dated = append(dated, s)
expiry[s.Name] = exp
}
sort.Slice(dated, func(i, j int) bool { return dated[i].Created > dated[j].Created }) // newest first
now := time.Now()
var doomed []Snapshot
for i, s := range dated {
if i < p.KeepMin {
continue // the floor: these survive however old they are
}
if expiry[s.Name].Before(now) {
doomed = append(doomed, s)
}
}
sort.Slice(doomed, func(i, j int) bool { return doomed[i].Created < doomed[j].Created }) // oldest first
return doomed, nil
}
// DestroySnapshot removes one snapshot. recursive also removes the same-named snapshot on
// every dataset beneath it, which is what a recursive policy created.
func (m *Manager) DestroySnapshot(ctx context.Context, full string, recursive bool) error {
dataset, snap, ok := strings.Cut(full, "@")
if !ok || !datasetRe.MatchString(dataset) || !snapRe.MatchString(snap) {
return errors.New("bad snapshot name")
}
args := []string{"zfs", "destroy"}
if recursive {
args = append(args, "-r")
}
out, code, err := m.run(ctx, append(args, full)...)
if err != nil {
return err
}
if code != 0 {
return errors.New(strings.TrimSpace(string(out)))
}
return nil
}
// SnapshotRecursive takes dataset@name, optionally of everything beneath it too.
func (m *Manager) SnapshotRecursive(ctx context.Context, dataset, name string, recursive bool) error {
if !datasetRe.MatchString(dataset) || !snapRe.MatchString(name) {
return errors.New("bad snapshot name")
}
args := []string{"zfs", "snapshot"}
if recursive {
args = append(args, "-r")
}
out, code, err := m.run(ctx, append(args, dataset+"@"+name)...)
if err != nil {
return err
}
if code != 0 {
return errors.New(strings.TrimSpace(string(out)))
}
return nil
}
// RunResult is what one pass of a policy did.
type RunResult struct {
Taken string `json:"taken"`
Pruned []string `json:"pruned"`
Skipped int `json:"skipped"` // snapshots that are not this policy's, left alone
}
// Run takes the snapshot and prunes what has aged out. It is the same code the scheduler
// calls, so "Run now" and the timer cannot drift apart.
func (m *Manager) Run(ctx context.Context, p *Policy) (RunResult, error) {
var res RunResult
if err := p.Validate(); err != nil {
return res, err
}
name := time.Now().Format(stampLayout)
if pre := strings.TrimSpace(p.Prefix); pre != "" {
name = pre + "-" + name
}
if p.KeepFor != "" {
name += ttlSep + strings.TrimSpace(strings.ToLower(p.KeepFor)) // zfSnap's convention: the retention lives in the name
}
if err := m.SnapshotRecursive(ctx, p.Dataset, name, p.Recursive); err != nil {
return res, err
}
res.Taken = p.Dataset + "@" + name
doomed, err := m.Doomed(ctx, p)
if err != nil {
return res, err // the snapshot was taken; the prune is what failed
}
for _, s := range doomed {
if err := m.DestroySnapshot(ctx, s.Name, p.Recursive); err != nil {
return res, fmt.Errorf("destroying %s: %w", s.Name, err)
}
res.Pruned = append(res.Pruned, s.Name)
}
return res, nil
}
// ---- what happened when it ran ----
//
// The policies are a definition the daemon owns; what a run did is written by whatever ran
// it, which is a cron job in its own process. Keeping the two apart means the CLI never has
// to write state.json behind the daemon's back — the last writer would win and one of them
// would lose its work.
// RunRecord is the outcome of one run of one schedule.
type RunRecord struct {
At time.Time `json:"at"`
Taken string `json:"taken,omitempty"`
Pruned []string `json:"pruned,omitempty"`
Error string `json:"error,omitempty"`
}
func runPath(dir, id string) string { return filepath.Join(dir, "zfs-runs", id+".json") }
// WriteRun records what a run did, beside the config.
func WriteRun(dir, id string, r RunRecord) error {
if err := os.MkdirAll(filepath.Join(dir, "zfs-runs"), 0o700); err != nil {
return err
}
b, err := json.Marshal(r)
if err != nil {
return err
}
p := runPath(dir, id)
tmp := p + ".tmp"
if err := os.WriteFile(tmp, b, 0o600); err != nil {
return err
}
return os.Rename(tmp, p)
}
// ReadRun is the last thing a schedule did, or the zero value if it has never run.
func ReadRun(dir, id string) RunRecord {
var r RunRecord
b, err := os.ReadFile(runPath(dir, id))
if err == nil {
_ = json.Unmarshal(b, &r)
}
return r
}
// ForgetRun drops a schedule's record, for when the schedule itself goes.
func ForgetRun(dir, id string) { _ = os.Remove(runPath(dir, id)) }
// ---- the cron file ----
// CronFile is where Ham Lab writes the snapshot schedules. One file it owns entirely and
// rewrites whole, so there is never a half-updated set of lines, and so a schedule removed
// in the UI is removed from the machine.
const CronFile = "/etc/cron.d/hamlab-zfs"
// CronFileText renders every schedule as cron lines. user is who they run as — the account
// Ham Lab itself runs as, which reaches zfs through sudo exactly as the daemon does, so the
// files it writes stay owned by that account rather than by root.
func CronFileText(policies []Policy, user, exe, config string) (string, error) {
var b strings.Builder
b.WriteString("# Ham Lab snapshot schedules — written by the ZFS tab, rewritten whole on every change.\n")
b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
b.WriteString("SHELL=/bin/sh\n")
b.WriteString("PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
for i := range policies {
p := &policies[i]
if !p.Enabled {
b.WriteString("# " + p.Name + " — paused\n")
continue
}
expr, err := p.CronExpr()
if err != nil {
return "", fmt.Errorf("%s: %w", p.Name, err)
}
name := strings.ReplaceAll(p.Name, "\n", " ")
fmt.Fprintf(&b, "# %s — %s, keep %s\n", name, p.Dataset, p.KeepFor)
fmt.Fprintf(&b, "%s %s %s -config %s zfs-snapshot %s\n\n", expr, user, exe, config, p.ID)
}
return b.String(), nil
}
// WriteCronFile puts the schedules on the machine, or removes the file when there are none.
func (m *Manager) WriteCronFile(ctx context.Context, policies []Policy, user, exe, config string) error {
live := false
for i := range policies {
if policies[i].Enabled {
live = true
}
}
if !live {
out, code, err := m.run(ctx, "rm", "-f", CronFile)
if err != nil {
return err
}
if code != 0 {
return errors.New(strings.TrimSpace(string(out)))
}
return nil
}
text, err := CronFileText(policies, user, exe, config)
if err != nil {
return err
}
return m.writeRoot(ctx, CronFile, text)
}
// WriteRootFile puts text at a root-owned path, through the same privilege escalation every
// other write here uses. Shared by the snapshot schedules and the image-update schedules,
// which both own a file in /etc/cron.d.
func (m *Manager) WriteRootFile(ctx context.Context, path, text string) error {
return m.writeRoot(ctx, path, text)
}
// RemoveRootFile deletes one, for when the last schedule of a kind goes away.
func (m *Manager) RemoveRootFile(ctx context.Context, path string) error {
out, code, err := m.run(ctx, "rm", "-f", path)
if err != nil {
return err
}
if code != 0 {
return errors.New(strings.TrimSpace(string(out)))
}
return nil
}
func (m *Manager) writeRoot(ctx context.Context, path, text string) error {
args := []string{"tee", path}
if m.sudo {
args = priv.Wrap(args...)
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
cmd.Stdin = strings.NewReader(text)
var errb bytes.Buffer
cmd.Stdout, cmd.Stderr = nil, &errb
if err := cmd.Run(); err != nil {
msg := strings.TrimSpace(errb.String())
if msg == "" {
msg = err.Error()
}
return errors.New(msg)
}
return nil
}
internal/storage/snapshots_test.go
+135
View File
@@ -0,0 +1,135 @@
package storage
import (
"strings"
"testing"
"time"
"hamlab/internal/cron"
)
func TestSnapTimeReadsZfSnapNames(t *testing.T) {
// the real thing off grid
taken, exp, ok := SnapTime("tank@2026-09-24_02.30.01--6w")
if !ok {
t.Fatal("zfSnap's own name must parse")
}
if taken.Format("2006-01-02 15:04:05") != "2026-09-24 02:30:01" {
t.Fatalf("taken = %v", taken)
}
if got := exp.Sub(taken); got != 6*7*24*time.Hour {
t.Fatalf("ttl = %v, want 6w", got)
}
// a labelled one from a policy with a prefix
if _, e, ok := SnapTime("tank/apps@nightly-2026-09-24_02.30.01--30d"); !ok || e.IsZero() {
t.Fatal("a prefixed name must parse and expire")
}
// no TTL: never expires
if _, e, ok := SnapTime("tank@2026-09-24_02.30.01"); !ok || !e.IsZero() {
t.Fatal("a name without a TTL must parse and never expire")
}
// things that are not ours
for _, n := range []string{"tank@before-upgrade", "tank@manual", "tank", "tank@2026-13-45_99.99.99--6w"} {
if _, _, ok := SnapTime(n); ok {
t.Errorf("%q must not parse as a scheduled snapshot", n)
}
}
}
func TestCronExprAndNextRun(t *testing.T) {
now := time.Date(2026, 9, 24, 9, 15, 0, 0, time.Local)
for _, c := range []struct {
p Policy
want string
}{
{Policy{Every: "daily", At: "02:30"}, "30 2 * * *"},
{Policy{Every: "hourly", At: "00:30"}, "30 * * * *"},
{Policy{Every: "weekly", At: "04:00", Weekday: 0}, "0 4 * * 0"},
{Policy{Every: "monthly", At: "05:15", Day: 12}, "15 5 12 * *"},
{Policy{Every: "custom", Cron: "*/15 9-17 * * 1-5"}, "*/15 9-17 * * 1-5"},
} {
got, err := c.p.CronExpr()
if err != nil || got != c.want {
t.Errorf("CronExpr(%+v) = %q, %v; want %q", c.p.Every, got, err, c.want)
}
}
// the next run is read with the same evaluator the Jobs tab uses
daily := Policy{Every: "daily", At: "02:30"}
if n := daily.NextRun(now); n.Hour() != 2 || n.Minute() != 30 || n.Day() != 25 {
t.Fatalf("NextRun = %v, want 25th 02:30", n)
}
// a custom expression that does not parse is refused, not silently ignored
if _, err := (&Policy{Every: "custom", Cron: "not a schedule"}).CronExpr(); err == nil {
t.Fatal("a bad cron expression must be refused")
}
if _, err := (&Policy{Every: "custom", Cron: "@reboot"}).CronExpr(); err == nil {
t.Fatal("@reboot is not a snapshot schedule")
}
}
func TestCronFileText(t *testing.T) {
ps := []Policy{
{ID: "abc", Name: "tank nightly", Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6w", Enabled: true},
{ID: "def", Name: "paused one", Dataset: "tank/apps", Every: "hourly", At: "00:05", KeepFor: "2d"},
}
text, err := CronFileText(ps, "core", "/usr/local/bin/hamlab", "/home/core/apps/hamlab/config.json")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(text, "30 2 * * * core /usr/local/bin/hamlab -config /home/core/apps/hamlab/config.json zfs-snapshot abc") {
t.Fatalf("the enabled schedule is missing or malformed:\n%s", text)
}
if strings.Contains(text, "zfs-snapshot def") {
t.Fatalf("a paused schedule must not be scheduled:\n%s", text)
}
// every line cron will read must parse as a schedule
for _, line := range strings.Split(text, "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") || strings.Contains(line, "=") {
continue
}
f := strings.Fields(line)
if err := cron.ValidSchedule(strings.Join(f[:5], " ")); err != nil {
t.Errorf("line %q: %v", line, err)
}
}
}
func TestValidateRefusesNonsense(t *testing.T) {
ok := Policy{Dataset: "tank/apps", Every: "daily", At: "02:30", KeepFor: "6w"}
if err := ok.Validate(); err != nil {
t.Fatalf("a sane policy: %v", err)
}
bad := []Policy{
{Dataset: "", Every: "daily", At: "02:30", KeepFor: "6w"},
{Dataset: "tank", Every: "yearly", At: "02:30", KeepFor: "6w"},
{Dataset: "tank", Every: "custom", KeepFor: "6w"},
{Dataset: "tank", Every: "custom", Cron: "not a schedule", KeepFor: "6w"},
{Dataset: "tank", Every: "daily", At: "25:00", KeepFor: "6w"},
{Dataset: "tank", Every: "daily", At: "02:30"}, // no retention
{Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6 weeks"}, // unparseable
{Dataset: "tank", Every: "monthly", At: "02:30", Day: 31, KeepFor: "6w"}, // not every month has one
{Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6w", Prefix: "a--b"},
}
for i, p := range bad {
if err := p.Validate(); err == nil {
t.Errorf("bad policy %d was accepted: %+v", i, p)
}
}
}
func TestParseAge(t *testing.T) {
for s, want := range map[string]time.Duration{
"6w": 6 * 7 * 24 * time.Hour, "30d": 30 * 24 * time.Hour,
"12h": 12 * time.Hour, "90m": 90 * time.Minute,
} {
if got, err := ParseAge(s); err != nil || got != want {
t.Errorf("ParseAge(%q) = %v, %v; want %v", s, got, err, want)
}
}
for _, s := range []string{"", "6", "w", "-1d", "0w", "6y", "six weeks"} {
if _, err := ParseAge(s); err == nil {
t.Errorf("ParseAge(%q) should fail", s)
}
}
}
internal/storage/storage.go
+926
View File
@@ -0,0 +1,926 @@
// Package storage is the Disks tab: block devices from lsblk, SMART health via
// smartctl (optional; the tab explains how to get it), and ZFS pools/datasets/snapshots
// when the zfs tools and module are present. Spinning disks are never woken just to
// read SMART (-n standby).
package storage
import (
"bytes"
"context"
"encoding/hex"
"encoding/json"
"errors"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"syscall"
"time"
"hamlab/internal/priv"
)
// ---- block devices ----
type Part struct {
Name string `json:"name"`
Type string `json:"type"` // part | crypt | lvm | md | …
FSType string `json:"fstype,omitempty"`
Mount string `json:"mount,omitempty"`
Label string `json:"label,omitempty"`
Size uint64 `json:"size"`
Used uint64 `json:"used,omitempty"`
FSSize uint64 `json:"fs_size,omitempty"`
}
type Device struct {
Name string `json:"name"`
Kind string `json:"kind"` // nvme | ssd | hdd | usb | emmc | sd | other
Model string `json:"model,omitempty"`
Vendor string `json:"vendor,omitempty"`
Serial string `json:"serial,omitempty"`
Tran string `json:"tran,omitempty"`
Size uint64 `json:"size"`
Rotational bool `json:"rotational"`
Removable bool `json:"removable"`
Temp float64 `json:"temp,omitempty"`
Parts []Part `json:"parts"`
Namespaces []string `json:"namespaces,omitempty"` // NVMe: the block devices behind this controller
}
type lsblkNode struct {
Name string `json:"name"`
Type string `json:"type"`
Size uint64 `json:"size"`
Model *string `json:"model"`
Serial *string `json:"serial"`
Vendor *string `json:"vendor"`
Rota bool `json:"rota"`
Tran *string `json:"tran"`
FSType *string `json:"fstype"`
Mountpoint *string `json:"mountpoint"`
Label *string `json:"label"`
RM bool `json:"rm"`
Children []lsblkNode `json:"children"`
}
func str(p *string) string {
if p == nil {
return ""
}
return strings.TrimSpace(*p)
}
func Devices(ctx context.Context) ([]Device, error) {
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
defer cancel()
var root struct {
Blockdevices []lsblkNode `json:"blockdevices"`
}
// lsblk knows labels and stacked devices (LVM, LUKS, md); without it (Alpine and other
// busybox systems don't ship it) or with one too old for JSON, /sys has the essentials.
out, err := exec.CommandContext(ctx, "lsblk", "-J", "-b", "-o", "NAME,TYPE,SIZE,MODEL,SERIAL,VENDOR,ROTA,TRAN,FSTYPE,MOUNTPOINT,LABEL,RM").Output()
if err != nil || json.Unmarshal(out, &root) != nil {
root.Blockdevices = sysBlockDevices()
}
devs := []Device{}
for _, n := range root.Blockdevices {
if n.Type != "disk" || n.Size == 0 || strings.HasPrefix(n.Name, "zd") || mmcAreaRe.MatchString(n.Name) {
continue
}
d := Device{Name: n.Name, Model: str(n.Model), Vendor: str(n.Vendor), Serial: str(n.Serial), Tran: str(n.Tran),
Size: n.Size, Rotational: n.Rota, Removable: n.RM, Temp: blockTemp(n.Name), Parts: []Part{}}
switch {
case strings.HasPrefix(n.Name, "mmcblk"):
d.Kind = mmcKind(n.Name)
if d.Model == "" {
d.Model = strings.TrimSpace(readStrFile("/sys/block/" + n.Name + "/device/name"))
}
case d.Tran == "usb":
d.Kind = "usb"
case d.Tran == "nvme" || strings.HasPrefix(n.Name, "nvme"):
d.Kind = "nvme"
case n.Rota:
d.Kind = "hdd"
case d.Tran == "sata" || d.Tran == "sas" || strings.HasPrefix(n.Name, "sd"):
d.Kind = "ssd"
default:
d.Kind = "other"
}
var walk func(nodes []lsblkNode)
walk = func(nodes []lsblkNode) {
for _, c := range nodes {
p := Part{Name: c.Name, Type: c.Type, FSType: str(c.FSType), Mount: str(c.Mountpoint), Label: str(c.Label), Size: c.Size}
if p.Mount != "" && p.Mount != "[SWAP]" {
var st syscall.Statfs_t
if syscall.Statfs(p.Mount, &st) == nil {
bs := uint64(st.Bsize)
p.FSSize, p.Used = st.Blocks*bs, (st.Blocks-st.Bfree)*bs
}
}
d.Parts = append(d.Parts, p)
walk(c.Children)
}
}
walk(n.Children)
devs = append(devs, d)
}
devs = foldNVMe(devs)
sort.Slice(devs, func(i, j int) bool { return devs[i].Name < devs[j].Name })
return devs, nil
}
// sysBlockDevices builds the same tree lsblk would from /sys/block and /proc/mounts.
func sysBlockDevices() []lsblkNode {
type mnt struct{ point, fstype string }
mounts := map[string]mnt{}
if b, err := os.ReadFile("/proc/mounts"); err == nil {
for _, line := range strings.Split(string(b), "\n") {
if f := strings.Fields(line); len(f) >= 3 && strings.HasPrefix(f[0], "/dev/") {
dev := f[0]
if real, err := filepath.EvalSymlinks(dev); err == nil { // /dev/disk/by-uuid/…, /dev/mapper/…
dev = real
}
if _, seen := mounts[filepath.Base(dev)]; !seen {
mounts[filepath.Base(dev)] = mnt{strings.ReplaceAll(f[1], `\040`, " "), f[2]}
}
}
}
}
if b, err := os.ReadFile("/proc/swaps"); err == nil {
for _, line := range strings.Split(string(b), "\n") {
if f := strings.Fields(line); len(f) > 0 && strings.HasPrefix(f[0], "/dev/") {
mounts[filepath.Base(f[0])] = mnt{"[SWAP]", "swap"}
}
}
}
sp := func(v string) *string {
if v = strings.TrimSpace(v); v == "" {
return nil
}
return &v
}
node := func(dir, name, typ string) lsblkNode {
n := lsblkNode{Name: name, Type: typ, Size: u64(strings.TrimSpace(readStrFile(dir+"/size"))) * 512}
if m, ok := mounts[name]; ok {
n.Mountpoint, n.FSType = sp(m.point), sp(m.fstype)
}
return n
}
var out []lsblkNode
ents, _ := os.ReadDir("/sys/block")
for _, e := range ents {
name := e.Name()
dir := "/sys/block/" + name
if strings.HasPrefix(name, "loop") || strings.HasPrefix(name, "ram") || strings.HasPrefix(name, "zram") ||
strings.HasPrefix(name, "dm-") || strings.HasPrefix(name, "md") || strings.HasPrefix(name, "sr") || strings.HasPrefix(name, "nbd") {
continue
}
n := node(dir, name, "disk")
n.Model, n.Vendor = sp(readStrFile(dir+"/device/model")), sp(readStrFile(dir+"/device/vendor"))
n.Serial = sp(readStrFile(dir + "/device/serial"))
n.Rota = strings.TrimSpace(readStrFile(dir+"/queue/rotational")) == "1"
n.RM = strings.TrimSpace(readStrFile(dir+"/removable")) == "1"
real, _ := filepath.EvalSymlinks(dir)
switch {
case strings.HasPrefix(name, "nvme"):
n.Tran = sp("nvme")
case strings.Contains(real, "/usb"):
n.Tran = sp("usb")
case strings.Contains(real, "/ata"):
n.Tran = sp("sata")
}
subs, _ := os.ReadDir(dir)
for _, sub := range subs {
if _, err := os.Stat(dir + "/" + sub.Name() + "/partition"); err == nil {
n.Children = append(n.Children, node(dir+"/"+sub.Name(), sub.Name(), "part"))
}
}
out = append(out, n)
}
return out
}
// An eMMC chip's boot and RPMB areas are hardware regions of the same chip, not drives;
// the kernel lists them as disks of their own (mmcblk0boot0, mmcblk0boot1, mmcblk0rpmb).
var mmcAreaRe = regexp.MustCompile(`^mmcblk\d+(boot\d+|rpmb|gp\d+)$`)
func mmcKind(name string) string {
if strings.TrimSpace(readStrFile("/sys/block/"+name+"/device/type")) == "MMC" {
return "emmc"
}
return "sd"
}
// mmcHealth is the eMMC answer to SMART: the chip's own wear and end-of-life registers,
// which the kernel publishes in /sys. SD cards have neither; smartctl speaks to neither.
func mmcHealth(s Smart) Smart {
dir := "/sys/block/" + s.Device + "/device/"
s.Available, s.Supported = true, true
s.SelfTestNote = "not a feature of eMMC or SD"
s.Serial = strings.TrimSpace(readStrFile(dir + "serial"))
s.Firmware = strings.TrimSpace(readStrFile(dir + "fwrev"))
// fwrev is eight raw bytes in hex; vendors usually put ASCII in them ("0001")
if b, err := hex.DecodeString(strings.TrimPrefix(s.Firmware, "0x")); err == nil {
if t := strings.TrimRight(string(b), "\x00 "); t != "" && strings.IndexFunc(t, func(r rune) bool { return r < 0x20 || r > 0x7e }) < 0 {
s.Firmware = t
}
}
s.Model = strings.TrimSpace(readStrFile(dir + "name"))
hx := func(v string) int { n, _ := strconv.ParseInt(strings.TrimPrefix(v, "0x"), 16, 32); return int(n) }
life := strings.Fields(readStrFile(dir + "life_time"))
eol := strings.TrimSpace(readStrFile(dir + "pre_eol_info"))
if len(life) != 2 || eol == "" {
s.Error = "This card reports no health data (SD cards don't; eMMC 5.0 and later does)."
return s
}
m := &MMCHealth{LifeA: hx(life[0]), LifeB: hx(life[1]), PreEOL: hx(eol)}
s.MMC = m
ok := m.PreEOL <= 1 && m.LifeA < 10 && m.LifeB < 10
s.Passed = &ok
if m.PreEOL == 2 {
s.Warnings = append(s.Warnings, "80% of reserved blocks consumed")
} else if m.PreEOL >= 3 {
s.Warnings = append(s.Warnings, "90% of reserved blocks consumed — replace soon")
}
if m.LifeA >= 10 || m.LifeB >= 10 {
s.Warnings = append(s.Warnings, "estimated lifetime reached")
}
return s
}
var nsRe = regexp.MustCompile(`^(nvme\d+)n\d+$`)
// foldNVMe merges nvme0n1, nvme0n2, … into one physical device "nvme0": one SSD, one
// SMART record, however many namespaces the controller presents (Apple exposes three).
func foldNVMe(devs []Device) []Device {
byCtrl := map[string]*Device{}
var out []Device
for _, d := range devs {
m := nsRe.FindStringSubmatch(d.Name)
if m == nil {
out = append(out, d)
continue
}
ctrl := m[1]
c, ok := byCtrl[ctrl]
if !ok {
nd := d
nd.Name = ctrl
nd.Parts = []Part{}
nd.Namespaces = nil
if mdl := strings.TrimSpace(readStrFile("/sys/class/nvme/" + ctrl + "/model")); mdl != "" {
nd.Model = mdl
}
if sn := strings.TrimSpace(readStrFile("/sys/class/nvme/" + ctrl + "/serial")); sn != "" {
nd.Serial = sn
}
if t := ctrlTemp(ctrl); t > 0 {
nd.Temp = t
}
nd.Size = 0
c = &nd
byCtrl[ctrl] = c
out = append(out, Device{}) // placeholder keeps order; filled below
c.Namespaces = []string{}
}
c.Size += d.Size
c.Namespaces = append(c.Namespaces, d.Name)
c.Parts = append(c.Parts, d.Parts...)
}
// replace placeholders with the folded controllers, in first-seen order
res := out[:0]
seen := map[string]bool{}
for _, d := range out {
if d.Name != "" {
res = append(res, d)
continue
}
for _, c := range byCtrl {
if !seen[c.Name] {
seen[c.Name] = true
res = append(res, *c)
break
}
}
}
return res
}
func readStrFile(p string) string {
b, err := os.ReadFile(p)
if err != nil {
return ""
}
return string(b)
}
func ctrlTemp(ctrl string) float64 {
dirs, _ := filepath.Glob("/sys/class/nvme/" + ctrl + "/hwmon*")
for _, d := range dirs {
if b, err := os.ReadFile(filepath.Join(d, "temp1_input")); err == nil {
if v, err := strconv.Atoi(strings.TrimSpace(string(b))); err == nil {
return float64(v) / 1000
}
}
}
return 0
}
func blockTemp(name string) float64 {
for _, pat := range []string{"/sys/block/" + name + "/device/hwmon*", "/sys/block/" + name + "/device/hwmon/hwmon*"} {
dirs, _ := filepath.Glob(pat)
for _, d := range dirs {
if b, err := os.ReadFile(filepath.Join(d, "temp1_input")); err == nil {
if v, err := strconv.Atoi(strings.TrimSpace(string(b))); err == nil {
return float64(v) / 1000
}
}
}
}
return 0
}
var devRe = regexp.MustCompile(`^[a-z][a-z0-9]*$`)
func validDevice(name string) bool {
if !devRe.MatchString(name) {
return false
}
if _, err := os.Stat("/sys/block/" + name); err == nil {
return true
}
_, err := os.Stat("/sys/class/nvme/" + name) // NVMe controller (nvme0), which is what smartctl wants anyway
return err == nil
}
// ---- SMART ----
type Attr struct {
ID int `json:"id"`
Name string `json:"name"`
Value int `json:"value"`
Worst int `json:"worst"`
Thresh int `json:"thresh"`
Raw string `json:"raw"`
Failed string `json:"failed,omitempty"`
Warn bool `json:"warn"`
}
type Smart struct {
Device string `json:"device"`
Available bool `json:"available"` // smartctl installed
Supported bool `json:"supported"` // device answered
Standby bool `json:"standby"` // spun down; not read
Passed *bool `json:"passed"`
Temp int `json:"temp,omitempty"`
PowerOnHours int `json:"power_on_hours,omitempty"`
PowerCycles int `json:"power_cycles,omitempty"`
Model string `json:"model,omitempty"`
Serial string `json:"serial,omitempty"`
Firmware string `json:"firmware,omitempty"`
Attrs []Attr `json:"attrs"`
NVMe *NVMeLog `json:"nvme,omitempty"`
MMC *MMCHealth `json:"mmc,omitempty"`
SelfTest string `json:"self_test,omitempty"`
SelfTestPct int `json:"self_test_remaining,omitempty"`
SelfTestOK bool `json:"self_test_supported"`
SelfTestNote string `json:"self_test_note,omitempty"`
Warnings []string `json:"warnings"`
Error string `json:"error,omitempty"`
FetchedAt int64 `json:"fetched_at"`
}
// MMCHealth is the eMMC EXT_CSD wear report. Life values are tenths: 1 = 0–10% of rated
// life used, 10 = 90–100%, 11 = exceeded, 0 = not reported. A is the SLC area, B the MLC.
type MMCHealth struct {
LifeA int `json:"life_a"`
LifeB int `json:"life_b"`
PreEOL int `json:"pre_eol"` // 1 normal · 2 warning · 3 urgent
}
type NVMeLog struct {
CriticalWarning int `json:"critical_warning"`
PercentageUsed int `json:"percentage_used"`
AvailableSpare int `json:"available_spare"`
SpareThreshold int `json:"spare_threshold"`
MediaErrors int64 `json:"media_errors"`
ErrorLogEntries int64 `json:"error_log_entries"`
DataUnitsRead uint64 `json:"data_read_bytes"`
DataUnitsWritten uint64 `json:"data_written_bytes"`
UnsafeShutdowns int64 `json:"unsafe_shutdowns"`
}
type Manager struct {
sudo bool
smartctl bool
mu sync.Mutex
cache map[string]Smart
}
// lookPath also checks the sbin directories, which a user's PATH often lacks.
func lookPath(bin string) (string, bool) {
if p, err := exec.LookPath(bin); err == nil {
return p, true
}
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin"} {
if _, err := os.Stat(d + "/" + bin); err == nil {
return d + "/" + bin, true
}
}
return "", false
}
func New(sudo bool) *Manager {
_, ok := lookPath("smartctl")
return &Manager{sudo: sudo, smartctl: ok, cache: map[string]Smart{}}
}
// HasSmartctl re-probes when it was missing at startup, so an install from the Disks
// tab takes effect without a daemon restart.
func (m *Manager) HasSmartctl() bool {
if !m.smartctl {
_, m.smartctl = lookPath("smartctl")
}
return m.smartctl
}
func (m *Manager) run(ctx context.Context, args ...string) ([]byte, int, error) {
if p, ok := lookPath(args[0]); ok {
args[0] = p
}
if m.sudo {
args = priv.Wrap(args...)
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
var out, errb bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errb
err := cmd.Run()
code := 0
var ee *exec.ExitError
if errors.As(err, &ee) {
code = ee.ExitCode()
err = nil // smartctl and zpool use exit codes as status; the caller decides
if len(out.Bytes()) == 0 && errb.Len() > 0 {
err = errors.New(strings.TrimSpace(strings.Split(errb.String(), "\n")[0])) // nothing usable came back; say why
}
}
if err != nil {
msg := strings.TrimSpace(errb.String())
if msg == "" {
msg = err.Error()
}
return out.Bytes(), code, errors.New(msg)
}
return out.Bytes(), code, nil
}
func num(v any) int64 {
switch x := v.(type) {
case float64:
return int64(x)
case json.Number:
n, _ := x.Int64()
return n
}
return 0
}
// Smart reads SMART for one device. Cached for 10 minutes; spinning disks in standby are
// left asleep and reported as such. force re-reads (and will wake the disk).
func (m *Manager) Smart(ctx context.Context, dev string, force bool) Smart {
s := Smart{Device: dev, Available: m.smartctl, Attrs: []Attr{}, Warnings: []string{}, FetchedAt: time.Now().UnixMilli()}
if !validDevice(dev) {
s.Error = "unknown device"
return s
}
if strings.HasPrefix(dev, "mmcblk") {
return mmcHealth(s)
}
if !m.HasSmartctl() {
return s
}
m.mu.Lock()
if c, ok := m.cache[dev]; ok && !force && time.Since(time.UnixMilli(c.FetchedAt)) < 10*time.Minute {
m.mu.Unlock()
return c
}
m.mu.Unlock()
args := []string{"smartctl", "-j", "-a"}
if !force {
args = append(args, "-n", "standby")
}
args = append(args, "/dev/"+dev)
out, code, err := m.run(ctx, args...)
if err != nil && len(out) == 0 {
s.Error = err.Error()
return s
}
var raw map[string]any
if json.Unmarshal(out, &raw) != nil {
s.Error = "smartctl: unreadable output"
return s
}
if code&2 != 0 { // -n standby: device is asleep (also "open failed", but JSON will say)
if msgs, ok := raw["smartctl"].(map[string]any); ok {
for _, mm := range asList(msgs["messages"]) {
if strings.Contains(strings.ToLower(asString(mm["string"])), "standby") {
s.Standby = true
}
}
}
if s.Standby {
m.mu.Lock()
m.cache[dev] = s
m.mu.Unlock()
return s
}
}
if st, ok := raw["smart_status"].(map[string]any); ok {
p := st["passed"] == true
s.Passed = &p
s.Supported = true
}
s.Model = asString(raw["model_name"])
s.Serial = asString(raw["serial_number"])
s.Firmware = asString(raw["firmware_version"])
if t, ok := raw["temperature"].(map[string]any); ok {
s.Temp = int(num(t["current"]))
}
if t, ok := raw["power_on_time"].(map[string]any); ok {
s.PowerOnHours = int(num(t["hours"]))
}
s.PowerCycles = int(num(raw["power_cycle_count"]))
if a, ok := raw["ata_smart_attributes"].(map[string]any); ok {
for _, row := range asList(a["table"]) {
at := Attr{ID: int(num(row["id"])), Name: asString(row["name"]), Value: int(num(row["value"])), Worst: int(num(row["worst"])), Thresh: int(num(row["thresh"])), Failed: asString(row["when_failed"])}
if rw, ok := row["raw"].(map[string]any); ok {
at.Raw = asString(rw["string"])
rv := num(rw["value"])
switch at.ID {
case 5, 187, 188, 196, 197, 198:
at.Warn = rv > 0
case 199:
at.Warn = rv > 0
case 10, 11:
at.Warn = rv > 100
}
if at.Warn {
s.Warnings = append(s.Warnings, at.Name+" = "+at.Raw)
}
}
if at.Failed != "" {
at.Warn = true
s.Warnings = append(s.Warnings, at.Name+" failed ("+at.Failed+")")
}
s.Attrs = append(s.Attrs, at)
}
}
if n, ok := raw["nvme_smart_health_information_log"].(map[string]any); ok {
l := &NVMeLog{CriticalWarning: int(num(n["critical_warning"])), PercentageUsed: int(num(n["percentage_used"])),
AvailableSpare: int(num(n["available_spare"])), SpareThreshold: int(num(n["available_spare_threshold"])),
MediaErrors: num(n["media_errors"]), ErrorLogEntries: num(n["num_err_log_entries"]), UnsafeShutdowns: num(n["unsafe_shutdowns"]),
DataUnitsRead: uint64(num(n["data_units_read"])) * 512000, DataUnitsWritten: uint64(num(n["data_units_written"])) * 512000}
s.NVMe = l
s.Supported = true
if l.CriticalWarning != 0 {
s.Warnings = append(s.Warnings, "critical warning flags set")
}
if l.MediaErrors > 0 {
s.Warnings = append(s.Warnings, "media errors: "+strconv.FormatInt(l.MediaErrors, 10))
}
if l.SpareThreshold > 0 && l.AvailableSpare < l.SpareThreshold {
s.Warnings = append(s.Warnings, "available spare below threshold")
}
if l.PercentageUsed >= 90 {
s.Warnings = append(s.Warnings, "endurance "+strconv.Itoa(l.PercentageUsed)+"% used")
}
}
// Can this drive run self-tests from here? ATA says so outright; NVMe drives prove it
// by exposing the self-test log, and some (WD SN580 among them) advertise the command
// but reject the log page, which makes the whole feature unusable.
if d, ok := raw["ata_smart_data"].(map[string]any); ok {
if caps, ok := d["capabilities"].(map[string]any); ok {
s.SelfTestOK = caps["self_tests_supported"] == true
if !s.SelfTestOK {
s.SelfTestNote = "drive reports no self-test support"
}
}
}
if _, ok := raw["nvme_self_test_log"]; ok {
s.SelfTestOK = true
} else if s.NVMe != nil {
s.SelfTestNote = "this drive's firmware doesn't expose the NVMe self-test log, so tests can't be run or read from here"
}
if msgs, ok := raw["smartctl"].(map[string]any); ok {
for _, mm := range asList(msgs["messages"]) {
if strings.Contains(asString(mm["string"]), "Self-test Log failed") {
s.SelfTestOK = false
s.SelfTestNote = "firmware rejects the self-test log page (" + asString(mm["string"]) + ")"
}
}
}
if d, ok := raw["ata_smart_data"].(map[string]any); ok {
if st, ok := d["self_test"].(map[string]any); ok {
if ss, ok := st["status"].(map[string]any); ok {
s.SelfTest = asString(ss["string"])
s.SelfTestPct = int(num(ss["remaining_percent"]))
}
}
}
if s.Passed != nil && !*s.Passed {
s.Warnings = append([]string{"SMART overall health: FAILED"}, s.Warnings...)
}
if !s.Supported && s.Error == "" {
s.Error = "device does not report SMART"
if msgs, ok := raw["smartctl"].(map[string]any); ok {
for _, mm := range asList(msgs["messages"]) {
if asString(mm["severity"]) == "error" {
s.Error = asString(mm["string"])
}
}
}
}
m.mu.Lock()
m.cache[dev] = s
m.mu.Unlock()
return s
}
func asList(v any) []map[string]any {
var out []map[string]any
if l, ok := v.([]any); ok {
for _, x := range l {
if mm, ok := x.(map[string]any); ok {
out = append(out, mm)
}
}
}
return out
}
func asString(v any) string {
if s, ok := v.(string); ok {
return s
}
return ""
}
// SelfTest starts a short or long self-test.
func (m *Manager) SelfTest(ctx context.Context, dev, kind string) error {
if !validDevice(dev) {
return errors.New("unknown device")
}
if kind != "short" && kind != "long" {
return errors.New("test must be short or long")
}
if !m.smartctl {
return errors.New("smartctl is not installed")
}
if sm := m.Smart(ctx, dev, false); sm.Supported && !sm.SelfTestOK {
return errors.New("self-tests aren't available on this drive: " + sm.SelfTestNote)
}
out, code, err := m.run(ctx, "smartctl", "-t", kind, "/dev/"+dev)
if err != nil {
return err
}
if code != 0 {
return errors.New(strings.TrimSpace(lastLine(string(out))))
}
m.mu.Lock()
delete(m.cache, dev)
m.mu.Unlock()
return nil
}
func lastLine(s string) string {
lines := strings.Split(strings.TrimSpace(s), "\n")
return lines[len(lines)-1]
}
// ---- ZFS ----
type Pool struct {
Name string `json:"name"`
Size uint64 `json:"size"`
Alloc uint64 `json:"alloc"`
Free uint64 `json:"free"`
Frag int `json:"frag"`
Cap int `json:"cap"`
Health string `json:"health"`
Dedup string `json:"dedup"`
Scan string `json:"scan,omitempty"`
Errors string `json:"errors,omitempty"`
Config string `json:"config,omitempty"` // vdev tree, verbatim
}
type Dataset struct {
Name string `json:"name"`
Type string `json:"type"`
Used uint64 `json:"used"`
Avail uint64 `json:"avail"`
Refer uint64 `json:"refer"`
UsedBySnap uint64 `json:"used_by_snapshots"`
Mount string `json:"mount,omitempty"`
Compress string `json:"compressratio"`
Snapshots int `json:"snapshots"`
}
type Snapshot struct {
Name string `json:"name"`
Used uint64 `json:"used"`
Created int64 `json:"created"` // unix ms
}
type ZFS struct {
Available bool `json:"available"`
Message string `json:"message,omitempty"`
Pools []Pool `json:"pools"`
Datasets []Dataset `json:"datasets"`
}
var poolRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_.-]*$`)
var datasetRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_.:/-]*$`)
var snapRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.:-]*$`)
func u64(s string) uint64 { v, _ := strconv.ParseUint(s, 10, 64); return v }
func atoi(s string) int { v, _ := strconv.Atoi(strings.TrimSuffix(s, "%")); return v }
// Pools is the pool list alone, for callers that poll: one `zpool list`, tried without
// root first (it needs none on Linux) so polling writes nothing to the auth log.
func (m *Manager) Pools(ctx context.Context) []Pool {
bin, ok := lookPath("zpool")
if !ok {
return nil
}
args := []string{"list", "-Hp", "-o", "name,size,alloc,free,frag,cap,health,dedup"}
out, err := exec.CommandContext(ctx, bin, args...).Output()
if err != nil {
var code int
if out, code, err = m.run(ctx, append([]string{"zpool"}, args...)...); err != nil || code != 0 {
return nil
}
}
return parsePools(out)
}
func parsePools(out []byte) []Pool {
pools := []Pool{}
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
f := strings.Split(line, "\t")
if len(f) < 8 {
continue
}
pools = append(pools, Pool{Name: f[0], Size: u64(f[1]), Alloc: u64(f[2]), Free: u64(f[3]), Frag: atoi(f[4]), Cap: atoi(f[5]), Health: f[6], Dedup: f[7]})
}
return pools
}
func (m *Manager) ZFS(ctx context.Context) ZFS {
z := ZFS{Pools: []Pool{}, Datasets: []Dataset{}}
if _, ok := lookPath("zpool"); !ok {
z.Message = "ZFS tools not installed"
return z
}
out, code, err := m.run(ctx, "zpool", "list", "-Hp", "-o", "name,size,alloc,free,frag,cap,health,dedup")
if err != nil || code != 0 {
msg := "zpool list failed"
if err != nil {
msg = err.Error()
}
z.Message = msg
return z
}
z.Available = true
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
f := strings.Split(line, "\t")
if len(f) < 8 {
continue
}
p := Pool{Name: f[0], Size: u64(f[1]), Alloc: u64(f[2]), Free: u64(f[3]), Frag: atoi(f[4]), Cap: atoi(f[5]), Health: f[6], Dedup: f[7]}
if st, _, err := m.run(ctx, "zpool", "status", p.Name); err == nil {
p.Scan, p.Errors, p.Config = parseStatus(string(st))
}
z.Pools = append(z.Pools, p)
}
snapCount := map[string]int{}
if so, _, err := m.run(ctx, "zfs", "list", "-t", "snapshot", "-H", "-o", "name"); err == nil {
for _, line := range strings.Split(strings.TrimSpace(string(so)), "\n") {
if ds, _, ok := strings.Cut(line, "@"); ok {
snapCount[ds]++
}
}
}
if do, _, err := m.run(ctx, "zfs", "list", "-Hp", "-t", "filesystem,volume", "-o", "name,type,used,avail,refer,usedbysnapshots,mountpoint,compressratio"); err == nil {
for _, line := range strings.Split(strings.TrimSpace(string(do)), "\n") {
f := strings.Split(line, "\t")
if len(f) < 8 {
continue
}
d := Dataset{Name: f[0], Type: f[1], Used: u64(f[2]), Avail: u64(f[3]), Refer: u64(f[4]), UsedBySnap: u64(f[5]), Compress: f[7], Snapshots: snapCount[f[0]]}
if f[6] != "-" && f[6] != "none" && f[6] != "legacy" {
d.Mount = f[6]
}
z.Datasets = append(z.Datasets, d)
}
}
return z
}
// parseStatus pulls the scan line, the errors line and the vdev table out of zpool status.
func parseStatus(text string) (scan, errs, config string) {
lines := strings.Split(text, "\n")
inConfig := false
var cfg []string
for i := 0; i < len(lines); i++ {
l := lines[i]
t := strings.TrimSpace(l)
switch {
case strings.HasPrefix(t, "scan:"):
scan = strings.TrimSpace(strings.TrimPrefix(t, "scan:"))
for i+1 < len(lines) && strings.HasPrefix(lines[i+1], "\t") && !strings.Contains(lines[i+1], ":") {
i++
scan += " " + strings.TrimSpace(lines[i])
}
case strings.HasPrefix(t, "errors:"):
errs = strings.TrimSpace(strings.TrimPrefix(t, "errors:"))
inConfig = false
case strings.HasPrefix(t, "config:"):
inConfig = true
case inConfig && t != "":
cfg = append(cfg, strings.TrimPrefix(l, "\t"))
}
}
return scan, errs, strings.Join(cfg, "\n")
}
func (m *Manager) Snapshots(ctx context.Context, dataset string) ([]Snapshot, error) {
if !datasetRe.MatchString(dataset) {
return nil, errors.New("bad dataset name")
}
out, code, err := m.run(ctx, "zfs", "list", "-t", "snapshot", "-Hp", "-d", "1", "-o", "name,used,creation", "-s", "creation", dataset)
if err != nil {
return nil, err
}
if code != 0 {
return nil, errors.New("zfs list failed")
}
snaps := []Snapshot{}
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
f := strings.Split(line, "\t")
if len(f) < 3 {
continue
}
c, _ := strconv.ParseInt(f[2], 10, 64)
snaps = append(snaps, Snapshot{Name: f[0], Used: u64(f[1]), Created: c * 1000})
}
return snaps, nil
}
// Scrub starts, pauses or stops a scrub. action: start | pause | stop.
func (m *Manager) Scrub(ctx context.Context, pool, action string) error {
if !poolRe.MatchString(pool) {
return errors.New("bad pool name")
}
args := []string{"zpool", "scrub"}
switch action {
case "start":
case "pause":
args = append(args, "-p")
case "stop":
args = append(args, "-s")
default:
return errors.New("unknown scrub action")
}
out, code, err := m.run(ctx, append(args, pool)...)
if err != nil {
return err
}
if code != 0 {
return errors.New(strings.TrimSpace(string(out)))
}
return nil
}
// CreateSnapshot takes dataset@name.
func (m *Manager) CreateSnapshot(ctx context.Context, dataset, name string) error {
if !datasetRe.MatchString(dataset) || !snapRe.MatchString(name) {
return errors.New("bad snapshot name")
}
out, code, err := m.run(ctx, "zfs", "snapshot", dataset+"@"+name)
if err != nil {
return err
}
if code != 0 {
return errors.New(strings.TrimSpace(string(out)))
}
return nil
}
internal/storage/sys_test.go
+22
View File
@@ -0,0 +1,22 @@
package storage
import (
"os"
"testing"
)
// The /sys fallback must find the same disks and partitions lsblk does.
func TestSysBlockDevices(t *testing.T) {
if _, err := os.Stat("/sys/block"); err != nil {
t.Skip("no /sys/block")
}
for _, d := range sysBlockDevices() {
if d.Name == "" || d.Type != "disk" {
t.Fatalf("bad node %+v", d)
}
t.Logf("%s size=%d tran=%s rota=%v parts=%d", d.Name, d.Size, str(d.Tran), d.Rota, len(d.Children))
for _, c := range d.Children {
t.Logf(" %s size=%d fs=%s mount=%s", c.Name, c.Size, str(c.FSType), str(c.Mountpoint))
}
}
}
internal/store/store.go
+95 -7
View File
@@ -5,6 +5,7 @@ package store
import (
"encoding/json"
"errors"
"log"
"os"
"sync"
"time"
@@ -15,9 +16,74 @@ type Terminal struct {
Name string `json:"name"`
}
// Session is one signed-in browser. The map key is the cookie's token; ID is what the
// UI sees and revokes by, so a session list never shows a usable token.
type Session struct {
ID string `json:"id"`
User string `json:"user"`
Expires time.Time `json:"expires"`
Created time.Time `json:"created,omitempty"`
LastSeen time.Time `json:"last_seen,omitempty"`
IP string `json:"ip,omitempty"`
Agent string `json:"agent,omitempty"`
}
// UnmarshalJSON also reads the pre-users form, a bare expiry time: such a session has no
// user yet; the server gives it to the migrated first account.
func (s *Session) UnmarshalJSON(b []byte) error {
if len(b) > 0 && b[0] == '"' {
return json.Unmarshal(b, &s.Expires)
}
type plain Session
return json.Unmarshal(b, (*plain)(s))
}
// UserState is what follows an account rather than the machine: how the console looks
// to that person and how their dashboard is laid out.
type UserState struct {
UI json.RawMessage `json:"ui,omitempty"` // theme, accents; opaque to the server
}
// RememberedMount is one bind mount of a container that was running when Ham Lab last
// looked. A compose stack records the path its file had *inside* whatever deployed it, so
// when that container is gone the path is the only way back to the file on disk.
type RememberedMount struct {
Container string `json:"container"`
Dest string `json:"dest"` // the path inside the container
Src string `json:"src"` // the path on the machine
Seen time.Time `json:"seen"`
}
type State struct {
Terminals []Terminal `json:"terminals"`
Sessions map[string]time.Time `json:"sessions"` // token -> expiry
Terminals []Terminal `json:"terminals"`
Sessions map[string]Session `json:"sessions"` // token -> session
Users map[string]*UserState `json:"users,omitempty"`
// UI is the pre-users machine-wide look; on first boot with users it moves to the first
// account. Kept so an old state file still reads.
UI json.RawMessage `json:"ui,omitempty"`
Alerts json.RawMessage `json:"alerts,omitempty"` // rules, channels and recent events (internal/api/alerts.go)
ZFS json.RawMessage `json:"zfs,omitempty"` // snapshot schedules (internal/api/zfs.go)
DockerJobs json.RawMessage `json:"docker_jobs,omitempty"` // image-update schedules (internal/api/dockersched.go)
Network json.RawMessage `json:"network,omitempty"` // speed test results (internal/api/network.go)
// Mounts remembers where a container's bind mounts pointed, so a stack deployed by an
// orchestrator that has since been removed can still be found on disk. See
// internal/api/stacks.go.
Mounts []RememberedMount `json:"mounts,omitempty"`
SamplerMs int `json:"sampler_ms,omitempty"` // how often this machine samples; the hub keeps every machine on the console's refresh rate
HistorySec int `json:"history_sec,omitempty"` // how far back the graphs reach; the hub keeps every machine on the same one
}
// UserState returns the prefs of an account, making an empty one if needed.
func (st *State) UserState(name string) *UserState {
if st.Users == nil {
st.Users = map[string]*UserState{}
}
u := st.Users[name]
if u == nil {
u = &UserState{}
st.Users[name] = u
}
return u
}
type Store struct {
@@ -33,12 +99,16 @@ func Open(path string) (*Store, error) {
return nil, err
}
if err == nil {
// State is conveniences (sessions, terminal names, layout). A file mangled by a power
// cut must not keep the console down: set it aside and start clean.
if err := json.Unmarshal(b, &s.state); err != nil {
return nil, err
s.state = State{}
_ = os.Rename(path, path+".corrupt")
log.Printf("state: %s unreadable (%v); kept as %s.corrupt, starting fresh", path, err, path)
}
}
if s.state.Sessions == nil {
s.state.Sessions = map[string]time.Time{}
s.state.Sessions = map[string]Session{}
}
return s, nil
}
@@ -63,9 +133,27 @@ func (s *Store) save() error {
if err != nil {
return err
}
tmp := s.path + ".tmp"
if err := os.WriteFile(tmp, b, 0o600); err != nil {
return WriteFileAtomic(s.path, b, 0o600)
}
// WriteFileAtomic writes beside path, syncs, then renames: after a crash the file is the
// old version or the new one, never empty or half-written.
func WriteFileAtomic(path string, data []byte, perm os.FileMode) error {
tmp := path + ".tmp"
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, perm)
if err != nil {
return err
}
return os.Rename(tmp, s.path)
_, err = f.Write(data)
if err == nil {
err = f.Sync()
}
if cerr := f.Close(); err == nil {
err = cerr
}
if err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, path)
}
internal/tasks/tasks.go
+260
View File
@@ -0,0 +1,260 @@
// Package tasks runs long commands (apt upgrades, and later anything else the UI kicks
// off) in the background, keeps their output, and lets any number of clients follow it
// live. It is also the task history: what ran, when, and how it ended.
package tasks
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"errors"
"os/exec"
"sort"
"sync"
"syscall"
"time"
)
const (
maxOutput = 1 << 20 // bytes kept per task
maxHistory = 200
)
type Task struct {
ID string
Kind string // "updates", "cron", …
Title string
Command string
Started time.Time
Finished time.Time
Running bool
ExitCode int
Err string
mu sync.Mutex
buf bytes.Buffer
subs map[chan []byte]struct{}
cancel context.CancelFunc
onCancel func()
}
type Snapshot struct {
ID string `json:"id"`
Kind string `json:"kind"`
Title string `json:"title"`
Command string `json:"command"`
Started time.Time `json:"started"`
Finished *time.Time `json:"finished,omitempty"`
Running bool `json:"running"`
ExitCode int `json:"exit_code"`
Err string `json:"error,omitempty"`
Output string `json:"output,omitempty"`
}
type Manager struct {
mu sync.Mutex
tasks map[string]*Task
}
func New() *Manager { return &Manager{tasks: map[string]*Task{}} }
func newID() string {
b := make([]byte, 6)
_, _ = rand.Read(b)
return hex.EncodeToString(b)
}
// Start launches cmd with its output captured into a new task. cmd must not have
// Stdout/Stderr set. The task ends when the process exits or Cancel is called.
func (m *Manager) Start(kind, title string, cmd *exec.Cmd) (*Task, error) {
ctx, cancel := context.WithCancel(context.Background())
t := &Task{ID: newID(), Kind: kind, Title: title, Command: cmd.String(), Started: time.Now(), Running: true,
subs: map[chan []byte]struct{}{}, cancel: cancel}
w := &writer{t: t}
cmd.Stdout, cmd.Stderr = w, w
// Its own process group: a task is usually `sh -c "a && b"`, and cancelling must reach a and
// b, not just the shell — killing only the shell left compose running and, because compose
// still held the output pipe, left the task "running" too. WaitDelay is the backstop for
// anything that escapes the group and keeps the pipe open.
if cmd.SysProcAttr == nil {
cmd.SysProcAttr = &syscall.SysProcAttr{}
}
cmd.SysProcAttr.Setpgid = true
cmd.WaitDelay = 5 * time.Second
if err := cmd.Start(); err != nil {
cancel()
return nil, err
}
m.mu.Lock()
m.tasks[t.ID] = t
m.prune()
m.mu.Unlock()
go func() {
done := make(chan error, 1)
go func() { done <- cmd.Wait() }()
var err error
select {
case err = <-done:
case <-ctx.Done():
t.mu.Lock()
hook := t.onCancel
t.mu.Unlock()
if hook != nil {
go hook()
}
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM) // asked first: compose and apt both stop cleanly
select {
case err = <-done:
case <-time.After(4 * time.Second):
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
_ = cmd.Process.Kill()
err = <-done
}
t.mu.Lock()
t.Err = "cancelled"
t.mu.Unlock()
}
t.mu.Lock()
t.Running, t.Finished = false, time.Now()
if err != nil {
var ee *exec.ExitError
if errors.As(err, &ee) {
t.ExitCode = ee.ExitCode()
} else {
t.ExitCode = -1
if t.Err == "" {
t.Err = err.Error()
}
}
}
for ch := range t.subs {
close(ch)
}
t.subs = map[chan []byte]struct{}{}
t.mu.Unlock()
cancel()
}()
return t, nil
}
// prune keeps the newest maxHistory finished tasks. Caller holds m.mu.
func (m *Manager) prune() {
if len(m.tasks) <= maxHistory {
return
}
var finished []*Task
for _, t := range m.tasks {
if !t.Running {
finished = append(finished, t)
}
}
sort.Slice(finished, func(i, j int) bool { return finished[i].Started.Before(finished[j].Started) })
for i := 0; i < len(finished) && len(m.tasks) > maxHistory; i++ {
delete(m.tasks, finished[i].ID)
}
}
func (m *Manager) Get(id string) (*Task, bool) {
m.mu.Lock()
defer m.mu.Unlock()
t, ok := m.tasks[id]
return t, ok
}
// List returns every task newest first, without output.
func (m *Manager) List() []Snapshot {
m.mu.Lock()
all := make([]*Task, 0, len(m.tasks))
for _, t := range m.tasks {
all = append(all, t)
}
m.mu.Unlock()
sort.Slice(all, func(i, j int) bool { return all[i].Started.After(all[j].Started) })
out := make([]Snapshot, 0, len(all))
for _, t := range all {
out = append(out, t.Snapshot(false))
}
return out
}
// RunningOf reports whether a task of the given kind is in flight.
func (m *Manager) RunningOf(kind string) (*Task, bool) {
m.mu.Lock()
defer m.mu.Unlock()
for _, t := range m.tasks {
if t.Kind == kind && t.Running {
return t, true
}
}
return nil, false
}
func (t *Task) Snapshot(withOutput bool) Snapshot {
t.mu.Lock()
defer t.mu.Unlock()
s := Snapshot{ID: t.ID, Kind: t.Kind, Title: t.Title, Command: t.Command, Started: t.Started, Running: t.Running, ExitCode: t.ExitCode, Err: t.Err}
if !t.Running {
f := t.Finished
s.Finished = &f
}
if withOutput {
s.Output = t.buf.String()
}
return s
}
// Subscribe returns the output so far plus a channel of further chunks, closed when the
// task ends. Call unsub when done following.
func (t *Task) Subscribe() (backlog []byte, ch chan []byte, unsub func()) {
t.mu.Lock()
defer t.mu.Unlock()
backlog = append([]byte(nil), t.buf.Bytes()...)
if !t.Running {
c := make(chan []byte)
close(c)
return backlog, c, func() {}
}
ch = make(chan []byte, 256)
t.subs[ch] = struct{}{}
return backlog, ch, func() {
t.mu.Lock()
if _, ok := t.subs[ch]; ok {
delete(t.subs, ch)
close(ch)
}
t.mu.Unlock()
}
}
func (t *Task) Cancel() { t.cancel() }
// OnCancel sets something to do, besides stopping the task's own processes, when the task is
// cancelled: whatever it started that isn't its child (a command inside a container).
func (t *Task) OnCancel(fn func()) {
t.mu.Lock()
t.onCancel = fn
t.mu.Unlock()
}
type writer struct{ t *Task }
func (w *writer) Write(p []byte) (int, error) {
t := w.t
t.mu.Lock()
defer t.mu.Unlock()
t.buf.Write(p)
if t.buf.Len() > maxOutput {
b := t.buf.Bytes()
t.buf = *bytes.NewBuffer(append([]byte("…\n"), b[len(b)-maxOutput*3/4:]...))
}
chunk := append([]byte(nil), p...)
for ch := range t.subs {
select {
case ch <- chunk:
default: // slow follower: drop the chunk rather than stall the process
}
}
return len(p), nil
}
internal/term/manager.go
+26 -2
View File
@@ -9,6 +9,7 @@ import (
"os/exec"
"sort"
"sync"
"sync/atomic"
"syscall"
"time"
@@ -68,6 +69,7 @@ type Manager struct {
shell string
home string
OnChange func([]Info) // called (without locks held) when the list changes
closing atomic.Bool
}
func NewManager(shell, home string) *Manager {
@@ -82,10 +84,21 @@ func newID() string {
// Create starts a new login shell. id may be empty.
func (m *Manager) Create(id, name string) (*Session, error) {
return m.CreateCmd(id, name, nil)
}
// CreateCmd is Create with something other than a login shell: argv nil is the shell, and
// anything else runs that instead — `docker exec -it …` for a container's console. Every
// session is the same thing afterwards, so attach, resize, the scrollback ring and the tab
// strip work without knowing which it is.
func (m *Manager) CreateCmd(id, name string, argv []string) (*Session, error) {
if id == "" {
id = newID()
}
cmd := exec.Command(m.shell, "-l")
if len(argv) > 0 {
cmd = exec.Command(argv[0], argv[1:]...)
}
cmd.Dir = m.home
cmd.Env = append(os.Environ(), "TERM=xterm-256color", "COLORTERM=truecolor", "HAMLAB=1")
ptmx, err := pty.StartWithSize(cmd, &pty.Winsize{Cols: defaultCol, Rows: defaultRow})
@@ -130,6 +143,15 @@ func (m *Manager) pump(s *Session) {
s.mu.Unlock()
_ = s.ptmx.Close()
m.changed()
// An exited shell stays listed for a while so the exit is visible, then reaps itself.
time.AfterFunc(10*time.Minute, func() {
m.mu.Lock()
cur, ok := m.sessions[s.ID]
m.mu.Unlock()
if ok && cur == s && !s.running {
_ = m.Close(s.ID)
}
})
}
// broadcast requires s.mu. Slow clients are dropped rather than blocking the PTY.
@@ -274,13 +296,15 @@ func (m *Manager) Close(id string) error {
}
func (m *Manager) changed() {
if m.OnChange != nil {
if m.OnChange != nil && !m.closing.Load() {
m.OnChange(m.List())
}
}
// Shutdown hangs up every shell. Called on daemon exit.
// Shutdown hangs up every shell. Called on daemon exit. The saved list is left as it
// was, because that list is what the next start re-opens.
func (m *Manager) Shutdown() {
m.closing.Store(true)
for _, s := range m.List() {
_ = m.Close(s.ID)
}
internal/tlsauto/tlsauto.go
+125
View File
@@ -0,0 +1,125 @@
// Package tlsauto makes a self-signed certificate for a node so hub↔node traffic is
// encrypted without any CA. The hub pins the certificate's fingerprint.
package tlsauto
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/hex"
"encoding/pem"
"errors"
"math/big"
"net"
"os"
"path/filepath"
"strings"
"time"
)
// Fingerprint formats a certificate's SHA-256 the way the UI shows it.
func Fingerprint(der []byte) string {
sum := sha256.Sum256(der)
h := hex.EncodeToString(sum[:])
var b strings.Builder
for i := 0; i < len(h); i += 2 {
if i > 0 {
b.WriteByte(':')
}
b.WriteString(strings.ToUpper(h[i : i+2]))
}
return b.String()
}
// Ensure returns the cert/key paths and fingerprint, generating them on first use.
func Ensure(dir string) (certPath, keyPath, fingerprint string, err error) {
certPath, keyPath = filepath.Join(dir, "cert.pem"), filepath.Join(dir, "key.pem")
if _, err := os.Stat(certPath); err == nil {
if _, err := os.Stat(keyPath); err == nil {
c, err := tls.LoadX509KeyPair(certPath, keyPath)
if err == nil && len(c.Certificate) > 0 {
return certPath, keyPath, Fingerprint(c.Certificate[0]), nil
}
}
}
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
return "", "", "", err
}
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 127))
if err != nil {
return "", "", "", err
}
host, _ := os.Hostname()
tmpl := &x509.Certificate{
SerialNumber: serial,
Subject: pkix.Name{CommonName: "hamlab@" + host, Organization: []string{"Ham Lab"}},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().AddDate(10, 0, 0),
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
DNSNames: []string{host, "localhost"},
IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
}
if addrs, err := net.InterfaceAddrs(); err == nil {
for _, a := range addrs {
if ipn, ok := a.(*net.IPNet); ok && !ipn.IP.IsLoopback() && ipn.IP.To4() != nil {
tmpl.IPAddresses = append(tmpl.IPAddresses, ipn.IP)
}
}
}
der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &priv.PublicKey, priv)
if err != nil {
return "", "", "", err
}
keyDER, err := x509.MarshalECPrivateKey(priv)
if err != nil {
return "", "", "", err
}
if err := os.MkdirAll(dir, 0o700); err != nil {
return "", "", "", err
}
if err := os.WriteFile(certPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0o644); err != nil {
return "", "", "", err
}
if err := os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER}), 0o600); err != nil {
return "", "", "", err
}
return certPath, keyPath, Fingerprint(der), nil
}
// PinnedConfig returns a client TLS config that accepts exactly one certificate.
func PinnedConfig(fingerprint string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, // we verify by fingerprint below, not by CA
VerifyPeerCertificate: func(raw [][]byte, _ [][]*x509.Certificate) error {
if len(raw) == 0 {
return errors.New("no certificate presented")
}
if Fingerprint(raw[0]) != fingerprint {
return errors.New("certificate fingerprint changed: " + Fingerprint(raw[0]))
}
return nil
},
}
}
// Peek connects and returns the fingerprint the server presents (trust on first use).
func Peek(addr string) (string, error) {
d := &net.Dialer{Timeout: 5 * time.Second}
conn, err := tls.DialWithDialer(d, "tcp", addr, &tls.Config{InsecureSkipVerify: true})
if err != nil {
return "", err
}
defer conn.Close()
certs := conn.ConnectionState().PeerCertificates
if len(certs) == 0 {
return "", errors.New("no certificate presented")
}
return Fingerprint(certs[0].Raw), nil
}
internal/updates/updates.go
+544
View File
@@ -0,0 +1,544 @@
// Package updates is the package-manager view: what can be upgraded, when the index
// was last refreshed, whether a reboot is pending, and the refresh/upgrade actions,
// which run as background tasks so their output can be followed live.
//
// Supported: apt (Debian, Ubuntu), dnf (Fedora, RHEL family), zypper (openSUSE, SLES),
// pacman (Arch) and apk (Alpine). All of them list, refresh, upgrade and install; pacman
// upgrades everything or nothing, because Arch doesn't support partial upgrades.
package updates
import (
"bufio"
"context"
"errors"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
"time"
"hamlab/internal/priv"
"hamlab/internal/tasks"
)
type Package struct {
Name string `json:"name"`
Current string `json:"current"`
Candidate string `json:"candidate"`
Suite string `json:"suite,omitempty"`
Security bool `json:"security"`
Description string `json:"description,omitempty"`
}
type Status struct {
Manager string `json:"manager"` // apt | dnf | zypper | pacman | apk | ""
CanUpgrade bool `json:"can_upgrade"`
Selective bool `json:"selective"` // individual packages can be upgraded
Packages []Package `json:"packages"`
SecurityCount int `json:"security_count"`
LastRefresh int64 `json:"last_refresh,omitempty"` // unix ms
LastUpgrade int64 `json:"last_upgrade,omitempty"`
RebootNeeded bool `json:"reboot_needed"`
RebootPkgs []string `json:"reboot_pkgs"`
AutoUpgrades bool `json:"auto_upgrades"`
RunningTask string `json:"running_task,omitempty"`
Sudo bool `json:"sudo"`
}
type Manager struct {
sudo bool
tasks *tasks.Manager
pm string
}
func New(sudo bool, tm *tasks.Manager) *Manager {
m := &Manager{sudo: sudo, tasks: tm}
for _, pm := range []string{"apt-get", "dnf", "zypper", "pacman", "apk"} {
if _, err := exec.LookPath(pm); err == nil {
m.pm = strings.TrimSuffix(pm, "-get")
break
}
}
return m
}
func (m *Manager) sudoArgs(args ...string) []string {
if m.sudo {
return priv.Wrap(args...)
}
return args
}
func out(ctx context.Context, args ...string) (string, error) {
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
cmd.Env = append(os.Environ(), "LC_ALL=C", "DEBIAN_FRONTEND=noninteractive")
b, err := cmd.Output()
if err != nil {
var ee *exec.ExitError
if errors.As(err, &ee) && len(ee.Stderr) > 0 {
return string(b), errors.New(strings.TrimSpace(string(ee.Stderr)))
}
return string(b), err
}
return string(b), nil
}
// newestMtime: the newest mtime among the paths matching any of the globs.
func newestMtime(globs ...string) int64 {
var best int64
for _, g := range globs {
matches, _ := filepath.Glob(g)
for _, m := range matches {
if t := mtimeMs(m); t > best {
best = t
}
}
}
return best
}
// dnfLastUpgrade: newest transaction in `dnf history` whose action includes an upgrade,
// else the newest transaction of any kind.
func dnfLastUpgrade(ctx context.Context) int64 {
text, err := out(ctx, "dnf", "history", "list")
if err != nil {
return 0
}
re := regexp.MustCompile(`(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+(\S+)`)
var newest, upgrade int64
for _, line := range strings.Split(text, "\n") {
m := re.FindStringSubmatch(line)
if m == nil {
continue
}
t, err := time.ParseInLocation("2006-01-02 15:04:05", m[1], time.Local)
if err != nil {
continue
}
ms := t.UnixMilli()
if ms > newest {
newest = ms
}
if strings.Contains(m[2], "U") && ms > upgrade {
upgrade = ms
}
}
if upgrade > 0 {
return upgrade
}
return newest
}
func mtimeMs(p string) int64 {
st, err := os.Stat(p)
if err != nil {
return 0
}
return st.ModTime().UnixMilli()
}
// ---- status ----
func (m *Manager) Status(ctx context.Context) Status {
s := Status{Manager: m.pm, Packages: []Package{}, RebootPkgs: []string{}, Sudo: m.sudo}
if t, ok := m.tasks.RunningOf("updates"); ok {
s.RunningTask = t.ID
}
s.CanUpgrade = m.sudo && m.pm != ""
s.Selective = m.pm != "pacman"
switch m.pm {
case "apt":
s.Packages = aptUpgradable(ctx)
s.LastRefresh = mtimeMs("/var/lib/apt/periodic/update-success-stamp")
if s.LastRefresh == 0 {
s.LastRefresh = mtimeMs("/var/lib/apt/lists/partial")
}
if s.LastRefresh == 0 {
s.LastRefresh = mtimeMs("/var/lib/apt/lists")
}
s.LastUpgrade = aptLastUpgrade()
s.AutoUpgrades = aptAutoUpgrades(ctx)
case "dnf":
s.Packages = dnfUpgradable(ctx)
s.LastRefresh = newestMtime("/var/cache/libdnf5/*/repodata", "/var/cache/dnf/*/repodata", "/var/cache/libdnf5", "/var/cache/dnf")
s.LastUpgrade = dnfLastUpgrade(ctx)
case "zypper":
s.Packages = zypperUpgradable(ctx)
s.LastRefresh = newestMtime("/var/cache/zypp/raw/*", "/var/cache/zypp/raw")
s.LastUpgrade = mtimeMs("/var/log/zypp/history")
case "pacman":
s.Packages = pacmanUpgradable(ctx)
s.LastRefresh = newestMtime("/var/lib/pacman/sync/*.db", "/var/lib/pacman/sync")
s.LastUpgrade = pacmanLastUpgrade()
case "apk":
s.Packages = apkUpgradable(ctx)
s.LastRefresh = newestMtime("/var/cache/apk/APKINDEX.*", "/var/lib/apk/APKINDEX.*")
s.LastUpgrade = mtimeMs("/lib/apk/db/installed")
}
for _, p := range s.Packages {
if p.Security {
s.SecurityCount++
}
}
if _, err := os.Stat("/run/reboot-required"); err == nil {
s.RebootNeeded = true
if b, err := os.ReadFile("/run/reboot-required.pkgs"); err == nil {
for _, l := range strings.Split(strings.TrimSpace(string(b)), "\n") {
if l = strings.TrimSpace(l); l != "" {
s.RebootPkgs = append(s.RebootPkgs, l)
}
}
}
} else if m.pm != "apt" {
s.RebootNeeded = kernelChanged()
}
return s
}
// kernelChanged: the running kernel isn't the newest installed one (dnf/pacman hosts).
func kernelChanged() bool {
b, err := os.ReadFile("/proc/sys/kernel/osrelease")
if err != nil {
return false
}
running := strings.TrimSpace(string(b))
ents, err := os.ReadDir("/lib/modules")
if err != nil {
return false
}
newest := ""
for _, e := range ents {
if e.Name() > newest {
newest = e.Name()
}
}
return newest != "" && newest != running
}
// ---- apt ----
var aptListRe = regexp.MustCompile(`^(\S+)/(\S+)\s+(\S+)\s+\S+\s+\[upgradable from: (\S+)\]`)
func aptUpgradable(ctx context.Context) []Package {
text, _ := out(ctx, "apt", "list", "--upgradable")
var pkgs []Package
for _, line := range strings.Split(text, "\n") {
m := aptListRe.FindStringSubmatch(line)
if m == nil {
continue
}
pkgs = append(pkgs, Package{Name: m[1], Suite: m[2], Candidate: m[3], Current: m[4],
Security: strings.Contains(m[2], "security")})
}
if len(pkgs) == 0 {
return []Package{}
}
// One apt-cache call for all the descriptions.
args := append([]string{"apt-cache", "show", "--no-all-versions"}, names(pkgs)...)
if text, err := out(ctx, args...); err == nil {
desc := map[string]string{}
cur := ""
for _, line := range strings.Split(text, "\n") {
if strings.HasPrefix(line, "Package: ") {
cur = strings.TrimPrefix(line, "Package: ")
} else if strings.HasPrefix(line, "Description-en: ") || strings.HasPrefix(line, "Description: ") {
if _, ok := desc[cur]; !ok {
_, v, _ := strings.Cut(line, ": ")
desc[cur] = v
}
}
}
for i := range pkgs {
pkgs[i].Description = desc[pkgs[i].Name]
}
}
sort.Slice(pkgs, func(i, j int) bool { return pkgs[i].Name < pkgs[j].Name })
return pkgs
}
func names(pkgs []Package) []string {
n := make([]string, len(pkgs))
for i, p := range pkgs {
n[i] = p.Name
}
return n
}
// aptLastUpgrade: the Start-Date of the last history.log entry that upgraded something.
func aptLastUpgrade() int64 {
f, err := os.Open("/var/log/apt/history.log")
if err != nil {
return 0
}
defer f.Close()
var last, start string
sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 0, 64<<10), 8<<20)
for sc.Scan() {
l := sc.Text()
switch {
case strings.HasPrefix(l, "Start-Date: "):
start = strings.TrimPrefix(l, "Start-Date: ")
case strings.HasPrefix(l, "Upgrade: "):
last = start
}
}
if last == "" {
return 0
}
t, err := time.ParseInLocation("2006-01-02 15:04:05", last, time.Local)
if err != nil {
return 0
}
return t.UnixMilli()
}
func aptAutoUpgrades(ctx context.Context) bool {
text, _ := out(ctx, "apt-config", "dump", "APT::Periodic::Unattended-Upgrade")
return strings.Contains(text, `"1"`)
}
// ---- dnf / zypper / pacman / apk ----
func dnfUpgradable(ctx context.Context) []Package {
text, _ := out(ctx, "dnf", "-q", "check-update") // exit 100 when updates exist; output still parses
var pkgs []Package
for _, line := range strings.Split(text, "\n") {
f := strings.Fields(line)
if len(f) != 3 || !strings.Contains(f[0], ".") {
continue
}
name := f[0][:strings.LastIndex(f[0], ".")]
pkgs = append(pkgs, Package{Name: name, Candidate: f[1], Suite: f[2], Security: strings.Contains(f[2], "security")})
}
if pkgs == nil {
return []Package{}
}
// check-update only names the candidate; one rpm query fills in what's installed.
args := append([]string{"rpm", "-q", "--qf", `%{NAME} %|EPOCH?{%{EPOCH}:}:{}|%{VERSION}-%{RELEASE}\n`}, names(pkgs)...)
if text, _ := out(ctx, args...); text != "" { // exits non-zero when any name isn't installed (new dependencies)
cur := map[string]string{}
for _, line := range strings.Split(text, "\n") {
if f := strings.Fields(line); len(f) == 2 {
cur[f[0]] = f[1] // several installed (kernels): the last listed is the newest
}
}
for i := range pkgs {
pkgs[i].Current = cur[pkgs[i].Name]
}
}
return pkgs
}
func pacmanUpgradable(ctx context.Context) []Package {
text, _ := out(ctx, "pacman", "-Qu")
var pkgs []Package
for _, line := range strings.Split(text, "\n") {
f := strings.Fields(line) // name old -> new
if len(f) == 4 && f[2] == "->" {
pkgs = append(pkgs, Package{Name: f[0], Current: f[1], Candidate: f[3]})
}
}
if pkgs == nil {
return []Package{}
}
return pkgs
}
// zypperUpgradable parses the table `zypper list-updates` prints:
// S | Repository | Name | Current Version | Available Version | Arch
func zypperUpgradable(ctx context.Context) []Package {
text, _ := out(ctx, "zypper", "--non-interactive", "--quiet", "list-updates")
pkgs := []Package{}
for _, line := range strings.Split(text, "\n") {
f := strings.Split(line, "|")
if len(f) < 5 || strings.TrimSpace(f[0]) != "v" {
continue
}
repo := strings.TrimSpace(f[1])
pkgs = append(pkgs, Package{Name: strings.TrimSpace(f[2]), Current: strings.TrimSpace(f[3]), Candidate: strings.TrimSpace(f[4]),
Suite: repo, Security: strings.Contains(strings.ToLower(repo), "security")})
}
return pkgs
}
var apkVerRe = regexp.MustCompile(`^(.+)-([^-]+-r\d+)\s+<\s+(\S+)`)
// apkUpgradable parses `apk version -l '<'`: "musl-1.2.4-r2 < 1.2.5-r0".
func apkUpgradable(ctx context.Context) []Package {
text, _ := out(ctx, "apk", "version", "-l", "<")
pkgs := []Package{}
for _, line := range strings.Split(text, "\n") {
if m := apkVerRe.FindStringSubmatch(strings.TrimSpace(line)); m != nil {
pkgs = append(pkgs, Package{Name: m[1], Current: m[2], Candidate: m[3]})
}
}
return pkgs
}
// pacmanLastUpgrade: the timestamp of the last "upgraded" line in pacman.log.
func pacmanLastUpgrade() int64 {
f, err := os.Open("/var/log/pacman.log")
if err != nil {
return 0
}
defer f.Close()
last := ""
sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 0, 64<<10), 1<<20)
for sc.Scan() {
if l := sc.Text(); strings.Contains(l, "[ALPM] upgraded ") && strings.HasPrefix(l, "[") {
if end := strings.IndexByte(l, ']'); end > 1 {
last = l[1:end]
}
}
}
for _, layout := range []string{"2006-01-02T15:04:05-0700", "2006-01-02 15:04"} {
if t, err := time.ParseInLocation(layout, last, time.Local); err == nil {
return t.UnixMilli()
}
}
return 0
}
// ---- actions ----
// Never starts with '-', so a name can't be read as an option by any of the managers.
var pkgNameRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9+._-]{0,127}$`)
func checkNames(pkgs []string) error {
for _, p := range pkgs {
if !pkgNameRe.MatchString(p) {
return errors.New("bad package name: " + p)
}
}
return nil
}
func (m *Manager) start(args ...string) (*tasks.Task, error) {
if m.pm == "" {
return nil, errors.New("no supported package manager on this host")
}
if !m.sudo {
return nil, errors.New("needs root: run Ham Lab as root, or give its user passwordless sudo or doas")
}
if t, ok := m.tasks.RunningOf("updates"); ok {
return nil, errors.New("another package task is running: " + t.Title)
}
full := priv.Wrap(append([]string{"env", "DEBIAN_FRONTEND=noninteractive", "APT_LISTCHANGES_FRONTEND=none", "LC_ALL=C"}, args...)...)
cmd := exec.Command(full[0], full[1:]...)
return m.tasks.Start("updates", strings.Join(args, " "), cmd)
}
func (m *Manager) Refresh() (*tasks.Task, error) {
switch m.pm {
case "apt":
return m.start("apt-get", "update")
case "dnf":
return m.start("dnf", "-y", "makecache", "--refresh")
case "zypper":
return m.start("zypper", "--non-interactive", "refresh")
case "pacman":
return m.start("pacman", "-Sy", "--noconfirm")
case "apk":
return m.start("apk", "update")
}
return m.start()
}
// rolling reports a distro whose supported upgrade path is a full distribution upgrade.
func rolling() bool {
b, _ := os.ReadFile("/etc/os-release")
return strings.Contains(string(b), "tumbleweed") || strings.Contains(string(b), "microos")
}
// Upgrade upgrades everything, or only the named packages. Nothing here can block on a
// question: apt keeps the current config file on a conflict, the rest run non-interactive.
func (m *Manager) Upgrade(pkgs []string) (*tasks.Task, error) {
if err := checkNames(pkgs); err != nil {
return nil, err
}
all := len(pkgs) == 0
var args []string
switch m.pm {
case "apt":
args = []string{"apt-get", "-y", "-o", "Dpkg::Options::=--force-confdef", "-o", "Dpkg::Options::=--force-confold"}
if all {
args = append(args, "dist-upgrade")
} else {
args = append(args, "install", "--only-upgrade")
}
case "dnf":
args = []string{"dnf", "-y", "upgrade"}
case "zypper":
args = []string{"zypper", "--non-interactive", "update"}
if all && rolling() {
args = []string{"zypper", "--non-interactive", "dist-upgrade"}
}
case "pacman":
if !all {
return nil, errors.New("pacman upgrades everything or nothing: Arch doesn't support partial upgrades")
}
args = []string{"pacman", "-Syu", "--noconfirm"}
case "apk":
args = []string{"apk", "upgrade"}
if all {
args = []string{"apk", "upgrade", "--update-cache"}
}
}
return m.start(append(args, pkgs...)...)
}
// Install installs packages (used by other tabs for optional tooling like smartmontools).
func (m *Manager) Install(pkgs []string) (*tasks.Task, error) {
if len(pkgs) == 0 {
return nil, errors.New("no packages")
}
if err := checkNames(pkgs); err != nil {
return nil, err
}
var args []string
switch m.pm {
case "apt":
args = []string{"apt-get", "-y", "--no-install-recommends", "install"}
case "dnf":
args = []string{"dnf", "-y", "install"}
case "zypper":
args = []string{"zypper", "--non-interactive", "install", "--no-recommends"}
case "pacman":
args = []string{"pacman", "-S", "--noconfirm", "--needed"}
case "apk":
args = []string{"apk", "add"}
}
return m.start(append(args, pkgs...)...)
}
func (m *Manager) Changelog(ctx context.Context, pkg string) (string, error) {
if !pkgNameRe.MatchString(pkg) {
return "", errors.New("bad package name")
}
switch m.pm {
case "apt":
text, err := out(ctx, "apt-get", "changelog", "-qq", pkg)
if len(text) > 64<<10 {
text = text[:64<<10] + "\n…"
}
return text, err
case "dnf":
return out(ctx, "dnf", "-q", "changelog", pkg)
}
return "", errors.New("changelogs not available for " + m.pm)
}
// Power runs systemctl reboot / poweroff.
func (m *Manager) Power(ctx context.Context, action string) error {
if action != "reboot" && action != "poweroff" {
return errors.New("unknown power action")
}
_, err := out(ctx, m.sudoArgs("systemctl", action)...)
return err
}
internal/vpn/edit.go
+575
View File
@@ -0,0 +1,575 @@
package vpn
// Adding, renaming and removing the devices that dial in.
//
// Only OpenWrt's UCI is written, because that is where the server here is configured and
// because a half-supported second path is worse than an honest refusal. A wg-quick host is
// reported as unmanaged and the buttons do not appear.
//
// Two things happen for every change, in the order that fails safest:
//
// - Adding: the configuration is written first and committed, then the peer is applied to
// the running tunnel. If applying fails the configuration is rolled back, so the two never
// disagree silently — a peer that works only after the next reboot is the worst outcome
// of the three.
// - Removing: the running tunnel first, because that is the moment access actually ends,
// then the configuration. If the second half fails the caller is told plainly that the
// device is off now but would return on a reboot.
//
// Nothing here restarts the network. `wg set` changes one peer on a live interface and
// touches nothing else, which matters rather a lot when the interface belongs to the router
// the house is behind.
import (
"context"
"errors"
"fmt"
"net"
"net/netip"
"os"
"os/exec"
"regexp"
"sort"
"strconv"
"strings"
)
// NameRe is what a device may be called: a short slug, because it also names the config
// section it is stored in.
var NameRe = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9 ._-]{0,30}$`)
// sectionRe guards the identifier that goes into a uci path.
var sectionRe = regexp.MustCompile(`^[a-zA-Z0-9_]{1,48}$`)
const sectionPrefix = "wgpeer_"
var (
ErrUnmanaged = errors.New("this tunnel is not configured through UCI, so Ham Lab will not edit it here")
ErrNoRoom = errors.New("no free address is left in this tunnel's range")
)
// NewPeer is what the caller asks for.
type NewPeer struct {
Iface string `json:"-"`
Name string `json:"name"`
// Address is the tunnel address to give the device, with or without a prefix. Empty means
// the next free one.
Address string `json:"address"`
// Endpoint is what the device will dial: the house's name or address, and the port.
Endpoint string `json:"endpoint"`
// DNS is what the device should use while connected. Empty leaves the line out.
DNS string `json:"dns"`
// Split routes only the home networks through the tunnel instead of everything.
Split bool `json:"split"`
}
// Created is a new device, and the one chance to read its configuration.
type Created struct {
Section string `json:"section"`
Name string `json:"name"`
PublicKey string `json:"public_key"`
Address string `json:"address"`
// Config is the complete client profile. It holds the private key, which exists nowhere
// else: it is generated here, handed over once and never stored.
Config string `json:"config"`
}
// uciPeer is one peer section as UCI holds it.
type uciPeer struct {
Section string
Iface string // the wireguard_<iface> type it belongs to
Desc string
PublicKey string
AllowedIPs []string
}
// uciPeers reads every peer section. The type of a peer section names its interface
// ("wireguard_wg0"), which is how a peer is tied to a tunnel.
func uciPeers(ctx context.Context) ([]uciPeer, error) {
if _, err := exec.LookPath("uci"); err != nil {
return nil, ErrUnmanaged
}
b, err := exec.CommandContext(ctx, "uci", "-q", "show", "network").Output()
if err != nil {
return nil, err
}
by := map[string]*uciPeer{}
var order []string
for _, line := range strings.Split(string(b), "\n") {
lhs, rhs, ok := strings.Cut(line, "=")
if !ok || !strings.HasPrefix(lhs, "network.") {
continue
}
parts := strings.Split(strings.TrimPrefix(lhs, "network."), ".")
v := strings.Trim(rhs, "'")
section := parts[0]
if len(parts) == 1 { // the section's own type
if !strings.HasPrefix(v, "wireguard_") {
continue
}
by[section] = &uciPeer{Section: section, Iface: strings.TrimPrefix(v, "wireguard_")}
order = append(order, section)
continue
}
p := by[section]
if p == nil {
continue
}
switch parts[1] {
case "description":
p.Desc = v
case "public_key":
p.PublicKey = v
case "allowed_ips":
p.AllowedIPs = strings.Fields(strings.ReplaceAll(v, ",", " "))
}
}
out := make([]uciPeer, 0, len(order))
for _, s := range order {
out = append(out, *by[s])
}
return out, nil
}
// Managed reports whether this tunnel's peers can be edited here, and labels each peer with
// the section that holds it so the caller has a safe identifier to act on.
func Managed(ctx context.Context, t *Tunnel) bool {
peers, err := uciPeers(ctx)
if err != nil {
return false
}
found := false
byKey := map[string]string{}
for _, p := range peers {
if p.Iface == t.Name {
byKey[p.PublicKey] = p.Section
found = true
}
}
for i := range t.Peers {
t.Peers[i].Section = byKey[t.Peers[i].PublicKey]
}
// A tunnel with no peers at all is still manageable if UCI is the thing configuring it.
if !found {
for _, line := range uciInterfaces(ctx) {
if line == t.Name {
return true
}
}
}
return found
}
// uciInterfaces is every interface UCI configures as wireguard.
func uciInterfaces(ctx context.Context) []string {
var out []string
if _, err := exec.LookPath("uci"); err != nil {
return out
}
b, err := exec.CommandContext(ctx, "uci", "-q", "show", "network").Output()
if err != nil {
return out
}
for _, line := range strings.Split(string(b), "\n") {
lhs, rhs, ok := strings.Cut(line, "=")
if !ok || strings.Trim(rhs, "'") != "wireguard" {
continue
}
parts := strings.Split(strings.TrimPrefix(lhs, "network."), ".")
if len(parts) == 2 && parts[1] == "proto" {
out = append(out, parts[0])
}
}
return out
}
// NextAddress is the lowest address in the tunnel's range that nothing is using. The
// interface's own address is taken, as is every address any peer is allowed.
func NextAddress(t Tunnel) (string, error) {
var base netip.Prefix
for _, a := range t.Addrs {
p, err := netip.ParsePrefix(a)
if err != nil || !p.Addr().Is4() {
continue
}
base = p
break
}
if !base.IsValid() {
return "", errors.New("this tunnel has no IPv4 range to allocate from")
}
taken := map[netip.Addr]bool{base.Addr(): true}
for _, p := range t.Peers {
for _, a := range p.AllowedIPs {
if ip, err := netip.ParsePrefix(strings.TrimSpace(a)); err == nil {
taken[ip.Addr()] = true
} else if ip, err := netip.ParseAddr(strings.TrimSpace(a)); err == nil {
taken[ip] = true
}
}
}
ip := base.Masked().Addr().Next() // .0 is the network
for base.Contains(ip) {
next := ip.Next()
if !base.Contains(next) { // the last address in the range is the broadcast
break
}
if !taken[ip] {
return ip.String(), nil
}
ip = next
}
return "", ErrNoRoom
}
// slug turns a device name into a section name, which may hold rather less.
func slug(name string) string {
var b strings.Builder
for _, r := range strings.ToLower(name) {
switch {
case r >= 'a' && r <= 'z', r >= '0' && r <= '9':
b.WriteRune(r)
case r == ' ', r == '-', r == '.', r == '_':
b.WriteByte('_')
}
}
s := strings.Trim(b.String(), "_")
if s == "" {
s = "peer"
}
if len(s) > 24 {
s = s[:24]
}
return s
}
// genKeys makes a private key, its public half and a pre-shared key. The private key is
// returned to be handed over once and is never written anywhere by this daemon.
func genKeys(ctx context.Context) (priv, pub, psk string, err error) {
run := func(name string, stdin string, args ...string) (string, error) {
cmd := exec.CommandContext(ctx, name, args...)
if stdin != "" {
cmd.Stdin = strings.NewReader(stdin)
}
out, err := cmd.Output()
return strings.TrimSpace(string(out)), err
}
if priv, err = run("wg", "", "genkey"); err != nil {
return "", "", "", fmt.Errorf("generating a key: %w", err)
}
if pub, err = run("wg", priv+"\n", "pubkey"); err != nil {
return "", "", "", fmt.Errorf("deriving the public key: %w", err)
}
if psk, err = run("wg", "", "genpsk"); err != nil {
return "", "", "", fmt.Errorf("generating a pre-shared key: %w", err)
}
return priv, pub, psk, nil
}
// AddPeer creates a device: it writes the peer into the configuration, applies it to the
// running tunnel, and returns the client profile once.
func AddPeer(ctx context.Context, t Tunnel, n NewPeer) (*Created, error) {
if !NameRe.MatchString(strings.TrimSpace(n.Name)) {
return nil, errors.New("a device name is 1–31 letters, digits, spaces, dots, dashes or underscores")
}
name := strings.TrimSpace(n.Name)
if strings.TrimSpace(n.Endpoint) == "" {
return nil, errors.New("the device needs an endpoint to dial: the house's address or name, and the port")
}
endpoint, err := checkEndpoint(n.Endpoint)
if err != nil {
return nil, err
}
addr := strings.TrimSpace(n.Address)
if addr == "" {
if addr, err = NextAddress(t); err != nil {
return nil, err
}
}
host, err := hostOf(addr)
if err != nil {
return nil, err
}
for _, p := range t.Peers {
for _, a := range p.AllowedIPs {
if strings.HasPrefix(strings.TrimSpace(a), host+"/") {
return nil, fmt.Errorf("%s is already used by %s", host, firstNonEmpty(p.Name, "another device"))
}
}
}
section := sectionPrefix + slug(name)
existing, err := uciPeers(ctx)
if err != nil {
return nil, ErrUnmanaged
}
taken := map[string]bool{}
for _, p := range existing {
taken[p.Section] = true
}
for i := 2; taken[section]; i++ {
section = fmt.Sprintf("%s%s%d", sectionPrefix, slug(name), i)
}
if !sectionRe.MatchString(section) {
return nil, errors.New("that name cannot be turned into a configuration section")
}
priv, pub, psk, err := genKeys(ctx)
if err != nil {
return nil, err
}
allowed := host + "/32"
// the configuration first, so a peer can never be live but unrecorded
set := [][]string{
{"set", "network." + section + "=wireguard_" + t.Name},
{"set", "network." + section + ".description=" + name},
{"set", "network." + section + ".public_key=" + pub},
{"set", "network." + section + ".preshared_key=" + psk},
{"set", "network." + section + ".allowed_ips=" + allowed},
}
for _, args := range set {
if err := uci(ctx, args...); err != nil {
_ = uci(ctx, "revert", "network")
return nil, fmt.Errorf("writing the configuration: %w", err)
}
}
if err := uci(ctx, "commit", "network"); err != nil {
_ = uci(ctx, "revert", "network")
return nil, fmt.Errorf("saving the configuration: %w", err)
}
// then the running tunnel; if this fails the configuration goes back as it was
if err := wgSet(ctx, t.Name, pub, psk, allowed); err != nil {
_ = uci(ctx, "delete", "network."+section)
_ = uci(ctx, "commit", "network")
return nil, fmt.Errorf("applying it to the running tunnel: %w", err)
}
return &Created{
Section: section, Name: name, PublicKey: pub, Address: host,
Config: profile(priv, psk, t, host, endpoint, n.DNS, n.Split),
}, nil
}
// EditPeer changes a device's label or the address it is allowed. The section keeps the name
// it was created with; the description is what is shown.
func EditPeer(ctx context.Context, t Tunnel, section, name, address string) error {
if !sectionRe.MatchString(section) {
return errors.New("that is not a device this can edit")
}
peers, err := uciPeers(ctx)
if err != nil {
return ErrUnmanaged
}
var me *uciPeer
for i := range peers {
if peers[i].Section == section && peers[i].Iface == t.Name {
me = &peers[i]
}
}
if me == nil {
return errors.New("no such device on this tunnel")
}
if name = strings.TrimSpace(name); name != "" {
if !NameRe.MatchString(name) {
return errors.New("a device name is 1–31 letters, digits, spaces, dots, dashes or underscores")
}
if err := uci(ctx, "set", "network."+section+".description="+name); err != nil {
_ = uci(ctx, "revert", "network")
return err
}
}
allowed := ""
if address = strings.TrimSpace(address); address != "" {
host, err := hostOf(address)
if err != nil {
_ = uci(ctx, "revert", "network")
return err
}
for _, p := range t.Peers {
if p.Section == section {
continue
}
for _, a := range p.AllowedIPs {
if strings.HasPrefix(strings.TrimSpace(a), host+"/") {
_ = uci(ctx, "revert", "network")
return fmt.Errorf("%s is already used by %s", host, firstNonEmpty(p.Name, "another device"))
}
}
}
allowed = host + "/32"
if err := uci(ctx, "set", "network."+section+".allowed_ips="+allowed); err != nil {
_ = uci(ctx, "revert", "network")
return err
}
}
if err := uci(ctx, "commit", "network"); err != nil {
_ = uci(ctx, "revert", "network")
return err
}
if allowed != "" { // the running tunnel has to be told, or the device keeps the old address
if err := wgSet(ctx, t.Name, me.PublicKey, "", allowed); err != nil {
return fmt.Errorf("saved, but the running tunnel refused the new address: %w", err)
}
}
return nil
}
// DeletePeer removes a device. The running tunnel goes first, because that is the moment the
// device actually loses its way in.
func DeletePeer(ctx context.Context, t Tunnel, section string) error {
if !sectionRe.MatchString(section) {
return errors.New("that is not a device this can remove")
}
peers, err := uciPeers(ctx)
if err != nil {
return ErrUnmanaged
}
var me *uciPeer
for i := range peers {
if peers[i].Section == section && peers[i].Iface == t.Name {
me = &peers[i]
}
}
if me == nil {
return errors.New("no such device on this tunnel")
}
if err := wgRemove(ctx, t.Name, me.PublicKey); err != nil {
return fmt.Errorf("removing it from the running tunnel: %w", err)
}
if err := uci(ctx, "delete", "network."+section); err != nil {
return fmt.Errorf("the device is disconnected, but removing it from the configuration failed, so it would come back on a reboot: %w", err)
}
if err := uci(ctx, "commit", "network"); err != nil {
return fmt.Errorf("the device is disconnected, but saving the configuration failed, so it would come back on a reboot: %w", err)
}
return nil
}
// ---- the small dangerous bits, each in one place ----
func uci(ctx context.Context, args ...string) error {
out, err := exec.CommandContext(ctx, "uci", args...).CombinedOutput()
if err != nil {
return fmt.Errorf("uci %s: %s", strings.Join(args, " "), strings.TrimSpace(string(out)))
}
return nil
}
// wgSet applies a peer to the running interface. The pre-shared key goes in through a file
// because that is the only way wg takes one, so the file is made readable by nobody else and
// removed immediately.
func wgSet(ctx context.Context, iface, pub, psk, allowed string) error {
args := []string{"set", iface, "peer", pub}
if psk != "" {
f, err := os.CreateTemp("", "hamlab-psk")
if err != nil {
return err
}
defer os.Remove(f.Name())
if err := f.Chmod(0o600); err != nil {
f.Close()
return err
}
if _, err := f.WriteString(psk + "\n"); err != nil {
f.Close()
return err
}
f.Close()
args = append(args, "preshared-key", f.Name())
}
args = append(args, "allowed-ips", allowed)
out, err := exec.CommandContext(ctx, "wg", args...).CombinedOutput()
if err != nil {
return fmt.Errorf("wg set: %s", strings.TrimSpace(string(out)))
}
return nil
}
func wgRemove(ctx context.Context, iface, pub string) error {
out, err := exec.CommandContext(ctx, "wg", "set", iface, "peer", pub, "remove").CombinedOutput()
if err != nil {
return fmt.Errorf("wg set: %s", strings.TrimSpace(string(out)))
}
return nil
}
// profile is the client's own configuration file, which is the only place its private key
// ever appears.
func profile(priv, psk string, t Tunnel, addr, endpoint, dns string, split bool) string {
prefix := 24
for _, a := range t.Addrs {
if p, err := netip.ParsePrefix(a); err == nil && p.Addr().Is4() {
prefix = p.Bits()
break
}
}
allowed := "0.0.0.0/0"
if split {
nets := []string{}
for _, a := range t.Addrs {
if p, err := netip.ParsePrefix(a); err == nil && p.Addr().Is4() {
nets = append(nets, p.Masked().String())
}
}
nets = append(nets, "10.0.0.0/8", "192.168.0.0/16", "172.16.0.0/12")
allowed = strings.Join(dedupe(nets), ", ")
}
var b strings.Builder
fmt.Fprintf(&b, "[Interface]\nPrivateKey = %s\nAddress = %s/%d\n", priv, addr, prefix)
if dns = strings.TrimSpace(dns); dns != "" {
fmt.Fprintf(&b, "DNS = %s\n", dns)
}
fmt.Fprintf(&b, "\n[Peer]\nPublicKey = %s\n", t.PublicKey)
if psk != "" {
fmt.Fprintf(&b, "PresharedKey = %s\n", psk)
}
fmt.Fprintf(&b, "Endpoint = %s\nAllowedIPs = %s\n", endpoint, allowed)
return b.String()
}
func dedupe(in []string) []string {
seen := map[string]bool{}
out := in[:0:0]
for _, s := range in {
if !seen[s] {
seen[s] = true
out = append(out, s)
}
}
sort.Strings(out)
return out
}
// hostOf takes an address with or without a prefix and returns the bare address.
func hostOf(s string) (string, error) {
s = strings.TrimSpace(s)
if p, err := netip.ParsePrefix(s); err == nil {
return p.Addr().String(), nil
}
a, err := netip.ParseAddr(s)
if err != nil {
return "", fmt.Errorf("%q is not an address", s)
}
return a.String(), nil
}
// checkEndpoint makes sure what the device will dial looks like a host and a port.
func checkEndpoint(s string) (string, error) {
s = strings.TrimSpace(s)
host, port, err := net.SplitHostPort(s)
if err != nil || host == "" {
return "", fmt.Errorf("%q should be a name or address and a port, like home.example.com:51820", s)
}
if n, err := strconv.Atoi(port); err != nil || n < 1 || n > 65535 {
return "", fmt.Errorf("%q is not a port", port)
}
return s, nil
}
func firstNonEmpty(a, b string) string {
if a != "" {
return a
}
return b
}
internal/vpn/edit_test.go
+126
View File
@@ -0,0 +1,126 @@
package vpn
import (
"strings"
"testing"
)
func tunnel(addrs []string, peers ...string) Tunnel {
t := Tunnel{Name: "wg0", PublicKey: "SERVERKEY=", Addrs: addrs}
for _, p := range peers {
t.Peers = append(t.Peers, Peer{PublicKey: p + "KEY=", AllowedIPs: []string{p}})
}
return t
}
func TestNextAddress(t *testing.T) {
// .1 is the interface itself and .2 and .3 are taken, so the next free one is .4
got, err := NextAddress(tunnel([]string{"172.21.46.1/24"}, "172.21.46.3/32", "172.21.46.2/32"))
if err != nil || got != "172.21.46.4" {
t.Errorf("NextAddress = %q, %v; want 172.21.46.4", got, err)
}
// a gap is filled before the end is reached
got, _ = NextAddress(tunnel([]string{"172.21.46.1/24"}, "172.21.46.2/32", "172.21.46.4/32"))
if got != "172.21.46.3" {
t.Errorf("NextAddress = %q, want the gap at .3", got)
}
// an empty tunnel starts after the server
got, _ = NextAddress(tunnel([]string{"10.9.0.1/24"}))
if got != "10.9.0.2" {
t.Errorf("NextAddress = %q, want 10.9.0.2", got)
}
// a full one says so rather than handing out a duplicate
full := tunnel([]string{"10.9.0.1/30"}, "10.9.0.2/32")
if _, err := NextAddress(full); err != ErrNoRoom {
t.Errorf("err = %v, want ErrNoRoom on a full range", err)
}
// IPv6-only, or nothing at all, is refused rather than guessed
if _, err := NextAddress(tunnel([]string{"fc00::1/64"})); err == nil {
t.Error("want an error when there is no IPv4 range")
}
}
func TestSlug(t *testing.T) {
for in, want := range map[string]string{
"ip12": "ip12",
"Mitchell's iPad": "mitchells_ipad",
"my-phone": "my_phone",
" spaced out ": "spaced__out",
"!!!": "peer",
strings.Repeat("a", 40): strings.Repeat("a", 24),
} {
if got := slug(in); got != want {
t.Errorf("slug(%q) = %q, want %q", in, got, want)
}
}
}
func TestProfile(t *testing.T) {
tun := tunnel([]string{"172.21.46.1/24"})
got := profile("PRIV=", "PSK=", tun, "172.21.46.4", "home.example.com:51820", "1.1.1.1, 1.0.0.1", false)
for _, want := range []string{
"PrivateKey = PRIV=",
"Address = 172.21.46.4/24", // the interface's own prefix, not a bare /32
"DNS = 1.1.1.1, 1.0.0.1",
"PublicKey = SERVERKEY=",
"PresharedKey = PSK=",
"Endpoint = home.example.com:51820",
"AllowedIPs = 0.0.0.0/0",
} {
if !strings.Contains(got, want) {
t.Errorf("profile is missing %q:\n%s", want, got)
}
}
// no DNS line at all when none was asked for, rather than an empty one
if strings.Contains(profile("P=", "", tun, "172.21.46.4", "h:1", "", false), "DNS") {
t.Error("an empty DNS should leave the line out")
}
// split routes the private ranges instead of everything
split := profile("P=", "", tun, "172.21.46.4", "h:1", "", true)
if strings.Contains(split, "0.0.0.0/0") {
t.Errorf("a split tunnel must not route everything:\n%s", split)
}
if !strings.Contains(split, "172.21.46.0/24") {
t.Errorf("a split tunnel should include the tunnel's own range:\n%s", split)
}
}
func TestCheckEndpoint(t *testing.T) {
for _, ok := range []string{"bigham.dev:51820", "1.2.3.4:51820", "[2001:db8::1]:51820"} {
if _, err := checkEndpoint(ok); err != nil {
t.Errorf("checkEndpoint(%q) = %v, want it accepted", ok, err)
}
}
for _, bad := range []string{"", "bigham.dev", ":51820", "bigham.dev:0", "bigham.dev:notaport", "bigham.dev:99999"} {
if _, err := checkEndpoint(bad); err == nil {
t.Errorf("checkEndpoint(%q) was accepted, want refused", bad)
}
}
}
func TestHostOf(t *testing.T) {
for in, want := range map[string]string{"172.21.46.4": "172.21.46.4", "172.21.46.4/32": "172.21.46.4", " 10.0.0.9/24 ": "10.0.0.9"} {
got, err := hostOf(in)
if err != nil || got != want {
t.Errorf("hostOf(%q) = %q, %v; want %q", in, got, err, want)
}
}
if _, err := hostOf("not-an-address"); err == nil {
t.Error("hostOf accepted nonsense")
}
}
// The name is what becomes a configuration section, so it is the input worth being strict
// about; everything refused here is refused before anything is written.
func TestNameRe(t *testing.T) {
for _, ok := range []string{"ip12", "My Phone", "a", "work-laptop.2"} {
if !NameRe.MatchString(ok) {
t.Errorf("%q should be a valid name", ok)
}
}
for _, bad := range []string{"", " leading", "semi;colon", "quote'd", "back`tick", "new\nline", "$(whoami)", strings.Repeat("a", 32)} {
if NameRe.MatchString(bad) {
t.Errorf("%q should be refused as a name", bad)
}
}
}
internal/vpn/vpn.go
+372
View File
@@ -0,0 +1,372 @@
// Package vpn reads this machine's WireGuard tunnels: the interfaces, and the peers on them
// with names put back on.
//
// The live state comes from `wg show all dump`, which is the kernel's own answer and the only
// place handshakes and byte counts exist. The kernel knows nothing about names, though — a
// peer there is a public key — so the configuration is read alongside it purely to label
// them: `# comments` in a wg-quick .conf, or the `description` of a UCI peer section on
// OpenWrt. Everything here is read-only, and no private or pre-shared key is ever read, let
// alone returned.
package vpn
import (
"bufio"
"context"
"net"
"os"
"os/exec"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
"hamlab/internal/priv"
)
// Peer is one end of a tunnel: a device that dials in, or the server this machine dials out
// to. Which it is depends on the tunnel; see Tunnel.Kind.
type Peer struct {
// Name is what the configuration calls it, where it says at all. The kernel has no idea.
Name string `json:"name,omitempty"`
PublicKey string `json:"public_key"`
// Endpoint is where its packets come from, empty until it has been heard from.
Endpoint string `json:"endpoint,omitempty"`
AllowedIPs []string `json:"allowed_ips"`
// Handshake is the last completed handshake, absent when there has never been one — a peer
// that is configured but has not connected since this tunnel came up. A pointer because
// omitempty does nothing for a zero time.Time, and a peer that has never been heard from
// would otherwise be reported as having last spoken in the year one.
Handshake *time.Time `json:"handshake,omitempty"`
// Online is a handshake inside the last three minutes. WireGuard is silent when idle, so
// this means "was talking recently", not "is connected" — there is no such thing here.
Online bool `json:"online"`
Rx int64 `json:"rx"`
Tx int64 `json:"tx"`
Keepalive int `json:"keepalive,omitempty"`
// PSK notes that a pre-shared key is in use, without the key itself going anywhere.
PSK bool `json:"psk,omitempty"`
// Section is the configuration section holding this peer, and the identifier anything that
// edits it uses. Empty when the peer is not in a configuration this can edit.
Section string `json:"section,omitempty"`
}
// Tunnel is one WireGuard interface.
type Tunnel struct {
Name string `json:"name"`
// Managed is whether devices on this tunnel can be added and removed from here, which
// today means its peers are configured through OpenWrt's UCI.
Managed bool `json:"managed"`
PublicKey string `json:"public_key"`
// Kind is "server" when peers dial in to this machine, "client" when this machine dials
// out through a peer, and "" when it is too new or too odd to tell. It is worked out from
// the shape of the peers, not from anything WireGuard states: a peer that is routed
// everything and has an endpoint of its own is an exit, and a tunnel with one of those is
// this machine going out through it.
Kind string `json:"kind"`
ListenPort int `json:"listen_port,omitempty"`
FwMark string `json:"fwmark,omitempty"`
Addrs []string `json:"addrs"`
MTU int `json:"mtu,omitempty"`
Peers []Peer `json:"peers"`
// Exit is the peer this machine goes out through, for a client tunnel.
Exit string `json:"exit,omitempty"`
}
// Report is every tunnel on this machine, and why there are none when there are none.
type Report struct {
At time.Time `json:"at"`
Tunnels []Tunnel `json:"tunnels"`
// Tool is false when wireguard-tools is not installed, Root false when the daemon cannot
// reach root — either one means the peers cannot be read, and the tab should say which.
Tool bool `json:"tool"`
Root bool `json:"root"`
Note string `json:"note,omitempty"`
}
const onlineWithin = 3 * time.Minute
// Look reads the tunnels on this machine.
func Look(ctx context.Context) Report {
r := Report{At: time.Now(), Tunnels: []Tunnel{}, Root: priv.Available()}
if _, err := exec.LookPath("wg"); err != nil {
r.Note = "wireguard-tools is not installed here, so there is nothing to read"
return r
}
r.Tool = true
args := []string{"wg", "show", "all", "dump"}
if priv.Available() {
args = priv.Wrap(args...)
}
cmd := exec.CommandContext(ctx, args[0], args[1:]...)
out, err := cmd.Output()
if err != nil {
if !r.Root {
r.Note = "reading WireGuard needs root, and this daemon has no way to it"
} else {
r.Note = "wg show failed: " + strings.TrimSpace(err.Error())
}
return r
}
r.Tunnels = parseDump(string(out))
names := peerNames()
for i := range r.Tunnels {
t := &r.Tunnels[i]
t.Addrs, t.MTU = ifaceAddrs(t.Name)
for j := range t.Peers {
p := &t.Peers[j]
p.Name = names[p.PublicKey]
p.Online = p.Handshake != nil && time.Since(*p.Handshake) < onlineWithin
}
t.Kind, t.Exit = classify(*t)
t.Managed = Managed(ctx, t)
sort.SliceStable(t.Peers, func(a, b int) bool {
pa, pb := t.Peers[a], t.Peers[b]
if pa.Online != pb.Online {
return pa.Online
}
if ha, hb := shookAt(pa), shookAt(pb); !ha.Equal(hb) {
return ha.After(hb)
}
return pa.Name < pb.Name
})
}
if len(r.Tunnels) == 0 {
r.Note = "no WireGuard interfaces are up on this machine"
}
return r
}
// parseDump reads `wg show all dump`. The first line of each interface is the interface
// itself, the rest are its peers; every line is tab-separated and positional.
//
// interface: name private-key public-key listen-port fwmark
// peer: name public-key psk endpoint allowed-ips handshake rx tx keepalive
//
// The private key is in the second column and is read past without being kept.
func parseDump(out string) []Tunnel {
// An empty list, never nil: this is JSON to a browser that will take its length, and a
// null there is a blank page rather than "no tunnels".
tunnels := []Tunnel{}
byName := map[string]int{}
sc := bufio.NewScanner(strings.NewReader(out))
sc.Buffer(make([]byte, 0, 64<<10), 1<<20)
for sc.Scan() {
f := strings.Split(sc.Text(), "\t")
if len(f) < 5 {
continue
}
name := f[0]
if len(f) == 5 { // an interface line
port, _ := strconv.Atoi(f[3])
byName[name] = len(tunnels)
tunnels = append(tunnels, Tunnel{Name: name, PublicKey: f[2], ListenPort: port, FwMark: off(f[4]), Peers: []Peer{}})
continue
}
if len(f) < 9 {
continue
}
i, ok := byName[name]
if !ok {
continue
}
hs, _ := strconv.ParseInt(f[5], 10, 64)
rx, _ := strconv.ParseInt(f[6], 10, 64)
tx, _ := strconv.ParseInt(f[7], 10, 64)
ka, _ := strconv.Atoi(f[8])
p := Peer{PublicKey: f[1], PSK: f[2] != "(none)", Endpoint: off(f[3]), Rx: rx, Tx: tx, Keepalive: ka}
if f[4] != "(none)" {
p.AllowedIPs = strings.Split(f[4], ",")
}
if hs > 0 {
at := time.Unix(hs, 0)
p.Handshake = &at
}
tunnels[i].Peers = append(tunnels[i].Peers, p)
}
return tunnels
}
// shookAt is a peer's last handshake, or the zero time when there has never been one.
func shookAt(p Peer) time.Time {
if p.Handshake == nil {
return time.Time{}
}
return *p.Handshake
}
// off turns wg's placeholders into an empty string.
func off(v string) string {
if v == "(none)" || v == "off" {
return ""
}
return v
}
// classify decides whether this machine is the one being dialled or the one dialling. A peer
// that is allowed every address and has an endpoint is a way out to the internet; a tunnel
// with one of those is this machine using it. Otherwise, if anything can reach us here, it is
// a server.
func classify(t Tunnel) (kind, exit string) {
for _, p := range t.Peers {
for _, a := range p.AllowedIPs {
if a == "0.0.0.0/0" || a == "::/0" {
if p.Endpoint != "" {
name := p.Name
if name == "" {
name = p.Endpoint
}
return "client", name
}
}
}
}
if t.ListenPort > 0 && len(t.Peers) > 0 {
return "server", ""
}
return "", ""
}
// ifaceAddrs is what the interface carries, and its MTU.
func ifaceAddrs(name string) ([]string, int) {
addrs := []string{}
mtu := 0
ifi, err := net.InterfaceByName(name)
if err != nil {
return addrs, mtu
}
mtu = ifi.MTU
got, err := ifi.Addrs()
if err != nil {
return addrs, mtu
}
for _, a := range got {
addrs = append(addrs, a.String())
}
return addrs, mtu
}
// ---- names ----
// peerNames maps a public key to what the configuration calls that peer. Two shapes are
// understood, which between them cover every machine here: wg-quick's .conf files, where the
// name is a comment, and OpenWrt's UCI, where it is the peer section's description.
func peerNames() map[string]string {
out := map[string]string{}
for k, v := range confNames() {
out[k] = v
}
for k, v := range uciNames() { // UCI wins: on OpenWrt it is the live configuration
out[k] = v
}
return out
}
// confNames reads /etc/wireguard/*.conf. In a wg-quick file a peer is named by the comment
// that introduces it, which is the convention every tool that writes these files follows:
//
// # phone
// [Peer]
// PublicKey = …
//
// confDir is where wg-quick keeps its files; a variable so the test can point it elsewhere.
var confDir = "/etc/wireguard"
func confNames() map[string]string {
out := map[string]string{}
files, _ := filepath.Glob(filepath.Join(confDir, "*.conf"))
for _, path := range files {
text, err := readMaybeRoot(path)
if err != nil {
continue
}
sc := bufio.NewScanner(strings.NewReader(text))
// last is the most recent comment line; when a [Peer] block opens, that comment becomes
// its pending name. Keeping the two apart is what lets each block take the comment that
// introduces it and ignore any comment written inside it.
last, pending := "", ""
inPeer := false
for sc.Scan() {
line := strings.TrimSpace(sc.Text())
switch {
case strings.EqualFold(line, "[Peer]"):
inPeer, pending, last = true, last, ""
case strings.HasPrefix(line, "["):
inPeer, pending, last = false, "", ""
case strings.HasPrefix(line, "#"):
if c := strings.TrimSpace(strings.TrimLeft(line, "#")); c != "" {
last = c
}
case inPeer && strings.HasPrefix(line, "PublicKey"):
// The separator is the first '='; the key's own base64 padding is the rest of
// the line, so everything after that first one is the key.
if _, v, ok := strings.Cut(line, "="); ok && pending != "" {
out[strings.TrimSpace(v)] = pending
}
pending = ""
}
}
}
return out
}
// readMaybeRoot reads a file, falling back to root when it is not ours to read — which
// /etc/wireguard always is, since it holds private keys. Only the comments and public keys in
// it are looked at; nothing else in the file is kept or returned.
func readMaybeRoot(path string) (string, error) {
b, err := os.ReadFile(path)
if err == nil {
return string(b), nil
}
if !os.IsPermission(err) || !priv.Available() {
return "", err
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
args := priv.Wrap("cat", path)
out, cerr := exec.CommandContext(ctx, args[0], args[1:]...).Output()
if cerr != nil {
return "", cerr
}
return string(out), nil
}
// uciNames reads OpenWrt's configuration: each peer is a section with a description and a
// public key. `uci show` is used rather than the file, so includes and overrides are applied.
func uciNames() map[string]string {
out := map[string]string{}
if _, err := exec.LookPath("uci"); err != nil {
return out
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
b, err := exec.CommandContext(ctx, "uci", "-q", "show", "network").Output()
if err != nil {
return out
}
desc, key := map[string]string{}, map[string]string{}
for _, line := range strings.Split(string(b), "\n") {
lhs, rhs, ok := strings.Cut(line, "=")
if !ok || !strings.HasPrefix(lhs, "network.") {
continue
}
parts := strings.Split(strings.TrimPrefix(lhs, "network."), ".")
if len(parts) != 2 {
continue
}
section, option := parts[0], parts[1]
v := strings.Trim(rhs, "'")
switch option {
case "description":
desc[section] = v
case "public_key":
key[section] = v
}
}
for section, k := range key {
if d := desc[section]; d != "" {
out[k] = d
}
}
return out
}
internal/vpn/vpn_test.go
+193
View File
@@ -0,0 +1,193 @@
package vpn
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// Real output of `wg show all dump`, from the two machines here: gate, where devices dial in,
// and hamtv, which dials out through Mullvad. Keys are the real public ones, which are public.
const gateDump = "wg0\tPRIVATEKEYWOULDBEHERE=\t4VwqgzHyelmVmNY+EANRWQggSSxvYYguJQMvUty9Sns=\t51820\toff\n" +
"wg0\tNeWxo9wfqgipBQLuP676WFcMlW5RDrOcanc9y8sT7kU=\t3KWsu0mdVb2l5B/j0qHE03yE9LXi3FJ1WAjWMuxaLAw=\t10.0.0.33:63573\t172.21.46.3/32\t1790565744\t134192164\t415977404\toff\n" +
"wg0\twUnRD990c6aglU0TCREtmfuUTPzqSmHrd3qTORpRJBg=\t4Pd5u2t6U9EdHlFlneg+hMr0G2v+IvoE/EqGfFdDPI8=\t(none)\t172.21.46.2/32\t0\t0\t0\toff\n"
const hamtvDump = "wg0\tPRIVATEKEYWOULDBEHERE=\tF/rgcHi+ZiaWPoSQ9iPcldIUbbMotE1hwSB/74Y9Qi4=\t39281\toff\n" +
"wg0\tzqsfGglzJPY657WMRxf/S4omG7+ZkSDIpDq+ggbc9yo=\t(none)\t23.234.72.2:51820\t0.0.0.0/0,::/0\t1790571246\t18206675520\t2687837240\t25\n"
func TestParseServerDump(t *testing.T) {
ts := parseDump(gateDump)
if len(ts) != 1 {
t.Fatalf("got %d tunnels, want 1", len(ts))
}
w := ts[0]
if w.Name != "wg0" || w.ListenPort != 51820 {
t.Errorf("interface = %s:%d, want wg0:51820", w.Name, w.ListenPort)
}
if w.PublicKey != "4VwqgzHyelmVmNY+EANRWQggSSxvYYguJQMvUty9Sns=" {
t.Errorf("public key = %q; the third column is the public one", w.PublicKey)
}
// The private key is column two of the interface line and must not appear anywhere.
if w.PublicKey == "PRIVATEKEYWOULDBEHERE=" || w.FwMark == "PRIVATEKEYWOULDBEHERE=" {
t.Error("the private key was kept")
}
if len(w.Peers) != 2 {
t.Fatalf("got %d peers, want 2", len(w.Peers))
}
p := w.Peers[0]
if p.PublicKey != "NeWxo9wfqgipBQLuP676WFcMlW5RDrOcanc9y8sT7kU=" || p.Endpoint != "10.0.0.33:63573" {
t.Errorf("first peer = %+v", p)
}
if !p.PSK {
t.Error("first peer has a pre-shared key and should say so")
}
if got := p.AllowedIPs; len(got) != 1 || got[0] != "172.21.46.3/32" {
t.Errorf("allowed ips = %v", got)
}
if p.Rx != 134192164 || p.Tx != 415977404 {
t.Errorf("counters = %d/%d", p.Rx, p.Tx)
}
if p.Handshake == nil || p.Handshake.Unix() != 1790565744 {
t.Errorf("handshake = %v", p.Handshake)
}
// The second peer has never connected: no endpoint, no handshake, nothing moved.
if q := w.Peers[1]; q.Endpoint != "" || q.Handshake != nil || q.Rx != 0 {
t.Errorf("second peer = %+v, want one that has never been heard from", q)
}
if kind, exit := classify(w); kind != "server" || exit != "" {
t.Errorf("classify = %q/%q, want server", kind, exit)
}
}
func TestParseClientDump(t *testing.T) {
ts := parseDump(hamtvDump)
if len(ts) != 1 || len(ts[0].Peers) != 1 {
t.Fatalf("got %d tunnels", len(ts))
}
w := ts[0]
p := w.Peers[0]
if len(p.AllowedIPs) != 2 || p.AllowedIPs[0] != "0.0.0.0/0" {
t.Errorf("allowed ips = %v, want everything", p.AllowedIPs)
}
if p.Keepalive != 25 {
t.Errorf("keepalive = %d, want 25", p.Keepalive)
}
if p.PSK {
t.Error("this peer has no pre-shared key")
}
// It routes everything and has an endpoint, so this machine goes out through it.
kind, exit := classify(w)
if kind != "client" || exit != "23.234.72.2:51820" {
t.Errorf("classify = %q/%q, want client through the endpoint", kind, exit)
}
}
// A tunnel that is up but has no peers at all is neither, and must not be guessed at.
func TestClassifyUndecided(t *testing.T) {
if kind, _ := classify(Tunnel{Name: "wg0", ListenPort: 51820}); kind != "" {
t.Errorf("classify with no peers = %q, want no opinion", kind)
}
}
// Peers are named by the comment above them in a wg-quick file. The public key's own base64
// padding is an '=' too, which is the thing to get wrong.
func TestConfNames(t *testing.T) {
dir := t.TempDir()
conf := `[Interface]
Address = 172.21.46.1/24
PrivateKey = SECRETSECRETSECRETSECRETSECRETSECRETSECRET=
ListenPort = 51820
# phone
[Peer]
PublicKey = NeWxo9wfqgipBQLuP676WFcMlW5RDrOcanc9y8sT7kU=
AllowedIPs = 172.21.46.3/32
# the laptop
[Peer]
PublicKey = wUnRD990c6aglU0TCREtmfuUTPzqSmHrd3qTORpRJBg=
AllowedIPs = 172.21.46.2/32
`
if err := os.WriteFile(filepath.Join(dir, "wg0.conf"), []byte(conf), 0o600); err != nil {
t.Fatal(err)
}
old := confDir
confDir = dir
defer func() { confDir = old }()
names := confNames()
if got := names["NeWxo9wfqgipBQLuP676WFcMlW5RDrOcanc9y8sT7kU="]; got != "phone" {
t.Errorf("first peer named %q, want phone", got)
}
if got := names["wUnRD990c6aglU0TCREtmfuUTPzqSmHrd3qTORpRJBg="]; got != "the laptop" {
t.Errorf("second peer named %q, want the laptop", got)
}
for k := range names {
if k == "SECRETSECRETSECRETSECRETSECRETSECRETSECRET=" {
t.Error("the private key was read as if it were a peer")
}
}
if len(names) != 2 {
t.Errorf("%d names, want 2: %v", len(names), names)
}
}
func TestOnlineIsRecentHandshake(t *testing.T) {
for _, tc := range []struct {
name string
hs time.Time
want bool
}{
{"just now", time.Now().Add(-30 * time.Second), true},
{"a while back", time.Now().Add(-10 * time.Minute), false},
{"never", time.Time{}, false},
} {
got := !tc.hs.IsZero() && time.Since(tc.hs) < onlineWithin
if got != tc.want {
t.Errorf("%s: online = %v, want %v", tc.name, got, tc.want)
}
}
}
// A peer that has never handshaken must carry no time at all, not the zero one. omitempty
// does nothing for a time.Time, so this is the difference between the page saying "never
// connected" and saying the device was last seen in the year one.
func TestNeverConnectedHasNoTimestamp(t *testing.T) {
w := parseDump(gateDump)[0]
b, err := json.Marshal(w.Peers[1])
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(b), "handshake") {
t.Errorf("json = %s, want no handshake field at all", b)
}
if strings.Contains(string(b), "0001-01-01") {
t.Errorf("json = %s, want no year-one timestamp", b)
}
// and the one that has connected still carries its time
if b, _ := json.Marshal(w.Peers[0]); !strings.Contains(string(b), "handshake") {
t.Errorf("json = %s, want the handshake of a peer that has one", b)
}
}
// A machine with wireguard-tools installed but no tunnel up produces an empty dump. That has
// to marshal as [] and not as null: the page takes the length of it, and null blanks the
// whole console rather than saying "no tunnels".
func TestEmptyDumpIsAnEmptyList(t *testing.T) {
for _, dump := range []string{"", "\n", " \n\n"} {
got := parseDump(dump)
if got == nil {
t.Fatalf("parseDump(%q) = nil, want an empty list", dump)
}
b, err := json.Marshal(got)
if err != nil {
t.Fatal(err)
}
if string(b) != "[]" {
t.Errorf("parseDump(%q) marshals to %s, want []", dump, b)
}
}
}
scripts/gitea-release.sh
Executable
+69
View File
@@ -0,0 +1,69 @@
#!/bin/sh
# Publish what `make release` built as a release on Gitea, which is where every Ham Lab
# looks for a newer build (Settings → Updates, and `update.url` in config.json).
#
# make release && scripts/gitea-release.sh [--force]
#
# The token needs write:repository and is read from $HAMLAB_PUBLISH_TOKEN, else from
# ~/.config/hamlab-publish-token. It is never printed.
#
# A release is a deliberate act, not part of `make deploy`: deploying pushes this build to
# the machines this hub owns, while publishing offers it to every install everywhere.
set -eu
REPO=${HAMLAB_REPO:-https://git.bigham.dev/api/v1/repos/mitchell/hamlab}
DIST=${DIST:-dist}
FORCE=0
[ "${1:-}" != "--force" ] || FORCE=1
die() { printf '%s\n' "$*" >&2; exit 1; }
[ -f "$DIST/VERSION" ] || die "no $DIST/VERSION — run: make release"
VERSION=$(head -n 1 "$DIST/VERSION" | tr -cd 'A-Za-z0-9._+-')
[ -n "$VERSION" ] || die "$DIST/VERSION is empty"
TOKEN=${HAMLAB_PUBLISH_TOKEN:-}
if [ -z "$TOKEN" ] && [ -r "$HOME/.config/hamlab-publish-token" ]; then
TOKEN=$(cat "$HOME/.config/hamlab-publish-token")
fi
[ -n "$TOKEN" ] || die "no publish token: set HAMLAB_PUBLISH_TOKEN or write ~/.config/hamlab-publish-token (Gitea → Settings → Applications, scope write:repository)"
api() { # method path [curl args...]
m=$1; p=$2; shift 2
curl -sS -X "$m" -H "Authorization: token $TOKEN" "$REPO$p" "$@"
}
SHA=$(git rev-parse HEAD)
echo "publishing $VERSION ($(echo "$SHA" | cut -c1-8)) to $REPO"
# An existing release for this version is replaced only when asked: re-publishing a version
# with different binaries is how an install ends up with something nobody can reproduce.
existing=$(api GET "/releases/tags/$VERSION" | python3 -c 'import json,sys
try: d = json.load(sys.stdin)
except Exception: d = {}
print(d.get("id") or "")' 2>/dev/null || true)
if [ -n "$existing" ]; then
[ "$FORCE" = 1 ] || die "$VERSION is already published (release $existing). Re-publish with --force."
echo " replacing release $existing"
api DELETE "/releases/$existing" -o /dev/null
fi
notes=$(git log -1 --pretty=%B | head -20)
payload=$(python3 -c 'import json,sys; print(json.dumps({"tag_name": sys.argv[1], "target": sys.argv[2], "name": sys.argv[1], "body": sys.argv[3], "draft": False, "prerelease": False}))' "$VERSION" "$SHA" "$notes")
id=$(printf '%s' "$payload" | api POST /releases -H 'Content-Type: application/json' -d @- | python3 -c 'import json,sys
d = json.load(sys.stdin)
if not d.get("id"): sys.exit("gitea refused the release: %s" % d.get("message", d))
print(d["id"])')
[ -n "$id" ] || die "could not create the release"
echo " release $id created"
n=0
for f in "$DIST"/hamlab-linux-* "$DIST/SHA256SUMS" "$DIST/install.sh" "$DIST/VERSION"; do
[ -f "$f" ] || continue
name=$(basename "$f")
api POST "/releases/$id/assets?name=$name" -F "attachment=@$f" -o /dev/null
n=$((n + 1))
echo " + $name"
done
echo "published $VERSION with $n assets"
echo " every Ham Lab pointed at this repo will offer it in Settings → Updates"
web/package-lock.json
Generated
+7
View File
@@ -11,6 +11,7 @@
"@xterm/addon-fit": "^0.10.0",
"@xterm/addon-web-links": "^0.11.0",
"@xterm/xterm": "^5.5.0",
"qrcode-generator": "^2.0.4",
"react": "^18.3.1",
"react-dom": "^18.3.1"
},
@@ -1551,6 +1552,12 @@
"node": "^10 || ^12 || >=14"
}
},
"node_modules/qrcode-generator": {
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/qrcode-generator/-/qrcode-generator-2.0.4.tgz",
"integrity": "sha512-mZSiP6RnbHl4xL2Ap5HfkjLnmxfKcPWpWe/c+5XxCuetEenqmNFf1FH/ftXPCtFG5/TDobjsjz6sSNL0Sr8Z9g==",
"license": "MIT"
},
"node_modules/react": {
"version": "18.3.1",
"resolved": "https://registry.npmjs.org/react/-/react-18.3.1.tgz",
web/package.json
+3 -1
View File
@@ -6,12 +6,14 @@
"scripts": {
"dev": "vite",
"build": "tsc --noEmit && vite build",
"icons": "node scripts/icons.mjs"
"icons": "node scripts/icons.mjs",
"test": "node scripts/test-graphcols.mjs"
},
"dependencies": {
"@xterm/addon-fit": "^0.10.0",
"@xterm/addon-web-links": "^0.11.0",
"@xterm/xterm": "^5.5.0",
"qrcode-generator": "^2.0.4",
"react": "^18.3.1",
"react-dom": "^18.3.1"
},
web/public/icons/apple-touch-icon.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 1.7 KiB

After

Width:  |  Height:  |  Size: 58 KiB

web/public/icons/icon-192.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 1.7 KiB

After

Width:  |  Height:  |  Size: 68 KiB

web/public/icons/icon-512-maskable.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.8 KiB

After

Width:  |  Height:  |  Size: 196 KiB

web/public/icons/icon-512.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4.4 KiB

After

Width:  |  Height:  |  Size: 346 KiB

web/public/icons/logo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 310 KiB

web/public/sw.js
+1 -1
View File
@@ -1,5 +1,5 @@
// Ham Lab service worker: cache hashed assets, never the API, network-first for the shell.
const CACHE = 'hamlab-v1';
const CACHE = 'hamlab-v3'; // bumped when files under /icons/ change: they are served cache-first
self.addEventListener('install', () => self.skipWaiting());
self.addEventListener('activate', (e) => {
e.waitUntil(caches.keys().then((ks) => Promise.all(ks.filter((k) => k !== CACHE).map((k) => caches.delete(k)))).then(() => self.clients.claim()));
web/scripts/icons.mjs
+63 -21
View File
@@ -1,26 +1,68 @@
// Renders the PWA icons with node-canvas (borrowed from ../../calorie-counter/node_modules).
// Renders the PWA icon set and the login logo from scripts/logo-source.png, with node-canvas
// (borrowed from ../../calorie-counter/node_modules). node scripts/icons.mjs
//
// The artwork is a rounded square on a transparent background, a little off-centre in its
// file. It is trimmed to its own edge, then drawn three ways:
// icon-192 / icon-512 / logo the artwork alone, transparent corners, filling the square
// apple-touch-icon iOS paints transparency black and cuts its own, tighter
// corners: a dark ground from the artwork's own palette, the
// artwork inset a little so its neon ring clears that cut
// icon-512-maskable Android may crop to a circle 80% across: the whole artwork
// sits inside that circle, on the same dark ground to the edge
import { createRequire } from 'node:module'
import { writeFileSync } from 'node:fs'
const require = createRequire(import.meta.url)
const { createCanvas } = require('/home/core/projects/calorie-counter/node_modules/canvas')
const { createCanvas, loadImage } = require('/home/core/projects/calorie-counter/node_modules/canvas')
function draw(size, { pad = 0, radius = 0 }) {
const c = createCanvas(size, size), g = c.getContext('2d')
g.fillStyle = '#111418'
if (radius) { g.beginPath(); g.roundRect(0, 0, size, size, radius); g.fill() } else g.fillRect(0, 0, size, size)
const inner = size - pad * 2
g.fillStyle = '#e3a33b'
g.font = `bold ${inner * 0.62}px "DejaVu Serif", serif`
g.textAlign = 'center'; g.textBaseline = 'middle'
g.fillText('H', size / 2, size / 2 + inner * 0.03)
// prompt caret
g.fillStyle = '#58a6ff'
g.font = `bold ${inner * 0.22}px "DejaVu Sans Mono", monospace`
g.fillText('>_', size / 2 + inner * 0.27, size / 2 + inner * 0.27)
return c.toBuffer('image/png')
const img = await loadImage(new URL('./logo-source.png', import.meta.url).pathname)
const src = createCanvas(img.width, img.height), sg = src.getContext('2d')
sg.drawImage(img, 0, 0)
const px = sg.getImageData(0, 0, img.width, img.height), d = px.data
// The export left the artwork at alpha 252 and a stray alpha-1 speck outside it: make what
// is there fully there, what isn't fully gone, and keep the anti-aliased edge between.
let x0 = img.width, y0 = img.height, x1 = 0, y1 = 0
for (let y = 0; y < img.height; y++) for (let x = 0; x < img.width; x++) {
const i = (y * img.width + x) * 4 + 3
if (d[i] >= 240) d[i] = 255; else if (d[i] < 8) d[i] = 0
if (d[i] > 128) { if (x < x0) x0 = x; if (x > x1) x1 = x; if (y < y0) y0 = y; if (y > y1) y1 = y }
}
writeFileSync('public/icons/icon-192.png', draw(192, {}))
writeFileSync('public/icons/icon-512.png', draw(512, {}))
writeFileSync('public/icons/icon-512-maskable.png', draw(512, { pad: 80 }))
writeFileSync('public/icons/apple-touch-icon.png', draw(180, {}))
console.log('icons written')
sg.putImageData(px, 0, 0)
// a square around the artwork's own centre, as wide as its longer side: never stretched
const side = Math.max(x1 - x0, y1 - y0) + 1, cx = (x0 + x1) / 2, cy = (y0 + y1) / 2
const art = createCanvas(side, side)
art.getContext('2d').drawImage(src, Math.round(cx - side / 2), Math.round(cy - side / 2), side, side, 0, 0, side, side)
// Shrinking six-fold in one step aliases the fine lines; halving until close does not.
function shrink(from, size) {
let cur = from
while (cur.width / 2 >= size) {
const next = createCanvas(Math.ceil(cur.width / 2), Math.ceil(cur.height / 2)), g = next.getContext('2d')
g.imageSmoothingEnabled = true; g.imageSmoothingQuality = 'high'; g.quality = 'best'; g.patternQuality = 'best'
g.drawImage(cur, 0, 0, next.width, next.height)
cur = next
}
return cur
}
function draw(size, { scale = 1, ground = false } = {}) {
const c = createCanvas(size, size), g = c.getContext('2d')
if (ground) { // the artwork's own night-green, lit a little behind the middle
const bg = g.createRadialGradient(size / 2, size / 2, 0, size / 2, size / 2, size * 0.72)
bg.addColorStop(0, '#0b2e2a'); bg.addColorStop(1, '#03110f')
g.fillStyle = bg; g.fillRect(0, 0, size, size)
}
const w = Math.round(size * scale), from = shrink(art, w)
g.imageSmoothingEnabled = true; g.imageSmoothingQuality = 'high'; g.quality = 'best'; g.patternQuality = 'best'
g.drawImage(from, (size - w) / 2, (size - w) / 2, w, w)
return c.toBuffer('image/png', { compressionLevel: 9 })
}
const out = {
'icon-192.png': draw(192),
'icon-512.png': draw(512),
'logo.png': draw(480),
'apple-touch-icon.png': draw(180, { scale: 0.95, ground: true }),
'icon-512-maskable.png': draw(512, { scale: 0.68, ground: true }),
}
for (const [name, buf] of Object.entries(out)) { writeFileSync(new URL('../public/icons/' + name, import.meta.url).pathname, buf); console.log(name, Math.round(buf.length / 1024) + ' KB') }
console.log(`artwork ${x1 - x0 + 1}×${y1 - y0 + 1} at ${x0},${y0} → ${side} px square`)
web/scripts/logo-source.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.5 MiB

web/scripts/test-graphcols.mjs
+42
View File
@@ -0,0 +1,42 @@
// node scripts/test-graphcols.mjs — the rule in src/graphcols.ts, on a curve scrolled one point at a time.
// (No test runner in this project: esbuild, which vite already brings, turns the one file into something node can load.)
import { buildSync } from 'esbuild'
import { pathToFileURL } from 'node:url'
import { mkdtempSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
const out = join(mkdtempSync(join(tmpdir(), 'graphcols-')), 'graphcols.mjs')
buildSync({ entryPoints: [new URL('../src/graphcols.ts', import.meta.url).pathname], outfile: out, format: 'esm', logLevel: 'error' })
const { columnsOf } = await import(pathToFileURL(out).href)
const fails = []
const check = (ok, what, detail) => { if (!ok) fails.push(what + (detail !== undefined ? ' — ' + JSON.stringify(detail).slice(0, 200) : '')) }
const CAP = 300, COLS = 90 // the net graph: 300 points of history in about 90 dot-columns
const base = Array.from({ length: CAP }, (_, i) => (i === 100 ? 900 : i === 180 ? 500 : i === 251 ? 700 : 10))
const spikes = (cols) => cols.filter((v) => v > 100).sort((a, b) => b - a).join(',')
// what Top drew before 2026-09-21: the nearest point for each column, counted from the right edge
const nearest = (values, cap, cols) => { const n = Math.min(values.length, cap), seen = []; for (let c = 0; c < cols; c++) { const i = Math.floor((c * cap) / cols) - (cap - n); if (i >= 0) seen.push(values[values.length - n + i]) } return seen }
let values = base.slice(), advance = CAP - 1, prev = null
const oldShapes = new Set(), newShapes = new Set()
for (let step = 0; step < 40; step++) {
oldShapes.add(spikes(nearest(values, CAP, COLS)))
const cols = columnsOf(values, advance, CAP, COLS)
newShapes.add(spikes(cols))
if (prev) check([0, 1].some((sh) => cols.slice(0, -1).every((v, j) => prev[j + sh] === undefined || prev[j + sh] === v)), `frame ${step} is the one before, slid left by a column at most`)
prev = cols
values.push(10); values.shift(); advance++
}
check(oldShapes.size > 1, 'the test reproduces the fault it guards against (spikes flickering under nearest-point drawing)', [...oldShapes])
check(newShapes.size === 1 && [...newShapes][0] === '900,700,500', 'every spike is drawn, at its height, on every frame', [...newShapes])
check(columnsOf(base, CAP - 1, CAP, COLS).length === COLS, 'a full curve fills the box')
const young = columnsOf(base.slice(-30), 29, CAP, COLS).length
check(young >= 8 && young <= 10, 'a tenth of a history takes a tenth of the box, from the right', young)
const wide = columnsOf(base, CAP - 1, CAP, 390)
check(wide.length === 390 && Math.max(...wide) === 900, 'a box wider than the history is filled, nothing invented', wide.length)
check(columnsOf([], 0, CAP, COLS).length === 0 && columnsOf([5], 0, 2, 50).length >= 1, 'empty and single-point curves')
if (fails.length) { console.error('graphcols: FAIL\n ' + fails.join('\n ')); process.exit(1) }
console.log('graphcols: ok')
web/src/Alerts.tsx
+132
View File
@@ -0,0 +1,132 @@
// Alerts: rules over sensors, watched by this hub for itself and every machine it manages.
import { useCallback, useEffect, useState } from 'react'
import { api, AlertChannel, AlertEvent, AlertRule, AlertsInfo } from './api'
import { accentFor } from './accents'
import { Btn, Loading, askYes, toast, useEscape } from './ui'
const OPS = ['>', '>=', '<', '<=', '==', '!=']
const ago = (ms: number) => { const s = Math.max(0, (Date.now() - ms) / 1000); return s < 90 ? `${Math.round(s)} s` : s < 5400 ? `${Math.round(s / 60)} min` : s < 129600 ? `${Math.round(s / 3600)} h` : `${Math.round(s / 86400)} d` }
const patience = (s: number) => (s === 0 ? 'at once' : s < 120 ? `for ${s} s` : `for ${Math.round(s / 60)} min`)
const show = (v: unknown) => (typeof v === 'number' ? String(Math.round(v * 10) / 10) : String(v))
const parse = (raw: string): number | boolean | string => (raw === 'true' ? true : raw === 'false' ? false : raw.trim() !== '' && !isNaN(Number(raw)) ? Number(raw) : raw)
// A rule names the machines it watches by the name the hub knows them by; on screen each is
// titled by its own hostname, like everywhere else.
function RuleDialog({ rule, machines, title, onSave, onClose }: { rule: AlertRule; machines: string[]; title: (n: string) => string; onSave: (r: AlertRule) => void; onClose: () => void }) {
const [r, setR] = useState({ ...rule, value: String(rule.value) })
const [hits, setHits] = useState<string>('')
useEscape(onClose)
useEffect(() => { // what the selector matches right now, across the fleet
if (!r.sensor.trim()) { setHits(''); return }
const t = window.setTimeout(async () => {
try {
const res = await fetch('/api/fleet/sensors/query', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ machines: '*', query: { x: r.sensor.trim() } }) })
if (!res.ok) { setHits((await res.json().catch(() => ({}))).error ?? 'not a valid pattern'); return }
const by = await res.json() as Record<string, { x?: unknown; error?: string }>
const n = Object.entries(by).map(([m, v]) => [m, v.error ? 0 : v.x && typeof v.x === 'object' ? Object.keys(v.x).length : v.x === null || v.x === undefined ? 0 : 1] as const)
setHits(r.sensor.trim() === 'hamlab.reachable' ? 'every machine: whether it answers the hub' : n.every(([, c]) => c === 0) ? 'matches nothing right now' : 'matches ' + n.filter(([, c]) => c).map(([m, c]) => `${c} on ${m}`).join(', '))
} catch { setHits('') }
}, 350)
return () => clearTimeout(t)
}, [r.sensor])
const all = !r.machines || r.machines === '*'
const picked = all ? [] : r.machines.split(',').map((x) => x.trim()).filter(Boolean)
return (
<div className="modal-back" onClick={onClose}>
<form className="modal picker" onClick={(e) => e.stopPropagation()} onSubmit={(e) => { e.preventDefault(); onSave({ ...r, name: r.name.trim(), sensor: r.sensor.trim(), value: parse(r.value) }) }}>
<h3>{rule.name ? 'Edit rule' : 'New rule'}</h3>
<label className="muted">Name<input autoFocus value={r.name} maxLength={64} onChange={(e) => setR({ ...r, name: e.target.value })} placeholder="Tank nearly full" /></label>
<label className="muted">Sensor — an ID or a pattern; the Widgets tab lists them all
<input className="mono" value={r.sensor} onChange={(e) => setR({ ...r, sensor: e.target.value })} placeholder="zfs.tank.used_pct" spellCheck={false} autoCapitalize="off" />
</label>
{hits && <div className="muted small">{hits}</div>}
<div className="rule-cond">
<select value={r.op} onChange={(e) => setR({ ...r, op: e.target.value })}>{OPS.map((o) => <option key={o}>{o}</option>)}</select>
<input className="mono" value={r.value} onChange={(e) => setR({ ...r, value: e.target.value })} placeholder="90 · true · unhealthy" spellCheck={false} autoCapitalize="off" />
<span className="muted">for</span>
<input className="mono" type="number" min={0} max={86400} value={r.for_sec} onChange={(e) => setR({ ...r, for_sec: Math.max(0, Number(e.target.value) || 0) })} style={{ width: 90 }} /><span className="muted">s</span>
</div>
<div className="chips">{(['info', 'warn', 'crit'] as const).map((s) => <button type="button" key={s} className={'chip' + (r.severity === s ? ' on' : '')} onClick={() => setR({ ...r, severity: s })}>{s === 'crit' ? 'critical' : s === 'warn' ? 'warning' : 'info'}</button>)}</div>
<div className="chips">
<button type="button" className={'chip' + (all ? ' on' : '')} onClick={() => setR({ ...r, machines: '*' })}>every machine</button>
{machines.map((m) => <button type="button" key={m} className={'chip' + (picked.includes(m) ? ' on' : '')} onClick={() => { const next = picked.includes(m) ? picked.filter((x) => x !== m) : [...picked, m]; setR({ ...r, machines: next.length ? next.join(',') : '*' }) }}>{title(m)}</button>)}
</div>
<div className="svc-actions"><button type="submit" className="btn on" disabled={!r.name.trim() || !r.sensor.trim()}>Save</button><span className="spacer" /><button type="button" className="btn" onClick={onClose}>Cancel</button></div>
</form>
</div>
)
}
export default function Alerts({ machines, titleOf: title }: { machines: string[]; titleOf: (n: string) => string }) {
const [a, setA] = useState<AlertsInfo | null>(null)
const [err, setErr] = useState('')
const [editing, setEditing] = useState<AlertRule | null>(null)
const [chan, setChan] = useState<AlertChannel>({ type: 'ntfy', url: '' })
const load = useCallback(async () => { try { const r = await api.alerts(); setA({ ...r, rules: r.rules ?? [], channels: r.channels ?? [], events: r.events ?? [], firing: r.firing ?? [] }); setErr('') } catch (e) { setErr(e instanceof Error ? e.message : 'failed') } }, [])
useEffect(() => { load(); const t = setInterval(() => { if (document.visibilityState === 'visible') load() }, 5000); return () => clearInterval(t) }, [load])
if (!a) return <div className="dash">{err ? <div className="error">{err}</div> : <Loading label="Loading alerts…" />}</div>
const save = async (patch: { rules?: AlertRule[]; channels?: AlertChannel[] }, done: string) => {
try { await api.saveAlerts(patch); toast(done); window.dispatchEvent(new Event('hamlab:alerts')); await load() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
}
const saveRule = (r: AlertRule) => { const exists = a.rules.some((x) => x.id === r.id); save({ rules: exists ? a.rules.map((x) => (x.id === r.id ? r : x)) : [...a.rules, r] }, exists ? 'rule saved' : 'rule added'); setEditing(null) }
const sevClass = (s: string) => (s === 'crit' ? 'bad' : s === 'warn' ? 'warn' : '')
return (
<div className="dash alerts">
<div className="dash-bar"><h2 className="section">Now</h2><span className="spacer" /><span className="muted small">checked {a.checked > 0 ? `${ago(a.checked)} ago` : 'not yet'} · every {a.every_sec} s · all machines</span></div>
{a.firing.length === 0 ? <section className="card all-clear"><span className="good">✓ All clear</span><span className="muted small"> — {a.rules.filter((r) => r.enabled).length} rules watching {machines.length + 1} machine{machines.length ? 's' : ''}</span></section> : (
<div className="cards">
{a.firing.map((f) => (
<section className={'card firing ' + sevClass(f.severity)} key={f.rule_id + f.machine + f.sensor}>
<div className="sub-head"><span className={'sdot ' + (f.severity === 'crit' ? 'bad' : 'warn')} /><span className="dname">{f.rule}</span><span className="spacer" /><span className="badge" style={{ color: accentFor(f.machine), borderColor: accentFor(f.machine) }}>{title(f.machine)}</span></div>
<div className="mono small">{f.sensor} = {show(f.value)}</div>
<div className="muted small">for {ago(f.since)}</div>
</section>
))}
</div>
)}
<div className="dash-bar"><h2 className="section">Rules</h2><span className="spacer" />
<button className="btn" onClick={() => setEditing({ id: 'r' + Date.now().toString(36), name: '', sensor: '', op: '>=', value: 90, for_sec: 300, severity: 'warn', enabled: true, machines: '*' })}>+ Rule</button>
</div>
<section className="card rules">
{a.rules.length === 0 && <div className="empty">No rules. Nothing is being watched.</div>}
{a.rules.map((r) => (
<div className={'rule' + (r.enabled ? '' : ' off')} key={r.id}>
<button className={'toggle' + (r.enabled ? ' on' : '')} title={r.enabled ? 'watching — tap to pause' : 'paused — tap to watch'} onClick={() => save({ rules: a.rules.map((x) => (x.id === r.id ? { ...x, enabled: !x.enabled } : x)) }, r.enabled ? `${r.name}: paused` : `${r.name}: watching`)}><i /></button>
<div className="rule-body" onClick={() => setEditing(r)}>
<div className="rule-name"><span className={'sdot ' + (r.severity === 'crit' ? 'bad' : r.severity === 'warn' ? 'warn' : 'ok')} />{r.name}{a.firing.some((f) => f.rule_id === r.id) && <span className="badge bad">firing</span>}</div>
<div className="mono small muted">{r.sensor} {r.op} {show(r.value)} · {patience(r.for_sec)}{r.machines && r.machines !== '*' ? ` · ${r.machines}` : ''}</div>
</div>
<button className="icon-btn" title="delete" onClick={async () => { if (await askYes(`Delete the rule "${r.name}"?`)) save({ rules: a.rules.filter((x) => x.id !== r.id) }, 'rule deleted') }}>✕</button>
</div>
))}
</section>
<div className="dash-bar"><h2 className="section">Where alerts go</h2><span className="spacer" />{a.channels.length > 0 && <Btn className="btn" onClick={async () => { try { await api.testAlerts(); toast('test sent') } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } }}>Send a test</Btn>}</div>
<section className="card">
<div className="muted small" style={{ marginBottom: 8 }}>Always here, and on the rail. To reach your phone: an <b>ntfy</b> topic URL (the ntfy app, from ntfy.sh or your own server), or a <b>webhook</b> — Home Assistant, Slack, anything that takes a POST. Sent when an alert fires and again when it clears.</div>
{a.channels.map((c, i) => <div className="row" key={i}><span className="badge" style={{ marginLeft: 0 }}>{c.type}</span><span className="k mono small">{c.url}</span><Btn className="btn danger" confirm={`Remove this ${c.type} channel?`} onClick={() => save({ channels: a.channels.filter((_, j) => j !== i) }, 'channel removed')}>Remove</Btn></div>)}
<form className="chan-add" onSubmit={(e) => { e.preventDefault(); if (!/^https?:\/\/./.test(chan.url.trim())) { toast('a channel needs an http(s) URL', 'err'); return } save({ channels: [...a.channels, { type: chan.type, url: chan.url.trim() }] }, 'channel added'); setChan({ ...chan, url: '' }) }}>
<select value={chan.type} onChange={(e) => setChan({ ...chan, type: e.target.value as AlertChannel['type'] })}><option value="ntfy">ntfy</option><option value="webhook">webhook</option></select>
<input className="mono" value={chan.url} onChange={(e) => setChan({ ...chan, url: e.target.value })} placeholder={chan.type === 'ntfy' ? 'https://ntfy.sh/your-secret-topic' : 'https://…/api/webhook/…'} spellCheck={false} autoCapitalize="off" />
<button className="btn on" disabled={!chan.url.trim()}>Add</button>
</form>
</section>
<div className="dash-bar"><h2 className="section">History</h2><span className="spacer" /><span className="muted small">the last {a.events.length}</span></div>
<section className="card">
{a.events.length === 0 && <div className="empty">Nothing has fired yet.</div>}
{a.events.map((e: AlertEvent, i) => (
<div className="ev" key={i}>
<span className={e.kind === 'firing' ? (e.severity === 'crit' ? 'warn-text bad-text' : 'warn-text') : 'good'}>{e.kind === 'firing' ? '▲' : '✓'}</span>
<span className="ev-main">{e.rule} <span className="muted">· {title(e.machine)} · </span><span className="mono small muted">{e.sensor} = {show(e.value)}</span></span>
<span className="muted small num" title={new Date(e.t).toLocaleString()}>{ago(e.t)} ago</span>
</div>
))}
</section>
{editing && <RuleDialog rule={editing} machines={machines} title={title} onSave={saveRule} onClose={() => setEditing(null)} />}
</div>
)
}
web/src/App.tsx
+24 -2
View File
@@ -2,15 +2,37 @@ import { useCallback, useEffect, useState } from 'react'
import { api, Me, Unauthenticated } from './api'
import Login from './Login'
import Shell from './Shell'
import { Dialogs } from './ui'
import { applyTheme, currentThemeId } from './themes'
import { getHistorySec, getRefreshMs, setHistorySec, setRefreshMs } from './prefs'
import { setAccents } from './accents'
import { setRailOrder } from './railorder'
export default function App() {
const [me, setMe] = useState<Me | null | undefined>(undefined)
const refresh = useCallback(async () => {
try { setMe(await api.me()) } catch (e) { if (e instanceof Unauthenticated) setMe(null) }
try {
const m = await api.meHere(); setMe(m)
api.uiPrefs().then((p) => {
if (p.theme && p.theme !== currentThemeId()) applyTheme(p.theme)
if (p.refreshMs && p.refreshMs !== getRefreshMs()) setRefreshMs(p.refreshMs)
if (p.accents && typeof p.accents === 'object') setAccents(p.accents, false)
if (Array.isArray(p.rail)) setRailOrder(p.rail, false)
}).catch(() => { /* keep local */ })
// The graph window is the machines' own setting; the copy here only labels the curves
// and spaces the live points, so take the hub's word for it at every login.
api.historyWindow().then((h) => { if (h.seconds && h.seconds !== getHistorySec()) setHistorySec(h.seconds) }).catch(() => { /* keep local */ })
} catch (e) { if (e instanceof Unauthenticated) setMe(null) }
}, [])
useEffect(() => { refresh() }, [refresh])
// Any page that gets a 401 from the hub sends us straight back to the login form.
useEffect(() => {
const h = () => setMe((m) => (m ? null : m))
window.addEventListener('hamlab:unauthenticated', h)
return () => window.removeEventListener('hamlab:unauthenticated', h)
}, [])
useEffect(() => {
if (!me) return
const t = setInterval(refresh, 30000)
@@ -19,5 +41,5 @@ export default function App() {
if (me === undefined) return <div className="center muted">…</div>
if (me === null) return <Login onLogin={refresh} />
return <Shell me={me} onLoggedOut={() => setMe(null)} />
return <><Dialogs /><Shell me={me} onLoggedOut={() => setMe(null)} /></>
}
web/src/Disks.tsx
+165
View File
@@ -0,0 +1,165 @@
import { useCallback, useEffect, useMemo, useState } from 'react'
import { api, SmartInfo, StorageDevice } from './api'
import { fmtBytes, fmtRate } from './fmt'
import TaskOutput from './TaskOutput'
import { askText, Btn, Loading, Meter, Spinner, toast } from './ui'
import { useRefreshMs } from './prefs'
import { Sensor, useSensorStream } from './sensors'
const pct = (v: number) => `${Math.round(v)}%`
const KIND: Record<string, string> = { nvme: 'NVMe', ssd: 'SSD', hdd: 'HDD', usb: 'USB', emmc: 'eMMC', sd: 'SD', other: 'disk' }
const hours = (h: number) => (h >= 24 * 365 ? `${(h / 8760).toFixed(1)} y` : h >= 48 ? `${Math.round(h / 24)} d` : `${h} h`)
const ago = (ms: number) => { const s = (Date.now() - ms) / 1000; return s < 3600 ? `${Math.round(s / 60)}m ago` : s < 86400 ? `${Math.round(s / 3600)}h ago` : `${Math.round(s / 86400)}d ago` }
// ---- SMART panel ----
function SmartPanel({ dev, rotational }: { dev: string; rotational: boolean }) {
const [s, setS] = useState<SmartInfo | null>(null)
const [err, setErr] = useState('')
const load = useCallback(async (force = false) => {
setS(null)
try { setS(await api.smart(dev, force)); setErr('') } catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
}, [dev])
useEffect(() => { load() }, [load])
if (err) return <div className="error">{err}</div>
if (!s) return <Loading label="Reading health…" />
if (!s.available) return <div className="notice">smartmontools isn't installed, so there's no SMART data. Install it from the button above.</div>
if (s.standby) return <div className="notice">Disk is spun down; SMART wasn't read so it stays asleep. <Btn className="btn" onClick={() => load(true)}>Wake and read</Btn></div>
if (s.error) return <div className="notice">{s.error}</div>
const ok = s.passed === true && s.warnings.length === 0
return (
<div className="smart">
<div className="props">
<div><span className="k">Health</span><span className={ok ? 'good' : 'warn-text'}>{s.passed === null ? 'unknown' : s.passed ? 'PASSED' : 'FAILED'}{s.warnings.length > 0 && ` · ${s.warnings.length} warning${s.warnings.length > 1 ? 's' : ''}`}</span></div>
{s.temp > 0 && <div><span className="k">Temperature</span><span>{s.temp}°C</span></div>}
{s.power_on_hours > 0 && <div><span className="k">Powered on</span><span>{hours(s.power_on_hours)} · {s.power_cycles} cycles</span></div>}
{s.firmware && <div><span className="k">Firmware</span><span className="mono">{s.firmware}</span></div>}
{s.serial && <div><span className="k">Serial</span><span className="mono">{s.serial}</span></div>}
{s.self_test && <div><span className="k">Self-test</span><span>{s.self_test}{s.self_test_remaining ? ` (${s.self_test_remaining}% left)` : ''}</span></div>}
<div><span className="k">Read</span><span className="muted">{ago(s.fetched_at)} <button className="link" onClick={() => load(true)}>refresh</button></span></div>
</div>
{s.warnings.length > 0 && <div className="notice attn">{s.warnings.join(' · ')}</div>}
{s.mmc && (
<div className="props">
<div><span className="k">Wear</span><span>{[['A', s.mmc.life_a], ['B', s.mmc.life_b]].filter(([, v]) => v).map(([k, v]) => `type ${k} ${v === 11 ? 'past rated life' : `${(v as number) * 10 - 10}–${(v as number) * 10}% used`}`).join(' · ') || 'not reported'}</span></div>
<div><span className="k">Reserve</span><span>{['not reported', 'normal', 'warning — 80% of spare blocks used', 'urgent — 90% of spare blocks used'][s.mmc.pre_eol] ?? s.mmc.pre_eol}</span></div>
</div>
)}
{s.nvme && (
<div className="props">
<div><span className="k">Endurance</span><span>{s.nvme.percentage_used}% used · spare {s.nvme.available_spare}% (threshold {s.nvme.spare_threshold}%)</span></div>
<div><span className="k">Written</span><span>{fmtBytes(s.nvme.data_written_bytes)} · read {fmtBytes(s.nvme.data_read_bytes)}</span></div>
<div><span className="k">Errors</span><span>{s.nvme.media_errors} media · {s.nvme.error_log_entries} logged · {s.nvme.unsafe_shutdowns} unsafe shutdowns</span></div>
</div>
)}
{s.attrs.length > 0 && (
<div className="attrs-wrap">
<table className="attrs">
<thead><tr><th>ID</th><th>Attribute</th><th>Raw</th><th>Value</th><th>Worst</th><th>Thresh</th></tr></thead>
<tbody>
{s.attrs.map((a) => (
<tr key={a.id} className={a.warn ? 'warn' : ''}>
<td className="num">{a.id}</td><td>{a.name.replace(/_/g, ' ')}{a.failed && <span className="badge bad">{a.failed}</span>}</td>
<td className="num raw">{a.raw}</td><td className="num">{a.value}</td><td className="num">{a.worst}</td><td className="num">{a.thresh}</td>
</tr>
))}
</tbody>
</table>
</div>
)}
{!s.self_test_supported && <div className="muted small">Self-tests: {s.self_test_note || 'not available on this drive'}.</div>}
{s.self_test_supported && <div className="svc-actions">
<Btn className="btn" confirm={`Start a short self-test on ${dev}?`} onClick={async () => { if (await api.smartTest(dev, 'short').then(() => true, (e) => { toast(String(e.message ?? e), 'err'); return false })) { toast('short test started'); load(true) } }}>Short test</Btn>
<Btn className="btn" confirm={`Start a long self-test on ${dev}? ${rotational ? 'This can take hours on a spinning disk.' : ''}`} onClick={async () => { if (await api.smartTest(dev, 'long').then(() => true, (e) => { toast(String(e.message ?? e), 'err'); return false })) { toast('long test started'); load(true) } }}>Long test</Btn>
</div>}
</div>
)
}
// ---- device card ----
function DeviceCard({ d, live, forceOpen }: { d: StorageDevice; live?: { read_bps: number; write_bps: number; util_pct: number }; forceOpen: boolean | null }) {
const [own, setOwn] = useState(false)
const open = forceOpen ?? own
const setOpen = (v: boolean) => setOwn(v)
return (
<section className="card disk">
<div className="sub-head">
<span className="badge">{KIND[d.kind] ?? d.kind}</span>
<span className="dname">{d.name}</span>
<span className="model">{[d.vendor, d.model].filter(Boolean).join(' ')}</span>
<span className="spacer" />
<span className="num">{fmtBytes(d.size)}{d.temp ? ` · ${d.temp.toFixed(0)}°C` : ''}</span>
</div>
{live && <div className="sub-head num muted">read {fmtRate(live.read_bps)} · write {fmtRate(live.write_bps)} · {pct(live.util_pct)} busy</div>}
<div className="muted small disk-summary">
{d.serial && <span>S/N {d.serial}</span>}
{d.namespaces && d.namespaces.length > 1 && <span>{d.namespaces.length} namespaces</span>}
<span>{d.parts.length} partition{d.parts.length === 1 ? '' : 's'}</span>
{d.parts.filter((p) => p.mount && p.mount !== '[SWAP]').length > 0 && <span>mounted at {d.parts.filter((p) => p.mount && p.mount !== '[SWAP]').map((p) => p.mount).join(', ')}</span>}
{d.parts.some((p) => p.mount === '[SWAP]') && <span>swap</span>}
</div>
{forceOpen === null && <div className="svc-actions"><button className="btn" onClick={() => setOpen(!open)}>{open ? 'Hide SMART' : 'SMART'}</button></div>}
{open && <SmartPanel dev={d.name} rotational={d.rotational} />}
</section>
)
}
// ---- page ----
export default function Disks() {
const [devs, setDevs] = useState<StorageDevice[] | null>(null)
const [hasSmart, setHasSmart] = useState(true)
const [err, setErr] = useState('')
const [task, setTask] = useState<string | null>(null)
const [allSmart, setAllSmart] = useState<boolean>(() => localStorage.getItem('hamlab.disks.smart') === '1')
const rate = useRefreshMs()
const load = useCallback(async () => {
try {
const r = await api.storage()
setDevs(r.devices); setHasSmart(r.smartctl); setErr('')
} catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
}, [])
useEffect(() => { load() }, [load])
// Live per-device rates come from the sensor stream, grouped by the kernel name in each sensor's labels.
const stream = useSensorStream<{ d: Record<string, Sensor> }>({ d: 'disk.*' }, { everyMs: rate })
const live = useMemo(() => {
const by: Record<string, { name: string; read_bps: number; write_bps: number; util_pct: number }> = {}
for (const x of Object.values(stream.data?.d ?? {})) {
const name = x.labels?.device
if (!name || typeof x.value !== 'number') continue
const row = (by[name] ??= { name, read_bps: 0, write_bps: 0, util_pct: 0 })
if (x.id.endsWith('.read')) row.read_bps = x.value
else if (x.id.endsWith('.write')) row.write_bps = x.value
else if (x.id.endsWith('.util')) row.util_pct = x.value
}
return { disks: Object.values(by) }
}, [stream.data])
if (!devs) return <div className="dash">{err ? <div className="error">{err}</div> : <Loading label="Scanning disks…" />}</div>
const liveFor = (dev: StorageDevice) => {
const names = dev.namespaces?.length ? dev.namespaces : [dev.name]
const rows = (live?.disks ?? []).filter((x) => names.includes(x.name))
if (rows.length === 0) return undefined
return { read_bps: rows.reduce((a, x) => a + x.read_bps, 0), write_bps: rows.reduce((a, x) => a + x.write_bps, 0), util_pct: Math.max(...rows.map((x) => x.util_pct)) }
}
return (
<div className="dash disks">
{err && <div className="error">{err}</div>}
<div className="dash-bar">
<h2 className="section">Devices · {devs.length}</h2>
<span className="spacer" />
{hasSmart && <button className={'chip' + (allSmart ? ' on' : '')} title="show SMART for every drive" onClick={() => { const v = !allSmart; setAllSmart(v); localStorage.setItem('hamlab.disks.smart', v ? '1' : '0') }}>{allSmart ? 'SMART: all open' : 'SMART: show all'}</button>}
{!hasSmart && <Btn className="btn" onClick={async () => { try { const t = await api.installPackages(['smartmontools']); setTask(t.id) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } }}>Install smartmontools</Btn>}
<Btn className="btn" title="reload" onClick={load}>↻</Btn>
</div>
{!hasSmart && <div className="notice">SMART health needs smartmontools on this host. Install it with the button, then reload.</div>}
{task && <TaskOutput id={task} onDone={(t) => { toast(`${t.title}: ${t.exit_code === 0 ? 'finished' : `exited ${t.exit_code}`}`, t.exit_code === 0 ? 'ok' : 'err'); load() }} />}
<div className="cards disk-cards">
{devs.map((d) => <DeviceCard key={d.name} d={d} live={liveFor(d)} forceOpen={allSmart ? true : null} />)}
</div>
</div>
)
}
web/src/Docker.tsx
+682
View File
@@ -0,0 +1,682 @@
// Docker: containers grouped by the stack that deployed them, with start / stop / restart,
// logs that follow, and inspect — then images, volumes and networks. Live CPU and memory
// come from the same container sensors everything else uses.
import { FormEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
import { api, DockerContainer, DockerEvent, DockerHealth, DockerInfo, DockerLogLine, ImageCheck, MachineUpdates, Stack, StackDiff, UpdateJob, UpdateJobRow, wsURLPath } from './api'
import TaskOutput from './TaskOutput'
import StackNew, { DupFrom } from './StackNew'
import { Sensor, fetchHistories, fmtValue, useSensorStream } from './sensors'
import { useHashSel } from './hashsel'
import { fmtWindowSec, useHistorySec, useRefreshMs } from './prefs'
import Sparkline from './Sparkline'
import { fmtBytes } from './fmt'
import { describeCron } from './crontext'
import { askText, askYes, Btn, CloseBtn, Loading, setUnsaved, toast, useEscape } from './ui'
type Tab = 'containers' | 'stacks' | 'updates' | 'images' | 'volumes' | 'networks'
const dot = (c: DockerContainer) => (c.state !== 'running' ? (c.state === 'exited' || c.state === 'dead' ? 'bad' : 'warn') : c.health === 'unhealthy' ? 'bad' : c.health === 'starting' ? 'warn' : 'ok')
const ports = (p: string) => [...new Set(p.split(', ').map((x) => x.replace(/^\[?::\]?:/, '').replace(/^0\.0\.0\.0:/, '').replace(/^[\d.]+:/, '')).filter((x) => x.includes('->')))].join(' ')
// A container's output, followed over a held connection the way the Services tab follows a
// journal. A reconnect resumes from the last line's timestamp, so a container that restarts
// — or a phone that slept — doesn't replay what is already on screen.
function LogView({ id }: { id: string }) {
const [lines, setLines] = useState<DockerLogLine[]>([])
const [err, setErr] = useState('')
const [live, setLive] = useState(false)
const [follow, setFollow] = useState(true)
const [paused, setPaused] = useState(false)
const box = useRef<HTMLDivElement>(null)
const last = useRef('')
const pausedRef = useRef(false)
pausedRef.current = paused
const pending = useRef<DockerLogLine[]>([])
// What a resume has already shown: `--since` is inclusive, and two lines can share a
// timestamp (a container writing to both streams at once), so the boundary is a set of
// lines rather than a moment. Bounded — only the lines around a reconnect can repeat.
const seen = useRef<{ set: Set<string>; q: string[] }>({ set: new Set(), q: [] })
useEffect(() => {
setLines([]); setErr(''); setLive(false); last.current = ''; pending.current = []
seen.current = { set: new Set(), q: [] }
let closed = false
let sock: WebSocket | null = null
let retry: number | undefined
let attempt = 0
const connect = () => {
if (closed) return
const q = last.current ? `since=${encodeURIComponent(last.current)}` : 'n=300'
// fixed when the log opens: a reconnect must not follow the console to another machine
const s = new WebSocket(wsURLPath(`/api/docker/containers/${encodeURIComponent(id)}/logs/tail?${q}`))
sock = s
s.onopen = () => { attempt = 0; setLive(true); setErr('') }
s.onmessage = (ev) => {
const l = JSON.parse(ev.data) as DockerLogLine
if (l.t) {
const k = l.t + '|' + (l.err ? 'e' : 'o') + '|' + l.m
const { set, q } = seen.current
if (set.has(k)) return
set.add(k); q.push(k)
if (q.length > 1000) for (const old of q.splice(0, 500)) set.delete(old)
if (l.t > last.current) last.current = l.t
}
if (pausedRef.current) { pending.current.push(l); return }
setLines((prev) => (prev.length > 2000 ? [...prev.slice(-1500), l] : [...prev, l]))
}
// a container that has stopped — or was never there — ends the stream at once, so
// back off rather than asking Docker the same question twice a second forever
s.onclose = () => { if (closed) return; setLive(false); retry = window.setTimeout(connect, Math.min(15000, 1000 * 2 ** attempt++)) }
s.onerror = () => { setErr('log stream disconnected'); s.close() }
}
connect()
return () => { closed = true; clearTimeout(retry); sock?.close() }
}, [id])
// Resume: flush what arrived while paused.
useEffect(() => {
if (paused || pending.current.length === 0) return
const p = pending.current; pending.current = []
setLines((prev) => [...prev, ...p].slice(-2000))
}, [paused])
useEffect(() => { if (follow && !paused && box.current) box.current.scrollTop = box.current.scrollHeight }, [lines, follow, paused])
return (
<div className="logs">
<div className="logs-bar">
<span className={'dot ' + (live ? 'ok' : 'bad')} /> <span className="muted">{live ? 'following' : 'reconnecting…'} · {lines.length} lines{err && ` · ${err}`}</span>
<span className="spacer" />
<button className={'chip' + (follow ? ' on' : '')} onClick={() => setFollow(!follow)}>follow</button>
<button className="chip" onClick={() => setPaused(!paused)}>{paused ? `Resume${pending.current.length ? ` (${pending.current.length})` : ''}` : 'Pause'}</button>
<button className="chip" onClick={() => setLines([])}>Clear</button>
</div>
<div className="log-lines" ref={box} onScroll={(e) => { const el = e.currentTarget; if (el.scrollHeight - el.scrollTop - el.clientHeight > 40) setFollow(false) }}>
{lines.map((l, i) => <div className={'log' + (l.err ? ' err' : '')} key={i}><span className="lt">{l.t ? new Date(l.t).toLocaleTimeString() : ''}</span><span className="lm">{l.m}</span></div>)}
{lines.length === 0 && !err && <div className="muted small" style={{ padding: 8 }}>No output yet.</div>}
</div>
</div>
)
}
// What a container is doing, and what can be done to it: its numbers over time, its health
// checks, what has happened to it lately, a shell inside it, and its limits.
function Detail({ c, live, onBack, onChanged }: { c: DockerContainer; live: Record<string, Sensor>; onBack: () => void; onChanged: () => void }) {
const [tab, setTab] = useState<'logs' | 'stats' | 'health' | 'events' | 'inspect'>('logs')
const [info, setInfo] = useState<Record<string, unknown> | null>(null)
const [limits, setLimits] = useState(false)
useEffect(() => { setInfo(null); if (tab === 'inspect' || limits) api.dockerInspect(c.name).then(setInfo).catch((e) => toast(e instanceof Error ? e.message : 'failed', 'err')) }, [tab, limits, c.name])
const act = (a: 'start' | 'stop' | 'restart' | 'remove') => async () => { try { await api.dockerAction(c.name, a); toast(`${c.name}: ${a} ok`); onChanged() } catch (e) { toast(`${c.name}: ${e instanceof Error ? e.message : 'failed'}`, 'err') } }
const running = c.state === 'running'
const seg = c.name.toLowerCase().replace(/[^a-z0-9-]+/g, '_').replace(/^_|_$/g, '')
const s = (k: string) => live[`container.${seg}.${k}`]
const mounts = (info?.mounts as { Source: string; Destination: string; RW: boolean }[] | undefined) ?? []
const shell = async () => {
try { const t = await api.dockerExec(c.name); toast(`shell open as “${t.name}” on the Terminal tab`); window.dispatchEvent(new CustomEvent('hamlab:terminal', { detail: t.id })) }
catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
}
return (
<div className="svc-detail">
<CloseBtn onClick={onBack} />
<div className="svc-head"><button className="btn back" onClick={onBack}>‹ Back</button><span className={'sdot ' + dot(c)} /><span className="dname">{c.name}</span>{c.health && c.health !== 'none' && <span className={'badge' + (c.health === 'healthy' ? '' : ' bad')}>{c.health}</span>}</div>
<div className="props">
<div><span className="k">Image</span><span className="mono small">{c.image}</span></div>
<div><span className="k">Status</span><span>{c.status}</span></div>
{c.stack && <div><span className="k">Stack</span><span>{c.stack}{c.service && c.service !== c.name ? ` · ${c.service}` : ''}</span></div>}
{running && <div><span className="k">Using</span><span>{fmtValue(s('cpu')?.value, '%')} of a core · {fmtValue(s('mem')?.value ?? c.mem, 'B')}</span></div>}
{ports(c.ports) && <div><span className="k">Ports</span><span className="mono small">{ports(c.ports)}</span></div>}
</div>
<div className="svc-actions">
{!running && <Btn className="btn on" onClick={act('start')}>Start</Btn>}
{!running && <Btn className="btn danger" confirm={`Remove the container ${c.name}? Its image, volumes and bind-mounted data stay; anything written inside the container itself is lost.${c.stack ? ` Deploying the ${c.stack} stack makes it again.` : ''}`} onClick={async () => { await act('remove')(); onBack() }}>Remove</Btn>}
{running && <Btn className="btn" onClick={shell}>Shell</Btn>}
{running && <Btn className="btn" confirm={`Restart ${c.name}?`} onClick={act('restart')}>Restart</Btn>}
<Btn className="btn" onClick={() => setLimits(true)}>Limits</Btn>
{running && <Btn className="btn danger" confirm={`Stop ${c.name}? It stays stopped until something starts it.`} onClick={act('stop')}>Stop</Btn>}
</div>
<div className="tabs">
<button className={tab === 'logs' ? 'on' : ''} onClick={() => setTab('logs')}>Logs</button>
<button className={tab === 'stats' ? 'on' : ''} onClick={() => setTab('stats')}>Stats</button>
<button className={tab === 'health' ? 'on' : ''} onClick={() => setTab('health')}>Health</button>
<button className={tab === 'events' ? 'on' : ''} onClick={() => setTab('events')}>Events</button>
<button className={tab === 'inspect' ? 'on' : ''} onClick={() => setTab('inspect')}>Inspect</button>
</div>
{tab === 'logs' && <LogView id={c.name} />}
{tab === 'stats' && <Stats seg={seg} live={live} />}
{tab === 'health' && <Health id={c.name} />}
{tab === 'events' && <Events id={c.name} />}
{tab === 'inspect' && (!info ? <Loading label="Inspecting…" /> : (
<div className="props stacked">
<div><span className="k">Restarts</span><span>{String(info.restart || 'no')} policy · restarted {String(info.restarts ?? 0)}×</span></div>
<div><span className="k">Network</span><span className="mono small">{Object.keys((info.networks as object) ?? {}).join(', ') || String(info.network_mode)}</span></div>
{Number(info.memory_limit) > 0 && <div><span className="k">Mem limit</span><span>{fmtValue(Number(info.memory_limit), 'B')}</span></div>}
{!!info.error && <div><span className="k">Error</span><span className="warn-text">{String(info.error)}</span></div>}
{!!info.compose_file && <div><span className="k">Compose</span><span className="mono small">{String(info.compose_file)}</span></div>}
{mounts.length > 0 && <div><span className="k">Mounts</span><span className="mono small">{mounts.map((m) => <div key={m.Destination}>{m.Source} → {m.Destination}{m.RW ? '' : ' (read only)'}</div>)}</span></div>}
<div><span className="k">Env <span className="muted">names only</span></span><span className="mono small muted">{((info.env_names as string[]) ?? []).join(' ') || '—'}</span></div>
</div>
))}
{limits && <LimitsDialog c={c} info={info} onClose={() => setLimits(false)} onSaved={() => { setLimits(false); onChanged() }} />}
</div>
)
}
// CPU and memory over time, from the container sensors the sampler already keeps.
function Stats({ seg, live }: { seg: string; live: Record<string, Sensor> }) {
const [hist, setHist] = useState<{ id: string; t: number[]; v: number[] }[] | null>(null)
const historySec = useHistorySec()
useEffect(() => {
let alive = true
fetchHistories(`container.${seg}.cpu,container.${seg}.mem`)
.then((h) => { if (alive) setHist(h) })
.catch(() => { if (alive) setHist([]) })
return () => { alive = false }
}, [seg])
const css = (n: string) => getComputedStyle(document.documentElement).getPropertyValue(n).trim() || '#3987e5'
const of = (k: string) => hist?.find((h) => h.id === `container.${seg}.${k}`)?.v ?? []
const cpu = of('cpu'), mem = of('mem')
if (hist === null) return <Loading label="Reading the history…" />
if (cpu.length < 2 && mem.length < 2) return <div className="notice">No history yet for this container — it is sampled while it runs, so give it a moment.</div>
return (
<div className="cards">
<section className="card">
<div className="sub-head"><span className="dname">CPU</span><span className="spacer" /><span className="num">{fmtValue(live[`container.${seg}.cpu`]?.value, '%')} of a core</span></div>
<Sparkline series={[{ name: 'cpu', values: cpu, color: css('--s1') }]} format={(v) => `${v.toFixed(1)}%`} height={64} />
<div className="muted small">the last {fmtWindowSec(historySec)}</div>
</section>
<section className="card">
<div className="sub-head"><span className="dname">Memory</span><span className="spacer" /><span className="num">{fmtValue(live[`container.${seg}.mem`]?.value, 'B')}</span></div>
<Sparkline series={[{ name: 'mem', values: mem, color: css('--s2') }]} format={(v) => fmtBytes(v)} height={64} />
<div className="muted small">the last {fmtWindowSec(historySec)}</div>
</section>
</div>
)
}
// The health check's verdict, and what its last few probes actually said.
function Health({ id }: { id: string }) {
const [h, setH] = useState<DockerHealth | null>(null)
const [err, setErr] = useState('')
useEffect(() => { setH(null); api.dockerHealth(id).then(setH).catch((e) => setErr(e instanceof Error ? e.message : 'failed')) }, [id])
if (err) return <div className="error">{err}</div>
if (!h) return <Loading label="Asking…" />
if (!h.configured) return <div className="notice">This image defines no health check, so Docker only knows whether the process is running — which it is ({h.state}). A health check is a <span className="mono">healthcheck:</span> block in the compose file.</div>
return (
<>
<div className="props">
<div><span className="k">Health</span><span className={h.status === 'healthy' ? '' : 'warn-text'}>{h.status}</span></div>
<div><span className="k">State</span><span>{h.state}</span></div>
{!!h.failing_streak && <div><span className="k">Failing</span><span className="warn-text">{h.failing_streak} probe{h.failing_streak === 1 ? '' : 's'} in a row</span></div>}
</div>
<div className="muted small" style={{ marginTop: 8 }}>The last {h.log?.length ?? 0} probes — Docker keeps five.</div>
{(h.log ?? []).slice().reverse().map((p, i) => (
<section className="card" key={i} style={{ marginTop: 6 }}>
<div className="sub-head">
<span className={'sdot ' + (p.ExitCode === 0 ? 'ok' : 'bad')} />
<span>{new Date(p.Start).toLocaleString()}</span>
<span className="spacer" />
<span className="muted small">exit {p.ExitCode} · {Math.max(0, new Date(p.End).getTime() - new Date(p.Start).getTime())} ms</span>
</div>
{p.Output.trim() && <pre className="run-out" style={{ maxHeight: 140 }}>{p.Output.trim()}</pre>}
</section>
))}
</>
)
}
// What has happened to this container lately, asked of Docker rather than kept here.
function Events({ id }: { id: string }) {
const [ev, setEv] = useState<DockerEvent[] | null>(null)
const [kept, setKept] = useState(0)
const [since, setSince] = useState('7d')
const [err, setErr] = useState('')
useEffect(() => { setEv(null); setErr(''); api.dockerEvents(id, since).then((r) => { setEv(r.events); setKept(r.kept) }).catch((e) => setErr(e instanceof Error ? e.message : 'failed')) }, [id, since])
const bad = (a: string, h?: string) => /die|kill|oom|destroy/.test(a) || (h !== undefined && h !== 'healthy')
return (
<>
<div className="chips" style={{ marginBottom: 8 }}>
{['24h', '7d', '30d'].map((x) => <button key={x} className={'chip' + (since === x ? ' on' : '')} onClick={() => setSince(x)}>{x}</button>)}
</div>
{err && <div className="error">{err}</div>}
{!ev && !err && <Loading label="Asking Docker…" />}
{ev && ev.length === 0 && <div className="notice">Nothing for this container in the last {since}. Ham Lab keeps its own record — Docker's own only reaches back a few minutes — so history starts from when this version was installed. {kept} events kept across every container.</div>}
{ev && ev.length > 0 && (
<div className="snap-list">
{ev.map((e, i) => (
<div className="snap" key={i}>
<span className={bad(e.action, e.health) ? 'warn-text' : ''}>{e.action}{e.health ? `: ${e.health}` : ''}</span>
<span className="muted">{new Date(e.t).toLocaleString()}</span>
<span className="muted small">{e.exit_code ? `exit ${e.exit_code}` : ''}{e.signal ? ` signal ${e.signal}` : ''}</span>
<span />
<span />
</div>
))}
</div>
)}
</>
)
}
// Memory, CPU and restart policy, changed on the running container. Docker does this live,
// which is the point — and also the catch, said plainly below.
function LimitsDialog({ c, info, onClose, onSaved }: { c: DockerContainer; info: Record<string, unknown> | null; onClose: () => void; onSaved: () => void }) {
const curMem = Number(info?.memory_limit ?? 0)
const [mem, setMem] = useState(curMem > 0 ? String(Math.round(curMem / (1 << 20))) + 'm' : '')
const [cpus, setCpus] = useState('')
const [restart, setRestart] = useState(String(info?.restart ?? ''))
const [err, setErr] = useState('')
const [busy, setBusy] = useState(false)
useEscape(onClose)
const save = async (e: FormEvent) => {
e.preventDefault(); setErr(''); setBusy(true)
try { await api.dockerLimits(c.name, { memory: mem, cpus, restart }); toast(`${c.name} updated`); onSaved() }
catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setBusy(false) }
}
return (
<div className="modal-back" onClick={onClose}>
<form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
<h3>Limits for {c.name}</h3>
<label className="muted">Memory — 512m, 2g, or 0 for no limit{curMem > 0 ? ` (now ${fmtBytes(curMem)})` : ' (no limit now)'}
<input className="mono" value={mem} onChange={(e) => setMem(e.target.value)} placeholder="leave blank to keep" spellCheck={false} />
</label>
<label className="muted">CPUs — 1.5 means one and a half cores, 0 for no limit
<input className="mono" value={cpus} onChange={(e) => setCpus(e.target.value)} placeholder="leave blank to keep" spellCheck={false} />
</label>
<label className="muted">Restart policy
<select value={restart} onChange={(e) => setRestart(e.target.value)}>
<option value="">leave as it is</option>
{['no', 'on-failure', 'unless-stopped', 'always'].map((p) => <option key={p} value={p}>{p}</option>)}
</select>
</label>
<div className="notice attn">
This changes the running container now, without recreating it. It does <b>not</b> edit the
compose file{c.stack ? <> — so deploying <b>{c.stack}</b> puts back whatever the file says</> : ''}.
Edit the stack to make it stick.
</div>
{err && <div className="error">{err}</div>}
<div className="svc-actions">
<button type="submit" className="btn on" disabled={busy || (!mem && !cpus && !restart)}>Apply</button>
<span className="spacer" />
<button type="button" className="btn" onClick={onClose}>Cancel</button>
</div>
</form>
</div>
)
}
function lineDiff(a: string, b: string): { s: ' ' | '+' | '-'; t: string }[] {
const x = a.split('\n'), y = b.split('\n'); const n = x.length, m = y.length
const L = Array.from({ length: n + 1 }, () => new Uint16Array(m + 1))
for (let i = n - 1; i >= 0; i--) for (let j = m - 1; j >= 0; j--) L[i][j] = x[i] === y[j] ? L[i + 1][j + 1] + 1 : Math.max(L[i + 1][j], L[i][j + 1])
const out: { s: ' ' | '+' | '-'; t: string }[] = []; let i = 0, j = 0
while (i < n && j < m) { if (x[i] === y[j]) { out.push({ s: ' ', t: x[i] }); i++; j++ } else if (L[i + 1][j] >= L[i][j + 1]) out.push({ s: '-', t: x[i++] }); else out.push({ s: '+', t: y[j++] }) }
while (i < n) out.push({ s: '-', t: x[i++] }); while (j < m) out.push({ s: '+', t: y[j++] })
return out
}
// What a Deploy would change: each file against what this console last deployed (or, before
// the first deploy from here, the copy the last save replaced). Only changed lines and a
// little around them; a .env shows which keys changed, never their values.
function DiffView({ files }: { files: StackDiff[] }) {
const changed = files.filter((f) => f.base && f.old !== f.new)
if (files.every((f) => !f.base)) return <div className="muted small">Nothing to compare with yet: this console hasn't deployed or saved this stack before.</div>
if (changed.length === 0) return <div className="muted small">No changes since {files.some((f) => f.base === 'deployed') ? 'the last deploy from here' : 'the last save'}: Deploy would only recreate what Docker itself thinks has changed.</div>
return <>{changed.map((f) => { const d = lineDiff(f.old, f.new); const keep = new Set<number>(); d.forEach((l, i) => { if (l.s !== ' ') for (let k = i - 2; k <= i + 2; k++) keep.add(k) })
return <div key={f.name}><div className="muted small">{f.name} · against {f.base === 'deployed' ? 'what was last deployed from here' : 'the copy the last save replaced'}</div>
<pre className="diff">{d.map((l, i) => (keep.has(i) ? <div key={i} className={l.s === '+' ? 'add' : l.s === '-' ? 'del' : ''}>{l.s} {l.t}</div> : keep.has(i - 1) ? <div key={i} className="muted"> ⋯</div> : null))}</pre></div> })}</>
}
// A .env is mostly secrets: keys shown, values hidden until asked for.
const maskEnv = (text: string) => text.split('\n').map((l) => { const i = l.indexOf('='); return i > 0 && !l.trim().startsWith('#') && l.length > i + 1 ? l.slice(0, i + 1) + '••••••••' : l }).join('\n')
// One stack: its compose file to edit, and the four things to do with it. Saving validates
// with compose itself and keeps the previous file beside it; deploying is a separate, asked-for act.
function StackDetail({ name, startTask, onBack, onChanged, onDuplicate }: { name: string; startTask?: string; onBack: () => void; onChanged: () => void; onDuplicate: (from: DupFrom) => void }) {
const [st, setSt] = useState<Stack | null>(null)
const [text, setText] = useState(''); const [orig, setOrig] = useState('')
const [err, setErr] = useState(''); const [task, setTask] = useState('')
const [file, setFile] = useState<number | 'env'>(0); const [sum, setSum] = useState('')
const [reveal, setReveal] = useState(false); const [diff, setDiff] = useState<StackDiff[] | null>(null)
const [upd, setUpd] = useState<{ images: ImageCheck[]; checked: number } | null>(null); const [checking, setChecking] = useState(false)
useEffect(() => { setUpd(null); setDiff(null); api.stackUpdates(name).then(setUpd).catch(() => {}) }, [name])
useEffect(() => { setReveal(false) }, [file, name])
const load = useCallback(async () => { try { const r = await api.stackFile(name, file); setSt(r.stack); setText(r.text); setOrig(r.text); setSum(r.sum); setErr('') } catch (e) { setErr(e instanceof Error ? e.message : 'failed') } }, [name, file])
useEffect(() => { setSt(null); setTask(startTask ?? ''); load() }, [load]) // eslint-disable-line react-hooks/exhaustive-deps
const dirty = text !== orig
useEffect(() => { setUnsaved(dirty ? `${name}'s compose file` : ''); return () => setUnsaved('') }, [dirty, name])
const save = async () => { try { await api.saveStack(name, file, text, sum); setOrig(text); setErr(''); setDiff(null); api.stackFile(name, file).then((r) => setSum(r.sum)).catch(() => {}); toast('saved — the running containers are unchanged until you deploy') } catch (e) { setErr(e instanceof Error ? e.message : 'failed') } }
const remove = async () => {
const typed = await askText(`Delete the stack "${name}"?\n\nIts containers are removed and its compose file${st?.has_env ? ' and .env' : ''} moved to .hamlab-trash beside it — moved, not erased. Volumes and bind-mounted data are not touched.\n\nType its name to confirm.`, '', 'Delete')
if (typed !== name) { if (typed !== null) toast('that is not its name — nothing was deleted', 'info'); return }
try { const r = await api.deleteStack(name); toast(`${name} deleted — its files are in ${r.trash}`); setUnsaved(''); onChanged(); onBack() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
}
const duplicate = async () => { if (!st) return; try { onDuplicate({ name, text: file === 0 ? orig : (await api.stackFile(name, 0)).text, hasEnv: !!st.has_env, files: st.host_files?.length ?? 1 }) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } }
const run = (a: 'deploy' | 'update' | 'recreate' | 'restart' | 'stop' | 'down') => async () => { try { setTask((await api.stackAction(name, a)).id) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } }
if (err && !st) return <div className="svc-detail"><CloseBtn onClick={onBack} /><button className="btn back" onClick={onBack}>‹ Back</button><div className="notice">{err}</div></div>
if (!st) return <div className="svc-detail"><Loading label="Reading the compose file…" /></div>
return (
<div className="svc-detail">
<CloseBtn onClick={onBack} />
<div className="svc-head"><button className="btn back" onClick={onBack}>‹ Back</button><span className={'sdot ' + (st.containers.length === 0 ? '' : st.running === st.containers.length ? 'ok' : st.running ? 'warn' : 'bad')} /><span className="dname">{st.name}</span><span className="muted small">{st.running}/{st.containers.length} running</span></div>
<div className="muted small mono" style={{ overflowWrap: 'anywhere' }}>{file === 'env' ? (st.host_file.replace(/[^/]+$/, '') + '.env') : (st.host_files?.[file] ?? st.host_file)}{st.held_by && <span className="muted"> · kept on <b>{st.held_by}</b>, deployed to this machine over {st.remote}</span>}{st.via && <span className="muted"> · compose runs inside <b>{st.via}</b>, as it was deployed</span>}{st.has_env && ' · has a .env beside it'}</div>
{((st.host_files?.length ?? 0) > 1 || st.has_env) && <div className="chips" title="this stack is deployed from several files, in this order; Deploy uses all of them">{st.host_files!.map((f, i) => <button key={f} className={'chip' + (i === file ? ' on' : '')} onClick={async () => { if (dirty && !(await askYes('Discard unsaved changes to this file?', 'Discard'))) return; setFile(i) }}>{f.split('/').pop()}</button>)}{st.has_env && <button className={'chip' + (file === 'env' ? ' on' : '')} title="the .env beside the compose file: variables compose substitutes" onClick={async () => { if (dirty && !(await askYes('Discard unsaved changes to this file?', 'Discard'))) return; setFile('env') }}>.env</button>}</div>}
{file === 'env' && !reveal ? <div className="env-masked"><pre className="editor stack-editor">{maskEnv(text)}</pre><button className="btn" onClick={() => setReveal(true)}>Reveal and edit</button></div> :
<textarea className="editor stack-editor" value={text} onChange={(e) => setText(e.target.value)} spellCheck={false} autoCapitalize="off" autoCorrect="off" />}
{err && <div className="error" style={{ whiteSpace: 'pre-wrap' }}>{err}{/changed on disk/.test(err) && <> <button className="link" onClick={async () => { if (await askYes('Reload the file from disk? Your text in the box is replaced.', 'Reload')) load() }}>reload it</button></>}</div>}
<div className="svc-actions">
<Btn className="btn on" disabled={!dirty} onClick={save}>Save</Btn>
{dirty && <button className="btn" onClick={() => { setText(orig); setErr('') }}>Revert</button>}
<span className="spacer" />
<Btn className="btn" disabled={dirty} title={dirty ? 'save first' : 'recreate what changed: docker compose up -d'} confirm={`Deploy ${st.name}? Containers whose definition changed are recreated (a few seconds down each).`} onClick={run('deploy')}>Deploy</Btn>
{st.containers.length > 0 && <Btn className="btn" disabled={dirty} title="pull newer images, then deploy" confirm={`Pull newer images for ${st.name} and redeploy? Anything with a new image is recreated.`} onClick={run('update')}>Update</Btn>}
{st.containers.length > 0 && <Btn className="btn" confirm={`Restart every container in ${st.name}?`} onClick={run('restart')}>Restart</Btn>}
{st.containers.length > 0 && <Btn className="btn" disabled={dirty} title="build the containers again from scratch: docker compose up -d --force-recreate" confirm={`Recreate every container in ${st.name}?\n\nThis is the way out of a container Docker refuses to start — one pinned to a network that no longer exists, say, which answers "network … not found" to an ordinary Deploy. Nothing in a volume or a bind mount is touched.`} onClick={run('recreate')}>Recreate</Btn>}
{st.running > 0 && <Btn className="btn danger" confirm={`Stop every container in ${st.name}? They stay stopped until deployed or started again.`} onClick={run('stop')}>Stop</Btn>}
</div>
<div className="svc-actions">
<span className="muted small">{st.containers.length === 0 ? 'Not deployed: no containers exist for this stack.' : ''}</span><span className="spacer" />
<button className="btn" disabled={dirty} title={dirty ? 'save first' : 'a new stack that starts as a copy of this one'} onClick={duplicate}>Duplicate…</button>
{st.containers.length > 0 && <Btn className="btn danger" title="docker compose down" confirm={`Take ${st.name} down? Its ${st.containers.length} container${st.containers.length === 1 ? ' is' : 's are'} removed (not just stopped). The compose file, volumes and bind-mounted data stay, and Deploy brings it back.`} onClick={run('down')}>Down</Btn>}
<button className="btn danger" onClick={remove}>Delete…</button>
</div>
{task && <TaskOutput id={task} onDone={() => { onChanged(); load(); setDiff(null); api.stackUpdates(name).then(setUpd).catch(() => {}) }} />}
<div className="stack-extra">
<div className="sub-head"><span className="k">What Deploy would change</span><span className="spacer" /><button className="link" onClick={async () => { try { setDiff((await api.stackDiff(name)).files) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } }}>{diff ? 'refresh' : 'show'}</button></div>
{diff && <DiffView files={diff} />}
<div className="sub-head"><span className="k">Images</span><span className="spacer" /><span className="muted small">{upd?.checked ? `checked ${new Date(upd.checked).toLocaleString()}` : 'not checked'}</span> <Btn className="link" busy={checking} onClick={async () => { setChecking(true); try { setUpd(await api.stackUpdates(name, true)) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } finally { setChecking(false) } }}>check for updates</Btn></div>
{(st.images ?? []).map((img) => { const c = upd?.images.find((x) => x.image === img); return <div className="img-row" key={img}><span className="mono small">{img}</span><span className="spacer" />{c ? <span className={'badge' + (c.state === 'update' ? ' attn' : '')} title={c.detail}>{c.state === 'update' ? 'update available' : c.state === 'current' ? 'up to date' : 'unknown'}</span> : <span className="muted small">—</span>}</div> })}
</div>
<div className="muted small">{st.containers.join(' · ')}</div>
</div>
)
}
export default function Docker() {
const [d, setD] = useState<DockerInfo | null>(null)
const [stacks, setStacks] = useState<Stack[] | null>(null)
const [err, setErr] = useState('')
const [tab, setTab] = useState<Tab>('containers')
const [filter, setFilter] = useState('')
const [sel, setSel] = useHashSel('docker')
const [making, setMaking] = useState<'' | 'stack' | 'container'>('')
const [dup, setDup] = useState<DupFrom | null>(null)
const [startTask, setStartTask] = useState<{ name: string; id: string } | null>(null)
const [pull, setPull] = useState(''); const [pullTask, setPullTask] = useState('')
const rate = useRefreshMs()
const load = useCallback(async () => { try { setD(await api.docker()); setErr('') } catch (e) { setErr(e instanceof Error ? e.message : 'failed') } api.stacks().then((r) => setStacks(r.stacks)).catch(() => setStacks([])) }, [])
useEffect(() => { load(); const t = setInterval(() => { if (document.visibilityState === 'visible') load() }, Math.max(5000, rate)); return () => clearInterval(t) }, [load, rate])
const live = useSensorStream<{ c: Record<string, Sensor> }>({ c: 'container.*' }, { everyMs: Math.max(2000, rate) })
const sensors = live.data?.c ?? {}
const byStack = useMemo(() => {
const f = filter.trim().toLowerCase(); const out = new Map<string, DockerContainer[]>()
for (const c of d?.containers ?? []) { if (f && !`${c.name} ${c.image} ${c.stack}`.toLowerCase().includes(f)) continue; const k = c.stack || 'started by hand'; out.set(k, [...(out.get(k) ?? []), c]) }
return out
}, [d, filter])
// what a new stack could join, and what it could collide with
const nets = useMemo(() => (d?.networks ?? []).map((n) => String(n.Name)).filter((n) => !['bridge', 'host', 'none'].includes(n)).sort(), [d])
const taken = useMemo(() => { const ports: Record<string, string> = {}; for (const c of d?.containers ?? []) for (const m of (c.ports ?? '').matchAll(/:(\d+)->/g)) ports[m[1]] = c.name; return { names: (d?.containers ?? []).map((c) => c.name), ports } }, [d])
// What the registry last said about each image, so a container can be marked without
// asking again. Only ever read from the cache here — checking is a deliberate act.
const [upd, setUpd] = useState<MachineUpdates | null>(null)
useEffect(() => { api.machineUpdates().then(setUpd).catch(() => setUpd(null)) }, [])
const imgState = useMemo(() => Object.fromEntries((upd?.images ?? []).map((i) => [i.image, i.state])), [upd])
const behind = (upd?.images ?? []).filter((i) => i.state === 'update').length
if (!d) return <div className="dash">{err ? <div className="notice">{err === 'not found' ? "This machine's Ham Lab is older than the Docker tab. Update it from Machines." : err}</div> : <Loading label="Asking Docker…" />}</div>
if (!d.available) return <div className="dash"><div className="notice">{d.message}</div></div>
const selStack = sel.startsWith('stack:') ? sel.slice(6) : ''
const cur = selStack ? undefined : d.containers.find((c) => c.name === sel)
const shownTab: Tab = selStack ? 'stacks' : tab
const up = d.containers.filter((c) => c.state === 'running').length
const sid = (c: DockerContainer, k: string) => sensors[`container.${c.name.toLowerCase().replace(/[^a-z0-9-]+/g, '_').replace(/^_|_$/g, '')}.${k}`]
const imageCols: [string, string][] = [['Repository', 'Repository'], ['Tag', 'Tag'], ['Size', 'Size'], ['CreatedSince', 'Created']]
return (
<div className={'svc docker' + (((cur || sel) && shownTab === 'containers') || selStack ? ' has-sel' : '') + (shownTab === 'updates' ? ' wide' : '')}>
<div className="svc-list">
<div className="svc-tools">
<div className="chips">{(['containers', 'stacks', 'updates', 'images', 'volumes', 'networks'] as Tab[]).map((t) => <button key={t} className={'chip' + (shownTab === t ? ' on' : '')} onClick={() => { setTab(t); if (sel) setSel('') }}>{t}{t === 'containers' ? ` · ${up}/${d.containers.length}` : t === 'stacks' ? ` · ${stacks?.length ?? '…'}` : t === 'updates' ? (behind > 0 ? ` · ${behind}` : '') : ` · ${d[t].length}`}</button>)}</div>
{(shownTab === 'containers' || shownTab === 'stacks') && <div className="svc-actions"><button className="btn" onClick={() => setMaking(shownTab === 'stacks' ? 'stack' : 'container')}>{shownTab === 'stacks' ? '+ New stack' : '+ New container'}</button></div>}
{shownTab === 'images' && <form className="chan-add" onSubmit={async (e) => { e.preventDefault(); try { setPullTask((await api.pullImage(pull.trim())).id) } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') } }}>
<input className="mono" value={pull} onChange={(e) => setPull(e.target.value)} placeholder="pull an image — nginx:alpine, ghcr.io/owner/name:tag" spellCheck={false} autoCapitalize="off" autoCorrect="off" />
<button className="btn on" disabled={!pull.trim()}>Pull</button>
{(() => { const unused = d.images.filter((im) => !im.in_use).length; return unused > 0 && <Btn className="btn danger" title="remove every image no container uses" confirm={`Remove all ${unused} image${unused === 1 ? '' : 's'} that no container uses? They can be pulled again, but a stack that is down will have to download its images before it next starts.`} onClick={async () => { try { toast((await api.pruneImages(true)).result); load() } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') } }}>Remove unused · {unused}</Btn> })()}
<Btn className="btn" title="remove dangling images: layers no tag points at any more" confirm="Remove dangling images? Only layers that no tag points at any more — nothing a container or a tag uses." onClick={async () => { try { toast((await api.pruneImages()).result); load() } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') } }}>Prune</Btn>
</form>}
{shownTab === 'volumes' && (() => { const n = d.volumes.filter((v) => v.anonymous && !((v.used_by as string[] | undefined) ?? []).length).length; return <div className="svc-actions"><span className="muted small">Deleting a volume deletes its data. Only one nothing uses can be deleted.</span><span className="spacer" />{n > 0 && <Btn className="btn danger" title="the nameless volumes an image's VOLUME line leaves behind each time its container is recreated" confirm={`Delete ${n} unused anonymous volume${n === 1 ? '' : 's'} and what is in them? Named volumes are not touched.`} onClick={async () => { try { toast((await api.pruneVolumes()).result); load() } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') } }}>Prune anonymous · {n}</Btn>}</div> })()}
{shownTab === 'networks' && (() => { const n = d.networks.filter((x) => !x.predefined && !((x.used_by as string[] | undefined) ?? []).length).length; return <div className="svc-actions"><span className="muted small">A network holds no data. Only one nothing is attached to can be removed.</span><span className="spacer" />{n > 0 && <Btn className="btn" title="remove every network no container is attached to" confirm={`Remove ${n} network${n === 1 ? '' : 's'} nothing is attached to? A stack that is only down counts as unattached — its network comes back on the next deploy, with a new subnet.`} onClick={async () => { try { toast((await api.pruneNetworks()).result || 'nothing to remove'); load() } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') } }}>Prune · {n}</Btn>}</div> })()}
{shownTab === 'images' && pullTask && <TaskOutput id={pullTask} onDone={() => load()} />}
{shownTab === 'containers' && <input value={filter} onChange={(e) => setFilter(e.target.value)} placeholder="filter containers, images, stacks" />}
</div>
<div className="svc-rows">
{shownTab === 'stacks' && (stacks ?? []).map((st) => (
<div key={st.name} className={'svc-row' + (st.name === selStack ? ' active' : '')} onClick={() => st.editable ? setSel('stack:' + st.name) : toast(st.note ?? 'not editable here', 'info')}>
<span className={'sdot ' + (st.containers.length === 0 ? '' : st.running === st.containers.length ? 'ok' : st.running ? 'warn' : 'bad')} />
<div className="svc-name"><div className="n">{st.name}</div><div className="d">{st.editable ? (st.held_by ? `on ${st.held_by}: ` : '') + st.host_file : st.note}</div></div>
<div className="svc-state">{st.containers.length === 0 ? 'not deployed' : `${st.running}/${st.containers.length}`}{!st.editable && <><br />read-only</>}</div>
</div>
))}
{shownTab === 'stacks' && stacks?.length === 0 && <div className="muted small" style={{ padding: 12 }}>No compose stacks here.</div>}
{shownTab === 'containers' && [...byStack].map(([stack, list]) => (
<div key={stack}>
<h3 className="stack-head">{stack} <span className="muted">· {list.filter((c) => c.state === 'running').length}/{list.length}</span></h3>
{list.map((c) => (
<div key={c.id} className={'svc-row' + (c.name === sel ? ' active' : '')} onClick={() => setSel(c.name)}>
<span className={'sdot ' + dot(c)} />
<div className="svc-name"><div className="n">{c.name}<UpdateDot state={imgState[c.image]} /></div><div className="d">{c.image}</div></div>
<div className="svc-state">{c.state === 'running' ? <>{fmtValue(sid(c, 'mem')?.value ?? c.mem, 'B')}<br />{fmtValue(sid(c, 'cpu')?.value, '%')}</> : c.state}</div>
</div>
))}
</div>
))}
{shownTab === 'containers' && byStack.size === 0 && <div className="muted small" style={{ padding: 12 }}>No container matches.</div>}
{shownTab === 'volumes' && (
<table className="dtable"><thead><tr><th>Volume</th><th>Used by</th><th /></tr></thead>
<tbody>{d.volumes.map((row) => { const name = String(row.Name), users = (row.used_by as string[] | undefined) ?? [], anon = !!row.anonymous
const remove = async () => {
if (anon) { if (!(await askYes(`Delete this anonymous volume and everything in it?\n\n${name}\n\nNothing uses it. It can't be brought back.`, 'Delete'))) return }
else { const typed = await askText(`Delete the volume "${name}" and everything in it?\n\nNothing uses it right now — but if it belongs to a stack that is only down, this is that stack's data. It can't be brought back.\n\nType its name to confirm.`, '', 'Delete'); if (typed !== name) { if (typed !== null) toast('that is not its name — nothing was deleted', 'info'); return } }
try { await api.removeVolume(name); toast('volume deleted'); load() } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') }
}
return <tr key={name}><td className="mono small" title={String(row.Mountpoint ?? '')}>{anon ? <>{name.slice(0, 12)}… <span className="muted">anonymous</span></> : name}</td><td className="small">{users.length ? users.join(', ') : <span className="muted">nothing</span>}</td><td className="row-end">{users.length ? <span className="badge">in use</span> : <button className="icon-btn" title="delete this volume and its data" onClick={remove}>✕</button>}</td></tr> })}
{d.volumes.length === 0 && <tr><td colSpan={3} className="muted small">No volumes.</td></tr>}</tbody></table>
)}
{shownTab === 'networks' && (
<table className="dtable"><thead><tr><th>Network</th><th>Driver</th><th>Used by</th><th /></tr></thead>
<tbody>{d.networks.map((row, i) => { const name = String(row.Name), users = (row.used_by as string[] | undefined) ?? [], own = !!row.predefined
return <tr key={i}><td className="mono small">{name}{own && <span className="muted"> · Docker's own</span>}</td><td className="mono small">{String(row.Driver ?? '')}</td><td className="small">{users.length ? users.join(', ') : <span className="muted">nothing</span>}</td><td className="row-end">{users.length ? <span className="badge" title="a container is attached to this network">in use</span> : own ? null : <Btn className="icon-btn" title="remove this network" confirm={`Remove the network ${name}? Nothing is attached to it. A stack that wants it back gets it recreated on its next deploy, with a new subnet.`} onClick={async () => { try { await api.removeNetwork(name); toast(`${name} removed`); load() } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') } }}>✕</Btn>}</td></tr> })}
{d.networks.length === 0 && <tr><td colSpan={4} className="muted small">No networks.</td></tr>}</tbody></table>
)}
{shownTab === 'images' && (
<table className="dtable"><thead><tr>{imageCols.map(([, h]) => <th key={h}>{h}</th>)}<th /></tr></thead>
<tbody>{d.images.map((row, i) => <tr key={i}>{imageCols.map(([k]) => <td key={k} className="mono small">{String(row[k] ?? '')}</td>)}<td className="row-end">{row.in_use ? <span className="badge" title="a container uses this image">in use</span> : <Btn className="icon-btn" title="remove this image" confirm={`Remove the image ${row.Repository}:${row.Tag}? Docker refuses if a container still uses it.`} onClick={async () => { const ref = row.Tag && row.Tag !== '<none>' ? `${row.Repository}:${row.Tag}` : String(row.ID); try { await api.removeImage(ref); toast(`${ref} removed`); load() } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') } }}>✕</Btn>}</td></tr>)}</tbody></table>
)}
</div>
</div>
{shownTab === 'containers' && !cur && sel && !selStack && <div className="svc-detail"><CloseBtn onClick={() => setSel('')} /><button className="btn back" onClick={() => setSel('')}>‹ Back</button><div className="notice">There is no container called <b className="mono">{sel}</b> on this machine. It may have been removed, renamed or recreated under another name. <button className="link" onClick={() => setSel('')}>Show them all</button></div></div>}
{shownTab === 'containers' && (cur ? <Detail c={cur} live={sensors} onBack={() => setSel('')} onChanged={load} /> : (sel && !selStack ? null : <div className="svc-empty muted">Select a container.</div>))}
{making && <StackNew key={'new:' + (dup?.name ?? making)} mode={making} from={dup ?? undefined} networks={nets} taken={taken} onClose={() => { setMaking(''); setDup(null) }} onCreated={async (name, deploy) => { setMaking(''); setDup(null); setTab('stacks'); if (deploy) { try { setStartTask({ name, id: (await api.stackAction(name, 'deploy')).id }) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } } else setStartTask(null); await load(); setSel('stack:' + name) }} />}
{shownTab === 'updates' && <Updates upd={upd} onChecked={setUpd} />}
{shownTab === 'stacks' && (selStack ? <StackDetail key={selStack + (startTask?.name === selStack ? startTask.id : '')} name={selStack} startTask={startTask?.name === selStack ? startTask.id : undefined} onBack={() => setSel('')} onChanged={load} onDuplicate={(f) => { setDup(f); setMaking('stack') }} /> : <div className="svc-empty muted">Select a stack to edit its compose file.</div>)}
</div>
)
}
// A container's image against the registry, as a single mark: nothing when nobody has
// asked, so an uncoloured list means "not checked" rather than "all current".
function UpdateDot({ state }: { state?: string }) {
if (!state || state === 'current') return null
const known = state === 'update'
return <span className={'updot' + (known ? ' behind' : ' unknown')} title={known ? 'a newer image has been published' : 'the registry could not say'}>{known ? '↑' : '?'}</span>
}
// What is out of date here, and the schedules that keep it from happening again.
function Updates({ upd, onChecked }: { upd: MachineUpdates | null; onChecked: (u: MachineUpdates) => void }) {
const [busy, setBusy] = useState(false)
const [jobs, setJobs] = useState<UpdateJobRow[] | null>(null)
const [file, setFile] = useState('')
const [editing, setEditing] = useState<Partial<UpdateJob> | null>(null)
const [stacks, setStacks] = useState<string[]>([])
const loadJobs = useCallback(async () => {
try { const r = await api.updateJobs(); setJobs(r.jobs); setFile(r.file) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err'); setJobs([]) }
}, [])
useEffect(() => { loadJobs(); api.stacks().then((r) => setStacks(r.stacks.map((s) => s.name))).catch(() => {}) }, [loadJobs])
const check = async () => {
setBusy(true)
try { onChecked(await api.machineUpdates(true)); toast('checked') } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } finally { setBusy(false) }
}
const run = async (j: UpdateJob) => {
if (!await askYes(`Run “${j.name}” now? It pulls ${j.stack}'s images${j.apply ? ' and deploys them, which restarts its containers' : ', without deploying them'}.`, 'Run now')) return
try { await api.runUpdateJob(j.id); toast(`${j.name}: done`); loadJobs() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
}
const remove = async (j: UpdateJob) => {
if (!await askYes(`Remove the schedule “${j.name}”? Nothing is updated or rolled back — the schedule simply stops.`, 'Remove')) return
try { await api.deleteUpdateJob(j.id); toast('removed'); loadJobs() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
}
const imgs = upd?.images ?? []
const behind = imgs.filter((i) => i.state === 'update')
// Updating is per stack — that is what compose can do — so an image row acts on whichever
// stacks use it, and a queue runs them one at a time rather than recreating half the
// machine at once.
const [queue, setQueue] = useState<string[]>([])
const [running, setRunning] = useState<{ stack: string; id: string } | null>(null)
const [done, setDone] = useState<string[]>([])
const stacksFor = useCallback((img: string) => upd?.stacks?.[img] ?? [], [upd])
const startNext = useCallback(async (rest: string[]) => {
const next = rest[0]
if (!next) { setRunning(null); setQueue([]); return }
setQueue(rest.slice(1))
try { setRunning({ stack: next, id: (await api.stackAction(next, 'update')).id }) }
catch (e) { toast(`${next}: ${e instanceof Error ? e.message : 'failed'}`, 'err'); setRunning(null); startNext(rest.slice(1)) }
}, [])
const updateStacks = async (names: string[], what: string) => {
const uniq = [...new Set(names)]
if (uniq.length === 0) { toast('nothing here names a stack this console can deploy', 'info'); return }
if (!await askYes(`Update ${what}?\n\nThis pulls the newer image${uniq.length > 1 ? 's' : ''} and recreates ${uniq.length === 1 ? `the ${uniq[0]} stack` : `these stacks, one at a time: ${uniq.join(', ')}`}. Their containers restart.`, 'Update')) return
setDone([]); startNext(uniq)
}
const behindStacks = [...new Set(behind.flatMap((i) => stacksFor(i.image)))]
return (
<div className="svc-detail">
<div className="dash-bar"><h2 className="section">What is out of date</h2><span className="spacer" />
<span className="muted small">{upd?.checked ? `checked ${new Date(upd.checked).toLocaleString()}` : 'not checked yet'}</span>
{behindStacks.length > 0 && <Btn className="btn on" disabled={!!running} onClick={() => updateStacks(behindStacks, `everything out of date — ${behind.length} image${behind.length === 1 ? '' : 's'} across ${behindStacks.length} stack${behindStacks.length === 1 ? '' : 's'}`)}>Update all · {behind.length}</Btn>}
<Btn className="btn" busy={busy} onClick={check}>Check now</Btn>
</div>
<div className="muted small">Each check is one request per image, which registries count — so it happens when asked, and the answer is kept for six hours.</div>
{imgs.length === 0 && <div className="notice" style={{ marginTop: 8 }}>Nothing checked yet on this machine.</div>}
{imgs.length > 0 && (
<section className="card" style={{ marginTop: 8 }}>
{behind.length === 0 && <div className="muted">Every image that could be checked is current.</div>}
<div className="upd-list">
{imgs.filter((i) => i.state !== 'current').map((i) => (
<div className="row" key={i.image}>
<span className="k mono small">{i.image}</span>
<span className={i.state === 'update' ? 'warn-text' : 'muted small'}>{done.some((d) => stacksFor(i.image).includes(d)) ? <span className="good">updated just now</span> : i.state === 'update' ? 'a newer image is published' : i.detail || 'unknown'}</span>
<span className="muted small">{stacksFor(i.image).join(' ')}</span>
{i.state === 'update' && !done.some((d) => stacksFor(i.image).includes(d)) && (
<Btn className="btn" disabled={!!running || stacksFor(i.image).length === 0} title={stacksFor(i.image).length ? `pull it and recreate ${stacksFor(i.image).join(', ')}` : 'no stack here uses this image'}
onClick={() => updateStacks(stacksFor(i.image), `${i.image}`)}>Update</Btn>
)}
</div>
))}
</div>
{behind.length === 0 && imgs.some((i) => i.state === 'current') && <div className="muted small" style={{ marginTop: 6 }}>{imgs.filter((i) => i.state === 'current').length} current.</div>}
{running && (
<div style={{ marginTop: 8 }}>
<div className="sub-head"><span className="dname">Updating {running.stack}</span><span className="spacer" /><span className="muted small">{queue.length > 0 ? `${queue.length} to follow` : 'last one'}</span></div>
<TaskOutput id={running.id} onDone={(t) => {
if (t.exit_code === 0 && !t.error) setDone((d) => [...d, running.stack])
else toast(`${running.stack}: update failed`, 'err')
startNext(queue)
}} />
</div>
)}
</section>
)}
<div className="dash-bar" style={{ marginTop: 14 }}><h2 className="section">Schedules</h2><span className="spacer" />
<Btn className="btn on" onClick={() => setEditing({ cron: '0 3 * * *', apply: true, enabled: true })}>+ New schedule</Btn>
</div>
<div className="muted small">Written to <span className="mono">{file || '/etc/cron.d/hamlab-docker'}</span>, which Ham Lab owns — so they keep running whether or not this console is up.</div>
{jobs === null && <Loading label="Reading…" />}
{jobs?.length === 0 && <div className="notice" style={{ marginTop: 8 }}>No schedule yet. Nothing on this machine updates itself.</div>}
<div className="cards" style={{ marginTop: 8 }}>
{(jobs ?? []).map(({ job, next, last }) => (
<section className="card" key={job.id}>
<div className="sub-head">
<span className={'dot ' + (last?.error ? 'bad' : job.enabled ? 'ok' : '')} />
<span className="dname">{job.name}</span>
{!job.enabled && <span className="badge">paused</span>}
<span className="spacer" />
<span className="muted small">{job.enabled && next ? new Date(next).toLocaleString() : 'not scheduled'}</span>
</div>
<div className="props" style={{ marginTop: 6 }}>
<div><span className="k">Stack</span><span>{job.stack}{job.services?.length ? ` · ${job.services.join(' ')}` : ''}</span></div>
<div><span className="k">Runs</span><span>{describeCron(job.cron) || <span className="mono small">{job.cron}</span>}{describeCron(job.cron) && <span className="muted mono small"> · {job.cron}</span>}</span></div>
<div><span className="k">Does</span><span>{job.apply ? 'pulls and deploys' : 'pulls only — you deploy'}</span></div>
<div><span className="k">Last run</span><span>{last?.at ? new Date(last.at).toLocaleString() : 'never'}</span></div>
</div>
{last?.error && <div className="error" style={{ marginTop: 6 }}>{last.error}</div>}
<div className="svc-actions">
<Btn className="btn on" onClick={() => run(job)}>Run now</Btn>
<button className="btn" onClick={() => setEditing(job)}>Edit</button>
<Btn className="btn" onClick={async () => { await api.saveUpdateJob({ ...job, enabled: !job.enabled }); loadJobs() }}>{job.enabled ? 'Pause' : 'Resume'}</Btn>
<Btn className="btn danger" onClick={() => remove(job)}>Remove</Btn>
</div>
</section>
))}
</div>
{editing && <JobDialog j={editing} stacks={stacks} onClose={() => setEditing(null)} onSaved={() => { setEditing(null); loadJobs() }} />}
</div>
)
}
function JobDialog({ j, stacks, onClose, onSaved }: { j: Partial<UpdateJob>; stacks: string[]; onClose: () => void; onSaved: () => void }) {
const [f, setF] = useState<Partial<UpdateJob>>(j)
const [svc, setSvc] = useState((j.services ?? []).join(' '))
const [err, setErr] = useState('')
const [busy, setBusy] = useState(false)
useEscape(onClose)
const set = (k: keyof UpdateJob, v: unknown) => setF((x) => ({ ...x, [k]: v }))
const save = async (e: FormEvent) => {
e.preventDefault(); setErr(''); setBusy(true)
try { await api.saveUpdateJob({ ...f, services: svc.split(/\s+/).filter(Boolean) }); onSaved() }
catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setBusy(false) }
}
return (
<div className="modal-back" onClick={onClose}>
<form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
<h3>{f.id ? 'Edit schedule' : 'Keep a stack up to date'}</h3>
<label className="muted">Stack
<select value={f.stack ?? ''} onChange={(e) => set('stack', e.target.value)} required>
<option value="">choose…</option>
{stacks.map((s) => <option key={s} value={s}>{s}</option>)}
</select>
</label>
<label className="muted">Name — optional
<input value={f.name ?? ''} onChange={(e) => set('name', e.target.value)} placeholder={f.stack ?? 'nightly'} maxLength={64} />
</label>
<label className="muted">Only these services — optional, space separated; blank means the whole stack
<input className="mono" value={svc} onChange={(e) => setSvc(e.target.value)} placeholder="all of them" spellCheck={false} />
</label>
<label className="muted">When — a cron expression
<input className="mono" value={f.cron ?? ''} onChange={(e) => set('cron', e.target.value)} placeholder="0 3 * * *" spellCheck={false} required />
</label>
<label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
<input type="checkbox" checked={!!f.apply} onChange={(e) => set('apply', e.target.checked)} style={{ width: 'auto' }} />
Deploy the new image, not just fetch it
</label>
<label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
<input type="checkbox" checked={!!f.enabled} onChange={(e) => set('enabled', e.target.checked)} style={{ width: 'auto' }} />
Enabled
</label>
<div className="notice">
{f.apply
? 'Deploying restarts the stack’s containers. On something that carries DNS or ingress, pick an hour you are asleep.'
: 'Pulling only means the new image is on the machine and the list shows it as available — nothing restarts until you deploy.'}
</div>
{err && <div className="error">{err}</div>}
<div className="svc-actions">
<button type="submit" className="btn on" disabled={busy || !f.stack || !f.cron}>Save</button>
<span className="spacer" />
<button type="button" className="btn" onClick={onClose}>Cancel</button>
</div>
</form>
</div>
)
}
web/src/Embed.tsx
+32
View File
@@ -0,0 +1,32 @@
// One widget, alone, for an iframe in someone else's dashboard (Glance, Home Assistant…):
// /embed/widget?type=gauge&sensor=cpu.usage&token=<read-only token>
// &machine=grid &title=… &theme=dark|light|cyberpunk|retro &every=<ms> &bare=1
// It stays live over the sensor stream. The token must be a read-only sensor token; a
// full-control token is refused in a URL by design.
import { useSensorStream, Source } from './sensors'
import { Widget, WidgetData, WidgetSpec, WidgetType, WIDGET_TYPES } from './widgets'
import { applyTheme } from './themes'
const p = new URLSearchParams(location.search)
if (p.get('theme')) applyTheme(p.get('theme')!)
if (p.get('bare') === '1') document.body.style.background = 'transparent' // sit on the host dashboard's own card
export default function Embed() {
const sensor = p.get('sensor') ?? ''
const asked = p.get('type') as WidgetType | null
const type: WidgetType = /[*?[]/.test(sensor) ? 'list' : asked && WIDGET_TYPES.some((t) => t.id === asked && !t.many) ? asked : 'stat'
const spec: WidgetSpec = { id: 'w', type, sensor, title: p.get('title') ?? undefined }
const src: Source = { machine: p.get('machine') || null, token: p.get('token') ?? undefined }
const every = Number(p.get('every')) || undefined
const live = useSensorStream<{ w: WidgetData }>(sensor ? { w: sensor } : null, { src, everyMs: every })
return (
<div className={'embed' + (p.get('bare') === '1' ? ' bare' : '')}>
{!sensor ? <div className="error">embed needs ?sensor=</div> : (
<div className={'tile w-' + type}>
<Widget spec={spec} data={live.data ? live.data.w : undefined} src={src} />
{live.error && !live.data && <div className="sub">{src.token ? live.error : 'no token in the URL (&token=…)'}</div>}
</div>
)}
</div>
)
}
web/src/Files.tsx
+254
View File
@@ -0,0 +1,254 @@
import { DragEvent, useCallback, useEffect, useRef, useState } from 'react'
import { api, FileEntry, FileInfo } from './api'
import { fmtBytes } from './fmt'
import { Btn, Loading, Spinner, askText, askYes, setUnsaved, toast, useEscape } from './ui'
import { useHashSel } from './hashsel'
const join = (dir: string, name: string) => (dir === '/' ? '/' + name : dir + '/' + name)
const fmtTime = (ms: number) => { const d = new Date(ms); const now = new Date(); return d.toDateString() === now.toDateString() ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) : d.toLocaleDateString(undefined, { year: d.getFullYear() === now.getFullYear() ? undefined : 'numeric', month: 'short', day: '2-digit' }) }
const isImage = (n: string) => /\.(png|jpe?g|gif|webp|svg|bmp|ico|avif)$/i.test(n)
const shellQuote = (s: string) => "'" + s.replace(/'/g, "'\\''") + "'"
const MAX_EDIT = 2 * 1024 * 1024
function icon(e: FileEntry): string {
if (e.type === 'dir') return e.target ? '🔗' : '📁'
if (e.type === 'link') return e.broken ? '⛓️‍💥' : '🔗'
if (e.type === 'other') return '⚙️'
if (isImage(e.name)) return '🖼️'
if (/\.(zip|gz|bz2|xz|zst|tar|tgz|7z|rar)$/i.test(e.name)) return '🗜️'
if (/\.(sh|bash|zsh|py|go|js|ts|tsx|rb|pl|php|rs|c|h|cpp|java|lua)$/i.test(e.name) || e.mode.includes('x')) return '📜'
if (/\.(log|txt|md|json|ya?ml|toml|ini|conf|cfg|env|service|timer)$/i.test(e.name)) return '📄'
return '📄'
}
type Sort = 'name' | 'size' | 'mtime'
// ---- editor / viewer ----
function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolean; onClose: () => void; onChanged: () => void }) {
const [info, setInfo] = useState<FileInfo | null>(null)
const [text, setText] = useState<string | null>(null)
const [orig, setOrig] = useState('')
const [err, setErr] = useState('')
const [saving, setSaving] = useState(false)
const name = path.split('/').pop() ?? path
const dirty = text !== null && text !== orig
useEffect(() => {
setInfo(null); setText(null); setErr('')
let stop = false
;(async () => {
try {
const i = await api.fileStat(path, sudo)
if (stop) return
setInfo(i)
if (i.text && i.size <= MAX_EDIT && !isImage(name)) {
const t = await api.fileRead(path, sudo)
if (stop) return
setText(t); setOrig(t)
}
} catch (e) { if (!stop) setErr(e instanceof Error ? e.message : 'failed') }
})()
return () => { stop = true }
}, [path, sudo, name])
const save = useCallback(async () => {
if (text === null || !dirty) return
setSaving(true)
try { await api.fileWrite(path, text, sudo); setOrig(text); toast(`${name} saved`); onChanged() }
catch (e) { toast(`save failed — ${e instanceof Error ? e.message : 'failed'}`, 'err') }
finally { setSaving(false) }
}, [text, dirty, path, sudo, name, onChanged])
useEffect(() => {
const h = (e: KeyboardEvent) => { if ((e.ctrlKey || e.metaKey) && e.key === 's') { e.preventDefault(); save() } }
window.addEventListener('keydown', h)
return () => window.removeEventListener('keydown', h)
}, [save])
const close = async () => { if (dirty && !await askYes('Discard unsaved changes?', 'Discard')) return; setUnsaved(''); onClose() }
// Unsaved edits are announced, so leaving the page, switching machine or closing the tab asks first.
useEffect(() => {
setUnsaved(dirty ? name : '')
if (!dirty) return
const h = (e: BeforeUnloadEvent) => { e.preventDefault(); e.returnValue = '' }
window.addEventListener('beforeunload', h)
return () => { window.removeEventListener('beforeunload', h); setUnsaved('') }
}, [dirty, name])
const dl = api.fileDownloadURL(path, sudo)
return (
<div className="viewer">
<div className="viewer-bar">
<button className="btn" onClick={close}>‹ Back</button>
<span className="vname mono" title={path}>{name}{dirty && <span className="dirty"> •</span>}</span>
<span className="spacer" />
{info && <span className="muted num">{fmtBytes(info.size)} · {info.perm} · {info.user}</span>}
<a className="btn" href={dl} download={name}>Download</a>
{text !== null && <Btn className="btn on" disabled={!dirty || saving} onClick={save}>Save</Btn>}
</div>
{err && <div className="error">{err}</div>}
{!info && !err && <Loading label="Opening…" />}
{info && text !== null && (
<textarea className="editor file-editor" value={text} spellCheck={false} onChange={(e) => setText(e.target.value)} />
)}
{info && text === null && isImage(name) && <div className="preview"><img src={api.fileReadURL(path, sudo)} alt={name} /></div>}
{info && text === null && !isImage(name) && !err && (
<div className="notice">{info.type === 'dir' ? 'This is a directory.' : info.size > MAX_EDIT ? `Too large to edit here (${fmtBytes(info.size)}). Download it instead.` : 'Binary file. Download it instead.'}</div>
)}
</div>
)
}
// ---- page ----
export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: string) => void }) {
const [hashPath, setHashPath] = useHashSel('files') // the folder is in the URL; without one, the last folder that opened
const [path, setPath] = useState<string>(() => (hashPath.startsWith('/') ? hashPath : sessionStorage.getItem('hamlab.files.path') || '/'))
useEffect(() => { setHashPath(path) }, [path, setHashPath])
useEffect(() => { if (hashPath.startsWith('/') && hashPath !== path) setPath(hashPath) }, [hashPath]) // eslint-disable-line react-hooks/exhaustive-deps
const [sudo, setSudo] = useState<boolean>(() => sessionStorage.getItem('hamlab.files.sudo') === '1')
const [entries, setEntries] = useState<FileEntry[] | null>(null)
const [err, setErr] = useState('')
const [hidden, setHidden] = useState(false)
const [sort, setSort] = useState<Sort>('name')
const [q, setQ] = useState('')
const [pathInput, setPathInput] = useState(path)
const [open, setOpen] = useState<string | null>(null)
const [menu, setMenu] = useState<string | null>(null)
const [upload, setUpload] = useState<{ name: string; pct: number; n: number; of: number } | null>(null)
const [drag, setDrag] = useState(false)
const fileInput = useRef<HTMLInputElement>(null)
const canSudo = useRef(true)
const load = useCallback(async (p = path, s = sudo) => {
try {
const l = await api.files(p, s)
setEntries(l.entries); setErr('')
canSudo.current = true
sessionStorage.setItem('hamlab.files.path', p) // only a folder that opened is worth coming back to
} catch (e) {
const msg = e instanceof Error ? e.message : 'failed'
setErr(msg); setEntries([])
}
}, [path, sudo])
useEffect(() => { setEntries(null); setPathInput(path); setMenu(null); load(path, sudo); sessionStorage.setItem('hamlab.files.sudo', sudo ? '1' : '0') }, [path, sudo, load])
useEffect(() => { const h = () => setMenu(null); window.addEventListener('click', h); return () => window.removeEventListener('click', h) }, [])
useEscape(() => setMenu(null), menu !== null)
const go = (p: string) => { setOpen(null); setPath(p || '/') }
const crumbs = path === '/' ? [] : path.slice(1).split('/')
const shown = (entries ?? []).filter((e) => (hidden || !e.name.startsWith('.')) && (!q || e.name.toLowerCase().includes(q.toLowerCase())))
.sort((a, b) => {
if ((a.type === 'dir') !== (b.type === 'dir')) return a.type === 'dir' ? -1 : 1
if (sort === 'size') return b.size - a.size
if (sort === 'mtime') return b.mtime - a.mtime
return a.name.localeCompare(b.name, undefined, { numeric: true, sensitivity: 'base' })
})
// ---- actions ----
const act = async (label: string, fn: () => Promise<unknown>) => {
try { await fn(); toast(label); await load() } catch (e) { toast(`${label} — ${e instanceof Error ? e.message : 'failed'}`, 'err') }
}
// A name is one path segment. ("../x" and "a/b" would quietly act somewhere else.)
const okName = (n?: string | null): n is string => { const t = n?.trim(); if (!t) return false; if (t.includes('/') || t === '.' || t === '..') { toast('a name can\'t contain / or be . or .. — use Move… to put it elsewhere', 'err'); return false } return true }
const mkdir = async () => { const n = await askText('New folder name'); if (okName(n)) act(`created ${n.trim()}`, () => api.fileMkdir(join(path, n.trim()), sudo)) }
const touch = async () => { const n = await askText('New file name'); if (okName(n)) act(`created ${n.trim()}`, () => api.fileTouch(join(path, n.trim()), sudo)).then(() => setOpen(join(path, n.trim()))) }
const rename = async (e: FileEntry) => { const n = await askText('Rename to', e.name); if (okName(n) && n.trim() !== e.name) act(`renamed to ${n.trim()}`, () => api.fileRename(join(path, e.name), join(path, n.trim()), sudo)) }
const move = async (e: FileEntry) => { const n = await askText('Move to (absolute path)', join(path, e.name)); if (n?.trim() && n.trim() !== join(path, e.name)) act(`moved to ${n.trim()}`, () => api.fileRename(join(path, e.name), n.trim(), sudo)) }
const chmod = async (e: FileEntry) => { const n = await askText(`Permissions for ${e.name} (octal)`, e.perm); if (n?.trim() && n.trim() !== e.perm) act(`chmod ${n.trim()} ${e.name}`, () => api.fileChmod(join(path, e.name), n.trim(), sudo)) }
const del = async (e: FileEntry) => {
const full = join(path, e.name)
if (e.type === 'dir' && !e.target) {
const typed = await askText(`Delete the folder "${e.name}" and everything in it?\nType its name to confirm.`)
if (typed !== e.name) return
act(`deleted ${e.name}`, () => api.fileDelete(full, true, sudo))
} else if (await askYes(`Delete ${e.name}?`)) act(`deleted ${e.name}`, () => api.fileDelete(full, false, sudo))
}
const copyPath = async (e: FileEntry) => { try { await navigator.clipboard.writeText(join(path, e.name)); toast('path copied') } catch { await askText('Path', join(path, e.name)) } }
const openTerm = (p: string) => { onOpenInTerminal?.(`cd ${shellQuote(p)}`); }
// ---- upload ----
const uploadFiles = async (files: FileList | File[]) => {
const list = Array.from(files)
for (let i = 0; i < list.length; i++) {
const f = list[i]
let overwrite = false
if ((entries ?? []).some((e) => e.name === f.name)) {
if (!await askYes(`"${f.name}" already exists here. Replace it?`)) continue
overwrite = true
}
setUpload({ name: f.name, pct: 0, n: i + 1, of: list.length })
try {
await api.fileUpload(path, f, sudo, (pct) => setUpload({ name: f.name, pct, n: i + 1, of: list.length }), overwrite)
} catch (e) { toast(`upload ${f.name} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); break }
}
setUpload(null)
toast(list.length === 1 ? `uploaded ${list[0].name}` : `uploaded ${list.length} files`)
load()
}
const onDrop = (e: DragEvent) => { e.preventDefault(); setDrag(false); if (e.dataTransfer.files.length) uploadFiles(e.dataTransfer.files) }
if (open) return <div className="files"><Viewer path={open} sudo={sudo} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
return (
<div className={'files' + (drag ? ' drag' : '')} onDragOver={(e) => { e.preventDefault(); setDrag(true) }} onDragLeave={() => setDrag(false)} onDrop={onDrop}>
<div className="files-bar">
<form className="crumbs" onSubmit={(e) => { e.preventDefault(); go(pathInput.trim()) }}>
<button type="button" className="crumb" onClick={() => go('/')}>/</button>
{crumbs.map((c, i) => <button type="button" key={i} className="crumb" onClick={() => go('/' + crumbs.slice(0, i + 1).join('/'))}>{c}</button>)}
<input className="path-input" value={pathInput} onChange={(e) => setPathInput(e.target.value)} onBlur={() => setPathInput(path)} spellCheck={false} />
</form>
<div className="files-tools">
<button className={'btn' + (sudo ? ' on' : '')} title="browse as root" onClick={() => setSudo(!sudo)}>{sudo ? 'root' : 'sudo'}</button>
<button className={'btn' + (hidden ? ' on' : '')} title="show hidden files" onClick={() => setHidden(!hidden)}>.hidden</button>
<select className="btn" value={sort} onChange={(e) => setSort(e.target.value as Sort)}><option value="name">name</option><option value="size">size</option><option value="mtime">modified</option></select>
<input className="filter" placeholder="filter" value={q} onChange={(e) => setQ(e.target.value)} />
<button className="btn" onClick={mkdir}>+ folder</button>
<button className="btn" onClick={touch}>+ file</button>
<button className="btn" onClick={() => fileInput.current?.click()}>Upload</button>
<input ref={fileInput} type="file" multiple style={{ display: 'none' }} onChange={(e) => { if (e.target.files?.length) uploadFiles(e.target.files); e.target.value = '' }} />
{onOpenInTerminal && <button className="btn" title="cd here in the active terminal" onClick={() => openTerm(path)}>&gt;_ here</button>}
<Btn className="btn" title="reload" onClick={() => load()}>↻</Btn>
</div>
</div>
{upload && <div className="upload-bar"><Spinner /> {upload.of > 1 ? `${upload.n}/${upload.of} · ` : ''}{upload.name} · {upload.pct}%<div className="meter"><div className="fill" style={{ width: `${upload.pct}%` }} /></div></div>}
{err && <div className="error" style={{ padding: '6px 12px' }}>{err}{/permission denied/i.test(err) && !sudo && <> · <button className="link" onClick={() => setSudo(true)}>retry as root</button></>}</div>}
<div className="file-list">
{entries === null && <Loading label="Listing…" />}
{path !== '/' && entries !== null && (
<div className="file-row up" onClick={() => go(path.slice(0, path.lastIndexOf('/')) || '/')}><span className="ficon">↩</span><span className="fname">..</span></div>
)}
{shown.map((e) => {
const full = join(path, e.name)
const isDir = e.type === 'dir'
return (
<div key={e.name} className={'file-row' + (e.broken ? ' broken' : '')} onClick={() => (isDir ? go(full) : setOpen(full))}>
<span className="ficon">{icon(e)}</span>
<span className="fname" title={e.target ? `→ ${e.target}` : e.name}>{e.name}{e.target && <span className="muted"> → {e.target}</span>}</span>
<span className="fmeta num">{isDir ? '' : fmtBytes(e.size)}</span>
<span className="fmeta num">{fmtTime(e.mtime)}</span>
<span className="fmeta num perm">{e.perm} {e.user}</span>
<button className="icon-btn more" onClick={(ev) => { ev.stopPropagation(); setMenu(menu === e.name ? null : e.name) }}>⋯</button>
{menu === e.name && (
<div className="ctx" onClick={(ev) => ev.stopPropagation()}>
{!isDir && <a href={api.fileDownloadURL(full, sudo)} download={e.name} onClick={() => setMenu(null)}>Download</a>}
{isDir && onOpenInTerminal && <button onClick={() => { setMenu(null); openTerm(full) }}>Open in terminal</button>}
<button onClick={() => { setMenu(null); rename(e) }}>Rename</button>
<button onClick={() => { setMenu(null); move(e) }}>Move…</button>
<button onClick={() => { setMenu(null); chmod(e) }}>Permissions ({e.perm})</button>
<button onClick={() => { setMenu(null); copyPath(e) }}>Copy path</button>
<button className="danger" onClick={() => { setMenu(null); del(e) }}>Delete</button>
</div>
)}
</div>
)
})}
{entries !== null && shown.length === 0 && !err && <div className="muted" style={{ padding: 12 }}>{q ? 'Nothing matches.' : 'Empty folder.'}{!hidden && entries.some((e) => e.name.startsWith('.')) ? ' (hidden files not shown)' : ''}</div>}
</div>
{drag && <div className="drop-hint">Drop to upload into {path}</div>}
</div>
)
}

Some files were not shown because too many files have changed in this diff Show More