Author SHA1 Message Date
Mitchell-JohnsonandClaude Fable 5.1 a2674bea6a thumbs: a GPU-access run already in flight refuses a second (409)
The sweep's battery hit the route three times; the second and third runs only collided
with the first's restart timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-13 18:00:39 -07:00
Mitchell-JohnsonandClaude Fable 5.1 134bbc991e PLAN.md: the whole-console sweep
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-13 17:56:16 -07:00
Mitchell-JohnsonandClaude Fable 5.1 285278abdb A sweep of the whole console: status codes, a data race, atomic writes, dead code
From the race detector, staticcheck, a garbage-in battery over every safe route and a
headless pass through every tab on two machines. Fixed:
- docker inspect/health of a container that isn't there answered 500; now 404, and a
  daemon that is down 503 (dockerErr, with a test).
- /api/virt/instances/{name} on a machine without Incus answered 502 "incus not found";
  it now says Incus isn't ready, like its siblings.
- Deleting a ZFS snapshot with a bad name answered 500; a bad name is 400, a snapshot
  that doesn't exist 404.
- The task list read a task's Running flag without the task's lock while the waiter
  goroutine cleared it: a data race, now read under the lock.
- The Virtualization export config and run record were written in place; a crash
  mid-write would have left a half file. Atomic now, like the config and state files.
- An unused function in kvm.go, one in incus/subnet.go, an unused field in files/move.go
  and a capitalised error string, from staticcheck.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-13 17:52:18 -07:00
Mitchell-JohnsonandClaude Fable 5.1 05161bf7ef PLAN.md: the Files edge sweep
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 20:32:34 -07:00
Mitchell-JohnsonandClaude Fable 5.1 1038c010fa Files: a sweep of the edges
Found by walking every handler and the UI with odd inputs, then a battery of requests
against the live hub. Fixed:
- A save over the 8 MiB cap was cut short in place: the file was truncated, then the
  first 8 MiB written, and the caller told 204. Now the body is read whole first and a
  body over the cap is a 413 with the file untouched.
- A link to a file showed the link's own time; now the file's, so the row reads right
  and the preview key follows the file.
- Reading a link to nothing answered 200 with an empty body (the open failed after the
  headers in root mode); now a 404 that names the target.
- A play request with an unknown codec or size was answered by inspecting the file
  first, so an audio-only file said "no video stream" instead of what was wrong; the
  choice is checked before any work, and is a 400.
- A stream that ended in failure was handed back to the next identical request for ten
  minutes; a failed one is now dropped and a fresh one made.
- A player that left while its stream was still finding a first segment made the loop
  try the next recipe; it stops. A stream closed mid-start is not started again.
- Ending a root-mode ffmpeg could wait forever: the daemon cannot signal root's
  processes, so the wait is bounded and the helper's own grace does the rest.
- A page going away (a phone closing the tab) releases its stream by a beacon, rather
  than ten minutes later by the sweeper.
- The path box resolves like a shell — relative to the folder shown, . and .. applied,
  trailing slashes dropped — in the tab and in the Move picker, with a test. A typed path
  that is a file opens it; a remembered folder that is gone falls back to / once.
- A dropped folder arrived as a nameless empty file; it is refused with a word. The
  upload toast counts what was actually uploaded.
- The viewer names a device, pipe or socket, and a dangling link, instead of calling
  them binary files.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 20:31:10 -07:00
Mitchell-JohnsonandClaude Fable 5.1 d200e5dc2e PLAN.md: re-encoding for the browser, on the machine's GPU
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 17:46:35 -07:00
Mitchell-JohnsonandClaude Fable 5.1 f3c547adfa Files: re-encode a video the browser can't play, on the machine's own GPU, and play that
The viewer tries the file as it is first. When the browser gives up — Safari and an MKV,
HEVC without a hardware decoder — or on a tap, a picker: codec (H.264, HEVC, AV1, or the
original codec repackaged) and size (original, 1080p, 720p, 480p), each codec labelled
with the engine this machine would use. POST /api/files/play runs ffmpeg into an HLS
event playlist of fragmented-MP4 segments under <data dir>/transcode/<id>/; Safari plays
HLS itself and everything else goes through hls.js, loaded only then. The recipes are
tried best first: a card on the bus before the integrated GPU, decoding on the same card
before decoding on the CPU and uploading, the software encoders last; a recipe that
produces no first segment is dropped for the next. The seekable range grows as ffmpeg
gets ahead; status says how far. Streams are reused for the same request, closed when
the player goes, swept after ten idle minutes, and the folder is emptied at start. Root
mode runs ffmpeg as root through a helper that ends when the daemon lets go of it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 17:42:25 -07:00
Mitchell-JohnsonandClaude Fable 5.1 aa4ae57c22 gofmt after the rename
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:54:52 -07:00
Mitchell-JohnsonandClaude Fable 5.1 af189b1fa1 internal/thumbs is internal/media, with Inspect: what a file holds, from ffprobe
The package was named for its first job; it now also decides what a file is for the
Files tab (image, video, audio) and reads its streams, which the viewer will need. Inspect
is the richer probe — codec, profile, pixel format, size, frame rate, HDR transfer, the
audio beside it, container and duration — run as root through sudo when asked.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:54:15 -07:00
Mitchell-JohnsonandClaude Fable 5.1 beec9269fe PLAN.md: video and audio playback in the Files viewer, as the file is
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:43:03 -07:00
Mitchell-JohnsonandClaude Fable 5.1 d7f56ee0b3 Files: a seek costs nothing; the backend reopens only when a read wants somewhere else
ServeContent seeks to the end for the size before it seeks to the range, which was
throwing away the stream Check had just opened.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:41:54 -07:00
Mitchell-JohnsonandClaude Fable 5.1 a9bcb88212 Files: play video and audio in the viewer, as the file is; reads answer Range requests
The viewer hands a video or audio file to the browser untouched — container, codec and
all — and the browser plays what it can; when it can't, the viewer says so and offers the
download. What the server adds is the part a player needs: /api/files/read now goes
through http.ServeContent for a real file of known size, so Range requests (seeking) and
HEAD work, and a download can resume. The root backend's helper takes an offset for that,
and a Seeker opens the backend once, at the range asked for, instead of streaming from
the top. Audio files are a preview kind too: their cover art is the thumbnail.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:40:42 -07:00
Mitchell-JohnsonandClaude Fable 5.1 c24f6f5cde Files: a move of something that isn't there is a 404, not a task that fails on arrival
The kernel answers EXDEV for two paths on different filesystems before it looks for the
source, so the cross-filesystem path was taken for a missing file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:30:04 -07:00
Mitchell-JohnsonandClaude Fable 5.1 b6d72cce9f PLAN.md: Move… is a folder picker; moves across filesystems copy then remove
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:29:27 -07:00
Mitchell-JohnsonandClaude Fable 5.1 fb3b8e4b4c Files: Move… opens a folder picker, and a move across filesystems copies then removes
The picker walks the machine like the tab does — back, forward, up, breadcrumbs, a path
box, a new folder — and "Move here" puts the file or folder in the folder being looked
at; it says why it can't when it can't (already here, into itself, a name clash). POST
/api/files/move is a rename when the two ends share a filesystem and otherwise a task:
on a NAS every move between datasets is a different filesystem, and a rename there is
EXDEV. The copy is built under a hidden temporary name beside its destination, renamed
into place once complete, and only then is the original removed, so a failure or a
cancel leaves the original as it was. Modes, times, symlinks and, as root, owners are kept.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:27:55 -07:00
Mitchell-JohnsonandClaude Fable 5.1 e5d3fb2afd PLAN.md: previews are cached in thumbs.db
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:05:17 -07:00
Mitchell-JohnsonandClaude Fable 5.1 e198471f8c Files previews: the cache is one database file, <data dir>/thumbs.db
bbolt instead of a tree of hash-named JPEGs: pure Go, so the binary stays static; one
file per machine; an entry carries where the preview came from (engine, codec, size,
when) beside the picture, and a count rides along so the tab can say what it holds
without walking anything. A miss never creates the file, so a machine nobody browses
with previews on never has one; a cache that cannot be opened is logged and previews
are still made and served, just not kept.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:04:12 -07:00
Mitchell-JohnsonandClaude Fable 5.1 51c718e8e3 PLAN.md: previews on the Files tab, shipped as v0.1.0-200
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 15:25:55 -07:00
Mitchell-JohnsonandClaude Fable 5.1 f8c9479e93 Files: previews — a toggle that shows a thumbnail per image and video, made by ffmpeg on the machine's GPU where it has one, cached under the data dir
A previews button on the Files tab turns the icon column into a 256 px thumbnail for
every image and video row, loaded as rows scroll near. internal/thumbs probes what the
machine can do rather than assuming it: ffmpeg and ffprobe, every /dev/dri/renderD*
opened and exercised, vainfo's decode profiles per node where it exists. A video goes to
the first GPU that lists its codec, with ffmpeg's own software fallback behind it and a
CPU retry behind that; images decode on the CPU, JPEGs at a reduced DCT size. Previews
are kept under <data dir>/thumbs/ keyed by path, size and mtime, with undecodable files
remembered too. Root mode makes them through the same root helper the file reads use.
Where ffmpeg is missing the tab offers to install it; where a render node exists but is
closed to the daemon's user, it offers a drop-in adding the group and a restart.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 15:20:27 -07:00
Mitchell-JohnsonandClaude Fable 5.1 e7034cd478 VIRTUALIZATION.md: grid's Incus removed for a fresh start; status
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 15:02:52 -07:00
Mitchell-JohnsonandClaude Opus 5.5 e068fbdb2f PLAN.md: copyparty retired ahead of the Files gaps; they stay as Files polish
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:54:27 -07:00
Mitchell-JohnsonandClaude Fable 5.1 faf444d8ce VIRTUALIZATION.md: storage decision revised — a Disk location picker, plain files on grid's root disk
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 14:36:06 -07:00
Mitchell-JohnsonandClaude Fable 5.1 ae72e45bf5 Virtualization: a disk-location picker on both create forms, locations by place, add or remove a location
Pools are shown by where their bytes land (plain files on this disk, a ZFS pool inside one
file, a dataset…) rather than by driver name, which read as the NAS pool. A location can be
added as a plain-files pool at any folder on the machine, and removed while nothing uses it.
The driver order at Install is as it was; the labels say what each one really is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 14:33:10 -07:00
Mitchell-JohnsonandClaude Opus 5.5 a9b4d7969e PLAN.md: CPU power on the Processes tab, shipped as v0.1.0-194
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:00:05 -07:00
Mitchell-JohnsonandClaude Opus 5.5 bfedf622be Processes: CPU power beside the clock, from the packages' energy counters
New sensor cpu.power: watts from the RAPL energy counters (Intel, and AMD
Zen through the same driver), summed over every package and averaged into
history. Parts of a package, psys and the MMIO duplicate are left out.

The counters are root-only, so a daemon that isn't root reads them through
`hamlab cpu-energy`, a root helper started once with sudo or doas and asked
over a pipe each sample, the same shape as the GPU client walk's helper.
It reads only the counters it finds itself. No counters or no root means
no reading, and the CPU box shows nothing extra.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 23:58:35 -07:00
Mitchell-JohnsonandClaude Opus 5.5 d5680e8e41 VGA console: stop importing spice-html5's demo stylesheet, which restyled every button
spice.css sets body, *, button and canvas for the whole document. Bundled
into the App chunk since 43ebacd (deployed as v0.1.0-188), it turned every
button into a gradient pill, shifted 15% right and 8% up with a 150px
minimum width, broke the rail and header layout, and bordered every graph
canvas. The console is already styled by the .vga rules in styles.css.

The file is removed from the vendored copy, and a new web test fails on any
stylesheet import outside an allow-list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 15:19:20 -07:00
Mitchell-JohnsonandClaude Opus 5.5 ea84ee40c7 PLAN.md: the 2375 item deployed as v0.1.0-191 and checked on every machine
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 15:00:19 -07:00
Mitchell-JohnsonandClaude Opus 5.5 24a9979432 Docker updates: a machine without Docker answers empty, not 500
The hub used to answer this for its nodes and hid the node's 500. Now the
node answers itself, so it needs the same guard the stack list has.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 14:58:13 -07:00
Mitchell-JohnsonandClaude Opus 5.5 c27d9f0ada Stacks: one machine, one set of files; default home is <data dir>/stacks
The hub no longer probes a node's Docker on tcp :2375, holds compose files
for a node, or runs compose against another machine's Docker. hubStacks,
nodeDocker, taskOnHub and the hub-side /api/nodes/{node}/docker/... routes
are gone; the generic node proxy hands those requests to the node's own
Ham Lab. Stack.remote / held_by and compose.Spec.Remote are removed.

New stacks default to <data dir>/stacks on every machine. Folders with two
or more deployed stacks are still offered, after it, never as default.
The ~/stacks fallback is gone. A missing home is skipped without a sudo find.

A schedule saved earlier with a tcp target is refused with a message
instead of being run against this machine's Docker.

PORTABILITY.md 3.1, 3.2, 3.3 and 9.2 done; 3.5 checked, no change needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 14:53:10 -07:00
64 changed files with 5325 additions and 647 deletions
+227 -28
View File
@@ -21,7 +21,7 @@ stub tab or a roadmap item.
| Guest logins | A **guest** is an account like any other, with `guest: true` and an expiry: `login`, sessions and prefs need no special case, and **it may do everything an account may do**. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. | Built 2026-09-27. A read-only version was built first and **cut the same day**: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is *an account*: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to. |
| Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
| Capability, not policy | **A tab offers what the selected machine can do, detected live — never what a rule says should run there.** Where the software is missing but installable, Ham Lab installs it; where it cannot be installed or the hardware lacks it, the tab says so in a sentence and offers what does work. Every action lands on the machine in the switcher. | Mitchell, 2026-10-09, after the Virtualization plan's first draft excluded the Mac mini and `hamtv` on the strength of `hamlet/CLAUDE.md`'s deployment rules: *"I want the option to do it anywhere I want, unless the hardware / software literally doesn't support it. Ham Lab needs to be built in a general-purpose way that can run on any Linux system."* Those rules bind the agent's unasked deployments, not the product. |
| Ham Lab's data lives in Ham Lab's folder | **Everything a tab manages sits under the machine's data directory (`config.DataDir()`), never at a convenient path elsewhere.** Existing material is imported by copying. The engine's runtime state (`/var/lib/docker`, `/var/lib/incus`) is the engine's; Ham Lab keeps what would rebuild it. | Mitchell, 2026-10-09, when the Virtualization plan pointed its ISO library at `/tank/software/isos/`: *"ALL data needs to be under a HamLab folder, because it manages its own data."* The same reason `core`'s compose files moved under `apps/hamlab/stacks/` on 2026-10-05. `grid`'s and the Mac's stacks at `~/stacks/<machine>/` are the standing exception. |
| Ham Lab's data lives in Ham Lab's folder | **Everything a tab manages sits under the machine's data directory (`config.DataDir()`), never at a convenient path elsewhere.** Existing material is imported by copying. The engine's runtime state (`/var/lib/docker`, `/var/lib/incus`) is the engine's; Ham Lab keeps what would rebuild it. | Mitchell, 2026-10-09, when the Virtualization plan pointed its ISO library at `/tank/software/isos/`: *"ALL data needs to be under a HamLab folder, because it manages its own data."* The same reason `core`'s compose files moved under `apps/hamlab/stacks/` on 2026-10-05. `grid`'s and the Mac's stacks at `~/stacks/<machine>/` were the standing exception; since 2026-10-10 the stacks code offers `<data dir>/stacks` as the default home everywhere and such folders only as discovered extras. |
## Layout
@@ -260,12 +260,13 @@ something long in one, switch away and back, refresh — it's all still there.
filesystem; compose is then run **inside that container**, so `.env`, relative paths
and the compose labels stay exactly as deployed (verified: labels identical after a
redeploy). Save validates with `compose config` first, keeps the previous file as
`<file>.hamlab-bak`, and never deploys by itself. A stack deployed from another host
(Dockhand on `core` driving `grid` over tcp) has no file on that machine, so the node
alone lists it read-only — but **the hub fills that in** (`hubStacks`): asked about a
node, it resolves the files on itself and runs compose here with `DOCKER_HOST` pointed
at the node's `tcp://…:2375`, the way it was deployed; the task runs on the hub and the
node's page follows it. Also per stack: every file it was deployed from (chips), its
`<file>.hamlab-bak`, and never deploys by itself. A stack whose file is not on the
machine that runs it is listed read-only, and that is the end of it: **since 2026-10-10
the hub holds no files for a node and never points compose at another machine's
Docker.** The `hubStacks` path — which resolved a node's files on the hub and deployed
over `tcp://…:2375`, the way Dockhand had — is gone, and every
`/api/nodes/<node>/docker/…` request is the node's own Ham Lab's to answer through the
generic proxy. Also per stack: every file it was deployed from (chips), its
**`.env`** (values masked until "Reveal and edit"; validated in place, put back if
compose objects), **what Deploy would change** (a line diff against the copy kept at the
last deploy from here, `<file>.hamlab-deployed`, else the copy the last save replaced;
@@ -278,9 +279,10 @@ something long in one, switch away and back, refresh — it's all still there.
form (image, ports, volumes, environment, restart, command, host networking) that writes
the compose for you: a container made here is a one-service stack, so it stays editable
and redeployable, where a bare `docker run` could only be thrown away and typed again.
New stacks go in a **home** — a folder where two or more stacks are already deployed from
(Dockhand's `stacks/core` here; `~/stacks` on a machine with none) — and are deployed the
way their neighbours were (inside `dockhand`, or over tcp to a node from the hub). Homes
New stacks go in a **home**. Since 2026-10-10 the default is Ham Lab's own
`<data dir>/stacks`, offered first on every machine; a folder where two or more stacks
are already deployed from is offered as well, and a stack made there is deployed the way
its neighbours were (inside an orchestrator's container, when that is how they run). Homes
are also scanned, so a stack with no containers ("not deployed": never deployed, or taken
down) is still listed. **Down** removes a stack's containers; **Delete…** (type its name)
takes it down and moves its own files to `<home>/.hamlab-trash/<name>-<time>/` — moved,
@@ -325,6 +327,14 @@ something long in one, switch away and back, refresh — it's all still there.
`grid` the sampler hung for 42 minutes in Go's poller on the Arc's `fan1_input` —
`readStr` now uses plain blocking syscalls and each GPU's sysfs reads run under a
3-second deadline (v0.1.0-168; `hamlet/docs/gotchas.md` has the mechanism).
- **CPU power on the Processes tab** — **shipped 2026-10-10** (v0.1.0-194,
`internal/metrics/cpupower.go`, the `cpu.power` sensor). Asked for beside the clock speed,
and that is where it sits: "3600 MHz · 89 W" in the cpu box's header. Watts are the RAPL
package energy counters' delta between samples, summed over packages; `psys`, a package's
parts and the MMIO duplicate are left out. The counters are root-only, so a daemon running
as `core` keeps a second root child, `hamlab cpu-energy`, beside `drm-clients`. Read on all
four Intel machines; the Mac mini has no CPU-only counter (its SMC reports the whole
machine's draw instead) and shows nothing.
- **The ARC on the ZFS tab** — **shipped 2026-10-02** (`internal/metrics/arc.go`,
`internal/storage/arc.go`, `GET /api/zfs/arc`, the Cache section in `ZFS.tsx`, `zfs.arc.*`
sensors). Asked for as "current ARC usage, including any types of caching services in the
@@ -615,6 +625,197 @@ something long in one, switch away and back, refresh — it's all still there.
so both modes share one implementation and one set of rules (absolute paths only;
`/`, `/etc`, `/usr`, … are never deleted or moved whole). Every write is logged with
the caller's IP.
- **Previews on the Files tab** — **shipped 2026-10-11** (v0.1.0-200, `internal/thumbs`,
`internal/api/thumbs.go`, `Thumb`/`ThumbNote` in `Files.tsx`). Asked for as a button that
toggles a file preview, "obviously not for everything", cached, "use whatever GPU is on the
machine we're looking at, and fall back to CPU", "a single small frame preview so it's
fast". A **previews** button beside `.hidden` turns the icon column into a 256 px
thumbnail for every image and video row (the listing says which: `preview: image|video`
per entry, decided by extension in one place, `thumbs.Kind`), loaded as rows scroll within
400 px of view — the list scrolls in a box of its own, so it has its own
IntersectionObserver rather than the browser's window-based lazy loading. A video thumb
carries a ▶ badge. The setting is in localStorage and follows the browser.
- **ffmpeg is the one engine**, for images and video alike; `GET /api/files/thumb` is one
ffprobe and one ffmpeg: `-ss` a tenth of the way in (30 s at most, `-noaccurate_seek`
so one keyframe is decoded), the first real video stream (cover art is skipped), one
frame, `scale` to fit 256, MJPEG q4 to a pipe. JPEGs decode at a reduced DCT size
(`-lowres`, enough halvings to stay at twice the preview) — a 24 MP photo is read as
1.5 MP. Thumbnails are the only thing Ham Lab asks a GPU to do, and it asks the way the
Processes tab reads one: **probed, never assumed.** The probe looks ffmpeg and ffprobe
up, opens every `/dev/dri/renderD*`, pushes a synthetic frame through each with VAAPI,
and reads `vainfo`'s decode profiles per node where it exists; CUDA is tried the same
way when `/dev/nvidiactl` is there (untested: nothing here has one). A video goes to the
first GPU listing its codec; ffmpeg's own software fallback sits behind that for a
profile the list did not distinguish, and a CPU retry behind a hardware run that fails
outright. Images are CPU only — one frame through libjpeg at a quarter size beats a
round trip through a GPU. Measured on `grid`: 1080p h264 0.4 s on the UHD 750 against
0.7 s on the CPU; the 8K Osmo HEVC 1.0 s against 2.9 s. `grid`'s two GPUs are listed in
node order and both decode the same set (h264, hevc, vp9, mjpeg, mpeg2 — **no AV1 on
either**, per vainfo, so AV1 is dav1d on the CPU).
- **Cached in one database file under the machine's data dir**, `<data dir>/thumbs.db`
(bbolt — pure Go, so the binary stays static; v0.1.0-202, the first build kept them as
a tree of hash-named JPEGs under `thumbs/`, replaced the same day when Mitchell asked
where they were kept: "stored in hamlab, maybe under a DB, up to you"). An entry is
keyed by path, size and mtime — a changed file gets a new preview, an unchanged one is
never decoded twice — and carries the engine, codec, source size and time beside the
picture; a count rides in the file so the previews button's tooltip can say "N kept
(M MB)" without walking anything. The URL carries `v=<mtime>-<size>` so the browser's
own cache is right too (7 days, ETag, 304). A file ffmpeg could not decode is kept with
ffmpeg's reason and no picture, answered as 422, retried only when the file changes; a
timeout is not kept. A miss never creates the file, so a machine nobody browses with
previews on never has one; a cache that cannot be opened is logged and previews are
still made and served, just not kept. Nothing expires: a preview is a few KB, and only
files somebody scrolled past are ever made. At most 2–4 ffmpegs at once per machine
(half the cores, clamped), one per file however many rows ask, and a preview's work
runs on a clock of its own so the row that asked first leaving does not cancel it for
the rest. Each ffmpeg runs under `nice -n 10`. `hamlet`'s backup job skips the file.
- **Root mode makes them as root** (`hamlab thumb`, the sibling of `hamlab fs`, caps on
stdin, JSON out), so a file only root can read gets a preview, and the probe runs as
root too — root opens every render node whatever groups the daemon has. Which matters:
a render node is `root:render 0660`, and on `core` and `hamtv` the `core` user is not in
`render`, so there the daemon's own previews are CPU and the tab says so, naming the GPU
and the owner, with an **Allow GPU access** button (confirms, then a task writes
`hamlab.service.d/gpu.conf` adding `render`/`video` as supplementary groups and
restarts Ham Lab three seconds later, the shape the Incus install uses for
`incus-admin`). Not pressed anywhere yet. Where ffmpeg is missing (the Mac mini, `gate`)
the note offers **Install ffmpeg with dnf/apk** through the Updates machinery
(`ffmpeg-free` on Fedora, `ffmpeg` elsewhere); the status re-probes when the task ends.
- *Not done, on purpose:* no cap on the cache and no clear button (not asked; it only
holds what was looked at); no tonemapping for HDR sources (a 10-bit HEVC frame comes out
flat); no preview in the viewer for a video (it still says "binary, download it").
- **Move… is a folder picker** — **shipped 2026-10-11** (v0.1.0-204, `MovePicker` in
`Files.tsx`, `internal/files/move.go`, `POST /api/files/move`). Asked for as: instead of
a path prompt, "an interactive file explorer, where I can go back/forward, etc and then
click some button, and it will move the file/folder to where I was". The picker opens in
the item's own folder and walks the machine the way the tab does — back and forward
through where it has been, up, breadcrumbs, a path box, folders only, `.hidden`
following the tab's toggle, **+ folder** to make the destination on the spot — and
**Move here** puts the item in the folder being looked at. It says why it can't when it
can't, with the button greyed: already here, a folder into itself, a name clash in that
folder (read from the listing before anything is tried). The row being moved shows in
its own folder, greyed. Rename stays a name prompt: it is a different question.
- **Across filesystems it copies, then removes.** `os.Rename` is EXDEV between
filesystems, and on `grid` every dataset is one — `/tank/media` to `/tank/files` was a
"invalid cross-device link" error before this, and the picker would have made that
move easy to attempt. `/api/files/move` renames when it can (204) and otherwise runs
`hamlab fs move` as a task (202 with the task; the tab shows its output and reloads
when it ends). The copy is built under a hidden name beside its destination
(`.<name>.hamlab-moving`), renamed into place once complete, and only then is the
original removed — a failure part-way removes the partial copy and leaves the original
as it was, and a cancel (SIGTERM to the helper) does the same. Modes, mtimes and
symlinks are kept; owners too when the backend is root. Devices, pipes and sockets are
refused outright rather than half-copied. Progress every 500 files or 512 MiB.
Verified on `grid` ext4 → ZFS with a tree holding a 0600 file, a symlink and an old
mtime, all intact on the other side.
- **Video and audio play in the viewer** — **shipped 2026-10-11** (v0.1.0-208). Asked for
as "keep it simple for now, and just send whatever codec the file is in to the client,
and let the client try to decode and watch it" — and that is the whole design. Opening a
video or audio file shows a `<video>`/`<audio>` with the browser's own controls, fed the
file as it is by `/api/files/read`; nothing is transcoded or remuxed, so what plays is
what the browser decodes (Safari: h264/hevc/aac in mp4/mov; Chrome: those plus vp9/av1 in
mkv/webm, hevc only where the machine has a hardware decoder; nothing plays MPEG-TS or a
GoPro `.360`'s raw EAC strip). When the element errors the viewer says so and leaves the
download. What the server gained is the part a player cannot do without: reads of a
real file with a known size go through `http.ServeContent`, so **Range requests** (every
seek, and Safari's first request) and HEAD work, and a download can resume. The root
backend's helper takes `--offset` for that, and `files.Seeker` opens the backend once, at
the range asked for, rather than streaming from the top and skipping; seeking costs
nothing until a read wants somewhere else. `/proc` files, which stat at size 0, keep the
plain streaming path, as does the editor's capped read. Audio became a preview kind at
the same time (`thumbs.Kind` → `audio`): its cover art, where it has any, is the row's
thumbnail, with a ♪ badge.
- *Not done, on purpose:* no playlist or next/previous, no subtitles. (Transcoding came
the same evening after all — the next entry — because Safari turned out not to play
the gaming captures.)
- **Re-encoding for the browser, on the machine's GPU** — **shipped 2026-10-11**
(v0.1.0-212, `internal/media/{encode,hls,ffrun,inspect}.go`, `internal/api/play.go`,
`Player.tsx`, `hls.js`). Asked for when Safari would not play the gaming captures:
"add GPU transcoding to a supported codec when the client browser can't play it. Each
machine should use its own GPU (if available) and transcode to probably H.264. Maybe we
need a quality selector, and a codec selector, up to you." What Safari actually refuses
is the **MKV container** — the captures checked are H.264 High 4:2:0, 18–36 Mbps — so
the picker has a "repackaged" choice beside the codecs, and a 36 Mbps capture over
WireGuard is the other reason to re-encode at all.
- **The viewer tries the file as it is first**, as before. When the browser gives up, or
on **Re-encode…** in the bar, a picker: **codec** — H.264, HEVC, AV1, or the original
codec repackaged — each labelled with the engine this machine would use (or greyed as
"not on this machine"), and **size** — original, 1080p, 720p, 480p; the choice is
remembered. Play starts the re-encode and the player follows it: "h264 High 2292×960
60 fps 36 Mbps in matroska → H.264 2292×960 5.0 Mbps on Intel Arc A380 (vaapi) · 7.3× ·
34% through — seek within what's done", then "all of it is ready". **Change…** drops
the stream and goes back to the picker.
- **HLS, made on the machine in the switcher.** `POST /api/files/play` inspects the file
(ffprobe, as root in root mode) and runs ffmpeg into an *event* playlist of 4 s
fragmented-MP4 segments under `<data dir>/transcode/<id>/`, answering once the first
segment exists (0.5–1 s on `grid`); the player reads the playlist and segments from
`/api/files/play/<id>/…` (only those three names are served), Safari natively and
everything else through hls.js, loaded only then. ffmpeg runs ahead at the engine's
pace, so the seekable range grows and the whole file is through long before it is
watched. Keyframes are forced every 4 s so segments cut cleanly; sound is AAC stereo
unless it already is AAC; HDR is tone-mapped (`tonemap_vaapi`, or zscale/hable on the
CPU). Nothing is kept: the same request within ten minutes reuses the stream, the
player closes it when it goes (and the "Change…" path does), the sweeper closes what
nobody touched for ten minutes, and the folder is emptied when the daemon starts. At
most three at once per machine (429 with the reason). Root mode runs ffmpeg as root
through `hamlab run-ffmpeg`, which ends when the daemon lets go of its stdin — so a
stream cannot outlive the console.
- **Engines are probed, never assumed**, the same probe the previews use, now also
reading each node's encode profiles from vainfo and ffmpeg's own encoder list. The
recipes for a codec are tried best first until one produces a first segment: a card on
the bus before the integrated GPU (the A380 before the UHD 750), decoding on the same
card so frames never leave it, then decoding on the CPU and uploading (a source the
card cannot decode — AV1 on `grid`), then libx264 / libx265 / SVT-AV1. The size is
never larger than the source, and the bit rate follows the output size (2/4/8/12/20
Mbps for 480p/720p/1080p/1440p/2160p, +25 % at 50 fps and up, ×0.6 for HEVC and AV1).
Measured on `grid` with a 2292×960 60 fps capture: H.264 7.3× realtime, HEVC 720p
11.7×, AV1 720p 13×, all on the Arc. `core` and `hamtv` re-encode on the CPU until
*Allow GPU access* is pressed (libx264 at 9.4× for a 720p→480p clip on `core`); the Mac
and `gate` need ffmpeg first, which the previews note offers to install.
- *Not done, on purpose:* no restart-from-a-seek beyond what is encoded (the encode is
faster than watching; a seek far ahead waits), no bit-rate ladder or adaptive
switching, no subtitles, no keeping a finished re-encode for next time.
- **Files edge sweep** — **2026-10-11** (v0.1.0-214), asked for as "a full bug sweep of
the Files section, every example and edge case". Every handler and both UI files read
through with odd inputs in mind, then a battery of ~60 requests against the live hub
(`sweep.py`, not kept: listings of files, missing and forbidden paths, NUL bytes, names
with spaces/#/%/?/umlauts, links to files, folders and nothing, a pipe, over-size saves,
uploads with bad names and clashes, renames and moves onto things, protected paths,
thumbnails of empty and mislabelled images, playback of an image and an audio-only mp4,
root-mode variants). One real data bug: **a save over the 8 MiB cap truncated the file
and kept the first 8 MiB, answering 204** — the body is now read whole first and refused
with 413, the file untouched. The rest: a link to a file showed the link's own time; a
link to nothing read as an empty 200; a bad codec or size was answered by inspecting
the file; a failed stream was handed back to the next identical request; a player
leaving mid-start made the recipe loop try the next engine; a root-mode ffmpeg's end
could wait forever; a page closing kept its stream for ten minutes (a beacon now
releases it at once); the path box took only absolute paths (it resolves like a shell
now, in the tab and the Move picker, with a test); a dropped folder became a nameless
empty file; the upload toast counted intentions; the viewer called a pipe a binary
file. Not bugs, left alone: `rm -rf` of a mount point empties it (the console is
root-capable by design); a folder of ten thousand rows is slow (no virtual list).
- **Whole-console sweep** — **2026-10-13** (v0.1.0-216), asked for as "a full sweep of
Ham Lab again, look for any bugs and fix them". Four nets, then reading: the Go race
detector over the whole suite (clean), staticcheck's development build (the release does
not know Go 1.26; it found only dead code and one capitalised error string), a
garbage-in battery over every route safe to poke — 202 routes, bad identifiers, empty
and malformed bodies, with the destructive ones held back — and a headless Chromium pass
through all twenty tabs on the hub and on `grid`, collecting exceptions, console errors
and failed requests (none). By hand: the auth middleware and sessions, token issue, the
hub→node and KVM proxies, every validation regex before a shell, the cron and cron.d
writers, config and state atomicity, every WebSocket and SSE loop's exit, the terminal
manager, the task manager, and every destructive confirmation in the UI. Fixed, all
small: docker inspect/health of a missing container was a 500 (404 now, a daemon that
is down 503), `/api/virt/instances/{name}` without Incus was a 502 (409 like its
siblings), a ZFS snapshot delete with a bad name was a 500 (400; 404 for one that does
not exist), the task list read a task's Running flag outside its lock (a real data race,
never seen), the Virtualization export config was written in place rather than
atomically, and the dead code. Nothing in auth, the proxies or the shell boundaries.
One self-inflicted finding: the battery's skip list left out `POST
/api/files/thumbs/gpu-access`, so the sweep pressed *Allow GPU access* on `core` three
times over — the drop-in landed and the hub restarted (that is what the headless pass
saw as a refused connection). Kept, since it is the fix the tab offers; `core` makes
previews and re-encodes on its UHD iGPU now. The handler refuses a second run while one
is in flight (409), since the second `systemd-run` only collided with the first's timer.
- **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth.
- **Notifications** — push on service failure / SMART warning / UPS event.
- **Terminal extras** — search, copy-mode, font/theme, split panes, upload via
@@ -709,12 +910,12 @@ what it has moved.
## To do
- **Grow the Files tab until copyparty can go** — asked for 2026-09-23 as two items (Files
polish, and deprecating copyparty), merged 2026-09-24: they are the same tab and the same
work. `copyparty` on `grid` (`copyparty.service`, `:3923`, `/etc/copyparty.conf`, serving
all of `/tank` as user `core` with one account behind NPM) overlaps **Files** almost
entirely. The aim is to stop the service and keep only Ham Lab, with general ease-of-use
polish along the way. What Files must gain first, in the order that matters:
- **Grow the Files tab** — asked for 2026-09-23 as two items (Files polish, and deprecating
copyparty), merged 2026-09-24. **copyparty was retired 2026-10-11** at Mitchell's call,
ahead of these gaps rather than after them: service, files and NPM host all gone, with a
rollback archive kept on `grid` (`hamlet`'s `docs/hosts.md` has the detail). Files is now
the only file browser on `grid`, so the list below is no longer parity — it is the polish
still owed, in the order that matters:
- **Multi-select and bulk actions.** copyparty selects many rows and moves, deletes or
downloads them together; Files acts on one row at a time.
- **Download a folder**, and a multi-file download, as a streamed zip or tar — the thing
@@ -722,17 +923,14 @@ what it has moved.
- **Resumable uploads.** copyparty's up2k chunks a file, survives a dropped connection
and skips what is already there; Files streams one PUT, so a phone upload that drops at
90% starts over.
- **A gallery.** Thumbnails for a folder of images and video, and play audio/video in
place — `/tank/media` and `/tank/immich` are most of what is browsed.
- **A gallery.** ~~Thumbnails for a folder of images and video~~ (the **previews** toggle,
2026-10-11 — see the shipped entry above; a grid view is not built), and play
audio/video in place — `/tank/media` and `/tank/immich` are most of what is browsed.
- **Search** below the current folder by name, since `find` in a terminal is the fallback
today.
- *Decide when built:* whether share links (copyparty's public per-path links) are wanted
at all — they are the one feature that hands out access without a login, and Ham Lab has
no notion of that; if not, say so in the docs rather than leaving a gap unexplained.
- *After:* `systemctl disable --now copyparty`, remove its NPM host, keep
`/etc/copyparty.conf` until a rollback is clearly not needed, then update
`hamlet`'s `docs/hosts.md` bare-metal list, `docs/gotchas.md` (the "`docker ps` is not
the inventory" note names it) and the `grid` row in `CLAUDE.md`.
- **Grow the Docker tab until Dockhand can go** — asked for 2026-09-23 and widened
2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the
@@ -780,12 +978,13 @@ what it has moved.
stacks that are merely down (`pihole_default`, `transmission_default`, `gluetun_default`
among them), and pruning would take them and hand them new subnets on the next deploy.
- **The `*:2375` question, which this is the moment to settle.** A stack deployed to
`grid` or the Mac still goes over `tcp://<node>:2375` from the hub. Decide whether the
hub should instead ask the node's own Ham Lab to deploy locally, which would end the
unauthenticated Docker socket exposure — its own item in `docs/pending.md`, currently
filed under "decided, no action", and this is the change that would make closing it
free.
- **The `*:2375` question — settled, and the code removed 2026-10-10.** The hub asks the
node's own Ham Lab; nothing probes or uses a node's Docker port. `hubStacks`,
`nodeDocker`, `taskOnHub`, the hub-side `/api/nodes/{node}/docker/…` routes and the
`remote` / `held_by` fields are gone. A schedule saved before this that still carries a
`remote` is refused with a message, never run against the wrong Docker. `PORTABILITY.md`
§3 has the audit it came from. Deployed as v0.1.0-191 and checked on all five machines
through the hub.
- **A Virtualization tab** — asked for 2026-09-23; **scoped, built and deployed 2026-10-09 — see
`VIRTUALIZATION.md`** for what shipped (phases 1 and 2) and what is left. Incus 6.0 LTS as the engine; the tab detects per machine whether Incus is
+8 -8
View File
@@ -44,11 +44,11 @@ default or phones home to this lab · **C** cosmetic (placeholder text, comments
| # | Sev | Where | Assumes | General form |
|---|---|---|---|---|
| 3.1 | **A** | `internal/api/stacks.go:483-509` `nodeDocker()` | For every node whose stacks aren't editable locally, the hub **probes `tcp://<node-host>:2375`** with `docker -H … version` and, if it answers, deploys to the node over an unauthenticated Docker socket. This is the Dockhand-era layout; on any other network an open 2375 is either absent or a security problem the console should not quietly use. | Remove the probe. The node's own Ham Lab deploys its stacks (the files live on the node, under its data dir). `PLAN.md` "To do" already names this decision. |
| 3.2 | **A** | `internal/api/stacks.go:439-479` `hubStacks()`, `:1036-1074` `homesFor()`, `web/src/Docker.tsx` "kept on `<hub>`, deployed to this machine over `tcp://…`" | A hub can hold compose files for a node and run compose remotely; new stacks for a node may be created **on the hub**. Only true here because Dockhand on `core` deployed `grid` and the Mac. | Same as 3.1: one machine, one set of files, one Docker. Hub-held stacks become an import path (copy to the node) rather than a steady state. |
| 3.3 | **B** | `internal/api/stacks.go:290-382` `homesOf()`; `:373-376` fallback `~/stacks` | Where new stacks go is **discovered** from where ≥2 stacks already are (Dockhand's `stacks/<machine>` habit), else `~/stacks`. Neither is Ham Lab's data directory, which the project's own rule says everything managed must live under. | Default home = `<DataDir>/stacks`. Discovered homes stay as *additional* places, never the default. |
| 3.1 | **done** 2026-10-10 | `internal/api/stacks.go` `nodeDocker()` (removed) | For every node whose stacks aren't editable locally, the hub **probes `tcp://<node-host>:2375`** with `docker -H … version` and, if it answers, deploys to the node over an unauthenticated Docker socket. This is the Dockhand-era layout; on any other network an open 2375 is either absent or a security problem the console should not quietly use. | Remove the probe. The node's own Ham Lab deploys its stacks (the files live on the node, under its data dir). `PLAN.md` "To do" already names this decision. |
| 3.2 | **done** 2026-10-10 | `hubStacks()`, `taskOnHub()`, the hub-side `/api/nodes/{node}/docker/…` routes, `Stack.remote`/`held_by` (all removed; the generic node proxy carries those requests) | A hub can hold compose files for a node and run compose remotely; new stacks for a node may be created **on the hub**. Only true here because Dockhand on `core` deployed `grid` and the Mac. | Same as 3.1: one machine, one set of files, one Docker. Hub-held stacks become an import path (copy to the node) rather than a steady state. |
| 3.3 | **done** 2026-10-10 | `internal/api/stacks.go` `homesOf()` — `<DataDir>/stacks` is always offered first and default; the `~/stacks` fallback is gone; `TestHomesDefaultToDataDir` | Where new stacks go is **discovered** from where ≥2 stacks already are (Dockhand's `stacks/<machine>` habit), else `~/stacks`. Neither is Ham Lab's data directory, which the project's own rule says everything managed must live under. | Default home = `<DataDir>/stacks`. Discovered homes stay as *additional* places, never the default. |
| 3.4 | **B** | `internal/api/stacks.go:100-203` remembered mounts, `internal/api/dockersched.go:772-795` `repoint()` | Machinery exists to find compose files **inside a removed orchestrator container's filesystem**. Generic in form, but it exists because of Dockhand and is exercised only here. | Keep (it is general), but it is not a reason to keep 3.1/3.2. |
| 3.5 | **B** | `internal/api/stacks.go:801` | Inside a `Via` container the command is literally `docker compose`, even when `dockerTool()` is `podman`. | Use the detected tool. |
| 3.5 | — | `internal/api/stacks.go` `stackAction`, `internal/compose/compose.go` | Inside a `Via` container the command is literally `docker compose`. | **Checked 2026-10-10, no change:** the CLI inside an orchestrator's container is that container's own `docker`, independent of what the host runs; the host's tool only chooses the `exec`. Both places now say so in a comment. |
| 3.6 | **B** | `internal/api/stacks.go:908` `docker buildx imagetools inspect` | Update checks need the **buildx** plugin. Docker CE on Debian has it; Podman and minimal Docker installs do not → every image reports "the registry didn't answer". | Fall back to `skopeo inspect` or a direct registry `HEAD` for the digest. |
| 3.7 | **B** | `Makefile:445-448` `deploy` | The machine you build on **is** the hub: installs locally, then `push-nodes`. | Fine for a developer; document that `deploy` means "I am the hub". A plain `make install` target for a single box. |
| 3.8 | **B** | `internal/nodes/nodes.go:316-323` | A hub pushes other architectures only if `<prefix>/hamlab-<arch>` siblings exist, which only `make deploy`/`install.sh`-from-dist put there. | A hub without siblings should fetch the right build from `update.url` instead of erroring. |
@@ -106,7 +106,7 @@ The installer supports systemd, procd and OpenRC (`internal/setup/setup.go:217-2
| # | Sev | Where | Assumes | General form |
|---|---|---|---|---|
| 7.1 | **A** | `internal/priv/priv.go`; every tab that calls `priv.Available()`; `web/src/Updates.tsx:2144` "give its user passwordless sudo or doas"; `web/src/Jobs.tsx:1883` | Full function needs the service user to have **unrestricted passwordless sudo**, which `hamlet/CLAUDE.md` grants on every host here. The installer neither creates nor offers a sudoers rule, so on a fresh box most tabs are read-only and the UI tells the user to go fix sudoers by hand. | `hamlab install --sudo` writes a **scoped** `/etc/sudoers.d/hamlab` naming the commands Ham Lab runs (smartctl, zpool, zfs, systemctl, docker, journalctl, tee for its cron files, `hamlab fs|space-walk|drm-clients|replace`), or installs as root. Either way the console works out of the box. |
| 7.1 | **A** | `internal/priv/priv.go`; every tab that calls `priv.Available()`; `web/src/Updates.tsx:2144` "give its user passwordless sudo or doas"; `web/src/Jobs.tsx:1883` | Full function needs the service user to have **unrestricted passwordless sudo**, which `hamlet/CLAUDE.md` grants on every host here. The installer neither creates nor offers a sudoers rule, so on a fresh box most tabs are read-only and the UI tells the user to go fix sudoers by hand. | `hamlab install --sudo` writes a **scoped** `/etc/sudoers.d/hamlab` naming the commands Ham Lab runs (smartctl, zpool, zfs, systemctl, docker, journalctl, tee for its cron files, `hamlab fs|space-walk|drm-clients|cpu-energy|replace`), or installs as root. Either way the console works out of the box. |
| 7.2 | **A** | `internal/api/server.go:1069-1080` `clientKey()`, `:1083-1092`, `:431-433` `secure()` | The **first `X-Forwarded-For` hop is trusted unconditionally**, because here Ham Lab always sits behind Nginx Proxy Manager. On a host exposed directly, a client sets the header and defeats the per-client login rate limiter and the session IP check. | Trust `X-Forwarded-*` only from configured proxy addresses (`trusted_proxies` in config), else use the socket peer. |
| 7.3 | **B** | `internal/api/server.go:1324` `X-Accel-Buffering: no` "nginx (NPM)" | Harmless header; comment names the lab's proxy. | Keep header; generalise comment. |
| 7.4 | **B** | `internal/api/guests.go:1-9`, `PLAN.md:21` | Guest logins are full-power ("he gives these to people he trusts"). A product default. | Keep, but it is a posture decision taken for this household; a read-only guest is the generic expectation and was cut. Document as a deliberate choice in README. |
@@ -125,7 +125,7 @@ The installer supports systemd, procd and OpenRC (`internal/setup/setup.go:217-2
| # | Sev | Where | Assumes | General form |
|---|---|---|---|---|
| 9.1 | **B** | `internal/config/config.go:115-117` `Candidates()`; `internal/setup/setup.go:413-422`; `README.md:38` | `~/apps/hamlab/config.json` is searched as a legacy location — the lab's pre-v0.2 path, kept so `core`, `grid` and the Mac keep working. | Keep for one more release with a one-time migration log line, then drop. A stranger never had it. |
| 9.2 | **B** | 3.3 above; `PLAN.md:24` "`grid`'s and the Mac's stacks at `~/stacks/<machine>/` are the standing exception" | The "everything under DataDir" rule has a lab-shaped exception baked into the stacks code. | Default to DataDir; treat `~/stacks` as a discovered home like any other. |
| 9.2 | **done** 2026-10-10 | 3.3 above; `PLAN.md:24` "`grid`'s and the Mac's stacks at `~/stacks/<machine>/` are the standing exception" | The "everything under DataDir" rule has a lab-shaped exception baked into the stacks code. | Default to DataDir; treat `~/stacks` as a discovered home like any other. |
| 9.3 | **C** | `internal/setup/setup.go:351` `ownFiles` | Correctly scoped; note only. | None. |
## 10. Build and release
@@ -157,8 +157,8 @@ Listed so the sweep is known to be complete, not because they need work.
## Suggested order of work
1. **Posture and safety first (A):** 7.2 trusted proxies; 7.1 installer-written scoped sudoers; 3.1 drop the `:2375` probe (with 3.2 as its consequence).
1. **Posture and safety first (A):** 7.2 trusted proxies; 7.1 installer-written scoped sudoers; ~~3.1 drop the `:2375` probe (with 3.2 as its consequence)~~ done 2026-10-10.
2. **"Any Linux" promises the README already makes (A):** 4.1 power, 4.4/4.5 cron on busybox and timer-only hosts, 4.2 Incus install on non-systemd, 4.3 a Services backend for OpenRC/procd, 2.10 the `tank` regex.
3. **Lab defaults that phone home or mislead (B):** 1.1–1.3 update/install URLs, 2.9 schedule defaults, 6.9/6.10 speed-test sources and an off switch, 8.1 memory limit, 3.3/9.2 stack home = DataDir.
3. **Lab defaults that phone home or mislead (B):** 1.1–1.3 update/install URLs, 2.9 schedule defaults, 6.9/6.10 speed-test sources and an off switch, 8.1 memory limit, ~~3.3/9.2 stack home = DataDir~~ done 2026-10-10.
4. **Breadth (B):** 5.1 zypper for Incus, 5.5 cgroup v1, 6.1/6.2 GPUs beyond Intel/AMD, 6.4 UPS order, 6.11/6.12 wg-quick/NetworkManager, 6.15 IPv6.
5. **Cosmetics (C):** every placeholder and comment in §1.10 and §2, then split lab facts out of `PLAN.md`, `VIRTUALIZATION.md`, `SENSORS.md`, `README.md`.
+6 -1
View File
@@ -12,7 +12,7 @@ added to the registry in `api.New`.
## IDs
```
cpu.usage cpu.core.<n>.usage cpu.temp load.1|5|15 uptime
cpu.usage cpu.core.<n>.usage cpu.temp cpu.power load.1|5|15 uptime
mem.total|used|avail|free|cached|buffers|used_pct swap.total|used|avail|used_pct
temp.<chip>.<label> fan.<chip>.<label>
net.<iface>.rx|tx|rx_total|tx_total|up
@@ -35,6 +35,11 @@ zfs.arc.size|target|min|max|mru|mfu|data|metadata|hit_pct|reads|misses|lifetime_
has run on that machine, and each one's `t` is when it ran — so a widget or an alert on
them is reading a measurement from that moment, not a live value. Nothing takes a speed
test on its own; something has to press the button.
- `cpu.power` is the CPU packages' own power in watts, from the kernel's RAPL energy
counters (Intel, and AMD Zen), summed over every package: cores, built-in graphics and
memory controller, not the whole machine. The counters are root-only, so a daemon that is
not root reads them through a helper started once with sudo or doas. It is absent on a
machine without the counters (ARM, the Mac mini) or without a way to root.
- `gpu.<name>.*` is each graphics device, named as lspci names it (`gpu.intel_arc_a380`;
labels carry `driver` and the PCI `bus`). `usage` is how busy the device was — from the
GT's idle residency on Intel, `gpu_busy_percent` on amdgpu — and is absent on a driver
+24 -21
View File
@@ -1,20 +1,17 @@
# Virtualization tab — plan (2026-10-09; phase 1 built the same day)
**Status:** phases 1 and 2 are **deployed to all five machines as v0.1.0-188** (2026-10-09).
Incus runs on `grid` (installed from the tab); the other machines show Install or, on `gate`,
unavailable. Phase 2 verified live on `grid` through the hub: ISO fetched by URL with its
sha256 checked, **New VM from ISO** booted Alpine with Secure Boot off, **the VGA console
drew it in a real browser** (headless Chromium, 1280×800 canvas, keyboard and mouse live),
a virtiofs mount appeared in a running VM, an export tarball landed in `virt/exports/` and the
nightly cron line is written, `instance.*` sensors flow and the "VM or container down" alert
default is in. Two things found and fixed on the way: spice-html5 wants the `binary`
websocket subprotocol agreed; and it opens one websocket per SPICE channel, so the bridge
holds **one console operation per VM** and gives each channel its own data websocket on it
(Incus refuses a second operation as "already connected"). **Left from phase 2:** the LAN
bridge step (`br0` on the uplink — needs a live test on a machine that can afford to drop
off the network for a moment, which is a decision, not a build), cloud-init presets beyond
"Use my SSH keys", and a backup line for `grid`'s `virt/exports/` (see `hamlet/docs/backups.md`).
`t-alpine` on `grid` is the live example; delete it from the tab when done.
**Status:** phases 1 and 2 are **deployed to all five machines as v0.1.0-196**. **No machine
has Incus installed as of 2026-10-11**: `grid`'s install of 2026-10-09 was removed at
Mitchell's request so he can install it himself from the tab and ask questions along the way
(`hamlet/docs/hosts.md` has what was removed and what the two days established). The tab
shows Install on `grid`, `core`, the Mac mini and `hamtv`, unavailable on `gate`. Both create
forms carry a **Disk location** picker with add/remove of locations; instance storage as
plain files on the machine's own disk is what he wants on `grid`. Everything in phase 2 was
verified live on `grid` before the removal: ISO fetch with checksum, New VM from ISO, the VGA
console in a real browser (headless Chromium; `binary` subprotocol, one console operation per
VM with a data websocket per SPICE channel), a virtiofs mount into a running VM, exports and
their cron line, `instance.*` sensors and the alert default. **Left:** the LAN bridge step
(his call), cloud-init presets beyond "Use my SSH keys", a backup line for `virt/exports/`.
A tab for creating, watching, consoling into and deleting **LXC containers and KVM virtual
machines**, the way Proxmox does with `pct` and `qm` — **per machine, like every other
@@ -236,12 +233,18 @@ like everything else.
## Layout decisions Ham Lab makes per machine (overridable in the Install dialog)
1. **Storage.** The instance pool's backing lives in Ham Lab's folder: Install creates a
sparse `virt/pool.img` and hands it to Incus as the pool's `source` (zfs or btrfs by the
detection above), or `virt/pool/` as a `dir` pool where neither is available. Sized from
the disk's free space, shown and editable. Bulk data is **bind-mounted in**, not stored in
the pool. The driver, path and size are recorded in `virt.json` so the page can say where
the data is and Install can rebuild the pool from the file after a reinstall.
1. **Storage — revised 2026-10-11.** The create forms carry a **Disk location** picker: a
list of the machine's Incus storage pools, each described by where its bytes land, with
"+ add a location" (any folder on the machine becomes a plain-files pool). On `grid` the
one location is `local`: plain files under `virt/pool/` on the root NVMe, each VM's disk
its own sparse file sized on its own, each container a directory, nothing pre-sized.
*The first design* (a ZFS pool inside one sparse file, handed to Incus as the pool's
source, for instant snapshots and clones) was built, ran for two days, and was taken out
at Mitchell's instruction: one pre-sized file holding every disk is an opaque middle
layer with a number on it that reads as a reservation, and "zfs · hamlab-virt" on the page
read as the NAS pool. The Install dialog still offers zfs/btrfs-in-a-file for a machine
that wants instant snapshots; the labels now say exactly what each is. Bulk data is still
**bind-mounted in**, not stored in the pool.
2. **Bridge subnet, computed.** Incus's `auto` picks a random `10.x.y.0/24`, and **this
LAN is `10.0.0.0/8`** — every host has a /8 on its NIC, so any 10.x bridge collides.
Ham Lab reads the machine's routes and picks the first free /24 from `192.168.0.0/16`,
+7
View File
@@ -31,6 +31,7 @@ import (
"hamlab/internal/cron"
"hamlab/internal/files"
"hamlab/internal/incus"
"hamlab/internal/media"
"hamlab/internal/metrics"
"hamlab/internal/nodes"
"hamlab/internal/priv"
@@ -82,10 +83,16 @@ func main() {
switch cmd { // commands that don't read the config
case "fs": // privileged helper for the Files tab (run as root by the daemon)
os.Exit(files.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
case "thumb": // privileged helper for the Files tab's previews in root mode (run as root by the daemon)
os.Exit(media.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
case "run-ffmpeg": // privileged helper for playback in root mode: ffmpeg as root, ending when stdin closes
os.Exit(media.RunFFmpegHelper(flag.Args()[1:], os.Stdin, os.Stderr))
case "space-walk": // helper for the Space tab: measure mountpoints, stream the tree back
os.Exit(space.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
case "drm-clients": // privileged helper for the GPU readings: who has the GPU open (run as root by the daemon)
os.Exit(metrics.DRMClientsHelper(os.Stdin, os.Stdout))
case "cpu-energy": // privileged helper for the CPU power reading: the packages' energy counters (run as root by the daemon)
os.Exit(metrics.CPUEnergyHelper(os.Stdin, os.Stdout))
case "install", "uninstall":
var err error
if cmd == "install" {
+1
View File
@@ -5,6 +5,7 @@ go 1.26.0
require (
github.com/coder/websocket v1.8.15
github.com/creack/pty v1.1.24
go.etcd.io/bbolt v1.5.0
golang.org/x/crypto v0.57.0
golang.org/x/term v0.46.0
)
+12
View File
@@ -2,9 +2,21 @@ github.com/coder/websocket v1.8.15 h1:6B2JPeOGlpff2Uz6vOEH1Vzpi0iUz20A+lPVhPHtNU
github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
go.etcd.io/bbolt v1.5.0/go.mod h1:mkltfYE5aUHQxUct9N9V+Kp7aSjFqjgrhcXIS70Lrdk=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+22 -3
View File
@@ -343,6 +343,21 @@ func (s *Server) dockerLogs(w http.ResponseWriter, r *http.Request) {
}
// dockerInspect is the parts of `docker inspect` worth reading on a phone.
// dockerErr answers for a docker command that failed: a thing that isn't there is a 404,
// the daemon being away a 503, anything else a 500.
func dockerErr(w http.ResponseWriter, err error) {
msg := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(err.Error()), "error: "))
low := strings.ToLower(msg)
switch {
case strings.Contains(low, "no such object") || strings.Contains(low, "no such container") || strings.Contains(low, "not found"):
jsonErr(w, 404, msg)
case strings.Contains(low, "cannot connect to the docker daemon") || strings.Contains(low, "is the docker daemon running"):
jsonErr(w, 503, msg)
default:
jsonErr(w, 500, msg)
}
}
func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
if !dockerRef.MatchString(id) {
@@ -352,9 +367,13 @@ func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
defer cancel()
out, err := dockerRun(ctx, "inspect", id)
if err != nil {
dockerErr(w, err)
return
}
var all []map[string]any
if err != nil || json.Unmarshal(out, &all) != nil || len(all) == 0 {
jsonErr(w, 500, fmt.Sprint("inspect failed: ", err))
if json.Unmarshal(out, &all) != nil || len(all) == 0 {
jsonErr(w, 500, "inspect: unreadable answer")
return
}
c := all[0]
@@ -486,7 +505,7 @@ func (s *Server) dockerHealth(w http.ResponseWriter, r *http.Request) {
defer cancel()
out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id)
if err != nil {
jsonErr(w, 500, err.Error())
dockerErr(w, err)
return
}
var st struct {
+8 -2
View File
@@ -55,11 +55,14 @@ type composeSpec struct {
Project string `json:"project"`
Files []string `json:"files"`
Via string `json:"via,omitempty"`
// Remote is read but never written: schedules made before v0.2 could point compose at
// another machine's Docker over tcp. Such a job is refused rather than run against this
// machine's Docker, which is where it would land without the variable.
Remote string `json:"remote,omitempty"`
}
func (c composeSpec) toSpec(tool string) compose.Spec {
return compose.Spec{Project: c.Project, Files: c.Files, Via: c.Via, Remote: c.Remote, Tool: tool}
return compose.Spec{Project: c.Project, Files: c.Files, Via: c.Via, Tool: tool}
}
const dockerCronFile = "/etc/cron.d/hamlab-docker"
@@ -169,6 +172,9 @@ func RunUpdate(ctx context.Context, db *store.Store, j UpdateJob, tool string) (
if len(j.Spec.Files) == 0 {
return "", errors.New("this schedule has no compose file recorded — open it in the Docker tab and save it again")
}
if j.Spec.Remote != "" {
return "", errors.New("this schedule was made to deploy to another machine over tcp, which this console no longer does — make it again on that machine")
}
spec := repoint(ctx, db, j.Spec.toSpec(tool))
var log strings.Builder
run := func(args ...string) error {
@@ -238,7 +244,7 @@ func (s *Server) dsPut(w http.ResponseWriter, r *http.Request) {
// seenFiles is the list as compose itself sees it: host paths normally, the
// orchestrator container's own paths when compose runs inside one. Getting this
// backwards gives a cron line that works from the console and fails at 3am.
j.Spec = composeSpec{Project: st.Name, Files: seenFiles(st, -1, ""), Via: st.Via, Remote: st.Remote}
j.Spec = composeSpec{Project: st.Name, Files: seenFiles(st, -1, ""), Via: st.Via}
if len(j.Spec.Files) == 0 {
jsonErr(w, 409, "that stack's compose file isn't anywhere this console can reach")
return
+123
View File
@@ -0,0 +1,123 @@
package api
import (
"bytes"
"errors"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"testing"
"hamlab/internal/files"
)
// A file read answers Range requests — what a <video> needs to seek — and HEAD, through
// the same handler that streams everything else.
func TestFilesReadRanges(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(dir, "clip.mp4")
data := make([]byte, 10000)
for i := range data {
data[i] = byte(i % 251)
}
if err := os.WriteFile(p, data, 0o644); err != nil {
t.Fatal(err)
}
s := &Server{fsLocal: files.Local{}}
get := func(method, extra string, hdr map[string]string) *httptest.ResponseRecorder {
r := httptest.NewRequest(method, "/api/files/read?path="+url.QueryEscape(p)+extra, nil)
for k, v := range hdr {
r.Header.Set(k, v)
}
w := httptest.NewRecorder()
s.filesRead(w, r)
return w
}
w := get("GET", "", nil)
if w.Code != 200 || w.Body.Len() != 10000 || w.Header().Get("Content-Type") != "video/mp4" || w.Header().Get("Accept-Ranges") != "bytes" {
t.Fatalf("whole: %d %d %q %q", w.Code, w.Body.Len(), w.Header().Get("Content-Type"), w.Header().Get("Accept-Ranges"))
}
w = get("GET", "", map[string]string{"Range": "bytes=100-199"})
if w.Code != 206 || w.Header().Get("Content-Range") != "bytes 100-199/10000" || w.Body.Len() != 100 || w.Body.Bytes()[0] != data[100] {
t.Fatalf("range: %d %q %d", w.Code, w.Header().Get("Content-Range"), w.Body.Len())
}
w = get("GET", "", map[string]string{"Range": "bytes=9990-"})
if w.Code != 206 || w.Body.Len() != 10 || w.Body.Bytes()[9] != data[9999] {
t.Fatalf("tail range: %d %d", w.Code, w.Body.Len())
}
w = get("HEAD", "", nil)
if w.Code != 200 || w.Body.Len() != 0 || w.Header().Get("Content-Length") != "10000" {
t.Fatalf("head: %d %d %q", w.Code, w.Body.Len(), w.Header().Get("Content-Length"))
}
if w := get("GET", "&max=50", nil); w.Code != 413 {
t.Fatalf("the editor's cap: %d", w.Code)
}
if w := get("GET", "", map[string]string{"Range": "bytes=20000-"}); w.Code != 416 {
t.Fatalf("a range past the end: %d", w.Code)
}
if err := os.Chmod(p, 0); err == nil && os.Geteuid() != 0 {
if w := get("GET", "", nil); w.Code != 403 {
t.Fatalf("unreadable: %d", w.Code)
}
}
}
// A write over the cap is refused whole: the file is not truncated first.
func TestFilesWriteCap(t *testing.T) {
p := filepath.Join(t.TempDir(), "f.txt")
if err := os.WriteFile(p, []byte("keep me"), 0o644); err != nil {
t.Fatal(err)
}
s := &Server{fsLocal: files.Local{}}
body := make([]byte, 8<<20+1)
r := httptest.NewRequest("PUT", "/api/files/write?path="+url.QueryEscape(p), bytes.NewReader(body))
w := httptest.NewRecorder()
s.filesWrite(w, r)
if w.Code != 413 {
t.Fatalf("got %d", w.Code)
}
if b, _ := os.ReadFile(p); string(b) != "keep me" {
t.Fatalf("the file was touched: %q", b)
}
r = httptest.NewRequest("PUT", "/api/files/write?path="+url.QueryEscape(p), bytes.NewReader([]byte("new")))
w = httptest.NewRecorder()
s.filesWrite(w, r)
if b, _ := os.ReadFile(p); w.Code != 204 || string(b) != "new" {
t.Fatalf("a write within the cap: %d %q", w.Code, b)
}
}
// A link to nothing is a 404, not an empty 200.
func TestFilesReadDanglingLink(t *testing.T) {
dir := t.TempDir()
l := filepath.Join(dir, "broken")
if err := os.Symlink("nowhere", l); err != nil {
t.Fatal(err)
}
s := &Server{fsLocal: files.Local{}}
r := httptest.NewRequest("GET", "/api/files/read?path="+url.QueryEscape(l), nil)
w := httptest.NewRecorder()
s.filesRead(w, r)
if w.Code != 404 {
t.Fatalf("got %d: %s", w.Code, w.Body.String())
}
}
// dockerErr tells a thing that isn't there from a daemon that isn't up from anything else.
func TestDockerErr(t *testing.T) {
for _, c := range []struct {
msg string
want int
}{
{"error: No such object: zzz", 404}, {"Error response from daemon: No such container: x", 404},
{"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?", 503},
{"something else entirely", 500},
} {
w := httptest.NewRecorder()
dockerErr(w, errors.New(c.msg))
if w.Code != c.want {
t.Errorf("%q → %d, want %d", c.msg, w.Code, c.want)
}
}
}
-14
View File
@@ -338,20 +338,6 @@ func (s *Server) listKVMs(w http.ResponseWriter, _ *http.Request) {
jsonOK(w, 200, out)
}
// kvmForMachine is the KVM attached to a machine, "" being the hub itself.
func (s *Server) kvmForMachine(machine string) (config.KVM, bool) {
var found config.KVM
ok := false
s.cfg.View(func(c *config.Config) {
for _, k := range c.KVMs {
if k.Machine == machine {
found, ok = k, true
}
}
})
return found, ok
}
func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
var body struct{ Name, URL, Machine string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
+169
View File
@@ -0,0 +1,169 @@
package api
// Playing a video the browser cannot play as it is: POST /api/files/play starts an HLS
// re-encode on this machine (internal/media), GET …/play/{id}/index.m3u8 and the segments
// feed the player, GET …/play/{id} is its progress, DELETE ends it. /api/files/play-caps
// says which codecs this machine can make and on what. Root mode (?sudo=1) reads the file
// and runs ffmpeg as root, through the same sudo the file reads use.
import (
"context"
"encoding/json"
"errors"
"io"
"log"
"net/http"
"path/filepath"
"strings"
"time"
"hamlab/internal/files"
"hamlab/internal/media"
)
const playLimit = 3 // re-encodes at once on one machine; a GPU does two or three, a CPU one
type playCaps struct {
Codecs []playCodec `json:"codecs"`
Qualities []media.Quality `json:"qualities"`
Error string `json:"error,omitempty"`
}
type playCodec struct {
media.Codec
Engine string `json:"engine,omitempty"` // "" when nothing here encodes it
}
// playCapsHandler: GET /api/files/play-caps[?sudo=1]
func (s *Server) playCapsHandler(w http.ResponseWriter, r *http.Request) {
root := r.URL.Query().Get("sudo") == "1"
if root && s.fsSudo == nil {
jsonErr(w, 403, "root access needs passwordless sudo on the host")
return
}
caps, err := s.thumbs.capsFor(r.Context(), root, false)
pc := playCaps{Codecs: []playCodec{}, Qualities: media.Qualities}
if err != nil {
pc.Error = err.Error()
} else if caps.Error != "" {
pc.Error = caps.Error
}
for _, c := range media.Codecs {
x := playCodec{Codec: c}
if pc.Error == "" {
x.Engine = media.EngineFor(caps, c.ID)
}
pc.Codecs = append(pc.Codecs, x)
}
jsonOK(w, 200, pc)
}
// playStart: POST /api/files/play[?sudo=1] {path, codec, quality}
func (s *Server) playStart(w http.ResponseWriter, r *http.Request) {
fsys, ok := s.fsFor(w, r)
if !ok {
return
}
root := r.URL.Query().Get("sudo") == "1"
var body struct{ Path, Codec, Quality string }
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
p, err := files.Clean(body.Path)
if err != nil {
jsonErr(w, 400, err.Error())
return
}
if media.Kind(p) != "video" {
jsonErr(w, 415, "not a video")
return
}
if err := media.ValidChoice(body.Codec, body.Quality); err != nil {
jsonErr(w, 400, err.Error())
return
}
if _, err := fsys.Stat(r.Context(), p); err != nil {
fsErr(w, err)
return
}
caps, err := s.thumbs.capsFor(r.Context(), root, false)
if err != nil {
jsonErr(w, 503, err.Error())
return
}
if caps.Error != "" {
jsonErr(w, 503, caps.Error)
return
}
ictx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
src, err := media.Inspect(ictx, caps, p, root)
cancel()
if err != nil {
playErr(w, err)
return
}
st, err := s.plays.Open(r.Context(), caps, root, p, body.Codec, body.Quality, src)
if err != nil {
playErr(w, err)
return
}
log.Printf("files: play %s as %s %s on %s (from %s)", p, st.Codec, st.Quality, st.Engine, clientIP(r))
jsonOK(w, 200, map[string]any{"stream": st, "url": "/api/files/play/" + st.ID + "/index.m3u8"})
}
func playErr(w http.ResponseWriter, err error) {
var de *media.DecodeError
switch {
case errors.As(err, &de):
jsonErr(w, 422, de.Msg)
case strings.Contains(err.Error(), "already being re-encoded"):
jsonErr(w, 429, err.Error())
case strings.HasPrefix(err.Error(), "unknown ") || strings.HasPrefix(err.Error(), "nothing on this machine encodes"):
jsonErr(w, 400, err.Error())
case errors.Is(err, context.DeadlineExceeded):
jsonErr(w, 504, "the file took too long to read")
default:
jsonErr(w, 500, err.Error())
}
}
// playStatus: GET /api/files/play/{id}
func (s *Server) playStatus(w http.ResponseWriter, r *http.Request) {
st, ok := s.plays.Find(r.PathValue("id"))
if !ok {
jsonErr(w, 404, "no such stream")
return
}
jsonOK(w, 200, st.Status())
}
// playStop: DELETE /api/files/play/{id}, or POST …/close — the form a page can send on its
// way out (navigator.sendBeacon), so a phone closing the tab releases the machine at once.
func (s *Server) playStop(w http.ResponseWriter, r *http.Request) {
s.plays.Close(r.PathValue("id"))
w.WriteHeader(204)
}
// playFile: GET /api/files/play/{id}/{file} — the playlist, the init segment, a segment.
func (s *Server) playFile(w http.ResponseWriter, r *http.Request) {
st, ok := s.plays.Find(r.PathValue("id"))
if !ok {
jsonErr(w, 404, "no such stream")
return
}
p, ok := st.File(r.PathValue("file"))
if !ok {
jsonErr(w, 404, "not a stream file")
return
}
switch filepath.Ext(p) {
case ".m3u8":
w.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
default:
w.Header().Set("Content-Type", "video/mp4")
}
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("X-Content-Type-Options", "nosniff")
http.ServeFile(w, r, p)
}
+114 -31
View File
@@ -35,6 +35,7 @@ import (
"hamlab/internal/files"
"hamlab/internal/hardware"
"hamlab/internal/incus"
"hamlab/internal/media"
"hamlab/internal/metrics"
"hamlab/internal/network"
"hamlab/internal/nodes"
@@ -79,6 +80,9 @@ type Server struct {
virt *incus.Manager
fsLocal files.FS
fsSudo files.FS // nil without passwordless sudo
thumbs *thumbSvc
plays *media.Streamer
self string // this binary, for helpers run as tasks
nodes *nodes.Manager
fingerprint string
version string
@@ -98,7 +102,10 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
if err != nil {
self = "hamlab"
}
s.self = self
s.space = space.New(self, filepath.Join(cfg.DataDir(), "space"))
s.thumbs = newThumbSvc(cfg.DataDir(), self)
s.plays = media.NewStreamer(cfg.DataDir(), self, playLimit)
s.hostName() // read it once now so the first request doesn't have to
if u, err := user.Current(); err == nil && u.Username != "" {
s.user = u.Username
@@ -220,20 +227,8 @@ func (s *Server) routes() *http.ServeMux {
m.HandleFunc("PUT /api/docker/schedules", s.auth(s.dsPut))
m.HandleFunc("DELETE /api/docker/schedules/{id}", s.auth(s.dsDelete))
m.HandleFunc("POST /api/docker/schedules/{id}/run", s.auth(s.dsRun))
m.HandleFunc("GET /api/nodes/{node}/docker/updates", s.auth(s.machineUpdates))
// The same, asked of a node through this hub: the hub answers for the stacks whose files
// it holds (deployed to the node over tcp) and passes everything else to the node.
m.HandleFunc("GET /api/nodes/{node}/docker/stacks", s.auth(s.stacksList))
m.HandleFunc("POST /api/nodes/{node}/docker/stacks", s.auth(s.stackCreate))
m.HandleFunc("GET /api/nodes/{node}/docker/stacks/homes", s.auth(s.stackHomes))
m.HandleFunc("POST /api/nodes/{node}/docker/stacks/{name}/delete", s.auth(s.stackDelete))
m.HandleFunc("GET /api/nodes/{node}/docker/stacks/{name}", s.auth(s.stackFile))
m.HandleFunc("PUT /api/nodes/{node}/docker/stacks/{name}", s.auth(s.stackSave))
m.HandleFunc("GET /api/nodes/{node}/docker/stacks/{name}/diff", s.auth(s.stackDiff))
m.HandleFunc("GET /api/nodes/{node}/docker/stacks/{name}/updates", s.auth(s.stackUpdates))
m.HandleFunc("POST /api/nodes/{node}/docker/stacks/{name}/{action}", s.auth(s.stackAction))
m.HandleFunc("GET /api/nodes/{node}/tasks/{id}", s.auth(s.taskOnHub(s.getTask)))
m.HandleFunc("GET /api/nodes/{node}/tasks/{id}/stream", s.auth(s.taskOnHub(s.streamTask)))
// Asked of a node through this hub, every one of these is the node's to answer: the
// generic /api/nodes/{name}/{rest...} proxy above carries them there unchanged.
m.HandleFunc("POST /api/docker/containers/{id}/{action}", s.auth(s.dockerAction))
m.HandleFunc("GET /api/docker/containers/{id}/logs", s.auth(s.dockerLogs))
m.HandleFunc("GET /api/docker/containers/{id}/logs/tail", s.auth(s.dockerTail))
@@ -271,6 +266,8 @@ func (s *Server) routes() *http.ServeMux {
m.HandleFunc("POST /api/virt/exports/run", s.auth(s.virtExportsRun))
m.HandleFunc("POST /api/virt/exports/delete", s.auth(s.virtExportDelete))
m.HandleFunc("GET /api/virt/sshkeys", s.auth(s.virtSSHKeys))
m.HandleFunc("POST /api/virt/pools", s.auth(s.virtPoolCreate))
m.HandleFunc("POST /api/virt/pools/delete", s.auth(s.virtPoolDelete))
m.HandleFunc("GET /api/alerts", s.auth(s.alertsGet))
m.HandleFunc("PUT /api/alerts", s.auth(s.alertsPut))
m.HandleFunc("POST /api/alerts/test", s.auth(s.alertsTest))
@@ -309,8 +306,19 @@ func (s *Server) routes() *http.ServeMux {
m.HandleFunc("POST /api/files/mkdir", s.auth(s.filesMkdir))
m.HandleFunc("POST /api/files/touch", s.auth(s.filesTouch))
m.HandleFunc("POST /api/files/rename", s.auth(s.filesRename))
m.HandleFunc("POST /api/files/move", s.auth(s.filesMove))
m.HandleFunc("POST /api/files/delete", s.auth(s.filesDelete))
m.HandleFunc("POST /api/files/chmod", s.auth(s.filesChmod))
m.HandleFunc("GET /api/files/thumb", s.auth(s.filesThumb))
m.HandleFunc("GET /api/files/thumbs", s.auth(s.thumbsStatus))
m.HandleFunc("POST /api/files/thumbs/install", s.auth(s.thumbsInstall))
m.HandleFunc("POST /api/files/thumbs/gpu-access", s.auth(s.thumbsGPUAccess))
m.HandleFunc("GET /api/files/play-caps", s.auth(s.playCapsHandler))
m.HandleFunc("POST /api/files/play", s.auth(s.playStart))
m.HandleFunc("GET /api/files/play/{id}", s.auth(s.playStatus))
m.HandleFunc("DELETE /api/files/play/{id}", s.auth(s.playStop))
m.HandleFunc("POST /api/files/play/{id}/close", s.auth(s.playStop))
m.HandleFunc("GET /api/files/play/{id}/{file}", s.auth(s.playFile))
m.HandleFunc("GET /api/hardware", s.auth(s.hardwareReport))
m.HandleFunc("GET /api/network", s.auth(s.networkOverview))
m.HandleFunc("GET /api/vpn", s.auth(s.vpnReport))
@@ -960,11 +968,7 @@ func (s *Server) updateAllMachines(w http.ResponseWriter, r *http.Request) {
}
func (s *Server) proxyNode(w http.ResponseWriter, r *http.Request) {
node := r.PathValue("node") // routes that also carry a {name} of their own call the machine {node}
if node == "" {
node = r.PathValue("name")
}
p, ok := s.nodes.Proxy(node)
p, ok := s.nodes.Proxy(r.PathValue("name"))
if !ok {
jsonErr(w, 404, "no such machine")
return
@@ -1928,21 +1932,15 @@ func (s *Server) filesRead(w http.ResponseWriter, r *http.Request) {
jsonErr(w, 400, "not a regular file (device, pipe or socket)")
return
}
if in.Broken {
jsonErr(w, 404, "a link to nothing: "+in.Target)
return
}
mx, _ := strconv.ParseInt(r.URL.Query().Get("max"), 10, 64)
if mx > 0 && in.Size > mx {
jsonErr(w, 413, "file too large")
return
}
f, err := fsys.Open(r.Context(), p)
if err != nil {
fsErr(w, err)
return
}
defer f.Close()
var body io.Reader = f
if mx > 0 { // stat can lie (size 0 on /proc); the cap must hold on the bytes actually sent
body = io.LimitReader(f, mx+1)
}
name := filepath.Base(p)
ct := mime.TypeByExtension(filepath.Ext(name))
if ct == "" {
@@ -1961,7 +1959,28 @@ func (s *Server) filesRead(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Disposition", "inline; filename*=UTF-8''"+url.PathEscape(name))
}
if in.Size > 0 && mx == 0 {
w.Header().Set("Content-Length", strconv.FormatInt(in.Size, 10))
// A real file of known size goes through net/http, which speaks Range: a video player
// can seek and a download can resume. /proc files say size 0 and take the plain path.
rs := files.NewSeeker(r.Context(), fsys, p, in.Size)
defer rs.Close()
if _, local := fsys.(files.Local); local { // cheap here, and turns "can't open" into a status rather than a cut-off body
if err := rs.Check(); err != nil {
fsErr(w, err)
return
}
}
http.ServeContent(w, r, name, time.UnixMilli(in.Mtime), rs)
return
}
f, err := fsys.Open(r.Context(), p)
if err != nil {
fsErr(w, err)
return
}
defer f.Close()
var body io.Reader = f
if mx > 0 { // stat can lie (size 0 on /proc); the cap must hold on the bytes actually sent
body = io.LimitReader(f, mx+1)
}
_, _ = io.Copy(w, body)
}
@@ -1975,7 +1994,14 @@ func (s *Server) filesWrite(w http.ResponseWriter, r *http.Request) {
if !ok {
return
}
if err := fsys.Write(r.Context(), p, io.LimitReader(r.Body, 8<<20)); err != nil {
// Whole in memory first: the write truncates the file, so a body over the cap must be
// refused before that, not cut short after it.
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 8<<20))
if err != nil {
jsonErr(w, 413, "too large to save here (8 MiB at most)")
return
}
if err := fsys.Write(r.Context(), p, bytes.NewReader(body)); err != nil {
fsErr(w, err)
return
}
@@ -2062,6 +2088,63 @@ func (s *Server) filesRename(w http.ResponseWriter, r *http.Request) {
})
}
// filesMove: POST {path, to} — to is the full destination path. A rename when the two
// ends share a filesystem (204); otherwise the copy-and-remove runs as a task (202 with the
// task), since a folder between datasets can take as long as it takes.
func (s *Server) filesMove(w http.ResponseWriter, r *http.Request) {
fsys, ok := s.fsFor(w, r)
if !ok {
return
}
var body struct{ Path, To string }
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
from, err := files.Clean(body.Path)
if err != nil {
jsonErr(w, 400, err.Error())
return
}
to, err := files.Clean(body.To)
if err != nil {
jsonErr(w, 400, err.Error())
return
}
if from == to {
jsonErr(w, 400, "it is already there")
return
}
if files.Inside(from, to) {
jsonErr(w, 400, "a folder can't be moved into itself")
return
}
// The kernel answers EXDEV for two paths on different filesystems before it looks for
// the source, so a missing file would otherwise become a task that fails on arrival.
if _, err := fsys.Stat(r.Context(), from); err != nil {
fsErr(w, err)
return
}
err = fsys.Rename(r.Context(), from, to)
if err == nil {
log.Printf("files: move %s -> %s (from %s)", from, to, clientIP(r))
w.WriteHeader(204)
return
}
if !files.IsCrossDevice(err) {
fsErr(w, err)
return
}
cmd := files.MoveCmd(context.Background(), fsys, s.self, from, to)
t, err := s.tasks.Start("files:move", "move "+filepath.Base(from)+" to "+filepath.Dir(to), cmd)
if err != nil {
jsonErr(w, 500, err.Error())
return
}
log.Printf("files: move %s -> %s across filesystems as task %s (from %s)", from, to, t.ID, clientIP(r))
jsonOK(w, 202, t.Snapshot(false))
}
func (s *Server) filesDelete(w http.ResponseWriter, r *http.Request) {
s.filesOp(w, r, func(fsys files.FS, body map[string]any, p string) error {
rec, _ := body["recursive"].(bool)
+77 -272
View File
@@ -18,11 +18,10 @@ package api
// machine would become unfindable at once. A remembered mount resolves with no container to
// run in, which is right: compose runs on the host against the host's copy of the file.
//
// A stack can also be deployed *from another machine*: Dockhand on the hub drives a node's
// Docker over tcp, so the node runs the containers and the hub holds the files. The node
// alone can only list such a stack read-only. The hub, asked about one of its nodes, fills
// in what the node couldn't (hubStacks): it resolves the files on itself and runs compose
// here with DOCKER_HOST pointed at the node.
// A stack is one machine's affair: its files and the Docker that runs it are on the same
// box, and that box's Ham Lab is the only thing that touches either. Asked about a node's
// stacks, a hub passes the request to the node's own Ham Lab and relays the answer; it holds
// no files for anyone else and never points compose at another machine's Docker.
import (
"bytes"
@@ -35,7 +34,6 @@ import (
"io"
"log"
"net/http"
"net/url"
"os"
"os/exec"
"path/filepath"
@@ -60,8 +58,6 @@ type stack struct {
Files []string `json:"files"` // every file it was deployed from, in order: -f a -f b
HostFiles []string `json:"host_files"` // the same, on the machine that holds them
Via string `json:"via"` // container compose runs inside; "" = on the host
Remote string `json:"remote"` // DOCKER_HOST compose is pointed at; "" = this machine's Docker
HeldBy string `json:"held_by"` // the machine holding the files, when it isn't the one running them
Editable bool `json:"editable"`
Note string `json:"note,omitempty"`
HasEnv bool `json:"has_env"`
@@ -282,7 +278,7 @@ func (s *Server) findStacks(ctx context.Context) ([]stack, error) {
st.resolve(maps)
stacks = append(stacks, *st)
}
stacks = append(stacks, scanHomes(ctx, s.homesOf(stacks, true), stacks)...)
stacks = append(stacks, scanHomes(ctx, s.homesOf(stacks), stacks)...)
sort.Slice(stacks, func(i, j int) bool { return stacks[i].Name < stacks[j].Name })
return stacks, nil
}
@@ -290,19 +286,23 @@ func (s *Server) findStacks(ctx context.Context) ([]stack, error) {
// ---- where stacks live ----
// A home is a directory of stacks, one folder each: where new ones are made, and where ones
// that aren't deployed (never were, or were taken down) are still found. Homes aren't
// configured; they are wherever this machine's stacks already are. With none at all, ~/stacks.
// that aren't deployed (never were, or were taken down) are still found. The default is
// Ham Lab's own: <data dir>/stacks, which is where everything this console manages lives.
// Any folder this machine already keeps stacks in — two or more deployed from it — is
// offered as well, so what was deployed before Ham Lab arrived stays editable and gains
// neighbours where it is, but it is never the default.
type stackHome struct {
Dir string `json:"dir"` // on the machine that holds the files
Dir string `json:"dir"` // on this machine
Inner string `json:"inner"` // the same directory as compose sees it (inside Via); Dir otherwise
Via string `json:"via"` // container compose runs inside
Remote string `json:"remote"` // DOCKER_HOST, when the containers run on another machine
HeldBy string `json:"held_by"` // the machine holding the files, when it isn't the one running them
Stacks int `json:"stacks"`
Default bool `json:"default"`
}
func (s *Server) homesOf(stacks []stack, withFallback bool) []stackHome {
// stacksDir is the home Ham Lab owns.
func (s *Server) stacksDir() string { return filepath.Join(s.cfg.DataDir(), "stacks") }
func (s *Server) homesOf(stacks []stack) []stackHome {
type tally struct {
home stackHome
deployed int
@@ -318,11 +318,10 @@ func (s *Server) homesOf(stacks []stack, withFallback bool) []stackHome {
if strings.Count(dir, "/") < 2 { // "/" or "/tmp": not somewhere to make folders in
continue
}
key := dir + "|" + st.Remote
t := by[key]
t := by[dir]
if t == nil {
t = &tally{home: stackHome{Dir: dir, Remote: st.Remote, HeldBy: st.HeldBy}, byVia: map[string]int{}, inner: map[string]string{}}
by[key] = t
t = &tally{home: stackHome{Dir: dir}, byVia: map[string]int{}, inner: map[string]string{}}
by[dir] = t
}
t.home.Stacks++
if len(st.Containers) > 0 {
@@ -335,11 +334,13 @@ func (s *Server) homesOf(stacks []stack, withFallback bool) []stackHome {
}
}
// A folder is a home when stacks are kept there as a habit: two or more deployed from it.
// One stack on its own (Dockhand's own compose file in ~/apps/dockhand) says nothing about
// where the next should go, and would have every compose file near it listed as a stack.
// New stacks are deployed the way most of their neighbours were.
var homes, lone []stackHome
for _, t := range by {
// One stack on its own (an orchestrator's own compose file in its app folder) says nothing
// about where the next should go, and would have every compose file near it listed as a
// stack — unless it is already in Ham Lab's folder. New stacks are deployed the way most
// of their neighbours were.
own := s.stacksDir()
var homes []stackHome
for dir, t := range by {
best, n := "", -1
for via, c := range t.byVia {
if c > n || (c == n && via > best) {
@@ -355,29 +356,23 @@ func (s *Server) homesOf(stacks []stack, withFallback bool) []stackHome {
if t.home.Inner == "" {
t.home.Inner = t.home.Dir
}
if t.deployed >= 2 {
if t.deployed >= 2 || dir == own {
homes = append(homes, t.home)
} else {
lone = append(lone, t.home)
}
}
if len(homes) == 0 {
homes = lone
}
sort.Slice(homes, func(i, j int) bool {
if (homes[i].Dir == own) != (homes[j].Dir == own) {
return homes[i].Dir == own
}
if homes[i].Stacks != homes[j].Stacks {
return homes[i].Stacks > homes[j].Stacks
}
return homes[i].Dir < homes[j].Dir
})
if len(homes) == 0 && withFallback {
if home, err := os.UserHomeDir(); err == nil {
homes = append(homes, stackHome{Dir: filepath.Join(home, "stacks"), Inner: filepath.Join(home, "stacks")})
if len(homes) == 0 || homes[0].Dir != own {
homes = append([]stackHome{{Dir: own, Inner: own}}, homes...)
}
}
if len(homes) > 0 {
homes[0].Default = true
}
return homes
}
@@ -394,6 +389,9 @@ func scanHomes(ctx context.Context, homes []stackHome, known []stack) []stack {
}
var out []stack
for _, h := range homes {
if _, err := os.Stat(h.Dir); errors.Is(err, os.ErrNotExist) {
continue // Ham Lab's own home before its first stack: nothing to find, and no sudo to spend finding it
}
args := []string{h.Dir, "-mindepth", "2", "-maxdepth", "2", "-type", "f", "("}
for i, n := range composeNames {
if i > 0 {
@@ -417,7 +415,7 @@ func scanHomes(ctx context.Context, homes []stackHome, known []stack) []stack {
continue
}
have[name], have[dir] = true, true
st := stack{Name: name, Containers: []string{}, Images: []string{}, HostFiles: []string{f}, HostFile: f, Via: h.Via, Remote: h.Remote, HeldBy: h.HeldBy, Editable: true, Note: "not deployed"}
st := stack{Name: name, Containers: []string{}, Images: []string{}, HostFiles: []string{f}, HostFile: f, Via: h.Via, Editable: true, Note: "not deployed"}
st.Files = []string{filepath.Join(h.Inner, filepath.Base(dir), filepath.Base(f))}
st.File = st.Files[0]
st.HasEnv = readable(st.envPath())
@@ -436,114 +434,23 @@ func contains(list []string, s string) bool {
return false
}
// hubStacks is a node's stack list with the hub's help: what the node could not find is
// looked for here, and if it is here — and the node's Docker answers over tcp, which is how
// it was deployed in the first place — the stack becomes editable from this hub.
func (s *Server) hubStacks(ctx context.Context, node string) ([]stack, error) {
code, b, err := s.nodes.Do(ctx, node, "GET", "/api/docker/stacks", nil, nil)
// stackFor finds the stack a request means.
func (s *Server) stackFor(ctx context.Context, r *http.Request) (stack, error) {
all, err := s.findStacks(ctx)
if err != nil {
return nil, err
}
var res struct{ Stacks []stack }
if code != 200 || json.Unmarshal(b, &res) != nil {
return nil, fmt.Errorf("%s answered %d", node, code)
}
remote, maps := "", []mountMap(nil)
for i := range res.Stacks {
st := &res.Stacks[i]
if st.Editable || len(st.Files) == 0 {
continue
}
if remote == "" {
remote, maps = s.nodeDocker(ctx, node), mountsFor(ctx, s.store)
}
if remote == "-" {
continue
}
note := st.Note
if st.resolve(maps); !st.Editable {
st.Note = note
continue
}
st.Remote, st.HeldBy = remote, s.hostName()
}
var held []stack
for _, st := range res.Stacks {
if st.HeldBy != "" {
held = append(held, st)
}
}
res.Stacks = append(res.Stacks, scanHomes(ctx, s.homesOf(held, false), res.Stacks)...)
sort.Slice(res.Stacks, func(i, j int) bool { return res.Stacks[i].Name < res.Stacks[j].Name })
return res.Stacks, nil
}
var (
nodeDockerMu sync.Mutex
nodeDockerOK = map[string]string{} // node → "tcp://host:2375", or "-" when it doesn't answer
)
// nodeDocker is the node's Docker endpoint as Dockhand-style deployments use it, if it answers.
func (s *Server) nodeDocker(ctx context.Context, node string) string {
nodeDockerMu.Lock()
defer nodeDockerMu.Unlock()
if v, ok := nodeDockerOK[node]; ok && v != "-" {
return v
}
ep := "-"
for _, n := range s.nodes.List() {
if n.Name != node {
continue
}
if u, err := url.Parse(n.URL); err == nil && u.Hostname() != "" {
cand := "tcp://" + u.Hostname() + ":2375"
c, cancel := context.WithTimeout(ctx, 5*time.Second)
if exec.CommandContext(c, dockerTool(), "-H", cand, "version", "--format", "{{.Server.Version}}").Run() == nil {
ep = cand
}
cancel()
}
}
nodeDockerOK[node] = ep
return ep
}
// stackFor finds the stack a request means: on this machine, or (on a hub route) one of a
// node's stacks that this hub holds the files for. handled is false when the request should
// simply be passed on to the node.
func (s *Server) stackFor(ctx context.Context, r *http.Request) (st stack, handled bool, err error) {
var all []stack
if node := r.PathValue("node"); node != "" {
all, err = s.hubStacks(ctx, node)
} else {
all, err = s.findStacks(ctx)
}
if err != nil {
return stack{}, true, err
return stack{}, err
}
for _, x := range all {
if x.Name == r.PathValue("name") {
if r.PathValue("node") != "" && x.Remote == "" {
return x, false, nil // the node can serve this one itself
}
return x, true, nil
return x, nil
}
}
return stack{}, true, errors.New("no such stack")
return stack{}, errors.New("no such stack")
}
func (s *Server) stacksList(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
defer cancel()
if node := r.PathValue("node"); node != "" {
stacks, err := s.hubStacks(ctx, node)
if err != nil {
s.proxyNode(w, r) // let the node's own answer (or failure) through
return
}
jsonOK(w, 200, map[string]any{"stacks": stacks})
return
}
if dockerTool() == "" { // no Docker is an empty answer, not a failure
jsonOK(w, 200, map[string]any{"stacks": []stack{}})
return
@@ -587,11 +494,7 @@ func target(r *http.Request, st stack) (host string, index int, ok bool) {
func (s *Server) stackFile(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
defer cancel()
st, handled, err := s.stackFor(ctx, r)
if !handled {
s.proxyNode(w, r)
return
}
st, err := s.stackFor(ctx, r)
if err != nil || !st.Editable {
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
return
@@ -610,8 +513,8 @@ func (s *Server) stackFile(w http.ResponseWriter, r *http.Request) {
jsonOK(w, 200, map[string]any{"stack": st, "text": string(b), "sum": sumOf(b), "index": i})
}
// composeCmd is `docker compose …` for this stack, run where its files make sense and
// pointed at the Docker that runs its containers. files are given as compose will see them.
// composeCmd is `docker compose …` for this stack, run where its files make sense. files are
// given as compose will see them.
func composeCmd(ctx context.Context, st stack, files []string, args ...string) *exec.Cmd {
return compose.Cmd(ctx, specOf(st, files), args...)
}
@@ -619,7 +522,7 @@ func composeCmd(ctx context.Context, st stack, files []string, args ...string) *
// specOf is a stack as the compose runner needs it — and as a scheduled update stores it,
// so the cron job and the console invoke compose identically.
func specOf(st stack, files []string) compose.Spec {
return compose.Spec{Project: st.Name, Files: files, Via: st.Via, Remote: st.Remote, Tool: dockerTool()}
return compose.Spec{Project: st.Name, Files: files, Via: st.Via, Tool: dockerTool()}
}
// seenFiles is the stack's file list as compose runs it, with file i replaced by a candidate.
@@ -647,11 +550,7 @@ func (s *Server) writeAsOwner(ctx context.Context, path, text string) error {
func (s *Server) stackSave(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
defer cancel()
st, handled, err := s.stackFor(ctx, r)
if !handled {
s.proxyNode(w, r)
return
}
st, err := s.stackFor(ctx, r)
if err != nil || !st.Editable {
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
return
@@ -723,11 +622,7 @@ func (s *Server) stackSave(w http.ResponseWriter, r *http.Request) {
func (s *Server) stackDiff(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
defer cancel()
st, handled, err := s.stackFor(ctx, r)
if !handled {
s.proxyNode(w, r)
return
}
st, err := s.stackFor(ctx, r)
if err != nil || !st.Editable {
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
return
@@ -777,11 +672,7 @@ func maskEnv(text string) string {
func (s *Server) stackAction(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
defer cancel()
st, handled, err := s.stackFor(ctx, r)
if !handled {
s.proxyNode(w, r)
return
}
st, err := s.stackFor(ctx, r)
if err != nil || !st.Editable {
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
return
@@ -794,17 +685,10 @@ func (s *Server) stackAction(w http.ResponseWriter, r *http.Request) {
flags := priv.Quote(append([]string{"-p", st.Name}, fl...))
base := func(args string) string {
if st.Via != "" {
env := ""
if st.Remote != "" {
env = "-e " + priv.Quote([]string{"DOCKER_HOST=" + st.Remote}) + " "
// The CLI inside the orchestrator's container is its own `docker`, whatever this host runs.
return fmt.Sprintf("%s exec %s docker compose %s %s", dockerTool(), priv.Quote([]string{st.Via}), flags, args)
}
return fmt.Sprintf("%s exec %s%s docker compose %s %s", dockerTool(), env, priv.Quote([]string{st.Via}), flags, args)
}
env := ""
if st.Remote != "" { // through env(1): a quoted NAME=value in front of a command is a command name to sh, not an assignment
env = "env " + priv.Quote([]string{"DOCKER_HOST=" + st.Remote}) + " "
}
return fmt.Sprintf("cd %s && %s%s compose %s %s", priv.Quote([]string{filepath.Dir(files[0])}), env, dockerTool(), flags, args)
return fmt.Sprintf("cd %s && %s compose %s %s", priv.Quote([]string{filepath.Dir(files[0])}), dockerTool(), flags, args)
}
// After a deploy that worked, keep a copy of exactly what was deployed: the next diff is against it.
remember := ""
@@ -860,7 +744,7 @@ func (s *Server) stackAction(w http.ResponseWriter, r *http.Request) {
_ = exec.CommandContext(ctx, dockerTool(), "exec", via, "pkill", "-TERM", "-f", pattern).Run()
})
}
stackUpdatesForget(r.PathValue("node") + "/" + st.Name)
stackUpdatesForget(st.Name)
log.Printf("stack %s: %s started as task %s (from %s)", st.Name, r.PathValue("action"), t.ID, clientIP(r))
jsonOK(w, 202, t.Snapshot(false))
}
@@ -885,20 +769,13 @@ var (
func stackUpdatesForget(key string) { updMu.Lock(); delete(updCache, key); updMu.Unlock() }
// checkImages asks the registry whether each image is still the one we hold. remote is a
// DOCKER_HOST for a machine whose images live elsewhere, "" for this one. Each image is one
// registry request, which Docker Hub counts — so this is only ever called when asked for.
func checkImages(ctx context.Context, images []string, remote string) []imageCheck {
local := func(args ...string) ([]byte, error) { // the Docker that holds the images
if remote != "" {
args = append([]string{"-H", remote}, args...)
}
return dockerRun(ctx, args...)
}
// checkImages asks the registry whether each image is still the one we hold. Each image is
// one registry request, which Docker Hub counts — so this is only ever called when asked for.
func checkImages(ctx context.Context, images []string) []imageCheck {
res := []imageCheck{}
for _, img := range images {
ic := imageCheck{Image: img, State: "unknown"}
out, err := local("image", "inspect", "-f", "{{json .RepoDigests}}", img)
out, err := dockerRun(ctx, "image", "inspect", "-f", "{{json .RepoDigests}}", img)
var have []string
if err != nil || json.Unmarshal(bytes.TrimSpace(out), &have) != nil || len(have) == 0 {
ic.Detail = "built or loaded here, not pulled from a registry"
@@ -930,31 +807,28 @@ func checkImages(ctx context.Context, images []string, remote string) []imageChe
}
// machineUpdates checks every image on this machine in one pass, so the container list can
// be coloured and the fleet can be asked "what is behind?" without walking stack by stack.
// be coloured and "what is behind?" answered without walking stack by stack.
func (s *Server) machineUpdates(w http.ResponseWriter, r *http.Request) {
node := r.PathValue("node")
if dockerTool() == "" { // no Docker is an empty answer, not a failure
jsonOK(w, 200, map[string]any{"images": []imageCheck{}, "checked": 0, "stacks": map[string][]string{}})
return
}
ctx, cancel := context.WithTimeout(r.Context(), 10*time.Minute)
defer cancel()
stacks, err := s.findStacks(ctx)
if node != "" {
stacks, err = s.hubStacks(ctx, node)
}
if err != nil {
jsonErr(w, 500, err.Error())
return
}
// one entry per image, however many stacks use it
type want struct{ remote string }
imgs := map[string]want{}
byImage := map[string][]string{} // image -> stacks
for _, st := range stacks {
for _, img := range st.Images {
imgs[img] = want{remote: st.Remote}
byImage[img] = append(byImage[img], st.Name)
}
}
refresh := r.URL.Query().Get("refresh") == "1"
key := "machine:" + node
key := "machine"
updMu.Lock()
c, ok := updCache[key]
updMu.Unlock()
@@ -967,15 +841,11 @@ func (s *Server) machineUpdates(w http.ResponseWriter, r *http.Request) {
return
}
var list []string
remote := ""
for img, wnt := range imgs {
for img := range byImage {
list = append(list, img)
if wnt.remote != "" {
remote = wnt.remote
}
}
sort.Strings(list)
res := checkImages(ctx, list, remote)
res := checkImages(ctx, list)
updMu.Lock()
updCache[key] = updAnswer{time.Now(), res}
updMu.Unlock()
@@ -988,16 +858,12 @@ func (s *Server) machineUpdates(w http.ResponseWriter, r *http.Request) {
func (s *Server) stackUpdates(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
defer cancel()
st, handled, err := s.stackFor(ctx, r)
if !handled {
s.proxyNode(w, r)
return
}
st, err := s.stackFor(ctx, r)
if err != nil {
jsonErr(w, 404, err.Error())
return
}
key := r.PathValue("node") + "/" + st.Name
key := st.Name
updMu.Lock()
c, ok := updCache[key]
updMu.Unlock()
@@ -1010,84 +876,38 @@ func (s *Server) stackUpdates(w http.ResponseWriter, r *http.Request) {
jsonOK(w, 200, map[string]any{"images": []imageCheck{}, "checked": 0})
return
}
res := checkImages(ctx, st.Images, st.Remote)
res := checkImages(ctx, st.Images)
updMu.Lock()
updCache[key] = updAnswer{time.Now(), res}
updMu.Unlock()
jsonOK(w, 200, map[string]any{"images": res, "checked": time.Now().UnixMilli()})
}
// taskOnHub serves a task this hub is running for one of its nodes' stacks; anything else
// under /api/nodes/<node>/tasks belongs to the node.
func (s *Server) taskOnHub(local http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if _, ok := s.tasks.Get(r.PathValue("id")); ok {
local(w, r)
return
}
s.proxyNode(w, r)
}
}
// ---- making and unmaking stacks ----
var stackNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]{0,62}$`)
// homesFor is where a new stack could go, seen from this request: this machine's homes, or —
// on a hub route — the node's own homes plus the ones this hub holds for it.
func (s *Server) homesFor(ctx context.Context, r *http.Request) ([]stackHome, error) {
node := r.PathValue("node")
if node == "" {
// homesFor is where a new stack could go on this machine.
func (s *Server) homesFor(ctx context.Context) ([]stackHome, error) {
stacks, err := s.findStacks(ctx)
if err != nil {
return nil, err
}
return s.homesOf(stacks, true), nil
}
stacks, err := s.hubStacks(ctx, node)
if err != nil {
return nil, err
}
var held []stack
for _, st := range stacks {
if st.HeldBy != "" {
held = append(held, st)
}
}
homes := s.homesOf(held, false)
if code, b, err := s.nodes.Do(ctx, node, "GET", "/api/docker/stacks/homes", nil, nil); err == nil && code == 200 {
var res struct{ Homes []stackHome }
if json.Unmarshal(b, &res) == nil {
for _, h := range res.Homes {
if len(homes) > 0 && h.Stacks == 0 {
continue // the node's empty ~/stacks is only worth offering when there is nothing else
}
h.Default = false
homes = append(homes, h)
}
}
}
if len(homes) > 0 {
homes[0].Default = true
}
return homes, nil
return s.homesOf(stacks), nil
}
func (s *Server) stackHomes(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
defer cancel()
if dockerTool() == "" && r.PathValue("node") == "" {
if dockerTool() == "" {
jsonOK(w, 200, map[string]any{"homes": []stackHome{}})
return
}
homes, err := s.homesFor(ctx, r)
homes, err := s.homesFor(ctx)
if err != nil {
jsonErr(w, 500, err.Error())
return
}
if homes == nil { // a machine with no Docker and nothing held for it: an empty list, never null
homes = []stackHome{}
}
jsonOK(w, 200, map[string]any{"homes": homes})
}
@@ -1114,7 +934,7 @@ func (s *Server) stackCreate(w http.ResponseWriter, r *http.Request) {
jsonErr(w, 400, "the compose file can't be empty")
return
}
homes, err := s.homesFor(ctx, r)
homes, err := s.homesFor(ctx)
if err != nil {
jsonErr(w, 500, err.Error())
return
@@ -1129,17 +949,7 @@ func (s *Server) stackCreate(w http.ResponseWriter, r *http.Request) {
jsonErr(w, 400, "that isn't one of the places this machine keeps stacks")
return
}
if r.PathValue("node") != "" && home.HeldBy == "" { // the node's own folder: the node makes it
r.Body, r.ContentLength = io.NopCloser(bytes.NewReader(raw)), int64(len(raw))
s.proxyNode(w, r)
return
}
var all []stack
if node := r.PathValue("node"); node != "" {
all, _ = s.hubStacks(ctx, node)
} else {
all, _ = s.findStacks(ctx)
}
all, _ := s.findStacks(ctx)
for _, st := range all {
if st.Name == name {
jsonErr(w, 409, "there is already a stack called "+name)
@@ -1201,7 +1011,7 @@ func (s *Server) stackCreate(w http.ResponseWriter, r *http.Request) {
_ = sudo("chmod", "600", filepath.Join(dir, ".env")) // it is where the secrets go
}
st := stack{Name: name, Containers: []string{}, Images: []string{}, HostFiles: []string{host}, HostFile: host, Files: []string{filepath.Join(home.Inner, name, "compose.yaml")},
Via: home.Via, Remote: home.Remote, HeldBy: home.HeldBy, Editable: true, Note: "not deployed", HasEnv: strings.TrimSpace(body.Env) != ""}
Via: home.Via, Editable: true, Note: "not deployed", HasEnv: strings.TrimSpace(body.Env) != ""}
st.File = st.Files[0]
if msg, err := composeCmd(ctx, st, seenFiles(st, -1, ""), "config", "-q").CombinedOutput(); err != nil {
undo()
@@ -1220,12 +1030,7 @@ func (s *Server) stackDelete(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 4*time.Minute)
defer cancel()
raw, _ := io.ReadAll(io.LimitReader(r.Body, 4096))
st, handled, err := s.stackFor(ctx, r)
if !handled {
r.Body, r.ContentLength = io.NopCloser(bytes.NewReader(raw)), int64(len(raw))
s.proxyNode(w, r)
return
}
st, err := s.stackFor(ctx, r)
if err != nil || !st.Editable {
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
return
@@ -1263,7 +1068,7 @@ func (s *Server) stackDelete(w http.ResponseWriter, r *http.Request) {
}
}
_ = run("rmdir", dir) // only if nothing else was in it
stackUpdatesForget(r.PathValue("node") + "/" + st.Name)
stackUpdatesForget(st.Name)
log.Printf("stack %s: taken down and %d file(s) moved to %s (from %s)", st.Name, moved, trash, clientIP(r))
jsonOK(w, 200, map[string]any{"trash": trash, "moved": moved})
}
+35
View File
@@ -7,6 +7,7 @@ import (
"testing"
"hamlab/internal/compose"
"hamlab/internal/config"
"hamlab/internal/store"
)
@@ -119,3 +120,37 @@ func TestRepointSurvivesAMissingContainer(t *testing.T) {
t.Errorf("unresolvable spec was changed: %+v", got)
}
}
// Where new stacks go: Ham Lab's own <data dir>/stacks is always offered, first and as the
// default, whether or not anything is in it yet. A folder this machine already keeps stacks
// in — two or more deployed from it — is offered too, but never ahead of Ham Lab's own.
func TestHomesDefaultToDataDir(t *testing.T) {
root := t.TempDir()
cfg := config.Default(filepath.Join(root, "data", "config.json"))
s := &Server{cfg: cfg}
own := filepath.Join(root, "data", "stacks")
legacy := filepath.Join(root, "home", "stacks", "box")
deployed := func(dir, name string) stack {
f := filepath.Join(dir, name, "compose.yaml")
return stack{Name: name, Containers: []string{name + "-1"}, Files: []string{f}, File: f, HostFiles: []string{f}, HostFile: f, Editable: true}
}
// Nothing deployed anywhere: one home, Ham Lab's own.
homes := s.homesOf(nil)
if len(homes) != 1 || homes[0].Dir != own || !homes[0].Default {
t.Fatalf("empty machine: homes = %+v, want just %s as default", homes, own)
}
// Two stacks deployed from a pre-existing folder: it is offered, after Ham Lab's own.
homes = s.homesOf([]stack{deployed(legacy, "a"), deployed(legacy, "b")})
if len(homes) != 2 || homes[0].Dir != own || !homes[0].Default || homes[1].Dir != legacy || homes[1].Default || homes[1].Stacks != 2 {
t.Fatalf("legacy folder: homes = %+v, want [%s (default), %s]", homes, own, legacy)
}
// One lone stack elsewhere says nothing about where the next should go; one in Ham Lab's
// own folder is already home.
homes = s.homesOf([]stack{deployed(filepath.Join(root, "home", "apps"), "orchestrator"), deployed(own, "c")})
if len(homes) != 1 || homes[0].Dir != own || homes[0].Stacks != 1 || !homes[0].Default {
t.Fatalf("lone stacks: homes = %+v, want just %s", homes, own)
}
}
+418
View File
@@ -0,0 +1,418 @@
package api
// Previews for the Files tab: GET /api/files/thumb hands back one small JPEG of an image or
// a video, made by internal/thumbs and kept in <data dir>/thumbs.db. The work is bounded
// two ways — a handful of ffmpegs at once, and one per file however many rows ask for it —
// because a folder of a thousand photos asks for every row the moment previews go on.
//
// Root mode (?sudo=1) makes the preview through the same root helper the file reads use,
// so a file only root can read gets a preview too, and the probe runs as root as well: root
// opens every render node, whatever groups the daemon's user has.
import (
"context"
"errors"
"log"
"net/http"
"os"
"os/exec"
"path/filepath"
"runtime"
"strconv"
"strings"
"sync"
"time"
"hamlab/internal/media"
"hamlab/internal/priv"
)
const thumbSize = 256
const thumbTimeout = 30 * time.Second
const capsTTL = 10 * time.Minute // how long a probe is believed; a refresh after an install or a group change does not wait
type thumbSvc struct {
cache *media.Cache
self string
sem chan struct{}
mu sync.Mutex
caps map[bool]*capsEntry // keyed by root
calls map[string]*thumbCall
}
type capsEntry struct {
mu sync.Mutex
caps media.Caps
err error
at time.Time
}
type thumbCall struct {
done chan struct{}
res media.Result
err error
}
func newThumbSvc(dataDir, self string) *thumbSvc {
n := runtime.NumCPU() / 2 // each ffmpeg is itself threaded; previews must not take the whole machine
if n < 2 {
n = 2
}
if n > 4 {
n = 4
}
return &thumbSvc{cache: media.NewCache(filepath.Join(dataDir, "thumbs.db")), self: self, sem: make(chan struct{}, n), caps: map[bool]*capsEntry{}, calls: map[string]*thumbCall{}}
}
// capsFor is what previews can be made with, probed once and kept for a while.
func (t *thumbSvc) capsFor(ctx context.Context, root, refresh bool) (media.Caps, error) {
t.mu.Lock()
e := t.caps[root]
if e == nil {
e = &capsEntry{}
t.caps[root] = e
}
t.mu.Unlock()
e.mu.Lock()
defer e.mu.Unlock()
if !refresh && !e.at.IsZero() && time.Since(e.at) < capsTTL {
return e.caps, e.err
}
if root {
e.caps, e.err = media.ProbeRoot(ctx, t.self)
} else {
e.caps, e.err = media.Probe(ctx), nil
}
e.at = time.Now()
mode := "as " + map[bool]string{false: "the daemon", true: "root"}[root]
switch {
case e.err != nil:
log.Printf("thumbs: probe %s failed: %v", mode, e.err)
case e.caps.Error != "":
log.Printf("thumbs: %s: %s", mode, e.caps.Error)
default:
log.Printf("thumbs: %s: ffmpeg %s, %s", mode, e.caps.Version, describeCaps(e.caps))
}
return e.caps, e.err
}
// describeCaps is the one line that says how previews get made here.
func describeCaps(c media.Caps) string {
if len(c.GPUs) == 0 {
return "CPU only"
}
parts := make([]string, 0, len(c.GPUs))
for _, g := range c.GPUs {
name := g.Name
if name == "" {
name = g.Device
}
p := name + " (" + g.API
if len(g.Codecs) > 0 {
p += ": " + strings.Join(g.Codecs, ", ")
}
parts = append(parts, p+")")
}
return "GPU " + strings.Join(parts, ", ")
}
// make produces a preview, sharing the work when the same file is asked for twice at once.
func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (media.Result, error) {
t.mu.Lock()
if c, ok := t.calls[key]; ok {
t.mu.Unlock()
select {
case <-c.done:
return c.res, c.err
case <-ctx.Done():
return media.Result{}, ctx.Err()
}
}
c := &thumbCall{done: make(chan struct{})}
t.calls[key] = c
t.mu.Unlock()
defer func() {
t.mu.Lock()
delete(t.calls, key)
t.mu.Unlock()
close(c.done)
}()
// The work itself runs on a context of its own: the first requester going away must not
// cancel a preview three other rows are waiting for.
wctx, cancel := context.WithTimeout(context.Background(), thumbTimeout+5*time.Second)
defer cancel()
select {
case t.sem <- struct{}{}:
defer func() { <-t.sem }()
case <-ctx.Done():
c.err = ctx.Err()
return c.res, c.err
}
caps, err := t.capsFor(wctx, root, false)
if err != nil {
c.err = err
return c.res, c.err
}
o := media.Opts{Size: thumbSize, Timeout: thumbTimeout}
if root {
c.res, c.err = media.MakeRoot(wctx, t.self, caps, path, o)
} else {
c.res, c.err = media.Make(wctx, caps, path, o)
}
return c.res, c.err
}
// ---- handlers ----
// filesThumb: GET /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1]. v is the client's idea of
// the file, from its listing; it makes a cache hit free of a stat, and a changed file a new URL.
func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
fsys, ok := s.fsFor(w, r)
if !ok {
return
}
p, ok := cleanQuery(w, r, "path")
if !ok {
return
}
if media.Kind(p) == "" {
jsonErr(w, 415, "no preview for this kind of file")
return
}
root := r.URL.Query().Get("sudo") == "1"
if mt, sz, ok := parseV(r.URL.Query().Get("v")); ok {
key := media.Key(p, sz, mt)
if r.Header.Get("If-None-Match") == `"`+key+`"` {
w.WriteHeader(304)
return
}
if s.serveCachedThumb(w, key) {
return
}
}
in, err := fsys.Stat(r.Context(), p)
if err != nil {
fsErr(w, err)
return
}
if in.Type == "dir" || in.Type == "other" || in.Broken {
jsonErr(w, 400, "not a regular file")
return
}
key := media.Key(p, in.Size, in.Mtime)
if s.serveCachedThumb(w, key) {
return
}
res, err := s.thumbs.make(r.Context(), root, key, p)
if err != nil {
var de *media.DecodeError
switch {
case errors.As(err, &de):
if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Failure: de.Msg}); err != nil {
log.Printf("thumbs: cache: %v", err)
}
log.Printf("thumbs: %s: %s", p, de.Msg)
jsonErr(w, 422, de.Msg)
case errors.Is(err, context.DeadlineExceeded):
log.Printf("thumbs: %s: timed out", p)
jsonErr(w, 504, "preview timed out")
case errors.Is(err, context.Canceled):
jsonErr(w, 499, "cancelled")
default:
jsonErr(w, 503, err.Error())
}
return
}
if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}); err != nil {
log.Printf("thumbs: cache: %v", err)
}
writeThumb(w, key, res.JPEG, res.Engine)
}
// serveCachedThumb answers from the cache when it can. A cache that cannot be read is a
// miss: the preview is made again rather than refused.
func (s *Server) serveCachedThumb(w http.ResponseWriter, key string) bool {
e, found, err := s.thumbs.cache.Get(key)
if err != nil {
log.Printf("thumbs: cache: %v", err)
}
switch {
case !found:
return false
case e.Failure != "":
jsonErr(w, 422, e.Failure)
default:
writeThumb(w, key, e.JPEG, "cache ("+e.Engine+")")
}
return true
}
func writeThumb(w http.ResponseWriter, key string, jpeg []byte, engine string) {
w.Header().Set("Content-Type", "image/jpeg")
w.Header().Set("Content-Length", strconv.Itoa(len(jpeg)))
w.Header().Set("Cache-Control", "private, max-age=604800")
w.Header().Set("ETag", `"`+key+`"`)
w.Header().Set("X-Hamlab-Thumb", engine)
w.Header().Set("X-Content-Type-Options", "nosniff")
_, _ = w.Write(jpeg)
}
func parseV(v string) (mtime, size int64, ok bool) {
a, b, found := strings.Cut(v, "-")
if !found {
return 0, 0, false
}
mtime, err1 := strconv.ParseInt(a, 10, 64)
size, err2 := strconv.ParseInt(b, 10, 64)
return mtime, size, err1 == nil && err2 == nil
}
type thumbStatus struct {
OK bool `json:"ok"`
Engine string `json:"engine"` // gpu | cpu | ""
Summary string `json:"summary"`
FFmpeg string `json:"ffmpeg,omitempty"`
GPUs []media.GPU `json:"gpus"`
Unusable []media.GPU `json:"unusable,omitempty"`
Error string `json:"error,omitempty"`
Install *thumbInstall `json:"install,omitempty"` // ffmpeg is missing and this can put it on
GPUFix bool `json:"gpu_fix,omitempty"` // a render node is there but closed to the daemon's user, and this can open it
User string `json:"user"`
Cache thumbCache `json:"cache"`
}
type thumbCache struct {
Path string `json:"path"`
Entries int64 `json:"entries"`
Bytes int64 `json:"bytes"`
}
type thumbInstall struct {
PM string `json:"pm"`
Package string `json:"package"`
}
// thumbsStatus: GET /api/files/thumbs[?sudo=1][&refresh=1] — how previews get made on this machine.
func (s *Server) thumbsStatus(w http.ResponseWriter, r *http.Request) {
root := r.URL.Query().Get("sudo") == "1"
if root && s.fsSudo == nil {
jsonErr(w, 403, "root access needs passwordless sudo on the host")
return
}
caps, err := s.thumbs.capsFor(r.Context(), root, r.URL.Query().Get("refresh") == "1")
st := thumbStatus{GPUs: []media.GPU{}, User: s.user, Cache: thumbCache{Path: s.thumbs.cache.Path}}
st.Cache.Entries, st.Cache.Bytes = s.thumbs.cache.Stats()
if root {
st.User = "root"
}
if err != nil {
st.Error = err.Error()
jsonOK(w, 200, st)
return
}
st.FFmpeg, st.GPUs, st.Unusable, st.Error = caps.Version, caps.GPUs, caps.Unusable, caps.Error
if st.GPUs == nil {
st.GPUs = []media.GPU{}
}
st.OK = caps.Error == ""
switch {
case !st.OK:
if pkg := ffmpegPackage(s.updates.PM()); pkg != "" && priv.Available() {
st.Install = &thumbInstall{PM: s.updates.PM(), Package: pkg}
}
st.Summary = st.Error
case len(caps.GPUs) > 0:
st.Engine = "gpu"
st.Summary = describeCaps(caps) + " · images on the CPU"
default:
st.Engine = "cpu"
st.Summary = "CPU only"
}
if !root {
for _, g := range caps.Unusable {
if g.Permission && gpuFixPossible() {
st.GPUFix = true
}
}
}
jsonOK(w, 200, st)
}
// ffmpegPackage is what to install for ffmpeg on a package manager. Fedora ships it as
// ffmpeg-free; the others under its own name.
func ffmpegPackage(pm string) string {
switch pm {
case "":
return ""
case "dnf":
if b, _ := os.ReadFile("/etc/os-release"); strings.Contains(string(b), "ID=fedora") {
return "ffmpeg-free"
}
}
return "ffmpeg"
}
const hamlabUnit = "/etc/systemd/system/hamlab.service"
// gpuFixPossible: the daemon is an ordinary user under systemd, with root to hand.
func gpuFixPossible() bool {
if os.Geteuid() == 0 || !priv.Available() {
return false
}
_, err := os.Stat(hamlabUnit)
return err == nil
}
// thumbsInstall: POST /api/files/thumbs/install — put ffmpeg on with the machine's package manager.
func (s *Server) thumbsInstall(w http.ResponseWriter, r *http.Request) {
pkg := ffmpegPackage(s.updates.PM())
if pkg == "" {
jsonErr(w, 400, "no supported package manager on this machine")
return
}
t, err := s.updates.Install([]string{pkg})
if err != nil {
jsonErr(w, 400, err.Error())
return
}
log.Printf("thumbs: installing %s as task %s (from %s)", pkg, t.ID, clientIP(r))
jsonOK(w, 202, t.Snapshot(false))
}
// gpuAccessScript gives hamlab.service the groups that own the render nodes, by a drop-in,
// then restarts it: a running process cannot gain a group.
const gpuAccessScript = `set -e
unit=` + hamlabUnit + `
[ -f "$unit" ] || { echo "no $unit: this machine does not run Ham Lab under systemd"; exit 1; }
groups=""
for g in render video; do
if getent group "$g" >/dev/null 2>&1; then groups="$groups $g"; fi
done
[ -n "$groups" ] || { echo "neither a render nor a video group exists here"; exit 1; }
mkdir -p "$unit.d"
printf '[Service]\nSupplementaryGroups=%s\n' "${groups# }" > "$unit.d/gpu.conf"
echo "hamlab.service: supplementary groups${groups} ($unit.d/gpu.conf)"
systemctl daemon-reload
echo 'Ham Lab restarts in 3 seconds to pick up its new groups; the page reconnects by itself.'
systemd-run --quiet --on-active=3 --unit=hamlab-gpu-restart systemctl restart hamlab.service
`
// thumbsGPUAccess: POST /api/files/thumbs/gpu-access — open the render nodes to the daemon.
func (s *Server) thumbsGPUAccess(w http.ResponseWriter, r *http.Request) {
if !gpuFixPossible() {
jsonErr(w, 400, "nothing to do: Ham Lab is root, has no root, or is not under systemd here")
return
}
if t, busy := s.tasks.RunningOf("thumbs:gpu"); busy { // a second run would only collide with the first's restart timer
jsonErr(w, 409, "already being done (task "+t.ID+")")
return
}
full := priv.Wrap("sh", "-c", gpuAccessScript)
t, err := s.tasks.Start("thumbs:gpu", "give Ham Lab access to the GPU", exec.Command(full[0], full[1:]...))
if err != nil {
jsonErr(w, 400, err.Error())
return
}
log.Printf("thumbs: GPU access drop-in as task %s, restart follows (from %s)", t.ID, clientIP(r))
jsonOK(w, 202, t.Snapshot(false))
}
+71
View File
@@ -48,6 +48,7 @@ func (s *Server) virtOverview(w http.ResponseWriter, r *http.Request) {
imgs, _ := s.virt.Images(ctx)
out["images"] = imgs
out["pools"] = s.virt.Pools(ctx)
out["default_pool"] = s.virt.DefaultPool(ctx)
out["networks"] = s.virt.Networks(ctx)
}
isos := s.virt.Library(s.virt.ISOsDir())
@@ -285,6 +286,9 @@ func (s *Server) virtInstance(w http.ResponseWriter, r *http.Request) {
jsonErr(w, 400, "bad name")
return
}
if !s.virtReady(w, r) {
return
}
list, err := s.virt.Instances(r.Context())
if err != nil {
jsonErr(w, 502, err.Error())
@@ -1094,3 +1098,70 @@ func (s *Server) virtSSHKeys(w http.ResponseWriter, _ *http.Request) {
}
jsonOK(w, 200, map[string]any{"keys": keys, "user": s.user})
}
// ---- locations: storage pools, added as plain-files pools at a path of his choosing ----
func (s *Server) virtPoolCreate(w http.ResponseWriter, r *http.Request) {
var body struct {
Name string `json:"name"`
Path string `json:"path"`
}
_ = readBody(r, &body, 8192)
p := filepath.Clean(body.Path)
name := strings.TrimSpace(body.Name)
if name == "" {
name = sensors.Seg(filepath.Base(p))
}
if !incus.ValidName(name) {
jsonErr(w, 400, "name: letters, digits and dashes, starting with a letter")
return
}
if !filepath.IsAbs(p) || p == "/" {
jsonErr(w, 400, "path: an absolute path to a folder on this machine")
return
}
if !s.virtReady(w, r) {
return
}
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Minute)
defer cancel()
if st, err := os.Stat(p); err == nil && !st.IsDir() {
jsonErr(w, 400, p+" is a file, not a folder")
return
} else if err != nil {
mk := priv.Wrap("mkdir", "-p", p) // Incus wants the folder there; it fills it as root
if out, err := exec.CommandContext(ctx, mk[0], mk[1:]...).CombinedOutput(); err != nil {
jsonErr(w, 500, "could not make "+p+": "+strings.TrimSpace(string(out)))
return
}
}
if out, err := s.virt.Cmd(ctx, "storage", "create", name, "dir", "source="+p).CombinedOutput(); err != nil {
jsonErr(w, 409, strings.TrimPrefix(strings.TrimSpace(string(out)), "Error: "))
return
}
log.Printf("virt: location %s added at %s (from %s)", name, p, clientIP(r))
jsonOK(w, 200, map[string]string{"name": name, "path": p})
}
func (s *Server) virtPoolDelete(w http.ResponseWriter, r *http.Request) {
var body struct {
Name string `json:"name"`
}
_ = readBody(r, &body, 4096)
if !incus.ValidName(body.Name) {
jsonErr(w, 400, "bad name")
return
}
if !s.virtReady(w, r) {
return
}
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Minute)
defer cancel()
// Incus refuses a pool anything still uses; that refusal is the answer
if out, err := s.virt.Cmd(ctx, "storage", "delete", body.Name).CombinedOutput(); err != nil {
jsonErr(w, 409, strings.TrimPrefix(strings.TrimSpace(string(out)), "Error: "))
return
}
log.Printf("virt: location %s removed (from %s)", body.Name, clientIP(r))
w.WriteHeader(204)
}
+9 -1
View File
@@ -265,7 +265,15 @@ func (s *Server) zfsSnapshotDelete(w http.ResponseWriter, r *http.Request) {
return
}
if err := s.storage.DestroySnapshot(r.Context(), body.Name, body.Recursive); err != nil {
jsonErr(w, 500, err.Error())
msg := err.Error()
switch {
case msg == "bad snapshot name":
jsonErr(w, 400, msg)
case strings.Contains(msg, "does not exist"):
jsonErr(w, 404, msg)
default:
jsonErr(w, 500, msg)
}
return
}
log.Printf("zfs snapshot %s destroyed by %s%s", body.Name, clientIP(r), map[bool]string{true: " (recursive)", false: ""}[body.Recursive])
+6 -14
View File
@@ -1,14 +1,13 @@
// Package compose runs docker compose the way a stack was deployed — which is not always
// simply "on this host". A stack deployed by an orchestrator running inside its own
// container records paths in that container's filesystem, so compose has to run in there;
// a stack deployed to another machine over tcp needs DOCKER_HOST pointed at it. Both the
// console and the scheduled updates it writes must invoke it identically, or a redeploy
// from one would not match the other's labels.
// container records paths in that container's filesystem, so compose has to run in there.
// Both the console and the scheduled updates it writes must invoke it identically, or a
// redeploy from one would not match the other's labels. Compose always drives this
// machine's own Docker: a stack's files and its containers are on the same box.
package compose
import (
"context"
"os"
"os/exec"
"path/filepath"
)
@@ -18,7 +17,6 @@ type Spec struct {
Project string // -p
Files []string // -f, in order
Via string // a container to run compose inside; "" is this host
Remote string // a DOCKER_HOST; "" is this machine's Docker
Tool string // "docker" or "podman"
}
@@ -33,18 +31,12 @@ func Cmd(ctx context.Context, s Spec, args ...string) *exec.Cmd {
base = append(base, "-f", f)
}
if s.Via != "" {
pre := []string{"exec"}
if s.Remote != "" {
pre = append(pre, "-e", "DOCKER_HOST="+s.Remote)
}
return exec.CommandContext(ctx, tool, append(append(append(pre, s.Via, "docker"), base...), args...)...)
// The CLI inside the orchestrator's container is its own `docker`, whatever this host runs.
return exec.CommandContext(ctx, tool, append(append([]string{"exec", s.Via, "docker"}, base...), args...)...)
}
cmd := exec.CommandContext(ctx, tool, append(base, args...)...)
if len(s.Files) > 0 {
cmd.Dir = filepath.Dir(s.Files[0])
}
if s.Remote != "" {
cmd.Env = append(os.Environ(), "DOCKER_HOST="+s.Remote)
}
return cmd
}
+73 -5
View File
@@ -15,6 +15,7 @@ import (
"io"
"os"
"os/exec"
"os/signal"
"os/user"
"path/filepath"
"sort"
@@ -23,6 +24,7 @@ import (
"syscall"
"time"
"hamlab/internal/media"
"hamlab/internal/priv"
)
@@ -37,6 +39,7 @@ type Entry struct {
Group string `json:"group"`
Target string `json:"target,omitempty"` // link target
Broken bool `json:"broken,omitempty"` // dangling link
Preview string `json:"preview,omitempty"` // "image" | "video": the Files tab can show a thumbnail of it
}
type Listing struct {
@@ -56,10 +59,12 @@ type FS interface {
List(ctx context.Context, path string) (Listing, error)
Stat(ctx context.Context, path string) (Info, error)
Open(ctx context.Context, path string) (io.ReadCloser, error)
OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) // Open from a byte offset: a range request, a player seeking
Write(ctx context.Context, path string, r io.Reader) error
Mkdir(ctx context.Context, path string) error
Touch(ctx context.Context, path string) error
Rename(ctx context.Context, from, to string) error
Move(ctx context.Context, from, to string, progress io.Writer) error // Rename, or across filesystems a copy and a removal; long, so callers run it as a task
Delete(ctx context.Context, path string, recursive bool) error
Chmod(ctx context.Context, path string, perm os.FileMode) error
}
@@ -137,7 +142,7 @@ func entryOf(dir, name string, lst os.FileInfo) Entry {
} else if st.IsDir() {
e.Type = "dir" // keep dir semantics for navigation; Target says it's a link
} else {
e.Size = st.Size()
e.Size, e.Mtime = st.Size(), st.ModTime().UnixMilli() // the file's, not the link's
}
case lst.IsDir():
e.Type = "dir"
@@ -146,6 +151,9 @@ func entryOf(dir, name string, lst os.FileInfo) Entry {
default:
e.Type = "other"
}
if e.Type == "file" || (e.Type == "link" && !e.Broken) {
e.Preview = media.Kind(name)
}
return e
}
@@ -224,6 +232,20 @@ func (Local) Open(_ context.Context, path string) (io.ReadCloser, error) {
return os.Open(path)
}
func (l Local) OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) {
f, err := l.Open(ctx, path)
if err != nil {
return nil, err
}
if offset > 0 {
if _, err := f.(*os.File).Seek(offset, io.SeekStart); err != nil {
f.Close()
return nil, err
}
}
return f, nil
}
// Write replaces the file's contents, keeping its mode; new files get 0644 (umask).
func (Local) Write(_ context.Context, path string, r io.Reader) error {
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
@@ -329,7 +351,15 @@ func (r *cmdReader) Close() error {
}
func (s Sudo) Open(ctx context.Context, path string) (io.ReadCloser, error) {
c := s.cmd(ctx, "read", path)
return s.open(ctx, "read", path)
}
func (s Sudo) OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) {
return s.open(ctx, "read", "--offset", strconv.FormatInt(offset, 10), path)
}
func (s Sudo) open(ctx context.Context, args ...string) (io.ReadCloser, error) {
c := s.cmd(ctx, args...)
errb := &strings.Builder{}
c.Stderr = errb
out, err := c.StdoutPipe()
@@ -380,6 +410,30 @@ func (s Sudo) Touch(ctx context.Context, path string) error {
func (s Sudo) Rename(ctx context.Context, from, to string) error {
return s.runJSON(ctx, nil, "rename", from, to)
}
// Move as root: the helper does the copying and talks on stdout.
func (s Sudo) Move(ctx context.Context, from, to string, progress io.Writer) error {
c := s.cmd(ctx, "move", from, to)
c.Stdout = progress
var errb strings.Builder
c.Stderr = &errb
if err := c.Run(); err != nil {
if msg := strings.TrimSpace(errb.String()); msg != "" {
return errors.New(msg)
}
return err
}
return nil
}
// MoveCmd is the command a task runs for a long move: this binary's helper, as root when
// the backend is.
func MoveCmd(ctx context.Context, fsys FS, self, from, to string) *exec.Cmd {
if s, ok := fsys.(Sudo); ok {
return s.cmd(ctx, "move", from, to)
}
return exec.CommandContext(ctx, self, "fs", "move", from, to)
}
func (s Sudo) Delete(ctx context.Context, path string, recursive bool) error {
if recursive {
return s.runJSON(ctx, nil, "delete", "-r", path)
@@ -394,11 +448,16 @@ func (s Sudo) Chmod(ctx context.Context, path string, perm os.FileMode) error {
func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer) int {
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
if len(args) < 2 {
return fail(errors.New("usage: hamlab fs <list|stat|read|write|mkdir|touch|rename|delete|chmod> …"))
return fail(errors.New("usage: hamlab fs <list|stat|read|write|mkdir|touch|rename|move|delete|chmod> …"))
}
ctx := context.Background()
var l Local
op := args[0]
var offset int64
if op == "read" && len(args) >= 4 && args[1] == "--offset" { // read --offset N <path>
offset, _ = strconv.ParseInt(args[2], 10, 64)
args = append([]string{"read"}, args[3:]...)
}
clean := func(i int) (string, error) {
if i >= len(args) {
return "", errors.New("path required")
@@ -425,7 +484,7 @@ func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer)
}
return emit(stdout, out, stderr)
case "read":
f, err := l.Open(ctx, p)
f, err := l.OpenAt(ctx, p, offset)
if err != nil {
return fail(err)
}
@@ -447,7 +506,7 @@ func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer)
return fail(err)
}
}
case "rename":
case "rename", "move":
from, err := clean(1)
if err != nil {
return fail(err)
@@ -456,6 +515,15 @@ func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer)
if err != nil {
return fail(err)
}
if op == "move" {
// A cancelled task is a SIGTERM: stop copying and take the partial copy away.
mctx, stop := signal.NotifyContext(ctx, syscall.SIGTERM, syscall.SIGINT)
defer stop()
if err := l.Move(mctx, from, to, stdout); err != nil {
return fail(err)
}
return 0
}
if err := l.Rename(ctx, from, to); err != nil {
return fail(err)
}
+197
View File
@@ -0,0 +1,197 @@
package files
import (
"context"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"syscall"
)
// IsCrossDevice reports a rename refused because the two ends are on different
// filesystems — on a NAS that is every move between datasets. The Sudo backend only has
// the helper's words for it.
func IsCrossDevice(err error) bool {
return errors.Is(err, syscall.EXDEV) || (err != nil && strings.Contains(err.Error(), "cross-device"))
}
// Inside reports whether to lies within from — a folder cannot be moved into itself.
func Inside(from, to string) bool {
return to == from || strings.HasPrefix(to, strings.TrimSuffix(from, "/")+"/")
}
// Move is Rename, and across filesystems a copy and a removal. The copy is built under a
// hidden temporary name beside where it will land and renamed into place only once it is
// complete, and the source is removed only after that: a failure part-way, or a cancel,
// leaves the source exactly as it was. Progress goes to w, a line at a time, for the task
// output that a long move is watched through.
func (Local) Move(ctx context.Context, from, to string, w io.Writer) error {
if protected[from] {
return errors.New("refusing to move " + from)
}
if Inside(from, to) {
return errors.New("a folder can't be moved into itself")
}
if _, err := os.Lstat(to); err == nil {
return errors.New("destination exists")
}
err := os.Rename(from, to)
if err == nil || !IsCrossDevice(err) {
return err
}
return moveByCopy(ctx, from, to, w)
}
// moveByCopy is the slow path: copy, rename into place, remove the original.
func moveByCopy(ctx context.Context, from, to string, w io.Writer) error {
st, err := os.Lstat(from)
if err != nil {
return err
}
tmp := filepath.Join(filepath.Dir(to), "."+filepath.Base(to)+".hamlab-moving")
_ = os.RemoveAll(tmp) // a previous attempt that died
fmt.Fprintf(w, "%s is on another filesystem: copying, then removing the original\n", filepath.Dir(to))
p := &progress{w: w}
if err := copyPath(ctx, from, tmp, st, p); err != nil {
fmt.Fprintln(w, "copy failed; removing the partial copy, the original is untouched")
_ = os.RemoveAll(tmp)
return err
}
if err := os.Rename(tmp, to); err != nil {
_ = os.RemoveAll(tmp)
return err
}
fmt.Fprintf(w, "copied %s; removing the original\n", p.String())
if err := os.RemoveAll(from); err != nil {
return fmt.Errorf("copied to %s, but the original could not be removed: %w", to, err)
}
fmt.Fprintf(w, "moved %s to %s\n", from, to)
return nil
}
type progress struct {
w io.Writer
files int64
bytes int64
shownFiles int64
shownBytes int64
}
func (p *progress) add(files, bytes int64) {
p.files += files
p.bytes += bytes
if p.files-p.shownFiles >= 500 || p.bytes-p.shownBytes >= 512<<20 {
fmt.Fprintf(p.w, "copying… %s\n", p.String())
p.shownFiles, p.shownBytes = p.files, p.bytes
}
}
func (p *progress) String() string {
return fmt.Sprintf("%d files, %s", p.files, fmtSize(p.bytes))
}
func fmtSize(b int64) string {
const unit = 1024
if b < unit {
return fmt.Sprintf("%d B", b)
}
div, exp := int64(unit), 0
for n := b / unit; n >= unit && exp < 4; n /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f %ciB", float64(b)/float64(div), "KMGTP"[exp])
}
// copyPath copies one entry of any kind, recursively for a directory, keeping mode, times
// and — when running as root — ownership. Devices, pipes and sockets are refused: a copy
// of one is not the thing itself.
func copyPath(ctx context.Context, src, dst string, st os.FileInfo, p *progress) error {
if err := ctx.Err(); err != nil {
return err
}
mode := st.Mode()
switch {
case mode&os.ModeSymlink != 0:
target, err := os.Readlink(src)
if err != nil {
return err
}
if err := os.Symlink(target, dst); err != nil {
return err
}
chown(dst, st)
p.add(1, 0)
return nil
case mode.IsDir():
if err := os.Mkdir(dst, mode.Perm()|0o700); err != nil { // writable while it fills, exact mode after
return err
}
ents, err := os.ReadDir(src)
if err != nil {
return err
}
for _, e := range ents {
lst, err := e.Info()
if err != nil {
return err
}
if err := copyPath(ctx, filepath.Join(src, e.Name()), filepath.Join(dst, e.Name()), lst, p); err != nil {
return err
}
}
chown(dst, st)
if err := os.Chmod(dst, mode.Perm()); err != nil {
return err
}
return os.Chtimes(dst, st.ModTime(), st.ModTime())
case mode.IsRegular():
n, err := copyFile(src, dst, st)
if err != nil {
return err
}
p.add(1, n)
return nil
default:
return fmt.Errorf("%s: can't copy a device, pipe or socket", src)
}
}
func copyFile(src, dst string, st os.FileInfo) (int64, error) {
in, err := os.Open(src)
if err != nil {
return 0, err
}
defer in.Close()
out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL, st.Mode().Perm())
if err != nil {
return 0, err
}
n, err := io.Copy(out, in)
if err != nil {
out.Close()
return n, err
}
if err := out.Close(); err != nil {
return n, err
}
chown(dst, st)
if err := os.Chmod(dst, st.Mode().Perm()); err != nil { // umask has no say
return n, err
}
return n, os.Chtimes(dst, st.ModTime(), st.ModTime())
}
// chown keeps the owner when this process may set one; an ordinary user's copy is the
// user's, which is what a rename would have left anyway.
func chown(dst string, st os.FileInfo) {
if os.Geteuid() != 0 {
return
}
if sys, ok := st.Sys().(*syscall.Stat_t); ok {
_ = os.Lchown(dst, int(sys.Uid), int(sys.Gid))
}
}
+195
View File
@@ -0,0 +1,195 @@
package files
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
"time"
)
func TestInside(t *testing.T) {
for _, c := range []struct {
from, to string
want bool
}{
{"/a/b", "/a/b", true}, {"/a/b", "/a/b/c", true}, {"/a/b", "/a/bc", false}, {"/a/b", "/a", false}, {"/", "/x", true},
} {
if got := Inside(c.from, c.to); got != c.want {
t.Errorf("Inside(%q, %q) = %v, want %v", c.from, c.to, got, c.want)
}
}
}
func TestIsCrossDevice(t *testing.T) {
if !IsCrossDevice(&os.LinkError{Op: "rename", Old: "/a", New: "/b", Err: syscall.EXDEV}) {
t.Fatal("the kernel's EXDEV")
}
if !IsCrossDevice(errors.New("rename /a /b: invalid cross-device link")) {
t.Fatal("the helper's words for it")
}
if IsCrossDevice(errors.New("permission denied")) || IsCrossDevice(nil) {
t.Fatal("anything else is not")
}
}
// The copy path, exercised directly: a tree with nested folders, a symlink, an odd mode
// and old mtimes lands whole under the new name, and the original is gone only then.
func TestMoveByCopy(t *testing.T) {
root := t.TempDir()
src := filepath.Join(root, "src", "album")
if err := os.MkdirAll(filepath.Join(src, "sub", "deep"), 0o755); err != nil {
t.Fatal(err)
}
old := time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC)
write := func(p, s string, mode os.FileMode) {
if err := os.WriteFile(p, []byte(s), mode); err != nil {
t.Fatal(err)
}
_ = os.Chmod(p, mode)
_ = os.Chtimes(p, old, old)
}
write(filepath.Join(src, "a.txt"), "hello", 0o644)
write(filepath.Join(src, "sub", "deep", "b.bin"), strings.Repeat("x", 100000), 0o600)
write(filepath.Join(src, "run.sh"), "#!/bin/sh\n", 0o755)
if err := os.Symlink("a.txt", filepath.Join(src, "link")); err != nil {
t.Fatal(err)
}
_ = os.Chtimes(filepath.Join(src, "sub"), old, old)
dst := filepath.Join(root, "dst", "moved")
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
t.Fatal(err)
}
var out strings.Builder
if err := moveByCopy(context.Background(), src, dst, &out); err != nil {
t.Fatal(err)
}
if _, err := os.Lstat(src); !os.IsNotExist(err) {
t.Fatal("the original is still there")
}
if b, _ := os.ReadFile(filepath.Join(dst, "a.txt")); string(b) != "hello" {
t.Fatal("a.txt")
}
if st, err := os.Stat(filepath.Join(dst, "sub", "deep", "b.bin")); err != nil || st.Size() != 100000 || st.Mode().Perm() != 0o600 || !st.ModTime().Equal(old) {
t.Fatalf("b.bin: %v %v", st, err)
}
if st, _ := os.Stat(filepath.Join(dst, "run.sh")); st.Mode().Perm() != 0o755 {
t.Fatal("mode lost")
}
if target, err := os.Readlink(filepath.Join(dst, "link")); err != nil || target != "a.txt" {
t.Fatal("symlink not copied as a symlink")
}
if st, _ := os.Stat(filepath.Join(dst, "sub")); !st.ModTime().Equal(old) {
t.Fatal("folder mtime lost")
}
if left, _ := filepath.Glob(filepath.Join(root, "dst", ".*hamlab-moving*")); len(left) != 0 {
t.Fatal("temporary copy left behind")
}
if !strings.Contains(out.String(), "moved ") || !strings.Contains(out.String(), "4 files") {
t.Fatalf("progress said: %q", out.String())
}
}
// A failure part-way leaves the original as it was and no partial copy.
func TestMoveByCopyFailsClean(t *testing.T) {
root := t.TempDir()
src := filepath.Join(root, "src")
if err := os.MkdirAll(src, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(src, "a"), []byte("a"), 0o644); err != nil {
t.Fatal(err)
}
if err := syscall.Mkfifo(filepath.Join(src, "pipe"), 0o644); err != nil {
t.Skip("no fifos here")
}
dst := filepath.Join(root, "dst")
err := moveByCopy(context.Background(), src, dst, &strings.Builder{})
if err == nil || !strings.Contains(err.Error(), "pipe") {
t.Fatalf("expected the pipe to be refused, got %v", err)
}
if _, err := os.Stat(filepath.Join(src, "a")); err != nil {
t.Fatal("the original was touched")
}
if _, err := os.Lstat(dst); !os.IsNotExist(err) {
t.Fatal("a partial copy was left")
}
if left, _ := filepath.Glob(filepath.Join(root, ".*hamlab-moving*")); len(left) != 0 {
t.Fatal("temporary copy left behind")
}
// a cancel is the same story
if err := os.Remove(filepath.Join(src, "pipe")); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if err := moveByCopy(ctx, src, dst, &strings.Builder{}); err == nil {
t.Fatal("a cancelled move must fail")
}
if _, err := os.Stat(filepath.Join(src, "a")); err != nil {
t.Fatal("the original was touched by a cancel")
}
}
// Move on one filesystem is a rename: nothing is copied, and the guards hold.
func TestMoveSameFilesystem(t *testing.T) {
root := t.TempDir()
src := filepath.Join(root, "f")
if err := os.WriteFile(src, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
var l Local
var out strings.Builder
if err := l.Move(context.Background(), src, filepath.Join(root, "g"), &out); err != nil {
t.Fatal(err)
}
if out.Len() != 0 {
t.Fatal("a rename has nothing to say")
}
if _, err := os.Stat(filepath.Join(root, "g")); err != nil {
t.Fatal("not renamed")
}
if err := l.Move(context.Background(), root, filepath.Join(root, "inside"), &out); err == nil {
t.Fatal("into itself")
}
if err := l.Move(context.Background(), filepath.Join(root, "g"), filepath.Join(root, "g"), &out); err == nil {
t.Fatal("onto itself")
}
if err := os.WriteFile(filepath.Join(root, "h"), nil, 0o644); err != nil {
t.Fatal(err)
}
if err := l.Move(context.Background(), filepath.Join(root, "g"), filepath.Join(root, "h"), &out); err == nil || !strings.Contains(err.Error(), "exists") {
t.Fatalf("onto an existing file: %v", err)
}
}
// A link to a file carries the file's size and time, so the row and the preview key follow
// the file; a link to a folder is a folder; a link to nothing says so.
func TestEntryOfLinks(t *testing.T) {
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "f"), []byte("12345"), 0o644); err != nil {
t.Fatal(err)
}
old := time.Date(2021, 1, 1, 0, 0, 0, 0, time.UTC)
_ = os.Chtimes(filepath.Join(dir, "f"), old, old)
_ = os.Mkdir(filepath.Join(dir, "d"), 0o755)
for _, l := range [][2]string{{"lf", "f"}, {"ld", "d"}, {"lx", "nowhere"}} {
if err := os.Symlink(l[1], filepath.Join(dir, l[0])); err != nil {
t.Fatal(err)
}
}
ent := func(n string) Entry { st, _ := os.Lstat(filepath.Join(dir, n)); return entryOf(dir, n, st) }
if e := ent("lf"); e.Type != "link" || e.Size != 5 || e.Mtime != old.UnixMilli() || e.Target != "f" || e.Broken {
t.Fatalf("link to file: %+v", e)
}
if e := ent("ld"); e.Type != "dir" || e.Target != "d" {
t.Fatalf("link to dir: %+v", e)
}
if e := ent("lx"); e.Type != "link" || !e.Broken || e.Preview != "" {
t.Fatalf("dangling: %+v", e)
}
}
+89
View File
@@ -0,0 +1,89 @@
package files
import (
"context"
"errors"
"io"
)
// Seeker is an FS file as http.ServeContent wants one — something to seek in — for a file
// whose size is already known from a stat. ServeContent seeks to the end for the size, to
// the start of the range asked for, then reads: so the backend is opened once, at that
// position, and only ever read forward. The Sudo backend's helper is started at the offset
// rather than streamed from the top and skipped.
type Seeker struct {
ctx context.Context
fsys FS
path string
size int64
pos int64 // where the next Read reads from
at int64 // where cur is positioned, when open
cur io.ReadCloser
}
func NewSeeker(ctx context.Context, fsys FS, path string, size int64) *Seeker {
return &Seeker{ctx: ctx, fsys: fsys, path: path, size: size}
}
// Check opens the file now, from the start, so a file that cannot be read is known before
// any headers go out. Seeking costs nothing; a Read from somewhere else reopens there.
func (s *Seeker) Check() error {
if s.cur != nil {
return nil
}
f, err := s.fsys.OpenAt(s.ctx, s.path, 0)
if err != nil {
return err
}
s.cur, s.at = f, 0
return nil
}
func (s *Seeker) Seek(offset int64, whence int) (int64, error) {
var pos int64
switch whence {
case io.SeekStart:
pos = offset
case io.SeekCurrent:
pos = s.pos + offset
case io.SeekEnd:
pos = s.size + offset
default:
return 0, errors.New("bad whence")
}
if pos < 0 {
return 0, errors.New("seek before start")
}
s.pos = pos // the stream, if open, stays where it is until a Read wants somewhere else
return pos, nil
}
func (s *Seeker) Read(p []byte) (int, error) {
if s.pos >= s.size {
return 0, io.EOF
}
if s.cur != nil && s.pos != s.at {
s.cur.Close()
s.cur = nil
}
if s.cur == nil {
f, err := s.fsys.OpenAt(s.ctx, s.path, s.pos)
if err != nil {
return 0, err
}
s.cur, s.at = f, s.pos
}
n, err := s.cur.Read(p)
s.pos += int64(n)
s.at += int64(n)
return n, err
}
func (s *Seeker) Close() error {
if s.cur == nil {
return nil
}
err := s.cur.Close()
s.cur = nil
return err
}
+112
View File
@@ -0,0 +1,112 @@
package files
import (
"context"
"io"
"os"
"path/filepath"
"testing"
)
import "strings"
type stringsBuilder = strings.Builder
type countingFS struct {
Local
opens []int64
}
func (c *countingFS) OpenAt(ctx context.Context, path string, off int64) (io.ReadCloser, error) {
c.opens = append(c.opens, off)
return c.Local.OpenAt(ctx, path, off)
}
// ServeContent's moves — size from the end, seek to the range, read forward — open the
// backend once, at the range; a seek to where the stream already is costs nothing.
func TestSeeker(t *testing.T) {
p := filepath.Join(t.TempDir(), "f")
data := make([]byte, 1000)
for i := range data {
data[i] = byte(i)
}
if err := os.WriteFile(p, data, 0o644); err != nil {
t.Fatal(err)
}
fs := &countingFS{}
s := NewSeeker(context.Background(), fs, p, int64(len(data)))
if n, _ := s.Seek(0, io.SeekEnd); n != 1000 {
t.Fatalf("end = %d", n)
}
if _, err := s.Seek(100, io.SeekStart); err != nil {
t.Fatal(err)
}
buf := make([]byte, 50)
if n, err := io.ReadFull(s, buf); err != nil || n != 50 || buf[0] != 100 || buf[49] != 149 {
t.Fatalf("read at 100: %d %v %v", n, err, buf[:2])
}
if _, err := s.Seek(150, io.SeekStart); err != nil { // where it already is
t.Fatal(err)
}
if n, _ := io.ReadFull(s, buf); n != 50 || buf[0] != 150 {
t.Fatal("read after a no-op seek")
}
if len(fs.opens) != 1 || fs.opens[0] != 100 {
t.Fatalf("opens = %v, want [100]", fs.opens)
}
if _, err := s.Seek(0, io.SeekStart); err != nil {
t.Fatal(err)
}
rest, err := io.ReadAll(s)
if err != nil || len(rest) != 1000 || rest[999] != data[999] {
t.Fatalf("read all from the start: %d %v", len(rest), err)
}
if len(fs.opens) != 2 || fs.opens[1] != 0 {
t.Fatalf("opens = %v, want [100 0]", fs.opens)
}
if n, err := s.Read(buf); n != 0 || err != io.EOF {
t.Fatal("past the end is EOF")
}
if _, err := s.Seek(-1, io.SeekStart); err == nil {
t.Fatal("before the start")
}
if err := s.Close(); err != nil {
t.Fatal(err)
}
// Check opens at the start and keeps that stream for a read from the start
s2 := NewSeeker(context.Background(), fs, p, 1000)
if err := s2.Check(); err != nil {
t.Fatal(err)
}
s2.Seek(0, io.SeekEnd)
s2.Seek(0, io.SeekStart)
if n, _ := io.ReadFull(s2, buf); n != 50 || buf[0] != 0 {
t.Fatal("read from the start after Check")
}
if len(fs.opens) != 3 {
t.Fatalf("Check's open was not reused: %v", fs.opens)
}
s2.Close()
if err := NewSeeker(context.Background(), fs, filepath.Join(t.TempDir(), "nope"), 5).Check(); err == nil {
t.Fatal("Check must report a file that can't be opened")
}
}
// The helper's read takes an offset, for the Sudo backend.
func TestHelperReadOffset(t *testing.T) {
p := filepath.Join(t.TempDir(), "f")
if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil {
t.Fatal(err)
}
var out, errb stringsBuilder
if rc := Helper([]string{"read", "--offset", "7", p}, nil, &out, &errb); rc != 0 {
t.Fatalf("rc %d: %s", rc, errb.String())
}
if out.String() != "789" {
t.Fatalf("got %q", out.String())
}
out.Reset()
if rc := Helper([]string{"read", p}, nil, &out, &errb); rc != 0 || out.String() != "0123456789" {
t.Fatalf("plain read: rc %d %q", rc, out.String())
}
}
+3 -2
View File
@@ -10,6 +10,7 @@ import (
"encoding/json"
"errors"
"fmt"
"hamlab/internal/store"
"os"
"os/exec"
"path/filepath"
@@ -91,7 +92,7 @@ func (m *Manager) SaveExportConfig(c ExportConfig) error {
}
c.Updated = time.Now()
b, _ := json.MarshalIndent(c, "", " ")
return os.WriteFile(m.exportConfigPath(), b, 0o644)
return store.WriteFileAtomic(m.exportConfigPath(), b, 0o644)
}
type ExportRun struct {
@@ -226,6 +227,6 @@ func (m *Manager) RunExports(ctx context.Context, names []string, w func(string)
}
run.Seconds = time.Since(start).Seconds()
b, _ := json.MarshalIndent(run, "", " ")
_ = os.WriteFile(m.exportRunPath(), b, 0o644)
_ = store.WriteFileAtomic(m.exportRunPath(), b, 0o644)
return run
}
+66 -6
View File
@@ -28,7 +28,7 @@ import (
)
const (
PoolName = "default"
PoolName = "local" // plain files on the machine's own disk, in Ham Lab's folder
BridgeName = "incusbr0"
ZPoolName = "hamlab-virt" // the ZFS pool made on the backing file; Incus is handed it by name
AdminGroup = "incus-admin"
@@ -615,11 +615,38 @@ type Pool struct {
Name string `json:"name"`
Driver string `json:"driver"`
Source string `json:"source"`
Where string `json:"where"` // in words: what disk this is, so a pool is picked by place, not by driver
Used uint64 `json:"used"`
Total uint64 `json:"total"`
UsedBy int `json:"used_by"`
}
// poolWhere describes a pool by where its bytes land.
func (m *Manager) poolWhere(driver, source string) string {
inHam := strings.HasPrefix(source, m.dir+"/") || source == m.dir
switch driver {
case "dir":
if inHam {
return "this machine's own disk — plain files, each disk sized on its own"
}
return "plain files at " + source
case "zfs":
if strings.HasSuffix(source, ".img") || source == ZPoolName {
return "a ZFS pool inside one file on this machine's disk"
}
if strings.Contains(source, "/") {
return "ZFS dataset " + source
}
return "ZFS pool " + source
case "btrfs":
if inHam {
return "btrfs inside one file on this machine's disk"
}
return "btrfs at " + source
}
return driver + " at " + source
}
type Network struct {
Name string `json:"name"`
Type string `json:"type"`
@@ -647,7 +674,7 @@ func (m *Manager) Pools(ctx context.Context) []Pool {
}
var out2 []Pool
for _, p := range raw {
pool := Pool{Name: p.Name, Driver: p.Driver, Source: p.Config["source"], UsedBy: len(p.UsedBy)}
pool := Pool{Name: p.Name, Driver: p.Driver, Source: p.Config["source"], Where: m.poolWhere(p.Driver, p.Config["source"]), UsedBy: len(p.UsedBy)}
if b, err := m.run(c, "query", "/1.0/storage-pools/"+p.Name+"/resources"); err == nil {
var res struct {
Space struct {
@@ -664,6 +691,24 @@ func (m *Manager) Pools(ctx context.Context) []Pool {
return out2
}
// DefaultPool is where a new instance's disk goes unless the form says otherwise: the pool
// on the default profile's root device.
func (m *Manager) DefaultPool(ctx context.Context) string {
c, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
out, err := m.run(c, "query", "/1.0/profiles/default")
if err != nil {
return ""
}
var p struct {
Devices map[string]map[string]string `json:"devices"`
}
if json.Unmarshal(out, &p) != nil {
return ""
}
return p.Devices["root"]["pool"]
}
func (m *Manager) Networks(ctx context.Context) []Network {
c, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
@@ -869,6 +914,7 @@ type CreateSpec struct {
CPU string `json:"cpu"`
Memory string `json:"memory"`
Disk string `json:"disk"`
Pool string `json:"pool"` // where the root disk goes; blank is the default profile's
Network string `json:"network"`
CloudInit string `json:"cloud_init"`
Profiles []string `json:"profiles"`
@@ -891,6 +937,8 @@ func CreateArgs(sp CreateSpec, kvm bool) ([]string, error) {
return nil, errors.New("memory: a size, like 2GiB")
case !ValidSize(sp.Disk):
return nil, errors.New("disk: a size, like 20GiB")
case sp.Pool != "" && !ValidName(sp.Pool):
return nil, errors.New("pool: a storage pool's name")
case !ValidNetwork(sp.Network):
return nil, errors.New("network: a bridge name")
case len(sp.CloudInit) > 256<<10:
@@ -912,8 +960,16 @@ func CreateArgs(sp CreateSpec, kvm bool) ([]string, error) {
if sp.Memory != "" {
args = append(args, "-c", "limits.memory="+sp.Memory)
}
// the root disk: the profile's, with the pool and size overridden where given
var root []string
if sp.Pool != "" {
root = append(root, "pool="+sp.Pool)
}
if sp.Disk != "" {
args = append(args, "-d", "root,size="+sp.Disk)
root = append(root, "size="+sp.Disk)
}
if len(root) > 0 {
args = append(args, "-d", "root,"+strings.Join(root, ","))
}
if sp.Network != "" {
args = append(args, "-n", sp.Network)
@@ -1104,6 +1160,7 @@ type FromISOSpec struct {
CPU string `json:"cpu"`
Memory string `json:"memory"`
Disk string `json:"disk"`
Pool string `json:"pool"`
Network string `json:"network"`
SecureBoot bool `json:"secureboot"`
Start bool `json:"start"`
@@ -1121,6 +1178,8 @@ func FromISOScript(sp FromISOSpec, kvm bool) (script string, params []string, er
return "", nil, errors.New("iso: pick one from the library")
case !ValidCPU(sp.CPU) || !ValidSize(sp.Memory) || !ValidSize(sp.Disk) || !ValidNetwork(sp.Network):
return "", nil, errors.New("cpu, memory, disk: a count and two sizes, like 2, 2GiB, 20GiB")
case sp.Pool != "" && !ValidName(sp.Pool):
return "", nil, errors.New("pool: a storage pool's name")
}
cpu, mem, disk := sp.CPU, sp.Memory, sp.Disk
if cpu == "" {
@@ -1140,13 +1199,14 @@ func FromISOScript(sp FromISOSpec, kvm bool) (script string, params []string, er
if sp.Start {
start = "1"
}
// $1 name $2 iso path $3 cpu $4 memory $5 disk $6 network $7 secureboot $8 start
// $1 name $2 iso path $3 cpu $4 memory $5 disk $6 network $7 secureboot $8 start $9 pool
script = `set -e
NET=""; [ -n "$6" ] && NET="-n $6"
incus init --empty --vm "$1" -c limits.cpu="$3" -c limits.memory="$4" -c security.secureboot="$7" -d root,size="$5" $NET
ROOT="root,size=$5"; [ -n "$9" ] && ROOT="root,pool=$9,size=$5"
incus init --empty --vm "$1" -c limits.cpu="$3" -c limits.memory="$4" -c security.secureboot="$7" -d "$ROOT" $NET
incus config device add "$1" iso0 disk source="$2" boot.priority=10
echo "made $1 with $2 as its first boot device"
[ "$8" = 1 ] && incus start "$1" && echo "started; open the VGA console to run the installer"
exit 0`
return script, []string{sp.Name, sp.ISO, cpu, mem, disk, sp.Network, sb, start}, nil
return script, []string{sp.Name, sp.ISO, cpu, mem, disk, sp.Network, sb, start, sp.Pool}, nil
}
+6 -3
View File
@@ -21,13 +21,13 @@ func TestInstallScriptShapes(t *testing.T) {
}{
{"grid: zfs, apt, docker", Status{State: "installable", PM: "apt", Arch: "x86_64", Drivers: []string{"zfs", "btrfs", "dir"}, Docker: true, ZFSHost: true, DiskFree: 700 << 30, MemTotal: 62 << 30},
InstallOpts{Driver: "zfs", PoolSizeGiB: 300, Subnet: "172.16.100.1/24", DockerRules: true, ARCMaxGiB: 16},
[]string{"apt-get", "zpool create", "incus storage create default zfs source=hamlab-virt", "ipv4.address='172.16.100.1/24'", "DOCKER-USER", "zfs_arc_max", "usermod -aG incus-admin 'core'", "SupplementaryGroups=incus-admin", "systemd-run"}},
[]string{"apt-get", "zpool create", "incus storage create local zfs source=hamlab-virt", "ipv4.address='172.16.100.1/24'", "DOCKER-USER", "zfs_arc_max", "usermod -aG incus-admin 'core'", "SupplementaryGroups=incus-admin", "systemd-run"}},
{"core: btrfs, apt, docker, no tools yet", Status{State: "installable", PM: "apt", Arch: "x86_64", Drivers: []string{"btrfs", "dir"}, Docker: true, DiskFree: 170 << 30, MemTotal: 8 << 30},
InstallOpts{Driver: "btrfs", PoolSizeGiB: 40, Subnet: "172.16.100.1/24", DockerRules: true},
[]string{"btrfs-progs", "mkfs.btrfs", "/etc/fstab", "incus storage create default btrfs source="}},
[]string{"btrfs-progs", "mkfs.btrfs", "/etc/fstab", "incus storage create local btrfs source="}},
{"mac: dir, dnf, arm64", Status{State: "installable", PM: "dnf", Arch: "aarch64", Drivers: []string{"dir"}, DiskFree: 200 << 30, MemTotal: 15 << 30},
InstallOpts{Driver: "dir", Subnet: "172.16.101.1/24"},
[]string{"dnf -y install incus qemu-system-aarch64-core edk2-aarch64", "incus storage create default dir source="}},
[]string{"dnf -y install incus qemu-system-aarch64-core edk2-aarch64", "incus storage create local dir source="}},
{"repair: already installed", Status{State: "repair", PM: "apt", Arch: "x86_64", Drivers: []string{"dir"}, DiskFree: 10 << 30, MemTotal: 4 << 30},
InstallOpts{Driver: "dir", Subnet: "172.16.100.1/24"},
[]string{"already installed"}},
@@ -75,6 +75,9 @@ func TestCreateArgs(t *testing.T) {
t.Fatal(err)
}
want := "launch images:debian/13 web-1 --vm -c limits.cpu=2 -c limits.memory=2GiB -d root,size=20GiB -n incusbr0"
if a, err := CreateArgs(CreateSpec{Name: "x", Image: "images:alpine/3.22", Pool: "local"}, true); err != nil || strings.Join(a, " ") != "init images:alpine/3.22 x -d root,pool=local" {
t.Fatalf("pool override: %v %v", a, err)
}
if got := strings.Join(args, " "); got != want {
t.Fatalf("got %q", got)
}
-4
View File
@@ -14,7 +14,6 @@ import (
"net/netip"
"os"
"os/exec"
"strings"
"time"
)
@@ -125,6 +124,3 @@ func dockerPools() []string {
}
return []string{"172.17.0.0/12", "192.168.0.0/16"}
}
// String form used in the layout file and on the page.
func prefixString(p netip.Prefix) string { return strings.TrimSpace(p.Masked().String()) }
+1 -1
View File
@@ -130,7 +130,7 @@ func (m *Manager) openVGA(ctx context.Context, client *http.Client, name string,
resp, err := client.Do(req)
if err != nil {
if isPermission(err) {
return nil, errors.New("Ham Lab's account can't reach Incus's socket; Finish setup on the Virtualization tab puts it in incus-admin")
return nil, errors.New("this account can't reach Incus's socket; Finish setup on the Virtualization tab puts Ham Lab in incus-admin")
}
return nil, err
}
+218
View File
@@ -0,0 +1,218 @@
package media
import (
"crypto/sha256"
"encoding/binary"
"encoding/hex"
"encoding/json"
"errors"
"os"
"strconv"
"sync"
"time"
bolt "go.etcd.io/bbolt"
)
// Cache keeps finished previews in one database file under Ham Lab's own data directory —
// thumbs.db, a bbolt key/value store: pure Go, so the binary stays static, and one file
// per machine rather than a tree of hash-named JPEGs. An entry is one file's preview keyed
// by its path, size and mtime, so a file that changes gets a new preview and an unchanged
// one is never decoded twice. A file ffmpeg could not decode is kept the same way, with
// ffmpeg's reason and no picture, so a folder of odd files does not re-run ffmpeg on every
// visit. Nothing expires: a preview is a few kilobytes, and only files somebody actually
// scrolled past are ever made. The file is opened the first time something is stored, so a
// machine nobody browses with previews on never has one.
type Cache struct {
Path string
mu sync.Mutex
db *bolt.DB
err error
}
// Entry is one cached preview, or the remembered reason there is none.
type Entry struct {
Path string `json:"path"`
Engine string `json:"engine,omitempty"`
Codec string `json:"codec,omitempty"`
Width int `json:"width,omitempty"`
Height int `json:"height,omitempty"`
Made time.Time `json:"made"`
Failure string `json:"failure,omitempty"` // set instead of JPEG when ffmpeg refused the file
JPEG []byte `json:"-"`
}
var (
bucketThumbs = []byte("thumbs")
bucketMeta = []byte("meta")
keyCount = []byte("count")
)
// Key identifies one file's preview. "v1" is the preview format: bump it to retire every
// cached preview at once.
func Key(path string, size, mtime int64) string {
h := sha256.Sum256([]byte("v1\x00" + path + "\x00" + strconv.FormatInt(size, 10) + "\x00" + strconv.FormatInt(mtime, 10)))
return hex.EncodeToString(h[:])
}
func NewCache(path string) *Cache { return &Cache{Path: path} }
// open the database once; an open that failed is not retried, and previews still get made
// and served — they are simply not kept, and the log says why.
func (c *Cache) open() (*bolt.DB, error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.db != nil || c.err != nil {
return c.db, c.err
}
db, err := bolt.Open(c.Path, 0o600, &bolt.Options{Timeout: 2 * time.Second})
if err != nil {
c.err = err
return nil, err
}
if err := db.Update(func(tx *bolt.Tx) error {
if _, err := tx.CreateBucketIfNotExists(bucketThumbs); err != nil {
return err
}
_, err := tx.CreateBucketIfNotExists(bucketMeta)
return err
}); err != nil {
db.Close()
c.err = err
return nil, err
}
c.db = db
return db, nil
}
// Close releases the file. Previews after it are made but not kept.
func (c *Cache) Close() error {
c.mu.Lock()
defer c.mu.Unlock()
if c.db == nil {
return nil
}
err := c.db.Close()
c.db, c.err = nil, errors.New("cache closed")
return err
}
// Get returns the entry for a key. A database that does not exist yet is a miss, not a reason
// to make one.
func (c *Cache) Get(key string) (Entry, bool, error) {
c.mu.Lock()
db := c.db
c.mu.Unlock()
if db == nil {
if _, err := os.Stat(c.Path); err != nil {
return Entry{}, false, nil
}
var err error
if db, err = c.open(); err != nil {
return Entry{}, false, err
}
}
var e Entry
found := false
err := db.View(func(tx *bolt.Tx) error {
v := tx.Bucket(bucketThumbs).Get([]byte(key))
if v == nil {
return nil
}
var err error
e, err = decode(v) // copies: a value is only valid inside the transaction
found = err == nil
return err
})
return e, found, err
}
// Put stores an entry, replacing any under the same key.
func (c *Cache) Put(key string, e Entry) error {
if len(e.JPEG) == 0 && e.Failure == "" {
return errors.New("empty preview")
}
if e.Made.IsZero() {
e.Made = time.Now()
}
db, err := c.open()
if err != nil {
return err
}
v, err := encode(e)
if err != nil {
return err
}
return db.Update(func(tx *bolt.Tx) error {
b := tx.Bucket(bucketThumbs)
k := []byte(key)
fresh := b.Get(k) == nil
if err := b.Put(k, v); err != nil {
return err
}
if fresh {
m := tx.Bucket(bucketMeta)
return m.Put(keyCount, be64(count(m)+1))
}
return nil
})
}
// Stats is how many previews are kept and how big the file is.
func (c *Cache) Stats() (entries int64, bytes int64) {
st, err := os.Stat(c.Path)
if err != nil {
return 0, 0
}
bytes = st.Size()
db, err := c.open()
if err != nil {
return 0, bytes
}
_ = db.View(func(tx *bolt.Tx) error { entries = count(tx.Bucket(bucketMeta)); return nil })
return entries, bytes
}
func count(m *bolt.Bucket) int64 {
if v := m.Get(keyCount); len(v) == 8 {
return int64(binary.BigEndian.Uint64(v))
}
return 0
}
func be64(n int64) []byte {
b := make([]byte, 8)
binary.BigEndian.PutUint64(b, uint64(n))
return b
}
// A value is a 4-byte header length, the header as JSON, then the JPEG bytes.
func encode(e Entry) ([]byte, error) {
hdr, err := json.Marshal(e)
if err != nil {
return nil, err
}
v := make([]byte, 4+len(hdr)+len(e.JPEG))
binary.BigEndian.PutUint32(v, uint32(len(hdr)))
copy(v[4:], hdr)
copy(v[4+len(hdr):], e.JPEG)
return v, nil
}
func decode(v []byte) (Entry, error) {
if len(v) < 4 {
return Entry{}, errors.New("short value")
}
n := int(binary.BigEndian.Uint32(v))
if 4+n > len(v) {
return Entry{}, errors.New("bad header length")
}
var e Entry
if err := json.Unmarshal(v[4:4+n], &e); err != nil {
return Entry{}, err
}
if rest := v[4+n:]; len(rest) > 0 {
e.JPEG = append([]byte(nil), rest...)
}
return e, nil
}
+222
View File
@@ -0,0 +1,222 @@
package media
// What to run when a file is to be re-encoded for a browser that cannot play it as it is:
// the codecs and sizes on offer, the output size and bit rate a choice comes to for a
// given source, the ways this machine could do it in the order worth trying, and the
// ffmpeg argument list for one of them. All decisions, no processes, so it can be tested.
import (
"fmt"
"math"
"sort"
"strconv"
)
// Codec is one target a stream may be re-encoded to. "copy" keeps the codec and only
// repackages: for a browser that lacks the container, not the decoder.
type Codec struct {
ID string `json:"id"`
Label string `json:"label"`
}
var Codecs = []Codec{{"h264", "H.264"}, {"hevc", "HEVC"}, {"av1", "AV1"}, {"copy", "Original codec, repackaged"}}
// Quality is a picture height; the bit rate follows from the size the output ends up at.
type Quality struct {
ID string `json:"id"`
Label string `json:"label"`
Height int `json:"height"` // 0: the source's own
}
var Qualities = []Quality{{"original", "Original size", 0}, {"1080p", "1080p", 1080}, {"720p", "720p", 720}, {"480p", "480p", 480}}
func qualityByID(id string) (Quality, bool) {
for _, q := range Qualities {
if q.ID == id {
return q, true
}
}
return Quality{}, false
}
// ValidChoice says whether a codec and quality are ones on offer — checked before any work.
func ValidChoice(codec, quality string) error {
if !codecKnown(codec) {
return fmt.Errorf("unknown codec %q", codec)
}
if _, ok := qualityByID(quality); !ok {
return fmt.Errorf("unknown quality %q", quality)
}
return nil
}
func codecKnown(id string) bool {
for _, c := range Codecs {
if c.ID == id {
return true
}
}
return false
}
// fit is the output: never larger than the source, even dimensions, and a bit rate by the
// size it comes to — a quarter more at 50 fps and up, less for the codecs that need less.
func fit(q Quality, src Info, codec string) (w, h, kbps int) {
w, h = src.Video.Width, src.Video.Height
if q.Height > 0 && h > q.Height {
w = int(math.Round(float64(w) * float64(q.Height) / float64(h)))
h = q.Height
}
w, h = w&^1, h&^1
switch {
case h >= 2160:
kbps = 20000
case h >= 1440:
kbps = 12000
case h >= 1080:
kbps = 8000
case h >= 720:
kbps = 4000
default:
kbps = 2000
}
if src.Video.FPS >= 50 {
kbps = kbps * 5 / 4
}
if codec == "hevc" || codec == "av1" {
kbps = kbps * 6 / 10
}
return w, h, kbps
}
// Recipe is one way to make the stream: the engine's name for the record, and the ffmpeg
// arguments that go before the input, the video filter, and the video codec.
type Recipe struct {
Engine string
Pre []string
VF string
Video []string
}
// recipes are the ways to try, best first: a card on the bus before the integrated GPU;
// on a GPU, decoding on the same card (frames never leave it) before decoding in software
// and uploading; the software encoders last. An empty list means nothing here can.
func recipes(c Caps, codec string, src Info, w, h, kbps int) []Recipe {
if codec == "copy" {
v := []string{"-c:v", "copy"}
if src.Video.Codec == "hevc" {
v = append(v, "-tag:v", "hvc1") // Safari reads hvc1, not hev1
}
return []Recipe{{Engine: "Original codec", Video: v}}
}
var out []Recipe
gpus := append([]GPU(nil), c.GPUs...)
sort.SliceStable(gpus, func(i, j int) bool { return !gpus[i].Integrated && gpus[j].Integrated })
rate := []string{"-b:v", strconv.Itoa(kbps) + "k", "-maxrate", strconv.Itoa(kbps*5/4) + "k", "-bufsize", strconv.Itoa(kbps*5/2) + "k"}
for i := range gpus {
g := &gpus[i]
if g.API != "vaapi" || !has(g.Encodes, codec) {
continue
}
video := append([]string{"-c:v", codec + "_vaapi"}, rate...)
switch codec {
case "h264":
video = append(video, "-profile:v", "high")
case "hevc":
video = append(video, "-tag:v", "hvc1")
}
name := g.Name
if name == "" {
name = g.Device
}
scale := fmt.Sprintf("scale_vaapi=w=%d:h=%d:format=nv12", w, h)
if has(g.Codecs, src.Video.Codec) {
vf := scale
if src.HDR() {
vf = "tonemap_vaapi=format=nv12:t=bt709:p=bt709:m=bt709," + vf
}
out = append(out, Recipe{Engine: name + " (vaapi)", Pre: []string{"-hwaccel", "vaapi", "-hwaccel_device", g.Device, "-hwaccel_output_format", "vaapi"}, VF: vf, Video: video})
}
out = append(out, Recipe{Engine: name + " (vaapi, decoded on the CPU)", Pre: []string{"-init_hw_device", "vaapi=va:" + g.Device, "-filter_hw_device", "va"},
VF: fmt.Sprintf("format=nv12,scale=%d:%d,hwupload", w, h), Video: video})
}
swVF := fmt.Sprintf("scale=%d:%d,format=yuv420p", w, h)
if src.HDR() {
swVF = fmt.Sprintf("zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable,zscale=t=bt709:m=bt709,scale=%d:%d,format=yuv420p", w, h)
}
cap := []string{"-maxrate", strconv.Itoa(kbps*5/4) + "k", "-bufsize", strconv.Itoa(kbps*5/2) + "k"}
switch {
case codec == "h264" && has(c.Encoders, "libx264"):
out = append(out, Recipe{Engine: "CPU (libx264)", VF: swVF, Video: append([]string{"-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-profile:v", "high"}, cap...)})
case codec == "hevc" && has(c.Encoders, "libx265"):
out = append(out, Recipe{Engine: "CPU (libx265)", VF: swVF, Video: append([]string{"-c:v", "libx265", "-preset", "fast", "-crf", "26", "-tag:v", "hvc1"}, cap...)})
case codec == "av1" && has(c.Encoders, "libsvtav1"):
out = append(out, Recipe{Engine: "CPU (SVT-AV1)", VF: swVF, Video: append([]string{"-c:v", "libsvtav1", "-preset", "8", "-crf", "35"}, cap...)})
}
return out
}
// EngineFor is the engine the picker will say a codec runs on, "" when none can.
func EngineFor(c Caps, codec string) string {
if c.Error != "" {
return ""
}
if codec == "copy" {
return "Original codec"
}
r := recipes(c, codec, Info{Video: &Track{Codec: "h264", Width: 1920, Height: 1080}}, 1920, 1080, 8000)
if len(r) == 0 {
return ""
}
return r[0].Engine
}
// hlsArgs is the whole ffmpeg argument list for one recipe: the input, the first video and
// audio streams, the picture through the recipe, sound as AAC unless it already is, a
// keyframe every segment, and an event playlist of fragmented-MP4 segments in dir.
func hlsArgs(dir, in string, src Info, r Recipe, copyCodec bool) []string {
a := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
a = append(a, r.Pre...)
a = append(a, "-i", in, "-map", "0:v:0", "-map", "0:a:0?", "-sn", "-dn")
if r.VF != "" {
a = append(a, "-vf", r.VF)
}
a = append(a, r.Video...)
switch {
case src.Audio == nil:
a = append(a, "-an")
case src.Audio.Codec == "aac":
a = append(a, "-c:a", "copy")
default:
a = append(a, "-c:a", "aac", "-b:a", "160k", "-ac", "2")
}
if !copyCodec {
fps := src.Video.FPS
if fps <= 0 {
fps = 30
}
gop := int(math.Round(fps * hlsSeconds))
if gop < 24 {
gop = 24
}
if gop > 300 {
gop = 300
}
a = append(a, "-g", strconv.Itoa(gop), "-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%d)", hlsSeconds))
}
a = append(a, "-f", "hls", "-hls_time", strconv.Itoa(hlsSeconds), "-hls_playlist_type", "event", "-hls_segment_type", "fmp4",
"-hls_fmp4_init_filename", "init.mp4", "-hls_segment_filename", dir+"/seg%05d.m4s", "-hls_flags", "independent_segments+temp_file",
"-progress", dir+"/progress", "-stats_period", "1", dir+"/index.m3u8")
return a
}
const hlsSeconds = 4
func has(list []string, x string) bool {
for _, v := range list {
if v == x {
return true
}
}
return false
}
+128
View File
@@ -0,0 +1,128 @@
package media
import (
"strings"
"testing"
)
func src(codec string, w, h int, fps float64) Info {
return Info{Duration: 100, Video: &Track{Codec: codec, Width: w, Height: h, FPS: fps}, Audio: &Track{Codec: "opus", Channels: 2}}
}
func TestFit(t *testing.T) {
q := func(id string) Quality { x, _ := qualityByID(id); return x }
for _, c := range []struct {
q string
in Info
codec string
w, h, kbs int
}{
{"1080p", src("hevc", 3840, 2160, 60), "h264", 1920, 1080, 10000},
{"original", src("hevc", 3840, 2160, 30), "h264", 3840, 2160, 20000},
{"720p", src("h264", 2292, 960, 60), "h264", 1718, 720, 5000}, // odd width made even
{"1080p", src("h264", 1280, 720, 30), "h264", 1280, 720, 4000}, // never upscaled
{"480p", src("h264", 1920, 1080, 30), "hevc", 852, 480, 1200}, // 853.3 rounds down, then even
{"original", src("av1", 640, 360, 24), "av1", 640, 360, 1200},
} {
w, h, k := fit(q(c.q), c.in, c.codec)
if w != c.w || h != c.h || k != c.kbs {
t.Errorf("%s %s %dx%d@%.0f → %dx%d %dk, want %dx%d %dk", c.q, c.codec, c.in.Video.Width, c.in.Video.Height, c.in.Video.FPS, w, h, k, c.w, c.h, c.kbs)
}
}
}
func gridCaps() Caps {
return Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac", "libsvtav1", "libx264", "libx265"},
GPUs: []GPU{
{API: "vaapi", Device: "/dev/dri/renderD128", Name: "Intel UHD Graphics 750", Integrated: true, Codecs: []string{"h264", "hevc", "vp9"}, Encodes: []string{"h264", "hevc"}},
{API: "vaapi", Device: "/dev/dri/renderD129", Name: "Intel Arc A380", Codecs: []string{"h264", "hevc", "vp9"}, Encodes: []string{"av1", "h264", "hevc"}},
}}
}
func engineNames(r []Recipe) []string {
out := make([]string, len(r))
for i := range r {
out[i] = r[i].Engine
}
return out
}
func TestRecipesOrder(t *testing.T) {
c := gridCaps()
// h264 from hevc: the card first, decoding on it; then the iGPU the same way; then the CPU
got := engineNames(recipes(c, "h264", src("hevc", 3840, 2160, 60), 1920, 1080, 10000))
want := []string{"Intel Arc A380 (vaapi)", "Intel Arc A380 (vaapi, decoded on the CPU)", "Intel UHD Graphics 750 (vaapi)", "Intel UHD Graphics 750 (vaapi, decoded on the CPU)", "CPU (libx264)"}
if strings.Join(got, "|") != strings.Join(want, "|") {
t.Fatalf("got %v", got)
}
// av1 source: neither card decodes it, so no same-card recipe, and only the Arc encodes av1
got = engineNames(recipes(c, "av1", src("av1", 1920, 1080, 30), 1920, 1080, 4800))
want = []string{"Intel Arc A380 (vaapi, decoded on the CPU)", "CPU (SVT-AV1)"}
if strings.Join(got, "|") != strings.Join(want, "|") {
t.Fatalf("av1: got %v", got)
}
// copy is one recipe, no filter, hvc1 for hevc
r := recipes(c, "copy", src("hevc", 1920, 1080, 30), 1920, 1080, 0)
if len(r) != 1 || r[0].VF != "" || !has(r[0].Video, "hvc1") || !has(r[0].Video, "copy") {
t.Fatalf("copy: %+v", r)
}
// a machine with no GPU and no x264: nothing for h264, something for av1
bare := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac", "libsvtav1"}}
if len(recipes(bare, "h264", src("h264", 1280, 720, 30), 1280, 720, 4000)) != 0 {
t.Fatal("no x264 means no h264")
}
if EngineFor(bare, "av1") != "CPU (SVT-AV1)" || EngineFor(bare, "h264") != "" || EngineFor(bare, "copy") != "Original codec" {
t.Fatal("engineFor")
}
if EngineFor(c, "hevc") != "Intel Arc A380 (vaapi)" {
t.Fatalf("engineFor on grid: %q", EngineFor(c, "hevc"))
}
// HDR gets tone-mapped on both paths
hdr := src("hevc", 3840, 2160, 60)
hdr.Video.ColorTransfer = "smpte2084"
r = recipes(c, "h264", hdr, 1920, 1080, 10000)
if !strings.HasPrefix(r[0].VF, "tonemap_vaapi") || !strings.Contains(r[len(r)-1].VF, "tonemap=hable") {
t.Fatalf("hdr: %q … %q", r[0].VF, r[len(r)-1].VF)
}
}
func TestHLSArgs(t *testing.T) {
in := src("h264", 1920, 1080, 60)
r := recipes(gridCaps(), "h264", in, 1920, 1080, 10000)[0]
a := strings.Join(hlsArgs("/d", "/v.mkv", in, r, false), " ")
for _, want := range []string{"-hwaccel vaapi -hwaccel_device /dev/dri/renderD129", "-i /v.mkv", "-map 0:v:0 -map 0:a:0?", "-vf scale_vaapi=w=1920:h=1080:format=nv12", "-c:v h264_vaapi -b:v 10000k -maxrate 12500k", "-c:a aac -b:a 160k -ac 2", "-g 240 -force_key_frames expr:gte(t,n_forced*4)", "-hls_segment_type fmp4", "-hls_segment_filename /d/seg%05d.m4s", "/d/index.m3u8"} {
if !strings.Contains(a, want) {
t.Errorf("missing %q in\n%s", want, a)
}
}
// aac stays as it is; no audio means none; copy sets no keyframes
in.Audio.Codec = "aac"
if a := strings.Join(hlsArgs("/d", "/v", in, r, false), " "); !strings.Contains(a, "-c:a copy") {
t.Error("aac should be copied")
}
in.Audio = nil
if a := strings.Join(hlsArgs("/d", "/v", in, r, false), " "); !strings.Contains(a, " -an ") {
t.Error("no audio stream → -an")
}
cp := recipes(gridCaps(), "copy", in, 1920, 1080, 0)[0]
if a := strings.Join(hlsArgs("/d", "/v", in, cp, true), " "); strings.Contains(a, "force_key_frames") || strings.Contains(a, "-vf") {
t.Error("copy must not ask for keyframes or filter")
}
}
func TestParseProgress(t *testing.T) {
b := []byte("frame=10\nout_time_us=1000000\nspeed=2.5x\nprogress=continue\nframe=20\nout_time_us=2500000\nspeed=3.1x\nprogress=continue\n")
s, sp := parseProgress(b)
if s != 2.5 || sp != 3.1 {
t.Fatalf("got %v %v", s, sp)
}
}
func TestValidChoice(t *testing.T) {
if err := ValidChoice("h264", "720p"); err != nil {
t.Fatal(err)
}
if ValidChoice("vp8", "720p") == nil || ValidChoice("h264", "9p") == nil || ValidChoice("", "") == nil {
t.Fatal("bad choices must be refused")
}
}
+188
View File
@@ -0,0 +1,188 @@
package media
// Running one long ffmpeg: as the daemon's user directly, or as root through this binary's
// run-ffmpeg helper when the file needs root to read. The helper is told to finish by its
// standard input closing, which also happens by itself if the daemon goes away — so a
// transcode never outlives the console that asked for it.
import (
"bytes"
"context"
"errors"
"io"
"os"
"os/exec"
"strings"
"sync"
"syscall"
"time"
"hamlab/internal/priv"
)
type ffproc struct {
cmd *exec.Cmd
in io.WriteCloser // the helper's stdin, root mode only
done chan struct{}
err error
tail *tailBuf
}
// startFF launches ffmpeg with args, under nice where there is one.
func startFF(c Caps, self string, root bool, args []string) (*ffproc, error) {
var argv []string
if root {
argv = priv.Wrap(append([]string{self, "run-ffmpeg"}, args...)...)
} else {
argv = append([]string{c.FFmpeg}, args...)
if c.Nice != "" {
argv = append([]string{c.Nice, "-n", "10"}, argv...)
}
}
cmd := exec.Command(argv[0], argv[1:]...)
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
p := &ffproc{cmd: cmd, done: make(chan struct{}), tail: &tailBuf{}}
cmd.Stderr = p.tail
cmd.Stdout = io.Discard
if root {
in, err := cmd.StdinPipe()
if err != nil {
return nil, err
}
p.in = in
}
if err := cmd.Start(); err != nil {
return nil, err
}
go func() {
p.err = cmd.Wait()
close(p.done)
}()
return p, nil
}
// exited reports whether ffmpeg has finished, and how.
func (p *ffproc) exited() (bool, error) {
select {
case <-p.done:
return true, p.err
default:
return false, nil
}
}
// end asks ffmpeg to finish and waits a little for it.
func (p *ffproc) end() {
if done, _ := p.exited(); done {
return
}
if p.in != nil {
p.in.Close()
} else {
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGTERM)
}
select {
case <-p.done:
case <-time.After(5 * time.Second):
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGKILL) // a root helper ignores this; it ends on its own within its own grace
select {
case <-p.done:
case <-time.After(10 * time.Second):
}
}
}
// lastWords is what ffmpeg said last, for the record.
func (p *ffproc) lastWords() string { return p.tail.last() }
// tailBuf keeps the last few KB written to it.
type tailBuf struct {
mu sync.Mutex
b []byte
}
const tailKeep = 4096
func (t *tailBuf) Write(b []byte) (int, error) {
t.mu.Lock()
defer t.mu.Unlock()
t.b = append(t.b, b...)
if len(t.b) > tailKeep {
t.b = t.b[len(t.b)-tailKeep:]
}
return len(b), nil
}
func (t *tailBuf) last() string {
t.mu.Lock()
defer t.mu.Unlock()
return lastLine(string(bytes.TrimSpace(t.b)))
}
// RunFFmpegHelper implements `hamlab run-ffmpeg <args…>`: ffmpeg as this user (root, when
// the daemon asked through sudo), finishing when stdin closes. Returns ffmpeg's exit code.
func RunFFmpegHelper(args []string, stdin io.Reader, stderr io.Writer) int {
ffmpeg, err := exec.LookPath("ffmpeg")
if err != nil {
io.WriteString(stderr, "ffmpeg is not installed\n")
return 1
}
argv := []string{ffmpeg}
if nice, err := exec.LookPath("nice"); err == nil {
argv = []string{nice, "-n", "10", ffmpeg}
}
cmd := exec.Command(argv[0], append(argv[1:], args...)...)
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Stderr = stderr
cmd.Stdout = io.Discard
if err := cmd.Start(); err != nil {
io.WriteString(stderr, err.Error()+"\n")
return 1
}
ctx, cancel := context.WithCancel(context.Background())
go func() {
_, _ = io.Copy(io.Discard, stdin) // returns when the daemon lets go
cancel()
}()
done := make(chan error, 1)
go func() { done <- cmd.Wait() }()
var werr error
select {
case werr = <-done:
case <-ctx.Done():
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM)
select {
case werr = <-done:
case <-time.After(5 * time.Second):
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
werr = <-done
}
}
var ee *exec.ExitError
if errors.As(werr, &ee) {
return ee.ExitCode()
}
if werr != nil {
return 1
}
return 0
}
// clearDir empties a folder without removing it.
func clearDir(dir string) error {
ents, err := os.ReadDir(dir)
if err != nil {
return err
}
for _, e := range ents {
if err := os.RemoveAll(dir + "/" + e.Name()); err != nil {
return err
}
}
return nil
}
func hasSegments(playlist string) bool {
b, err := os.ReadFile(playlist)
return err == nil && strings.Contains(string(b), "#EXTINF")
}
+344
View File
@@ -0,0 +1,344 @@
package media
// Playing what the browser cannot play as it is: an HLS stream, re-encoded on demand by
// ffmpeg on the machine's own GPU where it has one it can use, on its CPU otherwise. A
// stream is one file at one codec and size, in a folder of its own under
// <data dir>/transcode/: ffmpeg writes an event playlist and fragmented-MP4 segments there
// as fast as it can, and the player reads them as they land, so the range it can seek in
// grows until the whole file is through — on a GPU, several times faster than watching.
// Nothing is kept: a stream nobody has touched for a while is ended and its folder
// removed, and the folder is emptied when the daemon starts.
import (
"bufio"
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"log"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
"sync"
"time"
)
// Stream is one re-encode, in progress or finished.
type Stream struct {
ID string `json:"id"`
Path string `json:"path"`
Codec string `json:"codec"`
Quality string `json:"quality"`
Engine string `json:"engine"`
Width int `json:"width"`
Height int `json:"height"`
Kbps int `json:"kbps"`
Source Info `json:"source"`
Started time.Time `json:"started"`
Root bool `json:"root"`
key string
dir string
mu sync.Mutex
used time.Time
proc *ffproc
finished bool
failure string
gone bool // forgotten by the Streamer
}
// StreamStatus is what the player asks for while the file is still going through.
type StreamStatus struct {
Running bool `json:"running"`
Done bool `json:"done"`
Error string `json:"error,omitempty"`
Progress float64 `json:"progress"` // seconds of output written
Speed float64 `json:"speed"` // × realtime
Segments int `json:"segments"`
}
// Streamer keeps one machine's streams.
type Streamer struct {
dir string
self string
limit int
mu sync.Mutex
streams map[string]*Stream
}
const streamIdle = 10 * time.Minute
const firstSegmentWait = 40 * time.Second
var streamFile = regexp.MustCompile(`^(index\.m3u8|init\.mp4|seg\d{5}\.m4s)$`)
// NewStreamer empties the folder — a daemon restart orphans every stream — and starts the
// sweeper.
func NewStreamer(dataDir, self string, limit int) *Streamer {
m := &Streamer{dir: filepath.Join(dataDir, "transcode"), self: self, limit: limit, streams: map[string]*Stream{}}
_ = os.MkdirAll(m.dir, 0o755)
_ = clearDir(m.dir)
go func() {
for range time.Tick(30 * time.Second) {
m.Sweep(streamIdle)
}
}()
return m
}
// Open starts a stream, or hands back the one already serving the same request. It
// returns once the first segment is on disk, so the player's first fetch is answered.
func (m *Streamer) Open(ctx context.Context, c Caps, root bool, path, codec, quality string, src Info) (*Stream, error) {
if c.Error != "" {
return nil, errors.New(c.Error)
}
if err := ValidChoice(codec, quality); err != nil {
return nil, err
}
q, _ := qualityByID(quality)
if src.Video == nil {
return nil, &DecodeError{Msg: "no video stream"}
}
if src.Video.Width <= 0 || src.Video.Height <= 0 {
return nil, &DecodeError{Msg: "the picture's size is unknown"}
}
key := fmt.Sprintf("%s|%d|%.0f|%s|%s|%v", path, src.Size, src.Duration, codec, quality, root)
m.mu.Lock()
for _, s := range m.streams {
if s.key == key {
if s.failed() { // not the stream to hand back: let it go and make a fresh one
delete(m.streams, s.ID)
_ = os.RemoveAll(s.dir)
break
}
m.mu.Unlock()
s.touch()
return s, nil
}
}
running := 0
for _, s := range m.streams {
if s.running() {
running++
}
}
if running >= m.limit {
m.mu.Unlock()
return nil, fmt.Errorf("%d videos are already being re-encoded on this machine; one must finish or be closed first", running)
}
s := &Stream{ID: newID(), Path: path, Codec: codec, Quality: quality, Source: src, Started: time.Now(), Root: root, key: key, used: time.Now()}
s.dir = filepath.Join(m.dir, s.ID)
m.streams[s.ID] = s
m.mu.Unlock()
s.Width, s.Height, s.Kbps = fit(q, src, codec)
if codec == "copy" {
s.Width, s.Height, s.Kbps = src.Video.Width, src.Video.Height, int(src.BitRate/1000)
}
rs := recipes(c, codec, src, s.Width, s.Height, s.Kbps)
if len(rs) == 0 {
m.forget(s)
return nil, errors.New("nothing on this machine encodes " + codec)
}
var last string
for _, r := range rs {
if err := os.MkdirAll(s.dir, 0o755); err != nil { // the root helper's ffmpeg writes here too
m.forget(s)
return nil, err
}
_ = clearDir(s.dir)
args := hlsArgs(s.dir, path, src, r, codec == "copy")
p, err := startFF(c, m.self, root, args)
if err != nil {
last = r.Engine + ": " + err.Error()
continue
}
s.mu.Lock()
s.proc, s.Engine = p, r.Engine
s.mu.Unlock()
msg, ok := s.waitFirstSegment(ctx)
if ok {
go s.watch()
log.Printf("media: stream %s: %s → %s %s %dx%d %dk on %s", s.ID, filepath.Base(path), codec, quality, s.Width, s.Height, s.Kbps, r.Engine)
return s, nil
}
p.end()
if ctx.Err() != nil || s.closed() { // the caller left, or somebody closed it meanwhile: not the recipe's fault
m.forget(s)
return nil, errors.New("cancelled")
}
last = r.Engine + ": " + msg
log.Printf("media: stream %s on %s: %s", filepath.Base(path), r.Engine, msg)
}
m.forget(s)
return nil, &DecodeError{Msg: last}
}
// waitFirstSegment watches for the playlist's first segment, or ffmpeg giving up.
func (s *Stream) waitFirstSegment(ctx context.Context) (string, bool) {
deadline := time.Now().Add(firstSegmentWait)
playlist := filepath.Join(s.dir, "index.m3u8")
for {
if hasSegments(playlist) {
return "", true
}
if done, err := s.proc.exited(); done {
msg := s.proc.lastWords()
if msg == "" && err != nil {
msg = err.Error()
}
if msg == "" {
msg = "ffmpeg ended before writing a segment"
}
return msg, false
}
if time.Now().After(deadline) {
return "no segment after " + firstSegmentWait.String(), false
}
select {
case <-ctx.Done():
return ctx.Err().Error(), false
case <-time.After(250 * time.Millisecond):
}
}
}
// watch records how the encode ended.
func (s *Stream) watch() {
<-s.proc.done
s.mu.Lock()
s.finished = true
if s.proc.err != nil {
s.failure = s.proc.lastWords()
if s.failure == "" {
s.failure = s.proc.err.Error()
}
}
s.mu.Unlock()
if s.failure != "" {
log.Printf("media: stream %s ended early: %s", s.ID, s.failure)
}
}
func (m *Streamer) Find(id string) (*Stream, bool) {
m.mu.Lock()
defer m.mu.Unlock()
s, ok := m.streams[id]
return s, ok
}
// Close ends a stream and removes its folder.
func (m *Streamer) Close(id string) {
m.mu.Lock()
s, ok := m.streams[id]
m.mu.Unlock()
if !ok {
return
}
s.mu.Lock()
p := s.proc
s.mu.Unlock()
if p != nil {
p.end()
}
m.forget(s)
}
func (m *Streamer) forget(s *Stream) {
m.mu.Lock()
delete(m.streams, s.ID)
m.mu.Unlock()
s.mu.Lock()
s.gone = true
s.mu.Unlock()
_ = os.RemoveAll(s.dir)
}
func (s *Stream) closed() bool { s.mu.Lock(); defer s.mu.Unlock(); return s.gone }
func (s *Stream) failed() bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.finished && s.failure != ""
}
// Sweep closes what nobody has touched for a while.
func (m *Streamer) Sweep(idle time.Duration) {
m.mu.Lock()
var old []string
for id, s := range m.streams {
s.mu.Lock()
if time.Since(s.used) > idle {
old = append(old, id)
}
s.mu.Unlock()
}
m.mu.Unlock()
for _, id := range old {
log.Printf("media: stream %s idle, closed", id)
m.Close(id)
}
}
// File is the on-disk path of one of the stream's files — the playlist, the init segment
// or a media segment — and nothing else.
func (s *Stream) File(name string) (string, bool) {
if !streamFile.MatchString(name) {
return "", false
}
s.touch()
return filepath.Join(s.dir, name), true
}
func (s *Stream) touch() { s.mu.Lock(); s.used = time.Now(); s.mu.Unlock() }
func (s *Stream) running() bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.proc != nil && !s.finished
}
// Status is read from ffmpeg's own progress file and the playlist.
func (s *Stream) Status() StreamStatus {
s.mu.Lock()
st := StreamStatus{Running: s.proc != nil && !s.finished, Done: s.finished && s.failure == "", Error: s.failure}
s.mu.Unlock()
if b, err := os.ReadFile(filepath.Join(s.dir, "progress")); err == nil {
st.Progress, st.Speed = parseProgress(b)
}
if b, err := os.ReadFile(filepath.Join(s.dir, "index.m3u8")); err == nil {
st.Segments = strings.Count(string(b), "#EXTINF")
}
return st
}
// parseProgress takes the last out_time_us and speed from ffmpeg's -progress output.
func parseProgress(b []byte) (seconds, speed float64) {
sc := bufio.NewScanner(strings.NewReader(string(b)))
for sc.Scan() {
k, v, ok := strings.Cut(sc.Text(), "=")
if !ok {
continue
}
v = strings.TrimSpace(v)
switch k {
case "out_time_us":
if us, err := strconv.ParseInt(v, 10, 64); err == nil && us >= 0 {
seconds = float64(us) / 1e6
}
case "speed":
if x, err := strconv.ParseFloat(strings.TrimSuffix(v, "x"), 64); err == nil {
speed = x
}
}
}
return seconds, speed
}
func newID() string {
b := make([]byte, 8)
_, _ = rand.Read(b)
return hex.EncodeToString(b)
}
+85
View File
@@ -0,0 +1,85 @@
package media
import (
"context"
"os"
"os/exec"
"path/filepath"
"testing"
"time"
)
// The real thing, when ffmpeg is here: a generated clip goes through as H.264 and comes out
// as a playlist of segments, the status follows it to the end, and closing removes it all.
func TestStreamWithFFmpeg(t *testing.T) {
if _, err := exec.LookPath("ffmpeg"); err != nil {
t.Skip("no ffmpeg")
}
c := Probe(context.Background())
if c.Error != "" {
t.Skip(c.Error)
}
if EngineFor(c, "h264") == "" {
t.Skip("nothing here encodes h264")
}
dir := t.TempDir()
clip := filepath.Join(dir, "clip.mp4")
out, err := exec.Command("ffmpeg", "-nostdin", "-v", "error", "-f", "lavfi", "-i", "testsrc=size=640x360:rate=25:duration=9", "-f", "lavfi", "-i", "sine=frequency=440:duration=9",
"-c:v", "mpeg4", "-q:v", "5", "-c:a", "aac", "-shortest", clip).CombinedOutput()
if err != nil {
t.Skipf("could not make a clip: %s", out)
}
src, err := Inspect(context.Background(), c, clip, false)
if err != nil || src.Video == nil || src.Audio == nil || src.Video.Codec != "mpeg4" || src.Duration < 8 {
t.Fatalf("inspect: %+v %v", src, err)
}
m := NewStreamer(dir, "hamlab", 2)
s, err := m.Open(context.Background(), c, false, clip, "h264", "480p", src)
if err != nil {
t.Fatal(err)
}
if s.Width != 640 || s.Height != 360 || s.Engine == "" {
t.Fatalf("stream: %+v", s)
}
if p, ok := s.File("index.m3u8"); !ok {
t.Fatal("no playlist")
} else if !hasSegments(p) {
t.Fatal("Open returned before the first segment")
}
for _, bad := range []string{"../x", "progress", "seg1.m4s", "index.m3u8/", "init.mp4.tmp"} {
if _, ok := s.File(bad); ok {
t.Fatalf("%q must not be served", bad)
}
}
// the same request is the same stream
if again, err := m.Open(context.Background(), c, false, clip, "h264", "480p", src); err != nil || again.ID != s.ID {
t.Fatal("not reused")
}
var st StreamStatus
for i := 0; i < 120; i++ {
st = s.Status()
if st.Done || st.Error != "" {
break
}
time.Sleep(250 * time.Millisecond)
}
if !st.Done || st.Error != "" || st.Segments < 2 || st.Progress < 8 {
t.Fatalf("status at the end: %+v", st)
}
if init, _ := s.File("init.mp4"); !exists(init) {
t.Fatal("no init segment")
}
m.Close(s.ID)
if _, ok := m.Find(s.ID); ok || exists(s.dir) {
t.Fatal("close must forget and remove")
}
// an unplayable source is a DecodeError, and leaves nothing behind
bad := filepath.Join(dir, "bad.mkv")
_ = os.WriteFile(bad, []byte("not a video"), 0o644)
if _, err := Inspect(context.Background(), c, bad, false); err == nil {
t.Fatal("inspect of garbage")
}
if left, _ := os.ReadDir(m.dir); len(left) != 0 {
t.Fatalf("left behind: %v", left)
}
}
+142
View File
@@ -0,0 +1,142 @@
package media
import (
"bytes"
"context"
"encoding/json"
"errors"
"os/exec"
"strconv"
"strings"
"hamlab/internal/priv"
)
// Track is one stream of a media file, as ffprobe reports it.
type Track struct {
Index int `json:"index"`
Type string `json:"type"` // video | audio
Codec string `json:"codec"`
Profile string `json:"profile,omitempty"`
PixFmt string `json:"pix_fmt,omitempty"`
Width int `json:"width,omitempty"`
Height int `json:"height,omitempty"`
FPS float64 `json:"fps,omitempty"`
ColorTransfer string `json:"color_transfer,omitempty"` // smpte2084 / arib-std-b67 mean HDR
Channels int `json:"channels,omitempty"`
AttachedPic bool `json:"attached_pic,omitempty"`
}
// Info is what a file holds: the picture to take, the sound to go with it, and the whole.
type Info struct {
Container string `json:"container"` // "matroska,webm", "mov,mp4,m4a,3gp,3g2,mj2"
Duration float64 `json:"duration"`
BitRate int64 `json:"bit_rate,omitempty"` // bits per second, whole file
Size int64 `json:"size,omitempty"`
Video *Track `json:"video,omitempty"` // the first real picture stream; cover art is not it
Audio *Track `json:"audio,omitempty"` // the first sound stream
}
// HDR reports a picture that needs tone mapping to look right on an SDR screen.
func (i Info) HDR() bool {
return i.Video != nil && (i.Video.ColorTransfer == "smpte2084" || i.Video.ColorTransfer == "arib-std-b67")
}
type probeJSON struct {
Streams []struct {
Index int `json:"index"`
CodecType string `json:"codec_type"`
CodecName string `json:"codec_name"`
Profile string `json:"profile"`
PixFmt string `json:"pix_fmt"`
Width int `json:"width"`
Height int `json:"height"`
RFrameRate string `json:"r_frame_rate"`
AvgFrameRate string `json:"avg_frame_rate"`
ColorTransfer string `json:"color_transfer"`
Channels int `json:"channels"`
Disposition struct {
AttachedPic int `json:"attached_pic"`
} `json:"disposition"`
} `json:"streams"`
Format struct {
FormatName string `json:"format_name"`
Duration string `json:"duration"`
BitRate string `json:"bit_rate"`
Size string `json:"size"`
} `json:"format"`
}
// Inspect asks ffprobe what a file is. As root through sudo when asked, for a file only
// root may read. An error about the file itself is a DecodeError.
func Inspect(ctx context.Context, c Caps, path string, root bool) (Info, error) {
if c.FFprobe == "" {
return Info{}, errors.New("ffprobe is not installed")
}
argv := []string{c.FFprobe, "-v", "error", "-show_entries",
"stream=index,codec_type,codec_name,profile,pix_fmt,width,height,r_frame_rate,avg_frame_rate,color_transfer,channels,disposition:format=format_name,duration,bit_rate,size",
"-of", "json", path}
if root {
argv = priv.Wrap(argv...)
}
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
var errb bytes.Buffer
cmd.Stderr = &errb
out, err := cmd.Output()
if err != nil {
if ctx.Err() != nil {
return Info{}, ctx.Err()
}
msg := lastLine(errb.String())
if msg == "" {
msg = "ffprobe: " + err.Error()
}
return Info{}, &DecodeError{Msg: msg}
}
var pj probeJSON
if err := json.Unmarshal(out, &pj); err != nil {
return Info{}, &DecodeError{Msg: "ffprobe: unreadable answer"}
}
in := Info{Container: pj.Format.FormatName}
in.Duration, _ = strconv.ParseFloat(pj.Format.Duration, 64)
in.BitRate, _ = strconv.ParseInt(pj.Format.BitRate, 10, 64)
in.Size, _ = strconv.ParseInt(pj.Format.Size, 10, 64)
var art *Track
for _, s := range pj.Streams {
st := &Track{Index: s.Index, Type: s.CodecType, Codec: s.CodecName, Profile: s.Profile, PixFmt: s.PixFmt, Width: s.Width, Height: s.Height,
ColorTransfer: s.ColorTransfer, Channels: s.Channels, AttachedPic: s.Disposition.AttachedPic != 0}
st.FPS = fps(s.AvgFrameRate)
if st.FPS == 0 {
st.FPS = fps(s.RFrameRate)
}
switch {
case s.CodecType == "video" && st.AttachedPic:
if art == nil {
art = st
}
case s.CodecType == "video" && in.Video == nil:
in.Video = st
case s.CodecType == "audio" && in.Audio == nil:
in.Audio = st
}
}
if in.Video == nil {
in.Video = art // cover art is the only picture there is
}
return in, nil
}
// fps parses ffprobe's "60000/1001".
func fps(r string) float64 {
n, d, ok := strings.Cut(r, "/")
if !ok {
v, _ := strconv.ParseFloat(r, 64)
return v
}
nf, _ := strconv.ParseFloat(n, 64)
df, _ := strconv.ParseFloat(d, 64)
if df == 0 {
return 0
}
return nf / df
}
+26
View File
@@ -0,0 +1,26 @@
package media
import (
"fmt"
"os"
"os/user"
"strconv"
"syscall"
)
// ownerOf is "root:render 0660" for a device node, the thing to say when it cannot be opened.
func ownerOf(st os.FileInfo) string {
sys, ok := st.Sys().(*syscall.Stat_t)
if !ok {
return ""
}
u := strconv.Itoa(int(sys.Uid))
if x, err := user.LookupId(u); err == nil {
u = x.Username
}
g := strconv.Itoa(int(sys.Gid))
if x, err := user.LookupGroupId(g); err == nil {
g = x.Name
}
return fmt.Sprintf("%s:%s %04o", u, g, st.Mode().Perm())
}
+656
View File
@@ -0,0 +1,656 @@
// Package thumbs makes the Files tab's previews: one small JPEG for an image or a video,
// by ffmpeg, decoded on a GPU where the machine has one it can use and on the CPU
// otherwise.
//
// Nothing about the machine is assumed. A probe looks ffmpeg and ffprobe up, opens every
// /dev/dri/renderD* and exercises it (a VAAPI upload and download of a synthetic frame),
// so a node the daemon's user cannot open, or a driver that does not speak VAAPI, is
// reported once rather than tried per file. vainfo, where installed, says which codecs
// each node decodes, and a video goes to the first node that lists its codec; without
// vainfo the first working node is tried and ffmpeg falls back to software on its own when
// the stream turns out unsupported. A hardware run that fails outright is retried on the
// CPU before the file is called undecodable. Images are always decoded on the CPU: one
// frame of a JPEG at a reduced DCT size costs less than the trip through a GPU.
package media
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"time"
"hamlab/internal/metrics"
"hamlab/internal/priv"
)
// GPU is one hardware decoder as the probe found it.
type GPU struct {
API string `json:"api"` // vaapi | cuda
Device string `json:"device"` // /dev/dri/renderD128; "0" for cuda (its device index)
Name string `json:"name,omitempty"` // "Intel Arc A380"
Codecs []string `json:"codecs,omitempty"` // ffmpeg codec names it decodes; nil when nothing could say
Encodes []string `json:"encodes,omitempty"` // ffmpeg codec names it encodes, from vainfo
Integrated bool `json:"integrated,omitempty"` // on the CPU die; a card on the bus is preferred for encoding
Reason string `json:"reason,omitempty"` // Unusable only: why
Permission bool `json:"permission,omitempty"` // Unusable only: the node exists but this user may not open it
}
// Caps is what the machine can make previews with, as probed.
type Caps struct {
FFmpeg string `json:"ffmpeg,omitempty"`
FFprobe string `json:"ffprobe,omitempty"`
Nice string `json:"nice,omitempty"`
Version string `json:"version,omitempty"`
GPUs []GPU `json:"gpus"` // usable, in node order
Encoders []string `json:"encoders,omitempty"` // the encoders ffmpeg itself was built with that matter here
Unusable []GPU `json:"unusable,omitempty"` // render nodes that exist but cannot be used, with the reason
Error string `json:"error,omitempty"` // why nothing can be made at all
ProbedAt time.Time `json:"probed_at"`
}
// Opts shapes one preview.
type Opts struct {
Size int // the long edge in pixels; 256 when zero
Timeout time.Duration // for the whole job, ffprobe and every ffmpeg attempt; 30 s when zero
}
func (o Opts) defaults() Opts {
if o.Size <= 0 {
o.Size = 256
}
if o.Timeout <= 0 {
o.Timeout = 30 * time.Second
}
return o
}
// Result is a finished preview.
type Result struct {
JPEG []byte `json:"-"`
Engine string `json:"engine"` // "cpu", or "vaapi:/dev/dri/renderD129"
Codec string `json:"codec"`
Width int `json:"width"`
Height int `json:"height"`
}
// ErrUnsupported: not an image or a video by its name.
var ErrUnsupported = errors.New("not an image or video")
// A DecodeError is ffmpeg's verdict on the file itself. It will not change until the
// file does, so the caller may remember it.
type DecodeError struct{ Msg string }
func (e *DecodeError) Error() string { return e.Msg }
// ---- what gets a preview ----
var imageExt = set("png jpg jpeg jpe jfif gif webp bmp tif tiff avif heic heif hif jxl svg ico psd tga thm")
var videoExt = set("mp4 m4v mkv webm mov avi wmv flv mpg mpeg mpe m2v ts m2ts mts 3gp 3g2 ogv ogm mxf vob asf rm rmvb divx f4v mjpeg mjpg 360 osv insv lrf")
var audioExt = set("mp3 flac m4a m4b aac ogg oga opus wav wma aiff aif alac ape wv mka")
func set(s string) map[string]bool {
m := map[string]bool{}
for _, w := range strings.Fields(s) {
m[w] = true
}
return m
}
// Kind is "image", "video", "audio" or "" for a file name. An audio file's preview is its
// cover art, when it carries any.
func Kind(name string) string {
ext := strings.ToLower(strings.TrimPrefix(filepath.Ext(name), "."))
switch {
case imageExt[ext]:
return "image"
case videoExt[ext]:
return "video"
case audioExt[ext]:
return "audio"
}
return ""
}
// ---- the probe ----
// Probe finds out what this process can make previews with. It runs ffmpeg against every
// render node, so it takes a few hundred milliseconds; callers keep the answer.
func Probe(ctx context.Context) Caps {
c := Caps{ProbedAt: time.Now(), GPUs: []GPU{}}
c.FFmpeg, _ = exec.LookPath("ffmpeg")
c.FFprobe, _ = exec.LookPath("ffprobe")
c.Nice, _ = exec.LookPath("nice")
switch {
case c.FFmpeg == "" && c.FFprobe == "":
c.Error = "ffmpeg is not installed"
case c.FFmpeg == "":
c.Error = "ffmpeg is not installed (ffprobe is)"
case c.FFprobe == "":
c.Error = "ffprobe is not installed (ffmpeg is)"
}
if c.Error != "" {
return c
}
c.Version = ffmpegVersion(ctx, c.FFmpeg)
c.Encoders = encoders(ctx, c.FFmpeg)
accels := hwaccels(ctx, c.FFmpeg)
if accels["vaapi"] {
nodes, _ := filepath.Glob("/dev/dri/renderD*")
sort.Strings(nodes)
vainfo, _ := exec.LookPath("vainfo")
for _, dev := range nodes {
g := GPU{API: "vaapi", Device: dev, Name: metrics.RenderNodeName(dev), Integrated: metrics.RenderNodeIntegrated(dev)}
if err := openable(dev); err != nil {
g.Reason, g.Permission = err.Error(), errors.Is(err, os.ErrPermission)
c.Unusable = append(c.Unusable, g)
continue
}
if msg := hwWorks(ctx, c.FFmpeg, "vaapi=va:"+dev); msg != "" {
g.Reason = msg
c.Unusable = append(c.Unusable, g)
continue
}
if vainfo != "" {
g.Codecs, g.Encodes = vainfoCodecs(ctx, vainfo, dev)
}
c.GPUs = append(c.GPUs, g)
}
}
if accels["cuda"] && exists("/dev/nvidiactl") {
g := GPU{API: "cuda", Device: "0", Name: "NVIDIA GPU"}
if msg := hwWorks(ctx, c.FFmpeg, "cuda=cu:0"); msg != "" {
g.Reason = msg
c.Unusable = append(c.Unusable, g)
} else {
c.GPUs = append(c.GPUs, g)
}
}
return c
}
// openable: the node can be opened read-write, which is what a VAAPI driver needs. The
// usual reason it can't be is the owning group: a render node is root:render 0660 on
// most distributions, and a daemon running as an ordinary user is not in render.
func openable(dev string) error {
f, err := os.OpenFile(dev, os.O_RDWR, 0)
if err != nil {
if errors.Is(err, os.ErrPermission) {
owner := ""
if st, serr := os.Stat(dev); serr == nil {
owner = ownerOf(st)
}
if owner != "" {
return fmt.Errorf("%w: %s is %s", os.ErrPermission, dev, owner)
}
return fmt.Errorf("%w: cannot open %s", os.ErrPermission, dev)
}
return err
}
f.Close()
return nil
}
// hwWorks initialises a hardware device in ffmpeg and moves one synthetic frame through
// it. "" when it works, else the last thing ffmpeg said.
func hwWorks(ctx context.Context, ffmpeg, device string) string {
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
defer cancel()
name, rest, _ := strings.Cut(device, "=") // "vaapi", "va:/dev/…"
label, _, _ := strings.Cut(rest, ":") // "va"
cmd := exec.CommandContext(ctx, ffmpeg, "-nostdin", "-hide_banner", "-v", "error", "-init_hw_device", device, "-filter_hw_device", label,
"-f", "lavfi", "-i", "color=c=black:s=64x64:r=1:d=1", "-vf", "format=nv12,hwupload,hwdownload,format=nv12", "-frames:v", "1", "-f", "null", "-")
var errb bytes.Buffer
cmd.Stderr = &errb
if err := cmd.Run(); err != nil {
msg := lastLine(errb.String())
if msg == "" {
msg = err.Error()
}
return name + ": " + msg
}
return ""
}
func ffmpegVersion(ctx context.Context, ffmpeg string) string {
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
out, _ := exec.CommandContext(ctx, ffmpeg, "-version").Output()
f := strings.Fields(string(out)) // "ffmpeg version 7.1.5-0+deb13u1 Copyright …"
if len(f) >= 3 && f[1] == "version" {
v := f[2]
if i := strings.IndexAny(v, "-+~"); i > 0 {
v = v[:i]
}
return v
}
return ""
}
func hwaccels(ctx context.Context, ffmpeg string) map[string]bool {
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
out, _ := exec.CommandContext(ctx, ffmpeg, "-hide_banner", "-hwaccels").Output()
m := map[string]bool{}
for _, l := range strings.Split(string(out), "\n") {
l = strings.TrimSpace(l)
if l != "" && !strings.HasPrefix(l, "Hardware acceleration methods") {
m[l] = true
}
}
return m
}
var vaProfileRe = regexp.MustCompile(`VAProfile([A-Za-z0-9]+?)(?:Simple|Main|High|Profile|Baseline|Constrained|Scc|Advanced|Version|Stereo|Multiview)\w*\s*:\s*VAEntrypoint(VLD|EncSlice|EncSliceLP)\b`)
// wantedEncoders are the encoders this package may ask ffmpeg for.
var wantedEncoders = set("libx264 libx265 libsvtav1 libaom-av1 libopenh264 h264_vaapi hevc_vaapi av1_vaapi aac")
// encoders lists the wanted encoders this ffmpeg was built with.
func encoders(ctx context.Context, ffmpeg string) []string {
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
out, _ := exec.CommandContext(ctx, ffmpeg, "-hide_banner", "-encoders").Output()
var found []string
for _, l := range strings.Split(string(out), "\n") {
f := strings.Fields(l)
if len(f) >= 2 && wantedEncoders[f[1]] {
found = append(found, f[1])
}
}
sort.Strings(found)
return found
}
// vaCodec maps a VAProfile family to the ffmpeg codec it decodes.
var vaCodec = map[string]string{"H264": "h264", "HEVC": "hevc", "VP9": "vp9", "AV1": "av1", "MPEG2": "mpeg2video", "VC1": "vc1", "VP8": "vp8", "JPEG": "mjpeg", "MPEG4": "mpeg4", "H263": "h263"}
func vainfoCodecs(ctx context.Context, vainfo, dev string) (decodes, encodes []string) {
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
out, _ := exec.CommandContext(ctx, vainfo, "--display", "drm", "--device", dev).Output()
return parseVainfo(string(out))
}
// parseVainfo names the codecs the decode (VLD) and encode (EncSlice) profiles cover,
// the way ffmpeg names them.
func parseVainfo(text string) (decodes, encodes []string) {
dec, enc := map[string]bool{}, map[string]bool{}
for _, m := range vaProfileRe.FindAllStringSubmatch(text, -1) {
c, ok := vaCodec[m[1]]
if !ok {
continue
}
if m[2] == "VLD" {
dec[c] = true
} else {
enc[c] = true
}
}
return keys(dec), keys(enc)
}
func keys(m map[string]bool) []string {
if len(m) == 0 {
return nil
}
out := make([]string, 0, len(m))
for c := range m {
out = append(out, c)
}
sort.Strings(out)
return out
}
// pick is the GPU for a codec: the first that lists it, else the first that could not
// say (ffmpeg will fall back to software by itself if that guess is wrong), else none.
func (c Caps) pick(codec string) *GPU {
var unknown *GPU
for i := range c.GPUs {
g := &c.GPUs[i]
if g.Codecs == nil {
if unknown == nil {
unknown = g
}
continue
}
for _, x := range g.Codecs {
if x == codec {
return g
}
}
}
return unknown
}
// ---- making one ----
type fileInfo struct {
index string // the stream to take
codec string
w, h int
duration float64
}
type attempt struct {
gpu *GPU
lowres int
seek float64
}
func (a attempt) engine() string {
if a.gpu == nil {
return "cpu"
}
return a.gpu.API + ":" + a.gpu.Device
}
// Make produces the preview of one file.
func Make(ctx context.Context, c Caps, path string, o Opts) (Result, error) {
kind := Kind(path)
if kind == "" {
return Result{}, ErrUnsupported
}
if c.Error != "" {
return Result{}, errors.New(c.Error)
}
o = o.defaults()
ctx, cancel := context.WithTimeout(ctx, o.Timeout)
defer cancel()
fi, err := probeFile(ctx, c, path)
if err != nil {
return Result{}, err
}
var last string
for _, eng := range engines(c, kind, fi) {
for _, a := range attempts(eng, kind, fi, o.Size) {
data, msg, err := run(ctx, c, a, path, fi, o.Size)
if err == nil && len(data) > 0 {
return Result{JPEG: data, Engine: a.engine(), Codec: fi.codec, Width: fi.w, Height: fi.h}, nil
}
if ctx.Err() != nil {
return Result{}, ctx.Err() // the clock ran out: not the file's fault
}
if err != nil { // the decoder refused: try the next engine, not another seek
last = msg
break
}
last = "no frame at " + strconv.FormatFloat(a.seek, 'f', 1, 64) + " s"
}
}
if last == "" {
last = "could not decode"
}
return Result{}, &DecodeError{Msg: last}
}
// engines is the order to try: the fitting GPU first for a video, then the CPU.
func engines(c Caps, kind string, fi fileInfo) []*GPU {
if kind == "video" {
if g := c.pick(fi.codec); g != nil {
return []*GPU{g, nil}
}
}
return []*GPU{nil}
}
// attempts for one engine: a seek into a video, then from its start if that found no
// frame (a wrong duration in the header); a reduced-size decode of a JPEG, then full size.
func attempts(gpu *GPU, kind string, fi fileInfo, size int) []attempt {
if kind == "video" {
s := seekFor(fi.duration)
if s > 0 {
return []attempt{{gpu: gpu, seek: s}, {gpu: gpu}}
}
return []attempt{{gpu: gpu}}
}
if fi.codec == "mjpeg" && gpu == nil {
if l := lowresFor(fi.w, fi.h, size); l > 0 {
return []attempt{{lowres: l}, {}}
}
}
return []attempt{{gpu: gpu}}
}
// seekFor: a tenth of the way in, never more than 30 s, so a title card or a black lead-in
// is skipped and a long file still seeks to a keyframe near its start.
func seekFor(duration float64) float64 {
if duration <= 0 {
return 0
}
s := duration / 10
if s > 30 {
s = 30
}
if s < 0.5 {
return 0
}
return s
}
// lowresFor: how many halvings a JPEG decoder may take while the picture stays at least
// twice the preview size. libjpeg does this in the DCT, so a 24 MP photo decodes at 1.5 MP.
func lowresFor(w, h, size int) int {
m := w
if h < m {
m = h
}
l := 0
for l < 3 && m>>(l+1) >= size*2 {
l++
}
return l
}
type probeOut struct {
Streams []struct {
Index int `json:"index"`
Codec string `json:"codec_name"`
Width int `json:"width"`
Height int `json:"height"`
Disposition struct {
AttachedPic int `json:"attached_pic"`
} `json:"disposition"`
} `json:"streams"`
Format struct {
Duration string `json:"duration"`
} `json:"format"`
}
// probeFile asks ffprobe which stream to take and what is in it.
func probeFile(ctx context.Context, c Caps, path string) (fileInfo, error) {
cmd := exec.CommandContext(ctx, c.FFprobe, "-v", "error", "-select_streams", "v", "-show_entries", "stream=index,codec_name,width,height,disposition:format=duration", "-of", "json", path)
var errb bytes.Buffer
cmd.Stderr = &errb
out, err := cmd.Output()
if err != nil {
if ctx.Err() != nil {
return fileInfo{}, ctx.Err()
}
msg := lastLine(errb.String())
if msg == "" {
msg = "ffprobe: " + err.Error()
}
return fileInfo{}, &DecodeError{Msg: msg}
}
var po probeOut
if err := json.Unmarshal(out, &po); err != nil {
return fileInfo{}, &DecodeError{Msg: "ffprobe: unreadable answer"}
}
if len(po.Streams) == 0 {
return fileInfo{}, &DecodeError{Msg: "no video stream"}
}
s := po.Streams[0]
for _, x := range po.Streams { // cover art rides along as a video stream; the picture is not the film
if x.Disposition.AttachedPic == 0 {
s = x
break
}
}
fi := fileInfo{index: strconv.Itoa(s.Index), codec: s.Codec, w: s.Width, h: s.Height}
fi.duration, _ = strconv.ParseFloat(po.Format.Duration, 64)
return fi, nil
}
// run is one ffmpeg: the JPEG on success; on failure what ffmpeg said and the error.
// Empty output with a clean exit means the seek found nothing.
func run(ctx context.Context, c Caps, a attempt, path string, fi fileInfo, size int) ([]byte, string, error) {
args := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
if a.gpu != nil {
args = append(args, "-hwaccel", a.gpu.API, "-hwaccel_device", a.gpu.Device)
}
if a.lowres > 0 {
args = append(args, "-lowres", strconv.Itoa(a.lowres))
}
if a.seek > 0 {
args = append(args, "-noaccurate_seek", "-ss", strconv.FormatFloat(a.seek, 'f', 3, 64))
}
args = append(args, "-i", path, "-map", "0:"+fi.index, "-frames:v", "1",
"-vf", fmt.Sprintf("scale=w=%d:h=%d:force_original_aspect_ratio=decrease:flags=bicubic", size, size),
"-f", "image2pipe", "-c:v", "mjpeg", "-q:v", "4", "pipe:1")
name, argv := c.FFmpeg, args
if c.Nice != "" { // a burst of previews must not crowd out what the machine is for
name, argv = c.Nice, append([]string{"-n", "10", c.FFmpeg}, args...)
}
cmd := exec.CommandContext(ctx, name, argv...)
var out, errb bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errb
if err := cmd.Run(); err != nil {
msg := lastLine(errb.String())
if msg == "" {
msg = err.Error()
}
return nil, msg, err
}
return out.Bytes(), "", nil
}
// ---- the same, as root ----
// ProbeRoot is Probe run as root through this binary, for the Files tab's root mode: root
// opens every render node, whatever group the daemon's user is in.
func ProbeRoot(ctx context.Context, self string) (Caps, error) {
full := priv.Wrap(self, "thumb", "--probe")
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
var errb bytes.Buffer
cmd.Stderr = &errb
out, err := cmd.Output()
if err != nil {
if msg := lastLine(errb.String()); msg != "" {
return Caps{}, errors.New(msg)
}
return Caps{}, err
}
var c Caps
if err := json.Unmarshal(out, &c); err != nil {
return Caps{}, err
}
return c, nil
}
type helperOut struct {
Engine string `json:"engine,omitempty"`
Codec string `json:"codec,omitempty"`
Width int `json:"width,omitempty"`
Height int `json:"height,omitempty"`
JPEG []byte `json:"jpeg,omitempty"`
Error string `json:"error,omitempty"`
Permanent bool `json:"permanent,omitempty"` // a DecodeError: the file, not the moment
}
// MakeRoot is Make run as root through this binary, with the caps ProbeRoot found.
func MakeRoot(ctx context.Context, self string, c Caps, path string, o Opts) (Result, error) {
o = o.defaults()
caps, _ := json.Marshal(c)
full := priv.Wrap(self, "thumb", "--size", strconv.Itoa(o.Size), "--timeout", o.Timeout.String(), path)
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
cmd.Stdin = bytes.NewReader(caps)
var errb bytes.Buffer
cmd.Stderr = &errb
out, err := cmd.Output()
if err != nil {
if ctx.Err() != nil {
return Result{}, ctx.Err()
}
if msg := lastLine(errb.String()); msg != "" {
return Result{}, errors.New(msg)
}
return Result{}, err
}
var h helperOut
if err := json.Unmarshal(out, &h); err != nil {
return Result{}, err
}
if h.Error != "" {
if h.Permanent {
return Result{}, &DecodeError{Msg: h.Error}
}
return Result{}, errors.New(h.Error)
}
return Result{JPEG: h.JPEG, Engine: h.Engine, Codec: h.Codec, Width: h.Width, Height: h.Height}, nil
}
// Helper implements `hamlab thumb …`, the root side of the above. `--probe` prints Caps;
// `[--size N] [--timeout D] <path>` reads Caps from stdin and prints the preview as JSON.
// Returns the exit code.
func Helper(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
ctx := context.Background()
o := Opts{}
path := ""
for i := 0; i < len(args); i++ {
switch a := args[i]; {
case a == "--probe":
return emit(stdout, Probe(ctx), stderr)
case a == "--size" && i+1 < len(args):
o.Size, _ = strconv.Atoi(args[i+1])
i++
case a == "--timeout" && i+1 < len(args):
o.Timeout, _ = time.ParseDuration(args[i+1])
i++
case strings.HasPrefix(a, "-"):
return fail(errors.New("usage: hamlab thumb --probe | [--size N] [--timeout D] <path>"))
default:
path = a
}
}
if path == "" || !filepath.IsAbs(path) {
return fail(errors.New("usage: hamlab thumb --probe | [--size N] [--timeout D] <absolute path>"))
}
var c Caps
if err := json.NewDecoder(io.LimitReader(stdin, 1<<20)).Decode(&c); err != nil {
return fail(fmt.Errorf("caps on stdin: %w", err))
}
res, err := Make(ctx, c, path, o)
if err != nil {
var de *DecodeError
return emit(stdout, helperOut{Error: err.Error(), Permanent: errors.As(err, &de)}, stderr)
}
return emit(stdout, helperOut{Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}, stderr)
}
func emit(w io.Writer, v any, stderr io.Writer) int {
if err := json.NewEncoder(w).Encode(v); err != nil {
fmt.Fprintln(stderr, err.Error())
return 1
}
return 0
}
// ---- small helpers ----
func lastLine(s string) string {
lines := strings.Split(strings.TrimSpace(s), "\n")
return strings.TrimSpace(lines[len(lines)-1])
}
func exists(p string) bool { _, err := os.Stat(p); return err == nil }
+253
View File
@@ -0,0 +1,253 @@
package media
import (
"bytes"
"context"
"image"
"image/color"
"image/jpeg"
"image/png"
"os"
"os/exec"
"path/filepath"
"testing"
"time"
)
func TestKind(t *testing.T) {
for name, want := range map[string]string{"a.JPG": "image", "b.heic": "image", "c.mkv": "video", "GH010098.MP4": "video", "x.OSV": "video", "d.txt": "", "e": "", ".jpg": "image"} {
if got := Kind(name); got != want {
t.Errorf("Kind(%q) = %q, want %q", name, got, want)
}
}
}
func TestKeyFollowsTheFile(t *testing.T) {
a := Key("/x/a.jpg", 10, 20)
if a != Key("/x/a.jpg", 10, 20) || len(a) != 64 {
t.Fatal("key is not stable")
}
for _, b := range []string{Key("/x/b.jpg", 10, 20), Key("/x/a.jpg", 11, 20), Key("/x/a.jpg", 10, 21)} {
if b == a {
t.Fatal("key ignores part of the file's identity")
}
}
}
func TestParseVainfo(t *testing.T) {
text := `vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 25.2.3 ()
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileH264Main : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSliceLP
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileHEVCMain444_12 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileAV1Profile0 : VAEntrypointEncSliceLP
`
dec, enc := parseVainfo(text)
same := func(got, want []string) bool {
if len(got) != len(want) {
return false
}
for i := range want {
if got[i] != want[i] {
return false
}
}
return true
}
if !same(dec, []string{"h264", "hevc", "mjpeg", "mpeg2video", "vp9"}) {
t.Fatalf("decodes = %v", dec)
}
if !same(enc, []string{"av1", "h264", "hevc"}) {
t.Fatalf("encodes = %v", enc)
}
if d, e := parseVainfo(""); d != nil || e != nil {
t.Fatal("nothing from nothing")
}
}
func TestPickAndEngines(t *testing.T) {
igpu := GPU{API: "vaapi", Device: "/dev/dri/renderD128", Codecs: []string{"h264", "hevc"}}
arc := GPU{API: "vaapi", Device: "/dev/dri/renderD129", Codecs: []string{"av1", "h264", "hevc"}}
c := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", GPUs: []GPU{igpu, arc}}
if g := c.pick("av1"); g == nil || g.Device != arc.Device {
t.Fatal("av1 should go to the card that lists it")
}
if g := c.pick("h264"); g == nil || g.Device != igpu.Device {
t.Fatal("h264 should go to the first card that lists it")
}
if g := c.pick("vc1"); g != nil {
t.Fatal("a codec nobody lists is software")
}
// a card nothing could describe is tried, after the ones that answered
c.GPUs = append(c.GPUs, GPU{API: "cuda", Device: "0"})
if g := c.pick("vc1"); g == nil || g.API != "cuda" {
t.Fatal("an undescribed card is the guess of last resort")
}
// videos: GPU then CPU; images: CPU only
if e := engines(c, "video", fileInfo{codec: "hevc"}); len(e) != 2 || e[0] == nil || e[1] != nil {
t.Fatal("a video tries the GPU, then the CPU")
}
if e := engines(c, "image", fileInfo{codec: "mjpeg"}); len(e) != 1 || e[0] != nil {
t.Fatal("an image is CPU only")
}
}
func TestAttempts(t *testing.T) {
a := attempts(nil, "video", fileInfo{duration: 120}, 256)
if len(a) != 2 || a[0].seek != 12 || a[1].seek != 0 {
t.Fatalf("video attempts = %+v", a)
}
if a := attempts(nil, "video", fileInfo{duration: 7200}, 256); a[0].seek != 30 {
t.Fatalf("seek capped at 30 s, got %v", a[0].seek)
}
if a := attempts(nil, "video", fileInfo{duration: 2}, 256); len(a) != 1 || a[0].seek != 0 {
t.Fatalf("a short clip is taken from its start, got %+v", a)
}
if a := attempts(nil, "image", fileInfo{codec: "mjpeg", w: 6000, h: 4000}, 256); len(a) != 2 || a[0].lowres != 2 || a[1].lowres != 0 {
t.Fatalf("jpeg attempts = %+v", a)
}
if a := attempts(nil, "image", fileInfo{codec: "mjpeg", w: 800, h: 600}, 256); len(a) != 1 || a[0].lowres != 0 {
t.Fatalf("a small jpeg decodes whole, got %+v", a)
}
if a := attempts(nil, "image", fileInfo{codec: "png", w: 6000, h: 4000}, 256); len(a) != 1 || a[0].lowres != 0 {
t.Fatalf("only jpeg has lowres, got %+v", a)
}
}
func TestCache(t *testing.T) {
c := NewCache(filepath.Join(t.TempDir(), "thumbs.db"))
k := Key("/a", 1, 2)
if _, found, err := c.Get(k); found || err != nil {
t.Fatal("an empty cache found something, or complained")
}
if _, err := os.Stat(c.Path); err == nil {
t.Fatal("a miss must not create the database")
}
if n, b := c.Stats(); n != 0 || b != 0 {
t.Fatal("stats of nothing")
}
if err := c.Put(k, Entry{Path: "/a", Engine: "cpu", Codec: "png", Width: 640, Height: 360, JPEG: []byte("jpeg")}); err != nil {
t.Fatal(err)
}
e, found, err := c.Get(k)
if err != nil || !found || string(e.JPEG) != "jpeg" || e.Engine != "cpu" || e.Width != 640 || e.Path != "/a" || e.Made.IsZero() {
t.Fatalf("put then get: %+v %v %v", e, found, err)
}
k2 := Key("/b", 1, 2)
if err := c.Put(k2, Entry{Path: "/b", Failure: "no video stream"}); err != nil {
t.Fatal(err)
}
if e, found, _ := c.Get(k2); !found || e.JPEG != nil || e.Failure != "no video stream" {
t.Fatal("a failure is remembered with its reason")
}
if err := c.Put(k, Entry{}); err == nil {
t.Fatal("an empty preview must not be kept")
}
// replacing an entry does not count it twice
if err := c.Put(k, Entry{Path: "/a", JPEG: []byte("jpeg2")}); err != nil {
t.Fatal(err)
}
n, b := c.Stats()
if n != 2 || b == 0 {
t.Fatalf("stats = %d entries, %d bytes", n, b)
}
// survives a reopen
if err := c.Close(); err != nil {
t.Fatal(err)
}
c2 := NewCache(c.Path)
if e, found, _ := c2.Get(k); !found || string(e.JPEG) != "jpeg2" {
t.Fatal("not there after reopening")
}
if n, _ := c2.Stats(); n != 2 {
t.Fatal("count lost on reopen")
}
c2.Close()
}
// The real thing, when ffmpeg is here: a PNG and a generated video each come back as a
// JPEG no larger than the asked size, with the aspect kept.
func TestMakeWithFFmpeg(t *testing.T) {
if _, err := exec.LookPath("ffmpeg"); err != nil {
t.Skip("no ffmpeg")
}
c := Probe(context.Background())
if c.Error != "" {
t.Skip(c.Error)
}
dir := t.TempDir()
img := image.NewRGBA(image.Rect(0, 0, 640, 360))
for y := 0; y < 360; y++ {
for x := 0; x < 640; x++ {
img.Set(x, y, color.RGBA{uint8(x / 3), uint8(y), 90, 255})
}
}
pngPath := filepath.Join(dir, "a.png")
f, _ := os.Create(pngPath)
_ = png.Encode(f, img)
f.Close()
res, err := Make(context.Background(), c, pngPath, Opts{Size: 200})
if err != nil {
t.Fatal(err)
}
check(t, res, 200, 112, "cpu")
// a two-second test pattern, with the always-built-in mpeg4 encoder
vid := filepath.Join(dir, "v.mp4")
out, err := exec.Command("ffmpeg", "-nostdin", "-v", "error", "-f", "lavfi", "-i", "testsrc=size=320x180:rate=10:duration=2", "-c:v", "mpeg4", "-q:v", "5", vid).CombinedOutput()
if err != nil {
t.Skipf("could not make a test video: %s", out)
}
res, err = Make(context.Background(), c, vid, Opts{Size: 100})
if err != nil {
t.Fatal(err)
}
check(t, res, 100, 56, "")
if res.Codec != "mpeg4" || res.Width != 320 {
t.Fatalf("probe said %s %dx%d", res.Codec, res.Width, res.Height)
}
// not media, and media that isn't
if _, err := Make(context.Background(), c, filepath.Join(dir, "x.txt"), Opts{}); err != ErrUnsupported {
t.Fatalf("a .txt is unsupported, got %v", err)
}
bad := filepath.Join(dir, "bad.mp4")
_ = os.WriteFile(bad, []byte("not a video at all"), 0o644)
_, err = Make(context.Background(), c, bad, Opts{})
var de *DecodeError
if err == nil || !isDecodeErr(err, &de) {
t.Fatalf("garbage should be a DecodeError, got %v", err)
}
// the clock: a cancelled context is not the file's fault
ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
defer cancel()
if _, err := Make(ctx, c, pngPath, Opts{}); err == nil || isDecodeErr(err, &de) {
t.Fatalf("a timeout must not read as a decode failure, got %v", err)
}
}
func check(t *testing.T, res Result, w, h int, engine string) {
t.Helper()
im, err := jpeg.Decode(bytes.NewReader(res.JPEG))
if err != nil {
t.Fatalf("not a jpeg: %v", err)
}
if b := im.Bounds(); b.Dx() != w || b.Dy() < h-1 || b.Dy() > h+1 { // the short side rounds either way
t.Fatalf("got %dx%d, want %dx%d", b.Dx(), b.Dy(), w, h)
}
if engine != "" && res.Engine != engine {
t.Fatalf("engine %q, want %q", res.Engine, engine)
}
}
func isDecodeErr(err error, de **DecodeError) bool {
x, ok := err.(*DecodeError)
if ok {
*de = x
}
return ok
}
+259
View File
@@ -0,0 +1,259 @@
package metrics
// CPU power, from the RAPL energy counters the kernel keeps under /sys/class/powercap: Intel
// since Sandy Bridge, and AMD Zen through the same driver. Each counter is a running total in
// µJ, so power is the difference between two readings over the time between them. Every
// package is summed (`package-0`, and `package-1` on a two-socket board). The other top-level
// domains are left out. `psys` claims the whole platform but is only as good as the firmware's
// calibration, and on one machine here it reads lower than the package inside it. The MMIO
// copy of a package (`intel-rapl-mmio:0`) measures the same thing again. This is the CPU's own
// figure, shown beside its clock: cores, built-in graphics and memory controller, not disks,
// fans or the power supply.
//
// The counters are root-only since the PLATYPUS side channel (2020). So, like the GPU client
// walk, they are read in-process when the daemon is root, and otherwise by a root helper,
// `hamlab cpu-energy`, started once through sudo or doas and asked over a pipe at every
// sample: one privilege escalation per daemon, not one a second in the auth log. The helper
// takes nothing from the daemon but the newline that asks; it finds and reads the counters
// itself. Without root there is no reading, and nothing is shown.
import (
"bufio"
"encoding/json"
"io"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
"hamlab/internal/priv"
)
const raplRoot = "/sys/class/powercap"
var raplPkgRe = regexp.MustCompile(`^package-\d+$`)
// raplCount is one package's counter at one moment.
type raplCount struct {
Dir string `json:"d"`
UJ uint64 `json:"uj"`
}
type cpuPowerState struct {
probed bool
dirs []string // the package domains, found at the first sample
max map[string]uint64 // each counter's range: where it wraps to zero
useHelper bool // the counters are not readable in-process
helper *energyHelper
prev []raplCount
prevAt time.Time
}
// raplPackages lists the package domains under root, sorted. intel-rapl:0 is a package;
// intel-rapl:0:0 is a part of one (core, uncore, dram).
func raplPackages(root string) []string {
ents, err := os.ReadDir(root)
if err != nil {
return nil
}
var out []string
for _, e := range ents {
n := e.Name()
if !strings.HasPrefix(n, "intel-rapl:") || strings.Count(n, ":") != 1 {
continue
}
if raplPkgRe.MatchString(strings.TrimSpace(readStr(filepath.Join(root, n, "name")))) {
out = append(out, n)
}
}
sort.Strings(out)
return out
}
func readRAPL(root string, dirs []string) ([]raplCount, bool) {
out := make([]raplCount, 0, len(dirs))
for _, d := range dirs {
b, err := os.ReadFile(filepath.Join(root, d, "energy_uj"))
if err != nil {
return nil, false
}
v, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64)
if err != nil {
return nil, false
}
out = append(out, raplCount{d, v})
}
return out, len(out) > 0
}
// raplWatts is the power between two readings of the same counters. A counter that wrapped
// is carried across its range. One that went backwards with no known range, or a set of
// counters that changed between readings, gives no answer rather than a wrong one.
func raplWatts(prev, cur []raplCount, max map[string]uint64, dt float64) (float64, bool) {
if dt <= 0 || len(prev) == 0 || len(prev) != len(cur) {
return 0, false
}
var uj float64
for i := range cur {
if cur[i].Dir != prev[i].Dir {
return 0, false
}
if cur[i].UJ >= prev[i].UJ {
uj += float64(cur[i].UJ - prev[i].UJ)
continue
}
m := max[cur[i].Dir]
if m == 0 || prev[i].UJ > m {
return 0, false
}
uj += float64(m - prev[i].UJ + cur[i].UJ)
}
return uj / 1e6 / dt, true
}
// cpuPower is the CPU packages' power since the previous sample, in watts; 0 when this
// machine has no counters or no way to read them.
func (c *Collector) cpuPower(now time.Time) float64 {
p := &c.power
if !p.probed {
p.probed = true
p.dirs = raplPackages(raplRoot)
p.max = map[string]uint64{}
for _, d := range p.dirs {
if v, ok := readInt(filepath.Join(raplRoot, d, "max_energy_range_uj")); ok && v > 0 {
p.max[d] = uint64(v)
}
}
}
if len(p.dirs) == 0 {
return 0
}
var cur []raplCount
ok := false
if !p.useHelper {
if cur, ok = readRAPL(raplRoot, p.dirs); !ok && priv.Mode() != "root" && priv.Available() {
p.useHelper = true
}
}
if p.useHelper {
if p.helper == nil {
p.helper = &energyHelper{}
}
cur, ok = p.helper.read()
}
if !ok {
p.prev = nil
return 0
}
w, good := raplWatts(p.prev, cur, p.max, now.Sub(p.prevAt).Seconds())
p.prev, p.prevAt = cur, now
if !good {
return 0
}
return round1(w)
}
// energyHelper is the root child that answers each newline with the package counters, one
// JSON line. The same shape as the GPU client walk's helper, kept separate from it.
type energyHelper struct {
mu sync.Mutex
cmd *exec.Cmd
in io.WriteCloser
out *bufio.Reader
failedAt time.Time
}
func (h *energyHelper) read() ([]raplCount, bool) {
h.mu.Lock()
defer h.mu.Unlock()
if h.cmd == nil {
if !h.failedAt.IsZero() && time.Since(h.failedAt) < helperRetry {
return nil, false
}
if !h.start() {
h.failedAt = time.Now()
return nil, false
}
}
// a stuck child must not stall the sampler: it is killed, and tried again later
cmd := h.cmd
stop := time.AfterFunc(5*time.Second, func() { cmd.Process.Kill() })
defer stop.Stop()
if _, err := io.WriteString(h.in, "\n"); err != nil {
h.fail()
return nil, false
}
line, err := h.out.ReadBytes('\n')
if err != nil {
h.fail()
return nil, false
}
var cs []raplCount
if json.Unmarshal(line, &cs) != nil {
h.fail()
return nil, false
}
return cs, len(cs) > 0
}
func (h *energyHelper) start() bool {
self, err := os.Executable()
if err != nil {
return false
}
full := priv.Wrap(self, "cpu-energy")
cmd := exec.Command(full[0], full[1:]...)
in, err := cmd.StdinPipe()
if err != nil {
return false
}
out, err := cmd.StdoutPipe()
if err != nil {
return false
}
if err := cmd.Start(); err != nil {
return false
}
h.cmd, h.in, h.out = cmd, in, bufio.NewReader(out)
return true
}
func (h *energyHelper) fail() {
if h.cmd != nil {
h.in.Close()
h.cmd.Process.Kill()
go h.cmd.Wait()
}
h.cmd, h.in, h.out = nil, nil, nil
h.failedAt = time.Now()
}
// CPUEnergyHelper is `hamlab cpu-energy`: for every line read on stdin, one JSON line of the
// package counters on stdout. Ends when stdin does — when the daemon that started it has gone.
func CPUEnergyHelper(stdin io.Reader, stdout io.Writer) int {
return cpuEnergyHelper(raplRoot, stdin, stdout)
}
func cpuEnergyHelper(root string, stdin io.Reader, stdout io.Writer) int {
dirs := raplPackages(root)
sc := bufio.NewScanner(stdin)
w := bufio.NewWriter(stdout)
for sc.Scan() {
cs, ok := readRAPL(root, dirs)
if !ok {
cs = []raplCount{}
}
b, _ := json.Marshal(cs)
w.Write(b)
w.WriteByte('\n')
if w.Flush() != nil {
return 1
}
}
return 0
}
+106
View File
@@ -0,0 +1,106 @@
package metrics
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// A fake /sys/class/powercap: a package with its parts, the platform domain, and the MMIO
// copy of the same package. Only the package counts.
func fakeRAPL(t *testing.T, domains map[string][2]string) string {
t.Helper()
root := t.TempDir()
for dir, v := range domains {
d := filepath.Join(root, dir)
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
os.WriteFile(filepath.Join(d, "name"), []byte(v[0]+"\n"), 0o644)
os.WriteFile(filepath.Join(d, "energy_uj"), []byte(v[1]+"\n"), 0o644)
}
return root
}
func TestRAPLPackagesOnlyCountsPackages(t *testing.T) {
root := fakeRAPL(t, map[string][2]string{
"intel-rapl:0": {"package-0", "1000"},
"intel-rapl:0:0": {"core", "600"},
"intel-rapl:0:1": {"uncore", "10"},
"intel-rapl:1": {"psys", "300"},
"intel-rapl-mmio:0": {"package-0", "1000"},
})
if got := raplPackages(root); len(got) != 1 || got[0] != "intel-rapl:0" {
t.Fatalf("packages = %v, want [intel-rapl:0]", got)
}
two := fakeRAPL(t, map[string][2]string{"intel-rapl:0": {"package-0", "5"}, "intel-rapl:1": {"package-1", "7"}})
got := raplPackages(two)
if strings.Join(got, ",") != "intel-rapl:0,intel-rapl:1" {
t.Fatalf("two sockets: packages = %v", got)
}
cs, ok := readRAPL(two, got)
if !ok || len(cs) != 2 || cs[0].UJ != 5 || cs[1].UJ != 7 {
t.Fatalf("readRAPL = %+v, %v", cs, ok)
}
if got := raplPackages(filepath.Join(two, "nope")); len(got) != 0 {
t.Fatalf("no powercap: packages = %v, want none", got)
}
}
func TestRAPLWatts(t *testing.T) {
c := func(d string, uj uint64) raplCount { return raplCount{d, uj} }
max := map[string]uint64{"intel-rapl:0": 1_000_000_000}
cases := []struct {
name string
prev, cur []raplCount
dt float64
want float64
ok bool
}{
{"steady", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:0", 10_000_000)}, 2, 5, true},
{"two packages add", []raplCount{c("intel-rapl:0", 0), c("intel-rapl:1", 0)}, []raplCount{c("intel-rapl:0", 4_000_000), c("intel-rapl:1", 6_000_000)}, 1, 10, true},
{"wrapped", []raplCount{c("intel-rapl:0", 999_000_000)}, []raplCount{c("intel-rapl:0", 1_000_000)}, 1, 2, true},
{"backwards, range unknown", []raplCount{c("intel-rapl:1", 5_000_000)}, []raplCount{c("intel-rapl:1", 1)}, 1, 0, false},
{"counters changed", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:1", 9)}, 1, 0, false},
{"first reading", nil, []raplCount{c("intel-rapl:0", 9)}, 1, 0, false},
{"no time passed", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:0", 9)}, 0, 0, false},
}
for _, tc := range cases {
got, ok := raplWatts(tc.prev, tc.cur, max, tc.dt)
if ok != tc.ok || (ok && (got < tc.want-1e-9 || got > tc.want+1e-9)) {
t.Errorf("%s: got %v, %v; want %v, %v", tc.name, got, ok, tc.want, tc.ok)
}
}
}
// The helper answers every newline with the counters it found itself, and stops at EOF.
func TestCPUEnergyHelper(t *testing.T) {
root := fakeRAPL(t, map[string][2]string{"intel-rapl:0": {"package-0", "42"}, "intel-rapl:0:0": {"core", "7"}})
var out bytes.Buffer
if code := cpuEnergyHelper(root, strings.NewReader("\n\n"), &out); code != 0 {
t.Fatalf("exit %d", code)
}
lines := strings.Split(strings.TrimSpace(out.String()), "\n")
if len(lines) != 2 {
t.Fatalf("got %d lines, want 2: %q", len(lines), out.String())
}
var cs []raplCount
if err := json.Unmarshal([]byte(lines[1]), &cs); err != nil || len(cs) != 1 || cs[0] != (raplCount{"intel-rapl:0", 42}) {
t.Fatalf("line = %s (%v)", lines[1], err)
}
}
// History keeps the average of the samples that had a reading.
func TestFoldCPUPower(t *testing.T) {
b := []Sample{{T: 1, CPUPower: 4}, {T: 2}, {T: 3, CPUPower: 8}}
if got := fold(b).CPUPower; got != 6 {
t.Fatalf("folded CPU power = %v, want 6", got)
}
if got := fold([]Sample{{T: 1}, {T: 2}}).CPUPower; got != 0 {
t.Fatalf("no readings folded to %v, want 0", got)
}
}
+11
View File
@@ -32,6 +32,17 @@ func fold(b []Sample) Sample {
return t / uint64(len(b))
}
out.CPU = sum(func(s Sample) float64 { return s.CPU })
// CPU power: over the samples that have a reading, so a helper restart is not a dip to zero
var pw, pn float64
for _, s := range b {
if s.CPUPower > 0 {
pw, pn = pw+s.CPUPower, pn+1
}
}
out.CPUPower = 0
if pn > 0 {
out.CPUPower = round1(pw / pn)
}
out.Load[0] = sum(func(s Sample) float64 { return s.Load[0] })
out.Load[1] = sum(func(s Sample) float64 { return s.Load[1] })
out.Load[2] = sum(func(s Sample) float64 { return s.Load[2] })
+25
View File
@@ -250,6 +250,31 @@ func listGPUs(known []*gpuDev) []*gpuDev {
return out
}
// RenderNodeName is the GPU behind a render node ("/dev/dri/renderD128" → "Intel Arc A380"),
// named the same way the Processes tab names it; "" when sysfs has nothing for it.
func RenderNodeName(node string) string {
dev, err := filepath.EvalSymlinks("/sys/class/drm/" + filepath.Base(node) + "/device")
if err != nil {
return ""
}
driver := ueventValue(dev+"/uevent", "DRIVER")
if driver == "" {
driver = filepath.Base(readLink(dev + "/driver"))
}
return gpuName(dev, ueventValue(dev+"/uevent", "PCI_SLOT_NAME"), driver)
}
// RenderNodeIntegrated reports a render node whose GPU sits on the CPU die (PCI root bus 0,
// or no PCI bus at all), as opposed to a card on the bus.
func RenderNodeIntegrated(node string) bool {
dev, err := filepath.EvalSymlinks("/sys/class/drm/" + filepath.Base(node) + "/device")
if err != nil {
return true
}
pci := ueventValue(dev+"/uevent", "PCI_SLOT_NAME")
return pci == "" || strings.HasPrefix(pci, "0000:00:")
}
var gpuVendors = map[string]string{"0x8086": "Intel", "0x1002": "AMD", "0x10de": "NVIDIA", "0x1af4": "Virtio", "0x106b": "Apple"}
var (
+8 -2
View File
@@ -6,7 +6,9 @@
// lm-sensors, smartmontools or root. Sections whose source is absent (no fans, no
// UPS daemon, no hwmon at all on a VM) simply come back empty rather than failing.
// The only external commands are optional UPS daemons: pwrstat (CyberPower), upsc
// (NUT) and apcaccess (apcupsd), tried in that order.
// (NUT) and apcaccess (apcupsd), tried in that order. Two readings are root-only, the GPU
// client walk and the CPU energy counters; without root they come from small helpers of
// this binary started once through sudo or doas (gpu.go, cpupower.go), or not at all.
package metrics
import (
@@ -137,6 +139,7 @@ type Sample struct {
Interval int64 `json:"interval_ms"` // sampling interval in effect
CPU float64 `json:"cpu"`
Cores []float64 `json:"cores"`
CPUPower float64 `json:"cpu_power_w,omitempty"` // the CPU packages' own energy counters; 0 = no reading
Load [3]float64 `json:"load"`
Mem Mem `json:"mem"`
Swap Mem `json:"swap"`
@@ -177,6 +180,7 @@ type Collector struct {
lastTable *ProcTable
upsProbe func() *UPS // nil when no UPS daemon is installed
gpu gpuState
power cpuPowerState
prevARC arcPrev
subMu sync.Mutex
@@ -343,6 +347,7 @@ func (c *Collector) sample() {
s := Sample{T: now.UnixMilli(), Interval: c.Interval().Milliseconds(), Temps: []Temp{}, Fans: []Fan{}, Net: []Net{}, Disks: []Disk{}}
s.CPU, s.Cores = c.cpu()
s.CPUPower = c.cpuPower(now)
s.Load = loadavg()
s.Mem, s.Swap = meminfo()
s.Uptime = uptime()
@@ -1113,6 +1118,7 @@ type ProcTable struct {
CPUModel string `json:"cpu_model,omitempty"`
CPUMHz float64 `json:"cpu_mhz,omitempty"`
CPUMaxMHz float64 `json:"cpu_max_mhz,omitempty"`
CPUPowerW float64 `json:"cpu_power_w,omitempty"`
CPUTemp float64 `json:"cpu_temp,omitempty"`
CoreIDs []int `json:"core_ids,omitempty"` // for each logical CPU, the physical core it is a thread of (what the temperature sensors are named by)
Load [3]float64 `json:"load"`
@@ -1312,7 +1318,7 @@ func (c *Collector) Processes() ProcTable {
t.Mem, t.Swap = meminfo()
if n := len(c.ring); n > 0 {
last := c.ring[n-1]
t.CPU, t.Cores, t.Net, t.CPUTemp = last.CPU, last.Cores, last.Net, cpuTemp(last.Temps)
t.CPU, t.Cores, t.Net, t.CPUTemp, t.CPUPowerW = last.CPU, last.Cores, last.Net, cpuTemp(last.Temps), last.CPUPower
t.CoreIDs = coreIDs(len(last.Cores))
from := n - 90
if from < 0 {
+3
View File
@@ -62,6 +62,9 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
if c := cpuTemp(s.Temps); c > 0 {
f.add("cpu.temp", "CPU temperature", "temperature", "°C", c, t, none)
}
if s.CPUPower > 0 {
f.add("cpu.power", "CPU power", "power", "W", s.CPUPower, t, none)
}
for i, w := range []string{"1", "5", "15"} {
f.add("load."+w, "Load "+w+" min", "load", "", s.Load[i], t, none)
}
-1
View File
@@ -66,7 +66,6 @@ type Policy struct {
KeepMin int `json:"keep_min"`
Prefix string `json:"prefix"`
Enabled bool `json:"enabled"`
}
func (p *Policy) prefix() string { return strings.TrimSpace(p.Prefix) }
+9 -2
View File
@@ -146,7 +146,7 @@ func (m *Manager) prune() {
}
var finished []*Task
for _, t := range m.tasks {
if !t.Running {
if !t.running() {
finished = append(finished, t)
}
}
@@ -184,13 +184,20 @@ func (m *Manager) RunningOf(kind string) (*Task, bool) {
m.mu.Lock()
defer m.mu.Unlock()
for _, t := range m.tasks {
if t.Kind == kind && t.Running {
if t.Kind == kind && t.running() {
return t, true
}
}
return nil, false
}
// running reads the flag the waiter goroutine clears, under the task's own lock.
func (t *Task) running() bool {
t.mu.Lock()
defer t.mu.Unlock()
return t.Running
}
func (t *Task) Snapshot(withOutput bool) Snapshot {
t.mu.Lock()
defer t.mu.Unlock()
+3
View File
@@ -64,6 +64,9 @@ func New(sudo bool, tm *tasks.Manager) *Manager {
return m
}
// PM is the package manager found: apt | dnf | zypper | pacman | apk, or "".
func (m *Manager) PM() string { return m.pm }
func (m *Manager) sudoArgs(args ...string) []string {
if m.sudo {
return priv.Wrap(args...)
+7
View File
@@ -11,6 +11,7 @@
"@xterm/addon-fit": "^0.10.0",
"@xterm/addon-web-links": "^0.11.0",
"@xterm/xterm": "^5.5.0",
"hls.js": "^1.7.3",
"qrcode-generator": "^2.0.4",
"react": "^18.3.1",
"react-dom": "^18.3.1"
@@ -1426,6 +1427,12 @@
"node": ">=6.9.0"
}
},
"node_modules/hls.js": {
"version": "1.7.3",
"resolved": "https://registry.npmjs.org/hls.js/-/hls.js-1.7.3.tgz",
"integrity": "sha512-MsPlx6yVW4Qv4C7mEVou4/gk/5cN2dVxOTnrNjPSmyH2f0Ln+/UkG10Ax1PE6OvrmTVPKgQYozTtBaPOuCzq7A==",
"license": "Apache-2.0"
},
"node_modules/js-tokens": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz",
+2 -1
View File
@@ -7,12 +7,13 @@
"dev": "vite",
"build": "tsc --noEmit && vite build",
"icons": "node scripts/icons.mjs",
"test": "node scripts/test-graphcols.mjs"
"test": "node scripts/test-graphcols.mjs && node scripts/test-css-imports.mjs && node scripts/test-resolve-path.mjs"
},
"dependencies": {
"@xterm/addon-fit": "^0.10.0",
"@xterm/addon-web-links": "^0.11.0",
"@xterm/xterm": "^5.5.0",
"hls.js": "^1.7.3",
"qrcode-generator": "^2.0.4",
"react": "^18.3.1",
"react-dom": "^18.3.1"
+20
View File
@@ -0,0 +1,20 @@
// node scripts/test-css-imports.mjs — every stylesheet the UI imports is one somebody has read.
// A third-party stylesheet usually carries its demo page's global rules (body, *, button); on
// 2026-10-09 one restyled every button in the app. Add a file here only once its rules are
// known to touch nothing but its own component.
import { readdirSync, readFileSync, statSync } from 'node:fs'
import { join, relative } from 'node:path'
const allowed = new Set(['./styles.css', '@xterm/xterm/css/xterm.css'])
const src = new URL('../src', import.meta.url).pathname
const files = []
const walk = (d) => { for (const n of readdirSync(d)) { const p = join(d, n); if (statSync(p).isDirectory()) { if (n !== 'vendor') walk(p) } else if (/\.(ts|tsx)$/.test(n)) files.push(p) } }
walk(src)
const fails = []
for (const f of files) {
for (const m of readFileSync(f, 'utf8').matchAll(/import\s+(?:[^'"]*from\s+)?['"]([^'"]+\.css)['"]/g)) {
if (!allowed.has(m[1])) fails.push(`${relative(src, f)} imports ${m[1]}`)
}
}
if (fails.length) { console.error('css imports: not allowed:\n ' + fails.join('\n ')); process.exit(1) }
console.log(`css imports: ok (${files.length} files)`)
+16
View File
@@ -0,0 +1,16 @@
// node scripts/test-resolve-path.mjs — the path box resolves like a shell: relative to here,
// . and .. applied, trailing slashes dropped, never anything but an absolute path.
import { readFileSync } from 'node:fs'
const src = readFileSync(new URL('../src/Files.tsx', import.meta.url), 'utf8')
const m = src.match(/export function resolvePath[\s\S]*?\n}\n/)
if (!m) { console.error('resolvePath not found'); process.exit(1) }
const resolvePath = new Function(m[0].replace('export function', 'return function').replace(/: string\[\]/g, '').replace(/: string/g, '') + ' ')()
const cases = [
['/tank/media', '/x', '/tank/media'], ['/tank/media/', '/x', '/tank/media'], ['', '/here', '/here'], [' ', '/here', '/here'],
['sub', '/here', '/here/sub'], ['./sub/', '/here', '/here/sub'], ['..', '/a/b', '/a'], ['../..', '/a/b', '/'], ['../../..', '/a', '/'],
['/', '/a', '/'], ['a//b', '/', '/a/b'], ['/tank/./x/../y', '/', '/tank/y'], ['sp ace #1', '/d', '/d/sp ace #1'],
]
let bad = 0
for (const [input, here, want] of cases) { const got = resolvePath(input, here); if (got !== want) { bad++; console.error(`resolvePath(${JSON.stringify(input)}, ${here}) = ${got}, want ${want}`) } }
if (bad) process.exit(1)
console.log(`resolve path: ok (${cases.length} cases)`)
+2 -2
View File
@@ -355,7 +355,7 @@ function StackDetail({ name, startTask, onBack, onChanged, onDuplicate }: { name
<div className="svc-detail">
<CloseBtn onClick={onBack} />
<div className="svc-head"><button className="btn back" onClick={onBack}>‹ Back</button><span className={'sdot ' + (st.containers.length === 0 ? '' : st.running === st.containers.length ? 'ok' : st.running ? 'warn' : 'bad')} /><span className="dname">{st.name}</span><span className="muted small">{st.running}/{st.containers.length} running</span></div>
<div className="muted small mono" style={{ overflowWrap: 'anywhere' }}>{file === 'env' ? (st.host_file.replace(/[^/]+$/, '') + '.env') : (st.host_files?.[file] ?? st.host_file)}{st.held_by && <span className="muted"> · kept on <b>{st.held_by}</b>, deployed to this machine over {st.remote}</span>}{st.via && <span className="muted"> · compose runs inside <b>{st.via}</b>, as it was deployed</span>}{st.has_env && ' · has a .env beside it'}</div>
<div className="muted small mono" style={{ overflowWrap: 'anywhere' }}>{file === 'env' ? (st.host_file.replace(/[^/]+$/, '') + '.env') : (st.host_files?.[file] ?? st.host_file)}{st.via && <span className="muted"> · compose runs inside <b>{st.via}</b>, as it was deployed</span>}{st.has_env && ' · has a .env beside it'}</div>
{((st.host_files?.length ?? 0) > 1 || st.has_env) && <div className="chips" title="this stack is deployed from several files, in this order; Deploy uses all of them">{st.host_files!.map((f, i) => <button key={f} className={'chip' + (i === file ? ' on' : '')} onClick={async () => { if (dirty && !(await askYes('Discard unsaved changes to this file?', 'Discard'))) return; setFile(i) }}>{f.split('/').pop()}</button>)}{st.has_env && <button className={'chip' + (file === 'env' ? ' on' : '')} title="the .env beside the compose file: variables compose substitutes" onClick={async () => { if (dirty && !(await askYes('Discard unsaved changes to this file?', 'Discard'))) return; setFile('env') }}>.env</button>}</div>}
{file === 'env' && !reveal ? <div className="env-masked"><pre className="editor stack-editor">{maskEnv(text)}</pre><button className="btn" onClick={() => setReveal(true)}>Reveal and edit</button></div> :
<CodeEditor className="stack-editor" language={file === 'env' ? 'env' : 'yaml'} value={text} onChange={setText} />}
@@ -447,7 +447,7 @@ export default function Docker() {
{shownTab === 'stacks' && (stacks ?? []).map((st) => (
<div key={st.name} className={'svc-row' + (st.name === selStack ? ' active' : '')} onClick={() => st.editable ? setSel('stack:' + st.name) : toast(st.note ?? 'not editable here', 'info')}>
<span className={'sdot ' + (st.containers.length === 0 ? '' : st.running === st.containers.length ? 'ok' : st.running ? 'warn' : 'bad')} />
<div className="svc-name"><div className="n">{st.name}</div><div className="d">{st.editable ? (st.held_by ? `on ${st.held_by}: ` : '') + st.host_file : st.note}</div></div>
<div className="svc-name"><div className="n">{st.name}</div><div className="d">{st.editable ? st.host_file : st.note}</div></div>
<div className="svc-state">{st.containers.length === 0 ? 'not deployed' : `${st.running}/${st.containers.length}`}{!st.editable && <><br />read-only</>}</div>
</div>
))}
+229 -15
View File
@@ -1,10 +1,25 @@
import { DragEvent, useCallback, useEffect, useRef, useState } from 'react'
import { api, FileEntry, FileInfo } from './api'
import { DragEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
import { api, FileEntry, FileInfo, TaskInfo, ThumbStatus } from './api'
import { fmtBytes } from './fmt'
import { Btn, Loading, Spinner, askText, askYes, setUnsaved, toast, useEscape } from './ui'
import { useHashSel } from './hashsel'
import TaskOutput from './TaskOutput'
import Player from './Player'
const join = (dir: string, name: string) => (dir === '/' ? '/' + name : dir + '/' + name)
// A typed path, made absolute: relative to the folder being looked at, "." and ".." applied,
// trailing slashes dropped — what a shell would make of it.
export function resolvePath(input: string, here: string): string {
const raw = input.trim()
if (!raw) return here
const parts = (raw.startsWith('/') ? [] : here.split('/')).concat(raw.split('/'))
const out: string[] = []
for (const p of parts) {
if (p === '' || p === '.') continue
if (p === '..') out.pop(); else out.push(p)
}
return '/' + out.join('/')
}
const fmtTime = (ms: number) => { const d = new Date(ms); const now = new Date(); return d.toDateString() === now.toDateString() ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) : d.toLocaleDateString(undefined, { year: d.getFullYear() === now.getFullYear() ? undefined : 'numeric', month: 'short', day: '2-digit' }) }
const isImage = (n: string) => /\.(png|jpe?g|gif|webp|svg|bmp|ico|avif)$/i.test(n)
const shellQuote = (s: string) => "'" + s.replace(/'/g, "'\\''") + "'"
@@ -14,7 +29,9 @@ function icon(e: FileEntry): string {
if (e.type === 'dir') return e.target ? '🔗' : '📁'
if (e.type === 'link') return e.broken ? '⛓️‍💥' : '🔗'
if (e.type === 'other') return '⚙️'
if (isImage(e.name)) return '🖼️'
if (e.preview === 'image' || isImage(e.name)) return '🖼️'
if (e.preview === 'video') return '🎞️'
if (e.preview === 'audio') return '🎵'
if (/\.(zip|gz|bz2|xz|zst|tar|tgz|7z|rar)$/i.test(e.name)) return '🗜️'
if (/\.(sh|bash|zsh|py|go|js|ts|tsx|rb|pl|php|rs|c|h|cpp|java|lua)$/i.test(e.name) || e.mode.includes('x')) return '📜'
if (/\.(log|txt|md|json|ya?ml|toml|ini|conf|cfg|env|service|timer)$/i.test(e.name)) return '📄'
@@ -22,6 +39,163 @@ function icon(e: FileEntry): string {
}
type Sort = 'name' | 'size' | 'mtime'
const PREVIEWS_KEY = 'hamlab.files.previews'
const byName = (a: FileEntry, b: FileEntry) => a.name.localeCompare(b.name, undefined, { numeric: true, sensitivity: 'base' })
// A name is one path segment. ("../x" and "a/b" would quietly act somewhere else.)
const okName = (n?: string | null): n is string => { const t = n?.trim(); if (!t) return false; if (t.includes('/') || t === '.' || t === '..') { toast('a name can\'t contain / or be . or .. — use Move… to put it elsewhere', 'err'); return false } return true }
// ---- move: walk to the destination, then "Move here" ----
function MovePicker({ item, from, sudo, hidden, onClose, onMoved }: { item: FileEntry; from: string; sudo: boolean; hidden: boolean; onClose: () => void; onMoved: (task?: TaskInfo) => void }) {
const src = join(from, item.name)
const isFolder = item.type === 'dir' && !item.target
const [hist, setHist] = useState<string[]>([from]) // where the picker has been, for back and forward
const [at, setAt] = useState(0)
const dir = hist[at]
const [entries, setEntries] = useState<FileEntry[] | null>(null)
const [err, setErr] = useState('')
const [pathInput, setPathInput] = useState(from)
const [busy, setBusy] = useState(false)
const load = useCallback(async (p: string) => {
setEntries(null); setErr('')
try { const l = await api.files(p, sudo); setEntries(l.entries) } catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setEntries([]) }
}, [sudo])
useEffect(() => { setPathInput(dir); load(dir) }, [dir, load])
useEscape(onClose)
const go = (input: string) => {
const p = resolvePath(input, dir)
if (p === dir) return
setHist((h) => [...h.slice(0, at + 1), p]); setAt(at + 1)
}
const up = () => { if (dir !== '/') go(dir.slice(0, dir.lastIndexOf('/')) || '/') }
const folders = (entries ?? []).filter((e) => e.type === 'dir' && (hidden || !e.name.startsWith('.'))).sort(byName)
const crumbs = dir === '/' ? [] : dir.slice(1).split('/')
const dest = join(dir, item.name)
const why = dir === from ? 'it is already here'
: isFolder && (dir === src || dir.startsWith(src + '/')) ? 'a folder can\'t move into itself'
: (entries ?? []).some((e) => e.name === item.name) ? `something called ${item.name} is already there` : ''
const move = async () => {
setBusy(true)
try {
const t = await api.fileMove(src, dest, sudo)
if (t) toast(`${dir} is on another filesystem — copying ${item.name} there, then removing it here`, 'info')
else toast(`moved ${item.name} to ${dir}`)
onMoved(t)
} catch (e) { toast(`move — ${e instanceof Error ? e.message : 'failed'}`, 'err'); setBusy(false) }
}
const mkdir = async () => {
const n = await askText('New folder name')
if (!okName(n)) return
try { await api.fileMkdir(join(dir, n.trim()), sudo); go(join(dir, n.trim())) } catch (e) { toast(`create folder — ${e instanceof Error ? e.message : 'failed'}`, 'err') }
}
return (
<div className="modal-back" onClick={onClose}>
<div className="modal mover" onClick={(e) => e.stopPropagation()}>
<h3>Move {isFolder ? 'folder ' : ''}<span className="mono">{item.name}</span></h3>
<div className="mover-nav">
<button className="icon-btn" title="back" disabled={at === 0} onClick={() => setAt(at - 1)}>‹</button>
<button className="icon-btn" title="forward" disabled={at >= hist.length - 1} onClick={() => setAt(at + 1)}>›</button>
<button className="icon-btn" title="up one level" disabled={dir === '/'} onClick={up}>↑</button>
<form className="crumbs" onSubmit={(e) => { e.preventDefault(); go(pathInput) }}>
<button type="button" className="crumb" onClick={() => go('/')}>/</button>
{crumbs.map((c, i) => <button type="button" key={i} className="crumb" onClick={() => go('/' + crumbs.slice(0, i + 1).join('/'))}>{c}</button>)}
<input className="path-input" value={pathInput} onChange={(e) => setPathInput(e.target.value)} onBlur={() => setPathInput(dir)} spellCheck={false} autoCapitalize="off" autoCorrect="off" />
</form>
</div>
<div className="mover-list">
{entries === null && <Loading label="Listing…" />}
{err && <div className="error" style={{ margin: 10 }}>{err}</div>}
{folders.map((e) => {
const self = dir === from && e.name === item.name
return (
<div key={e.name} className={'mover-row' + (self ? ' self' : '')} onClick={() => { if (!self) go(join(dir, e.name)) }}>
<span className="ficon">{e.target ? '🔗' : '📁'}</span>
<span className="fname">{e.name}</span>
{self && <span className="muted small">this is what's moving</span>}
</div>
)
})}
{entries !== null && folders.length === 0 && !err && <div className="muted small" style={{ padding: 12 }}>No folders here — “Move here” puts it in this one.</div>}
</div>
<div className="modal-foot">
<div className="svc-actions">
<button className="btn" onClick={mkdir}>+ folder</button>
<span className="spacer" />
<button className="btn" onClick={onClose}>Cancel</button>
<Btn className="btn on" disabled={!!why || entries === null} busy={busy} onClick={move}>Move here</Btn>
</div>
<div className="muted small mono mover-dest">{why || `→ ${dest}`}</div>
</div>
</div>
</div>
)
}
// ---- previews ----
// Rows ask for their thumbnail only as they scroll near: a browser's own lazy loading
// watches the window, and this list scrolls inside a box of its own. One observer for the
// list, a callback per row, each fired once.
class Near {
private obs: IntersectionObserver
private cbs = new Map<Element, () => void>()
constructor(root: Element) {
this.obs = new IntersectionObserver((ents) => {
for (const e of ents) if (e.isIntersecting) { const cb = this.cbs.get(e.target); if (cb) { this.forget(e.target); cb() } }
}, { root, rootMargin: '400px 0px' })
}
watch(el: Element, cb: () => void) { this.cbs.set(el, cb); this.obs.observe(el); return () => this.forget(el) }
private forget(el: Element) { this.cbs.delete(el); this.obs.unobserve(el) }
}
function Thumb({ url, kind, near, fallback }: { url: string; kind: 'image' | 'video' | 'audio'; near: Near | null; fallback: string }) {
const ref = useRef<HTMLSpanElement>(null)
const [show, setShow] = useState(false)
const [failed, setFailed] = useState('')
useEffect(() => { const el = ref.current; if (!el || !near || show) return; return near.watch(el, () => setShow(true)) }, [near, show])
useEffect(() => { setFailed('') }, [url]) // a changed file is a new URL, and worth another try
const broken = failed === url
return (
<span ref={ref} className={'fthumb ' + kind} title={broken ? 'no preview for this file' : undefined}>
{show && !broken && <img src={url} alt="" decoding="async" onError={() => setFailed(url)} />}
{(!show || broken) && <span className="ficon">{fallback}</span>}
</span>
)
}
// What the previews button says about how they get made here, and the one-tap fixes.
function ThumbNote({ st, sudo, onChanged }: { st: ThumbStatus | null; sudo: boolean; onChanged: () => void }) {
const [task, setTask] = useState<{ id: string; restart: boolean } | null>(null)
const [waiting, setWaiting] = useState(false)
// the GPU fix ends by restarting Ham Lab: poll until it answers again, then re-read the status
useEffect(() => {
if (!waiting) return
const t = setInterval(async () => { try { await api.thumbStatus(sudo, true); setWaiting(false); onChanged() } catch { /* restarting */ } }, 3000)
return () => clearInterval(t)
}, [waiting, sudo, onChanged])
if (!st) return null
const blocked = (st.unusable ?? []).filter((g) => g.permission)
const start = async (label: string, fn: () => Promise<{ id: string }>, restart: boolean) => {
try { const t = await fn(); setTask({ id: t.id, restart }) } catch (e) { toast(`${label} — ${e instanceof Error ? e.message : 'failed'}`, 'err') }
}
return (
<>
{!st.ok && (
<div className="thumb-note notice">
<span>Previews need ffmpeg: {st.error || 'it is not installed on this machine'}.</span>
{st.install && !task && <Btn className="btn on" onClick={() => start('install', api.thumbInstall, false)}>Install ffmpeg with {st.install.pm}</Btn>}
</div>
)}
{st.ok && blocked.length > 0 && (
<div className="thumb-note notice">
<span>{blocked.map((g) => g.name || g.device).join(', ')} is here, but Ham Lab runs as {st.user} and can't open {blocked.length > 1 ? 'them' : 'it'} ({blocked[0].reason}). Previews of videos are on the CPU until it can.</span>
{st.gpu_fix && !task && !waiting && <Btn className="btn on" confirm="Ham Lab adds itself to the GPU's group and restarts. Open terminals are interrupted. Continue?" onClick={() => start('GPU access', api.thumbGPUAccess, true)}>Allow GPU access</Btn>}
{waiting && <span className="muted small"><Spinner /> Ham Lab is restarting; the page reconnects by itself.</span>}
</div>
)}
{task && <TaskOutput id={task.id} onDone={(t) => { const ok = t.exit_code === 0 && !t.error; if (ok && task.restart) setWaiting(true); else if (ok) onChanged(); else toast('failed — the output says where', 'err'); setTask(null) }} />}
</>
)
}
// ---- editor / viewer ----
function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolean; onClose: () => void; onChanged: () => void }) {
@@ -30,11 +204,12 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
const [orig, setOrig] = useState('')
const [err, setErr] = useState('')
const [saving, setSaving] = useState(false)
const [cantPlay, setCantPlay] = useState(false) // the browser tried the file as it is and gave up
const name = path.split('/').pop() ?? path
const dirty = text !== null && text !== orig
useEffect(() => {
setInfo(null); setText(null); setErr('')
setInfo(null); setText(null); setErr(''); setCantPlay(false)
let stop = false
;(async () => {
try {
@@ -92,8 +267,16 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
<textarea className="editor file-editor" value={text} spellCheck={false} onChange={(e) => setText(e.target.value)} />
)}
{info && text === null && isImage(name) && <div className="preview"><img src={api.fileReadURL(path, sudo)} alt={name} /></div>}
{info && text === null && !isImage(name) && !err && (
<div className="notice">{info.type === 'dir' ? 'This is a directory.' : info.size > MAX_EDIT ? `Too large to edit here (${fmtBytes(info.size)}). Download it instead.` : 'Binary file. Download it instead.'}</div>
{/* Video: the browser's own decoder first, and a re-encode on the machine when that fails. Audio as it is. */}
{info && text === null && !isImage(name) && info.preview === 'video' && <Player path={path} sudo={sudo} name={name} />}
{info && text === null && !isImage(name) && info.preview === 'audio' && !cantPlay && (
<div className="preview audio"><div className="audio-box"><div className="ficon">🎵</div><div className="mono small">{name}</div><audio controls preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => setCantPlay(true)} /></div></div>
)}
{info && text === null && !isImage(name) && info.preview === 'audio' && cantPlay && (
<div className="notice">This browser can't play {name} as it is — the codec isn't one it decodes. Download it instead.</div>
)}
{info && text === null && !isImage(name) && !info.preview && !err && (
<div className="notice">{info.type === 'dir' ? 'This is a directory.' : info.type === 'other' ? 'This is a device, pipe or socket — nothing to show.' : info.broken ? `This link points at nothing (${info.target}).` : info.size > MAX_EDIT ? `Too large to edit here (${fmtBytes(info.size)}). Download it instead.` : 'Binary file. Download it instead.'}</div>
)}
</div>
)
@@ -114,10 +297,19 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
const [pathInput, setPathInput] = useState(path)
const [open, setOpen] = useState<string | null>(null)
const [menu, setMenu] = useState<string | null>(null)
const [moving, setMoving] = useState<FileEntry | null>(null)
const [moveTask, setMoveTask] = useState<TaskInfo | null>(null)
const [upload, setUpload] = useState<{ name: string; pct: number; n: number; of: number } | null>(null)
const [drag, setDrag] = useState(false)
const fileInput = useRef<HTMLInputElement>(null)
const canSudo = useRef(true)
const [previews, setPreviews] = useState<boolean>(() => { try { return localStorage.getItem(PREVIEWS_KEY) === '1' } catch { return false } })
const [thumbs, setThumbs] = useState<ThumbStatus | null>(null)
const [listEl, setListEl] = useState<HTMLDivElement | null>(null)
const near = useMemo(() => (listEl && previews ? new Near(listEl) : null), [listEl, previews])
const readThumbs = useCallback((refresh = false) => { api.thumbStatus(sudo, refresh).then(setThumbs).catch((e) => setThumbs({ ok: false, engine: '', summary: '', gpus: [], error: e instanceof Error ? e.message : 'failed', user: '', cache: { path: '', entries: 0, bytes: 0 } })) }, [sudo])
useEffect(() => { try { localStorage.setItem(PREVIEWS_KEY, previews ? '1' : '0') } catch { /* private mode */ } }, [previews])
useEffect(() => { if (previews) readThumbs(); else setThumbs(null) }, [previews, readThumbs])
const load = useCallback(async (p = path, s = sudo) => {
try {
@@ -127,15 +319,22 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
sessionStorage.setItem('hamlab.files.path', p) // only a folder that opened is worth coming back to
} catch (e) {
const msg = e instanceof Error ? e.message : 'failed'
if (/not a directory/i.test(msg) && p !== '/') { // a file was typed: open it, from its folder
setPath(p.slice(0, p.lastIndexOf('/')) || '/'); setOpen(p); return
}
if (/no such file/i.test(msg) && p !== '/' && !sessionStorage.getItem('hamlab.files.lost')) { // gone since last time (another machine, a deleted folder)
sessionStorage.setItem('hamlab.files.lost', '1'); toast(`${p} isn't here — showing /`, 'info'); setPath('/'); return
}
setErr(msg); setEntries([])
}
sessionStorage.removeItem('hamlab.files.lost')
}, [path, sudo])
useEffect(() => { setEntries(null); setPathInput(path); setMenu(null); load(path, sudo); sessionStorage.setItem('hamlab.files.sudo', sudo ? '1' : '0') }, [path, sudo, load])
useEffect(() => { const h = () => setMenu(null); window.addEventListener('click', h); return () => window.removeEventListener('click', h) }, [])
useEscape(() => setMenu(null), menu !== null)
const go = (p: string) => { setOpen(null); setPath(p || '/') }
const go = (p: string) => { setOpen(null); setPath(resolvePath(p, path)) }
const crumbs = path === '/' ? [] : path.slice(1).split('/')
const shown = (entries ?? []).filter((e) => (hidden || !e.name.startsWith('.')) && (!q || e.name.toLowerCase().includes(q.toLowerCase())))
@@ -150,12 +349,10 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
const act = async (label: string, fn: () => Promise<unknown>): Promise<boolean> => {
try { await fn(); toast(label); await load(); return true } catch (e) { toast(`${label} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); return false }
}
// A name is one path segment. ("../x" and "a/b" would quietly act somewhere else.)
const okName = (n?: string | null): n is string => { const t = n?.trim(); if (!t) return false; if (t.includes('/') || t === '.' || t === '..') { toast('a name can\'t contain / or be . or .. — use Move… to put it elsewhere', 'err'); return false } return true }
const mkdir = async () => { const n = await askText('New folder name'); if (okName(n)) act(`created ${n.trim()}`, () => api.fileMkdir(join(path, n.trim()), sudo)) }
const touch = async () => { const n = await askText('New file name'); if (okName(n) && await act(`created ${n.trim()}`, () => api.fileTouch(join(path, n.trim()), sudo))) setOpen(join(path, n.trim())) } // opened only once it exists
const rename = async (e: FileEntry) => { const n = await askText('Rename to', e.name); if (okName(n) && n.trim() !== e.name) act(`renamed to ${n.trim()}`, () => api.fileRename(join(path, e.name), join(path, n.trim()), sudo)) }
const move = async (e: FileEntry) => { const n = await askText('Move to (absolute path)', join(path, e.name)); if (n?.trim() && n.trim() !== join(path, e.name)) act(`moved to ${n.trim()}`, () => api.fileRename(join(path, e.name), n.trim(), sudo)) }
const move = (e: FileEntry) => setMoving(e)
const chmod = async (e: FileEntry) => { const n = await askText(`Permissions for ${e.name} (octal)`, e.perm); if (n?.trim() && n.trim() !== e.perm) act(`chmod ${n.trim()} ${e.name}`, () => api.fileChmod(join(path, e.name), n.trim(), sudo)) }
const del = async (e: FileEntry) => {
const full = join(path, e.name)
@@ -171,6 +368,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
// ---- upload ----
const uploadFiles = async (files: FileList | File[]) => {
const list = Array.from(files)
let done = 0
for (let i = 0; i < list.length; i++) {
const f = list[i]
let overwrite = false
@@ -181,13 +379,22 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
setUpload({ name: f.name, pct: 0, n: i + 1, of: list.length })
try {
await api.fileUpload(path, f, sudo, (pct) => setUpload({ name: f.name, pct, n: i + 1, of: list.length }), overwrite)
done++
} catch (e) { toast(`upload ${f.name} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); break }
}
setUpload(null)
toast(list.length === 1 ? `uploaded ${list[0].name}` : `uploaded ${list.length} files`)
load()
if (done === 1) toast(`uploaded ${list.length === 1 ? list[0].name : '1 file'}`)
else if (done > 1) toast(`uploaded ${done} files`)
if (done > 0) load()
}
// A dropped folder arrives as a nameless 0-byte "file"; the browser gives no way to read it here.
const onDrop = (e: DragEvent) => {
e.preventDefault(); setDrag(false)
const items = Array.from(e.dataTransfer.items ?? [])
const files = Array.from(e.dataTransfer.files).filter((_, i) => !(items[i]?.webkitGetAsEntry?.()?.isDirectory))
if (files.length < e.dataTransfer.files.length) toast('folders can\'t be dropped — drop the files inside, or upload a zip', 'err')
if (files.length) uploadFiles(files)
}
const onDrop = (e: DragEvent) => { e.preventDefault(); setDrag(false); if (e.dataTransfer.files.length) uploadFiles(e.dataTransfer.files) }
if (open) return <div className="files"><Viewer path={open} sudo={sudo} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
@@ -202,6 +409,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
<div className="files-tools">
<button className={'btn' + (sudo ? ' on' : '')} title="browse as root" onClick={() => setSudo(!sudo)}>{sudo ? 'root' : 'sudo'}</button>
<button className={'btn' + (hidden ? ' on' : '')} title="show hidden files" onClick={() => setHidden(!hidden)}>.hidden</button>
<button className={'btn' + (previews ? ' on' : '')} title={'thumbnails for images and videos' + (thumbs ? ` — ${thumbs.summary}` + (thumbs.cache.entries ? ` · ${thumbs.cache.entries} kept (${fmtBytes(thumbs.cache.bytes)})` : '') : '')} onClick={() => setPreviews(!previews)}>previews</button>
<select className="btn" value={sort} onChange={(e) => setSort(e.target.value as Sort)}><option value="name">name</option><option value="size">size</option><option value="mtime">modified</option></select>
<input className="filter" placeholder="filter" value={q} onChange={(e) => setQ(e.target.value)} />
<button className="btn" onClick={mkdir}>+ folder</button>
@@ -213,10 +421,13 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
</div>
</div>
{previews && <ThumbNote st={thumbs} sudo={sudo} onChanged={() => readThumbs(true)} />}
{moveTask && <TaskOutput id={moveTask.id} onDone={(t) => { load(); if (t.exit_code === 0 && !t.error) toast(`${moveTask.title}: done`); else toast(`${moveTask.title}: failed`, 'err') }} />}
{moveTask && <div className="thumb-note"><button className="btn" onClick={() => setMoveTask(null)}>Hide output</button></div>}
{upload && <div className="upload-bar"><Spinner /> {upload.of > 1 ? `${upload.n}/${upload.of} · ` : ''}{upload.name} · {upload.pct}%<div className="meter"><div className="fill" style={{ width: `${upload.pct}%` }} /></div></div>}
{err && <div className="error" style={{ padding: '6px 12px' }}>{err}{/permission denied/i.test(err) && !sudo && <> · <button className="link" onClick={() => setSudo(true)}>retry as root</button></>}</div>}
<div className="file-list">
<div className={'file-list' + (previews ? ' thumbs' : '')} ref={setListEl}>
{entries === null && <Loading label="Listing…" />}
{path !== '/' && entries !== null && (
<div className="file-row up" onClick={() => go(path.slice(0, path.lastIndexOf('/')) || '/')}><span className="ficon">↩</span><span className="fname">..</span></div>
@@ -226,7 +437,9 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
const isDir = e.type === 'dir'
return (
<div key={e.name} className={'file-row' + (e.broken ? ' broken' : '')} onClick={() => (isDir ? go(full) : setOpen(full))}>
<span className="ficon">{icon(e)}</span>
{previews && e.preview && thumbs?.ok
? <Thumb url={api.fileThumbURL(full, sudo, e.mtime, e.size)} kind={e.preview} near={near} fallback={icon(e)} />
: <span className="ficon">{icon(e)}</span>}
<span className="fname" title={e.target ? `→ ${e.target}` : e.name}>{e.name}{e.target && <span className="muted"> → {e.target}</span>}</span>
<span className="fmeta num">{isDir ? '' : fmtBytes(e.size)}</span>
<span className="fmeta num">{fmtTime(e.mtime)}</span>
@@ -249,6 +462,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
{entries !== null && shown.length === 0 && !err && <div className="muted" style={{ padding: 12 }}>{q ? 'Nothing matches.' : 'Empty folder.'}{!hidden && entries.some((e) => e.name.startsWith('.')) ? ' (hidden files not shown)' : ''}</div>}
</div>
{drag && <div className="drop-hint">Drop to upload into {path}</div>}
{moving && <MovePicker item={moving} from={path} sudo={sudo} hidden={hidden} onClose={() => setMoving(null)} onMoved={(t) => { setMoving(null); if (t) setMoveTask(t); load() }} />}
</div>
)
}
+147
View File
@@ -0,0 +1,147 @@
// A video in the Files viewer. First the file as it is, handed to the browser's own
// decoder; when that fails — Safari and an MKV, HEVC without a hardware decoder — a
// picker: codec and size, and the machine re-encodes it to HLS on its own GPU (or CPU),
// which plays natively on Safari and through hls.js everywhere else. The same picker is
// one tap away even when the file plays, for a 36 Mbps capture over a slow link.
import { useCallback, useEffect, useRef, useState } from 'react'
import { api, apiPath, PlayCaps, PlayStatus, PlayStream } from './api'
import { Btn, Spinner } from './ui'
type Mode = 'native' | 'pick' | 'hls'
const CODEC_KEY = 'hamlab.play.codec'
const QUALITY_KEY = 'hamlab.play.quality'
const remember = (k: string, v: string) => { try { localStorage.setItem(k, v) } catch { /* private mode */ } }
const recall = (k: string, d: string) => { try { return localStorage.getItem(k) || d } catch { return d } }
const fmtMbps = (bps?: number) => (bps ? `${(bps / 1e6).toFixed(bps >= 10e6 ? 0 : 1)} Mbps` : '')
const describe = (s: PlayStream) => {
const v = s.source.video
const src = v ? `${v.codec}${v.profile ? ' ' + v.profile : ''} ${v.width}×${v.height}${v.fps ? ` ${Math.round(v.fps)} fps` : ''}${s.source.bit_rate ? ` ${fmtMbps(s.source.bit_rate)}` : ''} in ${s.source.container.split(',')[0]}` : s.source.container
const out = s.codec === 'copy' ? 'repackaged' : `${s.codec === 'h264' ? 'H.264' : s.codec.toUpperCase()} ${s.width}×${s.height} ${(s.kbps / 1000).toFixed(s.kbps >= 10000 ? 0 : 1)} Mbps`
return `${src} → ${out} on ${s.engine}`
}
export default function Player({ path, sudo, name }: { path: string; sudo: boolean; name: string }) {
const [mode, setMode] = useState<Mode>('native')
const [why, setWhy] = useState('') // what brought the picker up
const [caps, setCaps] = useState<PlayCaps | null>(null)
const [codec, setCodec] = useState(() => recall(CODEC_KEY, 'h264'))
const [quality, setQuality] = useState(() => recall(QUALITY_KEY, '1080p'))
const [stream, setStream] = useState<{ s: PlayStream; url: string } | null>(null)
const [status, setStatus] = useState<PlayStatus | null>(null)
const [err, setErr] = useState('')
const video = useRef<HTMLVideoElement>(null)
const streamRef = useRef(stream)
streamRef.current = stream
useEffect(() => { setMode('native'); setWhy(''); setStream(null); setStatus(null); setErr('') }, [path, sudo])
useEffect(() => { remember(CODEC_KEY, codec) }, [codec])
useEffect(() => { remember(QUALITY_KEY, quality) }, [quality])
useEffect(() => { if (mode === 'pick' && !caps) api.playCaps(sudo).then(setCaps).catch((e) => setCaps({ codecs: [], qualities: [], error: e instanceof Error ? e.message : 'failed' })) }, [mode, caps, sudo])
// a stream is the machine's work: let go of it when this player does — and when the page
// itself goes, by a beacon, the one request a browser still sends on its way out
const stop = useCallback(() => { const s = streamRef.current; if (s) { api.playStop(s.s.id).catch(() => { /* gone already */ }); setStream(null); setStatus(null) } }, [])
useEffect(() => () => stop(), [stop])
useEffect(() => {
const h = () => { const s = streamRef.current; if (s) navigator.sendBeacon(apiPath(`/api/files/play/${s.s.id}/close`)) }
window.addEventListener('pagehide', h)
return () => window.removeEventListener('pagehide', h)
}, [])
const start = async () => {
setErr('')
try {
stop()
const r = await api.playStart(path, codec, quality, sudo)
setStream({ s: r.stream, url: r.url }); setMode('hls')
} catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
}
// feed the stream to the element: Safari plays HLS itself, the rest through hls.js
useEffect(() => {
const v = video.current
if (mode !== 'hls' || !stream || !v) return
const url = apiPath(stream.url)
let hls: { destroy: () => void } | null = null
let gone = false
;(async () => {
if (v.canPlayType('application/vnd.apple.mpegurl')) {
v.src = url
} else {
const { default: Hls } = await import('hls.js')
if (gone) return
if (!Hls.isSupported()) { setErr('this browser has neither HLS nor MediaSource — nothing can play here'); return }
const h = new Hls({ maxBufferLength: 60, backBufferLength: 30 })
hls = h
h.on(Hls.Events.ERROR, (_e, d) => { if (d.fatal) setErr(`playback failed: ${d.details}`) })
h.loadSource(url)
h.attachMedia(v)
}
v.play().catch(() => { /* needs a tap on some phones */ })
})()
return () => { gone = true; hls?.destroy() }
}, [mode, stream])
// follow the re-encode while it runs
useEffect(() => {
if (mode !== 'hls' || !stream) return
let stopPolling = false
const tick = async () => { try { const st = await api.playStatus(stream.s.id); if (!stopPolling) { setStatus(st); if (st.running) setTimeout(tick, 2000) } } catch { /* ended */ } }
tick()
return () => { stopPolling = true }
}, [mode, stream])
const pick = (reason: string) => { setWhy(reason); setMode('pick') }
const codecs = caps?.codecs ?? []
return (
<div className="player">
{mode === 'native' && (
<>
<div className="play-bar"><span>as it is</span><span className="spacer" /><button className="btn" onClick={() => pick('')}>Re-encode…</button></div>
<div className="preview"><video controls playsInline preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => pick(`This browser can't play ${name} as it is — the container or codec isn't one it decodes.`)} /></div>
</>
)}
{mode === 'pick' && (
<div className="play-pick notice">
<div>{why || `Re-encode ${name} on this machine and play that instead.`}</div>
{caps?.error && <div className="error" style={{ marginTop: 8 }}>{caps.error}</div>}
{!caps && !err && <div className="muted small" style={{ marginTop: 8 }}><Spinner /> asking the machine what it can encode…</div>}
{caps && !caps.error && (
<div className="row">
<label>codec
<select value={codec} onChange={(e) => setCodec(e.target.value)}>
{codecs.map((c) => <option key={c.id} value={c.id} disabled={!c.engine}>{c.label}{c.engine ? ` — ${c.engine}` : ' — not on this machine'}</option>)}
</select>
</label>
<label>size
<select value={quality} onChange={(e) => setQuality(e.target.value)} disabled={codec === 'copy'}>
{caps.qualities.map((q) => <option key={q.id} value={q.id}>{q.label}</option>)}
</select>
</label>
<Btn className="btn on" disabled={!codecs.find((c) => c.id === codec)?.engine} onClick={start}>Play</Btn>
<a className="btn" href={api.fileDownloadURL(path, sudo)} download={name}>Download instead</a>
</div>
)}
{err && <div className="error" style={{ marginTop: 8 }}>{err}</div>}
</div>
)}
{mode === 'hls' && stream && (
<>
<div className="play-bar">
<span title={describe(stream.s)}>{describe(stream.s)}</span>
{status && status.running && <span>· {status.speed ? `${status.speed.toFixed(1)}×` : ''} {stream.s.source.duration ? `${Math.min(100, Math.round(status.progress / stream.s.source.duration * 100))}% through` : `${Math.round(status.progress)} s done`} — seek within what's done</span>}
{status?.done && <span>· all of it is ready</span>}
{status?.error && <span className="error">· stopped early: {status.error}</span>}
<span className="spacer" />
<button className="btn" onClick={() => { stop(); setMode('pick'); setWhy('') }}>Change…</button>
</div>
<div className="preview"><video ref={video} controls autoPlay playsInline /></div>
{err && <div className="error" style={{ margin: 12 }}>{err}</div>}
</>
)}
</div>
)
}
+1 -1
View File
@@ -101,7 +101,7 @@ export default function StackNew({ mode, from, networks = [], taken, onClose, on
{homes.length > 1 && <label className="muted">Where
<select value={home} onChange={(e) => setHome(e.target.value)}>{homes.map((h) => <option key={h.dir + h.via} value={h.dir}>{sameTail(h) ? h.dir : tail(h.dir)} · {h.stacks} stack{h.stacks === 1 ? '' : 's'}</option>)}</select>
</label>}
{picked && <div className="muted small mono" style={{ overflowWrap: 'anywhere' }}>{picked.dir}/{name || '…'}/compose.yaml{picked.held_by && ` — kept on ${picked.held_by}, deployed here over ${picked.remote}`}{picked.via && ` — compose runs inside ${picked.via}, like its neighbours`}</div>}
{picked && <div className="muted small mono" style={{ overflowWrap: 'anywhere' }}>{picked.dir}/{name || '…'}/compose.yaml{picked.via && ` — compose runs inside ${picked.via}, like its neighbours`}</div>}
{!from && <div className="tabs"><button className={how === 'form' ? 'on' : ''} onClick={() => setHow('form')}>Form</button><button className={how === 'compose' ? 'on' : ''} onClick={toCompose}>Compose</button></div>}
{how === 'form' ? <>
<label className="muted">Image<input className="mono" value={form.image} onChange={set('image')} placeholder="traefik/whoami:latest" spellCheck={false} autoCapitalize="off" autoCorrect="off" /></label>
+2 -1
View File
@@ -369,7 +369,8 @@ export default function Top() {
<div className="t-up">up {upfor(uptime)}</div>
</div>
<div className="t-cores">
<div className="t-cores-head"><b>{model || 'cpu'}</b><span className="spacer" />{pt?.cpu_mhz ? `${Math.round(pt.cpu_mhz)} MHz` : ''}</div>
<div className="t-cores-head"><b>{model || 'cpu'}</b><span className="spacer" />{pt?.cpu_mhz ? `${Math.round(pt.cpu_mhz)} MHz` : ''}
{pt?.cpu_power_w ? <span title="the CPU's own power reading: cores, built-in graphics and memory controller, not the whole machine">{pt.cpu_mhz ? ' · ' : ''}{pt.cpu_power_w < 10 ? pt.cpu_power_w.toFixed(1) : Math.round(pt.cpu_power_w)} W</span> : null}</div>
<div className="t-core">
<span className="t-core-n">CPU</span>
<Meter pct={cpu} grad="cpu" />
+3 -1
View File
@@ -4,7 +4,9 @@
import { useEffect, useRef, useState } from 'react'
import { wsURLPath } from './api'
import { Btn, useEscape } from './ui'
import './vendor/spice-html5/spice.css'
// No spice.css: upstream's stylesheet is for its own demo page and sets body, *, button and
// canvas for the whole document. Imported here it restyled every button in Ham Lab. The
// console is styled by the .vga rules in styles.css.
type Spice = typeof import('./vendor/spice-html5/src/main.js')
+41 -11
View File
@@ -81,7 +81,7 @@ export default function Virtualization() {
{tab === 'images' && <Images st={st} images={images} instances={instances} onTask={startTask} onNew={(image, vm) => { setTab('instances'); setMaking({ image, vm }) }} onChanged={load} />}
{tab === 'library' && <Library st={st} ov={ov} onTask={startTask} onChanged={load} onNewVM={(iso) => setFromIso(iso)} />}
</div>
{tab === 'instances' && (cur ? <Detail i={cur} st={st} onBack={() => setSel('')} onChanged={load} onTask={startTask} /> : <Summary ov={ov} />)}
{tab === 'instances' && (cur ? <Detail i={cur} st={st} onBack={() => setSel('')} onChanged={load} onTask={startTask} /> : <Summary ov={ov} onChanged={load} />)}
{making && <NewInstance st={st} ov={ov} preset={making} onClose={() => setMaking(null)} onStarted={(t) => { setMaking(null); startTask(t) }} />}
{fromIso !== null && <NewVMFromISO st={st} ov={ov} iso={fromIso} onClose={() => setFromIso(null)} onStarted={(t) => { setFromIso(null); startTask(t); setTab('instances') }} />}
</div>
@@ -151,11 +151,11 @@ function InstallDialog({ st, onClose, onStarted }: { st: VirtStatus; onClose: ()
<div className="notice">Everything goes under <span className="mono">{st.dir}</span>: the pool's backing file, ISOs, images, exports. Where is not a choice; how big is.</div>
<label className="muted">Pool driver
<select value={driver} onChange={(e) => setDriver(e.target.value)}>
{st.drivers.map((d) => <option key={d} value={d}>{d}{d === 'zfs' ? ' — snapshots and clones are instant' : d === 'btrfs' ? ' — snapshots, on a loop file' : ' — plain files, no snapshots to speak of'}</option>)}
{st.drivers.map((d) => <option key={d} value={d}>{d === 'dir' ? 'plain files on this disk — each VM\u2019s disk is its own file, sized on its own; snapshots are copies' : d === 'zfs' ? 'zfs — one pre-sized file holding every disk; snapshots and clones are instant' : 'btrfs — one pre-sized file holding every disk; snapshots are instant'}</option>)}
</select>
</label>
{driver !== 'dir' && (
<label className="muted">Pool size, GiB — a sparse file, grows as used · {fmtBytes(st.disk_free)} free
<label className="muted">Ceiling of that file, GiB — sparse, it grows only as disks fill it · {fmtBytes(st.disk_free)} free
<input type="number" min={5} max={Math.floor(st.disk_free / GiB)} value={size} onChange={(e) => setSize(e.target.value)} required />
</label>
)}
@@ -187,7 +187,17 @@ function InstallDialog({ st, onClose, onStarted }: { st: VirtStatus; onClose: ()
// ---- the right-hand pane when nothing is selected: what this machine's Incus is ----
function Summary({ ov }: { ov: VirtOverview }) {
// A location is a plain-files storage pool at a folder of his choosing.
async function addLocation(onDone: (name: string) => void) {
const path = await askText('Folder on this machine for instance disks', '/', 'Next')
if (!path) return
const base = path.split('/').filter(Boolean).pop() ?? 'disks'
const name = await askText(`A name for ${path}`, base.toLowerCase().replace(/[^a-z0-9-]+/g, '-').replace(/^-|-$/g, ''), 'Add')
if (!name) return
try { const r = await api.virtPoolCreate(name, path); toast(`${r.name} added at ${r.path}`); onDone(r.name) } catch (e) { toast(errMsg(e), 'err') }
}
function Summary({ ov, onChanged }: { ov: VirtOverview; onChanged: () => void }) {
const st = ov.status
return (
<div className="svc-detail">
@@ -198,9 +208,11 @@ function Summary({ ov }: { ov: VirtOverview }) {
{st.layout && <div><span className="k">Installed</span><span>{when(st.layout.attempted)} by Ham Lab {st.layout.hamlab_version ?? ''}</span></div>}
{!st.layout && <div><span className="k">Installed</span><span>outside Ham Lab — pools and networks below are as found</span></div>}
</div>
<h3 className="stack-head">Storage pools</h3>
<table className="virt-table"><thead><tr><th>Pool</th><th>Driver</th><th>Source</th><th className="num">Used</th><th className="num">Of</th><th className="num">Users</th></tr></thead>
<tbody>{(ov.pools ?? []).map((p) => <tr key={p.name}><td>{p.name}</td><td>{p.driver}</td><td className="mono small">{p.source}</td><td className="num">{p.total ? fmtBytes(p.used) : '—'}</td><td className="num">{p.total ? fmtBytes(p.total) : '—'}</td><td className="num">{p.used_by}</td></tr>)}</tbody></table>
<h3 className="stack-head">Disk locations <span className="muted">· where an instance\u2019s disk can go</span></h3>
<table className="virt-table"><thead><tr><th>Location</th><th>Where</th><th className="num">Used</th><th className="num">Free</th><th className="num">Disks</th><th /></tr></thead>
<tbody>{(ov.pools ?? []).map((p) => <tr key={p.name}><td>{p.name}{p.name === ov.default_pool && <span className="badge">default</span>}</td><td>{p.where}<div className="muted small mono">{p.driver} · {p.source}</div></td><td className="num">{p.total ? fmtBytes(p.used) : '—'}</td><td className="num">{p.total ? fmtBytes(p.total - p.used) : '—'}</td><td className="num">{p.used_by}</td>
<td className="num">{p.used_by === 0 && p.name !== ov.default_pool && <Btn className="btn danger" confirm={`Remove the location ${p.name}? Nothing uses it; the folder itself is left alone.`} onClick={async () => { try { await api.virtPoolDelete(p.name); toast('removed'); onChanged() } catch (e) { toast(errMsg(e), 'err') } }}>Remove</Btn>}</td></tr>)}</tbody></table>
<div className="svc-actions"><Btn className="btn" onClick={() => addLocation(onChanged)}>+ Add a location…</Btn><span className="muted small">a folder on this machine; VM disks become files in it, containers folders</span></div>
<h3 className="stack-head">Networks</h3>
<table className="virt-table"><thead><tr><th>Network</th><th>Type</th><th>Address</th><th>NAT</th><th className="num">Users</th></tr></thead>
<tbody>{(ov.networks ?? []).filter((n) => n.managed).map((n) => <tr key={n.name}><td>{n.name}</td><td>{n.type}</td><td className="mono small">{n.address ?? '—'}</td><td>{n.nat ? 'yes' : 'no'}</td><td className="num">{n.used_by}</td></tr>)}</tbody></table>
@@ -248,7 +260,7 @@ function Detail({ i, st, onBack, onChanged, onTask }: { i: VirtInstance; st: Vir
{i.description && <div><span className="k">About</span><span>{i.description}</span></div>}
{running && <div><span className="k">Using</span><span>{i.cpu_pct.toFixed(0)}% of a core · {fmtBytes(i.mem_usage)}{i.mem_peak ? ` (peak ${fmtBytes(i.mem_peak)})` : ''}{i.processes > 0 ? ` · ${i.processes} processes` : ''}</span></div>}
<div><span className="k">Limits</span><span>{i.cpu ? `${i.cpu} CPU` : 'all CPUs'} · {i.memory || 'unlimited memory'}</span></div>
{i.disk_total > 0 && <div><span className="k">Disk</span><span>{fmtBytes(i.disk_usage)} of {fmtBytes(i.disk_total)}</span></div>}
<div><span className="k">Disk</span><span>{i.disk_total > 0 ? `${fmtBytes(i.disk_usage)} of ${fmtBytes(i.disk_total)}` : i.disk_usage > 0 ? fmtBytes(i.disk_usage) : 'no limit'}{i.devices.root?.pool ? <span className="muted"> · on {i.devices.root.pool}</span> : ''}</span></div>
<div><span className="k">Network</span><span className="mono small">{i.ipv4.length ? i.ipv4.join(' ') : running ? 'no address yet' : '—'}</span></div>
<div><span className="k">Created</span><span>{when(i.created)}</span></div>
</div>
@@ -348,6 +360,20 @@ function Snapshots({ i, onChanged }: { i: VirtInstance; onChanged: () => void })
// ---- making one ----
// Where an instance's disk goes. One entry is still a choice shown; a new one can be added here.
function PoolPick({ ov, value, onChange }: { ov: VirtOverview; value: string; onChange: (v: string) => void }) {
const [extra, setExtra] = useState<{ name: string; where: string }[]>([])
const pools = [...(ov.pools ?? []).map((p) => ({ name: p.name, where: p.where + (p.total ? ` · ${fmtBytes(p.total - p.used)} free` : '') })), ...extra.filter((e) => !(ov.pools ?? []).some((p) => p.name === e.name))]
return (
<label className="muted">Disk location
<select value={value} onChange={(e) => { if (e.target.value === '+') addLocation((n) => { setExtra((x) => [...x, { name: n, where: 'just added' }]); onChange(n) }); else onChange(e.target.value) }}>
{pools.map((p) => <option key={p.name} value={p.name}>{p.name} — {p.where}</option>)}
<option value="+">+ add a location…</option>
</select>
</label>
)
}
function NewInstance({ st, ov, preset, onClose, onStarted }: { st: VirtStatus; ov: VirtOverview; preset: { image?: string; vm?: boolean }; onClose: () => void; onStarted: (t: TaskInfo) => void }) {
const [vm, setVm] = useState(!!preset.vm && st.kvm)
const [image, setImage] = useState(preset.image ?? '')
@@ -355,6 +381,7 @@ function NewInstance({ st, ov, preset, onClose, onStarted }: { st: VirtStatus; o
const [cpu, setCpu] = useState('')
const [mem, setMem] = useState('')
const [disk, setDisk] = useState('')
const [pool, setPool] = useState(ov.default_pool || ov.pools?.[0]?.name || '')
const nets = (ov.networks ?? []).filter((n) => n.managed || n.type === 'bridge')
const [net, setNet] = useState(nets.find((n) => n.name === 'incusbr0')?.name ?? nets[0]?.name ?? '')
const [cloud, setCloud] = useState('')
@@ -387,7 +414,7 @@ function NewInstance({ st, ov, preset, onClose, onStarted }: { st: VirtStatus; o
}, [q, local, remote])
const submit = async (e: FormEvent) => {
e.preventDefault(); setErr(''); setBusy(true)
try { onStarted(await api.virtCreate({ name: name.trim(), image: image.trim(), vm, cpu: cpu.trim(), memory: mem.trim(), disk: disk.trim(), network: net, cloud_init: showCloud ? cloud : '', profiles: [], description: desc.trim(), start })) }
try { onStarted(await api.virtCreate({ name: name.trim(), image: image.trim(), vm, cpu: cpu.trim(), memory: mem.trim(), disk: disk.trim(), pool, network: net, cloud_init: showCloud ? cloud : '', profiles: [], description: desc.trim(), start })) }
catch (e) { setErr(errMsg(e)); setBusy(false) }
}
return (
@@ -418,8 +445,9 @@ function NewInstance({ st, ov, preset, onClose, onStarted }: { st: VirtStatus; o
<div className="virt-form-row">
<label className="muted">CPUs<input className="mono" value={cpu} onChange={(e) => setCpu(e.target.value)} placeholder={vm ? '2' : 'all'} /></label>
<label className="muted">Memory<input className="mono" value={mem} onChange={(e) => setMem(e.target.value)} placeholder={vm ? '2GiB' : 'unlimited'} /></label>
<label className="muted">Root disk<input className="mono" value={disk} onChange={(e) => setDisk(e.target.value)} placeholder={vm ? '20GiB' : 'pool default'} /></label>
<label className="muted">Root disk<input className="mono" value={disk} onChange={(e) => setDisk(e.target.value)} placeholder={vm ? '20GiB' : 'no limit'} /></label>
</div>
<PoolPick ov={ov} value={pool} onChange={setPool} />
<div className="muted small">{fmtBytes(st.mem_avail)} free on this machine.{vm ? ' A VM’s memory is a reservation.' : ''}</div>
<label className="muted">Description — optional<input value={desc} onChange={(e) => setDesc(e.target.value)} maxLength={200} /></label>
<label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
@@ -556,6 +584,7 @@ function NewVMFromISO({ st, ov, iso, onClose, onStarted }: { st: VirtStatus; ov:
const [cpu, setCpu] = useState('2')
const [mem, setMem] = useState('2GiB')
const [disk, setDisk] = useState('20GiB')
const [pool, setPool] = useState(ov.default_pool || ov.pools?.[0]?.name || '')
const nets = (ov.networks ?? []).filter((n) => n.managed || n.type === 'bridge')
const [net, setNet] = useState(nets.find((n) => n.name === 'incusbr0')?.name ?? nets[0]?.name ?? '')
const [sb, setSb] = useState(true)
@@ -564,7 +593,7 @@ function NewVMFromISO({ st, ov, iso, onClose, onStarted }: { st: VirtStatus; ov:
useEscape(onClose)
const submit = async (e: FormEvent) => {
e.preventDefault(); setErr(''); setBusy(true)
try { onStarted(await api.virtFromISO({ name: name.trim(), iso: pick, cpu: cpu.trim(), memory: mem.trim(), disk: disk.trim(), network: net, secureboot: sb, start: true })) }
try { onStarted(await api.virtFromISO({ name: name.trim(), iso: pick, cpu: cpu.trim(), memory: mem.trim(), disk: disk.trim(), pool, network: net, secureboot: sb, start: true })) }
catch (e) { setErr(errMsg(e)); setBusy(false) }
}
return (
@@ -581,6 +610,7 @@ function NewVMFromISO({ st, ov, iso, onClose, onStarted }: { st: VirtStatus; ov:
<label className="muted">Memory<input className="mono" value={mem} onChange={(e) => setMem(e.target.value)} /></label>
<label className="muted">Disk<input className="mono" value={disk} onChange={(e) => setDisk(e.target.value)} /></label>
</div>
<PoolPick ov={ov} value={pool} onChange={setPool} />
<label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
<input type="checkbox" checked={sb} onChange={(e) => setSb(e.target.checked)} style={{ width: 'auto' }} />Secure Boot — Windows, Debian, Ubuntu and Fedora boot with it; Alpine, Arch, TrueNAS and most others need it off
</label>
+29 -8
View File
@@ -85,7 +85,7 @@ export interface GPU {
mhz?: number; req_mhz?: number; max_mhz?: number; mem_used?: number; mem_total?: number
clients: GPUClient[]; clients_ok: boolean; stalled?: boolean
}
export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_temp?: number; core_ids?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_power_w?: number; cpu_temp?: number; core_ids?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
export interface Sample {
t: number; interval_ms: number; cpu: number; cores: number[]; load: [number, number, number]
mem: { total: number; used: number; avail: number }; swap: { total: number; used: number; avail: number }
@@ -154,8 +154,8 @@ export interface Hardware { at: string; system: HWSystem; board: HWBoard; cpu: H
export interface DockerLogLine { t: string; m: string; err?: boolean }
export interface DockerInfo { available: boolean; message?: string; tool?: string; containers: DockerContainer[]; images: Record<string, unknown>[]; volumes: Record<string, unknown>[]; networks: Record<string, unknown>[] }
export interface Stack { name: string; containers: string[]; running: number; file: string; host_file: string; files: string[]; host_files: string[] | null; images: string[] | null; via: string; remote: string; held_by: string; editable: boolean; note?: string; has_env: boolean }
export interface StackHome { dir: string; inner: string; via: string; remote: string; held_by: string; stacks: number; default: boolean }
export interface Stack { name: string; containers: string[]; running: number; file: string; host_file: string; files: string[]; host_files: string[] | null; images: string[] | null; via: string; editable: boolean; note?: string; has_env: boolean }
export interface StackHome { dir: string; inner: string; via: string; stacks: number; default: boolean }
export interface ImageCheck { image: string; state: 'current' | 'update' | 'unknown'; detail?: string }
export interface StackDiff { name: string; base: '' | 'deployed' | 'saved'; old: string; new: string }
export interface AlertRule { id: string; name: string; sensor: string; op: string; value: number | boolean | string; for_sec: number; severity: 'info' | 'warn' | 'crit'; enabled: boolean; machines: string }
@@ -249,7 +249,14 @@ export interface ARCDetail {
}
export interface ARCView { detail: ARCDetail; live: ARCLive | null }
export interface FileEntry { name: string; type: 'dir' | 'file' | 'link' | 'other'; size: number; mode: string; perm: string; mtime: number; user: string; group: string; target?: string; broken?: boolean }
export interface FileEntry { name: string; type: 'dir' | 'file' | 'link' | 'other'; size: number; mode: string; perm: string; mtime: number; user: string; group: string; target?: string; broken?: boolean; preview?: 'image' | 'video' | 'audio' }
export interface PlayCaps { codecs: { id: string; label: string; engine?: string }[]; qualities: { id: string; label: string; height: number }[]; error?: string }
export interface PlayTrack { codec: string; profile?: string; pix_fmt?: string; width?: number; height?: number; fps?: number; channels?: number }
export interface PlayStream { id: string; codec: string; quality: string; engine: string; width: number; height: number; kbps: number; source: { container: string; duration: number; bit_rate?: number; video?: PlayTrack; audio?: PlayTrack } }
export interface PlayStatus { running: boolean; done: boolean; error?: string; progress: number; speed: number; segments: number }
export interface ThumbGPU { api: string; device: string; name?: string; codecs?: string[]; reason?: string; permission?: boolean }
// How previews get made on the machine: ok with an engine, or why not and what would fix it.
export interface ThumbStatus { ok: boolean; engine: 'gpu' | 'cpu' | ''; summary: string; ffmpeg?: string; gpus: ThumbGPU[]; unusable?: ThumbGPU[]; error?: string; install?: { pm: string; package: string }; gpu_fix?: boolean; user: string; cache: { path: string; entries: number; bytes: number } }
export interface FileInfo extends FileEntry { path: string; text: boolean }
const q = (path: string, sudo: boolean, extra = '') => `path=${encodeURIComponent(path)}${sudo ? '&sudo=1' : ''}${extra}`
@@ -260,17 +267,17 @@ export interface VirtStatus { state: 'ready' | 'repair' | 'installable' | 'unava
export interface VirtSnapshot { name: string; created: string; expires?: string; size?: number }
export interface VirtInstance { name: string; type: 'container' | 'virtual-machine'; status: string; arch: string; description?: string; image: string; os?: string; release?: string; cpu?: string; memory?: string; created: string; last_used: string; ipv4: string[]; snapshots: VirtSnapshot[]; profiles: string[]; ephemeral: boolean; cpu_pct: number; mem_usage: number; mem_peak: number; disk_usage: number; disk_total: number; pid: number; processes: number; config: Record<string, string>; devices: Record<string, Record<string, string>>; schedule?: string; expiry?: string }
export interface VirtImage { fingerprint: string; aliases: string[]; arch: string; type: string; size: number; created: string; last_used: string; description: string; os?: string; release?: string; variant?: string; cached: boolean; source?: string; public: boolean }
export interface VirtPool { name: string; driver: string; source: string; used: number; total: number; used_by: number }
export interface VirtPool { name: string; driver: string; source: string; where: string; used: number; total: number; used_by: number }
export interface VirtNetwork { name: string; type: string; managed: boolean; address?: string; nat: boolean; used_by: number }
export interface LibraryFile { name: string; size: number; modified: string; attached_to?: string[] }
export interface ExportConfig { enabled: boolean; hour: number; minute: number; keep: number; instance_only: boolean; instances?: string[]; updated?: string }
export interface ExportRun { at: string; exported: string[]; pruned: string[]; errors?: string[]; seconds: number }
export interface ExportFile { instance: string; name: string; size: number; modified: string }
export interface ExportsView { config: ExportConfig; cron: string; next?: string; last: ExportRun | null; files: ExportFile[]; dir: string; cron_file: string }
export interface FromISO { name: string; iso: string; cpu: string; memory: string; disk: string; network: string; secureboot: boolean; start: boolean }
export interface VirtOverview { status: VirtStatus; error?: string; instances?: VirtInstance[]; images?: VirtImage[]; pools?: VirtPool[]; networks?: VirtNetwork[]; library: { images: LibraryFile[]; isos: LibraryFile[] } }
export interface FromISO { name: string; iso: string; cpu: string; memory: string; disk: string; pool: string; network: string; secureboot: boolean; start: boolean }
export interface VirtOverview { status: VirtStatus; error?: string; instances?: VirtInstance[]; images?: VirtImage[]; pools?: VirtPool[]; default_pool?: string; networks?: VirtNetwork[]; library: { images: LibraryFile[]; isos: LibraryFile[] } }
export interface VirtInstallOpts { driver: string; pool_size_gib: number; subnet: string; docker_rules: boolean; arc_max_gib: number }
export interface VirtCreate { name: string; image: string; vm: boolean; cpu: string; memory: string; disk: string; network: string; cloud_init: string; profiles: string[]; description: string; start: boolean }
export interface VirtCreate { name: string; image: string; vm: boolean; cpu: string; memory: string; disk: string; pool: string; network: string; cloud_init: string; profiles: string[]; description: string; start: boolean }
export const api = {
procs: () => req<ProcTable>('GET', '/api/procs'),
@@ -292,6 +299,16 @@ export const api = {
},
fileReadURL: (path: string, sudo: boolean) => apiPath(`/api/files/read?${q(path, sudo)}`),
fileDownloadURL: (path: string, sudo: boolean) => apiPath(`/api/files/download?${q(path, sudo)}`),
// v carries the file's mtime and size, so a changed file is a new URL and an unchanged one a cached one
fileThumbURL: (path: string, sudo: boolean, mtime: number, size: number) => apiPath(`/api/files/thumb?${q(path, sudo, `&v=${mtime}-${size}`)}`),
thumbStatus: (sudo: boolean, refresh = false) => req<ThumbStatus>('GET', `/api/files/thumbs?${sudo ? 'sudo=1&' : ''}${refresh ? 'refresh=1' : ''}`),
thumbInstall: () => req<TaskInfo>('POST', '/api/files/thumbs/install'),
thumbGPUAccess: () => req<TaskInfo>('POST', '/api/files/thumbs/gpu-access'),
// playback of what the browser can't play as it is: an HLS re-encode on the machine
playCaps: (sudo: boolean) => req<PlayCaps>('GET', `/api/files/play-caps${sudo ? '?sudo=1' : ''}`),
playStart: (path: string, codec: string, quality: string, sudo: boolean) => req<{ stream: PlayStream; url: string }>('POST', `/api/files/play${sudo ? '?sudo=1' : ''}`, { path, codec, quality }),
playStatus: (id: string) => req<PlayStatus>('GET', `/api/files/play/${id}`),
playStop: (id: string) => req<void>('DELETE', `/api/files/play/${id}`),
fileWrite: async (path: string, text: string, sudo: boolean) => {
const r = await fetch(apiPath(`/api/files/write?${q(path, sudo)}`), { method: 'PUT', credentials: 'same-origin', body: text, headers: { 'Content-Type': 'application/octet-stream' } })
if (r.status === 401) throw new Unauthenticated()
@@ -308,6 +325,8 @@ export const api = {
fileMkdir: (path: string, sudo: boolean) => req<void>('POST', `/api/files/mkdir${sudo ? '?sudo=1' : ''}`, { path }),
fileTouch: (path: string, sudo: boolean) => req<void>('POST', `/api/files/touch${sudo ? '?sudo=1' : ''}`, { path }),
fileRename: (path: string, to: string, sudo: boolean) => req<void>('POST', `/api/files/rename${sudo ? '?sudo=1' : ''}`, { path, to }),
// a rename when the ends share a filesystem (nothing back), otherwise the copy-and-remove runs as a task (the task back)
fileMove: (path: string, to: string, sudo: boolean) => req<TaskInfo | undefined>('POST', `/api/files/move${sudo ? '?sudo=1' : ''}`, { path, to }),
fileDelete: (path: string, recursive: boolean, sudo: boolean) => req<void>('POST', `/api/files/delete${sudo ? '?sudo=1' : ''}`, { path, recursive }),
fileChmod: (path: string, mode: string, sudo: boolean) => req<void>('POST', `/api/files/chmod${sudo ? '?sudo=1' : ''}`, { path, mode }),
storage: () => req<{ devices: StorageDevice[]; zfs: ZFSInfo; smartctl: boolean }>('GET', '/api/storage'),
@@ -456,6 +475,8 @@ export const api = {
virtExportsRun: (instance = '') => req<TaskInfo>('POST', '/api/virt/exports/run', { instance }),
virtExportDelete: (instance: string, name: string) => req<void>('POST', '/api/virt/exports/delete', { instance, name }),
virtSSHKeys: () => req<{ keys: string[]; user: string }>('GET', '/api/virt/sshkeys'),
virtPoolCreate: (name: string, path: string) => req<{ name: string; path: string }>('POST', '/api/virt/pools', { name, path }),
virtPoolDelete: (name: string) => req<void>('POST', '/api/virt/pools/delete', { name }),
}
export function wsURLPath(path: string): string {
+36
View File
@@ -364,8 +364,44 @@ h2.section { margin: 0; font-size: 13px; font-weight: 600; color: var(--muted);
.file-editor { flex: 1; min-height: 0; border: 0; border-radius: 0; resize: none; font-size: 13px; }
.preview { flex: 1; display: flex; align-items: center; justify-content: center; overflow: auto; padding: 12px; }
.preview img { max-width: 100%; max-height: 100%; object-fit: contain; }
.preview video { max-width: 100%; max-height: 100%; background: #000; border-radius: 4px; outline: none; }
.player { flex: 1; min-height: 0; display: flex; flex-direction: column; }
.player .preview { flex: 1; min-height: 0; }
.play-bar { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; padding: 6px 12px; font-size: 12px; color: var(--muted); border-bottom: 1px solid var(--border); }
.play-bar select { padding: 4px 8px; background: var(--bg); border: 1px solid var(--border); border-radius: 5px; color: var(--fg); font-size: 12px; }
.play-pick { margin: 12px; }
.play-pick .row { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; margin-top: 10px; }
.play-pick label { display: flex; align-items: center; gap: 6px; font-size: 12px; color: var(--muted); }
.preview.audio { align-items: flex-start; }
.audio-box { display: flex; flex-direction: column; align-items: center; gap: 10px; width: min(560px, 100%); margin-top: 8vh; }
.audio-box .ficon { font-size: 48px; }
.audio-box audio { width: 100%; }
/* previews: the icon column becomes a thumbnail box for images and videos */
.file-list.thumbs .file-row { grid-template-columns: 72px 1fr 80px 80px 130px 32px; padding-top: 4px; padding-bottom: 4px; min-height: 52px; }
.fthumb { position: relative; width: 72px; height: 44px; display: flex; align-items: center; justify-content: center; background: var(--bg3); border-radius: 4px; overflow: hidden; }
.fthumb img { max-width: 100%; max-height: 100%; object-fit: contain; display: block; }
.fthumb.video::after, .fthumb.audio::after { content: '▶'; position: absolute; right: 2px; bottom: 2px; font-size: 9px; line-height: 1; padding: 2px 3px; border-radius: 3px; background: rgba(0,0,0,.6); color: #fff; }
.fthumb.audio::after { content: '♪'; }
.fthumb .ficon { font-size: 18px; }
.thumb-note { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; margin: 6px 10px 0; }
/* move: the folder picker */
.modal.mover { width: min(560px, 100%); height: min(76vh, 100%); overflow: hidden; }
.mover-nav { display: flex; align-items: center; gap: 2px; }
.mover-nav .icon-btn { font-size: 18px; min-width: 30px; line-height: 1; }
.mover-nav .icon-btn:disabled { opacity: .3; }
.mover-nav .crumbs { flex: 1; min-width: 0; margin-left: 4px; }
.modal.mover > .mover-list { flex: 1 1 auto; min-height: 80px; overflow-y: auto; margin: 0 -16px; border-top: 1px solid var(--border); }
.mover-row { display: flex; align-items: center; gap: 8px; padding: 9px 16px; cursor: pointer; border-bottom: 1px solid var(--border); min-height: 38px; }
.mover-row:hover { background: var(--bg3); }
.mover-row.self { opacity: .5; cursor: default; }
.mover-row .fname { flex: 1; }
.mover-dest { overflow-wrap: anywhere; }
.thumb-note .btn { padding-top: 3px; padding-bottom: 3px; }
.files-bar .task-out { margin: 6px 10px 0; max-height: 160px; }
@media (max-width: 800px) {
.file-row { grid-template-columns: 24px 1fr 70px 32px; } .file-row .fmeta:nth-child(4), .file-row .perm { display: none; }
.file-list.thumbs .file-row { grid-template-columns: 60px 1fr 70px 32px; min-height: 46px; }
.fthumb { width: 60px; height: 38px; }
.path-input { width: 100%; margin: 4px 0 0; }
.files-tools .filter { width: 80px; }
}
+8 -1
View File
@@ -1,9 +1,16 @@
# spice-html5, vendored
Upstream: https://gitlab.freedesktop.org/spice/spice-html5 — commit b372aab (2026-09-21),
copied unchanged on 2026-10-09 (`src/`, `thirdparty/`, `spice.css`). LGPL-3.0-or-later; the
copied unchanged on 2026-10-09 (`src/`, `thirdparty/`). LGPL-3.0-or-later; the
licence text is beside this file. It draws a VM's SPICE display into a canvas from a websocket
carrying the raw SPICE stream — which is what Incus's VGA console websocket is — so Ham Lab's
daemon bridges the two and this does the rest. Loaded lazily by `VgaConsole.tsx`, so the main
bundle does not carry it. Not on npm in a current form (the `@spice-project/spice-html5`
package stopped at 0.2.1 in 2019), hence the copy.
Upstream's `spice.css` is deliberately **not** here. It is the stylesheet of spice-html5's
demo page, with rules for `body`, `*`, `button` and `canvas` that apply to the whole
document. It was copied and imported on 2026-10-09 and turned every button in Ham Lab into
a gradient pill shifted off its place (v0.1.0-188 to -191). The console is styled by the
`.vga` rules in `src/styles.css`; `scripts/test-css-imports.mjs` fails on any new
stylesheet import that has not been looked at.
-168
View File
@@ -1,168 +0,0 @@
body
{
background-color: #999999;
color: #000000;
margin: 0;
padding: 0;
font-family: "Lucida Grande", "Lucida Sans Unicode", "Helvetica Neue", Helvetica, Arial, Verdana, sans-serif;
font-size: 12pt;
line-height: 1.5em;
}
* { margin: 0; }
button
{
cursor: pointer;
color: #000000;
background-color: #999999;
border: 1px solid #4a4a4a;
min-width: 150px;
border-radius: 10px;
background: -webkit-gradient(linear, left top, left bottom, from(#fff), to(#24414e));
background: -moz-linear-gradient(top, #fff, #24414e);
-ms-transform: translate(15%, -8%);
transform: translate(15%, -8%);
font-size: .90em;
font-weight: bolder;
padding: 0px 0px 3px 0px;
margin-top: 5px;
}
button:focus
{
outline: none;
}
button:hover
{
background-color: #24414e
}
button:active
{
background-color: #24414e;
transform: translate(14.85%, -7%);
}
.SidenavClosed
{
height: 100%;
width: 0;
position: fixed;
z-index: 1;
top: 0;
left: 0;
background-color: #bbbbbb;
box-shadow: 1px 0px #00000040;
overflow-x: hidden;
transition: 0.5s;
padding-top: 60px;
}
.SidenavOpen
{
height: 100%;
width: 200px !important;
position: fixed;
z-index: 1;
top: 0;
left: 0;
background-color: #bbbbbb;
box-shadow: 1px 0px #00000040;
overflow-x: hidden;
transition: 0.5s;
padding-top: 60px;
}
#Sidenav label
{
color: #000000;
margin-left: 3%;
text-shadow: 1px 1px 0px rgba(175, 210, 220, 0.8);
position: absolute;
font-size: .9em;
}
#Sidenav input
{
padding: 3px;
background-color: #fAfAfA;
border: 1px inset #999999;
outline: none;
float: right;
margin-right: 3%;
-moz-border-radius: 3px; -webkit-border-radius: 3px; border-radius: 3px;
box-sizing: border-box;
-moz-box-sizing: border-box;
-webkit-box-sizing: border-box;
position: relative;
max-width: 110px;
}
#Sidenav .closebtn
{
position: absolute;
top: 10px;
right: 10px;
font-size: 25px;
margin-left: 50px;
cursor: pointer;
color: #000000;
}
canvas
{
border: 1px solid #000000;
}
#login
{
background: #bbbbbb;
z-index: 1;
padding-top: 2px;
border-bottom: 2px solid #00000040;
}
#login p
{
margin-left: 27%;
display: inline;
font-size: large;
}
#spice-area
{
height: 100%;
width: 100%;
margin-left: auto;
margin-right: auto;
}
.spice-screen
{
min-height: 600px;
height: 100%;
text-align: center;
padding: 0;
background-color: #999999;
}
.spice-message
{
width: 90%;
height: 40%;
overflow: auto;
margin-left: auto;
margin-right: auto;
padding: 10px;
font-size: 8pt;
line-height: 1.1em;
font-family: 'Andale Mono', monospace;
background-color: #fAfAfA;
border: 1px inset #999999;
border-radius: 3px;
box-sizing: border-box;
-moz-box-sizing: border-box;
-webkit-box-sizing: border-box;
}
.spice-message p {
margin-bottom: 0em;
margin-top: 0em;
}
.spice-message-warning {
color: orange;
}
.spice-message-error {
color: red;
}