Compare commits
29
Commits
@@ -21,7 +21,7 @@ stub tab or a roadmap item.
|
|||||||
| Guest logins | A **guest** is an account like any other, with `guest: true` and an expiry: `login`, sessions and prefs need no special case, and **it may do everything an account may do**. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. | Built 2026-09-27. A read-only version was built first and **cut the same day**: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is *an account*: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to. |
|
| Guest logins | A **guest** is an account like any other, with `guest: true` and an expiry: `login`, sessions and prefs need no special case, and **it may do everything an account may do**. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. | Built 2026-09-27. A read-only version was built first and **cut the same day**: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is *an account*: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to. |
|
||||||
| Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
|
| Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
|
||||||
| Capability, not policy | **A tab offers what the selected machine can do, detected live — never what a rule says should run there.** Where the software is missing but installable, Ham Lab installs it; where it cannot be installed or the hardware lacks it, the tab says so in a sentence and offers what does work. Every action lands on the machine in the switcher. | Mitchell, 2026-10-09, after the Virtualization plan's first draft excluded the Mac mini and `hamtv` on the strength of `hamlet/CLAUDE.md`'s deployment rules: *"I want the option to do it anywhere I want, unless the hardware / software literally doesn't support it. Ham Lab needs to be built in a general-purpose way that can run on any Linux system."* Those rules bind the agent's unasked deployments, not the product. |
|
| Capability, not policy | **A tab offers what the selected machine can do, detected live — never what a rule says should run there.** Where the software is missing but installable, Ham Lab installs it; where it cannot be installed or the hardware lacks it, the tab says so in a sentence and offers what does work. Every action lands on the machine in the switcher. | Mitchell, 2026-10-09, after the Virtualization plan's first draft excluded the Mac mini and `hamtv` on the strength of `hamlet/CLAUDE.md`'s deployment rules: *"I want the option to do it anywhere I want, unless the hardware / software literally doesn't support it. Ham Lab needs to be built in a general-purpose way that can run on any Linux system."* Those rules bind the agent's unasked deployments, not the product. |
|
||||||
| Ham Lab's data lives in Ham Lab's folder | **Everything a tab manages sits under the machine's data directory (`config.DataDir()`), never at a convenient path elsewhere.** Existing material is imported by copying. The engine's runtime state (`/var/lib/docker`, `/var/lib/incus`) is the engine's; Ham Lab keeps what would rebuild it. | Mitchell, 2026-10-09, when the Virtualization plan pointed its ISO library at `/tank/software/isos/`: *"ALL data needs to be under a HamLab folder, because it manages its own data."* The same reason `core`'s compose files moved under `apps/hamlab/stacks/` on 2026-10-05. `grid`'s and the Mac's stacks at `~/stacks/<machine>/` are the standing exception. |
|
| Ham Lab's data lives in Ham Lab's folder | **Everything a tab manages sits under the machine's data directory (`config.DataDir()`), never at a convenient path elsewhere.** Existing material is imported by copying. The engine's runtime state (`/var/lib/docker`, `/var/lib/incus`) is the engine's; Ham Lab keeps what would rebuild it. | Mitchell, 2026-10-09, when the Virtualization plan pointed its ISO library at `/tank/software/isos/`: *"ALL data needs to be under a HamLab folder, because it manages its own data."* The same reason `core`'s compose files moved under `apps/hamlab/stacks/` on 2026-10-05. `grid`'s and the Mac's stacks at `~/stacks/<machine>/` were the standing exception; since 2026-10-10 the stacks code offers `<data dir>/stacks` as the default home everywhere and such folders only as discovered extras. |
|
||||||
|
|
||||||
## Layout
|
## Layout
|
||||||
|
|
||||||
@@ -260,12 +260,13 @@ something long in one, switch away and back, refresh — it's all still there.
|
|||||||
filesystem; compose is then run **inside that container**, so `.env`, relative paths
|
filesystem; compose is then run **inside that container**, so `.env`, relative paths
|
||||||
and the compose labels stay exactly as deployed (verified: labels identical after a
|
and the compose labels stay exactly as deployed (verified: labels identical after a
|
||||||
redeploy). Save validates with `compose config` first, keeps the previous file as
|
redeploy). Save validates with `compose config` first, keeps the previous file as
|
||||||
`<file>.hamlab-bak`, and never deploys by itself. A stack deployed from another host
|
`<file>.hamlab-bak`, and never deploys by itself. A stack whose file is not on the
|
||||||
(Dockhand on `core` driving `grid` over tcp) has no file on that machine, so the node
|
machine that runs it is listed read-only, and that is the end of it: **since 2026-10-10
|
||||||
alone lists it read-only — but **the hub fills that in** (`hubStacks`): asked about a
|
the hub holds no files for a node and never points compose at another machine's
|
||||||
node, it resolves the files on itself and runs compose here with `DOCKER_HOST` pointed
|
Docker.** The `hubStacks` path — which resolved a node's files on the hub and deployed
|
||||||
at the node's `tcp://…:2375`, the way it was deployed; the task runs on the hub and the
|
over `tcp://…:2375`, the way Dockhand had — is gone, and every
|
||||||
node's page follows it. Also per stack: every file it was deployed from (chips), its
|
`/api/nodes/<node>/docker/…` request is the node's own Ham Lab's to answer through the
|
||||||
|
generic proxy. Also per stack: every file it was deployed from (chips), its
|
||||||
**`.env`** (values masked until "Reveal and edit"; validated in place, put back if
|
**`.env`** (values masked until "Reveal and edit"; validated in place, put back if
|
||||||
compose objects), **what Deploy would change** (a line diff against the copy kept at the
|
compose objects), **what Deploy would change** (a line diff against the copy kept at the
|
||||||
last deploy from here, `<file>.hamlab-deployed`, else the copy the last save replaced;
|
last deploy from here, `<file>.hamlab-deployed`, else the copy the last save replaced;
|
||||||
@@ -278,9 +279,10 @@ something long in one, switch away and back, refresh — it's all still there.
|
|||||||
form (image, ports, volumes, environment, restart, command, host networking) that writes
|
form (image, ports, volumes, environment, restart, command, host networking) that writes
|
||||||
the compose for you: a container made here is a one-service stack, so it stays editable
|
the compose for you: a container made here is a one-service stack, so it stays editable
|
||||||
and redeployable, where a bare `docker run` could only be thrown away and typed again.
|
and redeployable, where a bare `docker run` could only be thrown away and typed again.
|
||||||
New stacks go in a **home** — a folder where two or more stacks are already deployed from
|
New stacks go in a **home**. Since 2026-10-10 the default is Ham Lab's own
|
||||||
(Dockhand's `stacks/core` here; `~/stacks` on a machine with none) — and are deployed the
|
`<data dir>/stacks`, offered first on every machine; a folder where two or more stacks
|
||||||
way their neighbours were (inside `dockhand`, or over tcp to a node from the hub). Homes
|
are already deployed from is offered as well, and a stack made there is deployed the way
|
||||||
|
its neighbours were (inside an orchestrator's container, when that is how they run). Homes
|
||||||
are also scanned, so a stack with no containers ("not deployed": never deployed, or taken
|
are also scanned, so a stack with no containers ("not deployed": never deployed, or taken
|
||||||
down) is still listed. **Down** removes a stack's containers; **Delete…** (type its name)
|
down) is still listed. **Down** removes a stack's containers; **Delete…** (type its name)
|
||||||
takes it down and moves its own files to `<home>/.hamlab-trash/<name>-<time>/` — moved,
|
takes it down and moves its own files to `<home>/.hamlab-trash/<name>-<time>/` — moved,
|
||||||
@@ -325,6 +327,14 @@ something long in one, switch away and back, refresh — it's all still there.
|
|||||||
`grid` the sampler hung for 42 minutes in Go's poller on the Arc's `fan1_input` —
|
`grid` the sampler hung for 42 minutes in Go's poller on the Arc's `fan1_input` —
|
||||||
`readStr` now uses plain blocking syscalls and each GPU's sysfs reads run under a
|
`readStr` now uses plain blocking syscalls and each GPU's sysfs reads run under a
|
||||||
3-second deadline (v0.1.0-168; `hamlet/docs/gotchas.md` has the mechanism).
|
3-second deadline (v0.1.0-168; `hamlet/docs/gotchas.md` has the mechanism).
|
||||||
|
- **CPU power on the Processes tab** — **shipped 2026-10-10** (v0.1.0-194,
|
||||||
|
`internal/metrics/cpupower.go`, the `cpu.power` sensor). Asked for beside the clock speed,
|
||||||
|
and that is where it sits: "3600 MHz · 89 W" in the cpu box's header. Watts are the RAPL
|
||||||
|
package energy counters' delta between samples, summed over packages; `psys`, a package's
|
||||||
|
parts and the MMIO duplicate are left out. The counters are root-only, so a daemon running
|
||||||
|
as `core` keeps a second root child, `hamlab cpu-energy`, beside `drm-clients`. Read on all
|
||||||
|
four Intel machines; the Mac mini has no CPU-only counter (its SMC reports the whole
|
||||||
|
machine's draw instead) and shows nothing.
|
||||||
- **The ARC on the ZFS tab** — **shipped 2026-10-02** (`internal/metrics/arc.go`,
|
- **The ARC on the ZFS tab** — **shipped 2026-10-02** (`internal/metrics/arc.go`,
|
||||||
`internal/storage/arc.go`, `GET /api/zfs/arc`, the Cache section in `ZFS.tsx`, `zfs.arc.*`
|
`internal/storage/arc.go`, `GET /api/zfs/arc`, the Cache section in `ZFS.tsx`, `zfs.arc.*`
|
||||||
sensors). Asked for as "current ARC usage, including any types of caching services in the
|
sensors). Asked for as "current ARC usage, including any types of caching services in the
|
||||||
@@ -615,6 +625,197 @@ something long in one, switch away and back, refresh — it's all still there.
|
|||||||
so both modes share one implementation and one set of rules (absolute paths only;
|
so both modes share one implementation and one set of rules (absolute paths only;
|
||||||
`/`, `/etc`, `/usr`, … are never deleted or moved whole). Every write is logged with
|
`/`, `/etc`, `/usr`, … are never deleted or moved whole). Every write is logged with
|
||||||
the caller's IP.
|
the caller's IP.
|
||||||
|
- **Previews on the Files tab** — **shipped 2026-10-11** (v0.1.0-200, `internal/thumbs`,
|
||||||
|
`internal/api/thumbs.go`, `Thumb`/`ThumbNote` in `Files.tsx`). Asked for as a button that
|
||||||
|
toggles a file preview, "obviously not for everything", cached, "use whatever GPU is on the
|
||||||
|
machine we're looking at, and fall back to CPU", "a single small frame preview so it's
|
||||||
|
fast". A **previews** button beside `.hidden` turns the icon column into a 256 px
|
||||||
|
thumbnail for every image and video row (the listing says which: `preview: image|video`
|
||||||
|
per entry, decided by extension in one place, `thumbs.Kind`), loaded as rows scroll within
|
||||||
|
400 px of view — the list scrolls in a box of its own, so it has its own
|
||||||
|
IntersectionObserver rather than the browser's window-based lazy loading. A video thumb
|
||||||
|
carries a ▶ badge. The setting is in localStorage and follows the browser.
|
||||||
|
- **ffmpeg is the one engine**, for images and video alike; `GET /api/files/thumb` is one
|
||||||
|
ffprobe and one ffmpeg: `-ss` a tenth of the way in (30 s at most, `-noaccurate_seek`
|
||||||
|
so one keyframe is decoded), the first real video stream (cover art is skipped), one
|
||||||
|
frame, `scale` to fit 256, MJPEG q4 to a pipe. JPEGs decode at a reduced DCT size
|
||||||
|
(`-lowres`, enough halvings to stay at twice the preview) — a 24 MP photo is read as
|
||||||
|
1.5 MP. Thumbnails are the only thing Ham Lab asks a GPU to do, and it asks the way the
|
||||||
|
Processes tab reads one: **probed, never assumed.** The probe looks ffmpeg and ffprobe
|
||||||
|
up, opens every `/dev/dri/renderD*`, pushes a synthetic frame through each with VAAPI,
|
||||||
|
and reads `vainfo`'s decode profiles per node where it exists; CUDA is tried the same
|
||||||
|
way when `/dev/nvidiactl` is there (untested: nothing here has one). A video goes to the
|
||||||
|
first GPU listing its codec; ffmpeg's own software fallback sits behind that for a
|
||||||
|
profile the list did not distinguish, and a CPU retry behind a hardware run that fails
|
||||||
|
outright. Images are CPU only — one frame through libjpeg at a quarter size beats a
|
||||||
|
round trip through a GPU. Measured on `grid`: 1080p h264 0.4 s on the UHD 750 against
|
||||||
|
0.7 s on the CPU; the 8K Osmo HEVC 1.0 s against 2.9 s. `grid`'s two GPUs are listed in
|
||||||
|
node order and both decode the same set (h264, hevc, vp9, mjpeg, mpeg2 — **no AV1 on
|
||||||
|
either**, per vainfo, so AV1 is dav1d on the CPU).
|
||||||
|
- **Cached in one database file under the machine's data dir**, `<data dir>/thumbs.db`
|
||||||
|
(bbolt — pure Go, so the binary stays static; v0.1.0-202, the first build kept them as
|
||||||
|
a tree of hash-named JPEGs under `thumbs/`, replaced the same day when Mitchell asked
|
||||||
|
where they were kept: "stored in hamlab, maybe under a DB, up to you"). An entry is
|
||||||
|
keyed by path, size and mtime — a changed file gets a new preview, an unchanged one is
|
||||||
|
never decoded twice — and carries the engine, codec, source size and time beside the
|
||||||
|
picture; a count rides in the file so the previews button's tooltip can say "N kept
|
||||||
|
(M MB)" without walking anything. The URL carries `v=<mtime>-<size>` so the browser's
|
||||||
|
own cache is right too (7 days, ETag, 304). A file ffmpeg could not decode is kept with
|
||||||
|
ffmpeg's reason and no picture, answered as 422, retried only when the file changes; a
|
||||||
|
timeout is not kept. A miss never creates the file, so a machine nobody browses with
|
||||||
|
previews on never has one; a cache that cannot be opened is logged and previews are
|
||||||
|
still made and served, just not kept. Nothing expires: a preview is a few KB, and only
|
||||||
|
files somebody scrolled past are ever made. At most 2–4 ffmpegs at once per machine
|
||||||
|
(half the cores, clamped), one per file however many rows ask, and a preview's work
|
||||||
|
runs on a clock of its own so the row that asked first leaving does not cancel it for
|
||||||
|
the rest. Each ffmpeg runs under `nice -n 10`. `hamlet`'s backup job skips the file.
|
||||||
|
- **Root mode makes them as root** (`hamlab thumb`, the sibling of `hamlab fs`, caps on
|
||||||
|
stdin, JSON out), so a file only root can read gets a preview, and the probe runs as
|
||||||
|
root too — root opens every render node whatever groups the daemon has. Which matters:
|
||||||
|
a render node is `root:render 0660`, and on `core` and `hamtv` the `core` user is not in
|
||||||
|
`render`, so there the daemon's own previews are CPU and the tab says so, naming the GPU
|
||||||
|
and the owner, with an **Allow GPU access** button (confirms, then a task writes
|
||||||
|
`hamlab.service.d/gpu.conf` adding `render`/`video` as supplementary groups and
|
||||||
|
restarts Ham Lab three seconds later, the shape the Incus install uses for
|
||||||
|
`incus-admin`). Not pressed anywhere yet. Where ffmpeg is missing (the Mac mini, `gate`)
|
||||||
|
the note offers **Install ffmpeg with dnf/apk** through the Updates machinery
|
||||||
|
(`ffmpeg-free` on Fedora, `ffmpeg` elsewhere); the status re-probes when the task ends.
|
||||||
|
- *Not done, on purpose:* no cap on the cache and no clear button (not asked; it only
|
||||||
|
holds what was looked at); no tonemapping for HDR sources (a 10-bit HEVC frame comes out
|
||||||
|
flat); no preview in the viewer for a video (it still says "binary, download it").
|
||||||
|
- **Move… is a folder picker** — **shipped 2026-10-11** (v0.1.0-204, `MovePicker` in
|
||||||
|
`Files.tsx`, `internal/files/move.go`, `POST /api/files/move`). Asked for as: instead of
|
||||||
|
a path prompt, "an interactive file explorer, where I can go back/forward, etc and then
|
||||||
|
click some button, and it will move the file/folder to where I was". The picker opens in
|
||||||
|
the item's own folder and walks the machine the way the tab does — back and forward
|
||||||
|
through where it has been, up, breadcrumbs, a path box, folders only, `.hidden`
|
||||||
|
following the tab's toggle, **+ folder** to make the destination on the spot — and
|
||||||
|
**Move here** puts the item in the folder being looked at. It says why it can't when it
|
||||||
|
can't, with the button greyed: already here, a folder into itself, a name clash in that
|
||||||
|
folder (read from the listing before anything is tried). The row being moved shows in
|
||||||
|
its own folder, greyed. Rename stays a name prompt: it is a different question.
|
||||||
|
- **Across filesystems it copies, then removes.** `os.Rename` is EXDEV between
|
||||||
|
filesystems, and on `grid` every dataset is one — `/tank/media` to `/tank/files` was a
|
||||||
|
"invalid cross-device link" error before this, and the picker would have made that
|
||||||
|
move easy to attempt. `/api/files/move` renames when it can (204) and otherwise runs
|
||||||
|
`hamlab fs move` as a task (202 with the task; the tab shows its output and reloads
|
||||||
|
when it ends). The copy is built under a hidden name beside its destination
|
||||||
|
(`.<name>.hamlab-moving`), renamed into place once complete, and only then is the
|
||||||
|
original removed — a failure part-way removes the partial copy and leaves the original
|
||||||
|
as it was, and a cancel (SIGTERM to the helper) does the same. Modes, mtimes and
|
||||||
|
symlinks are kept; owners too when the backend is root. Devices, pipes and sockets are
|
||||||
|
refused outright rather than half-copied. Progress every 500 files or 512 MiB.
|
||||||
|
Verified on `grid` ext4 → ZFS with a tree holding a 0600 file, a symlink and an old
|
||||||
|
mtime, all intact on the other side.
|
||||||
|
- **Video and audio play in the viewer** — **shipped 2026-10-11** (v0.1.0-208). Asked for
|
||||||
|
as "keep it simple for now, and just send whatever codec the file is in to the client,
|
||||||
|
and let the client try to decode and watch it" — and that is the whole design. Opening a
|
||||||
|
video or audio file shows a `<video>`/`<audio>` with the browser's own controls, fed the
|
||||||
|
file as it is by `/api/files/read`; nothing is transcoded or remuxed, so what plays is
|
||||||
|
what the browser decodes (Safari: h264/hevc/aac in mp4/mov; Chrome: those plus vp9/av1 in
|
||||||
|
mkv/webm, hevc only where the machine has a hardware decoder; nothing plays MPEG-TS or a
|
||||||
|
GoPro `.360`'s raw EAC strip). When the element errors the viewer says so and leaves the
|
||||||
|
download. What the server gained is the part a player cannot do without: reads of a
|
||||||
|
real file with a known size go through `http.ServeContent`, so **Range requests** (every
|
||||||
|
seek, and Safari's first request) and HEAD work, and a download can resume. The root
|
||||||
|
backend's helper takes `--offset` for that, and `files.Seeker` opens the backend once, at
|
||||||
|
the range asked for, rather than streaming from the top and skipping; seeking costs
|
||||||
|
nothing until a read wants somewhere else. `/proc` files, which stat at size 0, keep the
|
||||||
|
plain streaming path, as does the editor's capped read. Audio became a preview kind at
|
||||||
|
the same time (`thumbs.Kind` → `audio`): its cover art, where it has any, is the row's
|
||||||
|
thumbnail, with a ♪ badge.
|
||||||
|
- *Not done, on purpose:* no playlist or next/previous, no subtitles. (Transcoding came
|
||||||
|
the same evening after all — the next entry — because Safari turned out not to play
|
||||||
|
the gaming captures.)
|
||||||
|
- **Re-encoding for the browser, on the machine's GPU** — **shipped 2026-10-11**
|
||||||
|
(v0.1.0-212, `internal/media/{encode,hls,ffrun,inspect}.go`, `internal/api/play.go`,
|
||||||
|
`Player.tsx`, `hls.js`). Asked for when Safari would not play the gaming captures:
|
||||||
|
"add GPU transcoding to a supported codec when the client browser can't play it. Each
|
||||||
|
machine should use its own GPU (if available) and transcode to probably H.264. Maybe we
|
||||||
|
need a quality selector, and a codec selector, up to you." What Safari actually refuses
|
||||||
|
is the **MKV container** — the captures checked are H.264 High 4:2:0, 18–36 Mbps — so
|
||||||
|
the picker has a "repackaged" choice beside the codecs, and a 36 Mbps capture over
|
||||||
|
WireGuard is the other reason to re-encode at all.
|
||||||
|
- **The viewer tries the file as it is first**, as before. When the browser gives up, or
|
||||||
|
on **Re-encode…** in the bar, a picker: **codec** — H.264, HEVC, AV1, or the original
|
||||||
|
codec repackaged — each labelled with the engine this machine would use (or greyed as
|
||||||
|
"not on this machine"), and **size** — original, 1080p, 720p, 480p; the choice is
|
||||||
|
remembered. Play starts the re-encode and the player follows it: "h264 High 2292×960
|
||||||
|
60 fps 36 Mbps in matroska → H.264 2292×960 5.0 Mbps on Intel Arc A380 (vaapi) · 7.3× ·
|
||||||
|
34% through — seek within what's done", then "all of it is ready". **Change…** drops
|
||||||
|
the stream and goes back to the picker.
|
||||||
|
- **HLS, made on the machine in the switcher.** `POST /api/files/play` inspects the file
|
||||||
|
(ffprobe, as root in root mode) and runs ffmpeg into an *event* playlist of 4 s
|
||||||
|
fragmented-MP4 segments under `<data dir>/transcode/<id>/`, answering once the first
|
||||||
|
segment exists (0.5–1 s on `grid`); the player reads the playlist and segments from
|
||||||
|
`/api/files/play/<id>/…` (only those three names are served), Safari natively and
|
||||||
|
everything else through hls.js, loaded only then. ffmpeg runs ahead at the engine's
|
||||||
|
pace, so the seekable range grows and the whole file is through long before it is
|
||||||
|
watched. Keyframes are forced every 4 s so segments cut cleanly; sound is AAC stereo
|
||||||
|
unless it already is AAC; HDR is tone-mapped (`tonemap_vaapi`, or zscale/hable on the
|
||||||
|
CPU). Nothing is kept: the same request within ten minutes reuses the stream, the
|
||||||
|
player closes it when it goes (and the "Change…" path does), the sweeper closes what
|
||||||
|
nobody touched for ten minutes, and the folder is emptied when the daemon starts. At
|
||||||
|
most three at once per machine (429 with the reason). Root mode runs ffmpeg as root
|
||||||
|
through `hamlab run-ffmpeg`, which ends when the daemon lets go of its stdin — so a
|
||||||
|
stream cannot outlive the console.
|
||||||
|
- **Engines are probed, never assumed**, the same probe the previews use, now also
|
||||||
|
reading each node's encode profiles from vainfo and ffmpeg's own encoder list. The
|
||||||
|
recipes for a codec are tried best first until one produces a first segment: a card on
|
||||||
|
the bus before the integrated GPU (the A380 before the UHD 750), decoding on the same
|
||||||
|
card so frames never leave it, then decoding on the CPU and uploading (a source the
|
||||||
|
card cannot decode — AV1 on `grid`), then libx264 / libx265 / SVT-AV1. The size is
|
||||||
|
never larger than the source, and the bit rate follows the output size (2/4/8/12/20
|
||||||
|
Mbps for 480p/720p/1080p/1440p/2160p, +25 % at 50 fps and up, ×0.6 for HEVC and AV1).
|
||||||
|
Measured on `grid` with a 2292×960 60 fps capture: H.264 7.3× realtime, HEVC 720p
|
||||||
|
11.7×, AV1 720p 13×, all on the Arc. `core` and `hamtv` re-encode on the CPU until
|
||||||
|
*Allow GPU access* is pressed (libx264 at 9.4× for a 720p→480p clip on `core`); the Mac
|
||||||
|
and `gate` need ffmpeg first, which the previews note offers to install.
|
||||||
|
- *Not done, on purpose:* no restart-from-a-seek beyond what is encoded (the encode is
|
||||||
|
faster than watching; a seek far ahead waits), no bit-rate ladder or adaptive
|
||||||
|
switching, no subtitles, no keeping a finished re-encode for next time.
|
||||||
|
- **Files edge sweep** — **2026-10-11** (v0.1.0-214), asked for as "a full bug sweep of
|
||||||
|
the Files section, every example and edge case". Every handler and both UI files read
|
||||||
|
through with odd inputs in mind, then a battery of ~60 requests against the live hub
|
||||||
|
(`sweep.py`, not kept: listings of files, missing and forbidden paths, NUL bytes, names
|
||||||
|
with spaces/#/%/?/umlauts, links to files, folders and nothing, a pipe, over-size saves,
|
||||||
|
uploads with bad names and clashes, renames and moves onto things, protected paths,
|
||||||
|
thumbnails of empty and mislabelled images, playback of an image and an audio-only mp4,
|
||||||
|
root-mode variants). One real data bug: **a save over the 8 MiB cap truncated the file
|
||||||
|
and kept the first 8 MiB, answering 204** — the body is now read whole first and refused
|
||||||
|
with 413, the file untouched. The rest: a link to a file showed the link's own time; a
|
||||||
|
link to nothing read as an empty 200; a bad codec or size was answered by inspecting
|
||||||
|
the file; a failed stream was handed back to the next identical request; a player
|
||||||
|
leaving mid-start made the recipe loop try the next engine; a root-mode ffmpeg's end
|
||||||
|
could wait forever; a page closing kept its stream for ten minutes (a beacon now
|
||||||
|
releases it at once); the path box took only absolute paths (it resolves like a shell
|
||||||
|
now, in the tab and the Move picker, with a test); a dropped folder became a nameless
|
||||||
|
empty file; the upload toast counted intentions; the viewer called a pipe a binary
|
||||||
|
file. Not bugs, left alone: `rm -rf` of a mount point empties it (the console is
|
||||||
|
root-capable by design); a folder of ten thousand rows is slow (no virtual list).
|
||||||
|
- **Whole-console sweep** — **2026-10-13** (v0.1.0-216), asked for as "a full sweep of
|
||||||
|
Ham Lab again, look for any bugs and fix them". Four nets, then reading: the Go race
|
||||||
|
detector over the whole suite (clean), staticcheck's development build (the release does
|
||||||
|
not know Go 1.26; it found only dead code and one capitalised error string), a
|
||||||
|
garbage-in battery over every route safe to poke — 202 routes, bad identifiers, empty
|
||||||
|
and malformed bodies, with the destructive ones held back — and a headless Chromium pass
|
||||||
|
through all twenty tabs on the hub and on `grid`, collecting exceptions, console errors
|
||||||
|
and failed requests (none). By hand: the auth middleware and sessions, token issue, the
|
||||||
|
hub→node and KVM proxies, every validation regex before a shell, the cron and cron.d
|
||||||
|
writers, config and state atomicity, every WebSocket and SSE loop's exit, the terminal
|
||||||
|
manager, the task manager, and every destructive confirmation in the UI. Fixed, all
|
||||||
|
small: docker inspect/health of a missing container was a 500 (404 now, a daemon that
|
||||||
|
is down 503), `/api/virt/instances/{name}` without Incus was a 502 (409 like its
|
||||||
|
siblings), a ZFS snapshot delete with a bad name was a 500 (400; 404 for one that does
|
||||||
|
not exist), the task list read a task's Running flag outside its lock (a real data race,
|
||||||
|
never seen), the Virtualization export config was written in place rather than
|
||||||
|
atomically, and the dead code. Nothing in auth, the proxies or the shell boundaries.
|
||||||
|
One self-inflicted finding: the battery's skip list left out `POST
|
||||||
|
/api/files/thumbs/gpu-access`, so the sweep pressed *Allow GPU access* on `core` three
|
||||||
|
times over — the drop-in landed and the hub restarted (that is what the headless pass
|
||||||
|
saw as a refused connection). Kept, since it is the fix the tab offers; `core` makes
|
||||||
|
previews and re-encodes on its UHD iGPU now. The handler refuses a second run while one
|
||||||
|
is in flight (409), since the second `systemd-run` only collided with the first's timer.
|
||||||
- **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth.
|
- **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth.
|
||||||
- **Notifications** — push on service failure / SMART warning / UPS event.
|
- **Notifications** — push on service failure / SMART warning / UPS event.
|
||||||
- **Terminal extras** — search, copy-mode, font/theme, split panes, upload via
|
- **Terminal extras** — search, copy-mode, font/theme, split panes, upload via
|
||||||
@@ -709,12 +910,12 @@ what it has moved.
|
|||||||
|
|
||||||
## To do
|
## To do
|
||||||
|
|
||||||
- **Grow the Files tab until copyparty can go** — asked for 2026-09-23 as two items (Files
|
- **Grow the Files tab** — asked for 2026-09-23 as two items (Files polish, and deprecating
|
||||||
polish, and deprecating copyparty), merged 2026-09-24: they are the same tab and the same
|
copyparty), merged 2026-09-24. **copyparty was retired 2026-10-11** at Mitchell's call,
|
||||||
work. `copyparty` on `grid` (`copyparty.service`, `:3923`, `/etc/copyparty.conf`, serving
|
ahead of these gaps rather than after them: service, files and NPM host all gone, with a
|
||||||
all of `/tank` as user `core` with one account behind NPM) overlaps **Files** almost
|
rollback archive kept on `grid` (`hamlet`'s `docs/hosts.md` has the detail). Files is now
|
||||||
entirely. The aim is to stop the service and keep only Ham Lab, with general ease-of-use
|
the only file browser on `grid`, so the list below is no longer parity — it is the polish
|
||||||
polish along the way. What Files must gain first, in the order that matters:
|
still owed, in the order that matters:
|
||||||
- **Multi-select and bulk actions.** copyparty selects many rows and moves, deletes or
|
- **Multi-select and bulk actions.** copyparty selects many rows and moves, deletes or
|
||||||
downloads them together; Files acts on one row at a time.
|
downloads them together; Files acts on one row at a time.
|
||||||
- **Download a folder**, and a multi-file download, as a streamed zip or tar — the thing
|
- **Download a folder**, and a multi-file download, as a streamed zip or tar — the thing
|
||||||
@@ -722,17 +923,14 @@ what it has moved.
|
|||||||
- **Resumable uploads.** copyparty's up2k chunks a file, survives a dropped connection
|
- **Resumable uploads.** copyparty's up2k chunks a file, survives a dropped connection
|
||||||
and skips what is already there; Files streams one PUT, so a phone upload that drops at
|
and skips what is already there; Files streams one PUT, so a phone upload that drops at
|
||||||
90% starts over.
|
90% starts over.
|
||||||
- **A gallery.** Thumbnails for a folder of images and video, and play audio/video in
|
- **A gallery.** ~~Thumbnails for a folder of images and video~~ (the **previews** toggle,
|
||||||
place — `/tank/media` and `/tank/immich` are most of what is browsed.
|
2026-10-11 — see the shipped entry above; a grid view is not built), and play
|
||||||
|
audio/video in place — `/tank/media` and `/tank/immich` are most of what is browsed.
|
||||||
- **Search** below the current folder by name, since `find` in a terminal is the fallback
|
- **Search** below the current folder by name, since `find` in a terminal is the fallback
|
||||||
today.
|
today.
|
||||||
- *Decide when built:* whether share links (copyparty's public per-path links) are wanted
|
- *Decide when built:* whether share links (copyparty's public per-path links) are wanted
|
||||||
at all — they are the one feature that hands out access without a login, and Ham Lab has
|
at all — they are the one feature that hands out access without a login, and Ham Lab has
|
||||||
no notion of that; if not, say so in the docs rather than leaving a gap unexplained.
|
no notion of that; if not, say so in the docs rather than leaving a gap unexplained.
|
||||||
- *After:* `systemctl disable --now copyparty`, remove its NPM host, keep
|
|
||||||
`/etc/copyparty.conf` until a rollback is clearly not needed, then update
|
|
||||||
`hamlet`'s `docs/hosts.md` bare-metal list, `docs/gotchas.md` (the "`docker ps` is not
|
|
||||||
the inventory" note names it) and the `grid` row in `CLAUDE.md`.
|
|
||||||
|
|
||||||
- **Grow the Docker tab until Dockhand can go** — asked for 2026-09-23 and widened
|
- **Grow the Docker tab until Dockhand can go** — asked for 2026-09-23 and widened
|
||||||
2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the
|
2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the
|
||||||
@@ -780,12 +978,13 @@ what it has moved.
|
|||||||
stacks that are merely down (`pihole_default`, `transmission_default`, `gluetun_default`
|
stacks that are merely down (`pihole_default`, `transmission_default`, `gluetun_default`
|
||||||
among them), and pruning would take them and hand them new subnets on the next deploy.
|
among them), and pruning would take them and hand them new subnets on the next deploy.
|
||||||
|
|
||||||
- **The `*:2375` question, which this is the moment to settle.** A stack deployed to
|
- **The `*:2375` question — settled, and the code removed 2026-10-10.** The hub asks the
|
||||||
`grid` or the Mac still goes over `tcp://<node>:2375` from the hub. Decide whether the
|
node's own Ham Lab; nothing probes or uses a node's Docker port. `hubStacks`,
|
||||||
hub should instead ask the node's own Ham Lab to deploy locally, which would end the
|
`nodeDocker`, `taskOnHub`, the hub-side `/api/nodes/{node}/docker/…` routes and the
|
||||||
unauthenticated Docker socket exposure — its own item in `docs/pending.md`, currently
|
`remote` / `held_by` fields are gone. A schedule saved before this that still carries a
|
||||||
filed under "decided, no action", and this is the change that would make closing it
|
`remote` is refused with a message, never run against the wrong Docker. `PORTABILITY.md`
|
||||||
free.
|
§3 has the audit it came from. Deployed as v0.1.0-191 and checked on all five machines
|
||||||
|
through the hub.
|
||||||
|
|
||||||
- **A Virtualization tab** — asked for 2026-09-23; **scoped, built and deployed 2026-10-09 — see
|
- **A Virtualization tab** — asked for 2026-09-23; **scoped, built and deployed 2026-10-09 — see
|
||||||
`VIRTUALIZATION.md`** for what shipped (phases 1 and 2) and what is left. Incus 6.0 LTS as the engine; the tab detects per machine whether Incus is
|
`VIRTUALIZATION.md`** for what shipped (phases 1 and 2) and what is left. Incus 6.0 LTS as the engine; the tab detects per machine whether Incus is
|
||||||
|
|||||||
+8
-8
@@ -44,11 +44,11 @@ default or phones home to this lab · **C** cosmetic (placeholder text, comments
|
|||||||
|
|
||||||
| # | Sev | Where | Assumes | General form |
|
| # | Sev | Where | Assumes | General form |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| 3.1 | **A** | `internal/api/stacks.go:483-509` `nodeDocker()` | For every node whose stacks aren't editable locally, the hub **probes `tcp://<node-host>:2375`** with `docker -H … version` and, if it answers, deploys to the node over an unauthenticated Docker socket. This is the Dockhand-era layout; on any other network an open 2375 is either absent or a security problem the console should not quietly use. | Remove the probe. The node's own Ham Lab deploys its stacks (the files live on the node, under its data dir). `PLAN.md` "To do" already names this decision. |
|
| 3.1 | **done** 2026-10-10 | `internal/api/stacks.go` `nodeDocker()` (removed) | For every node whose stacks aren't editable locally, the hub **probes `tcp://<node-host>:2375`** with `docker -H … version` and, if it answers, deploys to the node over an unauthenticated Docker socket. This is the Dockhand-era layout; on any other network an open 2375 is either absent or a security problem the console should not quietly use. | Remove the probe. The node's own Ham Lab deploys its stacks (the files live on the node, under its data dir). `PLAN.md` "To do" already names this decision. |
|
||||||
| 3.2 | **A** | `internal/api/stacks.go:439-479` `hubStacks()`, `:1036-1074` `homesFor()`, `web/src/Docker.tsx` "kept on `<hub>`, deployed to this machine over `tcp://…`" | A hub can hold compose files for a node and run compose remotely; new stacks for a node may be created **on the hub**. Only true here because Dockhand on `core` deployed `grid` and the Mac. | Same as 3.1: one machine, one set of files, one Docker. Hub-held stacks become an import path (copy to the node) rather than a steady state. |
|
| 3.2 | **done** 2026-10-10 | `hubStacks()`, `taskOnHub()`, the hub-side `/api/nodes/{node}/docker/…` routes, `Stack.remote`/`held_by` (all removed; the generic node proxy carries those requests) | A hub can hold compose files for a node and run compose remotely; new stacks for a node may be created **on the hub**. Only true here because Dockhand on `core` deployed `grid` and the Mac. | Same as 3.1: one machine, one set of files, one Docker. Hub-held stacks become an import path (copy to the node) rather than a steady state. |
|
||||||
| 3.3 | **B** | `internal/api/stacks.go:290-382` `homesOf()`; `:373-376` fallback `~/stacks` | Where new stacks go is **discovered** from where ≥2 stacks already are (Dockhand's `stacks/<machine>` habit), else `~/stacks`. Neither is Ham Lab's data directory, which the project's own rule says everything managed must live under. | Default home = `<DataDir>/stacks`. Discovered homes stay as *additional* places, never the default. |
|
| 3.3 | **done** 2026-10-10 | `internal/api/stacks.go` `homesOf()` — `<DataDir>/stacks` is always offered first and default; the `~/stacks` fallback is gone; `TestHomesDefaultToDataDir` | Where new stacks go is **discovered** from where ≥2 stacks already are (Dockhand's `stacks/<machine>` habit), else `~/stacks`. Neither is Ham Lab's data directory, which the project's own rule says everything managed must live under. | Default home = `<DataDir>/stacks`. Discovered homes stay as *additional* places, never the default. |
|
||||||
| 3.4 | **B** | `internal/api/stacks.go:100-203` remembered mounts, `internal/api/dockersched.go:772-795` `repoint()` | Machinery exists to find compose files **inside a removed orchestrator container's filesystem**. Generic in form, but it exists because of Dockhand and is exercised only here. | Keep (it is general), but it is not a reason to keep 3.1/3.2. |
|
| 3.4 | **B** | `internal/api/stacks.go:100-203` remembered mounts, `internal/api/dockersched.go:772-795` `repoint()` | Machinery exists to find compose files **inside a removed orchestrator container's filesystem**. Generic in form, but it exists because of Dockhand and is exercised only here. | Keep (it is general), but it is not a reason to keep 3.1/3.2. |
|
||||||
| 3.5 | **B** | `internal/api/stacks.go:801` | Inside a `Via` container the command is literally `docker compose`, even when `dockerTool()` is `podman`. | Use the detected tool. |
|
| 3.5 | — | `internal/api/stacks.go` `stackAction`, `internal/compose/compose.go` | Inside a `Via` container the command is literally `docker compose`. | **Checked 2026-10-10, no change:** the CLI inside an orchestrator's container is that container's own `docker`, independent of what the host runs; the host's tool only chooses the `exec`. Both places now say so in a comment. |
|
||||||
| 3.6 | **B** | `internal/api/stacks.go:908` `docker buildx imagetools inspect` | Update checks need the **buildx** plugin. Docker CE on Debian has it; Podman and minimal Docker installs do not → every image reports "the registry didn't answer". | Fall back to `skopeo inspect` or a direct registry `HEAD` for the digest. |
|
| 3.6 | **B** | `internal/api/stacks.go:908` `docker buildx imagetools inspect` | Update checks need the **buildx** plugin. Docker CE on Debian has it; Podman and minimal Docker installs do not → every image reports "the registry didn't answer". | Fall back to `skopeo inspect` or a direct registry `HEAD` for the digest. |
|
||||||
| 3.7 | **B** | `Makefile:445-448` `deploy` | The machine you build on **is** the hub: installs locally, then `push-nodes`. | Fine for a developer; document that `deploy` means "I am the hub". A plain `make install` target for a single box. |
|
| 3.7 | **B** | `Makefile:445-448` `deploy` | The machine you build on **is** the hub: installs locally, then `push-nodes`. | Fine for a developer; document that `deploy` means "I am the hub". A plain `make install` target for a single box. |
|
||||||
| 3.8 | **B** | `internal/nodes/nodes.go:316-323` | A hub pushes other architectures only if `<prefix>/hamlab-<arch>` siblings exist, which only `make deploy`/`install.sh`-from-dist put there. | A hub without siblings should fetch the right build from `update.url` instead of erroring. |
|
| 3.8 | **B** | `internal/nodes/nodes.go:316-323` | A hub pushes other architectures only if `<prefix>/hamlab-<arch>` siblings exist, which only `make deploy`/`install.sh`-from-dist put there. | A hub without siblings should fetch the right build from `update.url` instead of erroring. |
|
||||||
@@ -106,7 +106,7 @@ The installer supports systemd, procd and OpenRC (`internal/setup/setup.go:217-2
|
|||||||
|
|
||||||
| # | Sev | Where | Assumes | General form |
|
| # | Sev | Where | Assumes | General form |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| 7.1 | **A** | `internal/priv/priv.go`; every tab that calls `priv.Available()`; `web/src/Updates.tsx:2144` "give its user passwordless sudo or doas"; `web/src/Jobs.tsx:1883` | Full function needs the service user to have **unrestricted passwordless sudo**, which `hamlet/CLAUDE.md` grants on every host here. The installer neither creates nor offers a sudoers rule, so on a fresh box most tabs are read-only and the UI tells the user to go fix sudoers by hand. | `hamlab install --sudo` writes a **scoped** `/etc/sudoers.d/hamlab` naming the commands Ham Lab runs (smartctl, zpool, zfs, systemctl, docker, journalctl, tee for its cron files, `hamlab fs|space-walk|drm-clients|replace`), or installs as root. Either way the console works out of the box. |
|
| 7.1 | **A** | `internal/priv/priv.go`; every tab that calls `priv.Available()`; `web/src/Updates.tsx:2144` "give its user passwordless sudo or doas"; `web/src/Jobs.tsx:1883` | Full function needs the service user to have **unrestricted passwordless sudo**, which `hamlet/CLAUDE.md` grants on every host here. The installer neither creates nor offers a sudoers rule, so on a fresh box most tabs are read-only and the UI tells the user to go fix sudoers by hand. | `hamlab install --sudo` writes a **scoped** `/etc/sudoers.d/hamlab` naming the commands Ham Lab runs (smartctl, zpool, zfs, systemctl, docker, journalctl, tee for its cron files, `hamlab fs|space-walk|drm-clients|cpu-energy|replace`), or installs as root. Either way the console works out of the box. |
|
||||||
| 7.2 | **A** | `internal/api/server.go:1069-1080` `clientKey()`, `:1083-1092`, `:431-433` `secure()` | The **first `X-Forwarded-For` hop is trusted unconditionally**, because here Ham Lab always sits behind Nginx Proxy Manager. On a host exposed directly, a client sets the header and defeats the per-client login rate limiter and the session IP check. | Trust `X-Forwarded-*` only from configured proxy addresses (`trusted_proxies` in config), else use the socket peer. |
|
| 7.2 | **A** | `internal/api/server.go:1069-1080` `clientKey()`, `:1083-1092`, `:431-433` `secure()` | The **first `X-Forwarded-For` hop is trusted unconditionally**, because here Ham Lab always sits behind Nginx Proxy Manager. On a host exposed directly, a client sets the header and defeats the per-client login rate limiter and the session IP check. | Trust `X-Forwarded-*` only from configured proxy addresses (`trusted_proxies` in config), else use the socket peer. |
|
||||||
| 7.3 | **B** | `internal/api/server.go:1324` `X-Accel-Buffering: no` "nginx (NPM)" | Harmless header; comment names the lab's proxy. | Keep header; generalise comment. |
|
| 7.3 | **B** | `internal/api/server.go:1324` `X-Accel-Buffering: no` "nginx (NPM)" | Harmless header; comment names the lab's proxy. | Keep header; generalise comment. |
|
||||||
| 7.4 | **B** | `internal/api/guests.go:1-9`, `PLAN.md:21` | Guest logins are full-power ("he gives these to people he trusts"). A product default. | Keep, but it is a posture decision taken for this household; a read-only guest is the generic expectation and was cut. Document as a deliberate choice in README. |
|
| 7.4 | **B** | `internal/api/guests.go:1-9`, `PLAN.md:21` | Guest logins are full-power ("he gives these to people he trusts"). A product default. | Keep, but it is a posture decision taken for this household; a read-only guest is the generic expectation and was cut. Document as a deliberate choice in README. |
|
||||||
@@ -125,7 +125,7 @@ The installer supports systemd, procd and OpenRC (`internal/setup/setup.go:217-2
|
|||||||
| # | Sev | Where | Assumes | General form |
|
| # | Sev | Where | Assumes | General form |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| 9.1 | **B** | `internal/config/config.go:115-117` `Candidates()`; `internal/setup/setup.go:413-422`; `README.md:38` | `~/apps/hamlab/config.json` is searched as a legacy location — the lab's pre-v0.2 path, kept so `core`, `grid` and the Mac keep working. | Keep for one more release with a one-time migration log line, then drop. A stranger never had it. |
|
| 9.1 | **B** | `internal/config/config.go:115-117` `Candidates()`; `internal/setup/setup.go:413-422`; `README.md:38` | `~/apps/hamlab/config.json` is searched as a legacy location — the lab's pre-v0.2 path, kept so `core`, `grid` and the Mac keep working. | Keep for one more release with a one-time migration log line, then drop. A stranger never had it. |
|
||||||
| 9.2 | **B** | 3.3 above; `PLAN.md:24` "`grid`'s and the Mac's stacks at `~/stacks/<machine>/` are the standing exception" | The "everything under DataDir" rule has a lab-shaped exception baked into the stacks code. | Default to DataDir; treat `~/stacks` as a discovered home like any other. |
|
| 9.2 | **done** 2026-10-10 | 3.3 above; `PLAN.md:24` "`grid`'s and the Mac's stacks at `~/stacks/<machine>/` are the standing exception" | The "everything under DataDir" rule has a lab-shaped exception baked into the stacks code. | Default to DataDir; treat `~/stacks` as a discovered home like any other. |
|
||||||
| 9.3 | **C** | `internal/setup/setup.go:351` `ownFiles` | Correctly scoped; note only. | None. |
|
| 9.3 | **C** | `internal/setup/setup.go:351` `ownFiles` | Correctly scoped; note only. | None. |
|
||||||
|
|
||||||
## 10. Build and release
|
## 10. Build and release
|
||||||
@@ -157,8 +157,8 @@ Listed so the sweep is known to be complete, not because they need work.
|
|||||||
|
|
||||||
## Suggested order of work
|
## Suggested order of work
|
||||||
|
|
||||||
1. **Posture and safety first (A):** 7.2 trusted proxies; 7.1 installer-written scoped sudoers; 3.1 drop the `:2375` probe (with 3.2 as its consequence).
|
1. **Posture and safety first (A):** 7.2 trusted proxies; 7.1 installer-written scoped sudoers; ~~3.1 drop the `:2375` probe (with 3.2 as its consequence)~~ done 2026-10-10.
|
||||||
2. **"Any Linux" promises the README already makes (A):** 4.1 power, 4.4/4.5 cron on busybox and timer-only hosts, 4.2 Incus install on non-systemd, 4.3 a Services backend for OpenRC/procd, 2.10 the `tank` regex.
|
2. **"Any Linux" promises the README already makes (A):** 4.1 power, 4.4/4.5 cron on busybox and timer-only hosts, 4.2 Incus install on non-systemd, 4.3 a Services backend for OpenRC/procd, 2.10 the `tank` regex.
|
||||||
3. **Lab defaults that phone home or mislead (B):** 1.1–1.3 update/install URLs, 2.9 schedule defaults, 6.9/6.10 speed-test sources and an off switch, 8.1 memory limit, 3.3/9.2 stack home = DataDir.
|
3. **Lab defaults that phone home or mislead (B):** 1.1–1.3 update/install URLs, 2.9 schedule defaults, 6.9/6.10 speed-test sources and an off switch, 8.1 memory limit, ~~3.3/9.2 stack home = DataDir~~ done 2026-10-10.
|
||||||
4. **Breadth (B):** 5.1 zypper for Incus, 5.5 cgroup v1, 6.1/6.2 GPUs beyond Intel/AMD, 6.4 UPS order, 6.11/6.12 wg-quick/NetworkManager, 6.15 IPv6.
|
4. **Breadth (B):** 5.1 zypper for Incus, 5.5 cgroup v1, 6.1/6.2 GPUs beyond Intel/AMD, 6.4 UPS order, 6.11/6.12 wg-quick/NetworkManager, 6.15 IPv6.
|
||||||
5. **Cosmetics (C):** every placeholder and comment in §1.10 and §2, then split lab facts out of `PLAN.md`, `VIRTUALIZATION.md`, `SENSORS.md`, `README.md`.
|
5. **Cosmetics (C):** every placeholder and comment in §1.10 and §2, then split lab facts out of `PLAN.md`, `VIRTUALIZATION.md`, `SENSORS.md`, `README.md`.
|
||||||
|
|||||||
+6
-1
@@ -12,7 +12,7 @@ added to the registry in `api.New`.
|
|||||||
## IDs
|
## IDs
|
||||||
|
|
||||||
```
|
```
|
||||||
cpu.usage cpu.core.<n>.usage cpu.temp load.1|5|15 uptime
|
cpu.usage cpu.core.<n>.usage cpu.temp cpu.power load.1|5|15 uptime
|
||||||
mem.total|used|avail|free|cached|buffers|used_pct swap.total|used|avail|used_pct
|
mem.total|used|avail|free|cached|buffers|used_pct swap.total|used|avail|used_pct
|
||||||
temp.<chip>.<label> fan.<chip>.<label>
|
temp.<chip>.<label> fan.<chip>.<label>
|
||||||
net.<iface>.rx|tx|rx_total|tx_total|up
|
net.<iface>.rx|tx|rx_total|tx_total|up
|
||||||
@@ -35,6 +35,11 @@ zfs.arc.size|target|min|max|mru|mfu|data|metadata|hit_pct|reads|misses|lifetime_
|
|||||||
has run on that machine, and each one's `t` is when it ran — so a widget or an alert on
|
has run on that machine, and each one's `t` is when it ran — so a widget or an alert on
|
||||||
them is reading a measurement from that moment, not a live value. Nothing takes a speed
|
them is reading a measurement from that moment, not a live value. Nothing takes a speed
|
||||||
test on its own; something has to press the button.
|
test on its own; something has to press the button.
|
||||||
|
- `cpu.power` is the CPU packages' own power in watts, from the kernel's RAPL energy
|
||||||
|
counters (Intel, and AMD Zen), summed over every package: cores, built-in graphics and
|
||||||
|
memory controller, not the whole machine. The counters are root-only, so a daemon that is
|
||||||
|
not root reads them through a helper started once with sudo or doas. It is absent on a
|
||||||
|
machine without the counters (ARM, the Mac mini) or without a way to root.
|
||||||
- `gpu.<name>.*` is each graphics device, named as lspci names it (`gpu.intel_arc_a380`;
|
- `gpu.<name>.*` is each graphics device, named as lspci names it (`gpu.intel_arc_a380`;
|
||||||
labels carry `driver` and the PCI `bus`). `usage` is how busy the device was — from the
|
labels carry `driver` and the PCI `bus`). `usage` is how busy the device was — from the
|
||||||
GT's idle residency on Intel, `gpu_busy_percent` on amdgpu — and is absent on a driver
|
GT's idle residency on Intel, `gpu_busy_percent` on amdgpu — and is absent on a driver
|
||||||
|
|||||||
+24
-21
@@ -1,20 +1,17 @@
|
|||||||
# Virtualization tab — plan (2026-10-09; phase 1 built the same day)
|
# Virtualization tab — plan (2026-10-09; phase 1 built the same day)
|
||||||
|
|
||||||
**Status:** phases 1 and 2 are **deployed to all five machines as v0.1.0-188** (2026-10-09).
|
**Status:** phases 1 and 2 are **deployed to all five machines as v0.1.0-196**. **No machine
|
||||||
Incus runs on `grid` (installed from the tab); the other machines show Install or, on `gate`,
|
has Incus installed as of 2026-10-11**: `grid`'s install of 2026-10-09 was removed at
|
||||||
unavailable. Phase 2 verified live on `grid` through the hub: ISO fetched by URL with its
|
Mitchell's request so he can install it himself from the tab and ask questions along the way
|
||||||
sha256 checked, **New VM from ISO** booted Alpine with Secure Boot off, **the VGA console
|
(`hamlet/docs/hosts.md` has what was removed and what the two days established). The tab
|
||||||
drew it in a real browser** (headless Chromium, 1280×800 canvas, keyboard and mouse live),
|
shows Install on `grid`, `core`, the Mac mini and `hamtv`, unavailable on `gate`. Both create
|
||||||
a virtiofs mount appeared in a running VM, an export tarball landed in `virt/exports/` and the
|
forms carry a **Disk location** picker with add/remove of locations; instance storage as
|
||||||
nightly cron line is written, `instance.*` sensors flow and the "VM or container down" alert
|
plain files on the machine's own disk is what he wants on `grid`. Everything in phase 2 was
|
||||||
default is in. Two things found and fixed on the way: spice-html5 wants the `binary`
|
verified live on `grid` before the removal: ISO fetch with checksum, New VM from ISO, the VGA
|
||||||
websocket subprotocol agreed; and it opens one websocket per SPICE channel, so the bridge
|
console in a real browser (headless Chromium; `binary` subprotocol, one console operation per
|
||||||
holds **one console operation per VM** and gives each channel its own data websocket on it
|
VM with a data websocket per SPICE channel), a virtiofs mount into a running VM, exports and
|
||||||
(Incus refuses a second operation as "already connected"). **Left from phase 2:** the LAN
|
their cron line, `instance.*` sensors and the alert default. **Left:** the LAN bridge step
|
||||||
bridge step (`br0` on the uplink — needs a live test on a machine that can afford to drop
|
(his call), cloud-init presets beyond "Use my SSH keys", a backup line for `virt/exports/`.
|
||||||
off the network for a moment, which is a decision, not a build), cloud-init presets beyond
|
|
||||||
"Use my SSH keys", and a backup line for `grid`'s `virt/exports/` (see `hamlet/docs/backups.md`).
|
|
||||||
`t-alpine` on `grid` is the live example; delete it from the tab when done.
|
|
||||||
|
|
||||||
A tab for creating, watching, consoling into and deleting **LXC containers and KVM virtual
|
A tab for creating, watching, consoling into and deleting **LXC containers and KVM virtual
|
||||||
machines**, the way Proxmox does with `pct` and `qm` — **per machine, like every other
|
machines**, the way Proxmox does with `pct` and `qm` — **per machine, like every other
|
||||||
@@ -236,12 +233,18 @@ like everything else.
|
|||||||
|
|
||||||
## Layout decisions Ham Lab makes per machine (overridable in the Install dialog)
|
## Layout decisions Ham Lab makes per machine (overridable in the Install dialog)
|
||||||
|
|
||||||
1. **Storage.** The instance pool's backing lives in Ham Lab's folder: Install creates a
|
1. **Storage — revised 2026-10-11.** The create forms carry a **Disk location** picker: a
|
||||||
sparse `virt/pool.img` and hands it to Incus as the pool's `source` (zfs or btrfs by the
|
list of the machine's Incus storage pools, each described by where its bytes land, with
|
||||||
detection above), or `virt/pool/` as a `dir` pool where neither is available. Sized from
|
"+ add a location" (any folder on the machine becomes a plain-files pool). On `grid` the
|
||||||
the disk's free space, shown and editable. Bulk data is **bind-mounted in**, not stored in
|
one location is `local`: plain files under `virt/pool/` on the root NVMe, each VM's disk
|
||||||
the pool. The driver, path and size are recorded in `virt.json` so the page can say where
|
its own sparse file sized on its own, each container a directory, nothing pre-sized.
|
||||||
the data is and Install can rebuild the pool from the file after a reinstall.
|
*The first design* (a ZFS pool inside one sparse file, handed to Incus as the pool's
|
||||||
|
source, for instant snapshots and clones) was built, ran for two days, and was taken out
|
||||||
|
at Mitchell's instruction: one pre-sized file holding every disk is an opaque middle
|
||||||
|
layer with a number on it that reads as a reservation, and "zfs · hamlab-virt" on the page
|
||||||
|
read as the NAS pool. The Install dialog still offers zfs/btrfs-in-a-file for a machine
|
||||||
|
that wants instant snapshots; the labels now say exactly what each is. Bulk data is still
|
||||||
|
**bind-mounted in**, not stored in the pool.
|
||||||
2. **Bridge subnet, computed.** Incus's `auto` picks a random `10.x.y.0/24`, and **this
|
2. **Bridge subnet, computed.** Incus's `auto` picks a random `10.x.y.0/24`, and **this
|
||||||
LAN is `10.0.0.0/8`** — every host has a /8 on its NIC, so any 10.x bridge collides.
|
LAN is `10.0.0.0/8`** — every host has a /8 on its NIC, so any 10.x bridge collides.
|
||||||
Ham Lab reads the machine's routes and picks the first free /24 from `192.168.0.0/16`,
|
Ham Lab reads the machine's routes and picks the first free /24 from `192.168.0.0/16`,
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ import (
|
|||||||
"hamlab/internal/cron"
|
"hamlab/internal/cron"
|
||||||
"hamlab/internal/files"
|
"hamlab/internal/files"
|
||||||
"hamlab/internal/incus"
|
"hamlab/internal/incus"
|
||||||
|
"hamlab/internal/media"
|
||||||
"hamlab/internal/metrics"
|
"hamlab/internal/metrics"
|
||||||
"hamlab/internal/nodes"
|
"hamlab/internal/nodes"
|
||||||
"hamlab/internal/priv"
|
"hamlab/internal/priv"
|
||||||
@@ -82,10 +83,16 @@ func main() {
|
|||||||
switch cmd { // commands that don't read the config
|
switch cmd { // commands that don't read the config
|
||||||
case "fs": // privileged helper for the Files tab (run as root by the daemon)
|
case "fs": // privileged helper for the Files tab (run as root by the daemon)
|
||||||
os.Exit(files.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
os.Exit(files.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||||
|
case "thumb": // privileged helper for the Files tab's previews in root mode (run as root by the daemon)
|
||||||
|
os.Exit(media.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||||
|
case "run-ffmpeg": // privileged helper for playback in root mode: ffmpeg as root, ending when stdin closes
|
||||||
|
os.Exit(media.RunFFmpegHelper(flag.Args()[1:], os.Stdin, os.Stderr))
|
||||||
case "space-walk": // helper for the Space tab: measure mountpoints, stream the tree back
|
case "space-walk": // helper for the Space tab: measure mountpoints, stream the tree back
|
||||||
os.Exit(space.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
os.Exit(space.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
|
||||||
case "drm-clients": // privileged helper for the GPU readings: who has the GPU open (run as root by the daemon)
|
case "drm-clients": // privileged helper for the GPU readings: who has the GPU open (run as root by the daemon)
|
||||||
os.Exit(metrics.DRMClientsHelper(os.Stdin, os.Stdout))
|
os.Exit(metrics.DRMClientsHelper(os.Stdin, os.Stdout))
|
||||||
|
case "cpu-energy": // privileged helper for the CPU power reading: the packages' energy counters (run as root by the daemon)
|
||||||
|
os.Exit(metrics.CPUEnergyHelper(os.Stdin, os.Stdout))
|
||||||
case "install", "uninstall":
|
case "install", "uninstall":
|
||||||
var err error
|
var err error
|
||||||
if cmd == "install" {
|
if cmd == "install" {
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ go 1.26.0
|
|||||||
require (
|
require (
|
||||||
github.com/coder/websocket v1.8.15
|
github.com/coder/websocket v1.8.15
|
||||||
github.com/creack/pty v1.1.24
|
github.com/creack/pty v1.1.24
|
||||||
|
go.etcd.io/bbolt v1.5.0
|
||||||
golang.org/x/crypto v0.57.0
|
golang.org/x/crypto v0.57.0
|
||||||
golang.org/x/term v0.46.0
|
golang.org/x/term v0.46.0
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -2,9 +2,21 @@ github.com/coder/websocket v1.8.15 h1:6B2JPeOGlpff2Uz6vOEH1Vzpi0iUz20A+lPVhPHtNU
|
|||||||
github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
|
github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
|
||||||
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
|
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
|
||||||
github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
|
github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
|
||||||
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
|
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||||
|
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||||
|
go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
|
||||||
|
go.etcd.io/bbolt v1.5.0/go.mod h1:mkltfYE5aUHQxUct9N9V+Kp7aSjFqjgrhcXIS70Lrdk=
|
||||||
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
|
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
|
||||||
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
|
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
|
||||||
|
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
|
||||||
|
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||||
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
|
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
|
||||||
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||||
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
|
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
|
||||||
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
|
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
|
|||||||
+22
-3
@@ -343,6 +343,21 @@ func (s *Server) dockerLogs(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// dockerInspect is the parts of `docker inspect` worth reading on a phone.
|
// dockerInspect is the parts of `docker inspect` worth reading on a phone.
|
||||||
|
// dockerErr answers for a docker command that failed: a thing that isn't there is a 404,
|
||||||
|
// the daemon being away a 503, anything else a 500.
|
||||||
|
func dockerErr(w http.ResponseWriter, err error) {
|
||||||
|
msg := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(err.Error()), "error: "))
|
||||||
|
low := strings.ToLower(msg)
|
||||||
|
switch {
|
||||||
|
case strings.Contains(low, "no such object") || strings.Contains(low, "no such container") || strings.Contains(low, "not found"):
|
||||||
|
jsonErr(w, 404, msg)
|
||||||
|
case strings.Contains(low, "cannot connect to the docker daemon") || strings.Contains(low, "is the docker daemon running"):
|
||||||
|
jsonErr(w, 503, msg)
|
||||||
|
default:
|
||||||
|
jsonErr(w, 500, msg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
|
||||||
id := r.PathValue("id")
|
id := r.PathValue("id")
|
||||||
if !dockerRef.MatchString(id) {
|
if !dockerRef.MatchString(id) {
|
||||||
@@ -352,9 +367,13 @@ func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
|
|||||||
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
|
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
out, err := dockerRun(ctx, "inspect", id)
|
out, err := dockerRun(ctx, "inspect", id)
|
||||||
|
if err != nil {
|
||||||
|
dockerErr(w, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
var all []map[string]any
|
var all []map[string]any
|
||||||
if err != nil || json.Unmarshal(out, &all) != nil || len(all) == 0 {
|
if json.Unmarshal(out, &all) != nil || len(all) == 0 {
|
||||||
jsonErr(w, 500, fmt.Sprint("inspect failed: ", err))
|
jsonErr(w, 500, "inspect: unreadable answer")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
c := all[0]
|
c := all[0]
|
||||||
@@ -486,7 +505,7 @@ func (s *Server) dockerHealth(w http.ResponseWriter, r *http.Request) {
|
|||||||
defer cancel()
|
defer cancel()
|
||||||
out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id)
|
out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
jsonErr(w, 500, err.Error())
|
dockerErr(w, err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
var st struct {
|
var st struct {
|
||||||
|
|||||||
@@ -55,11 +55,14 @@ type composeSpec struct {
|
|||||||
Project string `json:"project"`
|
Project string `json:"project"`
|
||||||
Files []string `json:"files"`
|
Files []string `json:"files"`
|
||||||
Via string `json:"via,omitempty"`
|
Via string `json:"via,omitempty"`
|
||||||
Remote string `json:"remote,omitempty"`
|
// Remote is read but never written: schedules made before v0.2 could point compose at
|
||||||
|
// another machine's Docker over tcp. Such a job is refused rather than run against this
|
||||||
|
// machine's Docker, which is where it would land without the variable.
|
||||||
|
Remote string `json:"remote,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c composeSpec) toSpec(tool string) compose.Spec {
|
func (c composeSpec) toSpec(tool string) compose.Spec {
|
||||||
return compose.Spec{Project: c.Project, Files: c.Files, Via: c.Via, Remote: c.Remote, Tool: tool}
|
return compose.Spec{Project: c.Project, Files: c.Files, Via: c.Via, Tool: tool}
|
||||||
}
|
}
|
||||||
|
|
||||||
const dockerCronFile = "/etc/cron.d/hamlab-docker"
|
const dockerCronFile = "/etc/cron.d/hamlab-docker"
|
||||||
@@ -169,6 +172,9 @@ func RunUpdate(ctx context.Context, db *store.Store, j UpdateJob, tool string) (
|
|||||||
if len(j.Spec.Files) == 0 {
|
if len(j.Spec.Files) == 0 {
|
||||||
return "", errors.New("this schedule has no compose file recorded — open it in the Docker tab and save it again")
|
return "", errors.New("this schedule has no compose file recorded — open it in the Docker tab and save it again")
|
||||||
}
|
}
|
||||||
|
if j.Spec.Remote != "" {
|
||||||
|
return "", errors.New("this schedule was made to deploy to another machine over tcp, which this console no longer does — make it again on that machine")
|
||||||
|
}
|
||||||
spec := repoint(ctx, db, j.Spec.toSpec(tool))
|
spec := repoint(ctx, db, j.Spec.toSpec(tool))
|
||||||
var log strings.Builder
|
var log strings.Builder
|
||||||
run := func(args ...string) error {
|
run := func(args ...string) error {
|
||||||
@@ -238,7 +244,7 @@ func (s *Server) dsPut(w http.ResponseWriter, r *http.Request) {
|
|||||||
// seenFiles is the list as compose itself sees it: host paths normally, the
|
// seenFiles is the list as compose itself sees it: host paths normally, the
|
||||||
// orchestrator container's own paths when compose runs inside one. Getting this
|
// orchestrator container's own paths when compose runs inside one. Getting this
|
||||||
// backwards gives a cron line that works from the console and fails at 3am.
|
// backwards gives a cron line that works from the console and fails at 3am.
|
||||||
j.Spec = composeSpec{Project: st.Name, Files: seenFiles(st, -1, ""), Via: st.Via, Remote: st.Remote}
|
j.Spec = composeSpec{Project: st.Name, Files: seenFiles(st, -1, ""), Via: st.Via}
|
||||||
if len(j.Spec.Files) == 0 {
|
if len(j.Spec.Files) == 0 {
|
||||||
jsonErr(w, 409, "that stack's compose file isn't anywhere this console can reach")
|
jsonErr(w, 409, "that stack's compose file isn't anywhere this console can reach")
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -0,0 +1,123 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"errors"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/url"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"hamlab/internal/files"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A file read answers Range requests — what a <video> needs to seek — and HEAD, through
|
||||||
|
// the same handler that streams everything else.
|
||||||
|
func TestFilesReadRanges(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
p := filepath.Join(dir, "clip.mp4")
|
||||||
|
data := make([]byte, 10000)
|
||||||
|
for i := range data {
|
||||||
|
data[i] = byte(i % 251)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(p, data, 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s := &Server{fsLocal: files.Local{}}
|
||||||
|
get := func(method, extra string, hdr map[string]string) *httptest.ResponseRecorder {
|
||||||
|
r := httptest.NewRequest(method, "/api/files/read?path="+url.QueryEscape(p)+extra, nil)
|
||||||
|
for k, v := range hdr {
|
||||||
|
r.Header.Set(k, v)
|
||||||
|
}
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
s.filesRead(w, r)
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
w := get("GET", "", nil)
|
||||||
|
if w.Code != 200 || w.Body.Len() != 10000 || w.Header().Get("Content-Type") != "video/mp4" || w.Header().Get("Accept-Ranges") != "bytes" {
|
||||||
|
t.Fatalf("whole: %d %d %q %q", w.Code, w.Body.Len(), w.Header().Get("Content-Type"), w.Header().Get("Accept-Ranges"))
|
||||||
|
}
|
||||||
|
w = get("GET", "", map[string]string{"Range": "bytes=100-199"})
|
||||||
|
if w.Code != 206 || w.Header().Get("Content-Range") != "bytes 100-199/10000" || w.Body.Len() != 100 || w.Body.Bytes()[0] != data[100] {
|
||||||
|
t.Fatalf("range: %d %q %d", w.Code, w.Header().Get("Content-Range"), w.Body.Len())
|
||||||
|
}
|
||||||
|
w = get("GET", "", map[string]string{"Range": "bytes=9990-"})
|
||||||
|
if w.Code != 206 || w.Body.Len() != 10 || w.Body.Bytes()[9] != data[9999] {
|
||||||
|
t.Fatalf("tail range: %d %d", w.Code, w.Body.Len())
|
||||||
|
}
|
||||||
|
w = get("HEAD", "", nil)
|
||||||
|
if w.Code != 200 || w.Body.Len() != 0 || w.Header().Get("Content-Length") != "10000" {
|
||||||
|
t.Fatalf("head: %d %d %q", w.Code, w.Body.Len(), w.Header().Get("Content-Length"))
|
||||||
|
}
|
||||||
|
if w := get("GET", "&max=50", nil); w.Code != 413 {
|
||||||
|
t.Fatalf("the editor's cap: %d", w.Code)
|
||||||
|
}
|
||||||
|
if w := get("GET", "", map[string]string{"Range": "bytes=20000-"}); w.Code != 416 {
|
||||||
|
t.Fatalf("a range past the end: %d", w.Code)
|
||||||
|
}
|
||||||
|
if err := os.Chmod(p, 0); err == nil && os.Geteuid() != 0 {
|
||||||
|
if w := get("GET", "", nil); w.Code != 403 {
|
||||||
|
t.Fatalf("unreadable: %d", w.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A write over the cap is refused whole: the file is not truncated first.
|
||||||
|
func TestFilesWriteCap(t *testing.T) {
|
||||||
|
p := filepath.Join(t.TempDir(), "f.txt")
|
||||||
|
if err := os.WriteFile(p, []byte("keep me"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s := &Server{fsLocal: files.Local{}}
|
||||||
|
body := make([]byte, 8<<20+1)
|
||||||
|
r := httptest.NewRequest("PUT", "/api/files/write?path="+url.QueryEscape(p), bytes.NewReader(body))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
s.filesWrite(w, r)
|
||||||
|
if w.Code != 413 {
|
||||||
|
t.Fatalf("got %d", w.Code)
|
||||||
|
}
|
||||||
|
if b, _ := os.ReadFile(p); string(b) != "keep me" {
|
||||||
|
t.Fatalf("the file was touched: %q", b)
|
||||||
|
}
|
||||||
|
r = httptest.NewRequest("PUT", "/api/files/write?path="+url.QueryEscape(p), bytes.NewReader([]byte("new")))
|
||||||
|
w = httptest.NewRecorder()
|
||||||
|
s.filesWrite(w, r)
|
||||||
|
if b, _ := os.ReadFile(p); w.Code != 204 || string(b) != "new" {
|
||||||
|
t.Fatalf("a write within the cap: %d %q", w.Code, b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A link to nothing is a 404, not an empty 200.
|
||||||
|
func TestFilesReadDanglingLink(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
l := filepath.Join(dir, "broken")
|
||||||
|
if err := os.Symlink("nowhere", l); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s := &Server{fsLocal: files.Local{}}
|
||||||
|
r := httptest.NewRequest("GET", "/api/files/read?path="+url.QueryEscape(l), nil)
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
s.filesRead(w, r)
|
||||||
|
if w.Code != 404 {
|
||||||
|
t.Fatalf("got %d: %s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// dockerErr tells a thing that isn't there from a daemon that isn't up from anything else.
|
||||||
|
func TestDockerErr(t *testing.T) {
|
||||||
|
for _, c := range []struct {
|
||||||
|
msg string
|
||||||
|
want int
|
||||||
|
}{
|
||||||
|
{"error: No such object: zzz", 404}, {"Error response from daemon: No such container: x", 404},
|
||||||
|
{"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?", 503},
|
||||||
|
{"something else entirely", 500},
|
||||||
|
} {
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
dockerErr(w, errors.New(c.msg))
|
||||||
|
if w.Code != c.want {
|
||||||
|
t.Errorf("%q → %d, want %d", c.msg, w.Code, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -338,20 +338,6 @@ func (s *Server) listKVMs(w http.ResponseWriter, _ *http.Request) {
|
|||||||
jsonOK(w, 200, out)
|
jsonOK(w, 200, out)
|
||||||
}
|
}
|
||||||
|
|
||||||
// kvmForMachine is the KVM attached to a machine, "" being the hub itself.
|
|
||||||
func (s *Server) kvmForMachine(machine string) (config.KVM, bool) {
|
|
||||||
var found config.KVM
|
|
||||||
ok := false
|
|
||||||
s.cfg.View(func(c *config.Config) {
|
|
||||||
for _, k := range c.KVMs {
|
|
||||||
if k.Machine == machine {
|
|
||||||
found, ok = k, true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
return found, ok
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
|
||||||
var body struct{ Name, URL, Machine string }
|
var body struct{ Name, URL, Machine string }
|
||||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
|
||||||
|
|||||||
@@ -0,0 +1,169 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
// Playing a video the browser cannot play as it is: POST /api/files/play starts an HLS
|
||||||
|
// re-encode on this machine (internal/media), GET …/play/{id}/index.m3u8 and the segments
|
||||||
|
// feed the player, GET …/play/{id} is its progress, DELETE ends it. /api/files/play-caps
|
||||||
|
// says which codecs this machine can make and on what. Root mode (?sudo=1) reads the file
|
||||||
|
// and runs ffmpeg as root, through the same sudo the file reads use.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"net/http"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"hamlab/internal/files"
|
||||||
|
"hamlab/internal/media"
|
||||||
|
)
|
||||||
|
|
||||||
|
const playLimit = 3 // re-encodes at once on one machine; a GPU does two or three, a CPU one
|
||||||
|
|
||||||
|
type playCaps struct {
|
||||||
|
Codecs []playCodec `json:"codecs"`
|
||||||
|
Qualities []media.Quality `json:"qualities"`
|
||||||
|
Error string `json:"error,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type playCodec struct {
|
||||||
|
media.Codec
|
||||||
|
Engine string `json:"engine,omitempty"` // "" when nothing here encodes it
|
||||||
|
}
|
||||||
|
|
||||||
|
// playCapsHandler: GET /api/files/play-caps[?sudo=1]
|
||||||
|
func (s *Server) playCapsHandler(w http.ResponseWriter, r *http.Request) {
|
||||||
|
root := r.URL.Query().Get("sudo") == "1"
|
||||||
|
if root && s.fsSudo == nil {
|
||||||
|
jsonErr(w, 403, "root access needs passwordless sudo on the host")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
caps, err := s.thumbs.capsFor(r.Context(), root, false)
|
||||||
|
pc := playCaps{Codecs: []playCodec{}, Qualities: media.Qualities}
|
||||||
|
if err != nil {
|
||||||
|
pc.Error = err.Error()
|
||||||
|
} else if caps.Error != "" {
|
||||||
|
pc.Error = caps.Error
|
||||||
|
}
|
||||||
|
for _, c := range media.Codecs {
|
||||||
|
x := playCodec{Codec: c}
|
||||||
|
if pc.Error == "" {
|
||||||
|
x.Engine = media.EngineFor(caps, c.ID)
|
||||||
|
}
|
||||||
|
pc.Codecs = append(pc.Codecs, x)
|
||||||
|
}
|
||||||
|
jsonOK(w, 200, pc)
|
||||||
|
}
|
||||||
|
|
||||||
|
// playStart: POST /api/files/play[?sudo=1] {path, codec, quality}
|
||||||
|
func (s *Server) playStart(w http.ResponseWriter, r *http.Request) {
|
||||||
|
fsys, ok := s.fsFor(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
root := r.URL.Query().Get("sudo") == "1"
|
||||||
|
var body struct{ Path, Codec, Quality string }
|
||||||
|
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
|
||||||
|
jsonErr(w, 400, "bad request")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p, err := files.Clean(body.Path)
|
||||||
|
if err != nil {
|
||||||
|
jsonErr(w, 400, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if media.Kind(p) != "video" {
|
||||||
|
jsonErr(w, 415, "not a video")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := media.ValidChoice(body.Codec, body.Quality); err != nil {
|
||||||
|
jsonErr(w, 400, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if _, err := fsys.Stat(r.Context(), p); err != nil {
|
||||||
|
fsErr(w, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
caps, err := s.thumbs.capsFor(r.Context(), root, false)
|
||||||
|
if err != nil {
|
||||||
|
jsonErr(w, 503, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if caps.Error != "" {
|
||||||
|
jsonErr(w, 503, caps.Error)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
ictx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
|
||||||
|
src, err := media.Inspect(ictx, caps, p, root)
|
||||||
|
cancel()
|
||||||
|
if err != nil {
|
||||||
|
playErr(w, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
st, err := s.plays.Open(r.Context(), caps, root, p, body.Codec, body.Quality, src)
|
||||||
|
if err != nil {
|
||||||
|
playErr(w, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
log.Printf("files: play %s as %s %s on %s (from %s)", p, st.Codec, st.Quality, st.Engine, clientIP(r))
|
||||||
|
jsonOK(w, 200, map[string]any{"stream": st, "url": "/api/files/play/" + st.ID + "/index.m3u8"})
|
||||||
|
}
|
||||||
|
|
||||||
|
func playErr(w http.ResponseWriter, err error) {
|
||||||
|
var de *media.DecodeError
|
||||||
|
switch {
|
||||||
|
case errors.As(err, &de):
|
||||||
|
jsonErr(w, 422, de.Msg)
|
||||||
|
case strings.Contains(err.Error(), "already being re-encoded"):
|
||||||
|
jsonErr(w, 429, err.Error())
|
||||||
|
case strings.HasPrefix(err.Error(), "unknown ") || strings.HasPrefix(err.Error(), "nothing on this machine encodes"):
|
||||||
|
jsonErr(w, 400, err.Error())
|
||||||
|
case errors.Is(err, context.DeadlineExceeded):
|
||||||
|
jsonErr(w, 504, "the file took too long to read")
|
||||||
|
default:
|
||||||
|
jsonErr(w, 500, err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// playStatus: GET /api/files/play/{id}
|
||||||
|
func (s *Server) playStatus(w http.ResponseWriter, r *http.Request) {
|
||||||
|
st, ok := s.plays.Find(r.PathValue("id"))
|
||||||
|
if !ok {
|
||||||
|
jsonErr(w, 404, "no such stream")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
jsonOK(w, 200, st.Status())
|
||||||
|
}
|
||||||
|
|
||||||
|
// playStop: DELETE /api/files/play/{id}, or POST …/close — the form a page can send on its
|
||||||
|
// way out (navigator.sendBeacon), so a phone closing the tab releases the machine at once.
|
||||||
|
func (s *Server) playStop(w http.ResponseWriter, r *http.Request) {
|
||||||
|
s.plays.Close(r.PathValue("id"))
|
||||||
|
w.WriteHeader(204)
|
||||||
|
}
|
||||||
|
|
||||||
|
// playFile: GET /api/files/play/{id}/{file} — the playlist, the init segment, a segment.
|
||||||
|
func (s *Server) playFile(w http.ResponseWriter, r *http.Request) {
|
||||||
|
st, ok := s.plays.Find(r.PathValue("id"))
|
||||||
|
if !ok {
|
||||||
|
jsonErr(w, 404, "no such stream")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p, ok := st.File(r.PathValue("file"))
|
||||||
|
if !ok {
|
||||||
|
jsonErr(w, 404, "not a stream file")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
switch filepath.Ext(p) {
|
||||||
|
case ".m3u8":
|
||||||
|
w.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
|
||||||
|
default:
|
||||||
|
w.Header().Set("Content-Type", "video/mp4")
|
||||||
|
}
|
||||||
|
w.Header().Set("Cache-Control", "no-store")
|
||||||
|
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||||
|
http.ServeFile(w, r, p)
|
||||||
|
}
|
||||||
+114
-31
@@ -35,6 +35,7 @@ import (
|
|||||||
"hamlab/internal/files"
|
"hamlab/internal/files"
|
||||||
"hamlab/internal/hardware"
|
"hamlab/internal/hardware"
|
||||||
"hamlab/internal/incus"
|
"hamlab/internal/incus"
|
||||||
|
"hamlab/internal/media"
|
||||||
"hamlab/internal/metrics"
|
"hamlab/internal/metrics"
|
||||||
"hamlab/internal/network"
|
"hamlab/internal/network"
|
||||||
"hamlab/internal/nodes"
|
"hamlab/internal/nodes"
|
||||||
@@ -79,6 +80,9 @@ type Server struct {
|
|||||||
virt *incus.Manager
|
virt *incus.Manager
|
||||||
fsLocal files.FS
|
fsLocal files.FS
|
||||||
fsSudo files.FS // nil without passwordless sudo
|
fsSudo files.FS // nil without passwordless sudo
|
||||||
|
thumbs *thumbSvc
|
||||||
|
plays *media.Streamer
|
||||||
|
self string // this binary, for helpers run as tasks
|
||||||
nodes *nodes.Manager
|
nodes *nodes.Manager
|
||||||
fingerprint string
|
fingerprint string
|
||||||
version string
|
version string
|
||||||
@@ -98,7 +102,10 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
self = "hamlab"
|
self = "hamlab"
|
||||||
}
|
}
|
||||||
|
s.self = self
|
||||||
s.space = space.New(self, filepath.Join(cfg.DataDir(), "space"))
|
s.space = space.New(self, filepath.Join(cfg.DataDir(), "space"))
|
||||||
|
s.thumbs = newThumbSvc(cfg.DataDir(), self)
|
||||||
|
s.plays = media.NewStreamer(cfg.DataDir(), self, playLimit)
|
||||||
s.hostName() // read it once now so the first request doesn't have to
|
s.hostName() // read it once now so the first request doesn't have to
|
||||||
if u, err := user.Current(); err == nil && u.Username != "" {
|
if u, err := user.Current(); err == nil && u.Username != "" {
|
||||||
s.user = u.Username
|
s.user = u.Username
|
||||||
@@ -220,20 +227,8 @@ func (s *Server) routes() *http.ServeMux {
|
|||||||
m.HandleFunc("PUT /api/docker/schedules", s.auth(s.dsPut))
|
m.HandleFunc("PUT /api/docker/schedules", s.auth(s.dsPut))
|
||||||
m.HandleFunc("DELETE /api/docker/schedules/{id}", s.auth(s.dsDelete))
|
m.HandleFunc("DELETE /api/docker/schedules/{id}", s.auth(s.dsDelete))
|
||||||
m.HandleFunc("POST /api/docker/schedules/{id}/run", s.auth(s.dsRun))
|
m.HandleFunc("POST /api/docker/schedules/{id}/run", s.auth(s.dsRun))
|
||||||
m.HandleFunc("GET /api/nodes/{node}/docker/updates", s.auth(s.machineUpdates))
|
// Asked of a node through this hub, every one of these is the node's to answer: the
|
||||||
// The same, asked of a node through this hub: the hub answers for the stacks whose files
|
// generic /api/nodes/{name}/{rest...} proxy above carries them there unchanged.
|
||||||
// it holds (deployed to the node over tcp) and passes everything else to the node.
|
|
||||||
m.HandleFunc("GET /api/nodes/{node}/docker/stacks", s.auth(s.stacksList))
|
|
||||||
m.HandleFunc("POST /api/nodes/{node}/docker/stacks", s.auth(s.stackCreate))
|
|
||||||
m.HandleFunc("GET /api/nodes/{node}/docker/stacks/homes", s.auth(s.stackHomes))
|
|
||||||
m.HandleFunc("POST /api/nodes/{node}/docker/stacks/{name}/delete", s.auth(s.stackDelete))
|
|
||||||
m.HandleFunc("GET /api/nodes/{node}/docker/stacks/{name}", s.auth(s.stackFile))
|
|
||||||
m.HandleFunc("PUT /api/nodes/{node}/docker/stacks/{name}", s.auth(s.stackSave))
|
|
||||||
m.HandleFunc("GET /api/nodes/{node}/docker/stacks/{name}/diff", s.auth(s.stackDiff))
|
|
||||||
m.HandleFunc("GET /api/nodes/{node}/docker/stacks/{name}/updates", s.auth(s.stackUpdates))
|
|
||||||
m.HandleFunc("POST /api/nodes/{node}/docker/stacks/{name}/{action}", s.auth(s.stackAction))
|
|
||||||
m.HandleFunc("GET /api/nodes/{node}/tasks/{id}", s.auth(s.taskOnHub(s.getTask)))
|
|
||||||
m.HandleFunc("GET /api/nodes/{node}/tasks/{id}/stream", s.auth(s.taskOnHub(s.streamTask)))
|
|
||||||
m.HandleFunc("POST /api/docker/containers/{id}/{action}", s.auth(s.dockerAction))
|
m.HandleFunc("POST /api/docker/containers/{id}/{action}", s.auth(s.dockerAction))
|
||||||
m.HandleFunc("GET /api/docker/containers/{id}/logs", s.auth(s.dockerLogs))
|
m.HandleFunc("GET /api/docker/containers/{id}/logs", s.auth(s.dockerLogs))
|
||||||
m.HandleFunc("GET /api/docker/containers/{id}/logs/tail", s.auth(s.dockerTail))
|
m.HandleFunc("GET /api/docker/containers/{id}/logs/tail", s.auth(s.dockerTail))
|
||||||
@@ -271,6 +266,8 @@ func (s *Server) routes() *http.ServeMux {
|
|||||||
m.HandleFunc("POST /api/virt/exports/run", s.auth(s.virtExportsRun))
|
m.HandleFunc("POST /api/virt/exports/run", s.auth(s.virtExportsRun))
|
||||||
m.HandleFunc("POST /api/virt/exports/delete", s.auth(s.virtExportDelete))
|
m.HandleFunc("POST /api/virt/exports/delete", s.auth(s.virtExportDelete))
|
||||||
m.HandleFunc("GET /api/virt/sshkeys", s.auth(s.virtSSHKeys))
|
m.HandleFunc("GET /api/virt/sshkeys", s.auth(s.virtSSHKeys))
|
||||||
|
m.HandleFunc("POST /api/virt/pools", s.auth(s.virtPoolCreate))
|
||||||
|
m.HandleFunc("POST /api/virt/pools/delete", s.auth(s.virtPoolDelete))
|
||||||
m.HandleFunc("GET /api/alerts", s.auth(s.alertsGet))
|
m.HandleFunc("GET /api/alerts", s.auth(s.alertsGet))
|
||||||
m.HandleFunc("PUT /api/alerts", s.auth(s.alertsPut))
|
m.HandleFunc("PUT /api/alerts", s.auth(s.alertsPut))
|
||||||
m.HandleFunc("POST /api/alerts/test", s.auth(s.alertsTest))
|
m.HandleFunc("POST /api/alerts/test", s.auth(s.alertsTest))
|
||||||
@@ -309,8 +306,19 @@ func (s *Server) routes() *http.ServeMux {
|
|||||||
m.HandleFunc("POST /api/files/mkdir", s.auth(s.filesMkdir))
|
m.HandleFunc("POST /api/files/mkdir", s.auth(s.filesMkdir))
|
||||||
m.HandleFunc("POST /api/files/touch", s.auth(s.filesTouch))
|
m.HandleFunc("POST /api/files/touch", s.auth(s.filesTouch))
|
||||||
m.HandleFunc("POST /api/files/rename", s.auth(s.filesRename))
|
m.HandleFunc("POST /api/files/rename", s.auth(s.filesRename))
|
||||||
|
m.HandleFunc("POST /api/files/move", s.auth(s.filesMove))
|
||||||
m.HandleFunc("POST /api/files/delete", s.auth(s.filesDelete))
|
m.HandleFunc("POST /api/files/delete", s.auth(s.filesDelete))
|
||||||
m.HandleFunc("POST /api/files/chmod", s.auth(s.filesChmod))
|
m.HandleFunc("POST /api/files/chmod", s.auth(s.filesChmod))
|
||||||
|
m.HandleFunc("GET /api/files/thumb", s.auth(s.filesThumb))
|
||||||
|
m.HandleFunc("GET /api/files/thumbs", s.auth(s.thumbsStatus))
|
||||||
|
m.HandleFunc("POST /api/files/thumbs/install", s.auth(s.thumbsInstall))
|
||||||
|
m.HandleFunc("POST /api/files/thumbs/gpu-access", s.auth(s.thumbsGPUAccess))
|
||||||
|
m.HandleFunc("GET /api/files/play-caps", s.auth(s.playCapsHandler))
|
||||||
|
m.HandleFunc("POST /api/files/play", s.auth(s.playStart))
|
||||||
|
m.HandleFunc("GET /api/files/play/{id}", s.auth(s.playStatus))
|
||||||
|
m.HandleFunc("DELETE /api/files/play/{id}", s.auth(s.playStop))
|
||||||
|
m.HandleFunc("POST /api/files/play/{id}/close", s.auth(s.playStop))
|
||||||
|
m.HandleFunc("GET /api/files/play/{id}/{file}", s.auth(s.playFile))
|
||||||
m.HandleFunc("GET /api/hardware", s.auth(s.hardwareReport))
|
m.HandleFunc("GET /api/hardware", s.auth(s.hardwareReport))
|
||||||
m.HandleFunc("GET /api/network", s.auth(s.networkOverview))
|
m.HandleFunc("GET /api/network", s.auth(s.networkOverview))
|
||||||
m.HandleFunc("GET /api/vpn", s.auth(s.vpnReport))
|
m.HandleFunc("GET /api/vpn", s.auth(s.vpnReport))
|
||||||
@@ -960,11 +968,7 @@ func (s *Server) updateAllMachines(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) proxyNode(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) proxyNode(w http.ResponseWriter, r *http.Request) {
|
||||||
node := r.PathValue("node") // routes that also carry a {name} of their own call the machine {node}
|
p, ok := s.nodes.Proxy(r.PathValue("name"))
|
||||||
if node == "" {
|
|
||||||
node = r.PathValue("name")
|
|
||||||
}
|
|
||||||
p, ok := s.nodes.Proxy(node)
|
|
||||||
if !ok {
|
if !ok {
|
||||||
jsonErr(w, 404, "no such machine")
|
jsonErr(w, 404, "no such machine")
|
||||||
return
|
return
|
||||||
@@ -1928,21 +1932,15 @@ func (s *Server) filesRead(w http.ResponseWriter, r *http.Request) {
|
|||||||
jsonErr(w, 400, "not a regular file (device, pipe or socket)")
|
jsonErr(w, 400, "not a regular file (device, pipe or socket)")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if in.Broken {
|
||||||
|
jsonErr(w, 404, "a link to nothing: "+in.Target)
|
||||||
|
return
|
||||||
|
}
|
||||||
mx, _ := strconv.ParseInt(r.URL.Query().Get("max"), 10, 64)
|
mx, _ := strconv.ParseInt(r.URL.Query().Get("max"), 10, 64)
|
||||||
if mx > 0 && in.Size > mx {
|
if mx > 0 && in.Size > mx {
|
||||||
jsonErr(w, 413, "file too large")
|
jsonErr(w, 413, "file too large")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
f, err := fsys.Open(r.Context(), p)
|
|
||||||
if err != nil {
|
|
||||||
fsErr(w, err)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
defer f.Close()
|
|
||||||
var body io.Reader = f
|
|
||||||
if mx > 0 { // stat can lie (size 0 on /proc); the cap must hold on the bytes actually sent
|
|
||||||
body = io.LimitReader(f, mx+1)
|
|
||||||
}
|
|
||||||
name := filepath.Base(p)
|
name := filepath.Base(p)
|
||||||
ct := mime.TypeByExtension(filepath.Ext(name))
|
ct := mime.TypeByExtension(filepath.Ext(name))
|
||||||
if ct == "" {
|
if ct == "" {
|
||||||
@@ -1961,7 +1959,28 @@ func (s *Server) filesRead(w http.ResponseWriter, r *http.Request) {
|
|||||||
w.Header().Set("Content-Disposition", "inline; filename*=UTF-8''"+url.PathEscape(name))
|
w.Header().Set("Content-Disposition", "inline; filename*=UTF-8''"+url.PathEscape(name))
|
||||||
}
|
}
|
||||||
if in.Size > 0 && mx == 0 {
|
if in.Size > 0 && mx == 0 {
|
||||||
w.Header().Set("Content-Length", strconv.FormatInt(in.Size, 10))
|
// A real file of known size goes through net/http, which speaks Range: a video player
|
||||||
|
// can seek and a download can resume. /proc files say size 0 and take the plain path.
|
||||||
|
rs := files.NewSeeker(r.Context(), fsys, p, in.Size)
|
||||||
|
defer rs.Close()
|
||||||
|
if _, local := fsys.(files.Local); local { // cheap here, and turns "can't open" into a status rather than a cut-off body
|
||||||
|
if err := rs.Check(); err != nil {
|
||||||
|
fsErr(w, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
http.ServeContent(w, r, name, time.UnixMilli(in.Mtime), rs)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
f, err := fsys.Open(r.Context(), p)
|
||||||
|
if err != nil {
|
||||||
|
fsErr(w, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
var body io.Reader = f
|
||||||
|
if mx > 0 { // stat can lie (size 0 on /proc); the cap must hold on the bytes actually sent
|
||||||
|
body = io.LimitReader(f, mx+1)
|
||||||
}
|
}
|
||||||
_, _ = io.Copy(w, body)
|
_, _ = io.Copy(w, body)
|
||||||
}
|
}
|
||||||
@@ -1975,7 +1994,14 @@ func (s *Server) filesWrite(w http.ResponseWriter, r *http.Request) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if err := fsys.Write(r.Context(), p, io.LimitReader(r.Body, 8<<20)); err != nil {
|
// Whole in memory first: the write truncates the file, so a body over the cap must be
|
||||||
|
// refused before that, not cut short after it.
|
||||||
|
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 8<<20))
|
||||||
|
if err != nil {
|
||||||
|
jsonErr(w, 413, "too large to save here (8 MiB at most)")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := fsys.Write(r.Context(), p, bytes.NewReader(body)); err != nil {
|
||||||
fsErr(w, err)
|
fsErr(w, err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -2062,6 +2088,63 @@ func (s *Server) filesRename(w http.ResponseWriter, r *http.Request) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// filesMove: POST {path, to} — to is the full destination path. A rename when the two
|
||||||
|
// ends share a filesystem (204); otherwise the copy-and-remove runs as a task (202 with the
|
||||||
|
// task), since a folder between datasets can take as long as it takes.
|
||||||
|
func (s *Server) filesMove(w http.ResponseWriter, r *http.Request) {
|
||||||
|
fsys, ok := s.fsFor(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var body struct{ Path, To string }
|
||||||
|
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
|
||||||
|
jsonErr(w, 400, "bad request")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
from, err := files.Clean(body.Path)
|
||||||
|
if err != nil {
|
||||||
|
jsonErr(w, 400, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
to, err := files.Clean(body.To)
|
||||||
|
if err != nil {
|
||||||
|
jsonErr(w, 400, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if from == to {
|
||||||
|
jsonErr(w, 400, "it is already there")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if files.Inside(from, to) {
|
||||||
|
jsonErr(w, 400, "a folder can't be moved into itself")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// The kernel answers EXDEV for two paths on different filesystems before it looks for
|
||||||
|
// the source, so a missing file would otherwise become a task that fails on arrival.
|
||||||
|
if _, err := fsys.Stat(r.Context(), from); err != nil {
|
||||||
|
fsErr(w, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
err = fsys.Rename(r.Context(), from, to)
|
||||||
|
if err == nil {
|
||||||
|
log.Printf("files: move %s -> %s (from %s)", from, to, clientIP(r))
|
||||||
|
w.WriteHeader(204)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !files.IsCrossDevice(err) {
|
||||||
|
fsErr(w, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
cmd := files.MoveCmd(context.Background(), fsys, s.self, from, to)
|
||||||
|
t, err := s.tasks.Start("files:move", "move "+filepath.Base(from)+" to "+filepath.Dir(to), cmd)
|
||||||
|
if err != nil {
|
||||||
|
jsonErr(w, 500, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
log.Printf("files: move %s -> %s across filesystems as task %s (from %s)", from, to, t.ID, clientIP(r))
|
||||||
|
jsonOK(w, 202, t.Snapshot(false))
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Server) filesDelete(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) filesDelete(w http.ResponseWriter, r *http.Request) {
|
||||||
s.filesOp(w, r, func(fsys files.FS, body map[string]any, p string) error {
|
s.filesOp(w, r, func(fsys files.FS, body map[string]any, p string) error {
|
||||||
rec, _ := body["recursive"].(bool)
|
rec, _ := body["recursive"].(bool)
|
||||||
|
|||||||
+81
-276
@@ -18,11 +18,10 @@ package api
|
|||||||
// machine would become unfindable at once. A remembered mount resolves with no container to
|
// machine would become unfindable at once. A remembered mount resolves with no container to
|
||||||
// run in, which is right: compose runs on the host against the host's copy of the file.
|
// run in, which is right: compose runs on the host against the host's copy of the file.
|
||||||
//
|
//
|
||||||
// A stack can also be deployed *from another machine*: Dockhand on the hub drives a node's
|
// A stack is one machine's affair: its files and the Docker that runs it are on the same
|
||||||
// Docker over tcp, so the node runs the containers and the hub holds the files. The node
|
// box, and that box's Ham Lab is the only thing that touches either. Asked about a node's
|
||||||
// alone can only list such a stack read-only. The hub, asked about one of its nodes, fills
|
// stacks, a hub passes the request to the node's own Ham Lab and relays the answer; it holds
|
||||||
// in what the node couldn't (hubStacks): it resolves the files on itself and runs compose
|
// no files for anyone else and never points compose at another machine's Docker.
|
||||||
// here with DOCKER_HOST pointed at the node.
|
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
@@ -35,7 +34,6 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"log"
|
"log"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -60,8 +58,6 @@ type stack struct {
|
|||||||
Files []string `json:"files"` // every file it was deployed from, in order: -f a -f b
|
Files []string `json:"files"` // every file it was deployed from, in order: -f a -f b
|
||||||
HostFiles []string `json:"host_files"` // the same, on the machine that holds them
|
HostFiles []string `json:"host_files"` // the same, on the machine that holds them
|
||||||
Via string `json:"via"` // container compose runs inside; "" = on the host
|
Via string `json:"via"` // container compose runs inside; "" = on the host
|
||||||
Remote string `json:"remote"` // DOCKER_HOST compose is pointed at; "" = this machine's Docker
|
|
||||||
HeldBy string `json:"held_by"` // the machine holding the files, when it isn't the one running them
|
|
||||||
Editable bool `json:"editable"`
|
Editable bool `json:"editable"`
|
||||||
Note string `json:"note,omitempty"`
|
Note string `json:"note,omitempty"`
|
||||||
HasEnv bool `json:"has_env"`
|
HasEnv bool `json:"has_env"`
|
||||||
@@ -282,7 +278,7 @@ func (s *Server) findStacks(ctx context.Context) ([]stack, error) {
|
|||||||
st.resolve(maps)
|
st.resolve(maps)
|
||||||
stacks = append(stacks, *st)
|
stacks = append(stacks, *st)
|
||||||
}
|
}
|
||||||
stacks = append(stacks, scanHomes(ctx, s.homesOf(stacks, true), stacks)...)
|
stacks = append(stacks, scanHomes(ctx, s.homesOf(stacks), stacks)...)
|
||||||
sort.Slice(stacks, func(i, j int) bool { return stacks[i].Name < stacks[j].Name })
|
sort.Slice(stacks, func(i, j int) bool { return stacks[i].Name < stacks[j].Name })
|
||||||
return stacks, nil
|
return stacks, nil
|
||||||
}
|
}
|
||||||
@@ -290,19 +286,23 @@ func (s *Server) findStacks(ctx context.Context) ([]stack, error) {
|
|||||||
// ---- where stacks live ----
|
// ---- where stacks live ----
|
||||||
|
|
||||||
// A home is a directory of stacks, one folder each: where new ones are made, and where ones
|
// A home is a directory of stacks, one folder each: where new ones are made, and where ones
|
||||||
// that aren't deployed (never were, or were taken down) are still found. Homes aren't
|
// that aren't deployed (never were, or were taken down) are still found. The default is
|
||||||
// configured; they are wherever this machine's stacks already are. With none at all, ~/stacks.
|
// Ham Lab's own: <data dir>/stacks, which is where everything this console manages lives.
|
||||||
|
// Any folder this machine already keeps stacks in — two or more deployed from it — is
|
||||||
|
// offered as well, so what was deployed before Ham Lab arrived stays editable and gains
|
||||||
|
// neighbours where it is, but it is never the default.
|
||||||
type stackHome struct {
|
type stackHome struct {
|
||||||
Dir string `json:"dir"` // on the machine that holds the files
|
Dir string `json:"dir"` // on this machine
|
||||||
Inner string `json:"inner"` // the same directory as compose sees it (inside Via); Dir otherwise
|
Inner string `json:"inner"` // the same directory as compose sees it (inside Via); Dir otherwise
|
||||||
Via string `json:"via"` // container compose runs inside
|
Via string `json:"via"` // container compose runs inside
|
||||||
Remote string `json:"remote"` // DOCKER_HOST, when the containers run on another machine
|
|
||||||
HeldBy string `json:"held_by"` // the machine holding the files, when it isn't the one running them
|
|
||||||
Stacks int `json:"stacks"`
|
Stacks int `json:"stacks"`
|
||||||
Default bool `json:"default"`
|
Default bool `json:"default"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) homesOf(stacks []stack, withFallback bool) []stackHome {
|
// stacksDir is the home Ham Lab owns.
|
||||||
|
func (s *Server) stacksDir() string { return filepath.Join(s.cfg.DataDir(), "stacks") }
|
||||||
|
|
||||||
|
func (s *Server) homesOf(stacks []stack) []stackHome {
|
||||||
type tally struct {
|
type tally struct {
|
||||||
home stackHome
|
home stackHome
|
||||||
deployed int
|
deployed int
|
||||||
@@ -318,11 +318,10 @@ func (s *Server) homesOf(stacks []stack, withFallback bool) []stackHome {
|
|||||||
if strings.Count(dir, "/") < 2 { // "/" or "/tmp": not somewhere to make folders in
|
if strings.Count(dir, "/") < 2 { // "/" or "/tmp": not somewhere to make folders in
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
key := dir + "|" + st.Remote
|
t := by[dir]
|
||||||
t := by[key]
|
|
||||||
if t == nil {
|
if t == nil {
|
||||||
t = &tally{home: stackHome{Dir: dir, Remote: st.Remote, HeldBy: st.HeldBy}, byVia: map[string]int{}, inner: map[string]string{}}
|
t = &tally{home: stackHome{Dir: dir}, byVia: map[string]int{}, inner: map[string]string{}}
|
||||||
by[key] = t
|
by[dir] = t
|
||||||
}
|
}
|
||||||
t.home.Stacks++
|
t.home.Stacks++
|
||||||
if len(st.Containers) > 0 {
|
if len(st.Containers) > 0 {
|
||||||
@@ -335,11 +334,13 @@ func (s *Server) homesOf(stacks []stack, withFallback bool) []stackHome {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
// A folder is a home when stacks are kept there as a habit: two or more deployed from it.
|
// A folder is a home when stacks are kept there as a habit: two or more deployed from it.
|
||||||
// One stack on its own (Dockhand's own compose file in ~/apps/dockhand) says nothing about
|
// One stack on its own (an orchestrator's own compose file in its app folder) says nothing
|
||||||
// where the next should go, and would have every compose file near it listed as a stack.
|
// about where the next should go, and would have every compose file near it listed as a
|
||||||
// New stacks are deployed the way most of their neighbours were.
|
// stack — unless it is already in Ham Lab's folder. New stacks are deployed the way most
|
||||||
var homes, lone []stackHome
|
// of their neighbours were.
|
||||||
for _, t := range by {
|
own := s.stacksDir()
|
||||||
|
var homes []stackHome
|
||||||
|
for dir, t := range by {
|
||||||
best, n := "", -1
|
best, n := "", -1
|
||||||
for via, c := range t.byVia {
|
for via, c := range t.byVia {
|
||||||
if c > n || (c == n && via > best) {
|
if c > n || (c == n && via > best) {
|
||||||
@@ -355,29 +356,23 @@ func (s *Server) homesOf(stacks []stack, withFallback bool) []stackHome {
|
|||||||
if t.home.Inner == "" {
|
if t.home.Inner == "" {
|
||||||
t.home.Inner = t.home.Dir
|
t.home.Inner = t.home.Dir
|
||||||
}
|
}
|
||||||
if t.deployed >= 2 {
|
if t.deployed >= 2 || dir == own {
|
||||||
homes = append(homes, t.home)
|
homes = append(homes, t.home)
|
||||||
} else {
|
|
||||||
lone = append(lone, t.home)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if len(homes) == 0 {
|
|
||||||
homes = lone
|
|
||||||
}
|
|
||||||
sort.Slice(homes, func(i, j int) bool {
|
sort.Slice(homes, func(i, j int) bool {
|
||||||
|
if (homes[i].Dir == own) != (homes[j].Dir == own) {
|
||||||
|
return homes[i].Dir == own
|
||||||
|
}
|
||||||
if homes[i].Stacks != homes[j].Stacks {
|
if homes[i].Stacks != homes[j].Stacks {
|
||||||
return homes[i].Stacks > homes[j].Stacks
|
return homes[i].Stacks > homes[j].Stacks
|
||||||
}
|
}
|
||||||
return homes[i].Dir < homes[j].Dir
|
return homes[i].Dir < homes[j].Dir
|
||||||
})
|
})
|
||||||
if len(homes) == 0 && withFallback {
|
if len(homes) == 0 || homes[0].Dir != own {
|
||||||
if home, err := os.UserHomeDir(); err == nil {
|
homes = append([]stackHome{{Dir: own, Inner: own}}, homes...)
|
||||||
homes = append(homes, stackHome{Dir: filepath.Join(home, "stacks"), Inner: filepath.Join(home, "stacks")})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if len(homes) > 0 {
|
|
||||||
homes[0].Default = true
|
|
||||||
}
|
}
|
||||||
|
homes[0].Default = true
|
||||||
return homes
|
return homes
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -394,6 +389,9 @@ func scanHomes(ctx context.Context, homes []stackHome, known []stack) []stack {
|
|||||||
}
|
}
|
||||||
var out []stack
|
var out []stack
|
||||||
for _, h := range homes {
|
for _, h := range homes {
|
||||||
|
if _, err := os.Stat(h.Dir); errors.Is(err, os.ErrNotExist) {
|
||||||
|
continue // Ham Lab's own home before its first stack: nothing to find, and no sudo to spend finding it
|
||||||
|
}
|
||||||
args := []string{h.Dir, "-mindepth", "2", "-maxdepth", "2", "-type", "f", "("}
|
args := []string{h.Dir, "-mindepth", "2", "-maxdepth", "2", "-type", "f", "("}
|
||||||
for i, n := range composeNames {
|
for i, n := range composeNames {
|
||||||
if i > 0 {
|
if i > 0 {
|
||||||
@@ -417,7 +415,7 @@ func scanHomes(ctx context.Context, homes []stackHome, known []stack) []stack {
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
have[name], have[dir] = true, true
|
have[name], have[dir] = true, true
|
||||||
st := stack{Name: name, Containers: []string{}, Images: []string{}, HostFiles: []string{f}, HostFile: f, Via: h.Via, Remote: h.Remote, HeldBy: h.HeldBy, Editable: true, Note: "not deployed"}
|
st := stack{Name: name, Containers: []string{}, Images: []string{}, HostFiles: []string{f}, HostFile: f, Via: h.Via, Editable: true, Note: "not deployed"}
|
||||||
st.Files = []string{filepath.Join(h.Inner, filepath.Base(dir), filepath.Base(f))}
|
st.Files = []string{filepath.Join(h.Inner, filepath.Base(dir), filepath.Base(f))}
|
||||||
st.File = st.Files[0]
|
st.File = st.Files[0]
|
||||||
st.HasEnv = readable(st.envPath())
|
st.HasEnv = readable(st.envPath())
|
||||||
@@ -436,114 +434,23 @@ func contains(list []string, s string) bool {
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
// hubStacks is a node's stack list with the hub's help: what the node could not find is
|
// stackFor finds the stack a request means.
|
||||||
// looked for here, and if it is here — and the node's Docker answers over tcp, which is how
|
func (s *Server) stackFor(ctx context.Context, r *http.Request) (stack, error) {
|
||||||
// it was deployed in the first place — the stack becomes editable from this hub.
|
all, err := s.findStacks(ctx)
|
||||||
func (s *Server) hubStacks(ctx context.Context, node string) ([]stack, error) {
|
|
||||||
code, b, err := s.nodes.Do(ctx, node, "GET", "/api/docker/stacks", nil, nil)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return stack{}, err
|
||||||
}
|
|
||||||
var res struct{ Stacks []stack }
|
|
||||||
if code != 200 || json.Unmarshal(b, &res) != nil {
|
|
||||||
return nil, fmt.Errorf("%s answered %d", node, code)
|
|
||||||
}
|
|
||||||
remote, maps := "", []mountMap(nil)
|
|
||||||
for i := range res.Stacks {
|
|
||||||
st := &res.Stacks[i]
|
|
||||||
if st.Editable || len(st.Files) == 0 {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if remote == "" {
|
|
||||||
remote, maps = s.nodeDocker(ctx, node), mountsFor(ctx, s.store)
|
|
||||||
}
|
|
||||||
if remote == "-" {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
note := st.Note
|
|
||||||
if st.resolve(maps); !st.Editable {
|
|
||||||
st.Note = note
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
st.Remote, st.HeldBy = remote, s.hostName()
|
|
||||||
}
|
|
||||||
var held []stack
|
|
||||||
for _, st := range res.Stacks {
|
|
||||||
if st.HeldBy != "" {
|
|
||||||
held = append(held, st)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
res.Stacks = append(res.Stacks, scanHomes(ctx, s.homesOf(held, false), res.Stacks)...)
|
|
||||||
sort.Slice(res.Stacks, func(i, j int) bool { return res.Stacks[i].Name < res.Stacks[j].Name })
|
|
||||||
return res.Stacks, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
var (
|
|
||||||
nodeDockerMu sync.Mutex
|
|
||||||
nodeDockerOK = map[string]string{} // node → "tcp://host:2375", or "-" when it doesn't answer
|
|
||||||
)
|
|
||||||
|
|
||||||
// nodeDocker is the node's Docker endpoint as Dockhand-style deployments use it, if it answers.
|
|
||||||
func (s *Server) nodeDocker(ctx context.Context, node string) string {
|
|
||||||
nodeDockerMu.Lock()
|
|
||||||
defer nodeDockerMu.Unlock()
|
|
||||||
if v, ok := nodeDockerOK[node]; ok && v != "-" {
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
ep := "-"
|
|
||||||
for _, n := range s.nodes.List() {
|
|
||||||
if n.Name != node {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if u, err := url.Parse(n.URL); err == nil && u.Hostname() != "" {
|
|
||||||
cand := "tcp://" + u.Hostname() + ":2375"
|
|
||||||
c, cancel := context.WithTimeout(ctx, 5*time.Second)
|
|
||||||
if exec.CommandContext(c, dockerTool(), "-H", cand, "version", "--format", "{{.Server.Version}}").Run() == nil {
|
|
||||||
ep = cand
|
|
||||||
}
|
|
||||||
cancel()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
nodeDockerOK[node] = ep
|
|
||||||
return ep
|
|
||||||
}
|
|
||||||
|
|
||||||
// stackFor finds the stack a request means: on this machine, or (on a hub route) one of a
|
|
||||||
// node's stacks that this hub holds the files for. handled is false when the request should
|
|
||||||
// simply be passed on to the node.
|
|
||||||
func (s *Server) stackFor(ctx context.Context, r *http.Request) (st stack, handled bool, err error) {
|
|
||||||
var all []stack
|
|
||||||
if node := r.PathValue("node"); node != "" {
|
|
||||||
all, err = s.hubStacks(ctx, node)
|
|
||||||
} else {
|
|
||||||
all, err = s.findStacks(ctx)
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
return stack{}, true, err
|
|
||||||
}
|
}
|
||||||
for _, x := range all {
|
for _, x := range all {
|
||||||
if x.Name == r.PathValue("name") {
|
if x.Name == r.PathValue("name") {
|
||||||
if r.PathValue("node") != "" && x.Remote == "" {
|
return x, nil
|
||||||
return x, false, nil // the node can serve this one itself
|
|
||||||
}
|
|
||||||
return x, true, nil
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return stack{}, true, errors.New("no such stack")
|
return stack{}, errors.New("no such stack")
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) stacksList(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) stacksList(w http.ResponseWriter, r *http.Request) {
|
||||||
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
|
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
if node := r.PathValue("node"); node != "" {
|
|
||||||
stacks, err := s.hubStacks(ctx, node)
|
|
||||||
if err != nil {
|
|
||||||
s.proxyNode(w, r) // let the node's own answer (or failure) through
|
|
||||||
return
|
|
||||||
}
|
|
||||||
jsonOK(w, 200, map[string]any{"stacks": stacks})
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if dockerTool() == "" { // no Docker is an empty answer, not a failure
|
if dockerTool() == "" { // no Docker is an empty answer, not a failure
|
||||||
jsonOK(w, 200, map[string]any{"stacks": []stack{}})
|
jsonOK(w, 200, map[string]any{"stacks": []stack{}})
|
||||||
return
|
return
|
||||||
@@ -587,11 +494,7 @@ func target(r *http.Request, st stack) (host string, index int, ok bool) {
|
|||||||
func (s *Server) stackFile(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) stackFile(w http.ResponseWriter, r *http.Request) {
|
||||||
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
|
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
st, handled, err := s.stackFor(ctx, r)
|
st, err := s.stackFor(ctx, r)
|
||||||
if !handled {
|
|
||||||
s.proxyNode(w, r)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if err != nil || !st.Editable {
|
if err != nil || !st.Editable {
|
||||||
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
|
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
|
||||||
return
|
return
|
||||||
@@ -610,8 +513,8 @@ func (s *Server) stackFile(w http.ResponseWriter, r *http.Request) {
|
|||||||
jsonOK(w, 200, map[string]any{"stack": st, "text": string(b), "sum": sumOf(b), "index": i})
|
jsonOK(w, 200, map[string]any{"stack": st, "text": string(b), "sum": sumOf(b), "index": i})
|
||||||
}
|
}
|
||||||
|
|
||||||
// composeCmd is `docker compose …` for this stack, run where its files make sense and
|
// composeCmd is `docker compose …` for this stack, run where its files make sense. files are
|
||||||
// pointed at the Docker that runs its containers. files are given as compose will see them.
|
// given as compose will see them.
|
||||||
func composeCmd(ctx context.Context, st stack, files []string, args ...string) *exec.Cmd {
|
func composeCmd(ctx context.Context, st stack, files []string, args ...string) *exec.Cmd {
|
||||||
return compose.Cmd(ctx, specOf(st, files), args...)
|
return compose.Cmd(ctx, specOf(st, files), args...)
|
||||||
}
|
}
|
||||||
@@ -619,7 +522,7 @@ func composeCmd(ctx context.Context, st stack, files []string, args ...string) *
|
|||||||
// specOf is a stack as the compose runner needs it — and as a scheduled update stores it,
|
// specOf is a stack as the compose runner needs it — and as a scheduled update stores it,
|
||||||
// so the cron job and the console invoke compose identically.
|
// so the cron job and the console invoke compose identically.
|
||||||
func specOf(st stack, files []string) compose.Spec {
|
func specOf(st stack, files []string) compose.Spec {
|
||||||
return compose.Spec{Project: st.Name, Files: files, Via: st.Via, Remote: st.Remote, Tool: dockerTool()}
|
return compose.Spec{Project: st.Name, Files: files, Via: st.Via, Tool: dockerTool()}
|
||||||
}
|
}
|
||||||
|
|
||||||
// seenFiles is the stack's file list as compose runs it, with file i replaced by a candidate.
|
// seenFiles is the stack's file list as compose runs it, with file i replaced by a candidate.
|
||||||
@@ -647,11 +550,7 @@ func (s *Server) writeAsOwner(ctx context.Context, path, text string) error {
|
|||||||
func (s *Server) stackSave(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) stackSave(w http.ResponseWriter, r *http.Request) {
|
||||||
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
st, handled, err := s.stackFor(ctx, r)
|
st, err := s.stackFor(ctx, r)
|
||||||
if !handled {
|
|
||||||
s.proxyNode(w, r)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if err != nil || !st.Editable {
|
if err != nil || !st.Editable {
|
||||||
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
|
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
|
||||||
return
|
return
|
||||||
@@ -723,11 +622,7 @@ func (s *Server) stackSave(w http.ResponseWriter, r *http.Request) {
|
|||||||
func (s *Server) stackDiff(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) stackDiff(w http.ResponseWriter, r *http.Request) {
|
||||||
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
|
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
st, handled, err := s.stackFor(ctx, r)
|
st, err := s.stackFor(ctx, r)
|
||||||
if !handled {
|
|
||||||
s.proxyNode(w, r)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if err != nil || !st.Editable {
|
if err != nil || !st.Editable {
|
||||||
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
|
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
|
||||||
return
|
return
|
||||||
@@ -777,11 +672,7 @@ func maskEnv(text string) string {
|
|||||||
func (s *Server) stackAction(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) stackAction(w http.ResponseWriter, r *http.Request) {
|
||||||
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
|
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
st, handled, err := s.stackFor(ctx, r)
|
st, err := s.stackFor(ctx, r)
|
||||||
if !handled {
|
|
||||||
s.proxyNode(w, r)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if err != nil || !st.Editable {
|
if err != nil || !st.Editable {
|
||||||
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
|
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
|
||||||
return
|
return
|
||||||
@@ -794,17 +685,10 @@ func (s *Server) stackAction(w http.ResponseWriter, r *http.Request) {
|
|||||||
flags := priv.Quote(append([]string{"-p", st.Name}, fl...))
|
flags := priv.Quote(append([]string{"-p", st.Name}, fl...))
|
||||||
base := func(args string) string {
|
base := func(args string) string {
|
||||||
if st.Via != "" {
|
if st.Via != "" {
|
||||||
env := ""
|
// The CLI inside the orchestrator's container is its own `docker`, whatever this host runs.
|
||||||
if st.Remote != "" {
|
return fmt.Sprintf("%s exec %s docker compose %s %s", dockerTool(), priv.Quote([]string{st.Via}), flags, args)
|
||||||
env = "-e " + priv.Quote([]string{"DOCKER_HOST=" + st.Remote}) + " "
|
|
||||||
}
|
|
||||||
return fmt.Sprintf("%s exec %s%s docker compose %s %s", dockerTool(), env, priv.Quote([]string{st.Via}), flags, args)
|
|
||||||
}
|
}
|
||||||
env := ""
|
return fmt.Sprintf("cd %s && %s compose %s %s", priv.Quote([]string{filepath.Dir(files[0])}), dockerTool(), flags, args)
|
||||||
if st.Remote != "" { // through env(1): a quoted NAME=value in front of a command is a command name to sh, not an assignment
|
|
||||||
env = "env " + priv.Quote([]string{"DOCKER_HOST=" + st.Remote}) + " "
|
|
||||||
}
|
|
||||||
return fmt.Sprintf("cd %s && %s%s compose %s %s", priv.Quote([]string{filepath.Dir(files[0])}), env, dockerTool(), flags, args)
|
|
||||||
}
|
}
|
||||||
// After a deploy that worked, keep a copy of exactly what was deployed: the next diff is against it.
|
// After a deploy that worked, keep a copy of exactly what was deployed: the next diff is against it.
|
||||||
remember := ""
|
remember := ""
|
||||||
@@ -860,7 +744,7 @@ func (s *Server) stackAction(w http.ResponseWriter, r *http.Request) {
|
|||||||
_ = exec.CommandContext(ctx, dockerTool(), "exec", via, "pkill", "-TERM", "-f", pattern).Run()
|
_ = exec.CommandContext(ctx, dockerTool(), "exec", via, "pkill", "-TERM", "-f", pattern).Run()
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
stackUpdatesForget(r.PathValue("node") + "/" + st.Name)
|
stackUpdatesForget(st.Name)
|
||||||
log.Printf("stack %s: %s started as task %s (from %s)", st.Name, r.PathValue("action"), t.ID, clientIP(r))
|
log.Printf("stack %s: %s started as task %s (from %s)", st.Name, r.PathValue("action"), t.ID, clientIP(r))
|
||||||
jsonOK(w, 202, t.Snapshot(false))
|
jsonOK(w, 202, t.Snapshot(false))
|
||||||
}
|
}
|
||||||
@@ -885,20 +769,13 @@ var (
|
|||||||
|
|
||||||
func stackUpdatesForget(key string) { updMu.Lock(); delete(updCache, key); updMu.Unlock() }
|
func stackUpdatesForget(key string) { updMu.Lock(); delete(updCache, key); updMu.Unlock() }
|
||||||
|
|
||||||
// checkImages asks the registry whether each image is still the one we hold. remote is a
|
// checkImages asks the registry whether each image is still the one we hold. Each image is
|
||||||
// DOCKER_HOST for a machine whose images live elsewhere, "" for this one. Each image is one
|
// one registry request, which Docker Hub counts — so this is only ever called when asked for.
|
||||||
// registry request, which Docker Hub counts — so this is only ever called when asked for.
|
func checkImages(ctx context.Context, images []string) []imageCheck {
|
||||||
func checkImages(ctx context.Context, images []string, remote string) []imageCheck {
|
|
||||||
local := func(args ...string) ([]byte, error) { // the Docker that holds the images
|
|
||||||
if remote != "" {
|
|
||||||
args = append([]string{"-H", remote}, args...)
|
|
||||||
}
|
|
||||||
return dockerRun(ctx, args...)
|
|
||||||
}
|
|
||||||
res := []imageCheck{}
|
res := []imageCheck{}
|
||||||
for _, img := range images {
|
for _, img := range images {
|
||||||
ic := imageCheck{Image: img, State: "unknown"}
|
ic := imageCheck{Image: img, State: "unknown"}
|
||||||
out, err := local("image", "inspect", "-f", "{{json .RepoDigests}}", img)
|
out, err := dockerRun(ctx, "image", "inspect", "-f", "{{json .RepoDigests}}", img)
|
||||||
var have []string
|
var have []string
|
||||||
if err != nil || json.Unmarshal(bytes.TrimSpace(out), &have) != nil || len(have) == 0 {
|
if err != nil || json.Unmarshal(bytes.TrimSpace(out), &have) != nil || len(have) == 0 {
|
||||||
ic.Detail = "built or loaded here, not pulled from a registry"
|
ic.Detail = "built or loaded here, not pulled from a registry"
|
||||||
@@ -930,31 +807,28 @@ func checkImages(ctx context.Context, images []string, remote string) []imageChe
|
|||||||
}
|
}
|
||||||
|
|
||||||
// machineUpdates checks every image on this machine in one pass, so the container list can
|
// machineUpdates checks every image on this machine in one pass, so the container list can
|
||||||
// be coloured and the fleet can be asked "what is behind?" without walking stack by stack.
|
// be coloured and "what is behind?" answered without walking stack by stack.
|
||||||
func (s *Server) machineUpdates(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) machineUpdates(w http.ResponseWriter, r *http.Request) {
|
||||||
node := r.PathValue("node")
|
if dockerTool() == "" { // no Docker is an empty answer, not a failure
|
||||||
|
jsonOK(w, 200, map[string]any{"images": []imageCheck{}, "checked": 0, "stacks": map[string][]string{}})
|
||||||
|
return
|
||||||
|
}
|
||||||
ctx, cancel := context.WithTimeout(r.Context(), 10*time.Minute)
|
ctx, cancel := context.WithTimeout(r.Context(), 10*time.Minute)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
stacks, err := s.findStacks(ctx)
|
stacks, err := s.findStacks(ctx)
|
||||||
if node != "" {
|
|
||||||
stacks, err = s.hubStacks(ctx, node)
|
|
||||||
}
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
jsonErr(w, 500, err.Error())
|
jsonErr(w, 500, err.Error())
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
// one entry per image, however many stacks use it
|
// one entry per image, however many stacks use it
|
||||||
type want struct{ remote string }
|
|
||||||
imgs := map[string]want{}
|
|
||||||
byImage := map[string][]string{} // image -> stacks
|
byImage := map[string][]string{} // image -> stacks
|
||||||
for _, st := range stacks {
|
for _, st := range stacks {
|
||||||
for _, img := range st.Images {
|
for _, img := range st.Images {
|
||||||
imgs[img] = want{remote: st.Remote}
|
|
||||||
byImage[img] = append(byImage[img], st.Name)
|
byImage[img] = append(byImage[img], st.Name)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
refresh := r.URL.Query().Get("refresh") == "1"
|
refresh := r.URL.Query().Get("refresh") == "1"
|
||||||
key := "machine:" + node
|
key := "machine"
|
||||||
updMu.Lock()
|
updMu.Lock()
|
||||||
c, ok := updCache[key]
|
c, ok := updCache[key]
|
||||||
updMu.Unlock()
|
updMu.Unlock()
|
||||||
@@ -967,15 +841,11 @@ func (s *Server) machineUpdates(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
var list []string
|
var list []string
|
||||||
remote := ""
|
for img := range byImage {
|
||||||
for img, wnt := range imgs {
|
|
||||||
list = append(list, img)
|
list = append(list, img)
|
||||||
if wnt.remote != "" {
|
|
||||||
remote = wnt.remote
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
sort.Strings(list)
|
sort.Strings(list)
|
||||||
res := checkImages(ctx, list, remote)
|
res := checkImages(ctx, list)
|
||||||
updMu.Lock()
|
updMu.Lock()
|
||||||
updCache[key] = updAnswer{time.Now(), res}
|
updCache[key] = updAnswer{time.Now(), res}
|
||||||
updMu.Unlock()
|
updMu.Unlock()
|
||||||
@@ -988,16 +858,12 @@ func (s *Server) machineUpdates(w http.ResponseWriter, r *http.Request) {
|
|||||||
func (s *Server) stackUpdates(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) stackUpdates(w http.ResponseWriter, r *http.Request) {
|
||||||
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
st, handled, err := s.stackFor(ctx, r)
|
st, err := s.stackFor(ctx, r)
|
||||||
if !handled {
|
|
||||||
s.proxyNode(w, r)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
jsonErr(w, 404, err.Error())
|
jsonErr(w, 404, err.Error())
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
key := r.PathValue("node") + "/" + st.Name
|
key := st.Name
|
||||||
updMu.Lock()
|
updMu.Lock()
|
||||||
c, ok := updCache[key]
|
c, ok := updCache[key]
|
||||||
updMu.Unlock()
|
updMu.Unlock()
|
||||||
@@ -1010,84 +876,38 @@ func (s *Server) stackUpdates(w http.ResponseWriter, r *http.Request) {
|
|||||||
jsonOK(w, 200, map[string]any{"images": []imageCheck{}, "checked": 0})
|
jsonOK(w, 200, map[string]any{"images": []imageCheck{}, "checked": 0})
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
res := checkImages(ctx, st.Images, st.Remote)
|
res := checkImages(ctx, st.Images)
|
||||||
updMu.Lock()
|
updMu.Lock()
|
||||||
updCache[key] = updAnswer{time.Now(), res}
|
updCache[key] = updAnswer{time.Now(), res}
|
||||||
updMu.Unlock()
|
updMu.Unlock()
|
||||||
jsonOK(w, 200, map[string]any{"images": res, "checked": time.Now().UnixMilli()})
|
jsonOK(w, 200, map[string]any{"images": res, "checked": time.Now().UnixMilli()})
|
||||||
}
|
}
|
||||||
|
|
||||||
// taskOnHub serves a task this hub is running for one of its nodes' stacks; anything else
|
|
||||||
// under /api/nodes/<node>/tasks belongs to the node.
|
|
||||||
func (s *Server) taskOnHub(local http.HandlerFunc) http.HandlerFunc {
|
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
if _, ok := s.tasks.Get(r.PathValue("id")); ok {
|
|
||||||
local(w, r)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
s.proxyNode(w, r)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---- making and unmaking stacks ----
|
// ---- making and unmaking stacks ----
|
||||||
|
|
||||||
var stackNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]{0,62}$`)
|
var stackNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]{0,62}$`)
|
||||||
|
|
||||||
// homesFor is where a new stack could go, seen from this request: this machine's homes, or —
|
// homesFor is where a new stack could go on this machine.
|
||||||
// on a hub route — the node's own homes plus the ones this hub holds for it.
|
func (s *Server) homesFor(ctx context.Context) ([]stackHome, error) {
|
||||||
func (s *Server) homesFor(ctx context.Context, r *http.Request) ([]stackHome, error) {
|
stacks, err := s.findStacks(ctx)
|
||||||
node := r.PathValue("node")
|
|
||||||
if node == "" {
|
|
||||||
stacks, err := s.findStacks(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
return s.homesOf(stacks, true), nil
|
|
||||||
}
|
|
||||||
stacks, err := s.hubStacks(ctx, node)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
var held []stack
|
return s.homesOf(stacks), nil
|
||||||
for _, st := range stacks {
|
|
||||||
if st.HeldBy != "" {
|
|
||||||
held = append(held, st)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
homes := s.homesOf(held, false)
|
|
||||||
if code, b, err := s.nodes.Do(ctx, node, "GET", "/api/docker/stacks/homes", nil, nil); err == nil && code == 200 {
|
|
||||||
var res struct{ Homes []stackHome }
|
|
||||||
if json.Unmarshal(b, &res) == nil {
|
|
||||||
for _, h := range res.Homes {
|
|
||||||
if len(homes) > 0 && h.Stacks == 0 {
|
|
||||||
continue // the node's empty ~/stacks is only worth offering when there is nothing else
|
|
||||||
}
|
|
||||||
h.Default = false
|
|
||||||
homes = append(homes, h)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if len(homes) > 0 {
|
|
||||||
homes[0].Default = true
|
|
||||||
}
|
|
||||||
return homes, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) stackHomes(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) stackHomes(w http.ResponseWriter, r *http.Request) {
|
||||||
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
|
ctx, cancel := context.WithTimeout(r.Context(), 40*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
if dockerTool() == "" && r.PathValue("node") == "" {
|
if dockerTool() == "" {
|
||||||
jsonOK(w, 200, map[string]any{"homes": []stackHome{}})
|
jsonOK(w, 200, map[string]any{"homes": []stackHome{}})
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
homes, err := s.homesFor(ctx, r)
|
homes, err := s.homesFor(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
jsonErr(w, 500, err.Error())
|
jsonErr(w, 500, err.Error())
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if homes == nil { // a machine with no Docker and nothing held for it: an empty list, never null
|
|
||||||
homes = []stackHome{}
|
|
||||||
}
|
|
||||||
jsonOK(w, 200, map[string]any{"homes": homes})
|
jsonOK(w, 200, map[string]any{"homes": homes})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1114,7 +934,7 @@ func (s *Server) stackCreate(w http.ResponseWriter, r *http.Request) {
|
|||||||
jsonErr(w, 400, "the compose file can't be empty")
|
jsonErr(w, 400, "the compose file can't be empty")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
homes, err := s.homesFor(ctx, r)
|
homes, err := s.homesFor(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
jsonErr(w, 500, err.Error())
|
jsonErr(w, 500, err.Error())
|
||||||
return
|
return
|
||||||
@@ -1129,17 +949,7 @@ func (s *Server) stackCreate(w http.ResponseWriter, r *http.Request) {
|
|||||||
jsonErr(w, 400, "that isn't one of the places this machine keeps stacks")
|
jsonErr(w, 400, "that isn't one of the places this machine keeps stacks")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if r.PathValue("node") != "" && home.HeldBy == "" { // the node's own folder: the node makes it
|
all, _ := s.findStacks(ctx)
|
||||||
r.Body, r.ContentLength = io.NopCloser(bytes.NewReader(raw)), int64(len(raw))
|
|
||||||
s.proxyNode(w, r)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
var all []stack
|
|
||||||
if node := r.PathValue("node"); node != "" {
|
|
||||||
all, _ = s.hubStacks(ctx, node)
|
|
||||||
} else {
|
|
||||||
all, _ = s.findStacks(ctx)
|
|
||||||
}
|
|
||||||
for _, st := range all {
|
for _, st := range all {
|
||||||
if st.Name == name {
|
if st.Name == name {
|
||||||
jsonErr(w, 409, "there is already a stack called "+name)
|
jsonErr(w, 409, "there is already a stack called "+name)
|
||||||
@@ -1201,7 +1011,7 @@ func (s *Server) stackCreate(w http.ResponseWriter, r *http.Request) {
|
|||||||
_ = sudo("chmod", "600", filepath.Join(dir, ".env")) // it is where the secrets go
|
_ = sudo("chmod", "600", filepath.Join(dir, ".env")) // it is where the secrets go
|
||||||
}
|
}
|
||||||
st := stack{Name: name, Containers: []string{}, Images: []string{}, HostFiles: []string{host}, HostFile: host, Files: []string{filepath.Join(home.Inner, name, "compose.yaml")},
|
st := stack{Name: name, Containers: []string{}, Images: []string{}, HostFiles: []string{host}, HostFile: host, Files: []string{filepath.Join(home.Inner, name, "compose.yaml")},
|
||||||
Via: home.Via, Remote: home.Remote, HeldBy: home.HeldBy, Editable: true, Note: "not deployed", HasEnv: strings.TrimSpace(body.Env) != ""}
|
Via: home.Via, Editable: true, Note: "not deployed", HasEnv: strings.TrimSpace(body.Env) != ""}
|
||||||
st.File = st.Files[0]
|
st.File = st.Files[0]
|
||||||
if msg, err := composeCmd(ctx, st, seenFiles(st, -1, ""), "config", "-q").CombinedOutput(); err != nil {
|
if msg, err := composeCmd(ctx, st, seenFiles(st, -1, ""), "config", "-q").CombinedOutput(); err != nil {
|
||||||
undo()
|
undo()
|
||||||
@@ -1220,12 +1030,7 @@ func (s *Server) stackDelete(w http.ResponseWriter, r *http.Request) {
|
|||||||
ctx, cancel := context.WithTimeout(r.Context(), 4*time.Minute)
|
ctx, cancel := context.WithTimeout(r.Context(), 4*time.Minute)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
raw, _ := io.ReadAll(io.LimitReader(r.Body, 4096))
|
raw, _ := io.ReadAll(io.LimitReader(r.Body, 4096))
|
||||||
st, handled, err := s.stackFor(ctx, r)
|
st, err := s.stackFor(ctx, r)
|
||||||
if !handled {
|
|
||||||
r.Body, r.ContentLength = io.NopCloser(bytes.NewReader(raw)), int64(len(raw))
|
|
||||||
s.proxyNode(w, r)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if err != nil || !st.Editable {
|
if err != nil || !st.Editable {
|
||||||
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
|
jsonErr(w, 404, "that stack's compose file isn't anywhere this console can reach")
|
||||||
return
|
return
|
||||||
@@ -1263,7 +1068,7 @@ func (s *Server) stackDelete(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
_ = run("rmdir", dir) // only if nothing else was in it
|
_ = run("rmdir", dir) // only if nothing else was in it
|
||||||
stackUpdatesForget(r.PathValue("node") + "/" + st.Name)
|
stackUpdatesForget(st.Name)
|
||||||
log.Printf("stack %s: taken down and %d file(s) moved to %s (from %s)", st.Name, moved, trash, clientIP(r))
|
log.Printf("stack %s: taken down and %d file(s) moved to %s (from %s)", st.Name, moved, trash, clientIP(r))
|
||||||
jsonOK(w, 200, map[string]any{"trash": trash, "moved": moved})
|
jsonOK(w, 200, map[string]any{"trash": trash, "moved": moved})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"hamlab/internal/compose"
|
"hamlab/internal/compose"
|
||||||
|
"hamlab/internal/config"
|
||||||
"hamlab/internal/store"
|
"hamlab/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -119,3 +120,37 @@ func TestRepointSurvivesAMissingContainer(t *testing.T) {
|
|||||||
t.Errorf("unresolvable spec was changed: %+v", got)
|
t.Errorf("unresolvable spec was changed: %+v", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Where new stacks go: Ham Lab's own <data dir>/stacks is always offered, first and as the
|
||||||
|
// default, whether or not anything is in it yet. A folder this machine already keeps stacks
|
||||||
|
// in — two or more deployed from it — is offered too, but never ahead of Ham Lab's own.
|
||||||
|
func TestHomesDefaultToDataDir(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
cfg := config.Default(filepath.Join(root, "data", "config.json"))
|
||||||
|
s := &Server{cfg: cfg}
|
||||||
|
own := filepath.Join(root, "data", "stacks")
|
||||||
|
legacy := filepath.Join(root, "home", "stacks", "box")
|
||||||
|
deployed := func(dir, name string) stack {
|
||||||
|
f := filepath.Join(dir, name, "compose.yaml")
|
||||||
|
return stack{Name: name, Containers: []string{name + "-1"}, Files: []string{f}, File: f, HostFiles: []string{f}, HostFile: f, Editable: true}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Nothing deployed anywhere: one home, Ham Lab's own.
|
||||||
|
homes := s.homesOf(nil)
|
||||||
|
if len(homes) != 1 || homes[0].Dir != own || !homes[0].Default {
|
||||||
|
t.Fatalf("empty machine: homes = %+v, want just %s as default", homes, own)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Two stacks deployed from a pre-existing folder: it is offered, after Ham Lab's own.
|
||||||
|
homes = s.homesOf([]stack{deployed(legacy, "a"), deployed(legacy, "b")})
|
||||||
|
if len(homes) != 2 || homes[0].Dir != own || !homes[0].Default || homes[1].Dir != legacy || homes[1].Default || homes[1].Stacks != 2 {
|
||||||
|
t.Fatalf("legacy folder: homes = %+v, want [%s (default), %s]", homes, own, legacy)
|
||||||
|
}
|
||||||
|
|
||||||
|
// One lone stack elsewhere says nothing about where the next should go; one in Ham Lab's
|
||||||
|
// own folder is already home.
|
||||||
|
homes = s.homesOf([]stack{deployed(filepath.Join(root, "home", "apps"), "orchestrator"), deployed(own, "c")})
|
||||||
|
if len(homes) != 1 || homes[0].Dir != own || homes[0].Stacks != 1 || !homes[0].Default {
|
||||||
|
t.Fatalf("lone stacks: homes = %+v, want just %s", homes, own)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,418 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
// Previews for the Files tab: GET /api/files/thumb hands back one small JPEG of an image or
|
||||||
|
// a video, made by internal/thumbs and kept in <data dir>/thumbs.db. The work is bounded
|
||||||
|
// two ways — a handful of ffmpegs at once, and one per file however many rows ask for it —
|
||||||
|
// because a folder of a thousand photos asks for every row the moment previews go on.
|
||||||
|
//
|
||||||
|
// Root mode (?sudo=1) makes the preview through the same root helper the file reads use,
|
||||||
|
// so a file only root can read gets a preview too, and the probe runs as root as well: root
|
||||||
|
// opens every render node, whatever groups the daemon's user has.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"log"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"hamlab/internal/media"
|
||||||
|
"hamlab/internal/priv"
|
||||||
|
)
|
||||||
|
|
||||||
|
const thumbSize = 256
|
||||||
|
const thumbTimeout = 30 * time.Second
|
||||||
|
const capsTTL = 10 * time.Minute // how long a probe is believed; a refresh after an install or a group change does not wait
|
||||||
|
|
||||||
|
type thumbSvc struct {
|
||||||
|
cache *media.Cache
|
||||||
|
self string
|
||||||
|
sem chan struct{}
|
||||||
|
mu sync.Mutex
|
||||||
|
caps map[bool]*capsEntry // keyed by root
|
||||||
|
calls map[string]*thumbCall
|
||||||
|
}
|
||||||
|
|
||||||
|
type capsEntry struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
caps media.Caps
|
||||||
|
err error
|
||||||
|
at time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
type thumbCall struct {
|
||||||
|
done chan struct{}
|
||||||
|
res media.Result
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func newThumbSvc(dataDir, self string) *thumbSvc {
|
||||||
|
n := runtime.NumCPU() / 2 // each ffmpeg is itself threaded; previews must not take the whole machine
|
||||||
|
if n < 2 {
|
||||||
|
n = 2
|
||||||
|
}
|
||||||
|
if n > 4 {
|
||||||
|
n = 4
|
||||||
|
}
|
||||||
|
return &thumbSvc{cache: media.NewCache(filepath.Join(dataDir, "thumbs.db")), self: self, sem: make(chan struct{}, n), caps: map[bool]*capsEntry{}, calls: map[string]*thumbCall{}}
|
||||||
|
}
|
||||||
|
|
||||||
|
// capsFor is what previews can be made with, probed once and kept for a while.
|
||||||
|
func (t *thumbSvc) capsFor(ctx context.Context, root, refresh bool) (media.Caps, error) {
|
||||||
|
t.mu.Lock()
|
||||||
|
e := t.caps[root]
|
||||||
|
if e == nil {
|
||||||
|
e = &capsEntry{}
|
||||||
|
t.caps[root] = e
|
||||||
|
}
|
||||||
|
t.mu.Unlock()
|
||||||
|
e.mu.Lock()
|
||||||
|
defer e.mu.Unlock()
|
||||||
|
if !refresh && !e.at.IsZero() && time.Since(e.at) < capsTTL {
|
||||||
|
return e.caps, e.err
|
||||||
|
}
|
||||||
|
if root {
|
||||||
|
e.caps, e.err = media.ProbeRoot(ctx, t.self)
|
||||||
|
} else {
|
||||||
|
e.caps, e.err = media.Probe(ctx), nil
|
||||||
|
}
|
||||||
|
e.at = time.Now()
|
||||||
|
mode := "as " + map[bool]string{false: "the daemon", true: "root"}[root]
|
||||||
|
switch {
|
||||||
|
case e.err != nil:
|
||||||
|
log.Printf("thumbs: probe %s failed: %v", mode, e.err)
|
||||||
|
case e.caps.Error != "":
|
||||||
|
log.Printf("thumbs: %s: %s", mode, e.caps.Error)
|
||||||
|
default:
|
||||||
|
log.Printf("thumbs: %s: ffmpeg %s, %s", mode, e.caps.Version, describeCaps(e.caps))
|
||||||
|
}
|
||||||
|
return e.caps, e.err
|
||||||
|
}
|
||||||
|
|
||||||
|
// describeCaps is the one line that says how previews get made here.
|
||||||
|
func describeCaps(c media.Caps) string {
|
||||||
|
if len(c.GPUs) == 0 {
|
||||||
|
return "CPU only"
|
||||||
|
}
|
||||||
|
parts := make([]string, 0, len(c.GPUs))
|
||||||
|
for _, g := range c.GPUs {
|
||||||
|
name := g.Name
|
||||||
|
if name == "" {
|
||||||
|
name = g.Device
|
||||||
|
}
|
||||||
|
p := name + " (" + g.API
|
||||||
|
if len(g.Codecs) > 0 {
|
||||||
|
p += ": " + strings.Join(g.Codecs, ", ")
|
||||||
|
}
|
||||||
|
parts = append(parts, p+")")
|
||||||
|
}
|
||||||
|
return "GPU " + strings.Join(parts, ", ")
|
||||||
|
}
|
||||||
|
|
||||||
|
// make produces a preview, sharing the work when the same file is asked for twice at once.
|
||||||
|
func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (media.Result, error) {
|
||||||
|
t.mu.Lock()
|
||||||
|
if c, ok := t.calls[key]; ok {
|
||||||
|
t.mu.Unlock()
|
||||||
|
select {
|
||||||
|
case <-c.done:
|
||||||
|
return c.res, c.err
|
||||||
|
case <-ctx.Done():
|
||||||
|
return media.Result{}, ctx.Err()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
c := &thumbCall{done: make(chan struct{})}
|
||||||
|
t.calls[key] = c
|
||||||
|
t.mu.Unlock()
|
||||||
|
defer func() {
|
||||||
|
t.mu.Lock()
|
||||||
|
delete(t.calls, key)
|
||||||
|
t.mu.Unlock()
|
||||||
|
close(c.done)
|
||||||
|
}()
|
||||||
|
// The work itself runs on a context of its own: the first requester going away must not
|
||||||
|
// cancel a preview three other rows are waiting for.
|
||||||
|
wctx, cancel := context.WithTimeout(context.Background(), thumbTimeout+5*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
select {
|
||||||
|
case t.sem <- struct{}{}:
|
||||||
|
defer func() { <-t.sem }()
|
||||||
|
case <-ctx.Done():
|
||||||
|
c.err = ctx.Err()
|
||||||
|
return c.res, c.err
|
||||||
|
}
|
||||||
|
caps, err := t.capsFor(wctx, root, false)
|
||||||
|
if err != nil {
|
||||||
|
c.err = err
|
||||||
|
return c.res, c.err
|
||||||
|
}
|
||||||
|
o := media.Opts{Size: thumbSize, Timeout: thumbTimeout}
|
||||||
|
if root {
|
||||||
|
c.res, c.err = media.MakeRoot(wctx, t.self, caps, path, o)
|
||||||
|
} else {
|
||||||
|
c.res, c.err = media.Make(wctx, caps, path, o)
|
||||||
|
}
|
||||||
|
return c.res, c.err
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- handlers ----
|
||||||
|
|
||||||
|
// filesThumb: GET /api/files/thumb?path=…&v=<mtime>-<size>[&sudo=1]. v is the client's idea of
|
||||||
|
// the file, from its listing; it makes a cache hit free of a stat, and a changed file a new URL.
|
||||||
|
func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
|
||||||
|
fsys, ok := s.fsFor(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p, ok := cleanQuery(w, r, "path")
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if media.Kind(p) == "" {
|
||||||
|
jsonErr(w, 415, "no preview for this kind of file")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
root := r.URL.Query().Get("sudo") == "1"
|
||||||
|
if mt, sz, ok := parseV(r.URL.Query().Get("v")); ok {
|
||||||
|
key := media.Key(p, sz, mt)
|
||||||
|
if r.Header.Get("If-None-Match") == `"`+key+`"` {
|
||||||
|
w.WriteHeader(304)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if s.serveCachedThumb(w, key) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
in, err := fsys.Stat(r.Context(), p)
|
||||||
|
if err != nil {
|
||||||
|
fsErr(w, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if in.Type == "dir" || in.Type == "other" || in.Broken {
|
||||||
|
jsonErr(w, 400, "not a regular file")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
key := media.Key(p, in.Size, in.Mtime)
|
||||||
|
if s.serveCachedThumb(w, key) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
res, err := s.thumbs.make(r.Context(), root, key, p)
|
||||||
|
if err != nil {
|
||||||
|
var de *media.DecodeError
|
||||||
|
switch {
|
||||||
|
case errors.As(err, &de):
|
||||||
|
if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Failure: de.Msg}); err != nil {
|
||||||
|
log.Printf("thumbs: cache: %v", err)
|
||||||
|
}
|
||||||
|
log.Printf("thumbs: %s: %s", p, de.Msg)
|
||||||
|
jsonErr(w, 422, de.Msg)
|
||||||
|
case errors.Is(err, context.DeadlineExceeded):
|
||||||
|
log.Printf("thumbs: %s: timed out", p)
|
||||||
|
jsonErr(w, 504, "preview timed out")
|
||||||
|
case errors.Is(err, context.Canceled):
|
||||||
|
jsonErr(w, 499, "cancelled")
|
||||||
|
default:
|
||||||
|
jsonErr(w, 503, err.Error())
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}); err != nil {
|
||||||
|
log.Printf("thumbs: cache: %v", err)
|
||||||
|
}
|
||||||
|
writeThumb(w, key, res.JPEG, res.Engine)
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveCachedThumb answers from the cache when it can. A cache that cannot be read is a
|
||||||
|
// miss: the preview is made again rather than refused.
|
||||||
|
func (s *Server) serveCachedThumb(w http.ResponseWriter, key string) bool {
|
||||||
|
e, found, err := s.thumbs.cache.Get(key)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("thumbs: cache: %v", err)
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case !found:
|
||||||
|
return false
|
||||||
|
case e.Failure != "":
|
||||||
|
jsonErr(w, 422, e.Failure)
|
||||||
|
default:
|
||||||
|
writeThumb(w, key, e.JPEG, "cache ("+e.Engine+")")
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeThumb(w http.ResponseWriter, key string, jpeg []byte, engine string) {
|
||||||
|
w.Header().Set("Content-Type", "image/jpeg")
|
||||||
|
w.Header().Set("Content-Length", strconv.Itoa(len(jpeg)))
|
||||||
|
w.Header().Set("Cache-Control", "private, max-age=604800")
|
||||||
|
w.Header().Set("ETag", `"`+key+`"`)
|
||||||
|
w.Header().Set("X-Hamlab-Thumb", engine)
|
||||||
|
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||||
|
_, _ = w.Write(jpeg)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseV(v string) (mtime, size int64, ok bool) {
|
||||||
|
a, b, found := strings.Cut(v, "-")
|
||||||
|
if !found {
|
||||||
|
return 0, 0, false
|
||||||
|
}
|
||||||
|
mtime, err1 := strconv.ParseInt(a, 10, 64)
|
||||||
|
size, err2 := strconv.ParseInt(b, 10, 64)
|
||||||
|
return mtime, size, err1 == nil && err2 == nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type thumbStatus struct {
|
||||||
|
OK bool `json:"ok"`
|
||||||
|
Engine string `json:"engine"` // gpu | cpu | ""
|
||||||
|
Summary string `json:"summary"`
|
||||||
|
FFmpeg string `json:"ffmpeg,omitempty"`
|
||||||
|
GPUs []media.GPU `json:"gpus"`
|
||||||
|
Unusable []media.GPU `json:"unusable,omitempty"`
|
||||||
|
Error string `json:"error,omitempty"`
|
||||||
|
Install *thumbInstall `json:"install,omitempty"` // ffmpeg is missing and this can put it on
|
||||||
|
GPUFix bool `json:"gpu_fix,omitempty"` // a render node is there but closed to the daemon's user, and this can open it
|
||||||
|
User string `json:"user"`
|
||||||
|
Cache thumbCache `json:"cache"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type thumbCache struct {
|
||||||
|
Path string `json:"path"`
|
||||||
|
Entries int64 `json:"entries"`
|
||||||
|
Bytes int64 `json:"bytes"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type thumbInstall struct {
|
||||||
|
PM string `json:"pm"`
|
||||||
|
Package string `json:"package"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// thumbsStatus: GET /api/files/thumbs[?sudo=1][&refresh=1] — how previews get made on this machine.
|
||||||
|
func (s *Server) thumbsStatus(w http.ResponseWriter, r *http.Request) {
|
||||||
|
root := r.URL.Query().Get("sudo") == "1"
|
||||||
|
if root && s.fsSudo == nil {
|
||||||
|
jsonErr(w, 403, "root access needs passwordless sudo on the host")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
caps, err := s.thumbs.capsFor(r.Context(), root, r.URL.Query().Get("refresh") == "1")
|
||||||
|
st := thumbStatus{GPUs: []media.GPU{}, User: s.user, Cache: thumbCache{Path: s.thumbs.cache.Path}}
|
||||||
|
st.Cache.Entries, st.Cache.Bytes = s.thumbs.cache.Stats()
|
||||||
|
if root {
|
||||||
|
st.User = "root"
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
st.Error = err.Error()
|
||||||
|
jsonOK(w, 200, st)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
st.FFmpeg, st.GPUs, st.Unusable, st.Error = caps.Version, caps.GPUs, caps.Unusable, caps.Error
|
||||||
|
if st.GPUs == nil {
|
||||||
|
st.GPUs = []media.GPU{}
|
||||||
|
}
|
||||||
|
st.OK = caps.Error == ""
|
||||||
|
switch {
|
||||||
|
case !st.OK:
|
||||||
|
if pkg := ffmpegPackage(s.updates.PM()); pkg != "" && priv.Available() {
|
||||||
|
st.Install = &thumbInstall{PM: s.updates.PM(), Package: pkg}
|
||||||
|
}
|
||||||
|
st.Summary = st.Error
|
||||||
|
case len(caps.GPUs) > 0:
|
||||||
|
st.Engine = "gpu"
|
||||||
|
st.Summary = describeCaps(caps) + " · images on the CPU"
|
||||||
|
default:
|
||||||
|
st.Engine = "cpu"
|
||||||
|
st.Summary = "CPU only"
|
||||||
|
}
|
||||||
|
if !root {
|
||||||
|
for _, g := range caps.Unusable {
|
||||||
|
if g.Permission && gpuFixPossible() {
|
||||||
|
st.GPUFix = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
jsonOK(w, 200, st)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ffmpegPackage is what to install for ffmpeg on a package manager. Fedora ships it as
|
||||||
|
// ffmpeg-free; the others under its own name.
|
||||||
|
func ffmpegPackage(pm string) string {
|
||||||
|
switch pm {
|
||||||
|
case "":
|
||||||
|
return ""
|
||||||
|
case "dnf":
|
||||||
|
if b, _ := os.ReadFile("/etc/os-release"); strings.Contains(string(b), "ID=fedora") {
|
||||||
|
return "ffmpeg-free"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "ffmpeg"
|
||||||
|
}
|
||||||
|
|
||||||
|
const hamlabUnit = "/etc/systemd/system/hamlab.service"
|
||||||
|
|
||||||
|
// gpuFixPossible: the daemon is an ordinary user under systemd, with root to hand.
|
||||||
|
func gpuFixPossible() bool {
|
||||||
|
if os.Geteuid() == 0 || !priv.Available() {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
_, err := os.Stat(hamlabUnit)
|
||||||
|
return err == nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// thumbsInstall: POST /api/files/thumbs/install — put ffmpeg on with the machine's package manager.
|
||||||
|
func (s *Server) thumbsInstall(w http.ResponseWriter, r *http.Request) {
|
||||||
|
pkg := ffmpegPackage(s.updates.PM())
|
||||||
|
if pkg == "" {
|
||||||
|
jsonErr(w, 400, "no supported package manager on this machine")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t, err := s.updates.Install([]string{pkg})
|
||||||
|
if err != nil {
|
||||||
|
jsonErr(w, 400, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
log.Printf("thumbs: installing %s as task %s (from %s)", pkg, t.ID, clientIP(r))
|
||||||
|
jsonOK(w, 202, t.Snapshot(false))
|
||||||
|
}
|
||||||
|
|
||||||
|
// gpuAccessScript gives hamlab.service the groups that own the render nodes, by a drop-in,
|
||||||
|
// then restarts it: a running process cannot gain a group.
|
||||||
|
const gpuAccessScript = `set -e
|
||||||
|
unit=` + hamlabUnit + `
|
||||||
|
[ -f "$unit" ] || { echo "no $unit: this machine does not run Ham Lab under systemd"; exit 1; }
|
||||||
|
groups=""
|
||||||
|
for g in render video; do
|
||||||
|
if getent group "$g" >/dev/null 2>&1; then groups="$groups $g"; fi
|
||||||
|
done
|
||||||
|
[ -n "$groups" ] || { echo "neither a render nor a video group exists here"; exit 1; }
|
||||||
|
mkdir -p "$unit.d"
|
||||||
|
printf '[Service]\nSupplementaryGroups=%s\n' "${groups# }" > "$unit.d/gpu.conf"
|
||||||
|
echo "hamlab.service: supplementary groups${groups} ($unit.d/gpu.conf)"
|
||||||
|
systemctl daemon-reload
|
||||||
|
echo 'Ham Lab restarts in 3 seconds to pick up its new groups; the page reconnects by itself.'
|
||||||
|
systemd-run --quiet --on-active=3 --unit=hamlab-gpu-restart systemctl restart hamlab.service
|
||||||
|
`
|
||||||
|
|
||||||
|
// thumbsGPUAccess: POST /api/files/thumbs/gpu-access — open the render nodes to the daemon.
|
||||||
|
func (s *Server) thumbsGPUAccess(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if !gpuFixPossible() {
|
||||||
|
jsonErr(w, 400, "nothing to do: Ham Lab is root, has no root, or is not under systemd here")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if t, busy := s.tasks.RunningOf("thumbs:gpu"); busy { // a second run would only collide with the first's restart timer
|
||||||
|
jsonErr(w, 409, "already being done (task "+t.ID+")")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
full := priv.Wrap("sh", "-c", gpuAccessScript)
|
||||||
|
t, err := s.tasks.Start("thumbs:gpu", "give Ham Lab access to the GPU", exec.Command(full[0], full[1:]...))
|
||||||
|
if err != nil {
|
||||||
|
jsonErr(w, 400, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
log.Printf("thumbs: GPU access drop-in as task %s, restart follows (from %s)", t.ID, clientIP(r))
|
||||||
|
jsonOK(w, 202, t.Snapshot(false))
|
||||||
|
}
|
||||||
@@ -48,6 +48,7 @@ func (s *Server) virtOverview(w http.ResponseWriter, r *http.Request) {
|
|||||||
imgs, _ := s.virt.Images(ctx)
|
imgs, _ := s.virt.Images(ctx)
|
||||||
out["images"] = imgs
|
out["images"] = imgs
|
||||||
out["pools"] = s.virt.Pools(ctx)
|
out["pools"] = s.virt.Pools(ctx)
|
||||||
|
out["default_pool"] = s.virt.DefaultPool(ctx)
|
||||||
out["networks"] = s.virt.Networks(ctx)
|
out["networks"] = s.virt.Networks(ctx)
|
||||||
}
|
}
|
||||||
isos := s.virt.Library(s.virt.ISOsDir())
|
isos := s.virt.Library(s.virt.ISOsDir())
|
||||||
@@ -285,6 +286,9 @@ func (s *Server) virtInstance(w http.ResponseWriter, r *http.Request) {
|
|||||||
jsonErr(w, 400, "bad name")
|
jsonErr(w, 400, "bad name")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if !s.virtReady(w, r) {
|
||||||
|
return
|
||||||
|
}
|
||||||
list, err := s.virt.Instances(r.Context())
|
list, err := s.virt.Instances(r.Context())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
jsonErr(w, 502, err.Error())
|
jsonErr(w, 502, err.Error())
|
||||||
@@ -1094,3 +1098,70 @@ func (s *Server) virtSSHKeys(w http.ResponseWriter, _ *http.Request) {
|
|||||||
}
|
}
|
||||||
jsonOK(w, 200, map[string]any{"keys": keys, "user": s.user})
|
jsonOK(w, 200, map[string]any{"keys": keys, "user": s.user})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---- locations: storage pools, added as plain-files pools at a path of his choosing ----
|
||||||
|
|
||||||
|
func (s *Server) virtPoolCreate(w http.ResponseWriter, r *http.Request) {
|
||||||
|
var body struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Path string `json:"path"`
|
||||||
|
}
|
||||||
|
_ = readBody(r, &body, 8192)
|
||||||
|
p := filepath.Clean(body.Path)
|
||||||
|
name := strings.TrimSpace(body.Name)
|
||||||
|
if name == "" {
|
||||||
|
name = sensors.Seg(filepath.Base(p))
|
||||||
|
}
|
||||||
|
if !incus.ValidName(name) {
|
||||||
|
jsonErr(w, 400, "name: letters, digits and dashes, starting with a letter")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(p) || p == "/" {
|
||||||
|
jsonErr(w, 400, "path: an absolute path to a folder on this machine")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !s.virtReady(w, r) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
if st, err := os.Stat(p); err == nil && !st.IsDir() {
|
||||||
|
jsonErr(w, 400, p+" is a file, not a folder")
|
||||||
|
return
|
||||||
|
} else if err != nil {
|
||||||
|
mk := priv.Wrap("mkdir", "-p", p) // Incus wants the folder there; it fills it as root
|
||||||
|
if out, err := exec.CommandContext(ctx, mk[0], mk[1:]...).CombinedOutput(); err != nil {
|
||||||
|
jsonErr(w, 500, "could not make "+p+": "+strings.TrimSpace(string(out)))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if out, err := s.virt.Cmd(ctx, "storage", "create", name, "dir", "source="+p).CombinedOutput(); err != nil {
|
||||||
|
jsonErr(w, 409, strings.TrimPrefix(strings.TrimSpace(string(out)), "Error: "))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
log.Printf("virt: location %s added at %s (from %s)", name, p, clientIP(r))
|
||||||
|
jsonOK(w, 200, map[string]string{"name": name, "path": p})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) virtPoolDelete(w http.ResponseWriter, r *http.Request) {
|
||||||
|
var body struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
}
|
||||||
|
_ = readBody(r, &body, 4096)
|
||||||
|
if !incus.ValidName(body.Name) {
|
||||||
|
jsonErr(w, 400, "bad name")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !s.virtReady(w, r) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
// Incus refuses a pool anything still uses; that refusal is the answer
|
||||||
|
if out, err := s.virt.Cmd(ctx, "storage", "delete", body.Name).CombinedOutput(); err != nil {
|
||||||
|
jsonErr(w, 409, strings.TrimPrefix(strings.TrimSpace(string(out)), "Error: "))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
log.Printf("virt: location %s removed (from %s)", body.Name, clientIP(r))
|
||||||
|
w.WriteHeader(204)
|
||||||
|
}
|
||||||
|
|||||||
+9
-1
@@ -265,7 +265,15 @@ func (s *Server) zfsSnapshotDelete(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
if err := s.storage.DestroySnapshot(r.Context(), body.Name, body.Recursive); err != nil {
|
if err := s.storage.DestroySnapshot(r.Context(), body.Name, body.Recursive); err != nil {
|
||||||
jsonErr(w, 500, err.Error())
|
msg := err.Error()
|
||||||
|
switch {
|
||||||
|
case msg == "bad snapshot name":
|
||||||
|
jsonErr(w, 400, msg)
|
||||||
|
case strings.Contains(msg, "does not exist"):
|
||||||
|
jsonErr(w, 404, msg)
|
||||||
|
default:
|
||||||
|
jsonErr(w, 500, msg)
|
||||||
|
}
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
log.Printf("zfs snapshot %s destroyed by %s%s", body.Name, clientIP(r), map[bool]string{true: " (recursive)", false: ""}[body.Recursive])
|
log.Printf("zfs snapshot %s destroyed by %s%s", body.Name, clientIP(r), map[bool]string{true: " (recursive)", false: ""}[body.Recursive])
|
||||||
|
|||||||
@@ -1,14 +1,13 @@
|
|||||||
// Package compose runs docker compose the way a stack was deployed — which is not always
|
// Package compose runs docker compose the way a stack was deployed — which is not always
|
||||||
// simply "on this host". A stack deployed by an orchestrator running inside its own
|
// simply "on this host". A stack deployed by an orchestrator running inside its own
|
||||||
// container records paths in that container's filesystem, so compose has to run in there;
|
// container records paths in that container's filesystem, so compose has to run in there.
|
||||||
// a stack deployed to another machine over tcp needs DOCKER_HOST pointed at it. Both the
|
// Both the console and the scheduled updates it writes must invoke it identically, or a
|
||||||
// console and the scheduled updates it writes must invoke it identically, or a redeploy
|
// redeploy from one would not match the other's labels. Compose always drives this
|
||||||
// from one would not match the other's labels.
|
// machine's own Docker: a stack's files and its containers are on the same box.
|
||||||
package compose
|
package compose
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"os"
|
|
||||||
"os/exec"
|
"os/exec"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
)
|
)
|
||||||
@@ -18,7 +17,6 @@ type Spec struct {
|
|||||||
Project string // -p
|
Project string // -p
|
||||||
Files []string // -f, in order
|
Files []string // -f, in order
|
||||||
Via string // a container to run compose inside; "" is this host
|
Via string // a container to run compose inside; "" is this host
|
||||||
Remote string // a DOCKER_HOST; "" is this machine's Docker
|
|
||||||
Tool string // "docker" or "podman"
|
Tool string // "docker" or "podman"
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -33,18 +31,12 @@ func Cmd(ctx context.Context, s Spec, args ...string) *exec.Cmd {
|
|||||||
base = append(base, "-f", f)
|
base = append(base, "-f", f)
|
||||||
}
|
}
|
||||||
if s.Via != "" {
|
if s.Via != "" {
|
||||||
pre := []string{"exec"}
|
// The CLI inside the orchestrator's container is its own `docker`, whatever this host runs.
|
||||||
if s.Remote != "" {
|
return exec.CommandContext(ctx, tool, append(append([]string{"exec", s.Via, "docker"}, base...), args...)...)
|
||||||
pre = append(pre, "-e", "DOCKER_HOST="+s.Remote)
|
|
||||||
}
|
|
||||||
return exec.CommandContext(ctx, tool, append(append(append(pre, s.Via, "docker"), base...), args...)...)
|
|
||||||
}
|
}
|
||||||
cmd := exec.CommandContext(ctx, tool, append(base, args...)...)
|
cmd := exec.CommandContext(ctx, tool, append(base, args...)...)
|
||||||
if len(s.Files) > 0 {
|
if len(s.Files) > 0 {
|
||||||
cmd.Dir = filepath.Dir(s.Files[0])
|
cmd.Dir = filepath.Dir(s.Files[0])
|
||||||
}
|
}
|
||||||
if s.Remote != "" {
|
|
||||||
cmd.Env = append(os.Environ(), "DOCKER_HOST="+s.Remote)
|
|
||||||
}
|
|
||||||
return cmd
|
return cmd
|
||||||
}
|
}
|
||||||
|
|||||||
+83
-15
@@ -15,6 +15,7 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
|
"os/signal"
|
||||||
"os/user"
|
"os/user"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"sort"
|
"sort"
|
||||||
@@ -23,20 +24,22 @@ import (
|
|||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"hamlab/internal/media"
|
||||||
"hamlab/internal/priv"
|
"hamlab/internal/priv"
|
||||||
)
|
)
|
||||||
|
|
||||||
type Entry struct {
|
type Entry struct {
|
||||||
Name string `json:"name"`
|
Name string `json:"name"`
|
||||||
Type string `json:"type"` // dir | file | link | other
|
Type string `json:"type"` // dir | file | link | other
|
||||||
Size int64 `json:"size"`
|
Size int64 `json:"size"`
|
||||||
Mode string `json:"mode"` // "-rw-r--r--"
|
Mode string `json:"mode"` // "-rw-r--r--"
|
||||||
Perm string `json:"perm"` // "0644"
|
Perm string `json:"perm"` // "0644"
|
||||||
Mtime int64 `json:"mtime"`
|
Mtime int64 `json:"mtime"`
|
||||||
User string `json:"user"`
|
User string `json:"user"`
|
||||||
Group string `json:"group"`
|
Group string `json:"group"`
|
||||||
Target string `json:"target,omitempty"` // link target
|
Target string `json:"target,omitempty"` // link target
|
||||||
Broken bool `json:"broken,omitempty"` // dangling link
|
Broken bool `json:"broken,omitempty"` // dangling link
|
||||||
|
Preview string `json:"preview,omitempty"` // "image" | "video": the Files tab can show a thumbnail of it
|
||||||
}
|
}
|
||||||
|
|
||||||
type Listing struct {
|
type Listing struct {
|
||||||
@@ -56,10 +59,12 @@ type FS interface {
|
|||||||
List(ctx context.Context, path string) (Listing, error)
|
List(ctx context.Context, path string) (Listing, error)
|
||||||
Stat(ctx context.Context, path string) (Info, error)
|
Stat(ctx context.Context, path string) (Info, error)
|
||||||
Open(ctx context.Context, path string) (io.ReadCloser, error)
|
Open(ctx context.Context, path string) (io.ReadCloser, error)
|
||||||
|
OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) // Open from a byte offset: a range request, a player seeking
|
||||||
Write(ctx context.Context, path string, r io.Reader) error
|
Write(ctx context.Context, path string, r io.Reader) error
|
||||||
Mkdir(ctx context.Context, path string) error
|
Mkdir(ctx context.Context, path string) error
|
||||||
Touch(ctx context.Context, path string) error
|
Touch(ctx context.Context, path string) error
|
||||||
Rename(ctx context.Context, from, to string) error
|
Rename(ctx context.Context, from, to string) error
|
||||||
|
Move(ctx context.Context, from, to string, progress io.Writer) error // Rename, or across filesystems a copy and a removal; long, so callers run it as a task
|
||||||
Delete(ctx context.Context, path string, recursive bool) error
|
Delete(ctx context.Context, path string, recursive bool) error
|
||||||
Chmod(ctx context.Context, path string, perm os.FileMode) error
|
Chmod(ctx context.Context, path string, perm os.FileMode) error
|
||||||
}
|
}
|
||||||
@@ -137,7 +142,7 @@ func entryOf(dir, name string, lst os.FileInfo) Entry {
|
|||||||
} else if st.IsDir() {
|
} else if st.IsDir() {
|
||||||
e.Type = "dir" // keep dir semantics for navigation; Target says it's a link
|
e.Type = "dir" // keep dir semantics for navigation; Target says it's a link
|
||||||
} else {
|
} else {
|
||||||
e.Size = st.Size()
|
e.Size, e.Mtime = st.Size(), st.ModTime().UnixMilli() // the file's, not the link's
|
||||||
}
|
}
|
||||||
case lst.IsDir():
|
case lst.IsDir():
|
||||||
e.Type = "dir"
|
e.Type = "dir"
|
||||||
@@ -146,6 +151,9 @@ func entryOf(dir, name string, lst os.FileInfo) Entry {
|
|||||||
default:
|
default:
|
||||||
e.Type = "other"
|
e.Type = "other"
|
||||||
}
|
}
|
||||||
|
if e.Type == "file" || (e.Type == "link" && !e.Broken) {
|
||||||
|
e.Preview = media.Kind(name)
|
||||||
|
}
|
||||||
return e
|
return e
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -224,6 +232,20 @@ func (Local) Open(_ context.Context, path string) (io.ReadCloser, error) {
|
|||||||
return os.Open(path)
|
return os.Open(path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (l Local) OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) {
|
||||||
|
f, err := l.Open(ctx, path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if offset > 0 {
|
||||||
|
if _, err := f.(*os.File).Seek(offset, io.SeekStart); err != nil {
|
||||||
|
f.Close()
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return f, nil
|
||||||
|
}
|
||||||
|
|
||||||
// Write replaces the file's contents, keeping its mode; new files get 0644 (umask).
|
// Write replaces the file's contents, keeping its mode; new files get 0644 (umask).
|
||||||
func (Local) Write(_ context.Context, path string, r io.Reader) error {
|
func (Local) Write(_ context.Context, path string, r io.Reader) error {
|
||||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
|
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
|
||||||
@@ -329,7 +351,15 @@ func (r *cmdReader) Close() error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s Sudo) Open(ctx context.Context, path string) (io.ReadCloser, error) {
|
func (s Sudo) Open(ctx context.Context, path string) (io.ReadCloser, error) {
|
||||||
c := s.cmd(ctx, "read", path)
|
return s.open(ctx, "read", path)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s Sudo) OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) {
|
||||||
|
return s.open(ctx, "read", "--offset", strconv.FormatInt(offset, 10), path)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s Sudo) open(ctx context.Context, args ...string) (io.ReadCloser, error) {
|
||||||
|
c := s.cmd(ctx, args...)
|
||||||
errb := &strings.Builder{}
|
errb := &strings.Builder{}
|
||||||
c.Stderr = errb
|
c.Stderr = errb
|
||||||
out, err := c.StdoutPipe()
|
out, err := c.StdoutPipe()
|
||||||
@@ -380,6 +410,30 @@ func (s Sudo) Touch(ctx context.Context, path string) error {
|
|||||||
func (s Sudo) Rename(ctx context.Context, from, to string) error {
|
func (s Sudo) Rename(ctx context.Context, from, to string) error {
|
||||||
return s.runJSON(ctx, nil, "rename", from, to)
|
return s.runJSON(ctx, nil, "rename", from, to)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Move as root: the helper does the copying and talks on stdout.
|
||||||
|
func (s Sudo) Move(ctx context.Context, from, to string, progress io.Writer) error {
|
||||||
|
c := s.cmd(ctx, "move", from, to)
|
||||||
|
c.Stdout = progress
|
||||||
|
var errb strings.Builder
|
||||||
|
c.Stderr = &errb
|
||||||
|
if err := c.Run(); err != nil {
|
||||||
|
if msg := strings.TrimSpace(errb.String()); msg != "" {
|
||||||
|
return errors.New(msg)
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// MoveCmd is the command a task runs for a long move: this binary's helper, as root when
|
||||||
|
// the backend is.
|
||||||
|
func MoveCmd(ctx context.Context, fsys FS, self, from, to string) *exec.Cmd {
|
||||||
|
if s, ok := fsys.(Sudo); ok {
|
||||||
|
return s.cmd(ctx, "move", from, to)
|
||||||
|
}
|
||||||
|
return exec.CommandContext(ctx, self, "fs", "move", from, to)
|
||||||
|
}
|
||||||
func (s Sudo) Delete(ctx context.Context, path string, recursive bool) error {
|
func (s Sudo) Delete(ctx context.Context, path string, recursive bool) error {
|
||||||
if recursive {
|
if recursive {
|
||||||
return s.runJSON(ctx, nil, "delete", "-r", path)
|
return s.runJSON(ctx, nil, "delete", "-r", path)
|
||||||
@@ -394,11 +448,16 @@ func (s Sudo) Chmod(ctx context.Context, path string, perm os.FileMode) error {
|
|||||||
func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer) int {
|
func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer) int {
|
||||||
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
|
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
|
||||||
if len(args) < 2 {
|
if len(args) < 2 {
|
||||||
return fail(errors.New("usage: hamlab fs <list|stat|read|write|mkdir|touch|rename|delete|chmod> …"))
|
return fail(errors.New("usage: hamlab fs <list|stat|read|write|mkdir|touch|rename|move|delete|chmod> …"))
|
||||||
}
|
}
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
var l Local
|
var l Local
|
||||||
op := args[0]
|
op := args[0]
|
||||||
|
var offset int64
|
||||||
|
if op == "read" && len(args) >= 4 && args[1] == "--offset" { // read --offset N <path>
|
||||||
|
offset, _ = strconv.ParseInt(args[2], 10, 64)
|
||||||
|
args = append([]string{"read"}, args[3:]...)
|
||||||
|
}
|
||||||
clean := func(i int) (string, error) {
|
clean := func(i int) (string, error) {
|
||||||
if i >= len(args) {
|
if i >= len(args) {
|
||||||
return "", errors.New("path required")
|
return "", errors.New("path required")
|
||||||
@@ -425,7 +484,7 @@ func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer)
|
|||||||
}
|
}
|
||||||
return emit(stdout, out, stderr)
|
return emit(stdout, out, stderr)
|
||||||
case "read":
|
case "read":
|
||||||
f, err := l.Open(ctx, p)
|
f, err := l.OpenAt(ctx, p, offset)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fail(err)
|
return fail(err)
|
||||||
}
|
}
|
||||||
@@ -447,7 +506,7 @@ func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer)
|
|||||||
return fail(err)
|
return fail(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
case "rename":
|
case "rename", "move":
|
||||||
from, err := clean(1)
|
from, err := clean(1)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fail(err)
|
return fail(err)
|
||||||
@@ -456,6 +515,15 @@ func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer)
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fail(err)
|
return fail(err)
|
||||||
}
|
}
|
||||||
|
if op == "move" {
|
||||||
|
// A cancelled task is a SIGTERM: stop copying and take the partial copy away.
|
||||||
|
mctx, stop := signal.NotifyContext(ctx, syscall.SIGTERM, syscall.SIGINT)
|
||||||
|
defer stop()
|
||||||
|
if err := l.Move(mctx, from, to, stdout); err != nil {
|
||||||
|
return fail(err)
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
if err := l.Rename(ctx, from, to); err != nil {
|
if err := l.Rename(ctx, from, to); err != nil {
|
||||||
return fail(err)
|
return fail(err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,197 @@
|
|||||||
|
package files
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"syscall"
|
||||||
|
)
|
||||||
|
|
||||||
|
// IsCrossDevice reports a rename refused because the two ends are on different
|
||||||
|
// filesystems — on a NAS that is every move between datasets. The Sudo backend only has
|
||||||
|
// the helper's words for it.
|
||||||
|
func IsCrossDevice(err error) bool {
|
||||||
|
return errors.Is(err, syscall.EXDEV) || (err != nil && strings.Contains(err.Error(), "cross-device"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Inside reports whether to lies within from — a folder cannot be moved into itself.
|
||||||
|
func Inside(from, to string) bool {
|
||||||
|
return to == from || strings.HasPrefix(to, strings.TrimSuffix(from, "/")+"/")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move is Rename, and across filesystems a copy and a removal. The copy is built under a
|
||||||
|
// hidden temporary name beside where it will land and renamed into place only once it is
|
||||||
|
// complete, and the source is removed only after that: a failure part-way, or a cancel,
|
||||||
|
// leaves the source exactly as it was. Progress goes to w, a line at a time, for the task
|
||||||
|
// output that a long move is watched through.
|
||||||
|
func (Local) Move(ctx context.Context, from, to string, w io.Writer) error {
|
||||||
|
if protected[from] {
|
||||||
|
return errors.New("refusing to move " + from)
|
||||||
|
}
|
||||||
|
if Inside(from, to) {
|
||||||
|
return errors.New("a folder can't be moved into itself")
|
||||||
|
}
|
||||||
|
if _, err := os.Lstat(to); err == nil {
|
||||||
|
return errors.New("destination exists")
|
||||||
|
}
|
||||||
|
err := os.Rename(from, to)
|
||||||
|
if err == nil || !IsCrossDevice(err) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return moveByCopy(ctx, from, to, w)
|
||||||
|
}
|
||||||
|
|
||||||
|
// moveByCopy is the slow path: copy, rename into place, remove the original.
|
||||||
|
func moveByCopy(ctx context.Context, from, to string, w io.Writer) error {
|
||||||
|
st, err := os.Lstat(from)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
tmp := filepath.Join(filepath.Dir(to), "."+filepath.Base(to)+".hamlab-moving")
|
||||||
|
_ = os.RemoveAll(tmp) // a previous attempt that died
|
||||||
|
fmt.Fprintf(w, "%s is on another filesystem: copying, then removing the original\n", filepath.Dir(to))
|
||||||
|
p := &progress{w: w}
|
||||||
|
if err := copyPath(ctx, from, tmp, st, p); err != nil {
|
||||||
|
fmt.Fprintln(w, "copy failed; removing the partial copy, the original is untouched")
|
||||||
|
_ = os.RemoveAll(tmp)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := os.Rename(tmp, to); err != nil {
|
||||||
|
_ = os.RemoveAll(tmp)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "copied %s; removing the original\n", p.String())
|
||||||
|
if err := os.RemoveAll(from); err != nil {
|
||||||
|
return fmt.Errorf("copied to %s, but the original could not be removed: %w", to, err)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "moved %s to %s\n", from, to)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type progress struct {
|
||||||
|
w io.Writer
|
||||||
|
files int64
|
||||||
|
bytes int64
|
||||||
|
shownFiles int64
|
||||||
|
shownBytes int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *progress) add(files, bytes int64) {
|
||||||
|
p.files += files
|
||||||
|
p.bytes += bytes
|
||||||
|
if p.files-p.shownFiles >= 500 || p.bytes-p.shownBytes >= 512<<20 {
|
||||||
|
fmt.Fprintf(p.w, "copying… %s\n", p.String())
|
||||||
|
p.shownFiles, p.shownBytes = p.files, p.bytes
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *progress) String() string {
|
||||||
|
return fmt.Sprintf("%d files, %s", p.files, fmtSize(p.bytes))
|
||||||
|
}
|
||||||
|
|
||||||
|
func fmtSize(b int64) string {
|
||||||
|
const unit = 1024
|
||||||
|
if b < unit {
|
||||||
|
return fmt.Sprintf("%d B", b)
|
||||||
|
}
|
||||||
|
div, exp := int64(unit), 0
|
||||||
|
for n := b / unit; n >= unit && exp < 4; n /= unit {
|
||||||
|
div *= unit
|
||||||
|
exp++
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%.1f %ciB", float64(b)/float64(div), "KMGTP"[exp])
|
||||||
|
}
|
||||||
|
|
||||||
|
// copyPath copies one entry of any kind, recursively for a directory, keeping mode, times
|
||||||
|
// and — when running as root — ownership. Devices, pipes and sockets are refused: a copy
|
||||||
|
// of one is not the thing itself.
|
||||||
|
func copyPath(ctx context.Context, src, dst string, st os.FileInfo, p *progress) error {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
mode := st.Mode()
|
||||||
|
switch {
|
||||||
|
case mode&os.ModeSymlink != 0:
|
||||||
|
target, err := os.Readlink(src)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := os.Symlink(target, dst); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
chown(dst, st)
|
||||||
|
p.add(1, 0)
|
||||||
|
return nil
|
||||||
|
case mode.IsDir():
|
||||||
|
if err := os.Mkdir(dst, mode.Perm()|0o700); err != nil { // writable while it fills, exact mode after
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ents, err := os.ReadDir(src)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, e := range ents {
|
||||||
|
lst, err := e.Info()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := copyPath(ctx, filepath.Join(src, e.Name()), filepath.Join(dst, e.Name()), lst, p); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
chown(dst, st)
|
||||||
|
if err := os.Chmod(dst, mode.Perm()); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return os.Chtimes(dst, st.ModTime(), st.ModTime())
|
||||||
|
case mode.IsRegular():
|
||||||
|
n, err := copyFile(src, dst, st)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
p.add(1, n)
|
||||||
|
return nil
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("%s: can't copy a device, pipe or socket", src)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyFile(src, dst string, st os.FileInfo) (int64, error) {
|
||||||
|
in, err := os.Open(src)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
defer in.Close()
|
||||||
|
out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL, st.Mode().Perm())
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
n, err := io.Copy(out, in)
|
||||||
|
if err != nil {
|
||||||
|
out.Close()
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
if err := out.Close(); err != nil {
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
chown(dst, st)
|
||||||
|
if err := os.Chmod(dst, st.Mode().Perm()); err != nil { // umask has no say
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
return n, os.Chtimes(dst, st.ModTime(), st.ModTime())
|
||||||
|
}
|
||||||
|
|
||||||
|
// chown keeps the owner when this process may set one; an ordinary user's copy is the
|
||||||
|
// user's, which is what a rename would have left anyway.
|
||||||
|
func chown(dst string, st os.FileInfo) {
|
||||||
|
if os.Geteuid() != 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if sys, ok := st.Sys().(*syscall.Stat_t); ok {
|
||||||
|
_ = os.Lchown(dst, int(sys.Uid), int(sys.Gid))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,195 @@
|
|||||||
|
package files
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"syscall"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestInside(t *testing.T) {
|
||||||
|
for _, c := range []struct {
|
||||||
|
from, to string
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{"/a/b", "/a/b", true}, {"/a/b", "/a/b/c", true}, {"/a/b", "/a/bc", false}, {"/a/b", "/a", false}, {"/", "/x", true},
|
||||||
|
} {
|
||||||
|
if got := Inside(c.from, c.to); got != c.want {
|
||||||
|
t.Errorf("Inside(%q, %q) = %v, want %v", c.from, c.to, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIsCrossDevice(t *testing.T) {
|
||||||
|
if !IsCrossDevice(&os.LinkError{Op: "rename", Old: "/a", New: "/b", Err: syscall.EXDEV}) {
|
||||||
|
t.Fatal("the kernel's EXDEV")
|
||||||
|
}
|
||||||
|
if !IsCrossDevice(errors.New("rename /a /b: invalid cross-device link")) {
|
||||||
|
t.Fatal("the helper's words for it")
|
||||||
|
}
|
||||||
|
if IsCrossDevice(errors.New("permission denied")) || IsCrossDevice(nil) {
|
||||||
|
t.Fatal("anything else is not")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The copy path, exercised directly: a tree with nested folders, a symlink, an odd mode
|
||||||
|
// and old mtimes lands whole under the new name, and the original is gone only then.
|
||||||
|
func TestMoveByCopy(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
src := filepath.Join(root, "src", "album")
|
||||||
|
if err := os.MkdirAll(filepath.Join(src, "sub", "deep"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
old := time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
|
write := func(p, s string, mode os.FileMode) {
|
||||||
|
if err := os.WriteFile(p, []byte(s), mode); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_ = os.Chmod(p, mode)
|
||||||
|
_ = os.Chtimes(p, old, old)
|
||||||
|
}
|
||||||
|
write(filepath.Join(src, "a.txt"), "hello", 0o644)
|
||||||
|
write(filepath.Join(src, "sub", "deep", "b.bin"), strings.Repeat("x", 100000), 0o600)
|
||||||
|
write(filepath.Join(src, "run.sh"), "#!/bin/sh\n", 0o755)
|
||||||
|
if err := os.Symlink("a.txt", filepath.Join(src, "link")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_ = os.Chtimes(filepath.Join(src, "sub"), old, old)
|
||||||
|
|
||||||
|
dst := filepath.Join(root, "dst", "moved")
|
||||||
|
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var out strings.Builder
|
||||||
|
if err := moveByCopy(context.Background(), src, dst, &out); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := os.Lstat(src); !os.IsNotExist(err) {
|
||||||
|
t.Fatal("the original is still there")
|
||||||
|
}
|
||||||
|
if b, _ := os.ReadFile(filepath.Join(dst, "a.txt")); string(b) != "hello" {
|
||||||
|
t.Fatal("a.txt")
|
||||||
|
}
|
||||||
|
if st, err := os.Stat(filepath.Join(dst, "sub", "deep", "b.bin")); err != nil || st.Size() != 100000 || st.Mode().Perm() != 0o600 || !st.ModTime().Equal(old) {
|
||||||
|
t.Fatalf("b.bin: %v %v", st, err)
|
||||||
|
}
|
||||||
|
if st, _ := os.Stat(filepath.Join(dst, "run.sh")); st.Mode().Perm() != 0o755 {
|
||||||
|
t.Fatal("mode lost")
|
||||||
|
}
|
||||||
|
if target, err := os.Readlink(filepath.Join(dst, "link")); err != nil || target != "a.txt" {
|
||||||
|
t.Fatal("symlink not copied as a symlink")
|
||||||
|
}
|
||||||
|
if st, _ := os.Stat(filepath.Join(dst, "sub")); !st.ModTime().Equal(old) {
|
||||||
|
t.Fatal("folder mtime lost")
|
||||||
|
}
|
||||||
|
if left, _ := filepath.Glob(filepath.Join(root, "dst", ".*hamlab-moving*")); len(left) != 0 {
|
||||||
|
t.Fatal("temporary copy left behind")
|
||||||
|
}
|
||||||
|
if !strings.Contains(out.String(), "moved ") || !strings.Contains(out.String(), "4 files") {
|
||||||
|
t.Fatalf("progress said: %q", out.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A failure part-way leaves the original as it was and no partial copy.
|
||||||
|
func TestMoveByCopyFailsClean(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
src := filepath.Join(root, "src")
|
||||||
|
if err := os.MkdirAll(src, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(src, "a"), []byte("a"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := syscall.Mkfifo(filepath.Join(src, "pipe"), 0o644); err != nil {
|
||||||
|
t.Skip("no fifos here")
|
||||||
|
}
|
||||||
|
dst := filepath.Join(root, "dst")
|
||||||
|
err := moveByCopy(context.Background(), src, dst, &strings.Builder{})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "pipe") {
|
||||||
|
t.Fatalf("expected the pipe to be refused, got %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(filepath.Join(src, "a")); err != nil {
|
||||||
|
t.Fatal("the original was touched")
|
||||||
|
}
|
||||||
|
if _, err := os.Lstat(dst); !os.IsNotExist(err) {
|
||||||
|
t.Fatal("a partial copy was left")
|
||||||
|
}
|
||||||
|
if left, _ := filepath.Glob(filepath.Join(root, ".*hamlab-moving*")); len(left) != 0 {
|
||||||
|
t.Fatal("temporary copy left behind")
|
||||||
|
}
|
||||||
|
// a cancel is the same story
|
||||||
|
if err := os.Remove(filepath.Join(src, "pipe")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
if err := moveByCopy(ctx, src, dst, &strings.Builder{}); err == nil {
|
||||||
|
t.Fatal("a cancelled move must fail")
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(filepath.Join(src, "a")); err != nil {
|
||||||
|
t.Fatal("the original was touched by a cancel")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move on one filesystem is a rename: nothing is copied, and the guards hold.
|
||||||
|
func TestMoveSameFilesystem(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
src := filepath.Join(root, "f")
|
||||||
|
if err := os.WriteFile(src, []byte("x"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var l Local
|
||||||
|
var out strings.Builder
|
||||||
|
if err := l.Move(context.Background(), src, filepath.Join(root, "g"), &out); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if out.Len() != 0 {
|
||||||
|
t.Fatal("a rename has nothing to say")
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(filepath.Join(root, "g")); err != nil {
|
||||||
|
t.Fatal("not renamed")
|
||||||
|
}
|
||||||
|
if err := l.Move(context.Background(), root, filepath.Join(root, "inside"), &out); err == nil {
|
||||||
|
t.Fatal("into itself")
|
||||||
|
}
|
||||||
|
if err := l.Move(context.Background(), filepath.Join(root, "g"), filepath.Join(root, "g"), &out); err == nil {
|
||||||
|
t.Fatal("onto itself")
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(root, "h"), nil, 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := l.Move(context.Background(), filepath.Join(root, "g"), filepath.Join(root, "h"), &out); err == nil || !strings.Contains(err.Error(), "exists") {
|
||||||
|
t.Fatalf("onto an existing file: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A link to a file carries the file's size and time, so the row and the preview key follow
|
||||||
|
// the file; a link to a folder is a folder; a link to nothing says so.
|
||||||
|
func TestEntryOfLinks(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, "f"), []byte("12345"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
old := time.Date(2021, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||||
|
_ = os.Chtimes(filepath.Join(dir, "f"), old, old)
|
||||||
|
_ = os.Mkdir(filepath.Join(dir, "d"), 0o755)
|
||||||
|
for _, l := range [][2]string{{"lf", "f"}, {"ld", "d"}, {"lx", "nowhere"}} {
|
||||||
|
if err := os.Symlink(l[1], filepath.Join(dir, l[0])); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ent := func(n string) Entry { st, _ := os.Lstat(filepath.Join(dir, n)); return entryOf(dir, n, st) }
|
||||||
|
if e := ent("lf"); e.Type != "link" || e.Size != 5 || e.Mtime != old.UnixMilli() || e.Target != "f" || e.Broken {
|
||||||
|
t.Fatalf("link to file: %+v", e)
|
||||||
|
}
|
||||||
|
if e := ent("ld"); e.Type != "dir" || e.Target != "d" {
|
||||||
|
t.Fatalf("link to dir: %+v", e)
|
||||||
|
}
|
||||||
|
if e := ent("lx"); e.Type != "link" || !e.Broken || e.Preview != "" {
|
||||||
|
t.Fatalf("dangling: %+v", e)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
package files
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Seeker is an FS file as http.ServeContent wants one — something to seek in — for a file
|
||||||
|
// whose size is already known from a stat. ServeContent seeks to the end for the size, to
|
||||||
|
// the start of the range asked for, then reads: so the backend is opened once, at that
|
||||||
|
// position, and only ever read forward. The Sudo backend's helper is started at the offset
|
||||||
|
// rather than streamed from the top and skipped.
|
||||||
|
type Seeker struct {
|
||||||
|
ctx context.Context
|
||||||
|
fsys FS
|
||||||
|
path string
|
||||||
|
size int64
|
||||||
|
pos int64 // where the next Read reads from
|
||||||
|
at int64 // where cur is positioned, when open
|
||||||
|
cur io.ReadCloser
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewSeeker(ctx context.Context, fsys FS, path string, size int64) *Seeker {
|
||||||
|
return &Seeker{ctx: ctx, fsys: fsys, path: path, size: size}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check opens the file now, from the start, so a file that cannot be read is known before
|
||||||
|
// any headers go out. Seeking costs nothing; a Read from somewhere else reopens there.
|
||||||
|
func (s *Seeker) Check() error {
|
||||||
|
if s.cur != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
f, err := s.fsys.OpenAt(s.ctx, s.path, 0)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
s.cur, s.at = f, 0
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Seeker) Seek(offset int64, whence int) (int64, error) {
|
||||||
|
var pos int64
|
||||||
|
switch whence {
|
||||||
|
case io.SeekStart:
|
||||||
|
pos = offset
|
||||||
|
case io.SeekCurrent:
|
||||||
|
pos = s.pos + offset
|
||||||
|
case io.SeekEnd:
|
||||||
|
pos = s.size + offset
|
||||||
|
default:
|
||||||
|
return 0, errors.New("bad whence")
|
||||||
|
}
|
||||||
|
if pos < 0 {
|
||||||
|
return 0, errors.New("seek before start")
|
||||||
|
}
|
||||||
|
s.pos = pos // the stream, if open, stays where it is until a Read wants somewhere else
|
||||||
|
return pos, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Seeker) Read(p []byte) (int, error) {
|
||||||
|
if s.pos >= s.size {
|
||||||
|
return 0, io.EOF
|
||||||
|
}
|
||||||
|
if s.cur != nil && s.pos != s.at {
|
||||||
|
s.cur.Close()
|
||||||
|
s.cur = nil
|
||||||
|
}
|
||||||
|
if s.cur == nil {
|
||||||
|
f, err := s.fsys.OpenAt(s.ctx, s.path, s.pos)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
s.cur, s.at = f, s.pos
|
||||||
|
}
|
||||||
|
n, err := s.cur.Read(p)
|
||||||
|
s.pos += int64(n)
|
||||||
|
s.at += int64(n)
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Seeker) Close() error {
|
||||||
|
if s.cur == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
err := s.cur.Close()
|
||||||
|
s.cur = nil
|
||||||
|
return err
|
||||||
|
}
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
package files
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
import "strings"
|
||||||
|
|
||||||
|
type stringsBuilder = strings.Builder
|
||||||
|
|
||||||
|
type countingFS struct {
|
||||||
|
Local
|
||||||
|
opens []int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *countingFS) OpenAt(ctx context.Context, path string, off int64) (io.ReadCloser, error) {
|
||||||
|
c.opens = append(c.opens, off)
|
||||||
|
return c.Local.OpenAt(ctx, path, off)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ServeContent's moves — size from the end, seek to the range, read forward — open the
|
||||||
|
// backend once, at the range; a seek to where the stream already is costs nothing.
|
||||||
|
func TestSeeker(t *testing.T) {
|
||||||
|
p := filepath.Join(t.TempDir(), "f")
|
||||||
|
data := make([]byte, 1000)
|
||||||
|
for i := range data {
|
||||||
|
data[i] = byte(i)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(p, data, 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
fs := &countingFS{}
|
||||||
|
s := NewSeeker(context.Background(), fs, p, int64(len(data)))
|
||||||
|
if n, _ := s.Seek(0, io.SeekEnd); n != 1000 {
|
||||||
|
t.Fatalf("end = %d", n)
|
||||||
|
}
|
||||||
|
if _, err := s.Seek(100, io.SeekStart); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
buf := make([]byte, 50)
|
||||||
|
if n, err := io.ReadFull(s, buf); err != nil || n != 50 || buf[0] != 100 || buf[49] != 149 {
|
||||||
|
t.Fatalf("read at 100: %d %v %v", n, err, buf[:2])
|
||||||
|
}
|
||||||
|
if _, err := s.Seek(150, io.SeekStart); err != nil { // where it already is
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if n, _ := io.ReadFull(s, buf); n != 50 || buf[0] != 150 {
|
||||||
|
t.Fatal("read after a no-op seek")
|
||||||
|
}
|
||||||
|
if len(fs.opens) != 1 || fs.opens[0] != 100 {
|
||||||
|
t.Fatalf("opens = %v, want [100]", fs.opens)
|
||||||
|
}
|
||||||
|
if _, err := s.Seek(0, io.SeekStart); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rest, err := io.ReadAll(s)
|
||||||
|
if err != nil || len(rest) != 1000 || rest[999] != data[999] {
|
||||||
|
t.Fatalf("read all from the start: %d %v", len(rest), err)
|
||||||
|
}
|
||||||
|
if len(fs.opens) != 2 || fs.opens[1] != 0 {
|
||||||
|
t.Fatalf("opens = %v, want [100 0]", fs.opens)
|
||||||
|
}
|
||||||
|
if n, err := s.Read(buf); n != 0 || err != io.EOF {
|
||||||
|
t.Fatal("past the end is EOF")
|
||||||
|
}
|
||||||
|
if _, err := s.Seek(-1, io.SeekStart); err == nil {
|
||||||
|
t.Fatal("before the start")
|
||||||
|
}
|
||||||
|
if err := s.Close(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// Check opens at the start and keeps that stream for a read from the start
|
||||||
|
s2 := NewSeeker(context.Background(), fs, p, 1000)
|
||||||
|
if err := s2.Check(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s2.Seek(0, io.SeekEnd)
|
||||||
|
s2.Seek(0, io.SeekStart)
|
||||||
|
if n, _ := io.ReadFull(s2, buf); n != 50 || buf[0] != 0 {
|
||||||
|
t.Fatal("read from the start after Check")
|
||||||
|
}
|
||||||
|
if len(fs.opens) != 3 {
|
||||||
|
t.Fatalf("Check's open was not reused: %v", fs.opens)
|
||||||
|
}
|
||||||
|
s2.Close()
|
||||||
|
if err := NewSeeker(context.Background(), fs, filepath.Join(t.TempDir(), "nope"), 5).Check(); err == nil {
|
||||||
|
t.Fatal("Check must report a file that can't be opened")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The helper's read takes an offset, for the Sudo backend.
|
||||||
|
func TestHelperReadOffset(t *testing.T) {
|
||||||
|
p := filepath.Join(t.TempDir(), "f")
|
||||||
|
if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var out, errb stringsBuilder
|
||||||
|
if rc := Helper([]string{"read", "--offset", "7", p}, nil, &out, &errb); rc != 0 {
|
||||||
|
t.Fatalf("rc %d: %s", rc, errb.String())
|
||||||
|
}
|
||||||
|
if out.String() != "789" {
|
||||||
|
t.Fatalf("got %q", out.String())
|
||||||
|
}
|
||||||
|
out.Reset()
|
||||||
|
if rc := Helper([]string{"read", p}, nil, &out, &errb); rc != 0 || out.String() != "0123456789" {
|
||||||
|
t.Fatalf("plain read: rc %d %q", rc, out.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -10,6 +10,7 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"hamlab/internal/store"
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -91,7 +92,7 @@ func (m *Manager) SaveExportConfig(c ExportConfig) error {
|
|||||||
}
|
}
|
||||||
c.Updated = time.Now()
|
c.Updated = time.Now()
|
||||||
b, _ := json.MarshalIndent(c, "", " ")
|
b, _ := json.MarshalIndent(c, "", " ")
|
||||||
return os.WriteFile(m.exportConfigPath(), b, 0o644)
|
return store.WriteFileAtomic(m.exportConfigPath(), b, 0o644)
|
||||||
}
|
}
|
||||||
|
|
||||||
type ExportRun struct {
|
type ExportRun struct {
|
||||||
@@ -226,6 +227,6 @@ func (m *Manager) RunExports(ctx context.Context, names []string, w func(string)
|
|||||||
}
|
}
|
||||||
run.Seconds = time.Since(start).Seconds()
|
run.Seconds = time.Since(start).Seconds()
|
||||||
b, _ := json.MarshalIndent(run, "", " ")
|
b, _ := json.MarshalIndent(run, "", " ")
|
||||||
_ = os.WriteFile(m.exportRunPath(), b, 0o644)
|
_ = store.WriteFileAtomic(m.exportRunPath(), b, 0o644)
|
||||||
return run
|
return run
|
||||||
}
|
}
|
||||||
|
|||||||
+66
-6
@@ -28,7 +28,7 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
PoolName = "default"
|
PoolName = "local" // plain files on the machine's own disk, in Ham Lab's folder
|
||||||
BridgeName = "incusbr0"
|
BridgeName = "incusbr0"
|
||||||
ZPoolName = "hamlab-virt" // the ZFS pool made on the backing file; Incus is handed it by name
|
ZPoolName = "hamlab-virt" // the ZFS pool made on the backing file; Incus is handed it by name
|
||||||
AdminGroup = "incus-admin"
|
AdminGroup = "incus-admin"
|
||||||
@@ -615,11 +615,38 @@ type Pool struct {
|
|||||||
Name string `json:"name"`
|
Name string `json:"name"`
|
||||||
Driver string `json:"driver"`
|
Driver string `json:"driver"`
|
||||||
Source string `json:"source"`
|
Source string `json:"source"`
|
||||||
|
Where string `json:"where"` // in words: what disk this is, so a pool is picked by place, not by driver
|
||||||
Used uint64 `json:"used"`
|
Used uint64 `json:"used"`
|
||||||
Total uint64 `json:"total"`
|
Total uint64 `json:"total"`
|
||||||
UsedBy int `json:"used_by"`
|
UsedBy int `json:"used_by"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// poolWhere describes a pool by where its bytes land.
|
||||||
|
func (m *Manager) poolWhere(driver, source string) string {
|
||||||
|
inHam := strings.HasPrefix(source, m.dir+"/") || source == m.dir
|
||||||
|
switch driver {
|
||||||
|
case "dir":
|
||||||
|
if inHam {
|
||||||
|
return "this machine's own disk — plain files, each disk sized on its own"
|
||||||
|
}
|
||||||
|
return "plain files at " + source
|
||||||
|
case "zfs":
|
||||||
|
if strings.HasSuffix(source, ".img") || source == ZPoolName {
|
||||||
|
return "a ZFS pool inside one file on this machine's disk"
|
||||||
|
}
|
||||||
|
if strings.Contains(source, "/") {
|
||||||
|
return "ZFS dataset " + source
|
||||||
|
}
|
||||||
|
return "ZFS pool " + source
|
||||||
|
case "btrfs":
|
||||||
|
if inHam {
|
||||||
|
return "btrfs inside one file on this machine's disk"
|
||||||
|
}
|
||||||
|
return "btrfs at " + source
|
||||||
|
}
|
||||||
|
return driver + " at " + source
|
||||||
|
}
|
||||||
|
|
||||||
type Network struct {
|
type Network struct {
|
||||||
Name string `json:"name"`
|
Name string `json:"name"`
|
||||||
Type string `json:"type"`
|
Type string `json:"type"`
|
||||||
@@ -647,7 +674,7 @@ func (m *Manager) Pools(ctx context.Context) []Pool {
|
|||||||
}
|
}
|
||||||
var out2 []Pool
|
var out2 []Pool
|
||||||
for _, p := range raw {
|
for _, p := range raw {
|
||||||
pool := Pool{Name: p.Name, Driver: p.Driver, Source: p.Config["source"], UsedBy: len(p.UsedBy)}
|
pool := Pool{Name: p.Name, Driver: p.Driver, Source: p.Config["source"], Where: m.poolWhere(p.Driver, p.Config["source"]), UsedBy: len(p.UsedBy)}
|
||||||
if b, err := m.run(c, "query", "/1.0/storage-pools/"+p.Name+"/resources"); err == nil {
|
if b, err := m.run(c, "query", "/1.0/storage-pools/"+p.Name+"/resources"); err == nil {
|
||||||
var res struct {
|
var res struct {
|
||||||
Space struct {
|
Space struct {
|
||||||
@@ -664,6 +691,24 @@ func (m *Manager) Pools(ctx context.Context) []Pool {
|
|||||||
return out2
|
return out2
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// DefaultPool is where a new instance's disk goes unless the form says otherwise: the pool
|
||||||
|
// on the default profile's root device.
|
||||||
|
func (m *Manager) DefaultPool(ctx context.Context) string {
|
||||||
|
c, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
out, err := m.run(c, "query", "/1.0/profiles/default")
|
||||||
|
if err != nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
var p struct {
|
||||||
|
Devices map[string]map[string]string `json:"devices"`
|
||||||
|
}
|
||||||
|
if json.Unmarshal(out, &p) != nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return p.Devices["root"]["pool"]
|
||||||
|
}
|
||||||
|
|
||||||
func (m *Manager) Networks(ctx context.Context) []Network {
|
func (m *Manager) Networks(ctx context.Context) []Network {
|
||||||
c, cancel := context.WithTimeout(ctx, 20*time.Second)
|
c, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
@@ -869,6 +914,7 @@ type CreateSpec struct {
|
|||||||
CPU string `json:"cpu"`
|
CPU string `json:"cpu"`
|
||||||
Memory string `json:"memory"`
|
Memory string `json:"memory"`
|
||||||
Disk string `json:"disk"`
|
Disk string `json:"disk"`
|
||||||
|
Pool string `json:"pool"` // where the root disk goes; blank is the default profile's
|
||||||
Network string `json:"network"`
|
Network string `json:"network"`
|
||||||
CloudInit string `json:"cloud_init"`
|
CloudInit string `json:"cloud_init"`
|
||||||
Profiles []string `json:"profiles"`
|
Profiles []string `json:"profiles"`
|
||||||
@@ -891,6 +937,8 @@ func CreateArgs(sp CreateSpec, kvm bool) ([]string, error) {
|
|||||||
return nil, errors.New("memory: a size, like 2GiB")
|
return nil, errors.New("memory: a size, like 2GiB")
|
||||||
case !ValidSize(sp.Disk):
|
case !ValidSize(sp.Disk):
|
||||||
return nil, errors.New("disk: a size, like 20GiB")
|
return nil, errors.New("disk: a size, like 20GiB")
|
||||||
|
case sp.Pool != "" && !ValidName(sp.Pool):
|
||||||
|
return nil, errors.New("pool: a storage pool's name")
|
||||||
case !ValidNetwork(sp.Network):
|
case !ValidNetwork(sp.Network):
|
||||||
return nil, errors.New("network: a bridge name")
|
return nil, errors.New("network: a bridge name")
|
||||||
case len(sp.CloudInit) > 256<<10:
|
case len(sp.CloudInit) > 256<<10:
|
||||||
@@ -912,8 +960,16 @@ func CreateArgs(sp CreateSpec, kvm bool) ([]string, error) {
|
|||||||
if sp.Memory != "" {
|
if sp.Memory != "" {
|
||||||
args = append(args, "-c", "limits.memory="+sp.Memory)
|
args = append(args, "-c", "limits.memory="+sp.Memory)
|
||||||
}
|
}
|
||||||
|
// the root disk: the profile's, with the pool and size overridden where given
|
||||||
|
var root []string
|
||||||
|
if sp.Pool != "" {
|
||||||
|
root = append(root, "pool="+sp.Pool)
|
||||||
|
}
|
||||||
if sp.Disk != "" {
|
if sp.Disk != "" {
|
||||||
args = append(args, "-d", "root,size="+sp.Disk)
|
root = append(root, "size="+sp.Disk)
|
||||||
|
}
|
||||||
|
if len(root) > 0 {
|
||||||
|
args = append(args, "-d", "root,"+strings.Join(root, ","))
|
||||||
}
|
}
|
||||||
if sp.Network != "" {
|
if sp.Network != "" {
|
||||||
args = append(args, "-n", sp.Network)
|
args = append(args, "-n", sp.Network)
|
||||||
@@ -1104,6 +1160,7 @@ type FromISOSpec struct {
|
|||||||
CPU string `json:"cpu"`
|
CPU string `json:"cpu"`
|
||||||
Memory string `json:"memory"`
|
Memory string `json:"memory"`
|
||||||
Disk string `json:"disk"`
|
Disk string `json:"disk"`
|
||||||
|
Pool string `json:"pool"`
|
||||||
Network string `json:"network"`
|
Network string `json:"network"`
|
||||||
SecureBoot bool `json:"secureboot"`
|
SecureBoot bool `json:"secureboot"`
|
||||||
Start bool `json:"start"`
|
Start bool `json:"start"`
|
||||||
@@ -1121,6 +1178,8 @@ func FromISOScript(sp FromISOSpec, kvm bool) (script string, params []string, er
|
|||||||
return "", nil, errors.New("iso: pick one from the library")
|
return "", nil, errors.New("iso: pick one from the library")
|
||||||
case !ValidCPU(sp.CPU) || !ValidSize(sp.Memory) || !ValidSize(sp.Disk) || !ValidNetwork(sp.Network):
|
case !ValidCPU(sp.CPU) || !ValidSize(sp.Memory) || !ValidSize(sp.Disk) || !ValidNetwork(sp.Network):
|
||||||
return "", nil, errors.New("cpu, memory, disk: a count and two sizes, like 2, 2GiB, 20GiB")
|
return "", nil, errors.New("cpu, memory, disk: a count and two sizes, like 2, 2GiB, 20GiB")
|
||||||
|
case sp.Pool != "" && !ValidName(sp.Pool):
|
||||||
|
return "", nil, errors.New("pool: a storage pool's name")
|
||||||
}
|
}
|
||||||
cpu, mem, disk := sp.CPU, sp.Memory, sp.Disk
|
cpu, mem, disk := sp.CPU, sp.Memory, sp.Disk
|
||||||
if cpu == "" {
|
if cpu == "" {
|
||||||
@@ -1140,13 +1199,14 @@ func FromISOScript(sp FromISOSpec, kvm bool) (script string, params []string, er
|
|||||||
if sp.Start {
|
if sp.Start {
|
||||||
start = "1"
|
start = "1"
|
||||||
}
|
}
|
||||||
// $1 name $2 iso path $3 cpu $4 memory $5 disk $6 network $7 secureboot $8 start
|
// $1 name $2 iso path $3 cpu $4 memory $5 disk $6 network $7 secureboot $8 start $9 pool
|
||||||
script = `set -e
|
script = `set -e
|
||||||
NET=""; [ -n "$6" ] && NET="-n $6"
|
NET=""; [ -n "$6" ] && NET="-n $6"
|
||||||
incus init --empty --vm "$1" -c limits.cpu="$3" -c limits.memory="$4" -c security.secureboot="$7" -d root,size="$5" $NET
|
ROOT="root,size=$5"; [ -n "$9" ] && ROOT="root,pool=$9,size=$5"
|
||||||
|
incus init --empty --vm "$1" -c limits.cpu="$3" -c limits.memory="$4" -c security.secureboot="$7" -d "$ROOT" $NET
|
||||||
incus config device add "$1" iso0 disk source="$2" boot.priority=10
|
incus config device add "$1" iso0 disk source="$2" boot.priority=10
|
||||||
echo "made $1 with $2 as its first boot device"
|
echo "made $1 with $2 as its first boot device"
|
||||||
[ "$8" = 1 ] && incus start "$1" && echo "started; open the VGA console to run the installer"
|
[ "$8" = 1 ] && incus start "$1" && echo "started; open the VGA console to run the installer"
|
||||||
exit 0`
|
exit 0`
|
||||||
return script, []string{sp.Name, sp.ISO, cpu, mem, disk, sp.Network, sb, start}, nil
|
return script, []string{sp.Name, sp.ISO, cpu, mem, disk, sp.Network, sb, start, sp.Pool}, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -21,13 +21,13 @@ func TestInstallScriptShapes(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{"grid: zfs, apt, docker", Status{State: "installable", PM: "apt", Arch: "x86_64", Drivers: []string{"zfs", "btrfs", "dir"}, Docker: true, ZFSHost: true, DiskFree: 700 << 30, MemTotal: 62 << 30},
|
{"grid: zfs, apt, docker", Status{State: "installable", PM: "apt", Arch: "x86_64", Drivers: []string{"zfs", "btrfs", "dir"}, Docker: true, ZFSHost: true, DiskFree: 700 << 30, MemTotal: 62 << 30},
|
||||||
InstallOpts{Driver: "zfs", PoolSizeGiB: 300, Subnet: "172.16.100.1/24", DockerRules: true, ARCMaxGiB: 16},
|
InstallOpts{Driver: "zfs", PoolSizeGiB: 300, Subnet: "172.16.100.1/24", DockerRules: true, ARCMaxGiB: 16},
|
||||||
[]string{"apt-get", "zpool create", "incus storage create default zfs source=hamlab-virt", "ipv4.address='172.16.100.1/24'", "DOCKER-USER", "zfs_arc_max", "usermod -aG incus-admin 'core'", "SupplementaryGroups=incus-admin", "systemd-run"}},
|
[]string{"apt-get", "zpool create", "incus storage create local zfs source=hamlab-virt", "ipv4.address='172.16.100.1/24'", "DOCKER-USER", "zfs_arc_max", "usermod -aG incus-admin 'core'", "SupplementaryGroups=incus-admin", "systemd-run"}},
|
||||||
{"core: btrfs, apt, docker, no tools yet", Status{State: "installable", PM: "apt", Arch: "x86_64", Drivers: []string{"btrfs", "dir"}, Docker: true, DiskFree: 170 << 30, MemTotal: 8 << 30},
|
{"core: btrfs, apt, docker, no tools yet", Status{State: "installable", PM: "apt", Arch: "x86_64", Drivers: []string{"btrfs", "dir"}, Docker: true, DiskFree: 170 << 30, MemTotal: 8 << 30},
|
||||||
InstallOpts{Driver: "btrfs", PoolSizeGiB: 40, Subnet: "172.16.100.1/24", DockerRules: true},
|
InstallOpts{Driver: "btrfs", PoolSizeGiB: 40, Subnet: "172.16.100.1/24", DockerRules: true},
|
||||||
[]string{"btrfs-progs", "mkfs.btrfs", "/etc/fstab", "incus storage create default btrfs source="}},
|
[]string{"btrfs-progs", "mkfs.btrfs", "/etc/fstab", "incus storage create local btrfs source="}},
|
||||||
{"mac: dir, dnf, arm64", Status{State: "installable", PM: "dnf", Arch: "aarch64", Drivers: []string{"dir"}, DiskFree: 200 << 30, MemTotal: 15 << 30},
|
{"mac: dir, dnf, arm64", Status{State: "installable", PM: "dnf", Arch: "aarch64", Drivers: []string{"dir"}, DiskFree: 200 << 30, MemTotal: 15 << 30},
|
||||||
InstallOpts{Driver: "dir", Subnet: "172.16.101.1/24"},
|
InstallOpts{Driver: "dir", Subnet: "172.16.101.1/24"},
|
||||||
[]string{"dnf -y install incus qemu-system-aarch64-core edk2-aarch64", "incus storage create default dir source="}},
|
[]string{"dnf -y install incus qemu-system-aarch64-core edk2-aarch64", "incus storage create local dir source="}},
|
||||||
{"repair: already installed", Status{State: "repair", PM: "apt", Arch: "x86_64", Drivers: []string{"dir"}, DiskFree: 10 << 30, MemTotal: 4 << 30},
|
{"repair: already installed", Status{State: "repair", PM: "apt", Arch: "x86_64", Drivers: []string{"dir"}, DiskFree: 10 << 30, MemTotal: 4 << 30},
|
||||||
InstallOpts{Driver: "dir", Subnet: "172.16.100.1/24"},
|
InstallOpts{Driver: "dir", Subnet: "172.16.100.1/24"},
|
||||||
[]string{"already installed"}},
|
[]string{"already installed"}},
|
||||||
@@ -75,6 +75,9 @@ func TestCreateArgs(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
want := "launch images:debian/13 web-1 --vm -c limits.cpu=2 -c limits.memory=2GiB -d root,size=20GiB -n incusbr0"
|
want := "launch images:debian/13 web-1 --vm -c limits.cpu=2 -c limits.memory=2GiB -d root,size=20GiB -n incusbr0"
|
||||||
|
if a, err := CreateArgs(CreateSpec{Name: "x", Image: "images:alpine/3.22", Pool: "local"}, true); err != nil || strings.Join(a, " ") != "init images:alpine/3.22 x -d root,pool=local" {
|
||||||
|
t.Fatalf("pool override: %v %v", a, err)
|
||||||
|
}
|
||||||
if got := strings.Join(args, " "); got != want {
|
if got := strings.Join(args, " "); got != want {
|
||||||
t.Fatalf("got %q", got)
|
t.Fatalf("got %q", got)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,7 +14,6 @@ import (
|
|||||||
"net/netip"
|
"net/netip"
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
"strings"
|
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -125,6 +124,3 @@ func dockerPools() []string {
|
|||||||
}
|
}
|
||||||
return []string{"172.17.0.0/12", "192.168.0.0/16"}
|
return []string{"172.17.0.0/12", "192.168.0.0/16"}
|
||||||
}
|
}
|
||||||
|
|
||||||
// String form used in the layout file and on the page.
|
|
||||||
func prefixString(p netip.Prefix) string { return strings.TrimSpace(p.Masked().String()) }
|
|
||||||
|
|||||||
@@ -130,7 +130,7 @@ func (m *Manager) openVGA(ctx context.Context, client *http.Client, name string,
|
|||||||
resp, err := client.Do(req)
|
resp, err := client.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if isPermission(err) {
|
if isPermission(err) {
|
||||||
return nil, errors.New("Ham Lab's account can't reach Incus's socket; Finish setup on the Virtualization tab puts it in incus-admin")
|
return nil, errors.New("this account can't reach Incus's socket; Finish setup on the Virtualization tab puts Ham Lab in incus-admin")
|
||||||
}
|
}
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,218 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/binary"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"strconv"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
bolt "go.etcd.io/bbolt"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Cache keeps finished previews in one database file under Ham Lab's own data directory —
|
||||||
|
// thumbs.db, a bbolt key/value store: pure Go, so the binary stays static, and one file
|
||||||
|
// per machine rather than a tree of hash-named JPEGs. An entry is one file's preview keyed
|
||||||
|
// by its path, size and mtime, so a file that changes gets a new preview and an unchanged
|
||||||
|
// one is never decoded twice. A file ffmpeg could not decode is kept the same way, with
|
||||||
|
// ffmpeg's reason and no picture, so a folder of odd files does not re-run ffmpeg on every
|
||||||
|
// visit. Nothing expires: a preview is a few kilobytes, and only files somebody actually
|
||||||
|
// scrolled past are ever made. The file is opened the first time something is stored, so a
|
||||||
|
// machine nobody browses with previews on never has one.
|
||||||
|
type Cache struct {
|
||||||
|
Path string
|
||||||
|
mu sync.Mutex
|
||||||
|
db *bolt.DB
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
// Entry is one cached preview, or the remembered reason there is none.
|
||||||
|
type Entry struct {
|
||||||
|
Path string `json:"path"`
|
||||||
|
Engine string `json:"engine,omitempty"`
|
||||||
|
Codec string `json:"codec,omitempty"`
|
||||||
|
Width int `json:"width,omitempty"`
|
||||||
|
Height int `json:"height,omitempty"`
|
||||||
|
Made time.Time `json:"made"`
|
||||||
|
Failure string `json:"failure,omitempty"` // set instead of JPEG when ffmpeg refused the file
|
||||||
|
JPEG []byte `json:"-"`
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
bucketThumbs = []byte("thumbs")
|
||||||
|
bucketMeta = []byte("meta")
|
||||||
|
keyCount = []byte("count")
|
||||||
|
)
|
||||||
|
|
||||||
|
// Key identifies one file's preview. "v1" is the preview format: bump it to retire every
|
||||||
|
// cached preview at once.
|
||||||
|
func Key(path string, size, mtime int64) string {
|
||||||
|
h := sha256.Sum256([]byte("v1\x00" + path + "\x00" + strconv.FormatInt(size, 10) + "\x00" + strconv.FormatInt(mtime, 10)))
|
||||||
|
return hex.EncodeToString(h[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewCache(path string) *Cache { return &Cache{Path: path} }
|
||||||
|
|
||||||
|
// open the database once; an open that failed is not retried, and previews still get made
|
||||||
|
// and served — they are simply not kept, and the log says why.
|
||||||
|
func (c *Cache) open() (*bolt.DB, error) {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
if c.db != nil || c.err != nil {
|
||||||
|
return c.db, c.err
|
||||||
|
}
|
||||||
|
db, err := bolt.Open(c.Path, 0o600, &bolt.Options{Timeout: 2 * time.Second})
|
||||||
|
if err != nil {
|
||||||
|
c.err = err
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := db.Update(func(tx *bolt.Tx) error {
|
||||||
|
if _, err := tx.CreateBucketIfNotExists(bucketThumbs); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
_, err := tx.CreateBucketIfNotExists(bucketMeta)
|
||||||
|
return err
|
||||||
|
}); err != nil {
|
||||||
|
db.Close()
|
||||||
|
c.err = err
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
c.db = db
|
||||||
|
return db, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Close releases the file. Previews after it are made but not kept.
|
||||||
|
func (c *Cache) Close() error {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
if c.db == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
err := c.db.Close()
|
||||||
|
c.db, c.err = nil, errors.New("cache closed")
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get returns the entry for a key. A database that does not exist yet is a miss, not a reason
|
||||||
|
// to make one.
|
||||||
|
func (c *Cache) Get(key string) (Entry, bool, error) {
|
||||||
|
c.mu.Lock()
|
||||||
|
db := c.db
|
||||||
|
c.mu.Unlock()
|
||||||
|
if db == nil {
|
||||||
|
if _, err := os.Stat(c.Path); err != nil {
|
||||||
|
return Entry{}, false, nil
|
||||||
|
}
|
||||||
|
var err error
|
||||||
|
if db, err = c.open(); err != nil {
|
||||||
|
return Entry{}, false, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var e Entry
|
||||||
|
found := false
|
||||||
|
err := db.View(func(tx *bolt.Tx) error {
|
||||||
|
v := tx.Bucket(bucketThumbs).Get([]byte(key))
|
||||||
|
if v == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
var err error
|
||||||
|
e, err = decode(v) // copies: a value is only valid inside the transaction
|
||||||
|
found = err == nil
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
return e, found, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Put stores an entry, replacing any under the same key.
|
||||||
|
func (c *Cache) Put(key string, e Entry) error {
|
||||||
|
if len(e.JPEG) == 0 && e.Failure == "" {
|
||||||
|
return errors.New("empty preview")
|
||||||
|
}
|
||||||
|
if e.Made.IsZero() {
|
||||||
|
e.Made = time.Now()
|
||||||
|
}
|
||||||
|
db, err := c.open()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
v, err := encode(e)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return db.Update(func(tx *bolt.Tx) error {
|
||||||
|
b := tx.Bucket(bucketThumbs)
|
||||||
|
k := []byte(key)
|
||||||
|
fresh := b.Get(k) == nil
|
||||||
|
if err := b.Put(k, v); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if fresh {
|
||||||
|
m := tx.Bucket(bucketMeta)
|
||||||
|
return m.Put(keyCount, be64(count(m)+1))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stats is how many previews are kept and how big the file is.
|
||||||
|
func (c *Cache) Stats() (entries int64, bytes int64) {
|
||||||
|
st, err := os.Stat(c.Path)
|
||||||
|
if err != nil {
|
||||||
|
return 0, 0
|
||||||
|
}
|
||||||
|
bytes = st.Size()
|
||||||
|
db, err := c.open()
|
||||||
|
if err != nil {
|
||||||
|
return 0, bytes
|
||||||
|
}
|
||||||
|
_ = db.View(func(tx *bolt.Tx) error { entries = count(tx.Bucket(bucketMeta)); return nil })
|
||||||
|
return entries, bytes
|
||||||
|
}
|
||||||
|
|
||||||
|
func count(m *bolt.Bucket) int64 {
|
||||||
|
if v := m.Get(keyCount); len(v) == 8 {
|
||||||
|
return int64(binary.BigEndian.Uint64(v))
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func be64(n int64) []byte {
|
||||||
|
b := make([]byte, 8)
|
||||||
|
binary.BigEndian.PutUint64(b, uint64(n))
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
|
||||||
|
// A value is a 4-byte header length, the header as JSON, then the JPEG bytes.
|
||||||
|
func encode(e Entry) ([]byte, error) {
|
||||||
|
hdr, err := json.Marshal(e)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
v := make([]byte, 4+len(hdr)+len(e.JPEG))
|
||||||
|
binary.BigEndian.PutUint32(v, uint32(len(hdr)))
|
||||||
|
copy(v[4:], hdr)
|
||||||
|
copy(v[4+len(hdr):], e.JPEG)
|
||||||
|
return v, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func decode(v []byte) (Entry, error) {
|
||||||
|
if len(v) < 4 {
|
||||||
|
return Entry{}, errors.New("short value")
|
||||||
|
}
|
||||||
|
n := int(binary.BigEndian.Uint32(v))
|
||||||
|
if 4+n > len(v) {
|
||||||
|
return Entry{}, errors.New("bad header length")
|
||||||
|
}
|
||||||
|
var e Entry
|
||||||
|
if err := json.Unmarshal(v[4:4+n], &e); err != nil {
|
||||||
|
return Entry{}, err
|
||||||
|
}
|
||||||
|
if rest := v[4+n:]; len(rest) > 0 {
|
||||||
|
e.JPEG = append([]byte(nil), rest...)
|
||||||
|
}
|
||||||
|
return e, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,222 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
// What to run when a file is to be re-encoded for a browser that cannot play it as it is:
|
||||||
|
// the codecs and sizes on offer, the output size and bit rate a choice comes to for a
|
||||||
|
// given source, the ways this machine could do it in the order worth trying, and the
|
||||||
|
// ffmpeg argument list for one of them. All decisions, no processes, so it can be tested.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"math"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Codec is one target a stream may be re-encoded to. "copy" keeps the codec and only
|
||||||
|
// repackages: for a browser that lacks the container, not the decoder.
|
||||||
|
type Codec struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Label string `json:"label"`
|
||||||
|
}
|
||||||
|
|
||||||
|
var Codecs = []Codec{{"h264", "H.264"}, {"hevc", "HEVC"}, {"av1", "AV1"}, {"copy", "Original codec, repackaged"}}
|
||||||
|
|
||||||
|
// Quality is a picture height; the bit rate follows from the size the output ends up at.
|
||||||
|
type Quality struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Label string `json:"label"`
|
||||||
|
Height int `json:"height"` // 0: the source's own
|
||||||
|
}
|
||||||
|
|
||||||
|
var Qualities = []Quality{{"original", "Original size", 0}, {"1080p", "1080p", 1080}, {"720p", "720p", 720}, {"480p", "480p", 480}}
|
||||||
|
|
||||||
|
func qualityByID(id string) (Quality, bool) {
|
||||||
|
for _, q := range Qualities {
|
||||||
|
if q.ID == id {
|
||||||
|
return q, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return Quality{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
// ValidChoice says whether a codec and quality are ones on offer — checked before any work.
|
||||||
|
func ValidChoice(codec, quality string) error {
|
||||||
|
if !codecKnown(codec) {
|
||||||
|
return fmt.Errorf("unknown codec %q", codec)
|
||||||
|
}
|
||||||
|
if _, ok := qualityByID(quality); !ok {
|
||||||
|
return fmt.Errorf("unknown quality %q", quality)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func codecKnown(id string) bool {
|
||||||
|
for _, c := range Codecs {
|
||||||
|
if c.ID == id {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// fit is the output: never larger than the source, even dimensions, and a bit rate by the
|
||||||
|
// size it comes to — a quarter more at 50 fps and up, less for the codecs that need less.
|
||||||
|
func fit(q Quality, src Info, codec string) (w, h, kbps int) {
|
||||||
|
w, h = src.Video.Width, src.Video.Height
|
||||||
|
if q.Height > 0 && h > q.Height {
|
||||||
|
w = int(math.Round(float64(w) * float64(q.Height) / float64(h)))
|
||||||
|
h = q.Height
|
||||||
|
}
|
||||||
|
w, h = w&^1, h&^1
|
||||||
|
switch {
|
||||||
|
case h >= 2160:
|
||||||
|
kbps = 20000
|
||||||
|
case h >= 1440:
|
||||||
|
kbps = 12000
|
||||||
|
case h >= 1080:
|
||||||
|
kbps = 8000
|
||||||
|
case h >= 720:
|
||||||
|
kbps = 4000
|
||||||
|
default:
|
||||||
|
kbps = 2000
|
||||||
|
}
|
||||||
|
if src.Video.FPS >= 50 {
|
||||||
|
kbps = kbps * 5 / 4
|
||||||
|
}
|
||||||
|
if codec == "hevc" || codec == "av1" {
|
||||||
|
kbps = kbps * 6 / 10
|
||||||
|
}
|
||||||
|
return w, h, kbps
|
||||||
|
}
|
||||||
|
|
||||||
|
// Recipe is one way to make the stream: the engine's name for the record, and the ffmpeg
|
||||||
|
// arguments that go before the input, the video filter, and the video codec.
|
||||||
|
type Recipe struct {
|
||||||
|
Engine string
|
||||||
|
Pre []string
|
||||||
|
VF string
|
||||||
|
Video []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// recipes are the ways to try, best first: a card on the bus before the integrated GPU;
|
||||||
|
// on a GPU, decoding on the same card (frames never leave it) before decoding in software
|
||||||
|
// and uploading; the software encoders last. An empty list means nothing here can.
|
||||||
|
func recipes(c Caps, codec string, src Info, w, h, kbps int) []Recipe {
|
||||||
|
if codec == "copy" {
|
||||||
|
v := []string{"-c:v", "copy"}
|
||||||
|
if src.Video.Codec == "hevc" {
|
||||||
|
v = append(v, "-tag:v", "hvc1") // Safari reads hvc1, not hev1
|
||||||
|
}
|
||||||
|
return []Recipe{{Engine: "Original codec", Video: v}}
|
||||||
|
}
|
||||||
|
var out []Recipe
|
||||||
|
gpus := append([]GPU(nil), c.GPUs...)
|
||||||
|
sort.SliceStable(gpus, func(i, j int) bool { return !gpus[i].Integrated && gpus[j].Integrated })
|
||||||
|
rate := []string{"-b:v", strconv.Itoa(kbps) + "k", "-maxrate", strconv.Itoa(kbps*5/4) + "k", "-bufsize", strconv.Itoa(kbps*5/2) + "k"}
|
||||||
|
for i := range gpus {
|
||||||
|
g := &gpus[i]
|
||||||
|
if g.API != "vaapi" || !has(g.Encodes, codec) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
video := append([]string{"-c:v", codec + "_vaapi"}, rate...)
|
||||||
|
switch codec {
|
||||||
|
case "h264":
|
||||||
|
video = append(video, "-profile:v", "high")
|
||||||
|
case "hevc":
|
||||||
|
video = append(video, "-tag:v", "hvc1")
|
||||||
|
}
|
||||||
|
name := g.Name
|
||||||
|
if name == "" {
|
||||||
|
name = g.Device
|
||||||
|
}
|
||||||
|
scale := fmt.Sprintf("scale_vaapi=w=%d:h=%d:format=nv12", w, h)
|
||||||
|
if has(g.Codecs, src.Video.Codec) {
|
||||||
|
vf := scale
|
||||||
|
if src.HDR() {
|
||||||
|
vf = "tonemap_vaapi=format=nv12:t=bt709:p=bt709:m=bt709," + vf
|
||||||
|
}
|
||||||
|
out = append(out, Recipe{Engine: name + " (vaapi)", Pre: []string{"-hwaccel", "vaapi", "-hwaccel_device", g.Device, "-hwaccel_output_format", "vaapi"}, VF: vf, Video: video})
|
||||||
|
}
|
||||||
|
out = append(out, Recipe{Engine: name + " (vaapi, decoded on the CPU)", Pre: []string{"-init_hw_device", "vaapi=va:" + g.Device, "-filter_hw_device", "va"},
|
||||||
|
VF: fmt.Sprintf("format=nv12,scale=%d:%d,hwupload", w, h), Video: video})
|
||||||
|
}
|
||||||
|
swVF := fmt.Sprintf("scale=%d:%d,format=yuv420p", w, h)
|
||||||
|
if src.HDR() {
|
||||||
|
swVF = fmt.Sprintf("zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable,zscale=t=bt709:m=bt709,scale=%d:%d,format=yuv420p", w, h)
|
||||||
|
}
|
||||||
|
cap := []string{"-maxrate", strconv.Itoa(kbps*5/4) + "k", "-bufsize", strconv.Itoa(kbps*5/2) + "k"}
|
||||||
|
switch {
|
||||||
|
case codec == "h264" && has(c.Encoders, "libx264"):
|
||||||
|
out = append(out, Recipe{Engine: "CPU (libx264)", VF: swVF, Video: append([]string{"-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-profile:v", "high"}, cap...)})
|
||||||
|
case codec == "hevc" && has(c.Encoders, "libx265"):
|
||||||
|
out = append(out, Recipe{Engine: "CPU (libx265)", VF: swVF, Video: append([]string{"-c:v", "libx265", "-preset", "fast", "-crf", "26", "-tag:v", "hvc1"}, cap...)})
|
||||||
|
case codec == "av1" && has(c.Encoders, "libsvtav1"):
|
||||||
|
out = append(out, Recipe{Engine: "CPU (SVT-AV1)", VF: swVF, Video: append([]string{"-c:v", "libsvtav1", "-preset", "8", "-crf", "35"}, cap...)})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// EngineFor is the engine the picker will say a codec runs on, "" when none can.
|
||||||
|
func EngineFor(c Caps, codec string) string {
|
||||||
|
if c.Error != "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
if codec == "copy" {
|
||||||
|
return "Original codec"
|
||||||
|
}
|
||||||
|
r := recipes(c, codec, Info{Video: &Track{Codec: "h264", Width: 1920, Height: 1080}}, 1920, 1080, 8000)
|
||||||
|
if len(r) == 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return r[0].Engine
|
||||||
|
}
|
||||||
|
|
||||||
|
// hlsArgs is the whole ffmpeg argument list for one recipe: the input, the first video and
|
||||||
|
// audio streams, the picture through the recipe, sound as AAC unless it already is, a
|
||||||
|
// keyframe every segment, and an event playlist of fragmented-MP4 segments in dir.
|
||||||
|
func hlsArgs(dir, in string, src Info, r Recipe, copyCodec bool) []string {
|
||||||
|
a := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
|
||||||
|
a = append(a, r.Pre...)
|
||||||
|
a = append(a, "-i", in, "-map", "0:v:0", "-map", "0:a:0?", "-sn", "-dn")
|
||||||
|
if r.VF != "" {
|
||||||
|
a = append(a, "-vf", r.VF)
|
||||||
|
}
|
||||||
|
a = append(a, r.Video...)
|
||||||
|
switch {
|
||||||
|
case src.Audio == nil:
|
||||||
|
a = append(a, "-an")
|
||||||
|
case src.Audio.Codec == "aac":
|
||||||
|
a = append(a, "-c:a", "copy")
|
||||||
|
default:
|
||||||
|
a = append(a, "-c:a", "aac", "-b:a", "160k", "-ac", "2")
|
||||||
|
}
|
||||||
|
if !copyCodec {
|
||||||
|
fps := src.Video.FPS
|
||||||
|
if fps <= 0 {
|
||||||
|
fps = 30
|
||||||
|
}
|
||||||
|
gop := int(math.Round(fps * hlsSeconds))
|
||||||
|
if gop < 24 {
|
||||||
|
gop = 24
|
||||||
|
}
|
||||||
|
if gop > 300 {
|
||||||
|
gop = 300
|
||||||
|
}
|
||||||
|
a = append(a, "-g", strconv.Itoa(gop), "-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%d)", hlsSeconds))
|
||||||
|
}
|
||||||
|
a = append(a, "-f", "hls", "-hls_time", strconv.Itoa(hlsSeconds), "-hls_playlist_type", "event", "-hls_segment_type", "fmp4",
|
||||||
|
"-hls_fmp4_init_filename", "init.mp4", "-hls_segment_filename", dir+"/seg%05d.m4s", "-hls_flags", "independent_segments+temp_file",
|
||||||
|
"-progress", dir+"/progress", "-stats_period", "1", dir+"/index.m3u8")
|
||||||
|
return a
|
||||||
|
}
|
||||||
|
|
||||||
|
const hlsSeconds = 4
|
||||||
|
|
||||||
|
func has(list []string, x string) bool {
|
||||||
|
for _, v := range list {
|
||||||
|
if v == x {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
@@ -0,0 +1,128 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func src(codec string, w, h int, fps float64) Info {
|
||||||
|
return Info{Duration: 100, Video: &Track{Codec: codec, Width: w, Height: h, FPS: fps}, Audio: &Track{Codec: "opus", Channels: 2}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFit(t *testing.T) {
|
||||||
|
q := func(id string) Quality { x, _ := qualityByID(id); return x }
|
||||||
|
for _, c := range []struct {
|
||||||
|
q string
|
||||||
|
in Info
|
||||||
|
codec string
|
||||||
|
w, h, kbs int
|
||||||
|
}{
|
||||||
|
{"1080p", src("hevc", 3840, 2160, 60), "h264", 1920, 1080, 10000},
|
||||||
|
{"original", src("hevc", 3840, 2160, 30), "h264", 3840, 2160, 20000},
|
||||||
|
{"720p", src("h264", 2292, 960, 60), "h264", 1718, 720, 5000}, // odd width made even
|
||||||
|
{"1080p", src("h264", 1280, 720, 30), "h264", 1280, 720, 4000}, // never upscaled
|
||||||
|
{"480p", src("h264", 1920, 1080, 30), "hevc", 852, 480, 1200}, // 853.3 rounds down, then even
|
||||||
|
{"original", src("av1", 640, 360, 24), "av1", 640, 360, 1200},
|
||||||
|
} {
|
||||||
|
w, h, k := fit(q(c.q), c.in, c.codec)
|
||||||
|
if w != c.w || h != c.h || k != c.kbs {
|
||||||
|
t.Errorf("%s %s %dx%d@%.0f → %dx%d %dk, want %dx%d %dk", c.q, c.codec, c.in.Video.Width, c.in.Video.Height, c.in.Video.FPS, w, h, k, c.w, c.h, c.kbs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func gridCaps() Caps {
|
||||||
|
return Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac", "libsvtav1", "libx264", "libx265"},
|
||||||
|
GPUs: []GPU{
|
||||||
|
{API: "vaapi", Device: "/dev/dri/renderD128", Name: "Intel UHD Graphics 750", Integrated: true, Codecs: []string{"h264", "hevc", "vp9"}, Encodes: []string{"h264", "hevc"}},
|
||||||
|
{API: "vaapi", Device: "/dev/dri/renderD129", Name: "Intel Arc A380", Codecs: []string{"h264", "hevc", "vp9"}, Encodes: []string{"av1", "h264", "hevc"}},
|
||||||
|
}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func engineNames(r []Recipe) []string {
|
||||||
|
out := make([]string, len(r))
|
||||||
|
for i := range r {
|
||||||
|
out[i] = r[i].Engine
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRecipesOrder(t *testing.T) {
|
||||||
|
c := gridCaps()
|
||||||
|
// h264 from hevc: the card first, decoding on it; then the iGPU the same way; then the CPU
|
||||||
|
got := engineNames(recipes(c, "h264", src("hevc", 3840, 2160, 60), 1920, 1080, 10000))
|
||||||
|
want := []string{"Intel Arc A380 (vaapi)", "Intel Arc A380 (vaapi, decoded on the CPU)", "Intel UHD Graphics 750 (vaapi)", "Intel UHD Graphics 750 (vaapi, decoded on the CPU)", "CPU (libx264)"}
|
||||||
|
if strings.Join(got, "|") != strings.Join(want, "|") {
|
||||||
|
t.Fatalf("got %v", got)
|
||||||
|
}
|
||||||
|
// av1 source: neither card decodes it, so no same-card recipe, and only the Arc encodes av1
|
||||||
|
got = engineNames(recipes(c, "av1", src("av1", 1920, 1080, 30), 1920, 1080, 4800))
|
||||||
|
want = []string{"Intel Arc A380 (vaapi, decoded on the CPU)", "CPU (SVT-AV1)"}
|
||||||
|
if strings.Join(got, "|") != strings.Join(want, "|") {
|
||||||
|
t.Fatalf("av1: got %v", got)
|
||||||
|
}
|
||||||
|
// copy is one recipe, no filter, hvc1 for hevc
|
||||||
|
r := recipes(c, "copy", src("hevc", 1920, 1080, 30), 1920, 1080, 0)
|
||||||
|
if len(r) != 1 || r[0].VF != "" || !has(r[0].Video, "hvc1") || !has(r[0].Video, "copy") {
|
||||||
|
t.Fatalf("copy: %+v", r)
|
||||||
|
}
|
||||||
|
// a machine with no GPU and no x264: nothing for h264, something for av1
|
||||||
|
bare := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac", "libsvtav1"}}
|
||||||
|
if len(recipes(bare, "h264", src("h264", 1280, 720, 30), 1280, 720, 4000)) != 0 {
|
||||||
|
t.Fatal("no x264 means no h264")
|
||||||
|
}
|
||||||
|
if EngineFor(bare, "av1") != "CPU (SVT-AV1)" || EngineFor(bare, "h264") != "" || EngineFor(bare, "copy") != "Original codec" {
|
||||||
|
t.Fatal("engineFor")
|
||||||
|
}
|
||||||
|
if EngineFor(c, "hevc") != "Intel Arc A380 (vaapi)" {
|
||||||
|
t.Fatalf("engineFor on grid: %q", EngineFor(c, "hevc"))
|
||||||
|
}
|
||||||
|
// HDR gets tone-mapped on both paths
|
||||||
|
hdr := src("hevc", 3840, 2160, 60)
|
||||||
|
hdr.Video.ColorTransfer = "smpte2084"
|
||||||
|
r = recipes(c, "h264", hdr, 1920, 1080, 10000)
|
||||||
|
if !strings.HasPrefix(r[0].VF, "tonemap_vaapi") || !strings.Contains(r[len(r)-1].VF, "tonemap=hable") {
|
||||||
|
t.Fatalf("hdr: %q … %q", r[0].VF, r[len(r)-1].VF)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHLSArgs(t *testing.T) {
|
||||||
|
in := src("h264", 1920, 1080, 60)
|
||||||
|
r := recipes(gridCaps(), "h264", in, 1920, 1080, 10000)[0]
|
||||||
|
a := strings.Join(hlsArgs("/d", "/v.mkv", in, r, false), " ")
|
||||||
|
for _, want := range []string{"-hwaccel vaapi -hwaccel_device /dev/dri/renderD129", "-i /v.mkv", "-map 0:v:0 -map 0:a:0?", "-vf scale_vaapi=w=1920:h=1080:format=nv12", "-c:v h264_vaapi -b:v 10000k -maxrate 12500k", "-c:a aac -b:a 160k -ac 2", "-g 240 -force_key_frames expr:gte(t,n_forced*4)", "-hls_segment_type fmp4", "-hls_segment_filename /d/seg%05d.m4s", "/d/index.m3u8"} {
|
||||||
|
if !strings.Contains(a, want) {
|
||||||
|
t.Errorf("missing %q in\n%s", want, a)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// aac stays as it is; no audio means none; copy sets no keyframes
|
||||||
|
in.Audio.Codec = "aac"
|
||||||
|
if a := strings.Join(hlsArgs("/d", "/v", in, r, false), " "); !strings.Contains(a, "-c:a copy") {
|
||||||
|
t.Error("aac should be copied")
|
||||||
|
}
|
||||||
|
in.Audio = nil
|
||||||
|
if a := strings.Join(hlsArgs("/d", "/v", in, r, false), " "); !strings.Contains(a, " -an ") {
|
||||||
|
t.Error("no audio stream → -an")
|
||||||
|
}
|
||||||
|
cp := recipes(gridCaps(), "copy", in, 1920, 1080, 0)[0]
|
||||||
|
if a := strings.Join(hlsArgs("/d", "/v", in, cp, true), " "); strings.Contains(a, "force_key_frames") || strings.Contains(a, "-vf") {
|
||||||
|
t.Error("copy must not ask for keyframes or filter")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseProgress(t *testing.T) {
|
||||||
|
b := []byte("frame=10\nout_time_us=1000000\nspeed=2.5x\nprogress=continue\nframe=20\nout_time_us=2500000\nspeed=3.1x\nprogress=continue\n")
|
||||||
|
s, sp := parseProgress(b)
|
||||||
|
if s != 2.5 || sp != 3.1 {
|
||||||
|
t.Fatalf("got %v %v", s, sp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidChoice(t *testing.T) {
|
||||||
|
if err := ValidChoice("h264", "720p"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if ValidChoice("vp8", "720p") == nil || ValidChoice("h264", "9p") == nil || ValidChoice("", "") == nil {
|
||||||
|
t.Fatal("bad choices must be refused")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,188 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
// Running one long ffmpeg: as the daemon's user directly, or as root through this binary's
|
||||||
|
// run-ffmpeg helper when the file needs root to read. The helper is told to finish by its
|
||||||
|
// standard input closing, which also happens by itself if the daemon goes away — so a
|
||||||
|
// transcode never outlives the console that asked for it.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"hamlab/internal/priv"
|
||||||
|
)
|
||||||
|
|
||||||
|
type ffproc struct {
|
||||||
|
cmd *exec.Cmd
|
||||||
|
in io.WriteCloser // the helper's stdin, root mode only
|
||||||
|
done chan struct{}
|
||||||
|
err error
|
||||||
|
tail *tailBuf
|
||||||
|
}
|
||||||
|
|
||||||
|
// startFF launches ffmpeg with args, under nice where there is one.
|
||||||
|
func startFF(c Caps, self string, root bool, args []string) (*ffproc, error) {
|
||||||
|
var argv []string
|
||||||
|
if root {
|
||||||
|
argv = priv.Wrap(append([]string{self, "run-ffmpeg"}, args...)...)
|
||||||
|
} else {
|
||||||
|
argv = append([]string{c.FFmpeg}, args...)
|
||||||
|
if c.Nice != "" {
|
||||||
|
argv = append([]string{c.Nice, "-n", "10"}, argv...)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cmd := exec.Command(argv[0], argv[1:]...)
|
||||||
|
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||||
|
p := &ffproc{cmd: cmd, done: make(chan struct{}), tail: &tailBuf{}}
|
||||||
|
cmd.Stderr = p.tail
|
||||||
|
cmd.Stdout = io.Discard
|
||||||
|
if root {
|
||||||
|
in, err := cmd.StdinPipe()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
p.in = in
|
||||||
|
}
|
||||||
|
if err := cmd.Start(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
go func() {
|
||||||
|
p.err = cmd.Wait()
|
||||||
|
close(p.done)
|
||||||
|
}()
|
||||||
|
return p, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// exited reports whether ffmpeg has finished, and how.
|
||||||
|
func (p *ffproc) exited() (bool, error) {
|
||||||
|
select {
|
||||||
|
case <-p.done:
|
||||||
|
return true, p.err
|
||||||
|
default:
|
||||||
|
return false, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// end asks ffmpeg to finish and waits a little for it.
|
||||||
|
func (p *ffproc) end() {
|
||||||
|
if done, _ := p.exited(); done {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if p.in != nil {
|
||||||
|
p.in.Close()
|
||||||
|
} else {
|
||||||
|
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGTERM)
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-p.done:
|
||||||
|
case <-time.After(5 * time.Second):
|
||||||
|
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGKILL) // a root helper ignores this; it ends on its own within its own grace
|
||||||
|
select {
|
||||||
|
case <-p.done:
|
||||||
|
case <-time.After(10 * time.Second):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// lastWords is what ffmpeg said last, for the record.
|
||||||
|
func (p *ffproc) lastWords() string { return p.tail.last() }
|
||||||
|
|
||||||
|
// tailBuf keeps the last few KB written to it.
|
||||||
|
type tailBuf struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
b []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
const tailKeep = 4096
|
||||||
|
|
||||||
|
func (t *tailBuf) Write(b []byte) (int, error) {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
t.b = append(t.b, b...)
|
||||||
|
if len(t.b) > tailKeep {
|
||||||
|
t.b = t.b[len(t.b)-tailKeep:]
|
||||||
|
}
|
||||||
|
return len(b), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *tailBuf) last() string {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
return lastLine(string(bytes.TrimSpace(t.b)))
|
||||||
|
}
|
||||||
|
|
||||||
|
// RunFFmpegHelper implements `hamlab run-ffmpeg <args…>`: ffmpeg as this user (root, when
|
||||||
|
// the daemon asked through sudo), finishing when stdin closes. Returns ffmpeg's exit code.
|
||||||
|
func RunFFmpegHelper(args []string, stdin io.Reader, stderr io.Writer) int {
|
||||||
|
ffmpeg, err := exec.LookPath("ffmpeg")
|
||||||
|
if err != nil {
|
||||||
|
io.WriteString(stderr, "ffmpeg is not installed\n")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
argv := []string{ffmpeg}
|
||||||
|
if nice, err := exec.LookPath("nice"); err == nil {
|
||||||
|
argv = []string{nice, "-n", "10", ffmpeg}
|
||||||
|
}
|
||||||
|
cmd := exec.Command(argv[0], append(argv[1:], args...)...)
|
||||||
|
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||||
|
cmd.Stderr = stderr
|
||||||
|
cmd.Stdout = io.Discard
|
||||||
|
if err := cmd.Start(); err != nil {
|
||||||
|
io.WriteString(stderr, err.Error()+"\n")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
go func() {
|
||||||
|
_, _ = io.Copy(io.Discard, stdin) // returns when the daemon lets go
|
||||||
|
cancel()
|
||||||
|
}()
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() { done <- cmd.Wait() }()
|
||||||
|
var werr error
|
||||||
|
select {
|
||||||
|
case werr = <-done:
|
||||||
|
case <-ctx.Done():
|
||||||
|
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM)
|
||||||
|
select {
|
||||||
|
case werr = <-done:
|
||||||
|
case <-time.After(5 * time.Second):
|
||||||
|
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||||
|
werr = <-done
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var ee *exec.ExitError
|
||||||
|
if errors.As(werr, &ee) {
|
||||||
|
return ee.ExitCode()
|
||||||
|
}
|
||||||
|
if werr != nil {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// clearDir empties a folder without removing it.
|
||||||
|
func clearDir(dir string) error {
|
||||||
|
ents, err := os.ReadDir(dir)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, e := range ents {
|
||||||
|
if err := os.RemoveAll(dir + "/" + e.Name()); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasSegments(playlist string) bool {
|
||||||
|
b, err := os.ReadFile(playlist)
|
||||||
|
return err == nil && strings.Contains(string(b), "#EXTINF")
|
||||||
|
}
|
||||||
@@ -0,0 +1,344 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
// Playing what the browser cannot play as it is: an HLS stream, re-encoded on demand by
|
||||||
|
// ffmpeg on the machine's own GPU where it has one it can use, on its CPU otherwise. A
|
||||||
|
// stream is one file at one codec and size, in a folder of its own under
|
||||||
|
// <data dir>/transcode/: ffmpeg writes an event playlist and fragmented-MP4 segments there
|
||||||
|
// as fast as it can, and the player reads them as they land, so the range it can seek in
|
||||||
|
// grows until the whole file is through — on a GPU, several times faster than watching.
|
||||||
|
// Nothing is kept: a stream nobody has touched for a while is ended and its folder
|
||||||
|
// removed, and the folder is emptied when the daemon starts.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/hex"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Stream is one re-encode, in progress or finished.
|
||||||
|
type Stream struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Path string `json:"path"`
|
||||||
|
Codec string `json:"codec"`
|
||||||
|
Quality string `json:"quality"`
|
||||||
|
Engine string `json:"engine"`
|
||||||
|
Width int `json:"width"`
|
||||||
|
Height int `json:"height"`
|
||||||
|
Kbps int `json:"kbps"`
|
||||||
|
Source Info `json:"source"`
|
||||||
|
Started time.Time `json:"started"`
|
||||||
|
Root bool `json:"root"`
|
||||||
|
|
||||||
|
key string
|
||||||
|
dir string
|
||||||
|
mu sync.Mutex
|
||||||
|
used time.Time
|
||||||
|
proc *ffproc
|
||||||
|
finished bool
|
||||||
|
failure string
|
||||||
|
gone bool // forgotten by the Streamer
|
||||||
|
}
|
||||||
|
|
||||||
|
// StreamStatus is what the player asks for while the file is still going through.
|
||||||
|
type StreamStatus struct {
|
||||||
|
Running bool `json:"running"`
|
||||||
|
Done bool `json:"done"`
|
||||||
|
Error string `json:"error,omitempty"`
|
||||||
|
Progress float64 `json:"progress"` // seconds of output written
|
||||||
|
Speed float64 `json:"speed"` // × realtime
|
||||||
|
Segments int `json:"segments"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Streamer keeps one machine's streams.
|
||||||
|
type Streamer struct {
|
||||||
|
dir string
|
||||||
|
self string
|
||||||
|
limit int
|
||||||
|
mu sync.Mutex
|
||||||
|
streams map[string]*Stream
|
||||||
|
}
|
||||||
|
|
||||||
|
const streamIdle = 10 * time.Minute
|
||||||
|
const firstSegmentWait = 40 * time.Second
|
||||||
|
|
||||||
|
var streamFile = regexp.MustCompile(`^(index\.m3u8|init\.mp4|seg\d{5}\.m4s)$`)
|
||||||
|
|
||||||
|
// NewStreamer empties the folder — a daemon restart orphans every stream — and starts the
|
||||||
|
// sweeper.
|
||||||
|
func NewStreamer(dataDir, self string, limit int) *Streamer {
|
||||||
|
m := &Streamer{dir: filepath.Join(dataDir, "transcode"), self: self, limit: limit, streams: map[string]*Stream{}}
|
||||||
|
_ = os.MkdirAll(m.dir, 0o755)
|
||||||
|
_ = clearDir(m.dir)
|
||||||
|
go func() {
|
||||||
|
for range time.Tick(30 * time.Second) {
|
||||||
|
m.Sweep(streamIdle)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
// Open starts a stream, or hands back the one already serving the same request. It
|
||||||
|
// returns once the first segment is on disk, so the player's first fetch is answered.
|
||||||
|
func (m *Streamer) Open(ctx context.Context, c Caps, root bool, path, codec, quality string, src Info) (*Stream, error) {
|
||||||
|
if c.Error != "" {
|
||||||
|
return nil, errors.New(c.Error)
|
||||||
|
}
|
||||||
|
if err := ValidChoice(codec, quality); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
q, _ := qualityByID(quality)
|
||||||
|
if src.Video == nil {
|
||||||
|
return nil, &DecodeError{Msg: "no video stream"}
|
||||||
|
}
|
||||||
|
if src.Video.Width <= 0 || src.Video.Height <= 0 {
|
||||||
|
return nil, &DecodeError{Msg: "the picture's size is unknown"}
|
||||||
|
}
|
||||||
|
key := fmt.Sprintf("%s|%d|%.0f|%s|%s|%v", path, src.Size, src.Duration, codec, quality, root)
|
||||||
|
m.mu.Lock()
|
||||||
|
for _, s := range m.streams {
|
||||||
|
if s.key == key {
|
||||||
|
if s.failed() { // not the stream to hand back: let it go and make a fresh one
|
||||||
|
delete(m.streams, s.ID)
|
||||||
|
_ = os.RemoveAll(s.dir)
|
||||||
|
break
|
||||||
|
}
|
||||||
|
m.mu.Unlock()
|
||||||
|
s.touch()
|
||||||
|
return s, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
running := 0
|
||||||
|
for _, s := range m.streams {
|
||||||
|
if s.running() {
|
||||||
|
running++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if running >= m.limit {
|
||||||
|
m.mu.Unlock()
|
||||||
|
return nil, fmt.Errorf("%d videos are already being re-encoded on this machine; one must finish or be closed first", running)
|
||||||
|
}
|
||||||
|
s := &Stream{ID: newID(), Path: path, Codec: codec, Quality: quality, Source: src, Started: time.Now(), Root: root, key: key, used: time.Now()}
|
||||||
|
s.dir = filepath.Join(m.dir, s.ID)
|
||||||
|
m.streams[s.ID] = s
|
||||||
|
m.mu.Unlock()
|
||||||
|
|
||||||
|
s.Width, s.Height, s.Kbps = fit(q, src, codec)
|
||||||
|
if codec == "copy" {
|
||||||
|
s.Width, s.Height, s.Kbps = src.Video.Width, src.Video.Height, int(src.BitRate/1000)
|
||||||
|
}
|
||||||
|
rs := recipes(c, codec, src, s.Width, s.Height, s.Kbps)
|
||||||
|
if len(rs) == 0 {
|
||||||
|
m.forget(s)
|
||||||
|
return nil, errors.New("nothing on this machine encodes " + codec)
|
||||||
|
}
|
||||||
|
var last string
|
||||||
|
for _, r := range rs {
|
||||||
|
if err := os.MkdirAll(s.dir, 0o755); err != nil { // the root helper's ffmpeg writes here too
|
||||||
|
m.forget(s)
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
_ = clearDir(s.dir)
|
||||||
|
args := hlsArgs(s.dir, path, src, r, codec == "copy")
|
||||||
|
p, err := startFF(c, m.self, root, args)
|
||||||
|
if err != nil {
|
||||||
|
last = r.Engine + ": " + err.Error()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
s.mu.Lock()
|
||||||
|
s.proc, s.Engine = p, r.Engine
|
||||||
|
s.mu.Unlock()
|
||||||
|
msg, ok := s.waitFirstSegment(ctx)
|
||||||
|
if ok {
|
||||||
|
go s.watch()
|
||||||
|
log.Printf("media: stream %s: %s → %s %s %dx%d %dk on %s", s.ID, filepath.Base(path), codec, quality, s.Width, s.Height, s.Kbps, r.Engine)
|
||||||
|
return s, nil
|
||||||
|
}
|
||||||
|
p.end()
|
||||||
|
if ctx.Err() != nil || s.closed() { // the caller left, or somebody closed it meanwhile: not the recipe's fault
|
||||||
|
m.forget(s)
|
||||||
|
return nil, errors.New("cancelled")
|
||||||
|
}
|
||||||
|
last = r.Engine + ": " + msg
|
||||||
|
log.Printf("media: stream %s on %s: %s", filepath.Base(path), r.Engine, msg)
|
||||||
|
}
|
||||||
|
m.forget(s)
|
||||||
|
return nil, &DecodeError{Msg: last}
|
||||||
|
}
|
||||||
|
|
||||||
|
// waitFirstSegment watches for the playlist's first segment, or ffmpeg giving up.
|
||||||
|
func (s *Stream) waitFirstSegment(ctx context.Context) (string, bool) {
|
||||||
|
deadline := time.Now().Add(firstSegmentWait)
|
||||||
|
playlist := filepath.Join(s.dir, "index.m3u8")
|
||||||
|
for {
|
||||||
|
if hasSegments(playlist) {
|
||||||
|
return "", true
|
||||||
|
}
|
||||||
|
if done, err := s.proc.exited(); done {
|
||||||
|
msg := s.proc.lastWords()
|
||||||
|
if msg == "" && err != nil {
|
||||||
|
msg = err.Error()
|
||||||
|
}
|
||||||
|
if msg == "" {
|
||||||
|
msg = "ffmpeg ended before writing a segment"
|
||||||
|
}
|
||||||
|
return msg, false
|
||||||
|
}
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
return "no segment after " + firstSegmentWait.String(), false
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err().Error(), false
|
||||||
|
case <-time.After(250 * time.Millisecond):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// watch records how the encode ended.
|
||||||
|
func (s *Stream) watch() {
|
||||||
|
<-s.proc.done
|
||||||
|
s.mu.Lock()
|
||||||
|
s.finished = true
|
||||||
|
if s.proc.err != nil {
|
||||||
|
s.failure = s.proc.lastWords()
|
||||||
|
if s.failure == "" {
|
||||||
|
s.failure = s.proc.err.Error()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
s.mu.Unlock()
|
||||||
|
if s.failure != "" {
|
||||||
|
log.Printf("media: stream %s ended early: %s", s.ID, s.failure)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Streamer) Find(id string) (*Stream, bool) {
|
||||||
|
m.mu.Lock()
|
||||||
|
defer m.mu.Unlock()
|
||||||
|
s, ok := m.streams[id]
|
||||||
|
return s, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
// Close ends a stream and removes its folder.
|
||||||
|
func (m *Streamer) Close(id string) {
|
||||||
|
m.mu.Lock()
|
||||||
|
s, ok := m.streams[id]
|
||||||
|
m.mu.Unlock()
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.mu.Lock()
|
||||||
|
p := s.proc
|
||||||
|
s.mu.Unlock()
|
||||||
|
if p != nil {
|
||||||
|
p.end()
|
||||||
|
}
|
||||||
|
m.forget(s)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Streamer) forget(s *Stream) {
|
||||||
|
m.mu.Lock()
|
||||||
|
delete(m.streams, s.ID)
|
||||||
|
m.mu.Unlock()
|
||||||
|
s.mu.Lock()
|
||||||
|
s.gone = true
|
||||||
|
s.mu.Unlock()
|
||||||
|
_ = os.RemoveAll(s.dir)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Stream) closed() bool { s.mu.Lock(); defer s.mu.Unlock(); return s.gone }
|
||||||
|
|
||||||
|
func (s *Stream) failed() bool {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
return s.finished && s.failure != ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sweep closes what nobody has touched for a while.
|
||||||
|
func (m *Streamer) Sweep(idle time.Duration) {
|
||||||
|
m.mu.Lock()
|
||||||
|
var old []string
|
||||||
|
for id, s := range m.streams {
|
||||||
|
s.mu.Lock()
|
||||||
|
if time.Since(s.used) > idle {
|
||||||
|
old = append(old, id)
|
||||||
|
}
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
m.mu.Unlock()
|
||||||
|
for _, id := range old {
|
||||||
|
log.Printf("media: stream %s idle, closed", id)
|
||||||
|
m.Close(id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// File is the on-disk path of one of the stream's files — the playlist, the init segment
|
||||||
|
// or a media segment — and nothing else.
|
||||||
|
func (s *Stream) File(name string) (string, bool) {
|
||||||
|
if !streamFile.MatchString(name) {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
s.touch()
|
||||||
|
return filepath.Join(s.dir, name), true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Stream) touch() { s.mu.Lock(); s.used = time.Now(); s.mu.Unlock() }
|
||||||
|
|
||||||
|
func (s *Stream) running() bool {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
return s.proc != nil && !s.finished
|
||||||
|
}
|
||||||
|
|
||||||
|
// Status is read from ffmpeg's own progress file and the playlist.
|
||||||
|
func (s *Stream) Status() StreamStatus {
|
||||||
|
s.mu.Lock()
|
||||||
|
st := StreamStatus{Running: s.proc != nil && !s.finished, Done: s.finished && s.failure == "", Error: s.failure}
|
||||||
|
s.mu.Unlock()
|
||||||
|
if b, err := os.ReadFile(filepath.Join(s.dir, "progress")); err == nil {
|
||||||
|
st.Progress, st.Speed = parseProgress(b)
|
||||||
|
}
|
||||||
|
if b, err := os.ReadFile(filepath.Join(s.dir, "index.m3u8")); err == nil {
|
||||||
|
st.Segments = strings.Count(string(b), "#EXTINF")
|
||||||
|
}
|
||||||
|
return st
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseProgress takes the last out_time_us and speed from ffmpeg's -progress output.
|
||||||
|
func parseProgress(b []byte) (seconds, speed float64) {
|
||||||
|
sc := bufio.NewScanner(strings.NewReader(string(b)))
|
||||||
|
for sc.Scan() {
|
||||||
|
k, v, ok := strings.Cut(sc.Text(), "=")
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
v = strings.TrimSpace(v)
|
||||||
|
switch k {
|
||||||
|
case "out_time_us":
|
||||||
|
if us, err := strconv.ParseInt(v, 10, 64); err == nil && us >= 0 {
|
||||||
|
seconds = float64(us) / 1e6
|
||||||
|
}
|
||||||
|
case "speed":
|
||||||
|
if x, err := strconv.ParseFloat(strings.TrimSuffix(v, "x"), 64); err == nil {
|
||||||
|
speed = x
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return seconds, speed
|
||||||
|
}
|
||||||
|
|
||||||
|
func newID() string {
|
||||||
|
b := make([]byte, 8)
|
||||||
|
_, _ = rand.Read(b)
|
||||||
|
return hex.EncodeToString(b)
|
||||||
|
}
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The real thing, when ffmpeg is here: a generated clip goes through as H.264 and comes out
|
||||||
|
// as a playlist of segments, the status follows it to the end, and closing removes it all.
|
||||||
|
func TestStreamWithFFmpeg(t *testing.T) {
|
||||||
|
if _, err := exec.LookPath("ffmpeg"); err != nil {
|
||||||
|
t.Skip("no ffmpeg")
|
||||||
|
}
|
||||||
|
c := Probe(context.Background())
|
||||||
|
if c.Error != "" {
|
||||||
|
t.Skip(c.Error)
|
||||||
|
}
|
||||||
|
if EngineFor(c, "h264") == "" {
|
||||||
|
t.Skip("nothing here encodes h264")
|
||||||
|
}
|
||||||
|
dir := t.TempDir()
|
||||||
|
clip := filepath.Join(dir, "clip.mp4")
|
||||||
|
out, err := exec.Command("ffmpeg", "-nostdin", "-v", "error", "-f", "lavfi", "-i", "testsrc=size=640x360:rate=25:duration=9", "-f", "lavfi", "-i", "sine=frequency=440:duration=9",
|
||||||
|
"-c:v", "mpeg4", "-q:v", "5", "-c:a", "aac", "-shortest", clip).CombinedOutput()
|
||||||
|
if err != nil {
|
||||||
|
t.Skipf("could not make a clip: %s", out)
|
||||||
|
}
|
||||||
|
src, err := Inspect(context.Background(), c, clip, false)
|
||||||
|
if err != nil || src.Video == nil || src.Audio == nil || src.Video.Codec != "mpeg4" || src.Duration < 8 {
|
||||||
|
t.Fatalf("inspect: %+v %v", src, err)
|
||||||
|
}
|
||||||
|
m := NewStreamer(dir, "hamlab", 2)
|
||||||
|
s, err := m.Open(context.Background(), c, false, clip, "h264", "480p", src)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if s.Width != 640 || s.Height != 360 || s.Engine == "" {
|
||||||
|
t.Fatalf("stream: %+v", s)
|
||||||
|
}
|
||||||
|
if p, ok := s.File("index.m3u8"); !ok {
|
||||||
|
t.Fatal("no playlist")
|
||||||
|
} else if !hasSegments(p) {
|
||||||
|
t.Fatal("Open returned before the first segment")
|
||||||
|
}
|
||||||
|
for _, bad := range []string{"../x", "progress", "seg1.m4s", "index.m3u8/", "init.mp4.tmp"} {
|
||||||
|
if _, ok := s.File(bad); ok {
|
||||||
|
t.Fatalf("%q must not be served", bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// the same request is the same stream
|
||||||
|
if again, err := m.Open(context.Background(), c, false, clip, "h264", "480p", src); err != nil || again.ID != s.ID {
|
||||||
|
t.Fatal("not reused")
|
||||||
|
}
|
||||||
|
var st StreamStatus
|
||||||
|
for i := 0; i < 120; i++ {
|
||||||
|
st = s.Status()
|
||||||
|
if st.Done || st.Error != "" {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
time.Sleep(250 * time.Millisecond)
|
||||||
|
}
|
||||||
|
if !st.Done || st.Error != "" || st.Segments < 2 || st.Progress < 8 {
|
||||||
|
t.Fatalf("status at the end: %+v", st)
|
||||||
|
}
|
||||||
|
if init, _ := s.File("init.mp4"); !exists(init) {
|
||||||
|
t.Fatal("no init segment")
|
||||||
|
}
|
||||||
|
m.Close(s.ID)
|
||||||
|
if _, ok := m.Find(s.ID); ok || exists(s.dir) {
|
||||||
|
t.Fatal("close must forget and remove")
|
||||||
|
}
|
||||||
|
// an unplayable source is a DecodeError, and leaves nothing behind
|
||||||
|
bad := filepath.Join(dir, "bad.mkv")
|
||||||
|
_ = os.WriteFile(bad, []byte("not a video"), 0o644)
|
||||||
|
if _, err := Inspect(context.Background(), c, bad, false); err == nil {
|
||||||
|
t.Fatal("inspect of garbage")
|
||||||
|
}
|
||||||
|
if left, _ := os.ReadDir(m.dir); len(left) != 0 {
|
||||||
|
t.Fatalf("left behind: %v", left)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,142 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"os/exec"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"hamlab/internal/priv"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Track is one stream of a media file, as ffprobe reports it.
|
||||||
|
type Track struct {
|
||||||
|
Index int `json:"index"`
|
||||||
|
Type string `json:"type"` // video | audio
|
||||||
|
Codec string `json:"codec"`
|
||||||
|
Profile string `json:"profile,omitempty"`
|
||||||
|
PixFmt string `json:"pix_fmt,omitempty"`
|
||||||
|
Width int `json:"width,omitempty"`
|
||||||
|
Height int `json:"height,omitempty"`
|
||||||
|
FPS float64 `json:"fps,omitempty"`
|
||||||
|
ColorTransfer string `json:"color_transfer,omitempty"` // smpte2084 / arib-std-b67 mean HDR
|
||||||
|
Channels int `json:"channels,omitempty"`
|
||||||
|
AttachedPic bool `json:"attached_pic,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Info is what a file holds: the picture to take, the sound to go with it, and the whole.
|
||||||
|
type Info struct {
|
||||||
|
Container string `json:"container"` // "matroska,webm", "mov,mp4,m4a,3gp,3g2,mj2"
|
||||||
|
Duration float64 `json:"duration"`
|
||||||
|
BitRate int64 `json:"bit_rate,omitempty"` // bits per second, whole file
|
||||||
|
Size int64 `json:"size,omitempty"`
|
||||||
|
Video *Track `json:"video,omitempty"` // the first real picture stream; cover art is not it
|
||||||
|
Audio *Track `json:"audio,omitempty"` // the first sound stream
|
||||||
|
}
|
||||||
|
|
||||||
|
// HDR reports a picture that needs tone mapping to look right on an SDR screen.
|
||||||
|
func (i Info) HDR() bool {
|
||||||
|
return i.Video != nil && (i.Video.ColorTransfer == "smpte2084" || i.Video.ColorTransfer == "arib-std-b67")
|
||||||
|
}
|
||||||
|
|
||||||
|
type probeJSON struct {
|
||||||
|
Streams []struct {
|
||||||
|
Index int `json:"index"`
|
||||||
|
CodecType string `json:"codec_type"`
|
||||||
|
CodecName string `json:"codec_name"`
|
||||||
|
Profile string `json:"profile"`
|
||||||
|
PixFmt string `json:"pix_fmt"`
|
||||||
|
Width int `json:"width"`
|
||||||
|
Height int `json:"height"`
|
||||||
|
RFrameRate string `json:"r_frame_rate"`
|
||||||
|
AvgFrameRate string `json:"avg_frame_rate"`
|
||||||
|
ColorTransfer string `json:"color_transfer"`
|
||||||
|
Channels int `json:"channels"`
|
||||||
|
Disposition struct {
|
||||||
|
AttachedPic int `json:"attached_pic"`
|
||||||
|
} `json:"disposition"`
|
||||||
|
} `json:"streams"`
|
||||||
|
Format struct {
|
||||||
|
FormatName string `json:"format_name"`
|
||||||
|
Duration string `json:"duration"`
|
||||||
|
BitRate string `json:"bit_rate"`
|
||||||
|
Size string `json:"size"`
|
||||||
|
} `json:"format"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Inspect asks ffprobe what a file is. As root through sudo when asked, for a file only
|
||||||
|
// root may read. An error about the file itself is a DecodeError.
|
||||||
|
func Inspect(ctx context.Context, c Caps, path string, root bool) (Info, error) {
|
||||||
|
if c.FFprobe == "" {
|
||||||
|
return Info{}, errors.New("ffprobe is not installed")
|
||||||
|
}
|
||||||
|
argv := []string{c.FFprobe, "-v", "error", "-show_entries",
|
||||||
|
"stream=index,codec_type,codec_name,profile,pix_fmt,width,height,r_frame_rate,avg_frame_rate,color_transfer,channels,disposition:format=format_name,duration,bit_rate,size",
|
||||||
|
"-of", "json", path}
|
||||||
|
if root {
|
||||||
|
argv = priv.Wrap(argv...)
|
||||||
|
}
|
||||||
|
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
|
||||||
|
var errb bytes.Buffer
|
||||||
|
cmd.Stderr = &errb
|
||||||
|
out, err := cmd.Output()
|
||||||
|
if err != nil {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return Info{}, ctx.Err()
|
||||||
|
}
|
||||||
|
msg := lastLine(errb.String())
|
||||||
|
if msg == "" {
|
||||||
|
msg = "ffprobe: " + err.Error()
|
||||||
|
}
|
||||||
|
return Info{}, &DecodeError{Msg: msg}
|
||||||
|
}
|
||||||
|
var pj probeJSON
|
||||||
|
if err := json.Unmarshal(out, &pj); err != nil {
|
||||||
|
return Info{}, &DecodeError{Msg: "ffprobe: unreadable answer"}
|
||||||
|
}
|
||||||
|
in := Info{Container: pj.Format.FormatName}
|
||||||
|
in.Duration, _ = strconv.ParseFloat(pj.Format.Duration, 64)
|
||||||
|
in.BitRate, _ = strconv.ParseInt(pj.Format.BitRate, 10, 64)
|
||||||
|
in.Size, _ = strconv.ParseInt(pj.Format.Size, 10, 64)
|
||||||
|
var art *Track
|
||||||
|
for _, s := range pj.Streams {
|
||||||
|
st := &Track{Index: s.Index, Type: s.CodecType, Codec: s.CodecName, Profile: s.Profile, PixFmt: s.PixFmt, Width: s.Width, Height: s.Height,
|
||||||
|
ColorTransfer: s.ColorTransfer, Channels: s.Channels, AttachedPic: s.Disposition.AttachedPic != 0}
|
||||||
|
st.FPS = fps(s.AvgFrameRate)
|
||||||
|
if st.FPS == 0 {
|
||||||
|
st.FPS = fps(s.RFrameRate)
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case s.CodecType == "video" && st.AttachedPic:
|
||||||
|
if art == nil {
|
||||||
|
art = st
|
||||||
|
}
|
||||||
|
case s.CodecType == "video" && in.Video == nil:
|
||||||
|
in.Video = st
|
||||||
|
case s.CodecType == "audio" && in.Audio == nil:
|
||||||
|
in.Audio = st
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if in.Video == nil {
|
||||||
|
in.Video = art // cover art is the only picture there is
|
||||||
|
}
|
||||||
|
return in, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// fps parses ffprobe's "60000/1001".
|
||||||
|
func fps(r string) float64 {
|
||||||
|
n, d, ok := strings.Cut(r, "/")
|
||||||
|
if !ok {
|
||||||
|
v, _ := strconv.ParseFloat(r, 64)
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
nf, _ := strconv.ParseFloat(n, 64)
|
||||||
|
df, _ := strconv.ParseFloat(d, 64)
|
||||||
|
if df == 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return nf / df
|
||||||
|
}
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"os/user"
|
||||||
|
"strconv"
|
||||||
|
"syscall"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ownerOf is "root:render 0660" for a device node, the thing to say when it cannot be opened.
|
||||||
|
func ownerOf(st os.FileInfo) string {
|
||||||
|
sys, ok := st.Sys().(*syscall.Stat_t)
|
||||||
|
if !ok {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
u := strconv.Itoa(int(sys.Uid))
|
||||||
|
if x, err := user.LookupId(u); err == nil {
|
||||||
|
u = x.Username
|
||||||
|
}
|
||||||
|
g := strconv.Itoa(int(sys.Gid))
|
||||||
|
if x, err := user.LookupGroupId(g); err == nil {
|
||||||
|
g = x.Name
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%s:%s %04o", u, g, st.Mode().Perm())
|
||||||
|
}
|
||||||
@@ -0,0 +1,656 @@
|
|||||||
|
// Package thumbs makes the Files tab's previews: one small JPEG for an image or a video,
|
||||||
|
// by ffmpeg, decoded on a GPU where the machine has one it can use and on the CPU
|
||||||
|
// otherwise.
|
||||||
|
//
|
||||||
|
// Nothing about the machine is assumed. A probe looks ffmpeg and ffprobe up, opens every
|
||||||
|
// /dev/dri/renderD* and exercises it (a VAAPI upload and download of a synthetic frame),
|
||||||
|
// so a node the daemon's user cannot open, or a driver that does not speak VAAPI, is
|
||||||
|
// reported once rather than tried per file. vainfo, where installed, says which codecs
|
||||||
|
// each node decodes, and a video goes to the first node that lists its codec; without
|
||||||
|
// vainfo the first working node is tried and ffmpeg falls back to software on its own when
|
||||||
|
// the stream turns out unsupported. A hardware run that fails outright is retried on the
|
||||||
|
// CPU before the file is called undecodable. Images are always decoded on the CPU: one
|
||||||
|
// frame of a JPEG at a reduced DCT size costs less than the trip through a GPU.
|
||||||
|
package media
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"hamlab/internal/metrics"
|
||||||
|
"hamlab/internal/priv"
|
||||||
|
)
|
||||||
|
|
||||||
|
// GPU is one hardware decoder as the probe found it.
|
||||||
|
type GPU struct {
|
||||||
|
API string `json:"api"` // vaapi | cuda
|
||||||
|
Device string `json:"device"` // /dev/dri/renderD128; "0" for cuda (its device index)
|
||||||
|
Name string `json:"name,omitempty"` // "Intel Arc A380"
|
||||||
|
Codecs []string `json:"codecs,omitempty"` // ffmpeg codec names it decodes; nil when nothing could say
|
||||||
|
Encodes []string `json:"encodes,omitempty"` // ffmpeg codec names it encodes, from vainfo
|
||||||
|
Integrated bool `json:"integrated,omitempty"` // on the CPU die; a card on the bus is preferred for encoding
|
||||||
|
Reason string `json:"reason,omitempty"` // Unusable only: why
|
||||||
|
Permission bool `json:"permission,omitempty"` // Unusable only: the node exists but this user may not open it
|
||||||
|
}
|
||||||
|
|
||||||
|
// Caps is what the machine can make previews with, as probed.
|
||||||
|
type Caps struct {
|
||||||
|
FFmpeg string `json:"ffmpeg,omitempty"`
|
||||||
|
FFprobe string `json:"ffprobe,omitempty"`
|
||||||
|
Nice string `json:"nice,omitempty"`
|
||||||
|
Version string `json:"version,omitempty"`
|
||||||
|
GPUs []GPU `json:"gpus"` // usable, in node order
|
||||||
|
Encoders []string `json:"encoders,omitempty"` // the encoders ffmpeg itself was built with that matter here
|
||||||
|
Unusable []GPU `json:"unusable,omitempty"` // render nodes that exist but cannot be used, with the reason
|
||||||
|
Error string `json:"error,omitempty"` // why nothing can be made at all
|
||||||
|
ProbedAt time.Time `json:"probed_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Opts shapes one preview.
|
||||||
|
type Opts struct {
|
||||||
|
Size int // the long edge in pixels; 256 when zero
|
||||||
|
Timeout time.Duration // for the whole job, ffprobe and every ffmpeg attempt; 30 s when zero
|
||||||
|
}
|
||||||
|
|
||||||
|
func (o Opts) defaults() Opts {
|
||||||
|
if o.Size <= 0 {
|
||||||
|
o.Size = 256
|
||||||
|
}
|
||||||
|
if o.Timeout <= 0 {
|
||||||
|
o.Timeout = 30 * time.Second
|
||||||
|
}
|
||||||
|
return o
|
||||||
|
}
|
||||||
|
|
||||||
|
// Result is a finished preview.
|
||||||
|
type Result struct {
|
||||||
|
JPEG []byte `json:"-"`
|
||||||
|
Engine string `json:"engine"` // "cpu", or "vaapi:/dev/dri/renderD129"
|
||||||
|
Codec string `json:"codec"`
|
||||||
|
Width int `json:"width"`
|
||||||
|
Height int `json:"height"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ErrUnsupported: not an image or a video by its name.
|
||||||
|
var ErrUnsupported = errors.New("not an image or video")
|
||||||
|
|
||||||
|
// A DecodeError is ffmpeg's verdict on the file itself. It will not change until the
|
||||||
|
// file does, so the caller may remember it.
|
||||||
|
type DecodeError struct{ Msg string }
|
||||||
|
|
||||||
|
func (e *DecodeError) Error() string { return e.Msg }
|
||||||
|
|
||||||
|
// ---- what gets a preview ----
|
||||||
|
|
||||||
|
var imageExt = set("png jpg jpeg jpe jfif gif webp bmp tif tiff avif heic heif hif jxl svg ico psd tga thm")
|
||||||
|
var videoExt = set("mp4 m4v mkv webm mov avi wmv flv mpg mpeg mpe m2v ts m2ts mts 3gp 3g2 ogv ogm mxf vob asf rm rmvb divx f4v mjpeg mjpg 360 osv insv lrf")
|
||||||
|
var audioExt = set("mp3 flac m4a m4b aac ogg oga opus wav wma aiff aif alac ape wv mka")
|
||||||
|
|
||||||
|
func set(s string) map[string]bool {
|
||||||
|
m := map[string]bool{}
|
||||||
|
for _, w := range strings.Fields(s) {
|
||||||
|
m[w] = true
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
// Kind is "image", "video", "audio" or "" for a file name. An audio file's preview is its
|
||||||
|
// cover art, when it carries any.
|
||||||
|
func Kind(name string) string {
|
||||||
|
ext := strings.ToLower(strings.TrimPrefix(filepath.Ext(name), "."))
|
||||||
|
switch {
|
||||||
|
case imageExt[ext]:
|
||||||
|
return "image"
|
||||||
|
case videoExt[ext]:
|
||||||
|
return "video"
|
||||||
|
case audioExt[ext]:
|
||||||
|
return "audio"
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- the probe ----
|
||||||
|
|
||||||
|
// Probe finds out what this process can make previews with. It runs ffmpeg against every
|
||||||
|
// render node, so it takes a few hundred milliseconds; callers keep the answer.
|
||||||
|
func Probe(ctx context.Context) Caps {
|
||||||
|
c := Caps{ProbedAt: time.Now(), GPUs: []GPU{}}
|
||||||
|
c.FFmpeg, _ = exec.LookPath("ffmpeg")
|
||||||
|
c.FFprobe, _ = exec.LookPath("ffprobe")
|
||||||
|
c.Nice, _ = exec.LookPath("nice")
|
||||||
|
switch {
|
||||||
|
case c.FFmpeg == "" && c.FFprobe == "":
|
||||||
|
c.Error = "ffmpeg is not installed"
|
||||||
|
case c.FFmpeg == "":
|
||||||
|
c.Error = "ffmpeg is not installed (ffprobe is)"
|
||||||
|
case c.FFprobe == "":
|
||||||
|
c.Error = "ffprobe is not installed (ffmpeg is)"
|
||||||
|
}
|
||||||
|
if c.Error != "" {
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
c.Version = ffmpegVersion(ctx, c.FFmpeg)
|
||||||
|
c.Encoders = encoders(ctx, c.FFmpeg)
|
||||||
|
accels := hwaccels(ctx, c.FFmpeg)
|
||||||
|
if accels["vaapi"] {
|
||||||
|
nodes, _ := filepath.Glob("/dev/dri/renderD*")
|
||||||
|
sort.Strings(nodes)
|
||||||
|
vainfo, _ := exec.LookPath("vainfo")
|
||||||
|
for _, dev := range nodes {
|
||||||
|
g := GPU{API: "vaapi", Device: dev, Name: metrics.RenderNodeName(dev), Integrated: metrics.RenderNodeIntegrated(dev)}
|
||||||
|
if err := openable(dev); err != nil {
|
||||||
|
g.Reason, g.Permission = err.Error(), errors.Is(err, os.ErrPermission)
|
||||||
|
c.Unusable = append(c.Unusable, g)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if msg := hwWorks(ctx, c.FFmpeg, "vaapi=va:"+dev); msg != "" {
|
||||||
|
g.Reason = msg
|
||||||
|
c.Unusable = append(c.Unusable, g)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if vainfo != "" {
|
||||||
|
g.Codecs, g.Encodes = vainfoCodecs(ctx, vainfo, dev)
|
||||||
|
}
|
||||||
|
c.GPUs = append(c.GPUs, g)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if accels["cuda"] && exists("/dev/nvidiactl") {
|
||||||
|
g := GPU{API: "cuda", Device: "0", Name: "NVIDIA GPU"}
|
||||||
|
if msg := hwWorks(ctx, c.FFmpeg, "cuda=cu:0"); msg != "" {
|
||||||
|
g.Reason = msg
|
||||||
|
c.Unusable = append(c.Unusable, g)
|
||||||
|
} else {
|
||||||
|
c.GPUs = append(c.GPUs, g)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
// openable: the node can be opened read-write, which is what a VAAPI driver needs. The
|
||||||
|
// usual reason it can't be is the owning group: a render node is root:render 0660 on
|
||||||
|
// most distributions, and a daemon running as an ordinary user is not in render.
|
||||||
|
func openable(dev string) error {
|
||||||
|
f, err := os.OpenFile(dev, os.O_RDWR, 0)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, os.ErrPermission) {
|
||||||
|
owner := ""
|
||||||
|
if st, serr := os.Stat(dev); serr == nil {
|
||||||
|
owner = ownerOf(st)
|
||||||
|
}
|
||||||
|
if owner != "" {
|
||||||
|
return fmt.Errorf("%w: %s is %s", os.ErrPermission, dev, owner)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("%w: cannot open %s", os.ErrPermission, dev)
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
f.Close()
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// hwWorks initialises a hardware device in ffmpeg and moves one synthetic frame through
|
||||||
|
// it. "" when it works, else the last thing ffmpeg said.
|
||||||
|
func hwWorks(ctx context.Context, ffmpeg, device string) string {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
name, rest, _ := strings.Cut(device, "=") // "vaapi", "va:/dev/…"
|
||||||
|
label, _, _ := strings.Cut(rest, ":") // "va"
|
||||||
|
cmd := exec.CommandContext(ctx, ffmpeg, "-nostdin", "-hide_banner", "-v", "error", "-init_hw_device", device, "-filter_hw_device", label,
|
||||||
|
"-f", "lavfi", "-i", "color=c=black:s=64x64:r=1:d=1", "-vf", "format=nv12,hwupload,hwdownload,format=nv12", "-frames:v", "1", "-f", "null", "-")
|
||||||
|
var errb bytes.Buffer
|
||||||
|
cmd.Stderr = &errb
|
||||||
|
if err := cmd.Run(); err != nil {
|
||||||
|
msg := lastLine(errb.String())
|
||||||
|
if msg == "" {
|
||||||
|
msg = err.Error()
|
||||||
|
}
|
||||||
|
return name + ": " + msg
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func ffmpegVersion(ctx context.Context, ffmpeg string) string {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
out, _ := exec.CommandContext(ctx, ffmpeg, "-version").Output()
|
||||||
|
f := strings.Fields(string(out)) // "ffmpeg version 7.1.5-0+deb13u1 Copyright …"
|
||||||
|
if len(f) >= 3 && f[1] == "version" {
|
||||||
|
v := f[2]
|
||||||
|
if i := strings.IndexAny(v, "-+~"); i > 0 {
|
||||||
|
v = v[:i]
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func hwaccels(ctx context.Context, ffmpeg string) map[string]bool {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
out, _ := exec.CommandContext(ctx, ffmpeg, "-hide_banner", "-hwaccels").Output()
|
||||||
|
m := map[string]bool{}
|
||||||
|
for _, l := range strings.Split(string(out), "\n") {
|
||||||
|
l = strings.TrimSpace(l)
|
||||||
|
if l != "" && !strings.HasPrefix(l, "Hardware acceleration methods") {
|
||||||
|
m[l] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
var vaProfileRe = regexp.MustCompile(`VAProfile([A-Za-z0-9]+?)(?:Simple|Main|High|Profile|Baseline|Constrained|Scc|Advanced|Version|Stereo|Multiview)\w*\s*:\s*VAEntrypoint(VLD|EncSlice|EncSliceLP)\b`)
|
||||||
|
|
||||||
|
// wantedEncoders are the encoders this package may ask ffmpeg for.
|
||||||
|
var wantedEncoders = set("libx264 libx265 libsvtav1 libaom-av1 libopenh264 h264_vaapi hevc_vaapi av1_vaapi aac")
|
||||||
|
|
||||||
|
// encoders lists the wanted encoders this ffmpeg was built with.
|
||||||
|
func encoders(ctx context.Context, ffmpeg string) []string {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
out, _ := exec.CommandContext(ctx, ffmpeg, "-hide_banner", "-encoders").Output()
|
||||||
|
var found []string
|
||||||
|
for _, l := range strings.Split(string(out), "\n") {
|
||||||
|
f := strings.Fields(l)
|
||||||
|
if len(f) >= 2 && wantedEncoders[f[1]] {
|
||||||
|
found = append(found, f[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(found)
|
||||||
|
return found
|
||||||
|
}
|
||||||
|
|
||||||
|
// vaCodec maps a VAProfile family to the ffmpeg codec it decodes.
|
||||||
|
var vaCodec = map[string]string{"H264": "h264", "HEVC": "hevc", "VP9": "vp9", "AV1": "av1", "MPEG2": "mpeg2video", "VC1": "vc1", "VP8": "vp8", "JPEG": "mjpeg", "MPEG4": "mpeg4", "H263": "h263"}
|
||||||
|
|
||||||
|
func vainfoCodecs(ctx context.Context, vainfo, dev string) (decodes, encodes []string) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
out, _ := exec.CommandContext(ctx, vainfo, "--display", "drm", "--device", dev).Output()
|
||||||
|
return parseVainfo(string(out))
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseVainfo names the codecs the decode (VLD) and encode (EncSlice) profiles cover,
|
||||||
|
// the way ffmpeg names them.
|
||||||
|
func parseVainfo(text string) (decodes, encodes []string) {
|
||||||
|
dec, enc := map[string]bool{}, map[string]bool{}
|
||||||
|
for _, m := range vaProfileRe.FindAllStringSubmatch(text, -1) {
|
||||||
|
c, ok := vaCodec[m[1]]
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if m[2] == "VLD" {
|
||||||
|
dec[c] = true
|
||||||
|
} else {
|
||||||
|
enc[c] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return keys(dec), keys(enc)
|
||||||
|
}
|
||||||
|
|
||||||
|
func keys(m map[string]bool) []string {
|
||||||
|
if len(m) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]string, 0, len(m))
|
||||||
|
for c := range m {
|
||||||
|
out = append(out, c)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// pick is the GPU for a codec: the first that lists it, else the first that could not
|
||||||
|
// say (ffmpeg will fall back to software by itself if that guess is wrong), else none.
|
||||||
|
func (c Caps) pick(codec string) *GPU {
|
||||||
|
var unknown *GPU
|
||||||
|
for i := range c.GPUs {
|
||||||
|
g := &c.GPUs[i]
|
||||||
|
if g.Codecs == nil {
|
||||||
|
if unknown == nil {
|
||||||
|
unknown = g
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, x := range g.Codecs {
|
||||||
|
if x == codec {
|
||||||
|
return g
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return unknown
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- making one ----
|
||||||
|
|
||||||
|
type fileInfo struct {
|
||||||
|
index string // the stream to take
|
||||||
|
codec string
|
||||||
|
w, h int
|
||||||
|
duration float64
|
||||||
|
}
|
||||||
|
|
||||||
|
type attempt struct {
|
||||||
|
gpu *GPU
|
||||||
|
lowres int
|
||||||
|
seek float64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a attempt) engine() string {
|
||||||
|
if a.gpu == nil {
|
||||||
|
return "cpu"
|
||||||
|
}
|
||||||
|
return a.gpu.API + ":" + a.gpu.Device
|
||||||
|
}
|
||||||
|
|
||||||
|
// Make produces the preview of one file.
|
||||||
|
func Make(ctx context.Context, c Caps, path string, o Opts) (Result, error) {
|
||||||
|
kind := Kind(path)
|
||||||
|
if kind == "" {
|
||||||
|
return Result{}, ErrUnsupported
|
||||||
|
}
|
||||||
|
if c.Error != "" {
|
||||||
|
return Result{}, errors.New(c.Error)
|
||||||
|
}
|
||||||
|
o = o.defaults()
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, o.Timeout)
|
||||||
|
defer cancel()
|
||||||
|
fi, err := probeFile(ctx, c, path)
|
||||||
|
if err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
var last string
|
||||||
|
for _, eng := range engines(c, kind, fi) {
|
||||||
|
for _, a := range attempts(eng, kind, fi, o.Size) {
|
||||||
|
data, msg, err := run(ctx, c, a, path, fi, o.Size)
|
||||||
|
if err == nil && len(data) > 0 {
|
||||||
|
return Result{JPEG: data, Engine: a.engine(), Codec: fi.codec, Width: fi.w, Height: fi.h}, nil
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return Result{}, ctx.Err() // the clock ran out: not the file's fault
|
||||||
|
}
|
||||||
|
if err != nil { // the decoder refused: try the next engine, not another seek
|
||||||
|
last = msg
|
||||||
|
break
|
||||||
|
}
|
||||||
|
last = "no frame at " + strconv.FormatFloat(a.seek, 'f', 1, 64) + " s"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if last == "" {
|
||||||
|
last = "could not decode"
|
||||||
|
}
|
||||||
|
return Result{}, &DecodeError{Msg: last}
|
||||||
|
}
|
||||||
|
|
||||||
|
// engines is the order to try: the fitting GPU first for a video, then the CPU.
|
||||||
|
func engines(c Caps, kind string, fi fileInfo) []*GPU {
|
||||||
|
if kind == "video" {
|
||||||
|
if g := c.pick(fi.codec); g != nil {
|
||||||
|
return []*GPU{g, nil}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return []*GPU{nil}
|
||||||
|
}
|
||||||
|
|
||||||
|
// attempts for one engine: a seek into a video, then from its start if that found no
|
||||||
|
// frame (a wrong duration in the header); a reduced-size decode of a JPEG, then full size.
|
||||||
|
func attempts(gpu *GPU, kind string, fi fileInfo, size int) []attempt {
|
||||||
|
if kind == "video" {
|
||||||
|
s := seekFor(fi.duration)
|
||||||
|
if s > 0 {
|
||||||
|
return []attempt{{gpu: gpu, seek: s}, {gpu: gpu}}
|
||||||
|
}
|
||||||
|
return []attempt{{gpu: gpu}}
|
||||||
|
}
|
||||||
|
if fi.codec == "mjpeg" && gpu == nil {
|
||||||
|
if l := lowresFor(fi.w, fi.h, size); l > 0 {
|
||||||
|
return []attempt{{lowres: l}, {}}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return []attempt{{gpu: gpu}}
|
||||||
|
}
|
||||||
|
|
||||||
|
// seekFor: a tenth of the way in, never more than 30 s, so a title card or a black lead-in
|
||||||
|
// is skipped and a long file still seeks to a keyframe near its start.
|
||||||
|
func seekFor(duration float64) float64 {
|
||||||
|
if duration <= 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
s := duration / 10
|
||||||
|
if s > 30 {
|
||||||
|
s = 30
|
||||||
|
}
|
||||||
|
if s < 0.5 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// lowresFor: how many halvings a JPEG decoder may take while the picture stays at least
|
||||||
|
// twice the preview size. libjpeg does this in the DCT, so a 24 MP photo decodes at 1.5 MP.
|
||||||
|
func lowresFor(w, h, size int) int {
|
||||||
|
m := w
|
||||||
|
if h < m {
|
||||||
|
m = h
|
||||||
|
}
|
||||||
|
l := 0
|
||||||
|
for l < 3 && m>>(l+1) >= size*2 {
|
||||||
|
l++
|
||||||
|
}
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
|
||||||
|
type probeOut struct {
|
||||||
|
Streams []struct {
|
||||||
|
Index int `json:"index"`
|
||||||
|
Codec string `json:"codec_name"`
|
||||||
|
Width int `json:"width"`
|
||||||
|
Height int `json:"height"`
|
||||||
|
Disposition struct {
|
||||||
|
AttachedPic int `json:"attached_pic"`
|
||||||
|
} `json:"disposition"`
|
||||||
|
} `json:"streams"`
|
||||||
|
Format struct {
|
||||||
|
Duration string `json:"duration"`
|
||||||
|
} `json:"format"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// probeFile asks ffprobe which stream to take and what is in it.
|
||||||
|
func probeFile(ctx context.Context, c Caps, path string) (fileInfo, error) {
|
||||||
|
cmd := exec.CommandContext(ctx, c.FFprobe, "-v", "error", "-select_streams", "v", "-show_entries", "stream=index,codec_name,width,height,disposition:format=duration", "-of", "json", path)
|
||||||
|
var errb bytes.Buffer
|
||||||
|
cmd.Stderr = &errb
|
||||||
|
out, err := cmd.Output()
|
||||||
|
if err != nil {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return fileInfo{}, ctx.Err()
|
||||||
|
}
|
||||||
|
msg := lastLine(errb.String())
|
||||||
|
if msg == "" {
|
||||||
|
msg = "ffprobe: " + err.Error()
|
||||||
|
}
|
||||||
|
return fileInfo{}, &DecodeError{Msg: msg}
|
||||||
|
}
|
||||||
|
var po probeOut
|
||||||
|
if err := json.Unmarshal(out, &po); err != nil {
|
||||||
|
return fileInfo{}, &DecodeError{Msg: "ffprobe: unreadable answer"}
|
||||||
|
}
|
||||||
|
if len(po.Streams) == 0 {
|
||||||
|
return fileInfo{}, &DecodeError{Msg: "no video stream"}
|
||||||
|
}
|
||||||
|
s := po.Streams[0]
|
||||||
|
for _, x := range po.Streams { // cover art rides along as a video stream; the picture is not the film
|
||||||
|
if x.Disposition.AttachedPic == 0 {
|
||||||
|
s = x
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fi := fileInfo{index: strconv.Itoa(s.Index), codec: s.Codec, w: s.Width, h: s.Height}
|
||||||
|
fi.duration, _ = strconv.ParseFloat(po.Format.Duration, 64)
|
||||||
|
return fi, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// run is one ffmpeg: the JPEG on success; on failure what ffmpeg said and the error.
|
||||||
|
// Empty output with a clean exit means the seek found nothing.
|
||||||
|
func run(ctx context.Context, c Caps, a attempt, path string, fi fileInfo, size int) ([]byte, string, error) {
|
||||||
|
args := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
|
||||||
|
if a.gpu != nil {
|
||||||
|
args = append(args, "-hwaccel", a.gpu.API, "-hwaccel_device", a.gpu.Device)
|
||||||
|
}
|
||||||
|
if a.lowres > 0 {
|
||||||
|
args = append(args, "-lowres", strconv.Itoa(a.lowres))
|
||||||
|
}
|
||||||
|
if a.seek > 0 {
|
||||||
|
args = append(args, "-noaccurate_seek", "-ss", strconv.FormatFloat(a.seek, 'f', 3, 64))
|
||||||
|
}
|
||||||
|
args = append(args, "-i", path, "-map", "0:"+fi.index, "-frames:v", "1",
|
||||||
|
"-vf", fmt.Sprintf("scale=w=%d:h=%d:force_original_aspect_ratio=decrease:flags=bicubic", size, size),
|
||||||
|
"-f", "image2pipe", "-c:v", "mjpeg", "-q:v", "4", "pipe:1")
|
||||||
|
name, argv := c.FFmpeg, args
|
||||||
|
if c.Nice != "" { // a burst of previews must not crowd out what the machine is for
|
||||||
|
name, argv = c.Nice, append([]string{"-n", "10", c.FFmpeg}, args...)
|
||||||
|
}
|
||||||
|
cmd := exec.CommandContext(ctx, name, argv...)
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||||
|
if err := cmd.Run(); err != nil {
|
||||||
|
msg := lastLine(errb.String())
|
||||||
|
if msg == "" {
|
||||||
|
msg = err.Error()
|
||||||
|
}
|
||||||
|
return nil, msg, err
|
||||||
|
}
|
||||||
|
return out.Bytes(), "", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- the same, as root ----
|
||||||
|
|
||||||
|
// ProbeRoot is Probe run as root through this binary, for the Files tab's root mode: root
|
||||||
|
// opens every render node, whatever group the daemon's user is in.
|
||||||
|
func ProbeRoot(ctx context.Context, self string) (Caps, error) {
|
||||||
|
full := priv.Wrap(self, "thumb", "--probe")
|
||||||
|
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
|
||||||
|
var errb bytes.Buffer
|
||||||
|
cmd.Stderr = &errb
|
||||||
|
out, err := cmd.Output()
|
||||||
|
if err != nil {
|
||||||
|
if msg := lastLine(errb.String()); msg != "" {
|
||||||
|
return Caps{}, errors.New(msg)
|
||||||
|
}
|
||||||
|
return Caps{}, err
|
||||||
|
}
|
||||||
|
var c Caps
|
||||||
|
if err := json.Unmarshal(out, &c); err != nil {
|
||||||
|
return Caps{}, err
|
||||||
|
}
|
||||||
|
return c, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type helperOut struct {
|
||||||
|
Engine string `json:"engine,omitempty"`
|
||||||
|
Codec string `json:"codec,omitempty"`
|
||||||
|
Width int `json:"width,omitempty"`
|
||||||
|
Height int `json:"height,omitempty"`
|
||||||
|
JPEG []byte `json:"jpeg,omitempty"`
|
||||||
|
Error string `json:"error,omitempty"`
|
||||||
|
Permanent bool `json:"permanent,omitempty"` // a DecodeError: the file, not the moment
|
||||||
|
}
|
||||||
|
|
||||||
|
// MakeRoot is Make run as root through this binary, with the caps ProbeRoot found.
|
||||||
|
func MakeRoot(ctx context.Context, self string, c Caps, path string, o Opts) (Result, error) {
|
||||||
|
o = o.defaults()
|
||||||
|
caps, _ := json.Marshal(c)
|
||||||
|
full := priv.Wrap(self, "thumb", "--size", strconv.Itoa(o.Size), "--timeout", o.Timeout.String(), path)
|
||||||
|
cmd := exec.CommandContext(ctx, full[0], full[1:]...)
|
||||||
|
cmd.Stdin = bytes.NewReader(caps)
|
||||||
|
var errb bytes.Buffer
|
||||||
|
cmd.Stderr = &errb
|
||||||
|
out, err := cmd.Output()
|
||||||
|
if err != nil {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return Result{}, ctx.Err()
|
||||||
|
}
|
||||||
|
if msg := lastLine(errb.String()); msg != "" {
|
||||||
|
return Result{}, errors.New(msg)
|
||||||
|
}
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
var h helperOut
|
||||||
|
if err := json.Unmarshal(out, &h); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
if h.Error != "" {
|
||||||
|
if h.Permanent {
|
||||||
|
return Result{}, &DecodeError{Msg: h.Error}
|
||||||
|
}
|
||||||
|
return Result{}, errors.New(h.Error)
|
||||||
|
}
|
||||||
|
return Result{JPEG: h.JPEG, Engine: h.Engine, Codec: h.Codec, Width: h.Width, Height: h.Height}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Helper implements `hamlab thumb …`, the root side of the above. `--probe` prints Caps;
|
||||||
|
// `[--size N] [--timeout D] <path>` reads Caps from stdin and prints the preview as JSON.
|
||||||
|
// Returns the exit code.
|
||||||
|
func Helper(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
|
||||||
|
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
|
||||||
|
ctx := context.Background()
|
||||||
|
o := Opts{}
|
||||||
|
path := ""
|
||||||
|
for i := 0; i < len(args); i++ {
|
||||||
|
switch a := args[i]; {
|
||||||
|
case a == "--probe":
|
||||||
|
return emit(stdout, Probe(ctx), stderr)
|
||||||
|
case a == "--size" && i+1 < len(args):
|
||||||
|
o.Size, _ = strconv.Atoi(args[i+1])
|
||||||
|
i++
|
||||||
|
case a == "--timeout" && i+1 < len(args):
|
||||||
|
o.Timeout, _ = time.ParseDuration(args[i+1])
|
||||||
|
i++
|
||||||
|
case strings.HasPrefix(a, "-"):
|
||||||
|
return fail(errors.New("usage: hamlab thumb --probe | [--size N] [--timeout D] <path>"))
|
||||||
|
default:
|
||||||
|
path = a
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if path == "" || !filepath.IsAbs(path) {
|
||||||
|
return fail(errors.New("usage: hamlab thumb --probe | [--size N] [--timeout D] <absolute path>"))
|
||||||
|
}
|
||||||
|
var c Caps
|
||||||
|
if err := json.NewDecoder(io.LimitReader(stdin, 1<<20)).Decode(&c); err != nil {
|
||||||
|
return fail(fmt.Errorf("caps on stdin: %w", err))
|
||||||
|
}
|
||||||
|
res, err := Make(ctx, c, path, o)
|
||||||
|
if err != nil {
|
||||||
|
var de *DecodeError
|
||||||
|
return emit(stdout, helperOut{Error: err.Error(), Permanent: errors.As(err, &de)}, stderr)
|
||||||
|
}
|
||||||
|
return emit(stdout, helperOut{Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}, stderr)
|
||||||
|
}
|
||||||
|
|
||||||
|
func emit(w io.Writer, v any, stderr io.Writer) int {
|
||||||
|
if err := json.NewEncoder(w).Encode(v); err != nil {
|
||||||
|
fmt.Fprintln(stderr, err.Error())
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- small helpers ----
|
||||||
|
|
||||||
|
func lastLine(s string) string {
|
||||||
|
lines := strings.Split(strings.TrimSpace(s), "\n")
|
||||||
|
return strings.TrimSpace(lines[len(lines)-1])
|
||||||
|
}
|
||||||
|
|
||||||
|
func exists(p string) bool { _, err := os.Stat(p); return err == nil }
|
||||||
@@ -0,0 +1,253 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"image"
|
||||||
|
"image/color"
|
||||||
|
"image/jpeg"
|
||||||
|
"image/png"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestKind(t *testing.T) {
|
||||||
|
for name, want := range map[string]string{"a.JPG": "image", "b.heic": "image", "c.mkv": "video", "GH010098.MP4": "video", "x.OSV": "video", "d.txt": "", "e": "", ".jpg": "image"} {
|
||||||
|
if got := Kind(name); got != want {
|
||||||
|
t.Errorf("Kind(%q) = %q, want %q", name, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestKeyFollowsTheFile(t *testing.T) {
|
||||||
|
a := Key("/x/a.jpg", 10, 20)
|
||||||
|
if a != Key("/x/a.jpg", 10, 20) || len(a) != 64 {
|
||||||
|
t.Fatal("key is not stable")
|
||||||
|
}
|
||||||
|
for _, b := range []string{Key("/x/b.jpg", 10, 20), Key("/x/a.jpg", 11, 20), Key("/x/a.jpg", 10, 21)} {
|
||||||
|
if b == a {
|
||||||
|
t.Fatal("key ignores part of the file's identity")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseVainfo(t *testing.T) {
|
||||||
|
text := `vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 25.2.3 ()
|
||||||
|
VAProfileMPEG2Simple : VAEntrypointVLD
|
||||||
|
VAProfileH264Main : VAEntrypointVLD
|
||||||
|
VAProfileH264High : VAEntrypointEncSliceLP
|
||||||
|
VAProfileJPEGBaseline : VAEntrypointVLD
|
||||||
|
VAProfileHEVCMain10 : VAEntrypointVLD
|
||||||
|
VAProfileHEVCMain10 : VAEntrypointEncSlice
|
||||||
|
VAProfileHEVCMain444_12 : VAEntrypointVLD
|
||||||
|
VAProfileVP9Profile2 : VAEntrypointVLD
|
||||||
|
VAProfileAV1Profile0 : VAEntrypointEncSliceLP
|
||||||
|
`
|
||||||
|
dec, enc := parseVainfo(text)
|
||||||
|
same := func(got, want []string) bool {
|
||||||
|
if len(got) != len(want) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for i := range want {
|
||||||
|
if got[i] != want[i] {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if !same(dec, []string{"h264", "hevc", "mjpeg", "mpeg2video", "vp9"}) {
|
||||||
|
t.Fatalf("decodes = %v", dec)
|
||||||
|
}
|
||||||
|
if !same(enc, []string{"av1", "h264", "hevc"}) {
|
||||||
|
t.Fatalf("encodes = %v", enc)
|
||||||
|
}
|
||||||
|
if d, e := parseVainfo(""); d != nil || e != nil {
|
||||||
|
t.Fatal("nothing from nothing")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPickAndEngines(t *testing.T) {
|
||||||
|
igpu := GPU{API: "vaapi", Device: "/dev/dri/renderD128", Codecs: []string{"h264", "hevc"}}
|
||||||
|
arc := GPU{API: "vaapi", Device: "/dev/dri/renderD129", Codecs: []string{"av1", "h264", "hevc"}}
|
||||||
|
c := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", GPUs: []GPU{igpu, arc}}
|
||||||
|
if g := c.pick("av1"); g == nil || g.Device != arc.Device {
|
||||||
|
t.Fatal("av1 should go to the card that lists it")
|
||||||
|
}
|
||||||
|
if g := c.pick("h264"); g == nil || g.Device != igpu.Device {
|
||||||
|
t.Fatal("h264 should go to the first card that lists it")
|
||||||
|
}
|
||||||
|
if g := c.pick("vc1"); g != nil {
|
||||||
|
t.Fatal("a codec nobody lists is software")
|
||||||
|
}
|
||||||
|
// a card nothing could describe is tried, after the ones that answered
|
||||||
|
c.GPUs = append(c.GPUs, GPU{API: "cuda", Device: "0"})
|
||||||
|
if g := c.pick("vc1"); g == nil || g.API != "cuda" {
|
||||||
|
t.Fatal("an undescribed card is the guess of last resort")
|
||||||
|
}
|
||||||
|
// videos: GPU then CPU; images: CPU only
|
||||||
|
if e := engines(c, "video", fileInfo{codec: "hevc"}); len(e) != 2 || e[0] == nil || e[1] != nil {
|
||||||
|
t.Fatal("a video tries the GPU, then the CPU")
|
||||||
|
}
|
||||||
|
if e := engines(c, "image", fileInfo{codec: "mjpeg"}); len(e) != 1 || e[0] != nil {
|
||||||
|
t.Fatal("an image is CPU only")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAttempts(t *testing.T) {
|
||||||
|
a := attempts(nil, "video", fileInfo{duration: 120}, 256)
|
||||||
|
if len(a) != 2 || a[0].seek != 12 || a[1].seek != 0 {
|
||||||
|
t.Fatalf("video attempts = %+v", a)
|
||||||
|
}
|
||||||
|
if a := attempts(nil, "video", fileInfo{duration: 7200}, 256); a[0].seek != 30 {
|
||||||
|
t.Fatalf("seek capped at 30 s, got %v", a[0].seek)
|
||||||
|
}
|
||||||
|
if a := attempts(nil, "video", fileInfo{duration: 2}, 256); len(a) != 1 || a[0].seek != 0 {
|
||||||
|
t.Fatalf("a short clip is taken from its start, got %+v", a)
|
||||||
|
}
|
||||||
|
if a := attempts(nil, "image", fileInfo{codec: "mjpeg", w: 6000, h: 4000}, 256); len(a) != 2 || a[0].lowres != 2 || a[1].lowres != 0 {
|
||||||
|
t.Fatalf("jpeg attempts = %+v", a)
|
||||||
|
}
|
||||||
|
if a := attempts(nil, "image", fileInfo{codec: "mjpeg", w: 800, h: 600}, 256); len(a) != 1 || a[0].lowres != 0 {
|
||||||
|
t.Fatalf("a small jpeg decodes whole, got %+v", a)
|
||||||
|
}
|
||||||
|
if a := attempts(nil, "image", fileInfo{codec: "png", w: 6000, h: 4000}, 256); len(a) != 1 || a[0].lowres != 0 {
|
||||||
|
t.Fatalf("only jpeg has lowres, got %+v", a)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCache(t *testing.T) {
|
||||||
|
c := NewCache(filepath.Join(t.TempDir(), "thumbs.db"))
|
||||||
|
k := Key("/a", 1, 2)
|
||||||
|
if _, found, err := c.Get(k); found || err != nil {
|
||||||
|
t.Fatal("an empty cache found something, or complained")
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(c.Path); err == nil {
|
||||||
|
t.Fatal("a miss must not create the database")
|
||||||
|
}
|
||||||
|
if n, b := c.Stats(); n != 0 || b != 0 {
|
||||||
|
t.Fatal("stats of nothing")
|
||||||
|
}
|
||||||
|
if err := c.Put(k, Entry{Path: "/a", Engine: "cpu", Codec: "png", Width: 640, Height: 360, JPEG: []byte("jpeg")}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
e, found, err := c.Get(k)
|
||||||
|
if err != nil || !found || string(e.JPEG) != "jpeg" || e.Engine != "cpu" || e.Width != 640 || e.Path != "/a" || e.Made.IsZero() {
|
||||||
|
t.Fatalf("put then get: %+v %v %v", e, found, err)
|
||||||
|
}
|
||||||
|
k2 := Key("/b", 1, 2)
|
||||||
|
if err := c.Put(k2, Entry{Path: "/b", Failure: "no video stream"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if e, found, _ := c.Get(k2); !found || e.JPEG != nil || e.Failure != "no video stream" {
|
||||||
|
t.Fatal("a failure is remembered with its reason")
|
||||||
|
}
|
||||||
|
if err := c.Put(k, Entry{}); err == nil {
|
||||||
|
t.Fatal("an empty preview must not be kept")
|
||||||
|
}
|
||||||
|
// replacing an entry does not count it twice
|
||||||
|
if err := c.Put(k, Entry{Path: "/a", JPEG: []byte("jpeg2")}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
n, b := c.Stats()
|
||||||
|
if n != 2 || b == 0 {
|
||||||
|
t.Fatalf("stats = %d entries, %d bytes", n, b)
|
||||||
|
}
|
||||||
|
// survives a reopen
|
||||||
|
if err := c.Close(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
c2 := NewCache(c.Path)
|
||||||
|
if e, found, _ := c2.Get(k); !found || string(e.JPEG) != "jpeg2" {
|
||||||
|
t.Fatal("not there after reopening")
|
||||||
|
}
|
||||||
|
if n, _ := c2.Stats(); n != 2 {
|
||||||
|
t.Fatal("count lost on reopen")
|
||||||
|
}
|
||||||
|
c2.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// The real thing, when ffmpeg is here: a PNG and a generated video each come back as a
|
||||||
|
// JPEG no larger than the asked size, with the aspect kept.
|
||||||
|
func TestMakeWithFFmpeg(t *testing.T) {
|
||||||
|
if _, err := exec.LookPath("ffmpeg"); err != nil {
|
||||||
|
t.Skip("no ffmpeg")
|
||||||
|
}
|
||||||
|
c := Probe(context.Background())
|
||||||
|
if c.Error != "" {
|
||||||
|
t.Skip(c.Error)
|
||||||
|
}
|
||||||
|
dir := t.TempDir()
|
||||||
|
img := image.NewRGBA(image.Rect(0, 0, 640, 360))
|
||||||
|
for y := 0; y < 360; y++ {
|
||||||
|
for x := 0; x < 640; x++ {
|
||||||
|
img.Set(x, y, color.RGBA{uint8(x / 3), uint8(y), 90, 255})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
pngPath := filepath.Join(dir, "a.png")
|
||||||
|
f, _ := os.Create(pngPath)
|
||||||
|
_ = png.Encode(f, img)
|
||||||
|
f.Close()
|
||||||
|
res, err := Make(context.Background(), c, pngPath, Opts{Size: 200})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
check(t, res, 200, 112, "cpu")
|
||||||
|
|
||||||
|
// a two-second test pattern, with the always-built-in mpeg4 encoder
|
||||||
|
vid := filepath.Join(dir, "v.mp4")
|
||||||
|
out, err := exec.Command("ffmpeg", "-nostdin", "-v", "error", "-f", "lavfi", "-i", "testsrc=size=320x180:rate=10:duration=2", "-c:v", "mpeg4", "-q:v", "5", vid).CombinedOutput()
|
||||||
|
if err != nil {
|
||||||
|
t.Skipf("could not make a test video: %s", out)
|
||||||
|
}
|
||||||
|
res, err = Make(context.Background(), c, vid, Opts{Size: 100})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
check(t, res, 100, 56, "")
|
||||||
|
if res.Codec != "mpeg4" || res.Width != 320 {
|
||||||
|
t.Fatalf("probe said %s %dx%d", res.Codec, res.Width, res.Height)
|
||||||
|
}
|
||||||
|
|
||||||
|
// not media, and media that isn't
|
||||||
|
if _, err := Make(context.Background(), c, filepath.Join(dir, "x.txt"), Opts{}); err != ErrUnsupported {
|
||||||
|
t.Fatalf("a .txt is unsupported, got %v", err)
|
||||||
|
}
|
||||||
|
bad := filepath.Join(dir, "bad.mp4")
|
||||||
|
_ = os.WriteFile(bad, []byte("not a video at all"), 0o644)
|
||||||
|
_, err = Make(context.Background(), c, bad, Opts{})
|
||||||
|
var de *DecodeError
|
||||||
|
if err == nil || !isDecodeErr(err, &de) {
|
||||||
|
t.Fatalf("garbage should be a DecodeError, got %v", err)
|
||||||
|
}
|
||||||
|
// the clock: a cancelled context is not the file's fault
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
|
||||||
|
defer cancel()
|
||||||
|
if _, err := Make(ctx, c, pngPath, Opts{}); err == nil || isDecodeErr(err, &de) {
|
||||||
|
t.Fatalf("a timeout must not read as a decode failure, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func check(t *testing.T, res Result, w, h int, engine string) {
|
||||||
|
t.Helper()
|
||||||
|
im, err := jpeg.Decode(bytes.NewReader(res.JPEG))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("not a jpeg: %v", err)
|
||||||
|
}
|
||||||
|
if b := im.Bounds(); b.Dx() != w || b.Dy() < h-1 || b.Dy() > h+1 { // the short side rounds either way
|
||||||
|
t.Fatalf("got %dx%d, want %dx%d", b.Dx(), b.Dy(), w, h)
|
||||||
|
}
|
||||||
|
if engine != "" && res.Engine != engine {
|
||||||
|
t.Fatalf("engine %q, want %q", res.Engine, engine)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func isDecodeErr(err error, de **DecodeError) bool {
|
||||||
|
x, ok := err.(*DecodeError)
|
||||||
|
if ok {
|
||||||
|
*de = x
|
||||||
|
}
|
||||||
|
return ok
|
||||||
|
}
|
||||||
@@ -0,0 +1,259 @@
|
|||||||
|
package metrics
|
||||||
|
|
||||||
|
// CPU power, from the RAPL energy counters the kernel keeps under /sys/class/powercap: Intel
|
||||||
|
// since Sandy Bridge, and AMD Zen through the same driver. Each counter is a running total in
|
||||||
|
// µJ, so power is the difference between two readings over the time between them. Every
|
||||||
|
// package is summed (`package-0`, and `package-1` on a two-socket board). The other top-level
|
||||||
|
// domains are left out. `psys` claims the whole platform but is only as good as the firmware's
|
||||||
|
// calibration, and on one machine here it reads lower than the package inside it. The MMIO
|
||||||
|
// copy of a package (`intel-rapl-mmio:0`) measures the same thing again. This is the CPU's own
|
||||||
|
// figure, shown beside its clock: cores, built-in graphics and memory controller, not disks,
|
||||||
|
// fans or the power supply.
|
||||||
|
//
|
||||||
|
// The counters are root-only since the PLATYPUS side channel (2020). So, like the GPU client
|
||||||
|
// walk, they are read in-process when the daemon is root, and otherwise by a root helper,
|
||||||
|
// `hamlab cpu-energy`, started once through sudo or doas and asked over a pipe at every
|
||||||
|
// sample: one privilege escalation per daemon, not one a second in the auth log. The helper
|
||||||
|
// takes nothing from the daemon but the newline that asks; it finds and reads the counters
|
||||||
|
// itself. Without root there is no reading, and nothing is shown.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"encoding/json"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"hamlab/internal/priv"
|
||||||
|
)
|
||||||
|
|
||||||
|
const raplRoot = "/sys/class/powercap"
|
||||||
|
|
||||||
|
var raplPkgRe = regexp.MustCompile(`^package-\d+$`)
|
||||||
|
|
||||||
|
// raplCount is one package's counter at one moment.
|
||||||
|
type raplCount struct {
|
||||||
|
Dir string `json:"d"`
|
||||||
|
UJ uint64 `json:"uj"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type cpuPowerState struct {
|
||||||
|
probed bool
|
||||||
|
dirs []string // the package domains, found at the first sample
|
||||||
|
max map[string]uint64 // each counter's range: where it wraps to zero
|
||||||
|
useHelper bool // the counters are not readable in-process
|
||||||
|
helper *energyHelper
|
||||||
|
prev []raplCount
|
||||||
|
prevAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// raplPackages lists the package domains under root, sorted. intel-rapl:0 is a package;
|
||||||
|
// intel-rapl:0:0 is a part of one (core, uncore, dram).
|
||||||
|
func raplPackages(root string) []string {
|
||||||
|
ents, err := os.ReadDir(root)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
var out []string
|
||||||
|
for _, e := range ents {
|
||||||
|
n := e.Name()
|
||||||
|
if !strings.HasPrefix(n, "intel-rapl:") || strings.Count(n, ":") != 1 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if raplPkgRe.MatchString(strings.TrimSpace(readStr(filepath.Join(root, n, "name")))) {
|
||||||
|
out = append(out, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func readRAPL(root string, dirs []string) ([]raplCount, bool) {
|
||||||
|
out := make([]raplCount, 0, len(dirs))
|
||||||
|
for _, d := range dirs {
|
||||||
|
b, err := os.ReadFile(filepath.Join(root, d, "energy_uj"))
|
||||||
|
if err != nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
v, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
out = append(out, raplCount{d, v})
|
||||||
|
}
|
||||||
|
return out, len(out) > 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// raplWatts is the power between two readings of the same counters. A counter that wrapped
|
||||||
|
// is carried across its range. One that went backwards with no known range, or a set of
|
||||||
|
// counters that changed between readings, gives no answer rather than a wrong one.
|
||||||
|
func raplWatts(prev, cur []raplCount, max map[string]uint64, dt float64) (float64, bool) {
|
||||||
|
if dt <= 0 || len(prev) == 0 || len(prev) != len(cur) {
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
var uj float64
|
||||||
|
for i := range cur {
|
||||||
|
if cur[i].Dir != prev[i].Dir {
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
if cur[i].UJ >= prev[i].UJ {
|
||||||
|
uj += float64(cur[i].UJ - prev[i].UJ)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
m := max[cur[i].Dir]
|
||||||
|
if m == 0 || prev[i].UJ > m {
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
uj += float64(m - prev[i].UJ + cur[i].UJ)
|
||||||
|
}
|
||||||
|
return uj / 1e6 / dt, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// cpuPower is the CPU packages' power since the previous sample, in watts; 0 when this
|
||||||
|
// machine has no counters or no way to read them.
|
||||||
|
func (c *Collector) cpuPower(now time.Time) float64 {
|
||||||
|
p := &c.power
|
||||||
|
if !p.probed {
|
||||||
|
p.probed = true
|
||||||
|
p.dirs = raplPackages(raplRoot)
|
||||||
|
p.max = map[string]uint64{}
|
||||||
|
for _, d := range p.dirs {
|
||||||
|
if v, ok := readInt(filepath.Join(raplRoot, d, "max_energy_range_uj")); ok && v > 0 {
|
||||||
|
p.max[d] = uint64(v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(p.dirs) == 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
var cur []raplCount
|
||||||
|
ok := false
|
||||||
|
if !p.useHelper {
|
||||||
|
if cur, ok = readRAPL(raplRoot, p.dirs); !ok && priv.Mode() != "root" && priv.Available() {
|
||||||
|
p.useHelper = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if p.useHelper {
|
||||||
|
if p.helper == nil {
|
||||||
|
p.helper = &energyHelper{}
|
||||||
|
}
|
||||||
|
cur, ok = p.helper.read()
|
||||||
|
}
|
||||||
|
if !ok {
|
||||||
|
p.prev = nil
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
w, good := raplWatts(p.prev, cur, p.max, now.Sub(p.prevAt).Seconds())
|
||||||
|
p.prev, p.prevAt = cur, now
|
||||||
|
if !good {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return round1(w)
|
||||||
|
}
|
||||||
|
|
||||||
|
// energyHelper is the root child that answers each newline with the package counters, one
|
||||||
|
// JSON line. The same shape as the GPU client walk's helper, kept separate from it.
|
||||||
|
type energyHelper struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
cmd *exec.Cmd
|
||||||
|
in io.WriteCloser
|
||||||
|
out *bufio.Reader
|
||||||
|
failedAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *energyHelper) read() ([]raplCount, bool) {
|
||||||
|
h.mu.Lock()
|
||||||
|
defer h.mu.Unlock()
|
||||||
|
if h.cmd == nil {
|
||||||
|
if !h.failedAt.IsZero() && time.Since(h.failedAt) < helperRetry {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
if !h.start() {
|
||||||
|
h.failedAt = time.Now()
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// a stuck child must not stall the sampler: it is killed, and tried again later
|
||||||
|
cmd := h.cmd
|
||||||
|
stop := time.AfterFunc(5*time.Second, func() { cmd.Process.Kill() })
|
||||||
|
defer stop.Stop()
|
||||||
|
if _, err := io.WriteString(h.in, "\n"); err != nil {
|
||||||
|
h.fail()
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
line, err := h.out.ReadBytes('\n')
|
||||||
|
if err != nil {
|
||||||
|
h.fail()
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
var cs []raplCount
|
||||||
|
if json.Unmarshal(line, &cs) != nil {
|
||||||
|
h.fail()
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
return cs, len(cs) > 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *energyHelper) start() bool {
|
||||||
|
self, err := os.Executable()
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
full := priv.Wrap(self, "cpu-energy")
|
||||||
|
cmd := exec.Command(full[0], full[1:]...)
|
||||||
|
in, err := cmd.StdinPipe()
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
out, err := cmd.StdoutPipe()
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if err := cmd.Start(); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
h.cmd, h.in, h.out = cmd, in, bufio.NewReader(out)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *energyHelper) fail() {
|
||||||
|
if h.cmd != nil {
|
||||||
|
h.in.Close()
|
||||||
|
h.cmd.Process.Kill()
|
||||||
|
go h.cmd.Wait()
|
||||||
|
}
|
||||||
|
h.cmd, h.in, h.out = nil, nil, nil
|
||||||
|
h.failedAt = time.Now()
|
||||||
|
}
|
||||||
|
|
||||||
|
// CPUEnergyHelper is `hamlab cpu-energy`: for every line read on stdin, one JSON line of the
|
||||||
|
// package counters on stdout. Ends when stdin does — when the daemon that started it has gone.
|
||||||
|
func CPUEnergyHelper(stdin io.Reader, stdout io.Writer) int {
|
||||||
|
return cpuEnergyHelper(raplRoot, stdin, stdout)
|
||||||
|
}
|
||||||
|
|
||||||
|
func cpuEnergyHelper(root string, stdin io.Reader, stdout io.Writer) int {
|
||||||
|
dirs := raplPackages(root)
|
||||||
|
sc := bufio.NewScanner(stdin)
|
||||||
|
w := bufio.NewWriter(stdout)
|
||||||
|
for sc.Scan() {
|
||||||
|
cs, ok := readRAPL(root, dirs)
|
||||||
|
if !ok {
|
||||||
|
cs = []raplCount{}
|
||||||
|
}
|
||||||
|
b, _ := json.Marshal(cs)
|
||||||
|
w.Write(b)
|
||||||
|
w.WriteByte('\n')
|
||||||
|
if w.Flush() != nil {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
@@ -0,0 +1,106 @@
|
|||||||
|
package metrics
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A fake /sys/class/powercap: a package with its parts, the platform domain, and the MMIO
|
||||||
|
// copy of the same package. Only the package counts.
|
||||||
|
func fakeRAPL(t *testing.T, domains map[string][2]string) string {
|
||||||
|
t.Helper()
|
||||||
|
root := t.TempDir()
|
||||||
|
for dir, v := range domains {
|
||||||
|
d := filepath.Join(root, dir)
|
||||||
|
if err := os.MkdirAll(d, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
os.WriteFile(filepath.Join(d, "name"), []byte(v[0]+"\n"), 0o644)
|
||||||
|
os.WriteFile(filepath.Join(d, "energy_uj"), []byte(v[1]+"\n"), 0o644)
|
||||||
|
}
|
||||||
|
return root
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRAPLPackagesOnlyCountsPackages(t *testing.T) {
|
||||||
|
root := fakeRAPL(t, map[string][2]string{
|
||||||
|
"intel-rapl:0": {"package-0", "1000"},
|
||||||
|
"intel-rapl:0:0": {"core", "600"},
|
||||||
|
"intel-rapl:0:1": {"uncore", "10"},
|
||||||
|
"intel-rapl:1": {"psys", "300"},
|
||||||
|
"intel-rapl-mmio:0": {"package-0", "1000"},
|
||||||
|
})
|
||||||
|
if got := raplPackages(root); len(got) != 1 || got[0] != "intel-rapl:0" {
|
||||||
|
t.Fatalf("packages = %v, want [intel-rapl:0]", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
two := fakeRAPL(t, map[string][2]string{"intel-rapl:0": {"package-0", "5"}, "intel-rapl:1": {"package-1", "7"}})
|
||||||
|
got := raplPackages(two)
|
||||||
|
if strings.Join(got, ",") != "intel-rapl:0,intel-rapl:1" {
|
||||||
|
t.Fatalf("two sockets: packages = %v", got)
|
||||||
|
}
|
||||||
|
cs, ok := readRAPL(two, got)
|
||||||
|
if !ok || len(cs) != 2 || cs[0].UJ != 5 || cs[1].UJ != 7 {
|
||||||
|
t.Fatalf("readRAPL = %+v, %v", cs, ok)
|
||||||
|
}
|
||||||
|
if got := raplPackages(filepath.Join(two, "nope")); len(got) != 0 {
|
||||||
|
t.Fatalf("no powercap: packages = %v, want none", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRAPLWatts(t *testing.T) {
|
||||||
|
c := func(d string, uj uint64) raplCount { return raplCount{d, uj} }
|
||||||
|
max := map[string]uint64{"intel-rapl:0": 1_000_000_000}
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
prev, cur []raplCount
|
||||||
|
dt float64
|
||||||
|
want float64
|
||||||
|
ok bool
|
||||||
|
}{
|
||||||
|
{"steady", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:0", 10_000_000)}, 2, 5, true},
|
||||||
|
{"two packages add", []raplCount{c("intel-rapl:0", 0), c("intel-rapl:1", 0)}, []raplCount{c("intel-rapl:0", 4_000_000), c("intel-rapl:1", 6_000_000)}, 1, 10, true},
|
||||||
|
{"wrapped", []raplCount{c("intel-rapl:0", 999_000_000)}, []raplCount{c("intel-rapl:0", 1_000_000)}, 1, 2, true},
|
||||||
|
{"backwards, range unknown", []raplCount{c("intel-rapl:1", 5_000_000)}, []raplCount{c("intel-rapl:1", 1)}, 1, 0, false},
|
||||||
|
{"counters changed", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:1", 9)}, 1, 0, false},
|
||||||
|
{"first reading", nil, []raplCount{c("intel-rapl:0", 9)}, 1, 0, false},
|
||||||
|
{"no time passed", []raplCount{c("intel-rapl:0", 0)}, []raplCount{c("intel-rapl:0", 9)}, 0, 0, false},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
got, ok := raplWatts(tc.prev, tc.cur, max, tc.dt)
|
||||||
|
if ok != tc.ok || (ok && (got < tc.want-1e-9 || got > tc.want+1e-9)) {
|
||||||
|
t.Errorf("%s: got %v, %v; want %v, %v", tc.name, got, ok, tc.want, tc.ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The helper answers every newline with the counters it found itself, and stops at EOF.
|
||||||
|
func TestCPUEnergyHelper(t *testing.T) {
|
||||||
|
root := fakeRAPL(t, map[string][2]string{"intel-rapl:0": {"package-0", "42"}, "intel-rapl:0:0": {"core", "7"}})
|
||||||
|
var out bytes.Buffer
|
||||||
|
if code := cpuEnergyHelper(root, strings.NewReader("\n\n"), &out); code != 0 {
|
||||||
|
t.Fatalf("exit %d", code)
|
||||||
|
}
|
||||||
|
lines := strings.Split(strings.TrimSpace(out.String()), "\n")
|
||||||
|
if len(lines) != 2 {
|
||||||
|
t.Fatalf("got %d lines, want 2: %q", len(lines), out.String())
|
||||||
|
}
|
||||||
|
var cs []raplCount
|
||||||
|
if err := json.Unmarshal([]byte(lines[1]), &cs); err != nil || len(cs) != 1 || cs[0] != (raplCount{"intel-rapl:0", 42}) {
|
||||||
|
t.Fatalf("line = %s (%v)", lines[1], err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// History keeps the average of the samples that had a reading.
|
||||||
|
func TestFoldCPUPower(t *testing.T) {
|
||||||
|
b := []Sample{{T: 1, CPUPower: 4}, {T: 2}, {T: 3, CPUPower: 8}}
|
||||||
|
if got := fold(b).CPUPower; got != 6 {
|
||||||
|
t.Fatalf("folded CPU power = %v, want 6", got)
|
||||||
|
}
|
||||||
|
if got := fold([]Sample{{T: 1}, {T: 2}}).CPUPower; got != 0 {
|
||||||
|
t.Fatalf("no readings folded to %v, want 0", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -32,6 +32,17 @@ func fold(b []Sample) Sample {
|
|||||||
return t / uint64(len(b))
|
return t / uint64(len(b))
|
||||||
}
|
}
|
||||||
out.CPU = sum(func(s Sample) float64 { return s.CPU })
|
out.CPU = sum(func(s Sample) float64 { return s.CPU })
|
||||||
|
// CPU power: over the samples that have a reading, so a helper restart is not a dip to zero
|
||||||
|
var pw, pn float64
|
||||||
|
for _, s := range b {
|
||||||
|
if s.CPUPower > 0 {
|
||||||
|
pw, pn = pw+s.CPUPower, pn+1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out.CPUPower = 0
|
||||||
|
if pn > 0 {
|
||||||
|
out.CPUPower = round1(pw / pn)
|
||||||
|
}
|
||||||
out.Load[0] = sum(func(s Sample) float64 { return s.Load[0] })
|
out.Load[0] = sum(func(s Sample) float64 { return s.Load[0] })
|
||||||
out.Load[1] = sum(func(s Sample) float64 { return s.Load[1] })
|
out.Load[1] = sum(func(s Sample) float64 { return s.Load[1] })
|
||||||
out.Load[2] = sum(func(s Sample) float64 { return s.Load[2] })
|
out.Load[2] = sum(func(s Sample) float64 { return s.Load[2] })
|
||||||
|
|||||||
@@ -250,6 +250,31 @@ func listGPUs(known []*gpuDev) []*gpuDev {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RenderNodeName is the GPU behind a render node ("/dev/dri/renderD128" → "Intel Arc A380"),
|
||||||
|
// named the same way the Processes tab names it; "" when sysfs has nothing for it.
|
||||||
|
func RenderNodeName(node string) string {
|
||||||
|
dev, err := filepath.EvalSymlinks("/sys/class/drm/" + filepath.Base(node) + "/device")
|
||||||
|
if err != nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
driver := ueventValue(dev+"/uevent", "DRIVER")
|
||||||
|
if driver == "" {
|
||||||
|
driver = filepath.Base(readLink(dev + "/driver"))
|
||||||
|
}
|
||||||
|
return gpuName(dev, ueventValue(dev+"/uevent", "PCI_SLOT_NAME"), driver)
|
||||||
|
}
|
||||||
|
|
||||||
|
// RenderNodeIntegrated reports a render node whose GPU sits on the CPU die (PCI root bus 0,
|
||||||
|
// or no PCI bus at all), as opposed to a card on the bus.
|
||||||
|
func RenderNodeIntegrated(node string) bool {
|
||||||
|
dev, err := filepath.EvalSymlinks("/sys/class/drm/" + filepath.Base(node) + "/device")
|
||||||
|
if err != nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
pci := ueventValue(dev+"/uevent", "PCI_SLOT_NAME")
|
||||||
|
return pci == "" || strings.HasPrefix(pci, "0000:00:")
|
||||||
|
}
|
||||||
|
|
||||||
var gpuVendors = map[string]string{"0x8086": "Intel", "0x1002": "AMD", "0x10de": "NVIDIA", "0x1af4": "Virtio", "0x106b": "Apple"}
|
var gpuVendors = map[string]string{"0x8086": "Intel", "0x1002": "AMD", "0x10de": "NVIDIA", "0x1af4": "Virtio", "0x106b": "Apple"}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
|
|||||||
@@ -6,7 +6,9 @@
|
|||||||
// lm-sensors, smartmontools or root. Sections whose source is absent (no fans, no
|
// lm-sensors, smartmontools or root. Sections whose source is absent (no fans, no
|
||||||
// UPS daemon, no hwmon at all on a VM) simply come back empty rather than failing.
|
// UPS daemon, no hwmon at all on a VM) simply come back empty rather than failing.
|
||||||
// The only external commands are optional UPS daemons: pwrstat (CyberPower), upsc
|
// The only external commands are optional UPS daemons: pwrstat (CyberPower), upsc
|
||||||
// (NUT) and apcaccess (apcupsd), tried in that order.
|
// (NUT) and apcaccess (apcupsd), tried in that order. Two readings are root-only, the GPU
|
||||||
|
// client walk and the CPU energy counters; without root they come from small helpers of
|
||||||
|
// this binary started once through sudo or doas (gpu.go, cpupower.go), or not at all.
|
||||||
package metrics
|
package metrics
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -137,6 +139,7 @@ type Sample struct {
|
|||||||
Interval int64 `json:"interval_ms"` // sampling interval in effect
|
Interval int64 `json:"interval_ms"` // sampling interval in effect
|
||||||
CPU float64 `json:"cpu"`
|
CPU float64 `json:"cpu"`
|
||||||
Cores []float64 `json:"cores"`
|
Cores []float64 `json:"cores"`
|
||||||
|
CPUPower float64 `json:"cpu_power_w,omitempty"` // the CPU packages' own energy counters; 0 = no reading
|
||||||
Load [3]float64 `json:"load"`
|
Load [3]float64 `json:"load"`
|
||||||
Mem Mem `json:"mem"`
|
Mem Mem `json:"mem"`
|
||||||
Swap Mem `json:"swap"`
|
Swap Mem `json:"swap"`
|
||||||
@@ -177,6 +180,7 @@ type Collector struct {
|
|||||||
lastTable *ProcTable
|
lastTable *ProcTable
|
||||||
upsProbe func() *UPS // nil when no UPS daemon is installed
|
upsProbe func() *UPS // nil when no UPS daemon is installed
|
||||||
gpu gpuState
|
gpu gpuState
|
||||||
|
power cpuPowerState
|
||||||
prevARC arcPrev
|
prevARC arcPrev
|
||||||
|
|
||||||
subMu sync.Mutex
|
subMu sync.Mutex
|
||||||
@@ -343,6 +347,7 @@ func (c *Collector) sample() {
|
|||||||
|
|
||||||
s := Sample{T: now.UnixMilli(), Interval: c.Interval().Milliseconds(), Temps: []Temp{}, Fans: []Fan{}, Net: []Net{}, Disks: []Disk{}}
|
s := Sample{T: now.UnixMilli(), Interval: c.Interval().Milliseconds(), Temps: []Temp{}, Fans: []Fan{}, Net: []Net{}, Disks: []Disk{}}
|
||||||
s.CPU, s.Cores = c.cpu()
|
s.CPU, s.Cores = c.cpu()
|
||||||
|
s.CPUPower = c.cpuPower(now)
|
||||||
s.Load = loadavg()
|
s.Load = loadavg()
|
||||||
s.Mem, s.Swap = meminfo()
|
s.Mem, s.Swap = meminfo()
|
||||||
s.Uptime = uptime()
|
s.Uptime = uptime()
|
||||||
@@ -1113,6 +1118,7 @@ type ProcTable struct {
|
|||||||
CPUModel string `json:"cpu_model,omitempty"`
|
CPUModel string `json:"cpu_model,omitempty"`
|
||||||
CPUMHz float64 `json:"cpu_mhz,omitempty"`
|
CPUMHz float64 `json:"cpu_mhz,omitempty"`
|
||||||
CPUMaxMHz float64 `json:"cpu_max_mhz,omitempty"`
|
CPUMaxMHz float64 `json:"cpu_max_mhz,omitempty"`
|
||||||
|
CPUPowerW float64 `json:"cpu_power_w,omitempty"`
|
||||||
CPUTemp float64 `json:"cpu_temp,omitempty"`
|
CPUTemp float64 `json:"cpu_temp,omitempty"`
|
||||||
CoreIDs []int `json:"core_ids,omitempty"` // for each logical CPU, the physical core it is a thread of (what the temperature sensors are named by)
|
CoreIDs []int `json:"core_ids,omitempty"` // for each logical CPU, the physical core it is a thread of (what the temperature sensors are named by)
|
||||||
Load [3]float64 `json:"load"`
|
Load [3]float64 `json:"load"`
|
||||||
@@ -1312,7 +1318,7 @@ func (c *Collector) Processes() ProcTable {
|
|||||||
t.Mem, t.Swap = meminfo()
|
t.Mem, t.Swap = meminfo()
|
||||||
if n := len(c.ring); n > 0 {
|
if n := len(c.ring); n > 0 {
|
||||||
last := c.ring[n-1]
|
last := c.ring[n-1]
|
||||||
t.CPU, t.Cores, t.Net, t.CPUTemp = last.CPU, last.Cores, last.Net, cpuTemp(last.Temps)
|
t.CPU, t.Cores, t.Net, t.CPUTemp, t.CPUPowerW = last.CPU, last.Cores, last.Net, cpuTemp(last.Temps), last.CPUPower
|
||||||
t.CoreIDs = coreIDs(len(last.Cores))
|
t.CoreIDs = coreIDs(len(last.Cores))
|
||||||
from := n - 90
|
from := n - 90
|
||||||
if from < 0 {
|
if from < 0 {
|
||||||
|
|||||||
@@ -62,6 +62,9 @@ func flatten(s Sample, want func(id string) bool) []sensors.Sensor {
|
|||||||
if c := cpuTemp(s.Temps); c > 0 {
|
if c := cpuTemp(s.Temps); c > 0 {
|
||||||
f.add("cpu.temp", "CPU temperature", "temperature", "°C", c, t, none)
|
f.add("cpu.temp", "CPU temperature", "temperature", "°C", c, t, none)
|
||||||
}
|
}
|
||||||
|
if s.CPUPower > 0 {
|
||||||
|
f.add("cpu.power", "CPU power", "power", "W", s.CPUPower, t, none)
|
||||||
|
}
|
||||||
for i, w := range []string{"1", "5", "15"} {
|
for i, w := range []string{"1", "5", "15"} {
|
||||||
f.add("load."+w, "Load "+w+" min", "load", "", s.Load[i], t, none)
|
f.add("load."+w, "Load "+w+" min", "load", "", s.Load[i], t, none)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -66,7 +66,6 @@ type Policy struct {
|
|||||||
KeepMin int `json:"keep_min"`
|
KeepMin int `json:"keep_min"`
|
||||||
Prefix string `json:"prefix"`
|
Prefix string `json:"prefix"`
|
||||||
Enabled bool `json:"enabled"`
|
Enabled bool `json:"enabled"`
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Policy) prefix() string { return strings.TrimSpace(p.Prefix) }
|
func (p *Policy) prefix() string { return strings.TrimSpace(p.Prefix) }
|
||||||
|
|||||||
@@ -106,8 +106,8 @@ func TestValidateRefusesNonsense(t *testing.T) {
|
|||||||
{Dataset: "tank", Every: "custom", KeepFor: "6w"},
|
{Dataset: "tank", Every: "custom", KeepFor: "6w"},
|
||||||
{Dataset: "tank", Every: "custom", Cron: "not a schedule", KeepFor: "6w"},
|
{Dataset: "tank", Every: "custom", Cron: "not a schedule", KeepFor: "6w"},
|
||||||
{Dataset: "tank", Every: "daily", At: "25:00", KeepFor: "6w"},
|
{Dataset: "tank", Every: "daily", At: "25:00", KeepFor: "6w"},
|
||||||
{Dataset: "tank", Every: "daily", At: "02:30"}, // no retention
|
{Dataset: "tank", Every: "daily", At: "02:30"}, // no retention
|
||||||
{Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6 weeks"}, // unparseable
|
{Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6 weeks"}, // unparseable
|
||||||
{Dataset: "tank", Every: "monthly", At: "02:30", Day: 31, KeepFor: "6w"}, // not every month has one
|
{Dataset: "tank", Every: "monthly", At: "02:30", Day: 31, KeepFor: "6w"}, // not every month has one
|
||||||
{Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6w", Prefix: "a--b"},
|
{Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6w", Prefix: "a--b"},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -146,7 +146,7 @@ func (m *Manager) prune() {
|
|||||||
}
|
}
|
||||||
var finished []*Task
|
var finished []*Task
|
||||||
for _, t := range m.tasks {
|
for _, t := range m.tasks {
|
||||||
if !t.Running {
|
if !t.running() {
|
||||||
finished = append(finished, t)
|
finished = append(finished, t)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -184,13 +184,20 @@ func (m *Manager) RunningOf(kind string) (*Task, bool) {
|
|||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
defer m.mu.Unlock()
|
defer m.mu.Unlock()
|
||||||
for _, t := range m.tasks {
|
for _, t := range m.tasks {
|
||||||
if t.Kind == kind && t.Running {
|
if t.Kind == kind && t.running() {
|
||||||
return t, true
|
return t, true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil, false
|
return nil, false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// running reads the flag the waiter goroutine clears, under the task's own lock.
|
||||||
|
func (t *Task) running() bool {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
return t.Running
|
||||||
|
}
|
||||||
|
|
||||||
func (t *Task) Snapshot(withOutput bool) Snapshot {
|
func (t *Task) Snapshot(withOutput bool) Snapshot {
|
||||||
t.mu.Lock()
|
t.mu.Lock()
|
||||||
defer t.mu.Unlock()
|
defer t.mu.Unlock()
|
||||||
|
|||||||
@@ -64,6 +64,9 @@ func New(sudo bool, tm *tasks.Manager) *Manager {
|
|||||||
return m
|
return m
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// PM is the package manager found: apt | dnf | zypper | pacman | apk, or "".
|
||||||
|
func (m *Manager) PM() string { return m.pm }
|
||||||
|
|
||||||
func (m *Manager) sudoArgs(args ...string) []string {
|
func (m *Manager) sudoArgs(args ...string) []string {
|
||||||
if m.sudo {
|
if m.sudo {
|
||||||
return priv.Wrap(args...)
|
return priv.Wrap(args...)
|
||||||
|
|||||||
Generated
+7
@@ -11,6 +11,7 @@
|
|||||||
"@xterm/addon-fit": "^0.10.0",
|
"@xterm/addon-fit": "^0.10.0",
|
||||||
"@xterm/addon-web-links": "^0.11.0",
|
"@xterm/addon-web-links": "^0.11.0",
|
||||||
"@xterm/xterm": "^5.5.0",
|
"@xterm/xterm": "^5.5.0",
|
||||||
|
"hls.js": "^1.7.3",
|
||||||
"qrcode-generator": "^2.0.4",
|
"qrcode-generator": "^2.0.4",
|
||||||
"react": "^18.3.1",
|
"react": "^18.3.1",
|
||||||
"react-dom": "^18.3.1"
|
"react-dom": "^18.3.1"
|
||||||
@@ -1426,6 +1427,12 @@
|
|||||||
"node": ">=6.9.0"
|
"node": ">=6.9.0"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"node_modules/hls.js": {
|
||||||
|
"version": "1.7.3",
|
||||||
|
"resolved": "https://registry.npmjs.org/hls.js/-/hls.js-1.7.3.tgz",
|
||||||
|
"integrity": "sha512-MsPlx6yVW4Qv4C7mEVou4/gk/5cN2dVxOTnrNjPSmyH2f0Ln+/UkG10Ax1PE6OvrmTVPKgQYozTtBaPOuCzq7A==",
|
||||||
|
"license": "Apache-2.0"
|
||||||
|
},
|
||||||
"node_modules/js-tokens": {
|
"node_modules/js-tokens": {
|
||||||
"version": "4.0.0",
|
"version": "4.0.0",
|
||||||
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz",
|
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz",
|
||||||
|
|||||||
+2
-1
@@ -7,12 +7,13 @@
|
|||||||
"dev": "vite",
|
"dev": "vite",
|
||||||
"build": "tsc --noEmit && vite build",
|
"build": "tsc --noEmit && vite build",
|
||||||
"icons": "node scripts/icons.mjs",
|
"icons": "node scripts/icons.mjs",
|
||||||
"test": "node scripts/test-graphcols.mjs"
|
"test": "node scripts/test-graphcols.mjs && node scripts/test-css-imports.mjs && node scripts/test-resolve-path.mjs"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@xterm/addon-fit": "^0.10.0",
|
"@xterm/addon-fit": "^0.10.0",
|
||||||
"@xterm/addon-web-links": "^0.11.0",
|
"@xterm/addon-web-links": "^0.11.0",
|
||||||
"@xterm/xterm": "^5.5.0",
|
"@xterm/xterm": "^5.5.0",
|
||||||
|
"hls.js": "^1.7.3",
|
||||||
"qrcode-generator": "^2.0.4",
|
"qrcode-generator": "^2.0.4",
|
||||||
"react": "^18.3.1",
|
"react": "^18.3.1",
|
||||||
"react-dom": "^18.3.1"
|
"react-dom": "^18.3.1"
|
||||||
|
|||||||
@@ -0,0 +1,20 @@
|
|||||||
|
// node scripts/test-css-imports.mjs — every stylesheet the UI imports is one somebody has read.
|
||||||
|
// A third-party stylesheet usually carries its demo page's global rules (body, *, button); on
|
||||||
|
// 2026-10-09 one restyled every button in the app. Add a file here only once its rules are
|
||||||
|
// known to touch nothing but its own component.
|
||||||
|
import { readdirSync, readFileSync, statSync } from 'node:fs'
|
||||||
|
import { join, relative } from 'node:path'
|
||||||
|
|
||||||
|
const allowed = new Set(['./styles.css', '@xterm/xterm/css/xterm.css'])
|
||||||
|
const src = new URL('../src', import.meta.url).pathname
|
||||||
|
const files = []
|
||||||
|
const walk = (d) => { for (const n of readdirSync(d)) { const p = join(d, n); if (statSync(p).isDirectory()) { if (n !== 'vendor') walk(p) } else if (/\.(ts|tsx)$/.test(n)) files.push(p) } }
|
||||||
|
walk(src)
|
||||||
|
const fails = []
|
||||||
|
for (const f of files) {
|
||||||
|
for (const m of readFileSync(f, 'utf8').matchAll(/import\s+(?:[^'"]*from\s+)?['"]([^'"]+\.css)['"]/g)) {
|
||||||
|
if (!allowed.has(m[1])) fails.push(`${relative(src, f)} imports ${m[1]}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (fails.length) { console.error('css imports: not allowed:\n ' + fails.join('\n ')); process.exit(1) }
|
||||||
|
console.log(`css imports: ok (${files.length} files)`)
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
// node scripts/test-resolve-path.mjs — the path box resolves like a shell: relative to here,
|
||||||
|
// . and .. applied, trailing slashes dropped, never anything but an absolute path.
|
||||||
|
import { readFileSync } from 'node:fs'
|
||||||
|
const src = readFileSync(new URL('../src/Files.tsx', import.meta.url), 'utf8')
|
||||||
|
const m = src.match(/export function resolvePath[\s\S]*?\n}\n/)
|
||||||
|
if (!m) { console.error('resolvePath not found'); process.exit(1) }
|
||||||
|
const resolvePath = new Function(m[0].replace('export function', 'return function').replace(/: string\[\]/g, '').replace(/: string/g, '') + ' ')()
|
||||||
|
const cases = [
|
||||||
|
['/tank/media', '/x', '/tank/media'], ['/tank/media/', '/x', '/tank/media'], ['', '/here', '/here'], [' ', '/here', '/here'],
|
||||||
|
['sub', '/here', '/here/sub'], ['./sub/', '/here', '/here/sub'], ['..', '/a/b', '/a'], ['../..', '/a/b', '/'], ['../../..', '/a', '/'],
|
||||||
|
['/', '/a', '/'], ['a//b', '/', '/a/b'], ['/tank/./x/../y', '/', '/tank/y'], ['sp ace #1', '/d', '/d/sp ace #1'],
|
||||||
|
]
|
||||||
|
let bad = 0
|
||||||
|
for (const [input, here, want] of cases) { const got = resolvePath(input, here); if (got !== want) { bad++; console.error(`resolvePath(${JSON.stringify(input)}, ${here}) = ${got}, want ${want}`) } }
|
||||||
|
if (bad) process.exit(1)
|
||||||
|
console.log(`resolve path: ok (${cases.length} cases)`)
|
||||||
+2
-2
@@ -355,7 +355,7 @@ function StackDetail({ name, startTask, onBack, onChanged, onDuplicate }: { name
|
|||||||
<div className="svc-detail">
|
<div className="svc-detail">
|
||||||
<CloseBtn onClick={onBack} />
|
<CloseBtn onClick={onBack} />
|
||||||
<div className="svc-head"><button className="btn back" onClick={onBack}>‹ Back</button><span className={'sdot ' + (st.containers.length === 0 ? '' : st.running === st.containers.length ? 'ok' : st.running ? 'warn' : 'bad')} /><span className="dname">{st.name}</span><span className="muted small">{st.running}/{st.containers.length} running</span></div>
|
<div className="svc-head"><button className="btn back" onClick={onBack}>‹ Back</button><span className={'sdot ' + (st.containers.length === 0 ? '' : st.running === st.containers.length ? 'ok' : st.running ? 'warn' : 'bad')} /><span className="dname">{st.name}</span><span className="muted small">{st.running}/{st.containers.length} running</span></div>
|
||||||
<div className="muted small mono" style={{ overflowWrap: 'anywhere' }}>{file === 'env' ? (st.host_file.replace(/[^/]+$/, '') + '.env') : (st.host_files?.[file] ?? st.host_file)}{st.held_by && <span className="muted"> · kept on <b>{st.held_by}</b>, deployed to this machine over {st.remote}</span>}{st.via && <span className="muted"> · compose runs inside <b>{st.via}</b>, as it was deployed</span>}{st.has_env && ' · has a .env beside it'}</div>
|
<div className="muted small mono" style={{ overflowWrap: 'anywhere' }}>{file === 'env' ? (st.host_file.replace(/[^/]+$/, '') + '.env') : (st.host_files?.[file] ?? st.host_file)}{st.via && <span className="muted"> · compose runs inside <b>{st.via}</b>, as it was deployed</span>}{st.has_env && ' · has a .env beside it'}</div>
|
||||||
{((st.host_files?.length ?? 0) > 1 || st.has_env) && <div className="chips" title="this stack is deployed from several files, in this order; Deploy uses all of them">{st.host_files!.map((f, i) => <button key={f} className={'chip' + (i === file ? ' on' : '')} onClick={async () => { if (dirty && !(await askYes('Discard unsaved changes to this file?', 'Discard'))) return; setFile(i) }}>{f.split('/').pop()}</button>)}{st.has_env && <button className={'chip' + (file === 'env' ? ' on' : '')} title="the .env beside the compose file: variables compose substitutes" onClick={async () => { if (dirty && !(await askYes('Discard unsaved changes to this file?', 'Discard'))) return; setFile('env') }}>.env</button>}</div>}
|
{((st.host_files?.length ?? 0) > 1 || st.has_env) && <div className="chips" title="this stack is deployed from several files, in this order; Deploy uses all of them">{st.host_files!.map((f, i) => <button key={f} className={'chip' + (i === file ? ' on' : '')} onClick={async () => { if (dirty && !(await askYes('Discard unsaved changes to this file?', 'Discard'))) return; setFile(i) }}>{f.split('/').pop()}</button>)}{st.has_env && <button className={'chip' + (file === 'env' ? ' on' : '')} title="the .env beside the compose file: variables compose substitutes" onClick={async () => { if (dirty && !(await askYes('Discard unsaved changes to this file?', 'Discard'))) return; setFile('env') }}>.env</button>}</div>}
|
||||||
{file === 'env' && !reveal ? <div className="env-masked"><pre className="editor stack-editor">{maskEnv(text)}</pre><button className="btn" onClick={() => setReveal(true)}>Reveal and edit</button></div> :
|
{file === 'env' && !reveal ? <div className="env-masked"><pre className="editor stack-editor">{maskEnv(text)}</pre><button className="btn" onClick={() => setReveal(true)}>Reveal and edit</button></div> :
|
||||||
<CodeEditor className="stack-editor" language={file === 'env' ? 'env' : 'yaml'} value={text} onChange={setText} />}
|
<CodeEditor className="stack-editor" language={file === 'env' ? 'env' : 'yaml'} value={text} onChange={setText} />}
|
||||||
@@ -447,7 +447,7 @@ export default function Docker() {
|
|||||||
{shownTab === 'stacks' && (stacks ?? []).map((st) => (
|
{shownTab === 'stacks' && (stacks ?? []).map((st) => (
|
||||||
<div key={st.name} className={'svc-row' + (st.name === selStack ? ' active' : '')} onClick={() => st.editable ? setSel('stack:' + st.name) : toast(st.note ?? 'not editable here', 'info')}>
|
<div key={st.name} className={'svc-row' + (st.name === selStack ? ' active' : '')} onClick={() => st.editable ? setSel('stack:' + st.name) : toast(st.note ?? 'not editable here', 'info')}>
|
||||||
<span className={'sdot ' + (st.containers.length === 0 ? '' : st.running === st.containers.length ? 'ok' : st.running ? 'warn' : 'bad')} />
|
<span className={'sdot ' + (st.containers.length === 0 ? '' : st.running === st.containers.length ? 'ok' : st.running ? 'warn' : 'bad')} />
|
||||||
<div className="svc-name"><div className="n">{st.name}</div><div className="d">{st.editable ? (st.held_by ? `on ${st.held_by}: ` : '') + st.host_file : st.note}</div></div>
|
<div className="svc-name"><div className="n">{st.name}</div><div className="d">{st.editable ? st.host_file : st.note}</div></div>
|
||||||
<div className="svc-state">{st.containers.length === 0 ? 'not deployed' : `${st.running}/${st.containers.length}`}{!st.editable && <><br />read-only</>}</div>
|
<div className="svc-state">{st.containers.length === 0 ? 'not deployed' : `${st.running}/${st.containers.length}`}{!st.editable && <><br />read-only</>}</div>
|
||||||
</div>
|
</div>
|
||||||
))}
|
))}
|
||||||
|
|||||||
+229
-15
@@ -1,10 +1,25 @@
|
|||||||
import { DragEvent, useCallback, useEffect, useRef, useState } from 'react'
|
import { DragEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
|
||||||
import { api, FileEntry, FileInfo } from './api'
|
import { api, FileEntry, FileInfo, TaskInfo, ThumbStatus } from './api'
|
||||||
import { fmtBytes } from './fmt'
|
import { fmtBytes } from './fmt'
|
||||||
import { Btn, Loading, Spinner, askText, askYes, setUnsaved, toast, useEscape } from './ui'
|
import { Btn, Loading, Spinner, askText, askYes, setUnsaved, toast, useEscape } from './ui'
|
||||||
import { useHashSel } from './hashsel'
|
import { useHashSel } from './hashsel'
|
||||||
|
import TaskOutput from './TaskOutput'
|
||||||
|
import Player from './Player'
|
||||||
|
|
||||||
const join = (dir: string, name: string) => (dir === '/' ? '/' + name : dir + '/' + name)
|
const join = (dir: string, name: string) => (dir === '/' ? '/' + name : dir + '/' + name)
|
||||||
|
// A typed path, made absolute: relative to the folder being looked at, "." and ".." applied,
|
||||||
|
// trailing slashes dropped — what a shell would make of it.
|
||||||
|
export function resolvePath(input: string, here: string): string {
|
||||||
|
const raw = input.trim()
|
||||||
|
if (!raw) return here
|
||||||
|
const parts = (raw.startsWith('/') ? [] : here.split('/')).concat(raw.split('/'))
|
||||||
|
const out: string[] = []
|
||||||
|
for (const p of parts) {
|
||||||
|
if (p === '' || p === '.') continue
|
||||||
|
if (p === '..') out.pop(); else out.push(p)
|
||||||
|
}
|
||||||
|
return '/' + out.join('/')
|
||||||
|
}
|
||||||
const fmtTime = (ms: number) => { const d = new Date(ms); const now = new Date(); return d.toDateString() === now.toDateString() ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) : d.toLocaleDateString(undefined, { year: d.getFullYear() === now.getFullYear() ? undefined : 'numeric', month: 'short', day: '2-digit' }) }
|
const fmtTime = (ms: number) => { const d = new Date(ms); const now = new Date(); return d.toDateString() === now.toDateString() ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) : d.toLocaleDateString(undefined, { year: d.getFullYear() === now.getFullYear() ? undefined : 'numeric', month: 'short', day: '2-digit' }) }
|
||||||
const isImage = (n: string) => /\.(png|jpe?g|gif|webp|svg|bmp|ico|avif)$/i.test(n)
|
const isImage = (n: string) => /\.(png|jpe?g|gif|webp|svg|bmp|ico|avif)$/i.test(n)
|
||||||
const shellQuote = (s: string) => "'" + s.replace(/'/g, "'\\''") + "'"
|
const shellQuote = (s: string) => "'" + s.replace(/'/g, "'\\''") + "'"
|
||||||
@@ -14,7 +29,9 @@ function icon(e: FileEntry): string {
|
|||||||
if (e.type === 'dir') return e.target ? '🔗' : '📁'
|
if (e.type === 'dir') return e.target ? '🔗' : '📁'
|
||||||
if (e.type === 'link') return e.broken ? '⛓️💥' : '🔗'
|
if (e.type === 'link') return e.broken ? '⛓️💥' : '🔗'
|
||||||
if (e.type === 'other') return '⚙️'
|
if (e.type === 'other') return '⚙️'
|
||||||
if (isImage(e.name)) return '🖼️'
|
if (e.preview === 'image' || isImage(e.name)) return '🖼️'
|
||||||
|
if (e.preview === 'video') return '🎞️'
|
||||||
|
if (e.preview === 'audio') return '🎵'
|
||||||
if (/\.(zip|gz|bz2|xz|zst|tar|tgz|7z|rar)$/i.test(e.name)) return '🗜️'
|
if (/\.(zip|gz|bz2|xz|zst|tar|tgz|7z|rar)$/i.test(e.name)) return '🗜️'
|
||||||
if (/\.(sh|bash|zsh|py|go|js|ts|tsx|rb|pl|php|rs|c|h|cpp|java|lua)$/i.test(e.name) || e.mode.includes('x')) return '📜'
|
if (/\.(sh|bash|zsh|py|go|js|ts|tsx|rb|pl|php|rs|c|h|cpp|java|lua)$/i.test(e.name) || e.mode.includes('x')) return '📜'
|
||||||
if (/\.(log|txt|md|json|ya?ml|toml|ini|conf|cfg|env|service|timer)$/i.test(e.name)) return '📄'
|
if (/\.(log|txt|md|json|ya?ml|toml|ini|conf|cfg|env|service|timer)$/i.test(e.name)) return '📄'
|
||||||
@@ -22,6 +39,163 @@ function icon(e: FileEntry): string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type Sort = 'name' | 'size' | 'mtime'
|
type Sort = 'name' | 'size' | 'mtime'
|
||||||
|
const PREVIEWS_KEY = 'hamlab.files.previews'
|
||||||
|
const byName = (a: FileEntry, b: FileEntry) => a.name.localeCompare(b.name, undefined, { numeric: true, sensitivity: 'base' })
|
||||||
|
// A name is one path segment. ("../x" and "a/b" would quietly act somewhere else.)
|
||||||
|
const okName = (n?: string | null): n is string => { const t = n?.trim(); if (!t) return false; if (t.includes('/') || t === '.' || t === '..') { toast('a name can\'t contain / or be . or .. — use Move… to put it elsewhere', 'err'); return false } return true }
|
||||||
|
|
||||||
|
// ---- move: walk to the destination, then "Move here" ----
|
||||||
|
function MovePicker({ item, from, sudo, hidden, onClose, onMoved }: { item: FileEntry; from: string; sudo: boolean; hidden: boolean; onClose: () => void; onMoved: (task?: TaskInfo) => void }) {
|
||||||
|
const src = join(from, item.name)
|
||||||
|
const isFolder = item.type === 'dir' && !item.target
|
||||||
|
const [hist, setHist] = useState<string[]>([from]) // where the picker has been, for back and forward
|
||||||
|
const [at, setAt] = useState(0)
|
||||||
|
const dir = hist[at]
|
||||||
|
const [entries, setEntries] = useState<FileEntry[] | null>(null)
|
||||||
|
const [err, setErr] = useState('')
|
||||||
|
const [pathInput, setPathInput] = useState(from)
|
||||||
|
const [busy, setBusy] = useState(false)
|
||||||
|
const load = useCallback(async (p: string) => {
|
||||||
|
setEntries(null); setErr('')
|
||||||
|
try { const l = await api.files(p, sudo); setEntries(l.entries) } catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setEntries([]) }
|
||||||
|
}, [sudo])
|
||||||
|
useEffect(() => { setPathInput(dir); load(dir) }, [dir, load])
|
||||||
|
useEscape(onClose)
|
||||||
|
const go = (input: string) => {
|
||||||
|
const p = resolvePath(input, dir)
|
||||||
|
if (p === dir) return
|
||||||
|
setHist((h) => [...h.slice(0, at + 1), p]); setAt(at + 1)
|
||||||
|
}
|
||||||
|
const up = () => { if (dir !== '/') go(dir.slice(0, dir.lastIndexOf('/')) || '/') }
|
||||||
|
const folders = (entries ?? []).filter((e) => e.type === 'dir' && (hidden || !e.name.startsWith('.'))).sort(byName)
|
||||||
|
const crumbs = dir === '/' ? [] : dir.slice(1).split('/')
|
||||||
|
const dest = join(dir, item.name)
|
||||||
|
const why = dir === from ? 'it is already here'
|
||||||
|
: isFolder && (dir === src || dir.startsWith(src + '/')) ? 'a folder can\'t move into itself'
|
||||||
|
: (entries ?? []).some((e) => e.name === item.name) ? `something called ${item.name} is already there` : ''
|
||||||
|
const move = async () => {
|
||||||
|
setBusy(true)
|
||||||
|
try {
|
||||||
|
const t = await api.fileMove(src, dest, sudo)
|
||||||
|
if (t) toast(`${dir} is on another filesystem — copying ${item.name} there, then removing it here`, 'info')
|
||||||
|
else toast(`moved ${item.name} to ${dir}`)
|
||||||
|
onMoved(t)
|
||||||
|
} catch (e) { toast(`move — ${e instanceof Error ? e.message : 'failed'}`, 'err'); setBusy(false) }
|
||||||
|
}
|
||||||
|
const mkdir = async () => {
|
||||||
|
const n = await askText('New folder name')
|
||||||
|
if (!okName(n)) return
|
||||||
|
try { await api.fileMkdir(join(dir, n.trim()), sudo); go(join(dir, n.trim())) } catch (e) { toast(`create folder — ${e instanceof Error ? e.message : 'failed'}`, 'err') }
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
<div className="modal-back" onClick={onClose}>
|
||||||
|
<div className="modal mover" onClick={(e) => e.stopPropagation()}>
|
||||||
|
<h3>Move {isFolder ? 'folder ' : ''}<span className="mono">{item.name}</span></h3>
|
||||||
|
<div className="mover-nav">
|
||||||
|
<button className="icon-btn" title="back" disabled={at === 0} onClick={() => setAt(at - 1)}>‹</button>
|
||||||
|
<button className="icon-btn" title="forward" disabled={at >= hist.length - 1} onClick={() => setAt(at + 1)}>›</button>
|
||||||
|
<button className="icon-btn" title="up one level" disabled={dir === '/'} onClick={up}>↑</button>
|
||||||
|
<form className="crumbs" onSubmit={(e) => { e.preventDefault(); go(pathInput) }}>
|
||||||
|
<button type="button" className="crumb" onClick={() => go('/')}>/</button>
|
||||||
|
{crumbs.map((c, i) => <button type="button" key={i} className="crumb" onClick={() => go('/' + crumbs.slice(0, i + 1).join('/'))}>{c}</button>)}
|
||||||
|
<input className="path-input" value={pathInput} onChange={(e) => setPathInput(e.target.value)} onBlur={() => setPathInput(dir)} spellCheck={false} autoCapitalize="off" autoCorrect="off" />
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
<div className="mover-list">
|
||||||
|
{entries === null && <Loading label="Listing…" />}
|
||||||
|
{err && <div className="error" style={{ margin: 10 }}>{err}</div>}
|
||||||
|
{folders.map((e) => {
|
||||||
|
const self = dir === from && e.name === item.name
|
||||||
|
return (
|
||||||
|
<div key={e.name} className={'mover-row' + (self ? ' self' : '')} onClick={() => { if (!self) go(join(dir, e.name)) }}>
|
||||||
|
<span className="ficon">{e.target ? '🔗' : '📁'}</span>
|
||||||
|
<span className="fname">{e.name}</span>
|
||||||
|
{self && <span className="muted small">this is what's moving</span>}
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
})}
|
||||||
|
{entries !== null && folders.length === 0 && !err && <div className="muted small" style={{ padding: 12 }}>No folders here — “Move here” puts it in this one.</div>}
|
||||||
|
</div>
|
||||||
|
<div className="modal-foot">
|
||||||
|
<div className="svc-actions">
|
||||||
|
<button className="btn" onClick={mkdir}>+ folder</button>
|
||||||
|
<span className="spacer" />
|
||||||
|
<button className="btn" onClick={onClose}>Cancel</button>
|
||||||
|
<Btn className="btn on" disabled={!!why || entries === null} busy={busy} onClick={move}>Move here</Btn>
|
||||||
|
</div>
|
||||||
|
<div className="muted small mono mover-dest">{why || `→ ${dest}`}</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- previews ----
|
||||||
|
|
||||||
|
// Rows ask for their thumbnail only as they scroll near: a browser's own lazy loading
|
||||||
|
// watches the window, and this list scrolls inside a box of its own. One observer for the
|
||||||
|
// list, a callback per row, each fired once.
|
||||||
|
class Near {
|
||||||
|
private obs: IntersectionObserver
|
||||||
|
private cbs = new Map<Element, () => void>()
|
||||||
|
constructor(root: Element) {
|
||||||
|
this.obs = new IntersectionObserver((ents) => {
|
||||||
|
for (const e of ents) if (e.isIntersecting) { const cb = this.cbs.get(e.target); if (cb) { this.forget(e.target); cb() } }
|
||||||
|
}, { root, rootMargin: '400px 0px' })
|
||||||
|
}
|
||||||
|
watch(el: Element, cb: () => void) { this.cbs.set(el, cb); this.obs.observe(el); return () => this.forget(el) }
|
||||||
|
private forget(el: Element) { this.cbs.delete(el); this.obs.unobserve(el) }
|
||||||
|
}
|
||||||
|
|
||||||
|
function Thumb({ url, kind, near, fallback }: { url: string; kind: 'image' | 'video' | 'audio'; near: Near | null; fallback: string }) {
|
||||||
|
const ref = useRef<HTMLSpanElement>(null)
|
||||||
|
const [show, setShow] = useState(false)
|
||||||
|
const [failed, setFailed] = useState('')
|
||||||
|
useEffect(() => { const el = ref.current; if (!el || !near || show) return; return near.watch(el, () => setShow(true)) }, [near, show])
|
||||||
|
useEffect(() => { setFailed('') }, [url]) // a changed file is a new URL, and worth another try
|
||||||
|
const broken = failed === url
|
||||||
|
return (
|
||||||
|
<span ref={ref} className={'fthumb ' + kind} title={broken ? 'no preview for this file' : undefined}>
|
||||||
|
{show && !broken && <img src={url} alt="" decoding="async" onError={() => setFailed(url)} />}
|
||||||
|
{(!show || broken) && <span className="ficon">{fallback}</span>}
|
||||||
|
</span>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// What the previews button says about how they get made here, and the one-tap fixes.
|
||||||
|
function ThumbNote({ st, sudo, onChanged }: { st: ThumbStatus | null; sudo: boolean; onChanged: () => void }) {
|
||||||
|
const [task, setTask] = useState<{ id: string; restart: boolean } | null>(null)
|
||||||
|
const [waiting, setWaiting] = useState(false)
|
||||||
|
// the GPU fix ends by restarting Ham Lab: poll until it answers again, then re-read the status
|
||||||
|
useEffect(() => {
|
||||||
|
if (!waiting) return
|
||||||
|
const t = setInterval(async () => { try { await api.thumbStatus(sudo, true); setWaiting(false); onChanged() } catch { /* restarting */ } }, 3000)
|
||||||
|
return () => clearInterval(t)
|
||||||
|
}, [waiting, sudo, onChanged])
|
||||||
|
if (!st) return null
|
||||||
|
const blocked = (st.unusable ?? []).filter((g) => g.permission)
|
||||||
|
const start = async (label: string, fn: () => Promise<{ id: string }>, restart: boolean) => {
|
||||||
|
try { const t = await fn(); setTask({ id: t.id, restart }) } catch (e) { toast(`${label} — ${e instanceof Error ? e.message : 'failed'}`, 'err') }
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
{!st.ok && (
|
||||||
|
<div className="thumb-note notice">
|
||||||
|
<span>Previews need ffmpeg: {st.error || 'it is not installed on this machine'}.</span>
|
||||||
|
{st.install && !task && <Btn className="btn on" onClick={() => start('install', api.thumbInstall, false)}>Install ffmpeg with {st.install.pm}</Btn>}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{st.ok && blocked.length > 0 && (
|
||||||
|
<div className="thumb-note notice">
|
||||||
|
<span>{blocked.map((g) => g.name || g.device).join(', ')} is here, but Ham Lab runs as {st.user} and can't open {blocked.length > 1 ? 'them' : 'it'} ({blocked[0].reason}). Previews of videos are on the CPU until it can.</span>
|
||||||
|
{st.gpu_fix && !task && !waiting && <Btn className="btn on" confirm="Ham Lab adds itself to the GPU's group and restarts. Open terminals are interrupted. Continue?" onClick={() => start('GPU access', api.thumbGPUAccess, true)}>Allow GPU access</Btn>}
|
||||||
|
{waiting && <span className="muted small"><Spinner /> Ham Lab is restarting; the page reconnects by itself.</span>}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{task && <TaskOutput id={task.id} onDone={(t) => { const ok = t.exit_code === 0 && !t.error; if (ok && task.restart) setWaiting(true); else if (ok) onChanged(); else toast('failed — the output says where', 'err'); setTask(null) }} />}
|
||||||
|
</>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
// ---- editor / viewer ----
|
// ---- editor / viewer ----
|
||||||
function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolean; onClose: () => void; onChanged: () => void }) {
|
function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolean; onClose: () => void; onChanged: () => void }) {
|
||||||
@@ -30,11 +204,12 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
|
|||||||
const [orig, setOrig] = useState('')
|
const [orig, setOrig] = useState('')
|
||||||
const [err, setErr] = useState('')
|
const [err, setErr] = useState('')
|
||||||
const [saving, setSaving] = useState(false)
|
const [saving, setSaving] = useState(false)
|
||||||
|
const [cantPlay, setCantPlay] = useState(false) // the browser tried the file as it is and gave up
|
||||||
const name = path.split('/').pop() ?? path
|
const name = path.split('/').pop() ?? path
|
||||||
const dirty = text !== null && text !== orig
|
const dirty = text !== null && text !== orig
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
setInfo(null); setText(null); setErr('')
|
setInfo(null); setText(null); setErr(''); setCantPlay(false)
|
||||||
let stop = false
|
let stop = false
|
||||||
;(async () => {
|
;(async () => {
|
||||||
try {
|
try {
|
||||||
@@ -92,8 +267,16 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
|
|||||||
<textarea className="editor file-editor" value={text} spellCheck={false} onChange={(e) => setText(e.target.value)} />
|
<textarea className="editor file-editor" value={text} spellCheck={false} onChange={(e) => setText(e.target.value)} />
|
||||||
)}
|
)}
|
||||||
{info && text === null && isImage(name) && <div className="preview"><img src={api.fileReadURL(path, sudo)} alt={name} /></div>}
|
{info && text === null && isImage(name) && <div className="preview"><img src={api.fileReadURL(path, sudo)} alt={name} /></div>}
|
||||||
{info && text === null && !isImage(name) && !err && (
|
{/* Video: the browser's own decoder first, and a re-encode on the machine when that fails. Audio as it is. */}
|
||||||
<div className="notice">{info.type === 'dir' ? 'This is a directory.' : info.size > MAX_EDIT ? `Too large to edit here (${fmtBytes(info.size)}). Download it instead.` : 'Binary file. Download it instead.'}</div>
|
{info && text === null && !isImage(name) && info.preview === 'video' && <Player path={path} sudo={sudo} name={name} />}
|
||||||
|
{info && text === null && !isImage(name) && info.preview === 'audio' && !cantPlay && (
|
||||||
|
<div className="preview audio"><div className="audio-box"><div className="ficon">🎵</div><div className="mono small">{name}</div><audio controls preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => setCantPlay(true)} /></div></div>
|
||||||
|
)}
|
||||||
|
{info && text === null && !isImage(name) && info.preview === 'audio' && cantPlay && (
|
||||||
|
<div className="notice">This browser can't play {name} as it is — the codec isn't one it decodes. Download it instead.</div>
|
||||||
|
)}
|
||||||
|
{info && text === null && !isImage(name) && !info.preview && !err && (
|
||||||
|
<div className="notice">{info.type === 'dir' ? 'This is a directory.' : info.type === 'other' ? 'This is a device, pipe or socket — nothing to show.' : info.broken ? `This link points at nothing (${info.target}).` : info.size > MAX_EDIT ? `Too large to edit here (${fmtBytes(info.size)}). Download it instead.` : 'Binary file. Download it instead.'}</div>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
)
|
)
|
||||||
@@ -114,10 +297,19 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
|||||||
const [pathInput, setPathInput] = useState(path)
|
const [pathInput, setPathInput] = useState(path)
|
||||||
const [open, setOpen] = useState<string | null>(null)
|
const [open, setOpen] = useState<string | null>(null)
|
||||||
const [menu, setMenu] = useState<string | null>(null)
|
const [menu, setMenu] = useState<string | null>(null)
|
||||||
|
const [moving, setMoving] = useState<FileEntry | null>(null)
|
||||||
|
const [moveTask, setMoveTask] = useState<TaskInfo | null>(null)
|
||||||
const [upload, setUpload] = useState<{ name: string; pct: number; n: number; of: number } | null>(null)
|
const [upload, setUpload] = useState<{ name: string; pct: number; n: number; of: number } | null>(null)
|
||||||
const [drag, setDrag] = useState(false)
|
const [drag, setDrag] = useState(false)
|
||||||
const fileInput = useRef<HTMLInputElement>(null)
|
const fileInput = useRef<HTMLInputElement>(null)
|
||||||
const canSudo = useRef(true)
|
const canSudo = useRef(true)
|
||||||
|
const [previews, setPreviews] = useState<boolean>(() => { try { return localStorage.getItem(PREVIEWS_KEY) === '1' } catch { return false } })
|
||||||
|
const [thumbs, setThumbs] = useState<ThumbStatus | null>(null)
|
||||||
|
const [listEl, setListEl] = useState<HTMLDivElement | null>(null)
|
||||||
|
const near = useMemo(() => (listEl && previews ? new Near(listEl) : null), [listEl, previews])
|
||||||
|
const readThumbs = useCallback((refresh = false) => { api.thumbStatus(sudo, refresh).then(setThumbs).catch((e) => setThumbs({ ok: false, engine: '', summary: '', gpus: [], error: e instanceof Error ? e.message : 'failed', user: '', cache: { path: '', entries: 0, bytes: 0 } })) }, [sudo])
|
||||||
|
useEffect(() => { try { localStorage.setItem(PREVIEWS_KEY, previews ? '1' : '0') } catch { /* private mode */ } }, [previews])
|
||||||
|
useEffect(() => { if (previews) readThumbs(); else setThumbs(null) }, [previews, readThumbs])
|
||||||
|
|
||||||
const load = useCallback(async (p = path, s = sudo) => {
|
const load = useCallback(async (p = path, s = sudo) => {
|
||||||
try {
|
try {
|
||||||
@@ -127,15 +319,22 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
|||||||
sessionStorage.setItem('hamlab.files.path', p) // only a folder that opened is worth coming back to
|
sessionStorage.setItem('hamlab.files.path', p) // only a folder that opened is worth coming back to
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
const msg = e instanceof Error ? e.message : 'failed'
|
const msg = e instanceof Error ? e.message : 'failed'
|
||||||
|
if (/not a directory/i.test(msg) && p !== '/') { // a file was typed: open it, from its folder
|
||||||
|
setPath(p.slice(0, p.lastIndexOf('/')) || '/'); setOpen(p); return
|
||||||
|
}
|
||||||
|
if (/no such file/i.test(msg) && p !== '/' && !sessionStorage.getItem('hamlab.files.lost')) { // gone since last time (another machine, a deleted folder)
|
||||||
|
sessionStorage.setItem('hamlab.files.lost', '1'); toast(`${p} isn't here — showing /`, 'info'); setPath('/'); return
|
||||||
|
}
|
||||||
setErr(msg); setEntries([])
|
setErr(msg); setEntries([])
|
||||||
}
|
}
|
||||||
|
sessionStorage.removeItem('hamlab.files.lost')
|
||||||
}, [path, sudo])
|
}, [path, sudo])
|
||||||
|
|
||||||
useEffect(() => { setEntries(null); setPathInput(path); setMenu(null); load(path, sudo); sessionStorage.setItem('hamlab.files.sudo', sudo ? '1' : '0') }, [path, sudo, load])
|
useEffect(() => { setEntries(null); setPathInput(path); setMenu(null); load(path, sudo); sessionStorage.setItem('hamlab.files.sudo', sudo ? '1' : '0') }, [path, sudo, load])
|
||||||
useEffect(() => { const h = () => setMenu(null); window.addEventListener('click', h); return () => window.removeEventListener('click', h) }, [])
|
useEffect(() => { const h = () => setMenu(null); window.addEventListener('click', h); return () => window.removeEventListener('click', h) }, [])
|
||||||
useEscape(() => setMenu(null), menu !== null)
|
useEscape(() => setMenu(null), menu !== null)
|
||||||
|
|
||||||
const go = (p: string) => { setOpen(null); setPath(p || '/') }
|
const go = (p: string) => { setOpen(null); setPath(resolvePath(p, path)) }
|
||||||
const crumbs = path === '/' ? [] : path.slice(1).split('/')
|
const crumbs = path === '/' ? [] : path.slice(1).split('/')
|
||||||
|
|
||||||
const shown = (entries ?? []).filter((e) => (hidden || !e.name.startsWith('.')) && (!q || e.name.toLowerCase().includes(q.toLowerCase())))
|
const shown = (entries ?? []).filter((e) => (hidden || !e.name.startsWith('.')) && (!q || e.name.toLowerCase().includes(q.toLowerCase())))
|
||||||
@@ -150,12 +349,10 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
|||||||
const act = async (label: string, fn: () => Promise<unknown>): Promise<boolean> => {
|
const act = async (label: string, fn: () => Promise<unknown>): Promise<boolean> => {
|
||||||
try { await fn(); toast(label); await load(); return true } catch (e) { toast(`${label} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); return false }
|
try { await fn(); toast(label); await load(); return true } catch (e) { toast(`${label} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); return false }
|
||||||
}
|
}
|
||||||
// A name is one path segment. ("../x" and "a/b" would quietly act somewhere else.)
|
|
||||||
const okName = (n?: string | null): n is string => { const t = n?.trim(); if (!t) return false; if (t.includes('/') || t === '.' || t === '..') { toast('a name can\'t contain / or be . or .. — use Move… to put it elsewhere', 'err'); return false } return true }
|
|
||||||
const mkdir = async () => { const n = await askText('New folder name'); if (okName(n)) act(`created ${n.trim()}`, () => api.fileMkdir(join(path, n.trim()), sudo)) }
|
const mkdir = async () => { const n = await askText('New folder name'); if (okName(n)) act(`created ${n.trim()}`, () => api.fileMkdir(join(path, n.trim()), sudo)) }
|
||||||
const touch = async () => { const n = await askText('New file name'); if (okName(n) && await act(`created ${n.trim()}`, () => api.fileTouch(join(path, n.trim()), sudo))) setOpen(join(path, n.trim())) } // opened only once it exists
|
const touch = async () => { const n = await askText('New file name'); if (okName(n) && await act(`created ${n.trim()}`, () => api.fileTouch(join(path, n.trim()), sudo))) setOpen(join(path, n.trim())) } // opened only once it exists
|
||||||
const rename = async (e: FileEntry) => { const n = await askText('Rename to', e.name); if (okName(n) && n.trim() !== e.name) act(`renamed to ${n.trim()}`, () => api.fileRename(join(path, e.name), join(path, n.trim()), sudo)) }
|
const rename = async (e: FileEntry) => { const n = await askText('Rename to', e.name); if (okName(n) && n.trim() !== e.name) act(`renamed to ${n.trim()}`, () => api.fileRename(join(path, e.name), join(path, n.trim()), sudo)) }
|
||||||
const move = async (e: FileEntry) => { const n = await askText('Move to (absolute path)', join(path, e.name)); if (n?.trim() && n.trim() !== join(path, e.name)) act(`moved to ${n.trim()}`, () => api.fileRename(join(path, e.name), n.trim(), sudo)) }
|
const move = (e: FileEntry) => setMoving(e)
|
||||||
const chmod = async (e: FileEntry) => { const n = await askText(`Permissions for ${e.name} (octal)`, e.perm); if (n?.trim() && n.trim() !== e.perm) act(`chmod ${n.trim()} ${e.name}`, () => api.fileChmod(join(path, e.name), n.trim(), sudo)) }
|
const chmod = async (e: FileEntry) => { const n = await askText(`Permissions for ${e.name} (octal)`, e.perm); if (n?.trim() && n.trim() !== e.perm) act(`chmod ${n.trim()} ${e.name}`, () => api.fileChmod(join(path, e.name), n.trim(), sudo)) }
|
||||||
const del = async (e: FileEntry) => {
|
const del = async (e: FileEntry) => {
|
||||||
const full = join(path, e.name)
|
const full = join(path, e.name)
|
||||||
@@ -171,6 +368,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
|||||||
// ---- upload ----
|
// ---- upload ----
|
||||||
const uploadFiles = async (files: FileList | File[]) => {
|
const uploadFiles = async (files: FileList | File[]) => {
|
||||||
const list = Array.from(files)
|
const list = Array.from(files)
|
||||||
|
let done = 0
|
||||||
for (let i = 0; i < list.length; i++) {
|
for (let i = 0; i < list.length; i++) {
|
||||||
const f = list[i]
|
const f = list[i]
|
||||||
let overwrite = false
|
let overwrite = false
|
||||||
@@ -181,13 +379,22 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
|||||||
setUpload({ name: f.name, pct: 0, n: i + 1, of: list.length })
|
setUpload({ name: f.name, pct: 0, n: i + 1, of: list.length })
|
||||||
try {
|
try {
|
||||||
await api.fileUpload(path, f, sudo, (pct) => setUpload({ name: f.name, pct, n: i + 1, of: list.length }), overwrite)
|
await api.fileUpload(path, f, sudo, (pct) => setUpload({ name: f.name, pct, n: i + 1, of: list.length }), overwrite)
|
||||||
|
done++
|
||||||
} catch (e) { toast(`upload ${f.name} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); break }
|
} catch (e) { toast(`upload ${f.name} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); break }
|
||||||
}
|
}
|
||||||
setUpload(null)
|
setUpload(null)
|
||||||
toast(list.length === 1 ? `uploaded ${list[0].name}` : `uploaded ${list.length} files`)
|
if (done === 1) toast(`uploaded ${list.length === 1 ? list[0].name : '1 file'}`)
|
||||||
load()
|
else if (done > 1) toast(`uploaded ${done} files`)
|
||||||
|
if (done > 0) load()
|
||||||
|
}
|
||||||
|
// A dropped folder arrives as a nameless 0-byte "file"; the browser gives no way to read it here.
|
||||||
|
const onDrop = (e: DragEvent) => {
|
||||||
|
e.preventDefault(); setDrag(false)
|
||||||
|
const items = Array.from(e.dataTransfer.items ?? [])
|
||||||
|
const files = Array.from(e.dataTransfer.files).filter((_, i) => !(items[i]?.webkitGetAsEntry?.()?.isDirectory))
|
||||||
|
if (files.length < e.dataTransfer.files.length) toast('folders can\'t be dropped — drop the files inside, or upload a zip', 'err')
|
||||||
|
if (files.length) uploadFiles(files)
|
||||||
}
|
}
|
||||||
const onDrop = (e: DragEvent) => { e.preventDefault(); setDrag(false); if (e.dataTransfer.files.length) uploadFiles(e.dataTransfer.files) }
|
|
||||||
|
|
||||||
if (open) return <div className="files"><Viewer path={open} sudo={sudo} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
|
if (open) return <div className="files"><Viewer path={open} sudo={sudo} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
|
||||||
|
|
||||||
@@ -202,6 +409,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
|||||||
<div className="files-tools">
|
<div className="files-tools">
|
||||||
<button className={'btn' + (sudo ? ' on' : '')} title="browse as root" onClick={() => setSudo(!sudo)}>{sudo ? 'root' : 'sudo'}</button>
|
<button className={'btn' + (sudo ? ' on' : '')} title="browse as root" onClick={() => setSudo(!sudo)}>{sudo ? 'root' : 'sudo'}</button>
|
||||||
<button className={'btn' + (hidden ? ' on' : '')} title="show hidden files" onClick={() => setHidden(!hidden)}>.hidden</button>
|
<button className={'btn' + (hidden ? ' on' : '')} title="show hidden files" onClick={() => setHidden(!hidden)}>.hidden</button>
|
||||||
|
<button className={'btn' + (previews ? ' on' : '')} title={'thumbnails for images and videos' + (thumbs ? ` — ${thumbs.summary}` + (thumbs.cache.entries ? ` · ${thumbs.cache.entries} kept (${fmtBytes(thumbs.cache.bytes)})` : '') : '')} onClick={() => setPreviews(!previews)}>previews</button>
|
||||||
<select className="btn" value={sort} onChange={(e) => setSort(e.target.value as Sort)}><option value="name">name</option><option value="size">size</option><option value="mtime">modified</option></select>
|
<select className="btn" value={sort} onChange={(e) => setSort(e.target.value as Sort)}><option value="name">name</option><option value="size">size</option><option value="mtime">modified</option></select>
|
||||||
<input className="filter" placeholder="filter" value={q} onChange={(e) => setQ(e.target.value)} />
|
<input className="filter" placeholder="filter" value={q} onChange={(e) => setQ(e.target.value)} />
|
||||||
<button className="btn" onClick={mkdir}>+ folder</button>
|
<button className="btn" onClick={mkdir}>+ folder</button>
|
||||||
@@ -213,10 +421,13 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{previews && <ThumbNote st={thumbs} sudo={sudo} onChanged={() => readThumbs(true)} />}
|
||||||
|
{moveTask && <TaskOutput id={moveTask.id} onDone={(t) => { load(); if (t.exit_code === 0 && !t.error) toast(`${moveTask.title}: done`); else toast(`${moveTask.title}: failed`, 'err') }} />}
|
||||||
|
{moveTask && <div className="thumb-note"><button className="btn" onClick={() => setMoveTask(null)}>Hide output</button></div>}
|
||||||
{upload && <div className="upload-bar"><Spinner /> {upload.of > 1 ? `${upload.n}/${upload.of} · ` : ''}{upload.name} · {upload.pct}%<div className="meter"><div className="fill" style={{ width: `${upload.pct}%` }} /></div></div>}
|
{upload && <div className="upload-bar"><Spinner /> {upload.of > 1 ? `${upload.n}/${upload.of} · ` : ''}{upload.name} · {upload.pct}%<div className="meter"><div className="fill" style={{ width: `${upload.pct}%` }} /></div></div>}
|
||||||
{err && <div className="error" style={{ padding: '6px 12px' }}>{err}{/permission denied/i.test(err) && !sudo && <> · <button className="link" onClick={() => setSudo(true)}>retry as root</button></>}</div>}
|
{err && <div className="error" style={{ padding: '6px 12px' }}>{err}{/permission denied/i.test(err) && !sudo && <> · <button className="link" onClick={() => setSudo(true)}>retry as root</button></>}</div>}
|
||||||
|
|
||||||
<div className="file-list">
|
<div className={'file-list' + (previews ? ' thumbs' : '')} ref={setListEl}>
|
||||||
{entries === null && <Loading label="Listing…" />}
|
{entries === null && <Loading label="Listing…" />}
|
||||||
{path !== '/' && entries !== null && (
|
{path !== '/' && entries !== null && (
|
||||||
<div className="file-row up" onClick={() => go(path.slice(0, path.lastIndexOf('/')) || '/')}><span className="ficon">↩</span><span className="fname">..</span></div>
|
<div className="file-row up" onClick={() => go(path.slice(0, path.lastIndexOf('/')) || '/')}><span className="ficon">↩</span><span className="fname">..</span></div>
|
||||||
@@ -226,7 +437,9 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
|||||||
const isDir = e.type === 'dir'
|
const isDir = e.type === 'dir'
|
||||||
return (
|
return (
|
||||||
<div key={e.name} className={'file-row' + (e.broken ? ' broken' : '')} onClick={() => (isDir ? go(full) : setOpen(full))}>
|
<div key={e.name} className={'file-row' + (e.broken ? ' broken' : '')} onClick={() => (isDir ? go(full) : setOpen(full))}>
|
||||||
<span className="ficon">{icon(e)}</span>
|
{previews && e.preview && thumbs?.ok
|
||||||
|
? <Thumb url={api.fileThumbURL(full, sudo, e.mtime, e.size)} kind={e.preview} near={near} fallback={icon(e)} />
|
||||||
|
: <span className="ficon">{icon(e)}</span>}
|
||||||
<span className="fname" title={e.target ? `→ ${e.target}` : e.name}>{e.name}{e.target && <span className="muted"> → {e.target}</span>}</span>
|
<span className="fname" title={e.target ? `→ ${e.target}` : e.name}>{e.name}{e.target && <span className="muted"> → {e.target}</span>}</span>
|
||||||
<span className="fmeta num">{isDir ? '' : fmtBytes(e.size)}</span>
|
<span className="fmeta num">{isDir ? '' : fmtBytes(e.size)}</span>
|
||||||
<span className="fmeta num">{fmtTime(e.mtime)}</span>
|
<span className="fmeta num">{fmtTime(e.mtime)}</span>
|
||||||
@@ -249,6 +462,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
|
|||||||
{entries !== null && shown.length === 0 && !err && <div className="muted" style={{ padding: 12 }}>{q ? 'Nothing matches.' : 'Empty folder.'}{!hidden && entries.some((e) => e.name.startsWith('.')) ? ' (hidden files not shown)' : ''}</div>}
|
{entries !== null && shown.length === 0 && !err && <div className="muted" style={{ padding: 12 }}>{q ? 'Nothing matches.' : 'Empty folder.'}{!hidden && entries.some((e) => e.name.startsWith('.')) ? ' (hidden files not shown)' : ''}</div>}
|
||||||
</div>
|
</div>
|
||||||
{drag && <div className="drop-hint">Drop to upload into {path}</div>}
|
{drag && <div className="drop-hint">Drop to upload into {path}</div>}
|
||||||
|
{moving && <MovePicker item={moving} from={path} sudo={sudo} hidden={hidden} onClose={() => setMoving(null)} onMoved={(t) => { setMoving(null); if (t) setMoveTask(t); load() }} />}
|
||||||
</div>
|
</div>
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,147 @@
|
|||||||
|
// A video in the Files viewer. First the file as it is, handed to the browser's own
|
||||||
|
// decoder; when that fails — Safari and an MKV, HEVC without a hardware decoder — a
|
||||||
|
// picker: codec and size, and the machine re-encodes it to HLS on its own GPU (or CPU),
|
||||||
|
// which plays natively on Safari and through hls.js everywhere else. The same picker is
|
||||||
|
// one tap away even when the file plays, for a 36 Mbps capture over a slow link.
|
||||||
|
import { useCallback, useEffect, useRef, useState } from 'react'
|
||||||
|
import { api, apiPath, PlayCaps, PlayStatus, PlayStream } from './api'
|
||||||
|
import { Btn, Spinner } from './ui'
|
||||||
|
|
||||||
|
type Mode = 'native' | 'pick' | 'hls'
|
||||||
|
const CODEC_KEY = 'hamlab.play.codec'
|
||||||
|
const QUALITY_KEY = 'hamlab.play.quality'
|
||||||
|
const remember = (k: string, v: string) => { try { localStorage.setItem(k, v) } catch { /* private mode */ } }
|
||||||
|
const recall = (k: string, d: string) => { try { return localStorage.getItem(k) || d } catch { return d } }
|
||||||
|
|
||||||
|
const fmtMbps = (bps?: number) => (bps ? `${(bps / 1e6).toFixed(bps >= 10e6 ? 0 : 1)} Mbps` : '')
|
||||||
|
const describe = (s: PlayStream) => {
|
||||||
|
const v = s.source.video
|
||||||
|
const src = v ? `${v.codec}${v.profile ? ' ' + v.profile : ''} ${v.width}×${v.height}${v.fps ? ` ${Math.round(v.fps)} fps` : ''}${s.source.bit_rate ? ` ${fmtMbps(s.source.bit_rate)}` : ''} in ${s.source.container.split(',')[0]}` : s.source.container
|
||||||
|
const out = s.codec === 'copy' ? 'repackaged' : `${s.codec === 'h264' ? 'H.264' : s.codec.toUpperCase()} ${s.width}×${s.height} ${(s.kbps / 1000).toFixed(s.kbps >= 10000 ? 0 : 1)} Mbps`
|
||||||
|
return `${src} → ${out} on ${s.engine}`
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function Player({ path, sudo, name }: { path: string; sudo: boolean; name: string }) {
|
||||||
|
const [mode, setMode] = useState<Mode>('native')
|
||||||
|
const [why, setWhy] = useState('') // what brought the picker up
|
||||||
|
const [caps, setCaps] = useState<PlayCaps | null>(null)
|
||||||
|
const [codec, setCodec] = useState(() => recall(CODEC_KEY, 'h264'))
|
||||||
|
const [quality, setQuality] = useState(() => recall(QUALITY_KEY, '1080p'))
|
||||||
|
const [stream, setStream] = useState<{ s: PlayStream; url: string } | null>(null)
|
||||||
|
const [status, setStatus] = useState<PlayStatus | null>(null)
|
||||||
|
const [err, setErr] = useState('')
|
||||||
|
const video = useRef<HTMLVideoElement>(null)
|
||||||
|
const streamRef = useRef(stream)
|
||||||
|
streamRef.current = stream
|
||||||
|
|
||||||
|
useEffect(() => { setMode('native'); setWhy(''); setStream(null); setStatus(null); setErr('') }, [path, sudo])
|
||||||
|
useEffect(() => { remember(CODEC_KEY, codec) }, [codec])
|
||||||
|
useEffect(() => { remember(QUALITY_KEY, quality) }, [quality])
|
||||||
|
useEffect(() => { if (mode === 'pick' && !caps) api.playCaps(sudo).then(setCaps).catch((e) => setCaps({ codecs: [], qualities: [], error: e instanceof Error ? e.message : 'failed' })) }, [mode, caps, sudo])
|
||||||
|
|
||||||
|
// a stream is the machine's work: let go of it when this player does — and when the page
|
||||||
|
// itself goes, by a beacon, the one request a browser still sends on its way out
|
||||||
|
const stop = useCallback(() => { const s = streamRef.current; if (s) { api.playStop(s.s.id).catch(() => { /* gone already */ }); setStream(null); setStatus(null) } }, [])
|
||||||
|
useEffect(() => () => stop(), [stop])
|
||||||
|
useEffect(() => {
|
||||||
|
const h = () => { const s = streamRef.current; if (s) navigator.sendBeacon(apiPath(`/api/files/play/${s.s.id}/close`)) }
|
||||||
|
window.addEventListener('pagehide', h)
|
||||||
|
return () => window.removeEventListener('pagehide', h)
|
||||||
|
}, [])
|
||||||
|
|
||||||
|
const start = async () => {
|
||||||
|
setErr('')
|
||||||
|
try {
|
||||||
|
stop()
|
||||||
|
const r = await api.playStart(path, codec, quality, sudo)
|
||||||
|
setStream({ s: r.stream, url: r.url }); setMode('hls')
|
||||||
|
} catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
|
||||||
|
}
|
||||||
|
|
||||||
|
// feed the stream to the element: Safari plays HLS itself, the rest through hls.js
|
||||||
|
useEffect(() => {
|
||||||
|
const v = video.current
|
||||||
|
if (mode !== 'hls' || !stream || !v) return
|
||||||
|
const url = apiPath(stream.url)
|
||||||
|
let hls: { destroy: () => void } | null = null
|
||||||
|
let gone = false
|
||||||
|
;(async () => {
|
||||||
|
if (v.canPlayType('application/vnd.apple.mpegurl')) {
|
||||||
|
v.src = url
|
||||||
|
} else {
|
||||||
|
const { default: Hls } = await import('hls.js')
|
||||||
|
if (gone) return
|
||||||
|
if (!Hls.isSupported()) { setErr('this browser has neither HLS nor MediaSource — nothing can play here'); return }
|
||||||
|
const h = new Hls({ maxBufferLength: 60, backBufferLength: 30 })
|
||||||
|
hls = h
|
||||||
|
h.on(Hls.Events.ERROR, (_e, d) => { if (d.fatal) setErr(`playback failed: ${d.details}`) })
|
||||||
|
h.loadSource(url)
|
||||||
|
h.attachMedia(v)
|
||||||
|
}
|
||||||
|
v.play().catch(() => { /* needs a tap on some phones */ })
|
||||||
|
})()
|
||||||
|
return () => { gone = true; hls?.destroy() }
|
||||||
|
}, [mode, stream])
|
||||||
|
|
||||||
|
// follow the re-encode while it runs
|
||||||
|
useEffect(() => {
|
||||||
|
if (mode !== 'hls' || !stream) return
|
||||||
|
let stopPolling = false
|
||||||
|
const tick = async () => { try { const st = await api.playStatus(stream.s.id); if (!stopPolling) { setStatus(st); if (st.running) setTimeout(tick, 2000) } } catch { /* ended */ } }
|
||||||
|
tick()
|
||||||
|
return () => { stopPolling = true }
|
||||||
|
}, [mode, stream])
|
||||||
|
|
||||||
|
const pick = (reason: string) => { setWhy(reason); setMode('pick') }
|
||||||
|
const codecs = caps?.codecs ?? []
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="player">
|
||||||
|
{mode === 'native' && (
|
||||||
|
<>
|
||||||
|
<div className="play-bar"><span>as it is</span><span className="spacer" /><button className="btn" onClick={() => pick('')}>Re-encode…</button></div>
|
||||||
|
<div className="preview"><video controls playsInline preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => pick(`This browser can't play ${name} as it is — the container or codec isn't one it decodes.`)} /></div>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
{mode === 'pick' && (
|
||||||
|
<div className="play-pick notice">
|
||||||
|
<div>{why || `Re-encode ${name} on this machine and play that instead.`}</div>
|
||||||
|
{caps?.error && <div className="error" style={{ marginTop: 8 }}>{caps.error}</div>}
|
||||||
|
{!caps && !err && <div className="muted small" style={{ marginTop: 8 }}><Spinner /> asking the machine what it can encode…</div>}
|
||||||
|
{caps && !caps.error && (
|
||||||
|
<div className="row">
|
||||||
|
<label>codec
|
||||||
|
<select value={codec} onChange={(e) => setCodec(e.target.value)}>
|
||||||
|
{codecs.map((c) => <option key={c.id} value={c.id} disabled={!c.engine}>{c.label}{c.engine ? ` — ${c.engine}` : ' — not on this machine'}</option>)}
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>size
|
||||||
|
<select value={quality} onChange={(e) => setQuality(e.target.value)} disabled={codec === 'copy'}>
|
||||||
|
{caps.qualities.map((q) => <option key={q.id} value={q.id}>{q.label}</option>)}
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<Btn className="btn on" disabled={!codecs.find((c) => c.id === codec)?.engine} onClick={start}>Play</Btn>
|
||||||
|
<a className="btn" href={api.fileDownloadURL(path, sudo)} download={name}>Download instead</a>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{err && <div className="error" style={{ marginTop: 8 }}>{err}</div>}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{mode === 'hls' && stream && (
|
||||||
|
<>
|
||||||
|
<div className="play-bar">
|
||||||
|
<span title={describe(stream.s)}>{describe(stream.s)}</span>
|
||||||
|
{status && status.running && <span>· {status.speed ? `${status.speed.toFixed(1)}×` : ''} {stream.s.source.duration ? `${Math.min(100, Math.round(status.progress / stream.s.source.duration * 100))}% through` : `${Math.round(status.progress)} s done`} — seek within what's done</span>}
|
||||||
|
{status?.done && <span>· all of it is ready</span>}
|
||||||
|
{status?.error && <span className="error">· stopped early: {status.error}</span>}
|
||||||
|
<span className="spacer" />
|
||||||
|
<button className="btn" onClick={() => { stop(); setMode('pick'); setWhy('') }}>Change…</button>
|
||||||
|
</div>
|
||||||
|
<div className="preview"><video ref={video} controls autoPlay playsInline /></div>
|
||||||
|
{err && <div className="error" style={{ margin: 12 }}>{err}</div>}
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
@@ -101,7 +101,7 @@ export default function StackNew({ mode, from, networks = [], taken, onClose, on
|
|||||||
{homes.length > 1 && <label className="muted">Where
|
{homes.length > 1 && <label className="muted">Where
|
||||||
<select value={home} onChange={(e) => setHome(e.target.value)}>{homes.map((h) => <option key={h.dir + h.via} value={h.dir}>{sameTail(h) ? h.dir : tail(h.dir)} · {h.stacks} stack{h.stacks === 1 ? '' : 's'}</option>)}</select>
|
<select value={home} onChange={(e) => setHome(e.target.value)}>{homes.map((h) => <option key={h.dir + h.via} value={h.dir}>{sameTail(h) ? h.dir : tail(h.dir)} · {h.stacks} stack{h.stacks === 1 ? '' : 's'}</option>)}</select>
|
||||||
</label>}
|
</label>}
|
||||||
{picked && <div className="muted small mono" style={{ overflowWrap: 'anywhere' }}>{picked.dir}/{name || '…'}/compose.yaml{picked.held_by && ` — kept on ${picked.held_by}, deployed here over ${picked.remote}`}{picked.via && ` — compose runs inside ${picked.via}, like its neighbours`}</div>}
|
{picked && <div className="muted small mono" style={{ overflowWrap: 'anywhere' }}>{picked.dir}/{name || '…'}/compose.yaml{picked.via && ` — compose runs inside ${picked.via}, like its neighbours`}</div>}
|
||||||
{!from && <div className="tabs"><button className={how === 'form' ? 'on' : ''} onClick={() => setHow('form')}>Form</button><button className={how === 'compose' ? 'on' : ''} onClick={toCompose}>Compose</button></div>}
|
{!from && <div className="tabs"><button className={how === 'form' ? 'on' : ''} onClick={() => setHow('form')}>Form</button><button className={how === 'compose' ? 'on' : ''} onClick={toCompose}>Compose</button></div>}
|
||||||
{how === 'form' ? <>
|
{how === 'form' ? <>
|
||||||
<label className="muted">Image<input className="mono" value={form.image} onChange={set('image')} placeholder="traefik/whoami:latest" spellCheck={false} autoCapitalize="off" autoCorrect="off" /></label>
|
<label className="muted">Image<input className="mono" value={form.image} onChange={set('image')} placeholder="traefik/whoami:latest" spellCheck={false} autoCapitalize="off" autoCorrect="off" /></label>
|
||||||
|
|||||||
+2
-1
@@ -369,7 +369,8 @@ export default function Top() {
|
|||||||
<div className="t-up">up {upfor(uptime)}</div>
|
<div className="t-up">up {upfor(uptime)}</div>
|
||||||
</div>
|
</div>
|
||||||
<div className="t-cores">
|
<div className="t-cores">
|
||||||
<div className="t-cores-head"><b>{model || 'cpu'}</b><span className="spacer" />{pt?.cpu_mhz ? `${Math.round(pt.cpu_mhz)} MHz` : ''}</div>
|
<div className="t-cores-head"><b>{model || 'cpu'}</b><span className="spacer" />{pt?.cpu_mhz ? `${Math.round(pt.cpu_mhz)} MHz` : ''}
|
||||||
|
{pt?.cpu_power_w ? <span title="the CPU's own power reading: cores, built-in graphics and memory controller, not the whole machine">{pt.cpu_mhz ? ' · ' : ''}{pt.cpu_power_w < 10 ? pt.cpu_power_w.toFixed(1) : Math.round(pt.cpu_power_w)} W</span> : null}</div>
|
||||||
<div className="t-core">
|
<div className="t-core">
|
||||||
<span className="t-core-n">CPU</span>
|
<span className="t-core-n">CPU</span>
|
||||||
<Meter pct={cpu} grad="cpu" />
|
<Meter pct={cpu} grad="cpu" />
|
||||||
|
|||||||
@@ -4,7 +4,9 @@
|
|||||||
import { useEffect, useRef, useState } from 'react'
|
import { useEffect, useRef, useState } from 'react'
|
||||||
import { wsURLPath } from './api'
|
import { wsURLPath } from './api'
|
||||||
import { Btn, useEscape } from './ui'
|
import { Btn, useEscape } from './ui'
|
||||||
import './vendor/spice-html5/spice.css'
|
// No spice.css: upstream's stylesheet is for its own demo page and sets body, *, button and
|
||||||
|
// canvas for the whole document. Imported here it restyled every button in Ham Lab. The
|
||||||
|
// console is styled by the .vga rules in styles.css.
|
||||||
|
|
||||||
type Spice = typeof import('./vendor/spice-html5/src/main.js')
|
type Spice = typeof import('./vendor/spice-html5/src/main.js')
|
||||||
|
|
||||||
|
|||||||
+41
-11
@@ -81,7 +81,7 @@ export default function Virtualization() {
|
|||||||
{tab === 'images' && <Images st={st} images={images} instances={instances} onTask={startTask} onNew={(image, vm) => { setTab('instances'); setMaking({ image, vm }) }} onChanged={load} />}
|
{tab === 'images' && <Images st={st} images={images} instances={instances} onTask={startTask} onNew={(image, vm) => { setTab('instances'); setMaking({ image, vm }) }} onChanged={load} />}
|
||||||
{tab === 'library' && <Library st={st} ov={ov} onTask={startTask} onChanged={load} onNewVM={(iso) => setFromIso(iso)} />}
|
{tab === 'library' && <Library st={st} ov={ov} onTask={startTask} onChanged={load} onNewVM={(iso) => setFromIso(iso)} />}
|
||||||
</div>
|
</div>
|
||||||
{tab === 'instances' && (cur ? <Detail i={cur} st={st} onBack={() => setSel('')} onChanged={load} onTask={startTask} /> : <Summary ov={ov} />)}
|
{tab === 'instances' && (cur ? <Detail i={cur} st={st} onBack={() => setSel('')} onChanged={load} onTask={startTask} /> : <Summary ov={ov} onChanged={load} />)}
|
||||||
{making && <NewInstance st={st} ov={ov} preset={making} onClose={() => setMaking(null)} onStarted={(t) => { setMaking(null); startTask(t) }} />}
|
{making && <NewInstance st={st} ov={ov} preset={making} onClose={() => setMaking(null)} onStarted={(t) => { setMaking(null); startTask(t) }} />}
|
||||||
{fromIso !== null && <NewVMFromISO st={st} ov={ov} iso={fromIso} onClose={() => setFromIso(null)} onStarted={(t) => { setFromIso(null); startTask(t); setTab('instances') }} />}
|
{fromIso !== null && <NewVMFromISO st={st} ov={ov} iso={fromIso} onClose={() => setFromIso(null)} onStarted={(t) => { setFromIso(null); startTask(t); setTab('instances') }} />}
|
||||||
</div>
|
</div>
|
||||||
@@ -151,11 +151,11 @@ function InstallDialog({ st, onClose, onStarted }: { st: VirtStatus; onClose: ()
|
|||||||
<div className="notice">Everything goes under <span className="mono">{st.dir}</span>: the pool's backing file, ISOs, images, exports. Where is not a choice; how big is.</div>
|
<div className="notice">Everything goes under <span className="mono">{st.dir}</span>: the pool's backing file, ISOs, images, exports. Where is not a choice; how big is.</div>
|
||||||
<label className="muted">Pool driver
|
<label className="muted">Pool driver
|
||||||
<select value={driver} onChange={(e) => setDriver(e.target.value)}>
|
<select value={driver} onChange={(e) => setDriver(e.target.value)}>
|
||||||
{st.drivers.map((d) => <option key={d} value={d}>{d}{d === 'zfs' ? ' — snapshots and clones are instant' : d === 'btrfs' ? ' — snapshots, on a loop file' : ' — plain files, no snapshots to speak of'}</option>)}
|
{st.drivers.map((d) => <option key={d} value={d}>{d === 'dir' ? 'plain files on this disk — each VM\u2019s disk is its own file, sized on its own; snapshots are copies' : d === 'zfs' ? 'zfs — one pre-sized file holding every disk; snapshots and clones are instant' : 'btrfs — one pre-sized file holding every disk; snapshots are instant'}</option>)}
|
||||||
</select>
|
</select>
|
||||||
</label>
|
</label>
|
||||||
{driver !== 'dir' && (
|
{driver !== 'dir' && (
|
||||||
<label className="muted">Pool size, GiB — a sparse file, grows as used · {fmtBytes(st.disk_free)} free
|
<label className="muted">Ceiling of that file, GiB — sparse, it grows only as disks fill it · {fmtBytes(st.disk_free)} free
|
||||||
<input type="number" min={5} max={Math.floor(st.disk_free / GiB)} value={size} onChange={(e) => setSize(e.target.value)} required />
|
<input type="number" min={5} max={Math.floor(st.disk_free / GiB)} value={size} onChange={(e) => setSize(e.target.value)} required />
|
||||||
</label>
|
</label>
|
||||||
)}
|
)}
|
||||||
@@ -187,7 +187,17 @@ function InstallDialog({ st, onClose, onStarted }: { st: VirtStatus; onClose: ()
|
|||||||
|
|
||||||
// ---- the right-hand pane when nothing is selected: what this machine's Incus is ----
|
// ---- the right-hand pane when nothing is selected: what this machine's Incus is ----
|
||||||
|
|
||||||
function Summary({ ov }: { ov: VirtOverview }) {
|
// A location is a plain-files storage pool at a folder of his choosing.
|
||||||
|
async function addLocation(onDone: (name: string) => void) {
|
||||||
|
const path = await askText('Folder on this machine for instance disks', '/', 'Next')
|
||||||
|
if (!path) return
|
||||||
|
const base = path.split('/').filter(Boolean).pop() ?? 'disks'
|
||||||
|
const name = await askText(`A name for ${path}`, base.toLowerCase().replace(/[^a-z0-9-]+/g, '-').replace(/^-|-$/g, ''), 'Add')
|
||||||
|
if (!name) return
|
||||||
|
try { const r = await api.virtPoolCreate(name, path); toast(`${r.name} added at ${r.path}`); onDone(r.name) } catch (e) { toast(errMsg(e), 'err') }
|
||||||
|
}
|
||||||
|
|
||||||
|
function Summary({ ov, onChanged }: { ov: VirtOverview; onChanged: () => void }) {
|
||||||
const st = ov.status
|
const st = ov.status
|
||||||
return (
|
return (
|
||||||
<div className="svc-detail">
|
<div className="svc-detail">
|
||||||
@@ -198,9 +208,11 @@ function Summary({ ov }: { ov: VirtOverview }) {
|
|||||||
{st.layout && <div><span className="k">Installed</span><span>{when(st.layout.attempted)} by Ham Lab {st.layout.hamlab_version ?? ''}</span></div>}
|
{st.layout && <div><span className="k">Installed</span><span>{when(st.layout.attempted)} by Ham Lab {st.layout.hamlab_version ?? ''}</span></div>}
|
||||||
{!st.layout && <div><span className="k">Installed</span><span>outside Ham Lab — pools and networks below are as found</span></div>}
|
{!st.layout && <div><span className="k">Installed</span><span>outside Ham Lab — pools and networks below are as found</span></div>}
|
||||||
</div>
|
</div>
|
||||||
<h3 className="stack-head">Storage pools</h3>
|
<h3 className="stack-head">Disk locations <span className="muted">· where an instance\u2019s disk can go</span></h3>
|
||||||
<table className="virt-table"><thead><tr><th>Pool</th><th>Driver</th><th>Source</th><th className="num">Used</th><th className="num">Of</th><th className="num">Users</th></tr></thead>
|
<table className="virt-table"><thead><tr><th>Location</th><th>Where</th><th className="num">Used</th><th className="num">Free</th><th className="num">Disks</th><th /></tr></thead>
|
||||||
<tbody>{(ov.pools ?? []).map((p) => <tr key={p.name}><td>{p.name}</td><td>{p.driver}</td><td className="mono small">{p.source}</td><td className="num">{p.total ? fmtBytes(p.used) : '—'}</td><td className="num">{p.total ? fmtBytes(p.total) : '—'}</td><td className="num">{p.used_by}</td></tr>)}</tbody></table>
|
<tbody>{(ov.pools ?? []).map((p) => <tr key={p.name}><td>{p.name}{p.name === ov.default_pool && <span className="badge">default</span>}</td><td>{p.where}<div className="muted small mono">{p.driver} · {p.source}</div></td><td className="num">{p.total ? fmtBytes(p.used) : '—'}</td><td className="num">{p.total ? fmtBytes(p.total - p.used) : '—'}</td><td className="num">{p.used_by}</td>
|
||||||
|
<td className="num">{p.used_by === 0 && p.name !== ov.default_pool && <Btn className="btn danger" confirm={`Remove the location ${p.name}? Nothing uses it; the folder itself is left alone.`} onClick={async () => { try { await api.virtPoolDelete(p.name); toast('removed'); onChanged() } catch (e) { toast(errMsg(e), 'err') } }}>Remove</Btn>}</td></tr>)}</tbody></table>
|
||||||
|
<div className="svc-actions"><Btn className="btn" onClick={() => addLocation(onChanged)}>+ Add a location…</Btn><span className="muted small">a folder on this machine; VM disks become files in it, containers folders</span></div>
|
||||||
<h3 className="stack-head">Networks</h3>
|
<h3 className="stack-head">Networks</h3>
|
||||||
<table className="virt-table"><thead><tr><th>Network</th><th>Type</th><th>Address</th><th>NAT</th><th className="num">Users</th></tr></thead>
|
<table className="virt-table"><thead><tr><th>Network</th><th>Type</th><th>Address</th><th>NAT</th><th className="num">Users</th></tr></thead>
|
||||||
<tbody>{(ov.networks ?? []).filter((n) => n.managed).map((n) => <tr key={n.name}><td>{n.name}</td><td>{n.type}</td><td className="mono small">{n.address ?? '—'}</td><td>{n.nat ? 'yes' : 'no'}</td><td className="num">{n.used_by}</td></tr>)}</tbody></table>
|
<tbody>{(ov.networks ?? []).filter((n) => n.managed).map((n) => <tr key={n.name}><td>{n.name}</td><td>{n.type}</td><td className="mono small">{n.address ?? '—'}</td><td>{n.nat ? 'yes' : 'no'}</td><td className="num">{n.used_by}</td></tr>)}</tbody></table>
|
||||||
@@ -248,7 +260,7 @@ function Detail({ i, st, onBack, onChanged, onTask }: { i: VirtInstance; st: Vir
|
|||||||
{i.description && <div><span className="k">About</span><span>{i.description}</span></div>}
|
{i.description && <div><span className="k">About</span><span>{i.description}</span></div>}
|
||||||
{running && <div><span className="k">Using</span><span>{i.cpu_pct.toFixed(0)}% of a core · {fmtBytes(i.mem_usage)}{i.mem_peak ? ` (peak ${fmtBytes(i.mem_peak)})` : ''}{i.processes > 0 ? ` · ${i.processes} processes` : ''}</span></div>}
|
{running && <div><span className="k">Using</span><span>{i.cpu_pct.toFixed(0)}% of a core · {fmtBytes(i.mem_usage)}{i.mem_peak ? ` (peak ${fmtBytes(i.mem_peak)})` : ''}{i.processes > 0 ? ` · ${i.processes} processes` : ''}</span></div>}
|
||||||
<div><span className="k">Limits</span><span>{i.cpu ? `${i.cpu} CPU` : 'all CPUs'} · {i.memory || 'unlimited memory'}</span></div>
|
<div><span className="k">Limits</span><span>{i.cpu ? `${i.cpu} CPU` : 'all CPUs'} · {i.memory || 'unlimited memory'}</span></div>
|
||||||
{i.disk_total > 0 && <div><span className="k">Disk</span><span>{fmtBytes(i.disk_usage)} of {fmtBytes(i.disk_total)}</span></div>}
|
<div><span className="k">Disk</span><span>{i.disk_total > 0 ? `${fmtBytes(i.disk_usage)} of ${fmtBytes(i.disk_total)}` : i.disk_usage > 0 ? fmtBytes(i.disk_usage) : 'no limit'}{i.devices.root?.pool ? <span className="muted"> · on {i.devices.root.pool}</span> : ''}</span></div>
|
||||||
<div><span className="k">Network</span><span className="mono small">{i.ipv4.length ? i.ipv4.join(' ') : running ? 'no address yet' : '—'}</span></div>
|
<div><span className="k">Network</span><span className="mono small">{i.ipv4.length ? i.ipv4.join(' ') : running ? 'no address yet' : '—'}</span></div>
|
||||||
<div><span className="k">Created</span><span>{when(i.created)}</span></div>
|
<div><span className="k">Created</span><span>{when(i.created)}</span></div>
|
||||||
</div>
|
</div>
|
||||||
@@ -348,6 +360,20 @@ function Snapshots({ i, onChanged }: { i: VirtInstance; onChanged: () => void })
|
|||||||
|
|
||||||
// ---- making one ----
|
// ---- making one ----
|
||||||
|
|
||||||
|
// Where an instance's disk goes. One entry is still a choice shown; a new one can be added here.
|
||||||
|
function PoolPick({ ov, value, onChange }: { ov: VirtOverview; value: string; onChange: (v: string) => void }) {
|
||||||
|
const [extra, setExtra] = useState<{ name: string; where: string }[]>([])
|
||||||
|
const pools = [...(ov.pools ?? []).map((p) => ({ name: p.name, where: p.where + (p.total ? ` · ${fmtBytes(p.total - p.used)} free` : '') })), ...extra.filter((e) => !(ov.pools ?? []).some((p) => p.name === e.name))]
|
||||||
|
return (
|
||||||
|
<label className="muted">Disk location
|
||||||
|
<select value={value} onChange={(e) => { if (e.target.value === '+') addLocation((n) => { setExtra((x) => [...x, { name: n, where: 'just added' }]); onChange(n) }); else onChange(e.target.value) }}>
|
||||||
|
{pools.map((p) => <option key={p.name} value={p.name}>{p.name} — {p.where}</option>)}
|
||||||
|
<option value="+">+ add a location…</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
function NewInstance({ st, ov, preset, onClose, onStarted }: { st: VirtStatus; ov: VirtOverview; preset: { image?: string; vm?: boolean }; onClose: () => void; onStarted: (t: TaskInfo) => void }) {
|
function NewInstance({ st, ov, preset, onClose, onStarted }: { st: VirtStatus; ov: VirtOverview; preset: { image?: string; vm?: boolean }; onClose: () => void; onStarted: (t: TaskInfo) => void }) {
|
||||||
const [vm, setVm] = useState(!!preset.vm && st.kvm)
|
const [vm, setVm] = useState(!!preset.vm && st.kvm)
|
||||||
const [image, setImage] = useState(preset.image ?? '')
|
const [image, setImage] = useState(preset.image ?? '')
|
||||||
@@ -355,6 +381,7 @@ function NewInstance({ st, ov, preset, onClose, onStarted }: { st: VirtStatus; o
|
|||||||
const [cpu, setCpu] = useState('')
|
const [cpu, setCpu] = useState('')
|
||||||
const [mem, setMem] = useState('')
|
const [mem, setMem] = useState('')
|
||||||
const [disk, setDisk] = useState('')
|
const [disk, setDisk] = useState('')
|
||||||
|
const [pool, setPool] = useState(ov.default_pool || ov.pools?.[0]?.name || '')
|
||||||
const nets = (ov.networks ?? []).filter((n) => n.managed || n.type === 'bridge')
|
const nets = (ov.networks ?? []).filter((n) => n.managed || n.type === 'bridge')
|
||||||
const [net, setNet] = useState(nets.find((n) => n.name === 'incusbr0')?.name ?? nets[0]?.name ?? '')
|
const [net, setNet] = useState(nets.find((n) => n.name === 'incusbr0')?.name ?? nets[0]?.name ?? '')
|
||||||
const [cloud, setCloud] = useState('')
|
const [cloud, setCloud] = useState('')
|
||||||
@@ -387,7 +414,7 @@ function NewInstance({ st, ov, preset, onClose, onStarted }: { st: VirtStatus; o
|
|||||||
}, [q, local, remote])
|
}, [q, local, remote])
|
||||||
const submit = async (e: FormEvent) => {
|
const submit = async (e: FormEvent) => {
|
||||||
e.preventDefault(); setErr(''); setBusy(true)
|
e.preventDefault(); setErr(''); setBusy(true)
|
||||||
try { onStarted(await api.virtCreate({ name: name.trim(), image: image.trim(), vm, cpu: cpu.trim(), memory: mem.trim(), disk: disk.trim(), network: net, cloud_init: showCloud ? cloud : '', profiles: [], description: desc.trim(), start })) }
|
try { onStarted(await api.virtCreate({ name: name.trim(), image: image.trim(), vm, cpu: cpu.trim(), memory: mem.trim(), disk: disk.trim(), pool, network: net, cloud_init: showCloud ? cloud : '', profiles: [], description: desc.trim(), start })) }
|
||||||
catch (e) { setErr(errMsg(e)); setBusy(false) }
|
catch (e) { setErr(errMsg(e)); setBusy(false) }
|
||||||
}
|
}
|
||||||
return (
|
return (
|
||||||
@@ -418,8 +445,9 @@ function NewInstance({ st, ov, preset, onClose, onStarted }: { st: VirtStatus; o
|
|||||||
<div className="virt-form-row">
|
<div className="virt-form-row">
|
||||||
<label className="muted">CPUs<input className="mono" value={cpu} onChange={(e) => setCpu(e.target.value)} placeholder={vm ? '2' : 'all'} /></label>
|
<label className="muted">CPUs<input className="mono" value={cpu} onChange={(e) => setCpu(e.target.value)} placeholder={vm ? '2' : 'all'} /></label>
|
||||||
<label className="muted">Memory<input className="mono" value={mem} onChange={(e) => setMem(e.target.value)} placeholder={vm ? '2GiB' : 'unlimited'} /></label>
|
<label className="muted">Memory<input className="mono" value={mem} onChange={(e) => setMem(e.target.value)} placeholder={vm ? '2GiB' : 'unlimited'} /></label>
|
||||||
<label className="muted">Root disk<input className="mono" value={disk} onChange={(e) => setDisk(e.target.value)} placeholder={vm ? '20GiB' : 'pool default'} /></label>
|
<label className="muted">Root disk<input className="mono" value={disk} onChange={(e) => setDisk(e.target.value)} placeholder={vm ? '20GiB' : 'no limit'} /></label>
|
||||||
</div>
|
</div>
|
||||||
|
<PoolPick ov={ov} value={pool} onChange={setPool} />
|
||||||
<div className="muted small">{fmtBytes(st.mem_avail)} free on this machine.{vm ? ' A VM’s memory is a reservation.' : ''}</div>
|
<div className="muted small">{fmtBytes(st.mem_avail)} free on this machine.{vm ? ' A VM’s memory is a reservation.' : ''}</div>
|
||||||
<label className="muted">Description — optional<input value={desc} onChange={(e) => setDesc(e.target.value)} maxLength={200} /></label>
|
<label className="muted">Description — optional<input value={desc} onChange={(e) => setDesc(e.target.value)} maxLength={200} /></label>
|
||||||
<label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
|
<label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
|
||||||
@@ -556,6 +584,7 @@ function NewVMFromISO({ st, ov, iso, onClose, onStarted }: { st: VirtStatus; ov:
|
|||||||
const [cpu, setCpu] = useState('2')
|
const [cpu, setCpu] = useState('2')
|
||||||
const [mem, setMem] = useState('2GiB')
|
const [mem, setMem] = useState('2GiB')
|
||||||
const [disk, setDisk] = useState('20GiB')
|
const [disk, setDisk] = useState('20GiB')
|
||||||
|
const [pool, setPool] = useState(ov.default_pool || ov.pools?.[0]?.name || '')
|
||||||
const nets = (ov.networks ?? []).filter((n) => n.managed || n.type === 'bridge')
|
const nets = (ov.networks ?? []).filter((n) => n.managed || n.type === 'bridge')
|
||||||
const [net, setNet] = useState(nets.find((n) => n.name === 'incusbr0')?.name ?? nets[0]?.name ?? '')
|
const [net, setNet] = useState(nets.find((n) => n.name === 'incusbr0')?.name ?? nets[0]?.name ?? '')
|
||||||
const [sb, setSb] = useState(true)
|
const [sb, setSb] = useState(true)
|
||||||
@@ -564,7 +593,7 @@ function NewVMFromISO({ st, ov, iso, onClose, onStarted }: { st: VirtStatus; ov:
|
|||||||
useEscape(onClose)
|
useEscape(onClose)
|
||||||
const submit = async (e: FormEvent) => {
|
const submit = async (e: FormEvent) => {
|
||||||
e.preventDefault(); setErr(''); setBusy(true)
|
e.preventDefault(); setErr(''); setBusy(true)
|
||||||
try { onStarted(await api.virtFromISO({ name: name.trim(), iso: pick, cpu: cpu.trim(), memory: mem.trim(), disk: disk.trim(), network: net, secureboot: sb, start: true })) }
|
try { onStarted(await api.virtFromISO({ name: name.trim(), iso: pick, cpu: cpu.trim(), memory: mem.trim(), disk: disk.trim(), pool, network: net, secureboot: sb, start: true })) }
|
||||||
catch (e) { setErr(errMsg(e)); setBusy(false) }
|
catch (e) { setErr(errMsg(e)); setBusy(false) }
|
||||||
}
|
}
|
||||||
return (
|
return (
|
||||||
@@ -581,6 +610,7 @@ function NewVMFromISO({ st, ov, iso, onClose, onStarted }: { st: VirtStatus; ov:
|
|||||||
<label className="muted">Memory<input className="mono" value={mem} onChange={(e) => setMem(e.target.value)} /></label>
|
<label className="muted">Memory<input className="mono" value={mem} onChange={(e) => setMem(e.target.value)} /></label>
|
||||||
<label className="muted">Disk<input className="mono" value={disk} onChange={(e) => setDisk(e.target.value)} /></label>
|
<label className="muted">Disk<input className="mono" value={disk} onChange={(e) => setDisk(e.target.value)} /></label>
|
||||||
</div>
|
</div>
|
||||||
|
<PoolPick ov={ov} value={pool} onChange={setPool} />
|
||||||
<label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
|
<label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
|
||||||
<input type="checkbox" checked={sb} onChange={(e) => setSb(e.target.checked)} style={{ width: 'auto' }} />Secure Boot — Windows, Debian, Ubuntu and Fedora boot with it; Alpine, Arch, TrueNAS and most others need it off
|
<input type="checkbox" checked={sb} onChange={(e) => setSb(e.target.checked)} style={{ width: 'auto' }} />Secure Boot — Windows, Debian, Ubuntu and Fedora boot with it; Alpine, Arch, TrueNAS and most others need it off
|
||||||
</label>
|
</label>
|
||||||
|
|||||||
+29
-8
@@ -85,7 +85,7 @@ export interface GPU {
|
|||||||
mhz?: number; req_mhz?: number; max_mhz?: number; mem_used?: number; mem_total?: number
|
mhz?: number; req_mhz?: number; max_mhz?: number; mem_used?: number; mem_total?: number
|
||||||
clients: GPUClient[]; clients_ok: boolean; stalled?: boolean
|
clients: GPUClient[]; clients_ok: boolean; stalled?: boolean
|
||||||
}
|
}
|
||||||
export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_temp?: number; core_ids?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
|
export interface ProcTable { t: number; cpu: number; cores: number[]; cpu_hist: number[]; cpu_model?: string; cpu_mhz?: number; cpu_max_mhz?: number; cpu_power_w?: number; cpu_temp?: number; core_ids?: number[]; load: [number, number, number]; mem: MemInfo; swap: MemInfo; net: NetIf[]; gpus?: GPU[]; uptime: number; procs: Proc[]; total: number; running: number }
|
||||||
export interface Sample {
|
export interface Sample {
|
||||||
t: number; interval_ms: number; cpu: number; cores: number[]; load: [number, number, number]
|
t: number; interval_ms: number; cpu: number; cores: number[]; load: [number, number, number]
|
||||||
mem: { total: number; used: number; avail: number }; swap: { total: number; used: number; avail: number }
|
mem: { total: number; used: number; avail: number }; swap: { total: number; used: number; avail: number }
|
||||||
@@ -154,8 +154,8 @@ export interface Hardware { at: string; system: HWSystem; board: HWBoard; cpu: H
|
|||||||
|
|
||||||
export interface DockerLogLine { t: string; m: string; err?: boolean }
|
export interface DockerLogLine { t: string; m: string; err?: boolean }
|
||||||
export interface DockerInfo { available: boolean; message?: string; tool?: string; containers: DockerContainer[]; images: Record<string, unknown>[]; volumes: Record<string, unknown>[]; networks: Record<string, unknown>[] }
|
export interface DockerInfo { available: boolean; message?: string; tool?: string; containers: DockerContainer[]; images: Record<string, unknown>[]; volumes: Record<string, unknown>[]; networks: Record<string, unknown>[] }
|
||||||
export interface Stack { name: string; containers: string[]; running: number; file: string; host_file: string; files: string[]; host_files: string[] | null; images: string[] | null; via: string; remote: string; held_by: string; editable: boolean; note?: string; has_env: boolean }
|
export interface Stack { name: string; containers: string[]; running: number; file: string; host_file: string; files: string[]; host_files: string[] | null; images: string[] | null; via: string; editable: boolean; note?: string; has_env: boolean }
|
||||||
export interface StackHome { dir: string; inner: string; via: string; remote: string; held_by: string; stacks: number; default: boolean }
|
export interface StackHome { dir: string; inner: string; via: string; stacks: number; default: boolean }
|
||||||
export interface ImageCheck { image: string; state: 'current' | 'update' | 'unknown'; detail?: string }
|
export interface ImageCheck { image: string; state: 'current' | 'update' | 'unknown'; detail?: string }
|
||||||
export interface StackDiff { name: string; base: '' | 'deployed' | 'saved'; old: string; new: string }
|
export interface StackDiff { name: string; base: '' | 'deployed' | 'saved'; old: string; new: string }
|
||||||
export interface AlertRule { id: string; name: string; sensor: string; op: string; value: number | boolean | string; for_sec: number; severity: 'info' | 'warn' | 'crit'; enabled: boolean; machines: string }
|
export interface AlertRule { id: string; name: string; sensor: string; op: string; value: number | boolean | string; for_sec: number; severity: 'info' | 'warn' | 'crit'; enabled: boolean; machines: string }
|
||||||
@@ -249,7 +249,14 @@ export interface ARCDetail {
|
|||||||
}
|
}
|
||||||
export interface ARCView { detail: ARCDetail; live: ARCLive | null }
|
export interface ARCView { detail: ARCDetail; live: ARCLive | null }
|
||||||
|
|
||||||
export interface FileEntry { name: string; type: 'dir' | 'file' | 'link' | 'other'; size: number; mode: string; perm: string; mtime: number; user: string; group: string; target?: string; broken?: boolean }
|
export interface FileEntry { name: string; type: 'dir' | 'file' | 'link' | 'other'; size: number; mode: string; perm: string; mtime: number; user: string; group: string; target?: string; broken?: boolean; preview?: 'image' | 'video' | 'audio' }
|
||||||
|
export interface PlayCaps { codecs: { id: string; label: string; engine?: string }[]; qualities: { id: string; label: string; height: number }[]; error?: string }
|
||||||
|
export interface PlayTrack { codec: string; profile?: string; pix_fmt?: string; width?: number; height?: number; fps?: number; channels?: number }
|
||||||
|
export interface PlayStream { id: string; codec: string; quality: string; engine: string; width: number; height: number; kbps: number; source: { container: string; duration: number; bit_rate?: number; video?: PlayTrack; audio?: PlayTrack } }
|
||||||
|
export interface PlayStatus { running: boolean; done: boolean; error?: string; progress: number; speed: number; segments: number }
|
||||||
|
export interface ThumbGPU { api: string; device: string; name?: string; codecs?: string[]; reason?: string; permission?: boolean }
|
||||||
|
// How previews get made on the machine: ok with an engine, or why not and what would fix it.
|
||||||
|
export interface ThumbStatus { ok: boolean; engine: 'gpu' | 'cpu' | ''; summary: string; ffmpeg?: string; gpus: ThumbGPU[]; unusable?: ThumbGPU[]; error?: string; install?: { pm: string; package: string }; gpu_fix?: boolean; user: string; cache: { path: string; entries: number; bytes: number } }
|
||||||
export interface FileInfo extends FileEntry { path: string; text: boolean }
|
export interface FileInfo extends FileEntry { path: string; text: boolean }
|
||||||
|
|
||||||
const q = (path: string, sudo: boolean, extra = '') => `path=${encodeURIComponent(path)}${sudo ? '&sudo=1' : ''}${extra}`
|
const q = (path: string, sudo: boolean, extra = '') => `path=${encodeURIComponent(path)}${sudo ? '&sudo=1' : ''}${extra}`
|
||||||
@@ -260,17 +267,17 @@ export interface VirtStatus { state: 'ready' | 'repair' | 'installable' | 'unava
|
|||||||
export interface VirtSnapshot { name: string; created: string; expires?: string; size?: number }
|
export interface VirtSnapshot { name: string; created: string; expires?: string; size?: number }
|
||||||
export interface VirtInstance { name: string; type: 'container' | 'virtual-machine'; status: string; arch: string; description?: string; image: string; os?: string; release?: string; cpu?: string; memory?: string; created: string; last_used: string; ipv4: string[]; snapshots: VirtSnapshot[]; profiles: string[]; ephemeral: boolean; cpu_pct: number; mem_usage: number; mem_peak: number; disk_usage: number; disk_total: number; pid: number; processes: number; config: Record<string, string>; devices: Record<string, Record<string, string>>; schedule?: string; expiry?: string }
|
export interface VirtInstance { name: string; type: 'container' | 'virtual-machine'; status: string; arch: string; description?: string; image: string; os?: string; release?: string; cpu?: string; memory?: string; created: string; last_used: string; ipv4: string[]; snapshots: VirtSnapshot[]; profiles: string[]; ephemeral: boolean; cpu_pct: number; mem_usage: number; mem_peak: number; disk_usage: number; disk_total: number; pid: number; processes: number; config: Record<string, string>; devices: Record<string, Record<string, string>>; schedule?: string; expiry?: string }
|
||||||
export interface VirtImage { fingerprint: string; aliases: string[]; arch: string; type: string; size: number; created: string; last_used: string; description: string; os?: string; release?: string; variant?: string; cached: boolean; source?: string; public: boolean }
|
export interface VirtImage { fingerprint: string; aliases: string[]; arch: string; type: string; size: number; created: string; last_used: string; description: string; os?: string; release?: string; variant?: string; cached: boolean; source?: string; public: boolean }
|
||||||
export interface VirtPool { name: string; driver: string; source: string; used: number; total: number; used_by: number }
|
export interface VirtPool { name: string; driver: string; source: string; where: string; used: number; total: number; used_by: number }
|
||||||
export interface VirtNetwork { name: string; type: string; managed: boolean; address?: string; nat: boolean; used_by: number }
|
export interface VirtNetwork { name: string; type: string; managed: boolean; address?: string; nat: boolean; used_by: number }
|
||||||
export interface LibraryFile { name: string; size: number; modified: string; attached_to?: string[] }
|
export interface LibraryFile { name: string; size: number; modified: string; attached_to?: string[] }
|
||||||
export interface ExportConfig { enabled: boolean; hour: number; minute: number; keep: number; instance_only: boolean; instances?: string[]; updated?: string }
|
export interface ExportConfig { enabled: boolean; hour: number; minute: number; keep: number; instance_only: boolean; instances?: string[]; updated?: string }
|
||||||
export interface ExportRun { at: string; exported: string[]; pruned: string[]; errors?: string[]; seconds: number }
|
export interface ExportRun { at: string; exported: string[]; pruned: string[]; errors?: string[]; seconds: number }
|
||||||
export interface ExportFile { instance: string; name: string; size: number; modified: string }
|
export interface ExportFile { instance: string; name: string; size: number; modified: string }
|
||||||
export interface ExportsView { config: ExportConfig; cron: string; next?: string; last: ExportRun | null; files: ExportFile[]; dir: string; cron_file: string }
|
export interface ExportsView { config: ExportConfig; cron: string; next?: string; last: ExportRun | null; files: ExportFile[]; dir: string; cron_file: string }
|
||||||
export interface FromISO { name: string; iso: string; cpu: string; memory: string; disk: string; network: string; secureboot: boolean; start: boolean }
|
export interface FromISO { name: string; iso: string; cpu: string; memory: string; disk: string; pool: string; network: string; secureboot: boolean; start: boolean }
|
||||||
export interface VirtOverview { status: VirtStatus; error?: string; instances?: VirtInstance[]; images?: VirtImage[]; pools?: VirtPool[]; networks?: VirtNetwork[]; library: { images: LibraryFile[]; isos: LibraryFile[] } }
|
export interface VirtOverview { status: VirtStatus; error?: string; instances?: VirtInstance[]; images?: VirtImage[]; pools?: VirtPool[]; default_pool?: string; networks?: VirtNetwork[]; library: { images: LibraryFile[]; isos: LibraryFile[] } }
|
||||||
export interface VirtInstallOpts { driver: string; pool_size_gib: number; subnet: string; docker_rules: boolean; arc_max_gib: number }
|
export interface VirtInstallOpts { driver: string; pool_size_gib: number; subnet: string; docker_rules: boolean; arc_max_gib: number }
|
||||||
export interface VirtCreate { name: string; image: string; vm: boolean; cpu: string; memory: string; disk: string; network: string; cloud_init: string; profiles: string[]; description: string; start: boolean }
|
export interface VirtCreate { name: string; image: string; vm: boolean; cpu: string; memory: string; disk: string; pool: string; network: string; cloud_init: string; profiles: string[]; description: string; start: boolean }
|
||||||
|
|
||||||
export const api = {
|
export const api = {
|
||||||
procs: () => req<ProcTable>('GET', '/api/procs'),
|
procs: () => req<ProcTable>('GET', '/api/procs'),
|
||||||
@@ -292,6 +299,16 @@ export const api = {
|
|||||||
},
|
},
|
||||||
fileReadURL: (path: string, sudo: boolean) => apiPath(`/api/files/read?${q(path, sudo)}`),
|
fileReadURL: (path: string, sudo: boolean) => apiPath(`/api/files/read?${q(path, sudo)}`),
|
||||||
fileDownloadURL: (path: string, sudo: boolean) => apiPath(`/api/files/download?${q(path, sudo)}`),
|
fileDownloadURL: (path: string, sudo: boolean) => apiPath(`/api/files/download?${q(path, sudo)}`),
|
||||||
|
// v carries the file's mtime and size, so a changed file is a new URL and an unchanged one a cached one
|
||||||
|
fileThumbURL: (path: string, sudo: boolean, mtime: number, size: number) => apiPath(`/api/files/thumb?${q(path, sudo, `&v=${mtime}-${size}`)}`),
|
||||||
|
thumbStatus: (sudo: boolean, refresh = false) => req<ThumbStatus>('GET', `/api/files/thumbs?${sudo ? 'sudo=1&' : ''}${refresh ? 'refresh=1' : ''}`),
|
||||||
|
thumbInstall: () => req<TaskInfo>('POST', '/api/files/thumbs/install'),
|
||||||
|
thumbGPUAccess: () => req<TaskInfo>('POST', '/api/files/thumbs/gpu-access'),
|
||||||
|
// playback of what the browser can't play as it is: an HLS re-encode on the machine
|
||||||
|
playCaps: (sudo: boolean) => req<PlayCaps>('GET', `/api/files/play-caps${sudo ? '?sudo=1' : ''}`),
|
||||||
|
playStart: (path: string, codec: string, quality: string, sudo: boolean) => req<{ stream: PlayStream; url: string }>('POST', `/api/files/play${sudo ? '?sudo=1' : ''}`, { path, codec, quality }),
|
||||||
|
playStatus: (id: string) => req<PlayStatus>('GET', `/api/files/play/${id}`),
|
||||||
|
playStop: (id: string) => req<void>('DELETE', `/api/files/play/${id}`),
|
||||||
fileWrite: async (path: string, text: string, sudo: boolean) => {
|
fileWrite: async (path: string, text: string, sudo: boolean) => {
|
||||||
const r = await fetch(apiPath(`/api/files/write?${q(path, sudo)}`), { method: 'PUT', credentials: 'same-origin', body: text, headers: { 'Content-Type': 'application/octet-stream' } })
|
const r = await fetch(apiPath(`/api/files/write?${q(path, sudo)}`), { method: 'PUT', credentials: 'same-origin', body: text, headers: { 'Content-Type': 'application/octet-stream' } })
|
||||||
if (r.status === 401) throw new Unauthenticated()
|
if (r.status === 401) throw new Unauthenticated()
|
||||||
@@ -308,6 +325,8 @@ export const api = {
|
|||||||
fileMkdir: (path: string, sudo: boolean) => req<void>('POST', `/api/files/mkdir${sudo ? '?sudo=1' : ''}`, { path }),
|
fileMkdir: (path: string, sudo: boolean) => req<void>('POST', `/api/files/mkdir${sudo ? '?sudo=1' : ''}`, { path }),
|
||||||
fileTouch: (path: string, sudo: boolean) => req<void>('POST', `/api/files/touch${sudo ? '?sudo=1' : ''}`, { path }),
|
fileTouch: (path: string, sudo: boolean) => req<void>('POST', `/api/files/touch${sudo ? '?sudo=1' : ''}`, { path }),
|
||||||
fileRename: (path: string, to: string, sudo: boolean) => req<void>('POST', `/api/files/rename${sudo ? '?sudo=1' : ''}`, { path, to }),
|
fileRename: (path: string, to: string, sudo: boolean) => req<void>('POST', `/api/files/rename${sudo ? '?sudo=1' : ''}`, { path, to }),
|
||||||
|
// a rename when the ends share a filesystem (nothing back), otherwise the copy-and-remove runs as a task (the task back)
|
||||||
|
fileMove: (path: string, to: string, sudo: boolean) => req<TaskInfo | undefined>('POST', `/api/files/move${sudo ? '?sudo=1' : ''}`, { path, to }),
|
||||||
fileDelete: (path: string, recursive: boolean, sudo: boolean) => req<void>('POST', `/api/files/delete${sudo ? '?sudo=1' : ''}`, { path, recursive }),
|
fileDelete: (path: string, recursive: boolean, sudo: boolean) => req<void>('POST', `/api/files/delete${sudo ? '?sudo=1' : ''}`, { path, recursive }),
|
||||||
fileChmod: (path: string, mode: string, sudo: boolean) => req<void>('POST', `/api/files/chmod${sudo ? '?sudo=1' : ''}`, { path, mode }),
|
fileChmod: (path: string, mode: string, sudo: boolean) => req<void>('POST', `/api/files/chmod${sudo ? '?sudo=1' : ''}`, { path, mode }),
|
||||||
storage: () => req<{ devices: StorageDevice[]; zfs: ZFSInfo; smartctl: boolean }>('GET', '/api/storage'),
|
storage: () => req<{ devices: StorageDevice[]; zfs: ZFSInfo; smartctl: boolean }>('GET', '/api/storage'),
|
||||||
@@ -456,6 +475,8 @@ export const api = {
|
|||||||
virtExportsRun: (instance = '') => req<TaskInfo>('POST', '/api/virt/exports/run', { instance }),
|
virtExportsRun: (instance = '') => req<TaskInfo>('POST', '/api/virt/exports/run', { instance }),
|
||||||
virtExportDelete: (instance: string, name: string) => req<void>('POST', '/api/virt/exports/delete', { instance, name }),
|
virtExportDelete: (instance: string, name: string) => req<void>('POST', '/api/virt/exports/delete', { instance, name }),
|
||||||
virtSSHKeys: () => req<{ keys: string[]; user: string }>('GET', '/api/virt/sshkeys'),
|
virtSSHKeys: () => req<{ keys: string[]; user: string }>('GET', '/api/virt/sshkeys'),
|
||||||
|
virtPoolCreate: (name: string, path: string) => req<{ name: string; path: string }>('POST', '/api/virt/pools', { name, path }),
|
||||||
|
virtPoolDelete: (name: string) => req<void>('POST', '/api/virt/pools/delete', { name }),
|
||||||
}
|
}
|
||||||
|
|
||||||
export function wsURLPath(path: string): string {
|
export function wsURLPath(path: string): string {
|
||||||
|
|||||||
@@ -364,8 +364,44 @@ h2.section { margin: 0; font-size: 13px; font-weight: 600; color: var(--muted);
|
|||||||
.file-editor { flex: 1; min-height: 0; border: 0; border-radius: 0; resize: none; font-size: 13px; }
|
.file-editor { flex: 1; min-height: 0; border: 0; border-radius: 0; resize: none; font-size: 13px; }
|
||||||
.preview { flex: 1; display: flex; align-items: center; justify-content: center; overflow: auto; padding: 12px; }
|
.preview { flex: 1; display: flex; align-items: center; justify-content: center; overflow: auto; padding: 12px; }
|
||||||
.preview img { max-width: 100%; max-height: 100%; object-fit: contain; }
|
.preview img { max-width: 100%; max-height: 100%; object-fit: contain; }
|
||||||
|
.preview video { max-width: 100%; max-height: 100%; background: #000; border-radius: 4px; outline: none; }
|
||||||
|
.player { flex: 1; min-height: 0; display: flex; flex-direction: column; }
|
||||||
|
.player .preview { flex: 1; min-height: 0; }
|
||||||
|
.play-bar { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; padding: 6px 12px; font-size: 12px; color: var(--muted); border-bottom: 1px solid var(--border); }
|
||||||
|
.play-bar select { padding: 4px 8px; background: var(--bg); border: 1px solid var(--border); border-radius: 5px; color: var(--fg); font-size: 12px; }
|
||||||
|
.play-pick { margin: 12px; }
|
||||||
|
.play-pick .row { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; margin-top: 10px; }
|
||||||
|
.play-pick label { display: flex; align-items: center; gap: 6px; font-size: 12px; color: var(--muted); }
|
||||||
|
.preview.audio { align-items: flex-start; }
|
||||||
|
.audio-box { display: flex; flex-direction: column; align-items: center; gap: 10px; width: min(560px, 100%); margin-top: 8vh; }
|
||||||
|
.audio-box .ficon { font-size: 48px; }
|
||||||
|
.audio-box audio { width: 100%; }
|
||||||
|
/* previews: the icon column becomes a thumbnail box for images and videos */
|
||||||
|
.file-list.thumbs .file-row { grid-template-columns: 72px 1fr 80px 80px 130px 32px; padding-top: 4px; padding-bottom: 4px; min-height: 52px; }
|
||||||
|
.fthumb { position: relative; width: 72px; height: 44px; display: flex; align-items: center; justify-content: center; background: var(--bg3); border-radius: 4px; overflow: hidden; }
|
||||||
|
.fthumb img { max-width: 100%; max-height: 100%; object-fit: contain; display: block; }
|
||||||
|
.fthumb.video::after, .fthumb.audio::after { content: '▶'; position: absolute; right: 2px; bottom: 2px; font-size: 9px; line-height: 1; padding: 2px 3px; border-radius: 3px; background: rgba(0,0,0,.6); color: #fff; }
|
||||||
|
.fthumb.audio::after { content: '♪'; }
|
||||||
|
.fthumb .ficon { font-size: 18px; }
|
||||||
|
.thumb-note { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; margin: 6px 10px 0; }
|
||||||
|
/* move: the folder picker */
|
||||||
|
.modal.mover { width: min(560px, 100%); height: min(76vh, 100%); overflow: hidden; }
|
||||||
|
.mover-nav { display: flex; align-items: center; gap: 2px; }
|
||||||
|
.mover-nav .icon-btn { font-size: 18px; min-width: 30px; line-height: 1; }
|
||||||
|
.mover-nav .icon-btn:disabled { opacity: .3; }
|
||||||
|
.mover-nav .crumbs { flex: 1; min-width: 0; margin-left: 4px; }
|
||||||
|
.modal.mover > .mover-list { flex: 1 1 auto; min-height: 80px; overflow-y: auto; margin: 0 -16px; border-top: 1px solid var(--border); }
|
||||||
|
.mover-row { display: flex; align-items: center; gap: 8px; padding: 9px 16px; cursor: pointer; border-bottom: 1px solid var(--border); min-height: 38px; }
|
||||||
|
.mover-row:hover { background: var(--bg3); }
|
||||||
|
.mover-row.self { opacity: .5; cursor: default; }
|
||||||
|
.mover-row .fname { flex: 1; }
|
||||||
|
.mover-dest { overflow-wrap: anywhere; }
|
||||||
|
.thumb-note .btn { padding-top: 3px; padding-bottom: 3px; }
|
||||||
|
.files-bar .task-out { margin: 6px 10px 0; max-height: 160px; }
|
||||||
@media (max-width: 800px) {
|
@media (max-width: 800px) {
|
||||||
.file-row { grid-template-columns: 24px 1fr 70px 32px; } .file-row .fmeta:nth-child(4), .file-row .perm { display: none; }
|
.file-row { grid-template-columns: 24px 1fr 70px 32px; } .file-row .fmeta:nth-child(4), .file-row .perm { display: none; }
|
||||||
|
.file-list.thumbs .file-row { grid-template-columns: 60px 1fr 70px 32px; min-height: 46px; }
|
||||||
|
.fthumb { width: 60px; height: 38px; }
|
||||||
.path-input { width: 100%; margin: 4px 0 0; }
|
.path-input { width: 100%; margin: 4px 0 0; }
|
||||||
.files-tools .filter { width: 80px; }
|
.files-tools .filter { width: 80px; }
|
||||||
}
|
}
|
||||||
|
|||||||
+8
-1
@@ -1,9 +1,16 @@
|
|||||||
# spice-html5, vendored
|
# spice-html5, vendored
|
||||||
|
|
||||||
Upstream: https://gitlab.freedesktop.org/spice/spice-html5 — commit b372aab (2026-09-21),
|
Upstream: https://gitlab.freedesktop.org/spice/spice-html5 — commit b372aab (2026-09-21),
|
||||||
copied unchanged on 2026-10-09 (`src/`, `thirdparty/`, `spice.css`). LGPL-3.0-or-later; the
|
copied unchanged on 2026-10-09 (`src/`, `thirdparty/`). LGPL-3.0-or-later; the
|
||||||
licence text is beside this file. It draws a VM's SPICE display into a canvas from a websocket
|
licence text is beside this file. It draws a VM's SPICE display into a canvas from a websocket
|
||||||
carrying the raw SPICE stream — which is what Incus's VGA console websocket is — so Ham Lab's
|
carrying the raw SPICE stream — which is what Incus's VGA console websocket is — so Ham Lab's
|
||||||
daemon bridges the two and this does the rest. Loaded lazily by `VgaConsole.tsx`, so the main
|
daemon bridges the two and this does the rest. Loaded lazily by `VgaConsole.tsx`, so the main
|
||||||
bundle does not carry it. Not on npm in a current form (the `@spice-project/spice-html5`
|
bundle does not carry it. Not on npm in a current form (the `@spice-project/spice-html5`
|
||||||
package stopped at 0.2.1 in 2019), hence the copy.
|
package stopped at 0.2.1 in 2019), hence the copy.
|
||||||
|
|
||||||
|
Upstream's `spice.css` is deliberately **not** here. It is the stylesheet of spice-html5's
|
||||||
|
demo page, with rules for `body`, `*`, `button` and `canvas` that apply to the whole
|
||||||
|
document. It was copied and imported on 2026-10-09 and turned every button in Ham Lab into
|
||||||
|
a gradient pill shifted off its place (v0.1.0-188 to -191). The console is styled by the
|
||||||
|
`.vga` rules in `src/styles.css`; `scripts/test-css-imports.mjs` fails on any new
|
||||||
|
stylesheet import that has not been looked at.
|
||||||
|
|||||||
Vendored
-168
@@ -1,168 +0,0 @@
|
|||||||
body
|
|
||||||
{
|
|
||||||
background-color: #999999;
|
|
||||||
color: #000000;
|
|
||||||
margin: 0;
|
|
||||||
padding: 0;
|
|
||||||
font-family: "Lucida Grande", "Lucida Sans Unicode", "Helvetica Neue", Helvetica, Arial, Verdana, sans-serif;
|
|
||||||
font-size: 12pt;
|
|
||||||
line-height: 1.5em;
|
|
||||||
}
|
|
||||||
|
|
||||||
* { margin: 0; }
|
|
||||||
|
|
||||||
button
|
|
||||||
{
|
|
||||||
cursor: pointer;
|
|
||||||
color: #000000;
|
|
||||||
background-color: #999999;
|
|
||||||
border: 1px solid #4a4a4a;
|
|
||||||
min-width: 150px;
|
|
||||||
border-radius: 10px;
|
|
||||||
background: -webkit-gradient(linear, left top, left bottom, from(#fff), to(#24414e));
|
|
||||||
background: -moz-linear-gradient(top, #fff, #24414e);
|
|
||||||
-ms-transform: translate(15%, -8%);
|
|
||||||
transform: translate(15%, -8%);
|
|
||||||
font-size: .90em;
|
|
||||||
font-weight: bolder;
|
|
||||||
padding: 0px 0px 3px 0px;
|
|
||||||
margin-top: 5px;
|
|
||||||
}
|
|
||||||
button:focus
|
|
||||||
{
|
|
||||||
outline: none;
|
|
||||||
}
|
|
||||||
button:hover
|
|
||||||
{
|
|
||||||
background-color: #24414e
|
|
||||||
}
|
|
||||||
button:active
|
|
||||||
{
|
|
||||||
background-color: #24414e;
|
|
||||||
transform: translate(14.85%, -7%);
|
|
||||||
}
|
|
||||||
|
|
||||||
.SidenavClosed
|
|
||||||
{
|
|
||||||
height: 100%;
|
|
||||||
width: 0;
|
|
||||||
position: fixed;
|
|
||||||
z-index: 1;
|
|
||||||
top: 0;
|
|
||||||
left: 0;
|
|
||||||
background-color: #bbbbbb;
|
|
||||||
box-shadow: 1px 0px #00000040;
|
|
||||||
overflow-x: hidden;
|
|
||||||
transition: 0.5s;
|
|
||||||
padding-top: 60px;
|
|
||||||
}
|
|
||||||
.SidenavOpen
|
|
||||||
{
|
|
||||||
height: 100%;
|
|
||||||
width: 200px !important;
|
|
||||||
position: fixed;
|
|
||||||
z-index: 1;
|
|
||||||
top: 0;
|
|
||||||
left: 0;
|
|
||||||
background-color: #bbbbbb;
|
|
||||||
box-shadow: 1px 0px #00000040;
|
|
||||||
overflow-x: hidden;
|
|
||||||
transition: 0.5s;
|
|
||||||
padding-top: 60px;
|
|
||||||
}
|
|
||||||
#Sidenav label
|
|
||||||
{
|
|
||||||
color: #000000;
|
|
||||||
margin-left: 3%;
|
|
||||||
text-shadow: 1px 1px 0px rgba(175, 210, 220, 0.8);
|
|
||||||
position: absolute;
|
|
||||||
font-size: .9em;
|
|
||||||
}
|
|
||||||
#Sidenav input
|
|
||||||
{
|
|
||||||
padding: 3px;
|
|
||||||
background-color: #fAfAfA;
|
|
||||||
border: 1px inset #999999;
|
|
||||||
outline: none;
|
|
||||||
float: right;
|
|
||||||
margin-right: 3%;
|
|
||||||
-moz-border-radius: 3px; -webkit-border-radius: 3px; border-radius: 3px;
|
|
||||||
box-sizing: border-box;
|
|
||||||
-moz-box-sizing: border-box;
|
|
||||||
-webkit-box-sizing: border-box;
|
|
||||||
position: relative;
|
|
||||||
max-width: 110px;
|
|
||||||
}
|
|
||||||
#Sidenav .closebtn
|
|
||||||
{
|
|
||||||
position: absolute;
|
|
||||||
top: 10px;
|
|
||||||
right: 10px;
|
|
||||||
font-size: 25px;
|
|
||||||
margin-left: 50px;
|
|
||||||
cursor: pointer;
|
|
||||||
color: #000000;
|
|
||||||
}
|
|
||||||
|
|
||||||
canvas
|
|
||||||
{
|
|
||||||
border: 1px solid #000000;
|
|
||||||
}
|
|
||||||
|
|
||||||
#login
|
|
||||||
{
|
|
||||||
background: #bbbbbb;
|
|
||||||
z-index: 1;
|
|
||||||
padding-top: 2px;
|
|
||||||
border-bottom: 2px solid #00000040;
|
|
||||||
}
|
|
||||||
#login p
|
|
||||||
{
|
|
||||||
margin-left: 27%;
|
|
||||||
display: inline;
|
|
||||||
font-size: large;
|
|
||||||
}
|
|
||||||
|
|
||||||
#spice-area
|
|
||||||
{
|
|
||||||
height: 100%;
|
|
||||||
width: 100%;
|
|
||||||
margin-left: auto;
|
|
||||||
margin-right: auto;
|
|
||||||
}
|
|
||||||
.spice-screen
|
|
||||||
{
|
|
||||||
min-height: 600px;
|
|
||||||
height: 100%;
|
|
||||||
text-align: center;
|
|
||||||
padding: 0;
|
|
||||||
background-color: #999999;
|
|
||||||
}
|
|
||||||
.spice-message
|
|
||||||
{
|
|
||||||
width: 90%;
|
|
||||||
height: 40%;
|
|
||||||
overflow: auto;
|
|
||||||
margin-left: auto;
|
|
||||||
margin-right: auto;
|
|
||||||
padding: 10px;
|
|
||||||
font-size: 8pt;
|
|
||||||
line-height: 1.1em;
|
|
||||||
font-family: 'Andale Mono', monospace;
|
|
||||||
background-color: #fAfAfA;
|
|
||||||
border: 1px inset #999999;
|
|
||||||
border-radius: 3px;
|
|
||||||
box-sizing: border-box;
|
|
||||||
-moz-box-sizing: border-box;
|
|
||||||
-webkit-box-sizing: border-box;
|
|
||||||
}
|
|
||||||
.spice-message p {
|
|
||||||
margin-bottom: 0em;
|
|
||||||
margin-top: 0em;
|
|
||||||
}
|
|
||||||
.spice-message-warning {
|
|
||||||
color: orange;
|
|
||||||
}
|
|
||||||
.spice-message-error {
|
|
||||||
color: red;
|
|
||||||
}
|
|
||||||
Reference in New Issue
Block a user