Author SHA1 Message Date
Mitchell-JohnsonandClaude Fable 5.1 a2674bea6a thumbs: a GPU-access run already in flight refuses a second (409)
The sweep's battery hit the route three times; the second and third runs only collided
with the first's restart timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-13 18:00:39 -07:00
Mitchell-JohnsonandClaude Fable 5.1 134bbc991e PLAN.md: the whole-console sweep
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-13 17:56:16 -07:00
Mitchell-JohnsonandClaude Fable 5.1 285278abdb A sweep of the whole console: status codes, a data race, atomic writes, dead code
From the race detector, staticcheck, a garbage-in battery over every safe route and a
headless pass through every tab on two machines. Fixed:
- docker inspect/health of a container that isn't there answered 500; now 404, and a
  daemon that is down 503 (dockerErr, with a test).
- /api/virt/instances/{name} on a machine without Incus answered 502 "incus not found";
  it now says Incus isn't ready, like its siblings.
- Deleting a ZFS snapshot with a bad name answered 500; a bad name is 400, a snapshot
  that doesn't exist 404.
- The task list read a task's Running flag without the task's lock while the waiter
  goroutine cleared it: a data race, now read under the lock.
- The Virtualization export config and run record were written in place; a crash
  mid-write would have left a half file. Atomic now, like the config and state files.
- An unused function in kvm.go, one in incus/subnet.go, an unused field in files/move.go
  and a capitalised error string, from staticcheck.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-13 17:52:18 -07:00
Mitchell-JohnsonandClaude Fable 5.1 05161bf7ef PLAN.md: the Files edge sweep
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 20:32:34 -07:00
Mitchell-JohnsonandClaude Fable 5.1 1038c010fa Files: a sweep of the edges
Found by walking every handler and the UI with odd inputs, then a battery of requests
against the live hub. Fixed:
- A save over the 8 MiB cap was cut short in place: the file was truncated, then the
  first 8 MiB written, and the caller told 204. Now the body is read whole first and a
  body over the cap is a 413 with the file untouched.
- A link to a file showed the link's own time; now the file's, so the row reads right
  and the preview key follows the file.
- Reading a link to nothing answered 200 with an empty body (the open failed after the
  headers in root mode); now a 404 that names the target.
- A play request with an unknown codec or size was answered by inspecting the file
  first, so an audio-only file said "no video stream" instead of what was wrong; the
  choice is checked before any work, and is a 400.
- A stream that ended in failure was handed back to the next identical request for ten
  minutes; a failed one is now dropped and a fresh one made.
- A player that left while its stream was still finding a first segment made the loop
  try the next recipe; it stops. A stream closed mid-start is not started again.
- Ending a root-mode ffmpeg could wait forever: the daemon cannot signal root's
  processes, so the wait is bounded and the helper's own grace does the rest.
- A page going away (a phone closing the tab) releases its stream by a beacon, rather
  than ten minutes later by the sweeper.
- The path box resolves like a shell — relative to the folder shown, . and .. applied,
  trailing slashes dropped — in the tab and in the Move picker, with a test. A typed path
  that is a file opens it; a remembered folder that is gone falls back to / once.
- A dropped folder arrived as a nameless empty file; it is refused with a word. The
  upload toast counts what was actually uploaded.
- The viewer names a device, pipe or socket, and a dangling link, instead of calling
  them binary files.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 20:31:10 -07:00
Mitchell-JohnsonandClaude Fable 5.1 d200e5dc2e PLAN.md: re-encoding for the browser, on the machine's GPU
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 17:46:35 -07:00
Mitchell-JohnsonandClaude Fable 5.1 f3c547adfa Files: re-encode a video the browser can't play, on the machine's own GPU, and play that
The viewer tries the file as it is first. When the browser gives up — Safari and an MKV,
HEVC without a hardware decoder — or on a tap, a picker: codec (H.264, HEVC, AV1, or the
original codec repackaged) and size (original, 1080p, 720p, 480p), each codec labelled
with the engine this machine would use. POST /api/files/play runs ffmpeg into an HLS
event playlist of fragmented-MP4 segments under <data dir>/transcode/<id>/; Safari plays
HLS itself and everything else goes through hls.js, loaded only then. The recipes are
tried best first: a card on the bus before the integrated GPU, decoding on the same card
before decoding on the CPU and uploading, the software encoders last; a recipe that
produces no first segment is dropped for the next. The seekable range grows as ffmpeg
gets ahead; status says how far. Streams are reused for the same request, closed when
the player goes, swept after ten idle minutes, and the folder is emptied at start. Root
mode runs ffmpeg as root through a helper that ends when the daemon lets go of it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 17:42:25 -07:00
Mitchell-JohnsonandClaude Fable 5.1 aa4ae57c22 gofmt after the rename
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:54:52 -07:00
Mitchell-JohnsonandClaude Fable 5.1 af189b1fa1 internal/thumbs is internal/media, with Inspect: what a file holds, from ffprobe
The package was named for its first job; it now also decides what a file is for the
Files tab (image, video, audio) and reads its streams, which the viewer will need. Inspect
is the richer probe — codec, profile, pixel format, size, frame rate, HDR transfer, the
audio beside it, container and duration — run as root through sudo when asked.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:54:15 -07:00
Mitchell-JohnsonandClaude Fable 5.1 beec9269fe PLAN.md: video and audio playback in the Files viewer, as the file is
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:43:03 -07:00
Mitchell-JohnsonandClaude Fable 5.1 d7f56ee0b3 Files: a seek costs nothing; the backend reopens only when a read wants somewhere else
ServeContent seeks to the end for the size before it seeks to the range, which was
throwing away the stream Check had just opened.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:41:54 -07:00
Mitchell-JohnsonandClaude Fable 5.1 a9bcb88212 Files: play video and audio in the viewer, as the file is; reads answer Range requests
The viewer hands a video or audio file to the browser untouched — container, codec and
all — and the browser plays what it can; when it can't, the viewer says so and offers the
download. What the server adds is the part a player needs: /api/files/read now goes
through http.ServeContent for a real file of known size, so Range requests (seeking) and
HEAD work, and a download can resume. The root backend's helper takes an offset for that,
and a Seeker opens the backend once, at the range asked for, instead of streaming from
the top. Audio files are a preview kind too: their cover art is the thumbnail.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:40:42 -07:00
Mitchell-JohnsonandClaude Fable 5.1 c24f6f5cde Files: a move of something that isn't there is a 404, not a task that fails on arrival
The kernel answers EXDEV for two paths on different filesystems before it looks for the
source, so the cross-filesystem path was taken for a missing file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:30:04 -07:00
Mitchell-JohnsonandClaude Fable 5.1 b6d72cce9f PLAN.md: Move… is a folder picker; moves across filesystems copy then remove
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:29:27 -07:00
Mitchell-JohnsonandClaude Fable 5.1 fb3b8e4b4c Files: Move… opens a folder picker, and a move across filesystems copies then removes
The picker walks the machine like the tab does — back, forward, up, breadcrumbs, a path
box, a new folder — and "Move here" puts the file or folder in the folder being looked
at; it says why it can't when it can't (already here, into itself, a name clash). POST
/api/files/move is a rename when the two ends share a filesystem and otherwise a task:
on a NAS every move between datasets is a different filesystem, and a rename there is
EXDEV. The copy is built under a hidden temporary name beside its destination, renamed
into place once complete, and only then is the original removed, so a failure or a
cancel leaves the original as it was. Modes, times, symlinks and, as root, owners are kept.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:27:55 -07:00
Mitchell-JohnsonandClaude Fable 5.1 e5d3fb2afd PLAN.md: previews are cached in thumbs.db
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:05:17 -07:00
Mitchell-JohnsonandClaude Fable 5.1 e198471f8c Files previews: the cache is one database file, <data dir>/thumbs.db
bbolt instead of a tree of hash-named JPEGs: pure Go, so the binary stays static; one
file per machine; an entry carries where the preview came from (engine, codec, size,
when) beside the picture, and a count rides along so the tab can say what it holds
without walking anything. A miss never creates the file, so a machine nobody browses
with previews on never has one; a cache that cannot be opened is logged and previews
are still made and served, just not kept.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 16:04:12 -07:00
40 changed files with 3126 additions and 217 deletions
+147 -7
View File
@@ -652,16 +652,23 @@ something long in one, switch away and back, refresh — it's all still there.
0.7 s on the CPU; the 8K Osmo HEVC 1.0 s against 2.9 s. `grid`'s two GPUs are listed in 0.7 s on the CPU; the 8K Osmo HEVC 1.0 s against 2.9 s. `grid`'s two GPUs are listed in
node order and both decode the same set (h264, hevc, vp9, mjpeg, mpeg2 — **no AV1 on node order and both decode the same set (h264, hevc, vp9, mjpeg, mpeg2 — **no AV1 on
either**, per vainfo, so AV1 is dav1d on the CPU). either**, per vainfo, so AV1 is dav1d on the CPU).
- **Cached under the machine's data dir**, `<data dir>/thumbs/<xx>/<sha256>.jpg`, keyed by - **Cached in one database file under the machine's data dir**, `<data dir>/thumbs.db`
path, size and mtime — a changed file gets a new preview, an unchanged one is never (bbolt — pure Go, so the binary stays static; v0.1.0-202, the first build kept them as
decoded twice, and the URL carries `v=<mtime>-<size>` so the browser's own cache is a tree of hash-named JPEGs under `thumbs/`, replaced the same day when Mitchell asked
right too (7 days, ETag, 304). A file ffmpeg could not decode is remembered beside the where they were kept: "stored in hamlab, maybe under a DB, up to you"). An entry is
JPEGs as `.err` with ffmpeg's reason, answered as 422, and retried only when the file keyed by path, size and mtime — a changed file gets a new preview, an unchanged one is
changes; a timeout is not remembered. Nothing expires: a preview is a few KB, and only never decoded twice — and carries the engine, codec, source size and time beside the
picture; a count rides in the file so the previews button's tooltip can say "N kept
(M MB)" without walking anything. The URL carries `v=<mtime>-<size>` so the browser's
own cache is right too (7 days, ETag, 304). A file ffmpeg could not decode is kept with
ffmpeg's reason and no picture, answered as 422, retried only when the file changes; a
timeout is not kept. A miss never creates the file, so a machine nobody browses with
previews on never has one; a cache that cannot be opened is logged and previews are
still made and served, just not kept. Nothing expires: a preview is a few KB, and only
files somebody scrolled past are ever made. At most 2–4 ffmpegs at once per machine files somebody scrolled past are ever made. At most 2–4 ffmpegs at once per machine
(half the cores, clamped), one per file however many rows ask, and a preview's work (half the cores, clamped), one per file however many rows ask, and a preview's work
runs on a clock of its own so the row that asked first leaving does not cancel it for runs on a clock of its own so the row that asked first leaving does not cancel it for
the rest. Each ffmpeg runs under `nice -n 10`. `hamlet`'s backup job skips the folder. the rest. Each ffmpeg runs under `nice -n 10`. `hamlet`'s backup job skips the file.
- **Root mode makes them as root** (`hamlab thumb`, the sibling of `hamlab fs`, caps on - **Root mode makes them as root** (`hamlab thumb`, the sibling of `hamlab fs`, caps on
stdin, JSON out), so a file only root can read gets a preview, and the probe runs as stdin, JSON out), so a file only root can read gets a preview, and the probe runs as
root too — root opens every render node whatever groups the daemon has. Which matters: root too — root opens every render node whatever groups the daemon has. Which matters:
@@ -676,6 +683,139 @@ something long in one, switch away and back, refresh — it's all still there.
- *Not done, on purpose:* no cap on the cache and no clear button (not asked; it only - *Not done, on purpose:* no cap on the cache and no clear button (not asked; it only
holds what was looked at); no tonemapping for HDR sources (a 10-bit HEVC frame comes out holds what was looked at); no tonemapping for HDR sources (a 10-bit HEVC frame comes out
flat); no preview in the viewer for a video (it still says "binary, download it"). flat); no preview in the viewer for a video (it still says "binary, download it").
- **Move… is a folder picker** — **shipped 2026-10-11** (v0.1.0-204, `MovePicker` in
`Files.tsx`, `internal/files/move.go`, `POST /api/files/move`). Asked for as: instead of
a path prompt, "an interactive file explorer, where I can go back/forward, etc and then
click some button, and it will move the file/folder to where I was". The picker opens in
the item's own folder and walks the machine the way the tab does — back and forward
through where it has been, up, breadcrumbs, a path box, folders only, `.hidden`
following the tab's toggle, **+ folder** to make the destination on the spot — and
**Move here** puts the item in the folder being looked at. It says why it can't when it
can't, with the button greyed: already here, a folder into itself, a name clash in that
folder (read from the listing before anything is tried). The row being moved shows in
its own folder, greyed. Rename stays a name prompt: it is a different question.
- **Across filesystems it copies, then removes.** `os.Rename` is EXDEV between
filesystems, and on `grid` every dataset is one — `/tank/media` to `/tank/files` was a
"invalid cross-device link" error before this, and the picker would have made that
move easy to attempt. `/api/files/move` renames when it can (204) and otherwise runs
`hamlab fs move` as a task (202 with the task; the tab shows its output and reloads
when it ends). The copy is built under a hidden name beside its destination
(`.<name>.hamlab-moving`), renamed into place once complete, and only then is the
original removed — a failure part-way removes the partial copy and leaves the original
as it was, and a cancel (SIGTERM to the helper) does the same. Modes, mtimes and
symlinks are kept; owners too when the backend is root. Devices, pipes and sockets are
refused outright rather than half-copied. Progress every 500 files or 512 MiB.
Verified on `grid` ext4 → ZFS with a tree holding a 0600 file, a symlink and an old
mtime, all intact on the other side.
- **Video and audio play in the viewer** — **shipped 2026-10-11** (v0.1.0-208). Asked for
as "keep it simple for now, and just send whatever codec the file is in to the client,
and let the client try to decode and watch it" — and that is the whole design. Opening a
video or audio file shows a `<video>`/`<audio>` with the browser's own controls, fed the
file as it is by `/api/files/read`; nothing is transcoded or remuxed, so what plays is
what the browser decodes (Safari: h264/hevc/aac in mp4/mov; Chrome: those plus vp9/av1 in
mkv/webm, hevc only where the machine has a hardware decoder; nothing plays MPEG-TS or a
GoPro `.360`'s raw EAC strip). When the element errors the viewer says so and leaves the
download. What the server gained is the part a player cannot do without: reads of a
real file with a known size go through `http.ServeContent`, so **Range requests** (every
seek, and Safari's first request) and HEAD work, and a download can resume. The root
backend's helper takes `--offset` for that, and `files.Seeker` opens the backend once, at
the range asked for, rather than streaming from the top and skipping; seeking costs
nothing until a read wants somewhere else. `/proc` files, which stat at size 0, keep the
plain streaming path, as does the editor's capped read. Audio became a preview kind at
the same time (`thumbs.Kind` → `audio`): its cover art, where it has any, is the row's
thumbnail, with a ♪ badge.
- *Not done, on purpose:* no playlist or next/previous, no subtitles. (Transcoding came
the same evening after all — the next entry — because Safari turned out not to play
the gaming captures.)
- **Re-encoding for the browser, on the machine's GPU** — **shipped 2026-10-11**
(v0.1.0-212, `internal/media/{encode,hls,ffrun,inspect}.go`, `internal/api/play.go`,
`Player.tsx`, `hls.js`). Asked for when Safari would not play the gaming captures:
"add GPU transcoding to a supported codec when the client browser can't play it. Each
machine should use its own GPU (if available) and transcode to probably H.264. Maybe we
need a quality selector, and a codec selector, up to you." What Safari actually refuses
is the **MKV container** — the captures checked are H.264 High 4:2:0, 18–36 Mbps — so
the picker has a "repackaged" choice beside the codecs, and a 36 Mbps capture over
WireGuard is the other reason to re-encode at all.
- **The viewer tries the file as it is first**, as before. When the browser gives up, or
on **Re-encode…** in the bar, a picker: **codec** — H.264, HEVC, AV1, or the original
codec repackaged — each labelled with the engine this machine would use (or greyed as
"not on this machine"), and **size** — original, 1080p, 720p, 480p; the choice is
remembered. Play starts the re-encode and the player follows it: "h264 High 2292×960
60 fps 36 Mbps in matroska → H.264 2292×960 5.0 Mbps on Intel Arc A380 (vaapi) · 7.3× ·
34% through — seek within what's done", then "all of it is ready". **Change…** drops
the stream and goes back to the picker.
- **HLS, made on the machine in the switcher.** `POST /api/files/play` inspects the file
(ffprobe, as root in root mode) and runs ffmpeg into an *event* playlist of 4 s
fragmented-MP4 segments under `<data dir>/transcode/<id>/`, answering once the first
segment exists (0.5–1 s on `grid`); the player reads the playlist and segments from
`/api/files/play/<id>/…` (only those three names are served), Safari natively and
everything else through hls.js, loaded only then. ffmpeg runs ahead at the engine's
pace, so the seekable range grows and the whole file is through long before it is
watched. Keyframes are forced every 4 s so segments cut cleanly; sound is AAC stereo
unless it already is AAC; HDR is tone-mapped (`tonemap_vaapi`, or zscale/hable on the
CPU). Nothing is kept: the same request within ten minutes reuses the stream, the
player closes it when it goes (and the "Change…" path does), the sweeper closes what
nobody touched for ten minutes, and the folder is emptied when the daemon starts. At
most three at once per machine (429 with the reason). Root mode runs ffmpeg as root
through `hamlab run-ffmpeg`, which ends when the daemon lets go of its stdin — so a
stream cannot outlive the console.
- **Engines are probed, never assumed**, the same probe the previews use, now also
reading each node's encode profiles from vainfo and ffmpeg's own encoder list. The
recipes for a codec are tried best first until one produces a first segment: a card on
the bus before the integrated GPU (the A380 before the UHD 750), decoding on the same
card so frames never leave it, then decoding on the CPU and uploading (a source the
card cannot decode — AV1 on `grid`), then libx264 / libx265 / SVT-AV1. The size is
never larger than the source, and the bit rate follows the output size (2/4/8/12/20
Mbps for 480p/720p/1080p/1440p/2160p, +25 % at 50 fps and up, ×0.6 for HEVC and AV1).
Measured on `grid` with a 2292×960 60 fps capture: H.264 7.3× realtime, HEVC 720p
11.7×, AV1 720p 13×, all on the Arc. `core` and `hamtv` re-encode on the CPU until
*Allow GPU access* is pressed (libx264 at 9.4× for a 720p→480p clip on `core`); the Mac
and `gate` need ffmpeg first, which the previews note offers to install.
- *Not done, on purpose:* no restart-from-a-seek beyond what is encoded (the encode is
faster than watching; a seek far ahead waits), no bit-rate ladder or adaptive
switching, no subtitles, no keeping a finished re-encode for next time.
- **Files edge sweep** — **2026-10-11** (v0.1.0-214), asked for as "a full bug sweep of
the Files section, every example and edge case". Every handler and both UI files read
through with odd inputs in mind, then a battery of ~60 requests against the live hub
(`sweep.py`, not kept: listings of files, missing and forbidden paths, NUL bytes, names
with spaces/#/%/?/umlauts, links to files, folders and nothing, a pipe, over-size saves,
uploads with bad names and clashes, renames and moves onto things, protected paths,
thumbnails of empty and mislabelled images, playback of an image and an audio-only mp4,
root-mode variants). One real data bug: **a save over the 8 MiB cap truncated the file
and kept the first 8 MiB, answering 204** — the body is now read whole first and refused
with 413, the file untouched. The rest: a link to a file showed the link's own time; a
link to nothing read as an empty 200; a bad codec or size was answered by inspecting
the file; a failed stream was handed back to the next identical request; a player
leaving mid-start made the recipe loop try the next engine; a root-mode ffmpeg's end
could wait forever; a page closing kept its stream for ten minutes (a beacon now
releases it at once); the path box took only absolute paths (it resolves like a shell
now, in the tab and the Move picker, with a test); a dropped folder became a nameless
empty file; the upload toast counted intentions; the viewer called a pipe a binary
file. Not bugs, left alone: `rm -rf` of a mount point empties it (the console is
root-capable by design); a folder of ten thousand rows is slow (no virtual list).
- **Whole-console sweep** — **2026-10-13** (v0.1.0-216), asked for as "a full sweep of
Ham Lab again, look for any bugs and fix them". Four nets, then reading: the Go race
detector over the whole suite (clean), staticcheck's development build (the release does
not know Go 1.26; it found only dead code and one capitalised error string), a
garbage-in battery over every route safe to poke — 202 routes, bad identifiers, empty
and malformed bodies, with the destructive ones held back — and a headless Chromium pass
through all twenty tabs on the hub and on `grid`, collecting exceptions, console errors
and failed requests (none). By hand: the auth middleware and sessions, token issue, the
hub→node and KVM proxies, every validation regex before a shell, the cron and cron.d
writers, config and state atomicity, every WebSocket and SSE loop's exit, the terminal
manager, the task manager, and every destructive confirmation in the UI. Fixed, all
small: docker inspect/health of a missing container was a 500 (404 now, a daemon that
is down 503), `/api/virt/instances/{name}` without Incus was a 502 (409 like its
siblings), a ZFS snapshot delete with a bad name was a 500 (400; 404 for one that does
not exist), the task list read a task's Running flag outside its lock (a real data race,
never seen), the Virtualization export config was written in place rather than
atomically, and the dead code. Nothing in auth, the proxies or the shell boundaries.
One self-inflicted finding: the battery's skip list left out `POST
/api/files/thumbs/gpu-access`, so the sweep pressed *Allow GPU access* on `core` three
times over — the drop-in landed and the hub restarted (that is what the headless pass
saw as a refused connection). Kept, since it is the fix the tab offers; `core` makes
previews and re-encodes on its UHD iGPU now. The handler refuses a second run while one
is in flight (409), since the second `systemd-run` only collided with the first's timer.
- **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth. - **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth.
- **Notifications** — push on service failure / SMART warning / UPS event. - **Notifications** — push on service failure / SMART warning / UPS event.
- **Terminal extras** — search, copy-mode, font/theme, split panes, upload via - **Terminal extras** — search, copy-mode, font/theme, split panes, upload via
+4 -2
View File
@@ -31,6 +31,7 @@ import (
"hamlab/internal/cron" "hamlab/internal/cron"
"hamlab/internal/files" "hamlab/internal/files"
"hamlab/internal/incus" "hamlab/internal/incus"
"hamlab/internal/media"
"hamlab/internal/metrics" "hamlab/internal/metrics"
"hamlab/internal/nodes" "hamlab/internal/nodes"
"hamlab/internal/priv" "hamlab/internal/priv"
@@ -41,7 +42,6 @@ import (
"hamlab/internal/store" "hamlab/internal/store"
"hamlab/internal/tasks" "hamlab/internal/tasks"
hterm "hamlab/internal/term" hterm "hamlab/internal/term"
"hamlab/internal/thumbs"
"hamlab/internal/tlsauto" "hamlab/internal/tlsauto"
"hamlab/internal/updates" "hamlab/internal/updates"
"hamlab/internal/web" "hamlab/internal/web"
@@ -84,7 +84,9 @@ func main() {
case "fs": // privileged helper for the Files tab (run as root by the daemon) case "fs": // privileged helper for the Files tab (run as root by the daemon)
os.Exit(files.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr)) os.Exit(files.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
case "thumb": // privileged helper for the Files tab's previews in root mode (run as root by the daemon) case "thumb": // privileged helper for the Files tab's previews in root mode (run as root by the daemon)
os.Exit(thumbs.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr)) os.Exit(media.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
case "run-ffmpeg": // privileged helper for playback in root mode: ffmpeg as root, ending when stdin closes
os.Exit(media.RunFFmpegHelper(flag.Args()[1:], os.Stdin, os.Stderr))
case "space-walk": // helper for the Space tab: measure mountpoints, stream the tree back case "space-walk": // helper for the Space tab: measure mountpoints, stream the tree back
os.Exit(space.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr)) os.Exit(space.Helper(flag.Args()[1:], os.Stdin, os.Stdout, os.Stderr))
case "drm-clients": // privileged helper for the GPU readings: who has the GPU open (run as root by the daemon) case "drm-clients": // privileged helper for the GPU readings: who has the GPU open (run as root by the daemon)
+1
View File
@@ -5,6 +5,7 @@ go 1.26.0
require ( require (
github.com/coder/websocket v1.8.15 github.com/coder/websocket v1.8.15
github.com/creack/pty v1.1.24 github.com/creack/pty v1.1.24
go.etcd.io/bbolt v1.5.0
golang.org/x/crypto v0.57.0 golang.org/x/crypto v0.57.0
golang.org/x/term v0.46.0 golang.org/x/term v0.46.0
) )
+12
View File
@@ -2,9 +2,21 @@ github.com/coder/websocket v1.8.15 h1:6B2JPeOGlpff2Uz6vOEH1Vzpi0iUz20A+lPVhPHtNU
github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg= github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s= github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE= github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
go.etcd.io/bbolt v1.5.0/go.mod h1:mkltfYE5aUHQxUct9N9V+Kp7aSjFqjgrhcXIS70Lrdk=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M= golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA= golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE= golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc= golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+22 -3
View File
@@ -343,6 +343,21 @@ func (s *Server) dockerLogs(w http.ResponseWriter, r *http.Request) {
} }
// dockerInspect is the parts of `docker inspect` worth reading on a phone. // dockerInspect is the parts of `docker inspect` worth reading on a phone.
// dockerErr answers for a docker command that failed: a thing that isn't there is a 404,
// the daemon being away a 503, anything else a 500.
func dockerErr(w http.ResponseWriter, err error) {
msg := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(err.Error()), "error: "))
low := strings.ToLower(msg)
switch {
case strings.Contains(low, "no such object") || strings.Contains(low, "no such container") || strings.Contains(low, "not found"):
jsonErr(w, 404, msg)
case strings.Contains(low, "cannot connect to the docker daemon") || strings.Contains(low, "is the docker daemon running"):
jsonErr(w, 503, msg)
default:
jsonErr(w, 500, msg)
}
}
func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) { func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id") id := r.PathValue("id")
if !dockerRef.MatchString(id) { if !dockerRef.MatchString(id) {
@@ -352,9 +367,13 @@ func (s *Server) dockerInspect(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
defer cancel() defer cancel()
out, err := dockerRun(ctx, "inspect", id) out, err := dockerRun(ctx, "inspect", id)
if err != nil {
dockerErr(w, err)
return
}
var all []map[string]any var all []map[string]any
if err != nil || json.Unmarshal(out, &all) != nil || len(all) == 0 { if json.Unmarshal(out, &all) != nil || len(all) == 0 {
jsonErr(w, 500, fmt.Sprint("inspect failed: ", err)) jsonErr(w, 500, "inspect: unreadable answer")
return return
} }
c := all[0] c := all[0]
@@ -486,7 +505,7 @@ func (s *Server) dockerHealth(w http.ResponseWriter, r *http.Request) {
defer cancel() defer cancel()
out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id) out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id)
if err != nil { if err != nil {
jsonErr(w, 500, err.Error()) dockerErr(w, err)
return return
} }
var st struct { var st struct {
+123
View File
@@ -0,0 +1,123 @@
package api
import (
"bytes"
"errors"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"testing"
"hamlab/internal/files"
)
// A file read answers Range requests — what a <video> needs to seek — and HEAD, through
// the same handler that streams everything else.
func TestFilesReadRanges(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(dir, "clip.mp4")
data := make([]byte, 10000)
for i := range data {
data[i] = byte(i % 251)
}
if err := os.WriteFile(p, data, 0o644); err != nil {
t.Fatal(err)
}
s := &Server{fsLocal: files.Local{}}
get := func(method, extra string, hdr map[string]string) *httptest.ResponseRecorder {
r := httptest.NewRequest(method, "/api/files/read?path="+url.QueryEscape(p)+extra, nil)
for k, v := range hdr {
r.Header.Set(k, v)
}
w := httptest.NewRecorder()
s.filesRead(w, r)
return w
}
w := get("GET", "", nil)
if w.Code != 200 || w.Body.Len() != 10000 || w.Header().Get("Content-Type") != "video/mp4" || w.Header().Get("Accept-Ranges") != "bytes" {
t.Fatalf("whole: %d %d %q %q", w.Code, w.Body.Len(), w.Header().Get("Content-Type"), w.Header().Get("Accept-Ranges"))
}
w = get("GET", "", map[string]string{"Range": "bytes=100-199"})
if w.Code != 206 || w.Header().Get("Content-Range") != "bytes 100-199/10000" || w.Body.Len() != 100 || w.Body.Bytes()[0] != data[100] {
t.Fatalf("range: %d %q %d", w.Code, w.Header().Get("Content-Range"), w.Body.Len())
}
w = get("GET", "", map[string]string{"Range": "bytes=9990-"})
if w.Code != 206 || w.Body.Len() != 10 || w.Body.Bytes()[9] != data[9999] {
t.Fatalf("tail range: %d %d", w.Code, w.Body.Len())
}
w = get("HEAD", "", nil)
if w.Code != 200 || w.Body.Len() != 0 || w.Header().Get("Content-Length") != "10000" {
t.Fatalf("head: %d %d %q", w.Code, w.Body.Len(), w.Header().Get("Content-Length"))
}
if w := get("GET", "&max=50", nil); w.Code != 413 {
t.Fatalf("the editor's cap: %d", w.Code)
}
if w := get("GET", "", map[string]string{"Range": "bytes=20000-"}); w.Code != 416 {
t.Fatalf("a range past the end: %d", w.Code)
}
if err := os.Chmod(p, 0); err == nil && os.Geteuid() != 0 {
if w := get("GET", "", nil); w.Code != 403 {
t.Fatalf("unreadable: %d", w.Code)
}
}
}
// A write over the cap is refused whole: the file is not truncated first.
func TestFilesWriteCap(t *testing.T) {
p := filepath.Join(t.TempDir(), "f.txt")
if err := os.WriteFile(p, []byte("keep me"), 0o644); err != nil {
t.Fatal(err)
}
s := &Server{fsLocal: files.Local{}}
body := make([]byte, 8<<20+1)
r := httptest.NewRequest("PUT", "/api/files/write?path="+url.QueryEscape(p), bytes.NewReader(body))
w := httptest.NewRecorder()
s.filesWrite(w, r)
if w.Code != 413 {
t.Fatalf("got %d", w.Code)
}
if b, _ := os.ReadFile(p); string(b) != "keep me" {
t.Fatalf("the file was touched: %q", b)
}
r = httptest.NewRequest("PUT", "/api/files/write?path="+url.QueryEscape(p), bytes.NewReader([]byte("new")))
w = httptest.NewRecorder()
s.filesWrite(w, r)
if b, _ := os.ReadFile(p); w.Code != 204 || string(b) != "new" {
t.Fatalf("a write within the cap: %d %q", w.Code, b)
}
}
// A link to nothing is a 404, not an empty 200.
func TestFilesReadDanglingLink(t *testing.T) {
dir := t.TempDir()
l := filepath.Join(dir, "broken")
if err := os.Symlink("nowhere", l); err != nil {
t.Fatal(err)
}
s := &Server{fsLocal: files.Local{}}
r := httptest.NewRequest("GET", "/api/files/read?path="+url.QueryEscape(l), nil)
w := httptest.NewRecorder()
s.filesRead(w, r)
if w.Code != 404 {
t.Fatalf("got %d: %s", w.Code, w.Body.String())
}
}
// dockerErr tells a thing that isn't there from a daemon that isn't up from anything else.
func TestDockerErr(t *testing.T) {
for _, c := range []struct {
msg string
want int
}{
{"error: No such object: zzz", 404}, {"Error response from daemon: No such container: x", 404},
{"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?", 503},
{"something else entirely", 500},
} {
w := httptest.NewRecorder()
dockerErr(w, errors.New(c.msg))
if w.Code != c.want {
t.Errorf("%q → %d, want %d", c.msg, w.Code, c.want)
}
}
}
-14
View File
@@ -338,20 +338,6 @@ func (s *Server) listKVMs(w http.ResponseWriter, _ *http.Request) {
jsonOK(w, 200, out) jsonOK(w, 200, out)
} }
// kvmForMachine is the KVM attached to a machine, "" being the hub itself.
func (s *Server) kvmForMachine(machine string) (config.KVM, bool) {
var found config.KVM
ok := false
s.cfg.View(func(c *config.Config) {
for _, k := range c.KVMs {
if k.Machine == machine {
found, ok = k, true
}
}
})
return found, ok
}
func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) { func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
var body struct{ Name, URL, Machine string } var body struct{ Name, URL, Machine string }
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil { if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
+169
View File
@@ -0,0 +1,169 @@
package api
// Playing a video the browser cannot play as it is: POST /api/files/play starts an HLS
// re-encode on this machine (internal/media), GET …/play/{id}/index.m3u8 and the segments
// feed the player, GET …/play/{id} is its progress, DELETE ends it. /api/files/play-caps
// says which codecs this machine can make and on what. Root mode (?sudo=1) reads the file
// and runs ffmpeg as root, through the same sudo the file reads use.
import (
"context"
"encoding/json"
"errors"
"io"
"log"
"net/http"
"path/filepath"
"strings"
"time"
"hamlab/internal/files"
"hamlab/internal/media"
)
const playLimit = 3 // re-encodes at once on one machine; a GPU does two or three, a CPU one
type playCaps struct {
Codecs []playCodec `json:"codecs"`
Qualities []media.Quality `json:"qualities"`
Error string `json:"error,omitempty"`
}
type playCodec struct {
media.Codec
Engine string `json:"engine,omitempty"` // "" when nothing here encodes it
}
// playCapsHandler: GET /api/files/play-caps[?sudo=1]
func (s *Server) playCapsHandler(w http.ResponseWriter, r *http.Request) {
root := r.URL.Query().Get("sudo") == "1"
if root && s.fsSudo == nil {
jsonErr(w, 403, "root access needs passwordless sudo on the host")
return
}
caps, err := s.thumbs.capsFor(r.Context(), root, false)
pc := playCaps{Codecs: []playCodec{}, Qualities: media.Qualities}
if err != nil {
pc.Error = err.Error()
} else if caps.Error != "" {
pc.Error = caps.Error
}
for _, c := range media.Codecs {
x := playCodec{Codec: c}
if pc.Error == "" {
x.Engine = media.EngineFor(caps, c.ID)
}
pc.Codecs = append(pc.Codecs, x)
}
jsonOK(w, 200, pc)
}
// playStart: POST /api/files/play[?sudo=1] {path, codec, quality}
func (s *Server) playStart(w http.ResponseWriter, r *http.Request) {
fsys, ok := s.fsFor(w, r)
if !ok {
return
}
root := r.URL.Query().Get("sudo") == "1"
var body struct{ Path, Codec, Quality string }
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
p, err := files.Clean(body.Path)
if err != nil {
jsonErr(w, 400, err.Error())
return
}
if media.Kind(p) != "video" {
jsonErr(w, 415, "not a video")
return
}
if err := media.ValidChoice(body.Codec, body.Quality); err != nil {
jsonErr(w, 400, err.Error())
return
}
if _, err := fsys.Stat(r.Context(), p); err != nil {
fsErr(w, err)
return
}
caps, err := s.thumbs.capsFor(r.Context(), root, false)
if err != nil {
jsonErr(w, 503, err.Error())
return
}
if caps.Error != "" {
jsonErr(w, 503, caps.Error)
return
}
ictx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
src, err := media.Inspect(ictx, caps, p, root)
cancel()
if err != nil {
playErr(w, err)
return
}
st, err := s.plays.Open(r.Context(), caps, root, p, body.Codec, body.Quality, src)
if err != nil {
playErr(w, err)
return
}
log.Printf("files: play %s as %s %s on %s (from %s)", p, st.Codec, st.Quality, st.Engine, clientIP(r))
jsonOK(w, 200, map[string]any{"stream": st, "url": "/api/files/play/" + st.ID + "/index.m3u8"})
}
func playErr(w http.ResponseWriter, err error) {
var de *media.DecodeError
switch {
case errors.As(err, &de):
jsonErr(w, 422, de.Msg)
case strings.Contains(err.Error(), "already being re-encoded"):
jsonErr(w, 429, err.Error())
case strings.HasPrefix(err.Error(), "unknown ") || strings.HasPrefix(err.Error(), "nothing on this machine encodes"):
jsonErr(w, 400, err.Error())
case errors.Is(err, context.DeadlineExceeded):
jsonErr(w, 504, "the file took too long to read")
default:
jsonErr(w, 500, err.Error())
}
}
// playStatus: GET /api/files/play/{id}
func (s *Server) playStatus(w http.ResponseWriter, r *http.Request) {
st, ok := s.plays.Find(r.PathValue("id"))
if !ok {
jsonErr(w, 404, "no such stream")
return
}
jsonOK(w, 200, st.Status())
}
// playStop: DELETE /api/files/play/{id}, or POST …/close — the form a page can send on its
// way out (navigator.sendBeacon), so a phone closing the tab releases the machine at once.
func (s *Server) playStop(w http.ResponseWriter, r *http.Request) {
s.plays.Close(r.PathValue("id"))
w.WriteHeader(204)
}
// playFile: GET /api/files/play/{id}/{file} — the playlist, the init segment, a segment.
func (s *Server) playFile(w http.ResponseWriter, r *http.Request) {
st, ok := s.plays.Find(r.PathValue("id"))
if !ok {
jsonErr(w, 404, "no such stream")
return
}
p, ok := st.File(r.PathValue("file"))
if !ok {
jsonErr(w, 404, "not a stream file")
return
}
switch filepath.Ext(p) {
case ".m3u8":
w.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
default:
w.Header().Set("Content-Type", "video/mp4")
}
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("X-Content-Type-Options", "nosniff")
http.ServeFile(w, r, p)
}
+103 -12
View File
@@ -35,6 +35,7 @@ import (
"hamlab/internal/files" "hamlab/internal/files"
"hamlab/internal/hardware" "hamlab/internal/hardware"
"hamlab/internal/incus" "hamlab/internal/incus"
"hamlab/internal/media"
"hamlab/internal/metrics" "hamlab/internal/metrics"
"hamlab/internal/network" "hamlab/internal/network"
"hamlab/internal/nodes" "hamlab/internal/nodes"
@@ -80,6 +81,8 @@ type Server struct {
fsLocal files.FS fsLocal files.FS
fsSudo files.FS // nil without passwordless sudo fsSudo files.FS // nil without passwordless sudo
thumbs *thumbSvc thumbs *thumbSvc
plays *media.Streamer
self string // this binary, for helpers run as tasks
nodes *nodes.Manager nodes *nodes.Manager
fingerprint string fingerprint string
version string version string
@@ -99,8 +102,10 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
if err != nil { if err != nil {
self = "hamlab" self = "hamlab"
} }
s.self = self
s.space = space.New(self, filepath.Join(cfg.DataDir(), "space")) s.space = space.New(self, filepath.Join(cfg.DataDir(), "space"))
s.thumbs = newThumbSvc(cfg.DataDir(), self) s.thumbs = newThumbSvc(cfg.DataDir(), self)
s.plays = media.NewStreamer(cfg.DataDir(), self, playLimit)
s.hostName() // read it once now so the first request doesn't have to s.hostName() // read it once now so the first request doesn't have to
if u, err := user.Current(); err == nil && u.Username != "" { if u, err := user.Current(); err == nil && u.Username != "" {
s.user = u.Username s.user = u.Username
@@ -301,12 +306,19 @@ func (s *Server) routes() *http.ServeMux {
m.HandleFunc("POST /api/files/mkdir", s.auth(s.filesMkdir)) m.HandleFunc("POST /api/files/mkdir", s.auth(s.filesMkdir))
m.HandleFunc("POST /api/files/touch", s.auth(s.filesTouch)) m.HandleFunc("POST /api/files/touch", s.auth(s.filesTouch))
m.HandleFunc("POST /api/files/rename", s.auth(s.filesRename)) m.HandleFunc("POST /api/files/rename", s.auth(s.filesRename))
m.HandleFunc("POST /api/files/move", s.auth(s.filesMove))
m.HandleFunc("POST /api/files/delete", s.auth(s.filesDelete)) m.HandleFunc("POST /api/files/delete", s.auth(s.filesDelete))
m.HandleFunc("POST /api/files/chmod", s.auth(s.filesChmod)) m.HandleFunc("POST /api/files/chmod", s.auth(s.filesChmod))
m.HandleFunc("GET /api/files/thumb", s.auth(s.filesThumb)) m.HandleFunc("GET /api/files/thumb", s.auth(s.filesThumb))
m.HandleFunc("GET /api/files/thumbs", s.auth(s.thumbsStatus)) m.HandleFunc("GET /api/files/thumbs", s.auth(s.thumbsStatus))
m.HandleFunc("POST /api/files/thumbs/install", s.auth(s.thumbsInstall)) m.HandleFunc("POST /api/files/thumbs/install", s.auth(s.thumbsInstall))
m.HandleFunc("POST /api/files/thumbs/gpu-access", s.auth(s.thumbsGPUAccess)) m.HandleFunc("POST /api/files/thumbs/gpu-access", s.auth(s.thumbsGPUAccess))
m.HandleFunc("GET /api/files/play-caps", s.auth(s.playCapsHandler))
m.HandleFunc("POST /api/files/play", s.auth(s.playStart))
m.HandleFunc("GET /api/files/play/{id}", s.auth(s.playStatus))
m.HandleFunc("DELETE /api/files/play/{id}", s.auth(s.playStop))
m.HandleFunc("POST /api/files/play/{id}/close", s.auth(s.playStop))
m.HandleFunc("GET /api/files/play/{id}/{file}", s.auth(s.playFile))
m.HandleFunc("GET /api/hardware", s.auth(s.hardwareReport)) m.HandleFunc("GET /api/hardware", s.auth(s.hardwareReport))
m.HandleFunc("GET /api/network", s.auth(s.networkOverview)) m.HandleFunc("GET /api/network", s.auth(s.networkOverview))
m.HandleFunc("GET /api/vpn", s.auth(s.vpnReport)) m.HandleFunc("GET /api/vpn", s.auth(s.vpnReport))
@@ -1920,21 +1932,15 @@ func (s *Server) filesRead(w http.ResponseWriter, r *http.Request) {
jsonErr(w, 400, "not a regular file (device, pipe or socket)") jsonErr(w, 400, "not a regular file (device, pipe or socket)")
return return
} }
if in.Broken {
jsonErr(w, 404, "a link to nothing: "+in.Target)
return
}
mx, _ := strconv.ParseInt(r.URL.Query().Get("max"), 10, 64) mx, _ := strconv.ParseInt(r.URL.Query().Get("max"), 10, 64)
if mx > 0 && in.Size > mx { if mx > 0 && in.Size > mx {
jsonErr(w, 413, "file too large") jsonErr(w, 413, "file too large")
return return
} }
f, err := fsys.Open(r.Context(), p)
if err != nil {
fsErr(w, err)
return
}
defer f.Close()
var body io.Reader = f
if mx > 0 { // stat can lie (size 0 on /proc); the cap must hold on the bytes actually sent
body = io.LimitReader(f, mx+1)
}
name := filepath.Base(p) name := filepath.Base(p)
ct := mime.TypeByExtension(filepath.Ext(name)) ct := mime.TypeByExtension(filepath.Ext(name))
if ct == "" { if ct == "" {
@@ -1953,7 +1959,28 @@ func (s *Server) filesRead(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Disposition", "inline; filename*=UTF-8''"+url.PathEscape(name)) w.Header().Set("Content-Disposition", "inline; filename*=UTF-8''"+url.PathEscape(name))
} }
if in.Size > 0 && mx == 0 { if in.Size > 0 && mx == 0 {
w.Header().Set("Content-Length", strconv.FormatInt(in.Size, 10)) // A real file of known size goes through net/http, which speaks Range: a video player
// can seek and a download can resume. /proc files say size 0 and take the plain path.
rs := files.NewSeeker(r.Context(), fsys, p, in.Size)
defer rs.Close()
if _, local := fsys.(files.Local); local { // cheap here, and turns "can't open" into a status rather than a cut-off body
if err := rs.Check(); err != nil {
fsErr(w, err)
return
}
}
http.ServeContent(w, r, name, time.UnixMilli(in.Mtime), rs)
return
}
f, err := fsys.Open(r.Context(), p)
if err != nil {
fsErr(w, err)
return
}
defer f.Close()
var body io.Reader = f
if mx > 0 { // stat can lie (size 0 on /proc); the cap must hold on the bytes actually sent
body = io.LimitReader(f, mx+1)
} }
_, _ = io.Copy(w, body) _, _ = io.Copy(w, body)
} }
@@ -1967,7 +1994,14 @@ func (s *Server) filesWrite(w http.ResponseWriter, r *http.Request) {
if !ok { if !ok {
return return
} }
if err := fsys.Write(r.Context(), p, io.LimitReader(r.Body, 8<<20)); err != nil { // Whole in memory first: the write truncates the file, so a body over the cap must be
// refused before that, not cut short after it.
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 8<<20))
if err != nil {
jsonErr(w, 413, "too large to save here (8 MiB at most)")
return
}
if err := fsys.Write(r.Context(), p, bytes.NewReader(body)); err != nil {
fsErr(w, err) fsErr(w, err)
return return
} }
@@ -2054,6 +2088,63 @@ func (s *Server) filesRename(w http.ResponseWriter, r *http.Request) {
}) })
} }
// filesMove: POST {path, to} — to is the full destination path. A rename when the two
// ends share a filesystem (204); otherwise the copy-and-remove runs as a task (202 with the
// task), since a folder between datasets can take as long as it takes.
func (s *Server) filesMove(w http.ResponseWriter, r *http.Request) {
fsys, ok := s.fsFor(w, r)
if !ok {
return
}
var body struct{ Path, To string }
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&body); err != nil {
jsonErr(w, 400, "bad request")
return
}
from, err := files.Clean(body.Path)
if err != nil {
jsonErr(w, 400, err.Error())
return
}
to, err := files.Clean(body.To)
if err != nil {
jsonErr(w, 400, err.Error())
return
}
if from == to {
jsonErr(w, 400, "it is already there")
return
}
if files.Inside(from, to) {
jsonErr(w, 400, "a folder can't be moved into itself")
return
}
// The kernel answers EXDEV for two paths on different filesystems before it looks for
// the source, so a missing file would otherwise become a task that fails on arrival.
if _, err := fsys.Stat(r.Context(), from); err != nil {
fsErr(w, err)
return
}
err = fsys.Rename(r.Context(), from, to)
if err == nil {
log.Printf("files: move %s -> %s (from %s)", from, to, clientIP(r))
w.WriteHeader(204)
return
}
if !files.IsCrossDevice(err) {
fsErr(w, err)
return
}
cmd := files.MoveCmd(context.Background(), fsys, s.self, from, to)
t, err := s.tasks.Start("files:move", "move "+filepath.Base(from)+" to "+filepath.Dir(to), cmd)
if err != nil {
jsonErr(w, 500, err.Error())
return
}
log.Printf("files: move %s -> %s across filesystems as task %s (from %s)", from, to, t.ID, clientIP(r))
jsonOK(w, 202, t.Snapshot(false))
}
func (s *Server) filesDelete(w http.ResponseWriter, r *http.Request) { func (s *Server) filesDelete(w http.ResponseWriter, r *http.Request) {
s.filesOp(w, r, func(fsys files.FS, body map[string]any, p string) error { s.filesOp(w, r, func(fsys files.FS, body map[string]any, p string) error {
rec, _ := body["recursive"].(bool) rec, _ := body["recursive"].(bool)
+49 -31
View File
@@ -1,7 +1,7 @@
package api package api
// Previews for the Files tab: GET /api/files/thumb hands back one small JPEG of an image or // Previews for the Files tab: GET /api/files/thumb hands back one small JPEG of an image or
// a video, made by internal/thumbs and kept under <data dir>/thumbs/. The work is bounded // a video, made by internal/thumbs and kept in <data dir>/thumbs.db. The work is bounded
// two ways — a handful of ffmpegs at once, and one per file however many rows ask for it — // two ways — a handful of ffmpegs at once, and one per file however many rows ask for it —
// because a folder of a thousand photos asks for every row the moment previews go on. // because a folder of a thousand photos asks for every row the moment previews go on.
// //
@@ -23,8 +23,8 @@ import (
"sync" "sync"
"time" "time"
"hamlab/internal/media"
"hamlab/internal/priv" "hamlab/internal/priv"
"hamlab/internal/thumbs"
) )
const thumbSize = 256 const thumbSize = 256
@@ -32,7 +32,7 @@ const thumbTimeout = 30 * time.Second
const capsTTL = 10 * time.Minute // how long a probe is believed; a refresh after an install or a group change does not wait const capsTTL = 10 * time.Minute // how long a probe is believed; a refresh after an install or a group change does not wait
type thumbSvc struct { type thumbSvc struct {
cache thumbs.Cache cache *media.Cache
self string self string
sem chan struct{} sem chan struct{}
mu sync.Mutex mu sync.Mutex
@@ -42,14 +42,14 @@ type thumbSvc struct {
type capsEntry struct { type capsEntry struct {
mu sync.Mutex mu sync.Mutex
caps thumbs.Caps caps media.Caps
err error err error
at time.Time at time.Time
} }
type thumbCall struct { type thumbCall struct {
done chan struct{} done chan struct{}
res thumbs.Result res media.Result
err error err error
} }
@@ -61,11 +61,11 @@ func newThumbSvc(dataDir, self string) *thumbSvc {
if n > 4 { if n > 4 {
n = 4 n = 4
} }
return &thumbSvc{cache: thumbs.Cache{Dir: filepath.Join(dataDir, "thumbs")}, self: self, sem: make(chan struct{}, n), caps: map[bool]*capsEntry{}, calls: map[string]*thumbCall{}} return &thumbSvc{cache: media.NewCache(filepath.Join(dataDir, "thumbs.db")), self: self, sem: make(chan struct{}, n), caps: map[bool]*capsEntry{}, calls: map[string]*thumbCall{}}
} }
// capsFor is what previews can be made with, probed once and kept for a while. // capsFor is what previews can be made with, probed once and kept for a while.
func (t *thumbSvc) capsFor(ctx context.Context, root, refresh bool) (thumbs.Caps, error) { func (t *thumbSvc) capsFor(ctx context.Context, root, refresh bool) (media.Caps, error) {
t.mu.Lock() t.mu.Lock()
e := t.caps[root] e := t.caps[root]
if e == nil { if e == nil {
@@ -79,9 +79,9 @@ func (t *thumbSvc) capsFor(ctx context.Context, root, refresh bool) (thumbs.Caps
return e.caps, e.err return e.caps, e.err
} }
if root { if root {
e.caps, e.err = thumbs.ProbeRoot(ctx, t.self) e.caps, e.err = media.ProbeRoot(ctx, t.self)
} else { } else {
e.caps, e.err = thumbs.Probe(ctx), nil e.caps, e.err = media.Probe(ctx), nil
} }
e.at = time.Now() e.at = time.Now()
mode := "as " + map[bool]string{false: "the daemon", true: "root"}[root] mode := "as " + map[bool]string{false: "the daemon", true: "root"}[root]
@@ -97,7 +97,7 @@ func (t *thumbSvc) capsFor(ctx context.Context, root, refresh bool) (thumbs.Caps
} }
// describeCaps is the one line that says how previews get made here. // describeCaps is the one line that says how previews get made here.
func describeCaps(c thumbs.Caps) string { func describeCaps(c media.Caps) string {
if len(c.GPUs) == 0 { if len(c.GPUs) == 0 {
return "CPU only" return "CPU only"
} }
@@ -117,7 +117,7 @@ func describeCaps(c thumbs.Caps) string {
} }
// make produces a preview, sharing the work when the same file is asked for twice at once. // make produces a preview, sharing the work when the same file is asked for twice at once.
func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (thumbs.Result, error) { func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (media.Result, error) {
t.mu.Lock() t.mu.Lock()
if c, ok := t.calls[key]; ok { if c, ok := t.calls[key]; ok {
t.mu.Unlock() t.mu.Unlock()
@@ -125,7 +125,7 @@ func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (thumb
case <-c.done: case <-c.done:
return c.res, c.err return c.res, c.err
case <-ctx.Done(): case <-ctx.Done():
return thumbs.Result{}, ctx.Err() return media.Result{}, ctx.Err()
} }
} }
c := &thumbCall{done: make(chan struct{})} c := &thumbCall{done: make(chan struct{})}
@@ -153,11 +153,11 @@ func (t *thumbSvc) make(ctx context.Context, root bool, key, path string) (thumb
c.err = err c.err = err
return c.res, c.err return c.res, c.err
} }
o := thumbs.Opts{Size: thumbSize, Timeout: thumbTimeout} o := media.Opts{Size: thumbSize, Timeout: thumbTimeout}
if root { if root {
c.res, c.err = thumbs.MakeRoot(wctx, t.self, caps, path, o) c.res, c.err = media.MakeRoot(wctx, t.self, caps, path, o)
} else { } else {
c.res, c.err = thumbs.Make(wctx, caps, path, o) c.res, c.err = media.Make(wctx, caps, path, o)
} }
return c.res, c.err return c.res, c.err
} }
@@ -175,13 +175,13 @@ func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
if !ok { if !ok {
return return
} }
if thumbs.Kind(p) == "" { if media.Kind(p) == "" {
jsonErr(w, 415, "no preview for this kind of file") jsonErr(w, 415, "no preview for this kind of file")
return return
} }
root := r.URL.Query().Get("sudo") == "1" root := r.URL.Query().Get("sudo") == "1"
if mt, sz, ok := parseV(r.URL.Query().Get("v")); ok { if mt, sz, ok := parseV(r.URL.Query().Get("v")); ok {
key := thumbs.Key(p, sz, mt) key := media.Key(p, sz, mt)
if r.Header.Get("If-None-Match") == `"`+key+`"` { if r.Header.Get("If-None-Match") == `"`+key+`"` {
w.WriteHeader(304) w.WriteHeader(304)
return return
@@ -199,16 +199,18 @@ func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
jsonErr(w, 400, "not a regular file") jsonErr(w, 400, "not a regular file")
return return
} }
key := thumbs.Key(p, in.Size, in.Mtime) key := media.Key(p, in.Size, in.Mtime)
if s.serveCachedThumb(w, key) { if s.serveCachedThumb(w, key) {
return return
} }
res, err := s.thumbs.make(r.Context(), root, key, p) res, err := s.thumbs.make(r.Context(), root, key, p)
if err != nil { if err != nil {
var de *thumbs.DecodeError var de *media.DecodeError
switch { switch {
case errors.As(err, &de): case errors.As(err, &de):
_ = s.thumbs.cache.PutFailure(key, de.Msg) if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Failure: de.Msg}); err != nil {
log.Printf("thumbs: cache: %v", err)
}
log.Printf("thumbs: %s: %s", p, de.Msg) log.Printf("thumbs: %s: %s", p, de.Msg)
jsonErr(w, 422, de.Msg) jsonErr(w, 422, de.Msg)
case errors.Is(err, context.DeadlineExceeded): case errors.Is(err, context.DeadlineExceeded):
@@ -221,21 +223,26 @@ func (s *Server) filesThumb(w http.ResponseWriter, r *http.Request) {
} }
return return
} }
if err := s.thumbs.cache.Put(key, res.JPEG); err != nil { if err := s.thumbs.cache.Put(key, media.Entry{Path: p, Engine: res.Engine, Codec: res.Codec, Width: res.Width, Height: res.Height, JPEG: res.JPEG}); err != nil {
log.Printf("thumbs: cache %s: %v", key[:8], err) log.Printf("thumbs: cache: %v", err)
} }
writeThumb(w, key, res.JPEG, res.Engine) writeThumb(w, key, res.JPEG, res.Engine)
} }
// serveCachedThumb answers from the cache when it can. A cache that cannot be read is a
// miss: the preview is made again rather than refused.
func (s *Server) serveCachedThumb(w http.ResponseWriter, key string) bool { func (s *Server) serveCachedThumb(w http.ResponseWriter, key string) bool {
data, why, found := s.thumbs.cache.Get(key) e, found, err := s.thumbs.cache.Get(key)
if err != nil {
log.Printf("thumbs: cache: %v", err)
}
switch { switch {
case !found: case !found:
return false return false
case data == nil: case e.Failure != "":
jsonErr(w, 422, why) jsonErr(w, 422, e.Failure)
default: default:
writeThumb(w, key, data, "cache") writeThumb(w, key, e.JPEG, "cache ("+e.Engine+")")
} }
return true return true
} }
@@ -265,13 +272,19 @@ type thumbStatus struct {
Engine string `json:"engine"` // gpu | cpu | "" Engine string `json:"engine"` // gpu | cpu | ""
Summary string `json:"summary"` Summary string `json:"summary"`
FFmpeg string `json:"ffmpeg,omitempty"` FFmpeg string `json:"ffmpeg,omitempty"`
GPUs []thumbs.GPU `json:"gpus"` GPUs []media.GPU `json:"gpus"`
Unusable []thumbs.GPU `json:"unusable,omitempty"` Unusable []media.GPU `json:"unusable,omitempty"`
Error string `json:"error,omitempty"` Error string `json:"error,omitempty"`
Install *thumbInstall `json:"install,omitempty"` // ffmpeg is missing and this can put it on Install *thumbInstall `json:"install,omitempty"` // ffmpeg is missing and this can put it on
GPUFix bool `json:"gpu_fix,omitempty"` // a render node is there but closed to the daemon's user, and this can open it GPUFix bool `json:"gpu_fix,omitempty"` // a render node is there but closed to the daemon's user, and this can open it
User string `json:"user"` User string `json:"user"`
CacheDir string `json:"cache_dir"` Cache thumbCache `json:"cache"`
}
type thumbCache struct {
Path string `json:"path"`
Entries int64 `json:"entries"`
Bytes int64 `json:"bytes"`
} }
type thumbInstall struct { type thumbInstall struct {
@@ -287,7 +300,8 @@ func (s *Server) thumbsStatus(w http.ResponseWriter, r *http.Request) {
return return
} }
caps, err := s.thumbs.capsFor(r.Context(), root, r.URL.Query().Get("refresh") == "1") caps, err := s.thumbs.capsFor(r.Context(), root, r.URL.Query().Get("refresh") == "1")
st := thumbStatus{GPUs: []thumbs.GPU{}, User: s.user, CacheDir: s.thumbs.cache.Dir} st := thumbStatus{GPUs: []media.GPU{}, User: s.user, Cache: thumbCache{Path: s.thumbs.cache.Path}}
st.Cache.Entries, st.Cache.Bytes = s.thumbs.cache.Stats()
if root { if root {
st.User = "root" st.User = "root"
} }
@@ -298,7 +312,7 @@ func (s *Server) thumbsStatus(w http.ResponseWriter, r *http.Request) {
} }
st.FFmpeg, st.GPUs, st.Unusable, st.Error = caps.Version, caps.GPUs, caps.Unusable, caps.Error st.FFmpeg, st.GPUs, st.Unusable, st.Error = caps.Version, caps.GPUs, caps.Unusable, caps.Error
if st.GPUs == nil { if st.GPUs == nil {
st.GPUs = []thumbs.GPU{} st.GPUs = []media.GPU{}
} }
st.OK = caps.Error == "" st.OK = caps.Error == ""
switch { switch {
@@ -389,6 +403,10 @@ func (s *Server) thumbsGPUAccess(w http.ResponseWriter, r *http.Request) {
jsonErr(w, 400, "nothing to do: Ham Lab is root, has no root, or is not under systemd here") jsonErr(w, 400, "nothing to do: Ham Lab is root, has no root, or is not under systemd here")
return return
} }
if t, busy := s.tasks.RunningOf("thumbs:gpu"); busy { // a second run would only collide with the first's restart timer
jsonErr(w, 409, "already being done (task "+t.ID+")")
return
}
full := priv.Wrap("sh", "-c", gpuAccessScript) full := priv.Wrap("sh", "-c", gpuAccessScript)
t, err := s.tasks.Start("thumbs:gpu", "give Ham Lab access to the GPU", exec.Command(full[0], full[1:]...)) t, err := s.tasks.Start("thumbs:gpu", "give Ham Lab access to the GPU", exec.Command(full[0], full[1:]...))
if err != nil { if err != nil {
+3
View File
@@ -286,6 +286,9 @@ func (s *Server) virtInstance(w http.ResponseWriter, r *http.Request) {
jsonErr(w, 400, "bad name") jsonErr(w, 400, "bad name")
return return
} }
if !s.virtReady(w, r) {
return
}
list, err := s.virt.Instances(r.Context()) list, err := s.virt.Instances(r.Context())
if err != nil { if err != nil {
jsonErr(w, 502, err.Error()) jsonErr(w, 502, err.Error())
+9 -1
View File
@@ -265,7 +265,15 @@ func (s *Server) zfsSnapshotDelete(w http.ResponseWriter, r *http.Request) {
return return
} }
if err := s.storage.DestroySnapshot(r.Context(), body.Name, body.Recursive); err != nil { if err := s.storage.DestroySnapshot(r.Context(), body.Name, body.Recursive); err != nil {
jsonErr(w, 500, err.Error()) msg := err.Error()
switch {
case msg == "bad snapshot name":
jsonErr(w, 400, msg)
case strings.Contains(msg, "does not exist"):
jsonErr(w, 404, msg)
default:
jsonErr(w, 500, msg)
}
return return
} }
log.Printf("zfs snapshot %s destroyed by %s%s", body.Name, clientIP(r), map[bool]string{true: " (recursive)", false: ""}[body.Recursive]) log.Printf("zfs snapshot %s destroyed by %s%s", body.Name, clientIP(r), map[bool]string{true: " (recursive)", false: ""}[body.Recursive])
+70 -7
View File
@@ -15,6 +15,7 @@ import (
"io" "io"
"os" "os"
"os/exec" "os/exec"
"os/signal"
"os/user" "os/user"
"path/filepath" "path/filepath"
"sort" "sort"
@@ -23,8 +24,8 @@ import (
"syscall" "syscall"
"time" "time"
"hamlab/internal/media"
"hamlab/internal/priv" "hamlab/internal/priv"
"hamlab/internal/thumbs"
) )
type Entry struct { type Entry struct {
@@ -58,10 +59,12 @@ type FS interface {
List(ctx context.Context, path string) (Listing, error) List(ctx context.Context, path string) (Listing, error)
Stat(ctx context.Context, path string) (Info, error) Stat(ctx context.Context, path string) (Info, error)
Open(ctx context.Context, path string) (io.ReadCloser, error) Open(ctx context.Context, path string) (io.ReadCloser, error)
OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) // Open from a byte offset: a range request, a player seeking
Write(ctx context.Context, path string, r io.Reader) error Write(ctx context.Context, path string, r io.Reader) error
Mkdir(ctx context.Context, path string) error Mkdir(ctx context.Context, path string) error
Touch(ctx context.Context, path string) error Touch(ctx context.Context, path string) error
Rename(ctx context.Context, from, to string) error Rename(ctx context.Context, from, to string) error
Move(ctx context.Context, from, to string, progress io.Writer) error // Rename, or across filesystems a copy and a removal; long, so callers run it as a task
Delete(ctx context.Context, path string, recursive bool) error Delete(ctx context.Context, path string, recursive bool) error
Chmod(ctx context.Context, path string, perm os.FileMode) error Chmod(ctx context.Context, path string, perm os.FileMode) error
} }
@@ -139,7 +142,7 @@ func entryOf(dir, name string, lst os.FileInfo) Entry {
} else if st.IsDir() { } else if st.IsDir() {
e.Type = "dir" // keep dir semantics for navigation; Target says it's a link e.Type = "dir" // keep dir semantics for navigation; Target says it's a link
} else { } else {
e.Size = st.Size() e.Size, e.Mtime = st.Size(), st.ModTime().UnixMilli() // the file's, not the link's
} }
case lst.IsDir(): case lst.IsDir():
e.Type = "dir" e.Type = "dir"
@@ -149,7 +152,7 @@ func entryOf(dir, name string, lst os.FileInfo) Entry {
e.Type = "other" e.Type = "other"
} }
if e.Type == "file" || (e.Type == "link" && !e.Broken) { if e.Type == "file" || (e.Type == "link" && !e.Broken) {
e.Preview = thumbs.Kind(name) e.Preview = media.Kind(name)
} }
return e return e
} }
@@ -229,6 +232,20 @@ func (Local) Open(_ context.Context, path string) (io.ReadCloser, error) {
return os.Open(path) return os.Open(path)
} }
func (l Local) OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) {
f, err := l.Open(ctx, path)
if err != nil {
return nil, err
}
if offset > 0 {
if _, err := f.(*os.File).Seek(offset, io.SeekStart); err != nil {
f.Close()
return nil, err
}
}
return f, nil
}
// Write replaces the file's contents, keeping its mode; new files get 0644 (umask). // Write replaces the file's contents, keeping its mode; new files get 0644 (umask).
func (Local) Write(_ context.Context, path string, r io.Reader) error { func (Local) Write(_ context.Context, path string, r io.Reader) error {
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
@@ -334,7 +351,15 @@ func (r *cmdReader) Close() error {
} }
func (s Sudo) Open(ctx context.Context, path string) (io.ReadCloser, error) { func (s Sudo) Open(ctx context.Context, path string) (io.ReadCloser, error) {
c := s.cmd(ctx, "read", path) return s.open(ctx, "read", path)
}
func (s Sudo) OpenAt(ctx context.Context, path string, offset int64) (io.ReadCloser, error) {
return s.open(ctx, "read", "--offset", strconv.FormatInt(offset, 10), path)
}
func (s Sudo) open(ctx context.Context, args ...string) (io.ReadCloser, error) {
c := s.cmd(ctx, args...)
errb := &strings.Builder{} errb := &strings.Builder{}
c.Stderr = errb c.Stderr = errb
out, err := c.StdoutPipe() out, err := c.StdoutPipe()
@@ -385,6 +410,30 @@ func (s Sudo) Touch(ctx context.Context, path string) error {
func (s Sudo) Rename(ctx context.Context, from, to string) error { func (s Sudo) Rename(ctx context.Context, from, to string) error {
return s.runJSON(ctx, nil, "rename", from, to) return s.runJSON(ctx, nil, "rename", from, to)
} }
// Move as root: the helper does the copying and talks on stdout.
func (s Sudo) Move(ctx context.Context, from, to string, progress io.Writer) error {
c := s.cmd(ctx, "move", from, to)
c.Stdout = progress
var errb strings.Builder
c.Stderr = &errb
if err := c.Run(); err != nil {
if msg := strings.TrimSpace(errb.String()); msg != "" {
return errors.New(msg)
}
return err
}
return nil
}
// MoveCmd is the command a task runs for a long move: this binary's helper, as root when
// the backend is.
func MoveCmd(ctx context.Context, fsys FS, self, from, to string) *exec.Cmd {
if s, ok := fsys.(Sudo); ok {
return s.cmd(ctx, "move", from, to)
}
return exec.CommandContext(ctx, self, "fs", "move", from, to)
}
func (s Sudo) Delete(ctx context.Context, path string, recursive bool) error { func (s Sudo) Delete(ctx context.Context, path string, recursive bool) error {
if recursive { if recursive {
return s.runJSON(ctx, nil, "delete", "-r", path) return s.runJSON(ctx, nil, "delete", "-r", path)
@@ -399,11 +448,16 @@ func (s Sudo) Chmod(ctx context.Context, path string, perm os.FileMode) error {
func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer) int { func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer) int {
fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 } fail := func(err error) int { fmt.Fprintln(stderr, err.Error()); return 1 }
if len(args) < 2 { if len(args) < 2 {
return fail(errors.New("usage: hamlab fs <list|stat|read|write|mkdir|touch|rename|delete|chmod> …")) return fail(errors.New("usage: hamlab fs <list|stat|read|write|mkdir|touch|rename|move|delete|chmod> …"))
} }
ctx := context.Background() ctx := context.Background()
var l Local var l Local
op := args[0] op := args[0]
var offset int64
if op == "read" && len(args) >= 4 && args[1] == "--offset" { // read --offset N <path>
offset, _ = strconv.ParseInt(args[2], 10, 64)
args = append([]string{"read"}, args[3:]...)
}
clean := func(i int) (string, error) { clean := func(i int) (string, error) {
if i >= len(args) { if i >= len(args) {
return "", errors.New("path required") return "", errors.New("path required")
@@ -430,7 +484,7 @@ func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer)
} }
return emit(stdout, out, stderr) return emit(stdout, out, stderr)
case "read": case "read":
f, err := l.Open(ctx, p) f, err := l.OpenAt(ctx, p, offset)
if err != nil { if err != nil {
return fail(err) return fail(err)
} }
@@ -452,7 +506,7 @@ func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer)
return fail(err) return fail(err)
} }
} }
case "rename": case "rename", "move":
from, err := clean(1) from, err := clean(1)
if err != nil { if err != nil {
return fail(err) return fail(err)
@@ -461,6 +515,15 @@ func Helper(args []string, stdin io.Reader, stdout io.Writer, stderr io.Writer)
if err != nil { if err != nil {
return fail(err) return fail(err)
} }
if op == "move" {
// A cancelled task is a SIGTERM: stop copying and take the partial copy away.
mctx, stop := signal.NotifyContext(ctx, syscall.SIGTERM, syscall.SIGINT)
defer stop()
if err := l.Move(mctx, from, to, stdout); err != nil {
return fail(err)
}
return 0
}
if err := l.Rename(ctx, from, to); err != nil { if err := l.Rename(ctx, from, to); err != nil {
return fail(err) return fail(err)
} }
+197
View File
@@ -0,0 +1,197 @@
package files
import (
"context"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"syscall"
)
// IsCrossDevice reports a rename refused because the two ends are on different
// filesystems — on a NAS that is every move between datasets. The Sudo backend only has
// the helper's words for it.
func IsCrossDevice(err error) bool {
return errors.Is(err, syscall.EXDEV) || (err != nil && strings.Contains(err.Error(), "cross-device"))
}
// Inside reports whether to lies within from — a folder cannot be moved into itself.
func Inside(from, to string) bool {
return to == from || strings.HasPrefix(to, strings.TrimSuffix(from, "/")+"/")
}
// Move is Rename, and across filesystems a copy and a removal. The copy is built under a
// hidden temporary name beside where it will land and renamed into place only once it is
// complete, and the source is removed only after that: a failure part-way, or a cancel,
// leaves the source exactly as it was. Progress goes to w, a line at a time, for the task
// output that a long move is watched through.
func (Local) Move(ctx context.Context, from, to string, w io.Writer) error {
if protected[from] {
return errors.New("refusing to move " + from)
}
if Inside(from, to) {
return errors.New("a folder can't be moved into itself")
}
if _, err := os.Lstat(to); err == nil {
return errors.New("destination exists")
}
err := os.Rename(from, to)
if err == nil || !IsCrossDevice(err) {
return err
}
return moveByCopy(ctx, from, to, w)
}
// moveByCopy is the slow path: copy, rename into place, remove the original.
func moveByCopy(ctx context.Context, from, to string, w io.Writer) error {
st, err := os.Lstat(from)
if err != nil {
return err
}
tmp := filepath.Join(filepath.Dir(to), "."+filepath.Base(to)+".hamlab-moving")
_ = os.RemoveAll(tmp) // a previous attempt that died
fmt.Fprintf(w, "%s is on another filesystem: copying, then removing the original\n", filepath.Dir(to))
p := &progress{w: w}
if err := copyPath(ctx, from, tmp, st, p); err != nil {
fmt.Fprintln(w, "copy failed; removing the partial copy, the original is untouched")
_ = os.RemoveAll(tmp)
return err
}
if err := os.Rename(tmp, to); err != nil {
_ = os.RemoveAll(tmp)
return err
}
fmt.Fprintf(w, "copied %s; removing the original\n", p.String())
if err := os.RemoveAll(from); err != nil {
return fmt.Errorf("copied to %s, but the original could not be removed: %w", to, err)
}
fmt.Fprintf(w, "moved %s to %s\n", from, to)
return nil
}
type progress struct {
w io.Writer
files int64
bytes int64
shownFiles int64
shownBytes int64
}
func (p *progress) add(files, bytes int64) {
p.files += files
p.bytes += bytes
if p.files-p.shownFiles >= 500 || p.bytes-p.shownBytes >= 512<<20 {
fmt.Fprintf(p.w, "copying… %s\n", p.String())
p.shownFiles, p.shownBytes = p.files, p.bytes
}
}
func (p *progress) String() string {
return fmt.Sprintf("%d files, %s", p.files, fmtSize(p.bytes))
}
func fmtSize(b int64) string {
const unit = 1024
if b < unit {
return fmt.Sprintf("%d B", b)
}
div, exp := int64(unit), 0
for n := b / unit; n >= unit && exp < 4; n /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f %ciB", float64(b)/float64(div), "KMGTP"[exp])
}
// copyPath copies one entry of any kind, recursively for a directory, keeping mode, times
// and — when running as root — ownership. Devices, pipes and sockets are refused: a copy
// of one is not the thing itself.
func copyPath(ctx context.Context, src, dst string, st os.FileInfo, p *progress) error {
if err := ctx.Err(); err != nil {
return err
}
mode := st.Mode()
switch {
case mode&os.ModeSymlink != 0:
target, err := os.Readlink(src)
if err != nil {
return err
}
if err := os.Symlink(target, dst); err != nil {
return err
}
chown(dst, st)
p.add(1, 0)
return nil
case mode.IsDir():
if err := os.Mkdir(dst, mode.Perm()|0o700); err != nil { // writable while it fills, exact mode after
return err
}
ents, err := os.ReadDir(src)
if err != nil {
return err
}
for _, e := range ents {
lst, err := e.Info()
if err != nil {
return err
}
if err := copyPath(ctx, filepath.Join(src, e.Name()), filepath.Join(dst, e.Name()), lst, p); err != nil {
return err
}
}
chown(dst, st)
if err := os.Chmod(dst, mode.Perm()); err != nil {
return err
}
return os.Chtimes(dst, st.ModTime(), st.ModTime())
case mode.IsRegular():
n, err := copyFile(src, dst, st)
if err != nil {
return err
}
p.add(1, n)
return nil
default:
return fmt.Errorf("%s: can't copy a device, pipe or socket", src)
}
}
func copyFile(src, dst string, st os.FileInfo) (int64, error) {
in, err := os.Open(src)
if err != nil {
return 0, err
}
defer in.Close()
out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL, st.Mode().Perm())
if err != nil {
return 0, err
}
n, err := io.Copy(out, in)
if err != nil {
out.Close()
return n, err
}
if err := out.Close(); err != nil {
return n, err
}
chown(dst, st)
if err := os.Chmod(dst, st.Mode().Perm()); err != nil { // umask has no say
return n, err
}
return n, os.Chtimes(dst, st.ModTime(), st.ModTime())
}
// chown keeps the owner when this process may set one; an ordinary user's copy is the
// user's, which is what a rename would have left anyway.
func chown(dst string, st os.FileInfo) {
if os.Geteuid() != 0 {
return
}
if sys, ok := st.Sys().(*syscall.Stat_t); ok {
_ = os.Lchown(dst, int(sys.Uid), int(sys.Gid))
}
}
+195
View File
@@ -0,0 +1,195 @@
package files
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
"time"
)
func TestInside(t *testing.T) {
for _, c := range []struct {
from, to string
want bool
}{
{"/a/b", "/a/b", true}, {"/a/b", "/a/b/c", true}, {"/a/b", "/a/bc", false}, {"/a/b", "/a", false}, {"/", "/x", true},
} {
if got := Inside(c.from, c.to); got != c.want {
t.Errorf("Inside(%q, %q) = %v, want %v", c.from, c.to, got, c.want)
}
}
}
func TestIsCrossDevice(t *testing.T) {
if !IsCrossDevice(&os.LinkError{Op: "rename", Old: "/a", New: "/b", Err: syscall.EXDEV}) {
t.Fatal("the kernel's EXDEV")
}
if !IsCrossDevice(errors.New("rename /a /b: invalid cross-device link")) {
t.Fatal("the helper's words for it")
}
if IsCrossDevice(errors.New("permission denied")) || IsCrossDevice(nil) {
t.Fatal("anything else is not")
}
}
// The copy path, exercised directly: a tree with nested folders, a symlink, an odd mode
// and old mtimes lands whole under the new name, and the original is gone only then.
func TestMoveByCopy(t *testing.T) {
root := t.TempDir()
src := filepath.Join(root, "src", "album")
if err := os.MkdirAll(filepath.Join(src, "sub", "deep"), 0o755); err != nil {
t.Fatal(err)
}
old := time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC)
write := func(p, s string, mode os.FileMode) {
if err := os.WriteFile(p, []byte(s), mode); err != nil {
t.Fatal(err)
}
_ = os.Chmod(p, mode)
_ = os.Chtimes(p, old, old)
}
write(filepath.Join(src, "a.txt"), "hello", 0o644)
write(filepath.Join(src, "sub", "deep", "b.bin"), strings.Repeat("x", 100000), 0o600)
write(filepath.Join(src, "run.sh"), "#!/bin/sh\n", 0o755)
if err := os.Symlink("a.txt", filepath.Join(src, "link")); err != nil {
t.Fatal(err)
}
_ = os.Chtimes(filepath.Join(src, "sub"), old, old)
dst := filepath.Join(root, "dst", "moved")
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
t.Fatal(err)
}
var out strings.Builder
if err := moveByCopy(context.Background(), src, dst, &out); err != nil {
t.Fatal(err)
}
if _, err := os.Lstat(src); !os.IsNotExist(err) {
t.Fatal("the original is still there")
}
if b, _ := os.ReadFile(filepath.Join(dst, "a.txt")); string(b) != "hello" {
t.Fatal("a.txt")
}
if st, err := os.Stat(filepath.Join(dst, "sub", "deep", "b.bin")); err != nil || st.Size() != 100000 || st.Mode().Perm() != 0o600 || !st.ModTime().Equal(old) {
t.Fatalf("b.bin: %v %v", st, err)
}
if st, _ := os.Stat(filepath.Join(dst, "run.sh")); st.Mode().Perm() != 0o755 {
t.Fatal("mode lost")
}
if target, err := os.Readlink(filepath.Join(dst, "link")); err != nil || target != "a.txt" {
t.Fatal("symlink not copied as a symlink")
}
if st, _ := os.Stat(filepath.Join(dst, "sub")); !st.ModTime().Equal(old) {
t.Fatal("folder mtime lost")
}
if left, _ := filepath.Glob(filepath.Join(root, "dst", ".*hamlab-moving*")); len(left) != 0 {
t.Fatal("temporary copy left behind")
}
if !strings.Contains(out.String(), "moved ") || !strings.Contains(out.String(), "4 files") {
t.Fatalf("progress said: %q", out.String())
}
}
// A failure part-way leaves the original as it was and no partial copy.
func TestMoveByCopyFailsClean(t *testing.T) {
root := t.TempDir()
src := filepath.Join(root, "src")
if err := os.MkdirAll(src, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(src, "a"), []byte("a"), 0o644); err != nil {
t.Fatal(err)
}
if err := syscall.Mkfifo(filepath.Join(src, "pipe"), 0o644); err != nil {
t.Skip("no fifos here")
}
dst := filepath.Join(root, "dst")
err := moveByCopy(context.Background(), src, dst, &strings.Builder{})
if err == nil || !strings.Contains(err.Error(), "pipe") {
t.Fatalf("expected the pipe to be refused, got %v", err)
}
if _, err := os.Stat(filepath.Join(src, "a")); err != nil {
t.Fatal("the original was touched")
}
if _, err := os.Lstat(dst); !os.IsNotExist(err) {
t.Fatal("a partial copy was left")
}
if left, _ := filepath.Glob(filepath.Join(root, ".*hamlab-moving*")); len(left) != 0 {
t.Fatal("temporary copy left behind")
}
// a cancel is the same story
if err := os.Remove(filepath.Join(src, "pipe")); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if err := moveByCopy(ctx, src, dst, &strings.Builder{}); err == nil {
t.Fatal("a cancelled move must fail")
}
if _, err := os.Stat(filepath.Join(src, "a")); err != nil {
t.Fatal("the original was touched by a cancel")
}
}
// Move on one filesystem is a rename: nothing is copied, and the guards hold.
func TestMoveSameFilesystem(t *testing.T) {
root := t.TempDir()
src := filepath.Join(root, "f")
if err := os.WriteFile(src, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
var l Local
var out strings.Builder
if err := l.Move(context.Background(), src, filepath.Join(root, "g"), &out); err != nil {
t.Fatal(err)
}
if out.Len() != 0 {
t.Fatal("a rename has nothing to say")
}
if _, err := os.Stat(filepath.Join(root, "g")); err != nil {
t.Fatal("not renamed")
}
if err := l.Move(context.Background(), root, filepath.Join(root, "inside"), &out); err == nil {
t.Fatal("into itself")
}
if err := l.Move(context.Background(), filepath.Join(root, "g"), filepath.Join(root, "g"), &out); err == nil {
t.Fatal("onto itself")
}
if err := os.WriteFile(filepath.Join(root, "h"), nil, 0o644); err != nil {
t.Fatal(err)
}
if err := l.Move(context.Background(), filepath.Join(root, "g"), filepath.Join(root, "h"), &out); err == nil || !strings.Contains(err.Error(), "exists") {
t.Fatalf("onto an existing file: %v", err)
}
}
// A link to a file carries the file's size and time, so the row and the preview key follow
// the file; a link to a folder is a folder; a link to nothing says so.
func TestEntryOfLinks(t *testing.T) {
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "f"), []byte("12345"), 0o644); err != nil {
t.Fatal(err)
}
old := time.Date(2021, 1, 1, 0, 0, 0, 0, time.UTC)
_ = os.Chtimes(filepath.Join(dir, "f"), old, old)
_ = os.Mkdir(filepath.Join(dir, "d"), 0o755)
for _, l := range [][2]string{{"lf", "f"}, {"ld", "d"}, {"lx", "nowhere"}} {
if err := os.Symlink(l[1], filepath.Join(dir, l[0])); err != nil {
t.Fatal(err)
}
}
ent := func(n string) Entry { st, _ := os.Lstat(filepath.Join(dir, n)); return entryOf(dir, n, st) }
if e := ent("lf"); e.Type != "link" || e.Size != 5 || e.Mtime != old.UnixMilli() || e.Target != "f" || e.Broken {
t.Fatalf("link to file: %+v", e)
}
if e := ent("ld"); e.Type != "dir" || e.Target != "d" {
t.Fatalf("link to dir: %+v", e)
}
if e := ent("lx"); e.Type != "link" || !e.Broken || e.Preview != "" {
t.Fatalf("dangling: %+v", e)
}
}
+89
View File
@@ -0,0 +1,89 @@
package files
import (
"context"
"errors"
"io"
)
// Seeker is an FS file as http.ServeContent wants one — something to seek in — for a file
// whose size is already known from a stat. ServeContent seeks to the end for the size, to
// the start of the range asked for, then reads: so the backend is opened once, at that
// position, and only ever read forward. The Sudo backend's helper is started at the offset
// rather than streamed from the top and skipped.
type Seeker struct {
ctx context.Context
fsys FS
path string
size int64
pos int64 // where the next Read reads from
at int64 // where cur is positioned, when open
cur io.ReadCloser
}
func NewSeeker(ctx context.Context, fsys FS, path string, size int64) *Seeker {
return &Seeker{ctx: ctx, fsys: fsys, path: path, size: size}
}
// Check opens the file now, from the start, so a file that cannot be read is known before
// any headers go out. Seeking costs nothing; a Read from somewhere else reopens there.
func (s *Seeker) Check() error {
if s.cur != nil {
return nil
}
f, err := s.fsys.OpenAt(s.ctx, s.path, 0)
if err != nil {
return err
}
s.cur, s.at = f, 0
return nil
}
func (s *Seeker) Seek(offset int64, whence int) (int64, error) {
var pos int64
switch whence {
case io.SeekStart:
pos = offset
case io.SeekCurrent:
pos = s.pos + offset
case io.SeekEnd:
pos = s.size + offset
default:
return 0, errors.New("bad whence")
}
if pos < 0 {
return 0, errors.New("seek before start")
}
s.pos = pos // the stream, if open, stays where it is until a Read wants somewhere else
return pos, nil
}
func (s *Seeker) Read(p []byte) (int, error) {
if s.pos >= s.size {
return 0, io.EOF
}
if s.cur != nil && s.pos != s.at {
s.cur.Close()
s.cur = nil
}
if s.cur == nil {
f, err := s.fsys.OpenAt(s.ctx, s.path, s.pos)
if err != nil {
return 0, err
}
s.cur, s.at = f, s.pos
}
n, err := s.cur.Read(p)
s.pos += int64(n)
s.at += int64(n)
return n, err
}
func (s *Seeker) Close() error {
if s.cur == nil {
return nil
}
err := s.cur.Close()
s.cur = nil
return err
}
+112
View File
@@ -0,0 +1,112 @@
package files
import (
"context"
"io"
"os"
"path/filepath"
"testing"
)
import "strings"
type stringsBuilder = strings.Builder
type countingFS struct {
Local
opens []int64
}
func (c *countingFS) OpenAt(ctx context.Context, path string, off int64) (io.ReadCloser, error) {
c.opens = append(c.opens, off)
return c.Local.OpenAt(ctx, path, off)
}
// ServeContent's moves — size from the end, seek to the range, read forward — open the
// backend once, at the range; a seek to where the stream already is costs nothing.
func TestSeeker(t *testing.T) {
p := filepath.Join(t.TempDir(), "f")
data := make([]byte, 1000)
for i := range data {
data[i] = byte(i)
}
if err := os.WriteFile(p, data, 0o644); err != nil {
t.Fatal(err)
}
fs := &countingFS{}
s := NewSeeker(context.Background(), fs, p, int64(len(data)))
if n, _ := s.Seek(0, io.SeekEnd); n != 1000 {
t.Fatalf("end = %d", n)
}
if _, err := s.Seek(100, io.SeekStart); err != nil {
t.Fatal(err)
}
buf := make([]byte, 50)
if n, err := io.ReadFull(s, buf); err != nil || n != 50 || buf[0] != 100 || buf[49] != 149 {
t.Fatalf("read at 100: %d %v %v", n, err, buf[:2])
}
if _, err := s.Seek(150, io.SeekStart); err != nil { // where it already is
t.Fatal(err)
}
if n, _ := io.ReadFull(s, buf); n != 50 || buf[0] != 150 {
t.Fatal("read after a no-op seek")
}
if len(fs.opens) != 1 || fs.opens[0] != 100 {
t.Fatalf("opens = %v, want [100]", fs.opens)
}
if _, err := s.Seek(0, io.SeekStart); err != nil {
t.Fatal(err)
}
rest, err := io.ReadAll(s)
if err != nil || len(rest) != 1000 || rest[999] != data[999] {
t.Fatalf("read all from the start: %d %v", len(rest), err)
}
if len(fs.opens) != 2 || fs.opens[1] != 0 {
t.Fatalf("opens = %v, want [100 0]", fs.opens)
}
if n, err := s.Read(buf); n != 0 || err != io.EOF {
t.Fatal("past the end is EOF")
}
if _, err := s.Seek(-1, io.SeekStart); err == nil {
t.Fatal("before the start")
}
if err := s.Close(); err != nil {
t.Fatal(err)
}
// Check opens at the start and keeps that stream for a read from the start
s2 := NewSeeker(context.Background(), fs, p, 1000)
if err := s2.Check(); err != nil {
t.Fatal(err)
}
s2.Seek(0, io.SeekEnd)
s2.Seek(0, io.SeekStart)
if n, _ := io.ReadFull(s2, buf); n != 50 || buf[0] != 0 {
t.Fatal("read from the start after Check")
}
if len(fs.opens) != 3 {
t.Fatalf("Check's open was not reused: %v", fs.opens)
}
s2.Close()
if err := NewSeeker(context.Background(), fs, filepath.Join(t.TempDir(), "nope"), 5).Check(); err == nil {
t.Fatal("Check must report a file that can't be opened")
}
}
// The helper's read takes an offset, for the Sudo backend.
func TestHelperReadOffset(t *testing.T) {
p := filepath.Join(t.TempDir(), "f")
if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil {
t.Fatal(err)
}
var out, errb stringsBuilder
if rc := Helper([]string{"read", "--offset", "7", p}, nil, &out, &errb); rc != 0 {
t.Fatalf("rc %d: %s", rc, errb.String())
}
if out.String() != "789" {
t.Fatalf("got %q", out.String())
}
out.Reset()
if rc := Helper([]string{"read", p}, nil, &out, &errb); rc != 0 || out.String() != "0123456789" {
t.Fatalf("plain read: rc %d %q", rc, out.String())
}
}
+3 -2
View File
@@ -10,6 +10,7 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"hamlab/internal/store"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
@@ -91,7 +92,7 @@ func (m *Manager) SaveExportConfig(c ExportConfig) error {
} }
c.Updated = time.Now() c.Updated = time.Now()
b, _ := json.MarshalIndent(c, "", " ") b, _ := json.MarshalIndent(c, "", " ")
return os.WriteFile(m.exportConfigPath(), b, 0o644) return store.WriteFileAtomic(m.exportConfigPath(), b, 0o644)
} }
type ExportRun struct { type ExportRun struct {
@@ -226,6 +227,6 @@ func (m *Manager) RunExports(ctx context.Context, names []string, w func(string)
} }
run.Seconds = time.Since(start).Seconds() run.Seconds = time.Since(start).Seconds()
b, _ := json.MarshalIndent(run, "", " ") b, _ := json.MarshalIndent(run, "", " ")
_ = os.WriteFile(m.exportRunPath(), b, 0o644) _ = store.WriteFileAtomic(m.exportRunPath(), b, 0o644)
return run return run
} }
-4
View File
@@ -14,7 +14,6 @@ import (
"net/netip" "net/netip"
"os" "os"
"os/exec" "os/exec"
"strings"
"time" "time"
) )
@@ -125,6 +124,3 @@ func dockerPools() []string {
} }
return []string{"172.17.0.0/12", "192.168.0.0/16"} return []string{"172.17.0.0/12", "192.168.0.0/16"}
} }
// String form used in the layout file and on the page.
func prefixString(p netip.Prefix) string { return strings.TrimSpace(p.Masked().String()) }
+1 -1
View File
@@ -130,7 +130,7 @@ func (m *Manager) openVGA(ctx context.Context, client *http.Client, name string,
resp, err := client.Do(req) resp, err := client.Do(req)
if err != nil { if err != nil {
if isPermission(err) { if isPermission(err) {
return nil, errors.New("Ham Lab's account can't reach Incus's socket; Finish setup on the Virtualization tab puts it in incus-admin") return nil, errors.New("this account can't reach Incus's socket; Finish setup on the Virtualization tab puts Ham Lab in incus-admin")
} }
return nil, err return nil, err
} }
+218
View File
@@ -0,0 +1,218 @@
package media
import (
"crypto/sha256"
"encoding/binary"
"encoding/hex"
"encoding/json"
"errors"
"os"
"strconv"
"sync"
"time"
bolt "go.etcd.io/bbolt"
)
// Cache keeps finished previews in one database file under Ham Lab's own data directory —
// thumbs.db, a bbolt key/value store: pure Go, so the binary stays static, and one file
// per machine rather than a tree of hash-named JPEGs. An entry is one file's preview keyed
// by its path, size and mtime, so a file that changes gets a new preview and an unchanged
// one is never decoded twice. A file ffmpeg could not decode is kept the same way, with
// ffmpeg's reason and no picture, so a folder of odd files does not re-run ffmpeg on every
// visit. Nothing expires: a preview is a few kilobytes, and only files somebody actually
// scrolled past are ever made. The file is opened the first time something is stored, so a
// machine nobody browses with previews on never has one.
type Cache struct {
Path string
mu sync.Mutex
db *bolt.DB
err error
}
// Entry is one cached preview, or the remembered reason there is none.
type Entry struct {
Path string `json:"path"`
Engine string `json:"engine,omitempty"`
Codec string `json:"codec,omitempty"`
Width int `json:"width,omitempty"`
Height int `json:"height,omitempty"`
Made time.Time `json:"made"`
Failure string `json:"failure,omitempty"` // set instead of JPEG when ffmpeg refused the file
JPEG []byte `json:"-"`
}
var (
bucketThumbs = []byte("thumbs")
bucketMeta = []byte("meta")
keyCount = []byte("count")
)
// Key identifies one file's preview. "v1" is the preview format: bump it to retire every
// cached preview at once.
func Key(path string, size, mtime int64) string {
h := sha256.Sum256([]byte("v1\x00" + path + "\x00" + strconv.FormatInt(size, 10) + "\x00" + strconv.FormatInt(mtime, 10)))
return hex.EncodeToString(h[:])
}
func NewCache(path string) *Cache { return &Cache{Path: path} }
// open the database once; an open that failed is not retried, and previews still get made
// and served — they are simply not kept, and the log says why.
func (c *Cache) open() (*bolt.DB, error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.db != nil || c.err != nil {
return c.db, c.err
}
db, err := bolt.Open(c.Path, 0o600, &bolt.Options{Timeout: 2 * time.Second})
if err != nil {
c.err = err
return nil, err
}
if err := db.Update(func(tx *bolt.Tx) error {
if _, err := tx.CreateBucketIfNotExists(bucketThumbs); err != nil {
return err
}
_, err := tx.CreateBucketIfNotExists(bucketMeta)
return err
}); err != nil {
db.Close()
c.err = err
return nil, err
}
c.db = db
return db, nil
}
// Close releases the file. Previews after it are made but not kept.
func (c *Cache) Close() error {
c.mu.Lock()
defer c.mu.Unlock()
if c.db == nil {
return nil
}
err := c.db.Close()
c.db, c.err = nil, errors.New("cache closed")
return err
}
// Get returns the entry for a key. A database that does not exist yet is a miss, not a reason
// to make one.
func (c *Cache) Get(key string) (Entry, bool, error) {
c.mu.Lock()
db := c.db
c.mu.Unlock()
if db == nil {
if _, err := os.Stat(c.Path); err != nil {
return Entry{}, false, nil
}
var err error
if db, err = c.open(); err != nil {
return Entry{}, false, err
}
}
var e Entry
found := false
err := db.View(func(tx *bolt.Tx) error {
v := tx.Bucket(bucketThumbs).Get([]byte(key))
if v == nil {
return nil
}
var err error
e, err = decode(v) // copies: a value is only valid inside the transaction
found = err == nil
return err
})
return e, found, err
}
// Put stores an entry, replacing any under the same key.
func (c *Cache) Put(key string, e Entry) error {
if len(e.JPEG) == 0 && e.Failure == "" {
return errors.New("empty preview")
}
if e.Made.IsZero() {
e.Made = time.Now()
}
db, err := c.open()
if err != nil {
return err
}
v, err := encode(e)
if err != nil {
return err
}
return db.Update(func(tx *bolt.Tx) error {
b := tx.Bucket(bucketThumbs)
k := []byte(key)
fresh := b.Get(k) == nil
if err := b.Put(k, v); err != nil {
return err
}
if fresh {
m := tx.Bucket(bucketMeta)
return m.Put(keyCount, be64(count(m)+1))
}
return nil
})
}
// Stats is how many previews are kept and how big the file is.
func (c *Cache) Stats() (entries int64, bytes int64) {
st, err := os.Stat(c.Path)
if err != nil {
return 0, 0
}
bytes = st.Size()
db, err := c.open()
if err != nil {
return 0, bytes
}
_ = db.View(func(tx *bolt.Tx) error { entries = count(tx.Bucket(bucketMeta)); return nil })
return entries, bytes
}
func count(m *bolt.Bucket) int64 {
if v := m.Get(keyCount); len(v) == 8 {
return int64(binary.BigEndian.Uint64(v))
}
return 0
}
func be64(n int64) []byte {
b := make([]byte, 8)
binary.BigEndian.PutUint64(b, uint64(n))
return b
}
// A value is a 4-byte header length, the header as JSON, then the JPEG bytes.
func encode(e Entry) ([]byte, error) {
hdr, err := json.Marshal(e)
if err != nil {
return nil, err
}
v := make([]byte, 4+len(hdr)+len(e.JPEG))
binary.BigEndian.PutUint32(v, uint32(len(hdr)))
copy(v[4:], hdr)
copy(v[4+len(hdr):], e.JPEG)
return v, nil
}
func decode(v []byte) (Entry, error) {
if len(v) < 4 {
return Entry{}, errors.New("short value")
}
n := int(binary.BigEndian.Uint32(v))
if 4+n > len(v) {
return Entry{}, errors.New("bad header length")
}
var e Entry
if err := json.Unmarshal(v[4:4+n], &e); err != nil {
return Entry{}, err
}
if rest := v[4+n:]; len(rest) > 0 {
e.JPEG = append([]byte(nil), rest...)
}
return e, nil
}
+222
View File
@@ -0,0 +1,222 @@
package media
// What to run when a file is to be re-encoded for a browser that cannot play it as it is:
// the codecs and sizes on offer, the output size and bit rate a choice comes to for a
// given source, the ways this machine could do it in the order worth trying, and the
// ffmpeg argument list for one of them. All decisions, no processes, so it can be tested.
import (
"fmt"
"math"
"sort"
"strconv"
)
// Codec is one target a stream may be re-encoded to. "copy" keeps the codec and only
// repackages: for a browser that lacks the container, not the decoder.
type Codec struct {
ID string `json:"id"`
Label string `json:"label"`
}
var Codecs = []Codec{{"h264", "H.264"}, {"hevc", "HEVC"}, {"av1", "AV1"}, {"copy", "Original codec, repackaged"}}
// Quality is a picture height; the bit rate follows from the size the output ends up at.
type Quality struct {
ID string `json:"id"`
Label string `json:"label"`
Height int `json:"height"` // 0: the source's own
}
var Qualities = []Quality{{"original", "Original size", 0}, {"1080p", "1080p", 1080}, {"720p", "720p", 720}, {"480p", "480p", 480}}
func qualityByID(id string) (Quality, bool) {
for _, q := range Qualities {
if q.ID == id {
return q, true
}
}
return Quality{}, false
}
// ValidChoice says whether a codec and quality are ones on offer — checked before any work.
func ValidChoice(codec, quality string) error {
if !codecKnown(codec) {
return fmt.Errorf("unknown codec %q", codec)
}
if _, ok := qualityByID(quality); !ok {
return fmt.Errorf("unknown quality %q", quality)
}
return nil
}
func codecKnown(id string) bool {
for _, c := range Codecs {
if c.ID == id {
return true
}
}
return false
}
// fit is the output: never larger than the source, even dimensions, and a bit rate by the
// size it comes to — a quarter more at 50 fps and up, less for the codecs that need less.
func fit(q Quality, src Info, codec string) (w, h, kbps int) {
w, h = src.Video.Width, src.Video.Height
if q.Height > 0 && h > q.Height {
w = int(math.Round(float64(w) * float64(q.Height) / float64(h)))
h = q.Height
}
w, h = w&^1, h&^1
switch {
case h >= 2160:
kbps = 20000
case h >= 1440:
kbps = 12000
case h >= 1080:
kbps = 8000
case h >= 720:
kbps = 4000
default:
kbps = 2000
}
if src.Video.FPS >= 50 {
kbps = kbps * 5 / 4
}
if codec == "hevc" || codec == "av1" {
kbps = kbps * 6 / 10
}
return w, h, kbps
}
// Recipe is one way to make the stream: the engine's name for the record, and the ffmpeg
// arguments that go before the input, the video filter, and the video codec.
type Recipe struct {
Engine string
Pre []string
VF string
Video []string
}
// recipes are the ways to try, best first: a card on the bus before the integrated GPU;
// on a GPU, decoding on the same card (frames never leave it) before decoding in software
// and uploading; the software encoders last. An empty list means nothing here can.
func recipes(c Caps, codec string, src Info, w, h, kbps int) []Recipe {
if codec == "copy" {
v := []string{"-c:v", "copy"}
if src.Video.Codec == "hevc" {
v = append(v, "-tag:v", "hvc1") // Safari reads hvc1, not hev1
}
return []Recipe{{Engine: "Original codec", Video: v}}
}
var out []Recipe
gpus := append([]GPU(nil), c.GPUs...)
sort.SliceStable(gpus, func(i, j int) bool { return !gpus[i].Integrated && gpus[j].Integrated })
rate := []string{"-b:v", strconv.Itoa(kbps) + "k", "-maxrate", strconv.Itoa(kbps*5/4) + "k", "-bufsize", strconv.Itoa(kbps*5/2) + "k"}
for i := range gpus {
g := &gpus[i]
if g.API != "vaapi" || !has(g.Encodes, codec) {
continue
}
video := append([]string{"-c:v", codec + "_vaapi"}, rate...)
switch codec {
case "h264":
video = append(video, "-profile:v", "high")
case "hevc":
video = append(video, "-tag:v", "hvc1")
}
name := g.Name
if name == "" {
name = g.Device
}
scale := fmt.Sprintf("scale_vaapi=w=%d:h=%d:format=nv12", w, h)
if has(g.Codecs, src.Video.Codec) {
vf := scale
if src.HDR() {
vf = "tonemap_vaapi=format=nv12:t=bt709:p=bt709:m=bt709," + vf
}
out = append(out, Recipe{Engine: name + " (vaapi)", Pre: []string{"-hwaccel", "vaapi", "-hwaccel_device", g.Device, "-hwaccel_output_format", "vaapi"}, VF: vf, Video: video})
}
out = append(out, Recipe{Engine: name + " (vaapi, decoded on the CPU)", Pre: []string{"-init_hw_device", "vaapi=va:" + g.Device, "-filter_hw_device", "va"},
VF: fmt.Sprintf("format=nv12,scale=%d:%d,hwupload", w, h), Video: video})
}
swVF := fmt.Sprintf("scale=%d:%d,format=yuv420p", w, h)
if src.HDR() {
swVF = fmt.Sprintf("zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable,zscale=t=bt709:m=bt709,scale=%d:%d,format=yuv420p", w, h)
}
cap := []string{"-maxrate", strconv.Itoa(kbps*5/4) + "k", "-bufsize", strconv.Itoa(kbps*5/2) + "k"}
switch {
case codec == "h264" && has(c.Encoders, "libx264"):
out = append(out, Recipe{Engine: "CPU (libx264)", VF: swVF, Video: append([]string{"-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-profile:v", "high"}, cap...)})
case codec == "hevc" && has(c.Encoders, "libx265"):
out = append(out, Recipe{Engine: "CPU (libx265)", VF: swVF, Video: append([]string{"-c:v", "libx265", "-preset", "fast", "-crf", "26", "-tag:v", "hvc1"}, cap...)})
case codec == "av1" && has(c.Encoders, "libsvtav1"):
out = append(out, Recipe{Engine: "CPU (SVT-AV1)", VF: swVF, Video: append([]string{"-c:v", "libsvtav1", "-preset", "8", "-crf", "35"}, cap...)})
}
return out
}
// EngineFor is the engine the picker will say a codec runs on, "" when none can.
func EngineFor(c Caps, codec string) string {
if c.Error != "" {
return ""
}
if codec == "copy" {
return "Original codec"
}
r := recipes(c, codec, Info{Video: &Track{Codec: "h264", Width: 1920, Height: 1080}}, 1920, 1080, 8000)
if len(r) == 0 {
return ""
}
return r[0].Engine
}
// hlsArgs is the whole ffmpeg argument list for one recipe: the input, the first video and
// audio streams, the picture through the recipe, sound as AAC unless it already is, a
// keyframe every segment, and an event playlist of fragmented-MP4 segments in dir.
func hlsArgs(dir, in string, src Info, r Recipe, copyCodec bool) []string {
a := []string{"-nostdin", "-hide_banner", "-loglevel", "error", "-y"}
a = append(a, r.Pre...)
a = append(a, "-i", in, "-map", "0:v:0", "-map", "0:a:0?", "-sn", "-dn")
if r.VF != "" {
a = append(a, "-vf", r.VF)
}
a = append(a, r.Video...)
switch {
case src.Audio == nil:
a = append(a, "-an")
case src.Audio.Codec == "aac":
a = append(a, "-c:a", "copy")
default:
a = append(a, "-c:a", "aac", "-b:a", "160k", "-ac", "2")
}
if !copyCodec {
fps := src.Video.FPS
if fps <= 0 {
fps = 30
}
gop := int(math.Round(fps * hlsSeconds))
if gop < 24 {
gop = 24
}
if gop > 300 {
gop = 300
}
a = append(a, "-g", strconv.Itoa(gop), "-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%d)", hlsSeconds))
}
a = append(a, "-f", "hls", "-hls_time", strconv.Itoa(hlsSeconds), "-hls_playlist_type", "event", "-hls_segment_type", "fmp4",
"-hls_fmp4_init_filename", "init.mp4", "-hls_segment_filename", dir+"/seg%05d.m4s", "-hls_flags", "independent_segments+temp_file",
"-progress", dir+"/progress", "-stats_period", "1", dir+"/index.m3u8")
return a
}
const hlsSeconds = 4
func has(list []string, x string) bool {
for _, v := range list {
if v == x {
return true
}
}
return false
}
+128
View File
@@ -0,0 +1,128 @@
package media
import (
"strings"
"testing"
)
func src(codec string, w, h int, fps float64) Info {
return Info{Duration: 100, Video: &Track{Codec: codec, Width: w, Height: h, FPS: fps}, Audio: &Track{Codec: "opus", Channels: 2}}
}
func TestFit(t *testing.T) {
q := func(id string) Quality { x, _ := qualityByID(id); return x }
for _, c := range []struct {
q string
in Info
codec string
w, h, kbs int
}{
{"1080p", src("hevc", 3840, 2160, 60), "h264", 1920, 1080, 10000},
{"original", src("hevc", 3840, 2160, 30), "h264", 3840, 2160, 20000},
{"720p", src("h264", 2292, 960, 60), "h264", 1718, 720, 5000}, // odd width made even
{"1080p", src("h264", 1280, 720, 30), "h264", 1280, 720, 4000}, // never upscaled
{"480p", src("h264", 1920, 1080, 30), "hevc", 852, 480, 1200}, // 853.3 rounds down, then even
{"original", src("av1", 640, 360, 24), "av1", 640, 360, 1200},
} {
w, h, k := fit(q(c.q), c.in, c.codec)
if w != c.w || h != c.h || k != c.kbs {
t.Errorf("%s %s %dx%d@%.0f → %dx%d %dk, want %dx%d %dk", c.q, c.codec, c.in.Video.Width, c.in.Video.Height, c.in.Video.FPS, w, h, k, c.w, c.h, c.kbs)
}
}
}
func gridCaps() Caps {
return Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac", "libsvtav1", "libx264", "libx265"},
GPUs: []GPU{
{API: "vaapi", Device: "/dev/dri/renderD128", Name: "Intel UHD Graphics 750", Integrated: true, Codecs: []string{"h264", "hevc", "vp9"}, Encodes: []string{"h264", "hevc"}},
{API: "vaapi", Device: "/dev/dri/renderD129", Name: "Intel Arc A380", Codecs: []string{"h264", "hevc", "vp9"}, Encodes: []string{"av1", "h264", "hevc"}},
}}
}
func engineNames(r []Recipe) []string {
out := make([]string, len(r))
for i := range r {
out[i] = r[i].Engine
}
return out
}
func TestRecipesOrder(t *testing.T) {
c := gridCaps()
// h264 from hevc: the card first, decoding on it; then the iGPU the same way; then the CPU
got := engineNames(recipes(c, "h264", src("hevc", 3840, 2160, 60), 1920, 1080, 10000))
want := []string{"Intel Arc A380 (vaapi)", "Intel Arc A380 (vaapi, decoded on the CPU)", "Intel UHD Graphics 750 (vaapi)", "Intel UHD Graphics 750 (vaapi, decoded on the CPU)", "CPU (libx264)"}
if strings.Join(got, "|") != strings.Join(want, "|") {
t.Fatalf("got %v", got)
}
// av1 source: neither card decodes it, so no same-card recipe, and only the Arc encodes av1
got = engineNames(recipes(c, "av1", src("av1", 1920, 1080, 30), 1920, 1080, 4800))
want = []string{"Intel Arc A380 (vaapi, decoded on the CPU)", "CPU (SVT-AV1)"}
if strings.Join(got, "|") != strings.Join(want, "|") {
t.Fatalf("av1: got %v", got)
}
// copy is one recipe, no filter, hvc1 for hevc
r := recipes(c, "copy", src("hevc", 1920, 1080, 30), 1920, 1080, 0)
if len(r) != 1 || r[0].VF != "" || !has(r[0].Video, "hvc1") || !has(r[0].Video, "copy") {
t.Fatalf("copy: %+v", r)
}
// a machine with no GPU and no x264: nothing for h264, something for av1
bare := Caps{FFmpeg: "ffmpeg", FFprobe: "ffprobe", Encoders: []string{"aac", "libsvtav1"}}
if len(recipes(bare, "h264", src("h264", 1280, 720, 30), 1280, 720, 4000)) != 0 {
t.Fatal("no x264 means no h264")
}
if EngineFor(bare, "av1") != "CPU (SVT-AV1)" || EngineFor(bare, "h264") != "" || EngineFor(bare, "copy") != "Original codec" {
t.Fatal("engineFor")
}
if EngineFor(c, "hevc") != "Intel Arc A380 (vaapi)" {
t.Fatalf("engineFor on grid: %q", EngineFor(c, "hevc"))
}
// HDR gets tone-mapped on both paths
hdr := src("hevc", 3840, 2160, 60)
hdr.Video.ColorTransfer = "smpte2084"
r = recipes(c, "h264", hdr, 1920, 1080, 10000)
if !strings.HasPrefix(r[0].VF, "tonemap_vaapi") || !strings.Contains(r[len(r)-1].VF, "tonemap=hable") {
t.Fatalf("hdr: %q … %q", r[0].VF, r[len(r)-1].VF)
}
}
func TestHLSArgs(t *testing.T) {
in := src("h264", 1920, 1080, 60)
r := recipes(gridCaps(), "h264", in, 1920, 1080, 10000)[0]
a := strings.Join(hlsArgs("/d", "/v.mkv", in, r, false), " ")
for _, want := range []string{"-hwaccel vaapi -hwaccel_device /dev/dri/renderD129", "-i /v.mkv", "-map 0:v:0 -map 0:a:0?", "-vf scale_vaapi=w=1920:h=1080:format=nv12", "-c:v h264_vaapi -b:v 10000k -maxrate 12500k", "-c:a aac -b:a 160k -ac 2", "-g 240 -force_key_frames expr:gte(t,n_forced*4)", "-hls_segment_type fmp4", "-hls_segment_filename /d/seg%05d.m4s", "/d/index.m3u8"} {
if !strings.Contains(a, want) {
t.Errorf("missing %q in\n%s", want, a)
}
}
// aac stays as it is; no audio means none; copy sets no keyframes
in.Audio.Codec = "aac"
if a := strings.Join(hlsArgs("/d", "/v", in, r, false), " "); !strings.Contains(a, "-c:a copy") {
t.Error("aac should be copied")
}
in.Audio = nil
if a := strings.Join(hlsArgs("/d", "/v", in, r, false), " "); !strings.Contains(a, " -an ") {
t.Error("no audio stream → -an")
}
cp := recipes(gridCaps(), "copy", in, 1920, 1080, 0)[0]
if a := strings.Join(hlsArgs("/d", "/v", in, cp, true), " "); strings.Contains(a, "force_key_frames") || strings.Contains(a, "-vf") {
t.Error("copy must not ask for keyframes or filter")
}
}
func TestParseProgress(t *testing.T) {
b := []byte("frame=10\nout_time_us=1000000\nspeed=2.5x\nprogress=continue\nframe=20\nout_time_us=2500000\nspeed=3.1x\nprogress=continue\n")
s, sp := parseProgress(b)
if s != 2.5 || sp != 3.1 {
t.Fatalf("got %v %v", s, sp)
}
}
func TestValidChoice(t *testing.T) {
if err := ValidChoice("h264", "720p"); err != nil {
t.Fatal(err)
}
if ValidChoice("vp8", "720p") == nil || ValidChoice("h264", "9p") == nil || ValidChoice("", "") == nil {
t.Fatal("bad choices must be refused")
}
}
+188
View File
@@ -0,0 +1,188 @@
package media
// Running one long ffmpeg: as the daemon's user directly, or as root through this binary's
// run-ffmpeg helper when the file needs root to read. The helper is told to finish by its
// standard input closing, which also happens by itself if the daemon goes away — so a
// transcode never outlives the console that asked for it.
import (
"bytes"
"context"
"errors"
"io"
"os"
"os/exec"
"strings"
"sync"
"syscall"
"time"
"hamlab/internal/priv"
)
type ffproc struct {
cmd *exec.Cmd
in io.WriteCloser // the helper's stdin, root mode only
done chan struct{}
err error
tail *tailBuf
}
// startFF launches ffmpeg with args, under nice where there is one.
func startFF(c Caps, self string, root bool, args []string) (*ffproc, error) {
var argv []string
if root {
argv = priv.Wrap(append([]string{self, "run-ffmpeg"}, args...)...)
} else {
argv = append([]string{c.FFmpeg}, args...)
if c.Nice != "" {
argv = append([]string{c.Nice, "-n", "10"}, argv...)
}
}
cmd := exec.Command(argv[0], argv[1:]...)
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
p := &ffproc{cmd: cmd, done: make(chan struct{}), tail: &tailBuf{}}
cmd.Stderr = p.tail
cmd.Stdout = io.Discard
if root {
in, err := cmd.StdinPipe()
if err != nil {
return nil, err
}
p.in = in
}
if err := cmd.Start(); err != nil {
return nil, err
}
go func() {
p.err = cmd.Wait()
close(p.done)
}()
return p, nil
}
// exited reports whether ffmpeg has finished, and how.
func (p *ffproc) exited() (bool, error) {
select {
case <-p.done:
return true, p.err
default:
return false, nil
}
}
// end asks ffmpeg to finish and waits a little for it.
func (p *ffproc) end() {
if done, _ := p.exited(); done {
return
}
if p.in != nil {
p.in.Close()
} else {
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGTERM)
}
select {
case <-p.done:
case <-time.After(5 * time.Second):
_ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGKILL) // a root helper ignores this; it ends on its own within its own grace
select {
case <-p.done:
case <-time.After(10 * time.Second):
}
}
}
// lastWords is what ffmpeg said last, for the record.
func (p *ffproc) lastWords() string { return p.tail.last() }
// tailBuf keeps the last few KB written to it.
type tailBuf struct {
mu sync.Mutex
b []byte
}
const tailKeep = 4096
func (t *tailBuf) Write(b []byte) (int, error) {
t.mu.Lock()
defer t.mu.Unlock()
t.b = append(t.b, b...)
if len(t.b) > tailKeep {
t.b = t.b[len(t.b)-tailKeep:]
}
return len(b), nil
}
func (t *tailBuf) last() string {
t.mu.Lock()
defer t.mu.Unlock()
return lastLine(string(bytes.TrimSpace(t.b)))
}
// RunFFmpegHelper implements `hamlab run-ffmpeg <args…>`: ffmpeg as this user (root, when
// the daemon asked through sudo), finishing when stdin closes. Returns ffmpeg's exit code.
func RunFFmpegHelper(args []string, stdin io.Reader, stderr io.Writer) int {
ffmpeg, err := exec.LookPath("ffmpeg")
if err != nil {
io.WriteString(stderr, "ffmpeg is not installed\n")
return 1
}
argv := []string{ffmpeg}
if nice, err := exec.LookPath("nice"); err == nil {
argv = []string{nice, "-n", "10", ffmpeg}
}
cmd := exec.Command(argv[0], append(argv[1:], args...)...)
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Stderr = stderr
cmd.Stdout = io.Discard
if err := cmd.Start(); err != nil {
io.WriteString(stderr, err.Error()+"\n")
return 1
}
ctx, cancel := context.WithCancel(context.Background())
go func() {
_, _ = io.Copy(io.Discard, stdin) // returns when the daemon lets go
cancel()
}()
done := make(chan error, 1)
go func() { done <- cmd.Wait() }()
var werr error
select {
case werr = <-done:
case <-ctx.Done():
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM)
select {
case werr = <-done:
case <-time.After(5 * time.Second):
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
werr = <-done
}
}
var ee *exec.ExitError
if errors.As(werr, &ee) {
return ee.ExitCode()
}
if werr != nil {
return 1
}
return 0
}
// clearDir empties a folder without removing it.
func clearDir(dir string) error {
ents, err := os.ReadDir(dir)
if err != nil {
return err
}
for _, e := range ents {
if err := os.RemoveAll(dir + "/" + e.Name()); err != nil {
return err
}
}
return nil
}
func hasSegments(playlist string) bool {
b, err := os.ReadFile(playlist)
return err == nil && strings.Contains(string(b), "#EXTINF")
}
+344
View File
@@ -0,0 +1,344 @@
package media
// Playing what the browser cannot play as it is: an HLS stream, re-encoded on demand by
// ffmpeg on the machine's own GPU where it has one it can use, on its CPU otherwise. A
// stream is one file at one codec and size, in a folder of its own under
// <data dir>/transcode/: ffmpeg writes an event playlist and fragmented-MP4 segments there
// as fast as it can, and the player reads them as they land, so the range it can seek in
// grows until the whole file is through — on a GPU, several times faster than watching.
// Nothing is kept: a stream nobody has touched for a while is ended and its folder
// removed, and the folder is emptied when the daemon starts.
import (
"bufio"
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"log"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
"sync"
"time"
)
// Stream is one re-encode, in progress or finished.
type Stream struct {
ID string `json:"id"`
Path string `json:"path"`
Codec string `json:"codec"`
Quality string `json:"quality"`
Engine string `json:"engine"`
Width int `json:"width"`
Height int `json:"height"`
Kbps int `json:"kbps"`
Source Info `json:"source"`
Started time.Time `json:"started"`
Root bool `json:"root"`
key string
dir string
mu sync.Mutex
used time.Time
proc *ffproc
finished bool
failure string
gone bool // forgotten by the Streamer
}
// StreamStatus is what the player asks for while the file is still going through.
type StreamStatus struct {
Running bool `json:"running"`
Done bool `json:"done"`
Error string `json:"error,omitempty"`
Progress float64 `json:"progress"` // seconds of output written
Speed float64 `json:"speed"` // × realtime
Segments int `json:"segments"`
}
// Streamer keeps one machine's streams.
type Streamer struct {
dir string
self string
limit int
mu sync.Mutex
streams map[string]*Stream
}
const streamIdle = 10 * time.Minute
const firstSegmentWait = 40 * time.Second
var streamFile = regexp.MustCompile(`^(index\.m3u8|init\.mp4|seg\d{5}\.m4s)$`)
// NewStreamer empties the folder — a daemon restart orphans every stream — and starts the
// sweeper.
func NewStreamer(dataDir, self string, limit int) *Streamer {
m := &Streamer{dir: filepath.Join(dataDir, "transcode"), self: self, limit: limit, streams: map[string]*Stream{}}
_ = os.MkdirAll(m.dir, 0o755)
_ = clearDir(m.dir)
go func() {
for range time.Tick(30 * time.Second) {
m.Sweep(streamIdle)
}
}()
return m
}
// Open starts a stream, or hands back the one already serving the same request. It
// returns once the first segment is on disk, so the player's first fetch is answered.
func (m *Streamer) Open(ctx context.Context, c Caps, root bool, path, codec, quality string, src Info) (*Stream, error) {
if c.Error != "" {
return nil, errors.New(c.Error)
}
if err := ValidChoice(codec, quality); err != nil {
return nil, err
}
q, _ := qualityByID(quality)
if src.Video == nil {
return nil, &DecodeError{Msg: "no video stream"}
}
if src.Video.Width <= 0 || src.Video.Height <= 0 {
return nil, &DecodeError{Msg: "the picture's size is unknown"}
}
key := fmt.Sprintf("%s|%d|%.0f|%s|%s|%v", path, src.Size, src.Duration, codec, quality, root)
m.mu.Lock()
for _, s := range m.streams {
if s.key == key {
if s.failed() { // not the stream to hand back: let it go and make a fresh one
delete(m.streams, s.ID)
_ = os.RemoveAll(s.dir)
break
}
m.mu.Unlock()
s.touch()
return s, nil
}
}
running := 0
for _, s := range m.streams {
if s.running() {
running++
}
}
if running >= m.limit {
m.mu.Unlock()
return nil, fmt.Errorf("%d videos are already being re-encoded on this machine; one must finish or be closed first", running)
}
s := &Stream{ID: newID(), Path: path, Codec: codec, Quality: quality, Source: src, Started: time.Now(), Root: root, key: key, used: time.Now()}
s.dir = filepath.Join(m.dir, s.ID)
m.streams[s.ID] = s
m.mu.Unlock()
s.Width, s.Height, s.Kbps = fit(q, src, codec)
if codec == "copy" {
s.Width, s.Height, s.Kbps = src.Video.Width, src.Video.Height, int(src.BitRate/1000)
}
rs := recipes(c, codec, src, s.Width, s.Height, s.Kbps)
if len(rs) == 0 {
m.forget(s)
return nil, errors.New("nothing on this machine encodes " + codec)
}
var last string
for _, r := range rs {
if err := os.MkdirAll(s.dir, 0o755); err != nil { // the root helper's ffmpeg writes here too
m.forget(s)
return nil, err
}
_ = clearDir(s.dir)
args := hlsArgs(s.dir, path, src, r, codec == "copy")
p, err := startFF(c, m.self, root, args)
if err != nil {
last = r.Engine + ": " + err.Error()
continue
}
s.mu.Lock()
s.proc, s.Engine = p, r.Engine
s.mu.Unlock()
msg, ok := s.waitFirstSegment(ctx)
if ok {
go s.watch()
log.Printf("media: stream %s: %s → %s %s %dx%d %dk on %s", s.ID, filepath.Base(path), codec, quality, s.Width, s.Height, s.Kbps, r.Engine)
return s, nil
}
p.end()
if ctx.Err() != nil || s.closed() { // the caller left, or somebody closed it meanwhile: not the recipe's fault
m.forget(s)
return nil, errors.New("cancelled")
}
last = r.Engine + ": " + msg
log.Printf("media: stream %s on %s: %s", filepath.Base(path), r.Engine, msg)
}
m.forget(s)
return nil, &DecodeError{Msg: last}
}
// waitFirstSegment watches for the playlist's first segment, or ffmpeg giving up.
func (s *Stream) waitFirstSegment(ctx context.Context) (string, bool) {
deadline := time.Now().Add(firstSegmentWait)
playlist := filepath.Join(s.dir, "index.m3u8")
for {
if hasSegments(playlist) {
return "", true
}
if done, err := s.proc.exited(); done {
msg := s.proc.lastWords()
if msg == "" && err != nil {
msg = err.Error()
}
if msg == "" {
msg = "ffmpeg ended before writing a segment"
}
return msg, false
}
if time.Now().After(deadline) {
return "no segment after " + firstSegmentWait.String(), false
}
select {
case <-ctx.Done():
return ctx.Err().Error(), false
case <-time.After(250 * time.Millisecond):
}
}
}
// watch records how the encode ended.
func (s *Stream) watch() {
<-s.proc.done
s.mu.Lock()
s.finished = true
if s.proc.err != nil {
s.failure = s.proc.lastWords()
if s.failure == "" {
s.failure = s.proc.err.Error()
}
}
s.mu.Unlock()
if s.failure != "" {
log.Printf("media: stream %s ended early: %s", s.ID, s.failure)
}
}
func (m *Streamer) Find(id string) (*Stream, bool) {
m.mu.Lock()
defer m.mu.Unlock()
s, ok := m.streams[id]
return s, ok
}
// Close ends a stream and removes its folder.
func (m *Streamer) Close(id string) {
m.mu.Lock()
s, ok := m.streams[id]
m.mu.Unlock()
if !ok {
return
}
s.mu.Lock()
p := s.proc
s.mu.Unlock()
if p != nil {
p.end()
}
m.forget(s)
}
func (m *Streamer) forget(s *Stream) {
m.mu.Lock()
delete(m.streams, s.ID)
m.mu.Unlock()
s.mu.Lock()
s.gone = true
s.mu.Unlock()
_ = os.RemoveAll(s.dir)
}
func (s *Stream) closed() bool { s.mu.Lock(); defer s.mu.Unlock(); return s.gone }
func (s *Stream) failed() bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.finished && s.failure != ""
}
// Sweep closes what nobody has touched for a while.
func (m *Streamer) Sweep(idle time.Duration) {
m.mu.Lock()
var old []string
for id, s := range m.streams {
s.mu.Lock()
if time.Since(s.used) > idle {
old = append(old, id)
}
s.mu.Unlock()
}
m.mu.Unlock()
for _, id := range old {
log.Printf("media: stream %s idle, closed", id)
m.Close(id)
}
}
// File is the on-disk path of one of the stream's files — the playlist, the init segment
// or a media segment — and nothing else.
func (s *Stream) File(name string) (string, bool) {
if !streamFile.MatchString(name) {
return "", false
}
s.touch()
return filepath.Join(s.dir, name), true
}
func (s *Stream) touch() { s.mu.Lock(); s.used = time.Now(); s.mu.Unlock() }
func (s *Stream) running() bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.proc != nil && !s.finished
}
// Status is read from ffmpeg's own progress file and the playlist.
func (s *Stream) Status() StreamStatus {
s.mu.Lock()
st := StreamStatus{Running: s.proc != nil && !s.finished, Done: s.finished && s.failure == "", Error: s.failure}
s.mu.Unlock()
if b, err := os.ReadFile(filepath.Join(s.dir, "progress")); err == nil {
st.Progress, st.Speed = parseProgress(b)
}
if b, err := os.ReadFile(filepath.Join(s.dir, "index.m3u8")); err == nil {
st.Segments = strings.Count(string(b), "#EXTINF")
}
return st
}
// parseProgress takes the last out_time_us and speed from ffmpeg's -progress output.
func parseProgress(b []byte) (seconds, speed float64) {
sc := bufio.NewScanner(strings.NewReader(string(b)))
for sc.Scan() {
k, v, ok := strings.Cut(sc.Text(), "=")
if !ok {
continue
}
v = strings.TrimSpace(v)
switch k {
case "out_time_us":
if us, err := strconv.ParseInt(v, 10, 64); err == nil && us >= 0 {
seconds = float64(us) / 1e6
}
case "speed":
if x, err := strconv.ParseFloat(strings.TrimSuffix(v, "x"), 64); err == nil {
speed = x
}
}
}
return seconds, speed
}
func newID() string {
b := make([]byte, 8)
_, _ = rand.Read(b)
return hex.EncodeToString(b)
}
+85
View File
@@ -0,0 +1,85 @@
package media
import (
"context"
"os"
"os/exec"
"path/filepath"
"testing"
"time"
)
// The real thing, when ffmpeg is here: a generated clip goes through as H.264 and comes out
// as a playlist of segments, the status follows it to the end, and closing removes it all.
func TestStreamWithFFmpeg(t *testing.T) {
if _, err := exec.LookPath("ffmpeg"); err != nil {
t.Skip("no ffmpeg")
}
c := Probe(context.Background())
if c.Error != "" {
t.Skip(c.Error)
}
if EngineFor(c, "h264") == "" {
t.Skip("nothing here encodes h264")
}
dir := t.TempDir()
clip := filepath.Join(dir, "clip.mp4")
out, err := exec.Command("ffmpeg", "-nostdin", "-v", "error", "-f", "lavfi", "-i", "testsrc=size=640x360:rate=25:duration=9", "-f", "lavfi", "-i", "sine=frequency=440:duration=9",
"-c:v", "mpeg4", "-q:v", "5", "-c:a", "aac", "-shortest", clip).CombinedOutput()
if err != nil {
t.Skipf("could not make a clip: %s", out)
}
src, err := Inspect(context.Background(), c, clip, false)
if err != nil || src.Video == nil || src.Audio == nil || src.Video.Codec != "mpeg4" || src.Duration < 8 {
t.Fatalf("inspect: %+v %v", src, err)
}
m := NewStreamer(dir, "hamlab", 2)
s, err := m.Open(context.Background(), c, false, clip, "h264", "480p", src)
if err != nil {
t.Fatal(err)
}
if s.Width != 640 || s.Height != 360 || s.Engine == "" {
t.Fatalf("stream: %+v", s)
}
if p, ok := s.File("index.m3u8"); !ok {
t.Fatal("no playlist")
} else if !hasSegments(p) {
t.Fatal("Open returned before the first segment")
}
for _, bad := range []string{"../x", "progress", "seg1.m4s", "index.m3u8/", "init.mp4.tmp"} {
if _, ok := s.File(bad); ok {
t.Fatalf("%q must not be served", bad)
}
}
// the same request is the same stream
if again, err := m.Open(context.Background(), c, false, clip, "h264", "480p", src); err != nil || again.ID != s.ID {
t.Fatal("not reused")
}
var st StreamStatus
for i := 0; i < 120; i++ {
st = s.Status()
if st.Done || st.Error != "" {
break
}
time.Sleep(250 * time.Millisecond)
}
if !st.Done || st.Error != "" || st.Segments < 2 || st.Progress < 8 {
t.Fatalf("status at the end: %+v", st)
}
if init, _ := s.File("init.mp4"); !exists(init) {
t.Fatal("no init segment")
}
m.Close(s.ID)
if _, ok := m.Find(s.ID); ok || exists(s.dir) {
t.Fatal("close must forget and remove")
}
// an unplayable source is a DecodeError, and leaves nothing behind
bad := filepath.Join(dir, "bad.mkv")
_ = os.WriteFile(bad, []byte("not a video"), 0o644)
if _, err := Inspect(context.Background(), c, bad, false); err == nil {
t.Fatal("inspect of garbage")
}
if left, _ := os.ReadDir(m.dir); len(left) != 0 {
t.Fatalf("left behind: %v", left)
}
}
+142
View File
@@ -0,0 +1,142 @@
package media
import (
"bytes"
"context"
"encoding/json"
"errors"
"os/exec"
"strconv"
"strings"
"hamlab/internal/priv"
)
// Track is one stream of a media file, as ffprobe reports it.
type Track struct {
Index int `json:"index"`
Type string `json:"type"` // video | audio
Codec string `json:"codec"`
Profile string `json:"profile,omitempty"`
PixFmt string `json:"pix_fmt,omitempty"`
Width int `json:"width,omitempty"`
Height int `json:"height,omitempty"`
FPS float64 `json:"fps,omitempty"`
ColorTransfer string `json:"color_transfer,omitempty"` // smpte2084 / arib-std-b67 mean HDR
Channels int `json:"channels,omitempty"`
AttachedPic bool `json:"attached_pic,omitempty"`
}
// Info is what a file holds: the picture to take, the sound to go with it, and the whole.
type Info struct {
Container string `json:"container"` // "matroska,webm", "mov,mp4,m4a,3gp,3g2,mj2"
Duration float64 `json:"duration"`
BitRate int64 `json:"bit_rate,omitempty"` // bits per second, whole file
Size int64 `json:"size,omitempty"`
Video *Track `json:"video,omitempty"` // the first real picture stream; cover art is not it
Audio *Track `json:"audio,omitempty"` // the first sound stream
}
// HDR reports a picture that needs tone mapping to look right on an SDR screen.
func (i Info) HDR() bool {
return i.Video != nil && (i.Video.ColorTransfer == "smpte2084" || i.Video.ColorTransfer == "arib-std-b67")
}
type probeJSON struct {
Streams []struct {
Index int `json:"index"`
CodecType string `json:"codec_type"`
CodecName string `json:"codec_name"`
Profile string `json:"profile"`
PixFmt string `json:"pix_fmt"`
Width int `json:"width"`
Height int `json:"height"`
RFrameRate string `json:"r_frame_rate"`
AvgFrameRate string `json:"avg_frame_rate"`
ColorTransfer string `json:"color_transfer"`
Channels int `json:"channels"`
Disposition struct {
AttachedPic int `json:"attached_pic"`
} `json:"disposition"`
} `json:"streams"`
Format struct {
FormatName string `json:"format_name"`
Duration string `json:"duration"`
BitRate string `json:"bit_rate"`
Size string `json:"size"`
} `json:"format"`
}
// Inspect asks ffprobe what a file is. As root through sudo when asked, for a file only
// root may read. An error about the file itself is a DecodeError.
func Inspect(ctx context.Context, c Caps, path string, root bool) (Info, error) {
if c.FFprobe == "" {
return Info{}, errors.New("ffprobe is not installed")
}
argv := []string{c.FFprobe, "-v", "error", "-show_entries",
"stream=index,codec_type,codec_name,profile,pix_fmt,width,height,r_frame_rate,avg_frame_rate,color_transfer,channels,disposition:format=format_name,duration,bit_rate,size",
"-of", "json", path}
if root {
argv = priv.Wrap(argv...)
}
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
var errb bytes.Buffer
cmd.Stderr = &errb
out, err := cmd.Output()
if err != nil {
if ctx.Err() != nil {
return Info{}, ctx.Err()
}
msg := lastLine(errb.String())
if msg == "" {
msg = "ffprobe: " + err.Error()
}
return Info{}, &DecodeError{Msg: msg}
}
var pj probeJSON
if err := json.Unmarshal(out, &pj); err != nil {
return Info{}, &DecodeError{Msg: "ffprobe: unreadable answer"}
}
in := Info{Container: pj.Format.FormatName}
in.Duration, _ = strconv.ParseFloat(pj.Format.Duration, 64)
in.BitRate, _ = strconv.ParseInt(pj.Format.BitRate, 10, 64)
in.Size, _ = strconv.ParseInt(pj.Format.Size, 10, 64)
var art *Track
for _, s := range pj.Streams {
st := &Track{Index: s.Index, Type: s.CodecType, Codec: s.CodecName, Profile: s.Profile, PixFmt: s.PixFmt, Width: s.Width, Height: s.Height,
ColorTransfer: s.ColorTransfer, Channels: s.Channels, AttachedPic: s.Disposition.AttachedPic != 0}
st.FPS = fps(s.AvgFrameRate)
if st.FPS == 0 {
st.FPS = fps(s.RFrameRate)
}
switch {
case s.CodecType == "video" && st.AttachedPic:
if art == nil {
art = st
}
case s.CodecType == "video" && in.Video == nil:
in.Video = st
case s.CodecType == "audio" && in.Audio == nil:
in.Audio = st
}
}
if in.Video == nil {
in.Video = art // cover art is the only picture there is
}
return in, nil
}
// fps parses ffprobe's "60000/1001".
func fps(r string) float64 {
n, d, ok := strings.Cut(r, "/")
if !ok {
v, _ := strconv.ParseFloat(r, 64)
return v
}
nf, _ := strconv.ParseFloat(n, 64)
df, _ := strconv.ParseFloat(d, 64)
if df == 0 {
return 0
}
return nf / df
}
@@ -1,4 +1,4 @@
package thumbs package media
import ( import (
"fmt" "fmt"
@@ -11,7 +11,7 @@
// the stream turns out unsupported. A hardware run that fails outright is retried on the // the stream turns out unsupported. A hardware run that fails outright is retried on the
// CPU before the file is called undecodable. Images are always decoded on the CPU: one // CPU before the file is called undecodable. Images are always decoded on the CPU: one
// frame of a JPEG at a reduced DCT size costs less than the trip through a GPU. // frame of a JPEG at a reduced DCT size costs less than the trip through a GPU.
package thumbs package media
import ( import (
"bytes" "bytes"
@@ -39,6 +39,8 @@ type GPU struct {
Device string `json:"device"` // /dev/dri/renderD128; "0" for cuda (its device index) Device string `json:"device"` // /dev/dri/renderD128; "0" for cuda (its device index)
Name string `json:"name,omitempty"` // "Intel Arc A380" Name string `json:"name,omitempty"` // "Intel Arc A380"
Codecs []string `json:"codecs,omitempty"` // ffmpeg codec names it decodes; nil when nothing could say Codecs []string `json:"codecs,omitempty"` // ffmpeg codec names it decodes; nil when nothing could say
Encodes []string `json:"encodes,omitempty"` // ffmpeg codec names it encodes, from vainfo
Integrated bool `json:"integrated,omitempty"` // on the CPU die; a card on the bus is preferred for encoding
Reason string `json:"reason,omitempty"` // Unusable only: why Reason string `json:"reason,omitempty"` // Unusable only: why
Permission bool `json:"permission,omitempty"` // Unusable only: the node exists but this user may not open it Permission bool `json:"permission,omitempty"` // Unusable only: the node exists but this user may not open it
} }
@@ -49,7 +51,8 @@ type Caps struct {
FFprobe string `json:"ffprobe,omitempty"` FFprobe string `json:"ffprobe,omitempty"`
Nice string `json:"nice,omitempty"` Nice string `json:"nice,omitempty"`
Version string `json:"version,omitempty"` Version string `json:"version,omitempty"`
GPUs []GPU `json:"gpus"` // usable, in preference order GPUs []GPU `json:"gpus"` // usable, in node order
Encoders []string `json:"encoders,omitempty"` // the encoders ffmpeg itself was built with that matter here
Unusable []GPU `json:"unusable,omitempty"` // render nodes that exist but cannot be used, with the reason Unusable []GPU `json:"unusable,omitempty"` // render nodes that exist but cannot be used, with the reason
Error string `json:"error,omitempty"` // why nothing can be made at all Error string `json:"error,omitempty"` // why nothing can be made at all
ProbedAt time.Time `json:"probed_at"` ProbedAt time.Time `json:"probed_at"`
@@ -93,6 +96,7 @@ func (e *DecodeError) Error() string { return e.Msg }
var imageExt = set("png jpg jpeg jpe jfif gif webp bmp tif tiff avif heic heif hif jxl svg ico psd tga thm") var imageExt = set("png jpg jpeg jpe jfif gif webp bmp tif tiff avif heic heif hif jxl svg ico psd tga thm")
var videoExt = set("mp4 m4v mkv webm mov avi wmv flv mpg mpeg mpe m2v ts m2ts mts 3gp 3g2 ogv ogm mxf vob asf rm rmvb divx f4v mjpeg mjpg 360 osv insv lrf") var videoExt = set("mp4 m4v mkv webm mov avi wmv flv mpg mpeg mpe m2v ts m2ts mts 3gp 3g2 ogv ogm mxf vob asf rm rmvb divx f4v mjpeg mjpg 360 osv insv lrf")
var audioExt = set("mp3 flac m4a m4b aac ogg oga opus wav wma aiff aif alac ape wv mka")
func set(s string) map[string]bool { func set(s string) map[string]bool {
m := map[string]bool{} m := map[string]bool{}
@@ -102,7 +106,8 @@ func set(s string) map[string]bool {
return m return m
} }
// Kind is "image", "video" or "" for a file name. // Kind is "image", "video", "audio" or "" for a file name. An audio file's preview is its
// cover art, when it carries any.
func Kind(name string) string { func Kind(name string) string {
ext := strings.ToLower(strings.TrimPrefix(filepath.Ext(name), ".")) ext := strings.ToLower(strings.TrimPrefix(filepath.Ext(name), "."))
switch { switch {
@@ -110,6 +115,8 @@ func Kind(name string) string {
return "image" return "image"
case videoExt[ext]: case videoExt[ext]:
return "video" return "video"
case audioExt[ext]:
return "audio"
} }
return "" return ""
} }
@@ -135,13 +142,14 @@ func Probe(ctx context.Context) Caps {
return c return c
} }
c.Version = ffmpegVersion(ctx, c.FFmpeg) c.Version = ffmpegVersion(ctx, c.FFmpeg)
c.Encoders = encoders(ctx, c.FFmpeg)
accels := hwaccels(ctx, c.FFmpeg) accels := hwaccels(ctx, c.FFmpeg)
if accels["vaapi"] { if accels["vaapi"] {
nodes, _ := filepath.Glob("/dev/dri/renderD*") nodes, _ := filepath.Glob("/dev/dri/renderD*")
sort.Strings(nodes) sort.Strings(nodes)
vainfo, _ := exec.LookPath("vainfo") vainfo, _ := exec.LookPath("vainfo")
for _, dev := range nodes { for _, dev := range nodes {
g := GPU{API: "vaapi", Device: dev, Name: metrics.RenderNodeName(dev)} g := GPU{API: "vaapi", Device: dev, Name: metrics.RenderNodeName(dev), Integrated: metrics.RenderNodeIntegrated(dev)}
if err := openable(dev); err != nil { if err := openable(dev); err != nil {
g.Reason, g.Permission = err.Error(), errors.Is(err, os.ErrPermission) g.Reason, g.Permission = err.Error(), errors.Is(err, os.ErrPermission)
c.Unusable = append(c.Unusable, g) c.Unusable = append(c.Unusable, g)
@@ -153,7 +161,7 @@ func Probe(ctx context.Context) Caps {
continue continue
} }
if vainfo != "" { if vainfo != "" {
g.Codecs = vainfoCodecs(ctx, vainfo, dev) g.Codecs, g.Encodes = vainfoCodecs(ctx, vainfo, dev)
} }
c.GPUs = append(c.GPUs, g) c.GPUs = append(c.GPUs, g)
} }
@@ -242,28 +250,61 @@ func hwaccels(ctx context.Context, ffmpeg string) map[string]bool {
return m return m
} }
var vaProfileRe = regexp.MustCompile(`VAProfile([A-Za-z0-9]+?)(?:Simple|Main|High|Profile|Baseline|Constrained|Scc|Advanced|Version|Stereo|Multiview)\w*\s*:\s*VAEntrypointVLD`) var vaProfileRe = regexp.MustCompile(`VAProfile([A-Za-z0-9]+?)(?:Simple|Main|High|Profile|Baseline|Constrained|Scc|Advanced|Version|Stereo|Multiview)\w*\s*:\s*VAEntrypoint(VLD|EncSlice|EncSliceLP)\b`)
// wantedEncoders are the encoders this package may ask ffmpeg for.
var wantedEncoders = set("libx264 libx265 libsvtav1 libaom-av1 libopenh264 h264_vaapi hevc_vaapi av1_vaapi aac")
// encoders lists the wanted encoders this ffmpeg was built with.
func encoders(ctx context.Context, ffmpeg string) []string {
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
out, _ := exec.CommandContext(ctx, ffmpeg, "-hide_banner", "-encoders").Output()
var found []string
for _, l := range strings.Split(string(out), "\n") {
f := strings.Fields(l)
if len(f) >= 2 && wantedEncoders[f[1]] {
found = append(found, f[1])
}
}
sort.Strings(found)
return found
}
// vaCodec maps a VAProfile family to the ffmpeg codec it decodes. // vaCodec maps a VAProfile family to the ffmpeg codec it decodes.
var vaCodec = map[string]string{"H264": "h264", "HEVC": "hevc", "VP9": "vp9", "AV1": "av1", "MPEG2": "mpeg2video", "VC1": "vc1", "VP8": "vp8", "JPEG": "mjpeg", "MPEG4": "mpeg4", "H263": "h263"} var vaCodec = map[string]string{"H264": "h264", "HEVC": "hevc", "VP9": "vp9", "AV1": "av1", "MPEG2": "mpeg2video", "VC1": "vc1", "VP8": "vp8", "JPEG": "mjpeg", "MPEG4": "mpeg4", "H263": "h263"}
func vainfoCodecs(ctx context.Context, vainfo, dev string) []string { func vainfoCodecs(ctx context.Context, vainfo, dev string) (decodes, encodes []string) {
ctx, cancel := context.WithTimeout(ctx, 10*time.Second) ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel() defer cancel()
out, _ := exec.CommandContext(ctx, vainfo, "--display", "drm", "--device", dev).Output() out, _ := exec.CommandContext(ctx, vainfo, "--display", "drm", "--device", dev).Output()
return parseVainfo(string(out)) return parseVainfo(string(out))
} }
// parseVainfo keeps the decode (VLD) profiles and names their codecs the way ffmpeg does. // parseVainfo names the codecs the decode (VLD) and encode (EncSlice) profiles cover,
func parseVainfo(text string) []string { // the way ffmpeg names them.
seen := map[string]bool{} func parseVainfo(text string) (decodes, encodes []string) {
dec, enc := map[string]bool{}, map[string]bool{}
for _, m := range vaProfileRe.FindAllStringSubmatch(text, -1) { for _, m := range vaProfileRe.FindAllStringSubmatch(text, -1) {
if c, ok := vaCodec[m[1]]; ok { c, ok := vaCodec[m[1]]
seen[c] = true if !ok {
continue
}
if m[2] == "VLD" {
dec[c] = true
} else {
enc[c] = true
} }
} }
out := make([]string, 0, len(seen)) return keys(dec), keys(enc)
for c := range seen { }
func keys(m map[string]bool) []string {
if len(m) == 0 {
return nil
}
out := make([]string, 0, len(m))
for c := range m {
out = append(out, c) out = append(out, c)
} }
sort.Strings(out) sort.Strings(out)
@@ -1,4 +1,4 @@
package thumbs package media
import ( import (
"bytes" "bytes"
@@ -41,20 +41,32 @@ func TestParseVainfo(t *testing.T) {
VAProfileH264High : VAEntrypointEncSliceLP VAProfileH264High : VAEntrypointEncSliceLP
VAProfileJPEGBaseline : VAEntrypointVLD VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointVLD VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileHEVCMain444_12 : VAEntrypointVLD VAProfileHEVCMain444_12 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileAV1Profile0 : VAEntrypointEncSliceLP VAProfileAV1Profile0 : VAEntrypointEncSliceLP
` `
got := parseVainfo(text) dec, enc := parseVainfo(text)
want := []string{"h264", "hevc", "mjpeg", "mpeg2video", "vp9"} same := func(got, want []string) bool {
if len(got) != len(want) { if len(got) != len(want) {
t.Fatalf("got %v, want %v", got, want) return false
} }
for i := range want { for i := range want {
if got[i] != want[i] { if got[i] != want[i] {
t.Fatalf("got %v, want %v", got, want) return false
} }
} }
return true
}
if !same(dec, []string{"h264", "hevc", "mjpeg", "mpeg2video", "vp9"}) {
t.Fatalf("decodes = %v", dec)
}
if !same(enc, []string{"av1", "h264", "hevc"}) {
t.Fatalf("encodes = %v", enc)
}
if d, e := parseVainfo(""); d != nil || e != nil {
t.Fatal("nothing from nothing")
}
} }
func TestPickAndEngines(t *testing.T) { func TestPickAndEngines(t *testing.T) {
@@ -107,30 +119,54 @@ func TestAttempts(t *testing.T) {
} }
func TestCache(t *testing.T) { func TestCache(t *testing.T) {
c := Cache{Dir: t.TempDir()} c := NewCache(filepath.Join(t.TempDir(), "thumbs.db"))
k := Key("/a", 1, 2) k := Key("/a", 1, 2)
if _, _, found := c.Get(k); found { if _, found, err := c.Get(k); found || err != nil {
t.Fatal("empty cache found something") t.Fatal("an empty cache found something, or complained")
} }
if err := c.Put(k, []byte("jpeg")); err != nil { if _, err := os.Stat(c.Path); err == nil {
t.Fatal("a miss must not create the database")
}
if n, b := c.Stats(); n != 0 || b != 0 {
t.Fatal("stats of nothing")
}
if err := c.Put(k, Entry{Path: "/a", Engine: "cpu", Codec: "png", Width: 640, Height: 360, JPEG: []byte("jpeg")}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if b, _, found := c.Get(k); !found || string(b) != "jpeg" { e, found, err := c.Get(k)
t.Fatal("put then get") if err != nil || !found || string(e.JPEG) != "jpeg" || e.Engine != "cpu" || e.Width != 640 || e.Path != "/a" || e.Made.IsZero() {
t.Fatalf("put then get: %+v %v %v", e, found, err)
} }
k2 := Key("/b", 1, 2) k2 := Key("/b", 1, 2)
if err := c.PutFailure(k2, "no video stream"); err != nil { if err := c.Put(k2, Entry{Path: "/b", Failure: "no video stream"}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if b, why, found := c.Get(k2); !found || b != nil || why != "no video stream" { if e, found, _ := c.Get(k2); !found || e.JPEG != nil || e.Failure != "no video stream" {
t.Fatal("a failure is remembered with its reason") t.Fatal("a failure is remembered with its reason")
} }
if err := c.Put(k, nil); err == nil { if err := c.Put(k, Entry{}); err == nil {
t.Fatal("an empty preview must not be kept") t.Fatal("an empty preview must not be kept")
} }
if left, _ := filepath.Glob(filepath.Join(c.Dir, "*", ".tmp-*")); len(left) != 0 { // replacing an entry does not count it twice
t.Fatal("temp files left behind") if err := c.Put(k, Entry{Path: "/a", JPEG: []byte("jpeg2")}); err != nil {
t.Fatal(err)
} }
n, b := c.Stats()
if n != 2 || b == 0 {
t.Fatalf("stats = %d entries, %d bytes", n, b)
}
// survives a reopen
if err := c.Close(); err != nil {
t.Fatal(err)
}
c2 := NewCache(c.Path)
if e, found, _ := c2.Get(k); !found || string(e.JPEG) != "jpeg2" {
t.Fatal("not there after reopening")
}
if n, _ := c2.Stats(); n != 2 {
t.Fatal("count lost on reopen")
}
c2.Close()
} }
// The real thing, when ffmpeg is here: a PNG and a generated video each come back as a // The real thing, when ffmpeg is here: a PNG and a generated video each come back as a
+11
View File
@@ -264,6 +264,17 @@ func RenderNodeName(node string) string {
return gpuName(dev, ueventValue(dev+"/uevent", "PCI_SLOT_NAME"), driver) return gpuName(dev, ueventValue(dev+"/uevent", "PCI_SLOT_NAME"), driver)
} }
// RenderNodeIntegrated reports a render node whose GPU sits on the CPU die (PCI root bus 0,
// or no PCI bus at all), as opposed to a card on the bus.
func RenderNodeIntegrated(node string) bool {
dev, err := filepath.EvalSymlinks("/sys/class/drm/" + filepath.Base(node) + "/device")
if err != nil {
return true
}
pci := ueventValue(dev+"/uevent", "PCI_SLOT_NAME")
return pci == "" || strings.HasPrefix(pci, "0000:00:")
}
var gpuVendors = map[string]string{"0x8086": "Intel", "0x1002": "AMD", "0x10de": "NVIDIA", "0x1af4": "Virtio", "0x106b": "Apple"} var gpuVendors = map[string]string{"0x8086": "Intel", "0x1002": "AMD", "0x10de": "NVIDIA", "0x1af4": "Virtio", "0x106b": "Apple"}
var ( var (
+9 -2
View File
@@ -146,7 +146,7 @@ func (m *Manager) prune() {
} }
var finished []*Task var finished []*Task
for _, t := range m.tasks { for _, t := range m.tasks {
if !t.Running { if !t.running() {
finished = append(finished, t) finished = append(finished, t)
} }
} }
@@ -184,13 +184,20 @@ func (m *Manager) RunningOf(kind string) (*Task, bool) {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
for _, t := range m.tasks { for _, t := range m.tasks {
if t.Kind == kind && t.Running { if t.Kind == kind && t.running() {
return t, true return t, true
} }
} }
return nil, false return nil, false
} }
// running reads the flag the waiter goroutine clears, under the task's own lock.
func (t *Task) running() bool {
t.mu.Lock()
defer t.mu.Unlock()
return t.Running
}
func (t *Task) Snapshot(withOutput bool) Snapshot { func (t *Task) Snapshot(withOutput bool) Snapshot {
t.mu.Lock() t.mu.Lock()
defer t.mu.Unlock() defer t.mu.Unlock()
-78
View File
@@ -1,78 +0,0 @@
package thumbs
import (
"crypto/sha256"
"encoding/hex"
"errors"
"os"
"path/filepath"
"strconv"
)
// Cache keeps finished previews on disk, under Ham Lab's own data directory: one JPEG per
// (path, size, mtime), so a file that changes gets a new preview and an unchanged one is
// never decoded twice. A file ffmpeg could not decode is remembered the same way, with
// ffmpeg's reason beside it, so a folder of odd files does not re-run ffmpeg on every
// visit. Nothing expires: a preview is a few kilobytes, and only files somebody actually
// scrolled past are ever made.
type Cache struct{ Dir string }
// Key identifies one file's preview. "v1" is the preview format: bump it to retire every
// cached preview at once.
func Key(path string, size, mtime int64) string {
h := sha256.Sum256([]byte("v1\x00" + path + "\x00" + strconv.FormatInt(size, 10) + "\x00" + strconv.FormatInt(mtime, 10)))
return hex.EncodeToString(h[:])
}
func (c Cache) file(key, ext string) string { return filepath.Join(c.Dir, key[:2], key+ext) }
// Get returns the preview, or the remembered reason it could not be made.
func (c Cache) Get(key string) (jpeg []byte, failure string, found bool) {
if len(key) < 2 {
return nil, "", false
}
if b, err := os.ReadFile(c.file(key, ".jpg")); err == nil && len(b) > 0 {
return b, "", true
}
if b, err := os.ReadFile(c.file(key, ".err")); err == nil {
return nil, string(b), true
}
return nil, "", false
}
// Put keeps a preview; PutFailure keeps the reason there is none.
func (c Cache) Put(key string, jpeg []byte) error {
if len(jpeg) == 0 {
return errors.New("empty preview")
}
return c.write(c.file(key, ".jpg"), jpeg)
}
func (c Cache) PutFailure(key, reason string) error {
return c.write(c.file(key, ".err"), []byte(reason))
}
// write lands the file whole or not at all: a reader never sees a half-written JPEG.
func (c Cache) write(dst string, b []byte) error {
if err := os.MkdirAll(filepath.Dir(dst), 0o700); err != nil {
return err
}
tmp, err := os.CreateTemp(filepath.Dir(dst), ".tmp-*")
if err != nil {
return err
}
if _, err := tmp.Write(b); err != nil {
tmp.Close()
os.Remove(tmp.Name())
return err
}
if err := tmp.Close(); err != nil {
os.Remove(tmp.Name())
return err
}
if err := os.Rename(tmp.Name(), dst); err != nil {
os.Remove(tmp.Name())
return err
}
return nil
}
+7
View File
@@ -11,6 +11,7 @@
"@xterm/addon-fit": "^0.10.0", "@xterm/addon-fit": "^0.10.0",
"@xterm/addon-web-links": "^0.11.0", "@xterm/addon-web-links": "^0.11.0",
"@xterm/xterm": "^5.5.0", "@xterm/xterm": "^5.5.0",
"hls.js": "^1.7.3",
"qrcode-generator": "^2.0.4", "qrcode-generator": "^2.0.4",
"react": "^18.3.1", "react": "^18.3.1",
"react-dom": "^18.3.1" "react-dom": "^18.3.1"
@@ -1426,6 +1427,12 @@
"node": ">=6.9.0" "node": ">=6.9.0"
} }
}, },
"node_modules/hls.js": {
"version": "1.7.3",
"resolved": "https://registry.npmjs.org/hls.js/-/hls.js-1.7.3.tgz",
"integrity": "sha512-MsPlx6yVW4Qv4C7mEVou4/gk/5cN2dVxOTnrNjPSmyH2f0Ln+/UkG10Ax1PE6OvrmTVPKgQYozTtBaPOuCzq7A==",
"license": "Apache-2.0"
},
"node_modules/js-tokens": { "node_modules/js-tokens": {
"version": "4.0.0", "version": "4.0.0",
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz", "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz",
+2 -1
View File
@@ -7,12 +7,13 @@
"dev": "vite", "dev": "vite",
"build": "tsc --noEmit && vite build", "build": "tsc --noEmit && vite build",
"icons": "node scripts/icons.mjs", "icons": "node scripts/icons.mjs",
"test": "node scripts/test-graphcols.mjs && node scripts/test-css-imports.mjs" "test": "node scripts/test-graphcols.mjs && node scripts/test-css-imports.mjs && node scripts/test-resolve-path.mjs"
}, },
"dependencies": { "dependencies": {
"@xterm/addon-fit": "^0.10.0", "@xterm/addon-fit": "^0.10.0",
"@xterm/addon-web-links": "^0.11.0", "@xterm/addon-web-links": "^0.11.0",
"@xterm/xterm": "^5.5.0", "@xterm/xterm": "^5.5.0",
"hls.js": "^1.7.3",
"qrcode-generator": "^2.0.4", "qrcode-generator": "^2.0.4",
"react": "^18.3.1", "react": "^18.3.1",
"react-dom": "^18.3.1" "react-dom": "^18.3.1"
+16
View File
@@ -0,0 +1,16 @@
// node scripts/test-resolve-path.mjs — the path box resolves like a shell: relative to here,
// . and .. applied, trailing slashes dropped, never anything but an absolute path.
import { readFileSync } from 'node:fs'
const src = readFileSync(new URL('../src/Files.tsx', import.meta.url), 'utf8')
const m = src.match(/export function resolvePath[\s\S]*?\n}\n/)
if (!m) { console.error('resolvePath not found'); process.exit(1) }
const resolvePath = new Function(m[0].replace('export function', 'return function').replace(/: string\[\]/g, '').replace(/: string/g, '') + ' ')()
const cases = [
['/tank/media', '/x', '/tank/media'], ['/tank/media/', '/x', '/tank/media'], ['', '/here', '/here'], [' ', '/here', '/here'],
['sub', '/here', '/here/sub'], ['./sub/', '/here', '/here/sub'], ['..', '/a/b', '/a'], ['../..', '/a/b', '/'], ['../../..', '/a', '/'],
['/', '/a', '/'], ['a//b', '/', '/a/b'], ['/tank/./x/../y', '/', '/tank/y'], ['sp ace #1', '/d', '/d/sp ace #1'],
]
let bad = 0
for (const [input, here, want] of cases) { const got = resolvePath(input, here); if (got !== want) { bad++; console.error(`resolvePath(${JSON.stringify(input)}, ${here}) = ${got}, want ${want}`) } }
if (bad) process.exit(1)
console.log(`resolve path: ok (${cases.length} cases)`)
+147 -14
View File
@@ -1,11 +1,25 @@
import { DragEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react' import { DragEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
import { api, FileEntry, FileInfo, ThumbStatus } from './api' import { api, FileEntry, FileInfo, TaskInfo, ThumbStatus } from './api'
import { fmtBytes } from './fmt' import { fmtBytes } from './fmt'
import { Btn, Loading, Spinner, askText, askYes, setUnsaved, toast, useEscape } from './ui' import { Btn, Loading, Spinner, askText, askYes, setUnsaved, toast, useEscape } from './ui'
import { useHashSel } from './hashsel' import { useHashSel } from './hashsel'
import TaskOutput from './TaskOutput' import TaskOutput from './TaskOutput'
import Player from './Player'
const join = (dir: string, name: string) => (dir === '/' ? '/' + name : dir + '/' + name) const join = (dir: string, name: string) => (dir === '/' ? '/' + name : dir + '/' + name)
// A typed path, made absolute: relative to the folder being looked at, "." and ".." applied,
// trailing slashes dropped — what a shell would make of it.
export function resolvePath(input: string, here: string): string {
const raw = input.trim()
if (!raw) return here
const parts = (raw.startsWith('/') ? [] : here.split('/')).concat(raw.split('/'))
const out: string[] = []
for (const p of parts) {
if (p === '' || p === '.') continue
if (p === '..') out.pop(); else out.push(p)
}
return '/' + out.join('/')
}
const fmtTime = (ms: number) => { const d = new Date(ms); const now = new Date(); return d.toDateString() === now.toDateString() ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) : d.toLocaleDateString(undefined, { year: d.getFullYear() === now.getFullYear() ? undefined : 'numeric', month: 'short', day: '2-digit' }) } const fmtTime = (ms: number) => { const d = new Date(ms); const now = new Date(); return d.toDateString() === now.toDateString() ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) : d.toLocaleDateString(undefined, { year: d.getFullYear() === now.getFullYear() ? undefined : 'numeric', month: 'short', day: '2-digit' }) }
const isImage = (n: string) => /\.(png|jpe?g|gif|webp|svg|bmp|ico|avif)$/i.test(n) const isImage = (n: string) => /\.(png|jpe?g|gif|webp|svg|bmp|ico|avif)$/i.test(n)
const shellQuote = (s: string) => "'" + s.replace(/'/g, "'\\''") + "'" const shellQuote = (s: string) => "'" + s.replace(/'/g, "'\\''") + "'"
@@ -17,6 +31,7 @@ function icon(e: FileEntry): string {
if (e.type === 'other') return '⚙️' if (e.type === 'other') return '⚙️'
if (e.preview === 'image' || isImage(e.name)) return '🖼️' if (e.preview === 'image' || isImage(e.name)) return '🖼️'
if (e.preview === 'video') return '🎞️' if (e.preview === 'video') return '🎞️'
if (e.preview === 'audio') return '🎵'
if (/\.(zip|gz|bz2|xz|zst|tar|tgz|7z|rar)$/i.test(e.name)) return '🗜️' if (/\.(zip|gz|bz2|xz|zst|tar|tgz|7z|rar)$/i.test(e.name)) return '🗜️'
if (/\.(sh|bash|zsh|py|go|js|ts|tsx|rb|pl|php|rs|c|h|cpp|java|lua)$/i.test(e.name) || e.mode.includes('x')) return '📜' if (/\.(sh|bash|zsh|py|go|js|ts|tsx|rb|pl|php|rs|c|h|cpp|java|lua)$/i.test(e.name) || e.mode.includes('x')) return '📜'
if (/\.(log|txt|md|json|ya?ml|toml|ini|conf|cfg|env|service|timer)$/i.test(e.name)) return '📄' if (/\.(log|txt|md|json|ya?ml|toml|ini|conf|cfg|env|service|timer)$/i.test(e.name)) return '📄'
@@ -25,6 +40,95 @@ function icon(e: FileEntry): string {
type Sort = 'name' | 'size' | 'mtime' type Sort = 'name' | 'size' | 'mtime'
const PREVIEWS_KEY = 'hamlab.files.previews' const PREVIEWS_KEY = 'hamlab.files.previews'
const byName = (a: FileEntry, b: FileEntry) => a.name.localeCompare(b.name, undefined, { numeric: true, sensitivity: 'base' })
// A name is one path segment. ("../x" and "a/b" would quietly act somewhere else.)
const okName = (n?: string | null): n is string => { const t = n?.trim(); if (!t) return false; if (t.includes('/') || t === '.' || t === '..') { toast('a name can\'t contain / or be . or .. — use Move… to put it elsewhere', 'err'); return false } return true }
// ---- move: walk to the destination, then "Move here" ----
function MovePicker({ item, from, sudo, hidden, onClose, onMoved }: { item: FileEntry; from: string; sudo: boolean; hidden: boolean; onClose: () => void; onMoved: (task?: TaskInfo) => void }) {
const src = join(from, item.name)
const isFolder = item.type === 'dir' && !item.target
const [hist, setHist] = useState<string[]>([from]) // where the picker has been, for back and forward
const [at, setAt] = useState(0)
const dir = hist[at]
const [entries, setEntries] = useState<FileEntry[] | null>(null)
const [err, setErr] = useState('')
const [pathInput, setPathInput] = useState(from)
const [busy, setBusy] = useState(false)
const load = useCallback(async (p: string) => {
setEntries(null); setErr('')
try { const l = await api.files(p, sudo); setEntries(l.entries) } catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setEntries([]) }
}, [sudo])
useEffect(() => { setPathInput(dir); load(dir) }, [dir, load])
useEscape(onClose)
const go = (input: string) => {
const p = resolvePath(input, dir)
if (p === dir) return
setHist((h) => [...h.slice(0, at + 1), p]); setAt(at + 1)
}
const up = () => { if (dir !== '/') go(dir.slice(0, dir.lastIndexOf('/')) || '/') }
const folders = (entries ?? []).filter((e) => e.type === 'dir' && (hidden || !e.name.startsWith('.'))).sort(byName)
const crumbs = dir === '/' ? [] : dir.slice(1).split('/')
const dest = join(dir, item.name)
const why = dir === from ? 'it is already here'
: isFolder && (dir === src || dir.startsWith(src + '/')) ? 'a folder can\'t move into itself'
: (entries ?? []).some((e) => e.name === item.name) ? `something called ${item.name} is already there` : ''
const move = async () => {
setBusy(true)
try {
const t = await api.fileMove(src, dest, sudo)
if (t) toast(`${dir} is on another filesystem — copying ${item.name} there, then removing it here`, 'info')
else toast(`moved ${item.name} to ${dir}`)
onMoved(t)
} catch (e) { toast(`move — ${e instanceof Error ? e.message : 'failed'}`, 'err'); setBusy(false) }
}
const mkdir = async () => {
const n = await askText('New folder name')
if (!okName(n)) return
try { await api.fileMkdir(join(dir, n.trim()), sudo); go(join(dir, n.trim())) } catch (e) { toast(`create folder — ${e instanceof Error ? e.message : 'failed'}`, 'err') }
}
return (
<div className="modal-back" onClick={onClose}>
<div className="modal mover" onClick={(e) => e.stopPropagation()}>
<h3>Move {isFolder ? 'folder ' : ''}<span className="mono">{item.name}</span></h3>
<div className="mover-nav">
<button className="icon-btn" title="back" disabled={at === 0} onClick={() => setAt(at - 1)}>‹</button>
<button className="icon-btn" title="forward" disabled={at >= hist.length - 1} onClick={() => setAt(at + 1)}>›</button>
<button className="icon-btn" title="up one level" disabled={dir === '/'} onClick={up}>↑</button>
<form className="crumbs" onSubmit={(e) => { e.preventDefault(); go(pathInput) }}>
<button type="button" className="crumb" onClick={() => go('/')}>/</button>
{crumbs.map((c, i) => <button type="button" key={i} className="crumb" onClick={() => go('/' + crumbs.slice(0, i + 1).join('/'))}>{c}</button>)}
<input className="path-input" value={pathInput} onChange={(e) => setPathInput(e.target.value)} onBlur={() => setPathInput(dir)} spellCheck={false} autoCapitalize="off" autoCorrect="off" />
</form>
</div>
<div className="mover-list">
{entries === null && <Loading label="Listing…" />}
{err && <div className="error" style={{ margin: 10 }}>{err}</div>}
{folders.map((e) => {
const self = dir === from && e.name === item.name
return (
<div key={e.name} className={'mover-row' + (self ? ' self' : '')} onClick={() => { if (!self) go(join(dir, e.name)) }}>
<span className="ficon">{e.target ? '🔗' : '📁'}</span>
<span className="fname">{e.name}</span>
{self && <span className="muted small">this is what's moving</span>}
</div>
)
})}
{entries !== null && folders.length === 0 && !err && <div className="muted small" style={{ padding: 12 }}>No folders here — “Move here” puts it in this one.</div>}
</div>
<div className="modal-foot">
<div className="svc-actions">
<button className="btn" onClick={mkdir}>+ folder</button>
<span className="spacer" />
<button className="btn" onClick={onClose}>Cancel</button>
<Btn className="btn on" disabled={!!why || entries === null} busy={busy} onClick={move}>Move here</Btn>
</div>
<div className="muted small mono mover-dest">{why || `→ ${dest}`}</div>
</div>
</div>
</div>
)
}
// ---- previews ---- // ---- previews ----
@@ -43,7 +147,7 @@ class Near {
private forget(el: Element) { this.cbs.delete(el); this.obs.unobserve(el) } private forget(el: Element) { this.cbs.delete(el); this.obs.unobserve(el) }
} }
function Thumb({ url, kind, near, fallback }: { url: string; kind: 'image' | 'video'; near: Near | null; fallback: string }) { function Thumb({ url, kind, near, fallback }: { url: string; kind: 'image' | 'video' | 'audio'; near: Near | null; fallback: string }) {
const ref = useRef<HTMLSpanElement>(null) const ref = useRef<HTMLSpanElement>(null)
const [show, setShow] = useState(false) const [show, setShow] = useState(false)
const [failed, setFailed] = useState('') const [failed, setFailed] = useState('')
@@ -100,11 +204,12 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
const [orig, setOrig] = useState('') const [orig, setOrig] = useState('')
const [err, setErr] = useState('') const [err, setErr] = useState('')
const [saving, setSaving] = useState(false) const [saving, setSaving] = useState(false)
const [cantPlay, setCantPlay] = useState(false) // the browser tried the file as it is and gave up
const name = path.split('/').pop() ?? path const name = path.split('/').pop() ?? path
const dirty = text !== null && text !== orig const dirty = text !== null && text !== orig
useEffect(() => { useEffect(() => {
setInfo(null); setText(null); setErr('') setInfo(null); setText(null); setErr(''); setCantPlay(false)
let stop = false let stop = false
;(async () => { ;(async () => {
try { try {
@@ -162,8 +267,16 @@ function Viewer({ path, sudo, onClose, onChanged }: { path: string; sudo: boolea
<textarea className="editor file-editor" value={text} spellCheck={false} onChange={(e) => setText(e.target.value)} /> <textarea className="editor file-editor" value={text} spellCheck={false} onChange={(e) => setText(e.target.value)} />
)} )}
{info && text === null && isImage(name) && <div className="preview"><img src={api.fileReadURL(path, sudo)} alt={name} /></div>} {info && text === null && isImage(name) && <div className="preview"><img src={api.fileReadURL(path, sudo)} alt={name} /></div>}
{info && text === null && !isImage(name) && !err && ( {/* Video: the browser's own decoder first, and a re-encode on the machine when that fails. Audio as it is. */}
<div className="notice">{info.type === 'dir' ? 'This is a directory.' : info.size > MAX_EDIT ? `Too large to edit here (${fmtBytes(info.size)}). Download it instead.` : 'Binary file. Download it instead.'}</div> {info && text === null && !isImage(name) && info.preview === 'video' && <Player path={path} sudo={sudo} name={name} />}
{info && text === null && !isImage(name) && info.preview === 'audio' && !cantPlay && (
<div className="preview audio"><div className="audio-box"><div className="ficon">🎵</div><div className="mono small">{name}</div><audio controls preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => setCantPlay(true)} /></div></div>
)}
{info && text === null && !isImage(name) && info.preview === 'audio' && cantPlay && (
<div className="notice">This browser can't play {name} as it is — the codec isn't one it decodes. Download it instead.</div>
)}
{info && text === null && !isImage(name) && !info.preview && !err && (
<div className="notice">{info.type === 'dir' ? 'This is a directory.' : info.type === 'other' ? 'This is a device, pipe or socket — nothing to show.' : info.broken ? `This link points at nothing (${info.target}).` : info.size > MAX_EDIT ? `Too large to edit here (${fmtBytes(info.size)}). Download it instead.` : 'Binary file. Download it instead.'}</div>
)} )}
</div> </div>
) )
@@ -184,6 +297,8 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
const [pathInput, setPathInput] = useState(path) const [pathInput, setPathInput] = useState(path)
const [open, setOpen] = useState<string | null>(null) const [open, setOpen] = useState<string | null>(null)
const [menu, setMenu] = useState<string | null>(null) const [menu, setMenu] = useState<string | null>(null)
const [moving, setMoving] = useState<FileEntry | null>(null)
const [moveTask, setMoveTask] = useState<TaskInfo | null>(null)
const [upload, setUpload] = useState<{ name: string; pct: number; n: number; of: number } | null>(null) const [upload, setUpload] = useState<{ name: string; pct: number; n: number; of: number } | null>(null)
const [drag, setDrag] = useState(false) const [drag, setDrag] = useState(false)
const fileInput = useRef<HTMLInputElement>(null) const fileInput = useRef<HTMLInputElement>(null)
@@ -192,7 +307,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
const [thumbs, setThumbs] = useState<ThumbStatus | null>(null) const [thumbs, setThumbs] = useState<ThumbStatus | null>(null)
const [listEl, setListEl] = useState<HTMLDivElement | null>(null) const [listEl, setListEl] = useState<HTMLDivElement | null>(null)
const near = useMemo(() => (listEl && previews ? new Near(listEl) : null), [listEl, previews]) const near = useMemo(() => (listEl && previews ? new Near(listEl) : null), [listEl, previews])
const readThumbs = useCallback((refresh = false) => { api.thumbStatus(sudo, refresh).then(setThumbs).catch((e) => setThumbs({ ok: false, engine: '', summary: '', gpus: [], error: e instanceof Error ? e.message : 'failed', user: '', cache_dir: '' })) }, [sudo]) const readThumbs = useCallback((refresh = false) => { api.thumbStatus(sudo, refresh).then(setThumbs).catch((e) => setThumbs({ ok: false, engine: '', summary: '', gpus: [], error: e instanceof Error ? e.message : 'failed', user: '', cache: { path: '', entries: 0, bytes: 0 } })) }, [sudo])
useEffect(() => { try { localStorage.setItem(PREVIEWS_KEY, previews ? '1' : '0') } catch { /* private mode */ } }, [previews]) useEffect(() => { try { localStorage.setItem(PREVIEWS_KEY, previews ? '1' : '0') } catch { /* private mode */ } }, [previews])
useEffect(() => { if (previews) readThumbs(); else setThumbs(null) }, [previews, readThumbs]) useEffect(() => { if (previews) readThumbs(); else setThumbs(null) }, [previews, readThumbs])
@@ -204,15 +319,22 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
sessionStorage.setItem('hamlab.files.path', p) // only a folder that opened is worth coming back to sessionStorage.setItem('hamlab.files.path', p) // only a folder that opened is worth coming back to
} catch (e) { } catch (e) {
const msg = e instanceof Error ? e.message : 'failed' const msg = e instanceof Error ? e.message : 'failed'
if (/not a directory/i.test(msg) && p !== '/') { // a file was typed: open it, from its folder
setPath(p.slice(0, p.lastIndexOf('/')) || '/'); setOpen(p); return
}
if (/no such file/i.test(msg) && p !== '/' && !sessionStorage.getItem('hamlab.files.lost')) { // gone since last time (another machine, a deleted folder)
sessionStorage.setItem('hamlab.files.lost', '1'); toast(`${p} isn't here — showing /`, 'info'); setPath('/'); return
}
setErr(msg); setEntries([]) setErr(msg); setEntries([])
} }
sessionStorage.removeItem('hamlab.files.lost')
}, [path, sudo]) }, [path, sudo])
useEffect(() => { setEntries(null); setPathInput(path); setMenu(null); load(path, sudo); sessionStorage.setItem('hamlab.files.sudo', sudo ? '1' : '0') }, [path, sudo, load]) useEffect(() => { setEntries(null); setPathInput(path); setMenu(null); load(path, sudo); sessionStorage.setItem('hamlab.files.sudo', sudo ? '1' : '0') }, [path, sudo, load])
useEffect(() => { const h = () => setMenu(null); window.addEventListener('click', h); return () => window.removeEventListener('click', h) }, []) useEffect(() => { const h = () => setMenu(null); window.addEventListener('click', h); return () => window.removeEventListener('click', h) }, [])
useEscape(() => setMenu(null), menu !== null) useEscape(() => setMenu(null), menu !== null)
const go = (p: string) => { setOpen(null); setPath(p || '/') } const go = (p: string) => { setOpen(null); setPath(resolvePath(p, path)) }
const crumbs = path === '/' ? [] : path.slice(1).split('/') const crumbs = path === '/' ? [] : path.slice(1).split('/')
const shown = (entries ?? []).filter((e) => (hidden || !e.name.startsWith('.')) && (!q || e.name.toLowerCase().includes(q.toLowerCase()))) const shown = (entries ?? []).filter((e) => (hidden || !e.name.startsWith('.')) && (!q || e.name.toLowerCase().includes(q.toLowerCase())))
@@ -227,12 +349,10 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
const act = async (label: string, fn: () => Promise<unknown>): Promise<boolean> => { const act = async (label: string, fn: () => Promise<unknown>): Promise<boolean> => {
try { await fn(); toast(label); await load(); return true } catch (e) { toast(`${label} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); return false } try { await fn(); toast(label); await load(); return true } catch (e) { toast(`${label} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); return false }
} }
// A name is one path segment. ("../x" and "a/b" would quietly act somewhere else.)
const okName = (n?: string | null): n is string => { const t = n?.trim(); if (!t) return false; if (t.includes('/') || t === '.' || t === '..') { toast('a name can\'t contain / or be . or .. — use Move… to put it elsewhere', 'err'); return false } return true }
const mkdir = async () => { const n = await askText('New folder name'); if (okName(n)) act(`created ${n.trim()}`, () => api.fileMkdir(join(path, n.trim()), sudo)) } const mkdir = async () => { const n = await askText('New folder name'); if (okName(n)) act(`created ${n.trim()}`, () => api.fileMkdir(join(path, n.trim()), sudo)) }
const touch = async () => { const n = await askText('New file name'); if (okName(n) && await act(`created ${n.trim()}`, () => api.fileTouch(join(path, n.trim()), sudo))) setOpen(join(path, n.trim())) } // opened only once it exists const touch = async () => { const n = await askText('New file name'); if (okName(n) && await act(`created ${n.trim()}`, () => api.fileTouch(join(path, n.trim()), sudo))) setOpen(join(path, n.trim())) } // opened only once it exists
const rename = async (e: FileEntry) => { const n = await askText('Rename to', e.name); if (okName(n) && n.trim() !== e.name) act(`renamed to ${n.trim()}`, () => api.fileRename(join(path, e.name), join(path, n.trim()), sudo)) } const rename = async (e: FileEntry) => { const n = await askText('Rename to', e.name); if (okName(n) && n.trim() !== e.name) act(`renamed to ${n.trim()}`, () => api.fileRename(join(path, e.name), join(path, n.trim()), sudo)) }
const move = async (e: FileEntry) => { const n = await askText('Move to (absolute path)', join(path, e.name)); if (n?.trim() && n.trim() !== join(path, e.name)) act(`moved to ${n.trim()}`, () => api.fileRename(join(path, e.name), n.trim(), sudo)) } const move = (e: FileEntry) => setMoving(e)
const chmod = async (e: FileEntry) => { const n = await askText(`Permissions for ${e.name} (octal)`, e.perm); if (n?.trim() && n.trim() !== e.perm) act(`chmod ${n.trim()} ${e.name}`, () => api.fileChmod(join(path, e.name), n.trim(), sudo)) } const chmod = async (e: FileEntry) => { const n = await askText(`Permissions for ${e.name} (octal)`, e.perm); if (n?.trim() && n.trim() !== e.perm) act(`chmod ${n.trim()} ${e.name}`, () => api.fileChmod(join(path, e.name), n.trim(), sudo)) }
const del = async (e: FileEntry) => { const del = async (e: FileEntry) => {
const full = join(path, e.name) const full = join(path, e.name)
@@ -248,6 +368,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
// ---- upload ---- // ---- upload ----
const uploadFiles = async (files: FileList | File[]) => { const uploadFiles = async (files: FileList | File[]) => {
const list = Array.from(files) const list = Array.from(files)
let done = 0
for (let i = 0; i < list.length; i++) { for (let i = 0; i < list.length; i++) {
const f = list[i] const f = list[i]
let overwrite = false let overwrite = false
@@ -258,13 +379,22 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
setUpload({ name: f.name, pct: 0, n: i + 1, of: list.length }) setUpload({ name: f.name, pct: 0, n: i + 1, of: list.length })
try { try {
await api.fileUpload(path, f, sudo, (pct) => setUpload({ name: f.name, pct, n: i + 1, of: list.length }), overwrite) await api.fileUpload(path, f, sudo, (pct) => setUpload({ name: f.name, pct, n: i + 1, of: list.length }), overwrite)
done++
} catch (e) { toast(`upload ${f.name} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); break } } catch (e) { toast(`upload ${f.name} — ${e instanceof Error ? e.message : 'failed'}`, 'err'); break }
} }
setUpload(null) setUpload(null)
toast(list.length === 1 ? `uploaded ${list[0].name}` : `uploaded ${list.length} files`) if (done === 1) toast(`uploaded ${list.length === 1 ? list[0].name : '1 file'}`)
load() else if (done > 1) toast(`uploaded ${done} files`)
if (done > 0) load()
}
// A dropped folder arrives as a nameless 0-byte "file"; the browser gives no way to read it here.
const onDrop = (e: DragEvent) => {
e.preventDefault(); setDrag(false)
const items = Array.from(e.dataTransfer.items ?? [])
const files = Array.from(e.dataTransfer.files).filter((_, i) => !(items[i]?.webkitGetAsEntry?.()?.isDirectory))
if (files.length < e.dataTransfer.files.length) toast('folders can\'t be dropped — drop the files inside, or upload a zip', 'err')
if (files.length) uploadFiles(files)
} }
const onDrop = (e: DragEvent) => { e.preventDefault(); setDrag(false); if (e.dataTransfer.files.length) uploadFiles(e.dataTransfer.files) }
if (open) return <div className="files"><Viewer path={open} sudo={sudo} onClose={() => setOpen(null)} onChanged={() => load()} /></div> if (open) return <div className="files"><Viewer path={open} sudo={sudo} onClose={() => setOpen(null)} onChanged={() => load()} /></div>
@@ -279,7 +409,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
<div className="files-tools"> <div className="files-tools">
<button className={'btn' + (sudo ? ' on' : '')} title="browse as root" onClick={() => setSudo(!sudo)}>{sudo ? 'root' : 'sudo'}</button> <button className={'btn' + (sudo ? ' on' : '')} title="browse as root" onClick={() => setSudo(!sudo)}>{sudo ? 'root' : 'sudo'}</button>
<button className={'btn' + (hidden ? ' on' : '')} title="show hidden files" onClick={() => setHidden(!hidden)}>.hidden</button> <button className={'btn' + (hidden ? ' on' : '')} title="show hidden files" onClick={() => setHidden(!hidden)}>.hidden</button>
<button className={'btn' + (previews ? ' on' : '')} title={'thumbnails for images and videos' + (thumbs ? ` — ${thumbs.summary}` : '')} onClick={() => setPreviews(!previews)}>previews</button> <button className={'btn' + (previews ? ' on' : '')} title={'thumbnails for images and videos' + (thumbs ? ` — ${thumbs.summary}` + (thumbs.cache.entries ? ` · ${thumbs.cache.entries} kept (${fmtBytes(thumbs.cache.bytes)})` : '') : '')} onClick={() => setPreviews(!previews)}>previews</button>
<select className="btn" value={sort} onChange={(e) => setSort(e.target.value as Sort)}><option value="name">name</option><option value="size">size</option><option value="mtime">modified</option></select> <select className="btn" value={sort} onChange={(e) => setSort(e.target.value as Sort)}><option value="name">name</option><option value="size">size</option><option value="mtime">modified</option></select>
<input className="filter" placeholder="filter" value={q} onChange={(e) => setQ(e.target.value)} /> <input className="filter" placeholder="filter" value={q} onChange={(e) => setQ(e.target.value)} />
<button className="btn" onClick={mkdir}>+ folder</button> <button className="btn" onClick={mkdir}>+ folder</button>
@@ -292,6 +422,8 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
</div> </div>
{previews && <ThumbNote st={thumbs} sudo={sudo} onChanged={() => readThumbs(true)} />} {previews && <ThumbNote st={thumbs} sudo={sudo} onChanged={() => readThumbs(true)} />}
{moveTask && <TaskOutput id={moveTask.id} onDone={(t) => { load(); if (t.exit_code === 0 && !t.error) toast(`${moveTask.title}: done`); else toast(`${moveTask.title}: failed`, 'err') }} />}
{moveTask && <div className="thumb-note"><button className="btn" onClick={() => setMoveTask(null)}>Hide output</button></div>}
{upload && <div className="upload-bar"><Spinner /> {upload.of > 1 ? `${upload.n}/${upload.of} · ` : ''}{upload.name} · {upload.pct}%<div className="meter"><div className="fill" style={{ width: `${upload.pct}%` }} /></div></div>} {upload && <div className="upload-bar"><Spinner /> {upload.of > 1 ? `${upload.n}/${upload.of} · ` : ''}{upload.name} · {upload.pct}%<div className="meter"><div className="fill" style={{ width: `${upload.pct}%` }} /></div></div>}
{err && <div className="error" style={{ padding: '6px 12px' }}>{err}{/permission denied/i.test(err) && !sudo && <> · <button className="link" onClick={() => setSudo(true)}>retry as root</button></>}</div>} {err && <div className="error" style={{ padding: '6px 12px' }}>{err}{/permission denied/i.test(err) && !sudo && <> · <button className="link" onClick={() => setSudo(true)}>retry as root</button></>}</div>}
@@ -330,6 +462,7 @@ export default function Files({ onOpenInTerminal }: { onOpenInTerminal?: (cmd: s
{entries !== null && shown.length === 0 && !err && <div className="muted" style={{ padding: 12 }}>{q ? 'Nothing matches.' : 'Empty folder.'}{!hidden && entries.some((e) => e.name.startsWith('.')) ? ' (hidden files not shown)' : ''}</div>} {entries !== null && shown.length === 0 && !err && <div className="muted" style={{ padding: 12 }}>{q ? 'Nothing matches.' : 'Empty folder.'}{!hidden && entries.some((e) => e.name.startsWith('.')) ? ' (hidden files not shown)' : ''}</div>}
</div> </div>
{drag && <div className="drop-hint">Drop to upload into {path}</div>} {drag && <div className="drop-hint">Drop to upload into {path}</div>}
{moving && <MovePicker item={moving} from={path} sudo={sudo} hidden={hidden} onClose={() => setMoving(null)} onMoved={(t) => { setMoving(null); if (t) setMoveTask(t); load() }} />}
</div> </div>
) )
} }
+147
View File
@@ -0,0 +1,147 @@
// A video in the Files viewer. First the file as it is, handed to the browser's own
// decoder; when that fails — Safari and an MKV, HEVC without a hardware decoder — a
// picker: codec and size, and the machine re-encodes it to HLS on its own GPU (or CPU),
// which plays natively on Safari and through hls.js everywhere else. The same picker is
// one tap away even when the file plays, for a 36 Mbps capture over a slow link.
import { useCallback, useEffect, useRef, useState } from 'react'
import { api, apiPath, PlayCaps, PlayStatus, PlayStream } from './api'
import { Btn, Spinner } from './ui'
type Mode = 'native' | 'pick' | 'hls'
const CODEC_KEY = 'hamlab.play.codec'
const QUALITY_KEY = 'hamlab.play.quality'
const remember = (k: string, v: string) => { try { localStorage.setItem(k, v) } catch { /* private mode */ } }
const recall = (k: string, d: string) => { try { return localStorage.getItem(k) || d } catch { return d } }
const fmtMbps = (bps?: number) => (bps ? `${(bps / 1e6).toFixed(bps >= 10e6 ? 0 : 1)} Mbps` : '')
const describe = (s: PlayStream) => {
const v = s.source.video
const src = v ? `${v.codec}${v.profile ? ' ' + v.profile : ''} ${v.width}×${v.height}${v.fps ? ` ${Math.round(v.fps)} fps` : ''}${s.source.bit_rate ? ` ${fmtMbps(s.source.bit_rate)}` : ''} in ${s.source.container.split(',')[0]}` : s.source.container
const out = s.codec === 'copy' ? 'repackaged' : `${s.codec === 'h264' ? 'H.264' : s.codec.toUpperCase()} ${s.width}×${s.height} ${(s.kbps / 1000).toFixed(s.kbps >= 10000 ? 0 : 1)} Mbps`
return `${src} → ${out} on ${s.engine}`
}
export default function Player({ path, sudo, name }: { path: string; sudo: boolean; name: string }) {
const [mode, setMode] = useState<Mode>('native')
const [why, setWhy] = useState('') // what brought the picker up
const [caps, setCaps] = useState<PlayCaps | null>(null)
const [codec, setCodec] = useState(() => recall(CODEC_KEY, 'h264'))
const [quality, setQuality] = useState(() => recall(QUALITY_KEY, '1080p'))
const [stream, setStream] = useState<{ s: PlayStream; url: string } | null>(null)
const [status, setStatus] = useState<PlayStatus | null>(null)
const [err, setErr] = useState('')
const video = useRef<HTMLVideoElement>(null)
const streamRef = useRef(stream)
streamRef.current = stream
useEffect(() => { setMode('native'); setWhy(''); setStream(null); setStatus(null); setErr('') }, [path, sudo])
useEffect(() => { remember(CODEC_KEY, codec) }, [codec])
useEffect(() => { remember(QUALITY_KEY, quality) }, [quality])
useEffect(() => { if (mode === 'pick' && !caps) api.playCaps(sudo).then(setCaps).catch((e) => setCaps({ codecs: [], qualities: [], error: e instanceof Error ? e.message : 'failed' })) }, [mode, caps, sudo])
// a stream is the machine's work: let go of it when this player does — and when the page
// itself goes, by a beacon, the one request a browser still sends on its way out
const stop = useCallback(() => { const s = streamRef.current; if (s) { api.playStop(s.s.id).catch(() => { /* gone already */ }); setStream(null); setStatus(null) } }, [])
useEffect(() => () => stop(), [stop])
useEffect(() => {
const h = () => { const s = streamRef.current; if (s) navigator.sendBeacon(apiPath(`/api/files/play/${s.s.id}/close`)) }
window.addEventListener('pagehide', h)
return () => window.removeEventListener('pagehide', h)
}, [])
const start = async () => {
setErr('')
try {
stop()
const r = await api.playStart(path, codec, quality, sudo)
setStream({ s: r.stream, url: r.url }); setMode('hls')
} catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
}
// feed the stream to the element: Safari plays HLS itself, the rest through hls.js
useEffect(() => {
const v = video.current
if (mode !== 'hls' || !stream || !v) return
const url = apiPath(stream.url)
let hls: { destroy: () => void } | null = null
let gone = false
;(async () => {
if (v.canPlayType('application/vnd.apple.mpegurl')) {
v.src = url
} else {
const { default: Hls } = await import('hls.js')
if (gone) return
if (!Hls.isSupported()) { setErr('this browser has neither HLS nor MediaSource — nothing can play here'); return }
const h = new Hls({ maxBufferLength: 60, backBufferLength: 30 })
hls = h
h.on(Hls.Events.ERROR, (_e, d) => { if (d.fatal) setErr(`playback failed: ${d.details}`) })
h.loadSource(url)
h.attachMedia(v)
}
v.play().catch(() => { /* needs a tap on some phones */ })
})()
return () => { gone = true; hls?.destroy() }
}, [mode, stream])
// follow the re-encode while it runs
useEffect(() => {
if (mode !== 'hls' || !stream) return
let stopPolling = false
const tick = async () => { try { const st = await api.playStatus(stream.s.id); if (!stopPolling) { setStatus(st); if (st.running) setTimeout(tick, 2000) } } catch { /* ended */ } }
tick()
return () => { stopPolling = true }
}, [mode, stream])
const pick = (reason: string) => { setWhy(reason); setMode('pick') }
const codecs = caps?.codecs ?? []
return (
<div className="player">
{mode === 'native' && (
<>
<div className="play-bar"><span>as it is</span><span className="spacer" /><button className="btn" onClick={() => pick('')}>Re-encode…</button></div>
<div className="preview"><video controls playsInline preload="metadata" src={api.fileReadURL(path, sudo)} onError={() => pick(`This browser can't play ${name} as it is — the container or codec isn't one it decodes.`)} /></div>
</>
)}
{mode === 'pick' && (
<div className="play-pick notice">
<div>{why || `Re-encode ${name} on this machine and play that instead.`}</div>
{caps?.error && <div className="error" style={{ marginTop: 8 }}>{caps.error}</div>}
{!caps && !err && <div className="muted small" style={{ marginTop: 8 }}><Spinner /> asking the machine what it can encode…</div>}
{caps && !caps.error && (
<div className="row">
<label>codec
<select value={codec} onChange={(e) => setCodec(e.target.value)}>
{codecs.map((c) => <option key={c.id} value={c.id} disabled={!c.engine}>{c.label}{c.engine ? ` — ${c.engine}` : ' — not on this machine'}</option>)}
</select>
</label>
<label>size
<select value={quality} onChange={(e) => setQuality(e.target.value)} disabled={codec === 'copy'}>
{caps.qualities.map((q) => <option key={q.id} value={q.id}>{q.label}</option>)}
</select>
</label>
<Btn className="btn on" disabled={!codecs.find((c) => c.id === codec)?.engine} onClick={start}>Play</Btn>
<a className="btn" href={api.fileDownloadURL(path, sudo)} download={name}>Download instead</a>
</div>
)}
{err && <div className="error" style={{ marginTop: 8 }}>{err}</div>}
</div>
)}
{mode === 'hls' && stream && (
<>
<div className="play-bar">
<span title={describe(stream.s)}>{describe(stream.s)}</span>
{status && status.running && <span>· {status.speed ? `${status.speed.toFixed(1)}×` : ''} {stream.s.source.duration ? `${Math.min(100, Math.round(status.progress / stream.s.source.duration * 100))}% through` : `${Math.round(status.progress)} s done`} — seek within what's done</span>}
{status?.done && <span>· all of it is ready</span>}
{status?.error && <span className="error">· stopped early: {status.error}</span>}
<span className="spacer" />
<button className="btn" onClick={() => { stop(); setMode('pick'); setWhy('') }}>Change…</button>
</div>
<div className="preview"><video ref={video} controls autoPlay playsInline /></div>
{err && <div className="error" style={{ margin: 12 }}>{err}</div>}
</>
)}
</div>
)
}
+13 -2
View File
@@ -249,10 +249,14 @@ export interface ARCDetail {
} }
export interface ARCView { detail: ARCDetail; live: ARCLive | null } export interface ARCView { detail: ARCDetail; live: ARCLive | null }
export interface FileEntry { name: string; type: 'dir' | 'file' | 'link' | 'other'; size: number; mode: string; perm: string; mtime: number; user: string; group: string; target?: string; broken?: boolean; preview?: 'image' | 'video' } export interface FileEntry { name: string; type: 'dir' | 'file' | 'link' | 'other'; size: number; mode: string; perm: string; mtime: number; user: string; group: string; target?: string; broken?: boolean; preview?: 'image' | 'video' | 'audio' }
export interface PlayCaps { codecs: { id: string; label: string; engine?: string }[]; qualities: { id: string; label: string; height: number }[]; error?: string }
export interface PlayTrack { codec: string; profile?: string; pix_fmt?: string; width?: number; height?: number; fps?: number; channels?: number }
export interface PlayStream { id: string; codec: string; quality: string; engine: string; width: number; height: number; kbps: number; source: { container: string; duration: number; bit_rate?: number; video?: PlayTrack; audio?: PlayTrack } }
export interface PlayStatus { running: boolean; done: boolean; error?: string; progress: number; speed: number; segments: number }
export interface ThumbGPU { api: string; device: string; name?: string; codecs?: string[]; reason?: string; permission?: boolean } export interface ThumbGPU { api: string; device: string; name?: string; codecs?: string[]; reason?: string; permission?: boolean }
// How previews get made on the machine: ok with an engine, or why not and what would fix it. // How previews get made on the machine: ok with an engine, or why not and what would fix it.
export interface ThumbStatus { ok: boolean; engine: 'gpu' | 'cpu' | ''; summary: string; ffmpeg?: string; gpus: ThumbGPU[]; unusable?: ThumbGPU[]; error?: string; install?: { pm: string; package: string }; gpu_fix?: boolean; user: string; cache_dir: string } export interface ThumbStatus { ok: boolean; engine: 'gpu' | 'cpu' | ''; summary: string; ffmpeg?: string; gpus: ThumbGPU[]; unusable?: ThumbGPU[]; error?: string; install?: { pm: string; package: string }; gpu_fix?: boolean; user: string; cache: { path: string; entries: number; bytes: number } }
export interface FileInfo extends FileEntry { path: string; text: boolean } export interface FileInfo extends FileEntry { path: string; text: boolean }
const q = (path: string, sudo: boolean, extra = '') => `path=${encodeURIComponent(path)}${sudo ? '&sudo=1' : ''}${extra}` const q = (path: string, sudo: boolean, extra = '') => `path=${encodeURIComponent(path)}${sudo ? '&sudo=1' : ''}${extra}`
@@ -300,6 +304,11 @@ export const api = {
thumbStatus: (sudo: boolean, refresh = false) => req<ThumbStatus>('GET', `/api/files/thumbs?${sudo ? 'sudo=1&' : ''}${refresh ? 'refresh=1' : ''}`), thumbStatus: (sudo: boolean, refresh = false) => req<ThumbStatus>('GET', `/api/files/thumbs?${sudo ? 'sudo=1&' : ''}${refresh ? 'refresh=1' : ''}`),
thumbInstall: () => req<TaskInfo>('POST', '/api/files/thumbs/install'), thumbInstall: () => req<TaskInfo>('POST', '/api/files/thumbs/install'),
thumbGPUAccess: () => req<TaskInfo>('POST', '/api/files/thumbs/gpu-access'), thumbGPUAccess: () => req<TaskInfo>('POST', '/api/files/thumbs/gpu-access'),
// playback of what the browser can't play as it is: an HLS re-encode on the machine
playCaps: (sudo: boolean) => req<PlayCaps>('GET', `/api/files/play-caps${sudo ? '?sudo=1' : ''}`),
playStart: (path: string, codec: string, quality: string, sudo: boolean) => req<{ stream: PlayStream; url: string }>('POST', `/api/files/play${sudo ? '?sudo=1' : ''}`, { path, codec, quality }),
playStatus: (id: string) => req<PlayStatus>('GET', `/api/files/play/${id}`),
playStop: (id: string) => req<void>('DELETE', `/api/files/play/${id}`),
fileWrite: async (path: string, text: string, sudo: boolean) => { fileWrite: async (path: string, text: string, sudo: boolean) => {
const r = await fetch(apiPath(`/api/files/write?${q(path, sudo)}`), { method: 'PUT', credentials: 'same-origin', body: text, headers: { 'Content-Type': 'application/octet-stream' } }) const r = await fetch(apiPath(`/api/files/write?${q(path, sudo)}`), { method: 'PUT', credentials: 'same-origin', body: text, headers: { 'Content-Type': 'application/octet-stream' } })
if (r.status === 401) throw new Unauthenticated() if (r.status === 401) throw new Unauthenticated()
@@ -316,6 +325,8 @@ export const api = {
fileMkdir: (path: string, sudo: boolean) => req<void>('POST', `/api/files/mkdir${sudo ? '?sudo=1' : ''}`, { path }), fileMkdir: (path: string, sudo: boolean) => req<void>('POST', `/api/files/mkdir${sudo ? '?sudo=1' : ''}`, { path }),
fileTouch: (path: string, sudo: boolean) => req<void>('POST', `/api/files/touch${sudo ? '?sudo=1' : ''}`, { path }), fileTouch: (path: string, sudo: boolean) => req<void>('POST', `/api/files/touch${sudo ? '?sudo=1' : ''}`, { path }),
fileRename: (path: string, to: string, sudo: boolean) => req<void>('POST', `/api/files/rename${sudo ? '?sudo=1' : ''}`, { path, to }), fileRename: (path: string, to: string, sudo: boolean) => req<void>('POST', `/api/files/rename${sudo ? '?sudo=1' : ''}`, { path, to }),
// a rename when the ends share a filesystem (nothing back), otherwise the copy-and-remove runs as a task (the task back)
fileMove: (path: string, to: string, sudo: boolean) => req<TaskInfo | undefined>('POST', `/api/files/move${sudo ? '?sudo=1' : ''}`, { path, to }),
fileDelete: (path: string, recursive: boolean, sudo: boolean) => req<void>('POST', `/api/files/delete${sudo ? '?sudo=1' : ''}`, { path, recursive }), fileDelete: (path: string, recursive: boolean, sudo: boolean) => req<void>('POST', `/api/files/delete${sudo ? '?sudo=1' : ''}`, { path, recursive }),
fileChmod: (path: string, mode: string, sudo: boolean) => req<void>('POST', `/api/files/chmod${sudo ? '?sudo=1' : ''}`, { path, mode }), fileChmod: (path: string, mode: string, sudo: boolean) => req<void>('POST', `/api/files/chmod${sudo ? '?sudo=1' : ''}`, { path, mode }),
storage: () => req<{ devices: StorageDevice[]; zfs: ZFSInfo; smartctl: boolean }>('GET', '/api/storage'), storage: () => req<{ devices: StorageDevice[]; zfs: ZFSInfo; smartctl: boolean }>('GET', '/api/storage'),
+26 -1
View File
@@ -364,13 +364,38 @@ h2.section { margin: 0; font-size: 13px; font-weight: 600; color: var(--muted);
.file-editor { flex: 1; min-height: 0; border: 0; border-radius: 0; resize: none; font-size: 13px; } .file-editor { flex: 1; min-height: 0; border: 0; border-radius: 0; resize: none; font-size: 13px; }
.preview { flex: 1; display: flex; align-items: center; justify-content: center; overflow: auto; padding: 12px; } .preview { flex: 1; display: flex; align-items: center; justify-content: center; overflow: auto; padding: 12px; }
.preview img { max-width: 100%; max-height: 100%; object-fit: contain; } .preview img { max-width: 100%; max-height: 100%; object-fit: contain; }
.preview video { max-width: 100%; max-height: 100%; background: #000; border-radius: 4px; outline: none; }
.player { flex: 1; min-height: 0; display: flex; flex-direction: column; }
.player .preview { flex: 1; min-height: 0; }
.play-bar { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; padding: 6px 12px; font-size: 12px; color: var(--muted); border-bottom: 1px solid var(--border); }
.play-bar select { padding: 4px 8px; background: var(--bg); border: 1px solid var(--border); border-radius: 5px; color: var(--fg); font-size: 12px; }
.play-pick { margin: 12px; }
.play-pick .row { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; margin-top: 10px; }
.play-pick label { display: flex; align-items: center; gap: 6px; font-size: 12px; color: var(--muted); }
.preview.audio { align-items: flex-start; }
.audio-box { display: flex; flex-direction: column; align-items: center; gap: 10px; width: min(560px, 100%); margin-top: 8vh; }
.audio-box .ficon { font-size: 48px; }
.audio-box audio { width: 100%; }
/* previews: the icon column becomes a thumbnail box for images and videos */ /* previews: the icon column becomes a thumbnail box for images and videos */
.file-list.thumbs .file-row { grid-template-columns: 72px 1fr 80px 80px 130px 32px; padding-top: 4px; padding-bottom: 4px; min-height: 52px; } .file-list.thumbs .file-row { grid-template-columns: 72px 1fr 80px 80px 130px 32px; padding-top: 4px; padding-bottom: 4px; min-height: 52px; }
.fthumb { position: relative; width: 72px; height: 44px; display: flex; align-items: center; justify-content: center; background: var(--bg3); border-radius: 4px; overflow: hidden; } .fthumb { position: relative; width: 72px; height: 44px; display: flex; align-items: center; justify-content: center; background: var(--bg3); border-radius: 4px; overflow: hidden; }
.fthumb img { max-width: 100%; max-height: 100%; object-fit: contain; display: block; } .fthumb img { max-width: 100%; max-height: 100%; object-fit: contain; display: block; }
.fthumb.video::after { content: '▶'; position: absolute; right: 2px; bottom: 2px; font-size: 9px; line-height: 1; padding: 2px 3px; border-radius: 3px; background: rgba(0,0,0,.6); color: #fff; } .fthumb.video::after, .fthumb.audio::after { content: '▶'; position: absolute; right: 2px; bottom: 2px; font-size: 9px; line-height: 1; padding: 2px 3px; border-radius: 3px; background: rgba(0,0,0,.6); color: #fff; }
.fthumb.audio::after { content: '♪'; }
.fthumb .ficon { font-size: 18px; } .fthumb .ficon { font-size: 18px; }
.thumb-note { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; margin: 6px 10px 0; } .thumb-note { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; margin: 6px 10px 0; }
/* move: the folder picker */
.modal.mover { width: min(560px, 100%); height: min(76vh, 100%); overflow: hidden; }
.mover-nav { display: flex; align-items: center; gap: 2px; }
.mover-nav .icon-btn { font-size: 18px; min-width: 30px; line-height: 1; }
.mover-nav .icon-btn:disabled { opacity: .3; }
.mover-nav .crumbs { flex: 1; min-width: 0; margin-left: 4px; }
.modal.mover > .mover-list { flex: 1 1 auto; min-height: 80px; overflow-y: auto; margin: 0 -16px; border-top: 1px solid var(--border); }
.mover-row { display: flex; align-items: center; gap: 8px; padding: 9px 16px; cursor: pointer; border-bottom: 1px solid var(--border); min-height: 38px; }
.mover-row:hover { background: var(--bg3); }
.mover-row.self { opacity: .5; cursor: default; }
.mover-row .fname { flex: 1; }
.mover-dest { overflow-wrap: anywhere; }
.thumb-note .btn { padding-top: 3px; padding-bottom: 3px; } .thumb-note .btn { padding-top: 3px; padding-bottom: 3px; }
.files-bar .task-out { margin: 6px 10px 0; max-height: 160px; } .files-bar .task-out { margin: 6px 10px 0; max-height: 160px; }
@media (max-width: 800px) { @media (max-width: 800px) {