From 16e85e1e428777ee14e149cecb37a14a41bd4b01 Mon Sep 17 00:00:00 2001 From: Mitchell-Johnson Date: Fri, 9 Oct 2026 12:11:33 -0700 Subject: [PATCH] =?UTF-8?q?Virtualization=20tab:=20scoping=20plan=20?= =?UTF-8?q?=E2=80=94=20Incus=20on=20grid,=20what=20is=20and=20isn't=20feas?= =?UTF-8?q?ible?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- PLAN.md | 16 ++--- VIRTUALIZATION.md | 175 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 181 insertions(+), 10 deletions(-) create mode 100644 VIRTUALIZATION.md diff --git a/PLAN.md b/PLAN.md index e6f2234..67f5c19 100644 --- a/PLAN.md +++ b/PLAN.md @@ -785,16 +785,12 @@ what it has moved. filed under "decided, no action", and this is the change that would make closing it free. -- **A Virtualization tab** — asked for 2026-09-23, not started, and **large**. Run and - manage **LXC containers and virtual machines** from the console, the way Proxmox does — - which is the comparison Ham Lab has invited since its first line. Nothing exists yet. - What a future session has to settle before writing code: which stack (libvirt/KVM + - `virsh` is the obvious one for VMs; LXC or Incus for containers, and Incus covers both); - whether it manages what is already installed or installs it; where disk images live - (`tank` on `grid`, local NVMe elsewhere); console access (SPICE/VNC proxied through the - daemon, or a serial console over the existing terminal machinery, which would reuse - `internal/term`); snapshots and clones; and what it does on a host with no virtualisation - at all, which is every host here today. Worth scoping as its own plan before starting. +- **A Virtualization tab** — asked for 2026-09-23; **scoped 2026-10-09 in `VIRTUALIZATION.md`**, + not started. Incus 6.0 LTS on `grid` (the only sensible VM host), containers and VMs in one + list with console/shell through the terminal machinery, snapshots, exports, and the old + Proxmox estate on `tank` as a later import. That file has the host probe, the decisions + (NVMe pool, `192.168.200.0/24` bridge because the LAN is a /8, the Docker `FORWARD` fix, + ARC cap) and the phased build. ## Open questions diff --git a/VIRTUALIZATION.md b/VIRTUALIZATION.md new file mode 100644 index 0000000..0bbea0b --- /dev/null +++ b/VIRTUALIZATION.md @@ -0,0 +1,175 @@ +# Virtualization tab — plan (2026-10-09, not started) + +A tab for creating, watching, consoling into and deleting **LXC containers and KVM virtual +machines**, the way Proxmox does with `pct` and `qm` — per machine, like every other page. +This is the scoping PLAN.md asked for. Nothing here is built. Live state of the hosts +stays in `hamlet/docs/hosts.md`. + +## Decision: Incus 6.0 LTS, on `grid` first + +| Option | Verdict | +|---|---| +| **Incus** (LXD fork, linuxcontainers.org) | **Chosen.** One daemon, one CLI and one JSON API for both containers and VMs. In Debian 13 as `incus` 6.0.4 LTS — `apt install incus` pulls `qemu-system-x86`, `swtpm`, `virtiofsd`, `incus-agent`, so VMs work out of the box. Fedora 42 has 6.23 for the Mac mini if that is ever wanted. Native ZFS storage driver (instant snapshots and clones), built-in image server (`images:debian/13`, `images:ubuntu/24.04`, Alpine, Fedora, Arch… as container *or* VM), snapshot schedules with expiry, `incus export` backups, per-instance CPU/memory limits, cloud-init, text and VGA console over websocket. Everything is reachable as `incus … --format json` or `incus query`, which is exactly how Ham Lab already drives Docker: `exec.Command`, no new Go dependency. | +| libvirt/KVM + `virsh`, plus LXC separately | Two daemons, two models, XML. libvirt's official Go binding needs cgo, which the static binary rules out (the pure-Go `go-libvirt` exists but then containers are a second project). Rejected. | +| Proxmox itself | Means reinstalling `grid`'s OS. No. | + +## Where it can run + +Probed 2026-10-09 (`/dev/kvm`, cpu flags, `kvm_intel`/`kvm` modules, RAM, disk). + +| Host | KVM | RAM | Verdict | +|---|---|---|---| +| `grid` | yes — i7-11700T, VT-x, `kvm_intel` loaded, nested on | 62 GiB; ZFS ARC is at 23 GiB and allowed 40 | **The VM host.** 790 GiB free on the NVMe root, 4.5 TiB free on `tank`. Cap the ARC at 16 GiB and ~30 GiB is free for guests with headroom for Immich and the encode jobs. | +| `core` | yes — N100, `kvm_intel` loaded | 7.5 GiB, ~4 GiB available | Small LXC containers only, never a VM. Second host, later. | +| Mac mini | yes — Asahi KVM, `/dev/kvm` present, 16K pages | 15 GiB, 8 GiB available | Technically workable (Fedora ships Incus 6.23 and `qemu-system-aarch64`; arm64 guests only, no x86 VMs; 4K-page guests on the 16K host are what `muvm` already does). **Excluded by the rule: single-core-performance workloads only.** | +| `hamtv` | yes | 3.7 GiB | In service for HamTV. Excluded. | +| `gate` | **no** `/dev/kvm` (OpenWrt) | 8 GiB | Excluded. The tab says "no virtualisation on this machine". | + +Nothing is installed anywhere today: no qemu, libvirt, lxc or incus on any host. + +## What the tab does (v1) + +Mirrors the Docker tab's shape: a list, a detail page, actions that are followed tasks, +destructive ones confirmed by typing the name. + +- **Instances** — containers and VMs in one list: state, IPv4, CPU and memory live, disk, + snapshots count, uptime. Badge for type. Click for detail. +- **Create** — pick container or VM; image from the `images:` remote (searchable, with the + handful most people want pinned: Debian 13, Ubuntu 24.04, Alpine, Fedora); name, CPUs, + memory, root disk size; network (NAT bridge, or LAN once `br0` exists — below); optional + cloud-init user-data (SSH key, packages, a first command); profile. Create, or create and + start. Collisions (name taken) named before Create. +- **Lifecycle** — start, stop, restart, force stop, rename, delete (type the name; the + instance's snapshots go with it, and the dialog says so). +- **Console** — two buttons, both open as tabs in the Terminal page through the existing + PTY manager (`term.CreateCmd`, the `docker exec` path): **Console** (`incus console`, + the serial console — works before the network is up, shows boot output) and **Shell** + (`incus exec -- login shell`). Works for containers and, via `incus-agent`, VMs. +- **Snapshots** — take, restore (confirmed), delete; a **schedule** per instance with an + expiry, which is Incus's own `snapshots.schedule`/`snapshots.expiry`, so no cron file of + ours this time. +- **Resources** — edit CPU and memory limits (live for containers; VMs take CPU hotplug and + memory balloon live with the agent, else on restart), grow the root disk, add a bind + mount (a `tank` path into a container; virtiofs into a VM). +- **Images** — what is cached locally, delete, aliases; **publish** a stopped instance as an + image (Proxmox's "convert to template"). +- **Storage and networks** — the pools with their fill, the bridges with their subnets. + Read-only in v1 beyond what Create needs. +- **Not installed here** — a host with KVM but no Incus gets an **Install** button that + runs the whole phase-0 recipe below as a task (the smartmontools-install precedent). A + host with no KVM gets one sentence. +- **Sensors** — `instance..cpu`, `.memory`, `.running` from Incus's state, like the + `container.*` sensors, so Alerts gets an "instance down" default and widgets can show one. + +## Not in v1, with reasons + +| Thing | Why not yet | +|---|---| +| **VGA console in the browser** | Incus hands the VM's SPICE socket over a websocket; drawing it needs `spice-html5` in the UI. Doable (the Incus UI does it), but text console + shell cover every Linux guest. Phase 3. | +| **Windows VMs** | Incus runs them (ISO repacked with virtio drivers by `distrobuilder`), but without a VGA console in the browser they are unusable from the tab. After the above. | +| **GPU passthrough to a VM** | Takes the Arc A380 from the host, and from Jellyfin. No. Sharing the GPU into a *container* (`gpu` device) is fine and is how `r720-jellyfin` used to run. Phase 3. | +| **Clustering / live migration** | One real host; the others are different sizes and one is arm64; no shared storage. Not planned. | +| **A fleet-wide instance list** | One machine per view. The switcher is the machine. | +| **Mac mini, `hamtv`, `gate`** | Above. | + +## Decisions baked in (say if any is wrong) + +1. **Storage: instances on `grid`'s NVMe, data on `tank`.** A ZFS pool for Incus made as a + 300 GiB loop file on the NVMe root (`incus storage create default zfs size=300GiB`) — fast + enough for VM root disks where raidz2 spinners are not, and still ZFS so snapshots and + clones are instant. Big data is bind-mounted in from `tank`, the same split Docker already + has (apps on `core`, data on `grid`). A second pool on `tank/incus` can be added later for + a guest that genuinely wants terabytes. **Consequence:** `grid`'s root NVMe is not backed + up anywhere (backups land *on* `grid`), so instances need the nightly export below. +2. **Network: NAT bridge first, pinned to `192.168.200.1/24`.** Incus's default picks a + random `10.x.y.0/24`, and **the LAN here is `10.0.0.0/8`** — every host has a /8 on its + NIC, so any 10.x bridge collides. `192.168.200.0/24` is clear of Docker's pools on both + hosts (`172.28.0.0/14` on `grid`; `172.19`–`172.29` and `192.168.80`–`143` on `core`). + Instances get `name.incus` DNS from Incus's dnsmasq. **LAN addresses in phase 2:** a + `br0` on `grid`'s `enp10s0` (the 10 GbE uplink) so an instance can take a lease from + `gate` and a Pi-hole name. That rewrites `/etc/network/interfaces` and drops `grid` off + the network for seconds — its own step, with the NanoKVM standing by. macvlan was + considered and rejected: no host network change, but host and guests cannot talk, so + Immich/Jellyfin/backups on `grid` could never reach an instance. +3. **Docker and Incus on one host:** Docker sets the `FORWARD` policy to `DROP`, which + silently kills the Incus bridge's traffic. Fix is two `DOCKER-USER` accept rules for + `incusbr0`, persisted — part of the install recipe, on `grid` and later `core`. +4. **ARC cap on `grid`:** `zfs_arc_max` from 40 GiB to 16 GiB so guest RAM is deterministic. +5. **Privilege:** `core` joins `incus-admin` (socket access, the `docker`-group shape) and + `hamlab.service` gets `SupplementaryGroups=incus-admin`; no sudo in the hot path. +6. **Backups:** nightly `incus export --instance-only` of every instance to + `/tank/backups/incus/` with a retention, as a cron line like the ZFS schedules + (`hamlab incus-export`), from phase 2. Plus the instance's own snapshot schedule. + +## The old Proxmox estate (restorable, phase 3) + +`/tank/backups/proxmox/` holds the retired `r720` node: `/etc/pve` (12 LXC confs, 2 VM +confs), 118 GiB of `vzdump` archives, and three raw VM disks. Latest of each: + +| ID | Name | Latest archive | Size | +|---|---|---|---| +| 100 | r720-docker | 2026-01-11 | 20.7 GB | +| 101 | r720-cockpit | 2026-01-11 | 439 MB | +| 102 | r720-wordpress-alexblaskovich | 2026-01-11 | 1.2 GB | +| 103 | r720-spoonterm | 2025-06-14 | 494 MB | +| 104 | r720-ollama | 2026-01-11 | 11.3 GB | +| 105 | r720-transmission | 2026-01-11 | 322 MB | +| 106 | r720-apache | 2026-01-11 | 347 MB | +| 107 | r720-nut | 2025-03-29 | 205 MB | +| 108 | r720-filebrowser | 2026-01-11 | 262 MB | +| 109 | r720-bookstack | 2025-04-27 | 516 MB | +| 110 | r720-jellyfin | 2026-01-11 | 2.2 GB | +| 111 | r720-wireguard | 2025-04-13 | 510 MB | +| 112 | r720-eggwiki | 2026-01-11 | 634 MB | +| VMs | steam (102), powerpanel (105/107/112) | `.vma.zst` 2024–2025; raw disks `vm-105`, `vm-112`, `base-104` | 19 GB, 1.9 GB ×6, 97 GiB | + +- **LXC:** feasible and scriptable. Unpack the rootfs (`tar --zstd -xpf … --numeric-owner`), + read `etc/vzdump/pct.conf` for arch/ostype, drop it, write `metadata.yaml`, pack, `incus + image import`, launch. Unprivileged uid shifting is Incus's job after that. A wizard in the + tab is plausible once the script works by hand. +- **VMs:** the raw disks import directly (`incus-migrate`, or an empty VM with the disk + imported into its root volume). The `.vma.zst` archives need Proxmox's `vma` extractor (or + the Python port) to become raw disks first. Most of these are the same `powerpanel` box + backed up six times, and `tank-space-audit.md` already has them down for deletion — decide + which, if any, are wanted before anything is restored. + +## Build plan + +| Phase | What | Effort | +|---|---|---| +| **0 — host prep on `grid`** (by hand, documented) | `apt install incus`; `incus admin init --preseed` with the pool and bridge above; `DOCKER-USER` rules; ARC cap; `incus-admin` group + unit edit; launch one Debian 13 container and one Debian 13 VM; verify console, shell, network, snapshot, export. Write it up in `hamlet/docs/hosts.md` and the preseed into `hamlet/scripts/`. | 1 session | +| **1 — the tab** | `internal/incus` (CLI wrapper, JSON types, `incus monitor` for live state), `internal/api/virt.go` (routes below), `Virtualization.tsx` (list, detail, create dialog, snapshot panel), rail entry `virt`, terminal integration, Install button, the no-KVM state. | 2 sessions | +| **2 — operations** | Snapshot schedules UI, nightly exports with retention, limits editor, bind mounts, `br0` on `grid` and the LAN network option, instance sensors + alert default, cloud-init presets. | 1–2 sessions | +| **3 — the rest** | Proxmox import wizard, publish-as-image, GPU device for containers, VGA console (`spice-html5`), `core` as a second (containers-only) host. | as wanted | + +### API sketch + +``` +GET /api/virt {installed, kvm, pools[], networks[], instances[]} +POST /api/virt/install task: the phase-0 recipe +GET /api/virt/images?remote=images&q= local cache + remote search +DELETE /api/virt/images/{fingerprint} +POST /api/virt/instances {name, type, image, cpu, memory, disk, network, cloudinit, profile, start} task +GET /api/virt/instances/{name} config, state, devices, snapshots +PATCH /api/virt/instances/{name} limits, description, devices +POST /api/virt/instances/{name}/{start|stop|restart|kill|delete|rename|publish|export} +POST /api/virt/instances/{name}/console → terminal {id} (incus console) +POST /api/virt/instances/{name}/exec → terminal {id} (incus exec -- sh -lc …) +POST /api/virt/instances/{name}/snapshots {name}; POST …/{snap}/restore; DELETE …/{snap} +PUT /api/virt/instances/{name}/schedule {cron, expiry} +POST /api/virt/import {path} task (export tarball, or phase-3 Proxmox archive) +``` + +Everything is per machine and reaches a node through the existing `/api/nodes/{name}/…` +proxy with no hub-side code; consoles attach over the existing `/api/terminals/{id}/attach`. +Tasks stream through `/api/tasks/{id}/stream` as Docker's do. + +## Sources + +- Incus on Debian: https://wiki.debian.org/Incus +- Incus and Docker on one host (the `FORWARD` drop): https://linuxcontainers.org/incus/docs/main/howto/network_bridge_firewalld/ +- Bridge options and `ipv4.address: auto`: https://linuxcontainers.org/incus/docs/main/reference/network_bridge/ +- Consoles (text and VGA): https://linuxcontainers.org/incus/docs/main/howto/instances_console/ +- ZFS storage driver: https://linuxcontainers.org/incus/docs/main/reference/storage_zfs/ +- macvlan's host-isolation: https://linuxcontainers.org/incus/docs/main/reference/network_macvlan/ +- Image format for the Proxmox import: https://linuxcontainers.org/incus/docs/main/reference/image_format/