Virtualization tab: scoping plan — Incus on grid, what is and isn't feasible
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Mitchell-Johnson
co-authored by
Claude Fable 5.1
Claude Fable 5.1
parent
1fbf345166
commit
16e85e1e42
@@ -785,16 +785,12 @@ what it has moved.
|
||||
filed under "decided, no action", and this is the change that would make closing it
|
||||
free.
|
||||
|
||||
- **A Virtualization tab** — asked for 2026-09-23, not started, and **large**. Run and
|
||||
manage **LXC containers and virtual machines** from the console, the way Proxmox does —
|
||||
which is the comparison Ham Lab has invited since its first line. Nothing exists yet.
|
||||
What a future session has to settle before writing code: which stack (libvirt/KVM +
|
||||
`virsh` is the obvious one for VMs; LXC or Incus for containers, and Incus covers both);
|
||||
whether it manages what is already installed or installs it; where disk images live
|
||||
(`tank` on `grid`, local NVMe elsewhere); console access (SPICE/VNC proxied through the
|
||||
daemon, or a serial console over the existing terminal machinery, which would reuse
|
||||
`internal/term`); snapshots and clones; and what it does on a host with no virtualisation
|
||||
at all, which is every host here today. Worth scoping as its own plan before starting.
|
||||
- **A Virtualization tab** — asked for 2026-09-23; **scoped 2026-10-09 in `VIRTUALIZATION.md`**,
|
||||
not started. Incus 6.0 LTS on `grid` (the only sensible VM host), containers and VMs in one
|
||||
list with console/shell through the terminal machinery, snapshots, exports, and the old
|
||||
Proxmox estate on `tank` as a later import. That file has the host probe, the decisions
|
||||
(NVMe pool, `192.168.200.0/24` bridge because the LAN is a /8, the Docker `FORWARD` fix,
|
||||
ARC cap) and the phased build.
|
||||
|
||||
|
||||
## Open questions
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
# Virtualization tab — plan (2026-10-09, not started)
|
||||
|
||||
A tab for creating, watching, consoling into and deleting **LXC containers and KVM virtual
|
||||
machines**, the way Proxmox does with `pct` and `qm` — per machine, like every other page.
|
||||
This is the scoping PLAN.md asked for. Nothing here is built. Live state of the hosts
|
||||
stays in `hamlet/docs/hosts.md`.
|
||||
|
||||
## Decision: Incus 6.0 LTS, on `grid` first
|
||||
|
||||
| Option | Verdict |
|
||||
|---|---|
|
||||
| **Incus** (LXD fork, linuxcontainers.org) | **Chosen.** One daemon, one CLI and one JSON API for both containers and VMs. In Debian 13 as `incus` 6.0.4 LTS — `apt install incus` pulls `qemu-system-x86`, `swtpm`, `virtiofsd`, `incus-agent`, so VMs work out of the box. Fedora 42 has 6.23 for the Mac mini if that is ever wanted. Native ZFS storage driver (instant snapshots and clones), built-in image server (`images:debian/13`, `images:ubuntu/24.04`, Alpine, Fedora, Arch… as container *or* VM), snapshot schedules with expiry, `incus export` backups, per-instance CPU/memory limits, cloud-init, text and VGA console over websocket. Everything is reachable as `incus … --format json` or `incus query`, which is exactly how Ham Lab already drives Docker: `exec.Command`, no new Go dependency. |
|
||||
| libvirt/KVM + `virsh`, plus LXC separately | Two daemons, two models, XML. libvirt's official Go binding needs cgo, which the static binary rules out (the pure-Go `go-libvirt` exists but then containers are a second project). Rejected. |
|
||||
| Proxmox itself | Means reinstalling `grid`'s OS. No. |
|
||||
|
||||
## Where it can run
|
||||
|
||||
Probed 2026-10-09 (`/dev/kvm`, cpu flags, `kvm_intel`/`kvm` modules, RAM, disk).
|
||||
|
||||
| Host | KVM | RAM | Verdict |
|
||||
|---|---|---|---|
|
||||
| `grid` | yes — i7-11700T, VT-x, `kvm_intel` loaded, nested on | 62 GiB; ZFS ARC is at 23 GiB and allowed 40 | **The VM host.** 790 GiB free on the NVMe root, 4.5 TiB free on `tank`. Cap the ARC at 16 GiB and ~30 GiB is free for guests with headroom for Immich and the encode jobs. |
|
||||
| `core` | yes — N100, `kvm_intel` loaded | 7.5 GiB, ~4 GiB available | Small LXC containers only, never a VM. Second host, later. |
|
||||
| Mac mini | yes — Asahi KVM, `/dev/kvm` present, 16K pages | 15 GiB, 8 GiB available | Technically workable (Fedora ships Incus 6.23 and `qemu-system-aarch64`; arm64 guests only, no x86 VMs; 4K-page guests on the 16K host are what `muvm` already does). **Excluded by the rule: single-core-performance workloads only.** |
|
||||
| `hamtv` | yes | 3.7 GiB | In service for HamTV. Excluded. |
|
||||
| `gate` | **no** `/dev/kvm` (OpenWrt) | 8 GiB | Excluded. The tab says "no virtualisation on this machine". |
|
||||
|
||||
Nothing is installed anywhere today: no qemu, libvirt, lxc or incus on any host.
|
||||
|
||||
## What the tab does (v1)
|
||||
|
||||
Mirrors the Docker tab's shape: a list, a detail page, actions that are followed tasks,
|
||||
destructive ones confirmed by typing the name.
|
||||
|
||||
- **Instances** — containers and VMs in one list: state, IPv4, CPU and memory live, disk,
|
||||
snapshots count, uptime. Badge for type. Click for detail.
|
||||
- **Create** — pick container or VM; image from the `images:` remote (searchable, with the
|
||||
handful most people want pinned: Debian 13, Ubuntu 24.04, Alpine, Fedora); name, CPUs,
|
||||
memory, root disk size; network (NAT bridge, or LAN once `br0` exists — below); optional
|
||||
cloud-init user-data (SSH key, packages, a first command); profile. Create, or create and
|
||||
start. Collisions (name taken) named before Create.
|
||||
- **Lifecycle** — start, stop, restart, force stop, rename, delete (type the name; the
|
||||
instance's snapshots go with it, and the dialog says so).
|
||||
- **Console** — two buttons, both open as tabs in the Terminal page through the existing
|
||||
PTY manager (`term.CreateCmd`, the `docker exec` path): **Console** (`incus console`,
|
||||
the serial console — works before the network is up, shows boot output) and **Shell**
|
||||
(`incus exec <n> -- login shell`). Works for containers and, via `incus-agent`, VMs.
|
||||
- **Snapshots** — take, restore (confirmed), delete; a **schedule** per instance with an
|
||||
expiry, which is Incus's own `snapshots.schedule`/`snapshots.expiry`, so no cron file of
|
||||
ours this time.
|
||||
- **Resources** — edit CPU and memory limits (live for containers; VMs take CPU hotplug and
|
||||
memory balloon live with the agent, else on restart), grow the root disk, add a bind
|
||||
mount (a `tank` path into a container; virtiofs into a VM).
|
||||
- **Images** — what is cached locally, delete, aliases; **publish** a stopped instance as an
|
||||
image (Proxmox's "convert to template").
|
||||
- **Storage and networks** — the pools with their fill, the bridges with their subnets.
|
||||
Read-only in v1 beyond what Create needs.
|
||||
- **Not installed here** — a host with KVM but no Incus gets an **Install** button that
|
||||
runs the whole phase-0 recipe below as a task (the smartmontools-install precedent). A
|
||||
host with no KVM gets one sentence.
|
||||
- **Sensors** — `instance.<name>.cpu`, `.memory`, `.running` from Incus's state, like the
|
||||
`container.*` sensors, so Alerts gets an "instance down" default and widgets can show one.
|
||||
|
||||
## Not in v1, with reasons
|
||||
|
||||
| Thing | Why not yet |
|
||||
|---|---|
|
||||
| **VGA console in the browser** | Incus hands the VM's SPICE socket over a websocket; drawing it needs `spice-html5` in the UI. Doable (the Incus UI does it), but text console + shell cover every Linux guest. Phase 3. |
|
||||
| **Windows VMs** | Incus runs them (ISO repacked with virtio drivers by `distrobuilder`), but without a VGA console in the browser they are unusable from the tab. After the above. |
|
||||
| **GPU passthrough to a VM** | Takes the Arc A380 from the host, and from Jellyfin. No. Sharing the GPU into a *container* (`gpu` device) is fine and is how `r720-jellyfin` used to run. Phase 3. |
|
||||
| **Clustering / live migration** | One real host; the others are different sizes and one is arm64; no shared storage. Not planned. |
|
||||
| **A fleet-wide instance list** | One machine per view. The switcher is the machine. |
|
||||
| **Mac mini, `hamtv`, `gate`** | Above. |
|
||||
|
||||
## Decisions baked in (say if any is wrong)
|
||||
|
||||
1. **Storage: instances on `grid`'s NVMe, data on `tank`.** A ZFS pool for Incus made as a
|
||||
300 GiB loop file on the NVMe root (`incus storage create default zfs size=300GiB`) — fast
|
||||
enough for VM root disks where raidz2 spinners are not, and still ZFS so snapshots and
|
||||
clones are instant. Big data is bind-mounted in from `tank`, the same split Docker already
|
||||
has (apps on `core`, data on `grid`). A second pool on `tank/incus` can be added later for
|
||||
a guest that genuinely wants terabytes. **Consequence:** `grid`'s root NVMe is not backed
|
||||
up anywhere (backups land *on* `grid`), so instances need the nightly export below.
|
||||
2. **Network: NAT bridge first, pinned to `192.168.200.1/24`.** Incus's default picks a
|
||||
random `10.x.y.0/24`, and **the LAN here is `10.0.0.0/8`** — every host has a /8 on its
|
||||
NIC, so any 10.x bridge collides. `192.168.200.0/24` is clear of Docker's pools on both
|
||||
hosts (`172.28.0.0/14` on `grid`; `172.19`–`172.29` and `192.168.80`–`143` on `core`).
|
||||
Instances get `name.incus` DNS from Incus's dnsmasq. **LAN addresses in phase 2:** a
|
||||
`br0` on `grid`'s `enp10s0` (the 10 GbE uplink) so an instance can take a lease from
|
||||
`gate` and a Pi-hole name. That rewrites `/etc/network/interfaces` and drops `grid` off
|
||||
the network for seconds — its own step, with the NanoKVM standing by. macvlan was
|
||||
considered and rejected: no host network change, but host and guests cannot talk, so
|
||||
Immich/Jellyfin/backups on `grid` could never reach an instance.
|
||||
3. **Docker and Incus on one host:** Docker sets the `FORWARD` policy to `DROP`, which
|
||||
silently kills the Incus bridge's traffic. Fix is two `DOCKER-USER` accept rules for
|
||||
`incusbr0`, persisted — part of the install recipe, on `grid` and later `core`.
|
||||
4. **ARC cap on `grid`:** `zfs_arc_max` from 40 GiB to 16 GiB so guest RAM is deterministic.
|
||||
5. **Privilege:** `core` joins `incus-admin` (socket access, the `docker`-group shape) and
|
||||
`hamlab.service` gets `SupplementaryGroups=incus-admin`; no sudo in the hot path.
|
||||
6. **Backups:** nightly `incus export --instance-only` of every instance to
|
||||
`/tank/backups/incus/` with a retention, as a cron line like the ZFS schedules
|
||||
(`hamlab incus-export`), from phase 2. Plus the instance's own snapshot schedule.
|
||||
|
||||
## The old Proxmox estate (restorable, phase 3)
|
||||
|
||||
`/tank/backups/proxmox/` holds the retired `r720` node: `/etc/pve` (12 LXC confs, 2 VM
|
||||
confs), 118 GiB of `vzdump` archives, and three raw VM disks. Latest of each:
|
||||
|
||||
| ID | Name | Latest archive | Size |
|
||||
|---|---|---|---|
|
||||
| 100 | r720-docker | 2026-01-11 | 20.7 GB |
|
||||
| 101 | r720-cockpit | 2026-01-11 | 439 MB |
|
||||
| 102 | r720-wordpress-alexblaskovich | 2026-01-11 | 1.2 GB |
|
||||
| 103 | r720-spoonterm | 2025-06-14 | 494 MB |
|
||||
| 104 | r720-ollama | 2026-01-11 | 11.3 GB |
|
||||
| 105 | r720-transmission | 2026-01-11 | 322 MB |
|
||||
| 106 | r720-apache | 2026-01-11 | 347 MB |
|
||||
| 107 | r720-nut | 2025-03-29 | 205 MB |
|
||||
| 108 | r720-filebrowser | 2026-01-11 | 262 MB |
|
||||
| 109 | r720-bookstack | 2025-04-27 | 516 MB |
|
||||
| 110 | r720-jellyfin | 2026-01-11 | 2.2 GB |
|
||||
| 111 | r720-wireguard | 2025-04-13 | 510 MB |
|
||||
| 112 | r720-eggwiki | 2026-01-11 | 634 MB |
|
||||
| VMs | steam (102), powerpanel (105/107/112) | `.vma.zst` 2024–2025; raw disks `vm-105`, `vm-112`, `base-104` | 19 GB, 1.9 GB ×6, 97 GiB |
|
||||
|
||||
- **LXC:** feasible and scriptable. Unpack the rootfs (`tar --zstd -xpf … --numeric-owner`),
|
||||
read `etc/vzdump/pct.conf` for arch/ostype, drop it, write `metadata.yaml`, pack, `incus
|
||||
image import`, launch. Unprivileged uid shifting is Incus's job after that. A wizard in the
|
||||
tab is plausible once the script works by hand.
|
||||
- **VMs:** the raw disks import directly (`incus-migrate`, or an empty VM with the disk
|
||||
imported into its root volume). The `.vma.zst` archives need Proxmox's `vma` extractor (or
|
||||
the Python port) to become raw disks first. Most of these are the same `powerpanel` box
|
||||
backed up six times, and `tank-space-audit.md` already has them down for deletion — decide
|
||||
which, if any, are wanted before anything is restored.
|
||||
|
||||
## Build plan
|
||||
|
||||
| Phase | What | Effort |
|
||||
|---|---|---|
|
||||
| **0 — host prep on `grid`** (by hand, documented) | `apt install incus`; `incus admin init --preseed` with the pool and bridge above; `DOCKER-USER` rules; ARC cap; `incus-admin` group + unit edit; launch one Debian 13 container and one Debian 13 VM; verify console, shell, network, snapshot, export. Write it up in `hamlet/docs/hosts.md` and the preseed into `hamlet/scripts/`. | 1 session |
|
||||
| **1 — the tab** | `internal/incus` (CLI wrapper, JSON types, `incus monitor` for live state), `internal/api/virt.go` (routes below), `Virtualization.tsx` (list, detail, create dialog, snapshot panel), rail entry `virt`, terminal integration, Install button, the no-KVM state. | 2 sessions |
|
||||
| **2 — operations** | Snapshot schedules UI, nightly exports with retention, limits editor, bind mounts, `br0` on `grid` and the LAN network option, instance sensors + alert default, cloud-init presets. | 1–2 sessions |
|
||||
| **3 — the rest** | Proxmox import wizard, publish-as-image, GPU device for containers, VGA console (`spice-html5`), `core` as a second (containers-only) host. | as wanted |
|
||||
|
||||
### API sketch
|
||||
|
||||
```
|
||||
GET /api/virt {installed, kvm, pools[], networks[], instances[]}
|
||||
POST /api/virt/install task: the phase-0 recipe
|
||||
GET /api/virt/images?remote=images&q= local cache + remote search
|
||||
DELETE /api/virt/images/{fingerprint}
|
||||
POST /api/virt/instances {name, type, image, cpu, memory, disk, network, cloudinit, profile, start} task
|
||||
GET /api/virt/instances/{name} config, state, devices, snapshots
|
||||
PATCH /api/virt/instances/{name} limits, description, devices
|
||||
POST /api/virt/instances/{name}/{start|stop|restart|kill|delete|rename|publish|export}
|
||||
POST /api/virt/instances/{name}/console → terminal {id} (incus console)
|
||||
POST /api/virt/instances/{name}/exec → terminal {id} (incus exec -- sh -lc …)
|
||||
POST /api/virt/instances/{name}/snapshots {name}; POST …/{snap}/restore; DELETE …/{snap}
|
||||
PUT /api/virt/instances/{name}/schedule {cron, expiry}
|
||||
POST /api/virt/import {path} task (export tarball, or phase-3 Proxmox archive)
|
||||
```
|
||||
|
||||
Everything is per machine and reaches a node through the existing `/api/nodes/{name}/…`
|
||||
proxy with no hub-side code; consoles attach over the existing `/api/terminals/{id}/attach`.
|
||||
Tasks stream through `/api/tasks/{id}/stream` as Docker's do.
|
||||
|
||||
## Sources
|
||||
|
||||
- Incus on Debian: https://wiki.debian.org/Incus
|
||||
- Incus and Docker on one host (the `FORWARD` drop): https://linuxcontainers.org/incus/docs/main/howto/network_bridge_firewalld/
|
||||
- Bridge options and `ipv4.address: auto`: https://linuxcontainers.org/incus/docs/main/reference/network_bridge/
|
||||
- Consoles (text and VGA): https://linuxcontainers.org/incus/docs/main/howto/instances_console/
|
||||
- ZFS storage driver: https://linuxcontainers.org/incus/docs/main/reference/storage_zfs/
|
||||
- macvlan's host-isolation: https://linuxcontainers.org/incus/docs/main/reference/network_macvlan/
|
||||
- Image format for the Proxmox import: https://linuxcontainers.org/incus/docs/main/reference/image_format/
|
||||
Reference in New Issue
Block a user