mitchell/hamlab
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 14 Wiki Activity
Files
e772f7fae2d87deca7a40a628df7698be619d1f7
hamlab/VIRTUALIZATION.md
T

23 KiB
Raw Blame History

Virtualization tab — plan (2026-10-09, revised the same day; not started)

A tab for creating, watching, consoling into and deleting LXC containers and KVM virtual machines, the way Proxmox does with pct and qm — per machine, like every other page. This is the scoping PLAN.md asked for. Nothing here is built. Live state of the hosts stays in hamlet/docs/hosts.md.

The governing rule: capability, not policy

Stated by Mitchell 2026-10-09, after the first draft of this plan excluded hosts on the strength of the homelab's deployment rules:

Anything that's done when a different machine is selected SHOULD GO ON THAT machine. I want the option to do it anywhere I want, unless the hardware / software literally doesn't support it. Ham Lab needs to be built in a general-purpose way that can run on any Linux system. If certain parts / libraries aren't available on that machine, those tabs should just report that.

So:

  • Every action lands on the machine in the switcher. There is no "VM host", no default machine, no cross-machine placement. grid is where he will most likely click first; the tab does not know or care.
  • The tab is gated by what the machine can do, detected live — never by a rule about what should run there. The hamlet/CLAUDE.md do-not-break rules (Mac mini for single-core workloads, hamtv not spare hardware) bind the agent's unasked deployments. They are not feature gates for the product, and the first draft was wrong to treat them as such.
  • Ham Lab owns the install and the data layout. It detects what is missing, offers to install it where it knows how, chooses and records where instance storage lives and which subnet the bridge takes, and runs the backups. Where it cannot install or the hardware cannot do the job, it says so in one sentence and offers what does work (containers without KVM, for instance).

Decision: Incus 6.0 LTS as the engine

Option Verdict
Incus (LXD fork, linuxcontainers.org) Chosen. One daemon, one CLI and one JSON API for both containers and VMs. Packaged by Debian 13 (incus 6.0.4 LTS — apt install incus pulls qemu-system-x86, swtpm, virtiofsd, incus-agent, so VMs work out of the box), Fedora (6.23), Arch, Alpine, Ubuntu (Zabbly repo); not by OpenWrt. Storage drivers for ZFS, btrfs, LVM and plain directories, so it fits a machine with or without ZFS. Built-in image server (images:debian/13, images:ubuntu/24.04, Alpine, Fedora, Arch… as container or VM, per architecture), snapshot schedules with expiry, incus export backups, per-instance CPU/memory limits, cloud-init, text and VGA console over websocket. Everything is reachable as incus … --format json or incus query, which is exactly how Ham Lab already drives Docker: exec.Command, no new Go dependency, no cgo.
libvirt/KVM + virsh, plus LXC separately Two daemons, two models, XML. libvirt's official Go binding needs cgo, which the static binary rules out (the pure-Go go-libvirt exists but then containers are a second project). Rejected.
Proxmox An OS, not a library. Rejected.

What each machine can do today

Probed 2026-10-09 (/dev/kvm, cpu flags, kvm modules, RAM, disk, package availability). These are facts the tab will detect at runtime, not a list of where instances are allowed. The last column is what the tab would show the day it ships.

Machine Arch KVM RAM (free) Incus from the package manager Pool driver available Tab state on first open
grid x86-64 yes — VT-x, kvm_intel loaded, nested on 62 GiB (~25; ARC holds 23 of the rest, max 40) apt: incus 6.0.4 zfs (2.3.9, tank) Install — containers and VMs
core x86-64 yes — N100, kvm_intel loaded 7.5 GiB (~4) apt: incus 6.0.4 btrfs or dir (ZFS was purged 2026-09-23) Install — containers and VMs; the create form shows free memory so a VM's size is an informed choice
Mac mini aarch64 yes — Asahi KVM, 16K pages 15 GiB (~8) dnf: incus 6.23, qemu-system-aarch64-core, edk2-aarch64 btrfs or dir Install — containers and VMs, arm64 images only (the picker filters by host arch). Whether a stock 4K-page guest kernel boots under the 16K host is untested; muvm does exactly that, so it is expected to.
hamtv x86-64 yes 3.7 GiB (~2.9) apt: incus 6.0.4 btrfs or dir Install — containers and small VMs
gate x86-64 no /dev/kvm 8 GiB (~6) none — OpenWrt's opkg has no Incus — Unavailable: "Incus isn't packaged for OpenWrt, and this kernel has no KVM." Nothing to install.

Nothing is installed anywhere today: no qemu, libvirt, lxc or incus on any host.

Detection the tab runs, in order

  1. incus on PATH and its socket answering → ready; read pools, networks, instances.
  2. Else: a package manager Ham Lab knows (apt, dnf, pacman, apk) that has an incus candidate → installable, with the Install dialog pre-filled (below).
  3. Else → unavailable, with the reason in one sentence (no package; no root; OpenWrt).
  4. Independently: /dev/kvm present and openable → VMs offered. Absent → containers only, said once on the page, and the create form has no VM option.
  5. Architecture from uname -m, so the image picker only shows images that can run here.
  6. Pool driver: zfs if the module and tools are present, else btrfs (installing btrfs-progs as part of Install), else dir. The dialog says which it picked and why.

What the tab does (v1)

Mirrors the Docker tab's shape: a list, a detail page, actions that are followed tasks, destructive ones confirmed by typing the name.

  • Install — one task that runs on the selected machine: install the package(s), incus admin init --preseed with the layout from the dialog, create the pool and bridge, add Ham Lab's user to incus-admin (socket access, the docker-group shape) and add it to the unit's SupplementaryGroups, apply the Docker fix where Docker is present (below), then restart Ham Lab. Idempotent, so it doubles as "repair". The dialog shows and lets him edit: pool driver (detected), pool size (proposed from free space), pool path, bridge subnet (computed), whether to apply the Docker rules.
  • Instances — containers and VMs in one list: type, state, IPv4, CPU and memory live, disk, snapshot count, uptime. Click for detail.
  • Create — container or VM (VM only where KVM is); image from the images: remote, filtered to this architecture, searchable, with the common ones pinned (Debian 13, Ubuntu 24.04, Alpine, Fedora); name, CPUs, memory (with this machine's free memory beside it), root disk size; network (the NAT bridge, or a LAN bridge if this machine has one); optional cloud-init user-data (SSH key, packages, a first command); profile. Create, or create and start. Collisions (name taken) named before Create.
  • Lifecycle — start, stop, restart, force stop, rename, delete (type the name; the instance's snapshots go with it, and the dialog says so).
  • Console — two buttons, both open as tabs in the Terminal page through the existing PTY manager (term.CreateCmd, the docker exec path): Console (incus console, the serial console — works before the network is up, shows boot output) and Shell (incus exec <n> -- login shell). Containers and, via incus-agent, VMs.
  • Snapshots — take, restore (confirmed), delete; a schedule per instance with an expiry, which is Incus's own snapshots.schedule/snapshots.expiry, so no cron file of ours this time.
  • Resources — edit CPU and memory limits (live for containers; VMs take CPU hotplug and memory balloon live with the agent, else on restart), grow the root disk, add a bind mount (a host path into a container; virtiofs into a VM).
  • Library — ISOs for VMs and images (container templates and VM images). Its own section below.
  • Storage and networks — the pools with their fill, the bridges with their subnets. Read-only in v1 beyond what Create and Install need.
  • Sensors — instance.<name>.cpu, .memory, .running from Incus's state, like the container.* sensors, so Alerts gets an "instance down" default and widgets can show one.

Library: ISOs and images (asked for 2026-10-09)

Proxmox has a storage view with ISO Images and CT Templates. Incus has the same two things under other names, and the tab gives them one home, a Library sub-tab, per machine like everything else.

Proxmox calls it Incus calls it What it is here
ISO Images an ISO file attached to a VM as a disk device (added as a ROM device; VMs only), or a custom storage volume of type iso A directory of .iso files on the machine that Ham Lab indexes — no copy into the pool
CT Templates images — the images: remote (linuxcontainers.org), the local image store, anything imported or published The local image store, with the remote as the catalogue to pull from

ISOs

  • One ISO directory per machine, chosen in the Install dialog and changeable later. Default is under Ham Lab's data directory; on grid it should be the existing /tank/software/isos/ — 105 GiB already there (linux/, windows/, truenas/, virtio_drivers/, pi/, hackberry/, sipeed/), scanned recursively.
  • Getting an ISO in: upload from the browser (the Files tab's PUT /api/files/upload with its progress bar, reused), download from a URL as a followed task with an optional checksum (the usual way — an ISO starts as a link), or import from a path already on the machine. Rename, delete (refused while a VM has it attached, naming the VM).
  • Using one: attach to / detach from a VM as a ROM device by path, with boot.priority set so it boots first; New VM from ISO makes an empty VM (incus init --empty --vm) with the ISO attached, starts it and opens the console. After install, Detach is one click and boot order falls back to the disk.
  • Honest limit: an ISO install is an interactive graphical installer, so it needs the VGA console in the browser (spice-html5 over Incus's VGA console websocket). Installers that run on a serial console (Debian with console=ttyS0, Alpine, Arch) work through the text console before that exists. Windows needs the virtio drivers ISO attached as a second ROM device as well — Incus's documented path is distrobuilder repack-windows to bake them in; the drivers are already in virtio_drivers/ on tank. So the ISO shelf and the VGA console ship together, in phase 2, moved up from phase 3.
  • ISO volumes (incus storage volume import <pool> <iso> <name> --type=iso) are the alternative: read-only, attachable to several VMs at once, but they copy the file into the pool. Not the default here — the directory honours the library that already exists and keeps 105 GiB out of a 300 GiB pool. Offered as an option only if a need shows up.

Images (container templates and VM images)

  • The catalogue: browse images: filtered to this machine's architecture, by type (container / VM), with the common ones pinned. Download copies one into the local store (incus image copy images:debian/13 local: --alias debian-13) so a later Create is instant and works without the internet. Images used by Create are cached by Incus anyway; this is just the explicit version, Proxmox's "Templates → Download".
  • The local store: every image here with alias, type, arch, size, origin (remote, published, imported), when it was last used, and which instances were made from it (Incus records the base image on each instance). Alias, delete (refused while an instance's base — Incus knows), "New instance from this image".
  • Import: a unified tarball or metadata + rootfs pair (a container template from anywhere, including the phase-3 Proxmox conversions), or a qcow2 / raw disk with metadata Ham Lab writes for a VM image — Home Assistant OS, OPNsense, a vendor cloud image — by upload, URL, or a path on the machine. Same three ways in as ISOs.
  • Publish: a stopped instance becomes an image with an alias — Proxmox's "convert to template", the one thing the Images section of v1 always had.
  • Where they live: Incus's image store, in the pool, so the pool's fill on the Storage view includes them and says so.

Not in v1, with reasons

Thing Why not yet
VGA console in the browser Incus hands the VM's SPICE socket over a websocket; drawing it needs spice-html5 in the UI. Doable (the Incus UI does it); text console + shell cover every Linux guest until then. Phase 2, with the ISO shelf — an ISO install is useless without it.
Windows VMs Incus runs them (ISO repacked with virtio drivers by distrobuilder), but without a VGA console in the browser they are unusable from the tab. After the above.
GPU passthrough to a VM Removes the card from the host and from whatever uses it there (Jellyfin on grid). Offered only with that warning, later. Sharing a GPU into a container (gpu device) leaves the host its card and is how r720-jellyfin used to run. Phase 3.
Clustering / live migration Incus clusters, but these machines differ in size and one is arm64, with no shared storage. Not planned.
A fleet-wide instance list One machine per view. The switcher is the machine.
A second engine (bare LXC on OpenWrt, libvirt-only hosts) Would double the backend for one router that has no KVM. The tab reports Incus unavailable there and stops.

Layout decisions Ham Lab makes per machine (overridable in the Install dialog)

  1. Storage. A pool on the machine's own disk, driver by the detection above, as a loop-backed image where the driver needs one (incus storage create default zfs size=…) — sized from free space, shown and editable. Bulk data is bind-mounted in, not stored in the pool: on grid that means /tank/… paths into instances, the same split Docker already has. A second pool on an existing dataset (tank/incus) is one extra command if a guest wants terabytes. The pool's path and driver are recorded in Ham Lab's state so the page can say where the data is.
  2. Bridge subnet, computed. Incus's auto picks a random 10.x.y.0/24, and this LAN is 10.0.0.0/8 — every host has a /8 on its NIC, so any 10.x bridge collides. Ham Lab reads the machine's routes and picks the first free /24 from 192.168.0.0/16, then 172.16.0.0/12, that is neither on-link nor routed nor inside Docker's configured pool (172.28.0.0/14 on grid; 172.19–172.29 and 192.168.80–143 live on core). Instances get name.incus DNS from Incus's dnsmasq.
  3. LAN addresses are a later, explicit step. A br0 on the machine's uplink lets an instance take a lease from gate and a Pi-hole name, but it rewrites the host's network config and drops it off the network for seconds — never part of Install, always its own confirmed action (with the NanoKVM standing by on grid). macvlan was considered and rejected: no host change, but host and guests cannot talk, so nothing on the host could reach an instance.
  4. Docker and Incus on one host. Docker sets the FORWARD policy to DROP, which silently kills the Incus bridge's traffic. Fix is two DOCKER-USER accept rules for the bridge, persisted — applied by Install wherever Docker is detected (grid, core, the Mac, hamtv all run it).
  5. Memory. A VM's memory is a hard reservation; the create form shows the machine's free memory and warns past it, and never refuses. On a ZFS host Install offers to cap the ARC (on grid, 40 GiB → 16 GiB) so guest RAM is deterministic; his choice.
  6. Privilege. Ham Lab's user joins incus-admin; no sudo in the hot path. Install itself goes through priv like the other installers.
  7. Backups. Nightly incus export --instance-only of every instance to a per-machine export directory with a retention, as a cron line like the ZFS schedules (hamlab incus-export), from phase 2. Default is under Ham Lab's data directory (so on core the nightly rsync already carries it); on grid he will point it at /tank/backups/incus/, since grid's own root disk is backed up nowhere.

The old Proxmox estate (importable, phase 3)

/tank/backups/proxmox/ holds the retired r720 node: /etc/pve (12 LXC confs, 2 VM confs), 118 GiB of vzdump archives, and three raw VM disks. Latest of each:

ID Name Latest archive Size
100 r720-docker 2026-01-11 20.7 GB
101 r720-cockpit 2026-01-11 439 MB
102 r720-wordpress-alexblaskovich 2026-01-11 1.2 GB
103 r720-spoonterm 2025-06-14 494 MB
104 r720-ollama 2026-01-11 11.3 GB
105 r720-transmission 2026-01-11 322 MB
106 r720-apache 2026-01-11 347 MB
107 r720-nut 2025-03-29 205 MB
108 r720-filebrowser 2026-01-11 262 MB
109 r720-bookstack 2025-04-27 516 MB
110 r720-jellyfin 2026-01-11 2.2 GB
111 r720-wireguard 2025-04-13 510 MB
112 r720-eggwiki 2026-01-11 634 MB
VMs steam (102), powerpanel (105/107/112) .vma.zst 2024–2025; raw disks vm-105, vm-112, base-104 19 GB, 1.9 GB ×6, 97 GiB
  • LXC: feasible and scriptable. Unpack the rootfs (tar --zstd -xpf … --numeric-owner), read etc/vzdump/pct.conf for arch/ostype, drop it, write metadata.yaml, pack, incus image import, launch. Unprivileged uid shifting is Incus's job after that. The result is an image in the Library, so a restored container is launched from there like any other template. An import action in the tab (pick an archive on this machine) is plausible once the script works by hand. x86-64 archives only run on x86-64 machines; the picker says so.
  • VMs: the raw disks import directly (incus-migrate, or an empty VM with the disk imported into its root volume). The .vma.zst archives need Proxmox's vma extractor (or the Python port) to become raw disks first. Most of these are the same powerpanel box backed up six times, and tank-space-audit.md already has them down for deletion — decide which, if any, are wanted before anything is restored.

Build plan

Phase What Effort
1 — detection, Install, the tab internal/incus (detection, CLI wrapper, JSON types, incus monitor for live state, the preseed builder and subnet picker), internal/api/virt.go (routes below), Virtualization.tsx (the three page states; list, detail, create dialog, snapshot panel), rail entry virt, terminal integration, sensors, and the Library's Images shelf (browse and download from images:, local store, import, publish). Tested by clicking Install on grid first, then core — two different pool drivers, both with Docker present. Then the Mac for the arm64 path. gate for the unavailable state. 2–3 sessions
2 — operations The Library's ISO shelf (directory, upload/URL/path, attach, New VM from ISO) with the VGA console (spice-html5), snapshot schedules UI, nightly exports with retention and the per-machine export directory, limits editor, bind mounts, the LAN-bridge step, instance alert default, cloud-init presets. 1–2 sessions
3 — the rest Proxmox import (archives become Library images), GPU device for containers, Windows via distrobuilder repack-windows, GPU passthrough to a VM with its warning. as wanted

API sketch

GET    /api/virt                               {state: ready|installable|unavailable, reason, kvm, arch,
                                                pool_drivers[], pools[], networks[], instances[], free_memory}
POST   /api/virt/install                       {driver, size, path, subnet, docker_rules, arc_max?}  task
GET    /api/virt/images?remote=images&q=       local cache + remote search, filtered to this arch
DELETE /api/virt/images/{fingerprint}
POST   /api/virt/instances                     {name, type, image, cpu, memory, disk, network, cloudinit, profile, start}  task
GET    /api/virt/instances/{name}              config, state, devices, snapshots
PATCH  /api/virt/instances/{name}              limits, description, devices
POST   /api/virt/instances/{name}/{start|stop|restart|kill|delete|rename|publish|export}
POST   /api/virt/instances/{name}/console      → terminal {id}   (incus console)
POST   /api/virt/instances/{name}/exec         → terminal {id}   (incus exec -- sh -lc …)
POST   /api/virt/instances/{name}/snapshots    {name}; POST …/{snap}/restore; DELETE …/{snap}
PUT    /api/virt/instances/{name}/schedule     {cron, expiry}
POST   /api/virt/import                        {path}  task (export tarball, or phase-3 Proxmox archive)

GET    /api/virt/isos                          the ISO directory, scanned: name, size, attached-to
PUT    /api/virt/isos/dir                      {path}  change this machine's ISO directory
POST   /api/virt/isos/fetch                    {url, sha256?}  task;  uploads reuse PUT /api/files/upload
DELETE /api/virt/isos/{name}                   refused while attached
POST   /api/virt/instances/{name}/iso          {iso, boot_first}  attach;  DELETE …/iso  detach
POST   /api/virt/instances/from-iso            {name, iso, cpu, memory, disk, network}  task → console
GET    /api/virt/images/remote?q=&type=        the `images:` catalogue for this arch
POST   /api/virt/images/download               {alias}  task (incus image copy images:… local:)
POST   /api/virt/images/import                 {path | url, alias, type}  task (tarball, or qcow2/raw + metadata)
GET    /api/virt/instances/{name}/vga          websocket: Incus VGA console → spice-html5  (phase 2)

Everything is per machine and reaches a node through the existing /api/nodes/{name}/… proxy with no hub-side code; consoles attach over the existing /api/terminals/{id}/attach. Tasks stream through /api/tasks/{id}/stream as Docker's do.

Sources

Reference in New Issue View Git Blame Copy Permalink