hamlab/VIRTUALIZATION.md

# Virtualization tab — plan (2026-10-09, not started)

A tab for creating, watching, consoling into and deleting **LXC containers and KVM virtual
machines**, the way Proxmox does with `pct` and `qm` — per machine, like every other page.
This is the scoping PLAN.md asked for. Nothing here is built. Live state of the hosts
stays in `hamlet/docs/hosts.md`.

## Decision: Incus 6.0 LTS, on `grid` first

| Option | Verdict |
|---|---|
| **Incus** (LXD fork, linuxcontainers.org) | **Chosen.** One daemon, one CLI and one JSON API for both containers and VMs. In Debian 13 as `incus` 6.0.4 LTS — `apt install incus` pulls `qemu-system-x86`, `swtpm`, `virtiofsd`, `incus-agent`, so VMs work out of the box. Fedora 42 has 6.23 for the Mac mini if that is ever wanted. Native ZFS storage driver (instant snapshots and clones), built-in image server (`images:debian/13`, `images:ubuntu/24.04`, Alpine, Fedora, Arch… as container *or* VM), snapshot schedules with expiry, `incus export` backups, per-instance CPU/memory limits, cloud-init, text and VGA console over websocket. Everything is reachable as `incus … --format json` or `incus query`, which is exactly how Ham Lab already drives Docker: `exec.Command`, no new Go dependency. |
| libvirt/KVM + `virsh`, plus LXC separately | Two daemons, two models, XML. libvirt's official Go binding needs cgo, which the static binary rules out (the pure-Go `go-libvirt` exists but then containers are a second project). Rejected. |
| Proxmox itself | Means reinstalling `grid`'s OS. No. |

## Where it can run

Probed 2026-10-09 (`/dev/kvm`, cpu flags, `kvm_intel`/`kvm` modules, RAM, disk).

| Host | KVM | RAM | Verdict |
|---|---|---|---|
| `grid` | yes — i7-11700T, VT-x, `kvm_intel` loaded, nested on | 62 GiB; ZFS ARC is at 23 GiB and allowed 40 | **The VM host.** 790 GiB free on the NVMe root, 4.5 TiB free on `tank`. Cap the ARC at 16 GiB and ~30 GiB is free for guests with headroom for Immich and the encode jobs. |
| `core` | yes — N100, `kvm_intel` loaded | 7.5 GiB, ~4 GiB available | Small LXC containers only, never a VM. Second host, later. |
| Mac mini | yes — Asahi KVM, `/dev/kvm` present, 16K pages | 15 GiB, 8 GiB available | Technically workable (Fedora ships Incus 6.23 and `qemu-system-aarch64`; arm64 guests only, no x86 VMs; 4K-page guests on the 16K host are what `muvm` already does). **Excluded by the rule: single-core-performance workloads only.** |
| `hamtv` | yes | 3.7 GiB | In service for HamTV. Excluded. |
| `gate` | **no** `/dev/kvm` (OpenWrt) | 8 GiB | Excluded. The tab says "no virtualisation on this machine". |

Nothing is installed anywhere today: no qemu, libvirt, lxc or incus on any host.

## What the tab does (v1)

Mirrors the Docker tab's shape: a list, a detail page, actions that are followed tasks,
destructive ones confirmed by typing the name.

- **Instances** — containers and VMs in one list: state, IPv4, CPU and memory live, disk,
  snapshots count, uptime. Badge for type. Click for detail.
- **Create** — pick container or VM; image from the `images:` remote (searchable, with the
  handful most people want pinned: Debian 13, Ubuntu 24.04, Alpine, Fedora); name, CPUs,
  memory, root disk size; network (NAT bridge, or LAN once `br0` exists — below); optional
  cloud-init user-data (SSH key, packages, a first command); profile. Create, or create and
  start. Collisions (name taken) named before Create.
- **Lifecycle** — start, stop, restart, force stop, rename, delete (type the name; the
  instance's snapshots go with it, and the dialog says so).
- **Console** — two buttons, both open as tabs in the Terminal page through the existing
  PTY manager (`term.CreateCmd`, the `docker exec` path): **Console** (`incus console`,
  the serial console — works before the network is up, shows boot output) and **Shell**
  (`incus exec <n> -- login shell`). Works for containers and, via `incus-agent`, VMs.
- **Snapshots** — take, restore (confirmed), delete; a **schedule** per instance with an
  expiry, which is Incus's own `snapshots.schedule`/`snapshots.expiry`, so no cron file of
  ours this time.
- **Resources** — edit CPU and memory limits (live for containers; VMs take CPU hotplug and
  memory balloon live with the agent, else on restart), grow the root disk, add a bind
  mount (a `tank` path into a container; virtiofs into a VM).
- **Images** — what is cached locally, delete, aliases; **publish** a stopped instance as an
  image (Proxmox's "convert to template").
- **Storage and networks** — the pools with their fill, the bridges with their subnets.
  Read-only in v1 beyond what Create needs.
- **Not installed here** — a host with KVM but no Incus gets an **Install** button that
  runs the whole phase-0 recipe below as a task (the smartmontools-install precedent). A
  host with no KVM gets one sentence.
- **Sensors** — `instance.<name>.cpu`, `.memory`, `.running` from Incus's state, like the
  `container.*` sensors, so Alerts gets an "instance down" default and widgets can show one.

## Not in v1, with reasons

| Thing | Why not yet |
|---|---|
| **VGA console in the browser** | Incus hands the VM's SPICE socket over a websocket; drawing it needs `spice-html5` in the UI. Doable (the Incus UI does it), but text console + shell cover every Linux guest. Phase 3. |
| **Windows VMs** | Incus runs them (ISO repacked with virtio drivers by `distrobuilder`), but without a VGA console in the browser they are unusable from the tab. After the above. |
| **GPU passthrough to a VM** | Takes the Arc A380 from the host, and from Jellyfin. No. Sharing the GPU into a *container* (`gpu` device) is fine and is how `r720-jellyfin` used to run. Phase 3. |
| **Clustering / live migration** | One real host; the others are different sizes and one is arm64; no shared storage. Not planned. |
| **A fleet-wide instance list** | One machine per view. The switcher is the machine. |
| **Mac mini, `hamtv`, `gate`** | Above. |

## Decisions baked in (say if any is wrong)

1. **Storage: instances on `grid`'s NVMe, data on `tank`.** A ZFS pool for Incus made as a
   300 GiB loop file on the NVMe root (`incus storage create default zfs size=300GiB`) — fast
   enough for VM root disks where raidz2 spinners are not, and still ZFS so snapshots and
   clones are instant. Big data is bind-mounted in from `tank`, the same split Docker already
   has (apps on `core`, data on `grid`). A second pool on `tank/incus` can be added later for
   a guest that genuinely wants terabytes. **Consequence:** `grid`'s root NVMe is not backed
   up anywhere (backups land *on* `grid`), so instances need the nightly export below.
2. **Network: NAT bridge first, pinned to `192.168.200.1/24`.** Incus's default picks a
   random `10.x.y.0/24`, and **the LAN here is `10.0.0.0/8`** — every host has a /8 on its
   NIC, so any 10.x bridge collides. `192.168.200.0/24` is clear of Docker's pools on both
   hosts (`172.28.0.0/14` on `grid`; `172.19`–`172.29` and `192.168.80`–`143` on `core`).
   Instances get `name.incus` DNS from Incus's dnsmasq. **LAN addresses in phase 2:** a
   `br0` on `grid`'s `enp10s0` (the 10 GbE uplink) so an instance can take a lease from
   `gate` and a Pi-hole name. That rewrites `/etc/network/interfaces` and drops `grid` off
   the network for seconds — its own step, with the NanoKVM standing by. macvlan was
   considered and rejected: no host network change, but host and guests cannot talk, so
   Immich/Jellyfin/backups on `grid` could never reach an instance.
3. **Docker and Incus on one host:** Docker sets the `FORWARD` policy to `DROP`, which
   silently kills the Incus bridge's traffic. Fix is two `DOCKER-USER` accept rules for
   `incusbr0`, persisted — part of the install recipe, on `grid` and later `core`.
4. **ARC cap on `grid`:** `zfs_arc_max` from 40 GiB to 16 GiB so guest RAM is deterministic.
5. **Privilege:** `core` joins `incus-admin` (socket access, the `docker`-group shape) and
   `hamlab.service` gets `SupplementaryGroups=incus-admin`; no sudo in the hot path.
6. **Backups:** nightly `incus export --instance-only` of every instance to
   `/tank/backups/incus/` with a retention, as a cron line like the ZFS schedules
   (`hamlab incus-export`), from phase 2. Plus the instance's own snapshot schedule.

## The old Proxmox estate (restorable, phase 3)

`/tank/backups/proxmox/` holds the retired `r720` node: `/etc/pve` (12 LXC confs, 2 VM
confs), 118 GiB of `vzdump` archives, and three raw VM disks. Latest of each:

| ID | Name | Latest archive | Size |
|---|---|---|---|
| 100 | r720-docker | 2026-01-11 | 20.7 GB |
| 101 | r720-cockpit | 2026-01-11 | 439 MB |
| 102 | r720-wordpress-alexblaskovich | 2026-01-11 | 1.2 GB |
| 103 | r720-spoonterm | 2025-06-14 | 494 MB |
| 104 | r720-ollama | 2026-01-11 | 11.3 GB |
| 105 | r720-transmission | 2026-01-11 | 322 MB |
| 106 | r720-apache | 2026-01-11 | 347 MB |
| 107 | r720-nut | 2025-03-29 | 205 MB |
| 108 | r720-filebrowser | 2026-01-11 | 262 MB |
| 109 | r720-bookstack | 2025-04-27 | 516 MB |
| 110 | r720-jellyfin | 2026-01-11 | 2.2 GB |
| 111 | r720-wireguard | 2025-04-13 | 510 MB |
| 112 | r720-eggwiki | 2026-01-11 | 634 MB |
| VMs | steam (102), powerpanel (105/107/112) | `.vma.zst` 2024–2025; raw disks `vm-105`, `vm-112`, `base-104` | 19 GB, 1.9 GB ×6, 97 GiB |

- **LXC:** feasible and scriptable. Unpack the rootfs (`tar --zstd -xpf … --numeric-owner`),
  read `etc/vzdump/pct.conf` for arch/ostype, drop it, write `metadata.yaml`, pack, `incus
  image import`, launch. Unprivileged uid shifting is Incus's job after that. A wizard in the
  tab is plausible once the script works by hand.
- **VMs:** the raw disks import directly (`incus-migrate`, or an empty VM with the disk
  imported into its root volume). The `.vma.zst` archives need Proxmox's `vma` extractor (or
  the Python port) to become raw disks first. Most of these are the same `powerpanel` box
  backed up six times, and `tank-space-audit.md` already has them down for deletion — decide
  which, if any, are wanted before anything is restored.

## Build plan

| Phase | What | Effort |
|---|---|---|
| **0 — host prep on `grid`** (by hand, documented) | `apt install incus`; `incus admin init --preseed` with the pool and bridge above; `DOCKER-USER` rules; ARC cap; `incus-admin` group + unit edit; launch one Debian 13 container and one Debian 13 VM; verify console, shell, network, snapshot, export. Write it up in `hamlet/docs/hosts.md` and the preseed into `hamlet/scripts/`. | 1 session |
| **1 — the tab** | `internal/incus` (CLI wrapper, JSON types, `incus monitor` for live state), `internal/api/virt.go` (routes below), `Virtualization.tsx` (list, detail, create dialog, snapshot panel), rail entry `virt`, terminal integration, Install button, the no-KVM state. | 2 sessions |
| **2 — operations** | Snapshot schedules UI, nightly exports with retention, limits editor, bind mounts, `br0` on `grid` and the LAN network option, instance sensors + alert default, cloud-init presets. | 1–2 sessions |
| **3 — the rest** | Proxmox import wizard, publish-as-image, GPU device for containers, VGA console (`spice-html5`), `core` as a second (containers-only) host. | as wanted |

### API sketch

```
GET    /api/virt                               {installed, kvm, pools[], networks[], instances[]}
POST   /api/virt/install                       task: the phase-0 recipe
GET    /api/virt/images?remote=images&q=       local cache + remote search
DELETE /api/virt/images/{fingerprint}
POST   /api/virt/instances                     {name, type, image, cpu, memory, disk, network, cloudinit, profile, start}  task
GET    /api/virt/instances/{name}              config, state, devices, snapshots
PATCH  /api/virt/instances/{name}              limits, description, devices
POST   /api/virt/instances/{name}/{start|stop|restart|kill|delete|rename|publish|export}
POST   /api/virt/instances/{name}/console      → terminal {id}   (incus console)
POST   /api/virt/instances/{name}/exec         → terminal {id}   (incus exec -- sh -lc …)
POST   /api/virt/instances/{name}/snapshots    {name}; POST …/{snap}/restore; DELETE …/{snap}
PUT    /api/virt/instances/{name}/schedule     {cron, expiry}
POST   /api/virt/import                        {path}  task (export tarball, or phase-3 Proxmox archive)
```

Everything is per machine and reaches a node through the existing `/api/nodes/{name}/…`
proxy with no hub-side code; consoles attach over the existing `/api/terminals/{id}/attach`.
Tasks stream through `/api/tasks/{id}/stream` as Docker's do.

## Sources

- Incus on Debian: https://wiki.debian.org/Incus
- Incus and Docker on one host (the `FORWARD` drop): https://linuxcontainers.org/incus/docs/main/howto/network_bridge_firewalld/
- Bridge options and `ipv4.address: auto`: https://linuxcontainers.org/incus/docs/main/reference/network_bridge/
- Consoles (text and VGA): https://linuxcontainers.org/incus/docs/main/howto/instances_console/
- ZFS storage driver: https://linuxcontainers.org/incus/docs/main/reference/storage_zfs/
- macvlan's host-isolation: https://linuxcontainers.org/incus/docs/main/reference/network_macvlan/
- Image format for the Proxmox import: https://linuxcontainers.org/incus/docs/main/reference/image_format/