Virtualization plan: every artefact under Ham Lab's data directory, one virt/ tree per machine

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

VIRTUALIZATION.md

@@ -26,11 +26,20 @@ So:
   single-core workloads, `hamtv` not spare hardware) bind the *agent's* unasked
   deployments. They are not feature gates for the product, and the first draft was wrong
   to treat them as such.
-- **Ham Lab owns the install and the data layout.** It detects what is missing, offers to
-  install it where it knows how, chooses and records where instance storage lives and which
-  subnet the bridge takes, and runs the backups. Where it cannot install or the hardware
-  cannot do the job, it says so in one sentence and offers what *does* work (containers
-  without KVM, for instance).
+- **Ham Lab owns its data, and its data lives in its own folder.** Everything this tab
+  manages — instance storage, ISOs, image tarballs, exports — sits under Ham Lab's data
+  directory on *that* machine (`config.DataDir()`, `~/apps/hamlab` on `core`, `grid` and
+  the Mac), in one `virt/` tree. Never a path elsewhere on the machine, however convenient
+  the existing folder. Stated 2026-10-09 after the first Library draft pointed the ISO shelf
+  at `/tank/software/isos/`: *"ALL data needs to be under a HamLab folder, because it
+  manages its own data."* This is the rule that moved `core`'s compose files under
+  `apps/hamlab/stacks/` on 2026-10-05; `grid`'s and the Mac's stacks, still at
+  `~/stacks/<machine>/`, are the outstanding exception and are not touched by this plan.
+  The engine's own runtime state (`/var/lib/incus`, like `/var/lib/docker`) is the engine's,
+  not Ham Lab's data: Ham Lab keeps what it would need to rebuild it.
+- **Ham Lab installs or reports.** It detects what is missing, offers to install it where it
+  knows how, and where it cannot or the hardware cannot do the job, says so in one sentence
+  and offers what *does* work (containers without KVM, for instance).
 
 ## Decision: Incus 6.0 LTS as the engine
 
@@ -78,8 +87,9 @@ destructive ones confirmed by typing the name.
   add Ham Lab's user to `incus-admin` (socket access, the `docker`-group shape) and add it
   to the unit's `SupplementaryGroups`, apply the Docker fix where Docker is present (below),
   then restart Ham Lab. Idempotent, so it doubles as "repair". The dialog shows and lets him
-  edit: pool driver (detected), pool size (proposed from free space), pool path, bridge
-  subnet (computed), whether to apply the Docker rules.
+  edit: pool driver (detected), pool size (proposed from free space), bridge subnet
+  (computed), whether to apply the Docker rules. Where it all goes is not a choice: Ham
+  Lab's folder.
 - **Instances** — containers and VMs in one list: type, state, IPv4, CPU and memory live,
   disk, snapshot count, uptime. Click for detail.
 - **Create** — container or VM (VM only where KVM is); image from the `images:` remote,
@@ -107,6 +117,37 @@ destructive ones confirmed by typing the name.
 - **Sensors** — `instance.<name>.cpu`, `.memory`, `.running` from Incus's state, like the
   `container.*` sensors, so Alerts gets an "instance down" default and widgets can show one.
 
+## Where the data lives
+
+One tree per machine, under Ham Lab's data directory, created by Install:
+
+```
+~/apps/hamlab/virt/
+  pool.img          the instance pool's backing file (zfs or btrfs loop; a plain directory
+                    pool/ when the driver is dir) — every instance's disks live in here
+  isos/             the ISO library: the files themselves
+  images/           the image library: template tarballs and VM disk images as files, the
+                    way Proxmox keeps CT templates — Incus's image store is loaded from here
+  exports/          nightly `incus export` tarballs, with retention
+  virt.json         what Ham Lab chose and when: driver, pool size, bridge subnet, ISO and
+                    image inventory, export schedule
+```
+
+Consequences, stated once:
+
+- The tree is on whatever disk holds Ham Lab's data directory — the boot NVMe on all three.
+  Install proposes the pool size from that disk's free space. On `grid` that disk has
+  790 GiB free; on `core`, 175 GiB.
+- Bulk data a guest needs is **bind-mounted in from where it already is** (`/tank/…` into an
+  instance on `grid`), the same split Docker has: the instance is Ham Lab's, the media is
+  not. That is a device on the instance, recorded in its config, not a second home.
+- On `core` the whole tree rides the existing nightly rsync to `grid` for free. On `grid`
+  and the Mac it is as backed up as the rest of their Ham Lab folders — which is to say the
+  exports are the backup, and the backup job will need to be told about them in phase 2.
+- Existing material elsewhere — the 105 GiB of ISOs at `/tank/software/isos/`, the Proxmox
+  archives — is **imported into the tree by copying**, file by file, on request. The
+  library never indexes a folder it does not own.
+
 ## Library: ISOs and images (asked for 2026-10-09)
 
 Proxmox has a storage view with **ISO Images** and **CT Templates**. Incus has the same two
@@ -115,19 +156,19 @@ like everything else.
 
 | Proxmox calls it | Incus calls it | What it is here |
 |---|---|---|
-| ISO Images | an ISO file attached to a VM as a `disk` device (added as a ROM device; VMs only), or a custom storage volume of type `iso` | **A directory of `.iso` files on the machine that Ham Lab indexes** — no copy into the pool |
-| CT Templates | **images** — the `images:` remote (linuxcontainers.org), the local image store, anything imported or published | **The local image store**, with the remote as the catalogue to pull from |
+| ISO Images | an ISO file attached to a VM as a `disk` device (added as a ROM device; VMs only), or a custom storage volume of type `iso` | **`virt/isos/` in Ham Lab's folder**, attached by path — no copy into the pool |
+| CT Templates | **images** — the `images:` remote (linuxcontainers.org), the local image store, anything imported or published | **`virt/images/` in Ham Lab's folder** — tarballs and disk images as files, loaded into Incus from there; the remote is the catalogue to pull from |
 
 ### ISOs
 
-- **One ISO directory per machine**, chosen in the Install dialog and changeable later.
-  Default is under Ham Lab's data directory; on `grid` it should be the existing
-  **`/tank/software/isos/`** — 105 GiB already there (`linux/`, `windows/`, `truenas/`,
-  `virtio_drivers/`, `pi/`, `hackberry/`, `sipeed/`), scanned recursively.
+- **The ISO library is `virt/isos/` in Ham Lab's folder**, on every machine, full stop. The
+  105 GiB at `/tank/software/isos/` on `grid` (`linux/`, `windows/`, `truenas/`,
+  `virtio_drivers/`…) is a *source* to import from — pick the ones wanted, Ham Lab copies them
+  in — not the library itself.
 - **Getting an ISO in:** upload from the browser (the Files tab's `PUT /api/files/upload`
   with its progress bar, reused), **download from a URL** as a followed task with an optional
   checksum (the usual way — an ISO starts as a link), or **import from a path** already on the
-  machine. Rename, delete (refused while a VM has it attached, naming the VM).
+  machine — which **copies** the file into `virt/isos/` and says so. Rename, delete (refused while a VM has it attached, naming the VM).
 - **Using one:** attach to / detach from a VM as a ROM device by path, with `boot.priority`
   set so it boots first; **New VM from ISO** makes an empty VM (`incus init --empty --vm`)
   with the ISO attached, starts it and opens the console. After install, Detach is one click
@@ -141,8 +182,8 @@ like everything else.
   and the VGA console ship together, in phase 2**, moved up from phase 3.
 - ISO *volumes* (`incus storage volume import <pool> <iso> <name> --type=iso`) are the
   alternative: read-only, attachable to several VMs at once, but they copy the file into the
-  pool. Not the default here — the directory honours the library that already exists and
-  keeps 105 GiB out of a 300 GiB pool. Offered as an option only if a need shows up.
+  pool. Not used here — a file in `virt/isos/` is already Ham Lab's, and a volume would copy it
+  a second time into the pool. Offered as an option only if a need shows up.
 
 ### Images (container templates and VM images)
 
@@ -161,8 +202,10 @@ like everything else.
   image — by upload, URL, or a path on the machine. Same three ways in as ISOs.
 - **Publish:** a stopped instance becomes an image with an alias — Proxmox's "convert to
   template", the one thing the Images section of v1 always had.
-- **Where they live:** Incus's image store, in the pool, so the pool's fill on the Storage
-  view includes them and says so.
+- **Where they live:** as files in `virt/images/` — the downloaded or imported tarball, the
+  published image exported to one (`incus image export`) — so the library is complete without
+  Incus and can be reloaded into a fresh install. Incus's own image store under
+  `/var/lib/incus/images/` is a cache derived from it, and the page says so.
 
 ## Not in v1, with reasons
 
@@ -177,13 +220,12 @@ like everything else.
 
 ## Layout decisions Ham Lab makes per machine (overridable in the Install dialog)
 
-1. **Storage.** A pool on the machine's own disk, driver by the detection above, as a
-   loop-backed image where the driver needs one (`incus storage create default zfs
-   size=…`) — sized from free space, shown and editable. Bulk data is **bind-mounted in**,
-   not stored in the pool: on `grid` that means `/tank/…` paths into instances, the same
-   split Docker already has. A second pool on an existing dataset (`tank/incus`) is one
-   extra command if a guest wants terabytes. The pool's path and driver are recorded in
-   Ham Lab's state so the page can say where the data is.
+1. **Storage.** The instance pool's backing lives in Ham Lab's folder: Install creates a
+   sparse `virt/pool.img` and hands it to Incus as the pool's `source` (zfs or btrfs by the
+   detection above), or `virt/pool/` as a `dir` pool where neither is available. Sized from
+   the disk's free space, shown and editable. Bulk data is **bind-mounted in**, not stored in
+   the pool. The driver, path and size are recorded in `virt.json` so the page can say where
+   the data is and Install can rebuild the pool from the file after a reinstall.
 2. **Bridge subnet, computed.** Incus's `auto` picks a random `10.x.y.0/24`, and **this
    LAN is `10.0.0.0/8`** — every host has a /8 on its NIC, so any 10.x bridge collides.
    Ham Lab reads the machine's routes and picks the first free /24 from `192.168.0.0/16`,
@@ -205,11 +247,10 @@ like everything else.
    the ARC (on `grid`, 40 GiB → 16 GiB) so guest RAM is deterministic; his choice.
 6. **Privilege.** Ham Lab's user joins `incus-admin`; no sudo in the hot path. Install
    itself goes through `priv` like the other installers.
-7. **Backups.** Nightly `incus export --instance-only` of every instance to a per-machine
-   export directory with a retention, as a cron line like the ZFS schedules
-   (`hamlab incus-export`), from phase 2. Default is under Ham Lab's data directory (so on
-   `core` the nightly rsync already carries it); on `grid` he will point it at
-   `/tank/backups/incus/`, since `grid`'s own root disk is backed up nowhere.
+7. **Backups.** Nightly `incus export --instance-only` of every instance to `virt/exports/`
+   with a retention, as a cron line like the ZFS schedules (`hamlab incus-export`), from
+   phase 2. On `core` that folder is inside the nightly rsync already; on `grid` and the Mac
+   the backup job gets a line for it in phase 2.
 
 ## The old Proxmox estate (importable, phase 3)
 
@@ -271,15 +312,15 @@ POST   /api/virt/instances/{name}/snapshots    {name}; POST …/{snap}/restore;
 PUT    /api/virt/instances/{name}/schedule     {cron, expiry}
 POST   /api/virt/import                        {path}  task (export tarball, or phase-3 Proxmox archive)
 
-GET    /api/virt/isos                          the ISO directory, scanned: name, size, attached-to
-PUT    /api/virt/isos/dir                      {path}  change this machine's ISO directory
+GET    /api/virt/isos                          virt/isos/, scanned: name, size, attached-to
+POST   /api/virt/isos/import                   {path}  task: copy a file on this machine into virt/isos/
 POST   /api/virt/isos/fetch                    {url, sha256?}  task;  uploads reuse PUT /api/files/upload
 DELETE /api/virt/isos/{name}                   refused while attached
 POST   /api/virt/instances/{name}/iso          {iso, boot_first}  attach;  DELETE …/iso  detach
 POST   /api/virt/instances/from-iso            {name, iso, cpu, memory, disk, network}  task → console
 GET    /api/virt/images/remote?q=&type=        the `images:` catalogue for this arch
 POST   /api/virt/images/download               {alias}  task (incus image copy images:… local:)
-POST   /api/virt/images/import                 {path | url, alias, type}  task (tarball, or qcow2/raw + metadata)
+POST   /api/virt/images/import                 {path | url, alias, type}  task: copy into virt/images/, then load into Incus
 GET    /api/virt/instances/{name}/vga          websocket: Incus VGA console → spice-html5  (phase 2)
 ```