oswald-infra/flatcar-todo.md

# Flatcar on Proxmox: what is left for you

The OpenTofu side is done: `nodes.<hv>.flatcar` in `terraform.tfvars` now creates
Flatcar VMs (see the Flatcar section in `opentofu/README.md` for how it works).
The items below need access or decisions I do not have. Nothing in this list
touches the existing LXCs/VMs.

## 1. Enable the Snippets content type on `local` (both nodes)

`local` already has `import` enabled (needed for the image), but not `snippets`
(needed for the Ignition file). On each node, as root:

```bash
pvesm set local --content backup,vztmpl,import,iso,snippets
```

Or in the UI: Datacenter → Storage → local → Edit → Content → tick Snippets.

Proxmox does **not** create the snippets directory when you enable the content
type -- you have to do it yourself, or the upload fails with `tee:
/var/lib/vz/snippets/<name>-ignition.json: No such file or directory`:

```bash
mkdir -p /var/lib/vz/snippets   # confirm the path via `local`'s entry in /etc/pve/storage.cfg
```

## 2. Let the provider SSH to the nodes as root (both nodes)

Proxmox has no API for uploading snippets, so bpg copies them over SSH. The
provider is configured (`providers.tf`, `ssh` block) to log in as `root` with
the ansible key from Infisical (`SSH_PRIVATE_KEY` / `SSH_PUBLIC_KEY`). On each
node append the public key to root's authorized_keys:

```bash
# from this machine; the key file is materialized by tofu at ~/.ssh/infra_ansible_key
ssh-copy-id -i ~/.ssh/infra_ansible_key.pub [email protected]   # pve
ssh-copy-id -i ~/.ssh/infra_ansible_key.pub [email protected]   # goose
```

(`~/.ssh/infra_ansible_key.pub` may not exist yet; paste `SSH_PUBLIC_KEY` from
Infisical into `/root/.ssh/authorized_keys` on the node instead.)

Then verify from here:

```bash
ssh -i ~/.ssh/infra_ansible_key [email protected] 'pvesm status'
```

If you would rather not give the provider root, bpg supports a non-root user
with a narrow sudoers file (`pvesm`, `qm`, `tee /var/lib/vz/snippets/*`). Change
`username` in the `ssh` blocks accordingly.

## 3. Declare a VM

Uncomment and adapt the `flatcar = { ... }` example under `nodes.goose` in
`terraform.tfvars`. Pick an unused vmid and IP. Optional per-VM config goes in
`butane/<name>.yaml`, referenced with `butane_file = "butane/<name>.yaml"` on
the VM (`butane/example.yaml` is a starting point; keep `variant: flatcar` and
`version: 1.0.0`). Inline YAML via `butane = <<-EOT ... EOT` also works.

I dry-ran a plan with the example VM declared (nothing applied): it creates
the image download, the Ignition snippet, the VM and its DNS record, and
changes nothing else.

Then:

```bash
infisical run --path /terraform -- tofu plan -out plan
infisical run --path /terraform -- tofu apply plan
```

The first apply on a node downloads the 500 MB Flatcar image into `local`
(`flatcar-stable-4593.2.5.img`), then creates the VM with that image imported
as its boot disk. Expect the download to take a minute or two.

Log in with `ssh ansible@<ip>` or `ssh core@<ip>` using the ansible key.

## 4. Things to know

- Ignition runs on first boot only. Editing the Butane file updates the snippet on the
  node but does nothing to a running VM. To apply, rebuild it:
  `tofu taint 'module.goose.proxmox_virtual_environment_vm.flatcar["<name>"]'`
  (this destroys the VM's disk; keep data in volumes you can recreate).
- Ignition snippets are plain text on the node. Do not put secrets in `butane`.
- Flatcar has no Python, so the existing Ansible plays will not run against it.
  These hosts land in a `flatcar` group in the generated inventory; only
  `raw`/`script` tasks work there. Ignition is the intended replacement.
- No QEMU guest agent: the module sets `agent.enabled = false` for Flatcar VMs,
  so plans never wait on it. Proxmox will not show the VM's IP in the UI.
- Bumping `flatcar_image.version` (in `vars.tf` default or tfvars) plus its
  sha512 from the release `.DIGESTS` file downloads a new image for future VMs
  only; existing VMs are not touched (`import_from` is ignored after creation).
  Flatcar itself auto-updates in place via its update engine.

## 5. If the VM does not boot

The Flatcar docs pass `--args "-machine sata=off"` when creating the VM. I left
it out: that option is root-only in Proxmox and the API token cannot set it.
If a Flatcar VM hangs at boot, check the serial console (`qm terminal <vmid>`)
first; if it looks firmware-related, set that arg once by hand as root
(`qm set <vmid> --args "-machine sata=off"`) and tell me so I can look for an
API-safe alternative.

## 6. oswl-connector: the headscale side

`oswl-connector` (goose, vmid 212, 192.168.7.62) runs tailscaled in a container
against `https://hs.atlashorizon.net`, advertising `192.168.7.0/24` and itself
as an exit node. Three things have to happen outside this repo:

1. **Create the preauth key and put it in Infisical** as `HEADSCALE_AUTHKEY`
   under `/terraform` (it is read by `secrets.tmpl` as `headscale_authkey`).
   It must be both ephemeral and reusable -- the VM keeps tailscaled's state in
   memory, so it re-registers on every boot:

   The key also carries the tag that drives route auto-approval (see 2), so
   the VM never has to pass `--advertise-tags` itself:

   ```bash
   headscale preauthkeys create --user <user> --ephemeral --reusable \
     --tags tag:oswl-connector --expiration 8760h
   ```

   Until that secret exists, `tofu plan` fails while rendering
   `butane/oswl_connector.yaml.tftpl` (`templatefile`: no value for
   "headscale_authkey").

2. **Route approval is automatic, via the key's tag.** The preauth key applies
   `tag:oswl-connector` as a forced tag at registration, and headscale's
   `autoApprovers` policy approves the subnet route and the exit node for that
   tag. This is what makes the node's ephemerality survivable: it re-registers
   on every boot and after every `tofu taint` rebuild, and each time the routes
   come back approved without anyone touching headscale.

   The policy this depends on:

   ```json
   {
     "tagOwners": {
       "tag:oswl-connector": ["<user>@"]
     },
     "autoApprovers": {
       "routes": {
         "192.168.7.0/24": ["tag:oswl-connector"]
       },
       "exitNode": ["tag:oswl-connector"]
     }
   }
   ```

   Two things worth knowing if it ever stops approving:

   - **`autoApprovers` must name the tag, not a user or a group.** Approval by
     username or group silently does not fire for a node whose identity is a
     forced tag ([#2891](https://github.com/juanfont/headscale/issues/2891),
     open as of 0.27.1).
   - **A tagged node is governed by the tag in ACLs, not by its user.** With
     headscale's default allow-all policy that changes nothing, but if you
     later write real ACL rules, `tag:oswl-connector` needs its own grants.

   To approve by hand (if the policy is ever removed):

   ```bash
   headscale nodes list
   headscale nodes approve-routes -i <id> -r 192.168.7.0/24,0.0.0.0/0,::/0
   ```

   Do not add `--advertise-tags` to `TS_EXTRA_ARGS` to try to set the tag from
   the client side: the key already sets it, and `tailscale up` fails outright
   if the registering user does not own the tag, which takes tailscaled down
   with it.

3. **Check the LAN's return path.** The VM has one NIC on VLAN 7, so clients
   reaching 192.168.7.0/24 through the tunnel arrive with a tailnet source
   address. The container runs with `--network host` and tailscaled installs
   its own SNAT rules by default (`--snat-subnet-routes`), so this works
   without touching the router; drop that default only if you add a route for
   the tailnet CIDR on the 192.168.7.0/24 gateway.

   That SNAT is only real if the netfilter modules are loaded -- the container
   has NET_ADMIN but no SYS_MODULE and no `/lib/modules`, so it cannot load
   them itself. The `modprobe -a` line in `tailscaled.service` loads them on
   the host, where the container picks them up through `--network host`. If
   tailscale ever reports "packet filter not installed" or
   "cannot create the ts-input chain", that line is what to check:

   ```bash
   lsmod | grep -E 'iptable|ip6table|nf_nat|nf_tables'
   docker exec tailscaled iptables -t nat -S | grep ts-postrouting
   ```

Note the same caveat as the Gitea runner: the auth key ends up in plaintext in
the rendered Ignition snippet under `/var/lib/vz/snippets` on goose.

## 7. Commit

Nothing is committed. The `opentofu` submodule has the module changes; commit
and push it first, then commit the parent repo (which records the new
submodule pointer plus `providers.tf`, `vars.tf`, `main.tf`,
`terraform.tfvars`, `secrets.tmpl`, `butane/`, and this file).