Flatcar on Proxmox: what is left for you

The OpenTofu side is done: nodes.<hv>.flatcar in terraform.tfvars now creates Flatcar VMs (see the Flatcar section in opentofu/README.md for how it works). The items below need access or decisions I do not have. Nothing in this list touches the existing LXCs/VMs.

1. Enable the Snippets content type on `local` (both nodes)

local already has import enabled (needed for the image), but not snippets (needed for the Ignition file). On each node, as root:

pvesm set local --content backup,vztmpl,import,iso,snippets

Or in the UI: Datacenter → Storage → local → Edit → Content → tick Snippets.

Proxmox does not create the snippets directory when you enable the content type -- you have to do it yourself, or the upload fails with tee: /var/lib/vz/snippets/<name>-ignition.json: No such file or directory:

mkdir -p /var/lib/vz/snippets   # confirm the path via `local`'s entry in /etc/pve/storage.cfg

2. Let the provider SSH to the nodes as root (both nodes)

Proxmox has no API for uploading snippets, so bpg copies them over SSH. The provider is configured (providers.tf, ssh block) to log in as root with the ansible key from Infisical (SSH_PRIVATE_KEY / SSH_PUBLIC_KEY). On each node append the public key to root's authorized_keys:

# from this machine; the key file is materialized by tofu at ~/.ssh/infra_ansible_key
ssh-copy-id -i ~/.ssh/infra_ansible_key.pub [email protected]   # pve
ssh-copy-id -i ~/.ssh/infra_ansible_key.pub [email protected]   # goose

(~/.ssh/infra_ansible_key.pub may not exist yet; paste SSH_PUBLIC_KEY from Infisical into /root/.ssh/authorized_keys on the node instead.)

Then verify from here:

ssh -i ~/.ssh/infra_ansible_key [email protected] 'pvesm status'

If you would rather not give the provider root, bpg supports a non-root user with a narrow sudoers file (pvesm, qm, tee /var/lib/vz/snippets/*). Change username in the ssh blocks accordingly.

3. Declare a VM

Uncomment and adapt the flatcar = { ... } example under nodes.goose in terraform.tfvars. Pick an unused vmid and IP. Optional per-VM config goes in butane/<name>.yaml, referenced with butane_file = "butane/<name>.yaml" on the VM (butane/example.yaml is a starting point; keep variant: flatcar and version: 1.0.0). Inline YAML via butane = <<-EOT ... EOT also works.

I dry-ran a plan with the example VM declared (nothing applied): it creates the image download, the Ignition snippet, the VM and its DNS record, and changes nothing else.

Then:

infisical run --path /terraform -- tofu plan -out plan
infisical run --path /terraform -- tofu apply plan

The first apply on a node downloads the 500 MB Flatcar image into local (flatcar-stable-4593.2.5.img), then creates the VM with that image imported as its boot disk. Expect the download to take a minute or two.

4. Things to know

Ignition runs on first boot only. Editing the Butane file updates the snippet on the node but does nothing to a running VM. To apply, rebuild it: tofu taint 'module.goose.proxmox_virtual_environment_vm.flatcar["<name>"]' (this destroys the VM's disk; keep data in volumes you can recreate).
Ignition snippets are plain text on the node. Do not put secrets in butane.
Flatcar has no Python, so the existing Ansible plays will not run against it. These hosts land in a flatcar group in the generated inventory; only raw/script tasks work there. Ignition is the intended replacement.
No QEMU guest agent: the module sets agent.enabled = false for Flatcar VMs, so plans never wait on it. Proxmox will not show the VM's IP in the UI.
Bumping flatcar_image.version (in vars.tf default or tfvars) plus its sha512 from the release .DIGESTS file downloads a new image for future VMs only; existing VMs are not touched (import_from is ignored after creation). Flatcar itself auto-updates in place via its update engine.

5. If the VM does not boot

The Flatcar docs pass --args "-machine sata=off" when creating the VM. I left it out: that option is root-only in Proxmox and the API token cannot set it. If a Flatcar VM hangs at boot, check the serial console (qm terminal <vmid>) first; if it looks firmware-related, set that arg once by hand as root (qm set <vmid> --args "-machine sata=off") and tell me so I can look for an API-safe alternative.

6. oswl-connector: the headscale side

oswl-connector (goose, vmid 212, 192.168.7.62) runs tailscaled in a container against https://hs.atlashorizon.net, advertising 192.168.7.0/24 and itself as an exit node. Three things have to happen outside this repo:

Create the preauth key and put it in Infisical as HEADSCALE_AUTHKEY under /terraform (it is read by secrets.tmpl as headscale_authkey). It must be both ephemeral and reusable -- the VM keeps tailscaled's state in memory, so it re-registers on every boot:

The key also carries the tag that drives route auto-approval (see 2), so the VM never has to pass --advertise-tags itself:
```
headscale preauthkeys create --user <user> --ephemeral --reusable \
  --tags tag:oswl-connector --expiration 8760h
```
Until that secret exists, tofu plan fails while rendering butane/oswl_connector.yaml.tftpl (templatefile: no value for "headscale_authkey").
Route approval is automatic, via the key's tag. The preauth key applies tag:oswl-connector as a forced tag at registration, and headscale's autoApprovers policy approves the subnet route and the exit node for that tag. This is what makes the node's ephemerality survivable: it re-registers on every boot and after every tofu taint rebuild, and each time the routes come back approved without anyone touching headscale.

The policy this depends on:
```
{
  "tagOwners": {
    "tag:oswl-connector": ["<user>@"]
  },
  "autoApprovers": {
    "routes": {
      "192.168.7.0/24": ["tag:oswl-connector"]
    },
    "exitNode": ["tag:oswl-connector"]
  }
}
```
Two things worth knowing if it ever stops approving:
- autoApprovers must name the tag, not a user or a group. Approval by username or group silently does not fire for a node whose identity is a forced tag (#2891, open as of 0.27.1).
- A tagged node is governed by the tag in ACLs, not by its user. With headscale's default allow-all policy that changes nothing, but if you later write real ACL rules, tag:oswl-connector needs its own grants.
To approve by hand (if the policy is ever removed):
```
headscale nodes list
headscale nodes approve-routes -i <id> -r 192.168.7.0/24,0.0.0.0/0,::/0
```
Do not add --advertise-tags to TS_EXTRA_ARGS to try to set the tag from the client side: the key already sets it, and tailscale up fails outright if the registering user does not own the tag, which takes tailscaled down with it.
Check the LAN's return path. The VM has one NIC on VLAN 7, so clients reaching 192.168.7.0/24 through the tunnel arrive with a tailnet source address. The container runs with --network host and tailscaled installs its own SNAT rules by default (--snat-subnet-routes), so this works without touching the router; drop that default only if you add a route for the tailnet CIDR on the 192.168.7.0/24 gateway.

That SNAT is only real if the netfilter modules are loaded -- the container has NET_ADMIN but no SYS_MODULE and no /lib/modules, so it cannot load them itself. The modprobe -a line in tailscaled.service loads them on the host, where the container picks them up through --network host. If tailscale ever reports "packet filter not installed" or "cannot create the ts-input chain", that line is what to check:
```
lsmod | grep -E 'iptable|ip6table|nf_nat|nf_tables'
docker exec tailscaled iptables -t nat -S | grep ts-postrouting
```

Note the same caveat as the Gitea runner: the auth key ends up in plaintext in the rendered Ignition snippet under /var/lib/vz/snippets on goose.

7. Commit

Nothing is committed. The opentofu submodule has the module changes; commit and push it first, then commit the parent repo (which records the new submodule pointer plus providers.tf, vars.tf, main.tf, terraform.tfvars, secrets.tmpl, butane/, and this file).

8.5 KiB Raw Blame History