oswald-infra/flatcar-todo.md

Flatcar on Proxmox: what is left for you

The OpenTofu side is done: nodes.<hv>.flatcar in terraform.tfvars now creates Flatcar VMs (see the Flatcar section in opentofu/README.md for how it works). The items below need access or decisions I do not have. Nothing in this list touches the existing LXCs/VMs.

1. Enable the Snippets content type on local (both nodes)

local already has import enabled (needed for the image), but not snippets (needed for the Ignition file). On each node, as root:

pvesm set local --content backup,vztmpl,import,iso,snippets

Or in the UI: Datacenter → Storage → local → Edit → Content → tick Snippets.

Proxmox does not create the snippets directory when you enable the content type -- you have to do it yourself, or the upload fails with tee: /var/lib/vz/snippets/<name>-ignition.json: No such file or directory:

mkdir -p /var/lib/vz/snippets   # confirm the path via `local`'s entry in /etc/pve/storage.cfg

2. Let the provider SSH to the nodes as root (both nodes)

Proxmox has no API for uploading snippets, so bpg copies them over SSH. The provider is configured (providers.tf, ssh block) to log in as root with the ansible key from Infisical (SSH_PRIVATE_KEY / SSH_PUBLIC_KEY). On each node append the public key to root's authorized_keys:

# from this machine; the key file is materialized by tofu at ~/.ssh/infra_ansible_key
ssh-copy-id -i ~/.ssh/infra_ansible_key.pub [email protected]   # pve
ssh-copy-id -i ~/.ssh/infra_ansible_key.pub [email protected]   # goose

(~/.ssh/infra_ansible_key.pub may not exist yet; paste SSH_PUBLIC_KEY from Infisical into /root/.ssh/authorized_keys on the node instead.)

Then verify from here:

ssh -i ~/.ssh/infra_ansible_key [email protected] 'pvesm status'

If you would rather not give the provider root, bpg supports a non-root user with a narrow sudoers file (pvesm, qm, tee /var/lib/vz/snippets/*). Change username in the ssh blocks accordingly.

3. Declare a VM

Uncomment and adapt the flatcar = { ... } example under nodes.goose in terraform.tfvars. Pick an unused vmid and IP. Optional per-VM config goes in butane/<name>.yaml, referenced with butane_file = "butane/<name>.yaml" on the VM (butane/example.yaml is a starting point; keep variant: flatcar and version: 1.0.0). Inline YAML via butane = <<-EOT ... EOT also works.

I dry-ran a plan with the example VM declared (nothing applied): it creates the image download, the Ignition snippet, the VM and its DNS record, and changes nothing else.

Then:

infisical run --path /terraform -- tofu plan -out plan
infisical run --path /terraform -- tofu apply plan

The first apply on a node downloads the 500 MB Flatcar image into local (flatcar-stable-4593.2.5.img), then creates the VM with that image imported as its boot disk. Expect the download to take a minute or two.

Log in with ssh ansible@<ip> or ssh core@<ip> using the ansible key.

4. Things to know

• Ignition runs on first boot only. Editing the Butane file updates the snippet on the node but does nothing to a running VM. To apply, rebuild it: tofu taint 'module.goose.proxmox_virtual_environment_vm.flatcar["<name>"]' (this destroys the VM's disk; keep data in volumes you can recreate).
• Ignition snippets are plain text on the node. Do not put secrets in butane.
• Flatcar has no Python, so the existing Ansible plays will not run against it. These hosts land in a flatcar group in the generated inventory; only raw/script tasks work there. Ignition is the intended replacement.
• No QEMU guest agent: the module sets agent.enabled = false for Flatcar VMs, so plans never wait on it. Proxmox will not show the VM's IP in the UI.
• Bumping flatcar_image.version (in vars.tf default or tfvars) plus its sha512 from the release .DIGESTS file downloads a new image for future VMs only; existing VMs are not touched (import_from is ignored after creation). Flatcar itself auto-updates in place via its update engine.

5. If the VM does not boot

The Flatcar docs pass --args "-machine sata=off" when creating the VM. I left it out: that option is root-only in Proxmox and the API token cannot set it. If a Flatcar VM hangs at boot, check the serial console (qm terminal <vmid>) first; if it looks firmware-related, set that arg once by hand as root (qm set <vmid> --args "-machine sata=off") and tell me so I can look for an API-safe alternative.

6. oswl-connector: the headscale side

oswl-connector (goose, vmid 212, 192.168.7.62) runs tailscaled in a container against https://hs.atlashorizon.net, advertising 192.168.7.0/24 and itself as an exit node. Three things have to happen outside this repo:

1. Create the preauth key and put it in Infisical as HEADSCALE_AUTHKEY under /terraform (it is read by secrets.tmpl as headscale_authkey). It must be both ephemeral and reusable -- the VM keeps tailscaled's state in memory, so it re-registers on every boot:

   headscale preauthkeys create --user <user> --ephemeral --reusable --expiration 8760h
   

   Until that secret exists, tofu plan fails while rendering butane/oswl_connector.yaml.tftpl (templatefile: no value for "headscale_authkey").

2. Approve the route and the exit node once the VM has registered -- headscale honours neither until enabled:

   headscale nodes list
   headscale nodes approve-routes -i <id> -r 192.168.7.0/24,0.0.0.0/0,::/0
   

   (On headscale older than 0.26 that is headscale routes list plus headscale routes enable -r <route-id>, once per route.)

   Because the node is ephemeral it re-registers on every boot, so this has to be redone after each reboot and after each tofu taint rebuild. Headscale can do it automatically, but the policy has to exist before the node advertises the tag -- tailscale up --advertise-tags fails outright if the tag is not owned by the registering user, which takes tailscaled down with it. So: add the policy first, confirm it loads, and only then add --advertise-tags=tag:oswl-connector to TS_EXTRA_ARGS in butane/oswl_connector.yaml.tftpl and rebuild the VM.

   {
     "tagOwners": {
       "tag:oswl-connector": ["<user>@"]
     },
     "autoApprovers": {
       "routes": {
         "192.168.7.0/24": ["tag:oswl-connector"]
       },
       "exitNode": ["tag:oswl-connector"]
     }
   }
   

   <user>@ is the user the preauth key belongs to (policy v2 spells usernames with a trailing @). Load the policy with policy.mode: file + policy.path in headscale's config.yaml, or headscale policy set -f policy.json in database mode.

   Two further caveats if you go that route:

   • List the tag in autoApprovers, not a user or a group. Approval by username or group silently does not fire for a node whose identity is a tag (#2891, open as of 0.27.1). Naming the tag is both the documented form and the workaround.
   • A tagged node is governed by the tag in ACLs, not by its user. With headscale's default allow-all policy that changes nothing, but if you later write real ACL rules, tag:oswl-connector needs its own grants.

3. Check the LAN's return path. The VM has one NIC on VLAN 7, so clients reaching 192.168.7.0/24 through the tunnel arrive with a tailnet source address. The container runs with --network host and tailscaled installs its own SNAT rules by default (--snat-subnet-routes), so this works without touching the router; drop that default only if you add a route for the tailnet CIDR on the 192.168.7.0/24 gateway.

Note the same caveat as the Gitea runner: the auth key ends up in plaintext in the rendered Ignition snippet under /var/lib/vz/snippets on goose.

7. Commit

Nothing is committed. The opentofu submodule has the module changes; commit and push it first, then commit the parent repo (which records the new submodule pointer plus providers.tf, vars.tf, main.tf, terraform.tfvars, secrets.tmpl, butane/, and this file).