# Flatcar on Proxmox: what is left for you The OpenTofu side is done: `nodes..flatcar` in `terraform.tfvars` now creates Flatcar VMs (see the Flatcar section in `opentofu/README.md` for how it works). The items below need access or decisions I do not have. Nothing in this list touches the existing LXCs/VMs. ## 1. Enable the Snippets content type on `local` (both nodes) `local` already has `import` enabled (needed for the image), but not `snippets` (needed for the Ignition file). On each node, as root: ```bash pvesm set local --content backup,vztmpl,import,iso,snippets ``` Or in the UI: Datacenter → Storage → local → Edit → Content → tick Snippets. Proxmox does **not** create the snippets directory when you enable the content type -- you have to do it yourself, or the upload fails with `tee: /var/lib/vz/snippets/-ignition.json: No such file or directory`: ```bash mkdir -p /var/lib/vz/snippets # confirm the path via `local`'s entry in /etc/pve/storage.cfg ``` ## 2. Let the provider SSH to the nodes as root (both nodes) Proxmox has no API for uploading snippets, so bpg copies them over SSH. The provider is configured (`providers.tf`, `ssh` block) to log in as `root` with the ansible key from Infisical (`SSH_PRIVATE_KEY` / `SSH_PUBLIC_KEY`). On each node append the public key to root's authorized_keys: ```bash # from this machine; the key file is materialized by tofu at ~/.ssh/infra_ansible_key ssh-copy-id -i ~/.ssh/infra_ansible_key.pub root@192.168.7.240 # pve ssh-copy-id -i ~/.ssh/infra_ansible_key.pub root@192.168.7.241 # goose ``` (`~/.ssh/infra_ansible_key.pub` may not exist yet; paste `SSH_PUBLIC_KEY` from Infisical into `/root/.ssh/authorized_keys` on the node instead.) Then verify from here: ```bash ssh -i ~/.ssh/infra_ansible_key root@192.168.7.241 'pvesm status' ``` If you would rather not give the provider root, bpg supports a non-root user with a narrow sudoers file (`pvesm`, `qm`, `tee /var/lib/vz/snippets/*`). Change `username` in the `ssh` blocks accordingly. ## 3. Declare a VM Uncomment and adapt the `flatcar = { ... }` example under `nodes.goose` in `terraform.tfvars`. Pick an unused vmid and IP. Optional per-VM config goes in `butane/.yaml`, referenced with `butane_file = "butane/.yaml"` on the VM (`butane/example.yaml` is a starting point; keep `variant: flatcar` and `version: 1.0.0`). Inline YAML via `butane = <<-EOT ... EOT` also works. I dry-ran a plan with the example VM declared (nothing applied): it creates the image download, the Ignition snippet, the VM and its DNS record, and changes nothing else. Then: ```bash infisical run --path /terraform -- tofu plan -out plan infisical run --path /terraform -- tofu apply plan ``` The first apply on a node downloads the 500 MB Flatcar image into `local` (`flatcar-stable-4593.2.5.img`), then creates the VM with that image imported as its boot disk. Expect the download to take a minute or two. Log in with `ssh ansible@` or `ssh core@` using the ansible key. ## 4. Things to know - Ignition runs on first boot only. Editing the Butane file updates the snippet on the node but does nothing to a running VM. To apply, rebuild it: `tofu taint 'module.goose.proxmox_virtual_environment_vm.flatcar[""]'` (this destroys the VM's disk; keep data in volumes you can recreate). - Ignition snippets are plain text on the node. Do not put secrets in `butane`. - Flatcar has no Python, so the existing Ansible plays will not run against it. These hosts land in a `flatcar` group in the generated inventory; only `raw`/`script` tasks work there. Ignition is the intended replacement. - No QEMU guest agent: the module sets `agent.enabled = false` for Flatcar VMs, so plans never wait on it. Proxmox will not show the VM's IP in the UI. - Bumping `flatcar_image.version` (in `vars.tf` default or tfvars) plus its sha512 from the release `.DIGESTS` file downloads a new image for future VMs only; existing VMs are not touched (`import_from` is ignored after creation). Flatcar itself auto-updates in place via its update engine. ## 5. If the VM does not boot The Flatcar docs pass `--args "-machine sata=off"` when creating the VM. I left it out: that option is root-only in Proxmox and the API token cannot set it. If a Flatcar VM hangs at boot, check the serial console (`qm terminal `) first; if it looks firmware-related, set that arg once by hand as root (`qm set --args "-machine sata=off"`) and tell me so I can look for an API-safe alternative. ## 6. oswl-connector: the headscale side `oswl-connector` (goose, vmid 212, 192.168.7.62) runs tailscaled in a container against `https://hs.atlashorizon.net`, advertising `192.168.7.0/24` and itself as an exit node. Three things have to happen outside this repo: 1. **Create the preauth key and put it in Infisical** as `HEADSCALE_AUTHKEY` under `/terraform` (it is read by `secrets.tmpl` as `headscale_authkey`). It must be both ephemeral and reusable -- the VM keeps tailscaled's state in memory, so it re-registers on every boot: The key also carries the tag that drives route auto-approval (see 2), so the VM never has to pass `--advertise-tags` itself: ```bash headscale preauthkeys create --user --ephemeral --reusable \ --tags tag:oswl-connector --expiration 8760h ``` Until that secret exists, `tofu plan` fails while rendering `butane/oswl_connector.yaml.tftpl` (`templatefile`: no value for "headscale_authkey"). 2. **Route approval is automatic, via the key's tag.** The preauth key applies `tag:oswl-connector` as a forced tag at registration, and headscale's `autoApprovers` policy approves the subnet route and the exit node for that tag. This is what makes the node's ephemerality survivable: it re-registers on every boot and after every `tofu taint` rebuild, and each time the routes come back approved without anyone touching headscale. The policy this depends on: ```json { "tagOwners": { "tag:oswl-connector": ["@"] }, "autoApprovers": { "routes": { "192.168.7.0/24": ["tag:oswl-connector"] }, "exitNode": ["tag:oswl-connector"] } } ``` Two things worth knowing if it ever stops approving: - **`autoApprovers` must name the tag, not a user or a group.** Approval by username or group silently does not fire for a node whose identity is a forced tag ([#2891](https://github.com/juanfont/headscale/issues/2891), open as of 0.27.1). - **A tagged node is governed by the tag in ACLs, not by its user.** With headscale's default allow-all policy that changes nothing, but if you later write real ACL rules, `tag:oswl-connector` needs its own grants. To approve by hand (if the policy is ever removed): ```bash headscale nodes list headscale nodes approve-routes -i -r 192.168.7.0/24,0.0.0.0/0,::/0 ``` Do not add `--advertise-tags` to `TS_EXTRA_ARGS` to try to set the tag from the client side: the key already sets it, and `tailscale up` fails outright if the registering user does not own the tag, which takes tailscaled down with it. 3. **Check the LAN's return path.** The VM has one NIC on VLAN 7, so clients reaching 192.168.7.0/24 through the tunnel arrive with a tailnet source address. The container runs with `--network host` and tailscaled installs its own SNAT rules by default (`--snat-subnet-routes`), so this works without touching the router; drop that default only if you add a route for the tailnet CIDR on the 192.168.7.0/24 gateway. That SNAT is only real if the netfilter modules are loaded -- the container has NET_ADMIN but no SYS_MODULE and no `/lib/modules`, so it cannot load them itself. The `modprobe -a` line in `tailscaled.service` loads them on the host, where the container picks them up through `--network host`. If tailscale ever reports "packet filter not installed" or "cannot create the ts-input chain", that line is what to check: ```bash lsmod | grep -E 'iptable|ip6table|nf_nat|nf_tables' docker exec tailscaled iptables -t nat -S | grep ts-postrouting ``` Note the same caveat as the Gitea runner: the auth key ends up in plaintext in the rendered Ignition snippet under `/var/lib/vz/snippets` on goose. ## 7. Commit Nothing is committed. The `opentofu` submodule has the module changes; commit and push it first, then commit the parent repo (which records the new submodule pointer plus `providers.tf`, `vars.tf`, `main.tf`, `terraform.tfvars`, `secrets.tmpl`, `butane/`, and this file).