# Flatcar on Proxmox: what is left for you The OpenTofu side is done: `nodes..flatcar` in `terraform.tfvars` now creates Flatcar VMs (see the Flatcar section in `opentofu/README.md` for how it works). The items below need access or decisions I do not have. Nothing in this list touches the existing LXCs/VMs. ## 1. Enable the Snippets content type on `local` (both nodes) `local` already has `import` enabled (needed for the image), but not `snippets` (needed for the Ignition file). On each node, as root: ```bash pvesm set local --content backup,vztmpl,import,iso,snippets ``` Or in the UI: Datacenter → Storage → local → Edit → Content → tick Snippets. Proxmox does **not** create the snippets directory when you enable the content type -- you have to do it yourself, or the upload fails with `tee: /var/lib/vz/snippets/-ignition.json: No such file or directory`: ```bash mkdir -p /var/lib/vz/snippets # confirm the path via `local`'s entry in /etc/pve/storage.cfg ``` ## 2. Let the provider SSH to the nodes as root (both nodes) Proxmox has no API for uploading snippets, so bpg copies them over SSH. The provider is configured (`providers.tf`, `ssh` block) to log in as `root` with the ansible key from Infisical (`SSH_PRIVATE_KEY` / `SSH_PUBLIC_KEY`). On each node append the public key to root's authorized_keys: ```bash # from this machine; the key file is materialized by tofu at ~/.ssh/infra_ansible_key ssh-copy-id -i ~/.ssh/infra_ansible_key.pub root@192.168.7.240 # pve ssh-copy-id -i ~/.ssh/infra_ansible_key.pub root@192.168.7.241 # goose ``` (`~/.ssh/infra_ansible_key.pub` may not exist yet; paste `SSH_PUBLIC_KEY` from Infisical into `/root/.ssh/authorized_keys` on the node instead.) Then verify from here: ```bash ssh -i ~/.ssh/infra_ansible_key root@192.168.7.241 'pvesm status' ``` If you would rather not give the provider root, bpg supports a non-root user with a narrow sudoers file (`pvesm`, `qm`, `tee /var/lib/vz/snippets/*`). Change `username` in the `ssh` blocks accordingly. ## 3. Declare a VM Uncomment and adapt the `flatcar = { ... }` example under `nodes.goose` in `terraform.tfvars`. Pick an unused vmid and IP. Optional per-VM config goes in `butane/.yaml`, referenced with `butane_file = "butane/.yaml"` on the VM (`butane/example.yaml` is a starting point; keep `variant: flatcar` and `version: 1.0.0`). Inline YAML via `butane = <<-EOT ... EOT` also works. I dry-ran a plan with the example VM declared (nothing applied): it creates the image download, the Ignition snippet, the VM and its DNS record, and changes nothing else. Then: ```bash infisical run --path /terraform -- tofu plan -out plan infisical run --path /terraform -- tofu apply plan ``` The first apply on a node downloads the 500 MB Flatcar image into `local` (`flatcar-stable-4593.2.5.img`), then creates the VM with that image imported as its boot disk. Expect the download to take a minute or two. Log in with `ssh ansible@` or `ssh core@` using the ansible key. ## 4. Things to know - Ignition runs on first boot only. Editing the Butane file updates the snippet on the node but does nothing to a running VM. To apply, rebuild it: `tofu taint 'module.goose.proxmox_virtual_environment_vm.flatcar[""]'` (this destroys the VM's disk; keep data in volumes you can recreate). - Ignition snippets are plain text on the node. Do not put secrets in `butane`. - Flatcar has no Python, so the existing Ansible plays will not run against it. These hosts land in a `flatcar` group in the generated inventory; only `raw`/`script` tasks work there. Ignition is the intended replacement. - No QEMU guest agent: the module sets `agent.enabled = false` for Flatcar VMs, so plans never wait on it. Proxmox will not show the VM's IP in the UI. - Bumping `flatcar_image.version` (in `vars.tf` default or tfvars) plus its sha512 from the release `.DIGESTS` file downloads a new image for future VMs only; existing VMs are not touched (`import_from` is ignored after creation). Flatcar itself auto-updates in place via its update engine. ## 5. If the VM does not boot The Flatcar docs pass `--args "-machine sata=off"` when creating the VM. I left it out: that option is root-only in Proxmox and the API token cannot set it. If a Flatcar VM hangs at boot, check the serial console (`qm terminal `) first; if it looks firmware-related, set that arg once by hand as root (`qm set --args "-machine sata=off"`) and tell me so I can look for an API-safe alternative. ## 6. oswl-connector: the headscale side `oswl-connector` (goose, vmid 212, 192.168.7.62) runs tailscaled in a container against `https://hs.atlashorizon.net`, advertising `192.168.7.0/24` and itself as an exit node. Three things have to happen outside this repo: 1. **Create the preauth key and put it in Infisical** as `HEADSCALE_AUTHKEY` under `/terraform` (it is read by `secrets.tmpl` as `headscale_authkey`). It must be both ephemeral and reusable -- the VM keeps tailscaled's state in memory, so it re-registers on every boot: ```bash headscale preauthkeys create --user --ephemeral --reusable --expiration 8760h ``` Until that secret exists, `tofu plan` fails while rendering `butane/oswl_connector.yaml.tftpl` (`templatefile`: no value for "headscale_authkey"). 2. **Approve the route and the exit node** once the VM has registered -- headscale honours neither until enabled: ```bash headscale nodes list headscale nodes approve-routes -i -r 192.168.7.0/24,0.0.0.0/0,::/0 ``` (On headscale older than 0.26 that is `headscale routes list` plus `headscale routes enable -r `, once per route.) Because the node is ephemeral it re-registers on every boot, so this has to be redone after each reboot and after each `tofu taint` rebuild. Headscale can do it automatically, but **the policy has to exist before the node advertises the tag** -- `tailscale up --advertise-tags` fails outright if the tag is not owned by the registering user, which takes tailscaled down with it. So: add the policy first, confirm it loads, and only then add `--advertise-tags=tag:oswl-connector` to `TS_EXTRA_ARGS` in `butane/oswl_connector.yaml.tftpl` and rebuild the VM. ```json { "tagOwners": { "tag:oswl-connector": ["@"] }, "autoApprovers": { "routes": { "192.168.7.0/24": ["tag:oswl-connector"] }, "exitNode": ["tag:oswl-connector"] } } ``` `@` is the user the preauth key belongs to (policy v2 spells usernames with a trailing `@`). Load the policy with `policy.mode: file` + `policy.path` in headscale's `config.yaml`, or `headscale policy set -f policy.json` in database mode. Two further caveats if you go that route: - **List the tag in `autoApprovers`, not a user or a group.** Approval by username or group silently does not fire for a node whose identity is a tag ([#2891](https://github.com/juanfont/headscale/issues/2891), open as of 0.27.1). Naming the tag is both the documented form and the workaround. - **A tagged node is governed by the tag in ACLs, not by its user.** With headscale's default allow-all policy that changes nothing, but if you later write real ACL rules, `tag:oswl-connector` needs its own grants. 3. **Check the LAN's return path.** The VM has one NIC on VLAN 7, so clients reaching 192.168.7.0/24 through the tunnel arrive with a tailnet source address. The container runs with `--network host` and tailscaled installs its own SNAT rules by default (`--snat-subnet-routes`), so this works without touching the router; drop that default only if you add a route for the tailnet CIDR on the 192.168.7.0/24 gateway. Note the same caveat as the Gitea runner: the auth key ends up in plaintext in the rendered Ignition snippet under `/var/lib/vz/snippets` on goose. ## 7. Commit Nothing is committed. The `opentofu` submodule has the module changes; commit and push it first, then commit the parent repo (which records the new submodule pointer plus `providers.tf`, `vars.tf`, `main.tf`, `terraform.tfvars`, `secrets.tmpl`, `butane/`, and this file).