AtlasHorizon/oswald-infra
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Run oswald-runner, ai and oswl-connector as Flatcar VMs

Browse Source 
Replaces the Debian VM that hosted the Gitea Actions runner and adds two
new hosts on goose, all provisioned by Ignition instead of cloud-init:

- oswald-runner: Gitea Actions runner, keeping its registration in
  /var/lib/oswald_runner on disk rather than running ephemeral, which was
  leaving stale offline runners behind whenever a job was interrupted.
- ai: LiteLLM proxy and the Gitea MCP server.
- oswl-connector: tailscaled against headscale, advertising 192.168.7.0/24
  and itself as an exit node.

Their registration tokens and keys come from Infisical through the new
butane_vars plumbing, so nothing secret is committed -- but each one does
land in plaintext in the rendered Ignition snippet on the node, which is
the tradeoff for having no cloud-init here.

The provider gains an ssh block on both nodes because Proxmox has no API
for uploading snippets. That, the snippets content type, and the headscale
preauth key all have to be set up by hand first; flatcar-todo.md has the
steps and the reasoning behind the awkward parts.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
nickandClaude Opus 5 committed 2026-09-22 10:02:23 +00:00
1 parent 5ee0114281
commit 90a2753a2a
11 files changed
+601 -42
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

flatcar-todo.md
+187
View File
@@ -0,0 +1,187 @@
# Flatcar on Proxmox: what is left for you
The OpenTofu side is done: `nodes.<hv>.flatcar` in `terraform.tfvars` now creates
Flatcar VMs (see the Flatcar section in `opentofu/README.md` for how it works).
The items below need access or decisions I do not have. Nothing in this list
touches the existing LXCs/VMs.
## 1. Enable the Snippets content type on `local` (both nodes)
`local` already has `import` enabled (needed for the image), but not `snippets`
(needed for the Ignition file). On each node, as root:
```bash
pvesm set local --content backup,vztmpl,import,iso,snippets
```
Or in the UI: Datacenter → Storage → local → Edit → Content → tick Snippets.
Proxmox does **not** create the snippets directory when you enable the content
type -- you have to do it yourself, or the upload fails with `tee:
/var/lib/vz/snippets/<name>-ignition.json: No such file or directory`:
```bash
mkdir -p /var/lib/vz/snippets # confirm the path via `local`'s entry in /etc/pve/storage.cfg
```
## 2. Let the provider SSH to the nodes as root (both nodes)
Proxmox has no API for uploading snippets, so bpg copies them over SSH. The
provider is configured (`providers.tf`, `ssh` block) to log in as `root` with
the ansible key from Infisical (`SSH_PRIVATE_KEY` / `SSH_PUBLIC_KEY`). On each
node append the public key to root's authorized_keys:
```bash
# from this machine; the key file is materialized by tofu at ~/.ssh/infra_ansible_key
ssh-copy-id -i ~/.ssh/infra_ansible_key.pub [email protected] # pve
ssh-copy-id -i ~/.ssh/infra_ansible_key.pub [email protected] # goose
```
(`~/.ssh/infra_ansible_key.pub` may not exist yet; paste `SSH_PUBLIC_KEY` from
Infisical into `/root/.ssh/authorized_keys` on the node instead.)
Then verify from here:
```bash
ssh -i ~/.ssh/infra_ansible_key [email protected] 'pvesm status'
```
If you would rather not give the provider root, bpg supports a non-root user
with a narrow sudoers file (`pvesm`, `qm`, `tee /var/lib/vz/snippets/*`). Change
`username` in the `ssh` blocks accordingly.
## 3. Declare a VM
Uncomment and adapt the `flatcar = { ... }` example under `nodes.goose` in
`terraform.tfvars`. Pick an unused vmid and IP. Optional per-VM config goes in
`butane/<name>.yaml`, referenced with `butane_file = "butane/<name>.yaml"` on
the VM (`butane/example.yaml` is a starting point; keep `variant: flatcar` and
`version: 1.0.0`). Inline YAML via `butane = <<-EOT ... EOT` also works.
I dry-ran a plan with the example VM declared (nothing applied): it creates
the image download, the Ignition snippet, the VM and its DNS record, and
changes nothing else.
Then:
```bash
infisical run --path /terraform -- tofu plan -out plan
infisical run --path /terraform -- tofu apply plan
```
The first apply on a node downloads the 500 MB Flatcar image into `local`
(`flatcar-stable-4593.2.5.img`), then creates the VM with that image imported
as its boot disk. Expect the download to take a minute or two.
Log in with `ssh ansible@<ip>` or `ssh core@<ip>` using the ansible key.
## 4. Things to know
- Ignition runs on first boot only. Editing the Butane file updates the snippet on the
node but does nothing to a running VM. To apply, rebuild it:
`tofu taint 'module.goose.proxmox_virtual_environment_vm.flatcar["<name>"]'`
(this destroys the VM's disk; keep data in volumes you can recreate).
- Ignition snippets are plain text on the node. Do not put secrets in `butane`.
- Flatcar has no Python, so the existing Ansible plays will not run against it.
These hosts land in a `flatcar` group in the generated inventory; only
`raw`/`script` tasks work there. Ignition is the intended replacement.
- No QEMU guest agent: the module sets `agent.enabled = false` for Flatcar VMs,
so plans never wait on it. Proxmox will not show the VM's IP in the UI.
- Bumping `flatcar_image.version` (in `vars.tf` default or tfvars) plus its
sha512 from the release `.DIGESTS` file downloads a new image for future VMs
only; existing VMs are not touched (`import_from` is ignored after creation).
Flatcar itself auto-updates in place via its update engine.
## 5. If the VM does not boot
The Flatcar docs pass `--args "-machine sata=off"` when creating the VM. I left
it out: that option is root-only in Proxmox and the API token cannot set it.
If a Flatcar VM hangs at boot, check the serial console (`qm terminal <vmid>`)
first; if it looks firmware-related, set that arg once by hand as root
(`qm set <vmid> --args "-machine sata=off"`) and tell me so I can look for an
API-safe alternative.
## 6. oswl-connector: the headscale side
`oswl-connector` (goose, vmid 212, 192.168.7.62) runs tailscaled in a container
against `https://hs.atlashorizon.net`, advertising `192.168.7.0/24` and itself
as an exit node. Three things have to happen outside this repo:
1. **Create the preauth key and put it in Infisical** as `HEADSCALE_AUTHKEY`
under `/terraform` (it is read by `secrets.tmpl` as `headscale_authkey`).
It must be both ephemeral and reusable -- the VM keeps tailscaled's state in
memory, so it re-registers on every boot:
```bash
headscale preauthkeys create --user <user> --ephemeral --reusable --expiration 8760h
```
Until that secret exists, `tofu plan` fails while rendering
`butane/oswl_connector.yaml.tftpl` (`templatefile`: no value for
"headscale_authkey").
2. **Approve the route and the exit node** once the VM has registered --
headscale honours neither until enabled:
```bash
headscale nodes list
headscale nodes approve-routes -i <id> -r 192.168.7.0/24,0.0.0.0/0,::/0
```
(On headscale older than 0.26 that is `headscale routes list` plus
`headscale routes enable -r <route-id>`, once per route.)
Because the node is ephemeral it re-registers on every boot, so this has to
be redone after each reboot and after each `tofu taint` rebuild. Headscale
can do it automatically, but **the policy has to exist before the node
advertises the tag** -- `tailscale up --advertise-tags` fails outright if
the tag is not owned by the registering user, which takes tailscaled down
with it. So: add the policy first, confirm it loads, and only then add
`--advertise-tags=tag:oswl-connector` to `TS_EXTRA_ARGS` in
`butane/oswl_connector.yaml.tftpl` and rebuild the VM.
```json
{
"tagOwners": {
"tag:oswl-connector": ["<user>@"]
},
"autoApprovers": {
"routes": {
"192.168.7.0/24": ["tag:oswl-connector"]
},
"exitNode": ["tag:oswl-connector"]
}
}
```
`<user>@` is the user the preauth key belongs to (policy v2 spells
usernames with a trailing `@`). Load the policy with `policy.mode: file` +
`policy.path` in headscale's `config.yaml`, or `headscale policy set -f
policy.json` in database mode.
Two further caveats if you go that route:
- **List the tag in `autoApprovers`, not a user or a group.** Approval by
username or group silently does not fire for a node whose identity is a
tag ([#2891](https://github.com/juanfont/headscale/issues/2891), open as
of 0.27.1). Naming the tag is both the documented form and the workaround.
- **A tagged node is governed by the tag in ACLs, not by its user.** With
headscale's default allow-all policy that changes nothing, but if you
later write real ACL rules, `tag:oswl-connector` needs its own grants.
3. **Check the LAN's return path.** The VM has one NIC on VLAN 7, so clients
reaching 192.168.7.0/24 through the tunnel arrive with a tailnet source
address. The container runs with `--network host` and tailscaled installs
its own SNAT rules by default (`--snat-subnet-routes`), so this works
without touching the router; drop that default only if you add a route for
the tailnet CIDR on the 192.168.7.0/24 gateway.
Note the same caveat as the Gitea runner: the auth key ends up in plaintext in
the rendered Ignition snippet under `/var/lib/vz/snippets` on goose.
## 7. Commit
Nothing is committed. The `opentofu` submodule has the module changes; commit
and push it first, then commit the parent repo (which records the new
submodule pointer plus `providers.tf`, `vars.tf`, `main.tf`,
`terraform.tfvars`, `secrets.tmpl`, `butane/`, and this file).