Run oswald-runner, ai and oswl-connector as Flatcar VMs

Replaces the Debian VM that hosted the Gitea Actions runner and adds two
new hosts on goose, all provisioned by Ignition instead of cloud-init:

- oswald-runner: Gitea Actions runner, keeping its registration in
  /var/lib/oswald_runner on disk rather than running ephemeral, which was
  leaving stale offline runners behind whenever a job was interrupted.
- ai: LiteLLM proxy and the Gitea MCP server.
- oswl-connector: tailscaled against headscale, advertising 192.168.7.0/24
  and itself as an exit node.

Their registration tokens and keys come from Infisical through the new
butane_vars plumbing, so nothing secret is committed -- but each one does
land in plaintext in the rendered Ignition snippet on the node, which is
the tradeoff for having no cloud-init here.

The provider gains an ssh block on both nodes because Proxmox has no API
for uploading snippets. That, the snippets content type, and the headscale
preauth key all have to be set up by hand first; flatcar-todo.md has the
steps and the reasoning behind the awkward parts.

Co-Authored-By: Claude Opus 5 <[email protected]>

butane/ai.yaml.tftpl

@@ -0,0 +1,47 @@
+variant: flatcar
+version: 1.0.0
+
+systemd:
+  units:
+    - name: gitea-mcp.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Gitea MCP Server
+        After=docker.service
+        Requires=docker.service
+        [Service]
+        TimeoutStartSec=0
+        ExecStartPre=-/usr/bin/docker rm --force gitea-mcp
+        ExecStart=/usr/bin/docker run --name gitea-mcp \
+          -e GITEA_HOST=https://git.atlashorizon.net \
+          -p 8080:8080 \
+          docker.gitea.com/gitea-mcp-server \
+          /app/gitea-mcp -t http --port 8080
+        ExecStop=/usr/bin/docker stop gitea-mcp
+        Restart=always
+        RestartSec=5s
+        [Install]
+        WantedBy=multi-user.target
+    - name: litellm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=LiteLLM proxy
+        After=docker.service
+        Requires=docker.service
+        [Service]
+        TimeoutStartSec=0
+        ExecStartPre=-/usr/bin/docker rm --force litellm
+        ExecStart=/usr/bin/docker run --name litellm \
+          -e DATABASE_URL=${litellm_database_url} \
+          -e STORE_MODEL_IN_DB=True \
+          -e LITELLM_MASTER_KEY=${litellm_master_key} \
+          -e LITELLM_SALT_KEY=${litellm_salt_key} \
+          -p 4000:4000 \
+          docker.litellm.ai/berriai/litellm:main-stable
+        ExecStop=/usr/bin/docker stop litellm
+        Restart=always
+        RestartSec=5s
+        [Install]
+        WantedBy=multi-user.target

butane/example.yaml

@@ -0,0 +1,36 @@
+# Example extra Butane config for a Flatcar VM. Reference it from tfvars with
+# `butane_file = "butane/<name>.yaml"`. It is merged into the base config the
+# module generates (hostname, core + ansible users with the shared SSH key).
+# Applied by Ignition on FIRST BOOT only: to change a running VM, edit this
+# file and taint the VM so it is rebuilt from the image.
+variant: flatcar
+version: 1.0.0
+
+systemd:
+  units:
+    # Example: run a container as a service.
+    - name: hello.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Hello container
+        After=docker.service
+        Requires=docker.service
+        [Service]
+        TimeoutStartSec=0
+        ExecStartPre=-/usr/bin/docker rm --force hello
+        ExecStart=/usr/bin/docker run --name hello --rm -p 8080:80 nginx:alpine
+        ExecStop=/usr/bin/docker stop hello
+        Restart=always
+        RestartSec=5s
+        [Install]
+        WantedBy=multi-user.target
+
+storage:
+  files:
+    # Example: a file on disk.
+    - path: /etc/motd
+      mode: 0644
+      contents:
+        inline: |
+          Managed by OpenTofu + Ignition. Config changes require a rebuild.

butane/oswald_runner.yaml.tftpl

@@ -0,0 +1,49 @@
+variant: flatcar
+version: 1.0.0
+
+systemd:
+  units:
+    # Gitea Actions runner. Registers once on first start and keeps its
+    # identity (the .runner file) in /var/lib/oswald_runner on the VM's own
+    # root disk -- not ephemeral, so it does not self-delete after each job
+    # (that caused stale offline entries whenever the container was
+    # interrupted before finishing a job; see flatcar-todo.md). Reusing the
+    # same registration also means the token never needs rotating.
+    #
+    # /var/lib/oswald_runner survives container restarts and VM reboots (it's
+    # on disk, not in the container), but not a VM rebuild (taint): that
+    # recreates the disk from the image, so the runner re-registers fresh
+    # next boot. Old entries from before this change, or from any future
+    # rebuild, still need manual cleanup in Gitea's runner admin page.
+    #
+    # GITEA_RUNNER_REGISTRATION_TOKEN is templated in from Infisical
+    # (GITEA_RUNNER_TOKEN in secrets.tmpl) by main.tf, not stored here. It
+    # still ends up in plaintext in the rendered Ignition snippet on the
+    # node's storage -- see the caveat in flatcar-todo.md.
+    - name: oswald_runner.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Gitea Actions runner
+        After=docker.service
+        Requires=docker.service
+        [Service]
+        TimeoutStartSec=0
+        ExecStartPre=-/usr/bin/docker rm --force oswald_runner
+        ExecStart=/usr/bin/docker run --name oswald_runner \
+          -e GITEA_INSTANCE_URL=https://git.atlashorizon.net \
+          -e GITEA_RUNNER_REGISTRATION_TOKEN=${gitea_runner_token} \
+          -e GITEA_RUNNER_NAME=oswald_runner \
+          -v /var/lib/oswald_runner:/data \
+          -v /var/run/docker.sock:/var/run/docker.sock \
+          docker.io/gitea/runner:3
+        ExecStop=/usr/bin/docker stop oswald_runner
+        Restart=always
+        RestartSec=5s
+        [Install]
+        WantedBy=multi-user.target
+
+storage:
+  directories:
+    - path: /var/lib/oswald_runner
+      mode: 0755

butane/oswl_connector.yaml.tftpl

@@ -0,0 +1,60 @@
+variant: flatcar
+version: 1.0.0
+
+# Headscale (Tailscale) node: advertises 192.168.7.0/24 as a subnet route and
+# offers itself as an exit node.
+#
+# Registered as an EPHEMERAL node: tailscaled runs with in-memory state, so the
+# VM re-authenticates on every boot and headscale drops the node shortly after
+# it goes offline. That requires the preauth key to be both --ephemeral and
+# --reusable, otherwise the second boot fails to register.
+#
+# HEADSCALE_AUTHKEY is templated in from Infisical (see secrets.tmpl) by
+# main.tf, not stored here. It still ends up in plaintext in the rendered
+# Ignition snippet on the node's storage -- see the caveat in flatcar-todo.md.
+
+systemd:
+  units:
+    - name: tailscaled.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Tailscale node (headscale subnet router + exit node)
+        After=docker.service network-online.target
+        Requires=docker.service
+        Wants=network-online.target
+        [Service]
+        TimeoutStartSec=0
+        # tun is not loaded by default on a stock Flatcar boot.
+        ExecStartPre=-/usr/sbin/modprobe tun
+        # Recommended by Tailscale for subnet routers and exit nodes: without
+        # this, UDP throughput through the node is roughly halved.
+        ExecStartPre=-/usr/bin/bash -c 'ethtool -K $(ip route show 0.0.0.0/0 | cut -d" " -f5) rx-udp-gro-forwarding on rx-gro-list off'
+        ExecStartPre=-/usr/bin/docker rm --force tailscaled
+        ExecStart=/usr/bin/docker run --name tailscaled \
+          --network host --cap-add NET_ADMIN --cap-add NET_RAW \
+          --device /dev/net/tun \
+          -e TS_AUTHKEY=${headscale_authkey} \
+          -e TS_HOSTNAME=oswl-connector \
+          -e TS_ROUTES=192.168.7.0/24 \
+          -e TS_ACCEPT_DNS=false \
+          -e "TS_EXTRA_ARGS=--login-server=https://hs.atlashorizon.net --advertise-exit-node" \
+          -e TS_USERSPACE=false \
+          docker.io/tailscale/tailscale:stable
+        ExecStop=/usr/bin/docker stop tailscaled
+        Restart=always
+        RestartSec=5s
+        [Install]
+        WantedBy=multi-user.target
+
+storage:
+  files:
+    # Forwarding must be on for the subnet route and the exit node to pass
+    # traffic at all; the container shares the host netns (--network host), so
+    # this has to be set on the host, not inside it.
+    - path: /etc/sysctl.d/99-tailscale.conf
+      mode: 0644
+      contents:
+        inline: |
+          net.ipv4.ip_forward = 1
+          net.ipv6.conf.all.forwarding = 1

butane/webserver.yaml

@@ -0,0 +1,75 @@
+# Butane config for a Flatcar VM running the official httpd (Apache) image
+# as a docker container. Applied by Ignition on FIRST BOOT only: to change a
+# running VM, edit this file and taint the VM so it is rebuilt from the image.
+variant: flatcar
+version: 1.0.0
+
+systemd:
+  units:
+    - name: webserver.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=httpd webserver container
+        After=docker.service
+        Requires=docker.service
+        [Service]
+        TimeoutStartSec=0
+        ExecStartPre=-/usr/bin/docker rm --force webserver
+        ExecStart=/usr/bin/docker run --name webserver --rm -p 80:80 httpd:latest
+        ExecStop=/usr/bin/docker stop webserver
+        Restart=always
+        RestartSec=5s
+        [Install]
+        WantedBy=multi-user.target
+    # Experimental: Flatcar ships no qemu-guest-agent binary, so this runs one
+    # from an Alpine container against the virtio-serial channel Proxmox
+    # exposes when the VM's `agent.enabled` is true (see qemu_guest_agent in
+    # tfvars). --network host is required so qemu-ga reports the host's real
+    # network interfaces/IP instead of the container's own bridge IP.
+    # --pid host + the /sbin/shutdown bind mount (see /opt/qga-shutdown.sh
+    # below) are required so guest-shutdown/reboot actually reach the host
+    # instead of Alpine's own busybox init inside the container.
+    # --privileged + host /dev is broader access than qemu-ga strictly needs,
+    # but keeps this simple for a first try.
+    - name: qemu-guest-agent.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=QEMU guest agent (containerized)
+        After=docker.service
+        Requires=docker.service
+        [Service]
+        TimeoutStartSec=0
+        ExecStartPre=-/usr/bin/docker rm --force qemu-guest-agent
+        ExecStart=/usr/bin/docker run --name qemu-guest-agent --rm --privileged \
+          --network host --pid host -v /dev:/dev \
+          -v /opt/qga-shutdown.sh:/sbin/shutdown:ro \
+          alpine:latest sh -c \
+          "apk add --no-cache qemu-guest-agent util-linux && exec qemu-ga -m virtio-serial -p /dev/virtio-ports/org.qemu.guest_agent.0"
+        Restart=always
+        RestartSec=5s
+        [Install]
+        WantedBy=multi-user.target
+
+storage:
+  files:
+    # qemu-ga always execs "/sbin/shutdown <flag> +0 <reason>" (flag is -H
+    # halt, -P powerdown, -r reboot) to handle guest-shutdown/guest-reboot.
+    # Inside the container that path is Alpine's own busybox shutdown, which
+    # only affects the container's namespace. Bind-mounted over /sbin/shutdown
+    # in the container (see qemu-guest-agent.service above), this shim uses
+    # nsenter to re-enter the real host's namespaces (PID 1 in the container
+    # is the host's actual systemd, since the container runs with --pid host)
+    # and calls the host's real systemctl instead.
+    - path: /opt/qga-shutdown.sh
+      mode: 0755
+      contents:
+        inline: |
+          #!/bin/sh
+          case "$1" in
+            -H) action=halt ;;
+            -r) action=reboot ;;
+            *)  action=poweroff ;;
+          esac
+          exec nsenter --target 1 --mount --uts --ipc --net -- systemctl "$action"