AtlasHorizon/oswald-infra
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fixed health check, added disabled openwebui

Browse Source
This commit is contained in:
nick committed 2026-09-23 06:45:19 +00:00
1 parent 5667842634
commit 4910bc3fad
3 files changed
+140 -31
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

butane/ai.yaml.tftpl
+89 -1
View File
@@ -13,7 +13,12 @@ systemd:
[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker rm --force gitea-mcp
ExecStart=/usr/bin/docker run --name gitea-mcp \
# The image ships a broken HEALTHCHECK: it is CMD-SHELL form but the
# image is distroless, so every probe fails on a missing /bin/sh and
# the container is reported unhealthy while serving fine. Exec-form
# health tests cannot be set from docker run, so the check is dropped;
# "/app/gitea-mcp -healthcheck" still works via docker exec.
ExecStart=/usr/bin/docker run --name gitea-mcp --no-healthcheck \
-e GITEA_HOST=https://git.atlashorizon.net \
-p 8080:8080 \
docker.gitea.com/gitea-mcp-server \
@@ -45,6 +50,89 @@ systemd:
RestartSec=5s
[Install]
WantedBy=multi-user.target
# ── Open WebUI (disabled) ──────────────────────────────────────────────
# Commented out until it is ready to launch. To re-enable:
# 1. strip the leading "# " from every line of the block below (this
# restores the original indentation exactly)
# 2. un-double the dollar signs in each interpolation: $$ back to $
# 3. re-add the seven OPENWEBUI_* keys to secrets.tmpl (listed below;
# they are removed while this is off so the generated JSON, which
# data.external parses, stays valid)
#
# OPENWEBUI_DATABASE_URL -> openwebui_database_url
# OPENWEBUI_SECRET_KEY -> openwebui_secret_key
# OPENWEBUI_S3_ACCESS_KEY -> openwebui_s3_access_key
# OPENWEBUI_S3_SECRET_KEY -> openwebui_s3_secret_key
# OPENWEBUI_OAUTH_CLIENT_ID -> openwebui_oauth_client_id
# OPENWEBUI_OAUTH_CLIENT_SECRET -> openwebui_oauth_client_secret
# OPENWEBUI_OIDC_PROVIDER_URL -> openwebui_oidc_provider_url
#
# The doubling in step 2 is not cosmetic: templatefile() substitutes on
# raw text before anything parses it as YAML, so an interpolation left
# intact here would still be resolved -- and would still fail the plan on
# a missing key -- even though it sits inside a comment.
# # Open WebUI.
# #
# # Prerequisites, all of which must exist before the VM is rebuilt:
# # - Postgres database, with "CREATE EXTENSION vector;" run inside it
# # - openwebui bucket in Garage, plus a key with r/w on it
# # - an embedding model in LiteLLM matching RAG_EMBEDDING_MODEL below
# # - an OIDC client at the provider, with the callback URI registered as
# # https://llm.atlashorizon.net/oauth/oidc/callback
# # - llm.atlashorizon.net terminating TLS and proxying to this host's
# # :3000 (not managed here -- see WEBUI_URL below)
# # - the OPENWEBUI_* secrets in Infisical (see secrets.tmpl)
# - name: open-webui.service
# enabled: true
# contents: |
# [Unit]
# Description=Open WebUI
# After=docker.service litellm.service
# Requires=docker.service
# [Service]
# TimeoutStartSec=0
# # Large image, so it is pulled up front: a slow first pull is then not
# # mistaken for a hung unit.
# ExecStartPre=/usr/bin/docker pull ghcr.io/open-webui/open-webui:v0.11.4
# ExecStartPre=-/usr/bin/docker rm --force open-webui
# # No volume is mounted. Every piece of durable state lives off-box --
# # Postgres for app data, pgvector for embeddings, Garage for uploads --
# # so /app/backend/data holds nothing worth keeping and the container
# # (and the whole VM) can be rebuilt at will. What stays local is cache
# # only, re-fetched on the next start.
# #
# # gitea-mcp already owns host :8080, so the UI is published on :3000.
# ExecStart=/usr/bin/docker run --name open-webui \
# -e DATABASE_URL=$${openwebui_database_url} \
# -e VECTOR_DB=pgvector \
# -e PGVECTOR_DB_URL=$${openwebui_database_url} \
# -e STORAGE_PROVIDER=s3 \
# -e S3_ENDPOINT_URL=http://s3.lan.oswaldtheotter.net:30188 \
# -e S3_REGION_NAME=garage \
# -e S3_BUCKET_NAME=openwebui \
# -e S3_ACCESS_KEY_ID=$${openwebui_s3_access_key} \
# -e S3_SECRET_ACCESS_KEY=$${openwebui_s3_secret_key} \
# -e WEBUI_SECRET_KEY=$${openwebui_secret_key} \
# -e ENABLE_OLLAMA_API=false \
# -e OPENAI_API_BASE_URL=http://192.168.7.61:4000/v1 \
# -e OPENAI_API_KEY=$${litellm_master_key} \
# -e RAG_EMBEDDING_ENGINE=openai \
# -e RAG_OPENAI_API_BASE_URL=http://192.168.7.61:4000/v1 \
# -e RAG_OPENAI_API_KEY=$${litellm_master_key} \
# -e RAG_EMBEDDING_MODEL=text-embedding-3-small \
# -e OAUTH_CLIENT_ID=$${openwebui_oauth_client_id} \
# -e OAUTH_CLIENT_SECRET=$${openwebui_oauth_client_secret} \
# -e OPENID_PROVIDER_URL=$${openwebui_oidc_provider_url} \
# -e OAUTH_PROVIDER_NAME=SSO \
# -e ENABLE_OAUTH_SIGNUP=true \
# -e WEBUI_URL=https://llm.atlashorizon.net \
# -p 3000:8080 \
# ghcr.io/open-webui/open-webui:v0.11.4
# ExecStop=/usr/bin/docker stop open-webui
# Restart=always
# RestartSec=5s
# [Install]
# WantedBy=multi-user.target
- name: dozzle-agent.service
enabled: true
contents: |
butane/oswl_connector.yaml.tftpl
+11 -2
View File
@@ -25,8 +25,17 @@ systemd:
Wants=network-online.target
[Service]
TimeoutStartSec=0
# tun is not loaded by default on a stock Flatcar boot.
ExecStartPre=-/usr/sbin/modprobe tun
# None of these are loaded by default on a stock Flatcar boot, and the
# container cannot load them itself: it has NET_ADMIN (enough to write
# iptables rules) but not SYS_MODULE, and the image ships no
# /lib/modules, so tailscaled's own modprobe fallback fails with
# "the filter table does not exist". Loading them here works because
# the container shares the host's network namespace (--network host),
# so it sees tables created on the host. Without the nat table in
# particular there is no ts-postrouting chain, which means no SNAT:
# the exit node cannot work at all and subnet-routed replies are sent
# back to tailnet addresses the LAN has no route for.
ExecStartPre=-/usr/sbin/modprobe -a tun iptable_filter iptable_nat ip6table_filter ip6table_nat nf_tables xt_mark
# Recommended by Tailscale for subnet routers and exit nodes: without
# this, UDP throughput through the node is roughly halved.
ExecStartPre=-/usr/bin/bash -c 'ethtool -K $(ip route show 0.0.0.0/0 | cut -d" " -f5) rx-udp-gro-forwarding on rx-gro-list off'
flatcar-todo.md
+40 -28
View File
@@ -112,33 +112,26 @@ as an exit node. Three things have to happen outside this repo:
It must be both ephemeral and reusable -- the VM keeps tailscaled's state in
memory, so it re-registers on every boot:
The key also carries the tag that drives route auto-approval (see 2), so
the VM never has to pass `--advertise-tags` itself:
```bash
headscale preauthkeys create --user <user> --ephemeral --reusable --expiration 8760h
headscale preauthkeys create --user <user> --ephemeral --reusable \
--tags tag:oswl-connector --expiration 8760h
```
Until that secret exists, `tofu plan` fails while rendering
`butane/oswl_connector.yaml.tftpl` (`templatefile`: no value for
"headscale_authkey").
2. **Approve the route and the exit node** once the VM has registered --
headscale honours neither until enabled:
2. **Route approval is automatic, via the key's tag.** The preauth key applies
`tag:oswl-connector` as a forced tag at registration, and headscale's
`autoApprovers` policy approves the subnet route and the exit node for that
tag. This is what makes the node's ephemerality survivable: it re-registers
on every boot and after every `tofu taint` rebuild, and each time the routes
come back approved without anyone touching headscale.
```bash
headscale nodes list
headscale nodes approve-routes -i <id> -r 192.168.7.0/24,0.0.0.0/0,::/0
```
(On headscale older than 0.26 that is `headscale routes list` plus
`headscale routes enable -r <route-id>`, once per route.)
Because the node is ephemeral it re-registers on every boot, so this has to
be redone after each reboot and after each `tofu taint` rebuild. Headscale
can do it automatically, but **the policy has to exist before the node
advertises the tag** -- `tailscale up --advertise-tags` fails outright if
the tag is not owned by the registering user, which takes tailscaled down
with it. So: add the policy first, confirm it loads, and only then add
`--advertise-tags=tag:oswl-connector` to `TS_EXTRA_ARGS` in
`butane/oswl_connector.yaml.tftpl` and rebuild the VM.
The policy this depends on:
```json
{
@@ -154,21 +147,28 @@ as an exit node. Three things have to happen outside this repo:
}
```
`<user>@` is the user the preauth key belongs to (policy v2 spells
usernames with a trailing `@`). Load the policy with `policy.mode: file` +
`policy.path` in headscale's `config.yaml`, or `headscale policy set -f
policy.json` in database mode.
Two things worth knowing if it ever stops approving:
Two further caveats if you go that route:
- **List the tag in `autoApprovers`, not a user or a group.** Approval by
- **`autoApprovers` must name the tag, not a user or a group.** Approval by
username or group silently does not fire for a node whose identity is a
tag ([#2891](https://github.com/juanfont/headscale/issues/2891), open as
of 0.27.1). Naming the tag is both the documented form and the workaround.
forced tag ([#2891](https://github.com/juanfont/headscale/issues/2891),
open as of 0.27.1).
- **A tagged node is governed by the tag in ACLs, not by its user.** With
headscale's default allow-all policy that changes nothing, but if you
later write real ACL rules, `tag:oswl-connector` needs its own grants.
To approve by hand (if the policy is ever removed):
```bash
headscale nodes list
headscale nodes approve-routes -i <id> -r 192.168.7.0/24,0.0.0.0/0,::/0
```
Do not add `--advertise-tags` to `TS_EXTRA_ARGS` to try to set the tag from
the client side: the key already sets it, and `tailscale up` fails outright
if the registering user does not own the tag, which takes tailscaled down
with it.
3. **Check the LAN's return path.** The VM has one NIC on VLAN 7, so clients
reaching 192.168.7.0/24 through the tunnel arrive with a tailnet source
address. The container runs with `--network host` and tailscaled installs
@@ -176,6 +176,18 @@ as an exit node. Three things have to happen outside this repo:
without touching the router; drop that default only if you add a route for
the tailnet CIDR on the 192.168.7.0/24 gateway.
That SNAT is only real if the netfilter modules are loaded -- the container
has NET_ADMIN but no SYS_MODULE and no `/lib/modules`, so it cannot load
them itself. The `modprobe -a` line in `tailscaled.service` loads them on
the host, where the container picks them up through `--network host`. If
tailscale ever reports "packet filter not installed" or
"cannot create the ts-input chain", that line is what to check:
```bash
lsmod | grep -E 'iptable|ip6table|nf_nat|nf_tables'
docker exec tailscaled iptables -t nat -S | grep ts-postrouting
```
Note the same caveat as the Gitea runner: the auth key ends up in plaintext in
the rendered Ignition snippet under `/var/lib/vz/snippets` on goose.