infra

Self-hosted git + CI/CD + AI stack on llm2 (192.168.1.120). Public at xor0.de, internal services at *.local.xor0.de and *.local.geist.md.
Compose stack lives at ~/infra/ on llm2. Credentials in SECRETS.md (git-ignored).

---

Architecture

Ingress

┌──────────────────────┐                   ┌─────────────────────────────┐
│                      │                   │                             │
│       Internet       │                   │             LAN             │
│                      │                   │                             │
└───────────┬──────────┘                   └──────────────┬──────────────┘
       *.xor0.de                                *.local.xor0.de
            │                       (via Cloudflare DNS)  │
            ▼                                             │
┌──────────────────────┐                                  │
│                      │                                  │
│    Cloudflare DNS    │                                  │
│  proxy OFF · DDNS    │                                  │
└───────────┬──────────┘                                  │
            │                                             │
            ▼                                             │
┌──────────────────────┐                                  │
│                      │                                  │
│ pfSense NAT :80/:443 │                                  │
│                      │                                  │
└───────────┬──────────┘                                  │
            │                                             │
            └───────────────────────────┐                 │
                                        ▼                 ▼
                                ┌───────────────────────────────┐
                                │       Local switch (L2)       │
                                └───────────────┬───────────────┘
                                                │
                                                ▼
                                ┌───────────────────────────────┐
                                │  Caddy — TLS + reverse proxy  │
                                │  host: llm2 · rose · .120     │
                                │  wildcard LE certs (DNS-01)   │
                                │  CrowdSec bouncer             │
                                └───────────────┬───────────────┘
                                                │  routes by Host header
       ┌───────────┬───────────┬───────────┬────┴──────┬───────────┬───────────┬───────────┐
       ▼           ▼           ▼           ▼           ▼           ▼           ▼           ▼
 ┌───────────┐┌───────────┐┌───────────┐┌───────────┐┌──────────┐┌──────────┐┌───────────┐┌───────────┐
 │  Forgejo  ││   Pages   ││Google MCP ││  SearXNG  ││  Mastra  ││  Mastra  ││ mirror-bot││  opencode │
 │git.xor0.de││pages.xor0 ││*.mcp.xor0 ││search.loc ││   API    ││  Studio  ││ → GitHub  ││opencode.lo│
 └───────────┘└───────────┘└───────────┘└───────────┘└──────────┘└──────────┘└───────────┘└───────────┘

     ─── inference hosts · OpenAI-compatible /v1 · reverse-proxied by Caddy ───────────────

 ┌───────────────────────────────────────┐     ┌───────────────────────────────────────┐
 │ llm2 · rose · 192.168.1.120   (local)  │     │ llm1 · Mac Studio · 192.168.1.150      │
 │ llama-server · Vulkan · systemd        │     │ OptiQ · MLX · remote LAN host          │
 │ https://llm.local + llm2.local         │     │ https://llm1.local  (Caddy → LAN)      │
 │ Gemma4-26B QAT q4_0 · KV q5_0          │     │ Qwen3.6-27B-OptiQ-4bit · int4 KV       │
 │ 4 slots: 1 planner + 3 coders          │     │ 1 slot · plain AR · 502 when asleep    │
 └───────────────────────────────────────┘     └───────────────────────────────────────┘

AI Stack

┌──────────────────┐                    ┌────────────────────────────────┐
│                  │                    │ llama-server · single · llm2   │
│  Mastra Studio   │                    │ rose · RX 7900 XT · Vulkan     │
│  (chat UI)       ├───CORS via Caddy──►│ ─ the one served model ─       │
│                  │        │           │ Gemma4-26B QAT q4_0 · KV q5_0  │
└──────────────────┘        │           │ 4-slot kv-unified · ctx 262k   │
                            │           │ MTP draft (Q8_0) · b10210      │
                            ▼           └────────────────▲───────────────┘
                   ┌──────────────────┐                 │
                   │   Mastra API     │                 │ inference
                   │   (agents)       ├─────────────────┘ (id_slot pinned)
                   │                  │
                   │                  ├──── web search ──►┌──────────────┐
                   └──────────────────┘                   │   SearXNG    │
                            │                             │  (JSON API)  │
                   ┌────────┴─────────┐                   └──────────────┘
                   │   Agent Slots    │
                   │  0: Chat (pinned)│
                   │  1: Orch (pinned)│
                   │  2-3: Workers    │
                   └──────────────────┘

  Second inference host — NOT in the Mastra path (llm-bench harness + direct /v1 clients):
                                       ┌────────────────────────────────┐
                                       │ OptiQ · MLX · llm1             │
                                       │ Mac Studio · Apple Silicon     │
                                       │ Qwen3.6-27B-OptiQ-4bit         │
                                       │ 1 slot · int4 KV · plain AR    │
                                       │ https://llm1.local (Caddy→LAN) │
                                       └────────────────────────────────┘

Benchmark host rose = llm2. It serves one model in single-model mode — /v1/models lists exactly that one entry, and switching models means editing llama-server.conf + a service restart. The second inference host, llm1 (Apple-Silicon Mac Studio, 192.168.1.150), serves Qwen3.6-27B-OptiQ-4bit under MLX/OptiQ — it is not part of the Mastra serving path above, but is now reverse-proxied by Caddy at https://llm1.local.xor0.de (in addition to loopback for the llm-bench harness). See Models & inference.

CI/CD + Mirroring

┌─────────────┐         ┌────────────────┐                 ┌─────────────────┐
│             │         │                │                 │                 │
│   Forgejo   ├─webhook►│   mirror-bot   │        ┌────────┤ sync_mirrors.py │
│             │         │                │        │        │                 │
└──────┬──────┘         └────────┬───────┘        │        └─────────────────┘
       │                         │                │                 ▲
       │                     dispatch             │                 │
  cron + push                    ▼                │                 │
       │                ┌────────────────┐ create + mirror          │
       │                │                │        │                 │
       ├───────────────►│ Actions Runner ├────────┼─runs────────────┘
   push│sync            │                │        │
       │                └────────────────┘        │
       │                                          │
       │                         ┌────────────────┘
       │                         ▼
       │                ┌────────────────┐
       │                │                │
       └───────────────►│     GitHub     │
                        │                │
                        └────────────────┘

---

Services

Container	Image	Purpose
forgejo	forgejo/forgejo:15	Git server, Actions, container registry. SQLite, HTTPS+PAT only (no SSH).
caddy	custom (see Dockerfile)	TLS termination, reverse proxy, CrowdSec IP banning. Wildcard cert via Cloudflare DNS-01.
crowdsec	crowdsecurity/crowdsec	Parses Caddy access logs; makes ban decisions consumed by the Caddy bouncer.
mirror-bot	python:3.12-alpine	Listens on http://mirror-bot:8080/webhook for Forgejo system webhook events. On repository.created: dispatches the sync-github-mirrors workflow.
forgejo-runner	forgejo/runner:11	Actions runner. Mounts host Docker socket. Labels: docker/ubuntu-latest → node:20-bookworm, deploy → docker:27-cli.
(Windows runner)	gitea-runner (native, off-host)	Native Windows host-mode runner on a dev box (not in this compose). Labels windows/windows-latest → :host. See windows-runner/.
(Google MCP proxy)	(via Caddy)	Reverse-proxies Google's hosted MCP servers (gmailmcp, calendarmcp, drivemcp) at *.mcp.xor0.de. LAN + Anthropic outbound range only.
llama-server	source build (machines/llm2/build-llamacpp.sh)	LLM inference server. Not a container — llama-server systemd service, Vulkan build of llama.cpp (currently b10210). Single-model mode: serves Gemma4-26B QAT q4_0 · KV q5_0 only (4-slot --kv-unified fleet, ctx 262144, MTP draft), so /v1/models lists exactly one entry. All params in machines/llm2/llama-server.conf. OpenAI-compatible API at llm.local.xor0.de. Deploy via machines/llm2/deploy.sh.
searxng	searxng/searxng:latest	Self-hosted meta search engine. JSON API for agents, HTML UI for humans. No API keys needed.
mastra	custom (see mastra/Dockerfile)	TypeScript agent framework (v1.x). Multi-agent orchestration with slot-pinned LLM access and SearXNG web search.
mastra-studio	custom (see mastra-studio/Dockerfile)	Mastra Studio dev playground / chat UI. Connects to Mastra API via Caddy (CORS).
opencode	custom (see opencode/Dockerfile)	Self-hosted opencode coding agent, headless (opencode serve). Both inference hosts registered as providers in opencode/opencode.json. Driven from the opencode Desktop app at opencode.local.xor0.de. See opencode.
central-db	postgres:17-alpine	Central house PostgreSQL. One shared server; per-app databases + owner roles provisioned by postgres/initdb.d/. Currently hosts llmbench (llm-bench tidy rows, read via DuckDB's postgres extension). LAN-only on :5432. Separate compose file: postgres/docker-compose.yml.
sonarqube	sonarqube:26.6.0.123539-community	Static code analysis (Community Build). Bundled Elasticsearch. Separate compose file: sonarqube/docker-compose.yml. Needs host vm.max_map_count >= 524288.
sonarqube-db	postgres:17-alpine	PostgreSQL backing store for SonarQube (its own private instance, separate from central-db). Holds all analysis history (include in backups). Part of sonarqube/docker-compose.yml.

---

Models & inference

Two inference hosts (see machines/ for per-host control scripts):

Host	Server	Model running	Serving config	Reachable at
llm2 — rose, 192.168.1.120, RX 7900 XT (20 GiB), Vulkan	llama-server single-model (systemd)	Gemma4-26B-A4B QAT q4_0 · KV q5_0 — the only served model (declared benchmark winner)	4-slot --kv-unified pool · ctx 262144 · KV q5_0 · MTP draft (gemma-4-26B-A4B-it-MTP-Q8_0, n-max 4) · ubatch 1024 · flash-attn	https://llm.local.xor0.de/v1 and https://llm2.local.xor0.de/v1 (OpenAI-compatible, LAN)
llm1 — Tobiass-Mac-Studio, 192.168.1.150, Apple Silicon, MLX	OptiQ (optiq serve)	Qwen3.6-27B-OptiQ-4bit — mixed-precision 4-bit weights, int4 KV	single slot (--max-concurrent 1) · --max-context auto · --kv-bits 4 · plain AR (no MTP) · --no-auth · binds 0.0.0.0:8080	https://llm1.local.xor0.de/v1 (Caddy → LAN) and http://<llm1>:8080/v1 direct. 502 when the Mac is asleep (Wake-on-LAN).

llm2 fleet sizing. The 4 slots come from the llm-bench agent_ctx capacity probe on rose: 4 coherent slots = 1 planner + 3 coders sharing one unified KV pool. --kv-unified is required so the pool is shared (a non-unified split would starve the deep planner). ctx-size is pinned to 262144 = Gemma4-26B's train context (build b10210 caps a unified pool at n_ctx_train).

llm2 serves one model. llama-server runs in single-model mode, so GET /v1/models returns exactly one entry — google/gemma-4-26B-A4B-it-qat-q4_0-gguf:Q5_0 — and nothing from the HF cache leaks in. The active model and every tunable live in machines/llm2/llama-server.conf.

Switching models is a conf edit + sudo systemctl restart llama-server (a full model load, not a hot swap). machines/llm2/models.ini is retained as a reference catalog — not deployed, not read by the service — recording params tested on this box for Qwen3.6-35B (APEX I-Compact, MTP IQ4_XS, heretic MTP-preserved, abliterated APEX), Gemma4-26B (Q4_K_M, QAT Q4_K_XL), Dolphin-Mistral-24B-Venice and Skyfall-31B-v4.2, each in ·kv-q5_0 (quality) and ·kv-q4_0 (lower-VRAM) variants. Copy a section's values into the conf to switch.

llm1 / OptiQ. Auto-started on interactive remote SSH login (trigger in machines/llm1/, sourced from ~remote/.bash_profile); it is a persistent daemon the harness only health-checks (never launches/kills). --max-context auto gives a growing, memory-bounded KV cache (validated coherent to 64k on 2026-07-23); a hard --max-context cap is not set — it would force a silently-evicting RotatingKVCache. The serve script itself lives in the llm-bench repo (scripts/llm1/serve.sh) as harness code; only the deploy/autostart wiring is here.

The autostart trigger (machines/llm1/optiq-login-start.sh) sets HOST=0.0.0.0 so OptiQ listens on the LAN, letting Caddy on llm2 reverse-proxy https://llm1.local.xor0.de → 192.168.1.150:8080. Security: OptiQ runs --no-auth, so binding 0.0.0.0 exposes the unauthenticated OpenAI API to the whole LAN (both via Caddy TLS and directly on :8080). This is LAN-only (pfSense forwards only 80/443) and matches llama-server's posture; to lock it to Caddy-only, add a Caddy IP-allowlist or basic-auth on the @llm1 block, plus a Mac firewall rule on :8080.

Two llm1 client-side requirements — every OpenAI-compatible client of llm1 has to satisfy both, so check these first when a client misbehaves:

1. Send an explicit max_tokens — measured, not folklore. With the field absent OptiQ returns exactly 512 tokens and finish_reason: "length"; with max_tokens: 2000 it returns 2000. A tool-using turn therefore ends mid-word with no visible answer. Mastra does this in mastra/src/mastra/gateways.ts (defaultMaxTokens: 16384); opencode derives it from limit.output.
2. Send a seed — a belt-and-braces measure, currently redundant. mlx_lm.server picks its generation path with _is_batchable = model.is_batchable and args.seed is None; without a seed it routes to BatchGenerator, which never quantizes the KV cache, so KV stays fp16 (~4× int4) and OOM-crashes the M1 against its 28 GB wired limit on a deep-context decode. The running server already forecloses that path via --max-concurrent 1 --kv-bits 4 (check with ssh llm1 'ps -eo args | grep [o]ptiq'), so the seed only matters if those flags are ever dropped. See llm-bench/runners/optiq-server.mjs.

llm1 is slow for multi-turn work. The int4-KV path has no usable prefix cache (prefix_cache_speedup ≈ 1.0 — cold 119,694 ms ≈ warm 119,542 ms), so every turn re-prefills the whole conversation at ~60 tok/s. It is a fine second opinion on short prompts; llm2 (2,392 tok/s prefill, 177.8× prefix cache) is the one to drive agent loops with.

---

opencode

A headless opencode server runs in the opencode container and is driven remotely from the opencode Desktop app on a dev box. Sessions, files and shell commands all live server-side; the Desktop app is only a client.

• Connect: point the Desktop app at https://opencode.local.xor0.de. No auth, matching the @llm / @llm1 posture — LAN-only by DNS. To add HTTP basic auth, set OPENCODE_SERVER_PASSWORD (and optionally OPENCODE_SERVER_USERNAME) on the service; verify the Desktop app accepts credentials before relying on it.
• Browser fallback: if the Desktop app cannot take a remote URL, change the container CMD from opencode serve to opencode web — same server and API, plus a browser UI on the same hostname.
• Workspace: opencode-workspace volume mounted at /workspace, empty on first start. Clone from Forgejo inside it; the container has a git.xor0.de → host-gateway entry so clones hit Caddy directly instead of relying on NAT hairpin.
• Sandbox: the agent has bash, but no Docker socket and no host filesystem. That isolation is the reason this is a container and not a systemd unit like llama-server.

Both inference hosts are registered as OpenAI-compatible providers in opencode/opencode.json; /models in the client switches between them. These values are load-bearing and must not be "tidied":

Setting	Value	Why
model-level temperature	true	A capability flag, not a value — but omitting it is not neutral. With it absent opencode treats the model as not supporting the parameter and strips temperature from the request body entirely, so the agent-level value below never leaves the container. Confirmed on the wire. top_p is not gated this way and is always sent.
agent.build/plan.temperature	1.0 (with top_p 0.95)	The fallback for any model that does not set its own. Gemma4 degenerates into repetition/looping below ~0.5 on coding tasks (see machines/llm2/llama-server.conf, which deliberately sets no sampling flags and leaves this to the caller). The agent block is global, not per-provider — a per-model options entry is how a second model escapes it (see the llm1 rows).
llm2 model limit.context	102400	The planner budget from the agent_ctx probe, not the full 262144 pool. Raising it to CTX would starve the three coder slots and Mastra's pinned slots 0/1.
llm2 model limit.output	8192	Drives the max_tokens opencode sends (it reserves a little — the wire value lands slightly under). Deliberately modest: this is the length of the rope a runaway generation gets. It was 32768, and on 2026-07-28 Gemma4 used every token of it — see below.
llm2 model options.repeat_penalty / repeat_last_n	1.1 / 256	Gemma4's GGUF defaults leave every repetition control off (repeat_penalty 1.0, dry_multiplier 0.0, presence/frequency 0.0). With the 1.0 temperature above there is otherwise nothing at all damping a loop. llama.cpp's OpenAI endpoint accepts both as extensions.
llm1 model limit.context	65536	Validated coherent to 64k; 128k is incoherent and ~13 tok/s.
llm1 model limit.output	8192	Sent as max_tokens; this alone defeats OptiQ's 512 default, which is the reason it must be set at all. A separate options.maxOutputTokens is not needed — it passes through as an unknown extra field and does nothing. Was 16384; at ~13 tok/s that is a twenty-minute runaway, worse in wall-clock than the llm2 incident.
llm1 model options.presence_penalty / presence_context_size	1.5 / 256	The only way to give llm1 any repetition damping. mlx_lm/server.py reads presence/repetition/frequency penalties from the request body with a hardcoded 0.0 default and never consults cli_args for them — unlike top_p/top_k/min_p, which do fall back to the CLI. So no server-side flag can supply one, and optiq's startup recipe (temperature=0.7, top_p=0.8, top_k=20, min_p=0.0) contains no penalty. presence_penalty rather than repetition_penalty because that is what the Qwen3.6 model card prescribes; the card leaves repetition_penalty at a no-op 1.0.
llm1 model options.temperature / top_p / top_k	0.7 / 0.8 / 20	Qwen3.6's published recipe. Per-model options override the agent-level temperature/top_p — verified on the wire, body showed 0.7/0.8 and not 1.0/0.95. Without these, llm1 inherits Gemma4's deliberately-high 1.0/0.95, which is well above Qwen3.6's recommended sampling and was compounding the missing penalty.
llm1 model options.seed	0	Free-form per-model options are merged verbatim into the request body. See requirement 2 above — currently redundant, kept as insurance.

Verifying what is actually sent. llama-server exposes live per-slot sampling params, which is the quickest way to check a client without touching the server: curl -s localhost:8090/slots while a generation is in flight and read the entry with is_processing: true. A correct opencode request shows temp=1.0 top_p=0.95 top_k=64. Note that idle slots retain the previous request's params, so only the processing slot is meaningful. Useful baseline: llm2's Gemma4 GGUF ships those same three values as its own defaults, so a request that omits temperature still lands on 1.0 — which means a missing temperature is invisible on llm2 and only shows up on llm1.

llm1 has no /slots equivalent, so verify it a different way. Two cheap checks, no proxy needed: mlx_lm validates known fields and tears down the connection on a bad value, while ignoring unknown fields outright — so repetition_penalty: "abc" returning HTTP 000 proves the field is parsed, where a made-up field returns 200. To see the whole body opencode actually sends, point the provider's baseURL at a small logging proxy that rewrites model and forwards to llm2; that is how the options-beats-agent precedence above was established. Undo it with git reset --hard origin/main on llm2 afterwards so the worktree stays clean.

One prompt occupies two slots, and they do not look alike. Alongside the temp=1.0 coding turn you will see a second slot at temp=0.5: opencode's internal title/summarise helper, which runs on small_model and hardcodes its own sampling — it is not configurable via the agent block and not a sign that the config failed to apply. It generates a handful of tokens, so the Gemma4 repetition warning does not bite there. Budget two slots per active session when sizing against N_PARALLEL=4.

A runaway generation looks exactly like a hang (observed 2026-07-28). Gemma4 produced 32,000 tokens in one unbroken response — 8m17s at 64 tok/s — and stopped only because it hit max_tokens. The client showed nothing the entire time: the model was inside a tool call, and opencode buffers tool-call arguments until the call closes, so there was no text to stream and no tool to display. Before assuming the server is stuck, check /slots — is_processing: true with a climbing n_decoded in journalctl -u llama-server means it is working, not wedged. The trigger was the sampler having no repetition control whatsoever (see the table); the fix was repeat_penalty plus a much smaller limit.output. Note also that the failed turn left the session at 91,460 tokens against a 102,400 limit — a runaway effectively ends the session even after it terminates, because every subsequent turn re-prefills the garbage.

opencode floats across whatever llama-server slots are free (no id_slot pinning — it has no hook for it). Mastra pins slots 0 and 1, so a busy Mastra plus three opencode agents saturates all four. Aggregate throughput plateaus at 4-way anyway: 116 → 116 → 171 → 179 tok/s at 1/2/4/8 concurrent.

---

Endpoints

Public (*.xor0.de — via pfSense NAT)

URL	Backend
https://git.xor0.de	Forgejo
https://pages.xor0.de	/srv/pages static files
https://<repo>.<user>.pages.xor0.de	/srv/pages/<user>/<repo>/
https://gmail-eckardt.mcp.xor0.de	Google Gmail MCP (restricted)
https://calendar-eckardt.mcp.xor0.de	Google Calendar MCP (restricted)
https://drive-eckardt.mcp.xor0.de	Google Drive MCP (restricted)

Internal (*.local.xor0.de — LAN only)

URL	Backend
https://llm.local.xor0.de/v1	llama-server on llm2 (OpenAI-compatible)
https://llm2.local.xor0.de/v1	llama-server on llm2 (alias of llm.local)
https://llm1.local.xor0.de/v1	OptiQ on llm1 (Mac Studio, over LAN — 502 when asleep)
https://search.local.xor0.de	SearXNG (search UI + JSON API)
https://mastra.local.xor0.de	Mastra API (agent endpoints)
https://mastra-studio.local.xor0.de	Mastra Studio (chat UI)
https://sonarqube.local.xor0.de	SonarQube (code analysis UI)
https://opencode.local.xor0.de	opencode server (Desktop app / HTTP API)

Internal (*.local.geist.md — LAN only)

URL	Backend
https://storybook.local.geist.md	geist-storybook (nginx, Storybook static)

---

Configuration

cp .env.example .env
# fill in: CLOUDFLARE_API_TOKEN, CROWDSEC_BOUNCER_API_KEY,
#          MIRROR_BOT_WEBHOOK_SECRET, MIRROR_BOT_FORGEJO_TOKEN

Live values for all secrets are in SECRETS.md.

---

First deploy

# 1. Copy repo to llm2
scp -r ~/repos/infra llm2:~/infra      # from Windows: scp -r "$env:USERPROFILE\repos\infra" llm2:~/infra

# 2. Fill in .env (see SECRETS.md for values)
ssh llm2 "cd ~/infra && cp .env.example .env && nano .env"

# 3. Build the custom Caddy image (first time only — CI handles rebuilds after)
ssh llm2 "cd ~/infra && docker build -f Dockerfile -t git.xor0.de/demonkoryu/caddy:latest . && docker push git.xor0.de/demonkoryu/caddy:latest"

# 4. Start everything
ssh llm2 "cd ~/infra && docker compose up -d"

pfSense:

• DDNS: Cloudflare, interface WAN, hostname @, domain xor0.de, token = CLOUDFLARE_API_TOKEN
• NAT port forwards: TCP 443 → 192.168.1.120:443, TCP 80 → 192.168.1.120:80

Cloudflare DNS:

• xor0.de A record → WAN IP, proxy OFF
• *.xor0.de CNAME → xor0.de, proxy OFF
• *.local.xor0.de A record → 192.168.1.120, proxy OFF (internal services)
• geist.md A record → 192.168.1.120, proxy OFF
• *.geist.md A record → 192.168.1.120, proxy OFF
• *.local.geist.md A record → 192.168.1.120, proxy OFF (internal services)

LE staging test (before going live): uncomment acme_ca staging line in Caddyfile, bring up Caddy, confirm (STAGING) Fake LE Root in cert issuer, then remove it.

---

Runner registration

One-time setup after first deploy:

# Get a registration token: Forgejo → Site Admin → Actions → Runners → Create new runner
ssh llm2 "cd ~/infra && mkdir -p runner-data && sudo chown -R 1001:1001 runner-data"
ssh llm2 "cd ~/infra && docker compose run --rm --entrypoint forgejo-runner runner \
  register --no-interactive \
    --instance http://forgejo:3000 \
    --token <TOKEN> \
    --name llm2-runner \
    --labels 'docker:docker://node:20-bookworm,ubuntu-latest:docker://node:20-bookworm,deploy:docker://docker:27-cli'"
ssh llm2 "cd ~/infra && docker compose up -d runner"

Windows host runner

A second, native Windows runner (host mode — jobs run directly on the box, no container) runs on a Windows dev machine for runs-on: windows workflows. Setup, install script, and runbook live in windows-runner/. All its state is under F:\ci; it runs as a Windows service.

---

Operations

# Update web services
ssh llm2 "cd ~/infra && docker compose pull && docker compose up -d"

# Update LLM server (systemd service, not a container)
# Rebuild llama.cpp only when bumping versions: ssh llm2 "bash ~/infra/machines/llm2/build-llamacpp.sh"
ssh llm2 "cd ~/infra && bash machines/llm2/deploy.sh"

# Update central PostgreSQL
ssh llm2 "cd ~/infra && docker compose -f postgres/docker-compose.yml pull && docker compose -f postgres/docker-compose.yml up -d"

# Update SonarQube
ssh llm2 "cd ~/infra && docker compose -f sonarqube/docker-compose.yml pull && docker compose -f sonarqube/docker-compose.yml up -d"

# Rebuild opencode (image is built locally — `pull` won't update it)
ssh llm2 "cd ~/infra && docker compose up -d --build opencode"

# Apply Caddyfile changes after a `git pull`.
# `caddy reload` alone is NOT enough: /etc/caddy/Caddyfile is a single-file bind mount, and git
# replaces the file (new inode) rather than rewriting it in place, so the container keeps reading the
# old copy and the reload reports "config is unchanged" while quietly changing nothing. A restart
# re-resolves the mount. Verify with:
#   docker compose exec caddy grep -c <new-hostname> /etc/caddy/Caddyfile
ssh llm2 "cd ~/infra && docker compose restart caddy"

# Logs
ssh llm2 "cd ~/infra && docker compose logs -f caddy"
ssh llm2 "cd ~/infra && docker compose logs -f forgejo"
ssh llm2 "cd ~/infra && docker compose logs -f mirror-bot"
ssh llm2 "cd ~/infra && docker compose logs -f opencode"
ssh llm2 "journalctl -u llama-server -f"
ssh llm2 "cd ~/infra && docker compose -f postgres/docker-compose.yml logs -f central-db"
ssh llm2 "cd ~/infra && docker compose -f sonarqube/docker-compose.yml logs -f sonarqube"

# Reload Caddy config without downtime
ssh llm2 "docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile"

# Backup
ssh llm2 "cd ~/infra && docker compose down && \
  tar czf infra-backup-\$(date +%F).tar.gz forgejo-data && \
  docker run --rm -v infra_caddy-data:/data -v \$(pwd):/backup alpine \
    tar czf /backup/caddy-data-\$(date +%F).tar.gz /data && \
  docker compose up -d"

# Add a new subdomain service: add a matcher + handle block in Caddyfile, then reload Caddy

---

Central PostgreSQL

The house relational DB — one shared postgres:17-alpine server (central-db), running as a separate compose project (postgres/docker-compose.yml) on the external services network. Per-app databases + owner roles are provisioned by postgres/initdb.d/10-app-databases.sh on first init; add a provision_app_db line there as new db-centric apps come up.

• Not HTTP → not behind Caddy. Reachable in-cluster at central-db:5432 (CI loaders, other containers on services) and on the LAN at 192.168.1.120:5432 (dev box, DuckDB ATTACH). The published port is bound to the LAN IP; pfSense forwards only 80/443, so :5432 never leaves the LAN.
• Current tenants: llmbench (role + database) holds the llm-bench tidy benchmark rows. The dashboard's build-time loader reads it once via DuckDB's postgres extension (ATTACH '…' AS pg (TYPE postgres)) and bakes a static snapshot, so the public page needs no per-visit DB connection.
• Secrets: POSTGRES_PASSWORD (superuser) + LLMBENCH_DB_PASSWORD (app role) in postgres/.env (per-service, git-ignored — copy from postgres/.env.example, generate with openssl rand -hex 32). Both are consumed only on first init; rotate an existing cluster with ALTER ROLE … PASSWORD.
• Backups: all data lives in the central-db-data volume — include it in any backup. A logical dump is docker compose -f postgres/docker-compose.yml exec central-db pg_dumpall -U postgres.
• Not consolidated (yet): SonarQube keeps its own private sonarqube-db. Folding it into central-db is a possible future migration, not done here.

# First deploy (the external `services` network must already exist — brought up by the main stack)
scp -r postgres llm2:~/infra/          # or rsync; ships compose + initdb.d
ssh llm2 "cd ~/infra/postgres && cp .env.example .env && nano .env"   # fill in both passwords
ssh llm2 "cd ~/infra && docker compose -f postgres/docker-compose.yml up -d"

# Verify the llmbench role + database were provisioned
ssh llm2 "docker exec central-db psql -U postgres -c '\\l' -c '\\du'"

Deploy/log commands are in Operations.

---

SonarQube

Static code analysis (Community Build), running as a separate compose project (sonarqube/docker-compose.yml): sonarqube + its own PostgreSQL (sonarqube-db) on the external services network, proxied by Caddy at https://sonarqube.local.xor0.de (LAN only). First login is admin/admin (forced password change). Deploy/log commands are in Operations.

• Host requirement: the bundled Elasticsearch needs vm.max_map_count >= 524288 or sonarqube boot-loops. Already satisfied on llm2 (/etc/sysctl.d/10-map-count.conf → 1048576); a fresh host must set it persistently.
• Secrets: SONARQUBE_DB_PASSWORD in sonarqube/.env (per-service, git-ignored — copy from sonarqube/.env.example).
• Backups: analysis history lives in the sonarqube-db-data volume — include it in any backup.

---

GitHub mirroring

Every repo created in Forgejo is automatically mirrored to github.com/demonkoryu/<repo> (private).

How it works:

1. Forgejo fires a system webhook on repository.created → mirror-bot dispatches the sync-github-mirrors workflow via the Forgejo API.
2. The workflow (mirror-bot/sync_mirrors.py) ensures every repo has a GitHub mirror: creates the GitHub repo if needed and adds a push mirror.
3. The same workflow runs on a 15-minute cron schedule as a catch-all for any repos that slipped through.
4. Each push to Forgejo syncs to GitHub immediately (push mirror with sync_on_commit: true).

System webhook (already configured): http://mirror-bot:8080/webhook, event: repository.

Forgejo Actions secrets (on the infra repo): MIRROR_FORGEJO_TOKEN, MIRROR_GITHUB_TOKEN.

---

Notes

• NAT reflection: CI jobs and mirror-bot call git.xor0.de from inside the host. Requires either pfSense NAT reflection enabled, or a DNS host override (git.xor0.de → 192.168.1.120) in pfSense DNS Resolver.
• Runner trust: the Actions runner mounts the host Docker socket — job containers run on the host daemon. Treat all workflow-enabled repos as trusted. Deploy jobs (label deploy) can build images and manage containers directly.
• Windows runner trust: the windows/windows-latest host-mode runner (windows-runner/) runs job steps as native processes under the demonkoryu user account — no isolation. Same rule: only trusted repos.
• LE rate limit: 5 duplicate certs/domain/week. Use staging for testing; the wildcard covers all subdomains so issuance is rare.
• Sub-subdomains: *.xor0.de does not cover *.foo.xor0.de. Each needs its own Caddy server block and cert (e.g., *.local.xor0.de for internal services).