Self-hosting OmniRoute as my LLM gateway

Every agent on my Mac mini — Claude Code, OpenCode, whatever I’m evaluating this week — needs to talk to a grab-bag of LLM providers, some domestic Chinese, some not. Instead of handing each tool its own set of keys and base URLs, I run OmniRoute as a single gateway: one endpoint, one key store, provider failover, and request logs I can actually read. It runs as one Docker container (diegosouzapw/omniroute:latest) on port 20128, via OrbStack, on a host that also runs Clash Verge in TUN mode and Tailscale.

That last sentence is where all three of my problems came from. Here’s the setup, and the potholes.

The compose file

The whole service fits in one file, and the comments carry the why:

services:
  omniroute:
    image: diegosouzapw/omniroute:latest
    container_name: omniroute
    restart: unless-stopped
    stop_grace_period: 40s
    ports:
      - "20128:20128"
    # DNS: the host's resolver path (Tailscale MagicDNS 100.100.100.100) returns
    # STALE/POISONED A records for CN provider APIs (open.bigmodel.cn ->
    # 8.208.31.28, api.minimaxi.com -> 47.79.117.67). Those IPs are dead
    # endpoints: TCP connects, then the TLS handshake is RST -> 502.
    # Pointing the container at a public resolver makes the Clash TUN
    # (dns-hijack: any:53) return fake-IPs (198.18.0.0/16) instead, and Clash
    # maps fake-IP -> real IP correctly on connect.
    dns:
      - 223.5.5.5
      - 119.29.29.29
    environment:
      HOSTNAME: 0.0.0.0
      PORT: 20128
      NEXT_PUBLIC_BASE_URL: http://192.168.1.11:20128
      # Chat admission control (see chatBodyAdmission.ts):
      # - MAX_HEAVY_IN_FLIGHT: concurrent heavyweight requests allowed process-wide.
      #   Default 1 serializes agent fan-out (OpenCode/Claude Code); 4 lets bursts queue.
      # - ADMISSION_QUEUE_MS: how long a heavy request waits for capacity before a
      #   retryable 503. Default 2000 (immediate 503 burns the client's retry budget).
      OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT: 4
      OMNIROUTE_CHAT_ADMISSION_QUEUE_MS: 20000
      # Provider rate limiter (open-sse/services/rateLimitManager.ts).
      # MAX_WAIT_MS does double duty: it is passed to Bottleneck as `expiration`,
      # i.e. the deadline for how long a request may run AFTER it leaves the
      # queue. Default 15000ms (15s) killed long reasoning models mid-flight
      # (glm-5.3, MiniMax-M3, k3) -> 504 RATE_LIMIT_EXECUTION_TIMEOUT; it is
      # NOT an upstream timeout and does not bound queue wait.
      # 120000 gives extended-thinking models room to produce a first token.
      RATE_LIMIT_MAX_WAIT_MS: 120000
      # Because MAX_WAIT_MS is also the queue-wait allowance, pair the raise
      # with a depth cap so a burst fast-rejects (RATE_LIMIT_QUEUE_FULL)
      # instead of growing the queue unbounded. Default 0 = unbounded.
      RATE_LIMIT_MAX_QUEUE_DEPTH: 100
    volumes:
      - omniroute-data:/app/data

volumes:
  omniroute-data:

Nothing exotic: one container, one named volume, four knobs. Each knob exists because a default bit me.

The 502s that weren’t the container’s fault

First symptom: random 502s, but only on the Chinese provider APIs — open.bigmodel.cn and api.minimaxi.com. Everything routed to non-CN providers was fine, which already smelled like network, not app.

My first instinct was nc -z open.bigmodel.cn 443 from the container. It succeeded. Case closed, network fine, must be OmniRoute’s bug — except it wasn’t. nc -z proves a TCP handshake completed, nothing more. When I split the layers properly, the picture changed: TCP connected, then the TLS handshake got RST. A dead endpoint that still answers SYN is exactly what a poisoned DNS record looks like.

Checking the resolver path explained the rest. The host resolves through Tailscale MagicDNS (100.100.100.100), and for those two domains it was returning stale records — open.bigmodel.cn -> 8.208.31.28, api.minimaxi.com -> 47.79.117.67 — IPs that accept TCP and die at TLS. The debugging move that saved me was comparing the host resolver’s answer against DNS-over-HTTPS truth, because port 443 is not hijacked by Clash:

curl -s -H 'accept: application/dns-json' \
  "https://223.5.5.5/resolve?name=open.bigmodel.cn&type=A"

DoH came back with different, working IPs. So the fix wasn’t in OmniRoute at all — it was deciding who the container asks for DNS. Pointing dns: at public resolvers (AliDNS 223.5.5.5, DNSPod 119.29.29.29) looks like it would bypass Clash, but it does the opposite, and understanding why requires knowing how the TUN’s fake-IP mode works:

┌──────────────┐   DNS: open.bigmodel.cn?    ┌─────────────────────────┐
│  omniroute   │ ──────────────────────────▶ │ Clash TUN (host)        │
│  (container) │   to 223.5.5.5:53           │ dns-hijack: any:53      │
└──────┬───────┘ ◀────────────────────────── └───────────┬─────────────┘
       │              fake-IP 198.18.x.x                 │
       │                                                 │ fake-IP -> real IP
       │ HTTPS to 198.18.x.x                             │ on connect
       ▼                                                 ▼
┌────────────────────────────────────────────────────────────────┐
│ Clash routing: rule match (direct for CN providers) -> real API │
└────────────────────────────────────────────────────────────────┘

dns-hijack: any:53 means Clash intercepts queries aimed at any resolver, UDP and TCP both, and answers with a fake-IP from 198.18.0.0/16. When the container then connects to that fake-IP, the connection lands on the TUN interface, and Clash maps the fake-IP back to the real destination and routes it per its rules. The container rides Clash’s routing correctly because its DNS was hijacked. The MagicDNS path, by contrast, answered the query for real — with a poisoned record — and Clash never got a say.

After the change, the 502s stopped. Not reduced — stopped.

Admission control for agent fan-out

Second problem, once traffic actually flowed: agents kept tripping over each other. Claude Code and OpenCode both fan out — subagents, parallel tool calls — and OmniRoute’s chat admission control (in chatBodyAdmission.ts) has two defaults tuned for a single human at a keyboard:

• OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT=1 — one heavyweight request in flight, process-wide. My “concurrent” agents were silently serializing behind each other.
• OMNIROUTE_CHAT_ADMISSION_QUEUE_MS=2000 — how long a queued request waits for capacity before a retryable 503. Two seconds sounds generous until a burst arrives; the immediate 503s were burning the clients’ retry budgets on what should have been a short wait.

Raising them to 4 and 20000 changes the posture from “reject fast” to “let bursts queue for capacity.” Fan-out actually runs in parallel, and a burst that would have cost a retry round-trip just waits its turn.

The Bottleneck expiration gotcha

Third problem, and the subtlest: long-reasoning models — glm-5.3, MiniMax-M3, kimi k3 — kept dying mid-generation with 504 RATE_LIMIT_EXECUTION_TIMEOUT. Fifteen seconds in, consistently. Nothing upstream was timing out; the providers were happy to keep streaming.

The culprit is RATE_LIMIT_MAX_WAIT_MS in rateLimitManager.ts, which does double duty. It’s passed to Bottleneck as expiration — the deadline for how long a request may run after it leaves the queue, not how long it may wait in line, and not an upstream timeout. Default 15000ms. An extended-thinking model that spends 20 seconds producing its first token isn’t slow; it’s normal, and the default was executing it mid-thought.

Raising it to 120000 gives those models room. But because the same value also serves as the queue-wait allowance, raising it alone would let a flood of requests pile up for two minutes each. So it ships paired with RATE_LIMIT_MAX_QUEUE_DEPTH: 100 (default 0, unbounded): a real overload now fast-rejects with RATE_LIMIT_QUEUE_FULL instead of growing the queue until everything times out anyway.

Final config summary

Checking it works

I deliberately didn’t verify any of this with one-off curls. A single successful request proves nothing when the failure mode is intermittent. The verification standard was OmniRoute’s own health data: every provider’s test_status green, and the request logs clean under sustained multi-agent load — real Claude Code and OpenCode sessions hammering it, not a loop of synthetic pings. That’s the state it’s been in since, restart: unless-stopped doing its one job quietly.

← Back to all posts