ShanHaiGateway · Shanhai Pass — strings a pile of unreliable upstreams into one OpenAI-compatible endpoint

01 / WHY

Why it looks like this

The backstory first. This is not a paper turned into code — it was squeezed out by free pools and the bill.

Every so often a vendor drops a generous free model. People farm it with a pile of accounts, resell access, the vendor notices — and the free tier dies.

I run the same playbook, with two differences: everything I learn is documented here for free; and I don't leech — the paid channels actually get paid (¥100+ a month out of pocket), and commercial lanes, vouchers and free pools share the load. In the October 2026 measurement this pushed 20 billion tokens a month for ≈ ¥170.

The price is a pile of failure modes. Every pit below was stepped on in production, and each pit bought back one rule — all recorded in the ops-semantics table. Shanhai Pass is the gateway that grew out of those rules: twisting free pools, vouchers and pay-as-you-go plans into one stable upstream pool.

You wire three upstream providers into a rotation, and then this starts happening:

production-incidents.log

001503An upstream has a bad day and the whole product 503s with it. retry can't save this.

002cachePrefix-cache hit rate 51–56% — exactly the in-pool sampling probability. Rotation is the ceiling.

0034d+An upstream silently delists a model; a backend idles at 0% success for four and a half days before anyone notices.

0041720×One disconnect storm prints 1720×503; healthy backends cool down one by one until no backend is left.

005keyKeys scattered everywhere, and the repo can never see daylight.

$→every pit bought back one rule

02 / FEATURES

What it does

The core in one sentence: make unreliable upstreams do reliable work together. Everything else serves that sentence.

FEAT-01

SWRR weighted rotation + isolated fallback chain

Weight, concurrency and timeout adjustable per backend. Fallback backends live in their own pool and never pollute the main one.

FEAT-02

Error-taxonomy-driven cooldown

403/429 exponential backoff 5m→10m→…→4h cap; saturation queues for 60s instead of throwing 503; client disconnects never trigger cooldown.

FEAT-03

Task-level session affinity

Prefers X-Job-Id, falls back to X-Session-Key, pins every turn of a task to the same backend. Measured prefix-cache hit rate 27% → 51%, peaking at 69%.

FEAT-04

Quirk layer

Vendor-specific error bodies / usage keys live in their own package; zero name-based branching in the kernel. Vendors without quirks need zero config.

FEAT-05

Multi-user downstream key billing

sk-user-* quota issuance + pluggable login (registry pattern) + a stock user console.

FEAT-06

Shadow capture observability

Single SQLite store for attempts/captures, auto-scrubbed before write; fine-grained /api/stats dashboard.

FEAT-07

Quota-window probe + auto-demotion

Queries each vendor's 5h/7d/30d quota window per backend name; the sidecar judges every 60s: exhausted → retired, restored → back in. Adding a subscription changes no mapping table.

FEAT-08

Sidecar slow-loop self-healing

An independent Python process, stdlib only, zero dependencies. Rate-limit/degrade decisions from per-window failure rates, dividing labor with the gateway's fast path.

FEAT-09

Docker all the way

Multi-stage build, non-root runtime, built-in healthcheck, zero secrets in the image.

03 / NUMBERS

Measured scale and bill

2026-10, measured in production. Read the numbers together with the disclaimers at the end.

monthly token throughput (single-machine production)

¥0

monthly cash bill (list price ≈ $3,800)

0MB

Docker image size, zero CGO

upstream 429s on main business (the free pool collapsing doesn't touch it)

Free pools / cache reads

Shoulder ~63% of throughput; commercial lanes carry only ~37% of real requests. The free pool errors constantly — main-business upstream 429s stay at 0.

Session affinity: reconnecting the prefix cache

before affinity27%

after affinity51%

peak69%

Rotation deals the second turn to another vendor and prefix caching dies. Pin back to the same lane and the hit rate doubles.

At a production burn of ~1B tokens/day, with route spreading, cache affinity (70% hit rate) and rolling rotation of "multiple voucher accounts + free grant pools":

Channel	Count	Monthly tokens	Monthly cash
Volcengine subscription	4	8B	2 paid in cash ¥99.8, the other 2 covered by channel vouchers (¥10 left on them)
opencode Go	2	1.2B (paid)	bought at half price, $5 each (≈¥70)
Free grant pools · credit earning · cache reads	—	~63%	¥0

→ ¥170 for 20B tokens ≈ ¥0.85 per 100M. Only 23% of tokens are billed at full fresh-token rates. Code alone (cache affinity + free-tier blending + per-request-size splitting) cuts list-price volume to 1/4–1/3.

Disclaimers — read them together with the numbers:

Promo prices have shelf lives. Vouchers and half-price deals can vanish anytime; back at list price it's ≈ ¥340/month — still 98%+ saved.

"Free/budget lanes carry 63%" fluctuates with upstream grants; measured range 37–57%. No specific quota is promised.

Free/granted quota is the upstream's gift, not this project's. What this project open-sources, forever: cache affinity, lane-switch retries, per-request-size splitting, and independent per-subscription metering with automatic demotion.

Numbers are single-machine production measurements from 2026-10 (daily burn 440M–1.3B tokens).

04 / QUICK START

To production in three minutes

Just want a working gateway: three Docker commands, swap the base_url, go. Weights, cooldown and affinity ship with sane defaults — the architecture section is optional.

bash — docker compose

# clone + prepare real config (into the mounted volume, never into git)
git clone https://github.com/ffseika0304/shanhai-gateway.git
cd shanhai-gateway
mkdir -p runtime/config runtime/data
cp config/backends.example.json runtime/config/backends.json
cp config/keys.example.json    runtime/config/keys.json

# start the container (host side defaults to :18016)
docker compose -f deploy/docker-compose.yml up -d --build

# smoke test
curl -s http://127.0.0.1:18016/health
{"status":"ok","version":"0.5.1"}

bash — Go ≥ 1.27

make build    # equivalent to: go build -o shg ./cmd/shg

cp config/backends.example.json config/backends.json
# fake credentials boot fine

export LISTEN=":18006"
export ROUTER_VALIDATE=0   # skip the startup outbound self-check

./shg &
curl -s http://127.0.0.1:18006/v1/chat/completions   -H 'Content-Type: application/json'   -d '{"model":"demo-model","messages":[{"role":"user","content":"hi"}]}'
# 200 means through

config/backends.json — minimal

{
  "accounts": [
    { "name": "vendor-a",
      "base_url": "https://api.example-llm.com/v1",
      "api_keys": ["sk-xxxxx"] }
  ],
  "backends": [
    {
      "name": "vendor-a-main",
      "account": "vendor-a",
      "model": "demo-model",
      "base_url": "https://api.example-llm.com/v1",
      "weight": 1,
      "concurrency": 8,
      "enabled": true,
      "is_primary": true
    }
  ]
}

The LAN port's POST /v1/chat/completions does not require a key (free lane); downstream key billing only activates on the public port / user console. Config hot-reloads on mtime — tune weights or add backends without restarting. Full field set: config/backends.example.json.

05 / OPS SEMANTICS

The rules table paid for by incidents

Pinning it into production, or rewriting the router? Read the "why" here first. Every magic number has an incident citation.

Client disconnects never cool down

Oct 3: one disconnect storm printed 1720×503 — client aborts booked as backend failures → healthy backends cooled down one by one → no backend left in the pool. Disconnects may only be classified as network (observed): no cooldown, no circuit.

"Model not found" is the backend's fault

When an upstream silently delists a model, the gateway sees 4xx/black-hole — a fallback backend once idled at 0% success for four and a half days before discovery. Such 4xx lands on the owning backend and walks cooldown/retirement, patched via model_expire / model_fallback identities.

Quota errors back off exponentially; saturation queues

403/429 exponential cooldown 5m→10m→…→4h cap, no repeated blind retries; saturation does not degrade — it queues up to 60s (QUEUE_WAIT) instead of throwing 503.

The blocklist is write-only; cleanup is manual

backends.blocked.json is the "permanently dead" list decided by startup self-check. No automatic unblock path: auto-rewhitening would use failure rate as evidence, while the blocklist's judgment is probe fact. Human review first, then delete from the file.

Strikes reset only on success — don't touch them

Oct 1, the 403 self-lock storm: strikes zeroed before quota recovered → the backend returned to the pool running on empty → instant 403 again → cooldown restart → loop. The whole backoff chain depends on "consecutive strikes zeroed only by success"; admin hot-editing deliberately provides no strike entry — by design, not an omission.

Session affinity = a preference layer outside rotation

Oct 10: measured prefix-cache hit rate 0% — SWRR deals consecutive turns of one conversation to different providers; prefix caches live provider-side. The affinity key prefers X-Job-Id (task-level — exactly the prefix-cache reuse unit), falling back to X-Session-Key. Only a final success with delivered content writes the table; eviction/keep decisions share one source with cooldown/circuit — no second health layer invented.

Per-vendor quirks go in the quirk layer; zero name branches in the kernel

Folding into universal rules hurts everyone else; if name==xxx sprinkles name checks into the kernel — both wrong roads. "Problems only one vendor has" go into internal/core/quirk/xxx.go; the kernel consults it at three fixed seams (egress decorate / error classify / usage parse). Quirks are data (config fields), not code branches.

The observability contract stays SLR-isomorphic

The attempts table carries a job_id column (idempotent ALTER migration for existing DBs); /api/stats exposes the isomorphic affinity{hits,misses,puts,pinned_jobs} object. Scripts attributing by job keep working after migration.

06 / FIXES

Troubleshooting

All actually asked before.

The gateway won't start, complaining "no backend config"

env points at the wrong directory. GATEWAY_CONFIG_DIR must contain backends.json; in docker, check that the mounted runtime/config/ really has files (docker logs shg shows how many backends were parsed at startup).

A backend never gets picked

Three fields in GET /api/stats: enabled (disabled in config), cooldown_remain_s (quota-class cooldown, up to 4h), circuit_open. Note circuit_open:false does not mean no cooldown — the two are separate counters.

Every backend is cooling down and all requests fail

This is exactly the shape rule #1 exists to prevent. Check fail_breakdown: lots of network failures usually means a client disconnect storm misfiring (old-version semantics) — confirm you're running the current version; a concentrated burst of 401s means dead keys — cooldown won't heal until you replace them.

The dashboard has data and I want the raw requests

Query the SQLite captures table (read the captures consumption contract in the repo first). Everything is scrubbed before storage (keys / intranet paths / phone numbers / Bearer headers → [mask]) — that's a feature, not a flaw.

The docker container keeps restarting

docker logs shg for the exit reason; most likely runtime/config/ is missing files or has invalid JSON. A failing healthcheck also makes the orchestrator mark it unhealthy, but restart: unless-stopped only watches process exit.