A self-hosted vLLM fallback that keeps sovereign routes alive

Why keep a runtime you hope not to use

Every failover graph has a last node. For EU tenants it is usually a second or third provider region inside the EEA, and for most outages that is enough. It is not enough when a whole provider is down, when the contract says Germany only and the German options run out, or when the policy says certain data may not reach third-party infrastructure at all. For those cases the last node should be a runtime you operate, on hardware in your own data centre or VPC, serving an open-weight model.

A self-hosted endpoint is the one place where you control every attribute the policy engine checks: the country, the operator, the retention, the network path and the keys. It also answers questions beyond failover. Our saudi-pdpl-sovereign pack requires customer-operated infrastructure for government-classified data, and the DORA register asks whether a service could be reintegrated in-house. A tested open-weight runtime is a straightforward yes to both. vLLM is a common choice because it serves an OpenAI-compatible API, so the gateway and your application treat it like any other endpoint.

Pick a model you may run and can afford to run

Licences come first. Several strong open-weight families ship under Apache 2.0, including Mistral's Small models, the Qwen3 family and OpenAI's gpt-oss. Others, such as Llama, use their own community licences with acceptable use terms your legal team should read. Record the licence and the exact weights you approved in the same inventory that lists your hosted models.

Then do the memory arithmetic, because it decides the hardware. Weights in BF16 take two bytes per parameter, so a 24 billion parameter model needs about 48 GB before it serves a single token. On one 80 GB GPU at 90% memory utilisation that leaves roughly 24 GB, less runtime overhead, for the KV cache, which holds the context of concurrent conversations. Split the model across two such GPUs with tensor parallelism and the same arithmetic leaves about 96 GB for cache. Quantised weights shrink the first number at some cost in quality, which your evaluation set should measure rather than assume. OpenAI describes gpt-oss-120b as fitting on a single 80 GB GPU because its weights ship quantised.

Serve it with nothing leaving the box

A sovereign fallback that phones home is not sovereign. Put the inference nodes on a network segment with outbound traffic denied, stage weights from an internal artefact store, and check them against pinned hashes before every start. Tell the Hugging Face libraries not to reach the hub, and turn off vLLM's anonymous usage statistics, which by default report hardware and model configuration, not prompts. Do not pass --trust-remote-code, which executes Python shipped inside a model repository; choose an architecture vLLM supports natively instead.

• Current vLLM releases leave request logging off unless you pass --enable-log-requests, and --enable-log-outputs adds generated text on top.
• Older releases logged requests unless you passed --disable-log-requests, so check the behaviour of the exact version you pin.
• Whichever version you run, send a test prompt and confirm nothing containing it reaches your log pipeline, because that is the claim you will make in a security questionnaire.

# Inference nodes: outbound traffic denied at the firewall.
export HF_HUB_OFFLINE=1
export VLLM_NO_USAGE_STATS=1
export DO_NOT_TRACK=1

cd /models/fallback-24b-instruct && sha256sum -c SHA256SUMS || exit 1

vllm serve /models/fallback-24b-instruct \
  --served-model-name fallback-drafting \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --api-key "$(cat /run/secrets/vllm_api_key)" \
  --host 10.20.0.15 --port 8000

Register it like any other endpoint

To the gateway, the runtime is an endpoint in the registry with the same attributes as a provider region: country, facility, operator, retention flag and the SHA-256 thumbprint of its TLS certificate, which goes into every receipt it serves. The policy decides when it may be used, the failover graph decides where it sits in the order, and the approval is per task class rather than wholesale.

endpoint: vllm-eu-customer
runtime: vllm
base_url: https://inference.fra-dc1.internal/v1
country: DE
facility: customer-dc-frankfurt-1
operator: customer
zdr: true                     # request logging off, usage statistics off
tls_cert_sha256: 5b7e...0c93
models:
  fallback-drafting:
    weights_sha256: 9f2c...e41a
    licence: Apache-2.0
    approved_task_classes: [extract, classify, summarise]
health:
  probe: synthetic_completion
  interval_seconds: 10

Know what it can do before you need it

The fallback will be weaker than the frontier model it replaces on some tasks, and that is acceptable if you know which ones. Run the same evaluation set you use for the primary model against the fallback, per task class, and compare. In our experience smaller open-weight models hold up well on extraction, classification and summarisation of structured documents, and fall further behind on long multi-step drafting and complex reasoning. Your results will differ; the point is to have them before an outage forces the question.

Approve the fallback only for the task classes where it meets your threshold. For the rest, the honest behaviour during an outage is the fail-closed error, not a degraded answer nobody evaluated. Write the decision down: for systems your inventory classifies as high-risk under the AI Act, the fallback model has to be on the approved list like any other, with its own test results in the technical documentation.

Keep it warm and keep it current

A runtime that serves no traffic decays. Drivers change, certificates expire, and someone reclaims the GPUs for a training job. Send it a small, steady share of real traffic for approved task classes, a few percent is enough, and probe it every ten seconds like any other endpoint. vLLM exposes a /health endpoint and Prometheus metrics at /metrics, so the dashboards that watch your providers can watch it too.

Size it for the traffic that cannot wait, not for full production load. If 15% of requests come from task classes that must keep running during an outage and peak load is 40 requests per second, the fallback has to sustain 6. Treat upgrades the way you treat a provider's model change: new weights get new hashes, a new evaluation run and a new approval before the registry points at them. Then add the runtime to the quarterly game day, so the last node in the graph is tested as often as the first.