Failover without leaving the jurisdiction: a design for outages

How prompts leave during an outage

Ask an engineering team where their EU traffic goes and they will show you the primary region. Ask where it goes when the primary is down and the answer is often a pause, followed by a look at a retry wrapper someone wrote during an incident two years ago. That wrapper is where residency programmes fail. Not in steady state, which everyone designs and reviews, but in the forty minutes of a provider incident when the system will do whatever it takes to return a 200.

The pattern repeats across the architecture reviews we run. A Frankfurt deployment starts returning 429s and 503s. The SDK retries, a fallback list takes over, and the last entry on that list is a global endpoint or a US region, added because it always has capacity. The application keeps working and nobody is paged. Weeks later someone notices the cross-region line on the invoice, and the DPO has to decide whether an incident log is now a record of an unlawful transfer. It usually is.

• Latency-sorted fallback picks the nearest healthy endpoint, and in a regional incident the nearest healthy endpoint is often on another continent.
• Retry wrappers in application code frequently call the provider's API directly, bypassing whatever routing controls sit in the gateway.
• Some provider deployment types route requests across regions at the provider's discretion, so read the deployment type in the documentation, not just the region name in the console.

Draw the graph per jurisdiction

The design rule is easy to state: failover targets are a compliance decision, so they belong in policy, not in a latency table. For each jurisdiction we keep a failover graph with a primary, an ordered set of secondaries and a last resort, all drawn from the same legal zone. For an EU tenant under the eu-gdpr-strict pack that means Frankfurt first, then Dublin, Paris or the Netherlands, and nothing outside the EEA. For a tenant whose contract says Germany only, the graph is shorter: two German facilities and, if they run one, a runtime in their own data centre.

At request time the router builds the candidate list as the intersection of three sets: registry endpoints that serve the tenant's model class, endpoints the policy allows for this tenant and this payload, and endpoints currently passing health checks. It walks that list in order. Because the policy is evaluated for each candidate, a misconfigured group cannot widen the zone. If someone adds a US endpoint to an EU group, the policy denies it and the denial lands in the logs, where a reviewer will see it.

tenant: versicherung-claims-de
policy: eu-gdpr-strict@4.2.0
model_class: claims-summary
failover:
  primary: bedrock-eu-central-1      # Frankfurt
  secondary:
    - bedrock-eu-west-1              # Dublin
    - bedrock-eu-west-3              # Paris
  last_resort: vllm-eu-customer      # customer data centre, EU
  on_exhausted: fail_closed          # never widen the zone
health:
  probe: synthetic_completion
  interval_seconds: 10
  open_circuit_after: 5              # consecutive failures
  half_open_after_seconds: 30
  max_ttft_ms: 4000

Fail closed, and say why

When every candidate in the graph is unavailable, the gateway returns an error. It does not look further. Customers question this choice more than any other, and it is the one I would defend first. An outage is recoverable: you lose some minutes of AI features, the provider publishes a post-incident report, and users retry. A transfer without a legal basis cannot be taken back. The prompt has been processed, it may sit in a provider's abuse-monitoring logs, and the controller has to assess whether it amounts to a breach that must be notified under Article 33.

The error should be useful. We return a 503 with a Retry-After header and a body that names the policy, every candidate considered and the reason each was rejected, so the on-call engineer sees a compliance decision rather than a mystery.

• Queue the work and retry with exponential backoff and jitter, so recovering endpoints are not flattened by a retry storm.
• Degrade the feature visibly: hide the draft button, or route the case to a person.
• Never catch the error and call the provider directly. Search the codebase for provider SDK clients created outside the gateway wrapper, because that is where the bypass usually lives.

HTTP/1.1 503 Service Unavailable
Content-Type: application/json
Retry-After: 30

{
  "error": {
    "type": "no_compliant_endpoint",
    "message": "All endpoints permitted by eu-gdpr-strict@4.2.0 for tenant versicherung-claims-de are unavailable. The request was not forwarded.",
    "candidates": [
      { "endpoint": "bedrock-eu-central-1", "rejected": "circuit open: 5 consecutive 503 responses" },
      { "endpoint": "bedrock-eu-west-1", "rejected": "circuit open: time to first token above 4000 ms" },
      { "endpoint": "bedrock-eu-west-3", "rejected": "model class claims-summary not deployed" },
      { "endpoint": "vllm-eu-customer", "rejected": "health probe failed" }
    ]
  }
}

Capacity is the real failover problem

A failover graph that points at regions with no spare capacity is a diagram, not a plan. Provider quotas are set per region and per model, and during a regional incident every customer of that provider fails over to the same neighbours at the same moment. If you have not reserved headroom there, your secondaries will answer with 429s exactly when you need them.

Do the arithmetic for your own peak. Take a claims team sending 40 requests per second with about 2,000 tokens per request, prompt and completion together. That is 80,000 tokens per second, or 4.8 million tokens per minute. If Dublin and Paris split the failover load evenly, each needs 2.4 million tokens per minute of headroom above its own peak, on the same model version. Where reserved throughput at that level costs too much, a self-hosted runtime in your own data centre is often the cheaper last resort, sized for the subset of traffic that cannot wait.

Model parity matters as much as capacity. A secondary that serves a different model, or a different version of the same one, will produce different outputs, and for systems your inventory classifies as high-risk, the eu-ai-act-high-risk pack refuses models outside the approved list anyway. Keep the same dated model version deployed across the graph, or treat cross-model failover as a degraded mode with its own evaluation results.

Health checks that mean something

A TCP check or a provider status page says little about whether a model endpoint will answer your request in time. We probe each endpoint every ten seconds with a short synthetic completion against the tenant's model class and track error rate and time to first token. A circuit opens after five consecutive failures or when time to first token crosses the tenant's threshold, stays open for thirty seconds, then lets a single probe through to test recovery.

Streaming sets a hard boundary. The gateway can fail over before the first token is sent, because the client has seen nothing yet. Once tokens are flowing, switching endpoints would stitch two models' outputs together, so we end the stream with an error and let the client retry with the same idempotency key. The latency cost of staying in the zone is small: Frankfurt to Dublin or the Netherlands is a round trip in the low tens of milliseconds, against roughly 80 to 100 ms to US East before any model compute.

Prove it with game days and receipts

Failover you have never exercised is a hope. Once a quarter, force the primary circuit open for a test tenant in production and watch where the traffic goes. Then check the evidence rather than the dashboard. Every response carries a signed receipt naming the resolved endpoint, the provider region id and the facility, so you can count receipts by region for the test window and confirm that each one sits inside the graph. The receipts are leaves in the hourly Merkle tree and ytt-verify checks them offline, so an auditor who trusts neither of us can run the same check.

• Every receipt in the window resolves to an endpoint inside the tenant's graph.
• Time to first token during failover stayed under the alerting threshold.
• Egress logs show no provider API calls from outside the gateway.
• With the whole graph forced down, the application degraded as designed and sent nothing anywhere else.

ytt-verify batch --bundle gameday-2026-08-20.tar.gz \
  --pubkey keys/yttsecunet-eu-2026q3.pub --strict --json \
  | jq -r '.receipts[].provider_region' | sort | uniq -c
#   612 eu-west-1
#   388 eu-west-3