Frankfurt to Virginia: the latency floor no CDN can buy back

Start with physics

Light in a vacuum covers about 300,000 km per second. Standard single-mode fibre has a refractive index of roughly 1.47, which slows light to about 200,000 km per second, or 200 km per millisecond. Every latency argument about cross-border inference starts from that number, because nothing in software can beat it.

The great-circle distance from Frankfurt to Ashburn, Virginia, where much of US East cloud capacity sits, is about 6,500 km. At 200 km per millisecond that is 32.5 ms one way and 65 ms for a round trip, assuming a perfectly straight fibre with zero switching. That is the floor. You will never see it.

Why the real number is 80 to 100 ms

Submarine cables land where landing stations and permits allow, not on the great circle. A packet from Frankfurt typically crosses to a coastal point in Western Europe, travels the cable to a landing on the US east coast, then runs terrestrial fibre inland. Each segment adds distance, and each router adds serialisation and queueing delay. Public inter-region measurements published by the large clouds and independent monitors consistently put Frankfurt to US East in the 80 to 100 ms range.

Compare that with Frankfurt to Amsterdam or Dublin, where round trips in the low tens of milliseconds are normal, and Frankfurt to a provider's Frankfurt region, where you are often in single digits. Routing an EU user's request to Virginia is not a small detour. It adds a fixed tax measured in whole multiples of the in-region cost.

The number also moves. Cable faults and maintenance reroute traffic onto longer paths, and BGP decisions made by carriers you have never heard of can shift a flow from one cable system to another overnight. An in-region path has fewer of these failure modes simply because it has fewer kilometres and fewer networks in it. Variance matters as much as the median: a p99 that jumps by 40 ms during a cable repair is the kind of regression that shows up in user complaints before it shows up on a dashboard.

Where the round trip lands in an LLM request

A single round trip of 90 ms sounds tolerable against a model that takes a second to answer. The issue is how many round trips you pay. A cold HTTPS request needs a TCP handshake and, with TLS 1.3, one more round trip for the TLS handshake before the first byte of the request is sent. That is two transatlantic round trips, roughly 180 ms, before the model sees anything. TLS 1.3 0-RTT resumption can remove one, but early data is replayable, which is why many API gateways do not accept it for POST requests.

Streaming changes the picture for output. Once the response stream is open, tokens arrive as the server emits them, so the round trip is paid roughly once for time to first token, not once per token. Users perceive time to first token most, and that is exactly where the transatlantic tax concentrates.

• Cold connection: TCP plus TLS 1.3 handshake, two round trips before the request.
• Warm pooled connection: one round trip for request and first response bytes.
• Agent loop with six sequential model or tool calls: six round trips minimum, around half a second of pure transit at 85 ms each.

Agents make it worse

The single-shot chat completion is no longer the typical workload. Agent frameworks call a model, parse a tool call, execute the tool, call the model again with the result, and repeat. Each step is sequential, because the next prompt depends on the previous answer. If your orchestrator runs in Frankfurt and the model runs in Virginia, every step crosses the Atlantic twice.

I have seen teams accept a US endpoint for a prototype because the demo felt fine, then discover that a six-step workflow carries half a second of avoidable network time in production. Residency and latency point the same way here: keep the model next to the orchestrator, and keep both next to the data subject.

What a gateway adds, and how to check it

Any proxy in the path costs something. Ours does policy evaluation, PII detection and receipt minting, and our production telemetry puts the added latency under 8 ms at p99. That is a YTT Secunet figure and you should not take it on faith. Measure time to first token directly against the provider's EU endpoint, then through the gateway, from the same machine, with connection reuse enabled, over a few hundred requests. Compare distributions, not single runs.

Two details make the comparison honest. First, reuse the client object across iterations, as the script below does, so the SDK keeps its HTTP connection pool warm; otherwise you are measuring handshakes, not the gateway. Second, keep the prompt and max_tokens tiny so model compute stays roughly constant and the difference you see is network plus proxy. Run it from the region your users are in, not from your laptop on a hotel network.

import time, statistics
from openai import OpenAI

client = OpenAI(
    base_url="https://eu.yttsecunet.com/v1",
    api_key="jr_live_...",
    default_headers={
        "X-YTT-Policy": "eu-gdpr-strict",
        "X-YTT-Tenant": "acme-gmbh",
    },
)

ttft = []
for _ in range(300):
    start = time.perf_counter()
    stream = client.chat.completions.create(
        model="mistral-small-latest",
        messages=[{"role": "user", "content": "Reply with one word."}],
        stream=True,
        max_tokens=4,
    )
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            ttft.append((time.perf_counter() - start) * 1000)
            break
    stream.close()

q = statistics.quantiles(ttft, n=100)
print(f"p50 {q[49]:.1f} ms  p99 {q[98]:.1f} ms")

The design conclusion

Physics gives you a simple rule: the cheapest millisecond is the one you never send across an ocean. In-jurisdiction routing is usually framed as a compliance cost. For European users it is also the fastest path, and the same is true for Gulf users served from the Gulf and Singapore users served from Singapore. When someone proposes a cross-border fallback for performance reasons, ask them to show the round-trip arithmetic first. It rarely survives the calculation.

The exception is capacity. When the only model that meets a quality bar is offered in one US region, the latency argument loses to the capability argument, and the residency argument has to be settled separately on its legal merits. Most of the time, though, an in-region model of comparable quality exists, and the work is in wiring routing and failover so it is used every time, not only when someone remembers to pick the right base URL.