What is an Inference Delivery Network?
An inference delivery network (IDN) routes AI inference requests across pooled GPU capacity the way a CDN routes content: one endpoint in front, many providers behind it, and every request served by the best available backend. OpenRelay is an IDN.
- One OpenAI-compatible endpoint
- Automatic backend routing
- Failover built in
- Per-token billing
Same outage, two setups
A provider goes down. Calling it directly, your requests fail until it comes back. Through an IDN, they move to healthy capacity and keep succeeding.
The CDN, applied to inference
CDNs solved content delivery by putting one entry point in front of distributed, interchangeable capacity. An IDN applies the same structure to a harder resource: GPU compute.
| CDN | IDN | Why it matters |
|---|---|---|
| Origin server | The model | The thing being delivered: weights, not files |
| Points of presence | GPU capacity across providers | Pooled supply instead of one vendor's region |
| Request routing | Model-to-backend routing | Every request goes to the best available backend |
| Cache hit | Warm model | A loaded model answers now; a cold one makes you wait |
| Failover to the next PoP | Failover to the next provider | A provider running dry is routine, not an incident |
| Egress billing | Token billing | Pay for what was delivered, not for standing capacity |
The term comes from a 2021 Inria research paper, Towards Inference Delivery Networks, which defined IDNs as networks of computing nodes coordinating to serve ML inference requests. OpenRelay is the production implementation.
AI development multiplied machine demand. Human attention stayed flat.
Machine demand, human-scale infrastructure
Coding agents burn tokens around the clock. Five engineers running agents produce the load of fifty, on capacity sized for five.
At-desk management
When a provider throttles, someone has to notice and re-route. The network should do that, or your headcount does.
Single-vendor exposure
One provider means their pricing, their rate limits, and their 3am outage. An IDN routes around a constrained backend and keeps serving.
Draw from the network, or add to it
The pool that serves inference also rents GPU VMs. Every GPU plugged in adds capacity for both.
Frequently asked questions
What is an inference delivery network (IDN)?
An inference delivery network is a network of GPU capacity, pooled across providers and locations, that routes each AI inference request to the best available backend. It applies the CDN model to inference: one endpoint in front, distributed capacity behind it, routing and failover handled by the network instead of by the application.
How is an IDN different from a CDN?
A CDN moves static and cached content close to users; the hard problems are bandwidth and cache placement. An IDN delivers computation: the scarce resource is GPU capacity, and the routing decision weighs model availability, load, and cost rather than geographic distance alone. The operating model is the same: one entry point, many interchangeable backends, automatic failover.
How is an IDN different from a single inference API provider?
A single provider serves models from their own capacity, so their capacity limits are your capacity limits. An IDN sits above providers and treats them as interchangeable supply. When one backend is saturated or down, requests route to another, without the application changing anything.
What is a token delivery network?
A narrower version of the same idea. A token delivery network serves LLM tokens. An inference delivery network covers every inference workload, LLMs included, which is how the 2021 Inria research below defined it.
Where does the term come from?
Researchers at Inria introduced inference delivery networks in a 2021 paper (arXiv:2105.02510, later published in IEEE/ACM Transactions on Networking), defining them as networks of computing nodes that coordinate to serve machine-learning inference requests. OpenRelay is the commercial implementation of that idea for production AI workloads.
Do I need to change my code to use one?
No. OpenRelay's endpoint is OpenAI-compatible: point your existing SDK at the OpenRelay base URL with an OpenRelay API key and requests route across the network. Billing is per token, with no minimum spend.
Point your SDK at the network
One base URL swap and your requests route across pooled GPU capacity, with failover handled for you and billing per token.