On this page

🔗The New Bottleneck We Uncovered

In the last post, we built a lean, mean Rust streaming bridge. It solved a huge problem: it eliminated the need for our application to wait for a full LLM response, dramatically improving the user's perceived latency. We celebrated. The spinners were gone. Life was good.

But then a new, more subtle problem emerged. We noticed that our provider's "time to first token" wasn't consistent. On a good day, it was 300ms. On a bad day, during peak hours, it could be over a second. Sometimes, a request would just hang for a moment before starting. Our p99 latency—what our unluckiest 10% of users experienced—was still unacceptably high.

The bottleneck was no longer our own stack; it was the variability of a single upstream provider. So, we asked a simple, powerful question: What if we didn't have to bet on just one?

🔗The Big Idea: Let Them Race

This is where the fan-out bridge pattern comes in. It’s a beautifully simple, almost brute-force solution to the problem of provider variability.

Instead of sending a request to a single LLM provider, we send the same request to multiple providers simultaneously. We then listen to all of them and stream back the response from whichever one answers first. The rest? We simply hang up on them.

This turns a liability (provider inconsistency) into a strength. We're no longer at the mercy of a single provider's momentary slowness. We're getting the best performance of our entire portfolio of models on every single request.

🔗The Architecture: A Race Coordinator

The architecture evolves slightly but remains conceptually simple:

Client → Rust Bridge (The Race Coordinator) → [Provider A, Provider B, Provider C]

Our Rust bridge is no longer just a simple pipe; it's now a race coordinator. It’s responsible for:

  1. Receiving a single request from the client.
  2. Translating that request into the specific API formats for multiple upstream providers (e.g., Azure OpenAI, Anthropic, a self-hosted HF model).
  3. Spawning concurrent tasks to send all requests simultaneously.
  4. "Listening" for the first byte from any of the providers.
  5. As soon as one "wins," immediately start relaying its stream back to the client.
  6. Crucially, cancelling the requests to all the losing providers to save on cost and resources.

🔗The Core Logic: How to Race Asynchronous Tasks in Rust

This might sound complicated, but Rust's powerful concurrency model, specifically the tokio::select! macro, makes this pattern astonishingly elegant.

tokio::select! allows you to wait on multiple asynchronous operations at once and act on the first one that completes.

Here’s a simplified sketch of how the handler would change:

// ... (imports and structs from previous post) ...

async fn stream_race_handler(State(state): State<AppState>, Json(body): Json<InferenceRequest>) -> impl IntoResponse {
    let (tx, rx) = mpsc::channel::<Result<Event, Infallible>>(64);

    // Spawn the race coordinator task
    tokio::spawn(async move {
        // We'll use another channel to signal the "winner"
        let (winner_tx, mut winner_rx) = mpsc::channel(1);

        // --- Task for Provider A (e.g., Azure OpenAI) ---
        let tx_a = winner_tx.clone();
        tokio::spawn(async move {
            // `send_request_and_get_stream` is a helper we'd write
            if let Ok(stream) = send_request_and_get_stream("azure_openai", &body).await {
                // We won if we can send our stream back
                let _ = tx_a.send(stream).await;
            }
        });

        // --- Task for Provider B (e.g., Anthropic) ---
        let tx_b = winner_tx.clone();
        tokio::spawn(async move {
            if let Ok(stream) = send_request_and_get_stream("anthropic", &body).await {
                let _ = tx_b.send(stream).await;
            }
        });

        // The Race!
        // `select!` waits for the first message on the winner_rx channel.
        // As soon as one of the tasks above sends its stream, this will complete.
        // The other tasks are dropped, and their network requests are cancelled.
        if let Some(mut winning_stream) = winner_rx.recv().await {
            // We have a winner! Now, we just forward its stream.
            while let Some(item) = winning_stream.next().await {
                 if tx.send(Ok(Event::default().data(item))).await.is_err() {
                     break; // Client disconnected
                 }
            }
        }
    });

    // The rest is the same as before...
    let sse_stream = tokio_stream::wrappers::ReceiverStream::new(rx);
    Sse::new(sse_stream).keep_alive(KeepAlive::new().interval(Duration::from_secs(10))).into_response()
}

The beauty of this is Rust's structured concurrency. When the select! block finishes, the other tokio::spawn tasks that didn't win are dropped. When they are dropped, their Future is cancelled, which propagates down and closes the underlying network connection. We don't have to manually clean anything up.

🔗The Hard Parts: This Isn't Free Magic

This pattern is incredibly powerful, but it introduces new, serious considerations.

  1. Cost Control (The #1 Problem): You are intentionally sending redundant requests. If you're not careful, this will double or triple your inference costs. You must implement a cost control strategy. A common one is the "hedged request":

    • Send the request to your primary, preferred provider (e.g., gpt-4o-mini).
    • If you don't receive a response within a short timeout (e.g., 250ms), then send the request to your secondary, more expensive/reliable provider (e.g., gpt-4o). This caps your costs while still protecting you from latency spikes.

  2. API Incompatibility: OpenAI, Anthropic, Cohere, and self-hosted models all have slightly different request payloads and streaming formats. Your bridge must act as an adapter layer, transforming a single incoming request into multiple, provider-specific formats. This adds development and maintenance overhead.

  3. Defining the "Winner": Is the winner the first to send back any byte? The first to send back a valid JSON chunk? The first to send back an error? Your logic needs to be robust enough to handle a provider that fails instantly versus one that just hangs.

🔗Observability: Your Race Tracker

Your observability game has to level up. You now need to track the race itself.

  • Logs: Your structured logs for each request must now include winning_provider, losing_providers, and the time_to_first_token_ms for each provider that was queried.
  • Metrics: Your Prometheus metrics need a new star player: a counter for provider_wins tagged by the provider's name (provider="azure"). This will quickly show you which provider is consistently the fastest. A histogram of the latency delta between the winner and the first loser is also invaluable for tuning your hedging strategy.

🔗How Your .NET/Python App Uses This

This is the most beautiful part: it doesn't change a thing.

The complexity of the race is completely encapsulated within the Rust bridge. Your downstream services still just call a single endpoint and read a standard SSE stream. They get the benefits of lower p99 latency and higher reliability for free, without having to change a single line of their own code.

🔗The New Production Checklist

  • [✓] Multi-Key Management: Do you have a secure way to load and manage API keys for all your providers?
  • [✓] Cost Control: Have you implemented a hedging strategy or another mechanism to prevent runaway costs?
  • [✓] Provider Adapters: Is your code modular enough to easily add or remove a provider without rewriting the core logic?
  • [✓] Win/Loss Dashboard: Do you have a dashboard that clearly shows which providers are winning and what their comparative latencies are?

🔗Beyond Latency: The Path to True Resilience

While we started this journey to crush p99 latency, the fan-out pattern gives us an even greater prize: resilience.

If your primary provider has a partial outage or performance degradation, your system doesn't go down. It doesn't even get slow. It transparently fails over to the next-fastest provider, healing itself in real-time. This is how you move from building a cool AI feature to building a truly robust, production-grade AI system that your users can depend on.