On this page

🔗The Problem We've All Faced

Your beautiful UI has a chat window. The user asks a question. A spinner appears. And then... silence.

For three, four, maybe five agonizing seconds, the user stares at that spinner, wondering if your app has crashed. Then, finally, a wall of text appears all at once. From a technical standpoint, it worked. From a user experience standpoint, it was a failure.

Waiting for a full LLM response before showing anything is performance malpractice. Users perceive latency from the moment they hit "Enter" to the moment they see the first word. Getting that first word on screen in under a second is the difference between an application that feels alive and one that feels broken.

This is where a tiny, hyper-efficient Rust bridge comes in. Instead of baking complex, finicky streaming logic into your main .NET or Python application, you can offload the entire problem to a small, specialized service that does one thing incredibly well: it pulls tokens from any LLM provider and pushes them downstream with almost zero overhead. It’s a streaming superpower for your existing stack.

🔗What This Post Covers

We're going to build a minimal Rust streaming proxy using the incredible Axum (web framework) and Reqwest (HTTP client) libraries. We'll cover:

  • How to correctly handle Server-Sent Events (SSE) and the magic headers that make streaming work reliably.
  • How to implement bulletproof backpressure so a slow client can't crash your service with a memory overflow.
  • Why this pattern is a game-changer for latency and user experience.
  • How your existing .NET or Python services can talk to this bridge effortlessly.

🔗The Architecture: A Simple, Powerful Pattern

The data flow is beautifully simple:

🔗Client (Browser/Service) → Rust Bridge (The Smart Pipe) → LLM Provider (OpenAI, Azure, etc.)

Our Rust bridge is just a smart pipe. It:

  1. Accepts a standard POST request from your application.
  2. Immediately opens a streaming connection to the upstream LLM provider.
  3. As soon as the first chunk of data arrives from the provider, it relays it to your client.
  4. It gracefully handles timeouts, retries, and backpressure.
  5. It emits clean, structured logs and metrics so you can see exactly what's happening.

🔗The Code: Less Scary Than You Think

Rust has a reputation for being complex, but for this kind of I/O-bound work, it's incredibly ergonomic. This is a complete, working implementation.

// A minimal, production-ready streaming bridge in Rust
use axum::{extract::State, routing::post, response::IntoResponse, Router, Json};
use axum::response::sse::{Event, KeepAlive, Sse};
use futures::{Stream, StreamExt, TryStreamExt};
use reqwest::Client;
use serde::{Deserialize, Serialize};
use std::{convert::Infallible, time::Duration};
use tokio::sync::mpsc;

#[derive(Clone)]
struct AppState {
    http_client: Client,
    upstream_api_key: String,
}

#[derive(Deserialize)]
struct InferenceRequest {
    prompt: String,
    #[serde(default)] model: String,
}

// Structs for the OpenAI-compatible payload
#[derive(Serialize)] struct UpstreamPayload<'a> { model: &'a str, stream: bool, messages: Vec<Message<'a>> }
#[derive(Serialize)] struct Message<'a> { role: &'a str, content: &'a str }

#[tokio::main]
async fn main() {
    let state = AppState {
        http_client: Client::new(),
        upstream_api_key: std::env::var("UPSTREAM_API_KEY").expect("UPSTREAM_API_KEY must be set"),
    };
    let app = Router::new().route("/v1/chat/stream", post(stream_chat_handler)).with_state(state);
    let listener = tokio::net::TcpListener::bind(("0.0.0.0", 8787)).await.unwrap();
    println!("🚀 Listening on http://0.0.0.0:8787");
    axum::serve(listener, app).await.unwrap();
}

async fn stream_chat_handler(State(state): State<AppState>, Json(body): Json<InferenceRequest>) -> impl IntoResponse {
    // A bounded channel is our backpressure mechanism. If the client is slow,
    // this channel will fill up, and the `send` call will block,
    // naturally pausing our reading from the upstream provider.
    let (tx, rx) = mpsc::channel::<Result<Event, Infallible>>(64);

    // This is the magic! Spawn a background task to handle the long-lived connection
    // to the provider. This lets our HTTP handler return immediately with the stream.
    tokio::spawn(async move {
        let payload = UpstreamPayload {
            model: if body.model.is_empty() { "gpt-4o-mini" } else { &body.model },
            stream: true,
            messages: vec![Message { role: "user", content: &body.prompt }],
        };

        let req = state.http_client
            .post("https://api.openai.com/v1/chat/completions")
            .bearer_auth(&state.upstream_api_key)
            .timeout(Duration::from_secs(30))
            .json(&payload);

        // Try to establish the connection
        match req.send().await {
            Ok(resp) => {
                let mut stream = resp.bytes_stream();
                // Read from the provider's stream chunk-by-chunk
                while let Some(item) = stream.next().await {
                    match item {
                        Ok(bytes) => {
                            // The provider's stream is often already in SSE format.
                            // We normalize it and forward each line.
                            for line in String::from_utf8_lossy(&bytes).lines() {
                                if !line.starts_with("data:") { continue; }
                                // `tx.send().await` provides the backpressure.
                                if tx.send(Ok(Event::default().data(line))).await.is_err() {
                                    // Client disconnected, so we stop.
                                    return;
                                }
                            }
                        }
                        Err(e) => {
                            let _ = tx.send(Ok(Event::default().event("error").data(format!("upstream_chunk_error: {e}")))).await;
                            return;
                        }
                    }
                }
            }
            Err(e) => {
                let _ = tx.send(Ok(Event::default().event("error").data(format!("request_error: {e}")))).await;
            }
        }
    });

    // Convert our receiver channel into a stream that Axum can serve as SSE
    let sse_stream = tokio_stream::wrappers::ReceiverStream::new(rx);
    Sse::new(sse_stream).keep_alive(KeepAlive::new().interval(Duration::from_secs(10))).into_response()
}

🔗The "Gotchas" That This Code Handles

  • Real Backpressure: The mpsc::channel is bounded (we gave it a size of 64). If the downstream client is slow to read, our tx.send().await will naturally pause, which stops us from reading from the upstream reqwest stream. This prevents our bridge from consuming gigabytes of memory trying to buffer for a slow client. It just works.
  • Correct Headers: For SSE to work through proxies like Nginx, you need the right headers. Axum's Sse response type handles the Content-Type: text/event-stream and Cache-Control: no-cache for you. You should also add X-Accel-Buffering: no in your proxy config.
  • Optional Token-Aware Chunking: For most apps, relaying the provider's chunks is fine. But if you really need perfect "typing" animations, you could use a library like tiktoken-rs to decode the chunks into tokens and then re-encode them one by one. This adds CPU overhead, so only do it if your UX absolutely demands it.

🔗Observability: Seeing the Stream

Log every request. At a minimum, capture: the start time, the upstream model, bytes in/out, time to first chunk, total time, and the number of chunks. Use Prometheus to export a histogram of your first-token latency and token-per-second rate. This is how you prove the bridge is fast.

🔗Benchmarking: Prove It's Fast

Don't take my word for it. You can hammer this service locally with a tool like bombardier.

# First, run your Rust bridge on port 8787
# Then, hit it with 50 concurrent connections
./bombardier -c 50 -n 2000 -m POST -H 'content-type: application/json' \
  -b '{"prompt":"Tell me a short story about a robot who learns to code.","model":"gpt-4o-mini"}' \
  http://127.0.0.1:8787/v1/chat/stream

Watch the latency numbers. They'll be low.

🔗Interop: How Your .NET/Python App Uses This

This is the best part. Your main application doesn't need to know anything about Rust. It just makes a simple HTTP call.

  • In .NET: Use HttpClient with HttpCompletionOption.ResponseHeadersRead. This lets you start reading the response stream as soon as the headers arrive. You can then use a StreamReader to read the SSE stream line-by-line.
  • In Python: Use httpx in async mode with stream=True. You can then use async for line in response.aiter_lines() to process the stream as it arrives.

🔗Your Production Checklist

  • Security: Add authentication (a simple bearer token is fine). Limit the request body size.
  • Resilience: Add a circuit breaker (tower-governor is great) for upstream failures and implement a smart retry policy.
  • Rate Limiting: Add per-IP and per-token rate limits to prevent abuse.
  • Deployment: Run it behind a proper TLS termination layer like Cloudflare or an ALB.
  • Observability: Implement structured logging with correlation IDs and export your Prometheus metrics.

🔗What's Next?

This pattern is a powerful tool in your belt. A reference repository with this code is coming soon, which I'll wire up to the other "Production AI in .NET" projects. Next, I'll explore an even more advanced version that can fan out requests to multiple LLM providers and stream back the response from whichever is fastest.

If you have a use case you'd like to see, ping me. Let's make our apps feel alive again.