Projects / Performance Engineering
Rust LLM Streaming Bridge
Async Rust service that fans tokens out to browsers and edge workers while keeping first-token latency under 50ms.
Overview
This project was about making LLM streaming feel fast without giving up reliability.
LLM products live or die by perceived latency. If the first token is slow, users feel the system is broken even when the total response time is acceptable. The bridge handled provider streaming, fan-out, retries, and instrumentation in a small Rust service built for low overhead.
Stack: Rust, Tokio, HTTP/2, gRPC, OpenTelemetry
Status: Production-ready
Built for
- AI applications where first-token latency matters.
- Teams routing between multiple LLM providers.
- Browser and edge clients that need clean streaming behavior.
- Systems that need telemetry around provider latency, retries, and failures.
What made it hard
Streaming looks simple until production traffic hits it.
Connections drop. Providers behave differently. Some clients read slowly. Retries can duplicate output if they are not handled carefully. Observability is often thin because teams focus on the model call and forget the transport layer.
The goal was to make streaming boring: fast, observable, and predictable.
What I built
Async Rust service
Built the bridge on Tokio with non-blocking I/O and careful backpressure handling so slow clients would not punish the rest of the system.
Provider abstraction
Added provider adapters for OpenAI, Azure OpenAI, and Anthropic-style streaming APIs so application code could consume a consistent stream shape.
Low-latency delivery
Optimized connection reuse, buffering, and token forwarding to reduce first-token delay and keep streams smooth under load.
Recovery behavior
Implemented retry, timeout, health check, and graceful degradation patterns around provider failures.
Production telemetry
Instrumented latency, stream duration, error rates, provider behavior, and throughput with OpenTelemetry.
Why I built it this way
A lot of AI applications treat streaming as a UI detail. It is not.
Streaming is part of the product experience, and it is also part of the reliability story. If the transport layer is slow, fragile, or opaque, every model call feels worse than it should.
Rust was a good fit because this was infrastructure work: many concurrent streams, low overhead, explicit control over memory and backpressure, and no tolerance for mystery latency.
Impact
- Sub-50ms first-token delivery on the optimized path.
- 10K+ concurrent streams supported by async architecture and backpressure control.
- Provider flexibility without rewriting the application layer.
- Better debugging through metrics and traces around the stream itself.
- A reusable bridge for AI products that need fast user-facing output.
Related content
More projects