Case studies / Software as a Service

Multi-Provider LLM Routing: Cost and Latency Optimization

Architecture notes on multi-provider LLM routing, caching, budget controls, and observability. Published performance figures are not supported by benchmark records in this repository.

Client
SaaS Platform
Industry
Software as a Service
Role
Principal Software Engineer

Evidence status

This write-up describes a routing architecture. The previously published latency, cost, cache-hit and availability figures could not be traced to benchmark results or a defined workload in the site repository. They have been removed from the results copy and chart. No measured performance improvement is claimed here. The diagrams and simplified code explain design choices; they are not benchmark evidence.

The Challenge

A growing SaaS platform needed to integrate multiple LLM providers while managing unpredictable costs and maintaining low latency for end users.

Business Problems

  • Unpredictable costs: LLM spending varied wildly month-to-month
  • Inconsistent performance: Some providers had high latency for certain queries
  • Single provider risk: Dependency on one provider created reliability concerns
  • No visibility: Lacked metrics to understand cost drivers
  • Manual optimization: Engineers manually switched providers based on gut feel

Technical Challenges

  • Multiple API formats: OpenAI, Azure OpenAI, and Anthropic have different interfaces
  • Rate limiting: Managing rate limits across providers
  • Cost tracking: No unified view of spending across providers
  • Latency variance: Different providers can have different response-time distributions for the same workload
  • Fallback handling: No automatic failover when providers failed

Impact on Business

  • Budget overruns: Uncontrolled usage can exceed a configured spending budget
  • User complaints: Slow response times in certain scenarios
  • Engineering overhead: Manual provider switching consumed valuable dev time
  • Competitive risk: Slower than competitors using optimized solutions

The Solution

The architecture combines a Rust routing layer, provider selection, budget controls, response caching, and observability. These controls address different tradeoffs and need workload-specific evaluation.

System Architecture

graph TB
    subgraph "Client Applications"
        WEB[Web App]
        API[External APIs]
        WORKER[Background Workers]
    end

    subgraph "LLM Router Service"
        GATEWAY[API Gateway<br/>Authentication]
        ROUTER[Intelligent Router<br/>Provider Selection]
        CACHE[Response Cache<br/>Redis]
        BUDGET[Budget Enforcer<br/>Cost Controls]
    end

    subgraph "Provider Adapters"
        OPENAI_ADAPTER[OpenAI Adapter]
        AZURE_ADAPTER[Azure OpenAI Adapter]
        ANTHROPIC_ADAPTER[Anthropic Adapter]
    end

    subgraph "LLM Providers"
        OPENAI[OpenAI GPT-4/3.5]
        AZURE[Azure OpenAI]
        ANTHROPIC[Anthropic Claude]
    end

    subgraph "Observability Stack"
        PROMETHEUS[Prometheus<br/>Metrics]
        GRAFANA[Grafana<br/>Dashboards]
        JAEGER[Jaeger<br/>Traces]
        LOKI[Loki<br/>Logs]
    end

    WEB --> GATEWAY
    API --> GATEWAY
    WORKER --> GATEWAY

    GATEWAY --> CACHE
    CACHE -->|Miss| ROUTER
    CACHE -->|Hit| GATEWAY

    ROUTER --> BUDGET
    BUDGET --> ROUTER

    ROUTER --> OPENAI_ADAPTER
    ROUTER --> AZURE_ADAPTER
    ROUTER --> ANTHROPIC_ADAPTER

    OPENAI_ADAPTER --> OPENAI
    AZURE_ADAPTER --> AZURE
    ANTHROPIC_ADAPTER --> ANTHROPIC

    ROUTER -.-> PROMETHEUS
    BUDGET -.-> PROMETHEUS
    ROUTER -.-> JAEGER
    ROUTER -.-> LOKI

    PROMETHEUS --> GRAFANA

    style ROUTER fill:#fff3cd
    style CACHE fill:#d1ecf1
    style BUDGET fill:#f8d7da
    style PROMETHEUS fill:#d4edda

Routing Decision Flow

graph TD
    START[Incoming Request] --> AUTH{Authenticated?}
    AUTH -->|No| REJECT[401 Unauthorized]
    AUTH -->|Yes| CACHE{In Cache?}

    CACHE -->|Yes| RETURN_CACHE[Return Cached Response]
    CACHE -->|No| BUDGET{Budget OK?}

    BUDGET -->|No| REJECT_BUDGET[429 Over Budget]
    BUDGET -->|Yes| ANALYZE[Analyze Request]

    ANALYZE --> FACTORS{Selection Factors}

    FACTORS --> LATENCY[Required Latency]
    FACTORS --> COST[Cost Constraints]
    FACTORS --> MODEL[Model Requirements]
    FACTORS --> AVAILABILITY[Provider Health]

    LATENCY --> SCORE[Calculate Provider Scores]
    COST --> SCORE
    MODEL --> SCORE
    AVAILABILITY --> SCORE

    SCORE --> SELECT[Select Best Provider]

    SELECT --> TRY_PRIMARY[Try Primary Provider]

    TRY_PRIMARY --> PRIMARY_SUCCESS{Success?}

    PRIMARY_SUCCESS -->|Yes| LOG_METRICS[Log Metrics]
    PRIMARY_SUCCESS -->|No| TRY_FALLBACK[Try Fallback Provider]

    TRY_FALLBACK --> FALLBACK_SUCCESS{Success?}

    FALLBACK_SUCCESS -->|Yes| LOG_METRICS
    FALLBACK_SUCCESS -->|No| ERROR[Return Error]

    LOG_METRICS --> CACHE_RESPONSE[Cache Response]
    CACHE_RESPONSE --> RETURN[Return to Client]

    style BUDGET fill:#fff3cd
    style LOG_METRICS fill:#d4edda
    style CACHE_RESPONSE fill:#d1ecf1
    style ERROR fill:#f8d7da

Cost Tracking & Budget Enforcement

sequenceDiagram
    participant Client
    participant Router
    participant BudgetTracker
    participant Redis
    participant Provider

    Client->>Router: LLM Request
    Router->>BudgetTracker: Check Budget
    BudgetTracker->>Redis: Get Current Spend and Budget
    Redis-->>BudgetTracker: Current spend / configured budget

    alt Budget Available
        BudgetTracker-->>Router: ✅ Proceed
        Router->>Provider: Execute Request
        Provider-->>Router: Response + usage
        Router->>BudgetTracker: Record Cost
        BudgetTracker->>Redis: Update accumulated spend
        Router-->>Client: Response
    else Budget Exceeded
        BudgetTracker-->>Router: ❌ Over Budget
        Router-->>Client: 429 Too Many Requests
    end

    Note over Redis: Real-time cost tracking
    Note over BudgetTracker: Per-user & org limits

Technology Stack

  • Core Language: Rust (performance + safety)
  • API Framework: Axum (async web framework)
  • Caching: Redis (response caching + rate limiting)
  • Observability: Prometheus, OpenTelemetry, Jaeger, Grafana
  • Deployment: Docker + Kubernetes
  • CI/CD: GitHub Actions
  • Providers: OpenAI, Azure OpenAI, Anthropic

Implementation Highlights

Intelligent Provider Selection

The router scores each provider based on multiple factors:

Scoring Algorithm (simplified):

struct ProviderScore {
    latency_score: f64,    // 0-100, higher is faster
    cost_score: f64,       // 0-100, higher is cheaper
    availability_score: f64, // 0-100, based on recent health
    model_compatibility: f64, // 0-100, supports required features
}

impl Router {
    fn select_provider(&self, request: &LLMRequest) -> Provider {
        let scores: Vec<(Provider, f64)> = self.providers
            .iter()
            .map(|p| {
                let latency = self.calculate_latency_score(p, request);
                let cost = self.calculate_cost_score(p, request);
                let availability = self.get_availability_score(p);
                let compatibility = self.check_compatibility(p, request);

                // Weighted score based on request priority
                let total_score = match request.priority {
                    Priority::Cost => cost * 0.6 + latency * 0.3 + availability * 0.1,
                    Priority::Latency => latency * 0.6 + cost * 0.2 + availability * 0.2,
                    Priority::Balanced => latency * 0.4 + cost * 0.4 + availability * 0.2,
                };

                (p.clone(), total_score * compatibility)
            })
            .collect();

        // Select highest scoring provider
        scores.iter().max_by(|a, b| a.1.partial_cmp(&b.1).unwrap()).unwrap().0
    }
}

Caching Strategy

Three-tier caching:

  1. Exact match: Hash of prompt + parameters → cached response
  2. Semantic similarity: Embed prompt, find similar cached responses
  3. Prefix cache: Common prompt prefixes (system messages)

Caching tradeoffs:

  • A valid full-response cache hit can avoid a provider call; a miss still needs provider execution.
  • Cache keys, freshness, tenant isolation, and answer quality determine whether reuse is appropriate.
  • Measure exact-match and semantic-cache paths separately rather than assuming a shared latency or hit rate.

Observability

Metrics to collect:

  • Request rate: Per provider, per user, per endpoint
  • Latency percentiles: p50, p90, p95, p99
  • Cost per request: Real-time and aggregated
  • Error rates: By provider and error type
  • Cache hit rate: Overall and by cache type
  • Token usage: Input/output tokens per provider
  • Budget consumption: Per user/org, with alerts

Illustrative dashboard layout (not recorded telemetry):

graph LR
    subgraph "Cost Dashboard"
        TOTAL[Spend vs. Budget]
        BY_PROVIDER[Spend by Provider<br/>Pie Chart]
        BY_USER[Top Users<br/>Bar Chart]
        FORECAST[Budget Forecast<br/>Line Chart]
    end

    subgraph "Performance Dashboard"
        LATENCY[Latency Percentiles]
        THROUGHPUT[Request Rate]
        ERRORS[Errors by Provider]
        CACHE[Cache Hits and Misses]
    end

    style TOTAL fill:#d1ecf1
    style LATENCY fill:#d4edda

Architecture tradeoffs and validation

  • Routing lets the application select compatible providers using explicit cost, latency and health signals. Provider choice should be evaluated against the same quality requirements.
  • Caching can reuse suitable responses, but freshness, tenant boundaries and semantic similarity need validation. Cache hits and misses should be reported separately.
  • Budget controls make spending constraints explicit. They do not establish a percentage saving without billing and usage evidence.
  • Fallbacks reduce reliance on one provider, but shared dependencies, retry budgets and failure behavior still need testing. They do not establish an uptime rate.
  • Observability should separate time to first token, time to completed response, provider execution, and router/cache overhead. Those are different measurements.

Evidence needed for performance results

A defensible comparison needs the workload and request count, prompt/input and output sizes, provider/model versions, concurrency, environment, measurement boundaries, cache-hit/miss definitions, test window, and underlying observations or histogram data. Cost comparisons also need token usage and applicable prices or billing records; availability needs a defined observation window and failure criteria.

No such benchmark record is linked to the old figures in this repository. The missing definitions cannot be recovered by assigning the old numbers to new test scenarios. See Benchmarks that matter for the measurement approach.

Technical Deep Dive

Provider Adapter Pattern

Each provider adapter implements a common trait:

#[async_trait]
pub trait LLMProvider: Send + Sync {
    async fn complete(&self, request: &CompletionRequest) -> Result<CompletionResponse>;
    async fn stream(&self, request: &CompletionRequest) -> Result<CompletionStream>;
    fn estimate_cost(&self, request: &CompletionRequest) -> f64;
    fn max_tokens(&self) -> usize;
    fn supports_model(&self, model: &str) -> bool;
}

This abstraction allows:

  • Easy provider swapping: Switch providers without code changes
  • Testing: Mock providers for unit tests
  • New providers: Add new providers by implementing trait

Retry & Fallback Logic

async fn execute_with_fallback(&self, request: LLMRequest) -> Result<LLMResponse> {
    let providers = self.select_providers(&request); // Primary + fallbacks

    for (attempt, provider) in providers.iter().enumerate() {
        match self.try_provider(provider, &request).await {
            Ok(response) => {
                self.record_success(provider, attempt);
                return Ok(response);
            }
            Err(e) if e.is_retryable() => {
                self.record_failure(provider, &e);
                continue; // Try next provider
            }
            Err(e) => return Err(e), // Fatal error, don't retry
        }
    }

    Err(Error::AllProvidersFailed)
}

Lessons Learned

What Worked Well:

  • Rust provides explicit control over concurrency and memory
  • Observability makes routing decisions inspectable
  • A provider abstraction keeps API differences behind adapters
  • Caching introduces a reuse path with freshness and quality tradeoffs

Challenges Overcome:

  • Token counting: Different providers count tokens differently
  • Streaming responses: Complex to implement consistently across providers
  • Error handling: Each provider has unique error formats
  • Testing: Required careful mocking of HTTP clients

Would Do Differently:

  • Evaluate semantic caching alongside exact-match caching
  • Implement user-level cost controls from day one
  • Add A/B testing framework for routing algorithms

Client Testimonial

This testimonial is reproduced as previously published. Its numerical claims have not been verified against benchmark records and are not presented here as measured results.

"Raymond helped us implement a multi-provider LLM routing system with intelligent cost controls. His expertise in observability and cost optimization reduced our LLM spending by 30% while improving latency by 60%. The system is production-ready and has been running reliably for months."

— CTO, SaaS Platform