On this page

🔗Why We Have to Talk About This

Every .NET team building with Azure OpenAI goes on the same painful journey.

It starts with excitement. You follow the official quickstart, and in about ten minutes, you have a console app calling GPT-4. It feels like magic. You quickly wrap it in an ASP.NET controller, and you've got an AI-powered API.

And then, reality hits.

The first time the Azure OpenAI API has a momentary blip, your application throws an ugly HttpRequestException and the user gets a 500 error. The first time a user closes their browser mid-request, you're left with a zombie process chewing up server resources. The first time you get a surprisingly large bill, you realize you have no idea which user or feature is responsible for all those tokens.

So you start bolting things on. You wrap the API call in a Polly retry policy. You thread a CancellationToken through your code. You add some basic logging. Before you know it, your clean, simple controller has become a tangled mess of production-hardening logic, and you've spent two weeks rewriting the sample code into something you can actually trust.

I've been on this journey, and frankly, it's a waste of time. We should be focusing on building great features, not re-solving the same infrastructure problems over and over. That's why I built this starter kit. It's a production-grade foundation that bakes in all the annoying, non-negotiable parts from day one.

🔗The Design Philosophy: Boring is Beautiful

This starter isn't about fancy algorithms. It's about building a stable, observable, and cost-conscious foundation for your AI features.

  • Resilience First: We use battle-tested libraries like Polly to build smart resilience policies. Simple retries for transient network errors, circuit breakers for more serious provider issues, and proper timeouts so one bad call doesn't bring down your whole service.
  • Observability is Not Optional: You can't manage what you can't measure. The starter is instrumented from top to bottom with OpenTelemetry, giving you structured logs, detailed traces, and crucial metrics out of the box. Debugging is no longer a guessing game.
  • Cost is a Feature: AI is not free. This design treats cost and latency as first-class citizens. It includes hooks to enforce budgets before a call is even made, preventing runaway bills.
  • Provider Agnostic: While it's built for Azure OpenAI, the core interfaces are provider-agnostic. A simple configuration change is all it takes to switch to the official OpenAI API or another provider, without rewriting your business logic.

🔗A Quick Tour of the API

The API surface is simple and intentional, covering the most common use cases.

  • POST /v1/chat: Your standard, request-response chat completion endpoint.
  • POST /v1/chat/stream: A streaming endpoint using Server-Sent Events (SSE) for real-time, token-by-token responses.
  • POST /v1/embeddings: An efficient endpoint for converting text into vector embeddings.

🔗Production-Ready Patterns, Baked In

  • Real Cancellation and Retries: We pass a CancellationToken through the entire stack, from the ASP.NET controller down to the HttpClient call. If a user cancels a request, the work actually stops. Retry policies are smart; we only retry idempotent calls (like GET or PUT with safe-guards) on transient failures. Non-idempotent calls that could have side effects are protected by a circuit breaker instead.

  • Streaming That Doesn't Suck: Streaming responses are critical for a good user experience. This starter uses IAsyncEnumerable<string> on the server, which maps cleanly to a Server-Sent Events stream in the controller. This means your frontend can start rendering the response the moment the first token arrives, and our traces can clearly show you that all-important "time to first token" latency.

  • Cost and Latency Budgets Before the Call: Don't wait for the bill to find out a request was too expensive. You can configure per-request or per-tenant budgets (e.g., "this user can't spend more than $0.05 per request"). The service does a pre-flight check based on the prompt's token count and short-circuits with a useful error before calling the provider if it would exceed the budget.

  • Observability You Can Actually Use: When things go wrong, you need answers. Logs are structured with crucial context: the prompt version, tokens in/out, provider name, model name, total latency, and the calculated cost of the call. OpenTelemetry traces give you a flame graph showing exactly where time was spent—was it retrieving data from your database, rendering the prompt, or waiting on the model?

🔗What You Get on Day One

This isn't a theoretical blog post. This is a link to a repository you can clone, compile, and deploy today.

  • A foundation you can trust, with the resilience and observability patterns you'd otherwise spend weeks building.
  • A suite of tests that not only check for correctness but also act as performance regression tests for latency, cost, and response quality.

In my next posts, I'll show how to wire this production-grade service into the RAG demo we built last month and how to bridge it to other components, like the Rust search indexer. But for now, you can start building on a foundation that's designed for the real world, not a quickstart demo.