On this page

🔗Why We Need to Have This Conversation

Let me tell you a story. A few months ago, we shipped a new RAG feature. It worked great in staging. The answers were crisp, the citations were accurate. A week after launch, a product manager comes to us and says, "The feature just feels... worse now."

We had no idea what they meant. The service was up. The error rate was zero. But when we dug in, we found the horrifying truth: a change in an upstream data source meant our retrieval step was silently pulling back irrelevant documents. The LLM, trying its best to be helpful, was just hallucinating answers based on bad context. The feature was fundamentally broken, but none of our traditional monitoring tools even blinked.

This is the new reality of building with LLMs. They fail silently. They fail weirdly. They fail by becoming slow, expensive, or just plain wrong in ways that a simple health check can't catch.

Observability for AI isn't a nice-to-have; it's our only defense against this quiet chaos. It's how we turn vague feelings of "it feels wrong" into a concrete, actionable signal that tells us exactly what broke, when it broke, and why.

🔗What You Absolutely Must Instrument

You need to make the invisible visible. This means instrumenting every critical step of the AI request lifecycle.

  • The Request Itself: For every single call, you need to know: Which provider and model did we use? Which prompt template and version was executed? How big were the inputs and outputs (in tokens)? What was the latency, and most importantly, what did it cost?
  • The Tools: If your LLM calls tools, you must track it. Log the tool's name, the arguments it was called with, how long it took to run, and whether it succeeded or failed. A flaky external API can be the root cause of your entire system's poor performance.
  • The RAG Pipeline: For RAG, you need to go deeper. How long did the retrieval step take? How many documents (k) did we retrieve? Did we re-rank them? Which sources did the documents come from? This is how you spot the problem I described earlier.
  • The Quality: This is the most overlooked piece. You need a way to track quality over time. This can be explicit user feedback (a thumbs up/down button), heuristic scores (Does the answer contain citations?), or labels from an offline evaluation process.

🔗The Three Pillars of AI Observability

Think of this as your detective's toolkit for debugging your AI.

🔗Pillar 1: Structured Logs - Your Ground Truth

Your first step is to log everything, but not as messy, unreadable strings. Log structured JSON for every request. This makes your logs machine-readable and searchable. Each log entry should include a correlation ID that ties it back to the parent request, along with the tenant or user ID. Crucially, log the prompt_version and the key template variables. This lets you see if a recent prompt change is the cause of a problem, without necessarily logging sensitive user data.

🔗Pillar 2: Traces - The Story of a Request

If logs are the detailed notes, traces are the timeline of the crime. Using a standard like OpenTelemetry, you can create a single trace for each incoming user request. Within that trace, you create "spans" for each major step:

  • assemble_prompt: How long did it take to retrieve data and render the prompt?
  • retrieve_documents: The RAG retrieval step.
  • llm_call: The actual call to the provider. This span should be rich with attributes: the model name, temperature, tokens in/out, and whether it was a cache hit.
  • tool_call: A separate span for each tool the LLM decides to use.

When a user complains that a request was slow, you can pull up its trace and see a beautiful flame graph that shows you exactly where the time went. No more guesswork.

🔗Pillar 3: Metrics - The Vital Signs of Your System

Metrics are your high-level dashboard, your system's pulse and blood pressure. You should use a tool like Prometheus to track three types of metrics:

  • Counters (the odometer): Total requests, errors (tagged by cause), and cache hits. These numbers only go up.
  • Histograms (the speed distribution): This is for things that have a duration, like latency. Don't just track the average; track the p50, p90, and p99. Your p99 latency tells you the experience of your unluckiest user. You should have histograms for end-to-end latency, first-token latency, and retrieval latency.
  • Gauges (the speedometer): A point-in-time value. Things like the current request queue depth, the number of concurrent requests, and your real-time token rate.

🔗From Data to Insight: The Dashboard

Don't just collect data; visualize it. Your dashboards should tell a story.

  • The "Golden Paths" Dashboard: This is your high-level operations view. It should show p50/p90 latency, error rate, cost per day, and cache hit rate. This is what you look at to see if the system is healthy right now.
  • The Quality Dashboard: This tracks your quality metrics over time. Is the "groundedness" score from your offline eval trending down? Is the user "thumbs down" rate trending up? This is how you catch silent failures.
  • The RAG Dashboard: This is a deep dive into your retrieval pipeline. It should show retrieval latency, the distribution of document sources being pulled back, and the average k.

🔗From Insight to Action: Budgets and Alerts

Observability without alerting is just a history lesson. You need to turn your insights into proactive signals.

  • Set Budgets: Configure per-tenant and per-feature dollar caps. If a single user is on track to burn through their monthly budget in the first two days, send an alert. Cost is a critical operational metric.
  • Set Regression Alerts: After every deployment, your CI/CD pipeline should watch the dashboards. If p90 latency spikes by 20% or your quality score dips by 10%, automatically flag the deploy as problematic.

🔗A Practical Sketch for .NET Teams

This isn't as hard as it sounds.

  • Use the OpenTelemetry .NET SDK to automatically instrument ASP.NET and HttpClient. This gives you basic traces and metrics for free.
  • Create a simple wrapper class around the OpenAIClient. This wrapper is responsible for starting and stopping spans, logging the structured JSON, and recording the Prometheus metrics.
  • Add a small feedback API (POST /v1/feedback) that lets users submit feedback, which you then store with the TraceId of the original request. This allows you to drill down from a "thumbs down" directly to the trace of the request that failed.

🔗What Really Matters

At the end of the day, it comes down to being able to answer these questions quickly:

  • How fast is our time-to-first-token, and what's our throughput under real-world load?
  • If quality suddenly drops, can we pinpoint the cause within minutes—was it a prompt change, a retrieval problem, or a flaky tool?
  • Are we staying within our cost and latency budgets?

Observability is the bedrock of professional AI engineering. It’s what gives you the confidence to deploy quickly, the clarity to debug effectively, and the insight to build a product that users can actually trust.