RAG on Azure with pgvector

Overview

This project turned a promising RAG prototype into a system the team could measure, debug, and trust.

The early version could retrieve documents and generate answers, but the team needed more than “it seems right.” They needed citations, freshness controls, better ranking, and dashboards that made answer quality visible over time.

Industry: Enterprise software
Duration: 4 months
Role: Principal Software Engineer
Stack: Azure, PostgreSQL, pgvector, FastAPI, Sentence Transformers, Prometheus, Grafana

Built for

• Teams that need AI answers with source citations.
• Enterprise users who will not trust a black-box answer without traceability.
• Product teams that need to measure retrieval quality before expanding usage.
• Engineering teams that need observability around latency, cost, and answer quality.

What made it hard

RAG quality is rarely one problem. It is usually a chain of small decisions that compound.

Bad chunking loses context. Weak ranking returns almost-right documents. Stale content creates confident but outdated answers. Missing evaluation turns every change into guesswork.

The hard part was building a system where the team could see those failures instead of arguing from anecdotes.

What I built

Retrieval pipeline

Implemented semantic retrieval with PostgreSQL and pgvector, including indexing, metadata filters, and query patterns designed for production latency.

Re-ranking and relevance tuning

Added a multi-stage ranking path so the system could retrieve broadly, then choose more carefully before generation.

Citation-ready answers

Designed the answer path around traceability. Generated responses carried source references so users and reviewers could inspect where the answer came from.

Offline evaluation

Built evaluation harnesses with ground-truth examples so changes to chunking, ranking, prompts, and model behavior could be measured before production rollout.

Observability dashboards

Added dashboards for latency, retrieval quality, answer confidence, cost, and system behavior over time.

Why I built it this way

A RAG system without evaluation eventually becomes a guessing game.

You can tune prompts forever and still not know whether the system is improving. You can swap embedding models and feel better because the demo looks cleaner. But unless the team can measure retrieval quality and trace answers back to sources, the system is hard to trust.

I built this around measurement first because production AI needs a feedback loop. The point was not just to answer questions. The point was to make answer quality visible enough that the team could improve it on purpose.

Impact

• 85% answer accuracy, up from 40% (+45 percentage points) on a 500-question evaluation set through better chunking, re-ranking, and evaluation.
• Citation support so users could inspect the source behind generated answers.
• Freshness controls for content updates and cache invalidation.
• Production dashboards covering accuracy, latency, and cost.
• A repeatable evaluation process for future changes.

Related content

• Blog: RAG on Azure with pgvector
• Case Study: RAG on Azure