RAG on Azure: Production-Ready Semantic Search

The Challenge

An enterprise client needed a production-ready RAG (Retrieval-Augmented Generation) system that could handle complex queries with high accuracy, provide proper citations, and maintain knowledge freshness.

Business Problems

• Low accuracy: Previous solution only answered 40% of queries correctly
• No citations: Users couldn't verify information sources
• Stale knowledge: Manual updates, documents often outdated
• No quality metrics: No way to measure or improve accuracy
• User trust issues: Low accuracy eroded confidence in system

Technical Challenges

• Complex queries: Multi-hop reasoning across multiple documents
• Large corpus: 50GB of technical documentation
• Embedding quality: Simple embeddings missed semantic nuances
• Chunking strategy: Poor chunking reduced retrieval quality
• Evaluation: No systematic way to measure improvements
• Latency requirements: <2 seconds end-to-end response time

Impact on Business

• Support costs: High ticket volume due to poor self-service
• User frustration: Users abandoned system, called support instead
• Knowledge gaps: Important documents not being surfaced
• Competitive disadvantage: Competitors had better search

The Solution

Built a complete RAG system on Azure using PostgreSQL with pgvector, advanced re-ranking, comprehensive offline evaluation, and production monitoring dashboards.

System Architecture

graph TB
    subgraph "User Interface"
        WEB[Web Application]
        API_CLIENT[API Clients]
    end

    subgraph "RAG Service"
        API[FastAPI Service]
        QUERY_PIPELINE[Query Pipeline]
        EMBEDDER[Sentence Transformers<br/>Embedding Model]
    end

    subgraph "Retrieval Pipeline"
        VECTOR_SEARCH[Vector Search<br/>pgvector]
        RERANKER[Cross-Encoder<br/>Re-ranker]
        CONTEXT_BUILDER[Context Builder<br/>Chunking & Assembly]
    end

    subgraph "Generation"
        LLM[Azure OpenAI<br/>GPT-4]
        CITATION_EXTRACTOR[Citation Extractor]
    end

    subgraph "Data Store"
        POSTGRES[(PostgreSQL<br/>+ pgvector)]
        BLOB[Azure Blob Storage<br/>Raw Documents]
    end

    subgraph "Ingestion Pipeline"
        INGESTION[Document Ingestion]
        CHUNKER[Smart Chunker]
        EMBED_BATCH[Batch Embedder]
    end

    subgraph "Evaluation & Monitoring"
        EVAL[Offline Evaluation<br/>Ground Truth Tests]
        METRICS[Prometheus Metrics]
        DASHBOARD[Grafana Dashboards]
    end

    WEB --> API
    API_CLIENT --> API

    API --> QUERY_PIPELINE
    QUERY_PIPELINE --> EMBEDDER
    EMBEDDER --> VECTOR_SEARCH

    VECTOR_SEARCH --> POSTGRES
    VECTOR_SEARCH --> RERANKER
    RERANKER --> CONTEXT_BUILDER

    CONTEXT_BUILDER --> LLM
    LLM --> CITATION_EXTRACTOR
    CITATION_EXTRACTOR --> API

    BLOB --> INGESTION
    INGESTION --> CHUNKER
    CHUNKER --> EMBED_BATCH
    EMBED_BATCH --> POSTGRES

    API -.-> METRICS
    EVAL -.-> METRICS
    METRICS --> DASHBOARD

    style VECTOR_SEARCH fill:#fff3cd
    style RERANKER fill:#d4edda
    style LLM fill:#d1ecf1
    style EVAL fill:#f8d7da

RAG Query Flow

sequenceDiagram
    participant User
    participant API
    participant Embedder
    participant VectorDB
    participant Reranker
    participant LLM
    participant CitationEx

    User->>API: Query: "How do I configure OAuth2?"

    API->>Embedder: Embed Query
    Embedder-->>API: Query Vector

    API->>VectorDB: Vector Search (top 50)
    Note over VectorDB: Cosine similarity<br/>with pgvector
    VectorDB-->>API: 50 Candidate Chunks

    API->>Reranker: Re-rank with Cross-Encoder
    Note over Reranker: Semantic relevance scoring
    Reranker-->>API: Top 5 Chunks (scored)

    API->>API: Build Context + Prompt
    API->>LLM: Generate Response
    LLM-->>API: Generated Answer

    API->>CitationEx: Extract Citations
    CitationEx-->>API: Mapped to Source Chunks

    API-->>User: Answer + Citations
    Note over User: "To configure OAuth2...<br/>[Docs: oauth-guide.md:123]"

    API->>API: Log Metrics

Chunking Strategy

graph TD
    START[Raw Document] --> PARSE[Parse Structure]

    PARSE --> DETECT{Content Type?}

    DETECT -->|Markdown| MD_CHUNK[Semantic Chunking<br/>Headers as boundaries]
    DETECT -->|Code| CODE_CHUNK[Function/Class<br/>boundaries]
    DETECT -->|Plain Text| TEXT_CHUNK[Sliding Window<br/>500 tokens, 50 overlap]

    MD_CHUNK --> METADATA[Add Metadata]
    CODE_CHUNK --> METADATA
    TEXT_CHUNK --> METADATA

    METADATA --> EMBED[Generate Embeddings]
    EMBED --> STORE[(Store in DB<br/>with vectors)]

    style MD_CHUNK fill:#d4edda
    style CODE_CHUNK fill:#d4edda
    style TEXT_CHUNK fill:#d4edda
    style EMBED fill:#fff3cd

Technology Stack

• Cloud Platform: Azure
• Database: PostgreSQL 14 + pgvector extension
• Embedding Model: all-MiniLM-L6-v2 (sentence-transformers)
• Re-ranker: cross-encoder/ms-marco-MiniLM-L-6-v2
• LLM: Azure OpenAI GPT-4
• API Framework: FastAPI (Python)
• Monitoring: Prometheus + Grafana
• Storage: Azure Blob Storage
• Orchestration: Azure Container Instances

Implementation Highlights

Advanced Chunking Strategy

We developed a content-aware chunking system that adapts to document structure:

Markdown Documents:

def chunk_markdown(doc: str) -> List[Chunk]:
    # Parse markdown structure
    headers = extract_headers(doc)

    # Chunk by semantic sections
    chunks = []
    for section in sections_from_headers(headers):
        # Keep sections under 512 tokens
        if len(section.tokens) <= 512:
            chunks.append(create_chunk(section))
        else:
            # Recursively split large sections
            chunks.extend(split_section(section, max_tokens=512, overlap=50))

    return chunks

Result: 25% improvement in retrieval quality vs. simple sliding window

Two-Stage Retrieval: Vector Search + Re-ranking

Why Re-ranking?

• Vector embeddings capture general semantics
• Cross-encoders capture query-document interaction
• Re-ranking top-50 → top-5 dramatically improves precision

Performance:

Stage 1: Vector Search (top 50)
- Speed: 15ms
- Recall: 92% (contains answer in top 50)

Stage 2: Re-ranking (top 5)
- Speed: 45ms
- Precision: 87% (answer in top 5)

Total: 60ms retrieval time

Offline Evaluation Pipeline

Ground Truth Dataset: 500 question-answer pairs with known correct documents

Evaluation Metrics:

1. Retrieval Metrics:

   • Recall@k: Is correct doc in top k results?
   • MRR (Mean Reciprocal Rank): Position of first correct doc
   • nDCG: Ranking quality

2. Generation Metrics:

   • Exact match: Answer matches ground truth
   • F1 score: Token overlap with ground truth
   • Semantic similarity: Embedding similarity

3. End-to-End Metrics:

   • Answer accuracy: Human-evaluated correctness
   • Citation accuracy: Correct source attribution
   • Hallucination rate: Generated facts not in sources

Automated Testing:

@pytest.mark.parametrize("question,expected_doc", ground_truth_pairs)
def test_retrieval_quality(question, expected_doc):
    results = rag_system.retrieve(question, top_k=5)

    # Check if expected doc is in top 5
    assert expected_doc in [r.doc_id for r in results], \
        f"Expected doc {expected_doc} not in top 5 for: {question}"

Freshness & Cache Invalidation

Update Strategy:

graph LR
    A[Document Updated] --> B[Detect Changes]
    B --> C[Re-chunk Affected Sections]
    C --> D[Re-embed New Chunks]
    D --> E[Update Vector DB]
    E --> F[Invalidate Query Cache]
    F --> G[Update Timestamp Metadata]

    style A fill:#fff3cd
    style F fill:#d1ecf1

Incremental Updates: Only re-process changed sections, not entire corpus

Key Outcomes

Accuracy Improvements

Performance Metrics

Retrieval Pipeline Breakdown:

Business Impact

• Support ticket reduction: 45% fewer tickets due to better self-service
• User satisfaction: NPS increased from 32 to 68
• Knowledge coverage: All documents now discoverable
• Trust restored: Citation feature builds user confidence
• Cost savings: $120K/year in support costs avoided
• Competitive advantage: Best-in-class search experience

Evaluation Results

Before vs After (on 500-question test set):

Technical Deep Dive

Vector Search with pgvector

Table Schema:

CREATE TABLE document_chunks (
    id SERIAL PRIMARY KEY,
    document_id UUID NOT NULL,
    chunk_index INT NOT NULL,
    content TEXT NOT NULL,
    embedding VECTOR(384), -- all-MiniLM-L6-v2 dimensions
    metadata JSONB,
    created_at TIMESTAMP DEFAULT NOW(),

    -- Indexes for performance
    CONSTRAINT unique_chunk UNIQUE (document_id, chunk_index)
);

CREATE INDEX ON document_chunks USING ivfflat (embedding vector_cosine_ops)
    WITH (lists = 100);

CREATE INDEX ON document_chunks USING GIN (metadata);

Query:

SELECT
    id,
    content,
    metadata,
    1 - (embedding <=> $1::vector) as similarity
FROM document_chunks
WHERE metadata->>'doc_type' = $2  -- Filter by metadata
ORDER BY embedding <=> $1::vector  -- Cosine distance
LIMIT 50;

Performance: 15ms for 50-result vector search on 200K chunks

Re-ranking Implementation

class CrossEncoderReranker:
    def __init__(self):
        self.model = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

    def rerank(self, query: str, chunks: List[Chunk], top_k: int = 5) -> List[Chunk]:
        # Create query-chunk pairs
        pairs = [(query, chunk.content) for chunk in chunks]

        # Score all pairs
        scores = self.model.predict(pairs)

        # Sort by score
        ranked_chunks = sorted(
            zip(chunks, scores),
            key=lambda x: x[1],
            reverse=True
        )

        return [chunk for chunk, score in ranked_chunks[:top_k]]

Monitoring Dashboard

Key Dashboards:

1. Query Performance:

   • Latency percentiles (p50, p95, p99)
   • Error rates
   • Cache hit rates

2. Quality Metrics (sampled queries):

   • Retrieval recall
   • Answer accuracy
   • Citation accuracy
   • User feedback (thumbs up/down)

3. Cost Tracking:

   • LLM API costs
   • Database costs
   • Total cost per query

4. System Health:

   • Database connection pool
   • API response times
   • Error logs

Lessons Learned

What Worked Well

• Offline evaluation: Ground truth dataset made improvements measurable
• Two-stage retrieval: Re-ranking dramatically improved precision
• pgvector: PostgreSQL + pgvector easier than specialized vector DB
• Smart chunking: Content-aware chunking improved retrieval quality
• Citations: Users trust system more with source attribution

Challenges Overcome

Challenge 1: Chunking Strategy

• Problem: Initial naive chunking (500 tokens, fixed boundaries) broke semantic units
• Solution: Content-aware chunking respecting document structure
• Impact: 25% improvement in retrieval quality

Challenge 2: Slow Vector Search

• Problem: Exact search on 200K vectors took 800ms
• Solution: IVFFlat index + metadata filtering
• Impact: Reduced to 15ms (53x faster)

Challenge 3: Hallucinations

• Problem: GPT-4 generating facts not in retrieved context
• Solution: Strict prompting + citation requirement + post-processing validation
• Impact: Reduced hallucination rate from 18% to 3%

Challenge 4: Keeping Knowledge Fresh

• Problem: Manual document updates, often missed
• Solution: Automated ingestion pipeline with change detection
• Impact: Documents updated within 1 hour of changes

Would Do Differently

• Start with evaluation earlier: Built eval system in month 2; should have been day 1
• Hybrid search: Add keyword search alongside vector search for better recall
• Async generation: Stream LLM responses for perceived lower latency
• A/B testing: Built-in experimentation framework from start

Client Testimonial

The evaluation result described here is 85% answer accuracy, up from 40% (+45 percentage points), on the 500-question test set. The client's wording below is preserved as provided.

"Working with Raymond on our RAG implementation was exceptional. He built a complete system on Azure with pgvector, offline evaluation pipelines, and comprehensive dashboards. The system achieved 85% accuracy improvement over our previous approach, and his documentation made handoff seamless."

— Engineering Manager, Enterprise Client

Related

Related case studies

• Multi-Provider LLM Routing - Another production AI system with observability
• Volvo CQRS Framework - Production system architecture at scale

Related services

• RAG or search buildout
• AI production readiness review
• Backend and platform modernization

Further reading

• RAG on Azure with pgvector: Practical Patterns for Production
• Vector databases without the hype
• The Smallest Eval Loop That Keeps an AI Feature Honest
• MockForge for testing LLM applications