On this page

🔗The Integration Challenge

AI-powered mocks are powerful, but they're useless if they don't fit into your team's workflow. You can't just drop a mock server into your infrastructure and expect everyone to use it. You need patterns that make mocks a natural part of development, testing, and CI/CD.

This post covers the production patterns we've seen work in real teams: how to integrate AI-powered mocks into CI/CD pipelines, how to manage deterministic vs non-deterministic data, how to handle test failures, and how to make mocks reliable enough for production-like testing.

🔗Pattern 1: CI/CD Integration with Deterministic Seeds

In CI/CD, you need deterministic test results. Flaky tests that pass or fail randomly are a productivity killer. But LLM-generated data is non-deterministic by default, which creates a tension.

The solution: use deterministic seeding for CI/CD, but allow variation in local development.

# .github/workflows/test.yml
- name: Run tests with mocked APIs
  env:
    MOCKFORGE_SEED: ${{ github.sha }}  # Deterministic seed from commit
    MOCKFORGE_PROVIDER: openai
    MOCKFORGE_MODEL: gpt-3.5-turbo
  run: |
    mockforge serve --seed $MOCKFORGE_SEED &
    sleep 2
    cargo test

The seed is derived from the commit SHA, which means:

  • Same commit = same test data = reproducible results
  • Different commits = different test data = coverage of edge cases
  • Failed tests can be reproduced locally using the same seed

This gives you the best of both worlds: deterministic CI/CD with realistic LLM-generated data.

🔗Pattern 2: Local Development with Variation

For local development, you want variation to catch edge cases. But you also want caching so you're not waiting on LLM calls every time you run tests.

# Local development: use timestamp-based seed for variation
export MOCKFORGE_SEED=$(date +%s)
export MOCKFORGE_CACHE_DIR=~/.mockforge/cache
export MOCKFORGE_CACHE_TTL=86400  # 24 hours

mockforge serve &

The timestamp-based seed ensures you get different data each day, but the cache ensures you're not regenerating on every test run. This balances variation with performance.

Pro tip: Clear the cache when you want fresh data:

rm -rf ~/.mockforge/cache

🔗Pattern 3: Test Data Lifecycle Management

Mock data has a lifecycle: generate, cache, validate, expire. Managing this lifecycle properly is crucial for reliability.

graph TD
    A[Test Request] -->|Check Cache| B{Cached?}
    B -->|Yes| C[Return Cached]
    B -->|No| D[Generate via LLM]
    D -->|Validate| E{Valid Schema?}
    E -->|No| F[Retry with Correction]
    E -->|Yes| G[Cache Result]
    F --> D
    G --> H[Return Data]
    C --> I[Check TTL]
    I -->|Expired?| D
    I -->|Valid| H

Cache TTL strategy:

  • Short TTL (1 hour) for development: catch schema changes quickly
  • Long TTL (30 days) for CI/CD: maximize cache hits, reduce costs
  • Manual invalidation: when upstream API changes are detected

Validation strategy:

  • Always validate generated data against schema
  • Retry with corrected prompt if validation fails
  • Fall back to faker if LLM generation fails repeatedly

🔗Pattern 4: Handling Non-Deterministic Failures

Even with deterministic seeds, LLM generation can occasionally fail or produce invalid data. Your tests need to handle this gracefully.

// Test with retry logic for mock generation failures
#[tokio::test]
async fn test_user_creation() {
    let user = mockforge::generate_user()
        .retry(3)
        .with_fallback(|_| {
            // Fall back to faker if LLM fails
            faker::user()
        })
        .await
        .expect("Failed to generate user");
    
    assert!(user.id.is_some());
    assert!(user.email.contains("@"));
}

The retry logic handles transient LLM failures. The fallback ensures tests don't fail just because the LLM had a bad day. This makes your test suite resilient to LLM variability.

🔗Pattern 5: Schema Evolution and Breaking Changes

APIs change over time. Your mocks need to evolve with them, but you also need to detect when changes break your tests.

MockForge's automatic API sync (covered in a previous post) helps here, but you also need patterns for handling schema evolution in your tests.

// Version-aware schema generation
let schema = load_schema("api-v1.yaml");
let mock_data = mockforge::generate(schema, version: "v1").await?;

// When API changes, tests using old schema still work
// New tests use new schema
let new_schema = load_schema("api-v2.yaml");
let new_mock_data = mockforge::generate(new_schema, version: "v2").await?;

Migration strategy:

  1. Keep old schema versions cached
  2. Generate mocks for both old and new schemas during transition
  3. Gradually migrate tests to new schema
  4. Deprecate old schema once migration is complete

🔗Pattern 6: Observability in Test Environments

Mock servers are infrastructure. They need observability like any other service.

// Instrument mock generation
#[instrument]
async fn generate_mock(schema: &Value) -> Result<Value> {
    let start = Instant::now();
    let result = generate_via_llm(schema).await;
    let duration = start.elapsed();
    
    // Track metrics
    metrics::generation_duration.observe(duration.as_secs_f64());
    metrics::generation_requests.inc();
    
    match &result {
        Ok(_) => metrics::generation_success.inc(),
        Err(_) => metrics::generation_failure.inc(),
    }
    
    result
}

Key metrics to track:

  • Generation latency (p50, p95, p99)
  • Success/failure rates
  • Cache hit rates
  • Cost per generation
  • Schema validation failures

Alerting thresholds:

  • Generation latency > 5 seconds
  • Failure rate > 5%
  • Cache hit rate < 50%
  • Daily cost > budget

🔗Pattern 7: Parallel Test Execution

When running tests in parallel, you need to ensure mock data doesn't conflict between test processes.

// Process-specific cache keys
fn cache_key(schema: &Value, seed: u64, process_id: &str) -> String {
    format!("{}:{}:{}", hash_schema(schema), seed, process_id)
}

The process ID ensures each test process has its own cache namespace. This prevents cache collisions and ensures parallel tests don't interfere with each other.

Alternative: Use a shared cache with proper locking. This is more efficient but requires coordination between processes.

🔗Pattern 8: Staging Environment Mocks

For staging environments, you want mocks that are realistic enough to catch integration issues, but you don't want to expose real user data.

// Staging mocks: LLM-generated with production-like patterns
let staging_mock = mockforge::generate(schema)
    .with_context("staging")  // Influences generation style
    .with_realistic_patterns(true)  // Use production-like patterns
    .await?;

The staging context influences the LLM prompt to generate data that matches production patterns (e.g., realistic user behavior, realistic data distributions) without using real data.

🔗Pattern 9: Error Handling and Fallbacks

LLM generation can fail for many reasons: API rate limits, network issues, invalid responses, cost budget exceeded. Your system needs graceful fallbacks.

enum MockGenerationStrategy {
    LLM,           // Primary: LLM-powered generation
    LLMCached,     // Fallback 1: Cached LLM data
    Faker,         // Fallback 2: Traditional faker
    Static,        // Fallback 3: Static example data
}

async fn generate_with_fallbacks(schema: &Value) -> Result<Value> {
    // Try LLM generation
    if let Ok(data) = generate_via_llm(schema).await {
        return Ok(data);
    }
    
    // Try cached LLM data
    if let Some(cached) = cache.get(schema).await {
        return Ok(cached);
    }
    
    // Fall back to faker
    if let Ok(data) = generate_via_faker(schema).await {
        return Ok(data);
    }
    
    // Last resort: static example
    if let Some(example) = schema.get("example") {
        return Ok(example.clone());
    }
    
    Err("All generation strategies failed")
}

This ensures your mocks always work, even when the LLM is unavailable. The fallback chain preserves functionality while degrading gracefully.

🔗Pattern 10: Cost Control in CI/CD

CI/CD pipelines can generate a lot of mock data. Without cost controls, you can easily exceed your budget.

# CI/CD with cost controls
- name: Run tests with budget
  env:
    MOCKFORGE_DAILY_BUDGET: $10.00
    MOCKFORGE_FALLBACK_ON_BUDGET: true
  run: |
    mockforge serve --budget $MOCKFORGE_DAILY_BUDGET &
    cargo test

When the budget is exceeded, the system automatically falls back to faker libraries. This ensures CI/CD never breaks your budget, even if there's a bug or unusual test volume.

Budget recommendations:

  • Development: $5-10/day (allows experimentation)
  • CI/CD: $20-50/day (covers normal test volume)
  • Staging: $50-100/day (more realistic data needed)

🔗Pattern 11: Schema Validation and Testing

Before using generated mocks in tests, validate that they conform to the schema. This catches LLM generation errors early.

#[test]
fn test_generated_mock_validates() {
    let schema = load_schema("user.yaml");
    let mock = mockforge::generate(&schema).await?;
    
    // Validate against schema
    let validator = jsonschema::JSONSchema::compile(&schema)?;
    let validation_result = validator.validate(&mock);
    
    assert!(validation_result.is_ok(), 
        "Generated mock doesn't validate against schema: {:?}", 
        validation_result.err());
}

Run these validation tests in CI/CD before running your actual test suite. This ensures you're not running tests with invalid mock data.

🔗Pattern 12: Documentation and Onboarding

Mocks are infrastructure. They need documentation like any other service.

Document:

  • How to start the mock server
  • How to configure generation (seeds, providers, models)
  • How to clear cache
  • How to debug generation failures
  • How to add new schemas
  • Cost expectations and budgets

Onboarding checklist:

  • Mock server runs locally
  • Tests use mocks correctly
  • Cache is configured
  • Cost budgets are set
  • Observability is set up
  • Team knows how to debug failures

🔗The Bottom Line

AI-powered mocks are powerful, but they're only valuable if they integrate seamlessly into your workflow. These patterns ensure mocks are:

  • Reliable: Deterministic in CI/CD, cached for performance
  • Resilient: Fallbacks when LLM fails, retries for transient errors
  • Observable: Metrics, logs, and alerts for monitoring
  • Cost-controlled: Budgets and fallbacks prevent overspending
  • Maintainable: Clear documentation, versioning, and lifecycle management

The goal isn't perfect mocks—it's mocks that work well enough that your team actually uses them. These patterns make that possible.