LLM-Powered Mock Data Generation: Beyond Faker Libraries

🔗The Limitations of Traditional Mock Data

Faker libraries are great for generating simple test data. Need a random email address? A fake name? A UUID? They've got you covered. But when you're mocking complex APIs with relationships, business logic constraints, and realistic data patterns, traditional fakers fall short.

Here's the problem: a faker library can give you a random email address, but it can't give you an email address that matches the user's name. It can give you a random date, but it can't give you a date that makes sense in the context of the business logic. It can generate a user object, but it can't ensure that the user's orders have realistic timestamps, or that the user's profile matches their purchase history.

Real APIs don't return random data. They return data that follows patterns, respects relationships, and makes business sense. When your mocks generate obviously fake, disconnected data, your tests pass but your assumptions are wrong. You're not testing against realistic scenarios.

🔗Enter LLM-Powered Generation

MockForge addresses this by using LLMs to generate intelligent, context-aware mock data. Instead of just filling in schema fields with random values, the system uses language models to synthesize realistic data that respects relationships, follows patterns, and makes business sense.

The key insight is that LLMs are incredibly good at understanding context and generating coherent, realistic data. Give an LLM a schema and a description of what you're modeling, and it can generate data that looks and behaves like production data—without exposing any real user information.

🔗The Architecture: Schema-Aware LLM Generation

The generation pipeline works like this:

graph TD
    A[OpenAPI Schema] -->|Parse| B[Schema Analysis]
    B -->|Extract Structure| C[Build LLM Prompt]
    C -->|Include Context| D[LLM Provider]
    D -->|Generate JSON| E[Validate Against Schema]
    E -->|Valid?| F[Return Mock Data]
    E -->|Invalid?| G[Retry with Correction]
    G --> D

The system starts with your OpenAPI schema. It analyzes the structure, identifies relationships between entities, and extracts any hints about what the data represents (from descriptions, examples, or field names). This information is assembled into a prompt that instructs the LLM to generate realistic JSON data conforming to the schema.

🔗Prompt Engineering: Making LLMs Generate Valid Data

The prompt construction is critical. You can't just say "generate a user object" and expect valid results. The prompt needs to:

1. Specify the exact JSON structure from the schema
2. Include field constraints (required fields, formats, enums)
3. Provide context about what the data represents
4. Request realistic values that make business sense
5. Maintain relationships between entities

Here's what a well-constructed prompt looks like:

Generate a realistic JSON object for a user registration API response.

Schema:
{
  "id": "string (UUID, required)",
  "email": "string (email format, required)",
  "firstName": "string (required)",
  "lastName": "string (required)",
  "createdAt": "string (ISO 8601 date-time, required)",
  "profile": {
    "bio": "string (optional, max 500 chars)",
    "avatarUrl": "string (URL format, optional)"
  }
}

Requirements:
- Generate realistic, believable data
- Ensure all required fields are present
- Use proper formats (UUID, email, ISO 8601 dates)
- firstName and lastName should match a realistic name
- createdAt should be a recent timestamp
- If bio is provided, it should be coherent and realistic

Return ONLY valid JSON, no explanation.

The LLM understands the schema constraints and generates data that's both valid and realistic. The firstName and lastName will be a matching name pair. The createdAt timestamp will be recent and realistic. The bio, if generated, will be coherent text that makes sense for a user profile.

🔗Cost Optimization: Keeping It Cheap

LLM API calls cost money. At OpenAI's pricing, generating mock data for every test could quickly become expensive. MockForge uses several strategies to keep costs manageable:

1. Caching Generated Data

The most important optimization: cache generated data by schema hash. If you're generating mock data for the same schema repeatedly, you don't need to call the LLM every time. Hash the schema structure, check the cache, and reuse the generated data if available.

// Simplified caching logic
fn generate_with_cache(schema: &Value, cache: &Cache) -> Value {
    let schema_hash = hash_schema(schema);
    
    if let Some(cached) = cache.get(&schema_hash) {
        return cached.clone();
    }
    
    let generated = generate_via_llm(schema);
    cache.set(&schema_hash, &generated);
    generated
}

2. Deterministic Seeding

For reproducible tests, you need deterministic data. MockForge supports seeding the generation process, so the same seed always produces the same data. This is crucial for test stability—you want your tests to be deterministic, not flaky because of random data.

The seeding works by including a seed value in the LLM prompt, which influences the generated data. Combined with caching, this means the first generation is LLM-powered, but subsequent generations with the same seed are cached and instant.

3. Batch Generation

When generating multiple related objects, you can batch them in a single LLM call. Instead of:

• Generate user (1 API call)
• Generate order (1 API call)
• Generate order item (1 API call)

You can do:

• Generate user with orders and order items (1 API call)

This reduces API calls and improves consistency, since the LLM can maintain relationships across the entire batch.

4. Provider Selection

Different LLM providers have different cost/quality trade-offs. For mock data generation, you often don't need GPT-4's sophistication. GPT-3.5-turbo or even Claude Haiku can generate perfectly good mock data at a fraction of the cost.

MockForge supports multiple providers (OpenAI, Anthropic, Ollama for self-hosted), allowing you to choose the right model for your use case and budget.

🔗Real-World Cost Analysis

Let's do the math. At OpenAI's pricing for GPT-3.5-turbo:

• Input: ~$0.50 per 1M tokens
• Output: ~$1.50 per 1M tokens

A typical mock generation request for a user object might be:

• Input prompt: ~500 tokens (schema + instructions)
• Generated output: ~200 tokens (JSON data)

Cost per generation: ~$0.0004 (less than half a cent).

If you're generating 1,000 mock objects:

• Without caching: ~$0.40
• With 50% cache hit rate: ~$0.20
• With 90% cache hit rate: ~$0.04

The caching strategy dramatically reduces costs. In practice, most teams see cache hit rates of 70-90% because they're generating data for the same schemas repeatedly.

🔗Relationship Awareness: The Hard Part

The real value of LLM-powered generation isn't just realistic individual objects—it's maintaining relationships across entities. A user's orders should have timestamps that make sense relative to their account creation. A product's price should match its category. An order's items should reference products that exist.

This is where LLM generation shines. The prompt can include context about related entities:

Generate a user object and their first order.

User schema: [user schema]
Order schema: [order schema]

Requirements:
- User's createdAt should be before order's createdAt
- Order's userId should match user's id
- Order items should reference valid product IDs from the products catalog
- Order total should equal sum of item prices

The LLM understands these relationships and generates data that respects them. This is almost impossible to achieve with traditional faker libraries without complex custom logic.

🔗Integration with OpenAPI

MockForge integrates seamlessly with OpenAPI specs. When you point it at an OpenAPI spec, it:

1. Parses all schemas from components/schemas
2. Identifies relationships via $ref references
3. Extracts hints from schema descriptions and examples
4. Generates prompts that include relationship context
5. Generates mock data for all endpoints

This means you can generate realistic mocks for your entire API from a single OpenAPI spec, with relationships maintained across all entities.

🔗Template Tokens: Hybrid Approach

MockForge also supports template tokens for common patterns:

{
  "id": "{{uuid}}",
  "email": "{{faker.email}}",
  "createdAt": "{{timestamp}}"
}

This hybrid approach gives you the best of both worlds: fast generation for simple fields (via templates), and LLM-powered generation for complex, relationship-aware data.

🔗Production Patterns: When to Use LLM Generation

Use LLM generation when:

• You need realistic data that respects business logic
• Relationships between entities matter for your tests
• You're building demos or staging environments
• Traditional fakers can't generate realistic enough data

Don't use LLM generation when:

• You need deterministic, reproducible data (use seeding + caching)
• Cost is a primary concern and simple fakers suffice
• You're generating data in tight loops (use caching aggressively)
• You need exact control over every field value

Hybrid approach: Use LLM generation for initial data generation, then cache aggressively. Use template tokens for simple fields, LLM generation for complex relationships. This gives you realistic data at a reasonable cost.

🔗Observability: Tracking Costs and Quality

Like any AI-powered feature, you need observability. MockForge tracks:

• Number of LLM API calls
• Tokens consumed (input + output)
• Cost per generation
• Cache hit rates
• Generation latency
• Schema hash distribution

This data helps you understand whether your caching strategy is effective and whether you're spending too much on generation. Set up alerts for unusual cost spikes or low cache hit rates.

🔗The Bottom Line

LLM-powered mock data generation bridges the gap between simple faker libraries and production-like test data. It's not about replacing fakers entirely—it's about using LLMs where they add real value: generating realistic, relationship-aware data that makes your mocks more trustworthy.

The cost is manageable with proper caching and provider selection. At ~$0.01 per 1,000 requests with good caching, it's a negligible expense compared to the value of more realistic test data.

For teams building complex APIs with relationships and business logic, this feature transforms mocks from "good enough" to "indistinguishable from production." That's the difference between tests that pass and tests that actually validate your assumptions.