Benchmarks that matter: latency, throughput, dollars

🔗Why We Need to Talk About This

Getting an LLM app from a cool demo to a real-world product is a tricky business. It’s easy to get mesmerized by the magic of text generation, tweaking prompts and models to get the "smartest" possible response.

But here’s the hard truth: your users don't care about the magic behind the curtain. They care about two things: Is it fast? And is it useful?

A slow, laggy UI is a recipe for frustration and abandonment. And for your business, a model that hemorrhages money on every request is a non-starter. That's why we need to stop obsessing over abstract accuracy scores and start benchmarking the things that actually matter to users and the bottom line. It’s all about the user's perception of speed and what you actually pay for.

🔗Scenarios You Can't Afford to Ignore

Your benchmarks are useless if they don't reflect how people will actually use your app. Here are the big ones you need to test.

• The "Typing" Effect: Streaming vs. Non-streaming Think about a chatbot. Is it better to wait three seconds in silence for the perfect, complete answer, or to see it start "typing" a response almost instantly? For users, that immediate feedback is gold. It feels responsive and alive. You absolutely must measure both the time to first token and the total generation time. You might be surprised how much a good streaming implementation can improve the perceived performance, even if the total time is a hair longer.

• The RAG Bottleneck: Cache Hit vs. Miss If you're using Retrieval-Augmented Generation (RAG), your retrieval step is often the Achilles' heel of latency. Pulling documents from a vector database takes time. A cache is your best friend here. You need to know the performance difference between a request where the data is already in a warm cache versus one that has to go fetch it from scratch. This tells you how effective your caching is and shines a bright light on your biggest bottleneck.

• The "Calling a Friend" Problem: Tool Use The moment your LLM needs to use a tool—like calling an external weather API or a flight checker—you've introduced a dependency you don't control. That third-party API can be slow, buggy, or down altogether. You have to benchmark these requests to understand the full latency picture and have a fallback plan for when (not if) the external tool fails or slows to a crawl.

🔗The Only Metrics That Really Count

Don't drown in data. Focus on this small, powerful set of numbers.

• Latency (p50/p90/p99): Don't just look at the average.

  • p50 (Median): This is your typical user's experience.
  • p90: This is what your slowest 10% of users see. If this number is high, things are starting to feel sluggish for a lot of people.
  • p99: This is your canary in the coal mine. A high p99 points to ugly outliers caused by cold starts, network hiccups, or other gremlins in the system. Measure this for both the first token and the total response time.

• Throughput: This isn't just "requests per second." It's "requests per second at a latency your users will tolerate." What happens when you get popular and 100 users hit your app at once? Does it fall over? Does latency spike to unusable levels? This is how you figure out when and how to scale.

• Cost: This one hits your wallet.

  • Provider Charges: This is the obvious one—the bill from OpenAI, Google, Anthropic, etc. It's directly tied to the tokens you send in and get back.
  • Infrastructure: This is the sneaky cost. It's the price of the servers, databases, and vector stores you run yourself. For self-hosted models, this can mean very expensive GPUs. Don't forget to factor this in!

🔗Making It Fast and Affordable: Your Game Plan

• Right-Size Your Model and Context Don't use a sledgehammer to crack a nut. Start with smaller, faster, cheaper models first. Only bump up to the giant, powerful models if you can prove that the quality improvement is absolutely necessary. And keep your contexts lean! Long prompts and lengthy RAG documents balloon your costs and latency. Be ruthless about trimming unnecessary context.

• Cache Everything. Aggressively. Caching is the undisputed hero of performance and cost-saving.

  • Cache full responses for identical requests.
  • Cache your RAG results. Don't re-fetch and re-embed the same documents over and over.
  • Measure your cache hit rate. A low hit rate means your caching strategy isn't working.

• Use a Real Load Testing Harness Testing one request at a time tells you nothing. You need to simulate a crowd. Use a tool like k6, Locust, or Artillery to hammer your application with concurrent users. Crucially, measure latency from the client side to capture the full story—network time and all. Store these results, graph them over time, and set up alerts. That’s how you catch performance regressions before your users do.