On this page

So you’ve got an ONNX model. You've done the hard part of training it or wrestling it out of some Python notebook, and now you want to run it fast in a real application. ONNX Runtime is a phenomenal tool for this, but dropping it into your code without a little care is a surefire way to get surprising performance issues and subtle bugs.

Let's be honest, the documentation can be a bit dense. So here's a no-nonsense guide—a set of patterns I've learned from shipping ONNX models in production—to help you get all the performance wins without the usual headaches.

🔗The Session Is Sacred: Memory and Startup

This is the most important rule. An InferenceSession is a heavy, expensive object to create. When you initialize it, ONNX Runtime loads the model graph, analyzes it, and performs a whole suite of powerful graph optimizations—fusing operations, selecting the best kernels for your hardware, and allocating memory. This can take hundreds of milliseconds, or even seconds, for a large model.

  • Treat It Like a Singleton: Never, ever create a new InferenceSession for each request. Create it once per model when your application starts up, and reuse that single instance for the entire lifetime of your app. This alone is your biggest performance win.
  • Crank Up the Optimizations: When creating the session, don't be shy. Set the graph optimization level to ORT_ENABLE_EXTENDED or ORT_ENABLE_ALL. This tells the runtime to be more aggressive about fusing operations, which can significantly speed things up.
  • Leave the Memory Arena Alone (Probably): By default, ONNX Runtime uses a memory arena to reuse large memory blocks between inference calls. This is a good thing! It reduces memory allocation overhead and prevents the garbage collector from going crazy. Only disable this if you have a very specific memory management problem and you've proven the arena is the cause.

🔗Threads and Load: Taming the Beast

Your model is now loaded, but how do you handle multiple requests at once without it all falling over?

  • Tuning Threads Isn't a Dark Art: You'll see IntraOpNumThreads and InterOpNumThreads in the session options. Here's what they mean in plain English:

    • IntraOp: "Inside the operation." How many threads to use for one big operation, like a massive matrix multiplication.
    • InterOp: "Between operations." How many independent operations to run in parallel. A good starting point for both is often Environment.ProcessorCount. But this is just a starting point! You have to benchmark. A CPU-bound model might love having all the cores, but if your app is doing other work, you might want to give it fewer.

  • Batching: Smaller and More Often is Better: If you're batching requests (and you often should be), resist the temptation to create huge batches. A massive batch of 128 might have great theoretical throughput, but it also means the last user in that batch has to wait for the other 127 to finish. This creates terrible tail latency. It's often better for user-facing applications to process smaller batches (say, 8 or 16) more frequently. Your p95 latency will thank you.

  • Guard Your Session with a Semaphore: What happens when 500 requests hit your web API at once? If you just let them all flood the InferenceSession, you'll exhaust your thread pool, memory will spike, and latency will go through the roof. Protect your model. Use a SemaphoreSlim(initialCount, maxCount) to cap the number of concurrent inference calls. Start with a maxCount equal to your IntraOpNumThreads and tune from there. This creates a predictable, stable system instead of a chaotic free-for-all.

🔗The "It Worked in Python" Problem: Conversion Gotchas

A model that's fast but wrong is worse than useless. Most conversion problems happen when exporting from PyTorch or TensorFlow.

  • Opset Mismatches are a Silent Killer: The ONNX spec versions up over time, adding new operations (opsets). If you export a model with opset 17 but try to run it with an older ONNX Runtime that only supports up to 15, you'll get cryptic errors. Make sure your runtime environment's opset support matches the version you exported with. When in doubt, target a recent but well-supported version.
  • Embrace Dynamic Axes: Don't export your model with a fixed batch size of 1. Use dynamic axes for the batch dimension (and sequence length, if applicable). This gives you a single model file that can handle a batch of 1, 8, or 32 without forcing you to re-export. Just make sure the names you give the dynamic axes in your export script match the names you use in your C# code.
  • Validate Your Numbers! Don't skip this. After you convert a model, run a few sample inputs through both the original Python model and the new ONNX model. Compare the outputs. They won't be identical due to floating-point differences, but they should be very close. Define an acceptable tolerance and write a test for it. This saves you from debugging a model that's producing nonsense.

🔗Your Quick Sanity Checklist

Pin this to your monitor before you deploy.

  • Is my session a singleton? Did I create it once at startup and warm it up with a dummy request so the first real user doesn't pay the startup cost?
  • Am I validating inputs? Am I checking tensor shapes and data types before passing them to the model to prevent cryptic crashes?
  • Have I tuned my threads and concurrency? Did I start with a reasonable thread count and add a SemaphoreSlim to prevent overload?
  • Have I measured what matters? Do I know my cold start latency, my warm p95 latency, and my throughput (QPS) under a realistic load?