On this page

🔗The Pain We All Know

Your team builds and deploys services in .NET. It's your bread and butter. Your entire operational stack—from CI/CD to monitoring to security—is built around it. Then, the machine learning team hands you their latest creation: a fantastic new model that's unfortunately trapped inside a Python script.

Suddenly, you're faced with a mountain of new complexity. Do you spin up a Python web server like Flask or FastAPI? How do you package and deploy it? How do you monitor it? Now you have two completely different tech stacks to maintain in production, doubling your operational surface area and your on-call headaches.

What if you could just take that trained model and run it directly inside your existing .NET application, with fantastic performance and zero Python servers?

This isn't a fantasy. This is what ONNX Runtime offers. It's a high-performance inference engine from Microsoft that lets you treat a trained model like any other DLL. It's the bridge that allows you to benefit from the rich Python ML ecosystem without paying the heavy operational price of hosting it.

🔗The Magic Trick: Converting Your Model to ONNX

ONNX is an open standard format for ML models. The first step is to get your model out of its native framework and into this universal format. This is usually a one-time step done by the person who trained the model.

The two most common paths are:

  • PyTorch → ONNX: Use the built-in torch.onnx.export function.
  • scikit-learn → ONNX: Use the skl2onnx library.

Here's a quick look at what it takes to export a simple PyTorch model. The key is to tell the exporter the names of your inputs/outputs and to define which axes are dynamic (like the batch size).

# A simple PyTorch model
import torch
from torch import nn

class SmallClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(in_features=8, out_features=2)
    def forward(self, x):
        return self.fc(x)

# Exporting it to ONNX
model = SmallClassifier().eval()
dummy_input = torch.randn(1, 8) # A sample input to trace the model's graph

torch.onnx.export(
    model,
    dummy_input,
    "model.onnx", # The output file
    input_names=["input"],
    output_names=["output"],
    opset_version=17, # A modern version of the ONNX spec
    dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}} # Allow variable batch sizes
)

That's it. The model.onnx file is now a self-contained, portable artifact that you can copy directly into your .NET project.

🔗Running it in .NET: Simpler Than You Think

Now for the fun part. Let's run this model from a C# application.

First, add the NuGet package: dotnet add package Microsoft.ML.OnnxRuntime

Then, the code is remarkably straightforward.

using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;

// The InferenceSession is the core object. Create it once and reuse it.
using var session = new InferenceSession("path/to/your/model.onnx");

// Prepare your input data in the shape the model expects [batch_size, num_features]
var inputTensor = new DenseTensor<float>(new[] { 1, 8 });
// Let's say our input is 8 floating point numbers
var inputData = new float[] { 1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f, 7.0f, 8.0f };
inputData.AsSpan().CopyTo(inputTensor.Buffer.Span);

// The input name "input" must match the name we used during export
var inputs = new List<NamedOnnxValue> {
    NamedOnnxValue.CreateFromTensor("input", inputTensor)
};

// Run the inference
using var results = session.Run(inputs);

// Get the output tensor by its name
var output = results.First(v => v.Name == "output").AsTensor<float>();

Console.WriteLine($"Output: {string.Join(", ", output.ToArray())}");

🔗What About the GPU?

If you're running on a machine with an NVIDIA GPU and have CUDA installed, you can get a massive performance boost. All you need to do is swap the base NuGet package for the GPU-enabled one and provide a single option when creating your session.

dotnet add package Microsoft.ML.OnnxRuntime.Gpu

// Just provide SessionOptions when creating the session
var opts = new SessionOptions();
opts.AppendExecutionProvider_CUDA(deviceId: 0); // Use the first GPU
using var session = new InferenceSession("model.onnx", opts);
// The rest of your code remains exactly the same.

🔗Productionizing: Cold Starts, Concurrency, and Control

Getting it running is easy. Making it robust is the real work.

  • The Session is a Singleton: The InferenceSession object is thread-safe for running inference (session.Run). It's expensive to create because it loads and optimizes the model. Create it once per model when your application starts up and reuse it for all requests.
  • Warm Up Your Engines: The very first Run call can be slower because the runtime needs to JIT-compile kernels and allocate memory. To avoid this "cold start" penalty hitting your first user, run a dummy inference call as part of your application's startup routine.
  • Control Your Concurrency: By default, ONNX Runtime might try to use all available CPU cores. In a web server handling many requests, this can lead to thread pool starvation. You can tune this with SessionOptions.IntraOpNumThreads. More simply, you can wrap your session.Run calls in a SemaphoreSlim to cap the number of concurrent inferences, preventing your server from getting overwhelmed.

🔗When Should I Stick With Python?

ONNX is not a silver bullet. There are valid reasons to keep a Python server in the loop.

  • Complex Pre/Post-processing: If your model requires complex data transformations (like specific audio processing or tokenization logic) that are a nightmare to port to .NET, it can be easier to keep that logic in Python.
  • Unsupported Dynamic Models: Some cutting-edge models use dynamic control flow or custom operators that don't export cleanly to the static graph format of ONNX.
  • Pure Research: If your team is iterating on model architecture daily, the export-and-test cycle can slow them down. In a pure research context, the simplicity of staying in Python outweighs the deployment benefits of ONNX.

But for the vast majority of stable models being served in production APIs, ONNX shines.

🔗Your Production Go-Live Checklist

  • Is my InferenceSession a singleton that's warmed up at application start?
  • Do I have a health check endpoint that runs a quick inference to verify the model is working?
  • Am I validating the shape and data type of my inputs to prevent crashes?
  • Have I implemented concurrency limits, timeouts, and cancellation tokens?
  • Are my logs structured, including the model version and inference latency?
  • Have I benchmarked my model's latency (p50/p95) and throughput under a realistic load?

🔗What to Measure

Forget abstract FLOPS. These are the numbers that matter for your business:

  • Latency: How long does it take, from the user's perspective? Measure both cold start and warm p50/p95 latency.
  • Throughput: How many requests per second can my service handle before latency spikes?
  • Utilization: Is my CPU/GPU actually being used effectively, or is it sitting idle waiting for data?

🔗Next Steps

This post covers the "how." In the follow-up, I’ll publish a reference repository containing a complete ASP.NET Web API that hosts an ONNX Runtime model, complete with all the production-hardening patterns discussed here: health checks, concurrency controls, and benchmark scripts. You'll be able to git clone and deploy a production-ready .NET inference server in minutes.