Why it matters

ONNX Runtime enables cross-platform inference. Understanding shapes portability.

Advertisement

The architecture

Model in ONNX format.

Execution providers: CUDA, TensorRT, CoreML, DirectML, CPU.

Optimizations + graph transforms.

ONNX Runtime stackONNX modelcross-frameworkExecution providersCUDA / TRT / CoreML / etc.Optimizationsgraph + kernelONNX Runtime GenAI extension supports LLM tokenization + KV cache + generation
ONNX Runtime.
Advertisement

How it works end to end

Onnxruntime-genai for LLM.

Multi-backend.

Web / mobile / server variants.