TensorRT-LLM — NVIDIA's Optimized LLM Inference

Why it matters

TensorRT-LLM often fastest on NVIDIA. Understanding enables optimal NVIDIA deployment.

The architecture

Model compilation: convert model to TensorRT engine. Optimizations baked in.

Runtime: efficient serving.

How it works end to end

Quantization: FP8, INT8, INT4 with post-training or QAT.

Tensor parallel + pipeline parallel across GPUs.

Speculative decoding + Medusa integration.

Triton Inference Server integration for serving.