Core concept
Vs static batching: fill batch fully before dispatch. Requests wait. Wasted compute on padding.
Advertisement
How it works
Complex to implement. Standard in vLLM + TGI + TRT-LLM.
Advertisement
Trade-offs + gotchas
All modern LLM serving.