Core concept
Attention kernel modified to handle non-contiguous blocks. Blocks shared between requests when possible (prefix caching).
Advertisement
How it works
Complex kernel. Requires framework support. vLLM + TRT-LLM implement.
Advertisement
Trade-offs + gotchas
LLM serving. Enables 10x+ throughput vs contiguous KV cache.