Core concept
Auto-scaling. Multi-model endpoints (share instance across models). Multi-container endpoints (pipeline). Shadow tests.
Advertisement
How it works
Persistent endpoints cost even when idle. Async + Serverless for spiky workloads. Cost varies dramatically.
Advertisement
Trade-offs + gotchas
Real-time: latency-sensitive apps. Async: hour-long inference (large images/audio). Serverless: bursty low volume. Batch: offline.