CPU SLM is at an inflection point. Hardware (AMX, AI accelerators), software (vLLM CPU, MLX), and models (Phi-3.5, Qwen 2.5) all converged. What's coming next.
Hardware trends
Intel Sapphire Rapids → Granite Rapids: more AMX. AMD Turin: bigger AVX-512 + AI accelerators. ARM Neoverse V3: better matrix ops. Apple M4: bigger neural engine. CPU inference for 7B-class SLMs at GPU-class speeds within 2 years.
Software stabilization
vLLM, llama.cpp, ONNX Runtime, MLX have converged on similar capabilities: INT4, FP8 cache, paged attention, MoE support. Differentiation now in ergonomics, not raw throughput. Standardization around GGUF and SafeTensors.
Model trends
More small models. Phi-4, Qwen 3, Llama 4 mini. The 1-7B range is the sweet spot for CPU. Quality continues to climb (Phi-3 was on-par with Llama 2 13B; Phi-3.5 close to Llama 3 8B). Compute scaling has slowed; data + post-training drives gains.
Quantization frontier
Aggressive sub-INT4 (Q3, Q2, ternary) for memory-constrained edge. AWQ + Marlin + per-channel + group quantization combinations. Quality near-FP16 at 4-5 bits. Edge cases (long-context, math, code) still suffer; ongoing research.
Application picture
Personal AI assistants on local hardware. Privacy-preserving inference at the edge. Air-gapped enterprise deployments. Code-assist running on the dev's own laptop. Voice agents on phones. CPU-class hardware is enough for many real applications. The cloud-only era is ending for moderate-quality LLM workloads.