All 24 articles, sorted alphabetically
Llama Chat Template
Llama 2's [INST]/[/INST] tags. Llama 3's newer role tokens.
Read article →SLM batching -- throughput for small-model serving
Deep-dive on SLM batching: the per-request inefficiency, static vs continuous batching (join/leave per decode step), the throughput-latency tradeoff, …
Read article →SLM distillation architecture
Deep-dive on small-language-model distillation: teacher forward, KD loss, temperature, curriculum, data augmentation, eval, and deployment.
Read article →Knowledge Distillation
How distillation transfers capability from a teacher model to a student, and why distilled models often beat directly-trained equivalents.
Read article →DPO alignment architecture for small language models
Deep-dive on Direct Preference Optimization for SLMs: preference-pair pipelines, frozen reference models and precomputed log-probs, the beta dial, LoR…
Read article →SLM early-exit inference architecture
Deep-dive on early-exit (adaptive-depth) inference for small language models: intermediate exit classifiers and confidence thresholds, shared LM heads…
Read article →Small Language Model Edge Deployment Architecture in Depth
A 2500-word walkthrough of SLM edge deployment: base model, distillation, quantization, edge runtime, router, cloud fallback, telemetry, OTA, and gove…
Read article →FlashAttention on-device
Deep-dive on FlashAttention for on-device small language models, the algorithmic reframing that computes attention without ever materializing the quad…
Read article →SLM function calling architecture
Deep-dive on reliable tool use from small language models: schema registries compiled to decoding grammars, streaming validation, repair loops, fallba…
Read article →GGUF runtime architecture
Deep-dive on the GGUF runtime behind llama.cpp and on-device inference: a single self-describing file holding metadata, tokenizer, and block-quantized…
Read article →Guided decoding architecture for SLMs
Deep-dive on constrained/structured generation for small models: compiling JSON schema, regex, and CFG to a finite-state automaton, per-step logit mas…
Read article →SLM KV cache architecture
Deep-dive on the KV cache in small-language-model serving: why decode is memory-bandwidth-bound, prefill vs decode, paged KV blocks and block tables (…
Read article →Multi-LoRA serving architecture
Deep-dive on multi-LoRA inference serving: the low-rank adapter decomposition and why the base GEMM is shareable, the adapter registry and GPU-residen…
Read article →On-device SLM NPU acceleration architecture
Deep-dive on running small language models on mobile NPUs: graph compilation and partitioning, the integer MAC array and on-chip SRAM, quantization fo…
Read article →PagedAttention architecture
Deep-dive on PagedAttention for SLM serving: fixed-size KV blocks mapped through per-sequence block tables, on-demand allocation with an O(1) free-lis…
Read article →Prefix caching for SLM serving
Deep-dive on prefix caching for small-language-model serving: block-hashing token prefixes, a radix index of cached KV blocks, paged and reference-cou…
Read article →Model Pruning
How pruning removes redundant weights or attention heads from models, and how sparsity accelerates inference.
Read article →Model Quantization
How quantization reduces model size and inference cost by using lower-precision weights and activations.
Read article →Speculative decoding
Deep-dive on speculative decoding: draft-verify mechanism, parallel verification exploiting the memory-bound insight, acceptance rate and speedup, exa…
Read article →SLM tensor parallelism architecture
Deep-dive on tensor parallelism for serving small language models: column- and row-parallel sharding of attention and MLP, two all-reduces per layer, …
Read article →Vocabulary trimming architecture
Deep-dive on tokenizer vocabulary trimming for small language models: why V x d dominates small-model budgets, corpus surveys, keep sets and byte-fall…
Read article →