Why architecture matters here
SLMs fail when teams distill without a target task or without the right data. A 7B distilled from a 70B on generic web text ends up mediocre everywhere. A 7B distilled with a curated task-specific dataset can match or beat the 70B on that task.
The architecture matters because each step has a design decision: teacher choice (bigger is not always better), dataset curation (task-specific beats generic), loss formulation (KL vs CE mix), temperature (how soft to make targets), and evaluation (task-specific benchmarks that reflect the product surface).
Getting the pipeline right yields models that ship and delight users at cost that scales.
The architecture: every piece explained
The top strip is the training loop. Teacher model is the large model whose behavior you want to compress. Dataset curation gathers task-representative prompts + expected outputs, augmented with synthetic teacher-generated examples. Teacher forward produces soft targets — full logit distributions — that carry more information than hard labels alone. Student model is the small architecture; often a purpose-designed compact model.
The middle row is the loss formulation. KD loss combines KL divergence between student and teacher distributions with a cross-entropy loss on hard labels; the mix is tuned. Temperature softens both distributions — higher T gives smoother targets that teach relative preferences. Curriculum orders examples from easy to hard so the student builds capability gradually. Data augmentation uses the teacher to generate additional supervised examples where labeled data is scarce.
The lower rows are practical delivery. Eval harness measures per-task metrics, latency, and cost against the teacher. Deployment target — edge, CPU, small GPU — drives quantization and architecture choices. Ops handles versioning, A/B testing, rollback, and feedback labels that flow back into the next distillation cycle.
End-to-end flow
End-to-end: a team distills a 70B teacher into a 3B student for on-device chat summarization. They curate 500k domain-specific summarization examples and augment with 2M teacher-generated pairs. Teacher forward produces soft logit distributions at temperature 4. Student trains for 3 epochs with KD loss = 0.7 KL + 0.3 CE. Curriculum starts with short passages, grows to long documents. Eval: ROUGE-L 0.42 vs teacher 0.45 — 6% gap at 1/20 the size. Latency on device: 40 tokens/s vs 4 for the teacher. Ship. A/B against the previous 3B baseline shows +3% user acceptance. Rollback flag ready. User feedback logs feed the next distillation.
What the student is actually fitting
The mechanics of the soft-target objective - the temperature-scaled softmax, the KL term, the T^2 factor that restores the gradient scale the temperature would otherwise shrink, and a worked numeric example - are derived in full in Distillation Math: Teacher to Student. This article does not repeat that derivation. What matters for a small student is the assumption the standard derivation quietly makes.
The classic objective minimizes the forward KL divergence, KL(teacher || student). That direction is mode-covering: the teacher's probability is the weight on each term, so wherever the teacher puts mass, the student is penalized for putting none. For a student of comparable capacity that is exactly right - you want the whole landscape reproduced. For a student twenty to fifty times smaller it is a trap. The student cannot represent every mode the teacher has, so a mass-covering objective forces it to smear probability across regions it cannot model. The symptom in a generative SLM is hedging: bland, averaged, over-general text, and non-trivial probability on continuations the teacher would never produce.
The reverse direction, KL(student || teacher), is mode-seeking. The student's own probability weights the terms, so it is penalized only for putting mass where the teacher has none, and it is free to abandon modes it cannot afford. It concentrates on a subset of the teacher's behavior and represents that subset well. For classification over a handful of classes the distinction rarely matters. For open-ended generation under a real capacity gap it is one of the largest levers available, and it is the core idea behind the MiniLLM and generalized-KD lines of work. Reverse KL has no closed form over sequences, so it is estimated from student samples - which is why it arrives bundled with on-policy training.
Token-level versus sequence-level distillation
Token-level KD is teacher forcing with a richer target. Both models run over the same fixed prefix and the loss matches the student's next-token distribution against the teacher's at every position. The signal is dense - one full distribution over the vocabulary per token - and the teacher contributes a single parallel forward pass, so it is cheap. It has two hard prerequisites: access to the teacher's logits, and a shared tokenizer, because otherwise the two [N, V] tensors do not index the same classes.
Sequence-level KD throws the logits away. The teacher decodes complete outputs for each prompt - greedy, beam, or sampled - and the student trains with ordinary maximum likelihood on those sequences. It looks like a downgrade and often is not. Decoding approximates the teacher's sequence distribution by its high-probability modes, so the resulting corpus is far less multi-modal than real text: for a given prompt there is essentially one way to answer instead of thousands. A small student fits that simplified, self-consistent target much better than it fits the messy real distribution. That is the mechanism behind sequence-level KD beating token-level KD for heavily compressed students.
Sequence-level KD also removes both prerequisites: no logit access means an API-only teacher works, and no shared vocabulary means you can distill across model families. The price is the shape of the compute. Token-level costs one extra parallel forward pass per batch; sequence-level costs autoregressive generation, which is serial and dominates the budget. Treat it as a one-time corpus build, not a per-step cost.
Caching teacher logits
If you go token-level over a fixed corpus, cache. A full fp16 distribution over a 128k-token vocabulary is roughly 256 KB per position, which is unstorable for any real corpus. Store the top-k logits with their indices instead - k in the range of tens to a few hundred - and renormalize over the retained entries at training time, optionally folding the discarded tail into one residual bucket. The truncation is not free: the tail you drop is precisely the low-probability structure the temperature was raised to expose, so k and T interact. Raise T and you must raise k.
On-policy distillation and exposure bias
Every offline scheme above trains the student on prefixes it did not generate - ground-truth text, or teacher output. At inference the student conditions on its own prefixes. That mismatch is exposure bias, and it compounds: one slightly off-distribution token puts the model in a state it never saw in training, which makes the next token worse. Small models drift off-manifold faster than large ones, and long outputs - multi-turn chat, agentic tool loops, long summaries - give the drift room to run.
On-policy distillation makes the training distribution the inference distribution by construction. Generate sequences from the current student, run the teacher over those student-generated tokens to get its distribution at each position, and minimize the divergence there. The student is corrected exactly where it actually goes wrong, rather than where a teacher-forced corpus said it might. Generalized knowledge distillation (GKD) is the standard framing, and it is also what makes reverse KL practical, since the student samples the estimator needs are already being drawn.
The cost is real: the teacher must be online, because you cannot precompute targets for sequences that do not exist yet, and every step contains a generation loop. Three mitigations hold up in practice. Mix the batches - alternate on-policy steps with cached offline steps, since a partial on-policy fraction captures most of the benefit at a fraction of the generation cost. Refresh rather than regenerate - sample a pool of student rollouts every N steps and reuse it, accepting some staleness, the same trade-off as replay staleness in RL. Sample with temperature - greedy rollouts explore a narrow band of states, and moderate-temperature sampling widens the set of states the teacher gets to correct.
Self-distillation and verifier-filtered rollouts
Push the idea further and the teacher can disappear. The student generates k candidates per prompt, a verifier keeps only those that pass - unit tests for code, a symbolic checker for math, a schema validator for structured output, exact match where a reference exists - and the student retrains on the survivors. This is rejection-sampling fine-tuning, and its appeal for SLMs is that the supervision is grounded in something that cannot be flattered. The failure mode is diversity collapse: each round trains the model on its own most-likely-and-correct outputs, entropy falls, and after a few rounds the model emits one answer per prompt and has lost the ability to recover from its own mistakes. Track output entropy and pass@k across rounds, not just pass@1.
Distilling rationales, not just answers
A transformer does a fixed amount of computation per token. A large model can afford enough serial depth to resolve a multi-step problem inside one forward pass; a 1B student cannot. Chain-of-thought distillation routes around the student's depth limit: training it to emit the teacher's intermediate reasoning lets it spend extra tokens where it lacks layers. That is why rationale distillation helps small students disproportionately - it buys serial compute the architecture does not otherwise provide.
The naive version teaches fluent nonsense. Train on every trace the teacher produced and the student learns the surface form of reasoning without the constraint that the reasoning entails the answer, which yields confident derivations that arrive at the wrong result. Filter on outcome: sample k traces per problem and keep only those whose final answer is verifiably correct. That works, but it silently reshapes the corpus - problems the teacher solves reliably are over-represented and hard problems disappear, so the student trains mostly on what was already easy.
Rationalization recovers part of the lost tail: when the teacher fails a problem, hand it the ground-truth answer and ask it to produce a justification. Those traces are post-hoc and can encode reasoning that only works because the answer was known, so keep the proportion small, tag the examples, and check whether removing them moves held-out accuracy.
Rationale length is a latency budget
On device, a 300-token rationale in front of a 20-token answer is a sixteen-fold increase in output tokens and therefore in wall-clock latency. Two mitigations. Train the student in two modes behind an explicit control token, so cheap requests skip the rationale entirely. Or bias the corpus toward brevity: among the correct traces for a problem, keep the shortest rather than the first, which distills a terser reasoning style instead of truncating one at inference. Watch the related tic - students copy teacher formatting faithfully, including preamble, bullet habits, and hedging boilerplate, so normalize the style of the corpus before training rather than trying to prompt it away afterward.
Choosing the teacher and building the corpus
The strongest teacher is not always the best teacher. As the gap between teacher and student widens, the teacher's distribution becomes something the student has no capacity to approximate, and transfer degrades even though the teacher improved. The standard remedy is a teacher-assistant chain: distill the frontier model into an intermediate model, then distill the intermediate into the target student, so each hop crosses a gap the receiving model can represent. A same-family teacher is worth a lot for token-level KD as well, because the matching tokenizer comes for free.
Tokenizer alignment is a hard constraint, not a detail. Token-level matching is meaningless across different vocabularies. Cross-tokenizer distillation requires aligning token boundaries - mapping through bytes, or matching spans by minimum edit distance - and every alignment heuristic loses signal at the seams. If teacher and student come from different families, sequence-level KD sidesteps the problem entirely and is usually the right call.
The prompt distribution is the capability. The student learns the teacher's behavior on the inputs you distill over and nothing else. The operational loop around that - curation, drift monitoring, scheduled re-distillation - is covered in Model distillation architecture, and the stage-by-stage pipeline view in SLM Distillation Architecture in Depth. What is specific to synthetic corpora is that teacher-generated prompts collapse in diversity fast: ask a model for a thousand support queries and you get a hundred paraphrases of ten. Force the spread with a seed grid - domain, persona, constraint, difficulty, length, and deliberate malformation - sample prompts at high temperature and answers at low, and deduplicate by embedding neighborhood rather than exact match. For the general data-quality machinery (near-duplicate detection, quality classifiers, mixing ratios) see Training Data for SLMs.
Prefer breadth to epochs. Teacher-generated data is cheap to regenerate and expensive to repeat: synthetic text is more self-similar than real text, so it overfits in fewer passes. Spare budget is better spent generating more prompts than running another epoch over the ones you have.