Why architecture matters here
SLMs fail when teams distill without a target task or without the right data. A 7B distilled from a 70B on generic web text ends up mediocre everywhere. A 7B distilled with a curated task-specific dataset can match or beat the 70B on that task.
The architecture matters because each step has a design decision: teacher choice (bigger is not always better), dataset curation (task-specific beats generic), loss formulation (KL vs CE mix), temperature (how soft to make targets), and evaluation (task-specific benchmarks that reflect the product surface).
Getting the pipeline right yields models that ship and delight users at cost that scales.
The architecture: every piece explained
The top strip is the training loop. Teacher model is the large model whose behavior you want to compress. Dataset curation gathers task-representative prompts + expected outputs, augmented with synthetic teacher-generated examples. Teacher forward produces soft targets — full logit distributions — that carry more information than hard labels alone. Student model is the small architecture; often a purpose-designed compact model.
The middle row is the loss formulation. KD loss combines KL divergence between student and teacher distributions with a cross-entropy loss on hard labels; the mix is tuned. Temperature softens both distributions — higher T gives smoother targets that teach relative preferences. Curriculum orders examples from easy to hard so the student builds capability gradually. Data augmentation uses the teacher to generate additional supervised examples where labeled data is scarce.
The lower rows are practical delivery. Eval harness measures per-task metrics, latency, and cost against the teacher. Deployment target — edge, CPU, small GPU — drives quantization and architecture choices. Ops handles versioning, A/B testing, rollback, and feedback labels that flow back into the next distillation cycle.
End-to-end flow
End-to-end: a team distills a 70B teacher into a 3B student for on-device chat summarization. They curate 500k domain-specific summarization examples and augment with 2M teacher-generated pairs. Teacher forward produces soft logit distributions at temperature 4. Student trains for 3 epochs with KD loss = 0.7 KL + 0.3 CE. Curriculum starts with short passages, grows to long documents. Eval: ROUGE-L 0.42 vs teacher 0.45 — 6% gap at 1/20 the size. Latency on device: 40 tokens/s vs 4 for the teacher. Ship. A/B against the previous 3B baseline shows +3% user acceptance. Rollback flag ready. User feedback logs feed the next distillation.
What the student is actually fitting
The mechanics of the soft-target objective - the temperature-scaled softmax, the KL term, the T^2 factor that restores the gradient scale the temperature would otherwise shrink, and a worked numeric example - are derived in full in Distillation Math: Teacher to Student. This article does not repeat that derivation. What matters for a small student is the assumption the standard derivation quietly makes.
The classic objective minimizes the forward KL divergence, KL(teacher || student). That direction is mode-covering: the teacher's probability is the weight on each term, so wherever the teacher puts mass, the student is penalized for putting none. For a student of comparable capacity that is exactly right - you want the whole landscape reproduced. For a student twenty to fifty times smaller it is a trap. The student cannot represent every mode the teacher has, so a mass-covering objective forces it to smear probability across regions it cannot model. The symptom in a generative SLM is hedging: bland, averaged, over-general text, and non-trivial probability on continuations the teacher would never produce.
The reverse direction, KL(student || teacher), is mode-seeking. The student's own probability weights the terms, so it is penalized only for putting mass where the teacher has none, and it is free to abandon modes it cannot afford. It concentrates on a subset of the teacher's behavior and represents that subset well. For classification over a handful of classes the distinction rarely matters. For open-ended generation under a real capacity gap it is one of the largest levers available, and it is the core idea behind the MiniLLM and generalized-KD lines of work. Reverse KL has no closed form over sequences, so it is estimated from student samples - which is why it arrives bundled with on-policy training.
Token-level versus sequence-level distillation
Token-level KD is teacher forcing with a richer target. Both models run over the same fixed prefix and the loss matches the student's next-token distribution against the teacher's at every position. The signal is dense - one full distribution over the vocabulary per token - and the teacher contributes a single parallel forward pass, so it is cheap. It has two hard prerequisites: access to the teacher's logits, and a shared tokenizer, because otherwise the two [N, V] tensors do not index the same classes.
Sequence-level KD throws the logits away. The teacher decodes complete outputs for each prompt - greedy, beam, or sampled - and the student trains with ordinary maximum likelihood on those sequences. It looks like a downgrade and often is not. Decoding approximates the teacher's sequence distribution by its high-probability modes, so the resulting corpus is far less multi-modal than real text: for a given prompt there is essentially one way to answer instead of thousands. A small student fits that simplified, self-consistent target much better than it fits the messy real distribution. That is the mechanism behind sequence-level KD beating token-level KD for heavily compressed students.
Sequence-level KD also removes both prerequisites: no logit access means an API-only teacher works, and no shared vocabulary means you can distill across model families. The price is the shape of the compute. Token-level costs one extra parallel forward pass per batch; sequence-level costs autoregressive generation, which is serial and dominates the budget. Treat it as a one-time corpus build, not a per-step cost.
Caching teacher logits
If you go token-level over a fixed corpus, cache. A full fp16 distribution over a 128k-token vocabulary is roughly 256 KB per position, which is unstorable for any real corpus. Store the top-k logits with their indices instead - k in the range of tens to a few hundred - and renormalize over the retained entries at training time, optionally folding the discarded tail into one residual bucket. The truncation is not free: the tail you drop is precisely the low-probability structure the temperature was raised to expose, so k and T interact. Raise T and you must raise k.
On-policy distillation and exposure bias
Every offline scheme above trains the student on prefixes it did not generate - ground-truth text, or teacher output. At inference the student conditions on its own prefixes. That mismatch is exposure bias, and it compounds: one slightly off-distribution token puts the model in a state it never saw in training, which makes the next token worse. Small models drift off-manifold faster than large ones, and long outputs - multi-turn chat, agentic tool loops, long summaries - give the drift room to run.
On-policy distillation makes the training distribution the inference distribution by construction. Generate sequences from the current student, run the teacher over those student-generated tokens to get its distribution at each position, and minimize the divergence there. The student is corrected exactly where it actually goes wrong, rather than where a teacher-forced corpus said it might. Generalized knowledge distillation (GKD) is the standard framing, and it is also what makes reverse KL practical, since the student samples the estimator needs are already being drawn.
The cost is real: the teacher must be online, because you cannot precompute targets for sequences that do not exist yet, and every step contains a generation loop. Three mitigations hold up in practice. Mix the batches - alternate on-policy steps with cached offline steps, since a partial on-policy fraction captures most of the benefit at a fraction of the generation cost. Refresh rather than regenerate - sample a pool of student rollouts every N steps and reuse it, accepting some staleness, the same trade-off as replay staleness in RL. Sample with temperature - greedy rollouts explore a narrow band of states, and moderate-temperature sampling widens the set of states the teacher gets to correct.
Self-distillation and verifier-filtered rollouts
Push the idea further and the teacher can disappear. The student generates k candidates per prompt, a verifier keeps only those that pass - unit tests for code, a symbolic checker for math, a schema validator for structured output, exact match where a reference exists - and the student retrains on the survivors. This is rejection-sampling fine-tuning, and its appeal for SLMs is that the supervision is grounded in something that cannot be flattered. The failure mode is diversity collapse: each round trains the model on its own most-likely-and-correct outputs, entropy falls, and after a few rounds the model emits one answer per prompt and has lost the ability to recover from its own mistakes. Track output entropy and pass@k across rounds, not just pass@1.
Distilling rationales, not just answers
A transformer does a fixed amount of computation per token. A large model can afford enough serial depth to resolve a multi-step problem inside one forward pass; a 1B student cannot. Chain-of-thought distillation routes around the student's depth limit: training it to emit the teacher's intermediate reasoning lets it spend extra tokens where it lacks layers. That is why rationale distillation helps small students disproportionately - it buys serial compute the architecture does not otherwise provide.
The naive version teaches fluent nonsense. Train on every trace the teacher produced and the student learns the surface form of reasoning without the constraint that the reasoning entails the answer, which yields confident derivations that arrive at the wrong result. Filter on outcome: sample k traces per problem and keep only those whose final answer is verifiably correct. That works, but it silently reshapes the corpus - problems the teacher solves reliably are over-represented and hard problems disappear, so the student trains mostly on what was already easy.
Rationalization recovers part of the lost tail: when the teacher fails a problem, hand it the ground-truth answer and ask it to produce a justification. Those traces are post-hoc and can encode reasoning that only works because the answer was known, so keep the proportion small, tag the examples, and check whether removing them moves held-out accuracy.
Rationale length is a latency budget
On device, a 300-token rationale in front of a 20-token answer is a sixteen-fold increase in output tokens and therefore in wall-clock latency. Two mitigations. Train the student in two modes behind an explicit control token, so cheap requests skip the rationale entirely. Or bias the corpus toward brevity: among the correct traces for a problem, keep the shortest rather than the first, which distills a terser reasoning style instead of truncating one at inference. Watch the related tic - students copy teacher formatting faithfully, including preamble, bullet habits, and hedging boilerplate, so normalize the style of the corpus before training rather than trying to prompt it away afterward.
Choosing the teacher and building the corpus
The strongest teacher is not always the best teacher. As the gap between teacher and student widens, the teacher's distribution becomes something the student has no capacity to approximate, and transfer degrades even though the teacher improved. The standard remedy is a teacher-assistant chain: distill the frontier model into an intermediate model, then distill the intermediate into the target student, so each hop crosses a gap the receiving model can represent. A same-family teacher is worth a lot for token-level KD as well, because the matching tokenizer comes for free.
Tokenizer alignment is a hard constraint, not a detail. Token-level matching is meaningless across different vocabularies. Cross-tokenizer distillation requires aligning token boundaries - mapping through bytes, or matching spans by minimum edit distance - and every alignment heuristic loses signal at the seams. If teacher and student come from different families, sequence-level KD sidesteps the problem entirely and is usually the right call.
The prompt distribution is the capability. The student learns the teacher's behavior on the inputs you distill over and nothing else. The operational loop around that - curation, drift monitoring, scheduled re-distillation - is covered in Model distillation architecture, and the stage-by-stage pipeline view in SLM Distillation Architecture in Depth. What is specific to synthetic corpora is that teacher-generated prompts collapse in diversity fast: ask a model for a thousand support queries and you get a hundred paraphrases of ten. Force the spread with a seed grid - domain, persona, constraint, difficulty, length, and deliberate malformation - sample prompts at high temperature and answers at low, and deduplicate by embedding neighborhood rather than exact match. For the general data-quality machinery (near-duplicate detection, quality classifiers, mixing ratios) see Training Data for SLMs.
Prefer breadth to epochs. Teacher-generated data is cheap to regenerate and expensive to repeat: synthetic text is more self-similar than real text, so it overfits in fewer passes. Spare budget is better spent generating more prompts than running another epoch over the ones you have.
Stacking distillation with pruning and quantization
Distillation, pruning, and quantization are usually presented as alternatives. They compose, and the order is prune, distill to heal, then quantize, with an evaluation gate between stages.
Pruning goes first because of initialization. Structured pruning - dropping whole layers, attention heads, or slices of the FFN width - produces a student whose weights are the teacher's weights, minus some. That student starts far closer to the teacher's function than a randomly initialized model of the same size, so the distillation run that follows is repairing damage rather than learning language from scratch, and it converges in a small fraction of the tokens. Evaluate immediately after pruning and again after healing, because the two stages fail differently and a single end-to-end number will not tell you which one hurt.
Quantization goes last because it is a lossy transform of finished weights. If int4 is the deployment target you have two paths. Post-training quantization followed by measurement is cheap and often sufficient. Quantization-aware training - keeping the KD loss active while the student's forward pass runs through fake-quant operators - costs a training run but lets the student adapt to the grid it will be stored on. Do not assume a distilled student is inherently quantization-robust: what breaks low-bit quantization is outlier activation channels, and distillation does not remove them. SLM edge quantization covers the formats and the outlier problem.
The rule violated most often: evaluate the artifact you ship. A model measured in bf16 and deployed in int4 has not been measured. Gate on the quantized weights, at production sampling settings, on the target hardware.
What does not survive compression
Compression loss is not uniform, and collapsing it into a single quality-gap number is how teams get surprised in production. Capability degrades in two distinct patterns.
Graceful degradation. Surface competence - fluency, formatting, tone, task shape, in-domain classification, extraction, routing, and summarization of text resembling the distillation corpus - declines slowly and roughly with size. These are the capabilities distillation transfers well, because they are functions of the input that a smaller model can approximate.
Cliffs. Other capabilities do not decline, they fall over. Long-tail factual recall is the clearest case: parametric knowledge needs parameters to live in, and no objective compresses a large model's memorized facts into a model with no room for them. Reasoning depth behaves the same way - problems needing more sequential steps than the student can externalize are not answered slightly worse, they are answered wrongly and confidently. The pattern repeats for robustness to paraphrase and adversarial phrasing, for following several simultaneous constraints, for long-context reliability, and for calibration, where the student often inherits the teacher's confidence without the teacher's competence.
The architectural consequence is to stop asking the student to hold knowledge. Distill the shape of the work - instruction following, tool selection, argument construction, output format, routing - and supply facts at inference through retrieval or tool calls. A 1B student that reliably calls the right tool and formats the result is a better product than a 3B student that half-remembers your documentation. Where the student is genuinely weak, route around it: a confidence threshold or a slice-based rule that escalates to the teacher keeps the tail correct while the student serves the bulk of traffic.
Evaluating a distilled student honestly
Distillation breaks evaluation in ways ordinary fine-tuning does not, because the training data came out of a model.
Benchmark leakage through the teacher
Your teacher was trained on the public internet, which contains the benchmarks. When you ask it to generate a synthetic corpus, some of what it emits is a reconstruction of material it memorized - including benchmark items, verbatim or lightly paraphrased. The student then trains on the test set through an intermediary, and its public benchmark scores become meaningless in a way that is invisible unless you go looking. Decontaminate the generated corpus against every eval set you intend to report, by n-gram overlap and by embedding similarity, not by exact string match. Then hold out a private evaluation built from your own traffic after the corpus was generated, and treat that number as the real one.
Judge circularity
Using the teacher to grade the student is the default and it is close to worthless. The student was trained to imitate the teacher's outputs, so a teacher-judge rewards exactly that imitation - including the teacher's stylistic preferences and its errors. You measure fidelity and report it as quality. Use a judge from a different model family, a programmatic check, or human review for anything you intend to act on.
Aggregates hide cliffs
Because degradation is non-uniform, a single average is the wrong instrument: a small drop in aggregate accuracy is routinely a large collapse on one slice, offset by noise elsewhere. Report per slice - per intent or task type, per input-length bucket, per language, per difficulty tier, and specifically on the rare cases you deliberately oversampled. Add a regression suite against the model you are replacing, not only against the teacher, because the question users care about is whether anything got worse today.
Finally, measure at deployment settings. Temperature is a training-time device; the shipped student runs at whatever the serving config says, and evaluating a distilled model at its distillation temperature is a common and entirely silent error. Latency, tokens per second on the actual device, and the token overhead of any rationale the student emits belong in the same table as the quality scores - those are the numbers distillation was performed to move.