Why architecture matters here
SLM edge quantization architecture matters because device constraints are real. A phone has 4-8GB usable RAM shared with the OS + apps; a laptop has 16-32GB. Fit or fail.
Cost is fixed at deploy time. Right format + runtime combination means a 3B model runs on a phone at usable speed; wrong combination means it doesn't run at all.
Reliability under thermal throttling is where careful design shows. Sustained inference heats the device; throttling kicks in; user experience degrades. Batching, pacing, and idle behavior all matter.
Why the edge changes the calculus
In a datacenter, quantization is a point on a cost curve. Halve the bytes per weight and more concurrent sequences fit on a card, tokens-per-dollar improves, and if quality falls further than you like you buy precision back with a larger instance. Every term in that equation is continuous and every choice is reversible. On a phone or a laptop none of it is. The hardware was chosen by the user, its memory was fixed at manufacture, and the model either fits inside the budget your process is permitted to hold or the operating system ends the process. There is no next instance size.
The ceiling is also lower than the spec sheet implies. Nameplate RAM is shared with the window compositor, whatever the user left open in a browser, and the OS itself; mobile platforms enforce a per-process footprint well below total physical memory and terminate anything that crosses it, usually without a warning you can catch and handle. The number that governs your design is the resident set your application is allowed on the worst device you have promised to support, minus the tokenizer, the KV cache at your real context length, the runtime's scratch buffers and the rest of your app's state. Weight bytes are the remainder, and the quantization recipe exists to land on that remainder exactly.
The second structural difference is that there is nothing to batch. Server inference amortizes the cost of streaming a weight matrix out of memory across many concurrent sequences, which raises arithmetic intensity until the hardware is compute-limited. One user on one device generates one token at a time, indefinitely: every weight is read to produce every token, and the batch dimension is one and will stay one. Decode speed is therefore close to a direct function of how many bytes the weights occupy. Taking a model from sixteen bits to four cuts the bytes moved per token by roughly four, and that ratio is essentially the whole speedup. Quantization at the edge is a memory-traffic optimization that also happens to shrink the footprint, not a compute optimization.
That framing predicts battery as well as speed. Fetching a byte from DRAM costs far more energy than the multiply-accumulate that consumes it, so bytes-per-token tracks drain better than any operation count, and it explains why a model that feels fine for one exchange becomes unpleasant across a long session: sustained traffic heats the package, and it is the sustained clock rather than the burst clock that the user experiences. The device-side dynamics of that are developed in on-device NPU acceleration; what belongs here is that the quantization recipe is what sets the traffic in the first place.
What the target silicon actually accelerates
Three classes of compute are in play, and they have different opinions about number formats. The ARM CPU cores are the only unit present on every device you will ever ship to: NEON gives you wide SIMD, and newer cores add integer dot-product and matrix-multiply extensions that make eight-bit multiply-accumulate genuinely fast. A mobile NPU is integer-first and reached only through the vendor's toolchain and driver, which means its availability is a per-SoC question rather than a per-platform one. The integrated GPU sits in between: real half-precision throughput reachable through Metal, Vulkan or OpenCL, sharing physical memory with the CPU so there is no host copy to pay for, but also no bandwidth of its own and a queue it shares with whatever is drawing the interface.
This is why a numerically inferior format with a kernel beats a better one without. Grouped int8 and int4 endure not because they minimize error but because someone has written and tuned a kernel for them on each of those paths. A format with no kernel on your target does not fail in an obvious way. The runtime loads it, unpacks every block to floating point, and calls the generic float matmul. The footprint saving survives that fallback; the arithmetic saving does not, and the unpacking cost is added on top of what you were already paying. It is entirely possible to ship a smaller model that generates more slowly than the one it replaced, and to spend a week looking for the regression somewhere else.
The check is cheap and it is not the one people run. Before committing to a format, confirm that the runtime version you intend to ship has a native kernel for that exact quantization type on that exact backend, and confirm it on the oldest core you support rather than the newest. Whether a file loads is a format question; whether it is fast is a kernel question, and in every one of these runtimes those two things are decided by different code written by different people at different times.
The architecture: every layer explained
Walk the diagram top to bottom.
SLM in FP16. The base — Phi-3.5-mini, Llama 3.2 3B, Qwen 3 3B. Foundation to quantize.
Quantizer. Tool chain converting to on-device format. llama.cpp's convert + quantize; MLC's compilation; coremltools for Apple.
Format Choice. INT4 (Q4_K_M in GGUF, standard); INT2 (Q2_K, extreme compression, quality risk); FP8 (limited edge support).
GGUF (llama.cpp). Cross-platform; Windows/Linux/macOS/Android/iOS. Flexible; widely supported. Q4_K_M is standard default.
MLC (mobile). Compiles model with TVM per target device. Best perf on Android/iOS with GPU/NPU.
Core ML (iOS/macOS). Apple's on-device ML. Integrates with Neural Engine + GPU. Best iOS integration.
Weight sharing. K-means clustering weights to a small codebook. Extra compression beyond bit-depth.
KV quantization. On-device long-context needs INT8/INT4 KV cache to fit.
Device Constraints. RAM budget (2-6GB for model); thermal throttling; battery drain rate.
Quality Eval. Task-specific benchmarks; language coverage; edge-case handling.
End-to-end deployment flow
Trace a deployment. Target: Phi-3.5-mini (3.8B) on iOS.
Download FP16 weights from Hugging Face. Convert to Core ML format via coremltools: quantize INT4 per-block, target Neural Engine + GPU compute.
Resulting Core ML package: ~2GB. Fits in app bundle or download-on-first-launch.
Alternative: llama.cpp GGUF Q4_K_M. ~2.3GB. Runs on iOS via a Metal-backed llama.cpp fork or as a library.
Benchmark on iPhone 15 Pro: Core ML gets 30 tokens/sec; llama.cpp gets 22. Core ML wins on Apple silicon.
Quality eval on Phi-3 benchmarks: MMLU 68 vs FP16 baseline 70 (acceptable), HumanEval 55 vs 58. Translation to target languages tested.
Deployment: bundle Core ML in the app. Cold start ~1s to load model. First-token latency ~200ms after warm.
Sustained use: thermal throttles after 5 minutes of continuous generation. App backs off with idle detection.
Battery: 20% drain per hour of active generation. Documented in app.
Format choice follows runtime support, not accuracy
Ranking candidate formats by their reported accuracy degradation is the wrong opening move for a device target. The ordering that survives contact with a shipping product runs the other way: choose the runtime your platform team can integrate, sign, ship and keep updated, then take the best format that runtime accelerates on the hardware in your support matrix. Accuracy differences between reasonable formats at the same bit-width are small next to the difference between a kernel that exists and one that does not.
The realistic options divide by reach. A GGUF artifact under llama.cpp buys the broadest span from one file, covering desktop operating systems and both mobile platforms with a CPU-first execution path and optional accelerator offload. ONNX Runtime covers a similarly wide range by delegating subgraphs to whichever execution provider is present, which makes the accelerator question configuration rather than a rebuild. Core ML offers the deepest integration on Apple hardware, including a path to the Neural Engine that nothing else reaches, at the cost of being Apple-only. LiteRT with vendor delegates is the mainstream Android route, where the quality of the experience depends heavily on which SoC vendor wrote the delegate. ExecuTorch keeps the export path native to a PyTorch training stack, which matters when the same team owns fine-tuning and deployment and wants one description of the model.
The trap underneath all of this is that a quantized artifact is not a portable model. It is a build for one runtime. The block layout, the group size, which tensors were held back at higher precision, and any statistics baked in during conversion are all properties of the toolchain that produced the file. Taking a Core ML package to Android is not a conversion; it is a fresh quantization from the original float checkpoint, yielding a different artifact with a different quality profile that has to be evaluated on its own terms. Teams usually learn this after iOS ships well and the Android build regresses on a behavior nobody thought to retest. Budget for one artifact and one evaluation run per runtime and accelerator combination you support, and treat "the same model on both platforms" as a claim that needs evidence rather than a description of what you did.
A slower version of the same trap is that the runtime is a moving dependency. Quantized formats gain new types, kernels get rewritten, and accumulation choices change between releases, so a file produced by one version of a quantizer can be read by a newer runtime and produce different output. Pin the runtime revision alongside the artifact hash, treat the pair as the unit you release, and re-run the on-device evaluation when either half moves.
Weight-only versus weight-and-activation on a CPU
On a discrete GPU, weight-only quantization is usually the right default. Bandwidth is the binding constraint, half-precision arithmetic is abundant, so you store weights at four bits, expand them inside the matmul, and leave activations in floating point. That reasoning does not transfer to the processor in a phone. ARM cores have no comparable surplus of float throughput, and their fastest route through a matrix multiply is the integer dot-product and matrix extensions. If the activations stay in floating point, the integer path is never entered, so a weight-only four-bit model on a CPU collects the footprint benefit and very little of the arithmetic benefit.
This is why CPU-targeted device builds tend to quantize activations too, most often to eight bits with scales derived at run time from the tensor in hand. The general tradeoff between scales computed live and scales frozen ahead of time is worked through in dynamic versus static quantization. The edge-specific consequence is that the live option is frequently the only one you can support honestly, for reasons that are about data access rather than performance, and which the next section takes up.
The corollary is that no single ranking of formats holds across a session. A weight-only four-bit build can be the fastest configuration while the integrated GPU is servicing the matmuls and the slowest one twenty seconds later when the GPU is saturated by the interface and the runtime falls back to the CPU. Both states occur on the same device, in the same conversation, without anything having changed in your code. Profile the fallback path as a first-class case, not an edge case.
Calibration data when nothing may leave the device
Any procedure that fits activation ranges needs inputs resembling production. In a server deployment you sample production traffic and the problem is administrative. On device, the premise usually forbids it: the prompts belong to the user, the reason the model is local at all is that they stay local, and in regulated categories a compliance position rests on that. The representative sample has to be manufactured instead, and how you manufacture it is a design decision with a quality consequence.
There are four realistic sources, roughly ordered by fidelity. An opt-in cohort of internal users or paid participants who contribute prompts under a separate agreement gives you real inputs at small volume. Synthetic prompts generated by a larger model against a written specification of the tasks the device model must serve are cheap and controllable, but systematically tidier than what users type, so ranges fitted to them under-represent the messy tail. Public corpora matched to your domain and language mix cover style but not your prompt template or system prompt, which is where a surprising amount of the activation distribution comes from. Finally, on-device statistics collection has the device compute activation histograms locally and return only aggregated ranges, never text; it is the cleanest answer and the most expensive to build, and it still needs consent and a privacy review.
Whichever you choose, audit the locale composition before you trust the result. A calibration set drawn from one language in a product shipping in twenty fits ranges to the wrong scripts, and the damage lands in the languages nobody calibrated, which is generally the same set nobody evaluated. The mechanics of range selection, percentile against min-max and per-tensor against per-channel, belong to calibration. What is particular to the edge is that you get one attempt. There is no loop that recalibrates against production traffic, because there is no production traffic you are permitted to observe, and that asymmetry is a large part of why device builds lean toward activation scales computed at run time.