SLM Edge Quantization Architecture in Depth

Why architecture matters here

SLM edge quantization architecture matters because device constraints are real. A phone has 4-8GB usable RAM shared with the OS + apps; a laptop has 16-32GB. Fit or fail.

Cost is fixed at deploy time. Right format + runtime combination means a 3B model runs on a phone at usable speed; wrong combination means it doesn't run at all.

Reliability under thermal throttling is where careful design shows. Sustained inference heats the device; throttling kicks in; user experience degrades. Batching, pacing, and idle behavior all matter.

Why the edge changes the calculus

In a datacenter, quantization is a point on a cost curve. Halve the bytes per weight and more concurrent sequences fit on a card, tokens-per-dollar improves, and if quality falls further than you like you buy precision back with a larger instance. Every term in that equation is continuous and every choice is reversible. On a phone or a laptop none of it is. The hardware was chosen by the user, its memory was fixed at manufacture, and the model either fits inside the budget your process is permitted to hold or the operating system ends the process. There is no next instance size.

The ceiling is also lower than the spec sheet implies. Nameplate RAM is shared with the window compositor, whatever the user left open in a browser, and the OS itself; mobile platforms enforce a per-process footprint well below total physical memory and terminate anything that crosses it, usually without a warning you can catch and handle. The number that governs your design is the resident set your application is allowed on the worst device you have promised to support, minus the tokenizer, the KV cache at your real context length, the runtime's scratch buffers and the rest of your app's state. Weight bytes are the remainder, and the quantization recipe exists to land on that remainder exactly.

The second structural difference is that there is nothing to batch. Server inference amortizes the cost of streaming a weight matrix out of memory across many concurrent sequences, which raises arithmetic intensity until the hardware is compute-limited. One user on one device generates one token at a time, indefinitely: every weight is read to produce every token, and the batch dimension is one and will stay one. Decode speed is therefore close to a direct function of how many bytes the weights occupy. Taking a model from sixteen bits to four cuts the bytes moved per token by roughly four, and that ratio is essentially the whole speedup. Quantization at the edge is a memory-traffic optimization that also happens to shrink the footprint, not a compute optimization.

That framing predicts battery as well as speed. Fetching a byte from DRAM costs far more energy than the multiply-accumulate that consumes it, so bytes-per-token tracks drain better than any operation count, and it explains why a model that feels fine for one exchange becomes unpleasant across a long session: sustained traffic heats the package, and it is the sustained clock rather than the burst clock that the user experiences. The device-side dynamics of that are developed in on-device NPU acceleration; what belongs here is that the quantization recipe is what sets the traffic in the first place.

What the target silicon actually accelerates

Three classes of compute are in play, and they have different opinions about number formats. The ARM CPU cores are the only unit present on every device you will ever ship to: NEON gives you wide SIMD, and newer cores add integer dot-product and matrix-multiply extensions that make eight-bit multiply-accumulate genuinely fast. A mobile NPU is integer-first and reached only through the vendor's toolchain and driver, which means its availability is a per-SoC question rather than a per-platform one. The integrated GPU sits in between: real half-precision throughput reachable through Metal, Vulkan or OpenCL, sharing physical memory with the CPU so there is no host copy to pay for, but also no bandwidth of its own and a queue it shares with whatever is drawing the interface.

This is why a numerically inferior format with a kernel beats a better one without. Grouped int8 and int4 endure not because they minimize error but because someone has written and tuned a kernel for them on each of those paths. A format with no kernel on your target does not fail in an obvious way. The runtime loads it, unpacks every block to floating point, and calls the generic float matmul. The footprint saving survives that fallback; the arithmetic saving does not, and the unpacking cost is added on top of what you were already paying. It is entirely possible to ship a smaller model that generates more slowly than the one it replaced, and to spend a week looking for the regression somewhere else.

The check is cheap and it is not the one people run. Before committing to a format, confirm that the runtime version you intend to ship has a native kernel for that exact quantization type on that exact backend, and confirm it on the oldest core you support rather than the newest. Whether a file loads is a format question; whether it is fast is a kernel question, and in every one of these runtimes those two things are decided by different code written by different people at different times.

The architecture: every layer explained

Walk the diagram top to bottom.

SLM in FP16. The base — Phi-3.5-mini, Llama 3.2 3B, Qwen 3 3B. Foundation to quantize.

Quantizer. Tool chain converting to on-device format. llama.cpp's convert + quantize; MLC's compilation; coremltools for Apple.

Format Choice. INT4 (Q4_K_M in GGUF, standard); INT2 (Q2_K, extreme compression, quality risk); FP8 (limited edge support).

GGUF (llama.cpp). Cross-platform; Windows/Linux/macOS/Android/iOS. Flexible; widely supported. Q4_K_M is standard default.

MLC (mobile). Compiles model with TVM per target device. Best perf on Android/iOS with GPU/NPU.

Core ML (iOS/macOS). Apple's on-device ML. Integrates with Neural Engine + GPU. Best iOS integration.

Weight sharing. K-means clustering weights to a small codebook. Extra compression beyond bit-depth.

KV quantization. On-device long-context needs INT8/INT4 KV cache to fit.

Device Constraints. RAM budget (2-6GB for model); thermal throttling; battery drain rate.

Quality Eval. Task-specific benchmarks; language coverage; edge-case handling.

End-to-end deployment flow

Trace a deployment. Target: Phi-3.5-mini (3.8B) on iOS.

Download FP16 weights from Hugging Face. Convert to Core ML format via coremltools: quantize INT4 per-block, target Neural Engine + GPU compute.

Resulting Core ML package: ~2GB. Fits in app bundle or download-on-first-launch.

Alternative: llama.cpp GGUF Q4_K_M. ~2.3GB. Runs on iOS via a Metal-backed llama.cpp fork or as a library.

Benchmark on iPhone 15 Pro: Core ML gets 30 tokens/sec; llama.cpp gets 22. Core ML wins on Apple silicon.

Quality eval on Phi-3 benchmarks: MMLU 68 vs FP16 baseline 70 (acceptable), HumanEval 55 vs 58. Translation to target languages tested.

Deployment: bundle Core ML in the app. Cold start ~1s to load model. First-token latency ~200ms after warm.

Sustained use: thermal throttles after 5 minutes of continuous generation. App backs off with idle detection.

Battery: 20% drain per hour of active generation. Documented in app.

Format choice follows runtime support, not accuracy

Ranking candidate formats by their reported accuracy degradation is the wrong opening move for a device target. The ordering that survives contact with a shipping product runs the other way: choose the runtime your platform team can integrate, sign, ship and keep updated, then take the best format that runtime accelerates on the hardware in your support matrix. Accuracy differences between reasonable formats at the same bit-width are small next to the difference between a kernel that exists and one that does not.

The realistic options divide by reach. A GGUF artifact under llama.cpp buys the broadest span from one file, covering desktop operating systems and both mobile platforms with a CPU-first execution path and optional accelerator offload. ONNX Runtime covers a similarly wide range by delegating subgraphs to whichever execution provider is present, which makes the accelerator question configuration rather than a rebuild. Core ML offers the deepest integration on Apple hardware, including a path to the Neural Engine that nothing else reaches, at the cost of being Apple-only. LiteRT with vendor delegates is the mainstream Android route, where the quality of the experience depends heavily on which SoC vendor wrote the delegate. ExecuTorch keeps the export path native to a PyTorch training stack, which matters when the same team owns fine-tuning and deployment and wants one description of the model.

The trap underneath all of this is that a quantized artifact is not a portable model. It is a build for one runtime. The block layout, the group size, which tensors were held back at higher precision, and any statistics baked in during conversion are all properties of the toolchain that produced the file. Taking a Core ML package to Android is not a conversion; it is a fresh quantization from the original float checkpoint, yielding a different artifact with a different quality profile that has to be evaluated on its own terms. Teams usually learn this after iOS ships well and the Android build regresses on a behavior nobody thought to retest. Budget for one artifact and one evaluation run per runtime and accelerator combination you support, and treat "the same model on both platforms" as a claim that needs evidence rather than a description of what you did.

A slower version of the same trap is that the runtime is a moving dependency. Quantized formats gain new types, kernels get rewritten, and accumulation choices change between releases, so a file produced by one version of a quantizer can be read by a newer runtime and produce different output. Pin the runtime revision alongside the artifact hash, treat the pair as the unit you release, and re-run the on-device evaluation when either half moves.

Weight-only versus weight-and-activation on a CPU

On a discrete GPU, weight-only quantization is usually the right default. Bandwidth is the binding constraint, half-precision arithmetic is abundant, so you store weights at four bits, expand them inside the matmul, and leave activations in floating point. That reasoning does not transfer to the processor in a phone. ARM cores have no comparable surplus of float throughput, and their fastest route through a matrix multiply is the integer dot-product and matrix extensions. If the activations stay in floating point, the integer path is never entered, so a weight-only four-bit model on a CPU collects the footprint benefit and very little of the arithmetic benefit.

This is why CPU-targeted device builds tend to quantize activations too, most often to eight bits with scales derived at run time from the tensor in hand. The general tradeoff between scales computed live and scales frozen ahead of time is worked through in dynamic versus static quantization. The edge-specific consequence is that the live option is frequently the only one you can support honestly, for reasons that are about data access rather than performance, and which the next section takes up.

The corollary is that no single ranking of formats holds across a session. A weight-only four-bit build can be the fastest configuration while the integrated GPU is servicing the matmuls and the slowest one twenty seconds later when the GPU is saturated by the interface and the runtime falls back to the CPU. Both states occur on the same device, in the same conversation, without anything having changed in your code. Profile the fallback path as a first-class case, not an edge case.

Calibration data when nothing may leave the device

Any procedure that fits activation ranges needs inputs resembling production. In a server deployment you sample production traffic and the problem is administrative. On device, the premise usually forbids it: the prompts belong to the user, the reason the model is local at all is that they stay local, and in regulated categories a compliance position rests on that. The representative sample has to be manufactured instead, and how you manufacture it is a design decision with a quality consequence.

There are four realistic sources, roughly ordered by fidelity. An opt-in cohort of internal users or paid participants who contribute prompts under a separate agreement gives you real inputs at small volume. Synthetic prompts generated by a larger model against a written specification of the tasks the device model must serve are cheap and controllable, but systematically tidier than what users type, so ranges fitted to them under-represent the messy tail. Public corpora matched to your domain and language mix cover style but not your prompt template or system prompt, which is where a surprising amount of the activation distribution comes from. Finally, on-device statistics collection has the device compute activation histograms locally and return only aggregated ranges, never text; it is the cleanest answer and the most expensive to build, and it still needs consent and a privacy review.

Whichever you choose, audit the locale composition before you trust the result. A calibration set drawn from one language in a product shipping in twenty fits ranges to the wrong scripts, and the damage lands in the languages nobody calibrated, which is generally the same set nobody evaluated. The mechanics of range selection, percentile against min-max and per-tensor against per-channel, belong to calibration. What is particular to the edge is that you get one attempt. There is no loop that recalibrates against production traffic, because there is no production traffic you are permitted to observe, and that asymmetry is a large part of why device builds lean toward activation scales computed at run time.

Smaller model at higher precision, or bigger model crushed

The comparison teams skip is the one their memory budget actually poses. The question is not how few bits a chosen model will tolerate; it is what the best quality available for a fixed number of megabytes resident is. Those are different questions, and only the second admits the answer that a smaller model at a comfortable precision might win. A two-billion-parameter model at four bits and a four-billion at two bits occupy comparable space, and the less-compressed of the two frequently comes out ahead, though not on aggregate benchmark scores, which flatten exactly the differences that matter.

The reason is that damage from aggressive compression is not spread evenly across capabilities. Fluent continuation of common text degrades gracefully, which is why a badly quantized model still reads well in a demo and why the person doing the demo believes it is fine. What breaks first is precision work: holding an output format, naming a tool and its arguments correctly, declining when it should decline, staying in the language it was addressed in, carrying a multi-step instruction through to the end. Those are the behaviors an on-device assistant is built out of, so the degradation concentrates on the load-bearing part of the product while the prose samples look healthy.

Run the comparison at equal footprint rather than equal parameter count. Fix the byte budget, assemble every candidate that fits inside it (several model sizes crossed with several bit-widths, deliberately including the higher-precision small options that get dismissed on reflex), and score them all on the device with your own task suite. The sweep costs a day of engineering and it routinely overturns the assumption that the largest model that fits is the best model that fits. Fix the right budget while you are at it: the resident cost is weights plus the KV cache at the context length you actually advertise, so a larger model is charged twice, once for its parameters and again for a proportionally larger cache per token. Whether that cache is itself quantized changes the arithmetic, and is treated in KV cache quantization.

Evaluating on the device, not the workstation

Perplexity is the default metric for quantization work because it is cheap, stable, and comparable across checkpoints. For deciding whether an edge build ships it is close to useless. It averages next-token likelihood over generic text, while your model has one or two narrow jobs and the only thing you need to know is whether it still does them. A quantized build can move a fraction of a point of perplexity and simultaneously lose the reliability of its structured output, and the metric has no way to report that, because valid and invalid JSON are both perfectly ordinary token sequences.

Build a task suite instead. Use the real prompts with the real system prompt and chat template, score them the way your product scores success, and include the specific failures already sitting in your bug tracker. Schema validity, exact match on extracted fields, correct tool selection, refusal behavior on the cases where refusal is correct, and language retention are all binary or near-binary signals, which is what you want when the question is whether to ship. Keep the suite small enough to run on a handset in a few minutes so that it can run on every build rather than once before a release.

Then run it on the device, with the artifact and runtime you intend to ship. A quantized model evaluated on a workstation is a different computation from the same file on an ARM core: different kernels, different accumulator widths, different decisions about which operators fall back and where the tensors travel when they do. Workstation numbers are a smoke test, useful for catching a conversion that went badly wrong and not much else. The number worth defending comes from a device farm that covers the oldest hardware you support and the model most of your users carry, exercised under the thermal conditions of a real session rather than from a cold start, because the throughput you measure in the first thirty seconds is not the throughput your users live with.

Shipping the weights inside an app

The last stretch is packaging, and it constrains format choice more than most teams anticipate. App stores cap the initial download, so a multi-gigabyte artifact is not going into the base bundle. That leaves store-managed on-demand asset delivery, which keeps the file out of the install size at the cost of a first-run wait and a fresh failure mode when the network is poor, or your own distribution with your own integrity checking, versioning and resume logic. Both are real engineering; neither is a build-system flag.

Whichever route you take, the file has to land on disk in a form the runtime can map directly. Memory-mapped loading is the reason a multi-gigabyte model appears to open instantly, and the mechanics are covered in the GGUF runtime. What matters at packaging time is its precondition: the weights must sit uncompressed and correctly aligned in the filesystem. Any step in your pipeline that compresses, re-packs or re-encodes the asset quietly converts a mapping into a full read and turns an instant open into a multi-second one. The symptom is a slow cold start, which nobody attributes to the packaging step, so assert the alignment in a build check rather than discovering it in a support ticket.

Plan updates as whole-artifact swaps rather than patches. A re-quantized model differs almost everywhere at the byte level, so binary deltas recover very little; assume a full download per model revision and make the frequency a product decision, because asking users to fetch a gigabyte is a thing they will notice. Keep the previous artifact on disk until the replacement has passed evaluation, because rollback is the only remedy you have for a model that regresses on one SoC family. You cannot hotfix a device you cannot reach, and the population that hits the regression is the population least able to tell you about it.

Budget storage as deliberately as memory. Users exhaust disk far more often than RAM, and an application holding two model revisions through an update is the one whose update fails on the devices that are already full. If your rollout keeps a spare copy, the peak storage requirement is twice the artifact, and that peak is what determines whether the update succeeds. Communicate the real number in your store listing rather than the steady-state one.