Large Language Models

Sentence Similarity with BERT and Flask

Sentence Similarity with BERT and Flask

LLM Agent Architecture in Depth

A 2500-word walkthrough of LLM agent architecture: planner, tool registry, executor, working memory, long-term memory, reflection, guardrails, streami…

LLM chunked prefill architecture

Deep-dive on chunked prefill for LLM serving: how splitting a long prompt into fixed-size token chunks under a single per-step token budget lets the s…

LLM continuous batching architecture

Deep-dive on continuous (iteration-level) batching for LLM serving: instead of static batches that wait for their slowest member, a scheduler recompos…

FlashAttention architecture

Deep-dive on FlashAttention: why the GPU memory hierarchy (HBM vs SRAM) makes standard attention memory-bound, how tiling and the online softmax compu…

LLM guided decoding architecture

Deep-dive on guided (constrained/structured) decoding: why validity can't be reliably obtained by prompting, how a schema or …

LLM KV-cache offloading architecture

Deep-dive on KV-cache offloading for LLM serving: paged block management, LRU eviction with reuse hints, prefix caching, pinned host staging, async co…

LLM KV-cache quantization architecture

Deep-dive on KV-cache quantization for LLM serving: why the KV cache dominates memory, quantize-on-write and dequantize-on-read, per-token vs per-chan…

Multi-LoRA serving architecture

Deep-dive on multi-LoRA serving: one shared base model with many small per-tenant LoRA adapters, the router and adapter cache/store that manage scarce…

PagedAttention architecture

Deep-dive on PagedAttention and KV-cache management in LLM serving: block managers and free pools, continuous-batching admission, prefix caching with …

Prefill/decode disaggregation architecture

Deep-dive on disaggregated LLM inference: why prefill and decode interfere, SLO-aware routing, layer-wise KV cache streaming over NVLink/RDMA, prefix …

LLM priority scheduling architecture

Deep-dive on priority scheduling for LLM inference: priority classes and deadlines, admission control, KV-cache-aware preemption (swap vs recompute), …

LLM prompt caching -- reusing computed context for cost and latency

Deep-dive on LLM prompt caching: KV cache reuse for repeated prompt prefixes, prefix matching, cost reduction (cached tokens billed less) and latency …

LLM model routing architecture

Deep-dive on LLM routing: feature extraction and learned routers, cascade vs predictive dispatch, quality gates and judge models, escalation ceilings,…

LLM sampling and decoding architecture

Deep-dive on LLM decoding: the logit vector, masking, repetition/frequency/presence penalties, temperature scaling, top-k vs top-p vs min-p truncation…

LLM Semantic Caching: Reusing Answers by Meaning, Not Bytes

How a semantic cache embeds queries, searches a vector index of prior Q&A, and serves on a similarity threshold — cutting cost…

Speculative decoding architecture

Deep-dive on speculative decoding: draft model + target verify + accept-reject, plus tree spec, Medusa heads, and self-speculation variants.

Structured output architecture

Deep-dive on structured LLM output: JSON Schema to FSM compilation, token-mask decoding, tokenizer lifting, vLLM/TGI/llama.cpp guided decoding, JSON m…

LLM tokenization architecture

Deep-dive on LLM tokenization: BPE training and vocabulary-size trades, the runtime encode pipeline with byte fallback, special-token security at trus…

LLM tool use -- connecting models to actions and data

Deep-dive on LLM tool use (function calling): the model requesting tools via structured output, tool schemas (name/description/params), the call-execu…

Positional Embeddings: RoPE, ALiBi, and the Quest for Perfect Long-Range Memory

Retrieval-Augmented Generation (RAG): Bridging the Gap Between a Model’s Training and Today’s News

Reinforcement Learning from Human Feedback (RLHF): The Secret Sauce That Made ChatGPT 'Helpful'

Speculative Decoding: How Using a Tiny Model to 'Guess' Makes the Big Model 3x Faster

The Hallucination Problem: Why LLMs Lie and How 'Fact-Checking' Layers Are Being Built

The LLM as the Ultimate Compiler: From Natural Language to Executable Code

Understanding Tokenization: Why 'Apple' Is One Token But 'antidisestablishmentarianism' Is Many

Vision-Language Models (VLM): How Transformers 'See' and Describe Images in Real-Time

Zero-Shot vs. Few-Shot Learning: Why the Best Models Don't Need Training Anymore