JVM JIT architecture — C1, C2, Graal, tiered compilation, deopt

Why architecture matters here

JIT surprises are common. Cold JVMs are slow; hot ones can hit a code cache limit and never recompile; a monomorphic call site turns polymorphic and deopts. Each has a specific fix rooted in understanding the pipeline.

The architecture matters because tuning happens at ops (code cache sizing, AOT precompile) and code (avoid megamorphic dispatch, allow escape-analysis-friendly patterns).

With the pipeline in mind, you can design services that reach steady state fast and stay there.

The architecture: every piece explained

The top strip is the compilation path. Class loader loads bytecode. Interpreter runs it warm-up while profiling. Hot methods escalate to C1 (client) — quick compilation with basic optimizations. Hotter ones escalate to C2 / Graal — aggressive optimizations, inlining, unroll, vectorize.

The middle row is the intelligence. Tiered compilation runs C1 and C2 concurrently, using profile data to decide when to recompile. Inline caches speed polymorphic dispatch by remembering recent target types. Escape analysis allows stack allocation and lock elision when objects don't escape. Deoptimization handles the case when an optimistic assumption breaks — a safepoint transitions execution back to the interpreter.

The lower rows are ops. Code cache holds compiled methods; when full, JIT stops compiling and performance drops. Diagnostics — JFR, JITWatch, PrintCompilation — show compilation events. Ops covers warmup strategies, code cache sizing, and AOT precompile options.

Starting interpreted is not a compromise

The obvious design - compile every method to machine code the first time it is called - is wrong for the JVM, and understanding why explains most of the rest of the pipeline. Compilation is not free. It costs CPU on a compiler thread, it costs memory in the code cache, and the code it produces is only good if the compiler knew something about how the method behaves. At the moment a method is first invoked, the JVM knows nothing about it.

It also knows that most methods will never matter. In a typical service the overwhelming majority of loaded methods run a handful of times: configuration parsing, bean wiring, one-shot initialization, error paths that never fire. Spending optimizing-compiler time on those buys nothing and delays the methods that do matter. Interpretation runs roughly an order of magnitude below optimized native code, which is irrelevant for a method called twice and unacceptable for one called ten million times.

So the interpreter does two jobs at once. It gets the program running immediately with zero compilation latency, and while it runs it gathers the evidence the optimizing compiler will need: how often each method is entered, how often each loop goes round, which branches are taken, which concrete types arrive at each call site, whether a cast has ever failed, whether a reference has ever been null. The JIT is not a translator bolted onto an interpreter - it is a compiler that uses the interpreter as its measurement instrument.

Counters, thresholds, and what hot actually means

Two counters drive the whole escalation. Each method carries an invocation counter, bumped on entry, and a backedge counter, bumped whenever control jumps backwards - that is, once per loop iteration. Interpreted and profiling-tier code test these against a threshold at the same points, and crossing it queues the method for compilation.

Two counters exist rather than one because they describe different kinds of hot. A method called a million times and a method called once containing a million-iteration loop are both worth compiling, but only the first is reachable by counting entries. The backedge counter catches the second, and it is also what triggers on-stack replacement, covered below.

The thresholds are tunable, they interact - under tiered compilation the trigger is a formula over both counters, scaled by how backed up the compiler queue currently is, not a flat constant - and their defaults have moved across JDK releases. Treat any specific threshold number you read anywhere as an example rather than a contract. What is stable is the shape: hotness is measured rather than declared, the measurement is cheap enough to leave permanently on, and a method that never crosses the threshold stays interpreted forever no matter how long the process lives.

One consequence catches people out. Counters can decay, because the runtime is deliberately distinguishing "called a lot recently" from "called a lot since three in the morning". A code path invoked steadily but slowly may never accumulate enough to trip anything, and can sit interpreted in a process that has been up for weeks.

The five tiers, and why tier 3 exists

Tiered compilation is usually summarized as "C1 first, then C2", which hides the interesting part. There are five levels.

The normal path is 0 to 3 to 4. Level 3 is the piece that needs explaining, because it is compiled code that is deliberately slower than it needs to be: it carries the same profiling instrumentation the interpreter carries, and recording a receiver type on every virtual call costs real cycles. The JVM pays that because C2 cannot do its job without the data, and gathering enough of it in the interpreter alone would take far longer - the method would spend its entire warm phase at interpreter speed instead of at compiled-with-instrumentation speed.

Level 1 exists for the opposite reason. If a method is trivial enough that C2 has nothing to add - a getter, a small final helper - then profiling it is pure overhead, so it is compiled once without instrumentation and never revisited. Levels 2 and 3 differ only in how much profiling they carry; the JVM drops to level 2 when the C2 queue is backed up, reasoning that if the eventual C2 compile is far away it is better to run fast now than to profile thoroughly for a compiler that will not reach you soon.

Disabling tiering is occasionally proposed as a tuning move. It is almost always the wrong lever: -XX:-TieredCompilation leaves you interpreting all the way to the C2 threshold, lengthening warmup considerably, in exchange for slightly less code cache and slightly fewer compiler cycles.

The compile queue is a background cost

Compilation runs on dedicated compiler threads, not on the thread that tripped the counter. The triggering thread enqueues a request and carries on at its current tier; some time later the compiled code is installed and subsequent calls pick it up. The number of compiler threads scales with available processors and is split between C1 and C2 workers.

Two things follow. First, hot is a request, not a promise of prompt service. During a burst of new code - startup, a lazily initialized subsystem, a deploy that invalidated everything - the queue grows and methods run at a lower tier for longer than the raw thresholds suggest. The JVM compensates by ordering the queue toward the hottest requests and by steering new compiles to level 2, but the latency is real.

Second, compilation competes with your application for CPU. This is invisible on a 32-core host and very visible in a container with a one- or two-CPU quota, where compiler threads and request threads contend for the same small allowance precisely during startup, when the queue is longest. A service that starts far more slowly under a tight CPU limit than the same image on a laptop is usually seeing this rather than slow I/O. -XX:CICompilerCount bounds the thread count if you need to, at the cost of a longer warmup. For reproducible measurement, -Xbatch makes compilation synchronous - the triggering thread blocks until the compile completes - which is useless in production and invaluable when you want a deterministic ordering of compilation log events.

Inlining is the optimization that enables the others

Replacing a call with the callee's body saves a little on its own: no call sequence, no frame setup, no return. That is not why it matters. It matters because every other optimization the compiler has operates inside a single compilation unit, and inlining is the only thing that makes that unit bigger. Constant propagation cannot cross a call boundary. Escape analysis cannot see that the object a factory returned is immediately dismantled by its caller - a dependency the escape analysis article works through in detail. Loop transforms cannot hoist anything out of a loop whose body is opaque. Inline the callee and all of them start working on the merged code.

So the compiler inlines aggressively, but not without limits, because the compilation unit cannot grow unboundedly: compile time rises superlinearly, register pressure climbs, and the code cache is finite. The governing heuristics are roughly these.

• Callee bytecode size. Two separate caps apply - a small one for call sites not known to be hot (-XX:MaxInlineSize) and a considerably larger one for sites the profile says are hot (-XX:FreqInlineSize). A method of a few dozen bytecodes inlines almost anywhere; one of a few hundred inlines only where it is demonstrably hot.
• Inlining depth. A cap on nesting levels (-XX:MaxInlineLevel), which is what stops a deeply layered abstraction from flattening all the way down.
• Caller size. Once the growing compilation unit passes its own size limit, further inlining stops regardless of how attractive an individual callee looks.
• Whether the callee is inlinable at all. A method too cold to have been compiled itself, or one already marked not compilable after repeated deoptimization, will be refused.
• How many receiver types the call site has seen. The one that dominates in practice, and the subject of the next section.

The practical rule that falls out of this is the opposite of the folk advice that manual inlining makes Java faster: keep hot methods small. A 900-line request handler cannot be inlined into anything and, worse, will exhaust the caller-size budget so that nothing inlines into it either. The compiler rewards code factored into small, monomorphic, frequently called methods.

Monomorphic, bimorphic, megamorphic

A virtual or interface call has no single target at compile time. What the JIT has instead is the receiver-type profile that the interpreter and the level-3 code recorded: the set of concrete classes actually observed arriving at that exact bytecode index.

Monomorphic, one type ever seen, is the good case. The compiler emits a cheap guard comparing the receiver's class against the recorded one and, on the fast path, inlines the target outright. The call has become straight-line code. If the guard ever fails, the method deoptimizes.

Bimorphic, two types, still works. The compiler emits two guards and inlines both bodies, or inlines the dominant one and leaves a call for the other. Code size for that site roughly doubles, which consumes inlining budget that other call sites in the same method wanted.

Megamorphic is where it stops. Past a small number of observed receivers the profile is no longer useful for speculation and the compiler emits real dynamic dispatch: load the class, index the vtable or search the itable, jump. The direct cost is a few cycles and an indirect branch the CPU's predictor will frequently miss. The indirect cost is far larger - the callee is not inlined, so the compilation unit ends there, and every optimization downstream of that call loses its inputs.

An inline cache is the runtime structure underneath this. A call site starts unresolved; on first execution it patches itself into a monomorphic cache holding one class and one target address; a miss either re-patches or, if misses keep happening, transitions the site permanently to a megamorphic stub. That transition is one-way. A site that has gone megamorphic does not return to a fast path when traffic becomes uniform again.

Profile pollution is the failure mode you will actually hit

Because the profile is attached to a bytecode index, it is per call site, not per caller. A shared helper - a generic collection utility, a logging wrapper, a stream operation used throughout the codebase - contains one call site that every caller funnels through. Each individual caller may be perfectly monomorphic, yet the shared site sees the union of all their types and goes megamorphic. The helper then compiles badly for everyone, including the one caller whose hot loop depends on it.

The fix is not clever: give the hot path its own copy of the code so it gets its own profile. Duplicating a small helper for the one caller that matters is a legitimate optimization, and one of the few places where writing slightly worse-looking Java produces meaningfully better machine code.

Speculation, class hierarchy analysis, and uncommon traps

The monomorphic guard above is one instance of a general strategy: assume the profile describes the future, compile aggressively on that assumption, and install something that bails out if it turns out not to.

Class hierarchy analysis is the strongest version. If an interface or abstract method currently has exactly one loaded implementation, there is no ambiguity at all and the compiler can devirtualize with no runtime guard whatsoever. What it does instead is record a dependency: this compiled method assumes no second implementor exists. If a class loader later produces one, the JVM invalidates every compiled method carrying that dependency. The assumption is enforced by the class loading machinery rather than by a check in the hot path, which makes it free right up until the moment it is not.

Uncommon traps handle the branch case. If a branch has never been taken during profiled execution, the compiler does not generate code for it. It generates a trap - an instruction that, if reached, throws execution out of the compiled method entirely. The never-taken branch then costs nothing at all: no code, no register pressure, no interference with block layout, and the compiler is free to treat everything downstream as though that path did not exist, folding constants and eliminating checks accordingly.

The same machinery covers a null check that has never seen null, a cast that has never failed, an array store that has never had the wrong element type, and an arithmetic path that has never overflowed. In each case the fast path is the observed one and the unobserved one is a trap.

This is what lets a warmed JVM beat ahead-of-time compiled code on abstraction-heavy programs, because an AOT compiler has no observations and no interpreter to bail out into, so it cannot speculate at all. The GraalVM article covers that comparison, and covers Graal as an alternative top tier: it plugs into HotSpot in place of C2 and changes the inlining and escape-analysis policies described here, while the tiers, counters, deoptimization, and code cache remain exactly the machinery on this page.

End-to-end flow

End-to-end: JVM starts. Interpreter runs the request path. Once the method's counters cross the tier-3 threshold, C1 compiles it with profiling and latency drops sharply. Once they cross the tier-4 threshold, C2 recompiles with aggressive inlining and escape analysis, and latency drops again. Warmup takes 2 minutes. Later, a call site that was monomorphic (always Impl1) sees Impl2 for the first time; deopts back to interpreter briefly then recompiles polymorphically. Ops watches PrintCompilation to spot patterns and sizes code cache to keep JIT active.

Deoptimization: what it costs and why it is silent

Every speculation above needs a way out, and that way out is deoptimization: discarding compiled code and handing execution back to the interpreter in the middle of a method.

The mechanism is more delicate than it sounds. The running thread is somewhere inside optimized machine code whose relationship to the original bytecode is not obvious - callees have been inlined into it, locals live in registers, objects may have been scalar-replaced out of existence entirely. To hand control back, the JVM consults metadata the compiler emitted alongside the code, describing for each possible bailout point the full interpreter state: which bytecode index in which method, and what belongs in each local slot and on each operand stack. From that it reconstructs a stack of interpreter frames, one per inlined method, materializes any object the optimizer had dissolved, and resumes interpreting. This runs at a safepoint, which is why a burst of deoptimization shows up in a pause log looking structurally identical to garbage collection.

The compiled method is then marked not entrant so that no new call enters it, threads already inside drain out, and the code is eventually reclaimed. The method drops to a lower tier, re-profiles with the new evidence, and is recompiled - normally without the assumption that just broke.

Why a healthy process can quietly get slower

A single deoptimization is cheap. A pattern of them is not, and the JVM's defence against that pattern is itself the failure mode. It counts traps per method and per bytecode index, and when one site has bailed out too many times it stops trusting speculation there and recompiles without it, permanently, for the remaining life of the process. If the churn continues, the method can be marked not compilable at that tier and simply left running at a lower one.

That is the mechanism behind the report that a service was fast for hours and then was not, with no deploy, no configuration change, and no traffic shift large enough to explain it. Typical causes: a second implementation of an interface loaded lazily hours in, invalidating a class-hierarchy dependency across a lot of compiled code at once; a traffic mix change that starts taking a branch the profile said never fired; an error path exercised for the first time in production, deoptimizing the hot method that encloses it; a plugin, script engine, or dynamically generated proxy appearing late and turning a monomorphic call site megamorphic. Nothing logs an error. Throughput steps down and stays there.

The signal lives in the compilation log rather than in application metrics: made not entrant lines appearing steadily rather than only during warmup, and the same method being recompiled again and again. A recompilation loop - compile, deoptimize, recompile - burns compiler CPU continuously and is worth alerting on directly.

On-stack replacement, and what it does to benchmarks

Standard compilation only helps the next call. A method entered once and still executing - main, a batch job's outer loop, a polling loop - would never benefit, because there is no next call and the frame currently on the stack is an interpreted one.

On-stack replacement solves that. When the backedge counter trips, the JVM compiles a version of the method specialized to a single entry point: the loop's bytecode index. It builds an entry that accepts the live interpreter frame's state, transplants those locals into a compiled frame, and jumps into the middle of the compiled loop. Execution continues in native code without the method ever having returned.

OSR code is a compromise. It is compiled for one bytecode index and cannot serve as a normal entry point, so a separate standard compilation happens later if the method is ever called again. Everything live at loop entry arrives as an incoming value rather than as something the compiler derived and can reason about, so the optimizer has less to work with. And the pre-loop part of the method is generated but unreachable in that version.

This matters more for measurement than for production. A hand-rolled benchmark that puts its workload inside one long loop in main is measuring OSR code. Real callers invoke the method normally and get the standard compilation, which is a different and usually better body of machine code. That gap, together with tier transitions, is exactly why JMH forks a fresh JVM, runs warmup iterations, and calls the benchmark method rather than looping inside it. A number produced by wrapping a loop in System.nanoTime() is not measuring what production will run.

The code cache is a bounded resource

Compiled methods live in the code cache: a fixed-size native region reserved at startup, entirely separate from the heap. It also holds the interpreter itself, adapters, stubs, and intrinsics. Modern JVMs segment it into non-method code, profiled code, and non-profiled code, which is good for instruction locality and is also an operational trap, because one segment can fill while the others still have room.

A sweeper reclaims space, evicting methods made not entrant by deoptimization and code that has gone cold. Normally it keeps up. When it cannot, the JVM prints a warning that the code cache is full and the compiler has been disabled, and that is precisely what it means. Compilation stops. Already-compiled methods keep running, but nothing new is compiled, nothing is recompiled after a deoptimization, and any method that becomes hot afterwards stays interpreted. The process does not crash and does not recover; it drifts progressively closer to interpreter speed as its compiled code is invalidated and never replaced.

What fills it: very large codebases and heavy framework stacks; aggressive inlining, since an inlined callee's code is duplicated into every caller it is inlined into; large numbers of distinct lambdas and method handles, each call site materializing its own class; and dynamically generated code from proxies, ORM layers, scripting engines, and instrumentation agents. Agents are a common surprise, because they add code the application's authors never wrote and never see.

-XX:ReservedCodeCacheSize raises the ceiling and -XX:+PrintCodeCache reports usage at exit, but this is better monitored than diagnosed after the fact. The JVM exposes code cache occupancy as a memory pool through the standard management interface, so it belongs on the same dashboard as heap and metaspace.

Warmup is a production concern

Everything above adds up to a process whose performance is a function of how long it has been running and what it has seen, and that has direct operational consequences.

A freshly started instance placed straight into a load balancer's rotation at full weight serves its first requests interpreted, at latencies that can be one or two orders of magnitude worse than steady state, while simultaneously spending scarce CPU on compilation. During a rolling deploy this appears as a p99 spike that tracks the rollout and vanishes when it completes. The mitigations are ordinary: readiness checks that do not pass until the instance has done real work, load balancer slow-start or gradual weight ramping, and a startup routine that exercises the main request paths before advertising readiness.

Warmup traffic has to resemble real traffic. Because the profile is what the compiler speculates on, warming with a narrow synthetic workload builds a profile shaped like that workload, and the first real request that disagrees deoptimizes the methods you just warmed. Driving one request type through a polymorphic handler is a reliable way to hand production a set of assumptions that break immediately.

If the warmup window itself is the problem rather than the traffic shaping, the options are architectural rather than tuning flags. Class-Data Sharing removes class loading and verification from startup while leaving the JIT intact; CRaC restores a process image checkpointed after warmup, so compiled code and accumulated profiles survive into the new process; GraalVM Native Image removes the JIT entirely and pays for it in peak throughput. They are three points on the same trade.

Reading what the JIT actually did

Speculation about JIT behavior is cheap and usually wrong. The tooling is good enough that you should not have to guess.

-XX:+PrintCompilation is the low-effort starting point: one line per compilation event carrying the compile identifier, the tier, the method, and flag characters worth learning. % marks an OSR compilation, s a synchronized method, ! a method with exception handlers, n a native wrapper, b a blocking rather than background compile. Lines reading made not entrant are the deoptimization trail. What you are looking for is not individual lines but the overall shape: does the stream quieten down after warmup, or does the same method keep reappearing forever?

-XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining answers the question that usually matters - why a particular call was not inlined - and prints a reason, such as the callee being too large, the site being megamorphic, or the callee never having executed often enough to qualify. It is the fastest way to confirm that an abstraction boundary is breaking a hot path.

For anything deeper, -XX:+LogCompilation writes a structured XML log that JITWatch consumes, giving inlining trees, deoptimization events, and - with hsdis installed - the generated assembly per method. That is the right tool for "show me exactly what this loop compiled to".

Pair all of it with a sampling profiler, because a compilation problem is only worth fixing if the method is where time actually goes. async-profiler resolves JIT-compiled frames and shows the real distribution; JFR records compilation and code cache events continuously and is the practical option for catching a deoptimization storm that only occurs in production at four in the morning.