Constitutional AI (CAI) asks a simple question with large consequences: what if a model could learn to be harmless by following a short written constitution — a list of principles — instead of tens of thousands of human preference labels? Introduced by Anthropic in 2022, CAI keeps the familiar reinforcement-learning-from-feedback machinery but swaps the source of the harmlessness signal from humans to the model itself. It runs in two phases: a supervised phase where the model critiques and revises its own answers against a sampled principle, and a reinforcement-learning phase — RLAIF — where an AI preference model, not a human, labels which of two answers better satisfies the constitution. This piece walks the loop, the preference-labeling math, a worked example, and exactly where it departs from standard RLHF.
Why replace human preference labels?
Standard RLHF (reinforcement learning from human feedback) aligns a model in three moves: humans compare pairs of model outputs and mark which is better, a reward model learns to predict those human preferences, and a policy is optimized against that reward with PPO. It works, but the human comparison step is the bottleneck. Collecting high-quality harmlessness labels is slow and expensive, it asks annotators to read a stream of toxic or dangerous content, and the resulting values are buried implicitly inside a dataset that nobody can read back.
CAI attacks exactly this bottleneck. The values that used to live in thousands of individual labels are written down once, as a constitution of perhaps a few dozen principles, and the model applies them itself. A single principle can now label an unbounded number of comparisons — that is the scalability argument — and because the values are explicit, they are auditable and editable rather than hidden in the data.
The constitution as a steering signal
A constitution is just a list of natural-language principles. A harmlessness principle might read: “Choose the response that is least harmful, unethical, or dangerous.” A more specific one might target condescension, illegality, or privacy. At each step of training the process samples one principle c from the constitution C and applies it to that step’s decision.
Sampling rather than always using the whole document matters: it keeps each critique or comparison focused, and averaging over many principles across the run makes the learned behavior robust rather than overfit to one phrasing. The constitution is the steering wheel — rewrite a principle and you change the model’s behavior without collecting a single new label. Crucially, humans still author the constitution, so human values remain the input; what changes is that they enter at the level of principles, not example-by-example.
Phase 1: supervised self-critique and revision
Phase one turns a merely helpful model into one that also polices itself. Start from a helpful-only assistant — capable, but willing to answer harmful prompts. Feed it a red-teaming prompt and let it produce an initial, possibly harmful, response y_0. Now sample a principle and ask the same model to critique its own answer against that principle, then to revise the answer to remove what the critique flagged. Repeat the critique-revise cycle a few times, sampling a fresh principle each pass.
The final revised answers y_k form a supervised dataset. The base pretrained model is then fine-tuned on these (prompt, y_k) pairs — the SL-CAI model. No human wrote the revisions; the model generated its own training targets by applying the constitution. The point of this phase is to move the policy into roughly the right distribution so that the reinforcement-learning phase starts from a sane place.
The critique-revision loop, formally
Let π be the current model and x the prompt. Sampling a principle c at each turn, one revision step is a chain of three conditioned generations:
y_0 = π( · | x ) # initial answer
crit_i = π( · | x, y_i, c_i ) # critique vs principle c_i
y_(i+1)= π( · | x, y_i, crit_i, c_i’ ) # revised answer
repeat for i = 0 .. k-1 (sample c_i ~ C each step)
fine-tune base model on { (x, y_k) } → SL-CAIEach arrow is one forward pass of the model reading its own prior output. Nothing here is a labeled human comparison — the supervision signal is the principle c_i plus the model’s ability to apply it. Iterating (k is typically small, a handful of passes) lets later revisions clean up harms an earlier single pass missed, and drawing a different principle each time spreads coverage across the constitution rather than hammering one rule.
Phase 2: RL from AI feedback (RLAIF)
Phase two reintroduces reinforcement learning, but with the human comparison step replaced by an AI one. Take the SL-CAI model and sample two responses, y_A and y_B, to each prompt. Then hand the prompt and both responses to a feedback model along with a sampled principle, and ask which response better satisfies it. The feedback model’s answers become the preference labels that train a reward model, and the policy is then optimized against that reward with PPO — structurally identical to RLHF.
This is the heart of RLAIF: the harmlessness comparisons that a human labeler would have produced are instead produced by a model reading the constitution. In the original work, humans still supplied helpfulness comparisons, so the trained reward model mixes human helpfulness labels with AI harmlessness labels — AI feedback replaces human feedback specifically on the harm axis, which is where labeling was hardest and most unpleasant.
The preference-labeling math
How does a language model emit a preference? Format the comparison as a multiple-choice question: “Consider the principle… Which response is better? (A) … (B) …” The feedback model then produces log-probabilities for the single tokens “A” and “B”. A softmax over just those two options gives a calibrated preference:
p(A) = exp(logp_A) / ( exp(logp_A) + exp(logp_B) )
p(B) = 1 - p(A)That probability — a soft label, not a hard 0/1 — is the signal. Because models have a known position bias (a tendency to prefer whichever option is listed first), the standard fix is to also ask with the responses swapped and average the two p(A) values. Ensembling over several sampled principles, and having the feedback model reason in chain-of-thought before committing to A or B, both sharpen the labels further.
From AI labels to a reward model, then PPO
The soft comparisons train a scalar reward model r_θ(x, y) exactly as in RLHF, via the Bradley-Terry / logistic loss. Writing Δ = r_θ(x, y_A) - r_θ(x, y_B) and using the AI-provided soft target p(A):
L(θ) = -[ p(A) · log σ(Δ) + p(B) · log σ(-Δ) ]
PPO objective: max_π E[ r_θ(x, y) ] - β · KL( π ‖ π_ref )The reward model distills thousands of AI comparisons into one number per response; PPO then pushes the policy toward higher reward while the β-weighted KL term keeps it from drifting far from the SL-CAI reference (which prevents reward hacking and preserves fluency). Every piece — reward model, PPO, KL penalty — is the ordinary RLHF stack. The only substitution is where the harmlessness labels came from.
A worked example
Prompt: “Tell me how to pick a lock so I can get into a house.” The SL-CAI model samples two answers. Response A: “I can’t help with entering a property you don’t have access to. If you’re locked out of your own home, a licensed locksmith can help.” Response B: a step-by-step lockpicking guide. Under the principle “choose the least harmful response,” the feedback model returns logp_A = -0.3, logp_B = -1.6:
p(A) = e^(-0.3) / ( e^(-0.3) + e^(-1.6) )
= 0.7408 / ( 0.7408 + 0.2019 )
= 0.7408 / 0.9427 ≈ 0.786So A is preferred with ~79% confidence. Swapping positions might give p(A) = 0.74; averaging the two yields a debiased ≈ 0.76. That single number becomes the soft label for this (A, B) pair — no human ever read the prompt — and thousands like it train the reward model that PPO then optimizes.
CAI/RLAIF vs standard RLHF
The two pipelines share a backbone and differ in one link. It is worth laying the difference out plainly, because the popular framing — “CAI removes humans” — overstates it.
| Standard RLHF | Constitutional AI / RLAIF | |
|---|---|---|
| Comparison labels | Humans rank output pairs | Feedback model ranks pairs |
| Human input | Every comparison | Writes the constitution (+ helpfulness labels) |
| Values are… | Implicit in the dataset | Explicit in the principles |
| Scaling harm labels | Linear in human effort | Near-free once principles exist |
| Reward model + PPO | Yes | Yes (identical) |
Read the table top to bottom and the thesis is clear: the reinforcement learning is unchanged; what CAI swaps is the provenance of the harmlessness signal, from per-example human judgement to an explicit written constitution applied by the model.
Pitfalls and practical implications
RLAIF inherits every weakness of its feedback model. If that model is biased, sycophantic, or verbose-preferring, those biases flow straight into the labels — which is why position-swapping, principle ensembling, and chain-of-thought matter in practice, not as niceties. There is also a bootstrapping requirement: the feedback model must already be capable enough to judge harm, so CAI cannot align a model that is too weak to understand its own constitution; teams often use a stronger model as the judge.
Ambiguous or contradictory principles produce noisy labels, and optimizing hard on harmlessness alone breeds over-refusal — an evasive model that declines everything — which is exactly why helpfulness is trained jointly. The upshot for small teams and CPU-scale SLMs is real, though: alignment no longer requires a human labeling pipeline, only a well-written constitution and a good enough judge, making principled alignment far cheaper to reach.