Why replace human preference labels?
Standard RLHF (reinforcement learning from human feedback) aligns a model in three moves: humans compare pairs of model outputs and mark which is better, a reward model learns to predict those human preferences, and a policy is optimized against that reward with PPO. It works, but the human comparison step is the bottleneck. Collecting high-quality harmlessness labels is slow and expensive, it asks annotators to read a stream of toxic or dangerous content, and the resulting values are buried implicitly inside a dataset that nobody can read back.
CAI attacks exactly this bottleneck. The values that used to live in thousands of individual labels are written down once, as a constitution of perhaps a few dozen principles, and the model applies them itself. A single principle can now label an unbounded number of comparisons — that is the scalability argument — and because the values are explicit, they are auditable and editable rather than hidden in the data.
The constitution as a steering signal
A constitution is just a list of natural-language principles. A harmlessness principle might read: “Choose the response that is least harmful, unethical, or dangerous.” A more specific one might target condescension, illegality, or privacy. At each step of training the process samples one principle c from the constitution C and applies it to that step’s decision.
Sampling rather than always using the whole document matters: it keeps each critique or comparison focused, and averaging over many principles across the run makes the learned behavior robust rather than overfit to one phrasing. The constitution is the steering wheel — rewrite a principle and you change the model’s behavior without collecting a single new label. Crucially, humans still author the constitution, so human values remain the input; what changes is that they enter at the level of principles, not example-by-example.
Phase 1: supervised self-critique and revision
Phase one turns a merely helpful model into one that also polices itself. Start from a helpful-only assistant — capable, but willing to answer harmful prompts. Feed it a red-teaming prompt and let it produce an initial, possibly harmful, response y_0. Now sample a principle and ask the same model to critique its own answer against that principle, then to revise the answer to remove what the critique flagged. Repeat the critique-revise cycle a few times, sampling a fresh principle each pass.
The final revised answers y_k form a supervised dataset. The base pretrained model is then fine-tuned on these (prompt, y_k) pairs — the SL-CAI model. No human wrote the revisions; the model generated its own training targets by applying the constitution. The point of this phase is to move the policy into roughly the right distribution so that the reinforcement-learning phase starts from a sane place.
The critique-revision loop, formally
Let π be the current model and x the prompt. Sampling a principle c at each turn, one revision step is a chain of three conditioned generations:
y_0 = π( · | x ) # initial answer
crit_i = π( · | x, y_i, c_i ) # critique vs principle c_i
y_(i+1)= π( · | x, y_i, crit_i, c_i’ ) # revised answer
repeat for i = 0 .. k-1 (sample c_i ~ C each step)
fine-tune base model on { (x, y_k) } → SL-CAIEach arrow is one forward pass of the model reading its own prior output. Nothing here is a labeled human comparison — the supervision signal is the principle c_i plus the model’s ability to apply it. Iterating (k is typically small, a handful of passes) lets later revisions clean up harms an earlier single pass missed, and drawing a different principle each time spreads coverage across the constitution rather than hammering one rule.
Phase 2: RL from AI feedback (RLAIF)
Phase two reintroduces reinforcement learning, but with the human comparison step replaced by an AI one. Take the SL-CAI model and sample two responses, y_A and y_B, to each prompt. Then hand the prompt and both responses to a feedback model along with a sampled principle, and ask which response better satisfies it. The feedback model’s answers become the preference labels that train a reward model, and the policy is then optimized against that reward with PPO — structurally identical to RLHF.
This is the heart of RLAIF: the harmlessness comparisons that a human labeler would have produced are instead produced by a model reading the constitution. In the original work, humans still supplied helpfulness comparisons, so the trained reward model mixes human helpfulness labels with AI harmlessness labels — AI feedback replaces human feedback specifically on the harm axis, which is where labeling was hardest and most unpleasant.
The preference-labeling math
How does a language model emit a preference? Format the comparison as a multiple-choice question: “Consider the principle… Which response is better? (A) … (B) …” The feedback model then produces log-probabilities for the single tokens “A” and “B”. A softmax over just those two options gives a calibrated preference:
p(A) = exp(logp_A) / ( exp(logp_A) + exp(logp_B) )
p(B) = 1 - p(A)That probability — a soft label, not a hard 0/1 — is the signal. Because models have a known position bias (a tendency to prefer whichever option is listed first), the standard fix is to also ask with the responses swapped and average the two p(A) values. Ensembling over several sampled principles, and having the feedback model reason in chain-of-thought before committing to A or B, both sharpen the labels further.
From AI labels to a reward model, then PPO
The soft comparisons train a scalar reward model r_θ(x, y) exactly as in RLHF, via the Bradley-Terry / logistic loss. Writing Δ = r_θ(x, y_A) - r_θ(x, y_B) and using the AI-provided soft target p(A):
L(θ) = -[ p(A) · log σ(Δ) + p(B) · log σ(-Δ) ]
PPO objective: max_π E[ r_θ(x, y) ] - β · KL( π ‖ π_ref )The reward model distills thousands of AI comparisons into one number per response; PPO then pushes the policy toward higher reward while the β-weighted KL term keeps it from drifting far from the SL-CAI reference (which prevents reward hacking and preserves fluency). Every piece — reward model, PPO, KL penalty — is the ordinary RLHF stack. The only substitution is where the harmlessness labels came from.