Constitutional AI
The 2022 Anthropic paper that introduced a method to align models using AI-generated feedback rather than human labels, scaling alignment securely and transparently.
Paper: Constitutional AI: Harmlessness from AI Feedback
Authors: Yuntao Bai, Saurav Kadavath, Saurabh Garg, Amanda Askell, Dawn Drain, Melissa Rubskova, Stanislav Fort, Tim Maxwell, Jackson Kernion, Nicholas Schiefer, Ben Mann, Tom Conerly, Sam Bowman, Deep Ganguli, Jared Kaplan, Colin Hernandez, Andy Jones, William Fedus · 2022
Read the paper
The Problem
Aligning large language models (making them helpful and harmless) relied heavily on Reinforcement Learning from Human Feedback (RLHF). This had severe limitations: humans are expensive, slow, and often disagree on what is "harmless." Furthermore, human labeling is a "black box"—you don't always know why a human preferred one answer over another. As models became more capable, relying entirely on human supervision to catch subtle safety issues became a critical bottleneck.
The Idea
Anthropic proposed Constitutional AI (CAI), replacing human feedback with AI feedback (RLAIF). Instead of humans rating outputs, humans write a "Constitution"—a list of explicit principles (e.g., "Choose the response that is less harmful," "Avoid racist tropes"). The AI is then prompted to evaluate its own outputs, or the outputs of other models, against this constitution. This makes the alignment process transparent (the rules are written in plain English) and infinitely scalable (the AI generates its own training data).
How It Works
The CAI process has two main phases:
- Supervised Phase (Self-Critique): The base model is prompted with a harmful request and generates a (likely harmful) response. It is then shown a principle from the Constitution and asked to critique its own response. Finally, it rewrites the response to conform to the principle. The model is fine-tuned on these self-revised, harmless responses.
- RL Phase (AI Feedback): The model generates two different responses to a prompt. An AI model (acting as a judge) is shown the Constitution and asked which response better adheres to the principles. This AI-generated preference data is used to train a Reward Model, which then optimizes the main model via standard PPO.
Why It Mattered
Constitutional AI proved that models had reached a level of capability where they could effectively supervise themselves. It solved the scaling bottleneck of RLHF, allowing Anthropic to align the Claude family of models faster and cheaper than competitors, while maintaining a rigorous, transparent safety posture.
What Came After
RLAIF (Reinforcement Learning from AI Feedback) became a standard tool used by all frontier labs. Generating synthetic preference data via strong "judge" models (like GPT-4 or Claude 3 Opus) is now the primary method for aligning open-source models, replacing expensive human data collection.