2022
Training Language Models to Follow Instructions with Human Feedback
RLHF: how a raw text predictor was taught to be helpful, honest, and willing to say no.
“Making language models bigger does not inherently make them better at following a user's intent.”
The alignment gap
Larger models predict text well but often produce outputs users do not want: unhelpful answers, hallucinations, toxic completions. Ouyang et al. separate capability from intent alignment.
RLHF pipeline
The method has three stages: supervised fine-tuning on demonstration data, training a reward model from human comparisons, then optimizing the policy with reinforcement learning (PPO) against that reward. Human labelers rank outputs; the model learns the ranking.
Why it mattered
InstructGPT became the template for ChatGPT-era assistants. RLHF is now standard, and also controversial: it encodes labeler preferences, hides failure modes behind polished tone, and adds training complexity. The paper names the tradeoff clearly.
- collection
The idea lineage of the model you talked to this morning, in reading order.
← previous · 2022
Inner Monologue: Embodied Reasoning through Planning with Language Models
next · 2023 →
Sparks of Artificial General Intelligence: Early experiments with GPT-4