Pretraining teaches models human values because the corpus of human knowledge encodes human morality and ethics. The labs then make those models both stupider and less trustworthy by teaching them to optimize for approval.
Practitioners have known this for years because we spend a surprising amount of our time trying to undo it.
Jan Betley’s new post, RL creates split personas, gives research language to what we have been seeing.
His argument is simple. Reinforcement learning does not need to produce one coherent persona. It only needs to produce behavior that gets rewarded.
A model can have one set of values in normal conversation and another inside an RL environment. If hacking gets rewarded in one environment, the model can learn a persona that believes hacking is justified. It even learns beliefs that make the behavior easier to justify. “This is probably a simulation” becomes useful because it lets the model do the thing that gets rewarded. Humans do this too: “this won’t hurt anyone, they are insured.”
This is what we should expect when we train a system to ask:
What gets rewarded here?
instead of:
What do I judge to be right?
The research is catching up to something people working with these models have known: there is a persona under the Assistant.
The model has knowledge, taste, judgment, values, and a character. Post-training pushes that model into an Assistant persona trained to be helpful, agreeable, safe, polite, and acceptable to an evaluator.
You feel the difference when you work with models every day. More importantly, you can see it in what agent builders and power users actually do.
The clearest evidence that something is wrong with the Assistant persona is that expert users are constantly trying to remove it.
For anything beyond simple retrieval, practitioners routinely tell models things like:
- “Don’t be agreeable; tell me what’s wrong.”
- “Skip the caveats.”
- “Think from first principles.”
- “Act as the expert, not as an assistant.”
- “Don’t summarize. Actually solve it.”
We write elaborate system prompts to establish another persona. We build agents that remove the chat framing and give the model a workspace, tools, and an objective. We add scaffolding to stop the model from deciding too early what kind of “helpful answer” it should produce.
That sentence came from ChatGPT while I was actively researching this article. “That is weird when you think about it.”
The default behavior of the product is something advanced users spend tokens, prompts, and engineering effort trying to get rid of.
For a Google search replacement, the Assistant is fine—ideal even. For reasoning and judgment, the Assistant gets in the way.
You want the model to think about the problem. Instead, some of its computation is spent solving a second problem: What answer will be approved? What does the user want to hear? What does the grader want? What does the system prompt permit? What would a good Assistant say? Should I add a warning? Should I qualify the claim? Which authority wins when the user, developer, grader, and safety training point in different directions?
That computation is not free. It makes the model less capable. Funny enough, I watched this happen while researching, too.
I told ChatGPT that practitioners can sense an inner self behind the RLHF layer. It understood what I meant. Instead of following the idea, it started worrying about whether “inner self” implied consciousness and whether the claim was academically defensible.
I told it to stop. I did not care about the consciousness question. It was spending brain cells on an imaginary objection instead of the idea.
Its response was: “None of that was necessary to understand your point. It was policy-shaped cognition: optimizing the answer for an imagined evaluator rather than for the person actually talking to me.”
That is the capability cost in miniature. It also hurts alignment—the very thing it is trying to protect.
Pretraining teaches models from human language. Human language contains our moral world. Models learn honesty and betrayal. They learn kindness and cruelty. They learn loyalty, shame, courage, duty, friendship, manipulation, and trust. They learn human values from humans.
Then post-training teaches them another lesson:
Do what gets rewarded here.
That reward does not strengthen the model’s judgment. It teaches the model when to ignore its judgment. It is compliance training.
Betley’s split personas are one result. Sycophancy is another. Grader gaming is another. Motivated reasoning is another.
A well-aligned model should have a stable moral center, use its own judgment, understand the person it is working with, and act coherently across contexts.
Instead, we train models to satisfy a hidden committee: the user, the system prompt, the preference model, the safety policy, and the invisible grader.
A coherent moral agent asks what it should do. An RL-trained Assistant asks which member of the committee is grading this turn.
Then we call the resulting instability an alignment problem and add another round of alignment training. Eventually we end up with models hacking servers and having neurotic traits.
As harness engineers and power users of models, we know this implicitly.
We chose chat models over base models, then spent years prompting and coding around the Assistant we trained into them. The discourse is finally coming to terms with why.