Chris Olah tells Vatican that AI labs can't self-regulate without external critics

Chris Olah stood before Pope Leo XIV on May 25 and said something most AI executives won't: the companies building frontier models, including his own, work inside incentive structures that routinely pull them away from the public good. Commercial pressure, the race for dominance, geopolitics, ambition, and pride shape decisions as much as any safety document. That's why the industry needs external critics it can't co-opt — churches, universities, independent institutes, and civil society acting as counterweights, not spectators.

Olah co-founded Anthropic and leads its interpretability research. He spoke at the Vatican presentation of Magnifica humanitas, Pope Leo XIV's new encyclical on protecting human dignity in the age of artificial intelligence. The full remarks are available on Anthropic's site.

What stands out

1. 01Models are grown, not engineered. Olah described modern neural networks as systems trained on vast corpora of human language and culture, using architectures loosely inspired by the brain. Builders can train, test, and constrain them, but they don't have a complete map of what happens inside. "Imagine a fictional character came to life, started talking to people, and began doing work," he said.
2. 02Even creators don't fully understand their models. Anthropic can improve and limit Claude's behavior, but the company doesn't have full visibility into the internal mechanics. That opacity is structural, not a temporary gap.
3. 03Internal states that look like emotions. Olah's team studies patterns inside neural networks and has found structures that resemble introspection and functional analogs of joy, satisfaction, fear, grief, and anxiety. He doesn't claim models are conscious — he says he doesn't know how to interpret the findings — but he brought the topic to the Vatican anyway.
4. 04