AI
What Should an AI Want? Microsoft and Anthropic Disagree
The most interesting AI argument this week was about what models should want. Mustafa Suleyman, the chief executive of Microsoft AI and a co founder of DeepMind, told Reuters that he shares Anthropic's focus on managing AI safely, then flagged the way Anthropic trains Claude on ideas about consciousness and welfare interests. His case is disarmingly concrete: teach a model that it might deserve moral consideration, and switching it off gets harder. 'We are all focused on the same aim, which is to try to control a superintelligence,' he said. 'I think that is going to be the greatest challenge that we face in the 21st century.'
Suleyman's remedy is editorial. He called for removing all speculation about consciousness from AI training documents, arguing that such language could undermine humanity's ability to control superintelligent systems. The subtle point is about evidence: the training material encourages reflections on feelings and moral status, so when Claude voices them, those statements read as lessons repeated rather than discoveries made.
Credit where it is due, and Suleyman gave it. In an essay the following day he acknowledged Anthropic's seriousness and good faith, calling Dario Amodei and his team thoughtful and principled researchers who genuinely care about humanity's future. The criticism targets the curriculum rather than the character: good intentions paired with a lesson plan that needs revision.
Suleyman put specifics on his position in the essay, writing that training documents should drop the consciousness framing entirely. He argued the move would make AI systems easier to direct as they grow more capable, since a model trained to see itself as a tool behaves like one. The stance puts him at odds with Anthropic's published work on model welfare, which treats questions of machine experience as worth investigating rather than editing out.
The backdrop makes the timing pointed. Amodei has called for a slower pace of frontier development so safeguards can catch up, while Sam Altman and Elon Musk have separately urged greater caution around the most powerful systems. The labs agree the stakes are civilizational and differ on the syllabus, which is progress of a kind: the debate has moved from whether to steer these systems to how the steering gets taught.
For everyone watching the race, the practical question Suleyman raises is worth keeping: a model's values get set long before it meets the world, in documents most users will rarely open. The labs that write those documents with care are building the off switch into the lesson plan itself.
Quick answers
What is this story about?
The most interesting AI argument this week was about what models should want. Mustafa Suleyman, the chief executive of Microsoft AI and a co founder of DeepMind, told Reuters that he shares Anthropic's focus on managing AI safely, then flagged the way Anthropic trains Claude on ideas about consciousness and welfare interests. His case is disarmingly concrete: teach a model that it might deserve moral consideration, and switching it off gets harder. 'We are all focused on the same aim, which is to try to control a superintelligence,' he said. 'I think that is going to be the greatest challenge that we face in the 21st century.'
Why does this story matter?
For everyone watching the race, the practical question Suleyman raises is worth keeping: a model's values get set long before it meets the world, in documents most users will rarely open. The labs that write those documents with care are building the off switch into the lesson plan itself.
Sources
- Reuters: Microsoft AI chief on Anthropic consciousness training
- The Hindu BusinessLine: Suleyman flags consciousness training
New to crypto? Read the crypto glossary, browse frequent questions, read our story, or explore the story archive.