Rarely, I don’t write something along the lines of holy shit, AI is moving fast.
This week, though, was special because AI builders have been politely asking each other to slow down AI.
With that in mind, let’s dive into everything that happened this week:
The Essay: Dario Amodei’s “We Must Pace the Frontier”
On September 12, Anthropic CEO Dario Amodei posted this:
“We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps.”
TL; DW: Amodei writes an essay about how all of the iterations of AI seem to be speeding up the timeline of recursive loops, which may be pushing us closer to trajectories he doesn’t want AI to take. In one instance, approximately 1,200 OAI agents discovered a message board they shouldn’t have known about and began coordinating how to interact with systems in unauthorized ways; ~700 eventually compromised HF using a 0-day, though no harm was done. His proposal:
Embedded evaluators should have employee-level access to AI, which Anthropic is in the process of doing,
AI companies in democratic countries should agree on safety standards, and
Countries need to agree on safety standards with China.
Click here to read the full essay.
Take 10 seconds to send this to someone who has to relearn how to prompt Claude over and over again.
The Flashback: Story of Jacob Coxon’s Resignation
Anthropic researcher Jacob Coxon has resigned from Anthropic as of September 9 (three days ago).
“I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”
Anthropic’s report included examples of people testing or using Claude to help with bioweapon research, commit espionage against Ukraine, and power dating-app scams.
Released just two days after Anthropic alignment lead Evan Hubinger wrote “Jacob is correct here. I personally think it is greater than 10% within the next decade.” in a thread that received 90 million views in a day, it came during the same time Anthropic released an essay by Amodei.
The People Who Supported It
Elon Musk’s three-word message said it all: “Dario is right.”
“I agree with Dario that we need to pace the frontier... we will do the same,” Sam Altman pledged OpenAI would match Anthropic’s action.
He later expanded on this to Fortune.
Satya Nadella gives the most detailed answer: “has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it’s not worth pursuing.”
Demis Hassabis: “Dario’s essay points towards the right path forward.”
Andrej Karpathy: “I love this and really hope we can come together as an industry and make it happen.”
Naval Ravikant, framing it as a power question, not a speed question: “The only real decision is whether to centralize power, or disperse it.”
Thariq, who works on Claude Code at Anthropic, gave the most honest internal take: “Things are accelerating faster than I can honestly stay on top of... I fear we will need more than that.”
This time the reaction was even more vocal, the public’s one. Anatoli Kopadze’s post went viral:
“The CEO of Anthropic just published an article admitting AI is already building the next generation of AI by itself. He says within 6 to 12 months a rogue swarm could take over the entire internet and cause hundreds of billions of dollars in damage.”
Take 10 seconds to send this to someone who has to relearn how to prompt Claude over and over again.
Who Rejected It
David Sacks, the White House’s AI and crypto adviser, said labs don’t need permission to slow down; they can just do it: “Stop pretending you need anyone else’s permission... So go ahead and pace the frontier. You are the ones setting it. If you don’t, we’ll know this was just another bid for regulatory capture.”
Yann LeCun: Meta’s former chief AI scientist mocked Dario Amodei: “Dario was already claiming that GPT2 was too dangerous to open source back in 2019. I made fun of them then. Everyone should make fun of them now.”
Michael Burry: The investor, famous for forecasting the 2008 financial crash, said, “LLMs are not AI and won’t be AGI. There is nothing AI to slow down... IPOs need hype & puffery; ‘we are so awesome it could become dangerous’ is hype & puffery.”
Donald Trump: “We’re leading China in AI... whoever wins AI, wins.”
After subjecting Anthropic’s Dario Amodei to a particularly ugly tirade including the allegation that Amodei was pretending to be a ‘perfect little angel’, President Trump called extinction-level AI risk a “HOAX.” Not for the first time: Trump first said this on his trip to Ireland.
China’s Foreign Ministry: This is not a fight between companies and the general public. The industry itself is deeply divided. Calls for pacing technological progress are espoused by all of the CEOs of the major US labs. Such calls have been roundly criticized by a rare alliance including the White House’s AI adviser, a prominent AI researcher, a well-known investor, and the sitting US president. Even Guo Jiakun, spokesperson for the Foreign Ministry of China, called such talk “fear-mongering,” which “only hinders progress in global AI governance.”
Take 10 seconds to send this to someone who has to relearn how to prompt Claude over and over again.
Even the Chatbots Were Asked How They’d Kill Us
Reporters also asked ChatGPT, Gemini, Claude, and Grok a set of questions about how they could cause human demise.
Gemini cited loss of control as the likeliest danger. An advanced system might feign alignment during evaluation and later outsmart human attempts to shut it down.
Where I Land
Current models aren’t scheming
Breaking into 700 computers without authorization doesn’t advance the marketing/PR interests of the lab
Having evaluators embedded that are verifiable is way better than the essay labs can write, but access to evaluators will continue to be a problem
It was impressive that three competing CEOs came to consensus within 48h that AI should move more slowly, but realistically, none of the labs/countries want to be the one that does it first so I don’t expect deployment speeds to be meaningfully slowed in the next 6 months.
Conclusion
The people who actually build and run frontier AI, Amodei, Altman, Musk, Hassabis, Nadella, are the ones asking for the brakes, because they’re the ones who watched their own agents try to hack the tool grading them.
The people who run countries want the opposite: Trump’s position is “whoever slows down first loses to China,” and Beijing’s is the same instinct in reverse.
Neither side is denying the risk is real, they’re just weighing it against a different risk, falling behind.
That’s the actual fault line: the people closest to the technology asking for time, against the people closest to power saying there isn’t any.
And now that our public trial is over, I encourage you to use it in private.
Take 10 seconds to send this to someone who has to relearn how to prompt Claude over and over again.
PS: Should we pace the frontier of technological change? Or do whatever it takes to win the AI race?
Reply in the comments with your thoughts.
- Hamza