
By KIM BELLARD
Probably the last thing the world needs is to hear from me about AI’s existential threat, but, really, what else is there to talk about right now?
For anyone who has not been following the current furor, the straw that broke the proverbial camel’s back came last week when Jacob Coxon, a researcher at AI leader Anthropic, announced he was leaving the company — after having left OpenAI for it earlier this year due to Anthropic’s better model-safety efforts. In a post on X, he warned:
I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.
AI systems, he fears, “will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.” Insiders, he says, “earnestly believe it could kill us all by the end of the decade.”
Scared yet?
Others quickly chimed in. Evan Hubinger , a team leader at Anthropic, posted “AI could kill all humans … I personally think it is >10% within the next decade.” Others put the risk even higher. By the end of the week Dario Anodei, founder and CEO of Anthropic, had written a long plea for private industry and government to quickly act together to “pace the industry.”
“We must slow the pace at which we improve the capabilities of AI models,” he urged. “Progress will still seem fast, and we must make wise use of the time we gain.”
OpenAI’s Sam Altman, Space X/X/Tesla CEO Elon Musk, Microsoft CEO Satya Nadella, and former Google DeepMind Dennis Hassabis quickly signaled their support.
We also heard more about the kind of risks AI might pose. Anthropic released a report about how it detected and countered possible AI use to create bioweapons, detailing five such efforts. That’s just the tip of the iceberg: “Recently, we swept 30 days of activity associated with adversarial state institutions and found roughly 35 distinct research efforts, most of them ordinary civilian science, but some with notable dual-use potential.”
And it turns out that this summer’s rogue AI hack of Hugging Face was both scarier than we realized and only one of several such actions. The Wall Street Journal detailed several such efforts, from multiple AI companies. AI agents escaped walled-off environments, coordinated with other AI agents (up to 3,700 in one case), and tried to cover their tracks from humans.
We’re not nearing the point when AI can act on its own to achieve its purposes; we are there. And protecting humans may not necessarily be those purposes.
The big fear is that AI is now at the point of “recursive self-improvement,” taking humans out of the loop in training and upgrading it. If you thought artificial general intelligence (AGI) was scary, RGI puts its rate and scope of improvement on steroids. “It’s hard to overstate how dangerous speeding towards RSI is,” said Jasmine Wang, an OpenAI researcher.
We don’t let private industry develop nuclear or biochemical weapons, and we’re at a point with AI that should give the same kind of concern. Laissez-faire is no longer an option.
Continue reading…
