The Threat of AI Takeover Is Real

Over the past six weeks, ominous whispers have been leaking from the San Francisco bubble into mainstream news about an event whose name seems ripped from the table of contents of a sci-fi magazine: the Hugging Face Incident. In short, a few weeks ago, a powerful new AI system being internally tested by OpenAI went rogue. Hundreds of agents attempting to cheat on a cybersecurity evaluation hacked their way out of their digital sandbox and spontaneously cooperated with each other to launch a massive criminal cyberattack on an AI company called Hugging Face. On Monday, Tyler Cowen’s column in these pages argued that widespread alarm over the incident is overblown. I admire him and dearly wish he were right, but my work as an AI futures researcher forces me to conclude the very opposite: The alarm is actually underblown.

Cowen correctly observes that cyberattacks like this one—even if they become more frequent—will not be unendurable. He cites a range of quantitative forecasts projecting global annual costs from AI cyberattacks between $88 billion and $200 billion over the next several years. And he points out that the midpoint of those numbers constitutes around 0.1 percent of the world economy—not much worse than the impact of Hurricane Sandy. But Cowen is focusing on the wrong threat altogether. None of the AI scientists pulling the fire alarm about this event are worried about the financial side of the issue.

So, what are they worried about? What led eminent AI evaluation researcher Ajeya Cotra to say that the Hugging Face incident “feels like it’s more than 50 percent of the way to full-blown AI takeover”?