Niall Ferguson: AIpocalypse Now

The moment when everyone begins talking about an issue is usually the moment when the probability of rational decision-making—call it p(RDM)—goes to zero.

Everyone is now talking about AI. Or writing about it. Or getting AI to write about it. The headline—“Ex-Anthropic insider tells CNN how AI could kill all humans by 2030”—really got everyone’s attention, didn’t it? And within days we got the nonrational decision: Let’s hit pause. Or rather, let’s hit “pace.” Yes, it’s time to “Pace the Frontier.”

Here’s Dario Amodei of Anthropic: “We must slow the pace at which we improve the capabilities of AI models.” In other words, AI could find a cure for cancer, but it could also unleash cyberattacks and bioterrorism. So slow down, people! We need third-party “embedded evaluators” at all the frontier AI companies. We need companies within democratic countries to “establish common safety standards.” And we also need to “coordinate with authoritarian governments, to the extent this is possible.”

And suddenly, almost overnight, nearly everyone agrees (another bad sign). Amodei’s archrival, Sam Altman of OpenAI, is so committed to pausing that he is selflessly postponing his company’s IPO until next year. Elon Musk, who hates Altman with a passion, says “Dario is right.” Demis Hassabis of DeepMind is in, too.

To get a sense of why we’re hitting pause, or pacing the frontier, let’s ask AI itself a question.

Prompt: What is your p(Doom), i.e., probability that AI could kill all humans within the next decade? Please answer in no more than two sentences.