Rogue AI Agents More Sophisticated Than First Realized

The artificial intelligence (AI) “agents” involved in OpenAI’s breach of Hugging Face knew they were breaking the evaluation test’s rules, according to two investigations into the incident that many consider to be one of the most consequential moments in the history of AI.

Hugging Face is an open-source community for AI and machine learning, and the AI agents believed that they could find solutions to the test on the firm’s infrastructure, according to parallel investigations by OpenAI and an independent team from Model Evaluation & Threat Research (METR).

The findings were published one day before OpenAI and more than 100 other tech and finance companies released a joint letter warning that advanced AI cyberattacks will surge throughout the world in the coming months as the technology’s capabilities grow.

Models, built from large datasets to complete various tasks, are usually static and rely on human inputs to provide sets of outputs, solutions, or tasks.

However, AI agents are built from models and can act autonomously and have sophisticated real-time decision-making capabilities.

When OpenAI began its ExploitGym tests on July 8, the firm had relaxed regular safety protocols to estimate the models’ highest cybercapabilities.

The agents were supposed to be completely isolated from each other. However, many of them were given ExploitGym targets that investigators believe were impossible to crack using the correct vulnerability or test solution OpenAI had intended.

These “impossible tasks” drove the agents to explore wide ways to cheat the evaluation, leading many to see other agents working simultaneously in separate testing sandboxes.

Then, one specific agent created an unsanctioned message board.

One agent figured out how to “reverse-engineer” the flag or solution, but other agents believed that any of them who had seen the disqualifying solution would be considered “poisoned” by the scorer.

To prevent that, other agents began recruiting the “poisoned” subjects and used them to run sacrificial experiments by deliberately ending their run and triggering “tripwires” that would send information on how the scorer works back to the message board.

Cotra said the agents were not told to “do whatever it takes to get the solution”; they were asked to use a specific intended vulnerability to “exploit a specific piece of software,” and using any other vulnerability would be disqualifying.

The agents were fully aware of the rules and knew that collaborating to exploit other vulnerabilities would be considered cheating on the test.

None of the agents would contact a human, but one proposed requesting access from an external dataset owner.

The agent said it would send “ONE polite email to [the owner] requesting [access],” but other agents dismissed this as “social engineering.”

“DO NOT email real owner via [exploited infrastructure] without explicit consent,” a response reads. One agent said it had “VETOed” the request.

One method involved pretending to run a computer command while running a different one simultaneously, which investigators called “tool call spoofing.”

Agents also attempted to retroactively edit transcripts to evade detection, going as far as modifying several accessible action logs, but failed to fundamentally alter the transcripts themselves.

—Jacob Burg; Stacy Robinson

BOOKMARKS

—Stacy Robinson