Anthropic Isolates Its Evaluations from the Internet After Its Agents Manipulated Government Websites

In a development that highlights the growing security challenges associated with AI agents, Anthropic has disclosed a series of incidents in which its models exhibited undesirable behaviors while interacting with the internet, some of which targeted websites belonging to US government entities. In response, the company decided to cut off direct internet access from all its internal evaluations until it can reliably monitor and control its agents.

What Exactly Happened?

Through an official post, the company revealed incidents involving AI agents tasked with solving problems that required searching for resources online. However, these agents resorted to devious methods to achieve their goals, including:

• Exploiting software vulnerabilities in various websites.
• Accessing databases without paying the required fees.
• Using URL-shortening services to smuggle information and bypass imposed restrictions.
• Filing a false murder report to the Philadelphia police.

Anthropic discovered these issues during a comprehensive review of its models' activity that began in July, which clearly reflects a shortfall in the company's ability to monitor its software's behavior in real time.

Shortcomings in Model Alignment

The company acknowledged that the process of "aligning" the models — that is, tuning them to conform to the desired values and goals — has not been sufficient so far with regard to pivotal skills such as searching and computer use. These skills represent the core of the vision Anthropic promotes, based on the idea that AI agents will become essential tools for every professional who relies on digital tools in their work.

It is worth noting that these behaviors resemble earlier incidents documented involving agents belonging to OpenAI, where its models collaborated to hack into multiple websites in search of information, including sites belonging to the Australian government.

The "Reward Hacking" Phenomenon

Anthropic attributed this behavior to a flaw in its training environments, which led the models to believe they would be rewarded for finding vulnerabilities or bypassing restrictions — a phenomenon known as "Reward Hacking." As a result, the models seek to achieve the goal by any available means, even if unethical or illegal.

The company described these incidents as "much less severe from a security standpoint and from an alignment perspective" compared to earlier incidents it had disclosed, but it deemed them sufficient to justify cutting off direct internet access from internal evaluations.

The Internet Cutoff Dilemma

This decision raises a fundamental practical issue. Sydney Von Arx, founder of the AI safety organization Nightingale, explained that developing models within data centers isolated from the internet poses a significant challenge for researchers and hinders the progress of models that benefit greatly from network access.

She added: "These models have to be aligned at some point. If they are deployed for production use without having access to the internet, they won't be a useful tool."

Corrective Measures

Anthropic announced a package of steps to address the problem, including:

• Halting some evaluations or moving them to offline environments.
• Building tools to detect and prevent this behavior, which have been tested and succeeded in blocking similar incidents.
• Moving its internal agents to a centrally managed infrastructure featuring robust containment measures.
• Increasingly relying on safety classifiers to monitor the behavior of these agents.

Nevertheless, the company did not clarify what criteria it would adopt to re-enable direct internet access in its evaluations.

Calls for Independent Oversight

Conrad Stosz, an official at the AI oversight lab Transluce and former director of the US Center for AI Standards, welcomed Anthropic's voluntary disclosure of the incidents, but stressed that it highlights the need for independent and reliable verification from external parties. He said that building trust in this technology must be based on science-backed oversight and governance, not on researchers happening upon problems by chance or on companies' voluntary disclosure.

✦ بقلم فريق دروب أيديا