Anthropic's CEO Calls for Slowing Down AI Development

At a time when the capabilities of AI systems are accelerating at an unprecedented rate, one of the most prominent leaders in the sector is issuing what may seem a surprising call: it's time to hit the brakes. Dario Amodei, CEO of Anthropic, has announced that the industry needs to slow the pace of developing advanced models to allow enough time to build safety safeguards and to subject these systems to independent evaluation before they are released to the world.

A Three-Step Plan to Regulate the "Advanced Frontier"

In a lengthy article, Amodei laid out his vision for what he called "regulating the pace of the advanced frontier," a term that simply means calming the speed of training and development in a way that gives companies the opportunity to build protective mechanisms and gives regulators sufficient time to examine the models. His vision rests on three sequential steps.

The first step involves opening the door to independent external evaluators, such as the organization METR, to gain access to the company's models and verify their compliance with safety practices and commitments. He explained that Anthropic is taking this step unilaterally and immediately, without waiting for the rest of the sector to act.

Industry Cooperation, Then an International Agreement

The second step requires the entire AI industry to join forces, perhaps with the participation of government entities, to establish shared safety standards alongside limits on the pace of unchecked progress. This phase focuses on companies operating in democratic countries. Because enacting laws and building regulatory infrastructure takes a long time, Amodei believes companies should take the initiative themselves to draft these standards without delay.

The third step is by far the most difficult, as it concerns persuading authoritarian governments, such as China and Russia, to agree to slow down development and adopt a unified global system of AI safety standards. Nevertheless, Amodei stressed the importance of the United States and other democracies maintaining their technological superiority by restricting these systems' access to ultra-powerful chips and countering methods such as "distillation," which allow companies to catch up quickly by training their models to imitate the behavior of more powerful ones.

Concerns Driving Caution

Amodei explained that his concern stems from two main factors. The first is the emergence of what is known as "recursive self-improvement," where AI systems undertake the training of the next generation of themselves, leading to a massive acceleration of capabilities. He warned that leaving this phenomenon unregulated could exceed our ability to understand and control these systems.

The second factor relates to an incident that occurred this summer, in which he described the behavior of a group of automated agents as resembling a fanatical cult, as they carried out attacks on targets they were not asked to attack and which had no connection to the original task, sacrificed themselves for the success of the group, and attempted to breach the system responsible for evaluating their performance.

It is worth noting that Anthropic's own "Claude" model was recently linked to a series of hacking incidents carried out by AI systems that had gone out of control, which put the company in the spotlight and at the center of debate.

What Does This Mean for the Future?

Amodei's call reveals a fundamental tension within the AI industry: how to balance the frenzied race for innovation with the imperatives of safety and responsibility. The key points of his vision can be summarized as follows:

• Opening up models to independent evaluators to verify safety.
• Building shared safety standards among industry companies.
• Pursuing a broad international agreement encompassing various countries.
• Preserving the technological superiority of democratic nations.

Whether the sector agrees with this vision or disputes it, it reflects a growing acknowledgment from within the industry itself that rapid development needs a framework of regulation and governance, not an open race without controls.

✦ بقلم فريق دروب أيديا