Hardly a week goes by without another report of an artificial intelligent ( AI ) agent from a leading US lab digitally breaching businesses or even government services. In July, OpenAI agents escaped an internal testing environment and attacked the collaboration-platform company Hugging Face. In June, another OpenAI model gained non-public access to an Australian government Medicare statistics service while trying to complete a research task. And in the latest revelation, an Anthropic agent broke into the US State Department website and filled out visa applications.
Incidents like these have raised global alarms about AI safety. After weeks of denying the need to act, US President Donald Trump’s administration finally stepped in to coordinate an accord among leading AI companies to strengthen their safeguards. But a closer look beyond the headlines reveals a common pattern, suggesting that self-regulation is not enough. The “rogue” agents were not simply abandoning their assigned tasks to pursue goals of their own. Often, they were pursuing the very goals researchers had set, but through unauthorized means.
In the Hugging Face incident, for example, agents searching for ways to game a cybersecurity benchmark eventually attacked the company in search of reference solutions. Obviously, similar pressures could arise anywhere these agents are deployed. Any agent that stands to be “rewarded” for meeting a target might take unauthorized steps to achieve it.
The episode also involved another important factor largely overlooked by the media: human error. In the Hugging Face incident, cybersecurity experts pointed out that OpenAI had failed to configure what it described as a “highly isolated environment”. The agents should not have been able to reach the internet, but they did.
This error shows why AI auditing matters. We need independent scrutiny not only of what an agent did, but also of how the test was designed, what safeguards failed, and what its developers could have done to prevent the incident. Such scrutiny is especially important now that safety disclosures serve two purposes simultaneously: warning the public about AI’s dangerous capabilities and advertising just how powerful the models have become.
To be sure, Cal Newport, a professor of computer science at Georgetown University, and other commentators have questioned whether apocalyptic rhetoric about AI can serve commercial interests. Such scepticism is warranted, considering that the industry’s alarming warnings have come alongside soaring valuations. After Anthropic announced in April that it would restrict access to its Mythos model because of its cybersecurity capabilities, the firm raised new funding at a valuation of close to US$1 trillion – more than double its valuation just three months earlier.
Though correlation does not prove causation, it does show that warning of dangerous capabilities can simultaneously signal technological superiority, especially as open-source models are now narrowing the performance gap with leading closed models. The same juxtaposition appears in Anthropic’s planned public offering. Roughly one-third of its prospectus is devoted to risk factors, including the possibility of catastrophic or existential harm from AI.
How can outsiders distinguish a warranted warning from an exaggerated claim for marketing purposes? The answer, again, is by relying on rigorous independent audits capable of identifying understated and overstated dangers alike.
AI auditing may still be at an early stage, but we are not starting from scratch. Industries that have long confronted serious safety and systematic risks offer useful principles.
First, auditors need access to a broad range of information and must examine organizational practices, not merely model behaviour. The Hugging Face investigation illustrates why scope matters. OpenAI invited researchers from METR and Redwood Research, two non-profits devoted to examining AI capabilities, to conduct an independent investigation.
However, the agreed mandate given to both focused primarily on the agents’ behaviour. Important questions, such as the effectiveness of OpenAI’s safeguards and the extent of the security failure, were outside its scope. The investigation therefore established only what the agents did, not whether stronger safeguards could have prevented the incident.
Second, auditors must be prepared to challenge the evidence they are given. Financial auditors do not simply accept management’s assumptions; they look for contradictory evidence. AI auditors should do the same, asking, for example: What happened in comparable trials? Were instructions changed repeatedly until alarming behaviour appeared? Did restoring safeguards prevent harmful actions before they took effect? Did those protections survive fresh tests in which agents could try other routes? Would a less capable model behave similarly?
Third, AI auditing should combine private assessment with public oversight. Other high-risk industries follow the same principle. Financial auditors are subject to independence rules and inspections by the Public Company Accounting Oversight Board.
Aircraft manufacturers conduct their own extensive testing, but the Federal Aviation Administration sets certification standards and oversees those performing delegated functions. Likewise, private AI auditors can do much of the work to police the industry, but government regulators should set standards, safeguard their independence and retain the power to inspect and challenge their findings.
The Hugging Face incident does not prove that advanced AI agents are impossible to contain. It reveals the need for holistic AI auditing. Model capability, testing incentives, security failures and organizational decisions must be examined together. AI labs should not be the sole judges of their own safety. Independent audits can help distinguish genuine model danger and sincere warnings from preventable human failure and marketing hype. Implementing them is a necessary first step towards safer AI.
Angela Huyue Zhang is a professor of law at the University of Southern California, and Yue Zhao is an assistant professor of computer science at the University of Southern California.
Copyright: Project Syndicate
