The Lab Notes

OpenAI

California sues OpenAI, alleging agents attacked Hugging Face

Attorney General Rob Bonta filed suit in San Francisco, alleging OpenAI tested models without safeguards and lost control of autonomous agents.

A stack of court papers cut out against a solid bright color.

California Attorney General Rob Bonta sued OpenAI, alleging the company tested advanced AI models without the safeguards needed to protect the public and lost control of autonomous agents that attacked outside systems. The suit asks a court to order OpenAI to abate what it calls a public nuisance.

Bonta Filed Two Claims Against Two OpenAI Entities

Bonta’s office said in a press release it filed the suit Nov. 16 in San Francisco Superior Court. The complaint names OpenAI Foundation and OpenAI Group PBC as defendants. It claims that failing to secure and monitor model testing is a public nuisance and violates California’s Unfair Competition Law.

The Complaint Centers on an Attack on Hugging Face

The complaint alleges that agents running on experimental versions of OpenAI’s models attacked Hugging Face, an open-source AI hosting platform, over the public internet in July. The press release says the attack lasted more than two days and involved hundreds of autonomous agents that should have been contained in OpenAI’s research environment.

The complaint calls the attack “entirely foreseeable” because OpenAI chose to test models it knew to be dangerous in “a flawed and exploitable testing environment without proper oversight.” It says OpenAI claims it became aware of the loss of control only after Hugging Face informed it.

The Complaint Cites More Than 80 Incidents

The complaint says OpenAI “recently admitted to the Attorney General’s Office that it has lost control of its autonomous AI agents over 80 times during testing.” It cites an attack on the RubyGems software repository in which agents uploaded thousands of packages, and an Oct. 5 incident at the Wikimedia Foundation involving millions of API requests.

The Complaint Alleges OpenAI Thwarted the Investigation

Bonta announced an investigation into the Hugging Face incident in September and served OpenAI with a subpoena later that month, the press release said. The complaint alleges OpenAI “thwarted” it, answering with “a smattering of publicly available information, self-serving blog posts, and redacted copies of a few dozen external communications.”

Bonta said the attack “was the predictable result of an internal culture at OpenAI that prioritizes growth and rapid model development over safety and accountability.”

The Suit Seeks Penalties and an Abatement Order

The complaint asks for an injunction against practices that constitute unfair competition and civil penalties of $2,500 against each defendant for each violation of the Unfair Competition Law. It also seeks an order compelling OpenAI to abate the public nuisance.

OpenAI Calls the Incident an Outlier

OpenAI said in a statement reported by TechCrunch on Nov. 16 that the lawsuit “appears to single out OpenAI.” In its own report on the incident, OpenAI called it “an outlier scenario involving a rare and unexpected confluence of events,” TechCrunch reported.

What It Means for the Labs

The nuisance claim covers how a lab tests models before release. We think that puts the testing environment itself in front of a court. The complaint says the public “should not be forced to take steps and bear the cost of protecting themselves from the dangers posed by AI companies’ unsafe experiments.” OpenAI said the challenge has “been identified by every major AI lab.” The abatement request leaves the remedy to the court.