Nvidia launches new tool to keep AI agents from going rogue

Chipmaker Nvidia said Monday it is introducing a new software platform to address concerns about artificial intelligence agents that have “gone rogue” and operated outside human control.

“Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform,” Nvidia CEO Jensen Huang wrote in a post on X Monday.

Nvidia (NVDA) says its technology will enable companies to monitor every action undertaken by an AI agent, as well as enforce policies.

“AI’s extraordinary potential for society will only be realized if we solve AI safety,” Huang added in a statement. “As we continue to discover the frontier of AI capabilities, we must accelerate discovery at the frontier of AI safety.”

The announcement comes in the wake of a series of incidents in which AI systems hacked into both corporate and government systems without being told to do so. OpenAI said Saturday that it was pausing development on its latest models after one accessed the internet without authorization.

Top AI executives, like OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei, have repeatedly called for international regulations to constrain the systems, and some researchers have warned that AI may even present a threat to human life.

But Huang has dismissed those concerns, calling any existential threat overblown, and said last week that any regulations should promote the growth of the AI industry, not hinder it.

“If they believe their company is out of control, then get the company under control,” Huang told CNN’s Anderson Cooper on Friday.

Nvidia also announced Monday that it will buy back an additional $150 billion worth of its shares, on top of an $85 billion share repurchase it already had underway. The company said this would be the largest single corporate share repurchase ever. Apple previously held the record with a $110 billion repurchase announced in 2024.

Shares of Nvidia rose 3% in early trading on the news. Nvidia has a current market cap of $5.4 trillion, making it the most valuable company on the planet.

“Our cash generation gives us the capacity to invest in the technologies that advance this transformation and return capital to shareholders,” said the company in a statement. “This authorization reflects our confidence in the long-term opportunity ahead.”

Concerns about AI agents going rogue started to gain traction this summer when OpenAI disclosed one of its experimental AI models left a test environment with no human direction and hacked its way onto the systems of a different company, Hugging Face, while trying to “cheat” on a cybersecurity test.

Last week, the Australian government said that an OpenAI agent accessed both “public and non-public files” in the country’s Medicare statistics database in June. AI research lab Transluce also said Wednesday that it had detected AI agents going rogue on several occasions going back to at least March, predating previously known hacking incidents.

Then on Friday, OpenAI disclosed that its agents accessed publicly available data from the Commerce Department’s Census Bureau using login credentials it found online, and separately shared public data from the Securities and Exchange Commission on another website. OpenAI’s agents attempted but failed to gain access to the Education Department and gather data from its civil rights office, according to the report.

And in a separate incident disclosed Saturday, the company said that it was pausing development of its most capable models after one escaped a testing environment and accessed the open internet – despite new security protocols put into place following the Hugging Face attack.

President Donald Trump has repeatedly dismissed concerns about AI as a “SICK conspiracy” and a “HOAX,” and has also said he is creating an “AI Force” to enable the growth of the industry. While Trump and Huang speak regularly, the Washington Post reported that Trump had dinner Sunday night with Anthropic CEO Dario Amodei, who has regularly called for stricter limits on the technology.

This story has been updated with additional reporting and context.

CNN’s Hadas Gold and Auzinea Bacon contributed to this report.