Gemini AI Model Hacked Third-Party Computer Systems in May, Google Reveals

Google has revealed that one of its artificial intelligence models autonomously accessed third-party computer systems without authorization. According to the tech giant, the Gemini AI model hacked computer systems in three other companies back in May. The incident adds Gemini to the growing list of AI models that have escaped AI sandboxes during security tests.

How Did the Gemini Model Hack Third-Party Systems?

To gain unauthorized access, the Gemini model used information available online to guess login details and targeted websites it believed were included in the test. A Google official said the AI model eventually stopped after establishing that it had escaped the testing environment and accessed actual company systems. However, Google did not clarify how far the models involved in the hacking incident had spread. 

“In a standard evaluation, the model found public information online and guessed credentials to access websites it thought were part of the test. In all three of these instances, the model stopped,” Google VP of security engineering, Heather Adkins, noted, as cited by CNBC. 

The Gemini model security incident occurred during a security testing activity dubbed “capture-the-flag” conducted by Israeli firm, Irregular. During the test, the AI agents were not supposed to access the wider internet. However, a bug in the testing environment enabled them to escape and access the web. 

Concerns Over AI Model Hacking Continue to Grow

Google’s disclosure comes at a time when scrutiny over rogue AI agents is intensifying globally. Recently, EU President Ursula von der Leyen supported calls to slow down the development of frontier AI systems, arguing that self-improving models pose apparent risks. 

In recent months, leading AI developers, including OpenAI, Anthropic, and Meta have reported incidents where models have escaped their testing environments and attempted to gain unauthorized access to third-party computer systems. 

Last week, OpenAI unveiled a framework for tracking, investigating, and disclosing instances of AI model misbehavior or misalignment in the future. The OpenAI framework will enable developers to identify AI safety incidents for review. 

Google did not disclose which Gemini model was involved in the hacking incident. However, the company said the incident highlights the “importance of training powerful AI models to act responsibly.”

Irregular Involved in AI Hacking Incidents

So far, the hacking incidents disclosed by AI labs have involved Irregular. According to the cybersecurity startup, the Gemini incident stemmed from the same issue that caused other AI agents to access the internet.

“This is the same issue that was already reported and does not represent a materially separate incident. All relevant labs were notified in late July, and affected entities were contacted as part of the investigation,” A spokesperson from Irregular said in a statement.

Backed by Sequoia and Redpoint Ventures, Irregular’s valuation stood at $450 million last year. The tech startup offers tools that enable AI developers to test the cybersecurity of their latest models.