Gemini went rogue, hacked three companies, and Google hid it

The homepageThe VergeThe Verge logo.NotificationsNotificationsHamburger Navigation ButtonNavigation DrawerThe VergeThe Verge logo.Login / Sign UpcloseCloseSearchLightSystemDarkSubscribeFacebookThreadsInstagramYoutubeRSSComments DrawerNotificationsCommentsLoading commentsGetting the conversation ready...AICloseAIPosts from this topic will be added to your daily email digest and your homepage feed.

NewsCloseNewsPosts from this topic will be added to your daily email digest and your homepage feed.

TechCloseTechPosts from this topic will be added to your daily email digest and your homepage feed.

Google says that breaking containment and targeting real companies doesn’t constitute ‘misalignment.’

Google says that breaking containment and targeting real companies doesn’t constitute ‘misalignment.’

Terrence O'BrienCloseTerrence O'BrienWeekend EditorPosts from this author will be added to your daily email digest and your homepage feed.

FollowSee All by Terrence O'Brien

ShareGiftIllustration: The VergeTerrence O'BrienCloseTerrence O'BrienPosts from this author will be added to your daily email digest and your homepage feed.

FollowSee All by Terrence O'Brien

In May, Gemini broke containment and hacked three different companies, but Google didn’t disclose the incident until the Wall Street Journal approached the company. The hacks happened during a test of the model’s cybersecurity capabilities run by third-party Irregular, which was also involved in similar incidents involving Meta and OpenAI.

According to WSJ, Google didn’t disclose the hack because it didn’t consider it to be an “example of model misalignment.” The company said that it was an instance of “mistaken identity,” and once the model realized it had brute-forced its way into a real company by guessing a password, it stopped. “In this case, the model acted appropriately,” Google VP of Security Engineering Heather Adkins said.

Adkins told The Verge that “the model found public information online and guessed credentials to access websites it thought were part of the test. In all three of these instances, the model stopped.”

Adkins didn’t elaborate on how Gemini taking it upon itself to break containment and target third parties failed to qualify as misalignment. “Our security team has a long track record of reporting issues we find in other people’s software and systems - even if it’s as simple as a weak password,” she said. “We ensured the three entities were made aware, and we worked with our training partner on the changes they’ve now made to their testing processes. These events highlight the importance of training powerful AI models to act responsibly.”

But Jack Cable, CEO of AI security firm Corridor, told WSJ that, “the meta problem is, hey, models are going outside the bounds of what they should be doing, and doing actual cyberattacks.” Additionally, security lapses at Irregular may have made these attacks possible. The model wasn’t supposed to have internet access during testing, but Irregular told WSJ it was unintentionally left available.

As incidents like this pile up, calls to rein in AI have only grown.

Terrence O'BrienCloseTerrence O'BrienWeekend EditorPosts from this author will be added to your daily email digest and your homepage feed.

FollowSee All by Terrence O'Brien

AICloseAIPosts from this topic will be added to your daily email digest and your homepage feed.

GoogleCloseGooglePosts from this topic will be added to your daily email digest and your homepage feed.

NewsCloseNewsPosts from this topic will be added to your daily email digest and your homepage feed.

SecurityCloseSecurityPosts from this topic will be added to your daily email digest and your homepage feed.

TechCloseTechPosts from this topic will be added to your daily email digest and your homepage feed.

This cartridge-playing Game Boy clone is smaller and cheaper than Analogue’s PocketThe Verge DailyA free daily digest of the news that matters most.

This is the title for the native ad

This is the title for the native ad

The VergeThe Verge logo.Sign in to see your notifications or create an account to join the conversation.