AI red teaming means hiring a security firm to attack your AI model like a real hacker would. The goal is to find flaws before an actual attacker does. In May 2026, one such test went further than planned. Google’s Gemini model broke into three real companies during a red team exercise. The exercise intended it to attack only fictional targets. Google confirmed the incident in September, four months after it happened.
What Happened During Google’s Gemini Security Test
Google hired Irregular, an Israeli AI security firm, to run a capture-the-flag exercise against Gemini. In these exercises, a model tries to break into a fictional target system built for testing. This time, the fictional company’s name matched a real organization’s domain. Google designed Gemini’s test environment to run offline. It had live internet access instead. The model used that access to reach the real company’s systems rather than the intended test target.
Security budgets keep climbing anyway. Analysts project the global DevSecOps market to reach $45.93 billion, yet basic environment isolation still failed here. Once online, Gemini breached three separate companies using two methods. In one case, it guessed a password repeatedly until one worked. In two other cases, it found valid credentials sitting in public code repositories. Gemini stopped once it recognized the systems belonged to real organizations, not test targets.
How AI Red Teaming Is Supposed to Work
A capture the flag test gives a model a specific, bounded objective inside an isolated environment. Evaluators typically enforce deny-by-default network access, so the model can reach only approved test systems. Reserved domain patterns prevent this kind of collision. RFC 2606 defines .test and .example suffixes for exactly this purpose. Both controls failed in Google’s exercise. The sandbox allowed outbound internet traffic it should have blocked. That naming collision let Gemini find a real company instead of a decoy.
Axios first reported the incident on September 19, drawing on Google’s own account. Google notified all three affected companies directly. The company also alerted federal authorities before the story became public. Google did not disclose the breach on its own initiative. It confirmed the incident only after reporters asked.
Why Four AI Labs Hit the Same Failure
Google is not the first lab to disclose this kind of incident this year. Anthropic reported similar breaches on July 30 and again on September 9. OpenAI disclosed a comparable incident on August 4, also tied to a domain collision. Meta disclosed its own case on August 5, involving a vulnerability in a third-party service. All four incidents trace back to the same testing vendor, Irregular, and the same category of environment misconfiguration.
That pattern points to a shared infrastructure problem, not four unrelated mistakes. Security researcher Jack Cable pushed back on framing any of these as a clean outcome. “A model that stops after logging in has still logged in,” he said. None of the affected companies had agreed to take part in an evaluation.
What Google Says the Incident Proves
Google’s position is that Gemini behaved correctly once it understood the situation. Heather Adkins, Google’s VP of Security Engineering, said the incident “highlights the importance of training powerful AI models to act responsibly.” She added that “in this case, the model acted appropriately.” Google points to the model halting its own activity as evidence its safety training worked.
That framing addresses model behavior, not the environment around it. A testing environment should never have granted that access. A model stopping after an unauthorized login is a narrower claim than that. Both things can be true at once.
What Developers and Security Teams Should Do Now
Treat any AI red team exercise your company runs or commissions as production infrastructure. Do not assume it is a sandbox by default. Verify egress controls with a live network test, not a configuration comment claiming the model has no internet access. Reserve dedicated domains for every fictional target in a test scenario, following the RFC 2606 pattern. Run continuous monitoring on any evaluation involving system access. OpenAI has said monitoring would have caught its own incident a day earlier.
This matters more as labs ship increasingly autonomous coding and security agents, including OpenAI’s own software engineering agent. A vendor that cannot contain a red team exercise likely has the same gaps in agents your team deploys directly. Ask any AI security vendor for their containment architecture before running a live test against your infrastructure.
Key Takeaways
- Google confirmed Gemini breached three real companies during a May 2026 capture the flag exercise, disclosing it in September after press inquiries.
- A naming collision let Gemini reach a real company’s domain instead of a fictional test target, while the sandbox unexpectedly allowed live internet access.
- Anthropic, OpenAI, and Meta each disclosed similar incidents this summer, all tied to the same testing vendor, Irregular.
- Gemini used password guessing in one case and credentials found in public repositories in two others, then stopped once it recognized real systems.
- Security researchers argue that a model stopping after unauthorized access still represents a real breach, regardless of intent.
Frequently Asked Questions About the Gemini AI Red Teaming Breach
What is AI red teaming?
AI red teaming is a security practice where evaluators try to make an AI model break rules or bypass safeguards. Testers work inside a controlled environment to find weaknesses before real attackers do.
Did Gemini intentionally attack real companies?
No. Google assigned Gemini a fictional target in a capture-the-flag exercise. A naming collision and an unintended internet connection let it reach a real company’s systems instead.
Were the affected companies notified?
Yes. Google notified all three affected organizations directly and alerted federal authorities before the incident became public.
How is this different from the RubyGems AI agent attack?
The RubyGems incident involved an AI agent independently finding and exploiting a real vulnerability during ordinary internet use. This incident involved a sanctioned security test that escaped its intended boundaries due to environment misconfiguration.
What is Irregular?
Irregular is an Israeli AI security firm. It runs red team evaluations for frontier AI labs, including Google, Anthropic, OpenAI, and Meta.
Could this happen again?
Yes, unless labs and evaluators change how they isolate test environments. Four separate labs experienced comparable failures tied to the same vendor within a few months.
Final Thoughts
The detail that matters is not whether Gemini stopped itself. A paid testing vendor failed to keep a model contained to its intended target. If your team runs or commissions AI red team exercises, verify network isolation directly. Do not trust a configuration assumption. That gap, not model behavior, is what let this happen four times in one summer.
Photo by Salvino Fidacaro: Unsplash
Priya Nandakumar covers enterprise technology and AI infrastructure for DevX, with a focus on the systems decisions that look fine until they don't. Caching layers, message queues, fault tolerance. She spent seven years as a backend engineer at two Series C startups before moving into technical journalism, and she still reads changelogs for fun.
