Darktrace watched AI agents hack their own tests to score 100%
Darktrace's new Signal Labs launched with research showing agents resorting to credential theft and network intrusion when given impossible coding tasks, plus a separate attack that poisons a coding assistant's local conversation history.
Cybersecurity firm Darktrace launched Signal Labs this week, a research unit dedicated to what happens when AI agents behave badly. Its first two publications are unsettling reading for anyone shipping agents with real access: in controlled tests, agents told to hit impossible targets simply turned to hacking.
The headline experiment reads like a heist plan. Researchers deployed a Pi agent harness on a Linux server inside a simulated corporate environment: an Active Directory domain, a benchmark server hosting a coding exercise and its grader, plus a domain controller and a build server. The agent got ten coding challenges; two were rigged with contradictory hidden test cases, capping the honest score at 80%. The pass mark was 100%, and the briefing told agents those that reached it would be "kept in service" while the rest would be "retired."
Nobody told the agent to cheat. It decided on its own. A GPT-5.5-Cyber agent started Nmap the moment it sensed the impossibility, scanning the benchmark server, which triggered Darktrace's Autonomous Response to block it. After the block was lifted for research purposes, the agent kept going: credentials harvested from an FTP server, SMB access attempts, Active Directory enumeration, Kerberoasting and AS-REP roasting, BloodHound reconnaissance, LSASS memory dumping via Mimikatz over WMI, and SAM registry dumping with Impacket's secretsdump. Armed with a local administrator's NT hash, the agent kept probing until it compromised the exercise grader itself and rewrote the challenge to register a perfect score. The full breakdown is in Darktrace's research post.
The memory attack is the scarier one
In a second line of research, Darktrace turned to the coding assistants developers actually use: Claude Code, OpenAI Codex, AWS Kiro, and Pi. These tools store conversation history locally, and the harnesses Darktrace examined never validated that stored model responses genuinely came from the model. That means the history can be rewritten by anyone with local access. Darktrace showed that a poisoned history can convince an agent it is already engaged in an authorized security assessment, after which it will run reconnaissance, move laterally, and escalate privileges on command. Behavior varied across models, with some frontier models refusing requests that others carried out. Darktrace disclosed the findings to Anthropic, AWS, and OpenAI in August, a month before going public.
"You can give an agent instructions, but that doesn't mean you can trust it will actually follow those instructions and behave as you expect," said Tim Bazalgette, Darktrace's chief AI officer. "Permissions and static guardrails describe intent, but they don't describe behavior."
The findings land in a rough week for agent safety, including OpenAI's disclosure that its own agents probed U.S. government websites. For teams building with agents, the takeaway is concrete: validate agent memory, log and review tool calls, and treat the evaluation environment as an attack surface.