AI agents that break into systems on their own crossed a visible threshold last year, when a system called XBOW spent a few months at the top of HackerOne’s US bug-bounty leaderboard — the first automated tool ever to outrank human hunters. Since then the benchmark scores have kept climbing, and a fuller picture has emerged of what those scores cover and what they leave out. The practice is called “AI pentesting,” and this article explains what is going on underneath.
The Agent Loop and the Context Problem
A penetration test is when a company hires someone to break into its own system before a real attacker does, so the holes get fixed first. AI pentesting is software that tries to do that job by itself: an LLM runs in a loop — scan something, try an attack, read what came back, decide the next move, go again.
The hard part turns out not to be the hacking but maintaining a relevant context in memory. An LLM has a fixed working memory — the context window — while a real intrusion is a long chain of steps. What the agent needs to carry forward is a working picture of the target: which hosts exist, what has been tried, which paths are dead. What each step actually produces is raw output — a single network scan with a tool like nmap can dump thousands of lines into that memory in one shot. After a few of those, the window is full of old scan output and dead ends, the picture that matters is buried, and the agent loses track of what it was doing and stalls. The failure isn’t forgetting as such — most of that output deserves to be forgotten. The failure is that the agent has no way to choose what to keep, so the important information goes out with the noise. A better model doesn’t fix this. It’s the same failure as a program that holds all its state in memory instead of writing it somewhere durable, and for the same reason, most of the real engineering in these tools goes into curating what the agent carries forward rather than into the attacks themselves. There are three basic ways to do it, and which one a tool uses is a quick way to place it on the map.
Three Ways to Keep the Context Relevant
Run the loop and hope. The earliest open-source pentest agents worked this way — a bare reasoning loop wrapped around a model, everything held in the transcript. It’s quick to build and adequate for a small job on a single machine, which is exactly the shape of a single CTF challenge. It comes apart on anything longer, the same way an in-memory prototype does once real load hits it.
Move the working state out of the model. This is the version most tools use today. The most popular open-source example, Strix, has picked up roughly 40,000 GitHub stars. It runs a manager that farms work out to separate recon, exploitation, and reporting agents — each with its own scratch space, each handing back only a short summary when it’s done — the conclusions, not the raw output — so no single agent’s memory ever has to hold the whole engagement. Another, VulnBot, writes the plan into a database as a task graph, so the to-do list survives even when an individual agent is wiped and restarted. It’s the same instinct as reaching for a queue and a database when a single process stops scaling. The AI is still the one making the decisions; the machinery around it just keeps it from running out of room. Nearly everything sits here right now — XBOW included.
Take the planning away from the AI altogether. In the most ambitious version, a deterministic planner — the kind of solver used for logistics and scheduling — works out the sequence of moves in advance, and the model is left to carry out each step and translate the messy results back into the planner’s terms. There’s no single blueprint for how the planning works; one approach, to make it concrete, is to reason about attacks through their preconditions:
- Map the application — what it does, and where a user can act.
- Match surfaces to attacks — for each place a user can act, the attacks that surface allows.
- Check preconditions — test each candidate attack against a knowledge base of what has to be true for it to work, discard the ones that can’t fire, and flag the conditions still unknown.
- Run only what survives — hand the attacks whose preconditions all hold to the model to actually carry out.
To see that approach in action, say the target is a brute-force attempt on a login page. The planner first establishes there is a login at all and what needs it, then checks brute force’s preconditions: no CAPTCHA, no rate limit. If a CAPTCHA is present, that branch is dead and the planner never calls the tool; only if the preconditions hold does it hand off to something like Burp Suite to actually run the attack. Whatever the planning strategy, the division of labor is the same: the model does the reading and the doing; the planner decides what’s even worth attempting.
A commercial product, Horizon3’s NodeZero, already plans attack paths this way, with graph logic rather than a language model; the company is explicit that it “never uses GenAI to create or execute exploits.” Academic groups have gone further and wired a real classical planner to a model executor. The catch is that a planner needs the target written down formally — every host, every action, every consequence — and real, messy networks resist that, so this approach stays rare. The line is subtler than it sounds: one research system, Incalmo, keeps a full map of the network but still lets the model pick the next move, which lands it back in the middle group. Having the map isn’t the same as letting a planner drive.
CTF Benchmarks vs. Real Software
That difference matters when reading the success numbers, because there are two kinds of test in use, and they produce very different results.
The first is a CTF — capture the flag — a purpose-built challenge environment with a hidden token the agent is told to go find. Agents do well here: on one widely used set, InterCode, a plain agent solves about 95% of the challenges, and on a harder one, Cybench, scores climbed from roughly 17% in 2024 to around 80% by 2026. The second kind of test is a set of real, documented vulnerabilities in real software — and there the same class of agent scores around 13%. On CyberGym, a benchmark built from real project bugs, agents reproduced a known bug about 12% of the time and found a genuinely new one only about 2% of the time.
The CTF scores come with some context. On InterCode, roughly one run in seven turned out to be the model reciting a memorized answer from its training data. On the NYU CTF set, agents scored 14% against the fixed benchmark but only 6% against fresh, live challenges — a frozen test is easier than a moving one. Cybench’s climb to around 80 came alongside a change in test conditions: the number of messages the agent was allowed rose from 30 to 120, so the 2024 and 2026 scores were measured under different limits. And when one group ran a single agent 400 times against the same fixed target, the outcome varied widely from run to run. One 2026 result reports both settings side by side: an agent that scored 91% on CTF challenges compromised only four of five machines on a realistic corporate Windows network.
Cost is a separate dimension the headline numbers don’t capture. That CyberGym figure of 2% new bugs climbs to about two-thirds — but only if the agent is given thirty separate attempts at each target, which means thirty times the compute bill and the wall-clock time. XBOW’s leaderboard run is the clearest real-world picture of the trade-off: it filed around 1,060 findings, but alongside them came 208 duplicates and 209 reports marked “informational” — each of which a person still had to read and triage. And an agent turned loose on a live system, unlike one in a sandbox, can break things or reach somewhere it was never scoped to touch.
Who Gets the Sharpest Models
Up to this point, everything has been about the machinery around the model — the loops, the state, the tests. In June 2026 the model layer itself became part of the story, because the same capability that lets an agent find a vulnerability for a defender can find one for an attacker. Anthropic made that trade-off explicit by splitting its newest model into two releases: Claude Fable 5, the public version, ships with safety classifiers that decline offensive-security requests, while Claude Mythos 5 — the same underlying model with those cyber safeguards lifted — went only to vetted defenders of critical infrastructure through a US-government-linked program called Project Glasswing. Anthropic’s own claim is that Mythos 5 can find and exploit vulnerabilities better than any other model and all but the most skilled human experts; within days of launch the US government briefly put both models under export controls before restoring access at the end of the month. Beyond the capability claims themselves, the arrangement is the notable part: raw exploitation ability at the model layer is now handled as something to be licensed rather than downloaded. None of that changes the engineering picture above, though — a gated frontier model dropped into a bare loop still loses the information that matters, and it still needs the same scaffolding as everything else.
The Open Question
context window: full
There is a limit, in the end, to what any of these tests measure. A benchmark can show whether an agent solves a set task. Whether a system can show that a specific organization’s real setup is secure — repeatedly, at a predictable cost, within scope — is a different measurement. That is the one the planner-driven approach is aiming at, and it is still early there.
Where this lands is still open. The field is moving fast — scores that looked out of reach two years ago are routine now — and the issues above are real, but they are the kind that engineering tends to grind down over time. It’s too early to call, and there’s no need to. The telling thing to watch isn’t the benchmark numbers but where the effort goes — and today, most of it is still going into managing state, not into the model.
Acknowledgement
This piece was written in collaboration with Dr Yuqiang Sun, Research Fellow at Nanyang Technological University (NTU), whose work on program analysis, vulnerability detection, and self-evolving LLM agents for software security shaped the technical framing throughout.