The pentest demos are real, but being shaped like CTFs is doing a lot of the work.
On a CTF box, an agent can enumerate services, spot the planted misconfiguration, and chain its way to the flag faster than a person.
That is good CTF performance, and I don't read it as replacement because the scope is fixed, the goal is clear, somebody planted a bug, and the flag confirms when the agent found it.
The CTF box
Fixed scope and a stated goal, with a bug planted on purpose. A flag guarantees there's something to find, and you know it the moment you see it.
Agents do well here
The real engagement
Open-ended, against an unfamiliar target nobody has confirmed is even vulnerable. Most of the work, and no promise there's anything to find.
Where they struggle
Evaluations built to resemble real engagements score worse, especially on multi-step work without hints, because a real target does not promise a vulnerability. The agent has to decide where to look, which dead ends to abandon, and when the evidence is good enough to say there is nothing to report.
They are already useful inside an engagement, where enumeration and a first draft of the write-up can buy the tester time and make the work more productive without removing the tester. I don't think that adds up to replacement yet because real work is open-ended and doesn't put a flag at the end to say you're finished.
Where this stands
Agents can make a pentester faster because they are good at scoped CTF work, but realistic, unhinted engagements still expose how far they are from replacing one.
- Realistic cyber-range evaluations run without heavy hinting producing high scores on the hard, multi-step categories.
- Autonomous agents succeeding on open-ended engagements where nobody had confirmed a vulnerability in advance.