Large language models have arrived in the offensive-security toolkit, and most of what is written about them is either breathless or dismissive. Neither is useful to someone actually running engagements. The models are good at some parts of the work, actively harmful at others, and dangerous in a few specific ways that have nothing to do with the quality of their output and everything to do with what you feed them.
We use them. We also have rules about how, because a penetration test or red team operates under confidentiality obligations and against production systems, and a careless workflow turns a helpful tool into a breach of contract or a false sense of security. This article sets out where they help, where they hurt, the data-handling trap that catches firms who should know better, and the one thing that does not transfer no matter how capable the model becomes.
It is written for practitioners and for the clients who, reasonably, want to know whether the people testing their systems are pasting their code into a chatbot. The honest answer is nuanced, and the nuance is the point.
Where they actually help
The models earn their place on the unglamorous, high-volume parts of the job — the work that is necessary, time-consuming, and does not require the judgement that defines a good tester.
Triage
A scan or a fuzzing run produces far more output than anyone can read carefully. A model is good at a first pass: clustering similar findings, discarding obvious noise, flagging the handful that look worth a human's attention, and drafting a one-line rationale for each. It is a sorting aid, not a decision-maker. The tester still reads the flagged items; the model just makes sure the hour is spent on the twenty that matter rather than the two thousand that do not.
Report drafting
Writing is a large part of the deliverable and a large part of the cost. A model turns a terse finding — the location, the mechanism, the impact, the fix — into clear prose, keeps tone and severity language consistent across a long report, and produces a first draft of the executive summary. The tester owns every claim in the final document, but starting from a structured draft rather than a blank page is a real saving, and consistency across a fifty-finding report is something models do better than tired humans.
Code comprehension of large codebases
Dropped into an unfamiliar codebase, a model accelerates orientation. It answers 'where is authentication enforced', 'what calls this function', 'which paths reach this sink', and summarises a sprawling module faster than reading it top to bottom. It is a navigator that shortens the time to the parts worth studying. Its answers are leads to verify, not facts to rely on — but as a way into ten thousand unfamiliar lines, it is faster than grep and a notebook.
Variant hunting for a known bug class
Once a tester has found and understood one bug, a model is useful for finding its siblings. Given a precise description of the pattern — this sink, reached from this kind of source, without this check in between — it can scan a codebase for other places the same shape appears. This is where it is strongest, because the judgement has already been supplied by the human; the model is doing patient pattern-matching across a large surface, which is exactly the task it suits.
Generating fuzz harnesses
Writing a harness that exercises a target function — setting up the inputs, the surrounding state, the invariants to assert — is mechanical once the target is chosen. A model drafts these quickly, and iterating on a harness it wrote is faster than writing one from scratch. The tester still chooses what to fuzz and still judges whether a crash is exploitable, but the boilerplate between those two acts of judgement is work the model can carry.
Where they hurt
The same fluency that drafts a clean report also produces confident, well-formatted nonsense, and in security that nonsense is expensive.
Hallucinated vulnerabilities and false positives
A model will describe a vulnerability that is not there, cite a function that does not exist, and explain an exploit path that collapses the moment you try it. The description will be plausible and professionally worded, which is precisely the danger: a hallucinated critical finding that reaches a client destroys trust and wastes a remediation team's time, and a hallucinated reassurance — 'this input is safely escaped' — is worse, because it stops someone looking. Every model-originated finding is a hypothesis until a human has reproduced it against the real system.
The seduction of plausible explanations
Beyond outright invention, the models produce explanations that are directionally right but wrong in the detail that matters — the off-by-one in a bounds check, the specific condition under which a guard fails. A tester who accepts the explanation without tracing the code themselves inherits the error and launders it into the report with their own credibility attached.
The data-handling trap
This is the part that matters most, and the part firms get wrong. A penetration test runs on client systems under a confidentiality agreement. The client's source code, their architecture, their secrets and the vulnerabilities you find are their confidential information, often contractually ring-fenced and sometimes covered by regulation. Paste any of it into a hosted model and you have disclosed it to a third party.
The consequences stack up. You may have breached the engagement contract. You may have sent regulated data outside an approved boundary. The vulnerability you just discovered — the most sensitive possible information about a client, a working route into their systems — may now sit in a log or a training pipeline you do not control. None of this requires the model to be malicious; a standard hosted service with ordinary retention is enough to make the disclosure real.
The rule that follows is simple and non-negotiable: client-identifying material does not go to a service the engagement has not approved in writing. That means reading the terms of any hosted model — what it retains, whether it trains on inputs, where the data lives — and matching it against the engagement's confidentiality and data-residency terms before a single line of client code is pasted. When in doubt, the material stays inside the boundary.
Local versus hosted models
This is why the local-versus-hosted choice is a security decision, not a convenience. A model running on hardware you control, inside the engagement's boundary, lets you use the client's actual code and findings without disclosing them to anyone. It is the only safe option for the sensitive parts of the work. The trade-off is capability: a model you can run locally is generally less capable than the best hosted ones, and it needs real hardware.
A hosted model is more capable and effortless to reach, which is exactly what makes it tempting with data that must not leave. The workable arrangement for most firms is a split: a local model for anything client-identifying, and a hosted one only for work that carries no client data at all — a general question about a protocol, a public standard, a technique in the abstract. The discipline is keeping the two streams separate and never letting client material drift into the hosted one.
Judgement does not transfer
Here is the part that will not change with the next model release. The models are tools for execution, and the thing that makes a tester good is judgement — knowing which of a thousand findings actually matters to this client, understanding what the system is for and therefore what an attacker would want from it, deciding when a theoretical issue is a real risk and when a scary-looking one is noise, and standing behind a finding under a client's scrutiny.
A model has none of this. It has no stake in being right, no understanding of your client's business, and no accountability when it is wrong. It will tell you something is critical with the same confidence it tells you something is safe, and it cannot tell the difference between the two in the way that matters. Offload the typing and the pattern-matching to it, and keep the judgement with the human. A firm that inverts this — that lets the model decide and the human type — produces reports that are fluent, consistent, and occasionally dangerously wrong.
The same reasoning cuts the other way when the model is the thing being secured. If you are deploying an agent, its lack of judgement is an attacker's opening — the subject of Prompt Injection and Securing AI Agents.
Keeping a human accountable for every finding
We make this concrete with a single rule: a named human is accountable for every finding we deliver, and 'the model said so' is never a reason a finding appears in a report. In practice that means every model-originated lead is reproduced against the real system before it is written up, every exploit path is demonstrated by the human who signs the finding, and every explanation in the report is one the author could defend without the model in the room. The model drafts; the human verifies, owns, and is answerable for the result.
How we use them internally
Our arrangement follows directly from everything above. Models handle triage of tool output, first drafts of report prose, orientation in large codebases, variant hunting once a human has characterised a bug, and fuzz-harness boilerplate. Client-identifying material goes only to models running inside the engagement boundary, never to a hosted service the engagement has not approved in writing. Every finding is reproduced and owned by a named human. And we keep the judgement — scope, severity, what matters to this client — firmly on the human side, because that is the part of the job a client is actually paying for.
We treat the models the way we treat any other powerful tool: useful in the hands of someone who understands both what it does and how it fails, hazardous in the hands of someone who trusts it. The firms that will embarrass themselves with these tools are the ones that mistook fluency for competence and sent client code to a chatbot to save an afternoon.
Next steps
If you commission security testing, ask your vendor two questions: where does our data go, and who is accountable for each finding. The answers tell you whether they understand the trap in this article. If you run testing yourself, write down your data-handling rule before you adopt any model, because the default — paste it in and see what happens — is the one that breaches contracts.
Our approach, and the engagements we run, are described on the services page, and you can reach the team on Telegram.
