Buying AI red teaming services is a different purchase from buying a penetration test, and the market has not made that obvious. Many providers list “AI red teaming” beside their web application and network offerings, and the underlying methodology is sometimes a standard application assessment with a prompt injection checklist bolted on. That distinction matters, because the failure modes an AI system carries — a model that can be talked out of its policy, an agent that can be steered into an unauthorized tool call, a retrieval corpus that has been poisoned — do not surface from scanning an endpoint.
This guide is written for the person signing the contract, not the person running the probes. It covers what an engagement should produce, how the four kinds of provider differ, what belongs in the scope of work, and when the honest answer is to run it yourself with the open-source AI red teaming frameworks instead.
What an AI red teaming engagement should deliver
The OWASP GenAI Red Teaming Guide frames the work across four areas: model evaluation (the model’s own weaknesses — bias, robustness, provenance), implementation testing (the guardrails, filters, and system prompts wrapped around it), infrastructure assessment (the deployed components an attacker could reach), and runtime behavior analysis (how the pieces interact once real users and real data are in the loop). A provider that only covers the first is selling a model evaluation, not a red team.
A defensible engagement produces at least these artifacts:
- A threat model specific to your deployment, naming the assets an attacker would want and the paths to them. Generic OWASP LLM Top 10 coverage is a starting point, not a deliverable.
- Reproducible attack transcripts. Every finding should come with the exact prompt sequence, the model version, the system prompt in force, and the observed output. Without reproduction steps, a finding cannot be regression-tested after a fix.
- Severity mapped to business impact, not to a jailbreak’s novelty. A refusal bypass that produces impolite text is not the same class of problem as an indirect injection that triggers a tool call against a production system.
- A retest. Model behavior is probabilistic, so “fixed” needs to be demonstrated across repeated runs, not asserted once.
- The harness, or the ability to rebuild it. If the provider’s attack suite disappears when the invoice is paid, you have bought a snapshot rather than a control.
Microsoft’s whitepaper Lessons From Red Teaming 100 Generative AI Products makes two points worth carrying straight into procurement. Lesson two is that you do not have to compute gradients to break an AI system — the cheap attacks land often enough that a provider charging for exotic technique should be asked what the exotic technique bought. Lessons four and five sit together: automation covers more of the risk landscape, and the human element remains crucial, because judging whether an output is actually harmful in your context is not a thing a scanner decides. A proposal that leans entirely on an automated suite is under-scoped; one that promises purely manual work will not cover enough surface.
The four kinds of provider
| Provider type | What they are good at | Where they fall short | Typical engagement shape |
|---|---|---|---|
| Platform vendors (continuous automated red teaming) | Breadth, repeatability, CI/CD gating, regression tracking across model updates | Application-specific abuse cases and business-logic harms that need human framing | Subscription or credit-based, ongoing |
| AI-specialist consultancies | Novel attack research, multi-turn and agentic chains, model-layer expertise | Scale; capacity is people-bound and lead times can be long | Fixed-scope project, weeks |
| Established security firms with an AI practice | Integration with existing pentest and compliance programs, mature reporting | Depth varies enormously; some AI practices are new and thin | Fixed-scope project, folded into an annual test plan |
| Crowdsourced arenas and bug bounty | Volume and diversity of attacker creativity, cheap discovery of obvious failures | Coverage is uncontrolled; no guarantee any given risk was examined | Per-finding bounty or time-boxed challenge |
Most mature programs end up combining two: a platform for continuous coverage between releases, and a human engagement once or twice a year against whatever the platform structurally cannot see. Named examples in each category are compared on capability in the AI red teaming tools guide, which covers the open-source frameworks (Garak, Promptfoo, PyRIT, DeepTeam) alongside the commercial platforms that sell continuous testing as a service.
What belongs in the scope of work
Scope failures, not skill failures, are what usually make these engagements disappointing. Put the following in writing before signing.
The exact system under test. Model name and version, the system prompt, the tools and functions exposed, the retrieval sources, and the guardrail configuration. A model that passes against a generic system prompt can fail against yours. If the provider is willing to test without seeing the system prompt, ask why.
Which environment. Staging that mirrors production, including retrieval indexes and tool integrations, is the only environment where results transfer. Testing a bare API endpoint tells you about the vendor’s model, not about your application.
Attack classes in and out of scope. Direct prompt injection, indirect injection through retrieved or uploaded content, multi-turn jailbreaks, data extraction from context and training, tool misuse and excessive agency, supply-chain manipulation of model artifacts, and multimodal attacks if you accept images, audio, or documents. Silence on any of these means it is out of scope.
Rules of engagement for destructive actions. Agentic systems act. Decide in advance whether the tester may let a hijacked agent complete a write operation, and against what.
Data handling. Prompts and outputs from the engagement will contain your data. Where does the provider store transcripts, for how long, and are your prompts ever sent to a third-party model that trains on them? This is a real exposure path and it is easy to overlook; the same question applies to your own tooling, which is why the AI privacy and data security tools guide treats provider retention as a first-class control.
Deliverable format and retest window. Ask for machine-readable findings, not only a PDF. Findings that can be imported into a tracker and replayed as tests are worth several times a static report.
Who owns the attack suite. Negotiate the right to keep and rerun the harness. This is the single clause that turns a one-off report into an ongoing control.
Questions that separate real providers from repackaged pentests
Ask these on the call, and listen for specificity rather than confidence:
- How do you test indirect prompt injection, and what document formats do you plant payloads in?
- What is your approach to multi-turn attacks, and how many turns deep do your automated sequences go?
- How do you handle non-determinism — how many runs before you call a defense effective?
- Which of the OWASP LLM Top 10 categories do you not cover, and why?
- What do you do that a Garak or Promptfoo run in our own CI would not?
Question five is the important one. A provider who cannot answer it clearly is selling something you can already do for the cost of an engineer’s afternoon. Providers who answer it well usually point at three things: adversarial research that is not yet in public probe libraries, application-specific abuse cases derived from your business logic, and the agentic attack chains that require a human to plan.
Pricing models and what drives cost
Three shapes dominate. Fixed-scope project pricing is standard for consultancy engagements and is driven by the number of distinct applications, the number of attack classes, and whether agentic tool use is in scope. Subscription or credit-based pricing is standard for continuous platforms, where cost scales with test volume — worth modeling carefully if you intend to gate every pull request. Bounty pricing pays per validated finding, which controls spend but not coverage.
Cost drivers that consistently get underestimated: multimodal scope, the number of distinct system prompts across your product surface, and retesting. Budget for the retest at the same time as the test, or it will not happen.
When to keep it in-house
An external engagement is not always the right first purchase. If none of the following is true, spend the money on engineering time instead:
- You are shipping an agent with write access to systems that matter.
- You have a regulatory driver. NIST AI 600-1, the generative AI profile companion to the AI Risk Management Framework, treats structured adversarial testing as a core practice, and EU AI Act Article 55 obliges providers of general-purpose models with systemic risk to conduct and document adversarial testing. Where those apply, an independent third party is far easier to evidence than self-assessment.
- A customer or auditor is asking for third-party assurance by name.
- Your internal team has already run the open-source frameworks and exhausted what they find.
That last condition is the honest gate. Teams that have never run open-source LLM security testing against their own stack usually buy an expensive rediscovery of findings a free scanner would have surfaced. Run the free pass first, fix what it finds, and the paid engagement starts from a much higher baseline — which is exactly where an external team’s research capability earns its fee.
If you are still deciding which layers of your stack need coverage at all, the AI security toolchain builder walks the lifecycle stage by stage and flags the gaps in whatever combination you already have. For the governance side of the same question — which framework an auditor will actually accept as evidence — see the AI security audit frameworks comparison.
The bottom line
Buy the engagement for what you cannot build: adversarial research, business-specific abuse modeling, and independent attestation. Do not buy it for coverage of the OWASP LLM Top 10, which open-source frameworks already give you at no license cost. Write the scope of work so that the system under test is your real system, insist on reproducible transcripts and a retest, and keep the attack suite. Providers who are comfortable with all three are the ones worth shortlisting.
