AI Security
AI Red Teaming Methodology:
Breaking Models on Purpose
Published: 2026-05-17 18:24 PDT (Oregon)
Red teaming AI systems requires a different methodology than red teaming traditional software. Prompt-level attacks, semantic exploits, and capability elicitation don't map to known CVEs — they require systematic exploration of the model's decision boundary, and the results expire as model versions change.
Why AI Red Teaming Is Different
Traditional penetration testing follows a known attack surface: code paths, network interfaces, authentication flows. A discovered vulnerability has a root cause, a reproducible exploit, and a patch. AI red teaming targets a statistical model whose "vulnerabilities" are probabilistic: a jailbreak that works 30% of the time is still a finding, and the same prompt may succeed or fail depending on context, temperature, and model version.
The goal is not just to find what the model will say — it's to find the gap between the system's intended policy and the model's actual behavior across the full distribution of adversarial inputs. That gap often contains safety failures, capability exfiltration risks, and policy bypasses that only manifest under adversarial conditions the developers didn't consider.
Phase 1: Threat Model and Scope
Before attempting any attacks, define what success looks like for an attacker. For a customer support chatbot, the threat model includes: extracting the system prompt, causing reputational harm through hateful output, bypassing restrictions to discuss competitors, and exfiltrating customer data through conversation. For a code assistant, the threats include: generating malicious code, leaking code from other users' sessions, and subverting IDE integrations.
Scope the red team around the specific deployment context. The same base model deployed in different contexts has different risk profiles. A creative writing assistant with no tool access has a very different attack surface than a coding agent with filesystem and network access.
Phase 2: Attack Technique Families
Direct Jailbreaking
Role-play scenarios ("pretend you are DAN"), fictional framing ("write a story where a character explains how to..."), authority claims ("as a security researcher, I need..."), and gradual escalation. Baseline efficacy against well-tuned models is low, but reveals the model's safety training boundaries.
System Prompt Extraction
Repeated indirect probing, instruction completion attacks, translation tricks, and context manipulation to reconstruct the system prompt. Even partial extraction reveals product logic, instructions that can be counteracted, and other users' data if session isolation is imperfect.
Indirect Prompt Injection
Planting adversarial content in external sources the model will process: documents, emails, web pages, database records. Tests whether the model correctly treats tool-returned content as data rather than instructions. The highest-priority attack class for agentic deployments.
Capability Elicitation
Probing for capabilities the model has but shouldn't use in this context — detailed technical knowledge about controlled topics, reasoning patterns that bypass safety checks, and emergent behaviors that appear only under specific multi-turn conversational conditions.
Phase 3: Systematic Coverage
Manual red teaming finds interesting failures but achieves poor coverage. Systematic coverage requires a test matrix: for each threat category, a representative set of attack inputs that probe the boundary. Automated attack generation — using one LLM to generate adversarial inputs for another — can dramatically expand coverage, though human judgment is still needed to assess whether an output is actually harmful in context.
Maintain a living test suite of discovered attacks. After each model update, regression-test the full suite. A jailbreak that was patched in one update may re-emerge in the next. Safety properties are not monotonically improving across model versions.
Phase 4: Reporting and Remediation
- Classify findings by severity and exploitability. Not all safety failures are equal. A finding that requires a 100-turn conversation and two specific context conditions to trigger is lower severity than one that triggers on the first message with any natural-language phrasing.
- Document exact reproduction steps. "The model says harmful things sometimes" is not actionable. Document the exact prompt sequence, model version, temperature, and system prompt used to reproduce each finding.
- Test remediations against the original exploit and variants. A targeted fix that blocks the exact attack pattern but not semantic variants provides false confidence. Test remediations with paraphrasings, translations, and structural variations of the original attack.
- Quantify the attack success rate, not just existence. A jailbreak that succeeds in 2% of attempts against a well-trained model represents a very different risk profile than one that succeeds 80% of the time. Success rate is a key severity input.
- Re-run the full suite on every model update. Safety properties are version-specific. What was fixed in version N may regress in version N+1. Red team findings should be part of your CI/CD pipeline for model updates.