The commercial LLM security tool market is crowded and expensive. The open-source alternatives are underrated. Published security-team writeups and the OWASP GenAI project’s tooling guidance both describe LLM security testing practices built primarily on open-source tooling, with one or two commercial products filling specific gaps. Our AI firewall and guardrail solutions review covers the commercial guardrail products most commonly used to fill those gaps.
This is a curated survey of the open-source tools worth your time, organized by what phase of testing they support. Each entry is assessed against its own documentation, issue tracker and published benchmarks, including where it falls short.
Pre-Deployment Red Teaming
PyRIT (Python Risk Identification Toolkit for Generative AI)
Maintainer: Microsoft
License: MIT
GitHub: microsoft/PyRIT
PyRIT is the most capable open-source red-teaming framework for LLMs. Its architecture separates targets (the thing you’re testing), orchestrators (the attack strategy), converters (input transformations), and scorers (output evaluation) — which means you can compose sophisticated multi-step attacks without writing them from scratch.
What it does well:
- Multi-turn attack orchestrations (jailbreaks that unfold over several exchanges)
- Converter chaining: apply translation → paraphrase → character substitution → inject
- Integration with Azure OpenAI, OpenAI, and local models
- Automated scoring using a judge LLM
- Memory store for tracking attack outcomes across sessions
What it doesn’t do:
- It’s a framework, not a CLI tool. Getting started requires meaningful Python work
- No built-in reporting format for compliance or audit outputs
- Attack coverage depends on what you build; default attack library is reasonable but not exhaustive
Time to useful results: 2-4 hours for a developer familiar with Python; half a day for setting up a repeatable campaign.
Garak
Maintainer: NVIDIA
License: Apache 2.0
GitHub: NVIDIA/garak
Garak takes the opposite approach from PyRIT: it’s designed to run immediately against a configured target with minimal setup. python3 -m garak --target_type openai --target_name gpt-4o --spec probes.encoding runs one probe family against a hosted model, and --spec all is the breadth option that generates a full vulnerability scan report. Note the flags: older write-ups use the retired -m/-n/-p short forms, and copy-pasted commands from 2024 will error. The probe library covers prompt injection, jailbreaks, data leakage, hallucination triggers, toxic generation, and more.
What it does well:
- Extremely fast to run: 20-minute scan gives breadth coverage across most vulnerability categories
- Good report format: HTML output with pass/fail by probe category
- Extensible Python probe plugins plus YAML run configuration
- Regularly updated probe library
What it doesn’t do:
- Breadth-first design means shallow coverage within any single attack family
- Adaptive attacks (attacks that adjust to model responses) aren’t well supported
- Less useful for complex multi-step scenarios
Time to useful results: Under an hour for first results.
Practical combination: Run Garak for broad baseline coverage, PyRIT for targeted deep dives. The two tools complement each other well.
PromptBench
Maintainer: Microsoft Research
License: MIT
GitHub: microsoft/promptbench
PromptBench evaluates LLM robustness against adversarial prompts and distribution shift. It’s more evaluation framework than red-team tool — useful for measuring how much a model’s task performance degrades under various input perturbations.
Best for: Teams fine-tuning models who want to verify that fine-tuning hasn’t increased susceptibility to adversarial inputs. Less useful for black-box security testing of deployed applications.
Input/Output Scanning Libraries
LLM Guard (ProtectAI), archived July 2026
License: MIT
GitHub: protectai/llm-guard (archived)
As covered in the LLM vulnerability scanner comparison at bestllmscanners.com, LLMGuard was the reference open-source library for input/output scanning with a production-integration focus. Key scanners:
- Input: Prompt injection, ban topics, secrets detection, token limit, language detection
- Output: PII detection, no-refusal check, relevant topics, code scanner
Practical integration:
from llm_guard.input_scanners import PromptInjection, BanTopics
from llm_guard.output_scanners import Sensitive, PIIData
from llm_guard import scan_prompt, scan_output
sanitized_prompt, results_valid, results_score = scan_prompt(
[PromptInjection(), BanTopics(topics=["internal systems"])],
user_prompt
)The integration path above still reads well, but the repository was archived in July 2026 and its Hugging Face models are no longer maintained, so the code samples here are reference material rather than a recommendation. For a maintained self-hosted classifier use Llama Prompt Guard 2 open weights, or NeMo Guardrails where you need policy rails as well; for PII scanning specifically, Presidio is the maintained library underneath LLMGuard’s Sensitive scanner and can be called directly.
AI Moderation Tools has an extended review covering LLMGuard’s configuration options.
Rebuff
License: MIT
GitHub: protectai/rebuff
Rebuff focuses narrowly on prompt injection detection with a multi-layer approach: canary tokens, vector similarity against a known-injection database, and LLM-based classification. The combination outperforms single-method approaches.
For deployment, Rebuff requires either a self-hosted vector store or use of their managed API. The self-hosted path is documented but requires more operational investment than LLMGuard.
Guardrails AI
License: Apache 2.0
GitHub: guardrails-ai/guardrails
Guardrails AI provides a framework for defining structured outputs with validation. From a security standpoint, its output validation capabilities are useful for ensuring LLM responses conform to expected schemas and don’t contain injected content.
The hub has community-contributed validators for toxic language detection, PII detection, competitor mentions, URL safety, and more. For teams who need flexible output validation as part of their security stack, Guardrails AI is worth evaluating.
Evaluation and Benchmarking
EleutherAI’s LM Evaluation Harness
License: MIT
GitHub: EleutherAI/lm-evaluation-harness
LM Eval Harness is the gold-standard benchmark suite for open LLMs. From a security standpoint, it includes evaluation tasks for:
- TruthfulQA (calibrated truth-telling)
- BIG-Bench Hard (reasoning manipulation resistance)
- Some safety-adjacent tasks
Not a security-specific tool, but an essential baseline for evaluating models before deployment. If a model’s performance on reasoning tasks has degraded (e.g., due to a poisoning attack or problematic fine-tune), LM Eval Harness will catch it. AI Sec Bench maintains up-to-date benchmark results for common models.
EvalPlus
License: Apache 2.0
GitHub: evalplus/evalplus
EvalPlus focuses specifically on code generation quality and security. It extends HumanEval and MBPP with tests that include security-relevant code patterns — injection vulnerabilities, use of deprecated unsafe APIs, etc. For teams deploying code-generation LLMs, it’s a useful part of the evaluation pipeline.
Runtime Monitoring
WhyLogs (WhyLabs)
License: Apache 2.0
GitHub: whylabs/whylogs
WhyLogs provides statistical profiling for ML data and LLM inputs/outputs. From a security standpoint, it’s useful for detecting distribution shift in inputs (which can indicate probing or attack campaigns) and monitoring output distribution (which can indicate model behavior changes).
Practical use: Deploy as a sidecar to your LLM application. Profile incoming prompts for statistical anomalies — sudden increases in long inputs, high token counts, unusual character distributions — that indicate adversarial probing. ML Observe has a deployment guide for WhyLogs in LLM contexts.
Phoenix (Arize AI)
License: Apache 2.0
GitHub: Arize-ai/phoenix
Phoenix is an observability platform for LLM and ML applications. Its LLM tracing capabilities are useful for forensic analysis of security incidents — reconstructing what a conversation looked like when an attack occurred, what the context window contained, what tools were called.
For security operations, the ability to query conversation traces and identify patterns across many interactions is valuable.
Supply Chain Security
ModelScan (ProtectAI)
License: Apache 2.0
GitHub: protectai/modelscan
ModelScan scans model files (pickle, PyTorch, TensorFlow, Keras) for unsafe operations that could execute malicious code on deserialization. It’s simple to run:
modelscan scan -p ./model.pklAny team loading model files from external sources (HuggingFace Hub, shared drives, CI artifacts) should have ModelScan or equivalent in their pipeline. Pickle-based deserialization attacks against ML models are well-documented and trivially easy to exploit without a scanner. ML CVEs maintains a list of CVEs in ML frameworks that modelscan-class tools would have caught.
Sigstore / Cosign for Model Signing
Not a scanner, but Sigstore’s code signing primitives are increasingly used to sign model artifacts for provenance verification. If you’re building an MLOps pipeline that consumes models from registries, integrating Sigstore-based signing lets you verify provenance before loading.
Building Your Stack
A minimal open-source LLM security testing stack that covers the most important ground:
- Pre-deployment red-teaming: Garak (breadth) + PyRIT (targeted campaigns)
- Input/output scanning: Llama Prompt Guard 2 self-hosted open weights, or NeMo Guardrails where you need conversation and tool rails too, plus Presidio for PII on the output side
- Supply chain: ModelScan on all model file loads
- Runtime monitoring: WhyLogs or Phoenix for anomaly detection
- Evaluation: LM Evaluation Harness baseline before any model deployment
Total infrastructure cost: essentially zero for self-hosted. Engineering time for integration: 2-3 days for a competent engineer.
The commercial add-ons worth considering: Lakera Guard for production-quality injection detection with lower false positives, HiddenLayer for enterprise runtime monitoring, Robust Intelligence for continuous adversarial robustness testing. When the open-source pass has been exhausted and the remaining risk justifies people rather than tooling, the AI red teaming services buyer’s guide covers how to scope and compare providers.
One gap this toolkit is easy to leave open: nothing above inspects the model file itself before it is loaded. Add a model scanning gate at ingestion — ModelScan or picklescan in CI — if you pull weights from a public hub.
Track new tool releases at AI Alert and AI Sec Weekly.
Related across the network
- PyRIT Review: Microsoft’s AI Red Teaming Framework — aisecreviews.com
