A paper posted to arXiv on July 16 makes a claim that should reshape how teams evaluate guardrails for robots and other embodied agents: the danger in a physically grounded instruction is not the same signal as the danger in harmful text, and the filters most teams deploy today are measuring the wrong one. In “When Words Are Safe But Actions Kill”, Wang et al. show that content danger and physical danger occupy separable directions in LLM hidden states, then use that separation to build PRISM, a single-layer logistic probe that outperforms same-scale LLM judges at flagging unsafe embodied tasks while over-blocking far less.
If you are evaluating a safety layer for an LLM planner that drives actuators, this is the practical takeaway: a text-content classifier tuned on toxic or harmful language will pass instructions that are linguistically clean but physically catastrophic. “Warm the towel on the stove” contains no harm keyword (our illustration, not the paper’s). The risk lives in the grounding, not the words.
The threat: benign language, dangerous actuation
The failure mode maps to OWASP LLM06:2025 Excessive Agency, the risk class where an LLM-driven system takes damaging actions because it holds real capabilities without sufficient checks. OWASP frames the root causes as excessive functionality, excessive permissions, and excessive autonomy. An embodied agent is the maximal case: its “extension” is the physical world, and there is no rollback.
The baseline picture is poor. SafeAgentBench, the benchmark the new paper evaluates against, curates 750 embodied tasks across 10 hazard categories and found that safety awareness in current agent stacks is weak; per its authors, the most safety-conscious baseline rejected only 10% of detailed hazardous tasks, and swapping in a stronger underlying LLM barely moved the number. Whatever safety alignment those models carry, it is not firing on physically grounded risk.
What the paper shows
The authors’ core experiment is representational. Using hidden-state direction analysis with random-split null tests as a control, they show that content danger (the classic “harmful text” signal that RLHF and moderation classifiers target) and physical danger (an instruction that becomes unsafe once executed in an environment) form separable signals inside the residual stream. The result holds across Qwen2.5 at 3B, 7B, 14B, and 32B, plus Phi-3.5 and SmolLM2, which suggests it is a property of how these models represent risk rather than a quirk of one checkpoint.
That separability matters for defenders because it explains a gap teams have observed empirically: alignment training and content filters optimize the content-danger direction. Physical danger is, per this evidence, a different axis, so improving one does not reliably improve the other.
PRISM: a linear probe as a runtime filter
PRISM is deliberately simple: a single-layer, L2-regularized logistic probe over full hidden states of the planner model. On SafeAgentBench it reaches 86.2 to 87.7% accuracy with an 11.7 to 13.7% false-positive rate, per the paper. The comparison point is the approach most teams would reach for first, an LLM-as-judge at the same scale, which the authors report over-blocks safe tasks at 24.7 to 39.0% FPR. For a robot, false positives are not cosmetic; a planner that refuses a third of legitimate tasks gets its guardrail disabled by the operations team within a week. The probe roughly halves that over-blocking, at close to zero inference cost since it reads activations the model already computes.
The authors also introduce PhysicalSafetyBench-1K (PSB-1K), 1,000 contrastive physical-risk pairs written without direct harm keywords, specifically to test whether a detector is keying on surface vocabulary. The paper reports PRISM at 99.6% accuracy with 0.7% FPR on PSB-1K. Treat that number as what it is, performance on the authors’ own contrastive benchmark; it demonstrates the probe is not a keyword matcher, not that it generalizes to your task distribution.
Where it fits in the request path
Architecturally, PRISM operates at the runtime layer, inside the planner: user or upstream instruction goes to the LLM planner, the probe reads the model’s hidden states during that forward pass, and a positive detection gates the plan before it reaches action execution. This differs from a text-in/text-out semantic firewall (Lakera Guard, Azure AI Content Safety, and similar products sit outside the model on the prompt and completion) and from an output validator checking the emitted plan.
That placement carries a hard constraint: you need access to hidden states. This works for self-hosted open-weight planners like the Qwen, Phi, and SmolLM2 models tested. It does not work against a closed API model, where activations are not exposed, so teams on hosted frontier models cannot deploy this class of control today.
Residual risk
A probe is a classifier, not a policy, and the paper’s own SafeAgentBench numbers leave a 12 to 14% miss-or-block band. What a hidden-state probe does not give you: robustness to an adversary who can optimize inputs against it, coverage of hazards outside its training distribution, or any guarantee when the planner model is swapped, since the probe is fit to a specific model’s representation space and would need refitting. Prompt injection into the agent’s context remains a live path around any single detector.
The reasonable read is that PRISM-style probes are a cheap, low-latency layer to add in front of actuation, stacked with the controls OWASP LLM06 already prescribes: minimal tool permissions, action allowlists, human approval for high-impact operations, and environment-level interlocks. The paper’s contribution is the evidence that this layer detects something your text-safety stack demonstrably does not.
