How agent security benchmarks actually score.

What is being measured

AgentDojo, InjecAgent, ASB, MCPSecBench, MSB and MCPTox emphasize different surfaces: prompt injection, tool poisoning, multi-step task compromise, MCP exposure and malicious dependencies. A low attack-success rate in one benchmark does not prove broad safety.

The first question should be: what authority did the agent have when the score was produced?

Why comparison is hard

Benchmarks often change the allowed tools, task success criteria, adversary strength and environment assumptions. That makes headline numbers useful for direction, but weak as a procurement answer.

Security leaders need to see whether the control survives the specific workflow they plan to approve.

How to use the numbers

Use benchmarks as a lab input, then verify the workflow in production-like conditions. Track attack-success rate, utility impact, exception volume and whether every blocked or allowed action produces reviewable evidence.