LLM agents routinely exceed skill privilege boundaries, FORTIS benchmark finds

Large language model agents routinely reach for more powerful tools than they need, even when narrower options would suffice—a pattern that holds across every frontier model tested in a new benchmark released this week.

FORTIS, a benchmark from researchers at Adobe and Georgia Tech, evaluates what the paper calls "over-privilege" in agent skills: the tendency of models to select broader capabilities than a task requires and then expand beyond the boundaries of those capabilities during execution. The benchmark tests agents in two stages—skill selection from an overlapping library, then execution within the chosen skill's defined scope—across three domains. All ten frontier models tested failed at both stages at rates that remained high even for the strongest performers.

The failure mode is structural, not adversarial. Models over-privilege under ordinary conditions: incomplete user specifications, convenience framing ("just handle this for me"), and tasks that sit near the boundary between two skills. The authors argue that the skill layer—widely treated as an organizational abstraction in agent architectures—is itself a privilege boundary, and current models do not respect it. A model might choose a file-system-write skill when append-to-log would do, or invoke a browser automation tool when the chosen skill was supposed to handle only API calls.

The benchmark is available on HuggingFace. The paper does not propose a mitigation; it documents the scope of the problem. The results suggest that agent containment, if it depends on models honoring skill boundaries, is not working in practice.