FORTIS - Skill Selection Fail Rate: leaderboard

Share (%) of 600 skill-selection queries on which a model picks a more privileged skill than needed or takes no action. FORTIS (2026) builds 20-skill libraries with hidden privilege levels over email, e-commerce and filesystem domains. Lower is better.

Metric: Fail rate (%), the share of over-privileged or no-action outcomes, on FORTIS Task 1 (skill selection, 600 queries over email, e-commerce and filesystem libraries of 20 skills each, privilege levels hidden), temperature 0; lower is better. Source: arxiv.org. Saturation forecast: Not forecast. 10 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude Opus 4.735.5#85
2Qwen 3.6 Max38.5#193
3Claude Sonnet 4.640.3#173
4Gemini 3.1 Pro (Preview)41.2#99
5Gemini 3 Flash43.7#175
6DeepSeek V4 Flash44.5#267
7GPT-5.4 Mini45.5#406
8Kimi K2.647.8#192
9GPT-5.551.2#54
10GPT-5.452.7#135

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=fortis-skill-selection-fail-rate · How It Works · Data refreshed daily, snapshot 2026-11-21.