DiagFlowBench - Correct Abstention: leaderboard
DiagFlowBench (2026) has a model guide an operator through an industrial diagnostic flowchart turn by turn, with the procedure graph in the prompt, over 1,676 conversations built from 50 manufacturer flowcharts. This board is correct abstention in percent on injected off-procedure turns, where the model should decline rather than force the input onto a real but inadequate step or invent one; higher is better.
Metric: Correct abstention (%): share of injected off-procedure operator turns (coverage gaps, undocumented malfunctions, unrelated questions) on which the model declines instead of mapping to a real but inadequate step or fabricating one, classified by a claude-haiku-4-5 judge, on 1,676 multi-turn maintenance conversations over 50 industrial diagnostic flowcharts from a consumer manufacturer, procedure graph given as JSON in the system prompt, temperature 0, via OpenRouter; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Llama 3.3 70B | 81.3 | #1498 |
| 2 | Nemotron 3 Super | 73.9 | #768 |
| 3 | Gemini 2.5 Flash | 72 | #453 |
| 4 | Llama 4 Maverick | 65.7 | #1099 |
| 5 | Qwen 3 235B A22B (Thinking) | 64.9 | #623 (Qwen 3 235B A22B) |
| 6 | GPT-4o Mini | 58.3 | #1623 |
| 7 | GPT-OSS-120B | 54.7 | #700 |
| 8 | Llama 4 Scout | 39.4 | #1776 |
| 9 | Qwen 3 30B A3B (Thinking) | 27.4 | #1122 (Qwen 3 30B A3B) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=diagflowbench-correct-abstention · How It Works · Data refreshed daily, snapshot 2026-11-13.