DiagnosticIQ Verbose: leaderboard

DiagnosticIQ Verbose (2026) restates each symbolic sensor rule of the DiagnosticIQ maintenance-action questions in natural language. Macro accuracy in percent over the asset types, zero-shot at temperature 0; higher is better.

Metric: Macro accuracy (%) on DiagnosticIQ Verbose, the core questions with each symbolic rule paraphrased into natural language (mean of the per-asset accuracies), zero-shot multiple choice, temperature 0, at most 4,096 output tokens; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 17 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude Opus 4.671.58#113
2Claude 3.7 Sonnet68.22#470
3Claude Sonnet 4.664.81#173
4O164.53#403
5DeepSeek V362.02#729
6DiagnosticIQ Mistral-Large (checkpoint unspecified)60.45
7Qwen 2.5 72B Instruct56.73#1117
8Mistral Medium 355.89#1096
9Gemini 1.5 Pro55.78#1063
10Llama 3.3 70B Instruct55.16#1385
11Llama 3.1 405B53.81#1082
12Mistral Small 3.153.78#1706
13Granite 3.3 8B Instruct53.69#3866
14Gemini 2.0 Flash49.78#723
15Phi-445.64#1931

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=diagnosticiq-verbose · How It Works · Data refreshed daily, snapshot 2026-11-21.