OnCallVal - Root Cause Identified: leaderboard
Agentic on-call incident response in a simulated production fleet; scored only from the final world state. OnCallVal v0.1.0: the agent receives one alert page, a simulated fleet and a 26/32/40-action budget through a 15-action API. The published baselines use nine episodes per fault family, 72 in total (24 medium, 32 hard, 16 severe); the bundled harness runs greedy decoding with no output repair, and the baseline section states no other settings. Resolved requires root cause, mitigation and no collateral change; scores are counts of the 72 episodes. No-collateral (72/72 for every baseline) and action counts are not admitted. The full 200-episode split has no published model scores. Question access: test.parquet (200 episodes) is public with its scorer; the 72-episode baseline subset is described by family/tier counts, not listed.
Metric: Episodes with correct root cause (of 72). Source: huggingface.co. Saturation forecast: Rough model projection: around 2026. 3 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 2.5 3B Instruct (Benchmark Sweep Run: source=OnCallVal; v0.1.0 public split; 72-episode baseline subset; run settings as published) | 16 | |
| 2 | Qwen 2.5 1.5B Instruct (Benchmark Sweep Run: source=OnCallVal; v0.1.0 public split; 72-episode baseline subset; run settings as published) | 10 | |
| 3 | Qwen 2.5 7B Instruct (Benchmark Sweep Run: source=OnCallVal; v0.1.0 public split; 72-episode baseline subset; run settings as published) | 8 |
Interactive version: theaggregate.ai/benchmark?slug=oncallval-root-cause-identified · How It Works · Data refreshed daily, snapshot 2026-11-07.