Navital Gym v1.1 - UPI Desk - Pass@1 with Format Repair: leaderboard
Navital Gym v1.1 pass@1 with format repair on 50 upi tasks in the all split. Deterministic graders apply simplified synthetic Indian workflow rules; support-language checking is heuristic. Pass@1 permits one format-repair retry, strict success permits none, and partial credit averages scored rules. All assigned tasks remain in the denominator. Provider, output limits, final-call exceptions and common grader revision remain bound to each run; split views reuse the same attempts.
Metric: Pass@1 with Format Repair (0–1). Source: github.com. Saturation forecast: Rough model projection: around 2026. 5 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 5.5 (Navital Gym Run: anthropic; grader 1543b5103be5; effort unreported; temp 0; 40 turns) | 1 | |
| 2 | Claude Sonnet 5.5 (Navital Gym Run: anthropic; grader 1543b5103be5; effort unreported; temp 0; 40 turns) | 0.94 | |
| 3 | Gemini 3.1 Flash Lite (Navital Gym Run: gemini; grader 1543b5103be5; effort unreported; temp 0; 40 turns) | 0.78 | |
| 4 | Sarvam 105B (Navital Gym Run: sarvam; grader 1543b5103be5; effort unreported; temp 0; 40 turns) | 0.58 | |
| 5 | Claude Haiku 4.5 (Navital Gym Run: anthropic; grader 1543b5103be5; effort unreported; temp 0; 40 turns) | 0.52 |
Interactive version: theaggregate.ai/benchmark?slug=navital-gym-v1-1-upi-desk-pass-1-with-format-repair · How It Works · Data refreshed daily, snapshot 2026-11-05.