AWARE-US (Pairwise Ranking) - S1 Unique Repair: leaderboard
Metric: Car match (%; share of the 41 persona-grounded car-search dialogues, four preference constraints of which exactly one can be dropped to make the query feasible, in which the agent relaxes the oracle's least important constraint and recommends the oracle's car; pairwise ranking: the model compares every pair of constraints in both orders and relaxes the one that loses most comparisons). Source: arxiv.org. Saturation forecast: Estimated already saturated. 3 models tracked.
Top models
| # | Model | Score |
|---|---|---|
| 1 | Gemma 2 9B (IT) | 24.4 |
| 2 | Qwen 3 8B | 17.1 |
| 3 | Llama 3 8B Instruct | 9.8 |
Interactive version: theaggregate.ai/benchmark?slug=aware-us-pairwise-ranking-s1-unique-repair · How It Works · Data refreshed daily, snapshot 2026-09-26.