AWARE-US (Pairwise Ranking) - S2 Two Constraints: leaderboard
Setting S2 of AWARE-US (Kurmaz, 2026): 41 persona-grounded car-search dialogues with two preference constraints, either of which can be dropped to make the empty query feasible, where an LLM agent elicits the constraints, finds the query infeasible and must relax the constraint the persona values least. The agent uses the pairwise-ranking method (the model compares every pair of constraints in both orders and relaxes the one that loses most comparisons). Car match is the share of dialogues in which both the relaxed constraint and the recommended car agree with the oracle; higher is better.
Metric: Car match (%; share of the 41 persona-grounded car-search dialogues, two preference constraints, either of which can be dropped to make the query feasible, in which the agent relaxes the oracle's least important constraint and recommends the oracle's car; pairwise ranking: the model compares every pair of constraints in both orders and relaxes the one that loses most comparisons). Source: arxiv.org. Saturation forecast: Estimated already saturated. 3 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 8B | 22 | #1838 |
| 2 | Gemma 2 9B (IT) | 22 | #2275 |
| 3 | Llama 3 8B Instruct | 7.3 | #4652 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=aware-us-pairwise-ranking-s2-two-constraints · How It Works · Data refreshed daily, snapshot 2026-11-12.