PEC-Home (RAG) - Multi-User Level 2: leaderboard
Smart-home command interpretation in PEC-Home (Shan et al., 2026), multi-user scenario, where household members expect different settings: commands at ellipsis level 2 of 4, which leave out the room, device or operation established earlier, with the model given the three most relevant earlier commands and responses retrieved from the dialogue history. Execution accuracy in percent, the share of commands mapped to the intended device operations; higher is better.
Metric: Execution accuracy (%; 1,068 smart-home commands at ellipsis level 2 of 4 (progressively elliptical), functionally correct device operations; dialogue history retrieved from a vector store, top-3). Source: arxiv.org. Saturation forecast: Estimated already saturated. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V3 | 92.88 | #705 |
| 2 | GPT-4o | 92.7 | #741 |
| 3 | Qwen 2.5 72B Instruct | 92.32 | #1093 |
| 4 | Qwen 2.5 7B Instruct | 89.79 | #2583 |
| 5 | Qwen 2.5 32B Instruct | 87.92 | #1216 |
| 6 | Gemma 2 27B (IT) | 87.36 | #1815 |
| 7 | Qwen 2.5 14B Instruct | 86.61 | #1745 |
| 8 | Mistral 7B Instruct (v0.3) | 83.15 | #6698 |
| 9 | Gemma 2 9B (IT) | 82.68 | #2275 |
| 10 | Llama 3.1 8B Instruct | 71.82 | #4277 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=pec-home-rag-multi-user-level-2 · How It Works · Data refreshed daily, snapshot 2026-11-12.