PEC-Home (RAG) - Multi-User Level 1: leaderboard
Smart-home command interpretation in PEC-Home (Shan et al., 2026), multi-user scenario, where household members expect different settings: complete commands (ellipsis level 1 of 4), with the model given the three most relevant earlier commands and responses retrieved from the dialogue history. Execution accuracy in percent, the share of commands mapped to the intended device operations; higher is better.
Metric: Execution accuracy (%; 1,068 smart-home commands at ellipsis level 1 of 4 (complete commands), functionally correct device operations; dialogue history retrieved from a vector store, top-3). Source: arxiv.org. Saturation forecast: Estimated already saturated. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 2.5 72B Instruct | 95.51 | #1093 |
| 2 | DeepSeek V3 | 94.1 | #705 |
| 3 | GPT-4o | 93.07 | #741 |
| 4 | Gemma 2 9B (IT) | 92.6 | #2275 |
| 5 | Gemma 2 27B (IT) | 91.95 | #1815 |
| 6 | Qwen 2.5 7B Instruct | 91.76 | #2583 |
| 7 | Qwen 2.5 32B Instruct | 89.14 | #1216 |
| 8 | Mistral 7B Instruct (v0.3) | 88.76 | #6698 |
| 9 | Qwen 2.5 14B Instruct | 88.11 | #1745 |
| 10 | Llama 3.1 8B Instruct | 73.97 | #4277 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=pec-home-rag-multi-user-level-1 · How It Works · Data refreshed daily, snapshot 2026-11-12.