GameReplica - Rule Consistency: leaderboard
Rule consistency in GameReplica (Qiao et al., 2026): an LLM judge compares the rule document that the agent induced from pixels and interaction with the reference rule checklist, scoring each rule consistent, partially consistent or inconsistent. Percent over 125 game replication tasks, three runs each, with a second judge reviewing the verdicts; higher is better.
Metric: Rule consistency (%; item-by-item agreement of the rule document the agent induces with the reference rule checklist, 1 / 0.5 / 0 per rule; 125 tasks (25 web games x 5 difficulty levels), game reached only through screenshots and mouse, keyboard and reset actions; mean of 3 runs per task; first-round judge gemini-3.6-flash, review by gpt-5.6-sol). Source: arxiv.org. Saturation forecast: Around March 2027. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.8 | 54.2 | #56 |
| 2 | GPT-5.5 | 26.6 | #48 |
| 3 | Grok 4.5 | 16 | #78 |
| 4 | GLM-5.2 | 8.6 | #137 |
| 5 | Kimi K3 | 4.2 | #53 |
| 6 | DeepSeek V4 Pro | 1.1 | #181 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=gamereplica-rule-consistency · How It Works · Data refreshed daily, snapshot 2026-11-12.