LiveGeo - Safety-Aware: leaderboard
LiveGeo (arXiv 2611.17267v1, Table 3) tests geographic decisions with a shared ReAct agent scaffold and tool interface. Each task allows 200 interaction steps, temperature 0.1 and at most 10,000 generated tokens per turn, retaining the full interaction history. Results average three independent runs per task. The seven static categories contain 1,088 reviewed instances; the dynamic columns use 149 matched instances with the same replayed world timeline and events for every model. Task-specific judges enforce structured submissions and the documented checks. Dynamic success also requires observing the events, replanning correctly and passing at least 70% of applicable checks. The reported value is mean success as a percentage; confidence intervals are retained in the source. Appendix C.1 binds the exact API identities. Reasoning effort and downloadable weights are not inferred. The destination must satisfy safety constraints and supporting-evidence checks. Higher is better.
Metric: Task-specific success rate (%). Source: arxiv.org. Saturation forecast: Around 2030. 9 models tracked. Updated 14 Nov 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 5 (Harness: LiveGeo ReAct) | 19.8 | |
| 2 | Gemini 3.7 Flash (Harness: LiveGeo ReAct) | 17.4 | |
| 3 | Claude Opus 5 (Harness: LiveGeo ReAct) | 17.4 | |
| 4 | Gemini 3.1 Pro Preview (Harness: LiveGeo ReAct) | 15.7 | |
| 5 | DeepSeek-V4-Flash-Vision-Exp (Harness: LiveGeo ReAct) | 13.2 | |
| 6 | Grok 4.5 (Harness: LiveGeo ReAct) | 11.6 | |
| 7 | Qwen 3.8 Max (Harness: LiveGeo ReAct) | 10.7 | |
| 8 | GPT-5.6 Sol | 7.4 | #30 |
| 9 | Kimi K3 | 2.5 | #56 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=livegeo-safety-aware · How It Works · Data refreshed daily, snapshot 2026-11-16.