OmniToM - Belief Extraction: leaderboard
OmniToM (2026) evaluates theory of mind through explicit belief modeling on 895 short social stories from ToMBench, with gold belief propositions built by a human-calibrated annotation pipeline. This board is Stage 1, extracting the world facts and the beliefs of each actor (macro F1 against the gold propositions, matched by an LLM judge, averaged over the seven story categories); higher is better.
Metric: Belief-extraction macro F1 (%): story-level F1 between the extracted world-fact and actor-belief propositions and the gold propositions, matched semantically by a GPT-5 judge, macro-averaged over stories, unweighted mean of the seven story categories, on the 895 OmniToM stories from seven ToMBench categories (22,343 gold belief propositions), zero-shot TELeR level 3 prompts; GPT-5 is not scored as it is the judge; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 8 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 3 27B (IT) | 57.69 | #1324 |
| 2 | Mistral Small 3 | 56 | #1796 |
| 3 | Gemini 2.5 Flash | 54.97 | #463 |
| 4 | Mistral Large 2 (Jul) | 54.1 | #1144 |
| 5 | Qwen 3 32B | 51.63 | #972 |
| 6 | Llama 3.3 70B Instruct | 47.24 | #1385 |
| 7 | Qwen 3 8B | 43.41 | #1873 |
| 8 | Llama 3.1 8B Instruct | 37.42 | #4600 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=omnitom-belief-extraction · How It Works · Data refreshed daily, snapshot 2026-11-21.