StakeBench - Commitment Detection: leaderboard

Market commitment detection in StakeBench (2026): given a prediction-market comment, decide whether its author holds a position in that market, using Polymarket and Manifold records rather than annotator labels. The score is accuracy in percent, macro-averaged over topic and platform splits; higher is better.

Metric: Accuracy (%) on G1: does a comment come from a user holding a position in the market (yes or no, chance 50), one-shot prompt, Polymarket and Manifold comments of resolved markets, macro-average over 18 topic and platform splits, greedy decoding; higher is better. Source: arxiv.org. Saturation forecast: Around December 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Qwen 3 32BNon-reasoning74.7#972
2Qwen 3 14BNon-reasoning74#1269
3DeepSeek R1 Distill Llama 8B72.1#7059
4Gemma 2 9B71.8#3201
5Gemini 2.5 Flash71.1#463
6Claude Haiku 4.569.7#583
7Qwen 3 8BNon-reasoning69.2#1873
8GPT-5.568.9#54
9Qwen 3 30B A3BNon-reasoning67#1156
10Fino1-14B64.8
11Fino1-8B60.1#6784
12Llama 3.2 3B Instruct56.7#7241
13Mistral 7B Instruct (v0.3)53.4#6750
14Fin-Chat-7B53.1
15FinMA-30B50.3

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=stakebench-commitment-detection · How It Works · Data refreshed daily, snapshot 2026-11-21.