ForecastGym - Last-state Accuracy (Tavily evidence): leaderboard
ForecastGym replays frozen web-search evidence for 150 resolved Polymarket and Kalshi questions (opened 14 September 2026) so every model forecasts from identical information: the score is the share of forecasts on the right side of 50% at each question's last evidence state, given Tavily's recorded results plus the model's own earlier forecasts and notes. Four models; the market's own price scores 71.9% on the same states.
Metric: Share of last-state forecasts on the correct side of 0.5 (%, read x100 from the printed 0-1 share; higher is better; a forecast of exactly 0.5 counts as half; ForecastGym first batch: 150 binary Polymarket and Kalshi questions opened 14 September 2026 and collected for 14 days, 614 frozen evidence states; the model reads each question's states offline in chronological order (main condition: Tavily's recorded evidence plus its previous forecast, rationale and persistent notes), one JSON-schema request per state with the same prompt and settings for every model; reasoning or thinking settings are not stated). Source: arxiv.org. Saturation forecast: Around February 2027. 4 models tracked. Updated 19 Nov 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V4.1 Flash | 66 | #69 |
| 2 | GPT-5.6 Terra | 64.2 | #72 |
| 3 | Gemini 3.8 Flash | 62.8 | #43 |
| 4 | Claude Sonnet 5 | 59.7 | #118 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=forecastgym-last-state-accuracy-tavily-evidence · How It Works · Data refreshed daily, snapshot 2026-11-20.