SynopticBench: leaderboard

SynopticBench (2026) tests whether vision-language models can write weather forecast discussions from numerical weather prediction charts over North America, with National Weather Service area forecast discussions as references. This board is an LLM-judge score from 0 to 1 (Gemini 2.5 Flash judge) on the 2024 and 2025 test years; higher is better.

Metric: LLM-judge score (0-1, Gemini-2.5-Flash judge) of a generated weather forecast discussion against the National Weather Service area forecast discussion, from forecast-chart images over North America, 2024-2025 test years; higher is better. Source: arxiv.org. Saturation forecast: Around June 2027. 5 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3.1 Pro (Preview)0.4185#99
2Llama 3.2 11B Vision (SynopticBench Checkpoint Unspecified)0.2623
3Qwen 2.5 VL 7B (SynopticBench checkpoint unspecified)0.2566
4LLaVA-v1.5-13B0.1896#6926
5LLaVA-v1.5-7B0.1565#7630

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=synopticbench · How It Works · Data refreshed daily, snapshot 2026-11-21.