MERRIN (Agentic Search): leaderboard

MERRIN (2026) asks 162 human-written questions whose single short answers depend on non-text web evidence (images, video, audio or tables) found among noisy and conflicting sources, with no hint of the modality in the question. Agentic setting in the multimodal search agent of the paper, whose page-reading and video tools call Gemini 3 Flash: accuracy over three runs, judged against the gold answer, in percent; higher is better.

Metric: Accuracy (%) in the paper's smolagents multimodal search agent (Serper web search, plus page-reading and video-watching tools that call Gemini 3 Flash), on MERRIN's 162 human-annotated questions with unique short answers that need non-text web evidence (image, video, audio or table) and multi-hop reasoning or conflict resolution across noisy sources; accuracy against the gold answer by an LLM judge with the BrowseComp prompt, mean of three runs; higher is better. Source: arxiv.org. Saturation forecast: Around September 2027. 9 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3.1 Pro (Preview)40.1#99
2Gemini 3 Pro39.9#145
3Gemini 3 Flash32.9#175
4GPT-5.4 Nano31.9#651
5GPT-5.4 Mini31.1#406
6Gemini 3.1 Flash Lite26.3#371
7Qwen 3 235B A22B23.3#628
8Qwen 3 30B A3B16.1#1156
9Qwen 3 4B10.5#2348

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=merrin-agentic-search · How It Works · Data refreshed daily, snapshot 2026-11-21.