MintEval (200-Task Subset): leaderboard

MintEval v0 (arXiv 2610.03080): the model turns a trader's colloquial instruction into a trading-strategy program, which is executed bar by bar against the reference program on identical BTCUSDT 15-minute data and frictions. ActionMatch is the share of active bars (either program in or targeting a position) on which the two programs target the same position; this board uses the all-task ActionMatch_all0 variant, assigning compile failures 0, while the paper’s headline ActionMatch averages only compiled runs. Open setting (instruction and interface only), temperature 0, on the stratified 200-task subset (40 per state-span quintile) that the frontier models ran. Percent, higher is better.

Metric: ActionMatch_all0 (%), open setting, compile failures scored 0, 200 stratified BTCUSDT tasks. Source: github.com. Saturation forecast: Around February 2027. 4 models tracked. Updated 14 Nov 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Opus 5.588.9#9
2Qwen 3.8 MaxLow81.9#66
3DeepSeek V4 ProHigh71.2#198
4GPT-5.4 Mini54.3#405

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=minteval-200-task-subset · How It Works · Data refreshed daily, snapshot 2026-11-20.