Ultralytics VLM Leaderboard: leaderboard
The Ultralytics VLM Leaderboard (suite ul37-v2) ranks vision language models and open-vocabulary detectors on drawing boxes around named objects, the task behind auto-annotation. Unlike COCO or RefCOCO, which use everyday photographs, its 360 images come 10 each from 36 real-world detection datasets on Ultralytics Platform, among them dental X-rays, retinal lesions, retail products and baggage scans, sent whole with each dataset's own class names. Every image goes through the platform's production annotation call, and SAM 3.1 tightens every model's boxes except SAM 3's. A timeout, refusal or invalid output counts as an image with no predictions. Boxes are scored with the Ultralytics package's detection metrics in the order the model listed them, and the datasets are averaged equally. Providers run at their default reasoning effort unless that misses the platform's 115-second request limit. This board is mAP50: average precision at an IoU of 0.5, averaged over the labeled classes.
Metric: mAP50 over 36 UL37 detection datasets (0-1). Source: www.ultralytics.com. Saturation forecast: Around 2029. 44 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 4 Carbon | 0.58 | #2 |
| 2 | GPT-6.1 Terra | 0.56 | #19 |
| 3 | Claude Fable 5.2 | 0.54 | #1 |
| 4 | Gemini 4 Argon | 0.54 | #11 |
| 5 | Claude Opus 5.5 | 0.53 | #5 |
| 6 | GPT-6 Astra | 0.52 | #6 |
| 7 | Gemini 3.8 Flash | 0.52 | #36 |
| 8 | GPT-6.1 Sol | 0.51 | #10 |
| 9 | Claude Sonnet 5.5 | 0.47 | #18 |
| 10 | Qwen 3.8 Max | 0.46 | #55 |
| 11 | Kimi K3.1 Pro | 0.44 | #28 |
| 12 | Muse Ember | 0.43 | #54 |
| 13 | Claude Haiku 5.5 | 0.43 | #64 |
| 14 | Claude Fable 5.1 | 0.43 | #14 |
| 15 | Qwen3.8-Flash-Next | 0.39 | #115 |
Interactive version: theaggregate.ai/benchmark?slug=ultralytics-vlm-leaderboard · How It Works · Data refreshed daily, snapshot 2026-11-12.