NVIDIA Nemotron 49B v1.5 Reproduction - bfcl_v3 - live_irrelevance: leaderboard
Berkeley Function Calling Leaderboard v3, live irrelevance detection, where the model should make no call, 882 entries; accuracy in percent for reasoning on and off. From NVIDIA's NeMo-Skills tutorial reproducing the Llama-3.3-Nemotron-Super-49B-v1.5 evaluations, with tool calls parsed server-side by the model's llama_nemotron_json parser; a run separate from the model card's results. Reasoning on uses temperature 0.6, top-p 0.95 and an empty system prompt; reasoning off uses temperature 0, top-p 1.0 and the /no_think system prompt; both allow up to 65,536 generated tokens.
Metric: Accuracy (%). Source: raw.githubusercontent.com. Saturation forecast: Around February 2027. 2 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Llama-3.3-nemotron-super-49B-v1.5 (Non-reasoning) | 85.6 | #903 (Llama-3.3-nemotron-super-49B-v1.5) |
| 2 | Llama-3.3-nemotron-super-49B-v1.5 (Reasoning) | 84.47 | #903 (Llama-3.3-nemotron-super-49B-v1.5) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=nvidia-nemotron-49b-v1-5-reproduction-bfcl-v3-live-irrelevance · How It Works · Data refreshed daily, snapshot 2026-11-12.