RedHat Gemma4 FP8 - Thinking - IFEval (0-shot, inst-level strict): leaderboard
RedHatAI Gemma4-31B-IT FP8-dynamic model-card evaluation at revision d4ab4f579dd3516f97d8a6a4c98d0653480bad15. Original google/gemma-4-31B-it and exact RedHatAI FP8-dynamic checkpoint remain distinct. Source reports scores averaged across three seed runs (1234,2345,3456), eight for AIME2025. vLLM OpenAI-compatible serving; server context32768. lm-eval IFEval/GSM8K Platinum/MMLU-Pro: zero-shot, temperature1.0, top_p0.95, top_k64, max_gen_toks32000. lighteval MATH500/AIME2025/GPQA Diamond/LiveCodeBenchv6: zero-shot, same sampling, requested max_new_tokens65536. The source does not resolve the lighteval requested-token/server-context mismatch; these are reported settings, not an inferred effective65536-token budget. BFCLv4 uses native function calling and all categories with thinking enabled; no separate BFCL generation budget is specified. No harness commit or evaluated checkpoint revision is reported. Card revision pins the evidence, not a model-weight evaluation revision. Recovery ratios are diagnostics and are not additional capability outcomes. Mode: With thinking. Native outcome: IFEval (0-shot, inst-level strict). Metric: Native accuracy (%).
Metric: Native accuracy (%). Source: huggingface.co. Saturation forecast: Around January 2027. 2 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 4 31B (IT) (Thinking) | 96.44 | #181 (Gemma 4 31B (IT)) |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=redhat-gemma4-fp8-thinking-ifeval-0-shot-inst-level-strict · How It Works · Data refreshed daily, snapshot 2026-10-27.