Open Japanese LLM v2 - M-IFEval-Ja: leaderboard
From LLM-jp's Open Japanese LLM Leaderboard v2, which runs open-weight models through llm-jp-eval v2 with vLLM. M-IFEval-Ja is the Japanese split of Lightblue's multilingual IFEval, prompts with verifiable formatting instructions run zero-shot; metric is prompt-level strict accuracy, so every instruction in a prompt must be followed. Upstream 0-1 scores are shown as 0-100, higher is better.
Metric: Prompt-Level Strict Accuracy (%). Source: huggingface.co. Saturation forecast: Around March 2028. 27 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | llm-jp-4.1-32B-a3B (Thinking) | 56.4 | #366 (llm-jp-4.1-32B-a3B) |
| 2 | llm-jp-4.1-33B (Thinking) | 53.49 | #256 (llm-jp-4.1-33B) |
| 3 | llm-jp-4.1-8B (Thinking) | 50.58 | #408 (llm-jp-4.1-8B) |
| 4 | Llama 3.3 70B Instruct | 47.09 | #541 |
| 5 | llm-jp-4-8B-Thinking | 45.35 | #529 |
| 6 | GPT-OSS-120B | 44.77 | #346 |
| 7 | llm-jp-4-32B-a3B-Thinking | 44.77 | #477 |
| 8 | Qwen 3.6 27B (Non-reasoning) | 44.19 | #182 (Qwen 3.6 27B) |
| 9 | Qwen 3 8B (Thinking) | 43.6 | #678 (Qwen 3 8B) |
| 10 | Qwen 3.5 27B (Non-reasoning) | 41.86 | #192 (Qwen 3.5 27B) |
| 11 | gemma-4-26B-A4B-it (Thinking) | 40.12 | |
| 12 | Llama-3.3-Swallow-70B-Instruct-v0.4 | 37.79 | #540 |
| 13 | Qwen 3 32B (Non-reasoning) | 37.21 | #449 (Qwen 3 32B) |
| 14 | Gemma 3 27B (IT) | 34.88 | #528 |
| 15 | GPT-OSS-20B | 33.14 | #523 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=open-japanese-llm-v2-m-ifeval-ja · How It Works · Data refreshed daily, snapshot 2026-11-05.