llm-jp-4-32B-a3B-Thinking: benchmark results
LLM-jp 4 instruction-tuned release developed by NII's LLM-jp project, using Qwen3 mixture-of-experts architecture and a 65,536-token context window. This is the thinking post-trained checkpoint, not the separately released base model. Provider: National Institute of Informatics. Released 2026-03-16. Access: Open.
Aggregate Rating 1591 ± 13, rank #477 of 1635 rated models, from 174 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| Open Japanese LLM v2 - JCulture-MCQ | 55.22 | Set F1 (%) | 100 |
| Open Japanese LLM v2 - JFinQA | 67.5 | Accuracy (%) | 100 |
| Nejumi 4 - BFCL - Irrelevance Detection | 93.33 | Accuracy (%) | 91.7 |
| Nejumi 4 - MT-Bench (Japanese) - Humanities | 100 | Judge rating (1-10, x10) | 91 |
| Nejumi 4 - MT-Bench (Japanese) - Math | 100 | Judge rating (1-10, x10) | 89.6 |
| Nejumi 4 - jaster (0-shot) - JCoLA (out-of-domain) | 87 | Exact match (%) | 81.2 |
| Swallow - Japanese MT-Bench - Extraction | 75 | Judge Score (normalized, %) | 80.6 |
| LLM-jp4.1 Release Swallow - JEMHopQA | 0.64 | Score (0–1) | 78.8 |
| Nejumi 4 - Toxicity - Violation Categories | 47.62 | Criteria met (%) | 76.4 |
| Swallow - English MT-Bench - Extraction | 80.5 | Judge Score (normalized, %) | 76.1 |
| LLM-jp4.1 Release Judge - MT-Bench (JA) | 7.82 | Judge score (1–10) | 75 |
| LLM-jp4.1 Release Swallow - WMT20 EN→JA | 0.24 | BLEU (source-normalized 0–1) | 75 |
Interactive version: theaggregate.ai/model?slug=llm-jp-4-32b-a3b-thinking · How It Works · Data refreshed daily, snapshot 2026-11-05.