llm-jp-4-8B-Thinking: benchmark results
LLM-jp 4 instruction-tuned release developed by NII's LLM-jp project, using Llama architecture and a 65,536-token context window. This is the thinking post-trained checkpoint, not the separately released base model. Provider: National Institute of Informatics. Released 2026-03-16. Access: Open.
Aggregate Rating 1574 ± 15, rank #529 of 1635 rated models, from 80 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| Open Japanese LLM v2 - JCoLA In-Domain | 79.65 | Accuracy (%) | 100 |
| LLM-jp4.1 Release Swallow - JEMHopQA | 0.66 | Score (0–1) | 96.2 |
| LLM-jp4.1 Release Swallow - JHumanEval @10 | 0.96 | Pass@10 (0–1) | 94.2 |
| Open Japanese LLM v2 - JBLiMP | 83.99 | Accuracy (%) | 87.5 |
| Open Japanese LLM v2 - JCoLA Out-of-Domain | 81.35 | Accuracy (%) | 87.5 |
| Swallow - English MT-Bench - Extraction | 81.8 | Judge Score (normalized, %) | 80.6 |
| FrameBench - Frame Identification - Japanese | 75 | Accuracy (%; Japanese FrameNet candidate frames) | 80 |
| Swallow - Japanese MT-Bench - Writing | 67 | Judge Score (normalized, %) | 75.4 |
| Open Japanese LLM v2 - M-IFEval-Ja | 45.35 | Prompt-Level Strict Accuracy (%) | 75 |
| LLM-jp4.1 Release Swallow - HumanEval+ @10 | 0.98 | Pass@10 (0–1) | 73.1 |
| Swallow - Post-trained Japanese - M-IFEval-Ja | 67.7 | Instruction-Level Strict Accuracy (%) | 70.1 |
| Swallow - Japanese MT-Bench - Extraction | 73.4 | Judge Score (normalized, %) | 69.4 |
Interactive version: theaggregate.ai/model?slug=llm-jp-4-8b-thinking · How It Works · Data refreshed daily, snapshot 2026-11-05.