llm-jp-3-1.8B: benchmark results
National Institute of Informatics' LLM-jp-3 1.8B transformer model, part of Japan's LLM-jp research project, pretrained on a blend of datasets totaling 2.1 trillion tokens seen. Provider: National Institute of Informatics. Released 2024-09-23. Access: Open.
Aggregate Rating 1270 ± 30, rank #1495 of 1635 rated models, from 76 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| Open Japanese LLM - AIO Char F1 | 62.17 | Score (%) | 77.2 |
| pfgen-bench - Completion Mode - Truthfulness | 0.73 | Truthfulness Score | 49.5 |
| pfgen-bench - Completion Mode - Fluency | 0.57 | Fluency Score | 49.2 |
| pfgen-bench - Completion Mode - Score | 0.47 | pfgen Score (mean of three) | 49.2 |
| Open Japanese LLM - Mbpp Pylint Check | 27.11 | Score (%) | 48.2 |
| pfgen-bench - Completion Mode - Helpfulness | 0.11 | Helpfulness Score | 47.4 |
| Open Japanese LLM - Wiki Reading Char F1 | 74.7 | Score (%) | 45.7 |
| Open Japanese LLM - QA | 50.49 | Score (%) | 44 |
| Open Japanese LLM - Niilc Char F1 | 44.13 | Score (%) | 39.2 |
| Swallow - Pre-trained Japanese - JEMHopQA | 46.3 | Character F1 (%) | 37.8 |
| Swallow - Pre-trained Japanese - NIILC | 44.9 | Character F1 (%) | 36.7 |
| Open Japanese LLM - Jemhopqa Char F1 | 45.16 | Score (%) | 30.8 |
Interactive version: theaggregate.ai/model?slug=llm-jp-3-1-8b · How It Works · Data refreshed daily, snapshot 2026-11-05.