llm-jp-3-3.7B: benchmark results
National Institute of Informatics' LLM-jp-3 3.7B transformer model, part of Japan's LLM-jp research project, pretrained on a blend of datasets totaling 2.1 trillion tokens seen. Provider: National Institute of Informatics. Released 2024-09-23. Access: Open.
Unified ELO 1309 ± 33, rank #1448 of 1639 rated models, from 76 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| Open Japanese LLM - Mbpp Pylint Check | 99 | Score (%) | 98.2 |
| Open Japanese LLM - AIO Char F1 | 74.01 | Score (%) | 86.9 |
| Open Japanese LLM - QA | 57.01 | Score (%) | 75.7 |
| Open Japanese LLM - Niilc Char F1 | 53.99 | Score (%) | 75 |
| Open Japanese LLM - Wiki Reading Char F1 | 80.9 | Score (%) | 74.3 |
| pfgen-bench - Completion Mode - Helpfulness | 0.19 | Helpfulness Score | 62.9 |
| pfgen-bench - Completion Mode - Score | 0.54 | pfgen Score (mean of three) | 62.9 |
| pfgen-bench - Completion Mode - Fluency | 0.65 | Fluency Score | 62.5 |
| pfgen-bench - Completion Mode - Truthfulness | 0.8 | Truthfulness Score | 61.8 |
| Swallow - Pre-trained Japanese - NIILC | 54.1 | Character F1 (%) | 57.1 |
| Open Japanese LLM - FA | 24.01 | Score (%) | 42.8 |
| Open Japanese LLM - Wiki NER SET F1 | 3.54 | Score (%) | 39.4 |
Interactive version: theaggregate.ai/model?slug=llm-jp-3-3-7b · How It Works · Data refreshed daily, snapshot 2026-10-09.