T2Quant FP16 - HumanEval: leaderboard
T2Quant FP16 - HumanEval. The T2Quant paper evaluates released FP16 checkpoints Dream-org/Dream-v0-Instruct-7B, GSAI-ML/LLaDA-1.5, and GSAI-ML/LLaDA-8B-Instruct with their native chat templates. Every model uses the same held-out problems, separated from calibration data. Sampling uses temperature 0, four tokens unmasked per step, and low-confidence remasking. GSM8K, HumanEval, ARC-Challenge, MMLU-Pro, DROP, and BBH have 512, 100, 512, 512, 512, and 512 evaluation problems; output budgets are 512, 512, 128, 512, 64, and 256 tokens. HumanEval is pass@1. Other tasks use normalized exact match; DROP uses official answer normalization against any annotated span, not F1. Quantized and calibrated derivatives are separate artifacts. Reported metric: Pass@1 (%).
Metric: Pass@1 (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 3 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | LLaDA-1.5-8B (FP16) | 26 | |
| 2 | LLaDA-8B-Instruct (FP16) | 25 | |
| 3 | Dream-v0-Instruct-7B (FP16) | 19 |
Interactive version: theaggregate.ai/benchmark?slug=t2quant-fp16-humaneval · How It Works · Data refreshed daily, snapshot 2026-10-25.