EditJudge-Bench - Transfer - EditRewardBench: leaderboard
EditJudge-Bench v1,1382curatedBlender before/after pairs,factual/counterfactual instructions. NativeAUROC or multiplechoice factual-instruction ranking percent. No parsing-failure dropping:asymmetric score2fail factual,8failcounterfactual;unsupportedlogitoperation.5;APIerrors score5;zero outputs retained. Exact publishedTOML backend/model/prompt/image-order/budget retained; source-unreported launcherflags remain unknown. Gemma31B is NVIDIA NVFP4;QwenThinking is its distinct checkpoint. Supervised judges retain own trainedartifact IDs,not their basemodels. EditRewardBench:1453strict-human-preference pairs,native agreementaccuracy,not rankcolumn. Transfer-specific logprob-simple/noq configs remain distinct.
Metric: Human Agreement (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 11 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | EditReward-Qwen2.5-VL-7B (EditJudge Run: config=editreward__qwen2.5-vl-7b;backend=editreward) | 60.6 | |
| 2 | Gemma-4-31B-IT-NVFP4 (EditJudge Run: config=vie__gemma-4-31b;backend=vllm,template=vie_tie_0shot,image_position=after,max_tokens=16,max_model_len=4096) | 59.5 | |
| 3 | Qwen3-VL-32B-Instruct (EditJudge Run: config=vie__qwen3-vl-32b;backend=vllm,template=vie_tie_0shot,image_position=after,max_tokens=16,max_model_len=4096) | 59.1 | |
| 4 | Qwen3-VL-Reranker-8B (EditJudge Run: config=reranker__qwen3-vl-reranker-8b;backend=reranker) | 58.3 | |
| 5 | Gemma-4-31B-IT-NVFP4 (EditJudge Run: config=logprob__gemma-4-31b;backend=vllm,template=binary_q_images_last,logprobs=True,max_model_len=4096) | 57.6 | |
| 6 | Qwen3-VL-8B-Instruct (EditJudge Run: config=logprob-noq__qwen3-vl-8b;backend=vllm,template=binary_tagged_images_last,logprobs=True,max_model_len=4096) | 56.8 | |
| 7 | Qwen2.5-VL-7B-Instruct (EditJudge Run: config=logprob-simple__qwen2.5-vl-7b;backend=vllm,template=binary_simple,image_position=before,logprobs=True,max_model_len=4096;unlogged_runtime_fields=image_position;values=reviewed_config_defaults_not_confirmed_launch) | 56.2 | |
| 8 | Qwen3-VL-8B-Instruct (EditJudge Run: config=vie__qwen3-vl-8b;backend=vllm,template=vie_tie_0shot,image_position=after,max_tokens=16,max_model_len=4096) | 56.1 | |
| 9 | Qwen3-VL-4B-Instruct (EditJudge Run: config=vie__qwen3-vl-4b;backend=vllm,template=vie_tie_0shot,image_position=after,max_tokens=16,max_model_len=4096) | 55.8 | |
| 10 | Qwen2.5-VL-7B-Instruct (EditJudge Run: config=vie__qwen2.5-vl-7b;backend=vllm,template=vie_tie_0shot,image_position=after,max_tokens=16,max_model_len=4096;unlogged_runtime_fields=all_launcher_flags;values=reviewed_config_defaults_not_confirmed_launch) | 50.7 |
Interactive version: theaggregate.ai/benchmark?slug=editjudge-bench-transfer-editrewardbench · How It Works · Data refreshed daily, snapshot 2026-11-12.