ArabFuncBench - Tool Selection Accuracy: leaderboard
ArabFuncBench (Sadouk and Gadi; dataset card revision c11e4e5e, 2026-09-10) measures zero-shot function calling in Arabic: 1,000 requests over 50 tools in five service domains (education, e-commerce, healthcare, Islamic services, government). Unlike the Berkeley Function Calling Leaderboard, whose requests are English, every request, tool description and argument value is written natively in Modern Standard Arabic rather than translated. The tool definitions (OpenAI JSON schema) go in the prompt and the model must answer with a JSON object; in 200 negative examples no tool fits and a null call is correct. The examples were generated with the Claude Haiku API and 20% were manually validated. Tool Selection Accuracy (0-1) credits the expected function, or a null call on a negative; replies that cannot be parsed and API calls that still fail after three attempts count as a null call, which is right only on the negatives. API models (Together AI's Turbo endpoints, OpenAI, Anthropic) ran at the provider's default temperature; ALLaM, Fanar and AceGPT ran locally in 4-bit NF4 quantization at temperature 0.1 with inputs truncated at 2,048 tokens. Argument Extraction F1 and Language Compliance Rate are not admitted.
Metric: Tool selection accuracy (fraction, 1,000 examples incl. 200 null-call negatives). Source: huggingface.co. Saturation forecast: Rough model projection: around 2026. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Llama 3.3 70B Instruct (Benchmark Sweep Run: source=ArabFuncBench; card row=Llama-3.3-70B; api-model=meta-llama/Llama-3.3-70B-Instruct-Turbo; served by Together AI, Turbo endpoint; prompted JSON tool calls; max tokens=500; temperature=provider default) | 1 | |
| 2 | Qwen 2.5 7B Turbo Instruct (Benchmark Sweep Run: source=ArabFuncBench; card row=Qwen2.5-7B; api-model=Qwen/Qwen2.5-7B-Instruct-Turbo; served by Together AI, Turbo endpoint; prompted JSON tool calls; max tokens=500; temperature=provider default) | 0.99 | |
| 3 | GPT-4o Mini (Benchmark Sweep Run: source=ArabFuncBench; card row=GPT-4o-mini; api-model=gpt-4o-mini; OpenAI API, undated alias; prompted JSON tool calls; max tokens=500; temperature=provider default) | 0.98 | |
| 4 | Claude Haiku 4.5 (Benchmark Sweep Run: source=ArabFuncBench; card row=Claude Haiku; api-model=claude-haiku-4-5; Anthropic API; prompted JSON tool calls; max tokens=500; temperature=provider default) | 0.92 | |
| 5 | ALLaM-7B-Instruct-preview (Benchmark Sweep Run: source=ArabFuncBench; card row=ALLaM-7B; api-model=humain-ai/ALLaM-7B-Instruct-preview; local transformers, bitsandbytes 4-bit NF4; prompted JSON tool calls; input truncated at 2048 tokens; max new tokens=300; temperature=0.1, sampled; instruction in the user turn) | 0.92 | |
| 6 | Fanar-1-9B-Instruct (Benchmark Sweep Run: source=ArabFuncBench; card row=Fanar-9B; api-model=QCRI/Fanar-1-9B-Instruct; local transformers, bitsandbytes 4-bit NF4; prompted JSON tool calls; input truncated at 2048 tokens; max new tokens=300; temperature=0.1, sampled) | 0.87 | |
| 7 | AceGPT-7B-chat (Benchmark Sweep Run: source=ArabFuncBench; card row=AceGPT-7B; api-model=FreedomIntelligence/AceGPT-7B-chat; local transformers, bitsandbytes 4-bit NF4; prompted JSON tool calls; input truncated at 2048 tokens; max new tokens=300; temperature=0.1, sampled) | 0.22 |
Interactive version: theaggregate.ai/benchmark?slug=arabfuncbench-tool-selection-accuracy · How It Works · Data refreshed daily, snapshot 2026-11-12.