{"paper":{"title":"IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"IslamicMMLU introduces a benchmark of 10,013 questions to test LLMs on Quran, Hadith, and Fiqh.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Ali Abdelaal, Mahmoud Fawzi, Mohammed Nader Al Haffar, Walid Magdy","submitted_at":"2026-03-24T22:18:16Z","abstract_excerpt":"Large language models are increasingly consulted for Islamic knowledge, yet no comprehensive benchmark evaluates their performance across core Islamic disciplines. We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (jurisprudence, 4,000 questions). Each track is formed of multiple types of questions to examine LLMs capabilities handling different aspects of Islamic knowledge. The benchmark is used to create the IslamicMMLU public leaderboard for evaluating LLMs, and we initially evaluate 2"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (4,000 questions).","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The questions accurately and representatively capture core Islamic knowledge without introducing selection bias or factual errors in their creation and translation.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"IslamicMMLU is a new benchmark with 10,013 questions across Quran, Hadith, and Fiqh tracks that measures LLM accuracy on Islamic topics, with top models at 93.8% and notable variation by track and model type.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"IslamicMMLU introduces a benchmark of 10,013 questions to test LLMs on Quran, Hadith, and Fiqh.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"45d8ae3c340197f6897f93d52b3e51da95d30b79e2ec096ee0491ff29749cc75"},"source":{"id":"2603.23750","kind":"arxiv","version":3},"verdict":{"id":"f6097c35-1796-4136-be9b-a4d04b81fda7","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T00:03:54.971051Z","strongest_claim":"We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (4,000 questions).","one_line_summary":"IslamicMMLU is a new benchmark with 10,013 questions across Quran, Hadith, and Fiqh tracks that measures LLM accuracy on Islamic topics, with top models at 93.8% and notable variation by track and model type.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The questions accurately and representatively capture core Islamic knowledge without introducing selection bias or factual errors in their creation and translation.","pith_extraction_headline":"IslamicMMLU introduces a benchmark of 10,013 questions to test LLMs on Quran, Hadith, and Fiqh."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2603.23750/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}