{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2025:7I3KCDAMH7KICXH44C26P7ZB2S","short_pith_number":"pith:7I3KCDAM","schema_version":"1.0","canonical_sha256":"fa36a10c0c3fd4815cfce0b5e7ff21d4899fadcd9832063dbbb245ef82a6109b","source":{"kind":"arxiv","id":"2503.10497","version":2},"attestation_state":"computed","paper":{"title":"MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Aosong Feng, Dairui Liu, Douglas Teodoro, Edison Marrese-Taylor, Fan Gao, Felix Juefei-Xu, Foutse Khomh, Heli Qi, Huitao Li, Irene Li, Jinghui Lu, Junjue Wang, Kunyu Yu, Lei Ma, Nan Liu, Osamu Yoshie, Qingcheng Zeng, Qingyu Chen, Randy Goebel, Ruihai Dong, Rui Yang, Shangding Gu, Shijian Lu, Weihao Xuan, Xiaofei Xie, Xin Li, Yuang Jiang, Yuekang Li, Yun Xing, Yunze Xiao, Yusuke Iwasawa, Yutaka Matsuo","submitted_at":"2025-03-13T15:59:20Z","abstract_excerpt":"Existing large language model (LLM) evaluation benchmarks primarily focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-linguistic reasoning abilities. This dual limitation makes it challenging to comprehensively assess LLMs' performance in the multilingual setting. To fill this gap, we introduce MMLU-ProX, a comprehensive benchmark covering 29 languages, built on an English benchmark. Each language version consists of 11,829 identical questions, enabling direct cross-linguistic comparisons. Additionally, to meet efficient evaluation needs,"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2503.10497","kind":"arxiv","version":2},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2025-03-13T15:59:20Z","cross_cats_sorted":[],"title_canon_sha256":"68a0aa5742c827d7183a6af5290d1f7a777f3389625c8029b370f1b0268fca6f","abstract_canon_sha256":"f580cce5636af75d19cfc01a218c8a74598ae2c985f3478139807a80c1b3a09b"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T11:09:46.079607Z","signature_b64":"xu8vLhII9IEHxjr5kI4a6DU5qSyFG6PrIQT0iMSEoXDX49vmGxyl76w6RmDKzD2shOrfJsR2T78Askcn1eBRAw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"fa36a10c0c3fd4815cfce0b5e7ff21d4899fadcd9832063dbbb245ef82a6109b","last_reissued_at":"2026-07-05T11:09:46.079101Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T11:09:46.079101Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Aosong Feng, Dairui Liu, Douglas Teodoro, Edison Marrese-Taylor, Fan Gao, Felix Juefei-Xu, Foutse Khomh, Heli Qi, Huitao Li, Irene Li, Jinghui Lu, Junjue Wang, Kunyu Yu, Lei Ma, Nan Liu, Osamu Yoshie, Qingcheng Zeng, Qingyu Chen, Randy Goebel, Ruihai Dong, Rui Yang, Shangding Gu, Shijian Lu, Weihao Xuan, Xiaofei Xie, Xin Li, Yuang Jiang, Yuekang Li, Yun Xing, Yunze Xiao, Yusuke Iwasawa, Yutaka Matsuo","submitted_at":"2025-03-13T15:59:20Z","abstract_excerpt":"Existing large language model (LLM) evaluation benchmarks primarily focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-linguistic reasoning abilities. This dual limitation makes it challenging to comprehensively assess LLMs' performance in the multilingual setting. To fill this gap, we introduce MMLU-ProX, a comprehensive benchmark covering 29 languages, built on an English benchmark. Each language version consists of 11,829 identical questions, enabling direct cross-linguistic comparisons. Additionally, to meet efficient evaluation needs,"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2503.10497","kind":"arxiv","version":2},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2503.10497/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2503.10497","created_at":"2026-07-05T11:09:46.079159+00:00"},{"alias_kind":"arxiv_version","alias_value":"2503.10497v2","created_at":"2026-07-05T11:09:46.079159+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2503.10497","created_at":"2026-07-05T11:09:46.079159+00:00"},{"alias_kind":"pith_short_12","alias_value":"7I3KCDAMH7KI","created_at":"2026-07-05T11:09:46.079159+00:00"},{"alias_kind":"pith_short_16","alias_value":"7I3KCDAMH7KICXH4","created_at":"2026-07-05T11:09:46.079159+00:00"},{"alias_kind":"pith_short_8","alias_value":"7I3KCDAM","created_at":"2026-07-05T11:09:46.079159+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":7,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2605.30107","citing_title":"Dial HEALTHDIAL for Advice: A Multilingual and Multi-Parallel Spoken Dialogue Dataset for Knowledge-Grounded Information Seeking","ref_index":5,"is_internal_anchor":false},{"citing_arxiv_id":"2606.19640","citing_title":"Creating Multilingual Mental Health Dialogue Datasets: Limits of Persona-Based Localization via Nationality and Language","ref_index":44,"is_internal_anchor":false},{"citing_arxiv_id":"2605.22567","citing_title":"LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance","ref_index":53,"is_internal_anchor":false},{"citing_arxiv_id":"2510.27269","citing_title":"Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?","ref_index":4,"is_internal_anchor":false},{"citing_arxiv_id":"2605.14040","citing_title":"Physics-R1: An Audited Olympiad Corpus and Recipe for Visual Physics Reasoning","ref_index":45,"is_internal_anchor":false},{"citing_arxiv_id":"2604.20720","citing_title":"COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Sampling","ref_index":79,"is_internal_anchor":false},{"citing_arxiv_id":"2604.06903","citing_title":"Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus","ref_index":13,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/7I3KCDAMH7KICXH44C26P7ZB2S","json":"https://pith.science/pith/7I3KCDAMH7KICXH44C26P7ZB2S.json","graph_json":"https://pith.science/api/pith-number/7I3KCDAMH7KICXH44C26P7ZB2S/graph.json","events_json":"https://pith.science/api/pith-number/7I3KCDAMH7KICXH44C26P7ZB2S/events.json","paper":"https://pith.science/paper/7I3KCDAM"},"agent_actions":{"view_html":"https://pith.science/pith/7I3KCDAMH7KICXH44C26P7ZB2S","download_json":"https://pith.science/pith/7I3KCDAMH7KICXH44C26P7ZB2S.json","view_paper":"https://pith.science/paper/7I3KCDAM","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2503.10497&json=true","fetch_graph":"https://pith.science/api/pith-number/7I3KCDAMH7KICXH44C26P7ZB2S/graph.json","fetch_events":"https://pith.science/api/pith-number/7I3KCDAMH7KICXH44C26P7ZB2S/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/7I3KCDAMH7KICXH44C26P7ZB2S/action/timestamp_anchor","attest_storage":"https://pith.science/pith/7I3KCDAMH7KICXH44C26P7ZB2S/action/storage_attestation","attest_author":"https://pith.science/pith/7I3KCDAMH7KICXH44C26P7ZB2S/action/author_attestation","sign_citation":"https://pith.science/pith/7I3KCDAMH7KICXH44C26P7ZB2S/action/citation_signature","submit_replication":"https://pith.science/pith/7I3KCDAMH7KICXH44C26P7ZB2S/action/replication_record"}},"created_at":"2026-07-05T11:09:46.079159+00:00","updated_at":"2026-07-05T11:09:46.079159+00:00"}