{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:JOQLOVT2PYLOUXAO6J5ENB2DIP","short_pith_number":"pith:JOQLOVT2","schema_version":"1.0","canonical_sha256":"4ba0b7567a7e16ea5c0ef27a46874343f35409355b95b7c151361dcf68af325f","source":{"kind":"arxiv","id":"2411.19799","version":1},"attestation_state":"computed","paper":{"title":"INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Abraham Diress, Aditya Kumar Dalmia, Alfonso Amayuelas, Angelika Romanou, Anna Sotnikova, Antoine Bosselut, Arshia Soltani Moakhar, Ayush Kumar Tarun, Azmine Toushik Wasi, Azril Hafizi Amirudin, Bardia Soltani Moakhar, B\\\"orje F. Karlsson, Christopher Klamm, Daniel Fernando Erazo Florez, Daniil Dzenhaliou, Danylo Boiko, Debjit Paul, Dominik Krzemi\\'nski, Drishti Sharma, Eldar Khalilov, Esther Ploeger, Fabian Farestam, Fajri Koto, Gabriel Adriano de Melo, Gal Cohen, Imanol Schlag, Jebish Purbey, Jekaterina Novikova, Jenny Chim, Joel Niklaus, Johan Samir Obando Ceron, Joseph Marvin Imperial, Maral Jabbarishiviari, Marjana Prifti Skenduli, Marzieh Fadaee, Michael Chang, Micol Altomare, Mike Zhang, Mohamed A. Haggag, Negar Foroutan, Perttu Isotalo, Ran Tamir, Rishabh Maheshwary, Roshan Santhosh, Sara Hooker, Sara Rydell, Selvan Sunitha Ravi, Serhan Yilmaz, Sharad Duwal, Shayekh Bin Islam, Shivalika Singh, Snegha A, Sree Harsha Nelaturu, Swati Rajwal, Syrielle Montariol, Thenuka Ovin Weerasinghe, Viraat Aryabumi, Yiyang Nan, Zeming Chen","submitted_at":"2024-11-29T16:03:14Z","abstract_excerpt":"The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2411.19799","kind":"arxiv","version":1},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2024-11-29T16:03:14Z","cross_cats_sorted":[],"title_canon_sha256":"6f83125031030729baf4c1958b5c7710e4c474554dafa7c7a88ddd4435a27888","abstract_canon_sha256":"4c2fc4a6b544e9cf97c6347c0c5a76b505d1ef1535b8c18c4943e13b5f92ba35"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T09:42:08.567327Z","signature_b64":"dVxm2c/+8665QCnCWP45urjGbXyh3ulqNn5qVSwur0H2Ottg+xV4mK/fpM+eKPb/+GLejFnXQnyqiF9CFQsPAw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"4ba0b7567a7e16ea5c0ef27a46874343f35409355b95b7c151361dcf68af325f","last_reissued_at":"2026-07-05T09:42:08.566653Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T09:42:08.566653Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Abraham Diress, Aditya Kumar Dalmia, Alfonso Amayuelas, Angelika Romanou, Anna Sotnikova, Antoine Bosselut, Arshia Soltani Moakhar, Ayush Kumar Tarun, Azmine Toushik Wasi, Azril Hafizi Amirudin, Bardia Soltani Moakhar, B\\\"orje F. Karlsson, Christopher Klamm, Daniel Fernando Erazo Florez, Daniil Dzenhaliou, Danylo Boiko, Debjit Paul, Dominik Krzemi\\'nski, Drishti Sharma, Eldar Khalilov, Esther Ploeger, Fabian Farestam, Fajri Koto, Gabriel Adriano de Melo, Gal Cohen, Imanol Schlag, Jebish Purbey, Jekaterina Novikova, Jenny Chim, Joel Niklaus, Johan Samir Obando Ceron, Joseph Marvin Imperial, Maral Jabbarishiviari, Marjana Prifti Skenduli, Marzieh Fadaee, Michael Chang, Micol Altomare, Mike Zhang, Mohamed A. Haggag, Negar Foroutan, Perttu Isotalo, Ran Tamir, Rishabh Maheshwary, Roshan Santhosh, Sara Hooker, Sara Rydell, Selvan Sunitha Ravi, Serhan Yilmaz, Sharad Duwal, Shayekh Bin Islam, Shivalika Singh, Snegha A, Sree Harsha Nelaturu, Swati Rajwal, Syrielle Montariol, Thenuka Ovin Weerasinghe, Viraat Aryabumi, Yiyang Nan, Zeming Chen","submitted_at":"2024-11-29T16:03:14Z","abstract_excerpt":"The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2411.19799","kind":"arxiv","version":1},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2411.19799/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2411.19799","created_at":"2026-07-05T09:42:08.566737+00:00"},{"alias_kind":"arxiv_version","alias_value":"2411.19799v1","created_at":"2026-07-05T09:42:08.566737+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2411.19799","created_at":"2026-07-05T09:42:08.566737+00:00"},{"alias_kind":"pith_short_12","alias_value":"JOQLOVT2PYLO","created_at":"2026-07-05T09:42:08.566737+00:00"},{"alias_kind":"pith_short_16","alias_value":"JOQLOVT2PYLOUXAO","created_at":"2026-07-05T09:42:08.566737+00:00"},{"alias_kind":"pith_short_8","alias_value":"JOQLOVT2","created_at":"2026-07-05T09:42:08.566737+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":11,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2606.06712","citing_title":"Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation","ref_index":44,"is_internal_anchor":false},{"citing_arxiv_id":"2605.27015","citing_title":"PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions","ref_index":16,"is_internal_anchor":false},{"citing_arxiv_id":"2605.07731","citing_title":"Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs","ref_index":58,"is_internal_anchor":false},{"citing_arxiv_id":"2605.08738","citing_title":"SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training","ref_index":85,"is_internal_anchor":false},{"citing_arxiv_id":"2605.11887","citing_title":"Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models","ref_index":22,"is_internal_anchor":false},{"citing_arxiv_id":"2601.02780","citing_title":"MiMo-V2-Flash Technical Report","ref_index":39,"is_internal_anchor":false},{"citing_arxiv_id":"2605.08738","citing_title":"SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training","ref_index":84,"is_internal_anchor":false},{"citing_arxiv_id":"2604.20549","citing_title":"Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection","ref_index":42,"is_internal_anchor":false},{"citing_arxiv_id":"2605.07731","citing_title":"Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs","ref_index":58,"is_internal_anchor":false},{"citing_arxiv_id":"2604.14448","citing_title":"MARCA: A Checklist-Based Benchmark for Multilingual Web Search","ref_index":23,"is_internal_anchor":false},{"citing_arxiv_id":"2505.09388","citing_title":"Qwen3 Technical Report","ref_index":31,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/JOQLOVT2PYLOUXAO6J5ENB2DIP","json":"https://pith.science/pith/JOQLOVT2PYLOUXAO6J5ENB2DIP.json","graph_json":"https://pith.science/api/pith-number/JOQLOVT2PYLOUXAO6J5ENB2DIP/graph.json","events_json":"https://pith.science/api/pith-number/JOQLOVT2PYLOUXAO6J5ENB2DIP/events.json","paper":"https://pith.science/paper/JOQLOVT2"},"agent_actions":{"view_html":"https://pith.science/pith/JOQLOVT2PYLOUXAO6J5ENB2DIP","download_json":"https://pith.science/pith/JOQLOVT2PYLOUXAO6J5ENB2DIP.json","view_paper":"https://pith.science/paper/JOQLOVT2","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2411.19799&json=true","fetch_graph":"https://pith.science/api/pith-number/JOQLOVT2PYLOUXAO6J5ENB2DIP/graph.json","fetch_events":"https://pith.science/api/pith-number/JOQLOVT2PYLOUXAO6J5ENB2DIP/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/JOQLOVT2PYLOUXAO6J5ENB2DIP/action/timestamp_anchor","attest_storage":"https://pith.science/pith/JOQLOVT2PYLOUXAO6J5ENB2DIP/action/storage_attestation","attest_author":"https://pith.science/pith/JOQLOVT2PYLOUXAO6J5ENB2DIP/action/author_attestation","sign_citation":"https://pith.science/pith/JOQLOVT2PYLOUXAO6J5ENB2DIP/action/citation_signature","submit_replication":"https://pith.science/pith/JOQLOVT2PYLOUXAO6J5ENB2DIP/action/replication_record"}},"created_at":"2026-07-05T09:42:08.566737+00:00","updated_at":"2026-07-05T09:42:08.566737+00:00"}