{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2026:XUBDE4GE24JW6HP2CIHVG4SLF6","short_pith_number":"pith:XUBDE4GE","schema_version":"1.0","canonical_sha256":"bd023270c4d7136f1dfa120f53724b2fbd52aec09ca6968cde3343443911d0bc","source":{"kind":"arxiv","id":"2601.18026","version":2},"attestation_state":"computed","paper":{"title":"CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Abdulhamid Abubakar, Ahmad Mustafid, Ahmad Mustapha Wali, Akshata A, Amit Agarwal, Amr Keleg, Andrew Yates, Atnafu Lambebo Tonja, Azmine Toushik Wasi, Azril Hafizi Amirudin, Benjamin Rice, Beno\\^it Sagot, Bruhan Kyomuhendo, Carol Muchemi, Casper Muziri, Catherine Arnett, Chalamalasetti Kranti, Cristina Aggazzotti, Cynthia Amol, Damian Stewart, Daniel Ruffinelli, David Anugraha, Dmitry Gaynullin, Ej Zhou, Esther Adenuga, Faisal Muhammad Adam, Fenal Ashokbhai Ilasariya, Filbert Aurelian Tjiaranata, Genta Indra Winata, Gouthami Vadithya, Hamada Nayel, Hande Celikkanat, Hend Al-Khalifa, Hitesh Laxmichand Patel, Idris Abdulmumin, Ikhlasul Akmal Hanif, Ilker Kesen, Ingrid Gabriela Franco Ramirez, Inshirah Idris, Jakhongir Saydaliev, Jean Maillard, Jesujoba O. Alabi, Joseph Marvin Imperial, Juan Pablo Mart\\'inez, Jun Kevin, Kamohelo Makaaka, Karan Dua, Kenton Murray, Khang Nguyen, Konstantin Dobler, Kun Kerdthaisong, Lanwenn ar C'horr, Laurie Burchell, Leshem Choshen, Luca Foppiano, Luis Frentzen Salim, Malte Ostendorff, Manuel Goul\\~ao, Mattes Ruckdeschel, Melika Nobakhtian, Michael Anugraha, Mike Zhang, Mithil Bangera, Muhammad Ravi Shulthan Habibi, My Chiffon Nguyen, Nadia Ghezaiel Hammouda, Nicholas Andrews, Nuhu Ibrahim, Pavel Stepachev, Pedro Ortiz Suarez, Quentin Pag\\`es, Rafael Mosquera-G\\'omez, Raia Abu Ahmad, Rasul Dent, Reem Alqifari, Rob van der Goot, Sara Hincapie-Monsalve, Sarah Luger, Saron Samuel, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad, Shu Okabe, Sotaro Takeshita, Sowmya Vajjala, Srikant Panda, Tack Hwa Wong, Thibault Cl\\'erice, Thom Vaughan, Tommaso Green, Vallerie Alexandra Putra, Verrah Otiende, Vicky Feliren, Vukosi Marivate, Weerayut Buaphet, Yassine Toughrai, Yeshil Bangera, Yiyuan Li","submitted_at":"2026-01-25T22:49:30Z","abstract_excerpt":"Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heterogeneous web data often used to train multilingual language models. In this paper, we introduce CommonLID, a community-driven, human-annotated LID benchmark for the web domain, covering 109 languages. Many of the included languages have been previously under-served, making CommonLID a key resource for developing more representative high-quality text corpora. We show CommonLID's value by using it, alongside five other"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2601.18026","kind":"arxiv","version":2},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2026-01-25T22:49:30Z","cross_cats_sorted":[],"title_canon_sha256":"5960ec8a45ccff55ac210147e8b7dfc768cd6ec615d3b979461dd0d56e3a118f","abstract_canon_sha256":"d4c794bf3be49ace26201155d50681e5bc6872f2a14c9dca3141f780511000e4"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-06-10T01:08:32.665730Z","signature_b64":"CGYapzpupfFifomULbEzTSMptDTsZTd9822knXGREZGGBAkbSGJuGotUOnA4EJ3Y5RxsCVHuUBFURLtL68TvCQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"bd023270c4d7136f1dfa120f53724b2fbd52aec09ca6968cde3343443911d0bc","last_reissued_at":"2026-06-10T01:08:32.665071Z","signature_status":"signed_v1","first_computed_at":"2026-06-10T01:08:32.665071Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Abdulhamid Abubakar, Ahmad Mustafid, Ahmad Mustapha Wali, Akshata A, Amit Agarwal, Amr Keleg, Andrew Yates, Atnafu Lambebo Tonja, Azmine Toushik Wasi, Azril Hafizi Amirudin, Benjamin Rice, Beno\\^it Sagot, Bruhan Kyomuhendo, Carol Muchemi, Casper Muziri, Catherine Arnett, Chalamalasetti Kranti, Cristina Aggazzotti, Cynthia Amol, Damian Stewart, Daniel Ruffinelli, David Anugraha, Dmitry Gaynullin, Ej Zhou, Esther Adenuga, Faisal Muhammad Adam, Fenal Ashokbhai Ilasariya, Filbert Aurelian Tjiaranata, Genta Indra Winata, Gouthami Vadithya, Hamada Nayel, Hande Celikkanat, Hend Al-Khalifa, Hitesh Laxmichand Patel, Idris Abdulmumin, Ikhlasul Akmal Hanif, Ilker Kesen, Ingrid Gabriela Franco Ramirez, Inshirah Idris, Jakhongir Saydaliev, Jean Maillard, Jesujoba O. Alabi, Joseph Marvin Imperial, Juan Pablo Mart\\'inez, Jun Kevin, Kamohelo Makaaka, Karan Dua, Kenton Murray, Khang Nguyen, Konstantin Dobler, Kun Kerdthaisong, Lanwenn ar C'horr, Laurie Burchell, Leshem Choshen, Luca Foppiano, Luis Frentzen Salim, Malte Ostendorff, Manuel Goul\\~ao, Mattes Ruckdeschel, Melika Nobakhtian, Michael Anugraha, Mike Zhang, Mithil Bangera, Muhammad Ravi Shulthan Habibi, My Chiffon Nguyen, Nadia Ghezaiel Hammouda, Nicholas Andrews, Nuhu Ibrahim, Pavel Stepachev, Pedro Ortiz Suarez, Quentin Pag\\`es, Rafael Mosquera-G\\'omez, Raia Abu Ahmad, Rasul Dent, Reem Alqifari, Rob van der Goot, Sara Hincapie-Monsalve, Sarah Luger, Saron Samuel, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad, Shu Okabe, Sotaro Takeshita, Sowmya Vajjala, Srikant Panda, Tack Hwa Wong, Thibault Cl\\'erice, Thom Vaughan, Tommaso Green, Vallerie Alexandra Putra, Verrah Otiende, Vicky Feliren, Vukosi Marivate, Weerayut Buaphet, Yassine Toughrai, Yeshil Bangera, Yiyuan Li","submitted_at":"2026-01-25T22:49:30Z","abstract_excerpt":"Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heterogeneous web data often used to train multilingual language models. In this paper, we introduce CommonLID, a community-driven, human-annotated LID benchmark for the web domain, covering 109 languages. Many of the included languages have been previously under-served, making CommonLID a key resource for developing more representative high-quality text corpora. We show CommonLID's value by using it, alongside five other"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2601.18026","kind":"arxiv","version":2},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2601.18026/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2601.18026","created_at":"2026-06-10T01:08:32.665152+00:00"},{"alias_kind":"arxiv_version","alias_value":"2601.18026v2","created_at":"2026-06-10T01:08:32.665152+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2601.18026","created_at":"2026-06-10T01:08:32.665152+00:00"},{"alias_kind":"pith_short_12","alias_value":"XUBDE4GE24JW","created_at":"2026-06-10T01:08:32.665152+00:00"},{"alias_kind":"pith_short_16","alias_value":"XUBDE4GE24JW6HP2","created_at":"2026-06-10T01:08:32.665152+00:00"},{"alias_kind":"pith_short_8","alias_value":"XUBDE4GE","created_at":"2026-06-10T01:08:32.665152+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":1,"internal_anchor_count":1,"sample":[{"citing_arxiv_id":"2605.18232","citing_title":"SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark","ref_index":9,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/XUBDE4GE24JW6HP2CIHVG4SLF6","json":"https://pith.science/pith/XUBDE4GE24JW6HP2CIHVG4SLF6.json","graph_json":"https://pith.science/api/pith-number/XUBDE4GE24JW6HP2CIHVG4SLF6/graph.json","events_json":"https://pith.science/api/pith-number/XUBDE4GE24JW6HP2CIHVG4SLF6/events.json","paper":"https://pith.science/paper/XUBDE4GE"},"agent_actions":{"view_html":"https://pith.science/pith/XUBDE4GE24JW6HP2CIHVG4SLF6","download_json":"https://pith.science/pith/XUBDE4GE24JW6HP2CIHVG4SLF6.json","view_paper":"https://pith.science/paper/XUBDE4GE","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2601.18026&json=true","fetch_graph":"https://pith.science/api/pith-number/XUBDE4GE24JW6HP2CIHVG4SLF6/graph.json","fetch_events":"https://pith.science/api/pith-number/XUBDE4GE24JW6HP2CIHVG4SLF6/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/XUBDE4GE24JW6HP2CIHVG4SLF6/action/timestamp_anchor","attest_storage":"https://pith.science/pith/XUBDE4GE24JW6HP2CIHVG4SLF6/action/storage_attestation","attest_author":"https://pith.science/pith/XUBDE4GE24JW6HP2CIHVG4SLF6/action/author_attestation","sign_citation":"https://pith.science/pith/XUBDE4GE24JW6HP2CIHVG4SLF6/action/citation_signature","submit_replication":"https://pith.science/pith/XUBDE4GE24JW6HP2CIHVG4SLF6/action/replication_record"}},"created_at":"2026-06-10T01:08:32.665152+00:00","updated_at":"2026-06-10T01:08:32.665152+00:00"}