{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2025:BMASMK2V2A52Q3PQ2AMVCJZAGW","short_pith_number":"pith:BMASMK2V","schema_version":"1.0","canonical_sha256":"0b01262b55d03ba86df0d01951272035bdacf225c9bec7246454cd9853df56b1","source":{"kind":"arxiv","id":"2503.10267","version":3},"attestation_state":"computed","paper":{"title":"An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Amanda Myntti, Andrey Kutuzov, Barry Haddow, Bhavitvya Malik, David Samuel, Dayy\\'an O'Brien, Du\\v{s}an Vari\\v{s}, Erik Henriksson, Farrokh Mehryary, Gema Ram\\'irez-S\\'anchez, Jan Haji\\v{c}, Jaume Zaragoza-Bernabeu, Jind\\v{r}ich Helcl, Joona Kyt\\\"oniemi, J\\\"org Tiedemann, Jousia Piha, Laurie Burchell, Liane Guillou, Mariia Fedorova, Marta Ba\\~n\\'on, Mateusz Klimaszewski, Mikko Aulamo, Nikita Moghe, Nikolay Arefyev, Ona de Gibert, Pavel Stepachev, Petter M{\\ae}hlum, Pinzhen Chen, Proyag Pal, Sampo Pyysalo, Stephan Oepen, Tereza Vojt\\v{e}chov\\'a, Veronika Laippala, Ville Komulainen, Vladislav Mikhailov","submitted_at":"2025-03-13T11:24:09Z","abstract_excerpt":"Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the qualit"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2503.10267","kind":"arxiv","version":3},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2025-03-13T11:24:09Z","cross_cats_sorted":[],"title_canon_sha256":"81a37379c6dbb3d4429714e1494c971a1644c87839991a7fedec08003dcc98ba","abstract_canon_sha256":"033dc34077a37edba1ddfbfaa9e0ef800e02b6a9c37273abe2812a006ce74266"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T11:15:45.245182Z","signature_b64":"uWFrj+FbHo2QeEAVcgSHYTsrfkd0/Ui2Z2qsbC7+dGfawZVnxtauOxjuUX2AowLJPA4RYgHoCEjiJl5BXB8xDw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"0b01262b55d03ba86df0d01951272035bdacf225c9bec7246454cd9853df56b1","last_reissued_at":"2026-07-05T11:15:45.244677Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T11:15:45.244677Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Amanda Myntti, Andrey Kutuzov, Barry Haddow, Bhavitvya Malik, David Samuel, Dayy\\'an O'Brien, Du\\v{s}an Vari\\v{s}, Erik Henriksson, Farrokh Mehryary, Gema Ram\\'irez-S\\'anchez, Jan Haji\\v{c}, Jaume Zaragoza-Bernabeu, Jind\\v{r}ich Helcl, Joona Kyt\\\"oniemi, J\\\"org Tiedemann, Jousia Piha, Laurie Burchell, Liane Guillou, Mariia Fedorova, Marta Ba\\~n\\'on, Mateusz Klimaszewski, Mikko Aulamo, Nikita Moghe, Nikolay Arefyev, Ona de Gibert, Pavel Stepachev, Petter M{\\ae}hlum, Pinzhen Chen, Proyag Pal, Sampo Pyysalo, Stephan Oepen, Tereza Vojt\\v{e}chov\\'a, Veronika Laippala, Ville Komulainen, Vladislav Mikhailov","submitted_at":"2025-03-13T11:24:09Z","abstract_excerpt":"Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the qualit"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2503.10267","kind":"arxiv","version":3},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2503.10267/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2503.10267","created_at":"2026-07-05T11:15:45.244738+00:00"},{"alias_kind":"arxiv_version","alias_value":"2503.10267v3","created_at":"2026-07-05T11:15:45.244738+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2503.10267","created_at":"2026-07-05T11:15:45.244738+00:00"},{"alias_kind":"pith_short_12","alias_value":"BMASMK2V2A52","created_at":"2026-07-05T11:15:45.244738+00:00"},{"alias_kind":"pith_short_16","alias_value":"BMASMK2V2A52Q3PQ","created_at":"2026-07-05T11:15:45.244738+00:00"},{"alias_kind":"pith_short_8","alias_value":"BMASMK2V","created_at":"2026-07-05T11:15:45.244738+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":1,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2511.07003","citing_title":"NiuTrans.LMT: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs","ref_index":1,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/BMASMK2V2A52Q3PQ2AMVCJZAGW","json":"https://pith.science/pith/BMASMK2V2A52Q3PQ2AMVCJZAGW.json","graph_json":"https://pith.science/api/pith-number/BMASMK2V2A52Q3PQ2AMVCJZAGW/graph.json","events_json":"https://pith.science/api/pith-number/BMASMK2V2A52Q3PQ2AMVCJZAGW/events.json","paper":"https://pith.science/paper/BMASMK2V"},"agent_actions":{"view_html":"https://pith.science/pith/BMASMK2V2A52Q3PQ2AMVCJZAGW","download_json":"https://pith.science/pith/BMASMK2V2A52Q3PQ2AMVCJZAGW.json","view_paper":"https://pith.science/paper/BMASMK2V","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2503.10267&json=true","fetch_graph":"https://pith.science/api/pith-number/BMASMK2V2A52Q3PQ2AMVCJZAGW/graph.json","fetch_events":"https://pith.science/api/pith-number/BMASMK2V2A52Q3PQ2AMVCJZAGW/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/BMASMK2V2A52Q3PQ2AMVCJZAGW/action/timestamp_anchor","attest_storage":"https://pith.science/pith/BMASMK2V2A52Q3PQ2AMVCJZAGW/action/storage_attestation","attest_author":"https://pith.science/pith/BMASMK2V2A52Q3PQ2AMVCJZAGW/action/author_attestation","sign_citation":"https://pith.science/pith/BMASMK2V2A52Q3PQ2AMVCJZAGW/action/citation_signature","submit_replication":"https://pith.science/pith/BMASMK2V2A52Q3PQ2AMVCJZAGW/action/replication_record"}},"created_at":"2026-07-05T11:15:45.244738+00:00","updated_at":"2026-07-05T11:15:45.244738+00:00"}