{"paper":{"title":"PanGu-$\\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Chao Zhang, Chen Li, Dasen Yan, Fang Peng, Fangqing Jiang, Gaojun Fan, Han Zhang, Hengtao Tao, Hui Wang, Jianfeng Yu, Jin Wang, Jun Wang, Kaisheng Wang, Lingfeng Deng, Mingyue Guo, Qi Guo, Qun Liu, Shanzhi Gu, Shaojie Zhang, Teng Su, Wei Zeng, Xiaoda Zhang, Xiaozhe Ren, Xin Jiang, Xinjing Huang, Xuefeng Jin, Yan Zhang, Yaowei Wang, Yehong Zhang, Yifan Yao, Yi Liao, Yonghong Tian, Yue Yu, Zexuan Yi, Zhe Lin, ZhenZhang Yang, Zhiwei Wang, Ziyan Gong","submitted_at":"2021-04-26T06:59:36Z","abstract_excerpt":"Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demonstrated strong performances on natural language understanding and generation with \\textit{few-shot in-context} learning. In this work, we present our practice on training large-scale autoregressive language models named PanGu-$\\alpha$, with up to 200 billion parameters. PanGu-$\\alpha$ is developed under the MindSpore and trained on a cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is imple"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2104.12369","kind":"arxiv","version":1},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2104.12369/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}