CNMBERT: A Model for Converting Hanyu Pinyin Abbreviations to Chinese Characters

Feng Cao; Zishuo Feng

REVIEW 3 major objections 6 minor 1 cited by

CNMBERT: A Model for Converting Hanyu Pinyin Abbreviations to Chinese Characters

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A BERT variant converts pinyin abbreviations to Chinese characters with 61.53% MRR, beating fine-tuned GPT models and ChatGPT-4o on a 10,373-sample benchmark.

desk verdict A genuinely new multi-mask trick for pinyin abbreviation conversion, but the headline SOTA claim rests on a hand-filtered test set and a confounded ablation. read the letter →

arxiv 2411.11770 v4 pith:4TSVLEBS submitted 2024-11-18 cs.CL cs.AI

Zishuo Feng , Feng Cao This is my paper · ORCID

classification cs.CLcs.AI

keywords pinyinabbreviationconversionChinesespellingcorrectionfill-masktaskmulti-maskstrategyMixtureofExpertsBERTnaturallanguageprocessingmeanreciprocalrank

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that converting Hanyu Pinyin abbreviations to Chinese characters is best treated as a fill-mask task, not a generation task. It proposes CNMBERT, which extends BERT with letter-specific mask tokens and mixture-of-experts layers, and reports that it reaches 61.53% MRR and 51.86% accuracy, surpassing fine-tuned Qwen models and ChatGPT-4o. If true, this gives a lightweight, fast model for decoding the abbreviated pinyin common on Chinese social media, with potential benefits for downstream tasks like named entity recognition and sentiment analysis.

What carries the argument

The multi-mask strategy maps each letter of the pinyin alphabet to a distinct mask token, so a masked position carries the first letter of the intended character's pinyin; this preserves consistency with BERT's masked-language-model pretraining while turning the abbreviation into usable evidence. The MoE layers, arranged in a pyramid-residual structure with a shared expert and routed experts, replace every other feedforward layer and allow different experts to handle different token clusters, including the special mask tokens. Together these mechanisms let the model convert abbreviations like 'fq' to '放弃' by constraining predictions to characters whose pinyin starts with the given letters.

What would settle it

Collect a fresh sample of pinyin abbreviations from raw Douyin, Bilibili, or Weibo comments without manual filtering, run CNMBERT and ChatGPT-4o on the same sentences, and compare MRR; if CNMBERT's advantage shrinks substantially or reverses on this unfiltered sample, the filtered test set is the reason for the reported gap.

Watch

Extended reading notes

Core claim

CNMBERT outperforms fine-tuned GPT models and ChatGPT-4o on pinyin abbreviation to character conversion, achieving a 61.53% MRR score and 51.86% accuracy on a constructed test set of 10,373 sentences. The central discovery is that replacing the generic [MASK] token with 26 pinyin-initial-specific mask tokens (e.g., [LETTER A]) allows a BERT model to use the first letter of each abbreviation as a strong constraint, while pyramid-structured MoE layers let different experts specialize on different token types. Ablations show the multi-mask strategy is the main driver: removing it drops MRR from 59.70 to 34.49, while adding MoE layers adds about 1.8 MRR points. The model also runs much faster and uses far less memory than the fine-tuned 14-billion-parameter baseline.

Load-bearing premise

The test set was manually filtered to remove cases with excessive ambiguity, so the reported performance gap over ChatGPT-4o may not generalize to unfiltered real-world pinyin abbreviations, where ambiguity is common.

Editorial extensions

If this is right

If the results hold, CNMBERT offers a practical, low-resource tool for expanding pinyin abbreviations in social media text, running at 3.20 QPS with 0.8GB memory on a single RTX 3090 Ti.
The multi-mask strategy is architecture-agnostic and can be applied to any BERT-based model, potentially improving other Chinese spelling correction tasks that involve masked characters.
Decoding abbreviations before downstream processing could improve named entity recognition and sentiment analysis on Chinese social media, where such abbreviations are frequent.
The model's performance on monosyllabic abbreviations (74.30% MRR) suggests that common single-letter substitutions, which are especially frequent in censored or informal contexts, are largely recoverable.
The reported sharp decline on longer abbreviations and low-context sentences defines a clear boundary: the method works best when context is rich and the abbreviation is short.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The manually filtered test set probably understates real-world difficulty; a fairer stress test would sample raw Douyin or Bilibili comments without removing ambiguous cases, and it is possible the gap over ChatGPT-4o narrows there.
Because the multi-mask strategy uses only first letters, extending it to include pinyin finals, tones, or syllable boundaries could specifically improve polysyllabic word conversion, where the paper reports weaker results.
The approach could transfer to other initialism-heavy languages or to English acronyms, as the paper hints, but the correspondence between letters and mask tokens would need re-deriving for each language's orthography.
The MoE layer's contribution is modest compared with the multi-mask strategy, so the practical value of the MoE component may lie more in scalability than in accuracy gains on this task.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is a small but legitimate contribution to a niche task—converting pinyin abbreviations like 'fq' for '放弃' into Chinese characters. The multi-mask idea is new: instead of a generic [MASK], each pinyin initial gets its own mask token, so the model knows the first letter of each target syllable. On their own 10,373-sample test set, the full model beats ChatGPT-4o by a wide margin (61.53 vs 41.90 MRR). I think the result is real for what it measures, but what it measures is a filtered subset of the task.

The good: the multi-mask strategy is simple, clearly explained, and applies to any BERT-style model. The ablations show multi-mask is the big win (59.70 with it, 34.49 without it—though see below). They also report memory and QPS, showing a 329M model beating a 14.8B fine-tuned Qwen on speed and memory, a practical plus. The writing is straightforward and the authors acknowledge the model's weakness on long or low-context abbreviations.

The soft spots: the test set is load-bearing. The authors manually removed cases where the replaced word would cause 'excessive ambiguity,' which removes exactly the hard cases in the real world. They don't report how many samples were filtered or what the distribution looks like after filtering. If you strip out ambiguous cases, you measure the easy half, and the gap to GPT-4o may narrow or vanish on unfiltered data. The ablation for '−Multi-Mask' is also confounded: that model lacks MoE as well, so you cannot separate the two components. They cite PinyinGPT as related work but never compare against it, which is the most direct competitor. And the 'predicting pinyin' experiment in Fig. 4 is near-trivial because the abbreviation already contains the first letters; it mostly shows the model learned the mapping, not that it resolves ambiguity.

The math and architecture description are fine. The citation pattern is reasonable, and the authors are honest about limitations. This paper is for people working on Chinese spelling correction, social media text normalization, and input method design. It deserves a serious referee, but the referee should insist on: a proper crossed ablation, a PinyinGPT comparison, and a released or unfiltered test set with a quantified report of what the manual filtering removed. My recommendation: send it out for review, expecting major revision.

Referee Report

3 major / 6 minor

Summary. The paper proposes CNMBERT, a BERT-based fill-mask model for converting Hanyu Pinyin abbreviations to Chinese characters. The model introduces a multi-mask strategy in which each pinyin initial letter is mapped to a distinct mask token, and replaces selected feed-forward layers with Pyramid-Residual Mixture-of-Experts layers. The authors construct a 10,373-sample test set from EXT, webtext2019zh, and Bilibili comment data, and report that CNMBERT achieves an MRR of 61.53% and an accuracy of 51.86%, outperforming fine-tuned Qwen2.5-14B and ChatGPT-4o. Ablations and UMAP visualizations are provided to support the contributions of the multi-mask and MoE components.

Significance. If the reported results are robust, CNMBERT would be a practically useful, low-resource alternative to large autoregressive models for pinyin-abbreviation conversion, with a clear application to Chinese spelling correction and social-media text normalization. The multi-mask idea of conditioning masked prediction on the pinyin initial letters is a simple and appealing design that aligns pre-training with the downstream task, and the MoE design is a sensible way to scale capacity while keeping the model compact. The paper also provides useful analysis of expert specialization and reports latency/memory advantages. However, the strength of the central empirical claim—that CNMBERT surpasses GPT-4o—depends on the validity of a self-constructed test set that was manually filtered to remove ambiguous cases, and on the correctness of an ablation that currently conflates two design choices. The significance is therefore conditional on additional evaluation and ablation evidence.

major comments (3)

[Section IV.A, Table V] The test set construction is the load-bearing component of the headline comparison. The authors state that they 'manually reviewed those data to ensure that the replaced words would not introduce excessive ambiguity' and further 'imposed constraints on the frequency of the different replaced words, ensuring that their replacement frequency in the overall dataset is less than 0.4%.' This manual filtering removes exactly the ambiguous cases where a conversion tool is most needed and where autoregressive models such as GPT-4o might be relatively stronger or weaker; the reported 19.63-point MRR advantage over ChatGPT-4o is therefore only established on a subset that is easier than natural pinyin-abbreviation usage. The paper does not quantify how many candidate examples were excluded during manual review, what distribution of ambiguity was removed, or how the 0.4% frequency cap interacts with the difficulty of the benchmark. Please report the filtering statistics, release the test set, and evaluate on an unfiltered random sample (or on real user-generated abbreviations) to support the generalization claim.
[Section IV.B and Section IV.E, Table V] The ablation labeled '−Multi-Mask' does not isolate the multi-mask strategy. In Section IV.B the authors trained three models: (a) without multi-mask and without MoE, (b) with multi-mask but without MoE, and (c) with both. In Table V, 'CNMBERT' corresponds to (c), '−MoE' to (b), and '−Multi-Mask' to (a). The difference between (a) and (b) therefore includes both the multi-mask strategy and the MoE layers simultaneously. The conclusion that 'the multi-mask strategy and the inclusion of MoE layers significantly enhance its performance' is supported, but the paper cannot attribute the 34.49-to-59.70 gain specifically to the multi-mask strategy because the MoE is also removed. Please add a model that has MoE layers but no multi-mask strategy, so that the individual contribution of the multi-mask strategy can be measured cleanly; the current labeling is misleading.
[Section IV.D and Section IV.C] The evaluation protocol for autoregressive models introduces an uncontrolled variable in the comparison. ChatGPT-4o, Qwen2.5-14b-Instruct, Qwen-FT, and Llama3-Chinese-8B are asked to 'list all the potential meanings in descending order of likelihood, separated by spaces,' and the outputs are then scored with MRR. The paper does not specify how the free-form text was parsed into a ranked list, how ties or formatting variations (e.g., punctuation, line breaks, extra explanations) were handled, or whether the same parsing was applied identically to all autoregressive baselines. Since part of the reported gap between CNMBERT (which produces a deterministic ranked list over a constrained vocabulary) and the autoregressive baselines could stem from parsing artifacts rather than task ability, please provide the parsing code and the exact instructions used to convert model outputs into ranked candidates, and report the impact of different parsing choices on the baseline scores.

minor comments (6)

[Equation (1)] The subscript of the loss is rendered as 'LM LM' in the text; this should be 'MLM'. Additionally, the definition of M as a set of index sets is correct but could be clarified by writing, for each letter c, M_c as the set of positions masked with [LETTER c], to avoid confusion between the alphabet and the mask-token set.
[Section IV.A] The training-data construction is underspecified. Please state how the 2 million sentences were drawn from the three sources, how a word was chosen for replacement in a sentence, whether replacement is always with the first letter of each character's pinyin (including tone-less initials), and what the exact meaning of the 'frequency < 0.4%' constraint is. This detail is needed for reproducibility and for interpreting the test-set difficulty.
[Section IV.B and Table V] The row label '−Multi-Mask' in Tables V and VII is confusing because it refers to the model that removes both the multi-mask strategy and MoE. Please rename the rows according to what is actually removed (e.g., '−Multi-Mask −MoE' for model (a) and '−MoE' for model (b)) or restructure the ablation table to separate the two factors.
[Section IV.B] The training protocol is ambiguous: the text says the authors 'extended the BERT-wwm-ext model' and then 'fine-tuned the model on the original MLM task' and then 'based on this, trained three CNMBERT models.' It is not clear whether the 32-epoch training applies to all three model variants, whether the multi-mask and MoE models are all initialized from the same WWM-fine-tuned BERT, or whether the WWM-fine-tuned model itself is one of the three. Please clarify the exact step-by-step training procedure.
[Section IV.D] Please specify the decoding vocabulary and candidate space for CNMBERT. The paper states beam size = 16 and topk = 10, but it does not say whether the model is constrained to output only Chinese characters, whether the vocabulary is restricted to characters appearing in the training data, or how the ranked list of character sequences is constructed from beam search. This information is necessary for interpreting the MRR numbers.
[Section V.E and Figure 4] The pinyin-prediction experiment reported in Figure 4 should be framed as a sanity check rather than as evidence of disambiguation ability. Since each abbreviation already contains the first letters of the target pinyin, a high pinyin MRR is expected if the model has learned the mapping from letters to pinyin syllables; it does not measure whether the model can resolve the inherent ambiguity between homophones. Consider rephrasing the conclusion in Section V.E accordingly.

Circularity Check

1 steps flagged · score 2.0 of 10

Central conversion result is not circular; only the auxiliary 'predicting pinyin' experiment (Fig. 4) is near-tautological.

self definitional [Section IV.E and Fig. 4]
"We tested whether the model could accurately predict the correct pinyin for both monosyllabic and polysyllabic words from its pinyin abbreviation... We found that for our model, the MRR score of predicting the pinyin of words was higher than the score of predicting the words themselves. This indicates that the model can vaguely infer the pinyin of corresponding Chinese characters from the pinyin abbreviations."

In the paper's own definition, a pinyin abbreviation is the case where 'the pinyin's first letters are used to replace Chinese characters.' Thus the model's input tokens are the initial letters of the target pinyin. Predicting the pinyin therefore only requires completing finals and tones (if any) from the initials that are already present in the input, while predicting characters requires additionally resolving homophones. Higher pinyin MRR than character MRR is expected by construction, so Fig. 4 does not demonstrate a separate 'infer the pinyin' ability. This is a minor diagnostic, not the paper's central SOTA claim, which is evaluated on character conversion and is not circular.

full rationale

The main derivation is self-contained: CNMBERT is a supervised encoder that maps a sequence containing pinyin-initial mask tokens to full Chinese characters, trained with a cross-entropy loss (Eq. 1) on data where the target characters are independently known. The multi-mask strategy explicitly injects the first letters of the pinyin as input features; that is the task definition, not circularity. The MoE layers are standard and externally grounded. The comparison against GPT-4o and Qwen-FT is an empirical benchmark on a held-out test set, not a derivation from the model's own outputs. The manual filtering of ambiguous test examples is a legitimate external-validity criticism, but it is not a circularity: the test labels are not derived from the model. The only near-circular element is the auxiliary 'predicting pinyin' analysis in Fig. 4, where the input abbreviation is definitionally the initial letters of the pinyin being predicted, making that finding unsurprising. Because this does not support the central conversion claim, the overall circularity is low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The ledger lists hand-chosen hyperparameters, domain assumptions about the abbreviation-to-pinyin mapping and test set representativeness, and no invented physical entities. The multi-mask tokens [LETTER X] are model vocabulary, not independent entities.

free parameters (5)

beam_size = 16
Used for beam search during inference; chosen by the authors without a reported validation set.
topk = 10
Used for candidate generation in MRR computation; choice affects reported MRR scores.
moe_expert_counts = layers 1/3/5: 2 experts top-k=1; layer 7: 4 experts top-k=1; layers 9/11/13/15: 8 experts top-k=2
Architecture choice for the MoE configuration; not derived from data.
learning_rate = 1e-5
Learning rate for CNMBERT fine-tuning; chosen by the authors.
training_epochs = 32
Number of training epochs; chosen without reported early stopping criterion.

assumptions (4)

domain assumption Pinyin abbreviations are exactly the first letters of the pinyin of the target characters
The whole task and data construction (Section III.A, Table II) assume this mapping; real social media abbreviations may vary.
ad hoc to paper Manual filtering of test data for low ambiguity yields an unbiased evaluation set
Section IV.A: the authors manually reviewed the test data to avoid excessive ambiguity, which removes hard cases and favors the model.
domain assumption Training corpora are representative of the test distribution
Section IV.A: training data comes from EXT, webtext2019zh, and Bilibili comments, but no evidence that these cover naturally occurring pinyin abbreviations.
domain assumption BERT MLM pretraining provides a good initialization for pinyin abbreviation prediction
The model is initialized from Chinese-BERT-wwm and further tuned on MLM with multi-mask; this assumes the pretrained representations transfer.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CNMBERT: A Model for Converting Hanyu Pinyin Abbreviations to Chinese Characters." pith.science (2026). https://pith.science/paper/4TSVLEBS

@misc{pith2026241111770,
  author       = {Pith},
  title        = {Pith review of: CNMBERT: A Model for Converting Hanyu Pinyin Abbreviations to Chinese Characters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4TSVLEBS}},
  note         = {Machine review of arXiv:2411.11770}
}

read the original abstract

The task of converting Hanyu Pinyin abbreviations to Chinese characters is a significant branch within the domain of Chinese Spelling Correction (CSC). It plays an important role in many downstream applications such as named entity recognition and sentiment analysis. This task typically involves text-length alignment and seems easy to solve; however, due to the limited information content in pinyin abbreviations, achieving accurate conversion is challenging. In this paper, we treat this as a fill-mask task and propose CNMBERT, which stands for zh-CN Pinyin Multi-mask BERT Model, as a solution to this issue. By introducing a multi-mask strategy and Mixture of Experts (MoE) layers, CNMBERT outperforms fine-tuned GPT models and ChatGPT-4o with a 61.53% MRR score and 51.86% accuracy on a 10,373-sample test dataset.

Figures

Figures reproduced from arXiv: 2411.11770 by the authors.

**Figure 2.** The overall architecture of the model and its workflow. The model use a 16-layer transformer architecture. For layers 0–15, some FFN layers are [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The scores using MRR@5 for predictions of words with different [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Results of predict monosyllabic and polysyllabic words’ pinyin. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: The feature space of experts in layer 3(a-c) and layer 7(d-f). [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

RAIR: Retrieval-Augmented Iterative Refinement for Chinese Spelling Correction
cs.CL 2025-04 reject novelty 6.0 of 10

RAIR combines a fine-tuned retriever, multi-turn length reflection, and adaptive selection to improve LLM-based Chinese spelling correction in equal-length and variable-length scenarios.

Reference graph

Works this paper leans on

26 extracted references · 24 canonical work pages · cited by 1 Pith paper

[1]

Yu and Z

J. Yu and Z. Li, ‘Chinese spelling error detection and correction based on language model, pronunciation, and shape’, in Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, 2014, pp. 220–223

work page 2014
[2]

Vaswani, ‘Attention is all you need’, Advances in Neural Information Processing Systems, 2017

A. Vaswani, ‘Attention is all you need’, Advances in Neural Information Processing Systems, 2017

work page 2017
[3]

Radford, ‘Improving language understanding by generative pre- training’, 2018

A. Radford, ‘Improving language understanding by generative pre- training’, 2018. Fig. 5. The feature space of experts in layer 3(a-c) and layer 7(d-f)

work page 2018
[4]

Y . Cui, W. Che, T. Liu, B. Qin, S. Wang, and G. Hu, ‘Revisiting Pre- Trained Models for Chinese Natural Language Processing’, in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 657–668

work page 2020
[5]

Devlin, M.-W

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ‘BERT: Pre- training of Deep Bidirectional Transformers for Language Understand- ing’, CoRR, vol. abs/1810.04805, 2018

arXiv 2018
[6]

Team, ‘Qwen2.5: A Party of Foundation Models’

Q. Team, ‘Qwen2.5: A Party of Foundation Models’. Sep-2024

work page 2024
[7]

Li et al., ‘Unified Named Entity Recognition as Word-Word Relation Classification’, arXiv [cs.CL]

J. Li et al., ‘Unified Named Entity Recognition as Word-Word Relation Classification’, arXiv [cs.CL]. 2021

work page 2021
[8]

Zhang, S

L. Zhang, S. Wang, and B. Liu, ‘Deep Learning for Sentiment Analysis: A Survey’, arXiv [cs.CL]. 2018

work page 2018

Show all 26 references

[9]

L. Liu, H. Wu, and H. Zhao, ‘Chinese Spelling Correction as Rephrasing Language Model’, arXiv [cs.CL]. 2024

2024
[10]

Wu et al., ‘Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation’, arXiv [cs.CL]

Y . Wu et al., ‘Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation’, arXiv [cs.CL]. 2016

2016
[11]

W. Che, Y . Feng, L. Qin, and T. Liu, ‘N-LTP: An Open-source Neural Language Technology Platform for Chinese’, pp. 42–49, Sep. 2020

2020
[12]

Du, ‘GPT2-Chinese: Tools for training GPT2 model in Chinese language’, GitHub repository

Z. Du, ‘GPT2-Chinese: Tools for training GPT2 model in Chinese language’, GitHub repository. GitHub, 2019

2019
[13]

Tan et al., ‘Exploring and Adapting Chinese GPT to Pinyin Input Method’, arXiv [cs.CL]

M. Tan et al., ‘Exploring and Adapting Chinese GPT to Pinyin Input Method’, arXiv [cs.CL]. 2022

2022
[14]

R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, ‘Adaptive mixtures of local experts’, Neural computation, vol. 3, no. 1, pp. 79–87, 1991

1991
[15]

Rajbhandari et al., ‘DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale’, arXiv [cs.LG]

S. Rajbhandari et al., ‘DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale’, arXiv [cs.LG]. 2022

2022
[16]

Dai et al., ‘DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models’, arXiv [cs.CL]

D. Dai et al., ‘DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models’, arXiv [cs.CL]. 2024

2024
[17]

E. J. Hu et al., ‘LoRA: Low-Rank Adaptation of Large Language Models’, arXiv [cs.CL]. 2021

2021
[18]

R. Luo, J. Xu, Y . Zhang, Z. Zhang, X. Ren, and X. Sun, ‘PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation’, CoRR, vol. abs/1906.11455, 2019

1906 arXiv
[19]

Zheng, R

Y . Zheng, R. Zhang, J. Zhang, Y . Ye, and Z. Luo, ‘LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models’, in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 3: System Demonstrations), 2024, pp. 400–410

2024
[20]

Rajbhandari, J

S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He, ‘ZeRO: Memory Op- timizations Toward Training Trillion Parameter Models’, arXiv [cs.LG]. 2020

2020
[21]

T. Wolf et al., ‘Transformers: State-of-the-Art Natural Language Pro- cessing’, in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020, pp. 38–45

2020
[22]

Loshchilov and F

I. Loshchilov and F. Hutter, ‘Decoupled Weight Decay Regularization’, arXiv [cs.LG]. 2019

2019
[23]

Y . Cui, Z. Yang, and X. Yao, ‘Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca’, arXiv preprint arXiv:2304. 08177, 2023

2023
[24]

Chase, LangChain

H. Chase, LangChain. 2022

2022
[25]

McInnes, J

L. McInnes, J. Healy, and J. Melville, ‘UMAP: Uniform Manifold Ap- proximation and Projection for Dimension Reduction’, arXiv [stat.ML]. 2020

2020
[26]

Xu, ‘NLP Chinese Corpus: Large Scale Chinese Corpus for NLP’

B. Xu, ‘NLP Chinese Corpus: Large Scale Chinese Corpus for NLP’. Zenodo, Sep-2019

2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Yu and Z

J. Yu and Z. Li, ‘Chinese spelling error detection and correction based on language model, pronunciation, and shape’, in Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, 2014, pp. 220–223

work page 2014

[2] [2]

Vaswani, ‘Attention is all you need’, Advances in Neural Information Processing Systems, 2017

A. Vaswani, ‘Attention is all you need’, Advances in Neural Information Processing Systems, 2017

work page 2017

[3] [3]

Radford, ‘Improving language understanding by generative pre- training’, 2018

A. Radford, ‘Improving language understanding by generative pre- training’, 2018. Fig. 5. The feature space of experts in layer 3(a-c) and layer 7(d-f)

work page 2018

[4] [4]

Y . Cui, W. Che, T. Liu, B. Qin, S. Wang, and G. Hu, ‘Revisiting Pre- Trained Models for Chinese Natural Language Processing’, in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 657–668

work page 2020

[5] [5]

Devlin, M.-W

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ‘BERT: Pre- training of Deep Bidirectional Transformers for Language Understand- ing’, CoRR, vol. abs/1810.04805, 2018

arXiv 2018

[6] [6]

Team, ‘Qwen2.5: A Party of Foundation Models’

Q. Team, ‘Qwen2.5: A Party of Foundation Models’. Sep-2024

work page 2024

[7] [7]

Li et al., ‘Unified Named Entity Recognition as Word-Word Relation Classification’, arXiv [cs.CL]

J. Li et al., ‘Unified Named Entity Recognition as Word-Word Relation Classification’, arXiv [cs.CL]. 2021

work page 2021

[8] [8]

Zhang, S

L. Zhang, S. Wang, and B. Liu, ‘Deep Learning for Sentiment Analysis: A Survey’, arXiv [cs.CL]. 2018

work page 2018