Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

NEZHA: Neural Contextualized Representation for Chinese Language Understanding

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NEZHA: Chinese language model tops four NLU benchmarks

desk verdict Honest engineering report with a genuinely new parameter-free relative positional encoding, but the state-of-the-art claim is undercut by copied baselines and single runs. read the letter →

arxiv 1909.00204 v3 pith:5SFZP4XK submitted 2019-08-31 cs.CL

Junqiu Wei , Xiaozhe Ren , Xiaoguang Li , Wenyong Huang , Yi Liao , Yasheng Wang , Jiashu Lin , Xin Jiang
show 2 more authors
Xiao Chen Qun Liu
This is my paper · ORCID
classification cs.CL
keywords ChineselanguageunderstandingpretrainedmodelfunctionalrelativepositionalencodingwholewordmaskingmixedprecisiontrainingLAMBoptimizerBERTnatural
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a Chinese-language pretrained model called NEZHA, built on BERT, and claims it reaches state-of-the-art accuracy on several Chinese natural language understanding tasks after fine-tuning. Its distinguishing feature is functional relative positional encoding, which computes attention weights with fixed sinusoidal functions of token distance instead of learned absolute position embeddings. NEZHA also combines whole word masking, mixed precision training, and the LAMB optimizer. A sympathetic reader would take away that a parameter-free relative position scheme is a practical upgrade for Chinese BERT-style models.

What carries the argument

Functional relative positional encoding (FRPE) is the central mechanism. For each pair of positions $i$ and $j$, it builds vectors $a_{ij}$ whose dimension $2k$ and $2k+1$ entries are $\sin((j-i)/10000^{2k/d_z})$ and $\cos((j-i)/10000^{2k/d_z})$, with $d_z$ the per-head hidden size. These fixed vectors are added to the key and value projections inside self-attention, replacing learned absolute position embeddings with a parameter-free sinusoidal encoding of relative distance. It carries the argument because ablation experiments show it produces the largest consistent gains, particularly when training sequences are short but downstream inputs are long, and because it is the one technique that distinguishes NEZHA from all compared baselines.

What would settle it

Run BERT-WWM, ERNIE-Baidu 2.0, and NEZHA under exactly the same fine-tuning settings and multiple random seeds on LCQMC, XNLI, PD-NER, and ChnSenti; if the NEZHA averages do not consistently beat the re-run baselines by a clear margin, the state-of-the-art claim is not supported. A second check: train on the same data with FRPE replaced by parametric absolute or parametric relative encodings and confirm that CMRC performance drops as in the ablation; if it does not, the encoding is not the driver.

Watch

Extended reading notes

Core claim

NEZHA is a Chinese BERT-style model pretrained on a combination of Chinese Wikipedia, an online encyclopedia corpus, and news text, using masked language modeling with whole word masking and next sentence prediction. The paper's central claim is that after fine-tuning, NEZHA achieves state-of-the-art or competitive performance on People's Daily NER, LCQMC, ChnSenti, and XNLI, with strong results on CMRC as well. The authors attribute the improvement mainly to functional relative positional encoding, which removes trainable positional parameters and instead injects relative position through sine and cosine functions inside self-attention. The ablation study supports this attribution by showing consistent gains for functional relative encoding over parametric absolute and parametric relative encodings, especially on the CMRC reading comprehension task.

Load-bearing premise

The load-bearing premise is that the scores for BERT-WWM and ERNIE-Baidu are fair baselines; they are taken from the original papers rather than re-run under NEZHA's own fine-tuning protocol, and the paper itself says the comparison may not be entirely fair.

Editorial extensions

If this is right

  • On the tasks studied, swapping parametric absolute positional embeddings for FRPE yields consistent downstream gains, so the same swap should help other BERT-style Chinese models.
  • Whole word masking with a word segmenter and FRPE are complementary; the NEZHA versions with whole word masking show the best XNLI and sentiment results among the compared models.
  • Mixed precision training and the LAMB optimizer let NEZHA use a total batch size of 14,400 for the base model, so large-batch pretraining is feasible without hand-tuned learning-rate schedules.
  • Because FRPE uses fixed sinusoidal functions, the model may extrapolate to input sequences longer than the 128 or 512 tokens used during pretraining, which matters for reading comprehension and long documents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because FRPE adds no trainable position parameters, a natural test is whether it also improves English BERT or multilingual Transformers, particularly for long-document tasks where absolute position embeddings are known to saturate.
  • The gains on CMRC when going from sequence length 128 to 512 suggest that FRPE's benefit is tied to relative distance information; an even longer pretraining sequence might push scores further or hit a plateau.
  • The quality of whole word masking depends on the Chinese word segmenter used; comparing it with n-gram or unigram-subword masking would show how much of the gain comes from the segmentation choice rather than the masking principle.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This manuscript reports the pre-training and fine-tuning of NEZHA, a Chinese contextualized language model built on BERT. The model combines functional relative positional encoding (FRPE) with whole-word masking, mixed-precision training, and the LAMB optimizer. The authors describe the pre-training corpora (Chinese Wikipedia, Baidu Baike, Chinese News), give full hyperparameters, and evaluate BASE and LARGE models on CMRC, XNLI, LCQMC, PD-NER, and ChnSenti. An ablation study (Table 5) isolates the effects of positional encoding scheme, whole-word masking, training sequence length, and corpus source. The abstract claims state-of-the-art results on PD-NER, LCQMC, ChnSenti, and XNLI; Section 3.2 notes that some baseline scores are copied from other papers, so the comparison may not be entirely fair.

Significance. If the empirical claims hold, the paper's main contribution is a well-specified and practical recipe for Chinese pre-training, with FRPE as a parameter-free alternative to learned positional encodings. The internal ablation on CMRC (Table 5) gives credible evidence that relative encodings, especially FRPE, help when the model is trained with sequence length 128, and the paper is unusually transparent about training details and about the limits of the benchmark comparison. However, the headline state-of-the-art claim is not yet supported by the evidence as presented: the margins over copied baselines are tiny, there are no variance estimates, and the paper itself concedes that the comparison may be unfair. The value of the paper is therefore primarily as an engineering/technical report; its scientific claims need tightening or additional experiments.

major comments (3)
  1. [Abstract; Section 3.2; Table 4] The state-of-the-art claim in the abstract is not established by Table 4 because most baseline scores are copied from Refs. [8] and [4] under different fine-tuning protocols, as the text concedes ('the comparison may not be entirely fair'). The protocol mismatch is visible even within the table: the authors' own BERTBASE-WWM reimplementation differs from the published [8] numbers (e.g., LCQMC test 87.16 vs 87.00; XNLI test 78.44 vs 78.20; ChnSenti test 94.58 vs 95.40). The best NEZHA variant beats the best reported baseline on test by at most +0.20 (ChnSenti 96.00 vs 95.80), +0.17 (XNLI 81.17 vs 81.00), and +0.04 (LCQMC 87.94 vs 87.90), and ties on PD-NER at 98.58. To support the claim, please re-run the baselines under the Table 3 protocol with several seeds or, failing that, replace 'state-of-the-art' with a qualified statement such as 'competitive' and specify the setting.
  2. [Section 3.2; Tables 4 and 5] All results in Tables 4 and 5 are single runs without error bars or significance tests. Given that the headline margins are below 0.2 points on test and that several dev comparisons favor the baseline (ERNIE-LARGE 2.0 dev: XNLI 82.60 vs 82.21, LCQMC 90.90 vs 90.87, ChnSenti 96.10 vs 95.92), seed-level variance could reverse every claimed advantage. Please report means and standard deviations over at least five fine-tuning seeds, or explicitly state that differences are within noise and do not claim superiority.
  3. [Abstract; Section 3.2] The abstract's 'state-of-the-art' claim is ambiguous between dev and test splits, and the body reports both. On test, NEZHA-LARGE-WWM is best on XNLI, LCQMC, and ChnSenti but only ties on PD-NER; on dev it is not best on XNLI, LCQMC, or ChnSenti. The claim should specify the split(s) being referenced and discuss the dev/test discrepancy.
minor comments (6)
  1. [Table 4] The PD-NER row for BERTBASE-WWM (in [8]) reports a test value of 65.10, which is inconsistent with the neighboring values around 98 and appears to be a data-entry error; please correct it and check the rest of the copied rows for alignment.
  2. [Section 3.2] The first sentence contains a typo, 'Goolge's BERT', which should read 'Google's BERT'.
  3. [Section 3.3; Table 5] In Table 5, comparing the rows 'News+Wiki+Baike, FRPE, SL:128' and 'News+Wiki+Baike, FRPE, WWM, SL:128' shows mixed effects: ChnSenti dev drops from 95.25 to 94.75 and CMRC F1 drops from 86.41 to 86.25, while other metrics improve; the text's statement that the techniques 'generally have positive contributions' is therefore too strong. Please qualify the claim or report seed-level variance.
  4. [Section 2.2] The claim that fixed sinusoidal encodings 'may allow extrapolation to sequence lengths longer than the ones encountered during training' is not tested; the SL:512 ablation only varies the training length, not extrapolation. Please add an extrapolation experiment or soften the statement.
  5. [Section 3.1] The exact versions of the crawled corpora, the cleaning pipeline, and the Jieba segmenter are not specified, which limits reproducibility of the pre-training data; please provide version/checksum details or links.
  6. [Equations (6)-(7)] The indexing with 2k and 2k+1 assumes that dz is even; please state this assumption explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NEZHA's claims are empirical benchmark results based on externally cited techniques, with no fitted parameter or self-citation chain driving the reported numbers.

full rationale

The paper's central claims are empirical: NEZHA combines BERT with functional relative positional encoding (Eqs. 6-7), whole word masking [8], mixed precision training [13], and the LAMB optimizer [14], and reports fine-tuned benchmark scores. None of these components is defined in terms of the downstream tasks; the functional relative positional encoding is a fixed sinusoidal function of relative position with no trainable parameters and no fitting to the evaluated datasets. The ablations compare model variants under NEZHA's own protocol and are internally consistent. The only explicit caveat is baseline comparability: 'Note that the part of the results are directly copied from the original papers [8, 4]. Due to the possible differences in the experimental setting or finetuning methods, the comparison may not be entirely fair' (Section 3.2). That is a fairness and validity limitation, not a circular reduction: the copied baseline numbers are not inputs to the model's construction, and they do not make the reported NEZHA scores equal to the inputs by definition. Citation [5] shares some authors with the present paper, but it appears only in the introduction as related work and is not load-bearing for NEZHA's results. Thus the derivation chain is self-contained, with no fitted parameter renamed as a prediction and no self-citation chain forcing the central claim.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The scientific content is empirical; the claims rest on hand-chosen training hyperparameters, on the quality of Jieba segmentation, on the representativeness of five Chinese tasks, and on the comparability of baseline numbers copied from other papers. No new theoretical entities are introduced.

free parameters (7)
  • Maximum learning rate (BASE) = 1.8e-4
    Hand-tuned pre-training hyperparameter listed in Section 3.1; affects convergence and final task performance but not fit to downstream metrics.
  • Maximum learning rate (LARGE) = 1e-4
    Hand-tuned pre-training hyperparameter listed in Section 3.1.
  • Number of warm-up steps = 1800
    Hand-chosen for both BASE and LARGE pre-training; Section 3.1.
  • Total batch size = 14400 (BASE), 5120 (LARGE)
    Hardware-dependent training choice; Section 3.1.
  • Masked and replaced token ratios = 12% masked, 1.5% replaced
    Inherited from BERT but chosen by hand; Section 2.3. Important for the MLM pre-training objective.
  • Wavelength scale 10000 in positional encoding = 10000
    Chosen to match the Transformer sinusoidal encoding; Eq. 6-7. Not fit to data.
  • Fine-tuning hyperparameters per task = 10 configurations in Table 3
    Batch sizes, learning rates, and epochs are hand-selected per downstream task; they influence the reported scores in Table 4.
assumptions (4)
  • domain assumption Jieba segmentation provides sufficiently accurate word boundaries for Whole Word Masking.
    Section 2.3; WWM quality depends on segmentation accuracy, and incorrect boundaries would change the masking distribution.
  • domain assumption Results reported in prior papers for BERT-WWM and ERNIE-Baidu are directly comparable to NEZHA runs.
    Section 3.2 acknowledges 'the comparison may not be entirely fair'; the central SOTA claim implicitly assumes comparability despite different fine-tuning settings.
  • domain assumption The five tasks (CMRC, XNLI, LCQMC, PD-NER, ChnSenti) are representative of Chinese NLU.
    Section 3.2; generalization to other Chinese tasks is extrapolated from these five benchmarks.
  • domain assumption Masked Language Modeling and Next Sentence Prediction are effective pre-training objectives for Chinese.
    Used throughout; standard practice inherited from BERT without re-validation on Chinese data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NEZHA: Neural Contextualized Representation for Chinese Language Understanding." pith.science (2026). https://pith.science/paper/5SFZP4XK

@misc{pith2026190900204,
  author       = {Pith},
  title        = {Pith review of: NEZHA: Neural Contextualized Representation for Chinese Language Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5SFZP4XK}},
  note         = {Machine review of arXiv:1909.00204}
}
read the original abstract

The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora. In this technical report, we present our practice of pre-training language models named NEZHA (NEural contextualiZed representation for CHinese lAnguage understanding) on Chinese corpora and finetuning for the Chinese NLU tasks. The current version of NEZHA is based on BERT with a collection of proven improvements, which include Functional Relative Positional Encoding as an effective positional encoding scheme, Whole Word Masking strategy, Mixed Precision Training and the LAMB Optimizer in training the models. The experimental results show that NEZHA achieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including named entity recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment classification (ChnSenti) and natural language inference (XNLI).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction

    cs.CL 2024-12 conditional novelty 5.0 of 10

    The ASR-EC benchmark on Chinese ASR errors shows that multimodal LLM augmentation corrects ASR output best, while prompting alone worsens CER.

Reference graph

Works this paper leans on

18 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [8]

    Pre-training with whole word masking for chinese bert

    Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Y ang , Shijin Wang, and Guoping Hu. Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101 , 2019

    arXiv 1906

  2. [4]

    Ernie 2.0: A continual pre-training framework for language understanding

    Y u Sun, Shuohuan Wang, Y ukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. arXiv preprint arXiv:1907.12412 , 2019

    arXiv 1907

  3. [1]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina T outanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Cha pter of the Association for Computational Linguistics: Human La nguage T echnologies, V olume 1 (Long and Short Papers), pages 4171–4186, 2019

    work page 2019

  4. [2]

    Deep contextualized word representations

    Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardne r, Christopher Clark, Kenton Lee, and Luke Zettle- moyer. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237, 2018

    work page 2018

  5. [3]

    Ernie: Enhanced representation through knowled ge integration

    Y u Sun, Shuohuan Wang, Y ukun Li, Shikun Feng, Xuyi Chen, H an Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowled ge integration. arXiv preprint arXiv:1904.09223 , 2019

    arXiv 1904

  6. [5]

    Ernie: Enhanced language representation with informative entities

    Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong S un, and Qun Liu. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129 , 2019

    arXiv 1905

  7. [6]

    Xlnet: Gener- alized autoregressive pretraining for language understan ding

    Zhilin Y ang, Zihang Dai, Yiming Y ang, Jaime Carbonell, R uslan Salakhutdinov, and Quoc V Le. Xlnet: Gener- alized autoregressive pretraining for language understan ding. arXiv preprint arXiv:1906.08237 , 2019

    arXiv 1906

  8. [7]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Jos hi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and V eselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. 2019

    work page 2019

Show all 18 references
  1. [9]

    Attention is all you need

    Ashish V aswani, Noam Shazeer, Niki Parmar, Jakob Uszkor eit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998–6008, 2017

    2017

  2. [10]

    Google’s ne ural machine translation system: Bridging the gap between human and machine translation

    Y onghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Moha mmad Norouzi, Wolfgang Macherey, Maxim Krikun, Y uan Cao, Qin Gao, Klaus Macherey, et al. Google’s ne ural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08...

    2016 arXiv

  3. [11]

    Self- attention with relative position representations

    Peter Shaw, Jakob Uszkoreit, and Ashish V aswani. Self- attention with relative position representations. In Proceedings of the 2018 Conference of the North American Cha pter of the Association for Computational Lin- guistics: Human Language T echnologies, V olume 2 (Short Pap...

    2018

  4. [12]

    Transformer-xl: Attentive language models beyond a fix ed-length context

    Zihang Dai, Zhilin Y ang, Yiming Y ang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fix ed-length context. arXiv preprint arXiv:1901.02860, 2019

    1901 arXiv

  5. [13]

    Mixed precision training

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gre gory Diamos, Erich Elsen, David Garcia, Boris Gins- burg, Michael Houston, Oleksii Kuchaiev, Ganesh V enkatesh , et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017

    arXiv

  6. [14]

    Reducing bert pre-training time from 3 days to 76 minutes

    Y ang Y ou, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Reducing bert pre-training time from 3 days to 76 minutes. arXiv preprint arXiv:1904.00962 , 2019

    1904 arXiv

  7. [15]

    Horovod: fast and easy distributed deep learning in TensorFlow

    Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799, 2018

    2018 arXiv

  8. [16]

    A span-extraction dataset for chinese machine reading compr ehension

    Yiming Cui, Ting Liu, Li Xiao, Zhipeng Chen, Wentao Ma, W anxiang Che, Shijin Wang, and Guoping Hu. A span-extraction dataset for chinese machine reading compr ehension. arXiv preprint arXiv:1810.07366 , 2018

    2018 arXiv

  9. [17]

    Xnli: Evaluating cross-lingual sentenc e representations

    Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina W illiams, Samuel R Bowman, Holger Schwenk, and V eselin Stoyanov. Xnli: Evaluating cross-lingual sentenc e representations. arXiv preprint arXiv:1809.05053 , 2018

    2018 arXiv

  10. [18]

    Lcqmc: A large- scale chinese question matching corpus

    Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Che n, Dongfang Li, and Buzhou Tang. Lcqmc: A large- scale chinese question matching corpus. In Proceedings of the 27th International Conference on Comput ational Linguistics, pages 1952–1962, 2018. 8

    1952

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.