REVIEW 4 major objections 5 minor 28 references
Generating Classical Chinese Poems from Vernacular Chinese
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read An unsupervised model can translate vernacular Chinese paragraphs into classical Chinese poems while preserving much of the input meaning, the paper argues.
desk verdict A genuinely new task (vernacular-to-classical poem generation) with two sensible, well-ablated fixes for the length gap; the main thing to check in review is whether the test poems were held out from the training corpus. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is phrase-segmentation-based padding: each poem line is split at traditional phrase boundaries and padded with a special token to double its length, letting a 28-character quatrain align with a roughly 70-character vernacular paragraph. The second mechanism is a reinforcement-learning anti-repetition loss: for a back-translated vernacular paragraph $S$, the model computes the repetition ratio $RR(S) = 1 - \frac{\mathrm{vocab}(S)}{\mathrm{len}(S)}$ and, following self-critical policy gradient training, down-weights sequences whose ratio exceeds a threshold $\tau$. Together, the two mechanisms target under-translation (missed trailing content) and over-translation (repeated renderings of the same source phrase).
What would settle it
Search the 163,000-poem training corpus for the 487 test poems; if any gold poem appears in training, the reported perplexity and BLEU improvements may be inflated by memorization rather than translation.
Extended reading notes
Core claim
The central discovery claimed is that the large length and abstraction gap between vernacular Chinese and classical poetry can be handled inside an unsupervised translation framework. The model builds a shared character embedding space, trains denoising language models for each side, and uses back-translation so the two sides reconstruct each other. The paper’s specific contribution is that inserting padding tokens at phrase boundaries (2-2-3 for seven-character quatrains) gives the decoder room to cover the second half of a vernacular paragraph, while a self-critical reinforcement objective penalizing high repetition ratio suppresses the model’s tendency to repeat content when expanding a terse poem. The paper reports consistent improvement over a naive transformer in perplexity and BLEU, and human evaluators rate the best model’s output as comparable to amateur human poetry, especially in preserving input meaning.
Load-bearing premise
The reported gains depend on the 487 test poems actually being absent from the 163,000-poem training set and on the online vernacular translations being faithful enough to serve as gold references.
Editorial extensions
If this is right
- Users can steer poem content by writing a vernacular paragraph, so generation control moves from isolated keywords to whole meaning.
- The same padding-plus-repetition-penalty design should transfer to other unsupervised tasks where source and target lengths differ sharply, such as summarization.
- Inputs rich in classical poetic imagery or descriptive wording yield better poems, while logical or philosophical inputs confuse the model.
- The best generated poems are close enough to amateur human poems that evaluators could not reliably distinguish them (65.8% mean accuracy).
- Low BLEU scores do not necessarily mean poor quality, because many different poems can express the same vernacular meaning.
Reading between the lines
- The discrimination test used 25 manually selected generated poems, so the “comparable to amateur poems” result probably describes the model’s best-case output rather than its typical output; a random sample would be a stricter test.
- A natural next experiment is to apply the same recipe to unsupervised summarization or style transfer with a long source and short target; the paper lists summarization as future work, but the mechanism suggests it.
- The reliance on paired vernacular translations as gold references means evaluation quality hinges on translator fidelity; using multiple translations per poem or human meaning-equivalence judgments would harden the comparison.
- The padding benefit is tied to the 2-2-3 quatrain rhythm; adapting to other classical forms would require their corresponding segmentation schemas and would test whether padding’s advantage is semantic or mainly positional.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a new task, generating classical Chinese quatrain poems from vernacular Chinese paragraphs, and adapts an unsupervised machine translation (UMT) framework to this task. To handle the length and abstraction gap between vernacular prose and classical poems, the authors introduce two modifications: phrase-segmentation-based padding, which pads poem lines according to the traditional 2-2-3 segmentation schema to alleviate under-translation, and a reinforcement learning loss, which penalizes repetitive back-translations to alleviate over-translation. Experiments compare an LSTM baseline, a naive transformer, and three transformer variants on 487 test poems paired with vernacular translations, reporting perplexity and BLEU improvements for the full model. The paper also reports human evaluations of fluency, semantic coherence, semantic preservability, and poeticness, plus a human discrimination test in which judges were asked to distinguish generated poems from human-written poems. The central claim is that the proposed model significantly improves translation quality and can generate poems comparable to amateur human poems.
Significance. If the results hold, this is a worthwhile contribution to computational creativity and Chinese poetry generation: it defines a useful and under-explored task, adapts UMT machinery in a non-trivial way, and provides a concrete mechanism for user control over poem semantics through the vernacular input. The paper also ships code and data, and the human evaluation is a genuine attempt to assess subjective quality. The reported improvements are internally consistent across automatic metrics and human judgments, and the analysis of input characteristics in Section 4.5 is a useful practical guide. The main caveat is that the quantitative evidence is only as strong as the test-set construction and the statistical reporting, both of which currently have gaps.
major comments (4)
- [Section 4.1, Table 1] The paper never states that the 487 test quatrains were excluded from the 163K training poems or the 19K validation poems. Section 4.1 describes the 163K poem corpus and the separate 487 test poems both as collected from online Tang and Song sources, but no deduplication or holdout procedure is reported. Since the UMT training objective in Eq. (2) includes reconstructing poems from their back-translated or noisy versions, any test poem present in the training or validation corpus would directly inflate perplexity and BLEU scores in Table 3. This is a load-bearing issue for the paper's quantitative claims, and it must be fixed by explicitly stating that the test poems were removed from both corpora and by reporting any deduplication checks.
- [Section 4.4, Tables 3 and 4] The headline claim of 'significantly improve' is not supported by any statistical test, confidence interval, or error bar. Table 3 reports mean perplexity and BLEU over five epochs after convergence, but BLEU differences between '+Anti OT' and '+Anti UT' are 6.08 vs 6.34, and the overall BLEU scores are all below 7, which is a small absolute difference. Table 4 reports human evaluation means without variance or inter-annotator agreement. The authors should provide paired bootstrap or permutation tests for the automatic metrics on the 487 test items, and report standard deviations, confidence intervals, and inter-annotator agreement for the human evaluations. Without these, the claim of significance remains unverified.
- [Section 5, 'Addressing Over-Translation'] The repetition-ratio result, a drop from 40.8% to 34.9% with the RL loss, is partly a check of the training objective rather than independent evidence of improved generation quality. Equation (5) defines the RL loss to penalize high repetition ratio (RR(S_TS) - tau), so the model is explicitly optimized to reduce RR on back-translated paragraphs. To substantiate that this reduction is not achieved at the expense of other qualities, the authors should report repetition ratio on held-out generated samples not used in training, and correlate the RL-induced change with the downstream metrics and human scores. This is not a reason to reject the paper, but it is a necessary clarification.
- [Section 4.6, Table 7] The human discrimination test uses 25 manually selected generated poems, which is a non-random and potentially biased sample, and reports only a mean accuracy of 65.8% with no confidence interval or per-evaluator variance. The manual selection is acknowledged, but its implications for the claim that generated poems are 'comparable to amateur poems' are not discussed. The authors should either use a random or stratified sample, or explicitly qualify that the comparability claim applies only to cherry-picked outputs. Reporting the distribution of accuracy values and inter-annotator agreement would also help interpret the result.
minor comments (5)
- [Section 4.5] The sentence 'We try to As no ground truth is available' is grammatically incomplete and should be revised.
- [Section 3.2.2 and Section 4.4] There are several typos: 'expending' should be 'expanding', and '+Anti OT' is described as 'mitigating over-fitting' in Section 4.4, but the intended term appears to be 'over-translation'.
- [Section 3.1, Eq. (1)] The notation S_N and T_N is not formally defined; the text says noise is added by 'drop, swap or blank a few words', but the exact noise process is not specified, which makes the language modeling loss difficult to reproduce.
- [Section 4.5 and Section 4.6] The selection criteria for the 20 literature fragments in Section 4.5 and for the 25 test poems in Section 4.6 are not specified, and it is unclear whether the evaluators were the same across the two human evaluations.
- [Table 4] The human evaluation table would be clearer if it reported scores for the expert and amateur groups separately, since the paper states that it anticipated inconsistencies between the groups but does not show the numbers.
Circularity Check
No significant circularity: central claims rest on external gold-standard metrics and human evaluation; the RL repetition-ratio result is a training-objective sanity check, not an independent prediction.
full rationale
The paper's derivation chain is self-contained against external benchmarks. The core UMT training (Eqs. 1-3) uses unaligned monolingual corpora, and the reported effectiveness evidence is BLEU/perplexity against gold poems (Table 3), human grading (Table 4), and a human discrimination test (Table 7), none of which are inputs to the training objective. The phrase-segmentation padding is motivated by an external prosodic schema (Ye, 1984) and is empirically compared across schemas. The closest candidate for circularity is the Section 5 repetition-ratio comparison: Eq. (5) defines the RL loss as a direct function of the repetition ratio RR from Eq. (4), so the observed drop from 40.8% to 34.9% is a consistency check that the RL term moved the quantity it was designed to move, not an independent confirmation; however, the paper's central claim does not rest on that check. I also flag a data-hygiene concern that is not circularity: Section 4.1 ('Training and Validation Sets' and 'Test Set') never states that the 487 test poems were excluded from the 163K/19K corpora, so automatic metrics could be inflated by contamination; that is a correctness risk, not a self-referential derivation, and it does not raise the circularity score.
Assumptions & free parameters
free parameters (5)
- alpha1 (LM loss scaling factor) =
not reported
- alpha2 (back-translation loss scaling factor) =
not reported
- alpha3 (RL loss scaling factor) =
not reported
- tau (repetition-ratio threshold) =
not reported
- padding factor =
2x line length
assumptions (4)
- domain assumption Seven-character quatrain lines follow a 2-2-3 phrase segmentation schema, and padding along those boundaries preserves phrase semantics.
- domain assumption Unsupervised MT with a shared character embedding space and back-translation can align the semantics of vernacular Chinese and classical Chinese without parallel data.
- domain assumption Test-set vernacular translations are faithful, high-quality translations of their paired gold poems.
- ad hoc to paper The RL loss in Eq. 5, which weights likelihood by (RR(S_TS) - tau), provides a valid training signal that improves rather than degrades overall poem quality.
Cite this review
Pith. "Pith review of Generating Classical Chinese Poems from Vernacular Chinese." pith.science (2026). https://pith.science/paper/45DLIGD4
@misc{pith2026190900279,
author = {Pith},
title = {Pith review of: Generating Classical Chinese Poems from Vernacular Chinese},
year = {2026},
howpublished = {\url{https://pith.science/paper/45DLIGD4}},
note = {Machine review of arXiv:1909.00279}
}
read the original abstract
Classical Chinese poetry is a jewel in the treasure house of Chinese culture. Previous poem generation models only allow users to employ keywords to interfere the meaning of generated poems, leaving the dominion of generation to the model. In this paper, we propose a novel task of generating classical Chinese poems from vernacular, which allows users to have more control over the semantic of generated poems. We adapt the approach of unsupervised machine translation (UMT) to our task. We use segmentation-based padding and reinforcement learning to address under-translation and over-translation respectively. According to experiments, our approach significantly improve the perplexity and BLEU compared with typical UMT models. Furthermore, we explored guidelines on how to write the input vernacular to generate better poems. Human evaluation showed our approach can generate high-quality poems which are comparable to amateur poems.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR
2015
-
[4]
Sameer Bansal, Herman Kamper, Adam Lopez, and Sharon Goldwater. 2017. Towards speech-to-text translation without speech recognition. In EACL
work page 2017
-
[5]
Pi-Chuan Chang, Michel Galley, and Christopher D. Manning. 2008. Optimizing chinese word segmentation for machine translation performance. In WMT@ACL
work page 2008
-
[6]
Kyunghyun Cho, Bart van Merrienboer, Çaglar G \"u lçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP
work page 2014
-
[7]
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. 2019. Unsupervised image captioning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
work page 2019
-
[8]
Jing He, Ming Zhou, and Long Jiang. 2012. Generating chinese classical poems with statistical machine translation models. In AAAI
work page 2012
Show all 28 references
-
[9]
Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735--1780
1997
-
[10]
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018 a . Unsupervised machine translation using monolingual corpora only. In ICLR
2018
-
[11]
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018 b . Phrase-based & neural unsupervised machine translation. In EMNLP
2018
-
[12]
Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016. Coverage embedding models for neural machine translation. In EMNLP
2016
-
[13]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In ACL
2001
-
[14]
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In ICLR
2018
-
[15]
Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1179--1195
2017
-
[16]
Baskaran Sankaran, Haitao Mi, Yaser Al-Onaizan, and Abe Ittycheriah. 2016. Temporal attention model for neural machine translation. CoRR, abs/1608.02927
2016 arXiv
-
[17]
Liu, Xiaohua Liu, and Hang Li
Zhaopeng Tu, Zhengdong Lu, Yang P. Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In ACL
2016
-
[18]
Gomez, Lukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS
2017
-
[19]
Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, and Enhong Chen. 2016. Chinese poetry generation with planning based neural network. In COLING
2016
-
[20]
Wei-Hung Weng, Yu-An Chung, and Peter Szolovits. 2019. https://doi.org/10.1145/3292500.3330710 Unsupervised clinical language translation . In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &\#38; Data Mining, KDD '19, pages 3121--3131, New ...
2019
-
[21]
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Ku...
2016 arXiv
-
[22]
Linli Xu, Liang Jiang, Chuan Qin, Zhe Wang, and Dongfang Du. 2018. How images inspire poems: Generating classical chinese poetry from images with memory networks. In AAAI
2018
-
[23]
Rui Yan. 2016. i, poet: Automatic poetry composition through recurrent neural networks with iterative polishing schema. In IJCAI
2016
-
[24]
Cheng Yang, Maosong Sun, Xiaoyuan Yi, and Wenhao Li. 2018. Stylistic chinese poetry generation via unsupervised style disentanglement. In EMNLP
2018
-
[25]
Jiaying Ye. 1984. Poem Criticism with Jialin. Zhong hua shu ju, Beijing, China
1984
-
[26]
Xiaoyuan Yi, Maosong Sun, Ruoyu Li, and Zonghan Yang. 2018. Chinese poetry generation with a working memory model. In IJCAI
2018
-
[27]
Xingxing Zhang and Mirella Lapata. 2014. Chinese poetry generation with recurrent neural networks. In EMNLP
2014
-
[28]
Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018. Style transfer as unsupervised machine translation. CoRR, abs/1808.07894
2018 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.