Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Generating Classical Chinese Poems from Vernacular Chinese

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read An unsupervised model can translate vernacular Chinese paragraphs into classical Chinese poems while preserving much of the input meaning, the paper argues.

desk verdict A genuinely new task (vernacular-to-classical poem generation) with two sensible, well-ablated fixes for the length gap; the main thing to check in review is whether the test poems were held out from the training corpus. read the letter →

arxiv 1909.00279 v1 pith:45DLIGD4 submitted 2019-08-31 cs.CL cs.AIcs.LG

classification cs.CLcs.AIcs.LG
keywords classicalChinesepoetrypoemgenerationvernacularunsupervisedmachinetranslationunder-translationover-translationphrase-segmentationpaddingreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that ordinary modern Chinese prose, not just a handful of keywords, can drive the generation of classical Chinese poems. Because no parallel poem–translation pairs are available, the task is framed as unsupervised machine translation between the two registers, trained on separate monolingual corpora with back-translation and language modeling. The paper claims that two additions—padding poem lines according to their traditional 2-2-3 phrase segmentation, and a reinforcement learning penalty on repeated characters—fix the under-translation and over-translation that a naive model suffers. Experiments show lower perplexity, higher BLEU, better human ratings of fluency and semantic preservation, and output that human judges find close to amateur poetry.

What carries the argument

The central mechanism is phrase-segmentation-based padding: each poem line is split at traditional phrase boundaries and padded with a special token to double its length, letting a 28-character quatrain align with a roughly 70-character vernacular paragraph. The second mechanism is a reinforcement-learning anti-repetition loss: for a back-translated vernacular paragraph $S$, the model computes the repetition ratio $RR(S) = 1 - \frac{\mathrm{vocab}(S)}{\mathrm{len}(S)}$ and, following self-critical policy gradient training, down-weights sequences whose ratio exceeds a threshold $\tau$. Together, the two mechanisms target under-translation (missed trailing content) and over-translation (repeated renderings of the same source phrase).

What would settle it

Search the 163,000-poem training corpus for the 487 test poems; if any gold poem appears in training, the reported perplexity and BLEU improvements may be inflated by memorization rather than translation.

Watch

Extended reading notes

Core claim

The central discovery claimed is that the large length and abstraction gap between vernacular Chinese and classical poetry can be handled inside an unsupervised translation framework. The model builds a shared character embedding space, trains denoising language models for each side, and uses back-translation so the two sides reconstruct each other. The paper’s specific contribution is that inserting padding tokens at phrase boundaries (2-2-3 for seven-character quatrains) gives the decoder room to cover the second half of a vernacular paragraph, while a self-critical reinforcement objective penalizing high repetition ratio suppresses the model’s tendency to repeat content when expanding a terse poem. The paper reports consistent improvement over a naive transformer in perplexity and BLEU, and human evaluators rate the best model’s output as comparable to amateur human poetry, especially in preserving input meaning.

Load-bearing premise

The reported gains depend on the 487 test poems actually being absent from the 163,000-poem training set and on the online vernacular translations being faithful enough to serve as gold references.

Editorial extensions

If this is right

  • Users can steer poem content by writing a vernacular paragraph, so generation control moves from isolated keywords to whole meaning.
  • The same padding-plus-repetition-penalty design should transfer to other unsupervised tasks where source and target lengths differ sharply, such as summarization.
  • Inputs rich in classical poetic imagery or descriptive wording yield better poems, while logical or philosophical inputs confuse the model.
  • The best generated poems are close enough to amateur human poems that evaluators could not reliably distinguish them (65.8% mean accuracy).
  • Low BLEU scores do not necessarily mean poor quality, because many different poems can express the same vernacular meaning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The discrimination test used 25 manually selected generated poems, so the “comparable to amateur poems” result probably describes the model’s best-case output rather than its typical output; a random sample would be a stricter test.
  • A natural next experiment is to apply the same recipe to unsupervised summarization or style transfer with a long source and short target; the paper lists summarization as future work, but the mechanism suggests it.
  • The reliance on paired vernacular translations as gold references means evaluation quality hinges on translator fidelity; using multiple translations per poem or human meaning-equivalence judgments would harden the comparison.
  • The padding benefit is tied to the 2-2-3 quatrain rhythm; adapting to other classical forms would require their corresponding segmentation schemas and would test whether padding’s advantage is semantic or mainly positional.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a new task, generating classical Chinese quatrain poems from vernacular Chinese paragraphs, and adapts an unsupervised machine translation (UMT) framework to this task. To handle the length and abstraction gap between vernacular prose and classical poems, the authors introduce two modifications: phrase-segmentation-based padding, which pads poem lines according to the traditional 2-2-3 segmentation schema to alleviate under-translation, and a reinforcement learning loss, which penalizes repetitive back-translations to alleviate over-translation. Experiments compare an LSTM baseline, a naive transformer, and three transformer variants on 487 test poems paired with vernacular translations, reporting perplexity and BLEU improvements for the full model. The paper also reports human evaluations of fluency, semantic coherence, semantic preservability, and poeticness, plus a human discrimination test in which judges were asked to distinguish generated poems from human-written poems. The central claim is that the proposed model significantly improves translation quality and can generate poems comparable to amateur human poems.

Significance. If the results hold, this is a worthwhile contribution to computational creativity and Chinese poetry generation: it defines a useful and under-explored task, adapts UMT machinery in a non-trivial way, and provides a concrete mechanism for user control over poem semantics through the vernacular input. The paper also ships code and data, and the human evaluation is a genuine attempt to assess subjective quality. The reported improvements are internally consistent across automatic metrics and human judgments, and the analysis of input characteristics in Section 4.5 is a useful practical guide. The main caveat is that the quantitative evidence is only as strong as the test-set construction and the statistical reporting, both of which currently have gaps.

major comments (4)
  1. [Section 4.1, Table 1] The paper never states that the 487 test quatrains were excluded from the 163K training poems or the 19K validation poems. Section 4.1 describes the 163K poem corpus and the separate 487 test poems both as collected from online Tang and Song sources, but no deduplication or holdout procedure is reported. Since the UMT training objective in Eq. (2) includes reconstructing poems from their back-translated or noisy versions, any test poem present in the training or validation corpus would directly inflate perplexity and BLEU scores in Table 3. This is a load-bearing issue for the paper's quantitative claims, and it must be fixed by explicitly stating that the test poems were removed from both corpora and by reporting any deduplication checks.
  2. [Section 4.4, Tables 3 and 4] The headline claim of 'significantly improve' is not supported by any statistical test, confidence interval, or error bar. Table 3 reports mean perplexity and BLEU over five epochs after convergence, but BLEU differences between '+Anti OT' and '+Anti UT' are 6.08 vs 6.34, and the overall BLEU scores are all below 7, which is a small absolute difference. Table 4 reports human evaluation means without variance or inter-annotator agreement. The authors should provide paired bootstrap or permutation tests for the automatic metrics on the 487 test items, and report standard deviations, confidence intervals, and inter-annotator agreement for the human evaluations. Without these, the claim of significance remains unverified.
  3. [Section 5, 'Addressing Over-Translation'] The repetition-ratio result, a drop from 40.8% to 34.9% with the RL loss, is partly a check of the training objective rather than independent evidence of improved generation quality. Equation (5) defines the RL loss to penalize high repetition ratio (RR(S_TS) - tau), so the model is explicitly optimized to reduce RR on back-translated paragraphs. To substantiate that this reduction is not achieved at the expense of other qualities, the authors should report repetition ratio on held-out generated samples not used in training, and correlate the RL-induced change with the downstream metrics and human scores. This is not a reason to reject the paper, but it is a necessary clarification.
  4. [Section 4.6, Table 7] The human discrimination test uses 25 manually selected generated poems, which is a non-random and potentially biased sample, and reports only a mean accuracy of 65.8% with no confidence interval or per-evaluator variance. The manual selection is acknowledged, but its implications for the claim that generated poems are 'comparable to amateur poems' are not discussed. The authors should either use a random or stratified sample, or explicitly qualify that the comparability claim applies only to cherry-picked outputs. Reporting the distribution of accuracy values and inter-annotator agreement would also help interpret the result.
minor comments (5)
  1. [Section 4.5] The sentence 'We try to As no ground truth is available' is grammatically incomplete and should be revised.
  2. [Section 3.2.2 and Section 4.4] There are several typos: 'expending' should be 'expanding', and '+Anti OT' is described as 'mitigating over-fitting' in Section 4.4, but the intended term appears to be 'over-translation'.
  3. [Section 3.1, Eq. (1)] The notation S_N and T_N is not formally defined; the text says noise is added by 'drop, swap or blank a few words', but the exact noise process is not specified, which makes the language modeling loss difficult to reproduce.
  4. [Section 4.5 and Section 4.6] The selection criteria for the 20 literature fragments in Section 4.5 and for the 25 test poems in Section 4.6 are not specified, and it is unclear whether the evaluators were the same across the two human evaluations.
  5. [Table 4] The human evaluation table would be clearer if it reported scores for the expert and amateur groups separately, since the paper states that it anticipated inconsistencies between the groups but does not show the numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central claims rest on external gold-standard metrics and human evaluation; the RL repetition-ratio result is a training-objective sanity check, not an independent prediction.

full rationale

The paper's derivation chain is self-contained against external benchmarks. The core UMT training (Eqs. 1-3) uses unaligned monolingual corpora, and the reported effectiveness evidence is BLEU/perplexity against gold poems (Table 3), human grading (Table 4), and a human discrimination test (Table 7), none of which are inputs to the training objective. The phrase-segmentation padding is motivated by an external prosodic schema (Ye, 1984) and is empirically compared across schemas. The closest candidate for circularity is the Section 5 repetition-ratio comparison: Eq. (5) defines the RL loss as a direct function of the repetition ratio RR from Eq. (4), so the observed drop from 40.8% to 34.9% is a consistency check that the RL term moved the quantity it was designed to move, not an independent confirmation; however, the paper's central claim does not rest on that check. I also flag a data-hygiene concern that is not circularity: Section 4.1 ('Training and Validation Sets' and 'Test Set') never states that the 487 test poems were excluded from the 163K/19K corpora, so automatic metrics could be inflated by contamination; that is a correctness risk, not a self-referential derivation, and it does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The method's load-bearing ingredients are inherited UMT assumptions, a poetic segmentation convention, the test-set gold standard, and several unstated hyperparameters (loss weights, tau, padding factor).

free parameters (5)
  • alpha1 (LM loss scaling factor) = not reported
    Scaling factor in composite loss L = alpha1*Llm + alpha2*Lbt (Eq. 3). Value is chosen by hand; no grid or selection procedure reported.
  • alpha2 (back-translation loss scaling factor) = not reported
    Scaling factor for the back-translation loss in Eq. 3 and Eq. 6. Value is not reported.
  • alpha3 (RL loss scaling factor) = not reported
    Scaling factor for the RL repetition-penalty loss in Eq. 6. Value is not reported.
  • tau (repetition-ratio threshold) = not reported
    Threshold in Eq. 5 that decides whether a generated vernacular paragraph is rewarded or punished. It is a manually set hyperparameter.
  • padding factor = 2x line length
    Based on the observed vernacular/poem ratio of 73.3/32.0, poem lines are padded to twice their original length during training. This choice is not tuned or varied.
assumptions (4)
  • domain assumption Seven-character quatrain lines follow a 2-2-3 phrase segmentation schema, and padding along those boundaries preserves phrase semantics.
    Adopted from Ye (1984), used in Section 3.2.1 and Section 5. The paper's padding scheme and its superiority over 2-3-2 and 3-2-2 schemas (Table 8) depend on this poetic convention being the right one for the corpus.
  • domain assumption Unsupervised MT with a shared character embedding space and back-translation can align the semantics of vernacular Chinese and classical Chinese without parallel data.
    Central method borrowed from Lample et al. 2018b, used throughout Section 3.1. The whole experiment depends on this transfer being possible across registers with a large length gap.
  • domain assumption Test-set vernacular translations are faithful, high-quality translations of their paired gold poems.
    Used in Section 4.1 and Section 4.4; gold-reference metrics and semantic-preservation judgments are only valid if these online translations really are faithful.
  • ad hoc to paper The RL loss in Eq. 5, which weights likelihood by (RR(S_TS) - tau), provides a valid training signal that improves rather than degrades overall poem quality.
    This is a novel loss choice whose gradient behavior is not analyzed; it is justified only empirically, and the threshold tau is unstated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generating Classical Chinese Poems from Vernacular Chinese." pith.science (2026). https://pith.science/paper/45DLIGD4

@misc{pith2026190900279,
  author       = {Pith},
  title        = {Pith review of: Generating Classical Chinese Poems from Vernacular Chinese},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/45DLIGD4}},
  note         = {Machine review of arXiv:1909.00279}
}
read the original abstract

Classical Chinese poetry is a jewel in the treasure house of Chinese culture. Previous poem generation models only allow users to employ keywords to interfere the meaning of generated poems, leaving the dominion of generation to the model. In this paper, we propose a novel task of generating classical Chinese poems from vernacular, which allows users to have more control over the semantic of generated poems. We adapt the approach of unsupervised machine translation (UMT) to our task. We use segmentation-based padding and reinforcement learning to address under-translation and over-translation respectively. According to experiments, our approach significantly improve the perplexity and BLEU compared with typical UMT models. Furthermore, we explored guidelines on how to write the input vernacular to generate better poems. Human evaluation showed our approach can generate high-quality poems which are comparable to amateur poems.

Figures

Figures reproduced from arXiv: 1909.00279 by the authors.

Figure 1
Figure 1. An example of the training procedures of our model. Here we depict two procedures, namely back [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A real example to show the effectiveness [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples of different padding schemas. The results show our 2-2-3 segmentation￾schema greatly outperforms 2-3-2 and 3-2-2 schema in both perplexity and BLEU scores. Note that the BLEU scores of 2-3-2 and 3-2-2 schema remains almost the same as our naive baseline (Without padding). According to the observation, we have the following conclusions: 1) Although padding better aligns the vernacular paragraph to the poem, … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 20 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR

  4. [4]

    Sameer Bansal, Herman Kamper, Adam Lopez, and Sharon Goldwater. 2017. Towards speech-to-text translation without speech recognition. In EACL

  5. [5]

    Pi-Chuan Chang, Michel Galley, and Christopher D. Manning. 2008. Optimizing chinese word segmentation for machine translation performance. In WMT@ACL

  6. [6]

    Kyunghyun Cho, Bart van Merrienboer, Çaglar G \"u lçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP

  7. [7]

    Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. 2019. Unsupervised image captioning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  8. [8]

    Jing He, Ming Zhou, and Long Jiang. 2012. Generating chinese classical poems with statistical machine translation models. In AAAI

Show all 28 references
  1. [9]

    Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735--1780

  2. [10]

    Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018 a . Unsupervised machine translation using monolingual corpora only. In ICLR

  3. [11]

    Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018 b . Phrase-based & neural unsupervised machine translation. In EMNLP

  4. [12]

    Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016. Coverage embedding models for neural machine translation. In EMNLP

  5. [13]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In ACL

  6. [14]

    Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In ICLR

  7. [15]

    Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel

    Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1179--1195

  8. [16]

    Baskaran Sankaran, Haitao Mi, Yaser Al-Onaizan, and Abe Ittycheriah. 2016. Temporal attention model for neural machine translation. CoRR, abs/1608.02927

  9. [17]

    Liu, Xiaohua Liu, and Hang Li

    Zhaopeng Tu, Zhengdong Lu, Yang P. Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In ACL

  10. [18]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS

  11. [19]

    Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, and Enhong Chen. 2016. Chinese poetry generation with planning based neural network. In COLING

  12. [20]

    Wei-Hung Weng, Yu-An Chung, and Peter Szolovits. 2019. https://doi.org/10.1145/3292500.3330710 Unsupervised clinical language translation . In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &\#38; Data Mining, KDD '19, pages 3121--3131, New ...

  13. [21]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Ku...

  14. [22]

    Linli Xu, Liang Jiang, Chuan Qin, Zhe Wang, and Dongfang Du. 2018. How images inspire poems: Generating classical chinese poetry from images with memory networks. In AAAI

  15. [23]

    Rui Yan. 2016. i, poet: Automatic poetry composition through recurrent neural networks with iterative polishing schema. In IJCAI

  16. [24]

    Cheng Yang, Maosong Sun, Xiaoyuan Yi, and Wenhao Li. 2018. Stylistic chinese poetry generation via unsupervised style disentanglement. In EMNLP

  17. [25]

    Jiaying Ye. 1984. Poem Criticism with Jialin. Zhong hua shu ju, Beijing, China

  18. [26]

    Xiaoyuan Yi, Maosong Sun, Ruoyu Li, and Zonghan Yang. 2018. Chinese poetry generation with a working memory model. In IJCAI

  19. [27]

    Xingxing Zhang and Mirella Lapata. 2014. Chinese poetry generation with recurrent neural networks. In EMNLP

  20. [28]

    Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018. Style transfer as unsupervised machine translation. CoRR, abs/1808.07894

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.