{"id":"32c2dfc5-9c69-4ed0-b370-fe04051f92df","arxiv_id":"1909.00279","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An unsupervised machine-translation model with phrase padding and an anti-repetition reward generates classical Chinese quatrains from vernacular paragraphs.","lead":"This paper builds a system that turns modern Chinese paragraphs into classical-style poems without needing paired training examples. If it holds up, users can steer poem meaning by how they write the paragraph instead of tweaking keywords.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central risk is test-set leakage: Section 4.1 never states that the 487 test quatrains were excluded from the 163K-poem training/validation corpus, so the reported perplexity and BLEU gains could reflect memorization rather than genuine translation.","rationale":"The reader's weakest assumption identified both test-set leakage and translation fidelity as the fragile points. I focus on the leakage half because it is the more load-bearing: without a holdout guarantee, the central quantitative comparisons in Section 4.4 are uninterpretable, and those comparisons are what the paper uses to argue that both proposed mechanisms help. The possibility is not remote, since the training and test poems both come from Tang/Song sources and the paper gives no deduplication statement. The human evaluation partially survives this concern, because the discrimination test is on a separate manually selected set, but the claim 'comparable to amateur poems' still depends on cherry-picked examples and lacks a baseline scoring of human amateur poems on the same rubric. Those are secondary issues. A conditional verdict remains appropriate: the flaw is in missing verification, not in the core architecture, and a concrete overlap check plus a re-run on a deduplicated test set would resolve it. I therefore leave the reader's CONDITIONAL verdict unchanged.","tokens_in":10447,"tokens_out":6129,"duration_ms":65190,"concrete_test":"Using the released code and data, normalize the 487 test poems and all poems in the training (163K) and validation (19K) sets by removing punctuation and whitespace, then check for exact duplicates and near-duplicates under a small character-level edit distance (e.g., distance <= 2) to catch re-formatting. If any overlap is found, remove all overlapping test items and recompute Table 3 (perplexity and BLEU) on the deduplicated test set. If the perplexity/BLEU improvements of +Anti OT&UT versus the baselines shrink appreciably or reverse, the paper's quantitative headline fails. Even a clean zero-overlap result should be reported explicitly, along with the overlap count and the recomputed metrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 reports that the poem corpus (163K quatrains from Tang and Song poems) is split into training and validation sets, and that the 487 test poems were also collected from online Tang/Song sources. The paper does not state that the 487 test poems were removed from the 163K training corpus or the 19K validation corpus. This is exactly the condition required for the automatic metrics to be meaningful. Since the UMT training objective in Eq. (2) includes reconstructing poems from their noisy/back-translated versions, any test poem present in training is a target the model was explicitly optimized to output; perplexity (probability of the gold poem) and BLEU (n-gram overlap with the gold poem) would then be inflated, and the comparisons in Table 3 between +Anti OT&UT and the baselines would no longer support the paper's claim of improved semantic preservation. The paper also asserts that the test-set vernacular translations are 'high quality' without describing a verification or deduplication procedure, but the more decisive missing support is the holdout statement. If overlap exists, the principal quantitative evidence collapses, though the human evaluations might still stand; the issue is fixable by reporting and removing duplicates, so it warrants a conditional acceptance rather than an outright rejection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new task, generating classical Chinese quatrain poems from vernacular Chinese paragraphs, and adapts an unsupervised machine translation (UMT) framework to this task. To handle the length and abstraction gap between vernacular prose and classical poems, the authors introduce two modifications: phrase-segmentation-based padding, which pads poem lines according to the traditional 2-2-3 segmentation schema to alleviate under-translation, and a reinforcement learning loss, which penalizes repetitive back-translations to alleviate over-translation. Experiments compare an LSTM baseline, a naive transformer, and three transformer variants on 487 test poems paired with vernacular translations, reporting perplexity and BLEU improvements for the full model. The paper also reports human evaluations of fluency, semantic coherence, semantic preservability, and poeticness, plus a human discrimination test in which judges were asked to distinguish generated poems from human-written poems. The central claim is that the proposed model significantly improves translation quality and can generate poems comparable to amateur human poems.","tokens_in":10763,"tokens_out":2592,"duration_ms":26715,"significance":"If the results hold, this is a worthwhile contribution to computational creativity and Chinese poetry generation: it defines a useful and under-explored task, adapts UMT machinery in a non-trivial way, and provides a concrete mechanism for user control over poem semantics through the vernacular input. The paper also ships code and data, and the human evaluation is a genuine attempt to assess subjective quality. The reported improvements are internally consistent across automatic metrics and human judgments, and the analysis of input characteristics in Section 4.5 is a useful practical guide. The main caveat is that the quantitative evidence is only as strong as the test-set construction and the statistical reporting, both of which currently have gaps.","major_comments":[{"comment":"The paper never states that the 487 test quatrains were excluded from the 163K training poems or the 19K validation poems. Section 4.1 describes the 163K poem corpus and the separate 487 test poems both as collected from online Tang and Song sources, but no deduplication or holdout procedure is reported. Since the UMT training objective in Eq. (2) includes reconstructing poems from their back-translated or noisy versions, any test poem present in the training or validation corpus would directly inflate perplexity and BLEU scores in Table 3. This is a load-bearing issue for the paper's quantitative claims, and it must be fixed by explicitly stating that the test poems were removed from both corpora and by reporting any deduplication checks.","section":"Section 4.1, Table 1"},{"comment":"The headline claim of 'significantly improve' is not supported by any statistical test, confidence interval, or error bar. Table 3 reports mean perplexity and BLEU over five epochs after convergence, but BLEU differences between '+Anti OT' and '+Anti UT' are 6.08 vs 6.34, and the overall BLEU scores are all below 7, which is a small absolute difference. Table 4 reports human evaluation means without variance or inter-annotator agreement. The authors should provide paired bootstrap or permutation tests for the automatic metrics on the 487 test items, and report standard deviations, confidence intervals, and inter-annotator agreement for the human evaluations. Without these, the claim of significance remains unverified.","section":"Section 4.4, Tables 3 and 4"},{"comment":"The repetition-ratio result, a drop from 40.8% to 34.9% with the RL loss, is partly a check of the training objective rather than independent evidence of improved generation quality. Equation (5) defines the RL loss to penalize high repetition ratio (RR(S_TS) - tau), so the model is explicitly optimized to reduce RR on back-translated paragraphs. To substantiate that this reduction is not achieved at the expense of other qualities, the authors should report repetition ratio on held-out generated samples not used in training, and correlate the RL-induced change with the downstream metrics and human scores. This is not a reason to reject the paper, but it is a necessary clarification.","section":"Section 5, 'Addressing Over-Translation'"},{"comment":"The human discrimination test uses 25 manually selected generated poems, which is a non-random and potentially biased sample, and reports only a mean accuracy of 65.8% with no confidence interval or per-evaluator variance. The manual selection is acknowledged, but its implications for the claim that generated poems are 'comparable to amateur poems' are not discussed. The authors should either use a random or stratified sample, or explicitly qualify that the comparability claim applies only to cherry-picked outputs. Reporting the distribution of accuracy values and inter-annotator agreement would also help interpret the result.","section":"Section 4.6, Table 7"}],"minor_comments":[{"comment":"The sentence 'We try to As no ground truth is available' is grammatically incomplete and should be revised.","section":"Section 4.5"},{"comment":"There are several typos: 'expending' should be 'expanding', and '+Anti OT' is described as 'mitigating over-fitting' in Section 4.4, but the intended term appears to be 'over-translation'.","section":"Section 3.2.2 and Section 4.4"},{"comment":"The notation S_N and T_N is not formally defined; the text says noise is added by 'drop, swap or blank a few words', but the exact noise process is not specified, which makes the language modeling loss difficult to reproduce.","section":"Section 3.1, Eq. (1)"},{"comment":"The selection criteria for the 20 literature fragments in Section 4.5 and for the 25 test poems in Section 4.6 are not specified, and it is unclear whether the evaluators were the same across the two human evaluations.","section":"Section 4.5 and Section 4.6"},{"comment":"The human evaluation table would be clearer if it reported scores for the expert and amateur groups separately, since the paper states that it anticipated inconsistencies between the groups but does not show the numbers.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The most serious concern is the missing holdout statement for the 487 test poems in Section 4.1. If the test poems overlap with the 163K training corpus or the 19K validation corpus, the main quantitative results in Table 3 collapse, because the model was optimized to reconstruct those exact poems. This is fixable by reporting a deduplication procedure and re-running the evaluation on a guaranteed disjoint test set. I would also ask the authors to address the statistical-reporting gap and the circularity of the repetition-ratio evidence. The paper is a reasonable fit for the journal's interest in creative language generation, but the quantitative claims need to be placed on firmer ground before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper introduces a genuinely new task, generates classical Chinese quatrains from vernacular paragraphs, and adapts unsupervised machine translation to a setting with a large source/target length gap. The two mechanisms are not conceptually deep—phrase-segmentation-based padding and an RL anti-repetition loss—but they are appropriate and the ablations are honest. I found the 2-2-3 vs 2-3-2 vs 3-2-2 padding-schema comparison particularly convincing: it shows the padding works for the right reason, not merely because it lengthens the sequence. The human evaluation is also a real effort: 30 native speakers, expert and amateur groups, four criteria. The paper earns its claim that the combined model is the best of the tested variants.\n\nThe soft spots are real but mostly fixable. The biggest is the test-set holdout ambiguity. Section 4.1 says the training corpus contains 163K Tang/Song quatrains and the test set is 487 seven-character quatrains from the same online sources, but it never states that the test poems were removed from the training/validation corpora. If they overlap, the perplexity and BLEU gains are inflated. I do not regard this as evidence of misconduct, but the omission is serious because the automatic metrics are the paper's central quantitative support. The stress-test note is on target here.\n\nThe other concerns are smaller. The word 'significantly' in the abstract is not backed by any significance test; the gains are consistent across metrics, but there are no error bars or p-values. The RL loss directly targets repetition ratio, so the 40.8% to 34.9% drop is partly a check of the training objective rather than an independent result—though BLEU and human judgments come from external gold poems and do not have that circularity. The human evaluation reports means without variance or inter-annotator agreement, and the discrimination test uses 25 manually selected poems. These are quality-of-evidence issues, not fatal flaws.\n\nThe citation pattern looks fine, and the code and data are public, which counts for a lot. The paper is written clearly and the authors acknowledge known weaknesses in their discussion, including the structural weakness of generated poems.\n\nWho should read this: people working on Chinese poem generation, on UMT for length-asymmetric tasks, and anyone thinking about unsupervised summarization. It deserves a serious referee, not a desk reject. My recommendation: engage with it, but require the authors to state explicitly whether any of the 487 test poems appear in the 163K training corpus or the 19K validation corpus, and if so, re-run the experiments after deduplication. I would also ask for error bars and a random rather than hand-picked discrimination sample.","headline":"A genuinely new task (vernacular-to-classical poem generation) with two sensible, well-ablated fixes for the length gap; the main thing to check in review is whether the test poems were held out from the training corpus.","tokens_in":852,"tokens_out":1944,"would_cite":true,"duration_ms":34673,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An unsupervised model can translate vernacular Chinese paragraphs into classical Chinese poems while preserving much of the input meaning, the paper argues.","keywords":["classical Chinese poetry","poem generation","vernacular Chinese","unsupervised machine translation","under-translation","over-translation","phrase-segmentation padding","reinforcement learning"],"falsifier":"Search the 163,000-poem training corpus for the 487 test poems; if any gold poem appears in training, the reported perplexity and BLEU improvements may be inflated by memorization rather than translation.","tokens_in":10264,"feed_emoji":"📜","tokens_out":7457,"duration_ms":123728,"temperature":0.7,"pith_summary":"The paper proposes that ordinary modern Chinese prose, not just a handful of keywords, can drive the generation of classical Chinese poems. Because no parallel poem–translation pairs are available, the task is framed as unsupervised machine translation between the two registers, trained on separate monolingual corpora with back-translation and language modeling. The paper claims that two additions—padding poem lines according to their traditional 2-2-3 phrase segmentation, and a reinforcement learning penalty on repeated characters—fix the under-translation and over-translation that a naive model suffers. Experiments show lower perplexity, higher BLEU, better human ratings of fluency and semantic preservation, and output that human judges find close to amateur poetry.","feed_headline":"Vernacular Chinese paragraphs become classical Chinese poems","feed_subtitle":"Phrase-level padding and a repetition penalty lift output near amateur-level verse, human judges say.","key_machinery":"The central mechanism is phrase-segmentation-based padding: each poem line is split at traditional phrase boundaries and padded with a special token to double its length, letting a 28-character quatrain align with a roughly 70-character vernacular paragraph. The second mechanism is a reinforcement-learning anti-repetition loss: for a back-translated vernacular paragraph $S$, the model computes the repetition ratio $RR(S) = 1 - \\frac{\\mathrm{vocab}(S)}{\\mathrm{len}(S)}$ and, following self-critical policy gradient training, down-weights sequences whose ratio exceeds a threshold $\\tau$. Together, the two mechanisms target under-translation (missed trailing content) and over-translation (repeated renderings of the same source phrase).","core_discovery":"The central discovery claimed is that the large length and abstraction gap between vernacular Chinese and classical poetry can be handled inside an unsupervised translation framework. The model builds a shared character embedding space, trains denoising language models for each side, and uses back-translation so the two sides reconstruct each other. The paper’s specific contribution is that inserting padding tokens at phrase boundaries (2-2-3 for seven-character quatrains) gives the decoder room to cover the second half of a vernacular paragraph, while a self-critical reinforcement objective penalizing high repetition ratio suppresses the model’s tendency to repeat content when expanding a terse poem. The paper reports consistent improvement over a naive transformer in perplexity and BLEU, and human evaluators rate the best model’s output as comparable to amateur human poetry, especially in preserving input meaning.","pith_inferences":["The discrimination test used 25 manually selected generated poems, so the “comparable to amateur poems” result probably describes the model’s best-case output rather than its typical output; a random sample would be a stricter test.","A natural next experiment is to apply the same recipe to unsupervised summarization or style transfer with a long source and short target; the paper lists summarization as future work, but the mechanism suggests it.","The reliance on paired vernacular translations as gold references means evaluation quality hinges on translator fidelity; using multiple translations per poem or human meaning-equivalence judgments would harden the comparison.","The padding benefit is tied to the 2-2-3 quatrain rhythm; adapting to other classical forms would require their corresponding segmentation schemas and would test whether padding’s advantage is semantic or mainly positional."],"forward_implications":["Users can steer poem content by writing a vernacular paragraph, so generation control moves from isolated keywords to whole meaning.","The same padding-plus-repetition-penalty design should transfer to other unsupervised tasks where source and target lengths differ sharply, such as summarization.","Inputs rich in classical poetic imagery or descriptive wording yield better poems, while logical or philosophical inputs confuse the model.","The best generated poems are close enough to amateur human poems that evaluators could not reliably distinguish them (65.8% mean accuracy).","Low BLEU scores do not necessarily mean poor quality, because many different poems can express the same vernacular meaning."],"supporting_citations":[{"why":"Supplies the unsupervised machine translation framework (shared embeddings, denoising language modeling, back-translation) that the model adapts.","marker":"Lample et al., 2018b"},{"why":"Source of the reinforcement learning idea of penalizing repeated output to curb over-translation.","marker":"Paulus et al., 2018"},{"why":"Provides the self-critical policy gradient training used to optimize the repetition-ratio loss.","marker":"Rennie et al., 2017"},{"why":"Defines the 2-2-3 phrase segmentation schema for seven-character quatrains on which the padding is based.","marker":"Ye, 1984"},{"why":"Supports that respecting phrase segments preserves semantic content, motivating placement of padding tokens.","marker":"Chang et al., 2008"},{"why":"Provides the transformer architecture used as the base model and as a baseline.","marker":"Vaswani et al., 2017"},{"why":"Defines BLEU, the n-gram overlap metric used to score generated poems against gold poems.","marker":"Papineni et al., 2001"}],"fun_headline_variants":["Unsupervised MT turns vernacular prose into classical Chinese poems","Phrase padding and RL yield classical poems from vernacular Chinese","Vernacular to classical poetry: unsupervised translation with padding","Classical poems from casual Chinese via back-translation and padding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on the 487 test poems actually being absent from the 163,000-poem training set and on the online vernacular translations being faithful enough to serve as gold references.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised MT turns vernacular prose into classical Chinese poems","Phrase padding and RL yield classical poems from vernacular Chinese","Vernacular to classical poetry: unsupervised translation with padding","Classical poems from casual Chinese via back-translation and padding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00103,"raw_usage":{"total_tokens":4283,"prompt_tokens":833,"completion_tokens":3450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":3382}},"tokens_in":449,"tokens_out":3450,"duration_ms":24271,"temperature":1.0,"reasoning_tokens":3382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:56:49.433771+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Search the 163,000-poem training corpus for the 487 test poems; if any gold poem appears in training, the reported perplexity and BLEU improvements may be inflated by memorization rather than translation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the reinforcement learning idea of penalizing repeated output to curb over-translation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports that respecting phrase segments preserves semantic content, motivating placement of padding tokens."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BLEU, the n-gram overlap metric used to score generated poems against gold poems."}],"review_version":1}