{"id":"9890e5dd-3cfb-4175-a0db-d4453903312f","arxiv_id":"2411.13237","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A block-inverse-prompting loop with revise and rewrite lets GLM-10B-Chinese produce traditional-form poems that human raters scored above GPT-4 and domain-specific systems.","lead":"This paper introduces BIPro, a method that makes a weaker Chinese language model improve its own poetry by revising and rewriting sentences with its block-generation ability. It shows that a 10-billion-parameter model can beat much larger direct-generation systems like GPT-4 on constrained Chinese poetry, suggesting a low-cost route to higher-quality constrained text.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Open-domain 'outperform' claim rests on a 0.14 overall-score margin over Shisanbai (5.27 vs 5.13) with no significance test; the central superiority claim may be statistically unsupported.","rationale":"The reader's weakest assumption identified the BIPro scorer as the key risk. That is a legitimate internal-validity concern: if the perplexity-based scorer does not track poetic quality, the revise/rewrite loop may not explain the observed gains. However, the most load-bearing concern is more basic: the empirical claim of superiority is not backed by inferential statistics. Table 1 shows open-domain overall scores of 5.27 for BIPro versus 5.13 for Shisanbai and 5.22 versus 5.16 in AR. Given the reported variances (Table 4), these differences are small enough to be sampling noise, and the paper provides no paired test, confidence interval, or effect size. The central contribution statement in the introduction explicitly asserts that BIPro 'enabl[es] the less advanced block generative model GLM-10B to outperform both superior generative systems and domain-specific systems'; that assertion requires statistical support. The reader's rationale already mentions 'no significance tests accompany the headline score differences' but does not elevate it to the weakest assumption. I partially agree with the reader: the scorer concern matters, but without a significance test the superiority claim is not yet established, and the concrete check can settle whether it lands. The proposed test uses the promised open-source data. If the difference over Shisanbai is not significant, the paper's headline should be weakened; if it is significant, the concern dissolves. Since this is a fixable evidential gap rather than a demonstrated falsehood, the conditional verdict remains appropriate, with the significance test as a required condition.","tokens_in":13910,"tokens_out":8501,"duration_ms":94740,"concrete_test":"Obtain the open-sourced review data (github.com/xz-keg/BiPro, referenced in Appendix A.6). For the 42 open-domain titles, compute per-title mean overall scores for each system, averaging over reviewers. Run a two-sided paired Wilcoxon signed-rank test between BIPro and Shisanbai, and also BIPro vs GPT-4 and BIPro vs GLM-4, on the 42 paired title-level means; repeat for AR scores and with a paired t-test as robustness. Also compute the 95% confidence interval for the mean per-title difference. If the BIPro-Shisanbai p-value exceeds 0.05, the headline 'outperform domain-specific systems' claim is not supported; if it survives, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that BIPro outperforms both direct generative systems (GPT-4, GLM-4) and domain-specific systems (Yusheng, Shisanbai) in open-domain poem generation. The supporting evidence is the human-evaluation table (Table 1): BIPro's open-domain overall score is 5.27 vs Shisanbai's 5.13 (difference 0.14) and AR 5.22 vs 5.16 (difference 0.06). Table 4 reports standard deviations around 1.0-1.2 for these scores. With 42 titles and 10 reviewers, the standard error of the mean difference is on the order of 0.1, so the margin over Shisanbai is within plausible sampling noise. The paper reports no significance test, no paired per-title analysis, and no inter-annotator agreement statistic. The abstract states that BIPro 'outperform[s] both most advanced direct generative systems ... and best domain-specific systems', but the open-domain margin over Shisanbai is not established as reliable. This is more load-bearing than the scorer-reliability question: even if the scorer is imperfect, the empirical superiority claim is what the paper asserts, and it currently rests on an untested small margin. The same issue affects the conclusion that BIPro narrows the gap to human poems, since no confidence intervals or tests accompany those comparisons either.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BIPro, a zero-shot framework for constrained generation of traditional-form Chinese poems. The method uses a block generative model (GLM-10B-Chinese) and two inverse-prompting operations, revise and rewrite, both scored by the same model's perplexity under a BIPro prompt, to iteratively refine a poem. Two human evaluations are reported: an open-domain challenge comparing BIPro with GPT-4, GLM-4, Baidu Poetry Helper, Yusheng, and Shisanbai, and a parallel-title challenge comparing BIPro with direct GLM-10B generation, GPT-4, and human poems from the Daily Poem section. The authors claim that BIPro achieves the highest human scores in the open-domain task and narrows the gap to human-authored poems in the parallel task.","tokens_in":14143,"tokens_out":6245,"duration_ms":66886,"significance":"The core idea is interesting and potentially useful: exploiting a block model's ability to fill masked spans to revise and rewrite constrained text is a novel twist on inverse prompting. The paper ships code and review data, uses two human evaluations in a difficult literary domain, and reports a large improvement over its own direct-generation baseline (overall 4.65→5.54 in the parallel challenge, Table 1). The evaluation design is thoughtful (anonymous presentation, random ordering, Answer Ranking scores). However, the central claim that BIPro outperforms all baselines is not yet statistically supported, and several methodological asymmetries need to be addressed before the claims can be taken at face value. The skeptical concern about the absence of significance testing lands, and the self-scorer circularity is a real limitation that the paper does not currently resolve.","major_comments":[{"comment":"The headline claim that BIPro 'outperforms' Shisanbai in the open-domain challenge rests on an overall-score margin of 5.27 vs 5.13 (AR 5.22 vs 5.16), while Table 4 reports standard deviations around 1.0–1.2. No significance test, confidence interval, or paired per-title analysis is reported. With 42 titles and 10 reviewers, this margin is within plausible sampling noise; the ambiguity only grows if the reported SDs are across title-level means rather than across reviewer scores. I recommend a paired statistical test or bootstrap confidence interval on per-title means for both the open-domain and parallel challenges, and a corresponding softening of 'outperform' if the margin is not significant.","section":"§4.2, Tables 1 and 4"},{"comment":"Format constraints are verified asymmetrically. BIPro generations are required to satisfy all eight Pingshui constraints, while GPT-4, GLM-4, and Baidu are checked only on the first two constraints and repeatedly prompted until they produce a poem that satisfies those two. This means the direct systems are not held to the same tonal and rhyming requirements, so their lower Format scores may partly reflect unmet constraints rather than intrinsic quality. An apples-to-apples comparison should either apply the full verifier to all systems (reporting the pass rate) or otherwise control for constraint satisfaction; this is load-bearing because Format is one of the four rated dimensions and the comparison with direct systems is central to the paper's claims.","section":"Appendix A.5"},{"comment":"The revise and rewrite loop accepts a replacement only when s, the GLM-10B perplexity under the BIPro prompt, improves. Since the same model is both generator and scorer, the internal objective is not tied to any external standard. The paper provides no evidence that this self-perplexity correlates with human aesthetic judgments, and no ablation separates the contribution of the beam-based constraint generation, the revise operation, the rewrite operation, and the self-scorer. The large human-evaluated improvement over direct GLM-10B (Table 1) could in principle come mostly from the constrained beam search or from the iterative refinement rather than from the scorer as described. I would like to see an ablation that replaces the scorer with a random or prior-based selection rule, and a correlation analysis between s and human scores on the collected data.","section":"§3.4, Algorithm 1"},{"comment":"The human evaluation is the only evidence for the central claims, yet the paper reports no reviewer-agreement statistic and does not clarify the unit of analysis in Table 4. With 10 reviewers and subjective aesthetic scores, reviewer disagreement could drive the variance. Please report inter-annotator agreement (e.g., ICC or Krippendorff's alpha) and state explicitly whether the values in Table 4 are standard deviations across reviewer scores or across title-level means. This also affects the appropriate significance test for the claims.","section":"§A.3, Table 4"}],"minor_comments":[{"comment":"The notation p/p_{k-1} is used but never defined; I assume it denotes the poem with the k-1-th sentence removed, but this should be stated. The scorer signature is also inconsistent: s(p) appears alongside s(prompt, target), so the reader cannot tell what the second argument in s(p/p_{k-1}, p'_k-1) means until reading §3.4.","section":"Algorithm 1"},{"comment":"The caption says 'variance' but the entries are shown as mean ± standard deviation. Please use consistent terminology and indicate which quantity is reported.","section":"Table 4"},{"comment":"The 'Direct Generation' baseline is not direct autoregressive generation: it uses the same beam-based constrained generation method as BIPro but without the BIPro scorer and without revise/rewrite. This is actually a useful ablation baseline, but the name is misleading and should be clarified in the main text.","section":"§A.1.3"},{"comment":"The single case study where a BIPro poem receives a higher score than its human counterpart (6.70 vs 6.20) is not evidence for the general claim of narrowing the gap; this is only an anecdote. The aggregate results in Table 1 are the relevant evidence, and the case-study paragraph should not be phrased as if it supports the overall conclusion.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The author is affiliated with Zhipu.AI, the company behind GLM-4, one of the comparison systems; this potential conflict of interest is not discussed in the manuscript. The open-sourced code and review data are a significant strength and materially help reproducibility. Please also ensure the statistical-significance request is treated as a substantive requirement rather than a stylistic suggestion, since the margin over Shisanbai is small relative to the reported variance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: BIPro is a real and reasonably new method — use a block generative model to score inverse prompts directly by masking, then iteratively revise and rewrite the poem. The parallel-challenge result is genuinely impressive: same GLM-10B base, direct generation scores 4.65 overall, BIPro scores 5.54, closing more than half the gap to the human Daily Poem average of 6.37. That is a large, believable effect. Credit where due: the paper is honest about its limitations, ships code and data, and the core idea of avoiding fragile natural-language inverse transformations is a clean extension of inverse prompting.\n\nThe soft spots are concentrated in the evaluation, not the method. The open-domain claim that BIPro beats the best domain-specific systems rests on a 5.27 vs 5.13 overall margin over Shisanbai (AR 5.22 vs 5.16). With reported SDs around 1.0–1.2 and 10 reviewers, the standard error of that difference is roughly 0.1, so the margin is not distinguishable from noise. No significance test, no paired per-title analysis, no inter-annotator agreement. That is load-bearing because the abstract says BIPro \"outperform[s]\" the domain-specific systems. Second, format constraints are checked asymmetrically: BIPro must satisfy all eight Pingshui constraints, while GPT-4, GLM-4, and Baidu are checked only on the first two. That makes the format and overall comparisons unfair in a way that could go either direction, but it is uncontrolled. Third, the scorer is the same GLM-10B model doing the generating, so the rewrite loop may be optimizing the model's own perplexity rather than human-perceived quality. The large parallel gain says the overall framework works, but the paper does not isolate how much the scorer contributes versus the beam search and multi-draft structure. Minor: the abstract says \"proficient poets\" while the body says \"amateur poets\"; the memorization check is applied to direct GLM-10B but not to BIPro, even though the same base model could copy famous poems.\n\nWho is this for? Anyone working on constrained generation, especially with weaker models and iterative revision. It deserves a serious referee: the idea is worth engaging with and the parallel-challenge result is solid enough to justify the evaluation burden. A reviewer should push for significance testing, symmetric constraint checks, a memorization check, and an ablation that removes the self-scorer. If those land, this becomes a useful paper.","headline":"BIPro is a legitimate new constrained-generation idea with a strong same-base-model result, but the open-domain 'outperform' claim rests on a 0.14-point margin without a significance test and on uneven constraint checking.","tokens_in":14696,"tokens_out":2458,"would_cite":false,"duration_ms":30397,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BIPro uses revise-and-rewrite loops to make the weaker GLM-10B-Chinese model write traditional-form Chinese poems that human poets rate above GPT-4 and domain-specific systems.","keywords":["Chinese poetry generation","constrained text generation","block generative models","inverse prompting","zero-shot generation","poem revision","human evaluation","Pingshui rhyme scheme"],"falsifier":"A concrete falsifier: from the open-sourced review data, compute for each accepted revision the change in BIPro score and the change in human overall score; if these changes are not positively correlated across the hundreds of revisions, the selection rule described in Algorithm 1 is not carrying the reported quality gain.","tokens_in":13637,"feed_emoji":"📜","tokens_out":9889,"duration_ms":92200,"temperature":0.7,"pith_summary":"The paper tries to show that constrained writing—producing text that must obey strict form rules—does not require the largest or most specialized language model if the model can generate text in the middle of a passage and is given a way to revise itself. BIPro wraps the block generative model GLM-10B-Chinese in two loops: after each new line is written, the previous line is masked and regenerated, and after the whole poem exists, every line is rewritten in rounds; candidates are kept only when the model's own perplexity score improves. In human evaluations by proficient amateur poets, BIPro's zero-shot poems scored 5.27 overall (5.22 by answer ranking) on 42 open-domain titles, ahead of GPT-4, GLM-4, Baidu Poetry Helper, Yusheng, and Shisanbai, and 5.54/5.43 on 87 parallel titles, closing part of the gap to human Daily Poem poems at 6.37/6.42. If the result holds, it would mean an untapped potential of block generative models for constrained generation tasks beyond poetry.","feed_headline":"Weaker model beats GPT-4 at Chinese poetry via self-revision","feed_subtitle":"Human poets rank its zero-shot poems above GPT-4, GLM-4, and specialist poetry systems.","key_machinery":"The load-bearing mechanism is the block generative model's ability to infill masked spans, combined with the BIPro scorer. For a candidate replacement, the input is rearranged into a BIPro prompt (title, surrounding lines, blank position) and the target line is scored by its perplexity under GLM-10B-Chinese; lower perplexity means the line fits the context better, and Algorithm 1 accepts a revision only when this score improves. The generation phase uses beam search with a Pingshui format verifier that checks eight structural constraints (sentence count, line length, tone categories, rhyme, and internal tone patterns), replacing beams that violate the constraints with good beams from other candidates. The revise step then masks the previous line and regenerates it after the next line exists, while the rewrite step cycles over all lines for up to 20 rounds, so the poem evolves the way a human draft does.","core_discovery":"The central discovery is that the bottleneck in constrained generation is not model scale or domain training but the lack of a revision mechanism. BIPro directly computes the inverse-prompting score that earlier inverse prompting could only approximate: instead of transforming the prompt into an inverse natural-language form, it masks the target text and lets the block generative model score the candidate by its perplexity against the surrounding context and the title. This makes a search-and-evaluate loop possible: beam search with a Pingshui format verifier generates constraint-satisfying candidate lines, revise improves a line immediately after its successor is written, and rewrite cycles through all lines after the poem is complete. Using GLM-10B-Chinese with no fine-tuning and zero-shot prompts, the framework turns a model that scores 19.7 percent on a Chinese common-knowledge benchmark into a poem generator that human poets rank above much stronger direct generators and above systems trained specifically on classical poetry; in the parallel challenge it even outscored the human reference poem in one case (6.70/7.25 vs 6.20/6.00).","pith_inferences":["Beyond the paper, the BIPro scoring signal could be tested as a general text-quality metric: if perplexity under a mask-conditioned block model correlates with human judgments beyond poetry, the same loop could revise prose, lyrics, or constrained dialogue.","An extension the paper does not run is to use a stronger block generative model than GLM-10B-Chinese; the limitations section itself suggests that with better block models the method may surpass human poems, so a direct scaling study would be a natural next test.","The evaluation used a small pool of amateur poets with monetary incentives tied to predicting others' scores; a larger pre-registered study with professional poets would clarify whether the advantage over GPT-4 persists outside this reviewer population.","Because the scorer weight can be adjusted (title score vs neighboring-line score), a practical extension is to expose a user-controllable weight to shift poems toward relevance or toward formal and aesthetic quality, something the paper does not explore."],"forward_implications":["The paper's conclusion is that block generative models have untapped potential for constrained generation: the same GLM-10B base model moves from 4.65/4.37 (direct) to 5.54/5.43 (BIPro) in the parallel challenge.","BIPro's open-domain poems score highest overall and by answer ranking among the six systems, even though each detailed sub-score is not always the best; the gain comes from balancing format, relevance, informativeness, and aesthetics.","The gap to short-listed human poems narrows from 1.72 overall points for GPT-4 to 0.83 points for BIPro in the parallel challenge.","The framework is zero-shot: no priming examples and no additional training are used, so the revise-and-rewrite loop itself, not domain-specific learning, is what carries the improvement."],"supporting_citations":[{"why":"Introduces inverse prompting, the scoring idea BIPro generalizes to block generative models by masking the prompt instead of transforming it.","marker":"Zou et al., 2021"},{"why":"Defines GLM, the block generative architecture whose middle-text generation BIPro builds on and whose Chinese 10B version is the base model.","marker":"Du et al., 2022"},{"why":"Reports GPT-4, the leading direct generative baseline that BIPro is compared against in both challenges.","marker":"Achiam et al., 2023"},{"why":"Describes Yusheng, a domain-specific traditional-poem system trained on over one million poems, used as a strong baseline.","marker":"Ma et al., 2023"},{"why":"Gives the Answer Ranking method that turns reviewers' self-predictions into the AR scores reported alongside overall scores.","marker":"Kong et al., 2022"},{"why":"Documents the Pingshui rhyme-and-tone system whose eight rules define the format constraint that the verifier enforces.","marker":"Nie, 1982"}],"fun_headline_variants":["Zero-shot Chinese poetry: weaker model beats GPT-4 via self-revision","Self-revision powers a 10B model past GPT-4 in Chinese poem generation","Block inverse prompting turns a mid-size model into a top poet","Human poets rank zero-shot poems from a small model above GPT-4","Why a weaker model outshines GPT-4 at Chinese poetry: it revises"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model's own uncertainty score for a candidate line (lower perplexity under the BIPro prompt) is a reliable guide to poetic quality, since the loop only keeps revisions that improve that score; if it does not track what human poets value, the reported gains would not be explained by the method as described.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot Chinese poetry: weaker model beats GPT-4 via self-revision","Self-revision powers a 10B model past GPT-4 in Chinese poem generation","Block inverse prompting turns a mid-size model into a top poet","Human poets rank zero-shot poems from a small model above GPT-4","Why a weaker model outshines GPT-4 at Chinese poetry: it revises"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1406,"prompt_tokens":970,"completion_tokens":436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":349}},"tokens_in":586,"tokens_out":436,"duration_ms":5006,"temperature":1.0,"reasoning_tokens":349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:40:07.996277+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier: from the open-sourced review data, compute for each accepted revision the change in BIPro score and the change in human overall score; if these changes are not positively correlated across the hundreds of revisions, the selection rule described in Algorithm 1 is not carrying the reported quality gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the Pingshui rhyme-and-tone system whose eight rules define the format constraint that the verifier enforces."}],"review_version":1}