REVIEW 4 major objections 4 minor 30 references
BIPro: Zero-shot Chinese Poem Generation via Block Inverse Prompting Constrained Generation Framework
T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read BIPro uses revise-and-rewrite loops to make the weaker GLM-10B-Chinese model write traditional-form Chinese poems that human poets rate above GPT-4 and domain-specific systems.
desk verdict BIPro is a legitimate new constrained-generation idea with a strong same-base-model result, but the open-domain 'outperform' claim rests on a 0.14-point margin without a significance test and on uneven constraint checking. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the block generative model's ability to infill masked spans, combined with the BIPro scorer. For a candidate replacement, the input is rearranged into a BIPro prompt (title, surrounding lines, blank position) and the target line is scored by its perplexity under GLM-10B-Chinese; lower perplexity means the line fits the context better, and Algorithm 1 accepts a revision only when this score improves. The generation phase uses beam search with a Pingshui format verifier that checks eight structural constraints (sentence count, line length, tone categories, rhyme, and internal tone patterns), replacing beams that violate the constraints with good beams from other candidates. The revise step then masks the previous line and regenerates it after the next line exists, while the rewrite step cycles over all lines for up to 20 rounds, so the poem evolves the way a human draft does.
What would settle it
A concrete falsifier: from the open-sourced review data, compute for each accepted revision the change in BIPro score and the change in human overall score; if these changes are not positively correlated across the hundreds of revisions, the selection rule described in Algorithm 1 is not carrying the reported quality gain.
Extended reading notes
Core claim
The central discovery is that the bottleneck in constrained generation is not model scale or domain training but the lack of a revision mechanism. BIPro directly computes the inverse-prompting score that earlier inverse prompting could only approximate: instead of transforming the prompt into an inverse natural-language form, it masks the target text and lets the block generative model score the candidate by its perplexity against the surrounding context and the title. This makes a search-and-evaluate loop possible: beam search with a Pingshui format verifier generates constraint-satisfying candidate lines, revise improves a line immediately after its successor is written, and rewrite cycles through all lines after the poem is complete. Using GLM-10B-Chinese with no fine-tuning and zero-shot prompts, the framework turns a model that scores 19.7 percent on a Chinese common-knowledge benchmark into a poem generator that human poets rank above much stronger direct generators and above systems trained specifically on classical poetry; in the parallel challenge it even outscored the human reference poem in one case (6.70/7.25 vs 6.20/6.00).
Load-bearing premise
The load-bearing premise is that the model's own uncertainty score for a candidate line (lower perplexity under the BIPro prompt) is a reliable guide to poetic quality, since the loop only keeps revisions that improve that score; if it does not track what human poets value, the reported gains would not be explained by the method as described.
Editorial extensions
If this is right
- The paper's conclusion is that block generative models have untapped potential for constrained generation: the same GLM-10B base model moves from 4.65/4.37 (direct) to 5.54/5.43 (BIPro) in the parallel challenge.
- BIPro's open-domain poems score highest overall and by answer ranking among the six systems, even though each detailed sub-score is not always the best; the gain comes from balancing format, relevance, informativeness, and aesthetics.
- The gap to short-listed human poems narrows from 1.72 overall points for GPT-4 to 0.83 points for BIPro in the parallel challenge.
- The framework is zero-shot: no priming examples and no additional training are used, so the revise-and-rewrite loop itself, not domain-specific learning, is what carries the improvement.
Reading between the lines
- Beyond the paper, the BIPro scoring signal could be tested as a general text-quality metric: if perplexity under a mask-conditioned block model correlates with human judgments beyond poetry, the same loop could revise prose, lyrics, or constrained dialogue.
- An extension the paper does not run is to use a stronger block generative model than GLM-10B-Chinese; the limitations section itself suggests that with better block models the method may surpass human poems, so a direct scaling study would be a natural next test.
- The evaluation used a small pool of amateur poets with monetary incentives tied to predicting others' scores; a larger pre-registered study with professional poets would clarify whether the advantage over GPT-4 persists outside this reviewer population.
- Because the scorer weight can be adjusted (title score vs neighboring-line score), a practical extension is to expose a user-controllable weight to shift poems toward relevance or toward formal and aesthetic quality, something the paper does not explore.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BIPro, a zero-shot framework for constrained generation of traditional-form Chinese poems. The method uses a block generative model (GLM-10B-Chinese) and two inverse-prompting operations, revise and rewrite, both scored by the same model's perplexity under a BIPro prompt, to iteratively refine a poem. Two human evaluations are reported: an open-domain challenge comparing BIPro with GPT-4, GLM-4, Baidu Poetry Helper, Yusheng, and Shisanbai, and a parallel-title challenge comparing BIPro with direct GLM-10B generation, GPT-4, and human poems from the Daily Poem section. The authors claim that BIPro achieves the highest human scores in the open-domain task and narrows the gap to human-authored poems in the parallel task.
Significance. The core idea is interesting and potentially useful: exploiting a block model's ability to fill masked spans to revise and rewrite constrained text is a novel twist on inverse prompting. The paper ships code and review data, uses two human evaluations in a difficult literary domain, and reports a large improvement over its own direct-generation baseline (overall 4.65→5.54 in the parallel challenge, Table 1). The evaluation design is thoughtful (anonymous presentation, random ordering, Answer Ranking scores). However, the central claim that BIPro outperforms all baselines is not yet statistically supported, and several methodological asymmetries need to be addressed before the claims can be taken at face value. The skeptical concern about the absence of significance testing lands, and the self-scorer circularity is a real limitation that the paper does not currently resolve.
major comments (4)
- [§4.2, Tables 1 and 4] The headline claim that BIPro 'outperforms' Shisanbai in the open-domain challenge rests on an overall-score margin of 5.27 vs 5.13 (AR 5.22 vs 5.16), while Table 4 reports standard deviations around 1.0–1.2. No significance test, confidence interval, or paired per-title analysis is reported. With 42 titles and 10 reviewers, this margin is within plausible sampling noise; the ambiguity only grows if the reported SDs are across title-level means rather than across reviewer scores. I recommend a paired statistical test or bootstrap confidence interval on per-title means for both the open-domain and parallel challenges, and a corresponding softening of 'outperform' if the margin is not significant.
- [Appendix A.5] Format constraints are verified asymmetrically. BIPro generations are required to satisfy all eight Pingshui constraints, while GPT-4, GLM-4, and Baidu are checked only on the first two constraints and repeatedly prompted until they produce a poem that satisfies those two. This means the direct systems are not held to the same tonal and rhyming requirements, so their lower Format scores may partly reflect unmet constraints rather than intrinsic quality. An apples-to-apples comparison should either apply the full verifier to all systems (reporting the pass rate) or otherwise control for constraint satisfaction; this is load-bearing because Format is one of the four rated dimensions and the comparison with direct systems is central to the paper's claims.
- [§3.4, Algorithm 1] The revise and rewrite loop accepts a replacement only when s, the GLM-10B perplexity under the BIPro prompt, improves. Since the same model is both generator and scorer, the internal objective is not tied to any external standard. The paper provides no evidence that this self-perplexity correlates with human aesthetic judgments, and no ablation separates the contribution of the beam-based constraint generation, the revise operation, the rewrite operation, and the self-scorer. The large human-evaluated improvement over direct GLM-10B (Table 1) could in principle come mostly from the constrained beam search or from the iterative refinement rather than from the scorer as described. I would like to see an ablation that replaces the scorer with a random or prior-based selection rule, and a correlation analysis between s and human scores on the collected data.
- [§A.3, Table 4] The human evaluation is the only evidence for the central claims, yet the paper reports no reviewer-agreement statistic and does not clarify the unit of analysis in Table 4. With 10 reviewers and subjective aesthetic scores, reviewer disagreement could drive the variance. Please report inter-annotator agreement (e.g., ICC or Krippendorff's alpha) and state explicitly whether the values in Table 4 are standard deviations across reviewer scores or across title-level means. This also affects the appropriate significance test for the claims.
minor comments (4)
- [Algorithm 1] The notation p/p_{k-1} is used but never defined; I assume it denotes the poem with the k-1-th sentence removed, but this should be stated. The scorer signature is also inconsistent: s(p) appears alongside s(prompt, target), so the reader cannot tell what the second argument in s(p/p_{k-1}, p'_k-1) means until reading §3.4.
- [Table 4] The caption says 'variance' but the entries are shown as mean ± standard deviation. Please use consistent terminology and indicate which quantity is reported.
- [§A.1.3] The 'Direct Generation' baseline is not direct autoregressive generation: it uses the same beam-based constrained generation method as BIPro but without the BIPro scorer and without revise/rewrite. This is actually a useful ablation baseline, but the name is misleading and should be clarified in the main text.
- [§4.3] The single case study where a BIPro poem receives a higher score than its human counterpart (6.70 vs 6.20) is not evidence for the general claim of narrowing the gap; this is only an anecdote. The aggregate results in Table 1 are the relevant evidence, and the case-study paragraph should not be phrased as if it supports the overall conclusion.
Circularity Check
No circularity: the internal GLM-10B self-scorer is an optimization heuristic; the headline superiority claim rests on independent human ratings.
full rationale
The paper's claimed derivation is: generate candidate poem sentences with a block generative model, use a BIPro scorer based on the same model's perplexity to select among candidates during revise/rewrite, and then evaluate the final poems through human ratings by proficient poets. The internal scorer is the same model that generates text, so Algorithm 1's replacement criterion 's(...) > s(p)' guarantees improvement only in the model's own perplexity, not in human-perceived quality. However, the paper does not present the BIPro score as the evidence for its headline claim; the central claim that BIPro 'outperform[s] both most advanced direct generative systems ... and best domain-specific systems' is supported by the human-evaluation results in Table 1, which are external to the scoring loop. Thus no prediction reduces by construction to a fitted input or to the scorer's self-preference. The citation to Zou et al. (2021) is prior work by the same author, but it is used as background methodology for inverse prompting, not as the load-bearing evidence for the empirical superiority claim; that claim is independently grounded in human ratings. The small margin over Shisanbai and absence of significance testing is a statistical-support concern, not a circularity concern. No specific circular step can be identified from the paper's own equations or cited evidence.
Assumptions & free parameters
free parameters (4)
- Beam size k =
6
- Maximal rewrite rounds =
20
- Scorer weighting scheme =
unspecified weights
- Match sentence choice =
next for odd, previous for even
assumptions (6)
- domain assumption GLM-10B-Chinese has reliable block (infill) generation ability
- domain assumption Model perplexity is a valid quality scorer
- domain assumption The Pingshui verifier correctly encodes all relevant constraints
- domain assumption Human reviewers are able to judge poem quality
- domain assumption The collected titles are representative
- standard math Answer Ranking aggregation is valid
Cite this review
Pith. "Pith review of BIPro: Zero-shot Chinese Poem Generation via Block Inverse Prompting Constrained Generation Framework." pith.science (2026). https://pith.science/paper/DPPU2Y7K
@misc{pith2026241113237,
author = {Pith},
title = {Pith review of: BIPro: Zero-shot Chinese Poem Generation via Block Inverse Prompting Constrained Generation Framework},
year = {2026},
howpublished = {\url{https://pith.science/paper/DPPU2Y7K}},
note = {Machine review of arXiv:2411.13237}
}
read the original abstract
Recently, generative pre-trained models have made significant strides, particularly highlighted by the release of ChatGPT and GPT-4, which exhibit superior cross-domain capabilities. However, these models still face challenges on constrained writing tasks like poem generation under open-domain titles. In response to this challenge, we introduce Block Inverse Prompting (BIPro) constrained generation framework. BIPro leverages two block inverse prompting methods, revise and rewrite, that mimic the process of human text writing using block generative models. It significantly improves the zero-shot generation quality on the formidable constrained generation task of open-domain traditional-form Chinese poem generation. Based on a less powerful block generative model GLM-10B-Chinese, poems composed via BIPro without priming or additional training outperform both most advanced direct generative systems like GPT-4 or GLM-4 and best domain-specific systems such as Yusheng, Shisanbai, or Baidu Poetry Helper in human evaluation by proficient poets. Finally, BIPro considerably narrows the gap between AI-generated works and short-listed human literary arts in another human evaluation, unveiling the promising potential of block generative models in improving the quality of constrained generation.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
-
[2]
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M \'e rouane Debbah, \'E tienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. The falcon series of open language models. arXiv preprint arXiv:2311.16867
arXiv 2023
-
[3]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609
arXiv 2023
-
[4]
Baidu Research . 2023. http://research.baidu.com/Blog/index-view?id=183 Ernie bot: Baidu's knowledge-enhanced large language model built on full ai stack technology
work page 2023
-
[5]
Maria Teresa Baldassarre, Danilo Caivano, Berenice Fernandez Nieto, Domenico Gigante, and Azzurra Ragone. 2023. The social impact of generative ai: An analysis on chatgpt. In Proceedings of the 2023 ACM Conference on Information Technology for Social Good, pages 363--373
work page 2023
-
[6]
Zekun Deng, Hao Yang, and Jun Wang. 2024. Can ai write classical chinese poetry like humans? an empirical study inspired by turing test. arXiv preprint arXiv:2401.04952
arXiv 2024
-
[7]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
arXiv 2018
-
[8]
Duo Du, Yanling Zhang, and Jiao Ge. 2023. Effect of ai generated content advertising on consumer engagement. In International Conference on Human-Computer Interaction, pages 121--129. Springer
work page 2023
Show all 30 references
-
[9]
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: general language model pretraining with autoregressive blank infilling. pages 320--335
2022
-
[10]
Cristina Garbacea and Qiaozhu Mei. 2022. Why is constrained neural language generation particularly challenging? arXiv preprint arXiv:2206.05395
2022 arXiv
-
[11]
Yuqing Kong, Yunqi Li, Yubo Zhang, Zhihuan Huang, and Jinzhao Wu. 2022. Eliciting thinking hierarchy without a prior. Advances in Neural Information Processing Systems, 35:13329--13341
2022
-
[12]
Chuang Liu, Renren Jin, Yuqi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng, Shuting Zhang, Jianxiang Peng, Peiyi Zhang, Qingqing Lyu, et al. 2023. M3ke: A massive multi-level multi-subject knowledge evaluation benchmark for chinese large language models. arXiv preprint arXiv:2305.10263
2023 arXiv
-
[13]
Jingkun Ma, Runzhe Zhan, and Derek F Wong. 2023. Yu sheng: Human-in-loop classical chinese poetry generation system. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 57--66
2023
-
[14]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26
2013
-
[15]
Hongyin Nie. 1982. Pingshui. Knowledge of Literature and History, (1):97--100
1982
-
[16]
Nora I Parker and PL Newbigging. 1963. Magnitude and decrement of the m \"u ller-lyer illusion as a function of pre-training. Canadian Journal of Psychology/Revue canadienne de psychologie, 17(1):134
1963
-
[17]
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training
2018
-
[18]
Piotr Sawicki, Marek Grzes, Fabricio Goes, Dan Brown, Max Peeperkorn, and Aisha Khatun. 2023. Bits of grass: Does gpt already know how to write like whitman? arXiv preprint arXiv:2305.11064
2023 arXiv
-
[19]
Anthony Seow. 2002. The writing process and process writing. Methodology in language teaching: An anthology of current practice, pages 315--320
2002
-
[20]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805
2023 arXiv
-
[21]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971
2023 arXiv
-
[22]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288
2023 arXiv
-
[23]
Zhuohan Xie, Jey Han Lau, and Trevor Cohn. 2019. From shakespeare to li-bai: Adapting a sonnet model to chinese poetry. In Proceedings of the 17th Annual Workshop of the Australasian Language Technology Association, pages 10--18
2019
-
[24]
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305
2023 arXiv
-
[25]
Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, and Jie Tang. 2021. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. AI Open, 2:65--68
2021
-
[26]
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414
2022 arXiv
-
[27]
Guo Zhipeng, Xiaoyuan Yi, Maosong Sun, Wenhao Li, Cheng Yang, Jiannan Liang, Huimin Chen, Yuhui Zhang, and Ruoyu Li. 2019. Jiuge: A human-machine collaborative chinese classical poetry generation system. In Proceedings of the 57th annual meeting of the association for computat...
2019
-
[28]
Xu Zou, Da Yin, Qingyang Zhong, Hongxia Yang, Zhilin Yang, and Jie Tang. 2021. https://doi.org/10.1145/3447548.3467418 Controllable Generation from Pre-Trained Language Models via Inverse Prompting , page 2450–2460. Association for Computing Machinery, New York, NY, USA
2021
-
[29]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[30]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.