pith. machine review for the scientific record. sign in

arxiv: 2604.14162 · v1 · submitted 2026-03-23 · 💻 cs.CL · cs.LG

Recognition: 2 theorem links

· Lean Theorem

Decoupling Scores and Text: The Politeness Principle in Peer Review

Yingxuan Wen
Authors on Pith no claims yet

Pith reviewed 2026-05-15 00:58 UTC · model grok-4.3

classification 💻 cs.CL cs.LG
keywords peer reviewacceptance predictionpoliteness principlescore modelstext modelssentiment analysisICLR submissionsreview reliability
0
0 comments X

The pith

Numerical scores predict ICLR acceptance at 91 percent accuracy while review text reaches only 81 percent because politeness masks the rejection signal.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a dataset of more than 30,000 ICLR submissions from 2021 to 2025 and tests whether numerical review scores or the written review text better forecast final acceptance decisions. Score-only models reach 91 percent accuracy, but text-only models top out at 81 percent even when powered by large language models. The gap occurs because reviews of rejected papers contain more positive than negative sentiment words, following the politeness principle that softens criticism. Score distributions in the 9 percent of cases where scores fail show high kurtosis and negative skew, indicating that one or two low scores often decide rejection even when the average sits near the threshold. This decoupling shows authors why they cannot reliably read outcomes from text comments alone.

Core claim

Models trained on numerical scores alone predict acceptance with 91 percent accuracy across the ICLR dataset, while models using only review text achieve 81 percent accuracy at best. Cases where score models err exhibit score distributions with high kurtosis and negative skewness, revealing that individual low scores override average scores to drive rejection. Text models lag because rejected submissions still receive reviews with a net positive sentiment balance, a pattern the authors label the politeness principle that conceals the decisive negative judgment from authors.

What carries the argument

The politeness principle, the tendency for review text to contain more positive than negative sentiment words even for rejected papers, which decouples the written feedback from the numerical scores that carry the rejection signal.

If this is right

  • Individual low scores decide rejection even when the average score is borderline.
  • Review text alone supplies little reliable information about acceptance outcomes.
  • Large language models do not close the performance gap when processing review text.
  • The 9 percent of score-model failures trace to distinctive high-kurtosis score distributions.
  • Sentiment counts in rejection reviews show consistent positive bias across the dataset.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Review platforms could reduce author confusion by requiring separate entry fields for scores and open text.
  • The same politeness masking may occur in grant reviews or hiring letters, suggesting a broader pattern worth testing.
  • Authors could be advised to treat numerical scores as the primary signal and ignore sentiment wording.
  • Replicating the analysis on post-2025 ICLR data would show whether the politeness pattern has changed.

Load-bearing premise

The constructed dataset of ICLR 2021-2025 submissions accurately represents the full review process without selection bias or missing metadata that could affect acceptance labels.

What would settle it

A new dataset from another conference where text-only models match or exceed score-only accuracy after controlling for review length and reviewer identity.

Figures

Figures reproduced from arXiv: 2604.14162 by Yingxuan Wen.

Figure 1
Figure 1. Figure 1: Overview of the research framework. It integrates large-scale dataset construction, multi [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The count and rate of accepted papers combined from 2021 to 2025. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prediction accuracy across different rating, arranged in ascending order of average score. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Statistical profile of review ratings across three sample categories (Hard Samples, Simple [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average sentiment score for six high-level [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of sentiments ratios across each aspect, illustrating that positive sentiment dominates nearly all aspects, even for rejected pa￾pers (Hard/Simple Reject). whereas text-based models stagnate at 81%. Our diagnostic analysis attributes this gap to a structural contradiction between the two modalities. Statistically, the Hard Samples are defined by negative skewness and high kurto￾sis, representing… view at source ↗
read the original abstract

Authors often struggle to interpret peer review feedback, deriving false hope from polite comments or feeling confused by specific low scores. To investigate this, we construct a dataset of over 30,000 ICLR 2021-2025 submissions and compare acceptance prediction performance using numerical scores versus text reviews. Our experiments reveal a significant performance gap: score-based models achieve 91% accuracy, while text-based models reach only 81% even with large language models, indicating that textual information is considerably less reliable. To explain this phenomenon, we first analyze the 9% of samples that score-based models fail to predict, finding their score distributions exhibit high kurtosis and negative skewness, which suggests that individual low scores play a decisive role in rejection even when the average score falls near the borderline. We then examine why text-based accuracy significantly lags behind scores from a review sentiment perspective, revealing the prevalence of the Politeness Principle: reviews of rejected papers still contain more positive than negative sentiment words, masking the true rejection signal and making it difficult for authors to judge outcomes from text alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper constructs a dataset of over 30,000 ICLR 2021-2025 submissions and compares acceptance prediction using numerical review scores versus full review text. It reports that score-based models achieve 91% accuracy while text-based models (including LLMs) reach only 81%, attributing the gap to high kurtosis and negative skewness in mispredicted score distributions and to the Politeness Principle, whereby rejected papers' reviews contain more positive than negative sentiment words, masking rejection signals.

Significance. If the empirical gap and explanations hold after addressing data-construction details, the work offers a large-scale demonstration that numerical scores are substantially more predictive of acceptance than textual content in peer review, with direct implications for review guidelines and author interpretation of feedback. The scale of the dataset and direct comparison against LLMs constitute clear strengths.

major comments (2)
  1. [§3] §3 (Dataset Construction): No information is provided on scraping method, inclusion/exclusion rules, handling of incomplete reviews, withdrawn submissions, or validation of acceptance labels against official ICLR decisions. This is load-bearing for the central 91%-vs-81% claim and the kurtosis analysis, as selection bias or label noise could artifactually produce the observed gap.
  2. [§4] §4 (Experiments): The manuscript reports accuracy numbers but supplies no details on model architectures, training procedures, feature extraction for text models, cross-validation strategy, or how acceptance labels were obtained. Without these, the possibility of data leakage or post-hoc threshold choices cannot be ruled out.
minor comments (2)
  1. [Abstract and §5.2] The abstract and §5.2 refer to 'high kurtosis and negative skewness' without reporting the numerical values or the exact statistical test used; adding these would improve reproducibility.
  2. [Figures 4-6] Figure captions and axis labels in the sentiment-analysis plots should explicitly state the sentiment lexicon or model employed.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We agree that additional details on dataset construction and experimental procedures are necessary for reproducibility and to fully support our claims. We will revise the manuscript to address these points.

read point-by-point responses

  1. Referee: [§3] §3 (Dataset Construction): No information is provided on scraping method, inclusion/exclusion rules, handling of incomplete reviews, withdrawn submissions, or validation of acceptance labels against official ICLR decisions. This is load-bearing for the central 91%-vs-81% claim and the kurtosis analysis, as selection bias or label noise could artifactually produce the observed gap.

    Authors: We appreciate this observation. The current manuscript indeed lacks these specifics, which we will add in the revision. We scraped the data using the official OpenReview API for ICLR conferences 2021-2025. Inclusion criteria required submissions with at least one complete review containing both numerical scores and text; incomplete reviews and withdrawn submissions were excluded. Acceptance labels were validated by matching against the official ICLR acceptance lists published on the conference website. We will include a new subsection detailing these steps, along with statistics on excluded items to address potential bias concerns. revision: yes

  2. Referee: [§4] §4 (Experiments): The manuscript reports accuracy numbers but supplies no details on model architectures, training procedures, feature extraction for text models, cross-validation strategy, or how acceptance labels were obtained. Without these, the possibility of data leakage or post-hoc threshold choices cannot be ruled out.

    Authors: We agree that these details are essential. In the revised Section 4, we will specify: score-based models use logistic regression and random forests on the average and individual scores; text-based models include TF-IDF with logistic regression, fine-tuned BERT, and zero-shot GPT-4 prompting. Training used 5-fold stratified cross-validation with fixed random seeds. Acceptance labels were directly from ICLR metadata, with no post-hoc thresholding—predictions used a 0.5 probability threshold on the trained models. We will also release the code and data splits to eliminate leakage concerns. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical comparison against external acceptance labels

full rationale

The paper's core result is an empirical measurement: models trained on numerical review scores achieve 91% accuracy predicting ICLR acceptance, while text-based models reach only 81%. Acceptance labels are external ICLR decisions, not derived from the fitted models or from the same textual features being evaluated. No equations, self-citations, ansatzes, or uniqueness theorems are invoked to justify the performance gap; the kurtosis and sentiment analyses are post-hoc observations on the same dataset rather than load-bearing derivations that reduce to the inputs by construction. The work is therefore self-contained against external benchmarks and receives the default non-circularity finding.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The claim rests on standard supervised learning assumptions plus the untested premise that sentiment word counts reliably capture politeness masking without domain-specific validation.

free parameters (1)
  • model hyperparameters and feature thresholds
    Accuracy numbers depend on choices of classifiers, text embeddings, and sentiment lexicons that are not specified in the abstract.
axioms (1)
  • domain assumption Acceptance decisions are determined primarily by the numerical scores rather than unmeasured factors
    The 91% figure treats scores as near-ground-truth predictors without quantifying residual variance from area chairs or other signals.

pith-pipeline@v0.9.0 · 5478 in / 1278 out tokens · 34127 ms · 2026-05-15T00:58:33.201649+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

    work page

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

    work page

  3. [3]

    Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. https://doi.org/10.18653/v1/D19-1371 S ci BERT : A pretrained language model for scientific text . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615--3620, Hong Kong, China...

    work page doi:10.18653/v1/d19-1371 2019

  4. [4]

    Prabhat Kumar Bharti, Meith Navlakha, Mayank Agarwal, and Asif Ekbal. 2023. https://doi.org/10.1007/s10579-023-09662-3 Politepeer: does peer review hurt? a dataset to gauge politeness intensity in the peer reviews . 58(4):1291–1313

    work page doi:10.1007/s10579-023-09662-3 2023

  5. [5]

    Penelope Brown. 1987. Politeness: Some universals in language usage, volume 4. Cambridge university press

    work page 1987

  6. [6]

    Souvic Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020. https://doi.org/10.1145/3383583.3398541 Aspect-based sentiment analysis of scientific reviews . In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, JCDL '20, page 207–216, New York, NY, USA. Association for Computing Machinery

    work page doi:10.1145/3383583.3398541 2020

  7. [7]

    Zhongfen Deng, Hao Peng, Congying Xia, Jianxin Li, Lifang He, and Philip Yu. 2020. https://doi.org/10.18653/v1/2020.coling-main.555 Hierarchical bi-directional self-attention networks for paper review rating recommendation . In Proceedings of the 28th International Conference on Computational Linguistics, pages 6302--6314, Barcelona, Spain (Online). Inter...

    work page doi:10.18653/v1/2020.coling-main.555 2020

  8. [8]

    Gustavo L\' u cius Fernandes and Pedro O. S. Vaz-de Melo. 2022. https://doi.org/10.1145/3529372.3530935 Between acceptance and rejection: challenges for an automatic peer review process . New York, NY, USA. Association for Computing Machinery

    work page doi:10.1145/3529372.3530935 2022

  9. [9]

    Gustavo L \'u cius Fernandes and Pedro OS Vaz-de Melo. 2024. Enhancing the examination of obstacles in an automated peer review system. International Journal on Digital Libraries, 25(2):341--364

    work page 2024

  10. [10]

    Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019. https://doi.org/10.18653/v1/N19-1129 Does my rebuttal matter? insights from a major NLP conference . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pap...

    work page doi:10.18653/v1/n19-1129 2019

  11. [11]

    Tirthankar Ghosal, Rajeev Verma, Asif Ekbal, and Pushpak Bhattacharyya. 2019. https://doi.org/10.18653/v1/P19-1106 D eep S enti P eer: Harnessing sentiment in review texts to recommend peer review decisions . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1120--1130, Florence, Italy. Association for Compu...

    work page doi:10.18653/v1/p19-1106 2019

  12. [12]

    Xinyu Hua, Zhe Hu, and Lu Wang. 2019. https://doi.org/10.18653/v1/P19-1255 Argument generation with retrieval, planning, and realization . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2661--2672, Florence, Italy. Association for Computational Linguistics

    work page doi:10.18653/v1/p19-1255 2019

  13. [13]

    Junjie Huang, Win bin Huang, Yi Bu, Qi Cao, Huawei Shen, and Xueqi Cheng. 2023. https://doi.org/10.1016/j.joi.2023.101427 What makes a successful rebuttal in computer science conferences?: A perspective on social interaction . Journal of Informetrics, 17(3):101427

    work page doi:10.1016/j.joi.2023.101427 2023

  14. [14]

    Maximilian Idahl and Zahra Ahmadi. 2025. https://doi.org/10.18653/v1/2025.naacl-demo.44 O pen R eviewer: A specialized large language model for generating critical scientific paper reviews . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Dem...

    work page doi:10.18653/v1/2025.naacl-demo.44 2025

  15. [15]

    Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. https://doi.org/10.18653/v1/N18-1149 A dataset of peer reviews ( P eer R ead): Collection, insights and NLP applications . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Ling...

    work page doi:10.18653/v1/n18-1149 2018

  16. [16]

    Amir Hossein Kargaran, Nafiseh Nikeghbal, Jing Yang, and Nedjma Ousidhoum. 2025. Insights from the iclr peer review and rebuttal process. arXiv preprint arXiv:2511.15462

    work page arXiv 2025

  17. [17]

    Youfang Leng, Li Yu, and Jie Xiong. 2019. https://doi.org/10.1145/3340555.3353766 Deepreviewer: Collaborative grammar and innovation neural network for automatic paper review . In 2019 International Conference on Multimodal Interaction, ICMI '19, page 395–403, New York, NY, USA. Association for Computing Machinery

    work page doi:10.1145/3340555.3353766 2019

  18. [18]

    Rui Li, Jia-Chen Gu, Po-Nien Kung, Heming Xia, Xiangwen Kong, Zhifang Sui, Nanyun Peng, and 1 others. 2025. Llm-reval: Can we trust llm reviewers yet? arXiv preprint arXiv:2510.12367

    work page arXiv 2025

  19. [19]

    Kai Lu, Shixiong Xu, Jinqiu Li, Kun Ding, and Gaofeng Meng. 2025. https://openreview.net/forum?id=s7HUJamWqX Agent reviewers: Domain-specific multimodal agents with shared memory for paper review . In Forty-second International Conference on Machine Learning

    work page 2025

  20. [20]

    Minghui Meng, Ruxue Han, Jiangtao Zhong, Haomin Zhou, and Chengzhi Zhang. 2023. https://doi.org/10.59494/dsi.2023.1.4 Aspect-based sentiment analysis of online peer reviews and prediction of paper acceptance results . Data Science and Informetrics, 81 1 1:0

    work page doi:10.59494/dsi.2023.1.4 2023

  21. [21]

    Ana Carolina Ribeiro, Amanda Sizo, Henrique Lopes Cardoso, and Lu\' s Paulo Reis. 2021. https://doi.org/10.1007/978-3-030-86230-5_60 Acceptance decision prediction in peer-review through sentiment analysis . In Progress in Artificial Intelligence: 20th EPIA Conference on Artificial Intelligence, EPIA 2021, Virtual Event, September 7–9, 2021, Proceedings, ...

    work page doi:10.1007/978-3-030-86230-5_60 2021

  22. [22]

    Ke Wang and Xiaojun Wan. 2018. https://doi.org/10.1145/3209978.3210056 Sentiment analysis of peer review texts for scholarly papers . In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '18, page 175–184, New York, NY, USA. Association for Computing Machinery

    work page doi:10.1145/3209978.3210056 2018

  23. [23]

    Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. 2020. https://doi.org/10.18653/v1/2020.inlg-1.44 R eview R obot: Explainable paper review generation based on knowledge synthesis . In Proceedings of the 13th International Conference on Natural Language Generation, pages 384--397, Dublin, Ireland. Association for Computa...

    work page doi:10.18653/v1/2020.inlg-1.44 2020

  24. [24]

    Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. https://aclanthology.org/2024.lrec-main.816/ Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340--9351, Torino, Ital...

    work page 2024

  25. [25]

    Changjia Zhu, Junjie Xiong, Renkai Ma, Zhicong Lu, Yao Liu, and Lingyao Li. 2025. https://api.semanticscholar.org/CorpusID:281310346 When your reviewer is an llm: Biases, divergence, and prompt injection risks in peer review . ArXiv, abs/2509.09912

    work page arXiv 2025

  26. [26]

    Zhenzhen Zhuang, Jiandong Chen, Hongfeng Xu, Yuwen Jiang, and Jialiang Lin. 2025. https://doi.org/10.1016/j.inffus.2025.103332 Large language models for automated scholarly paper review: A survey . Information Fusion, 124:103332

    work page doi:10.1016/j.inffus.2025.103332 2025