REVIEW 4 major objections 5 minor 2 cited by
Human preference over LLM rationales is driven by correctness, plausibility, and completeness.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 17:11 UTC pith:4S5JITTJ
load-bearing objection Useful diagnostic framing for rationale evaluation, but the LLM-judged attribute scores are a load-bearing liability that the paper's own appendix exposes. the 4 major comments →
Rethinking Human Preference Evaluation of LLM Rationales
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that the coarse signal of human preference over rationales decomposes into measurable attributes, and that three of them—correctness, plausibility, and completeness—consistently dominate across datasets and judge models. On that basis, the authors argue that attribute-specific ELO rankings, computed from scores assigned by LLM judges, are a more informative evaluation than conventional binary-preference ELO. The resulting rankings largely agree with overall human preferences but reveal unexpected per-attribute weaknesses, such as Claude-v1's high repetition and GPT-4 falling behind GPT-3.5-Turbo on arithmetic accuracy and self-consistency.
What carries the argument
The load-bearing machinery has two parts. First, a taxonomy of 12 rationale attributes (faithfulness, hallucination, repetition, informativeness, plausibility, self-consistency, source consistency, grammar, arithmetic accuracy, conciseness, completeness, correctness), scored by automated metrics, LLM judges, and human annotators. Second, a statistical bridge: SHAP values on a LightGBM model trained to predict human preference from the attribute scores identify which attributes matter, and attribute-specific ELO ratings turn those scores into per-dimension model rankings. The SHAP-LightGBM step is what turns 'preference' into an explainable function of attributes.
Load-bearing premise
That the LLM judges' attribute scores are accurate and unbiased measures of rationale quality; if those scores are systematically biased (e.g., rewarding confident or verbose text), the SHAP importance ordering and attribute-ELO rankings inherit that bias.
What would settle it
Have a fresh set of independent human annotators score the same rationales on the twelve attributes, recompute the SHAP ordering and attribute-specific ELO from human scores only, and compare with the LLM-judge results. If the top three attributes change or the per-attribute model rankings flip, the paper's conclusions are an artifact of judge bias rather than a property of human preference.
If this is right
- Attribute-level scores give a decomposition of 'preference' that can guide targeted model improvement, e.g., penalizing repetition or boosting arithmetic accuracy.
- Evaluators can focus limited annotation effort on the three most predictive attributes—correctness, plausibility, completeness—followed by informativeness and conciseness.
- Attribute-specific ELO rankings can be used to compare models on dimensions that a single ELO number masks, giving model developers diagnostic feedback.
- Using multiple LLM judges and reporting their outputs transparently mitigates judge bias in fine-grained evaluation.
Where Pith is reading between the lines
- If the attribute-importance result generalizes, preference data used for RLHF could be weighted by attributes rather than binary outcomes, potentially aligning models with what humans actually value.
- The same SHAP-plus-ELO pipeline could be applied to other free-text generation tasks—e.g., planning steps, code explanations, or commonsense answers—to check whether the same three attributes dominate, or whether the importance ordering shifts by domain.
- The finding that GPT-3.5-Turbo outperforms GPT-4 on arithmetic accuracy and self-consistency suggests that newer, larger models are not uniformly better on all rationale dimensions; model selection for interpretability tasks should be attribute-aware, not just win-rate-aware.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a fine-grained, attribute-based framework for evaluating LLM-generated rationales. It defines 12 rationale attributes, measures them with ROSCOE heuristics, three LLM judges (GPT-4o, Gemini 2.5-Flash, OLMo 2-32b), and author-provided human annotations, then uses SHAP on a LightGBM model to identify which attributes predict human preference in MT-Bench and Chatbot Arena. It also computes attribute-specific ELO ratings to compare models on individual rationale qualities. The headline findings are that Correctness, Plausibility, and Completeness are top predictors of human preference across datasets and judges, and that attribute-specific ELO reveals, e.g., GPT-3.5-Turbo surpassing GPT-4 on Arithmetic Accuracy and Self-Consistency.
Significance. If the central claim holds, the paper makes a useful contribution: it moves beyond opaque binary preference judgments toward interpretable, attribute-level evaluation, and it offers practical guidance on which attributes to prioritize. The paper has several strengths: it uses multiple LLM judges including an open-source model, provides full prompt templates in the appendix, applies SHAP to capture nonlinear feature interactions, and candidly acknowledges limitations in Section 5.1. The proposed attribute-ELO framework is a sensible way to make model comparisons more diagnostic. However, the significance is conditional on the validity of the LLM-judged attribute scores, and the paper's own evidence (Appendix A.7) shows that these scores can be unreliable. The human validation currently available is too limited to resolve this concern.
major comments (4)
- [Section 4.1, Figure 1; Section 5.1; Appendix A.7] The central Q2 finding that Correctness, Plausibility, and Completeness are top predictors of human preference is derived entirely from SHAP analysis in which the features are LLM-judged attribute scores. The paper's own Appendix A.7 shows GPT-4o assigning a perfect Correctness score of 1.0 to a rationale with an algebraic error. If LLM judges share systematic stylistic biases or make factual errors, the SHAP rankings describe judge preferences rather than human preferences. Section 5.1 acknowledges 'potential factual errors' and 'inherent bias of the judge models,' but the paper does not quantify agreement between each LLM judge and human attribute scores, nor does it show that the SHAP ranking is stable when individual judges are excluded. Without this validation, the headline claim is not robust to the primary measurement instrument.
- [Appendix A.5; Section 5.1] The human annotation validation is too thin to support the claim that LLM attribute scores are unbiased measures of rationale quality. The annotations were done by the three co-first authors on a randomly sampled subset, but the paper does not report the sample size, the sampling procedure, or inter-annotator agreement. The appendix shows SHAP plots based on these human scores, but with no error bars or uncertainty quantification. Since the human annotations are the only direct evidence about the attributes themselves, the paper needs independent annotators, per-attribute reliability statistics, and a sufficiently large sample for the human-based SHAP analysis to be meaningful.
- [Section 4, Datasets] The external validity of the 'across datasets' claim is limited by dataset construction. Chatbot Arena is filtered by GPT-4o to select mathematical and logical questions, introducing another LLM-judgment step into the pipeline, and MT-Bench contributes only 80 unique questions. The paper should report the filter's agreement with human judgment or provide a human-verified subset, and should temper the generalizability claims given the small MT-Bench sample. As written, the results may reflect the filter's selection bias as much as human preference.
- [Section 4.1, Section A.6] The attribute-specific ELO findings—particularly that GPT-3.5-Turbo outperforms GPT-4 on Arithmetic Accuracy and Self-Consistency—are computed from the same LLM-judged attribute scores that Appendix A.7 shows to be unreliable. The paper presents these as 'unique insights about models' strengths and weaknesses,' but without verification against human attribute scores or a targeted error analysis, these specific claims are fragile. At minimum, the paper should report judge-by-judge ELO rankings to show whether the finding is consistent across all three LLM judges.
minor comments (5)
- [Section 3.2, Appendix A.3] The prompt instructions say high scores are always best, even for Hallucination and Repetition. This is counterintuitive and should be clarified in the main text, since SHAP direction interpretation depends on knowing that a high Hallucination score means absence of hallucination.
- [Appendix A.5] Please state the exact number of rationales annotated by the human annotators and describe the random sampling procedure. Currently the reader cannot tell how much data the human-based SHAP plots are based on.
- [Appendix A.6.2] Some figure captions appear inconsistent: Figures 74 and 75 are in the Human Annotator section but their captions say 'LLM Judges,' and Figures 76 and 77 use both 'Lower is better' and 'Higher is better' without explanation. These should be corrected.
- [Abstract, Section 4] The dataset name is written inconsistently as 'MT Bench' and 'MT-Bench' across the abstract and main text. Please standardize.
- [Section 5.2] The sentence 'the practical utility of human nature' appears to be a typo; likely 'human-AI collaboration' or 'human utility' was intended.
Circularity Check
No significant circularity: the SHAP analysis is a descriptive regression of externally defined attribute scores on human preference labels; no step reduces to its own input by construction.
full rationale
The claimed derivation chain is: (1) identify attributes from prior literature; (2) measure them with ROSCOE metrics, LLM judges, and human annotations; (3) train LightGBM with these attribute scores as features to predict human binary preference labels; (4) run SHAP to rank feature importance; (5) compute attribute-specific ELO from the same attribute scores. None of these steps defines an attribute in terms of the preference label, none fits a parameter to a subset and then predicts a forced function of that fit, and no load-bearing premise is justified by a self-citation. The closest concern is that LLM judges may have their own biases and the paper's own Appendix A.7 shows GPT-4o miscoring Correctness; however that is a measurement-validity limitation, explicitly acknowledged in Section 5.1, not a circular reduction. The human preference labels (MT-Bench/Chatbot Arena) are external gold-standard data, and the attribute scores are independent inputs to the model; the SHAP result is a correlation/importance statement, not a prediction that is algebraically identical to the inputs. The attribute-specific ELO rankings are a re-indexing of the same LLM-judged scores, but they are presented as a re-evaluation, not as an independent prediction of human preference. Therefore no specific circular step can be quoted, and the score is 0.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption Human preference labels in MT-Bench and Chatbot Arena are a valid gold standard for rationale quality.
- domain assumption LLM-judge scores for the 12 attributes are accurate, unbiased measures of those attributes.
- standard math SHAP values on a LightGBM model trained on attribute features validly quantify the importance of each attribute in explaining human preference.
- domain assumption The filtering of Chatbot Arena using GPT-4o to select mathematical and logical questions preserves a representative sample of rationales.
Cite this review
Pith. "Pith review of Rethinking Human Preference Evaluation of LLM Rationales." pith.science (2026). https://pith.science/paper/4S5JITTJ
@misc{pith2026250911026,
author = {Pith},
title = {Pith review of: Rethinking Human Preference Evaluation of LLM Rationales},
year = {2026},
howpublished = {\url{https://pith.science/paper/4S5JITTJ}},
note = {Machine review of arXiv:2509.11026}
}
read the original abstract
Large language models (LLMs) often generate natural language rationales -- free-form explanations that help improve performance on complex reasoning tasks and enhance interpretability for human users. However, evaluating these rationales remains challenging. While recent work has relied on binary preference judgments from humans or LLM judges, such evaluations are often opaque and coarse-grained, offering limited insight into what makes one rationale better than another. In this work, we rethink preference evaluation for LLM-generated rationales by asking: (1) What attributes define good rationales? (2) Can human preferences be explained by these attributes? (3) Can attribute-based evaluation overcome the limitations of binary comparisons? We identify a set of key rationale attributes from prior literature and assess them using automatic metrics, LLM judgments, and human annotations. We then analyze two standard human preference datasets MT Bench and Chatbot Arena using SHAP to identify which attributes best explain human preference outcomes. Finally, we re-evaluate model-generated rationales using attribute-specific ELO scores, revealing more nuanced model comparisons and insights. Our findings suggest that fine-grained attribute evaluations can better characterize rationale quality and guide future research toward more interpretable and reliable evaluation practices.
Figures
Forward citations
Cited by 2 Pith papers
-
CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents
A trained router plus conformal budget calibration lets coding agents choose cheap recovery vs. escalation after a failed attempt, producing a cost–quality frontier from a single model.
-
CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents
Budget-calibrated recovery routing with conformal risk control lets coding agents match always-escalate solve rates at about 35% of the cost.
Reference graph
Works this paper leans on
-
[2]
URL https://arxiv. org/abs/2210.04982. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference,
-
[4]
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei
URL https://arxiv.org/ abs/1706.03741. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30,
-
[5]
URLhttps://arxiv.org/abs/2212.07919. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,
-
[8]
URL https://arxiv. org/abs/2305.07095. 7 Accepted to the First Workshop on the Application of LLM Explainability to Reasoning and Planning at COLM 2025 Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wa...
Pith/arXiv arXiv 2025
-
[11]
URL https://arxiv.org/abs/2501.00656. OpenAI. Learning to reason with LLMs. https://openai.com/index/ learning-to-reason-with-llms/, September
-
[12]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al
Accessed: 2025-06-23. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744,
2025
-
[13]
URL https://arxiv.org/ abs/2304.10703. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741,
-
[15]
Peifeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren
URLhttps://arxiv.org/abs/2311.02805. Peifeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. Pinto: Faithful language reasoning using prompt-generated rationales,
-
[16]
URL https://arxiv.org/ abs/2211.01562. 8 Accepted to the First Workshop on the Application of LLM Explainability to Reasoning and Planning at COLM 2025 Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information p...
Pith/arXiv arXiv 2025
-
[18]
URL https://arxiv.org/abs/2306.05685. 9 Accepted to the First Workshop on the Application of LLM Explainability to Reasoning and Planning at COLM 2025 A Appendix A.1 ROSCOE Metrics ROSCOE Metric At- tributes Description faithfulness Mean alignment from the hypothesis chain to the context sentences; higher scores indicate better grounding by the context. f...
Pith/arXiv arXiv 2025
-
[2016]
cc/paper files/paper/2016/file/10a5ab2db37feedfdeaab192ead4ac0e-Paper.pdf
URL https://proceedings.neurips. cc/paper files/paper/2016/file/10a5ab2db37feedfdeaab192ead4ac0e-Paper.pdf. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pra...
2016
-
[2017]
Scott M Lundberg and Su-In Lee
URL https://proceedings.neurips.cc/paper files/paper/2017/ file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf. Scott M Lundberg and Su-In Lee. A unified approach to interpreting model pre- dictions. In I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.),Advances in Neural Information Processing Systems 30, pp....
2017
-
[2019]
URL https: //arxiv.org/abs/1906.02361. Sahana Ramnath, Brihi Joshi, Skyler Hallinan, Ximing Lu, Liunian Harold Li, Aaron Chan, Jack Hessel, Yejin Choi, and Xiang Ren. Tailoring self-rationalizers with multi-reward distillation,
Pith/arXiv arXiv 1906
-
[2020]
doi: 10.18653/v1/2020.findings-emnlp.390
Associ- ation for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.390. URL https://aclanthology.org/2020.findings-emnlp.390/. Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, and Fei Liu. Decipherpref: Analyzing influential factors in human preference judgments via gpt-4. arXiv preprint arXiv:2305.14702,
Pith/arXiv arXiv 2020
-
[2022]
URL https://arxiv. org/abs/2112.08674. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822,
-
[2023]
URLhttps://arxiv.org/abs/2305.18029. Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues.arXiv preprint arXiv:2402.14762,
-
[2024]
Paul Christiano, Jan Leike, Tom B
URLhttps://arxiv.org/abs/2403.04132. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences,
-
[2025]
Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. Leakage-adjusted simulata- bility: Can models generate non-trivial explanations of their behavior in natural lan- guage? In Trevor Cohn, Yulan He, and Yang Liu (eds.),Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4351–4367, Online, November
2020
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.