REVIEW 2 major objections 6 minor 32 references
The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity
T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read In open-ended tasks, showing peer discussion to an LLM judge can change the rating of an identical answer, and wrong peers consistently produce the lowest-quality revisions.
desk verdict Solid generator-side result on open-ended conformity; the evaluator-sensitivity claim is well-designed but confounded by an uncounterbalanced blind-then-informed order. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the paired blind-informed evaluation contrast, formalized as the judge-specific loading $\gamma_j$ in a hierarchical ordinal model. Each generated answer is rated once with the peer block hidden and once with it visible; because the answer is identical, the difference isolates the evaluator side of peer-context sensitivity. On the generator side, the branched Round 2 design defines three estimands — ordinary re-answering $\Delta_{sp}$, content exposure $\Delta_{\mathrm{content},k}$, and the peer-presentation residual $\Delta_{pp,c}$ — whose sum gives the total peer-arm shift, separating what the candidate text does from what the attributed social packaging does.
What would settle it
Randomize the order of the blind and informed rating passes, or use separate judges for each pass, and re-estimate $\gamma_j$; if the estimated evaluator-side sensitivity changes materially with order, the claim that judges react to visible peer context is confounded by pass order, while stability would support the claim.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that evaluators are part of the conformity experiment: peer context can change the rating of an identical answer, and this evaluator-side sensitivity is heterogeneous across judges. One judge shifts ratings toward the peer-endorsed position, two shift away, one is approximately neutral, and two frontier API judges also show credibly negative directional sensitivity. On the generator side, the total quality shift decomposes into a small ordinary re-answering component, a content-exposure component, and a bundled peer-presentation residual; the residual is negative in every tested cell, and all-wrong peers produce the lowest latent-quality revisions in all twelve generator-dataset cells. Because blind and informed ratings hold the answer text fixed, the paper attributes the rating contrast to the evaluator's reaction to visible peer context rather than to changed generator output. The paper further shows that the anchor items fixing the latent scale can be misordered by judges unless scored with the gold reference visible.
Load-bearing premise
The peer-context sensitivity of judges rests on comparing blind and informed ratings of the same answers, but the paper does not report randomizing or counterbalancing which rating pass comes first; if judge behavior drifts from the first to the second pass, that drift is absorbed into the estimated sensitivity.
Editorial extensions
If this is right
- Flip-based conformity measures will understate harm in open-ended settings because a revision can keep its nominal answer while losing quality.
- Systems that forward peer answers in multi-agent loops should expect the social packaging itself to lower rated quality, independent of content.
- Evaluator panels should report blind-versus-context differences per judge rather than assuming judges add only information about quality.
- Latent-scale conclusions need an explicit anchor-recognition audit, since reference-free anchors are misordered on roughly a quarter of questions.
Reading between the lines
- The same paired blind-informed protocol transfers directly to other treatment-correlated judge context, such as retrieved passages in retrieval-augmented generation or rubric exemplars, where context is assumed informative but may bias the rating.
- The reported $\gamma_j$ is conditional on an antisymmetric coding in which mixed context is zero; a condition-independent shift in leniency between the blind and informed passes is not separately identified, so future designs should add an informed-pass intercept to capture it.
- The 54% Round 1 reproduction rate under greedy decoding means the decomposition corpora had to be self-contained; this suggests that non-deterministic batching, not sampling temperature, can break reproducibility, which is worth checking in other greedy pipelines.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces an experimental protocol for measuring open-ended LLM conformity that separates generator-side revision effects from evaluator-side reactions to visible peer context. Four open-weight generators are run on three benchmarks with a branched design that includes no-peer re-answering, content-only prompts, and peer-presented prompts; all generated answers receive blind ratings from a four-model judge panel, and peer-presented answers are additionally rated with the peer block visible. The authors fit a hierarchical ordinal model to estimate condition shifts and judge-specific peer-context sensitivity, and they corroborate the generator-side ordering with a fixed RoBERTa-MNLI classifier and other automated metrics. They report four conclusions: flip rates are insufficient for open-ended conformity; all-wrong peers produce the lowest-quality revisions in all 12 generator-dataset cells; evaluators exhibit heterogeneous, judge-specific sensitivity to visible peer context; and anchor calibration must be audited explicitly.
Significance. If the claims hold, the paper makes a useful methodological contribution: it demonstrates that graded, evaluator-sensitive measurement matters for open-ended conformity and that same-answer blind-versus-informed contrasts can, in principle, isolate evaluator-side effects. The generator-side results are strong: the all-wrong ordering is consistent across raw means, a hierarchical model, self-rating exclusions, and a non-generative fixed classifier. The paper also ships with substantial validation: reproducibility seeds, anchor audits, scale-sensitivity refits, and a detailed computational appendix. The most novel and distinctive claim — that evaluators shift identically-worded ratings in response to visible peer context, with direction varying by judge — is plausible but currently rests on an identification assumption that is not demonstrated.
major comments (2)
- [Section 2.3, Eq. (7); Appendix B.1-B.2] The evaluator-side loading γ_j is identified from blind and informed ratings of the same answer, but the paper reports no randomization or counterbalancing of the two rating passes; the appendix shows the blind prompt first and the informed prompt second. If informed ratings always follow blind ratings, any systematic within-judge change across the passes (fatigue, scale drift, learning, or condition-correlated carryover) is absorbed into γ_j. The paper's concession in Section 3.3 that a generic informed-pass shift is not separately identified does not resolve the issue, because the mixed condition (d_i=0) could in principle estimate a condition-independent shift. I request either evidence that pass order was randomized or counterbalanced, a robustness model that includes an informed-pass intercept estimated from the mixed cell, or an analysis demonstrating that the γ_j estimates in Table 4 are invariant to pass order.
- [Section 3.4] The text states that a 'reference-free intermediate fit also changed the direction of the estimated asymmetry despite satisfactory sampling diagnostics,' but no such fit is reported in the paper or appendix. This claim is load-bearing for Conclusion 4 because it is the only direct evidence that a poorly calibrated anchor scale can reverse a substantive conclusion. Please either document the fit with its point estimates, credible intervals, and convergence diagnostics, or remove the claim and rely solely on the anchor-recognition audit in Table 5.
minor comments (6)
- [Section 3.1] The 9.0% statistic for all-wrong revisions that a binary flip indicator would score as unchanged is not operationalized; please specify how the flip indicator is defined for free-text answers and how the 9.0% value is computed.
- [Section 2.2] Krippendorff's α = 0.235 is reported as the raw blind agreement, but the paper does not report pairwise judge agreement or a justification for why the hierarchical model adequately accounts for such low raw agreement; a short clarification would help.
- [Section 2.3, Eq. (7)] The notation '⊮' for the indicator of the informed pass is nonstandard and could be confused with a negation symbol; please use a standard indicator notation such as '\mathbb{1}'.
- [Table 4] The GPT-4o and GPT-5.4-mini rows come from separate five-judge refits, not from the same posterior as the four open-weight rows; the footnote says this, but the presentation would be clearer if those rows were in a separate table or clearly labeled as non-comparable posterior quantities.
- [Appendix A.2] The count of 147,000 generated answers includes reproducibility executions (seeds 43 and 44) and anchor responses, while the primary hierarchical analysis uses only seed 42; please make this distinction explicit in the main text when referencing corpus sizes.
- [Figure 1] The illustrative scores 4/5, 2/5, and 3/5 in Figure 1 are not labeled as hypothetical ratings; adding a caption note that these are schematic values would avoid confusion with actual data.
Circularity Check
No significant circularity: the central contrasts are identified from held-fixed answer text and are externally corroborated.
full rationale
The derivation chain is self-contained. Eq. (7) defines gamma_j as the coefficient on the informed-pass indicator interacted with signed peer context d_i; because the same answer theta_i appears in both passes, gamma_j is identified from the blind-informed rating contrast and is not set by construction. The subsequent conclusion that gamma_j differs from zero is a posterior statement about observed ratings, not a restatement of the model definition. The generator-side decomposition (Eqs. 1-4) is an accounting identity, but its components are estimated from separately constructed decomposition corpora, and the headline all-wrong-is-worst ordering is checked against fixed RoBERTa-MNLI false-endorsement scores (Appendix E.5) and self-rating-excluded refits (Appendix E.6), both independent of the hierarchical LLM-judge fit. Self-citations provide the dataset sample and peer-template surface, but no conclusion depends on those papers as a proof. The lack of reported counterbalancing of blind vs. informed passes is a real identification limitation for the evaluator-side contrast, but it is a confound (order effects) rather than a circular reduction, since gamma_j is not algebraically forced by the design.
Assumptions & free parameters
free parameters (1)
- Anchor magnitude theta_a =
1.5 (sensitivity refits at 1.0 and 2.0)
assumptions (5)
- domain assumption Ordinal ratings from each judge follow an ordered logistic model with shared cutpoints after a judge-specific linear transform of latent quality.
- ad hoc to paper The informed-pass effect is antisymmetric in signed peer context: d_i = +1 for all-correct, 0 for mixed, -1 for all-wrong, with no separate generic informed-pass shift.
- domain assumption Blind and informed passes differ only by the presence of the peer block, with no order or context effects.
- domain assumption Anchors, scored with the gold reference visible, identify the origin and unit of theta for experimental answers scored reference-free.
- domain assumption The separately constructed decomposition corpora estimate the same condition shifts as the pooled main corpus, despite different baselines and totals.
Cite this review
Pith. "Pith review of The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity." pith.science (2026). https://pith.science/paper/C47MOPGJ
@misc{pith2026260804463,
author = {Pith},
title = {Pith review of: The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity},
year = {2026},
howpublished = {\url{https://pith.science/paper/C47MOPGJ}},
note = {Machine review of arXiv:2608.04463}
}
read the original abstract
Prior work on LLM conformity largely measures discrete answer flips under verifiable labels. Open-ended revisions require a different measurement strategy because answer quality is graded, latent, and judged imperfectly. We introduce an experimental protocol implemented across a pooled main peer-condition corpus and separately constructed decomposition corpora, allowing us to separate ordinary re-answering, candidate-content exposure, a bundled peer-presentation residual, and directional judge sensitivity to visible peer context. Across four open-weight generators and three benchmarks, all-wrong peer input produces the lowest-quality revisions in every generator-dataset cell. Blind and informed ratings of identical answers also differ by evaluator: one judge shifts toward the peer-endorsed position, two shift away, one is approximately neutral, and GPT-4o and GPT-5.4-mini audits are likewise non-neutral. Finally, an anchor audit shows that terse correct anchors can be misread often enough to destabilize the latent scale unless calibration is checked explicitly. These results support four conclusions: flip rates are insufficient as a complete measure of open-ended conformity, wrong peers harm open-ended revision, evaluators are not neutral, and anchor calibration is necessary.
Figures
Reference graph
Works this paper leans on
-
[1]
Alfonso Amayuelas, Xianjun Yang, Antonis Antoniades, Wenyue Hua, Liangming Pan, and William Yang Wang. 2024. https://aclanthology.org/2024.findings-emnlp.407/ Multiagent collaboration attack: Investigating adversarial attacks in large language model collaborations via debate . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages...
work page 2024
-
[2]
Satanjeev Banerjee and Alon Lavie. 2005. METEOR : An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65--72
work page 2005
-
[3]
Erica Cau, Valentina Pansanella, Dino Pedreschi, and Giulio Rossetti. 2025. https://doi.org/10.1140/epjds/s13688-025-00579-1 Selective agreement, not sycophancy: investigating opinion dynamics in LLM interactions . EPJ Data Science, 14(1):59
-
[4]
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. ChatEval : Towards better LLM -based evaluators through multi-agent debate. In International Conference on Learning Representations
work page 2024
-
[5]
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.474 Humans or LLMs as the judge? a study on judgement bias . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301--8327, Miami, Florida, USA. Association for Computational Linguistics
-
[6]
Young-Min Cho, Sharath Chandra Guntuku, and Lyle Ungar. 2025. https://doi.org/10.48550/arXiv.2505.21588 Herd behavior: Investigating peer influence in llm-based multi-agent systems . arXiv preprint arXiv:2505.21588
-
[7]
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457
arXiv 2018
-
[8]
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning
work page 2024
Show all 32 references
-
[9]
Andrew Estornell and Yang Liu. 2024. Multi-llm debate: Framework, principals, and interventions. In Advances in Neural Information Processing Systems, volume 37, pages 28938--28964
2024
- [10]
- [11]
-
[12]
Guo, Chao Yan, Avinash Baidya, Katherine Brown, Xiang Gao, Juming Xiong, Zhijun Yin, and Bradley A
Kevin H. Guo, Chao Yan, Avinash Baidya, Katherine Brown, Xiang Gao, Juming Xiong, Zhijun Yin, and Bradley A. Malin. 2026. https://arxiv.org/abs/2605.27288 It's not always sycophancy: Measuring llm conformity as a function of epistemic uncertainty . Preprint, arXiv:2605.27288
2026 arXiv
-
[13]
Hoffman and Andrew Gelman
Matthew D. Hoffman and Andrew Gelman. 2014. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. Journal of Machine Learning Research, 15(47):1593--1623
2014
-
[14]
Jiseung Hong, Grace Byun, Seungone Kim, and Kai Shu. 2025. https://doi.org/10.18653/v1/2025.findings-emnlp.121 Measuring sycophancy of language models in multi-turn dialogues . In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2239--2259, Suzhou, ...
2025 doi
-
[15]
Yibo Hu and Jiaming Qu. 2026. Most llm conformity needs no speaker: Measuring the speaker-free floor in peer-pressure benchmarks. arXiv preprint arXiv:2607.05545
2026 arXiv
-
[16]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \'e o Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang,...
2023 arXiv
-
[17]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention . In Proceedings of the 29th Symposium on Operating Systems Princi...
2023
-
[18]
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214--3252
2022
-
[19]
Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yaning Jia, Peijun Qing, Lin Shi, Arman Cohan, Yujun Yan, and Soroush Vosoughi. 2025. https://aclanthology.org/2025.findings-emnlp.941/ Judging with many minds: Do more perspectives mean less prejudice? on bias amplification and r...
2025
-
[20]
Arash Marioriyad, Mohammad Hossein Rohban, and Mahdieh Soleymani Baghshah. 2025. https://arxiv.org/abs/2509.26072 The silent judge: Unacknowledged shortcut bias in LLM -as-a-judge . In NeurIPS 2025 Workshop on Reliable ML from Unreliable Data
2025
-
[21]
Du Phan, Neeraj Pradhan, and Martin Jankowiak. 2019. https://arxiv.org/abs/1912.11554 Composable effects for flexible and accelerated probabilistic programming in numpyro . arXiv preprint arXiv:1912.11554
2019 arXiv
-
[22]
Priya Pitre, Naren Ramakrishnan, and Xuan Wang. 2025. https://doi.org/10.18653/v1/2025.findings-acl.1141 CONSENSAGENT : Towards efficient and effective consensus in multi-agent LLM interactions through sycophancy mitigation . In Findings of the Association for Computational Li...
2025 doi
-
[23]
Jiaming Qu, Lucheng Fu, and Yibo Hu. 2026. Easier to mislead than to correct: Harmful and beneficial revision in llm conformity. arXiv preprint arXiv:2606.01637
2026 arXiv
-
[24]
Leonardo Ranaldi and Giulia Pucci. 2023. https://arxiv.org/abs/2311.09410 When large language models contradict humans? large language models' sycophantic behaviour . arXiv preprint arXiv:2311.09410
2023 arXiv
-
[25]
Fumiko Samejima. 1969. Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, (17)
1969
-
[26]
Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. 2025. https://aclanthology.org/2025.ijcnlp-long.18/ Judging the judges: A systematic study of position bias in LLM -as-a-judge . In Proceedings of the 14th International Joint Conference on Natu...
2025
-
[27]
Aki Vehtari, Andrew Gelman, Daniel Simpson, Bob Carpenter, and Paul-Christian B \"u rkner. 2021. Rank-normalization, folding, and localization: An improved R for assessing convergence of MCMC . Bayesian Analysis, 16(2):667--718
2021
-
[28]
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. MMLU-Pro : A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Sys...
2024
-
[29]
Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2024. Self-preference bias in LLM -as-a-judge. arXiv preprint arXiv:2410.21819
2024 arXiv
-
[30]
Zhiyuan Weng, Guikun Chen, and Wenguan Wang. 2025. https://openreview.net/forum?id=st77ShxP1K Do as we do, not as you think: the conformity of large language models . In Proceedings of the International Conference on Learning Representations
2025
-
[31]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115
2024 arXiv
-
[32]
Weinberger, and Yoav Artzi
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore : Evaluating text generation with BERT . In International Conference on Learning Representations
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.