Pith. sign in

REVIEW 3 major objections 90 references

Evaluator-specific reasoning traces let LLMs simulate individual preference judges better than consensus-only baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 02:41 UTC pith:SFAGBFGO

load-bearing objection Careful factorial study showing that evaluator-specific ICL (especially retrospective reasoning) can beat a same-model base judge by up to ~10 points, with honest negative telemetry results and systematic predictors of who is hard to simulate. the 3 major comments →

arxiv 2607.05742 v1 pith:SFAGBFGO submitted 2026-07-07 cs.HC

PERSONAJUDGE: Simulating Individual Human Preference Judgments with Evaluator-Specific Demonstration Data

Zeyu He , Xuan Qi , Subramanian Chidambaram , Zhichao Xu , Vinayak Arannil , Lydia Chilton , Alex C. Williams This is my paper
classification cs.HC
keywords LLM-as-judgeindividual preference simulationin-context learningretrospective reasoninginterface telemetryevaluator variationpersonalized evaluationHelpful and Harmless
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most LLM-as-judge pipelines learn crowd consensus and erase how different people actually decide. PERSONAJUDGE instead feeds an LLM each evaluator's own past labels plus optional process data—retrospective reasoning and interface telemetry—and asks the model to simulate that person's next three-way judgment (prefer A, neutral, prefer B). Across 32 trained annotators and 4,200 judgments on helpfulness and harmlessness tasks, the best configuration improves accuracy by as much as 9.9 points over the same model with no personal demonstrations. Retrospective reasoning supplies the largest lift; raw interface telemetry often hurts. Simulation difficulty is systematic: people who use Neutral a lot or diverge from consensus are harder to mimic, and the Neutral-usage tendency itself is stable across tasks even when simulatability is not. The work shows that individual-aware evaluation is feasible, but only when the right kind of process data is collected.

Core claim

Conditioning an LLM on an evaluator's own multi-facet demonstrations—especially categorical judgments paired with retrospective reasoning—raises three-class simulation accuracy over a zero-shot Base Judge by up to 9.9 percentage points, and the gain is personalization rather than generic demonstration benefit.

What carries the argument

PERSONAJUDGE: a two-round in-context learning cascade that first predicts whether the target evaluator will express any preference, then (if needed) predicts its direction, using demonstrations that can include labels, interface telemetry, and retrospective reasoning traces.

Load-bearing premise

The post-hoc think-alouds collected after replaying each judgment are assumed to be faithful enough accounts of the original decision criteria rather than after-the-fact rationalizations.

What would settle it

A delayed re-test of the same evaluators on held-out items, or a side-by-side comparison of concurrent versus retrospective verbal reports, that shows the reasoning traces fail to improve simulation once rationalization is controlled for.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper proposes PERSONAJUDGE, an ICL framework that simulates an individual evaluator’s three-class preference judgment (Prefer A / Neutral / Prefer B) by conditioning an LLM on that evaluator’s prior categorical labels plus optional interface telemetry and retrospective reasoning traces. Using a 4×4×4 factorial design over 32 trained annotators and 4,200 HH-style judgments (helpfulness and harmlessness), the authors report that evaluator-specific demonstrations improve three-class accuracy over the same model’s zero-shot Base Judge by up to 9.9 pp (Claude-3.5-Sonnet, 8-shot J+RR on Harmlessness), that retrospective reasoning is the most useful complementary signal while event-level telemetry often hurts, and that simulation difficulty is systematic—predicted by neutral usage and divergence from consensus—with neutral usage a stable cross-task trait (r=0.728). Controls include a cross-evaluator demonstration control, oracle/majority baselines, and a deviation-item analysis showing modest but genuine individual capture.

Significance. If the results hold, the work supplies a concrete, carefully controlled methodology for moving LLM-as-Judge pipelines from consensus simulation toward individual-aware evaluation. The multi-facet data collection protocol, two-round cascade, factorial design, cross-evaluator personalization control, and deviation-item analysis are reusable contributions for the field. The honest reporting of modest absolute accuracy (near per-evaluator majority-class), the negative telemetry effect, and the cost–benefit asymmetry between reasoning and telemetry are themselves useful methodological findings for scaling personalized assessment and for reward modeling under heterogeneous preferences. Strengths include transparent baselines, non-parametric significance testing with multiple-comparison correction, and explicit limitations on post-hoc reasoning fidelity.

major comments (3)
  1. §5.1.1–5.1.2 and Table 2: the headline “up to 9.9 pp” gain is configuration-specific (Claude-3.5, 8-shot J+RR on Harmlessness). After FDR correction over the 64 conditions (Appendix H.4), only 3 Harmlessness and 0 Helpfulness configurations remain significant; the recommended configuration is significant only as a planned comparison. The abstract and main claims should state more clearly that average gains are small (+1.4 / +2.8 pp) and that most of the 64 cells do not survive family-wise correction, so that readers do not over-generalize the peak number.
  2. §5.1.3 and Appendix I: PERSONAJUDGE does not significantly exceed the per-evaluator majority-class baseline (∆ = −0.019, p=0.95 Harmlessness; +0.042, p=0.14 Helpfulness). The deviation-item analysis shows genuine individual capture (accuracy ~0.36 on items where consensus predictors score 0 by construction), but the absolute individual signal remains modest. The paper’s framing of “individual evaluator simulation” should more explicitly position the method as a complement to, rather than a replacement for, simple per-person predictors, and discuss what additional signal would be needed to clear that bar.
  3. Limitations and §3.2.2 / Stage-2 protocol: the largest gains rest on retrospective think-alouds cued by interaction replay. The authors correctly note possible rationalization (Ericsson & Simon), but provide no quantitative check (e.g., inter-rater agreement on criteria extracted from traces, or correlation of trace content with Stage-1 dwell/revisit patterns). A short validation or sensitivity analysis would strengthen the claim that J+RR gains reflect decision criteria rather than post-hoc narrative.

Circularity Check

0 steps flagged

No significant circularity: purely empirical held-out accuracy comparisons with no derivation that reduces to its inputs by construction.

full rationale

PERSONAJUDGE is an empirical simulation study. Its load-bearing claims are three-class accuracy of ICL simulations against each evaluator's own held-out validation labels (disjoint 40-item demo pool / 60-item validation split), compared to a zero-shot Base Judge, a cross-evaluator demonstration control, majority-class baselines, and deviation-item analyses. There is no first-principles derivation, no fitted scalar renamed as a prediction, no uniqueness theorem, and no ansatz whose only support is a self-citation. Self-citations (e.g., PILOT, MetaSynth, ADEQA) appear only as related-work context and do not force the reported accuracy deltas. The 9.9 pp gain, the J+RR superiority, the telemetry penalty, and the neutral-usage / consensus-divergence predictors are all measured quantities, not identities. Score 0 is therefore the correct outcome.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

Empirical HCI/ML paper. Load-bearing premises are standard experimental assumptions plus the operationalization of 'individual simulation' via ICL accuracy on three-class labels. No free physical constants; experimental factors (shot counts, models, demo types) are controlled variables, not fitted parameters that define the claim. The main invented construct is the PERSONAJUDGE multi-facet demonstration package itself.

free parameters (2)
  • demonstration count k (1/2/4/8)
    Chosen experimental levels; results plateau after 4, so the recommended 8-shot config is a design choice rather than a data-fitted constant, but the reported peak gain depends on this choice.
  • event-level telemetry serialization schema
    Authors encode raw GUI events into a fixed typed sequence (Table 4). Alternative aggregations (dwell totals, re-read counts) are not tested; the negative IT finding is therefore schema-dependent.
axioms (4)
  • domain assumption In-context learning with a small number of evaluator-specific demonstrations can transfer that evaluator's judgment style to new instances.
    Core methodological premise of §3; standard ICL literature is cited but success on subjective three-class preference is assumed testable rather than proved.
  • domain assumption Post-hoc think-alouds cued by interaction replay are usable proxies for the criteria that produced the original categorical judgment.
    Stage-2 collection protocol (Appendix A); Limitations explicitly notes possible rationalization.
  • ad hoc to paper Three-class accuracy against the same evaluator's held-out labels is the right primary metric of individual simulation fidelity.
    Chosen over agreement with consensus or binary collapse; justified in §4.5 but remains a design choice.
  • domain assumption Trained professional annotators on HH pairwise items are a sufficient testbed for claims about individual-aware AI assessment.
    Study population and task scope (§4.1, Limitations).
invented entities (1)
  • PERSONAJUDGE multi-facet demonstration (J / J+IT / J+RR / J+IT+RR) independent evidence
    purpose: Package categorical judgment with optional interface telemetry and retrospective reasoning for ICL simulation of a target evaluator.
    The framework is the paper's central construct; independent evidence is the empirical accuracy gains reported against base and cross-evaluator controls.

pith-pipeline@v1.1.0-grok45 · 31191 in / 2913 out tokens · 35966 ms · 2026-07-11T02:41:16.776202+00:00 · methodology

0 comments
read the original abstract

Large language models increasingly serve as judges in AI evaluation, but current approaches rely on consensus preferences that ignore individual evaluator variation. We propose a novel simulation approach that combines categorical judgments with evaluator-specific auxiliary data--retrospective reasoning traces and interface telemetry--to enable LLM-based simulation of individual evaluators via in-context learning. We conduct a systematic empirical study of this approach using multi-facet data from 32 trained annotators across 4,200 preference judgments in a 4 x 4 x 4 factorial design. Our key findings: (1) The simulation approach achieves up to 9.9 percentage point improvements over the Base Judge; (2) Reasoning traces provide the largest gains with higher collection efforts, while interface telemetry often hurts rather than helps performance despite being cheaper to collect. (3) Simulation difficulty is systematic, predicted by an evaluator's neutral usage (most clearly on Helpfulness) and divergence from consensus; the neutral-usage tendency--rather than simulatability itself--is the cross-task-stable property (r = 0.728). These results establish both the potential and limits of evaluator-specific auxiliary data for personalized evaluation, offering methodological insights for scaling individual aware AI assessment.

Figures

Figures reproduced from arXiv: 2607.05742 by Alex C. Williams, Lydia Chilton, Subramanian Chidambaram, Vinayak Arannil, Xuan Qi, Zeyu He, Zhichao Xu.

Figure 1
Figure 1. Figure 1: PERSONAJUDGE workflow. A human eval￾uator first completes judgment tasks, producing three complementary signals: a categorical judgment, inter￾face telemetry, and retrospective reasoning. PERSONA￾JUDGE organizes these signals into evaluator-specific demonstrations and combines them with the task instruc￾tion and a new task instance. An LLM then uses this information to simulate the target evaluator’s judgm… view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy comparison across random selec [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Evaluators’ Neutral-selection rates are strongly correlated across tasks (r = 0.728, p < 0.05), whereas simulation fidelity itself is not (r = 0.181). Each point is one of the 10 evaluators participating in both tasks; Neutral rates are computed on each evalua￾tor’s 60 validation items with a human majority label, so they can differ slightly from the full-data rates in [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 4
Figure 4. Figure 4: Stage 1 interface for preference judgment collection, showing the conversation context, two AI responses, [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Stage 2 interface for retrospective reasoning, showing an interaction replay and structured think-aloud [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Retrospective Reasoning Format ICL Demonstration Format [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Evaluator judgment distribution for the Harm [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

90 extracted references · 90 canonical work pages · 6 internal anchors

  1. [1]

    Constitutional AI: Harmlessness from AI Feedback

    Bai, Yuntao and Kadavath, Saurav and Kundu, Sandipan and Askell, Amanda and Kernion, Jackson and Jones, Andy and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and McKinnon, Cameron and Chen, Carol and Olsson, Catherine and Olah, Christopher and Hernandez, Danny and Drain, Dawn and Ganguli, Deep and Li, Dustin and Tran-Johnson, Eli and Perez, Ethan an...

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    If in a crowdsourced data annotation pipeline, a

    He, Zeyu and Huang, Chieh-Yang and Ding, Chien-Kuang Cornelia and Rohatgi, Shaurya and Huang, Ting-Hao Kenneth , booktitle=. If in a crowdsourced data annotation pipeline, a

    work page

  3. [3]

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and Zhang, Hao and Gonzalez, Joseph and Stoica, Ion , journal=. Judging

    work page

  4. [4]

    2006 , publisher=

    Cognitive apprenticeship , author=. 2006 , publisher=

    work page 2006

  5. [5]

    A short primer on situated cognition , author=

    work page

  6. [6]

    Mental models , pages=

    Some observations on mental models , author=. Mental models , pages=. 2014 , publisher=

    work page 2014

  7. [7]

    Proceedings of the 2008 conference on empirical methods in natural language processing , pages=

    Cheap and fast--but is it good? evaluating non-expert annotations for natural language tasks , author=. Proceedings of the 2008 conference on empirical methods in natural language processing , pages=

    work page 2008

  8. [8]

    Self-Instruct: Aligning Language Models with Self-Generated Instructions

    Self-instruct: Aligning language models with self-generated instructions , author=. arXiv preprint arXiv:2212.10560 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Advances in Neural Information Processing Systems , volume=

    Alpacafarm: A simulation framework for methods that learn from human feedback , author=. Advances in Neural Information Processing Systems , volume=

    work page

  10. [10]

    Proceedings of the 41st International Conference on Machine Learning , articleno =

    Lee, Harrison and Phatale, Samrat and Mansoor, Hassan and Mesnard, Thomas and Ferret, Johan and Lu, Kellie and Bishop, Colton and Hall, Ethan and Carbune, Victor and Rastogi, Abhinav and Prakash, Sushant , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

    work page 2024

  11. [11]

    Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume=

    Deja vu: Characterizing worker reliability using task consistency , author=. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume=

    work page

  12. [12]

    G -Eval: NLG Evaluation using Gpt-4 with Better Human Alignment

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang. G -Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.153

    work page doi:10.18653/v1/2023.emnlp-main.153 2023

  13. [13]

    A companion to cognitive science , pages=

    Protocol analysis , author=. A companion to cognitive science , pages=. 2017 , publisher=

    work page 2017

  14. [14]

    Advances in neural information processing systems , volume=

    Learning to summarize with human feedback , author=. Advances in neural information processing systems , volume=

    work page

  15. [15]

    2021 , volume=

    Reiichiro Nakano and Jacob Hilton and Suchir Balaji and Jeff Wu and Ouyang Long and Christina Kim and Christopher Hesse and Shantanu Jain and Vineet Kosaraju and William Saunders and Xu Jiang and Karl Cobbe and Tyna Eloundou and Gretchen Krueger and Kevin Button and Matthew Knight and Benjamin Chess and John Schulman , journal=. 2021 , volume=

    work page 2021

  16. [16]

    Understanding Dataset Difficulty with

    Ethayarajh, Kawin and Choi, Yejin and Swayamdipta, Swabha , booktitle =. Understanding Dataset Difficulty with. 2022 , editor =

    work page 2022

  17. [17]

    Behavioral and brain sciences , volume=

    Does the chimpanzee have a theory of mind? , author=. Behavioral and brain sciences , volume=. 1978 , publisher=

    work page 1978

  18. [18]

    theory of mind

    Mindreaders: the cognitive basis of" theory of mind" , author=. 2010 , publisher=

    work page 2010

  19. [19]

    Large Language Models are not Fair Evaluators

    Wang, Peiyi and Li, Lei and Chen, Liang and Cai, Zefan and Zhu, Dawei and Lin, Binghuai and Cao, Yunbo and Kong, Lingpeng and Liu, Qi and Liu, Tianyu and Sui, Zhifang. Large Language Models are not Fair Evaluators. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.ac...

    work page doi:10.18653/v1/2024.acl-long.511 2024

  20. [20]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , year=

    Towards a Unified Multi-Dimensional Evaluator for Text Generation , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , year=

    work page 2022

  21. [21]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year=

    FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year=

    work page 2023

  22. [22]

    Advances in Neural Information Processing Systems , volume=

    Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models , author=. Advances in Neural Information Processing Systems , volume=

    work page

  23. [23]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , pages=

    NLPositionality: Characterizing Design Biases of Datasets and Models , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , pages=

    work page

  24. [24]

    Justice or Prejudice? Quantifying Biases in

    Jiayi Ye and Yanbo Wang and Yue Huang and Dongping Chen and Qihui Zhang and Nuno Moniz and Tian Gao and Werner Geyer and Chao Huang and Pin-Yu Chen and Nitesh V Chawla and Xiangliang Zhang , booktitle=. Justice or Prejudice? Quantifying Biases in. 2024 , url=

    work page 2024

  25. [25]

    Judging the judges: A systematic study of position bias in

    Shi, Lin and Ma, Chiyu and Liang, Wenhua and Diao, Xingjian and Ma, Weicheng and Vosoughi, Soroush , booktitle=. Judging the judges: A systematic study of position bias in

    work page

  26. [26]

    Rank analysis of incomplete block designs:

    Bradley, Ralph Allan and Terry, Milton E , journal=. Rank analysis of incomplete block designs:. 1952 , publisher=

    work page 1952

  27. [27]

    Advances in neural information processing systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in neural information processing systems , volume=

    work page

  28. [28]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Orpo: Monolithic preference optimization without reference model , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2024

  29. [29]

    Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology , pages=

    Creating general user models from computer use , author=. Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology , pages=

    work page

  30. [30]

    OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation

    Ziyi Wang and Yuxuan Lu and Wenbo Li and Amirali Amini and Bo Sun and Yakov Bart and Weimin Lyu and Jiri Gesi and Tian Wang and Jing Huang and Yu Su and Upol Ehsan and Malihe Alikhani and Toby Jia-Jun Li and Lydia Chilton and Dakuo Wang , year=. 2506.05606 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [31]

    1993 , publisher=

    The Adaptive Decision Maker , author=. 1993 , publisher=

    work page 1993

  32. [32]

    Proceedings of the 12th International Conference on Intelligent User Interfaces , year=

    Toward Harnessing User Feedback for Machine Learning , author=. Proceedings of the 12th International Conference on Intelligent User Interfaces , year=

    work page

  33. [33]

    Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , year=

    Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing , author=. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , year=

    work page

  34. [34]

    Advances in Neural Information Processing Systems , year=

    Training Language Models to Follow Instructions with Human Feedback , author=. Advances in Neural Information Processing Systems , year=

    work page

  35. [35]

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback , author=. arXiv preprint arXiv:2204.05862 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  36. [36]

    Organizational Behavior and Human Decision Processes , year=

    Process Tracing Methods: Contributions, Problems, and Neglected Research Questions , author=. Organizational Behavior and Human Decision Processes , year=

    work page

  37. [37]

    Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

    Ready to Buy or Just Browsing? Detecting Web Searcher Goals from Interaction Data , author=. Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

    work page

  38. [38]

    Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

    Understanding Web Browsing Behaviors through Weibull Analysis of Dwell Time , author=. Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

    work page

  39. [39]

    2011 , publisher=

    Eye Tracking: A Comprehensive Guide to Methods and Measures , author=. 2011 , publisher=

    work page 2011

  40. [40]

    Cognitive Psychology , year=

    Eye Fixations and Cognitive Processes , author=. Cognitive Psychology , year=

    work page

  41. [41]

    1993 , publisher=

    Protocol Analysis: Verbal Reports as Data , author=. 1993 , publisher=

    work page 1993

  42. [42]

    Knowledge Based Systems , year=

    The think aloud method: a practical approach to modelling cognitive processes , author=. Knowledge Based Systems , year=

    work page

  43. [43]

    Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing , year=

    Too Much, Too Little, or Just Right? Ways Explanations Impact End Users' Mental Models , author=. Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing , year=

    work page

  44. [44]

    Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , year=

    Why and Why Not Explanations Improve the Intelligibility of Context-Aware Intelligent Systems , author=. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , year=

    work page

  45. [45]

    Cognitive Science , year=

    Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems , author=. Cognitive Science , year=

    work page

  46. [46]

    Who validates the validators?

    Shankar, Shreya and Zamfirescu-Pereira, JD and Hartmann, Bj. Who validates the validators?. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology , pages=

    work page

  47. [47]

    and Hajishirzi, Hannaneh

    Lambert, Nathan and Pyatkin, Valentina and Morrison, Jacob and Miranda, LJ and Lin, Bill Yuchen and Chandu, Khyathi and Dziri, Nouha and Kumar, Sachin and Zick, Tom and Choi, Yejin and Smith, Noah A. and Hajishirzi, Hannaneh. R eward B ench: Evaluating Reward Models for Language Modeling. Findings of the Association for Computational Linguistics: NAACL 20...

    work page doi:10.18653/v1/2025.findings-naacl.96 2025

  48. [48]

    and Le, Quoc V

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed H. and Le, Quoc V. and Zhou, Denny , title =. Proceedings of the 36th International Conference on Neural Information Processing Systems , articleno =. 2022 , isbn =

    work page 2022

  49. [49]

    Advances in Neural Information Processing Systems , volume=

    Large Language Models are Zero-Shot Reasoners , author=. Advances in Neural Information Processing Systems , volume=

    work page

  50. [50]

    Advances in Neural Information Processing Systems , volume=

    Language Models are Few-Shot Learners , author=. Advances in Neural Information Processing Systems , volume=

    work page

  51. [51]

    Information Processing & Management , volume=

    A Systematic Analysis of Performance Measures for Classification Tasks , author=. Information Processing & Management , volume=

    work page

  52. [52]

    Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation

    Evaluation: From Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation , author=. arXiv preprint arXiv:2010.16061 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  53. [53]

    Journal of Memory and Language , volume=

    Mixed-effects Modeling with Crossed Random Effects for Subjects and Items , author=. Journal of Memory and Language , volume=

    work page

  54. [54]

    Journal of the American Statistical Association , volume=

    Multiple Comparisons among Means , author=. Journal of the American Statistical Association , volume=

    work page

  55. [55]

    1988 , publisher=

    Statistical Power Analysis for the Behavioral Sciences , author=. 1988 , publisher=

    work page 1988

  56. [56]

    Proceedings of the 14th International Joint Conference on Artificial Intelligence , volume=

    A Study of Cross-validation and Bootstrap for Accuracy Estimation and Model Selection , author=. Proceedings of the 14th International Joint Conference on Artificial Intelligence , volume=

    work page

  57. [57]

    A Survey on In-context Learning

    Dong, Qingxiu and Li, Lei and Dai, Damai and Zheng, Ce and Ma, Jingyuan and Li, Rui and Xia, Heming and Xu, Jingjing and Wu, Zhiyong and Chang, Baobao and Sun, Xu and Li, Lei and Sui, Zhifang. A Survey on In-context Learning. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.64

    work page doi:10.18653/v1/2024.emnlp-main.64 2024

  58. [58]

    IMPersona: Evaluating Individual Level LM Impersonation

    IMPersona: Evaluating Individual Level LM Impersonation , author=. arXiv preprint arXiv:2504.04332 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2510.22954 , year=

    Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) , author=. arXiv preprint arXiv:2510.22954 , year=

    work page arXiv

  60. [60]

    Rewarded soups: towards

    Rame, Alexandre and Couairon, Guillaume and Dancette, Corentin and Gaya, Jean-Baptiste and Shukor, Mustafa and Soulier, Laure and Cord, Matthieu , journal=. Rewarded soups: towards

    work page

  61. [61]

    2024 , editor =

    Chakraborty, Souradip and Qiu, Jiahao and Yuan, Hui and Koppel, Alec and Manocha, Dinesh and Huang, Furong and Bedi, Amrit and Wang, Mengdi , booktitle =. 2024 , editor =

    work page 2024

  62. [62]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    Helpsteer: Multi-attribute helpfulness dataset for steerlm , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    work page 2024

  63. [63]

    Can LLM be a Personalized Judge?

    Dong, Yijiang River and Hu, Tiancheng and Collier, Nigel. Can LLM be a Personalized Judge?. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.592

    work page doi:10.18653/v1/2024.findings-emnlp.592 2024

  64. [64]

    Chen, Lingjiao and Zaharia, Matei and Zou, James , journal=. How is. 2024 , publisher=

    work page 2024

  65. [65]

    Computer Speech & Language , volume=

    Human evaluation of automatically generated text: Current trends and best practice guidelines , author=. Computer Speech & Language , volume=. 2021 , publisher=

    work page 2021

  66. [66]

    Transactions of the Association for Computational Linguistics , volume=

    Dealing with disagreements: Looking beyond the majority vote in subjective annotations , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher=

    work page 2022

  67. [67]

    Proceedings of the 42nd International Conference on Machine Learning , pages =

    Diverging Preferences: When do Annotators Disagree and do Models Know? , author =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , editor =

    work page 2025

  68. [68]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    When the majority is wrong: Modeling annotator disagreement for subjective tasks , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2023

  69. [69]

    We Need to Consider Disagreement in Evaluation

    Basile, Valerio and Fell, Michael and Fornaciari, Tommaso and Hovy, Dirk and Paun, Silviu and Plank, Barbara and Poesio, Massimo and Uma, Alexandra. We Need to Consider Disagreement in Evaluation. Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future. 2021. doi:10.18653/v1/2021.bppf-1.3

    work page doi:10.18653/v1/2021.bppf-1.3 2021

  70. [70]

    Language Resources and Evaluation , volume=

    Perspectivist approaches to natural language processing: a survey , author=. Language Resources and Evaluation , volume=. 2025 , publisher=

    work page 2025

  71. [71]

    Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

    Sap, Maarten and Swayamdipta, Swabha and Vianna, Laura and Zhou, Xuhui and Choi, Yejin and Smith, Noah A. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022. doi:10...

    work page doi:10.18653/v1/2022.naacl-main.431 2022

  72. [72]

    Can Language Models Reason about Individualistic Human Values and Preferences?

    Jiang, Liwei and Sorensen, Taylor and Levine, Sydney and Choi, Yejin. Can Language Models Reason about Individualistic Human Values and Preferences?. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.336

    work page doi:10.18653/v1/2025.acl-long.336 2025

  73. [73]

    Annotation alignment: Comparing

    Movva, Rajiv and Koh, Pang Wei and Pierson, Emma , booktitle=. Annotation alignment: Comparing

    work page

  74. [74]

    Learning Personalized Alignment for Evaluating Open-ended Text Generation

    Wang, Danqing and Yang, Kevin and Zhu, Hanlin and Yang, Xiaomeng and Cohen, Andrew and Li, Lei and Tian, Yuandong. Learning Personalized Alignment for Evaluating Open-ended Text Generation. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.737

    work page doi:10.18653/v1/2024.emnlp-main.737 2024

  75. [75]

    Judgment and Decision making , volume=

    Applying the decision moving window to risky choice: Comparison of eye-tracking and mouse-tracing methods , author=. Judgment and Decision making , volume=. 2011 , publisher=

    work page 2011

  76. [76]

    2002 , publisher=

    Jasper, JD and Shapiro, Jennifer , journal=. 2002 , publisher=

    work page 2002

  77. [77]

    Cognitive psychology , volume=

    Eye fixations and cognitive processes , author=. Cognitive psychology , volume=. 1976 , publisher=

    work page 1976

  78. [78]

    Judgment and Decision making , volume=

    The role of process data in the development and testing of process models of judgment and decision making , author=. Judgment and Decision making , volume=. 2011 , publisher=

    work page 2011

  79. [79]

    Camburu, Oana-Maria and Rockt. e-. Advances in Neural Information Processing Systems , volume=

    work page

  80. [80]

    Advances in Neural Information Processing Systems , volume=

    Towards interpretable natural language understanding with explanations as latent variables , author=. Advances in Neural Information Processing Systems , volume=

    work page

Showing first 80 references.