Pith. sign in

REVIEW 4 major objections 5 minor 81 references

Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding dialog-act and maxim analysis to LLM judge prompts improves preference accuracy on multi-turn conversations.

desk verdict Useful framework for multi-turn LLM judging, but the causal role of dialog acts and maxims is unproven due to a missing length-matched control. read the letter →

arxiv 2505.20451 v1 pith:YOSWOSHT submitted 2025-05-26 cs.CL

Sahana Ramnath , Anurag Mudgil , Brihi Joshi , Skyler Hallinan , Xiang Ren This is my paper
classification cs.CL
keywords LLM-as-a-judgemulti-turnconversationevaluationdialogactsGriceanmaximspreferencejudgmentsrewardmodelsjuryofjudgesconversational
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Amulet, a training-free prompting framework that improves how LLM judges decide which assistant response is better in long, multi-turn conversations. Amulet asks the judge to first analyze each turn's dialog acts — the communicative dimensions and functions such as request, inform, thank — and to compare the two candidate responses against twelve Gricean maxims of good conversation. On four preference datasets, these prompts alone outperform plain input/output and explain-then-choose judges, and combining the two prompts into a jury with a fallback mechanism yields further gains that also beat several fine-tuned reward models. The paper argues this works because real human-assistant conversations shift intent frequently — about 60 to 70 percent of consecutive human turns change dialog acts — and because in roughly 75 percent of instances the preferred response can be distinguished by dialog acts or maxim satisfaction. If right, Amulet offers a cheap, drop-in upgrade for evaluating assistants and collecting preference data.

What carries the argument

The load-bearing objects are two linguistic annotation schemes dropped into the judge prompt: dialog acts, defined as communicative dimensions with functions (for example, Task/Set Question or Social Obligations Management/Thanking), and twelve Gricean sub-maxims (Quantity-1, Quantity-2, Quality, Relevance-1/2, Manner-1/2, Benevolence-1/2, Transparency-1/2/3). The mechanism is the two-vote jury pipeline: each instance is judged twice with response positions swapped to counter position bias, and the jury resolves disagreement by consulting AMULET-DA first, then AMULET-MAXIM, then either the vanilla explain judge or a reward model to break remaining ties. This pipeline converts the two signal sources into a decision, and the paper's win-tie-loss statistics show the jury raises the win rate and breaks ties relative to the single DA judge.

What would settle it

Run AMULET-DA, AMULET-MAXIM, and a control prompt of matched length that simply instructs the judge to reason step by step about which response better continues the conversation, on the same four datasets. If the control matches or exceeds Amulet's accuracies, the improvement is not attributable to dialog acts and maxims, and the paper's linguistic framing is not the active ingredient.

Watch

Extended reading notes

Core claim

Amulet's central claim is that preference judgments in complex multi-turn conversations become more accurate when the judge is forced to make its reasoning linguistic: first labeling each turn with dialog acts (dimensions like Task or Social Obligations Management plus functions like Request or Thanking), then scoring which candidate response better satisfies twelve conversational maxims covering quantity, quality, relevance, manner, benevolence, and transparency. On HH-TEST, WildFeedback, Nectar, and HH-TRAIN with GPT-4o, the DA prompt and the MAXIM prompt together outperform the vanilla I/O baseline, while AMULET-MAXIM alone underperforms; the authors interpret this as maxims capturing distinctions that dialog acts miss. The full system is a jury: use AMULET-DA's two votes; if they disagree, use AMULET-MAXIM; if that still ties, fall back to the vanilla judge or a reward model. This DA-then-MAXIM-then-fallback pipeline achieves the paper's best accuracies, beating all three state-of-the-art reward models in most settings, and the ordering of DA before MAXIM is not critical.

Load-bearing premise

The load-bearing premise is that the accuracy gains come from the dialog-act and maxim constructs themselves, rather than from the longer, more detailed reasoning prompt; the experiments compare against a bare prompt and a prompt that adds an explanation, but not against a length-matched step-by-step reasoning prompt without dialog acts and maxims.

Editorial extensions

If this is right

  • LLM-based evaluation of multi-turn assistant responses can be made more accurate without any fine-tuning, just by adding dialog-act and maxim analysis to the prompt.
  • The DA-then-MAXIM jury beats the plain LLM judge and, in most settings, also beats the strong reward models tested, so it can serve as a lightweight alternative when reward-model compute is unavailable.
  • Dialog acts and maxim satisfaction distinguish chosen from rejected responses in roughly 75 percent of instances, which means the signal is broadly available in existing preference data.
  • Ordering the jury as MAXIM-then-DA gives nearly the same accuracies, so the benefit comes from combining the two perspectives rather than from a specific decision order.
  • The framework carries over to Claude and Qwen, not just GPT-4o, and to conversations with seven or more human turns.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The experiments do not include a length-matched control prompt without dialog acts and maxims, so part of the gain may come from longer structured reasoning; a step-by-step baseline would isolate the linguistic content.
  • If the linguistic annotations are the active ingredient, the same signals could be used to clean preference datasets — for instance, flagging instances where both responses satisfy the same maxims and are near-ties — rather than only to judge them.
  • A natural extension is to distill the DA and maxim reasoning into a small fine-tuned judge, which would make the accuracy gain available without API costs.
  • The reported annotation quality (84 percent of DA turns and 96 percent of maxim judgements correct in the authors' qualitative check) suggests the method's output could double as interpretable explanations, though the paper notes that natural-language explanations may still hallucinate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AMULET, a prompt-based framework for LLM-as-judge preference evaluation in multi-turn conversations. It defines AMULET-DA, which asks the judge to label each turn with ISO-style dialog acts, and AMULET-MAXIM, which asks the judge to compare two candidate responses on twelve Gricean sub-maxims. It also introduces juries that combine these prompts with each other, with a vanilla judge, or with reward models. Experiments on HH-TEST/HH-TRAIN, NECTAR, and WILDFEEDBACK with GPT-4O and other models report accuracy improvements over I/O and W-EXPL baselines and over three reward models. The paper also reports descriptive analyses of dialog-act shifts and maxim asymmetries in these datasets.

Significance. If the improvements are real and specifically attributable to the dialog-act and maxim constructs, AMULET would be a useful, training-free addition to LLM-judge evaluation. The paper's strengths include the use of four multi-turn preference datasets, explicit prompts in appendices, a two-vote position-bias mitigation, and comparison against strong reward models. The descriptive statistics about intent shifts are potentially valuable for dataset design. However, the manuscript's causal claim is not yet established, because the experimental design does not separate the effect of the linguistic constructs from the effect of longer, more structured prompts, and because no statistical tests accompany the accuracy differences.

major comments (4)
  1. [§4.2, §4.4, Appendices F–G] The central claim that dialog acts and maxims improve accuracy is not supported without a length-matched or structure-matched control. AMULET-DA and AMULET-MAXIM prompts are much longer than the I/O and W-EXPL baselines; they contain taxonomies, definitions, few-shot examples, and structured per-turn output formats. A generic 'analyze each turn step by step and then decide' prompt that matches the length and reasoning structure without using DA or maxim vocabulary is needed to attribute the gains to the linguistic constructs. This control is required because the abstract and conclusion state a causal role for dialog acts and maxims.
  2. [§4.4, Table 2, Limitations] The paper repeatedly uses the word 'significant' for the accuracy improvements, but no significance tests are reported. The Limitations section states that only single runs were performed, and the single rerun of AMULET-DA on HH-TEST dropped from 59.8% to 58%. With dataset sizes ranging from 460 to 8,210 instances and differences of a few percentage points, the statistical significance of the reported gains cannot be assessed. The authors should report multiple runs, bootstrap confidence intervals, or a paired significance test, and restrict 'significant' to results with a defined statistical procedure.
  3. [§3.1, §3.2, Appendix H.1] The descriptive insights about dialog-act shift rates and maxim differentiation are computed from GPT-4O's own DA and MAXIM annotations, with the human validation performed by the authors on only 30 conversations for DA and 7 conversations for maxims. Since the abstract highlights the '60 to 70 percent' intent-change statistic as a finding about the data, the paper should either obtain substantially larger and independently annotated validation or explicitly present these numbers as model-based estimates with appropriate uncertainty. The current validation sample is too small to support the strength of the claims in Section 3.
  4. [§4.4, Table 2] The text says AMULET-DA outperforms the baseline LLM judges in almost all datasets/settings, but the table shows that AMULET-DA is worse than I/O on WILDFEEDBACK with both ≥4 and ≥7 human turns (72.7 vs. 75.1 and 72.8 vs. 75.0), and AMULET-MAXIM is below both baselines in most settings. The individual component results should be reported honestly, and the central empirical claim should be based on the jury methods, which do show consistent improvements.
minor comments (5)
  1. [Limitations] There is a duplicated phrase in the Limitations section: 'we rerun AMULET-DA with HH-TEST on GPT-4O. We find that the second run results in an accuracy of yields an accuracy of 58%' should read 'results in an accuracy of 58%'.
  2. [§4.2] The sentence 'we use refer to prior work' contains a grammatical error; it should be 'we refer to prior work'.
  3. [Appendix F] In the AMULET-DA prompt, the numbered list of communicative dimensions jumps from (4) Time Management to (6) Own Communication Management, omitting (5); the numbering should be sequential.
  4. [Appendix D vs. Table 12] Appendix D reports the final NECTAR evaluation set size as 6,531, while Table 12 lists 6,513 instances; these numbers should be reconciled.
  5. [§3.2, Table 1] The statement that the chosen response satisfies more maxims than the rejected response in '~80% of the instances within both categories' is not accurate for all cells; for example, on HH-TEST with same DA, the chosen satisfies more maxims in 15.9/22.8 ≈ 69.7% of instances, not approximately 80%.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the accuracy claims are tested against external human preference labels and the DA/maxim analysis is not used to define the evaluation metric.

full rationale

The paper's central claim is that AMULET-DA and AMULET-MAXIM prompts improve LLM-judge accuracy on multi-turn preference data. This claim is tested against external human preference labels (chosen/rejected) from ANTHROPIC, WILDFEEDBACK, and NECTAR (Section 4.1, Table 2), so the evaluation does not reduce to the framework's own outputs. The DA and MAXIM annotations are intermediate reasoning artifacts produced by the same GPT-4O model that issues the final judgment, and the Section 3 descriptive statistics (e.g., 'humans change their dialog acts ... ~73% of the time') are computed without large-scale human validation of those annotations; these are validity and interpretability limitations, not circular reductions, because the final accuracy is still measured against independently collected preference labels and no equation or fitted parameter defines the prediction in terms of the target. The paper cites Zhao et al. (2024), on which coauthor Xiang Ren appears, for real-world conversation motivation and dataset selection, but that citation is not load-bearing for the framework's central accuracy result and the cited work is an external resource. The absence of a length-matched generic-reasoning control is a genuine experimental design concern: it means the linguistic constructs may not be the active ingredient behind the accuracy gains. However, that is a question of causal attribution and experimental control, not circularity: the AMULET prompts are not constructed from the test labels, and the observed predictions are not equivalent to the inputs by definition. The Limitations section also candidly acknowledges single-run experiments, possible hallucination, and the lack of certainty about training-data contamination; these are empirical caveats, not circular steps. No self-definitional equation, fitted-input-renamed-as-prediction, or author-imported uniqueness theorem appears in the derivation chain. Under the stated rules, the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The framework introduces no fitted model parameters. The hand-chosen dataset filters and prompt design choices are listed. The theoretical assumptions that the dialog-act taxonomy and the twelve maxims are valid and sufficient for preference judgments, and that the auto-generated annotations are accurate enough, are stated as domain assumptions.

free parameters (3)
  • Minimum human turns threshold = 4 (7 in subsets)
    Hand-chosen filter; conversations with fewer than 4 human turns are removed (Appendix D), altering dataset difficulty and comparability.
  • WILDFEEDBACK turn length cap = 300 words
    Instances with any human or assistant turn longer than 300 words are dropped (Appendix D), changing the composition of the WildFeedback evaluation set.
  • NECTAR instance cap = 100000
    Only the first 100,000 NECTAR instances are considered (Appendix D), a manual truncation controlling evaluation size.
assumptions (5)
  • domain assumption The Bunt (2019) dialog-act taxonomy, with its dimensions and functions, is a valid and sufficient representation of communicative intents in human-assistant conversations.
    Used throughout the AMULET-DA prompt (Appendix F); the effectiveness of DA-based judgment depends on this taxonomy capturing the relevant distinctions.
  • domain assumption The twelve maxims from Miehling et al. (2024) are the right set of conversational principles to differentiate good from bad responses for preference judgments.
    Adopted in the AMULET-MAXIM prompt (Appendix G); if the maxim set misses key quality dimensions, the method's differentiation power is limited.
  • domain assumption GPT-4O's automatically generated DA and MAXIM annotations are sufficiently accurate to support both the descriptive analyses (Section 3) and the final judgments.
    Only 30 conversations (194 turns) and 7 conversations (84 maxim judgments) are manually checked (Appendix H.1); the descriptive claims about human intent changes rely entirely on these model-generated labels.
  • domain assumption The preference labels in ANTHROPIC, WILDFEEDBACK and NECTAR are reliable ground truth and are not contaminated by the evaluated models.
    The authors state contamination risk is low but cannot rule out GPT-4O having seen HH-TRAIN (Limitations section).
  • domain assumption The two-vote position-swapped aggregation requires both votes to agree for a correct judgment; ties are counted as losses in the accuracy metric.
    This is a conservative design choice; the reported accuracies are sensitive to this definition (Section 4.1, Table 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries." pith.science (2026). https://pith.science/paper/YOSWOSHT

@misc{pith2026250520451,
  author       = {Pith},
  title        = {Pith review of: Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YOSWOSHT}},
  note         = {Machine review of arXiv:2505.20451}
}
read the original abstract

Today, large language models are widely used as judges to evaluate responses from other language models. Hence, it is imperative to benchmark and improve these LLM-judges on real-world language model usage: a typical human-assistant conversation is lengthy, and shows significant diversity in topics, intents, and requirements across turns, e.g. social interactions, task requests, feedback. We present Amulet, a framework that leverages pertinent linguistic concepts of dialog-acts and maxims to improve the accuracy of LLM-judges on preference data with complex, multi-turn conversational context. Amulet presents valuable insights about (a) the communicative structures and intents present in the conversation (dialog acts), and (b) the satisfaction of conversational principles (maxims) by the preference responses, and uses them to make judgments. On four challenging datasets, Amulet shows that (a) humans frequently (60 to 70 percent of the time) change their intents from one turn of the conversation to the next, and (b) in 75 percent of instances, the preference responses can be differentiated via dialog acts and/or maxims, reiterating the latter's significance in judging such data. Amulet can be used either as a judge by applying the framework to a single LLM, or integrated into a jury with different LLM judges; our judges and juries show strong improvements on relevant baselines for all four datasets.

Figures

Figures reproduced from arXiv: 2505.20451 by the authors.

Figure 1
Figure 1. Real-world language model usage typically in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. AMULET-DA uses dialog-acts to analyze communicative structures in the conversation. In the example above, the second human turn “Okay thanks. And what are some typical forms required?” has the structures of feedback/positive, social/thanking and task/question. AMULET-MAXIM analyzes which conversational principles are satisfied by the preference responses. In the example above, Response-1 is better than Response-2 at… view at source ↗
Figure 3
Figure 3. Graphs for Section 3.1: (a) Frequency of most common functions in WILDFEEDBACK, (b) Distribution of #conversations where the human turns’ #DA′ s ≥ x for WILDFEEDBACK, (c) % of consecutive human turns with different DA’s, (d) % of consecutive assistant turns with different DA’s when the corresponding consecutive human DA’s are different. (acronyms, eg:- PQ is Propositional Question, SQ is Set Question, etc. in Append… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: For each maxim on the x-axis, we measure the % of conversations in the dataset where (a) the chosen [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Voting pipeline in AMULET-LM-JURY and AMULET-RM-JURY. which response is better (Rpref ∈ {R1, R2}); the W-EXPL judge outputs Rpref , as well as a natural language explanation for the same. We design these prompts based on prior works such as Zheng et al. (2023); Lee et …
Figure 6
Figure 6. Figure 6: Frequency of most common functions in HH-T [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]
Figure 7
Figure 7. Figure 7: Frequency of most common dimensions in HH-T [PITH_FULL_IMAGE:figures/full_fig_p029_7.png]
Figure 8
Figure 8. Figure 8: #DA′ s ≥ x for human turns for HH-TRAIN, HH-TEST and NECTAR 29 [PITH_FULL_IMAGE:figures/full_fig_p029_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

81 extracted references · 34 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  3. [3]

    Afra Feyza Akyurek, Ekin Akyurek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, and Niket Tandon. 2023. https://doi.org/10.18653/v1/2023.acl-long.427 RL 4 F : Generating natural language feedback with reinforcement learning for repairing model outputs . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    doi:10.18653/v1/2023.acl-long.427 2023

  4. [4]

    James Allen and Mark Core. 1997. Draft of damsl: Dialog act markup in several layers

    work page 1997

  5. [5]

    Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf. Accessed: (May 12, 2024)

    work page 2024

  6. [6]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022 a . Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  7. [7]

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022 b . Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073

    arXiv 2022

  8. [8]

    Su Lin Blodgett, Solon Barocas, Hal Daum \'e III, and Hanna Wallach. 2020. https://doi.org/10.18653/v1/2020.acl-main.485 Language (technology) is power: A critical survey of `` bias '' in NLP . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454--5476, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.485 2020

Show all 81 references
  1. [9]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

    1952

  2. [10]

    Harry Bunt. 2011. Multifunctionality in dialogue. Computer Speech & Language, 25(2):222--245

    2011

  3. [11]

    Harry Bunt. 2019. Guidelines for using iso standard 24617-2

    2019

  4. [12]

    Bao Chen, Yuanjie Wang, Zeming Liu, and Yuhang Guo. 2023. Automatic evaluate dialogue appropriateness by using dialogue act. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7361--7372

    2023

  5. [13]

    Guiming Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301--8327

    2024

  6. [14]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  7. [15]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, and 1 others. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Con...

  8. [16]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. https://arxiv.org/abs/2310.01377 Ultrafeedback: Boosting language models with high-quality feedback . Preprint, arXiv:2310.01377

    2023 arXiv

  9. [17]

    Luigi Daniele and Suphavadeeprasit. 2023. https://huggingface.co/datasets/LDJnr/Capybara Amplify-instruct: Synthetically generated diverse multi-turn conversations for efficient llm training. arXiv preprint arXiv:(coming soon)

    2023

  10. [18]

    Qingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui, and Furu Wei. 2024 a . https://arxiv.org/abs/2410.06961 Self-boosting large language models with synthetic preference data . Preprint, arXiv:2410.06961

    2024 arXiv

  11. [19]

    Yijiang River Dong, Tiancheng Hu, and Nigel Collier. 2024 b . https://arxiv.org/abs/2406.11657 Can llm be a personalized judge? Preprint, arXiv:2406.11657

    2024 arXiv

  12. [20]

    Nicolai Dorka. 2024. Quantile regression for distributional reward models in rlhf. arXiv preprint arXiv:2409.10164

    2024 arXiv

  13. [21]

    Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V -usable information. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988--6008. PMLR

    2022

  14. [22]

    Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. https://doi.org/10.18653/v1/2023.acl-long.656 From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models . In Proceedings of the 61st An...

    2023 doi

  15. [23]

    R Frederking. 1996. Grice’s maxims: do the right thing. Frederking, RE

    1996

  16. [24]

    Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556--6576

    2024

  17. [25]

    Herbert Paul Grice. 1975. Logic and conversation. Syntax and semantics, 3:43--58

    1975

  18. [26]

    Umang Gupta, Jwala Dhamala, Varun Kumar, Apurv Verma, Yada Pruksachatkun, Satyapriya Krishna, Rahul Gupta, Kai-Wei Chang, Greg Ver Steeg, and Aram Galstyan. 2022. https://doi.org/10.18653/v1/2022.findings-acl.55 Mitigating gender bias in distilled language models via counterfa...

    2022 doi

  19. [27]

    The application of grice maxims in conversation: A pragmatic study

    Md Mahroof Hossain. The application of grice maxims in conversation: A pragmatic study. Journal of English Language Teaching and Applied Linguistics, 3(10):32--40

  20. [28]

    Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024. Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval, pages 364--381. Springer

    2024

  21. [29]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint ar...

    2023 arXiv

  22. [30]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  23. [31]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1--38

    2023

  24. [32]

    Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren. 2021. https://doi.org/10.18653/v1/2021.naacl-main.296 On transferability of bias mitigation effects in language model fine-tuning . In Proceedings of the 2021 Conference of...

    2021 doi

  25. [33]

    Jaehun Jung, Faeze Brahman, and Yejin Choi. 2024. Trust or escalate: Llm judges with provable guarantees for human agreement. arXiv preprint arXiv:2407.18370

    2024 arXiv

  26. [34]

    Atoosa Kasirzadeh and Iason Gabriel. 2023. In conversation with artificial intelligence: aligning language models with human values. Philosophy & Technology, 36(2):27

    2023

  27. [35]

    Prometheus: Inducing fine-grained evaluation capability in language models

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and 1 others. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Rep...

  28. [36]

    Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024. https://arxiv.org/abs/2404.16019 The prism alignment project: What participatory, repr...

    2024 arXiv

  29. [37]

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics ACL 2024, pages 517--545

    2024

  30. [38]

    Lea Krause and Piek TJM Vossen. 2024. The gricean maxims in nlp-a survey. In Proceedings of the 17th International Natural Language Generation Conference, pages 470--485

    2024

  31. [39]

    Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2025. https://arxiv.org/abs/2505.06120 Llms get lost in multi-turn conversation . Preprint, arXiv:2505.06120

    2025 arXiv

  32. [40]

    Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. https://arxiv.org/abs/2406.18629 Step-dpo: Step-wise preference optimization for long-chain reasoning of llms . Preprint, arXiv:2406.18629

    2024 arXiv

  33. [41]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2025. https://aclanthology.org/2025.findings-naacl.96/ R eward B ench: Eva...

    2025

  34. [42]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and 1 others. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In Forty-first Internationa...

  35. [43]

    Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, and Youngjae Yu. 2024. Aligning large language models by on-policy self-judgment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2024

  36. [44]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, and 1 others. 2024. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594

    2024

  37. [45]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

    2023

  38. [46]

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations

  39. [47]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2022 doi

  40. [48]

    Zachary C Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31--57

    2018

  41. [49]

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451

    2024 arXiv

  42. [50]

    Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. 2024. Uncertainty-aware reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847

    2024 arXiv

  43. [51]

    Shikib Mehri, Jinho Choi, Luis Fernando D'Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila, Dilek Hakkani-Tur, Zekang Li, Verena Rieser, and 1 others. 2022. Report from the nsf future directions workshop on automatic evaluation of dialog: Research directions ...

    2022 arXiv

  44. [52]

    Erik Miehling, Manish Nagireddy, Prasanna Sattigeri, Elizabeth Daly, David Piorkowski, and John Richards. 2024. Language models in dialogue: Conversational maxims for human-ai interactions. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14420--14437

    2024

  45. [53]

    Xiaoyu Tan Minghao Yang, Chao Qu. 2024. [https://huggingface.co/infly/INF-ORM-Llama3.1-70B](https://huggingface.co/infly/INF-ORM-Llama3.1-70B) Inf-orm-llama3.1-70b

    2024

  46. [54]

    Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2023. Recent advances in deep learning based dialogue systems: A systematic survey. Artificial intelligence review, 56(4):3055--3155

    2023

  47. [55]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing sys...

    2022

  48. [56]

    Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. 2020. https://doi.org/10.18653/v1/2020.acl-main.333 Towards holistic and automatic evaluation of open-domain dialogue generation . In Proceedings of the 58th Annual Meeting of the Association for Computa...

    2020 doi

  49. [57]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  50. [58]

    Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. 2024. Offsetbias: Leveraging debiased data for tuning evaluators. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1043--1067

    2024

  51. [59]

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.97 Large language models are effective text rankers with pairwise...

    2024 doi

  52. [60]

    Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, SM Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, and Amitava Das. 2023. The troubling emergence of hallucination in large language models-an extensive definition, quantification, and prescriptive remediations....

    2023

  53. [61]

    Oscar Sainz, Jon Campos, Iker Garc \' a-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023,...

    2023

  54. [62]

    Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingu...

    2024

  55. [63]

    Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, and Chelsea Finn. 2025. https://arxiv.org/abs/2502.19312 Fspo: Few-shot preference optimization of synthetic preference data in llms elicits effective personalization to real...

    2025 arXiv

  56. [64]

    Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. https://doi.org/10.18653/v1/P19-1159 Mitigating gender bias in natural language processing: Literature review . In Proceeding...

    2019 doi

  57. [65]

    Gemma Team. 2024 a . https://doi.org/10.34740/KAGGLE/M/3301 Gemma

    2024 doi

  58. [66]

    Qwen Team. 2024 b . https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  59. [67]

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796

    2024 arXiv

  60. [68]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024 a . Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP

    2024

  61. [69]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.511 Large language models are not fair evaluators . In Proceedings of the 62nd Annual Meeting o...

    2024 doi

  62. [70]

    Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. https://doi.org/10.18653/v1/2021.eancs-1.3 A comprehensive assessment of dialog evaluation metrics . In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15--33, Online. Association for...

    2021 doi

  63. [71]

    Lei Yu, Meng Cao, Jackie Chi Kit Cheung, and Yue Dong. 2024. Mechanistic understanding and mitigation of language model non-factual hallucinations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7943--7956

    2024

  64. [72]

    Chen Zhang, Luis Fernando D’Haro, Qiquan Zhang, Thomas Friedrichs, and Haizhou Li. 2022. Fined-eval: Fine-grained automatic dialogue-level evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3336--3355

    2022

  65. [73]

    Shuo Zhang, Liangming Pan, Junzhou Zhao, and William Yang Wang. 2023 a . Mitigating language model hallucination with interactive question-knowledge alignment. arXiv preprint arXiv:2305.13669, 3

    2023 arXiv

  66. [74]

    Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. 2023 b . Alleviating hallucinations of large language models through induced hallucinations. arXiv preprint arXiv:2312.15710

    2023 arXiv

  67. [75]

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. https://arxiv.org/abs/2405.01470 Wildchat: 1m chatgpt interaction logs in the wild . Preprint, arXiv:2405.01470

    2024 arXiv

  68. [76]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697--12706. PMLR

    2021

  69. [77]

    Lmsys-chat-1m: A large-scale real-world llm conversation dataset

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, and 1 others. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. In The Twelfth International Conference on Learning Representations

  70. [78]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  71. [79]

    Judgelm: Fine-tuned large language models are scalable judges

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations

  72. [80]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  73. [81]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.