Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

This paper argues that generic 'improved' prompt templates can hurt LLM task accuracy, so prompt edits should be treated as regression risks and tested on task-specific suites before deployment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:42 UTC pith:OEKNMEPM

load-bearing objection Useful practical warning with reproducible code; the regression phenomenon is real but the paper overclaims the mechanism and has an internal data contradiction. the 4 major comments →

arxiv 2601.22025 v2 pith:OEKNMEPM submitted 2026-01-29 cs.CL cs.AIcs.IRcs.SE

When Generic Prompt Improvements Hurt: Evaluation-Driven Iteration for LLM Applications

Daniel Commey This is my paper
classification cs.CL cs.AIcs.IRcs.SE
keywords LLM evaluationprompt engineeringregression testingretrieval-augmented generationprompt sensitivityevaluation-driven iterationstructured extractioninstruction-following
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to show that prompt 'improvements' are not monotonic: what helps one LLM task can hurt another. Using a small, reproducible local evaluation harness with two open-weight models, it finds that adding generic helpfulness rules degraded structured extraction pass rate from 100% to 90% and RAG citation/content compliance from 93.3% to 80% for Llama 3, while improving instruction-following. A four-condition ablation attributes the degradation to generic rules conflicting with task-specific constraints, not to the system wrapper itself. The broader point is that teams should validate every prompt change against a task-specific golden suite rather than assume best-practice templates help. The paper therefore proposes the Minimum Viable Evaluation Suite (MVES) as a tiered standard for doing that validation.

Core claim

On the paper's own terms, the central discovery is that a generic 'improved' prompt template trades off behaviors: it can help open-ended instruction-following while degrading structured tasks. In local experiments with deterministic decoding, replacing task-specific extraction prompts with a generic system wrapper plus explicit rules lowered Llama 3's extraction all-pass rate by 10 percentage points and its RAG compliance by 13.3 points. The ablation's key result is that adding the system wrapper alone changed nothing — the regressions appeared only when generic rules were appended to the user prompt, where they conflicted with task-specific grounding and JSON-format constraints. This is of

What carries the argument

The carrying mechanism is a four-condition ablation (baseline, baseline plus system wrapper, baseline plus generic rules, full generic prompt) run over three small curated test suites: extraction, RAG citation compliance, and instruction-following. The comparison isolates the source of the regression: the system wrapper is inert, and the degradation comes from generic helpfulness/verbosity rules colliding with task-specific constraints. Around this, the paper builds the Minimum Viable Evaluation Suite (MVES), a tiered structure linking application type to failure modes, metrics, artifacts, and validation evidence, organized in a Define-Test-Diagnose-Fix loop.

Load-bearing premise

The small hand-built suites (20 extraction, 15 RAG, 15 instruction cases) and the hand-chosen baseline and improved prompts are representative enough that the measured pass-rate shifts reflect a general phenomenon rather than small-sample noise or prompt-selection artifacts.

What would settle it

Run the same four-condition ablation on a suite of 400-600 cases per condition, as the paper's own power guideline recommends for detecting 5% differences; if the generic-rules condition stops degrading extraction and RAG compliance at scale, the central non-monotonicity claim would not generalize beyond the paper's selected cases. The raw logs and scripts are provided so this can be checked directly.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Prompt changes should be treated like code changes: run the task-specific suite before deployment, because the same edit can improve one behavior and break another.
  • Small golden sets (tens of cases) can surface regressions that informal spot checks miss, provided they cover the task's main failure modes.
  • Generic system wrappers are not automatically harmful; the harm in these experiments came from generic rules appended to the user prompt conflicting with task-specific constraints.
  • Explicit output-contract prompts (JSON-only, cite-only-provided-sources) can improve strict extraction and grounding even when they feel restrictive.
  • Instruction-following tasks can benefit from generic rule scaffolding, so the same template may be right for one module and wrong for another.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the conflict mechanism generalizes, the same non-monotonicity should appear for other 'helpfulness' additions such as role-priming sentences or few-shot examples that imply verbosity, not just system wrappers.
  • The all-pass-rate framing likely understates production risk: a prompt that degrades RAG compliance by 13 points on 15 cases could produce a much higher rate of uncited or unsupported claims at scale, where edge cases dominate.
  • A testable extension is to run the identical ablation on domain-stratified suites of 400-600 cases per condition; the paper's own power guideline implies that would determine whether the observed trade-off holds beyond these hand-picked examples.
  • One could also instrument each suite per quality dimension (grounding vs. helpfulness vs. format) to predict which template edits will trade off against which dimensions, turning the finding into a diagnostic rather than a warning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This technical report proposes the Minimum Viable Evaluation Suite (MVES), a tiered framework for application-level LLM evaluation, and combines it with a substantial synthesis of test-set design, metrics, RAG evaluation, and LLM-as-judge methods. The empirical core (Section 12) compares a task-specific baseline prompt with a generic “improved” prompt across three small suites (extraction 20, RAG 15, instruction 15) using Llama 3 8B and Qwen 2.5 7B under Ollama. The paper reports non-monotonic effects: the generic prompt degrades extraction and RAG pass rates in several conditions while improving instruction-following, and a four-condition ablation is used to attribute the degradation to generic rules rather than the system wrapper. All artifacts, scripts, and raw logs are said to be in a public repository.

Significance. The practical message — that generic prompt changes can trade off one task against another and should be regression-tested — is a valuable and broadly concordant existence claim. The paper's strengths are its reproducible local harness, explicit failure taxonomy, detailed LLM-as-judge bias treatment, and an unusually honest set of threats-to-validity statements, including an admission that the suites are small and model coverage narrow. If the reported effects are corrected and re-statisticized, the paper would be a useful practitioner-oriented demonstration rather than a decisive benchmark study. At present, however, the quantitative headline, the mechanistic conclusion, and the failure analysis contain internal contradictions that prevent the results from being accepted as stated.

major comments (4)
  1. [Abstract vs. §12.4 and Table 10] The abstract claims the largest observed decline is Qwen 2.5 on RAG from 26/30 to 9/30 when generic rules are appended to the user prompt. No result in the body supports this. Table 10 reports Qwen RAG at 93.3% (baseline) and 86.7% (improved), and Table 13 reports Qwen RAG A/B/C at 93.3% and D at 86.7%. There is no condition with 26/30 or 9/30. This is a direct contradiction in the paper's central quantitative claim and must be corrected before any further review.
  2. [§12.9, §14.4, Table 13] The mechanism attribution is not supported by the RAG ablation. Table 13 shows Llama RAG A=93.3%, B=93.3%, C=93.3%, D=80.0%, and Qwen RAG A/B/C=93.3%, D=86.7%. If generic rules alone caused the RAG regression, condition C should drop; it does not. The regression appears only when the wrapper and rules are combined in D. Thus the conclusion that “generic rules, not system wrappers, cause regressions” is unsupported for the RAG suite, which is one of the two headline results. The extraction result does show a rules effect (C drops 100→90 for Llama), but the RAG result does not. The prompt texts for C and D are not provided in the paper, so the reader cannot verify whether the rule wording is identical across conditions. The authors should either revise the mechanistic claim to “the combined wrapper+rules prompt caused RAG degradation” or supply the actual prompts and add an ablation that i
  3. [§12.4, §12.9, §5.7, §12.10] The effect sizes are stated as precise percentage-point declines despite being based on 15–20 cases per suite, and the paper itself states in §5.7 that detecting a 5% absolute difference requires 400–600 cases. For example, Llama RAG at 93.3%→80% is 14/15→12/15, and Qwen RAG at 93.3%→86.7% is 14/15→13/15. These are two- or three-case shifts and are well within binomial sampling variability. The N=5 repetitions in the ablation do not help because the runs are deterministic and identical. The authors should report raw counts, confidence intervals, or an exact paired test (e.g., McNemar) and, more generally, frame the results as qualitative existence demonstrations rather than calibrated effect sizes.
  4. [Table 11 vs. Tables 10 and 13] The failure categorization for Llama RAG is inconsistent with the reported pass rates. Table 11 lists 2/15 RAG failures under the improved prompt, which would yield a pass rate of 13/15 ≈ 86.7%, but Tables 10 and 13 report 80% (12/15). The table either omits a third failure category or the pass-rate numbers are incorrect. Since the failure table is used to argue that generic helpfulness pressure causes unsupported claims, this discrepancy undermines the diagnostic narrative and should be reconciled.
minor comments (5)
  1. [Abstract, §12.4] The abstract says “expanded 30-case-per-suite ablations,” but the suites contain 20, 15, and 15 cases. Clarify whether this refers to 30 total conditions or something else.
  2. [§12.9] The statement that the system wrapper has “no effect” (A vs. B) is an absence-of-evidence claim given the sample size. Rephrase as “no effect was observed in this suite” and avoid asserting a true null.
  3. [References] Reference [13] lists the author as “Chin-Yun Lin”; the correct name is Chin-Yew Lin. Also check the ROUGE citation details.
  4. [Appendix A.2] The helpfulness rubric contains the phrase “hallmarks hallucinations”; this should be “hallucinates” or “contains hallucinations.”
  5. [Table 10] Add raw counts (e.g., 14/15) to the percentage cells. Given the small suite sizes, percentages alone overstate precision and make paired comparisons harder to verify.

Circularity Check

0 steps flagged

No significant circularity: the empirical prompt-comparison and ablation results are not derived from their inputs by construction.

full rationale

The paper's central claim—that generic prompt additions can degrade structured-task performance while improving instruction-following—is supported by direct, reproducible experiments comparing hand-specified prompt conditions on fixed suites. No parameter is fitted to the reported outcomes and then renamed a prediction; the pass rates in Tables 10 and 13 are measured outputs, not consequences of the prompt definitions. The four-condition ablation is a controlled comparison, and even if its causal interpretation is debatable (the RAG degradation appears only in condition D, not C), that is a correctness/statistical-power concern, not circularity. The MVES thresholds are explicitly labeled as heuristics ('these are heuristics derived from general-purpose RAG'), and the paper's own threats-to-validity section calls the results 'existence proofs' rather than universal quantitative claims. There are no self-citations bearing on the load-bearing argument, no imported uniqueness theorem, and no renamed known result. The derivation chain is therefore self-contained with respect to its empirical inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

Central claim rests on small hand-built suites, hand-chosen prompts, manual failure coding, deterministic local inference, and the assumption that two quantized open models represent general behavior. No free parameters are fitted; MVES thresholds are heuristics rather than fitted values, and no new physical entities are introduced.

axioms (5)
  • domain assumption Small hand-curated suites (20 extraction / 15 RAG / 15 instruction cases) are sufficient to detect prompt regressions without confidence intervals.
    §12.10 admits small suite; §5.7 states 400-600 cases needed for 5% differences, yet the paper reports 10-13 pp differences on 15-20 cases and no CIs. This assumption underlies all quantitative claims.
  • domain assumption Manual failure categorization (markdown wrapper, unsupported claim, refusal noncompliance) is accurate and complete enough to identify the mechanism.
    §12.6 describes hand-categorized failures; the statement that generic rules (not wrapper) cause degradation relies on this coding.
  • ad hoc to paper The baseline prompts are representative task-specific best practice and the 'improved' prompt is a fair generic improvement, not a strawman.
    §12.2 constructs both prompt sets by hand; the extraction baseline already contains the strong JSON-only contract, so adding generic helpful rules is loaded to produce conflict.
  • domain assumption Deterministic decoding at temperature=0 eliminates meaningful output variance, so pass/fail outcomes observed once (or five times) transfer to other runs.
    §12.1/§12.9 report deterministic outputs on the author's local stack; this assumption may fail on other hardware/software stacks as §1.1 itself notes.
  • domain assumption The models and quantization (Q4_K_M) represent general LLM prompt sensitivity; cloud models may differ.
    Threats to validity in §12.10 explicitly limit model coverage; the existence proof generalizes only if model class behavior is representative.

pith-pipeline@v1.3.0-alltime-deepseek · 24432 in / 12883 out tokens · 132020 ms · 2026-08-03T06:42:44.636648+00:00 · methodology

0 comments
read the original abstract

Evaluating Large Language Model (LLM) applications differs from conventional software testing because outputs are probabilistic, semantically variable, and sensitive to prompt and model changes. This technical report proposes the Minimum Viable Evaluation Suite (MVES), an audit-oriented structure for application-level LLM evaluation. MVES links application categories to failure modes, metrics, required artifacts, and validation evidence across general LLM applications, retrieval-augmented systems, and agentic workflows. We pair the framework with a reproducible local evaluation harness covering structured extraction, RAG citation/content-compliance, and instruction-following checks. Using Ollama with Llama 3 8B Instruct and Qwen 2.5 7B Instruct, we evaluate five prompt conditions over expanded 30-case-per-suite ablations. The results show that, in the tested local conditions, generic prompt additions do not produce monotonic improvements: stronger output-contract prompts improve strict extraction for both models, while RAG citation/content-compliance declines under some generic-rule conditions. The largest observed decline occurs for Qwen 2.5 on RAG when generic rules are appended to the user prompt, from 26/30 to 9/30. These findings support evaluation-driven prompt iteration: prompt changes should be treated as potential regression risks and tested against task-specific suites before deployment. The accompanying repository contains the test suites, prompt variants, evaluation harness, raw result logs, and scripts needed to reproduce the reported local ablations.

Figures

Figures reproduced from arXiv: 2601.22025 by Daniel Commey.

Figure 1
Figure 1. Figure 1: Evaluation pipeline overview: inputs flow through the application to produce model [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Mi-Memory: A Lifecycle Memory Framework for Personal AI

    cs.AI 2026-07 conditional novelty 5.0

    Mi-Memory proposes a four-role lifecycle framework for personal AI memory with an audit contract of typed evidence, traces, strategy artifacts, and rollback records; modules are benchmarked separately, but the contrac...

Reference graph

Works this paper leans on

42 extracted references · 19 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Constitu- tional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    Ragas: Automated evaluation of retrieval augmented generation.arXiv preprint arXiv:2309.15217, 2023

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation.arXiv preprint arXiv:2309.15217, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Ka- davath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Pith/arXiv arXiv 2022

  4. [4]

    Language model evaluation harness.https://github.com/EleutherAI/lm-evaluation-harness, 2023

    Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. Language model evaluation harness.https://github.com/EleutherAI/lm-evaluation-harness, 2023

    2023

  5. [5]

    On calibration of modern neural networks.International Conference on Machine Learning, pages 1321–1330, 2017

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks.International Conference on Machine Learning, pages 1321–1330, 2017

    2017

  6. [6]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. arXiv version; later appeared in 2021 venues

    Pith/arXiv arXiv 2009

  7. [7]

    Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks.arXiv preprint arXiv:2305.10160, 2023

    Alon Jacovi, Avi Caciularu, Jonathan Mamou, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks.arXiv preprint arXiv:2305.10160, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Survey of hallucination in natural language generation.ACM Computing Surveys, 55(12):1–38, 2023

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.ACM Computing Surveys, 55(12):1–38, 2023

    2023

  9. [9]

    Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022

    SauravKadavath, TomConerly, AmandaAskell, TomHenighan, DawnDrain, EthanPerez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022

    Pith/arXiv arXiv 2022

  10. [10]

    Computing krippendorff’s alpha-reliability.Departmental Papers (ASC), 2011

    Klaus Krippendorff. Computing krippendorff’s alpha-reliability.Departmental Papers (ASC), 2011

    2011

  11. [11]

    Retrieval- augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing Systems, 33:9459–9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing Systems, 33:9459–9474, 2020

    2020

  12. [12]

    Holistic evaluation of language models.Transactions on Machine Learning Research, 2023

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models.Transactions on Machine Learning Research, 2023

    2023

  13. [13]

    Rouge: A package for automatic evaluation of summaries.Text Summa- rization Branches Out, pages 74–81, 2004

    Chin-Yun Lin. Rouge: A package for automatic evaluation of summaries.Text Summa- rization Branches Out, pages 74–81, 2004

    2004

  14. [14]

    Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958, 2022

    Pith/arXiv arXiv 2022

  15. [15]

    On faithfulness and factuality in abstractive summarization.arXiv preprint arXiv:2005.00661, 2020

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization.arXiv preprint arXiv:2005.00661, 2020. 36

    Pith/arXiv arXiv 2005

  16. [16]

    Factscore: Fine-grained atomic evalu- ation of factual precision in long form text generation.arXiv preprint arXiv:2305.14251, 2023

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evalu- ation of factual precision in long form text generation.arXiv preprint arXiv:2305.14251, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Openai evals.https://github.com/openai/evals, 2023

    OpenAI. Openai evals.https://github.com/openai/evals, 2023

    2023

  18. [18]

    Training language mod- els to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language mod- els to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022

    2022

  19. [19]

    Llm evaluators recognize and favor their own generations.arXiv preprint arXiv:2404.13076, 2024

    Arjun Panickssery, Samuel R Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations.arXiv preprint arXiv:2404.13076, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    Ares: An auto- mated evaluation framework for retrieval-augmented generation systems.arXiv preprint arXiv:2311.09476, 2023

    Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. Ares: An auto- mated evaluation framework for retrieval-augmented generation systems.arXiv preprint arXiv:2311.09476, 2023. arXiv version; later appeared in 2024 venues

    Pith/arXiv arXiv 2023

  22. [22]

    Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark.arXiv preprint arXiv:2310.18018, 2023

    Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de La- calle, and Eneko Agirre. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark.arXiv preprint arXiv:2310.18018, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Bleurt: Learning robust metrics for text generation.arXiv preprint arXiv:2004.04696, 2020

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation.arXiv preprint arXiv:2004.04696, 2020

    Pith/arXiv arXiv 2004

  24. [24]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.arXiv preprint arXiv:2206.04615, 2022

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.arXiv preprint arXiv:2206.04615, 2022. BIG-bench; later appeared in Transactions on Machine Learning Research (TMLR)

    Pith/arXiv arXiv 2022

  25. [25]

    Large language models are not fair evaluators.arXiv preprint arXiv:2305.17926, 2023

    Peiyi Wang, Lei Li, Liang Chen, Feifan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators.arXiv preprint arXiv:2305.17926, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675, 2020

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675, 2020

    Pith/arXiv arXiv 1904

  27. [27]

    Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Promptbench: Towards eval- uating the robustness of large language models on adversarial prompts.arXiv preprint arXiv:2306.04528, 2023

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. Promptbench: Towards eval- uating the robustness of large language models on adversarial prompts.arXiv preprint arXiv:2306.04528, 2023. 37 A Appendix A.1 Full Compatibility Checklist •Data Privacy: Ensure PII is stripped f...

    Pith/arXiv arXiv 2023

  29. [29]

    Separate retrieval and generation evaluation

  30. [30]

    Measure retrieval Recall@k and Precision@k

  31. [31]

    Evaluate faithfulness to retrieved documents

  32. [32]

    Check for correct but unsupported” responses

  33. [33]

    Verify citation accuracy and coverage

  34. [34]

    Test out-of-scope queries (information not in knowledge base)

  35. [35]

    38 A.5 LLM-as-Judge Checklist

    Monitor retrieval latency and index freshness. 38 A.5 LLM-as-Judge Checklist

  36. [36]

    Use a different model than the one being evaluated

  37. [37]

    Provide explicit rubrics in the evaluation prompt

  38. [38]

    Request chain-of-thought reasoning before scores

  39. [39]

    Randomize presentation order for comparisons

  40. [40]

    Validate scores against human judgments on a sample

  41. [41]

    Use multiple judge models where feasible

  42. [42]

    Document known biases in your report. 39