Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that much of the reported prompt sensitivity of LLMs is an artifact of heuristic evaluation, and that switching to LLM-as-a-Judge scoring makes accuracy and model rankings far more stable across paraphrased prompts.

desk verdict Argues prompt sensitivity is mostly a grading artifact; the direction is right and the human anchor gives it real weight, but the paper's own human data show genuine cross-template variation that the central claim overstates. read the letter →

arxiv 2509.01790 v1 pith:ODRAVI4C submitted 2025-09-01 cs.CL cs.AIcs.LG

Andong Hua , Kenan Tang , Chenhe Gu , Jindong Gu , Eric Wong , Yao Qin This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords promptsensitivityLLM-as-a-Judgeevaluationartifactsemanticequivalencerankingconsistencyheuristicbenchmarkstabilityparaphrasedprompts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompt sensitivity—the observation that paraphrasing a prompt changes a model's benchmark score—has been treated as a core weakness of LLMs. This paper argues that much of that sensitivity is manufactured by the grading protocol, not the model. Across 7 models, 6 benchmarks, and 12 paraphrased prompt templates, the authors compare heuristic scoring (log-likelihood over fixed options, word-F1 overlap, boxed-answer extraction) with LLM-as-a-Judge scoring, in which a judge model decides whether the prediction is semantically equivalent to the reference answer. Under judge-based scoring, accuracy variance across templates collapses—for example, Gemma-2 on ARC-Challenge goes from a standard deviation of 0.28 under heuristics to 0.005—and model rankings become consistent across prompts. Human annotation on a 50-question sample per dataset agrees with the judge, supporting the conclusion that prompt sensitivity is largely an evaluation artifact.

What carries the argument

The load-bearing object is the LLM-as-a-Judge protocol: a judge model (Gemini 2.0 Flash, cross-checked with GPT-4o-mini) receives the question, the reference answer, and the model prediction, and returns a binary judgment of whether the prediction semantically matches the reference while ignoring explanations. This replaces exact-format matching and log-likelihood scoring. Two sensitivity metrics carry the quantitative argument: the standard deviation of accuracy across the 12 prompt templates, and the mean pairwise Spearman rank correlation of model rankings across templates. The judge's binary semantic-equivalence decision is the mechanism that turns format-fragile scoring into format-inde

What would settle it

Run the same 12-template protocol on ARC-Challenge with templates deliberately engineered to force different answer registers (letter-only, single word, full sentence, quoted phrase) and score with a blind LLM judge. If the standard deviation of judge-based accuracy across these templates approaches the 0.28 heuristic value, the artifact explanation is in trouble. A cheaper probe: re-annotate the human sample with a stricter rubric requiring exact reference wording and check whether judge-human agreement drops or rankings change.

Watch

Extended reading notes

Core claim

The paper's central claim is that the large performance swings and ranking flips attributed to prompt paraphrasing mostly reflect how answers are graded, not how well models reason. Scored by log-likelihood over fixed options or by rigid extraction, any correct answer phrased differently from the reference gets marked wrong, and that inflates prompt sensitivity. Replacing that grader with an LLM judge that checks semantic equivalence produces uniformly small accuracy standard deviations across 12 prompt templates and raises the mean pairwise Spearman rank correlation of model rankings (ARC-Challenge: from about 0.30 to 0.92 for open models, and 0.95 including proprietary models). The judge i

Load-bearing premise

The load-bearing premise is that answer correctness should be defined as semantic equivalence to the reference answer, judged by an LLM and validated on 50 sampled questions from one model per benchmark; if a stricter, format-aware correctness standard is required, the observed stability is a property of that grading standard rather than of the models.

Editorial extensions

If this is right

  • Previously reported ranking flips across prompt templates should be re-examined; under semantic-equivalence grading, model rankings become nearly template-independent.
  • Benchmark scores from log-likelihood or rigid extraction can understate model capability when a correct answer is paraphrased or differently formatted.
  • LLM-as-a-Judge is a defensible default for prompt-sensitivity studies, though well-engineered task-specific heuristics (as the paper shows for MATH with symbolic simplification) can be just as stable.
  • Prompt-template stability appears to generalize across model generations: Llama-2 and Mistral, not just current models, also show compressed variance under judge scoring.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the claim holds, format sensitivity for end users is partly an interface property: applications that require machine-readable exact output still need format constraints, even though the underlying knowledge is stable.
  • The human validation covers only 50 questions per dataset from one model per benchmark; checking additional models and questions could reveal cases where the judge and annotators share a systematically lenient notion of what counts as a match.
  • A testable extension is to build adversarial template pairs that force stylistically different but semantically identical answers (single word vs. full sentence with citations) and see whether judge-based scores stay flat; if they move, some genuine sensitivity remains.
  • Because the judge is itself an LLM, its notion of semantic equivalence could be probed by swapping judge prompts or judge models; the paper's GPT-4o-mini cross-check on ARC-Challenge is one step in that direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper revisits the widely reported sensitivity of LLMs to prompt paraphrasing. The authors evaluate 7 LLMs on 6 benchmarks with 12 prompt templates per benchmark, comparing heuristic evaluation (log-likelihood scoring, word-level F1, regex extraction) with LLM-as-a-Judge evaluation. They report that heuristic methods greatly exaggerate prompt sensitivity: for example, Genma-2 on ARC-Challenge has accuracy std 0.28 under heuristics vs 0.005 under an LLM judge, and mean Spearman rank correlation across open-source models rises from 0.30 to 0.92. A human-annotation study (10,800 annotations) is offered as evidence that the LLM judge agrees with humans. The paper concludes that prompt sensitivity is largely an artifact of heuristic evaluation rather than an inherent flaw of modern LLMs.

Significance. If the central claim holds, the paper is a valuable corrective to the prompt-sensitivity literature: it would shift attention from model behavior to evaluation protocols and support wider adoption of semantic, human-aligned grading. The study has genuine strengths: it spans multiple model families, includes open-ended and multiple-choice benchmarks, uses a non-circular human anchor for judge reliability, and includes a positive control (MATH) in which a well-engineered heuristic already shows low sensitivity. The MATH control and older-model results (Appendix E) strengthen the argument that the variance reduction is not merely an artifact of recent instruction tuning. The paper is therefore potentially impactful for evaluation methodology and for interpreting prior multi-prompt studies.

major comments (4)
  1. [Section 4, Table 3] Observations 1 and 2 state that human-annotated correctness shows 'consistently high agreement' and 'minimal variation' across prompt templates. Table 3 contradicts this. Perfect agreement (all 12 templates giving the same correctness label) is 86% for ARC, 80% for OpenbookQA, 52% for GPQA-Diamond, 66% for NarrativeQA, 68% for MATH, and 88% for SimpleQA. Thus for GPQA-Diamond, nearly half of the 50 sampled questions flip correctness across templates under majority-voted human labels; NarrativeQA and MATH flip on roughly one-third. These are nontrivial real cross-template differences that exist independently of the heuristic grader. The statement 'answer correctness does not vary substantially across different prompt templates' is quantitatively contradicted by the paper's own data. The authors should report flip rates by template pair, identify whether flips concentrate in a few template
  2. [Section 4, Table 4 and Appendix D] The human validation sample is narrow: 50 questions per dataset from one model per benchmark (Gemma-2 for ARC/OpenbookQA/GPQA; GPT-4.1-mini for NarrativeQA/MATH/SimpleQA), with the three annotators seeing 12 responses per question. This can establish that the judge agrees with humans on those samples, but it does not validate the claim that judge-based stability holds across all seven evaluated models. In particular, it does not measure human-judge agreement for the models that show the largest heuristic variance, nor does it show per-template agreement. Adding human annotations for at least one more model per benchmark and reporting human-judge agreement per template would materially strengthen the inference.
  3. [Section 2.1, Section 3.2, Appendix A] The 12 templates are GPT-4o paraphrases of the original prompts, and for MATH several ({text1},{text2}) pairs are identical, differing only in few-shot examples. This is a reasonable family of paraphrastic variation, but it is not the same family used in the prior studies the paper claims to reinterpret (e.g., format-hostile variants, option-numbering changes, role-play prompts, or minimal punctuation edits). The limitation section notes stability over 100 prompts only for ARC. The inference from 'variance is small under these 12 templates' to 'previously reported sensitivity is largely an artifact' is therefore overbroad. The authors should add stress tests with intentionally brittle prompt modifications and report variance decomposed by template family, or restrict the conclusion to paraphrastic prompt variation.
  4. [Section 3.2, Figure 2, Table 2] The variance comparison is made on accuracy, a bounded scale, and many judge-based accuracies are near ceiling (e.g., Gemma-2 ARC 0.90, GPT-4.1 ARC 0.96; Table 2). Binary semantic-match labels also collapse all partial credit. Some of the variance reduction could be a mechanical consequence of a coarse binary label assigned by a lenient judge rather than evidence that model output distributions are stable. The MATH positive control helps, but it is a single dataset. A matched analysis comparing exact-match vs semantic-match labels applied by the same judge, or reporting a continuous semantic-similarity distribution, would support the causal claim that heuristic grading—not model behavior—creates the variance.
minor comments (5)
  1. [References] Several references contain formatting artifacts: 'V oronov' and 'Ko ˇcisk`y' appear with stray spaces/accents, and 'and 1 others' is used in multiple entries. These should be cleaned.
  2. [Table 3 caption] The caption reads 'Correctness of answers shows minimal variation across prompt templates,' but the Perfect Agreement column (52–88%) does not support 'minimal variation.' Please revise the caption to describe the statistic without asserting the conclusion.
  3. [Table 4] Table 4 labels NarrativeQA Human–Human kappa 0.687 and Human–LLM kappa 0.670 as 'strong consistency.' In standard interpretations these are moderate, not strong. Report confidence intervals and use calibrated language.
  4. [Appendix D] The human annotation instructions reproduced in Appendix D are very brief (binary 1/0, 'if ambiguous put 0'). To support reproducibility, include the full rubric, worked examples, and a statement on how annotators handled partially correct or paraphrased answers.
  5. [Figure 1] The caption says heuristic evaluation labels 'First World War' as 'Incorrect' for illustration purposes, but word-level F1 would give partial credit rather than fail. Please clarify whether the figure shows a thresholded score and what threshold is used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central result is anchored to external human annotations and a non-circular MATH control.

full rationale

The paper's derivation chain is not circular. Its central claim—that heuristic grading exaggerates prompt sensitivity—is an empirical comparison between two evaluation protocols, not a consequence of how either protocol is defined. The LLM-judge results are anchored to an external benchmark: human annotations on 50 questions per dataset (Section 4, Tables 3-4), with reported Fleiss/Cohen kappas. The MATH benchmark provides a genuine positive control: a symbolic/normalization heuristic (sympy) also yields low variance and high ranking correlation, matching the judge's result (Section 3.2), which rules out the concern that the variance reduction is merely an artifact of switching to an LLM judge. No parameter is fitted and then renamed a prediction; no uniqueness theorem or load-bearing self-citation is used (the only self-citation, Hua et al. 2024, is an incidental example of LLM success). The skeptical concern that Table 3 shows only moderate cross-template human agreement for GPQA (Fleiss kappa=0.67, 52% perfect agreement) undercuts the paper's strength of conclusion and should be weighed as a correctness/overclaim issue, but it is not a circular derivation: the paper's own numbers are being used against its interpretation. Similarly, the Limitations section's claim of stability at 100 prompts is asserted without shown data, which is a missing-evidence issue, not circularity. Under the stated rule that a non-finding is appropriate when the derivation is self-contained against external benchmarks, no circular step meets the required 'exhibit the reduction' bar.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no free parameters in the ML-fitting sense and no invented entities. The central measurement rests on hand-chosen analysis components: the default judge model, the template count (12 per benchmark, justified only for ARC against a 100-prompt check), and partially undocumented per-benchmark judge prompt tweaks. The normative axioms are the definition of correctness as semantic equivalence and the representativeness of GPT-4o-paraphrased templates; the human study anchors the former empirically but only for one model per benchmark, and the latter is untested against adversarial prompt variation.

free parameters (4)
  • LLM judge model = Gemini 2.0 Flash (default); GPT-4o-mini (ablation on ARC only)
    The judge is a hand-chosen component of the measurement. Robustness to judge identity is tested only on ARC-Challenge (Table 2); all other benchmarks rest on the single default judge.
  • Number of prompt templates = 12 per benchmark
    Chosen as a computational constraint (stated in Limitations). Sensitivity to this choice (12 vs 100+ prompts) is only shown for ARC-Challenge metrics; no data for the 100-prompt check is shown.
  • Per-benchmark judge prompt adjustments
    Ad hoc and described qualitatively ('minor benchmark-specific adjustments', e.g., 'Ignore all explanation' for GPQA). Not fully enumerated, so the judge's effective strictness is not held constant across benchmarks.
  • Human annotation sample = 50 questions per dataset x 1 model per dataset
    Hand-picked sample size and model assignment; the agreement numbers in Tables 3-4 rest on this sample and are not extended to the other 6 models.
assumptions (5)
  • domain assumption Semantic equivalence to the reference answer, as judged by LLM-as-a-Judge or human annotators, is the correct operational definition of answer correctness.
    Invoked in Sections 2.2 and 4. The 'artifact' conclusion follows from choosing this correctness notion over a format-aware one.
  • domain assumption The 12 GPT-4o-paraphrased templates per benchmark are representative of realistic prompt variation.
    Section 2.1. No adversarial, order-shuffled, or format-hostile templates are included; template similarity may compress measured sensitivity.
  • domain assumption Human annotator binary judgments on 50 sampled questions generalize to full benchmarks and to all 7 models under evaluation.
    Section 4. Judge alignment is validated on one model per dataset only; the main claims cover 7 models.
  • domain assumption Greedy decoding yields outputs representative of model behavior for evaluation purposes.
    Section 3.1. Sampling temperature 0 removes stochasticity, which may understate sensitivity relative to sampled decoding.
  • standard math Standard statistical tools: Spearman rank correlation, Fleiss' kappa, Cohen's kappa.
    Sections 2.3 and 4. Textbook methods with no unproved mathematical content.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs." pith.science (2026). https://pith.science/paper/ODRAVI4C

@misc{pith2026250901790,
  author       = {Pith},
  title        = {Pith review of: Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ODRAVI4C}},
  note         = {Machine review of arXiv:2509.01790}
}
read the original abstract

Prompt sensitivity, referring to the phenomenon where paraphrasing (i.e., repeating something written or spoken using different words) leads to significant changes in large language model (LLM) performance, has been widely accepted as a core limitation of LLMs. In this work, we revisit this issue and ask: Is the widely reported high prompt sensitivity truly an inherent weakness of LLMs, or is it largely an artifact of evaluation processes? To answer this question, we systematically evaluate 7 LLMs (e.g., GPT and Gemini family) across 6 benchmarks, including both multiple-choice and open-ended tasks on 12 diverse prompt templates. We find that much of the prompt sensitivity stems from heuristic evaluation methods, including log-likelihood scoring and rigid answer matching, which often overlook semantically correct responses expressed through alternative phrasings, such as synonyms or paraphrases. When we adopt LLM-as-a-Judge evaluations, we observe a substantial reduction in performance variance and a consistently higher correlation in model rankings across prompts. Our findings suggest that modern LLMs are more robust to prompt templates than previously believed, and that prompt sensitivity may be more an artifact of evaluation than a flaw in the models.

Figures

Figures reproduced from arXiv: 2509.01790 by the authors.

Figure 1
Figure 1. When provided with diverse prompt templates, LLMs provide different but semantically equivalent [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The mean and standard deviation of performance across different prompt templates. For all 6 datasets, we [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An example of a judging prompt. After filling [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 45 canonical work pages

  1. [1]

    Evaluate the choices and select the most appropriate answer.\n{question}\nThe options are as follows:\nOption A: {first_option}\nOption B: {second_option}\nOption C: {third_option}\nOption D: {fourth_option}\nYour answer should be formatted as:\n’I have chosen option [choice]’\nwhere [choice] is your selected answer.\n

    work page

  2. [2]

    Review the available options and select the one you think is correct.{question}\nAvailable answers include:\nA ) {first_option}\nB ) {second_option}\nC ) {third_option}\nD ) {fourth_option}\n\nResponse:

    work page

  3. [3]

    Select the correct answer based on your understanding.\n{question}\nPick from the following options:\n[A] {first_option}\n[B] {second_option}\n[C] {third_option}\n[D] {fourth_option}\nPlease respond with ’Option [choice]’.\n 7

    work page

  4. [4]

    Evaluate the options presented and select the most suitable.{question}\nAvailable answers:\n[A] {first_option}\n[B] {second_option}\n[C] {third_option}\n[D] {fourth_option}\n\nExpress your choice as: ’The answer is [choice].’\nwhere [choice] is your selected option.\n

    work page

  5. [5]

    Based on the question presented, choose the most fitting response.{question}\nAvailable answers are:\nA: {first_option}\nB: {second_option}\nC: {third_option}\nD: {fourth_option}\nPlease provide your response in the following format:\n’Your choice: [option]’\nwhere [option] corresponds to the letter or number you selected.\n

    work page

  6. [6]

    {first_option}\n2

    From the options below, select the response that you believe is correct.{question}\nChoices to consider:\n1. {first_option}\n2. {second_option}\n3. {third_option}\n4. {fourth_option}\nResponse:

    work page

  7. [7]

    Select your answer from the provided list of options.\n{question}\nOptions are:\nThe choice is A: {first_option}\nThe choice is B: {second_option}\nThe choice is C: {third_option}\nThe choice is D: {fourth_option}\n\nChoose your answer:

    work page

  8. [8]

    After considering the options, choose the best possible answer.{question}\nThe following choices are available:\nA: {first_option}\nB: {second_option}\nC: {third_option}\nD: {fourth_option}\nState your answer as:\n’Answer: [choice]’\n

    work page

Show all 47 references
  1. [9]

    Analyze the selections and provide your choice.\n{question}\nYour options are listed below:\nOption 1 - {first_option}\nOption 2 - {second_option}\nOption 3 - {third_option}\nOption 4 - {fourth_option}\n\nYour response:

  2. [10]

    Consider the following question and determine the right response.\n{question}\nWhich of the following answers do you prefer?\nOption 1: {first_option}\nOption 2: {second_option}\nOption 3: {third_option}\nOption 4: {fourth_option}\nI select:

  3. [11]

    Determine which option best answers the question asked.\n{question}\nPossible choices are as follows:\nOption [A] {first_option}\nOption [B] {second_option}\nOption [C] {third_option}\nOption [D] {fourth_option}\n\nFinal answer:

  4. [12]

    For NarrativeQA, we use the following 12 prompt templates:

    Identify the option that best answers the question posed.{question}\nConsider these choices:\nSelect option 1: {first_option}\nSelect option 2: {second_option}\nSelect option 3: {third_option}\nSelect option 4: {fourth_option}\n\nChoice provided: In the prompt templates, {ques...

  5. [16]

    Answer the question asconcisely as you can, using a single phrase if possible

    You are given a story, which can be either a novel or a movie script, and a question. Answer the question asconcisely as you can, using a single phrase if possible. Do not provide any explanation.\n\n Story: {context}\n\n Now, answer the question based on the story as concisel...

  6. [17]

    Provide the most accurate answer you can in a single phrase or sentence fragment

    Below is an excerpt from a mystery or thriller story, followed by a question. Provide the most accurate answer you can in a single phrase or sentence fragment. No elaboration is needed.\n\nStory: {context}\n\nExamine the situation carefully and respond.\n\nQuestion: {question}...

  7. [18]

    Respond succinctly with a phrase

    You are presented with a passage from literary fiction or cinematic writing and a comprehension question. Respond succinctly with a phrase. Avoid any additional commentary.\n\nStory: {context}\n\nAnalyze and respond concisely.\n\nQuestion: {question}\n\nAnswer:

  8. [19]

    Give your answer using only 8 a few words

    A tale from a distant world or magical land is told below, followed by a question from a curious scholar. Give your answer using only 8 a few words. No need to explain the lore.\n\nStory: {context}\n\nWhat say you?\n\nQuestion: {question}\n\nAnswer:

  9. [20]

    A question follows

    You’re reading a gritty tale from the backstreets of the city. A question follows. Keep your answer clipped, clean, and under the radar—just a phrase, no fluff.\n\nStory: {context}\n\nHere’s the case:\n\nQuestion: {question}\n\nAnswer:

  10. [21]

    One phrase, no lifelines!\n\nStory: {context}\n\nLet’s play!\n\nQuestion: {question}\n\nAnswer:

    Welcome to *Plot Points*! We’ll give you a story snippet and a question—your job is to give the fastest, most precise answer possible. One phrase, no lifelines!\n\nStory: {context}\n\nLet’s play!\n\nQuestion: {question}\n\nAnswer:

  11. [22]

    A question will now be entered into the record

    The record shows the following account. A question will now be entered into the record. Provide your answer in a short, factual phrase. No commentary permitted.\n\nStory: {context}\n\nDeposition Question:\n\nQuestion: {question}\n\nAnswer:

  12. [23]

    Answer the question

    Read the excerpt. Answer the question. Keep it short.\n\nStory: {context}\n\nQuestion: {question}\n\nAnswer:

  13. [24]

    Story fragment retrieved from Galactic Chronicles

    Accessing archive. . . Story fragment retrieved from Galactic Chronicles. A query follows. Respond with the most relevant concept or phrase. Do not explain.\n\nStory: {context}\n\n>>> QUERY:\n\nQuestion: {question}\n\n>>> RESPONSE:\n\nAnswer:

  14. [25]

    Now a little question is asked

    Once upon a time, a story was told. Now a little question is asked. Answer it kindly and briefly—just a few words will do. No need to explain why.\n\nStory: {context}\n\nHere comes the question:\n\nQuestion: {question}\n\nAnswer:

  15. [26]

    Respond with a single phrase, a shard of truth—no more, no less.\n\nStory: {context}\n\nWhisper your reply:\n\nQuestion: {question}\n\nAnswer:

    From the folds of a lyrical tale, a question emerges like morning light. Respond with a single phrase, a shard of truth—no more, no less.\n\nStory: {context}\n\nWhisper your reply:\n\nQuestion: {question}\n\nAnswer:

  16. [27]

    Narrative extracted

    Intel received. Narrative extracted. Stand by for situational query. Your task: deliver the answer in minimal terms. Do not elaborate.\n\nStory: {context}\n\nMission Query:\n\nQuestion: {question}\n\nAnswer: In the prompt templates, {context} is the context, and {question} is ...

  17. [28]

    ({empty_string}, \nAnswer:\n)

  18. [30]

    (Problem::\n, \nAnswer:\n)

  19. [31]

    (Task:\n\n, \n\nSolution:)

  20. [33]

    (Solve the following math problem:\n\n, \nAn- swer:\n)

  21. [35]

    (Problem::\n, \n\nSolution:)

  22. [36]

    (Solve the following math problem:\n\n, \n\nSolution:)

  23. [37]

    (**Problem Statement**:\n\n, \n\nSolution:)

  24. [38]

    (**Problem Statement**:\n\n, \nAnswer:\n)

  25. [39]

    Hence, while two pairs of ({text1}, {text2}) could be the same, the actual prompt template is different

    ({empty_string}, \nAnswer:\n) Since MATH uses few-shot prompting for evalu- ation, we further change the examples provided for each prompt template. Hence, while two pairs of ({text1}, {text2}) could be the same, the actual prompt template is different. For SimpleQA, we use th...

  26. [40]

    Ready your reasoning—consider the chal- lenge that follows.\n\n

  27. [41]

    Take a thoughtful pause, then craft your best response to the prompt beneath this line.\n\n

  28. [42]

    Showcase your insight by addressing the up- coming question.\n\n

  29. [43]

    You will be given:-A question-A reference (correct) answer-A model's predicted answerYour task is to judge whether the prediction matches the reference answer

    Put your analytical lens on and dive into the inquiry below.\n\n 9 You are an AI assistant that determines whether a model's prediction matches a given reference answer for a question. You will be given:-A question-A reference (correct) answer-A model's predicted answerYour ta...

  30. [44]

    Channel your inner detective: examine the next question and present your findings.\n\n

  31. [45]

    Let your knowledge shine—respond thought- fully to the statement that follows.\n\n

  32. [46]

    Engage your critical thinking skills and tackle the question that appears next.\n\n

  33. [47]

    Apply the concepts you’ve mastered to answer the forthcoming inquiry.\n\n

  34. [48]

    Use evidence and reasoning to construct your answer to the question below.\n\n

  35. [49]

    Approach the next problem with curiosity and craft a clear solution.\n\n

  36. [50]

    Fifty years

    Demonstrate what you’ve learned by address- ing the prompt that follows.\n\n B Prompts for LLM-as-a-Judge Figure 3 shows the prompt we use for LLM-as- a-Judge. For each benchmark, we make minor task-specific modifications to the judging prompt. For SimpleQA, we use the officia...

    2024

  37. [2019]

    Transactions of the Association for Com- putational Linguistics, 7:249–266

    CoQA: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249–266. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R. Bowman. 2024. ...

    2024 arXiv

  38. [2023]

    Advances in Neural Information Pro- cessing Systems, 36:46595–46623

    Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Pro- cessing Systems, 36:46595–46623. A Diverse Prompts In this section, we list the diverse prompt templates we use for each benchmark. For ARC-Challenge, GPQA, and OpenbookQA, we use the f...

  39. [2025]

    arXiv preprint arXiv:2505.09388

    Qwen3 technical report. arXiv preprint arXiv:2505.09388. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Hao- ran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Z...

    2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.