{"id":"b7c72506-0067-49e9-b300-14fcdbf9f28a","arxiv_id":"2509.01790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using LLM-as-a-Judge grading instead of heuristic answer matching dramatically reduces measured prompt sensitivity and stabilizes model rankings across prompt templates, with human annotations confirming the judge's view.","lead":"A new study argues that the famously large performance swings of LLMs under reworded prompts are mostly caused by the scoring software, not by the models themselves. When grading is done by an LLM judge validated against human checks instead of rigid answer matching, scores and model rankings stay far more stable across prompt phrasings than past work reported.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own human-annotation data show substantial cross-template variation (e.g., 48% of GPQA questions flip across templates), contradicting the claim that prompt sensitivity is largely an artifact.","rationale":"After reading the paper and the reader's verdict, I focused on the central claim that prompt sensitivity is largely an artifact of heuristic evaluation. The strongest evidence for this is the human annotation study in Section 4, which is supposed to anchor the LLM-judge as a valid ground-truth. However, Table 3 itself reveals substantial cross-template variation in human correctness judgments: GPQA-Diamond has Fleiss' kappa = 0.67 and only 52% 'perfect agreement' across 12 templates, meaning that for 48% of the 50 sampled questions, the majority-voted human label differs across templates. NarrativeQA is similar (66% perfect agreement). These are not 'consistently high' as the paper states, and they directly contradict Observation 1 that 'answer correctness does not vary substantially across different prompt templates.' This is an internal, quantitative inconsistency, not a matter of differing norms about correctness. If human annotators themselves flip their correctness judgments across prompts on a large fraction of questions, then a non-trivial part of prompt sensitivity is a real model behavior, not a grading artifact. The LLM-judge's agreement with the human majority vote (Table 4) does not eliminate this variation; it may even inherit the same template-dependent biases. Therefore, the paper's conclusion that prompt sensitivity is 'largely an artifact' is not supported by its own data. This concern is more load-bearing than the generic worry about judge validity, because it uses the paper's own reported numbers to expose an overstatement. I would still keep the verdict CONDITIONAL rather than REJECT, because the paper's direction (heuristic grading exaggerates sensitivity) may survive with revised claims and more careful analysis, and because the human sample is small (50 questions). The concrete test would be to release the raw data and recompute the flip rates; if the flip rates are as high as they appear, the paper must weaken its conclusion. The reader's weakest_assumption focused on LLM-judge validity, which is related but not identical; the reader did mention the moderate kappa in the rationale, so agreement is partial.","tokens_in":12691,"tokens_out":8249,"duration_ms":91433,"concrete_test":"Ask the authors to release the raw human annotation matrix (question × template × annotator) and compute: (1) the percentage of the 50 GPQA-Diamond and 50 NarrativeQA questions for which the majority-voted correctness label differs between any pair of templates; (2) the maximum pairwise difference in human-annotated accuracy across templates for each dataset. If the flip rate exceeds 20% or the max pairwise accuracy difference exceeds 10 percentage points on either dataset, the claim of 'minimal variation' fails. Alternatively, re-run the human study with 200+ questions per dataset and report per-template accuracy; the current 50-question sample is insufficient to support the strong conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 3 reports Fleiss' kappa for cross-template agreement of human-annotated correctness. For GPQA-Diamond, kappa = 0.67 and 'perfect agreement' (all 12 templates same label) occurs for only 52% of the 50 questions; for NarrativeQA, kappa = 0.73 with 66% perfect agreement. These are moderate, not 'consistently high' as claimed in Observation 1. Thus for nearly half of GPQA questions, human annotators (majority-voted) judged the answer correct under some templates and incorrect under others. This is direct evidence that at least some prompt sensitivity is real and not an artifact of heuristic grading. The paper's central conclusion that 'answer correctness does not vary substantially across different prompt templates' is contradicted by its own Table 3. The LLM-judge may agree with the human majority vote (Table 4), but that does not erase the cross-template variation in the human labels. Consequently, the claim that prompt sensitivity is 'largely an artifact' is overstated; the data show a non-negligible real component. This is an internal, quantitative inconsistency, not a matter of external norms.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper revisits the widely reported sensitivity of LLMs to prompt paraphrasing. The authors evaluate 7 LLMs on 6 benchmarks with 12 prompt templates per benchmark, comparing heuristic evaluation (log-likelihood scoring, word-level F1, regex extraction) with LLM-as-a-Judge evaluation. They report that heuristic methods greatly exaggerate prompt sensitivity: for example, Genma-2 on ARC-Challenge has accuracy std 0.28 under heuristics vs 0.005 under an LLM judge, and mean Spearman rank correlation across open-source models rises from 0.30 to 0.92. A human-annotation study (10,800 annotations) is offered as evidence that the LLM judge agrees with humans. The paper concludes that prompt sensitivity is largely an artifact of heuristic evaluation rather than an inherent flaw of modern LLMs.","tokens_in":12821,"tokens_out":4924,"duration_ms":59434,"significance":"If the central claim holds, the paper is a valuable corrective to the prompt-sensitivity literature: it would shift attention from model behavior to evaluation protocols and support wider adoption of semantic, human-aligned grading. The study has genuine strengths: it spans multiple model families, includes open-ended and multiple-choice benchmarks, uses a non-circular human anchor for judge reliability, and includes a positive control (MATH) in which a well-engineered heuristic already shows low sensitivity. The MATH control and older-model results (Appendix E) strengthen the argument that the variance reduction is not merely an artifact of recent instruction tuning. The paper is therefore potentially impactful for evaluation methodology and for interpreting prior multi-prompt studies.","major_comments":[{"comment":"Observations 1 and 2 state that human-annotated correctness shows 'consistently high agreement' and 'minimal variation' across prompt templates. Table 3 contradicts this. Perfect agreement (all 12 templates giving the same correctness label) is 86% for ARC, 80% for OpenbookQA, 52% for GPQA-Diamond, 66% for NarrativeQA, 68% for MATH, and 88% for SimpleQA. Thus for GPQA-Diamond, nearly half of the 50 sampled questions flip correctness across templates under majority-voted human labels; NarrativeQA and MATH flip on roughly one-third. These are nontrivial real cross-template differences that exist independently of the heuristic grader. The statement 'answer correctness does not vary substantially across different prompt templates' is quantitatively contradicted by the paper's own data. The authors should report flip rates by template pair, identify whether flips concentrate in a few template","section":"Section 4, Table 3"},{"comment":"The human validation sample is narrow: 50 questions per dataset from one model per benchmark (Gemma-2 for ARC/OpenbookQA/GPQA; GPT-4.1-mini for NarrativeQA/MATH/SimpleQA), with the three annotators seeing 12 responses per question. This can establish that the judge agrees with humans on those samples, but it does not validate the claim that judge-based stability holds across all seven evaluated models. In particular, it does not measure human-judge agreement for the models that show the largest heuristic variance, nor does it show per-template agreement. Adding human annotations for at least one more model per benchmark and reporting human-judge agreement per template would materially strengthen the inference.","section":"Section 4, Table 4 and Appendix D"},{"comment":"The 12 templates are GPT-4o paraphrases of the original prompts, and for MATH several ({text1},{text2}) pairs are identical, differing only in few-shot examples. This is a reasonable family of paraphrastic variation, but it is not the same family used in the prior studies the paper claims to reinterpret (e.g., format-hostile variants, option-numbering changes, role-play prompts, or minimal punctuation edits). The limitation section notes stability over 100 prompts only for ARC. The inference from 'variance is small under these 12 templates' to 'previously reported sensitivity is largely an artifact' is therefore overbroad. The authors should add stress tests with intentionally brittle prompt modifications and report variance decomposed by template family, or restrict the conclusion to paraphrastic prompt variation.","section":"Section 2.1, Section 3.2, Appendix A"},{"comment":"The variance comparison is made on accuracy, a bounded scale, and many judge-based accuracies are near ceiling (e.g., Gemma-2 ARC 0.90, GPT-4.1 ARC 0.96; Table 2). Binary semantic-match labels also collapse all partial credit. Some of the variance reduction could be a mechanical consequence of a coarse binary label assigned by a lenient judge rather than evidence that model output distributions are stable. The MATH positive control helps, but it is a single dataset. A matched analysis comparing exact-match vs semantic-match labels applied by the same judge, or reporting a continuous semantic-similarity distribution, would support the causal claim that heuristic grading—not model behavior—creates the variance.","section":"Section 3.2, Figure 2, Table 2"}],"minor_comments":[{"comment":"Several references contain formatting artifacts: 'V oronov' and 'Ko ˇcisk`y' appear with stray spaces/accents, and 'and 1 others' is used in multiple entries. These should be cleaned.","section":"References"},{"comment":"The caption reads 'Correctness of answers shows minimal variation across prompt templates,' but the Perfect Agreement column (52–88%) does not support 'minimal variation.' Please revise the caption to describe the statistic without asserting the conclusion.","section":"Table 3 caption"},{"comment":"Table 4 labels NarrativeQA Human–Human kappa 0.687 and Human–LLM kappa 0.670 as 'strong consistency.' In standard interpretations these are moderate, not strong. Report confidence intervals and use calibrated language.","section":"Table 4"},{"comment":"The human annotation instructions reproduced in Appendix D are very brief (binary 1/0, 'if ambiguous put 0'). To support reproducibility, include the full rubric, worked examples, and a statement on how annotators handled partially correct or paraphrased answers.","section":"Appendix D"},{"comment":"The caption says heuristic evaluation labels 'First World War' as 'Incorrect' for illustration purposes, but word-level F1 would give partial credit rather than fail. Please clarify whether the figure shows a thresholded score and what threshold is used.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful and mostly defensible point, but the current wording overclaims. The Table 3 perfect-agreement numbers are the strongest internal evidence against the 'largely artifact' framing, and the authors need to either weaken the conclusion or provide an analysis showing that the flips are small/benign and unrelated to model capability. I would not reject: the central idea is sound and the human-anchored judge evaluation is a meaningful contribution, but the load-bearing inconsistency must be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's headline is that most prompt sensitivity in leaderboard-style evals is an artifact of heuristic grading, not model instability. The direction is probably right, and the study is the most systematic test I've seen: 7 models, 6 benchmarks, 12 templates each, with an LLM judge backstopped by human annotations on a sample. The MATH result works as a genuine positive control—a strong heuristic matches the judge in low sensitivity—and the jump in mean Spearman on ARC (0.30 to 0.92) is too large to wave away. That part deserves credit.\n\nBut the central claim is overstated, and the paper's own Table 3 shows it. Fleiss kappa for cross-template agreement of human-annotated correctness is 0.67 on GPQA and 0.73 on NarrativeQA, with 'perfect agreement' across all 12 templates for only 52% and 66% of questions. That means on nearly half of GPQA questions, human labels change correctness depending on template. Calling kappa >0.6 'consistently high' is generous; 0.67 is moderate. This is direct evidence that some prompt sensitivity is real, not an artifact of heuristics. The conclusion 'answer correctness does not vary substantially across templates' doesn't follow from the paper's own data.\n\nOther soft spots, in order: the human anchor covers 50 questions per dataset and one model per benchmark, so judge reliability outside that sample is assumed; judge accuracy near ceiling (ARC 0.90) means part of the variance compression is mechanical; the paraphrase-overlooked mechanism doesn't explain the multiple-choice log-likelihood results, where the output is a fixed letter and matching is exact; and there's a numeric inconsistency in the abstract/intro—accuracy 'varies by only 0.17' with a reported stdev of 0.005, but with 12 values a range of 0.17 puts a floor on stdev well above 0.005. No code or data is released, which matters for a methodological claim like this.\n\nWorth a serious referee? Yes. The question is important, the experimental design is substantial, and the MATH control plus human annotations are real evidence. But it needs revision: release the artifacts, widen the human validation across models, fix the inconsistency, and soften 'largely an artifact' to something like 'partly an artifact'—the data support that.","headline":"Argues prompt sensitivity is mostly a grading artifact; the direction is right and the human anchor gives it real weight, but the paper's own human data show genuine cross-template variation that the central claim overstates.","tokens_in":13448,"tokens_out":2233,"would_cite":true,"duration_ms":23725,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that much of the reported prompt sensitivity of LLMs is an artifact of heuristic evaluation, and that switching to LLM-as-a-Judge scoring makes accuracy and model rankings far more stable across paraphrased prompts.","keywords":["prompt sensitivity","LLM-as-a-Judge","evaluation artifact","semantic equivalence","ranking consistency","heuristic evaluation","benchmark stability","paraphrased prompts"],"falsifier":"Run the same 12-template protocol on ARC-Challenge with templates deliberately engineered to force different answer registers (letter-only, single word, full sentence, quoted phrase) and score with a blind LLM judge. If the standard deviation of judge-based accuracy across these templates approaches the 0.28 heuristic value, the artifact explanation is in trouble. A cheaper probe: re-annotate the human sample with a stricter rubric requiring exact reference wording and check whether judge-human agreement drops or rankings change.","tokens_in":12450,"feed_emoji":"🎯","tokens_out":11971,"duration_ms":110321,"temperature":0.7,"pith_summary":"Prompt sensitivity—the observation that paraphrasing a prompt changes a model's benchmark score—has been treated as a core weakness of LLMs. This paper argues that much of that sensitivity is manufactured by the grading protocol, not the model. Across 7 models, 6 benchmarks, and 12 paraphrased prompt templates, the authors compare heuristic scoring (log-likelihood over fixed options, word-F1 overlap, boxed-answer extraction) with LLM-as-a-Judge scoring, in which a judge model decides whether the prediction is semantically equivalent to the reference answer. Under judge-based scoring, accuracy variance across templates collapses—for example, Gemma-2 on ARC-Challenge goes from a standard deviation of 0.28 under heuristics to 0.005—and model rankings become consistent across prompts. Human annotation on a 50-question sample per dataset agrees with the judge, supporting the conclusion that prompt sensitivity is largely an evaluation artifact.","feed_headline":"Prompt sensitivity is mostly a grading artifact, not a model flaw","feed_subtitle":"Switching to an LLM judge cuts accuracy variance and stabilizes model rankings across prompt templates.","key_machinery":"The load-bearing object is the LLM-as-a-Judge protocol: a judge model (Gemini 2.0 Flash, cross-checked with GPT-4o-mini) receives the question, the reference answer, and the model prediction, and returns a binary judgment of whether the prediction semantically matches the reference while ignoring explanations. This replaces exact-format matching and log-likelihood scoring. Two sensitivity metrics carry the quantitative argument: the standard deviation of accuracy across the 12 prompt templates, and the mean pairwise Spearman rank correlation of model rankings across templates. The judge's binary semantic-equivalence decision is the mechanism that turns format-fragile scoring into format-inde","core_discovery":"The paper's central claim is that the large performance swings and ranking flips attributed to prompt paraphrasing mostly reflect how answers are graded, not how well models reason. Scored by log-likelihood over fixed options or by rigid extraction, any correct answer phrased differently from the reference gets marked wrong, and that inflates prompt sensitivity. Replacing that grader with an LLM judge that checks semantic equivalence produces uniformly small accuracy standard deviations across 12 prompt templates and raises the mean pairwise Spearman rank correlation of model rankings (ARC-Challenge: from about 0.30 to 0.92 for open models, and 0.95 including proprietary models). The judge i","pith_inferences":["If the claim holds, format sensitivity for end users is partly an interface property: applications that require machine-readable exact output still need format constraints, even though the underlying knowledge is stable.","The human validation covers only 50 questions per dataset from one model per benchmark; checking additional models and questions could reveal cases where the judge and annotators share a systematically lenient notion of what counts as a match.","A testable extension is to build adversarial template pairs that force stylistically different but semantically identical answers (single word vs. full sentence with citations) and see whether judge-based scores stay flat; if they move, some genuine sensitivity remains.","Because the judge is itself an LLM, its notion of semantic equivalence could be probed by swapping judge prompts or judge models; the paper's GPT-4o-mini cross-check on ARC-Challenge is one step in that direction."],"forward_implications":["Previously reported ranking flips across prompt templates should be re-examined; under semantic-equivalence grading, model rankings become nearly template-independent.","Benchmark scores from log-likelihood or rigid extraction can understate model capability when a correct answer is paraphrased or differently formatted.","LLM-as-a-Judge is a defensible default for prompt-sensitivity studies, though well-engineered task-specific heuristics (as the paper shows for MATH with symbolic simplification) can be just as stable.","Prompt-template stability appears to generalize across model generations: Llama-2 and Mistral, not just current models, also show compressed variance under judge scoring."],"supporting_citations":[{"why":"The large-scale prior study reporting significant accuracy differences across prompt variants, whose claims the paper re-explains as evaluation artifacts.","marker":"Mizrahi et al. (2024)"},{"why":"Showed that no prompt format consistently outperforms across models, a key prior evidence of ranking instability that the paper attributes to grading.","marker":"Voronov et al. (2024)"},{"why":"Prior multi-prompt evaluation work modeling prompt sensitivity as inherent to model behavior, which the paper contrasts with its evaluation-artifact explanation.","marker":"Polo et al. (2024)"},{"why":"Established the LLM-as-a-judge methodology and its alignment with human judgments, supplying the evaluation protocol adopted here.","marker":"Zheng et al. (2023)"},{"why":"ARC-Challenge, the benchmark where the paper's headline variance compression (Gemma-2 stdev 0.28 to 0.005) and rank-correlation improvement are shown.","marker":"Clark et al. (2018)"},{"why":"MATH benchmark, whose well-engineered symbolic heuristic matches LLM-as-a-Judge stability and serves as the counterexample showing heuristics can be stable.","marker":"Hendrycks et al. (2021b)"},{"why":"SimpleQA, a recently proposed factuality benchmark with no official heuristic parser, evaluated only by LLM judge in the paper.","marker":"Wei et al. (2024)"},{"why":"Supplies the Fleiss kappa used to quantify human-annotator agreement, evidence that answer correctness is stable across templates.","marker":"Fleiss (1971)"},{"why":"LongBench source of the word-level F1 normalization used for the NarrativeQA heuristic, the rigid matching the paper blames for inflated sensitivity.","marker":"Bai et al. (2024)"}],"fun_headline_variants":["Prompt sensitivity is an artifact of evaluation, not a model flaw","Bad grading inflates LLM prompt sensitivity, new study shows","LLM judges cut prompt variance, exposing model robustness","Prompt sensitivity: a grading artifact, not an LLM weakness","Swap log-likelihood for LLM judge, prompt sensitivity disappears"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that answer correctness should be defined as semantic equivalence to the reference answer, judged by an LLM and validated on 50 sampled questions from one model per benchmark; if a stricter, format-aware correctness standard is required, the observed stability is a property of that grading standard rather than of the models.","fun_headline_variants_meta":{"raw":{"variants":["Prompt sensitivity is an artifact of evaluation, not a model flaw","Bad grading inflates LLM prompt sensitivity, new study shows","LLM judges cut prompt variance, exposing model robustness","Prompt sensitivity: a grading artifact, not an LLM weakness","Swap log-likelihood for LLM judge, prompt sensitivity disappears"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1267,"prompt_tokens":728,"completion_tokens":539,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":454}},"tokens_in":472,"tokens_out":539,"duration_ms":5778,"temperature":1.0,"reasoning_tokens":454,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:11:45.748443+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 12-template protocol on ARC-Challenge with templates deliberately engineered to force different answer registers (letter-only, single word, full sentence, quoted phrase) and score with a blind LLM judge. If the standard deviation of judge-based accuracy across these templates approaches the 0.28 heuristic value, the artifact explanation is in trouble. A cheaper probe: re-annotate the human sample with a stricter rubric requiring exact reference wording and check whether judge-human agreement drops or rankings change.","supporting_citations":[],"review_version":1}