{"id":"e63e564e-8006-4400-863e-fb33e2382f8b","arxiv_id":"2608.11694","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Meaning-preserving rephrasing of benchmark problems flips model answers in both directions, and the net loss is larger for stronger models.","lead":"This paper introduces BenchDrift, a system that rephrases benchmark questions in ways that keep the answer the same, then measures how often a model's correctness flips. Across eight models and three benchmarks it finds that scores can swing by tens of points, and that stronger models tend to lose more correct answers to rephrasing than weaker models gain.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that stronger models are more wording-dependent is not yet separated from the algebraic link between drift directions and baseline accuracy; conditional rates are missing.","rationale":"The paper is a careful measurement study with open code and data, an explicit taxonomy, and honest limitations. The basic observation that large two-way drift occurs under rephrasing is credible and is supported by the role-swap checks in Section 4.2, which show that the measured drift is not strongly sensitive to which LLM plays the generator, validator, or judge. I focused on the asymmetry and sign-change claim rather than the validator issue because it is the load-bearing step for the paper's most provocative conclusion—that stronger models are more wording-dependent—and because it can be checked immediately from the released data. The validator concern in Appendix A is real, but it is at least partially bounded by the role-swap agreement; the denominator-confounding issue is not addressed anywhere in the paper. The reader's weakest_assumption named the validator, but the reader's rationale also identified the mechanical link between drift directions and baseline accuracy, so my concern is a different, though related, load-bearing issue. Since the fix is an additional analysis on already-released data rather than new experiments, and since the drift measurements themselves are not invalidated, the CONDITIONAL verdict remains appropriate.","tokens_in":13721,"tokens_out":4507,"duration_ms":49840,"concrete_test":"From the released per-problem data, compute for each of the 24 model–benchmark pairs the conditional positive-drift rate Pos/(1−Rep) and the conditional negative-drift rate Neg/Rep. Then fit the constant-rate null model Pos = (1−Rep)r_p and Neg = Rep r_n, and compare it against a model in which conditional rates vary with baseline accuracy (e.g., bootstrap confidence intervals for r_p and r_n across model-strength bins, or a likelihood-ratio test). If r_n and r_p are approximately flat across Rep, the Section 4.1 'changes sign' claim is a denominator artifact; if r_n rises significantly as Rep increases, the model-level claim survives. Separately, report total drift Pos+Neg conditional on Rep to test whether stronger models have genuinely larger wording dependence rather than merely more correct answers to lose.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's second headline finding—that phrasing sensitivity changes sign as models improve and that the best models are the most wording-dependent—relies on comparing Pos and Neg as fractions of the full problem set N. By the definitions in Section 2.2, Pos can only count problems the model got wrong on the original phrasing and Neg can only count problems it got right, so Pos ≤ 1−Rep and Neg ≤ Rep. Under a null model in which every baseline failure has a fixed recovery rate r_p and every baseline success a fixed fragility rate r_n, Neg − Pos = Rep(r_n + r_p) − r_p, a linear function of Rep. The observed near-linear relation in Section 4.1 (Pearson r = 0.98, slope 1.08) is therefore exactly the shape such a baseline-only null model predicts. The paper never reports the conditional rates—positive drift among original failures and negative drift among original successes—nor does it fit a constant-rate null model, so the sign-change result and the claim that 'the best models on a benchmark are the ones whose scores depend most on the wording' are not yet separated from a mechanical consequence of the drift denominators. This concern is independent of the validator issue in Appendix A: even if every validated variation truly preserves the answer, the headline asymmetry could still be an artifact of how the rates are normalized. The released per-problem data are sufficient to resolve this directly.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BenchDrift, a pipeline that generates variations of benchmark problems along four axes (linguistic, referential, pragmatic, structural), validates with an LLM that each variation preserves the ground-truth answer, and measures how often correctness flips relative to the original phrasing. Over eight models and three benchmarks (GSM8K, MMLU, MATH-Hard), the authors report an average best-to-worst accuracy span of 74.7 percentage points, a two-way drift in which weak models gain more than they lose and strong models lose far more than they gain, and transformation-level fragility rankings that are consistent across models. They also report reliability checks that swap generator/validator/judge models, a subsampling analysis of the number of variations needed, and a comparison with prompt optimizers. The paper's central claims are that benchmark scores are not stable estimates of capability and that the best models are the most wording-dependent.","tokens_in":13872,"tokens_out":11633,"duration_ms":105692,"significance":"If the findings hold, the paper provides a valuable methodology and measurement: it turns a single-point benchmark score into an interval, attributes flips to concrete transformations, and shows cross-model agreement about which rephrasings are dangerous. The open release of code, data, and the variation taxonomy is a strength, as are the bootstrap CIs, the role-swap reliability checks, and the permutation-based axis test. However, the two headline interpretive claims—the sign change with model strength and the assertion that the best models depend most on wording—rest on unconditional drift rates whose denominators impose algebraic caps, and the answer-preservation guarantee relies on LLM validator/judge instruments with no human validation. The paper itself flags the latter in Appendix A. These issues are addressable with additional analyses on the released data, but they currently leave the strongest conclusions underdetermined.","major_comments":[{"comment":"The claim that phrasing sensitivity changes sign as models improve (and the derived statement that the best models are the most wording-dependent) is based on comparing Pos and Neg as unconditional fractions of the problem count N. By definition, positive drift can only occur on problems the model answered incorrectly on the original phrasing, and negative drift only on problems it answered correctly, so Pos ≤ 1−Rep and Neg ≤ Rep. Under a baseline-only null model with constant recovery rate r_p for original failures and constant fragility rate r_n for original successes, Neg − Pos = Rep(r_n + r_p) − r_p, which is linear in Rep with slope r_n + r_p. The reported near-linear relation (Pearson r = 0.98, slope 1.08) is exactly the curve such a null produces, so it does not by itself establish a model-dependent change in sensitivity. The manuscript never reports the conditional rates—positive drift among original failures, Pos/(1−Rep), and negative drift among original successes, Neg/Rep—nor fits a constant-rate null model. Recomputing from Table 2 shows the direction can reverse under conditioning: GPT-OSS-20B on GSM8K has a conditional recovery rate of 80.8% (4.2/5.2) versus a conditional fragility rate of 59.9% (56.8/94.8). The authors should report the conditional rates for all 24 pairs, fit the constant-rate null, and re-evaluate the sign-change and 'most wording-dependent' conclusions on that basis.","section":"Section 2.2 and Section 4.1"},{"comment":"The validator and the judge are LLMs, and the paper states that no human validation was run. The authors correctly note that a judge that is directionally more permissive (e.g., more willing to accept borderline answers on variations than on the original) could inflate or deflate the asymmetry, which is a central finding. The role-swap checks in Section 4.2 demonstrate agreement among LLM instruments, not agreement with human judgment, so a shared blind spot remains possible. Because every drift measurement and the 74.7-point span assume that each surviving variation truly preserves the answer, an unvalidated validator confounds wording effects with actual answer changes. The manuscript should include a human-annotation study on a stratified sample of variations (across axes and benchmarks) reporting validator precision and judge directional bias, and should show that the headline drift rates and the asymmetry are robust to a conservative re-analysis restricted to human-confirmed answer-preserving variations.","section":"Appendix A and Section 2.5"},{"comment":"The statement that role swaps 'shift average negative drift by at most 11.9 percentage points and average positive drift by at most 2.9, both within the per-cell confidence intervals of Table 2' appears to be incorrect. The per-cell (or per-configuration) confidence intervals in Table 3 are around ±3.7–4.3 percentage points, while the main configuration reports negative drift of 45.9±4.3 and the fully swapped configuration reports 34.0±3.8; the difference of 11.9 points is outside the reported interval. Since this sentence is the basis for the claim that the measurement is stable across pipeline roles, the authors should correct the CI comparison (e.g., report overlap tests or the standard error of the difference).","section":"Section 4.2"}],"minor_comments":[{"comment":"The notation 'Rep.' is used in the identities Best = Rep. + Pos. and Worst = Rep. − Neg. before it is defined; define it at first use (e.g., when drift is introduced).","section":"Section 2.2"},{"comment":"Confidence intervals are only shown for the main configuration; report CIs for each role-swap row so the reader can assess the overlap claims directly.","section":"Table 3"},{"comment":"The y-axes are truncated (starting at 8% and 5%) without break symbols; this can exaggerate relative differences—add an axis break or start at 0 while keeping the printed values.","section":"Figure 3"},{"comment":"The comparison with DSPy and GEPA uses only 15 GSM8K problems and a single target model; the text should clearly state that this is an illustrative case study rather than a general efficiency comparison.","section":"Section 4.8"},{"comment":"The permutation test reports p<0.001 but does not state the test statistic; specify the quantity being permuted (e.g., spread of axis-level drift rates) and the test direction.","section":"Section 4.3"},{"comment":"The statement that 'models more often lose correct answers to rephrasing than they gain new ones' is an aggregate over 24 model–benchmark pairs; make explicit that this is an average and that the direction reverses for weaker models below the 60% baseline.","section":"Abstract and Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the journal and the open release of code and data is valuable. In my view, the conditional-rates reanalysis and the human-validation study are necessary before the headline claims can be accepted; the authors' own limitations section already identifies the human-validation gap. The CI comparison error in Section 4.2 should also be corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful measurement framework. The authors build a four-axis, answer-preserving rephrasing pipeline, validate variations with an LLM validator, run role-swap controls, attribute every flip to a specific transformation, and release the code and data. That part is solid, and the existence of large two-way drift—some failures becoming successes and some successes failures under rephrasing—is credible. Second, the paper's flashier claim, that better models are more wording-dependent because their reported scores lose more to rephrasing, is not yet supported. The reason is simple and mostly algebraic. Pos and Neg are computed as fractions of the full problem set. A problem can only contribute to Pos if the model got the original wrong, and only to Neg if it got the original right. So the asymmetry Neg−Pos is linearly related to baseline accuracy even when the underlying per-problem fragility and recovery rates are constant. The paper reports r=0.98 between that asymmetry and Rep, which is exactly the shape a null model predicts. They never report the conditional rates—recovery among original failures, fragility among original successes—so the sign-change result is confounded with the mechanical effect of the denominators. This is fixable from the released per-problem data and should be fixed before publication.\n\nThe other soft spot is the validator. The answer-preservation guarantee is load-bearing, and the validator is an LLM with no human agreement study. The authors admit this in Appendix A, and the role-swap checks show LLMs agree with each other, not that any of them track human judgment. That's a real limitation, but not a fatal one for a first pass.\n\nMinor issues: the prompt-optimizer comparison is on 15 problems, so treat it as illustrative. And the 'fragility belongs to the rephrasing not the model' claim is a bit strong with mean pairwise Spearman ρ=0.77.\n\nWho is this for? Anyone building or using benchmarks. The audit method is reusable and the interval around a score (Best/Rep/Worst) is a good way to report uncertainty. I'd bring it to reading group. It deserves a serious referee, but the referee should demand the conditional-rate analysis and ideally a small human validation of the validator. With those, it could be a useful reference.","headline":"Useful, well-built audit method; the 'stronger models are more wording-dependent' claim is confounded by rate definitions and needs conditional-rate analysis before it can be believed.","tokens_in":14483,"tokens_out":4578,"would_cite":true,"duration_ms":44111,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark score is computed from one phrasing of each problem, but rephrasing a problem while keeping its answer fixed routinely flips an LLM's correctness in both directions, so the reported number is just one point inside a wide range.","keywords":["benchmark drift","wording sensitivity","meaning-preserving rephrasing","LLM evaluation","positive and negative drift","prompt robustness","BenchDrift"],"falsifier":"Take a random sample of the variations the validator accepted, ask independent human annotators whether each one still has the same ground-truth answer, re-measure drift using only human-approved variations, and compare. If the 74.7-point average range collapses or the positive–negative asymmetry reverses, the original numbers were set by the validator's blind spot rather than by the models' wording sensitivity.","tokens_in":13430,"feed_emoji":"🔀","tokens_out":11517,"duration_ms":96135,"temperature":0.7,"pith_summary":"The paper's central claim is that a benchmark score, computed from one phrasing of each problem, is not a stable estimate of what an LLM can do. Meaning-preserving rephrasing routinely flips correctness in both directions: some wrong answers become right and some right answers become wrong, across eight models and three benchmarks. Averaged over the 24 model–benchmark pairs, worst-case accuracy (correct under every phrasing) and best-case accuracy (correct under at least one) are separated by 74.7 percentage points. The paper further argues that phrasing sensitivity does not fade as models get stronger; instead it changes sign, so strong models lose far more correct answers to rephrasing than they gain, while weak models gain more than they lose. If this is right, leaderboard scores can rank wording luck as much as capability.","feed_headline":"Rephrasing flips LLM answers, hiding a 74.7-point score range","feed_subtitle":"The same problem, reworded, can swing a model's accuracy by nearly 75 points.","key_machinery":"The central object is BenchDrift, a four-role pipeline: a generator proposes meaning-preserving rephrasings of each problem, a validator keeps only those that still yield the original ground-truth answer, the target model answers the original and all surviving variations, and a judge scores the answers. The argument is carried by the drift identities $Best = Rep. + Pos.$ and $Worst = Rep. - Neg.$, where $Pos.$ is the fraction of problems wrong on the original but right on some rephrasing and $Neg.$ is the fraction right on the original but wrong on some rephrasing. A shared denominator makes the two directions comparable, and every flip is tied to one of four axes — linguistic, referential, pragmatic, and structural — and to a specific transformation within that axis. These identities turn the reported score into one point in an exactly computable interval.","core_discovery":"The paper establishes that correctness under meaning-preserving rephrasing behaves like a two-way drift, with every flip attributed to a specific transformation. For a fixed problem and answer, a model's accuracy can be as low as the share of problems it gets right under every tested rephrasing and as high as the share it gets right under at least one, and the reported single-phrasing score sits inside that range. Across GSM8K, MMLU, and MATH-Hard, this range averages 74.7 percentage points. The direction of the drift tracks baseline accuracy almost linearly (Pearson $r = 0.98$, slope 1.08): above a 60% baseline, negative drift exceeds positive drift in all twelve model–benchmark pairs, so the strongest models are precisely the ones whose scores depend most on the wording they were given. Models also agree on which rephrasings cost the most correct answers (mean pairwise Spearman $\\rho = 0.77$), and rephrasing breaks a substantial share of answers the model was most confident about, whether the rephrasing makes the problem shorter or longer.","pith_inferences":["Editorial inference: if fragility belongs to the rephrasing, then benchmark items could be ranked by how many models they break, and weighting items by this fragility would produce a more stable evaluation score; the paper does not construct such an item-level ranking.","One testable extension the paper leaves implicit is whether drift on one benchmark predicts drift on another for the same model; the near-linear asymmetry result suggests it might, but no cross-benchmark correlation is reported.","Because the authors find rephrasing breaks confident answers, a selective-prediction system that only answers when the model shows high confidence would still lose accuracy under rephrasing; this follows from their data but is not tested in the paper."],"forward_implications":["A single benchmark score should be read as one point inside a range; across the 24 model–benchmark pairs, the worst-to-best gap averages 74.7 percentage points.","For models above a roughly 60% baseline, rephrasing costs more correct answers than it recovers, so the reported score tends to overstate how much of a model's success is robust to wording.","Because models largely agree on which rephrasings break the most answers, a wording transformation that hurts one model is likely to hurt others; fragility is a property of the rephrasing, not just of the model.","Model confidence cannot filter fragile answers: even among the answers the model was most confident about, almost one in five is lost to a meaning-preserving rephrasing.","An audit using only five rephrased variations per problem recovers about three-quarters of the drifting problems, so measuring drift does not require a full-scale generation campaign."],"supporting_citations":[{"why":"Supplies GSM8K, one of the three benchmarks whose problems are rephrased and whose drift is measured.","marker":"Cobbe et al., 2021"},{"why":"Supplies MMLU, the multi-domain factual benchmark used as a second testbed for drift.","marker":"Hendrycks et al., 2021"},{"why":"Provides the chain-of-thought prompting protocol used for all target-model answers, with temperature 0 to remove sampling noise.","marker":"Wei et al., 2022"},{"why":"Supplies the LLM-as-judge setup that scores answers and accepts equivalent forms such as '12', 'twelve', and 'a dozen'.","marker":"Zheng et al., 2023"},{"why":"Prior evidence that prompt formatting alone moves benchmark accuracy; the paper extends this to meaning-preserving rephrasing.","marker":"Sclar et al., 2024"},{"why":"Prior behavioral testing showing that templated edits expose hidden failures; the paper's transformation taxonomy builds on this idea.","marker":"Ribeiro et al., 2020"}],"fun_headline_variants":["Benchmark scores drift 74.7 points when LLMs see reworded problems","Same question, new wording: LLM accuracy swings by 74.7 points","Two-way drift: rephrasing flips LLM corrects and errors alike","Better LLMs, more wording-sensitive: 74.7-point score swing","Rephrasing benchmarks reveals model fragility, 74.7-point gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated validator and judge reliably identify which reworded problems truly keep the original answer and which answers are correct; the paper reports no human validation and cannot rule out a shared blind spot, so any drift measured could mix wording effects with actual answer changes.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark scores drift 74.7 points when LLMs see reworded problems","Same question, new wording: LLM accuracy swings by 74.7 points","Two-way drift: rephrasing flips LLM corrects and errors alike","Better LLMs, more wording-sensitive: 74.7-point score swing","Rephrasing benchmarks reveals model fragility, 74.7-point gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001245,"raw_usage":{"total_tokens":5167,"prompt_tokens":1064,"completion_tokens":4103,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":3998}},"tokens_in":680,"tokens_out":4103,"duration_ms":24566,"temperature":1.0,"reasoning_tokens":3998,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:31:18.629531+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the variations the validator accepted, ask independent human annotators whether each one still has the same ground-truth answer, re-measure drift using only human-approved variations, and compare. If the 74.7-point average range collapses or the positive–negative asymmetry reverses, the original numbers were set by the validator's blind spot rather than by the models' wording sensitivity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MMLU, the multi-domain factual benchmark used as a second testbed for drift."}],"review_version":1}