{"id":"1ca9b14b-c96d-4324-a66c-99339ce3efa6","arxiv_id":"2505.12938","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Submitting one solution per paraphrased variant improves Pass@k over repeating solutions to the original prompt, under a uniform-spread model and in APPS experiments.","lead":"The paper proposes a Variator agent that rewrites a problem into k paraphrased variants and submits one solution per variant, claiming better Pass@k than submitting k solutions to the original. It supports this with a theoretical bound and with coding and cybersecurity experiments on OpenAI o3-mini and Claude 3.7 Sonnet.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's regret guarantee is false: under the paper's own model, the Pass@k gap between Repeater and Variator can exceed (w/4)^k, so the claim that Variator only slightly underperforms on easy challenges is unproven.","rationale":"The paper aims to show that generating equivalent variants and submitting one solution per variant boosts Pass@k, with Theorem 1 as the central theoretical justification. The reader judged the theoretical core sound, but the regret guarantee in Theorem 1 is demonstrably false under the paper's own assumptions. The proof's step from 'p_o − p_v is maximized at p_o = 1' to 'Pass@k(Repeater) − Pass@k(Variator) is maximized there' is invalid because the function u_k is nonlinear; the counterexample above shows the claimed bound can be exceeded. This is an internal inconsistency, not a matter of interpretation. The reader's concern about variant equivalence is legitimate, but it is secondary: even granting full equivalence, the theoretical claim about slight underperformance on easy challenges is not established. The empirical evaluation in Section 5.2 also relies on plug-in estimates from eqs. (4)–(5) rather than an end-to-end run of the Variator agent, so the false theorem is not rescued by the experiments. Given that the central theoretical guarantee contains a concrete counterexample, the current version should not be accepted as written; it requires a corrected theorem, a revised proof, and re-verification of the downstream claims. I would therefore move the verdict from conditional to reject-as-written, while noting the paper's core idea may be salvageable after correction.","tokens_in":17771,"tokens_out":17093,"duration_ms":165760,"concrete_test":"Recompute the regret inequality at p_o = 0.95, w = 0.2, k = 2 using eq. (7) and eqs. (4)–(5). If p_v = 0.921875 and the gap 0.9975 − 0.993896 = 0.003604 exceeds 0.0025, the bound fails. A fuller check is to compute max_{p_o ∈ [0,1]} [u_k(p_o) − u_k(p_v(p_o))] for several values of w and k and compare it with (w/4)^k; if it ever exceeds (w/4)^k, the proof's monotonicity step is invalid and the theorem must be corrected.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's theoretical core rests on Theorem 1. Its performance guarantee (eq. 1) is correct, but the regret guarantee (eq. 2) is not. The proof in Appendix E claims that because p_o − p_v is maximized at p_o = 1, the Pass@k gap is also maximized there. This inference is invalid because u_k(p) = 1 − (1−p)^k is nonlinear in p. A direct counterexample using the paper's own model: take w = 0.2, k = 2, p_o = 0.95. From eq. (7), p_v = 1 − (1+0.2−0.95)^2 / (4·0.2) = 0.921875. Then Pass@2(Repeater) = 1 − 0.05^2 = 0.9975, while Pass@2(Variator) = 1 − 0.078125^2 = 0.993896, so the gap is 0.003604, which exceeds (w/4)^k = 0.0025. Thus the theorem's 'regardless of p_o' regret bound is false, and the paper's assertion that Variator is always nearly as good as Repeater is unsupported. This is an internal inconsistency in the proof, not merely a disagreement with prior results.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a \"Variator\" agent that paraphrases a given task into k semantically equivalent variants, generates one candidate solution per variant, and evaluates success under the Pass@k metric. The authors claim that this leverages LLM inconsistency to outperform a baseline \"Repeater\" agent that submits k solutions to the original task. They support this with a probabilistic model in Theorem 1, claiming a performance guarantee Pass@k(Variator) >= 1 - (1 - w/4)^k and a regret guarantee Pass@k(Variator) >= Pass@k(Repeater) - (w/4)^k for all p_o. They also report experiments on APPS coding problems and two CTF challenges showing inconsistency in reasoning models, plus a private-dataset experiment intended to remove memorization effects. The paper's central theoretical conclusion is that, under the stated model, Variator approaches perfect Pass@k for large k while only slightly underperforming Repeater on easy challenges.","tokens_in":18039,"tokens_out":5815,"duration_ms":58944,"significance":"If the central claims were correct, the paper would make a useful and practically relevant contribution: it would turn the widely observed phenomenon of prompt sensitivity into a cheap, task-agnostic Pass@k improvement, with a simple closed-form theoretical justification and empirical support on frontier reasoning models. The paper also has genuine strengths: it demonstrates inconsistency in reasoning models for coding and cybersecurity tasks, reports statistically significant p-values for the inconsistency effect, and attempts to control for memorization with a private-dataset construction. However, the regret guarantee of Theorem 1 is false as stated, and the empirical Pass@k comparison is a plug-in computation rather than a direct end-to-end measurement. Because these two issues bear directly on the paper's main claims, the paper cannot be accepted in its current form.","major_comments":[{"comment":"","section":"Section 5.1 and Appendix E, Eq. (2)"},{"comment":"","section":"Section 5.2 and Eqs. (4)-(5)"},{"comment":"","section":"Section 3.2 and Section 5.2"},{"comment":"","section":"Section 5.2, dataset selection"}],"minor_comments":[{"comment":"","section":"Appendix A"},{"comment":"","section":"Appendix C.1"},{"comment":"","section":"References"},{"comment":"","section":"Section 5.2"},{"comment":"","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's main theoretical result, the regret guarantee in Theorem 1, is internally inconsistent with the model it uses; this is the kind of error that must be fixed before publication. The plug-in evaluation in Section 5.2 is a further concern because it does not directly measure the claimed Pass@k improvement. On the positive side, the demonstration of inconsistency in reasoning models and the private-dataset design are valuable and could form the basis of a solid revised paper. The authors may also want to add a code/data availability statement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The stress-test note is correct, and it matters. The Variator idea—generate k paraphrased variants, submit one solution per variant—is clean, task-agnostic, and practically motivated. The symmetry-breaking intuition is real: hard challenges get more upside from small success-rate increases than easy challenges lose from small decreases. The paper also does something genuinely useful in showing that the inconsistency effect persists in frontier reasoning models on coding and CTF tasks, with careful null-hypothesis testing and expert validation of variant equivalence. The Appendix F prefix-failure experiment is clever and worth a look.\n\nThe problem is Theorem 1. The performance guarantee (1) appears correct: p_v is minimized at w/4, so Pass@k(Variator) ≥ 1 − (1 − w/4)^k. But the regret guarantee (2) is false. The proof claims that because p_o − p_v is maximized at p_o = 1, the Pass@k gap is also maximized there. That inference fails because u_k(p) = 1 − (1 − p)^k is nonlinear. The paper's own equations give a counterexample: take w = 0.2, k = 2, p_o = 0.95. Then p_v = 1 − (1+0.2−0.95)^2/(4·0.2) = 0.921875, so Pass@2(Repeater) = 0.9975 and Pass@2(Variator) = 0.993896. The gap is 0.003604, greater than (w/4)^k = 0.0025. The theorem's 'regardless of p_o' claim is therefore unsupported, and the paper's assertion that Variator only slightly underperforms on easy challenges needs a different proof or a weaker statement.\n\nThis is not a cosmetic issue. The regret guarantee is what justifies the claim that Variator wins on average over a mixed benchmark. Without it, the theoretical argument is incomplete.\n\nThe empirical side also has soft spots. The Pass@k numbers in Table 1 are computed by plugging estimated p_o and p_v into equations (4) and (5), not by actually running the Repeater and Variator agents end-to-end. There are no confidence intervals or significance tests. The public-APPS gains are small, the subset is a post-hoc selection, and the 'private dataset' is really a selected median variant used as the original, which reduces but does not eliminate the memorization concern. The equivalence assumption is also fragile: about 6% of coding variants were flagged non-equivalent by experts, and the Section 5.2 Variator runs apply no verification.\n\nEven with these caveats, the paper deserves a serious referee. The method is novel, the inconsistency demonstration is solid, and the core approach is checkable. I would send it to review, but I would expect major revision: fix or replace the regret bound, run the actual agents end-to-end with error bars, and release the code and variant data. As written, the theoretical claim is false and the empirical protocol is weaker than the conclusions require.","headline":"The Variator idea is genuinely interesting, but the paper's Theorem 1 regret guarantee is false under the paper's own model, and the empirical Pass@k numbers are plug-in estimates rather than end-to-end runs.","tokens_in":18619,"tokens_out":2969,"would_cite":false,"duration_ms":32331,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a model's inconsistent performance across reworded versions of the same task can be harvested: generate k equivalent variants and submit one solution per variant, and Pass@k rises even when each variant is, on…","keywords":["LLM inconsistency","Pass@k","prompt paraphrasing","variant generation","reasoning models","coding benchmarks","cybersecurity CTF","APPS dataset"],"falsifier":"Take a set of tasks, generate variants with the paper's prompt, have experts or an automated equivalence check discard every non-equivalent variant, and compare Variator against Repeater on the verified subset; if the Pass@k advantage vanishes or reverses once only provably equivalent variants are used, the symmetry-breaking claim is not the reason for the empirical gain.","tokens_in":17511,"feed_emoji":"🎲","tokens_out":5382,"duration_ms":50534,"temperature":0.7,"pith_summary":"This paper claims that a language model's inconsistency—the way its success on a task changes when the prompt is reworded or given a new backstory—can be turned into a measurable performance gain. The proposed Variator agent generates $k$ equivalent variants of a task and submits one candidate solution for each, while the Repeater baseline submits $k$ solutions to the original task. The paper proves that under a symmetric noise model the Variator's Pass@k approaches 1 exponentially in $k$, regardless of the original task's difficulty, and never falls far below the baseline. Experiments on coding benchmarks and private variant datasets support the claim, and the paper shows the underlying inconsistency persists in frontier reasoning models across coding and cybersecurity.","feed_headline":"Vary the prompt k ways, and Pass@k climbs even on hard problems","feed_subtitle":"Submitting one answer per rewritten variant beats k answers to the original—proven under a symmetric noise model.","key_machinery":"The load-bearing object is the Variator agent: an LLM-based generator that rewrites a task into $k$ paraphrased variants (changed backstory, wording, notation, or architecture) while preserving the input-output format, followed by one solver run per variant. The argument rests on the Pass@k transform $u_k(p)=1-(1-p)^k$, which for $k>1$ is convex and pushes intermediate probabilities toward 1, and on the piecewise formula for the clipped mean variant success rate $p_v$: $(w+p_o)^2/(4w)$ for hard tasks, $p_o$ in the middle zone, and $1-(1+w-p_o)^2/(4w)$ for easy tasks. Combining these yields the exponential guarantees of Theorem 1 and explains why the average loss on easy tasks is outweighed by the Pass@k amplification of gains on hard tasks.","core_discovery":"On the paper's own terms, the central discovery is a symmetry-breaking effect: when variant success rates are centered on the original success rate $p_o$ with symmetric noise clipped to $[0,1]$, averaging happens before the Pass@k transform, and the convexity of $u_k(p)=1-(1-p)^k$ for $k>1$ amplifies gains on hard tasks more than it penalizes losses on easy ones. Theorem 1 states that if $P_v=[p_o+W]_0^1$ with $W$ uniform on $[-w,w]$, then for every $p_o$, $\\mathrm{Pass@k}(\\mathrm{Variator}) \\ge 1-(1-w/4)^k$ and $\\mathrm{Pass@k}(\\mathrm{Variator}) \\ge \\mathrm{Pass@k}(\\mathrm{Repeater})-(w/4)^k$. The proof computes the clipped mean $p_v$, which exceeds $p_o$ for hard tasks where clipping at 0 occurs, equals $p_o$ in the middle range, and falls slightly below $p_o$ for easy tasks where clipping at 1 occurs. The paper also reports that Variator outperforms Repeater on the APPS benchmark for sufficiently large $k$ despite memorization headwinds, and that the advantage grows on a private variant dataset free of memorization.","pith_inferences":["If variant equivalence is not verified during deployment, the measured gains may overstate real-world gains: roughly 6% of coding variants in the paper's manual check were judged non-equivalent, and the autonomous agent applies no such check.","An automatic equivalence checker (for example, running the original test suite or a learned equivalence model on the generated variants) could raise Variator's success further while making the guarantee honest.","The same mechanism should transfer to other verifiable free-form domains—mathematical proof generation, hardware or network configuration, data-cleaning scripts—wherever \"correct\" can be tested automatically and paraphrases can be generated.","The failure-induction experiment in the appendix suggests that early token-path decisions drive inconsistency; if so, prompt paraphrasing could also serve as a cheap robustness probe for identifying which tasks a model solves for fragile reasons."],"forward_implications":["For any challenge, regardless of how hard the original is, Variator's Pass@k is guaranteed to approach 1 exponentially in $k$; with spread $w=0.2$, $k=10$ already guarantees at least 40% success.","Variator is never much worse than Repeater: the regret bound $(w/4)^k$ decays exponentially, so for reasonable $k$ the penalty is negligible even when variants are, on average, no easier than the original.","On the APPS competition-level subset, Variator beats Repeater for $k\\ge 5$ with o3-mini and $k\\ge 10$ with Claude 3.7 Sonnet (extended thinking), despite public-benchmark memorization favoring the original task.","On the private variant dataset, where memorization is absent, the improvement is larger, with gains of 3 to 5 percentage points at $k=20$.","Because inconsistency persists in frontier reasoning models in both coding and cybersecurity, the method is likely to stay effective for future model generations, not just current ones."],"supporting_citations":[{"why":"Defines the Pass@k metric that the whole comparison is built on.","marker":"[13]"},{"why":"Supplies the APPS coding benchmark used for the public empirical evaluation.","marker":"[23]"},{"why":"Documents the prompting-language sensitivity that the paper generalizes into a usable effect.","marker":"[2]"},{"why":"Shows prompt sensitivity is greater for long-output tasks such as coding, motivating the chosen domains.","marker":"[7]"},{"why":"Provides evidence of test-set memorization, which the paper identifies as a headwind for its public-dataset comparison.","marker":"[6]"},{"why":"System card for the o3-mini reasoning model used in the experiments.","marker":"[17]"},{"why":"System card for Claude 3.7 Sonnet (extended thinking) used in the experiments.","marker":"[18]"}],"fun_headline_variants":["Turn LLM inconsistency into a Pass@k boost by varying tasks","Inconsistency pays off: Vary the prompt, boost Pass@k","Varied prompts beat repeated tries for Pass@k","Symmetry-breaking: One answer per variant lifts Pass@k"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on LLM-generated variants being equivalent to the original task—same solution set and comparable difficulty; if a nontrivial fraction are easier or solve different problems, the measured Pass@k gains are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Turn LLM inconsistency into a Pass@k boost by varying tasks","Inconsistency pays off: Vary the prompt, boost Pass@k","Varied prompts beat repeated tries for Pass@k","Symmetry-breaking: One answer per variant lifts Pass@k"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000898,"raw_usage":{"total_tokens":3882,"prompt_tokens":972,"completion_tokens":2910,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":2837}},"tokens_in":588,"tokens_out":2910,"duration_ms":23626,"temperature":1.0,"reasoning_tokens":2837,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:23:13.837059+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of tasks, generate variants with the paper's prompt, have experts or an automated equivalence check discard every non-equivalent variant, and compare Variator against Repeater on the verified subset; if the Pass@k advantage vanishes or reverses once only provably equivalent variants are used, the symmetry-breaking claim is not the reason for the empirical gain.","supporting_citations":[{"cited_title":"The language of prompting: What linguistic properties make a prompt successful?, 2023","cited_arxiv_id":null,"evidence_quote":"Documents the prompting-language sensitivity that the paper generalizes into a usable effect."},{"cited_title":"PertEval: Unveiling real knowledge capacity of LLMs with knowledge-invariant perturbations","cited_arxiv_id":null,"evidence_quote":"Provides evidence of test-set memorization, which the paper identifies as a headwind for its public-dataset comparison."},{"cited_title":"OpenAI o3-mini system card, 2025","cited_arxiv_id":null,"evidence_quote":"System card for the o3-mini reasoning model used in the experiments."},{"cited_title":"Claude 3.7 Sonnet system card, 2025","cited_arxiv_id":null,"evidence_quote":"System card for Claude 3.7 Sonnet (extended thinking) used in the experiments."}],"review_version":1}