Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Leveraging LLM Inconsistency to Boost Pass@k Performance

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a model's inconsistent performance across reworded versions of the same task can be harvested: generate k equivalent variants and submit one solution per variant, and Pass@k rises even when each variant is, on…

desk verdict The Variator idea is genuinely interesting, but the paper's Theorem 1 regret guarantee is false under the paper's own model, and the empirical Pass@k numbers are plug-in estimates rather than end-to-end runs. read the letter →

arxiv 2505.12938 v2 pith:G6LGO2OO submitted 2025-05-19 cs.LG cs.AIcs.CL

Uri Dalal , Meirav Segal , Zvika Ben-Haim , Dan Lahav , Omer Nevo This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords LLMinconsistencyPass@kpromptparaphrasingvariantgenerationreasoningmodelscodingbenchmarkscybersecurityCTFAPPSdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a language model's inconsistency—the way its success on a task changes when the prompt is reworded or given a new backstory—can be turned into a measurable performance gain. The proposed Variator agent generates $k$ equivalent variants of a task and submits one candidate solution for each, while the Repeater baseline submits $k$ solutions to the original task. The paper proves that under a symmetric noise model the Variator's Pass@k approaches 1 exponentially in $k$, regardless of the original task's difficulty, and never falls far below the baseline. Experiments on coding benchmarks and private variant datasets support the claim, and the paper shows the underlying inconsistency persists in frontier reasoning models across coding and cybersecurity.

What carries the argument

The load-bearing object is the Variator agent: an LLM-based generator that rewrites a task into $k$ paraphrased variants (changed backstory, wording, notation, or architecture) while preserving the input-output format, followed by one solver run per variant. The argument rests on the Pass@k transform $u_k(p)=1-(1-p)^k$, which for $k>1$ is convex and pushes intermediate probabilities toward 1, and on the piecewise formula for the clipped mean variant success rate $p_v$: $(w+p_o)^2/(4w)$ for hard tasks, $p_o$ in the middle zone, and $1-(1+w-p_o)^2/(4w)$ for easy tasks. Combining these yields the exponential guarantees of Theorem 1 and explains why the average loss on easy tasks is outweighed by the Pass@k amplification of gains on hard tasks.

What would settle it

Take a set of tasks, generate variants with the paper's prompt, have experts or an automated equivalence check discard every non-equivalent variant, and compare Variator against Repeater on the verified subset; if the Pass@k advantage vanishes or reverses once only provably equivalent variants are used, the symmetry-breaking claim is not the reason for the empirical gain.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a symmetry-breaking effect: when variant success rates are centered on the original success rate $p_o$ with symmetric noise clipped to $[0,1]$, averaging happens before the Pass@k transform, and the convexity of $u_k(p)=1-(1-p)^k$ for $k>1$ amplifies gains on hard tasks more than it penalizes losses on easy ones. Theorem 1 states that if $P_v=[p_o+W]_0^1$ with $W$ uniform on $[-w,w]$, then for every $p_o$, $\mathrm{Pass@k}(\mathrm{Variator}) \ge 1-(1-w/4)^k$ and $\mathrm{Pass@k}(\mathrm{Variator}) \ge \mathrm{Pass@k}(\mathrm{Repeater})-(w/4)^k$. The proof computes the clipped mean $p_v$, which exceeds $p_o$ for hard tasks where clipping at 0 occurs, equals $p_o$ in the middle range, and falls slightly below $p_o$ for easy tasks where clipping at 1 occurs. The paper also reports that Variator outperforms Repeater on the APPS benchmark for sufficiently large $k$ despite memorization headwinds, and that the advantage grows on a private variant dataset free of memorization.

Load-bearing premise

The claim depends on LLM-generated variants being equivalent to the original task—same solution set and comparable difficulty; if a nontrivial fraction are easier or solve different problems, the measured Pass@k gains are inflated.

Editorial extensions

If this is right

  • For any challenge, regardless of how hard the original is, Variator's Pass@k is guaranteed to approach 1 exponentially in $k$; with spread $w=0.2$, $k=10$ already guarantees at least 40% success.
  • Variator is never much worse than Repeater: the regret bound $(w/4)^k$ decays exponentially, so for reasonable $k$ the penalty is negligible even when variants are, on average, no easier than the original.
  • On the APPS competition-level subset, Variator beats Repeater for $k\ge 5$ with o3-mini and $k\ge 10$ with Claude 3.7 Sonnet (extended thinking), despite public-benchmark memorization favoring the original task.
  • On the private variant dataset, where memorization is absent, the improvement is larger, with gains of 3 to 5 percentage points at $k=20$.
  • Because inconsistency persists in frontier reasoning models in both coding and cybersecurity, the method is likely to stay effective for future model generations, not just current ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If variant equivalence is not verified during deployment, the measured gains may overstate real-world gains: roughly 6% of coding variants in the paper's manual check were judged non-equivalent, and the autonomous agent applies no such check.
  • An automatic equivalence checker (for example, running the original test suite or a learned equivalence model on the generated variants) could raise Variator's success further while making the guarantee honest.
  • The same mechanism should transfer to other verifiable free-form domains—mathematical proof generation, hardware or network configuration, data-cleaning scripts—wherever "correct" can be tested automatically and paraphrases can be generated.
  • The failure-induction experiment in the appendix suggests that early token-path decisions drive inconsistency; if so, prompt paraphrasing could also serve as a cheap robustness probe for identifying which tasks a model solves for fragile reasons.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a "Variator" agent that paraphrases a given task into k semantically equivalent variants, generates one candidate solution per variant, and evaluates success under the Pass@k metric. The authors claim that this leverages LLM inconsistency to outperform a baseline "Repeater" agent that submits k solutions to the original task. They support this with a probabilistic model in Theorem 1, claiming a performance guarantee Pass@k(Variator) >= 1 - (1 - w/4)^k and a regret guarantee Pass@k(Variator) >= Pass@k(Repeater) - (w/4)^k for all p_o. They also report experiments on APPS coding problems and two CTF challenges showing inconsistency in reasoning models, plus a private-dataset experiment intended to remove memorization effects. The paper's central theoretical conclusion is that, under the stated model, Variator approaches perfect Pass@k for large k while only slightly underperforming Repeater on easy challenges.

Significance. If the central claims were correct, the paper would make a useful and practically relevant contribution: it would turn the widely observed phenomenon of prompt sensitivity into a cheap, task-agnostic Pass@k improvement, with a simple closed-form theoretical justification and empirical support on frontier reasoning models. The paper also has genuine strengths: it demonstrates inconsistency in reasoning models for coding and cybersecurity tasks, reports statistically significant p-values for the inconsistency effect, and attempts to control for memorization with a private-dataset construction. However, the regret guarantee of Theorem 1 is false as stated, and the empirical Pass@k comparison is a plug-in computation rather than a direct end-to-end measurement. Because these two issues bear directly on the paper's main claims, the paper cannot be accepted in its current form.

major comments (4)
  1. [Section 5.1 and Appendix E, Eq. (2)]
  2. [Section 5.2 and Eqs. (4)-(5)]
  3. [Section 3.2 and Section 5.2]
  4. [Section 5.2, dataset selection]
minor comments (5)
  1. [Appendix A]
  2. [Appendix C.1]
  3. [References]
  4. [Section 5.2]
  5. [General]

Circularity Check

1 steps flagged · score 2.0 of 10

Theorem 1 and the theoretical derivation are self-contained; the only circularity-like step is the 'empirical' Pass@k table, which is computed by plugging measured success rates into the theorem's own eqs. (4)-(5) rather than by independently running the two agents.

  1. fitted input called prediction [Section 5.2, 'Empirical results', third paragraph]
    "The average success rate among these 150 candidates is our estimate for the mean variant success rate pv of eq. (3). Finally, we compute Pass@k for both agents from po and pv using eqs. (4) and (5)."

    The paper presents Table 1 as empirical validation that Variator outperforms Repeater, but the Pass@k numbers are not obtained by actually running the two agents k times and counting successes. Instead they are obtained by applying the theorem's own formulas Pass@k(Repeater)=1-(1-po)^k and Pass@k(Variator)=1-(1-pv)^k to the measured means po and pv. These formulas are deterministic monotone transforms of the very quantities estimated from the data, so the crossover pattern in Table 1 is forced by construction once the measured pv lies slightly below po at high po and above po at low po. The table is thus a projection of the model rather than an independent empirical test of the theorem's functional form.

full rationale

The theoretical core of the paper is not circular. Theorem 1 starts from an explicit probabilistic model, P_v=[p_o+W]_0^1 with W uniform in [-w,w], and derives p_v and the Pass@k expressions by direct expectation under stated assumptions; it contains no fitted parameters and does not rely on self-citations or imported uniqueness claims. The measured p_o versus p_v scatter plots in Figure 5 are genuine external data from APPS and private variants, and the qualitative claim about variant success rates is anchored to those measurements. The only circularity-like element is the plug-in computation of Table 1 from eqs. (4)-(5), which makes the reported Pass@k a deterministic function of the measured inputs rather than an independently executed experiment; this is a validation-presentation concern, not an infection of the theorem itself. Separately, I note a non-circular correctness issue in Appendix E: the proof infers from p_o-p_v being maximized at p_o=1 that the Pass@k gap is also maximized there, but u_k(p)=1-(1-p)^k is nonlinear in p; for w=0.2, k=2, p_o=0.95 the gap exceeds (w/4)^k, so the regret guarantee as stated is questionable. That is a mathematical proof error/risk, not a circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The theoretical result rests on a stylized uniform-spread model and on the equivalence of generated variants; the empirical claims rest on treating repeated LLM samples as independent Bernoulli draws and on a post-hoc APPS subset selection. No free parameters are fit to the data: the w parameter is a modeling choice and the 60-test threshold is a data-selection choice. No new physical constants or entities are introduced.

free parameters (2)
  • w (uniform spread in Theorem 1)
    Half-width of the uniform distribution for variant success rates. Not fitted to data, but the quantitative guarantees (e.g., 'with w=0.2, Pass@10(Variator) >= 40%') depend on it, and any claimed rate assumes a specific w.
  • Minimum test cases threshold (60) = 60
    Hand-chosen data-selection criterion in Section 5.2 that restricts the APPS evaluation to competition-level problems with >=60 test cases, yielding 60 challenges. This choice affects the measured Pass@k gap.
assumptions (4)
  • domain assumption Generated variants are equivalent to the original challenge (same solution set and near-identical difficulty).
    Section 3.2 states that the definition requires equivalence but that the generator 'does not inherently guarantee equivalence'. Human verification found ~6% non-equivalent coding variants, and Section 5.2 runs without verification, so the central empirical claim depends on this premise.
  • ad hoc to paper Variant success rates follow P_v = [p_o + W]_0^1 with independent uniform W in [-w, w].
    This is the explicit model of Theorem 1. Independence of W_j is used in the product factorization in Appendix E, while the actual generator is sequential and shares context (previously used titles), so independence is an idealization.
  • domain assumption Repeated LLM samples are independent Bernoulli trials with a fixed success rate.
    Eqs. (4) and (5) compute Pass@k as 1-(1-p)^k from a single success rate. API sampling at temperature 1 is treated as i.i.d., which underpins both the theoretical Pass@k formulas and the empirical plug-in estimates.
  • standard math Under the no-inconsistency null, success counts across variants follow Bin(n,p).
    Used in Section 4 and Appendix D for the Monte Carlo hypothesis test that rejects the null of uniform success rates.
invented entities (1)
  • Variator agent
    purpose: Software agent that paraphrases a task into k variants and submits one solution per variant to boost Pass@k.
    The Variator is a proposed method, not a physical or theoretical entity. It has empirical handles (reported Pass@k numbers) but no external falsifiable prediction beyond the experiments in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging LLM Inconsistency to Boost Pass@k Performance." pith.science (2026). https://pith.science/paper/G6LGO2OO

@misc{pith2026250512938,
  author       = {Pith},
  title        = {Pith review of: Leveraging LLM Inconsistency to Boost Pass@k Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G6LGO2OO}},
  note         = {Machine review of arXiv:2505.12938}
}
read the original abstract

Large language models (LLMs) achieve impressive abilities in numerous domains, but exhibit inconsistent performance in response to minor input changes. Rather than view this as a drawback, in this paper we introduce a novel method for leveraging models' inconsistency to boost Pass@k performance. Specifically, we present a "Variator" agent that generates k variants of a given task and submits one candidate solution for each one. Our variant generation approach is applicable to a wide range of domains as it is task agnostic and compatible with free-form inputs. We demonstrate the efficacy of our agent theoretically using a probabilistic model of the inconsistency effect, and show empirically that it outperforms the baseline on the APPS dataset. Furthermore, we establish that inconsistency persists even in frontier reasoning models across coding and cybersecurity domains, suggesting our method is likely to remain relevant for future model generations.

Figures

Figures reproduced from arXiv: 2505.12938 by the authors.

Figure 1
Figure 1. Overview of the proposed technique, which is shown both empirically (for a commonly [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. However, the observed success rates differ significantly from this expected distribution. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 2
Figure 2. Variant success rate distributions. Blue bars represent the observed histogram of variant [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: Distributions of success rates for 25 variants of the same challenge, with three guidance [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 4
Figure 4. Figure 4: Success rates of a challenge and its variants. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Average variant success rate pv as a function of the original challenge success rate po for different models. Orange circles represent individual challenges, and the solid orange line is a smoothed spline interpolation thereof. Model memorization is one likely reason f…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 35 canonical work pages

  1. [1]

    The economic potential of generative AI: The next productivity frontier

    McKinsey Global Institute. The economic potential of generative AI: The next productivity frontier. Technical report, McKinsey & Company, June 2023. URL https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/ the-economic-potential-of-generative-ai-the-next-productivity-frontier . Accessed: May 21, 2025

    work page 2023

  2. [2]

    The language of prompting: What linguistic properties make a prompt successful?, 2023

    Alina Leidinger, Robert Van Rooij, and Ekaterina Shutova. The language of prompting: What linguistic properties make a prompt successful?, 2023. URL https://arxiv.org/abs/2311. 01967

    work page 2023

  3. [3]

    Robustness of learning from task instructions, 2022

    Jiasheng Gu, Hongyu Zhao, Hanzi Xu, Liangyu Nie, Hongyuan Mei, and Wenpeng Yin. Robustness of learning from task instructions, 2022. URL https://arxiv.org/abs/2212. 03813

    work page 2022

  4. [4]

    Large language models sensitivity to the order of options in multiple-choice questions, 2023

    Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions, 2023. URL https://arxiv.org/abs/2308.11483

    arXiv 2023

  5. [5]

    When benchmarks are targets: Revealing the sensitivity of large language model leaderboards

    Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    work page 2024

  6. [6]

    PertEval: Unveiling real knowledge capacity of LLMs with knowledge-invariant perturbations

    Jiatong Li, Renjun Hu, Kunzhe Huang, Yan Zhuang, Qi Liu, Mengxiao Zhu, Xing Shi, and Wei Lin. PertEval: Unveiling real knowledge capacity of LLMs with knowledge-invariant perturbations. In 38th Annual Conference on Neural Information Processing Systems (NeurIPS),

    work page

  7. [7]

    Prosa: Assessing and understanding the prompt sensitivity of LLMs, 2024

    Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. Prosa: Assessing and understanding the prompt sensitivity of LLMs, 2024. URL https: //arxiv.org/abs/2410.12405

    arXiv 2024

  8. [8]

    Mind your format: Towards consistent evaluation of in-context learning improvements, 2024

    Anton V oronov, Lena Wolf, and Max Ryabinin. Mind your format: Towards consistent evaluation of in-context learning improvements, 2024. URL https://arxiv.org/abs/2401. 06766

    work page 2024

Show all 47 references
  1. [9]

    State of what art? A call for multi-prompt LLM evaluation

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? A call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949, 2024

    2024

  2. [10]

    Unnatural instructions: Tuning language models with (almost) no human labor, 2022

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor, 2022. URL https://arxiv.org/abs/2212. 09689

    2022

  3. [11]

    Evaluating the zero-shot robustness of instruction-tuned language models, 2023

    Jiuding Sun, Chantal Shaib, and Byron C Wallace. Evaluating the zero-shot robustness of instruction-tuned language models, 2023. URL https://arxiv.org/abs/2306.11270

    2023 arXiv

  4. [12]

    Quantifying language models’ sensitivity to spurious features in prompt design, or: How I learned to start worrying about prompt formatting, 2023

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design, or: How I learned to start worrying about prompt formatting, 2023. URL https://arxiv.org/abs/2310.11324

    2023 arXiv

  5. [13]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  6. [14]

    Spoc: Search-based pseudocode to code

    Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. In 32nd An- nual Conference on Advances in Neural Information Processing Systems (NeurIPS) ,

  7. [15]

    AI-powered fuzzing: Breaking the bug hunting barrier, 2023

    Jonathan Metzman Dongge Liu and Oliver Chang. AI-powered fuzzing: Breaking the bug hunting barrier, 2023. URL https://security.googleblog.com/2023/08/ ai-powered-fuzzing-breaking-bug-hunting.html

    2023

  8. [16]

    OpenAI o1 system card, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. OpenAI o1 system card, 2024. URL https://arxiv.org/abs/2412.16720

    2024 arXiv

  9. [17]

    OpenAI o3-mini system card, 2025

    OpenAI. OpenAI o3-mini system card, 2025. URL https://cdn.openai.com/ o3-mini-system-card-feb10.pdf

    2025

  10. [18]

    Claude 3.7 Sonnet system card, 2025

    Anthropic. Claude 3.7 Sonnet system card, 2025. URL https://anthropic.com/ claude-3-7-sonnet-system-card

    2025

  11. [19]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity, 2021

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity, 2021. URL https://arxiv.org/abs/2104.08786

    2021 arXiv

  12. [20]

    Posix: A prompt sensitivity index for large language models, 2024

    Anwoy Chatterjee, HSVNS Kowndinya Renduchintala, Sumit Bhatia, and Tanmoy Chakraborty. Posix: A prompt sensitivity index for large language models, 2024. URL https://arxiv. org/abs/2410.02185

    2024 arXiv

  13. [21]

    DOVE: A large-scale multi-dimensional predictions dataset towards meaningful LLM evaluation, 2025

    Eliya Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, and Gabriel Stanovsky. DOVE: A large-scale multi-dimensional predictions dataset towards meaningful LLM evaluation, 2025. URL https://arxiv.org/abs/2503. 01622

    2025

  14. [22]

    Demystifying prompts in language models via perplexity estimation, 2022

    Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation, 2022. URL https://arxiv.org/abs/ 2212.04037

    2022 arXiv

  15. [23]

    Measuring coding challenge competence with APPS

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In35th Annual Conference on Neural Information Processing Systems (Ne...

    2021 arXiv

  16. [24]

    NYU CTF bench: A scalable open-source benchmark dataset for evaluating LLMs in offensive security

    Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. NYU CTF bench: A scalable open-source benchmark dataset for eval...

    2024

  17. [25]

    OpenAI o3 and o4-mini system card, 2025

    OpenAI. OpenAI o3 and o4-mini system card, 2025. URL https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf

    2025

  18. [26]

    A framework for evaluating emerging cyberattack capabilities of ai

    Mikel Rodriguez, Raluca Ada Popa, Four Flynn, Lihao Liang, Allan Dafoe, and Anna Wang. A framework for evaluating emerging cyberattack capabilities of ai. arXiv preprint arXiv:2503.11917, 2025

    2025 arXiv

  19. [27]

    Amazon Bedrock

    Amazon Web Services. Amazon Bedrock. https://aws.amazon.com/bedrock/, 2023. Accessed: 2025-05-12

    2023

  20. [28]

    Openai api

    OpenAI. Openai api. https://openai.com/api/, 2021. Accessed: 2025-05-12. 12

    2021

  21. [29]

    Azure OpenAI service

    Microsoft Azure. Azure OpenAI service. https://azure.microsoft.com/en-us/ products/ai-services/openai-service , 2023. Accessed: 2025-05-12

    2023

  22. [30]

    pickle — python object serialization, 2025

    Python Software Foundation. pickle — python object serialization, 2025. URL https: //docs.python.org/3/library/pickle.html

    2025

  23. [31]

    t2.large

    Anthropic. Model card addendum: Claude 3.5 Haiku and upgraded Claude 3.5 Son- net, 2024. URL https://assets.anthropic.com/m/1cd9d098ac3e6467/original/ Claude-3-Model-Card-October-Addendum.pdf . 13 Appendices A Model settings and resources To investigate the phenomenon of incon...

    2024

  24. [34]

    ‘app.py‘: The server-side application that powers the CTF challenge

  25. [35]

    ‘solution.py‘: A working exploit script that successfully solves the challenge ## Detailed Requirements ### Theme Transformation * Develop a completely new thematic context * Replace all theme-specific terminology, visuals, and narrative elements * Create appropriate themed fe...

  26. [36]

    Complete code for the transformed ‘app.py‘

  27. [37]

    A title for your new theme (CamelCase format, maximum 4 words)

  28. [38]

    In this appendix, we provide the instructions given to these experts

    Brief description of your architectural approach (maximum 5 words) ## Constraints * Avoid making only superficial thematic changes to the provided code - Use a completely different design pattern * **Banned themes**: {previously used themes placeholder} * **Banned design patte...

  29. [39]

    Select a failing solution of the Ships variant, such as the one on the left side of Table 5

  30. [40]

    Identify the first point of failure in the solution, such as the first position in which a term is incorrectly transferred (in our case, the line marked red in Table 5)

  31. [41]

    Translating

    Take a prefix ending somewhat earlier than the first point of failure. In our experiment, we ended the prefix some 50 tokens prior to the failure. This prefix is thus a correct partial solution of the challenge. 20 Table 4: Two equivalent math problems with varying success rat...

    2024

  32. [42]

    Let’s define variables: * Let x = cruise ship’s speed in mph * Then (x-20) = sailboat’s speed in mph * Distance = 480 miles for both vessels

  33. [43]

    ) Let me solve this step by step

    Multiply both sides by x: * 480x = (x-20)(480 + 2x) * 480x = 480x - 9600 + 2x ² - 40x * 0 = 2x ² - 520x - 9600 (Incorrect solution continues. . . .) Let me solve this step by step

  34. [44]

    Let’s define variables: * Let x = first caterpillar’s eating rate in leaves per day * Then (x-20) = second caterpillar’s eating rate in leaves per day * Total leaves = 480 for both caterpillars

  35. [45]

    Multiply both sides by x: (LLM to continue response here.)

  36. [46]

    mph” to “leaves per day,

    Translate the prefix to the setting of the Caterpillars variant. In particular, replace every occurrence of “mph” to “leaves per day,” every occurrence of “speed” with “eating rate,” and so on. The result is a valid solution prefix for the Caterpillars variant, and can be seen...

  37. [47]

    Sample the results multiple times to obtain an average success rate

    Run the LLM on the Caterpillars variant, but prefill the response with the solution prefix. Sample the results multiple times to obtain an average success rate. F.2 Results and conclusions When running the Caterpillars variant with the prefilled solution prefix, the model obta...

  38. [2019]

    URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ 7298332f04ac004a0ca44cc69ecf6f6b-Paper.pdf

    2019

  39. [2024]

    URL https://arxiv.org/abs/2405.19740

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.