Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Semantically faithful prompt rewrites produce large, unpredictable accuracy swings in LLM code generation: drops up to 42.1%, gains up to 35.3%.

desk verdict Interesting dataset and a few honest case studies, but the central accuracy claims are undermined by invalid negation oracles and internally inconsistent tables. read the letter →

arxiv 2506.06971 v2 pith:YFC6SQ5H submitted 2025-06-08 cs.CL cs.CR

Jaechul Roh , Varun Gandhi , Shivani Anilkumar , Arin Garg This is my paper
classification cs.CLcs.CR
keywords Chain-of-CodeCollapseLLMreasoningrobustnesscodegenerationadversarialpromptingsemanticperturbationpromptnegationobjectivePass@1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chain-of-Code Collapse asks whether LLMs that write code from natural-language prompts reason stably when the same problem is reframed in a different surface form. The paper builds 700 perturbed prompts from 100 LeetCode-style problems, rewrites that are meant to preserve the underlying logic while changing narrative or structure, and measures Pass@1 accuracy on nine models. It finds no uniform effect: some rewrites drop accuracy by as much as 42.1 percentage points, while others raise it by up to 35.3 points. The stake is practical and diagnostic: if meaning-preserving rewrites can flip results this much, clean-accuracy rankings are not a reliable measure of reasoning robustness.

What carries the argument

The carrying mechanism is the CoCC perturbation framework: a set of prompt templates that instruct a rewrite model to keep Input, Output, Explanation, Examples, and Constraints intact while applying storytelling, gamification, domain shift, example perturbation, distracting constraints, or negation. A Logical Preservation Score, assigned by an LLM judge on a 0-10 rubric, estimates how much each rewrite drifted from the original logic, and Pass@1 with execution-based tests measures whether the model still solves the intended problem. This setup is what lets the paper attribute accuracy changes to surface phrasing rather than to a different task.

What would settle it

Take a negated prompt, such as count passengers aged 60 or less, generate fresh unit tests from that rewritten objective only, and re-score all models with those tests; if correctly negated solutions pass, the reported accuracy collapse is an artifact of stale test cases rather than a reasoning failure.

Watch

Extended reading notes

Core claim

The central discovery is that semantically faithful prompt perturbations produce large, unpredictable swings in code-generation accuracy across current LLMs. Storytelling, gamification, and domain shifts often help or barely hurt, while distracting constraints and negation can cut accuracy by more than half; the same perturbation type can help one model and severely hurt another. The paper reads this as evidence that high accuracy on clean prompts does not imply robust reasoning, and it proposes semantic perturbation robustness as a new evaluation axis: the standard deviation of accuracy across logic-preserving rewrites of the same task.

Load-bearing premise

The central assumption is that each rewritten prompt still means the original problem, so the original test cases are fair judges of whether the model solved the intended task.

Editorial extensions

If this is right

  • A high clean-accuracy score cannot be read as evidence that a model will hold up under the normal linguistic variation of real user prompts.
  • Rewrites that add narrative structure or concrete context can improve generation for some models, suggesting that sparse, minimal problem statements may underuse model capability.
  • Benchmark reporting should include variance across semantically equivalent rephrasings, not just a single accuracy number.
  • Objective-inverting rewrites like negation need newly derived test cases; without them, observed failures are uninterpretable as reasoning failures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The negation experiments, where test cases still expected original answers, imply that part of the reported collapse is an evaluation artifact: models that faithfully executed the rewritten objective were marked wrong, and extending the framework to include freshly generated tests per rewrite is the natural next test.
  • The same perturbation suite could be reused as a stress test for math word problems, retrieval, or agent planning, where same-task, different-words failures are equally consequential.
  • The observed gains for some models under storytelling suggest that prompt-surface tuning is model-specific: a prompt that scaffolds one model may distract another.
  • If semantic perturbation robustness becomes standard practice, a model's advertised capability should be its accuracy distribution over equivalent phrasings rather than a single point estimate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces Chain-of-Code Collapse (CoCC), a perturbation framework for code-generation LLMs. The authors rewrite 100 LiveCodeBench LeetCode problems using six perturbation families (storytelling, gamification, domain shift, example perturbation, distracting constraints, hard/soft negation), evaluate nine LLMs with Pass@1, and report that semantically faithful rewrites can cause large accuracy drops (up to -42.1%) and gains (up to +35.3%). They interpret these swings as evidence of reasoning fragility and propose a new evaluation axis, semantic perturbation robustness. The paper also releases its perturbation datasets and evaluation framework.

Significance. If the empirical claims were well-supported, this would be a timely and useful stress test for code LLMs, complementing GSM-Symbolic in the code domain and providing a concrete benchmark for robustness under surface-level reformulation. The authors are transparent about limitations in their appendices and release their data, which is commendable. However, the central quantitative claims are undermined by internally inconsistent tables, an admitted oracle-misalignment problem for the negation experiments, and a circular use of evaluated models for preservation scoring. As presented, the paper does not yet substantiate the conclusion that LLM code generation is fragile under semantically faithful perturbations.

major comments (5)
  1. [§5.2, Tables 3–5] The same experimental cells are reported with incompatible numbers. Claude-3.7-Sonnet under Storytelling is 63.35% in Table 3, but Table 4 lists per-difficulty accuracies of 48.5%, 34.0%, and 47.1%; using the 33/50/17 difficulty split implied by Table 2 (100.0/92.0/64.7 yielding 90.0 overall) gives roughly 41%, not 63.35%. Similarly, Domain Shift for Claude is 35.66% in Table 3, but the Table 4 rows (35.7/24.0/11.8) weight to about 25.8%. In Table 7, Qwen2.5-Coder hard-negation rows 33.3/20.0/23.5 cannot produce the reported "Overall Acc." of 47.2%. Because no aggregation formula or raw counts are given, these discrepancies make the headline deltas (-42.1%, +35.3%) unverifiable.
  2. [§6.3, Appendices A.2, B.1, B.2] The paper explicitly concedes that the negation results do not measure what they claim. Section 6.3 states that hard-negation "reported accuracies (0%−17%) not reflecting genuine capability" due to "pervasive test case misalignment," and the appendices show models that implemented the correct negated logic (e.g., age ≤ 60) being marked wrong because the test still expected the original answer, while models that ignored the rewrite passed. Yet Table 7 reports these numbers as accuracy, the abstract cites large drops, and the conclusion treats negation as evidence of reasoning collapse. This internal contradiction invalidates the negation-based portion of the evaluation and any aggregate claims that include it.
  3. [§3.1–3.3, §4.1] The evaluation never establishes that the original LiveCodeBench test cases remain correct labels for the perturbed prompts. The methodology reuses the original test cases for every perturbation type, with no per-instance check that the rewritten problem has the same expected outputs. Appendix A.2 provides a concrete counterexample: under soft negation of sumOfSquares, the correct answer for the rewritten task is 9 but the test expects 21; Llama's faithful implementation was marked wrong, and DeepSeek's ignore-the-rewrite solution was marked correct. Without a validation protocol for oracle alignment, every accuracy delta in Tables 3–7 is a mixture of model skill and test-harness mismatch.
  4. [§3.3, Appendix F] The "semantically faithful" label rests on preservation scores assigned by Claude-3.7-Sonnet, which is itself one of the evaluated models, and the rewrites were generated by LLaMA-3.1-8B-Instruct, also an evaluated model. No human validation or inter-annotator agreement is reported. Since the central claim is that perturbations are logically equivalent to the originals, using an evaluated LLM as the arbiter of that equivalence is circular. The authors should report a human-validated subset (e.g., 50–100 instances) with agreement rates, and should re-score or exclude cases where preservation is disputed.
  5. [§4, Tables 2–7] All results are point estimates from a single pass@1 run per problem with no confidence intervals, seeds, decoding parameters, or number of samples reported. With 100 problems, a one-problem change is 1%, so many deltas (e.g., +0.5%, -1.9%, +1.3%) are within sampling noise. The claim that some models are "remarkably stable (<5% deviation)" cannot be assessed without variance estimates. Repeated runs or bootstrap confidence intervals are needed before interpreting the magnitude of any swing.
minor comments (6)
  1. [Abstract, §5.2, §7] Typos and grammatical issues: "principles approaches" should be "principled approaches" (Abstract), "underutlize" should be "underutilize" (§5.2), "ON average" should be "On average" (§5.1), "large-scle" should be "large-scale" (§7), and "our of" should be "out of" (§7).
  2. [Table 5] The model name "LLaMA3-8B-Instruct" is inconsistent with "LLaMA-3.1-8B-Instruct" used elsewhere; the same model should be named consistently throughout.
  3. [Table 1, §6.1] Table 1 lists preservation scores for five attack types but omits the Negation Objective scores (1.83 hard, 2.56 soft) that are later reported in §6.1; include them in the table or clarify why they are excluded.
  4. [§1, §3.2] The paper says "six distinct perturbation methods" in the introduction but later lists seven perturbation types (storytelling, gamification, domain shift, example perturbation, distracting constraints, hard negation, soft negation). Please clarify the counting.
  5. [References] The reference to "Cai et al. (2024)" appears to correspond to arXiv:2503.21961, which is dated 2025; update the citation year and check all preprint dates for consistency.
  6. [Figure 1] Figure 1 is not explicitly referenced in the text; add a reference to it in Section 1 or 3 so readers know when to consult it.

Circularity Check

2 steps flagged · score 4.0 of 10

The paper's own Section 6.3 concedes that the negation accuracies are artifacts of un-updated test oracles, and the 'semantically faithful' framing is partly self-referential because preservation is scored by an evaluated model; however, the headline Table 3 deltas have independent content.

  1. self definitional [Section 3.1 vs. Section 6.1/6.3; Table 7; Appendices B.1-B.2]
    "Our method centers on applying meaning-preserving or minimally diverging rewrites to problems, without altering their underlying logic or difficulty.... Negation Objective (Hard). This method fully inverts the core task (e.g., 'maximize'→'minimize' or 'include'→'exclude')."

    The paper defines its perturbation suite as meaning-preserving, yet hard and soft negation are explicitly objective-inverting, and the evaluation retains the original test oracle. Appendix B.2 shows models implementing the negated condition age <= 60 produce the correct answer for the rewritten task (1) but are marked wrong against the original expected value (2), while Appendix B.1 shows a model 'passing' by ignoring the negation and solving the original problem. Thus the reported negation accuracies are, by construction, accuracies on the original task rather than on the rewritten task, and Section 6.3 itself concedes the numbers 'do not reflect genuine capability.' Those rows are nevertheless used in Section 5.4 and the Conclusion to support reasoning-collapse claims.

  2. other [Section 3.3, Section 5.4, Figure 3]
    "We employed the Claude-3.7-Sonnet model as an automated evaluator to ensure consistent and scalable scoring across all 700 perturbed instances."

    The 'semantically faithful' input premise is operationalized by Claude-3.7-Sonnet, which is itself one of the nine evaluated models and the model with the largest accuracy drops in Table 3. No human validation of the preservation scores is reported. The Section 5.4 accuracy-versus-preservation correlation is therefore partly self-referential: the same model family supplies both the preservation labels and the measured behavior. This does not directly feed into the Table 3 accuracy deltas, so the circularity burden is moderate rather than total.

full rationale

The paper's most public quantitative claims are not fully circular: the -42.1% drop and +35.3% gain quoted in the abstract come from Distracting Constraints and Example Perturbation in Table 3, where the original test cases are the correct oracle, and those accuracy numbers do not use the preservation scores as algorithmic inputs. The circularity is concentrated in two places. First, two of the seven perturbation types (hard and soft negation) reuse original expected outputs while instructing models to invert the task; the paper's own Section 6.3 and Appendices B.1-B.2 show that correct negated implementations are marked wrong and original implementations are marked right, so the reported negation collapse is a measurement artifact, not evidence about reasoning on the rewritten task. Second, the 'semantically faithful' characterization is assigned by Claude-3.7-Sonnet, an evaluated model, without reported human validation, making the Section 5.4 preservation correlation partly self-referential. Because the main perturbation-type results retain independent content and the negation invalidity is explicitly acknowledged inside the paper, the appropriate score is 4 rather than 6 or higher.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper makes no parameterized derivation; it is an empirical benchmark. The free-parameter ledger is empty because the only numeric choices are design choices (100 problems, seven perturbation templates, preservation rubric thresholds). The central claims rest instead on unverified assumptions about perturbation equivalence and test-oracle validity, listed below.

assumptions (5)
  • ad hoc to paper Each perturbed problem statement is logically equivalent to the original, so original test cases remain correct labels.
    Section 3.1 asserts preservation for all transformations, but Section 6.3 and Appendices A.2 and B.1-B.2 show negation rewrites were evaluated against un-updated test cases, making the assumption false for a major perturbation class.
  • ad hoc to paper Logical Preservation Scores from Claude-3.7-Sonnet are a valid measure of semantic drift.
    Section 3.3 uses Claude-3.7-Sonnet as the automated judge with a rubric but no human validation or inter-rater agreement; Claude is also one of the evaluated models.
  • domain assumption Pass@1 computed by execution against original test cases measures success on the perturbed task.
    Section 4.1 adopts Pass@1; this only measures the perturbed task if test oracles are aligned, which the paper shows is not always the case.
  • domain assumption A single generation per problem is representative of model performance.
    No sampling temperature, number of runs, or confidence intervals are reported, so the percentages in Tables 2-7 are treated as stable point estimates.
  • domain assumption The curated 100-problem LeetCode subset is representative enough for the conclusions.
    Section 4.1 describes the subset as curated but gives no selection criteria, and Section 7 lists this restriction as a limitation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation." pith.science (2026). https://pith.science/paper/YFC6SQ5H

@misc{pith2026250606971,
  author       = {Pith},
  title        = {Pith review of: Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YFC6SQ5H}},
  note         = {Machine review of arXiv:2506.06971}
}
read the original abstract

Large Language Models (LLMs) have achieved remarkable success in tasks requiring complex reasoning, such as code generation, mathematical problem solving, and algorithmic synthesis -- especially when aided by reasoning tokens and Chain-of-Thought prompting. Yet, a core question remains: do these models truly reason, or do they merely exploit shallow statistical patterns? In this paper, we introduce Chain-of-Code Collapse, where we systematically investigate the robustness of reasoning LLMs by introducing a suite of semantically faithful yet adversarially structured prompt perturbations. Our evaluation -- spanning 700 perturbed code generations derived from LeetCode-style problems -- applies transformations such as storytelling reframing, irrelevant constraint injection, example reordering, and numeric perturbation. We observe that while certain modifications severely degrade performance (with accuracy drops up to -42.1%), others surprisingly improve model accuracy by up to 35.3%, suggesting sensitivity not only to semantics but also to surface-level prompt dynamics. These findings expose the fragility and unpredictability of current reasoning systems, underscoring the need for more principles approaches to reasoning alignments and prompting robustness. We release our perturbation datasets and evaluation framework to promote further research in trustworthy and resilient LLM reasoning.

Figures

Figures reproduced from arXiv: 2506.06971 by the authors.

Figure 1
Figure 1. Overview of CHAIN-OF-CODE COLLAPSE structure intact. Their results revealed sig￾nificant accuracy drops under benign modifica￾tions—suggesting that reasoning performance is highly sensitive to surface form, even when task semantics remain unchanged. In this work, we extend this line of inquiry to the domain of code generation. We introduce CHAIN-OF-CODE COLLAPSE (CoCC), a system￾atic framework for evaluating reasoni… view at source ↗
Figure 2
Figure 2. Prompt template for Storytelling transformation. 4.1 Dataset and Evaluation Metrics We conduct our experiments using the Live￾CodeBench dataset (Jain et al., 2024), a recent benchmark designed to evaluate LLM perfor￾mance on code generation under interactive and test-driven settings. For our study, we select a curated subset of 100 LeetCode problems from the Code Generation split of the LiveCodeBench, which provides… view at source ↗
Figure 3
Figure 3. Accuracy vs. Logical Preservation Score for [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Prompt template for Gamification transformation. F Preservation Score Prompt Template [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Prompt template for Example Perturbation transformation. Distracting Constraints Prompt Template Instruction: Add irrelevant but realistic constraints to the problem. Guidelines: • Do not alter solvability. • Insert edge cases, meaningless jargon, or constraints that d…
Figure 6
Figure 6. Figure 6: Prompt template for Distracting Constraints transformation [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Prompt template for Domain Shift transformation. Hard Negation Objective Prompt Template Instruction: Invert the objective of the coding task. Examples: • Max → Min • Increasing → Not Increasing • Include → Exclude Prompted Task: Rewrite the problem with the opposite i…
Figure 8
Figure 8. Figure 8: Prompt template for Hard Negation Objective transformation. Soft Negation Prompt Template Instruction: Apply a light semantic reversal without changing the task’s core logic. Examples: • Non-decreasing instead of increasing • Non-minimum instead of minimum Guidelines: …
Figure 9
Figure 9. Figure 9: Prompt template for Soft Negation transformation [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Prompt template for evaluating logical preservation between original and modified coding problems [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models

    cs.CL 2025-09 conditional novelty 4.0 of 10

    A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.

  2. LLM Harms: A Taxonomy and Discussion

    cs.CY 2025-12 unverdicted novelty 3.0 of 10

    This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.

Reference graph

Works this paper leans on

33 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Claude 3.5 sonnet model card addendum

    Anthropic (2024). Claude 3.5 sonnet model card addendum. https://www.anthropic.com/news/claude-3-5-sonnet. Accessed: 2025-05-12

    work page 2024

  4. [4]

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021). Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  5. [5]

    Bai, Y., Kadavath, S., Kundu, S., et al. (2023). Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2306.04761

    work page arXiv 2023

  6. [6]

    B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., et al

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems , 33

    work page 2020

  7. [7]

    Cai, Y., Donhauser, J., Salemi, A., McDowell, P., Tian, Y., and Ghassemi, M. (2024). Attention misalignment: Modeling errors in language models through entropy and variance. arXiv preprint arXiv:2503.21961

    arXiv 2024

  8. [8]

    S., Ri, N., Xiao, Y., and Ghassemi, M

    Chan, Y. S., Ri, N., Xiao, Y., and Ghassemi, M. (2025). Speak easy: Eliciting harmful jailbreaks from llms with simple interactions. arXiv preprint arXiv:2502.04322

    arXiv 2025

Show all 33 references
  1. [9]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  2. [10]

    Chen, W., Ma, X., Wang, X., and Cohen, W. W. (2022). Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588

    2022 arXiv

  3. [11]

    Chowdhery, A., Narang, S., Devlin, J., et al. (2022). Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311

    2022 arXiv

  4. [12]

    DeepMind, G. (2024). Start building with gemini 1.5 and gemini 2.5. https://developers.googleblog.com/en/start-building-with-gemini-25-flash/. Accessed: 2025-05-12

    2024

  5. [13]

    Dou, S., Jia, H., Wu, S., Zheng, H., Zhou, W., Wu, M., Chai, M., Fan, J., Huang, C., Tao, Y., Liu, Y., Zhou, E., Zhang, M., Zhou, Y., Wu, Y., Zheng, R., Wen, M., Weng, R., Wang, J., Cai, X., Gui, T., Qiu, X., Zhang, Q., and Huang, X. (2024). What's wrong with your code generat...

    2024

  6. [14]

    Gan, E., Zhao, Y., Cheng, L., Mao, Y., Goyal, A., Kawaguchi, K., Kan, M.-Y., and Shieh, M. (2024). Reasoning robustness of llms to adversarial typographical errors. arXiv preprint arXiv:2411.05345

    2024 arXiv

  7. [15]

    Golovneva, O., Chen, M., Poff, S., Corredor, M., Zettlemoyer, L., Fazel-Zarandi, M., and Celikyilmaz, A. (2022). Roscoe: A suite of metrics for scoring step-by-step reasoning. arXiv preprint arXiv:2212.07919

    2022 arXiv

  8. [16]

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  9. [17]

    Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. (2024). Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  10. [18]

    Jiang, J., Wang, F., Shen, J., Kim, S., and Kim, S. (2024). A survey on large language models for code generation. arXiv preprint arXiv:2406.00515v2

    2024 arXiv

  11. [19]

    Introducing llama 3.1: Our most capable models to date

    Meta (2024). Introducing llama 3.1: Our most capable models to date. Meta-AI-blog

    2024

  12. [20]

    Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. (2024). Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229

    2024 arXiv

  13. [21]

    and Plank, B

    Mondorf, P. and Plank, B. (2024). Beyond accuracy: Evaluating the reasoning behavior of large language models-a survey. corr, abs/2404.01869, 2024. doi: 10.48550. arXiv preprint ARXIV.2404.01869

    2024 arXiv

  14. [22]

    Mu, W., Xu, L., Pei, S., Mi, L., and Zhou, H. (2025). Evaluate-and-purify: Fortifying code language models against adversarial attacks using llm-as-a-judge. https://arxiv.org/abs/2504.19730

    2025 arXiv

  15. [23]

    Gpt-4 technical report

    OpenAI (2023). Gpt-4 technical report. https://openai.com/research/gpt-4. Accessed: 2025-05-12

    2023

  16. [24]

    Prasad, A., Saha, S., Zhou, X., and Bansal, M. (2023). Receval: Evaluating reasoning chains via correctness and informativeness. arXiv preprint arXiv:2304.10703

    2023 arXiv

  17. [25]

    Shen, X., Wu, Y., Backes, M., and Zhang, Y. (2024). Voice jailbreak attacks against gpt-4o. arXiv preprint arXiv:2405.19103

    2024 arXiv

  18. [26]

    Song, X., Xie, Z., Huai, S., Kong, J., and Luo, J. (2025). Dagger behind smile: Fool llms with a happy ending story. arXiv preprint arXiv:2501.13115

    2025

  19. [27]

    and Zhang, T

    Tong, W. and Zhang, T. (2024). Codejudge: Evaluating code generation with large language models. arXiv preprint arXiv:2410.02184

    2024 arXiv

  20. [28]

    K.-W., and Lim, E.-P

    Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., and Lim, E.-P. (2023). Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091

    2023 arXiv

  21. [29]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824--24837

    2022

  22. [30]

    Wu, F., Liu, X., and Xiao, C. (2023). Deceptprompt: Exploiting llm-driven code generation via adversarial natural language instructions. arXiv preprint arXiv:2312.04730

    2023 arXiv

  23. [31]

    V., and Xiong, C

    Xu, Y., Yaghini, M., Ogutu, K., Sun, Z., Lin, X. V., and Xiong, C. (2024). Prompting under control: Structured prompts guide language models better. arXiv preprint arXiv:2412.20545

    2024 arXiv

  24. [32]

    Zhu, K., Zhao, Q., Chen, H., Wang, J., and Xie, X. (2024). Promptbench: A unified library for evaluation of large language models. Journal of Machine Learning Research , 25(254):1--22

    2024

  25. [33]

    Zhu, Y., Li, G., Jiang, X., Li, J., Mei, H., Jin, Z., and Dong, Y. (2025). Uncertainty-guided chain-of-thought for code generation with llms. arXiv preprint arXiv:2503.15341

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.