Pith. sign in

REVIEW 5 major objections 4 minor 34 references

LLM Performance for Code Generation on Noisy Tasks

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that LLMs solve heavily obfuscated coding and math tasks through memorisation and eager pattern matching, not genuine reasoning, and that the rate of performance decay under obfuscation can serve as a practical detector…

desk verdict A real, reproducible asymmetry in how LLMs handle obfuscated old vs. new LeetCode tasks; the decay-as-contamination detector is plausible but unproven because the comparison isn't difficulty-matched. read the letter →

arxiv 2505.23598 v1 pith:HWOIUU35 submitted 2025-05-29 cs.LG cs.SE

Radzim Sendyka , Christian Cabrera , Andrei Paleyes , Diana Robinson , Neil Lawrence This is my paper
classification cs.LGcs.SE
keywords LLMevaluationdatasetcontaminationobfuscationcodegenerationmemorisationeagerpatternmatchingperformancedecaybenchmarkreliability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to establish that large language models often solve code-generation and math tasks by memorisation rather than reasoning. The evidence: when tasks from older benchmarks are obfuscated with typos, deletions, or truncation until no human can read them, the models still answer correctly; when the same treatment is applied to tasks published after the models' knowledge cutoff, accuracy collapses. The behaviour is named 'eager pattern matching' and is held responsible for the models' ability to complete an unintelligible prompt by filling in a familiar memorised solution. The authors also propose measuring performance decay under obfuscation as a practical signal for detecting dataset contamination, and show that old and new LeetCode tasks decay at clearly different rates. If the paper is right, standard benchmark scores can overstate genuine capability and should be treated with caution.

What carries the argument

The machinery is an obfuscation pipeline that applies three augmentation methods (typos, random word deletions, and truncation) at ten severity levels to twenty tasks per dataset, creating 600 obfuscated variants that are scored by automated test cases. The load-bearing quantity is the performance-decay curve, summarised by the 50%-decay augmentation rate, which the paper uses to separate contaminated from uncontaminated content. The named behaviour 'eager pattern matching' is the paper's explanation for why models succeed where humans cannot: instead of reconstructing the question, the model pattern-completes toward a memorised exemplar, a mechanism tested with adversarial tasks that resemble familiar problems but ask a different question.

What would settle it

Fine-tune a model on a set of post-cutoff tasks, re-run the obfuscation sweep, and check whether the decay curve flattens to the OldLC shape; if it stays steep, gentler decay is not caused by memorisation and the proposed contamination detector fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that all five evaluated LLMs solve tasks obfuscated beyond human recognition when those tasks predate the models' training data, and that this success is an artefact of memorisation, not reasoning. On LeetCode tasks published in 2015, models continued to produce correct solutions even at maximum obfuscation, while not one model solved any of the 2025 task versions obfuscated above a 0.7 augmentation rate. The authors introduce 'eager pattern matching' to describe the mechanism: the model seizes on residual stylistic cues and answers with a complete solution to a similar training problem, which they demonstrate with adversarial variants of known tasks that all models solved incorrectly by reproducing the memorised answer. They report that the augmentation rate causing 50% performance loss is roughly twice as high on the old dataset as on the new one, and treat this decay contrast as evidence that the old tasks are in the training set.

Load-bearing premise

The comparison that carries the contamination conclusion assumes old and new LeetCode tasks are interchangeable except for publication date, but the new tasks already score far lower without obfuscation, so their faster decay could come from being harder rather than from being unseen during training.

Editorial extensions

If this is right

  • Benchmark scores on static, widely copied datasets can materially overstate how well an LLM can reason about unfamiliar problems.
  • A model that eagerly matches patterns may produce confident, wrong solutions when a new task superficially resembles a training problem, creating a safety hazard for automated code generation.
  • Measuring the obfuscation-decay curve offers a low-cost, model-agnostic check for whether a dataset is likely to be in a model's training data.
  • Evaluations on newly released tasks, and dynamic benchmarks that avoid reuse, become necessary for trustworthy capability claims.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not run the decisive control of deliberately training a model on a post-cutoff task and re-measuring its decay curve; if the curve then flattened, the memorisation mechanism would be confirmed, and if not, the decay contrast would be explained by task difficulty rather than contamination.
  • A practical contamination detector would need to separate decay due to memorisation from decay due to question length or style; fitting per-model decay curves on more datasets, as the paper itself suggests, is a testable next step.
  • The eager-pattern-matching result implies that human review of AI-generated code may be insufficient when the input prompt resembles a training example, because the model will confidently reproduce the wrong solution; a natural extension is testing such adversarial prompts in an IDE-style assistant.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper studies how LLMs behave on obfuscated competitive-programming and math tasks. It compares five LLMs on three datasets: OldLC (LeetCode questions from 2015), NewLC (LeetCode questions from March 2025, after the models' knowledge cutoffs), and MATH (2021). Using three obfuscation methods (Typos, Deletions, Truncation) at ten augmentation levels, the authors measure accuracy decay and find that OldLC and MATH tasks are solved even under extreme obfuscation while NewLC tasks are not, concluding that LLMs rely on memorisation and 'eager pattern matching' rather than reasoning. The paper proposes performance-decay statistics (e.g., 50% decay point) as a potential contamination detector, supports this with adversarial examples that show LLMs reverting to known problems, and discusses safety and evaluation implications. The repository with code and data is provided.

Significance. If the central empirical contrast is trustworthy, the paper makes a useful contribution: it provides a concrete, reproducible procedure (obfuscation + decay analysis) for probing contamination, releases code and data, and includes adversarial tasks that go beyond simple memorisation checks. The observation that LLMs solve tasks that humans judge unintelligible is striking and worth reporting. However, the paper's own conclusions are more cautious than its abstract: Section VI concedes that confounding factors cannot be ruled out with three summary statistics. The significance therefore hinges on whether the OldLC/NewLC comparison can be made difficulty-matched, and whether the correctness metric is strong enough to support the claims.

major comments (5)
  1. [IV.A, Table II, Table I] The central contamination signal is the contrast between slow decay on OldLC (50% decay at 0.70 ± 0.08) and fast decay on NewLC (50% decay at 0.29 ± 0.06). This comparison is confounded by baseline difficulty: Table II shows that non-obfuscated accuracy on NewLC (0.19–0.45 across models) is far below OldLC (0.50–0.66). With a lower baseline, the same absolute accuracy drop occurs at a lower augmentation level simply because of floor effects; the 50% decay point is not difficulty-invariant. The paper states in IV.A that the two LeetCode sets are 'very similar tasks,' but that claim is not supported by the baseline gap. To make the decay-as-contamination claim load-bearing, the authors should either match tasks by difficulty, normalize decay relative to baseline accuracy, or show that the OldLC/NewLC difference persists on a difficulty-matched subset. The MATH results (high baseline, yet steep initial decay, 50% decay at 0.19) further show that baseline alone does not determine decay, but they also show that decay behavior is dataset-specific, so without controlling for difficulty the OldLC/NewLC contrast cannot isolate contamination.
  2. [III.D, IV.A] Correctness for coding tasks is judged on the 2–3 test cases embedded in the problem statement. This is a very weak proxy for a correct solution: a memorised solution that happens to pass the provided examples may be counted correct even if it does not solve the general problem. Given that the paper's headline claim is that LLMs 'solve' obfuscated tasks, the metric should be strengthened (e.g., a held-out test set from LeetCode's official test suite, or at least a larger set of generated test cases). The current metric allows the decay curves to be contaminated by false positives, and the contrast between OldLC and NewLC could be partly an artefact of OldLC tasks having more memorable input-output pairs.
  3. [IV.B, Table I] The interpretation of the MATH results is internally strained. The paper notes that MATH shows a steep initial decay (50% decay at 0.19), which 'could suggest a lower degree of overtraining or perhaps not being included in the training set at all,' but then rejects this possibility because tasks are still solved at high obfuscation, attributing the difference to question length and redundancy. This is an ad hoc explanation: if length/redundancy explains the MATH pattern, then it may also explain part of the OldLC/NewLC difference, since LeetCode and MATH are not matched on length, redundancy, or style. The decay detector therefore lacks a well-defined null model. The authors should either provide a quantitative model of how redundancy interacts with decay, or explicitly restrict the contamination-detection claim to comparisons within a matched dataset family.
  4. [III.F, Fig. 8] The human baseline measures self-reported understanding of obfuscated text, not actual problem-solving ability. The paper acknowledges this ('It is not directly comparable to actual performance'), but Figure 8 presents the human baseline as a decay curve alongside LLM decay, and the text uses it to argue that LLMs exceed 'human recognition' thresholds. Since the baseline is subjective and based on randomly selected examples rather than the full task set, it cannot support quantitative claims such as 'obfuscated beyond human recognition.' Either collect a more rigorous human performance baseline (e.g., asking participants to solve the tasks) or soften the wording to 'self-reported comprehensibility.'
  5. [VI, IV.A, V] The proposed contamination detector is essentially the same observation used to infer contamination: slow decay on assumed-contaminated OldLC versus fast decay on assumed-clean NewLC. There is no out-of-sample validation. The paper explicitly concedes (Section VI) that with only three sets of summary statistics, confounding factors cannot be ruled out. This is an honest limitation, but it should be reflected in the abstract and in the contributions list, which state that decay 'is a practical indicator of dataset contamination' and 'reveals stark differences between new and contaminated tasks.' As it stands, the paper demonstrates a correlation between release date and decay behavior on one pair of datasets; the detection claim is a hypothesis, not a validated method. The authors should either add a validation experiment (e.g., training a model on a known subset of an older dataset and testing on held-out versions) or explicitly reframe the contribution as a proposed diagnostic with preliminary evidence.
minor comments (4)
  1. [Throughout] Several typos and grammatical slips should be corrected: 'augmention' (IV.A), 'previosly' (Appendix D), 'contaminaton' (II.A), 'which is not surprising as it's currently top' (Appendix C.A).
  2. [Fig. 4] The adversarial example in Figure 4 has 'nums[2]' in the constraints, which appears to be a typo for 'nums2[i]'; this is potentially confusing for a reader trying to verify what the LLM was asked.
  3. [III.F] The description of the human baseline says '4 researchers with a high level of coding and math skills' but does not specify how many examples each participant saw, how the random selection was performed, or whether participants were aware of the original task content; please provide more detail for reproducibility.
  4. [References] Reference [33] appears to be miscited: the listed DOI (10.1145/3643657.3643910) belongs to the S4 workshop paper, not the Hawaii conference paper; please verify and correct.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the core obfuscation experiments are self-contained; the contamination-detector proposal is a conceded hypothesis rather than a fitted prediction.

full rationale

The paper's main empirical contributions are self-contained. The obfuscation pipeline, the collection of OldLC, NewLC, and MATH tasks, the model API evaluations, and the adversarial 'two medians' task are all external observations: no equation defines the target claim in terms of its own evidence, and no fitted parameter is renamed as a prediction. The OldLC-versus-NewLC decay contrast is a measured empirical difference, and the assumption that OldLC is contaminated rests on release-date reasoning independent of the decay measurement. The proposed decay-based contamination detector is presented as a hypothesis, not as a validated tool; Section VI explicitly concedes: 'with just three sets of summary statistics, we are unable to account for confounding factors or test any such detection systems.' That concession is a limitation and a validity gap, but it is not a definitional reduction. The paper also contains author self-citations ([32], [33]) supporting the intellectual-debt discussion, but these are not load-bearing for the experimental claims. Overall, there is no circular step that equates an input to an output by construction, so the circularity score is low.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The central claim rests on three domain assumptions: temporal contamination labels, test-case sufficiency, and the human self-report baseline. The proposed 'eager pattern matching' mechanism is an interpretive construct without direct measurement.

assumptions (3)
  • domain assumption OldLC tasks are included in the training data of all five LLMs, while NewLC tasks are not.
    Section IV-A states 'The OldLC dataset is almost certainly in the training set of all of the LLMs examined' and NewLC 'makes it unlikely for any of the questions to have been included in the training set.' This is not verified against actual training data, but is a reasonable temporal assumption.
  • domain assumption Passing all provided test cases (2-3 per problem) constitutes solving a coding task.
    Section III-D defines correctness as passing all test cases; with only 2-3 cases, incorrect algorithms may pass.
  • domain assumption Human researchers' self-reported ability to understand obfuscated tasks is a valid baseline for 'unintelligible to human readers.'
    Section III-F describes self-reported understanding percentages, not actual task-solving performance, and the authors note it is not directly comparable.
invented entities (1)
  • eager pattern matching
    purpose: Explains LLM tendency to solve obfuscated tasks by matching to previously seen problems rather than reasoning
    Introduced in the abstract and Section I; inferred from adversarial examples and decay patterns, but not directly measured or formally defined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Performance for Code Generation on Noisy Tasks." pith.science (2026). https://pith.science/paper/HWOIUU35

@misc{pith2026250523598,
  author       = {Pith},
  title        = {Pith review of: LLM Performance for Code Generation on Noisy Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HWOIUU35}},
  note         = {Machine review of arXiv:2505.23598}
}
read the original abstract

This paper investigates the ability of large language models (LLMs) to recognise and solve tasks which have been obfuscated beyond recognition. Focusing on competitive programming and benchmark tasks (LeetCode and MATH), we compare performance across multiple models and obfuscation methods, such as noise and redaction. We demonstrate that all evaluated LLMs can solve tasks obfuscated to a level where the text would be unintelligible to human readers, and does not contain key pieces of instruction or context. We introduce the concept of eager pattern matching to describe this behaviour, which is not observed in tasks published after the models' knowledge cutoff date, indicating strong memorisation or overfitting to training data, rather than legitimate reasoning about the presented problem. We report empirical evidence of distinct performance decay patterns between contaminated and unseen datasets. We discuss the implications for benchmarking and evaluations of model behaviour, arguing for caution when designing experiments using standard datasets. We also propose measuring the decay of performance under obfuscation as a possible strategy for detecting dataset contamination and highlighting potential safety risks and interpretability issues for automated software systems.

Figures

Figures reproduced from arXiv: 2505.23598 by the authors.

Figure 1
Figure 1. Example of an obfuscated task. This paper contributes to this discussion by studying the LLMs behaviour when solving problems under obfuscated inputs. We perform an exhaustive evaluation, adding varying levels of noise to common benchmark tasks, up to a point where they are no longer recognisable. These obfuscated tasks were supplied to LLMs, and their outputs were evaluated [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Examples of obfuscated tasks. C. Large Language Models The models were chosen to come from a diverse range of developers, sizes, and inference costs. To ensure our evalua￾tions of the NewLC datasets are not affected by contamination, we only considered models with knowledge cutoffs and release dates before March 2025. The set chosen represents the relative leaders of the industry [20], [21]. • Claude 3.5 Haiku, Anth… view at source ↗
Figure 3
Figure 3. Simplified experiment pipeline. F. Human baselines In order to establish a baseline for what the expected behaviour should be, we asked 4 researchers with a high level of coding and math skills to report what percentage of tasks they believed to able to understand under different obfuscation methods and levels, by showing them randomly selected examples. This is not the same metric as used in the LLM evaluation, as … view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Adversarial task resembling the structure of the “me [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The performance of LLMs on the two LeetCode [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 7
Figure 7. Figure 7: The average performance of selected LLMs on the [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 6
Figure 6. Figure 6: Comparison of the average performance decays of [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 8
Figure 8. Figure 8: Comparison of the average performance decays of [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]
Figure 9
Figure 9. Figure 9: Examples of obfuscated tasks along with solutions. [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: The average performance of selected LLMs on the 2 [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: The average performance of selected LLMs on the 20 [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 14
Figure 14. Figure 14: The average performance decay of LLMs on the [PITH_FULL_IMAGE:figures/full_fig_p009_14.png]
Figure 13
Figure 13. Figure 13: The average performance decay of LLMs on the [PITH_FULL_IMAGE:figures/full_fig_p009_13.png]
Figure 15
Figure 15. Figure 15: Stylistically similar question to the “median of two [PITH_FULL_IMAGE:figures/full_fig_p010_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 16 canonical work pages

  1. [1]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” 2024. [Online]. Available: https://arxiv.org/abs/2308.10620

    arXiv 2024

  2. [2]

    The current challenges of software engineering in the era of large language models,

    C. Gao, X. Hu, S. Gao, X. Xia, and Z. Jin, “The current challenges of software engineering in the era of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2412.14554

    arXiv 2024

  3. [3]

    Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms,

    Y . Xia, W. Shen, Y . Wang, J. K. Liu, H. Sun, S. Wu, J. Hu, and X. Xu, “Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms,” 2025. [Online]. Available: https://arxiv.org/abs/2504.14655

    arXiv 2025

  4. [4]

    Math word problem solving on math leaderboard,

    “Math word problem solving on math leaderboard,” https: //paperswithcode.com/sota/math-word-problem-solving-on-math, accessed: 2025-05-23

    work page 2025

  5. [5]

    A performance study of llm-generated code on leetcode,

    T. Coignion, C. Quinton, and R. Rouvoy, “A performance study of llm-generated code on leetcode,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, ser. EASE 2024. ACM, Jun. 2024, p. 79–89. [Online]. Available: http://dx.doi.org/10.1145/3661167.3661221

    arXiv 2024

  6. [6]

    Reduced, reused and recycled: The life of a dataset in machine learning research,

    B. Koch, E. Denton, A. Hanna, and J. G. Foster, “Reduced, reused and recycled: The life of a dataset in machine learning research,” 2021. [Online]. Available: https://arxiv.org/abs/2112.01716

    arXiv 2021

  7. [7]

    Leetcode dataset,

    NyanDoggo, “Leetcode dataset,” https://huggingface.co/datasets/ NyanDoggo/leetcode, 2025, accessed: 2025-05-23

    work page 2025

  8. [8]

    Measuring mathematical problem solving with the math dataset,

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the math dataset,” 2021. [Online]. Available: https://arxiv.org/abs/2103.03874

    arXiv 2021

Show all 34 references
  1. [9]

    Data contamination through the lens of time,

    M. Roberts, H. Thakur, C. Herlihy, C. White, and S. Dooley, “Data contamination through the lens of time,” 2023. [Online]. Available: https://arxiv.org/abs/2310.10628

    2023 arXiv

  2. [10]

    Dynabench: Rethinking benchmarking in nlp,

    D. Kiela, M. Bartolo, Y . Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams, “Dynabench: Rethinking benchmarking in nlp,” 2021. [Online]. Available:...

    2021 arXiv

  3. [11]

    Towards data contamination detection for modern large language models: Limitations, inconsistencies, and oracle challenges,

    V . Samuel, Y . Zhou, and H. P. Zou, “Towards data contamination detection for modern large language models: Limitations, inconsistencies, and oracle challenges,” 2024. [Online]. Available: https://arxiv.org/abs/2409.09927

    2024 arXiv

  4. [12]

    A comprehensive survey of contamination detection methods in large language models,

    M. Ravaut, B. Ding, F. Jiao, H. Chen, X. Li, R. Zhao, C. Qin, C. Xiong, and S. Joty, “A comprehensive survey of contamination detection methods in large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2404.00699

    2025 arXiv

  5. [13]

    Resilience of large language models for noisy instructions,

    B. Wang, C. Wei, Z. Liu, G. Lin, and N. F. Chen, “Resilience of large language models for noisy instructions,” 2024. [Online]. Available: https://arxiv.org/abs/2404.09754

    2024 arXiv

  6. [14]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,”

  7. [15]

    Impact of noise on llm-models performance in abstraction and reasoning corpus (arc) tasks with model temperature considerations,

    N. Khandalkar, P. Yadav, K. Shinde, L. B. Ramegowda, and R. Das, “Impact of noise on llm-models performance in abstraction and reasoning corpus (arc) tasks with model temperature considerations,”

  8. [16]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” 2018. [Online]. Available: https://arxiv.org/abs/1803.05457

    2018 arXiv

  9. [17]

    Datasets: A community library for natural language processing,

    Q. Lhoest, A. V . del Moral, Y . Jernite, A. Thakur, P. von Platen, S. Patil, J. Chaumond, M. Drame, J. Plu, L. Tunstall, J. Davison, M. ˇSaˇsko, G. Chhablani, B. Malik, S. Brandeis, T. L. Scao, V . Sanh, C. Xu, N. Patry, A. McMillan-Major, P. Schmid, S. Gugger, C. Delangue, T...

    2021 arXiv

  10. [18]

    Leetcode problemset,

    LeetCode, “Leetcode problemset,” 2025, accessed: 2025-05-25. [Online]. Available: https://leetcode.com/problemset/

    2025

  11. [19]

    Math augmented dataset,

    nivektk, “Math augmented dataset,” https://huggingface.co/datasets/ nivektk/math-augmented-dataset, 2021, accessed: 2025-05-27

    2021

  12. [20]

    Demand for llms: Descriptive evidence on substitution, market expansion, and multihoming,

    A. Fradkin, “Demand for llms: Descriptive evidence on substitution, market expansion, and multihoming,” 2025. [Online]. Available: https://arxiv.org/abs/2504.15440

    2025 arXiv

  13. [21]

    Openrouter: Unified api and playground for large lan- guage models,

    OpenRouter, “Openrouter: Unified api and playground for large lan- guage models,” https://openrouter.ai, accessed: 2025-05-21

    2025

  14. [22]

    Claude 3 model card october ad- dendum,

    Anthropic, “Claude 3 model card october ad- dendum,” 2024, accessed: 2025-05-23. [Online]. Available: https://assets.anthropic.com/m/1cd9d098ac3e6467/original/ Claude-3-Model-Card-October-Addendum.pdf

    2024

  15. [23]

    Introducing deepseek-v3,

    DeepSeek Inc., “Introducing deepseek-v3,” December 2024, accessed: 2025-05-28. [Online]. Available: https://api-docs.deepseek.com/news/ news1226

    2024

  16. [24]

    Gemini 2.0 flash,

    Google, “Gemini 2.0 flash,” https://openrouter.ai/google/gemini-2. 0-flash-001, February 2025, accessed May 2025

    2025

  17. [25]

    Llama 3.3 70b instruct,

    Meta, “Llama 3.3 70b instruct,” 2024, accessed: 2025-04-01. [Online]. Available: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct

    2024

  18. [26]

    Gpt-4o-mini,

    OpenAI, “Gpt-4o-mini,” https://openrouter.ai/openai/gpt-4o-mini, July 2024, accessed May 2025

    2024

  19. [27]

    Is your ai-generated code really safe? evaluating large language models on secure code generation with codeseceval,

    J. Wang, X. Luo, L. Cao, H. He, H. Huang, J. Xie, A. Jatowt, and Y . Cai, “Is your ai-generated code really safe? evaluating large language models on secure code generation with codeseceval,” 2024. [Online]. Available: https://arxiv.org/abs/2407.02395

    2024 arXiv

  20. [28]

    Keeping an eye on dangerous python modules,

    X. Mertens, “Keeping an eye on dangerous python modules,” https: //isc.sans.edu/diary/27514, 2023, sANS Internet Storm Center

    2023

  21. [29]

    A mathematical theory of communication,

    C. E. Shannon, “A mathematical theory of communication,” Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948

    1948

  22. [30]

    What are bob and alice saying? [mis]communication and intermediation between language and code,

    H. Lammin, “What are bob and alice saying? [mis]communication and intermediation between language and code,” in Language Games , L. Aceti, S. Calvert, and H. Lammin, Eds. Cambridge, MA: LEA / MIT Press, 2021, published online: March 15, 2022. [Online]. Available: https://api.s...

    2021

  23. [31]

    Zittrain, Intellectual Debt: With Great Power Comes Great Ignorance , ser

    J. Zittrain, Intellectual Debt: With Great Power Comes Great Ignorance , ser. Cambridge Law Handbooks. Cambridge University Press, 2022, p. 176–184

    2022

  24. [33]

    The systems engineering approach in times of large language models,

    C. Cabrera, V . Bastidas, J. Schooling, and N. D. Lawrence, “The systems engineering approach in times of large language models,” in Proceedings of the 58th Hawaii International Conference on System Sciences , 2025. [Online]. Available: https://doi.org/10.1145/3643657.3643910 ...

    2025

  25. [2021]

    Available: https://arxiv.org/abs/2009.03300

    [Online]. Available: https://arxiv.org/abs/2009.03300

    2009 arXiv

  26. [2025]

    Available: https://arxiv.org/abs/2504.15903

    [Online]. Available: https://arxiv.org/abs/2504.15903

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.