Pith. sign in

REVIEW 4 major objections 5 minor 51 references

Memorization Diagnostics for Code LLMs Should be Scale-Aware

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The standard tests for code memorization stop working as models grow, and large-model failures under I/O isomorphisms are serialization errors, not lost algorithms.

desk verdict Probe saturation at scale is real and worth publishing, but the I/O isomorphism protocol's boolean-output handling is internally inconsistent and needs fixing before the central numbers are trusted. read the letter →

arxiv 2608.12771 v1 pith:4QUXSHGD submitted 2026-08-13 cs.SE cs.AI

Prateek Kumar Rajput , Abdoul Aziz Bonkoungou , Alberick Euraste Djiré , Xunzhu Tang , Yewei Song , Iyiola Emmanuel Olatunji , El Hacen Diallo , Jacques Klein
show 1 more author
Tegawendé F. Bissyandé
This is my paper · ORCID
classification cs.SEcs.AI
keywords codegenerationLLMmemorizationbenchmarkcontaminationrepresentationalloadI/Oisomorphismmetamorphictestingopcodeanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that standard memorization diagnostics for code LLMs, synonym-fuzzed prompts and likelihood-shift probes such as CoDeC, lose their power as model size increases. On a scale axis of dense models, these probes separate seen from unseen problems at small sizes but fade at frontier scale, even on benchmarks likely present in training data. To interpret what a pass@k drop means, the authors hold the algorithmic task provably fixed with invertible numeric I/O isomorphisms and add only representational load. The paper's central claim is that scaled models keep the algorithmic core under this load and fail mainly at re-encoding outputs, a compliance failure rather than a forgotten algorithm, while smaller models abandon whole solution families. If true, contamination-based memorization findings on large models need re-reading, and future evaluations should separate representational load from recall.

What carries the argument

The load-bearing object is the I/O isomorphism: an invertible affine map $T_\theta(t)=at+b$ with $a\neq 0$, stated in the prompt as an encode/decode contract. Because the transform is bijective, the metamorphic equivalence $f(x)=y \iff T_\theta(f(T_\theta^{-1}(x')))=y'$ guarantees that any drop in accuracy is due to representational handling, not task change. The diagnostic lens is opcode entropy: generated Python is compiled to bytecode, the Shannon entropy of the opcode-frequency distribution $H(\pi)=-\sum_{o}\pi(o)\log_2\pi(o)$ is computed, and per-problem Jensen-Shannon divergence compares pooled Iso distributions against Original. Near-zero JSD is read as same algorithmic family with different serialization; large JSD is read as solution-family collapse. Variants Iso (Enc only) and Iso (Dec only) decompose the load, localizing the dominant cost to decoder-side contract compliance.

What would settle it

Take a transformed problem where a frontier model's Iso generations pass but use different algorithmic families, such as an O($n^{2}$) and an O(n log n) sort on the same decoded input, compile both to bytecode, and compute their pooled JSD: if near-zero JSD coexists with genuinely different asymptotic strategies, the opcode proxy cannot support the compliance-not-competence conclusion.

Watch

Extended reading notes

Core claim

The paper demonstrates probe saturation at scale: synonym fuzzing at 20–40% costs frontier models under eight pass@1 points, and CoDeC's seen-versus-unseen AUC falls from 100% on small checkpoints to 75% at 340B–405B scale. Then, using an affine integer isomorphism $T_\theta(t)=at+b$ appended as an encode/decode contract, correctness is preserved by construction, $f(x)=y \iff T_\theta(f(T_\theta^{-1}(x')))=y'$, so any accuracy difference is representational, not task-level. Under this Iso contract, frontier-model pass@1 drops 14–30 absolute points despite the provably unchanged task, but pooled opcode Jensen-Shannon divergence between Original and Iso stays near zero for scaled models, while small models on hard tasks show large JSD spikes. The authors conclude that scaled models narrow rather than lose their route from specification to code: the same loops and control flow survive and the failure is output-side serialization of the encoded contract, whereas smaller models collapse to different solution families or no valid logic.

Load-bearing premise

The whole argument that scaled models keep their algorithms rests on assuming a program's bytecode opcode histogram faithfully reflects its algorithmic strategy; the paper itself admits two behaviorally different programs can share the same histogram.

Editorial extensions

If this is right

  • Contaminated-benchmark score gaps on large dense models can no longer be read as direct memorization evidence, since the same pass@k drop appears under a provably unseen isomorphic restatement of the task.
  • Evaluations should report solution-space stability, such as opcode JSD, alongside correctness, because a bare pass@k drop conflates a narrowed representational channel with a forgotten algorithm.
  • Decoder-side contract compliance is the binding constraint at scale, so benchmarks that stress output formatting and encoding will discriminate model capability better than lexical prompt fuzzing.
  • The isomorphism protocol provides a reusable oracle: any bijective value transform yields a provable metamorphic relation, so transformed tests need no ground-truth labels beyond the original task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the narrow-channel pattern transfers beyond numeric tasks, then frontier-model failures on format-heavy or API-contract-heavy code are better modeled as interface-serialization defects than as knowledge gaps, which would shift where debugging and prompting effort should go.
  • The probe-saturation result implies that contamination estimates calibrated on small models may overstate the memorization share of frontier benchmark scores; re-running contamination audits per scale tier would reveal where the estimate changes.
  • A testable extension is to construct bijections on non-numeric spaces, such as string reversals with explicit mappings, to see whether the decoder-side bottleneck is a general property of autoregressive generation or specific to arithmetic contracts.
  • Mixture-of-experts architectures, excluded from this study, could split the difference: if routing disperses probability mass differently, decoder-side probes might recover signal at high nominal parameter counts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that standard memorization diagnostics for code LLMs, both encoder-side (synonym fuzzing, dead-code insertion) and decoder-side (CoDeC likelihood scoring), lose discriminative power as dense models become larger. It introduces an I/O-isomorphism protocol that applies invertible value transforms (affine, base-conversion, cubic) to numeric test cases, appends an encode/decode contract to the prompt, and uses the metamorphic relation f(x)=y iff T(f(T^{-1}(x')))=y' to attribute performance drops to "representational load" rather than to loss of the underlying algorithm. Across MBPP, EffiBench, and BigOBench, the paper reports that frontier models lose 14-30 absolute pass@1 points under the isomorphic contract, while opcode-entropy analysis shows near-zero Jensen-Shannon divergence from the original condition, interpreted as preservation of the algorithmic core with failure only in output serialization (the 'narrow channel'). Smaller models instead show large opcode divergence, interpreted as abandonment of solution families. The paper concludes that separate representational load from memorization and reports that the existing probes saturate at scale.

Significance. If the central claims hold, the paper makes a useful methodological contribution: the I/O-isomorphism protocol is a clean metamorphic-testing instrument for numeric coding tasks, and the distinction between competence failure (losing the algorithm) and compliance failure (mis-serializing the interface) is practically important for interpreting pass@k drops. The paper is also valuable empirically for showing that lexical and likelihood-based probes that work on small models may not transfer to frontier dense models, and it ships code and data. The mathematical identity underlying the isomorphism is sound for numeric I/O. However, the two most load-bearing empirical inferences - the size of the Iso drop and the interpretation of near-zero opcode JSD as preserved algorithmic strategy - rest on methodological choices that the paper itself partially acknowledges as limitations. The boolean-output inconsistency in the running example and the proxy nature of opcode entropy require attention before the compliance-versus-competence conclusion can be accepted.

major comments (4)
  1. [§5.4 and Figure 2] The metamorphic guarantee is stated for integer-valued I/O via y' = T(y), but the running example in Figure 2 uses boolean outputs: the Iso test cases list raw True/False, not T(True) or T(False), and the passing solution returns raw booleans despite the prompt's instruction to 're-encode every output'. The failing solution attempts to encode booleans as integer expressions, which evaluates back to True in Python and therefore fails on both original and Iso cases. This means the Iso condition is ill-posed for boolean-output tasks: the model receives contradictory instructions (encode every output, but test cases show unencoded booleans). The paper never states how boolean- or string-valued outputs in MBPP, EffiBench, or BigOBench were mapped through T or how the oracle graded them, and Section 8.3 restricts the metamorphic guarantee to numeric I/O. If a substantial fraction of benchmark tasks have non-integer outputs, the magnitude of the Iso drops in Table 3, the encoder/decoder decomposition in RQ3a, and the compliance-failure interpretation in RQ3b are all contaminated by this protocol gap. The authors should either restrict all Iso analyses to tasks with integer outputs or define an explicit boolean/string encoding and enforce it consistently in the oracle.
  2. [§5.5 and RQ3b] The central conclusion that scaled models 'attempt the same algorithmic core' under Iso rests on near-zero Jensen-Shannon divergence of pooled opcode-frequency histograms. The paper's own Section 9 acknowledges that two behaviorally different programs can share an opcode histogram and that JSD ignores opcode ordering, so the metric is not a fingerprint of algorithmic strategy. The calibration check with Syn-20/Syn-40 shows only that near-zero JSD can accompany preserved logic; it does not establish that near-zero JSD under Iso implies preserved logic, because an Iso-induced change of solution strategy could in principle leave the opcode histogram unchanged. Since RQ3b uses this inference to separate compliance failure from competence failure, the claim needs additional support, for example semantic equivalence checking of generated programs on randomized inputs, control-flow graph comparison, or execution-based verification of the algorithmic core on a sample of tasks.
  3. [§6.2, Table 4] The CoDeC scale trend is based on dataset-level AUC computed from only two seen and two unseen datasets per model. With four datasets, AUC can take only coarse values (multiples of 25 percentage points), and a single mislabeled dataset changes the reported AUC by 25 points. The paper itself refers to Llama-3.1-70B's AUC of 25% as 'likely an anomaly'. This makes the RQ2 conclusion ('discriminative power appears to weaken with scale') fragile; at minimum the authors should provide per-dataset score distributions, bootstrap confidence intervals, or additional seen/unseen datasets to support the monotonic trend.
  4. [§5.2 and Table 3] The paper reports pass@1 but generates n=5 completions per (task, condition) at temperature T=0.0, without stating how pass@1 is derived. It is unclear whether the reported value is the mean per-sample accuracy (c/n averaged over tasks), the unbiased pass@k estimator with k=1, or best-of-n accuracy (pass@5 mislabeled as pass@1). Since all quantitative claims in Section 6 use this metric, the derivation must be stated explicitly; the difference matters, especially under near-deterministic decoding where the five samples may be near-identical.
minor comments (5)
  1. [Abstract and §6.1] The abstract and introduction list dead-code insertion as one of the encoder-side probes studied, but Section 6.1 reports only synonym fuzzing; dead code appears only in the prompt-length ablation in Section 7.2. Please clarify which results correspond to dead-code insertion as a memorization probe.
  2. [Table 3] The table caption states that '±' denotes the half-width of a 95% bootstrap CI, but only the Original column reports this interval; the other condition columns show no uncertainty. Please either report intervals for all conditions or explain why they are omitted.
  3. [Figure 2] The failing solution in Figure 2 returns expressions such as '(0*2)-94 == -94', which evaluate to the boolean True in Python; the caption's phrase 'all four returns collapse to the same value' is therefore correct but potentially confusing to readers who see the model as attempting to comply with the encoding contract. Re-state the failure mode in terms of the model's inability to return an encoded integer rather than a boolean expression.
  4. [§6.1] For very small models such as StarCoder2-15B and CodeLlama-13B, baseline accuracy is close to zero on BigOBench and EffiBench, so relative percentage changes are difficult to interpret; the paper acknowledges this but it would help to read the absolute drops alongside the relative ones in the discussion.
  5. [§5.5] The definition of the pooled opcode distribution is not fully formal: please clarify how the five per-generation frequency vectors are pooled (sum of counts followed by renormalization) and whether the same pooling is applied for the Original and condition distributions before computing JSD.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical measurements with disclosed proxy caveats.

full rationale

The paper's derivation chain is self-contained with respect to its central claims. The I/O isomorphism protocol rests on the bijective affine transform and the biconditional f(x)=y ⇔ Tθ(f(Tθ^{-1}(x')))=y' (Section 5.4), a mathematical identity rather than a fitted relation; the observed pass@1 drops are empirical measurements against MBPP, EffiBench, and BigOBench. The synonym-fuzzing baseline follows Djiré et al. [10], a prior paper sharing authors, but the present contribution is the new observation that this baseline saturates at scale; that observation is measured, not inherited from [10], so the self-citation is not load-bearing. The CoDeC reproduction is based on the external method of Zawalski et al. [13]. The opcode-JSD analysis is the only step where a conclusion ('same algorithmic core') is inferred from a proxy, but Section 5.5 defines the proxy explicitly and Section 9 discloses its limits (two behaviorally different programs can share an opcode histogram; JSD ignores ordering), which makes this a construct-validity caveat rather than a circular definition. No parameter is fitted to the target conclusion, and no 'prediction' is equivalent to its input by construction. The boolean-output inconsistency in Fig. 2 and the numeric-only scope admitted in Section 8.3 are potential correctness/validity threats, but they do not reduce the derivation to its own inputs. Overall circularity score: 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 2 invented entities

The central claim relies on the opcode proxy, the dense-model scale axis, contamination designations by date, and the assumption that random transforms are unseen. None of these are circular: they are stated assumptions with acknowledged limits. The only hand-chosen protocol constants are the transform families and synonym rates, which are ablated or held fixed.

free parameters (1)
  • Affine isomorphism constants a,b = a=2, b=-94 in the running example; random per instance in general
    Protocol parameters chosen to define the representational load, not fitted to data. The ablation over base-conversion and cubic families shows the observed drops are not specific to the affine family.
assumptions (4)
  • domain assumption Opcode-frequency entropy is a valid proxy for algorithmic strategy.
    RQ3b's conclusion that scaled models preserve algorithms rests on near-zero JSD between opcode distributions. The paper acknowledges in Section 9 that behaviorally different programs can share histograms and that JSD is invariant to opcode ordering.
  • domain assumption Dense parameter count is a valid scale axis for the study.
    The paper deliberately excludes MoE models because effective capacity is ambiguous (Section 5.1). The findings therefore hold only for dense architectures, an explicitly scoped assumption.
  • domain assumption Publication dates of benchmarks relative to model release indicate contamination status.
    MBPP is labeled potentially contaminated because it predates every model; EffiBench and BigOBench are labeled unseen because they postdate most cutoffs. For closed models, training cutoffs are not disclosed, so contamination designations are probable rather than certain, as the paper admits in Section 5.3 and Section 9.
  • domain assumption Randomly parameterized I/O transforms are unseen in training.
    The claim that the model cannot memorize its way through the isomorphic layer assumes the specific a,b parameters were not encountered during training. This is plausible but not verifiable for closed models.
invented entities (2)
  • representational load independent evidence
    purpose: Names the interface burden imposed by unfamiliar formulations of a fixed task, treated as distinct from memorization.
    Operationalized through I/O isomorphisms: pass@1 drops under Iso are attributed to it, and opcode JSD provides a measurable handle separating it from algorithmic loss.
  • narrow channel independent evidence
    purpose: Describes the decoder-side bottleneck where scaled models preserve the algorithm but mis-serialize the output contract.
    Supported by Figure 2's running example and the Iso(Enc-only) versus Iso(Dec-only) decomposition in RQ3a, though it remains a metaphor rather than a formal construct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Memorization Diagnostics for Code LLMs Should be Scale-Aware." pith.science (2026). https://pith.science/paper/4QUXSHGD

@misc{pith2026260812771,
  author       = {Pith},
  title        = {Pith review of: Memorization Diagnostics for Code LLMs Should be Scale-Aware},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4QUXSHGD}},
  note         = {Machine review of arXiv:2608.12771}
}
read the original abstract

The extent to which large language models for code rely on memorization over genuine understanding remains highly debated. While current literature frequently reports widespread memorization, evaluating the underlying probing techniques across dense architectures reveals a severe breakdown in their utility at scale. Traditional encoder-style probes using perturbations such as synonym fuzzing or dead-code insertion struggle to expose memorization in scaled models, even on known-contaminated benchmarks, and decoder-style probes that rely on log probabilities show similar performance degradation. The specific mode of failure for these probes, particularly why such techniques disrupt smaller models but fail to impact larger ones, motivates us to untangle representation load from memorization rather than treating them as a single phenomenon. By applying invertible mathematical transforms to numeric problems, we isolate these two factors and reveal that scaled encoders successfully absorb substantial representation load while still converging on the correct family of solutions. In practical software engineering, this ability to adapt to varying surface forms is what truly matters for usability and generalizability in LLM and agentic applications. Whether a specific solution was seen during training becomes a much less pressing question because although memorization inflates scores on contaminated benchmarks, factoring out representation load makes it debatable how much we should truly care if a functional answer was originally memorized. Future evaluations must therefore be built around separating these phenomena rather than relying on methodologies that quietly entangle them.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 24 canonical work pages

  1. [1]

    arXiv preprint arXiv:2107.03374 (2021)

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  2. [2]

    arXiv preprint arXiv:2108.07732 (2021) 26

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.: Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021) 26

    arXiv 2021

  3. [3]

    Jimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K.: Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023)

    arXiv 2023

  4. [4]

    Zhang, L., He, S., Zhang, C., Kang, Y., Li, B., Xie, C., Wang, J., Wang, M., Huang, Y., Fu, S., et al.: Swe-bench goes live! arXiv preprint arXiv:2505.23419 (2025)

    arXiv 2025

  5. [5]

    arXiv preprint arXiv:2506.12286 (2025)

    Liang, S., Garg, S., Moghaddam, R.Z.: The swe-bench illusion: When state-of- the-art llms remember instead of reason. arXiv preprint arXiv:2506.12286 (2025)

    arXiv 2025

  6. [6]

    naturalizing

    Chakraborty, S., Ahmed, T., Ding, Y., Devanbu, P.T., Ray, B.: Natgen: generative pre-training by “naturalizing” source code. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 18–30 (2022)

    work page 2022

  7. [7]

    In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P.,et al.: Recode: Robustness evaluation of code genera- tion models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13818–13843 (2023)

    work page 2023

  8. [8]

    In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp

    Mastropaolo, A., Pascarella, L., Guglielmi, E., Ciniselli, M., Scalabrino, S., Oliveto, R., Bavota, G.: On the robustness of code generation techniques: An empirical study on github copilot. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 2149–2160 (2023). IEEE

    work page 2023

Show all 51 references
  1. [9]

    ACM Transactions on Software Engineering and Methodology (2024)

    Chen, J., Zhenhao, L., Xing, H., Xin, X.: Nlperturbator: Studying the robust- ness of code llms to natural language variations. ACM Transactions on Software Engineering and Methodology (2024)

    2024

  2. [10]

    arXiv preprint arXiv:2505.03019 (2025)

    Djir´ e, A.E., Kabor´ e, A.K., Barr, E.T., Klein, J., Bissyand´ e, T.F.: Memorization or interpolation? detecting llm memorization through input perturbation analysis. arXiv preprint arXiv:2505.03019 (2025)

    2025 arXiv

  3. [11]

    arXiv preprint arXiv:2604.13997 (2026)

    Euraste, D.A., Kader, K.A., Samhi, J., Barr, E.T., Klein, J., Bissyand´ e, T.F.: Learned or memorized? quantifying memorization advantage in code llms. arXiv preprint arXiv:2604.13997 (2026)

    2026 arXiv

  4. [12]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Riddell, M., Ni, A., Cohan, A.: Quantifying contamination in evaluating code generation capabilities of language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14116–14137 (2024)

    2024

  5. [13]

    arXiv preprint arXiv:2510.27055 (2025) 27

    Zawalski, M., Boubdir, M., Ba lazy, K., Nushi, B., Ribalta, P.: Detecting data contamination in llms via in-context learning. arXiv preprint arXiv:2510.27055 (2025) 27

    2025 arXiv

  6. [14]

    In: The Twelfth International Conference on Learning Representations (2023)

    Oren, Y., Meister, N., Chatterji, N.S., Ladhak, F., Hashimoto, T.: Proving test set contamination in black-box language models. In: The Twelfth International Conference on Learning Representations (2023)

    2023

  7. [15]

    Transactions of the Association for Computational Linguistics13, 809–830 (2025)

    Golchin, S., Surdeanu, M.: Data contamination quiz: A tool to detect and esti- mate contamination in large language models. Transactions of the Association for Computational Linguistics13, 809–830 (2025)

    2025

  8. [16]

    In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pp

    Nie, Y., Wang, C., Wang, K., Xu, G., Xu, G., Wang, H.: Decoding secret mem- orization in code llms through token-level characterization. In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pp. 2880–2892 (2025). IEEE

    2025

  9. [17]

    arXiv preprint arXiv:2101.00027 (2020)

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al.: The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020)

    2020 arXiv

  10. [18]

    Journal of machine learning research21(140), 1–67 (2020)

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research21(140), 1–67 (2020)

    2020

  11. [19]

    Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology (1998)

    Chen, T.Y., Cheung, S.C., Yiu, S.M.: Metamorphic testing: A new approach for generating next test cases. Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology (1998)

    1998

  12. [20]

    IEEE Transactions on Software Engineering42(9), 805–824 (2016)

    Segura, S., Fraser, G., Sanchez, A.B., Ruiz-Cort´ es, A.: A survey on metamorphic testing. IEEE Transactions on Software Engineering42(9), 805–824 (2016)

    2016

  13. [21]

    In: 30th USENIX Security Symposium (USENIX Security 21), pp

    Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U.,et al.: Extracting training data from large language models. In: 30th USENIX Security Symposium (USENIX Security 21), pp. 2633–2650 (2021)

    2021

  14. [22]

    In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., Carlini, N.: Deduplicating training data makes language models better. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424–8445 (2022)

    2022

  15. [23]

    In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, pp

    Coignion, T., Quinton, C., Rouvoy, R.: A performance study of llm-generated code on leetcode. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, pp. 79–89 (2024)

    2024

  16. [24]

    In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp

    Matton, A., Sherborne, T., Aumiller, D., Tommasone, E., Alizadeh, M., He, J., Ma, R., Voisin, M., Gilsenan-McMahon, E., Gall´ e, M.: On leakage of code gen- eration evaluation datasets. In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 13215–13223 ...

    2024

  17. [25]

    In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp

    Sainz, O., Campos, J., Garc´ ıa-Ferrero, I., Etxaniz, J., Lacalle, O.L., Agirre, E.: Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10776–10787 (2023)

    2023

  18. [26]

    arXiv preprint arXiv:2310.16789 (2023)

    Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., Zettle- moyer, L.: Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789 (2023)

    2023 arXiv

  19. [27]

    In: Findings of the Association for Computational Linguistics: ACL 2023, pp

    Mattern, J., Mireshghallah, F., Jin, Z., Sch¨ olkopf, B., Sachan, M., Berg- Kirkpatrick, T.: Membership inference attacks against language models via neighbourhood comparison. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 11330–11343 (2023)

    2023

  20. [28]

    Advances in Neural Information Processing Systems36, 39321–39362 (2023)

    Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tram` er, F., Carlini, N.: Counter- factual memorization in neural language models. Advances in Neural Information Processing Systems36, 39321–39362 (2023)

    2023

  21. [29]

    arXiv preprint arXiv:2308.03296 (2023)

    Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al.: Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296 (2023)

    2023 arXiv

  22. [30]

    Advances in Neural Information Processing Systems36, 28072–28090 (2023)

    Biderman, S., Prashanth, U., Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., Raff, E.: Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems36, 28072–28090 (2023)

    2023

  23. [31]

    arXiv preprint arXiv:2201.02177 (2022)

    Power, A., Burda, Y., Edwards, H., Babuschkin, I., Misra, V.: Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 (2022)

    2022 arXiv

  24. [32]

    arXiv preprint arXiv:2301.05217 (2023)

    Nanda, N., Chan, L., Lieberum, T., Smith, J., Steinhardt, J.: Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217 (2023)

    2023 arXiv

  25. [33]

    Advances in Neural Information Processing Systems35, 34651–34663 (2022)

    Liu, Z., Kitouni, O., Nolte, N.S., Michaud, E., Tegmark, M., Williams, M.: Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems35, 34651–34663 (2022)

    2022

  26. [34]

    Advances in Neural Information Processing Systems35, 38274–38290 (2022)

    Tirumala, K., Markosyan, A., Zettlemoyer, L., Aghajanyan, A.: Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems35, 38274–38290 (2022)

    2022

  27. [35]

    arXiv preprint arXiv:2403.07974 (2024)

    Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., Stoica, I.: Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 (2024)

    2024 arXiv

  28. [36]

    Advances in neural information processing systems36, 21558–21572 (2023)

    Liu, J., Xia, C.S., Wang, Y., Zhang, L.: Is your code generated by chatgpt 29 really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems36, 21558–21572 (2023)

    2023

  29. [37]

    arXiv preprint arXiv:2403.19114 (2024)

    Xia, C.S., Deng, Y., Zhang, L.: Top leaderboard ranking= top coding profi- ciency, always? evoeval: Evolving coding benchmarks via llm. arXiv preprint arXiv:2403.19114 (2024)

    2024 arXiv

  30. [38]

    arXiv preprint arXiv:2001.08361 (2020)

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  31. [39]

    arXiv preprint arXiv:2206.07682 (2022)

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.: Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022)

    2022 arXiv

  32. [40]

    Advances in neural information processing systems33, 1877–1901 (2020)

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A.,et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020)

    2020

  33. [41]

    Advances in Neural Information Processing Systems37, 11506–11544 (2024)

    Huang, D., Qing, Y., Shang, W., Cui, H., Zhang, J.M.: Effibench: Benchmarking the efficiency of automatically generated code. Advances in Neural Information Processing Systems37, 11506–11544 (2024)

    2024

  34. [42]

    Chambon, P., Roziere, B., Sagot, B., Synnaeve, G.: Bigo (bench)–can llms generate code with controlled time and space complexity? arXiv preprint arXiv:2503.15242 (2025)

    2025 arXiv

  35. [43]

    In: First Conference on Language Modeling (2024)

    Rein, D., Hou, B.L., Stickland, A.C., Petty, J., Pang, R.Y., Dirani, J., Michael, J., Bowman, S.R.: Gpqa: A graduate-level google-proof q&a benchmark. In: First Conference on Language Modeling (2024)

    2024

  36. [44]

    In: International Conference on Machine Learning, pp

    Biderman, S., Schoelkopf, H., Anthony, Q.G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M.A., Purohit, S., Prashanth, U.S., Raff, E.,et al.: Pythia: A suite for analyzing large language models across training and scaling. In: International Conference on Machine Learning, pp...

    2023

  37. [45]

    Advances in neural information processing systems30(2017)

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems30(2017)

    2017

  38. [46]

    OpenAI blog1(8), 9 (2019)

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.,et al.: Language models are unsupervised multitask learners. OpenAI blog1(8), 9 (2019)

    2019

  39. [47]

    Advances in neural information processing systems36, 70293– 70332 (2023)

    Dziri, N., Lu, X., Sclar, M., Li, X.L., Jiang, L., Lin, B.Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R.,et al.: Faith and fate: Limits of transformers on 30 compositionality. Advances in neural information processing systems36, 70293– 70332 (2023)

    2023

  40. [48]

    Ethayarajh, K.: How contextual are contextualized word representations? com- paring the geometry of bert, elmo, and gpt-2 embeddings. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natura...

    2019

  41. [49]

    arXiv preprint arXiv:2412.04604 (2024)

    Chollet, F., Knoop, M., Kamradt, G., Landers, B.: Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604 (2024)

    2024 arXiv

  42. [50]

    arXiv preprint arXiv:2505.11831 (2025)

    Chollet, F., Knoop, M., Kamradt, G., Landers, B., Pinkard, H.: Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831 (2025)

    2025 arXiv

  43. [51]

    arXiv preprint arXiv:1911.01547 (2019) 31

    Chollet, F.: On the measure of intelligence. arXiv preprint arXiv:1911.01547 (2019) 31

    2019 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.