{"id":"73eeb08f-9de7-435d-b34b-12458b1826d5","arxiv_id":"2608.12771","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Encoder-side synonym fuzzing and decoder-side log-likelihood probes lose discriminative power on large dense code LLMs, while reversible I/O transforms show scaled models preserve algorithmic structure and fail mainly on output serialization.","lead":"This paper tests whether common memorization probes for code-generating AI models still work as models get bigger, and finds they lose their signal at scale. It introduces a new test that changes the problem's wording using reversible math transforms, showing large models keep the algorithm and only fail to format the answer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The I/O-isomorphism oracle is internally inconsistent for boolean-output tasks: §5.4's metamorphic guarantee requires y′=T(y), but Fig. 2 grades raw True/False while instructing 're-encode every output', so reported Iso drops may be spec artifacts.","rationale":"The reader's weakest assumption—opcode-frequency entropy as a proxy for algorithmic strategy—is valid and explicitly acknowledged in Section 9, and it does threaten the 'narrow channel' interpretation in RQ3b. However, the more load-bearing problem is that the measurement itself may be corrupted for a large class of tasks: the I/O-isomorphism oracle is internally inconsistent for boolean outputs, as shown by the paper's own Figure 2. Section 5.4's biconditional is the mathematical basis of the paper's central method; it requires the output space to be closed under Tθ. For boolean outputs, Tθ is undefined unless booleans are mapped to integers, in which case the oracle must compare against the encoded integer. The published example instead compares against raw booleans while instructing the model to encode its outputs, creating an impossible specification. A model that complies fails; a model that ignores the instruction passes. This is an unacknowledged internal inconsistency, not merely a proxy-validity issue, and it directly affects the reported pass@1 drops, the decoder-side bottleneck claim (RQ3a), and the compliance-versus-competence reading (RQ3b). The paper also reports Iso results on EffiBench and BigOBench without clarifying the output types or explaining how an integer bijection applies to non-numeric outputs, while Section 8.3 concedes the guarantee holds only for numeric I/O. The proposed concrete test—partitioning tasks by output type and recomputing the drops on integer-output tasks only—would settle whether the central quantitative claims survive. This is why the verdict remains CONDITIONAL rather than ACCEPT, and why the reader's already-conditional verdict is reinforced rather than overturned.","tokens_in":22096,"tokens_out":13373,"duration_ms":139793,"concrete_test":"Download the released Zenodo dataset and partition every task by output datatype. For each boolean-output task, verify whether the Iso oracle's expected value equals T(0/1) (e.g., −92/−94) or the raw boolean; then recompute Table 3's Iso, Iso(Enc-only), and Iso(Dec-only) drops using only tasks whose outputs are integers mapped through Tθ. If the drops shrink below the claimed 14–30 points or shift the decoder-side bottleneck, the central RQ3 claims must be revised. If the drops persist on integer-output tasks, the paper's main empirical pattern survives this particular objection.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.4 defines the protocol via integer transforms Tθ and the biconditional f(x)=y ⇔ Tθ(f(Tθ^{-1}(x′)))=y′. This equivalence is sound only when y is an integer (or is explicitly mapped to one). The paper's own running example (Fig. 2) violates this: the Iso test case is is_woodall(672)==True, not ==T(True) (which would be −92), yet the contract tells the model to 're-encode every output' and illustrates boolean encoding ('if the answer is 0, output −94'). A model that follows that instruction returns an encoded integer expression; the oracle demands a raw boolean. The passing solution in Fig. 2 only passes by ignoring the output-encoding instruction. The paper never states how boolean- or string-valued outputs in MBPP, EffiBench, or BigOBench were handled, and Section 8.3 restricts the metamorphic guarantee to numeric I/O. If a large fraction of tasks have non-integer outputs, the magnitude of the Iso drop in Table 3, the encoder/decoder decomposition (RQ3a), and the compliance-failure interpretation (RQ3b) are contaminated by an ill-posed prompt rather than a genuine representational load.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that standard memorization diagnostics for code LLMs, both encoder-side (synonym fuzzing, dead-code insertion) and decoder-side (CoDeC likelihood scoring), lose discriminative power as dense models become larger. It introduces an I/O-isomorphism protocol that applies invertible value transforms (affine, base-conversion, cubic) to numeric test cases, appends an encode/decode contract to the prompt, and uses the metamorphic relation f(x)=y iff T(f(T^{-1}(x')))=y' to attribute performance drops to \"representational load\" rather than to loss of the underlying algorithm. Across MBPP, EffiBench, and BigOBench, the paper reports that frontier models lose 14-30 absolute pass@1 points under the isomorphic contract, while opcode-entropy analysis shows near-zero Jensen-Shannon divergence from the original condition, interpreted as preservation of the algorithmic core with failure only in output serialization (the 'narrow channel'). Smaller models instead show large opcode divergence, interpreted as abandonment of solution families. The paper concludes that separate representational load from memorization and reports that the existing probes saturate at scale.","tokens_in":22319,"tokens_out":6040,"duration_ms":63624,"significance":"If the central claims hold, the paper makes a useful methodological contribution: the I/O-isomorphism protocol is a clean metamorphic-testing instrument for numeric coding tasks, and the distinction between competence failure (losing the algorithm) and compliance failure (mis-serializing the interface) is practically important for interpreting pass@k drops. The paper is also valuable empirically for showing that lexical and likelihood-based probes that work on small models may not transfer to frontier dense models, and it ships code and data. The mathematical identity underlying the isomorphism is sound for numeric I/O. However, the two most load-bearing empirical inferences - the size of the Iso drop and the interpretation of near-zero opcode JSD as preserved algorithmic strategy - rest on methodological choices that the paper itself partially acknowledges as limitations. The boolean-output inconsistency in the running example and the proxy nature of opcode entropy require attention before the compliance-versus-competence conclusion can be accepted.","major_comments":[{"comment":"The metamorphic guarantee is stated for integer-valued I/O via y' = T(y), but the running example in Figure 2 uses boolean outputs: the Iso test cases list raw True/False, not T(True) or T(False), and the passing solution returns raw booleans despite the prompt's instruction to 're-encode every output'. The failing solution attempts to encode booleans as integer expressions, which evaluates back to True in Python and therefore fails on both original and Iso cases. This means the Iso condition is ill-posed for boolean-output tasks: the model receives contradictory instructions (encode every output, but test cases show unencoded booleans). The paper never states how boolean- or string-valued outputs in MBPP, EffiBench, or BigOBench were mapped through T or how the oracle graded them, and Section 8.3 restricts the metamorphic guarantee to numeric I/O. If a substantial fraction of benchmark tasks have non-integer outputs, the magnitude of the Iso drops in Table 3, the encoder/decoder decomposition in RQ3a, and the compliance-failure interpretation in RQ3b are all contaminated by this protocol gap. The authors should either restrict all Iso analyses to tasks with integer outputs or define an explicit boolean/string encoding and enforce it consistently in the oracle.","section":"§5.4 and Figure 2"},{"comment":"The central conclusion that scaled models 'attempt the same algorithmic core' under Iso rests on near-zero Jensen-Shannon divergence of pooled opcode-frequency histograms. The paper's own Section 9 acknowledges that two behaviorally different programs can share an opcode histogram and that JSD ignores opcode ordering, so the metric is not a fingerprint of algorithmic strategy. The calibration check with Syn-20/Syn-40 shows only that near-zero JSD can accompany preserved logic; it does not establish that near-zero JSD under Iso implies preserved logic, because an Iso-induced change of solution strategy could in principle leave the opcode histogram unchanged. Since RQ3b uses this inference to separate compliance failure from competence failure, the claim needs additional support, for example semantic equivalence checking of generated programs on randomized inputs, control-flow graph comparison, or execution-based verification of the algorithmic core on a sample of tasks.","section":"§5.5 and RQ3b"},{"comment":"The CoDeC scale trend is based on dataset-level AUC computed from only two seen and two unseen datasets per model. With four datasets, AUC can take only coarse values (multiples of 25 percentage points), and a single mislabeled dataset changes the reported AUC by 25 points. The paper itself refers to Llama-3.1-70B's AUC of 25% as 'likely an anomaly'. This makes the RQ2 conclusion ('discriminative power appears to weaken with scale') fragile; at minimum the authors should provide per-dataset score distributions, bootstrap confidence intervals, or additional seen/unseen datasets to support the monotonic trend.","section":"§6.2, Table 4"},{"comment":"The paper reports pass@1 but generates n=5 completions per (task, condition) at temperature T=0.0, without stating how pass@1 is derived. It is unclear whether the reported value is the mean per-sample accuracy (c/n averaged over tasks), the unbiased pass@k estimator with k=1, or best-of-n accuracy (pass@5 mislabeled as pass@1). Since all quantitative claims in Section 6 use this metric, the derivation must be stated explicitly; the difference matters, especially under near-deterministic decoding where the five samples may be near-identical.","section":"§5.2 and Table 3"}],"minor_comments":[{"comment":"The abstract and introduction list dead-code insertion as one of the encoder-side probes studied, but Section 6.1 reports only synonym fuzzing; dead code appears only in the prompt-length ablation in Section 7.2. Please clarify which results correspond to dead-code insertion as a memorization probe.","section":"Abstract and §6.1"},{"comment":"The table caption states that '±' denotes the half-width of a 95% bootstrap CI, but only the Original column reports this interval; the other condition columns show no uncertainty. Please either report intervals for all conditions or explain why they are omitted.","section":"Table 3"},{"comment":"The failing solution in Figure 2 returns expressions such as '(0*2)-94 == -94', which evaluate to the boolean True in Python; the caption's phrase 'all four returns collapse to the same value' is therefore correct but potentially confusing to readers who see the model as attempting to comply with the encoding contract. Re-state the failure mode in terms of the model's inability to return an encoded integer rather than a boolean expression.","section":"Figure 2"},{"comment":"For very small models such as StarCoder2-15B and CodeLlama-13B, baseline accuracy is close to zero on BigOBench and EffiBench, so relative percentage changes are difficult to interpret; the paper acknowledges this but it would help to read the absolute drops alongside the relative ones in the discussion.","section":"§6.1"},{"comment":"The definition of the pooled opcode distribution is not fully formal: please clarify how the five per-generation frequency vectors are pooled (sum of counts followed by renormalization) and whether the same pooling is applied for the Original and condition distributions before computing JSD.","section":"§5.5"}],"recommendation":"major_revision","confidential_remarks":"The boolean-output inconsistency in Section 5.4/Figure 2 is the most serious issue: it affects the validity of the Iso protocol on a potentially large fraction of benchmark tasks. If the authors re-run the protocol restricted to integer-output tasks (or properly encode boolean outputs), the central numeric claim may survive, but the current manuscript does not establish it. The opcode-JSD inference is also weaker than the text suggests; the paper's own Section 9 admits the relevant limitation. I recommend major revision rather than rejection because the core idea is sound and the fixes are within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know about this paper for two reasons. First, it asks a timely empirical question: do the probes we use to detect memorization in code LLMs still work as models scale? The answer, for synonym fuzzing and CoDeC, is no, and that saturation result is convincing and worth taking seriously. Second, the paper's proposed replacement, an I/O isomorphism protocol with opcode analysis, is clever but has a load-bearing bug in how it handles non-integer outputs.\n\nThe new things here are real. The demonstration that encoder-side synonym fuzzing and decoder-side CoDeC lose discriminative signal at scale, with AUC falling from 100% to 75% and synonym drops under 5 points at the frontier versus 20+ at small scale, is a useful counterweight to the memorization-heavy narrative. The I/O isomorphism idea, applying bijective transforms to preserve the algorithm while changing the interface, is a genuinely new evaluation axis. The opcode divergence analysis, showing that scaled models retain the same solution families under load while smaller models drop them, is a nice attempt to separate compliance from competence. The paper is clearly written, reports code/data, and includes honest threats-to-validity and ablations (dead-code length control, alternative transforms).\n\nThe soft spots are not minor. The main one: Section 5.4's metamorphic guarantee is stated only for numeric outputs, yet the paper applies it to benchmarks with boolean outputs and grades raw True/False while instructing the model to re-encode every output. The running example in Figure 2 is internally inconsistent: the contract says to output encoded integers, the test cases show raw booleans, and the passing solution only passes by ignoring the output-encoding instruction. If many tasks have boolean or string outputs, the reported Iso drops, the encoder/decoder decomposition (RQ3a), and the compliance-failure interpretation (RQ3b) are all contaminated by an ill-posed prompt rather than a clean representational load. The authors never state how they handled boolean outputs. This is fixable by either restricting the study to integer-output tasks or explicitly mapping booleans through T and grading the encoded output, but until then the magnitudes in Table 3 are suspect.\n\nThere are also smaller gaps: pass@1 is reported while five completions are generated, with no statement of how it is derived; the CoDeC AUC rests on only two seen and two unseen datasets with no error bars; and the opcode-entropy proxy is a histogram-level measure, as the authors themselves note in Section 9.\n\nNet: the qualitative point about probe saturation is likely right, and the paper deserves a serious referee. I would not reject it, but the boolean handling must be fixed or scoped away before the numeric claims can be relied upon. Take it as a conditional accept with major revision.","headline":"Probe saturation at scale is real and worth publishing, but the I/O isomorphism protocol's boolean-output handling is internally inconsistent and needs fixing before the central numbers are trusted.","tokens_in":22933,"tokens_out":4099,"would_cite":false,"duration_ms":40972,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The standard tests for code memorization stop working as models grow, and large-model failures under I/O isomorphisms are serialization errors, not lost algorithms.","keywords":["code generation","LLM memorization","benchmark contamination","representational load","I/O isomorphism","metamorphic testing","opcode analysis"],"falsifier":"Take a transformed problem where a frontier model's Iso generations pass but use different algorithmic families, such as an O($n^{2}$) and an O(n log n) sort on the same decoded input, compile both to bytecode, and compute their pooled JSD: if near-zero JSD coexists with genuinely different asymptotic strategies, the opcode proxy cannot support the compliance-not-competence conclusion.","tokens_in":21885,"feed_emoji":"🧠","tokens_out":6611,"duration_ms":64517,"temperature":0.7,"pith_summary":"This paper argues that standard memorization diagnostics for code LLMs, synonym-fuzzed prompts and likelihood-shift probes such as CoDeC, lose their power as model size increases. On a scale axis of dense models, these probes separate seen from unseen problems at small sizes but fade at frontier scale, even on benchmarks likely present in training data. To interpret what a pass@k drop means, the authors hold the algorithmic task provably fixed with invertible numeric I/O isomorphisms and add only representational load. The paper's central claim is that scaled models keep the algorithmic core under this load and fail mainly at re-encoding outputs, a compliance failure rather than a forgotten algorithm, while smaller models abandon whole solution families. If true, contamination-based memorization findings on large models need re-reading, and future evaluations should separate representational load from recall.","feed_headline":"Probes for code memorization stop working on large models","feed_subtitle":"Synonym and likelihood tests saturate at scale; I/O transforms show failures are compliance, not lost algorithms.","key_machinery":"The load-bearing object is the I/O isomorphism: an invertible affine map $T_\\theta(t)=at+b$ with $a\\neq 0$, stated in the prompt as an encode/decode contract. Because the transform is bijective, the metamorphic equivalence $f(x)=y \\iff T_\\theta(f(T_\\theta^{-1}(x')))=y'$ guarantees that any drop in accuracy is due to representational handling, not task change. The diagnostic lens is opcode entropy: generated Python is compiled to bytecode, the Shannon entropy of the opcode-frequency distribution $H(\\pi)=-\\sum_{o}\\pi(o)\\log_2\\pi(o)$ is computed, and per-problem Jensen-Shannon divergence compares pooled Iso distributions against Original. Near-zero JSD is read as same algorithmic family with different serialization; large JSD is read as solution-family collapse. Variants Iso (Enc only) and Iso (Dec only) decompose the load, localizing the dominant cost to decoder-side contract compliance.","core_discovery":"The paper demonstrates probe saturation at scale: synonym fuzzing at 20–40% costs frontier models under eight pass@1 points, and CoDeC's seen-versus-unseen AUC falls from 100% on small checkpoints to 75% at 340B–405B scale. Then, using an affine integer isomorphism $T_\\theta(t)=at+b$ appended as an encode/decode contract, correctness is preserved by construction, $f(x)=y \\iff T_\\theta(f(T_\\theta^{-1}(x')))=y'$, so any accuracy difference is representational, not task-level. Under this Iso contract, frontier-model pass@1 drops 14–30 absolute points despite the provably unchanged task, but pooled opcode Jensen-Shannon divergence between Original and Iso stays near zero for scaled models, while small models on hard tasks show large JSD spikes. The authors conclude that scaled models narrow rather than lose their route from specification to code: the same loops and control flow survive and the failure is output-side serialization of the encoded contract, whereas smaller models collapse to different solution families or no valid logic.","pith_inferences":["If the narrow-channel pattern transfers beyond numeric tasks, then frontier-model failures on format-heavy or API-contract-heavy code are better modeled as interface-serialization defects than as knowledge gaps, which would shift where debugging and prompting effort should go.","The probe-saturation result implies that contamination estimates calibrated on small models may overstate the memorization share of frontier benchmark scores; re-running contamination audits per scale tier would reveal where the estimate changes.","A testable extension is to construct bijections on non-numeric spaces, such as string reversals with explicit mappings, to see whether the decoder-side bottleneck is a general property of autoregressive generation or specific to arithmetic contracts.","Mixture-of-experts architectures, excluded from this study, could split the difference: if routing disperses probability mass differently, decoder-side probes might recover signal at high nominal parameter counts."],"forward_implications":["Contaminated-benchmark score gaps on large dense models can no longer be read as direct memorization evidence, since the same pass@k drop appears under a provably unseen isomorphic restatement of the task.","Evaluations should report solution-space stability, such as opcode JSD, alongside correctness, because a bare pass@k drop conflates a narrowed representational channel with a forgotten algorithm.","Decoder-side contract compliance is the binding constraint at scale, so benchmarks that stress output formatting and encoding will discriminate model capability better than lexical prompt fuzzing.","The isomorphism protocol provides a reusable oracle: any bijective value transform yields a provable metamorphic relation, so transformed tests need no ground-truth labels beyond the original task."],"supporting_citations":[{"why":"Defines the synonym-fuzzing protocol that the paper reproduces; the encoder-side probe whose discriminative power is shown to saturate at scale.","marker":"[10]"},{"why":"Defines CoDeC and the seen/unseen corpus designations; the decoder-side probe whose AUC collapses as model size grows.","marker":"[13]"},{"why":"Supplies MBPP, the benchmark likely present in training data, used to test whether probes behave differently on contaminated versus post-cutoff benchmarks.","marker":"[2]"},{"why":"Supplies EffiBench, a post-training-cutoff efficiency benchmark used as an unseen control.","marker":"[41]"},{"why":"Supplies BigOBench, a post-cutoff algorithmic-complexity benchmark used to detect strategy shifts under representational load.","marker":"[42]"},{"why":"Supplies the Pile-based seen datasets (wikipedia, hackernews) for the CoDeC contamination axis.","marker":"[17]"},{"why":"Supplies LiveCodeBench as a guaranteed-unseen competitive-programming dataset for the CoDeC probe.","marker":"[35]"},{"why":"Establishes metamorphic testing, the foundation for treating the I/O isomorphism as a relation-preserving oracle.","marker":"[19]"}],"fun_headline_variants":["Scale-aware probes show code LLMs adapt, not just memorize","Memorization probes fail on large code models: Iso test reveals why","Code memorization diagnostics must be scale-aware, study shows","Large code LLMs evade memorization probes via surface adaptation","Iso transforms separate code memorization from representation load"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument that scaled models keep their algorithms rests on assuming a program's bytecode opcode histogram faithfully reflects its algorithmic strategy; the paper itself admits two behaviorally different programs can share the same histogram.","fun_headline_variants_meta":{"raw":{"variants":["Scale-aware probes show code LLMs adapt, not just memorize","Memorization probes fail on large code models: Iso test reveals why","Code memorization diagnostics must be scale-aware, study shows","Large code LLMs evade memorization probes via surface adaptation","Iso transforms separate code memorization from representation load"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000793,"raw_usage":{"total_tokens":3526,"prompt_tokens":1011,"completion_tokens":2515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":2431}},"tokens_in":627,"tokens_out":2515,"duration_ms":17196,"temperature":1.0,"reasoning_tokens":2431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:40:19.554311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a transformed problem where a frontier model's Iso generations pass but use different algorithmic families, such as an O($n^{2}$) and an O(n log n) sort on the same decoded input, compile both to bytecode, and compute their pooled JSD: if near-zero JSD coexists with genuinely different asymptotic strategies, the opcode proxy cannot support the compliance-not-competence conclusion.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems37, 11506–11544 (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies EffiBench, a post-training-cutoff efficiency benchmark used as an unseen control."},{"cited_title":"Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology (1998)","cited_arxiv_id":null,"evidence_quote":"Establishes metamorphic testing, the foundation for treating the I/O isomorphism as a relation-preserving oracle."}],"review_version":1}