{"id":"1f84fa01-4b0a-4f6c-865d-6a8646fade8a","arxiv_id":"2505.08389","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors define contamination resistance as a benchmark property and show that most tested LLMs score near zero on Caesar-cipher encoding and decoding when the shift is not 3 and the text is random nonsense.","lead":"This paper introduces a benchmark based on Caesar ciphers that is designed to resist test-data contamination, and reports that most large language models fail most of its tasks. A generalist might read it because it offers a cheap, repeatable way to check whether AI models generalize or merely reproduce memorized patterns from training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's contamination resistance is inferred, not measured; task-level contamination (the ubiquitous shift-3 Caesar mapping) is not controlled and may drive the only success.","rationale":"The reader identified the right neighborhood: the claim that generated cipher texts have not been encountered during pretraining is unverified. My concern sharpens this. The paper's own evidence—Table 12 shows all tested models can define a Caesar cipher and most spontaneously use shift=3—indicates the shift-3 mapping itself is in the training distribution. Instance-level novelty does not remove this task-level contamination. The benchmark may still be useful as a source of novel instances, and the empirical finding that models fail on non-3 shifts is worth reporting. However, the central claim that the benchmark is an 'excellent example of a contamination resistant benchmark' is not established by the presented data. Because the conditional verdict already reflects unresolved risks around contamination and reproducibility, my analysis does not move the verdict; it strengthens the condition that must be met before the contribution can be accepted as stated.","tokens_in":19239,"tokens_out":5809,"duration_ms":62542,"concrete_test":"Run a fresh evaluation using 100 newly generated, never-published random nonce-word plaintexts under shifts 1–25 (excluding 3, 6, 9, 12) and the same 25 natural phrases, on GPT-4o and LLaMA3.1-70B with the base and open prompts. If exact-match accuracy is near zero for all non-3 shifts while shift=3 natural remains high, task-level contamination is confirmed and the contamination-resistance claim must be weakened to 'instance-level novelty only'; if accuracy is high for some other shift, the shift-3 effect is not a contamination signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines contamination resistance through instance-level dynamism: 'users can generate an infinite number of unique texts.' The empirical case in Section 5.1, however, attributes the shift-3 versus shift-6/9/12 gap to the prevalence of shift-3 ciphers in pretraining corpora, citing McCoy et al. (2024a). That is task-level contamination: the model may have memorized the mapping rule, not just particular strings. Instance-level novelty does not control for this. The paper's own Table 12 is direct evidence: all six models correctly define a Caesar cipher and four of six spontaneously use shift=3 examples. Section 5.2 further explains GPT-4o's natural-text advantage at shift=3 as memorized plaintext-to-cipher mappings. Thus the only setting in which GPT-4o succeeds is one the authors themselves identify as contaminated; the benchmark's resistance claim is therefore not demonstrated. The abstract's 'when contamination is controlled' conclusion is an inference from the absence of success in other conditions, not a measurement of contamination. A plausible alternative reading is that LLMs lack a general shift parameter and rely on memorized canonical examples; that reading still supports 'inconsistent competence' but not 'contamination resistant benchmark.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a definition of \"contamination resistance\" for evaluation benchmarks, proposes a Caesar-cipher-based benchmark as an instance, and tests six LLMs across four shift values, three prompting strategies, and two plain-text types (natural English and random words). The main empirical finding is that most models achieve near-zero exact-match accuracy on most cells, while GPT-4o shows partial success when the shift is 3 and the plain text is natural English; the authors attribute this shift-dependent success to contamination and summarize the overall pattern as \"inconsistent competence.\" The paper also includes pilot analyses of prompt effects, output-position accuracy, few-shot prompting, and two reasoning models (QwQ-32B and DeepSeek-R1), plus an integrity check for shift-direction ambiguity.","tokens_in":19437,"tokens_out":7680,"duration_ms":73885,"significance":"If the central claim were established, the benchmark would be a valuable, nearly free dynamic evaluation that controls for memorization, and the observed failures would provide evidence about LLM generalization on a simple linear mapping. The paper has real strengths: the experimental protocol is clearly specified, code and data are made available, the broad failure pattern (all non-GPT-4o models at near-zero accuracy) is robust across models and prompts, and the integrity check for shift direction is a thoughtful addition. However, the central claim as stated—that the benchmark is contamination resistant and that the results demonstrate behavior \"when contamination is controlled\"—is not supported by the evidence. The only condition with substantial success (shift=3, natural English) is, by the paper's own analysis, a condition in which task-level contamination is likely. This gap reduces the force of the empirical contribution unless the claim is reframed or directly tested.","major_comments":[{"comment":"The contamination-resistance argument is instance-level, not task-level. Section 3 says the benchmark is contamination resistant because \"users can generate an infinite number of unique texts,\" which prevents memorization of particular strings. It does not prevent a model from having memorized the general Caesar-cipher rule or a canonical shift value. The paper's own Section 5.1 attributes GPT-4o's shift-3 advantage to the prevalence of shift-3 ciphers in web corpora (citing McCoy et al., 2024a), and Table 12 shows that four of six models spontaneously give shift-3 examples when asked to define the Caesar cipher. Thus the one condition in which the benchmark \"works\" (GPT-4o, shift=3, natural text) is, according to the authors' own reasoning, a contaminated condition. The abstract's claim that models \"struggle with this benchmark when contamination is controlled\" is therefore an inference from the absence of success in the other cells, not a demonstrated measurement of contamination. I recommend either limiting the claim to instance-level contamination resistance and revising the abstract and conclusion accordingly, or directly testing task-level contamination, e.g., with a controlled pretraining experiment on small models or with per-instance membership checks.","section":"Section 3; Section 5.1; Table 12"},{"comment":"The sample size of 25 plain texts per cell is too small to support the precision of the quantitative claims. For example, GPT-4o's exact-match accuracy of 0.82 on the natural, shift=3, open-prompt cell (Table 2) has an approximate standard error of sqrt(0.82*0.18/25)=0.077, so a reported difference like 0.36 vs. 0.34 between shift=12 and other conditions (Table 13) is well within noise. No confidence intervals or significance tests are reported anywhere, yet the paper makes fine-grained comparative statements about prompt effects, output-position accuracies (Section 5.4), and few-shot improvements (Section 5.5). The broad qualitative pattern—most models at 0.00 in most cells—is robust, but the quantitative details about GPT-4o should be either supported by larger sample sizes or presented with appropriate uncertainty estimates and cautious language.","section":"Section 3; Tables 2 and 13"},{"comment":"The attribution of the shift-3 advantage to contamination is circumstantial rather than demonstrated. The paper relies on McCoy et al. (2024a) for the claim that shift-3 ciphers are more prevalent in pretraining corpora, but that citation does not establish that the specific test instances, or the general shift-3 rule, are in the training data of the tested models. The alternative explanation—that the models have a canonical shift bias or have learned the rule as a known task—is equally consistent with Table 12, which shows that all tested models can define a Caesar cipher and most give shift-3 examples. The paper should distinguish \"contamination\" from \"task familiarity,\" and if the contamination claim is to be maintained, it needs a more direct form of evidence, such as a controlled experiment in which pretraining data are known or a membership-style test on the specific cipher texts.","section":"Section 5.1; Section 5.2"}],"minor_comments":[{"comment":"The caption says \"first (id=1) and second (id=2)\", but Table 5 and the surrounding text use id=0,1,2 for the first, second, and third output characters; this caption should say \"second (id=1) and third (id=2)\".","section":"Table 8 caption"},{"comment":"There are several typos: \"straight forward\" should be \"straightforward\" (Section 2.2 and Section 4); Table 2 has \"a verage\" instead of \"average\"; Section 5.7 has \"an addition set\" instead of \"an additional set\"; Appendix B has \"4bit\" instead of \"4-bit\".","section":"Multiple sections"},{"comment":"The sentence \"We only experimented with the code prompt using GPT-4o\" is followed immediately by a figure reference; consider moving Figure 2 into the sentence that discusses the code-prompt results for readability.","section":"Section 5.3"},{"comment":"The transition between the decoding query shown in Table 15 and the encoding query \"rakibo zlmqwe\" is confusing. State explicitly that the decoding task used the query from Table 15 and that the encoding task used a different (reverse) query, so readers do not think the same query was used for both tasks.","section":"Section 5.6"},{"comment":"Tables 5 and 6 use different column labels (\"id\" vs. \"char_id\") for the same character-position variable; unify the notation.","section":"Section 5.4"},{"comment":"Use \"contamination-resistant\" consistently when the phrase is used as an adjective; the text currently switches between \"contamination resistant\" and \"contamination-resistant.\"","section":"Throughout"},{"comment":"The conclusion that \"fine-tuning models on Caesar ciphers may not be a viable solution\" (Section 5.5) goes beyond the data, since the experiment only tests zero-shot and 50-shot prompting with the base prompt on GPT-4o, not fine-tuning. Either remove the sentence or label it as speculation.","section":"Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a thesis-derived manuscript with a useful, easily reproducible benchmark design, but the central claim of contamination resistance is currently over-stated relative to the evidence. The authors may be able to address this by reframing the claim around instance-level novelty and by softening the \"contamination controlled\" language, or by adding a direct test of task-level contamination (e.g., a small controlled pretraining study). The statistical precision of the quantitative claims also needs strengthening. Despite the rough writing, the empirical pattern is interesting and the benchmark concept is worth publishing if the claims are appropriately scoped."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before reading. The Caesar-cipher benchmark itself is cheap and reusable, and the failure pattern they document—models doing fine at shift 3 only when the input looks like natural English—is real and worth explaining. But the headline \"contamination resistant\" is not something they demonstrate; it is an assumption supported by the infinitude of the cipher space, and the only clean success (GPT-4o, shift=3, natural text) is one they attribute to the model having memorized that exact mapping.\n\nWhat is actually new here is modest but real. They propose a concrete list of criteria for contamination resistance and instantiate them with a benchmark that varies shift (3,6,9,12), plain-text type (natural vs. random words), prompt format, and few-shot context, across six models. The random-word condition and the multi-model comparison go beyond McCoy et al. 2024a, which they cite for the shift-3 prevalence effect. I also want to credit them for the integrity check in Section 5.7 (allowing both shift directions) and for the straightforward Limitations section. The paper does not overclaim about the code or data; they ship it.\n\nThe soft spots are in proportion to the claim. The key one is that the contamination-resistance property is never measured. They argue that the space of possible cipher texts is infinite, so no model can memorize all instances—that gives instance-level novelty, but it does not rule out task-level contamination. The model could have memorized the rule \"shift by 3 for natural English\", and indeed their own Table 12 shows four of six models reach for shift=3 examples when describing a Caesar cipher. So the benchmark as run is better described as \"a task where the models show no consistent ability to generalize the shift parameter,\" not \"contamination controlled.\" The abstract's \"when contamination is controlled\" is an inference from zero performance elsewhere, not a direct measurement.\n\nSample size is a secondary issue: 25 plain texts per cell gives large binomial uncertainty, and they do not report intervals. Table 2 pools encoding and decoding even though Table 3 shows those behave differently; that should be stated up front. None of these are fatal to the empirical observation, but they weaken the numerical specifics.\n\nWho should read it: anyone building evaluation sets, and people studying how LLMs generalize on simple algorithmic mappings. It is a good discussion piece for a reading group, less a landmark. I would send it to a serious reviewer—the benchmark is reusable and the masking of the contamination claim is a fixable overstatement—but I would not accept it as-is. Ask for uncertainty estimates, a control that tries to teach the model the rule (e.g., few-shot with shift=6 after showing shift=3), and a rewritten abstract that says \"uncontrolled task-level contamination remains plausible\" instead of \"contamination is controlled.\"","headline":"A cheap, honest benchmark paper whose shift-3 failure pattern is real, but whose 'contamination resistant' label is an inference, not a measurement—worth refereeing, not accepting as-is.","tokens_in":19968,"tokens_out":3564,"would_cite":false,"duration_ms":36857,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a benchmark built from Caesar ciphers is contamination resistant, because every test instance can be freshly generated, and that once contamination is controlled, current LLMs fail at this simple linear-mapping task.","keywords":["contamination resistance","Caesar cipher","benchmark","large language models","data leakage","memorization","inconsistent competence","generalization"],"falsifier":"Take one of the paper's shift=6, shift=9, or shift=12 cipher-text/plaintext pairs with a random-word plaintext and search the released training corpora of LLaMA and Qwen for that exact string; if it is found, the contamination-resistance claim for those instances fails. Alternatively, prompt GPT-4o with a fresh random-word plaintext at a shift the paper did not use, such as 5 or 7, and count exact-match successes: high accuracy on unseen shifts would show the failure at 6, 9, and 12 is not simply contamination.","tokens_in":19022,"feed_emoji":"🔐","tokens_out":6974,"duration_ms":62868,"temperature":0.7,"pith_summary":"The paper introduces the concept of contamination resistance: a benchmark that keeps assessing the same capability while generating an infinite supply of fresh test instances, so memorization from pre-training gives no advantage. It proposes Caesar ciphers as such a benchmark, since any text can be shifted by any number of alphabet positions at essentially zero curation cost. When the authors test widely used LLMs under this controlled setting, most score near zero on exact-match accuracy, and GPT-4o succeeds mainly on shift=3 with natural English words. They interpret this as evidence that benchmark scores are inflated by contamination and that current models lack genuine generalization on a trivial linear mapping.","feed_headline":"LLMs fail Caesar ciphers once contamination is controlled","feed_subtitle":"A shifting-letter test shows current models memorize familiar examples but can't generalize to fresh ones.","key_machinery":"The load-bearing object is the Caesar-cipher instance generator: a plaintext (natural English words or random nonsense words) combined with one of four shift values (3, 6, 9, 12), posed as an encode/decode prompt. The generator's infinitude is the actual mechanism of contamination resistance—each text-shift pair is unique, so a model cannot have memorized the instance, and any correct answer must come from computing the shift. The paper also uses exact-match accuracy and character error rate as evaluation metrics, and a set of prompt variants (open, base, dict, code, plus zero- and few-shot settings) to distinguish genuine computation from memorized routines. The verbalization, lookup-table, and code behaviors serve as probes that separate a correct reasoning chain from a correct final answer.","core_discovery":"The central discovery is that the Caesar cipher satisfies a definition of contamination resistance: it probes genuine capabilities (logical deduction, arithmetic reasoning, generalization), admits infinitely many fresh instances so memorization gives no advantage, keeps task complexity constant across instances, and costs almost nothing to update. Empirically, when the authors control for contamination, widely used LLMs perform at or near zero exact-match accuracy on simple encoding/decoding; GPT-4o succeeds mainly at shift=3 with natural-language words, and its performance collapses for shifts 6, 9, and 12 and for random nonsense words. The authors attribute the shift=3 advantage to contamination, noting that shift=3 examples are prevalent in web corpora and that models volunteering definitions of the cipher tend to use shift=3 examples. They call the overall pattern \"inconsistent competence\": models can verbalize correct letter-by-letter reasoning and still output the wrong answer, can produce correct lookup tables and code while failing the final answer, and do not benefit from 50-shot demonstrations that already contain all letter mappings. The conclusion is that apparent LLM competence on benchmarks can be inflated by memorization, and that contamination-resistant benchmarks reveal a lack of true generalization.","pith_inferences":["The infinite-instance construction extends beyond Caesar ciphers: any parameterized family of deterministic text transformations (other affine ciphers, Vigenère with random keys, simple formatting rules) could generate contamination-resistant probes for symbol manipulation at near-zero curation cost.","A stronger test than the paper's would draw a fresh random shift at query time and compare models on shifts never seen in pre-training; the paper's use of only four shifts leaves the space of possible tasks sparsely sampled.","The lookup-table/code discrepancy suggests these scaffolding prompts may themselves be memorized routines; this can be probed by asking models to build a lookup table for a nonstandard alphabet, which training data would not contain.","If contamination is indeed what makes shift=3 easier, evaluation protocols should report scores on never-seen instances separately from static benchmarks, and models with a large gap should be flagged as relying on memorization."],"forward_implications":["A continuously regenerated Caesar-cipher test set can serve as a low-cost contamination control: any model that scores high on it must be computing the shift, not retrieving memorized text.","Performance gaps between shift=3 and other shifts, and between natural and nonsense words, should be read as evidence of memorization rather than task difficulty.","The observation that correct reasoning chains coexist with wrong final answers means model explanations cannot be taken at face value as evidence of competence.","Because 50-shot demonstrations covering all letter mappings did not help GPT-4o, in-context learning of a deterministic linear function appears unreliable in these models.","Larger models' lower character error rates (LLaMA3.1-70B vs 8B, Qwen2.5-32B vs 7B) suggest scale helps partially, but not enough to reach reliable zero-shot performance."],"supporting_citations":[{"why":"Supplies the key supporting observation that LLM performance tracks the probability of input/output in pre-training data, and that shift=3 cipher examples are more prevalent in web corpora; the paper's contamination explanation of the shift effect rests on this.","marker":"McCoy et al., 2024a"},{"why":"Establishes test-set contamination as a threat to LLM evaluation reliability, motivating the proposed contamination-resistance criteria.","marker":"Sainz et al., 2023"},{"why":"Shows LLMs are few-shot learners, which sets the expectation that 50 demonstrations covering all letter mappings should let GPT-4o solve a linear mapping; the paper's negative few-shot result is meaningful against this baseline.","marker":"Brown et al., 2020"},{"why":"Provides the LLaMA3.1-8B/70B models used in the experiments and their training details.","marker":"Dubey et al., 2024"},{"why":"Provides the Qwen2.5-7B/32B models used in the experiments and their training details.","marker":"Yang et al., 2024"},{"why":"Demonstrates that transformers can learn simple function classes in-context, which the paper tests against Caesar-cipher few-shot prompting.","marker":"Garg et al., 2022"},{"why":"Describes private benchmarking as an existing approach to contamination; the paper compares contamination resistance with this confidentiality-based alternative.","marker":"Chandran et al., 2024"},{"why":"Represents the dynamic-benchmark strategy that the paper critiques and improves on via infinite instance generation.","marker":"Jain et al., 2024"}],"fun_headline_variants":["Caesar cipher test exposes LLM memorization","When contamination is controlled, LLMs fail simple ciphers","LLMs stumble on fresh Caesar ciphers","Memorization, not reasoning, behind LLM benchmark scores","New benchmark reveals LLMs can't generalize to novel ciphers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's contamination resistance rests on the assumption that the freshly generated cipher texts with shifts 6, 9, and 12 and random nonsense-word plaintexts were never present in the tested models' training data; the paper does not verify this and relies instead on the infinitude of the cipher space.","fun_headline_variants_meta":{"raw":{"variants":["Caesar cipher test exposes LLM memorization","When contamination is controlled, LLMs fail simple ciphers","LLMs stumble on fresh Caesar ciphers","Memorization, not reasoning, behind LLM benchmark scores","New benchmark reveals LLMs can't generalize to novel ciphers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1323,"prompt_tokens":944,"completion_tokens":379,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":298}},"tokens_in":560,"tokens_out":379,"duration_ms":3529,"temperature":1.0,"reasoning_tokens":298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:55:34.544978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the paper's shift=6, shift=9, or shift=12 cipher-text/plaintext pairs with a random-word plaintext and search the released training corpora of LLaMA and Qwen for that exact string; if it is found, the contamination-resistance claim for those instances fails. Alternatively, prompt GPT-4o with a fresh random-word plaintext at a shift the paper did not use, such as 5 or 7, and count exact-match successes: high accuracy on unseen shifts would show the failure at 6, 9, and 12 is not simply contamination.","supporting_citations":[{"cited_title":"What can transformers learn in-context? A case study of simple function classes","cited_arxiv_id":null,"evidence_quote":"Demonstrates that transformers can learn simple function classes in-context, which the paper tests against Caesar-cipher few-shot prompting."}],"review_version":1}