{"id":"8d206d02-b755-4e89-bf6b-2daba0cbcae0","arxiv_id":"2502.07813","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A benchmark that encodes prompts in secret codes and measures how much accuracy models lose, showing most LLMs, especially open-source ones, struggle on this two-step compositional task.","lead":"CryptoX wraps standard AI reasoning tests (math, code, science, long-context retrieval) in coded prompts where some words are replaced by emoji or Morse-style symbols, then asks models to decode and answer. It tests dozens of large language models and finds that most, especially open-source ones, lose a large share of accuracy once the prompt is encoded, while top closed-source models like o1 and Gemini 2.0 Flash Thinking degrade far less.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CryptoBench's own Appendix G shows models answering correctly without decoding, so the claim that CryptoBench forces compositional reasoning is unsupported; scores may measure option-based shortcuts.","rationale":"The reader's weakest-assumption identification is correct and is independently supported by the paper's own appendix. The most damaging evidence is not external: Figures 35-37 document correct answers despite wrong or absent decoding for the very model used in the case study. If decoding is not necessary, the AUC and accuracy numbers confound compositional reasoning with partial-prompt retrieval from intact options, robustness to character-level perturbation, and memorization of public benchmarks. This is a construct-validity failure, not merely noise. It bears directly on the strongest claim: \"Most existing LLMs have weak CR abilities\" cannot be concluded from a task that at least some models solve without doing the supposed CR. The paper does have useful elements: the encoding framework is flexible, the multi-turn ablation points to composition costs, and the open/closed gap may be robust as a robustness finding. The appropriate response is to revise the framing and add quantitative evidence on causal dependence on decoding. I therefore keep the reader's CONDITIONAL verdict rather than escalating to REJECT, because the issue is measurable and fixable and the benchmark remains usable as a stress test.","tokens_in":33107,"tokens_out":7163,"duration_ms":74403,"concrete_test":"Run a systematic short-circuit audit on all 285 Crypto-MMLU-Alpha items at encoding level 10, using the released codebook to automatically check whether the model's response contains a correct decoding of the question, then cross-tabulate answer accuracy against decoding correctness. If answer accuracy on items with wrong or missing decoded question is substantially above the 25% random-choice baseline, or above accuracy on the same items with answer options removed, then decoding is not causally necessary and the benchmark does not isolate compositional reasoning. Repeat for o1, Gemini-2.0-Flash-Thinking, and Qwen2.5-72B-Instruct, and report the short-circuit rate. This directly tests the Section 3.1 \"forces models\" claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 claims CryptoBench \"forces models to conduct compositional reasoning, requiring them to decode the question before answering it.\" Appendix G contradicts this. In Figure 35, Qwen2.5-72B-Instruct gives the correct answer after mistranslating the encoded question and, on the Crypto-MMLU single-stage task, \"skipped the translation step entirely and directly answered the question.\" Figure 37 reports correct answers despite incorrect decoding in both conditions. The authors label these cases \"overfitting,\" but the implications for construct validity are stronger: if correct answers can be produced without decoding, then item scores do not require the compositional chain the benchmark is named for. This is aggravated by Algorithm 1, which encodes only 5 or 10 words in prompts of hundreds of characters; most of the question and all answer options remain in plaintext, so a model can exploit residual text and public MMLU answer associations. The logit-lens evidence in Section 4 is collected on Qwen2.5-3B/7B and Llama-3.1-8B, not on the high-scoring models, so it does not establish that o1 or Gemini-2.0-Flash-Thinking scores track decoded reasoning. The central claim that CryptoBench quantifies compositional reasoning and reveals a CR gap is therefore unsupported as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CryptoX, a framework that transforms existing benchmarks (MATH, MMLU, MBPP, BBH, Needle) by encrypting 5 or 10 words in each prompt using fixed codebooks and by adding answer-projection rules (numeric and alpha transformations). It constructs CryptoBench with 21 sub-datasets and roughly 7K cases, evaluates 20+ open- and closed-source LLMs, and reports that accuracy declines as more words are encoded, that closed-source models outperform open-source models, and that an AUC score provides a better ranking than average accuracy. The paper also presents logit-lens, neuron-activation, and reasoning-stage analyses on smaller open models to argue that models decode encoded words before solving the original problem.","tokens_in":33328,"tokens_out":3971,"duration_ms":38358,"significance":"If the benchmark measures a distinct compositional-reasoning ability, it would be a useful tool: it is grounded in established datasets, ships code, and the monotone accuracy decline with encoding level is a crisp and reproducible empirical finding. The paper is also transparent in providing detailed case studies in the appendix. However, the construct-validity premise is not established, and the paper's own appendix contradicts the central claim that the benchmark forces models to decode before answering. The evaluation of math answers relies on an unvalidated LLM-as-judge, and the mechanistic analyses are limited to small, low-scoring models. These issues make the headline conclusion ('most LLMs have weak compositional reasoning abilities') premature, although the benchmark itself may be salvageable with additional controls and more modest claims.","major_comments":[{"comment":"Section 3.1 states that 'CryptoBench forces models to conduct compositional reasoning, requiring them to decode the question before answering it,' but the paper's own case studies contradict this. In Figure 35, Qwen2.5-72B-Instruct answers correctly despite mistranslating the encoded question, and on the Crypto-MMLU single-stage task it 'skipped the translation step entirely and directly answered the question.' Figure 37 reports correct answers despite incorrect decoding in both conditions. Consequently, item-level scores do not require the intended decode-then-answer chain, and the central claim that CryptoBench quantifies compositional reasoning is unsupported. Please either redesign items so that correct answers require decoding (e.g., encode all content words and the answer options, or require the decoded question to be echoed in the response), or reframe the benchmark as measuring robustness to input corruption and explicitly state that construct-validity evidence is still needed.","section":"Section 3.1, Appendix G (Figures 35-37)"},{"comment":"With only 5 or 10 words encoded in prompts that are hundreds to thousands of characters long (e.g., Crypto-BBH average length ranges from 1585.3 to 3517.24 characters; Crypto-MMLU from 627.97 to 1333.6), most of the question and all answer options remain in plaintext. This leaves ample residual cues for models to answer correctly, especially for public benchmarks like MMLU that are likely present in pretraining corpora. The case studies in Appendix G are consistent with this alternative explanation: the model often answers correctly without decoding or after failed decoding. To support the compositional-reasoning interpretation, please add control conditions in which all content words are encoded (or a random-permutation control) and show that performance still tracks the number of encoded words; otherwise, the observed decline may reflect token-level perturbation sensitivity or partial-prompt pattern matching rather than compositional reasoning.","section":"Algorithm 1, Table 1"},{"comment":"Scores for Crypto-Math and Crypto-HighResolution are obtained with an LLM-as-judge (Doubao-Pro-256K) without reporting any validation, inter-judge agreement, comparison against exact match, or human ratings. Since mathematical answers are a significant part of the reported results, please provide evidence that the judge is reliable (e.g., agreement rates on a sample, or a comparison with exact-match with tolerance), or report exact-match numbers with clearly stated normalization rules for mathematical expressions.","section":"Section 3, 'Evaluation Metrics'"},{"comment":"The logit-lens and neuron-activation analyses are run only on Qwen2.5-3B/7B, Llama-3.1-8B, and similar small models, which are among the lowest-performing models in Table 2 (e.g., Qwen2.5-7B-Instruct has AUC 1.24). The paper draws general conclusions that 'LLMs' summarize subtask answers and that 'the layers of LLMs' exhibit a hierarchical pattern of subtask execution, but the highest-scoring models (o1, o3-mini, Gemini-2.0-Flash-Thinking) are not analyzed. Please restrict the mechanistic claims to the studied models or extend the analysis to at least one high-performing model before making claims about LLM compositional reasoning in general.","section":"Section 4, Appendices D-F"}],"minor_comments":[{"comment":"The paper uses the term 'overfitting' to describe correct answers despite decoding errors, but the evidence is equally consistent with memorization of public MMLU questions or with the model simply ignoring the encoding. Please clarify the terminology and discuss the possibility of benchmark contamination explicitly.","section":"Appendix G.2"},{"comment":"The sampling parameters are given as wide ranges ('temperature T = 0.7 to 1.0 and top p = 0.75 to 1.0') without per-model values. Please report the exact settings and number of runs for each model, especially because temperature affects the variance reported in Table 5.","section":"Section 3, 'LLMs'"},{"comment":"There are several typos and heading errors: 'Compostional Reasoning' (Section 5.1), 'Performane' (Appendix C.2), 'Thr Effect' (Appendix C.5), 'Instrcut' (Figure 6), 'nueron' (Section 4.2), and 'chararcter' (Figure 2).","section":"Throughout"},{"comment":"Table 2 is visually dense and uses color coding (green/blue/orange) that may not render reliably in print or for color-blind readers; consider splitting the 0/5/10 columns into separate rows or adding explicit numeric highlighting.","section":"Table 2"},{"comment":"The Spearman correlations with Chatbot Arena (0.61, 0.57, 0.19) are reported without the number of models or p-values; please add these statistics to make the correlation comparison interpretable.","section":"Appendix C.8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript may also benefit from citing prior work on cipher-based instruction following and robustness benchmarks (e.g., CipherChat, CipherBench, and related encoded-prompt evaluations), since the framing of being 'the first' to combine cryptographic encodings with existing benchmarks is easy to overstate. This is not a blocker for the review, but the editors may wish to ask the authors to position the work more carefully relative to that literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the benchmark is real and the empirical decay curve is clear; the construct-validity claim that it measures compositional reasoning is not supported, and the paper's own appendix gives you the counterexamples. Still worth a serious referee, but the authors need to reposition the central claim.\n\nWhat's new: CryptoBench takes MATH, MMLU, MBPP, BBH, and Needle, encodes 5 or 10 words with a codebook, and adds answer transformations. That gives a clean spectrum of difficulty, and the paper shows accuracy dropping monotonically as encoding increases, with a large gap between top closed and open models. The AUC aggregation over encoding levels is a reasonable way to summarize the curve, and the Spearman correlation with Chatbot Arena (0.61) is genuinely interesting—better than MMLU's 0.19. The logit-lens and neuron-activation analyses are a nice extra, though not load-bearing.\n\nWhere it wobbles: the central premise. Section 3.1 says CryptoBench 'forces models to conduct compositional reasoning, requiring them to decode the question before answering it.' Appendix G contradicts that directly. In Figures 35 and 37, Qwen2.5-72B-Instruct answers correctly after mistranslating or even skipping the decoding step; Figure 36 shows skipping entirely. The authors call this 'overfitting,' but the implication is stronger: if correct answers don't require the decode step, then item scores don't measure the compositional chain the benchmark is named for. Algorithm 1 encodes only 5 or 10 words in prompts that are hundreds of characters long, so the rest of the question and all the answer options remain in plaintext. A model can lean on residual text and its MMLU answer associations, which looks like exactly what the case studies show. The logit-lens evidence is collected on Qwen2.5-3B/7B and Llama-3.1-8B, not on the high-scoring models, so it doesn't rescue the claim for o1 or Gemini. Also: no repeated-run error bars, the LLM-as-judge for math answers isn't validated, the AUC in Table 2 is computed from three points, and the repo doesn't pin code versions. The 'for the first time' framing in the abstract is overblown too; encoding words of existing prompts is a known robustness/instruction-following trick.\n\nSo the empirical result stands as a stress-test finding, not as a compositional-reasoning measurement. The paper deserves a serious referee—the benchmark is useful and the correlation with human preference is worth chasing—but the authors need to temper the 'forces compositional reasoning' language, add seeded repeated runs and judge validation, and either show that correct answers without decoding are rare or re-frame the benchmark as measuring robustness to input perturbation plus reasoning.","headline":"A usable stress-test benchmark whose central 'compositional reasoning' claim is undercut by its own case studies; worth refereeing, but only after the authors reposition the claim.","tokens_in":33915,"tokens_out":2851,"would_cite":true,"duration_ms":27432,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark that encodes prompts reveals that most LLMs are weak at compositional reasoning, with open-source models trailing closed-source ones by a large margin.","keywords":["compositional reasoning evaluation","instruction encryption","cipher-based prompting","large language models","benchmark AUC","mechanistic interpretability","logit lens","open-source vs closed-source models"],"falsifier":"The paper's own Appendix G case studies show models producing correct answers even when they fail to decode the question or skip the decoding step outright. A direct test of the central mechanism would measure, per item, whether final-answer accuracy tracks decoding accuracy (for example, ROUGE or BLEU against the true plaintext question); if many items are answered correctly despite failed decoding, the benchmark is measuring something other than decode-then-reason compositionality.","tokens_in":32882,"feed_emoji":"🧩","tokens_out":6620,"duration_ms":59945,"temperature":0.7,"pith_summary":"The paper argues that standard reasoning benchmarks are saturated and miss a distinct ability: compositional reasoning, the capacity to combine separate subtasks in a single model run. Its proposal, CryptoX, rewrites existing benchmark questions by encoding a variable number of words with a cipher (for instance, emoji substitution) and adding answer-projection rules, so that a model must decode the prompt and then solve the original task in one pass. The resulting CryptoBench covers 21 sub-datasets and roughly 7,000 cases drawn from MATH, MMLU, BBH, MBPP, and long-context needle retrieval, tested on more than 20 models. The central finding is that accuracy falls sharply as more words are encoded, that most LLMs show weak compositional reasoning, and that open-source models score far below closed-source ones, with o1, o3-mini, and Gemini-2.0-Flash-Thinking the strongest. A sympathetic reader would care because the benchmark claims to expose a capability gap that ordinary accuracy numbers hide, and the accompanying mechanistic analyses suggest the gap reflects how models order decoding, reasoning, and answer summarization across layers.","feed_headline":"Encoded prompts expose a wide reasoning gap in LLMs","feed_subtitle":"Most models solve encoded questions poorly, and open-source models trail closed ones, with the gap widening as encoding grows.","key_machinery":"The engine of the method is instruction encryption: a codebook (base Morse, emoji Morse, or emoji shuffle) randomly replaces a chosen number of words in a prompt, and the encoding rules are explicitly included in the prompt. Instruction transformation then adds further projection steps from the decoded answer to the required output, such as mapping A–D to 1–4 or appending the first character of the option text. Performance is summarized as an area-under-curve (AUC) score computed as the number of encoded words ranges from 0 to 10, which the paper argues is more sensitive than a single accuracy value. On the mechanism side, logit lens and neuron activation analysis identify which layers track decoded words versus answer options, and reasoning-stage analysis labels layer bands as shallow (decomposition), intermediate (subtask solving), and deep (summarization).","core_discovery":"CryptoBench quantifies compositional reasoning by forcing a model to perform two coupled subtasks in one run: decode a cipher-transformed question using rules stated in the prompt, then answer the original question under an additional output transformation (for example, convert the chosen option letter to its alphabet position). The authors report that no tested model handles this well at high encoding levels, that the gap between open-source and closed-source models is large — o1 reaches 83.7% average accuracy while the smallest open-source models fall to near zero — and that model size, instruction tuning, and architecture all shift performance. Their mechanistic experiments indicate that models decompose the problem, decode in early-to-middle layers, solve the base task, and summarize subtask answers in later layers, and that neuron activation for encoding rules precedes the logit-lens rise of decoded words. The paper's conclusion is that compositional reasoning is a measurable ability distinct from the skills tapped by vanilla benchmarks, and that CryptoBench's area-under-curve score over encoding levels is a more discriminating metric than single accuracy numbers.","pith_inferences":["Editorial inference: Appendix G's cases where a model answers correctly despite failed or skipped decoding suggest that CryptoBench scores may partly reflect memorized or shortcut answers, so the benchmark's construct validity as a pure measure of compositional reasoning remains an open question.","Editorial inference: The framework is codebook-agnostic, so an immediate extension would be to test whether a model's ranking under emoji shuffle transfers to structurally different ciphers (for example, Huffman coding or RSA-style encryption), which would show whether the measured ability is generic compositionality or specific familiarity with the cipher format.","Editorial inference: The multi-turn versus single-turn ablation implies that decomposing a task into separate prompts is easier than composing the same subtasks in one context; a natural application is to use that gap as a diagnostic for how much context-composition overhead a model carries.","Editorial inference: The logit-lens finding that decoded-word probability peaks before answer-option probability suggests a testable intervention: explicitly forcing a model to output the plaintext question before answering should raise accuracy if the mechanism is causal, and should leave accuracy unchanged if decoding is epiphenomenal."],"forward_implications":["Standard benchmarks such as MMLU can be stress-tested by re-encoding prompts with a fresh codebook, yielding a difficulty curve instead of a single accuracy point.","Model rankings change under CryptoBench relative to vanilla accuracy, so a model's advertised reasoning skill on familiar benchmarks does not reliably predict its performance when prompts are superficially transformed.","The strong drop in accuracy as encoded-word count rises implies that improving compositional reasoning—not just base-task skill—could be a lever for better generalization.","The observed layer-level ordering (decode, then reason, then summarize) suggests that architectures or inference procedures that enforce explicit decomposition may improve CR performance.","AUC over encoding levels gives benchmark designers a graded metric that keeps some discrimination even when models saturate the vanilla task."],"supporting_citations":[{"why":"Supplies the MMLU questions that become Crypto-MMLU and serves as the vanilla baseline.","marker":"[Hendrycks et al., 2020a]"},{"why":"Supplies the MATH500 subset used for Crypto-Math.","marker":"[Hendrycks et al., 2021]"},{"why":"Supplies the BBH tasks used for Crypto-BBH.","marker":"[Suzgun et al., 2022]"},{"why":"Supplies the MBPP-sanitized code tasks used for Crypto-MBPP.","marker":"[Austin et al., 2021]"},{"why":"Supplies the long-context needle-retrieval setup behind Crypto-Needle-30K.","marker":"[gkamradt, 2023]"},{"why":"Provides the logit-lens technique used to trace decoding and answer-option probabilities across layers.","marker":"[Nostalgebraist, 2020]"},{"why":"Motivates the encryption-style transformation of prompts as the core of the evaluation method.","marker":"[Mushtaq et al., 2017]"},{"why":"Grounds the definition of compositional reasoning as decomposition and recombination of simpler components.","marker":"[Hou et al., 2023]"}],"fun_headline_variants":["CryptoBench: Encoded prompts reveal LLM reasoning gap","Two-step cipher test: open-source LLMs far behind closed","O1 leads, open-source near zero on CryptoBench encoding","Compositional reasoning quantified: CryptoBench breaks it down","LLMs fail to decode-and-reason: CryptoBench scores low"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model's accuracy on prompts that must be decoded before answering is a valid measure of compositional reasoning as a distinct ability, separate from instruction following, resilience to input perturbations, or memorization of benchmark answers; if that premise gives way, the scores blend unrelated capabilities.","fun_headline_variants_meta":{"raw":{"variants":["CryptoBench: Encoded prompts reveal LLM reasoning gap","Two-step cipher test: open-source LLMs far behind closed","O1 leads, open-source near zero on CryptoBench encoding","Compositional reasoning quantified: CryptoBench breaks it down","LLMs fail to decode-and-reason: CryptoBench scores low"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000429,"raw_usage":{"total_tokens":2183,"prompt_tokens":924,"completion_tokens":1259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":1173}},"tokens_in":540,"tokens_out":1259,"duration_ms":12176,"temperature":1.0,"reasoning_tokens":1173,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T18:29:58.864083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The paper's own Appendix G case studies show models producing correct answers even when they fail to decode the question or skip the decoding step outright. A direct test of the central mechanism would measure, per item, whether final-answer accuracy tracks decoding accuracy (for example, ROUGE or BLEU against the true plaintext question); if many items are answered correctly despite failed decoding, the benchmark is measuring something other than decode-then-reason compositionality.","supporting_citations":[],"review_version":1}