{"id":"03ba1fbf-b284-422d-ae56-1f4667b4f130","arxiv_id":"2505.23833","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A symbol-remapping benchmark shows that LLMs' arithmetic and symbolic reasoning accuracy drops sharply when familiar digits and operators are replaced, revealing heavy reliance on memorized tokens rather than abstract rules.","lead":"Researchers built a math-style benchmark that replaces digits and operators with random symbols to separate real reasoning from token memorization in large language models. They report that current models, including large API models and agent frameworks, collapse on non-decimal arithmetic and symbolic remapping, so the benchmark's memory dependence score may help track progress on genuine abstraction.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proof of Theorem 3.8 (Appendix A.2.2, Step 4) asserts the key implication rather than deriving it: small accuracy drop under remapping does not by itself imply an invariant abstraction mapping, so Delta's validity as a memory-dependence measure is unproven.","rationale":"The paper's advertised contribution is a theoretically grounded benchmark, and Section 3.5 claims Theorems 3.7-3.9 validate Gamma and Delta. Theorem 3.8 is the only result connecting Delta to abstraction, and its proof in Appendix A.2.2 is a placeholder: Step 2 restates the definition of Delta, Step 3 uses Gamma, and Step 4 writes 'Let f be the abstraction mapping. For small δ: f(c) ≈ f(M(c))' as though the necessary invariance were already established. The conclusion of the theorem is exactly the invariance needed for Delta to measure abstraction, so the argument is circular at the load-bearing point. I agree with the reader that this is the weakest assumption. The empirical findings (high Delta for LLMs, low Delta for humans on small subsets, fine-tuning results on remapped data) are suggestive and the benchmark may still be useful, but they do not independently verify Theorem 3.8. A high Delta could also reflect tokenization or prompt-format difficulties introduced by remapping, and a low Delta could reflect in-context adaptation to the new symbols rather than a representation-invariant rule. Thus the central claim that Delta robustly measures memory dependence is not established. The correct disposition remains CONDITIONAL: accept only if the theorem is either removed from the framing or replaced with a genuine derivation, and if the code and data are released with a commit hash and confidence intervals. No change to the reader's verdict is needed.","tokens_in":28426,"tokens_out":6925,"duration_ms":84137,"concrete_test":"Independently re-derive Theorem 3.8 from Definitions 3.1-3.6, replacing Step 4 of Appendix A.2.2 with a genuine derivation: from |P(Ĥ(c,r)=q) - P(Ĥ(M(c),r)=M(q))| ≤ δ and Γ ≥ γ, prove that there exists an abstraction map f with f(c) ≈ f(M(c)) and Re(f(c),r)=q with probability at least γ-δ. If the derivation cannot be completed without an additional assumption on Ĥ's architecture, the theorem is unsupported. To confirm the failure concretely, build a baseline wrapper that recovers the symbol mapping M from the remapped few-shot examples in the prompt and translates the question back to the original symbols before answering; if this baseline attains Γ ≈ 1 and Δ ≈ 0 while using no invariant abstraction, Theorem 3.8's implication is false for realistic LLM behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Delta measures memory dependence and that low Delta with high Gamma indicates Rule-Inductive abstraction rests entirely on Theorem 3.8. Its proof in Appendix A.2.2 contains the conclusion as an unproved assertion: Step 2 gives |P(Ĥ(c,r)=q) - P(Ĥ(M(c),r)=M(q))| ≤ δ from the definition of Delta, and Step 4 immediately concludes f(c) ≈ f(M(c)). This is a non-sequitur. Equality of output distributions under two input encodings does not imply that the model's internal abstraction mapping f is invariant. A model could achieve low Delta by using the remapped few-shot examples in the prompt to memorize the new symbol mapping, by exploiting cues that M preserves (string length, position, or unremapped operator tokens), or by applying a token-level nearest-neighbor rule, none of which is the invariant abstraction the theorem claims to certify. No additional assumption or derivation connects Delta to f-invariance, so the theoretical framework does not establish that Delta measures abstraction rather than something correlated with it. The empirical results are suggestive, but they do not independently verify the theorem; the central theoretical claim is unsupported at its load-bearing step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a theoretical framework for measuring abstract reasoning in LLMs, formalizing abstraction as a mapping f and reasoning as a rule application Re, and introducing two metrics: Gamma, the accuracy on original symbols, and Delta, the accuracy drop after systematic symbol remapping. The authors state three theorems claiming that Gamma measures rule-given reasoning potential, that low Delta combined with high Gamma indicates rule-inductive abstraction, and that a weighted combination of the two metrics yields a valid score. They build a benchmark of 82 symbolic sub-datasets across six categories, evaluate 7B-70B open models, API models, and agent frameworks under direct and chain-of-thought prompting, and report that LLMs perform poorly on non-decimal arithmetic, that operand remapping causes larger drops than operator remapping, and that fine-tuning and human baselines confirm a large abstraction gap. The central claim is that Delta robustly measures memory dependence and that its combination with Gamma diagnoses true abstraction versus memorization.","tokens_in":28733,"tokens_out":4608,"duration_ms":54205,"significance":"If the theoretical claims were established, the proposed benchmark would provide a principled diagnostic for abstraction in LLMs, and the empirical pattern—near-zero non-decimal arithmetic performance, high operand-specific memory dependence, and limited gains from chain-of-thought and multi-agent frameworks—would be a valuable and timely stress test for the field. The paper ships a substantial evaluation: many models, a human baseline, a fine-tuning control, and a public code release commitment, all of which strengthen the empirical contribution. However, the theoretical validation is the central advertised contribution, and it is not actually established: the proofs in Appendix A.2 largely restate definitions, and the key step in the proof of Theorem 3.8 is a non-sequitur. The benchmark's empirical findings are suggestive, but they cannot be interpreted through the proposed Delta metric without a valid theory or additional controls, so the main claim currently overreaches the evidence.","major_comments":[{"comment":"The proof of Theorem 3.8 asserts the key implication rather than deriving it: from |P(Ĥ(c,r)=q) − P(Ĥ(M(c),r)=M(q))| ≤ δ, Step 4 immediately concludes f(c) ≈ f(M(c)) and then Re(f(c),r) ≈ Re(f(M(c)),r). This does not follow. Equality of output distributions under two input encodings does not imply that the model's internal abstraction mapping f is invariant to M; a model could achieve low Delta by memorizing the remapped symbols from the few-shot examples in the prompt, by exploiting cues that M preserves (string length, position, operator tokens, or the structure of the examples), or by applying a token-level nearest-neighbor rule. None of these is the invariant rule-based reasoning the theorem claims to certify. Since Theorem 3.8 is the load-bearing justification for interpreting Delta as a memory-dependence measure, this gap invalidates the central theoretical claim. The authors either need to add an explicit axiom connecting output invariance to f-invariance and justify it, or provide independent evidence—e.g., testing with multiple disjoint mappings per instance, removing remapped examples from the prompt, or controlling for cue preservation—that low Delta cannot be explained by these alternative mechanisms.","section":"Appendix A.2.2, Theorem 3.8, Step 4"},{"comment":"Theorems 3.7 and 3.9 are restatements of the definitions rather than substantive validation. Theorem 3.7's proof is: Gamma ≥ γ by definition, then by the law of large numbers the empirical proportion approximates the probability; this does not establish that high Gamma indicates 'Rule-Given potential' beyond the definitional statement that the model answers correctly with high probability. Theorem 3.9 defines F as a weighted sum and then verifies three algebraic cases; calling this a theorem adds no mathematical content. The paper repeatedly advertises a 'rigorous theoretical foundation' and 'foundational theorems that validate our approach,' but the appendix shows that the theorems reduce to definitions and the single non-definitional step (Theorem 3.8) is unjustified. The authors should either substantially strengthen the theory or explicitly reframe the contribution as an empirical benchmark with a stated, unproven interpretive assumption.","section":"Section 3.5 and Appendix A.2.1/A.2.3"},{"comment":"Delta is confounded by floor and ceiling effects and by the model's ability to adapt in-context. If baseline accuracy Gamma is near zero (as in NBR for most models), the accuracy on the remapped set cannot drop much, so Delta can be small or even negative for reasons unrelated to abstraction. The theorem's condition Gamma ≥ γ is intended to mitigate this, but the paper reports and interprets Delta values for many configurations with low Gamma (e.g., NBR rows in Table 3). Conversely, when Gamma is high, a large Delta may reflect task difficulty under novel symbols or the model's use of short in-context examples to relearn the mapping, not necessarily 'memory dependence' on the original symbols. The paper should report Delta only for configurations where the floor/ceiling constraints are checked, or should normalize Delta relative to chance or to a control condition with an equally novel but structurally different task, before drawing conclusions such as 'operand-specific memorization.' As written, the empirical Delta values in Table 2 and Figure 4 cannot be uniquely attributed to memory dependence.","section":"Definition 3.6 and Section 5.1, Table 3"},{"comment":"There is an inconsistency between the formal definition of Delta and its implementation. Definition 3.6 keeps the rule string r_i in the original symbolic space and only remaps the concrete instances and answers, whereas Appendix A.5 describes an operator-specific remapping (M_operator) applied to the operator symbols, and Table 2 reports 'MemDep op' computed from such remapping. The theorem in Section 3.5 is about the definition with fixed r_i; the benchmark's operator-remapping variant is not covered by the theorem. The paper should clarify which operationalization is used for each reported number, and whether the theorem is intended to cover remapping of the rule tokens. If it is, the proof needs to be extended; if not, the claim that Delta measures memory dependence for the operator-remapping results is not theoretically backed.","section":"Definition 3.6 vs. Appendix A.5"}],"minor_comments":[{"comment":"The abstract contains a typo: 'mathematic framework' should be 'mathematical framework'.","section":"Abstract"},{"comment":"The phrase 'Llama seris' should be 'Llama series'.","section":"Section 1"},{"comment":"Theorems 3.7, 3.8, and 3.9 are presented in the main text without proofs, with proofs deferred to the appendix; since the theorems are not substantive, the paper should consider demoting them to 'propositions' or explicitly labeling which parts are assumptions.","section":"Section 3.5 / Appendix A.2"},{"comment":"The abbreviation 'BC-SR' is introduced without a clear definition; it appears to refer to the task category range from Basic Computation to Symbolic Reasoning, and should be spelled out at first use.","section":"Section 4.1"},{"comment":"The text claims the benchmark comprises 82 sub-datasets, but summing the named datasets in Appendix A.6 (4 BC + 16 EC + 1 MA + 12 NBR + 7 SMA + 43 SR) gives 83 entries, with at least one duplicate name; the count should be reconciled or the list corrected.","section":"Appendix A.6"},{"comment":"The candidate symbol pool is curated from the Llama-2 7B tokenizer vocabulary, but the same remapped strings are used for all evaluated models; the paper should note that tokenization differences across models may affect how 'novel' the remapped symbols are for each model.","section":"Appendix A.5"},{"comment":"The composite score F(Γ,Δ) defined in Theorem 3.9 is not used in any experiment or analysis; the paper should either use it or state explicitly that it is only a formal construct.","section":"Section 3.5 and Section 5"}],"recommendation":"major_revision","confidential_remarks":"The empirical benchmark and results are potentially useful as a stress test, and the human baseline and fine-tuning control are commendable. However, the theoretical claims as written are not supportable: the proof of Theorem 3.8 contains a non-sequitur at its decisive step, and the other 'theorems' are definitional. I would advise the editor that the paper is fixable in scope only if the authors substantially rewrite the theory section to state assumptions explicitly and either prove the needed bridge or drop the claim that Delta is a theoretically grounded measure of abstraction. If the authors are unwilling to weaken the theoretical framing, the paper should be rejected; if they can reframe the contribution as an empirical benchmark with clearly labeled interpretive assumptions and add the requested controls, a revised version could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical core of this paper is worth engaging with; the theoretical framing is not. The authors built a broad symbol-remapping benchmark, ran it across a wide model zoo including agents, added a human baseline and a fine-tuning control, and produced a concrete finding: performance drops more when operands are remapped than when operators are. That operand-vs-operator asymmetry is new to me and, if it replicates, could steer data augmentation and evaluation design. The benchmark itself is a real artifact, and the paper is transparent about failure modes and includes honest case studies. That part is solid work. The soft spot is the theory, and it is load-bearing. Delta is just the accuracy drop under remapping. Theorem 3.8 claims that small Delta plus high Gamma certifies rule-inductive abstraction, but the proof in Appendix A.2.2 jumps from equality of output distributions to invariance of the internal abstraction mapping f. That step is asserted, not derived. A model could achieve low Delta by memorizing the remapped symbols from the few-shot prompt, by exploiting cues the mapping preserves, or by applying a token-level nearest-neighbor rule, none of which is the abstraction the theorem claims to certify. The other theorems are also mostly restatements: 3.7 is the definition of Gamma plus the law of large numbers, and 3.9 defines a weighted sum and calls it valid. So the central theoretical justification does not hold up. The paper would be more honest if it presented Gamma and Delta as pragmatic diagnostics and dropped the theorem language, or replaced the theorems with a genuine derivation connecting Delta to a formal notion of invariance. As it stands, the theory section overclaims. There are also smaller issues: no confidence intervals on the Delta scores, and the code/data are only promised, not shipped with a hash. Who gets value from this? People building or using LLM reasoning benchmarks will find the dataset and the operand-memorization result useful, even if they ignore the theory. It deserves a serious referee: the empirical contribution is substantial enough that a good reviewer could help the authors rework the framing into something defensible. I would not desk-reject it, but I would send it back with a clear request to either fix or remove the theoretical claims.","headline":"Useful remapping benchmark with an empirical finding worth replicating, wrapped in a theory section that asserts rather than proves its central claim.","tokens_in":628,"tokens_out":1186,"would_cite":false,"duration_ms":32988,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current large language models, including 70B-scale models and agent frameworks, lack genuine abstract reasoning: they collapse on non-decimal arithmetic and depend heavily on memorized symbols when tasks are remapped.","keywords":["abstract reasoning","large language models","memory dependence","symbol remapping","rule induction","benchmark design","chain-of-thought prompting","non-decimal arithmetic"],"falsifier":"Run the paper's released generation scripts with a deliberately non-abstracting system—for example a retrieval model that answers by matching the nearest few-shot example—and check whether it attains near-zero $\\Delta$ on the remapped tasks. If such a system satisfies the paper's criterion of small $\\Delta$ with large $\\Gamma$ on the seen mapping while failing every genuinely novel rule, the claimed equivalence between low memory dependence and rule-inductive abstraction is refuted.","tokens_in":28252,"feed_emoji":"🧠","tokens_out":12217,"duration_ms":112551,"temperature":0.7,"pith_summary":"The paper tries to establish a theoretically grounded way to tell when a large language model is really applying an abstract rule and when it is leaning on memorized symbols. It defines abstraction as a mapping from concrete instances to abstract patterns, and reasoning as applying a rule to those patterns, then derives two metrics: $\\Gamma$, accuracy on the task as normally posed, and $\\Delta$, the drop in accuracy when the task's symbols are systematically remapped. The central theorem states that small $\\Delta$ together with large $\\Gamma$ certifies rule-inductive abstraction rather than memorization. Evaluated on a new benchmark across 7B-70B open models, API models, and multi-agent frameworks, the framework finds near-zero performance on non-decimal arithmetic and high $\\Delta$ nearly everywhere, so the paper concludes that current LLMs still lack robust abstract reasoning. The reason to care is that $\\Delta$ offers a concrete, repeatable diagnostic for abstraction that ordinary accuracy benchmarks cannot provide.","feed_headline":"70B LLMs score near zero on non-decimal arithmetic","feed_subtitle":"A two-metric symbol-remapping test separates true rule learning from memorized operands","key_machinery":"The load-bearing mechanism is the symbol-remapping protocol feeding the paired metrics: the Abstract Reasoning Score $\\Gamma$ is accuracy on the original task, the remapped score $\\Gamma_M$ is accuracy after a bijective symbol mapping swaps surface tokens (for instance binary '0' and '1' become 'A' and 'B'), and the Memory Dependence Score $\\Delta = \\Gamma - \\Gamma_M$ measures the gap. Because remapping preserves the abstract structure of the task while changing its tokens, a model that reasons about the rule should be invariant to the relabeling, whereas a model that memorized token associations should degrade; comparing operand remapping against operator remapping then isolates which kind of token the model depends on.","core_discovery":"The paper's central claim is that genuine abstract reasoning decomposes into two measurable components, and that the second—symbol independence—is what separates reasoning from memorization. Formally, abstraction is a mapping $f: C \\to A$ from concrete instance strings to abstract features, reasoning is a function $\\mathrm{Re}: A \\times R \\to Q$ that applies a rule to an abstract feature, and the composite $H = \\mathrm{Re} \\circ f$ is the object under evaluation. The Abstract Reasoning Score $\\Gamma$ is accuracy on the original test set, and the Memory Dependence Score $\\Delta = \\Gamma - \\Gamma_M$ is the accuracy loss when a bijective symbol mapping $M$ relabels the instances while preserving task structure. Theorem 3.8 asserts that small $\\Delta$ together with large $\\Gamma$ certifies rule-inductive abstraction—the model's abstraction mapping is invariant under remapping—and the benchmark this motivates applies operand-only, operator-only, and full remappings across 82 sub-datasets in six task categories. The empirical corollary is that current LLMs sit firmly on the memorization side of this divide, with operand-specific memory dependence exceeding operator dependence for nearly every model tested.","pith_inferences":["A direct test the paper leaves open is training against $\\Delta$ itself: using the accuracy drop under random remappings as a reward signal would show whether the metric can be gamed or whether it tracks genuine abstraction.","Because the remappings are character-level, $\\Delta$ likely underestimates dependence on subword and multi-token patterns; a remapping that operates on words or morphemes might expose even larger abstraction gaps.","The paper's fine-tuning result implies that low $\\Delta$ on a single remapping is fragile evidence: a stronger criterion would require robustness across several independent remappings of the same task, which the released generation scripts make directly testable.","Models pretrained on tokenizers or corpora with more symbol diversity might show systematically lower $\\Delta$, a prediction that would connect this benchmark to questions about how training distribution shapes abstraction."],"forward_implications":["Accuracy on familiar benchmarks is not evidence of abstraction: the same models that score well on decimal arithmetic collapse to near zero on the same operations in other number bases.","Chain-of-thought prompting raises accuracy on familiar multi-step problems but does not close the abstraction gap, and in several models it increases $\\Delta$, suggesting procedural guidance can deepen token dependence.","Multi-agent frameworks reach the highest raw scores on some tasks while showing the highest memory dependence of any group evaluated, so more interaction does not by itself produce symbol-invariant reasoning.","Operand-specific memorization dominates across every model class: remapping numbers hurts more than remapping operators, pointing to where training-data reliance is strongest.","Fine-tuning on remapped symbols improves accuracy for the same remapping structure but not for an unseen one, so exposure to mapped data teaches the specific mapping rather than a generalizable rule."],"supporting_citations":[{"why":"Supplies the ARC benchmark and the intelligence-as-abstraction viewpoint that motivates the paper's formal framework; the paper explicitly positions its text-based remapping benchmark against ARC's visual format.","marker":"(Chollet, 2019)"},{"why":"Provides GSM8K, used both as the Math Application task category and as the familiar-domain benchmark whose high scores contrast with the paper's low abstraction scores.","marker":"(Cobbe et al., 2021)"},{"why":"The GSM-Symbolic result that symbol perturbation degrades LLM math performance is the direct empirical precedent for using symbol remapping as a probe.","marker":"(Mirzadeh et al., 2024)"},{"why":"Introduces chain-of-thought prompting, the main intervention the paper tests and reports as insufficient to close the abstraction gap.","marker":"(Wei et al., 2022)"},{"why":"Counterfactual-task evidence that LLMs recite rather than reason, cited to justify why accuracy alone cannot distinguish memorization from abstraction.","marker":"(Wu et al., 2024)"},{"why":"Token-bias analysis showing LLMs are not yet genuine reasoners, which the paper leans on to motivate the memory-dependence measure.","marker":"(Jiang et al., 2024)"}],"fun_headline_variants":["LLMs flunk symbol-remapping test for abstraction","Memory dependence metric separates true rule learning from memorization","Non-decimal arithmetic near zero for 70B LLMs in new benchmark","Abstract reasoning gap: LLMs rely on symbols, not patterns","Symbol remapping exposes LLM memorization over rule abstraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole diagnostic rests on an assumption asserted rather than proved in Step 4 of the proof of Theorem 3.8 in Appendix A.2.2: a model whose accuracy barely drops under symbol remapping is presumed to be operating on a symbol-invariant abstraction, when it could equally have learned the new symbols from the few-shot examples inside the prompt.","fun_headline_variants_meta":{"raw":{"variants":["LLMs flunk symbol-remapping test for abstraction","Memory dependence metric separates true rule learning from memorization","Non-decimal arithmetic near zero for 70B LLMs in new benchmark","Abstract reasoning gap: LLMs rely on symbols, not patterns","Symbol remapping exposes LLM memorization over rule abstraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1641,"prompt_tokens":1026,"completion_tokens":615,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":530}},"tokens_in":642,"tokens_out":615,"duration_ms":6435,"temperature":1.0,"reasoning_tokens":530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:14:25.307258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's released generation scripts with a deliberately non-abstracting system—for example a retrieval model that answers by matching the nearest few-shot example—and check whether it attains near-zero $\\Delta$ on the remapped tasks. If such a system satisfies the paper's criterion of small $\\Delta$ with large $\\Gamma$ on the seen mapping while failing every genuinely novel rule, the claimed equivalence between low memory dependence and rule-inductive abstraction is refuted.","supporting_citations":[],"review_version":1}