{"id":"78e4c335-98df-4d7e-8e05-3473555b48a1","arxiv_id":"2502.03568","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM performance on naturalistic reasoning tasks tracks performance on equivalent Python code simulation, but the effect is partly driven by pattern matching and memorization rather than faithful execution.","lead":"This paper tests whether asking a large language model to 'run' short Python programs can stand in for harder natural-language reasoning tests. It finds the two performances track each other, but also that models often solve tasks by pattern-matching rather than by actually simulating the steps.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proxy claim rests on hand-authored equivalence; the paper never validates per-instance correspondence, so the observed trend correlation may reflect shared task difficulty rather than a transferable proxy.","rationale":"I agree with the reader's identification of the weakest assumption: the paired naturalistic and synthetic prompts are assumed equivalent in algorithmic content and difficulty, and most pairs were authored in-house. My stress-test sharpens this into a concrete evidential gap. The paper reports aggregate accuracy trends across a control variable, but a proxy claim requires that performance on the synthetic prompt predicts performance on its naturalistic counterpart at the level of individual instances. Aggregate similarity can arise simply because both tasks scale with the number of operations, which is a property of task complexity, not of the code-simulation mechanism. The paper's own external pair and its sorting results show that the two sides can diverge substantially for some models, which undermines the sufficient-condition framing. The proposed per-instance contingency test would settle the question using data the paper already claims to release. Because the reader's verdict is already CONDITIONAL and this concern reinforces rather than overturns that assessment, I recommend no verdict change: the paper should be accepted conditionally on releasing artifacts and demonstrating per-instance correspondence, not just trend-level similarity.","tokens_in":19681,"tokens_out":3524,"duration_ms":34492,"concrete_test":"Run a per-instance contingency analysis on the released logs: for each model and each of the five paired benchmarks, compute P(naturalistic correct | synthetic correct) and P(synthetic correct | naturalistic correct), plus the per-instance correlation of correctness across the two prompt forms. To control for construction bias, do the same on a held-out external naturalistic benchmark (e.g., Kim & Schuster object tracking or a new domain) with code equivalents generated mechanically from the naturalistic annotations rather than hand-authored. If the conditional probabilities are not substantially above the base rates, the paired benchmarks only show comparable average difficulty, not that code simulation transfers to naturalistic reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that code simulation is a proxy for naturalistic reasoning because 'correctly simulating each step is a sufficient condition to solve them correctly' (Abstract, Section 1). The evidential core is the paired benchmarks of Section 3.1, with correlations visually inferred from Figures 3 and 4. The load-bearing assumption, stated in Section 3.1, is that each synthetic/naturalistic pair encodes the same computation. Five of the six pairs were authored in-house for this paper; only object tracking uses an external source (Kim & Schuster, 2023). If the two prompts in a pair share surface features deliberately inserted by the same authors—same number of steps, same variable-to-agent mapping, same arithmetic operations—then a model that is sensitive to any of these features will score similarly on both sides even if it never engages the underlying algorithm. The comparison therefore does not distinguish 'code is a proxy for reasoning' from 'our templates are easy for LLMs in correlated ways.' The paper's own results reinforce the worry: the only external pair shows a large synthetic/naturalistic gap for Llama-3.1-405B (Figure 8), and in sorting, GPT-4 and GPT-4o find the synthetic task easier for more challenging instances while the naturalistic task behaves differently (Section 4.1). These divergences are inconsistent with a sufficient-condition reading of the proxy claim. What is missing is a per-instance check: do the same models make correlated errors on the paired prompts, or do similar aggregate accuracies arise from independent errors? Without that, the evidence supports only 'these task families are similarly difficult on average', which is a much weaker claim than a proxy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that code simulation can serve as a scalable proxy for evaluating LLM reasoning on naturalistic tasks. It introduces five paired benchmark families (straight-line/good exchange, critical path/critical good exchange, parallel path/clique exchange, nested loops/recurring calculation, sorting/ranking objects) plus a pairing of an external entity-tracking benchmark with code, and evaluates GPT-4, GPT-4o, and Llama-3.1-405B with Chain of Thought. Additional experiments on pure code simulation across open and closed models identify memorisation and 'lazy execution' as failure modes and propose Chain of Simulation (CoSm) prompting. The central claim is that performance on synthetic code correlates with performance on naturalistic tasks, so synthetic data can complement handcrafted benchmarks.","tokens_in":19957,"tokens_out":6119,"duration_ms":50900,"significance":"If substantiated, the proxy claim is practically valuable: synthetic code generation is cheaper and more scalable than human-authored naturalistic prompts, and the paper's detailed public code and appendices support reproducibility. The paper also makes a useful empirical observation that LLM code execution is fragile and affected by memorisation, with a proposed mitigation (CoSm). However, the central evidence is currently qualitative and correlational: no statistical tests, no per-instance error correspondence, and several divergences that complicate the 'sufficient condition' framing. The contribution is therefore promising but not yet rigorous enough to support the strongest claims.","major_comments":[{"comment":"The central proxy claim rests on the assumption that each synthetic/naturalistic pair encodes the same computation and is of comparable difficulty, but this equivalence is never validated per instance. Five of the six paired tasks were authored in-house for this paper; only the object-tracking pair uses an external source (Kim & Schuster, 2023). The reported evidence consists of aggregate accuracy trends against a control variable, so the observed correlation could be produced by shared surface features (e.g., number of steps, agent-variable mapping, arithmetic operations) rather than by the model executing the same algorithm in both modalities. The paper should report a per-instance measure of correspondence, such as the conditional probability of correct synthetic execution given correct naturalistic performance (or a confusion matrix over instances), and should include at least one held-out or externally sourced pair beyond object tracking.","section":"Section 3.1, Figures 3-4 and 8"},{"comment":"Several reported results conflict with the 'sufficient condition' reading of the proxy claim. In the sorting tasks, GPT-4 and GPT-4o find the synthetic condition easier for longer inputs while their naturalistic ranking performance behaves differently (Section 4.1, Figure 4). In the only externally sourced pair, Llama-3.1-405B shows a large synthetic advantage on object tracking (Appendix A, Figure 8). Moreover, the paper itself presents evidence that the object-tracking dataset has been memorised verbatim by GPT models (Appendix A, Figure 9), so this external benchmark cannot serve as a clean validation of the proxy. The authors should either explain these divergences within the proposed framework or explicitly moderate the claim to a task- and model-specific correlation, rather than a general sufficient condition.","section":"Section 4.1 and Appendix A, Figures 4 and 8"},{"comment":"The paper repeatedly describes the relationship between synthetic and naturalistic performance as 'strongly correlated,' but no correlation coefficient, significance test, or confidence interval is reported; the claim is based on visual inspection of trends over a small number of control points. Given that some trends are non-monotonic (e.g., sorting for GPT-4/GPT-4o), the paper should quantify the association (e.g., Spearman's rho with uncertainty) and, ideally, report per-instance agreement between the two conditions. At minimum, standard errors or error bars for the three runs should be shown in Figures 3 and 4.","section":"Section 4.1, Figures 3-4"},{"comment":"The manuscript explicitly states that logical-{and,or} instructions are excluded after observing that performance 'considerably drops' for them. This is a post hoc exclusion of an experimental condition and should be presented as a limitation in the main text, with the excluded results shown or linked so readers can assess whether the scope of the proxy claim is narrower than suggested. The abstract speaks of 'common constructs in programming,' which could include logical operations; the paper should clarify that the current evidence covers add/sub/mov and not logical operations.","section":"Appendix B.2"}],"minor_comments":[{"comment":"The formula for Levenshtein-based prediction error, 1/N Σ|ψ(y_i|x,p_i) ∩ Ω(p_i)|, is not standard notation; please define the distance explicitly (e.g., Levenshtein distance divided by length) and avoid the set-intersection notation.","section":"Section 3, metrics"},{"comment":"The sentence 'We study GPT-3.5-Turbo, GPT-4, and Llama-3-70B and run 3 independent runs of 30 experiments each are shown in Figure 5' is grammatically incomplete; please revise.","section":"Section 4.2"},{"comment":"The claim that the Straight line result 'proves' the synthetic task a faithful proxy is too strong for correlational evidence; suggest 'is consistent with' or 'suggests.'","section":"Section 4.1"},{"comment":"The text refers to 'Figure 2 (left)' for both the Straight line/Good exchange and the Critical path tasks; please clarify the panel labels so readers can locate the referenced examples.","section":"Figure 2"},{"comment":"The sentence 'which can be found in the code material and in particular in which can be found in the code material' contains a duplicated phrase; please correct.","section":"Appendix A.2"},{"comment":"The full naturalistic prompts are not included in the appendices; to support the equivalence assumption, the exact prompts (or a representative set) should be made available in the repository or an appendix.","section":"Benchmark prompts"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical benchmark contribution; the authors are transparent about data and code. The main risk to acceptance is the gap between the strength of the proxy claim and the correlational evidence. I would be willing to reconsider after the authors add per-instance validation and statistical measures. The paper explicitly positions itself as an extension of La Malfa et al. (2024), so there is no novelty disclosure problem. However, the post hoc exclusion in Appendix B.2 and the memorisation of the external object-tracking benchmark should be addressed directly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Rough take: if you build or use LLM reasoning benchmarks, this is worth knowing. The idea that naturalistic tasks can be evaluated through equivalent code simulation is not new — the authors explicitly extend their own 2024 work — but the paper adds a paired five-family benchmark suite, a CoSm prompt that forces explicit tracing, and a careful catalogue of failure modes: lazy execution on long sorting inputs, memorisation of standard algorithm templates, and a repeated-element bug in Bubble Sort. Those observations are real and honestly reported.\n\nThe central proxy claim is plausible but not as strong as the abstract bills it. The five paired families show correlated trends in Figures 3 and 4, but the analysis is qualitative: no correlation coefficients, no significance tests, and only 3 runs of 30 items. The stress-test worry deserves weight: five of the six pairs were authored in-house, so the correlation may partly reflect shared surface features rather than a transferable proxy property. The one external pair (object tracking) largely supports the proxy, but Llama-3.1-405B shows a large synthetic/naturalistic gap there, and the sorting results for GPT-4/GPT-4o are non-monotonic. Without a per-instance check of whether errors on the synthetic prompt predict errors on the naturalistic twin, the evidence supports 'similar average difficulty' more than 'code simulation is a proxy for reasoning.' That weaker claim is still useful.\n\nMinor issues: the logical-instruction condition is dropped post hoc in Appendix B.2 (documented, but still a drop), and the promised code/data are only reachable via unclickable hyperlinks in the PDF. For a benchmarking paper, releasing artifacts is not optional.\n\nOverall: a solid, incremental evaluation contribution with an honest limitations section. It deserves peer review, and I would expect it to be accepted after quantitative correlation analysis and released code. I'd bring it to reading group if you care about benchmark design.","headline":"A useful, honest evaluation paper whose core proxy claim is plausible but not yet proven; the paired-task evidence is correlational and lacks per-instance checks.","tokens_in":20520,"tokens_out":2909,"would_cite":true,"duration_ms":26018,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that correctly simulating equivalent code is a scalable proxy for assessing core LLM reasoning, and it demonstrates this on five paired benchmarks.","keywords":["code simulation","LLM reasoning evaluation","synthetic benchmarks","paired naturalistic tasks","memorisation","chain of simulation","sorting algorithms","entity tracking"],"falsifier":"Take an independently authored naturalistic reasoning task with a code version encoding the same algorithm, vary difficulty, and check whether model accuracy on the two forms tracks each other for held-out instances; if code accuracy stays high while naturalistic accuracy collapses (or vice versa), the proxy claim fails. A second check: feed Bubble Sort inputs with controlled duplicate frequencies—if accuracy no longer drops when duplicates are repeated, the lazy-execution explanation is wrong.","tokens_in":19510,"feed_emoji":"🧮","tokens_out":5579,"duration_ms":49232,"temperature":0.7,"pith_summary":"Many reasoning, planning, and problem-solving tasks are algorithmic: if a model can correctly simulate each step, it can solve the task. This paper argues that code simulation is therefore a scalable proxy for assessing core LLM reasoning, and it builds five pairs of naturalistic and synthetic prompts that encode the same computation. Across GPT-4, GPT-4o, and Llama-3.1-405B, accuracy on the synthetic code tracks accuracy on the naturalistic task as difficulty grows, suggesting that cheaply generated Python programs can stand in for handcrafted reasoning benchmarks. The experiments also show that simulator performance is fragile, dropping sharply when slight algorithmic variations block memorisation, and that a minimal step-by-step 'Chain of Simulation' prompt partially restores it.","feed_headline":"Synthetic code can stand in for LLM reasoning tests at scale","feed_subtitle":"Paired naturalistic and coding tasks track model accuracy closely, exposing memorisation and pattern shortcuts.","key_machinery":"The central object is the paired prompt: each naturalistic reasoning task is translated into an equivalent Python program whose ground-truth output is fixed by an interpreter, so the same question is asked in two surface forms. The pairs are built from programming constructs chosen to mirror cognitive-load building blocks: straight-line programs for sequential tracking, critical paths for ignoring irrelevant instructions, parallel independent paths for tracking multiple variables, nested loops for recurring calculations, and sorting algorithms for ranking. Difficulty is controlled by a single variable—number of instructions or exchanges, nesting depth, or vector length—so accuracy can be plotted against algorithmic load. The other load-bearing component is the Chain of Simulation (CoSm) prompt, which instructs the model to simulate the program instruction by instruction and report the trace, and is used to distinguish genuine execution from memorised or pattern-matched answers.","core_discovery":"The paper's central claim is that 'code simulation is a scalable proxy for assessing some core reasoning capabilities of LLMs,' grounded in the observation that correctly simulating each step is sufficient to solve many high-order tasks. To support this, the authors pair five naturalistic settings—good exchange, critical good exchange, clique exchange, recurring calculation, and ranking objects—with synthetic Python programs that encode the same operations: straight-line code, code with critical paths, parallel paths, nested loops, and sorting routines. They report strong performance correlations between each naturalistic task and its code counterpart as the number of operations, nesting depth, or objects grows, with the naturalistic version usually noisier and sometimes harder. They further document a 'lazy execution regime' in which models such as GPT-3.5-Turbo stop simulating long sorting inputs and instead output the expected sorted sequence, and they show that memorisation, rather than execution, drives accuracy on familiar routines like Fibonacci and ascending Bubble Sort: slight anonymised variations collapse accuracy. A simple prompting extension, Chain of Simulation (CoSm), which explicitly asks the model to report the trace at each iteration, partially offsets these failures.","pith_inferences":["Not stated in the paper but implied: the proxy will only hold for tasks whose reasoning is fully captured by an algorithm; tasks requiring world knowledge, pragmatics, or ambiguity will not have faithful code equivalents.","Because five of the six paired tasks were authored in-house, an independent third-party set of naturalistic–code pairs is needed to rule out that the correlation is an artifact of task construction.","The repeated-element sorting failure suggests a concrete testable intervention: decoding with duplicate-aware constraints or presence-penalty tuning should recover the dropped items if the explanation is probabilistic.","If the proxy generalizes, the same correlation should appear across programming languages or pseudocode formulations; if it is Python-specific, the mechanism is closer to memorised syntax than to general simulation ability."],"forward_implications":["Synthetic Python programs can be generated at scale to probe LLM reasoning, reducing the cost of handcrafted naturalistic benchmarks.","Performance on a code simulation task can be used to predict performance on the paired naturalistic task, making code a cheap screening tool for model capability.","Benchmark designers should anonymise and vary familiar algorithms, because accuracy on well-known routines can reflect memorisation rather than execution.","Adding an explicit trace instruction (CoSm) is a lightweight, model-agnostic way to reduce lazy execution and make code-simulation scores more faithful.","The failure modes identified—dropping repeated elements and pattern-matching known routines—point to concrete decoding and prompting interventions worth testing."],"supporting_citations":[{"why":"Supplies the external object-tracking naturalistic benchmark that the authors pair with an equivalent coding task, providing the one paired test not authored in-house.","marker":"Kim & Schuster, 2023"},{"why":"Defines Chain of Thought prompting, the standard baseline used for every model in the paired and synthetic evaluations.","marker":"Wei et al., 2022b"},{"why":"Provides cognitive load theory, which motivates the critical-path and parallel-path task designs as extraneous and intrinsic load.","marker":"Sweller & Chandler, 1991"},{"why":"Earlier code simulation challenges that this work extends into a proxy for naturalistic reasoning tasks.","marker":"La Malfa et al., 2024"},{"why":"The memorisation-detection method the authors test and show fails on code, supporting the fragility claim.","marker":"Shi et al., 2023"}],"fun_headline_variants":["Code simulation scales LLM reasoning tests, but exposes fragility","Synthetic code tasks reveal LLM reasoning shortcuts and memorisation","LLM 'lazy execution' exposed by code simulation proxy","Code simulation as a cost-effective proxy for LLM reasoning","Scaling reasoning benchmarks: code simulation vs handcrafted tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Each paired naturalistic and synthetic prompt is assumed to be truly equivalent in the computation it requires and in difficulty, so that observed correlations reflect the proxy property rather than the way the pairs were constructed.","fun_headline_variants_meta":{"raw":{"variants":["Code simulation scales LLM reasoning tests, but exposes fragility","Synthetic code tasks reveal LLM reasoning shortcuts and memorisation","LLM 'lazy execution' exposed by code simulation proxy","Code simulation as a cost-effective proxy for LLM reasoning","Scaling reasoning benchmarks: code simulation vs handcrafted tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1427,"prompt_tokens":954,"completion_tokens":473,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":389}},"tokens_in":570,"tokens_out":473,"duration_ms":4689,"temperature":1.0,"reasoning_tokens":389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T04:29:19.130855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an independently authored naturalistic reasoning task with a code version encoding the same algorithm, vary difficulty, and check whether model accuracy on the two forms tracks each other for held-out instances; if code accuracy stays high while naturalistic accuracy collapses (or vice versa), the proxy claim fails. A second check: feed Bubble Sort inputs with controlled duplicate frequencies—if accuracy no longer drops when duplicates are repeated, the lazy-execution explanation is wrong.","supporting_citations":[{"cited_title":"and Chandler, P","cited_arxiv_id":null,"evidence_quote":"Provides cognitive load theory, which motivates the critical-path and parallel-path task designs as extraneous and intrinsic load."}],"review_version":1}