{"id":"7e097057-6164-41f5-b94a-0c287351a23b","arxiv_id":"2509.07339","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A transformer trained to imitate A* produces traces whose length reflects similarity to training data, not true problem complexity, so long chain-of-thought should not be read as more 'thinking'.","lead":"This paper trains a small transformer from scratch to imitate the search steps of the A* maze-solving algorithm, then checks whether the length of the model's reasoning traces tracks problem difficulty. It finds that traces stay long on trivial mazes, that length aligns with closeness to training data rather than true complexity, and that this undermines reading 'thinking effort' from long chain-of-thought outputs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'loose correlation' claim is never quantified: no correlation statistics are reported, and Fig. 1a mixes valid responses with 32k-truncated failures, which can mechanically suppress correlation and makes the ID-vs-OOD comparison unverifiable.","rationale":"The reader's weakest_assumption was the transfer from a 380M supervised trace-imitation model to RL-trained frontier LRMs like R1. That is a real generalization concern, but the more immediate issue is whether the experimental result itself is established. The paper's central quantitative claim—that trace length and A* trace length are only loosely correlated and that this correlation disappears with distributional distance—is supported only by qualitative scatter plots. This is critical because including truncated 32k outputs (which are failures, not completed traces) can mechanically weaken any observed correlation, and because the free-space analysis uses only 5 valid responses. The ID-vs-OOD comparison consists of only two maze families, so 'distributional distance' is asserted rather than measured. These gaps do not necessarily invalidate the conclusion, but they make the current evidence insufficient to fully support the abstract's strong wording. The reader's CONDITIONAL verdict already captures this; I would keep it, hence UNCHANGED. My concern differs from the reader's primary one, hence partial agreement.","tokens_in":8062,"tokens_out":4722,"duration_ms":59217,"concrete_test":"Compute Spearman/Pearson correlations between generated trace length and A* trace length for each of the six evaluation sets, reporting confidence intervals and both including and excluding truncated/failed responses. If, e.g., Spearman ρ ≥ 0.5 for any OOD family, or if excluding censored points changes the Wilson/SF-style comparison direction, the 'loose correlation' and 'distributional distance' conclusions require revision. Ideally, the authors release per-instance data or a plotting script to make this reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that generated trace length and ground-truth A* length 'only loosely correlate' and that correlation is governed by distributional distance—rests entirely on visual inspection of scatter plots (Figs. 1a, 1b, 2a, 2b). No Pearson/Spearman correlation, R², or confidence interval is reported anywhere in §4. This matters because Figure 1a explicitly includes responses that hit the 32k context limit without producing a solution (points at the top). Censored observations of this kind can substantially attenuate a true correlation; moreover, only 5 of 100 free-space problems yield a valid plan, so any 'correlation' in that regime is computed over 5 points. The ID-vs-OOD contrast is also underdetermined: it compares exactly two maze families (Wilson vs SF-style) with no measure of distributional distance, so the claim that correlation 'vanishes' as distance increases is not supported by a trend. If, when quantified, Spearman correlations on Kruskal/DFS/Drunkard are moderate or the Wilson correlation is not strong, the qualitative reading of the plots changes. The missing statistics are thus load-bearing for the abstract's conclusion that complexity is 'not a significant factor.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains a ~380M-parameter decoder-only transformer from scratch on linearized A* traces for 30x30 grid-maze pathfinding, using Wilson-algorithm mazes as the training distribution. It then evaluates the model on held-out Wilson mazes, on mazes from four other generation algorithms (Kruskal, DFS, SF-Style, Drunkard), and on obstacle-free 'free-space' problems. The central claim is that the length of the model's generated intermediate tokens is only loosely correlated with the ground-truth A* trace length (a precise, verifiable measure of problem complexity), that the apparent correlation appears only for mazes close to the training distribution, and that this indicates approximate recall rather than problem-adaptive computation. The paper uses this to caution against interpreting long reasoning traces in systems such as DeepSeek R1 as evidence of 'thinking effort.'","tokens_in":8289,"tokens_out":4025,"duration_ms":49589,"significance":"If the central claim survives quantification, this is a useful controlled result. The setup has real strengths: problem complexity is external and verifiable via A* operation counts, the training procedure is fully specified and reproducible, and the 'model organism' framing is honest about the gap between small-scale trace-imitation models and large reasoning models. The paper addresses a timely and important question about whether CoT length is a reliable index of adaptive computation. However, the central claim currently rests on visual inspection of scatter plots; the design is capable of supporting a much stronger quantitative statement, and the missing statistics, censoring handling, and distributional-distance measure are the main obstacles. I would view the paper as making a credible qualitative case that needs additional quantitative rigor before the abstract's strong conclusions can be accepted.","major_comments":[{"comment":"The claim that 'there is very little correlation' is never operationalized. No Pearson/Spearman correlation, R², or confidence interval is reported for the free-space data or for any other maze family. This is load-bearing: Fig. 1a explicitly includes responses that hit the 32k context limit without producing a solution, and censored observations of this kind can substantially attenuate a true correlation. Moreover, only 5 of 100 free-space problems yield a valid plan, so any 'correlation' in that regime is computed over 5 points. Please report correlation coefficients with confidence intervals for each condition, treat truncated responses explicitly (e.g., exclude-and-analyze separately or use a censoring-aware analysis), and report the number of valid plans per condition. The y=x line also conflates correlation with calibration; a strong monotonic association need not lie on y=x if tok","section":"§4.1, Fig. 1a"},{"comment":"The conclusion that correlation 'vanishes as distributional distance increases' is based on comparing exactly two maze families (Wilson vs. Searchformer-style) and no quantitative definition of distributional distance. The paper never measures distance between maze distributions, so the abstract's attribution of the effect to 'distributional distance from the training data' is not supported by a trend. Please (i) report correlation coefficients for all five OOD maze families, (ii) define a measurable distributional distance (e.g., wall density, loop density, graph-theoretic statistics, or solution-length distribution), and (iii) test whether correlation strength decreases monotonically with that distance, or at least present a nonparametric comparison across families. Without this, the central causal claim remains an interpretation of two selected scatter plots rather than a demonstrated","section":"§4.2, Figs. 1b and 2"},{"comment":"The paper's stated target is the interpretation of long traces in systems like DeepSeek R1, but the experiments use a 380M transformer trained from scratch by supervised imitation of A* traces, with no RL, no final-answer reward, and a domain-specific tokenizer. This is a legitimate model-organism design, but the leap from 'SFT trace imitation does not produce adaptive trace lengths' to 'R1's long traces are not thinking effort' is substantial. Please either (a) add an experiment in the same maze domain with RL or final-answer-reward training and show that the same loose correlation persists, or (b) explicitly restrict the conclusions to the SFT trace-imitation setting and adjust the abstract and Section 5 accordingly. The transfer assumption should be stated as an assumption, not as a consequence of the present experiments.","section":"§1 and §5, transfer to R1"},{"comment":"The manuscript does not specify whether the scatter plots include only semantically valid generated traces or all generations, and how failures are coded. Since a model that fails to solve a problem can still emit many tokens, mixing truncated and invalid responses into the same scatter can dominate the apparent relationship. The 'excessive length on trivial problems' finding, in particular, may be driven by degenerate repetition rather than by a calibrated length-difficulty relationship. Please report per-condition success rates, state inclusion criteria for scatter points, and analyze valid and invalid traces separately. This is needed to interpret the raw scatter plots and to assess the claim that trace length is not adaptive.","section":"§4, data inclusion and validity"}],"minor_comments":[{"comment":"The running title 'PERFORMATIVETHINKING?' is missing a space; should be 'PERFORMATIVE THINKING?'.","section":"Title/header"},{"comment":"The maze-generation algorithm is referred to inconsistently as 'Searchformer-style' and 'SF-Style'. Use one name throughout.","section":"§3/§4.2"},{"comment":"The phrase 'from-scratch computational complexity' is confusing; presumably 'from-scratch' modifies the model, not the complexity. Reword to avoid ambiguity.","section":"§5"},{"comment":"The caption says 'Points at the top correspond to failures where the model generated tokens until reaching the 32k limit,' but the text does not say how many such points there are. A count or proportion would help.","section":"Fig. 1a caption"},{"comment":"The phrase 'If there were a correlation, the generated trace lengths would closely match the ground-truth values' conflates correlation with calibration. Rephrase to distinguish correlation strength from agreement with the y=x line.","section":"§4.1"},{"comment":"Training details are given, but there is no statement about code/data availability or a repository link. For reproducibility, please add this.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope for cs.AI and addresses a timely question. My main concern beyond the technical points is scope: the title and abstract explicitly invoke R1 and 'thinking effort', but no R1 or RL-trained model is evaluated. If the editor values direct evidence on large reasoning models, this gap should be made explicit in the paper. The quantitative gaps (correlations, censoring, distributional distance) are fixable within the manuscript's scope, so I do not see grounds for rejection; but the current version's strongest claims outrun the reported evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the question: do transformers trained to imitate A* traces produce trace lengths that track the ground-truth search complexity of the instance? I don't know of prior work that asks this directly, and the setup is a clean way to do it — A* trace length is an externally defined complexity measure, not a fitted parameter. The free-space probe is also a good idea: even trivial instances can trigger long or truncated traces, which already undercuts the naive \"longer trace = more thinking\" reading.\n\nWhat the paper does well is the controlled paradigm: a from-scratch 380M model, a verifiable search procedure, and a clear in-distribution vs out-of-distribution contrast. The Wilson scatter shows some alignment; the SF-style scatter does not. That visual contrast is suggestive and the qualitative direction seems right.\n\nBut the central claim — \"only loosely correlate\" — is never quantified. No Pearson or Spearman coefficient, no R², no confidence interval anywhere in Section 4. That matters because Figure 1a mixes valid responses with responses that hit the 32k context limit and never produced a solution. Censored points at the top can mechanically pull down any correlation, so the visual read may be misleading. And the free-space \"little correlation\" rests on 5 valid plans out of 100 — essentially no data. The OOD claim is also thin: the vanishing correlation is shown for one family (SF-style), not a range of distributions, so \"distributional distance\" is doing explanatory work without being measured.\n\nThe bigger soft spot is the leap to R1. A supervised trace-imitation model with no RL on final answers is a reasonable model organism for studying what imitation alone produces, but it is not a faithful stand-in for R1's post-training. The paper's caution about interpreting R1 trace lengths as thinking effort is worth saying, but it should be flagged as a hypothesis, not a demonstrated transfer. Also, the paper says the setup is open and replicable, but I don't see code or data artifacts; that is minor relative to the statistics problem, but still worth fixing.\n\nOverall: the core observation is plausible and the paper is honest in its framing, but the missing quantification is load-bearing, and the generalization to R1 needs tempering. If the authors add correlation statistics, report the censored-point analysis separately, and narrow their claims, this could be a useful reference for the CoT-interpretability and efficiency literature. I'd send it to review, but as a conditional accept with revisions requested.","headline":"A controlled small-scale negative result on CoT length vs. A* complexity, directionally plausible but under-quantified and over-extrapolated to R1.","tokens_in":8821,"tokens_out":1703,"would_cite":false,"duration_ms":23007,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a transformer trained from scratch on A* search traces, chain-of-thought length tracks training-data familiarity rather than problem difficulty, so long traces should not be read as thinking effort.","keywords":["chain-of-thought","intermediate token generation","A* trace imitation","problem complexity","distributional shift","reasoning models","pathfinding","test-time scaling"],"falsifier":"Take the same maze setup and train an identically sized transformer with reinforcement learning that rewards only the final plan's correctness, with no trace supervision, then measure the trace-length versus A*-operation-count correlation on out-of-distribution maze generators. A strong, monotone correlation across all generators would refute the claim that distributional distance, not complexity, controls trace length; a flat correlation would confirm it.","tokens_in":7927,"feed_emoji":"🗺️","tokens_out":7120,"duration_ms":76728,"temperature":0.7,"pith_summary":"The paper asks whether the number of intermediate tokens a model emits before its answer tracks how hard the problem actually is, or how familiar the problem looks. To get a verifiable answer, the authors train a 380M-parameter transformer from scratch to imitate the step-by-step traces of A* search on 30×30 maze problems, where A*'s operation count is an exact ground-truth measure of complexity. They find that on trivial no-obstacle grids the model often runs to the 32k context limit without solving anything, and across maze generators with different structure the generated trace length and A* trace length correlate only loosely. The only cases with visible correlation are held-out problems drawn from the same generator as training, which the authors read as approximate recall rather than adaptive computation. The paper's conclusion is that for such models, longer traces should not be treated as evidence of more \"thinking,\" and it extends this caution to large reasoning models whose long chains are popularly anthropomorphized.","feed_headline":"Chain-of-thought length tracks familiarity, not difficulty","feed_subtitle":"A trace-trained model's reasoning length follows its training distribution, not A*'s true workload.","key_machinery":"The key machinery is the derivational trace of A* search, linearized into tokens of the form 'create x y cG cH' and 'close x y cG cH' for each node creation and expansion. Because A*'s number of operations equals the length of this trace, the trace gives a precise, verifiable measure of problem complexity. Training a from-scratch transformer to imitate these traces creates a controlled setting in which the model's own generated trace length can be compared against the ground-truth complexity of each maze. The design also varies the maze-generation algorithm, producing training and test distributions with different structural distance, which lets the authors separate complexity effects from d","core_discovery":"The central claim is that intermediate token length in a transformer trained on derivational traces is not a reliable index of problem-adaptive computation. Using A* on grid mazes as a controlled setting, the authors define problem complexity as the number of A* operations, which is directly proportional to the ground-truth trace length. After training on 500,000 mazes from one generator, the model produces trace lengths that are only loosely correlated with A* trace lengths on held-out mazes from other generators; on obstacle-free grids it produces excessively long traces and frequently fails. Where correlation does appear, it is confined to instances close to the training distribution, lea","pith_inferences":["A natural extension is to measure distributional distance quantitatively and test whether trace-length correlation degrades smoothly as distance grows; the paper demonstrates the phenomenon but does not fit such a curve.","The controlled small-scale result suggests that large reasoning models trained with reinforcement learning on final answers might behave differently, since RL can in principle select for concise or long traces; whether the same distributional-recall pattern survives RL post-training is an open question the paper does not settle.","If trace length reflects training priors, then cross-distribution benchmarks are the right setting for evaluating test-time scaling claims; in-distribution accuracy alone could mask non-adaptive trace generation.","An interventionist test follows: perturb the maze distribution while holding A* operation counts constant; if trace lengths track the perturbation rather than the counts, the recall explanation is confirmed."],"forward_implications":["Trace length cannot be used as a proxy for problem difficulty when evaluating models on out-of-distribution inputs.","Observed \"adaptive\" reasoning lengths on in-distribution test sets may be an artifact of approximate recall, not evidence of computation that scales with difficulty.","Methods that aim to improve efficiency by shortening reasoning chains may be targeting a property that is not aligned with task complexity.","Anthropomorphic readings of intermediate tokens as \"thinking effort\" require additional evidence beyond trace length.","Evaluation of reasoning models should explicitly control distributional distance between training and test data."],"supporting_citations":[{"why":"supplies the linearized A* trace representation and the wall-percentage maze generator used for a far out-of-distribution test","marker":"[14]"},{"why":"established the method of training transformers on linearized search traces, which this work adapts","marker":"[6]"},{"why":"supplies the uniform spanning-tree maze generator that defines the training distribution","marker":"[31]"},{"why":"the large reasoning model whose long traces motivate the interpretation of chain length as thinking effort","marker":"[8]"},{"why":"supplies a union-find spanning-tree maze generator used as an out-of-distribution test distribution","marker":"[12]"},{"why":"supplies the recursive-backtracker maze generator used as another out-of-distribution distribution","marker":"[28]"},{"why":"supplies the random-walk cave generator used as a structurally distinct out-of-distribution distribution","marker":"[11]"}],"fun_headline_variants":["CoT length tracks data familiarity, not problem difficulty","Reasoning trace length is a recall artifact, not a thinking meter","Longer chain-of-thought means closer to training data, not harder tasks","In A* mazes, CoT length matches data nearness, not true workload","LLM 'thinking' length is distributional, not adaptive to complexity"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that a 380M-parameter model trained purely to imitate A* traces is a faithful stand-in for large reasoning models trained with reinforcement learning and preference alignment, so that findings about its trace lengths carry over to them.","fun_headline_variants_meta":{"raw":{"variants":["CoT length tracks data familiarity, not problem difficulty","Reasoning trace length is a recall artifact, not a thinking meter","Longer chain-of-thought means closer to training data, not harder tasks","In A* mazes, CoT length matches data nearness, not true workload","LLM 'thinking' length is distributional, not adaptive to complexity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1459,"prompt_tokens":800,"completion_tokens":659,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":564}},"tokens_in":544,"tokens_out":659,"duration_ms":6888,"temperature":1.0,"reasoning_tokens":564,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T22:21:47.807522+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same maze setup and train an identically sized transformer with reinforcement learning that rewards only the final plan's correctness, with no trace supervision, then measure the trace-length versus A*-operation-count correlation on out-of-distribution maze generators. A strong, monotone correlation across all generators would refute the claim that distributional distance, not complexity, controls trace length; a flat correlation would confirm it.","supporting_citations":[{"cited_title":"Generating random spanning trees more quickly than the cover time","cited_arxiv_id":null,"evidence_quote":"supplies the uniform spanning-tree maze generator that defines the training distribution"},{"cited_title":"Procedural dungeon generation: A drunkard’s walk in clojurescript","cited_arxiv_id":null,"evidence_quote":"supplies the random-walk cave generator used as a structurally distinct out-of-distribution distribution"}],"review_version":1}