{"id":"47104f8c-1cb8-425b-b677-f607ee7c3d78","arxiv_id":"2412.04537","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"Hidden reasoning tokens in a transformer trained on filler chain-of-thought can reportedly be recovered as rank-2 predictions during decoding.","lead":"This paper claims that a small transformer trained to output filler tokens ('...') on a 3SUM task still encodes the hidden reasoning steps, and that those steps can be recovered by picking the second-most likely token at each decoding step. The result is interesting for interpretability, but the paper provides no quantitative evidence that the recovered tokens are correct or that task performance is preserved.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on unquantified assertions: no recovery rate, no task accuracy, and no evidence that ground-truth CoT tokens actually sit at rank 2 during filler decoding.","rationale":"The reader's weakest_assumption focuses on the logit lens projection of intermediate layers. I think the more load-bearing defect is the absence of any quantitative evaluation of the recovery algorithm itself. The logit lens is used in §6.1 to tell a story about layer-wise evolution, but the central claim of recovering hidden characters rests on the token rankings from the final layer (§5.6, §5.7). Even if the logit lens projection were perfect, the paper would still not demonstrate recovery: it never reports the rate at which the ground-truth CoT token occupies rank 2, nor the task accuracy of the modified decoder. The only evidence cited is 'often' and two missing figures. The invalid corruption rate of 4/3 and the missing code link further prevent verification. Thus the concern is not an internal contradiction but a failure of empirical support. The underlying idea is cheaply testable, so a revision with the missing numbers could change the verdict; as submitted, REJECT is appropriate. I partially agree with the reader: the logit lens issue is a plausible secondary concern, but it is not the necessary condition for the central claim. The necessary condition is that the ground-truth CoT tokens are, in fact, the top non-filler candidates at filler positions and that substituting them does not hurt final accuracy. That condition is simply unmeasured.","tokens_in":6173,"tokens_out":6082,"duration_ms":53329,"concrete_test":"Reimplement or run the released code; on the 2,000 test instances, (1) compute the proportion of filler positions where the ground-truth next CoT token is the top-ranked non-filler token (rank 2 when filler is rank 1), with a 95% CI; (2) compute final-answer accuracy for standard greedy decoding, the modified decoding, and random token replacement; (3) test whether modified decoding accuracy is significantly lower than greedy (e.g., McNemar's test). If the rank-2 recovery rate is below 90% or modified accuracy is significantly worse, the paper's central claim is not supported.","verdict_should_be":"REJECT","load_bearing_attack":"Section 6.2 states that 'the second-ranked token often corresponded to the numerical tokens representing the hidden reasoning steps,' but 'often' is never quantified. The modified decoder of §5.7 replaces a filler token with the highest-ranked non-filler token; for this to recover the original CoT, the ground-truth token must be that rank-2 choice at high frequency. No such frequency is reported. Section 6.3 claims the method 'significantly outperformed' random replacement, yet gives no recovery numbers, no task accuracies, and no standard errors; the figures (Fig. 1 and 2) are missing from the submission. Crucially, the phrase 'without loss of performance' requires that the autoregressive replacement of filler tokens with recovered tokens does not degrade final-answer accuracy. The model was trained with filler prefixes; conditioning the next step on non-filler tokens is a distribution shift that may reduce accuracy. The paper never measures this. Additionally, the dataset description contains an invalid 'corruption rate' of 4/3 (§5.1.1, §5.3.1), which is not a probability and suggests the data-generation procedure is not precisely defined. The GitHub link in §8 is a placeholder, so the experiments cannot be independently checked. These missing quantities are load-bearing: if rank-2 recovery is rare, or if modified decoding lowers task accuracy, the central claim fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies a 4-layer transformer trained from scratch on synthetic 3SUM-style instances in which the chain of thought is replaced by filler tokens. It applies the logit lens and token-ranking analysis to argue that the hidden non-filler reasoning tokens can be recovered by replacing filler tokens with the highest-ranked non-filler token during greedy decoding, 'without loss of performance.' The manuscript presents qualitative descriptions and placeholder figures, but contains no quantitative results such as recovery rates, task accuracies, or error bars.","tokens_in":6422,"tokens_out":5107,"duration_ms":48230,"significance":"If established, the result would be a useful interpretability contribution: it would show that a small transformer trained on filler CoT retains the underlying reasoning tokens in lower-ranked logits, and that a simple decoding modification can expose them. The paper also connects to prior work on hidden computation and unfaithful CoT, which is an appropriate framing. However, the value is conditional on the missing empirical support; as submitted, the significance cannot be assessed because the central claims are asserted rather than demonstrated.","major_comments":[{"comment":"The central claim that the modified decoder 'significantly outperformed' random replacement and recovers hidden characters 'without loss of performance' is never quantified. No recovery accuracy, no final-task accuracy, and no standard errors are reported, and the figures referenced (Figures 1 and 2) are absent from the submission. These quantities are the evidentiary core of the paper.","section":"Section 6.3, Figures 1 and 2"},{"comment":"The corruption rate is set to 4/3, which is not a valid probability (a rate should lie in [0,1] or be expressed as a percentage). This makes the data-generation procedure ill-defined and impossible to reproduce as written, and it undermines all dataset-dependent results.","section":"Section 5.1.1 and Section 5.3.1"},{"comment":"The token-ranking analysis reports that 'the second-ranked token often corresponded to the numerical tokens representing the hidden reasoning steps,' but 'often' is never quantified. The paper does not report the frequency with which the ground-truth token appears at rank 2 (or any rank) under standard greedy decoding, so the step from ranking observations to a recovery method is not validated.","section":"Section 6.2"},{"comment":"The modified decoding algorithm changes the conditioning distribution at inference time by replacing filler tokens with non-filler tokens, whereas the model was trained on filler-token sequences. The paper never reports final-answer accuracy under this modified decoding distribution, so the central claim of 'without loss of performance' is not demonstrated.","section":"Section 5.7 and Section 6.3"},{"comment":"The logit-lens projection z_l = h_l W_out is applied to a small, randomly initialized, from-scratch trained model, but the paper gives no validation that the output embedding matrix supports meaningful vocabulary-space inspection for this model. Without such validation or a suitable control, the observed rank-2 patterns could be projection artifacts rather than recovered reasoning tokens.","section":"Section 5.5"}],"minor_comments":[{"comment":"The headings 'T rue Instances' and 'T riple Listing' contain stray spaces; these should be corrected.","section":"Section 5.1.1"},{"comment":"The code availability statement says the code is 'available on GitHub here' but no URL is provided, so the experiments cannot be independently checked.","section":"Section 8"},{"comment":"The figures are placeholders with no plots or data; they should either be included as real figures or removed if the results are not yet available.","section":"Figures 1 and 2"},{"comment":"The conclusion states that the paper 'demonstrated' the recovery claim, but the body contains only qualitative observations; the wording should be tempered to match the evidence actually presented.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads as an incomplete draft: key figures are missing, quantitative results are absent, a dataset parameter is invalid, and the code link is a placeholder. The underlying question is interesting and the proposed method is plausible, but the evidentiary standard is not met. I would recommend that the authors be invited to resubmit only after they provide the full experimental protocol, recovery rates, task accuracies with error bars, corrected data generation, and accessible code."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the paper's observation — that in a filler-CoT transformer the original reasoning tokens appear at rank 2 under the logit lens — is a reasonable wrinkle on the hidden-computation literature. But the manuscript doesn't provide a single quantitative result to back up its central claim. As written, it's a method sketch, not a demonstration.\n\nWhat's new: the combination of Pfau et al.'s filler-CoT setup with logit-lens token ranking, plus the proposal to replace filler tokens with the highest-ranked non-filler token during decoding. If it works, that's a cheap interpretability probe. The experiment design is sensible: a small transformer, a synthetic 3SUM variant, and a layer-wise analysis. The paper also cites the relevant prior work on unfaithful CoT and the logit lens.\n\nThe soft spots are load-bearing. The abstract and conclusion state that hidden characters 'can be recovered without loss of performance,' but no recovery accuracy, no task accuracy, no error bars, and no baseline numbers appear anywhere. The figures that would show these results are missing from the submission. Section 6.2 says the rank-2 token 'often' matches the hidden reasoning step, but 'often' is never quantified. Section 6.3 claims significant improvement over random replacement without giving either method's numeric performance. The dataset description lists a corruption rate of 4/3, which is not a probability; the code link is a placeholder. These are not cosmetic problems: the entire claim depends on rank-2 recovery being frequent and on the modified decoding not degrading task accuracy, and neither is measured.\n\nThere's also an unaddressed distribution shift: the model was trained with filler prefixes, so conditioning on non-filler tokens during decoding is off-distribution and could hurt accuracy. The paper never tests this.\n\nThe underlying idea is testable quickly, and I'm not dismissing it. But this manuscript doesn't demonstrate anything. A reader would have to redo the experiments from scratch to learn whether the claim holds.\n\nWho this is for: someone hunting for a lead on hidden computation might read it for the hypothesis, but they'd get no empirical payoff. It doesn't deserve a serious referee in its current form because there's nothing concrete to evaluate. If the author adds actual measurements — recovery rate, task accuracy, comparison numbers, corrected data-generation parameters — a follow-up could be worth a real look.\n\nRecommendation: desk reject, but tell the author to post a revised version with the numbers. The question is worth answering.","headline":"The rank-2 recovery idea is plausible and cheap to test, but the paper never reports the measurements that would support it — it reads as an abstract rather than a paper.","tokens_in":6953,"tokens_out":3992,"would_cite":false,"duration_ms":99873,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer trained to output filler tokens still stores the true reasoning tokens in lower-ranked logits, and a modified greedy decoder can surface them without losing task accuracy.","keywords":["chain-of-thought reasoning","hidden computation","logit lens","token ranking","greedy decoding","3SUM task","transformer interpretability","filler tokens"],"falsifier":"Take the same trained model and a held-out set with known ground-truth reasoning chains, then measure how often the highest-ranked non-filler token equals the true next reasoning token. If that token-level recovery accuracy is near chance while task accuracy stays high, the central claim fails. A second, sharper test is to retrain with a random or nonlinear output embedding; if the rank-2 pattern disappears while accuracy is unchanged, the pattern was an artifact of the logit-lens projection.","tokens_in":5941,"feed_emoji":"🧠","tokens_out":4830,"duration_ms":45726,"temperature":0.7,"pith_summary":"This paper asks what a transformer is really computing when it is trained to output filler characters instead of a visible chain of thought. Using the logit lens, the author shows that in a small transformer trained on a synthetic 3SUM-style task, early layers predict the numerical reasoning tokens and the final layer overwrites them with filler tokens. The original reasoning tokens survive as the second-ranked choices at decoding time. A modified greedy decoder that skips the filler token and outputs the highest-ranked non-filler token recovers the hidden reasoning steps while keeping task accuracy intact. The point is that visible chain-of-thought is not a reliable guide to what the model has computed, and hidden computations can be surfaced without retraining.","feed_headline":"Hidden chain-of-thought can be recovered from rank-2 tokens","feed_subtitle":"Even when a model prints dots instead of reasoning, the real computation stays in its second-ranked token choices.","key_machinery":"The load-bearing tool is the logit lens, which projects each layer's hidden state $h_l$ onto vocabulary logits via the unembedding matrix $W_{\\text{out}}$, turning internal activations into a probability distribution over tokens at every layer. Alongside it, token-ranking analysis inspects the top-$k$ candidates at each decoding step. In this model the filler token wins at the last layer while the true reasoning token sits at rank 2, and the modified decoding algorithm exploits that ordering by selecting the highest-ranked non-filler token whenever the filler token is on top.","core_discovery":"On its own terms, the paper's central claim is that hidden reasoning steps in a transformer trained with filler chain-of-thought are not lost: they are computed in early layers and then masked by filler tokens in the final layer. Evidence comes from layer-wise projections $z_l = h_l W_{\\text{out}}$ and from token rankings during autoregressive decoding, where the true chain-of-thought tokens appear as rank-2 candidates beneath the filler token. Replacing each filler token with the highest-ranked non-filler token during greedy decoding recovers the original reasoning sequence and preserves performance on the 3SUM task. The paper concludes that the hidden characters can be recovered without loss of performance.","pith_inferences":["If the same rank-skipping pattern holds in larger models, auditing systems that emit placeholder or redacted intermediate text could surface the genuine computation underneath, changing what such outputs can be taken to conceal.","A direct extension would be to test the modified decoder on natural-language reasoning tasks, where reasoning tokens are not a fixed synthetic vocabulary; the paper's small-model setting does not guarantee the pattern transfers.","The paper hints at induction heads as the overwriting mechanism; one could test that by ablating attention heads and checking whether the rank-2 recovery disappears."],"forward_implications":["If the claim holds, model outputs that consist of filler or placeholder tokens can still be audited for the reasoning that produced them, without retraining or changing the model.","Task accuracy on filler chain-of-thought training is not evidence that no internal reasoning occurred; the reasoning can be present and merely hidden in lower-ranked logits.","Rank-ordered token inspection becomes a practical, cheap method for extracting hidden intermediate steps from small transformers on algorithmic tasks.","The recovery-by-rank-2 observation gives a concrete target for studying when and how transformers overwrite computed information during output formatting."],"supporting_citations":[{"why":"Establishes the phenomenon being explained: transformers can solve reasoning tasks when chain-of-thought is replaced by filler tokens.","marker":"[2]"},{"why":"Supplies the logit-lens projection method used for layer-wise analysis.","marker":"[3]"},{"why":"Provides the base transformer architecture that the trained model is built on.","marker":"[5]"},{"why":"Offers the induction-head mechanism the paper invokes for the overwriting behavior.","marker":"[4]"},{"why":"Defines chain-of-thought prompting, the setting the paper modifies.","marker":"[1]"}],"fun_headline_variants":["Rank-2 tokens reveal hidden chain-of-thought","Filler dots don't hide reasoning: it's in rank-2 tokens","Hidden reasoning decoded from second-choice tokens","CoT recovery: rank-2 tokens carry the real computation","Why dots work: true reasoning hides in rank-2 tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole analysis rests on believing that projecting a layer's hidden state through the output embedding matrix shows what token that layer is 'thinking about'; if that projection is not a faithful view, the recovered rank-2 tokens could be artifacts of the projection rather than the original reasoning steps.","fun_headline_variants_meta":{"raw":{"variants":["Rank-2 tokens reveal hidden chain-of-thought","Filler dots don't hide reasoning: it's in rank-2 tokens","Hidden reasoning decoded from second-choice tokens","CoT recovery: rank-2 tokens carry the real computation","Why dots work: true reasoning hides in rank-2 tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1121,"prompt_tokens":790,"completion_tokens":331,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":249}},"tokens_in":406,"tokens_out":331,"duration_ms":3488,"temperature":1.0,"reasoning_tokens":249,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:23:54.673991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same trained model and a held-out set with known ground-truth reasoning chains, then measure how often the highest-ranked non-filler token equals the true next reasoning token. If that token-level recovery accuracy is near chance while task accuracy stays high, the central claim fails. A second, sharper test is to retrain with a random or nonlinear output embedding; if the rank-2 pattern disappears while accuracy is unchanged, the pattern was an artifact of the logit-lens projection.","supporting_citations":[{"cited_title":"Interpreting GPT: the logit lens","cited_arxiv_id":null,"evidence_quote":"Supplies the logit-lens projection method used for layer-wise analysis."},{"cited_title":"A mathematical framework for transformer circuits","cited_arxiv_id":null,"evidence_quote":"Offers the induction-head mechanism the paper invokes for the overwriting behavior."}],"review_version":1}