{"id":"9460dfe9-cdbe-49ac-a401-4cb0e55702be","arxiv_id":"2507.03327","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Masking loss on early training tokens and injecting embeddings from an auxiliary reader improves LLM accuracy on several reasoning benchmarks.","lead":"This paper tests two training tweaks that give large language models a quiet reading phase before they answer: masking the loss on early tokens, and adding an auxiliary model that feeds context embeddings into the main model. On several reasoning benchmarks, both tweaks raise accuracy by several points, but the results come without error bars or release details.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central empirical claim is unprotected against benchmark contamination: READQ's loss masking may amplify memorization of answer-bearing tokens from eval sets that are present in the continued-pretraining corpus, and no decontamination is reported.","rationale":"The reader's weakest assumption is exactly the absence of decontamination, and I agree it is the most load-bearing condition. The central claim is empirical, so its validity depends on the evaluation being uncontaminated. The concern is sharpened by the mechanism of READQ itself: loss masking on the first K tokens redistributes the training gradient toward later tokens; in a leaked benchmark item, the answer is typically among those later tokens. This means the same contaminated corpus could inflate READQ more than the baseline, so 'same-data baseline' does not fully control for leakage. The paper also omits K, error bars, and code, which compounds the issue, but contamination is the primary threat. The proposed concrete test is feasible and would settle the question. Because the reader already conditioned acceptance on this issue and I find no independent reason to move the verdict, I recommend keeping the conditional verdict unchanged.","tokens_in":5532,"tokens_out":4802,"duration_ms":59690,"concrete_test":"Compute 13-gram (and exact-match) overlap between each evaluation set and the exact training corpora listed in Appendix A.1, then retrain the READQ and baseline models on the decontaminated subset (all overlapping sequences removed) and re-run the same benchmarks. If the READQ-minus-baseline deltas collapse toward zero on the clean subset, contamination is the explanation; if the multi-point gains persist on fully non-overlapping data, the concern is resolved. A complementary check is to evaluate on a newly created, never-published benchmark in the same formats.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (abstract; §4) is that READQ and READQBUDDY deliver multi-point accuracy gains on ARC-C, HellaSwag, OpenBookQA, PubMedQA, Winogrande, and MedQA. For this claim to hold, the gains must come from the method rather than from memorization of test items. Appendix A.1 lists the continued-pretraining corpus as Fineweb, Pile, Cosmopedia, UltraBooks, and DeepMind Math; these are public, web-scale collections, and the evaluated benchmarks are standard public datasets that commonly appear in such corpora. The paper never describes a decontamination step. The baseline is trained on the same data, but this does not neutralize the risk: READQ masks the loss on the first K tokens (§3.1), which shifts the training signal toward later, context-rich tokens. For a leaked benchmark instance, those later tokens are frequently the answer tokens, so READQ can preferentially strengthen memorization of correct answers relative to an unmasked baseline. The effect magnitude would depend on K, and K is never reported. Thus the headline result is not yet distinguishable from contamination-amplified memorization. This is a load-bearing, testable concern, not a claim that the authors acted improperly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two training-time modifications intended to give LLMs a 'silent reading' phase before generation. READQ masks the next-token prediction loss on the first K tokens of each sequence, and READQBUDDY adds an auxiliary model whose penultimate-layer embedding is summed into the main model's input embedding at every step. The authors report multi-point accuracy gains over same-data baselines on ARC-Challenge, HellaSwag, OpenBookQA, PubMedQA, and Winogrande for a 3B model, and on MedQA and related scientific benchmarks for a 70B model. The paper is an early-stage empirical report; no code, checkpoints, repeated seeds, or decontamination analysis are provided.","tokens_in":5851,"tokens_out":4064,"duration_ms":49573,"significance":"If the reported gains are real and reproducible, this would be a notably cheap way to improve performance on several standard reasoning benchmarks, and the 70B MedQA improvement would be practically interesting. The experimental design has a real strength: every comparison is against a baseline trained on the same data, so the method, rather than the data mixture, is the intended variable. The paper also honestly flags that READQBUDDY may be under-trained and that the training recipe was not exhaustively tuned. However, the evidence as presented is not yet sufficient to support the headline claim: there are no repeated seeds or uncertainty estimates, the key hyperparameter K is never reported, the READQBUDDY architecture is underspecified, and no decontamination check is described for a training corpus composed of public web-scale datasets.","major_comments":[{"comment":"The masking window length K is never reported. READQ's behavior is defined entirely by K, and the contamination risk depends on how much of each sequence is masked, because masking shifts the training signal to later, context-rich tokens. Without K, the experiments are irreproducible and the mechanism cannot be assessed. Please report K for every configuration and include a sensitivity analysis over K.","section":"§3.1, App. A.1"},{"comment":"All reported numbers appear to come from a single training run, with no error bars, no multiple seeds, and no statistical tests. Several benchmarks are small (OpenBookQA has roughly 500 examples, PubMedQA roughly 1000), so differences of 2–10 points cannot be evaluated for significance without variance estimates. Please provide at least three independent runs per condition and report standard deviations or bootstrap confidence intervals.","section":"§4, Tables 1 and 2"},{"comment":"The continued-pretraining corpus consists of Fineweb, the Pile, Cosmopedia, UltraBooks, and DeepMind Math, all public and web-scale, while the evaluated benchmarks are standard public datasets that commonly appear in such corpora. The paper describes no decontamination step. This is load-bearing because READQ's loss masking preferentially weights later tokens, which in a leaked benchmark instance are frequently the answer tokens; the measured gains are therefore not yet distinguishable from contamination-amplified memorization. Please report an n-gram overlap analysis between the training mixture and each test set, and rerun the main comparisons on any uncontaminated subset.","section":"§4, App. A.1"},{"comment":"READQBUDDY is underspecified to the point of irreproducibility. Table 1 reports SW16 and SW64 variants, but Algorithm 1 contains no sliding-window parameter; the paper does not state the buddy model's context limit, the layer index from which the embedding is taken, the projection layer's initialization and hidden size, or how the 'entire input context' is handled when it exceeds the buddy's window. These details are necessary to reproduce the method and to interpret the difference between READQ and READQBUDDY.","section":"§3.2, Algorithm 1"}],"minor_comments":[{"comment":"The column headers 'P UBMEDQA' and 'W INOGRANDE' contain spacing errors that make the table difficult to read.","section":"Table 1"},{"comment":"The abbreviations SW16 and SW64 appear in the table but are explained only in the caption; please define them in the body of Section 3.2.","section":"Table 1, §3.2"},{"comment":"The phrase 'multiple point accuracy boost' is unclear; consider rewording to 'multi-point accuracy improvements' or giving specific numbers.","section":"Abstract"},{"comment":"The discussion of Rho-1 is brief and does not state how READQ's loss masking differs from Rho-1's selective token loss, which would help position the contribution.","section":"§2"},{"comment":"The bibliography entry for DeepMind Math appears as 'Grefenstette Saxton and Kohli Hill' with garbled author order; the correct citation should be checked.","section":"References"},{"comment":"The 70B experiment uses a different training mixture ('pretraining and supervised fine-tuning') from the 3B experiments; this difference is stated only in the appendix and should be noted in the main text where Table 2 is discussed.","section":"App. A.1, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like an early technical report. The central risk is that the headline results are artifacts of evaluation noise or benchmark contamination; both concerns are concrete and testable, and the authors should be asked to supply repeated runs and a contamination analysis before further consideration. The title's claim of 'decoupling comprehension and reasoning' overstates what is shown, since the paper provides no mechanistic evidence for a distinct comprehension phase."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a plausible core idea and one reasonably original component, but the headline numbers are not yet trustworthy because the evaluation is open to a specific, testable contamination failure.\n\nWhat's new: READQ is a fixed-window loss mask that skips the first K tokens of each sequence. That's close to Rho-1, which the paper cites, but READQ is simpler and the paper doesn't compare against it. READQBUDDY, where a 1B reader's embeddings are added to the 3B generator's input, is a genuinely different architecture and the more interesting piece. The paper uses same-data baselines, which is correct, and it is honest that these are early results with an under-tuned recipe.\n\nSoft spots: The main one is contamination. The continued-pretraining corpus is Fineweb, Pile, Cosmopedia, UltraBooks, and DeepMind Math—all public web-scale sets—and the test benchmarks are standard public datasets. No decontamination is reported. The stress-test concern is real and load-bearing: READQ masks the first K tokens, so the training signal concentrates on later, context-rich tokens, which in a leaked example are often the answer tokens. The baseline also trains on the same leaked data, so the baseline can also memorize, but READQ differentially amplifies the answer-token loss and could inflate the gain. Without a decontamination analysis, the 3–8 point boosts are not distinguishable from memorization amplification.\n\nAlso soft: K is never reported. That is the central hyperparameter. Single run, no error bars, no seeds. No comparison to Rho-1 or re-reading, so the reader can't tell if the fixed-window mask does anything beyond token-loss weighting. The 'silent reading' framing is post-hoc; the actual mechanism could simply be loss reweighting.\n\nThe citation pattern is fine. The CEPO self-citation is in related work, not load-bearing.\n\nWho's this for? Anyone working on token-level training curricula or auxiliary reader architectures. The idea is worth a serious referee because if the gains survive decontamination, it's a cheap and useful training change. But as submitted, the missing K and decontamination analysis make it impossible to accept the claims. My recommendation: send it to review, but require the authors to report K, add variance over seeds, and run a decontamination check (removing overlapping n-grams from the corpus and re-running). If the gains disappear, the paper becomes a negative result about contamination—that's still informative.\n\nVerdict: conditional reject for this version, but a legitimate candidate after revision.","headline":"A simple training trick and a genuinely different auxiliary-reader architecture, but the headline gains are not yet distinguishable from contamination-amplified memorization because K is unreported and no decontamination is described.","tokens_in":6337,"tokens_out":2478,"would_cite":false,"duration_ms":26479,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Giving LLMs a silent-reading phase—by masking early-token loss and adding a small reading buddy—improves reasoning accuracy by several points.","keywords":["silent reading","loss masking","READQ","READQBUDDY","continued pretraining","LLM reasoning","reasoning benchmarks","auxiliary model"],"falsifier":"Run the same continued-pretraining recipe on a corpus from which all six evaluation benchmarks have been explicitly removed, then evaluate on a private set of new questions in the same formats; if the multi-point gains disappear, memorized test items are responsible, not silent reading. If the gains persist on the private set, the mechanism is confirmed.","tokens_in":5350,"feed_emoji":"📖","tokens_out":9530,"duration_ms":107018,"temperature":0.7,"pith_summary":"LLM reasoning is usually trained purely on next-token prediction, then steered at inference time by explicit chain-of-thought. This paper claims that an earlier, silent stage—comprehension before any response token is predicted—is being skipped, and that making room for it in training improves accuracy. The first technique, READQ, masks the next-token loss for the first few tokens after each beginning-of-sequence marker, creating a gradient-free reading window. The second, READQBUDDY, adds a small auxiliary model that reads the input and injects its context embedding into the generator at every step. Across ARC-Challenge, HellaSwag, OpenBookQA, PubMedQA, and Winogrande, the authors report multi-point accuracy gains over the same continued-pretraining recipe without the techniques, including an 8-point gain on MedQA with a 70B model. If the gains are real, they come with no extra inference-time compute for READQ and only a small auxiliary model for READQBUDDY, making this an unusually cheap way to buy reasoning quality.","feed_headline":"Silent reading before answering lifts LLM reasoning scores","feed_subtitle":"Giving the model a loss-free reading window before it predicts boosts accuracy on five reasoning benchmarks.","key_machinery":"The machinery is next-token prediction loss masking, extended by an auxiliary-model embedding injection. READQ's loss mask sets the loss contribution of the first K predictions after each BOS token to zero, which both removes noisy gradients from context-poor early tokens and, the paper hypothesizes, lets the model learn to build an internal representation before it has to generate. READQBUDDY's buddy is a second, smaller LLM that reads the entire input in parallel; its penultimate-layer embedding is summed with the main model's embedding-layer output at every generation step, so the context summary is available when every output token is predicted. That persistent embedding sum is what carries the READQBUDDY result, while the zero-loss window carries READQ.","core_discovery":"The paper's central claim is that comprehension and reasoning can be separated in an LLM's training objective: a model should first read quietly, without being forced to predict, and only then think and speak. READQ implements the reading phase by zeroing the autoregressive loss on the first K predictions after each beginning-of-sequence token, so the model is never penalized for failing to guess words it has no context to know. READQBUDDY extends the reading phase across the whole response by running a second, smaller LLM over the input and adding that model's penultimate-layer embedding to the main model's input embedding at every step. The authors report that both techniques improve accuracy over the same training recipe on every benchmark they test, that the gains hold on a base model with no instruction tuning, and that the effect scales to a 70B model, where MedQA rises from 66.8% to 74.7%. They present the results as early evidence and note that their READQBUDDY recipe likely under-trains the auxiliary component, which they read as headroom rather than a ceiling.","pith_inferences":["Not claimed in the paper, but a direct extension: the loss-mask window size K has not been swept, and the reported gains are from a single fixed choice; a proper sweep could reveal the effect is even larger or peaks near zero.","If the two-phase view is correct, then RL-heavy reasoning training may be optimizing the wrong phase; the highest-value configuration could be reinforcement learning applied on top of a silent-reading base, which the paper neither confirms nor rules out.","Because READQBUDDY's buddy reads the whole input once and its embedding is reused at every step, the method should shine on long-context, retrieval-augmented, or multi-turn tasks where the model must keep a large context in mind; testing that is an obvious next experiment.","The benchmark suite is multiple-choice and classification heavy; open-ended generation is untested, so whether silent reading improves free-form reasoning quality rather than answer selection is an open question."],"forward_implications":["READQ can be added to any continued-pretraining recipe with no change to the architecture and no extra inference-time compute, so any future training run can capture the reported gain just by choosing a loss-mask window size.","The loss mask concentrates gradients on tokens the model has enough context to learn from, so the same training budget should produce a model that is better calibrated on early-context and long-context material.","The buddy architecture means a small auxiliary model can supply persistent context to a large generator, so the main model does not need to re-read or hold all context in its own activations.","The 70B MedQA result indicates the effect is not an artifact of small-scale training: on dense specialized reasoning, the silent-reading window can move accuracy by multiple points.","Silent reading and explicit chain-of-thought target different phases of response generation, so combining READQ with a thinking-trace stage is a natural next step the paper proposes."],"supporting_citations":[{"why":"Prior work showing re-reading input improves LLM reasoning; the motivation for a silent-reading phase.","marker":"Xu et al. (2024)"},{"why":"Supplies the Llama 3.2 3B and Llama 3.1 70B base models on which all experiments and ablations run.","marker":"Grattafiori et al. (2024)"},{"why":"Chain-of-thought prompting, the explicit thinking-trace mechanism the paper positions its silent-reading phase against.","marker":"Wei et al. (2023)"},{"why":"Token-level loss reweighting that motivates the claim that early context-poor tokens produce noisy gradients.","marker":"Lin et al. (2025)"},{"why":"MedQA, the medical benchmark where READQ produces the 8-point gain at 70B scale.","marker":"Jin et al. (2021)"},{"why":"Fineweb, one of the continued-pretraining corpora used to train the evaluated models.","marker":"Penedo et al. (2024)"}],"fun_headline_variants":["Loss-free reading before answering boosts LLM reasoning","Silent reading phase lifts LLM accuracy on benchmarks","Reading buddy architecture improves LLM reasoning at 70B","Quiet reading step boosts LLM performance across scales"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation benchmarks are not part of the continued-pretraining corpus, so the measured accuracy gains reflect the reading techniques rather than the model having already memorized the test questions.","fun_headline_variants_meta":{"raw":{"variants":["Loss-free reading before answering boosts LLM reasoning","Silent reading phase lifts LLM accuracy on benchmarks","Reading buddy architecture improves LLM reasoning at 70B","Quiet reading step boosts LLM performance across scales"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2825,"prompt_tokens":959,"completion_tokens":1866,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1803}},"tokens_in":575,"tokens_out":1866,"duration_ms":16951,"temperature":1.0,"reasoning_tokens":1803,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:13:26.185372+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same continued-pretraining recipe on a corpus from which all six evaluation benchmarks have been explicitly removed, then evaluate on a private set of new questions in the same formats; if the multi-point gains disappear, memorized test items are responsible, not silent reading. If the gains persist on the private set, the mechanism is confirmed.","supporting_citations":[{"cited_title":"What disease does this patient have? a large-scale open domain question answering dataset from medical exams","cited_arxiv_id":null,"evidence_quote":"MedQA, the medical benchmark where READQ produces the 8-point gain at 70B scale."}],"review_version":1}