{"id":"8f172ec3-9748-43fb-9fb0-ff6a1526d44f","arxiv_id":"2608.10928","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.","lead":"ThinkRetrieve is a method that retrieves solved example problems and injects them into a reasoning model's thinking trace at every step, instead of letting the model simply think longer. Across five reasoning models and four benchmarks, it improved accuracy over sequential test-time scaling, with the largest gains on hard competition math.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing prompt-only control conflates retrieval content with the re-checking instruction; the central attribution to retrieved exemplars is not yet established.","rationale":"The reader's weakest assumption was that dense cosine similarity reliably retrieves transferable exemplars. My concern is adjacent but distinct: the experimental design does not isolate whether the retrieved exemplar content, rather than the elaborate continuation prompt c, drives the gains. This is load-bearing because the paper's central claim is specifically that retrieval-augmented reasoning traces are what make test-time compute more effective. The Rand baseline helps but cannot settle the question, since random exemplars can actively harm the model and thus mask a prompt-only improvement. I do not see this as a fatal flaw; the paper includes strong supporting evidence (per-step injection beats static ICL, random retrieval underperforms, compute-matched self-consistency loses, answer-leakage controls are clean). But the missing prompt-only arm is a concrete, inexpensive experiment that would confirm or refute the causal attribution. The reader's CONDITIONAL verdict already accommodates this uncertainty, so I recommend no change.","tokens_in":19360,"tokens_out":8825,"duration_ms":87440,"concrete_test":"Run a prompt-only control on AIME 2025 and MATH-500 with Qwen3-1.7B and Qwen3-4B: use Algorithm 1 exactly, but omit the retrieval step and replace the exemplar slot in Appendix K with a neutral placeholder (e.g., 'Let me re-examine my reasoning'), at the same budgets B in {2K, 4K, 8K, 16K, 22K, 32K} and the same three seeds. Compare best-across-budget accuracy against ThinkRetrieve, sequential TTS, and Rand. If prompt-only accuracy is within 1-2 points of ThinkRetrieve, the central attribution to retrieved content fails; if it matches sequential TTS, the retrieval attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ThinkRetrieve's central claim is that injecting retrieved solved exemplars into the reasoning trace is what improves test-time scaling. But the experiments never compare against the continuation prompt alone. The prompt c from Section 3.2.1 (Appendix K) is a strong, domain-specific instruction: 'Wait, let me check again... Now coming back to the original question, here are the key points I can take from example question...' This prompt is applied only to ThinkRetrieve and the Rand baseline, not to sequential TTS. The reported gains therefore conflate two variables: (a) the content of the retrieved exemplar and (b) the directive re-checking prompt itself. The Rand baseline cannot resolve this: random exemplars may actively confuse the model, masking a prompt-driven improvement, while static ICL (S-ICL) prepends exemplars without c and also cannot separate the prompt effect. If a no-exemplar version of c already recovers most of the AIME/MATH-500 gains, then the paper's attribution of the gains to retrieval quality is unsupported, and the method could reduce to prompt engineering plus a domain-matched memory rather than retrieval-augmented reasoning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ThinkRetrieve, a test-time scaling framework for large reasoning models. Rather than extending a reasoning trace by sequential self-reflection alone, ThinkRetrieve iteratively closes the think block, prompts the model for an intermediate answer, encodes it together with the original question, retrieves the nearest solved exemplar from an external corpus, and injects that exemplar into the trace together with a fixed continuation prompt. The procedure repeats until a token budget is exhausted. Experiments cover five reasoning models and four benchmarks, comparing against standard thinking, sequential test-time scaling, static input-level ICL, and random per-step retrieval, with additional ablations on retrieval encoder, query formulation, self-consistency at matched compute, and answer-distinct filtering. The paper reports that ThinkRetrieve wins on every (model, benchmark) cell, with gains up to +13.4 absolute points on AIME 2025, and that retrieved exemplars reduce predictive entropy and per-step confidence reversal.","tokens_in":19613,"tokens_out":5009,"duration_ms":48442,"significance":"If the central attribution holds, the contribution is practically significant: it provides a lightweight, model-agnostic intervention that makes additional test-time tokens more effective than self-reflection alone across several open reasoning model families. The paper deserves credit for including multiple controls that go beyond a single baseline: random per-step retrieval, static ICL, compute-matched self-consistency, an answer-distinct retrieval filter, per-problem help/hurt tests, and a top-1 exemplar audit. These controls jointly make a credible case that the gains are not merely answer leakage or extra compute. However, the causal role of the retrieved exemplar content is not yet fully isolated, because no condition applies the continuation prompt alone; the absence of that control is the main load-bearing gap. The manuscript also overstates certainty in small-sample cells without error bars.","major_comments":[{"comment":"The central attribution claim is not established because no condition applies the continuation prompt c without a retrieved exemplar. The prompt template in Appendix K is a strong, domain-specific instruction: 'Wait, let me check again... Here is a similar question and its solution... Now coming back to the original question...' This prompt is used in ThinkRetrieve and in the Rand baseline, but not in sequential TTS or static ICL, which use different continuation instructions. The reported gains over sequential TTS therefore conflate two variables: the content of the retrieved exemplar and the directive to re-read and extract key points. The Rand baseline cannot resolve this, because random exemplars may actively confuse the model and mask a prompt-driven improvement. The authors should add a prompt-only control that inserts c at each step boundary with no exemplar, or with a neutral placeholder, while keeping all other mechanics fixed. If prompt-only already recovers most of the gains on AIME 2025 and MATH-500, the paper's central claim is unsupported; if it does not, the attribution to exemplar content is validated.","section":"§3.2.1, Appendix K"},{"comment":"The headline result reports best accuracy across thinking budgets, averaged over three seeds, without standard errors, confidence intervals, or per-budget seed-level values. This is especially consequential for AIME 2025, which has only 30 problems: a difference such as Qwen3-8B's 71.1 versus 68.9 is less than one problem per seed and cannot be distinguished from noise on this evidence. The claim that ThinkRetrieve wins on every (model, benchmark) cell needs uncertainty quantification, such as binomial confidence intervals or paired significance tests, at least for the small-AIME and near-saturated cells. In addition, selecting the best budget post hoc from a grid of budgets can inflate measured gains; the paper should report results at a fixed budget or otherwise correct for this selection, and should show the underlying per-seed numbers.","section":"Table 1, Fig. 4"},{"comment":"The leakage audit is in part circular. After removing all corpus entries with cosine similarity above 0.90, the maximum retained similarity is necessarily below 0.90, so the statement that 'zero queries exceed the decontamination threshold' is true by construction and cannot by itself validate the threshold. The relevant evidence would be the distribution of similarities to the unfiltered corpus and an assessment of whether pairs just below 0.90, in the 0.88-0.90 band, are genuine near-duplicates or structurally analogous but mathematically distinct problems. Section 5.1 provides a few qualitative examples, but the paper should report unfiltered similarity statistics and either justify the 0.90 threshold independently of the observed maximum or provide a systematic audit of the band just below it.","section":"§4, corpus decontamination and Table 4"}],"minor_comments":[{"comment":"The compute-matched self-consistency comparison is restricted to a stratified subset of MATH-500 levels 4 and 5 and reports single point estimates without confidence intervals; because this comparison is used to rule out 'compute alone' as the explanation, the appendix should state the subset size and report variance or a paired test.","section":"Table 3, Appendix E"},{"comment":"The paper says results are averaged over three seeds and reports 'best accuracy across budgets', but it does not specify how many budget values were evaluated per cell or how ties across budgets were broken; this should be stated so that the oracle-selection issue is transparent.","section":"§4, Tables 1 and 2"},{"comment":"The answer-distinct retrieval ablation reports that accuracy is 'essentially unchanged' and 'marginally higher' without giving the actual numbers; the per-seed accuracies for the filtered and unfiltered runs should be reported explicitly, since this control is central to ruling out answer leakage.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the method is plausible, but the missing prompt-only control is a load-bearing gap that should be resolved before acceptance. I do not see evidence of misconduct or deliberate omission; the issue looks like an incomplete ablation. The small AIME sample and oracle budget selection also need to be addressed with proper uncertainty reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ThinkRetrieve is a solid, unusually thorough paper on a real problem—sequential test-time scaling often plateaus or degrades, and the authors propose injecting solved exemplars into the reasoning trace at each step. The per-step injection of complete worked solutions is genuinely new relative to the cited RAG-for-reasoning work, which retrieves facts or static demos. The experiments are extensive: five models, four benchmarks, and a decent set of controls.\n\nWhat impressed me: the random-retrieval and static-ICL baselines, the answer-distinct retrieval control, the compute-matched comparison against self-consistency, and the per-problem help/hurt decomposition with McNemar. The decontamination audit is careful. The gains are consistent across every (model, benchmark) cell, which makes the central phenomenon hard to dismiss. The paper also gives a plausible mechanism—lower answer entropy—and supports it with measurements.\n\nThe soft spots are real but not fatal. The biggest one: there is no prompt-only control. The continuation prompt c (Appendix K) is a strong, domain-specific instruction—\"Wait, let me check again... here are the key points I can take from example question...\"—and it is applied only to ThinkRetrieve and the random baseline, not to sequential TTS. The Rand baseline cannot cleanly separate the prompt effect because random exemplars may actively mislead. A no-exemplar version of c would tell you how much of the gain comes from the prompt alone. If that recovers most of the AIME/MATH-500 improvement, the \"retrieval content\" attribution is overclaimed. This is fixable and should be required in revision.\n\nTwo smaller issues: Table 1 reports best accuracy across budgets without variance, and AIME 2025 is only 30 problems, so the headline +13.4 is noisier than it looks. The decontamination threshold of 0.90 sits close to the observed max similarity (0.898), but the answer-distinct control goes a long way toward mitigating that worry. The self-citation in the motivation is mildly present but not egregious.\n\nWho this is for: anyone working on test-time scaling, LLM reasoning, or retrieval-augmented generation. It deserves a serious referee. The missing control is the main thing to ask for; everything else is revision-level polish.\n\nMy recommendation: send it to review. Have the referee request the prompt-only ablation and per-budget error bars. If the prompt-only control collapses the effect, the paper still contributes a useful result about prompt engineering; either way it is worth referee time.","headline":"A thorough, well-controlled study of per-step exemplar retrieval for test-time scaling that deserves review, though the missing prompt-only control leaves the central attribution to retrieval content not fully established.","tokens_in":20116,"tokens_out":2843,"would_cite":true,"duration_ms":26042,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Injecting retrieved solved examples into a reasoning model's trace at each step beats sequential self-reflection on every model and benchmark tested.","keywords":["test-time scaling","retrieval-augmented generation","large reasoning models","chain-of-thought","in-context learning","exemplar retrieval","mathematical reasoning","reasoning drift"],"falsifier":"Construct a test set of 100 multi-step problems whose solution procedures are absent from the example corpus (for example, novel problem classes or deliberately mismatched domains), run ThinkRetrieve and sequential test-time scaling at matched token budgets, and check whether ThinkRetrieve falls to or below the sequential baseline; if it does not, the claim that retrieval quality drives the gains would be falsified. A more direct probe: corrupt the retrieved exemplar's solution steps while keeping its surface form and measure whether the model adopts the corrupted procedure more often than it corrects it.","tokens_in":19222,"feed_emoji":"🧠","tokens_out":12413,"duration_ms":89974,"temperature":0.7,"pith_summary":"Sequential test-time scaling, which lets a reasoning model 'think longer' by generating more chain-of-thought tokens, runs into diminishing returns and outright collapse: longer traces drift, repeat themselves, and compound errors. ThinkRetrieve replaces some of that self-reflection with retrieval: at each reasoning step, the model states its current intermediate answer, that answer is encoded together with the original question, and the nearest solved problem from an external corpus is injected into the thinking trace as a worked example. The paper's central claim is that this procedural scaffolding makes test-time compute more effective than self-reflection alone, and that the effect is consistent: ThinkRetrieve improves accuracy on every (model, benchmark) cell across five 1.5B–8B reasoning models and four benchmarks, with the largest gains on the hardest set (up to +13.4 absolute points on AIME 2025). It also keeps accuracy stable or increasing as the thinking budget grows, whereas sequential test-time scaling plateaus or degrades. A sympathetic reader would care because it offers a training-free way to make smaller reasoning models substantially more reliable on hard math problems.","feed_headline":"Solved examples injected mid-reasoning beat thinking longer in AI math","feed_subtitle":"ThinkRetrieve raises AIME 2025 accuracy by up to 13.4 points and stays stable as the token budget grows.","key_machinery":"The central object is the interleaved reasoning–retrieval trajectory $\\tau_k = (z_1, e_1, z_2, e_2, \\ldots, z_k, e_k)$, where $z_t$ are model-generated reasoning steps and $e_t$ is the solved exemplar retrieved after step $t$. At each step the model is prompted to produce an intermediate answer $y_t$, and the retrieval query is formed by jointly encoding the test question with $y_t$: $q_t = \\mathrm{Enc}(x_{\\mathrm{test}}, y_t)$. The most similar corpus entry, selected by cosine similarity against pre-encoded question–solution pairs, is formatted as an in-context example and injected into the trace together with a fixed continuation prompt that asks the model to extract key takeaways from the example. Exemplar tokens count against the same token budget $B$ as generated tokens, so ThinkRetrieve produces fewer model-generated tokens than sequential test-time scaling at a matched budget; the design isolates retrieval quality, not extra compute, as the cause of the gains. The loop repeats until the budget is exhausted, and the final answer is generated from the fully augmented trajectory.","core_discovery":"The paper claims that interleaving retrieved solved examples into a reasoning model's thinking trace changes what test-time compute buys: instead of relying on the model's own self-reflection to extend a line of reasoning, ThinkRetrieve periodically closes the thinking block, asks for an intermediate answer, retrieves a structurally similar solved problem from a large example bank, and reopens thinking with that exemplar and a fixed continuation prompt placed in the trace. The retrieved exemplar is a full question–solution pair, so it demonstrates how to reason, not merely which facts to use. Empirically the claim is that this wins on every (model, benchmark) cell across five models (1.5B–8B) on GSM-8K, MATH-500, AIME 2025, and SciQ, with an absolute gain of up to +13.4 points on AIME 2025 and relative gains up to 60%, while maintaining monotonic or stable accuracy as the token budget grows—in contrast to sequential test-time scaling, which plateaus or collapses. The paper further claims the gains come from reduced answer uncertainty: predictive entropy drops by roughly 0.55 nats, per-step confidence decreases monotonically under ThinkRetrieve, and a per-problem help/hurt decomposition shows the method helps far more than it hurts, with help concentrated on hard, procedurally transferable problems.","pith_inferences":["The low-entropy anchoring that drives the gains is a double-edged sword: if a retrieved exemplar is subtly wrong in a load-bearing way, the model may lock in a confident wrong answer. A testable extension would inject deliberately flawed exemplars and measure whether the model adopts or corrects the flaw.","The framework's dependence on a matched example corpus suggests that the transfer bottleneck is corpus construction, not model size; building high-coverage procedural corpora for code generation or formal proof could extend the method to non-math domains, a direction the paper flags as open.","Because exemplar tokens occupy the same budget as generated tokens, the method implicitly trades generation for retrieval; an adaptive policy that decides when to retrieve versus when to continue thinking could outperform a fixed per-step schedule, but the paper does not explore this.","The paper evaluates only open-weight models up to 8B; the mechanism may behave differently at scales where sequential self-reflection is more reliable, and the paper itself notes that even Qwen3-32B shows non-monotonic TTS returns, leaving open whether ThinkRetrieve's advantage persists at larger scale."],"forward_implications":["Sequential test-time scaling is not the only way to spend inference compute: grounding the trace with retrieved worked examples yields stable, sometimes monotonic gains where self-reflection plateaus or collapses, such as DeepSeek-R1-1.5B on GSM-8K staying at 84% while sequential TTS falls from 83% to 52%.","Both design choices matter: per-step in-trace injection is necessary (static input-level ICL underperforms) and semantic retrieval relevance is necessary (random per-step retrieval underperforms).","At matched total token budget, a single ThinkRetrieve pass beats TTS with self-consistency majority voting for k in {2,4,8} on MATH-500 hard problems, ruling out compute alone as the explanation.","The method preferentially helps hard, procedurally rich problems (level 5, Intermediate Algebra) and is neutral on already-saturated subjects, meaning it adds guidance where the model is uncertain without destabilizing competent reasoning.","The gains survive an answer-leakage control: excluding corpus entries whose final answer matches the gold answer does not reduce accuracy, so the benefit reflects structural and procedural similarity, not answer copying."],"supporting_citations":[{"why":"Provides the sequential test-time scaling baseline (budget forcing) that ThinkRetrieve extends and compares against.","marker":"Muennighoff et al., 2025"},{"why":"Documents diminishing and negative returns of longer reasoning traces, motivating the need for external procedural grounding.","marker":"Ghosal et al., 2025a"},{"why":"Establishes test-time scaling as an alternative to parameter scaling, the paradigm the paper targets.","marker":"Snell et al., 2024"},{"why":"Defines retrieval-augmented generation, the static-retrieval approach ThinkRetrieve contrasts with.","marker":"Lewis et al., 2020"},{"why":"RAT interleaves retrieval with reasoning for factual knowledge, the baseline ThinkRetrieve distinguishes from procedural exemplar retrieval.","marker":"Wang et al., 2024c"},{"why":"Search-o1 dynamically triggers search during reasoning, another dynamic-retrieval baseline the paper positions against.","marker":"Li et al., 2025"},{"why":"Supplies the sentence encoder used to embed retrieval queries and corpus entries.","marker":"Wang et al., 2022"},{"why":"Provides the synthetic math corpus from which the example bank is filtered.","marker":"LI et al., 2024"},{"why":"Supplies the MATH-500 benchmark used to measure gains on competition-level problems.","marker":"Hendrycks et al., 2021"}],"fun_headline_variants":["Inject solved examples into AI reasoning: 60% better on AIME","Retrieval-augmented reasoning: up to 60% relative gain on AIME","ThinkRetrieve: inject solved examples into thinking, not just tokens","AI math: mid-reasoning exemplars lift AIME up to 60% relative"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains depend on dense cosine similarity, computed with a pre-trained sentence encoder over a query formed by concatenating the test question with the model's intermediate answer, reliably retrieving a solved example whose solution procedure transfers to the test problem; when retrieval selects a misleading exemplar, the injected trace can anchor the model on a confident wrong answer, a failure mode the paper acknowledges in its Limitations.","fun_headline_variants_meta":{"raw":{"variants":["Inject solved examples into AI reasoning: 60% better on AIME","Retrieval-augmented reasoning: up to 60% relative gain on AIME","ThinkRetrieve: inject solved examples into thinking, not just tokens","AI math: mid-reasoning exemplars lift AIME up to 60% relative"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000972,"raw_usage":{"total_tokens":4161,"prompt_tokens":1006,"completion_tokens":3155,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":3069}},"tokens_in":622,"tokens_out":3155,"duration_ms":20365,"temperature":1.0,"reasoning_tokens":3069,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:05:31.580239+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set of 100 multi-step problems whose solution procedures are absent from the example corpus (for example, novel problem classes or deliberately mismatched domains), run ThinkRetrieve and sequential test-time scaling at matched token budgets, and check whether ThinkRetrieve falls to or below the sequential baseline; if it does not, the claim that retrieval quality drives the gains would be falsified. A more direct probe: corrupt the retrieved exemplar's solution steps while keeping its surface form and measure whether the model adopts the corrupted procedure more often than it corrects it.","supporting_citations":[],"review_version":1}