{"id":"60dc8f59-beed-40ab-aa17-b192d557766e","arxiv_id":"2607.13988","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Turn-level rewards from TD changes in a frozen reference model's gold-answer log-probability improve long-horizon search-agent RL on closed- and open-web benchmarks.","lead":"TRACE turns a frozen language model into a progress meter for web-search agents, rewarding each tool call by how much closer the transcript gets to the correct answer. On deep-research benchmarks it lifts Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6 using RL alone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TRACE's dense reward is vulnerable to the policy writing the gold answer into its private reasoning, which the frozen reference model sees when scoring prefixes; if this occurs, gains may reflect reward hacking rather than credit assignment.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing risk: gold-answer log-probability under a frozen reference model is only a valid dense reward if the policy cannot manufacture high prefix scores through its own generated text. Since the paper explicitly includes private reasoning in the assistant turn and never masks it from the reference scorer, the reward is, in principle, hackable. This is the single most important threat to the central claim because the headline result — 'pure RL improves tool use' — depends on the dense TD signal measuring evidence-gathering progress rather than the policy's ability to echo candidate answers in its reasoning. The paper has genuine independent support: controlled comparisons against GRPO/GSPO/GiGRPO, ablations of the log-ratio transform, and an offline diagnostic in Table 6. None of these controls isolates the policy's own reasoning tokens. The offline diagnostic in §A.2 correlates cumulative credits with final reference-model score and outcome reward, but it does not test whether those credits would persist if reasoning were masked. The qualitative examples in §A.5 are helpful but cherry-picked and cannot establish that the policy did not, on average, learn to place answer-like strings in reasoning. I therefore agree with the reader's conditional verdict. The concern does not move the verdict because it is a testable hypothesis, not a demonstrated failure; but it should be explicitly addressed before the method is treated as established. My proposed check directly settles it: if a reasoning-masked reference scorer reproduces the result, the concern is empirically refuted; if not, the central claim weakens substantially.","tokens_in":25739,"tokens_out":5324,"duration_ms":55979,"concrete_test":"Re-run the Qwen3-4B BrowseComp-Plus TRACE training with the reference scorer modified to mask all assistant reasoning tokens before computing \\bar\\ell_k, keeping tool calls and observations intact. If the resulting accuracy stays near 35.6 and the learned tool-use behavior matches the original run, the concern is refuted. If accuracy falls toward the GRPO baseline of 30.0, or if the turn-level credits decorrelate from final correctness in an offline recomputation on stored rollouts, the reported gains are substantially attributable to reasoning leakage rather than to the credit-assignment signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central premise (§3.2) is that a transcript with relevant evidence makes the gold answer easier for a frozen reference model to predict, so gold-answer log-probability is a monotone progress signal. But TRACE scores prefixes S_k that include the policy's own assistant-turn text: §2.1 defines actions as policy-generated assistant tokens, and §A.3 states that 'the assistant's reasoning, tool call, and final-answer formatting tokens are part of the assistant turn.' Algorithm 1 never strips private reasoning before computing \\bar\\ell_k in Eq. 5. Therefore the policy can inflate V(S_k) and earn positive TD credit \\delta_k = V(S_{k+1}) - V(S_k) simply by emitting the gold answer, or a high-likelihood paraphrase, in its private chain of thought before the next tool call — even if the tool observation added no evidence. The terminal verifier only checks the final <answer> span, so a failed rollout can still receive large dense turn rewards for reasoning that guesses y*. This is a direct failure mode of the monotone-progress assumption, not a peripheral implementation detail. The paper provides no countermeasure (e.g., masking reasoning tokens from the reference scorer), no sensitivity analysis for this failure mode, and no trajectory-level statistic for whether y* appears in private reasoning before the evidence that justifies it. The qualitative examples in §A.5 show nicely aligned credits on a handful of successful/failed trajectories, but they do not quantify how often the reference-model jump is caused by the policy's own text rather than by the tool observation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TRACE, a dense turn-level reward mechanism for long-horizon agentic RL. At each tool-call boundary, a frozen reference model scores the prefix by the gold answer's average log-probability; these scores are converted into log-ratio state values, and Temporal-Difference changes in those values supply per-turn credit. The turn-level signal is combined with a standard GRPO outcome advantage. On closed-web BrowseComp-Plus, the authors report gains from 7.2 to 35.6 (Qwen3-4B) and 8.4 to 42.6 (Qwen3-30B-A3B), with transfer to open-web benchmarks, all without cold-start SFT or live-web training. The paper includes controlled comparisons against GRPO, GSPO, and GiGRPO, ablations of the credit format and hyperparameters, and qualitative trajectory analyses.","tokens_in":26130,"tokens_out":6985,"duration_ms":67840,"significance":"If the reported gains are robust, TRACE is a practically significant contribution: it provides dense, critic-free, label-free turn credit for long-horizon tool-use agents while keeping the verifiable outcome reward as the final anchor. The controlled setup is a strength: the RL baselines share the same backbone, browser interface, training data, and terminal reward, and the ablations directly target the design choices. The qualitative examples in Appendix A.5 concretely illustrate the intended credit-localization behavior. However, the central empirical claim rests on single training runs, and the mechanism's core assumption—that gold-answer log-probability on the full prefix is a monotone progress signal—is exposed to a concrete reward-hacking failure mode through the policy's private reasoning. These issues need to be addressed before the headline claims can be accepted at face value.","major_comments":[{"comment":"The central premise is that a transcript containing relevant evidence makes the gold answer easier for the frozen reference model. However, the reference scorer computes log π_ref(y*_t | S_k, y*_<t) on prefixes that include the policy's private reasoning (Section A.3 explicitly counts reasoning and formatting tokens as part of the assistant turn), and Algorithm 1 performs no masking of reasoning tokens. A policy can therefore raise V(S_k) and obtain positive δ_k = V(S_{k+1}) − V(S_k) by writing y* or a high-likelihood paraphrase into its chain of thought before any evidence-adding tool observation, while still failing the final verifier. The paper provides no countermeasure, no quantification of how often y* appears in reasoning before the supporting evidence, and no sensitivity analysis. Since the headline gains are presented as evidence for credit assignment, this failure mode needs to","section":"§3.2, Eq. (5), Algorithm 1, §A.3"},{"comment":"The manuscript states 'controlled ablations are single training runs, so small differences should be read as directional rather than as variance-adjusted conclusions.' This caveat covers all controlled RL runs, not just ablations. In particular, the Table 2 comparison between 'remaining gap' (34.6) and 'log-ratio' (35.5) is a 0.9-point difference from one seed, and Figure 5(a) shows 35.6 vs 34.5 across coefficient settings. Without repeated seeds or error bars, the central ordering of methods cannot be distinguished from noise. At minimum, the primary TRACE-vs-GRPO comparison and the credit-format ablation should be run with multiple seeds and reported with variance.","section":"§4.1, Tables 1–2, Figures 3–5"},{"comment":"The contribution is empirical, yet no controlled comparison is made to the closest prior methods that derive process rewards from log-likelihood ratios or return decomposition (e.g., Yuan et al. 2024; Arjona-Medina et al. 2019). Without an adapted baseline, the observed gains cannot be attributed specifically to the TD/log-ratio formulation rather than to any dense frozen-model signal. A controlled variant implementing a log-ratio process-reward baseline on the 4B model would substantially strengthen the attribution of the improvement.","section":"Related Work, Table 1"}],"minor_comments":[{"comment":"The abstract says the 'one-step log-ratio TD component telescopes across redundant tool calls,' but the deployed reward uses the K-step backup (K=3) and terminal fill, which intentionally break the endpoint-only telescoping property. The paper acknowledges this in §3.3, but the abstract is potentially misleading.","section":"Abstract, §3.3"},{"comment":"The x-axis label says 'step k' but the ablation varies the look-ahead horizon K; please rename the axis to avoid confusion with the turn index k.","section":"Figure 5(b)"},{"comment":"The 'Legacy transform stabilizer' parameter is not used in the reported log-ratio runs but is listed in the TRACE configuration table. Please mark it clearly as a legacy/disabled parameter or remove it to avoid ambiguity.","section":"Table 4"},{"comment":"The notion of 'final-answer opener' used to score answer tokens is introduced informally and not reflected in the notation of S_k. Please state explicitly how the opener is constructed and whether it is included in every prefix used in Eq. (5).","section":"§2.1, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The private-reasoning exploit is the central unresolved issue. I would not reject outright because it is empirically testable—masking reasoning tokens or reporting the frequency of y* in pre-evidence reasoning would settle it. The single-run reporting is also below the usual bar for a journal claiming method-level improvements; please require repeated seeds for the main comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TRACE gives you a concrete way to turn a frozen reference model's gold-answer log-probability into per-turn reward for long-horizon agents. That's the thing to know. The recipe—log-ratio state value, TD differences at tool-call boundaries, K-step backup, terminal anchor—is more specific than prior work like Yuan et al. (2024) or Wang et al. (2026a), and the controlled comparison against GRPO, GSPO, and GiGRPO at two scales is genuinely informative. The ablations of credit format (raw delta vs remaining gap vs log-ratio) are well designed, and the qualitative examples in A.5 actually show the credits tracking the decisive search/open call and the misleading find call. The paper is also honest about its scope and about the single-run nature of the results.\n\nThe soft spots are real. The most important is the reward-hacking channel: the prefixes used to score answer log-prob include the assistant's own private reasoning tokens. If the policy writes the gold answer or a high-likelihood paraphrase in its chain-of-thought before the next tool call, the reference model's score jumps and the TD reward is positive even though the tool observation added nothing. The paper neither masks reasoning tokens from the reference scorer nor reports any statistic about how often the jump is driven by the policy's own text rather than the tool observation. That is a direct crack in the monotone-progress assumption. Second, all benchmark numbers are single runs with no error bars; the gaps are large, but without variance we can't know how much of TRACE's advantage is signal. Third, the closest log-ratio process-reward methods are cited but not run as controlled baselines, so we don't get a direct read on the marginal contribution of the specific TD formulation. Fourth, hyperparameters (including K, gamma_td, lambda_term, alpha_turn) are tuned on the evaluation benchmark, which inflates the headline numbers somewhat. None of these are fatal on their own, but together they mean the paper's strongest claim—TRACE substantially improves base-model tool use—is directionally supported rather than nailed down.\n\nWho should read it: anyone working on credit assignment for agentic RL, process reward modeling, or search-agent post-training. It's worth a serious referee: the method is novel enough, the experiments are relevant, and the failure modes are testable rather than hand-wavy. I'd send it out, and I'd ask the authors to address the reasoning-masking issue, add repeated-seed runs, and include at least one controlled baseline from the implicit-process-reward family.","headline":"TRACE is a plausible dense-credit recipe for agentic RL, but single-run evidence and a private-reasoning reward-hacking channel mean the headline gains should be treated as provisional, not established.","tokens_in":26633,"tokens_out":3138,"would_cite":true,"duration_ms":29406,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For long-horizon agents, a frozen model's answer confidence becomes a per-tool reward","keywords":["credit assignment","temporal difference","reinforcement learning","LLM agents","tool use","dense reward","log-probability","long-horizon search"],"falsifier":"Construct a rollout in which the policy copies the gold answer into its private reasoning before any search and check whether the frozen reference model's gold-answer log-probability spikes; if it does, the monotonicity premise fails. Alternatively, run the method on a long-horizon task with long or open-ended outputs, which the paper identifies as outside its current scope, and see whether turn credit tracks real progress.","tokens_in":25633,"feed_emoji":"🔍","tokens_out":7149,"duration_ms":60351,"temperature":0.7,"pith_summary":"The paper claims that turn-level credit for long-horizon agents can be computed from a frozen reference model alone, with no critic, process labels, or strong judge. The key idea is that a trajectory prefix makes the gold answer more predictable as it gathers evidence, so the temporal-difference change in the gold answer's log-probability is a valid dense reward for each tool call. Combined with an outcome-level advantage, this signal lets pure RL train competitive web-search agents: on the closed-web BrowseComp-Plus benchmark, a 4B model rises from 7.2 to 35.6 and a 30B-A3B model from 8.4 to 42.6. The paper argues this matters because outcome-only training cannot distinguish useful actions inside a failed rollout from the eventual mistake, and TRACE supplies that distinction without extra supervision.","feed_headline":"Frozen model's answer confidence becomes a per-tool reward","feed_subtitle":"Outcome-only RL leaves useful actions in failed rollouts unrewarded; TRACE's dense credit lifts BrowseComp-Plus from 7.2 to 35.6.","key_machinery":"The central object is the log-ratio state value V(S_k) = log(d_0 / d_k), where d_k is the remaining gap in the frozen reference model's average gold-answer log-probability for prefix k. The defining identity is telescoping: summing per-turn TD changes delta_k = V(S_{k+1}) - V(S_k) yields only V(S_T) - V(S_0), so total credit is endpoint-determined and trajectory padding cannot inflate it. This value plays the role a learned critic would play, but it is computed from a frozen model and combined with an outcome-level advantage.","core_discovery":"On the paper's own terms, the central discovery is that a frozen reference model's gold-answer log-probability, transformed into a log-ratio state value, gives a stable and telescope-friendly progress signal for agentic reinforcement learning. TRACE scores each prefix by the average gold-answer log-probability, defines the remaining gap with a small offset, sets the state value to the log ratio of the initial gap to the current gap, and rewards each tool call by the one-step temporal-difference change in that value, with a K-step backup and a terminal outcome anchor. Because the one-step changes telescope, redundant tool calls cannot inflate cumulative credit. The paper reports that this den","pith_inferences":["Inference: the same log-ratio TD credit should apply to any task with a compact verifiable answer, such as grounded single-answer tool use; the paper only tests search, so the breadth is open.","Inference: the method's main risk is reward hacking by answer leakage into private reasoning; a controlled test of reference-model sensitivity to that leakage would tell whether the monotonicity premise holds.","Inference: the telescoping property doubles as a diagnostic tool for post-hoc analysis, pinpointing the exact tool call that destroyed a secured answer, which the paper's qualitative examples illustrate without formalizing."],"forward_implications":["Outcome-only RL for long-horizon agents can be made dense without a critic, step labels, or a strong judge, making verifier-plus-progress training practical at scale.","Base chat models already contain useful search primitives; a dense credit signal can organize them into effective multi-turn behavior without cold-start SFT or agentic mid-training.","The learned search strategy transfers across corpora, retrieval engines, and languages, suggesting the dense signal is not merely overfitting a closed corpus.","Training curves start improving earlier and converge faster, indicating turn-level credit accelerates long-horizon learning rather than only rescaling final rewards."],"fun_headline_variants":["Reward each tool call from the gold-answer gap","TRACE: dense credit for long-horizon agents","Frozen model confidence becomes per-tool reward","No critic needed: turn-level rewards lift scores 5x"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a transcript with relevant evidence makes the gold answer easier for a fixed reference model to predict; if that monotonicity fails, the dense TD rewards are misaligned.","fun_headline_variants_meta":{"raw":{"variants":["Reward each tool call from the gold-answer gap","TRACE: dense credit for long-horizon agents","Frozen model confidence becomes per-tool reward","No critic needed: turn-level rewards lift scores 5x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000146,"raw_usage":{"total_tokens":1063,"prompt_tokens":833,"completion_tokens":230,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":173}},"tokens_in":577,"tokens_out":230,"duration_ms":3106,"temperature":1.0,"reasoning_tokens":173,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T03:06:00.694197+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a rollout in which the policy copies the gold answer into its private reasoning before any search and check whether the frozen reference model's gold-answer log-probability spikes; if it does, the monotonicity premise fails. Alternatively, run the method on a long-horizon task with long or open-ended outputs, which the paper identifies as outside its current scope, and see whether turn credit tracks real progress.","supporting_citations":[],"review_version":1}