{"id":"a621b48a-eb29-43ce-aae1-60734ed881c5","arxiv_id":"2608.03223","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ADRS converts a frozen model's privileged skill-conditioned token scores into calibrated, return-gated reward shaping for agentic reinforcement learning, reporting consistent long-horizon gains.","lead":"This paper proposes ADRS, a training method that lets an LLM agent re-score its own previous decisions using extra task skills during training, and then uses those scores to shape the reward before policy updates. The goal is denser per-token feedback for long-horizon agent tasks, and the paper reports gains on household, web, and search agent benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main table and appendix report contradictory Search/ALFWorld numbers for the same 150-step setting, undermining the SOTA claim.","rationale":"The reader's weakest_assumption concerns the TVA gate's batch-local covariance. That is a reasonable concern but not the most load-bearing: the paper includes ADRS-no-TVA variants that still outperform baselines, so the core 'ADRS improves performance' claim does not depend on the gate. The most load-bearing issue is the direct numerical contradiction between Table 2 and Table 10 for the same 150-step comparisons. Table 10 says Search step-150 ADRS-global is 38.3 vs GRPO 38.4, while Table 2 says ADRS achieves 45.0 vs GRPO 36.4; Table 10 says ALFWorld primary ADRS beats a local GRPO of 84.4 by 10.1 points, while Table 2 lists GRPO at 75.0. The text explicitly identifies the primary Search row as the global-normalized η=0.05 run, so the discrepancy cannot be explained away as a different configuration. Table 7's dev-set numbers (≈36.9 at step 150 for η=0.05) further suggest the Table 2 value of 45.0 is not reproducible from the stated setup. Because the headline claim is 'new state-of-the-art across all three benchmarks,' an unresolved contradiction in the supporting table is fatal to the paper as written. Propositions 1–3 appear mathematically correct, and the ALFWorld/WebShop gains may be real, but the current manuscript does not present a coherent, verifiable set of results. The verdict should therefore be REJECT pending reconciliation; if the authors can demonstrate that 45.0 is the correct matched Search number and explain the 84.4 baseline, a CONDITIONAL acceptance could be reconsidered.","tokens_in":22721,"tokens_out":5658,"duration_ms":56445,"concrete_test":"Re-run or inspect the training/evaluation logs for Qwen2.5-3B on Search at step 150 using the primary ADRS configuration (global normalization, η=0.05) and the paired GRPO baseline, evaluating on the full 51K validation set; report the macro-average. Also run the same GRPO configuration used for Table 2 and the 'local GRPO' of Table 10 on ALFWorld to determine which baseline (75.0 or 84.4) is correct. If Search ADRS-global is 38.3 and GRPO is 38.4, the SOTA claim for Search is false; if ALFWorld GRPO is 84.4, the 94.5 result is a smaller gain than claimed.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—new SOTA on all three benchmarks with Qwen2.5-3B—rests on Table 2 (Search 45.0, ALFWorld 94.5, WebShop 87.5/76.6). But Table 10 reports matched fixed-step comparisons that directly contradict these: Search Step 150: ADRS-global 38.3 vs GRPO 38.4 (delta −0.1), i.e., no gain; ALFWorld Step 150: primary ADRS 94.5 vs a 'local GRPO' 84.4, whereas Table 2 lists GRPO at 75.0 for the same common 150-step comparison. Table 7's development-set sweep for the described primary Search config (global normalization, η=0.05) also tops out around 36.9 at step 150, not 45.0. These are not different ablations: the text states the primary 3B Search row is the global-normalized η=0.05 run. The appendix therefore contradicts the main table on the exact quantities used to support 'consistent improvements' and 'new state-of-the-art.' If Table 10 is correct, ADRS fails to improve Search and the ALFWorld gain is against a different baseline, so the headline empirical claim is not self-consistent. This is more load-bearing than the TVA gate, since ADRS without TVA still shows gains and the gate is only a secondary component.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ADRS, a reward-shaping layer for multi-turn LLM agents. After a skill-free rollout, the same frozen policy snapshot re-scores the realized tokens under a task-matched privileged skill. The method (i) centers and scale-normalizes teacher log-probabilities within each interaction step, (ii) computes a scalar 'Teacher Value Advantage' (TVA) gate from the within-group covariance between teacher confidence and realized return, and (iii) adds the gated, zero-sum token signal to the base token rewards before the GRPO/GiGPO advantage operator, so that the privileged signal enters the native reward-to-advantage-to-policy path rather than a separate auxiliary loss. The paper reports new state-of-the-art numbers on ALFWorld, Search-based QA, and WebShop with Qwen2.5-3B, plus ablations on GiGPO, reduced data, unseen tasks, and longer training. Three propositions are stated and proved: TVA sign equals within-group confidence-return covariance, the stepwise teacher reward is zero-sum, and the ADRS token path is locally first-order equivalent to a detached token auxiliary coefficient at the behavior-policy point.","tokens_in":23083,"tokens_out":6181,"duration_ms":61701,"significance":"If the empirical results were self-consistent, the paper would make a useful contribution. The idea of converting privileged self-distillation scores into pre-advantage reward shaping is clean, the zero-sum representation is a nice algebraic property, and the three propositions are correct though modest and local. The ablations showing gains beyond TVA and on a GiGPO backbone are informative, and the authors explicitly acknowledge the batch-local, non-invariance scope of their identities. However, the headline empirical claim is not currently credible because the main table and the appendix report contradictory numbers for the same 150-step setting, especially for Search-based QA. The claimed Search improvement is tiny (+0.4 points in Table 2, -0.1 points in Table 10), and the appendix is the only place where fixed-step comparisons are given. Before the SOTA claim can be accepted, the numerical inconsistencies must be resolved and the robustness of the remaining gains established with variance information.","major_comments":[{"comment":"Table 2 reports Qwen2.5-3B Search macro-average 45.0 for ADRS, a +0.4 improvement over the strongest baseline (GRPO+OPSD, 44.6). Table 10 reports Search Step 150: GRPO 38.4, ADRS-global 38.3, delta -0.1. §A.5 states that the primary 3B Search row is the global-normalized η=0.05 run, and Table 7 shows that configuration reaching only 36.9 at step 150 on the development set (peak 38.1). These numbers cannot describe the same reported run. If Table 10 is the correct full-evaluation fixed-step comparison, ADRS does not improve Search over GRPO and the new-SOTA claim in §5.2 is false. If Table 2 is correct, Table 10 is mislabeled or uses a different protocol. This must be resolved before the headline claim can be evaluated.","section":"§5.2, Table 2 vs. Appendix C.2, Table 10; §A.5; Table 7"},{"comment":"The same ALFWorld comparison is reported with different baselines. Table 2 lists GRPO at 75.0 and SDAR at 84.4 for Qwen2.5-3B at 150 steps; the text says ADRS improves by 10.1 points over the strongest baseline, which is actually SDAR (84.4→94.5). Table 10, however, labels the reference as 'Local GRPO 84.4' and gives the same +10.1 improvement over it. Unless 'Local GRPO' is a matched-configuration run that differs from the GRPO row of Table 2, this is a direct contradiction. The reader cannot tell which baseline is being compared, and the wording in §5.2 ('strongest baselines') is therefore ambiguous.","section":"Table 2 vs. Table 10, ALFWorld row"},{"comment":"All headline numbers are single runs with no error bars, seeds, or variance estimates, despite small claimed differences (e.g., +0.4 points on Search). In addition, the configuration is selected per benchmark after observing results: Table 8 shows global vs. per-sequence normalization was chosen because 'global normalization is stronger on ALFWorld and full Search, while per-sequence normalization is stronger on WebShop,' and §A.5 states the Search row is the η=0.05 run selected from the development sweep. With only a 0.4-point margin in Table 2 and a -0.1-point margin in Table 10, this is a multiple-comparison problem. Report seeds, error bars, and the full configuration-selection protocol so the 'consistent gains' claim can be assessed.","section":"§5.2, §A.5, Table 8, Table 9"},{"comment":"Proposition 1 correctly identifies d_g as a within-group covariance of soft confidence and return. However, the same batch returns are used both to compute the gate and to define the rewards/advantages being gated, so the 'association' is true by construction for that batch. The paper provides no out-of-sample validation that this batch-local covariance is a reliable signal across held-out steps, tasks, or training iterations. If the covariance is noise, the gate degenerates to a random per-group scaling. I would ask for a temporally or cross-validated version of the reliability claim, or a clear statement that TVA is a heuristic whose usefulness is established only by the ablation results.","section":"§4.2, Eqs. (4)-(6); Proposition 1"}],"minor_comments":[{"comment":"Typo: 'available at the following the link' should be 'available at the following link'.","section":"Abstract"},{"comment":"The axis labels in Figure 10 appear garbled, with multiple trailing y-tick sequences ('50 75 100 125 150' etc.) that do not correspond to a single coordinate axis. Please regenerate the figure with clean axes.","section":"Figure 10"},{"comment":"Each table should state in the caption whether the evaluation is on the development subset or the full 51K set, and at which optimizer-step cadence (e.g., every 15 vs. every 75 steps). Currently Table 7's development-set numbers and Table 2's full-set numbers can be confused.","section":"Tables 7 and 10"},{"comment":"The proof refers to an 'exact old-logprob condition' but the statement of Proposition 3 does not define this condition explicitly. Add it to the proposition statement for readability.","section":"Proposition 3"},{"comment":"The sentence 'Compared with the strongest baselines ... +10.1 percentage points on ALFWorld' should name the baseline (SDAR, not GRPO) explicitly, since the GRPO row of Table 2 is 75.0 and the SDAR row is 84.4.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the main-table/appendix inconsistency. If Table 10 is the correct full-evaluation result, then the Search SOTA claim is unsupported and the 'consistent improvements' claim fails for one of the three benchmarks; the paper would need to be substantially reframed. If Table 2 is correct, the appendix must be corrected. I would also ask the editor to ensure the authors disclose whether the 'Local GRPO' baseline in Table 10 is a re-run or the same GRPO as in Table 2. The TVA circularity concern is secondary but worth requesting an out-of-sample test. The method and theory are interesting enough to merit revision rather than outright rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a plausible incremental method with a real empirical claim, but the paper has an internal inconsistency in its headline numbers that the authors need to fix. If Table 10 is right, the Search gain vanishes and the ALFWorld gain is against a different baseline.\n\nWhat's actually new: the specific combination of within-step centering/normalization, the batch-local TVA gate, and pre-advantage injection is not present in the OPSD/SDAR/RLSD line. The three propositions are correct. Proposition 2's zero-sum representation and Proposition 3's local equivalence to a detached auxiliary objective are honest and useful framing. The ALFWorld and WebShop results show sizable gains over GRPO and SDAR, and the ablations on data efficiency, transfer, and 300-step training suggest the mechanism does something.\n\nNow the soft spots, in order of severity. The stress-test concern holds up on reading. Table 2 reports Qwen2.5-3B Search at 45.0 vs GRPO 36.4, and ALFWorld at 94.5 vs GRPO 75.0. Table 10's fixed-step comparisons list Search step 150 as ADRS-global 38.3 vs GRPO 38.4 (delta -0.1), and ALFWorld primary ADRS 94.5 vs a \"local GRPO\" of 84.4. The text says the primary Search row is the global-normalized eta=0.05 run, so the discrepancy can't be waved off as a different ablation. The appendix's dev-set sweep (Table 7) also tops out near 36.9 at step 150 for that config, not 45.0. This is load-bearing: the \"new state-of-the-art on all three benchmarks\" and \"consistent improvements\" claims rest on the Table 2 numbers, and the appendix contradicts them on the exact quantities. It may be a reporting error, but the paper doesn't acknowledge or explain it.\n\nSecond, the TVA gate is computed from the same batch it gates. The paper itself calls the proposition observational and batch-local. There is no out-of-sample check that the confidence-return covariance generalizes across steps, tasks, or iterations. If that association is noise, the gate degrades to a random per-group scaling. This is a legitimate concern, but secondary to the table inconsistency.\n\nThird, there are no error bars, and the per-benchmark normalization/eta choices are selected from validation. The matched-pair tables partially justify this, but it weakens the \"consistent\" claim.\n\nThe theory is modest but correct, and the authors are upfront about the limits of Proposition 3. The paper is worth a serious referee, because the empirical contradiction might be fixable and the underlying method deserves independent verification. If the Search numbers can't be reconciled, the SOTA claim should be withdrawn. My recommendation: send to peer review, but the authors must reconcile the tables before acceptance.\n\nFor the reading group, I'd bring it as a case study in how an appendix can invalidate a headline result. I wouldn't cite it until the discrepancy is resolved.","headline":"A plausible incremental method with a correct but modest theory, undercut by an internal inconsistency between the main table and the appendix that, if real, kills the Search SOTA claim and muddies ALFWorld.","tokens_in":23552,"tokens_out":4168,"would_cite":false,"duration_ms":40289,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ADRS converts training-only skill text into per-token rewards for agentic RL, reporting state-of-the-art ALFWorld success of 94.5% with Qwen2.5-3B without using skills at test time.","keywords":["reinforcement learning","self-distillation","LLM agents","reward shaping","credit assignment","GRPO","teacher value advantage"],"falsifier":"Train ADRS with the TVA gate replaced by a random per-group scalar drawn from the same marginal distribution, keeping everything else fixed; if the reported ALFWorld/WebShop deltas over GRPO persist, the gate is not the source of the gains. Alternatively, record the gate's signed $d_g$ on held-out trajectories across training steps: if it frequently disagrees in sign with held-out return differences on the units it gates, the reliability claim fails.","tokens_in":22630,"feed_emoji":"🤖","tokens_out":5154,"duration_ms":43594,"temperature":0.7,"pith_summary":"ADRS claims that a frozen policy's own rescoring of its student-generated tokens under privileged skill text can be converted into reliable token-level credit for agentic RL, by centering teacher scores within each step, gating them by the batch-local association between teacher confidence and realized return, and adding the result to base rewards before GRPO/GiGPO advantage construction. If right, this turns training-only procedural knowledge into a policy that needs no skills at test time and consistently beats sparse-outcome RL baselines across three long-horizon benchmarks. The paper reports new best results with Qwen2.5-3B: 94.5% ALFWorld success, 45.0% Search macro-average, and 87.5/76.6 WebShop score/success, with gains persisting under reduced data, unseen tasks, and 300-step training.","feed_headline":"Self-distilled rewards push LLM agent success to 94.5 percent","feed_subtitle":"Training-only skill text becomes per-token credit, beating sparse-reward baselines on ALFWorld, WebShop, and search QA without test-time ski","key_machinery":"The carrying mechanism is 'pre-advantage reward shaping' built from three pieces: the within-step centered and scale-normalized teacher score $\\hat q_{s,j}$, which satisfies $\\sum_j \\hat q_{s,j} = 0$ and makes teacher preferences comparable across interaction steps; the Teacher Value Advantage (TVA) gate $m_u = \\mathrm{sg}[\\sigma(\\tau \\tilde d_g)]$, where $d_g = \\mathrm{Cov}_g(\\alpha, R)/(\\bar\\alpha_g(1-\\bar\\alpha_g))$ measures whether units with higher teacher confidence also get higher realized returns; and the shaped reward $\\tilde r = r_{\\mathrm{base}} + \\eta m \\hat q$ added before the GRPO/GiGPO advantage operator. The zero-sum property (Proposition 2) ensures the trajectory-level outco","core_discovery":"The central claim is that privileged teacher scores become useful credit only when they are (1) made comparable across steps by removing step-level offset and scale, (2) discounted when their within-group confidence-return covariance is low via the Teacher Value Advantage gate, and (3) injected into the native reward-to-advantage path rather than a separate auxiliary objective. In ADRS the teacher reward $r_{s,j}^T = \\eta m_{s,j} \\hat q_{s,j}$ is added to the base token reward before advantage estimation; because the centered signal is zero-sum within a step (Proposition 2), it preserves trajectory ordering and only redistributes update strength across tokens. Proposition 1 shows the gate is","pith_inferences":["The TVA gate's dependence on a batch-local covariance is the part most likely to behave differently under distribution shift; a natural test is to log $\\mathrm{Cov}(\\alpha, R)$ on held-out batches and check whether its sign and magnitude predict held-out returns before trusting it as a reliability measure.","The zero-sum shaping signal is deliberately not potential-based, so unlike classical potential-based shaping it does not come with a policy-invariance guarantee; the paper only claims the sampled-trajectory ordering is preserved, and it would be worth checking whether the shaping survives discounting or off-policy reuse.","One could extend ADRS to use the teacher's token-level confidence directly as a per-token $\\alpha$ rather than a unit-level average, turning the gate into a richer token-level reliability map; the current design broadcasts the gate to all tokens in a unit.","The action-object diagnostic suggests the token modulation concentrates contrast on task-bearing tokens; a sharper test would be to measure whether the same separation appears on failed trajectories, where the teacher signal is most needed."],"forward_implications":["On the reported benchmarks with Qwen2.5-3B, ADRS raises ALFWorld success from the strongest baseline's 84.4% to 94.5%, Search macro-average from 44.6% to 45.0%, and WebShop score/success from 85.0/68.0 to 87.5/76.6, giving new state-of-the-art numbers under the SDAR evaluation setup.","The gains hold when the RL backbone is changed from GRPO to GiGPO, when training data is cut to 60% of the full set, and when evaluated on an unseen ALFWorld split, suggesting the credit signal rather than a specific optimizer produces the advantage.","Skills are used only during post-rollout rescoring; the deployed policy receives no skill text, so the reported improvements are not a test-time prompting artifact.","Because ADRS is a credit-construction layer, it composes with the standard clipped actor update and existing KL/entropy terms, meaning it can be dropped into current agentic RL training loops with only a teacher-scale hyperparameter $\\eta$ to set."],"supporting_citations":[{"why":"Establishes the on-policy self-distillation recipe (same model, different contexts) that ADRS extends with reward shaping.","marker":"[40]"},{"why":"Provides the closest baseline, the skill provider, and the benchmark/evaluation protocol that ADRS follows.","marker":"[15]"},{"why":"Defines the GRPO group-relative advantage backbone that ADRS augments with the teacher reward.","marker":"[24]"},{"why":"Supplies the GiGPO step-aware credit backbone and the L3 anchor-state granularity used by step-level TVA.","marker":"[5]"},{"why":"Contributes the ALFWorld embodied-control benchmark that measures long-horizon procedural success.","marker":"[25]"},{"why":"Contributes the WebShop web-navigation benchmark with score and success metrics.","marker":"[38]"},{"why":"Defines the Search-based QA protocol and four-tool-turn evidence-gathering setting used for evaluation.","marker":"[9]"},{"why":"Supplies the potential-based shaping theory that ADRS contrasts against to justify its zero-sum two-path credit construction.","marker":"[19]"}],"fun_headline_variants":["Self-distilled rewards turn sparse into dense LLM agent credit","ADRS: Calibrated teacher scores for token-level RL credit","Confidence-gated teacher rewards boost long-horizon agentic RL","Privileged skills become per-token credit in self-distilled RL","Return-aware reward shaping improves LLM agents on long tasks"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The method assumes that the empirical within-group covariance between teacher confidence and realized return, computed on the very batch being trained on, is a reliable indicator of when the teacher signal should be trusted.","fun_headline_variants_meta":{"raw":{"variants":["Self-distilled rewards turn sparse into dense LLM agent credit","ADRS: Calibrated teacher scores for token-level RL credit","Confidence-gated teacher rewards boost long-horizon agentic RL","Privileged skills become per-token credit in self-distilled RL","Return-aware reward shaping improves LLM agents on long tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00053,"raw_usage":{"total_tokens":2419,"prompt_tokens":803,"completion_tokens":1616,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1525}},"tokens_in":547,"tokens_out":1616,"duration_ms":12618,"temperature":1.0,"reasoning_tokens":1525,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:10:59.749885+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ADRS with the TVA gate replaced by a random per-group scalar drawn from the same marginal distribution, keeping everything else fixed; if the reported ALFWorld/WebShop deltas over GRPO persist, the gate is not the source of the gains. Alternatively, record the gate's signed $d_g$ on held-out trajectories across training steps: if it frequently disagrees in sign with held-out return differences on the units it gates, the reliability claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GiGPO step-aware credit backbone and the L3 anchor-state granularity used by step-level TVA."},{"cited_title":"WebShop: Towards scalable real-world web interaction with grounded language agents","cited_arxiv_id":null,"evidence_quote":"Contributes the WebShop web-navigation benchmark with score and success metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the potential-based shaping theory that ADRS contrasts against to justify its zero-sum two-path credit construction."}],"review_version":1}