{"id":"ad35bd78-53be-4c6d-b390-ee8df63923fa","arxiv_id":"2607.25308","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CAST converts a game solver's per-action cost-to-go changes into turn-level RL credits for LLM agents and reports gains over outcome-only RLVR on three games plus zero-shot transfer to ALFWorld and WebShop.","lead":"CAST asks a game solver to score each move an LLM agent makes, turning \"how much closer to winning did that move get\" into a per-turn training signal. On three games it reports the best results among trained baselines, plus the best zero-shot transfer to two everyday agent benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2.1's equivalence fails for the deployed deterministic solvers: as τ→0 the cross-entropy coefficient β→0 while the CAST update stays finite, so the update is reward shaping, not distillation.","rationale":"The reader's weakest_assumption identifies Assumption C.1 (soft-optimal solver) as the key hinge, and my analysis agrees: the theorem is only valid for a softmax/Boltzmann teacher, while the actual solvers are deterministic. This is not a manufactured objection; the paper itself concedes in C.5.7 (A1) that the distillation interpretation degrades when the solver deviates from soft optimality. The concern is load-bearing because the paper's abstract and Section 2 present 'logit-free on-policy distillation' as a central contribution. If the equivalence fails for the evaluated solvers, the method reduces to potential-based reward shaping—a sound but older idea (Ng et al., 1999). The empirical results remain interesting and the abstractions (asinh, RMS, combining with GRPO) may still be useful, which is why this is a CONDITIONAL rather than a REJECT. I agree with the reader's overall assessment; my concrete test provides a direct way to verify whether the theoretical equivalence actually describes the implemented update. The reader's other concern about Assumption C.3 (GRPO unbiasedness) is also real but secondary: it affects only the task-return term, and the paper's main novel signal is the shaping term. The lack of error bars and code verification is a valid reproducibility concern but does not pinpoint a specific technical flaw as sharply as the soft-optimal mismatch does.","tokens_in":27287,"tokens_out":12867,"duration_ms":143536,"concrete_test":"On Sokoban, instrument one training batch and compute the actual CAST parameter update Δθ_CAST from Eq. 8. For several temperatures τ ∈ {0.01, 0.1, 1, 10}, define a Boltzmann teacher π_τ(a|s) ∝ exp(Q(s,a)/τ) from the solver's Q-values, set β = ατ/(√2(RMS+ε)), and compute the gradient ∇θ J(θ) of Eq. 17 (with the frozen-visitation surrogate). Measure the cosine similarity between Δθ_CAST and ∇θ J(θ) over a fixed parameter subset. If no τ yields similarity near 1 (e.g., >0.9), the theorem's equivalence does not describe the deployed update, confirming the mechanism is reward shaping rather than logit-free distillation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central theoretical claim is that maximizing the CAST advantage equals on-policy distillation (Theorem 2.1, Eq. 9). This rests on Assumption C.1 (soft-optimal solver, Eq. 14), which gives A^{π_Solver} = τ log π_Solver (Lemma C.5). But the three deployed solvers—weighted A* for Sokoban, multi-source reverse BFS for Rush Hour, and a peek-free CSP solver for Minesweeper—are deterministic optimal planners. For a deterministic policy, π_Solver(a|s) is 1 for one action and 0 for all others; it cannot be written as exp(Q/τ)/Z for any finite τ>0. The paper's own C.5.7 (A1) concedes the 'distillation interpretation degrades proportionally,' and Prop. C.12 identifies the actual mechanism as potential-based reward shaping (Ng et al., 1999). Moreover, in the deterministic limit the identity degenerates: for any suboptimal action with Q-gap Δ>0, log π_Solver ≈ -Δ/τ diverges, while the CAST update α h(Ã) remains O(1) and independent of τ. The theorem's coefficient β = ατ/(√2(RMS+ε)) scales linearly with τ, so the claimed cross-entropy gradient is not identifiable from the implemented update. Thus the headline 'logit-free OPD' is a statement about a hypothetical Boltzmann teacher, not about the solver actually used. The empirical method is sound but is explained by a well-known mechanism, substantially weakening the novelty of the theoretical contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CAST (Credit Assignment from Solver Teachers) augments GRPO/DAPO policy-gradient training of LLM game agents with a turn-level scalar signal derived from a game solver's state-value change: for each sampled action the agent compares the solver's cost-to-go before and after, applies asinh compression and batch RMS normalization, and adds the result to the group-relative outcome advantage (Eq. 8). The paper claims (Theorem 2.1, Eq. 9) that, under a soft-optimal-solver assumption, this update is equivalent to on-policy distillation from the solver using only scalar values, with no teacher logits. Experiments on Qwen3-4B-Instruct-2507 across Sokoban, Minesweeper, and Rush Hour report the best Avg@4 among trained baselines on all three games for both in-domain and unseen difficulty (ID average 62.1 vs 44.7 for DAPO), faster attainment of DAPO's peak, and the highest zero-shot averages on ALFWorld (37.9) and WebShop (22.7). Ablations examine the weight, the asinh transformation, RMS normalization, a learned value network as solver, and overhead.","tokens_in":27550,"tokens_out":8921,"duration_ms":80212,"significance":"The empirical comparison is well controlled: same base model, same sparse 0/1 reward, same rollout budget, and a clean DAPO-backbone ablation; the gains are consistent across games and transfer. The paper ships code and states its assumptions explicitly, which is commendable. If the theoretical equivalence held, the logit-free distillation result would be a significant contribution to RLVR. However, as detailed below, the theorem's key assumption is not satisfied by the deployed deterministic solvers, and the actual mechanism is potential-based reward shaping (Ng et al., 1999), which is standard and policy-invariant but not the claimed distillation novelty. The empirical contribution remains publishable if the theoretical claims are reframed honestly.","major_comments":[{"comment":"Assumption C.1 (Eq. 14) requires the solver to be a soft-optimal policy with finite temperature τ>0, which yields the identity A^{π_Solver}=τ log π_Solver (Lemma C.5). The three deployed solvers are deterministic optimal/near-optimal planners (Appendix C.3: weighted A*, multi-source reverse BFS, peek-free CSP search). A deterministic policy cannot be expressed as exp(Q/τ)/Z with finite Q-values and τ>0; as τ→0, the right side of Lemma C.5 diverges for any suboptimal action (approximately -Δ/τ), while the implemented update α h(Ã_{i,t}) in Eq. 8 is O(1) and contains no τ. Hence the coefficient β in Eq. 9 is not identifiable from the implemented update, and the 'logit-free on-policy distillation' interpretation does not apply to the actual experiments. The paper itself concedes this in (A1) of Appendix C.5.7 and then identifies the real mechanism as potential-based reward shaping (Prop. C.","section":"§2.2 Theorem 2.1 / App. C.5, Assumption C.1"},{"comment":"Assumption C.3 states that GRPO's group-relative return (Eq. 2) is an unbiased estimator of the task advantage up to a positive scaling constant. This is asserted, not proved. With G=8 rollouts per prompt and 0/1 sparse returns, any group in which all or none of the trajectories succeed has σ_R=0, so every member receives advantage 0 regardless of which actions were taken; the resulting gradient estimate is zero even when the true policy gradient is nonzero. Thus Eq. 2 is not an unbiased estimator of A^{π_θ}_{task} in finite samples. Step 2 of the proof of Theorem 2.1 depends on this identification. Please either provide a rigorous finite-sample statement (with the exact conditions under which unbiasedness holds), or weaken the theorem to an approximation and validate the approximation empirically (e.g., by comparing to a learned critic or Monte-Carlo advantage estimator).","section":"App. C.5.2, Assumption C.3 / Eq. 2"},{"comment":"Proposition C.12 interprets the solver signal as potential-based reward shaping (Ng et al., 1999) and leverages that theorem's optimal-policy invariance. However, Eq. 5's treatment of dead states is not a potential difference: when N(s_{t+1})=∞, the paper caps the transition at the finite penalty -N(s_t). This cap is not of the form Φ(s_t)-Φ(s_{t+1}) for any well-defined potential Φ, so the shaping invariance guarantee does not literally hold for dead-state transitions. Those transitions are exactly the large-magnitude signals that the asinh compression is designed to dampen (Fig. 4), so the discrepancy is not negligible. The shaping interpretation and the PDL-based bound of Eq. 23 should be stated for the uncapped signal, with a separate analysis (or at least an explicit acknowledgment) for the capped version used in training.","section":"§2.2 Eq. 5 / Prop. C.12"}],"minor_comments":[{"comment":"Table 1 reports means over 3 runs without standard deviations or significance tests. Given the headline 'best in every game,' please add error bars or confidence intervals.","section":"Table 1"},{"comment":"Section 2.2 defines N(s) as the minimum work to reach a win, but Minesweeper's solver (Appendix C.3) is a deterministic peek-free effort estimate K(s), not an optimal cost-to-go; this mismatch should be acknowledged where Eq. 4 is introduced.","section":"§2.2, Eq. 4"},{"comment":"Eq. 9 writes J(θ) as a fixed objective, but β contains the batch-level RMS_B(g), which is data-dependent. The 'equality' in Theorem 2.1 is therefore at best a per-batch surrogate; clarify this in the statement.","section":"Theorem 2.1, Eq. 9"},{"comment":"The claim that the student can surpass the solver is not tested; there is no comparison between CAST's success rate and the solver's own success rate on the evaluation tiers. If this is a contribution, add such a comparison.","section":"§2.2, 'Why the student can surpass the solver'"},{"comment":"Minor text artifacts: ' /envel⌢pe' appears in author affiliations; the citation 'Shridhar et al.' is missing a year and venue; several other references have inconsistent formatting.","section":"Front matter / references"}],"recommendation":"major_revision","confidential_remarks":"The empirical core is solid and well controlled; the main risk is overclaiming. If the authors can rewrite the theory section to present potential-based reward shaping as the mechanism and reposition 'logit-free distillation' as a motivating approximation for near-optimal stochastic solvers, the paper would be acceptable. I would not accept the current version with Theorem 2.1 as stated, because the equivalence does not hold for the implemented method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the empirical core is real and worth engaging. CAST — adding a solver's cost-to-go difference as a turn-level advantage inside DAPO/GRPO — gives a consistent lift on all three games (ID average 44.7→62.1 vs. the DAPO backbone), reaches DAPO's peak 1.7–2.0× faster, transfers zero-shot to ALFWorld and WebShop, and the DQN variant shows the method doesn't depend on an exact solver. That alone is a useful contribution, and the ablations are well designed. Second, the theory as written overreaches. Theorem 2.1 claims the update is logit-free on-policy distillation, but that equivalence rests on Assumption C.1, which requires the solver to be soft-optimal with finite τ. The deployed solvers are deterministic planners — weighted A*, reverse BFS, peek-free CSP — so the softmax identity A = τ log π does not hold for them. In the deterministic limit τ→0, log π diverges while the CAST update stays finite. The paper itself concedes this in C.5.7 (A1) and then, in Prop C.12, identifies the actual mechanism as potential-based reward shaping (Ng et al., 1999). That is an older, sound mechanism and it explains the gains; the 'OPD' framing is packaging, not substance. Other soft spots are minor: Assumption C.3 asserts GRPO's group-normalized returns are unbiased, which isn't proven and isn't literally true for finite groups; the main tables lack error bars or significance tests; the code is a URL without a hash; and the Minesweeper solver is a heuristic completion cost rather than exact cost-to-go. None of these undercuts the central empirical finding. Bottom line: the paper deserves a serious referee. I'd send it out with a request to reposition the theory — drop the logit-free OPD language or prove it for deterministic solvers (unlikely), present CAST as reward shaping with a robustness analysis, and add variance estimates. After that revision it would be a solid paper.","headline":"Strong empirical method, over-sold theory: the solver-advantage signal works, but the 'logit-free distillation' claim reduces to reward shaping once you look at the solver being used.","tokens_in":28200,"tokens_out":2348,"would_cite":true,"duration_ms":25581,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a game solver's one-step cost-to-go change supplies cheap turn-level credit for LLM agents, and that maximizing this solver advantage is equivalent to logit-free on-policy distillation, yielding consistent gains over","keywords":["credit assignment","RLVR","on-policy distillation","solver advantage","process supervision","LLM agents","game solvers","zero-shot transfer"],"falsifier":"Record the distribution of one-step cost-to-go changes A = N(s_t) − N(s_{t+1}) over a CAST training run. If a substantial share of steps have |A| > 1 — for instance, transitions into dead states with penalty −N(s_t) — Assumption C.2 fails and the constant-beta distillation equivalence of Theorem 2.1 is quantitatively inaccurate; the method would then have to be explained by the robustified step-dependent beta or by reward shaping rather than by logit-free distillation.","tokens_in":26977,"feed_emoji":"🎮","tokens_out":8666,"duration_ms":81101,"temperature":0.7,"pith_summary":"Long-horizon game agents trained with reinforcement learning from verifiable rewards (RLVR) face a credit-assignment problem: a terminal win/loss signal does not say which actions mattered. CAST proposes to query a game solver on each state the LLM visits, define the solver advantage as the one-step drop in the solver's cost-to-go, and add that scalar as dense turn-level credit alongside the trajectory reward. The paper's theoretical claim is that under a soft-optimal solver assumption this update is equivalent to on-policy distillation from the solver without teacher logits, because the scalar advantage already encodes the teacher's log-preference. Empirically, CAST outperforms trained baselines on Sokoban, Minesweeper, and Rush Hour under both in-domain and unseen-difficulty evaluation and attains the best zero-shot averages on ALFWorld and WebShop. If correct, the result means process supervision for LLM agents can come from existing solvers or learned value functions at negligible training overhead.","feed_headline":"Per-turn solver signal lifts LLM game agents above outcome-only RL","feed_subtitle":"Per-turn solver credit beats outcome-only RL in three games and transfers to new domains.","key_machinery":"The central object is the shifted solver advantage, eA(s_t,a_t) = N(s_t) − N(s_{t+1}), where N(s) is the solver's cost-to-go: how much work remains to win. Each LLM-sampled action is scored this way, then passed through asinh compression and batch RMS normalization to tame extremes and align scales across games. The load-bearing identity A = τ log π_solver(a|s) converts the scalar advantage into the teacher's log-preference, making the per-step signal a cross-entropy distillation penalty in the implicit objective while the GRPO outcome advantage anchors task return.","core_discovery":"The central discovery is that the one-step drop in a solver's cost-to-go, N(s_t) − N(s_{t+1}), is a usable turn-level credit signal. The paper proves that, under a soft-optimal solver with small advantages, the update maximizing this shifted solver advantage equals the gradient of task return minus a cross-entropy to the solver. This is logit-free distillation: a single scalar carries the teacher's log-preference, so no logits are needed. The closed-form optimal policy tilts the solver distribution by the task advantage, letting the student surpass the solver. Empirically, CAST gets the best Avg@4 on every game in both settings, reaches DAPO's peak in 1.7–2.0x fewer steps, and transfers zero","pith_inferences":["The practical mechanism may be more modest than the headline: for the deterministic near-optimal solvers actually deployed, the paper's own Proposition C.12 identifies the update as potential-based reward shaping, and the distillation interpretation degrades proportionally. The theorem likely explains why shaping helps rather than proving the deployed algorithm distills.","A natural extension is to test whether the method works in non-game agentic benchmarks where no exact solver exists but a learned value model can be trained; the DQN variant on Rush Hour suggests the bottleneck is value quality, not solver exactness.","The zero-shot transfer results invite a controlled study of what game-trained agents actually learn — e.g., whether the WebShop gains come from a general 'check constraints before committing' behavior rather than from game-specific action patterns.","The paper's robustified-coefficient corollary implies that the asinh transformation is not just a stabilizer but an adaptive distillation-temperature schedule; one could test this by measuring beta_eff across training and correlating it with performance."],"forward_implications":["If the central claim holds, long-horizon LLM agents can be trained with dense turn-level credit using only scalar solver values — no teacher logits, learned reward models, or expensive search.","Training with the solver advantage reaches the outcome-only backbone's peak validation performance in 1.7–2.0x fewer steps and continues to a higher plateau on Sokoban, Minesweeper, and Rush Hour.","The benefit transfers zero-shot: game-trained CAST agents beat all trained baselines on ALFWorld and WebShop, suggesting general planning abilities rather than game-specific memorization.","A learned value network can replace the exact solver and retain most of the gain, so the mechanism should apply wherever a reliable state-value estimate exists.","Because the implicit objective is cross-entropy plus a task-return tilt, the student is not restricted to imitating the solver; in the closed-form solution the task advantage lets the student exceed the teacher."],"fun_headline_variants":["No logits needed: solver's cost-change gives LLM agents turn-level credit","Solver value drop per move teaches LLM agents, outperforms sparse reward RL","One scalar per turn: solver's cost-drop signals better than outcome-only RL","Solver's per-turn value drop: logit-free distillation that beats outcome-only RL","Turn-level solver credit improves LLM agents in Sokoban, Minesweeper, Rush Hour"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the solver acts as a soft-optimal teacher with a positive temperature, so its advantage equals a scaled log-probability; the deployed deterministic near-optimal solvers sit at the hard-max limit, and the paper concedes the distillation interpretation degrades proportionally.","fun_headline_variants_meta":{"raw":{"variants":["No logits needed: solver's cost-change gives LLM agents turn-level credit","Solver value drop per move teaches LLM agents, outperforms sparse reward RL","One scalar per turn: solver's cost-drop signals better than outcome-only RL","Solver's per-turn value drop: logit-free distillation that beats outcome-only RL","Turn-level solver credit improves LLM agents in Sokoban, Minesweeper, Rush Hour"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001215,"raw_usage":{"total_tokens":4838,"prompt_tokens":750,"completion_tokens":4088,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":3979}},"tokens_in":494,"tokens_out":4088,"duration_ms":26395,"temperature":1.0,"reasoning_tokens":3979,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:51:18.443440+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the distribution of one-step cost-to-go changes A = N(s_t) − N(s_{t+1}) over a CAST training run. If a substantial share of steps have |A| > 1 — for instance, transitions into dead states with penalty −N(s_t) — Assumption C.2 fails and the constant-beta distillation equivalence of Theorem 2.1 is quantitatively inaccurate; the method would then have to be explained by the robustified step-dependent beta or by reward shaping rather than by logit-free distillation.","supporting_citations":[],"review_version":1}