{"id":"2923f427-75fc-4e2f-b38a-c7ff8991d298","arxiv_id":"2607.27973","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Interleaving action-conditioned next-observation supervision with group RL on a shared LLM backbone consistently lifts long-horizon agent success over pure policy optimization.","lead":"TAPO adds next-observation prediction on the same LLM backbone during agent RL, reusing ordinary rollouts. It is a cheap plug-in that raises success on WebShop and ALFWorld without extra data or test-time cost.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Gains may reflect extra supervised updates on rollout text rather than action-conditioned transition knowledge; matched-step and non-causal controls are missing.","rationale":"The reader’s strongest claim is correctly scoped as a consistent empirical improvement under a no-extra-data interleaved recipe. The load-bearing soft spot is exactly the mechanism/fairness assumption they flag: gains attributed to internalized (s,a)→s' knowledge without matched gradient-step budgets or non-causal auxiliary controls. I independently checked Algorithm 1, I-default, 150-iteration protocol, Tables 3–4, and Figs. 3–5; they support efficacy and a mild dose-response but do not close the confound. No stronger internal inconsistency (math, circular eval, or broken baselines within the GRPO/GiGPO grid) appears. Table 2 is already caveated. Capability-tax note is honest. Thus the reader’s CONDITIONAL verdict and medium correctness_risk stand; no upgrade to REJECT and no downgrade to unconditional ACCEPT. Concrete matched-step + shuffled-target runs would settle the concern cleanly.","tokens_in":17429,"tokens_out":739,"duration_ms":40096,"concrete_test":"On WebShop + Qwen2.5-1.5B + GRPO (same seeds/hyperparams as Table 1): (A) train vanilla GRPO to the same total optimizer steps and similar wall-clock as TAPO-I=4; (B) run TAPO but with L_TS targets replaced by randomly shuffled s_{t+1} from the same batch (or next-obs prediction conditioned on s_t only). If either control reaches success rates within ~2 points of TAPO-GRPO’s 66.2%, the transition-dynamics mechanism claim does not hold and the method reduces to extra supervised updates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that interleaving rollout-derived action-conditioned next-observation SL with GRPO/GiGPO improves agent success via better modeling of (s,a)→s' dynamics (Abstract; §4.1–4.2; contributions). Fairness of that claim rests on the auxiliary signal being specifically causal/transitional, not merely extra gradient steps or generic language modeling on observations already in the rollouts.\n\nAlgorithm 1 and I=4 imply ~25% more parameter updates than the pure-RL baseline at the same 150 iterations and same environment samples (Computing Details). Table 4 and Fig. 3 show full-process TS beats early-only and rises faster, and Table 3’s non-monotone I-sweep weakly argues against pure step-count, but neither matches total gradient steps/wall-clock of an extended GRPO run nor replaces L_TS with a non-causal control (e.g., predict shuffled s_{t+1}, or s_{t+1} from s_t alone without a_t). Fig. 4 (lower next-state PPL) and Fig. 5 (one CoT trace) are consistent with the mechanism story but do not isolate it. If a matched-budget or non-causal auxiliary closes the gap in Table 1, TAPO remains a useful regularizer whose “transition-aware” justification and plug-and-play generality are overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes TAPO, a post-training recipe for LLM agents that interleaves standard group-based policy optimization (GRPO or GiGPO) with an auxiliary action-conditioned next-observation prediction loss on a shared backbone, using only (s_t, a_t, s_{t+1}) triplets already present in RL rollouts. No extra expert data, environment samples, or inference-time modules are required. On WebShop and ALFWorld, with Qwen2.5-1.5B/7B and both GRPO and GiGPO, TAPO reports consistent gains in score/success rate over pure policy-optimization baselines (Table 1), with supporting analyses of alternation interval I (Table 3), full-process vs early-only transition supervision (Table 4), training dynamics (Fig. 3), next-state perplexity (Fig. 4), and a qualitative WebShop trace (Fig. 5). A limited reference comparison to Early Experience and RWML appears in Table 2.","tokens_in":17734,"tokens_out":1504,"duration_ms":35296,"significance":"If the gains hold under tighter controls, TAPO is a practically useful, low-overhead plug-in for agentic RL: it reuses existing rollouts, needs no separate world-model stage, and improves long-horizon success across two standard benchmarks, two model scales, and two algorithms. Strengths include multi-seed reporting (mean±std over 3 seeds), hyperparameter alignment with the GiGPO setup, an explicit algorithm box (Algorithm 1), prompt templates, PPL definition, and an anonymized code link. The connection to auxiliary-prediction RL and recent theory that goal-conditioned agents encode predictive structure is appropriate. The main scientific value is empirical and engineering rather than a new theoretical guarantee.","major_comments":[{"comment":"The central mechanistic claim—that gains come from internalized (s,a)→s' dynamics rather than extra supervised updates on rollout text—is not isolated. Algorithm 1 with default I=4 implies roughly one transition-supervision update per four RL updates (~25% more parameter updates at fixed 150 iterations and identical environment samples; Computing Details). Table 4 and Fig. 3 show full-process TS beats early-only and rises faster, and Table 3’s I-sweep is non-monotone, but neither matches total gradient steps/wall-clock of an extended pure-GRPO run nor replaces L_TS (Eq. 6) with a non-causal control (e.g., predict shuffled s_{t+1}, or s_{t+1} from s_t without a_t). Fig. 4 (lower PPL) and Fig. 5 (one CoT) are consistent with the story but do not establish causality. Without at least one matched-budget and one non-causal auxiliary control on WebShop (and ideally ALFWorld), the “transition-a","section":"§4.1–4.2, Eq. (6), Algorithm 1, Tables 3–4, Figs. 3–5"},{"comment":"Table 2 presents TAPO (93.6%) against Early Experience (82.8%) and RWML (90.1%) on ALFWorld with Qwen2.5-7B, but footnote 1 correctly notes mismatched objectives, schedules, and protocols. The main text still frames TAPO as “competitive with recent closely related” methods. Either move Table 2 to appendix as purely referential, or run a controlled re-implementation under a shared backbone, rollout budget, and evaluation protocol. As written, the comparison is not load-bearing evidence for superiority and risks overclaim.","section":"§5.3, Table 2"},{"comment":"Several Table 1 improvements are modest relative to reported variance (e.g., GiGPO 7B ALFWorld 90.8±1.3 → 93.6±1.4; GiGPO 1.5B ALFWorld 86.7±1.7 → 88.4±3.8). With only three seeds, overlapping intervals weaken the “consistently improves … across … all combinations” claim in the Abstract and §5.2. Please report pairwise significance tests or bootstrap CIs, or soften language where gaps are within noise, and clarify whether evaluation uses the same held-out task splits and decoding settings for all methods.","section":"Abstract, §5.2, Table 1"}],"minor_comments":[{"comment":"Appendix B.1 (GSM8K capability tax: base 59.1, GRPO 57.9, TAPO-GRPO 56.5) is important for the plug-and-play claim and is only briefly flagged in the Conclusion. A short main-text sentence and, if possible, one more OOD probe would help readers weigh the trade-off.","section":"Appendix B.1, §6"},{"comment":"Notation: f_θ(s_t, a_t) for next-observation prediction vs π_θ for the policy on the same θ is clear in prose but could be stated once as a shared LM head with different prompt formats (policy tags vs <prediction> tags).","section":"§4.1.2"},{"comment":"Fig. 3 axis labels appear as Unicode glyph runs in the manuscript source; ensure the camera-ready figures render readable “Training Steps” / “Episode Mean Reward” / “Eval Success Rate” labels.","section":"Figure 3"},{"comment":"Related work correctly distinguishes decoupled world-model pre-stages [19,20] from interleaved auxiliary TS; a one-sentence contrast with classical auxiliary-task RL (UNREAL, SPR) already cited in the introduction would tighten §2.3.","section":"§2.3"},{"comment":"Typos/style: “s(modI)” in Algorithm 1 line 15 should be the iteration counter; “F orty-second” in Ref. [14]; arXiv IDs and concurrent-work dates should be double-checked before camera-ready.","section":"Algorithm 1, References"}],"recommendation":"major_revision","confidential_remarks":"The empirical recipe looks real and useful; my major_revision is driven by missing mechanism controls and the soft Table 2 comparison, not by disbelief in the success-rate lifts. If the authors add matched-step and non-causal ablations and tone Table 2, this is close to a solid workshop/conference contribution. Scope fit for a serious ML journal is borderline—more systems/empirical agent RL than deep theory—but acceptable if controls land."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: TAPO is a practical agent-RL recipe, not a new theory of world models. They take the (s,a,s') tuples you already get from on-policy rollouts, run standard teacher-forced next-observation SL on the shared backbone every I policy steps, and report consistent lifts over GRPO and GiGPO on WebShop and ALFWorld for 1.5B and 7B Qwen2.5. No extra env samples, no expert demos, no test-time cost.\n\nWhat is actually new is the design point, not the auxiliary idea. UNREAL/DeepMDP/SPR and the concurrent Early Experience / RWML lines already own “predict the next observation.” TAPO’s contribution is interleaving that loss with group-based multi-turn RL on one θ, with a clear Algorithm 1, prompts, matched baseline hyperparameters, anonymous code, and a 2×2×2 empirical grid with three-seed error bars. The I-sweep, full-process vs early-only ablation, training curves, and next-state PPL drop are the right supporting checks. The capability-tax note on GSM8K is honest and small.\n\nSoft spots, in proportion. The stress-test lands: at I=4 you get roughly a quarter more parameter updates on the same rollouts, and there is no matched-step pure-RL run and no non-causal control (shuffled s' or s-only prediction). Fig. 4 and the single WebShop CoT are consistent with “transition awareness” but do not isolate it from “extra supervised updates on trajectory text.” Table 2 is explicitly reference-only and should not be read as a head-to-head. Scope is two text environments. None of that sinks the main claim as stated—gains over pure policy opt under a fixed recipe—but it does mean the “transition-aware” story is still partly interpretive.\n\nMath is standard GRPO/GiGPO plus NLL; citations are appropriate; circularity is low. This is for people building multi-turn agent post-training stacks who want a cheap regularizer. I would send it to referees. Engage if you care about agent RL practice; skim the mechanism section if you only care about world-model theory.","headline":"Clean plug-and-play recipe that reuses rollout (s,a,s') for next-obs SL interleaved with GRPO/GiGPO; gains look real, mechanism isolation is the soft spot.","tokens_in":18459,"tokens_out":580,"would_cite":true,"duration_ms":16578,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Reusing rollout transitions as next-observation prediction on a shared model improves LLM agent RL without extra data or test cost.","keywords":["LLM agents","reinforcement learning","transition supervision","next-observation prediction","group-based policy optimization","WebShop","ALFWorld","auxiliary objectives"],"falsifier":"Train a control that matches TAPO’s extra gradient steps and token budget but replaces next-observation prediction with a non-causal auxiliary loss (for example, reconstructing the current observation or predicting a shuffled next observation); if task success and next-state perplexity no longer improve together, the causal-transition story fails.","tokens_in":18231,"feed_emoji":"🔄","tokens_out":880,"duration_ms":23296,"temperature":0.7,"pith_summary":"Standard reinforcement learning for language-model agents mostly trains on sparse end-of-task rewards and ignores the dense, action-conditioned feedback the environment already returns after every step. This paper argues that those transitions are a free supervisory signal: by alternating ordinary policy updates with teacher-forced prediction of the next observation given state and action, on the same shared backbone and the same rollouts, the agent becomes more sensitive to action consequences and plans better over long horizons. The method, TAPO, is presented as a lightweight plug-in for existing group-based agent RL algorithms. On WebShop and ALFWorld, across 1.5B and 7B models and two policy optimizers, it consistently raises success rates and scores while adding no expert data, no extra environment samples, and no inference overhead. A sympathetic reader cares because long-horizon agent training is bottlenecked by sparse rewards; turning every interaction into both a policy sample and a dynamics sample is a cheap way to tighten that bottleneck.","feed_headline":"Same rollouts, next-state prediction: LLM agents plan better","feed_subtitle":"Alternating policy RL with free transition supervision lifts success on WebShop and ALFWorld at no extra test cost.","key_machinery":"TAPO: an alternating loop that, every I policy updates, reuses the same (state, action, next-state) triples from RL rollouts to minimize a teacher-forced next-observation loss on the identical model parameters that generate actions.","core_discovery":"The paper claims that interleaving standard group-based policy optimization with rollout-derived, action-conditioned next-observation prediction on a shared backbone systematically improves multi-step agent task performance over pure policy optimization, without requiring extra expert data, extra sampling, or any change at test time.","pith_inferences":["If next-observation supervision is the active ingredient, similar free gains should appear in other interactive LLM settings (tool use, multi-turn dialogue, code agents) wherever the environment returns a textual next state.","Matching total gradient steps with a non-causal auxiliary loss would cleanly separate “dynamics learning” from “extra supervised training,” a control the paper leaves open.","The mild general-capability tax suggests future schedules may need explicit retention objectives or selective transition sampling rather than uniform interleaving."],"forward_implications":["Existing group-based agent RL recipes (GRPO, GiGPO and similar) can be strengthened by periodic next-observation supervision on their own rollouts.","Long-horizon agents can acquire useful predictive knowledge of environment feedback without a separate world-model pre-training stage or expert trajectories.","Full-process interleaving beats using transition supervision only as an early warm-up.","Gains appear across model scales (1.5B and 7B) and both shopping and embodied household benchmarks.","The same recipe leaves a modest out-of-domain capability cost that is already partly present under pure policy RL."],"fun_headline_variants":["TAPO interleaves RL with next-observation prediction on shared backbone","Rollout-derived transition supervision improves LLM agent task success","Action-conditioned next-state prediction aids multi-step agent policies","Same rollouts train transitions and policy for better WebShop agents","Alternating policy opt and transition learning lifts ALFWorld results"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The performance gains come from the model internalizing genuine action-to-next-observation dynamics, not merely from receiving extra supervised gradient steps on the same trajectories.","fun_headline_variants_meta":{"raw":{"variants":["TAPO interleaves RL with next-observation prediction on shared backbone","Rollout-derived transition supervision improves LLM agent task success","Action-conditioned next-state prediction aids multi-step agent policies","Same rollouts train transitions and policy for better WebShop agents","Alternating policy opt and transition learning lifts ALFWorld results"]},"model":"grok-4.5","effort":"low","cost_usd":0.005376,"raw_usage":{"total_tokens":1431,"prompt_tokens":748,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":53764000,"prompt_tokens_details":{"text_tokens":748,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":613,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":748,"tokens_out":70,"duration_ms":11792,"temperature":1.0,"reasoning_tokens":613,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T21:42:18.487924+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train a control that matches TAPO’s extra gradient steps and token budget but replaces next-observation prediction with a non-causal auxiliary loss (for example, reconstructing the current observation or predicting a shuffled next observation); if task success and next-state perplexity no longer improve together, the causal-transition story fails.","supporting_citations":[],"review_version":1}