{"id":"04a274ed-1a3a-413b-9d33-26dab597850e","arxiv_id":"2412.00293","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Adaptformer extends LEAP-style energy-based planning with a learned intrinsic sub-goal curriculum and entropy-regularized stochastic policy, enabling generalization to multi-goal and other out-of-distribution missions.","lead":"Adaptformer is a new offline planner that combines a learned goal proposer, a GAN-style discriminator, and an entropy-regularized stochastic policy to solve long-horizon robot missions not seen during training. It reports up to 25% higher success than the prior LEAP method in multi-goal maze tasks and one case of learning from random walks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GA module's training signal is missing from Algorithm 1, so its ability to propose correct multi-goal sub-goals at test time is unsupported and load-bearing.","rationale":"I concur with the reader's identified weakest assumption: the GA module's transfer to unseen multi-goal configurations is the linchpin of the paper's central claim. My concern sharpens this by noting that the GA module's training objective (Eq. 3) is not implemented in Algorithm 1; the gradient updates only include the policy loss and discriminator loss. This means the paper does not actually demonstrate that the GA is trained to predict goals, making its test-time behavior even more ungrounded. The proposed experiment directly tests whether the GA can propose correct sub-goals for goal configurations outside its training distribution, using an oracle control to isolate the GA's contribution. The reader's verdict of CONDITIONAL is appropriate; no change is needed, but the authors should add this test and clarify the GA's training objective.","tokens_in":11886,"tokens_out":10532,"duration_ms":94688,"concrete_test":"Train Adaptformer on single-goal demonstrations in GoToObjMazeS4G1 with goals sampled only from the left half of the maze. Evaluate on two-goal tasks (GoToObjMazeS4G2) where both goals lie in the previously unseen right half. Measure success rate over at least 50 maps. As an oracle control, repeat the evaluation while conditioning the policy on the true goal positions (bypassing the GA). If Adaptformer with GA succeeds barely above chance while the oracle control succeeds at a high rate, the GA module fails to generalize to unseen goal configurations, invalidating a central claim. If both succeed, the GA transfer holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Adaptformer generalizes to multi-goal missions and unseen skills requires the Goal Augmentation (GA) module (Section IV-B) to propose correct sub-goals at test time for goal configurations never seen in training. This is the load-bearing assumption, yet it is not secured by the training procedure described. Equation (3) defines an objective for the GA module, max_ω log P(Ĝ|G_train; π_ω), but Algorithm 1 optimizes only L_π (Eq. 5) and L_D (Eq. 6); no term corresponding to Eq. 3 appears in the update rules. The text says ω is contained in θ, but the policy loss L_NLL conditions on S_g (sub-goals) and does not directly supervise the GA's proposed goals. Thus it is unclear how the GA is trained to output goal proposals that match the demonstration goals. At test time, the GA must output a set of multiple goal states even though it was only ever presented with single-goal demonstrations, requiring a compositional generalization that no mechanism in the paper ensures. If the GA proposes wrong waypoints, the energy minimization in Algorithm 2 will guide the agent to irrelevant states, and the multi-goal success rates would collapse. The paper's own ablation in Table II shows that removing goal conditioning reduces GoToObjMazeS4G2 success from 53% to 32%, confirming the GA is critical. Without an explicit training signal for the GA and an evaluation of its transfer to disjoint goal configurations, the central generalization claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adaptformer, an offline sequence-model planner that learns an energy-based heuristic over trajectories, augments goal conditioning with a learned Goal Augmentation (GA) module and a state discriminator that produces sub-goals, and performs planning via iterative Gibbs-style energy minimization with a deliberately stochastic policy. The authors claim up to 25% improvement over LEAP in multi-goal maze reachability, successful transfer from single-goal demonstrations to multi-goal missions, and the emergence of skills such as door opening and obstacle unblocking that were absent from the training demonstrations. Evaluation is conducted in BabyAI/Minigrid environments plus a hardware deployment on an AGILEX robot, with ablations and a random-demonstration experiment.","tokens_in":12170,"tokens_out":5435,"duration_ms":54281,"significance":"If the central claims are established, the paper would make a useful contribution to offline behavioral planning: the stochastic energy-minimization formulation and the idea of learning an intrinsic sub-goal curriculum are both interesting, and the reported generalization from single-goal training to multi-goal test missions would be a meaningful step beyond the deterministic, oracle-conditioned planning of LEAP. The paper deserves credit for including ablations, a LEAP-without-goal-conditioning baseline, a random-demonstration experiment, and a real-robot deployment. However, the main generalization claim currently rests on an underspecified Goal Augmentation module and on high-variance empirical results without statistical testing, so the significance is promising rather than conclusive.","major_comments":[{"comment":"Equation (3) defines a training objective for the Goal Augmentation module, max_ω log P(Ĝ|G_train; π_ω), but Algorithm 1 contains no term corresponding to this objective; the only updates are Lπ(θ) and LD(ϕ). The statement that \"ω is contained in θ\" does not explain how gradients from Eq. (5a) reach the GA, because L_NLL conditions on S_g rather than on the GA's proposals Ĝ. Since the multi-goal and out-of-distribution claims depend on the GA proposing correct waypoints at test time, the paper must specify the GA training signal and provide evidence that the learned proposals transfer to unseen goal configurations.","section":"Section IV-B/E, Algorithm 1"},{"comment":"The discussion states that \"our model demonstrates goal-unblocking capabilities and adapts to environments with closed doors, even without explicit demonstrations of door-opening actions,\" but this is contradicted by the KeyCorridorS3R3 row of Table I, where Adaptformer scores 18% versus LEAP's 21% and LEAP⊖GC's 16%. The authors should reconcile this contradiction or restrict the claim to the environments in which the numbers actually support it.","section":"Section V-D and Table I"},{"comment":"Success rates are reported as mean ± variance, but no confidence intervals or significance tests are provided; for example, GoToObjMazeS4G2 (53±16% versus 37±29%) and GoToObjMazeS4G2Close (48±18% versus 23±20%) have overlapping or very wide error ranges, so the \"up to 25% improvement\" claim is not statistically supported as reported. The authors should report standard errors or confidence intervals and, where possible, paired tests across the 50 maps and 150 runs.","section":"Table I and Section V-A"},{"comment":"The MultiRoomN2S4 row supporting the \"learning from random trajectories\" claim reports 71% versus 0% but gives no variance, no number of random demonstrations, no description of how many seeds were used, and no explanation of how a 100-step random walk without open/pickup/drop actions leads to a policy with 71% success. Because this row is the sole evidence for the sub-optimal-demonstration claim, the full protocol and repeated-seed statistics are needed.","section":"Section V-B (Randomly Collected Trajectory, Table I)"}],"minor_comments":[{"comment":"The λ1 update is written without an expectation over the dataset; it should average the entropy constraint violation over trajectories, rather than using a single timestep's entropy.","section":"Algorithm 1, line 13"},{"comment":"The notation \"{g_i}_{i=1}^N ∈ G_train ∼ T\" is confusing; the goal samples should be described as drawn from the goal sets appearing in the offline trajectories T.","section":"Section IV-B"},{"comment":"Algorithm 2 says \"a, s ∼ Eθ(T_i\\I)\" even though Eθ is a scalar energy, not a distribution; this should be \"a, s ∼ πθ(·|T_i\\I, S_g)\" or otherwise clarified.","section":"Section IV-F and Algorithm 2"},{"comment":"The energy function in Algorithm 2 adds the entropy term −λ1H(πθ) to the negative log-likelihood, whereas Section IV-D defines the energy as the sum of negative log-likelihoods only; the relationship between the constrained objective in Eq. (4) and the inference-time energy should be stated explicitly.","section":"Section IV-D and Algorithm 2"},{"comment":"The paper does not report values for the key hyperparameters λ1, λ2, β, planning horizon H, Gibbs iterations K, or the upper bound on goal proposals; these should be given in the text or an appendix to make the experiments reproducible.","section":"General reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The missing GA training signal is a genuine and load-bearing gap, not a stylistic issue; the central generalization claim cannot be evaluated until the authors specify how Eq. (3) is optimized and how the GA transfers to unseen goal configurations. The empirical contradictions and lack of statistical testing strengthen the case for major revision. I would be willing to review a revised version that addresses these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick read of Adaptformer. The genuinely new piece is the learned sub-goal curriculum: instead of an oracle goal like LEAP, they train a goal-augmentation module with a discriminator and entropy regularization to propose waypoints, and they evaluate the single-goal-to-multi-goal transfer. That's a real combination I haven't seen in this exact form. The multi-goal maze numbers are encouraging—up to 25% over LEAP on GoToObjMazeS4G2Close, and the random-walk-trained MultiRoom result (71% vs 0%) is the kind of thing that makes you sit up. The ablations are also useful: taking out goal conditioning, entropy, or the discriminator each drops performance, which supports the design.\n\nNow the soft spots, in proportion. The biggest one is the GA module. Equation (3) defines its objective, but Algorithm 1 never optimizes that term. The text says omega is inside theta, so the GA could be trained implicitly through the action NLL loss, but the paper doesn't say that, and it matters: the GA must propose a set of sub-goals for multi-goal missions from single-goal demonstrations. That's a compositional generalization claim backed by no mechanism and no direct evaluation. The ablation shows removing goal conditioning cuts GoToObjMazeS4G2 from 53% to 32%, so the whole framework leans on this unexplained piece.\n\nSecond, the statistics are thin. Variances are large (53±16 vs 37±29), no confidence intervals or significance tests, and many pairwise differences are within noise. Third, the abstract's door-opening/adaptation claim sits awkwardly next to KeyCorridorS3R3, where LEAP beats Adaptformer (21% vs 18%). That's not fatal, but it should be reconciled. Finally, no code is released and baselines are only LEAP variants; comparisons to DT or modern offline RL are borrowed from LEAP's paper, not run here.\n\nWho is this for? Researchers working on offline sequence-model planning and goal-conditioned RL. If the GA training is fixed and code appears, this could become a useful baseline. As it stands, the idea is promising but the central generalization claim is not established. I'd send it to review, definitely—the method is original enough and the random-walk result is worth checking—but I'd insist on code, significance testing, and a rewritten GA section. The limitations paragraph is honest and the overall thinking is coherent.","headline":"Promising planner idea with a load-bearing training detail left unexplained; worth a close look but not yet a reliable citation.","tokens_in":12711,"tokens_out":2932,"would_cite":false,"duration_ms":29042,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adaptformer plans through unseen multi-goal missions using only single-goal demonstrations.","keywords":["Adaptformer","sequence models","offline reinforcement learning","energy-based models","goal conditioning","sub-goal curriculum","long-horizon planning","generalization"],"falsifier":"Train Adaptformer on single-goal-reaching demonstrations in maps where all goals lie in the left half of the grid, then evaluate it in a maze where the only goals are on the right half. If the goal-augmentation module's top-ranked sub-goal proposals never enter the right half and the success rate drops to near zero, then the intrinsic sub-goal curriculum does not transfer to unseen goal configurations, and the central generalization claim is false.","tokens_in":11637,"feed_emoji":"🧭","tokens_out":5613,"duration_ms":44662,"temperature":0.7,"pith_summary":"Adaptformer is an offline planner that aims to complete long-horizon, multi-goal missions in environments never seen during training, using only demonstrations of single-goal reaching. Its central claim is that a masked-language-model energy function, guided by a learned sub-goal curriculum and an entropy-constrained stochastic policy, can generalize to out-of-distribution tasks, including skills like opening doors or moving obstacles that were never demonstrated. If true, this would mean autonomous agents can adapt to novel instructions at runtime without new data or reward engineering. The paper reports up to a 25% improvement in success rate over LEAP for multi-goal maze reachability, and shows the model completing tasks that LEAP never solves.","feed_headline":"Sequence planner beats state-of-the-art by 25% on multi-goal mazes","feed_subtitle":"A learned sub-goal curriculum and stochastic policy unlock skills like door-opening that never appeared in training.","key_machinery":"The central object is the energy-based trajectory model $E_\\theta(T) = \\sum_{t=1}^{H} -\\log \\pi_\\theta(a_t | T_{\\setminus t}, S_g)$, a masked-language-model-style pseudo-likelihood that assigns low energy to good action sequences. Three auxiliary mechanisms carry the generalization: the Goal-Augmentation module, an MLP that proposes sub-goals $\\hat{G}$ trained to maximize $\\log P(\\hat{G} | G_{\\text{train}}; \\pi_\\omega)$; the state discriminator $D_\\phi$, which pushes generated state sequences to be in-distribution; and the entropy lower bound $\\beta$ that forces the Gaussian policy $\\pi_\\theta$ to remain stochastic. Planning at test time iteratively samples masked timesteps and refills them via Gibbs sampling from the energy model, which dynamically re-proposes sub-goals and prevents stalling.","core_discovery":"The paper introduces Adaptformer, a stochastic and adaptive planner that frames planning as iterative minimization of a learned energy function over sequences of actions. Rather than conditioning on an oracle-provided final goal, Adaptformer learns a goal-augmentation module that proposes intrinsic sub-goals from the current state, instruction, and image embedding; a state discriminator encourages the generated state sequences to stay in-distribution while exploring diverse paths; and an entropy constraint keeps the policy stochastic enough to escape loops and discover new skills. The combination, trained by alternating policy and discriminator updates on offline demonstrations, lets the model plan over horizons longer than its training trajectories and take actions—such as unblocking a path or opening a door—that never appeared in the demonstrations. Empirically, Adaptformer outperforms the LEAP baseline by up to 25% in multi-goal maze tasks, and in a multi-room instruction-completion setting it succeeds where LEAP fails completely.","pith_inferences":["The goal-augmentation module is the linchpin: if its sub-goal proposals do not transfer to goal configurations outside the training distribution, the whole generalization story collapses, so ablating or probing this module on unseen goal layouts is the most direct stress test.","Because the policy is trained on fully observable images, the method's success under partial observability is unresolved; the same energy-minimization machinery might be combined with a belief-state encoder to handle limited field-of-view missions.","The reported 71%-vs-0% result on random-walk training hints that stochastic exploration alone can compensate for poor demonstration quality in simple environments, which could open the door to learning from suboptimal human teleoperation data if the same holds in messy real-world scenes.","Applying the same masked-energy planning loop to continuous control or manipulation tasks would test whether the sub-goal curriculum generalizes beyond gridworld navigation."],"forward_implications":["Long-horizon missions can be solved with the same model that was trained on short single-goal demonstrations, because the learned sub-goal curriculum decomposes the mission into achievable pieces.","Agents can exhibit skills that were never demonstrated, such as moving an obstacle or opening a door, when the stochastic policy and in-distribution sub-goals induce those actions.","The margin over LEAP grows with environment size and complexity, suggesting the approach scales better to larger maps.","Training on random-walk trajectories is sufficient for simple exploration tasks, which removes the requirement for optimal demonstrations in those settings.","The ablation study shows each component—goal conditioning, discriminator, and entropy regularization—contributes about a 7–10% success-rate drop when removed."],"supporting_citations":[{"why":"the baseline method this work extends; supplies the iterative energy-minimization planning loop and the comparison set","marker":"[5]"},{"why":"the sequence-modeling baseline whose goal-conditioned failure motivates Adaptformer's sub-goal curriculum","marker":"[4]"},{"why":"provides the masked-language-model energy interpretation used for trajectory scoring","marker":"[22]"},{"why":"motivates conditioning on achievable sub-goals to avoid stalling","marker":"[18]"},{"why":"supplies the entropy-constrained dual-form training and the Gaussian policy parameterization","marker":"[23]"},{"why":"defines the multi-goal maximum-entropy exploration setting the paper addresses","marker":"[16]"},{"why":"provides the GAN-style alternate training used for the state discriminator","marker":"[20]"}],"fun_headline_variants":["Adaptformer: energy-based planning with adaptive sub-goals","Adaptformer beats SOTA by 25% on mazes via adaptive planning","Sequence model planner adapts to novel missions via sub-goals","Long-horizon planning beyond training data with Adaptformer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The goal-augmentation module, trained only on single-goal demonstrations, is assumed to propose useful sub-goals for multi-goal missions with goals it never saw; if that transfer fails, the policy navigates toward irrelevant waypoints and the generalization claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Adaptformer: energy-based planning with adaptive sub-goals","Adaptformer beats SOTA by 25% on mazes via adaptive planning","Sequence model planner adapts to novel missions via sub-goals","Long-horizon planning beyond training data with Adaptformer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00038,"raw_usage":{"total_tokens":2068,"prompt_tokens":1048,"completion_tokens":1020,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":948}},"tokens_in":664,"tokens_out":1020,"duration_ms":9612,"temperature":1.0,"reasoning_tokens":948,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:31:52.751178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Adaptformer on single-goal-reaching demonstrations in maps where all goals lie in the left half of the grid, then evaluate it in a maze where the only goals are on the right half. If the goal-augmentation module's top-ranked sub-goal proposals never enter the right half and the success rate drops to near zero, then the intrinsic sub-goal curriculum does not transfer to unseen goal configurations, and the central generalization claim is false.","supporting_citations":[{"cited_title":"Planning with sequence models through iterative energy minimization,","cited_arxiv_id":null,"evidence_quote":"the baseline method this work extends; supplies the iterative energy-minimization planning loop and the comparison set"},{"cited_title":"Decision transformer: Re- inforcement learning via sequence modeling,","cited_arxiv_id":null,"evidence_quote":"the sequence-modeling baseline whose goal-conditioned failure motivates Adaptformer's sub-goal curriculum"},{"cited_title":"Waypoint transformer: Reinforcement learning via supervised learning with intermediate targets,","cited_arxiv_id":null,"evidence_quote":"motivates conditioning on achievable sub-goals to avoid stalling"},{"cited_title":"Online decision transformer,","cited_arxiv_id":null,"evidence_quote":"supplies the entropy-constrained dual-form training and the Gaussian policy parameterization"},{"cited_title":"Maximum entropy gain exploration for long horizon multi-goal reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"defines the multi-goal maximum-entropy exploration setting the paper addresses"}],"review_version":1}