{"id":"b0c3faa3-6303-4a5f-bad9-c01270903fde","arxiv_id":"2505.19761","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A prompt-shared hierarchical offline RL pipeline for LLM agents improves long-horizon task scores on ScienceWorld and ALFWorld over non-hierarchical baselines.","lead":"The paper presents GLIDER, a hierarchical training framework that splits an LLM agent into a high-level planner that sets sub-goals and a low-level executor that acts, sharing one model with LoRA. Tests on ScienceWorld and ALFWorld report consistent gains over non-hierarchical RL fine-tuning baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Low-level intrinsic reward “completion” is generated by a hand-written LLM prompt (Appendix D) and never validated; noisy or gameable completion labels would directly undermine the central hierarchy claim.","rationale":"The reader’s weakest_assumption already identified the intrinsic-reward construction as load-bearing and noted the Appendix D judge. My independent reading of Section 3.2, Eq. (2), Appendix D, and Algorithm 1 confirms this is the single most fragile link: the low-level policy (and hence the entire hierarchical decomposition) is trained on completion labels whose generation is delegated to an LLM prompt whose accuracy is unmeasured. The paper’s own text contains evidence supporting the concern: Appendix D shows the judge is prompt-based rather than “easily accessible from observation,” and Algorithm 1’s Step 12 both says “Fix low-level” and admits parameter sharing prevents strict freezing, so judge errors can propagate into both offline and online stages. This is not an external-consensus disagreement; it is an internal gap between the stated method (observation-derived reward) and the implemented method (LLM-judge reward), with no validation. The rest of the paper’s identified issues (missing error bars, missing ArCHer baseline, unstated hyperparameters, data-mixture selection from the eval curve) are real but secondary; the judge-validation issue directly threatens the central claim that hierarchy, rather than the extra LLM verifier, causes the gains. A conditional verdict is appropriate: the concern is concrete and testable, and the paper can resolve it with a focused validation experiment, but the evidence as written is insufficient to accept the mechanism claim unconditionally.","tokens_in":22057,"tokens_out":1773,"duration_ms":16943,"concrete_test":"On a held-out set of ScienceWorld and ALFWorld subtask-action-observation triples, compare the Appendix D “Check Subtask Complete” LLM judge against ground-truth completion labels derived from environment state (e.g., the explicit goal conditions used by the benchmark success checker). Report precision/recall per subtask type and per benchmark. Then retrain the low-level policy with ground-truth completion labels (keeping everything else fixed) and re-run Table 1 unseen-task evaluation; if the score drops materially (e.g., >5 points) or stays the same, the hierarchy claim is either inflated or unverified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.2 asserts that sub-task completion “can be easily accessible from the environment observation, without requiring any manual design or domain knowledge,” and Eq. (2) defines the low-level reward r-hat as this binary completion signal. Yet Appendix D shows the implementation uses a separate “Check Subtask Complete” LLM prompt: given the subtask, initial observation, action sequence, and final observation, a frozen LLM outputs True/False. This is a learned/probed reward annotator, not a direct observation read-off, and its accuracy is never measured. The low-level policy is trained entirely on r-hat (Eq. 2, Eq. 4–7 applied to D_l), so if the judge is noisy or gameable—e.g., it says True whenever the final location matches a keyword, or False for long/nested action sequences—the low-level skills are trained against a corrupted signal. The central comparison (Table 1: Llama-3-8B GLIDER 68.34 unseen vs. ETO 52.33) then conflates the value of hierarchy with the value of an extra LLM-based verifier. The paper provides no accuracy statistics, no ablations replacing the prompt judge with ground-truth completion labels, and no discussion of judge failure modes. Because the low-level policy is also frozen (claimed) or partially updated (Algorithm 1 says “Fix low-level” then admits parameter sharing prevents strict freezing) during O2O, any judge bias propagates into online adaptation claims in Sec. 4.4.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GLIDER, a hierarchical LLM agent framework that pairs a high-level subtask planner with a low-level action executor sharing a single LoRA-tuned backbone, and trains it via behavior cloning, offline RL (IQL-style actor-critic), and optional offline-to-online fine-tuning. Experiments on ScienceWorld and ALFWorld report consistent gains over prompt-based and fine-tuned baselines across Mistral-7B, Gemma-7B, and Llama-3-8B, together with ablations that separate the effects of the hierarchy, the training stages, model scale, and the expert-to-medium data mixture ratio.","tokens_in":22397,"tokens_out":6083,"duration_ms":62070,"significance":"If the central claims hold, GLIDER offers a parameter-efficient recipe for injecting temporal abstraction into LLM-based agents, and the central ablation comparing hierarchical and non-hierarchical versions on the same pipeline and backbones is a well-designed test of the hierarchy hypothesis. The cross-backbone and cross-scale consistency are genuine strengths, and the release of code would support reproducibility. However, the paper's headline claims are currently undercut by an unvalidated LLM completion judge that supplies the low-level reward, by an internal inconsistency about freezing the low-level policy in offline-to-online adaptation, and by the absence of any statistical error analysis. These issues are load-bearing because they directly affect the attribution of the reported performance gains to the proposed hierarchy.","major_comments":[{"comment":"The paper states that subtask completion 'can be easily accessible from the environment observation, without requiring any manual design or domain knowledge,' and Eq. (2) defines the low-level reward r-hat as this binary completion signal. Appendix D, however, shows that completion is determined by a hand-written 'Check Subtask Complete' LLM prompt that receives the subtask, initial observation, action sequence, and final observation. No accuracy statistics, failure-mode analysis, or ablation against ground-truth completion labels is provided. Since the low-level policy is trained entirely on r-hat through Eqs. (4)-(7), a noisy or gameable judge directly threatens the central hierarchy claim; at minimum, the revision should report judge agreement with ground-truth labels on a held-out sample and include an ablation that replaces the prompt judge with oracle completion labels.","section":"Section 3.2, Eq. (2), Appendix D"},{"comment":"The offline-to-online description says the task-agnostic low-level policy is frozen and only the high-level policy is fine-tuned, but Algorithm 1 says 'Fix low-level policy' and the Appendix A text then admits that parameter sharing prevents strict freezing, so low-level performance is instead maintained by continually training on offline demonstration data. This internal inconsistency undermines the attribution of the O2O results in Section 4.4 to frozen, transferable skills. Please specify the exact parameter update scheme (for example, which LoRA modules or prompts are updated) and report an ablation with truly frozen low-level parameters, or reinterpret the O2O claims accordingly.","section":"Section 3.5 and Algorithm 1, line 12"},{"comment":"No evaluation seeds, error bars, confidence intervals, or significance tests are reported anywhere, despite the small evaluation sets shown in Table 3 (for example, 211 unseen ScienceWorld episodes). The headline comparison of Llama-3-8B GLIDER at 68.34 versus ETO at 52.33 could in principle be within run-to-run noise. Please run at least three to five seeds per condition and report mean and standard deviation, or provide pairwise significance tests; this is essential for the paper's repeated claim of 'consistent performance gains.'","section":"Table 1, Figures 3-5"},{"comment":"The expert-to-medium 1:2 mixture ratio used for all main experiments appears to have been selected because it gives the best point estimate in Figure 5. If this ratio was chosen after inspecting test performance, the reported gains carry an optimism bias, and the claim that 1:2 is best needs a defined selection protocol or a held-out validation split. Moreover, without error bars, the differences among 68.34 (1:2), 60.31 (1:1), and 60.29 (1:5) may be noise; the revision should report variance and justify the selection procedure.","section":"Section 4.5 and Figure 5"},{"comment":"The claim that the framework eliminates 'the necessity for any manual or task-specific design' is not supported by the implementation: the subtask labels in expert demonstrations are produced by GPT-4 decomposition (Appendix B), the completion judge is a custom LLM prompt, and the hierarchy interval c is a hyperparameter. Please either soften these autonomy claims or provide evidence that these design choices are not task-specific.","section":"Section 3.2 and Appendix B"}],"minor_comments":[{"comment":"The baseline is called 'SwiftSage' in the related work but appears as 'SwitchSage' in Table 1 and elsewhere; please standardize the name.","section":"Section 4.1 and Table 1"},{"comment":"The \\toggle-off and \\toggle-on markup artifacts appear in the rendered table; these should be removed.","section":"Table 1"},{"comment":"The hyperparameter table lists 'batch size' twice with different values (64 and a per-device batch size of 2) and also gives gradient accumulation steps; please clarify the effective global batch size.","section":"Appendix C, Table 5"},{"comment":"The text describes the setting as 'Online-to-Offline' when it should be 'Offline-to-Online' to match Section 3.5.","section":"Appendix B, O2O task setups"},{"comment":"The Konda and Tsitsiklis (1999) entry lists page range 75993-76005, which appears incorrect; please verify and correct the citation details.","section":"References"},{"comment":"The low-level prompt asks the model to 'determine if the subtask is completed (true/false),' while the completion reward is generated separately by the 'Check Subtask Complete' prompt; please clarify how these two completion signals relate and whether the judge is the sole source of r-hat.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper is directionally promising, and the hierarchical-vs-non-hierarchical ablation on the same pipeline is a real strength. However, the unvalidated LLM completion judge and the missing statistical rigor are substantial obstacles to accepting the headline claims. I would like the revision to provide judge validation, seed-level variance, and a clear statement of the actual freezing scheme in O2O; with those additions, the central claim may well be supported. I also recommend that the editor check whether the baselines (NAT, ETO) are retrained under the same per-backbone settings or whether scores are copied from prior publications, since Table 1 does not state this."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea is good: one LoRA-tuned LLM playing both planner and executor, separated only by a prompt, with offline SFT then IQL-style RL, is a genuinely parameter-efficient hierarchy, and the paper's internal ablation (w/ Hier vs w/o Hier on the same pipeline) supports that the hierarchy itself is what buys the gain. Second, the low-level reward signal—the thing the entire low-level policy is trained on—is not the “easily accessible from observation” signal Section 3.2 claims. Appendix D reveals it is a hand-written LLM prompt asking a frozen model to judge subtask completion True/False. That judge's accuracy is never checked. If it is noisy or gameable, the low-level skills are trained on garbage, and the headline gains over ETO conflate hierarchy with an unvalidated verifier.\n\nWhat the paper does well: the shared-backbone architecture is a real parameter-efficiency win, and the three-stage pipeline (SFT, offline RL, O2O) is coherent. The ablations across three backbones and three model scales are a plus, and the w/ Hier vs w/o Hier comparison is the right one—it holds in every training stage, which is encouraging.\n\nThe soft spots are real. No error bars or seeds anywhere; a single run could move a few points, and the ALFWorld margins are already small. The 1:2 expert-to-medium data mixture looks like it was picked from the Figure 5 curve and then used for the headline numbers—that is at minimum post-hoc selection, and they should show robustness around that choice. ArCHer, the closest hierarchical LLM baseline, is cited but never compared. The paper says it freezes low-level skills during O2O, but Algorithm 1 admits parameter sharing prevents strict freezing and instead trains low-level on offline data; that contradiction needs resolving. And the hyperparameters c (hierarchy interval), the IQL expectile tau, and the length-regularization lambda in Eq. 3 are not reported—the table's lambda=0.99 appears to be the advantage weight, not the length penalty.\n\nWho is this for: people working on LLM agents and hierarchical RL. It is useful as a design recipe and as a cautionary example of reward validation. It deserves a serious referee—the architecture and the question are worthy—but the paper needs seeds, reported hyperparameters, a validated or replaced low-level reward, and ideally an ArCHer comparison before the claims are credible. My own verdict is skeptical, not because the idea is bad but because the measurement is incomplete. Send it to review, but expect major revision.","headline":"A genuinely neat parameter-efficient hierarchy, undermined by an unvalidated LLM-based reward judge and missing statistics.","tokens_in":22916,"tokens_out":2357,"would_cite":false,"duration_ms":25855,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An offline hierarchical RL framework makes LLM agents plan sub-tasks and act on them, beating non-hierarchical baselines on long-horizon benchmarks.","keywords":["hierarchical reinforcement learning","offline RL","LLM agent","temporal abstraction","sub-task planning","ScienceWorld","ALFWorld","parameter-efficient fine-tuning"],"falsifier":"If a head-to-head rerun of the exact Llama-3-8B tables (ScienceWorld seen/unseen, ALFWorld seen/unseen, and the three online adaptation curves) were run with the sub-task-completion judge replaced by an objective state-checker (or with a judge shown to be systematically noisy), and the margins over ETO collapsed, the paper's central mechanism would be identified as reward noise rather than hierarchy.","tokens_in":21886,"feed_emoji":"🧠","tokens_out":3163,"duration_ms":27809,"temperature":0.7,"pith_summary":"GLIDER claims that LLM agents for long-horizon interactive tasks improve when the policy is split into a high-level planner that proposes sub-tasks every c steps and a low-level executor that turns each sub-task into primitive actions. The paper argues that this divide-and-conquer structure creates temporal abstraction, making exploration easier and giving the agent a chain-of-thought-like decomposition to follow. Training proceeds offline in three stages: behavior cloning, offline RL fine-tuning with an actor-critic, and optional online fine-tuning of the high-level policy only. The payoff claimed is consistent gains over strong fine-tuning baselines on ScienceWorld and ALFWorld, with the largest margins on unseen tasks, plus fast adaptation to new tasks by reusing the low-level skills.","feed_headline":"Hierarchical offline RL boosts LLM agents on long-horizon tasks","feed_subtitle":"A prompt-driven planner plus executor lifts Llama-3-8B to 68.34 on unseen ScienceWorld tasks, vs 52.33 for a strong baseline.","key_machinery":"The central object is the two-level hierarchical policy with temporal abstraction: every c environment steps the high-level policy proposes one sub-task goal gt, and the low-level policy executes c primitive actions against that goal, collecting sub-task-completion intrinsic reward. The training machinery is a three-stage pipeline of SFT (behavior cloning with length regularization), ORL (offline actor-critic; IQL-style asymmetric expectile value learning, and advantage-weighted actor updates), and O2O (online fine-tuning of the high level only while the low level is refreshed with offline demonstrations). What carries the argument is that shared parameters with a hierarchy prompt make the hierarchy cheap, and the intrinsic reward makes the low level task-agnostic enough to transfer.","core_discovery":"The core claim is that a parameter-efficient hierarchy—where high-level planner and low-level executor share one LoRA-tuned LLM backbone differentiated only by a prompt—is a reliable way to lift LLM agent performance on long-horizon decision tasks. The high-level policy receives the task description and observation and emits a sub-task goal every c steps; the low-level policy then emits primitive actions for those c steps and receives a binary intrinsic reward indicating sub-task completion derived from the observation. Both policies are trained offline from a mixture of expert and medium-quality trajectories, first by behavior cloning and then by token-level advantage-weighted policy optimization with sentence-level critics. The paper reports that with Llama-3-8B, GLIDER reaches 77.43 on seen and 68.34 on unseen ScienceWorld tasks, versus 57.90 and 52.33 for ETO; it also reports consistent gains on ALFWorld and in ablations across backbones and model scales.","pith_inferences":["A testable extension suggested by the paper's own logic: if the low-level skills are truly task-agnostic, then a planner trained on one benchmark should bootstrap a low-level library trained on another, which the paper does not directly demonstrate.","The hierarchical structure may also function as an implicit reward-shaping mechanism: the binary sub-task completion signal gives the low-level policy dense feedback where environment rewards are sparse, which the non-hierarchical baselines lack.","The claim that sub-task completion is 'easily accessible from observation' would be stronger if it were verified across language-domains with adversarial distractors, where a verification LLM prompt could confidently misjudge completion.","The framework's authors have implicitly connected it to the idea of language-generality for sub-tasks (e.g., reusing 'monitor until done' patterns), which suggests a next step: measuring how much of the gain survives translation to environments whose observations do not naturally decompose into named sub-tasks."],"forward_implications":["If the method works as claimed, hierarchical sub-task decomposition plus offline RL could be the default recipe for grounding LLM policies in sparse-reward, long-horizon environments.","Unseen tasks should inherit the low-level sub-task library, so generalization to new tasks becomes mostly a planning problem, not an action-generation problem.","Offline-to-online adaptation should need far fewer environment interactions than monolithic fine-tuning, since only the high-level policy must be re-adapted to a new reward.","Even smaller models (Llama-3B) are claimed to outperform larger non-hierarchical models, which would make hierarchical decomposition a partial substitute for model scale.","The ablation results imply that SFT alone is not enough, and that offline RL alone is better than SFT alone, so the main source of capability is the reinforcement fine-tuning stage."],"supporting_citations":[{"why":"Supplies the ScienceWorld benchmark and its dense reward structure, which the method's primary evaluation depends on.","marker":"(Wang et al., 2022)"},{"why":"Supplies the ALFWorld benchmark and the sparse binary reward setting that tests the method's credit-assignment claims.","marker":"(Shridhar et al., 2021)"},{"why":"Provides the ETO baseline and the contrastive-trajectory exploration paradigm that GLIDER claims to surpass on unseen tasks.","marker":"(Song et al., 2024)"},{"why":"Provides the IQL asymmetric value-learning loss used in the offline critic, which is central to the RL fine-tuning stage.","marker":"(Kostrikov et al., 2022)"},{"why":"Provides the AWAC advantage-weighted actor update and the offline-to-online paradigm that GLIDER relies on for adaptation.","marker":"(Nair et al., 2020)"},{"why":"Provides LoRA, the parameter-efficient tuning method that lets the two policy levels share the frozen backbone.","marker":"(Hu et al., 2022)"},{"why":"Defines the ReAct prompting baseline that GLIDER's hierarchy is contrasted against in experiments.","marker":"(Yao et al., 2023b)"}],"fun_headline_variants":["GLIDER: LLM agents chunk long tasks into sub-goals for better decisions","One LLM, two roles: planner and doer, via offline RL hierarchy","Offline hierarchical RL lifts LLM agents on long-horizon benchmarks","LLMs divide tasks into sub-goals, conquer sparse-reward decisions","GLIDER: prompt-differentiated hierarchy improves LLM decision-making"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Sub-task completion can be read off the observation reliably, and this reading needs no manual design, because the low-level policy is trained entirely on binary intrinsic rewards built from that judgement; the appendix shows the judgement is actually made by a hand-written LLM prompt.","fun_headline_variants_meta":{"raw":{"variants":["GLIDER: LLM agents chunk long tasks into sub-goals for better decisions","One LLM, two roles: planner and doer, via offline RL hierarchy","Offline hierarchical RL lifts LLM agents on long-horizon benchmarks","LLMs divide tasks into sub-goals, conquer sparse-reward decisions","GLIDER: prompt-differentiated hierarchy improves LLM decision-making"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1612,"prompt_tokens":965,"completion_tokens":647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":581,"tokens_out":647,"duration_ms":6812,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:06:33.172923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a head-to-head rerun of the exact Llama-3-8B tables (ScienceWorld seen/unseen, ALFWorld seen/unseen, and the three online adaptation curves) were run with the sub-task-completion judge replaced by an objective state-checker (or with a judge shown to be systematically noisy), and the margins over ETO collapsed, the paper's central mechanism would be identified as reward noise rather than hierarchy.","supporting_citations":[{"cited_title":"ScienceWorld : Is your agent smarter than a 5th grader? In Proceedings of Empirical Methods in Natural Language Processing, pp.\\ 11279--11298, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the ScienceWorld benchmark and its dense reward structure, which the method's primary evaluation depends on."},{"cited_title":"ALFWorld : Aligning text and embodied environments for interactive learning","cited_arxiv_id":null,"evidence_quote":"Supplies the ALFWorld benchmark and the sparse binary reward setting that tests the method's credit-assignment claims."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ETO baseline and the contrastive-trajectory exploration paradigm that GLIDER claims to surpass on unseen tasks."},{"cited_title":"Offline reinforcement learning with implicit Q -learning","cited_arxiv_id":null,"evidence_quote":"Provides the IQL asymmetric value-learning loss used in the offline critic, which is central to the RL fine-tuning stage."}],"review_version":1}