{"id":"f98d852d-c52d-4a9c-953e-d60f040d1e3d","arxiv_id":"2411.17135","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"CoREN uses an LLM offline to estimate dense action rewards, filters them through three consistency checks, and aligns them to sparse success labels to train a small offline RL agent for household instruction-following tasks.","lead":"This paper trains a small embodied agent by using a large language model offline to score every action in pre-collected trajectories, then distills those scores into a compact policy. The method, CoREN, reports success rates on the VirtualHome benchmark close to online LLM-based agents, while using only a 117M-parameter policy at deployment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reward orchestrator in Eq. (10) is underdetermined: one trajectory-level success label cannot identify per-step weights, so the claimed domain-grounded dense reward may be a global rescaling; Table 5's small gap over majority voting makes this concern concrete.","rationale":"The paper's strongest empirical claim could be true: a 117M policy trained offline with LLM-derived dense rewards may indeed approach 8B online LLM agents on VirtualHome. I do not dispute that possibility. My concern is narrower and mechanistic: the only supervision used to turn three LLM reward streams into a 'domain-grounded' reward is Eq. (10)'s trajectory-level return-to-success alignment. With one scalar target per trajectory and per-step weights, the solution space is huge; the loss can be minimized by rescaling that does not improve stepwise reward quality. The near-parity of simple majority voting in Table 5 (60.8 vs 66.4 SR fine-grained; 55.2 vs 57.6 abstract) shows that a large part of the headline performance is already present without the orchestrator. The proposed scalar-only experiment would directly test whether the orchestrator's conditioning on (o_t,l_t,i) matters. This is not an accusation of data fabrication or dishonesty; it is an identifiability question about the stated learning objective. The reader's CONDITIONAL verdict is appropriate, and my concern mostly reinforces it rather than moving it. I agree with the reader's identification of insufficient per-step grounding, but I separate it from the cross-domain relabeling issue, which is a second, independent problem; hence 'partial' agreement.","tokens_in":25042,"tokens_out":9675,"duration_ms":96104,"concrete_test":"Train a stripped-down CoREN where Ψθ is replaced by two scalar weights shared across all timesteps—one for success-flagged trajectories and one for failures—while keeping the same rC, rS, rT and Eq. (10) objective. Run at least 5 seeds on the single-domain VirtualHome split and compare SR/CGC to Table 1. If scalar-only matching comes within ~2-3 SR points of CoREN, the per-step weighting is not the source of the gains and the 'domain-grounded dense reward' claim is unsupported. Additionally, if per-step LLM reward annotations are released, measure the correlation of r̂ with a ground-truth per-step relevance label (e.g., whether the action appears in the expert program at that step); if correlation is not higher than the raw majority vote, alignment is not grounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (10) trains the reward orchestrator by matching the discounted sum of per-step rewards Σ_t γ^t r̂(o_t,l_t|i) to the scalar success flag α f_s(i,τ). That is a single aggregate constraint per trajectory, but Ψθ outputs three softmax weights for every timestep. The per-step weights are therefore not identifiable from the loss: a global rescaling that makes successful trajectories' returns positive and failed ones negative can minimize Eq. (10) without ever correcting a wrong per-step LLM score. The resulting dense reward is not shown to be domain-grounded in the sense claimed. Table 5 makes this concrete: simple majority voting (no learned weights) already reaches 60.8/55.2 SR versus CoREN's 66.4/57.6, leaving a small margin that could be a trajectory-level bias rather than genuine stepwise grounding. Since the central claim is that LLM knowledge is distilled into a well-grounded dense reward, this underdetermination is load-bearing: the reported policy gains do not yet demonstrate the claimed grounding mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoREN, an offline RL framework for embodied instruction-following. Instead of using an LLM as an online agent, CoREN uses an LLM to generate dense stepwise rewards for offline trajectories, then combines three reward estimates (contextual, structural, temporal consistency) via a learned 'reward orchestrator' that is trained to make discounted trajectory returns match sparse success labels. The resulting reward-augmented dataset is used to train a compact 117M GPT2-based policy with CQL/DDQN. Experiments on VirtualHome (and ALFRED in the appendix) report that CoREN outperforms offline RL baselines and is comparable to online LLM agents with much larger policy models.","tokens_in":25245,"tokens_out":6374,"duration_ms":59712,"significance":"If the results hold, the paper demonstrates a useful direction: LLM commonsense can be distilled into a small offline policy, avoiding online LLM inference cost and latency. The framework is original in combining multiple consistency filters with a learned ensemble, and the authors provide extensive ablations (consistency types, LLM choices, ensemble scheme, number of prompts) plus an additional ALFRED experiment. The appendix includes full prompt templates and hyperparameters, which aids reproducibility. The main limitations are evidential: no error bars or seeds are reported, and the cross-domain protocol relabels old trajectories rather than executing in a genuinely new environment. These issues must be addressed before the central claims are fully supported.","major_comments":[{"comment":"All results are single point estimates with no standard deviations, confidence intervals, or number of seeds for any RL agent. The claims of 'significantly outperforms' and 'comparable performance' are therefore not backed by statistical evidence. This is especially problematic for the small margins in Table 5 (e.g., 66.4 vs 60.8 fine-grained SR, and 57.6 vs 55.2 abstract SR). Please report mean and standard deviation over at least 5-10 seeds and, for headline comparisons, provide a significance test.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The cross-domain protocol relabels the sparse success flags of existing trajectories from the original environment and retrains only the reward orchestrator. Since the observations, actions, and dynamics remain those of the original layout, a trajectory that was feasible in the old layout is not guaranteed to be feasible or goal-achieving in the new layout; relabeling it as successful does not simulate a new environment. The claim of 'relatively robust performance across domain shifts' (Table 2) is therefore not established. The paper should execute the relabeled trajectories in the VirtualHome simulator with the new object locations, or explicitly reframe the experiment as a label-shift robustness test and not as a domain shift.","section":"Section 4.2, cross-domain evaluation"},{"comment":"The orchestrator is trained only on a per-trajectory aggregate loss: the discounted sum of stepwise rewards is matched to the scalar success flag. Because Psi_theta outputs per-timestep weights, there are many weight assignments that satisfy the aggregate constraint for a given trajectory, and the loss does not directly reward per-step accuracy. The paper should provide either an identifiability argument or a per-step reward quality evaluation (e.g., correlation with human or expert stepwise scores), or an ablation that separates the contribution of per-step weighting from a simple trajectory-level offset or scaling. The modest gain over majority voting in Table 5 makes this distinction important.","section":"Section 3.3, Eq. (10)"},{"comment":"The offline dataset appears to contain only one expert trajectory per task labeled as success, with random perturbations labeled as failure. If the success flag is effectively 'this is the expert trajectory' rather than a verified goal-completion label, then the reward orchestrator may learn to identify the expert trajectory instead of generalizable task success. Please clarify whether success labels are verified by environment execution, and if not, discuss the consequence for the claimed 'domain-grounded' reward.","section":"Section 4.1 / Appendix A.2"}],"minor_comments":[{"comment":"The framework name is inconsistent: 'COREN' appears in the abstract and tables, 'CORE N' in Figure 1 and Algorithm 1, and 'CoREN' in several places. Please use a single spelling consistently.","section":"Throughout"},{"comment":"The arrow symbols '← −' in lines 7-9 and 20-21 appear to be formatting artifacts; please use standard assignment arrows.","section":"Algorithm 1"},{"comment":"The text states fs(i, tau) in {-1, 1}, while Section 4.1 defines success/failure flags as 1/0. Please clarify the normalization and the role of alpha in Eq. (10).","section":"Appendix B.1.3"},{"comment":"In Query 2, the answer lists 'cat' although 'cat' is not among the visible objects listed in the query. Please correct this example.","section":"Table 7"},{"comment":"The sentence 'Table Table 14' contains a duplicated word; fix the cross-reference.","section":"Appendix D.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the empirical setup is extensive, but the paper as submitted does not yet provide enough evidence for its strong claims. The lack of seeds/error bars is a serious omission in an RL paper, and the cross-domain experiment as designed is not a true domain shift. The underdetermination concern about Eq. (10) is real and should be addressed with additional analysis rather than text alone. If the authors can add a proper multi-seed evaluation, a valid cross-domain protocol, and evidence that the learned weights improve per-step reward accuracy, the paper could be publishable; otherwise the central claims remain unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nThe paper is worth a serious look, but its central mechanism is weaker than advertised. The new thing is a two-stage pipeline: an LLM produces three per-step reward streams filtered by contextual, structural, and temporal consistency, and a learned orchestrator combines them by fitting the trajectory return to a binary success flag. This combination is genuinely new, and the motivation—distilling LLM knowledge into a small offline policy—is practical. The VirtualHome and ALFRED results show the full method beating offline-RL baselines and staying competitive with online LLM agents, which is interesting if it holds up.\n\nThe biggest soft spot is the reward orchestrator. Eq. (10) fits a trajectory-level scalar to a binary label, while the orchestrator outputs per-step weights. That loss cannot identify the per-step weights; any global rescaling of the reward streams that separates successful from failed returns will minimize it. The claimed 'domain-grounded' dense reward is therefore not demonstrated. Table 5 makes the concern concrete: majority voting alone gets 60.8/55.2 SR versus CoREN's 66.4/57.6, so the learned ensemble adds only a few points. That margin could come from a trajectory-level bias, not from genuinely better stepwise credit assignment. The paper needs to show, e.g., per-step reward quality on held-out trajectories or a simpler baseline that adjusts only the global scale.\n\nThe cross-domain experiment is another real problem. They relabel sparse rewards on trajectories from the old environment instead of collecting or simulating trajectories in the new layout. If the old trajectories contain actions that are no longer feasible (e.g., grabbing an apple from the table when the apple is now in the fridge), the relabeled success flags are wrong, and the reported cross-domain robustness doesn't follow. This needs a proper protocol.\n\nMinor but important: all tables report point estimates with no seeds or error bars, which is a serious omission for RL; the prompt-count ablation (Appendix D.3) appears to select N on the test set; and the dataset construction text is ambiguous about how many successful trajectories exist.\n\nNone of this makes the paper a waste of time. The consistency filters are a sensible idea, the ablations show they matter, and the authors are honest about dependence on LLM capability. The paper just needs to close the gap between the claims and the evidence.\n\nMy recommendation: send it to peer review, with the expectation of major revisions. The referee should demand seeds, a corrected cross-domain protocol, and a direct test of whether the orchestrator's per-step weights are identifiable from the trajectory-level loss.\n\nBest.","headline":"A promising but under-supported method: the consistency-guided reward ensemble is new, but the orchestrator's per-step grounding is not demonstrated and the cross-domain test is flawed.","tokens_in":25794,"tokens_out":2960,"would_cite":false,"duration_ms":26141,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoREN shows that a large language model can be used only as a training-time reward teacher, distilling its commonsense into a 117M-parameter offline RL agent that matches online LLM agents with 8B parameters on household…","keywords":["offline reinforcement learning","large language models","reward estimation","embodied agents","instruction following","consistency ensemble","VirtualHome","reward shaping"],"falsifier":"In the cross-domain setting, replace relabeled old trajectories with trajectories actually executed in the new layout and retrain the orchestrator; if CoREN's SR drops to the level of the RL baselines, the claim that relabeling faithfully simulates the new environment is false.","tokens_in":24812,"feed_emoji":"🤖","tokens_out":8187,"duration_ms":65740,"temperature":0.7,"pith_summary":"The paper asks whether a large language model can be used not as an embodied agent but as a teacher: it scores each action in an offline dataset, and those scores are distilled into a small policy network via offline reinforcement learning. The difficulty is that LLM scores reflect commonsense, not the specific layout of the target house, so they can reward actions that make sense in general but are wrong here. CoREN addresses this with three consistency filters — agreement across different prompts, correct answers to object-relevance questions, and backward verification of the LLM's own high-value action sequence — followed by a learned blend of the filtered rewards that is aligned to the trajectory's binary success flag. The claim is that this two-stage process yields dense rewards grounded enough to train a 117M-parameter agent that matches online LLM agents with 8B parameters on VirtualHome instruction-following, and that degrades only mildly when objects move to new locations.","feed_headline":"LLM teacher distills into a 117M offline RL agent","feed_subtitle":"Using three consistency filters plus sparse labels, CoREN rivals 8B online LLM agents on VirtualHome.","key_machinery":"The central object is the consistency-guided reward ensemble, defined by three consistency-constrained reward estimates r_C, r_S, r_T computed from an LLM, and a learned reward orchestrator Ψ_θ that blends them into a unified stepwise reward r̂(o,l|i) = ⟨r(o,l|i), w(o,l|i)⟩. Contextual consistency takes a majority vote over N differently prompted LLM reward estimates; structural consistency only lets a reward count if the LLM correctly answers MDP-specific queries such as which visible objects are relevant to the instruction; temporal consistency drops rewards for actions the LLM rated as high-value when backward verification says the resulting sequence cannot accomplish the instruction. The orchestrator is a small network that outputs per-step weights (w_C, w_S, w_T) and is trained to minimize the squared difference between the discounted return of the unified rewards and the trajectory's binary success flag. This machinery carries the argument because the weights turn three individually biased commonsense score streams into one reward that is aligned with the actual outcome labels, and only the orchestrator needs retraining when the environment domain shifts.","core_discovery":"The central discovery, as the authors state it, is that consistency-guided reward ensemble (CoREN) makes LLM-based reward estimation work in the offline setting, where the LLM never interacts with the environment. The paper reports that CoREN outperforms all offline RL agents it is compared with and achieves performance comparable to state-of-the-art LLM-based agents using 8B parameters, despite the agent policy having only 117M parameters and the LLM being used only for training. The mechanism attributed for this is the ensemble of spatio-temporally consistent rewards: contextual (majority vote over diverse prompts), structural (gating rewards by the LLM's answers to object-relevance queries about the environment), and temporal (holding the LLM's high-value actions to a backward verification that they can accomplish the instruction), merged by a reward orchestrator trained to align the trajectory return with the sparse success flag.","pith_inferences":["If CoREN's orchestrator really captures domain grounding from sparse labels alone, the same scheme could densify rewards for any long-horizon instruction-following benchmark that ships only binary success flags, not just embodied household simulators.","The cross-domain experiment relabels old trajectories rather than collecting new ones; a stronger test would be retraining the orchestrator on trajectories actually executed in the new layout, since the reported robustness may be partly an artifact of the relabeling shortcut if those relabeled trajectories contain actions infeasible in the new layout.","The large gap between LLaMA3-8B (SR 12.0) and GPT-4 Turbo/Gemini (SR 65.6/66.4) as reward estimators suggests CoREN is a distillation channel whose ceiling tracks the teacher's commonsense and spatial reasoning, so stronger or domain-tuned LLMs should raise the ceiling proportionally.","One immediately testable extension: apply CoREN to a dataset with corrupted success flags (e.g., 20% flipped) and measure SR; if performance barely moves, the orchestrator's alignment is not what grounds the reward, contradicting the paper's mechanism."],"forward_implications":["On VirtualHome, CoREN reaches 66.4% fine-grained SR and 57.6% abstract SR, beating the strongest RL baseline (Self-Consistency) by 20 percentage points in SR and coming within a few points of SayCan-Gemini and ProgPrompt-Gemini.","Under cross-domain shifts, CoREN retains most of its performance (fine-grained SR 60.0 versus 66.4 in the single domain) by only retraining the orchestrator on relabeled trajectories, while the RL baselines drop to as low as 2.5 SR.","Each consistency mechanism contributes: ablations that remove temporal, structural, or contextual consistency lower fine-grained SR to 53.6–64.8 from 66.4.","The learned ensemble outperforms averaging (53.6 SR) and majority voting (60.8 SR), so the sparse-reward alignment step is doing real work.","On ALFRED (appendix), CoREN achieves 72.0 fine-grained SR versus the best RL baseline's 46.4, showing the recipe transfers to a different household benchmark."],"supporting_citations":[{"why":"Supplies VirtualHome, the benchmark environment with 58 actions and the household tasks on which all main results are measured.","marker":"(Puig et al., 2018)"},{"why":"The conservative Q-learning algorithm that trains the 117M-parameter agent on the reward-augmented offline dataset.","marker":"(Kumar et al., 2020b)"},{"why":"Lafite-RL, the closest baseline that also turns LLM action evaluations into rewards for an RL agent and that CoREN must beat.","marker":"(Chu et al., 2023)"},{"why":"RDLM, the LLM-based trajectory-return evaluation baseline that CoREN outperforms by large margins.","marker":"(Kwon et al., 2023)"},{"why":"Self-Consistency, the strongest RL-agent baseline, which votes over chain-of-thought reward candidates and which CoREN reports beating by 20 SR points.","marker":"(Wang et al., 2023)"},{"why":"SayCan, the online LLM-plus-affordance agent that CoREN nears at fine-grained SR despite a much smaller policy.","marker":"(Ichter et al., 2022)"},{"why":"ProgPrompt, the programmatic-prompt LLM agent that sets the highest fine-grained SR among the online baselines.","marker":"(Singh et al., 2023)"},{"why":"LLM-Planner, the retrieval-augmented planner baseline used in both VirtualHome and ALFRED comparisons.","marker":"(Song et al., 2023b)"}],"fun_headline_variants":["LLM teaches offline RL agent, 117M policy rivals 8B agents","Offline RL agent trained by LLM rewards matches 8B agents","LLM reward teacher yields 117M agent that rivals 8B","Tiny 117M agent matches 8B LLM agents via offline LLM rewards"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a binary success flag on each trajectory, together with the LLM's commonsense action scores, is enough to learn a dense reward that is truly grounded in the environment, and that reusing old trajectories with relabeled flags fairly simulates a new environment layout.","fun_headline_variants_meta":{"raw":{"variants":["LLM teaches offline RL agent, 117M policy rivals 8B agents","Offline RL agent trained by LLM rewards matches 8B agents","LLM reward teacher yields 117M agent that rivals 8B","Tiny 117M agent matches 8B LLM agents via offline LLM rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000908,"raw_usage":{"total_tokens":3888,"prompt_tokens":914,"completion_tokens":2974,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":2889}},"tokens_in":530,"tokens_out":2974,"duration_ms":17022,"temperature":1.0,"reasoning_tokens":2889,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:28:12.094546+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In the cross-domain setting, replace relabeled old trajectories with trajectories actually executed in the new layout and retrain the orchestrator; if CoREN's SR drops to the level of the RL baselines, the claim that relabeling faithfully simulates the new environment is false.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RDLM, the LLM-based trajectory-return evaluation baseline that CoREN outperforms by large margins."},{"cited_title":"Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu","cited_arxiv_id":null,"evidence_quote":"SayCan, the online LLM-plus-affordance agent that CoREN nears at fine-grained SR despite a much smaller policy."}],"review_version":1}