{"id":"3840bee2-db8a-44f9-9ff5-1e051568ae91","arxiv_id":"2505.10010","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A benchmark for offline RL with LLM-generated imaginary rollouts shows existing algorithms reach only 35.44% success on hard tasks versus 64.37% when trained on real rollouts.","lead":"ImagineBench is a new benchmark that tests how well offline reinforcement learning can learn from synthetic 'imaginary' rollouts generated by a large language model instead of real environment data. It spans five environments across locomotion, manipulation, and navigation, and finds that standard offline RL algorithms still lag far behind training on real rollouts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hard-task gap may be an artifact of low-quality LLM rollouts (Tab. 2: 25.8% goal consistency in BabyAI), not a proven algorithm limitation; no quality-filtered baseline is provided.","rationale":"The reader's weakest assumption is that rollout quality is measured only in BabyAI, so the headline gap may be an artifact of the specific Llama-2-7b pipeline. My concern is stronger and more specific: even in BabyAI, the hard-task imaginary rollouts have only 25.8% goal consistency, so the majority of the synthetic data used to train the w/IR baselines is off-goal. The comparison to 'Real' (which uses genuine rollouts for the novel tasks) therefore cannot separate algorithmic failure from data quality failure. The paper needs a quality-controlled baseline to justify its central interpretation that existing offline RL algorithms are suboptimal at using imaginary rollouts. Without such a control, the 35.44% vs 64.37% gap is an unsurprising consequence of training on mostly incorrect synthetic data. The benchmark itself remains a useful resource, so a conditional acceptance with a request for this analysis is appropriate; hence the reader's CONDITIONAL verdict stands unchanged.","tokens_in":15050,"tokens_out":7092,"duration_ms":63965,"concrete_test":"Apply the three quality filters from Sec. 5.2 (goal consistency, transition correctness, dynamics legality) to the released BabyAI hard-task imaginary rollouts. Retrain all w/IR baselines on the real training data plus only the filtered high-quality imaginary rollouts, reusing the same hyperparameters and seeds, and evaluate on hard tasks. If filtered w/IR success approaches the 'Real' 64.37% level, the original gap is due to synthetic data quality rather than algorithm deficiency, and the abstract's 'need for algorithm advancements' claim should be reframed as a need for better LLM imagination. If filtered w/IR remains near 35.44%, the algorithm gap is substantiated. Report per-environment filtered/unfiltered results as a supplementary table.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim that existing offline RL algorithms are suboptimal at leveraging LLM-imaginary rollouts rests on a 35.44% vs 64.37% gap on hard tasks (Abstract, Sec. 5.4, Fig. 6). This comparison is confounded: the w/IR methods train on imaginary rollouts for novel tasks, while the 'Real' baseline trains on actual environment rollouts for those same novel tasks. For hard tasks, the imaginary rollouts are of poor quality even in the one environment where quality is measured (BabyAI, Tab. 2): goal consistency 25.8%, transition correctness 72.9%, dynamics legality 66.8%. Training on mostly off-goal or illegal synthetic data naturally underperforms training on genuine task rollouts. The paper provides no control for rollout quality—no filtering, no quality weighting, no oracle upper bound using only the correct imaginary rollouts—so the gap cannot be attributed to algorithmic deficiency. Further, quality statistics are reported only for BabyAI (Tab. 2), leaving the utility of imaginary rollouts in MuJoCo, Meta-world, CLEVR-Robot, and LIBERO unexamined; the headline average across environments may therefore reflect dataset quality differences rather than a general algorithm limitation. The paper's own Sec. 6 acknowledges that LLM imagination quality is a limiting factor, undercutting the interpretation that algorithms are the primary bottleneck.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ImagineBench, a benchmark for offline reinforcement learning with LLM-imaginary rollouts. It provides real and synthetic datasets across five environments (MuJoCo, Meta-world, CLEVR-Robot, BabyAI, LIBERO), organizes tasks into Training, Rephrasing, Easy, and Hard levels, and evaluates seven offline RL algorithms (BC, CQL, BCQ, TD3+BC, PRDC, COMBO, SAC) with and without imaginary rollouts. The central empirical claim is that existing offline RL algorithms underutilize LLM-imaginary rollouts, as evidenced by a 35.44% success rate on hard tasks versus 64.37% when training on real rollouts for the same tasks. The authors also report rollout quality statistics, outline future directions, and state that code and datasets will be released.","tokens_in":15332,"tokens_out":3653,"duration_ms":32170,"significance":"The benchmark is a potentially valuable community resource: it targets an emerging research direction, covers diverse domains, includes language-conditioned tasks with graded difficulty, and provides a standardized comparison protocol. The release of code and datasets, if complete and reproducible, would lower the barrier for future RLIM research. However, the headline conclusion that existing algorithms are 'suboptimal' at leveraging imaginary rollouts is currently under-supported: the quality statistics are collected only for BabyAI, and the headline gap is confounded by dropout-quality differences rather than isolating algorithmic design. With additional per-environment quality measurements and quality-filtered control experiments, the benchmark's central finding could become a solid contribution; as written, the evidence is suggestive but not conclusive.","major_comments":[{"comment":"Quality statistics (consistency, transition correctness, dynamics legality) are reported only for BabyAI, yet the abstract and Section 5.4 draw conclusions about all five environments. Please provide analogous statistics for MuJoCo, Meta-world, CLEVR-Robot, and LIBERO, or explicitly argue why these metrics are not needed for those environments; without this, the headline average gap may simply reflect where LLM rollouts are poor rather than a general algorithmic limitation.","section":"Sec. 5.2, Table 2"},{"comment":"The headline comparison (35.44% vs. 64.37% on hard tasks) compares training on noisy LLM rollouts with training on oracle real rollouts for the same novel tasks. Because the imaginary rollouts are known to be only 25.8% goal-consistent (Table 2), the observed gap does not isolate an algorithmic limitation. Please add a control that trains with a quality-filtered subset of imaginary rollouts (e.g., only goal-consistent rollouts) or with synthetic rollouts of matched quality, to distinguish data-quality effects from algorithmic insufficiency.","section":"Sec. 5.4, Fig. 6"},{"comment":"The claim that 'policies augmented with LLM-generated imaginary rollouts exhibit consistently higher performance on novel tasks' is not supported by the displayed bars; many pairs show w/ IR below w/o IR (e.g., BabyAI training, CQL on Meta-world rephrasing, BC on CLEVR-Robot easy). Please provide a quantitative summary across all tasks and algorithms, including error bars and significance tests, and temper or qualify the claim accordingly.","section":"Sec. 5.3, Fig. 5"},{"comment":"The 'Real' condition is not an algorithm baseline but an oracle that has access to real rollouts for the novel tasks; it is useful as an upper bound but should be described as such. The current presentation implies it is a competitor, which conflates the benchmark's evaluation question with a data-availability assumption.","section":"Sec. 5.4, Fig. 6"}],"minor_comments":[{"comment":"Please fix typographical and formatting errors, including 'encodeing' (Sec. 4.1), 'culmulative' (Sec. 3), 'CLRVR-Robot' (Table 1), and '1,3000' (Table 1).","section":"Throughout"},{"comment":"The evaluation metrics ('5cm positional accuracy', '85% semantic consistency') are environment-specific free parameters; please justify these thresholds or cite prior conventions so that results are reproducible.","section":"Sec. 4.3"},{"comment":"The figure contains stray '/uni000000...' tokens in the subfigure captions; these should be removed in the camera-ready version.","section":"Fig. 5"},{"comment":"The sentence 'Each training batch uniformly samples equal proportions of data from the real and LLM-imaginary datasets' should specify whether this holds across difficulty levels and how the real dataset's size is matched to the imaginary dataset's size.","section":"Sec. 5.1"},{"comment":"Related work covers KALM, URI, AgentTrek, and InCLET; consider also discussing recent benchmarks for language-conditioned RL to clarify the novelty of ImagineBench relative to those efforts.","section":"Sec. 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript describes a likely useful benchmark and the empirical infrastructure seems solid, but the central claim needs stronger empirical support. The quality metrics in Table 2 already suggest a confound, so adding per-environment quality statistics and quality-filtered controls is essential. If the code and datasets are fully released, this could become a standard evaluation suite for RLIM; the current version is not yet ready for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the resource itself is real: paired real and imaginary rollout datasets across five environments, hierarchical task levels, baseline runs with seven offline RL algorithms, and public code. That is a genuinely useful contribution to the small but growing RLIM subfield. Second, the headline result—that existing offline RL algorithms only hit 35.44% on hard tasks versus 64.37% with real rollouts—is not yet supported as an algorithmic limitation, because the imaginary rollouts used for hard tasks are of poor quality and the paper provides no control for that.\n\nWhat it does well: the benchmark design is thoughtful. Real and imaginary datasets are paired, instructions are split into training/rephrasing/easy/hard levels, and the evaluation protocol is explicit (5cm positional accuracy, 85% semantic consistency, etc.). Running seven baselines across five domains is real work. The code and data being released matters. The KALM self-citation is fine; this is a benchmark paper extending that line, and the debt is properly acknowledged.\n\nThe soft spots are concentrated in the interpretation. Table 2 measures rollout quality only in BabyAI, and there the hard-task numbers are bad: 25.8% goal consistency, 72.9% transition correctness, 66.8% legality. Training mostly on off-goal or illegal synthetic data will naturally underperform training on real rollouts. Without a quality-filtered baseline—e.g., using only the goal-consistent imaginary rollouts, or weighting by a quality score—the 35.44% vs 64.37% gap cannot be assigned to algorithmic deficiency. The paper's own Section 6 says LLM imagination quality is a limiting factor, which undercuts the stronger reading of the gap. The claim that imaginary rollouts give \"consistently higher performance\" on novel tasks is also too strong: in Figure 5, several w/IR bars are below their w/oIR counterparts (MuJoCo and Meta-world show mixed results). Minor but real: Table 1 has a typo in the LIBERO hard imaginary rollout count (\"1,3000\"), and the quality statistics section should at least state that only BabyAI was measured.\n\nWho is this for? Anyone developing offline RL algorithms for language-conditioned policies, or researching ways to filter and weight LLM-generated experience. They will want the datasets. The paper deserves a serious referee: the benchmark contribution is solid and the interpretation of the gap needs to be fixed, not discarded. I would send it out, with a request for quality-filtered or quality-weighted baselines, either a corrected consistency claim or a careful reanalysis of Fig. 5, and a note about the dataset typo. After those revisions it could become a standard reference in this niche.","headline":"A genuinely useful benchmark resource for RL from LLM rollouts, but the headline algorithm-gap claim is confounded by low rollout quality and needs a quality-filtered control before it will convince anyone.","tokens_in":15869,"tokens_out":1533,"would_cite":true,"duration_ms":16888,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces ImagineBench, the first comprehensive benchmark for reinforcement learning with LLM-imaginary rollouts, and reports that existing offline RL algorithms reach only 35.44% success on hard unseen tasks versus 64.37% when…","keywords":["imaginary rollouts","reinforcement learning","large language models","offline RL","benchmark","language-conditioned policy","task generalization"],"falsifier":"Replace the Llama-2-7b generator with a different fine-tuned LLM, or add a simple filter that discards imaginary rollouts failing the paper's own consistency and legality checks, and rerun the hard-task benchmarks. If hard-task success rises to the 64.37% real-rollout level under either change, the reported 35.44% figure is driven by rollout generation quality rather than by a limitation of existing offline RL algorithms.","tokens_in":14869,"feed_emoji":"🤖","tokens_out":6371,"duration_ms":57145,"temperature":0.7,"pith_summary":"The paper introduces ImagineBench, which it calls the first comprehensive benchmark for reinforcement learning with LLM-imaginary rollouts: synthetic trajectories an LLM generates after being fine-tuned on real environment data. The benchmark supplies paired real and imaginary datasets covering locomotion, robotic manipulation, and navigation, with natural-language instructions at four difficulty levels. Using six offline RL algorithms plus SAC, the paper finds that adding imaginary rollouts generally improves performance on unseen tasks but leaves a clear gap: 35.44% success on hard tasks versus 64.37% when the same methods train on real rollouts. The central claim is that RLIM is promising but lacks both a standard evaluation resource and algorithms suited to hybrid real-imaginary data, and that ImagineBench supplies the former to accelerate work on the latter.","feed_headline":"First benchmark for RL from LLM rollouts exposes a 29-point gap","feed_subtitle":"ImagineBench provides five environments and paired real/imaginary datasets to attack the gap on unseen tasks.","key_machinery":"The load-bearing object is the RLIM pipeline itself. An LLM is fine-tuned on real rollout-instruction pairs through three supervised tasks, namely dynamics prediction, rollout explanation, and rollout generation, and is then prompted with a goal-oriented prompt to produce imaginary rollouts for novel goals. ImagineBench wraps this pipeline in standardized assets: real and imaginary datasets per environment, natural-language instructions, success-rate evaluation functions, and a four-level task taxonomy of training, rephrasing, easy, and hard tasks. The taxonomy is what makes the benchmark's claim testable: it separates preserved skills from genuine generalization and forces harder tasks to require composition of behaviors.","core_discovery":"The central discovery is that reinforcement learning from imaginary rollouts works well enough to be worth benchmarking, but existing offline RL algorithms are not yet good at using LLM-imaginary rollouts. On unseen tasks, policies trained with imaginary rollouts outperform policies trained without them in most environments, yet they reach only 35.44% average success on hard tasks compared with 64.37% for policies trained on real rollouts collected for the same novel goals. The paper also reports that imaginary-rollout quality degrades with task difficulty: in BabyAI, goal consistency falls from 88.0% on rephrased goals to 25.8% on hard goals, while transition correctness and legality stay above roughly 65%. This quality degradation, together with the performance gap, motivates the paper's agenda: better offline RL algorithms for imaginary data, filtering of low-quality rollouts, fast online adaptation, and multi-modal imagination.","pith_inferences":["The rollout-quality statistics are reported only for BabyAI; if the same consistency and legality measurements were made in Meta-world, LIBERO, and MuJoCo, the hard-task gap might shrink or grow, so the 35.44% versus 64.37% figure should be read as tied to the Llama-2-7b fine-tuning pipeline.","A testable extension is to use the benchmark to compare different LLM backbones or fine-tuning recipes; if a stronger LLM raises hard-task success toward the real-rollout level, the bottleneck is imagination quality rather than offline RL algorithm design.","The paper leaves implicit the natural next step of turning ImagineBench into an online-adaptation benchmark, where a policy trained on imaginary rollouts is fine-tuned with a small budget of real interactions to directly measure the fast-adaptation direction it lists as future work.","Because the benchmark uses vectorized state spaces, its instructions may not transfer to pixel observations; extending RLIM to vision-language models would require new state encoders and evaluation functions that the current benchmark does not provide."],"forward_implications":["Adding LLM-imaginary rollouts to real data improves success on unseen tasks for most offline RL algorithms, so LLM-generated experience is a usable source of synthetic training data.","Existing offline RL algorithms leave a large gap on hard tasks, so algorithms designed specifically for hybrid real-imaginary data are needed rather than simply more data.","The benchmark's hierarchical task levels provide a common protocol for comparing future RLIM methods.","The quality statistics imply that imaginary rollouts are most reliable for rephrasing tasks and degrade as goals require composition, pointing to rollout filtering as a concrete algorithmic lever.","The public datasets lower the computational cost of developing new RLIM algorithms."],"supporting_citations":[{"why":"Supplies KALM, the RLIM recipe of supervised fine-tuning for dynamics prediction, rollout explanation, and rollout generation that ImagineBench's imaginary-rollout collection follows.","marker":"[Pang et al., 2024b]"},{"why":"Shows LLMs can generate control trajectories from tutorial texts, a prior RLIM demonstration that the benchmark standardizes.","marker":"[Chen et al., 2024]"},{"why":"Extends RLIM-style synthetic rollouts to browser automation, one of the inconsistent evaluation protocols ImagineBench responds to.","marker":"[Xu et al., 2025]"},{"why":"Provides Meta-World, a benchmark environment and source of real rollouts used in ImagineBench.","marker":"[Yu et al., 2019]"},{"why":"Provides BabyAI, the navigation environment used for the imaginary-rollout quality statistics in Table 2.","marker":"[Chevalier-Boisvert et al., 2019]"},{"why":"Provides LIBERO, whose Object suite and novel sequential tasks appear in the benchmark.","marker":"[Liu et al., 2023]"},{"why":"Supplies MuJoCo, the physics simulator and HalfCheetah locomotion environment in the benchmark.","marker":"[Todorov et al., 2012]"},{"why":"Supplies CLEVR-Robot, the object-manipulation environment with discrete actions in the benchmark.","marker":"[Research, 2019]"},{"why":"Contributes CQL, a core offline RL baseline whose performance on hybrid real-imaginary data the benchmark reports.","marker":"[Kumar et al., 2020]"},{"why":"Contributes BCQ, an early conservative offline RL method evaluated across the benchmark.","marker":"[Fujimoto et al., 2019]"}],"fun_headline_variants":["Imaginary rollouts: RL gap of 29 points on hard tasks","New benchmark shows RL lags on LLM-generated rollouts","LLM rollouts: existing RL algorithms miss 29-point mark","ImagineBench: first benchmark for LLM-rollout RL","Hard tasks expose 29-point RL gap with imaginary data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that LLM-imaginary rollouts produced by the paper's specific Llama-2-7b fine-tuning recipe are representative of LLM-imaginary rollouts in general; if those rollouts are unusually good or bad, the measured algorithm gap is an artifact of that pipeline rather than a property of offline RL algorithms.","fun_headline_variants_meta":{"raw":{"variants":["Imaginary rollouts: RL gap of 29 points on hard tasks","New benchmark shows RL lags on LLM-generated rollouts","LLM rollouts: existing RL algorithms miss 29-point mark","ImagineBench: first benchmark for LLM-rollout RL","Hard tasks expose 29-point RL gap with imaginary data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000896,"raw_usage":{"total_tokens":3888,"prompt_tokens":998,"completion_tokens":2890,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":2801}},"tokens_in":614,"tokens_out":2890,"duration_ms":22612,"temperature":1.0,"reasoning_tokens":2801,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:18:29.679521+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the Llama-2-7b generator with a different fine-tuned LLM, or add a simple filter that discards imaginary rollouts failing the paper's own consistency and legality checks, and rerun the hard-task benchmarks. If hard-task success rises to the 64.37% real-rollout level under either change, the reported 35.44% figure is driven by rollout generation quality rather than by a limitation of existing offline RL algorithms.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows LLMs can generate control trajectories from tutorial texts, a prior RLIM demonstration that the benchmark standardizes."},{"cited_title":"H., and Bengio, Y","cited_arxiv_id":null,"evidence_quote":"Provides BabyAI, the navigation environment used for the imaginary-rollout quality statistics in Table 2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides LIBERO, whose Object suite and novel sequential tasks appear in the benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MuJoCo, the physics simulator and HalfCheetah locomotion environment in the benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies CLEVR-Robot, the object-manipulation environment with discrete actions in the benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes CQL, a core offline RL baseline whose performance on hybrid real-imaginary data the benchmark reports."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes BCQ, an early conservative offline RL method evaluated across the benchmark."}],"review_version":1}