{"id":"80f38d90-f1a1-4b53-9545-cf103c84e074","arxiv_id":"2603.11395","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ARROW, a DreamerV3-based continual RL method with dual distribution-matching replay buffers, reduces forgetting on sequential Atari while matching forward transfer on Procgen CoinRun variants.","lead":"ARROW extends DreamerV3 with a dual short-term/long-term replay buffer so a world model forgets less when tasks arrive sequentially. Continual RL agents that scale without huge memory would matter for lifelong robotics and game agents.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The dual-buffer claim rests on an unauditable sampling procedure whose ability to preserve task diversity without labels is not demonstrated by the recoverable evidence.","rationale":"The reader's weakest_assumption correctly isolates the unlabeled distribution-matching premise that distinguishes ARROW from FIFO DreamerV3. The corrupted extract prevents verification of the sampling equations, tables, error bars, and ablations that would support or refute that premise, so the concern remains live. No stronger internal inconsistency is visible in the recoverable abstract and structure; the contribution is a scoped methods extension with plausible neuroscience motivation. Therefore the verdict stays CONDITIONAL (pending reproducible artifacts and the proposed ablation) rather than moving to REJECT or ACCEPT. Agreement with the reader is full on the identity of the load-bearing soft spot.","tokens_in":30267,"tokens_out":432,"duration_ms":4997,"concrete_test":"Recover or re-implement the long-term sampling rule exactly as described; run an ablation that replaces distribution-matching with uniform or random retention inside a same-size dual buffer on the sequential Atari suite; if forgetting metrics become statistically indistinguishable from FIFO DreamerV3, the diversity-preservation premise fails and the strongest claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim (less forgetting on sequential Atari at matched buffer size, comparable forward transfer on Procgen) is defined relative to FIFO DreamerV3 by the dual short/long-term design. The long-term buffer is asserted to preserve task diversity via distribution-matching sampling that does not use task labels or boundaries. Because the manuscript extract is heavily corrupted, the precise sampling criterion, distance metric, update rule, and any ablation that isolates this component from mere dual-buffer capacity cannot be recovered or checked. Without that, it is possible that the reported gains arise from capacity allocation or hyperparameter tuning rather than from the claimed diversity-preserving mechanism. The premise that unlabeled distribution matching actually tracks task diversity (rather than collapsing to a non-representative subset) is therefore the single least-secure load-bearing condition for the central empirical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes ARROW, a model-based continual RL method that extends DreamerV3 with a dual replay architecture: a short-term buffer for recent experience and a fixed-size long-term buffer whose contents are maintained by distribution-matching sampling intended to preserve task diversity without task labels or boundaries. Motivated by neuroscience-style replay into a world model rather than directly into the policy, ARROW is evaluated in two sequential settings—Atari games (little shared structure) and Procgen CoinRun variants (shared structure allowing transfer)—and is claimed to exhibit substantially less forgetting than model-free and model-based baselines with the same total replay capacity, while maintaining comparable forward transfer on the structured setting.","tokens_in":30521,"tokens_out":1229,"duration_ms":16454,"significance":"If the dual-buffer, unlabeled distribution-matching design is shown to be the causal driver of reduced forgetting at matched memory budgets, the work would be a useful contribution to continual model-based RL: it targets the practical memory bottleneck of replay, couples replay to a world model in a bio-inspired way, and reports gains on both non-transfer and transfer continual settings. Strengths include an explicit same-capacity comparison axis against FIFO DreamerV3 and model-free baselines, and a clear problem framing (scalability of replay under sequential tasks). The significance is currently limited by narrow environment coverage (Atari subset + CoinRun variants), dependence on the DreamerV3 stack, and incomplete isolation of the claimed diversity-preserving sampling rule from capacity allocation and other free parameters.","major_comments":[{"comment":"The central empirical claim (less forgetting at matched total buffer size) is defined relative to FIFO DreamerV3 by the long-term distribution-matching buffer. The manuscript does not make the sampling criterion, distance/feature space, insertion/eviction rule, and short-vs-long sampling ratio fully auditable as a reproducible algorithm. Without a precise statement of this procedure (and any dependence on latent features of the world model), it is not possible to verify that the buffer tracks task diversity rather than collapsing to a non-representative subset—the load-bearing premise of ARROW versus plain dual capacity.","section":"Method / dual-buffer design"},{"comment":"There is no ablation that isolates distribution-matching sampling from (i) simply splitting capacity into two FIFO buffers and (ii) alternative long-term selection rules (e.g., reservoir, random, surprise-based). Same total capacity is treated as the primary fair axis, but short-term capacity, long-term capacity, and the sampling ratio are free parameters. Without these controls, reported gains on Atari forgetting may be attributable to capacity allocation or hyperparameter search rather than the claimed diversity-preserving mechanism.","section":"Experiments / ablations"},{"comment":"Evaluation is restricted to an Atari subset (tasks without shared structure) and Procgen CoinRun variants (shared structure). Continual RL conclusions about forgetting vs. forward transfer are sensitive to task order, number of tasks, and domain diversity. The manuscript should either expand the suite (more Atari games, additional Procgen games, or a standard continual RL benchmark with reported task sequences) or substantially qualify the scope of the claim that ARROW is robust across the two continual regimes.","section":"Experiments / evaluation settings"},{"comment":"Metrics for forgetting and forward transfer must be defined explicitly (e.g., average performance drop after subsequent tasks; normalized transfer relative to single-task or from-scratch baselines) and reported with variance over seeds and, where applicable, task orders. The abstract’s “substantially less forgetting” / “comparable forward transfer” language needs corresponding tables with effect sizes and statistical support; otherwise the strongest claim remains only directionally supported.","section":"Results / metrics"}],"minor_comments":[{"comment":"Neuroscience inspiration is motivational; keep it clearly separated from algorithmic claims so readers do not treat hippocampal replay as evidence for the specific dual-buffer rule.","section":"Introduction"},{"comment":"List all free parameters (buffer capacities, sampling ratio, DreamerV3 hyperparameters held fixed vs. tuned) in one place, and state the total memory budget in comparable units (transitions / bytes) across ARROW and baselines.","section":"Method / experimental setup"},{"comment":"Clarify whether task boundaries or labels are ever used at train or eval time; the abstract stresses unlabeled operation, which should be restated in the experimental protocol.","section":"Experiments"},{"comment":"Related work should more sharply position ARROW against other continual model-based RL and replay-selection methods (reservoir, generative replay, prioritization) so the novelty of unlabeled distribution matching for world-model replay is clear.","section":"Related work"},{"comment":"The provided manuscript extract is heavily corrupted (garbled characters, broken equations). Ensure the camera-ready source is clean, with numbered equations for the sampling objective and readable figures for learning curves and forgetting metrics.","section":"Presentation"}],"recommendation":"major_revision","confidential_remarks":"The extract available for review is severely garbled, which reduces confidence in fine-grained checks of the sampling equations and tables; the recommendation of major_revision assumes the authors can supply a clean manuscript with the missing algorithmic detail and ablations. Scope is a solid methods contribution for a continual-RL / model-based RL venue if those gaps are closed; without isolation of the distribution-matching rule, the paper risks reading as a DreamerV3 dual-buffer engineering note rather than a mechanism result."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: ARROW is a clean, scoped extension of DreamerV3 for continual RL—short-term FIFO plus a long-term buffer kept by distribution-matching sampling—and the abstract-level claim is that this cuts forgetting on sequential Atari at matched buffer size while keeping forward transfer on Procgen CoinRun variants. That is real subfield progress, not a theory rewrite.\n\nWhat is new is the combination: replay into the world model (not the policy), dual buffers, and unlabeled distribution matching to preserve task diversity without task IDs. The neuroscience framing is motivational and fine. They compare against model-free and model-based baselines with the same total buffer size, which is the right control. The two settings (no shared structure vs shared structure) are a sensible split for forgetting vs transfer.\n\nSoft spots, in proportion. The manuscript extract is badly corrupted, so the exact sampling criterion, distance metric, update rule, and ablations that isolate “distribution matching” from mere dual capacity are hard to audit. That is the load-bearing design choice; if gains come from capacity allocation or free hyperparameters (buffer sizes, short/long mix, Dreamer defaults), the story weakens. Benchmarks are narrow—Atari subset and CoinRun variants only—and code release is not clear from what we have. None of that makes the central idea incoherent; it just means the evidence is thinner than a top-venue methods paper usually needs.\n\nNo circular math, no invented physics. Free parameters are the usual buffer and sampling knobs. Citation pattern looks standard for Dreamer/continual RL.\n\nWho it is for: people already working on model-based continual RL or memory-efficient replay. A reading group that cares about lifelong agents would get a useful discussion out of the dual-buffer design and the Atari forgetting numbers. I would not cite it in the next year unless I am deep in that niche, but a serious editor should send it to referees rather than desk-reject. Ask for clearer sampling equations, ablations, broader sequences, and artifacts. Worth engaging if you work in the area; skip if you only want foundational RL theory.","headline":"Solid scoped methods paper: DreamerV3 plus dual short/long-term distribution-matching replay for continual RL, with useful but narrow empirical claims that still deserve a referee.","tokens_in":31153,"tokens_out":541,"would_cite":false,"duration_ms":9499,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"ARROW reduces catastrophic forgetting in continual RL by replaying experiences into a world model via dual short- and long-term buffers.","keywords":["continual reinforcement learning","world models","experience replay","catastrophic forgetting","DreamerV3","distribution matching","model-based RL","replay buffers"],"falsifier":"On the same sequential Atari suite, swap ARROW’s long-term distribution-matching sampler for random or FIFO sampling of equal capacity and check whether average forgetting rises to the level of the equal-size DreamerV3 FIFO baseline.","tokens_in":31132,"feed_emoji":"🧠","tokens_out":854,"duration_ms":17896,"temperature":0.7,"pith_summary":"Continual reinforcement learning asks agents to acquire new skills without erasing old ones, yet standard fixed-size replay buffers still forget and grow expensive in memory. This paper claims that replaying experiences to a predictive world model—rather than only to the policy—can retain past performance with the same memory budget, following a pattern known from neuroscience. ARROW extends DreamerV3 with two complementary buffers: a short-term buffer of recent experience and a long-term buffer whose contents are kept diverse by distribution-matching sampling instead of simple first-in-first-out replacement. On sequential Atari games that share little structure, the method forgets substantially less than model-free and model-based baselines that use equal-sized FIFO buffers, while matching forward transfer on related Procgen CoinRun variants. The result points to model-based agents plus carefully curated replay as a practical path to more scalable continual RL.","feed_headline":"Dual buffers cut forgetting in continual RL agents","feed_subtitle":"Same-size memory keeps past Atari skills by matching experience diversity into a world model.","key_machinery":"ARROW’s dual replay buffer: a short-term buffer holding recent experience and a long-term buffer that preserves task diversity by distribution-matching sampling rather than FIFO. Together they feed the world model so imagined rollouts remain accurate across task switches without expanding memory or requiring task labels.","core_discovery":"Compared with model-free and model-based baselines that use replay buffers of the same size, ARROW—an extension of DreamerV3 with a dual short-term and long-term distribution-matching buffer—exhibits substantially less forgetting on sequential tasks without shared structure (Atari) while maintaining comparable forward transfer on tasks with shared structure (Procgen CoinRun variants).","pith_inferences":["The same dual-buffer pattern could transfer to other generative world-model architectures beyond Dreamer-style agents.","If long-term sampling collapses under much longer task sequences, adding surprise or coverage criteria would be a direct control experiment.","Gains that are stronger on unstructured than structured tasks imply the method mainly shields against interference rather than amplifying transfer.","The approach may also cut memory in offline-to-online continual settings where a single FIFO buffer already saturates."],"forward_implications":["Equal memory budgets can yield lower forgetting on disjoint sequential tasks when replay preserves diversity rather than recency alone.","World-model training benefits more from diversity-preserving replay than policy-only training under the same capacity.","Continual agents need not observe task boundaries if sampling tracks distribution shift well enough to stabilize the world model.","Replay directed at a predictive model, rather than only at the policy, is a usable design lever for reducing interference.","Compact dual buffers make further scaling of model-based continual RL more practical than large undifferentiated FIFO stores."],"fun_headline_variants":["Dual buffers slash forgetting in continual RL world models","ARROW dual replay keeps Atari skills with same-size memory","Short- and long-term buffers cut forgetting vs same-size baselines","Distribution-matching dual buffers curb skill loss in DreamerV3","ARROW dual-buffer world models retain past tasks better on Atari"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That a fixed-size long-term buffer, updated only by distribution-matching sampling without task labels or boundaries, will keep a representative mix of past tasks and not collapse to a non-representative subset.","fun_headline_variants_meta":{"raw":{"variants":["Dual buffers slash forgetting in continual RL world models","ARROW dual replay keeps Atari skills with same-size memory","Short- and long-term buffers cut forgetting vs same-size baselines","Distribution-matching dual buffers curb skill loss in DreamerV3","ARROW dual-buffer world models retain past tasks better on Atari"]},"model":"grok-4.5","effort":"low","cost_usd":0.004628,"raw_usage":{"total_tokens":1352,"prompt_tokens":775,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":46280000,"prompt_tokens_details":{"text_tokens":775,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":507,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":775,"tokens_out":70,"duration_ms":4620,"temperature":1.0,"reasoning_tokens":507,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T22:56:00.383492+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same sequential Atari suite, swap ARROW’s long-term distribution-matching sampler for random or FIFO sampling of equal capacity and check whether average forgetting rises to the level of the equal-size DreamerV3 FIFO baseline.","supporting_citations":[],"review_version":2}