{"id":"c5a0f29c-0aa1-49ca-8f88-004eb53314a7","arxiv_id":"2411.19787","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CAREL improves instruction-following RL sample efficiency by aligning observation sequences with instruction tokens via an X-CLIP style contrastive loss and masking completed subtasks.","lead":"A new training recipe, CAREL, adds a video-text retrieval style contrastive loss to instruction-following reinforcement learning, plus a mechanism that masks already completed parts of a natural language instruction. In MiniGrid/BabyAI grid-world tasks, the approach reports faster learning and better generalization on unseen goal combinations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The success-reward threshold in §2.1 is never reported; if it was tuned per task, the reported sample-efficiency gains (Table 1) may reflect threshold choice rather than CAREL's mechanism.","rationale":"The reader's weakest_assumption points to the unspecified success-reward threshold, and I concur that this is the most load-bearing soft spot. The method's auxiliary loss is only defined on episodes that pass this threshold; the entire contrastive alignment signal depends on a correct separation of aligned and misaligned trajectory-instruction pairs. If the threshold is miscalibrated, the auxiliary loss can actively harm learning, yet the paper gives no value and no ablation over it. This is not merely a presentation issue: the central empirical claim, superior sample efficiency, is supported by point estimates at selected frame counts, and an unreported hyperparameter that controls which experiences feed the auxiliary loss could be responsible for a substantial part of the observed gap. The paper does have genuine strengths: the X-CLIP-style loss is clearly motivated, the instruction-tracking heuristic is novel, and the ablations in A.2.1-A.2.4 show the auxiliary coefficient and tracking components matter. The two-seed statistics and the vague 'halfway point of convergence' rule already weaken the quantitative support, but the threshold omission is more fundamental because it hides a core design choice. I therefore agree with the reader's CONDITIONAL verdict: with the threshold specified and a sensitivity analysis provided, the claim would be testable; without it, the results are not fully reproducible and the mechanism's contribution is not isolated.","tokens_in":13188,"tokens_out":12265,"duration_ms":110351,"concrete_test":"Obtain the code or ask the authors for the exact success-threshold fraction from §2.1, then run GoToSeqS5R2 and OpenDoorsOrderN4 with the threshold set to 0.0, 0.5, 0.9, and 1.0 times the maximum achievable reward while keeping all other hyperparameters fixed, and compare success rates at the exact frame counts used in Table 1 (10M and 3M). If the CAREL-vs-baseline gap changes by more than the two-seed run-to-run variance across threshold values, the reported gain is threshold-dependent and the paper must disclose and justify the chosen value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 defines success for the auxiliary loss as exceeding 'a fraction of the maximum achievable reward' but never gives the fraction. The X-CLIP loss in Eq. 12 uses successful episodes as the only positives; negatives are all other episodes in the same batch. If the threshold is too permissive, near-failed episodes with marginal rewards enter the positive set, so observation sequences are not cleanly aligned with the instruction and the contrastive signal is corrupted. If too strict, the positive batch shrinks and the auxiliary gradient becomes noisy or absent early in training. Because the headline gains (Table 1: 73% vs 41% on GoToSeqS5R2 at 10M frames; 79% vs 24% on OpenDoorsOrderN4 at 3M frames) are point estimates over two seeds, a threshold tuned to favor CAREL on these tasks cannot be ruled out. No sensitivity analysis is provided, and the abstract's link to code does not appear in the full text, so the central sample-efficiency claim currently rests on an unreported hyperparameter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces CAREL, a training-time framework for instruction-following reinforcement learning. It augments a base RL agent with a multi-grained X-CLIP-style contrastive loss that aligns successful episode-observation sequences with their textual instructions (Eq. 12), and an 'instruction tracking' mechanism that masks completed sub-tasks from the instruction once their similarity to the partial episode spikes (Eq. 14). The paper reports experiments on BabyAI with the BabyAI baseline, on MiniGrid with a modified SHELM baseline, and comparisons with LISA, and claims superior sample efficiency and systematic generalization. An appendix provides ablations for the auxiliary-loss coefficient, action embeddings, similarity-score choice, and the interaction between CAREL and instruction tracking.","tokens_in":13399,"tokens_out":6314,"duration_ms":59687,"significance":"Language grounding in RL is an important and active problem, and CAREL is a simple, architecture-agnostic proposal: a video-text-retrieval-style auxiliary loss plus a goal-masking heuristic. The manuscript is self-contained in its derivation of the auxiliary loss, gives algorithmic pseudocode for both components, and includes several relevant ablations (lambda_C, action embeddings, choice of similarity score, and CAREL-plus-IT interaction). The reported gains, such as 73% vs 41% at 10M frames on GoToSeqS5R2, are substantial. However, the central empirical claim is currently supported by point estimates from only two seeds for the main BabyAI results, an unreported success-reward threshold, and an undefined sampling-budget convention. The significance of the contribution is therefore conditional on the requested revisions.","major_comments":[{"comment":"The success-reward threshold used to select episodes for the auxiliary loss is never quantified. The text states only that a trajectory is successful if its aggregated reward exceeds 'a fraction of the maximum achievable reward', but the fraction is not reported for any task. This is load-bearing because the contrastive positives in Eq. (12) are exactly these successful episodes: a permissive threshold includes poorly aligned trajectories and corrupts the contrastive signal, while a strict threshold shrinks the positive batch and can eliminate the auxiliary gradient early in training. Please report the threshold used for every environment, state whether it was tuned per task, and provide a sensitivity analysis (e.g., sweeping the fraction) or an ablation that removes the threshold dependence.","section":"§2.1"},{"comment":"The main quantitative results in Table 1 are point estimates over two random seeds with no error bars or per-seed values, whereas Table 2 reports standard deviations. The reported differences are large, but with two seeds one cannot assess whether the ordering is stable. Please report all individual runs, add at least five seeds per condition, and include standard errors or confidence intervals; if computational cost is a concern, provide the raw curves in an appendix.","section":"§3.1, Table 1"},{"comment":"The sample budgets in Table 1 are said to be 'roughly the halfway point of the samples required for model convergence', but no convergence criterion is defined. This is not reproducible and, if chosen post hoc, can exaggerate the method's advantage. The same issue appears in the SHELM comparison ('the midpoint of convergence') and in Table 3. Please define the convergence rule (e.g., a threshold on the moving average of success rate over a fixed window), report the resulting convergence points for each method, and show the full learning curves with variance bands.","section":"§3.1.1"},{"comment":"The LISA comparison in Table 3 is not an equivalent sample-efficiency comparison. LISA is an imitation-learning method that consumes 100k expert trajectories (reported as 7M and 1.5M frames), while CAREL is trained online from environment interaction. Reporting both in the same 'Frames' column conflates environment steps with demonstration data and obscures differences in supervision. Please compare on a common data axis (e.g., number of trajectories), make explicit the supervision source for LISA, and, if possible, include a supervised or offline baseline trained on the same number of expert demonstrations.","section":"§3.2, Table 3"},{"comment":"The full CAREL method in Figure 3 differs from the vanilla CAREL baseline by both instruction tracking and the addition of action embeddings to the observation representations. Consequently, the improvement attributed to instruction tracking is confounded with a representation change. The ablation in Appendix A.2.2 separates action embeddings only for CAREL+IT, so please also report CAREL with action embeddings but without IT, and BabyAI+IT with and without action embeddings, so the contribution of each component can be isolated.","section":"§3.1.2"}],"minor_comments":[{"comment":"Line 6 of Algorithm 2 writes S_t_ci <- -mean(V_ci x~_t) with a minus sign, but Eq. (14) and the surrounding text describe a positive spike in similarity; the sign should be fixed.","section":"Algorithm 2"},{"comment":"The abstract says 'Our code base is available here', but the full text contains no URL or repository identifier; please provide a working link.","section":"Abstract"},{"comment":"The text states that all results are smoothed before plotting, but the smoothing window is not specified; please report it or provide the unsmoothed curves.","section":"Figures 2-4"},{"comment":"The slice notation [.,1] and [1,.] is used without definition; it should be defined explicitly, for example as selecting the column or row of the similarity matrix.","section":"Eqs. (5)-(9)"},{"comment":"The evaluation protocol for 'systematic generalization' is not described in detail; please state how the unseen-task split is constructed and which color/type combinations are held out.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has already appeared in TMLR according to the header, but I reviewed the submitted content as is. The requested revisions focus on reproducibility of the core results: the unreported success threshold, the two-seed main evaluation, and the undefined convergence-budget convention are all fixable without changing the method. I do not see a derivation-level circularity, but the self-referential threshold in Eq. (14) deserves an explicit sensitivity check if instruction tracking is to be presented as a generally usable component."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is sound and worth taking seriously. CAREL transfers X-CLIP's multi-grained contrastive objective to instruction-following RL by treating successful episodes as positive pairs, and it adds an instruction-tracking mechanism that masks completed subtasks when a similarity spike is detected (Eq. 14). That masking heuristic is the genuinely new piece, and the gains on sequential tasks like GoToSeqS5R2 and OpenDoorsOrderN4 are substantial and in line with what the mechanism is supposed to do. The ablations on the loss coefficient, action embeddings, and similarity score choice are useful, and the SHELM comparison with error bars gives some confidence that the effect generalizes beyond BabyAI's own baseline.\n\nThe soft spots are mostly about reporting discipline. The success-threshold fraction in Section 2.1 is never given, and the stress-test note is right that this is a potential source of tuning. If the threshold was adjusted per task, the headline numbers could reflect that choice rather than the method's mechanism. That does not sink the paper, because the improvements appear across several tasks and the appendix shows robustness to the loss coefficient, but it is a real reproducibility gap. The main BabyAI results rely on two seeds with no error bars, and Table 1 reports success at an undefined \"halfway point\" of convergence, which makes the numbers hard to interpret. The code link is mentioned in the abstract but no URL appears in the full text, so I could not check the implementation. These are addressable issues, not fatal ones.\n\nThe paper is aimed at researchers working on language-conditioned RL, especially those using BabyAI or similar instruction-following benchmarks. The instruction-tracking idea is worth citing as a baseline or inspiration. If I were handling this as an editor, I would send it to review, but I would insist on reporting the success threshold, adding more seeds or at least error bars on the main curves, and defining the convergence-reporting rule. As it stands, treat the sample-efficiency gains as promising but not fully pinned down.","headline":"A solid empirical paper with a genuinely new tracking heuristic; the sample-efficiency claims are plausible but the reporting of key hyperparameters needs to be tighter.","tokens_in":13949,"tokens_out":2502,"would_cite":true,"duration_ms":23223,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CAREL claims that an X-CLIP-style cross-modal contrastive auxiliary loss plus instruction masking gives instruction-following RL agents a substantial boost in sample efficiency and systematic generalization, with reported gains like 73%…","keywords":["reinforcement learning","instruction following","cross-modal alignment","contrastive learning","X-CLIP","sample efficiency","systematic generalization","goal-conditioned RL"],"falsifier":"Train CAREL on GoToSeqS5R2 with the success threshold deliberately set to 0% of maximum reward (all trajectories become positives) and separately to 100%; if the success-rate curves remain close to Table 1, the threshold is not what drives the gains, while if they collapse, the unspecified threshold is a critical hyperparameter. A second falsifier targets instruction tracking: replace the moving-average spike rule of Eq. 14 with random masking of one subtask per episode under the same probability schedule; if random masking matches the CAREL+IT curve, the similarity-based tracking is not the source of the improvement.","tokens_in":13004,"feed_emoji":"🎯","tokens_out":6902,"duration_ms":51774,"temperature":0.7,"pith_summary":"This paper argues that instruction-following reinforcement learning agents learn faster and generalize better when the model is trained, in addition to the reward signal, to align its observations with the textual instruction through a contrastive cross-modal auxiliary loss. The key claim is that borrowing a multi-grained alignment objective from video-text retrieval, where the instruction plays the role of the text and the observation sequence the role of the video, gives the agent a grounding signal that reward alone fails to provide. Combined with a tracking mechanism that masks completed parts of the instruction, the framework is reported to raise success rates substantially at fixed frame budgets, for instance from 41% to 73% on a sequenced goal-reaching task and from 24% to 79% on an ordered door-opening task. If correct, this is a training-time recipe that can be bolted onto existing dual-encoder agents to make them more sample-efficient without changing the environment or the reward structure.","feed_headline":"Auxiliary text-vision loss lifts RL success up to 3x","feed_subtitle":"A contrastive alignment loss plus instruction masking makes instruction-following agents learn faster on unseen tasks.","key_machinery":"The engine of CAREL is a contrastive cross-modal loss adapted from X-CLIP, a video-text retrieval method. For a batch of successful episodes, it computes four similarity matrices between local and global representations of the observation sequence and the instruction: episode-instruction, episode-word, observation-instruction, and observation-word. These are aggregated with softmax attention, and the final score feeds a symmetric InfoNCE-style loss that pulls each episode's representation close to its own instruction and repels it from other instructions in the batch. The second mechanism, instruction tracking, builds on the same similarity scores: for each subtask parsed from the instruction, it keeps a moving average of the episode-word similarity, and when the current score spikes above $k$ times the running average (Eq. 14), the subtask's tokens are masked from the instruction with a probability that ramps up with training progress. The auxiliary loss is what makes the similarity scores trustworthy enough to drive masking, so the two mechanisms are designed to reinforce each other.","core_discovery":"The paper proposes CAREL, a framework that adds a cross-modal contrastive auxiliary loss to the RL objective of an instruction-following agent. The loss is the X-CLIP objective, which computes similarity between an episode (a sequence of observations with action embeddings) and an instruction at four granularities: episode-instruction, episode-word, observation-instruction, and observation-word, and then combines them into a single per-pair score. During training, only trajectories that achieve a high enough reward are treated as positive pairs; the loss pulls matched episode-instruction pairs together in embedding space and pushes unmatched pairs apart. On top of the loss, CAREL's instruction tracking uses the per-word similarity scores produced during the forward pass to detect when a subtask has been completed, and masks the corresponding tokens, along with their connecting conjunctions, out of the instruction given to the policy for the remainder of the episode. The reported experiments, on navigation and manipulation tasks with procedurally generated instructions and unseen color and type combinations, show higher success rates at the same frame budgets than the BabyAI baseline and, when applied to the SHELM baseline, comparable or better mean rewards with faster convergence. The authors frame the contribution as a way to inject grounding into the representations of a dual-encoder RL agent without altering its architecture.","pith_inferences":["Because the auxiliary loss labels episodes by reward threshold and not by instruction semantics, the same recipe might apply to any goal-conditioned RL agent that receives a textual description and has a dual-encoder architecture, not just the BabyAI and SHELM settings tested here; the main barrier would be calibrating the success threshold per task.","The instruction-tracking masking rule has a natural connection to curriculum learning: as the masking probability grows with training progress, the agent sees progressively shorter instructions, which may implicitly encourage it to schedule its own subgoals; a testable extension would be to measure whether the learned policy becomes better at decomposing long instructions in zero-shot generalizati","The X-CLIP loss is symmetric in the sense that it aligns both directions, episode-to-instruction and instruction-to-episode; one could exploit the same scores to provide dense reward shaping during the episode, which the paper does not do.","If the success threshold in Section 2.1 is miscalibrated, the contrastive positives are mislabeled; a robustness study sweeping that threshold would clarify whether the method's gains are sensitive to this otherwise unspecified hyperparameter."],"forward_implications":["On the BabyAI benchmark, equipping the baseline with CAREL lifts the success rate at fixed frame budgets from 41% to 73% on GoToSeqS5R2 (10M frames) and from 24% to 79% on OpenDoorsOrderN4 (3M frames); similar gains appear across all five tested tasks.","Adding instruction tracking on top gives a further boost on the sequential tasks, with the full method reaching 93% versus 76% for the baseline on GoToSeqS5R2 and 97% versus 62% on OpenDoorsOrderN4 at convergence.","The framework transfers to a different base model: applying CAREL to SHELM improves sample efficiency on MiniGrid-GoToDoor (mean reward 0.83 versus 0.50 at 1.5M frames) and on PutNear.","CAREL matches or beats the imitation-learning baseline LISA in success rate on the two tasks LISA was designed for.","The implementation adds only a modest compute overhead (about 420 versus 669 frames per second on average in BabyAI), so the sample-efficiency gains are not bought by drastically slower updates."],"supporting_citations":[{"why":"Supplies the X-CLIP contrastive loss that CAREL adapts as its auxiliary objective.","marker":"Ma et al. (2022)"},{"why":"Defines the BabyAI benchmark and the base dual-encoder agent that CAREL modifies.","marker":"Chevalier-Boisvert et al. (2018)"},{"why":"Provides the CLIP contrastive learning formulation that grounds the loss family and the similarity scoring.","marker":"Radford et al. (2021)"},{"why":"Surveys video-text retrieval and motivates the multi-grained alignment property CAREL transfers to RL.","marker":"Zhu et al. (2023)"},{"why":"Provides the SHELM baseline used to show CAREL transfers to RGB observations and a different architecture.","marker":"Paischer et al. (2023)"},{"why":"Provides the LISA imitation-learning baseline against which CAREL's sample efficiency is compared.","marker":"Garg et al. (2022)"}],"fun_headline_variants":["Cross-modal auxiliary loss boosts instruction-following RL","CAREL: text-vision loss plus instruction tracking for RL","Contrastive loss and token masking improve RL generalization","Instruction tracking with auxiliary loss accelerates RL learning","CAREL: grounding instructions via video-text retrieval-style loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The auxiliary loss labels a trajectory as a positive example only if it crosses a success threshold defined as a fraction of the maximum achievable reward, a number the paper never specifies; the whole contrastive signal assumes this threshold cleanly separates episodes that genuinely match the instruction from those that do not, and the paper offers no evidence the same threshold works across tasks without tuning.","fun_headline_variants_meta":{"raw":{"variants":["Cross-modal auxiliary loss boosts instruction-following RL","CAREL: text-vision loss plus instruction tracking for RL","Contrastive loss and token masking improve RL generalization","Instruction tracking with auxiliary loss accelerates RL learning","CAREL: grounding instructions via video-text retrieval-style loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":3153,"prompt_tokens":930,"completion_tokens":2223,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2146}},"tokens_in":546,"tokens_out":2223,"duration_ms":13948,"temperature":1.0,"reasoning_tokens":2146,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:48:31.363254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train CAREL on GoToSeqS5R2 with the success threshold deliberately set to 0% of maximum reward (all trajectories become positives) and separately to 100%; if the success-rate curves remain close to Table 1, the threshold is not what drives the gains, while if they collapse, the unspecified threshold is a critical hyperparameter. A second falsifier targets instruction tracking: replace the moving-average spike rule of Eq. 14 with random masking of one subtask per episode under the same probability schedule; if random masking matches the CAREL+IT curve, the similarity-based tracking is not the source of the improvement.","supporting_citations":[{"cited_title":"Deep learning for video-text retrieval: a review","cited_arxiv_id":null,"evidence_quote":"Surveys video-text retrieval and motivates the multi-grained alignment property CAREL transfers to RL."},{"cited_title":"Semantic HELM: A Human-Readable Memory for Reinforcement Learning","cited_arxiv_id":"2306.09312","evidence_quote":"Provides the SHELM baseline used to show CAREL transfers to RGB observations and a different architecture."},{"cited_title":"LISA: Learning Interpretable Skill Abstractions from Language","cited_arxiv_id":"2203.00054","evidence_quote":"Provides the LISA imitation-learning baseline against which CAREL's sample efficiency is compared."}],"review_version":1}