{"id":"c6908a77-6954-4852-bc77-dbc3c4853444","arxiv_id":"2506.07006","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CARoL measures task similarity by state-transition prediction errors and uses those similarities to weight prior policies, value functions, or actor-critic knowledge when adapting to a new task.","lead":"A robot learning method called CARoL decides which previously learned skills to reuse for a new task by checking how well each old model of the robot's dynamics predicts the new task's state transitions. It then blends the most relevant old policies or value functions into a new policy, showing faster convergence in two simulations and on a real off-road vehicle.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's loss (Eq. 5) contains no target reward, so CARoL's reported 'adaptation' may be supervised imitation of source policies; the paper does not show it exceeds the best source on unseen targets.","rationale":"The reader's weakest_assumption concerns fidelity of the low-dimensional transition function as a proxy for knowledge relevance. That is a valid concern, but I identify a more fundamental issue: even if the transition-based weights perfectly rank source knowledge, the policy-based and actor-critic CARoL algorithms never use the target reward, so the claimed adaptation cannot be a process of learning the new task. The value-based variant does use reward through Eq. (8), but the two headline experiments (CarRacing and the real-world vehicle) rely on Algorithm 1/Algorithm 3 objectives that omit it. This is not a matter of tuning or approximation; it is a structural property of the loss. The concrete test above would directly show whether CARoL's final policy improves over the best source on truly unseen targets. If it does not, the paper's central claim must be softened to 'context-weighted source-policy selection' rather than 'context-aware adaptation for robot learning.' I keep the verdict CONDITIONAL because the deficiency is addressable by adding a reward term to the objective, as CARoL+ does, and by reporting the proposed comparison. The reader's proxy concern should also be investigated, but it is secondary: a perfect similarity measure cannot rescue an algorithm that never optimizes the target reward.","tokens_in":16321,"tokens_out":5285,"duration_ms":63885,"concrete_test":"For each out-of-source CarRacing target (μ=0.2, 0.7, 1.3, 1.7, 2.3), report the zero-shot average reward of each source policy SK1–SK3 on that target, alongside the final average reward of CARoL after the 2e6-step adaptation. If CARoL's final reward is not significantly higher than the best single SK's zero-shot reward on the same target, then Algorithm 1 provides no task-specific adaptation beyond source-policy selection. Run the same comparison for LunarLander targets i and j (outside the source square); if CARoL cannot beat the best source there, the actor-critic variant also lacks genuine adaptation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central demonstration of CARoL in CarRacing (Section V.A) uses Algorithm 1, whose only loss is the weighted KL divergence of Eq. (5). No reward from the target task appears in the objective, and the target policy is updated purely by regression toward a mixture of source policies. Similarly, Algorithm 3 in LunarLander combines Eq. (5) with Eq. (9), where Eq. (9) evaluates source critics but again contains no target reward; the only reward-aware variant is CARoL+ of Section V.B.2, which is not the method presented as CARoL. Consequently, the claimed 'faster convergence and higher rewards' are not evidence of learning the target task: the policy can at best interpolate among source behaviors. For target configurations far from all sources (e.g., CarRacing μ=0.2 or 2.3, or LunarLander targets i and j), a reward-free blend of source policies has no mechanism to discover new behavior that outperforms the best source. The paper's own Figure 4 suggests CARoL can beat source dash lines in some panels, but if that is real, the mechanism is unexplained and may be an artifact of evaluation noise, since the training loss does not reward improvement. This undermines the strong claim that context-aware weighting 'enables efficient learning of a new task' rather than merely selecting among already-trained policies.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CARoL proposes to accelerate RL adaptation to a new task by (i) learning a transition function for each source task, (ii) computing softmax similarity weights from the prediction error of these transition functions on target trajectories, and (iii) using those weights to adapt source policies, value functions, or actor-critic pairs to the target. The paper presents three algorithmic variants (policy-based, value-based, actor-critic), validates them in CarRacing and LunarLander, and reports a physical ground-vehicle experiment in which a policy adapted from two simulated off-road skills achieves 5/5 successful traversals of a rocky testbed. The central claim is that state-transition prediction error is a useful contextual marker for ranking source knowledge, and that the weighted adaptation yields faster convergence and higher reward than indiscriminate distillation, meta-learning, or learning from scratch.","tokens_in":16551,"tokens_out":6261,"duration_ms":71452,"significance":"The core idea is simple and interpretable: use the prediction error of learned source transition models on target trajectories as a task-similarity measure. This is a sensible alternative to latent environment encodings, and the weights are computed from independent target samples rather than fitted to target performance, so the core mechanism is not circular. The paper also provides a useful comparison against uniform policy distillation, and the real-world off-road deployment is a valuable practicality check. However, the central claim that CARoL enables efficient RL learning of a new task is not currently supported: the policy-based and actor-critic variants never use the target reward, and the main simulation results are single-curve comparisons without statistical grounding. The contribution would be significant if reframed as context-aware knowledge fusion or if the reward-based CARoL+ variant were made the primary method, but as written the evidence does not match the strength of the claims.","major_comments":[{"comment":"Among the three proposed algorithms, only the value-based variant (Eq. (8)) includes the target reward. Algorithm 1 minimizes only the weighted KL divergence in Eq. (5), and Algorithm 3 adds Eq. (9), which evaluates the target actor with source critics; neither term depends on the target reward. Remark 1 explicitly states that standard RL losses are omitted and defers reward-based learning to the CARoL+ variant. Since the CarRacing experiments (Section V.A.2) and the real-world experiment (Section V.C.2) use Algorithm 1, the reported \"faster convergence and higher rewards\" measure reward-free policy distillation, not RL-based learning of a new task. With no target reward, there is no mechanism to discover behavior that outperforms the best achievable weighted combination of source behaviors. The abstract's claim that CARoL \"demonstrates faster convergence and higher rewards when learning policies for new tasks\" therefore overstates what is shown. Please either reframe the contribution as context-aware policy fusion/adaptation without reward-based learning, or make CARoL+ (with target reward) the main method and report its performance on all targets.","section":"Section IV.B.1, Eq. (5); Section IV.B.3, Eq. (9); Algorithms 1 and 3"},{"comment":"In CarRacing, the transition function used for context is trained on speed and gyroscope readings only, while the policy consumes stacked camera images. The similarity measure Y_i in Eq. (3) therefore ranks source tasks based on a low-dimensional projection of the dynamics that may omit control-relevant features such as track curvature or visual friction cues. The central premise of Section IV is that transition probabilities determine the knowledge needed to solve an MDP; this premise requires f_i to approximate the dynamics that actually dictate policy choice. Please provide evidence that Y_i correctly ranks source tasks by their usefulness on the target, for example by plotting Y_i against the measured return of each source policy on each target task, or by training the context model on the full observation and comparing the resulting weights. Without such validation, the similarity-ranking claim is not fully supported.","section":"Section V.A.2 and Eq. (3)"},{"comment":"The main experimental claims rest on single learning curves with no seeds, confidence intervals, or significance tests. Statements such as \"CARoL demonstrates a faster convergence speed compared to other methods\" and \"CARoL generally achieves the best performance\" are not substantiated by any measure of variability. Please report mean and variance over at least five random seeds for the simulation experiments, and specify how many evaluation episodes are used per curve point. This is particularly important because the reward-free objective gives no a priori reason for CARoL to beat every source policy, so the observed advantages need statistical support.","section":"Section V.A.3, Section V.B.3, Figures 4 and 6"},{"comment":"The paper acknowledges that on target task (i) and on some trials of (j), Learning from Scratch (LfS) outperforms CARoL, and that only CARoL+ recovers the gap. This is exactly the failure mode one would expect from an adaptation objective that contains no target reward. The abstract and conclusion should scope the claims accordingly: CARoL can effectively blend source knowledge for target configurations inside or near the source envelope, but it cannot discover new behavior for targets far outside it. At present, this limitation appears only as a caveat in the experimental analysis, which conflicts with the paper's general claim that CARoL \"enables efficient learning of a new task.\"","section":"Section V.B.3 and Figure 5"}],"minor_comments":[{"comment":"Equation (6) contains unbalanced parentheses and the KL divergence is not written with explicit sums over actions. Please rewrite it as D(pi_i(s), pi_g(s|theta_g)) = sum_a Phi(pi_i(s)/T)_a * ln( Phi(pi_i(s)/T)_a / Phi(pi_g(s|theta_g))_a ).","section":"Section IV.B, Eq. (6)"},{"comment":"The description \"Greedy method to choose action a with explorations or argmax_a(Q_g(s,a|psi_g))\" is ambiguous. Please specify the epsilon-greedy schedule or the exploration mechanism used during data collection.","section":"Algorithm 2, line 4"},{"comment":"The captions state that dashed lines show source knowledge (SK) rewards, but they do not identify which dash style corresponds to SK1, SK2, SK3, and SK4, nor do they map subfigures (a)–(h) or (a)–(j) to the target configurations listed in the text. Adding this information would make the figures much easier to interpret.","section":"Figures 4 and 6 captions"},{"comment":"The footnote \"Code will be publicly available on/github\" is an incomplete sentence. Please either provide a complete repository URL or state that code will be released upon publication.","section":"Section I and Code Availability"},{"comment":"The table headers T1, T2, T3, T4 refer to source tasks whose configurations are described only in the text or in Figure 5. Adding the source-task parameters directly in the table headers would improve readability.","section":"Tables I and II"}],"recommendation":"major_revision","confidential_remarks":"The main issue is framing rather than a fundamental flaw in the similarity mechanism. The transition-prediction-error idea is sound and worth publishing, but the current manuscript claims RL-based learning while the primary algorithms are reward-free distillation. I believe this is fixable by either reframing the contribution or by promoting CARoL+ to the main method and rerunning the key comparisons. I did not elevate the lack of a real-world baseline to a separate major comment, but the authors should at least discuss why no comparison to a reward-based adaptation method was feasible in the physical experiment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is genuinely neat: train one state-transition model per source task, measure its prediction error on target trajectories, turn those errors into softmax weights, and use the weights to prioritize source policies or value functions during adaptation. That specific mechanism, applied consistently across policy, value, and actor-critic variants, is new relative to CAPS, RMA, and the other cited work. Second, the paper's strongest claims outrun its method. Algorithm 1's loss (Eq. 5) contains no target reward; it is pure weighted KL divergence to source policies. Algorithm 3 adds a weighted sum of source critics, again with no target reward. So what is actually demonstrated is context-aware distillation or knowledge selection, not reinforcement learning of a new task. The stress-test note is right: for a target far from all sources, a reward-free blend of source policies has no mechanism to beat the best source. The paper's own Figure 4 suggests CARoL sometimes beats the source dash lines, but that is unexplained and could easily be evaluation noise, since the training loss does not reward improvement.\n\nThat said, the paper does several things well. The similarity weights in Tables I and II correlate sensibly with friction differences and configuration-space distances, and the matching source for the sanity-check cases gets nearly all the weight. The method is simple, interpretable, and algorithm-agnostic. The real-world off-road transfer, with CARoL reaching 5/5 successes versus 3/5 and 2/5 for the source policies, is a useful demonstration even without an adaptation baseline. The authors also include CARoL+, which adds standard TD3 loss, and it does best in LunarLander—that variant is the honest version of the paper's claims.\n\nSoft spots, in proportion: the reward-free framing is the biggest issue, but it is fixable by either adding a reward-aware variant as the headline method or reframing CARoL as 'context-aware knowledge selection and bootstrapping' rather than learning. The absence of seeds, error bars, and code makes the quantitative curves hard to trust; the real-world experiment lacks any adaptation baseline (though cost is a fair excuse); and the MAML baseline is admittedly undertuned. The low-dimensional transition model (speed and gyroscope only) while the CarRacing policy consumes stacked camera images is a real potential mismatch—if dynamics visible only in pixels matter, the similarity proxy could mis-rank sources.\n\nOverall: the core mechanism is internally consistent and not circular, since the weights come from independent target samples and fixed source transition models. The paper deserves a serious referee. I would send it out, conditional on heavy revision: re-frame the contribution, run multiple seeds, add CARoL+ in CarRacing, compare against CAPS or a simple context-label baseline, and release code.","headline":"CARoL is a clean context-aware knowledge-reuse trick, but its headline claim of 'learning a new task' overstates what the reward-free Algorithm 1 actually does; worth refereeing with a demand for re-framing and better baseline coverage.","tokens_in":17098,"tokens_out":1510,"would_cite":true,"duration_ms":19473,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A robot can rank prior tasks by how well their learned state-transition models predict a new task's dynamics, then weight prior policies accordingly.","keywords":["reinforcement learning","robot learning","knowledge transfer","state transition","context awareness","policy adaptation","off-road navigation","multi-task learning"],"falsifier":"Take three tasks with identical low-dimensional state representation but a hidden dynamics parameter that only affects high-dimensional observations. If two source tasks show nearly equal transition-prediction errors on target trajectories yet measurably different transfer performance, the proxy fails; a concrete check would withhold wheel-slip information from $f_i$ while letting the policy observe camera images, then test whether target tasks with different slip regimes receive incorrect weights.","tokens_in":16042,"feed_emoji":"🤖","tokens_out":5802,"duration_ms":62308,"temperature":0.7,"pith_summary":"This paper argues that the right way to decide which previously learned robotic skill matters for a new task is to compare how the tasks move: the state transitions in the system's dynamics. It proposes CARoL, which learns a separate transition-prediction function for each source task, then scores a source task by how well that function predicts the target task's observed trajectories. Those scores become weights that bias the target policy toward the most relevant source policies or value functions. The authors show the weighting scheme fits policy-based, value-based, and actor-critic reinforcement learning, and they report faster adaptation in two simulated control tasks and on a physical ground vehicle navigating rocky terrain.","feed_headline":"State-transition errors rank old robot skills for new tasks","feed_subtitle":"CARoL weights prior policies by how well their learned dynamics predict the new task, shortening adaptation.","key_machinery":"The central object is the state-transition prediction function $f_i(s,a \\mid \\phi_i)$, a learned map from a state-action pair to the next state, trained separately on each source task's data. It serves as the context representation because it approximates the transition probability $P_i$, which the paper identifies as the factor that determines what knowledge an MDP needs. CARoL's similarity measure is the prediction error $Y_i$ of $f_i$ on the target task's trajectories, and the softmax of negative errors produces the weights $w_i$ that reweight source knowledge in the losses $L_P$, $L_Q$, and $L_{AC}$. The method's key move is to measure task relatedness through dynamics rather than through environment or robot encodings.","core_discovery":"The central claim is that in a Markov decision process the transition probability is the contextual marker that ties prior knowledge to a new task: source tasks whose state transitions resemble the target's are the ones whose policies or value functions should be reused. CARoL operationalizes this by training a small state-transition model $f_i(s,a)$ for each source task and computing $Y_i$, the sum of squared prediction errors of that model on trajectories sampled from the target task. Low error means high similarity, and the errors are turned into softmax weights $w_i$. Those weights enter three adaptation losses: a KL-divergence loss pulling the target policy toward weighted source policies, a Bellman target built from weighted source $Q$-functions, and an actor-critic loss that combines both. The paper reports that the resulting weights track the true task configurations, that CARoL converges faster than learning from scratch or fusing all source knowledge equally, and that it transfers policies from simulation to a real off-road vehicle using only a small set of manually collected trajectories.","pith_inferences":["A natural extension the paper does not develop is online reweighting: if target trajectories arrive incrementally during deployment, the same $Y_i$ computation could update the weights continuously, turning CARoL from a one-shot pre-adaptation step into a real-time task detector.","The ranking claim suggests a sharp test not reported in the paper: the ordering of similarity weights should predict the ordering of transfer performance of the source policies on held-out target tasks.","Because the context model in the car experiment uses only speed and gyroscope readings while the policy consumes stacked camera images, the method implicitly bets that control-relevant dynamics are visible in the low-dimensional state; testing on a representation that hides a control-relevant variable would bound the method's scope."],"forward_implications":["If the similarity weights are correct, a robot can rank its prior knowledge without training a transition model on the new task, using only a small set of target trajectories.","The same weighting scheme can be inserted into policy-based, value-based, or actor-critic algorithms, so existing RL pipelines can absorb it without changing their core learning mechanism.","When the target task matches a source task, the method assigns near-unit weight to that source, giving the framework a built-in sanity check during adaptation.","Combining the weighted prior-knowledge loss with a standard RL loss, as the paper's CARoL+ variant does, preserves fast convergence while recovering in cases where no source task is close.","If the weights are computed from a source task that is itself poorly learned, the quality of the source knowledge, not the similarity measure, becomes the limiting factor for the final target policy."],"supporting_citations":[{"why":"Supplies the PPO algorithm used to train the source policies and the target policy in the car and real-vehicle experiments.","marker":"[26]"},{"why":"Supplies the TD3 actor-critic algorithm used to train source actors and critics in the lunar-lander experiments.","marker":"[4]"},{"why":"Provides the policy distillation baseline that fuses all source policies equally, the indiscriminate-fusion comparison point for CARoL.","marker":"[25]"},{"why":"Provides the MAML meta-learning baseline that adapts a shared initialization to the target task.","marker":"[3]"},{"why":"Provides the multi-physics simulator used to train the two source policies that are adapted to the physical ground vehicle.","marker":"[29]"},{"why":"Provides the terrain encoder that produces part of the real-world observation space for the vehicle.","marker":"[9]"}],"fun_headline_variants":["State-transition errors pick which robot skills to reuse","CARoL: Learn new tasks faster by ranking old knowledge","Weigh prior policies by prediction error for quick adaptation","Context-aware robot learning adapts via state dynamics","Adaptive robot learning selects useful skills via transition errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a source task's transition-prediction error on the target's trajectories is a faithful proxy for how useful that source's policy or value function will be on the target; if the learned dynamics miss control-relevant features, the weights can rank knowledge incorrectly even though the rest of the pipeline is sound.","fun_headline_variants_meta":{"raw":{"variants":["State-transition errors pick which robot skills to reuse","CARoL: Learn new tasks faster by ranking old knowledge","Weigh prior policies by prediction error for quick adaptation","Context-aware robot learning adapts via state dynamics","Adaptive robot learning selects useful skills via transition errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1285,"prompt_tokens":949,"completion_tokens":336,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":260}},"tokens_in":565,"tokens_out":336,"duration_ms":4143,"temperature":1.0,"reasoning_tokens":260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:44:12.509468+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take three tasks with identical low-dimensional state representation but a hidden dynamics parameter that only affects high-dimensional observations. If two source tasks show nearly equal transition-prediction errors on target trajectories yet measurably different transfer performance, the proxy fails; a concrete check would withhold wheel-slip information from $f_i$ while letting the policy observe camera images, then test whether target tasks with different slip regimes receive incorrect weights.","supporting_citations":[{"cited_title":"Chrono: An open source multi-physics dynamics engine","cited_arxiv_id":null,"evidence_quote":"Provides the multi-physics simulator used to train the two source policies that are adapted to the physical ground vehicle."}],"review_version":1}