{"id":"68f015ab-1c5f-4242-9f17-80bd4f5ec92a","arxiv_id":"2501.15495","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"EF-OnTL lets teams of imperfect RL agents share selected experience online, guided by uncertainty and surprise, and is reported to match or beat expert-based advice baselines in four testbeds.","lead":"This engineering thesis proposes EF-OnTL, a way for reinforcement-learning agents that are all still learning to swap their own experience tuples with each other, with no fixed expert teacher. It also introduces sars-RND, an uncertainty estimator that reacts to the action taken, and tests the combination in four simulated environments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on cross-agent comparability of sars-RND uncertainty scores, but delta-confidence subtracts uncalibrated per-agent estimators; no evidence establishes that a higher delta-conf corresponds to a transferable knowledge gap.","rationale":"The reader's strongest concern is exactly the load-bearing one: cross-agent comparability of sars-RND scores. I agree that this assumption is neither validated nor derivable from the architecture. The thesis' own limitation statements cover communication fault-tolerance and agent homogeneity, but not calibration of the uncertainty estimators. The reviewer's proposed mechanism—subtractive delta-confidence in §4.5.2 and average-uncertainty source selection in Eq. 4.1—is only meaningful if two different agents' raw RND errors measure the same thing on the same scale. Since each estimator is independently initialized and trained on different local data, raw error magnitudes are not directly comparable. The internal sars-RND vs RND experiment (§3.4.1) demonstrates sensitivity to action changes within a single model, but cross-agent comparability is a strictly stronger property. The reader's conditional verdict is therefore appropriate, and my analysis reinforces it. The concrete test I propose is directly actionable with the public code: compare per-agent uncertainties on a shared tuple set and test whether rank-based or normalized delta-conf changes the reported results. Given the centrality of this assumption to the distinctive contribution, the verdict should remain CONDITIONAL pending that check.","tokens_in":47165,"tokens_out":3047,"duration_ms":32261,"concrete_test":"Using the public EF-OnTL code, run the Cart-Pole or MT-PP configuration with two or more agents at a fixed transfer step. Extract each agent's sars-RND score for a common set of tuples drawn from all agents' transfer buffers. Check (a) the inter-agent ranges and variances of scores on the same tuples, and (b) rank correlation between delta-conf rankings and an independent proxy for knowledge gap, such as per-state visit counts or target TD-error. Then re-run the headline experiments after z-score normalizing each agent's sars-RND scores before computing delta-conf and U. If the ranking is dominated by scale differences, or if normalization removes the reported advantage, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that EF-OnTL improves convergence by transferring uncertainty-filtered experience—requires that sars-RND uncertainties from different agents be on a common scale, because both source selection (Eq. 4.1, average uncertainty U) and content selection (delta-conf = u_target(o_s) - u_source(o_s), §4.5.2) subtract or compare raw scalar outputs. Each agent runs its own sars-RND predictor/target pair, initialized randomly and trained only on its own experience; RND-type errors are not calibrated and their scale depends on network initialization, optimizer trajectory, and data distribution. If agent A's estimator is systematically lower than B's for the same tuples, B will always appear 'more knowledgeable' and A's buffer will be selected or filtered incorrectly. The thesis says agents use 'a common uncertainty estimator methodology' (§1.3) and a 'standardised architecture' (§4.5.2), but that only guarantees the same architecture, not the same scale. §7.2 flags communication and homogeneity limitations but does not flag this calibration assumption. The internal comparison of sars-RND vs RND (§3.4.1) only shows sensitivity to action changes within one estimator; it never checks whether scores are comparable across agents. Thus the key mechanism could select experiences based on estimator scale rather than actual epistemic gap. This concern is prior to the empirical issues of missing confidence intervals and random-transfer controls: if delta-conf does not measure a knowledge gap, the framework's distinctive contribution loses its mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The thesis proposes Expert-Free Online Transfer Learning (EF-OnTL), a framework for online experience sharing among homogeneous RL agents without a fixed expert. Each agent maintains a transfer buffer of (s,a,r,s') tuples labelled by a new uncertainty estimator, sars-RND. At regular transfer steps, a source agent is chosen by average uncertainty or best performance, and each target samples B tuples selected by delta-confidence and/or TD-error (TCS). The thesis also introduces sars-RND, an extension of RND that takes action, reward, and next state into account. The evaluation compares EF-OnTL against no-transfer, OCMAS, RCMP, QMIX, and multi-agent PA-DDPG in Cart-Pole, MT-PP, HFO, and 3R2S, reporting better or equal convergence and final performance in these benchmarks.","tokens_in":47488,"tokens_out":9498,"duration_ms":85159,"significance":"EF-OnTL addresses a real gap: agent-to-agent transfer learning has mostly been teacher-student action advice, often with an expert, while experience sharing has mostly been offline pretraining. Showing that imperfect agents can mutually accelerate learning through filtered experience, with a public code repository (Section 5.3) and ablations of SS, TCS, and B (Section 6.3.2), would be a useful empirical contribution, especially for the real-world 3R2S demand scenario. No machine-checked proofs or parameter-free derivations are involved; the contribution is algorithmic and empirical, and the code is openly available. The claim is not derived from fitted equations, so the external circularity burden is low; the main concerns are internal validity: the uncertainty-filtering mechanism is not isolated from data-quantity effects, and the cross-agent comparability of sars-RND outputs is assumed rather than demonstrated.","major_comments":[{"comment":"The central mechanism presupposes that raw sars-RND outputs from different agents are directly comparable, but no evidence establishes this. In Eq. (4.1) the source is selected by minimizing average uncertainty u_i, and in Section 4.5.2 the content filter is delta-confidence = u_target(oi_s) - u_source(oi_s). Each agent runs its own sars-RND predictor/target pair, initialized randomly and trained only on that agent's experience; RND-type prediction errors are uncalibrated and their scale depends on network initialization, optimizer trajectory, and data distribution. A 'common uncertainty estimator methodology' (Section 1.3) and a 'standardised architecture' (Section 4.5.2) do not by themselves put the estimators on a common scale. If one agent's estimator is systematically lower, that agent will always be selected as source and tuples will be filtered by estimator scale rather than by a transferable knowledge gap. Section 7.2 flags communication and homogeneity limitations but does not flag this calibration assumption. The internal comparison of sars-RND and RND in Section 3.4.1 only shows sensitivity to action changes within one estimator; it never checks whether scores are comparable across agents. I request either a cross-agent calibration study (for example, evaluating all estimators on a common held-out tuple set and reporting the scale distributions) or an explicit normalization step, together with a validation that delta-confidence ordering correlates with measured transfer utility.","section":"Section 4.5.2 and Eq. (4.1)"},{"comment":"No random-experience transfer control is included. EF-OnTL is compared with no-transfer, OCMAS, RCMP, QMIX, and multi-agent PA-DDPG, but none of these isolates the effect of TCS-based uncertainty filtering from the simple effect of adding B extra tuples to the target's learning process. A control that transfers B uniformly random tuples from the source buffer would show whether delta-confidence and sars-RND provide benefit beyond data quantity or implicit regularization. Because the central claim is specifically that uncertainty-filtered experience accelerates convergence, this control is load-bearing and should be added.","section":"Section 6.3.1, Figures 6.5-6.10"},{"comment":"The headline comparisons do not report confidence intervals or significance tests. Figure 6.7 reports evaluation metrics on 500 test episodes but no variance or seed-level test, and the number of independent runs used in Chapter 6 is not stated in the reviewed text. With per-environment tuned configurations in Table 6.3, point-estimate learning curves are insufficient to support the abstract's 'better or equal performance' claim across four benchmarks. The 50-run average curves in Section 3.3 show that the authors can compute such statistics; they should do so for the Chapter 6 comparisons.","section":"Section 6.1.2 and Figures 6.5-6.10"},{"comment":"EF-OnTL's hyperparameters are tuned per environment, and the sensitivity analysis covers only Cart-Pole and MT-PP. Transfer budget B varies from 5,000 to 100 across environments, and TF, SS, and TCS also change; for HFO and 3R2S only one configuration is evaluated. The ablation in Section 6.3.2 therefore does not establish that the reported improvements are robust to B and TF in the more complex environments, which weakens the generality of the 'better or equal' claim as stated in the abstract.","section":"Tables 6.3-6.4 and Section 6.3.2"}],"minor_comments":[{"comment":"The normalization used in Figure 3.12 is not defined; the text should state exactly how RND and sars-RND uncertainties are aligned before comparison, because the two estimators have different input dimensions and are not obviously on the same scale.","section":"Section 3.4.1"},{"comment":"There are numerous spacing and typographical errors, such as 'inEF-OnTL' and 'frameworkEF-OnTL' in the abstract, and inconsistent rendering of 'sars-RND' across the text; a careful copy-edit is needed.","section":"Abstract and throughout"},{"comment":"The column header 'sigma pass' is not defined; it appears to denote the standard deviation of passengers served per vehicle, but this should be stated explicitly in the caption or text.","section":"Table 3.2"},{"comment":"The sentence 'which is more than doubled on the server' is ambiguous: if the MT-PP training and testing phase takes about 7 hours on the laptop, the server should take less time, so either the direction or the wording needs correction.","section":"Section 5.1.1"},{"comment":"The RCMP baseline is described in Section 2.4 as a modification that replaces the single optimal teacher with multiple trained agents and majority voting, but Section 6.1.1 does not restate this modification; the evaluation section should define the exact RCMP setup used so the comparison is reproducible.","section":"Section 2.4 and Section 6.1.1"}],"recommendation":"major_revision","confidential_remarks":"I see no grounds for rejection. The framework is plausible, the empirical scope is broad, and the code is publicly available, but the cross-agent calibration assumption and the absence of a random-experience control are genuine threats to internal validity. Both are addressable with additional experiments or explicit normalization, so I recommend major revision rather than rejection. If the venue is a journal, the thesis-style framing and length will also need to be reworked for a paper format."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth taking seriously. EF-OnTL replaces a fixed expert with dynamically selected learning agents that share experience buffers, and it uses uncertainty estimates both to pick the source and to filter what gets transferred. The combination is new relative to OCMAS, RCMP, and Gerstgrasser et al., and sars-RND is a plausible extension of RND to full transition tuples. The thesis also ships code, evaluates across four environments including a real-data ride-sharing simulator, and reports comparisons against established baselines. The offline feasibility study is a sensible precursor, and the EF-OnTL design is clearly specified.\n\nThe main soft spot is the unverified cross-agent comparability of sars-RND scores. Source selection compares average uncertainties across agents, and the delta-confidence criterion subtracts two independently trained estimators. The thesis asserts a standardised architecture and a common methodology, but it never shows that the scales actually match. A high delta-conf could reflect estimator scale rather than a knowledge gap. That is not fatal on its own, because the empirical gains might survive even with coarse ranking, but it leaves the mechanism underdetermined. The paper also reports no calibration analysis, no random-experience transfer control, and no significance tests or confidence intervals on the headline learning curves. The 3 percent served-requests improvement in 3R2S is incremental, and without error bars it is hard to know if it is real. The thesis itself flags communication and homogeneity assumptions in Section 7.2, but it does not flag the calibration assumption behind delta-confidence, and that omission should be fixed.\n\nThis is a solid combination of known ingredients plus a new estimator, evaluated broadly. It deserves a serious referee, but the referee should ask for a calibration analysis, a random-transfer control, and error bars on the main comparisons. The central argument is plausible, not proven. I would not cite it as an established result in the next year, but I would bring it to a reading group to discuss the mechanism. Send it to review with the expectation of major revision.","headline":"A promising but not yet established framework for expert-free online experience sharing; the core mechanism rests on an unverified cross-agent comparability of uncertainty scores.","tokens_in":48020,"tokens_out":2475,"would_cite":false,"duration_ms":24418,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Online transfer learning works between imperfect agents, no expert needed","keywords":["multi-agent reinforcement learning","online transfer learning","experience sharing","epistemic uncertainty","random network distillation","source selection","deep reinforcement learning"],"falsifier":"Run the same task with two agents whose sars-RND networks are initialised to produce deliberately different uncertainty scales, for example by initialising one predictor near zero and the other with large weights; if EF-OnTL's source selection or tuple ranking shifts substantially and performance degrades, the cross-agent comparability assumption fails. A second check is to compute delta-confidence after normalising each agent's uncertainties to a common distribution and measure how much the selected transferred batch changes; if the selected tuples differ, the raw scores are not doing the claimed work.","tokens_in":46927,"feed_emoji":"🤖","tokens_out":4720,"duration_ms":44315,"temperature":0.7,"pith_summary":"This thesis claims that online transfer learning in multi-agent reinforcement learning does not require a fixed expert. It proposes EF-OnTL, a framework in which each learning agent periodically shares a buffer of past interactions, each labelled with epistemic uncertainty, and every receiving agent filters those interactions to fill its own knowledge gaps. Source agents are selected on the fly by average uncertainty or recent performance, and transferred tuples are chosen by the gap between source and target uncertainty or by expected surprise. The author argues that EF-OnTL improves on or matches action-advice baselines across four environments of increasing complexity, with larger gains in harder tasks. If correct, the framework gives a practical way for imperfect agents to accelerate each other's convergence without a designated teacher.","feed_headline":"No-expert agents speed up learning by exchanging curated experience","feed_subtitle":"New framework picks the best source on the fly and filters tuples by uncertainty, beating advice baselines in four environments.","key_machinery":"The load-bearing component is the Transfer Core Engine, a per-agent module that exchanges transfer buffers, chooses a temporary source by Source Selection Criteria (lowest average uncertainty or best recent performance), and filters the incoming buffer by Transfer Content Selection Criteria based on delta-confidence and expected surprise approximated by TD-error. The second load-bearing component is sars-RND, an estimator that distils a random target network and reports prediction error conditioned on state, action, reward, and next state, which gives finer-grained epistemic uncertainty than state-only RND during online learning.","core_discovery":"EF-OnTL is an online experience-sharing framework where no agent is assumed to be an expert. At transfer steps every agent publishes its uncertainty-labelled interaction buffer; a common source is selected using either lowest average epistemic uncertainty or best recent cumulative reward. Each target then scores the source's tuples by delta-confidence, defined as target uncertainty minus source uncertainty on the same tuple, or by a combined normalised loss-and-confidence measure, and optimises its policy on the top B tuples. The thesis also introduces sars-RND, a Random Network Distillation variant that estimates epistemic uncertainty from the full state-action-reward-next-state tuple rather than from the state alone, so that rarely tried actions in familiar states still register as uncertain. Experiments across Cart-Pole, Multi-Team Predator-Prey, Half Field Offense, and a ride-sharing simulator show better or equal performance against no-transfer, OCMAS action-advice sharing, and RCMP expert-advice baselines, with improvements growing with environment complexity.","pith_inferences":["Beyond the paper: a direct test of the cross-agent comparability assumption would be to calibrate each agent's sars-RND uncertainty to a common scale before computing delta-confidence; if the transferred-batch ranking changes materially, the raw gap is not measuring the intended knowledge gap.","Beyond the paper: EF-OnTL could be extended to heterogeneous agents with different reward models by inserting a state and reward mapping function, a direction the thesis leaves open.","Beyond the paper: in large fleets, global source selection could be localised to communication neighbourhoods, and whether the loss in source quality is acceptable is an empirical question the thesis does not answer.","Beyond the paper: because the reported benefit scales with task difficulty, one would expect EF-OnTL to show its largest advantage in sparse-reward, high-dimensional MARL problems, which the four benchmarks only partially cover."],"forward_implications":["Agents can transfer knowledge online without any fixed expert, so target performance is not capped by a single teacher's policy quality.","Experience sharing can match or beat action-advice methods, meaning targets do not need their policies overridden to benefit from others' knowledge.","The reported performance gap between EF-OnTL and baselines grows with environment complexity, so the framework matters most where exploration is hardest.","sars-RND provides a usable fine-grained uncertainty signal during online training, unlike state-visit counters or state-only RND.","Each target receives a personalised batch tailored to its own knowledge gaps rather than a uniform broadcast from the source."],"supporting_citations":[{"why":"Random Network Distillation, the base uncertainty estimator that sars-RND extends with action, reward, and next-state inputs.","marker":"[70]"},{"why":"OCMAS, the online confidence-moderated action-advice baseline that EF-OnTL is compared against in the no-expert setting.","marker":"[52]"},{"why":"RCMP, the expert-advice baseline that anchors the comparison against fixed-teacher transfer.","marker":"[42]"},{"why":"Supplies the TD-error expected-surprise criterion used by EF-OnTL to filter transferred experiences.","marker":"[53]"},{"why":"Uncertainty-based source and content selection in an agent-to-agent transfer setting, a direct precedent for EF-OnTL's delta-confidence filtering.","marker":"[114]"},{"why":"Confidence-based advice selection from a trained classifier, used as prior work on filtering by confidence.","marker":"[115]"},{"why":"Performance-based and state-visit source selection criteria that inspire EF-OnTL's Best Performance source selection.","marker":"[49]"},{"why":"Prioritised experience replay, which motivates using expected surprise as a selection signal for impactful tuples.","marker":"[104]"}],"fun_headline_variants":["No-expert agents speed up multi-agent RL by sharing filtered experiences","Uncertainty-guided transfer boosts multi-agent RL without expert advice","Multi-agent RL learns faster by sharing uncertainty-filtered experiences","Expert-free transfer: agents share experience to accelerate multi-agent RL","Shared uncertainty data accelerates multi-agent RL training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the uncertainty numbers produced by different agents' sars-RND estimators can be compared directly, so that a large gap between a target's and a source's uncertainty on the same tuple reliably marks the knowledge the target is missing.","fun_headline_variants_meta":{"raw":{"variants":["No-expert agents speed up multi-agent RL by sharing filtered experiences","Uncertainty-guided transfer boosts multi-agent RL without expert advice","Multi-agent RL learns faster by sharing uncertainty-filtered experiences","Expert-free transfer: agents share experience to accelerate multi-agent RL","Shared uncertainty data accelerates multi-agent RL training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000401,"raw_usage":{"total_tokens":2117,"prompt_tokens":993,"completion_tokens":1124,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":1042}},"tokens_in":609,"tokens_out":1124,"duration_ms":8099,"temperature":1.0,"reasoning_tokens":1042,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:14:06.357919+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same task with two agents whose sars-RND networks are initialised to produce deliberately different uncertainty scales, for example by initialising one predictor near zero and the other with large weights; if EF-OnTL's source selection or tuple ranking shifts substantially and performance degrades, the cross-agent comparability assumption fails. A second check is to compute delta-confidence after normalising each agent's uncertainties to a common distribution and measure how much the selected transferred batch changes; if the selected tuples differ, the raw scores are not doing the claimed work.","supporting_citations":[{"cited_title":"Interactive Reinforcement Learning with Dynamic Reuse of Prior Knowledge from Human/Agent's Demonstration","cited_arxiv_id":"1805.04493","evidence_quote":"Uncertainty-based source and content selection in an agent-to-agent transfer setting, a direct precedent for EF-OnTL's delta-confidence filtering."},{"cited_title":"Improving reinforcement learning with confidence-based demonstrations","cited_arxiv_id":null,"evidence_quote":"Confidence-based advice selection from a trained classifier, used as prior work on filtering by confidence."}],"review_version":1}