{"id":"6dcca1c1-9b50-4bbe-8f52-0a7466d7de08","arxiv_id":"2505.19637","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"AELA improves MARL training by starting with truncated episodes and lengthening them when action-entropy falls, showing gains over QMIX and VDN on SMAC and predator-prey tasks.","lead":"This paper proposes AELA, a training method for multi-agent reinforcement learning that starts agents with short episodes and gradually lengthens them as their policies stabilize, measured by the entropy of their action choices. The authors report faster convergence and higher final win rates on StarCraft and predator-prey benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The entropy-based adaptive schedule is not isolated from simple truncation plus tuned growth: Appendix C's w calibration makes the EL trajectory essentially predetermined, so the central claim that adaptive entropy assessment drives gains is unsupported.","rationale":"I read the paper in good faith: it proposes a plausible MARL training heuristic, shows a monotonicity argument for why shorter episodes increase visits to secure states, and reports median win-rate curves where AELA often improves over QMIX/VDN. For the central claim to hold, the adaptive entropy-based increase must be a meaningful contributor to the improvements, not a decorative addition to simple truncation. That condition is the least secure part of the paper. The reader's weakest assumption identifies the same gap: AELA is never compared with a fixed short-episode schedule, so the observed gains might come from truncation and hyperparameter tuning rather than from the entropy trigger. I agree with this concern. I would sharpen it: the paper's own design in Appendix C largely predetermines the growth rate by choosing w so that EL reaches Emax around 80% of training, and the entropy slope is likely negative throughout early training, making the adaptive rule nearly equivalent to a time-based schedule. This is not an internal inconsistency, but it is a correctness risk for the novelty claim. The proposed ablation directly settles whether lines 22-25 of Algorithm 1 change the outcomes. Because the method is cheap to test and the authors provide enough implementation detail to reproduce it, the appropriate verdict remains conditional rather than reject: the paper should not be accepted as establishing 'adaptive' gains without this control. No change from the reader's CONDITIONAL verdict is needed.","tokens_in":14654,"tokens_out":4175,"duration_ms":44552,"concrete_test":"On 6h_vs_8z and MMM2, keep the Appendix C hyperparameters (EL0, w, Emax) and run three variants with at least 5 shared seeds: (A) AELA exactly as in Algorithm 1; (B) a fixed schedule that increments EL by 1 every round(Lbar*w) updates regardless of entropy slope; (C) constant EL = EL0 for the entire run. Report median test win rate every 10k steps with confidence intervals, plus the realized EL trajectory. If (B) matches (A) within noise, the entropy trigger is not the cause of AELA's gains; if (C) also matches, the gains come from truncation alone.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that adaptively adjusting episode length based on an entropy slope improves convergence speed and final performance. This requires the entropy-triggered growth rule (Algorithm 1 lines 22-25, Section 4.2) to be doing real work. The theoretical results do not supply that: Theorem 1 and Corollary 1 justify only the benefits of truncation, not the adaptive component, and Theorem 2's regret model does not model the scheduling rule at all. The experiments never isolate the adaptive trigger. All baselines use the fixed maximum episode length, so AELA's gains could come entirely from the initial truncation and from per-scenario tuning of EL0 and w. Appendix C makes this risk concrete: w is chosen so that EL reaches its maximum at roughly 80% of training time (Eq. 38-40), and for 6h_vs_8z both EL0 and w are specially adjusted because dead-ends are frequent. Moreover, early in training, softmax Q-value entropy generally has a negative slope, so the entropy trigger is likely active during most of the early windows; if so, AELA's EL trajectory is nearly a fixed schedule, and the entropy measurement is not the causal mechanism. The paper does not report how often the slope was negative or compare against a fixed short-episode schedule or a predetermined growth schedule. Therefore the strongest empirical claim overreaches the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adaptive Episode Length Adjustment (AELA) for multi-agent reinforcement learning, in which episodes are initialized at a fraction of the maximum episode length and extended by one step whenever a linear fit to the entropy of a window of softmax Q-values has negative slope. The authors motivate the approach with Lemma 1 and Theorem 1, which state that the probability of being in a secure state is non-increasing in the interaction step and that reducing the episode length increases or preserves the expected number of secure-state visits, and with Theorem 2, which claims that lower dead-end probability reduces regret. They evaluate AELA combined with VDN and QMIX on a modified predator-prey task and six SMAC maps, reporting faster convergence and higher final win rates, particularly for QMIX on 6h_vs_8z and Corridor, where standard QMIX is reported as being close to zero.","tokens_in":14932,"tokens_out":9300,"duration_ms":95722,"significance":"The problem addressed is real and under-explored: episode-length management is a potentially cheap and general intervention for MARL, and the reported observation that QMIX fails on 6h_vs_8z and Corridor while AELA-QMIX succeeds is empirically striking. The manuscript contains a reasonable amount of experimental work, including six SMAC maps, a modified predator-prey environment, and a detailed hyperparameter discussion in Appendix C. However, the theoretical results are largely a restatement of the monotonicity of the secure-state probability, and the experimental design does not isolate the entropy-adaptive mechanism from the initial truncation and the tuned growth schedule. Therefore the headline claim that entropy-based assessment drives the improvements is currently unsupported, and the significance of the contribution depends on whether the missing ablations confirm that attribution.","major_comments":[{"comment":"The theoretical support in Section 4.1 does not establish a benefit of the adaptive schedule. Theorem 1 (Appendix A, Eq. (14)-(27)) is a re-arrangement of Lemma 1: with N_total fixed, N_s(EL) = (N_total/EL) * sum_{l=1}^{EL} P_s(l), and because P_s(l) is non-increasing in l, N_s(EL) is non-increasing in EL. Corollary 1 then defines P_d = 1 - N_s/N_total, so the stated decrease in P_d is a restatement of the same monotonicity and carries no information about learning over time. Theorem 2's proof in Appendix B assumes P_d and P_g are constant, omits discounting, mixes realized and expected rewards, and drops the term -sum_{t=1}^T r_t between Eq. (30) and Eq. (31). Consequently, the proof does not rigorously show that reducing dead-end probability reduces regret under the proposed schedule. This matters because Section 4.2 explicitly says that AELA is built on these theorems.","section":"Section 4.1 / Appendix A / Appendix B"},{"comment":"The central empirical claim is not isolated. AELA differs from the fixed-maximum-length baselines in two ways: it starts from a reduced EL0 and it grows EL through the entropy trigger in Algorithm 1, lines 22-25. The experiments in Section 5 never compare AELA against (i) training with a fixed short episode length, (ii) a predetermined growth schedule, or (iii) a random growth schedule, so the reported gains could come entirely from EL0 and from the per-scenario tuning of w described in Appendix C. This risk is concrete: Eq. (38)-(40) choose w so that EL reaches its maximum at roughly 80% of training time, and for 6h_vs_8z both EL0 (15% of E_max) and w (900) are specially adjusted. In addition, the premise in Section 4.2 that a decreasing entropy trend indicates policy convergence is asserted without evidence, and the paper does not report how often the entropy slope was negative or how sensitive the results are to the slope test, so the entropy-based growth rule is not shown to be the cause of the observed improvements.","section":"Section 5 / Appendix C / Algorithm 1"},{"comment":"Statistical significance is not demonstrated. Figures 1 and 2 report median test returns and win rates over five runs, but they show no variance bands, confidence intervals, or significance tests. The abstract and Section 5.2 use phrases such as 'significantly superior performance' and single out 6h_vs_8z and Corridor as cases where QMIX is 'close to zero' while AELA-QMIX is 'significantly higher'; with only five runs and no measure of spread, this wording overstates what the reported evidence establishes. At minimum, the paper should report per-run curves or error bars and state the number of seeds used in each displayed curve.","section":"Section 5.2 / Figures 1-2"}],"minor_comments":[{"comment":"The acronym is introduced inconsistently: the abstract and title use 'Adaptive Episode Length Adjustment (AELA)', while the Introduction defines 'Adaptive Episode Length Approach (AELA)'.","section":"Abstract / Section 1"},{"comment":"The sentence 'The experiment is conducted over 16 episodes' is unclear, since the following sentence refers to five independent runs; this should be rephrased to specify what was averaged over what.","section":"Section 5.1"},{"comment":"The symbol T is used for the number of time steps within an episode in Eq. (10) and also for the total training horizon in Definition 4; this overloaded notation makes the formulas hard to follow.","section":"Section 4.2 / Definition 4"},{"comment":"The y-axis label reads 'T est episode length' in the manuscript text; this is a typo for 'Test episode length'.","section":"Figure 6"},{"comment":"Assumption 1 does not specify the sign of r_g, and for the argument around Eq. (36) to work for k=T the empty sum must be handled and r_g must be assumed positive; this should be stated explicitly.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the empirical phenomenon is potentially interesting, but the paper currently frames the entropy-adaptive rule as the cause of the improvements without an ablation that separates truncation from adaptation. If the authors cannot provide such ablations, the contribution should be reframed as 'initial episode truncation helps in certain SMAC maps', which is a weaker claim than the current one. I would not reject outright because the experiments are extensive enough to be reanalyzed, but acceptance should require the missing control conditions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"For a first pass: this is a practical training heuristic, and the headline result is believable enough to want to see more. The idea is simple — start with short episodes, lengthen them when Q-value entropy slopes downward — and it is genuinely new in the MARL setting, as far as the citations show. Applied to VDN and QMIX, it produces visibly faster convergence on easy SMAC maps and, more strikingly, turns near-zero win rates on 6h_vs_8z and Corridor into meaningful ones. That pattern is worth taking seriously, and the paper writes up the dead-end avoidance intuition clearly. The theoretical Section 4.1 is honest about being a motivation: Theorem 1 is just the monotonicity of P_s rearranged, and the proof is correct. It does what it claims, but it only justifies truncation, not the adaptive rule.\n\nThe soft spots are real and central. The paper never shows that the entropy-triggered growth rule does any work. Every baseline uses the fixed maximum episode length, so the gains could come entirely from starting with short episodes and then growing according to a schedule. The stress-test note makes this concrete: Appendix C chooses w so that EL reaches maximum at roughly 80% of training, and for 6h_vs_8z both EL0 and w are hand-tuned. Early in training softmax entropy almost always decreases, so the trigger is likely active in most early windows anyway. That makes AELA's trajectory close to a predetermined growth schedule, and no comparison is run against a fixed short-episode schedule or a predetermined growth schedule. This is a missing control, not a fatal flaw, but it directly undercuts the claim that entropy assessment is what drives the improvements.\n\nThe other weaknesses are standard but worth naming: median-only curves over five runs, no error bars or statistical tests, and per-scenario hyperparameter selection for the very maps where the gains are largest. Theorem 2's regret proof is too ad hoc to be load-bearing — constant P_d and P_g, no discounting, realized rewards treated as expectations — and the paper would be better off dropping it or rewriting it as a heuristic argument. That said, the authors do not hide their tuning choices; Appendix C is unusually transparent.\n\nWho is this for? Anyone working on training tricks for cooperative MARL. The paper deserves a serious referee because the idea is cheap to test and the initial evidence is promising. My recommendation: send it to review, and ask the authors for the obvious ablation — compare against a fixed short-episode schedule and a predetermined growth schedule, with error bars. If the entropy rule survives that, this is a useful contribution. If not, the paper still has a modest, honest story about truncation in MARL.","headline":"AELA is a simple, plausible MARL training heuristic with real-looking gains on hard SMAC maps, but the entropy trigger is never isolated from plain truncation plus tuned growth, so the central claim overreaches the evidence.","tokens_in":15485,"tokens_out":1505,"would_cite":false,"duration_ms":10845,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AELA proposes to start MARL training with short episodes and lengthen them only as Q-value entropy falls; the paper claims this speeds convergence and lifts final win rates, rescuing QMIX on maps where it otherwise wins almost never.","keywords":["multi-agent reinforcement learning","episode length adjustment","dead-end states","entropy-based convergence assessment","value decomposition","QMIX","VDN","SMAC"],"falsifier":"Compare AELA-QMIX on 6h_vs_8z to a version that keeps the episode length fixed at the initial value for the entire run. If the fixed-short version matches AELA's win rate, the entropy-triggered lengthening is not responsible for the reported recovery from near-zero wins.","tokens_in":14419,"feed_emoji":"🤖","tokens_out":7791,"duration_ms":137449,"temperature":0.7,"pith_summary":"This paper proposes Adaptive Episode Length Adjustment (AELA), a way to train multi-agent reinforcement learning agents by starting each episode short and only lengthening it once the policy shows signs of converging, measured by falling entropy of action-value distributions. The authors argue that short episodes keep agents away from dead-end states early in training, where no policy can recover, while gradually increasing length lets them eventually learn long-horizon coordination. They test AELA on top of QMIX and VDN in SMAC and a modified predator-prey environment. Their central claim is that AELA improves both convergence speed and final performance, and in maps like 6h_vs_8z and Corridor it turns win rates near zero for QMIX into substantially higher win rates. If true, it offers a general, plug-in training schedule that helps MARL without changing rewards, network architecture, or value factorization.","feed_headline":"Adaptive episode lengths lift MARL win rates from near zero","feed_subtitle":"Starting agents on short episodes, then extending them as policy entropy falls, speeds convergence and raises win rates.","key_machinery":"The central object is the entropy of the agents' action distributions, computed from Q-values via softmax and summed over batch, time, and agents. AELA collects this total entropy over a window of size w, fits a line, and increments episode length by one when the slope is negative; the initial length EL0 and window w are the only added hyper-parameters. This entropy-triggered length schedule carries the argument because it is what turns the theoretical claim—shorter episodes do not reduce secure-state visit counts—into an algorithm that gradually extends the horizon as learning stabilizes. The supporting theoretical machinery is Lemma 1's monotonicity of secure-state probability over interaction steps, which yields Theorem 1 (expected secure visits do not decrease when episode length is reduced) and Theorem 2 (lower dead-end probability reduces regret under a goal-dominance assumption).","core_discovery":"Under AELA, the episode length starts at a fraction of the task maximum (typically one quarter, or 15% in a dead-end-heavy scenario) and increases by one step whenever a linear fit to the entropy of Q-values over a window of updates has negative slope. The paper argues, under a Dec-POMDP model, that the probability of being in a secure (non-dead-end) state is non-increasing in the interaction step, so truncating episodes keeps expected secure-state visits constant or higher, and under the assumption that goal reward dominates intermediate rewards, lower dead-end probability lowers regret. The empirical discovery is that applying this schedule to QMIX and VDN improves convergence speed in easy SMAC maps and final win rate in hard maps, with the largest effect in 6h_vs_8z and Corridor, where original QMIX nearly always loses but AELA-QMIX wins at a significantly higher rate. The authors support the dead-end explanation with snapshots showing that QMIX's final policy in 6h_vs_8z wanders into a dead-end after losing units, while AELA's policy ends episodes earlier and avoids that state.","pith_inferences":["The paper does not compare AELA against a fixed short-episode schedule or a random extension schedule, so the adaptive entropy trigger's standalone contribution is an open question, not a demonstrated result.","If the gains come mainly from the initial truncation, a simpler schedule that keeps episodes at the initial length over the full run could match AELA with fewer hyper-parameters; the SMAC maps used here would make that comparison straightforward.","Because policy entropy is already computed in policy-gradient MARL methods, the same trigger could be lifted directly into those settings, which the paper does not test.","In tasks with dense intermediate rewards or where long horizons are essential for cooperation, truncation can cut off credit assignment, so a boundary condition on when AELA helps is a natural follow-up."],"forward_implications":["On easy SMAC maps (3m, 2s_vs_1sc), AELA reaches the same near-100% win rate as QMIX/VDN but faster.","On hard maps (MMM2, 3s5z_vs_3s6z, 6h_vs_8z, Corridor), AELA improves final win rate; for 6h_vs_8z and Corridor, AELA-QMIX wins where QMIX's win rate is near zero.","AELA concentrates collected samples in early interaction steps, which the paper argues reduces visits to dead-end states and lets agents learn strategies like luring.","Because AELA changes no rewards and no network architecture, it can be layered onto value-decomposition MARL methods, demonstrated with QMIX and VDN.","The theory implies the benefit is tied to dead-end-rich environments; in dead-end-free tasks, truncating episodes should not reduce secure-state visits, though the paper does not claim it would improve them."],"supporting_citations":[{"why":"QMIX is the value-decomposition baseline that AELA wraps; its near-zero win rate in 6h_vs_8z and Corridor provides the contrast for the main result.","marker":"[17]"},{"why":"VDN is the other value-decomposition baseline AELA is combined with and compared against.","marker":"[23]"},{"why":"SMAC supplies the evaluation scenarios and win-rate metric used for the main experimental claims.","marker":"[18]"},{"why":"Single-agent time-limit work cited to motivate why truncating episodes can benefit learning despite reduced long-term experience.","marker":"[12]"},{"why":"Adaptive trajectory truncation work cited as evidence that adaptive schedules beat fixed-length approaches, motivating AELA's design.","marker":"[14]"},{"why":"Dead-end and secure-exploration concepts that the paper's Definition 1, Definition 2, and secure-exploration argument build on.","marker":"[2]"},{"why":"Trajectory truncation in Monte Carlo RL cited as prior support for the benefit of cutting episodes short.","marker":"[13]"},{"why":"Cited as the source of the modified predator-prey environment used as the second evaluation suite.","marker":"[20]"}],"fun_headline_variants":["Start short, extend on entropy drop: MARL wins","Entropy-guided episode lengths lift MARL win rates","Adaptive episode lengths: from near-zero to winning in MARL","First MARL method to adapt episode length by learning progress"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that when the agents' choices become more predictable (lower entropy), the right next step is to let episodes run longer; if that trigger does not matter, the improvement might simply come from starting with short episodes and tuning two extra hyper-parameters.","fun_headline_variants_meta":{"raw":{"variants":["Start short, extend on entropy drop: MARL wins","Entropy-guided episode lengths lift MARL win rates","Adaptive episode lengths: from near-zero to winning in MARL","First MARL method to adapt episode length by learning progress"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1555,"prompt_tokens":1027,"completion_tokens":528,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":460}},"tokens_in":643,"tokens_out":528,"duration_ms":4062,"temperature":1.0,"reasoning_tokens":460,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:10:10.783887+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare AELA-QMIX on 6h_vs_8z to a version that keeps the episode length fixed at the initial value for the entire run. If the fixed-short version matches AELA's win rate, the entropy-triggered lengthening is not responsible for the reported recovery from near-zero wins.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"QMIX is the value-decomposition baseline that AELA wraps; its near-zero win rate in 6h_vs_8z and Corridor provides the contrast for the main result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VDN is the other value-decomposition baseline AELA is combined with and compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Single-agent time-limit work cited to motivate why truncating episodes can benefit learning despite reduced long-term experience."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Adaptive trajectory truncation work cited as evidence that adaptive schedules beat fixed-length approaches, motivating AELA's design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Dead-end and secure-exploration concepts that the paper's Definition 1, Definition 2, and secure-exploration argument build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Trajectory truncation in Monte Carlo RL cited as prior support for the benefit of cutting episodes short."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited as the source of the modified predator-prey environment used as the second evaluation suite."}],"review_version":1}