Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss

Emirhan Gazi; Eren \"Ozaltun; Feyza Orak; Kemal Devrim Kafadar; Kubilay Ka\u{g}an K\"om\"urc\"u; Mahmud Efnan \c{S}anl{\i}; Naz{\i}m Kemal \"Ure

REVIEW 3 major objections 5 minor 26 references

This paper claims that weighting an observation-predictor's loss by the RL critic's advantage estimate prevents the performance collapse that standard predictors suffer when inter-agent communication drops, yielding more than 20% higher mea

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 16:37 UTC pith:OZOVSOJA

load-bearing objection The idea is neat and the writeup is transparent, but the key experiment isn't controlled: batch size and loss both change, so the 20% gains can't yet be credited to value-aware weighting. the 3 major comments →

arxiv 2607.17914 v1 pith:OZOVSOJA submitted 2026-07-20 cs.MA cs.LGcs.RO

Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss

Kemal Devrim Kafadar , Eren \"Ozaltun , Mahmud Efnan \c{S}anl{\i} , Feyza Orak , Emirhan Gazi , Kubilay Ka\u{g}an K\"om\"urc\"u , Naz{\i}m Kemal \"Ure This is my paper

classification cs.MA cs.LGcs.RO

keywords multi-agent reinforcement learningcommunication dropoutobservation imputationvalue-aware predictionadvantage weightingproximal policy optimizationpartial observabilitycoordination robustness

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to fix a practical failure mode in multi-agent reinforcement learning: when communication links drop, agents fall back on internal predictors that estimate missing observations, but standard predictors spend their capacity on exploration noise and on behaviors the policy has already discarded. The authors' proposal is to weight each predictor training transition by the actor-critic's advantage estimate, so the predictor is forced to learn the intentional, high-return dynamics the policy is actively reinforcing. They integrate this advantage-weighted objective into the MARO observation-imputation framework and call the result Value-Aware MARO. Across five multi-agent particle tasks, it matches the baseline when communication is perfect and, in the high-loss regimes where the baseline collapses, raises mean returns by over 20% and cuts return variance by about 65%.

Core claim

The central claim is that the predictor's negative log-likelihood objective, reweighted by normalized ReLU(1 + λ Â_t) advantage weights, explicitly couples the predictor to the policy's evolution. The paper derives this from a first-order Taylor expansion of the predictor loss under the policy-parameter update: the change in the predictor loss is proportional to (1 + λ Â_t) times the original NLL, with Â_t the generalized advantage estimate; the heuristic surrogate drops the per-sample policy-gradient norm into a fixed λ = 1.0 and normalizes weights across the minibatch. Empirically, this objective prevents the performance collapse that the unweighted MARO predictor exhibits when the communi

What carries the argument

The load-bearing object is the advantage-weighted negative log-likelihood objective J'_batch(φ) = E_t[w_t · (−log p_φ(Δo_t | o_t))], with w_t = normalized ReLU(1 + λ Â_t). Here Δo_t is the change in the joint observation from t to t+1, p_φ is an LSTM-based predictive distribution, and Â_t is the generalized advantage estimate produced by the actor-critic. The ReLU prevents negative-advantage transitions from inverting the gradient; the minibatch normalization keeps the update scale stable as advantage magnitudes fluctuate; and the fixed λ = 1.0 means the method needs no per-task tuning of the weighting strength. This objective is what converts a reconstruction model into one that tracks the

Load-bearing premise

The central claim collapses if the performance gain comes from the smaller predictor batch size (10 instead of 32) or the synchronization with the RL minibatch, rather than from the advantage weighting itself—the paper does not include the needed control.

What would settle it

Run the baseline, unweighted MARO predictor with batch size 10 and the same RL-synchronized update schedule under p=0.0 and p=0.2. If its returns match the reported Value-Aware MARO within noise, then advantage weighting is not the cause of the improvement; if it still collapses, the weighting mechanism is supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

When communication probability is 0.0 or 0.2, Value-Aware MARO keeps returns near their high-communication levels in HearSee, SimpleSpread-2, and SimpleSpread-4, where the unweighted baseline drops sharply.
It reduces performance variance substantially—for example, in HearSee at p=0.0, the standard deviation drops from ±26.9 to ±2.9.
The benefit is realized without modifying the RL algorithm; the same value-aware predictor improves results under both MAPPO and IPPO training.
Because both baseline and proposed predictors are trained at full communication (p=1.0), the learned advantage weights transfer from online training to degraded execution, indicating training-time weighting, not test-time weighting, drives the robustness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The reported comparison also changes the predictor batch size from 32 to 10 when adding the advantage weights; unless an unweighted batch-10 control is run, part of the improvement could be attributed to more frequent, better-aligned predictor updates rather than the weighting.
The same reweighting recipe could be applied to latent forward models or world models in MARL, which would show whether the value-aware mechanism generalizes beyond observation-delta prediction.
A direct diagnostic would be to measure predictor error separately on high- versus low-advantage transitions: if the objective works as intended, error on high-advantage transitions should drop faster, and the paper does not report such a breakdown.
The method trains under full communication, so its advantage estimates are computed from intact observations; training under randomized dropout could create a mismatch between the value signal during training and the imputed observations during execution, which would be a separate regime worth testing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The idea is neat and the writeup is transparent, but the key experiment isn't controlled: batch size and loss both change, so the 20% gains can't yet be credited to value-aware weighting.

read the letter

Quick take: this is a simple, plausible idea — weight the predictor's negative log-likelihood with the critic's advantage — and the writeup is refreshingly transparent. But the main comparison changes two things at once: the loss weighting and the predictor batch size (10 vs 32), plus the buffer/update alignment. So the central claim that advantage weighting prevents performance collapse is not established by the evidence as presented.

What the paper does well: the extension of MARO is clean and new relative to that baseline; the derivation in Eqs. 6–12 is explicitly labeled heuristic, which is honest; code is provided; the p=1.0 results confirm both methods share the same underlying policy; and reporting IPPO results is a nice touch. In the three tasks where MARO collapses (HS, SXY-2, SXY-4), the gains at p=0.0 and p=0.2 are large, and the variance numbers are lower.

The soft spots are real. First, the batch-size confound is load-bearing. Appendix II Table III lists predictor batch size as 10 for Value-Aware and 32 for the baseline, and describes the new "structurally aligned" training flow. A smaller batch size changes gradient noise and update frequency; either could improve the predictor independently of the value weighting. Any claim that the gap is "caused solely by the predictors" is true only in the narrow sense that the policy is identical; it is not true that it is caused solely by the objective. The missing control — unweighted NLL at batch 10 with the same alignment — is the single most important experiment for this paper. Second, there is no number of seeds anywhere. The ± in Table I is presumably over evaluation episodes, not repeated training runs, so the abstract's 64.7% variance reduction is ambiguous. Third, the novelty is overstated by omission: value-aware model learning has a prior literature (Farahmand et al. and successors) that is not cited. None of these are fatal to the idea, and the paper's own transparency (disclosing the batch size, labeling the derivation heuristic) makes this look like an oversight rather than a dodge.

Who this is for: people building robust MARL systems under communication dropout. The method is easy to implement on top of MARO. I'd send this to peer review with clear revision requirements: a batch-size-matched control, multi-seed reporting, and a proper related-work pass. As it stands, it's a conditional accept, not a clean one.

Referee Report

3 major / 5 minor

Summary. The paper proposes Value-Aware MARO, an extension of the MARO observation-imputation architecture in which the LSTM predictor is trained with a negative log-likelihood objective weighted by w_t ∝ ReLU(1 + λ Â_t), where Â_t is a PPO/GAE advantage estimate. A Taylor-expansion argument (Sec. IV-B) motivates this weighting as biasing the predictor toward high-advantage transitions. Experiments use MAPPO and IPPO on five MPE tasks across communication probabilities p ∈ {1.0, 0.8, 0.6, 0.4, 0.2, 0.0}. The paper reports that Value-Aware MARO prevents the baseline's performance collapse under severe communication loss, with mean-return improvements over 20% and variance reduction averaging 64.7% relative to unweighted MARO.

Significance. If the causal claim were supported, this would be a useful and low-cost contribution: it reuses the critic's advantage signal, requires no architectural change, and is orthogonal to the predictor architecture. The paper includes source code and evaluates across five tasks and two RL algorithms, which is a strength. I also agree with the stress-test assessment that the method is not circular: the advantage weights are supplied by the critic, so the predictor is not fitting to its own output. The main obstacle is experimental: the comparison with MARO changes more than the loss weighting, so the reported gains cannot currently be attributed to the value-aware objective. The derivation is heuristic in important places, and the paper should be transparent about that. These issues are fixable with a matched control and proper statistical reporting.

major comments (3)

[V-B, Appendix II (Table III)] The central claim is that advantage weighting drives the improvement, but the proposed method and the baseline differ in two predictor-training variables simultaneously. Table III reports batch size 10 for Value-Aware MARO and 32 for MARO, and states that the Value-Aware predictor is updated in alignment with the RL minibatch ('we keep them structurally aligned rather than using a separate buffer'). Thus the Sec. V-B statement that performance gaps at low p are 'caused solely by the predictors' is too strong: the predictor differs in loss weighting, batch size, and update alignment. A smaller batch size changes gradient noise and update frequency and can independently affect predictor quality. The authors should add a control consisting of the unweighted NLL objective with batch size 10 and the same alignment; if that control matches the reported Value-Aware results, the advantage weight
[Section IV-B, Eqs. (9)-(12)] Equation (10) states that the PPO update for a single transition is ∆θ ∝ α Â ∇θ log πθ(a|o). This is not the PPO update: L_CLIP is an expectation over the minibatch and includes the clipping operation. Evaluating the ratio at θ = θ_old gives the expected policy gradient, not a per-sample update, and the clipping operation is not captured. The Taylor expansion also treats o as fixed when differentiating J(φ) in Eq. (7), omitting the dependence of the state distribution on θ. Consequently Eqs. (11)-(12) are a heuristic motivation rather than a derivation. The paper should either supply a correct derivation or explicitly present the weighting as an empirical design choice; the controlled experiment in the previous comment is then what carries the burden.
[Table I and Abstract] The reported mean ± std values are not accompanied by the number of independent runs or any significance test. The abstract's quantitative claims—'average improvement in mean returns of more than 20%' and 'reduces performance variance by a mean of 64.7%'—depend on the standard deviations in Table I, but without a seed count one cannot assess whether the variance reduction is meaningful or how stable the mean improvements are. Please report the number of seeds, per-seed results or error bars in the training curves, and a significance test (e.g., Mann-Whitney U or bootstrap over seeds).

minor comments (5)

[Abstract / Section V-B] The phrase 'average improvement in mean returns of more than 20%' needs a precise definition: specify whether the average is over tasks, p-levels, or both. Table I supports several aggregations, and the current wording is ambiguous.
[Section V-B / Fig. 4] SBF and SL are not shown in Fig. 4 because the baseline does not collapse. Table I shows Value-Aware is slightly worse at p=0.0 in these tasks (-424.6 vs -422.0 for SBF; -27.4 vs -27.0 for SL). The paper should explicitly note that the method does not help in non-collapsing tasks; 'performs comparably' is accurate but should be quantified.
[Section IV-B, Eq. (3)] The notation E_t and E_{t'} is not defined. Specify that the normalization in w_t is over the current minibatch.
[Appendix II, Table II/III] The fact that the Value-Aware predictor batch size is set equal to the RL batch size is central to the comparison and should be stated in the main text, not only in the appendix.
[Section IV-B, Eq. (12)] The hyperparameter λ is fixed at 1.0 across all tasks. A sensitivity study over λ would strengthen the claimed robustness; otherwise it remains an unexamined free parameter.

Circularity Check

0 steps flagged

No significant circularity: the advantage-weighted objective is an explicit heuristic, not a self-referential fit; the main weakness is an experimental confound, not a circular derivation.

full rationale

The paper's derivation (Sec. IV-B, Eqs. 4-13) is explicitly heuristic. The proposed predictor objective J'_batch(phi) = E_t[ w_t (-log p_phi(Delta o_t | o_t)) ] uses advantage weights A_t computed by the critic, an external quantity relative to the predictor's own output. The derivation from the Taylor expansion to the final weighted NLL is labeled by the authors as 'a heuristic approximation,' and no step asserts that the conclusion is contained in the premise. The predictor is trained with p=1.0 ground-truth observations, so its training inputs do not include the low-communication evaluation returns that the paper claims to improve. No fitted parameter is renamed as a prediction, no load-bearing self-citation or uniqueness theorem is invoked, and the MARO baseline is external prior work. The only self-citation ([18], Ure et al.) appears in a related-work list and is not load-bearing. The main weakness is a confound, not circularity: Appendix II Table III changes the predictor batch size (10 vs. 32) and buffer/update alignment together with the weighting, so Sec. V-B's claim that performance gaps are 'caused solely by the predictors' is not fully isolated from the training-procedure change. That is a causal-attribution / soundness concern and does not make the derivation circular. Accordingly, the circularity score is 1 rather than 0, reflecting the minor self-citation and the unisolated comparison, but there are no circular derivation steps.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical or architectural entities. Its dependence is on (a) a heuristic simplification of the derived weight, (b) the external critic's advantage signal, and (c) a set of standard PPO/actor-critic assumptions. The most fragile item is the heuristic collapse of the sample-dependent policy-gradient norm into a fixed λ.

free parameters (1)

λ (advantage weight) = 1.0 (hand-set, untuned)
Scales the advantage contribution in the weight w_t = ReLU(1 + λ Â_t). The derivation absorbs α||∇θ log π||² into this constant, but no principled value is derived; λ=1.0 is chosen across all tasks. The paper does not report sensitivity to λ.

axioms (4)

standard math Log-derivative trick and PPO surrogate gradient are used to connect predictor loss change to policy parameter updates.
Invoked in Eq. (7) and (9); these are standard identities/approximations in policy gradient theory.
standard math First-order Taylor expansion of the predictor loss J(φ) with respect to policy parameters θ is valid.
Used in Eq. (6); requires the policy update Δθ to be small, which PPO enforces via clipping, so reasonable but not verified in the experiments.
ad hoc to paper The per-sample factor α||∇θ log π_θ(a|o)||² can be treated as a constant λ ≥ 0 for all transitions.
Explicitly stated in Sec. IV-B as a 'heuristic approximation.' This changes the derived objective: the actual expansion gives α Â ||∇θ log π||² (-log p), not λ Â (-log p). The simplification is not justified and is load-bearing for the final loss.
domain assumption Transitions with high advantage are the ones that matter for robust coordination under communication loss.
This is the motivational premise of the method: weighting by advantage focuses the predictor on 'intentional, high-return dynamics.' It is plausible but not independently established; the experiments are the only evidence.

pith-pipeline@v1.3.0-alltime-deepseek · 12027 in / 15506 out tokens · 104871 ms · 2026-08-01T16:37:06.446913+00:00 · methodology

0 comments

read the original abstract

Robust multi-agent coordination relies heavily on inter-agent communication, which is frequently disrupted by physical and environmental constraints in real-world deployments. To maintain operation during these intermittent communication failures, agents can employ internal prediction models to estimate missing shared state information. However, predictors trained with standard reconstruction objectives treat all transitions equally. In a Reinforcement Learning context, this forces the model to waste capacity learning stochastic exploration noise and the outdated dynamics of suboptimal policies. In this paper, we propose a value-aware extension of Multi-Agent Observation Sharing under Communication Dropout (MARO) to patch communication gaps; we refer to this method as Value-Aware MARO. By dynamically weighting the predictor's loss function using advantage estimates derived from the underlying actor-critic architecture, our objective explicitly couples the predictor's learning process to the policy's evolution. This formulation focuses the model's capacity on the intentional, high-return dynamics actively reinforced by the agents. We evaluate our framework on several tasks within the Multi-Agent Particle Environment under varying communication reliability levels. Experimental results demonstrate that our approach maintains performance under declining communication reliability, particularly below 40%. While our method performs comparably in tasks where the baseline already maintains high coordination, our value-aware weighting effectively prevents the performance collapse observed in the standard predictor during high-attrition scenarios. In these environments, our method achieves an average improvement in mean returns of more than 20% and reduces performance variance by a mean of 64.7% compared to the standard unweighted baseline.

Figures

Figures reproduced from arXiv: 2607.17914 by Emirhan Gazi, Eren \"Ozaltun, Feyza Orak, Kemal Devrim Kafadar, Kubilay Ka\u{g}an K\"om\"urc\"u, Mahmud Efnan \c{S}anl{\i}, Naz{\i}m Kemal \"Ure.

**Figure 1.** Figure 1: At each step, it utilizes the available joint observation from the previous timestep, ot−1, to estimate the forward transition ∆ˆ ot−1 ∼ pϕ(∆ot−1|ot−1), computing an expected current full observation oˆt ≈ ot−1 + ∆ˆ ot−1. The predictor then evaluates the integrity of the communication link. If the signal is fully intact, the policy receives the ground-truth observation. However, upon detecting signal los… view at source ↗

**Figure 1.** Figure 1: MARLxPredictor Integration During Execution: The predictor intercepts local observations to patch missing shared data during communication failures, providing a reconstructed state to the policy where o i t is local observation of agent i at time t and ht−1 is the hidden state of the LSTM-based predictor network. PPO TRAINING PREDICTOR TRAINING Critic Model 𝑉𝜔(𝑜𝑡) Value Loss ℒ𝑉(𝜔) Policy Loss ℒ𝑐𝑙𝑖𝑝(θ) Act… view at source ↗

**Figure 3.** Figure 3: MAPPOxMARO comparisons between performances [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Figure 4: Robustness profile across three coordination tasks: [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 5.** Figure 5: Representative illustrations of the MPE tasks, demon [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

**Figure 6.** Figure 6: Extended training performance with MAPPO across varying communication levels where [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

**Figure 7.** Figure 7: Evaluation curves for IPPO during training of each task are given as comparisons between MARO and Value-Aware [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

26 extracted references · 5 linked inside Pith

[1]

Tarmac: Targeted multi-agent communication,

A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pineau, “Tarmac: Targeted multi-agent communication,” in International Conference on machine learning. PMLR, 2019, pp. 1538–1546

2019
[2]

Learning to communicate with deep multi-agent reinforcement learning,

J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,”Advances in neural information processing systems, vol. 29, 2016

2016
[3]

Proximal policy optimization algorithms,

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

Pith/arXiv arXiv 2017
[4]

Message-dropout: An efficient training method for multi-agent deep reinforcement learning,

W. Kim, M. Cho, and Y . Sung, “Message-dropout: An efficient training method for multi-agent deep reinforcement learning,” inProceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 6079–6086

2019
[5]

Multiagent cooperative search learning with intermittent communication,

R. Zhang, J. Wang, J. Ge, and Q. Huang, “Multiagent cooperative search learning with intermittent communication,”IEEE Intelligent Systems, vol. 39, no. 2, pp. 11–20, 2024

2024
[6]

Multi-agent reinforcement learning for cooperative search under aperiodically intermittent communication,

L. Fu, J. Wang, and H. Luo, “Multi-agent reinforcement learning for cooperative search under aperiodically intermittent communication,” Expert Systems with Applications, vol. 280, p. 127526, 2025

2025
[7]

Decentralized multi-agent reinforcement learning with global state prediction,

J. Bloom, P. Paliwal, A. Mukherjee, and C. Pinciroli, “Decentralized multi-agent reinforcement learning with global state prediction,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 8854–8861

2023
[8]

Centralized training with hybrid execu- tion in multi-agent reinforcement learning via predictive observation imputation,

P. P. Santos, D. S. Carvalho, M. Vasco, A. Sardinha, P. A. Santos, A. Paiva, and F. S. Melo, “Centralized training with hybrid execu- tion in multi-agent reinforcement learning via predictive observation imputation,”Artificial Intelligence, p. 104404, 2025

2025
[9]

Pagnet: Pluggable adaptive generative networks for information completion in multi-agent communication,

Z. Zhang, B. Cheng, Z. Wang, Y . Zhou, G. Li, P. Lu, B. He, and J. Chen, “Pagnet: Pluggable adaptive generative networks for information completion in multi-agent communication,”arXiv preprint arXiv:2502.03845, 2025

Pith/arXiv arXiv 2025
[10]

Control of a quadrotor with reinforcement learning,

J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter, “Control of a quadrotor with reinforcement learning,”IEEE Robotics and Automation Letters, vol. 2, no. 4, pp. 2096–2103, 2017

2096
[11]

Au- tonomous drone racing with deep reinforcement learning,

Y . Song, M. Steinweg, E. Kaufmann, and D. Scaramuzza, “Au- tonomous drone racing with deep reinforcement learning,” in2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021, pp. 1205–1212

2021
[12]

Multi-agent actor-critic for mixed cooperative-competitive environments,

R. Lowe, Y . I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mor- datch, “Multi-agent actor-critic for mixed cooperative-competitive environments,”Advances in neural information processing systems, vol. 30, 2017

2017
[13]

Counterfactual multi-agent policy gradients,

J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” inProceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018

2018
[14]

Monotonic value function factorisation for deep multi- agent reinforcement learning,

T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi- agent reinforcement learning,”Journal of Machine Learning Research, vol. 21, no. 178, pp. 1–51, 2020

2020
[15]

Value-decomposition networks for cooperative multi-agent learning,

P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V . Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls,et al., “Value-decomposition networks for cooperative multi-agent learning,” arXiv preprint arXiv:1706.05296, 2017

Pith/arXiv arXiv 2017
[16]

Learning attentional communication for multi- agent cooperation,

J. Jiang and Z. Lu, “Learning attentional communication for multi- agent cooperation,”Advances in neural information processing sys- tems, vol. 31, 2018

2018
[17]

R-maddpg for partially observable environments and limited communication,

R. E. Wang, M. Everett, and J. P. How, “R-maddpg for partially observable environments and limited communication,”arXiv preprint arXiv:2002.06684, 2020

Pith/arXiv arXiv 2002
[18]

Online heterogeneous multiagent learning under limited communication with applications to forest fire management,

N. K. Ure, S. Omidshafiei, B. T. Lopez, A.-a. Agha-Mohammadi, J. P. How, and J. Vian, “Online heterogeneous multiagent learning under limited communication with applications to forest fire management,” in2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2015, pp. 5181–5188

2015
[19]

Communication in multi-agent reinforcement learning: Intention sharing,

W. Kim, J. Park, and Y . Sung, “Communication in multi-agent reinforcement learning: Intention sharing,” inInternational conference on learning representations, 2020

2020
[20]

The surprising effectiveness of ppo in cooperative multi-agent games,

C. Yu, A. Velu, E. Vinitsky, J. Gao, Y . Wang, A. Bayen, and Y . Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,”Advances in neural information processing systems, vol. 35, pp. 24 611–24 624, 2022

2022
[21]

High- dimensional continuous control using generalized advantage estima- tion,

J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High- dimensional continuous control using generalized advantage estima- tion,”arXiv preprint arXiv:1506.02438, 2015

Pith/arXiv arXiv 2015
[22]

Pettingzoo: Gym for multi-agent reinforcement learning,

J. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan, L. S. Santos, C. Dieffendahl, C. Horsch, R. Perez-Vicente,et al., “Pettingzoo: Gym for multi-agent reinforcement learning,”Advances in Neural Information Processing Systems, vol. 34, pp. 15 032–15 043, 2021. APPENDIXI PHYSICALDYNAMICS OFMPE We utilize the MPE [22], a continuous-space phys...

2021
[23]

State transitions follow semi-implicit Euler integration (dt= 0.1s), where actions apply forces (F= 5.0N), which leads the predictor to model second-order temporal dynamics

Kinematics and Inertia:Agents are circular entities with massm= 1.0kg and radiusr= 0.05. State transitions follow semi-implicit Euler integration (dt= 0.1s), where actions apply forces (F= 5.0N), which leads the predictor to model second-order temporal dynamics
[24]

The velocity update is: vt+1 =v t ×(1−0.25) + F m dt(14)

Friction and Damping:A linear damping factor of 0.25simulates air resistance, where agents retain 75% of their velocity per step. The velocity update is: vt+1 =v t ×(1−0.25) + F m dt(14)
[25]

When distanced < ri +r j, a100N contact force is applied via a log-sum-exp formulation for differentiable gradients, and these interactions cause high-frequency velocity changes

Collision Physics:MPE uses a soft-repulsion model. When distanced < ri +r j, a100N contact force is applied via a log-sum-exp formulation for differentiable gradients, and these interactions cause high-frequency velocity changes. Fig. 5 illustrates the environments, highlighting their spe- cific semantics and communication-based dependencies. See Agent He...
[26]

6: Extended training performance with MAPPO across varying communication levels wherep∈ {1.0,0.8,0.6,0.4,0.2,0.0}

For further details about the environments, models and experiments; please check the github repository at: https://github.com/robust-comm-marl-IROS2026/Value- Aware-Prediction-Under-Communication-Loss -14 -12 -10 -8 p=1.0 -14 -12 -10 p=0.8 -14 -12 -10 p=0.6 0.0 0.8 1.6 -13 -12 -10 -9 p=0.4 0.0 0.8 1.6 Training step (×106) -13 -12 -10 p=0.2 0.0 0.8 1.6 -14...

[1] [1]

Tarmac: Targeted multi-agent communication,

A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pineau, “Tarmac: Targeted multi-agent communication,” in International Conference on machine learning. PMLR, 2019, pp. 1538–1546

2019

[2] [2]

Learning to communicate with deep multi-agent reinforcement learning,

J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,”Advances in neural information processing systems, vol. 29, 2016

2016

[3] [3]

Proximal policy optimization algorithms,

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

Pith/arXiv arXiv 2017

[4] [4]

Message-dropout: An efficient training method for multi-agent deep reinforcement learning,

W. Kim, M. Cho, and Y . Sung, “Message-dropout: An efficient training method for multi-agent deep reinforcement learning,” inProceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 6079–6086

2019

[5] [5]

Multiagent cooperative search learning with intermittent communication,

R. Zhang, J. Wang, J. Ge, and Q. Huang, “Multiagent cooperative search learning with intermittent communication,”IEEE Intelligent Systems, vol. 39, no. 2, pp. 11–20, 2024

2024

[6] [6]

Multi-agent reinforcement learning for cooperative search under aperiodically intermittent communication,

L. Fu, J. Wang, and H. Luo, “Multi-agent reinforcement learning for cooperative search under aperiodically intermittent communication,” Expert Systems with Applications, vol. 280, p. 127526, 2025

2025

[7] [7]

Decentralized multi-agent reinforcement learning with global state prediction,

J. Bloom, P. Paliwal, A. Mukherjee, and C. Pinciroli, “Decentralized multi-agent reinforcement learning with global state prediction,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 8854–8861

2023

[8] [8]

Centralized training with hybrid execu- tion in multi-agent reinforcement learning via predictive observation imputation,

P. P. Santos, D. S. Carvalho, M. Vasco, A. Sardinha, P. A. Santos, A. Paiva, and F. S. Melo, “Centralized training with hybrid execu- tion in multi-agent reinforcement learning via predictive observation imputation,”Artificial Intelligence, p. 104404, 2025

2025

[9] [9]

Pagnet: Pluggable adaptive generative networks for information completion in multi-agent communication,

Z. Zhang, B. Cheng, Z. Wang, Y . Zhou, G. Li, P. Lu, B. He, and J. Chen, “Pagnet: Pluggable adaptive generative networks for information completion in multi-agent communication,”arXiv preprint arXiv:2502.03845, 2025

Pith/arXiv arXiv 2025

[10] [10]

Control of a quadrotor with reinforcement learning,

J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter, “Control of a quadrotor with reinforcement learning,”IEEE Robotics and Automation Letters, vol. 2, no. 4, pp. 2096–2103, 2017

2096

[11] [11]

Au- tonomous drone racing with deep reinforcement learning,

Y . Song, M. Steinweg, E. Kaufmann, and D. Scaramuzza, “Au- tonomous drone racing with deep reinforcement learning,” in2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021, pp. 1205–1212

2021

[12] [12]

Multi-agent actor-critic for mixed cooperative-competitive environments,

R. Lowe, Y . I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mor- datch, “Multi-agent actor-critic for mixed cooperative-competitive environments,”Advances in neural information processing systems, vol. 30, 2017

2017

[13] [13]

Counterfactual multi-agent policy gradients,

J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” inProceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018

2018

[14] [14]

Monotonic value function factorisation for deep multi- agent reinforcement learning,

T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi- agent reinforcement learning,”Journal of Machine Learning Research, vol. 21, no. 178, pp. 1–51, 2020

2020

[15] [15]

Value-decomposition networks for cooperative multi-agent learning,

P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V . Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls,et al., “Value-decomposition networks for cooperative multi-agent learning,” arXiv preprint arXiv:1706.05296, 2017

Pith/arXiv arXiv 2017

[16] [16]

Learning attentional communication for multi- agent cooperation,

J. Jiang and Z. Lu, “Learning attentional communication for multi- agent cooperation,”Advances in neural information processing sys- tems, vol. 31, 2018

2018

[17] [17]

R-maddpg for partially observable environments and limited communication,

R. E. Wang, M. Everett, and J. P. How, “R-maddpg for partially observable environments and limited communication,”arXiv preprint arXiv:2002.06684, 2020

Pith/arXiv arXiv 2002

[18] [18]

Online heterogeneous multiagent learning under limited communication with applications to forest fire management,

N. K. Ure, S. Omidshafiei, B. T. Lopez, A.-a. Agha-Mohammadi, J. P. How, and J. Vian, “Online heterogeneous multiagent learning under limited communication with applications to forest fire management,” in2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2015, pp. 5181–5188

2015

[19] [19]

Communication in multi-agent reinforcement learning: Intention sharing,

W. Kim, J. Park, and Y . Sung, “Communication in multi-agent reinforcement learning: Intention sharing,” inInternational conference on learning representations, 2020

2020

[20] [20]

The surprising effectiveness of ppo in cooperative multi-agent games,

C. Yu, A. Velu, E. Vinitsky, J. Gao, Y . Wang, A. Bayen, and Y . Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,”Advances in neural information processing systems, vol. 35, pp. 24 611–24 624, 2022

2022

[21] [21]

High- dimensional continuous control using generalized advantage estima- tion,

J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High- dimensional continuous control using generalized advantage estima- tion,”arXiv preprint arXiv:1506.02438, 2015

Pith/arXiv arXiv 2015

[22] [22]

Pettingzoo: Gym for multi-agent reinforcement learning,

J. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan, L. S. Santos, C. Dieffendahl, C. Horsch, R. Perez-Vicente,et al., “Pettingzoo: Gym for multi-agent reinforcement learning,”Advances in Neural Information Processing Systems, vol. 34, pp. 15 032–15 043, 2021. APPENDIXI PHYSICALDYNAMICS OFMPE We utilize the MPE [22], a continuous-space phys...

2021

[23] [23]

State transitions follow semi-implicit Euler integration (dt= 0.1s), where actions apply forces (F= 5.0N), which leads the predictor to model second-order temporal dynamics

Kinematics and Inertia:Agents are circular entities with massm= 1.0kg and radiusr= 0.05. State transitions follow semi-implicit Euler integration (dt= 0.1s), where actions apply forces (F= 5.0N), which leads the predictor to model second-order temporal dynamics

[24] [24]

The velocity update is: vt+1 =v t ×(1−0.25) + F m dt(14)

Friction and Damping:A linear damping factor of 0.25simulates air resistance, where agents retain 75% of their velocity per step. The velocity update is: vt+1 =v t ×(1−0.25) + F m dt(14)

[25] [25]

When distanced < ri +r j, a100N contact force is applied via a log-sum-exp formulation for differentiable gradients, and these interactions cause high-frequency velocity changes

Collision Physics:MPE uses a soft-repulsion model. When distanced < ri +r j, a100N contact force is applied via a log-sum-exp formulation for differentiable gradients, and these interactions cause high-frequency velocity changes. Fig. 5 illustrates the environments, highlighting their spe- cific semantics and communication-based dependencies. See Agent He...

[26] [26]

6: Extended training performance with MAPPO across varying communication levels wherep∈ {1.0,0.8,0.6,0.4,0.2,0.0}

For further details about the environments, models and experiments; please check the github repository at: https://github.com/robust-comm-marl-IROS2026/Value- Aware-Prediction-Under-Communication-Loss -14 -12 -10 -8 p=1.0 -14 -12 -10 p=0.8 -14 -12 -10 p=0.6 0.0 0.8 1.6 -13 -12 -10 -9 p=0.4 0.0 0.8 1.6 Training step (×106) -13 -12 -10 p=0.2 0.0 0.8 1.6 -14...