{"id":"4c147f73-620a-45e7-a534-5c66badc30c3","arxiv_id":"2501.01140","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A decentralized MARL method that communicates observation-prediction error as an auxiliary message improves performance on out-of-distribution warehouse tasks.","lead":"Multi-agent reinforcement learning agents often fail when the environment changes in ways they never saw while training. This paper proposes letting each agent send a message that encodes how surprising its latest observation is, alongside the usual reward-driven communication, and tests it in a simulated warehouse.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The OOD gain is not yet attributable to 'unexpectedness': no control isolates UES content from channel capacity, hyperparameter differences, and the extra learned UEM.","rationale":"The reader's weakest assumption already points at the missing isolation of the UES message content. I agree, and I sharpen the concern by noting that hyperparameter differences (learning rate, entropy coefficient) and the 5+5 message split are additional confounds beyond channel capacity. This is load-bearing because Table 1 is the only evidence for the central claim; without a content-matched communication control, the result is compatible with a broader 'extra observation-dependent input helps' explanation. I do not see an internal inconsistency or a fatal flaw: the method is clearly described, the numbers consistently favor UES+R over the decentralized baselines, and the centralized MAPPO comparison is a reasonable upper-bound reference. The correct remedy is an additional ablation and a fully specified fine-tuning protocol, not rejection. A code release would also help, but the decisive scientific issue is the missing control. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":9364,"tokens_out":7430,"duration_ms":80913,"concrete_test":"Re-run RWARE (Section 3.1) with a single shared hyperparameter configuration (e.g., lr=0.0005, entropy coefficient=0.01) and fix all message lengths at 10 bits. Add two controls to Table 1: (a) IA2C+M(R,5)+M(RAND,5), where the second 5-bit channel is fixed random noise or a fixed random projection of the sender's observation; and (b) IA2C+M(R,5)+M(OBS,5), where the second 5-bit channel is a learned or fixed 5-bit projection of the sender's local observation embedding g(o_t). Train both for 10M steps, fine-tune under Goal-Shift and Shelf-Shift using the exact 10-batch protocol, and report per-seed results. If UES+R's margin over these controls shrinks to within seed noise, the gain cannot be attributed specifically to the unexpectedness encoding; if it survives, the proposed mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that communicating observation-prediction discrepancy (UES) as an auxiliary message drives the OOD gains in Table 1. The experiments do not yet isolate that mechanism. The compared methods differ in several dimensions at once: +M(UES) uses lr=0.001 while +M(UES+R) and +M(R) use 0.0005; +M(UES+R) uses an entropy coefficient of 0.05 versus 0.01 for the other methods; +M(UES+R) splits its 10-bit message into 5 reward bits and 5 UES bits, so neither component has the 10-bit capacity of the single-channel baselines; and only the UES variants add the forward-dynamics model and autoencoder of Section 2.2. The paper equalizes total message length (Section A.2) but does not control for what the auxiliary 5 bits encode. Because the UES message is a deterministic function of the sender's local observations, an equally sized message carrying raw local observations, a random projection of them, or a fixed random signal would test whether 'surprise' content, rather than extra observation-dependent input or extra exploration, is responsible. Since the autoencoder is trained only on reconstruction loss (Eq. 3) and UES is never ablated against a content-matched communication channel, the observed improvement could be explained by additional channel capacity for local observation sharing or by the higher entropy coefficient. If the content-matched controls match UES+R's performance, the paper's specific claim about communicating unexpectedness is not supported by the current experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Unexpectedness Encoding Scheme with Reward (UES+R), a decentralized MARL communication method for out-of-distribution adaptation. Each agent maintains a forward dynamics model that predicts the next observation; the prediction error is compressed by an autoencoder trained only on reconstruction loss, and the resulting continuous vector is concatenated with a reward-trained discrete message. The combined message is fed into the agents' policies. Experiments in a two-agent Multi-Robot Warehouse environment compare IA2C, IA2C with reward-only messages, IA2C with UES-only messages, IA2C with UES+R, and centralized MAPPO, in the training distribution and after Goal-Shift and Shelf-Shift. The reported results show UES+R outperforms the decentralized baselines in all settings and is competitive with MAPPO in most settings. The paper interprets these gains as evidence that communicating observation-prediction discrepancy helps agents adapt to distribution shifts.","tokens_in":9632,"tokens_out":5880,"duration_ms":53756,"significance":"The proposed mechanism is original and, if substantiated, practically useful: it uses a reward-free surprise signal to share information about distribution shift in a decentralized setting, and the architecture is simple and additive to any MARL communication baseline. The paper also provides a clear statement of the broadcast limitation and points to compression as future work. However, the empirical support is currently too narrow to establish the central claim. The experiments are restricted to a single two-agent environment, use only five seeds, report no significance tests, and—most importantly—do not isolate the UES message content from channel capacity, hyperparameter differences, or the extra learned forward model. The contribution is credible but not yet convincingly demonstrated.","major_comments":[{"comment":"The ablations do not use matched hyperparameters. IA2C+M(UES) is trained with learning rate 0.001, while IA2C+M(R) and IA2C+M(UES+R) use 0.0005; IA2C+M(UES+R) uses an entropy coefficient of 0.05, whereas all other methods use 0.01. Because the proposed method differs from the baselines on these dimensions, the improvements in Table 1 cannot currently be attributed to the UES message content. Please rerun all methods with identical learning rates and entropy coefficients, or provide a sensitivity analysis showing the reported ranking is invariant to these choices.","section":"Appendix A.2"},{"comment":"No control isolates the content of the UES message. IA2C+M(UES+R) sends a 5-bit UES component plus a 5-bit reward component, while IA2C+M(UES) and IA2C+M(R) each send 10 bits of a single type. The observed gains could therefore come from the extra observation-dependent channel, from the particular split of capacity, or from increased exploration (via the higher entropy coefficient), rather than from 'unexpectedness' as defined in Eq. (1). Add content-matched controls—for example, a 5-bit random projection of the local observation, or a fixed random signal, concatenated with the same 5-bit reward message—to test the specific claim that the prediction-error content is the driver.","section":"Section 3.3, Table 1"},{"comment":"The statistical support is weak. The study uses only one environment (two-agent RWARE), five seeds, and no significance tests. Some comparisons are also close: in Shelf-Shift, IA2C+M(UES+R) (1.45±0.07) and IA2C+M(R) (1.33±0.12) overlap at one standard deviation. Please report confidence intervals or significance tests and either increase the number of seeds or add an additional environment/task before making general claims about robust OOD adaptation.","section":"Section 3.2, Table 1"},{"comment":"It is not specified whether the forward dynamics model f and the autoencoder are updated during the 10-batch fine-tuning phase. If they are updated, the 'unexpectedness' signal is no longer a fixed measure of distribution shift; if they are frozen, the stale predictor may be responsible for the observed adaptation. Please state this explicitly and, ideally, compare both variants.","section":"Section 3.2"}],"minor_comments":[{"comment":"Equation (1) defines x_t = f(o_{t-1}, m_{t-2}, a_{t-1}) - o_t, but Appendix A.2 defines x_t = g(hat o_t) - g(o_t) using a random linear projection g. Please unify these definitions.","section":"Eq. (1) vs Appendix A.2"},{"comment":"The sentence 'we omitted the the superscript' contains a typo, and Eq. (4) does not show the entropy regularization mentioned in the text; please add the entropy term or clarify the notation.","section":"Section 2.3"},{"comment":"The claim that 'merely using reward or UES individually led to performances on par with or even worse than IA2C' is too strong given that the error bars overlap for some of the comparisons in Table 1; please soften the wording or add statistical evidence.","section":"Section 3.4"},{"comment":"The limitations paragraph mentions the broadcast assumption but not the absence of a content-matched ablation for the UES message; this omission should be acknowledged.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper is currently at a conditional stage. The central idea is interesting, but the lack of content-matched controls and hyperparameter matching makes the headline claim unproven. The authors should be asked for more rigorous ablations, additional seeds or environments, and a reproducibility check before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the UES+R paper. The idea is simple and worth a look: agents run a forward dynamics model and communicate the discrepancy between predicted and actual observations, compressed by an autoencoder, alongside a reward-trained message. That is distinct from Abu et al.'s reward-based confusion, and the combination is nicely motivated. The paper writes things up clearly, and the learning curves and table are internally consistent: UES+R is better than either component alone and matches or beats the centralized MAPPO baseline in most settings. Credit where due: the method is easy to understand, the ablations separate reward-only and UES-only messages, and the authors admit the broadcast assumption and the lack of message interpretability.\n\nThe soft spots are method-level, not fatal. The biggest one is that the experiments do not isolate the 'unexpectedness' content. UES+R differs from +M(R) in three ways at once: an extra 5-bit channel carries UES output, the actor-critic learning rate and entropy coefficient are different (0.0005/0.05 vs 0.0005/0.01), and the message split changes capacity. So even if the numbers hold, the contribution of the surprise signal itself is confounded. A content-matched control—same number of bits carrying raw local observations, a random projection, or a fixed random signal—would settle that. The paper does not do this. The reader's conditional verdict is reasonable. There is also no code release, only five seeds, no significance tests, and one environment. These are real problems, but not disqualifying; they make the result suggestive rather than conclusive.\n\nThe claim itself is not circular: UES is defined by prediction error, not fitted to the performance metric, and the forward model is trained on the same distribution, so it is a measure of novelty relative to training experience. That is a sensible definition of unexpectedness. The math in Eq. 1–3 is straightforward, and the derivation is honest. The paper's own limitation paragraph about broadcasting is a fair acknowledgment, though it does not fix the confound.\n\nOverall, this is a within-subfield paper that a serious referee should see. It is not a breakthrough, and the central attribution needs a cleaner experiment, but the idea is coherent and the direction is useful. If I were editing, I would send it to peer review with a request for the content-matched ablation and hyperparameter egalization before publication.","headline":"A plausible, clearly written communication scheme for OOD MARL, but the current experiments do not yet isolate the 'unexpectedness' mechanism from channel capacity and hyperparameter confounds.","tokens_in":10194,"tokens_out":1827,"would_cite":false,"duration_ms":18254,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that cooperative agents can adapt to out-of-distribution changes by communicating a compressed version of their observation-prediction error, and demonstrates the benefit in a multi-robot warehouse task.","keywords":["multi-agent reinforcement learning","out-of-distribution generalization","agent communication","decentralized training","forward dynamics model","prediction error","few-shot adaptation","multi-robot warehouse"],"falsifier":"Compare UES+R with a control that concatenates the reward-driven message with an encoding of random noise of the same length and trains it under the same 10M-step protocol and 10-episode fine-tuning. If the control matches UES+R's delivered-shelf counts under Goal-Shift and Shelf-Shift, then the semantic content of the unexpectedness encoding is not what drives the reported improvement.","tokens_in":9144,"feed_emoji":"🤖","tokens_out":8964,"duration_ms":76527,"temperature":0.7,"pith_summary":"Out-of-distribution situations are events an agent never saw in training, and a multi-agent team can adapt to them only if one agent's surprise can reach the others. This paper claims that surprise can be generated automatically: each agent learns a forward model that predicts its next local observation, measures the difference between the prediction and the observation it actually receives, compresses that difference with an autoencoder, and broadcasts the compressed vector as a message. When this 'unexpectedness encoding' is concatenated with a message trained by environmental reward, two decentralized agents in the Multi-Robot Warehouse environment deliver more shelves on the training distribution and after two designed distribution shifts than either message type alone or no communication. In the reported experiments, the combined scheme reaches 2.51 delivered shelves per episode in training, 2.29 after a goal-layout shift, and 1.45 after a shelf-layout shift. If the claim is right, communicating prediction error is a practical route to few-shot adaptation in partially observable cooperative tasks.","feed_headline":"Surprise signals let warehouse robots adapt to shifted layouts","feed_subtitle":"Decentralized agents that share prediction errors beat no-communication baselines and match centralized training.","key_machinery":"The load-bearing mechanism is the Unexpectedness Encoding Module (UEM). At each step, agent $i$'s forward model $f$ predicts the next projected observation $g(\\hat{o}^i_t)$ from the previous projected observation $g(o^i_{t-1})$, the previous action $a^i_{t-1}$, and the messages it received; the difference between this prediction and the actual projected observation $g(o^i_t)$ is the 'unexpectedness' vector. A reconstruction-trained autoencoder maps that vector into a short continuous message, which is concatenated with a reward-trained binary message to form the agent's broadcast. This concatenated message is the only channel through which one agent's unexpected experience can reach another, and the paper's ablations show that removing either half of it removes the benefit.","core_discovery":"The central claim is that in cooperative tasks with partial observability, the discrepancy between what an agent expects to observe and what it actually observes is a communicable signal that helps the whole team adapt to distribution shift. The proposed UES+R scheme computes this discrepancy on a linear projection of the observation, $\\mathbf{x}^i_t = g(\\hat{o}^i_t) - g(o^i_t)$, where the prediction $\\hat{o}^i_t = f(o^i_{t-1}, m^{-i}_{t-2}, a^i_{t-1})$ comes from a forward dynamics module trained by an $\\ell^2$ prediction loss. An autoencoder trained only on reconstruction loss compresses $\\mathbf{x}^i_t$ into a fixed-length message, and this message is concatenated with a separate binary message trained through the same actor-critic objective as the actions. In the two-agent Multi-Robot Warehouse environment, the paper reports that this combination delivers more requested shelves on the training distribution and after both designed distribution shifts than either message type alone or no communication, and it attributes the gain to the complementarity of surprise-oriented and reward-oriented information.","pith_inferences":["The method's signal depends on the forward model's prediction error remaining meaningful during adaptation; if the forward model is updated too aggressively while fine-tuning, the discrepancy shrinks and the message may stop flagging novelty. A direct test is to freeze the UEM weights during the 10-episode fine-tuning window and compare with updating them.","Because the surprise vector is computed from each agent's own $3\\times3$ local observation, shifts that change only the reward function or another agent's hidden strategy would not be detected by this channel. Extending UES to predict rewards or joint states might cover those cases.","The paper assumes messages are broadcast to all agents; in larger teams the concatenated message length grows with team size. An attentional selection mechanism over received messages is the natural scaling fix, a direction the paper names as a limitation."],"forward_implications":["Decentralized agents can approach the robustness of centralized training in these warehouse shifts without sharing raw observations.","Reward-driven and surprise-driven messages are complementary: each alone performs about as well as silence, while the pair does better.","A short autoencoded surprise vector of fixed length is enough to carry the adaptation signal, so the scheme does not need to transmit full observations.","The same few-shot protocol can be applied to both beneficial shifts (extra goal tiles) and harmful shifts (shelves moved toward walls), giving a reusable evaluation template."],"supporting_citations":[{"why":"introduces the prior confusion-based communication idea that UES+R contrasts and extends","marker":"Abu et al. 2021"},{"why":"supplies the Multi-Robot Warehouse benchmark and its task protocol","marker":"Papoudakis et al. 2020"},{"why":"provides the A2C algorithm used as the base learner for all decentralized methods","marker":"Mnih et al. 2016"},{"why":"provides MAPPO, the centralized-training baseline treated as an upper bound","marker":"Yu et al. 2021"},{"why":"gives the Dec-POMDP formalization that frames the partial-observability problem","marker":"Oliehoek, Spaan, and Vlassis 2008"},{"why":"establishes the reward-driven communication setup in which message bits are trained with the same objective as actions","marker":"Jaques et al. 2019"}],"fun_headline_variants":["Share prediction errors to adapt agents to unseen shifts","Surprise signals let robot teams adapt to new layouts","Communicating unexpectedness improves multi-agent adaptation","Robots that broadcast surprise adapt to distribution shift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the difference between what a forward model predicts and what an agent actually observes, after compression by a reconstruction-trained autoencoder, still conveys information that helps other agents adapt to a distribution shift; the paper does not separately test whether the encoded content matters beyond supplying an extra communication channel.","fun_headline_variants_meta":{"raw":{"variants":["Share prediction errors to adapt agents to unseen shifts","Surprise signals let robot teams adapt to new layouts","Communicating unexpectedness improves multi-agent adaptation","Robots that broadcast surprise adapt to distribution shift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1422,"prompt_tokens":906,"completion_tokens":516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":457}},"tokens_in":522,"tokens_out":516,"duration_ms":5218,"temperature":1.0,"reasoning_tokens":457,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:34:12.034827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare UES+R with a control that concatenates the reward-driven message with an encoding of random noise of the same length and trains it under the same 10M-step protocol and 10-episode fine-tuning. If the control matches UES+R's delivered-shelf counts under Goal-Shift and Shelf-Shift, then the semantic content of the unexpectedness encoding is not what drives the reported improvement.","supporting_citations":[{"cited_title":"Collaboration Promotes Group Resilience in Multi-Agent RL","cited_arxiv_id":"2111.06614","evidence_quote":"introduces the prior confusion-based communication idea that UES+R contrasts and extends"},{"cited_title":"P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K","cited_arxiv_id":null,"evidence_quote":"provides the A2C algorithm used as the base learner for all decentralized methods"},{"cited_title":"A.; Spaan, M","cited_arxiv_id":null,"evidence_quote":"gives the Dec-POMDP formalization that frames the partial-observability problem"},{"cited_title":"Z.; and De Freitas, N","cited_arxiv_id":null,"evidence_quote":"establishes the reward-driven communication setup in which message bits are trained with the same objective as actions"}],"review_version":1}