{"id":"61419487-ea52-4e96-8878-c8063f346269","arxiv_id":"2501.10593","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A new benchmark shows that IPPO, a standard MARL algorithm, cannot reliably infer a partner's changing goal from its actions alone.","lead":"This paper introduces ColorGrid, a grid-world benchmark where two agents cooperate while the target color changes over time and is visible to only one of them. It shows that a widely used multi-agent reinforcement learning algorithm, IPPO, fails to solve the hidden-goal version and releases code for others to try.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure 3 compares IPPO follower against A* copying follower under different leaders, confounding follower ability with leader predictability.","rationale":"The reader's weakest assumption concerns seed count and hyperparameter search. While those are legitimate, the more load-bearing issue is the uncontrolled leader policy in the central comparison of Figure 3. The paper's headline negative result, 'ColorGrid is unsolved by IPPO,' rests on the follower's inability to outperform a simple copying heuristic. But that heuristic is evaluated with a different, cleaner leader (A*) than the IPPO-trained follower is trained and evaluated with. If the IPPO leader is a less informative communication partner, the follower's task is harder independent of its goal-inference capacity. This is not a mere statistical robustness issue; it is a potential systematic confound. The proposed cross-evaluation test would settle whether the deficit survives when the leader is held fixed. The paper remains a useful benchmark contribution, so a conditional acceptance is still appropriate, but the required revision should include a matched-leader comparison or a clear statement that the negative result is specific to the tested leader policy.","tokens_in":10433,"tokens_out":5057,"duration_ms":49134,"concrete_test":"Cross-evaluate in the neutral setting: (1) take the best IPPO-trained follower from Figure 3 (seed 0) and run it with the A* leader that the A* copying baseline uses; (2) take the A* copying follower and run it with the frozen IPPO leader used to train the IPPO follower. Compare per-episode returns. If the IPPO follower's score approaches the A* copying baseline when paired with the A* leader, the Figure 3 deficit is an artifact of the different leaders rather than of IPPO's goal-inference capability. Run at least 5 seeds for each pairing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 (Figure 3) trains the IPPO follower with a frozen IPPO leader, but the A* copying baseline is paired with the A* leader. The two leaders have similar cumulative reward, but comparable reward does not imply comparable trajectory statistics. The IPPO leader may be slower to switch goals, take non-shortest paths, or hesitate, making the hidden goal less identifiable from observations than the deterministic A* leader does. The follower's task difficulty is therefore not matched across conditions, so the reported gap between IPPO and A* copying conflates the follower's inference ability with the leader's expressiveness. The claim 'IPPO is insufficient to learn to infer the leader's goal' requires holding the leader fixed; the paper does not do that.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ColorGrid, a PettingZoo-based 32x32 grid-world MARL environment in which a leader and a follower collect colored blocks, the goal color can switch during an episode, and the follower may be denied access to the goal color. The authors release the environment code, model checkpoints, and trajectory visualizations. They evaluate Independent PPO (IPPO) under symmetric and asymmetric goal information, with ablations over reward magnitude, penalty annealing, an auxiliary goal-prediction loss, goal concatenation location, distance and potential-field reward shaping, and goal-switch probability. Their central empirical claim is that the asymmetric setting with non-stationary hidden goals is currently unsolved by IPPO, and that an A* 'copying' follower substantially outperforms all trained IPPO followers. The paper presents this as a benchmark motivation for future MARL goal-inference algorithms.","tokens_in":10738,"tokens_out":3045,"duration_ms":33372,"significance":"If the central negative result is reliable, the paper would provide a useful benchmark artifact for the under-studied problem of real-time hidden-goal inference in cooperative MARL. The strongest parts of the submission are concrete: the environment is implemented and released, the state representation and reward structure are clearly specified, the neural architecture and training details are described in enough detail to reproduce, and several ablations (penalty annealing, auxiliary loss, goal concatenation position) go beyond a single environment demonstration. However, the load-bearing claim that IPPO is insufficient for goal inference rests on a small number of seeds and on an unmatched comparison between different leader policies; the manuscript itself acknowledges in Section 5.1 the need for 'running more experiments with different seeds and determining optimal hyperparameter configurations.' The benchmark contribution is sound as an environment release, but the paper's main negative result needs stronger statistical support and a deconfounded experimental design before it can support the stated conclusions.","major_comments":[{"comment":"The central evidence that the IPPO follower fails to infer the leader's goal is reported for a single seed. The Figure 3 caption states 'We use seed 0 for these comparisons,' while the A* baseline scores are averaged over 100 seeds. Without error bars, confidence intervals, or multiple seeds for the IPPO curves, the claim that IPPO is 'insufficient to learn to infer the leader's goal' cannot be distinguished from a seed-dependent training outcome. This is load-bearing because the abstract and Section 5 both present the negative result as the paper's main empirical finding.","section":"§4.2, Figure 3"},{"comment":"The comparison between the trained IPPO follower and the A* copying follower confounds follower inference ability with leader expressiveness. The IPPO follower is trained against a frozen IPPO leader, while the A* copying baseline is paired with an A* leader. The text notes that the two leaders have similar cumulative reward, but similar reward does not imply similar trajectory statistics; the IPPO leader may switch goals more slowly, wander, or take non-shortest paths, making the hidden goal harder or easier to infer. The claim 'IPPO is insufficient to learn to infer the leader's goal' requires holding the leader policy fixed across follower conditions, or at least controlling the information content of the leader's trajectories. As written, the reported gap conflates the follower's inference capability with the leader policy used to generate the demonstrations.","section":"§4.2, Figure 3"},{"comment":"The quantitative support for the effect of penalty annealing and the auxiliary loss is based on Table 2, which averages converged rewards over only 3 seeds and reports no variance, confidence intervals, or per-seed values. Since the paper's broad claim is that ColorGrid is 'unsolved by IPPO,' the absence of any uncertainty quantification is a material gap: with three seeds and no error bars, the reader cannot assess whether the differences between the four rows (e.g., 48.8 vs. 32.8) are real effects or training noise. The paper should either provide many more seeds with variance reporting or substantially soften the scope of the claim.","section":"§4, Table 2; Appendix A.2"},{"comment":"The 'unsolved by IPPO' conclusion is stated despite the use of a single fixed hyperparameter configuration (Appendix A.2) and no systematic hyperparameter search. Section 5.1 explicitly lists 'determining optimal hyperparameter configurations' as future work, which is appropriate, but the abstract and Section 4.2 nevertheless assert that the environment is 'currently unsolved by IPPO.' That assertion is stronger than the evidence supports: without a hyperparameter sensitivity analysis or at least a demonstration that reasonable variations do not change the outcome, the negative result could be an artifact of the chosen learning rate, entropy coefficient, network size, or other fixed settings. The authors should either provide such a sensitivity analysis or rephrase the claim as 'unsolved under the tested default configuration.'","section":"§4, §5.1"}],"minor_comments":[{"comment":"In the paragraph on customizability, the sentence 'The is also customizable' is incomplete and should be finished or removed.","section":"§3.1"},{"comment":"The text says 'see 3.1 for the computation explanation' of the goal-switch probability, but the derivation appears in Appendix A.3; the cross-reference should point to the appendix.","section":"§3.1"},{"comment":"The default goal-switch probability is given as 2.08% in the main text and Appendix A.3, but Section 4 states 'goal switch probability of 2%' when describing the default settings; these numbers should be made consistent.","section":"§3.1, Appendix A.3"},{"comment":"Equation (1) defines c_i but does not define the predicted probability \\hat c_i; please add a sentence clarifying that \\hat c_i is the auxiliary network's softmax output for color i.","section":"§3.2.1, Eq. (1)"},{"comment":"The statement that A* with an admissible heuristic 'is guaranteed to perform optimally' is imprecise for graph search unless the heuristic is also consistent; for this grid setting the intended meaning is clear, but the wording should be corrected.","section":"§4.1, footnote 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about its own limitations (Section 5.1 explicitly mentions seeds and hyperparameters), and the code release is a concrete asset. The main risk is that the headline negative result is currently under-powered and confounded; I believe this is fixable within the manuscript's scope by adding multi-seed experiments with matched leader policies and variance reporting. I do not see evidence of a deeper methodological flaw or a fabricated result, so rejection is not warranted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. ColorGrid is a genuinely useful little testbed: non-stationary goals plus asymmetric information in a simple grid, with code released. The environment fills a real gap — GoalCycle is stationary, Coingrid's goals are hidden but fixed, MPE is non-stationary but fully observed. The cost-of-exploration ablation (positive/neutral/negative EV) is a nice idea, and the paper is honestly written; the limitations section even admits the seed problem. The soft spot is the paper's main claim. Figure 3, the key evidence that IPPO fails, compares an IPPO follower trained against a frozen IPPO leader against an A* copying follower paired with an A* leader. Those are different leaders with different trajectory statistics; comparable cumulative reward does not mean comparable goal-inference difficulty. So the comparison confounds follower ability with leader predictability. On top of that, the IPPO runs use seed 0, Table 2 averages 3 seeds with no error bars, and hyperparameters are fixed without a search. The benchmark construction is sound, but the 'unsolved by IPPO' conclusion is provisional. This is fixable. The code release means others can rerun. The authors should either hold the leader fixed across both follower conditions or report both leader types, add seeds and variance, and temper the abstract. As is, the environment contribution stands; the negative result does not. Bottom line: worth a serious referee, because a new benchmark with released code deserves scrutiny, but the authors need to shore up the evidence before the difficulty claim is credible.","headline":"Useful new benchmark, but the 'IPPO cannot solve it' claim rests on a confounded leader comparison and a single seed.","tokens_in":11102,"tokens_out":2514,"would_cite":true,"duration_ms":25685,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard cooperative MARL, IPPO, fails to learn a partner's changing hidden goal in the new ColorGrid benchmark.","keywords":["multi-agent reinforcement learning","non-stationary environment","goal inference","hidden goals","asymmetric information","IPPO","benchmark environment","social learning"],"falsifier":"Train an IPPO follower in the asymmetric, neutral-expected-reward ColorGrid setting across at least ten seeds using the paper's reported hyperparameters, and compare mean final reward to the A* copying follower baseline averaged over 100 seeds; if any seed or the mean reaches or exceeds the A* follower's reward, the claim that ColorGrid is unsolved by IPPO is falsified.","tokens_in":10269,"feed_emoji":"🎯","tokens_out":9307,"duration_ms":83901,"temperature":0.7,"pith_summary":"This paper introduces ColorGrid, a two-agent grid-world benchmark in which one agent (the leader) knows the current goal block color while the other (the follower) must infer it from the leader's movements, and the goal color switches over time. The paper's central claim is that IPPO, a standard cooperative multi-agent reinforcement learning algorithm, cannot solve the asymmetric, non-stationary setting: with the goal hidden from the follower, trained followers fall behind a trivial baseline that simply copies the last color the leader collected. This matters because real human-AI collaboration often requires an assistant to infer a changing, unspoken human goal, and the paper argues that current state-of-the-art MARL methods lack this capability. The paper also reports that the cost of exploration—whether random block collection has positive, neutral, or negative expected reward—strongly shapes what the follower learns, with only the neutral case producing partial goal-following behavior.","feed_headline":"New benchmark shows IPPO cannot infer a partner's shifting goal","feed_subtitle":"In ColorGrid, a follower that copies the leader's last move outperforms all IPPO-trained agents.","key_machinery":"The load-bearing mechanism is the combination of three design choices in ColorGrid: (1) a non-stationary goal, where the goal block color switches with probability $2/3 \\cdot 1/32 \\approx 2.08\\%$ per time step; (2) asymmetric information, where the leader sees the goal one-hot vector but the follower receives zeros in its place; and (3) a tunable cost of exploration, set by choosing the goal reward ($+1$, $+2$, or $+4$) against a fixed $-1$ penalty for incorrect blocks. These components force the follower to infer the leader's intent from trajectory observations alone. The paper pairs the environment with an IPPO implementation that includes an LSTM for the asymmetric follower, an auxiliary cross-entropy loss for predicting the goal color, and penalty annealing that ramps the incorrect-block penalty from $0$ to $1$ between 4M and 10M training steps.","core_discovery":"In ColorGrid's asymmetric mode, the leader always knows the goal color and receives a $+1$ reward for collecting a matching block and $-1$ for others, while the follower sees only the board and the leader's trajectory; there is no message channel. Training a follower with IPPO against a frozen IPPO-trained leader yields poor behavior across all reward structures tested: with positive expected value the follower collects every block, with negative expected value it collects none, and only with neutral expected value does it sometimes pursue the correct color. Averaged over three seeds in the symmetric setting, IPPO converges to positive reward only when the incorrect-block penalty is annealed in over training and when an auxiliary supervised goal-prediction loss is used. In the asymmetric setting, the A* copying follower—which routes to the last color picked up by the leader—significantly outperforms all IPPO-trained followers, and the paper concludes that ColorGrid with non-stationary and asymmetric goals is currently unsolved by IPPO.","pith_inferences":["A natural extension of the paper's finding is that an explicit belief-tracking module—for example an online inverse-RL head that maintains a posterior over the leader's goal—could close the gap, since the leader's trajectory provably contains the relevant information (the copy-the-leader baseline succeeds with it).","Because the paper's own switch-probability ablation shows difficulty persists even when the goal never switches, the IPPO failure is likely driven more by sparse rewards and the hidden-goal credit-assignment problem than by non-stationarity itself; a testable prediction is that varying block sparsity alone will reproduce the failure.","The paper's cost-of-exploration results suggest a transferable design principle for human-AI assistance: when wrong actions are costly, assistants trained with independent PPO become harmlessly inactive, so shaping or an explicit inference signal is needed to make them useful rather than merely safe."],"forward_implications":["If the paper's claim is right, state-of-the-art independent PPO cannot learn to track a changing hidden goal from a partner's trajectory, so MARL benchmarks for human-AI assistance must include non-stationary and asymmetric objectives to expose this gap.","The cost of exploration becomes a first-order design lever: a pessimistic reward setting makes IPPO followers collapse to inaction, which mirrors high-stakes human-robot settings where conservative assistants may be harmless but unhelpful.","Penalty annealing and a supervised goal-prediction auxiliary loss are necessary for IPPO to learn even the symmetric version of ColorGrid with sparse rewards, so these ingredients should be part of any fair comparison on this benchmark.","The A* copying baseline provides an upper reference point for the inference problem: the information needed to act correctly is present in the leader's trajectory, so the failure is one of learning, not of observability."],"supporting_citations":[{"why":"Supplies the PPO algorithm whose independent-agent variant (IPPO) is the paper's main test subject.","marker":"[25]"},{"why":"Provides the result that independent learning can solve cooperative MARL tasks, the claim the paper's negative finding qualifies.","marker":"[4]"},{"why":"Documents strong PPO performance in cooperative multi-agent games, justifying IPPO as the benchmark algorithm.","marker":"[36]"},{"why":"Supplies the base network architecture and the auxiliary goal-prediction loss, and the closest prior environment with a hidden goal order.","marker":"[20]"},{"why":"Defines the assistive watch-and-help setting that ColorGrid extends by making the goal non-stationary within an episode.","marker":"[23]"},{"why":"Introduces a hidden-goal grid environment whose goals stay fixed, marking the novelty of ColorGrid's changing goal.","marker":"[24]"},{"why":"Provides the standard fully-observable cooperative coordination benchmark that motivates ColorGrid's non-stationary twist.","marker":"[3]"}],"fun_headline_variants":["IPPO fails at goal inference in ColorGrid's non-stationary tests","Simple copy strategy beats IPPO in ColorGrid","ColorGrid: IPPO can't track a shifting partner's goal","New multi-agent benchmark shows IPPO falls short on hidden goals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that IPPO cannot solve ColorGrid rests on a few training runs: the headline comparisons in Figure 3 use a single seed, and Table 2 averages only three seeds with one fixed hyperparameter set; if those runs are unrepresentative, the unsolved claim could fail.","fun_headline_variants_meta":{"raw":{"variants":["IPPO fails at goal inference in ColorGrid's non-stationary tests","Simple copy strategy beats IPPO in ColorGrid","ColorGrid: IPPO can't track a shifting partner's goal","New multi-agent benchmark shows IPPO falls short on hidden goals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000362,"raw_usage":{"total_tokens":1947,"prompt_tokens":930,"completion_tokens":1017,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":945}},"tokens_in":546,"tokens_out":1017,"duration_ms":8966,"temperature":1.0,"reasoning_tokens":945,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:02:31.893913+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an IPPO follower in the asymmetric, neutral-expected-reward ColorGrid setting across at least ten seeds using the paper's reported hyperparameters, and compare mean final reward to the A* copying follower baseline averaged over 100 seeds; if any seed or the mean reaches or exceeds the A* follower's reward, the claim that ColorGrid is unsolved by IPPO is falsified.","supporting_citations":[{"cited_title":"The surprising effectiveness of ppo in cooperative, multi-agent games, 2021","cited_arxiv_id":null,"evidence_quote":"Documents strong PPO performance in cooperative multi-agent games, justifying IPPO as the benchmark algorithm."},{"cited_title":"Emergent social learning via multi-agent reinforcement learning, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the base network architecture and the auxiliary goal-prediction loss, and the closest prior environment with a hidden goal order."},{"cited_title":"Tenenbaum, Sanja Fidler, and Antonio Torralba","cited_arxiv_id":null,"evidence_quote":"Defines the assistive watch-and-help setting that ColorGrid extends by making the goal non-stationary within an episode."},{"cited_title":"Modeling others using oneself in multi-agent reinforcement learning, 2018","cited_arxiv_id":null,"evidence_quote":"Introduces a hidden-goal grid environment whose goals stay fixed, marking the novelty of ColorGrid's changing goal."}],"review_version":1}