{"id":"99d1ec5a-5961-4e7f-9331-219fbd631b9c","arxiv_id":"2412.15388","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MARC, a relational critic using spatial relations without distances, improves sample efficiency and asymptotic performance in collaborative MARL tasks over several baselines, though not on all continuous tasks.","lead":"This paper introduces MARC, a multi-agent reinforcement learning critic that turns each agent's observation into a graph of spatial relations (left, right, top, bottom, adjacent, aligned) and processes it with a graph neural network. Across six collaborative tasks it reports faster learning and better final scores than several standard MARL algorithms, especially in grid-world tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The relational abstraction removes distance information, so near and far configurations map to the same critic input; Section 3.2 never proves this abstraction is reward/transition-preserving, and Section 4.5's continuous failure shows the cost.","rationale":"The reader's weakest assumption identified the removal of distances and velocities as load-bearing, and Section 4.5's own admission of 1–2 extra steps and collisions supports that. My stress-test sharpens this into an internal consistency concern: the described abstraction can make near and far states literally indistinguishable in the critic, which would undermine even the discrete-domain results rather than merely limiting continuous performance. This is more severe than 'the abstraction may discard some necessary information' because it predicts a specific failure mode—no value distinction between approaching and retreating while the relation set is unchanged—that should have made the discrete learning curves much worse than reported. The paper does not provide a formal abstraction-soundness argument; the appendix only demonstrates translation invariance of individual predicates, not that the pooled representation is a sufficient statistic. However, I do not reject the paper because the empirical results may still be valid if the implementation includes more information than the text states (e.g., entity counts, identity features, or implicit distance through adjacency thresholds), and the authors have released code. The appropriate response is a conditional acceptance asking for the minimal diagnostic and a precise statement of what the encoder actually consumes. The generalization comparison and continuous overclaim noted by the reader are real but secondary; the distance-blindness concern is the most load-bearing because it attacks the central mechanism itself.","tokens_in":19845,"tokens_out":8903,"duration_ms":92140,"concrete_test":"Run a minimal diagnostic: a single agent, a single fruit, a 1D grid, sparse reward, with the exact MARC observation encoder. Log the critic input e(o_t) for states with fruit at distances 2 through 10 while keeping all qualitative predicates fixed (agent at x=1, fruit at x=3..10). If the encoder is distance-free as described, e(o_t) must be identical across these distances. Then, after training, check whether the learned Q-values for 'move right' and 'move left' differ at those states. If they are identical, the abstraction cannot support the claimed navigation behavior and the discrete results are unexplained. If e(o_t) differs, inspect the code to identify the leaked distance/velocity feature and revise Section 3.2 accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a relational state abstraction built from qualitative spatial relations improves sample efficiency without discarding decision-relevant information. Section 3.2 makes the abstraction precise: entity features are non-spatial, edges are Boolean predicates (left/right/top/bottom/adjacent/aligned) with no distance attributes, and the critic input is a max-pooled R-GCN embedding. In a deterministic grid, moving one cell toward a fruit changes no predicate until the agent crosses the fruit's row/column or becomes adjacent. In a minimal two-entity world, the abstract observation for 'agent at (1,1), fruit at (5,1)' is identical to 'agent at (2,1), fruit at (5,1)' even though the optimal action differs by distance. Because max-pooling also discards which entity is where and how many neighbors satisfy each relation, the abstract state is not a sufficient statistic for returns: the same abstract state can transition to the same abstract state under 'move right' and 'move left', so bootstrapped Q-values for the two actions are indistinguishable in the far regime. The appendix proves only that the predicate left(a,b) is translation-invariant; it does not prove the pooled representation preserves the value function or transition structure. Section 4.5 confirms the abstraction has a real cost in continuous domains (1–2 extra steps and collisions due to missing proximity/velocity), but the same distance-blindness should also degrade discrete navigation. The strong discrete results therefore require an explanation: either the implementation leaks distance or velocity through entity features or edge construction, or the reported gains are not attributable to the abstraction as described.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MARC (Multi-Agent Relational Critic), an actor-critic MARL method whose critic encodes observations as a graph over agents and environment objects. The graph edges are Boolean spatial predicates (left/right/top/bottom/adjacent/aligned) and the node features are non-spatial attributes; the representation is passed through a shared R-GCN and max-pooled before each agent's value head. The authors evaluate MARC on four discrete collaborative tasks (a new Collaborative Pick and Place environment, two Level-Based Foraging variants, and Wolfpack) and one continuous task (Target), comparing against MAAC, GA-AC, QMIX, MAA2C, MAPPO, and InforMARL. They report improved sample efficiency and asymptotic performance on the discrete tasks, competitive performance on the continuous task, generalization results on Level-Based Foraging, and ablations over relation sets, graph architectures, and entity granularity. They also show that the relational encoder can be combined with a different backbone (MAPPO).","tokens_in":20195,"tokens_out":3453,"duration_ms":33820,"significance":"If the reported discrete-task results are robust, the paper makes a useful empirical contribution: it shows that a cheap relational abstraction, implemented purely in the critic, can improve sample efficiency and asymptotic performance in cooperative MARL tasks with spatial structure. The strengths of the paper include public code for MARC and the new CPP environment, a reasonably broad set of baselines, explicit ablations on the relation set and graph architecture, and a supporting experiment that combines the encoder with MAPPO. The significance is tempered, however, because the central claim is stated more broadly than the evidence supports: the continuous-domain results do not show an asymptotic advantage, and the abstraction is not shown to preserve decision-relevant information in general.","major_comments":[{"comment":"The claim in Section 4.3 that 'asymptotically, MARC is competitive and outperforms all baselines across the implemented tasks' is contradicted by the continuous Target results in Table 1. For 3 agents, MARC (212.7 ± 5.7) trails MAAC (236.1 ± 2.9), GA-AC (236.6 ± 3.5), and MAA2C (233.5 ± 2.1); for 7 agents, MARC (468.2 ± 4.2) trails MAAC (527.9 ± 5.4) and GA-AC (530.6 ± 3.4). The abstract and Section 4.3 need to restrict the 'outperforms all baselines' claim to the discrete tasks, or provide a substantive argument for why the continuous shortfall is acceptable for the paper's central claim.","section":"Section 4.3, Table 1, Figure 2"},{"comment":"The paper asserts that removing absolute positions and distances 'does not sacrifice essential information for decision-making' (Section 3.2), but the appendix only proves translation invariance of individual spatial predicates, not that the pooled abstract state preserves the optimal value function or transition structure. In the Boolean-edge representation with max-pooling, configurations with different distances to a target can map to the same abstract state even when the optimal action differs, so the abstraction is not shown to be a sufficient statistic for returns. Section 4.5 itself reports that in the continuous domain MARC takes 1-2 extra steps and suffers collisions because it lacks fine-grained proximity and velocity information. The authors should either provide a value-preservation argument for the discrete settings or explicitly qualify the 'without sacrificing essential information' claim as an empirical design assumption whose failure modes are documented in the continuous case.","section":"Section 3.2 and Appendix 'Invariances of the State Abstraction'"},{"comment":"The continuous-domain results are hard to interpret because Table 1 reports a single 'MARC' entry, while Figure 7 in the appendix plots two variants, 'MARC (5 relations)' and 'MARC (9 relations)', and the technical appendix says the finer-grained octagonal relations improve asymptotic performance. The main text should specify which MARC variant is used in Table 1 and explain whether the 5-relation or 9-relation results correspond to the reported numbers. This is necessary for reproducibility and for assessing the paper's claim that the default abstraction is competitive in continuous domains.","section":"Section 4.5 and Table 1 vs. Figure 7"}],"minor_comments":[{"comment":"The generalization evaluation compares MARC only against MAPPO and only on a single source configuration (LBF-10x10-4a-4f-coop). Reporting error bars or multiple seeds for the generalization numbers, and ideally comparing against one additional baseline, would make the claim of 'potential for generalization' stronger. As written, the evidence is suggestive but thin.","section":"Section 4.4"},{"comment":"The hyperparameter search for MARC, MAAC, and GA-AC is performed on LBF-10x10-4a-4f-coop and then applied across all tasks; the text should explicitly acknowledge that this could favor the selected method on that task and that per-task tuning might change the relative results in other environments.","section":"Technical Appendix, Table 2"},{"comment":"The MAA2C result for 7 agents in Table 1 has a standard deviation of 393.4, which is several times larger than the mean; this suggests highly unstable runs and should be discussed or flagged in the table caption, since it materially affects the comparison in the 7-agent setting.","section":"Section 4.5"},{"comment":"There are minor formatting and typographical issues: axis labels in Figure 2 show '0.00 0.25 0.50...' instead of a proper tick format, the reference for Abdel-Aziz et al. has a misplaced 'In', and some numerical values are given as '1e6' rather than '10^6'. These should be cleaned up in the final version.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical architecture paper with a useful negative-positive mix: it shows a relational critic helps on discrete spatial tasks, but the authors overclaim in the abstract and Section 4.3. The main fixes needed are to qualify the central claim to discrete domains, clarify the relation-set ambiguity in the continuous experiments, and temper the 'without sacrificing essential information' assertion, which is contradicted by their own continuous-domain analysis. The novelty is moderate—the R-GCN critic is a fairly direct adaptation of existing relational inductive biases to MARL—but the new CPP environment and the ablation study provide enough empirical content for a workshop-quality contribution; with the claims tightened, it could be acceptable for the main track."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: MARC is a legitimate empirical contribution. The specific recipe — object-level graph with six qualitative spatial relations, no distances, a shared R-GCN encoder, max-pooling, per-agent critic heads — is not in the cited literature, and the paper ships code and a new heterogeneous CPP environment. The discrete experiments are the strong part: on LBF variants, CPP, and Wolfpack, MARC learns faster and reaches higher asymptotic reward than MAAC, GA-AC, MAA2C, QMIX, and MAPPO. The ablations are useful too: local relations alone fail, full-grid entities hurt sample efficiency, and R-GAT adds nothing over R-GCN. That is a coherent story.\n\nThe soft spots are real but manageable. First, the abstract claims 'improvements in both sample efficiency and asymptotic performance' across the board, and the continuous Target task does not support that. MARC trails MAAC, GA-AC, and MAA2C by about 10% final reward in the 3-agent case, and the paper's own explanation — agents lack fine-grained proximity and velocity — is an honest admission that the abstraction discards decision-relevant information in continuous domains. The authors should either restrict the headline claim to discrete spatial tasks or present the continuous results as competitive-but-not-superior. Second, the stress-test point about distance-blindness is theoretically valid: near and far configurations can produce identical abstract states, so the critic cannot represent the true value function in general. The paper never proves the abstraction is reward/transition-preserving, and it doesn't need to for an empirical claim, but it should acknowledge that the discrete gains may rely on tasks where direction-to-target is more informative than distance-to-target. The continuous failure makes that caveat necessary. Third, minor issues: text says alpha=0.05 while Table 2 says 0.01; the generalization comparison with MAPPO is not fully specified (retrained per scenario or just evaluated?); and the appendix typo 'INFORMAL' for InforMARL should be fixed.\n\nOverall, the paper deserves a serious referee. The empirical evidence for the discrete domain is solid, the recipe is new and simple, and the ablations are informative. I would ask for revision to align the abstract with the actual continuous results, clarify the generalization protocol, and fix the alpha inconsistency. It is not a desk reject.","headline":"Solid empirical recipe with a real inductive bias, but the abstract overclaims the continuous results and the distance-free abstraction has a known cost that should be stated up front.","tokens_in":20748,"tokens_out":3301,"would_cite":true,"duration_ms":30311,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A critic that represents the state as a graph of relative spatial relations between agents and objects—discarding absolute positions, distances, and velocities—learns collaborative multi-agent tasks faster and to competitive final…","keywords":["multi-agent reinforcement learning","relational state abstraction","spatial inductive bias","graph neural networks","relational graph convolutional networks","sample efficiency","collaborative tasks","heterogeneous agents"],"falsifier":"In the continuous Target task, augment MARC's graph with distance and velocity edge features and compare final reward to the published lean version; if the augmented critic clearly outperforms it, the abstraction's core assumption is violated. The paper's own report that MARC takes 1-2 extra steps and collides more often in that domain is a concrete symptom to look for.","tokens_in":19656,"feed_emoji":"🤖","tokens_out":6181,"duration_ms":50781,"temperature":0.7,"pith_summary":"This paper asks whether a deliberately lossy representation—keeping only the relative spatial arrangement of agents and objects and throwing away absolute coordinates, distances, and velocities—can make multi-agent reinforcement learning more sample-efficient without hurting final performance. The answer, according to the authors, is yes: their MARC architecture transforms each observation into a spatial graph and processes it with a shared relational graph network in the critic, and on four discrete collaborative tasks it learns as fast or faster than six baselines, with up to a 7.3x step advantage over the second-best method. On a novel heterogeneous pick-and-place task it ends with 69.9% higher reward, and on a continuous navigation domain it beats the graph-based InforMARL baseline while staying competitive with attention-based methods. The paper also reports improved generalization to changed numbers of agents and objects, and an ablation shows the specific choice of coarse spatial relations matters: local-only relations and full-grid graphs both underperform.","feed_headline":"Spatial graph critic beats MARL baselines with 7x fewer steps","feed_subtitle":"MARC keeps only relative positions of agents and objects, drops distances and velocities, and still learns collaborative tasks fastest.","key_machinery":"The central object is the relational state abstraction: an observation is turned into a spatial graph whose nodes are agents and environment objects, and whose directed edges are six spatial predicates (left, right, top, bottom, adjacent, aligned) evaluated from relative coordinates. Absolute positions, distances, and velocities are deliberately dropped, which makes the representation translation-invariant and invariant to entity ordering. This graph is passed through a shared R-GCN (Relational Graph Convolutional Network) layer with per-relation weight matrices, followed by channel-wise max-pooling to produce a fixed-size observation encoding; each agent then has its own small MLP head that combines this encoding with the joint action to output a Q-value in a soft actor-critic loop.","core_discovery":"The paper claims that replacing an agent's raw observation with a graph that keeps only relative spatial relationships between entities—left/right/top/bottom, adjacency, and alignment—and processing that graph through a shared relational graph neural network in the critic makes multi-agent reinforcement learning dramatically more sample-efficient while matching or beating state-of-the-art baselines on final performance. On the discrete tasks, MARC reaches the same performance as the best baseline in as little as 1/7.3 of the environment steps, and on a new heterogeneous pick-and-place environment it ends with 69.9% higher reward. The same principle carries over to a continuous navigation task, where MARC beats the graph-based InforMARL baseline but trails the attention-based MAAC and GA-AC slightly, with the paper attributing the small gap to the loss of distance and velocity information.","pith_inferences":["A natural next step, unstated in the paper, is to learn the relation set itself rather than fixing six predicates; the paper's own ablation shows that both too-coarse and too-fine relation sets hurt, suggesting there is an optimal granularity that could be discovered per task.","Because the shared R-GCN is the only cross-agent information channel, MARC effectively implements implicit coordination through a learned global representation; one could test whether the learned entity embeddings become interpretable role or state codes.","The paper only tests collaborative reward settings; the same abstraction should be probed in mixed or competitive games, where relative spatial structure also matters but the value decomposition may behave differently.","The continuous-domain trade-off (faster learning but 1-2 extra steps and collisions) suggests a hybrid that starts with the lean graph and gradually adds distance edges during training; this is a testable extension the paper does not explore."],"forward_implications":["If MARC's claim holds, then a minimal relational bias—just six spatial predicates, no distances—can replace explicit agent-to-agent communication or attention in collaborative settings, lowering the bar for sample-efficient MARL.","The reported 7.3x step advantage on LBF-15x15 suggests that on large-agent collaborative tasks, relational abstraction can be the difference between feasibility and impracticality.","The abstraction transfers across grid and continuous domains, and the paper's MAPPO+RC appendix result indicates the encoder is backbone-agnostic; other MARL algorithms can bolt it on.","The translation invariance means policies learned in one absolute layout should transfer to the same relative layout elsewhere, which the generalization experiments (varying agent/fruit counts) begin to support.","The fixed sparse-relation graph also cuts the critic's input dimensionality relative to joint-observation critics, which is part of why the method is computationally cheap as well as sample-efficient."],"supporting_citations":[{"why":"Supplies the R-GCN update used as the shared relational encoder in the critic.","marker":"Schlichtkrull et al. 2018"},{"why":"Provides the MAAC baseline and the SAC-style actor-critic training loop that MARC builds on.","marker":"Iqbal and Sha 2019"},{"why":"Provides the InforMARL graph-based baseline and the continuous Target task setup used for comparison.","marker":"Nayak et al. 2023"},{"why":"Supplies the soft actor-critic target updates and entropy-regularized policy loss.","marker":"Haarnoja et al. 2018"},{"why":"Provides the QMIX value-decomposition baseline.","marker":"Rashid et al. 2018"},{"why":"Provides the MAPPO baseline, and is also combined with the relational critic in the appendix's backbone-agnostic experiment.","marker":"Yu et al. 2022"},{"why":"Provides the Level-based Foraging environment, modified by the paper with persistent trees as obstacles.","marker":"Christianos, Schäfer, and Albrecht 2020"},{"why":"Provides the multi-agent particle environment from which the continuous Target task is derived.","marker":"Lowe et al. 2017"}],"fun_headline_variants":["Relational state abstraction cuts MARL steps by 7x","Spatial graph critic boosts MARL sample efficiency 7x","MARC: relational critic achieves 7x fewer steps in tasks","Dropping distances, MARC learns collaborative tasks 7x faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the relative spatial layout of entities, without distances, velocities, or absolute locations, carries enough information for near-optimal decisions in these tasks; if a task truly requires knowing exactly how far or how fast something is, this abstraction deletes the information the policy needs.","fun_headline_variants_meta":{"raw":{"variants":["Relational state abstraction cuts MARL steps by 7x","Spatial graph critic boosts MARL sample efficiency 7x","MARC: relational critic achieves 7x fewer steps in tasks","Dropping distances, MARC learns collaborative tasks 7x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000345,"raw_usage":{"total_tokens":1875,"prompt_tokens":908,"completion_tokens":967,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":901}},"tokens_in":524,"tokens_out":967,"duration_ms":5548,"temperature":1.0,"reasoning_tokens":901,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:27:45.086881+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In the continuous Target task, augment MARC's graph with distance and velocity edge features and compare final reward to the published lean version; if the augmented critic clearly outperforms it, the abstraction's core assumption is violated. The paper's own report that MARC takes 1-2 extra steps and collides more often in that domain is a concrete symptom to look for.","supporting_citations":[{"cited_title":"N.; Bloem, P.; van den Berg, R.; Titov, I.; and Welling, M","cited_arxiv_id":null,"evidence_quote":"Supplies the R-GCN update used as the shared relational encoder in the critic."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MAAC baseline and the SAC-style actor-critic training loop that MARC builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the InforMARL graph-based baseline and the continuous Target task setup used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the soft actor-critic target updates and entropy-regularized policy loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the QMIX value-decomposition baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Level-based Foraging environment, modified by the paper with persistent trees as obstacles."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-agent particle environment from which the continuous Target task is derived."}],"review_version":1}