{"id":"e01b5461-fd8c-4cea-ab7e-e2fa6334027a","arxiv_id":"2501.04180","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HIVEX is an open-source benchmark with five Unity-based multi-agent environments for ecological tasks, plus trained PPO baselines and a Hugging Face leaderboard.","lead":"This paper introduces HIVEX, a suite of five simulated 3D environments for multi-agent reinforcement learning, aimed at wind farms, wildfires, reforestation, and ocean plastic cleanup. It includes procedural terrain, subtasks, PPO baselines, and a public leaderboard, offering a common testbed for MARL research on ecological challenges.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-shot generalization claim rests on a single fixed test seed (6000) versus training seed (5000), with no leakage or multi-seed analysis; this is the weakest load-bearing assumption.","rationale":"The central claim is that HIVEX is a benchmark for multi-agent research; a benchmark's value depends on its evaluation protocol. If test scenarios are not meaningfully held out, baseline numbers are not trustworthy evidence of generalization. The reader flagged the same assumption; I agree. The paper does include detailed specifications, trained PPO baselines, hosted models, and candid limitations, so the issue is not fabrication or internal inconsistency in the environment designs. However, Appendix A.3 reveals only two seeds are configured, making the generalization claim weak and the reported single-seed test results insufficient to support the paper's headline evaluation. The proposed multi-seed check would settle whether the concern lands. The verdict remains conditional: if the multi-seed check passes, the suite is usable for benchmarking; if it fails, the benchmark's central generalization evaluation is unsupported.","tokens_in":65396,"tokens_out":5368,"duration_ms":57586,"concrete_test":"Re-run the published protocol for one environment (e.g., Drone-Based Reforestation, terrain level 5, task 0) training PPO on seeds 5000-5019 and evaluating on the disjoint held-out seeds 6000-6019, reporting mean and standard deviation of test cumulative reward per seed; also compute structural distance (e.g., terrain heightmap correlation) between train and test seeds. If held-out seed rewards drop below the training-seed reward by more than the run-to-run variance, or if any test seed's layout is a near-duplicate of a training seed, the single-seed evaluation in the paper does not establish zero-shot generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central evaluation claim (Section 1) is that procedural generation differentiates training and test scenarios by seed, so test rewards measure zero-shot generalization. However, the hyperparameters in Appendix A.3 fix env_settings.seed to 5000 for training and 6000 for testing for every environment and condition, i.e., a single training seed and a single test seed. The reported \"average and error margin\" over three runs therefore characterizes run-to-run noise on one test seed, not performance across the procedural distribution. If the Unity RNG produces layouts that share structure across nearby seeds, or if the categorical terrain_level/pattern parameters dominate the layout regardless of seed, the held-out seed may still be structurally similar to training conditions. The paper provides no leakage test (e.g., comparing train-seed vs held-out-seed rewards, or structural distance between heightmaps/trash maps) and no per-seed variance. Without this, the baseline test rewards cannot support the claim that HIVEX evaluates open-ended, zero-shot generalization rather than memorization of a specific seeded configuration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"HIVEX is presented as an open-source MARL benchmark suite of five Unity-based environments addressing ecological challenges: Wind Farm Control, Wildfire Resource Management, Drone-Based Reforestation, Ocean Plastic Collection, and Aerial Wildfire Suppression. Each environment has a main task plus subtasks, procedural layout or terrain patterns, PPO baselines, and hosted models with a Hugging Face leaderboard. The paper claims that seed-differentiated procedural generation yields zero-shot test scenarios and that HIVEX simulates real-world critical ecological challenges.","tokens_in":65612,"tokens_out":5096,"duration_ms":53997,"significance":"If the suite is usable and the evaluation claims hold, HIVEX would fill a genuine gap: most existing MARL benchmarks are grid-based or abstract, and few target ecological applications. The paper's strengths include detailed environment and reward specifications, explicit hyperparameters for every environment and task, scalability experiments across agent counts, hosted models, a community leaderboard, and a candid limitations section. However, the empirical support for zero-shot generalization is thin, and the artifact links are not pinned to specific versions, so the benchmark's value depends on the requested revisions.","major_comments":[{"comment":"The central \"zero-shot test scenarios\" claim is not supported by the evaluation protocol. Every training configuration in Appendix A.3 fixes env_settings.seed to 5000 and testing seed to 6000, and Section 6 reports averages over three runs. Consequently, the reported test rewards characterize one held-out seed per condition, not the procedural distribution, and the run-to-run average cannot separate policy variance from seed variance. Please evaluate over multiple test seeds, report per-seed results with error bars, and include a leakage or structural-similarity check between training and test seeds (for example, the train-seed versus held-out-seed reward gap, or a distance measure between generated heightmaps, wind fields, or trash maps). Without such evidence, the claims of open-ended, zero-shot generalization are unsubstantiated.","section":"Section 1; Appendix A.3"},{"comment":"The central artifact is not verifiable because all URLs are organization-level links without commit hashes, release tags, or file paths: github.com/hivex-research, huggingface.co/hivex-research, and the hivex-leaderboard space. Section 6 also refers to a \"hivex-results repository\" without providing a link. For a benchmark paper whose main contribution is a usable environment suite, code, environment binaries, trained checkpoints, and leaderboard configuration must be pinned to specific versions (for example, commit hashes or DOIs) so that results can be reproduced and the leaderboard entries can be audited.","section":"Section 6; Abstract; Appendix A.1"},{"comment":"Several reward definitions contradict their own descriptions, which affects the semantics of the tasks. In A.5.3, \"Close to Other Vessel\" is described as a positive reward of 1 but Equation 18 gives 10, and \"Collide with Trash\" is described as -1 but Equation 23 gives -100. In A.5.4, Equations 47-48 normalize the closest-tree distance by the environment half-extend (200) and then reward 100 when the normalized value is at most 20, which corresponds to a physical distance of 4000 meters, contradicting the stated \"within a 20 meter radius\" condition. In A.5.5, Equation 53 applies the time-step burning penalty only when all trees are burning, whereas the description states the penalty applies while the fire is burning. These inconsistencies should be corrected or explicitly reconciled.","section":"Appendix A.5"},{"comment":"The paper states that all environments were trained and tested three times and that the average and error margin are reported, but neither Figure 14 nor the Appendix A.7 tables show error margins or run-level values. The only visible variation is between adjacent task/terrain cells, not between repeated runs. Please either report the per-run values and error bars or revise the text to state that only averages are shown.","section":"Section 6; Figure 14; Appendix A.7"}],"minor_comments":[{"comment":"The sentence beginning \"Straight forward applications However, we think...\" is incomplete and should be rephrased.","section":"Section 3.1"},{"comment":"The limitations section candidly acknowledges that the simulations are simplified and that wind, fire, and vessel dynamics differ substantially from reality, yet the conclusion calls HIVEX a benchmark that \"simulates real-world critical ecological challenges.\" Consider softening the conclusion to \"simplified models inspired by\" or \"abstractions of\" these challenges.","section":"Section 8; Section 9"},{"comment":"The Aerial Wildfire Suppression configuration omits the testing comments for learning_rate and learning_rate_schedule that appear in the other environment configs; please clarify whether the test runs use a zero learning rate and a constant schedule.","section":"Appendix A.3.6"},{"comment":"The energy-depletion penalty is described as \"-1/(episode * length/2)\" and \"-1/episode * length,\" which is ambiguous; the equations in A.5.4 suggest -1/1000 and -1/2000, so the text should be clarified.","section":"Section 4.4.2"},{"comment":"There is an unresolved cross-reference \"(Appendix: Learning Algorithm ??)\" in Section 6 that should be fixed.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The main gap is not a fatal mathematical error but an evaluation-protocol gap: single-seed test evaluation cannot support the zero-shot generalization claim, and the unpinned artifact links prevent independent verification. Both are fixable within the manuscript's scope. If the authors add multi-seed evaluation, leakage checks, and version-pinned artifacts, the paper could become a useful benchmark contribution. I would not recommend acceptance in the current form. The paper also does not compare PPO against any other MARL algorithm; for a leaderboard paper this may be acceptable, but an editor may want at least one independent baseline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The package itself is the contribution: five Unity-based multi-agent environments (wind farm control, wildfire resource management, drone reforestation, ocean plastic, aerial suppression), each with a main task, subtasks, patterns or terrain levels, PPO baselines, hosted models, and a leaderboard. That combination is genuinely not in the cited literature. The environment tables, reward equations, task lists, and hyperparameters are unusually detailed and mostly internally consistent. The appendix shows real learning curves and non-trivial test rewards, and the authors' Limitations section is candid about sim-to-real gaps: turbines turn too fast, water planes turn unrealistically, camera resolution is low. That honesty earns credit.\n\nWhere it is soft: the paper's central evaluation claim is that procedural generation differentiates training and testing by seed, so test rewards measure zero-shot generalization. Looking at Appendix A.3, training uses seed 5000 and testing uses seed 6000 for every environment and condition. Three runs at one test seed measure run-to-run noise, not performance across the procedural distribution. There is no multi-seed evaluation, no leakage check, and no structural comparison of train versus test layouts. This does not sink the benchmark release, but it does mean the \"zero-shot generalization\" wording in Section 1 and the conclusion is not supported by the reported results. The authors should either add a proper multi-seed evaluation or soften the claim.\n\nSecond, the artifact is not checkable from the paper: code, models, and leaderboard live at organization-level URLs with no commit hashes, so a referee cannot confirm the environments match the specifications. The paper also asserts \"no open-source benchmark ... closely mimics real-world ecological challenges\" without a systematic search; that is an overstatement, especially given Melting Pot and other suites the authors themselves cite. There are minor internal mismatches too, e.g., the DBR visual observation is 16x16 in Table 5 but the process diagram says 32x32, and some appendix equation references are off. These are minor and correctable.\n\nWho is this for: MARL researchers who want ecologically themed cooperative benchmark tasks, especially people interested in Unity/ML-Agents environments and PPO baselines. It is not a paper with a new algorithm or a hard theoretical result. As a benchmark release it deserves serious review, conditional on the repository being live and inspectable. My recommendation: send it to peer review, but the reviewers should require multi-seed test evaluation, pinned releases with hashes, and a toned-down generalization claim before acceptance.","headline":"A well-specified ecological MARL benchmark release with honest limitations; the main caveat is that the zero-shot generalization claim rests on one training seed and one test seed, and the code is not verifiable from the manuscript.","tokens_in":66107,"tokens_out":1236,"would_cite":false,"duration_ms":15975,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HIVEX is a new open benchmark for multi-agent AI on ecological crises.","keywords":["multi-agent reinforcement learning","benchmark suite","ecological challenges","procedural generation","zero-shot generalization","PPO baselines","wildfire management","ocean plastic collection"],"falsifier":"Train on the published training seeds, then evaluate the same checkpoint on both the published test seeds and a replay of the training seeds; statistically indistinguishable returns would mean the seed split does not create a generalization barrier. A supporting check is to try to predict the seed or terrain level from the agent's observations, since a successful predictor would reveal structural leakage between splits.","tokens_in":65194,"feed_emoji":"🌍","tokens_out":8959,"duration_ms":85161,"temperature":0.7,"pith_summary":"The paper presents HIVEX, a new open-source benchmark suite for multi-agent reinforcement learning aimed at ecological problems that are currently underrepresented in MARL benchmarks. Its central claim is that no existing open environment closely mimics real-world critical ecological challenges with the combination of subtasks, procedural terrain and layout generation, visual richness, and zero-shot test evaluation that HIVEX provides. To that end it releases five environments—Wind Farm Control, Wildfire Resource Management, Drone-Based Reforestation, Ocean Plastic Collection, and Aerial Wildfire Suppression—each with a main task and isolated subtasks, trained PPO baselines, and community-facing resources such as hosted models and a leaderboard. A sympathetic reader would take the suite's value to be a shared testbed where progress on climate-adjacent problems is measured on scenarios generated from unseen seeds rather than memorized fixed levels.","feed_headline":"New benchmark pits multi-agent AI against five ecological crises","feed_subtitle":"Each world generates unseen test scenarios, so progress is scored by generalization, not memorization.","key_machinery":"The central object is the HIVEX suite itself: five 3D multi-agent environments built on a common training pipeline and released as a public benchmark. The mechanism that carries the zero-shot generalization claim is seed-based procedural generation—layouts, terrain elevations, wind and fire fields, and object placements are drawn randomly, with training and evaluation differentiated by seed so that test scenarios are claimed to be unseen. The learning machinery is Proximal Policy Optimization (PPO), an on-policy algorithm that constrains how far each update can move the policy, here used with parameter sharing so a single policy controls all agents in an environment. Reward decomposition into per-agent, neighborhood, and collective terms is what defines the behaviors each environment is meant to elicit.","core_discovery":"Summarized on the paper's own terms, the discovery is that a single open suite can stage five high-impact ecological scenarios as multi-agent reinforcement learning problems while keeping the evaluation close to real-world conditions. The load-bearing design is procedural generation: environment features such as terrain elevation, noise fields, trash distributions, and fire behavior are sampled from random distributions, and training and testing are separated by seed values, so test results are meant to reflect zero-shot generalization to scenarios not seen during training. Each environment combines vector and visual observations, discrete and continuous actions, and individual, neighborhood, or collective reward terms, with between two and nine tasks per environment. The reported PPO baselines show that the environments are learnable and that reward tends to decline as terrain elevation or task complexity increases, which the paper reads as evidence that the suite introduces genuine difficulty variation rather than a single solvable level.","pith_inferences":["As an editorial extension, the seed split can be turned into a direct memorization probe: train on the supplied training seeds, then replay those same training seeds at evaluation time; if test-seed rewards and training-seed rewards are statistically indistinguishable, the benchmark is measuring memorization, not generalization.","The paper does not propose it, but the decoupled subtask rewards would let researchers study transfer and curriculum learning across the five environments, asking whether skills learned in one scenario accelerate learning in another.","Because the reported baselines decline on open-field and dense-forest terrain, an adversarial-seed stress test that starts fire fronts or pollution patches near the village or station would likely expose failure modes invisible in average test rewards.","The same procedural-generation machinery could be inverted to generate difficulty curricula ordered by terrain elevation, although the paper leaves that direction unstated."],"forward_implications":["New MARL algorithms can be compared on the same five tasks with hosted baselines and a community leaderboard, replacing self-chosen environments with a shared ecological-problem testbed.","Zero-shot test scores under the seed split become a measurable indicator of whether an agent generalizes to new wind fields, fire fronts, terrains, and trash distributions.","The subtask-and-reward decomposition allows researchers to isolate skills such as resource allocation, energy management, or village protection before combining them in each main task.","Scalability results suggest these environments can support studies of how performance changes with agent count for wind farm, reforestation, and wildfire suppression settings.","By reducing abstraction and adding visual richness, the suite may draw a broader range of disciplines into MARL work on ecological problems."],"supporting_citations":[{"why":"Supplies the PPO algorithm used for every baseline training and test run in the paper.","marker":"Schulman et al. (2017)"},{"why":"Supplies the agent-training toolkit and environment platform on which the HIVEX environments are built.","marker":"Juliani et al. (2020)"},{"why":"Its multi-substrate, many-scenario benchmark structure influenced the design of HIVEX's tasks and scenarios.","marker":"Leibo et al. (2021)"},{"why":"Survey of MARL applications used to argue that no existing benchmark targets critical ecological challenges.","marker":"Ning & Xie (2024)"},{"why":"Data on fire-driven forest loss used to justify wildfires as a priority scenario in the suite.","marker":"Tyukavina et al. (2022)"},{"why":"Catalogs machine-learning applications to climate change, motivating the suite's focus on ecological problems.","marker":"Rolnick et al. (2022)"}],"fun_headline_variants":["Multi-agent AI faces five ecological crises in new open benchmark","Procedural environments push multi-agent AI to generalize on ecology","Five real-world eco challenges now benchmark multi-agent AI","HIVEX suite: multi-agent RL benchmark tackles real ecological problems","New benchmark tests multi-agent AI generalization on ecological crises"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that seed-based procedural generation yields test scenarios the policy has not seen during training, so reported zero-shot test rewards measure generalization rather than memorization.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent AI faces five ecological crises in new open benchmark","Procedural environments push multi-agent AI to generalize on ecology","Five real-world eco challenges now benchmark multi-agent AI","HIVEX suite: multi-agent RL benchmark tackles real ecological problems","New benchmark tests multi-agent AI generalization on ecological crises"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1252,"prompt_tokens":934,"completion_tokens":318,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":237}},"tokens_in":550,"tokens_out":318,"duration_ms":3574,"temperature":1.0,"reasoning_tokens":237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:39:32.772783+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train on the published training seeds, then evaluate the same checkpoint on both the published test seeds and a replay of the training seeds; statistically indistinguishable returns would mean the seed split does not create a generalization barrier. A supporting check is to try to predict the seed or terrain level from the agent's observations, since a successful predictor would reveal structural leakage between splits.","supporting_citations":[],"review_version":1}