REVIEW 4 major objections 5 minor 173 references
HIVEX: A High-Impact Environment Suite for Multi-Agent Research (extended version)
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read HIVEX is a new open benchmark for multi-agent AI on ecological crises.
desk verdict A well-specified ecological MARL benchmark release with honest limitations; the main caveat is that the zero-shot generalization claim rests on one training seed and one test seed, and the code is not verifiable from the manuscript. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the HIVEX suite itself: five 3D multi-agent environments built on a common training pipeline and released as a public benchmark. The mechanism that carries the zero-shot generalization claim is seed-based procedural generation—layouts, terrain elevations, wind and fire fields, and object placements are drawn randomly, with training and evaluation differentiated by seed so that test scenarios are claimed to be unseen. The learning machinery is Proximal Policy Optimization (PPO), an on-policy algorithm that constrains how far each update can move the policy, here used with parameter sharing so a single policy controls all agents in an environment. Reward decomposition into per-agent, neighborhood, and collective terms is what defines the behaviors each environment is meant to elicit.
What would settle it
Train on the published training seeds, then evaluate the same checkpoint on both the published test seeds and a replay of the training seeds; statistically indistinguishable returns would mean the seed split does not create a generalization barrier. A supporting check is to try to predict the seed or terrain level from the agent's observations, since a successful predictor would reveal structural leakage between splits.
Extended reading notes
Core claim
Summarized on the paper's own terms, the discovery is that a single open suite can stage five high-impact ecological scenarios as multi-agent reinforcement learning problems while keeping the evaluation close to real-world conditions. The load-bearing design is procedural generation: environment features such as terrain elevation, noise fields, trash distributions, and fire behavior are sampled from random distributions, and training and testing are separated by seed values, so test results are meant to reflect zero-shot generalization to scenarios not seen during training. Each environment combines vector and visual observations, discrete and continuous actions, and individual, neighborhood, or collective reward terms, with between two and nine tasks per environment. The reported PPO baselines show that the environments are learnable and that reward tends to decline as terrain elevation or task complexity increases, which the paper reads as evidence that the suite introduces genuine difficulty variation rather than a single solvable level.
Load-bearing premise
The load-bearing premise is that seed-based procedural generation yields test scenarios the policy has not seen during training, so reported zero-shot test rewards measure generalization rather than memorization.
Editorial extensions
If this is right
- New MARL algorithms can be compared on the same five tasks with hosted baselines and a community leaderboard, replacing self-chosen environments with a shared ecological-problem testbed.
- Zero-shot test scores under the seed split become a measurable indicator of whether an agent generalizes to new wind fields, fire fronts, terrains, and trash distributions.
- The subtask-and-reward decomposition allows researchers to isolate skills such as resource allocation, energy management, or village protection before combining them in each main task.
- Scalability results suggest these environments can support studies of how performance changes with agent count for wind farm, reforestation, and wildfire suppression settings.
- By reducing abstraction and adding visual richness, the suite may draw a broader range of disciplines into MARL work on ecological problems.
Reading between the lines
- As an editorial extension, the seed split can be turned into a direct memorization probe: train on the supplied training seeds, then replay those same training seeds at evaluation time; if test-seed rewards and training-seed rewards are statistically indistinguishable, the benchmark is measuring memorization, not generalization.
- The paper does not propose it, but the decoupled subtask rewards would let researchers study transfer and curriculum learning across the five environments, asking whether skills learned in one scenario accelerate learning in another.
- Because the reported baselines decline on open-field and dense-forest terrain, an adversarial-seed stress test that starts fire fronts or pollution patches near the village or station would likely expose failure modes invisible in average test rewards.
- The same procedural-generation machinery could be inverted to generate difficulty curricula ordered by terrain elevation, although the paper leaves that direction unstated.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. HIVEX is presented as an open-source MARL benchmark suite of five Unity-based environments addressing ecological challenges: Wind Farm Control, Wildfire Resource Management, Drone-Based Reforestation, Ocean Plastic Collection, and Aerial Wildfire Suppression. Each environment has a main task plus subtasks, procedural layout or terrain patterns, PPO baselines, and hosted models with a Hugging Face leaderboard. The paper claims that seed-differentiated procedural generation yields zero-shot test scenarios and that HIVEX simulates real-world critical ecological challenges.
Significance. If the suite is usable and the evaluation claims hold, HIVEX would fill a genuine gap: most existing MARL benchmarks are grid-based or abstract, and few target ecological applications. The paper's strengths include detailed environment and reward specifications, explicit hyperparameters for every environment and task, scalability experiments across agent counts, hosted models, a community leaderboard, and a candid limitations section. However, the empirical support for zero-shot generalization is thin, and the artifact links are not pinned to specific versions, so the benchmark's value depends on the requested revisions.
major comments (4)
- [Section 1; Appendix A.3] The central "zero-shot test scenarios" claim is not supported by the evaluation protocol. Every training configuration in Appendix A.3 fixes env_settings.seed to 5000 and testing seed to 6000, and Section 6 reports averages over three runs. Consequently, the reported test rewards characterize one held-out seed per condition, not the procedural distribution, and the run-to-run average cannot separate policy variance from seed variance. Please evaluate over multiple test seeds, report per-seed results with error bars, and include a leakage or structural-similarity check between training and test seeds (for example, the train-seed versus held-out-seed reward gap, or a distance measure between generated heightmaps, wind fields, or trash maps). Without such evidence, the claims of open-ended, zero-shot generalization are unsubstantiated.
- [Section 6; Abstract; Appendix A.1] The central artifact is not verifiable because all URLs are organization-level links without commit hashes, release tags, or file paths: github.com/hivex-research, huggingface.co/hivex-research, and the hivex-leaderboard space. Section 6 also refers to a "hivex-results repository" without providing a link. For a benchmark paper whose main contribution is a usable environment suite, code, environment binaries, trained checkpoints, and leaderboard configuration must be pinned to specific versions (for example, commit hashes or DOIs) so that results can be reproduced and the leaderboard entries can be audited.
- [Appendix A.5] Several reward definitions contradict their own descriptions, which affects the semantics of the tasks. In A.5.3, "Close to Other Vessel" is described as a positive reward of 1 but Equation 18 gives 10, and "Collide with Trash" is described as -1 but Equation 23 gives -100. In A.5.4, Equations 47-48 normalize the closest-tree distance by the environment half-extend (200) and then reward 100 when the normalized value is at most 20, which corresponds to a physical distance of 4000 meters, contradicting the stated "within a 20 meter radius" condition. In A.5.5, Equation 53 applies the time-step burning penalty only when all trees are burning, whereas the description states the penalty applies while the fire is burning. These inconsistencies should be corrected or explicitly reconciled.
- [Section 6; Figure 14; Appendix A.7] The paper states that all environments were trained and tested three times and that the average and error margin are reported, but neither Figure 14 nor the Appendix A.7 tables show error margins or run-level values. The only visible variation is between adjacent task/terrain cells, not between repeated runs. Please either report the per-run values and error bars or revise the text to state that only averages are shown.
minor comments (5)
- [Section 3.1] The sentence beginning "Straight forward applications However, we think..." is incomplete and should be rephrased.
- [Section 8; Section 9] The limitations section candidly acknowledges that the simulations are simplified and that wind, fire, and vessel dynamics differ substantially from reality, yet the conclusion calls HIVEX a benchmark that "simulates real-world critical ecological challenges." Consider softening the conclusion to "simplified models inspired by" or "abstractions of" these challenges.
- [Appendix A.3.6] The Aerial Wildfire Suppression configuration omits the testing comments for learning_rate and learning_rate_schedule that appear in the other environment configs; please clarify whether the test runs use a zero learning rate and a constant schedule.
- [Section 4.4.2] The energy-depletion penalty is described as "-1/(episode * length/2)" and "-1/episode * length," which is ambiguous; the equations in A.5.4 suggest -1/1000 and -1/2000, so the text should be clarified.
- [Section 6] There is an unresolved cross-reference "(Appendix: Learning Algorithm ??)" in Section 6 that should be fixed.
Circularity Check
No significant circularity: this is a benchmark presentation with no derivation chain whose predictions reduce to fitted inputs; the seed-based zero-shot claim is an evaluation validity issue, not a circularity.
full rationale
This paper does not derive a quantitative prediction from fitted parameters. It presents five Unity-based multi-agent environments, reward functions, PPO baselines, and hosted models. The central claim that procedural generation differentiates training and evaluation scenarios by seed values is an experimental design assumption, not a circular step: the paper does not define the test objective in terms of the training outcome, nor does it fit a parameter to the test data and then rename the fit a prediction. The self-citations in Section 5 ('our previous work has shown significant performance improvements when introducing communication mechanisms in earlier versions of HIVEX environments') establish design lineage but are not load-bearing for the benchmark's independent content, which is the public suite, baselines, and leaderboard. The most serious concern identified by the skeptical reading is that test evaluation uses a single fixed seed (6000) versus a single training seed (5000), with no leakage or multi-seed analysis; that is a threat to the zero-shot generalization interpretation, but it is a correctness or robustness concern, not circularity under the definitions used here. No equation in the paper is equivalent by construction to another, and no result is forced by a self-citation chain. Accordingly, the honest finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- Across-environment reward weights =
e.g., border or obstacle penalty -100, pick-up water +100, fire out +10, per-pebble +1
- PPO hyperparameters =
learning_rate 0.0003, gamma 0.9 to 0.995, lambda 0.95, hidden units 64 to 512
- WRM broken-power-law constants =
dthresh = 200, s = 270, a = 5
- DBR seed-quality constants =
dotmax = 75, dotmin = 2.5, sdrm = 20, drm = 10, dcharge = 7.5
- Episode lengths and agent counts =
episodes 500 to 5000, agents 3 to 9, scalability up to 16 or 12
assumptions (4)
- domain assumption Unity ML-Agents simulations are sufficiently realistic proxies for real ecological tasks; simplified dynamics still yield transferable agent behavior.
- domain assumption PPO with parameter sharing is a suitable baseline to demonstrate learnability and difficulty of the environments.
- domain assumption Seed-based procedural generation divides training and evaluation sets, so test scenarios are unseen.
- domain assumption The reported environments can be reproduced from the linked GitHub and Hugging Face resources.
Cite this review
Pith. "Pith review of HIVEX: A High-Impact Environment Suite for Multi-Agent Research (extended version)." pith.science (2026). https://pith.science/paper/ROE3F632
@misc{pith2026250104180,
author = {Pith},
title = {Pith review of: HIVEX: A High-Impact Environment Suite for Multi-Agent Research (extended version)},
year = {2026},
howpublished = {\url{https://pith.science/paper/ROE3F632}},
note = {Machine review of arXiv:2501.04180}
}
read the original abstract
Games have been vital test beds for the rapid development of Agent-based research. Remarkable progress has been achieved in the past, but it is unclear if the findings equip for real-world problems. While pressure grows, some of the most critical ecological challenges can find mitigation and prevention solutions through technology and its applications. Most real-world domains include multi-agent scenarios and require machine-machine and human-machine collaboration. Open-source environments have not advanced and are often toy scenarios, too abstract or not suitable for multi-agent research. By mimicking real-world problems and increasing the complexity of environments, we hope to advance state-of-the-art multi-agent research and inspire researchers to work on immediate real-world problems. Here, we present HIVEX, an environment suite to benchmark multi-agent research focusing on ecological challenges. HIVEX includes the following environments: Wind Farm Control, Wildfire Resource Management, Drone-Based Reforestation, Ocean Plastic Collection, and Aerial Wildfire Suppression. We provide environments, training examples, and baselines for the main and sub-tasks. All trained models resulting from the experiments of this work are hosted on Hugging Face. We also provide a leaderboard on Hugging Face and encourage the community to submit models trained on our environment suite.
Figures
Figures from the paper (234 more)
Reference graph
Works this paper leans on
-
[1]
This reward corresponds to the performance of each wind turbine and is being calculated as described in equation 4
Generate Energy - This is a positive reward given at each time-step, in the range [0, 1]. This reward corresponds to the performance of each wind turbine and is being calculated as described in equation 4. Orienting the wind turbine against the wind yields a high reward
-
[2]
We remap the angle between the wind direction and the turbine’s orientation linearly from [0, 90] degrees to [0, 1] reward and from [90, 180] degrees to [1, 0] reward
Avoid Damage - This is a positive reward given at each time-step, in the range [0, 1]. We remap the angle between the wind direction and the turbine’s orientation linearly from [0, 90] degrees to [0, 1] reward and from [90, 180] degrees to [1, 0] reward. Orienting the wind turbine so that the rotor blades are parallel to the wind direction yields high rew...
-
[3]
Updating performance P with drag and wind force: P (θ) =P (θ) +d + W (θ) × aturbine (3)
-
[4]
Clamping performance P (θ) between 0 and 1 is the reward R(θ): R(θ) = max(0, min(1, P(θ))) (4) Here, max(0, min(1, P(θ))) limits P (θ) within the interval [0, 1], ensuring it neither falls below 0 nor exceeds 1
-
[5]
The steps to calculate the close to other vessel reward can be found in Equation 18
Close to Other Vessel - This is a positive reward of 1 given at each time step when the distance to the other vessel is smaller than or equal to 10. The steps to calculate the close to other vessel reward can be found in Equation 18
-
[6]
The reward given is the delta between the previous nearby trash field population count and the current
Nearby Trash Count Delta - This is a positive reward given when the nearby trash field population is higher than it has been until this time step. The reward given is the delta between the previous nearby trash field population count and the current. A nearby trash field population count is calculated by finding all floating plastic pebbles around a vesse...
-
[7]
Reward Calculation
Collide with Trash - This is a negative reward of −1 given when the agent-controlled vessel is colliding with a floating plastic pebble. Reward Calculation
-
[8]
The reward is the delta between the old and the new height, if the new height is higher than the old
High Landscape Point Delta - This is a reward given every time a higher point on the terrain landscape has been found. The reward is the delta between the old and the new height, if the new height is higher than the old. The reward range is [0, 40], as 40 is the environment’s height boundary. The steps to calculate the high landscape point delta reward ca...
Show all 173 references
-
[9]
• θ — The angle between the wind turbine orientation and the wind direction at the turbine
Generate Energy - First, we need to describe how performance is calculated for each wind turbine: Let us define: • aturbine = 0.1 — Acceleration of the turbine motor. • θ — The angle between the wind turbine orientation and the wind direction at the turbine. • P (θ) = 0.0 — Pe...
-
[10]
Calculation of wind force W based on angle θ: W (θ) = 0 if θ <0.5 Map(θ, 0.5, 1, 0, 1) if 0.5 ≤ θ ≤ 1 (1) The ”Map” function linearly interpolates the value of force from 0 to 1 as angle increases from 0.5 to 1
-
[11]
Calculation of drag: d = −0.1 × P (θ) (2)
-
[14]
Calculation steps:
Avoid Damage The avoid damage reward R(θ) can be calculated as follows: Let us define: • θ — The angle between the wind turbine orientation and the wind direction at the turbine. Calculation steps:
-
[15]
Calculation of avoid damage reward based on angle θ: R(θ) = θ 90 if 0 ≤ θ ≤ 90 2 − θ 90 if 90 < θ≤ 180 (5) A.5.2 W ILDFIRE RESOURCE MANAGEMENT Reward Description 44 Preprint
-
[16]
This reward is weighted by the resources distributed by self to self
Watch Tower Performance- This is a positive reward given at each time step, correspond- ing to the performance of the agent-controlled watch tower only. This reward is weighted by the resources distributed by self to self. Equation 9 describes how the individual perfor- mance ...
-
[17]
This reward is weighted by the resources distributed by self to neighbouring watch towers
Neighbour Performance - This is a positive reward given at each time step, correspond- ing to the sum of the performance of the neighbouring agent-controlled watch towers. This reward is weighted by the resources distributed by self to neighbouring watch towers. Equa- tion 10 ...
-
[18]
Equation 12 describes how the collective performance and reward are calculated
Collective Performance - This is a positive reward given at each time step, corresponding to the sum of the performance of all agent-controlled watch towers. Equation 12 describes how the collective performance and reward are calculated. Reward Calculation
-
[19]
Let us define: • dthresh = 200— Threshold distance to a fire, used for normalization
Watch Tower Performance - First, we need to calculate the performance of each watch tower agent. Let us define: • dthresh = 200— Threshold distance to a fire, used for normalization. • ⃗ x0 and ⃗ x1 — 3D vector positions of the closest observed fire at timesteps 0 and 1, respe...
-
[20]
The remapped distance factor based on the direction of movement is given by: d′ 1normalized = 0.5 − 0.5 × d1normalized if m 0.5 + 0.5 × d1normalized if not m (6)
-
[21]
The adjusted distance factor using the broken power law is: d′′ 1normalized = 1 + d′ 1normalized × 1000 s a − 1 2 (7)
-
[22]
• rsupporting — The amount of supporting resources from self only
This is the watch tower performance metric p: P = d′′ 1normalized (8) Now, we can calculate the reward R(p, rdistributed, rsupporting) by defining: • rdistributed — Total supporting resources distributed from self and others. • rsupporting — The amount of supporting resources ...
-
[23]
Let us define the following: • pi — Represents the performance metric for the i-th watch tower
Neighbour Performance Reward: We now describe how the neighbour reward is calculated. Let us define the following: • pi — Represents the performance metric for the i-th watch tower. • n — The number of neighbouring watch towers is 3. • Rneighbourhood — The neighbour reward acr...
-
[24]
Sum over neighbouring watch towers individual performance: Rneighbourhood(n, pi) = nX i=1 pi (10)
-
[25]
Let us define the following: • pi — Represents the performance metric for the i-th watch tower
Collective Performance Reward: We now describe how the collective reward is calculated. Let us define the following: • pi — Represents the performance metric for the i-th watch tower. • n — The total number of watch towers. • Rcollective — The collective reward across all watc...
-
[26]
Compute the Mean Squared Error MSE (n, pi) of watch tower performances: MSE(n, pi) = 1 n nX i=1 p2 i (11)
-
[27]
Calculate the collective reward: Rcollective(n, pi) = 1− 1 − p MSE(n, pi) (12) A.5.3 O CEAN PLASTIC COLLECTION Reward Description
-
[28]
Collect Trash - This is a positive reward of 1 given for each floating plastic pebble col- lected
-
[29]
The lowest trash count is scaled by 0.01
Lowest Collected Trash Count - This is a positive reward given at each time step for the lowest collected trash count amongst all agents. The lowest trash count is scaled by 0.01. The steps to calculate the lowest collected trash count reward can be found in Equation 15
-
[30]
Crossed Border - This is a negative reward of −100 given when the border is crossed
-
[31]
Collided with Other Vessel- This is a negative reward of−100 given when colliding with other vessel
-
[35]
Calculation steps:
Collect Trash - To calculate the Collect Trash reward, let us define the following: • rt — Reward for each trash pebble collected. Calculation steps:
-
[36]
Get the number of collected trash pebbles: rt = NX i=1 I(pi is collected) (13)
-
[37]
• T — Set of all agents lowest collected trash count
Lowest Collected Trash Count - To calculate the lowest collected trash count reward, let us define the following: 46 Preprint • a = 0.01 — Lowest trash count factor. • T — Set of all agents lowest collected trash count. Calculation steps:
-
[38]
, tn), where t ∈ T (14)
Get the lowest trash count from all agents: M (T ) = min(t1, t2, . . . , tn), where t ∈ T (14)
-
[39]
Calculate the lowest collected trash count reward R(T ): R(T ) =M (T ) × a (15)
-
[40]
• ⃗ p— The vessel position
Crossed Border - To calculate the Crossed Border reward, let us define the following: • eh = 200— The environment half extend. • ⃗ p— The vessel position. • rcb — Crossed boundary reward. Calculation steps:
-
[41]
We can now calculate the Crossed Border reward: rcb = −100 if (px > ehor px < −eh or py > ehor py < −eh) 0 otherwise (16)
-
[42]
• Np — Neighbouring vessel positions
Collided with Other Vessel - To calculate the Collided with Other Vessel reward, let us define the following: • ⃗ p— The vessel position. • Np — Neighbouring vessel positions. • rc — Collision reward. Calculation steps:
-
[43]
We can now calculate the Collided with Other Vessel reward: rc = −100 if ∃⃗ n∈ Np such that ⃗ pcollides with ⃗ n 0 otherwise (17)
-
[44]
• dthresh — Distance threshold to closest neighbouring vessel
Close to Other Vessel - To calculate the lowest collected trash count reward, let us define the following: • d — Distance to closest neighbouring vessel. • dthresh — Distance threshold to closest neighbouring vessel. Calculation steps:
-
[45]
r = 10 if d < dthresh 0 otherwise (18)
Calculate close to other vessel reward r. r = 10 if d < dthresh 0 otherwise (18)
-
[46]
• P — All floating plastic pebble positions, { ⃗ p1, ⃗ p2,
Nearby Trash Count Delta - To calculate the nearby trash count delta reward, let us define the following: • dthreshold = 25— Trash count nearby distance threshold. • P — All floating plastic pebble positions, { ⃗ p1, ⃗ p2, . . . , ⃗ pn} ∈P . • ntcold = 0— Old nearby trash coun...
-
[47]
The nearby trash count is calculated by considering only floating plastic pebbles with a distance below dthreshold: ntccurrent = nX i=1 [dist(pi) < dthreshold] (19) 47 Preprint
-
[48]
If the current nearby trash count ntccurrent is larger than the old nearby trash count ntcold, the difference between the two is the reward r(ntcdifference): ntcdifference = ntccurrent − ntcold (20) r(ntcdifference) = max(0, ntcdifference) (21)
-
[49]
Finally the old nearby trash count ntcold is updated with the current nearby trash count ntccurrent: ntcold = ntccurrent (22)
-
[50]
• Pt — All trash pebble positions
Collide with Trash - To calculate the Collide with Trash reward, let us define the following: • ⃗ p— The vessel position. • Pt — All trash pebble positions. • rp — Collision reward. Calculation steps:
-
[51]
We can now calculate the Collide with Trash reward: rc = −100 if ∃⃗ n∈ Pt such that ⃗ pcollides with ⃗ pt 0 otherwise (23) A.5.4 D RONE -BASED REFORESTATION Reward Description
-
[52]
Drop Seed - This is a positive reward given at each seed drop. The drop seed reward consists of the quality of the drop location, a seed reward, in the range of[0, 20] and distance to other seeds and existing trees, a distance reward, in the range of [0, 10]. Therefore, the re...
-
[53]
The deplete energy reward at each time step is higher when carrying a seed than if not carrying a seed
Deplete Energy Holding Seed - This is a negative reward of−1/(episode length/2) given at each time step if the drone is carrying a seed. The deplete energy reward at each time step is higher when carrying a seed than if not carrying a seed. The episode length is 2000
2000
-
[54]
The episode length is 2000
Deplete Energy No Seed - This is a negative reward of−1/(episode length) given at each time step if the drone is not carrying a seed. The episode length is 2000
2000
-
[55]
There are two tasks in which this reward is given
Pick-up Seed - This is an optional positive reward given when a drone is returned to the drone station. There are two tasks in which this reward is given. In ”Subtask: Pick-up Seed at Base” a reward of 100 is given and in ”Subtask: Explore Furthest Distance and Return to Base”...
-
[56]
If the distance to the drone station at time-step t−1 is larger than the current distance, this reward is given at incremental steps of 2.5
Incremental Running Back - After a seed has been dropped, this reward is given in- crementally when flying back to the drone station. If the distance to the drone station at time-step t−1 is larger than the current distance, this reward is given at incremental steps of 2.5. Th...
-
[57]
The steps to calculate the group-up reward can be found in Equation 42
Group-up - This is a positive reward of 10, given at each time-step, if the distance to any neighbouring drone is smaller than 5. The steps to calculate the group-up reward can be found in Equation 42
-
[58]
The reward is the delta between the old and the new potential, if the new potential is higher than the old
High Fertility Location Delta- This is a reward given every time a higher fertility potential seed drop location has been found. The reward is the delta between the old and the new potential, if the new potential is higher than the old. The range of the reward is [0, 1]. The s...
-
[60]
The reward is the delta between the old distance and the new, if the new distance is further than the old
Far Distance Explored Delta - This is a reward given every time a further distance has been explored. The reward is the delta between the old distance and the new, if the new distance is further than the old. The reward range is [0, 200], as 200 is the environment’s half exten...
-
[61]
The reward given is 100
Find Close Tree - This is a reward given when a tree has been found within a 20 meter radius. The reward given is 100. The steps to calculate the find close tree reward can be found in Equation 24. Reward Calculation
-
[62]
To calculate the seed drop reward, let us define the following: • dotmax = 75— Maximum distance to other trees
Drop Seed - To calculate the drop seed reward, we need to calculate the actual seed drop reward and a distance reward. To calculate the seed drop reward, let us define the following: • dotmax = 75— Maximum distance to other trees. • dotmin = 2.5 – Minimum distance to other tre...
-
[63]
This ensures that the newly dropped seed is far enough from existing and seeds dropped in the past, but also that the seed is not too far away from the existing forest
The following condition needs to hold true for this reward to be larger than 0. This ensures that the newly dropped seed is far enough from existing and seeds dropped in the past, but also that the seed is not too far away from the existing forest. (dotmin ≤ det ≤ dotmax) and ...
-
[64]
rs(det, dotmin, dotmax) =Remap(det, dotmin, dotmax, 1, 0) (25)
First, we remap the distance to existing and new trees to [1, 0] so that a high reward is given when the seed is dropped close to existing or new trees. rs(det, dotmin, dotmax) =Remap(det, dotmin, dotmax, 1, 0) (25)
-
[65]
Let us define: • sdd — Seed drop distance to drone station
Applying Multiplier: rs(det, dotmin, dotmax) =rs(det, dotmin, dotmax) × sdrm (26) We now describe how the distance reward is calculated. Let us define: • sdd — Seed drop distance to drone station. • ew = 200— Environment half extend. • drm = 10— Distance reward multiplier. • r...
-
[66]
0 < rs(det, dotmin, dotmax) (27)
The seed drop reward needs to be larger than 0 for the distance reward to be applied. 0 < rs(det, dotmin, dotmax) (27)
-
[67]
sddnormalized = sdd/ew (28) rd(sddnormalized, drm) =sddnormalized × drm (29) The total reward for dropping a seed consists of the drop seed reward 24 and the distance reward 27
Calculate the distance reward using the normalized seed drop distance to the drone station. sddnormalized = sdd/ew (28) rd(sddnormalized, drm) =sddnormalized × drm (29) The total reward for dropping a seed consists of the drop seed reward 24 and the distance reward 27. • rs — ...
-
[68]
• derholding seed(episode lengthmax) — Deplete energy reward while holding a seed
Deplete Energy Holding Seed - To calculate the deplete energy holding seed reward, let us define the following: 49 Preprint • episode length max = 2000— Max episode length. • derholding seed(episode lengthmax) — Deplete energy reward while holding a seed. derholding seed(episo...
2000
-
[69]
• derno seed(episode lengthmax) — Deplete energy reward without holding a seed
Deplete Energy No Seed - To calculate the deplete energy no seed reward, let us define the following: • episode length max = 2000— Max episode length. • derno seed(episode lengthmax) — Deplete energy reward without holding a seed. derno seed(episode lengthmax) =−1/(episode len...
2000
-
[70]
• d — Drone station position
Pick-up Seed - To calculate the Pick-up Seed reward, let us define the following: • p — Drone position. • d — Drone station position. • rps — Pick-up seed reward. Calculation steps:
-
[71]
We can now calculate the Pick-up Seed reward: rps = 1 if distance(p, d) = 0 0 otherwise (33)
-
[72]
Let us define the following: • d0 — Current distance to drone station at time-step 0 in incremental steps
Incremental Running Back - To calculate the incremental running back reward we need to calculate the seed drop reward 24 and distance reward 27. Let us define the following: • d0 — Current distance to drone station at time-step 0 in incremental steps. • ⃗ p0 — Current position...
-
[73]
The current distance d0 is calculated as follows: d0 = vuut nX i=1 (p0i − dpi)2/s (34) If d0 < d−1 continue with next step
The condition for the reward to be given is that the current distance from the drone to the drone station is smaller than in time-step t−1. The current distance d0 is calculated as follows: d0 = vuut nX i=1 (p0i − dpi)2/s (34) If d0 < d−1 continue with next step. (35)
-
[74]
rbm = (rs + rd)/(sdrm + drm) (36)
Let us first calculate the running back multiplier rbm by normalizing the sum of seed drop and distance rewards. rbm = (rs + rd)/(sdrm + drm) (36)
-
[75]
We can now calculate the reward for running back to the drone station: rrb = (rp × rbm)/(dinit − dcharge/s) (37) 50 Preprint
-
[76]
Finally, we need to ensure that the reward rrb is equal to or above 0 and equal to or below rp: rrb = 0 if rrb ≤ 0 rp if rrb > rp rrb otherwise (38)
-
[77]
• dthresh = 5— Distance threshold to closest drone
Group-up - To calculate the group-up reward we need to define the following: • nc — Closest neighbour. • dthresh = 5— Distance threshold to closest drone. • ⃗ p— Current local drone position. • dcn — Distance to closest neighbour • rgu — Reward for grouping up. Calculation steps:
-
[78]
Let us calculate the distance to the closest neighbour: dcn = vuut nX i=1 (pi − nci)2 (39)
-
[79]
We can now calculate the reward for grouping up: rgu = 0 if dthresh ≤ dcn 10 otherwise (40)
-
[80]
• dotmin = 2.5 – Minimum distance to other trees
High Fertility Location Delta - To calculate the high fertility location delta reward, let us define the following: • dotmax = 75— Maximum distance to other trees. • dotmin = 2.5 – Minimum distance to other trees. • dnt — Closest distance to new trees. • det — Closest distance...
-
[81]
If det is smaller or equal todotmax, det is larger or equal todotmin and dnt is larger or equal to dotmin, then follow the next calculation step, otherwise the reward rf lis 0
-
[82]
Calculate the current potential: potcurrent = Map(det, dotmin, dotmax, 1, 0) (41)
-
[83]
We can now calculate the high fertility location reward: rf l= potcurrent − potold if potold < potcurrent, delta of current and old potential 0 otherwise (42)
-
[84]
• hold = 0— Old height, initialized as 0
High Landscape Point Delta - To calculate the high landscape point delta reward, let us define the following: • ⃗ p— Current local drone position. • hold = 0— Old height, initialized as 0. • hcurrent — Current height. • h(⃗ x) — Get height at position ⃗ x. • rh — Height delta ...
-
[85]
Calculate the current height: hcurrent = h(⃗ p) (43)
-
[86]
We can now calculate the hight landscape point delta reward: rf l= hcurrent − hold if hold < hcurrent, delta of current and old height 0 otherwise (44)
-
[87]
• dold = 0— Old furthest distance to drone station, initialized as 0
Far Distance Explored Delta - To calculate the far distance explored delta reward, let us define the following: • ⃗ p— Current local drone position. • dold = 0— Old furthest distance to drone station, initialized as 0. • dcurrent — Current furthest distance to drone station. •...
-
[88]
Calculate the current furthest distance: dcurrent = d(⃗ p) if d(⃗ p) > dold dold otherwise (45)
-
[89]
We can now calculate the far distance delta reward: rf d= dcurrent − dold if dold < dcurrent, delta of current and old furthest distance 0 otherwise (46)
-
[90]
• ew = 200— Environment half extend
Find Close Tree - To calculate the find close tree reward, let us define the following: • ⃗ p— Current local drone position. • ew = 200— Environment half extend. • dcet — Distance to closest existing tree. • cet(⃗ x) — Get closest existing tree given a location. • rct — Find c...
-
[91]
Let us calculate the distance to the closest existing tree and normalize using the environ- ment half extend: dcet = cet(⃗ p)/ew (47)
-
[92]
If dcet < 20 a reward of 100 is given: rct = 100 if dcet ≤ 20 0 otherwise (48) A.5.5 A ERIAL WILDFIRE SUPPRESSION Reward Description
-
[93]
The border is a square around the island in the size of 1500 by 1500
Crossed Border - This is a negative reward of−100 given when the border of the environ- ment is crossed. The border is a square around the island in the size of 1500 by 1500. The island is 1200 by 1200
-
[94]
The island is 1200 by 1200 and there is a girdle of water around the island with a width of 300
Pick-up Water - This is a positive reward of 1 given when the agent steers the aeroplane towards the water. The island is 1200 by 1200 and there is a girdle of water around the island with a width of 300
-
[95]
Fire Out - This is a positive reward of 10 given when the fire on the whole island dies out, with or without the active assistance of the agent
-
[96]
Too Close to Village - This is a negative reward of −50 given when the fire is closer than 150 to the centre of the village
-
[97]
Time Step Burning - This is a negative reward of−0.01 given at each time-step, while the fire is burning
-
[98]
52 Preprint
Find Fire - This is a positive reward of 100 given when a burning tree has been found. 52 Preprint
-
[99]
Find Village - This is a positive reward of 100 given when the village has been found, and the distance between the current local aeroplane position and the village is less than 150
-
[100]
Extinguishing Tree - This is a positive reward in the range of[0, 5] given for each tree that has been in the state burning in time-step t−1 and is now extinguished by dropping water at its location
-
[101]
Reward Calculation
Preparing Tree - This is a positive reward in the range of [0, 1] given for each tree that has been in the state not burning in time-step t−1 and is now wet by dropping water at its location. Reward Calculation
-
[102]
• ⃗ p— The drone position
Crossed Border - To calculate the Crossed Border reward, let us define the following: • eh = 750— The environment half extend. • ⃗ p— The drone position. • rcb — Crossed boundary reward. Calculation steps:
-
[103]
We can now calculate the Crossed Border reward: rcb = −100 if (px > ehor px < −eh or py > ehor py < −eh) 0 otherwise (49)
-
[104]
• ih = 600— Island half extend
Pick-up Water - To calculate the Pick-up Water reward, let us define the following: • eh = 750— The environment half extend. • ih = 600— Island half extend. • ⃗ p— The drone position. • rpw — Pick-up Water reward. Calculation steps:
-
[105]
We can now calculate the Pick-up Water reward: rpw = 1 if (px < ehor px > −eh or py < ehor py > −eh) and (px > ihor px < −ih or py > ihor py < −ih) 0 otherwise (50)
-
[106]
• rnb — No burning tree reward
Fire Out - To calculate the Fire Out reward, let us define the following: • T — All tree states. • rnb — No burning tree reward. Calculation steps:
-
[107]
We can now calculate the Fire Out reward: rnb = 10 if ∀t ∈ T, t̸= ”burning” 0 otherwise (51)
-
[108]
• rcv — Too Close to Village reward
Too Close to Village - To calculate the Too Close to Village reward, let us define the following: • Tc — All tree states, closer to or equal to 150 to the village. • rcv — Too Close to Village reward. Calculation steps:
-
[109]
We can now calculate the Fire Out reward: rcc = −50 if ∃t ∈ Tc, t= ”burning” 0 otherwise (52)
-
[110]
53 Preprint • rtsb — Time Step Burning reward
Time Step Burning - To calculate the Time Step Burning reward, let us define the following: • T — All tree states. 53 Preprint • rtsb — Time Step Burning reward. Calculation steps:
-
[111]
We can now calculate the Time Step Burning reward: rtsb = −0.01 if ∀t ∈ T, t= ”burning” 0 otherwise (53)
-
[112]
• dt = 150— Distance threshold
Find Fire - To calculate the Find Fire reward, let us define the following: • ⃗ p— The drone position. • dt = 150— Distance threshold. • T — All tree states. • rf — Find Fire reward. Calculation steps:
-
[113]
We can now calculate the Find Fire reward: rf = 100 if ∃t ∈ T such that distance(p) < dt meters and t = ”burning” 0 otherwise (54)
-
[114]
• dt = 150— Distance threshold
Find Village - To calculate the Find Village reward, let us define the following: • ⃗ p— The drone position. • dt = 150— Distance threshold. • rv — Find Village reward. Calculation steps:
-
[115]
We can now calculate the Find Village reward: rv = 100 if distance(⃗ p) ≤ dt meters 0 otherwise (55)
-
[116]
• re — Extinguish Tree reward
Extinguishing Tree - To calculate the Extinguish Tree reward, let us define the following: • T — All tree states. • re — Extinguish Tree reward. Calculation steps:
-
[117]
We can now calculate the Extinguish Tree reward: re = 5 X t∈T I(tprevious = ”burning” and tcurrent = ”extinguished”) (56)
-
[118]
• rp — Preparing Tree reward
Preparing Tree - To calculate the Preparing Tree reward, let us define the following: • T — All tree states. • rp — Preparing Tree reward. Calculation steps:
-
[119]
We can now calculate the Preparing Tree reward: re = X t∈T I(tprevious = ”not Burning” and tcurrent = ”wet”) (57) A.6 T ASK DESCRIPTION AND REWARD SCALE A.6.1 W IND FARM CONTROL Task Description
-
[120]
The agent’s goal is to rotate the wind turbine to be oriented against the wind direction and hence generate energy
Main Task: Generate Energy - This is the main task of the environment. The agent’s goal is to rotate the wind turbine to be oriented against the wind direction and hence generate energy. 54 Preprint
-
[121]
Reward Scale Table 8: Main- and Sub-Task Reward Scale Task Reward 1
Subtask: Avoid Damage - This is a subtask to turn the wind turbine 90 degrees away so that the wind turbine rotor blades are parallel to the wind direction, avoiding damage to the wind turbine’s rotor blades. Reward Scale Table 8: Main- and Sub-Task Reward Scale Task Reward 1. 2
-
[122]
Avoid Damage 0 1 A.6.2 W ILDFIRE RESOURCE MANAGEMENT Task Descriptions
-
[123]
The goal of the agent is to distribute a total of 1.0 resources at each time step to self or neighbouring watch towers
Main Task: Distribute Resources - This is the main task of the environment. The goal of the agent is to distribute a total of 1.0 resources at each time step to self or neighbouring watch towers. If the agent is out of resources, it has to remove resources from self or neighbo...
-
[124]
Subtask: Keep All - This is a subtask with the same goal as the main task, however distributing resources to self yields higher rewards than distributing them to neighbouring watch towers
-
[125]
Reward Scale Table 9: Main- and Sub-Task Reward Scale Task Reward 1
Subtask: Distribute All - This is a subtask with the same goal as the main task, however distributing resources to neighbouring watch towers yields higher rewards than distributing them to self. Reward Scale Table 9: Main- and Sub-Task Reward Scale Task Reward 1. 2. 3
-
[126]
Watch Tower Performance 1 10 1
-
[127]
Neighbourhood Performance 1 1 10
-
[128]
Collective Performance 1 1 1 A.6.3 O CEAN PLASTIC COLLECTION Task Description
-
[129]
Main Task: Plastic Collection - This is the main task of the environment. The goal for the agent is to accelerate and steer the plastic collection vessel to collect as many float- ing plastic pebbles as possible while avoiding crashing into other vessels and crossing the envir...
-
[130]
Subtask: Find Highest Polluted Area - This is a subtask with the goal of finding the highest trash population area in a given scenario
-
[131]
Subtask: Group Up - This is a subtask with the goal of finding other vessels and staying close to other vessels while collecting as many floating plastic pebbles as possible
-
[132]
55 Preprint Reward Scale Table 10: Main- and Sub-Task Reward Scale Task Reward 1
Subtask: Avoid Plastic - This is a subtask with the goal of avoiding floating plastic peb- bles. 55 Preprint Reward Scale Table 10: Main- and Sub-Task Reward Scale Task Reward 1. 2. 3. 4
-
[133]
Collect Trash 1 1 1 -1
-
[134]
Global Lowest Trash Collected 1 1 1 0
-
[135]
Crossed Border 1 1 1 1
-
[136]
Collided with Other Vessel 1 1 1 1
-
[137]
Close to Other Vessel 0 0 1 0
-
[138]
Nearby Trash Count Delta 0 1 0 0
-
[139]
Collide with Trash 0 0 0 1 56 Preprint A.6.4 D RONE -BASED REFORESTATION Task Description
-
[140]
Main Task: Maximize Collective Planted Tree Count - This is the main task of the environment. The goal for the agent is to pick up a seed and re-charge batteries at the drone station, explore to find fertile ground for the seed, that is, a location that is close to existing tr...
-
[141]
Subtask: Find Closest Forest Perimeter - This is a subtask with the goal of finding the closest forest perimeter
-
[142]
In this subtask, the initial position of drones is random instead of at the drone station
Subtask: Pick-up Seed at Base - This is a subtask with the goal of going back to the drone station, picking up a seed, and recharging the battery. In this subtask, the initial position of drones is random instead of at the drone station
-
[143]
Subtask: Drop Seed - This is a subtask with the goal of finding the most fertile soil and dropping a seed
-
[144]
Subtask: Find Highest Potential Seed Drop Location - This is a subtask with the goal of finding soil with the highest fertility
-
[145]
Subtask: Find Highest Point on Landscape - This is a subtask with the goal of finding the highest point on the landscape
-
[146]
Reward Scale Table 11: Main- and Sub-Task Reward Scale Task Reward 1
Subtask: Explore Furthest Distance and Return to Base - This is a subtask with the goal of exploring the furthest from the drone station and returning. Reward Scale Table 11: Main- and Sub-Task Reward Scale Task Reward 1. 2. 3. 4. 5. 6. 7. 8
-
[147]
Drop Seed 1 0 0 0 1 0 0 0
-
[148]
Deplete Energy Holding Seed 1 1 1 1 1 1 1 1
-
[149]
Deplete Energy No Seed 1 1 1 1 1 1 1 1
-
[150]
Pick-up Seed 1 0 100 1 1 0 0 0-200
-
[151]
Incremental Running Back 1 0 0 1 1 0 0 1
-
[152]
High Fertility Location Delta 0 0 0 0 0 1 0 0
-
[153]
High Landscape Point Delta 0 0 0 0 0 0 1 0
-
[154]
Far Distance Explored Delta 0 0 0 0 0 0 0 1
-
[155]
Find Close Tree 0 1 0 0 0 0 0 0 A.6.5 A ERIAL WILDFIRE SUPPRESSION Task Description
-
[156]
The goal for the agent is to pick up water and extinguish as many burning trees as possible or prepare a forest that is not yet burning
Main Task: Minimize Time Fire Burning and Prevent Fire From Moving Towards Village - This is the main task of the environment. The goal for the agent is to pick up water and extinguish as many burning trees as possible or prepare a forest that is not yet burning. A secondary g...
-
[157]
Subtask: Maximize Extinguished Burning Trees - This is a subtask with the goal of extinguishing as many burning trees as possible
-
[158]
Subtask: Maximize Preparing Non-Burning Trees - This is a subtask with the goal of preparing as many non-burning trees as possible
-
[159]
57 Preprint
Subtask: Minimize Time Fire Burning - This is a subtask with the goal of minimizing the time of trees burning. 57 Preprint
-
[160]
Subtask: Protect Village - This is a subtask with the goal of protecting the village from approaching fire
-
[161]
Subtask: Pick Up water - This is a subtask with the goal of picking up water
-
[162]
Subtask: Drop Water - This is a subtask with the goal of dropping water anywhere
-
[163]
Subtask: Find Fire - This is a subtask with the goal of finding a burning tree
-
[164]
Reward Scale Table 12: Main- and Sub-Task Reward Scale Task Reward 1
Subtask: Find Village - This is a subtask with the goal of finding the village. Reward Scale Table 12: Main- and Sub-Task Reward Scale Task Reward 1. 2. 3. 4. 5. 6. 7. 8. 9
-
[165]
Crossed Border 1 1 1 1 1 1 1 1 1
-
[166]
Pick-up Water 1 1 1 1 1 100 1 0 0
-
[167]
Fire Out 1 1 1 1 1 0 0 0 0
-
[168]
Too Close to Vil- lage 1 1 1 1 10 0 0 0 0
-
[169]
Time Step Burn- ing 0 0 0 1 0 0 0 0 0
-
[170]
Find Fire 0 0 0 0 0 0 0 1 0
-
[171]
Find Village 0 0 0 0 0 0 0 0 1 Drop Water
-
[172]
Extinguishing Tree 1 10 1 1 1 1 1 0 0
-
[173]
Preparing Tree 1 1 5 1 1 1 1 0 0 58 Preprint A.7 A DDITIONAL RESULTS A.7.1 W IND FARM CONTROL : T RAIN & T EST METRICS 0 2 4 6 8 Step 1e6 0.9 1.0 1.1Value Entropy source training test 0 2 4 6 8 Step 1e6 0 5 10Value Extrinsic Value Estimate source training test 0 2 4 6 8 Step 1...
-
[2016]
James MacCarthy, Sasha Tyukavina, Mikaela Weisse, and Nancy Harris
URL https://link.springer.com/book/10.1007/978-3-319-31858-5 . James MacCarthy, Sasha Tyukavina, Mikaela Weisse, and Nancy Harris. New Data Confirms: Forest Fires Are Getting Worse, August 2022. URL https://www.wri.org/insights/ global-trends-forest-fires . MAJA J. MATARIC. Us...
2022 doi
-
[2019]
ISSN 2673-6187
Frontiers in Remote Sensing, 3, March 2022. ISSN 2673-6187. doi: 10.3389/frsen.2022. 825190. URL https://www.frontiersin.org/journals/remote-sensing/ articles/10.3389/frsen.2022.825190/full. Publisher: Frontiers. CRED / UCLouvain. EM-DAT - The international disaster database, ...
2022
-
[2020]
arXiv:1910.05789 [cs, stat]
URL http://arxiv.org/abs/1910.05789. arXiv:1910.05789 [cs, stat]. Intergovernmental Panel on Climate Change.Managing the Risks of Extreme Events and Disasters to Advance Climate Change Adaptation: Special Report of the Intergovernmental Panel on Climate Change. Cambridge Unive...
1910 arXiv
-
[2248]
URL https://ieeexplore.ieee.org/ document/10107729
doi: 10.1109/TWC.2023.3268082. URL https://ieeexplore.ieee.org/ document/10107729. Conference Name: IEEE Transactions on Wireless Communications. J¨org L¨assig, Kristian Kersting, and Katharina Morik. Computational Sustainability | SpringerLink,
2023
-
[6461]
URL https://www.liebertpub.com/doi/full/ 10.1089/big.2014.0026
doi: 10.1089/big.2014.0026. URL https://www.liebertpub.com/doi/full/ 10.1089/big.2014.0026. Publisher: Mary Ann Liebert, Inc., publishers. James D. Ford, Simon E. Tilleard, Lea Berrang-Ford, Malcolm Araos, Robbert Biesbroek, Alexan- dra C. Lesnikowski, Graham K. MacDonald, Ang...
2014
-
[7454]
URL http://link.springer.com/10.1007/ s10458-005-2631-2
doi: 10.1007/s10458-005-2631-2. URL http://link.springer.com/10.1007/ s10458-005-2631-2 . Georgios Papoudakis, Filippos Christianos, Lukas Sch¨afer, and Stefano V . Albrecht. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks, November 2021. U...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.