{"id":"28e316fd-0cb0-451e-bebe-9367c26289f4","arxiv_id":"2507.18867","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LIGHT injects decision-tree-extracted human knowledge into individual intrinsic rewards and improves sparse-reward MARL performance over QMIX, VDN, QTRAN, LIIR, and MASER.","lead":"The paper proposes LIGHT, a training method that turns simple human rules into per-agent bonus rewards for multi-agent reinforcement learning. In sparse-reward tasks like StarCraft battles, LIGHT learns faster and reaches higher win rates than standard algorithms, while its behavior aligns with the human rules it was trained to follow.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dual loss in Eqs. 6-8 is internally inconsistent: if individual Q_i are trained only on r_i (Eq. 7), monotonic mixing makes greedy actions maximize intrinsic, not team, reward; the paper neither proves nor specifies the gradient flow.","rationale":"The reader identified the joint training scheme as the weakest assumption, but framed it as an unproven risk to IGM. My read is more specific: as written, the two objectives can be internally inconsistent. If the individual Q-networks are updated only by Eq. 7, monotonic mixing makes the greedy joint action the argmax of intrinsic reward, so the extrinsic team reward never influences action selection; if both losses update theta_i, the paper leaves unanalyzed how the small auxiliary intrinsic TD loss (lambda_K=0.02) interacts with the team TD loss. Either way, the central claim that LIGHT maximizes joint action value while incorporating human expertise is not established by the equations. The empirical results could still hold if the hand-coded rules happen to be good heuristics, but the stated mechanism would be wrong. This concern is more decisive than the circular behavior-alignment analysis or the missing transfer experiment, and it can be settled by the proposed gradient-flow check plus a small conflicting-reward experiment. I therefore retain the reader's CONDITIONAL verdict rather than moving to REJECT, because the flaw is concrete but fixable by clarifying the update scheme or providing a proof/experiment; the condition should explicitly require resolving this objective inconsistency.","tokens_in":11925,"tokens_out":9663,"duration_ms":109025,"concrete_test":"Run a tabular one-agent or two-agent gridworld MDP with known team reward r_ext and a hand-coded human-knowledge rule that prefers a suboptimal action in exactly one state. Train LIGHT with the stated losses and log greedy actions from Q_i and Q_tot. If the greedy action follows the rule-preference while the team-optimal action differs, the Eq. 6/7 split is confirmed. Also instrument the released code to check whether dL_TD/dtheta_i is nonzero; if it is zero, Q_i are decoupled from r_ext by construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The joint training scheme is the load-bearing weakness. Eq. 6 trains Q_tot on R_t = r_ext + lambda*(1/N)*sum_i r_i (Eq. 5), while Eq. 7 defines L_i(theta_i) with a TD target using only the intrinsic reward r_i. In the QMIX/VDN architectures used here, Q_tot is a monotone function of the Q_i, so IGM forces the greedy joint action to equal the tuple of individual argmax_a Q_i. Therefore, as soon as Q_i are updated by Eq. 7 (the paper says 'individual Q-values update the intrinsic reward value', and Eq. 8 adds L_i to the total loss), action selection is determined by intrinsic reward alone; the extrinsic team reward r_ext cannot affect which joint action is chosen. Concretely, if a rule prefers action B while the environment rewards A, the agent selects B even if Q_tot is trained on R_t, because Q_tot is monotone in Q_i and Q_i(B) > Q_i(A). The text also conflates lambda in Eq. 5 with lambda_K in Eq. 8, and Table II omits lambda, so the actual weighting is unknown. If instead L_TD backpropagates into theta_i (as 'end-to-end' in Fig. 1 suggests), Eq. 7 is a competing auxiliary objective whose effect on credit assignment is unanalyzed. Either reading leaves the central claim 'maximizing the joint action value' unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LIGHT, a reward-shaping framework for cooperative multi-agent reinforcement learning under the CTDE paradigm. LIGHT computes an individual intrinsic reward for each agent as the negative Euclidean distance between the agent's action-value distribution and a 'soft logic rule' distribution derived from human knowledge. This intrinsic reward is added, with a weighting coefficient, to the extrinsic team reward, and the overall pipeline is trained with a combined TD loss for the mixing network and an individual Q-learning loss per agent. The method is evaluated on Level-Based Foraging (LBF) and StarCraft Multi-Agent Challenge (SMAC) in both dense- and sparse-reward settings, against five baselines, and is also analyzed through ablations and behavior-alignment statistics.","tokens_in":12291,"tokens_out":4719,"duration_ms":46425,"significance":"If the empirical claims hold, LIGHT provides a simple, plug-in method for injecting human knowledge into value-decomposition MARL algorithms, with reported gains on sparse-reward benchmarks and better alignment of agent behavior with human rules. The paper includes five-seed comparisons, ablations, and visualizations of the learned intrinsic rewards, which are informative. However, the strength of the contribution is undercut by an ambiguity in the joint training objective that may violate the IGM principle, by the circularity of the behavior-alignment analysis, and by missing implementation details that prevent reproduction. These issues are addressable, but they require substantial revision rather than copy editing.","major_comments":[{"comment":"The two-loss training scheme is internally inconsistent as specified. Eq. (6) trains Q_tot on R_t = r_ext + lambda * (1/N) * sum_i r_i,t, while Eq. (7) trains each Q_i with a TD target that uses only the intrinsic reward r_i,t. For the QMIX/VDN architectures used in the paper, Q_tot is a monotone function of the individual Q_i, so under the IGM principle the greedy joint action is the tuple of individual argmax actions. If the Q_i are updated solely by Eq. (7), action selection is driven exclusively by the intrinsic reward, and the extrinsic team reward cannot influence the chosen joint action. If instead gradients from Eq. (6) flow into the individual parameters in an end-to-end manner, the paper must state this explicitly and analyze the resulting gradient flow, since Eq. (7) then acts as an auxiliary objective whose effect on credit assignment is unexamined. The manuscript provides neither a proof that the combined objective preserves IGM nor an empirical check, such as comparing argmax_a Q_tot(tau,a) with the tuple of argmax_{a_i} Q_i(tau_i,a_i) on collected states. Without this, the central claim that LIGHT 'maximizes the joint action value' is unsupported.","section":"Section III-B, Eqs. (5)-(8)"},{"comment":"The behavior-alignment analysis is circular. Eq. (4) defines the intrinsic reward as the negative Euclidean distance between the agent's action distribution and the human-knowledge distribution H(o_i^t), and Eq. (8) trains the agents to maximize this reward. Measuring the fraction of actions consistent with the human-knowledge rules therefore confirms by construction, not empirically, that the learned policies align with the injected rules. The claim that LIGHT 'can efficiently capture the given human knowledge' is a tautology. The authors should test alignment on held-out human-labeled states or compare the consistency metric of LIGHT against a variant where the intrinsic reward is not directly derived from the same rules (as in the 'LIGHT w/o knowledge' ablation), and demonstrate a distributional difference.","section":"Section VI-D, Fig. 10"},{"comment":"Three implementation details that are load-bearing for reproducibility are missing. (i) The weight lambda in Eq. (5) is never given in Table II or anywhere in the text; only lambda_K = 0.02 is mentioned in Section VI. (ii) The probability p in Algorithm 1, and its role in defining the soft-logic-rule distribution H(o_i^t), is not mathematically specified; it is unclear whether p is a fixed constant, a learned quantity, or a function of health. (iii) The human-knowledge rule for the LBF experiments is never presented, so the LBF results cannot be reproduced. The authors should report lambda, define p precisely, and provide the full rule formulation for both benchmarks.","section":"Section III-B and Table II"},{"comment":"The claim that LIGHT 'achieves impressive performance on all scenarios' is not backed by statistical analysis. Only five seeds are used and no confidence intervals or significance tests are reported; in Fig. 3, VDN appears to match LIGHT before 0.5M steps, and in Fig. 5 the dense-reward results saturate for all methods. A formal comparison with error bars or paired tests is needed to support the superiority claim, especially for the sparse-reward SMAC maps.","section":"Section VI-A, Figs. 3-6"}],"minor_comments":[{"comment":"The summation index 'ri,jt' appears to be a typo for r_i,t; please fix it.","section":"Eq. (5)"},{"comment":"The output line 'Get the probability p, where argmax_a p = M' is unclear; p should be defined as a distribution or a scalar, and its mapping to H(o_i^t) should be stated.","section":"Algorithm 1"},{"comment":"The phrase 'before 0.5 steps' lacks the time unit (presumably 0.5 million steps); please correct it.","section":"Section VI-A, Fig. 3"},{"comment":"The y-axis 'Similarity' in Fig. 10(b) is not defined in the text; specify how action-consistency with human knowledge is computed.","section":"Fig. 10"},{"comment":"In Fig. 4 caption it says '3-agent & 3-food' but the surrounding text refers to '4-agent & 2-food'; please reconcile the description.","section":"Section VI-B"},{"comment":"Reference [28] appears to have the author ordering garbled; also check that all citations in the text, such as [27] near the IGM definition, point to the correct entries.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: LIGHT is a reasonable new combination — decision-tree soft rules used to shape per-agent intrinsic rewards inside QMIX/VDN — and the reported win-rate improvements over QMIX, VDN, QTRAN, LIIR and MASER on sparse-reward SMAC and LBF are plausible. The behavior-alignment analysis is circular, but that is a side exhibit, not the main claim. The main claim is empirical, and five seeds with ablations give it some weight. The paper deserves a serious referee, but it needs a real revision before the results can be trusted.\n\nWhat is actually new: using generalized human expertise, extracted offline as decision-tree rules, to define an intrinsic reward as negative distance between agent action distribution and soft-rule distribution. That specific combination with value decomposition I have not seen before. The paper is honest that the rules come from the authors' prior line of work, and the ablation showing human knowledge matters (random distributions hurt) is informative. Good that they test on both LBF and SMAC, dense and sparse reward.\n\nNow the soft spots, in order of severity. First, the training objective is underspecified and possibly inconsistent. Eq. 6 defines L(θ) as a TD loss on R_t = r_ext + λ(1/N)Σr_i, but says it updates only the mixing network. Eq. 7 defines L_i(θ_i) as a TD loss on intrinsic reward only, and Eq. 8 sums them. If L_TD gradients do not flow into θ_i, then the individual Q-functions are trained exclusively on intrinsic reward, and under monotonic mixing the greedy joint action is chosen by intrinsic reward alone — the extrinsic team reward cannot affect action selection. If instead L_TD does backprop into θ_i, then Eq. 7 is an auxiliary objective with unknown interference, and neither the paper's text nor Fig. 1 makes the gradient flow clear. This is not a minor omission; it determines whether the method is maximizing the team reward or only human preference. They need to state exactly which parameters each loss touches, or prove IGM preservation, or at least provide an ablation where Eq. 7 is removed.\n\nSecond, key hyperparameters are missing. The λ in Eq. 5 does not appear in Table II; the probability p in Algorithm 1 is left undefined; and the LBF human rule is never given. Without those, reproducing the results is guesswork. The paper should release code, with the rule files, as part of the submission.\n\nThird, the behavior-alignment analysis in Section VI-D is tautological: Eq. 4 defines the intrinsic reward as negative distance to the human-knowledge distribution and Eq. 8 trains agents to maximize it, so observing alignment is expected. It is fine as an interpretability check, but it should be labeled as such, not as evidence that human knowledge improves learning. The improvement claim rests on the win-rate curves, not on this analysis.\n\nMinor: the abstract promises \"better knowledge reusability across different sparse-reward tasks,\" but there is no transfer experiment. Either add one or soften the claim.\n\nOverall: the core idea is sound and the empirical direction is promising, but the missing training-objective specification is load-bearing. I would send this to peer review, and I would expect a major revision that clarifies the loss structure, fills in the missing hyperparameters, and releases the rules and code. If the authors can show the gradient flow and reproduce the gains with a clean implementation, this could be a useful contribution.","headline":"LIGHT is a plausible new combination of decision-tree human rules and intrinsic-reward MARL with promising empirical gains, but the training objective is underspecified in a way that determines whether the method actually optimizes team reward.","tokens_in":12779,"tokens_out":2552,"would_cite":false,"duration_ms":24368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LIGHT injects human-knowledge preferences as per-agent intrinsic rewards and reports state-of-the-art results on sparse-reward MARL benchmarks.","keywords":["multi-agent reinforcement learning","intrinsic reward","human expertise","sparse rewards","value decomposition","centralized training decentralized execution","Level-Based Foraging","StarCraft Multi-Agent Challenge"],"falsifier":"On the 5m SMAC sparse-reward map, retrain LIGHT with the human-knowledge rule inverted - recommend attack whenever health is below 15 instead of retreat. If the win-rate curve does not fall markedly below LIGHT with the correct rule, the knowledge signal is not what is driving the reported improvement; if it does fall, the mechanism is confirmed.","tokens_in":11715,"feed_emoji":"🧠","tokens_out":6318,"duration_ms":61469,"temperature":0.7,"pith_summary":"LIGHT is a proposal for sparse-reward cooperative multi-agent reinforcement learning. The paper argues that a few soft logic rules distilled from human knowledge can be turned into dense, per-agent intrinsic rewards that guide exploration more effectively than hand-designed shaping or generic intrinsic motivation. The reward for each agent is the negative Euclidean distance between a transformed action distribution from the rules and the agent's own action-value distribution, and the whole module is trained end to end inside a value-decomposition algorithm. If the claim is right, MARL agents in tasks with only a team reward can learn faster, align their behavior with human preferences, and reuse the same knowledge across related tasks, without needing a fully specified dense reward function.","feed_headline":"Human-knowledge rewards beat five MARL baselines","feed_subtitle":"LIGHT turns human rules into per-agent intrinsic rewards and beats five baselines on sparse-reward benchmarks.","key_machinery":"The load-bearing object is the intrinsic reward in Eq. (4): $r_i^t = -\\|\\varphi_i(H(o_i^t)) - \\varphi_i(A_i^t)\\|_2$, the negative Euclidean distance between a transformed soft-logic-rule action distribution and the agent's own action-value distribution. It converts a few human-supplied rules into a dense, time-varying per-agent learning signal. The other piece of machinery is the two-loss training scheme: the mixing network is updated on the shaped team reward $R_t = r_t^{ex} + \\lambda \\frac{1}{N}\\sum_i r_i^t$ (Eqs. 5-6), while each individual Q-network is updated on its intrinsic reward (Eq. 7), with the total loss $L = L_{TD}(\\theta) + \\lambda_K L_i(\\theta_i)$ (Eq. 8). This is the mechanism by which human knowledge enters the credit assignment loop.","core_discovery":"On its own terms, the paper's central discovery is that human expertise can be carried by a differentiable distance penalty rather than by hand-coded rewards or hard constraints. LIGHT computes, for each agent, $r_i^t = -\\|\\varphi_i(H(o_i^t)) - \\varphi_i(A_i^t)\\|_2$, the negative Euclidean distance between the human-knowledge action distribution and the agent's own action-value distribution, and adds the average of these signals to the team reward for the mixing network while training each individual Q-network on its intrinsic reward. The authors report that LIGHT outperforms QMIX, VDN, QTRAN, LIIR, and MASER on Level-Based Foraging and on sparse-reward StarCraft scenarios, that plugging LIGHT onto QMIX or VDN improves the backbone, and that the trained agents' actions match the human-knowledge rules more often than QMIX's do.","pith_inferences":["If the improvement is real, one natural next step is to learn the rule extraction together with the policy rather than fixing rules beforehand; that would let the intrinsic reward track the agents' evolving competence.","The distance signal essentially acts as a dynamic reward bias, so its effect should be equivalent to potential-based shaping only when the two distributions stay close. Quantifying that bias and testing whether LIGHT preserves the optimal policy under arbitrary rule mistakes would connect it to standard reward-shaping invariance results.","A direct prediction of the paper's mechanism is that LIGHT's advantage over plain QMIX should shrink as extrinsic rewards become denser, because the human-knowledge signal then mostly duplicates information the environment already provides."],"forward_implications":["LIGHT can be dropped onto any value-decomposition backbone: the paper shows LIGHT-QMIX and LIGHT-VDN both beat their fine-tuned backbones on sparse-reward SMAC maps.","Agents trained with LIGHT complete 4m and 2m_vs_1z episodes in fewer steps than QMIX and match human-knowledge rules more often.","Removing either the intrinsic reward or the human-knowledge distribution degrades performance, so both components contribute to the reported gains.","The method requires only a few soft logic rules as prior knowledge, not a fully specified dense reward function.","Because the module is end-to-end and parameterized, the same knowledge can be reused across tasks that share rules."],"supporting_citations":[{"why":"Provides the monotonic value-decomposition backbone that LIGHT augments and the QMIX baseline it must beat.","marker":"[9]"},{"why":"Supplies the additive value-decomposition baseline and the backbone used in the LIGHT-VDN variant.","marker":"[8]"},{"why":"Defines the IGM consistency criterion that value-decomposition methods, including LIGHT, are expected to satisfy.","marker":"[27]"},{"why":"Introduces the individual intrinsic reward paradigm and the LIIR baseline against which LIGHT is compared.","marker":"[18]"},{"why":"Supplies the MASER baseline and the sparse-reward SMAC configuration that LIGHT adopts for its experiments.","marker":"[19]"},{"why":"Shows that generalizable multi-agent coordination skills can be discovered from offline data, the basis for extracting the human-knowledge rules.","marker":"[29]"},{"why":"Provides the decision-tree extraction technique used to turn trained behaviors or offline data into interpretable rules.","marker":"[25]"}],"fun_headline_variants":["Human expertise as distance penalties for MARL rewards","LIGHT beats five MARL baselines with human-rule rewards","Distance-to-human-rule intrinsic rewards trump baselines","Turn human knowledge into per-agent rewards in MARL","Human-rule distance drives better multi-agent exploration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that training each agent's Q-network on the intrinsic reward alone (Eq. 7), while training the mixing network on the shaped team reward (Eq. 6), keeps the two objectives consistent enough that individual utilities still factor the true team value; the paper offers no proof or empirical check that this preserves the IGM principle, and if the losses conflict the reported gains could vanish.","fun_headline_variants_meta":{"raw":{"variants":["Human expertise as distance penalties for MARL rewards","LIGHT beats five MARL baselines with human-rule rewards","Distance-to-human-rule intrinsic rewards trump baselines","Turn human knowledge into per-agent rewards in MARL","Human-rule distance drives better multi-agent exploration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2731,"prompt_tokens":930,"completion_tokens":1801,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1724}},"tokens_in":546,"tokens_out":1801,"duration_ms":13484,"temperature":1.0,"reasoning_tokens":1724,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:06:51.383611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the 5m SMAC sparse-reward map, retrain LIGHT with the human-knowledge rule inverted - recommend attack whenever health is below 15 instead of retreat. If the win-rate curve does not fall markedly below LIGHT with the correct rule, the knowledge signal is not what is driving the reported improvement; if it does fall, the mechanism is confirmed.","supporting_citations":[{"cited_title":"QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Provides the monotonic value-decomposition backbone that LIGHT augments and the QMIX baseline it must beat."},{"cited_title":"Value-decomposition networks for cooperative multi-agent learning based on team reward,","cited_arxiv_id":null,"evidence_quote":"Supplies the additive value-decomposition baseline and the backbone used in the LIGHT-VDN variant."},{"cited_title":"QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Defines the IGM consistency criterion that value-decomposition methods, including LIGHT, are expected to satisfy."},{"cited_title":"Liir: Learning individual intrinsic reward in multi-agent reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Introduces the individual intrinsic reward paradigm and the LIIR baseline against which LIGHT is compared."},{"cited_title":"MASER: Multi-agent reinforcement learning with subgoals generated from experience replay buffer,","cited_arxiv_id":null,"evidence_quote":"Supplies the MASER baseline and the sparse-reward SMAC configuration that LIGHT adopts for its experiments."},{"cited_title":"Discovering generalizable multi-agent coordination skills from multi-task offline data,","cited_arxiv_id":null,"evidence_quote":"Shows that generalizable multi-agent coordination skills can be discovered from offline data, the basis for extracting the human-knowledge rules."},{"cited_title":"Extracting decision tree from trained deep reinforcement learning in traffic signal control,","cited_arxiv_id":null,"evidence_quote":"Provides the decision-tree extraction technique used to turn trained behaviors or offline data into interpretable rules."}],"review_version":1}