{"id":"5c980e01-6986-4ce6-ac0a-759e7a5dcc30","arxiv_id":"2411.14264","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Reward decomposition in multi-agent RL shows packet delivery ratio is the dominant reward component for XR codec adaptation, and the adaptive MHA-QMIX variant improves simulated KPIs over the APS baseline.","lead":"This paper adds reward decomposition, an explainability lens, to multi-agent reinforcement learning agents that choose video data rates for virtual reality users on 5G networks, and it reports that the packet delivery ratio reward dominates the others in simulation. A smart generalist might read it to see how explainable AI can reveal which network quality metric matters most and how that insight can be turned into a better controller.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'PDR is primary contributor' claim is not yet supported: Q1/Q2/Q3 magnitudes are compared across hand-set reward scales with no normalization or counterfactual ablation, so the central explainability conclusion may be an artifact.","rationale":"The paper's central claim has two parts: (1) PDR is the primary contributor to optimal performance, and (2) MHA-QMIX achieves significant KPI gains over APS. The first is the novel explanatory contribution and is the most vulnerable. Figure 7 compares Q magnitudes across three reward components; for such a comparison to carry meaning, the reward components must share a scale, or the analysis must be invariant to rescaling. Since the decomposed Q targets are the raw r_c values, changing the hand-set constants in Eqs. (15)-(17) changes the Q magnitudes, so the observed Q2>Q1>Q3 ordering is not a property of the wireless environment alone. The adaptive weights in Eq. (35) are a softmax of reward statistics and therefore encode the same arbitrary scale, so they do not provide independent evidence. A simple renormalization or ablation experiment would settle this. The theoretical appendix does not rescue the claim: even if the proof were valid, it preserves reward scales; and Eq. (39) does not hold for a shared wireless channel, while the min-based team rewards in Eqs. (18)-(20) do not match the additive assumption in Definition A.1, so the theorem is not applicable to this system. I do not accuse the authors of anything; the architectures are plausible, the tables are internally consistent, and the MHA-QMIX gains may well be real. But the headline explanatory conclusion should be conditional on a scale-invariance/ablation check. This matches the reader's conditional verdict, so no verdict change is needed; the stated conditions should explicitly include the normalization/ablation test described above.","tokens_in":30016,"tokens_out":7428,"duration_ms":71835,"concrete_test":"Retrain the best three algorithms (DQMIX/MH-DQMIX/MHA-DQMIX) under three reward conditions: (i) each r_c min-max normalized to [0,1] over the logged reward distribution; (ii) r = r_p only (PDR-only); (iii) r = r_d + r_th (no PDR). Compare Fig. 7 rank order and Table IV KPIs with mean +/- 95% CI over at least 10 independent seeds. If Q2 remains dominant after normalization and PDR-only matches or beats the composite on XR index/jitter/delay/PLR while no-PDR degrades, PDR-primary is supported; if the rank order flips or PDR-only underperforms, the conclusion is a reward-scale artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that the PDR reward is the primary contributor to optimal performance. The evidence is Fig. 7 (Q2>Q1>Q3), the MSX/RDFX plots, and the adaptive weight behavior in Figs. 12-13. None of these establish cross-component comparability. Rewards in Eqs. (15)-(17) are hand-set in different units: delay is a staircase taking {1,0.75,0.5,0.25,0}; PDR is {1,0.5,0}; DRR is a continuous throughput ratio in [0,1]. A decomposed Q^c is learned against target r_c, so scaling r_c by a constant scales Q^c approximately by the same constant. Hence Q2>Q1 may simply reflect that the PDR reward was assigned near-1 values more often in the simulated favourable regime, not that PDR causally dominates the optimal policy. The adaptive weights in Eq. (35) are softmax over mean+sinh(std) of these same rewards, so they inherit the scale arbitrariness and cannot independently confirm PDR primacy. Theorem 1 does not fix this: its proof in Appendix A is linearity of expectation for additive rewards and preserves reward scales; moreover, Eq. (39) assumes P(s'|s,a)=sum_i F_i(s'|s_i,a_i), which is not true for a shared wireless channel, and the actual team rewards in Eqs. (18)-(20) are minima over agents, not the sum assumed in Definition A.1. The empirical KPI gains of MHA-QMIX in Table IV are plausible but also lack error bars and multiple seeds. The most load-bearing defect is therefore that the headline conclusion is scale-dependent and has no counterfactual support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes explainable multi-agent reinforcement learning (MARL) for XR codec adaptation in 5G networks. It extends reward decomposition to value-function-factorization MARL algorithms (VDN, QMIX, QTRAN), introduces multi-headed and adaptive variants, defines a new explainability metric RDFX, and evaluates the methods in an ns-3 New Radio simulator. The central claims are that, for XR codec adaptation, the Packet Delivery Ratio (PDR) reward component is the primary contributor to performance, and that the best proposed algorithm (MHA-QMIX) achieves notable KPI gains over an Adjust Packet Size baseline.","tokens_in":30386,"tokens_out":3339,"duration_ms":34109,"significance":"If the PDR-importance finding were robust, it would provide concrete guidance for reward design in wireless XR applications and would give practitioners a principled reason to emphasize reliability terms over throughput and delay terms. The paper also makes several constructive algorithmic contributions: a multi-headed architecture that reduces the parameter overhead of decomposed value-function-factorization methods, an online adaptive weighting scheme inspired by multi-task learning, and a new metric (RDFX) for explaining action preferences over ordered action sets. The ns-3 evaluation covers a realistic 5G XR scenario with three user types and three distances. However, these strengths are partly offset by the fragility of the headline interpretability conclusion, which rests on cross-component comparisons of Q-values whose reward scales are not commensurable. The empirical KPI gains are plausible but lack statistical support, and no comparison with non-decomposed versions of the same MARL algorithms is reported, leaving the effect of reward decomposition itself unclear.","major_comments":[{"comment":"The claim that PDR is the primary contributor to optimal performance, based on Q2 > Q1 > Q3 in Fig. 7, is not established because the decomposed Q-values are trained against reward components with different hand-chosen scales and saturation patterns. The delay reward in Eq. (15) is a staircase taking values in {1, 0.75, 0.5, 0.25, 0}, the PDR reward in Eq. (16) takes {1, 0.5, 0}, and the data-rate reward in Eq. (17) is a continuous ratio in [0,1]. Since a decomposed Q-function is approximately proportional to the scale of its target reward, the observed ordering may simply reflect that the PDR reward is near 1 more often in the simulated favorable regime, rather than a causal dominance of PDR in the optimal policy. To support the headline conclusion, the authors should either normalize the reward components to comparable scales before comparing Q-values, run counterfactual ablations (e.g., train without the PDR component, or re-weight components so that all have similar ranges), or both. Without such evidence, the abstract and Section VII statements about PDR being \"the primary contributor\" are scale artifacts.","section":"Section VI-C, Fig. 7, Eqs. (15)-(17)"},{"comment":"The theoretical foundation of the paper depends on an additive decomposability assumption that is not satisfied by the system model used in the simulations. Definition A.1 (Eq. (38)-(39)) assumes both the team reward and the transition function decompose additively over agents, i.e., P(s'|s,a) = Σ_i F_i(s'|s_i,a_i). In the simulated 5G scenario, agents share a common wireless channel, so the next state cannot be written as a sum of per-agent transition functions. In addition, the team rewards used in the experiments are minima over agents, as defined in Eqs. (18)-(20), not the sum required by Eq. (38). Therefore Theorem 1, as proved in Appendix A, does not apply to the actual problem setting. The authors should either relax the theorem to cover the min-based and shared-channel case, clearly state that Theorem 1 holds only for an idealized additive model and is not the basis for the simulation results, or provide a separate argument for the simulation setting.","section":"Appendix A, Definition A.1 and Theorem 1"},{"comment":"The adaptive importance weights in Eq. (35) are computed as a softmax over mean plus sinh(std) of the raw reward components. Because the raw rewards have different scales, as described above, this weight update inherently upweights components with larger mean or variance. The observation in Figs. 12-13 that w2 (PDR) dominates is therefore partly a consequence of the reward construction and does not provide independent confirmation of PDR's causal importance. To make the adaptive mechanism a valid source of evidence about reward importance, the authors should demonstrate that the weight dynamics are stable under rescaling of the reward components, or they should compute weights on normalized reward signals. At minimum, the text should acknowledge this scale dependence and avoid presenting the adaptive weights as corroboration of the PDR-primary conclusion.","section":"Section V-H, Eq. (35), Figs. 12-13"},{"comment":"The performance results in Table IV are reported as single percentage gains without error bars, confidence intervals, or information about the number of random seeds used. The abstract and Section VI-E describe these gains as \"significant average gains,\" but no statistical test or variance measure supports that wording. Additionally, the paper does not compare the decomposed algorithms against their non-decomposed counterparts (VDN, QMIX, QTRAN), so it is unclear whether reward decomposition itself degrades, preserves, or improves learning. The authors should report mean and standard deviation over multiple independent training runs and, ideally, include the non-decomposed baselines so that the effect of the proposed architectural and adaptive changes is isolated.","section":"Section VI-E, Table IV, Fig. 14"}],"minor_comments":[{"comment":"The phrase \"compared to the initial composite reward\" is ambiguous; the intended contrast is between the PDR component and a composite reward that also includes delay and data-rate components, but the sentence as written suggests a comparison between the PDR reward and the initial composite reward. Please rephrase for clarity.","section":"Abstract"},{"comment":"The condition \"if ϵ ≥ N(0, 1)\" uses the normal-distribution notation in what appears to be an epsilon-greedy exploration check. This is likely a typographical error; the intended condition is probably a uniform random draw compared against epsilon. Please correct the pseudocode.","section":"Algorithm 1, line 10"},{"comment":"The definitions of MSX+ and MSX- would benefit from a worked example or a small illustrative figure. The set notation in Eqs. (10)-(12) is dense, and the accompanying text does not fully explain how the sets X+ and X- are constructed in practice.","section":"Section III-F"},{"comment":"The potential-shaping function ϕ(x, y, z) appears to have three arguments in its definition but is later evaluated as ϕ(x, y) with two arguments, and the text states \"we assume y = z_norm.\" Please align the notation to avoid confusion.","section":"Section V-C, Eq. (23)"},{"comment":"The architecture diagrams are very dense and the small font makes them difficult to read, especially the multi-headed variants in Fig. 4. Consider enlarging the panels or separating them into multiple figures for the camera-ready version.","section":"Figures 2-4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the algorithmic contributions are potentially useful. The main obstacle to publication is the lack of support for the central interpretability claim: the PDR-primary conclusion rests on comparing Q-values of reward components with different hand-set scales, and the adaptive weights and theoretical theorem inherit the same issue. If the authors add controlled ablations or re-frame the claim as an observation about the chosen reward parameterization, the paper could become acceptable. I would not reject on the basis of current consensus; the issue is internal evidence, not disagreement with a consensus view."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering paper that does useful work—extending reward decomposition to VDN/QMIX/QTRAN, sharing layers via multi-headed architectures, introducing the RDFX metric, and showing consistent KPI gains in a simulated 5G XR scenario. The PDR-importance conclusion, however, is not established; it likely reflects hand-tuned reward scales rather than causal contribution.\n\nWhat's new: the architectural combination (reward decomposition + VFF MARL + multi-headed shared layers) is a natural but nontrivial extension, and the RDFX metric is a sensible adaptation of RDX to ordered action spaces. The adaptive weighting via reward statistics is an interesting idea. The empirical work is reasonably thorough across algorithms and distances, and the gains over the APS baseline are plausible.\n\nSoft spots: the biggest issue is the abstract's claim that 'the Packet Delivery Ratio reward is the primary contributor to optimal performance.' Q1 (delay), Q2 (PDR), and Q3 (data rate) are learned against hand-scaled reward functions in different units: delay is a staircase capped at 1, PDR is {1, 0.5, 0}, DRR is a continuous [0,1] ratio. Comparing their magnitudes is comparing apples and oranges—higher Q2 may simply mean the PDR reward was assigned near-1 values more often. The adaptive weights in Eq. (35) are computed from the same reward statistics, so they cannot independently confirm PDR primacy. No counterfactual ablation (e.g., removing PDR from the reward) is provided. The theorem in Appendix A is also fragile: Eq. (39) assumes a decomposable transition, which is false for a shared wireless channel, and the actual team rewards are minima over agents, not the additive form assumed in Definition A.1. So the theory does not back the conclusion. Minor but relevant: no error bars or multiple seeds, and no non-decomposed VDN/QMIX/QTRAN baselines.\n\nBottom line: worth serious review. The engineering contributions stand on their own; the explainability story needs either toning down or real support—normalization, counterfactual analysis, or reward-scale robustness checks. For readers interested in XR codec adaptation or MARL reward design, the paper is informative. For the claim that PDR is causally primary, treat it as a hypothesis, not a finding.","headline":"Useful engineering extension of reward decomposition to MARL, but the headline 'PDR is primary' claim is scale-dependent and not yet supported.","tokens_in":30933,"tokens_out":2605,"would_cite":false,"duration_ms":25243,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that decomposing rewards in value-function-factorization MARL reveals the packet delivery ratio as the primary driver of optimal XR codec adaptation, and that an adaptive multi-headed QMIX variant exploiting this insight…","keywords":["Explainable multi-agent reinforcement learning","Reward decomposition","Value function factorization","XR codec adaptation","5G/6G networks","Quality of experience","RDFX"],"falsifier":"Retrain the same agents after swapping the reward scales, for example making the delay reward finely graded and the PDR reward coarse; if the relative Q-value ranking flips while network KPIs stay the same, the 'PDR is primary' conclusion is an artifact of reward scaling rather than an intrinsic property of the task.","tokens_in":29788,"feed_emoji":"📡","tokens_out":6771,"duration_ms":61086,"temperature":0.7,"pith_summary":"This paper tries to establish that reward decomposition can make multi-agent reinforcement learning explainable in a wireless setting, and that the resulting explanations are not just descriptive but useful for improving policy performance. Focusing on extended reality (XR) codec adaptation over 5G, the authors split the joint reward into delay, packet delivery ratio (PDR), and data-rate ratio components and modify three value-function-factorization algorithms, VDN, QMIX, and QTRAN, to output separate Q-values for each component. They claim that the PDR reward is the primary contributor to optimal performance compared with the initial composite reward that also included delay and data-rate terms. They also introduce a multi-headed architecture to cut computation, a metric called RDFX for explaining codec parameter choices, and an adaptive weighting scheme that uses reward means and variances during training. If these claims hold, network operators can use the decomposition both to explain why a codec agent chooses an action and to prune or reweight reward components accordingly.","feed_headline":"PDR reward, not delay, drives XR codec MARL policy","feed_subtitle":"Decomposed Q-values show packet delivery ratio dominates; MHA-QMIX cuts packet loss 67.9% vs baseline.","key_machinery":"The machinery is reward decomposition carried through value function factorization: each agent outputs C separate action-value heads, one per reward component, and these are combined per component into joint Q-values via additive summation for VDN or per-component mixing networks for QMIX and QTRAN. This rests on Theorem 1, which states that the joint action-value function factorizes as a weighted sum of per-agent, per-component Q-functions. Multi-headed shared layers reduce the number of networks, while RDFX defines the difference between the chosen action's Q-value and the expected Q-value of increasing or decreasing the codec parameter, and the adaptive loss uses weights computed from the mean and variance of each episode's reward sequence, rebalancing gradient training rates.","core_discovery":"On the paper's own terms, the central discovery is that a composite reward in value-function-factorization MARL can be decomposed into per-component Q-functions, and the resulting per-component values carry actionable information about reward design. Across agents, distances, and algorithms, the PDR component Q2 has the highest Q-value, with delay second and data-rate ratio lowest; MSX analysis often requires only the PDR component to explain the chosen action. The authors leverage this observation online with adaptive weights, giving the best algorithm, MHA-QMIX, average gains over the Adjust Packet Size baseline of up to 10.7% in XR index, 41.4% in jitter, 33.3% in delay, and 67.9% in packet loss ratio.","pith_inferences":["Inference: The paper's conclusion that PDR is primary rests on comparing Q-values whose reward scales are hand-chosen, so a normalized-reward ablation would determine whether the conclusion is about the environment or about the reward encoding.","Inference: A direct test of the explanation is to train a PDR-only variant and compare it against the composite reward; the decomposition predicts comparable KPI performance with fewer reward components.","Inference: The adaptive weighting via mean-plus-sinh-variance may be the more general contribution, since it is a parameter-free way to balance multiple reward terms online and could apply beyond MARL to any multi-objective reinforcement learning loss.","Inference: If deployed online, per-episode reward-importance weights could interact with off-policy replay because the loss surface changes as the weights move; the paper does not analyze this stability question."],"forward_implications":["Reward designers for XR codec adaptation should treat PDR as the dominant objective, with delay secondary and data-rate ratio least important, if the decomposition reflects true contribution.","The adaptive mechanism can replace post-hoc explainability: reward importance weights computed during training can be fed back into the loss without a separate explanation phase.","Multi-headed architectures give comparable or better KPIs than vanilla decomposed versions while reducing the number of networks, so explainability need not cost extra inference structures.","The decomposition framework and RDFX metric carry over to other wireless parameter-selection problems where actions are ordered values rather than discrete semantic actions.","Benchmarking against the Adjust Packet Size baseline at 400 m shows the largest gains, suggesting the biggest benefit appears in path-loss-limited conditions."],"supporting_citations":[{"why":"Defines reward decomposition into per-component Q-functions, the formalism the paper extends to multi-agent value-function-factorization algorithms.","marker":"[13]"},{"why":"Introduces VDN and the additive factorization assumption that becomes the base for the decomposed VDN variant.","marker":"[30]"},{"why":"Introduces QMIX with monotonic mixing, which the paper extends to per-reward mixing networks.","marker":"[32]"},{"why":"Introduces QTRAN and its joint and state-value factorization losses, adapted here for reward decomposition.","marker":"[31]"},{"why":"Defines the Adjust Packet Size baseline that the proposed algorithms are compared against in the KPI evaluations.","marker":"[15]"},{"why":"Supplies the gradient-balancing idea used in the adaptive loss weighting scheme.","marker":"[48]"},{"why":"Defines the XR Quality Index used both in the composite reward and as a KPI for evaluating the policies.","marker":"[45]"},{"why":"Provides the extended reality and cloud gaming traffic characteristics used to configure the simulations.","marker":"[44]"},{"why":"Earlier MARL codec adaptation work whose composite reward and setting this paper decomposes and extends.","marker":"[17]"}],"fun_headline_variants":["Reward decomposition reveals PDR drives XR codec MARL","MHA-QMIX cuts packet loss 67.9% with decomposed rewards","Explainable MARL: Packet delivery ratio dominates codec choice","PDR reward, not delay, steers MARL for XR codecs","Decomposed Q-values: PDR key to XR codec adaptation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the three decomposed Q-value heads are directly comparable in magnitude, so the largest one marks the truly most important reward rather than merely the reward with the most favorable hand-chosen scale, and that the multi-agent transition function splits additively across agents as assumed in the theorem's proof.","fun_headline_variants_meta":{"raw":{"variants":["Reward decomposition reveals PDR drives XR codec MARL","MHA-QMIX cuts packet loss 67.9% with decomposed rewards","Explainable MARL: Packet delivery ratio dominates codec choice","PDR reward, not delay, steers MARL for XR codecs","Decomposed Q-values: PDR key to XR codec adaptation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000998,"raw_usage":{"total_tokens":4273,"prompt_tokens":1042,"completion_tokens":3231,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":3147}},"tokens_in":658,"tokens_out":3231,"duration_ms":22086,"temperature":1.0,"reasoning_tokens":3147,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:21:43.820208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same agents after swapping the reward scales, for example making the delay reward finely graded and the PDR reward coarse; if the relative Q-value ranking flips while network KPIs stay the same, the 'PDR is primary' conclusion is an artifact of reward scaling rather than an intrinsic property of the task.","supporting_citations":[{"cited_title":"Explainable reinforcement learning via reward decomposition,","cited_arxiv_id":null,"evidence_quote":"Defines reward decomposition into per-component Q-functions, the formalism the paper extends to multi-agent value-function-factorization algorithms."},{"cited_title":"Monotonic Value Function Factorisation for Deep Multi- Agent Reinforcement Learning,","cited_arxiv_id":null,"evidence_quote":"Introduces QMIX with monotonic mixing, which the paper extends to per-reward mixing networks."},{"cited_title":"QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning,","cited_arxiv_id":null,"evidence_quote":"Introduces QTRAN and its joint and state-value factorization losses, adapted here for reward decomposition."},{"cited_title":"Enhancing 5G QoS Management for XR Traffic Through XR Loopback Mechanism,","cited_arxiv_id":null,"evidence_quote":"Defines the Adjust Packet Size baseline that the proposed algorithms are compared against in the KPI evaluations."},{"cited_title":"Gradnorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-balancing idea used in the adaptive loss weighting scheme."},{"cited_title":"XR Quality Index: Evaluating RAN Transmission Quality for XR services over 5G and beyond,","cited_arxiv_id":null,"evidence_quote":"Defines the XR Quality Index used both in the composite reward and as a KPI for evaluating the policies."},{"cited_title":"Study on XR (Extended Reality) Evaluations for NR,","cited_arxiv_id":null,"evidence_quote":"Provides the extended reality and cloud gaming traffic characteristics used to configure the simulations."},{"cited_title":"Extended Reality (XR) Codec Adaptation in 5G using Multi-Agent Reinforcement Learning with Attention Action Selection,","cited_arxiv_id":null,"evidence_quote":"Earlier MARL codec adaptation work whose composite reward and setting this paper decomposes and extends."}],"review_version":1}