{"id":"978c85c0-5008-442e-ae81-ef5242d344a1","arxiv_id":"2412.16079","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper proposes DSWM and ASWM, two Stackelberg-game-inspired algorithms that reweight node contributions in federated learning, and reports modest AUC gains for small-data nodes on three MedMNIST datasets.","lead":"The paper models distributed learning as a three-player Stackelberg game and introduces two algorithms, DSWM and ASWM, that reweight each node's contribution to the global model each round. On three medical image datasets, the adaptive version reports higher AUC for small-data nodes at a small cost to the large-data leader.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported ASWM AUC gains may be an artifact of selecting contribution weights on local test sets (Algorithm 1 lines 8, 15-16; Section 3.2), leaking evaluation data into training and invalidating the headline 2.713% improvement.","rationale":"The paper's central claim is a quantitative result: ASWM improves underrepresented nodes' AUC by 2.713% with only 0.441% decrease for larger nodes. My stress-test focuses on the experimental protocol rather than the game-theoretic framing. The decisive flaw is that contribution weights are selected and trained using local test-set losses, making the reported AUC a selected statistic. This is the same concern the reader identified, and I agree. The concern is load-bearing because the entire mechanism of ASWM is the learned weight; if the weights are fitted to the test set, the method's apparent fairness gain does not generalize. I considered whether other issues—lack of code, absent standard deviations, no significance tests—could be more central, but they are secondary; the leakage alone is sufficient to invalidate the quantitative claim. The paper does include a limitation statement about sequential updates, but that concerns scalability, not the test-set leakage. There is no formal verification or released code that could independently support the numbers. Therefore the verdict should remain REJECT, and no change to the reader's verdict is needed. The proposed concrete test, re-running with a separate validation split, would settle whether the reported gain is real. If the authors can show the gap persists without test-set leakage and with variance reported, the claim could be reconsidered.","tokens_in":10828,"tokens_out":4188,"duration_ms":34764,"concrete_test":"Re-run the three MedMNIST experiments with the same data splits and 10-seed protocol, but modify Algorithm 1 so the losses used for weight selection (lines 8 and 15-16) are computed on a held-out validation split (e.g., 10% of each node's local data), while the reported AUC is computed only on an untouched test split. Then compare ASWM vs PWFedAvg on the test split. If the mean ASWM improvement is no longer about 2.7% or its bootstrap 95% CI includes zero, the headline gain was an artifact of selecting weights on the test set. Also report per-run AUCs and standard deviations; without those, the magnitude of any real effect cannot be assessed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—ASWM improves underrepresented nodes by 2.713% AUC at a cost of 0.441% for larger nodes—rests on the weight-selection mechanism described in Section 3.2 and Algorithm 1. In DSWM, the contribution weight is chosen from C={0.1,...,1.0} as the one that minimizes the node's loss; in ASWM, a neural network is trained via experience replay to predict weights that minimize that same loss. The paper repeatedly specifies that this loss is computed on the node's local test set: the Stackelberg formulation in Section 2 defines L_0, L_1, L_2 as losses on local test sets; Algorithm 1 lines 8 and 15-16 store losses computed on DL, DF1, DF2; and Section 3.2 states that the input vectors include utility functions 'as loss functions on local test sets.' The reported AUC values are measured on those same test sets. Consequently, the weight selection and the policy training are directly optimizing the evaluation metric. The observed 2.713% average improvement is thus a fitted quantity, not an unbiased estimate of the method's performance on unseen data. This is the paper's central claim, so the result is unsupported as reported. The concern is not merely about missing significance tests or variance; even with 10 runs, the leakage would bias all runs in the same direction. Note that if the authors intended a validation split, the algorithm description is inaccurate and must be corrected; as written, it is test-set leakage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper frames distributed machine learning with imbalanced data as a three-player Stackelberg evolutionary game, with a leader node and two follower nodes. It proposes two algorithms, DSWM and ASWM, which set per-round contribution weights in the federated averaging update. The methods are evaluated on BreastMNIST, DermaMNIST, and BloodMNIST in a simulated three-node setup, and the authors report that ASWM improves underrepresented (follower) nodes by about 2.713% AUC on average over PWFedAvg while decreasing the leader node's AUC by only about 0.441%. The central quantitative claim is that ASWM improves fairness for small-data nodes with only a modest cost to the large-data node.","tokens_in":11225,"tokens_out":3834,"duration_ms":36150,"significance":"If the reported results were obtained with an unbiased evaluation protocol, the paper would offer a practical and useful mechanism for improving fairness in federated learning with imbalanced medical data. The idea of treating contribution-weight selection as a Stackelberg game is reasonable, and the use of three public medical datasets with simulated imbalance is a strength. However, the central empirical claim is currently undermined by a direct leakage of test-set information into the weight-selection and policy-training procedures, and the paper does not report any uncertainty estimates despite stating that experiments were repeated ten times. With the protocol as written, the 2.713% improvement cannot be interpreted as an unbiased estimate of the method's performance on unseen data. The paper would need a corrected evaluation protocol and substantially more statistical rigor before its claims can be accepted.","major_comments":[{"comment":"The paper's central quantitative claim is invalidated by test-set leakage. The loss functions in Eqs. (1)-(3) are explicitly defined as losses on the local test sets of the leader and followers. DSWM is described in Section 3.2 as selecting the contribution weight c that yields the lowest loss, and Algorithm 1 lines 8 and 15-16 compute UL and UFi on the local datasets DL and DFi and then store them in the experience replay buffer used to train the policy networks at line 19. The reported AUCs in Tables 2-4 are measured on those same test sets. Therefore, both DSWM and ASWM directly optimize the evaluation metric during weight selection and policy training, so the reported 2.713% average improvement is a fitted quantity rather than an unbiased estimate of performance on unseen data. The experiments must be redone with a separate validation split for weight selection and policy training, with the test sets used exactly once at the end.","section":"Section 2, Section 3.2, Algorithm 1"},{"comment":"There is an internal contradiction about which data are used for loss evaluation. The paragraph after Algorithm 1 says that 'the leader applies this weight to the model updates and then evaluates the resulting performance on a validation set,' but Section 2 and the algorithm's input and data-flow description identify the losses as being computed on local test sets. If a validation split was in fact used, the algorithm description is incorrect and must be corrected; if not, the test-set leakage is confirmed. Either way, the current text does not support a valid evaluation protocol.","section":"Section 3.2, Algorithm 1"},{"comment":"The claim that ASWM 'significantly' favors underrepresented nodes is not supported by any statistical evidence. The experiments are stated to be repeated ten times in Section 3.3, but Tables 2-4 report only point estimates of AUC with no standard deviations, confidence intervals, or paired tests. Differences of 0.01-0.03 AUC could easily be within run-to-run variability, especially given the small number of repetitions. The paper must report mean and variance across runs and, preferably, paired comparisons against PWFedAvg.","section":"Section 4, Tables 2-4"},{"comment":"The game-theoretic formulation is asserted but not actually used. The followers' problems in Eqs. (2) and (3) are mutually recursive, and no existence, uniqueness, or computation of the claimed Stackelberg equilibrium is provided. Algorithm 1 is described as an actor-critic method with experience replay, but no connection is shown between the algorithm's weight updates and the Stackelberg equilibrium defined in Section 2. This mismatch weakens the central framing of the paper; the authors should either provide a formal analysis of the game or substantially soften the claims that the algorithm implements a Stackelberg equilibrium.","section":"Section 2, Eqs. (2)-(3), Section 3.2"}],"minor_comments":[{"comment":"There are grammatical errors such as 'an Stackelberg evolutionary game' in the abstract and 'a inspiration' in the conclusion; these should be corrected before resubmission.","section":"Abstract, Section 5"},{"comment":"The notation for contribution weights is inconsistent: Section 2 introduces C0, C1, C2, while Section 3.1 uses Ck and Algorithm 1 uses C = {cF1, cF2}. The leader's own contribution weight is never clearly specified in the global update formula, so it is unclear how the leader's weight is set.","section":"Section 3.1"},{"comment":"The experimental setup omits many details needed for reproducibility, including model architectures, local training epochs, learning rates, number of communication rounds T, the Dirichlet concentration parameter, and the Gaussian noise parameters. The authors state that class distributions were generated using a Dirichlet model, but no code or data-generation script is provided.","section":"Section 3.3"},{"comment":"Figure 1 is described in the text only briefly; panels (c) and (d) are said to show contribution-weight selection for ASWM and DSWM, but the axes and the meaning of individual curves are not explained in the caption or the text.","section":"Figure 1"},{"comment":"Some references appear to be cited for claims they do not directly support, such as a course note on the Dirichlet distribution [19] being used as the methodological basis for the data-generation model; a more standard statistics textbook reference would be appropriate.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The test-set leakage is the decisive issue: the headline empirical result is currently an artifact of the evaluation protocol. The defect is fixable in principle by rerunning the experiments with a proper validation split and reporting uncertainty, but the current version should not be accepted. I found no evidence of deliberate misconduct; the description appears to be an honest but flawed protocol. The fit to the journal is acceptable if the claims can be re-established under a sound evaluation scheme."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the headline 2.713% AUC gain for underrepresented nodes is a fitted number, not an unbiased result. Section 3.2 and Algorithm 1 compute the loss used for weight selection on DL, DF1, DF2, and the text explicitly calls these local test sets. Both DSWM and ASWM optimize weights against those same test sets, and the AUC table is measured on the same sets. So the improvement over PWFedAvg is exactly what you'd expect from selecting weights on the test set; it says nothing about how the method would behave on unseen data. The stress-test note is right, and it is not a minor footnote. It is the central quantitative claim.\n\nThat said, the paper isn't worthless. Framing federated reweighting as a Stackelberg game with one leader and two followers is a reasonable and mildly fresh way to think about node contribution. The two algorithms are concrete, and the DSWM grid search over C plus the ASWM policy network are clearly specified. The comparison to FedAvg and PWFedAvg on three MedMNIST datasets is appropriate for a first look, and the motivation (small clinics get outperformed by large ones) is real. I'd credit the authors for making the mechanism explicit enough that the leakage is easy to spot.\n\nThe soft spots beyond the leakage: no code, no variances or significance tests, and only three nodes. The 'evolutionary' part is mostly experience replay; the evolutionary game theory isn't really used. The paper even says 'validation set' in one sentence in Section 3.2, which might mean the authors intended a validation split; if so, the algorithm description is wrong and must be corrected. As written, it is test-set leakage.\n\nWho is this for? Someone thinking about fairness-oriented federated aggregation might want to read the Stackelberg formulation, but they should not cite the empirical results. The paper deserves a serious referee only because the core idea is plausible and the flaw is fixable: replace test losses with validation losses, add variances, release code. If the authors fix that, there could be a real contribution.\n\nMy recommendation: if this lands on my desk, I'd send it to peer review with a strong note about the evaluation protocol, not desk reject. The idea is worth a revision, but the current numbers should not be trusted.","headline":"The 2.713% AUC gain is an artifact of selecting weights on local test sets; the Stackelberg idea is plausible but the empirical claim is unsupported.","tokens_in":11686,"tokens_out":3588,"would_cite":false,"duration_ms":33131,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By modeling distributed learning as a Stackelberg evolutionary game and reweighting each node's contribution every round, this paper claims that underrepresented nodes gain 2.713% average AUC while the largest node loses only 0.441%.","keywords":["federated learning","decentralized machine learning","Stackelberg game","evolutionary game theory","data imbalance","non-IID data","contribution weighting","AUC"],"falsifier":"Split each node's data into training, validation, and a never-touched test set, and let ASWM select contribution weights using validation loss instead of test-set loss; if the held-out AUC improvement over PWFedAvg falls below the reported 2.713% or reverses, the central claim fails. A simpler check on DSWM: if the weight chosen by lowest test loss is not the weight with lowest held-out loss in repeated Dirichlet draws, the selection rule is overfitting the evaluation set.","tokens_in":10648,"feed_emoji":"⚖️","tokens_out":8973,"duration_ms":67906,"temperature":0.7,"pith_summary":"Distributed machine learning suffers when participants hold very unequal datasets: the node with the most data dominates the shared model, and smaller nodes end up with worse predictions. This paper argues that the imbalance is a strategic, sequential interaction and models it as a Stackelberg evolutionary game, with the large-data node as leader and smaller nodes as followers. It introduces two algorithms, DSWM and ASWM, that replace the usual data-size weights in federated averaging with per-round contribution weights chosen to minimize each player's loss. On three medical image datasets, the adaptive version raises the Area Under the Curve (AUC) of underrepresented nodes by an average of 2.713% compared to precision-weighted federated averaging, while the leader loses only 0.441% on average. If the result holds, dynamic game-theoretic weighting is a practical way to make distributed medical learning fairer without centralizing data.","feed_headline":"Game-theoretic weighting lifts small-data nodes' AUC by 2.7%","feed_subtitle":"Adaptive Stackelberg weighting costs larger nodes only 0.44% AUC, making distributed medical learning fairer.","key_machinery":"The load-bearing object is the contribution weight $C_k$ that replaces the data count $n_k$ in the federated averaging update, together with the Stackelberg equilibrium procedure that selects it: the leader minimizes $L_0(C_0,C_1^*(C_0),C_2^*(C_0))$ and each follower minimizes its own loss given earlier choices. DSWM implements this by brute-force grid search over ten weights; ASWM replaces the search with two neural networks whose inputs include previous contribution weights, previous global weights, and local test-set losses, trained through experience replay. This mechanism is what lets a small node amplify its own update when its loss is high, counterbalancing the leader's data advantage.","core_discovery":"The paper's central claim is that the performance gap between large- and small-data nodes in distributed learning is not fixed: it can be reshaped by letting each node strategically choose how much influence its update has on the global model. Formally, the authors rewrite federated averaging as $w_{t+1} = \\sum_k C_k w^k_t / \\sum_k C_k$, replacing dataset sizes $n_k$ with contribution weights $C_k$, and define a Stackelberg game where the leader picks weights first, anticipating the followers' best responses. DSWM searches a fixed grid $C = \\{0.1, 0.2, \\dots, 1.0\\}$ each round and selects the weight with the lowest local loss; ASWM trains neural networks with experience replay to predict those weights. Across BreastMNIST, DermaMNIST, and BloodMNIST simulations with one leader and two followers, ASWM consistently yields the highest AUC on the smallest nodes, and the aggregate improvement over PWFedAvg is the reported 2.713% with a 0.441% average decrease for the large node.","pith_inferences":["A deployment-grade version would need a privacy-safe source for the utility losses; if nodes cannot reveal local test-set losses without leaking information, the mechanism may require secure aggregation or a trusted coordinator.","The average 2.713% gain is uneven across datasets and nodes, so per-seed intervals and worst-case analysis would show whether the benefit is reliable or driven by one favorable configuration.","The sequential leader-follower updates likely incur extra synchronization compared with simultaneous FedAvg; a wall-clock and communication-cost comparison would test whether the fairness gain survives efficiency constraints.","A natural stress test is to vary the number of followers and the Dirichlet concentration parameter to map where ASWM's advantage over PWFedAvg disappears; the paper's data-generation setup makes this a direct next experiment."],"forward_implications":["The smallest nodes gain most: Node 3 improves by roughly 5.97% on DermaMNIST, 2.02% on BloodMNIST, and 1.20% on BreastMNIST relative to PWFedAvg, so the method targets exactly the participants who are currently worst off.","The leader's cost is bounded: its AUC decreases by at most about 1.34% in these experiments, with an average drop of 0.441%, so adopting ASWM does not require sacrificing the large node's model.","Because the weighting replaces only the scalar coefficients in federated averaging, the scheme can be inserted into existing distributed training loops without centralizing data or changing local model architectures.","The Stackelberg formulation generalizes to $n$ leaders and $m$ followers, so the same per-round weighting logic is not limited to the three-node simulations tested here."],"supporting_citations":[{"why":"Supplies the FedAvg baseline and the non-IID convergence setting that the proposed methods must beat.","marker":"[17]"},{"why":"Supplies the PWFedAvg baseline; all reported AUC improvements are measured against it.","marker":"[23]"},{"why":"Provides the Stackelberg strategy definition used to formalize leader-follower contribution weighting.","marker":"[25]"},{"why":"Provides the Stackelberg evolutionary game framework that justifies dynamic per-round adaptation.","marker":"[26]"},{"why":"Supplies the experience replay mechanism used in training the contribution-weight predictors.","marker":"[8]"},{"why":"Supplies the Dirichlet distribution used to generate imbalanced node data in the experiments.","marker":"[19]"}],"fun_headline_variants":["Stackelberg weighting lifts small-data nodes' AUC by 2.7%","ASWM improves underserved nodes' AUC 2.7%; big nodes lose 0.44%","Fairer federated learning via Stackelberg game weighting","ASWM boosts small-node AUC 2.7%, big nodes drop 0.44%","Stackelberg evolutionary game narrows AUC gap in federated learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The protocol assumes that choosing contribution weights by minimizing each node's loss computed on its local test set does not leak evaluation information and that the chosen weights generalize to unseen data; if the weights are overfitted to those test sets, the reported AUC gains are artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Stackelberg weighting lifts small-data nodes' AUC by 2.7%","ASWM improves underserved nodes' AUC 2.7%; big nodes lose 0.44%","Fairer federated learning via Stackelberg game weighting","ASWM boosts small-node AUC 2.7%, big nodes drop 0.44%","Stackelberg evolutionary game narrows AUC gap in federated learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000787,"raw_usage":{"total_tokens":3485,"prompt_tokens":969,"completion_tokens":2516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":2411}},"tokens_in":585,"tokens_out":2516,"duration_ms":16086,"temperature":1.0,"reasoning_tokens":2411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:48:28.024354+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split each node's data into training, validation, and a never-touched test set, and let ASWM select contribution weights using validation loss instead of test-set loss; if the held-out AUC improvement over PWFedAvg falls below the reported 2.713% or reverses, the central claim fails. A simpler check on DSWM: if the weight chosen by lowest test loss is not the weight with lowest held-out loss in repeated Dirichlet draws, the selection rule is overfitting the evaluation set.","supporting_citations":[{"cited_title":"Simaan and J","cited_arxiv_id":null,"evidence_quote":"Provides the Stackelberg strategy definition used to formalize leader-follower contribution weighting."},{"cited_title":"Stackelberg evolutionary game theory: how to manage evolving systems","cited_arxiv_id":null,"evidence_quote":"Provides the Stackelberg evolutionary game framework that justifies dynamic per-round adaptation."},{"cited_title":"On the dirichlet distribution","cited_arxiv_id":null,"evidence_quote":"Supplies the Dirichlet distribution used to generate imbalanced node data in the experiments."}],"review_version":1}