{"id":"0f9c1f0a-5932-44ec-bbee-248c1e73f9c0","arxiv_id":"2506.16731","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"The paper designs a federated learning incentive scheme that pays agents based on the total variation distance of their label distribution from an IID reference, and claims this induces high-quality participation.","lead":"Federated learning agents often have no incentive to share high-quality data, so this paper proposes a payment mechanism that rewards agents for reducing the statistical distance between their local data distribution and a reference distribution.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Payment_k in §5.1 requires each agent's private non-IID degree δ_k, but the server only receives model updates and no estimation or reporting protocol is given; the mechanism cannot be evaluated, so the central equilibrium and implementability claims are unsupported.","rationale":"The paper's headline contribution is a practical incentive mechanism, not just a convergence bound. The convergence analysis (Theorem 2) and the generalization-gap bound (Theorem 3) are plausible and could be useful even if the mechanism fails; the TV-distance reformulation is a legitimate theoretical exercise. But the mechanism layer depends on evaluating Payment_k, and the explicit definition of Payment_k is a function of δ_k and δ_k'. Those quantities are private by the federated-learning premise: the server receives model updates, not label histograms. The paper contains no inference procedure and no formal reporting game in which δ_k is a report subject to the peer-prediction scoring. Therefore the claimed pure Nash equilibrium (Theorem 5) and truthful-reporting property are properties of a game whose payoff function the principal cannot compute; as stated, the scheme is not a mechanism. This is more basic than the flawed derivative argument in Lemma 9 or the hand-picked Φ/Υ ranges: even a repaired concavity proof or better hyperparameter analysis would not make the payment evaluable. The reader's weakest assumption identifies the same gap, and I agree with the REJECT verdict: the central mechanism is unimplementable as written. I do not see the gap as an internal inconsistency in the convergence algebra, but as a missing ingredient (an estimator or a report/verification protocol) that is load-bearing for the paper's stated application. The concrete test above makes the gap visible: when the server must compute payments without the hidden δ values, the computation is impossible, and the claimed experiments are circular because they use the constructing δ values in the payment formula.","tokens_in":28472,"tokens_out":6149,"duration_ms":66790,"concrete_test":"Run the Section 6 experiment with the server's input restricted to model updates and D_c, removing the true δ_k values used to generate client datasets. Require the server to compute Payment_k from §5.1 without accessing those values. If no explicit estimator of δ_k from updates is supplied and the computation halts, the mechanism is unimplementable; if the authors provide such an estimator, test it for unbiasedness on all four datasets and compare the resulting utilities to Figure 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing gap is that the payment rule defined in Section 5.1 is not computable from server-visible information. Payment_k(e_k,e'_k) is defined as f(Q/(Φδ_k^2(e_k)+Φδ_k'^2(e'_k)+Υ)), where δ_k is the total-variation distance between agent k's private label distribution p^(k) and the reference p^(c) (Eq. 1). The server's only data are the uploaded models and an auxiliary/validation set D_c (Section 4.2); it never observes p^(k). Section 5.1 asserts that 'the learner can evaluate the value of δ_k', but no estimator from gradients/updates and no report-and-verify protocol is provided. The experiments do not repair this: Section 6 constructs clients with known δ values and then feeds those same values into the payment formula, so the evaluation is circular for the mechanism itself. If the intended payment were instead based on the observed validation gap F_c(w_k)-F_c(w_k'), that is a different payment rule than the one analyzed, and the equilibrium analysis would need to be re-derived; Theorem 3's upper bound cannot serve as the payment because it is an inequality involving unobserved δ terms. Because payments cannot be evaluated, agents cannot anticipate their utility and the two-stage game in Section 5.2 is not a well-defined mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an incentive mechanism for federated learning in which payments reward agents for reducing the distance between their local label distribution and a reference distribution. The authors reformulate the convergence bound for FedAvg using a total-variation-based non-IID degree (Theorem 2), derive an upper bound on the generalization-loss gap between any two agents (Theorem 3), and use that bound to define a payment function in Section 5.1. They model the interaction as a two-stage Stackelberg game and claim existence of a pure Nash equilibrium (Theorem 5). Experiments on MNIST, FashionMNIST, CIFAR-10, and CIFAR-100 illustrate the effect of non-IID degree on accuracy and show utility curves that are claimed to confirm the theoretical equilibrium.","tokens_in":28802,"tokens_out":8978,"duration_ms":87413,"significance":"If the mechanism were implementable, it would address a real gap in the FL incentive literature, where most schemes reward sample quantity rather than data quality or heterogeneity. The convergence-bound reformulation in Theorem 2 is a useful conceptual link, and the idea of using peer prediction to score generalization gaps is creative. The paper also provides fairly detailed proof sketches. However, the central payment rule is not computable from information the server actually holds, and the equilibrium proof contains a derivative error. These are not presentation issues; they block the main claims. As it stands, the contribution is a theoretical skeleton whose mechanism cannot be instantiated or tested, and the experimental validation is circular because the payment formula is fed the same private δ values it would need to observe.","major_comments":[{"comment":"The payment function is defined as Payment_k(e_k,e'_k) = f(Q/(Φδ_k^2(e_k)+Φδ_{k'}^2(e_{k'})+Υ)), where δ_k is the total-variation distance between agent k's private label distribution p^{(k)} and the reference p^{(c)} defined in Eq. (1). The learner only receives uploaded models and an auxiliary set D_c (Section 4.2); it never observes p^{(k)}. The statement that 'the learner can evaluate the value of δ_k' is unsupported, and no estimator from model updates or report-and-verify protocol is given. Consequently, the payment cannot be computed, agents cannot anticipate a well-defined utility, and the two-stage game in Section 5.2 is not a well-defined mechanism as stated. The experiments in Section 6 do not repair this: they construct clients with known δ values and feed those values into the same payment formula, so the utility curves in Figure 2 are true by construction rather than evidence that the mechanism is implementable.","section":"Section 5.1, payment function after Eq. (2)"},{"comment":"Lemma 9 claims that ∂f(e_k,e_{k'})/∂e_{k'} is the same as ∂f(e_k,e_{k'})/∂e_k. For the payment f(Q/(Φδ_k^2(e_k)+Φδ_{k'}^2(e_{k'})+Υ)), the derivative with respect to e_k depends on δ_k and δ'_k through Φδ_k^2, while the derivative with respect to e_{k'} depends on δ_{k'} and δ'_{k'}; these are not equal unless the two agents have identical effort-distance functions and effort levels. The bounds d1_{k'} and d2_k asserted in the proof therefore do not follow. Since the 'well-behaved' condition (Definition 3) is what Theorem 5 relies on, the existence proof for a pure Nash equilibrium is incomplete.","section":"Appendix A.10, Lemma 9"},{"comment":"Theorem 4's statement says the optimal effort level e*_k is 1, but the displayed definition gives e*_k = 0 or Ŷe_k, not 1. In addition, the proof's first-order condition, ∂f(Ŷe_k,e_{k'})/∂δ_k + c d'(δ_k(Ŷe_k)) = 0, omits the chain-rule factor dδ_k/de_k and the sign of the derivative of the cost term c·d(|δ_k(0)-δ_k(e_k)|) that appears in Eq. (11). As written, the condition does not follow from the utility derivative. This affects the best-response characterization used in the backward-induction argument.","section":"Theorem 4 and its proof (Section 5.2, Appendix A.9)"},{"comment":"Theorem 2 states the bound as 16(E−1)G^2η_t^2(1+2η_tL)^{2(E−1)}Ψ with Ψ = ∑_k p_k (∑_i |p^{(k)}(y=i)−p^{(c)}(y=i)|)^2 = 4δ_k^2, which equals 64(E−1)G^2η_t^2(1+2η_tL)^{2(E−1)}∑_k p_k δ_k^2. Example 1 instead writes B = 16(E−1)G^2η_t^2(1+2η_tL)^{2(E−1)}∑_k p_k δ_k^2 and labels it 'Upper bound shown in Theorem 2'. These two expressions differ by a factor of 4, so the simplified bound used in the learner's payoff function is not justified by Theorem 2 as stated.","section":"Theorem 2 versus Example 1"},{"comment":"Both Theorem 2 and Lemma 6 rely on the assumption that the per-class Lipschitz constants are equal, L = L_{x|y=i} for all i and i', but this assumption only appears inside the proofs in Appendix A.2 and A.4 and is not listed among Assumptions 1–4 in Section 3.1. If per-class Lipschitz constants differ, the displayed bounds do not follow. The assumption should be promoted to a numbered assumption in the main text, with a discussion of which loss functions satisfy it.","section":"Assumptions in Theorem 2 and Lemma 6"}],"minor_comments":[{"comment":"The quantity in Eq. (1) is the total variation distance (half the L1 distance between label distributions), not the Wasserstein distance. The terminology should be corrected throughout, or the Wasserstein formulation should actually be used and the proof adjusted accordingly.","section":"Eq. (1) and throughout"},{"comment":"The notation is inconsistent: the payment is defined as Payment_k(e_k,e'_k) but then written as f(e_k,e_{k'}) in the text and in Assumption 5. Please use a single notation consistently.","section":"Section 5.1"},{"comment":"The derivation of Υ in Appendix B.4 yields Υ = 2G^2/μ only when E is even and a negative value when E is odd. Since the paper later requires Υ > 0, the assumption that E is even should be stated in the main text where Theorem 3 is introduced, not only in the appendix.","section":"Section 6 and Appendix B.4"},{"comment":"The claim that free-riding does not exist in the proposed setting is not established: the proof argues that the coefficient Q decouples the model's performance from agents' rewards, but the argument implicitly assumes uniqueness of the Nash equilibrium, which is not proven.","section":"Remark 2 / Remark 5"},{"comment":"The claim that utility 'remains stable' as training progresses is not evidence of a Nash equilibrium, because the payment formula depends only on the known δ values and the randomly selected peer, not on the training dynamics; the stability is essentially built into the formula. The experiments should test an actual strategic interaction or compare against a baseline mechanism.","section":"Section 6.2, Figure 2"}],"recommendation":"reject","confidential_remarks":"The implementability gap in Section 5.1 is foundational: the payment rule is defined in terms of a private quantity, and no estimation or incentive-compatible reporting protocol is provided. The proof of Lemma 9 also contains a clear mathematical error that invalidates the equilibrium claim as written. I do not see a repair that stays within the current framework; the mechanism would need to be redefined using observable quantities, and the equilibrium analysis re-derived. The paper might be suitable for resubmission if the authors develop an observable payment rule and correct the equilibrium proof."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2506.16731. The paper has a genuinely interesting idea: use a label-distribution distance δ_k between each agent's local data and the reference distribution to design payments that reward agents for making their data more IID. Theorems 2 and 3, which express convergence and generalization gaps in terms of δ_k^2, are new and derived with standard techniques. They are not earth-shaking, but they are a legitimate repackaging that could be useful.\n\nThe problem is the mechanism. Payment_k is defined as f(Q/(Φδ_k^2(e_k)+Φδ_k'^2(e'_k)+Υ)), and δ_k is a property of agent k's private data. The server only sees model updates and a validation set; nothing in the paper explains how the server obtains δ_k, and the single sentence saying 'the learner can evaluate δ_k' is not an estimator or protocol. So the payment rule is not well-defined from the server's perspective, and agents cannot anticipate utility. This is not a minor gap: it breaks the central claims about truthful reporting and equilibrium. The experiments feed known δ values into the same formula, so the 'confirmation' that utility peaks at the optimal effort is true by construction.\n\nThe equilibrium analysis also has a concrete error. Lemma 9 in the appendix claims ∂f(e_k,e'_k)/∂e_k' is identical to ∂f(e_k,e'_k)/∂e_k. It isn't unless the δ functions for both agents have the same derivative at the relevant points. That lemma is what establishes the 'well-behaved' utilities needed for Theorem 5 and Brouwer. So the existence proof is unsupported as written.\n\nThere are smaller issues: they call δ a Wasserstein distance but it's actually total variation; the bound in Remark 1 is asserted to be as tight as existing ones but no quantitative comparison is given; and the parameter choices for Φ, Υ in experiments look hand-picked.\n\nWhat's worth salvaging? The bound reformulation itself is probably correct in spirit, and the broad idea of conditioning incentives on distance to a reference distribution is worth taking seriously. But as a mechanism paper, this is not close to ready. A serious referee would send it back for a major overhaul, or more likely reject.\n\nMy recommendation: reject, but with a clear path. If they can either give a practical way to estimate δ_k from server-visible information or redefine the payment using an observable proxy like the validation loss gap and redo the game theory, the paper could have a second life. As it stands, the central claim is unsupported.","headline":"The convergence-bound formalism is a real contribution, but the payment rule as written is not computable from server-visible information, and the equilibrium proof has a hole.","tokens_in":29286,"tokens_out":5494,"would_cite":false,"duration_ms":54824,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a Wasserstein-distance measure of how non-IID each agent's data are can drive payments making truthful, high-effort participation a stable equilibrium in federated learning.","keywords":["federated learning","incentive mechanism","data heterogeneity","non-IID degree","Wasserstein distance","peer prediction","Stackelberg game","Nash equilibrium"],"falsifier":"Run the protocol with a server that has access only to model updates and a public reference distribution, and ask it to compute the payments; if the resulting payments do not match $f(Q/(\\Phi\\delta_k^2+\\Phi\\delta_{k'}^2+\\Upsilon))$ for ground-truth $\\delta$ values in a simulated setting, the mechanism is not implementable as specified.","tokens_in":28241,"feed_emoji":"💰","tokens_out":10471,"duration_ms":101203,"temperature":0.7,"pith_summary":"This paper tries to show that a federated learning platform can pay each participating agent according to how far its local data are from a reference IID distribution, and that this payment rule makes truthful, high-effort participation a stable outcome. The non-IID degree of an agent is measured by the quantity the paper calls the Wasserstein distance, $\\delta_k=\\frac{1}{2}\\sum_i |p^{(k)}(y=i)-p^{(c)}(y=i)|$, and the paper proves this quantity appears in the FL convergence bound and in the generalization-loss gap between any two agents. Rewards are built from a peer-prediction scoring function that pays an agent more when its model's loss gap to a randomly chosen peer is smaller. The interaction is formalized as a two-stage Stackelberg game, and the paper proves that a pure Nash equilibrium in effort levels exists when utility functions are well-behaved. If correct, this gives a route to rewarding data quality in FL without moving raw data off the agents.","feed_headline":"Pay agents for fixing non-IID data, and federated learning speeds up","feed_subtitle":"A payment rule tied to each agent's label-distribution gap makes high-effort participation a stable equilibrium.","key_machinery":"The load-bearing object is the quantity the paper calls the Wasserstein distance, defined by the label-proportion sum $\\delta_k = \\frac{1}{2}\\sum_{i=1}^I |p^{(k)}(y=i)-p^{(c)}(y=i)|$. It appears in Theorem 2's divergence bound, so Example 1 becomes a FedAvg convergence bound with $\\sum_k p_k\\delta_k^2$ as an explicit cost of heterogeneity. The same $\\delta_k$ enters Theorem 3's bound on the generalization-loss gap, $\\Phi\\delta_k^2+\\Phi\\delta_{k'}^2+\\Upsilon$, and the payment rule inverts that bound: $f(Q/(\\Phi\\delta_k^2+\\Phi\\delta_{k'}^2+\\Upsilon))$ pays more when the gap is predicted to be small. The randomly chosen peer $k'$, drawn from the peer-prediction approach, prevents coordinated strategic behavior, while the Stackelberg stage sets the payment coefficient $Q$ to maximize the learner's payoff.","core_discovery":"On its own terms, the paper's central discovery is that the non-IID degree $\\delta_k$ is not just a description of heterogeneity but a quantity that controls both sides of the incentive problem. Theorem 2 bounds the divergence between local and averaged models by a constant multiple of $\\sum_k p_k \\delta_k^2$, which turns generic FL convergence bounds into bounds that explicitly penalize heterogeneous data; Theorem 3 bounds the generalization-loss gap between two agents by $\\Phi\\delta_k^2+\\Phi\\delta_{k'}^2+\\Upsilon$. That gap bound is used to define the payment $Payment_k(e_k,e_{k'}) = f(Q/(\\Phi\\delta_k^2(e_k)+\\Phi\\delta_{k'}^2(e_{k'})+\\Upsilon))$, so an agent earns more by reducing its own $\\delta_k$ relative to its peer. The paper then models agents and learner as a Stackelberg game, characterizes each agent's optimal effort by a first-order condition, and proves that if utility functions are well-behaved, a pure Nash equilibrium in effort levels exists in which no agent can profitably deviate.","pith_inferences":["The paper does not provide an inference procedure for $\\delta_k$ from model updates; the most direct extension is to estimate $\\delta_k$ from the server's observed gradient norms or predicted label distributions, and the mechanism's deployability depends on that estimate being accurate.","Because the payment is based on an upper bound rather than realized contribution, an agent could in principle game the bound by overfitting the reference validation set without improving its true non-IID degree; comparing payments to leave-one-out influence would test whether this matters.","The same $\\delta_k$-to-convergence link could be used outside incentives, for example as a client-selection or data-valuation criterion in federated learning systems that already assume honest participation."],"forward_implications":["Theorem 2 converts the divergence term in existing FL convergence bounds into a weighted sum of squared non-IID degrees, so any agent that lowers $\\delta_k$ directly tightens the global convergence bound.","The peer-prediction scoring rule ties payment to the generalization-loss gap with a randomly chosen peer, which the paper argues prevents coordinated untruthful behavior and makes high-effort participation stable.","The Stackelberg formulation gives the learner a principled way to set the initial payment coefficient $Q$ by maximizing its own payoff subject to non-negative agent utilities.","Because the scoring uses model responses and a public reference distribution, the mechanism is designed to reward data quality without requiring agents to reveal raw data."],"supporting_citations":[{"why":"Supplies the FedAvg convergence bound that Example 1 rewrites in terms of the non-IID degree $\\delta_k$.","marker":"[Li et al., 2019]"},{"why":"Provides the generic SGD recursion (Lemma 1) from which the convergence analysis starts.","marker":"[Stich, 2018]"},{"why":"Defines the FedAvg protocol and aggregation rule used throughout the paper.","marker":"[McMahan et al., 2017]"},{"why":"Provides the empirical evidence that non-IID local data degrade FL accuracy and convergence, motivating the metric.","marker":"[Zhao et al., 2018]"},{"why":"Introduces the hypothesis-elicitation and peer-prediction perspective for FL incentives that the scoring functions build on.","marker":"[Liu and Wei, 2020]"},{"why":"Supplies the multi-task peer-prediction machinery used to design random-peer scoring and truthfulness guarantees.","marker":"[Shnayder et al., 2016]"},{"why":"Provides the original peer-prediction method for eliciting truthful feedback without ground truth.","marker":"[Miller et al., 2005]"},{"why":"Used in the proof of Theorem 5 to show that the best-response correspondence has a fixed point.","marker":"[Brouwer, 1911]"},{"why":"Supplies the definition of well-behaved utility functions and the collaboration-game baseline that Theorem 5 relies on.","marker":"[Blum et al., 2021]"},{"why":"Provides the data-sharing incentive baseline and the free-riding discussion that the paper contrasts with its equilibrium.","marker":"[Karimireddy et al., 2022]"}],"fun_headline_variants":["Reward agents for low data gap, FL converges faster","Payment scheme turns data heterogeneity into incentive","Stackelberg equilibrium boosts quality in federated learning","Paying agents for less skewed data speeds FedAvg","Data-gap-based rewards stabilize high-effort FL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the learner can actually evaluate the payment function, which requires knowing each agent's private label distribution or, equivalently, its non-IID degree $\\delta_k$; the paper does not describe how the server obtains these values from the model updates it receives, so if they are unavailable the payment scheme cannot be computed.","fun_headline_variants_meta":{"raw":{"variants":["Reward agents for low data gap, FL converges faster","Payment scheme turns data heterogeneity into incentive","Stackelberg equilibrium boosts quality in federated learning","Paying agents for less skewed data speeds FedAvg","Data-gap-based rewards stabilize high-effort FL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1367,"prompt_tokens":982,"completion_tokens":385,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":311}},"tokens_in":598,"tokens_out":385,"duration_ms":4292,"temperature":1.0,"reasoning_tokens":311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:20:21.893067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the protocol with a server that has access only to model updates and a public reference distribution, and ask it to compute the payments; if the resulting payments do not match $f(Q/(\\Phi\\delta_k^2+\\Phi\\delta_{k'}^2+\\Upsilon))$ for ground-truth $\\delta$ values in a simulated setting, the mechanism is not implementable as specified.","supporting_citations":[{"cited_title":"A fairness-aware incentive scheme for federated learning","cited_arxiv_id":null,"evidence_quote":"Introduces the hypothesis-elicitation and peer-prediction perspective for FL incentives that the scoring functions build on."},{"cited_title":"Informed truthfulness in multi-task peer prediction","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-task peer-prediction machinery used to design random-peer scoring and truthfulness guarantees."},{"cited_title":"Eliciting informative feedback: The peer-prediction method","cited_arxiv_id":null,"evidence_quote":"Provides the original peer-prediction method for eliciting truthful feedback without ground truth."},{"cited_title":"\\\"U ber abbildung von mannigfaltigkeiten","cited_arxiv_id":null,"evidence_quote":"Used in the proof of Theorem 5 to show that the best-response correspondence has a fixed point."},{"cited_title":"One for one, or all for all: Equilibria and optimality of collaboration in federated learning","cited_arxiv_id":null,"evidence_quote":"Supplies the definition of well-behaved utility functions and the collaboration-game baseline that Theorem 5 relies on."}],"review_version":2}