{"id":"77b2dfbd-ceba-4c2a-97be-04601bd07951","arxiv_id":"2608.12108","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LIGHTYEAR selects each client's aggregation set by scoring how similarly models behave on private validation data with a neural tangent kernel, improving robustness to heterogeneous and malfunctioning clients in peer-to-peer federated learning.","lead":"A federated learning system that lets each client pick which other clients' updates to merge, based on how similarly models behave on its own data rather than how close their parameters are. It survives malfunctioning clients in tests across five medical and vision datasets, though a few low-malfunction cases favor baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Final-layer NTK agreement (Eq. 14) is invariant to the final classification head, so a head-only malicious update can score A=1, pass Eq. 16, and corrupt the aggregate via Eq. 17.","rationale":"The paper proposes an interesting function-space selection mechanism and provides broad experiments on three malfunction types. The reader's weakest assumption—that the NTK agreement score reliably indicates update benefit—is the right place to look, and I found a concrete reason it can fail: the final-layer NTK of a linear-head network is the feature Gram matrix, invariant to the final-layer weights. A malicious or malfunctioning client that keeps the feature extractor and corrupts only the output head is mathematically indistinguishable from a benign update under Eq. 14, yet can destroy client i's model through Eq. 17. This undermines the central claim of robust, behavior-aware update selection, independent of the empirical issues noted by the reader. The paper's own tables also contradict 'consistently outperforms' in low-malfunction settings, but the mechanism flaw is more fundamental and testable. The proposed FEMNIST head-flip test would settle whether the authors' implementation uses logits (making the flaw real) or post-softmax outputs (which would weaken the counterexample). The correct verdict remains CONDITIONAL: the method is plausible and strong on the evaluated attacks, but it should not be accepted as-is without either modifying the agreement criterion to be sensitive to output behavior, using the full NTK, or explicitly scoping the robustness claims to attacks that perturb the feature extractor. I therefore keep the reader's CONDITIONAL verdict unchanged, with the head-invariance concern as a specific required condition.","tokens_in":35136,"tokens_out":13270,"duration_ms":114431,"concrete_test":"On FEMNIST with the paper's two-layer CNN, train a benign client i; construct a neighbor j by copying θ_i's feature extractor and flipping the sign of the final-layer weight matrix. Compute A(θ_i, θ_j; V_i) per Eq. 14 and the accuracy of θ_j on V_i. If A ≥ 0.6 (likely ≈ 1) while accuracy is at or below chance, run LIGHTYEAR with j as the only selected neighbor for one aggregation round and record client i's test accuracy. Repeat with a randomly re-initialized head. If either the agreement remains above threshold or the test accuracy drops substantially, the selection rule is blind to head-only malfunctions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the agreement score A(θ_i, θ_j; V_i) in Eq. 14 measures predictive alignment. For the architectures used (DenseNet, EfficientNet, TransUNet, CNN), the final layer is linear: h = W_L φ(x). The Jacobian in Eq. 11 is J_θ(x) = ∂h/∂θ^{(L)} = φ(x), which is independent of W_L. Hence the kernel in Eq. 12 and its centered/normalized version in Eq. 15 depend only on the feature extractor φ, not on the output head. Two models with identical features and opposite heads produce A = 1. A malfunctioning client that copies client i's feature extractor (available in the P2P exchange) and flips the sign of the final head has chance-level accuracy on V_i but A ≈ 1, so it passes the τ = 0.6 threshold in Eq. 16. Aggregating it via Eq. 17 moves θ_i's head toward the flipped head and collapses accuracy. The stated intuition in §4.5 ('similar local predictive sensitivities') conflates sensitivity of outputs w.r.t. final-layer parameters—which is just the feature map—with beneficial predictive behavior. The attacks evaluated (ANA, SFA, random init) perturb the feature extractor, which is why Fig. 7 shows separation; a head-only attack is invisible to the selector. Separately, the headline 'consistently outperforms' is contradicted by the paper's own tables in low-malfunction regimes (e.g., XRay ANA with 1 malfunction: LIGHTYEAR 84.4 vs AFA 86.8; Ultrasound SFA with 1 malfunction: LIGHTYEAR 83.6 vs CFL 86.6), but the head-invariance is the more fundamental correctness risk.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LIGHTYEAR, a peer-to-peer federated learning method for personalized aggregation under data heterogeneity and client malfunctions. Instead of comparing updates in parameter space, each client computes an agreement score between its own final-layer neural tangent kernel and that of each neighbor on a local validation set (Eqs. 11-15), selects neighbors whose score exceeds a threshold tau (Eq. 16), and aggregates selected updates with a round-decayed mixing coefficient gamma^t (Eq. 17). The authors evaluate on five datasets (FEMNIST, Camelyon17, ISIC19, ultrasound, chest X-ray) with ANA, sign-flipping, random, and dynamic malfunctions against nine baselines, reporting mean accuracy/Dice with standard deviations, and claim consistent superiority.","tokens_in":35505,"tokens_out":5205,"duration_ms":46920,"significance":"If the method works as claimed, the function-space selection idea and P2P evaluation are a useful contribution. Strengths include a broad benchmark spanning classification and segmentation, three malfunction types plus a dynamic setting, inclusion of two P2P baselines, open-source code, and a detailed reproducibility appendix. However, the central equivalence between the final-layer NTK agreement score and predictive alignment is not established and, as detailed below, is false for head-only corruptions. The empirical claim of consistent superiority is also contradicted by the appendix tables in low-malfunction regimes. With those issues addressed, the paper could be a solid contribution to robust personalized federated learning.","major_comments":[{"comment":"The final-layer NTK agreement score is invariant to the final linear layer, which undermines the selection mechanism for head-only corruptions. For a network with h = W_L phi(x), the Jacobian in Eq. (11) is dh/dtheta^(L) = phi(x) plus a bias-constant term, independent of W_L; consequently the kernel in Eq. (12), its centered/normalized form in Eq. (13), and the score in Eq. (14) depend only on the feature extractor. A malfunctioning client that copies the feature extractor and flips the sign of the classification head has chance-level accuracy on V_i but score A = 1, passes the threshold in Eq. (16), and corrupts the aggregate in Eq. (17). The attacks in §4.2 (ANA, SFA, random init) all perturb the feature extractor, so the separation shown in Fig. 7 does not test this failure mode. The manuscript should either prove a property that rules out this case, change the score to include output-space agreement on V_i, or compute the NTK over all layers; as it stands, the central claim that the score measures predictive behavior on the target domain is not supported.","section":"§4.5, Eqs. (11)-(15)"},{"comment":"The abstract and §7 claim that LIGHTYEAR consistently outperforms all baseline approaches, but several low-malfunction rows in the paper's own tables contradict this. For example, Ultrasound ANA with one malfunction: AFA 86.1 vs LIGHTYEAR 83.1 (Table 13); XRay ANA with one malfunction: AFA 86.8 vs LIGHTYEAR 84.4 (Table 17); Ultrasound SFA with one malfunction: CFL 86.6 vs LIGHTYEAR 83.6 (Table 14); XRay dynamic with one malfunction: AFA 87.4 vs LIGHTYEAR 82.7 (Table 20). No significance tests are reported, and the reported standard deviations overlap in most of these cases. The authors should replace the global superiority claim with a precise statement of the regimes where LIGHTYEAR is superior, e.g., high malfunction rates, and report paired significance tests or effect sizes.","section":"§7 and Appendix C, Tables 13, 14, 17, 18, 20"}],"minor_comments":[{"comment":"Eq. (5) is presented as an equality, but the target error of the corrupted model is at most the sum of the two displayed terms by the triangle inequality; Eq. (9) correctly uses an inequality. Please correct Eq. (5) or state it as an upper bound.","section":"§4.3, Eq. (5)"},{"comment":"Eq. (17) mixes notation: neighbor models are written theta_j while the client model is written bar_theta_i. Please clarify whether neighbors send their raw local models or their aggregated bar_theta_j updates.","section":"§4.7, Eq. (17)"},{"comment":"Several table captions are mismatched: Table 9 is numbered as dynamically changing malfunction but reports ANA, Table 10 reports SFA, Table 11 reports Random, and Table 12 spells Dyanmic. Please fix the captions.","section":"Appendix C, Tables 9-12"},{"comment":"The dataset name casing is inconsistent (Femnist vs FEMNIST) and the x-axis labels in Fig. 7 are duplicated; please harmonize the labels across figures and tables.","section":"Figures 6 and 7"},{"comment":"The text calls gamma^t a regularization parameter, but it is a time-decay mixing coefficient rather than a penalty on a norm; consider renaming it to avoid confusion with regularization in optimization.","section":"§4.7"}],"recommendation":"major_revision","confidential_remarks":"The head-invariance issue is a correctness risk that the authors should address before publication, and the empirical superiority claim needs to be scoped to the regimes where it actually holds. Given the broad evaluation and released code, I see this as a major revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nQuick take: LIGHTYEAR is a reasonable P2P FL selection rule with a large experiment suite, but the load-bearing agreement score has a concrete blind spot the paper misses: it is invariant to the final classification head. That matters for the robustness claim.\n\nWhat's new: using final-layer NTK agreement on a local validation set to pick a personalized aggregation set per client is a legitimate new application of known components (NTK, CKA, P2P). The empirical sweep is wide: five datasets, nine baselines, ANA/SFA/random attacks, plus a dynamic scenario. Code link is provided, and the ablations on gamma and tau are useful. For the attacks they consider, the separation shown in Figure 7 is real.\n\nThe soft spots, in order. First, the head-invariance issue. For a linear final layer, dh/dW_L = phi(x), so the kernel and the agreement score depend only on the feature extractor. A malicious client that copies the victim's feature extractor and flips the sign of the final head gets agreement ~1, passes the threshold, and poisons the aggregate via Eq. 17. The paper's attacks all perturb the feature extractor, so the selector never sees this. This is not a corner case; it is exactly the kind of malfunction the paper claims to handle. Second, the headline 'consistently outperforms' is contradicted by the paper's own tables in low-malfunction regimes: e.g., Ultrasound ANA with 1 malfunction has AFA 86.1 vs LIGHTYEAR 83.1; XRay ANA 1 has AFA 86.8 vs 84.4. The advantage is real in high-malfunction settings, but not universal. Third, there are no significance tests, and the reported stds suggest some differences are within noise.\n\nThe math is otherwise straightforward, and the paper is not circular: the score is computed, not fitted, and only tau and gamma are tuned. But the central premise that this score measures predictive alignment is only partially true, and the head-invariance breaks it.\n\nWho should read it: anyone working on robust aggregation for cross-silo FL, especially in medical imaging. It deserves a serious referee, but the authors need to either use the full NTK, add a head-sensitive term, or at least characterize the head-only attack and explain why it is out of scope.\n\nMy recommendation: engage with it, accept for peer review, and expect major revision if the head-invariance is not addressed.","headline":"A plausible P2P FL selection rule with broad experiments, undermined by a concrete blind spot: the final-layer NTK agreement score ignores the classification head, so a head-only malicious update passes the filter.","tokens_in":36044,"tokens_out":3121,"would_cite":false,"duration_ms":27770,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across five datasets and nine baselines, LIGHTYEAR claims that choosing aggregation partners by an NTK agreement score in function space, evaluated on private validation data, keeps federated clients accurate even when most neighbors…","keywords":["federated learning","personalized aggregation","neural tangent kernel","function space","peer-to-peer learning","Byzantine robustness","non-IID data","client drift"],"falsifier":"Measure, across many random client partitions and malfunction settings, the correlation between each incoming update's NTK agreement score and the actual change in that client's held-out test error after aggregating the update; if a substantial number of low-scoring updates improve performance or high-scoring updates degrade it, the selection premise fails. A direct adversarial test would craft a malicious model whose final-layer kernel on the validation set matches the victim's kernel, but whose predictions on held-out target data are wrong; if LIGHTYEAR accepts it, the score is not safety-critical.","tokens_in":34922,"feed_emoji":"🛡️","tokens_out":7034,"duration_ms":60647,"temperature":0.7,"pith_summary":"The paper proposes LIGHTYEAR, a peer-to-peer federated learning method in which each client decides which neighbor updates to accept by evaluating, on its own private validation set, how similarly the two models respond to small parameter changes. The claim is that this final-layer neural tangent kernel (NTK) agreement score captures predictive behavior better than parameter-space distance, so it separates beneficial updates from updates caused by data heterogeneity or malfunctioning clients. If the claim holds, federated systems can personalize aggregation per client without a central server ever seeing local data, and remain stable even when the majority of clients are corrupted. The paper supports the claim with experiments on five datasets spanning classification and segmentation, comparing against nine baselines including FedAvg, robust aggregation methods, and P2P approaches.","feed_headline":"Federated clients choose updates by behavior, not parameters","feed_subtitle":"An NTK agreement score on private data lets each client accept only the updates that help its own domain.","key_machinery":"The load-bearing object is the empirical final-layer NTK matrix $K_\\theta(V_i)$, whose entry $(a,b)$ is $\\frac{1}{C}\\langle J_\\theta(x_a), J_\\theta(x_b)\\rangle$ for validation inputs $x_a,x_b$ and output dimension $C$; it is centered and Frobenius-normalized with $H=I-\\frac{1}{|V_i|}\\mathbf{1}\\mathbf{1}^{\\top}$. The agreement score is the Frobenius inner product of the two normalized kernel matrices, $A(\\theta_i,\\theta_j;V_i)$, a centered kernel alignment. Selection applies the threshold $\\tau$ (fixed at 0.6 in the experiments) to build $S_i$, and aggregation applies $\\bar\\theta_i^{(t+1)}=\\bar\\theta_i^{(t)}+\\gamma_t\\frac{1}{|S_i|}\\sum_{j\\in S_i}(\\theta_j-\\bar\\theta_i^{(t)})$, with $\\gamma_t$ interpreted as a round-dependent decay that stabilizes training under heterogeneity.","core_discovery":"The central discovery is that the final-layer NTK agreement score, defined as the centered Frobenius-normalized kernel alignment between two models evaluated on the local validation set, is a reliable selector for personalized federated aggregation. The paper argues that models with similar local predictive sensitivities on a client's data will not harm that client's target domain, whereas parameter-space proximity gives no such guarantee. Using this score, each client selects the subset of neighbor updates whose agreement exceeds a threshold $\\tau$, then averages the selected updates with a round-dependent contraction factor $\\gamma_t$; setting $\\gamma=1$ reduces the rule to standard FedAvg over the selected set. Empirically, the paper finds that LIGHTYEAR maintains stable accuracy and Dice scores across all datasets and malfunction types, including cases where malfunctioning clients outnumber healthy ones, while centralized baselines collapse to near-random performance on segmentation tasks.","pith_inferences":["If the paper is right, the same final-layer NTK alignment could be applied in centralized FL by having the server evaluate incoming updates on a public proxy validation set, at the cost of weakening the privacy assumption that motivated the P2P design.","Because the score uses only the final layer, it measures task-head alignment rather than representation shifts; a natural extension is to test whether full-network or intermediate-layer NTK alignment improves selection in realistic covariate-shift settings.","The threshold $\\tau$ is fixed globally at 0.6; per-client or round-adaptive thresholds could push the method further in highly heterogeneous federations without changing the machinery.","A minimal-validation-size analysis would tell practitioners how much local data each client must hold before the agreement score is trustworthy."],"forward_implications":["A client can reject corrupted or misaligned updates without any server-side anomaly detection, because the decision is made locally on its own data.","Federated training remains usable when more than half of participants malfunction, a regime where parameter-space robust aggregators such as Krum and clustered methods degrade sharply.","The same agreement score can be reused as a per-client stopping criterion or trust score, since it separates benign from malfunctioning updates across datasets and attack types.","The regularized aggregation with $\\gamma_t<1$ provides a simple remedy for client drift that is independent of the selection mechanism.","The method's quadratic communication cost in the number of clients confines it to small, reliability-critical federations, which the paper argues is the right regime for medical imaging."],"supporting_citations":[{"why":"Introduces the neural tangent kernel, the mathematical object whose final-layer restriction defines the agreement score.","marker":"[21]"},{"why":"Provides the domain-adaptation error bound used to split target error into exchangeability and corruption terms.","marker":"[5]"},{"why":"Supplies the function-space versus parameter-space distinction that motivates selecting updates by behavior.","marker":"[6]"},{"why":"Defines FedAvg, the naive all-updates aggregation rule the paper starts from and the primary centralized baseline.","marker":"[31]"},{"why":"Prior evidence that NTK is an effective tool for model optimization in heterogeneous FL, cited to justify the NTK choice.","marker":"[45]"},{"why":"BALANCE, a P2P Byzantine-robust baseline that LIGHTYEAR is compared against and must outperform.","marker":"[15]"},{"why":"SCCLIP, a P2P remove-then-clip aggregation baseline used in the comparison.","marker":"[42]"},{"why":"Krum, a parameter-space robust aggregation baseline whose failures under malfunctions motivate function-space selection.","marker":"[7]"}],"fun_headline_variants":["NTK-guided aggregation personalizes federated updates","Function-space NTK scoring enhances robust FL","P2P NTK agreement selects helpful updates","LIGHTYEAR: choose neighbors by NTK alignment","Robust FL: pick updates based on behavior"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that two models which look similar in how they would respond to small training changes on a client's own data will actually help that client's task, an assumption the paper states as intuitive rather than proving.","fun_headline_variants_meta":{"raw":{"variants":["NTK-guided aggregation personalizes federated updates","Function-space NTK scoring enhances robust FL","P2P NTK agreement selects helpful updates","LIGHTYEAR: choose neighbors by NTK alignment","Robust FL: pick updates based on behavior"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00096,"raw_usage":{"total_tokens":4116,"prompt_tokens":995,"completion_tokens":3121,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":3049}},"tokens_in":611,"tokens_out":3121,"duration_ms":21066,"temperature":1.0,"reasoning_tokens":3049,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:15:47.696254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, across many random client partitions and malfunction settings, the correlation between each incoming update's NTK agreement score and the actual change in that client's held-out test error after aggregating the update; if a substantial number of low-scoring updates improve performance or high-scoring updates degrade it, the selection premise fails. A direct adversarial test would craft a malicious model whose final-layer kernel on the validation set matches the victim's kernel, but whose predictions on held-out target data are wrong; if LIGHTYEAR accepts it, the score is not safety-critical.","supporting_citations":[{"cited_title":"Neural tangent kernel: Conver- gence and generalization in neural networks.Advances in neural information process- ing systems, 31, 2018","cited_arxiv_id":null,"evidence_quote":"Introduces the neural tangent kernel, the mathematical object whose final-layer restriction defines the agreement score."},{"cited_title":"A theory of learning from different domains.Machine learning, 79:151–175, 2010","cited_arxiv_id":null,"evidence_quote":"Provides the domain-adaptation error bound used to split target error into exchangeability and corruption terms."},{"cited_title":"Neural tangent kernel empowered federated learning","cited_arxiv_id":null,"evidence_quote":"Prior evidence that NTK is an effective tool for model optimization in heterogeneous FL, cited to justify the NTK choice."},{"cited_title":"Byzantine-robust decentralized federated learning","cited_arxiv_id":null,"evidence_quote":"BALANCE, a P2P Byzantine-robust baseline that LIGHTYEAR is compared against and must outperform."},{"cited_title":"Byzantine-robust decentralized learning via remove- then-clip aggregation","cited_arxiv_id":null,"evidence_quote":"SCCLIP, a P2P remove-then-clip aggregation baseline used in the comparison."}],"review_version":1}