{"id":"c6211aea-2487-453d-be8b-d4cc7b4f2816","arxiv_id":"2508.20557","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AdaFD uses loss-based client weighting and L2 distillation to improve federated text classification when clients hold data from different domains, outperforming existing methods in two settings.","lead":"A new federated learning method, AdaFD, improves global text classifiers when each client holds data from a different language domain by weighting client contributions based on training loss. The paper also introduces a benchmark for multi-domain non-IID text data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported AdaFD gains conflate adaptive weighting with the switch to L2 distillation; no ablation isolates the adaptive component.","rationale":"The reader's weakest assumption and my own analysis converge on the same load-bearing point: the adaptive weighting mechanism is not isolated from other changes, and its underlying premise is asserted without direct evidence. I agree with the reader that this is the central vulnerability. The paper's experiments vary both the weighting rule and the distillation loss relative to the baselines, so the reported improvements cannot be uniquely attributed to the proposed adaptive ensemble. A single ablation, uniform weights with the same L2 distillation, would resolve whether the novelty is doing any work. The missing statistical support (no error bars, no described significance test despite p<0.001 claims) further weakens the headline result but is secondary to the confound. Since the reader already recommends a conditional verdict and this concern reinforces that recommendation without adding a different failure mode, the appropriate verdict is unchanged: the paper should be accepted only if the authors add the isolating ablation and either justify or repair the loss-alignment premise and the tau inconsistency.","tokens_in":19144,"tokens_out":3876,"duration_ms":37146,"concrete_test":"Run the homogeneous and heterogeneous setups from Table I with the ENWC weights in Eq. (8) replaced by uniform weights w_k=1/K, keeping the L2 distillation of Eq. (10), beta=5, 5 rounds, 3 epochs, and identical seeds. If uniform-weight-L2 matches ENWC-L2 global F1 within 0.2 points, the adaptive weighting contributes nothing. Also compute the Spearman correlation between each client's l_k^min and that client's F1 on the global test set across domains; a non-positive correlation would falsify the loss-as-alignment premise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AdaFD's central contribution is the adaptive ensemble in Eqs. (6)-(8), yet the experiments compare against baselines that differ in two respects: weighting rule and distillation objective. FedAvg, DS-FL, MHAT, and FedKD distill with KL/entropy-based losses (Eqs. 3-4), while AdaFD uses the L2 loss of Eq. (10); the derivation of Eq. (10) as the tau-to-infinity limit is at odds with the reported tau=1 in Section VI-B. Tables I and II therefore cannot attribute the 0.5-2.0 F1 gain to the adaptive weights; the L2 objective alone may account for it. No ablation keeps L2 distillation and replaces ENWC weights with equal or data-size weights. Section VII-E asserts that training loss reflects 'fitting quality of the global data distribution' without reporting a correlation; with one domain per client, low local loss can simply indicate local overfitting. The caption claim p<0.001 is unsupported by any reported replicates or significance test. These gaps leave the central claim conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaFD, an adaptive federated distillation framework for multi-domain non-IID textual data. It introduces a benchmark in which five Amazon review domains are assigned to five clients, creating both label and language-domain heterogeneity. AdaFD aggregates client logits on public data using weights derived from client training losses via three strategies (RNWC, ENWC, LLMWC) and distills the central model with an L2 loss that the authors derive as a large-temperature limit of KL divergence. Experiments report F1 scores on client-specific and global test sets in homogeneous and heterogeneous settings, at original and downsampled scales, claiming improvements over FedAvg, DS-FL, MHAT, and FedKD. The code is released.","tokens_in":19268,"tokens_out":5290,"duration_ms":39019,"significance":"The multi-domain non-IID benchmark is a useful contribution: most prior federated NLP work models non-IID only through label skew, and the proposed domain-based partitioning is more realistic for text. The adaptive weighting idea, using training loss as a proxy for a client's contribution to the global distribution, is interesting and plausibly useful. If the reported improvements are robust, the method is a practical addition to federated distillation. The code release and the systematic comparison across two settings, two data scales, and three weighting variants are also strengths. However, the empirical evidence as presented does not yet isolate the adaptive-weighting contribution, and several statistical and reproducibility issues need to be resolved before the central claim can be accepted.","major_comments":[{"comment":"The reported gains cannot be attributed to the adaptive weighting scheme because AdaFD differs from every baseline in two respects simultaneously: the weighting rule and the distillation objective. FedAvg, DS-FL, MHAT, and FedKD use KL/entropy-based distillation (Eqs. 3-4), while AdaFD uses the L2 loss of Eq. (10). No ablation holds the L2 objective fixed and replaces the ENWC/RNWC weights with equal weights or data-size weights. Without such an ablation, the 0.5-2.0 F1 improvements in Tables I and II may be caused by the distillation objective alone. Please add an ablation that isolates the adaptive weighting component.","section":"§V-A, §V-B, Tables I-II"},{"comment":"Eq. (10) is justified as the approximation of KL divergence when the temperature τ tends to infinity, but Section VI-B states that the temperature τ is set to 1 in the experiments. This is internally inconsistent: either the temperature is not actually used in the L2 loss, in which case the derivation should be corrected, or the experiments use τ=1, in which case the claimed connection to the τ→∞ limit is not what was implemented. Please clarify the role of τ and reconcile Eq. (10) with the reported setting.","section":"§V-B, §VI-B"},{"comment":"The table captions claim that AdaFD achieves statistically significant improvements over baselines with p<0.001, but the paper reports a single F1 number per condition with no variance, no number of replicates, and no description of the significance test. This makes the p-value unverifiable and the robustness claim unsupported. Please report means and standard deviations across at least several independent runs and describe the exact statistical test used, including any multiple-comparison correction.","section":"Tables I and II captions"},{"comment":"The core assumption that a client's minimum training loss (Eq. 6) or exponentially scaled training loss (Eq. 7) reflects how well that client's predictions align with the global data distribution is asserted in Section VII-E but not supported by quantitative evidence. Section VII-E only shows loss curves and concludes that training losses are 'reasonable and effective' without reporting a correlation between loss and contribution quality. In addition, Section VII-D selects β=5 because it yields the highest F1 on the test set, which is a form of test-set fitting. Please provide an empirical correlation or a direct ablation supporting the loss-to-weight mapping, and select β on a validation split or report sensitivity without test-set optimization.","section":"§VII-C, §VII-D, §VII-E"},{"comment":"The LLMWC branch in Algorithm 1 is incomplete: the pseudocode says 'generate the weights by LLM' but never assigns the generated weights to a variable and never shows how the ensemble logits bz are computed in that branch, since Eq. (8) appears only in the else branch. This makes the LLMWC results in Tables I and II non-reproducible. Please specify the full LLMWC procedure, including the weight variable, normalization, and how the weighted sum is formed.","section":"Algorithm 1, §V-A"},{"comment":"The construction of the global test set is under-specified. The text says it is 'constructed from all domains' and 'sampled with α=0.8' and that its size is consistent with the client-specific test sets, but it does not state whether this set is disjoint from the client-specific test sets, how it relates to the 8:1:1 split of private data, or whether the same set was used for model selection such as choosing β. Please describe the exact sampling procedure and ensure that no test data are used for hyperparameter selection.","section":"§VI-D.2"}],"minor_comments":[{"comment":"Equation (9) uses σ without defining it; please state that σ denotes the softmax function.","section":"§V-B"},{"comment":"There are minor language issues: 'can not capture' should be 'cannot capture', and the Index Terms entry 'natural language process' should be 'natural language processing'.","section":"Abstract, Index Terms"},{"comment":"The axis labels and legends in Figures 4-6 are difficult to read in the current rendering; please provide higher-resolution versions so that the reported trends are legible.","section":"Figures 4-6"},{"comment":"The sentence 'the adaptive ensemble hyperparameter β and temperature τ is set to 5 and 1' has a subject-verb agreement issue and should be rephrased as 'β and τ are set to 5 and 1, respectively.'","section":"Section VI-B"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper contributes a genuinely new benchmark—multi-domain input non-IID for federated distillation, using five Amazon review domains—plus a simple loss-based adaptive weighting scheme (RNWC, ENWC) and an LLM variant. The experiments cover homogeneous and heterogeneous PLM settings at two data scales, and the code is public. That is real value.\n\nThe problem is that the central claim isn't actually tested. AdaFD changes two things relative to baselines: the weighting rule and the distillation objective (L2 instead of KL at τ=1). The gains in Tables I and II could come from either alone. There is no ablation that holds L2 fixed and swaps ENWC for equal or data-size weights. The stress-test note has this right. And Fig. 4 shows weights converging to equality by round 5 anyway, which suggests the adaptive component may be doing less work as training proceeds.\n\nThe smaller issues line up: p<0.001 appears in both table captions with no replicates, no error bars, and no described test. β is tuned on the test set (β=5 is the best of the values tried), which is a mild form of fitting to the data. Algorithm 1's LLMWC branch is placed outside the loop in a way that makes the pseudocode ambiguous. Section V-B says L2 is the τ→∞ limit, but VI-B reports τ=1. And the claim that local training loss is a proxy for fit to the global distribution is asserted in VII-E; the figure there shows AdaFD clients with higher loss than Local models yet better global F1, so the proxy deserves a direct test rather than an assumption.\n\nNone of this is fatal. The benchmark is a useful resource. The idea is sensible. The writing is clear, and I see no circular reasoning or invented entities. The self-citation to FedID is to a shared-author paper but is contextual and appropriate. I'd cite the benchmark in my own work, and I'd bring the paper to a reading group as a 'worth knowing about, needs an ablation' case.\n\nRecommendation: send to peer review. The benchmark alone justifies a referee's time. Make the L2-fixed weighting ablation, variance reporting, and a clean explanation of τ and the LLMWC path conditions of acceptance. If the ablation shows the adaptive weights matter, this becomes a solid modest contribution; if not, the benchmark still stands.","headline":"New multi-domain non-IID benchmark for federated distillation, but the central claim that adaptive weighting helps is unproven because no ablation isolates it from the switch to L2 distillation.","tokens_in":19851,"tokens_out":2826,"would_cite":true,"duration_ms":27151,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Client training losses can replace data-size weights in federated distillation.","keywords":["federated distillation","non-IID","multi-domain","natural language processing","adaptive ensemble","training loss weighting","benchmark","heterogeneous models"],"falsifier":"Run AdaFD on the released benchmark while recording, for each round, each client's training-loss rank and the change in global F1 when that client's weight is increased; if clients with lower training loss are not positively correlated with improvements in global F1, or if a deliberately overfitted client with near-zero loss on its own domain receives the largest weight and drags global F1 down, the central claim is falsified.","tokens_in":18855,"feed_emoji":"📊","tokens_out":4958,"duration_ms":48634,"temperature":0.7,"pith_summary":"This paper claims that non-IID data in NLP federated learning should be understood as diversity in language domain as well as diversity in labels, and that earlier benchmarks missed the input-side heterogeneity. It introduces a benchmark built from five product-review domains and proposes AdaFD, a federated distillation method that weights each client's public-data predictions by a function of that client's private training loss. Two loss-based weighting schemes, RNWC and ENWC, plus a large-language-model-based variant, are compared with baselines that weight clients equally or by data size. In sentiment classification experiments, the loss-weighted variants achieve the best global F1 score among decentralized baselines in both same-architecture and different-architecture settings.","feed_headline":"Adaptive loss weighting beats fixed weights in federated NLP","feed_subtitle":"New benchmark varies language domain and labels; exponential-loss weighting tops baselines on global F1 in both settings.","key_machinery":"The central object is the adaptive ensemble weighting of client logits: weights $w^k = (1/l^k_{\\min})/\\|v\\|_1$ for RNWC and $w^k = e^{-\\beta l^k_{\\min}}/\\|v\\|_1$ for ENWC, where $l^k_{\\min}$ is client $k$'s minimum training loss over local epochs, $\\beta$ is a sharpness hyperparameter, and $v$ is the vector of unnormalized scores. These weights replace the equal or data-size weights used in earlier federated distillation methods, and they produce the aggregated prediction $\\hat{z} = \\sum_k w^k z^k$ that distills the central model through the squared-error objective $\\|\\hat{z} - \\tilde{z}\\|_2^2$. The weighting is what adapts: a client whose training loss is low is taken to fit the global distribution better and therefore receives more influence in the ensemble.","core_discovery":"The central claim is that adaptive ensemble weights derived from local training losses make federated distillation more robust to multi-domain non-IID text data than equal weighting or data-size weighting. Specifically, the paper proposes two weighting formulas: a reciprocal of each client's minimum training loss (RNWC) and an exponentially decaying function of that loss (ENWC), both normalized to sum to one and used to aggregate client logits on a shared public dataset. It further claims that distilling the central model with a squared-error loss, an approximation of KL divergence at high temperature, lets the global model track the global distribution even when clients hold different domains and different model architectures. The reported evidence is that AdaFD with ENWC achieves the best global F1 among decentralized baselines in homogeneous and heterogeneous settings, on both original and downsampled data, in both client-specific and global test sets.","pith_inferences":["Beyond the paper, the same weighting could be tested on label-diverse-only non-IID data to see whether the loss proxy helps when domain misalignment is not the main difficulty.","A natural extension is to compare rank-based weighting, which uses only the ordering of losses, with value-based weighting, to separate the information in loss magnitudes from the information in their order.","The paper's five domains are all product reviews; a harder test would use structurally different text types, where a low training loss might reflect domain narrowness rather than alignment with the global distribution.","One could also examine whether the server's access to per-client losses leaks information about local data difficulty, a privacy question the paper does not address."],"forward_implications":["If the loss proxy holds, AdaFD's weighting should transfer to other text classification benchmarks without re-engineering the aggregation step.","The two loss-based schemes reach the reported global F1 without LLM API calls, so they are cheaper than the LLM-based variant while matching or beating it.","The best global F1 occurs at five communication rounds and declines with more rounds, so early stopping is a practical component of the method.","The benchmark's protocol—Dirichlet-skewed labels plus domain-specific clients—gives the community a common setting for comparing federated distillation under input-and-output non-IID.","The L2 distillation objective approximates KL divergence as temperature grows, removing the need to tune a temperature parameter."],"supporting_citations":[{"why":"Defines federated learning and the local-update/central-aggregation loop that AdaFD builds on.","marker":"[3]"},{"why":"Federated interactive distillation for large pretrained language models, the line of work AdaFD extends.","marker":"[9]"},{"why":"Introduces federated distillation and augmentation under non-IID private data, establishing the server-side ensemble distillation setup.","marker":"[10]"},{"why":"DS-FL baseline that weights clients by private data size, a direct comparison target for AdaFD.","marker":"[13]"},{"why":"MHAT baseline that weights clients by private data size, another comparison target.","marker":"[14]"},{"why":"FedKD baseline that weights clients by private data size, the third comparison target.","marker":"[15]"},{"why":"Provides the Dirichlet-distribution construction for label non-IID, used to build the benchmark.","marker":"[44]"},{"why":"Supports the motivation that languages differ across domains in vocabulary and communication goals.","marker":"[16]"}],"fun_headline_variants":["Adaptive federated distillation tackles domain and label shifts","Exponential loss weighting improves federated distillation on diverse domains","New benchmark reveals domain diversity in federated text learning","Loss-based adaptive weighting outperforms fixed in federated distillation","Federated distillation adapts to multi-domain non-IID text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a client's minimum local training loss (or its exponential transform) reliably indicates how much that client's predictions should count in the global ensemble; if low loss can coincide with predictions that hurt the global distribution, the adaptive weighting provides no benefit.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive federated distillation tackles domain and label shifts","Exponential loss weighting improves federated distillation on diverse domains","New benchmark reveals domain diversity in federated text learning","Loss-based adaptive weighting outperforms fixed in federated distillation","Federated distillation adapts to multi-domain non-IID text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001355,"raw_usage":{"total_tokens":5493,"prompt_tokens":930,"completion_tokens":4563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":4482}},"tokens_in":546,"tokens_out":4563,"duration_ms":30942,"temperature":1.0,"reasoning_tokens":4482,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:43:07.992423+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AdaFD on the released benchmark while recording, for each round, each client's training-loss rank and the change in global F1 when that client's weight is increased; if clients with lower training loss are not positively correlated with improvements in global F1, or if a deliberately overfitted client with near-zero loss on its own domain receives the largest weight and drags global F1 down, the central claim is falsified.","supporting_citations":[{"cited_title":"FedID: Federated interactive distillation for large-scale pretraining language models,","cited_arxiv_id":null,"evidence_quote":"Federated interactive distillation for large pretrained language models, the line of work AdaFD extends."},{"cited_title":"Communication-efficient on-device machine learning: Federated distil- lation and augmentation under non-iid private data,","cited_arxiv_id":null,"evidence_quote":"Introduces federated distillation and augmentation under non-IID private data, establishing the server-side ensemble distillation setup."},{"cited_title":"Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data,","cited_arxiv_id":null,"evidence_quote":"DS-FL baseline that weights clients by private data size, a direct comparison target for AdaFD."},{"cited_title":"Mhat: An efficient model-heterogenous aggregation training scheme for federated learning,","cited_arxiv_id":null,"evidence_quote":"MHAT baseline that weights clients by private data size, another comparison target."},{"cited_title":"Preserving privacy in federated learning with ensem- ble cross-domain knowledge distillation,","cited_arxiv_id":null,"evidence_quote":"FedKD baseline that weights clients by private data size, the third comparison target."},{"cited_title":"Who speaks what language to whom and when?","cited_arxiv_id":null,"evidence_quote":"Supports the motivation that languages differ across domains in vocabulary and communication goals."}],"review_version":2}