REVIEW 3 major objections 4 minor 12 references
Capacity Confounds and Coverage Guarantees in Adaptive Sub-model Federated Learning
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Sub-model federated learning cannot recover client data heterogeneity from update divergence once device capacities differ: across two corrected estimators, multiple datasets, and all seeds, the estimates are dominated by capacity…
desk verdict Solid negative result with a load-bearing overclaim: the capacity-confound interpretation needs an equal-capacity positive control that the paper never runs. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the normalized gradient divergence estimator $\hat{H}^{(t)}_i = \|m^{(t)}_i \odot (\Delta^{(t)}_i - \bar{\Delta}^{(t)})\|^2 / (\|m^{(t)}_i \odot \bar{\Delta}^{(t)}\|^2 + \epsilon)$, computed on the coordinates client $i$ trained, smoothed by an exponential moving average, and fed into the allocation rule $p^{(t)}_i = \min(p^{\max}_i, p_{\min} + \gamma \tilde{H}^{(t)}_i / (\bar{H}^{(t)} + \epsilon))$. The load-bearing pieces are the nested width masks inherited from HeteroFL-style sub-model extraction, the coverage guarantee that restores full width to the largest-capacity clients, and Proposition 3, which proves that under capped allocation the uncovered coordinates stay frozen at initialization. Together they show that sub-model updates differ across width tiers by an order of magnitude in size and norm, dwarfing any data-driven variation; that frozen output-path parameters are what make uniform allocation collapse; and that coverage plus budget, not heterogeneity targeting, carries the accuracy.
What would settle it
Run the same federated training protocol with all clients at equal full capacity while keeping the same ground-truth label-divergence partition, and correlate the update-divergence estimates with total-variation divergence across clients; a clearly positive correlation would support the capacity-confound interpretation, whereas a near-zero or negative one would show the estimator fails to measure data heterogeneity even without capacity variation, and the confound framing would need revision.
Extended reading notes
Core claim
The paper's central claim is that update-based estimators of client heterogeneity are confounded by capacity whenever sub-model width varies with device capability. In the system-heterogeneous setting, the smoothed normalized gradient divergence $\tilde{H}^{(t)}_i$ correlates with device capacity at $r$ between $-0.84$ and $-0.72$ across two corrected estimators, two datasets, and all seeds, while partial correlations with ground-truth label divergence after removing capacity are near zero or negative, so the allocations do not track true heterogeneity. A second claim is structural: capped allocation freezes uncovered parameters, since coordinates outside the widest trained slice remain at random initialization for the entire run while participating in every forward pass (Proposition 3), progressively corrupting the global model, and a coverage guarantee that assigns full width to the highest-capacity clients eliminates the degradation. A third claim, established by a matched-budget control, is that adaptive allocation contributes nothing beyond its capacity budget: random time-varying allocation at the same average capacity matches HAS-FL on CIFAR-10 and EMNIST and beats it on Shakespeare, where the adaptive policy uses the most capacity and achieves the lowest accuracy.
Load-bearing premise
The load-bearing premise is that the update-divergence estimator would recover data heterogeneity if device capacities were equal, a positive control the paper never runs, leaving open that the estimator is simply a weak data signal in general.
Editorial extensions
If this is right
- Any method that estimates client data statistics from sub-model updates in a system-heterogeneous federation is measuring capacity rather than data, so heterogeneity-aware allocation built on such estimates cannot work as intended.
- Methods that size or shape sub-models from training-derived signals, such as capability-driven pruning ratios, data-driven channel importance, magnitude-based composition, or learned sparse ratios, inherit the same confound and should validate against a capacity-stratified analysis.
- Parameter coverage is the decisive design element in sub-model federated learning: preserving coverage of output-path parameters prevents the freeze-at-initialization collapse that destroys uniform allocation on high-class-count tasks.
- Adaptive allocation schemes should be reported against a matched-budget random control, which in these experiments fully explained the benefits attributed to the adaptive policy.
- Sub-model training still buys the participation of resource-constrained devices at quadratically reduced compute and communication cost, but at a genuine accuracy cost of roughly 8 to 13 points against full-model training.
Reading between the lines
- Likely generalization beyond the paper's measurements: the confound should afflict any update statistic, including norms, cosine similarity, and direction agreement, because the mechanism is the order-of-magnitude gap in update size and norm across width tiers, so equal-model-size clustering and client-selection methods remain safe only when every client trains the identical architecture.
- Testable extension: an equal-capacity federation with the same label-divergence partition would separate the two readings of the paper's negative result, with a positive correlation there completing the confound story and a null result indicating divergence-based heterogeneity signals should be abandoned outright.
- Natural next step the paper names but does not pursue: estimating data statistics within capacity strata, using fixed-width probe batches, or having clients report statistics directly, with the paper's reproducible-partition partial-correlation protocol as the evaluation harness.
- Design rule suggested by the coverage analysis: any federated aggregation should guarantee that every parameter participating in inference is eventually updated by some client, with priority on output-path parameters, since frozen interior units degrade less than frozen classifier coordinates.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper uses HAS-FL, an adaptive sub-model federated-learning framework, as a test case for a broader question: can a server estimate client data heterogeneity from sub-model updates when client model width varies with device capacity? The authors report three findings: (i) update-divergence estimates are dominated by capacity rather than by label-distribution divergence, based on correlations with ground-truth TV distance and capacity constraints in Table 5; (ii) capped allocation can leave parameters frozen at initialization, formalized as Proposition 3, and a coverage guarantee removes the resulting collapse; (iii) a matched-budget random-allocation control matches or outperforms the heterogeneity-aware policy on three benchmarks. The paper concludes that the apparent benefits of adaptive sub-model allocation come from capacity budgeting and parameter coverage rather than from heterogeneity estimation.
Significance. If fully established, the capacity-confound result is an important negative result: it would invalidate a natural and repeatedly suggested design direction in system-heterogeneous federated learning. The paper has real strengths: Proposition 3 is a direct and machine-checkable proof, the estimator is validated against ground-truth label divergence computed from actual partitions, experiments are multi-seed and reproducible with released code, and the matched-budget random control is a strong and appropriate comparison. However, the paper's strongest claim is currently overstated relative to its own numbers, and the confound interpretation would be substantially strengthened by an equal-capacity positive control that the paper does not run.
major comments (3)
- [Abstract and Section 4.4 (Table 5)] The abstract and Section 4.4 claim that 'no data signal remains once capacity is controlled for,' but Table 5 reports partial correlations r(H, TV | p_max) of -0.60, -0.47, -0.40, -0.26, -0.28, and 0.10 across the six runs. These values are not all near zero, and two are substantial in magnitude; only one is positive. The evidence supports 'no consistent positive data signal' or 'a weak and inconsistent data signal,' not 'no data signal remains.' Similarly, the text's statement that r(p, TV | p_max) is approximately zero is contradicted by the -0.36 and -0.46 entries in the same table. These are the paper's headline quantitative claims, so the wording and conclusions should be revised to match the reported numbers.
- [Section 4.4] The central 'confound' interpretation is missing a positive control: an experiment in which client capacities are held equal across clients while label divergence still varies, using the same estimator. Without such a baseline, the strong negative correlations between H and p_max are equally consistent with the alternative explanation that the proposed update-divergence estimator is simply a poor or negatively biased proxy for label divergence under sub-model training, independent of capacity variation. Since Section 4.7 explicitly frames the paper as a warning about capacity changing the meaning of update-based signals, the paper should either add the equal-capacity control or explicitly limit the claim to the system-heterogeneous setting.
- [Abstract and Section 4.4] The abstract and Section 4.4 claim that the confound persists 'across two corrected estimators,' and the text says 'we verified that both the coordinate-restricted and the common-core variants of the estimator exhibit the coupling.' However, only one estimator is defined (Eq. (4)), and Table 5 reports a single set of correlations. The common-core variant is never defined and its results are not reported. Either define the second estimator and give its correlations, or revise the claim to refer to one estimator.
minor comments (4)
- [Table 4 and Section 4.3] The text states that HeteroFL provides 'full coverage by construction,' but with 10 of 20 clients sampled per round, a static policy cannot guarantee per-round coverage unless a full-width client is sampled in every round. If 'full coverage' is meant in the union-of-masks sense over the whole federation, please state this explicitly; otherwise clarify how HeteroFL avoids rounds in which no sampled client covers the outer channels.
- [Section 4.1 and Eq. (6)] The allocation rule depends on free parameters p_min, gamma, beta, T_adapt, and T_norm, with values p_min=0.4 and gamma=0.25 chosen without a sensitivity analysis. Since the central negative result concerns the estimator rather than the allocation rule, this is not blocking, but a sentence on the robustness of the conclusions to these choices would help.
- [Table 5] The seed labels s42 through s44 are unusual and the paper does not explain why seeds are not numbered 1-3. Please clarify, since the reader cannot tell whether these are three separate initializations or a subset of a larger set.
- [Section 4.6 and Table 7] The Shakespeare benchmark uses only 20 role-clients, and the paper acknowledges this limitation. It would be useful to state explicitly whether the three seeds correspond to different train/test splits of the same roles or to different roles altogether, since this affects the interpretation of the reported standard deviations.
Circularity Check
No circularity: the central confound result is validated against external ground truth, Proposition 3 follows from definitions, and the matched-budget control is an independent comparison.
full rationale
The paper's central claim—that update-divergence estimates of client heterogeneity are dominated by device capacity—is not circular. In Section 4.4 (Table 5), the estimator \hat H_i from Eq. (4) is validated against ground-truth total-variation divergence computed from the actual Dirichlet-generated label partitions, an external benchmark independent of the estimator's construction. The negative correlations with p_max and the partial correlations r(\tilde H, TV | p_max) are reported empirical quantities, not fitted targets. The 'no data signal remains' interpretation is a causal claim that is arguably under-supported by the missing equal-capacity positive control—a correctness/validity concern, not circularity—but the numerical finding itself is not derived from its own conclusion. Proposition 3 is a direct corollary of the nested-mask construction (Algorithm 2) and the coordinate-wise aggregation rule (Algorithm 1, line 18): if no client mask covers a coordinate, its aggregated update is zero, and induction from w^(0) gives the stated result. This is a theorem from definitions, not a prediction fitted to observations. The matched-budget random control compares HAS-FL to a random allocation drawn with the same average capacity; the control is an independent intervention, and matching the budget is the point of the experiment, not a way of forcing the outcome. The paper contains no load-bearing self-citations: no cited 'uniqueness theorem' or prior work by the present authors is invoked to forbid alternative explanations. The 'inherited risk' discussion in Section 4.7 classifies other methods by whether their model size varies, which is a logical inference from the measured confound, not a renaming of the result under new coordinates. Overall, the derivation chain is self-contained, with the main weakness being an empirical identification gap rather than circularity.
Assumptions & free parameters
free parameters (5)
- p_min =
0.4
- gamma =
0.25
- beta =
0.9
- T_adapt =
5
- T_norm =
20
assumptions (5)
- domain assumption Nested width masks: a client with capacity p trains the leading p-fraction of channels in every layer, following Algorithm 2 and the HeteroFL/FjORD construction.
- domain assumption Ground-truth heterogeneity is the total-variation (TV) distance between a client's label distribution and the global label distribution.
- domain assumption The update-divergence estimator of Eq. (4) would recover data heterogeneity in an equal-capacity federation (positive control).
- domain assumption At least one client has p_max equal to 1.0 so the coverage guarantee can restore full coverage.
- standard math Standard federated optimization assumptions: local SGD steps, bounded gradients implicitly, and i.i.d. mini-batch sampling per client.
Cite this review
Pith. "Pith review of Capacity Confounds and Coverage Guarantees in Adaptive Sub-model Federated Learning." pith.science (2026). https://pith.science/paper/CGXHKH2O
@misc{pith2026260807157,
author = {Pith},
title = {Pith review of: Capacity Confounds and Coverage Guarantees in Adaptive Sub-model Federated Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/CGXHKH2O}},
note = {Machine review of arXiv:2608.07157}
}
read the original abstract
Sub-model federated learning lets resource-constrained clients train width-reduced versions of a global model, but existing methods allocate capacity by device resources alone. A natural next step, allocating capacity by each client's data heterogeneity as estimated from the updates the server already observes, has been repeatedly suggested. We ask whether that step is possible, using HAS-FL, an adaptive capacity-allocation framework, as a test case. Our findings are threefold. First, validated against ground-truth label-distribution divergence on reproducible partitions, update-divergence estimates of client heterogeneity are dominated by capacity rather than data: across two corrected estimators, multiple datasets, and all seeds, the estimates correlate strongly and negatively with device capacity, and no data signal remains once capacity is controlled for. This previously undocumented confound affects any method estimating client statistics from sub-model updates. Second, adaptive allocation has a hidden failure mode: when every client is capped below full width, the uncovered parameters stay at random initialization and progressively corrupt the global model. A simple coverage guarantee removes the failure and explains why uniform allocation collapses. Third, a matched-budget control settles what adaptivity contributes: random allocation to the same average budget performs no differently on both image benchmarks, and on the naturally partitioned text benchmark the adaptive policy is the weakest of the three strategies while consuming the most capacity. Sub-model training remains valuable because it admits constrained clients at quadratically reduced cost, but what protects accuracy is parameter coverage rather than allocation intelligence. Its apparent benefits come from capacity budgeting and coverage, and future designs need heterogeneity signals separable from capacity effects.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[3]
Exploiting shared representations for personalized federated learning,
27 Moayedikia and Troncoso Lora L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, “Exploiting shared representations for personalized federated learning,” inInternational Conference on Machine Learning, 2021, pp. 2089–2099. S. Horvath, S. Laskaridis, M. Alber, and N. D. Lane, “FjORD: Fair and accurate feder- ated learning under heterogeneous targets...
work page 2021
-
[8]
On the convergence and stability of distributed sub- model training,
Y. Deng, F. Qiao, and M. Mahdavi, “On the convergence and stability of distributed sub- model training,”arXiv preprint arXiv:2511.06132,
-
[9]
Communication-efficient federated learning via knowledge distillation,
C. Wu, F. Wu, L. Lyu, Y. Huang, and X. Xie, “Communication-efficient federated learning via knowledge distillation,”Nature Communications, vol. 13, no. 1, p. 2032,
-
[12]
Client Selection in Federated Learning based on Gradients Importance
O. Marnissi, H. El Hammouti, and E. H. Bergou. Client selection in federated learning based on gradients importance.arXiv preprint arXiv:2111.11204,
-
[2009]
Adaptive personalized federated learning,
Y. Deng, M. M. Kamani, and M. Mahdavi, “Adaptive personalized federated learning,” arXiv preprint arXiv:2003.13461,
arXiv 2003
-
[2017]
Sparse communication for distributed gradient descent,
A. F. Aji and K. Heafield, “Sparse communication for distributed gradient descent,” in Conference on Empirical Methods in Natural Language Processing, 2017, pp. 440–445. 28 Capacity Confounds in Adaptive Sub-model Federated Learning T. Vogels, S. P. Karimireddy, and M. Jaggi, “PowerSGD: Practical low-rank gradient com- pression for distributed optimizatio...
work page 2017
-
[2020]
Federated learning for mobile keyboard prediction,
A. Hard, K. Rao, R. Mathews, et al., “Federated learning for mobile keyboard prediction,” arXiv preprint arXiv:1811.03604,
-
[2021]
LEAF: A benchmark for federated settings,
S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Koneˇ cn´ y, H. B. McMahan, V. Smith, and A. Tal- walkar, “LEAF: A benchmark for federated settings,”arXiv preprint arXiv:1812.01097,
Show all 12 references
-
[2022]
Distributed learning of deep neural networks using independent subnet training,
B. Yuan, C. R. Wolfe, C. Dun, Y. Tang, A. Kyrillidis, and C. M. Jermaine, “Distributed learning of deep neural networks using independent subnet training,”arXiv preprint arXiv:1910.02120,
1910 arXiv
-
[2023]
FedCP: Separating feature information for personalized federated learning via conditional policy,
J. Zhang, Y. Hua, H. Wang, T. Song, Z. Xue, R. Ma, and H. Guan, “FedCP: Separating feature information for personalized federated learning via conditional policy,” inACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 3249–3261. Y. Tan, G. Long, L. Liu, T. ...
2023
-
[2024]
J. Xue, S. Sun, M. Liu, Y. Wang, Z. Liu, and J. Wang. Learnable sparse customization in heterogeneous edge computing.arXiv preprint arXiv:2412.07216,
-
[2026]
Efficient personalized federated learning via sparse model-adaptation,
D. Chen, L. Yao, D. Gao, B. Ding, and Y. Li, “Efficient personalized federated learning via sparse model-adaptation,” inInternational Conference on Machine Learning, 2023, pp. 5234–5256. A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning with theoretical...
2023
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.