Measuring Heterogeneity in Machine Learning with Distributed Energy Distance

Baocheng Geng; Joseph A. Casey; Keren Li; Mengchen Fan; Roman Shterenberg; Zhong Chen

REVIEW 3 major objections 4 minor 1 cited by

Measuring Heterogeneity in Machine Learning with Distributed Energy Distance

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a Taylor-approximated energy distance, normalized into an Energy Coefficient H, can quantify feature heterogeneity across distributed-learning nodes at linear cost, and that H can set penalty weights for aligning…

desk verdict The FL heterogeneity idea is sensible, but the paper's load-bearing multivariate Taylor approximation is wrong for d>1 and the derivation is deferred, so the current version doesn't support its claims. read the letter →

arxiv 2501.16174 v1 pith:77LX6UU5 submitted 2025-01-27 stat.ML cs.AIcs.DCcs.LG

Mengchen Fan , Baocheng Geng , Roman Shterenberg , Joseph A. Casey , Zhong Chen , Keren Li This is my paper · ORCID

classification stat.MLcs.AIcs.DCcs.LG

keywords energydistancefeatureheterogeneitydistributedlearningfederatedTaylorapproximationCoefficientHnon-IIDdatameasure

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles feature heterogeneity in distributed and federated learning, the setting where different nodes see different marginal distributions of predictors. It proposes the energy distance, normalized as the Energy Coefficient H, as a sensitive measure of these discrepancies, and shows that H is near zero for identically distributed clients and large for deliberately shifted feature distributions. Because the exact energy distance costs O($n^{2}$ d) pairwise operations, the paper derives Taylor approximations using means, variances, skewnesses, and kurtoses that reduce the cost to O(nd), and it reports simulations where these approximations track empirical values while speeding computation. The authors argue that H can act as a penalty weight between guest and host models, allowing heterogeneous nodes to align predictions and improve convergence. The central claim is that moment-based summaries can preserve the quantitative behavior of energy distance at scale.

What carries the argument

The central object is the Energy Coefficient H = $D^{2}$(X,Y)/(2E||X−Y||), a normalized energy distance in [0,1] that is zero only when the two feature distributions are identical. The load-bearing mechanism is the Taylor expansion of g(z) = $\sqrt$(z) around the mean of (X−Y)^2: formula (2) approximates E||X−X'|| as $\sqrt$(2)σ(1−(γ4+4)/16), formula (3) handles one-dimensional X versus Y, and formula (4) extends the idea to d dimensions by summing per-feature moments. This machinery is what converts a quadratic pairwise-distance computation into a linear-cost summary of marginal moments, and every H value reported in the MNIST experiments is computed through it.

What would settle it

Compute the empirical energy distance between two independent samples from a d-dimensional standard normal, with d = 100 and n = $10^{4}$, and compare it with the value predicted by Eq (4); the exact expected distance is about $\sqrt$(2d), while the approximation's leading term is (3/4)$\sqrt$(2d), so a ratio near 1.33 would show a systematic 25 percent gap in the high-dimensional regime the paper targets.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the energy distance, normalized as H = $D^{2}$(X,Y)/(2E||X−Y||), is a sensitive, non-parametric measure of feature heterogeneity among nodes in distributed and federated learning, and that this measure can be made scalable with a moment-based Taylor approximation. The one-dimensional formulas (2) and (3) and the d-dimensional formula (4) express expected Euclidean distances through means, variances, skewnesses, and kurtoses, reducing the empirical energy statistic's O($n^{2}$ d) pairwise computation to O(nd). The paper reports that H computed this way is near zero for identically distributed MNIST clients and large (mean 0.657) when clients receive different feature types, and that the high-H regime coincides with slower, less stable FedAvg convergence. It then argues that H can set the penalty coefficient between guest and host models, making heterogeneity itself a tunable input to coordination.

Load-bearing premise

Everything rests on the assumption that the multivariate squared Euclidean norm can be approximated from marginal moments alone: Eq (4) ignores all covariances between feature dimensions and fixes the Gaussian baseline at 3/4 of sqrt(nu), a coefficient calibrated in one dimension that does not match high-dimensional Gaussians, while the derivation is deferred to an extended version of the manuscript.

Editorial extensions

If this is right

If the approximation is accurate enough, every pair of nodes in a federated system can be assigned an H value in [0,1] at linear cost, turning heterogeneity from an unmeasured nuisance into a monitored quantity.
MNIST experiments indicate that low H tracks stable FedAvg convergence, so H could flag clients whose feature distributions are drifting before accuracy collapses.
Because H is zero only when feature distributions match, it can support hypothesis tests for whether two nodes are sampling from the same distribution without permutation tests.
The proposed penalty-weighting scheme makes H a tunable input to distributed collaborative learning, allowing guest models to align with a host model proportionally to measured feature discrepancy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The approximation's 3/4 Gaussian baseline is univariate in origin: for d-dimensional Gaussians the exact expected distance coefficient tends to 1 as d grows, so Eq (4) may systematically understate distances in high dimension; a Monte Carlo check against d = 100 standard normals would settle this directly.
The paper proposes H-based penalty weights but does not run an experiment that uses them; the convergence results only correlate high H with poor FedAvg performance, leaving open whether H-weighted penalties actually improve alignment and against what baseline.
If H is computed from marginal moments only, the denominator 2E||X−Y|| and numerator D^2 may inherit correlated moment-estimation errors; whether the normalized H is more robust than raw energy distance to such errors is an untested consequence of its construction.
The treatment of representation heterogeneity via one energy distance per response value works for discrete labels but is acknowledged as open for continuous responses; a natural extension would be a functional or kernel-smoothed H over response values.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my read. The novel piece is applying energy distance to feature heterogeneity in federated learning and proposing the normalized coefficient H. That is a sensible diagnostic, and the MNIST experiment does show H separates the mixed from the block-allocated clients. The computational speedup claim is also real: replacing pairwise distances with moment-based estimates is O(nd) instead of O(n^2 d).

The trouble is the paper's load-bearing math. The multivariate Taylor approximation, Eq. (4), is not a valid expansion of E||X-Y|| for d>1. For iid standard Gaussians it predicts (3/4)√(2d) while the exact value is 2Γ((d+1)/2)/Γ(d/2) ≈ √(2d). That's a ~25% error that persists in high dimensions, exactly the regime the paper targets. The formula uses only marginal moments, so it is blind to correlations and not rotation-invariant, though energy distance itself is. This is not a minor gap: every H value in Section 3.2 is computed with Eq. (4), so the heterogeneity numbers are unreliable.

There are other soft spots. The derivation is explicitly deferred to an 'extended version' (Section 2.3), which is honest but means the central claim is unverified. The paper also states energy distance is scale-invariant, which it is not; it scales linearly with the data. The penalty-weighting scheme in Section 4 is proposed but never tested. The single learning experiment shows the known effect that feature heterogeneity hurts FedAvg, not that H drives better aggregation.

What's good: the univariate approximations (2)-(3) are standard delta-method results and are likely fine for near-Gaussian data with small kurtosis. The comparison of H across distributions in Figure 2 is a reasonable sanity check. If the authors correct the multivariate formula, or restrict the method to univariate or independent features, the paper could become useful.

My bottom line: I would not send this to peer review as it stands. The central approximation is demonstrably wrong in the multivariate case, and the authors have not supplied the derivation. The idea is worth a rewrite, but this version doesn't support its claims.

Referee Report

3 major / 4 minor

Summary. The paper proposes using energy distance to measure feature heterogeneity across clients in federated and distributed learning, summarized by a normalized Energy Coefficient H. To avoid the quadratic cost of empirical energy distance, it introduces Taylor approximations based on summary statistics (Eqs. 2–4), reports simulations comparing approximate and exact formulas, and uses the Taylor approximation to compute H for MNIST clients in two feature-distribution regimes. It also proposes, without experiments, that H can be used to set penalty coefficients between guest and host models for better coordination.

Significance. If the Taylor approximations were sound, the O(nd) estimator and the normalized H coefficient would be useful practical tools for distributed learning; the univariate Gaussian formulas (5) and (7) appear correct, and the computational-time comparisons are a reasonable sanity check. However, the multivariate approximation used to compute every H value in the federated experiment has a leading-order error in high dimensions and ignores covariance structure, and the proposed penalty-weighting scheme is untested. The paper's central quantitative claims are therefore not established.

major comments (3)

[Section 2.3, Eq. (4)] The multivariate Taylor approximation is not a valid approximation to E||X−Y|| for d>1. For X,Y iid N(0,I_d), ν_XY=2d and all correction terms vanish, so Eq. (4) gives E||X−Y||≈0.75√(2d). The exact value is E||X−Y||=2Γ((d+1)/2)/Γ(d/2), which tends to √(2d) as d grows; the formula therefore has a relative error of about 25% in high dimensions and is already substantially off at d=10 (3.35 vs 4.36). The formula also uses only marginal moments and the scalar sums δμ1 and δμ2, so it ignores all covariances and is not invariant to orthogonal rotations, contradicting the invariance claimed in Section 2.1. Since Section 2.3 defers the derivation to an extended version, the approximation cannot be verified, and every H value in Section 3.2 inherits this error.
[Section 3.2] The federated-learning experiment computes H with the Taylor approximate method, i.e., with Eq. (4), but provides no validation against the empirical energy statistic (1) or exact expected distances on the MNIST feature space. The conclusion that the mean H of 0.657 'indicates the feature distributions between clients are different' is therefore not supported by independent evidence; at minimum, a comparison with the exact estimator would be needed before drawing any conclusion about heterogeneity or its effect on accuracy.
[Section 4] The proposed use of H to set penalty coefficients between guest and host models is stated only as a possibility ('can be used', 'enables dynamic adjustments'). No algorithm, simulation, or theoretical analysis tests this mechanism, yet the abstract lists 'enhancing coordination' as a contribution. This claim is currently unsupported.

minor comments (4)

[Section 2.1] D^2 is stated to be 'invariant to scaling'; in fact, D^2(cX,cY)=c D^2(X,Y) for common scaling c>0. Only the normalized coefficient H is scale-invariant.
[Section 3.2] The experiment does not specify whether H is computed on raw pixel vectors, what preprocessing is used before computing moments, or how client-pair H values are aggregated; these details are needed for reproducibility.
[Figure 5 caption] The caption says 'MINST' instead of 'MNIST'.
[Section 3.1] The text discusses Bernoulli distributions with p=0.05 and p=0.1, but Bernoulli is not listed in the experimental setup and is not identified in any panel of Figure 2; please clarify whether these results appear in the figure and how they were generated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Taylor approximation and adjusted formulas are derived from moment expansions and external Gaussian baselines, and no reported H value reduces to a fitted input.

full rationale

The paper's central quantities—Eqs. (2)–(4) and the adjusted Eqs. (6) and (8)—are obtained by a second-order Taylor expansion of g(z)=sqrt(z) around a variance-type quantity, or by taking the exact Gaussian expectation Eqs. (5) and (7) as an external baseline and adding skewness/kurtosis correction terms. These formulas are not fitted to the paper's own energy-distance outputs, and none of the H values reported in Section 3.2 is a re-statement of a fitted parameter. The self-citation (Fan et al., 2024) appears only in a list of representation-alignment approaches and is not used to justify the Taylor formulas, the energy coefficient, or the MNIST experiments, so it is not load-bearing. The deferred-derivation note ('Detailed derivations will be provided in the extended version of this manuscript') is a missing-proof or correctness concern, not circularity. Likewise, the dimension-dependent constant 3/4 in Eq. (4) may be inaccurate for d>1, but that is an accuracy issue: it follows from the paper's own univariate expansion rather than from a hidden reuse of the quantity being predicted. No specific reduction by construction or self-citation chain is exhibited, so per the hard rules no circularity is claimed.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on (i) the accuracy of a second-order Taylor expansion of sqrt applied to squared Euclidean distances, (ii) an unstated independence or no-covariance assumption in the multivariate formula, (iii) the correctness of the paper's stated metric properties, and (iv) the proxy validity of H for federated-learning degradation. None of these are provided with proof, and one stated property, scale invariance, is false. No invented entities are introduced.

assumptions (5)

domain assumption The second-order Taylor expansion of the square-root function around the mean of the squared distance is sufficiently accurate.
Invoked in Section 2.3 Eqs (2)-(4). The remainder is not bounded, and Section 3.1 admits notable discrepancies for heavy-tailed distributions such as Bernoulli and exponential.
ad hoc to paper The multivariate energy-distance approximation ignores all covariance structure between feature dimensions.
Eq (4) in Section 2.3 sums marginal moments only and never states an independence assumption; this is false for correlated features such as real images.
domain assumption Energy distance is invariant to scaling, rotation, and translation.
Section 2.1 asserts invariance to scaling; energy distance is homogeneous of degree 1 under scaling, so the stated property is incorrect and underlies later interpretability claims.
domain assumption The energy coefficient H computed on raw features is a reliable proxy for the feature heterogeneity that degrades federated learning.
Section 3.2 constructs two MNIST splits and observes a correlation between H and FedAvg accuracy; no causal or general relationship is established.
standard math The energy-distance test statistic asymptotically follows a known distribution under the null hypothesis.
Section 2.1 relies on the Szekely and Rizzo (2005) result, cited in the paper, to claim direct p-value computation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Measuring Heterogeneity in Machine Learning with Distributed Energy Distance." pith.science (2026). https://pith.science/paper/77LX6UU5

@misc{pith2026250116174,
  author       = {Pith},
  title        = {Pith review of: Measuring Heterogeneity in Machine Learning with Distributed Energy Distance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/77LX6UU5}},
  note         = {Machine review of arXiv:2501.16174}
}

read the original abstract

In distributed and federated learning, heterogeneity across data sources remains a major obstacle to effective model aggregation and convergence. We focus on feature heterogeneity and introduce energy distance as a sensitive measure for quantifying distributional discrepancies. While we show that energy distance is robust for detecting data distribution shifts, its direct use in large-scale systems can be prohibitively expensive. To address this, we develop Taylor approximations that preserve key theoretical quantitative properties while reducing computational overhead. Through simulation studies, we show how accurately capturing feature discrepancies boosts convergence in distributed learning. Finally, we propose a novel application of energy distance to assign penalty weights for aligning predictions across heterogeneous nodes, ultimately enhancing coordination in federated and distributed settings.

Figures

Figures reproduced from arXiv: 2501.16174 by the authors.

**Figure 1.** Computational time. Exponential (β = 0.1, 1, 10), standard Student’s t (df = 5), Beta (α = β = 0.5), and Gamma (k = 1, θ = 2). Two experimental setups were examined: (1) comparing samples drawn from the same type of distribution, and (2) comparing samples from each distribution to those drawn from one of the normal distributions. The computational methods assessed included the empirical energy distance formula (1), … view at source ↗

**Figure 2.** Energy coefficient H for various distributions. These findings underscore the necessity of adapting computational methods based on 10 [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

**Figure 3.** Mixed MNIST Inputs Feature-Based Allocation: Data was distributed by type. For example, Client 1 exclusively received black numerals on white backgrounds and the values were within the specific scale, while Client 2 received white numerals on black backgrounds and the values were within another specific scale [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Different Feature Distribution MNIST Input [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

**Figure 5.** Figure 5: Test accuracy of MINST dataset for Federated Learning with different feature [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

PFedDST: Personalized Federated Learning with Decentralized Selection Training
cs.LG 2025-02 reject novelty 4.0 of 10

A decentralized personalized federated learning method that scores peers by loss, header similarity, and recency reports faster convergence, but its own CIFAR-100 result contradicts the accuracy claim.

Reference graph

Works this paper leans on

21 extracted references · 20 canonical work pages · cited by 1 Pith paper

[1]

Alam, S., L. Liu, M. Yan, and M. Zhang (2022). Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction. Advances in neural information processing systems\/ 35 , 29677--29690

work page 2022
[2]

Chen, J. and A. Zhang (2021). Hetmaml: Task-heterogeneous model-agnostic meta-learning for few-shot learning across modalities. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pp.\ 191--200

work page 2021
[3]

Fan, M., B. Geng, K. Li, X. Wang, and P. K. Varshney (2024). Interpretable data fusion for distributed learning: A representative approach via gradient matching. In 2024 27th International Conference on Information Fusion (FUSION) , pp.\ 1--8

work page 2024
[4]

Cheng, S

Geng, B., X. Cheng, S. Brahma, D. Kellen, and P. K. Varshney (2021). Collaborative human decision making with heterogeneous agents. IEEE Transactions on Computational Social Systems\/ 9\/ (2), 469--479

work page 2021
[5]

Shayea, M

Gures, E., I. Shayea, M. Ergen, M. H. Azmi, and A. A. El-Saleh (2022). Machine learning-based load balancing algorithms in future heterogeneous networks: A survey. IEEE Access\/ 10 , 37689--37717

work page 2022
[6]

Hampel, F. R., E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel (2005). Robust Statistics: The Approach Based on Influence Functions . Wiley Series in Probability and Statistics. Hoboken, NJ, USA: John Wiley & Sons, Inc

work page 2005
[7]

El-Khamy, J

Hao, W., M. El-Khamy, J. Lee, J. Zhang, K. J. Liang, C. Chen, and L. C. Duke (2021). Towards fair federated learning with zero-shot data augmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp.\ 3310--3319

work page 2021
[8]

Kamm, S., S. S. Veekati, T. M \"u ller, N. Jazdi, and M. Weyrich (2023). A survey on machine learning based analysis of heterogeneous data in industrial automation. Computers in Industry\/ 149 , 103930

work page 2023

Show all 21 references

[9]

Li, T., S. Hu, A. Beirami, and V. Smith (2021). Ditto: Fair and robust federated learning through personalization. In International conference on machine learning , pp.\ 6357--6368. PMLR

2021
[10]

Li, T., A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith (2020). Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems\/ 2 , 429--450

2020
[11]

Li, W., X. Wang, G. Li, B. Geng, and P. K. Varshney (2023). Nn-copula-cd: A copula-guided interpretable neural network for change detection in heterogeneous remote sensing images. arXiv preprint arXiv:2303.17448\/

2023 arXiv
[12]

Lin, T., L. Kong, S. U. Stich, and M. Jaggi (2020). Ensemble distillation for robust model fusion in federated learning. Advances in neural information processing systems\/ 33 , 2351--2363

2020
[13]

Zhang, and J

Liu, F., G. Zhang, and J. Lu (2020). Heterogeneous domain adaptation: An unsupervised approach. IEEE transactions on neural networks and learning systems\/ 31\/ (12), 5588--5602

2020
[14]

Shen, J., X. Zhen, Q. Wang, and M. Worring (2023). Episodic multi-task learning with heterogeneous neural processes. Advances in Neural Information Processing Systems\/ 36 , 75214--75228

2023
[15]

Sz \'e kely, G. J. (2003). E-statistics: The energy of statistical samples. Bowling Green State University, Department of Mathematics and Statistics Technical Report\/ 3\/ (05), 1--18

2003
[16]

Sz \'e kely, G. J. and M. L. Rizzo (2005). A new test for multivariate normality. Journal of Multivariate Analysis\/ 93\/ (1), 58--80

2005
[17]

Tyagi, S. and P. Sharma (2020). Taming resource heterogeneity in distributed ml training with dynamic batching. In 2020 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS) , pp.\ 188--194. IEEE

2020
[18]

Lin, and Y.-C

Xu, Y.-Y., C.-S. Lin, and Y.-C. F. Wang (2023). Bias-eliminating augmentation learning for debiased federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp.\ 20442--20452

2023
[19]

Tong, and S

Yan, Y., X. Tong, and S. Wang (2023). Clustered federated learning in heterogeneous environment. IEEE Transactions on Neural Networks and Learning Systems\/

2023
[20]

Zhang, H., Q. Hou, T. Wu, S. Cheng, and J. Liu (2023). Data augmentation based federated learning. IEEE Internet of Things Journal\/

2023
[21]

Xiong, Y

Zheng, L., J. Xiong, Y. Zhu, and J. He (2022). Contrastive learning with complex heterogeneity. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pp.\ 2594--2604

2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Alam, S., L. Liu, M. Yan, and M. Zhang (2022). Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction. Advances in neural information processing systems\/ 35 , 29677--29690

work page 2022

[2] [2]

Chen, J. and A. Zhang (2021). Hetmaml: Task-heterogeneous model-agnostic meta-learning for few-shot learning across modalities. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pp.\ 191--200

work page 2021

[3] [3]

Fan, M., B. Geng, K. Li, X. Wang, and P. K. Varshney (2024). Interpretable data fusion for distributed learning: A representative approach via gradient matching. In 2024 27th International Conference on Information Fusion (FUSION) , pp.\ 1--8

work page 2024

[4] [4]

Cheng, S

Geng, B., X. Cheng, S. Brahma, D. Kellen, and P. K. Varshney (2021). Collaborative human decision making with heterogeneous agents. IEEE Transactions on Computational Social Systems\/ 9\/ (2), 469--479

work page 2021

[5] [5]

Shayea, M

Gures, E., I. Shayea, M. Ergen, M. H. Azmi, and A. A. El-Saleh (2022). Machine learning-based load balancing algorithms in future heterogeneous networks: A survey. IEEE Access\/ 10 , 37689--37717

work page 2022

[6] [6]

Hampel, F. R., E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel (2005). Robust Statistics: The Approach Based on Influence Functions . Wiley Series in Probability and Statistics. Hoboken, NJ, USA: John Wiley & Sons, Inc

work page 2005

[7] [7]

El-Khamy, J

Hao, W., M. El-Khamy, J. Lee, J. Zhang, K. J. Liang, C. Chen, and L. C. Duke (2021). Towards fair federated learning with zero-shot data augmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp.\ 3310--3319

work page 2021

[8] [8]

Kamm, S., S. S. Veekati, T. M \"u ller, N. Jazdi, and M. Weyrich (2023). A survey on machine learning based analysis of heterogeneous data in industrial automation. Computers in Industry\/ 149 , 103930

work page 2023