REVIEW 4 major objections 5 minor 58 references
The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read FedMUD decomposes only the model update — not the full weights — using block-wise Kronecker factors and aggregation-aware products, and converges to a stationary point of the federated objective with a faster bound than FedLMT.
desk verdict Useful paper with three solid techniques; the convergence proof leans on an unverified assumption, so treat the theory as suggestive rather than definitive. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
There are three mechanisms, one per issue. MUD freezes the base parameters and trains low-rank factors $U\in\mathbb{R}^{m\times r}$, $V\in\mathbb{R}^{n\times r}$ whose product is the communication content; $V$ starts at zero so the first transmitted update is exactly the local change. BKD tiles the update into $k^2$ blocks and writes each block as $U\otimes V$ with square factors, which gives a rank bound of $\operatorname{rank}(U)\operatorname{rank}(V)$ per block and dynamic compression through $k$. AAD decouples every trainable product $UV^\top$ into $U\tilde V^\top + \tilde U V^\top$ with fixed random tilde factors, so that the server's average of factors recovers the same matrix as the average of recovered matrices. The convergence proof's load-bearing step is Lemma 1: under the assumption that at least one averaged factor per layer has smallest singular value at least $\psi_{uv}$, the gradient norm in factored space dominates $\psi_{uv}^2\|\nabla f_l(\bar w)\|_F^2$, which converts the factored update into progress on the original objective.
What would settle it
Record the smallest singular values of the averaged factors $\bar U_l, \bar V_l$ across rounds for FedMUD on a non-IID dataset such as CIFAR-100 with Dirichlet-0.1 partitions; if any layer ever has both values at machine precision, Assumption 4 is violated and the stated bound no longer applies. A second test: fix identical total parameter budgets for FedMUD with $s=1$ and FedLMT, and compare required rounds to reach a target accuracy; if the smaller reset interval does not converge faster in practice, Remark 3's claimed advantage is contradicted.
Extended reading notes
Core claim
The central discovery is that the information loss in low-rank federated learning comes from three separate, correctable sources, and the paper corrects all three. Decomposing whole parameters corrupts the static base weights; FedMUD freezes $W$ and learns $U_t(V_t)^\top$ as the update only. Ordinary matrix-product factors cap the recovered rank at $r$; Block-wise Kronecker Decomposition partitions the update into $k^2$ blocks, each represented as a Kronecker product, lifting the rank upper bound to full rank with compression ratio $2k/\sqrt{mn}$. Direct aggregation of factors is biased, because averaging $U_i(V_i)^\top$ over clients differs from $\bar U(\bar V)^\top$ by the cross term $\operatorname{A}(\Delta U)\operatorname{A}(\Delta V)^\top$; Aggregation-Aware Decomposition writes $UV^\top = U\tilde V^\top + \tilde U V^\top$ with fixed $\tilde U,\tilde V$, making the cross term vanish. Theorem 1 bounds the average squared gradient norm by $\frac{2}{\eta c T}(f(\bar w^0)-f(w^*)) + O(\eta^{2-c})\bigl(1+(L-\rho)O(\Gamma_u^4+\Gamma_v^4)\bigr)$, with the $\Gamma$ terms growing in the reset interval $S$; Remark 3 concludes that FedLMT, the $S\ge T$ special case, has a slower rate bound.
Load-bearing premise
At every iteration and every decomposed layer, at least one of the two server-averaged factor matrices must have a smallest singular value bounded away from zero by $\psi_{uv}$, which the paper supports by citing random-matrix theory rather than proving it for aggregated, re-initialized factors. If $\psi_{uv}$ can drift to zero, Lemma 1 and the convergence bound lose their grip.
Editorial extensions
If this is right
- At compression ratio $1/32$, FedMUD+BKD+AAD reaches 89.0%, 86.6%, 75.9%, and 41.2% accuracy on FMNIST, SVHN, CIFAR-10, and CIFAR-100 (Non-IID-1), narrowing the gap to FedAvg to 1.3, 3.1, 5.1, and 8.6 points respectively.
- Setting the reset interval $s=1$ is both the theoretical optimum (smallest $\Gamma_u,\Gamma_v$) and the empirical best choice; increasing $s$ moves the method toward FedLMT's slower behavior.
- AAD is not specific to matrix products or to BKD; the paper states it applies to any product of trainable parameters that has to be aggregated, including Kronecker products, so the technique transfers to other compressed-training schemes.
- With $\rho=L$ the theorem collapses to the FedAvg rate, so FedMUD adds the low-rank machinery without changing the standard non-convex convergence order $O(1/\sqrt{T})$ (when $\eta=1/\sqrt{T}$ and $c\to 1$).
- Compared with FedBAT, a quantization method that compresses only the uplink, FedMUD+BKD+AAD still leads by up to 3.8 points on CIFAR-100 Non-IID-1, suggesting the decomposition approach remains competitive with quantization methods.
Reading between the lines
- The paper's 'faster than FedLMT' argument compares upper bounds (Remark 3) rather than tracking all constants, so a reader should not read it as a proof of a smaller rate exponent; a direct test is to plot FedMUD with $s=1$ against FedLMT at matched parameter counts and compare actual rounds to a target accuracy.
- Assumption 4 is the fragile link: the paper supports it by citing random-matrix theory, but the averaged factors $\bar U, \bar V$ are correlated through training and reinitialization, so the uniform lower bound $\psi_{uv}$ remains an assumption that an empirical trace of smallest singular values could confirm or break.
- Because BKD improves the rank-per-parameter tradeoff that also limits LoRA-style adapters, the block-Kronecker structure suggests a direct extension to parameter-efficient fine-tuning of large models; the paper does not explore that direction.
- The reset mechanism in MUD, which periodically merges the low-rank update into the frozen base and reinitializes the factors, acts like a sequence of low-rank corrections; a natural untested prediction is that the number of resets, not the rank alone, controls the model's effective capacity in the low-communication regime.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes three complementary techniques for communication-efficient federated learning with low-rank decomposition: Model Update Decomposition (MUD), Block-wise Kronecker Decomposition (BKD), and Aggregation-Aware Decomposition (AAD). MUD decomposes only the model update instead of the full model parameters, BKD increases the rank upper bound of the recovered matrix at a fixed compression ratio, and AAD removes the aggregation bias that arises when sub-matrices are averaged directly. The paper presents a convergence theorem for FedMUD under smoothness, bounded-gradient, bounded-factor, and a positive-smallest-singular-value assumption, and it claims that FedMUD converges faster than FedLMT. Experiments on FMNIST, SVHN, CIFAR-10, and CIFAR-100 under non-IID partitions show that the full method FedMUD+BKD+AAD improves accuracy over the baselines, with ablations for reset interval, initialization scale, and compression ratio.
Significance. If the theoretical claims were fully established, this would be a useful contribution to communication-efficient FL: MUD avoids introducing decomposition error into the frozen base model, BKD provides a higher rank upper bound than standard low-rank decomposition, and AAD gives an exact, linear aggregation property (Eqs. 6-9) that is clean and easy to verify. The AAD derivation is crisp, the experimental study is reasonably complete, and the code is released. The convergence proof is structurally coherent under the stated assumptions. However, the central speed-up claim over FedLMT currently rests on comparing upper-bound constants rather than on a proven rate separation, and the proof relies on Assumption 4 being satisfied uniformly over training, which is asserted rather than established for the algorithm's actual iterates. The empirical gains are credible, but they do not by themselves settle the theoretical claims.
major comments (4)
- [Section 4, Assumption 4 (Eq. (28)) and Lemmas 1, 6] Assumption 4 is load-bearing and is not verified for the actual algorithm. It requires a uniform lower bound psi_uv^2 on delta_min(bar_U_l)^2 + delta_min(bar_V_l)^2 at every iteration and every decomposed layer. The text justifies this by Marchenko-Pastur theory, but that theory provides a high-probability lower bound for a single random matrix with i.i.d. entries, whereas bar_U and bar_V are server averages of data-dependent factor matrices that are correlated through training and periodically reinitialized. At reset times bar_V_l = 0, so the condition reduces to delta_min(U0)^2 >= psi_uv^2 for a single shared random initialization; no deterministic or uniform-over-rounds guarantee is given. Lemma 1 and Lemma 6 use this assumption to convert factored-space gradient norms into original-space gradient norms, so Theorem 1 is conditional on an unproven property. If delta_min is small or zero for some layer or round, the learning-rate restriction eta <= (psi_uv^2/2)^(1/(c-1)) forces eta to be very small or the bound to be vacuous. Please provide a proof for the averaged, correlated factors, weaken the assumption to a verifiable condition, or add empirical monitoring of delta_min across all layers and rounds.
- [Appendix D.3, Lemma 4 (Eq. (44)) and Lemma 5 (Eq. (51))] The proof of Lemma 4 uses Gamma_v^2 as if it were a deterministic upper bound on ||V||_F^2, but Lemma 2 only establishes an expectation bound, E||V_{i,l}^t||_F^2 <= Gamma_v^2. In Eq. (44), the step E||A_l||_F^2 <= eta^2 Gamma_v^2 E||sum||^2 is not justified without an almost-sure bound on ||bar_V_l^{t_e}||_F^2 or an independence argument that is not available. The same issue appears in Eq. (51) for Lemma 5, where the bound E||A_l||_F^2 <= eta^2 G^2 Gamma_v^4 is obtained by treating products of expectation-bounded norms as if they were almost-surely bounded. Since Lemma 4 feeds into Lemma 6 and hence into Theorem 1, the convergence proof needs repair, for example by deriving an almost-sure version of Lemma 2 or by adjusting the constants in the stated bounds.
- [Section 4, Remark 3 and Theorem 1 (Eq. (10))] The claim that FedMUD converges faster than FedLMT is not established by Theorem 1. The theorem bounds the average squared gradient norm by a constant depending on Gamma_u and Gamma_v, and Remark 2 notes that Gamma_u and Gamma_v grow with S. Remark 3 then concludes that FedLMT, which corresponds to a much larger S, converges slower. This is a comparison of upper-bound constants, not a proof of a rate separation. Two algorithms can have the same O(1/sqrt(T)) rate with different constants, and a tighter upper bound does not imply faster actual convergence. To substantiate the advertised speed-up, the paper should provide a lower-bound construction or formally prove a rate separation, or it should explicitly state the claim as a comparison of theoretical upper bounds rather than as a proven convergence-rate advantage.
- [Section 4, Remark 1 and Remark 3] The informal 'c -> 1' limit used to claim an O(1/sqrt(T)) rate is not a rigorous corollary of the theorem as stated. For fixed c in (1,2), the learning rate must satisfy eta <= (psi_uv^2/2)^(1/(c-1)); when psi_uv^2/2 < 1, the right-hand side tends to 0 as c approaches 1, so one cannot simply let c -> 1 after setting eta = 1/sqrt(T). A formal statement should specify how c depends on T or assume psi_uv is bounded away from 0 independently of T. In addition, Remark 3 reduces FedMUD to FedLMT by setting S >= T, W_l = 0, and initializing both U and V randomly, whereas the FedMUD algorithm described in Section 3.1 and Appendix D.1 initializes V to zero; the reduction is therefore from a variant of FedMUD rather than from the default method.
minor comments (5)
- [Appendix D.2, Eq. (27)] Assumption 3 appears to contain a typo: the condition ||U_l||_F^2 <= epsilon_v^2 should presumably read ||V_l||_F^2 <= epsilon_v^2, since the second inequality governs the V matrices.
- [Appendix A, Table 3] The entry for EF21-P on CIFAR-100 (IID) reads '28.1(±04.)' and should be '28.1(±0.4)'.
- [Section 5.3] The sentence beginning 'ADD decouples the multiplication...' uses 'ADD' instead of 'AAD'.
- [Appendix D.4, Eq. (71)] The final big-O simplification in Eq. (71) hides terms involving Gamma_u^2 + Gamma_v^2 and Gamma_u^2 Gamma_v^2 inside O(eta^(2-c))(1 + (L-rho)O(Gamma_u^4 + Gamma_v^4)); the authors should clarify how these terms are absorbed into the stated big-O notation.
- [Section 5.4 and Appendix D.1] The reset interval s is defined in rounds in the experiments, while the theory uses S iterations between resets; the paper should state the relationship (e.g., S = s * E) explicitly to avoid confusion.
Circularity Check
No significant circularity: the FedMUD convergence proof is a conditional optimization analysis under explicit assumptions, and the FedLMT comparison is a limiting-case reduction rather than a fitted or self-citational loop.
full rationale
The central derivation (Theorem 1) is built from Lemmas 1-6, which are linear-algebra identities and drift bounds under Assumptions 1-4; no parameter is fitted to data and then reported as a prediction. Assumption 4 is a stated regularity condition, not a consequence of Theorem 1; the paper cites Marchenko-Pastur theory via FedLMT (Liu et al., 2024a) for plausibility, but the cited work is external, not authored by the present authors, so this is imported support rather than a self-citation chain. The relation between FedMUD and FedLMT in Remark 3 is a parameter identification (S >= T, W_l = 0, and random U,V reduces FedMUD to FedLMT), not a circular definition. The empirical comparison includes the authors' own FedBAT baseline, but FedBAT plays no role in the convergence proof and is not load-bearing. The main weaknesses are non-circularity concerns: Assumption 4 is not verified for server-averaged, training-correlated factor matrices (especially at reset times when V=0), and the speed-up over FedLMT in Remark 3 compares upper-bound constants rather than realized convergence rates. These are soundness and inference caveats, not reductions of the claimed result to its inputs.
Assumptions & free parameters
free parameters (3)
- Reset interval s =
1 (default), occasionally 2 or 4
- Initialization scale a for sub-matrices =
Tuned across {0.01, 0.05, 0.1, 0.5, 1, 5, 10}
- Compression ratio / rank r or block count k =
1/32 in main experiments; varying compression tested
assumptions (5)
- standard math Assumption 1: Each loss function f_i is differentiable and L_s-smooth.
- standard math Assumption 2: Stochastic gradients are unbiased with bounded variance and norm.
- domain assumption Assumption 3: Norms of U and V are bounded during initialization and training, with epsilon_u, epsilon_v much smaller than kappa_u, kappa_v.
- ad hoc to paper Assumption 4: At least one of the averaged factor matrices U_bar_l and V_bar_l has a positive smallest singular value, uniformly bounded below by psi_uv, at every iteration.
- domain assumption For BKD, the matrix dimensions are assumed to factor as m = k a^2 and n = k b^2; padding or reshaping is used otherwise.
Cite this review
Pith. "Pith review of The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning." pith.science (2026). https://pith.science/paper/RCLPCXOQ
@misc{pith2026250523176,
author = {Pith},
title = {Pith review of: The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/RCLPCXOQ}},
note = {Machine review of arXiv:2505.23176}
}
read the original abstract
To improve the training efficiency of federated learning (FL), previous research has employed low-rank decomposition techniques to reduce communication overhead. In this paper, we seek to enhance the performance of these low-rank decomposition methods. Specifically, we focus on three key issues related to decomposition in FL: what to decompose, how to decompose, and how to aggregate. Subsequently, we introduce three novel techniques: Model Update Decomposition (MUD), Block-wise Kronecker Decomposition (BKD), and Aggregation-Aware Decomposition (AAD), each targeting a specific issue. These techniques are complementary and can be applied simultaneously to achieve optimal performance. Additionally, we provide a rigorous theoretical analysis to ensure the convergence of the proposed MUD. Extensive experimental results show that our approach achieves faster convergence and superior accuracy compared to relevant baseline methods. The code is available at https://github.com/Leopold1423/fedmud-icml25.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Federated dynamic sparse training: Computing less, communicating less, yet learning better
Bibikar, S., Vikalo, H., Wang, Z., and Chen, X. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022
work page 2022
-
[2]
Large-scale machine learning with stochastic gradient descent
Bottou, L. Large-scale machine learning with stochastic gradient descent. In International Conference on Computational Statistics (COMPSTAT), pp.\ 177--186. Physica-Verlag, 2010
work page 2010
-
[3]
Bouacida, N., Hou, J., Zang, H., and Liu, X. Adaptive federated dropout: Improving communication efficiency and generalization for federated learning. In 2021 IEEE Conference on Computer Communications Workshops, INFOCOM Workshops , pp.\ 1--6. IEEE , 2021
work page 2021
-
[4]
Caldas, S., Kone c n \'y , J., McMahan, H. B., and Talwalkar, A. Expanding the reach of federated learning by reducing client resource requirements. CoRR, abs/1812.07210, 2018
arXiv 2018
-
[5]
On generalization bounds of a family of recurrent neural networks
Chen, M., Li, X., and Zhao, T. On generalization bounds of a family of recurrent neural networks. In The 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), 2020
work page 2020
-
[6]
Dadras, A., Stich, S. U., and Yurtsever, A. Personalized federated learning via low-rank matrix factorization. In Advances in Neural Information Processing Systems (NeurIPS) Workshop OPT, 2024
work page 2024
-
[7]
Dorfman, R., Vargaftik, S., Ben - Itzhak, Y., and Levy, K. Y. Docofl: Downlink compression for cross-device federated learning. In International Conference on Machine Learning (ICML), 2023
work page 2023
-
[8]
Edalati, A., Tahaei, M. S., Kobyzev, I., Nia, V. P., Clark, J. J., and Rezagholizadeh, M. Krona: Parameter efficient tuning with kronecker adapter. In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2023
work page 2023
Show all 58 references
-
[9]
Deep sparse rectifier neural networks
Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp.\ 315--323, 2011
2011
-
[10]
EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compression
Gruntkowska, K., Tyurin, A., and Richt \' a rik, P. EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compression. In International Conference on Machine Learning (ICML), 2023
2023
-
[11]
Lora+: Efficient low rank adaptation of large models
Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models. In International Conference on Machine Learning (ICML), 2024
2024
-
[12]
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR , pp.\ 770--778. IEEE Computer Society, 2016
2016
-
[13]
H \" o nig, R., Zhao, Y., and Mullins, R. D. Dadaquant: Doubly-adaptive quantization for communication-efficient federated learning. In International Conference on Machine Learning (ICML), 2022
2022
-
[14]
J., Shen, Y., Wallis, P., Allen - Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W
Hu, E. J., Shen, Y., Wallis, P., Allen - Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022
2022
-
[15]
Fedpara: Low-rank hadamard product for communication-efficient federated learning
Hyeon - Woo, N., Ye - Bin, M., and Oh, T. Fedpara: Low-rank hadamard product for communication-efficient federated learning. In International Conference on Learning Representations (ICLR), 2022
2022
-
[16]
and Szegedy, C
Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015
2015
-
[17]
Sparse random networks for communication-efficient federated learning
Isik, B., Pase, F., G \" u nd \" u z, D., Weissman, T., and Zorzi, M. Sparse random networks for communication-efficient federated learning. In International Conference on Learning Representations (ICLR), 2023
2023
-
[18]
P., Rebjock, Q., Stich, S
Karimireddy, S. P., Rebjock, Q., Stich, S. U., and Jaggi, M. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning (ICML), 2019
2019
-
[19]
B., Yu, F
Kone c n \'y , J., McMahan, H. B., Yu, F. X., Richt \' a rik, P., Suresh, A. T., and Bacon, D. Federated learning: Strategies for improving communication efficiency. CoRR, 2016
2016
-
[20]
and Hinton, G
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009
2009
-
[21]
and Yang, X
Le, Y. and Yang, X. Tiny imagenet visual recognition challenge, 2015. Stanford CS231n Course Project
2015
-
[22]
Federated learning on non-iid data silos: An experimental study
Li, Q., Diao, Y., Chen, Q., and He, B. Federated learning on non-iid data silos: An experimental study. In International Conference on Data Engineering (ICDE), 2022
2022
-
[23]
Adaptive low-precision training for embeddings in click-through rate prediction
Li, S., Guo, H., Hou, L., Zhang, W., Tang, X., Tang, R., Zhang, R., and Li, R. Adaptive low-precision training for embeddings in click-through rate prediction. In Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI) , Washington, DC, USA, 2023
2023
-
[24]
Masked random noise for communication-efficient federated learning
Li, S., Cheng, Y., Wang, H., Tang, X., Xu, S., Luo, W., Li, Y., Liu, D., He, X., and Li, R. Masked random noise for communication-efficient federated learning. In ACM International Conference on Multimedia (ACM MM), 2024 a
2024
-
[25]
Embedding compression in recommender systems: A survey
Li, S., Guo, H., Tang, X., Tang, R., Hou, L., Li, R., and Zhang, R. Embedding compression in recommender systems: A survey. ACM Comput. Surv. , 56 0 (5): 0 130:1--130:21, 2024 b
2024
-
[26]
Mixed-precision embeddings for large-scale recommendation models
Li, S., Hu, Z., Tang, X., Wang, H., Xu, S., Luo, W., Li, Y., He, X., and Li, R. Mixed-precision embeddings for large-scale recommendation models. arXiv preprint arXiv:2409.20305, 2024 c
2024 arXiv
-
[27]
Fedbat: Communication-efficient federated learning via learnable binarization
Li, S., Xu, W., Wang, H., Tang, X., Qi, Y., Xu, S., Luo, W., Li, Y., He, X., and Li, R. Fedbat: Communication-efficient federated learning via learnable binarization. In International Conference on Machine Learning (ICML), 2024 d
2024
-
[28]
Beyond zero initialization: Investigating the impact of non-zero initialization on lora fine-tuning dynamics
Li, S., Luo, X., Tang, X., Wang, H., Chen, H., Luo, W., Li, Y., He, X., and Li, R. Beyond zero initialization: Investigating the impact of non-zero initialization on lora fine-tuning dynamics. In International Conference on Machine Learning (ICML), 2025 a
2025
-
[29]
Bora: Towards more expressive low-rank adaptation with block diversity
Li, S., Luo, X., Wang, H., Tang, X., Cui, Z., Liu, D., Li, Y., xiuqiang He, and Li, R. Bora: Towards more expressive low-rank adaptation with block diversity. CoRR, 2025 b
2025
-
[30]
On the convergence of fedavg on non-iid data
Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations (ICLR), 2020
2020
-
[31]
Relora: High-rank training through low-rank updates
Lialin, V., Muckatira, S., Shivagunde, N., and Rumshisky, A. Relora: High-rank training through low-rank updates. In International Conference on Learning Representations (ICLR), 2024
2024
-
[32]
Liu, J., Zhou, Y., Wu, D., Hu, M., Guizani, M., and Sheng, Q. Z. Fedlmt: Tackling system heterogeneity of federated learning via low-rank model training with theoretical guarantees. In International Conference on Machine Learning (ICML), 2024 a
2024
-
[33]
Mgr: Multi-generator based rationalization
Liu, W., Wang, H., Wang, J., Li, R., Li, X., Zhang, Y., and Qiu, Y. Mgr: Multi-generator based rationalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 12771--12787, 2023 a
2023
-
[34]
D-separation for causal self-explanation
Liu, W., Wang, J., Wang, H., Li, R., Deng, Z., Zhang, Y., and Qiu, Y. D-separation for causal self-explanation. Advances in Neural Information Processing Systems, 36: 0 43620--43633, 2023 b
2023
-
[35]
Is the mmi criterion necessary for interpretability? degenerating non-causal features to plain noise for self-rationalization
Liu, W., Deng, Z., Niu, Z., Wang, J., Wang, H., Zhang, Y., and Li, R. Is the mmi criterion necessary for interpretability? degenerating non-causal features to plain noise for self-rationalization. Advances in Neural Information Processing Systems, 37: 0 117636--117656, 2024 b
2024
-
[36]
Enhancing the rationale-input alignment for self-explaining rationalization
Liu, W., Wang, H., Wang, J., Deng, Z., Zhang, Y., Wang, C., and Li, R. Enhancing the rationale-input alignment for self-explaining rationalization. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pp.\ 2218--2230. IEEE, 2024 c
2024
-
[37]
Exploring practical gaps in using cross entropy to implement maximum mutual information criterion for rationalization
Liu, W., Deng, Z., Niu, Z., Wang, J., Wang, H., and Li, R. Exploring practical gaps in using cross entropy to implement maximum mutual information criterion for rationalization. Transactions of the Association for Computational Linguistics, 13: 0 577--594, 2025 a
2025
-
[38]
Breaking free from mmi: A new frontier in rationalization by probing input utilization
Liu, W., Deng, Z., Niu, Z., Wang, J., Wang, H., Zeng, Z., and Li, R. Breaking free from mmi: A new frontier in rationalization by probing input utilization. In The Thirteenth International Conference on Learning Representations, 2025 b
2025
-
[39]
Malik, O. A. and Becker, S. Low-rank tucker decomposition of large tensors using tensorsketch. In Advances in Neural Information Processing Systems (NeurIPS), 2018
2018
-
[40]
A survey on lora of large language models
Mao, Y., Ge, Y., Fan, Y., Xu, W., Mi, Y., Hu, Z., and Gao, Y. A survey on lora of large language models. CoRR, 2024
2024
-
[41]
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2017
2017
-
[42]
Resource-adaptive federated learning with all-in-one neural composition
Mei, Y., Guo, P., Zhou, M., and Patel, V. Resource-adaptive federated learning with all-in-one neural composition. In Advances in Neural Information Processing Systems (NeurIPS), 2022
2022
-
[43]
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2011
2011
-
[44]
Novikov, A., Izmailov, P., Khrulkov, V., Figurnov, M., and Oseledets, I. V. Tensor train decomposition on tensorflow (T3F) . CoRR, 2018
2018
-
[45]
Qiu, X., Fern \' a ndez - Marqu \' e s, J., de Gusmao, P. P. B., Gao, Y., Parcollet, T., and Lane, N. D. Zerofl: Efficient on-device training for federated learning with local sparsity. In International Conference on Learning Representations (ICLR), 2022
2022
-
[46]
Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., and Pedarsani, R. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020
2020
-
[47]
N., Kingsbury, B., Sindhwani, V., Arisoy, E., and Ramabhadran, B
Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., and Ramabhadran, B. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In IEEE International Conference on Acoustics, Speech and Signal Processing, (ICASSP) , 2013
2013
-
[48]
Stripelis, D., Gupta, U., Ver Steeg, G., and Ambite, J. L. Federated progressive sparsification (purge-merge-tune)+. In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2022
2022
-
[49]
Improving lora in privacy-preserving federated learning
Sun, Y., Li, Z., Li, Y., and Ding, B. Improving lora in privacy-preserving federated learning. In International Conference on Learning Representations (ICLR), 2024
2024
-
[50]
Doublesqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression
Tang, H., Yu, C., Lian, X., Zhang, T., and Liu, J. Doublesqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression. In International Conference on Machine Learning (ICML), 2019
2019
-
[51]
Decoupling general and personalized knowledge in federated learning via additive and low-rank decomposition
Wu, X., Liu, X., Niu, J., Wang, H., Tang, S., Zhu, G., and Su, H. Decoupling general and personalized knowledge in federated learning via additive and low-rank decomposition. In ACM International Conference on Multimedia (ACM MM), 2024
2024
-
[52]
Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, 2017
2017
-
[53]
Communication-efficient federated learning with binary neural networks
Yang, Y., Zhang, Z., and Yang, Q. Communication-efficient federated learning with binary neural networks. IEEE J. Sel. Areas Commun. , 39 0 (12): 0 3836--3850, 2021
2021
-
[54]
Fedhm: Efficient federated learning for heterogeneous models via low-rank factorization
Yao, D., Pan, W., Wan, Y., Jin, H., and Sun, L. Fedhm: Efficient federated learning for heterogeneous models via low-rank factorization. CoRR, 2021
2021
-
[55]
Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning
Yu, H., Yang, S., and Zhu, S. Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2019
2019
-
[56]
H., Hoang, T
Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K. H., Hoang, T. N., and Khazaeni, Y. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning (ICML), 2019
2019
-
[57]
M., and Gu, Q
Zou, D., Long, P. M., and Gu, Q. On the global convergence of training deep linear resnets. In International Conference on Learning Representations (ICLR), 2020
2020
-
[58]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.