REVIEW 2 major objections 2 minor 38 references
OSDTW: Optimal Shared Depth and Task Weighting for Long-Tailed Recognition
T0 review · 2 major / 2 minor · reviewed 2026-06-30 · grok-4.3
Pith's one-line read OSDTW derives a bias-variance decomposition of generalization error to select shared encoder depth and task weights for long-tailed recognition.
desk verdict OSDTW gives a Fisher-derived proxy for choosing shared depth and task weights in long-tailed nets, but the block-diagonal approximation is the part that needs checking. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The factorized model for head and tail label groups, which lets the generalization error be written as the sum of task-wise terms, together with the block-diagonal Fisher approximation that produces the second-order bias-variance expansion.
What would settle it
On a standard long-tailed benchmark, compare test accuracy obtained with depth and weights chosen by minimizing the decomposed error terms against accuracy obtained with heuristic or grid-search choices; the proxy is falsified if the selected values do not improve or match the best heuristic performance.
Extended reading notes
Core claim
Under a block-diagonal Fisher approximation, the expected generalization error of the factorized model admits a computable second-order expansion that decomposes into encoder variance, encoder bias, and decoder variance; this bias-variance decomposition serves as a proxy for selecting the shared depth and task weights in the three-stage training pipeline.
Load-bearing premise
The block-diagonal Fisher approximation is accurate enough that the second-order error expansion reliably ranks choices of shared depth and task weights.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that long-tailed recognition can be improved by decomposing the problem into head and tail tasks via a shared encoder and task-specific decoders. Under a factorized model, the KL-divergence generalization error decomposes additively into task-wise terms. A three-stage pipeline (independent task training, weighted joint training, branch assembly) is proposed, and under a block-diagonal Fisher approximation a second-order expansion of expected generalization error is derived that decomposes into encoder variance, encoder bias, and decoder variance; this expansion serves as a computable proxy for selecting shared depth and task weights. Experiments on standard long-tailed benchmarks are reported to outperform strong baselines.
Significance. If the block-diagonal Fisher-based proxy reliably ranks shared-depth and task-weight choices, the work would replace heuristic hyper-parameter selection with a principled, bias-variance grounded procedure. The explicit decomposition into encoder/decoder terms could also clarify the sources of the head-tail trade-off. The three-stage pipeline and factorized KL argument are technically interesting even if the proxy requires further validation.
major comments (2)
- [Derivation of second-order expansion (under block-diagonal Fisher)] The central claim that the second-order expansion supplies an independent, computable proxy for hyper-parameter selection rests on the block-diagonal Fisher approximation. Because the encoder is shared, head and tail gradients are coupled through the same parameters; any residual off-block entries in the true Fisher (arising from feature overlap or class dependence) would invalidate the additive variance/bias decomposition used for ranking. No quantitative bound on the approximation error or regime of validity is supplied in the derivation.
- [Factorized model and task-wise objective] The factorized model is invoked both to obtain additive KL terms and to justify block-diagonality. The mutual-exclusivity assumption between head and tail label groups is plausible for the output layer but does not automatically extend to the shared encoder features; residual statistical dependence would couple the encoder-variance term across tasks and undermine the proxy.
minor comments (2)
- [Abstract and experimental section] The abstract states that the proxy 'enables efficient hyper-parameter search' but provides no concrete complexity comparison or timing results; a table or paragraph quantifying the cost of the three-stage pipeline versus grid search would strengthen the efficiency claim.
- [Notation and preliminaries] Notation for the task-wise objectives and the Fisher blocks should be introduced with explicit equation numbers in the main text rather than only in appendices, to make the bias-variance decomposition self-contained.
Simulated Author's Rebuttal
We thank the referee for the detailed and constructive review. We address each major comment below and describe the revisions we will make.
read point-by-point responses
-
Referee: [Derivation of second-order expansion (under block-diagonal Fisher)] The central claim that the second-order expansion supplies an independent, computable proxy for hyper-parameter selection rests on the block-diagonal Fisher approximation. Because the encoder is shared, head and tail gradients are coupled through the same parameters; any residual off-block entries in the true Fisher (arising from feature overlap or class dependence) would invalidate the additive variance/bias decomposition used for ranking. No quantitative bound on the approximation error or regime of validity is supplied in the derivation.
Authors: We acknowledge that the block-diagonal Fisher approximation is central to the additive decomposition and that no theoretical bound on the approximation error is derived. The shared encoder does induce parameter coupling, and off-block terms could in principle affect the ranking. While a general quantitative bound is technically difficult to obtain for deep networks, we will revise the manuscript to add an empirical analysis section that quantifies the magnitude of off-diagonal Fisher blocks on the benchmark datasets and discusses the observed regimes of validity. This will support the practical reliability of the proxy without claiming theoretical guarantees. revision: yes
-
Referee: [Factorized model and task-wise objective] The factorized model is invoked both to obtain additive KL terms and to justify block-diagonality. The mutual-exclusivity assumption between head and tail label groups is plausible for the output layer but does not automatically extend to the shared encoder features; residual statistical dependence would couple the encoder-variance term across tasks and undermine the proxy.
Authors: The factorized model p(y|x) = p_head(y_head|x) p_tail(y_tail|x) is introduced specifically to establish additive decomposition of the KL-based generalization error at the task level; it does not claim to remove all statistical dependence in the shared encoder features. The block-diagonal Fisher approximation is applied separately as a computational device for the bias-variance proxy. We will revise the text to clarify this distinction, explain how the three-stage pipeline (independent pre-training followed by weighted joint training) reduces feature-level coupling in practice, and note that the proxy remains an empirical tool whose utility is demonstrated by improved hyper-parameter selection over heuristics. revision: yes
Circularity Check
Derivation chain is self-contained with no reductions to inputs by construction
full rationale
The provided abstract and description show a forward derivation: factorized model yields additive KL terms for task-wise objectives, followed by an independent block-diagonal Fisher approximation enabling a second-order expansion into encoder/decoder variance and bias terms. This expansion is then used as a proxy for hyper-parameter selection. No quoted equations or steps reduce the proxy back to fitted inputs, self-citations, or ansatzes by definition; the selection mechanism retains independent grounding from the stated approximations rather than tautologically reproducing the original objective.
Assumptions & free parameters
free parameters (2)
- shared depth
- task weights
assumptions (2)
- domain assumption Factorized model captures mutual exclusivity and statistical dependence between head and tail label groups
- ad hoc to paper Block-diagonal Fisher approximation is accurate enough for the second-order expansion
Cite this review
Pith. "Pith review of OSDTW: Optimal Shared Depth and Task Weighting for Long-Tailed Recognition." pith.science (2026). https://pith.science/paper/WZIMGCDW
@misc{pith2026260524969,
author = {Pith},
title = {Pith review of: OSDTW: Optimal Shared Depth and Task Weighting for Long-Tailed Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/WZIMGCDW}},
note = {Machine review of arXiv:2605.24969}
}
read the original abstract
Long-tailed recognition suffers from a persistent head--tail trade-off: improving tail performance often degrades head accuracy and can increase training instability. Despite strong empirical results from re-weighting, decoupled training, and multi-expert methods, key design choices about representation sharing between head and tail classes and supervision weighting across class groups remain largely heuristic. In this work, we propose OSDTW, a principled task-decomposition framework that partitions the original single-label recognition problem into a head task and a tail task, implemented with a shared encoder and task-specific decoders. To handle the mutual exclusivity and statistical dependence between the two label groups, we introduce a factorized model and show that the resulting Kullback--Leibler divergence-based generalization error can be written as the sum of task-wise terms up to an additive constant, yielding a well-defined task-wise objective. We further develop a three-stage training pipeline: independent task training to estimate task-wise optima and the Fisher information matrix, weighted joint training to learn a shared encoder, and branch assembly to construct the final decoupled model. Under a block-diagonal Fisher approximation, we derive a computable second-order expansion of the expected generalization error, decomposing it into encoder variance, encoder bias, and decoder variance. This bias--variance decomposition provides a computable proxy to select the shared depth and task weights, enabling efficient hyper-parameter search. Experiments on standard long-tailed benchmarks demonstrate the effectiveness of the proposed approach over strong baselines.
Figures
Reference graph
Works this paper leans on
-
[1]
Advances in Neural Information Pro- cessing Systems32(2019)
Cao, K., Wei, C., Gaidon, A., Arechiga, N., Ma, T.: Learning imbalanced datasets with label-distribution-aware margin loss. Advances in Neural Information Pro- cessing Systems32(2019)
work page 2019
-
[2]
In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Chen, S., Hou, Y., Cui, Y., Che, W., Liu, T., Yu, X.: Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 7870–7881 (2020)
work page 2020
- [3]
-
[4]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Cui, J., Zhong, Z., Liu, S., Yu, B., Jia, J.: Parametric contrastive learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 715–724 (2021)
work page 2021
-
[5]
IEEE Transactions on Pattern Analysis and Machine Intelligence 46(12), 7463–7474 (2023)
Cui, J., Zhong, Z., Tian, Z., Liu, S., Yu, B., Jia, J.: Generalized parametric con- trastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 46(12), 7463–7474 (2023)
work page 2023
-
[6]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Cui, Y., Jia, M., Lin, T.Y., Song, Y., Belongie, S.: Class-balanced loss based on effective number of samples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9268–9277 (2019)
work page 2019
-
[7]
In: In- ternational Conference on Learning Representations (2021),https://openreview
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: In- ternational Conference on Learning Representations (2021),https://openreview. net/forum?id=YicbFdNTTy
work page 2021
-
[8]
In: The Twelfth In- ternational Conference on Learning Representations (2024),https://openreview
Guo, H., Greengard, P., Xing, E., Kim, Y.: LQ-loRA: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In: The Twelfth In- ternational Conference on Learning Representations (2024),https://openreview. net/forum?id=xw29VvOMmU
work page 2024
Show all 38 references
-
[9]
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalablevisionlearners.In:ProceedingsoftheIEEE/CVFConferenceonComputer Vision and Pattern Recognition. pp. 16000–16009 (2022) 14 C. Chu et al
2022
-
[10]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Hong, Y., Han, S., Choi, K., Seo, S., Kim, B., Chang, B.: Disentangling label distribution for long-tailed visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6626–6636 (2021)
2021
-
[11]
Hsu,Y.C.,Hua,T.,Chang,S.,Lou,Q.,Shen,Y.,Jin,H.:Languagemodelcompres- sionwithweightedlow-rankfactorization.In:InternationalConferenceonLearning Representations (2022),https://openreview.net/forum?id=uPv9Y3gmAI5
2022
-
[12]
In: International Conferenceon LearningRepresentations(2020),https://openreview.net/forum? id=r1gRTCVFvB
Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., Kalantidis, Y.: De- coupling representation and classifier for long-tailed recognition. In: International Conferenceon LearningRepresentations(2020),https://openreview.net/forum? id=r1gRTCVFvB
2020
-
[13]
In: International Confer- ence on Machine Learning
Kim, S., Hooper, C.R.C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M.W., Keutzer, K.: Squeezellm: Dense-and-sparse quantization. In: International Confer- ence on Machine Learning. pp. 23901–23923. PMLR (2024)
2024
-
[14]
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny im- ages. Tech. rep., University of Toronto, Toronto, Ontario (2009),https://www.cs. toronto.edu/~kriz/learning-features-2009-TR.pdf
2009
-
[15]
In: International Conference on Learning Representa- tions (2020),https://openreview.net/forum?id=HkgaETNtDB
Lee, C., Cho, K., Kang, W.: Mixout: Effective regularization to finetune large-scale pretrained language models. In: International Conference on Learning Representa- tions (2020),https://openreview.net/forum?id=HkgaETNtDB
2020
-
[16]
In: Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition
Li, M., Cheung, Y.m., Lu, Y.: Long-tailed visual recognition via gaussian clouded logit adjustment. In: Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition. pp. 6929–6938 (2022)
2022
-
[17]
In: Proceedings of the IEEE International Conference on Computer Vision
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object de- tection. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2980–2988 (2017)
2017
-
[18]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., Yu, S.X.: Large-scale long-tailed recognition in an open world. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2537–2546 (2019)
2019
-
[19]
In: International Conference on Machine Learning
Martens, J., Grosse, R.: Optimizing neural networks with kronecker-factored ap- proximate curvature. In: International Conference on Machine Learning. pp. 2408–
-
[20]
In: International Conference on Learning Represen- tations (2021),https://openreview.net/forum?id=37nvvqkCo5
Menon, A.K., Jayasumana, S., Rawat, A.S., Jain, H., Veit, A., Kumar, S.: Long-tail learning via logit adjustment. In: International Conference on Learning Represen- tations (2021),https://openreview.net/forum?id=37nvvqkCo5
2021
-
[21]
Advances in Neural Information Processing Sys- tems32(2019)
Raghu, M., Zhang, C., Kleinberg, J., Bengio, S.: Transfusion: Understanding trans- fer learning for medical imaging. Advances in Neural Information Processing Sys- tems32(2019)
2019
-
[22]
In: ProceedingsoftheIEEE/CVFConferenceonComputerVisionandPatternRecog- nition
Rangwani, H., Mondal, P., Mishra, M., Asokan, A.R., Babu, R.V.: Deit-lt: Dis- tillation strikes back for vision transformer training on long-tailed datasets. In: ProceedingsoftheIEEE/CVFConferenceonComputerVisionandPatternRecog- nition. pp. 23396–23406 (2024)
2024
-
[23]
In: European Con- ference on Computer Vision
Touvron, H., Cord, M., Jégou, H.: Deit iii: Revenge of the vit. In: European Con- ference on Computer Vision. pp. 516–533. Springer (2022)
2022
-
[24]
Van der Vaart, A.W.: Asymptotic statistics, vol. 3. Cambridge University Press (2000)
2000
-
[25]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8769–8778 (2018) Optim...
2018
-
[26]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Wang, J., Lukasiewicz, T., Hu, X., Cai, J., Xu, Z.: Rsg: A simple but effective mod- ule for learning imbalanced datasets. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3784–3793 (2021)
2021
-
[27]
In: International Conference on Learning Representations (2021),https://openreview.net/forum?id=D9I3drBz4UC
Wang, X., Lian, L., Miao, Z., Liu, Z., Yu, S.: Long-tailed recognition by rout- ing diverse distribution-aware experts. In: International Conference on Learning Representations (2021),https://openreview.net/forum?id=D9I3drBz4UC
2021
-
[28]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Xu, Z., Liu, R., Yang, S., Chai, Z., Yuan, C.: Learning imbalanced data with vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15793–15803 (2023)
2023
-
[29]
In: 2025 Asia Pacific Workshop on Data Science and Information Theory (APWDSIT)
Zhang, Q., Chu, C., Fu, H., Peng, T., Huang, S.L.: Asymptotic analysis for optimal source weights in multi-source transfer learning. In: 2025 Asia Pacific Workshop on Data Science and Information Theory (APWDSIT). pp. 1–6. IEEE (2025)
2025
-
[30]
CoRR abs/2601.10779(2026)
Zhang, Q., Chu, C., Fu, H., Peng, T., Wu, Y., Huang, G., Li, Y., Huang, S.: Unified optimization of source weights and transfer quanti- ties in multi-source transfer learning: An asymptotic framework. CoRR abs/2601.10779(2026). https://doi.org/10.48550/ARXIV.2601.10779,https:/...
2026 doi
-
[31]
In: In- ternational Conference on Machine Learning
Zhang, Q., Chu, C., Peng, T., Li, Q., Luo, X., Jiang, Z., Huang, S.L.: Lora-da: Data-aware initialization for low-rank adaptation via asymptotic analysis. In: In- ternational Conference on Machine Learning. PMLR (2026)
2026
-
[32]
Advances in Neural Information Processing Systems38, 25528–25563 (2026)
Zhang, Q., Fu, H., Huang, G., Liang, Y., Chu, C., Peng, T., Wu, Y., Li, Q., Li, Y., Huang, S.L.: A high-dimensional statistical method for optimizing transfer quanti- ties in multi-source transfer learning. Advances in Neural Information Processing Systems38, 25528–25563 (2026)
2026
-
[33]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Zhang, S., Li, Z., Yan, S., He, X., Sun, J.: Distribution alignment: A unified frame- work for long-tail visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2361–2370 (2021)
2021
-
[34]
Advances in Neural Information Processing Systems35, 34077–34090 (2022)
Zhang, Y., Hooi, B., Hong, L., Feng, J.: Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. Advances in Neural Information Processing Systems35, 34077–34090 (2022)
2022
-
[35]
IEEE Transactions on Pattern Analysis and Machine Intelligence45(9), 10795–10816 (2023)
Zhang, Y., Kang, B., Hooi, B., Yan, S., Feng, J.: Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence45(9), 10795–10816 (2023)
2023
-
[36]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Zhong, Z., Cui, J., Liu, S., Jia, J.: Improving calibration for long-tailed recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16489–16498 (2021)
2021
-
[37]
IEEE Transactions on Pattern Analysis and Machine Intelligence40(6), 1452–1464 (2017)
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence40(6), 1452–1464 (2017)
2017
-
[38]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Zhou, B., Cui, Q., Wei, X.S., Chen, Z.M.: Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9719– 9728 (2020)
2020
Reviewed June 30, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.