Pith. sign in

REVIEW 2 major objections 2 minor 38 references

OSDTW: Optimal Shared Depth and Task Weighting for Long-Tailed Recognition

T0 review · 2 major / 2 minor · reviewed 2026-06-30 · grok-4.3

Pith's one-line read OSDTW derives a bias-variance decomposition of generalization error to select shared encoder depth and task weights for long-tailed recognition.

desk verdict OSDTW gives a Fisher-derived proxy for choosing shared depth and task weights in long-tailed nets, but the block-diagonal approximation is the part that needs checking. read the letter →

arxiv 2605.24969 v1 pith:WZIMGCDW submitted 2026-05-24 cs.LG cs.AI

classification cs.LGcs.AI
keywords long-tailedrecognitiontaskdecompositionsharedencoderweightingFisherapproximationbias-variancegeneralizationerror
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper partitions long-tailed recognition into head and tail tasks implemented with a shared encoder and task-specific decoders. It introduces a factorized model so that the KL-based generalization error decomposes into task-wise terms up to a constant. Under a block-diagonal Fisher approximation, a second-order expansion of the expected generalization error is derived that splits into encoder variance, encoder bias, and decoder variance. This decomposition supplies a computable proxy for choosing the shared depth and task weights inside a three-stage training pipeline.

What carries the argument

The factorized model for head and tail label groups, which lets the generalization error be written as the sum of task-wise terms, together with the block-diagonal Fisher approximation that produces the second-order bias-variance expansion.

What would settle it

On a standard long-tailed benchmark, compare test accuracy obtained with depth and weights chosen by minimizing the decomposed error terms against accuracy obtained with heuristic or grid-search choices; the proxy is falsified if the selected values do not improve or match the best heuristic performance.

Watch

Extended reading notes

Core claim

Under a block-diagonal Fisher approximation, the expected generalization error of the factorized model admits a computable second-order expansion that decomposes into encoder variance, encoder bias, and decoder variance; this bias-variance decomposition serves as a proxy for selecting the shared depth and task weights in the three-stage training pipeline.

Load-bearing premise

The block-diagonal Fisher approximation is accurate enough that the second-order error expansion reliably ranks choices of shared depth and task weights.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper claims that long-tailed recognition can be improved by decomposing the problem into head and tail tasks via a shared encoder and task-specific decoders. Under a factorized model, the KL-divergence generalization error decomposes additively into task-wise terms. A three-stage pipeline (independent task training, weighted joint training, branch assembly) is proposed, and under a block-diagonal Fisher approximation a second-order expansion of expected generalization error is derived that decomposes into encoder variance, encoder bias, and decoder variance; this expansion serves as a computable proxy for selecting shared depth and task weights. Experiments on standard long-tailed benchmarks are reported to outperform strong baselines.

Significance. If the block-diagonal Fisher-based proxy reliably ranks shared-depth and task-weight choices, the work would replace heuristic hyper-parameter selection with a principled, bias-variance grounded procedure. The explicit decomposition into encoder/decoder terms could also clarify the sources of the head-tail trade-off. The three-stage pipeline and factorized KL argument are technically interesting even if the proxy requires further validation.

major comments (2)
  1. [Derivation of second-order expansion (under block-diagonal Fisher)] The central claim that the second-order expansion supplies an independent, computable proxy for hyper-parameter selection rests on the block-diagonal Fisher approximation. Because the encoder is shared, head and tail gradients are coupled through the same parameters; any residual off-block entries in the true Fisher (arising from feature overlap or class dependence) would invalidate the additive variance/bias decomposition used for ranking. No quantitative bound on the approximation error or regime of validity is supplied in the derivation.
  2. [Factorized model and task-wise objective] The factorized model is invoked both to obtain additive KL terms and to justify block-diagonality. The mutual-exclusivity assumption between head and tail label groups is plausible for the output layer but does not automatically extend to the shared encoder features; residual statistical dependence would couple the encoder-variance term across tasks and undermine the proxy.
minor comments (2)
  1. [Abstract and experimental section] The abstract states that the proxy 'enables efficient hyper-parameter search' but provides no concrete complexity comparison or timing results; a table or paragraph quantifying the cost of the three-stage pipeline versus grid search would strengthen the efficiency claim.
  2. [Notation and preliminaries] Notation for the task-wise objectives and the Fisher blocks should be introduced with explicit equation numbers in the main text rather than only in appendices, to make the bias-variance decomposition self-contained.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive review. We address each major comment below and describe the revisions we will make.

read point-by-point responses
  1. Referee: [Derivation of second-order expansion (under block-diagonal Fisher)] The central claim that the second-order expansion supplies an independent, computable proxy for hyper-parameter selection rests on the block-diagonal Fisher approximation. Because the encoder is shared, head and tail gradients are coupled through the same parameters; any residual off-block entries in the true Fisher (arising from feature overlap or class dependence) would invalidate the additive variance/bias decomposition used for ranking. No quantitative bound on the approximation error or regime of validity is supplied in the derivation.

    Authors: We acknowledge that the block-diagonal Fisher approximation is central to the additive decomposition and that no theoretical bound on the approximation error is derived. The shared encoder does induce parameter coupling, and off-block terms could in principle affect the ranking. While a general quantitative bound is technically difficult to obtain for deep networks, we will revise the manuscript to add an empirical analysis section that quantifies the magnitude of off-diagonal Fisher blocks on the benchmark datasets and discusses the observed regimes of validity. This will support the practical reliability of the proxy without claiming theoretical guarantees. revision: yes

  2. Referee: [Factorized model and task-wise objective] The factorized model is invoked both to obtain additive KL terms and to justify block-diagonality. The mutual-exclusivity assumption between head and tail label groups is plausible for the output layer but does not automatically extend to the shared encoder features; residual statistical dependence would couple the encoder-variance term across tasks and undermine the proxy.

    Authors: The factorized model p(y|x) = p_head(y_head|x) p_tail(y_tail|x) is introduced specifically to establish additive decomposition of the KL-based generalization error at the task level; it does not claim to remove all statistical dependence in the shared encoder features. The block-diagonal Fisher approximation is applied separately as a computational device for the bias-variance proxy. We will revise the text to clarify this distinction, explain how the three-stage pipeline (independent pre-training followed by weighted joint training) reduces feature-level coupling in practice, and note that the proxy remains an empirical tool whose utility is demonstrated by improved hyper-parameter selection over heuristics. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

Derivation chain is self-contained with no reductions to inputs by construction

full rationale

The provided abstract and description show a forward derivation: factorized model yields additive KL terms for task-wise objectives, followed by an independent block-diagonal Fisher approximation enabling a second-order expansion into encoder/decoder variance and bias terms. This expansion is then used as a proxy for hyper-parameter selection. No quoted equations or steps reduce the proxy back to fitted inputs, self-citations, or ansatzes by definition; the selection mechanism retains independent grounding from the stated approximations rather than tautologically reproducing the original objective.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The approach rests on a factorized probabilistic model and a block-diagonal Fisher approximation whose validity is not independently verified in the abstract; task weights and shared depth are treated as selectable via the derived proxy rather than free parameters fitted directly to final accuracy.

free parameters (2)
  • shared depth
    Hyper-parameter whose optimal value is chosen via the second-order proxy rather than exhaustive search.
  • task weights
    Weights used in joint training whose values are selected by the same bias-variance proxy.
assumptions (2)
  • domain assumption Factorized model captures mutual exclusivity and statistical dependence between head and tail label groups
    Invoked to write the KL generalization error as sum of task-wise terms.
  • ad hoc to paper Block-diagonal Fisher approximation is accurate enough for the second-order expansion
    Required to obtain the computable bias-variance decomposition used for hyper-parameter selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OSDTW: Optimal Shared Depth and Task Weighting for Long-Tailed Recognition." pith.science (2026). https://pith.science/paper/WZIMGCDW

@misc{pith2026260524969,
  author       = {Pith},
  title        = {Pith review of: OSDTW: Optimal Shared Depth and Task Weighting for Long-Tailed Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WZIMGCDW}},
  note         = {Machine review of arXiv:2605.24969}
}
read the original abstract

Long-tailed recognition suffers from a persistent head--tail trade-off: improving tail performance often degrades head accuracy and can increase training instability. Despite strong empirical results from re-weighting, decoupled training, and multi-expert methods, key design choices about representation sharing between head and tail classes and supervision weighting across class groups remain largely heuristic. In this work, we propose OSDTW, a principled task-decomposition framework that partitions the original single-label recognition problem into a head task and a tail task, implemented with a shared encoder and task-specific decoders. To handle the mutual exclusivity and statistical dependence between the two label groups, we introduce a factorized model and show that the resulting Kullback--Leibler divergence-based generalization error can be written as the sum of task-wise terms up to an additive constant, yielding a well-defined task-wise objective. We further develop a three-stage training pipeline: independent task training to estimate task-wise optima and the Fisher information matrix, weighted joint training to learn a shared encoder, and branch assembly to construct the final decoupled model. Under a block-diagonal Fisher approximation, we derive a computable second-order expansion of the expected generalization error, decomposing it into encoder variance, encoder bias, and decoder variance. This bias--variance decomposition provides a computable proxy to select the shared depth and task weights, enabling efficient hyper-parameter search. Experiments on standard long-tailed benchmarks demonstrate the effectiveness of the proposed approach over strong baselines.

Figures

Figures reproduced from arXiv: 2605.24969 by the authors.

Figure 1
Figure 1. Toy illustration of boundary uncertainty under finite samples. Dashed bands indicate the range of decision boundaries induced by resampling. To address these gaps, we take two steps. First, we adopt logit adjustment to handle train/test prior shift [20], and formulate the objective as minimizing K-L divergence-based generalization error [29–32]. Second, we characterize head/tail task statistics via the Fisher inform… view at source ↗
Figure 2
Figure 2. Three-stage training with a shared encoder of depth C and task-specific de￾coders. This stage provides task-level statistics such as Fisher blocks, which are then used to select the shared depth C and the task weights wA, wB. Stage 2 performs weighted training on the full network to obtain an optimal shared representation, using the objective (ϕ, ˆ ψˆ) = arg max ϕ,ψ h wA X x∈XN A log PXA;(ϕ,ψ)(x) + wB X x∈XN B log P… view at source ↗
Figure 3
Figure 3. ImageNet-LT accuracy of ViT-B under different task weights with full sharing (C = 12). On ImageNet-LT with full sharing (C = 12), [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: CIFAR-100-IR100 accuracy and generalization error of ViT-B across shared depth C and task weight wA, with the optimal combination highlighted by a white box. On CIFAR-100-IR100, [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages

  1. [1]

    Advances in Neural Information Pro- cessing Systems32(2019)

    Cao, K., Wei, C., Gaidon, A., Arechiga, N., Ma, T.: Learning imbalanced datasets with label-distribution-aware margin loss. Advances in Neural Information Pro- cessing Systems32(2019)

  2. [2]

    In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    Chen, S., Hou, Y., Cui, Y., Che, W., Liu, T., Yu, X.: Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 7870–7881 (2020)

  3. [3]

    Wiley (2001)

    Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley (2001)

  4. [4]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Cui, J., Zhong, Z., Liu, S., Yu, B., Jia, J.: Parametric contrastive learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 715–724 (2021)

  5. [5]

    IEEE Transactions on Pattern Analysis and Machine Intelligence 46(12), 7463–7474 (2023)

    Cui, J., Zhong, Z., Tian, Z., Liu, S., Yu, B., Jia, J.: Generalized parametric con- trastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 46(12), 7463–7474 (2023)

  6. [6]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Cui, Y., Jia, M., Lin, T.Y., Song, Y., Belongie, S.: Class-balanced loss based on effective number of samples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9268–9277 (2019)

  7. [7]

    In: In- ternational Conference on Learning Representations (2021),https://openreview

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: In- ternational Conference on Learning Representations (2021),https://openreview. net/forum?id=YicbFdNTTy

  8. [8]

    In: The Twelfth In- ternational Conference on Learning Representations (2024),https://openreview

    Guo, H., Greengard, P., Xing, E., Kim, Y.: LQ-loRA: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In: The Twelfth In- ternational Conference on Learning Representations (2024),https://openreview. net/forum?id=xw29VvOMmU

Show all 38 references
  1. [9]

    He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalablevisionlearners.In:ProceedingsoftheIEEE/CVFConferenceonComputer Vision and Pattern Recognition. pp. 16000–16009 (2022) 14 C. Chu et al

  2. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Hong, Y., Han, S., Choi, K., Seo, S., Kim, B., Chang, B.: Disentangling label distribution for long-tailed visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6626–6636 (2021)

  3. [11]

    Hsu,Y.C.,Hua,T.,Chang,S.,Lou,Q.,Shen,Y.,Jin,H.:Languagemodelcompres- sionwithweightedlow-rankfactorization.In:InternationalConferenceonLearning Representations (2022),https://openreview.net/forum?id=uPv9Y3gmAI5

  4. [12]

    In: International Conferenceon LearningRepresentations(2020),https://openreview.net/forum? id=r1gRTCVFvB

    Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., Kalantidis, Y.: De- coupling representation and classifier for long-tailed recognition. In: International Conferenceon LearningRepresentations(2020),https://openreview.net/forum? id=r1gRTCVFvB

  5. [13]

    In: International Confer- ence on Machine Learning

    Kim, S., Hooper, C.R.C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M.W., Keutzer, K.: Squeezellm: Dense-and-sparse quantization. In: International Confer- ence on Machine Learning. pp. 23901–23923. PMLR (2024)

  6. [14]

    Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny im- ages. Tech. rep., University of Toronto, Toronto, Ontario (2009),https://www.cs. toronto.edu/~kriz/learning-features-2009-TR.pdf

  7. [15]

    In: International Conference on Learning Representa- tions (2020),https://openreview.net/forum?id=HkgaETNtDB

    Lee, C., Cho, K., Kang, W.: Mixout: Effective regularization to finetune large-scale pretrained language models. In: International Conference on Learning Representa- tions (2020),https://openreview.net/forum?id=HkgaETNtDB

  8. [16]

    In: Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition

    Li, M., Cheung, Y.m., Lu, Y.: Long-tailed visual recognition via gaussian clouded logit adjustment. In: Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition. pp. 6929–6938 (2022)

  9. [17]

    In: Proceedings of the IEEE International Conference on Computer Vision

    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object de- tection. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2980–2988 (2017)

  10. [18]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., Yu, S.X.: Large-scale long-tailed recognition in an open world. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2537–2546 (2019)

  11. [19]

    In: International Conference on Machine Learning

    Martens, J., Grosse, R.: Optimizing neural networks with kronecker-factored ap- proximate curvature. In: International Conference on Machine Learning. pp. 2408–

  12. [20]

    In: International Conference on Learning Represen- tations (2021),https://openreview.net/forum?id=37nvvqkCo5

    Menon, A.K., Jayasumana, S., Rawat, A.S., Jain, H., Veit, A., Kumar, S.: Long-tail learning via logit adjustment. In: International Conference on Learning Represen- tations (2021),https://openreview.net/forum?id=37nvvqkCo5

  13. [21]

    Advances in Neural Information Processing Sys- tems32(2019)

    Raghu, M., Zhang, C., Kleinberg, J., Bengio, S.: Transfusion: Understanding trans- fer learning for medical imaging. Advances in Neural Information Processing Sys- tems32(2019)

  14. [22]

    In: ProceedingsoftheIEEE/CVFConferenceonComputerVisionandPatternRecog- nition

    Rangwani, H., Mondal, P., Mishra, M., Asokan, A.R., Babu, R.V.: Deit-lt: Dis- tillation strikes back for vision transformer training on long-tailed datasets. In: ProceedingsoftheIEEE/CVFConferenceonComputerVisionandPatternRecog- nition. pp. 23396–23406 (2024)

  15. [23]

    In: European Con- ference on Computer Vision

    Touvron, H., Cord, M., Jégou, H.: Deit iii: Revenge of the vit. In: European Con- ference on Computer Vision. pp. 516–533. Springer (2022)

  16. [24]

    Van der Vaart, A.W.: Asymptotic statistics, vol. 3. Cambridge University Press (2000)

  17. [25]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8769–8778 (2018) Optim...

  18. [26]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, J., Lukasiewicz, T., Hu, X., Cai, J., Xu, Z.: Rsg: A simple but effective mod- ule for learning imbalanced datasets. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3784–3793 (2021)

  19. [27]

    In: International Conference on Learning Representations (2021),https://openreview.net/forum?id=D9I3drBz4UC

    Wang, X., Lian, L., Miao, Z., Liu, Z., Yu, S.: Long-tailed recognition by rout- ing diverse distribution-aware experts. In: International Conference on Learning Representations (2021),https://openreview.net/forum?id=D9I3drBz4UC

  20. [28]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Xu, Z., Liu, R., Yang, S., Chai, Z., Yuan, C.: Learning imbalanced data with vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15793–15803 (2023)

  21. [29]

    In: 2025 Asia Pacific Workshop on Data Science and Information Theory (APWDSIT)

    Zhang, Q., Chu, C., Fu, H., Peng, T., Huang, S.L.: Asymptotic analysis for optimal source weights in multi-source transfer learning. In: 2025 Asia Pacific Workshop on Data Science and Information Theory (APWDSIT). pp. 1–6. IEEE (2025)

  22. [30]

    CoRR abs/2601.10779(2026)

    Zhang, Q., Chu, C., Fu, H., Peng, T., Wu, Y., Huang, G., Li, Y., Huang, S.: Unified optimization of source weights and transfer quanti- ties in multi-source transfer learning: An asymptotic framework. CoRR abs/2601.10779(2026). https://doi.org/10.48550/ARXIV.2601.10779,https:/...

  23. [31]

    In: In- ternational Conference on Machine Learning

    Zhang, Q., Chu, C., Peng, T., Li, Q., Luo, X., Jiang, Z., Huang, S.L.: Lora-da: Data-aware initialization for low-rank adaptation via asymptotic analysis. In: In- ternational Conference on Machine Learning. PMLR (2026)

  24. [32]

    Advances in Neural Information Processing Systems38, 25528–25563 (2026)

    Zhang, Q., Fu, H., Huang, G., Liang, Y., Chu, C., Peng, T., Wu, Y., Li, Q., Li, Y., Huang, S.L.: A high-dimensional statistical method for optimizing transfer quanti- ties in multi-source transfer learning. Advances in Neural Information Processing Systems38, 25528–25563 (2026)

  25. [33]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhang, S., Li, Z., Yan, S., He, X., Sun, J.: Distribution alignment: A unified frame- work for long-tail visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2361–2370 (2021)

  26. [34]

    Advances in Neural Information Processing Systems35, 34077–34090 (2022)

    Zhang, Y., Hooi, B., Hong, L., Feng, J.: Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. Advances in Neural Information Processing Systems35, 34077–34090 (2022)

  27. [35]

    IEEE Transactions on Pattern Analysis and Machine Intelligence45(9), 10795–10816 (2023)

    Zhang, Y., Kang, B., Hooi, B., Yan, S., Feng, J.: Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence45(9), 10795–10816 (2023)

  28. [36]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhong, Z., Cui, J., Liu, S., Jia, J.: Improving calibration for long-tailed recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16489–16498 (2021)

  29. [37]

    IEEE Transactions on Pattern Analysis and Machine Intelligence40(6), 1452–1464 (2017)

    Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence40(6), 1452–1464 (2017)

  30. [38]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhou, B., Cui, Q., Wei, X.S., Chen, Z.M.: Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9719– 9728 (2020)

Pith tools

Reviewed June 30, 2026 · model on record in the stance chip above.