REVIEW 3 major objections 5 minor 34 references
Contrastive Learning through Auxiliary Branch for Video Object Detection
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Adding a training-only contrastive branch to a video object detector raises its accuracy on ImageNet VID to 84.0% mAP (ResNet-101) and 85.2% mAP (ResNeXt-101) with zero additional inference cost.
desk verdict A simple, credible training-only contrastive auxiliary branch for video object detection, but the headline gain rests on an unreproduced baseline and single-run numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is CAB (contrastive auxiliary branch), a small trainable head attached to an intermediate backbone feature map: a 3x3 convolution, ReLU, adaptive average pooling, and a two-layer projection to a 128-dimensional vector, followed by the InfoNCE loss with temperature $\tau=0.1$. This branch is the only addition to the TROI/SELSA detector, and it is removed at inference. The second piece is DLW (dynamic loss weighting), the linearly decaying schedule $w(t)=w\,\max(0,1-t/k)$ that lets the auxiliary task dominate early and vanish by step $k=25,000$. Together they shift the backbone toward features that keep frames of the same video close in embedding space, without changing the detector's inference-time architecture or computational cost.
What would settle it
Re-run the TROI baseline and CLAB under identical conditions—same code, seeds, hyperparameters, FGFA split, and evaluation script—for at least three seeds. If the average gap is below about 0.5 mAP, or if the variance is comparable to the reported 2.0-point difference, the claim of a consistent gain from the auxiliary branch would be falsified.
Extended reading notes
Core claim
CLAB's central claim is that a training-only contrastive auxiliary branch can make a video detector's backbone features more resistant to the degradations common in video—motion blur, occlusion, unusual poses—and that this transfers into higher detection accuracy at zero inference cost. The auxiliary branch, attached to an intermediate backbone feature map, projects it to a 128-dimensional embedding and applies the InfoNCE loss with frames from the same video as positives and frames from other videos as negatives. The auxiliary loss is combined with the standard detection losses through a dynamic weight $w(t)=w\,\max(0,1-t/k)$ that decays linearly to zero at step $k$, the midpoint of training. On ImageNet VID, CLAB reaches 84.0% mAP with ResNet-101 and 85.2% mAP with ResNeXt-101, exceeding its TROI baseline by 2.0 and 0.9 points respectively and matching or beating published CNN-based video object detectors that use no post-processing.
Load-bearing premise
The entire 2.0-point gain rests on the assumption that the 82.0% mAP TROI baseline was reproduced under exactly the same training recipe, data split, hyperparameters, and evaluation protocol as the CLAB runs; the paper reports one run per configuration with no variance or seed information.
Editorial extensions
If this is right
- On the ImageNet VID validation set, CLAB with ResNet-101 reports 84.0% mAP and with ResNeXt-101 85.2% mAP, both above published CNN-based detectors evaluated without post-processing.
- The method adds zero inference-time cost: CAB and DLW are removed at test time, so the deployed model is identical in compute to TROI while the backbone is trained to be more robust.
- The ablation attributes 1.4 mAP of the 2.0-point gain to the contrastive branch and 0.6 mAP to the dynamic loss weighting, indicating that both components contribute.
- Because the auxiliary branch is attached to an intermediate feature map and trained with same-video positives, the learned robustness is specific to the detection backbone and does not require optical flow or external memory at inference.
- Loss weight and temperature ablations show CLAB stays above the TROI baseline across tested hyperparameter values, suggesting the gain is not limited to a single setting.
Reading between the lines
- CLAB's principle—a cheap contrastive head whose weight is annealed to zero—could transfer to other video recognition tasks such as action recognition or multi-object tracking, where same-video positives are equally natural and inference cost is a constraint.
- The reliance on a single run and the unstated variance across seeds leaves open the possibility that part of the 2.0-point gain is noise; re-running with several seeds and reporting mean and standard deviation would settle how robust the margin is.
- If the gain is real, it suggests that backbone representations, rather than temporal aggregation modules, are a bottleneck for robustness to video degradation; that would motivate contrastive pretraining objectives for video detectors more broadly.
- The method should combine with post-processing refinements like Seq-NMS, since the gains come from a stronger backbone and do not depend on the scoring stage; combining may push the absolute mAP further.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CLAB, a training-only auxiliary-branch method for video object detection. A contrastive InfoNCE loss is applied to intermediate backbone features of video frames, treating frames from the same video as positives and frames from other videos as negatives, and the auxiliary loss weight decays linearly to zero by the midpoint of training (DLW). The method is built on SELSA+TROI and evaluated on ImageNet VID (FGFA split), reporting 84.0 mAP with ResNet-101 and 85.2 mAP with ResNeXt-101, which the authors describe as state-of-the-art for CNN-based models without extra inference cost. Ablations attribute +1.4 mAP to the contrastive auxiliary branch and +0.6 mAP to dynamic loss weighting over a nominal TROI baseline of 82.0 mAP.
Significance. If the reported gains are reproducible, the paper makes a useful and practical contribution: a conceptually simple, inference-cost-free training technique that improves video object detection, with ablations of temperature and loss weight that provide concrete guidance for practitioners. The method is not circular, since performance is measured on an external benchmark, and the reported hyperparameter sensitivity is informative. However, the central claim of a consistent ~2.0 mAP gain rests on a single unverified comparison to a literature baseline and on single-run numbers, so the significance cannot be fully assessed at present.
major comments (3)
- [§4.1, Table 2] The baseline 'Temporal ROI Baseline [11]' is reported as 82.0 mAP, but the text does not state whether this number was obtained by the authors' own re-training of TROI under the exact protocol used for CLAB (FGFA split, 7 epochs, batch size 16 on two A100 GPUs, SGD lr 0.01, one target frame plus two support frames, 30 support frames at inference, NMS 0.5, shorter side 600). If 82.0 is taken from the TROI publication, the reported +1.4 and +2.0 gains are not controlled comparisons and could reflect differences in training recipe or evaluation rather than the effect of CAB/DLW. Please state explicitly how the baseline was produced; if it was not retrained in-house, retrain it under identical conditions and report the resulting number.
- [§4.3, Tables 2-4] All experimental results are single runs; no seeds, variance estimates, or released code/configurations are provided. The claimed total gain of 2.0 mAP is a difference between two single numbers, and the individual contributions (1.4 and 0.6 mAP) are small relative to typical run-to-run variation in this setting. Please report results from at least three seeds per configuration (mean ± std) and, ideally, release training configuration files and seeds so that the reproducibility of the main claim can be assessed.
- [§3.1, Eq. (1); §4.1] There are related inconsistencies in the formulation of the contrastive loss. Eq. (1) defines the InfoNCE loss with a single positive sample zi' per anchor, whereas §4.1 states that each image has 2 positive pairs because T=3 frames are sampled per video; if the implemented loss uses both support frames as positives, Eq. (1) must sum over the set of positives in the numerator. In addition, the denominator's indicator 1[k≠i] compares a frame index k with a video index i and should instead exclude all frames from the same video (e.g., 1[v(k)≠i] with v(k) the video of frame k). The negative-pair count '48 negative pairs' is also inconsistent with N=16 videos and T=3 frames, which gives (16−1)×3 = 45 negatives per anchor. Please correct the equation, the indicator notation, and the pair counts so that the method is unambiguous and reproducible.
minor comments (5)
- [Abstract, Section 1] The word 'constrastive' is a typo and should be 'contrastive' in both the abstract and the introduction.
- [Figure 2] The label 'B frames' near the input tensor is unexplained and conflicts with the notation 'T frames per video'; please use a single consistent notation for the number of frames per video.
- [Table 1] No sources are given for the baseline mAP values of the compared methods; please add a citation for each value or state that the numbers are taken from the corresponding papers.
- [§4.2] The sentence 'SELSA and TROI are simplified versions of our approach' is confusing, since CLAB is built upon TROI rather than being a simplification of SELSA/TROI; please rephrase to avoid this inversion.
- [§3.2, §4.1] Eq. (2) defines k=25,000 as the midpoint of training and the text calls it the midpoint; please state the total number of training steps or the per-epoch step count so that the schedule can be verified, since '7 epochs' alone does not determine k.
Circularity Check
No circularity; the method is evaluated against an external benchmark and does not reduce to its own inputs.
full rationale
The paper contains no circular derivation. CLAB adds an InfoNCE auxiliary loss computed on backbone features plus a linearly decaying loss weight to a Faster R-CNN / SELSA / TROI detector, and the reported 84.0% / 85.2% mAP values are measured on the external ImageNet VID validation set with standard mAP. No fitted parameter is renamed as a prediction: the temperature tau=0.1, loss weight w=0.005, and decay step k=25,000 are selected through ablations and reported as chosen hyperparameters, not as independently predicted quantities. This is a hyperparameter-selection and reproducibility concern (no seeds, no variance, single runs, and no code), but it is not circularity. The TROI baseline of 82.0 is cited from prior work, and the paper does not state whether the baseline was retrained under the identical recipe; that is a verification gap, not a circular step. All citations are to external prior work and the reference list contains no self-citations, so no load-bearing argument reduces to a self-citation chain. One internal inconsistency exists in Section 4.1, where the paper states each image contains 48 negative pairs while N=16 videos and T=3 frames give 45 cross-video negatives per anchor; this is an arithmetic/correctness issue, not circularity. The central SOTA claim therefore rests on an external benchmark comparison and is not equivalent to its inputs by construction.
Assumptions & free parameters
free parameters (4)
- initial auxiliary loss weight w =
0.005
- InfoNCE temperature tau =
0.1
- decay step k =
25,000
- number of support frames at training =
2
assumptions (3)
- domain assumption InfoNCE loss with positive pairs from the same video and negatives from different videos improves detection features
- domain assumption The TROI baseline with 82.0 mAP is reproduced faithfully
- domain assumption ImageNet VID validation set is the correct benchmark and the FGFA split is used
invented entities (1)
-
CLAB (Contrastive Learning through Auxiliary Branch) and DLW (Dynamic Loss Weighting)
independent evidence
Cite this review
Pith. "Pith review of Contrastive Learning through Auxiliary Branch for Video Object Detection." pith.science (2026). https://pith.science/paper/TXQGVJNW
@misc{pith2026250820551,
author = {Pith},
title = {Pith review of: Contrastive Learning through Auxiliary Branch for Video Object Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/TXQGVJNW}},
note = {Machine review of arXiv:2508.20551}
}
read the original abstract
Video object detection is a challenging task because videos often suffer from image deterioration such as motion blur, occlusion, and deformable shapes, making it significantly more difficult than detecting objects in still images. Prior approaches have improved video object detection performance by employing feature aggregation and complex post-processing techniques, though at the cost of increased computational demands. To improve robustness to image degradation without additional computational load during inference, we introduce a straightforward yet effective Contrastive Learning through Auxiliary Branch (CLAB) method. First, we implement a constrastive auxiliary branch using a contrastive loss to enhance the feature representation capability of the video object detector's backbone. Next, we propose a dynamic loss weighting strategy that emphasizes auxiliary feature learning early in training while gradually prioritizing the detection task as training converges. We validate our approach through comprehensive experiments and ablation studies, demonstrating consistent performance gains. Without bells and whistles, CLAB reaches a performance of 84.0% mAP and 85.2% mAP with ResNet-101 and ResNeXt-101, respectively, on the ImageNet VID dataset, thus achieving state-of-the-art performance for CNN-based models without requiring additional post-processing methods.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[11]
In: Proceedings of the AAAI Conference on Artificial Intelligence
Gong,T.,Chen,K.,Wang,X.,Chu,Q.,Zhu,F.,Lin,D.,Yu,N.,Feng,H.:Temporal roi align for video object recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 1442–1450 (2021)
work page 2021
-
[1]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition
Araslanov, N., Roth, S.: Single-stage semantic segmentation from image labels. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 4253–4262 (2020)
work page 2020
-
[2]
In: Proceedings of the European Conference on Computer Vision (ECCV)
Bertasius, G., Torresani, L., Shi, J.: Object detection in video with spatiotempo- ral sampling networks. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 331–346 (2018)
work page 2018
-
[3]
arXiv preprint arXiv:1706.05587 (2017)
Chen, L.C.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)
arXiv 2017
-
[4]
In: International conference on machine learning
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for con- trastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
2020
-
[5]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Chen,Y.,Cao,Y.,Hu,H.,Wang,L.:Memoryenhancedglobal-localaggregationfor video object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10337–10346 (2020)
work page 2020
-
[6]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Cui, Y., Jiang, C., Wang, L., Wu, G.: Mixformer: End-to-end tracking with itera- tive mixed attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13608–13618 (2022)
work page 2022
-
[7]
Computer Vision and Image Understanding219, 103406 (2022)
Dave, I., Gupta, R., Rizve, M.N., Shah, M.: Tclr: Temporal contrastive learning for video representation. Computer Vision and Image Understanding219, 103406 (2022)
work page 2022
Show all 34 references
-
[8]
In: 2009 IEEE conference on computer vision and pattern recognition
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
2009
-
[9]
In: Proceedings of the IEEE/CVF international conference on computer vision
Deng, J., Pan, Y., Yao, T., Zhou, W., Li, H., Mei, T.: Relation distillation net- works for video object detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 7023–7032 (2019)
2019
-
[10]
arXiv preprint arXiv:1812.02224 (2018)
Du, Y., Czarnecki, W.M., Jayakumar, S.M., Farajtabar, M., Pascanu, R., Lakshmi- narayanan, B.: Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.02224 (2018)
2018 arXiv
-
[12]
In: Computer Vision–ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16
Han, M., Wang, Y., Chang, X., Qiao, Y.: Mining inter-video proposal relations for video object detection. In: Computer Vision–ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16. pp. 431–446. Springer (2020)
2020
-
[13]
arXiv preprint arXiv:1602.08465 (2016)
Han, W., Khorrami, P., Paine, T.L., Ramachandran, P., Babaeizadeh, M., Shi, H., Li, J., Yan, S., Huang, T.S.: Seq-nms for video object detection. arXiv preprint arXiv:1602.08465 (2016)
2016 arXiv
-
[14]
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
Hashmi, K.A., Pagani, A., Stricker, D., Afzal, M.Z.: Boxmask: Revisiting bound- ing box supervision for video object detection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2030–2040 (2023)
2023
-
[15]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729–9738 (2020)
2020
-
[16]
He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016) 12 L. Rakotoarivony
2016
-
[17]
Advances in neural information processing systems32 (2019)
Lin, X., Baweja, H., Kantor, G., Held, D.: Adaptive auxiliary task weighting for re- inforcement learning. Advances in neural information processing systems32 (2019)
2019
-
[18]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Liu, X., Nejadasl, F.K., van Gemert, J.C., Booij, O., Pintea, S.L.: Objects do not disappear: Video object detection by single-frame object location anticipation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 6950–6961 (October 2023)
2023
-
[19]
arXiv preprint arXiv:1807.03748 (2018)
Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748 (2018)
2018 arXiv
-
[20]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Qian, R., Meng, T., Gong, B., Yang, M.H., Wang, H., Belongie, S., Cui, Y.: Spatiotemporal contrastive video representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6964–6974 (2021)
2021
-
[21]
arXiv preprint arXiv:1506.01497 (2015)
Ren, S.: Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497 (2015)
2015 arXiv
-
[22]
IEEE Access (2023)
Roh, S.D., Chung, K.S.: Diffusionvid: Denoising object boxes with spatio-temporal conditioning for video object detection. IEEE Access (2023)
2023
-
[23]
International journal of computer vision115, 211–252 (2015)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recog- nition challenge. International journal of computer vision115, 211–252 (2015)
2015
-
[24]
In: Proceedings of the IEEE international conference on com- puter vision
Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks. In: Proceedings of the IEEE international conference on com- puter vision. pp. 4068–4076 (2015)
2015
-
[25]
In: Euro- pean Conference on Computer Vision
Wang, H., Tang, J., Liu, X., Guan, S., Xie, R., Song, L.: Ptseformer: Progressive temporal-spatial enhanced transformer towards video object detection. In: Euro- pean Conference on Computer Vision. pp. 732–747. Springer (2022)
2022
-
[26]
In: Proceedings of the European conference on computer vision (ECCV)
Wang, S., Zhou, Y., Yan, J., Deng, Z.: Fully motion-aware network for video object detection. In: Proceedings of the European conference on computer vision (ECCV). pp. 542–557 (2018)
2018
-
[27]
In: Proceedings of the IEEE/CVF international conference on computer vision
Wu, H., Chen, Y., Wang, N., Zhang, Z.: Sequence level semantics aggregation for video object detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9217–9225 (2019)
2019
-
[28]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Xie,S.,Girshick,R.,Dollár,P.,Tu,Z.,He,K.:Aggregatedresidualtransformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1492–1500 (2017)
2017
-
[29]
In: European conference on computer vision
Xu, G., Liu, Z., Li, X., Loy, C.C.: Knowledge distillation meets self-supervision. In: European conference on computer vision. pp. 588–604. Springer (2020)
2020
-
[30]
In: Proceedings of the IEEE/CVF international conference on computer vision
Zhai, X., Oliver, A., Kolesnikov, A., Beyer, L.: S4l: Self-supervised semi-supervised learning. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1476–1485 (2019)
2019
-
[31]
Zhang, Y., Sun, P., Jiang, Y., Yu, D., Yuan, Z., Luo, P., Liu, W., Wang, X.: Bytetrack: Multi-object tracking by associating every detection box (2021)
2021
-
[32]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2881–2890 (2017)
2017
-
[33]
In: Proceedings of the IEEE international conference on computer vision
Zhu, X., Wang, Y., Dai, J., Yuan, L., Wei, Y.: Flow-guided feature aggregation for video object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 408–417 (2017)
2017
-
[34]
In: Proceedings of the ieee/cvf conference on computer vision and pattern recognition
Zhuang, C., She, T., Andonian, A., Mark, M.S., Yamins, D.: Unsupervised learning from video with deep neural embeddings. In: Proceedings of the ieee/cvf conference on computer vision and pattern recognition. pp. 9563–9572 (2020)
2020
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.