REVIEW 3 major objections 7 minor 1 cited by
Alignment-Free RGB-T Salient Object Detection: A Large-scale Dataset and Progressive Correlation Network
T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper introduces UVT20K, the largest unaligned RGB-thermal saliency dataset with 20,000 real-world pairs, and PCNet, a network that estimates a homography to align the two views and then progressively models inter- and intra-modal…
desk verdict Good dataset, overstated method claim: UVT20K is a real contribution, but the paper's own Table 2 refutes the 'optimal on all eight datasets' sentence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Semantics-guided Homography Estimation (SHE) module: a pre-trained iterative homography estimator (IHN) is adapted to RGB-T inputs by inserting S-Adapters into each encoder layer; the adapters use a down-up bottleneck whose fusion unit multiplies features by the sigmoid of channel-pooled semantic features, so the network learns to align object regions rather than the whole frame. The Inter- and Intra-Modal Correlation (IIMC) module: a transformer attention operation $C(Q,V)=softmax(QK^T/\sqrt{d_k})V+Q$ is applied first between the RGB feature masked by the warped thermal map (and by semantics) and the unwarped thermal feature, and then a second attention pass propagates the inter-modal result across the whole RGB feature. Together they implement the progressive correlation idea that the ablations show is responsible for the gains.
What would settle it
Take the UVT20K test set, split it by the challenge attributes into samples with and without depth-varying salient objects (or into TI/TC versus non-TI/TC), and compare PCNet's F-measure on the two groups. A substantial drop on the depth-varying or partially visible groups would indicate the single-homography alignment, rather than the correlation modules, is the bottleneck. A simpler check is to run PCNet on a small set of pairs where the thermal view shows a large parallax or a different object arrangement.
Extended reading notes
Core claim
The central claim is that alignment-free RGB-T SOD is best handled by a progressive pipeline: explicitly estimate the geometric transform between the two cameras, warp the thermal image, compute cross-modal correlations only on the thus-aligned overlapping region, and then propagate those correlations across the full object within the RGB modality. The paper further claims that a sufficiently large and diverse unaligned training set is necessary for this to work, and that UVT20K provides it. In support, PCNet achieves the best reported results on all eight evaluated datasets, with an F-measure of 0.822 on UVT20K, 0.686 on UVT2000, and 0.879 on VT821, and it shows the smallest performance gap between VT5000 and un-VT5000 among all compared methods.
Load-bearing premise
The method assumes that a single global geometric mapping, estimated by a network pre-trained on aerial imagery and fine-tuned with a small adapter, can line up the parts of the RGB and thermal images that show the same object, even when the scene has depth changes or the object is only partly visible in the thermal picture.
Editorial extensions
If this is right
- Training on UVT20K transfers: PCNet's F-measure on UVT2000 rises from 0.646 when trained on weakly aligned un-VT5000 to 0.686 when trained on UVT20K, showing that large-scale unaligned data helps beyond a specific benchmark.
- The dataset's scribbles and boundary annotations enable weakly supervised and boundary-aware training and evaluation for unaligned RGB-T SOD, which no previous benchmark supports.
- The gap between aligned and weakly aligned versions of the same data shrinks: PCNet's average drop from VT5000 to un-VT5000 is 3.8%, versus 9.1% for SACNet, implying explicit alignment makes models substantially more tolerant of misalignment.
- PCNet also tops aligned benchmarks, indicating that learning to align does not hurt, and can even help, when the input is already aligned.
Reading between the lines
- Because the homography is a single global transform, the approach should saturate on scenes with strong depth discontinuities; a useful extension is to replace the global warp with piecewise or learned dense correspondences for the TI/TC challenge subsets.
- The four new challenge attributes (SL, CSO, TI, TC) could be used as auxiliary supervision or as a gating signal to decide when the thermal branch should be trusted—something the paper does not explore.
- The dataset's two capture devices allow a domain-shift experiment: training on Hikvision pairs and testing on FLIR pairs (and vice versa) would measure how much of the reported gain comes from dataset scale rather than device-specific appearance.
- A testable extension of the progressive correlation idea is to apply the same align-then-correlate order to other unaligned multimodal tasks, such as RGB-depth saliency or visible-infrared tracking, where the parallax assumption is similar.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces UVT20K, a dataset of 20,000 unaligned RGB-Thermal image pairs with 407 scenes, 1,256 object categories, and multiple annotation types (saliency masks, scribbles, boundaries, and 15 challenge attributes), split into 10,000 training and 10,000 test pairs. It also proposes PCNet, composed of a Semantics-guided Homography Estimation (SHE) module that warps the thermal image to the RGB frame and an Inter- and Intra-Modal Correlation (IIMC) module that first correlates aligned regions and then propagates correlation within the RGB modality. The method is evaluated on UVT20K, UVT2000, three weakly aligned datasets, and three aligned datasets, with ablation studies and a training-set comparison.
Significance. If the dataset is released with a clean scene-disjoint split, UVT20K is a substantial community resource: it is an order of magnitude larger than existing unaligned RGB-T SOD benchmarks, includes a training/test partition, and provides annotations beyond binary masks. The authors deserve credit for releasing code and dataset, for training all compared alignment-free methods on the same UVT20K training set, and for including a comparison of training sets. PCNet's components are individually ablated and show consistent gains. However, the paper's strongest method claim—state-of-the-art on all eight datasets—is not supported by its own Table 2, and the current random pair-level split may allow scene leakage; both issues must be addressed before the results can be relied on.
major comments (3)
- [Comparison with State-of-the-Art Methods / Table 2] The text states that 'our method achieves optimal performance on all eight datasets,' but Table 2 contains two counterexamples. On un-VT1000, OSRNet has Sm=0.980 and Fm=0.951 versus PCNet's Sm=0.922 and Fm=0.904, so PCNet is best only on Em. On VT5000, SACNet has Em=0.957 and Fm=0.901 versus PCNet's Em=0.956 and Fm=0.899. The paper acknowledges only the VT5000 weakness, not the un-VT1000 one, which is large in magnitude. This is a load-bearing claim rather than a wording issue; please correct the summary statement, report per-dataset wins and losses, and temper the conclusion accordingly.
- [Dataset Splits] The split is described as randomly selecting 10,000 pairs for training and taking the remaining 10,000 for testing. Since the dataset comprises 407 scenes and, from the collection process, likely multiple frames per scene, a pair-level random split can place frames from the same scene in both training and test sets, yielding near-duplicate test content and optimistic results. Please state whether the split is scene-disjoint; if it is not, provide a scene-disjoint split or show that no scene appears in both partitions. This is essential for the benchmark claim.
- [Semantics-guided Homography Estimation (SHE) / Effect of SHE] The paper does not report any direct measure of alignment quality for SHE, so the claim that it aligns common regions rests only on downstream SOD metrics. Because the dataset explicitly includes TI and TC challenges where salient objects are partially or wholly absent from the thermal frame, a global homography cannot align those cases; the paper should report per-challenge performance or alignment error on a subset to show where SHE helps and where it fails, and whether IIMC compensates for alignment failures.
minor comments (7)
- [Comparison with State-of-the-Art Methods] The sentence 'Compared to the sub-optimal method (i.e., CA VER), our PCNet achieves an improvement of 6.2%, 2.8%, and 7.8%...' does not specify which two datasets are meant, and the values do not match simple averages over UVT20K and UVT2000; please clarify the calculation.
- [Experiment Setup / Table 2] All metrics are reported for a single run; for differences as small as 0.001–0.002 (e.g., VT1000 Em), please provide variance across seeds or significance information.
- [Dataset Annotation] The phrase 'rank and select the top 20,000 annotated samples' needs a precise ranking criterion (e.g., annotation agreement score) to reassure readers that the final dataset is not cherry-picked.
- [Methodology / Eq. (7)] In Eq. (7), M(I'_t) is described as a mapping and resize operation, but the dimensions of f_i_rgb, M(I'_t), and f_s are not given; state the feature-level shapes and how the mask is applied.
- [Table 1] Table 1 lists UVT20K's year as '-'; use the actual release year and clarify what is meant by the 'Training Subset' column entries for the weakly aligned datasets.
- [Comparison with State-of-the-Art Methods] The 'smallest performance gap' claim between weakly aligned and aligned datasets is only illustrated for VT5000; report the corresponding gaps for VT821 and VT1000 as well.
- [Ablation Studies / ID7] In ablation ID7, 'corresponding visible images' should be clarified as replacing the thermal branch with a second RGB input, since the current wording is ambiguous.
Circularity Check
No significant circularity: the dataset claim and PCNet results are empirical, held-out evaluations with no construction-level reduction to inputs.
full rationale
The paper's central contributions are a new dataset (UVT20K) and a network (PCNet). The dataset claim ('UVT20K is the largest multi-modal SOD dataset') is an empirical comparison of scale, scene count, and category count against existing datasets, not a derivation. The method claim is supported by quantitative comparisons on held-out test sets (UVT20K test, UVT2000, un-VT821, un-VT1000, un-VT5000, VT821, VT1000, VT5000) and by ablations; there is no fitted quantity that is then renamed as a prediction. The SHE module borrows an external homography estimator (IHN, Cao et al. 2022) and adapts it with an S-Adapter; this is a standard transfer-learning setup, not an ansatz smuggled in via self-citation. The paper does cite the authors' prior work (e.g., Wang et al. 2024a for UVT2000 and SACNet), but these citations are used as baselines and comparative benchmarks, not as load-bearing proof of the current method's validity. The ablation against the un-VT5000 training set is a fair control and does not reduce to the UVT20K contribution by construction. The skeptical observation that the text's 'optimal performance on all eight datasets' claim is contradicted by Table 2 (e.g., OSRNet outperforms PCNet on un-VT1000 Sm and Fm) is a correctness and reporting inconsistency, not a circularity: the numbers are external evidence, and the claim could be false without being circular. Overall, the derivation chain is self-contained empirical evaluation, with only minor, non-load-bearing self-citations, so the circularity score is low.
Assumptions & free parameters
assumptions (5)
- domain assumption The pre-trained IHN homography estimator can be adapted to RGB-T image pairs using a lightweight S-Adapter, without losing its alignment capability.
- domain assumption A planar homography warp is sufficient to align the common regions between the RGB and thermal images in the dataset.
- domain assumption The dataset construction process (quality filtering and majority-vote selection) yields accurate saliency annotations.
- standard math Evaluation metrics (E-measure, S-measure, F-measure) are the accepted standards for SOD.
- domain assumption Swin Transformer features are effective for both RGB and thermal modalities.
Cite this review
Pith. "Pith review of Alignment-Free RGB-T Salient Object Detection: A Large-scale Dataset and Progressive Correlation Network." pith.science (2026). https://pith.science/paper/JDUDAEIY
@misc{pith2026241214576,
author = {Pith},
title = {Pith review of: Alignment-Free RGB-T Salient Object Detection: A Large-scale Dataset and Progressive Correlation Network},
year = {2026},
howpublished = {\url{https://pith.science/paper/JDUDAEIY}},
note = {Machine review of arXiv:2412.14576}
}
read the original abstract
Alignment-free RGB-Thermal (RGB-T) salient object detection (SOD) aims to achieve robust performance in complex scenes by directly leveraging the complementary information from unaligned visible-thermal image pairs, without requiring manual alignment. However, the labor-intensive process of collecting and annotating image pairs limits the scale of existing benchmarks, hindering the advancement of alignment-free RGB-T SOD. In this paper, we construct a large-scale and high-diversity unaligned RGB-T SOD dataset named UVT20K, comprising 20,000 image pairs, 407 scenes, and 1256 object categories. All samples are collected from real-world scenarios with various challenges, such as low illumination, image clutter, complex salient objects, and so on. To support the exploration for further research, each sample in UVT20K is annotated with a comprehensive set of ground truths, including saliency masks, scribbles, boundaries, and challenge attributes. In addition, we propose a Progressive Correlation Network (PCNet), which models inter- and intra-modal correlations on the basis of explicit alignment to achieve accurate predictions in unaligned image pairs. Extensive experiments conducted on unaligned and aligned datasets demonstrate the effectiveness of our method.Code and dataset are available at https://github.com/Angknpng/PCNet.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmark
A vision-language model trained on a new 260K-pair real infrared-text dataset beats general VLMs on a 9-task infrared Q&A benchmark, but the benchmark is in-distribution.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Cao, S.-Y.; Hu, J.; Sheng, Z.; and Shen, H.-L. 2022. Iterative deep homography estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1879--1888
work page 2022
-
[4]
Cong, R.; Zhang, K.; Zhang, C.; Zheng, F.; Zhao, Y.; Huang, Q.; and Kwong, S. 2022. Does thermal really always matter for RGB-T salient object detection? IEEE Transactions on Multimedia, 25: 6971--6982
work page 2022
-
[5]
Fan, D.; Wang, W.; Cheng, M.; and Shen, J. 2019. Shifting More Attention to Video Salient Object Detection. In IEEE Conference on Computer Vision and Pattern Recognition , 8554--8564
work page 2019
-
[6]
Fan, D.-P.; Lin, Z.; Zhang, Z.; Zhu, M.; and Cheng, M.-M. 2020. Rethinking RGB-D salient object detection: Models, data sets, and large-scale benchmarks. IEEE Transactions on neural networks and learning systems, 32(5): 2075--2089
work page 2020
-
[7]
Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, 2790--2799. PMLR
2019
-
[8]
Huo, F.; Zhu, X.; Zhang, L.; Liu, Q.; and Shu, Y. 2021. Efficient context-guided stacked refinement network for RGB-T salient object detection. IEEE Transactions on Circuits and Systems for Video Technology, 32(5): 3111--3124
work page 2021
Show all 35 references
-
[9]
Huo, F.; Zhu, X.; Zhang, Q.; Liu, Z.; and Yu, W. 2022. Real-time one-stream semantic-guided refinement network for RGB-thermal salient object detection. IEEE Transactions on Instrumentation and Measurement, 71: 1--12
2022
-
[10]
Li, S.; Xu, M.; Ren, Y.; and Wang, Z. 2017. Closed-form optimization on saliency-guided image compression for HEVC-MSP. IEEE Transactions on Multimedia, 20(1): 155--170
2017
-
[11]
Liu, N.; Zhang, N.; Shao, L.; and Han, J. 2021 a . Learning selective mutual attention and contrast for RGB-D saliency detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12): 9026--9042
2021
-
[12]
Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021 b . Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 10012--10022
2021
-
[13]
Liu, Z.; Tan, Y.; He, Q.; and Xiao, Y. 2021 c . SwinNet: Swin transformer drives edge-aware RGB-D and RGB-T salient object detection. IEEE Transactions on Circuits and Systems for Video Technology, 32(7): 4486--4497
2021
-
[14]
Lu, A.; Li, C.; Yan, Y.; Tang, J.; and Luo, B. 2021. RGBT tracking via multi-adapter network with hierarchical divergence loss. IEEE Transactions on Image Processing, 30: 5613--5625
2021
-
[15]
Ma, S.; Song, K.; Dong, H.; Tian, H.; and Yan, Y. 2023. Modal complementary fusion network for RGB-T salient object detection. Applied Intelligence, 53(8): 9038--9055
2023
-
[16]
Pang, Y.; Zhao, X.; Zhang, L.; and Lu, H. 2023. CAVER: Cross-Modal View-Mixed Transformer for Bi-Modal Salient Object Detection. IEEE Trans. Image Process. , 32: 892--904
2023
-
[17]
Piao, Y.; Ji, W.; Li, J.; Zhang, M.; and Lu, H. 2019. Depth-induced multi-scale recurrent attention network for saliency detection. In Proceedings of the IEEE/CVF international conference on computer vision, 7254--7263
2019
-
[18]
Qin, Z.; Chen, J.; Chen, C.; Chen, X.; and Li, X. 2023. Unifusion: Unified multi-view fusion transformer for spatial-temporal representation in bird's-eye-view. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 8690--8699
2023
-
[19]
Tang, B.; Liu, Z.; Tan, Y.; and He, Q. 2022. HRTransNet: HRFormer-driven two-modality salient object detection. IEEE Transactions on Circuits and Systems for Video Technology, 33(2): 728--742
2022
-
[20]
Tang, J.; Fan, D.; Wang, X.; Tu, Z.; and Li, C. 2019. RGBT salient object detection: Benchmark and a novel cooperative ranking approach. IEEE Transactions on Circuits and Systems for Video Technology, 30(12): 4421--4433
2019
-
[21]
Tu, Z.; Li, Z.; Li, C.; Lang, Y.; and Tang, J. 2021. Multi-interactive dual-decoder for RGB-thermal salient object detection. IEEE Transactions on Image Processing, 30: 5678--5691
2021
-
[22]
Tu, Z.; Li, Z.; Li, C.; and Tang, J. 2022. Weakly alignment-free RGBT salient object detection with deep correlation network. IEEE Transactions on Image Processing, 31: 3752--3764
2022
-
[23]
Tu, Z.; Ma, Y.; Li, Z.; Li, C.; Xu, J.; and Liu, Y. 2020. RGBT Salient Object Detection: A Large-scale Dataset and Benchmark. CoRR, abs/2007.03262
2020 arXiv
-
[24]
Tu, Z.; Xia, T.; Li, C.; Wang, X.; Ma, Y.; and Tang, J. 2019. RGB-T image saliency detection via collaborative graph learning. IEEE Transactions on Multimedia, 22(1): 160--173
2019
-
[25]
Vaswani, A. 2017. Attention is All You Need. arXiv preprint arXiv:1706.03762
2017 arXiv
-
[26]
Wang, G.; Li, C.; Ma, Y.; Zheng, A.; Tang, J.; and Luo, B. 2018. RGB-T saliency detection benchmark: Dataset, baselines, analysis and a novel approach. In Image and Graphics Technologies and Applications: 13th Conference on Image and Graphics Technologies and Applications, IGT...
2018
-
[27]
Wang, J.; Song, K.; Bao, Y.; Huang, L.; and Yan, Y. 2021. CGFNet: Cross-guided fusion network for RGB-T salient object detection. IEEE Transactions on Circuits and Systems for Video Technology, 32(5): 2949--2961
2021
-
[28]
Wang, K.; Lin, D.; Li, C.; Tu, Z.; and Luo, B. 2024 a . Alignment-Free RGBT Salient Object Detection: Semantics-guided Asymmetric Correlation Network and A Unified Benchmark. IEEE Transactions on Multimedia
2024
-
[29]
Wang, K.; Tu, Z.; Li, C.; Zhang, C.; and Luo, B. 2024 b . Learning Adaptive Fusion Bank for Multi-modal Salient Object Detection. IEEE Transactions on Circuits and Systems for Video Technology
2024
-
[30]
Wang, W.; Lai, Q.; Fu, H.; Shen, J.; Ling, H.; and Yang, R. 2022. Salient Object Detection in the Deep Learning Era: An In-Depth Survey. IEEE Trans. Pattern Anal. Mach. Intell. , 44(6): 3239--3259
2022
-
[31]
Yuan, M.; Shi, X.; Wang, N.; Wang, Y.; and Wei, X. 2024. Improving RGB-infrared object detection with cascade alignment-guided transformer. Information Fusion, 105: 102246
2024
-
[32]
Zhang, P.; Liu, W.; Wang, D.; Lei, Y.; Wang, H.; and Lu, H. 2020. Non-rigid object tracking via deep multi-scale spatial-temporal discriminative saliency maps. Pattern Recognit., 100: 107130
2020
-
[33]
Zhao, Y.; Huang, X.; and Zhang, Z. 2021. Deep lucas-kanade homography for multimodal image alignment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 15950--15959
2021
-
[34]
Zhou, W.; Zhu, Y.; Lei, J.; Yang, R.; and Yu, L. 2023. LSNet: Lightweight spatial boosting network for detecting salient objects in RGB-thermal images. IEEE Transactions on Image Processing, 32: 1329--1340
2023
-
[35]
Zhu, Y.; Li, C.; Tang, J.; Luo, B.; and Wang, L. 2021. RGBT tracking by trident fusion network. IEEE Transactions on Circuits and Systems for Video Technology, 32(2): 579--592
2021
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.