{"id":"ccec408d-8f04-424f-ad93-c8b0eff7f772","arxiv_id":"2608.11564","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RGB-HS aligns a thermal encoder with a frozen RGB foundation model at structural and semantic levels, using brightness-contrast weighting, and improves thermal depth estimation on MS2.","lead":"This paper trains a thermal-image depth estimator by borrowing visual knowledge from a frozen RGB-image model, using multi-level feature alignment and a simple quality check on the RGB side. The method needs no RGB input at run time and reports small but consistent accuracy gains on a driving benchmark, especially at night and in rain.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Verification module's confidence scaling is term-dependent and contradicted by the alignment ablation; central mechanism unestablished.","rationale":"The reader's weakest assumption is that C_RGB is a valid proxy for RGB teacher reliability; reading Section III-C and Eq. 11 confirms this is exactly where the argument is least secure. The problem is even more concrete than the reader stated: the scaling is applied to the student features rather than teacher tokens, and its effect is inconsistent across the three alignment losses, with the KL term potentially amplifying rather than suppressing unreliable supervision. The ablation table aggravates the issue: alignment alone makes AbsRel worse, so the verification scalar is responsible for the full primary-metric gain. No code, error bars, or external validation are provided, and the paper's own table contradicts its textual claim that each alignment marginally improves AbsRel. These are internal consistency and mechanism-support problems, not disagreements with consensus. The aggregate Table I result is still plausible if the mechanism is confirmed, so the reader's CONDITIONAL verdict remains appropriate; I would not move it without additional evidence.","tokens_in":12192,"tokens_out":10968,"duration_ms":118276,"concrete_test":"Release the training code and run a control experiment that replaces C_RGB in Section III-C with (i) a constant 1.0 and (ii) an independent random per-image scalar matched to the mean and variance of C_RGB over the MS2 training set, keeping all losses and hyperparameters identical. If either control reproduces Table V's verification gain (AbsRel 0.114 to 0.105), then the brightness-contrast confidence is not the operative mechanism and the 'quality-aware filtering' claim fails. If neither control reproduces the gain, C_RGB carries real information, but the authors should still report per-loss gradient norms to resolve the cosine and KL inconsistencies.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the verification module, which is the only component that improves AbsRel in Table V (0.114 to 0.105), does not implement the claimed 'filtering' and its confidence signal is not validated. In Section III-C, C_RGB is a single per-image scalar, and the method applies F_THR <- C_RGB * F_THR. This scales the student features, not the RGB teacher tokens, so the description 'weighting tokens from the RGB branch' is inaccurate. The effect is term-dependent: the cosine latent loss (Eq. 5) is scale-invariant, so C_RGB has no effect there; the correlation loss (Eq. 2) has gradients that scale as C_RGB^2, down-weighting low-confidence images; but the KL loss (Eq. 6) uses softmax(C * f_THR / T) as the student logits, so for C in [0,1] the student distribution is flattened and KL(P_RGB || P_THR) increases, giving low-confidence images stronger, not weaker, alignment pressure. In addition, Table IV shows L_map or L_latent alone increases AbsRel relative to the DINOv3-ViT-B/16 baseline (0.111 to 0.116/0.116/0.114), directly contradicting the text's claim that each alignment 'marginally improves' AbsRel. Thus the final improvement cannot currently be attributed to the proposed quality-aware hierarchical supervision; it may be an artifact of per-image loss reweighting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RGB-HS, a framework for thermal-image depth estimation built on the MSCRF baseline. It replaces the thermal encoder with a DINOv3 ViT-B/16 foundation model, adds a frozen parallel RGB teacher encoder of the same architecture, and aligns the two branches with a map-level correlation loss and latent-level cosine and KL losses. A verification module computes a brightness-contrast confidence score C_RGB per RGB image and scales the thermal features before computing alignment losses. Experiments on the MS2 dataset are reported for monocular and stereo depth estimation, including ablations over encoder scale, alignment losses, and verification, plus complexity comparisons.","tokens_in":12543,"tokens_out":5798,"duration_ms":54590,"significance":"If the results hold, RGB-HS would be a useful demonstration that frozen RGB foundation features can be transferred to thermal depth estimation using a simple training-time teacher, and the efficiency analysis is a strength: RGB-HS uses fewer parameters and FLOPs than MSCRF while improving RMSE (Tables VI and I). The paper also provides a thorough comparison to prior thermal depth methods on both standard and cleaned MS2 splits. However, the central mechanism is not currently established: the alignment ablation contradicts the text, and the verification operation's implementation does not match the claimed filtering behavior, so the empirical improvement is not yet attributable to the proposed hierarchical supervision. The reproducible experimental setup is a further strength, but missing error bars make the small reported gains hard to assess.","major_comments":[{"comment":"The text states that adding either L_map or L_latent 'marginally improves' the baseline, but Table IV reports AbsRel 0.116 for each loss alone and 0.114 for both, versus 0.111 for the DINOv3 ViT-B/16 backbone without alignment. The alignment module therefore degrades AbsRel in every configuration shown, so the claim that the two alignment terms are complementary and improve performance is contradicted by the paper's own table. This is load-bearing because Table V attributes the final improvement to verification on top of these alignment terms; the authors must correct the text or provide corrected experiments, and report variance across multiple runs.","section":"Section IV-D, Table IV"},{"comment":"The verification module is described as 'weighting tokens from the RGB branch' and 'filtering unreliable RGB supervision', but the only implemented operation is F_THR <- C_RGB * F_THR, which scales the thermal student features. This is not equivalent to down-weighting the teacher: the cosine loss in Eq. (5) is invariant to this scaling, while in Eq. (6) scaling f_THR down flattens Softmax(f_THR/T) and increases KL(P_RGB || P_THR), so low-confidence images receive stronger, not weaker, distributional alignment pressure. Since Table V shows that verification is the sole component that improves AbsRel (0.114 to 0.105), the paper does not establish that the proposed quality-aware supervision causes the improvement; an alternative per-image reweighting explanation is not ruled out. The authors should validate C_RGB against an independent measure of RGB feature reliability, or redesign the mechanism so its effect matches the claimed filtering behavior.","section":"Section III-C, Eqs. (5), (6), (11), (12)"},{"comment":"The key numerical claims rest on small performance differences without error bars or significance tests (e.g., AbsRel 0.105 vs 0.111 in Table I, and the tied 0.072 AbsRel with RGB-MDE in Table II). Given that the claimed improvement is the basis of the paper's central claim, the authors should report mean and standard deviation over multiple training runs or otherwise demonstrate that the differences are not within run-to-run noise.","section":"Tables I and II"}],"minor_comments":[{"comment":"The text says 'reducing AbsRel and RMSE by 0.407 and 1.262' when comparing RGB-HS (Mono) with MSCRF (Stereo), but Table I shows an AbsRel reduction of 0.003 (0.120 to 0.117); the value 0.407 is the SqRel reduction. The sentence should refer to SqRel and RMSE, or the numbers should be corrected.","section":"Section IV-C"},{"comment":"C_RGB is called a 'probabilistic indicator' of visual quality, but it is a hand-designed heuristic score and not a calibrated probability; the wording should be softened accordingly.","section":"Section III-C"},{"comment":"There are several typographical issues: 'proposeRGB-HS' is missing a space, 'Bright-contrast' is awkward, and Table I contains the run-together entry '0.2070.8720.949' which should be separated.","section":"Abstract and Figure 2"},{"comment":"The statement that 'performance improves with larger ViT capacities' is only partially supported: ViT-L/16 has RMSE 3.939, worse than ViT-B/16's 3.826, so the relationship is non-monotonic and should be qualified or discussed.","section":"Section IV-D, Table III"},{"comment":"The notation f_THR,c and f_RGB,c is undefined: it should be stated whether these are vectors, maps, or scalars, and which norm is applied to the Gram-matrix difference; currently the expression is ambiguous.","section":"Section III-B, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a computer vision journal. The main risk is not novelty but the internal inconsistency between the ablation results and the verification mechanism; I would ask for a revised ablation study and a direct validation of the verification confidence before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. The paper reports a genuine but small improvement in thermal depth estimation on MS2: with the same DINOv3 ViT-B backbone, the full RGB-HS gets AbsRel 0.105 vs 0.111 for the backbone alone, and 0.105 vs 0.120 for the original MSCRF. On the cleaned MS2 split it ties RGB-MDE at 0.072 AbsRel with a slightly better RMSE. The second thing: the paper's claimed central mechanism, the brightness–contrast 'verification' module, is implemented in a way that contradicts its own description and likely does not do the down-weighting it is supposed to do.\n\nWhat is genuinely new is the specific combination: map-level Gram-matrix correlation loss plus latent-level cosine and KL losses, with a per-image brightness–contrast scalar used to modulate the alignment. That combination is not in GTDE, RGB-MDE, or ThermoStereoRT. The paper is readable, the baseline comparisons are transparent, and the authors cite the right prior work. The DINOv3 encoder replacement alone accounts for most of the gain (AbsRel 0.120 to 0.111), which is a useful data point.\n\nNow the soft spots, and the stress-test note is right about them. The verification module scales the student features (F_THR <- C_RGB * F_THR), not the RGB teacher tokens. The text says the module 'weights tokens from the RGB branch' or 'filters unreliable RGB supervision.' It does neither. The cosine loss is scale-invariant, so the latent cosine term is completely unaffected by C_RGB. The correlation loss does scale as C^2 and down-weights low-confidence images. But the KL term applies softmax to C*f_THR/T, and for C in [0,1] a smaller C flattens the student distribution, which increases KL divergence and gives low-quality RGB images stronger alignment pressure. So the module's net effect is uncertain and likely not what the authors intend. On top of that, Table IV contradicts the text: adding L_map or L_latent alone worsens AbsRel from 0.111 to 0.116, while the text says each one 'marginally improves' error. There are no error bars, results are on a single dataset, and the code is not released.\n\nWho should read this? If you work on thermal perception or cross-modal distillation, it is worth a look as an incremental data point and as a cautionary example of a confidence mechanism that can backfire. It deserves a serious referee—the empirical claim is testable and the flaws are fixable—but it needs major revision to realign the description with the implementation and to show, not just assert, that the verification works through the claimed channel. I would send it out, expecting a heavy revision.","headline":"Real but modest distillation gain for thermal depth; the verification module is misimplemented and likely does not do what the paper claims—referee-worthy but needs heavy revision.","tokens_in":13080,"tokens_out":4267,"would_cite":false,"duration_ms":38956,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hierarchical supervision from a frozen RGB foundation model improves depth estimation from thermal images, reducing the stereo baseline's RMSE from 5.068 to 3.548 on MS2 while matching the top RGB-distilled method on the cleaned split.","keywords":["thermal depth estimation","hierarchical supervision","knowledge distillation","foundation models","DINOv3","RGB-thermal alignment","MS2 dataset","brightness-contrast verification"],"falsifier":"Retrain RGB-HS with $C_{\\text{RGB}}$ replaced by a constant or by random per-image weights; if the error stays within noise of the verified model, the verification module is not doing the claimed work. A sharper test is to compute a per-image measure of alignment quality (for example, agreement between the teacher's RGB features and a strong thermal-only depth estimate) and check whether $C_{\\text{RGB}}$ predicts it better than chance; the paper reports only end-task validation numbers, not this correlation.","tokens_in":11970,"feed_emoji":"🌡️","tokens_out":8037,"duration_ms":78136,"temperature":0.7,"pith_summary":"The paper claims that a thermal-image depth network learns significantly better when a frozen RGB foundation model supervises it at two complementary levels: feature-map correlations for local structure and global latent statistics for semantic context. On the MS2 driving benchmark, this hierarchical supervision cuts the stereo baseline's RMSE from 5.068 to 3.548 and absolute relative error from 0.120 to 0.105; even the monocular version beats the previous stereo baseline. The paper also adds a brightness-contrast confidence signal that scales thermal features before alignment, intended to down-weight RGB supervision in low-quality images. If the claim holds, RGB foundation models trained on ordinary images become a reusable source of supervision for sensors that operate in the dark, and no RGB input is needed at inference.","feed_headline":"Frozen RGB encoder teaches thermal depth estimation","feed_subtitle":"Hierarchical alignment plus quality checks lift night and rain depth, with no RGB needed at test time.","key_machinery":"The load-bearing mechanism is a two-level alignment loss on a shared DINOv3 ViT-B/16 backbone: a map-level correlation loss compares per-channel self-correlation of thermal and RGB feature maps to transfer local structure, and a latent-level loss combining cosine similarity and KL divergence on max-pooled channel vectors transfers global semantic distributions. Around this sits a verification module that computes a brightness-contrast confidence $C_{\\text{RGB}}$ from the RGB luminance map (average of normalized mean luminance and contrast) and scales the thermal features by $C_{\\text{RGB}}$ before alignment, so low-quality RGB images contribute less. During inference the RGB branch, alignment, and verification are all discarded; only the thermal encoder-decoder runs.","core_discovery":"The central claim is that the rich hierarchies inside an RGB foundation model can be transferred to a thermal encoder through a teacher-student scheme in which the RGB branch is frozen and the thermal branch is trained to match it at both map level and latent level. Concretely, with DINOv3 ViT-B/16 as the shared encoder, RGB-HS aligns reshaped patch tokens via a channel-wise correlation loss and aligns global pooled embeddings via cosine similarity plus KL divergence, then scales thermal features by a brightness-contrast confidence $C_{\\text{RGB}}$ before computing these losses. On MS2 this yields the best thermal-only numbers in the main table (stereo AbsRel 0.105, RMSE 3.548), and on the cleaned MS2 split it reaches AbsRel 0.072 with RMSE 2.595, matching RGB-MDE's AbsRel while using fewer parameters and fewer FLOPs. The paper's diagnosis is that prior methods either ignored the RGB modality or ignored the intermediate tokens of the foundation model and treated all RGB supervision as equally reliable; RGB-HS addresses both omissions.","pith_inferences":["The brightness-contrast confidence is a hand-designed proxy; a learned confidence head or teacher-student feature-agreement score might replace it, and the ablation shows there is room to test whether a better signal widens the gap.","Because the correlation loss is shift-invariant and the latent loss ignores spatial layout, a position-aware or attention-based alignment might recover fine boundaries better in rain, where AbsRel (0.124) still trails daytime performance (0.098).","The method needs paired RGB-thermal training data but no correspondence at inference; adapting the alignment to unpaired or non-co-located RGB-thermal data would test whether the transfer is geometric or purely semantic.","All conclusions rest on one benchmark; evaluating RGB-HS on additional thermal datasets or on near-infrared input would show whether the gain generalizes across sensors."],"forward_implications":["RGB-HS in monocular mode (AbsRel 0.117, RMSE 3.806) beats the previous stereo baseline MSCRF (AbsRel 0.120, RMSE 5.068), so stereo input is not needed to benefit from RGB-supervised foundation priors.","Adding verification after alignment improves stereo AbsRel from 0.114 to 0.105 and RMSE from 3.646 to 3.548, so quality-weighted alignment contributes beyond the losses alone.","Using both alignment levels together beats either alone (AbsRel 0.114 vs 0.116 for each), indicating structural and semantic supervision are complementary.","On the cleaned MS2 split, RGB-HS matches RGB-MDE's AbsRel (0.072) and RMSElog (0.100) while using a smaller model (249M vs 666M) and fewer FLOPs (0.23T vs 0.72T).","Replacing the trained-from-scratch Swin-L encoder with a pre-trained DINOv3 ViT-B/16 gives most of the gain; the full RGB-HS reaches these numbers with an 86M encoder versus MSCRF's 197M."],"supporting_citations":[{"why":"Supplies the MS2 dataset, the MSCRF baseline architecture, and the baseline numbers every result is measured against.","marker":"[9]"},{"why":"DINOv3 ViT-B/16 is the frozen teacher and thermal student encoder; its hierarchical tokens are the objects being aligned.","marker":"[11]"},{"why":"RGB-MDE is the confidence-aware distillation method that defines the comparison on the cleaned MS2 split.","marker":"[16]"},{"why":"Defines the cleaned MS2 protocol used for Table II, setting the evaluation the paper ties on.","marker":"[42]"}],"fun_headline_variants":["Frozen RGB model guides thermal depth via hierarchy","Thermal depth learns from frozen RGB foundation model","Hierarchical RGB supervision boosts thermal depth estimation","Teacher-student with frozen RGB encoder for thermal depth","RGB foundation model teaches thermal depth without RGB at test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-designed brightness-contrast confidence $C_{\\text{RGB}}$ tracks how much the RGB teacher's features should be trusted for alignment, and that scaling thermal features by it genuinely down-weights unreliable supervision.","fun_headline_variants_meta":{"raw":{"variants":["Frozen RGB model guides thermal depth via hierarchy","Thermal depth learns from frozen RGB foundation model","Hierarchical RGB supervision boosts thermal depth estimation","Teacher-student with frozen RGB encoder for thermal depth","RGB foundation model teaches thermal depth without RGB at test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1318,"prompt_tokens":950,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":296}},"tokens_in":566,"tokens_out":368,"duration_ms":4270,"temperature":1.0,"reasoning_tokens":296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:35:02.154463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain RGB-HS with $C_{\\text{RGB}}$ replaced by a constant or by random per-image weights; if the error stays within noise of the verified model, the verification module is not doing the claimed work. A sharper test is to compute a per-image measure of alignment quality (for example, agreement between the teacher's RGB features and a strong thermal-only depth estimate) and check whether $C_{\\text{RGB}}$ predicts it better than chance; the paper reports only end-task validation numbers, not this correlation.","supporting_citations":[{"cited_title":"Deep Depth Estimation from Thermal Image: Dataset, Benchmark, and Challenges","cited_arxiv_id":"2503.22060","evidence_quote":"Defines the cleaned MS2 protocol used for Table II, setting the evaluation the paper ties on."},{"cited_title":"Monother-depth: Enhancing thermal depth estimation via confidence- aware distillation,","cited_arxiv_id":null,"evidence_quote":"RGB-MDE is the confidence-aware distillation method that defines the comparison on the cleaned MS2 split."}],"review_version":1}