REVIEW 5 major objections 6 minor 29 references
Right Answer, Wrong Heat: Explanation-Aware Evaluation and Thermal-Grounded Feedback for MLLMs on Infrared Images
T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Correct answers on infrared images often rest on visible-light cues, not heat evidence
desk verdict A well-scoped and honest empirical study of infrared MLLM explanation grounding, but every headline number flows through an LLM judge validated on only 48 human examples, so the magnitudes should be treated as provisional until the artifacts ship. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the structured output o = (a, e, V, u) and the strict output-level groundedness rule φ = (1−p_f) 1[τ≥2 ∨ γ=0] 1[ε≥2] (1−h), where τ is a 0–3 thermal-grounding score, ε is evidence grounding, h flags visible-cue misuse, and p_f is parse failure. This rule, combined with the Dual-LLM Consensus Judge, converts the abstract notion of 'thermal grounding' into a measurable label. The input manipulation Translated-RGB (replacing the infrared image with a visible-like rendering) isolates whether explanations rely on thermal evidence or on transferred visible-light priors. The TGF loop uses four role-specific feedback agents and a lexicographic selection score ψ(o) to repair explanations while keeping the answer fixed.
What would settle it
A larger human study (hundreds of items, multiple annotators) that shows the judge's thermal-grounding labels disagree substantially with human judgments on the same outputs, or that shows the judge rewards keywords like 'heat' and 'temperature' even when the explanation does not actually cite image evidence, would undermine the hidden-failure and erosion findings.
Extended reading notes
Core claim
The central claim is that general-purpose MLLMs often answer infrared questions correctly while justifying them with visible-light cues rather than thermal evidence. Using a structured output that separates answer, explanation, evidence list, and uncertainty, and scoring explanations with a Dual-LLM Consensus Judge, the paper reports a mean IR-only grounded-at-correct rate (F@C) of 0.631, meaning 36.9% of correct answers fail the output-level grounding criterion. Withholding the infrared image and showing only a translated visible rendering reduces mean thermal-grounding score from 1.441 to 0.752 (Δτ_sub = −0.689) while accuracy changes only from 0.716 to 0.700. This erosion is most pronounced for more capable models and disappears when infrared remains available alongside the visible view. The paper further proposes Thermal-Grounded Feedback (TGF), a multi-agent, training-free loop that improves explanation grounding on paired-input settings without altering answers.
Load-bearing premise
The Dual-LLM Consensus Judge's thermal-grounding scores are valid measurements of whether an explanation really uses thermal evidence.
Editorial extensions
If this is right
- If the central claim is correct, reporting answer accuracy alone on infrared benchmarks is misleading; evaluations should report EQ, F@C, and thermal-grounding score separately.
- The observed erosion under modality substitution implies that improving accuracy by translating infrared to visible-like renderings can actively reduce the trustworthiness of explanations, even when accuracy is preserved.
- The capability-associated pattern suggests that stronger models may be more prone to adopting visible-light priors when infrared is withheld, so capability scaling does not automatically improve thermal grounding.
- Training-free, output-only feedback like TGF offers a practical mitigation for closed-source models where weight access and fine-tuning are impossible.
- The framework generalizes to any domain where the evidence modality matters, such as medical imaging or remote sensing, where a correct answer must also be justified by the right sensory evidence.
Reading between the lines
- The Dual-LLM Consensus Judge may be rewarding thermal-sounding language rather than true heat-based reasoning; a larger human-validated set would clarify this, but the paper's 48-item anchor leaves this as an open risk.
- The negative correlation between capability and substitution erosion (r = −0.903) could be partly a ceiling effect: stronger models have higher baseline thermal scores, so they have more room to drop under substitution.
- A testable extension would apply the same explanation-aware evaluation to other sensor modalities (e.g., depth maps, multispectral images) where a correct answer could likewise be justified by non-modal cues.
- The TGF repair could be evaluated with a human study that directly compares revised explanations against original ones for thermal validity, rather than only LLM preference judgments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an explanation-aware evaluation framework for general-purpose MLLMs applied to infrared images, separating answer correctness, output-level explanation groundedness, and thermal grounding. Using IF-Bench, 11 models, and five input settings, it reports that correct IR-only answers frequently fail thermal grounding, that replacing the IR image with a translated visible-like rendering erodes thermal grounding with little accuracy change, and that this erosion correlates with model capability only when IR is withheld. It also introduces Thermal-Grounded Feedback (TGF), a training-free multi-agent loop, and reports local validation improving explanation-side metrics while keeping answers fixed.
Significance. If the measurement foundation holds, the framework fills a real gap: answer-only accuracy cannot distinguish a heat-grounded explanation from a visible-light shortcut, and the proposed metrics (EQ, F@C, HF, Δτ_sub) are concrete and well-defined. The authors are appropriately cautious about the capability association, explicitly labeling it descriptive and partially baseline-coupled, and they provide a transparent lexical baseline (LHJ). The central empirical claims, however, are entirely mediated by a Dual-LLM Consensus Judge that is validated on only a 48-item human anchor, with no released prompts or confidence intervals; this makes the headline numbers conditional on an unverified instrument. The TGF mitigation also contains a partial circularity that needs to be addressed.
major comments (5)
- [Dual-LLM Consensus Judge, Table 1] The entire measurement chain for the paper's central numbers (F@C = 0.631, HF = 36.9%, Δτ_sub = −0.689, r = −0.903) runs through the Dual-LLM Consensus Judge, yet the only validation is a 48-item blind human anchor described as 'preliminary', with no confidence intervals, no item-selection procedure, and no released judge prompts. Because the claims are comparative across settings and models, a setting-dependent judge bias could manufacture both the hidden-failure rate and the substitution erosion; for example, a judge that already expects thermal claims to be unsupported in Translated-RGB will assign lower τ there regardless of explanation content. Please provide a larger per-setting human validation, report confidence intervals for τ, F@C, and Δτ_sub, and release the judge prompts and the judged outputs.
- [Eq. (3) and Translated Visible Input] In Translated-RGB the model receives only a visible-like rendering, making color and texture cues the only available evidence. The visible-cue misuse flag h = 1 in Eq. (3) forces ϕ = 0 whenever such cues support a thermal conclusion, but whether this should be scored as misuse in a setting with no IR image is a judgment call that the 48-item anchor does not resolve. If the judge over-flags visible-language descriptions in the translated condition, the claimed substitution erosion Δτ_sub is inflated. Please validate the h labeling separately in Translated-RGB with per-setting human annotations, and report how Δτ_sub changes under alternative definitions of visible-cue misuse.
- [Thermal-Grounded Feedback, Table 5] The mitigation claim is evaluated with the same deterministic rubric that the feedback loop optimizes (the judge agent and Eq. (7)), and the independent preference check is also an LLM judge rather than human ratings. The statement that TGF is 'evaluated without circularity' is only partially true: the final evaluation judge differs from the loop judge, but the scoring criteria are the loop's own objectives. Please add a human-rated before/after sample and show that the F@C/EQ gains are not simply convergence of the loop to its own rubric.
- [Eq. (3), Aggregate Metrics] The thresholds τ≥2 and ε≥2 in Eq. (3) are hand-set, and no sensitivity analysis is reported. Since F@C, HF, and Δτ_sub all depend on these thresholds, the headline numbers may shift substantially with the chosen cutoffs. Please report how the main conclusions vary as the thresholds move (e.g., τ≥1 vs. τ≥3, ε≥1 vs. ε≥3) and state whether the qualitative findings survive.
- [Abstract, Experiments, Table 2] The paper occasionally conflates output-level groundedness with thermal grounding: F@C and HF are defined via ϕ in Eq. (3), which includes ε, h, and the γ gate, so Translated-RGB can show higher F@C than IR-only (0.661 vs. 0.631 in Table 2) while τ drops by 0.689. The abstract and conclusion should state clearly that the erosion finding is about thermal grounding τ, not the output-level groundedness captured by F@C and HF.
minor comments (6)
- [Table 2] The caption refers to shaded cells indicating improvement over the same model's IR-only value, but the shading is not visible in the text; please use bold or arrows to mark improvements.
- [Benchmark and input settings] The infrared-to-RGB translation model is cited but not described; please specify its architecture, training data, or output characteristics, since it is the key manipulation in Translated-RGB.
- [Table 1] Please report the item distribution of the 48-item human anchor across the five input settings and the ten dimensions; this is needed to assess whether the calibration covers the settings where the headline comparisons are made.
- [Eq. (3)] The thermal-gating indicator γ is introduced only as 'whether the question dimension is thermal-gated'; please define how γ is assigned to each question dimension.
- [Figure 6] The before-explanation quotes the input-format tag 'irplustranslatedrgb'; please clarify whether such filename or format cues were visible to models in all settings, since the model may be reading the input tag rather than the image content.
- [References] Several references are to 2026 proceedings or preprints; please verify that all cited works are publicly accessible or clearly marked as preprints.
Circularity Check
TGF mitigation result is measured with the same rubric the feedback loop optimizes, so the reported improvement is partly built in; the core diagnostic findings rest on a small human anchor and are not circular.
-
self definitional
[Methodology, 'Thermal-Grounded Feedback', Eq. (7); Experiments, 'Experimental Setup' and 'TGF Mitigates Explanation-Side Failures', Table 5.]
"Its before/after table applies the same deterministic rubric as the feedback loop to track repairable changes, while a separate paired LLM preference evaluation provides a semantic check of whether the revised explanations are actually preferred over no-feedback outputs."
Eq. (7) defines the selector score ψ(o) = (ϕ(o), 1[τ(o)≥2], 1[ϵ(o)≥2], 1[h(o)=0], uok(o), τ(o)+ϵ(o)); the loop keeps the candidate with the lexicographically largest ψ. Table 5's reported gains in F@C, EQ, τ, evidence grounding, and parse failures are computed from the same rubric components. With the answer fixed, raising ϕ among correct answers mechanically raises F@C, and the loop is selected to raise exactly those components, so the 0.875→0.939 F@C improvement is largely the selection objective restated as an outcome.
full rationale
The central diagnostic claims (F@C=0.631, hidden-failure rate 36.9%, Δτ_sub=-0.689, capability correlation r=-0.903) are measurements produced by the Dual-LLM Consensus Judge, which is anchored to 48 blind human labels (κ=0.80-0.89). That preliminary anchor gives the diagnostic an independent, if thin, external grounding; no equation in the paper makes those numbers equal to their inputs by construction, so I do not flag the judge itself as circular. The one concrete construction-level circularity is in the mitigation experiment: the TGF selector in Eq. (7) lexicographically maximizes exactly the components (ϕ, τ, ϵ, h, uncertainty, τ+ϵ) that Table 5 uses as its before/after metrics, and the Experimental Setup section states that the table 'applies the same deterministic rubric as the feedback loop.' Thus the reported improvement is partly a tautology of the selection rule, not independent evidence of better thermal reasoning. The paper's own caveats about the capability association being 'partially baseline-coupled' are appropriately disclosed and are not circularity. Overall, partial circularity confined to the TGF mitigation claim; score 6.
Assumptions & free parameters
free parameters (2)
- Thermal grounding threshold tau >= 2 in Eq. 3 =
2 on a 0-3 scale
- Evidence grounding threshold epsilon >= 2 in Eq. 3 =
2 on a 0-3 scale
assumptions (4)
- domain assumption The Dual-LLM Consensus Judge can reliably measure thermal grounding from text output alone.
- domain assumption Translated-RGB from IF-Bench is a valid IR-withholding probe that preserves task content.
- domain assumption When infrared is present alongside visible rendering, models have access to and can use thermal evidence.
- ad hoc to paper Strict output-level groundedness is adequately captured by the binary rule phi in Eq. 3 with the chosen thresholds.
Cite this review
Pith. "Pith review of Right Answer, Wrong Heat: Explanation-Aware Evaluation and Thermal-Grounded Feedback for MLLMs on Infrared Images." pith.science (2026). https://pith.science/paper/5LX7T2SO
@misc{pith2026260809145,
author = {Pith},
title = {Pith review of: Right Answer, Wrong Heat: Explanation-Aware Evaluation and Thermal-Grounded Feedback for MLLMs on Infrared Images},
year = {2026},
howpublished = {\url{https://pith.science/paper/5LX7T2SO}},
note = {Machine review of arXiv:2608.09145}
}
read the original abstract
General-purpose multimodal large language models (MLLMs) are increasingly applied to infrared images, where they are commonly scored by answer accuracy alone. However, a correct answer does not ensure that the model's explanation is grounded in infrared thermal evidence. We introduce an explanation-aware evaluation framework that separates answer correctness, output-level explanation groundedness, and thermal grounding for infrared visual questions. Using a Dual-LLM Consensus Judge with a preliminary human-anchor calibration check, we find that correct answers can still rely on weak or visible-light evidence; withholding the original infrared image and showing only a visible-like rendering erodes thermal grounding with little accuracy change; and this erosion is observed most strongly for more capable models but disappears when infrared remains available. We further propose Thermal-Grounded Feedback (TGF), a training-free feedback loop that diagnoses explanation-side failures and revises the explanation while preserving the selected answer. On local paired-input validation, TGF improves explanation-side grounding without changing answers. These findings suggest that future trustworthy MLLMs for infrared scene understanding should be evaluated and developed to produce thermally grounded explanations rather than merely accurate answers.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
- [1]
-
[2]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[3]
Infrared-LLaVA: Enhancing Understanding of Infrared Images in Multi-Modal Large Language Models , author =. 2024 , booktitle =
work page 2024
-
[4]
Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception
Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception , author =. 2026 , journal =. doi:10.48550/arXiv.2605.21882 , url =
work page Pith review arXiv doi:10.48550/arxiv.2605.21882 2026
-
[5]
ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery , author =. 2026 , journal =. doi:10.48550/arXiv.2602.14989 , url =
- [6]
-
[7]
Infrared and Visible Image Fusion with Language-Driven Loss in CLIP Embedding Space , author =. 2025 , booktitle =. doi:10.1145/3746027.3755182 , url =
arXiv 2025
-
[8]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Text-if: Leveraging semantic text guidance for degradation-aware and interactive image fusion , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Show all 29 references
-
[9]
2025 , journal =
TextFusion: Unveiling the Power of Textual Semantics for Controllable Image Fusion , author =. 2025 , journal =. doi:10.1016/j.inffus.2024.102790 , url =
2025
-
[10]
arXiv preprint arXiv:2401.00421 , year=
From text to pixels: A context-aware semantic synergy solution for infrared and visible image fusion , author=. arXiv preprint arXiv:2401.00421 , year=
- [11]
-
[12]
2026 , journal =
TSPFusion: Text-Guided Semantic Perception for Infrared and Visible Image Fusion , author =. 2026 , journal =. doi:10.1016/j.infrared.2025.106324 , url =
2026
-
[13]
2026 , journal =
TSDGFusion: A Text and Semantic Dual-Guided Model for Infrared and Visible Image Fusion , author =. 2026 , journal =. doi:10.1016/j.displa.2025.103266 , url =
2026
-
[14]
2026 , journal =
TOFusion: Text-Guided and Object-Aware Infrared and Visible Image Fusion , author =. 2026 , journal =. doi:10.1016/j.patcog.2026.113569 , url =
2026
-
[15]
Infrared Physics & Technology , pages=
FDFusion: Efficient text-guided infrared-visible image fusion via fine-tuned lightweight VLM and Dual-branch feature modeling , author=. Infrared Physics & Technology , pages=. 2026 , publisher=
2026
-
[16]
Displays , volume=
CLIPFusion: Infrared and visible image fusion network based on image--text large model and adaptive learning , author=. Displays , volume=. 2025 , publisher=
2025
-
[17]
2026 , booktitle =
CtrlFuse: Mask-Prompt Guided Controllable Infrared and Visible Image Fusion , author =. 2026 , booktitle =. doi:10.1609/aaai.v40i11.37884 , url =
2026 doi
-
[18]
2026 , booktitle =
ControlFuse: Instruction-guided Multi-Granularity Controllable Image Fusion , author =. 2026 , booktitle =. doi:10.1609/aaai.v40i16.38321 , url =
2026 doi
-
[19]
ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=
RIS-FUSION: Rethinking Text-Driven Infrared and Visible Image Fusion from the Perspective of Referring Image Segmentation , author=. ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2026 , organization=
2026
-
[20]
IEEE Transactions on Geoscience and Remote Sensing , year=
MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusion , author=. IEEE Transactions on Geoscience and Remote Sensing , year=
-
[21]
European Conference on Computer Vision , pages=
Depicting beyond scores: Advancing image quality assessment through multi-modal language models , author=. European Conference on Computer Vision , pages=. 2024 , organization=
2024
-
[22]
Proceedings of the AAAI conference on artificial intelligence , volume=
Exploring clip for assessing the look and feel of images , author=. Proceedings of the AAAI conference on artificial intelligence , volume=
-
[23]
IEEE Transactions on Pattern Analysis and Machine Intelligence , year=
EvaNet: Towards More Efficient and Consistent Infrared and Visible Image Fusion Assessment , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , year=
-
[24]
2025 , booktitle =
DEAL: Data-Efficient Adversarial Learning for High-Quality Infrared Imaging , author =. 2025 , booktitle =
2025
-
[25]
2026 , journal =
RTPSeg: A multi-modality dataset for LiDAR point cloud semantic segmentation assisted with RGB-thermal images in autonomous driving , author =. 2026 , journal =. doi:10.1016/j.isprsjprs.2026.01.008 , url =
2026 doi
-
[26]
European Conference on Computer Vision , pages=
Instructir: High-quality image restoration following human instructions , author=. European Conference on Computer Vision , pages=. 2024 , organization=
2024
-
[27]
International Conference on Learning Representations , volume=
An intelligent agentic system for complex image restoration problems , author=. International Conference on Learning Representations , volume=
-
[28]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Jarvisir: Elevating autonomous driving perception with intelligent image restoration , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[29]
International Journal of Computer Vision , volume=
Multi-agent image restoration , author=. International Journal of Computer Vision , volume=. 2026 , publisher=
2026
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.