{"id":"2ec2d34c-6310-4c4c-94e1-479ec83f3744","arxiv_id":"2608.09145","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Many correct answers from vision-language models on infrared images are not grounded in thermal evidence, and a training-free feedback loop can repair the explanations.","lead":"This paper introduces a way to score multimodal AI models on infrared images that checks not just whether the answer is correct, but whether the explanation relies on real thermal evidence. It finds many correct answers are justified with visible-light cues, and proposes a feedback loop that rewrites explanations without changing answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline numbers all flow through the Dual-LLM Consensus Judge's thermal scores, which rest on a 48-item human anchor with no released prompts or code; a setting-dependent judge bias could manufacture both the hidden-failure rate and the substitution erosion.","rationale":"I read the paper as making a measurement claim: correct infrared answers hide weak thermal grounding, and replacing the infrared image with a visible rendering erodes thermal grounding. The entire measurement rests on the Dual-LLM Consensus Judge's τ and h labels. The reader's weakest-assumption analysis identifies exactly this: the judge is validated on only 48 human items, with no larger validation or released artifacts, so all headline statistics inherit judge bias. My stress-test agrees and sharpens the mechanism. The concern is not that LLM judges are generally unreliable, but that this specific design can create a spurious setting effect: the judge sees or knows whether the input is IR-only or Translated-RGB, and the paper does not reveal the prompt, the images shown to the judge, or the selection of anchor items. If the judge penalizes thermal language more in the visible-rendering condition, Δτ_sub is inflated; if it over-flags any color/texture reference as visible-cue misuse, the strict groundedness rule propagates that inflation into F@C. The magnitude of the claimed effect is small enough that a 0.3-0.5 point label bias can explain a meaningful share of it. I also give the paper credit where it earned it: the capability-paradox discussion is appropriately hedged with baseline-coupled correlations and a closed-source control; the TGF mitigation is tested with answers fixed, and the paired preference evaluation provides an independent semantic check. Those parts are not my concern. My concern is squarely the validity of the dependent variable in the diagnostic experiments. The reader's CONDITIONAL verdict is appropriate: the paper is plausible and internally consistent, but the central measurement needs external validation. A larger human study and a label-swap audit would either confirm the effect or expose it as an evaluator artifact. Therefore I do not move the verdict; I would keep CONDITIONAL until the judge is audited.","tokens_in":11012,"tokens_out":5349,"duration_ms":57353,"concrete_test":"Run a controlled judge-bias audit. Take or generate 100 matched explanation pairs from same models and questions, where the explanation text is held fixed, and submit each explanation to the Dual-LLM Consensus Judge under two conditions: (a) labeled 'infrared image' and (b) labeled 'translated visible rendering'. If τ or h changes materially with the label alone, the judge is using context priors instead of explanation content. Independently, recruit three trained annotators to label 150 outputs stratified by model tier and setting, and report per-setting human-judge κ with confusion matrices. If the Translated-RGB κ is below 0.7, or if Δτ_sub recomputed on human-only labels differs from -0.689 by more than 20%, the erosion claim should be revised. Also release the judge prompts and the 48 anchor items so the calibration can be audited.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Everything the paper wants to show empirically is filtered through one instrument. F@C=0.631, HF=36.9%, and Δτ_sub=-0.689 are not raw model outputs; they are judged labels from a Dual-LLM Consensus Judge that was checked against a 48-item blind human anchor. The paper itself calls this calibration 'preliminary', and no confidence intervals, item-selection procedure, or judge prompts/code are given. The risk is not generic 'LLM judge unreliable'; it is setting-specific. Thermal grounding τ is defined as whether the explanation uses valid infrared evidence, but the judge evaluates each output in the context of the input image and setting, and the prompt is unreleased. In Translated-RGB, the image is a visible-like rendering; an LLM judge that already expects thermal claims to be unsupported there will assign lower τ regardless of explanation content. Conversely, 'visible-cue misuse' (h=1) is defined by the paper as visible-light evidence used to support a thermal conclusion, but in Translated-RGB color/texture cues are the only available evidence; whether this should even be scored as misuse is a judgment call the 48-item anchor does not resolve. Since Eq. (3) forces h to zero strict groundedness, a judge that over-flags visible cues in the translated condition inflates the claimed erosion. The same instrument also produces the τ differences behind r=-0.903. Without a second, larger, per-setting human validation, all headline measurements remain conditional on an unverified evaluator. This is not a reproducibility nit: a small setting-specific judge bias of 0.3-0.5 points is of the same magnitude as Δτ_sub=-0.689 and close to the difference between the mean IR-only τ=1.441 and the τ≥2 threshold used in strict groundedness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an explanation-aware evaluation framework for general-purpose MLLMs applied to infrared images, separating answer correctness, output-level explanation groundedness, and thermal grounding. Using IF-Bench, 11 models, and five input settings, it reports that correct IR-only answers frequently fail thermal grounding, that replacing the IR image with a translated visible-like rendering erodes thermal grounding with little accuracy change, and that this erosion correlates with model capability only when IR is withheld. It also introduces Thermal-Grounded Feedback (TGF), a training-free multi-agent loop, and reports local validation improving explanation-side metrics while keeping answers fixed.","tokens_in":11322,"tokens_out":5748,"duration_ms":54933,"significance":"If the measurement foundation holds, the framework fills a real gap: answer-only accuracy cannot distinguish a heat-grounded explanation from a visible-light shortcut, and the proposed metrics (EQ, F@C, HF, Δτ_sub) are concrete and well-defined. The authors are appropriately cautious about the capability association, explicitly labeling it descriptive and partially baseline-coupled, and they provide a transparent lexical baseline (LHJ). The central empirical claims, however, are entirely mediated by a Dual-LLM Consensus Judge that is validated on only a 48-item human anchor, with no released prompts or confidence intervals; this makes the headline numbers conditional on an unverified instrument. The TGF mitigation also contains a partial circularity that needs to be addressed.","major_comments":[{"comment":"The entire measurement chain for the paper's central numbers (F@C = 0.631, HF = 36.9%, Δτ_sub = −0.689, r = −0.903) runs through the Dual-LLM Consensus Judge, yet the only validation is a 48-item blind human anchor described as 'preliminary', with no confidence intervals, no item-selection procedure, and no released judge prompts. Because the claims are comparative across settings and models, a setting-dependent judge bias could manufacture both the hidden-failure rate and the substitution erosion; for example, a judge that already expects thermal claims to be unsupported in Translated-RGB will assign lower τ there regardless of explanation content. Please provide a larger per-setting human validation, report confidence intervals for τ, F@C, and Δτ_sub, and release the judge prompts and the judged outputs.","section":"Dual-LLM Consensus Judge, Table 1"},{"comment":"In Translated-RGB the model receives only a visible-like rendering, making color and texture cues the only available evidence. The visible-cue misuse flag h = 1 in Eq. (3) forces ϕ = 0 whenever such cues support a thermal conclusion, but whether this should be scored as misuse in a setting with no IR image is a judgment call that the 48-item anchor does not resolve. If the judge over-flags visible-language descriptions in the translated condition, the claimed substitution erosion Δτ_sub is inflated. Please validate the h labeling separately in Translated-RGB with per-setting human annotations, and report how Δτ_sub changes under alternative definitions of visible-cue misuse.","section":"Eq. (3) and Translated Visible Input"},{"comment":"The mitigation claim is evaluated with the same deterministic rubric that the feedback loop optimizes (the judge agent and Eq. (7)), and the independent preference check is also an LLM judge rather than human ratings. The statement that TGF is 'evaluated without circularity' is only partially true: the final evaluation judge differs from the loop judge, but the scoring criteria are the loop's own objectives. Please add a human-rated before/after sample and show that the F@C/EQ gains are not simply convergence of the loop to its own rubric.","section":"Thermal-Grounded Feedback, Table 5"},{"comment":"The thresholds τ≥2 and ε≥2 in Eq. (3) are hand-set, and no sensitivity analysis is reported. Since F@C, HF, and Δτ_sub all depend on these thresholds, the headline numbers may shift substantially with the chosen cutoffs. Please report how the main conclusions vary as the thresholds move (e.g., τ≥1 vs. τ≥3, ε≥1 vs. ε≥3) and state whether the qualitative findings survive.","section":"Eq. (3), Aggregate Metrics"},{"comment":"The paper occasionally conflates output-level groundedness with thermal grounding: F@C and HF are defined via ϕ in Eq. (3), which includes ε, h, and the γ gate, so Translated-RGB can show higher F@C than IR-only (0.661 vs. 0.631 in Table 2) while τ drops by 0.689. The abstract and conclusion should state clearly that the erosion finding is about thermal grounding τ, not the output-level groundedness captured by F@C and HF.","section":"Abstract, Experiments, Table 2"}],"minor_comments":[{"comment":"The caption refers to shaded cells indicating improvement over the same model's IR-only value, but the shading is not visible in the text; please use bold or arrows to mark improvements.","section":"Table 2"},{"comment":"The infrared-to-RGB translation model is cited but not described; please specify its architecture, training data, or output characteristics, since it is the key manipulation in Translated-RGB.","section":"Benchmark and input settings"},{"comment":"Please report the item distribution of the 48-item human anchor across the five input settings and the ten dimensions; this is needed to assess whether the calibration covers the settings where the headline comparisons are made.","section":"Table 1"},{"comment":"The thermal-gating indicator γ is introduced only as 'whether the question dimension is thermal-gated'; please define how γ is assigned to each question dimension.","section":"Eq. (3)"},{"comment":"The before-explanation quotes the input-format tag 'irplustranslatedrgb'; please clarify whether such filename or format cues were visible to models in all settings, since the model may be reading the input tag rather than the image content.","section":"Figure 6"},{"comment":"Several references are to 2026 proceedings or preprints; please verify that all cited works are publicly accessible or clearly marked as preprints.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's diagnostic idea is sound and the authors hedge the capability finding responsibly, but the empirical contribution is entirely mediated by an unreleased LLM judge validated on only 48 items. The reviewers should require release of judge prompts and judged outputs, a larger per-setting human validation, and confidence intervals before the central numbers can be accepted. The TGF mitigation also needs a human-rated check to rule out rubric-circularity. These are fixable within the scope of the manuscript, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a competent, well-scoped study with a genuinely useful decomposition. It separates answer correctness, output-level explanation groundedness, and thermal grounding for infrared MLLMs, and shows that a large share of correct answers rest on visible-light cues rather than thermal evidence. The substitution finding—that replacing the infrared image with a visible-like rendering erodes thermal grounding while barely changing accuracy—is a real and instructive result. The authors also deserve credit for being honest about the capability association: they explicitly call it descriptive, note the mechanical baseline-coupling in the r = −0.903 correlation, and show it weakens under partial correlation. That is more careful than most papers in this space.\n\nThe main soft spot is exactly where the reader put it. F@C = 0.631, HF = 36.9%, and Δτ_sub = −0.689 are not raw model outputs; they are labels from the Dual-LLM Consensus Judge, and the human anchor is 48 items. No prompts, no code, no item selection procedure, and no confidence intervals are provided. The stress-test worry that a setting-specific judge bias could inflate both the hidden-failure rate and the erosion is not a nit: the judge's visible-cue misuse rule is context-dependent, and the 48-item anchor does not resolve how the judge treats visible cues in Translated-RGB where color and texture are the only available evidence. That said, this is not a load-bearing flaw that invalidates the paper. The direction of the effects is internally consistent, the paper itself flags the calibration as preliminary, and the transparent lexical heuristic gives a lower-bound sanity check even if it under-detects. The TGF mitigation is also measured with a rubric that overlaps with the feedback loop's objectives, but they do add a separate paired LLM preference check, which helps. What is missing is external validation and release of artifacts.\n\nWho is this for? Anyone working on evaluation of multimodal models in non-RGB domains, and specifically the infrared MLLM community. It would get a serious referee from me. I'd ask the authors to release the judge prompts, the 48-item anchor set, and cluster-bootstrapped confidence intervals on the key deltas before publication. The framework deserves to be in the literature, but the headline numbers should be reproducible, not just plausible.\n\nSend it to review with a request for major revision on reproducibility, not on conceptual grounds.","headline":"A well-scoped and honest empirical study of infrared MLLM explanation grounding, but every headline number flows through an LLM judge validated on only 48 human examples, so the magnitudes should be treated as provisional until the artifacts ship.","tokens_in":11914,"tokens_out":1592,"would_cite":true,"duration_ms":18997,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Correct answers on infrared images often rest on visible-light cues, not heat evidence","keywords":["infrared image understanding","multimodal large language models","explanation grounding","thermal evidence","evaluation framework","feedback loop","visible-light substitution","grounded-at-correct rate"],"falsifier":"A larger human study (hundreds of items, multiple annotators) that shows the judge's thermal-grounding labels disagree substantially with human judgments on the same outputs, or that shows the judge rewards keywords like 'heat' and 'temperature' even when the explanation does not actually cite image evidence, would undermine the hidden-failure and erosion findings.","tokens_in":10796,"feed_emoji":"🔥","tokens_out":1327,"duration_ms":12892,"temperature":0.7,"pith_summary":"The paper argues that answer accuracy is an inadequate measure of whether a multimodal language model actually understands infrared imagery. It introduces an evaluation framework that separates whether an answer is correct from whether the explanation is grounded in thermal evidence, and shows that a large share of correct answers are not thermally grounded. The authors also demonstrate that when the original infrared image is replaced with a visible-like rendering, thermal grounding drops sharply while accuracy barely changes, and they propose a training-free feedback loop to revise explanations toward thermal evidence without changing answers.","feed_headline":"Right answers, wrong heat: AI explanations miss thermal evidence","feed_subtitle":"A new evaluation shows correct infrared answers often cite visible-light cues, and offers a fix.","key_machinery":"The central object is the structured output o = (a, e, V, u) and the strict output-level groundedness rule φ = (1−p_f) 1[τ≥2 ∨ γ=0] 1[ε≥2] (1−h), where τ is a 0–3 thermal-grounding score, ε is evidence grounding, h flags visible-cue misuse, and p_f is parse failure. This rule, combined with the Dual-LLM Consensus Judge, converts the abstract notion of 'thermal grounding' into a measurable label. The input manipulation Translated-RGB (replacing the infrared image with a visible-like rendering) isolates whether explanations rely on thermal evidence or on transferred visible-light priors. The TGF loop uses four role-specific feedback agents and a lexicographic selection score ψ(o) to repair explanations while keeping the answer fixed.","core_discovery":"The central claim is that general-purpose MLLMs often answer infrared questions correctly while justifying them with visible-light cues rather than thermal evidence. Using a structured output that separates answer, explanation, evidence list, and uncertainty, and scoring explanations with a Dual-LLM Consensus Judge, the paper reports a mean IR-only grounded-at-correct rate (F@C) of 0.631, meaning 36.9% of correct answers fail the output-level grounding criterion. Withholding the infrared image and showing only a translated visible rendering reduces mean thermal-grounding score from 1.441 to 0.752 (Δτ_sub = −0.689) while accuracy changes only from 0.716 to 0.700. This erosion is most pronounced for more capable models and disappears when infrared remains available alongside the visible view. The paper further proposes Thermal-Grounded Feedback (TGF), a multi-agent, training-free loop that improves explanation grounding on paired-input settings without altering answers.","pith_inferences":["The Dual-LLM Consensus Judge may be rewarding thermal-sounding language rather than true heat-based reasoning; a larger human-validated set would clarify this, but the paper's 48-item anchor leaves this as an open risk.","The negative correlation between capability and substitution erosion (r = −0.903) could be partly a ceiling effect: stronger models have higher baseline thermal scores, so they have more room to drop under substitution.","A testable extension would apply the same explanation-aware evaluation to other sensor modalities (e.g., depth maps, multispectral images) where a correct answer could likewise be justified by non-modal cues.","The TGF repair could be evaluated with a human study that directly compares revised explanations against original ones for thermal validity, rather than only LLM preference judgments."],"forward_implications":["If the central claim is correct, reporting answer accuracy alone on infrared benchmarks is misleading; evaluations should report EQ, F@C, and thermal-grounding score separately.","The observed erosion under modality substitution implies that improving accuracy by translating infrared to visible-like renderings can actively reduce the trustworthiness of explanations, even when accuracy is preserved.","The capability-associated pattern suggests that stronger models may be more prone to adopting visible-light priors when infrared is withheld, so capability scaling does not automatically improve thermal grounding.","Training-free, output-only feedback like TGF offers a practical mitigation for closed-source models where weight access and fine-tuning are impossible.","The framework generalizes to any domain where the evidence modality matters, such as medical imaging or remote sensing, where a correct answer must also be justified by the right sensory evidence."],"supporting_citations":[{"why":"Provides IF-Bench, the 680-question infrared VQA benchmark and the infrared-to-RGB translation model used for the Translated-RGB setting.","marker":"(Zhang et al. 2026)"},{"why":"Supplies the agentic restoration framework (perception, planning, expert execution, reflection) that inspires the multi-agent design of TGF.","marker":"(Zhu et al. 2025b)"},{"why":"Another agentic restoration system cited as inspiration for separating functional roles in the feedback loop.","marker":"(Lin et al. 2025)"},{"why":"Additional agentic restoration work that motivates the multi-agent structure of TGF.","marker":"(Jiang et al. 2026)"}],"fun_headline_variants":["Correct but clueless: AI infrared answers often miss thermal evidence","Wrong heat: How AI answers correctly but explains with visible light","Thermal grounding: AI gets answers right but explanations wrong on IR","New feedback loop fixes AI explanations on infrared without changing answers","AI right answers on infrared often rely on visible light, not heat"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Dual-LLM Consensus Judge's thermal-grounding scores are valid measurements of whether an explanation really uses thermal evidence.","fun_headline_variants_meta":{"raw":{"variants":["Correct but clueless: AI infrared answers often miss thermal evidence","Wrong heat: How AI answers correctly but explains with visible light","Thermal grounding: AI gets answers right but explanations wrong on IR","New feedback loop fixes AI explanations on infrared without changing answers","AI right answers on infrared often rely on visible light, not heat"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000428,"raw_usage":{"total_tokens":2185,"prompt_tokens":934,"completion_tokens":1251,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":1163}},"tokens_in":550,"tokens_out":1251,"duration_ms":8189,"temperature":1.0,"reasoning_tokens":1163,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:39:07.595155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A larger human study (hundreds of items, multiple annotators) that shows the judge's thermal-grounding labels disagree substantially with human judgments on the same outputs, or that shows the judge rewards keywords like 'heat' and 'temperature' even when the explanation does not actually cite image evidence, would undermine the hidden-failure and erosion findings.","supporting_citations":[{"cited_title":"2025 , journal =","cited_arxiv_id":null,"evidence_quote":"Another agentic restoration system cited as inspiration for separating functional roles in the feedback loop."}],"review_version":1}