{"id":"26b0622b-4fb1-48d0-824e-1965a6c459d0","arxiv_id":"2606.14741","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Language feedback can improve or worsen VLM visual grounding depending on the model, but the paper's 'thermal' findings rest on simple grayscale conversion rather than real thermal imagery.","lead":"This paper tests how well five AI models that read images and text can find people in photos degraded by fog, smoke, or a grayscale 'thermal' filter, and whether giving the model a text hint about its mistake helps it recover. It reports that such feedback helps one model but hurts another, and that cropping can hurt, not help, in the simulated thermal condition.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ2 language feedback is oracle-assisted: the prompt is generated from ground-truth error directions, so the +47.3%/-5.1% divergence does not demonstrate a deployable dynamic attention mechanism.","rationale":"The reader's verdict is REJECT, and I agree with that overall. The reader's weakest_assumption focuses on the thermal grayscale proxy, which is a legitimate external-validity concern. However, I identify a more fundamental internal-validity threat: the RQ2 feedback protocol uses ground-truth error directions. This is mentioned in the reader's rationale but not singled out as the weakest point. The oracle nature directly undermines the paper's central concept of 'language as dynamic attention' because in actual emergency use no such oracle exists. Even if the thermal condition were made realistic, the feedback experiments would not support the claimed dynamic-attention mechanism. The paper could be reframed as a controlled study of oracle-assisted correction on synthetic degradations, but as submitted it does not support the stated conclusions. Thus no change to the reader's REJECT verdict is needed, and the concern is partially aligned with the reader's list of issues.","tokens_in":12324,"tokens_out":4244,"duration_ms":47299,"concrete_test":"Re-run RQ2 on the same R1-failed thermal samples, but replace the oracle error-direction prompt with a non-oracle prompt, e.g., 'Your previous prediction was incorrect. Please re-examine the image and provide a corrected bounding box.' If the +47.3% Gemini gain and -5.1% Qwen2-VL change disappear or fall within noise, the central claim does not hold for realistic feedback. Additionally, run a version where the direction is chosen randomly or by a separate model with no access to ground truth, and compare the Gemini/Qwen divergence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that language feedback effectiveness is model-dependent rests on RQ2's iterative protocol. In §5.2.1, the R2 prompt is 'Your previous prediction was [too far left/right/too high/low]' — this is constructed from the known difference between the R1 predicted box and the ground-truth box. That is oracle information. In a real emergency deployment, no error direction is available because the ground truth is unknown. Therefore the Gemini +47.3% improvement and the Qwen2-VL -5.1% degradation measure each model's ability to follow an explicit correction command when the correct direction is supplied, not its ability to use language as unprivileged dynamic attention. The paper never labels this oracle-assisted, and the Limitations section does not mention it. Even if the thermal proxy were replaced with real sensor data, these results would not support 'language as dynamic attention for Emergency Visual Analysis'; they would only support 'oracle-steered correction' as a controlled upper bound. The model-dependent conclusion may be an artifact of how different models interpret directional corrections when the directions are perfectly correct, not of their general response to language feedback.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RefCOCO-Degraded, a benchmark of 3,811 COCO images transformed into four conditions (Clean, Fog, Smoke, and Thermal, where 'thermal' is a luma grayscale conversion). It evaluates several VLMs across four research questions: visual grounding under degradation (RQ1), iterative language feedback as a dynamic attention mechanism (RQ2), posture-classification VQA with and without cropping (RQ3), and hallucination behavior measured by an author-defined H-Score (RQ4). The headline claims are that thermal imagery causes severe grounding degradation (e.g., Qwen2-VL -63.6% IoU), that language feedback is model-dependent (+47.3% for Gemini vs -5.1% for Qwen2-VL in thermal), that cropping produces a 'Thermal Paradox' (-26% for Gemini), and that BLIP-2 is unsafe because its H-Score increases under degradation. The paper concludes that language feedback can serve as dynamic attention for some models but not others, and that model-condition validation is required.","tokens_in":12615,"tokens_out":5081,"duration_ms":54469,"significance":"The proposed benchmark and the multi-model comparison target a relevant and under-studied problem: VLM reliability in emergency conditions. The paper is transparent about some limitations, and the degradation pipeline is systematic. If the central findings were valid, they would offer useful evidence that language feedback is model- and condition-dependent, and the 'Thermal Paradox' could inform deployment choices. However, the two load-bearing pillars of the work—the thermal proxy and the oracle-assisted RQ2 protocol—undermine the central claims as stated. The paper provides no code, data, or statistical inference, and the H-Score is an unvalidated metric. The potential significance is real, but the current evidence does not establish it.","major_comments":[{"comment":"The R2 feedback prompt is constructed from the known difference between the R1 prediction and the ground-truth box ('Your previous prediction was [too far left/right/too high/low]'). This is oracle feedback: in a real emergency deployment, the error direction is unknown because the ground truth is not available. Therefore the central model-dependence result—Gemini +47.3% vs Qwen2-VL -5.1% in thermal—measures the models' ability to follow perfectly correct directional correction commands, not their ability to use language as an unprivileged dynamic-attention signal. The paper never labels RQ2 as oracle-assisted, and §8 does not mention this limitation. The main claim is not supported by the current protocol; a non-oracle feedback condition (e.g., 'your previous box was wrong; re-estimate the location' without giving the direction) or an explicit framing as an upper-bound study is needed.","section":"§5.2.1, RQ2 method"},{"comment":"The 'Thermal' condition is defined as I_thermal = 0.299R + 0.587G + 0.114B, i.e., a standard luma grayscale conversion. Real thermal imagery encodes emitted long-wave infrared radiation, not visible-light luminance; contrast, noise, thermal signatures, and object appearances are fundamentally different. All thermal-specific conclusions—the RQ1 collapse, the RQ2 +47.3% recovery, the Thermal Paradox (-26%), and the BLIP-2 hallucination increase—depend entirely on this grayscale proxy. The limitations sentence in §8 that synthetic degradation 'might not perfectly align' understates the severity: the experiments cannot be described as thermal without validation on real thermal sensor data, or the claims must be reframed as 'luminance/grayscale degradation.'","section":"§4.2, thermal condition definition"},{"comment":"RQ3 uses only 50 annotated samples per model, as acknowledged in §8. For n=50, the standard error of an accuracy estimate is at most about 7 percentage points (at 50% accuracy), so several reported differences—Qwen2-VL Clean -12%, Qwen2-VL Thermal +2%, LLaVA 0%—are within typical sampling noise. No confidence intervals or significance tests are given. The 'Thermal Paradox' (-26% for Gemini, from 94% to 68%) has a 95% CI width of roughly ±12 percentage points at this sample size, making the claimed 'catastrophic failure' borderline rather than definitive. The paper must either substantially increase the sample size, report uncertainty, and provide inferential statistics, or soften the 'catastrophic' language accordingly.","section":"§5.3/Table 5 and §8"},{"comment":"The H-Score is introduced as H = Fabrication + 0.5×Overconfidence - 0.3×Uncertainty, with weights that are justified neither theoretically nor empirically, and 'Fabrication' is detected via unspecified keyword matching. There is no validation, no inter-annotator agreement measure, and no comparison against established hallucination metrics such as CHAIR or POPE. The safety-relevant conclusion that BLIP-2 is 'dangerous' and 'unsuitable for emergency deployment' rests entirely on this unvalidated composite. The paper should validate the metric against existing benchmarks, report the component counts separately, present qualitative examples, or avoid strong safety claims until this is done.","section":"§5.4/Table 6, H-Score"}],"minor_comments":[{"comment":"The abstract and Section 3.2 state that five VLMs are evaluated, but Table 3 (RQ1) reports results only for Gemini, Qwen2-VL, and Kosmos-2. BLIP-2 and LLaVA are missing from the main grounding benchmark. Please clarify the per-RQ model coverage or add the missing rows.","section":"Abstract/Table 3"},{"comment":"The prompt template, decoding parameters, temperature, and exact model versions/checkpoints are not provided. This is essential for reproducibility, especially for proprietary models like Gemini where the version name is the only control.","section":"§5.1.2"},{"comment":"No confidence intervals, standard deviations, or significance tests are reported in any table. The paper should include these throughout, especially for the small-sample RQ3 and RQ2 gain percentages.","section":"General"},{"comment":"Equations are not numbered, which makes referencing difficult. Also, Figure 2 (coverage matrix) and Figure 5 (pipeline) are not described in the text; please add in-text references and captions that make the figures self-contained.","section":"§4.2/§5.3"},{"comment":"The H-Score formula uses an en-dash in the table but a minus sign in the text; please standardize the notation. Also, the description 'Higher H-Score = more dangerous' conflicts with the negative values in Table 6; clarify the intended interpretation.","section":"§3.3"},{"comment":"References [18] and [19] appear to be dated after the other citations; please verify the publication years and ensure all URLs/arXiv IDs are correct.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central empirical contribution is not supported by the current protocol: RQ2 uses oracle direction information, and the 'thermal' condition is a grayscale proxy. These are not local fixable issues; they require re-running the experiments under a non-oracle protocol and either real thermal data or a reframing of all thermal-related claims. The unvalidated H-Score and the 50-sample RQ3 further weaken the safety conclusions. I see a potentially interesting benchmark and a relevant question, but the present manuscript does not establish its stated claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline you need: the paper's central claim is not supported by its own protocol. The RefCOCO-Degraded benchmark is a real addition, and the empirical pattern of model-dependent responses to feedback is worth having, but the 'thermal' condition is grayscale and the feedback loop is oracle-driven. That makes the strong conclusions about emergency deployment premature.\n\nWhat's good: the dataset construction is straightforward and reproducible — 3,811 base images with fog, smoke, and grayscale variants and clean controls. The observation that language feedback helps Gemini (+47.3% in the thermal/grayscale setting) while hurting Qwen2-VL (-5.1%) is interesting and, as far as I can tell, not already in the literature. The 'Thermal Paradox' — cropping helps on RGB but hurts on grayscale — is a useful caution even if the thermal label is wrong. The paper is clearly written and the four research questions are well structured.\n\nWhere it falls apart: the thermal condition is not thermal. It's the standard luminance equation (0.299R+0.587G+0.114B), so every thermal-specific conclusion is actually about grayscale. That could be fine if the paper said 'grayscale,' but it sells itself as emergency thermal analysis. RQ2's feedback is oracle-assisted: the prompt tells the model its error direction, which requires knowing the ground truth. That measures ability to follow a correction command, not ability to use language as an unprivileged dynamic attention signal. This is never acknowledged, and the Limitations section doesn't mention it. Additionally, RQ3 has only 50 samples per model, the RQ1 table omits BLIP-2 and LLaVA without explanation, and none of the tables report error bars. The H-Score is an ad hoc composite with no validation.\n\nThe paper is not incoherent on its own terms — the authors likely believe the protocol is fair — but the gap between what they claim and what the experiments support is large. The right fix is to reframe the paper as a study of synthetic fog/smoke/grayscale robustness, explicitly label the feedback results as oracle-assisted upper bounds, and add uncertainty quantification if the sample sizes allow.\n\nMy call: this deserves a serious referee because the benchmark and the phenomenon of model-dependent feedback are worth discussing, and the review process could steer the authors toward an honest reframing. But as submitted, it's a reject with clear instructions for major revision.\n\nI'd bring it to a reading group as a case study in how easy it is to overclaim from synthetic degradation experiments.","headline":"A useful benchmark, but the 'thermal' findings are grayscale and the feedback loop is oracle-assisted — the paper overclaims what it tests.","tokens_in":13078,"tokens_out":3759,"would_cite":false,"duration_ms":40497,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that natural language feedback can serve as a dynamic attention mechanism for vision-language models in degraded emergency imagery, but only for some models and only in some conditions — Gemini gains 47.3% in thermal groun","keywords":["vision-language models","visual grounding","language feedback","dynamic attention","emergency visual analysis","thermal imagery","hallucination","RefCOCO-Degraded"],"falsifier":"Re-run the same four research questions on real thermal camera images (for example, from emergency response footage) instead of the synthetic luminance proxy. If real thermal images do not reproduce the +47.3% vs -5.1% feedback divergence, the Thermal Paradox, and BLIP-2's increased hallucination, then the central claims are artifacts of the chosen simulation.","tokens_in":12195,"feed_emoji":"🚨","tokens_out":3764,"duration_ms":35014,"temperature":0.7,"pith_summary":"The paper tries to establish whether iterative natural-language prompts can redirect a vision-language model's attention to recover accuracy lost under fog, smoke, and thermal degradation. It builds RefCOCO-Degraded, a 15,244-image benchmark, and finds the answer is conditional: language feedback yields large gains for Gemini in thermal conditions but hurts Qwen2-VL under the same protocol. It also reports a Thermal Paradox, where cropping helps RGB classification but causes catastrophic accuracy drops in thermal imagery, and identifies BLIP-2 as dangerously prone to increased hallucination under degradation. If these results hold, emergency AI designers cannot assume prompt-based attention or cropping strategies transfer across models or conditions; they must validate each model-condition pair.","feed_headline":"Language feedback lifts Gemini 47% but hurts Qwen2-VL in thermal tests","feed_subtitle":"A 15,244-image degraded benchmark shows recovery tricks don't transfer across vision-language models or conditions.","key_machinery":"The study's engine is RefCOCO-Degraded, a benchmark that takes 3,811 base images and applies four conditions: Clean, Fog, Smoke, and a Thermal proxy formed by grayscale luminance conversion. The central mechanism tested is iterative language feedback: when a first grounding prediction has IoU below 0.5, the model is told the error direction ('too far left', 'too high') and asked to correct its bounding box, up to three rounds. A second mechanism is explicit spatial attention via ground-truth cropping in posture classification. The paper also defines a Hallucination Score (H-Score) combining fabrication counts, overconfident language, and appropriate uncertainty.","core_discovery":"The central claim is that language feedback acts as a dynamic attention mechanism for some vision-language models but not others. Using a three-round iterative protocol applied only to samples where the initial grounding prediction failed (IoU below 0.5), Gemini improves by 47.3% in thermal conditions while Qwen2-VL degrades by 5.1% under the identical protocol. The paper also finds that cropping, a form of explicit spatial attention, is not universally beneficial: it improves RGB posture classification but reduces thermal accuracy by 26% for Gemini, because thermal images lose color and texture and therefore rely heavily on contextual cues. Finally, BLIP-2's hallucination score increases un","pith_inferences":["The paper's thermal conclusions rest on a luminance-proxy simulation rather than real infrared sensor data; if real thermal cameras produce different visual statistics, the Thermal Paradox and the model-dependent feedback gap could change or disappear.","The model-dependence of language feedback suggests that a model's ability to follow directional language may correlate with its instruction-following strength or grounding architecture, but the paper does not isolate that mechanism.","LLaVA's constant 64% accuracy across all conditions and cropping states hints at a possible majority-class collapse; a direct test would be to check its per-class performance on a balanced posture dataset.","The H-Score is a surface-language heuristic; whether it predicts real operational harm, such as false triage decisions, is an untested extension."],"forward_implications":["For thermal emergency scenarios, Gemini appears to benefit substantially from iterative language feedback, while Qwen2-VL does not; deploying the same feedback protocol across models is unsafe.","Cropping should be avoided in thermal imagery, at least for Gemini, because it removes contextual cues needed for posture inference; full-image processing is the safer default.","BLIP-2's increasing hallucination under degradation, with no uncertainty hedging, makes it a poor candidate for emergency use unless mitigation is added.","The RefCOCO-Degraded benchmark provides a reusable protocol for testing other vision-language models and degradation types before deployment.","All conclusions are tied to the tested model versions; newer releases may behave differently."],"fun_headline_variants":["Language feedback: Gemini +47% thermal, Qwen2-VL -5.1%","Thermal paradox: cropping helps RGB, fails in thermal vision","HorusEye: language feedback is model-dependent in emergency analysis","Dynamic attention via language: works for some VLMs, not others"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The thermal condition is modeled as grayscale luminance conversion (0.299R + 0.587G + 0.114B), not as a real thermal sensor; if true thermal imagery behaves differently, every thermal-specific conclusion could collapse.","fun_headline_variants_meta":{"raw":{"variants":["Language feedback: Gemini +47% thermal, Qwen2-VL -5.1%","Thermal paradox: cropping helps RGB, fails in thermal vision","HorusEye: language feedback is model-dependent in emergency analysis","Dynamic attention via language: works for some VLMs, not others"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000903,"raw_usage":{"total_tokens":3711,"prompt_tokens":718,"completion_tokens":2993,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":2926}},"tokens_in":462,"tokens_out":2993,"duration_ms":19750,"temperature":1.0,"reasoning_tokens":2926,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T04:49:34.987911+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same four research questions on real thermal camera images (for example, from emergency response footage) instead of the synthetic luminance proxy. If real thermal images do not reproduce the +47.3% vs -5.1% feedback divergence, the Thermal Paradox, and BLIP-2's increased hallucination, then the central claims are artifacts of the chosen simulation.","supporting_citations":[],"review_version":1}