{"id":"2426832a-3b1e-4770-be4e-73f47c9c2c10","arxiv_id":"2412.20750","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A small dataset of sensor images plus positive and negative answer examples markedly improves VLM performance on thermal, depth, and X-ray understanding without retraining the model architecture.","lead":"This paper builds a benchmark to test how well vision-language models understand thermal, depth, and X-ray images, and proposes a cheap fine-tuning trick with correct and deliberately wrong examples to teach them. The benchmark and method aim to make AI vision systems usable in sensor-rich real-world settings like robotics and security.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline VS-TDX gains may largely reflect GPT-4o style-matching: fine-tuning data and test questions are generated by the same pipeline (Sections III-B and IV), so an independent non-GPT-4o evaluation is required before accepting the generalization claim.","rationale":"The reader's weakest assumption identifies the same vulnerability, and I agree. I looked for stronger internal concerns, such as the DNA loss being a variant of DPO without a reference model or hyperparameters being selected on the test benchmark, but none is as load-bearing as the train/test generator coupling. The abstract and Table II assert a large, architecture-invariant improvement, yet Table II's only sensor-understanding measurement is VS-TDX. Section III-B states that every question/answer pair is generated with ChatGPT/GPT-4o (2024-08-06) using sensor-knowledge prompts, and Section IV states that the fine-tuning pairs are generated using the same methodology. The training distribution is therefore a small sample of the benchmark-generation distribution. A model fine-tuned on 3,600 such pairs can increase its probability of selecting GPT-4o's favored option pattern, which would raise accuracy even if no physical sensor knowledge is acquired. The fact that SFT alone helps less and preference learning helps more is consistent with learning a robust ranking over GPT-4o-generated answer options, not necessarily sensor understanding. The human study validates only 45 items, so it cannot rule out systematic style cues across 10,160 items. The benchmark is described as public but no release link or artifact is provided, which prevents an independent audit. The cross-model consistency and RGB sanity check are real supporting evidence, but they still evaluate on the same benchmark or on a non-sensor benchmark; only an external sensor-understanding evaluation can break the confound. The proposed test does this by removing GPT-4o from the evaluation side. If the test passes, the paper's central claim is credible; if it fails, the appropriate verdict is conditional on re-benchmarking with an independent ground-truth source. Because this concern does not overturn the paper's direction but does substantiate the reader's conditional verdict, I recommend UNCHANGED.","tokens_in":26944,"tokens_out":4680,"duration_ms":51644,"concrete_test":"Build a held-out evaluation set of about 600 images disjoint from both VS-TDX and the fine-tuning set, with questions and positive/negative answer sets written by human experts or generated by a different LLM such as Claude-3.5 or Gemini using identical sensor-knowledge prompts but no GPT-4o involvement. Evaluate Phantom-7B base, +SFT, and +SAFT on this set. If the SAFT-vs-base and SAFT-vs-SFT deltas on Sensor Understanding largely disappear, for example drop more than half of the Table II gap, then the VS-TDX gains are substantially pipeline-style artifacts and the central claim is overstated. If comparable gains persist, the concern is refuted. As a complementary check, shuffle option order per question on VS-TDX and confirm that accuracy is invariant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SAFT/DNA consistently delivers superior performance and generalization for sensor understanding (abstract; Table II). The load-bearing condition is that VS-TDX measures sensor understanding rather than familiarity with the GPT-4o QA-generation process. That condition is not secure. Both the benchmark (Section III-B) and the 3,600 fine-tuning pairs (Section IV) are produced with the same GPT-4o prompt template, including the same sensor-knowledge text and the same instruction to write a correct answer plus plausible sensor-mismatched negatives. Fine-tuning on such pairs can therefore exploit generator-specific regularities, such as distractor wording, correct-answer style, and predictable semantic cues, without learning thermal, depth, or X-ray physics. The only human validation is 20 crowd workers answering 45 sampled questions at 95.1%, and the benchmark is not actually released, so scale validity cannot be checked. Because the headline numbers, for example Phantom-7B depth Sensor Understanding rising from 29.6 to 86.1 in Table II, are measured entirely on VS-TDX, a shared-generation confound would materially overstate the method's benefit. The real-world thermal test and RGB sanity check are encouraging, but they are qualitative or non-sensor and do not establish that the core sensor-understanding metric generalizes across thermal, depth, and X-ray.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that current VLMs lack deep understanding of non-RGB sensor images (thermal, depth, X-ray) and proposes two contributions: (1) VS-TDX, a new multiple-choice benchmark with about 10,160 QA pairs generated from about 6,791 public sensor images using GPT-4o with sensor-knowledge prompts; and (2) Sensor-Aware Attributes Fine-Tuning (SAFT) with Diverse Negative Attributes (DNA) optimization, a preference-based loss that contrasts one positive answer against k sensor-mismatched negatives, trained with QLoRA on a small set of 600 images. On VS-TDX, the authors report large gains after SAFT across three model families (Phantom-7B, Qwen2-VL-7B, InternVL2-8B), with ablations on the number of negatives k and the number of training images n, a comparison against DPO/IPO/SimPO, a standard-RGB benchmark sanity check, and a qualitative real-thermal-camera test. The central claim is that DNA optimization delivers consistent, architecture-invariant, resource-efficient improvements in sensor understanding.","tokens_in":27134,"tokens_out":4796,"duration_ms":50001,"significance":"If the reported gains reflect genuine sensor-physics understanding rather than style matching to the QA generator, the contribution is practically significant: the method is architecture-agnostic, requires only 200 images per modality, uses standard QLoRA, and shows consistent direction across three families and three modalities. The ablations on k and n and the baselines against DPO/IPO/SimPO are appropriate and informative, and the authors are honest about the limitation of focusing on three modalities. The real-world thermal test and RGB-benchmark sanity check are useful first steps toward external validity. However, the paper does not yet secure its load-bearing assumption that VS-TDX measures sensor understanding independently of the GPT-4o generation pipeline used to create both the training and test material, and the absence of uncertainty quantification weakens the 'consistently superior' claim.","major_comments":[{"comment":"The VS-TDX benchmark and the 3,600 fine-tuning pairs are generated with the same GPT-4o prompt template, including the same sensor-knowledge text, the same question-type instructions, and the same instruction to produce plausible sensor-mismatched negative answers. Because the headline Sensor Understanding gains in Table II (e.g., Phantom-7B depth 29.6 to 86.1) are measured exclusively on VS-TDX, part or most of the improvement may reflect the model learning generator-specific regularities such as distractor phrasing, correct-answer style, and predictable semantic cues rather than thermal, depth, or X-ray physics. This concern is reinforced by Table I, where GPT-4o, the generator, already obtains the highest scores on the benchmark. To support the stated generalization claim, the authors should evaluate on an independently constructed test set (for example, human-written QA pairs over the same source images, or questions generated by a different model and prompt) and show that SAFT gains persist there. Without such a test, the magnitude of the claimed improvement is not identifiable.","section":"Sections III-B and IV-A; Appendix Fig. A3"},{"comment":"All accuracy numbers are single-run point estimates with no error bars, confidence intervals, or multiple-seed averages. Several comparisons central to the 'consistently superior' claim are small in absolute terms: for example, Table VI shows Phantom-7B n=200 thermal Sensor Understanding of 85.1 for SFT+IPO versus 86.4 for the proposed method, and Table V shows MMBench of 79.8 versus 80.2. Since the benchmark is multiple-choice, binomial confidence intervals are easy to compute, and a three-seed average for the main Table II comparisons would quantify the robustness of the +20-point improvements. Without such quantification, the precision of the reported differences is overstated.","section":"Tables I, II, IV, VI"},{"comment":"The abstract and introduction describe VS-TDX as 'the first comprehensive, public benchmark,' but the manuscript provides no release URL, dataset download link, or availability statement for the benchmark itself. A benchmark that is not actually released cannot serve the claimed community role, and the paper's central evaluation cannot be independently reproduced. The authors should release the benchmark (including images and QA pairs with clear usage terms) or revise the 'public' claim in the abstract and text.","section":"Abstract and Section V-A"},{"comment":"The motivation for DNA optimization states that 'Supervised Fine-Tuning (SFT) was largely ineffective in improving model performance for vision sensor understanding,' but Table II shows that SFT alone produces substantial gains on the same benchmark, for example Phantom-7B depth Sensor Understanding rising from 29.6 to 65.3 and thermal from 50.6 to 78.5. This internal inconsistency should be resolved: either the claim should be tempered (e.g., SFT improves but plateaus and still underperforms DNA), or the empirical basis for 'largely ineffective' should be presented in the reported results. As written, the motivation for the method is overstated relative to its own data.","section":"Section IV-A1 and Table II"}],"minor_comments":[{"comment":"The phrase 'rewardede9.4/hr' contains a typo; it should read 'rewarded €9.4/hr' or equivalent.","section":"Appendix (Human Evaluation)"},{"comment":"The inner-ring sample counts are difficult to read because the labels overlap in the current rendering; consider placing counts outside the ring or using a table.","section":"Figure 3"},{"comment":"The hyperparameters alpha=2, beta=0.2, and k=3 are fixed without any sensitivity analysis for alpha or beta; a small sweep or at least a statement of why these values are robust would strengthen the resource-constrained claim.","section":"Section V-A"},{"comment":"The human evaluation is based on only 45 questions answered by 20 participants; this is a reasonable spot-check but should be described as such rather than as a general validation of the full 10,160-item benchmark.","section":"Section V-B and Appendix Figure A1"},{"comment":"The prompt in Figure A3 asks the model to 'Make a strategy to create challenging questions when it is not known at the time that the image is from a <sensor_type> image,' which is ambiguous: it could mean the question writer should not rely on the sensor label, or that the question should be challenging to someone who does not know the sensor type. Clarifying this instruction would make the benchmark-generation procedure more transparent.","section":"Appendix (Input Prompts)"}],"recommendation":"major_revision","confidential_remarks":"The shared-pipeline confound between the training set and the VS-TDX evaluation is the central validity risk, and I do not think it can be dismissed without an external, non-GPT-4o test set. The other issues (no error bars, unreleased benchmark, SFT-motivation inconsistency) are fixable in revision. The paper's direction is credible and the method is simple enough to be reproducible, so I am not recommending rejection; I would like to see the revised version with an external evaluation before accepting the headline generalization claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is an applied, benchmark-plus-method paper, and the benchmark artifact is genuinely new. For thermal, depth, and X-ray images, there is no existing VLM reasoning benchmark of this scope, and the authors put together roughly ten thousand QA pairs across six tasks. The DNA loss is a straightforward extension of DPO—multi-negative, margin-augmented, averaging over k negatives—but the combination with a small sensor-specific fine-tuning set is reasonable and clearly described. The experiments are broad: ten VLMs on the benchmark, fine-tuning on 200 images per sensor for three model families, ablations on k and n, comparisons against DPO/IPO/SimPO, an RGB sanity check, and a test on a real thermal camera. That is real evidence.\n\nThe soft spot is exactly where the stress-test note lands. Both the benchmark questions and the fine-tuning pairs are generated by the same GPT-4o prompt template with the same sensor-knowledge text. Fine-tuning on such pairs can exploit generator regularities—distractor wording, correct-answer phrasing—without learning sensor physics. So the headline numbers, like Phantom-7B depth Sensor Understanding going from 29.6 to 86.1, are measured entirely on a benchmark that shares a generator with the training set. For Qwen2-VL the depth gain to 59.8 is still the weakest of the three modalities, which is consistent with some genuine but partial learning; still, the shared-pipeline design means the magnitude of the claimed improvement is not secure.\n\nThe other weaknesses are real but minor by comparison: no error bars or multiple seeds; the benchmark is called public but no release link or artifact appears in the paper; the human validation is only 45 questions at 95.1%; hyperparameters (alpha, beta, k, LoRA rank) were selected with the benchmark as the signal. None of these kill the direction. The method is cheap, architecture-agnostic, and the qualitative and RGB checks suggest it is not just overfitting to the benchmark style.\n\nBottom line: this is a paper for teams working on VLMs for robotics, autonomous driving, and medical imaging. It deserves a serious referee. I would send it out, but with a request that the benchmark and training data be released and that the authors add an evaluation on independently generated or human-curated sensor questions, plus multiple-seed variance. If that holds up, it is a solid applied contribution.","headline":"A genuinely new sensor-VLM benchmark and a cheap fine-tuning recipe, but the headline gains are partly confounded by a shared GPT-4o generation pipeline for train and test.","tokens_in":27785,"tokens_out":2308,"would_cite":true,"duration_ms":24543,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision-language models can learn sensor physics from 200 images per sensor plus diverse negative examples.","keywords":["vision-language models","sensor understanding","thermal imaging","depth imaging","X-ray imaging","negative-sample optimization","efficient fine-tuning","VS-TDX benchmark"],"falsifier":"Train the same SAFT/DNA recipe on a fresh, fully human-authored multiple-choice benchmark for thermal, depth, and X-ray images where no large language model wrote the questions or answers; if the roughly twenty-to-thirty-point sensor-understanding gains mostly disappear, the reported improvement was in part an artifact of matching the question generator's style.","tokens_in":26665,"feed_emoji":"🌡️","tokens_out":5395,"duration_ms":50363,"temperature":0.7,"pith_summary":"The paper argues that state-of-the-art vision-language models (VLMs) do not truly understand non-RGB sensor images; they fall back on RGB-trained priors, reading brightness in a thermal image as reflected light instead of emitted heat. To make the failure measurable, it introduces VS-TDX, a benchmark of about 10,000 question-answer pairs over thermal, depth, and X-ray images, organized into four perception tasks and two understanding tasks. To fix the failure, it proposes fine-tuning with the Diverse Negative Attributes (DNA) loss: alongside the correct sensor-matched answer, the model is trained to push down the probability of several plausible but sensor-mismatched answers. On three open VLMs fine-tuned with only 200 images per sensor, sensor-understanding scores rise roughly twenty to thirty points, and the gains hold without hurting standard RGB benchmarks. If correct, this means specialized sensor reasoning can be added to existing VLMs cheaply, without new architectures or large datasets.","feed_headline":"Two hundred images per sensor unlock vision-language sensor reasoning","feed_subtitle":"A benchmark plus negative-example tuning lifts sensor understanding by as much as 30 points across three model families.","key_machinery":"Two artifacts carry the argument. The VS-TDX benchmark supplies roughly 6,791 sensor images and about 10,160 multiple-choice question-answer pairs across thermal, depth, and X-ray, divided into Existence, Counting, Position, General Description, Contextual Understanding, and Sensor Understanding; the negative answers are deliberately sensor-mismatched, plausible but physically wrong. The DNA optimization loss is a preference objective: with reward $r(x,y)=\\alpha\\log\\pi_\\theta(y|x)/|y|$, it averages $\\log\\sigma(r(x,y^+)-r(x,y_i^-)-\\beta)$ over $k$ negative answers, forcing the positive answer's reward to exceed each negative's by a margin $\\beta$, and it is combined with standard cross-entropy supervised fine-tuning loss. This explicit widening of the probability gap between sensor-matched and sensor-mismatched answers is what counters the model's RGB priors, and fine-tuning is done with QLoRA so the base architecture stays untouched.","core_discovery":"The central claim is that the RGB-centric bias of current VLMs, not any architectural limit, is what blocks sensor understanding, and that this bias can be corrected with minimal data by contrasting correct answers against diverse wrong ones. Concretely, on the VS-TDX benchmark, DNA-optimized Phantom-7B improves Sensor Understanding from 50.6 to 86.4 on thermal and from 29.6 to 86.1 on depth; Qwen2-VL-7B and InternVL2-8B show similar lifts, and DNA outperforms plain supervised fine-tuning and preference methods such as DPO, IPO, and SimPO, especially when training data is cut to 50 images per sensor. The paper also reports that the fine-tuned model's performance on standard RGB benchmarks (MME, MMBench, MMMU, SEED) is essentially unchanged, and that it interprets thermal images from an unseen commercial camera in physically grounded terms, reading residual heat patterns rather than lighting effects.","pith_inferences":["Because DNA is architecture-invariant and needs only a small contrast set, the same recipe is a natural candidate for other non-RGB modalities such as LiDAR, hyperspectral, and SAR, which the paper lists as future work but does not demonstrate.","The benchmark and the training set are generated by the same GPT-4o prompt pipeline, so part of the measured gain may be the model imitating the generator's phrasing rather than learning sensor physics; an independent human-authored benchmark would separate those.","The margin loss over diverse negatives is essentially a general learn-what-not-to-say objective, so it may benefit other reasoning domains where models fall back on surface priors, though nothing in the paper tests that."],"forward_implications":["With 200 images per sensor and DNA fine-tuning, sensor understanding jumps from 50.6 to 86.4 on thermal and from 29.6 to 86.1 on depth for Phantom-7B, with similar gains on Qwen2-VL-7B and InternVL2-8B.","DNA optimization beats plain supervised fine-tuning and strong preference baselines such as DPO, IPO, and SimPO, and the margin holds when only 50 images per sensor are available.","VS-TDX scores expose a large machine-human gap: the best closed VLM reaches 73.3 percent on the sampled sensor-understanding questions while human participants reach 95.1 percent.","Fine-tuning with DNA does not degrade general RGB performance, so the sensor gains are not bought by overfitting to the sensor data.","The learned sensor reasoning transfers to an unseen thermal camera in a real-world test, suggesting the improvement is not tied to the exact training images."],"supporting_citations":[{"why":"Supplies the direct preference optimization baseline and the conceptual template for DNA optimization.","marker":"[50]"},{"why":"Provides the Bradley-Terry preference model used to define the probability that a positive answer beats a negative one.","marker":"[49]"},{"why":"Provides the triplet-loss margin idea that DNA adapts as the separation margin beta.","marker":"[51]"},{"why":"Supplies QLoRA, the parameter-efficient fine-tuning method that keeps the base VLM architecture unchanged.","marker":"[65]"},{"why":"GPT-4o is the generator of both benchmark and training question-answer pairs and the strongest closed-source baseline in the evaluation.","marker":"[9]"},{"why":"Qwen2-VL-7B is one of the three open model families used to demonstrate that DNA gains transfer across architectures.","marker":"[33]"},{"why":"InternVL2-8B is one of the three fine-tuning testbeds and the source of the failure examples motivating the benchmark.","marker":"[22]"},{"why":"Phantom-7B is the main fine-tuning testbed across the ablations and the model used in the real-world thermal camera test.","marker":"[45]"}],"fun_headline_variants":["Fifty sensor images correct VLM RGB bias on thermal and depth","Sensor-aware fine-tuning: 50 images beat DPO and other preference methods","RGB-centric bias is the real blocker for VLM sensor understanding","Minimal sensor data overcomes RGB bias, lifts thermal scores by 36 points","Data-efficient sensor tuning: 50 images, no architecture change, beat DPO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The VS-TDX benchmark's ground truth and the fine-tuning set come from the same GPT-4o-driven pipeline, with human validation on only 45 of about 10,000 questions, so the large measured gains could partly reflect the model adopting the generator's answer style instead of learning sensor physics.","fun_headline_variants_meta":{"raw":{"variants":["Fifty sensor images correct VLM RGB bias on thermal and depth","Sensor-aware fine-tuning: 50 images beat DPO and other preference methods","RGB-centric bias is the real blocker for VLM sensor understanding","Minimal sensor data overcomes RGB bias, lifts thermal scores by 36 points","Data-efficient sensor tuning: 50 images, no architecture change, beat DPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000908,"raw_usage":{"total_tokens":3897,"prompt_tokens":935,"completion_tokens":2962,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":2864}},"tokens_in":551,"tokens_out":2962,"duration_ms":20538,"temperature":1.0,"reasoning_tokens":2864,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:12:22.002574+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same SAFT/DNA recipe on a fresh, fully human-authored multiple-choice benchmark for thermal, depth, and X-ray images where no large language model wrote the questions or answers; if the roughly twenty-to-thirty-point sensor-understanding gains mostly disappear, the reported improvement was in part an artifact of matching the question generator's style.","supporting_citations":[{"cited_title":"Rank analysis of incomplete block designs,","cited_arxiv_id":null,"evidence_quote":"Provides the Bradley-Terry preference model used to define the probability that a positive answer beats a negative one."},{"cited_title":"Unifesp x-ray body part classifier compe- tition,","cited_arxiv_id":null,"evidence_quote":"Supplies QLoRA, the parameter-efficient fine-tuning method that keeps the base VLM architecture unchanged."},{"cited_title":"(2024) Hello gpt-4o","cited_arxiv_id":null,"evidence_quote":"GPT-4o is the generator of both benchmark and training question-answer pairs and the strongest closed-source baseline in the evaluation."},{"cited_title":"(2024) Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy","cited_arxiv_id":null,"evidence_quote":"InternVL2-8B is one of the three fine-tuning testbeds and the source of the failure examples motivating the benchmark."}],"review_version":1}