Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-language models can learn sensor physics from 200 images per sensor plus diverse negative examples.

desk verdict A genuinely new sensor-VLM benchmark and a cheap fine-tuning recipe, but the headline gains are partly confounded by a shared GPT-4o generation pipeline for train and test. read the letter →

arxiv 2412.20750 v4 pith:UCFUNI4R submitted 2024-12-30 cs.CV

Sangyun Chung , Youngjoon Yu , Se Yeon Kim , Youngchae Chee , Yong Man Ro This is my paper · ORCID
classification cs.CV
keywords vision-languagemodelssensorunderstandingthermalimagingdepthX-raynegative-sampleoptimizationefficientfine-tuningVS-TDXbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that state-of-the-art vision-language models (VLMs) do not truly understand non-RGB sensor images; they fall back on RGB-trained priors, reading brightness in a thermal image as reflected light instead of emitted heat. To make the failure measurable, it introduces VS-TDX, a benchmark of about 10,000 question-answer pairs over thermal, depth, and X-ray images, organized into four perception tasks and two understanding tasks. To fix the failure, it proposes fine-tuning with the Diverse Negative Attributes (DNA) loss: alongside the correct sensor-matched answer, the model is trained to push down the probability of several plausible but sensor-mismatched answers. On three open VLMs fine-tuned with only 200 images per sensor, sensor-understanding scores rise roughly twenty to thirty points, and the gains hold without hurting standard RGB benchmarks. If correct, this means specialized sensor reasoning can be added to existing VLMs cheaply, without new architectures or large datasets.

What carries the argument

Two artifacts carry the argument. The VS-TDX benchmark supplies roughly 6,791 sensor images and about 10,160 multiple-choice question-answer pairs across thermal, depth, and X-ray, divided into Existence, Counting, Position, General Description, Contextual Understanding, and Sensor Understanding; the negative answers are deliberately sensor-mismatched, plausible but physically wrong. The DNA optimization loss is a preference objective: with reward $r(x,y)=\alpha\log\pi_\theta(y|x)/|y|$, it averages $\log\sigma(r(x,y^+)-r(x,y_i^-)-\beta)$ over $k$ negative answers, forcing the positive answer's reward to exceed each negative's by a margin $\beta$, and it is combined with standard cross-entropy supervised fine-tuning loss. This explicit widening of the probability gap between sensor-matched and sensor-mismatched answers is what counters the model's RGB priors, and fine-tuning is done with QLoRA so the base architecture stays untouched.

What would settle it

Train the same SAFT/DNA recipe on a fresh, fully human-authored multiple-choice benchmark for thermal, depth, and X-ray images where no large language model wrote the questions or answers; if the roughly twenty-to-thirty-point sensor-understanding gains mostly disappear, the reported improvement was in part an artifact of matching the question generator's style.

Watch

Extended reading notes

Core claim

The central claim is that the RGB-centric bias of current VLMs, not any architectural limit, is what blocks sensor understanding, and that this bias can be corrected with minimal data by contrasting correct answers against diverse wrong ones. Concretely, on the VS-TDX benchmark, DNA-optimized Phantom-7B improves Sensor Understanding from 50.6 to 86.4 on thermal and from 29.6 to 86.1 on depth; Qwen2-VL-7B and InternVL2-8B show similar lifts, and DNA outperforms plain supervised fine-tuning and preference methods such as DPO, IPO, and SimPO, especially when training data is cut to 50 images per sensor. The paper also reports that the fine-tuned model's performance on standard RGB benchmarks (MME, MMBench, MMMU, SEED) is essentially unchanged, and that it interprets thermal images from an unseen commercial camera in physically grounded terms, reading residual heat patterns rather than lighting effects.

Load-bearing premise

The VS-TDX benchmark's ground truth and the fine-tuning set come from the same GPT-4o-driven pipeline, with human validation on only 45 of about 10,000 questions, so the large measured gains could partly reflect the model adopting the generator's answer style instead of learning sensor physics.

Editorial extensions

If this is right

  • With 200 images per sensor and DNA fine-tuning, sensor understanding jumps from 50.6 to 86.4 on thermal and from 29.6 to 86.1 on depth for Phantom-7B, with similar gains on Qwen2-VL-7B and InternVL2-8B.
  • DNA optimization beats plain supervised fine-tuning and strong preference baselines such as DPO, IPO, and SimPO, and the margin holds when only 50 images per sensor are available.
  • VS-TDX scores expose a large machine-human gap: the best closed VLM reaches 73.3 percent on the sampled sensor-understanding questions while human participants reach 95.1 percent.
  • Fine-tuning with DNA does not degrade general RGB performance, so the sensor gains are not bought by overfitting to the sensor data.
  • The learned sensor reasoning transfers to an unseen thermal camera in a real-world test, suggesting the improvement is not tied to the exact training images.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because DNA is architecture-invariant and needs only a small contrast set, the same recipe is a natural candidate for other non-RGB modalities such as LiDAR, hyperspectral, and SAR, which the paper lists as future work but does not demonstrate.
  • The benchmark and the training set are generated by the same GPT-4o prompt pipeline, so part of the measured gain may be the model imitating the generator's phrasing rather than learning sensor physics; an independent human-authored benchmark would separate those.
  • The margin loss over diverse negatives is essentially a general learn-what-not-to-say objective, so it may benefit other reasoning domains where models fall back on surface priors, though nothing in the paper tests that.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that current VLMs lack deep understanding of non-RGB sensor images (thermal, depth, X-ray) and proposes two contributions: (1) VS-TDX, a new multiple-choice benchmark with about 10,160 QA pairs generated from about 6,791 public sensor images using GPT-4o with sensor-knowledge prompts; and (2) Sensor-Aware Attributes Fine-Tuning (SAFT) with Diverse Negative Attributes (DNA) optimization, a preference-based loss that contrasts one positive answer against k sensor-mismatched negatives, trained with QLoRA on a small set of 600 images. On VS-TDX, the authors report large gains after SAFT across three model families (Phantom-7B, Qwen2-VL-7B, InternVL2-8B), with ablations on the number of negatives k and the number of training images n, a comparison against DPO/IPO/SimPO, a standard-RGB benchmark sanity check, and a qualitative real-thermal-camera test. The central claim is that DNA optimization delivers consistent, architecture-invariant, resource-efficient improvements in sensor understanding.

Significance. If the reported gains reflect genuine sensor-physics understanding rather than style matching to the QA generator, the contribution is practically significant: the method is architecture-agnostic, requires only 200 images per modality, uses standard QLoRA, and shows consistent direction across three families and three modalities. The ablations on k and n and the baselines against DPO/IPO/SimPO are appropriate and informative, and the authors are honest about the limitation of focusing on three modalities. The real-world thermal test and RGB-benchmark sanity check are useful first steps toward external validity. However, the paper does not yet secure its load-bearing assumption that VS-TDX measures sensor understanding independently of the GPT-4o generation pipeline used to create both the training and test material, and the absence of uncertainty quantification weakens the 'consistently superior' claim.

major comments (4)
  1. [Sections III-B and IV-A; Appendix Fig. A3] The VS-TDX benchmark and the 3,600 fine-tuning pairs are generated with the same GPT-4o prompt template, including the same sensor-knowledge text, the same question-type instructions, and the same instruction to produce plausible sensor-mismatched negative answers. Because the headline Sensor Understanding gains in Table II (e.g., Phantom-7B depth 29.6 to 86.1) are measured exclusively on VS-TDX, part or most of the improvement may reflect the model learning generator-specific regularities such as distractor phrasing, correct-answer style, and predictable semantic cues rather than thermal, depth, or X-ray physics. This concern is reinforced by Table I, where GPT-4o, the generator, already obtains the highest scores on the benchmark. To support the stated generalization claim, the authors should evaluate on an independently constructed test set (for example, human-written QA pairs over the same source images, or questions generated by a different model and prompt) and show that SAFT gains persist there. Without such a test, the magnitude of the claimed improvement is not identifiable.
  2. [Tables I, II, IV, VI] All accuracy numbers are single-run point estimates with no error bars, confidence intervals, or multiple-seed averages. Several comparisons central to the 'consistently superior' claim are small in absolute terms: for example, Table VI shows Phantom-7B n=200 thermal Sensor Understanding of 85.1 for SFT+IPO versus 86.4 for the proposed method, and Table V shows MMBench of 79.8 versus 80.2. Since the benchmark is multiple-choice, binomial confidence intervals are easy to compute, and a three-seed average for the main Table II comparisons would quantify the robustness of the +20-point improvements. Without such quantification, the precision of the reported differences is overstated.
  3. [Abstract and Section V-A] The abstract and introduction describe VS-TDX as 'the first comprehensive, public benchmark,' but the manuscript provides no release URL, dataset download link, or availability statement for the benchmark itself. A benchmark that is not actually released cannot serve the claimed community role, and the paper's central evaluation cannot be independently reproduced. The authors should release the benchmark (including images and QA pairs with clear usage terms) or revise the 'public' claim in the abstract and text.
  4. [Section IV-A1 and Table II] The motivation for DNA optimization states that 'Supervised Fine-Tuning (SFT) was largely ineffective in improving model performance for vision sensor understanding,' but Table II shows that SFT alone produces substantial gains on the same benchmark, for example Phantom-7B depth Sensor Understanding rising from 29.6 to 65.3 and thermal from 50.6 to 78.5. This internal inconsistency should be resolved: either the claim should be tempered (e.g., SFT improves but plateaus and still underperforms DNA), or the empirical basis for 'largely ineffective' should be presented in the reported results. As written, the motivation for the method is overstated relative to its own data.
minor comments (5)
  1. [Appendix (Human Evaluation)] The phrase 'rewardede9.4/hr' contains a typo; it should read 'rewarded €9.4/hr' or equivalent.
  2. [Figure 3] The inner-ring sample counts are difficult to read because the labels overlap in the current rendering; consider placing counts outside the ring or using a table.
  3. [Section V-A] The hyperparameters alpha=2, beta=0.2, and k=3 are fixed without any sensitivity analysis for alpha or beta; a small sweep or at least a statement of why these values are robust would strengthen the resource-constrained claim.
  4. [Section V-B and Appendix Figure A1] The human evaluation is based on only 45 questions answered by 20 participants; this is a reasonable spot-check but should be described as such rather than as a general validation of the full 10,160-item benchmark.
  5. [Appendix (Input Prompts)] The prompt in Figure A3 asks the model to 'Make a strategy to create challenging questions when it is not known at the time that the image is from a <sensor_type> image,' which is ambiguous: it could mean the question writer should not rely on the sensor label, or that the question should be challenging to someone who does not know the sensor type. Clarifying this instruction would make the benchmark-generation procedure more transparent.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline VS-TDX gains may largely reflect GPT-4o generator matching: SAFT training labels and benchmark ground truth come from the same GPT-4o prompt pipeline, so the benchmark is not an independent sensor-understanding measure.

  1. other [Section IV, first paragraph of 'Proposed Method' (SAFT dataset construction); cross-ref Section III-B]
    "Specifically, we construct a supplementary dataset of 600 vision sensor images—200 for each sensor type (thermal, depth, and X-ray). For each of these images, we generate a corresponding set of positive and diverse negative question-answer pairs using the same methodology as the VS-TDX benchmark (Section III-B), resulting in 3,600 training samples. None of these images or question-answer pairs are included in the VS-TDX evaluation benchmark, ensuring an unbiased evaluation."

    The reduction is structural: VS-TDX ground-truth questions and answers are generated by ChatGPT/GPT-4o using a sensor-knowledge prompt that instructs creation of a correct answer plus plausible sensor-mismatched negatives (Section III-B, Fig. A3). The SAFT training targets are produced by 'the same methodology', i.e., the same GPT-4o prompt template and generator. The 'Sensor Understanding' score therefore measures agreement with the GPT-4o answer-generation distribution, and SAFT is explicitly trained to increase reward on exactly that distribution. Disjoint train/test images do not break the loop because the label-generating function is identical.

full rationale

No equation-level circularity is present in the DNA loss itself: Equations (1)-(5) define a standard preference-optimization objective with multiple negatives and a margin, and the method's improvement over SFT is a genuine empirical comparison. The self-citation of Phantom-7B [45] is not load-bearing because similar gains appear on Qwen2-VL-7B and InternVL2-8B. The RGB sanity check (Table V) and the qualitative real-thermal test are independent evidence but do not validate the core VS-TDX metric against the GPT-4o-generator confound. Because the fine-tuning data and the benchmark labels are generated by the same GPT-4o pipeline, the central generalization claim is partially circular: the measured sensor-understanding gains can be substantially attributed to distributional alignment with the benchmark's generator rather than to robust physical understanding. The result is not fully forced by definition, so a score of 6 rather than 8-10 is appropriate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central evaluation rests on the benchmark being a correct, unbiased measure of sensor understanding, plus several modeling choices (the reward definition, the margin, the negative-sample count) that are either assumed or hand-tuned. The paper introduces no new physical entities; its free parameters are training and loss constants selected without a fully independent validation loop.

free parameters (4)
  • alpha = 2.0
    Scaling factor in the reward r_phi(x,y) = alpha * log pi_theta(y|x) / |y| (Eq. 1). The text says the default is 1.0, but all experiments use alpha=2, and no sensitivity analysis is reported, so its value is a hand-chosen tuning constant.
  • beta = 0.2
    Margin in the DNA loss (Eq. 5) that forces the positive reward to exceed the average negative reward. The choice 0.2 is stated without a sensitivity study, making it a free parameter selected to make the method work.
  • k (number of negatives per instance) = 3
    The ablation in Table III sweeps k=1,2,3 and the best setting is then used for all models. Since the same benchmark evaluates both the ablation and the final numbers, the benchmark served as the model-selection signal for this parameter.
  • LoRA rank and learning rate = rank 256 for Phantom, 64 for Qwen/InternVL; lr 2e-5
    Per-model QLoRA configurations in Section V-A2 are chosen by the authors without a tuning protocol; they affect the fine-tuning outcome and are not justified by an external validation set.
assumptions (4)
  • domain assumption The generated VS-TDX question-answer pairs are factually correct and unambiguous.
    Section III-B: pairs are generated by ChatGPT/GPT-4o with sensor knowledge prompts, then reviewed by six human annotators; only 45 of about 10,160 questions were independently validated by 20 crowd workers. The correctness of the full benchmark is an unproven assumption that the central evaluation relies on.
  • domain assumption Multiple-choice accuracy on VS-TDX is a valid measure of sensor-specific understanding.
    The paper defines sensor understanding through its own benchmark and does not validate against an external sensor-understanding benchmark. If the questions can be answered from RGB priors or question wording, the metric does not measure sensor understanding.
  • domain assumption Average token log-probability, used as the reward in Eq. 1, is a valid proxy for answer quality.
    The DNA loss assumes the model's own per-token log-likelihood is a well-calibrated reward for correctness across answers of different lengths. No calibration evidence is provided.
  • domain assumption QLoRA fine-tuning on 3,600 samples does not degrade general VLM capabilities.
    Partially tested: Table V shows similar MME, MMBench, MMMU, and SEED scores on Phantom-7B before and after fine-tuning, but the check is one model, one run, without error bars.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking." pith.science (2026). https://pith.science/paper/UCFUNI4R

@misc{pith2026241220750,
  author       = {Pith},
  title        = {Pith review of: Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UCFUNI4R}},
  note         = {Machine review of arXiv:2412.20750}
}
read the original abstract

Large-scale Vision-Language Models (VLMs) have achieved notable progress in aligning visual inputs with text. However, their ability to deeply understand the unique physical properties of non-RGB vision sensor images remains limited. In this paper, we revisit and analyze these limitations and introduce a novel, cost-efficient paradigm that significantly advances sensor image understanding-without requiring extensive training data or any modifications to the existing VLM architectures. Specifically, we propose Sensor-Aware Attributes Fine-Tuning (SAFT) with the Diverse Negative Attributes (DNA) optimization, which leverages minimal sensor-specific data to enable robust learning of non-RGB characteristics and overcome RGB-centric biases inherent in current VLMs. In addition, we present VS-TDX-the first comprehensive, public benchmark designed to rigorously evaluate VLMs' sensor-specific understanding across diverse and realistic scenarios. Through extensive experiments on VLMs and various sensor modalities, we validate that our method consistently delivers superior performance and generalization under resource-constrained and architecture-invariant settings. Our approach provides a practical advance towards scalable deployment of VLMs in increasingly sensor-diverse real-world environments.

Figures

Figures reproduced from arXiv: 2412.20750 by the authors.

Figure 1
Figure 1. Examples of vision sensor-related questions and re [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed VS-TDX benchmark and its evaluation flow. The VS-TDX benchmark is designed to [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. VS-TDX benchmark data source distribution. The outer [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: VS-TDX benchmark dataset generation pipeline. Public thermal, depth, and X-ray datasets are fed into a large-scale [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The proposed framework for Sensor-Aware Attributes Fine-Tuning. A Vision-Language Model takes an input image [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Log probabilities of positive and negative answers [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Qualitative visualization examples comparing different Phantom-7B model variants (Base Model, SFT, and SAFT(Ours)) [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Real-world thermal images captured by the thermal camera, evaluated using different Phantom-7B variants (Base Model [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis

    cs.CV 2026-07 conditional novelty 6.5 of 10

    Fabricating appearance-varied images and modality contexts from RGB lets LMMs improve perception and understanding on real and synthetic non-RGB modalities without in-modality training.

Reference graph

Works this paper leans on

73 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Grounding language models to images for multimodal inputs and outputs,

    J. Y . Koh, R. Salakhutdinov, and D. Fried, “Grounding language models to images for multimodal inputs and outputs,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 17 283–17 300

    work page 2023

  2. [2]

    Harnessing multi-modal large language models for measuring and interpreting color differences,

    Z. Wang, Y . Long, Q. Jiang, C. Huang, and X. Cao, “Harnessing multi-modal large language models for measuring and interpreting color differences,”IEEE Transactions on Image Processing, 2025

    work page 2025

  3. [3]

    3vl: Using trees to improve vision-language models’ interpretability,

    N. Yellinek, L. Karlinsky, and R. Giryes, “3vl: Using trees to improve vision-language models’ interpretability,”IEEE Transactions on Image Processing, vol. 34, pp. 495–509, 2025

    work page 2025

  4. [4]

    Timechat: A time-sensitive multimodal large language model for long video understanding,

    S. Ren, L. Yao, S. Li, X. Sun, and L. Hou, “Timechat: A time-sensitive multimodal large language model for long video understanding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 313–14 323

    2024

  5. [5]

    Who, what and where: Composite-semantics instance search for story videos,

    J. Guo, A. Lu, Z. Wu, Z. Wang, and C. Liang, “Who, what and where: Composite-semantics instance search for story videos,”IEEE Transactions on Image Processing, 2025

    work page 2025

  6. [6]

    Exploring language hierarchy for video grounding,

    X. Ding, N. Wang, S. Zhang, Z. Huang, X. Li, M. Tang, T. Liu, and X. Gao, “Exploring language hierarchy for video grounding,”IEEE Transactions on Image Processing, vol. 31, pp. 4693–4706, 2022

    work page 2022

  7. [7]

    Adaptive spatio- temporal graph enhanced vision-language representation for video qa,

    W. Jin, Z. Zhao, X. Cao, J. Zhu, X. He, and Y . Zhuang, “Adaptive spatio- temporal graph enhanced vision-language representation for video qa,” IEEE Transactions on Image Processing, vol. 30, pp. 5477–5489, 2021

    work page 2021

  8. [8]

    mplug-docowl: Modularized multimodal large language model for document understanding,

    J. Ye, A. Hu, H. Xu, Q. Ye, M. Yan, Y . Dan, C. Zhao, G. Xu, C. Li, J. Tianet al., “mplug-docowl: Modularized multimodal large language model for document understanding,”arXiv preprint arXiv:2307.02499, 2023

    arXiv 2023

Show all 73 references
  1. [9]

    (2024) Hello gpt-4o

    OpenAI. (2024) Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. [Online]. Available: https://openai.com/index/hello-gpt-4o/

    2024

  2. [10]

    Gpt-driver: Learning to drive with gpt,

    J. Mao, Y . Qian, H. Zhao, and Y . Wang, “Gpt-driver: Learning to drive with gpt,”arXiv preprint arXiv:2310.01415, 2023

    2023 arXiv

  3. [11]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model,

    Z. Xu, Y . Zhang, E. Xie, Z. Zhao, Y . Guo, K.-Y . K. Wong, Z. Li, and H. Zhao, “Drivegpt4: Interpretable end-to-end autonomous driving via large language model,”IEEE Robotics and Automation Letters, 2024

    2024

  4. [12]

    Vlm-auto: Vlm-based autonomous driving assistant with human-like behavior and understanding for complex road scenes,

    Z. Guo, Z. Yagudin, A. Lykov, M. Konenkov, and D. Tsetserukou, “Vlm-auto: Vlm-based autonomous driving assistant with human-like behavior and understanding for complex road scenes,” 2024. [Online]. Available: https://arxiv.org/abs/2405.05885

    2024 arXiv

  5. [13]

    Langloc: Language-driven localization via formatted spatial description genera- tion,

    W. Shi, C. Chen, K. Li, Y . Xiong, X. Cao, and Z. Zhou, “Langloc: Language-driven localization via formatted spatial description genera- tion,”IEEE Transactions on Image Processing, 2025

    2025

  6. [14]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices,

    X. Chu, L. Qiao, X. Lin, S. Xu, Y . Yang, Y . Hu, F. Wei, X. Zhang, B. Zhang, X. Weiet al., “Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices,”arXiv preprint arXiv:2312.16886, 2023

    2023 arXiv

  7. [15]

    Trafficvlm: A controllable visual language model for traffic video captioning,

    Q. M. Dinh, M. K. Ho, A. Q. Dang, and H. P. Tran, “Trafficvlm: A controllable visual language model for traffic video captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2024, pp. 7134–7143

    2024

  8. [16]

    Pretraining vision-language model for difference visual question answering in longitudinal chest x-rays,

    Y . Cho, T. Kim, H. Shin, S. Cho, and D. Shin, “Pretraining vision-language model for difference visual question answering in longitudinal chest x-rays,” 2024. [Online]. Available: https: //arxiv.org/abs/2402.08966

    2024 arXiv

  9. [17]

    Physically grounded vision-language models for robotic manipulation,

    J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh, “Physically grounded vision-language models for robotic manipulation,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 12 462–12 469

    2024

  10. [18]

    Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,

    W. Huang, C. Wang, Y . Li, R. Zhang, and L. Fei-Fei, “Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,” 2024. [Online]. Available: https://arxiv.org/abs/2409. 01652

    2024

  11. [19]

    Aha: A vision-language- model for detecting and reasoning over failures in robotic manipulation,

    J. Duan, W. Pumacay, N. Kumar, Y . R. Wang, S. Tian, W. Yuan, R. Krishna, D. Fox, A. Mandlekar, and Y . Guo, “Aha: A vision-language- model for detecting and reasoning over failures in robotic manipulation,”

  12. [20]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. L...

    2023 arXiv

  13. [21]

    A3vlm: Actionable articulation-aware vision language model,

    S. Huang, H. Chang, Y . Liu, Y . Zhu, H. Dong, P. Gao, A. Boularias, and H. Li, “A3vlm: Actionable articulation-aware vision language model,” 2024. [Online]. Available: https://arxiv.org/abs/2406.07549

    2024 arXiv

  14. [22]

    (2024) Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy

    OpenGVLab. (2024) Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy. https://internvl.github.io/blog/ 2024-07-02-InternVL-2.0/. [Online]. Available: https://internvl.github. io/blog/2024-07-0...

    2024

  15. [23]

    Spatialbot: Precise spatial understanding with vision language models,

    W. Cai, I. Ponomarenko, J. Yuan, X. Li, W. Yang, H. Dong, and B. Zhao, “Spatialbot: Precise spatial understanding with vision language models,” arXiv preprint arXiv:2406.13642, 2024

    2024 arXiv

  16. [24]

    Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving,

    W. Wang, J. Xie, C. Hu, H. Zou, J. Fan, W. Tong, Y . Wen, S. Wu, H. Deng, Z. Liet al., “Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving,”arXiv preprint arXiv:2312.09245, 2023

    2023

  17. [25]

    Onellm: One framework to align all modalities with language,

    J. Han, K. Gong, Y . Zhang, J. Wang, K. Zhang, D. Lin, Y . Qiao, P. Gao, and X. Yue, “Onellm: One framework to align all modalities with language,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 584–26 595

    2024

  18. [26]

    Xraygpt: Chest radiographs summarization using medical vision-language models,

    O. Thawkar, A. Shaker, S. S. Mullappilly, H. Cholakkal, R. M. Anwer, S. Khan, J. Laaksonen, and F. S. Khan, “Xraygpt: Chest radiographs summarization using medical vision-language models,”arXiv preprint arXiv:2306.07971, 2023

    2023 arXiv

  19. [27]

    Llms can evolve continually on modality for x-modal reasoning,

    J. Yu, H. Xiong, L. Zhang, H. Diao, Y . Zhuge, L. Hong, D. Wang, H. Lu, Y . He, and L. Chen, “Llms can evolve continually on modality for x-modal reasoning,”arXiv preprint arXiv:2410.20178, 2024

    2024 arXiv

  20. [28]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

    2023

  21. [29]

    Llava-next: Improved reasoning, ocr, and world knowledge,

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024. [Online]. Available: https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  22. [30]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” inInternational conference on machine learning. PMLR, 2023, pp. 19 730–19 742

    2023

  23. [31]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms,

    Z. Cheng, S. Leng, H. Zhang, Y . Xin, X. Li, G. Chen, Y . Zhu, W. Zhang, Z. Luo, D. Zhaoet al., “Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms,”arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  24. [32]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. Heet al., “Minicpm-v: A gpt-4v level mllm on your phone,”arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  25. [33]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Geet al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,”arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  26. [34]

    Imagebind: One embedding space to bind them all,

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 180–15 190

    2023

  27. [35]

    Pandagpt: One model to instruction-follow them all,

    Y . Su, T. Lan, H. Li, J. Xu, Y . Wang, and D. Cai, “Pandagpt: One model to instruction-follow them all,”arXiv preprint arXiv:2305.16355, 2023

    2023 arXiv

  28. [36]

    Mme: A comprehensive evaluation benchmark for multimodal large language models,

    C. Fu, P. Chen, Y . Shen, Y . Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y . Wu, and R. Ji, “Mme: A comprehensive evaluation benchmark for multimodal large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2306.13394

    2024 arXiv

  29. [37]

    Seed-bench: Benchmarking multimodal llms with generative comprehension,

    B. Li, R. Wang, G. Wang, Y . Ge, Y . Ge, and Y . Shan, “Seed-bench: Benchmarking multimodal llms with generative comprehension,”arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  30. [38]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sunet al., “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  31. [39]

    Q-bench: A benchmark for general-purpose foundation models on low-level vision,

    H. Wu, Z. Zhang, E. Zhang, C. Chen, L. Liao, A. Wang, C. Li, W. Sun, Q. Yan, G. Zhaiet al., “Q-bench: A benchmark for general-purpose foundation models on low-level vision,”arXiv preprint arXiv:2309.14181, 2023

    2023 arXiv

  32. [40]

    Q-bench ++: A benchmark for multi-modal foundation models on low-level vision from single images to pairs,

    Z. Zhang, H. Wu, E. Zhang, G. Zhai, and W. Lin, “Q-bench ++: A benchmark for multi-modal foundation models on low-level vision from single images to pairs,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 10 404–10 418, 2024

    2024

  33. [41]

    Mmbench: Is your multi-modal model an all-around player?

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liuet al., “Mmbench: Is your multi-modal model an all-around player?” inEuropean Conference on Computer Vision. Springer, 2025, pp. 216–233. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 18

    2025

  34. [42]

    The reviewing of object files: Object-specific integration of information,

    D. Kahneman, A. Treisman, and B. J. Gibbs, “The reviewing of object files: Object-specific integration of information,”Cognitive psychology, vol. 24, no. 2, pp. 175–219, 1992

    1992

  35. [43]

    D. E. Broadbent,Perception and communication. Elsevier, 2013

    2013

  36. [44]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” 2023

    2023

  37. [45]

    Phan- tom of latent for large language and vision models,

    B.-K. Lee, S. Chung, C. W. Kim, B. Park, and Y . M. Ro, “Phan- tom of latent for large language and vision models,”arXiv preprint arXiv:2409.14713, 2024

    2024 arXiv

  38. [46]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wanget al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,”arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  39. [47]

    (2024) Claude 3.5 sonnet

    Anthropic. (2024) Claude 3.5 sonnet. https://www.anthropic.com/ news/claude-3-5-sonnet. [Online]. Available: https://www.anthropic. com/news/claude-3-5-sonnet

    2024

  40. [48]

    Prolific,

    “Prolific,” https://www.prolific.com/, Prolific, 2025

    2025

  41. [49]

    Rank analysis of incomplete block designs,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs,”Biometrika, 1952. [Online]. Available: https://api. semanticscholar.org/CorpusID:115965399

    1952

  42. [50]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  43. [51]

    Facenet: A unified embedding for face recognition and clustering,

    F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun. 2015, p. 815–823. [Online]. Available: http://dx.doi.org/10. 1109/CVPR.2015.7298682

    2015

  44. [52]

    Target-aware dual adversarial learning and a multi-scenario multi- modality benchmark to fuse infrared and visible for object detection,

    J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong, and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi- modality benchmark to fuse infrared and visible for object detection,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2022

  45. [53]

    thermal dogs and people x6ejw dataset,

    Roboflow, “thermal dogs and people x6ejw dataset,” https://universe. roboflow.com/object-detection/thermal-dogs-and-people-x6ejw, nov 2022, visited on 2023-03-29. [Online]. Available: https://universe. roboflow.com/object-detection/thermal-dogs-and-people-x6ejw

    2022

  46. [54]

    pet dataset,

    harang, “pet dataset,” https://universe.roboflow.com/harang/pet-kjl3x, jul 2024, visited on 2024-10-28. [Online]. Available: https://universe. roboflow.com/harang/pet-kjl3x

    2024

  47. [55]

    Thermal dataset,

    Visual, “Thermal dataset,” https://universe.roboflow.com/visual-iqhyh/ thermal-duv93, dec 2023, visited on 2024-10-22. [Online]. Available: https://universe.roboflow.com/visual-iqhyh/thermal-duv93

    2023

  48. [56]

    Hit-uav: A high-altitude infrared thermal dataset for unmanned aerial vehicle-based object detection,

    J. Suo, T. Wang, X. Zhang, H. Chen, W. Zhou, and W. Shi, “Hit-uav: A high-altitude infrared thermal dataset for unmanned aerial vehicle-based object detection,”Scientific Data, vol. 10, p. 227, 2023

    2023

  49. [57]

    animal-detection-flir-extra dataset,

    one, “animal-detection-flir-extra dataset,” https://universe.roboflow.com/ one-rphct/animal detection flir extra, apr 2023, visited on 2024-10-

    2023

  50. [58]

    chips-thermal-face-dataset,

    james cook, “chips-thermal-face-dataset,” https://www.kaggle.com/ datasets/kagglechip/chips-thermal-face-dataset, apr 2020, visited on 2024-10-28. [Online]. Available: https://www.kaggle.com/datasets/ kagglechip/chips-thermal-face-dataset

    2020

  51. [59]

    Available: https://universe.roboflow.com/one-rphct/animal detection flir extra

    [Online]. Available: https://universe.roboflow.com/one-rphct/animal detection flir extra

  52. [60]

    Diode: A dense indoor and outdoor depth dataset,

    I. Vasiljevic, N. Kolkin, S. Zhang, R. Luo, H. Wang, F. Z. Dai, A. F. Daniele, M. Mostajabi, S. Basart, M. R. Walter, and G. Shakhnarovich, “Diode: A dense indoor and outdoor depth dataset,” 2019. [Online]. Available: https://arxiv.org/abs/1908.00463

    2019 arXiv

  53. [61]

    Ifsod dataset,

    NJUST, “Ifsod dataset,” https://universe.roboflow.com/njust-oxpbo/ ifsod, aug 2023, visited on 2024-10-28. [Online]. Available: https://universe.roboflow.com/njust-oxpbo/ifsod

    2023

  54. [62]

    Diml/cvl rgb-d dataset: 2m rgb-d images of natural indoor and outdoor scenes,

    J. Cho, D. Min, Y . Kim, and K. Sohn, “Diml/cvl rgb-d dataset: 2m rgb-d images of natural indoor and outdoor scenes,” 2021. [Online]. Available: https://arxiv.org/abs/2110.11590

    2021 arXiv

  55. [63]

    Indoor segmen- tation and support inference from rgbd images,

    P. K. Nathan Silberman, Derek Hoiem and R. Fergus, “Indoor segmen- tation and support inference from rgbd images,” inECCV, 2012

    2012

  56. [64]

    X-ray baggage detection dataset,

    Malek, “X-ray baggage detection dataset,” https://universe.roboflow. com/malek-mhnrl/x-ray-baggage-detection, apr 2022, visited on 2024- 11-11. [Online]. Available: https://universe.roboflow.com/malek-mhnrl/ x-ray-baggage-detection

    2022

  57. [65]

    Unifesp x-ray body part classifier compe- tition,

    F. Eduardo Farina, “Unifesp x-ray body part classifier compe- tition,” 2022. [Online]. Available: https://kaggle.com/competitions/ unifesp-x-ray-body-part-classifier

    2022

  58. [66]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,”

  59. [67]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  60. [68]

    Simpo: Simple preference optimization with a reference-free reward,

    Y . Meng, M. Xia, and D. Chen, “Simpo: Simple preference optimization with a reference-free reward,”arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  61. [69]

    TE-SQ1 Thermal Camera,

    “TE-SQ1 Thermal Camera,” https://www.i3-thermalexpert.com/ products/te-sq1/, i3 Thermal Expert, 2025. SANGYUN CHUNGreceived the B.S. degree from Hanyang University, Seoul, South Korea, in 2023. He is currently working toward the Ph.D. degree in electronic engineering, Korea Ad...

    2025

  62. [70]

    A general theoretical paradigm to understand learning from human preferences,

    M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos, “A general theoretical paradigm to understand learning from human preferences,” 2023. [Online]. Available: https://arxiv.org/abs/2310.12036

    2023 arXiv

  63. [2019]

    Available: https://arxiv.org/abs/1711.05101

    [Online]. Available: https://arxiv.org/abs/1711.05101

    arXiv

  64. [2024]

    Available: https://arxiv.org/abs/2410.00371

    [Online]. Available: https://arxiv.org/abs/2410.00371

    arXiv

  65. [2025]

    His research interests include deep learning and multimodal large language mod- els

    He is currently working toward the Masters degree in electronic engineering, Korea Advanced Institute of Science and Technology (KAIST), Dae- jeon, South Korea. His research interests include deep learning and multimodal large language mod- els. JOURNAL OF LATEX CLASS FILES, V...

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.